A novel all-fiber Mach–Zehnder interferometer (MZI), which consists of lateral core fusion splicing of a short section of side-polished single mode fiber (SMF) between two SMFs was proposed and demonstrated. A simple fiber side-polished platform was built to control the side polished depth through a microscope. The sensitivity of the fiber MZI structure to the surrounding refractive index (RI) can be greatly improved with the increase of the side-polished depth, but has no effect on the temperature sensitivity. The sensor with a polished depth of 44.2 μm measured RI sensitivity up to -118.0 nm/RIU (RI unit) in the RI range from 1.333 to 1.387, which agrees well with simulation results by using the beam propagation method (BPM). In addition, the fiber MZI structure also can achieve simultaneous measurement of both RI and temperature. These results show its potential for use in-line fiber type sensing application.