A compact fiber-optic vector magnetometer based on directional scattering between polarized plasmon waves and ferro-magnetic nanoparticles is demonstrated. The sensor configuration reported in this work uses a short section of tilted fiber Bragg grating (TFBG) coated with a nanometer scale gold film and packaged with a magnetic fluid (Fe3O4) inside a capillary. The transmission spectrum of the sensor provides a fine comb of narrowband resonances that overlap with a broader absorption of the surface plasmonresonance (SPR). The wavelength of the SPR attenuation in transmission shows high sensitivity to slight perturbations by magnetic fields, due to the strong directional scattering between the SPR attenuated cladding modes and the magnetic fluid near the fiber surface. Both the orientation (2 nm/deg) and the intensity (1.8 nm/mT) of magnetic fields can be determined unambiguously from the TFBG spectrum. Temperature cross sensitivity can be referenced out by monitoring the wavelength of the core mode resonance simultaneously.