In this chapter, the authors present an efficient retrieval technique for human face images based on a bag of facial features. A visual vocabulary is built beforehand using an invariant descriptor computed on detected image regions. The vocabulary is refined in two ways to make the retrieval system more efficient. Firstly, the visual vocabulary is minimized by only using facial features selected on face regions which are detected by an accurate face detector. Secondly, three criteria, namely Inverted-Occurrence-Frequency Weights, Average Feature Location Distance and Reliable Nearest-Neighbors, are calculated in advance to make the on-line retrieval procedure more efficient and precise. The proposed system is experimented on the Caltech Human Face Database. The results show that this technique is very effective and efficient on face image retrieval.