This research proposes a facial expression recognition system using evolutionary Particle Swarm Optimization (PSO)-based feature optimization. The system first employs modified Local Binary Patterns, which conduct horizontal and vertical neighbourhood pixel comparison, to generate a discriminative initial facial representation. Then, a PSO variant embedded with the concept of a micro Genetic Algorithm (mGA), called mGA-embedded PSO, is proposed to perform feature optimization. It incorporates a non-replaceable memory, a small-population secondary swarm, a new velocity updating strategy, a sub-dimension based in-depth local facial feature search, and a cooperation of local exploitation and global exploration search mechanism to mitigate the premature convergence problem of conventional PSO. Multiple classifiers are used for recognizing seven facial expressions. Based on a comprehensive study using within- and cross-domain images from the CK+ and MMI benchmark databases, respectively, the empirical results indicate that our proposed system outperforms other state-of-the-art PSO variants, conventional PSO, classical GA, and other related facial expression recognition models reported in the literature by a significant margin.