Finding correspondences between two related feature point sets is a basic task in computer vision and pattern recognition. In this paper, we present a novel method for point pattern matching via spectral graph analysis. In particular, we aim to render the spectral matching algorithm more robust for positional jitter and outlier. A local structural descriptor, namely the spectral context, is proposed to describe the attribute domain of point sets, which is fundamentally different from the previous methods. Furthermore, the approximate distance order is defined and employed as the metric for geometric consistency of neighboring points in this work. By combining these two novel ingredients, we formulate feature point set matching as an optimization problem with one-to-one constraints. The correspondences are then obtained by maximizing the given objective function via the technique of probabilistic relaxation. Comparative experiments conducted on both synthetic and real data demonstrate the effectiveness of the proposed method, especially in the presence of positional jitter and outliers.