Optical vortex beams carrying orbital angular momentum are being widely investigated for boosting the information capacity of communication systems by virtue of their unbounded state space for spatial mode division multiplexing or high-dimensional encoding. Vortex mode sorters are a critical component in such systems and a spiral transformation scheme working in the paraxial regime for vortex mode sorting has been proposed very recently to achieve high modal resolution with unity efficiency. Here we modify the spiral transformation scheme by developing a phase corrector in the nonparaxial regime, which is confirmed to have a more accurate phase-correcting function in implementing the spiral transformation and eventually lead to a better mode separation, especially in the case of a compact vortex mode sorter. Based on the above principle, a home-made compact vortex mode sorter is demonstrated by integrating the two phase elements implementing the spiral transformation onto the opposite sides of a thin quartz plate as flat diffractive optical elements, which achieves high-resolution and high-efficiency vortex mode sorting as expected. The modified optical spiral transformation and the compact design of vortex mode sorter based on this scheme therefore provide a much better tool for the effective mode separation, which can enable new applications in both classical and quantum information systems based on vortex modes.