Abstract: As the extensive application of nanotechnology and the deepened cognition of nanometer measurement, the measuring technologies of displacement with nanometer resolution were paid extensively attention to. On the basis of researching in working principle of the reflex-grating measurement, a miniature single-grating displacement measuring system with nanometer resolution was designed as well as realized. The overall design, optical structure, and software algorithm of the system were elaborated. In the end, comparative experiments were done with the help of auxiliary instruments which mainly included a capacitive displacement sensor named ASP-10-ILA. Experimental results show that the system can measure accurately when the two signals are not absolutely orthogonal, and the resolution of the system is 1 nm theoretically; the mean values of the measuring system deviates from the reference values by a maximum of 118 nm, and deviates from the fitting-line by no more than 100 nm when the grating displacement experiments are done within the range of capacitive displacement sensor; the deviations of the results are less than 5 ppm when the grating has displacement larger than 10 mm.