Abstract: In order to ensure the interpolation precision of the high precision photoelectric rotary encoder in the harsh working conditions, the real-time compensation processing system based on digital potentiometer was designed. On the basis of Moire fringe photoelectric signal mathematical model, the spatial distribution of the subdivision error due to signal amplitude deviation and DC drift was revealed. Error discipline and calculation formula was obtained. From several production aspects, such as encoder opto-mechanical alignment, uniformity of the code disk and photosensitive element debugging, fundamental characteristic of the encoder photoelectric signal subdivision error was pointed out. The resolution of high precision photoelectric encoder was constrained. From the Moire fringe signal form outputted by encoder photosensitive element, digital potentiometer look-up table with 0.1 resolution was constructed and key algorithm of real-time compensation was designed. A 23 bit encoder was selected as the experimental object and the compensation processing system was tested at -40-60 ℃. The experimental results showed that the DC drift was less than 1.2% and amplitude difference was less than 2% and the automatic compensation time was about 3 s. It can meet the encoder resolution(0.154) and real-time requirements. The method can be applied in the encoder system and improve the environmental adaptability and angle measurement reliability.