Abstract: Due to differences between grey distribution in infrared polarization images were large and feature information was not obvious, accuracy of general registration algorithms based on region or features were hard to satisfy requirement of infrared polarization information analysis. According to matrix rank as a measure of image similarity, a patch-registration method based on matrix recovery theory was proposed. Transform matrix was composed by image patches without registration. The transform matrix could be decomposed to a low-rank matrix and a sparse matrix, objective function was sum of nuclear norm of the transformation low-rank matrix and 1 norm of the transformation sparse matrix. Registration parameter was achieved by augmented Lagrange multiplier method when the value of object function was the smallest. Finally, registration result was acquired from registration parameters in each region which had been averaged. The experiment result shows that the algorithm is not sensitive to noise. Error of its registration parameters is less than 0.02 pixel.