Abstract: In order to improve the accuracy of the multi-sensor automatic target recognition algorithm,based on line group matching, an image matching algorithm using linear group geometric primitives was proposed. Line group was used to represent the target. Every three lines in line group can compose a triangle. An affine invariant, which consisted of the ratio of these triangles' area, was considered as a distinct feature to describe the line group. The best matching line group was found among the candidate line groups in real-time infrared image. The corresponding points between target and real-time image can be obtained through the matching lines. Then the location of the target can be computed through affine transformation. Finally, the position of the target in the real-time map was detected by the same-named point; in addition, a method of linear purification in the real-time infrared image was proposed. The extracted primary lines provided a condition which ensured the effectiveness and efficiency of the proposed recognition algorithm. Experimental result shows that the proposed algorithm has a high matching accuracy in multi-sensor target recognition.