Abstract: For infrared image which is characterized by low SNR and serious background and noise disturbance, according to the correlation of moving target frame-to-frame and the irrelevance of noise in sequential images, based on Open Source Computer Vision Library, a detection algorithm for small target was proposed and the detected target was tracked. By using energy accumulation method, background of image was got and then removed from original image, image SNR was improved. Noise was removed via the correlation of target frame-to-frame and the movement information. At last, the detected target was tracked by Kalman filter algorithm. The experimental results show that this detection algorithm can effectively find small moving target from sequential images, the tracking algorithm can track in real time and forecast the position accurately when target is covered.