Abstract: In order to solve the problem that the target template of standard Mean Shift tracking can only be built from a single image, and difficult to update, an algorithm combining improved Mean Shift with incremental Support Vector Machine for infrared target tracking was proposed. First, target was described using gray histogram of the target region. Then, in order to solve the problem of target lost in tracking caused by target size obviously changing, target localization was started using standard Mean Shift, and then image moment feature of the sub image for secondary search was combined to calculate the tracking window size for next frame. Meanwhile, according to the problem of target occlusion easily lead to tracking failure, machine learning theory was introduced and incremental support vector machine was used to update target template adaptively, thus target tracking problem was converted to a problem of classification between the target and the background. Experiments show that the improved algorithm proposed in this paper performs well even if greatly change occurs in target pose, size or partial occlusion happens.