Abstract: The delay and pertinence of observation sequence in the infrared tracking system affect estimation to the target estate. In the course of tracking, delay of observation sequence induces the error between the measurement and truth, and then estimation based on the measurement is inaccurate, at the same time, pertinence of observation sequence affects the estimation result too. To assure the veracity of prediction, firstly, the system model was modified by ellipsoidal set form to overcome the pertinence of observation sequence, and at the same time, the innovation re-organization technique was introduced to eliminate the observation delay. With the innovation re-organization, the system structure was changed and a model without delay was obtained. At last, a delayed ellipsoidal set filter(DESF) was presented. Result of simulation shows that, DESF algorithm could overcome the effect of pertinence of observation sequence, and compared with filter designed without innovation re-organization, the tracking precision is better.