Abstract: Interference imaging spectroscopy (IIS) is a novel high technology integrated by the two-beam interference technique, 2D imaging technique and spectroscopy with high resolution. The wide-field-of- view polarized type is an attractive scheme in IIS. It employs polarized components to realize interference and thus the image quality and spectrum reconstruction are principally determined by the characteristics of these polarization components. In this work, theoretical analysis of the interference visibility was performed by considering the polarization deviations and the retardation deviation of elements. The theoretical expression of interference visibility was deduced by polarization tracing method. The varying characteristics of interference visibility with incident angle, wavelength, azimuth angle, polarization direction and phase retardation were presented and the permissible deviation was quantitatively analyzed. This work would provide suggestions for system design, hyperspectral reconstruction and engineering practice of spectrometers composed of polarization components.