Abstract: The time and frequency domain characteristics of three types of modulation formats, Carrier-Suppressed Return to Zero-On Off Keying(CSRZ-OOK), Carrier-Suppressed Return to Zero-Differential Phase Shift Keying(CSRZ-DPSK) and Carrier-Suppressed Return to Zero- Differential Quadrature Phase Shift Keying (CSRZ-DQPSK) based on carrier-suppressed return to zero formats, were analyzed firstly and the transmission performance of the three types of modulation formats was simulated in 40 Gbit/s single channel optical fiber transmission system. The tolerance of the three types of modulation formats to chromatic dispersion(CD), polarization mode dispersion(PMD) and nonlinear effect was evaluated by means of Q value, eye diagrams and bit error rate(BER). The results show that CSRZ-DQPSK has very good tolerance to chromatic dispersion, polarization mode dispersion and nonlinear effect compared with CSRZ-OOK and CSRZ-DPSK.