Abstract: Circular polarization control error caused by system design and intensity fluctuation as well as phase noise induced during the transmission of optical signal in atmospheric turbulence will directly lead to performance impairments of circular polarization shift keying(CPolSK) system in FSO. Therefore, on the basis of roundly considering the three parameters, which are the intensity scintillation, the phase noise and the circular polarization control error, the mathematical statistical model of the CPolSK system channel was established based on the Gamma-Gamma atmospheric channel model, and the closed expression of the average channel capacity was derived. Simulations were given to analyze the effects of intensity scintillation, phase fluctuation and circular polarization control error on channel capacity of CPolSK system. Results show that the increase of both intensity fluctuation and phase noise will cause the degradation of the channel capacity, while raising the average SNR can improve the channel capacity in a certain degree. It is also noted that the average channel capacity will significantly decrease when the circular polarization control error is larger than 8.