Abstract: High-order Quadrature Amplitude Modulation (QAM) in a limited bandwidth can transmit more information than either BPSK or QPSK system. Therefore, the technology is widely used in cable communication systems. While optical scintillation affected the high-order QAM system easily. The multi-mode blind signal equalization is used to reduce the influence of scintillation caused by atmospheric turbulence. In simulation, the turbulence channel was modeled by Gamma-Gamma distribution. The constellation of receiving signal shows the different points dispersions under different channel noise situations. Results show that the transmitted signal was primarily affected by multiplicative noise. A 64-QAM experimental system is built to demonstrate the feasibility of the system under real situation.