Abstract: Design and manufacture of higher rejection rate filters which meet the needs of accurate measurement of BaTiO3 single crystal film high-speed electro-optic modulator waveguide devices were focused on. The filters were used to improve the signal-to-noise ratio of the test system and eliminate stray light interference. The films were prepared by the depositing method of dual ion beam sputtering, Nb and SiO2 were chosen as deposition materials. The coating was designed and optimized with the help of Macleod and TFCalc software, orthogonal matrix method was used to optimize the process parameters of assist ion source. The absorption of film was reduced by using optimized process parameters, the problem of film thickness accuracy controlling was solved by using the method of real-time calibration of the deposition rate. Test transmission spectrum of the filter, center wavelength is 1 550.1 nm, the pass band width is 5.1 nm, the wavelength interval of optical density from -0.1 dB to - 30 dB is 1.9 nm, the using requirements of the waveguide accurate measurement system is achieved.