Abstract: In order to reduce noise in the laser-pumped magnetometer, the virtual instrument technology was applied to the detection module for the first time. With Labview programming tools and the PXIe-bus based control system, the integrated design of the data acquiring module, the signal processing module and the signal generating module was completed. Analysis of the three modules was performed with emphasis on the digital signal processing algorithm. By improving the structure of the low-pass filter, the conflict between bandwidth and noise level was resolved. By using trapezoidal integration incremental PID control algorithm, the system's robustness and anti-interference ability were greatly improved. The control system noise level reached the magnitude of the 0.01 V. Combined with laser frequency and power stabilization technique, the noise introduced by the the laser frequency drift and power jitter was effectively eliminated. The magnetometer noise was reduced to 0.7 pT/Hz0.5@1 Hz.