Objective The measurement of grating pitch is a key technological requirement for achieving high-precision positioning in nanometer-level measurement techniques. It finds wide applications in fields such as micro-nano fabrication, precision manufacturing, and microelectronics, which are based on nanotechnology research. To enhance the precision of grating pitch measurements, calibration can be achieved through two methods involving digital processing of interference signals. The phase information of grating interference signals can be characterized through digital methods in both time and frequency domains. Traditional phase unwrapping methods in time-frequency domains typically involve global analysis of signals, which are more suitable for small discrete points and stable signals. However, they are susceptible to noise when handling non-stationary and nonlinear signals, leading to phase loss issues and insufficient phase information. Addressing the limitations of traditional algorithms, it is crucial to research optimized algorithms that effectively improve signal robustness and measurement stability while reducing the impact of nonlinear factors on phase accuracy. Therefore, this paper proposes an advanced method for high-precision digital calibration of dual grating pitch using Improved Complete Ensemble Empirical Mode Decomposition with Adaptive Noise combined with Hilbert Transform (ICEEMDAN-HT).

Methods This paper proposes an advanced digital calibration method for high-precision dual grating pitch using ICEEMDAN-HT. The theoretical analysis begins with the calibration principles of the dual grating homodyne interferometer and discusses the interference signal nonlinearity (Fig.1). It further elaborates on the principles and methods of digital calibration based on ICEEMDAN-HT (Fig.2). Simulation is conducted to decompose and reconstruct signals, comparing them with other algorithms to validate the completeness and stability of the ICEEMDAN method (Tab.1). Two sets of interference signals are collected from the dual grating homodyne interferometer calibration system (Fig.7). The ICEEMDAN-HT method is employed for signal reconstruction and phase extraction (Fig.8). Atomic force microscope scanning results are calibrated using a self-traceable grating serve as the standard measure (Fig.9). Comparison of calibration results between this method and others assesses the effectiveness and completeness of ICEEMDAN-HT digital calibration method (Tab.3).

Results and Discussions Simulation results indicate that the RMSE of the ICEEMDAN method is the lowest among the three methods at 0.029, with SNR and CC also the highest at 15.012 and 99.87%, respectively (Fig.6). In all three aspects, ICEEMDAN outperforms CEEMD and EMD, with a correlation coefficient close to 99.9%, indicating nearly complete restoration of the original noise-free signal with minimal information loss (Tab.1). This demonstrates that ICEEMDAN is the most effective in handling noisy nonlinear signals, showcasing superior completeness and stability in signal reconstruction. Validation through dual grating homodyne interferometer calibration experiments shows that ICEEMDAN-HT method achieves a relative error as low as 0.40%, surpassing other signal decomposition methods in calibration accuracy (Fig.8). These results illustrate that ICEEMDAN-HT digital calibration method effectively enhances signal robustness and measurement stability, mitigating the impact of nonlinear factors on phase accuracy. This research holds significant practical implications for real-time signal measurement and instantaneous processing efficiency.

Conclusions This paper addresses the nonlinear influences in dual grating distance measurement and proposes a digital calibration method based on ICEEMDAN-HT built upon existing EMD algorithms. Validation is conducted through dual grating homodyne interferometer calibration experiments, comparing the ICEEMDAN-HT method with several existing signal decomposition and reconstruction methods. The relative error of calibration with ICEEMDAN-HT can be reduced to as low as 0.40%, demonstrating higher calibration accuracy compared to other signal decomposition methods. The results indicate that the ICEEMDAN-HT digital calibration method effectively enhances signal robustness and measurement stability against nonlinear factors affecting phase accuracy. This research holds significant practical implications for real-time signal measurement and instantaneous processing efficiency.