Objective The limited resolution of infrared devices, constrained by cost and manufacturing technology, remains a challenge. While deep learning-based single image super-resolution (SISR) has shown promise in enhancing image resolution, its application in real-world infrared images is hindered by the complexity of actual degradation, including spatial non-uniform blur caused by optical aberration and assembly error, as well as variations in the blur kernel due to environmental temperature changes. A deep learning-based approach for infrared imaging degradation model identification and super-resolution reconstruction is proposed to tackle these challenges. This method entails solving the degradation model using a convolutional neural network to describe the evolution of blur kernels, along with a super-resolution reconstruction method that adheres to the constraints of the degradation model and incorporates online learning of degradation parameters.
Methods Images of calibration targets are captured using an infrared camera placed in a high and low temperature chamber, along with a portable target simulator placed outside it (Fig.1-2). These images are utilized to calibrate the blur kernels. A convolutional neural network (CNN) is employed to construct a model that characterizes the relationship between blur kernel, pixel coordinate, and operating temperature (Fig.3). The model is trained using the calibrated blur kernels. Additionally, a super-resolution network is developed and trained (Fig.4). The operating temperature is initially estimated using the low-resolution image. Next, the initial blur kernels are estimated by inputting the operating temperature into the kernel model. Subsequently, super-resolution reconstruction is conducted based on the estimated blur kernels, and the reconstructed image is utilized to refine the operating temperature and blur kernel estimation. Iterative processes improve the accuracy of blur kernel estimation, leading to enhanced reconstruction outcomes.
Results and Discussions The blur kernels of the infrared imaging system exhibit significant variation in response to temperature changes and spatial locations (Fig.6). The trained blur kernel model effectively predicts blur kernels using temperature and pixel coordinate inputs (Fig.7). The average PSNR between predicted and actual blur kernels across different operational temperatures is consistently high, with a minimum of 32.2 dB and an average of 37.1 dB, indicating precise predictions (Fig.8). The calibration and modeling of blur kernels provide valuable prior information for super-resolution reconstruction, resulting in enhanced reconstruction outcomes. Consequently, the proposed algorithm produces visually appealing results with improved detail (Fig.10-11) and enhances objective quality evaluation metrics such as the natural image quality evaluator (NIQE), perception-based image quality evaluator (PIQE), and blind/referenceless image spatial quality evaluator (BRISQUE) (Tab.1).
Conclusions A novel approach is proposed for infrared super-resolution imaging, including degradation model identification and iterative super-resolution reconstruction. The degradation model is based on a convolutional neural network and is solved using offline calibration data. It can predict blur kernels across various temperatures and spatial positions, reducing the need for extensive calibration work. Online degradation parameter correction is achieved through an iterative optimization network alternating between estimating the blur kernel and reconstructing the super-resolution image. By leveraging the degradation model, the complex high-dimensional blur kernel estimation problem is simplified into a low-dimensional operating temperature estimation problem, streamlining the solution process. Through iterations, the accuracy of blur kernel estimation improves, leading to superior super-resolution reconstruction outcomes. Experimental results demonstrate that calibrating and modeling blur kernels enhance prior information for super-resolution reconstruction, yielding superior results. Additionally, the proposed method adapts to a wider temperature range, reducing the stringency of athermalization design requirements for infrared optical systems.
DownLoad: