Objective In recent years, infrared imaging systems have been increasingly used in industry, security, and remote sensing. However, the resolution of infrared devices is still quite limited due to its cost and manufacturing technology restrictions. To increase image resolution, deep learning-based single image super-resolution (SISR) has gained much interest and made significant progress in simulated images. However, when applied to real-world images, most approaches suffer a performance drop, such as over-sharpening or over-smoothing. The main reason is that these methods assume that blur kernels are spatially invariant across the whole image. But such an assumption is rarely applicable for infrared images, whose blur kernels are usually spatially variant due to factors such as lens aberrations and thermal defocus. To address this issue, a blur kernel calibration method is proposed to estimate spatially-variant blur kernels, and a patch-based super-resolution (SR) algorithm is designed to reconstruct super-resolution images.
Methods Parallel light tube and motorized rotating platform are used to establish target image acquisition environment, and then images of multi-circle target at different positions are gathered (Fig.1). Based on sub-pixel accurate circle center detection, the camera pose parameters are solved, and high-resolution target images are synthesized according to the parameters. High-resolution and low-resolution target image pairs are fed into the blur kernel estimation network to obtain accurate blur kernels (Fig.3). In addition, a patch-based super-resolution algorithm is designed, which decomposes the test image into overlapping patches, reconstructs each of them separately using estimated kernels, and finally merges them according to Euclidean distances (Fig.4).
Results and Discussions The experimental results show that the blur caused by the optical system is not negligible and varies slowly with spatial position (Fig.6). The proposed method, which calibrates blur kernels in a laboratory setting, can obtain a more accurate blur kernel estimation result. As a consequence, the proposed patch-based super-resolution algorithm can produce more visually pleasant results with more reliable details (Fig.7-8), and can also boost objective quality evaluation indicators such as natural image quality evaluator (NIQE), perception based image quality evaluator (PIQE), and blind/referenceless image spatial quality evaluator (BRISQUE) (Tab.1). SR experiments on 4-bar targets with different spatial frequencies show that the proposed method can distinguish the target with spatial frequency of 3.57 cycles/mrad, while comparison methods can just distinguish that of 3.05 cycles/mrad under the same conditions (Fig.9).
Conclusions A blur kernel calibration method is proposed to estimate spatially-variant blur kernels, and a patch-based super-resolution algorithm is designed to implement super-resolution reconstruction. The experimental results show that image blur caused by the optical system changes slowly with the spatial position. As a result, one blur kernel can be estimated for each image patch, instead of densely estimated for each pixel, thereby reducing the complexity of calibration and memory consumption during reconstruction. Thanks to the accurate blur kernel estimation, the proposed super-resolution algorithm outperforms the comparison methods in both qualitative and quantitative results. Furthermore, the blur kernel calibration method is easy to implement in engineering applications. For any infrared camera, only dozens of multi-circle target images covering all areas of the focal plane are needed to complete the calibration process. When real-time performance is required, the proposed blur kernel calibration method can also be combined with other lightweight non-blind super-resolution methods to achieve a real-time performance. In the future, the problem of image blur caused by thermal defocusing will be studied to expand the scope of the method.
DownLoad: