Infrared and visible image fusion of convolutional neural network and NSST

Traditional multi-scale infrared and visible image fusion methods couldnot be well applied to all kinds of complex image environments, because the extracted image features were fixed. However, deep learning could independently select appropriate image features to solve the unicity in feature extraction of multi-scale methods. Therefore, an infrared and visible image fusion method based on the combination of convolutional neural network and non-subsampled shear wave transform (NSST) was proposed. Firstly, the binary classification map of the infrared target and background was extracted by convolutional neural network, and the classification map was accurately segmented by frequency-tuned (FT) saliency detection algorithm. At the same time, the NSST was used to decompose the source image in multiple scales and directions; Secondly, the target saliency combined with adaptive fuzzy logic algorithm was used for the fusion of low frequency sub-bands, and the high frequency coefficient local variance contrast method was used for the fusion of high frequency sub-bands; Finally, the fused image was obtained through the inverse transformation of NSST. The experiment results show that compared with the traditional image fusion algorithm, this method improves objective evaluation indicators such as information entropy, average gradient, spatial frequency, mutual information and cross entropy at least increased by 0.01%, 0.30%, 1.43%, 2.32%, 1.14%, respectively. The contrast of fusion image is greatly improved, and the background details are enriched, which is more conducive to human eye recognition. It can be widely used in electro-optical reconnaissance, electro-optical warning, multi-sensor information fusion and other electro-optical information fields.