Currently, visible and infrared image fusion (VIF) technology has a wide range of applications in road safety monitoring, anti-surveillance, etc. However, the traditional image fusion algorithms in the feature fusion process will have limitations such as part of the information is lost, etc. For this reason, this paper proposes an infrared visible image fusion algorithm based on the double-branching and decomposition of the results. The algorithm firstly adopts the dense block method, extracts visible image features, and uses a feature pyramid network to extract infrared features. The algorithm firstly adopts the dense block method to extract the visible image features, and uses the feature pyramid network to extract the infrared features, then, based on the deep learning network structure to extract the image information of different modalities, and designs the fusion network constrained by the three loss functions of the gradient loss, intensity loss and decomposition loss, so as to obtain a good fusion effect of the image. The experimental results show that the proposed algorithm achieves the optimal value in five indexes, and reaches sub-optimal value in one index, indicating that the proposed algorithm fuses the images with the optimal value and sub-optimal value. At the same time, the proposed algorithm retains the main thermal radiation information of infrared images better than other algorithms such as DenseFuse and IFCNN, which is superior to some extent.