DMMFnet: A Dual-Branch Multimodal Medical Image Fusion Network Using Super Token and Channel-Spatial Attention

Abstract: Multimodal medical image fusion leverages the correlation between different modal images to enhance the information contained within a single medical image. Existing fusion methods often fail to effectively extract multiscale features from medical images and establish long-distance relationships between deep feature blocks. To address these issues, we propose DMMFnet, an encoder-decoder fusion network that utilizes shared and private encoders to extract shared and private features. DMMFnet is based on super token sampling and channel-spatial attention. The shared encoder and decoder use a transformer structure with super token sampling technology to effectively integrate information from different modalities, improving processing efficiency and enhancing the ability to capture key features. The private encoder consists of invertible neural networks and transformer modules, designed to extract local and global features, respectively. A novel transformer module refines attention distribution and feature aggregation to capture superpixel-level global correlations, ensuring that the network effectively captures essential global information, thereby enhancing the quality of the fused image. Experimental results, comparing DMMFnet with nine leading fusion methods, indicate that DMMFnet significantly improves various evaluation metrics and achieves superior visual effects, demonstrating its advanced fusion capability.

Yukun Zhang, Lei Wang, Muhammad Tahir, Zizhen Huang, Yaolong Han, Shanliang Yang, Shilong Liu and Muhammad Imran Saeed. “DMMFnet: A Dual-Branch Multimodal Medical Image Fusion Network Using Super Token and Channel-Spatial Attention”. International Journal of Advanced Computer Science and Applications (IJACSA) 15.8 (2024). http://dx.doi.org/10.14569/IJACSA.2024.0150869

@article{Zhang2024,
title = {DMMFnet: A Dual-Branch Multimodal Medical Image Fusion Network Using Super Token and Channel-Spatial Attention},
journal = {International Journal of Advanced Computer Science and Applications},
doi = {10.14569/IJACSA.2024.0150869},
url = {http://dx.doi.org/10.14569/IJACSA.2024.0150869},
year = {2024},
publisher = {The Science and Information Organization},
volume = {15},
number = {8},
author = {Yukun Zhang and Lei Wang and Muhammad Tahir and Zizhen Huang and Yaolong Han and Shanliang Yang and Shilong Liu and Muhammad Imran Saeed}
}