Abstract
Multimodal image fusion is an important area of research with various applications in computer vision. This research proposes a modification to convolutional layers by fusing two different modalities of images. A novel architecture that uses adaptive fusion mechanisms to learn the optimal weightage of different modalities at each convolutional layer is introduced in the research. The proposed method is evaluated on a publicly available dataset, and the experimental results show that the performance of the proposed method outperforms state-of-the-art methods in terms of various evaluation metrics.
References
Yu Zhang, Yu Liu, Peng Sun, Han Yan, Xiaolin Zhao, and Li Zhang. 2020. IFCNN: A general image fusion framework based on convolutional neural network. Information Fusion 54: 99–118. https://doi.org/10.1016/j.inffus.2019.07.011
Jameel Ahmed Bhutto, Lianfang Tian, Qiliang Du, Zhengzheng Sun, Lubin Yu, and Muhammad Faizan Tahir. 2022. CT and MRI Medical Image Fusion Using Noise-Removal and Contrast Enhancement Scheme with Convolutional Neural Network. Entropy 24, 3: 393. https://doi.org/10.3390/e24030393
Ahmed Sabeeh Yousif, Zaid Omar, and Usman Ullah Sheikh. 2022. An improved approach for medical image fusion using sparse representation and Siamese convolutional neural network. Biomedical Signal Processing and Control 72: 103357. https://doi.org/10.1016/j.bspc.2021.103357
Han Xu and Jiayi Ma. 2021. EMFusion: An unsupervised enhanced medical image fusion network. Information Fusion 76: 177–186. https://doi.org/10.1016/j.inffus.2021.06.001
Zeyu Wang, Xiongfei Li, Haoran Duan, Yanchi Su, Xiaoli Zhang, and Xinjiang Guan. 2021. Medical image fusion based on convolutional neural networks and non-subsampled contourlet transform. Expert Systems with Applications 171: 114574. https://doi.org/10.1016/j.eswa.2021.114574
Kunpeng Wang, Mingyao Zheng, Hongyan Wei, Guanqiu Qi, and Yuanyuan Li. 2020. Multi-Modality Medical Image Fusion Using Convolutional Neural Network and Contrast Pyramid. Sensors 20, 8: 2169. https://doi.org/10.3390/s20082169
Hao Zhang, Han Xu, Xin Tian, Junjun Jiang, and Jiayi Ma. 2021. Image fusion meets deep learning: A survey and perspective. Information Fusion 76: 323–336. https://doi.org/10.1016/j.inffus.2021.06.008
Gargi J Trivedi and Rajesh Sanghvi. 2022. Medical Image Fusion Using CNN with Automated Pooling. Indian Journal Of Science And Technology 15, 42: 2267–2274. https://doi.org/10.17485/ijst/v15i42.1812
D. Sunderlin Shibu and S. Suja Priyadharsini. 2021. Multi scale decomposition based medical image fusion using convolutional neural network and sparse representation. Biomedical Signal Processing and Control 69: 102789. https://doi.org/10.1016/j.bspc.2021.102789
Yi Li, Junli Zhao, Zhihan Lv, and Jinhua Li. 2021. Medical image fusion method by deep learning. International Journal of Cognitive Computing in Engineering 2: 21–29. https://doi.org/10.1016/j.ijcce.2020.12.004
Velmurugan Subbiah Parvathy, Sivakumar Pothiraj, and Jenyfal Sampson. 2020. A novel approach in multimodality medical image fusion using optimal shearlet and deep learning. International Journal of Imaging Systems and Technology 30, 4: 847–859. https://doi.org/10.1002/ima.22436
Gargi Trivedi,and Rajesh Sanghvi. 2023. Optimizing Image Fusion Using Modified Principal Component Analysis Algorithm and Adaptive Weighting Scheme Int. J. Advanced Networking and Applications. 15(1): 5769 – 5774. https://doi.org/10.35444/IJANA.2023.15103
Jiaheng Xie, Bin Zhang, Jian Ma, Daniel Zeng, and Jenny Lo-Ciganic. 2021. Readmission Prediction for Patients with Heterogeneous Medical History: A Trajectory-Based Deep Learning Approach. ACM Transactions on Management Information Systems 13, 2: 1–27. https://doi.org/10.1145/3468780
Aixia Guo, Michael Pasque, Francis Loh, Douglas L. Mann, and Philip R. O. Payne. 2020. Heart Failure Diagnosis, Readmission, and Mortality Prediction Using Machine Learning and Artificial Intelligence Models. Current Epidemiology Reports 7, 4: 212–219. https://doi.org/10.1007/s40471-020-00259-w
Bobak J. Mortazavi, Nicholas S. Downing, Emily M. Bucholz, Kumar Dharmarajan, Ajay Manhapra, Shu-Xia Li, Sahand N. Negahban, and Harlan M. Krumholz. 2016. Analysis of Machine Learning Techniques for Heart Failure Readmissions. Circulation: Cardiovascular Quality and Outcomes 9, 6: 629–640. https://doi.org/10.1161/circoutcomes.116.003039
Boshu Ru, Xi Tan, Yu Liu, Kartik Kannapur, Dheepan Ramanan, Garin Kessler, Dominik Lautsch, and Gregg Fonarow. 2023. Comparison of Machine Learning Algorithms for Predicting Hospital Readmissions and Worsening Heart Failure Events in Patients With Heart Failure With Reduced Ejection Fraction: Modeling Study. JMIR Formative Research 7: e41775. https://doi.org/10.2196/41775
Dibaba Adeba Debal and Tilahun Melak Sitote. 2022. Chronic kidney disease prediction using machine learning techniques. Journal of Big Data 9, 1. https://doi.org/10.1186/s40537-022-00657-5
Amadou Wurry Jallow, Adama N. S. Bah, Karamo Bah, Chien-Yeh Hsu, and Kuo-Chung Chu. 2022. Machine Learning Approach for Chronic Kidney Disease Risk Prediction Combining Conventional Risk Factors and Novel Metabolic Indices. Applied Sciences 12, 23: 12001. https://doi.org/10.3390/app122312001
Hasnain Iftikhar, Murad Khan, Zardad Khan, Faridoon Khan, Huda M Alshanbari, and Zubair Ahmad. 2023. A Comparative Analysis of Machine Learning Models: A Case Study in Predicting Chronic Kidney Disease. Sustainability 15, 3: 2754. https://doi.org/10.3390/su15032754
Qiong Bai, Chunyan Su, Wen Tang, and Yike Li. 2022. Machine learning to predict end stage kidney disease in chronic kidney disease. Scientific Reports 12, 1. https://doi.org/10.1038/s41598-022-12316-z
Md. Ariful Islam, Md. Ziaul Hasan Majumder, and Md. Alomgeer Hussein. 2023. Chronic kidney disease prediction based on machine learning algorithms. Journal of Pathology Informatics 14: 100189. https://doi.org/10.1016/j.jpi.2023.100189
Aristidis G. Vrahatis, Konstantina Skolariki, Marios G. Krokidis, Konstantinos Lazaros, Themis P. Exarchos, and Panagiotis Vlamos. 2023. Revolutionizing the Early Detection of Alzheimer’s Disease through Non-Invasive Biomarkers: The Role of Artificial Intelligence and Deep Learning. Sensors 23, 9: 4184. https://doi.org/10.3390/s23094184
Ruoxuan Cui and Manhua Liu. 2019. RNN-based longitudinal analysis for diagnosis of Alzheimer’s disease. Computerized Medical Imaging and Graphics 73: 1–10. https://doi.org/10.1016/j.compmedimag.2019.01.005
Ali Ezzati and Richard B. Lipton. 2020. Machine Learning Predictive Models Can Improve Efficacy of Clinical Trials for Alzheimer’s Disease. Journal of Alzheimer’s Disease 74, 1: 55–63. https://doi.org/10.3233/jad-190822
Sayantan Kumar, Inez Oh, Suzanne Schindler, Albert M Lai, Philip R O Payne, and Aditi Gupta. 2021. Machine learning for modeling the progression of Alzheimer disease dementia using clinical data: a systematic literature review. JAMIA Open 3,4 https://doi.org/10.1093/jamiaopen/ooab052
JungHo Kong, Doyeon Ha, Juhun Lee, Inhae Kim, Minhyuk Park, Sin-Hyeog Im, Kunyoo Shin, and Sanguk Kim. 2022. Network-based machine learning approach to predict immunotherapy response in cancer patients. Nature Communications 13, 1. https://doi.org/10.1038/s41467-022-31535-6
Yujie You, Xin Lai, Yi Pan, Huiru Zheng, Julio Vera, Suran Liu, Senyi Deng, and Le Zhang. 2022. Artificial intelligence in cancer target identification and drug discovery. Signal Transduction and Targeted Therapy 7, 1. https://doi.org/10.1038/s41392-022-00994-0
Vesna Cuplov and Nicolas André. 2020. Machine Learning Approach to Forecast Chemotherapy-Induced Haematological Toxicities in Patients with Rhabdomyosarcoma. Cancers 12, 7: 1944. https://doi.org/10.3390/cancers12071944
Raihan Rafique, S.M. Riazul Islam, and Julhash U. Kazi. 2021. Machine learning in the prediction of cancer therapy. Computational and Structural Biotechnology Journal 19: 4003–4017. https://doi.org/10.1016/j.csbj.2021.07.003
Yu Liu, Xun Chen, Juan Cheng, Hu Peng, and Zengfu Wang. 2018. Infrared and visible image fusion with convolutional neural networks. International Journal of Wavelets, Multiresolution and Information Processing 16, 03: 1850018. https://doi.org/10.1142/s0219691318500182
Yongzhi Long, Haitao Jia, Yida Zhong, Yadong Jiang, and Yuming Jia. 2021. RXDNFuse: A aggregated residual dense network for infrared and visible image fusion. Information Fusion 69: 128–141. https://doi.org/10.1016/j.inffus.2020.11.009
Jiayi Ma, Wei Yu, Pengwei Liang, Chang Li, and Junjun Jiang. 2019. FusionGAN: A generative adversarial network for infrared and visible image fusion. Information Fusion 48: 11–26. https://doi.org/10.1016/j.inffus.2018.09.004
Alexander Toet. 2017. The TNO Multiband Image Data Collection. Data in Brief 15: 249–251. https://doi.org/10.1016/j.dib.2017.09.038
The Whole Brain Atlas. The Whole Brain Atlas. Retrieved from https://www.med.harvard.edu/aanlib/
Gargi Trivedi and Rajesh Sanghvi. 2023.On solution of non-instantaneous impulsive Hilfer fractional integro-differential evolution system. MATHEMATICA APPLICANDA.51(1);3-20.10.14708/ma. v50i2.7168
Xing, Xiaoxue, Cheng Liu, Cong Luo, and Tingfa Xu. "Infrared and visible image fusion based on nonlinear enhancement and NSST decomposition." EURASIP Journal on Wireless Communications and Networking 2020 (2020): 1-17.
Feng, Zi-Jun, Xiao-Ling Zhang, Li-Yong Yuan, and Jia-Nan Wang. "Infrared target detection and location for visual surveillance using fusion scheme of visible and infrared images." Mathematical Problems in Engineering 2013 (2013).