Accurate and early stroke lesion segmentation is necessary for better results for patients and successful treatment. A novel deep learning model using a multi-modality architecture that combines spatial, channel, and temporal information from perfusion MRI and CT images is proposed in this work for the deep segmentation of stroke lesions. The model addresses cross-domain generalization by integrating dynamic perfusion information from the ISLES 2022 dataset, such as Diffusion Weighted Imaging (DWI) and CT Perfusion (CTP) images alongside high-resolution anatomical data obtained from the ATLAS v2.0 dataset. We enhance the proposed approach by adding a hybrid spatial channel temporal attention transformer block to a UNet encoder-decoder architecture. Multi-scale features that are patch-embedded and enhanced with positional encoding are extracted by the encoder. To increase sensitivity to anatomical features and corresponding pathological changes, the attention module integrates temporal encoding of perfusion parameters like Cerebral Blood Flow (CBF), Cerebral Blood Volume (CBV), Mean Transit Time (MTT), and Time-To-Maximum (Tmax) with joint modeling of spatial and channel dependencies. The experimental results contribute to reliable clinical application by exhibiting increased accuracy and robustness over traditional UNet and unimodal transformer models.
@article{sulaiman2026,
author = {Sadiya Sulaiman and Roshni Thanka M. and Jemima Jebaseeli T. and Nader Salam and Bijolin Edwin E.},
title = {{Multimodal Stroke Lesion Segmentation with Hybrid Attention and Transformer Architecture}},
journal = {Journal of Innovative Image Processing},
volume = {8},
number = {1},
pages = {250-271},
year = {2026},
publisher = {Inventive Research Organization},
doi = {10.36548/jiip.2026.1.014},
url = {https://doi.org/10.36548/jiip.2026.1.014}
}
Copy Citation
- World Health Organization, “Global Health Estimates: Leading Causes of Death,” WHO, 2023. [Online]. Available: https://www.who.int/data/gho/data/themes/mortality-and-global-health-estimates
- Maier, Andreas, Christopher Syben, Tobias Lasser, and Christian Riess. "A Gentle Introduction to Deep Learning in Medical Image Processing." Zeitschrift für Medizinische Physik 29, no. 2 (2019): 86-101.
- Maier, Oskar, Bjoern H. Menze, Janina Von der Gablentz, Levin Häni, Mattias P. Heinrich, Matthias Liebrand, Stefan Winzeck et al. "ISLES 2015-A Public Evaluation Benchmark for Ischemic Stroke Lesion Segmentation from Multispectral MRI." Medical image analysis 35 (2017): 250-269.
- Abbasi, Hossein, Maysam Orouskhani, Samaneh Asgari, and Sara Shomal Zadeh. "Automatic Brain Ischemic Stroke Segmentation with Deep Learning: A Review." Neuroscience Informatics 3, no. 4 (2023): 100145.
- Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. "U-net: Convolutional Networks for Biomedical Image Segmentation." In International Conference on Medical image computing and computer-assisted intervention, Cham: Springer international publishing, 2015, 234-241.
- Chen, Jieneng, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L. Yuille, and Yuyin Zhou. "Transunet: Transformers Make Strong Encoders for Medical Image Segmentation." arXiv preprint arXiv:2102.04306 (2021).
- Ma, Danqing, Meng Wang, Ao Xiang, Zongqing Qi, and Qin Yang. "Transformer-Based Classification Outcome Prediction for Multimodal Stroke Treatment." In 2024 IEEE 2nd International Conference on Sensors, Electronics and Computer Engineering (ICSECE), IEEE, 2024, 383-386.
- Xu, Jie, Keren Shen, Zhuo Yu, Huizhe Lu, Te Lin, Yaozi Song, and Likang Luo. "Development of a Diagnostic Model for Acute Ischemic Stroke Early Identification Based on SE-ResNeXt." (2024).
- Warner, John J., Robert A. Harrington, Ralph L. Sacco, and Mitchell SV Elkind. "Guidelines for the Early Management of Patients with Acute Ischemic Stroke: 2019 Update to the 2018 Guidelines for the Early Management of Acute Ischemic Stroke." Stroke 50, no. 12 (2019): 3331-3332.
- El-Koussy, Marwan, Gerhard Schroth, Caspar Brekenfeld, and Marcel Arnold. "Imaging of Acute Ischemic Stroke." European neurology 72, no. 5-6 (2014): 309-316.
- Muir, Keith W., Alastair Buchan, Rudiger von Kummer, Joachim Rother, and Jean-Claude Baron. "Imaging of acute Stroke." The Lancet Neurology 5, no. 9 (2006): 755-768.
- Prust, Morgan L., Rachel Forman, and Bruce Ovbiagele. "Addressing Disparities in the Global Epidemiology of Stroke." Nature Reviews Neurology 20, no. 4 (2024): 207-221.
- Komosar, Aleksa, Darko Stefanović, and Srđan Sladojević. "An Overview of Image Processing in Biomedicine Using U-Net Convolutional Neural Network Architecture." Journal of Computer and Forensic Sciences 3, no. 1 (2024): 5-20.
- Chen, Jieneng, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L. Yuille, and Yuyin Zhou. "Transunet: Transformers Make Strong Encoders for Medical Image Segmentation." arXiv preprint arXiv:2102.04306 (2021).
- Liew, Sook-Lei, Julia M. Anglin, Nick W. Banks, Matt Sondag, Kaori L. Ito, Hosung Kim, Jennifer Chan et al. "A Large, Open Source Dataset of Stroke Anatomical Brain Images and Manual Lesion Segmentations." Scientific data 5, no. 1 (2018): 180011.
- Hakim, Arsany, Søren Christensen, Stefan Winzeck, Maarten G. Lansberg, Mark W. Parsons, Christian Lucas, David Robben, Roland Wiest, Mauricio Reyes, and Greg Zaharchuk. "Predicting Infarct Core from Computed Tomography Perfusion in Acute Ischemia with Machine Learning: Lessons from the ISLES Challenge." Stroke (2021): STROKEAHA-120.
- Kamnitsas, Konstantinos, Christian Ledig, Virginia FJ Newcombe, Joanna P. Simpson, Andrew D. Kane, David K. Menon, Daniel Rueckert, and Ben Glocker. "Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentation." Medical image analysis 36 (2017): 61-78.
- Zhang, Ling, Xiaosong Wang, Dong Yang, Thomas Sanford, Stephanie Harmon, Baris Turkbey, Bradford J. Wood et al. "Generalizing Deep Learning for Medical Image Segmentation to Unseen Domains Via Deep Stacked Transformation." IEEE transactions on medical imaging 39, no. 7 (2020): 2531-2540.
- Kamnitsas, Hernandez Petzsche, M.R., de la Rosa, E., Hanning, U. et al. ISLES 2022: A Multi-Center Magnetic Resonance Imaging Stroke Lesion Segmentation Dataset. Sci Data 9, 762 (2022). https://doi.org/10.1038/s41597-022-01875-5
- Konstantinos, Christian Ledig, Virginia FJ Newcombe, Joanna P. Simpson, Andrew D. Kane, David K. Menon, Daniel Rueckert, and Ben Glocker. "Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentation." Medical image analysis 36 (2017): 61-78.
- Wang, Wenxuan, Chen Chen, Meng Ding, Hong Yu, Sen Zha, and Jiangyun Li. "Transbts: Multimodal Brain Tumor Segmentation Using Transformer." In International conference on medical image computing and computer-assisted intervention, Cham: Springer International Publishing, 2021, 109-119.
- Chen, Jieneng, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L. Yuille, and Yuyin Zhou. "Transunet: Transformers Make Strong Encoders for Medical Image Segmentation." arXiv preprint arXiv:2102.04306 (2021).
- Pacal, Ishak, Ali Algarni, Bilal Bayram, and Suat Ince. "FA-UNet: A FasterNet and Attention-Gated Hybrid Network for Precise Ischemic Stroke Segmentation." Journal of Integrative Neuroscience 24, no. 10 (2025): 40100.
- ATLAS v2.0 https://fcon_1000.projects.nitrc.org/indi/retro/atlas.html