Multimodal Learning for Breast Cancer Biomarker Prediction Using Whole Slide Histopathology Images
PDFAbstract
Globally, breast cancer remains a significant health challenge that has a direct effect on women's cancer morbidity and mortality. The estrogen receptor (ER), progesterone receptor (PR), and human epidermal growth factor receptor 2 (HER2) are important factors that help doctor to determine the best treatment for each woman. When using immunohistochemistry and genomic assays to look for markers, it is a relatively long and slow process that varies from individual to individual. The aim of this study is to develop a deep-learning framework to predict directly the ER, PR and HER2 status of H&E-stained histopathology images. The technique entails downsampling Level-1 slide images from the TCGA-BRCA cohort, followed by using a pre-trained ResNet50 architecture to extract histological features to enhance the accuracy of biomarker prediction. We train a multi-output classification model using XGBoost that adds a classifier chain. We use a mixture of clinical and genetic data as well as image features. This joint computational method shows promise in enhancing the accuracy of biomarker predictions and enabling doctors to customize breast cancer treatment for individual patients.
Cite this article
Shah, Vinita, and Miral Patel. "Multimodal Learning for Breast Cancer Biomarker Prediction Using Whole Slide Histopathology Images." Journal of Innovative Image Processing 8, no. 1 (2026): 175-189. doi: 10.36548/jiip.2026.1.010
Copy Citation
Shah, V., & Patel, M. (2026). Multimodal Learning for Breast Cancer Biomarker Prediction Using Whole Slide Histopathology Images. Journal of Innovative Image Processing, 8(1), 175-189. https://doi.org/10.36548/jiip.2026.1.010
Copy Citation
Shah, Vinita, et al. "Multimodal Learning for Breast Cancer Biomarker Prediction Using Whole Slide Histopathology Images." Journal of Innovative Image Processing, vol. 8, no. 1, 2026, pp. 175-189. DOI: 10.36548/jiip.2026.1.010.
Copy Citation
Shah V, Patel M. Multimodal Learning for Breast Cancer Biomarker Prediction Using Whole Slide Histopathology Images. Journal of Innovative Image Processing. 2026;8(1):175-189. doi: 10.36548/jiip.2026.1.010
Copy Citation
V. Shah, and M. Patel, "Multimodal Learning for Breast Cancer Biomarker Prediction Using Whole Slide Histopathology Images," Journal of Innovative Image Processing, vol. 8, no. 1, pp. 175-189, Mar. 2026, doi: 10.36548/jiip.2026.1.010.
Copy Citation
Shah, V. and Patel, M. (2026) 'Multimodal Learning for Breast Cancer Biomarker Prediction Using Whole Slide Histopathology Images', Journal of Innovative Image Processing, vol. 8, no. 1, pp. 175-189. Available at: https://doi.org/10.36548/jiip.2026.1.010.
Copy Citation
@article{shah2026,
author = {Vinita Shah and Miral Patel},
title = {{Multimodal Learning for Breast Cancer Biomarker Prediction Using Whole Slide Histopathology Images}},
journal = {Journal of Innovative Image Processing},
volume = {8},
number = {1},
pages = {175-189},
year = {2026},
publisher = {Inventive Research Organization},
doi = {10.36548/jiip.2026.1.010},
url = {https://doi.org/10.36548/jiip.2026.1.010}
}
Copy Citation
Keywords
Breast Cancer
Histopathology
Deep Learning
H&E Images
Biomarker Prediction
ER
PR
HER2
References
- Allison, Kimberly H., M. Elizabeth H. Hammond, Mitchell Dowsett, Shannon E. McKernin, Lisa A. Carey, Patrick L. Fitzgibbons, Daniel F. Hayes et al. "Estrogen and Progesterone Receptor Testing in Breast Cancer: ASCO/CAP Guideline Update." Journal of Clinical Oncology 38, no. 12 (2020): 1346-1366.
- Bray, Freddie, Jacques Ferlay, Isabelle Soerjomataram, Rebecca L. Siegel, Lindsey A. Torre, and Ahmedin Jemal. "Global Cancer Statistics 2018: GLOBOCAN Estimates Of Incidence and Mortality Worldwide For 36 Cancers In 185 Countries." CA: a cancer journal for clinicians 68, no. 6 (2018): 394-424.
- Charles, M., and C. G. A. Network. "Comprehensive Molecular Portraits of Human Breast Tumors." Nature 490, no. 7418 (2012): 61-70.
- Elmore, Joann G., Raymond L. Barnhill, David E. Elder, Gary M. Longton, Margaret S. Pepe, Lisa M. Reisch, Patricia A. Carney et al. "Pathologists’ Diagnosis of Invasive Melanoma and Melanocytic Proliferations: Observer Accuracy and Reproducibility Study." bmj 357 (2017).
- Hammond, M. Elizabeth H., Daniel F. Hayes, Mitch Dowsett, D. Craig Allred, Karen L. Hagerty, Sunil Badve, Patrick L. Fitzgibbons et al. "American Society of Clinical Oncology/College of American Pathologists Guideline Recommendations for Immunohistochemical Testing of Estrogen and Progesterone Receptors in Breast Cancer." Journal of clinical oncology 28, no. 16 (2010): 2784-2795.
- Harbeck, N., F. Penault-Llorca, J. Cortes, M. Gnant, N. Houssami, P. Poortmans, K. Ruddy, J. Tsang, and F. Cardoso. Breast Cancer. Nature Reviews Disease Primers 5: 66. 2019.
- Gurcan, Metin N., Laura E. Boucheron, Ali Can, Anant Madabhushi, Nasir M. Rajpoot, and Bulent Yener. "Histopathological Image Analysis: A Review." IEEE reviews in biomedical engineering 2 (2009): 147-171.
- Lambin, Philippe, Ralph TH Leijenaar, Timo M. Deist, Jurgen Peerlings, Evelyn EC De Jong, Janita Van Timmeren, Sebastian Sanduleanu et al. "Radiomics: The Bridge between Medical Imaging and Personalized Medicine." Nature reviews Clinical oncology 14, no. 12 (2017): 749-762.
- Litjens, Geert, Thijs Kooi, Babak Ehteshami Bejnordi, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen Awm Van Der Laak, Bram Van Ginneken, and Clara I. Sánchez. "A Survey on Deep Learning in Medical Image Analysis." Medical image analysis 42 (2017): 60-88.
- Mavaddat, Nasim, Antonis Antoniou, Daniel Easton, and Douglas F. Easton. “Prediction of Breast Cancer Risk Based on Profiling with Common Genetic Variants.” Journal of the National Cancer Institute 107, no. 5 (2015): djv036.
- Expression–Based, Gene. "Gene Expression and Benefit of Chemotherapy in Women with Node-Negative, Estrogen Receptor–Positive Breast Cancer." Gene Expression 2: 54.
- Perou, Charles M., Therese Sørlie, Michael B. Eisen, Matt Van De Rijn, Stefanie S. Jeffrey, Christian A. Rees, Jonathan R. Pollack et al. "Molecular Portraits of Human Breast Tumours." nature 406, no. 6797 (2000): 747-752.
- Rakha, Emad A., Gary M. Tse, and Cecily M. Quinn. "An Update on the Pathological Classification of Breast Cancer." Histopathology 82, no. 1 (2023): 5-16.
- Schnitt, Stuart J. "Classification and Prognosis of Invasive Breast Cancer: From Morphology to Molecular Taxonomy." Modern pathology 23 (2010): S60-S64.
- Slamon, Dennis J., Gary M. Clark, Steven G. Wong, Wendy J. Levin, Axel Ullrich, and William L. McGuire. "Human Breast Cancer: Correlation of Relapse and Survival with Amplification of the HER-2/Neu Oncogene." science 235, no. 4785 (1987): 177-182.
- Slamon, Dennis J., Brian Leyland-Jones, Steven Shak, Hank Fuchs, Virginia Paton, Alex Bajamonde, Thomas Fleming et al. "Use of Chemotherapy Plus a Monoclonal Antibody Against HER2 for Metastatic Breast Cancer that Overexpresses HER2." New England journal of medicine 344, no. 11 (2001): 783-792.
- Sparano, Joseph A., Robert J. Gray, Della F. Makower, Kathleen I. Pritchard, Kathy S. Albain, Daniel F. Hayes, Charles E. Geyer Jr et al. "Adjuvant Chemotherapy Guided by a 21-Gene Expression Assay in Breast Cancer." New England Journal of Medicine 379, no. 2 (2018): 111-121.
- Swain, Sandra M., José Baselga, Sung-Bae Kim, Jungsil Ro, Vladimir Semiglazov, Mario Campone, Eva Ciruelos et al. "Pertuzumab, Trastuzumab, and Docetaxel in HER2-Positive Metastatic Breast Cancer." New England journal of medicine 372, no. 8 (2015): 724-734.
- World Health Organization. World Cancer Report 2020: Cancer Research for Cancer Prevention. Lyon: International Agency for Research on Cancer, 2020.
- Love, Michael I., Wolfgang Huber, and Simon Anders. "Moderated Estimation of Fold Change and Dispersion for RNA-Seq Data with DESeq2." Genome biology 15, no. 12 (2014): 550.
- Couture, Heather D., Lindsay A. Williams, Joseph Geradts, Sarah J. Nyante, Ebonee N. Butler, J. S. Marron, Charles M. Perou, Melissa A. Troester, and Marc Niethammer. "Image Analysis with Deep Learning to Predict Breast Cancer Grade, ER Status, Histologic Subtype, and Intrinsic Subtype." NPJ breast cancer 4, no. 1 (2018): 30.
- Jaber, Mustafa I., Bing Song, Clive Taylor, Charles J. Vaske, Stephen C. Benz, Shahrooz Rabizadeh, Patrick Soon-Shiong, and Christopher W. Szeto. "A Deep Learning Image-Based Intrinsic Molecular Subtype Classifier of Breast Tumors Reveals Tumor Heterogeneity that May Affect Survival." Breast Cancer Research 22, no. 1 (2020): 12.
- Rawat, Rishi R., Itzel Ortega, Preeyam Roy, Fei Sha, Darryl Shibata, Daniel Ruderman, and David B. Agus. "Deep learned tissue “Fingerprints” Classify Breast Cancers by ER/PR/Her2 Status from H&E Images." Scientific reports 10, no. 1 (2020): 7275.
- Bychkov, Dmitrii, Nina Linder, Riku Turkki, Stig Nordling, Panu E. Kovanen, Clare Verrill, Margarita Walliander, Mikael Lundin, Caj Haglund, and Johan Lundin. "Deep Learning Based Tissue Analysis Predicts Outcome in Colorectal Cancer." Scientific reports 8, no. 1 (2018): 3395.
- Shamai, Gil, Amir Livne, António Polónia, Edmond Sabo, Alexandra Cretu, Gil Bar-Sela, and Ron Kimmel. "Deep Learning-Based Image Analysis Predicts PD-L1 Status from H&E-Stained Histopathology Images in Breast Cancer." Nature Communications 13, no. 1 (2022): 6753.
- Wang, Xiaoxiao, Chong Zou, Yi Zhang, Xiuqing Li, Chenxi Wang, Fei Ke, Jie Chen et al. "Prediction of BRCA Gene Mutation in Breast Cancer Based on Deep Learning and Histopathology Images." Frontiers in Genetics 12 (2021): 661109.
- Tafavvoghi, Masoud, Anders Sildnes, Mehrdad Rakaee, Nikita Shvetsov, Lars Ailo Bongo, Lill-Tove Rasmussen Busund, and Kajsa Møllersen. "Deep Learning-Based Classification of Breast Cancer Molecular Subtypes from H&E Whole-Slide Images." Journal of Pathology Informatics 16 (2025): 100410.
- Valieris, Renan, Luan Martins, Alexandre Defelicibus, Adriana Passos Bueno, Cynthia Aparecida Bueno de Toledo Osorio, Dirce Carraro, Emmanuel Dias-Neto, Rafael A. Rosales, Jose Marcio Barros de Figueiredo, and Israel Tojal da Silva. "Weakly-Supervised Deep Learning Models Enable HER2-Low Prediction from H &E-Stained Slides." Breast Cancer Research 26, no. 1 (2024): 124.
- El Nahhas, Omar SM, Chiara ML Loeffler, Zunamys I. Carrero, Marko van Treeck, Fiona R. Kolbinger, Katherine J. Hewitt, Hannah S. Muti et al. "Regression-Based Deep-Learning Predicts Molecular Biomarkers from Pathology Slides." Nature communications 15, no. 1 (2024): 1253.
- Wahab, Noorul, Michael Toss, Islam M. Miligy, Mostafa Jahanifar, Nehal M. Atallah, Wenqi Lu, Simon Graham et al. "AI-Enabled Routine H&E Image Based Prognostic Marker for Early-Stage Luminal Breast Cancer." npj Precision Oncology 7, no. 1 (2023): 122.
- Niehues, Jan Moritz, Philip Quirke, Nicholas P. West, Heike I. Grabsch, Marko van Treeck, Yoni Schirris, Gregory P. Veldhuizen et al. "Generalizable Biomarker Prediction from Cancer Pathology Slides with Self-Supervised Deep Learning: A Retrospective Multi-Centric Study." Cell reports medicine 4, no. 4 (2023).
- He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. "Deep Residual Learning for Image Recognition." In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, 770-778.
- He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. "Identity Mappings in Deep Residual Networks." In European conference on computer vision, Cham: Springer International Publishing, 2016, 630-645.
- Niehues, Jan Moritz, Philip Quirke, Nicholas P. West, Heike I. Grabsch, Marko van Treeck, Yoni Schirris, Gregory P. Veldhuizen et al. "Generalizable Biomarker Prediction from Cancer Pathology Slides with Self-Supervised Deep Learning: A Retrospective Multi-Centric Study." Cell reports medicine 4, no. 4 (2023).
- Akbarnejad, Amir, Nilanjan Ray, Penny J. Barnes, and Gilbert Bigras. "Toward Accurate Deep Learning-Based Prediction of Ki67, ER, PR, and HER2 Status From H&E-Stained Breast Cancer Images." Applied Immunohistochemistry & Molecular Morphology 33, no. 3 (2025): 131-141.
- Hou, Jiaxin, Ranran Zhang, Yaoqin Xie, Chao Li, and Wenjian Qin. "Multimodal Deep Learning for Cancer Prognosis Prediction with Clinical Information Prompts Integration." npj Digital Medicine (2025).
