Abstract
Despite the state-of-the-art results obtained with transformer-based models like AraBERT, Arabic NER still appears to be very sensitive to the setting of hyperparameters. This usually requires a lot of manual tuning; the process is inefficient, time-consuming, and often results in suboptimal performance. In the context of this paper, an automatic framework for hyperparameter optimization was suggested using metaheuristic algorithms: PSO and TLBO. This is the first comparative research on the efficiency of metaheuristic algorithms for the optimization of Arabic NER. In this regard, this paper has applied these algorithms using the Wojood dataset and the aubmindlab/bert-base-arabertv2 to optimize the main hyperparameters: learning rate, batch size, and the number of epochs. The results revealed that PSO outperforms TLBO and a traditional approach for all test Micro-F1 scores, achieving a maximum of 0.8813, compared to 0.8755 for TLBO. Furthermore, PSO outperformed TLBO in terms of mean performance: 0.8708 versus 0.8561, with better convergence stability: a standard deviation of 0.0164 versus 0.0286. Although TLBO converged slightly faster than PSO, PSO demonstrated more robust generalization and reliability across runs. Overall, the findings confirm that metaheuristic optimization can substantially improve Arabic NER, with PSO standing out as the most effective, stable, and efficient optimizer for transformer-based Arabic NLP models.
References
Antoun, Wissam, Fady Baly, and Hazem Hajj. "Arabert: Transformer-based model for arabic language understanding." arXiv preprint arXiv:2003.00104 (2020).
Reynolds, Craig W. "Flocks, Herds and Schools: A Distributed Behavioral Model." In Proceedings of the 14th annual conference on Computer graphics and interactive techniques, 25-34. 1987.
Heppner, Frank C., and Ulf Grenander. 1990. “A Stochastic Nonlinear Model for Coordinated Bird Flocks.” In The Ubiquity of Chaos, edited by Stanley Krasner, 233–238. Washington, DC: AAAS Publications.
Eberhart, Russell, and James Kennedy. "A New Optimizer Using Particle Swarm Theory." In MHS'95. Proceedings of the sixth international symposium on micro machine and human science, pp. 39-43. Ieee, 1995.
Marini, Federico, and Beata Walczak. "Particle Swarm Optimization (PSO). A Tutorial." Chemometrics and intelligent laboratory systems 149 (2015): 153-165.
Kaveh, Ali, and Taha Bakhshpoori. "Teaching-Learning-Based Optimization Algorithm." In Metaheuristics: Outlines, MATLAB codes and examples, pp. 41-49. Cham: Springer International Publishing, 2019.
Rao, R. Venkata, Vimal J. Savsani, and Dipakkumar P. Vakharia. "Teaching–Learning-Based Optimization: A Novel Method for Constrained Mechanical Design Optimization Problems." Computer-aided design 43, no. 3 (2011): 303-315.
Balla, Husamelddin, and Sarah Jane Delany. "Exploration of Approaches to Arabic Named Entity Recognition." (2020). 2-16.
Muhammad, Marwa, Muhammad Rohaim, Alaa Hamouda, and Salah Abdel-Mageid. "A Comparison between Conditional Random Field and Structured Support Vector Machine for Arabic Named Entity Recognition." Journal of Computer Science 16, no. 1 (2020): 117-125.
Hudhud, Mohammad, Hamed Abdelhaq, Fadi Mohsen, A. Rocha, L. Steels, and J. van den Herik. "ArabiaNer: A System to Extract Named Entities from Arabic Content." In ICAART (1), pp. 489-497. 2021.
Alshammari, Nasser, and Saad Alanazi. "The Impact of Using Different Annotation Schemes on Named Entity Recognition." Egyptian Informatics Journal 22, no. 3 (2021): 295-302.
Elgamal, Marwa, Mohamed Abou-Kreisha, Reda Abo Elezz, and Salwa Hamada. "An Ontology-Based Name Entity Recognition NER and NLP Systems in Arabic Storytelling." Al-Azhar Bulletin of Science 31, no. 2-B (2020): 31-38.
Alkhatib, Manar, and Khaled Shaalan. "Boosting Arabic Named Entity Recognition Transliteration with Deep Learning." In FLAIRS, pp. 484-488. 2020.
Pasha, Arfath, Mohamed Al-Badrashiny, Mona T. Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. "Madamira: A Fast, Comprehensive Tool for Morphological Analysis and Disambiguation of Arabic." In Lrec, vol. 14, no. 2014, pp. 1094-1101. 2014.
Balla, Husamelddin, and Sarah Jane Delany. "Exploration of Approaches to Arabic Named Entity Recognition." (2020).
Helwe, Chadi, and Shady Elbassuoni. "Arabic Named Entity Recognition Via Deep Co-Learning." Artificial Intelligence Review 52, no. 1 (2019): 197-215.
Alkhatib, Manar, and Khaled Shaalan. "Boosting Arabic Named Entity Recognition Transliteration with Deep Learning." In FLAIRS, pp. 484-488. 2020.
Al-Smadi, Mohammad, Saad Al-Zboon, Yaser Jararweh, and Patrick Juola. "Transfer Learning for Arabic Named Entity Recognition with Deep Neural Networks." Ieee Access 8 (2020): 37736-37745.
Antoun, Wissam, Fady Baly, and Hazem Hajj. "Arabert: Transformer-Based Model for Arabic Language Understanding." arXiv preprint arXiv:2003.00104 (2020).
Abdul-Mageed, Muhammad, and AbdelRahim Elmadany. "ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic." In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 7088-7105. 2021.
Boudjellal, Nada, Huaping Zhang, Asif Khan, Arshad Ahmad, Rashid Naseem, Jianyun Shang, and Lin Dai. "ABioNER: A BERT‐Based Model for Arabic Biomedical Named‐Entity Recognition." Complexity 2021, no. 1 (2021): 1–6.
Kumar, Sunisth, Davide Liu, and Alexandre Boulenger. "Cross-Lingual NER for Financial Transaction Data in Low-Resource Languages." arXiv preprint arXiv:2307.08714 (2023).
Benajiba, Yassine, and Paolo Rosso. "Arabic Named Entity Recognition Using Conditional Random Fields." In Proc. of Workshop on HLT & NLP within the Arabic World, LREC, vol. 8, 143-153. 2008.
Salah, Ramzi Esmail, and Lailatul Qadri Binti Zakaria. "Building the Classical Arabic named Entity Recognition Corpus (CANERCorpus)." In 2018 Fourth International Conference on Information Retrieval and Knowledge Management (CAMP), pp. 1-8. IEEE, 2018.
Walker, Christopher, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2005. “ACE 2005 Multilingual Training Corpus.” Linguistic Data Consortium. https://catalog.ldc.upenn.edu/LDC2006T06.
Weischedel, Ralph, M. Palmer, M. Marcus, E. Hovy, S. Pradhan, L. Ramshaw, N. Xue et al. OntoNotes Release 5.0 LDC2013T19. Web Download. Philadelphia: Linguistic Data Consortium, 2013. 2013.
Jarrar, Mustafa. "The Arabic Ontology–An Arabic Wordnet with Ontologically Clean Content." Applied ontology 16, no. 1 (2021): 1-26.
Eiben, Agoston E., and James E. Smith. Introduction to Evolutionary Computing. springer, 2015.
Rajwar, Kanchan, Kusum Deep, and Swagatam Das. "An Exhaustive Review of the Metaheuristic Algorithms for Search and Optimization: Taxonomy, Applications, and Open Challenges." Artificial Intelligence Review 56, no. 11 (2023): 13187-13257.
Khalilia, Mohammed. 2024. COMP9312: Named Entity Recognition Project[Computer software]. GitHub. https://github.com/mohammedkhalilia/COMP9312.
Clerc, Maurice, and James Kennedy. "The Particle Swarm-Explosion, Stability, and Convergence in a Multidimensional Complex Space." IEEE transactions on Evolutionary Computation 6, no. 1 (2002): 58-73.