IRO Journals
Journal of Ubiquitous Computing and Communication Technologies

A Language-Routed Speech-to-Action Framework for Code-Switched Indic Voice Assistants

Open Access
Published: 11 November, 2026
Pages: 428-443
  • Shubham Pankaj Badhe , Shubham Pankaj Badhe
    Department of Artificial Intelligence and Data Science, PVG’s College of Engineering, Technology and Management, Savitribai Phule Pune University, Pune, India.
    Department of Artificial Intelligence and Data Science, PVG’s College of Engineering, Technology and Management, Savitribai Phule Pune University, Pune, India.
  • Kabeer Ahmed Merchant , Kabeer Ahmed Merchant
    Department of Artificial Intelligence and Data Science, PVG’s College of Engineering, Technology and Management, Savitribai Phule Pune University, Pune, India.
    Department of Artificial Intelligence and Data Science, PVG’s College of Engineering, Technology and Management, Savitribai Phule Pune University, Pune, India.
  • Shreyas Nishad Deshmukh , Shreyas Nishad Deshmukh
    Department of Artificial Intelligence and Data Science, PVG’s College of Engineering, Technology and Management, Savitribai Phule Pune University, Pune, India.
    Department of Artificial Intelligence and Data Science, PVG’s College of Engineering, Technology and Management, Savitribai Phule Pune University, Pune, India.
  • Shravani Bankar , Shravani Bankar
    Department of Artificial Intelligence and Data Science, PVG’s College of Engineering, Technology and Management, Savitribai Phule Pune University, Pune, India.
    Department of Artificial Intelligence and Data Science, PVG’s College of Engineering, Technology and Management, Savitribai Phule Pune University, Pune, India.
  • Minakshi Atre Minakshi Atre
    Head of Department, Department of Artificial Intelligence and Data Science, PVG’s College of Engineering, Technology and Management, Savitribai Phule Pune University, Pune, India.
    Head of Department, Department of Artificial Intelligence and Data Science, PVG’s College of Engineering, Technology and Management, Savitribai Phule Pune University, Pune, India.
view PDF
view PDF

How to Cite

Badhe, Shubham Pankaj, Kabeer Ahmed Merchant, Shreyas Nishad Deshmukh, Shravani Bankar, and Minakshi Atre. 2026. “A Language-Routed Speech-to-Action Framework for Code-Switched Indic Voice Assistants”. Journal of Ubiquitous Computing and Communication Technologies 8 (4): 428-43. https://doi.org/10.36548/jucct.2026.4.009.
ACM ACS APA ABNT Chicago Harvard IEEE MLA Turabian Vancouver AMA

Download Citation

Endnote/Zotero/Mendeley (RIS) BibTeX

Keywords

Indic ASR
Multilingual Voice Assistant
Code-Switching
Agentic Tool Orchestration
Speech Language Models.

Abstract

There are 22 Scheduled Languages in India, along with code-switching, and accents in different dialects make voice the most viable medium for many users of digital public services. There have been remarkable strides in Automatic Speech Recognition (ASR) technology for Indic languages, but most assessments are still limited to word error rate (WER). Conversely, LLM agents rely on clean input text and thus conceal the compound failure when the spoken and code-switched object is mapped to the Application Programming Interface (API). In this paper, we introduce IndicAgent-ASR, a cascade of acoustic-to-agent system that performs language/code switch detection, routes speech to Indic or English ASR models, canonicalizes multilingual slots, plans for tool call or dependent tool call, runs a safety filter, and generates a spoken response using Indic text-to-speech. We specify the architecture, an end-to-end flowchart, a turn-level algorithm, and an easily reproducible software stack. Published benchmarking results include multilingual and code-switching baselines for MUCS 2021 with WER scores of 30.73% and 32.45%, best-performing accent models with WER ranging from 10.5% to 20.5% in the LAHAJA benchmark, and parameter filling accuracy of 60.6% (English) against 39.2% (Indic) in VoiceAgentBench using the strongest ASR-LLM pipeline, where sequential dependent calls drop to 10.0% and 4.3%. The feasibility prototype of the Hindi/Hinglish and Marathi language features mandi, weather, e-health record, and balance check tools, but denies voice fund transfer and prompt injection tools. The evidence supports keeping a strong Indic ASR front-end in the cascade and treating dependent tools and safety as first-class modules.

References

  1. Javed, Tahir, Janki Nawale, Eldho George, Sakshi Joshi, Kaushal Bhogale, Deovrat Mehendale, Ishvinder Sethi et al. "Indicvoices: Towards building an inclusive multilingual speech dataset for indian languages." In Findings of the Association for Computational Linguistics: 2024, vol. ACL 2024, 10740-10782.
  2. Baevski, Alexei, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. "wav2vec 2.0: A framework for self-supervised learning of speech representations." Advances in neural information processing systems. 2020, vol. 33: 12449-12460.
  3. Javed, Tahir, Sumanth Doddapaneni, Abhigyan Raman, Kaushal Santosh Bhogale, Gowtham Ramesh, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh M. Khapra. "Towards building asr systems for the next billion users." In Proceedings of the aaai conference on artificial intelligence, 20222, vol. 36, no. 10: 10813-10821.
  4. Radford, Alec, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. "Robust speech recognition via large-scale weak supervision." In International conference on machine learning, 2023, vol. 202:28492-28518.
  5. Bawa, Anshul, Pranav Khadpe, Pratik Joshi, Kalika Bali, and Monojit Choudhury. "Do multilingual users prefer chat-bots that code-mix? Let's nudge and find out!." Proceedings of the ACM on Human-Computer Interaction. 2020. vol. 4 no: 1-23.
  6. Javed, Tahir, Kaushal Bhogale, Abhigyan Raman, Pratyush Kumar, Anoop Kunchukuttan, and Mitesh M. Khapra. "Indicsuperb: A speech processing universal performance benchmark for indian languages." In Proceedings of the AAAI conference on artificial intelligence, 2023, vol. 37, no. 11: 12942-12950.
  7. Sankar, Ashwin, Sparsh Jain, Nikhil Narasimhan, Devilal Choudhary, Dhairya Suman, Mohammed Safi Ur Rahman Khan, Anoop Kunchukuttan, Mitesh M. Khapra, and Raj Dabre. "Towards building large scale datasets and state-of-the-art automatic speech translation systems for 14 Indian languages." In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics. 2025, vol. 1 no: 32945-32966.
  8. Hemant, P., and Meera Narvekar. "Development of a code-switched Hindi-Marathi dataset and transformer-based architecture for enhanced speech recognition using dynamic switching algorithms." Applied Acoustics.2025, vol. 230 no: 110408.
  9. Jain, Palash, and Anirban Bhowmick. "Analyzing code-switching scenarios in india’s diverse linguistic landscape using end-to-end asr systems with vitb-hebic." Computers and Electrical Engineering. 2025, vol. 122: 109978.
  10. Sethiya, Nivedita, Puneet Walia, and Chandresh Kumar Maurya. "Indic-S2ST: a Multilingual and Multimodal Many-to-Many Indic Speech-to-Speech Translation Dataset." In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics.2025, no. 3766-3775.