IRO Journals
Journal of Ubiquitous Computing and Communication Technologies

ORION: A Hierarchical Surgical World Model for Real-Time, Multi-Agent Operating-Room Intelligence

Open Access
Published: 12 August, 2026
Pages: 276-298
  • Dev Mehta , Dev Mehta
    SRM Institute of Science and Technology, Chennai, India
    SRM Institute of Science and Technology, Chennai, India
  • Charulatha R.T. Charulatha R.T.
    SRM Institute of Science and Technology, Chennai, India
    SRM Institute of Science and Technology, Chennai, India
view PDF
view PDF

How to Cite

Mehta, Dev, and Charulatha R.T. 2026. “ORION: A Hierarchical Surgical World Model for Real-Time, Multi-Agent Operating-Room Intelligence”. Journal of Ubiquitous Computing and Communication Technologies 8 (3): 276-98. https://doi.org/10.36548/jucct.2026.3.008.
ACM ACS APA ABNT Chicago Harvard IEEE MLA Turabian Vancouver AMA

Download Citation

Endnote/Zotero/Mendeley (RIS) BibTeX

Keywords

Surgical Intelligence
Digital Twin
Surgical World Model
Multi-Agent Systems
Surgical Phase Recognition
Operating-Room Analytics

Abstract

Modern operating suites generate dense, heterogeneous data streams endoscopic video, multi-parameter vitals, anesthesia registers, device telemetry, intraoperative imaging, and robotic kinematics yet the overwhelming majority of these signals are discarded, with the complete clinical reality of a procedure routinely collapsed into a single postoperative note. This documentation bottleneck loses roughly ninety-nine percent of the granular information produced during surgery, preventing systematic audit, objective skill assessment, and longitudinal research. We present ORION (Operating Room Intelligence and Optimization Network), an AI-native platform that transforms multi-sensory input into a continuously updated digital twin of the patient and operating environment. ORION is organised as a twelve-layer Hierarchical Surgical World Model (HSWM) spanning real-time perception, relational scene-graph construction, a Gaussian-splatting digital twin, temporal workflow modelling, predictive risk forecasting, knowledge grounding, and multi-agent clinical reasoning. A generative component, SurgWorld, mitigates kinematic-data scarcity by synthesising physically plausible surgical video and recovering pseudo-kinematics through an inverse-dynamics model. Because ORION is presented as a proposed architectural framework, the reported Cholec80 and AutoLaparo figures up to 94.6% and 89.5% accuracy at 238.7 FPS for the underlying state-space model are drawn from the constituent methods' published evaluations, not a new experiment; prospective validation of the integrated system is future work. The reporting layer is designed to compress postoperative documentation to under thirty seconds. We further describe the low-latency execution architecture (NVIDIA Holoscan and Triton) required to stay below the 50 ms perceptual threshold, and map the platform onto India's CDSCO Software-as-a-Medical-Device regulatory pathway

References

  1. Özsoy, Ege, Chantal Pellegrini, Tobias Czempiel, Felix Tristram, Kun Yuan, David Bani-Harouni, Ulrich Eck, Benjamin Busam, Matthias Keicher, and Nassir Navab. ”Mm-or: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environments.” IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2025, 19378-19389.
  2. Özsoy, Ege, Arda Mamur, Felix Tristram, Chantal Pellegrini, Magdalena Wysocki, Benjamin Busam, and Nassir Navab. ”Egoexor: An Ego-Exo-Centric Operating Room Dataset for Surgical Activity Understanding.” Advances in Neural Information Processing Systems 2026, vol. 38.
  3. NVIDIA. (2026). Deploy NVIDIA Holoscan: Real-Time Sensor AI for Medical Imaging and Industrial Inspection. NVIDIA Developer. https://www.spheron.network/blog/nvidia-holoscan-gpu-cloud-sensor-ai-guide/
  4. Shin, Jongmin, Enki Cho, Ka Young Kim, Jung Yong Kim, Seong Tae Kim, and Namkee Oh. ”Towards Holistic Surgical Scene Graph.” In International Conference on Medical Image Computing and Computer-Assisted Intervention, Cham: Springer Nature Switzerland, 2025, 617-626.
  5. Henriques, Angelo, Korab Hoxha, Daniel Zapp, Peter C. Issa, Nassir Navab, and M. Ali Nasseri. ”Decoding the Surgical Scene: A Scoping Review of Scene Graphs in Surgery.” Medical Image Analysis 2026, 104083.
  6. Oh, Sukju, and Sukkyu Sun. ”SurgicalMamba: Dual-Path SSD with State Regramming for Online Surgical Phase Recognition.”, 2026, arXiv preprint arXiv:2605.14889.
  7. Che, Chengan, Chao Wang, Xinyue Chen, Sophia Tsoka, and Luis C. Garcia-Peraza-Herrera. ”A Stitch in Time: Learning Procedural Workflow Via Self-Supervised Plackett-Luce Ranking.” IEEE/CVF Conference on Computer Vision and Pattern Recognition 2026, 17000-17010.
  8. Wang, Ziyi, Bo Lu, Yonghao Long, Fangxun Zhong, Tak-Hong Cheung, Qi Dou, and Yunhui Liu. ”Autolaparo: A New Dataset of Integrated Multi-Tasks for Image-Guided Surgical Automation in Laparoscopic Hysterectomy.” In International Conference on Medical Image Computing and Computer-Assisted Intervention, Cham: Springer Nature Switzerland 2022, 486-496.
  9. Rueckert, Tobias, David Rauber, Raphaela Maerkl, Leonard Klausmann, Suemeyye R. Yildiran, Max Gutbrod, Danilo Weber Nunes et al. ”Comparative Validation of Surgical Phase Recognition, Instrument Keypoint Estimation, and Instrument Instance Segmentation in Endoscopy: Results of the PhaKIR 2024 challenge.” Medical Image Analysis 2026, 103945.
  10. Long, Jun, Junkun Hong, Zidong Wang, Tingxuan Chen, Yunfei Chen, and Liu Yang. ”SPHASE: Multi-Modal and Multi-Branch Surgical Phase Segmentation Framework Based on Temporal Convolutional Network.” IEEE International Conference on Bioinformatics and Biomedicine (BIBM) 2023, 586-593.
  11. He, Yufan, Pengfei Guo, Mengya Xu, Zhaoshuo Li, Andriy Myronenko, Dillan Imans, Bingjie Liu et al. ”Cosmos-H-Surgical: Learning Surgical Robot Policies from Videos via World Modeling.”, 2025, arXiv preprint arXiv:2512.23162.
  12. Scalable Surgical World Modeling via Latent Action Prediction, 2025. OpenReview - https://openreview.net/forum?id=aYQYOVm2AB
  13. Lee, Gyubok, Elea Bach, Eric Yang, Tom Pollard, Alistair Johnson, Edward Choi, and Jong Ha Lee. ”FHIR-Agentbench: Benchmarking LLM Agents for Realistic Interoperable EHR Question Answering.”, 2025, arXiv preprint arXiv:2509.19319.
  14. De Maio, Carmen, Giuseppe Fenza, Domenico Furno, Teodoro Grauso, and Vincenzo Loia. ”Privacy-Preserving Healthcare Data Interactions: A Multi-Agent Approach Using LLMs.” Journal of Communications Software and Systems 2025, vol. 21, no. 1, 13-22.
  15. Borkowski, Andrew A., and Alon Ben-Ari. ”Multiagent AI Systems in Health Care: Envisioning Next-Generation Intelligence.” Federal Practitioner 2025, vol. 42, no. 5, 188.
  16. Agent orchestration: LangGraph vs CrewAI vs Temporal. (2026). Nirmitee.io Technical Report - https://nirmitee.io/blog/langgraph-crewai-temporal-custom-orchestration-healthcare-agents-2026/
  17. Fazzari, Edoardo, and Cesare Stefanini. ”Deep Reinforcement Learning for Surgical Robotics with State and Image Information: A Survey.” (2026).
  18. Ravi, Nikhila, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr et al. ”Sam 2: Segment Anything in Images and Videos.” In International Conference on Learning Representations 2025, vol. 2025, 28085-28128.
  19. Isensee, Fabian, Paul F. Jäger, Simon AA Kohl, Jens Petersen, and Klaus H. Maier-Hein. ”Automated Design of Deep Learning Methods for Biomedical Image Segmentation.”, 2019 arXiv preprint arXiv:1904.08128.
  20. NVIDIA. (2024). Cosmos World Foundation Model Platform for Physical AI. NVIDIA. https://research.nvidia.com/publication/2025-01_cosmos-world-foundation-modelplatform-physical-ai
  21. Central Drugs Standard Control Organization. (2017). Medical Devices Rules, 2017. Government of India. https://cdsco.gov.in/opencms/resources/UploadCDSCOWeb/ 2022/m_device/Medical%20Devices%20Rules,%202017.pdf
  22. Central Drugs Standard Control Organization. (2025). Draft Guidance Document On Medical Device Software Under MDR 2017. Government of India. https://cdsco.gov. in/opencms/export/sites/CDSCO_WEB/Pdf-documents/Guidance-document-on-MedicalDevice-Software-under-MDR-2017.pdf
  23. International Electrotechnical Commission. (2015). IEC 62304: Medical device software - Software life-cycle processes. IEC.
  24. International Organization for Standardization. (2019). ISO 14971: Medical devices - Application of risk management to medical devices. ISO.
  25. International Organization for Standardization. (2016). ISO 13485: Medical devices - Quality management systems - Requirements for regulatory purposes. ISO.