Transforming E-commerce Experiences with Augmented Reality and Computer Vision: A Framework for Virtual Try-Ons and Home Decor Visualization

Abstract

In this paper, we present an end-to-end mobile system that integrates computer vision and augmented reality to enable AR views of home decor items in e-commerce as well as real-time virtual try-ons of t-shirts and eyewear. The system is implemented using Django with SQLite for backend integration and Flutter for cross-platform mobile release. The system solves the three main e-commerce problems of low engagement, uncertainty, and product misfit. Improving preprocessing for the VITON-HD model is one of the major contributions of this research. To accomplish appropriate data preparation, we refined human parsing, cloth segmentation, and pose estimation. The LabelMe tool was used to create a personal dataset of 1000 labeled images for training, and 200 images were used for testing. Body parts like hair, face, neck, upper_body, lower_body, left_hand, right_hand, and skirt were labeled in the images. YOLOv8x, which was trained using a human parsing model for this dataset, achieved mAP50/mAP50–95 scores of 0.899/0.808 for masks and 0.915/0.833 for bounding boxes, respectively. Better synthesis outcomes and efficient segmentation were made possible by the model. To properly place and size 2D glasses using alpha blending, we employed dlib's 68-point facial landmark detector to identify eye regions during the glasses try-on process. A robust model-viewer-plus package serves as the foundation for the AR visualization of home decor product functionality, enabling the use of augmented reality to display 3D models in the real world. By combining these features into a mobile application, the framework provides a simple and engaging way to encourage user satisfaction and confidence when shopping online.