Real-Time Emotion Recognition in AR-Based HCI Using Deep Learning
Abstract
Augmented Reality (AR) worlds need that the interaction mechanisms should be intuitive and adapt to the emotional condition of the users to achieve better engagement.Current AR-based Human-Computer Interaction (HCI) systems primarily rely on gestures and voice commands, lacking the emotional intelligence needed for context-aware interaction. This study proposes a real-time emotion recognition system for AR-based HCI using deep learning. A multimodal framework combining Convolutional Neural Networks (CNNs) for spatial feature extraction and Long Short-Term Memory (LSTM) networks for temporal modeling is introduced. The system processes real-time video streams from AR headset cameras, applying transfer learning, pruning, and quantization-aware training for computational efficiency. A custom AR dataset of 5,000 samples across seven emotions was developed, with dual-labeling (self-report and expert annotation) to ensure reliability. The proposed system achieved 94.2% classification accuracy with an average processing latency of 15.3 ms per frame, outperforming baselines by 8.7% in accuracy and 23% in efficiency. User studies (N=50, demographically diverse) reported 89% satisfaction. This research establishes a foundation for emotion-aware AR interfaces, with potential applications in education, healthcare, gaming, and social platforms.