ADAPTIVE HYBRID ENSEMBLE FRAMEWORK FOR REAL-TIME ANOMALY DETECTION IN LARGE-SCALE DATA STREAMS
Abstract
This paper presents an adaptive ensemble framework for real-time anomaly detection in large-scale data streams, addressing the challenges of concept drift, high-velocity data processing, and computational efficiency in modern distributed systems. We propose a Hybrid Statistical-Machine Learning Anomaly Detection (HSML-AD) algorithm that combines sliding window-based statistical analysis with incremental machine learning techniques. The framework employs a three-tier architecture: (1) lightweight statistical pre-filtering using modified Z-score and interquartile range methods, (2) adaptive feature extraction through exponential moving averages, and (3) ensemble classification using online random forest with dynamic weight adjustment based on recent prediction accuracy. Experimental evaluation on five benchmark datasets (KDD Cup 99, NSL-KDD, CICIDS2017, Yahoo S5, and Numenta Anomaly Benchmark) demonstrates that HSML-AD achieves an average F1-score of 94.3%, precision of 93.8%, and recall of 94.7%, outperforming baseline methods including Isolation Forest (F1: 87.2%), LSTM-Autoencoder (F1: 89.6%), and SPOT (F1: 86.4%). The algorithm maintains processing throughput of 127,000 records per second with average latency of 7.8 milliseconds on commodity hardware. The novelty lies in the adaptive weight mechanism that dynamically adjusts ensemble components based on data stream characteristics and recent performance, coupled with a memory-efficient incremental learning strategy that limits model size to 45 MB while maintaining detection accuracy. The proposed framework is applicable to network intrusion detection, IoT sensor monitoring, financial fraud detection, and industrial system health monitoring, particularly in resource-constrained environments requiring real-time processing.