Development of a Hybrid Anomaly-Based Intrusion Detection System Using Autoencoder and Isolation Forest

Abstract

As cyberattacks advance in sophistication and fluidity, modern intrusion detection systems (IDS) must progress from static, signature-based models to adaptive models that can detect known and zero-day threats (Labonne, 2020; Ali et al., 2023). This study proposes a hybrid anomaly-based IDS that integrates an Autoencoder (AE) for deep feature representation with an Isolation Forest (IF) for statistical anomaly scoring. The objective is to enhance detection performance for both known and zero-day attacks within an unsupervised learning framework. The proposed model is an adaptation of the hybrid AE–IF framework devised by Mohammed and Telek (2023) and the fog-computing adaptation by Sadaf and Sultana (2020). The hybrid model deploys a logical fusion framework—Hybrid OR and Hybrid AND—dynamically balancing precision and recall in anomaly detection. The model is implemented in Python and trained using unsupervised learning on two benchmark datasets, NSL-KDD and CICIDS2017, so the model operates without any prior knowledge of attack signatures (Engelen et al., 2021). Experimental responses to model anomaly detection capabilities support that the Hybrid OR configuration of the model provided the most balanced anomaly detection performance scores recording F1-score of 0.85 and 0.69 on NSL-KDD and CICIDS2017 datasets respectively—both hybrid performance metrics outperforming the stand-alone AE and IF models. These results are in line with more recent evidence supporting the position that combining feature-reconstruction learning with statistical isolation increases the resilience and adaptability towards network intrusion detection (Elsaid & Binbusayyis, 2024; Alhassan et al., 2024). While performance declined on the more complex CICIDS2017 dataset, the hybrid approach demonstrated improved generalisation relative to individual models. The results suggest that logical fusion of representation learning and isolation-based scoring provides a lightweight and adaptable framework for network intrusion detection, although further validation in live network environments is required before operational deployment (Abuabed et al., 2023; Wang et al., 2023).