Enhancing Network Intrusion Detection Systems Using Deep Learning-Based Anomaly Detection Models
Abstract:
Deep learning has revolutionised pattern recognition by integrating feature extraction and classification into end-to-end models, eliminating the need for manual feature engineering. This survey paper reviews recent advancements in deep learning–based anomaly detection for network intrusion detection systems (NIDS). The review focuses on three major classes of models: convolutional neural networks (CNNs), recurrent architectures (particularly LSTMs), and autoencoders. Each class demonstrates distinct strengths in capturing spatial, temporal, and latent representations of network traffic. The paper synthesises key studies applying these models to anomaly-based intrusion detection, compares reported performance across commonly used benchmark datasets, and discusses their effectiveness in detecting previously unseen attacks. Finally, it highlights ongoing challenges such as class imbalance and concept drift, and outlines future research directions, including adversarial training and online adaptation, to enhance deep-learning-driven NIDS.
KeyWords:
Intrusion detection, anomaly detection, deep learning, convolutional neural networks, recurrent neural networks, autoencoders, cybersecurity.
References:
- Goodfellow, I., Bengio, Y., Courville, A. and Bengio, Y., 2016. Deep learning (Vol. 1, No. 2). Cambridge: MIT press.
- LeCun, Y., Bengio, Y. and Hinton, G., 2015. Deep learning. nature, 521(7553), pp.436-444.
- Hochreiter, S. and Schmidhuber, J., 1997. Long short-term memory. Neural computation, 9(8), pp.1735-1780.
- Denning, D.E., 1987. An intrusion-detection model. IEEE Transactions on software engineering, (2), pp.222-232.
- Wang, W., Sheng, Y., Wang, J., Zeng, X., Ye, X., Huang, Y. and Zhu, M., 2017. HAST-IDS: Learning hierarchical spatial-temporal features using deep neural networks to improve intrusion detection. IEEE access, 6, pp.1792-1806.
- Vinayakumar, R., Alazab, M., Soman, K.P., Poornachandran, P., Al-Nemrat, A. and Venkatraman, S., 2019. Deep learning approach for intelligent intrusion detection system. IEEE access, 7, pp.41525-41550.
- Moustafa, N. and Slay, J., 2015, November. UNSW-NB15: a comprehensive data set for network intrusion detection systems (UNSW-NB15 network data set). In 2015, military communications and information systems conference (MilCIS) (pp. 1-6). IEEE.
- Sharafaldin, I., Lashkari, A.H., and Ghorbani, A.A., 2018. Toward generating a new intrusion detection dataset and intrusion traffic characterization. ICISSp, 1(2018), pp.108-116.
- Rani, M., and Gagandeep, 2022. Effective network intrusion detection by addressing class imbalance with deep neural networks multimedia tools and applications. Multimedia Tools and Applications, 81(6), pp.8499-8518.
- Nawaz, M. H., Ahsan, A., Khan, I. U., Wang, Y., Ahmad, M., & Akhtar, M. S. (2025). Mitigating Message Injection Attacks in Internet of Vehicles Using Deep Learning Based Intrusion Detection System. ICCK Transactions on Advanced Computing and Systems, 1(4), 208-221.
- Al-Turaiki, I. and Altwaijry, N., 2021. A convolutional neural network for improved anomaly-based network intrusion detection. Big Data, 9(3), pp.233-252.
- Abdallah, M., An Le Khac, N., Jahromi, H. and Delia Jurcut, A., 2021, August. A hybrid CNN-LSTM based approach for anomaly detection systems in SDNs. In Proceedings of the 16th International Conference on Availability, Reliability and Security (pp. 1-7).
- Aljanabi, M. and Kumaran, N., 2024. Effective Intrusion Detection through Hybrid CNN-LSTM and Grey Wolf Optimization for Feature Selection in Complex Network Environments. GK International Journal of Advanced Research in Engineering and Technology, 1(1), pp.22-32.
- Gwon, H., Lee, C., Keum, R. and Choi, H., 2019. Network intrusion detection based on LSTM and feature embedding. arXiv preprint arXiv:1911.11552.
- Bhuyan, M.H., Bhattacharyya, D.K. and Kalita, J.K., 2013. Network anomaly detection: methods, systems, and tools. Ieee communications surveys & tutorials, 16(1), pp.303-336.
- Diro, A.A. and Chilamkurti, N., 2018. Distributed attack detection scheme using deep learning approach for Internet of Things. Future Generation Computer Systems, 82, pp.761-768.
- Shone, N., Ngoc, T.N., Phai, V.D. and Shi, Q., 2018. A deep learning approach to network intrusion detection. IEEE transactions on emerging topics in computational intelligence, 2(1), pp.41-50.
- Kim, J., Shin, N., Jo, S.Y. and Kim, S.H., 2017, February. Method of intrusion detection using deep neural network. In 2017 IEEE international conference on big data and smart computing (BigComp) (pp. 313-316). IEEE.
- Duong, H.T., Le, V.T. and Hoang, V.T., 2023. Deep learning-based anomaly detection in video surveillance: A survey. Sensors, 23(11), p.5024.
- Al-Zewairi, M., Almajali, S., & Ayyash, M. (2020). Unknown security attack detection using shallow and deep ANN classifiers. Electronics, 9(12), 2006.
- Javaid, A., Niyaz, Q., Sun, W. and Alam, M., 2016, May. A deep learning approach for network intrusion detection system. In Proceedings of the 9th EAI International Conference on Bio-inspired Information and Communications Technologies (formerly BIONETICS) (pp. 21-26).
- Kwon, D., Kim, H., Kim, J., Suh, S.C., Kim, I., and Kim, K.J., 2019. A survey of deep learning-based network anomaly detection. Cluster Computing, 22(Supple 1), pp.949-961.
- Khalaf, L.I., Alhamadani, B., Ismael, O.A., Radhi, A.A., Ahmed, S.R. and Algburi, S., 2024, May. Deep learning-based anomaly detection in network traffic for cyber threat identification. In Proceedings of the Cognitive Models and Artificial Intelligence Conference (pp. 303-309).
- Barrak, A., Petrillo, F. and Jaafar, F., 2022. Serverless on machine learning: A systematic mapping study. IEEE Access, 10, pp.99337-99352.
- Naqash, T., Shah, S.H. and Islam, M.N.U., 2022. Statistical analysis-based intrusion detection system for ultra-high-speed software-defined network. International Journal of Parallel Programming, 50(1), pp.89-114.
- Li, Z., Qin, Z., Huang, K., Yang, X. and Ye, S., 2017. Intrusion Detection Using Convolutional Neural Networks for Representation Learning. In: Neural Information Processing, Lecture Notes in Computer Science, vol. 10638, pp.858–866. Springer. doi:10.1007/978-3-319-70139-4_87.
- Narmadha, S., & Balaji, N. V. (2025). Improved network anomaly detection system using optimized autoencoder− LSTM. Expert Systems with Applications, 273, 126854.
- Singh, A., & Jang-Jaccard, J. (2022). Autoencoder-based unsupervised intrusion detection using multi-scale convolutional recurrent networks. arXiv preprint arXiv:2204.03779.
- Shafi, S. M., & Chinnappan, S. K. (2024). Hybrid transformer-CNN and LSTM model for lung disease segmentation and classification. PeerJ Computer Science, 10, e2444.
- Ahmed, S. F., Alam, M. S. B., Hassan, M., Rozbu, M. R., Ishtiak, T., Rafa, N., Mofijur, M., & Gandomi, A. H. (2023). Deep learning modelling techniques: current progress, applications, advantages, and challenges. Artificial Intelligence Review, 56(11), 13521-13617.
- Tan, Y., Hu, C., Zhang, K., Zheng, K., Davis, E. and Park, J., 2020. LSTM-based anomaly detection for non-linear dynamical system. IEEE Access. doi: 10.1109/ACCESS.2020.2999065.
- Yin, C., Zhu, Y., Fei, J. and He, X., 2017. A deep learning approach for intrusion detection using recurrent neural networks. IEEE Access, 5, pp.21954–21961. doi: 10.1109/ACCESS.2017.2762418.
- Al-Qatf, M., Lasheng, Y., Al-Habib, M. and Al-Sabahi, K., 2018. Deep learning approach combining sparse autoencoder with SVM for network intrusion detection. IEEE Access, 6, pp.52843–52856. doi: 10.1109/ACCESS.2018.2869577.
- Mirsky, Y., Doitshman, T., Elovici, Y. and Shabtai, A., 2018. Kitsune: An ensemble of autoencoders for online network intrusion detection. Proceedings of the Network and Distributed System Security Symposium (NDSS). doi: 10.14722/ndss.2018.23211.
- Wu, P. and Guo, H., 2019. LuNet: A deep neural network for network intrusion detection. arXiv preprint arXiv:1909.10031.
- Aygun, R.C. and Yavuz, A.G., 2017. Network anomaly detection with stochastically improved autoencoder based models. In 4th IEEE International Conference on Cyber Security and Cloud Computing (CSCloud) and 3rd IEEE International Conference on Scalable and Smart Cloud (SSC), pp.193–198.
- Yan, H., Zhang, W., Chen, Q., Li, X., Sun, W., Li, H. and Lin, X., 2023. RECESS Vaccine for Federated Learning: Proactive Defense Against Model Poisoning Attacks. arXiv preprint arXiv:2310.05431.