Unsupervised Machine Learning Methods for Anomaly Detection in Network Packets

Hyoseong Park, Dongil Shin, Chulgyun Park, Jisoo Jang, Dongkyoo Shin

Traditional intrusion detection systems (IDS) based on packet signatures are widely used in network security but often fail to detect previously unseen attacks. To overcome this limitation, machine learning-based methods have been explored to identify anomalous patterns in network traffic indicative of unknown intrusions. In this study, we propose an IDS model based on the Long Short-Term Memory Autoencoder (LSTM-AE), specifically a Convolutional Neural Network Bidirectional LSTM Autoencoder (CNN-BiLSTM-AE). The model integrates convolutional layers for spatial feature extraction and bidirectional LSTM layers to capture temporal dependencies in both directions. By leveraging CNNs to extract key spatial features and BiLSTM to model sequential patterns, the proposed architecture enables effective differentiation between normal and malicious traffic. Anomalies are detected by computing reconstruction loss during inference and applying a predefined threshold to classify traffic. The experimental results demonstrate that the CNN-BiLSTM-AE model achieves high detection performance, with an accuracy of 98.1% and an F1-score of 98.3%, highlighting its effectiveness in identifying previously unknown intrusions.