Comparative Analysis of Machine Learning Algorithms for Malicious Network-Traffic Classification

Byron Wladimir Oviedo-Bayas, Stefany Michelle Perachimba Panezo, Jorge Humberto Guanin-Fajardo, Stalin Daniel Carreño Sandoya

The classification of malicious network-traffic is critical to cybersecurity. However, to the best of our knowledge, no previous studies have performed a comparative analysis of supervised algorithms for classifying malicious traffic, specifically within the network environment of UTEQ, an academic setting with distinctive traffic patterns and security policies. This study compared the performance of four supervised machine learning algorithms (K-Nearest Neighbors, Decision Tree, SVM-RBF, and SVM-Polynomial) using the CRISP-DM methodology. The dataset consisted of 1182 records with 30 variables from Hillstone Networks firewall logs at UTEQ, representing three categories: Normal (74.3%), Botnet_Activity (16.4%), and Other_Malware (9.3%). Preprocessing techniques included SMOTE balancing and Relief-based feature selection (reducing the variables to eight). The area under the curve (AUC) was used as a primary discriminant metric under two complementary one-vs-rest aggregation schemes. Using a support-weighted AUC, K-Nearest Neighbors (k = 7) obtained the highest value (AUC = 0.6147), followed by SVM-Polynomial (0.5846), Decision Tree (0.5724), and SVM-RBF (0.5784), with SVM-RBF obtaining the highest accuracy on the unified eight-feature test set (73.8%). Using a macro-averaged AUC, SVM-Polynomial obtained the highest value (0.6166), closely followed by KNN (0.6133). All AUC values fell within a narrow range (0.57–0.62). A class-wise analysis showed that no single model provides strong discrimination for all traffic classes, and that algorithm selection for operational deployment should be guided by the specific class of interest rather than by a single aggregate metric.