Open Access

ARTICLE

Anomaly Detection in Imbalanced Encrypted Traffic with Few Packet Metadata-Based Feature Extraction

Min-Gyu Kim¹, Hwankuk Kim^2,*

1 Department of Financial Information Security, Kookmin University, Seoul, 02707, Republic of Korea
2 Department of Information Security Cryptography Mathematics, Kookmin University, Seoul, 02707, Republic of Korea

* Corresponding Author: Hwankuk Kim. Email:

(This article belongs to the Special Issue: Advanced Security for Future Mobile Internet: A Key Challenge for the Digital Transformation)

Computer Modeling in Engineering & Sciences 2024, 141(1), 585-607. https://doi.org/10.32604/cmes.2024.051221

Received 29 February 2024; Accepted 19 June 2024; Issue published 20 August 2024

Abstract

In the IoT (Internet of Things) domain, the increased use of encryption protocols such as SSL/TLS, VPN (Virtual Private Network), and Tor has led to a rise in attacks leveraging encrypted traffic. While research on anomaly detection using AI (Artificial Intelligence) is actively progressing, the encrypted nature of the data poses challenges for labeling, resulting in data imbalance and biased feature extraction toward specific nodes. This study proposes a reconstruction error-based anomaly detection method using an autoencoder (AE) that utilizes packet metadata excluding specific node information. The proposed method omits biased packet metadata such as IP and Port and trains the detection model using only normal data, leveraging a small amount of packet metadata. This makes it well-suited for direct application in IoT environments due to its low resource consumption. In experiments comparing feature extraction methods for AE-based anomaly detection, we found that using flow-based features significantly improves accuracy, precision, F1 score, and AUC (Area Under the Receiver Operating Characteristic Curve) score compared to packet-based features. Additionally, for flow-based features, the proposed method showed a 30.17% increase in F1 score and improved false positive rates compared to Isolation Forest and OneClassSVM. Furthermore, the proposed method demonstrated a 32.43% higher AUC when using packet features and a 111.39% higher AUC when using flow features, compared to previously proposed oversampling methods. This study highlights the impact of feature extraction methods on attack detection in imbalanced, encrypted traffic environments and emphasizes that the one-class method using AE is more effective for attack detection and reducing false positives compared to traditional oversampling methods.

---

Keywords

---