Open Access

ARTICLE

TLSmell: Direct Identification on Malicious HTTPs Encryption Traffic with Simple Connection-Specific Indicators

by Zhengqiu Weng^1,2, Timing Chen^1,*, Tiantian Zhu¹, Hang Dong¹, Dan Zhou¹, Osama Alfarraj³

1 School of Computer Science and Technology, Zhejiang University of Technology, Hangzhou, 310023, China
2 Department of Information Technology, Wenzhou Polytechnic, Wenzhou, 325035, China
3 Computer Science Department, Community College, King Saud University, Riyadh, 11437, Saudi Arabia

* Corresponding Author: Timing Chen. Email:

Computer Systems Science and Engineering 2021, 37(1), 105-119. https://doi.org/10.32604/csse.2021.015074

Received 05 November 2020; Accepted 13 December 2020; Issue published 05 February 2021

Abstract

Internet traffic encryption is a very common traffic protection method. Most internet traffic is protected by the encryption protocol called transport layer security (TLS). Although traffic encryption can ensure the security of communication, it also enables malware to hide its information and avoid being detected. At present, most of the malicious traffic detection methods are aimed at the unencrypted ones. There are some problems in the detection of encrypted traffic, such as high false positive rate, difficulty in feature extraction, and insufficient practicability. The accuracy and effectiveness of existing methods need to be improved. In this paper, we present TLSmell, a framework that conducts malicious encrypted HTTPs traffic detection with simple connection-specific indicators by using different classifiers based online training. We perform deep packet analysis of encrypted traffic through data pre-processing to extract effective features, and then the online training algorithm is used for training and prediction. Without decrypting the original traffic, high-precision malicious traffic detection and analysis are realized, which can guarantee user privacy and communication security. At the same time, since there is no need to decrypt the traffic in advance, the efficiency of detecting malicious HTTPs traffic will be greatly improved. Combined with the traditional detection and analysis methods, malicious HTTPs traffic is screened, and suspicious traffic is further analyzed by the expert through the context of suspicious behaviors, thereby improving the overall performance of malicious encrypted traffic detection.

---

Keywords

---