Open Access

ARTICLE

EDSM-Based Binary Protocol State Machine Reversing

by Shen Wang^1,*, Fanghui Sun¹, Hongli Zhang¹, Dongyang Zhan^1,2, Shuang Li³, Jun Wang¹

1 School of Cyberspace Science, Harbin Institute of Technology, Harbin, 150001, China
2 The Ohio State University, Columbus, 43202, USA
3 Guangzhou University, Guangzhou, 510006, China

* Corresponding Author: Shen Wang. Email:

Computers, Materials & Continua 2021, 69(3), 3711-3725. https://doi.org/10.32604/cmc.2021.016562

Received 05 January 2021; Accepted 12 July 2021; Issue published 24 August 2021

Abstract

Internet communication protocols define the behavior rules of network components when they communicate with each other. With the continuous development of network technologies, many private or unknown network protocols are emerging in endlessly various network environments. Herein, relevant protocol specifications become difficult or unavailable to translate in many situations such as network security management and intrusion detection. Although protocol reverse engineering is being investigated in recent years to perform reverse analysis on the specifications of unknown protocols, most existing methods have proven to be time-consuming with limited efficiency, especially when applied on unknown protocol state machines. This paper proposes a state merging algorithm based on EDSM (Evidence-Driven State Merging) to infer the transition rules of unknown protocols in form of state machines with high efficiency. Compared with another classical state machine inferring method based on Exbar algorithm, the experiment results demonstrate that our proposed method could run faster, especially when dealing with massive training data sets. In addition, this method can also make the state machines have higher similarities with the reference state machines constructed from public specifications.

---

Keywords

---