Open Access

ARTICLE

Dis-NDVW: Distributed Network Asset Detection and Vulnerability Warning Platform

Leilei Li¹, Yansong Wang², Dongjie Zhu^2,*, Xiaofang Li³, Haiwen Du⁴, Yixuan Lu², Rongning Qu³, Russell Higgs⁵

1 Artificial Intelligence Academy, Wuxi Vocational College of Science and Technology, Wuxi, 214068, China
2 School of Computer Science and Technology, Harbin Institute of Technology, Weihai, 264209, China
3 Department of Mathematics, Harbin Institute of Technology, Weihai, 264209, China
4 School of Astronautics, Harbin Institute of Technology, Harbin, 150001, China
5 School of Mathematics and Statistics, University College Dublin, Dublin, DO4 V1W8, Ireland

* Corresponding Author: Dongjie Zhu. Email:

Computers, Materials & Continua 2023, 76(1), 771-791. https://doi.org/10.32604/cmc.2023.038268

Received 05 December 2022; Accepted 14 April 2023; Issue published 08 June 2023

Abstract

With the rapid development of Internet technology, the issues of network asset detection and vulnerability warning have become hot topics of concern in the industry. However, most existing detection tools operate in a single-node mode and cannot parallelly process large-scale tasks, which cannot meet the current needs of the industry. To address the above issues, this paper proposes a distributed network asset detection and vulnerability warning platform (Dis-NDVW) based on distributed systems and multiple detection tools. Specifically, this paper proposes a distributed message subscription and publication system based on Zookeeper and Kafka, which endows Dis-NDVW with the ability to parallelly process large-scale tasks. Meanwhile, Dis-NDVW combines the RangeAssignor, RoundRobinAssignor, and StickyAssignor algorithms to achieve load balancing of task nodes in a distributed detection cluster. In terms of a large-scale task processing strategy, this paper proposes a task partitioning method based on First-In-First-Out (FIFO) queue. This method realizes the parallel operation of task producers and task consumers by dividing pending tasks into different queues according to task types. To ensure the data reliability of the task cluster, Dis-NDVW provides a redundant storage strategy for master-slave partition replicas. In terms of distributed storage, Dis-NDVW utilizes a distributed elastic storage service based on ElasticSearch to achieve distributed storage and efficient retrieval of big data. Experimental verification shows that Dis-NDVW can better meet the basic requirements of ultra-large-scale detection tasks.

---

Keywords

---