Open Access

ARTICLE

A Resource-constrained Edge IoT Device Data-deduplication Method with Dynamic Asymmetric Maximum

Ye Yang¹, Xiaofang Li², Dongjie Zhu^3,*, Hao Hu³, Haiwen Du⁴, Yundong Sun⁴, Weiguo Tian³, Yansong Wang³, Ning Cao¹, Gregory M.P. O’Hare⁵

1 School of Artificial Intelligence, Wuxi Vocational College of Science and Technology, Wuxi, 214028, China
2 Department of Mathematics, Harbin Institute of Technology, Weihai, 264209, China
3 School of Computer Science and Technology, Harbin Institute of Technology, Weihai, 264200, China
4 School of Astronautics, Harbin Institute of Technology, Harbin, 150001, China
5 School of Computer Science, University College Dublin, Dublin, Dublin4, Ireland

* Corresponding Author: Dongjie Zhu. Email:

Intelligent Automation & Soft Computing 2021, 30(2), 481-494. https://doi.org/10.32604/iasc.2021.019201

Received 06 April 2021; Accepted 13 May 2021; Issue published 11 August 2021

Abstract

Smart vehicles use sophisticated sensors to capture real-time data. Due to the weak communication capabilities of wireless sensors, these data need to upload to the cloud for processing. Sensor clouds can resolve these drawbacks. However, there is a large amount of redundant data in the sensor cloud, occupying a large amount of storage space and network bandwidth. Deduplication can yield cost savings by storing one data copy. Chunking is essential because it can determine the performance of deduplication. Content-Defined Chunking (CDC) can effectively solve the problem of chunk boundaries shifted, but it occupies a lot of computing resources and has become a bottleneck in deduplication technology. This paper proposes a Dynamic Asymmetric Maximum algorithm (DAM), which uses the maximum value as the chunk boundaries and reducing the impact of the low-entropy string. It also uses the perfect hash algorithm to optimize the chunk search. Experiments show that our solution can effectively detect low-entropy strings in redundant data, save storage resources, and improve sensor clouds system throughput.

---

Keywords

---