Open AccessOpen Access


A Resource-constrained Edge IoT Device Data-deduplication Method with Dynamic Asymmetric Maximum

Ye Yang1, Xiaofang Li2, Dongjie Zhu3,*, Hao Hu3, Haiwen Du4, Yundong Sun4, Weiguo Tian3, Yansong Wang3, Ning Cao1, Gregory M.P. O’Hare5

1 School of Artificial Intelligence, Wuxi Vocational College of Science and Technology, Wuxi, 214028, China
2 Department of Mathematics, Harbin Institute of Technology, Weihai, 264209, China
3 School of Computer Science and Technology, Harbin Institute of Technology, Weihai, 264200, China
4 School of Astronautics, Harbin Institute of Technology, Harbin, 150001, China
5 School of Computer Science, University College Dublin, Dublin, Dublin4, Ireland

* Corresponding Author: Dongjie Zhu. Email:

Intelligent Automation & Soft Computing 2021, 30(2), 481-494.


Smart vehicles use sophisticated sensors to capture real-time data. Due to the weak communication capabilities of wireless sensors, these data need to upload to the cloud for processing. Sensor clouds can resolve these drawbacks. However, there is a large amount of redundant data in the sensor cloud, occupying a large amount of storage space and network bandwidth. Deduplication can yield cost savings by storing one data copy. Chunking is essential because it can determine the performance of deduplication. Content-Defined Chunking (CDC) can effectively solve the problem of chunk boundaries shifted, but it occupies a lot of computing resources and has become a bottleneck in deduplication technology. This paper proposes a Dynamic Asymmetric Maximum algorithm (DAM), which uses the maximum value as the chunk boundaries and reducing the impact of the low-entropy string. It also uses the perfect hash algorithm to optimize the chunk search. Experiments show that our solution can effectively detect low-entropy strings in redundant data, save storage resources, and improve sensor clouds system throughput.


Cite This Article

Y. Yang, X. Li, D. Zhu, H. Hu, H. Du et al., "A resource-constrained edge iot device data-deduplication method with dynamic asymmetric maximum," Intelligent Automation & Soft Computing, vol. 30, no.2, pp. 481–494, 2021.

This work is licensed under a Creative Commons Attribution 4.0 International License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
  • 1028


  • 535


  • 0


Share Link

WeChat scan