Vol.68, No.1, 2021, pp.727-741, doi:10.32604/cmc.2021.016462
Improving Cache Management with Redundant RDDs Eviction in Spark
  • Yao Zhao1, Jian Dong1,*, Hongwei Liu1, Jin Wu2, Yanxin Liu1
1 School of Computer Science and Technology, Harbin Institute of Technology, Harbin, 150001, China
2 School of Engineering, University of Georgia, Athens, 30602, USA
* Corresponding Author: Jian Dong. Email:
Received 02 January 2021; Accepted 03 February 2021; Issue published 22 March 2021
Efficient cache management plays a vital role in in-memory data-parallel systems, such as Spark, Tez, Storm and HANA. Recent research, notably research on the Least Reference Count (LRC) and Most Reference Distance (MRD) policies, has shown that dependency-aware caching management practices that consider the application’s directed acyclic graph (DAG) perform well in Spark. However, these practices ignore the further relationship between RDDs and cached some redundant RDDs with the same child RDDs, which degrades the memory performance. Hence, in memory-constrained situations, systems may encounter a performance bottleneck due to frequent data block replacement. In addition, the prefetch mechanisms in some cache management policies, such as MRD, are hard to trigger. In this paper, we propose a new cache management method called RDE (Redundant Data Eviction) that can fully utilize applications’ DAG information to optimize the management result. By considering both RDDs’ dependencies and the reference sequence, we effectively evict RDDs with redundant features and perfect the memory for incoming data blocks. Experiments show that RDE improves performance by an average of 55% compared to LRU and by up to 48% and 20% compared to LRC and MRD, respectively. RDE also shows less sensitivity to memory bottlenecks, which means better availability in memory-constrained environments.
Dependency-aware; cache management; in-memory computing; spark
Cite This Article
Y. Zhao, J. Dong, H. Liu, J. Wu and Y. Liu, "Improving cache management with redundant rdds eviction in spark," Computers, Materials & Continua, vol. 68, no.1, pp. 727–741, 2021.
This work is licensed under a Creative Commons Attribution 4.0 International License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.