Yao Zhao¹, Jian Dong^1,*, Hongwei Liu¹, Jin Wu², Yanxin Liu¹

CMC-Computers, Materials & Continua, Vol.68, No.1, pp. 727-741, 2021, DOI:10.32604/cmc.2021.016462 - 22 March 2021

Abstract Efficient cache management plays a vital role in in-memory data-parallel systems, such as Spark, Tez, Storm and HANA. Recent research, notably research on the Least Reference Count (LRC) and Most Reference Distance (MRD) policies, has shown that dependency-aware caching management practices that consider the application’s directed acyclic graph (DAG) perform well in Spark. However, these practices ignore the further relationship between RDDs and cached some redundant RDDs with the same child RDDs, which degrades the memory performance. Hence, in memory-constrained situations, systems may encounter a performance bottleneck due to frequent data block replacement. In addition,… More >

Suzhen Wang¹, Yanpiao Zhang¹, Lu Zhang¹, Ning Cao^{2, *}, Chaoyi Pang³

CMC-Computers, Materials & Continua, Vol.56, No.3, pp. 415-431, 2018, DOI:10.3970/cmc.2018.03716

Abstract Spark is a fast unified analysis engine for big data and machine learning, in which the memory is a crucial resource. Resilient Distribution Datasets (RDDs) are parallel data structures that allow users explicitly persist intermediate results in memory or on disk, and each one can be divided into several partitions. During task execution, Spark automatically monitors cache usage on each node. And when there is a RDD that needs to be stored in the cache where the space is insufficient, the system would drop out old data partitions in a least recently used (LRU) fashion… More >