Open Access

ARTICLE

Core – An Optimal Data Placement Strategy in Hadoop for Data Intentitive Applications Based on Cohesion Relation

Vengadeswaran^∗, Balasundaram

Department of Computer Applications, National Institute of Technology, Tiruchirappalli 620015, India

* Corresponding Author: E-mail:

Computer Systems Science and Engineering 2019, 34(1), 47-60. https://doi.org/10.32604/csse.2019.34.047

Abstract

The tremendous growth of data being generated today is making storage and computing a mammoth task. With its distributed processing capability Hadoop gives an efficient solution for such large data. Hadoop’s default data placement strategy places the data blocks randomly across the nodes without considering the execution parameters resulting in several lacunas such as increased execution time, query latency etc., Also, most of the data required for a task execution may not be locally available which creates data-locality problem. Hence we propose an innovative data placement strategy based on dependency of data blocks across the nodes. Our strategy dynamically analyses the history log and establishes relationship between various tasks and blocks required for each task through Block Dependency Graph (BDG). Then Our CORE-Algorithm re-organizes the HDFS layout by redistributing the data blocks to give an optimal data placement, resulting in improved performance for Big Data sets in distributed environment. This strategy is tested in 20-node cluster with different real-world MR applications. The results conclude that proposed strategy reduces the query execution time by 23%, improves the data locality by 50.7%, compared to default.

---

Keywords

---

Citations