Open Access

ARTICLE

Design and Implementation of Log Data Analysis Management System Based on Hadoop

by Dunhong Yao, Yu Chen

1 School of Computer Science and Engineering, Huaihua University, Huaihua, 418000, China
2 Key Laboratory of Wuling-Mountain Health Big Data Intelligent Processing and Application in Hunan Province Universities, Huaihua, 418000, China
3 Key Laboratory of Intelligent Control Technology for Wuling-Mountain Ecological Agriculture in Hunan Province, Huaihua, 418000, China
4 School of Computer Science and Engineering, Yulin Normal University, Yulin, 537000, China

* Corresponding Author: Dunhong Yao. Email:

Journal of Information Hiding and Privacy Protection 2020, 2(2), 59-65. https://doi.org/10.32604/jihpp.2020.010223

Received 12 July 2020; Accepted 25 July 2020; Issue published 11 November 2020

Abstract

With the rapid development of the Internet, many enterprises have launched their network platforms. When users browse, search, and click the products of these platforms, most platforms will keep records of these network behaviors, these records are often heterogeneous, and it is called log data. To effectively to analyze and manage these heterogeneous log data, so that enterprises can grasp the behavior characteristics of their platform users in time, to realize targeted recommendation of users, increase the sales volume of enterprises’ products, and accelerate the development of enterprises. Firstly, we follow the process of big data collection, storage, analysis, and visualization to design the system, then, we adopt HDFS storage technology, Yarn resource management technology, and gink load balancing technology to build a Hadoop cluster to process the log data, and adopt MapReduce processing technology and data warehouse hive technology analyze the log data to obtain the results. Finally, the obtained results are displayed visually, and a log data analysis system is successfully constructed. It has been proved by practice that the system effectively realizes the collection, analysis and visualization of log data, and can accurately realize the recommendation of products by enterprises. The system is stable and effective.

---

Keywords

---