Open Access

ARTICLE

Optimizing Big Data Retrieval and Job Scheduling Using Deep Learning Approaches

Bao Rong Chang¹, Hsiu-Fen Tsai^2,*, Yu-Chieh Lin¹

1 Department of Computer Science and Information Engineering, National University of Kaohsiung, Kaohsiung, Taiwan
2 Department of Fragrance and Cosmetic Science, Kaohsiung Medical University, Kaohsiung, Taiwan

* Corresponding Author: Hsiu-Fen Tsai. Email:

(This article belongs to the Special Issue: Hybrid Intelligent Methods for Forecasting in Resources and Energy Field)

Computer Modeling in Engineering & Sciences 2023, 134(2), 783-815. https://doi.org/10.32604/cmes.2022.020128

Received 05 November 2021; Accepted 28 March 2022; Issue published 31 August 2022

Abstract

Big data analytics in business intelligence do not provide effective data retrieval methods and job scheduling that will cause execution inefficiency and low system throughput. This paper aims to enhance the capability of data retrieval and job scheduling to speed up the operation of big data analytics to overcome inefficiency and low throughput problems. First, integrating stacked sparse autoencoder and Elasticsearch indexing explored fast data searching and distributed indexing, which reduces the search scope of the database and dramatically speeds up data searching. Next, exploiting a deep neural network to predict the approximate execution time of a job gives prioritized job scheduling based on the shortest job first, which reduces the average waiting time of job execution. As a result, the proposed data retrieval approach outperforms the previous method using a deep autoencoder and Solr indexing, significantly improving the speed of data retrieval up to 53% and increasing system throughput by 53%. On the other hand, the proposed job scheduling algorithm defeats both first-in-first-out and memory-sensitive heterogeneous early finish time scheduling algorithms, effectively shortening the average waiting time up to 5% and average weighted turnaround time by 19%, respectively.

---

Keywords

---