Open Access

ARTICLE

Fast Access and Retrieval of Big Data Based on Unique Identification

by Wenshun Sheng^1,*, Aiping Xu², Shengli Wu³

1 Pujiang Institute, Nanjing Tech University, Nanjing, 211200, China
2 School of Computer, Wuhan University, Wuhan, 430072, China
3 School of Computing, Ulster University, Belfast, BT370QB, United Kingdom

* Corresponding Author: Wenshun Sheng. Email:

Intelligent Automation & Soft Computing 2022, 32(3), 1781-1795. https://doi.org/10.32604/iasc.2022.022571

Received 11 August 2021; Accepted 18 October 2021; Issue published 09 December 2021

Abstract

In big data applications, the data are usually stored in data files, whose data file structures, field structures, data types and lengths are not uniform. Therefore, if these data are stored in the traditional relational database, it is difficult to meet the requirements of fast storage and access. To solve this problem, we propose the mapping model between the source data file and the target HBase file. Our method solves the heterogeneity of the file object and the universality of the storage conversion. Firstly, based on the mapping model, we design “RowKey”, generation rules and algorithm. Then according to the mapping rules of data file fields with the HBase table column, the data in the data file are transformed into HBase. Finally, the retrieved keywords in “RowKey” are stored and used to achieve fast data retrieval by prefix matching or keyword matching method. Our method has been applied to different projects, which shows these results can be applied to the data conversion from regular row store data file to HBase distributed large data storage and has strong commonality. The method can be widely used in HBase big data storage applications.

---

Keywords

---