Submit

Login Login

Register Register

Home / Journals / CMES / Online First / doi:10.32604/cmes.2024.052622
Journal Menu
Special Issues
Table of Content

All issues

Online First

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

Open Access

ARTICLE

DPAL-BERT: A Faster and Lighter Question Answering Model

Lirong Yin1, Lei Wang1, Zhuohang Cai2, Siyu Lu2,*, Ruiyang Wang2, Ahmed AlSanad3, Salman A. AlQahtani3, Xiaobing Chen4, Zhengtong Yin5, Xiaolu Li6, Wenfeng Zheng2,3,*
1 Department of Geography and Anthropology, Louisiana State University, Baton Rouge, LA 70803, USA
2 School of Automation, University of Electronic Science and Technology of China, Chengdu, 610054, China
3 College of Computer and Information Sciences, King Saud University, Riyadh, 11574, Saudi Arabia
4 School of Electrical and Computer Engineering, Louisiana State University, Baton Rouge, LA 70803, USA
5 College of Resources and Environmental Engineering, Guizhou University, Guiyang, 550025, China
6 School of Geographical Sciences, Southwest University, Chongqing, 400715, China
* Corresponding Author: Siyu Lu. Email: email; Wenfeng Zheng. Email: email
(This article belongs to the Special Issue: Emerging Artificial Intelligence Technologies and Applications)

Computer Modeling in Engineering & Sciences https://doi.org/10.32604/cmes.2024.052622

Received 09 April 2024; Accepted 02 July 2024; Published online 23 July 2024

Abstract

Recent advancements in natural language processing have given rise to numerous pre-training language models in question-answering systems. However, with the constant evolution of algorithms, data, and computing power, the increasing size and complexity of these models have led to increased training costs and reduced efficiency. This study aims to minimize the inference time of such models while maintaining computational performance. It also proposes a novel Distillation model for PAL-BERT (DPAL-BERT), specifically, employs knowledge distillation, using the PAL-BERT model as the teacher model to train two student models: DPAL-BERT-Bi and DPAL-BERT-C. This research enhances the dataset through techniques such as masking, replacement, and n-gram sampling to optimize knowledge transfer. The experimental results showed that the distilled models greatly outperform models trained from scratch. In addition, although the distilled models exhibit a slight decrease in performance compared to PAL-BERT, they significantly reduce inference time to just 0.25% of the original. This demonstrates the effectiveness of the proposed approach in balancing model performance and efficiency.

Keywords

DPAL-BERT; question answering systems; knowledge distillation; model compression; BERT; Bi-directional long short-term memory (BiLSTM); knowledge information transfer; PAL-BERT; training efficiency; natural language processing

  • 55

    View

  • 9

    Download

  • 0

    Like

Related articles
Copyright© 2024 Tech Science Press
© 1997-2024 TSP (Henderson, USA) unless otherwise stated

Share Link