Abstract
Given the prevalence of pre-trained language models (PLMs) within the field of natural language processing, it has become evident that the conventional two-stage approach of
‘pre-training’
-then-
‘fine-tuning’
consistently yields commendable outcomes. Nevertheless, most publicly accessible PLMs are pre-trained on extensive, general-purpose datasets, thereby failing to address the substantial domain dissimilarity between the source and target data. This discrepancy has significant implications for the adaptability of PLMs to specific domains. To address this issue, our study proposes AdaFT, an efficient domain-adaptive fine-tuning framework that seeks to enhance the traditional fine-tuning process, thus bridging the gap between the source and target domains. This is particularly beneficial for enabling PLMs to better align with the specialized context of the Chinese financial domain. In contrast to the standard two-stage paradigm, AdaFT incorporates two additional stages:
’multi-task further pre-training’
and
’multi-model parameter fusion.’
In the first phase, the PLM undergoes a rapid, multi-task, parallel learning process, which effectively augments its proficiency in Chinese financial domain-related tasks. In the subsequent stage, we introduce an adaptive multi-model parameter fusion (AdaMFusion) strategy to amalgamate the knowledge acquired from the extended pre-training. To efficiently allocate weights for AdaMFusion, we have developed a local search algorithm with a decreasing step length, i.e., Local Search with Decreasing Step size (LSDS). The combination of AdaMFusion and LSDS algorithm strikes a balance between efficiency and performance, making it suitable for most scenarios. We also find that the optimal weighting factor assigned to a model to be fused is positively correlated with the performance improvement of that model on the target task after further pre-training. We demonstrate that further pre-training is generally effective, and further pre-training on domain-relevant corpora is more effective than on task-relevant corpora. Our extensive experiments, utilizing BERT (Bidirectional Encoder Representations from Transformers) as an illustrative example, indicate that Chinese BERT-base trained under the AdaFT framework attains an accuracy rate of 94.95% in the target task, marking a substantial 3.12% enhancement when compared to the conventional fine-tuning approach. Furthermore, our results demonstrate that AdaFT remains effective when applied to BERT-based variants, such as Chinese ALBERT-base.