Quote
A. Dada et al., “On the Impact of Cross-Domain Data on German Language Models,” in Findings of the Association for Computational Linguistics: EMNLP 2023, Stroudsburg: Association for Computational Linguistics (ACL), 2023, pp. 13801–13813. doi: 10.18653/v1/2023.findings-emnlp.922. Available: https://aclanthology.org/2023.findings-emnlp.922/
Content
Traditionally, large language models have been either trained on general web crawls or domain-specific data. However, recent successes of generative large language models, have shed light on the benefits of cross-domain datasets. To examine the significance of prioritizing data diversity over quality, we present a German dataset comprising texts from five domains, along with another dataset aimed at containing high-quality data. By training a series of models ranging between 122M and 750M parameters on both datasets, we conduct a comprehensive benchmark on multiple downstream tasks. Our findings demonstrate that the models trained on the cross-domain dataset outperform those trained on quality data alone, leading to improvements up to 4.45% over the previous state-of-the-art.
About the publication
Linked publications and references
References and Relationships
DOI 10.18653/v1/2023.findings-emnlp.922