A comparative study of model architectures on multilingual comprehension capabilities
Yangchen Ji
Sinopec Research Institute of Petroleum Processing
DOI: https://doi.org/10.59429/esta.v12i2.10557
Keywords: Large language models; Monolingual training; Model evaluation
Abstract
This study investigates the differences in Chinese and English comprehension capabilities between the open-source large language models Llama3-8B (Meta) and GLM-4-9B (Zhipu AI) using a controlled variable approach. Employing a unified BPE tokenizer and cleaned monolingual corpora (850M tokens for Chinese, 1.1B tokens for English), both models were pretrained under identical hyperparameters (learning rate: 3e-5, batch size: 32, epochs: 10). Performance was evaluated primarily using the F1 score on text classification tasks. Results indicate that GLM-4-9B significantly outperformed Llama3-8B on Chinese tasks (F1: 92.3% vs. 89.7%), while both models exhibited comparable performance on English tasks (F1: 94.1% vs. 93.8%). This suggests that the Autoregressive Blank Infilling objective in GLM’s architecture may be better suited to the syntactic characteristics of Chinese. The study notes that limitations in dataset scale and hyperparameter optimization depth warrant further validation of these conclusions
References
[1]Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; Polosukhin, I. Attention Is All You Need. arXiv August 2, 2023. https://doi.org/10.48550/arXiv.1706.03762.
[2]Limisiewicz, T.; Balhar, J.; Mareček, D. Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages. arXiv May 26, 2023. https://doi.org/10.48550/arXiv.2305.17179.
[3]Lample, G.; Conneau, A. Cross-Lingual Language Model Pretraining. arXiv January 22, 2019. https://doi.org/10.48550/arXiv.1901.07291.
[4]AI@Meta. Llama 3 Model Card. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md.
[5]Ainslie, J.; Lee-Thorp, J.; Jong, M. de; Zemlyanskiy, Y.; Lebrón, F.; Sanghai, S. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. arXiv December 23, 2023. https://doi.org/10.48550/arXiv.2305.13245.
[6]GLM, T.; Zeng, A.; Xu, B.; Wang, B.; Zhang, C.; Yin, D.; Zhang, D.; Rojas, D.; Feng, G.; Zhao, H.; Lai, H.; Yu, H.; Wang, H.; Sun, J.; Zhang, J.; Cheng, J.; Gui, J.; Tang, J.; Zhang, J.; Sun, J.; Li, J.; Zhao, L.; Wu, L.; Zhong, L.; Liu, M.; Huang, M.; Zhang, P.; Zheng, Q.; Lu, R.; Duan, S.; Zhang, S.; Cao, S.; Yang, S.; Tam, W. L.; Zhao, W.; Liu, X.; Xia, X.; Zhang, X.; Gu, X.; Lv, X.; Liu, X.; Liu, X.; Yang, X.; Song, X.; Zhang, X.; An, Y.; Xu, Y.; Niu, Y.; Yang, Y.; Li, Y.; Bai, Y.; Dong, Y.; Qi, Z.; Wang, Z.; Yang, Z.; Du, Z.; Hou, Z.; Wang, Z. ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools. arXiv July 30, 2024. https://doi.org/10.48550/arXiv.2406.12793.
[7]Du, Z.; Qian, Y.; Liu, X.; Ding, M.; Qiu, J.; Yang, Z.; Tang, J. GLM: General Language Model Pretraining with Autoregressive Blank Infilling. arXiv March 17, 2022. https://doi.org/10.48550/arXiv.2103.10360.
[8]Sennrich, R.; Haddow, B.; Birch, A. Neural Machine Translation of Rare Words with Subword Units. arXiv June 10, 2016. https://doi.org/10.48550/arXiv.1508.07909.
[9]Kudo, T.; Richardson, J. SentencePiece: A Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing. arXiv August 19, 2018. https://doi.org/10.48550/arXiv.1808.06226.