# WangchanLion and WangchanX MRC Eval

Source: [WangchanLion and WangchanX MRC Eval](<https://www.sota2.com/research/paper/68fa72fc1e209131cea59cb0>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-27T06:33:04.779Z

Authors: Wannaphong Phatthiyaphaibun, Surapon Nonesung, Patomporn Payoungkhamdee, Peerat Limkonchotiwat, Can Udomcharoenchaikit, Jitkapat Sawatphol, Chompakorn Chaksangchaichot, Ekapol Chuangsuwanich, Sarana Nutanong

Published: 2024-03-24T12:49:30.000Z

## Abstract

This technical report describes the development of WangchanLion, an instruction fine-tuned model focusing on Machine Reading Comprehension (MRC) in the Thai language. Our model is based on SEA-LION and a collection of instruction following datasets. To promote open research and reproducibility, we publicly release all training data, code, and the final model weights under the Apache-2 license. To assess the contextual understanding capability, we conducted extensive experimental studies using two Thai MRC datasets, XQuAD and Iapp\_wiki\_qa\_squad. Experimental results demonstrate the model's ability to comprehend the context and produce an answer faithful to the reference one in 0-shot and 1-shot settings. In addition, our evaluation goes beyond the traditional MRC. We propose a new evaluation scheme assessing the answer's correctness, helpfulness, conciseness, and contextuality. Our code is available publicly at https://github.com/vistec-AI/WangchanLion.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Average evaluation across 7 tasks on Thai Culture (test)](<https://www.sota2.com/research/sota/average-evaluation-across-7-tasks-on-thai-culture-test.md>): Average evaluation across 7 tasks; Thai Culture (test); BERTScore: 68.8.

- [Average Evaluation across 7 Tasks on General (test)](<https://www.sota2.com/research/sota/average-evaluation-across-7-tasks-on-general-test.md>): Average evaluation across 7 tasks; General (test); BERTScore: 72.2.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2403.16127>)

[PDF](<https://arxiv.org/pdf/2403.16127.pdf>)
