# Raising Bars, Not Parameters: LilMoo Compact Language Model for Hindi

Source: [Raising Bars, Not Parameters: LilMoo Compact Language Model for Hindi](<https://www.sota2.com/research/paper/69a90de82d603e6de3e4065d>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-03-06T05:00:05.533Z

Authors: Shiza Fatimah, Aniket Sen, Sophia Falk, Florian Mai, Lucie Flek, Nicholas Kluge Corr\\^ea

Published: 2026-03-03T20:31:25.000Z

## Abstract

The dominance of large multilingual foundation models has widened linguistic inequalities in Natural Language Processing (NLP), often leaving low-resource languages underrepresented. This paper introduces LilMoo, a 0.6-billion-parameter Hindi language model trained entirely from scratch to address this gap. Unlike prior Hindi models that rely on continual pretraining from opaque multilingual foundations, LilMoo is developed through a fully transparent and reproducible pipeline optimized for limited compute environments. We construct a high-quality Hindi corpus (GigaLekh) filtered through both heuristic and learned (LLM-as-a-judge) methods, complemented by bilingual augmentation with curated English data. Using this dataset, we explore various training recipes for small-scale language models. Across comprehensive evaluation suites, LilMoo consistently outperforms comparably sized multilingual baselines such as Qwen2.5-0.5B and Qwen3-0.6B, demonstrating that well-designed language-specific pretraining can rival large multilingual models at the sub-billion-parameter range.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Multitask Language Understanding on MILU](<https://www.sota2.com/research/sota/multitask-language-understanding-on-milu.md>): Multitask Language Understanding; MILU; Accuracy: 28.

- [Question Answering on ARC](<https://www.sota2.com/research/sota/question-answering-on-arc.md>): Question Answering; ARC; Accuracy: 29.

- [Multitask Language Understanding on MMLU (Accuracy)](<https://www.sota2.com/research/sota/multitask-language-understanding-on-mmlu-accuracy.md>): Multitask Language Understanding; MMLU; Accuracy: 27.

- [Physical Commonsense Reasoning on PIQA](<https://www.sota2.com/research/sota/physical-commonsense-reasoning-on-piqa.md>): Physical Commonsense Reasoning; PIQA; Accuracy: 58.

- [Commonsense Reasoning on HellaSwag](<https://www.sota2.com/research/sota/commonsense-reasoning-on-hellaswag.md>): Commonsense Reasoning; HellaSwag; Accuracy: 32.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2603.03508>)

[PDF](<https://arxiv.org/pdf/2603.03508.pdf>)
