# COLE: a Comprehensive Benchmark for French Language Understanding Evaluation

Source: [COLE: a Comprehensive Benchmark for French Language Understanding Evaluation](<https://www.sota2.com/research/paper/68fa7bee1e209131cea8b627>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T05:52:16.996Z

Authors: David Beauchemin, Yan Tremblay, Mohamed Amine Youssef, Richard Khoury

Published: 2025-10-06T17:26:41.000Z

## Abstract

To address the need for a more comprehensive evaluation of French Natural Language Understanding (NLU), we introduce COLE, a new benchmark composed of 23 diverse task covering a broad range of NLU capabilities, including sentiment analysis, paraphrase detection, grammatical judgment, and reasoning, with a particular focus on linguistic phenomena relevant to the French language. We benchmark 94 large language models (LLM), providing an extensive analysis of the current state of French NLU. Our results highlight a significant performance gap between closed- and open-weights models and identify key challenging frontiers for current LLMs, such as zero-shot extractive question-answering (QA), fine-grained word sense disambiguation, and understanding of regional language variations. We release COLE as a public resource to foster further progress in French language modelling.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2510.05046>)

[PDF](<https://arxiv.org/pdf/2510.05046.pdf>)
