# GEM: Geometric Entropy Mixing for Optimal LLM Data Curation Source: [GEM: Geometric Entropy Mixing for Optimal LLM Data Curation]() Collection index: [llms.txt]() Data updated: 2026-06-02T05:00:05.013Z Authors: Yue Min, Ziyun Qiao, Ruining Chen, Yujun Li Published: 2026-04-27T06:42:28.000Z ## Abstract LLM pre-training efficacy increasingly depends on data composition rather than sheer volume. Yet, optimal mixing is hindered by categorization flaws: human taxonomies suffer from ontological misalignment, and Euclidean clustering fails to address embedding anisotropy. We introduce GEM (Geometric Entropy Mixing), a framework reformulating data curation as a variational problem on the hypersphere augmented with a mixing-balance regularizer. By decoupling the generative prior and optimizing the objective via a provable MM (Minorize-Maximize) algorithm, GEM effectively counteracts the cluster collapse to discover balanced semantic structures invisible to Euclidean heuristics. We employ teacher-student distillation to scale this geometric fidelity to web-scale corpora and introduce the Geometric Influence Score (GIS) for interpretable taxonomy generation. Experiments with 1.1B-parameter models demonstrate that GEM establishes a new state-of-the-art when integrated into mixing strategies like DoReMi and RegMix, improving average downstream accuracy by up to 1.2% and offering a robust coordinate system for predictable data mixing. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Zero-shot Evaluation on AI2 OLMES zero-shot](): Zero-shot Evaluation; AI2 OLMES zero-shot; ScienceQA: 35.96. ## Original sources and implementation [Original paper]() [PDF]()