List your company

Hot-Start Chinese Language Modeling:Visual Glyphs Accelerate Sample-Efficient Learning

About

In this work, we study whether rendering Chinese characters as visual glyph images, rather than discrete token IDs as mainstream LLMs do, providing an inductive bias for character-level language modeling. Our central finding gives a double-edged insight: visual inputs produce a pronounced hot-start effect, more than doubling early-stage accuracy within the first epoch (at 0.4% of total training steps) (12.3% visual inputs vs. 5.8% index-based baseline), yet both approaches converge to essentially identical final accuracy (39%). This pattern holds across resolutions as low as 8x8 pixels, partial cropping up to 50%, and model scales from 110M to 1.78B parameters. The mechanism we identify is that glyph rendering pre-encodes radical-based structure into embedding space before any training (cosine similarity 0.27 vs. 0.002 for random embeddings), enabling faster alignment but not higher final capacity. Our results clarify both the promise and fundamental limitation of visual representations as inductive biases for Chinese language modeling.

Key insights

What problem does this paper study?

It tests whether rendering Chinese characters as images (visual glyphs) instead of arbitrary token IDs helps language models learn. The key finding is a hot-start effect: visual inputs double early accuracy but converge to the same final accuracy as token IDs.

How does this differ from prior work?

Unlike prior glyph-augmented methods that keep token IDs as primary input, this work replaces IDs entirely with visual glyphs to isolate learning dynamics. At 8,200 samples, 8×8 visual input reaches 12.34% accuracy versus 5.84% for the index baseline.

What are the practical uses of these findings?

The authors suggest visual glyph initialization could serve as a lightweight warm-start for low-resource Chinese fine-tuning. On C-Eval, visual models at 10K samples surpass the fully trained index baseline (25.0% vs 22.3%).

What limitations do the authors acknowledge?

The index-based baseline uses randomly initialized character-level embeddings, not subword tokenization or pretrained representations. The comparison against BPE-tokenized or pretrained baselines remains future work; the authors hypothesize the hot-start advantage would be reduced there.

Related benchmarks