# Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation

Source: [Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation](<https://www.sota2.com/research/paper/68fa6f931e209131cea465dc>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-02T02:51:24.940Z

Authors: Marius Mosbach, Tiago Pimentel, Shauli Ravfogel, Dietrich Klakow, Yanai Elazar

Published: 2023-05-26T13:55:17.000Z

## Abstract

Few-shot fine-tuning and in-context learning are two alternative strategies for task adaptation of pre-trained language models. Recently, in-context learning has gained popularity over fine-tuning due to its simplicity and improved out-of-domain generalization, and because extensive evidence shows that fine-tuned models pick up on spurious correlations. Unfortunately, previous comparisons of the two approaches were done using models of different sizes. This raises the question of whether the observed weaker out-of-domain generalization of fine-tuned models is an inherent property of fine-tuning or a limitation of the experimental setup. In this paper, we compare the generalization of few-shot fine-tuning and in-context learning to challenge datasets, while controlling for the models used, the number of examples, and the number of parameters, ranging from 125M to 30B. Our results show that fine-tuned language models can in fact generalize well out-of-domain. We find that both approaches generalize similarly; they exhibit large variation and depend on properties such as model size and the number of examples, highlighting that robust task adaptation remains a challenge.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Cross-lingual Cultural Consistency on BLEnD All 8 Languages](<https://www.sota2.com/research/sota/cross-lingual-cultural-consistency-on-blend-all-8-languages.md>): Cross-lingual Cultural Consistency; BLEnD All 8 Languages; Max Sigma: 0.017.

- [Cross-lingual Cultural Consistency on BLEnD Higher-Resource](<https://www.sota2.com/research/sota/cross-lingual-cultural-consistency-on-blend-higher-resource.md>): Cross-lingual Cultural Consistency; BLEnD Higher-Resource; Max Sigma: 0.025.

- [Cross-lingual Cultural Consistency on BLEnD Lower-Resource](<https://www.sota2.com/research/sota/cross-lingual-cultural-consistency-on-blend-lower-resource.md>): Cross-lingual Cultural Consistency; BLEnD Lower-Resource; Max Sigma: 0.02.

- [Cross-lingual Cultural Consistency on BLEnD Indo-European](<https://www.sota2.com/research/sota/cross-lingual-cultural-consistency-on-blend-indo-european.md>): Cross-lingual Cultural Consistency; BLEnD Indo-European; Max Sigma: 0.021.

- [Cross-lingual Cultural Consistency on BLEnD Non-Indo-European](<https://www.sota2.com/research/sota/cross-lingual-cultural-consistency-on-blend-non-indo-european.md>): Cross-lingual Cultural Consistency; BLEnD Non-Indo-European; Max Sigma: 0.022.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2305.16938>)

[PDF](<https://arxiv.org/pdf/2305.16938.pdf>)

[Implementation](<https://github.com/uds-lsv/llmft>)
