# Bidirectional Language Models Are Also Few-shot Learners Source: [Bidirectional Language Models Are Also Few-shot Learners]() Collection index: [llms.txt]() Data updated: 2025-10-28T13:56:57.308Z Authors: Ajay Patel, Bryan Li, Mohammad Sadegh Rasooli, Noah Constant, Colin Raffel, Chris Callison-Burch Published: 2022-09-29T01:35:57.000Z ## Abstract Large language models such as GPT-3 (Brown et al., 2020) can perform arbitrary tasks without undergoing fine-tuning after being prompted with only a few labeled examples. An arbitrary task can be reformulated as a natural language prompt, and a language model can be asked to generate the completion, indirectly performing the task in a paradigm known as prompt-based learning. To date, emergent prompt-based learning capabilities have mainly been demonstrated for unidirectional language models. However, bidirectional language models pre-trained on denoising objectives such as masked language modeling produce stronger learned representations for transfer learning. This motivates the possibility of prompting bidirectional models, but their pre-training objectives have made them largely incompatible with the existing prompting paradigm. We present SAP (Sequential Autoregressive Prompting), a technique that enables the prompting of bidirectional models. Utilizing the machine translation task as a case study, we prompt the bidirectional mT5 model (Xue et al., 2021) with SAP and demonstrate its few-shot and zero-shot translations outperform the few-shot translations of unidirectional models like GPT-3 and XGLM (Lin et al., 2021), despite mT5's approximately 50% fewer parameters. We further show SAP is effective on question answering and summarization. For the first time, our results demonstrate prompt-based learning is an emergent property of a broader class of language models, rather than only unidirectional models. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Multi-hop QA on WebQSP](): Multi-hop QA; WebQSP; Accuracy: 63.3. - [Open-domain QA on WebQuestions](): Open-domain QA; WebQuestions; Accuracy: 48.7. - [Single-hop QA on SimpleQuestions](): Single-hop QA; SimpleQuestions; Accuracy: 0.2. - [Multi-hop QA on CompWebQ](): Multi-hop QA; CompWebQ; Accuracy: 37.6. ## Original sources and implementation [Original paper]() [PDF]()