# Bitune: Leveraging Bidirectional Attention to Improve Decoder-Only LLMs

Source: [Bitune: Leveraging Bidirectional Attention to Improve Decoder-Only LLMs](<https://www.sota2.com/research/paper/68fa7afa1e209131cea86133>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T05:30:46.040Z

Authors: Dawid J. Kopiczko, Tijmen Blankevoort, Yuki M. Asano

Published: 2024-05-23T17:59:22.000Z

## Abstract

Decoder-only large language models typically rely solely on masked causal attention, which limits their expressiveness by restricting information flow to one direction. We propose Bitune, a method that enhances pretrained decoder-only LLMs by incorporating bidirectional attention into prompt processing. We evaluate Bitune in instruction-tuning and question-answering settings, showing significant improvements in performance on commonsense reasoning, arithmetic, and language understanding tasks. Furthermore, extensive ablation studies validate the role of each component of the method, and demonstrate that Bitune is compatible with various parameter-efficient finetuning techniques and full model finetuning.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2405.14862>)

[PDF](<https://arxiv.org/pdf/2405.14862.pdf>)
