# Differentiable Instruction Optimization for Cross-Task Generalization Source: [Differentiable Instruction Optimization for Cross-Task Generalization]() Collection index: [llms.txt]() Data updated: 2026-01-02T03:05:27.035Z Authors: Masaru Isonuma, Junichiro Mori, Ichiro Sakata Published: 2023-06-16T17:49:34.000Z ## Abstract Instruction tuning has been attracting much attention to achieve generalization ability across a wide variety of tasks. Although various types of instructions have been manually created for instruction tuning, it is still unclear what kind of instruction is optimal to obtain cross-task generalization ability. This work presents instruction optimization, which optimizes training instructions with respect to generalization ability. Rather than manually tuning instructions, we introduce learnable instructions and optimize them with gradient descent by leveraging bilevel optimization. Experimental results show that the learned instruction enhances the diversity of instructions and improves the generalization ability compared to using only manually created instructions. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Zero-shot cross-task generalization on Super-NaturalInstructions (test)](): Zero-shot cross-task generalization; Super-NaturalInstructions (test); ROUGE-L: 34.97. - [Cross-task generalization on Super-NaturalInstructions (one-shot)](): Cross-task generalization; Super-NaturalInstructions one-shot; ROUGE-L: 37.85. ## Original sources and implementation [Original paper]() [PDF]() [Implementation]()