# Variational Pretraining for Semi-supervised Text Classification Source: [Variational Pretraining for Semi-supervised Text Classification]() Collection index: [llms.txt]() Data updated: 2025-10-25T21:36:37.771Z Authors: Suchin Gururangan, Tam Dang, Dallas Card, Noah A. Smith Published: 2019-06-05T18:40:37.000Z ## Abstract We introduce VAMPIRE, a lightweight pretraining framework for effective text classification when data and computing resources are limited. We pretrain a unigram document model as a variational autoencoder on in-domain, unlabeled data and use its internal states as features in a downstream classifier. Empirically, we show the relative strength of VAMPIRE against computationally expensive contextual embeddings and other popular semi-supervised baselines under low resource settings. We also find that fine-tuning to in-domain data is crucial to achieving decent performance from contextual embeddings when working with limited supervision. We accompany this paper with code to pretrain and use VAMPIRE embeddings in downstream tasks. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Multi-class text classification on Yelp](): Multi-class text classification; Yelp; Micro-F1: 55.1. - [Multi-class Text Classification on AG News](): Multi-class text classification; AG-News; Micro-F1: 0.876. - [Text Classification on Yahoo](): Text Classification; Yahoo; Micro F1 Score: 64.4. ## Original sources and implementation [Original paper]() [PDF]()