# AstroLLaVA: towards the unification of astronomical data and natural language Source: [AstroLLaVA: towards the unification of astronomical data and natural language]() Collection index: [llms.txt]() Data updated: 2025-10-27T04:37:04.552Z Authors: Sharaf Zaman, Michael J. Smith, Pranav Khetarpal, Rishabh Chakrabarty, Michele Ginolfi, Marc Huertas-Company, Maja Jab{\\l}o\\'nska, Sandor Kruk, Matthieu Le Lain, Sergio Jos\\'e Rodr\\'iguez M\\'endez, Dimitrios Tanoglidis Published: 2025-04-11T14:36:31.000Z ## Abstract We present AstroLLaVA, a vision language model for astronomy that enables interaction with astronomical imagery through natural dialogue. By fine-tuning the LLaVA model on a diverse dataset of $\\sim$30k images with captions and question-answer pairs sourced from NASA's \`Astronomy Picture of the Day', the European Southern Observatory, and the NASA/ESA Hubble Space Telescope, we create a model capable of answering open-ended questions about astronomical concepts depicted visually. Our two-stage fine-tuning process adapts the model to both image captioning and visual question answering in the astronomy domain. We demonstrate AstroLLaVA's performance on an astronomical visual question answering benchmark and release the model weights, code, and training set to encourage further open source work in this space. Finally, we suggest a roadmap towards general astronomical data alignment with pre-trained language models, and provide an open space for collaboration towards this end for interested researchers. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. ## Original sources and implementation [Original paper]() [PDF]()