# Leveraging Semantic Segmentation Masks with Embeddings for Fine-Grained Form Classification Source: [Leveraging Semantic Segmentation Masks with Embeddings for Fine-Grained Form Classification]() Collection index: [llms.txt]() Data updated: 2025-10-26T02:47:16.604Z Authors: Taylor Archibald, Tony Martinez Published: 2024-05-23T04:28:50.000Z ## Abstract Efficient categorization of historical documents is crucial for fields such as genealogy, legal research, and historical scholarship, where manual classification is impractical for large collections due to its labor-intensive and error-prone nature. To address this, we propose a representational learning strategy that integrates semantic segmentation and deep learning models such as ResNet, CLIP, Document Image Transformer (DiT), and masked auto-encoders (MAE), to generate embeddings that capture document features without predefined labels. To the best of our knowledge, we are the first to evaluate embeddings on fine-grained, unsupervised form classification. To improve these embeddings, we propose to first employ semantic segmentation as a preprocessing step. We contribute two novel datasets$\\unicode{x2014}$the French 19th-century and U.S. 1950 Census records$\\unicode{x2014}$to demonstrate our approach. Our results show the effectiveness of these various embedding techniques in distinguishing similar document types and indicate that applying semantic segmentation can greatly improve clustering and classification results. The census datasets are available at https://github.com/tahlor/census\_forms ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. ## Original sources and implementation [Original paper]() [PDF]()