CoMER: Modeling Coverage for Transformer-based Handwritten Mathematical Expression Recognition
About
The Transformer-based encoder-decoder architecture has recently made significant advances in recognizing handwritten mathematical expressions. However, the transformer model still suffers from the lack of coverage problem, making its expression recognition rate (ExpRate) inferior to its RNN counterpart. Coverage information, which records the alignment information of the past steps, has proven effective in the RNN models. In this paper, we propose CoMER, a model that adopts the coverage information in the transformer decoder. Specifically, we propose a novel Attention Refinement Module (ARM) to refine the attention weights with past alignment information without hurting its parallelism. Furthermore, we take coverage information to the extreme by proposing self-coverage and cross-coverage, which utilize the past alignment information from the current and previous layers. Experiments show that CoMER improves the ExpRate by 0.61%/2.09%/1.59% compared to the current state-of-the-art model, and reaches 59.33%/59.81%/62.97% on the CROHME 2014/2016/2019 test sets.
Key insights
What problem does CoMER solve?
Transformers for handwritten math recognition over-parse or under-parse image regions because they lack coverage, i.e., tracking which areas were already aligned. CoMER adds coverage to the transformer decoder via an Attention Refinement Module without losing parallelism.
How does CoMER differ from prior work?
Unlike prior RNN coverage attention or a transformer attempt that hurt parallelism, CoMER uses self-coverage and cross-coverage from different decoder layers. On CROHME 2014, fusion-coverage reaches 59.33% ExpRate versus 55.17% for the BTTR baseline.
What are the practical uses of these findings?
The paper documents benchmark results on CROHME 2014/2016/2019 datasets only. It mentions online education, automatic scoring, and formula image searching as downstream applications but does not document any production deployment.
What limitations and future work do the authors note?
The authors state ARM is a general attention refinement framework and propose extending it to tasks like machine translation, text summarization, and image captioning. Evaluation is limited to the CROHME datasets.