CREPE: Coordinate-Aware End-to-End Document Parser

In this study, we formulate an OCR-free sequence generation model for visualdocument understanding (VDU). Our model not only parses text from documentimages but also extracts the spatial coordinates of the text based on themulti-head architecture. Named as Coordinate-aware End-to-end Document Parser(CREPE), our method uniquely integrates these capabilities by introducing aspecial token for OCR text, and token-triggered coordinate decoding. We alsoproposed a weakly-supervised framework for cost-efficient training, requiringonly parsing annotations without high-cost coordinate annotations. Ourexperimental evaluations demonstrate CREPE's state-of-the-art performances ondocument parsing tasks. Beyond that, CREPE's adaptability is furtherhighlighted by its successful usage in other document understanding tasks suchas layout analysis, document visual question answering, and so one. CREPE'sabilities including OCR and semantic parsing not only mitigate errorpropagation issues in existing OCR-dependent methods, it also significantlyenhance the functionality of sequence generation models, ushering in a new erafor document understanding studies.