PatchFinder: Leveraging Visual Language Models for Accurate Information Retrieval using Model Uncertainty

Roman Colman, Minh Vu, Manish Bhattarai, Martin Ma, Hari Viswanathan, Daniel O'Malley, Javier Santos; Proceedings of the Winter Conference on Applications of Computer Vision (WACV), 2025, pp. 9128-9137

For decades corporations and governments have relied on scanned documents to record vast amounts of information. However extracting this information is a slow and tedious process due to the sheer volume and complexity of these records. The rise of Vision Language Models (VLMs) presents a way to efficiently and accurately extract the information out of these documents. The current automated workflow often requires a two-step approach involving the extraction of information using optical character recognition software and subsequent usage of large language models for processing this information. Unfortunately these methods encounter significant challenges when dealing with noisy scanned documents often requiring computationally expensive language models to handle high information density effectively. In this study we propose PatchFinder an algorithm that builds upon VLMs to improve information extraction. First we devise a confidence-based score called Patch Confidence based on the Maximum Softmax Probability of the VLMs' output to measure the model's confidence in its predictions. Using this metric PatchFinder determines a suitable patch size partitions the input document into overlapping patches and generates confidence-based predictions for the target information. Our experimental results show that PatchFinder leveraging Phi-3v a 4.2 billion parameter VLM achieves an accuracy of 94% on our dataset of 190 noisy scanned documents outperforming ChatGPT-4o by 18.5 percentage points.