Invoice OCR with doctr: practical production knobs, two-pass recognition, and rollout checklist

“Open it and Ctrl+F for ‘INV-2024-00817’, the scan is unchanged, but the text is selectable.”

A concise tour for engineers and product owners building invoice OCR pipelines with doctr (Mindee’s python-doctr). Practical knobs, trade-offs, and an operational checklist you can act on after running the notebook examples.

Prerequisites and what to expect

Code: the examples use the doctr Python package and its predictor builders (DocumentFile, ocr_predictor, detection_predictor, recognition_predictor, kie_predictor).
Hardware: a GPU is recommended for reasonable throughput. The notebook includes T4 tuning hints. CPU-only runs are possible for small tests.
Version note: layout detection and some convenience helpers expect doctr >= 1.0 in the notebook examples.

What the pipeline actually does

At a practical level the flow separates responsibilities you already care about:

Detection, where is text on the page (axis‑aligned boxes or 4‑point polygons)?
Recognition, what are the words inside those crops?
Structure & export, reading order, tables, labeled fields (KIE, Key Information Extraction), and searchable PDFs/JSON/hOCR for downstream systems.

doctr returns a Document object with blocks, lines, words and geometries so downstream extractors (table parsers, KIE models, or LLMs) can use spatial context, not just raw text.

Production-minded tricks up front

If you only take one thing away: instrument and optimise for the business fields that matter (invoice id, totals, bank details) and be strategic about cost vs accuracy.

Start with a cheap baseline and escalate selectively. Mobilenet detectors and recognizers use far less GPU memory and run faster. The notebook shows mobilenet variants as much cheaper on the synthetic page it uses. Use a heavyweight model only where it improves downstream SLAs.
Two‑pass recognition (selective reprocessing) is the usual pattern. Run a fast recognizer for all crops, then re-run low‑confidence words through a stronger model. The notebook uses CONF_GATE = 0.85 as an example gate.
Batch smartly. Detection is memory bound because of large feature maps, so keep detector batch sizes small. Recognition crops are tiny so recognition batch sizes can be large. Example knobs from the notebook: ocr_predictor(pretrained=True, det_bs=4, reco_bs=1024). On a T4 start at det_bs=2, reco_bs=512 and increase reco_bs until you hit OOM.
Measure, don’t eyeball. Sweep detection postprocessing thresholds (bin_thresh / box_thresh pairs) on a small labelled set and pick F1 for your target fields rather than trusting a visual check.

How the notebook builds a realistic test case

Rather than using a random scan, the notebook synthesizes invoice pages (example pixel A4 = (1240, 1754), roughly an A4 at ~150 DPI in the demo) from lists of ground-truth lines. A few representative strings used in the examples are:

“NORTHWIND TRADING CO.”
“Invoice No: INV-2024-00817”, “Date: 14/03/2024”, “Due Date: 13/04/2024”
Line items and totals, e.g. “Servo controller board Rev C”, qty “12”, unit price “84.50”, amount “1014.00”; “Subtotal 2943.00”; “TOTAL DUE 3531.60”
Payment: “Bank: Lloyds Sort Code: 30-96-26 Account: 41775302”

It then applies scan degradations to simulate phone photos and flatbed scans. Example perturbation parameters used in the notebook include rotation angles in degrees [0.4, -0.3, 13.0], noise levels [6.0, 8.0], and jpeg_quality=72. Outputs produced in the demo are named like “invoice_p1.png”, “invoice_p2.png”, “invoice_rotated.png” and “invoice.pdf”.

Build, benchmark, iterate

The notebook treats OCR as engineering: pick detection and recognition architectures, benchmark combinations, and rank them with a quick metric.

Example default builder: build_ocr(det=”db_resnet50″, reco=”crnn_vgg16_bn”).
Model tuples evaluated in the demo include (examples):
- (“db_mobilenet_v3_large”, “crnn_mobilenet_v3_small”)
- (“fast_base”, “crnn_vgg16_bn”)
- (“db_resnet50”, “crnn_vgg16_bn”)
- (“db_resnet50”, “parseq”)
Timing is measured with warmups (timeit(fn, *args, warmup=1, runs=3)).
Model ranking uses an order‑insensitive bag metric (bag_accuracy), essentially token multiset recall after normalization. The notebook’s normalization keeps characters matching [^\w@:./+-] (the demo’s norm regex) when comparing tokens.

Key postprocessing hooks and why they matter

Small pipeline changes often yield outsized gains.

Objectness vs recognition confidence. The detector’s objectness_score shows whether a box likely contains text. Recognition confidence is the recognizer’s softmax score for the word. Use objectness to drop hallucinated boxes and recognition confidence to triage reprocessing or human review.
PadBoxesHook (example params dx=0.004, dy=0.006) expands crops before recognition. The notebook notes “Recognition often improves when crops aren’t cut flush to the glyphs.”
DropTinyBoxesHook (min_h=0.006, min_w=0.004) filters speckle detections so they don’t waste recognition compute.
Tune detection postprocessing. The demo sweeps bin_thresh / box_thresh pairs: (0.1, 0.05), (0.3, 0.1), (0.5, 0.2), (0.7, 0.4), (0.9, 0.6). Lower thresholds recover faint text at the cost of noise. Higher thresholds produce fewer, cleaner boxes. Sweep on a labelled set and pick the tradeoff that matches your recall/precision needs.

Rotation, skew and page geometry

doctr supports axis‑aligned boxes and 4‑point polygons. The notebook documents three practical strategies for non‑straight pages:

assume_straight_pages=True, fastest, but it degrades with even moderate skew (single‑digit degrees).
assume_straight_pages=False, returns 4‑point polygons for robustness to rotation and perspective.
straighten_pages=True then assume_straight_pages, de‑skew first, then treat as axis‑aligned.

Choose the strategy based on your data. If most pages are scanned relatively straight, the speed of axis‑aligned processing is attractive. If photos and skew are common, prefer polygons or a de‑skew step.

Layout detection, KIE and table heuristics

Detecting regions gives you structure, not just text. The demo shows ocr_predictor(pretrained=True, detect_layout=True) to tag Title, Text, Table, Page‑header and Page‑footer regions. For robust field extraction the notebook recommends training a multi‑class detector so KIE returns labeled fields (invoice_no, date, totals) directly rather than relying on brittle regular expressions.

The notebook includes a quick regex mapping for one-off extraction (FIELDS), for example:

“invoice_no”: r”Invoice\s*No[:\s]*([A-Z0-9\-]+)”
“date”: r”\bDate[:\s]*(\d{2}/\d{2}/\d{4})”
“due_date”: r”Due\s*Date[:\s]*(\d{2}/\d{2}/\d{4})”
“vat_id”: r”VAT\s*(GB[\s\d]{8, })”
“total_due”: r”TOTAL\s*DUE\s*([\d., ]+)”

For table extraction the demo uses a pragmatic 1‑D clustering of left edges inside a vertical band (detect_columns). That heuristic works well for predictable invoice tables with fixed columns. It fails for complex tables with merged or spanning cells, rotated tables, or irregular grids, in those cases use a dedicated table parser or a learned table model.

Exporting the artifacts you need

Common outputs covered by the notebook:

Plain text and JSON exports for ingestion into downstream systems.
hOCR (HTML with OCR geometry) so text and positions are preserved in a searchable HTML layer.
synthesize() to re‑render recognized text into detected boxes for visual verification.
Searchable PDFs: the helper make_searchable_pdf(pages_np, doc_result, out_path, dpi=150) creates a raster image page with an invisible text layer so you can select/search text in the PDF. Example output: “invoice_searchable.pdf”, open it and Ctrl+F for “INV-2024-00817”.

Performance knobs and deployment pointers

Practical rules from the demo and common production practice:

Detector batch sizes are memory‑bound because of large feature maps. Recognition batches can be much larger because crops are tiny. Tune det_bs and reco_bs per GPU.
Cheap wins (ordered as in the notebook):
1. Swap to db_mobilenet_v3_large + crnn_mobilenet_v3_small, mobilenet variants are far cheaper on compute and memory for clean documents.
2. Pass fewer, larger batches rather than many tiny calls for bulk workloads, but be mindful of latency and OOM tradeoffs.
3. assume_straight_pages=True and disable orientation checks when your data allows it.
4. Lower PDF scale if source text is large to save processing cost.
5. Try half precision: predictor = predictor.half(), but validate accuracy first (some ops or models may not be fully FP16‑safe).
Deployment templates in the repo include a FastAPI example (api/ with /detection /recognition /ocr /kie routes), a Streamlit demo (demo/app.py), a Hugging Face Space (huggingface.co/spaces/mindee/doctr), and GPU‑ready Docker images (ghcr.io/mindee/doctr).

What the demo cautions, and what it leaves for you to solve

The notebook is explicit that many numbers are illustrative: “these numbers are for ONE synthetic page; always benchmark on your own data.” Keep that front of mind.

Topics you should plan for before going live:

Labeling for KIE: the demo points to training scripts (references/detection/train_pytorch.py, references/recognition/train_pytorch.py). A practical approach is to start with a pilot of a few hundred annotated documents per template to validate feasibility, and scale annotations (thousands) as you cover more variability (fonts, layouts, skews).
Human‑in‑the‑loop: flag low‑confidence tokens (recognition confidence) and low‑objectness boxes into a review queue. Closed‑loop systems should store reviewer corrections back to your training set for active learning.
Privacy & compliance: invoices contain PII and bank details. Add access controls, encrypted storage, redaction, and a retention policy, the demo is an engineering template, not a security posture.
Multi‑language and vocab gaps: many stock checkpoints are trained on Latin/French vocabularies. If your documents use other alphabets or unusual symbols, fine‑tune the recognizer with a wider VOCAB (see doctr.datasets.VOCABS) or collect targeted training samples.
Monitoring and SLAs: instrument per‑field F1, confidence histograms, throughput, tail latency and error rates. Production systems require automated alerts for drift and regressions.

Operational checklist for rollout

Benchmark on representative documents. Synthetic pages are good for development but not for SLAs.
Pin doctr and model checkpoint versions. APIs evolve and detect_layout/KIE helpers changed across releases.
Set up monitoring: per‑field confidence, per‑model latency, and F1 on a held‑out validation set.
Design a human review queue keyed by recognition confidence and objectness. Log corrections for active learning.
Fine‑tune recognizers if your alphabet or symbol set is out‑of-distribution.
Secure the pipeline: container scanning, secrets management for storage and bank details, and a data retention policy.

Key takeaways, questions you’ll ask

Can I get both accurate text and geometry for downstream extraction?

Yes. doctr returns a Document object with blocks, lines, words and geometries; synthesize() and hOCR/JSON exports preserve geometry so table parsers and KIE can use spatial context. Action: export hOCR/JSON and validate table alignment on 50 representative pages before integrating downstream.
Do I need the heaviest models everywhere?

No. Use a two‑pass approach: run a fast recognizer for all crops and reprocess only low‑confidence crops with a stronger model (PARSeq in the demo). This concentrates compute where it affects SLAs (invoice numbers, totals) and reduces cost for the rest.
How should I tune detector postprocessing?

Sweep bin_thresh and box_thresh pairs (the notebook tries pairs from (0.1, 0.05) up to (0.9, 0.6)) on a small labelled set and optimise per‑field F1. Use objectness to drop noisy boxes and PadBoxesHook to improve recognition when crops are tight.
Will off‑the‑shelf checkpoints handle my language or symbols?

Maybe not. Many stock checkpoints use Latin/French vocabs. Run a quick sample validation: if you see systematic OOV characters or missed glyphs, fine‑tune with an expanded VOCAB (see doctr.datasets.VOCABS) and targeted examples.
Can I make searchable PDFs from scans?

Yes. The demo’s make_searchable_pdf(…) overlays an invisible text layer on the raster image (example output “invoice_searchable.pdf”). It makes text selectable/searchable without altering the visual fidelity of the scan.

Where to look next

Start by running the notebook examples to reproduce the synthetic pipeline and then swap in a small representative batch of your own invoices. Consult the doctr project pages for API details and deployment examples:

doctr documentation: https://mindee.github.io/doctr
doctr GitHub: https://github.com/mindee/doctr
Example deployments and images referenced in the demo: ghcr.io/mindee/doctr and huggingface.co/spaces/mindee/doctr
Notebook code used in the demo (runnable examples) is available in the referenced GitHub notebook in the demo repo.

If your team cares about searchability, structured extraction and predictable costs, the pattern the notebook demonstrates, geometry‑preserving OCR, selective reprocessing, careful postprocessing and targeted fine‑tuning, is a pragmatic path from scanned images to reliable, auditable document intelligence.