NeoMME: Compact Single‑Tower Multimodal Encoders for Visual‑Document Retrieval

H Company’s NeoMME: compact single‑tower multimodal encoders (262, 937, 906 and 793, 715, 032 params) built for visual‑document retrieval

TL;DR for business leaders

Accuracy vs size: H Company reports that NeoMME‑260M reaches nDCG@10 = 0.523 on ViDoRe v3, and the 800M reaches 0.556. The 260M result is reported to be within 0.002 of a 3.75B‑parameter baseline on the same benchmark.
Deployment focus: NeoMME is presented as a representation‑first design with Apache 2.0 checkpoints, day‑zero Hugging Face Transformers support, CPU‑friendly query encoding (78.3 ms reported), and fast GPU indexing (51.3 pages/sec on a single L40S at 2048×2048 for the 260M model).
Tradeoffs: it omits a causal decoder, so there is no on‑model generation, and it uses late‑interaction for fine‑grained matches. That gives good recall but increases index size unless you apply pooling or quantization.

What NeoMME is, and how it’s different

H Company describes NeoMME as a family of single‑tower, bidirectional multimodal encoders at two sizes: 262, 937, 906 and 793, 715, 032 parameters. “Single‑tower” here means a single Transformer stack ingests both multilingual text tokens and raw image patches in the same layers, rather than using separate vision and text towers or a large generative decoder. The family intentionally drops a causal decoder to focus on representation quality and compute efficiency for retrieval and search tasks. H Company reports the models were trained from random initialization.

Why that matters for product teams: fewer parameters dedicated to generation lowers inference cost for retrieval pipelines, and a single Transformer reduces architectural glue when your job is to produce embeddings for indexing and search rather than to generate text on the fly.

Key specs (compact)

Exact parameter counts: 262, 937, 906 and 793, 715, 032.
Image tokenization: non‑overlapping 32×32 RGB patches, projected by a 2‑layer MLP trained from scratch.
Text embedding: ALBERT‑style factorized lookup (256‑dim) projected to model width.
Context length: 16, 384 tokens (H Company notes this is roughly equivalent to the tokenized representation of two 4K images after patching and border handling).
Attention: mostly symmetric sliding‑window (local) attention with global attention in every sixth and the final layer. Engineering optimizations include grouped‑query attention, query‑key normalization, gated attention, 2D rotary positional encodings, and squared‑ReLU MLPs.
Tokenizer: whitespace‑unconstrained BPE with a 131, 072 token vocabulary. H Company reports it emits 44.4% fewer tokens than “ModernBERT” on a 14‑language FLORES‑200 devtest subset.

These are compact models relative to many multimodal retrievers that repurpose large generative backbones or separate vision towers. The design targets efficient retrieval rather than multimodal generation.

Pretraining approach, what “discrete masked diffusion” means here

H Company describes the pretraining objective as a discrete masked‑diffusion denoising objective. In plain terms, the model is trained to recover masked discrete tokens through a staged noisy corruption process rather than via a single masked token prediction. The authors say the objective can be conditioned on visible image patches during multimodal examples.

Reported corruption schedules: text‑only segments sample corruption rates uniformly from 0→1, while multimodal segments sample from 0.30→1. In a heavy‑masking cross‑modal ablation (90% masking), H Company reports that including visible page patches during pretraining raised masked‑token recovery accuracy by 38.4 percentage points for the 260M model and 40.5 points for the 800M model. That is a large margin and supports joint multimodal conditioning when your downstream task includes visual‑document understanding.

Retrieval heads and evaluation highlights

NeoMME‑Retriever is a retrieval fine‑tune that jointly trains two heads in a single forward pass. One is a mean‑pooled dense head, described with “Matryoshka” nested widths. The other is a late‑interaction head that projects every token and patch to 128 dimensions so token‑level matching can be performed at search time. That dual output gives teams the option to serve dense retrieval, late‑interaction, or hybrid strategies without separate encodings.

Reported benchmark results (as provided by H Company):

ViDoRe v3 (nDCG@10): 260M = 0.523; 800M = 0.556. H Company notes the 260M run is within 0.002 of ColQwen2.5‑v0.2 (a 3.75B‑parameter model) and is 26.1 points above the next best sub‑300M model on that benchmark.
ViDoRe v1/v2: H Company reports paired numbers “0.860/0.522 and 0.874/0.559 nDCG@5” (the original report’s formatting is ambiguous about exact mappings; ask H Company to clarify which model maps to which score).
BEIR‑15 (text retrieval, late interaction): NeoMME reports 0.4881 and 0.5126 for the two sizes. By contrast LateOn (149M parameters) is reported at 0.5722. H Company points out supervision scale differs sharply (NeoMME’s retrieval supervision ~430K text query examples versus some comparators using what are reported as ~660M contrastive examples), which materially affects BEIR outcomes.

Two practical notes. First, small absolute deltas in nDCG can be meaningful depending on task noise and sample size, so reproduce results on your data slice. Second, BEIR is a heterogeneous text retrieval benchmark and differences in supervision type and scale can explain why a model tuned for visual‑document retrieval might lag a heavily contrastive text retriever on BEIR’s text tasks.

Deployability: throughput, latency, and index storage

Indexing throughput (reported): NeoMME‑260M indexes 51.3 pages/sec on a single NVIDIA L40S at 2048×2048 input. By comparison the reported ColModernVBERT indexing speed was 26.0 pages/sec on the same hardware and input, which is roughly a 1.97× throughput improvement for NeoMME‑260M in that measurement.
Query encoding (reported): NeoMME‑260M encodes a query in 78.3 ms on a CPU‑only host.
Late‑interaction index size (reported): a 2048×2048 page yields 4, 162 vectors and is reported as ≈1.5 MB per ViDoRe v3 document in float32. H Company demonstrates hierarchical token pooling and quantization options that drastically shrink storage. A pool factor of 10 with int8 queries/docs yields ≈39.0 kB/page (39.4× reduction, retaining 99.16% of baseline nDCG@10). An aggressive variant (pool factor 8, int8 queries with binary documents) is reported at ≈6.0 kB/page (255.5× reduction, retaining 95.19%).

Note on the bytes math: H Company reports the late‑interaction projection dimension as 128, and it also reports 4, 162 vectors per page and ≈1.5 MB per page in float32. If you compute 4, 162 vectors × 128 dims × 4 bytes/float32, the raw size is about 2.13 MB. That discrepancy suggests the reported ≈1.5 MB either reflects a different stored dimensionality, additional packing, or a different counting convention. Ask for the exact vector dimensionality and storage format (float32 vs float16, any header or metadata, sparse packing) when validating index‑size claims.

What this design gains, and what it gives up

Advantages

Representation efficiency: fewer parameters tied to generation reduce runtime cost for retrieval‑only services.
Operational simplicity: a single Transformer that handles both modalities simplifies encoding pipelines and may reduce integration complexity.
Practical deployability: H Company reports Apache 2.0 licensing and day‑zero Transformers support, plus CPU‑friendly encoding and fast GPU indexing numbers.

Tradeoffs

No on‑model autoregressive generation: NeoMME omits a causal decoder, so use cases requiring captions, summaries, or conversational multimodal outputs need a separate generative model.
Potential modality interference: single‑tower joint training can reduce opportunities to reuse large vision‑only backbones when those are valuable for a domain.
Late‑interaction index cost: per‑token vectors improve fine‑grained recall but increase index size. Compression (pooling and quant) is effective but workload‑dependent and requires validation.
Benchmark gaps: NeoMME’s BEIR numbers are lower than some text‑focused retrievers, which underscores that supervision type, scale, and evaluation domain matter.

When NeoMME makes sense for product teams

Good fit

Visual‑document search at scale: scanned contracts, invoices, manuals with diagrams, and multi‑page documents where token‑level matches matter.
Retrieval layers for enterprise KBs where CPU‑friendly query latency and permissive licensing are operational priorities.
Teams that prefer a compact, single model for embedding generation and want both dense and late‑interaction outputs from one forward pass.

Poor fit (without extra components)

Applications that require native multimodal generation, such as captions, summaries, or chat, since you’ll need to pair NeoMME with a separate decoder or generative model.
Vision‑specialist tasks that benefit from very large pretrained vision backbones unless you’re prepared to fine‑tune extensively.

Concrete pilot checklist (for engineering and product teams)

Reproduce accuracy: run NeoMME‑Retriever on a representative 5k, 10k sample of your documents and measure nDCG@k or recall against your ground truth.
Measure encode latency: validate the reported ≈78.3 ms CPU query encode on your target CPU instance and measure GPU indexing throughput on your target GPU (L40S if you have it) with your input shapes.
Index cost experiment: build a late‑interaction index for a 10k document slice, measure raw float32 size, then evaluate hierarchical pooling factors {4, 8, 10} with int8 quantization to map size versus nDCG drop.
Compare variants: run dense‑only versus late‑interaction retrieval and a hybrid pipeline to determine the sweet spot of recall versus storage for your corpus.
Operational checks: validate the provided repo and model card and check Hugging Face Transformers integration on your runtime stack.

Artifacts to request from H Company (must‑have for confident adoption)

Model card and repository with license file (Apache 2.0), tokenizer, and exact model configs to confirm parameter counts and embedding dims.
Training and ablation logs, including the 90% masking cross‑modal ablation table, and the corpus description used for pretraining and retrieval supervision.
Benchmark scripts and raw logs for ViDoRe v1/v2/v3 and BEIR runs, with exact dataset versions and seeds so you can reproduce numbers.
Indexing and throughput benchmark scripts, including GPU/CPU hardware configs, batch sizes, and precision settings used to produce the L40S and 78.3 ms numbers.
Details on late‑interaction storage calculations, including vector dimensionality, packing format, and any header or metadata, to reconcile the 4, 162 vectors → ≈1.5 MB report.

Key questions, and short, practical answers

Is NeoMME production‑ready?

H Company reports Apache 2.0 checkpoints and day‑zero Hugging Face Transformers support, plus concrete latency and throughput numbers, which positions NeoMME for production testing. Verify performance and compliance on your data and infrastructure before rollout.
Does multimodal pretraining help retrieval?

H Company’s ablation reports large gains (≈38.4-40.5 points at 90% masking) from conditioning on visible image patches. That suggests joint multimodal pretraining materially improves masked‑token recovery and cross‑modal representations for visual‑document tasks.
Can the 260M model match much larger retrievers?

H Company reports NeoMME‑260M at nDCG@10 = 0.523 on ViDoRe v3, within 0.002 of a 3.75B‑parameter comparison on that benchmark. That efficiency claim is worth reproducing on your corpus, because small leaderboard deltas can depend on dataset alignment and evaluation noise.
Will late‑interaction break my storage budget?

Out of the box the late‑interaction index is large (H Company reports ~4, 162 vectors and ≈1.5 MB per page in float32), but hierarchical pooling plus quantization examples in the release show reductions to ≈39.0 kB or even ≈6.0 kB per page while keeping most retrieval quality. Validate pooling and quantization tradeoffs on your workload.
What should I ask H Company before a pilot?

Request the model card, training and ablation logs, benchmark scripts and raw logs, exact tokenizer scripts, and precise storage and packing details for late‑interaction vectors so you can reproduce the reported throughput, latency, and index‑size numbers.

Bottom line

NeoMME is presented as a pragmatic, representation‑first multimodal retriever family: compact parameterizations, single‑tower joint encoding, and an operational focus (Apache 2.0, Transformers support, CPU query encoding). For teams building visual‑document search or embedding layers for enterprise knowledge, it’s worth a short pilot that reproduces the reported accuracy, latency, and index‑size numbers on your data. Before you commit, obtain the model card, benchmark logs, and storage format details from H Company and validate the pooling and quantization tradeoffs for your corpus. That due diligence is the fastest route from promising numbers to reliable production value.