The number, and the page it came from
Before a review, someone still has to find the figure in a filing, a contract, or a Treasury bulletin and show the page it came from. A confident paragraph with no citation is not an answer a finance partner can use. That gap, not chatbot fluency, is what document agents are being sold to close.
Nace AI’s bid is Drex for Search, a hosted retrieval layer for people and for coding agents. On the BrowseComp-Plus chart Nace published, the entry it places first is only a little more accurate than its own, and far more expensive in search calls: 95.18 percent at 209.27 calls per question, versus 92.05 percent at 12.61. About one-sixteenth the searches, for a gap of about three points. The announcement headline rounds the call count to 12.6. The chart uses 12.61.
Nace describes that gap as 16.6 times fewer searches. The division is right. The English is not. The leader used about 16.6 times as many calls. On Nace’s numbers, Drex used about 94 percent fewer.
These figures come from Nace’s product page and October 2026 evaluation report, and from a write-up that disclosed Nace sponsorship. They are not an independent audit.
What Nace says it shipped
Nace describes Drex as decision models applied to document and evidence search, built to work over more than a million files. The loop has three moves. Classify: a quick lookup, or a job for an agent that reads documents and queries tables. Squeeze: narrow the workspace from folder to file, section, and page, and inside a spreadsheet to table and column. Answer: send only the kept pages and tables to the language model, with a citation that names the file and the page, row, or timestamp.
Listed file types include PDFs and scans, spreadsheets, CSV and Parquet, audio and video, Word, PowerPoint, Markdown, JSON, and plain text. Workspaces are isolated, each with its own index and knowledge graph, and can be shared with invited members. Named starter domains include tax, M&A, SEC 10-K filings, and contracts, or a customer’s own set.
Stated deployment options are Nace’s managed cloud, a dedicated cloud, the customer’s own AWS or Azure account, or on-prem and air-gapped. Files can be uploaded, or left in the customer’s S3 bucket or Azure Blob container. Zero retention is said to be available with the customer’s own storage. What that excludes is not defined. Pricing, rate limits, service levels, and indexing time are not stated.
Access, on one key, is a CLI, Python and TypeScript SDKs, MCP for Claude Code, Codex, Cursor, and OpenCode, or a REST API. Pairing models named in the announcement are GPT-5, Gemini 3.5 Flash, and an open-weight Qwen 27B.
Later charts use other labels, including GPT-6 sol, GPT-6 luna, Gemini 3.1 Pro, Opus 5.5, and Qwen3.8-27B. Those names are printed as Nace published them. They have not been independently confirmed. They are not aliases for the pairing list above, and they should not be rewritten into the GPT-5.1 and Claude Opus 4.5 names from a December 2025 OfficeQA summary.
Read the boards that follow as a snapshot Nace says it took on October 20, 2026. Ranks this close can move without a product change. A score on one benchmark does not travel to another with a similar name. The materials include no customer case study and no account of how Drex fails.
What Nace claims about the method
Nace’s foil for ordinary retrieval is a slogan: “chunk, embed, top-k, hope.” The misses it hangs on that pipeline will be familiar. The answer sits in a table on page 23. The question needs two hops across files. The file is a scanned PDF. Those are examples. They are not a published error study.
Classic retrieval, fairly described, means fetching text chunks and handing them to a model. Whether Drex skips embedding, ranking, or chunking is not shown. The mechanism claim is narrower. Nace says Drex scores given options with a calibrated probability each, in one forward pass. Training data, model size, and the calibration method are not described.
The scaling line is bolder. Nace says depth is fixed, so a million files take as many steps as a thousand. That would matter for AI automation if it holds on a real pile of scans and spreadsheets. Nothing public here describes the index, the branching factor, or what counts as a step. Fixed depth is a product claim. It has not been shown as a scaling law.
High accuracy at a moderate search bill
BrowseComp-Plus, as Nace describes it, is 830 multi-hop questions over about 100, 000 fixed web documents. The paper id attached to that description is arXiv:2508.06600. With GPT-5, Nace reports 92.05 percent accuracy, second of 86 entries, at 12.61 search calls per question. A separate single-pass run is reported at 87.32 percent with 8.93 calls. Nace calls that the best of any entry under 10 calls. Against a Qwen3-Embed-8B baseline using the same GPT-5 answerer, Nace reports 71.69 percent, a gain of about 20.4 points. A BM25 baseline on the same chart sits at 57.59 percent.
The accuracy leader on that chart, a hybrid associated with AI21, is the 95.18 percent system at 209.27 calls. Two lower-scoring entries used slightly fewer calls than 12.61: Sail Research at 90.72 percent and 12.53 calls, and Mixedbread Search at 90.48 percent and 11.53 calls. LightOn Console is listed at 86.27 percent and 9.70 calls. The rows support high accuracy at a moderate call count. They do not show the fewest searches, and they do not show first place.
That 92 percent does not answer a different test. BrowseComp, described by Rise Data Labs and tied to arXiv:2504.12516, is an open-web task: short answers that are easy to check and hard to find, run with web search. Scores on that page were unavailable when checked. BrowseComp-V³, in a paper dated February 13, 2026 (arXiv:2602.12876), is another exam: 300 cross-modal questions, with even strong models reported around 36 percent. A fixed-corpus score is not evidence that browsing agents have solved the open web.
Three more boards
Nace has said it ranked first or second on three of four public leaderboards. That sentence does not match the later detail. As Nace lays the boards out, MADQA is called first, BrowseComp-Plus second, and EnterpriseRAG-Bench third. VisDocAgentBench is a head-to-head recall comparison. Nace does not state a rank for it.
- MADQA. Agents answer over more than 800 PDFs and cite pages. Nace reports 94.7 percent with Gemini 3.5 Flash, first on the October 20 read. Progress Agentic RAG sits at 94.3 percent, a lead of 0.4 points. With Qwen 27B, Nace reports 93.4 percent, tied on the chart with Mixedbread Agentic Search. OpenAI File Search with GPT-5 is listed at 49.6 percent.
- EnterpriseRAG-Bench. Described as more than 500, 000 internal company documents. Nace reports an overall score of 88.22 and third place, behind Flexor.ai at 91.48 and CDL at 90.91, and about 27.2 points above OpenAI File Search at 61.03.
- VisDocAgentBench. Find one target page among 2, 375 arXiv pages. Recall-at-1 means the first retrieved page is the target. Nace reports 77.5 percent with a Qwen 27B planner, against 67.5 percent for Claude Opus 5 in the write-up. On the chart, the score falls from 97.5 percent on direct questions to 77.5 percent when the answer needs one intermediate fact, and to 57.5 percent when it needs two. Forty questions at each level, 120 in all. The chart note says paper agents are shown with their better reported setup, so the gap may not be same-protocol. A 10-point gap on 120 questions is a signal. It is not a large-sample verdict. The score bends as the hop count rises.
Same bulletins, different exam
A December 9, 2025 summary of a Databricks post, carried by daily.dev, described OfficeQA as an open-source grounded-reasoning benchmark: 89, 000 pages of U.S. Treasury Bulletins, 246 questions, human solvers averaging about 50 minutes each. On that account, GPT-5.1 and Claude Opus 4.5 were under 45 percent on the full set and under 25 percent on the hardest questions, even with advanced document parsing.
OfficeQA Pro, as Nace uses it, is a 133-question set over the same document family, linked as arXiv:2603.08655. Shared paper stock is not a shared score scale. Nace’s pair accuracies, from the mid-40s to around 70 percent, cannot be read as beating that earlier baseline. Different question counts, possibly different splits, different model labels, no shared protocol.
What Nace did publish is nine agent-and-model pairs: shipped accuracy versus the same harness with Drex, plus agent turns. Accuracy rose on eight of the nine. Turns fell on all nine. The exception is the pair labeled Codex with GPT-6 luna, which moved from 54.9 percent to 53.4 percent while turns fell from 16.8 to 11.3. Accuracy did not improve across the board.
The published rows, labels as printed:
- Gemini CLI with Gemini 3.1 Pro: 54.9 percent to 61.7 percent. Turns 22.9 to 10.4.
- Codex with GPT-6 sol: 66.2 percent to 70.7 percent. Turns 14.3 to 8.6.
- Hermes Agent with Opus 5.5: 64.3 percent to 68.4 percent. Turns 8.5 to 3.7.
- Hermes Agent with Qwen3.8-27B: 56.4 percent to 60.2 percent. Turns 20.1 to 6.5.
- Claude Code with Opus 5.5: 66.5 percent to 69.9 percent. Turns 18.5 to 9.1.
- pi with Opus 5.5: 65.4 percent to 68.4 percent. Turns 7.7 to 3.7.
- OpenCode with Qwen3.8-27B: 51.1 percent to 53.4 percent. Turns 23.7 to 10.0.
- pi with Qwen3.8-27B: 43.6 percent to 45.9 percent. Turns 24.3 to 11.1.
- Codex with GPT-6 luna: 54.9 percent to 53.4 percent. Turns 16.8 to 11.3.
Nace’s prose also cites 6 percent more answers right, 62 percent fewer steps, 56 percent fewer tokens, and 2.3 times faster. Those aggregates are not reproducible from the rows as a simple average. The weighting is not shown. Tokens and latency are not in the pair table, so those two claims cannot be checked from the rows. The table itself still says something useful: most pairs got a bit more accurate, and every pair took fewer turns.
Grade the citation, not the demo
The case for AI agents in document-heavy work is extraction, retrieval over a large corpus, and a citation a skeptical partner will accept. Fifty minutes of human time per Treasury question, if that December 2025 figure holds as a picture of the work, is the cost you are trying to compress. Fewer turns matter because turns are latency, token spend, and chances to wander.
A renewal term on page 14 is useless if the amendment that changes it sits in another file. In contracts, filings, tax, and deal documents, a wrong page is an expensive page. A system that returns the file and the page is easier to govern than one that returns a confident paragraph and a pile of chunks.
None of that is a customer case study. Rise Data Labs, writing about browsing agents, makes a point that applies past its own benchmark business: agents get judged on what they can find, and published sets get studied. A public score is a clue. It is not a procurement result.
A sane pilot is small and mean. Pick a workspace you already understand, preferably contracts or filings your team can grade by hand. Measure citation accuracy, not just whether the final sentence sounds right. Count turns and tokens yourself. Include one scanned PDF, one ugly spreadsheet, and one question that needs two documents. Write down what happens when the first page is wrong. If the vendor cannot show that failure, you have not finished the evaluation. You have finished the demo.
Questions worth asking before a pilot
Are these leaderboard ranks an independent audit?
No. They are Nace’s reports from leaderboards it says it read on October 20, 2026, and from its October 2026 evaluation report. A 0.4-point lead, like the one Nace reports on MADQA, can move without a product change.
Did Drex use fewer searches than everyone on BrowseComp-Plus?
No. On the chart Nace published, the accuracy leader used about 16.6 times as many calls. Sail Research and Mixedbread used slightly fewer calls than Drex, and scored a bit lower.
Is a 92 percent BrowseComp-Plus score the same as solving BrowseComp?
No. BrowseComp-Plus, as Nace describes it, is a fixed corpus. BrowseComp is an open-web task, and BrowseComp-V³ is a 300-question multimodal set where strong models were still reported near 36 percent in early 2026.
Did accuracy rise on every OfficeQA Pro pair Nace published?
No. Eight of nine rose. The Codex pair labeled GPT-6 luna fell from 54.9 percent to 53.4 percent, even as its turns dropped. Turns fell for all nine.
Can OfficeQA Pro be treated as a win over the Databricks OfficeQA baseline?
No. The December 2025 description of OfficeQA is 246 questions, with frontier models under 45 percent. Pro is a 133-question protocol in Nace’s materials. Same document family is plausible. Same score scale is not established.
What should a two-week pilot actually measure?
Grade citation accuracy yourself on a scan, a messy spreadsheet, and a question that needs two documents, and count turns and tokens rather than trusting a vendor average. Require a written case for a wrong first page. Price, indexing time, and what zero retention excludes are still unpublished, and a leaderboard will not supply them.