AI labs have a data trust problem that their policies haven’t solved
When Anthropic said in June 2026 it would retain Fable usage logs for 30 days to defend against “complex and novel attacks, ” the reaction from large customers was immediate: several changed how they used the model, some blocked it outright, and the episode revived a long-standing question for enterprise buyers, what does “we won’t train on your data” actually mean in practice?
Why a 30‑day log policy mattered
The detail is small; the stakes are not. Reporting by Matthias Bastian at The Decoder (September 15, 2026) shows that customers including Nvidia, Palantir, and Booz Allen Hamilton restricted or altered their use of Anthropic’s Fable after the June announcement.
“As a company, you know, we believe ZDR [Zero Data Retention] should be on by default, “
, Justin Boitano, Nvidia VP of Enterprise AI (as reported).
Public reporting and company statements say Nvidia limited Fable to less‑sensitive tasks and used its own Nemotron models for proprietary work. Palantir blocked Fable deployment through its software until Anthropic granted stronger zero‑data‑retention guarantees. Booz Allen banned Fable for proprietary cybersecurity development, with CTO Bill Vass saying, “We worry a little bit that [Fable] might be learning from some of our code.” OpenAI, meanwhile, in August moved to let GPT‑5.6 Cyber customers store security logs on their own servers. Anthropic announced a “similar program” rolling out to select customers that fall.
Zero Data Retention (ZDR): a clear slogan, a fuzzy guarantee
Zero Data Retention (ZDR) is meant to be simple: the provider does not keep your inputs. In reality, vendors and buyers attach different meanings. Does ZDR mean inputs are never written to disk? Never used for training? No telemetry retained? Those are separate technical and legal promises.
Both OpenAI and Anthropic say they collect telemetry and technical usage data from enterprise customers. OpenAI calls some enterprise telemetry “de‑identified” and says it runs business data through automated classifiers and security tools “to better understand how our services are used, ” framing the resulting classifications as metadata that “do not contain any of the business data itself.” Even so, customers are uneasy because metadata, timestamps, token counts, classification labels, embeddings (numeric vectors representing text), or behavioral signals, can leak statistical structure or reveal sensitive patterns when combined or analyzed.
How user inputs can influence models, a practical taxonomy
Researchers have sketched a spectrum of ways customer data can feed back into models. The modes are familiar to ML engineers but are rarely spelled out in procurement language:
- Direct pretraining, ingesting raw inputs into the training corpus (highest risk of verbatim memorization).
- Fine‑tuning and distillation, using derived features or model outputs to shape a smaller model’s behavior.
- Reward‑model training / RLHF, using explicit user feedback as signals during reinforcement learning stages.
- User traces → RL environments, converting logs, commit histories, or session traces into environments or tasks the model optimizes against (a more invasive and under‑discussed path).
“De‑identification is weak, “
, John Schulman (reported). He warned users can be traced back “with just a small number of bits” and that de‑identification does not reliably protect against IP leakage. Schulman (now at Thinking Machines) also emphasized companies vary in how aggressively they train on user data and called for clearer norms on disclosure.
Sarah Hooker (formerly at Cohere and Google DeepMind) added that there are “clever synthetic data techniques that can generate distributional equivalent data while preserving privacy.” In plain language: labs can sometimes generate data that preserves the statistical shape of your inputs without keeping verbatim copies, which helps utility but can still embed your domain patterns into future models. Hooker warned companies with valuable IP, “If you are a company with IP you have a limited window to build your own intelligence that leverages your IP. Otherwise you are fueling a frontier lab which will encroach on your vertical sooner or later.”
Proof by controversy: the Buckmaster / Codex episode
The abstract debate got sharply practical with Tristan Buckmaster and Levent Alpöge’s Navier‑Stokes work. The researchers had uploaded drafts through OpenAI’s Codex during their research. Initial statements left open the possibility that anonymized derivatives of those prompts could have contributed to system improvements. After an internal investigation, OpenAI updated its post saying Buckmaster’s Codex prompts from the two months before the September 8, 2026 publication “could not have influenced the system in any way, including through training.”
That exchange shows the real damage vector: the perception that a provider absorbed unique research or code corrodes trust more than any technical nuance. Even when a vendor publishes a corrective investigation, the reputational and contractual fractures can persist.
Why de‑identification and synthetic data aren’t magic
There’s a body of ML security literature showing models can memorize and leak training data. Membership‑inference and model inversion attacks show how much can be reconstructed without raw logs. Notable work includes studies on extracting training data from language models and classical membership‑inference research, for example Nicholas Carlini and colleagues’ extraction studies, and earlier membership‑inference/model‑inversion papers. These results don’t mean every model will leak your secrets tomorrow, but they show de‑identification and simple redaction are imperfect.
More robust protections such as differential privacy (DP) offer measurable guarantees, but DP reduces utility and is hard to apply at foundation‑model scale without significant engineering and accuracy costs. Synthetic data can reduce direct exposure, but unless it is generated under strict privacy constraints, for example with DP, it can preserve statistical signals that a sophisticated lab could use to approximate your domain.
What the industry needs to fix next
Short answer: precise guarantees and verifiable controls. Vendors should publish clear telemetry contracts, offer customer‑owned logging and dedicated instances, and support enforceable non‑training clauses. Independent technical and legal audits must become standard for enterprise AI deals.
Procurement teams should stop treating ZDR as a checkbox and start asking exact questions: what fields are collected, how long they are retained, how they’re used, and how the vendor proves compliance. Labs need to adopt a culture of explicit disclosure about how customer interactions influence model development.
Final thought
The Fable episode is not just about a 30‑day window, it exposed a persistent misalignment between how frontier labs evolve models and how enterprises must protect IP and comply with rules. Until vendors replace fuzzy marketing with precise, auditable promises and buyers insist on them, these trust fissures will keep widening, quietly, project by project.
Practical checklist for procurement teams (fast action)
- Must ask immediately: Get a written list of telemetry fields, a non‑training clause, and a customer log‑storage option.
- Within 30 days: Update procurement templates to include audit rights and threat‑modeling requirements for AI use.
- Within 90 days: Pilot private instances or VPC deployments for sensitive teams and run a reconstruction/membership‑inference test with an independent firm.
- Ongoing: Require SOC 2 + technical red‑team evidence annually, and negotiate exit provisions (data extraction, immediate telemetry cessation) in every contract.
Protecting IP in the age of large models is as much a legal and governance exercise as it is a technical one. Treat vendor claims as starting points for negotiation, not final answers, and build procurement muscle so your company captures AI’s productivity upside without subsidizing a competitor’s future product.
Key questions, quick answers
-
Can I trust a vendor’s “we won’t train on your data” promise?
Trust depends on contract and technical controls. Marketing claims are weak; require explicit non‑training clauses, telemetry disclosures, and audit rights written into the agreement.
-
Does zero data retention eliminate risk?
ZDR reduces exposure but doesn’t erase it. Vendors may still collect metadata or derive synthetic data, and de‑identification can be broken or leak statistical patterns.
-
Could short‑term logs (e.g., 30 days) leak IP?
Yes. Even ephemeral logs can be used for debugging, creating reward signals, or deriving embeddings that capture structure. Whether this leads to substantive IP transfer depends on vendor practices and contractual limits.
-
Are vendors transparent about collected metadata?
Not consistently. Some vendors describe telemetry as “de‑identified, ” but buyers should demand a precise field list and the ability to opt out of nonessential collection.
-
Do customer inputs drive major frontier capability gains?
Most historical capability gains stem from compute, model scale, and diverse pretraining data. Experts argue customer data is unlikely to be the primary source of frontier leaps but can still leak domain‑specific IP or fine‑tune behaviors.