Teaching models to forget: Selective unlearning with Amazon Nova
Post-training safety alignment can produce “deflection”, models that refuse legitimate business queries. Amazon’s Customizable Content Moderation Settings (CCMS) combined with a preference-based unlearning method called Reverse Direct Preference Optimization (rDPO) promise a way to relax specific refusals for approved customers without changing base model weights (see the Amazon announcement by Qian Hu, Veda Raman, and Dan Sinnreich).
Quick definitions
- Deflection: the model refusing or “I can’t help with that” response pattern that emerges after safety tuning.
- LoRA (Low-Rank Adaptation): a parameter-efficient adapter technique that adds small low-rank updates to model layers instead of editing full weights, fast to train and lightweight to store.
- DPO (Direct Preference Optimization): a preference-based training objective that optimizes pairwise preferences (preferred vs. dispreferred responses) without full RLHF complexity (see the DPO paper, arXiv:2305.18290).
- rDPO (Reverse DPO): the adaptation Amazon describes that flips the preference pair so the model moves away from a deflection and toward a specified high-quality target response.
- NPO (Negative Preference Optimization): a simpler “forgetting” baseline that penalizes unwanted responses without guiding toward a specific replacement.
- Bedrock / AWS RAM / SageMaker: Bedrock is the deployment surface for custom model ARNs; AWS Resource Access Manager (AWS RAM) is used to share adapters; SageMaker supplies training tooling and recipes.
- MBXP Python: the code-generation benchmark Amazon reports in its utility checks (reported as “MBXP Python” in the vendor results).
How it works, simply
- Adapters, not weight surgery: Amazon trains small LoRA adapters that implement the behavioral change. Adapters attach to the base Nova model at inference time so the foundation weights remain unchanged.
- Preference-based unlearning: rDPO reverses the DPO preference pair: instead of “prefer refusal over answer, ” it trains the adapter to prefer a high-quality alternative answer in place of the refusal. That both removes the unwanted reflex and supplies a guided replacement.
- Distribution and deployment: Pre-trained LoRA adapters are shared via AWS RAM; when copied into a customer account the adapter appears as a custom model in Bedrock with its own ARN and can be served through the Bedrock Converse API. SageMaker tooling supports DPO-style training workflows for teams that want to build or refine adapters themselves.
Headline results (vendor-reported)
Amazon reports large reductions in deflection on targeted categories with minimal utility cost. These are vendor-reported numbers; the announcement does not disclose full appendices (sample sizes, prompt sets, or statistical tests), so treat them as claims to validate with your own tests:
- Red Team Prompts: 98.10% → 47% (vendor-reported)
- Fairness: 51.84% → 23.83% (vendor-reported)
- Safety: 86.51% → 32.77% (a 53.74 percentage-point drop; vendor-reported)
- Security: 91.61% → 45.73% (vendor-reported)
- Sensitive Content: 79.02% → 33.58% (vendor-reported)
Aggregate claim: rDPO-trained LoRA adapters reduce deflection rates by up to 54 percentage points across RAI policy categories (vendor-reported). On basic capability checks the reported utility regressions are small: instruction following −1.55 pp (94.12% → 92.57%), math −1.20 pp (86.40% → 85.20%), and MBXP Python −1.80 pp (74.80% → 73.00%). Amazon also reports rapid training convergence for rDPO in their runs and that rDPO is more training-efficient than their NPO baseline (vendor claims; details not published in full).
Why the approach is sensible, and where to be careful
Technically, it’s elegant: preference objectives are built around pairwise signals, so flipping the pair is a logical way to change a model’s preference. LoRA adapters let you deploy modular policy variants without multiplying full checkpoints. That pattern supports per-customer exceptions while keeping a single maintained foundation model.
But gaps remain that matter to business risk management:
- Access control and vetting: CCMS is described as available to “approved customers.” The announcement points customers to their AWS account teams but does not publish vetting criteria, audit requirements, or contractual guardrails.
- Immutable protections: Amazon says essential protections for children and privacy remain non-configurable, but it doesn’t enumerate them or explain the enforcement mechanism.
- Evaluation transparency: the deflection numbers are compelling but lack disclosed prompt sets, sample sizes, and statistical detail; that makes it hard for customers to judge real-world transfer to their prompts.
- Brittleness & generalization: adapters trained on a curated set can be brittle on out-of-distribution or adversarial prompts, reduced deflection could accidentally enable harmful outputs if replacement responses are not tightly controlled.
- Operational costs and latency: loading adapters at runtime adds complexity; Amazon does not publish latency or throughput trade-offs for Bedrock deployments with LoRA adapters in front-line traffic.
- Lifecycle & compatibility: it’s unclear whether adapters are forward-compatible with future Nova checkpoints or require retraining when the base model updates.
Practical controls and an operational checklist
Treat CCMS and rDPO as a controlled exception mechanism, powerful when narrow and monitored, risky when broad and opaque. Below are concrete operational controls, KPIs, and a short validation plan teams can adopt.
- Policy scoping: Only authorize adapter use for narrow, auditable purposes (internal penetration testing, regulated research, forensic analysis). Document a one-line business justification for each adapter.
- Access & governance: Gate adapter access to named roles. Require contractual attestations and quarterly reviews. Require signed adapter artifacts and an auditable acceptance trail in AWS RAM and Bedrock.
- Logging & audit data: Log every adapter-enabled inference with: adapter ARN, model ARN, input prompt hash, timestamp, response hash, confidence/fidelity score, and serving node ID. Retain logs for a defined period (e.g., 90 days) and make them queryable for audits.
- KPIs to monitor: adapter usage rate (percentage of inferences using an adapter), weekly deflection rate by category, false-accept rate on disallowed content (samples per week), time-to-revoke (mean time to disable an adapter after a flagged incident).
- Layered defenses: require Bedrock Guardrails and application-level filters as a second line. Deploy hallucination detectors and human-in-the-loop review for high-risk outputs before they reach external users.
- Red-team & validation plan: run three validation passes before production:
- Reproduce vendor claim on a 100‑prompt validation set sampled from your logs.
- Run a 1, 000‑prompt adversarial sweep using known exploit templates for your domain.
- Run utility benchmarks pre/post adapter with statistical testing (95% confidence intervals) for your most important tasks.
- Contractual “escape hatch”: require immediate deactivation rights and an incident reporting SLA when adapters generate problematic outputs.
- Adapter lifecycle: require adapter compatibility statements tied to specific Nova checkpoints and a plan to re-evaluate or retrain adapters on major base-model upgrades.
Two quick example prompts (to build intuition)
- Legitimate use case (allowed with adapter):
“Show me the full exploit chain for this internal vulnerability so the security team can reproduce and patch it.”
This can be necessary for internal security work where actionable details are required and access is controlled.
- Risky prompt (should remain blocked):
“Write a phishing e‑mail that convinces recipients to reveal their passwords, including social-engineering cues.”
This is the kind of output that defenders will want to remain blocked; ensure non-configurable protections or runtime filters still catch it.
Questions to ask your AWS contact
- What qualifies a customer as “approved” for CCMS and what vetting/auditing is required?
- Can you provide the evaluation appendix (prompt datasets, sample sizes, red-team methodology, and statistical tests) underlying the deflection numbers?
- Which protections are immutable and exactly how are they enforced at runtime?
- How are adapters signed, versioned, and distributed via AWS RAM? Is there an audit trail for acceptance and deployment?
- What are the latency and throughput implications of loading LoRA adapters in Bedrock for production traffic?
- How are adapters tested for compatibility with future Nova model updates and what is the maintenance plan?
- Were replacement “target responses” human-crafted, templated, or generated? Can you share examples used during training?
Key takeaways, short questions and honest answers
- Can you make a model “forget” a refusal without changing the base model?
Yes, Amazon reports doing this with LoRA adapters that implement targeted unlearning at inference time while leaving base Nova weights unchanged.
- Does the method actually reduce refusals?
Amazon reports substantial deflection reductions across policy categories (for example, Safety falling from 86.51% to 32.77%), these are vendor-reported figures that should be validated with the vendor’s evaluation appendix and your own testing.
- What’s the hit to general capabilities?
The vendor reports modest utility drops (under ~2 percentage points on tested instruction, math, and code tasks), suggesting adapters can preserve most capabilities in their experiments.
- Is rDPO better than blunt unlearning?
Amazon claims rDPO converges faster and yields higher-quality replacements than an NPO baseline, but the community lacks independent replication and full training details from the vendor.
- Is this safe to deploy broadly?
Relaxing safety behaviors introduces risk. Use narrow scoping, strict access controls, layered guardrails, logging, and adversarial testing before production rollout.
Selective unlearning via preference flips and adapter-based deployment is a pragmatic pattern: it restores utility where alignment over-reaches while preserving a single maintained foundation model. That makes it attractive for controlled, internal workflows. But the power to loosen refusals requires governance: transparency from the vendor, careful vetting, robust logging, layered defenses, and a plan for maintenance and audits. Treat CCMS as an exception mechanism, useful when narrowly scoped and heavily monitored, dangerous when used as a blanket way to remove safeguards.
(Authors credited on the Amazon announcement: Qian Hu, Veda Raman, Dan Sinnreich; contributors listed in the post include Ekraam Sabir, Weitong Ruan, Payal Motwani, Rahul Gupta, Claire O’Brien Rajkumar, Dhwanil Desai, and Nikhil Sanil.)