Generative AI Customization Spectrum: When to Use Prompting, RAG, Distillation, or Full Training

“The head chef’s tasting menu is flawless but takes 45 minutes per plate and costs a fortune in ingredients. Teach the line cook the three bestsellers, serve them in 10 minutes at a third of the cost.”

That image captures the trade-off every executive faces with generative AI: higher fidelity usually brings higher cost, slower delivery, and more operational burden. AWS frames that trade-off as an 8-step “generative AI customization spectrum” to help teams choose how much customization a workload actually needs, from calling foundation models as-is through prompt work, retrieval and distillation, up to fully custom foundation-model training with Amazon Nova Forge. The guidance is practical: start with the cheapest, simplest fix and only escalate when you see clear failure signals.

Why this matters for product and engineering leaders

Every rung you climb buys better domain fit and control, and it costs you compute, developer time, governance effort, and latency. Treat customization like product development. Ship the minimum that fixes the core user problem, instrument the result, then iterate. The vendor maps each step to managed services (Amazon Bedrock, Amazon SageMaker, Amazon Nova Forge) and supplies signals and customer examples to make the decision operational rather than theoretical. Those mappings and metrics are vendor‑reported, so validate them in your environment before budgeting large projects.

The 8-step customization spectrum (AWS framing)

USE (Steps 1-2)
- Call a foundation model as‑is, no weight changes.
- Prompt engineering / in‑context learning, improve system prompts, few‑shot examples, or chain‑of‑thought to get better answers without touching weights.
ENHANCE (Steps 3-5)
- Retrieval‑Augmented Generation (RAG), attach private or frequently changing documents at inference time to reduce hallucination.
- Prompt caching and context optimization, precompute static prefixes (system instructions, long context) to cut latency and token cost.
- Model distillation, train a smaller “student” model to mimic a larger teacher for cheaper, faster inference.
TRAIN (Steps 6-8)
- Fine‑tuning, parameter‑efficient methods (PEFT) or full weight updates using labeled examples.
- Continued pre‑training (CPT), self‑supervised training on large unlabeled domain corpora to instill deeper domain knowledge.
- Custom foundation model training, build from checkpoints and curated data mixes (Amazon Nova Forge is the vendor offering here).

Definitions on first use: Retrieval‑Augmented Generation (RAG), Parameter‑Efficient Fine‑Tuning (PEFT, e.g., LoRA), Continued Pre‑Training (CPT), Direct Preference Optimization (DPO), Reinforcement Fine‑Tuning (RFT).

When to move up the staircase, practical escalation signals

Prompt engineering → RAG: you repeatedly hit a practical prompt-length ceiling in production (a common heuristic teams report is ~2, 000 tokens, depending on model and cost), or the model keeps failing on facts that live in private or rapidly changing sources.
RAG → Distillation or Fine‑tuning: cost-per-request or latency at scale is unsustainable, and retrieval isn’t meeting your SLA or throughput goals.
Fine‑tuning → Continued pre‑training: the model lacks structural domain knowledge (reasoning patterns or representations) that labeled examples and retrieval cannot correct.
CPT → Full custom foundation training: you need architectural changes, formal data‑mixing guarantees to avoid catastrophic forgetting, or integrated reinforcement workflows that existing fine‑tuning pipelines can’t deliver.

Data and scale heuristics (use as guardrails, not laws)

Prompt ceiling: many teams use ~2k tokens as a practical heuristic, though model, costing, and workflow specifics change that threshold.
Fine‑tuning labeled data: PEFT methods often work with a few thousand labeled examples (sometimes with only hundreds for simple tasks), and full fine‑tuning typically needs tens of thousands, depending on task complexity and label quality.
DPO (Direct Preference Optimization): trains on pairs of preferred vs. non‑preferred responses rather than relying on a separately learned reward model, useful where preference data exists.
Continued pre‑training (CPT): vendor guidance places mid‑training at 1B+ tokens and full CPT in the 1T+ tokens range, these are order‑of‑magnitude signals that depend heavily on model size, the domain, and training recipe. Expect hundreds of millions to billions of tokens for modest domain adaptation, and much larger corpora for general‑purpose domain pretraining.

Vendor‑reported tooling and performance notes (validate before you plan)

Vendor announcements contain specific performance claims and feature timelines that are useful directional signals, but they require validation on your workloads. For example, the Amazon Bedrock Model Distillation general availability announcement (May 2025) reports student models up to 500 percent faster and up to 75 percent less expensive than the teacher, with less than 2 percent accuracy loss. Treat such figures as vendor‑reported results and test them on representative data. Distillation outcomes vary a lot by task and student architecture.

Similarly, AWS reports the following vendor timelines and tooling availability, and these should be confirmed against official documentation and release notes before procurement or roadmapping:

Reinforcement Fine‑Tuning (RFT) became available for Amazon Nova models in December 2025 and expanded to some open‑weight models in February 2026 (vendor‑reported).
Nova Forge SDK (“pip install amzn-nova-forge”) appears on GitHub as of March 2026 (vendor‑reported).
Custom Amazon Nova models trained after July 2025 can reportedly be served with pay‑per‑token inference on Amazon Bedrock without provisioned throughput (vendor‑reported).

Customer outcomes, two short, annotated vignettes and a summary

Vendor case studies show how teams applied different rungs. Treat these as vendor‑reported outcomes and validate the details in each customer’s writeup before using them as a planning baseline.

RAG at contact centers (vendor‑reported): Fractal Analytics used RAG to handle 200K+ monthly queries and reported a 10-15% reduction in call handling time plus 30% call deflection (vendor‑reported). That pattern, retrieval reducing search and lookup latency and hallucination, is typical for knowledge‑heavy support flows where documents change frequently.
Fine‑tuning for product integrations (vendor‑reported): Trellix fine‑tuned models and reported saving over 40 hours of development time per integration and a 90% reduction in time‑to‑market for new security integrations (vendor‑reported). This maps to an obvious play: when repeated, narrow engineering tasks dominate effort, fine‑tuning can codify those behaviors and reduce manual work.

Other vendor‑reported examples include EXL (80% reduction in underwriting costs via a RAG assistant), inGenious.ai (sub‑1s chatbot responses and 80% improved comprehension using Nova, baseline not disclosed), Sonrai (50% faster research with 5x fewer errors, up to $20k savings per experiment), and more. Use these reports as inspiration, not guarantees. Request methodology and baselines from vendors and, when possible, corroborate with independent benchmarks or customer references.

Common mistakes that push teams to premature complexity

“We have a lot of data.”, Volume alone is not a reason to choose CPT. Ask whether a few thousand labeled examples or a RAG approach would solve the failure mode first.
“RAG is too slow.”, Latency issues often have cheaper fixes: prompt caching, context optimization, localized inference, or distillation. Measure before you train.
Skipping governance: moving private data into model weights (via CPT or full fine‑tuning) materially changes your compliance surface. Plan encryption, access control, and auditability before you upload sensitive corpora.
Failing to pilot: don’t train at scale until small representative pilots validate accuracy and operational costs.

Instrumenting failure modes, a short checklist

Track prompt and token counts per request and their distribution over time.
Log latency percentiles (p50, p95, p99) and cost‑per‑request.
Record hallucination incidents with the source of truth and whether RAG would have fixed them.
Measure accuracy by intent and type, and track drift over time.
Keep a labeled‑data inventory, including counts, quality, and provenance.

Practical roadmap, what to do, in order

Instrument current failures (use the checklist above) and define success criteria for each experiment.
Start with better prompts and prompt evaluation tooling. Iterate system prompts and few‑shot examples and only move on when you hit persistent token or accuracy ceilings.
Introduce RAG if your failures are access to private or rapidly changing knowledge. Use managed knowledge bases to keep facts external and updatable.
If latency or cost is the blocker at scale, benchmark distillation on representative workloads and measure accuracy against cost and latency tradeoffs.
Only pursue PEFT or full fine‑tuning when labeled data and workload analysis indicate they will fix issues that retrieval and prompt engineering cannot.
Reserve CPT or full custom foundation training for cases requiring deep domain representations, architectural change, or integrated RL pipelines, and budget governance, validation, and long‑term maintenance.

Decision cheat‑sheet (one‑line triggers)

If failures are phrasing or formatting or you haven’t tried system prompts: Prompt engineering first.
If failures are factual errors tied to private or changing documents: RAG next.
If cost or latency is the blocker at scale: Try prompt caching, distillation, or edge inference before full fine‑tuning.
If the model lacks domain reasoning that labels and retrieval can’t fix: Consider PEFT, then full fine‑tuning, then CPT depending on data scale.
If you need new architecture, strict data‑mixing guarantees, or integrated RL workflows: Evaluate full custom foundation training (vendor‑reported offerings exist for this stage).

Key questions, quick, honest answers

When should I try prompt engineering first?

Start here in nearly all cases, it’s the cheapest, fastest way to improve behavior. Move beyond it when prompts regularly hit your practical token ceiling (a common heuristic is ~2, 000 tokens) or when the model fails on facts it cannot retrieve.
When does RAG outperform fine‑tuning?

When the problem is access to private or frequently changing documents. RAG keeps facts external and updatable, avoiding the cost and governance overhead of retraining weights.
How much labeled data do I need to fine‑tune?

PEFT methods can work with a few thousand labeled examples (sometimes fewer for simple tasks), and full fine‑tuning typically needs tens of thousands, depending on task complexity and label quality.
When is continued pre‑training justified?

If the model systematically lacks deep domain structure and retrieval or fine‑tuning can’t fix it. Vendor guidance points to 1B+ tokens for mid‑training and 1T+ for full CPT as order‑of‑magnitude signals, exact needs vary with model size and domain.
Are vendor performance claims (like distillation speedups) reliable for planning?

They’re useful directional signals but must be validated on your task. For instance, the Amazon Bedrock Model Distillation announcement (May 2025) reports large speed and cost gains with minimal accuracy loss, so test with representative data before scaling.

Final, pragmatic commitments for leaders

If you build a multi‑quarter AI roadmap, commit to three operational rules:

Instrument everything: within your first sprint, collect token counts, latency p95 and p99, hallucination logs, and a labeled‑data inventory.
Validate vendor claims with small pilots: run short, representative POCs to confirm accuracy, latency, and cost tradeoffs before approving large training jobs.
Governance as a hard constraint: moving data into weights changes your compliance obligations, require encryption, access controls, and auditability before you train.

Think of customization like product feature work. Ship the minimal fix that meets the user need, measure outcomes, then invest in heavier interventions only when the ROI and governance plan are clear. Keep the tasting menu for when it truly matters, and let the line cook win the routine days.