Multi-turn RL on AWS: Nova Forge + SageMaker HyperPod production pipeline and cost trade-offs

TL;DR, Executive summary

What this recipe does: it stitches Amazon Nova Forge, SageMaker HyperPod, and AWS primitives into an event-driven pipeline that trains multi-turn reinforcement learning agents, agents that learn across entire interaction sequences instead of scoring single replies.

Who should care: product and ML teams building agents that must orchestrate APIs, query databases, handle mid-run errors, or otherwise depend on downstream outcomes to judge earlier actions.

Single-line trade-off: high fidelity and control for multi-step behavior at the cost of P5-class GPU spend and nontrivial operational work, AWS flags roughly $786, $1, 180 per hour for the recommended HyperPod configurations (see the AWS post’s cost breakdown).

Decision checklist, Do this if you need durable multi-step orchestration, delayed-reward learning, and you can budget P5-level GPU runs. Don’t do this if your problem is limited to single-turn response tuning (RLHF) or you can’t tolerate significant GPU spend or extra infra complexity.

Why multi-turn RL, and why Amazon’s recipe matters

Training a model only for single replies (RLHF) is like grading chess moves in isolation. A strong opening only matters if it helps you win later. Multi-turn RL trains policies across whole conversations so agents learn to plan, call tools, and recover from mistakes mid-run. That becomes essential when rewards arrive several steps after a decision.

AWS shows a production-ready pattern that pairs Nova Forge (the Nova Forge SDK and Nova model APIs) with SageMaker HyperPod (EKS-backed P5 GPU training) and managed AWS services to run multi-turn RL at scale. The pattern is event-driven: upload a dataset to S3, EventBridge fires Step Functions, Nova Forge deploys a per-run runtime, reward workers score responses, and HyperPod runs distributed updates (GRPO).

Who needs this stack

  • Teams building agents that must call internal APIs, update databases, manage transactions, or execute multi-step human workflows.
  • ML groups that want reproducible, production-grade training loops with checkpointing, observability, and reward routing.
  • Not a fit if you only need better single-turn outputs, are on a tight budget, or cannot provision P5 GPU capacity.

How the pipeline fits together, a step-by-step flow

  1. Upload a .jsonl training file to S3 under training-data/.
  2. Amazon EventBridge detects the S3 PutObject and notifies Step Functions.
  3. Step Functions drives a per-run runtime deployment via the Nova Forge SDK: Lambda conversation proxy, SQS FIFO request/response routing, a DynamoDB conversation-state table, and ECS Fargate reward workers.
  4. HyperPod (running on an EKS cluster backed by P5 instances) provides vLLM generation replicas for fast inference and separate training pods that perform GRPO weight updates using rewards from the Fargate workers.
  5. Training proceeds until max_steps or convergence; checkpoints and logs are stored in S3 and visible via Step Functions and CloudWatch.
  6. After the run you can destroy the ephemeral runtime (Phase 2) to avoid further GPU costs; Phase 1 (the foundational CDK stack) remains if you choose, but it’s lightweight comparatively.

Key components (plain language)

  • SageMaker HyperPod: EKS-backed cluster that runs training pods and vLLM generation replicas on ml.p5.48xlarge (or ml.p5en.48xlarge) instances.
  • Nova Forge SDK: per-run deployer and message proxy. It wires SQS FIFO queues, Lambda proxies, DynamoDB state, and ECS reward workers.
  • ECS Fargate reward workers: your custom environments that score model responses and return reward signals.
  • Step Functions + EventBridge: event-driven orchestration that glues S3 uploads to runtime deployments.
  • S3 / DynamoDB / SQS: storage, conversation state, and message routing, respectively.

Defaults, quotas, and the real cost (what you must plan for)

The sample repo ships conservative local defaults in cdk.json for safety during initial experiments and a recommended production baseline in the docs. For example, cdk.json defaults include instance_type: ml.p5.48xlarge and instance_count: 10, but the quickstart CLI examples often deploy with instance_count=1 so you can test locally without P5 capacity. In short: local test = 1; production baseline = 10 (request 12-14 for headroom).

AWS calls out compute as the dominant line item. The AWS post reports an approximate running cost of $786, $1, 180 per hour for the HyperPod configurations cited (10-12 ml.p5.48xlarge instances). Other infrastructure costs are small by comparison, EKS control plane (~$0.10/hr), NAT Gateway (~$0.045/hr), and ECS Fargate reward workers (~$0.15, $0.30/hr plus data transfer), but they add up over many runs. That cost range and the “destroy when idle” warning come directly from the AWS blog post’s cost breakdown.

Practical budgeting recipe: Estimate = (# P5 instances × instance $/hr × run hours) + modest infra (EKS control plane, NAT, Fargate). Because instance $/hr varies by region and contract, treat the AWS range as a planning anchor and request GPU quotas early. Plan 2-4 weeks to secure production quotas and scheduling.

Training mechanics and knobs you need to know

  • Training methods: RFT_MULTITURN_FULL (full fine-tuning) or RFT_MULTITURN_LORA (LoRA parameter-efficient updates). LoRA is faster and cheaper for iteration. Full fine-tuning is heavier but may be needed for larger model changes.
  • GRPO: the guide uses Group Relative Policy Optimization (GRPO) for weight updates. GRPO aggregates reward signals across training pods and groups updates to stabilize multi-turn policy learning in distributed setups.
  • vLLM generation replicas: these replicas serve fast batched inference during rollout/generation. Training pods perform the GRPO updates; generation replicas do not perform weight updates.
  • Defaults in the sample: generation_replicas: 4, global_batch_size: 64, max_steps: 10 (adjust for your task). The sample environment uses “wordle” as a compact multi-turn demo.
  • Subscription: an Amazon Nova Forge subscription is required to access the Nova Forge SDK and training APIs.

Getting started (commands and practical sequence)

Quickstart commands from the sample repository (good for a lab run):

  • git clone https://github.com/aws-samples/nova-multi-turn-rl-infra.git
  • cd nova-multi-turn-rl-infra
  • pip install -r requirements.txt (Python 3.12+)
  • cdk deploy -c instance_count=1 -c max_steps=20 (local test)
  • For production-style runs use cdk deploy -c instance_count=10 (after you’ve requested P5 quotas)

Deployment timing estimates (rough): full CDK deploy ~30-40 minutes total: EKS cluster creation ~15 minutes, HyperPod Helm + codebuild steps ~5 minutes, HyperPod provisioning ~15-25 minutes (P5 capacity dependent), Lambda container builds ~5 minutes.

To trigger a run: upload a .jsonl file to S3 training-data/. Example lines used by the demo include id and metadata (prompt and answer). For Wordle the authors report convergence within ~50-100 steps with average reward rising from near-zero to around 0.6-0.8, an illustrative sanity check reported in the AWS post (where “step” refers to a training update/iteration in the demo run).

Observability, debugging, and common failure modes

Operational visibility centers on Step Functions and CloudWatch logs. Step Functions shows per-run progress and links to the log groups for each step. CloudWatch captures Lambda, CodeBuild, and container logs.

Troubleshooting checklist (first things to inspect)

  • If training never starts: check Step Functions execution details for the failure reason and review CloudWatch logs linked from that step.
  • If reward messages are missing or stuck: verify SQS queues and dead-letter queues. Example: aws sqs receive-message –queue-url $DLQ_URL –max-number-of-messages 5.
  • If HyperPod pods are pending/unscheduled: check node capacity with kubectl get nodes and pods with kubectl get pods -n kubeflow. P5 capacity or scheduling constraints are common culprits.
  • If Fargate reward workers crash: inspect ECS task logs in CloudWatch and Lambda proxy logs for malformed payloads or permission errors.
  • Use the sample diagnostic helper: from rft_infra import check_all_queues; check_all_queues(), it helps surface queue and routing issues the Nova Forge SDK deploys.

Typical reasons messages land in the DLQ include timeouts in the reward worker, permission errors between Lambda and SQS, malformed JSON payloads, and transient network issues. Start with the DLQ and then follow links to the CloudWatch logs for the failing component.

Security and compliance practicalities

  • Apply least-privilege IAM. Training pods, reward workers, and deployment Lambdas should have narrow roles limited to the S3 prefixes, DynamoDB tables, and SQS queues they need.
  • Encrypt sensitive artifacts. Use KMS for S3 objects and DynamoDB encryption where conversation logs contain sensitive or regulated data.
  • Network posture: use VPC endpoints for S3 to avoid NAT egress, keep training nodes in private subnets, and segregate reward workers if they need different network access.
  • Data governance: sanitize PII before using production data for reward signals, and define retention policies for conversation logs and checkpoints.

Where this pattern shines, and where you must engineer further

Strengths: it gives you a managed-plus-customizable pattern to train agents that reason across turns, with clear per-run isolation via deployed runtime stacks and observability through Step Functions and CloudWatch.

Design areas you must address yourself: fine-grained IAM mappings, production autoscaling policies and throughput testing for SQS/DynamoDB limits, long-term checkpointing and disaster recovery, and concrete cost-containment tactics such as scale-to-zero automation and mixed-instance scheduling.

Practical recommendations before you flip the switch

  • Prototype with non-sensitive, compact tasks. The Wordle demo is ideal to validate the pipeline, reward routing, and convergence before you port production logic.
  • Request P5 quotas early, plan 2-4 weeks for approvals and testing in your target region.
  • Prefer LoRA (RFT_MULTITURN_LORA) for quick iteration. Move to full fine-tuning when your use case justifies the cost or requires architecture changes.
  • Instrument everything: Step Functions and CloudWatch, SQS DLQs, DynamoDB backups, and routine snapshotting of checkpoints in S3.
  • Plan cost guardrails: automated teardown of Phase 2 after runs, alerts on long-lived HyperPod instances, and conservative runtimes for experiments.

Quick operational commands

  • Deploy local test: cdk deploy -c instance_count=1 -c max_steps=20
  • Request production deploy (after quotas): cdk deploy -c instance_count=10
  • Check Kubernetes pods: kubectl get pods -n kubeflow
  • Inspect dead-letter queue: aws sqs receive-message –queue-url $DLQ_URL –max-number-of-messages 5
  • Cleanup infra: ./cleanup.sh (or ./cleanup.sh –retain-data to keep S3 artifacts)

Resources and pointers

Sample infrastructure repo: github.com/aws-samples/nova-multi-turn-rl-infra

Nova Forge SDK repo: github.com/aws/nova-forge-sdk

Check the AWS blog post that documents the recipe and the cost breakdown for the quoted hourly ranges and Wordle convergence observations (the post contains step-by-step screenshots and configuration defaults).

Key questions leaders will ask, with straight answers

  • Do I need multi-turn RL instead of RLHF for my agent?

    Yes if three conditions hold: (1) decisions have delayed rewards or downstream effects; (2) the agent must orchestrate multi-step operations (APIs, DB changes); and (3) you require robust error recovery and sequential planning. If your needs are limited to single-turn quality improvements, RLHF or supervised fine-tuning may be sufficient and far cheaper.

  • What are the minimum compute and subscription requirements?

    An Amazon Nova Forge subscription is required to access the Nova Forge SDK and model APIs. The sample recommends a production baseline of ~10 ml.p5.48xlarge instances (request 12-14 for headroom); the repo defaults to 1 for safe local testing.

  • How much will this cost to run?

    AWS reports roughly $786, $1, 180 per hour for the HyperPod P5 configurations used in the sample (10-12 ml.p5.48xlarge), with other infra costs minor by comparison. Estimate total cost as: (# P5 instances × instance $/hr × run hours) + modest infra and storage fees. Treat the AWS range as a planning anchor and optimize run time aggressively.

  • How long does deployment take?

    Expect a full CDK deploy of foundational infra in ~30-40 minutes. Per-run runtime deployments are fast, infrastructure setup and reward worker deployment typically finish in a few minutes; HyperPod provisioning for large instance requests can be the slowest external factor.

  • Can I run custom reward environments?

    Yes. The Nova Forge SDK supports custom environments (set use_custom_env true and specify custom_env_id); it deploys reward workers on ECS Fargate that you can implement to score episodes and return reward signals.

Next practical step

If you’re evaluating this for production: clone the sample repo, start with instance_count=1 to validate the control plane and runtime wiring, request P5 quotas for your region, instrument the pipeline with DLQ alerts and CloudWatch dashboards, and run the Wordle demo end-to-end to inspect logs, reward traces, and convergence behavior. You’ll learn far more from a real run than from diagrams alone.

0