In an early scoring scheme for a Wordle-playing agent, solving the puzzle on the third guess paid more than solving it on the first. The model was not being clever. The scoreboard was lying.
Most leadership teams skip this part of agent training. They ask whether the model can write, classify, or call a tool. The harder question is what the system treats as a win once the work takes five steps and the useful signal shows up only at the end.
The job is the episode, not the reply
Ordinary fine-tuning wants a labeled answer. Single-turn reinforcement learning, the RLHF and RLAIF pattern most teams already know, puts one reward on one output. AWS draws a harder line for multi-turn reinforcement learning on SageMaker AI: train a policy across a sequence of observations and actions, and aim at cumulative reward for the whole episode.
Real agentic work has that shape. The agent looks, acts, sees what changed, and acts again. A decent move on step two can still wreck the close. A clumsy middle step can be the one that makes the ending possible. AWS points to the cases that show up in AI for business: back-and-forth work, outcomes that depend on a chain of actions, multi-step tool use, mistakes the agent is allowed to recover from, and models that handle a single step cleanly but fail end to end.
Support diagnosis fits. So does a coding agent that tests a fix and changes course. So does a research agent that refines a search instead of stopping at the first hit. Anyone looking at AI for sales will recognize the same skeleton: research the account, check the CRM, draft, send, read the reply, adjust. Score the email alone and you train a copywriter. Score the meeting, or the opportunity that actually moved, and you are training an AI agent.
That is also why “just use a bigger model” is a weak plan for internal tools. Public models were not trained on your approval path, your claims codes, or the odd system only your team knows how to query. Multi-turn training is a bet that a smaller policy can learn your loop and lift finish rates. It does not bid for SI.
Pay for the solved word, not the green tiles
The classroom in AWS’s hands-on guide is Wordle, not a live workflow. Six guesses, a hidden five-letter word, feedback in green, yellow, and gray. Few rules, a hard dependence on earlier moves, and a result you can score without arguing about taste. Frontier models already play this game well. The intended use is a small, cheaper model on a procedure those public models never saw. That business workflow was not the run. Hold every solve rate below to that standard.
The first reward borrowed per-guess points and added them across the game. A loss paid about +0.49. A first-guess solve paid +0.99. A third-guess solve paid +1.61. A slow win beat a fast one, and a loss still paid positive. An agent trained on that board will collect points, not solutions.
A partial repair still broke the order. With a single style weight of 0.15, a fully shaped third-guess win at +1.45 outscored a minimally shaped second-guess win at +1.25. How nicely the model played was allowed to overturn how fast it finished.
The version that held is outcome-dominant. Finish first. Style second, and only inside the tier already earned.
A solve on guess n starts at 1.05 + 0.5 × (6 − n) / 5. That is 1.55 on guess one, 1.45 on guess two, 1.35 on guess three, and 1.05 on guess six. Adjacent solves sit 0.1 apart. Six failed guesses score −0.5. Never making a valid guess scores −1.5.
A shaping term, for things like using the clues, is clipped to a range of −1 to 1, then multiplied by 0.04 after a win or 0.15 after a loss. The biggest style bonus on a win is 0.04, smaller than the 0.1 gap between adjacent solves. Style can nudge. It cannot promote a loss over a win, or a slow win over a fast one.
The scores reported as typical already include that small adjustment. A third-guess solve at +1.375 is the 1.35 outcome plus 0.025 of style credit. A sixth-guess solve at +1.069 is 1.05 plus 0.019. A six-guess loss at −0.427 is −0.5 plus a modest style credit that still leaves the game in the red. A first-guess solve sits at +1.550. No valid guess sits at −1.500.
Even the worst-styled solve lands at 1.01, which is 1.05 minus 0.04. That clears the 1.0 cutoff used for the pass rate, so every real solve counted. A loss cannot get there.
The formula matters less than the habit. Start with the ending, add shaping sparingly, and check the numbers on real trajectories before you trust a curve. If your AI automation program pays for activity (emails sent, tools called, tokens spent), you will get activity.
A short loop, not a labeled script
SageMaker AI runs the training job and hosts the policy model, the model sampled to choose each action. Amazon Bedrock AgentCore hosts the agent, the game, and the reward. The agent plays through one tool and returns one score when the episode ends. The same style of job can take a reward per turn. This example did not. One game, one score.
There is no labeled winning path to imitate. The dataset is a situation, not a script: 640 training words and 128 held-out words, no overlap, and no target answers. In this run the update method was Group Relative Policy Optimization. The model plays the same secret word several times (eight by default, so a batch of 128 is 1, 024 games per step), and a small LoRA adapter moves based on how those attempts score against each other. Several plausible paths can win. The trainer only needs to know which endings were better.
The team picked GPT-OSS-20B because, of the four bases listed then, it had the lowest per-token price, and its reasoning effort can be set per request. That catalog, and which regions host which model, moves. Check the current page before you treat any list as a decision. In the guide’s account, you pick the base and the service provisions training and sampling capacity, rather than handing you a cluster to size. The guide also states that the dataset, the agent, and the trained model stay in the customer’s account and are not used to train the base models.
The chart is not the model you would ship
With sampling on, the guide reports a jump from 61 percent to 84 percent on held-out words at default reasoning effort, and from 15 percent to 69 percent at the low effort used in training. Default, in this setup, means medium. The 84 percent is specifically medium effort at temperature 1.0, the mean of two runs at 80 percent and 88 percent, against 61 percent for the base model. Same teaching set. With only 128 words, treat it as directional, not a stable benchmark.
The training chart is a different measurement, and the gap between the two is the finding. Reward climbed to about +0.57 within 15 steps and held there. The validation solve rate jumped from 2 percent to 31 percent in the first 10 steps, then wandered between 35 and 48 percent. On 128 words, the guide treats that wander as mostly noise. The curve peaked at 48 percent on step 90 and ended at 37 percent.
That 37 percent is temperature 0, greedy decoding, at the low reasoning effort baked into the agent. The training job locks validation there so the curve stays comparable from step to step. Temperature 1.0 is how they would actually serve the model: let it sample, instead of always taking the single most likely token. At temperature 0, both the base model and the trained one often finished without a solve.
Read the matched comparisons, not a single headline. At low effort, the effort they trained at, the temperature-0 curve ended at 37 percent, while letting the model sample produced 69 percent, up from 15 percent for the base. Raising effort to medium, still with sampling on, produced the 84 percent, up from 61 percent. Medium effort added 15 points for the trained model and 46 points for the base model relative to low effort. Low effort used 63 to 83 percent fewer sample tokens per game than medium.
Each 128-word evaluation took four to seven minutes. The jobs behind the serving-style result cost about $1.20 in tokens and about an hour. Green-light a model from the training chart alone and you can reject a policy that is much stronger at the settings you will ship. Or you trust a number that exists only when the model is forced to be deterministic.
Reasoning effort made the comparison awkward in practice. It lived in the agent’s environment, including a “Reasoning: low” hint in the system prompt. Changing it meant redeploying, and redeploying during a live job restarts the runtime and fails the job. They did not test whether more than 100 steps would help at serving settings. They also did not test a curriculum that starts games with prior guesses, or a group larger than the default eight. Those are open ideas, not results.
What the bill actually measures
Do not budget from the figures below. They are GPT-OSS-20B list prices in US East (N. Virginia) as quoted in the hands-on guide at the time of that write-up, not a current quote. Check the SageMaker pricing page for rates actually in force.
Three meters make up the training bill. Prefill is input, and it piles up because every turn re-reads the history. Sample is what the model generates, including reasoning tokens. Train is the backward pass that updates the adapter. Train grows with trajectory length, and it was most of the money: $231.59 of the $338.50 token total.
They had to raise the sampling cap to 8, 192 tokens, the service maximum, because the 4, 096 default often ran out before the first tool call. Longer trajectories cost more. They land on the meter that already dominates.
- Prefill: 491, 616, 806 tokens at $0.12 per million, $58.99
- Sample: 159, 737, 098 tokens at $0.30 per million, $47.92
- Train: 643, 308, 554 tokens at $0.36 per million, $231.59
- Total: 1, 294, 662, 458 tokens, $338.50
AgentCore runtime, billed separately for compute, came to about $79 from CloudWatch usage at Northern Virginia list prices. The guide does not break that $79 into components. The 100-step job ran 12 hours and 40 minutes. Eight attempts per word and 128 words a step would be 102, 400 games. The guide reports 105, 486 rollouts and does not itemize the few thousand above that clean count. Of those, 51 never made a valid guess. Rare. That is why “no valid guess” was the harshest score, and it is still a failure mode to price into the reward, not one to find later in a dashboard.
Training and evaluation stop billing when they finish. The runtime, its role, the metrics app, and the S3 objects do not. Token price is not the whole system. These jobs also fail in boring ways: role setup, sampling connectivity, and credential lifetime. Have an engineer read the guide before a pilot becomes a standing job.
This is not Training Sessions
SageMaker later added Training Sessions, a related customization path. Managed multi-turn reinforcement learning predates that launch. It is not this job. VentureBeat’s account of the later launch lists a different model and region mix, including Mumbai for some models and a Qwen 3.8 variant, and it does not mention Nova Lite. A reported default of two simultaneous fine-tuning sessions is not a quota for multi-turn evaluation.
What to ask before you fund a run
- What is multi-turn reinforcement learning actually optimizing?
The outcome of a whole episode, a chain of observations and actions, rather than one labeled reply. AWS draws that line explicitly against single-turn RLHF and RLAIF.
- Why did the first Wordle reward fail?
Summed per-guess points let a third-guess win outscore a first-guess win, and paid a loss a positive score. The revised reward makes the ending dominate: the largest style bonus on a win, 0.04, is smaller than the 0.1 gap between adjacent solves.
- Did the model really reach 84 percent?
That is what the hands-on guide reports on held-out words at default effort, which in this setup is medium, with sampling on at temperature 1.0. It is the mean of runs at 80 and 88 percent, up from 61 percent for the base model. The training job’s temperature-0 curve, at the low effort used in training, ended at 37 percent. The gap is the finding.
- What did it cost?
Train tokens dominated: $231.59 of a $338.50 token bill, at GPT-OSS-20B list prices in the guide at the time of that write-up, not a current quote. AgentCore runtime added about $79, and the runtime, storage, and metrics plumbing keep costing money after the job stops. Do not budget from these figures.
- Is this the same product as SageMaker Training Sessions?
No. Managed multi-turn reinforcement learning predates that launch, and the model lists and regions do not match. Do not copy a quota or a region table from one onto the other.
- Will this transfer to sales, support, or an internal tool?
AWS describes that class of task as the reason the method exists. The Wordle run does not show results on one. A fitting pipeline is not demonstrated performance.
- Should we just train longer?
Unknown. The team did not test past 100 steps at serving settings, and they did not test a curriculum or a larger group of rollouts per case.
- How do we know the reward is safe to train on?
Hand-score a few dozen real trajectories and confirm a bad ending cannot outrank a good one, at the temperature, reasoning budget, and tools you will actually serve. If a slow win beats a fast one on paper, the model will learn the paper.
If you take one habit from the game, make it this: write down the business ending you are willing to pay for, score a few dozen real trajectories by hand, and confirm that a bad ending cannot outrank a good one. Then evaluate at the temperature, the reasoning budget, and the tools you will actually serve. The rest is infrastructure. The score is the strategy.
When the state itself is sealed
A reward only works on what the policy can actually observe, and some interfaces are built so part of that state stays closed.