A small model can copy the loop. This demo does not price it.
A shopper wants a blender under $80, shipping tomorrow, still valid with a coupon. A capable agent does not answer in one breath. It searches, reads listings, recalculates the discount, checks the constraint, and only then recommends. Each of those steps can be its own call to a frontier model.
Hold that scene lightly. The evidence here is not live commerce. It comes from ShoppingBench, a controlled agent benchmark. Product fields are sampled first, then GPT-4.1 writes the user instruction. Coupon and budget tasks add synthesized voucher rules. Gold targets exist before the agent runs, which is why a program can score the result. Useful for training. Not a transcript of real customers, real coupons, or a carrier’s cutoff.
The cost worry is still a fair hypothesis. A chatbot answers once. An AI agent for sales or support can search, read, refine, verify, and decide before anyone sees a product. If every step is a billed frontier call, spend scales with the loop, not just with customers. Whether that erases the return is a traffic question. An AWS walkthrough by Giuseppe Zappia, Ankita Agarwal, Amin Dashti, and Michael Oguike does not answer it. No cost per task, latency, throughput, or break-even volume is published. Those depend on sequence length, concurrency, and how often the agent is used. Anyone turning the demo into a finance case is inventing the spreadsheet.
What the walkthrough does show is narrower, and more useful for AI automation: a frontier teacher can write solved multi-step trajectories, and a 4 billion parameter student can be tuned to imitate that behavior closely enough to matter on this task.
Copy the behavior, not the weights
Sequence-level knowledge distillation, SeqKD for short, is the plain end of a fancy idea. The teacher writes the target text. The student is fine-tuned on it. You do not need the teacher’s weights, a reinforcement-learning stack, or its token probabilities (the logits an API does not hand you). If the only access you have is an API, this is the method that still works.
For an agent, the thing being copied is a trajectory, not a finished paragraph. That trajectory is an ordered episode of reasoning, tool calls, and observations from the environment. The sequence is the behavior.
The recipe has three moves.
- A frontier teacher attempts each task, often more than once.
- A programmatic scorer throws out the runs that fail. That filter is rejection sampling: generate several attempts, keep the solved ones.
- The surviving trajectories become training rows. The student is trained on what it should say and do next, not on the catalog text the tools already returned.
That last cut matters. You want a policy. You do not want a memory of product pages the agent will see again at inference time.
SeqKD fits when you lack clean human labels, the teacher is closed or API-only, and you want a portable first step. Methods that use the teacher’s token probabilities, or that train the student on its own rollouts, can go further. They also ask for access or infrastructure SeqKD does not.
Where the score actually moved
The walkthrough’s slice is 750 synthesized problems, 250 each for product, shop, and voucher intents, split once into 426 training problems and 324 held-out problems. The held-out set is 103 product, 105 shop, and 116 voucher. The published benchmark is broader than that slice. It also covers knowledge questions and multi-product, same-seller tasks, and its sandbox exposes six tools, including product search, product details, discount math, web knowledge, recommend, and terminate. If a reproduction drops tools, its scores are not comparable to the full benchmark. Read what follows as one AWS run on a shopping subset.
The teacher is GLM-5.3 on Amazon Bedrock, reasoning on, temperature 1.0, working over a catalog of about 2.7 million products. The walkthrough says it was sampled twice per training problem and reports 850 trajectories. A clean double of 426 would be 852, and the writeup does not explain the two-run gap. It also does not say how many of the 850 survived the filter, only that solved runs expanded into about 2, 600 training rows and 130 validation rows. Validation holds out whole problems, about five percent, so the student is not graded on later steps of a task it already saw.
The student is Qwen3.5-4B, tuned with LoRA (a small adapter on a shared base, not a rewrite of every weight) through serverless model customization on Amazon SageMaker AI. The authors describe the teacher as a mixture-of-experts model with roughly 40 billion active parameters. Active parameters are the ones used on a given step, not the teacher’s full size, and that figure is their description. It explains why the gap looked bridgeable. It is not a license to call a 4B model a tenth of a frontier model, or a tenth of the bill.
Quality is Absolute Success Rate: the share of held-out problems solved, averaged across the three intents, with a missing trajectory counted as a failure. In this setup a problem counts as solved only when the recommendation meets the scorer’s constraints. On product tasks in the published benchmark, that is thresholded similarity (title, price range, and feature overlap), not exact identity. Shop success cares that the count matches and the products come from one seller. The paper also tracks CAR, a graded relevance score. This run does not report it. These figures are from a single LoRA run. The authors say to expect a few points of run-to-run variation.
- Untuned Qwen3.5-4B: product 0.456, shop 0.314, voucher 0.121, average 0.297.
- Distilled student: product 0.563, shop 0.638, voucher 0.422, average 0.541.
- GLM-5.3 teacher, sampled at temperature 1.0: product 0.553, shop 0.619, voucher 0.362, average 0.511.
- Same teacher, greedy at temperature 0: product 0.544, shop 0.638, voucher 0.439, average 0.540.
- An unnamed medium-sized model, labeled frontier in the walkthrough: product 0.583, shop 0.667, voucher 0.388, average 0.546.
The gap from 0.297 to 0.541 is 0.244, which the walkthrough rounds to about 24 points. The lift is uneven, and that unevenness is the point. Product was already the base model’s best intent and moved the least, from 0.456 to 0.563. Shop and voucher, where the untuned 4B model was weakest, moved the most.
Read the teacher comparison in the same spirit. The student at 0.541 ties the greedy teacher at 0.540, and it beats the temperature-1.0 teacher at 0.511, the decoding mode that actually wrote the training traces. Matching a mode you did not train on, while beating the mode you did, is interesting. It is also the sort of gap that a few points of variation can swallow. Against the unnamed medium model the student is slightly behind on product and shop, ahead on voucher, and nearly tied on average. The walkthrough never names that model, so the comparison is weak, and “frontier” is doing unpaid work. The authors’ reading is that the 4B student recovered essentially all of its teacher’s task quality. Fair as a description of this one table. Not a replicated equivalence, and not a ShoppingBench leaderboard result.
Keep the thinking between the clicks
A natural instinct is to strip the traces and train only on tool calls. The walkthrough tried that. Average success fell from 0.541 to 0.415, a gap of 0.126, reported as about 13 points.
The reasoning between the actions isn’t narration to discard, it is part of the behavior that solves the task.
The “why this filter, why this shop, why stop” is load-bearing. For AI agents, that monologue belongs in the policy. It is not a UX extra you can peel off.
Specialization has a cost of its own. On IFEval, a general instruction-following check, prompt-strict score fell from 0.819 on the base model to 0.741 on the student, and instruction-strict fell from 0.872 to 0.813. The walkthrough describes that as roughly 6 to 8 points, and reads it as an expected tax for a narrower job, not as severe overfitting. One regression check. Reason enough to keep a general eval in the loop before you point the model at work outside the shopping job.
When imitation runs out of road
Distillation transfers a distribution the student can already represent, not raw capability it lacks.
If the task sits above what a 4B model can do, the failure looks familiar: traces get longer and accuracy stays flat. A longer meeting that still does not decide. The shopping subset was workable here because the untuned 4B model was already solving some of it. A student that copies format without gaining the skill has hit a capacity wall, and more of the same trajectories will not punch through it.
SeqKD also tends to plateau on the hardest problems. The next neighborhood is on-policy learning, where the student trains on its own rollouts and a verifier scores them. SageMaker’s recipe collection already documents related methods, including preference optimization, verifiable rewards, and multi-turn reinforcement learning with tool use. Availability is not a result. Those recipes were not run on this benchmark here, and they were not shown to beat this student.
Questions worth asking before you distill
-
Why do AI agents get expensive faster than a normal model call?
Because one completed task is a loop. Search, read, refine, verify, and decide can each be a separate billed call, so cost scales with steps, not just with customers. This demo does not price that loop. You can describe a shift from per-token API charges to the hours a hosted endpoint is up. You cannot quote a savings number, a latency, or a break-even volume from it.
-
When is sequence-level distillation the right first move?
When you do not have high-quality labels, the teacher is API-only, and you want a supervised-tuning step that does not require weights, token probabilities, or a reinforcement-learning stack. Judge it on held-out tasks. In the reported run the student’s average of 0.541 sat next to a greedy teacher at 0.540 and an unnamed medium model at 0.546, while beating the sampled teacher that wrote the traces, at 0.511. Encouraging. Not a replicated match.
-
Should you throw away the reasoning and keep only the tool calls?
Not on this evidence. Training on tool calls alone dropped average success from 0.541 to 0.415. Budget for a specialization check too: instruction-following on IFEval slipped by roughly 6 to 8 points, a tax for the narrower job.
-
What if the small model still fails the hard cases?
That is the capacity wall, or the SeqKD plateau. A larger student, or a later on-policy stage with a verifier, is the honest next step. More imitation of traces the student cannot represent will mostly make the traces longer.
Hire the experiment, not the headline
For AI for business, and especially AI for sales, the useful lesson is narrower than “small models catch frontier models.” A shopping or sales agent is a multi-step worker. If a frontier teacher can already solve the job, and a smaller open model is in range of that job, you can harvest solved trajectories and fine-tune the student on the behavior, reasoning included.
Judge the result the way you would judge a new sales hire. Held-out tasks. A scorer that knows the gold answer. A check that general instruction-following did not collapse. A cost model built from your traffic, not from a vendor’s silence. A strong reason to run the experiment on your catalog. A weak reason to retire the frontier model on faith.