Choosing the right OpenAI model on Amazon Bedrock: prioritize cost-per-success over price-per-token

Beyond the price per token: Choosing the right OpenAI model on Amazon Bedrock for your workload

On a 60-question slice of the AIME math benchmark, a lower per‑token model ended up costing roughly 7× more per correct answer than a newer model deployed on Amazon Bedrock, a compact demonstration of the main point: dollars per million tokens are an input to procurement, not the decision itself.

Engineers from Amazon Bedrock and OpenAI published an open, reproducible benchmark harness (https://github.com/openai-on-aws/benchmarks-openai) that runs identical Responses API calls against different backends and reports outcome‑normalized costs. They report cost per correct answer, cost per passing agent trajectory, and cost per passing professional deliverable. The authors are Nick McCarthy, Sudeesh Sasidharan, Sharadha Kandasubramanian, and Saurabh Trikande.

Executive summary

Headline finding: When you normalize for successful outcomes, model rankings can flip relative to nominal token price. In several snapshots the Bedrock-deployed gpt‑5.6 family, especially gpt‑5.6‑luna after a July 30, 2026 repricing, produced a lower observed cost per successful outcome than cheaper-per-token baselines (gpt‑5.4‑mini / gpt‑5.4‑nano).
Important caveats: Bedrock runs in these tests used “reasoning disabled” while the OpenAI API baselines ran at defaults. GDPval outputs were capped and truncated for some models. Sample sizes are modest (e.g., AIME N=60, DeepSearchQA N=50, GDPval N=48) so expect statistical noise.
Practical action: Run the provided harness against your workflows with your pricing snapshot and region, measure tokens, turns, retries and human rework time, and compute cost per successful outcome before you commit at scale.

How they compute “cost per successful outcome”

The authors convert logged token usage into dollars using a pricing snapshot, then divide total spend by the number of successful results under a predefined pass threshold. The generic formula is:

cost_per_success = (mean_input_tokens × input_price_per_token + mean_output_tokens × output_price_per_token) × average_calls_per_attempt ÷ pass_rate + human_rework_cost (if included).

Conversion from the per‑1M token rates used in the report to a per‑token price is straightforward. For example, using the Bedrock prices reported in the harness: luna input = $0.22 per 1M tokens → $0.00000022 per token; luna output = $1.32 per 1M tokens → $0.00000132 per token.

Illustrative (not reported) example: if a single call averaged 500 input tokens and 50 output tokens, the model cost per run on luna would be 500×0.00000022 + 50×0.00000132 = $0.000176. If the pass rate for that task were 75%, cost_per_success (excluding human rework) would be $0.000176 / 0.75 ≈ $0.000235. The authors applied the same arithmetic to the actual logged token counts and pass rates from each benchmark to produce their reported cost‑per‑success numbers (see the harness for the exact logged JSONs and calculations).

What they tested and the top-line numbers

The harness compares three gpt‑5.6 variants on Amazon Bedrock (gpt‑5.6‑luna, gpt‑5.6‑terra, gpt‑5.6‑sol) against two OpenAI API baselines (gpt‑5.4‑mini and gpt‑5.4‑nano). Benchmarks and sample sizes:

DeepSearchQA (multi‑step agent web research): N = 50 live questions
AIME (single‑call math): N = 60
MMLU‑Pro (professional MMLU variant): N = 140
GPQA Diamond (graduate‑level science): N = 198
GDPval (rubric‑graded professional deliverables): N = 48 (graded by rubrics authored by professionals averaging 14 years of experience)

Selected results (snapshots and exact figures from the harness)

AIME (single call)

Pass definition: exact numeric correctness as per AIME conventions; N = 60.

gpt‑5.6‑sol solved 75% of the sample vs gpt‑5.4‑mini at 37%.
Using the Bedrock pricing snapshot after July 30, 2026, the recorded cost per correct answer was: gpt‑5.6‑luna $0.0021 vs gpt‑5.4‑mini $0.0139, roughly a 6-7× advantage for luna on cost per success in this slice.

DeepSearchQA (multi‑turn agent)

Pass definition: F1 ≥ 0.7 on the target extraction; N = 50 live questions.

gpt‑5.4‑mini averaged 7.6 turns per question and, in some episodes, accumulated ≈114, 000 input tokens per question because the harness resends full conversation context each turn.
gpt‑5.6‑terra averaged ≈50, 000 input tokens per question and achieved mean F1 = 0.50 vs mini 0.39.
Cost per passing answer (F1 ≥ 0.7): terra $0.31 vs mini $0.40. luna $0.05 vs mini $0.40. gpt‑5.4‑nano had a pass rate of 18% on this sample and an observed cost per passing answer $0.07 vs luna $0.05.
Key takeaway: agentic turn count and token growth dominate costs when full-history resend is used. A model that needs fewer turns can be far cheaper per successful trajectory even if its per‑token rate is higher.

GDPval (professional deliverables)

Pass definition: ≥70% weighted rubric points; N = 48 deliverables.

Observed pass rates: luna 27/48 (56%), mini 20/48 (42%), nano ≈35%.
Recorded cost per passing deliverable: luna $0.010, mini $0.030, nano $0.012.
Note: outputs were capped at 8, 192 tokens. Truncation counts, 6 luna, 9 terra, 7 sol, 0 mini, 1 nano, biased results against higher‑capacity models for long outputs. Rerun with a higher cap to measure the true tradeoff for long deliverables.

Latency and throughput (us-west-2 snapshot, July 2026)

Median time‑to‑first‑token (TTFT): Bedrock averaged 21% lower for luna and 5% lower for terra compared with the OpenAI API baselines.
Throughput for outputs ≥500 tokens: luna throughput on Bedrock averaged 43% higher, terra +4%.
Observed worst‑case TTFT‑to‑median ratios: 2.1-2.5× on Bedrock vs 4.6-6.6× on the OpenAI API. Note, sol runs used us‑east‑1 and showed long, variable TTFT in these snapshots.
These are point‑in‑time, region‑specific snapshots. Teams should measure p95/p99 latency in their target region under expected concurrency.

“The per-token price is one input to a decision, not the decision.”

Why token price alone misleads

Accuracy multiplies costs. A cheaper model that fails often forces retries, additional agent turns, or human post‑editing. Those add token and labor spend that token sticker price doesn’t capture.
Design choices change billed tokens. The harness used client‑managed history with store:false, so it resends full conversation each turn. If you have T turns and each turn resends prior context of length L, billed input ≈ L×T×(T+1)/2, roughly quadratic growth. Example: a 10‑turn dialog with 500 tokens per turn produces ≈27, 500 billed input tokens just from context resends.
Deployment knobs shift both quality and spend. In these runs Bedrock models were executed with reasoning disabled while OpenAI API baselines used default settings. That asymmetry reduces Bedrock spend and capability in this snapshot. Parity runs, both sides with reasoning on or off, are necessary to fully isolate model‑intrinsic differences.
Hard truncation biases results. Output caps trimmed larger models more frequently, which can reduce pass rates even when a model could have succeeded given enough output budget.

Methodological caveats you must consider

Configuration asymmetry: the Bedrock runs used reasoning disabled and the OpenAI API baselines ran at their defaults. This materially affects accuracy and token use. Reproduce runs in parity to understand true deltas.
Region and endpoint differences: sol runs in us‑east‑1 while the latency snapshot compares us‑west‑2. Compare each model in the same region and load conditions.
Small sample sizes: several benchmarks (AIME N=60, DeepSearchQA N=50, GDPval N=48) are modest. Report confidence intervals, bootstrap or binomial, before declaring winners for your workload.
Autorater limitations: GDPval outputs were autorated by gpt‑5.5 with frozen prompts. Autorating is reproducible but can introduce model‑to‑model bias. Validate against human graders on a held‑out subset.
Pricing snapshot dependent: the July 30, 2026 Bedrock repricing (luna −80%, terra −20%) is central to the cost outcomes here. Confirm current regional pricing when you run the harness yourself.

Decision guidance distilled

gpt‑5.6‑luna: benchmark first for high‑volume, low‑complexity tasks and interactive apps. In these snapshots it frequently delivered the best cost‑per‑success after the July 30, 2026 repricing.
gpt‑5.6‑terra: a good candidate for agentic workloads where fewer turns and higher F1 matter. Its higher per‑token price can be offset by lower turn counts.
gpt‑5.6‑sol: choose for the hardest single‑call, accuracy‑gated tasks, for example deep reasoning like AIME. Expect longer and more variable TTFT in these runs.
gpt‑5.4‑mini / gpt‑5.4‑nano: remain attractive for trivial workloads and where you can tightly control context and turns. Their lower pass rates on accuracy‑sensitive tasks can increase total cost per usable result.

Practical engineering levers to lower billed tokens and total cost

Use a state store or retrieval to avoid resending full history each turn.
Summarize or compress conversation history at intervals to cap context growth.
Set conservative max_output_tokens for drafts and expand only when necessary, or paginate long outputs.
Include human post‑edit time in your cost model. Sometimes a higher‑quality model plus light human edit is cheaper than many retries on a low‑cost model.
Benchmark in the target region and at expected concurrency to uncover true latency tail behavior and throughput constraints.

Minimal experiment for procurement and product teams (one‑week run)

Pick 20-100 representative tasks and define a clear pass criterion for each (exact answer, F1 threshold, or rubric score ≥70%).
Run the harness (https://github.com/openai-on-aws/benchmarks-openai) against the candidate models in your target region using your pricing snapshot; log input/output tokens, turns, TTFT, and passes.
Compute cost_per_success = (mean_input_tokens × input_price_per_token + mean_output_tokens × output_price_per_token) / pass_rate, and add any human rework time cost.
Run a parity configuration, reasoning on vs off, to see how the reasoning flag moves the needle on both quality and cost.
Validate autorater labels against human judgments on a 10-20% subset to estimate grader agreement.

Key questions (and honest, short answers)

How should I pick a model for my workload?

Measure cost per outcome that matters to your business, correct answer, passing rubric, or successful agent trajectory. Include token spend, retries, turns, latency impacts, and human rework time, not just dollars per million tokens.
My agents resend full history. How much does that matter?

It can dominate your invoice. With client‑resend history, billed input approximately grows quadratically with turn count. Switching to a state store, retrieval, or periodic summaries can reduce token bills dramatically.
Do price changes flip model rankings?

Yes. The Bedrock repricing on July 30, 2026, luna −80%, terra −20%, materially improved luna’s cost‑per‑success in these snapshots. Always rerun benchmarks with current prices and your region.
Are these pass/fail results definitive?

No. The harness is reproducible, but the runs in these snapshots include asymmetries, small sample sizes, and some truncation. Treat the reported figures as guidance and reproduce them against your workloads.
What if my deliverables get truncated by token caps?

Truncation biased these results against larger models, several luna/terra/sol outputs were capped at 8, 192 tokens. Rerun with a higher output cap to measure the real trade‑off between completeness and cost for long outputs.

Next step

Clone the harness at https://github.com/openai-on-aws/benchmarks-openai, run it against a small but representative slice of your workload in your target region using your current pricing snapshot, and report three numbers back to stakeholders: mean tokens per session, pass rate, and cost per successful outcome (including human rework). Procurement should require cost‑per‑success reporting for any model purchase or long‑term commitment, because token sticker price alone will not protect your unit economics.