Amazon Bedrock Prompt Caching: Reduce Costs and Time‑to‑First‑Token for Long‑Context AI Workloads

Optimizing cost and latency with Amazon Bedrock prompt caching

TL;DR: Amazon Bedrock’s prompt caching snapshots a model’s partially processed prefix (via a cachePoint in the Converse API), so repeated requests that share that prefix avoid reprocessing. That can dramatically cut billed input tokens and reduce Time-to-First-Token (TTFT) for long-context workloads, but the savings and performance depend on model thresholds, TTLs, region routing, and how you instrument the cache.

Roadmap: how prompt caching works, the production patterns that benefit, billing and a numeric worked example, operational caveats (TTL, token thresholds, eviction, tenant isolation), a TTFT test checklist, and clear next steps you can act on today.

How prompt caching works, the short version

  • Place a cachePoint marker after the static prefix of your request.
  • First request: the model processes the full prefix and writes a checkpoint (cache write).
  • Subsequent requests with the same prefix: the model reads the checkpoint and starts generation from the cached internal state (cache read).

The Converse API uses a model-agnostic cachePoint syntax across supported families, and cache entries are scoped to an AWS account and Region. The AWS walkthrough and sample repo show concrete examples and metrics (see the sample repository for runnable notebooks).

“Note: Boto3 1.43.0 or later is required for the ttl parameter in cachePoint used in Scenario 4 (Mixed TTL).”

Why this matters for long-context applications

If your app resubmits the same large document every turn, such as legal contracts, product manuals, or tool schemas, you’re re-billing the model for identical tokens. Prompt caching stops that repeated billing for the prefix, and you pay a smaller read cost when the cached prefix is reused within its TTL.

Practical patterns that benefit most

  • Large static documents: cache manuals, contracts, or knowledge dumps and only send the question each turn.
  • System prompts: long policy or behavior prompts that don’t change often.
  • Tool definitions: large toolConfig or tools arrays that would otherwise be reprocessed each turn.
  • Mixed TTLs: assign longer TTLs to stable content and shorter TTLs to session context (longer TTL checkpoints must appear before shorter ones in the same request).
  • Tenant isolation: prefix cache keys per tenant so different customers don’t share checkpoints.
  • Framework integration: LangChain provides helpers (ChatBedrockConverse.create_cache_point()) to generate cachePoints from templates and surfaces usage metadata.

Billing and the exact token categories to watch

The API reports usage fields you should treat separately when calculating cost: inputTokens and outputTokens plus two cache-specific fields, cacheWriteInputTokens and cacheReadInputTokens. Billing is computed from these categories using model-specific rates on the Amazon Bedrock pricing page; the walkthrough and sample outputs show how those counts appear in responses.

Per the Bedrock prompt caching walkthrough and pricing notes, cache token categories behave like this:

  • cacheWriteInputTokens, tokens written to cache during the initial request, reported as billed at a higher rate than standard input tokens. The walkthrough reports about 25% higher for standard cache writes, and documents writes with a 1-hour TTL billed at 100% higher, that is roughly 2×.
  • cacheReadInputTokens, tokens read from cache on subsequent requests, reported as billed at a substantially lower rate, the walkthrough reports about 90% lower than standard input tokens.

Because pricing and model support change, confirm absolute $/token rates on the Amazon Bedrock pricing page before you run ROI math.

A worked (numeric) example that makes the savings obvious

Use these variables:

  • tokens = size of the cached prefix (e.g., 10, 000)
  • n = number of queries that reuse the prefix (e.g., 10)
  • C = standard input token cost per token (use the Bedrock pricing page for $/token)
  • m_write = cacheWrite multiplier (walkthrough reports ~1.25 × input cost for standard writes)
  • m_read = cacheRead multiplier (walkthrough reports ~0.10 × input cost for reads)

Cost without caching: cost_no_cache = n × tokens × C

Cost with caching: cost_cache = tokens × C × m_write + (n − 1) × tokens × C × m_read

Plugging the walkthrough multipliers (tokens = 10, 000; n = 10):

cost_no_cache = 10 × 10, 000 × C = 100, 000 × C

cost_cache = 10, 000 × C × 1.25 + 9 × 10, 000 × C × 0.10 = 12, 500 × C + 9, 000 × C = 21, 500 × C

That arithmetic gives a reduction of (100, 000 − 21, 500) / 100, 000 ≈ 78.5% on input-token cost in the TTL window. The AWS walkthrough presents a similar worked example and cites “approximately 75%” savings for a near-identical scenario; your exact dollar savings depend on Bedrock’s current $/token rates and model selection, so run this formula with your C to get accurate dollar results.

Token thresholds, TTLs, and scope, things that break naïve caching

  • Model minimum token thresholds: some models require a minimum number of tokens in a checkpoint before a cache write is created. The walkthrough cites examples such as Anthropic Claude Sonnet 4.5/4.6: 1, 024 tokens and Opus models: 4, 096 tokens. These are model-dependent, confirm current values in the Bedrock docs.
  • TTL defaults and limits: default TTL in the walkthrough is 5 minutes, and select models support up to 1 hour. Mixed TTLs are supported in a single request, but long-TTL checkpoints must appear before short-TTL ones because cachePoints are applied sequentially and later checkpoints can interact with or overwrite earlier cached state.
  • Cache scoping: cache entries are scoped to an AWS account and Region, so identical content sent to different Regions or accounts will produce separate cache writes.

Sample API usage metadata (example outputs)

First request (cache write expected):

{ “inputTokens”: 28, “outputTokens”: 253, “cacheWriteInputTokens”: 1898, “cacheReadInputTokens”: 0 }

Subsequent request (cache read expected):

{ “inputTokens”: 28, “outputTokens”: 294, “cacheWriteInputTokens”: 0, “cacheReadInputTokens”: 1898 }

Those JSON snippets are sample outputs from the AWS walkthrough and illustrate how Bedrock reports cacheWrite/cacheRead token counts alongside standard usage fields. Treat them as examples, reproduce them in your region and model to see the exact numbers.

Latency (TTFT), what to expect and how to benchmark

Cache hits usually reduce Time-to-First-Token, but the benefit depends on prefix size and environment. The walkthrough notes small prefixes, around 2k, 5k tokens, might not show statistically significant TTFT gains across a few runs, while very large prefixes, greater than 10k tokens, show clearer reductions. Latency is empirical, measure it in your target region and concurrency profile.

Quick TTFT benchmark checklist:

  • Run 50-100 repeated requests for several prefix sizes, for example 2k, 5k, 10k, 20k tokens.
  • Include warm-up runs to remove cold-start effects.
  • Test with target concurrency to capture real contention effects.
  • Record p50/p95 TTFT, cache hit rate, and request timestamps; compare cache write vs read medians.

Production best practices and operational caveats

  • Choose TTLs by content lifecycle: domain and tool definitions, 1 hour; session context, 5 minutes; dynamic real-time data, don’t cache.
  • Instrument everything: emit cacheWriteInputTokens, cacheReadInputTokens, cache hit rate, and TTFT into CloudWatch or your observability stack and set alarms for sudden hit-rate drops.
  • Cache eviction & consistency: confirm cache capacity, eviction policy, and any per-account or Region limits in Bedrock documentation. Plan for pre-warming high-value prefixes after deployments or region failovers to avoid a flood of cold writes.
  • Cross-Region routing: if requests for the same prefix land in multiple Regions, you’ll generate more cache writes. Enforce region affinity or request stickiness for heavy reuse workloads.
  • Content versioning and cache busting: include a content version or hash near the cachePoint so you can force cache writes when documents or tool schemas change.
  • Security and compliance: cached checkpoints hold processed content. Confirm retention, encryption, and IAM controls with security and legal teams. Treat tenant hashing as a cache-key technique, not a substitute for access control or PII governance.

Tenant isolation, do it right

The walkthrough recommends prepending a SHA-256 hash of tenant_id to your cached prefix so tenants get distinct cache keys without separate AWS accounts. A SHA-256 hex string is 64 characters and the samples say this typically adds roughly 16 tokens, however token counts depend on the model tokenizer, so measure changes in tokenization empirically.

Alternatives and practical tips:

  • Use a truncated deterministic hash, first 8-12 hex chars, to reduce token inflation, and test for collision risk against your tenant population.
  • Keep a server-side mapping of tenant_id → short synthetic prefix, for example “T-1234”, to avoid adding long hex strings into the prompt.
  • Always measure token deltas with the tokenizer used by your target model; small prefix changes can alter tokenization in unexpected ways.

Implementation notes, what to check before you roll

  • LangChain: ChatBedrockConverse.create_cache_point() is available in LangChain’s Bedrock helpers to insert cachePoint blocks; responses include usage metadata fields for cache creation and reads so you can instrument hit and miss behavior.
  • SDKs: sample code and scenarios reference Boto3 1.43.0 or later for the ttl parameter in cachePoint, verify your environment matches the sample requirements.
  • Repository with runnable examples and notebooks: https://github.com/aws-samples/amazon-bedrock-samples/tree/main/introduction-to-bedrock/prompt-caching, run these in a dev Region to reproduce the walkthrough metrics.

Key questions, quick, honest answers (and what to do next)