Cost‑efficient text‑to‑SQL: LoRA adapters + Nova Micro on Amazon Bedrock

TL;DR — Base LLMs can generate SQL, but they struggle with custom dialects, proprietary schemas, and strict correctness rules. The cost‑efficient pattern is to use parameter‑efficient fine‑tuning (PEFT) with LoRA (Low‑Rank Adaptation) adapters to capture domain behavior, then serve those adapters on-demand with Amazon Bedrock (Nova Micro). This gives production‑grade behavior at interactive latency while turning model hosting from a fixed infrastructure bill into pay‑per‑use. Example: 22,000 queries/month at ~800 input + 60 output tokens averages about $0.80/month in inference cost (see explicit math below).

Why base models aren’t enough for production text‑to‑SQL

Large foundation models are great at common SQL patterns, but even small differences in SQL dialects, vendor functions, or database schemas can create brittle behavior. The result is incorrect queries, failing dashboards, or dangerous generated DDL/DML. Fine‑tuning is usually necessary to reach production‑grade accuracy — but hosting a fully custom model 24/7 inflates operational costs.

Think of the foundation model as a reliable car and LoRA as a plugin GPS tuned to your city: small, focused, and cheap to install. PEFT (parameter‑efficient fine‑tuning) modifies only a small part of the model via adapters so you keep the large pre‑trained model frozen and add domain knowledge cheaply.

The pattern: LoRA adapters + Nova Micro on Bedrock

• LoRA / PEFT: fine‑tune a small adapter that captures domain rules and schema signals without updating the full model.
• Amazon Nova Micro: a compact foundation model offering low latency and low per‑token pricing for interactive text‑to‑SQL.
• Amazon Bedrock on‑demand inference: serverless, pay‑per‑token inference so you only pay when users query the system.

Combine the three and you get: small, cheap adapters that are applied at inference time against a managed, serverless model. The tradeoff is a modest latency and throughput hit from adapter application versus the operational savings of not hosting a persistent custom model.

“Fine‑tuning is necessary to reach production‑grade accuracy on custom SQL dialects, but persistent hosting increases continuous costs.”

Two practical implementation paths

• Amazon Bedrock model customization — Managed customization and serverless deployment. Best when your priority is speed of iteration, minimal infra, and pay‑per‑use economics.
• Amazon SageMaker AI training jobs — Full control over recipes, hyperparameters, and infra (example instance: ml.g5.48xlarge). Best when you need reproducibility, custom training logic, or VPC isolation for compliance.

Pick Bedrock for most pilots and light to medium production loads. Choose SageMaker when you need precise control over training runs, data residency, or specialized hardware tuning.

Example dataset, hyperparameters, and training costs

The working demo used the sql-create-context dataset (a curated mix of WikiSQL and Spider): roughly 78,000 NL→SQL examples spanning many schemas. Training examples were formatted for Bedrock’s conversation JSONL schema.

Sample hyperparameters used for Nova Micro + LoRA (Bedrock path): epochs = 5, batch size = 1, learning rate = 1e‑5, warmup steps = 10. Typical small runs completed in ~2–3 hours on Bedrock customization jobs for the dataset slice used.

SageMaker example: ml.g5.48xlarge, ~20,000 lines, 2 epochs, batch size 64, ~4 hours total. Compute cost for that run: 16.288 USD/hour × 4 hours ≈ $65.15.

Bedrock customization pricing example (illustrative): if training costs are $0.001 per 1,000 tokens, then 2,000 examples × 5 epochs × ~800 tokens ≈ 8,000,000 tokens → 8,000 × $0.001 = $8.

Serving costs — explicit math for the $0.80/month example

Reported Nova Micro token prices used here (verify current pricing before running the numbers): input tokens = $0.000035 per 1,000 tokens; output tokens = $0.00014 per 1,000 tokens.

• Queries per month: 22,000
• Input tokens per query: 800 → monthly input tokens = 22,000 × 800 = 17,600,000
• Input cost = 17,600,000 / 1,000 × $0.000035 = 17,600 × $0.000035 = $0.616
• Output tokens per query: 60 → monthly output tokens = 22,000 × 60 = 1,320,000
• Output cost = 1,320,000 / 1,000 × $0.00014 = 1,320 × $0.00014 = $0.1848
• Total monthly inference = $0.616 + $0.1848 ≈ $0.80

Per‑query cost with these assumptions: (800 input + 60 output) tokens cost ≈ $0.0000364 per query. Scaling to larger volumes:

• 100k queries → ≈ $3.64/month
• 500k queries → ≈ $18.20/month
• 1M queries → ≈ $36.40/month

These numbers underscore that, with the reported per‑token pricing and compact token footprints, on‑demand inference can remain economical even at high volumes. Large‑scale migrations or extremely tight latency requirements may still justify reserved capacity or persistent hosting — run your own break‑even model with your token averages and target SLAs.

Latency and throughput tradeoffs

Applying LoRA adapters at inference increases both cold starts and token‑generation latency, but remains interactive for many UIs. Measured examples:

• Cold start Time to First Token (TTFT): ~639 ms (≈34% slower than base).
• Normal TTFT (steady state): ~380 ms (≈7% slower).
• End‑to‑end latency: ~477 ms average.
• Token generation (output tokens/sec): ≈183 tps (≈27% slower than base).

Mitigations for cold starts include warm pools, low‑frequency keep‑alives, or client‑side caching of recent queries. If your product requires sub‑200ms TTFT, test the pattern early and consider co‑located inference or reserved capacity.

“Applying LoRA adapters at inference adds latency but remains suitable for interactive applications while enabling meaningful cost savings.”

Evaluation: metrics and LLM‑as‑a‑judge

Build an evaluation pipeline with multiple lenses:

• Exact match: string equality with ground truth (useful but brittle).
• Execution accuracy: run generated SQL in a sandbox and compare result sets to ground truth (most important practically).
• Syntactic validity: does the query parse and execute without errors?
• Schema validity: uses only known tables/columns and respects schema constraints.
• Human review sampling: random audits of production outputs for safety and edge cases.

An automated LLM‑as‑a‑judge can speed iteration: generate SQL, then prompt a high‑quality judge LLM (e.g., Claude or equivalent) to score correctness against ground truth. Calibrate the judge against a human‑labeled validation set to set pass/fail thresholds. Still, keep human audits in the loop for safety and tricky edge cases.

Operational checklist and safety controls

• Data & compliance: validate Bedrock/SageMaker data flows, encryption, and residency requirements. Use VPC isolation on SageMaker if needed.
• SQL safety: run generated queries in a read‑only sandbox or use a query sanitizer/allowlist. Enforce least‑privilege DB credentials and deny DDL/DML where possible.
• Monitoring: log generated SQL, execution results, latency p95/p99, error rates, and human review flags.
• CI for adapters: version adapters, test against a stable validation set, and automate rollout with canary percentages.
• Rollback & governance: approval gates for adapter updates and a fast rollback path to the base model if errors spike.

How to run a 2‑week pilot (practical)

1. Pick 2–3 target dashboards or user personas and identify the most common 50–200 query intents each.
2. Prepare 1,000 labeled NL→SQL examples drawn from those schemas (include edge cases and forbidden operations).
3. Fine‑tune a LoRA adapter on Nova Micro via Bedrock customization (or SageMaker if you need VPC/compliance), using a few epochs and low learning rate.
4. A/B test the fine‑tuned adapter vs base model on execution accuracy and latency; run generated SQL in a sandboxed DB with read‑only creds.
5. Measure: execution accuracy, TTFT p50/p95, user satisfaction, and rate of human escalations. Iterate and expand if metrics meet thresholds.

Decision checklist — when to pick Bedrock vs SageMaker

• Choose Bedrock if you want rapid iteration, serverless on‑demand inference, and low ops overhead.
• Choose SageMaker if you need full control over training, VPC isolation, or very specific reproducibility and hardware needs.
• Consider persistent hosting when you require extremely low latency SLAs, predictable high throughput with custom model footprints, or when cloud provider pricing/contracting makes reserved capacity cheaper. Always run a break‑even analysis with your real token averages.

Common questions (short answers)

How necessary is fine‑tuning for text‑to‑SQL?

Fine‑tuning with PEFT like LoRA is usually necessary to reach production‑grade accuracy on custom dialects and proprietary schemas.

Is on‑demand Bedrock inference genuinely cost‑efficient?

Yes for many pilots and moderate‑scale workloads. With the token prices used here, even 1M queries/month can be only a few dozen dollars. Validate with your token averages and up‑to‑date pricing.

How do we prevent unsafe SQL generation?

Use training data that discourages destructive operations, enforce least‑privilege DB roles, run queries in a read‑only sandbox, and add post‑generation sanitizers/allowlists for banned operations.

Next steps by role

• CTO: sponsor a 2‑week pilot and allocate budget for a small SageMaker or Bedrock customization run.
• ML Engineer: reproduce the Bedrock LoRA notebook, fix random seeds, and log execution accuracy vs base model.
• Product Manager: define success criteria (target execution accuracy, acceptable latency p95, human review rate) and pick initial schemas for the pilot.

Resources & reproducibility notes

Sample notebooks and training recipes for both Bedrock customization and SageMaker training are available in AWS example repositories; use them as a starting point and verify the framework versions and random seeds. Key reproducibility points: fix random seeds, log hardware specs (instance type and run duration), and snapshot adapter artifacts in storage (S3) with metadata.

One final practical reminder: token prices, instance costs, and feature availability change. Treat the numbers above as illustrative and re‑run the cost math with your specific token averages, latency needs, and compliance constraints before making a long‑term decision.

Key takeaways

• LoRA adapters + Nova Micro on Bedrock deliver a pragmatic balance: production accuracy with pay‑per‑use economics and modest latency tradeoffs.
• Bedrock customization is fastest for pilots; SageMaker is best for fine‑grained control and strict compliance needs.
• Build robust evaluation and safety pipelines: execution accuracy, LLM‑as‑a‑judge with calibration, human audits, sandboxed execution, and least‑privilege DB access.

“Combining LoRA with Bedrock on‑demand inference lets organizations keep custom behavior while paying by use instead of for idle infrastructure.”