NVIDIA Nemotron 3.5 Lightning now available in Amazon SageMaker JumpStart
If your business runs always‑on agents that repeatedly perform the same structured tasks, such as parsing forms, validating fields, and issuing API calls, you face a clear trade‑off: call a large frontier model and pay in latency and cost, or use a smaller model and risk missing edge cases. NVIDIA’s Nemotron 3.5 Lightning aims for the middle ground: an open, distilled foundation model built to handle the high‑volume, agentic steps inside a system‑of‑models architecture. The model was released on 08/11/2026 and is deployable through Amazon SageMaker JumpStart (see the NVIDIA developer site, the Hugging Face model card, the NeMo Gym repo, and the AWS SageMaker JumpStart announcement for primary references).
What Lightning is (and what it isn’t)
NVIDIA describes it as:
“Nemotron 3.5 Lightning is a publicly available foundation model distilled from NVIDIA’s frontier Nemotron 3 Ultra and developed with the Nemotron Coalition.”
Per NVIDIA and the Hugging Face model card, Lightning is a distilled, hybrid Mixture‑of‑Experts (MoE) model. That means the model contains many experts at scale but activates only a subset for each forward pass, trading dense compute for conditional capacity.
- Model: NVIDIA Nemotron 3.5 Lightning (distilled from Nemotron 3 Ultra)
- Architecture: Hybrid Mixture‑of‑Experts (MoE)
- Parameters: 30B total / ~3B active per forward pass (per the Hugging Face model card)
- Context window: up to 1, 000, 000 tokens (vendor claim)
- Decoding & speed: Multi‑Token Prediction (MTP) and speculative decoding (called “DFlash” by NVIDIA) to reduce per‑token latency
- Variants: BF16 and NVFP4
- License: OpenMDW‑1.1 (customers may post‑train and retain weights)
Plain language: Lightning is built to be the fast, repeatable worker inside an agent pipeline. It handles structured outputs (JSON, tool calls, schema‑validated responses) and high QPS steps cheaply. It is not meant to replace a frontier model for deep planning. NVIDIA recommends routing high‑volume steps to Lightning and keeping larger models for complex planning.
Where Lightning fits in a real system
Think in tiers. Use Lightning as the production workhorse for predictable, frequent agent actions, like validating input, normalizing values, calling an API, and formatting a JSON response. Route only the costly planning passes, policy, long‑horizon reasoning, and user‑facing synthesis, to a larger dense model. NVIDIA’s NeMo Switchyard concept mirrors this routing approach: use the right model for the right step to save money and latency without sacrificing final quality.
This pattern lowers per‑task cost and tail latency, but it requires more engineering: routing logic, model selection policies, fallback rules, and governance for post‑training and weights management.
Benchmarks and vendor claims, read the fine print
NVIDIA and partner materials headline two figures:
- “Up to 4x higher” throughput for high‑volume agentic workloads (vendor‑reported).
- “Up to 30% faster” task completion (vendor‑reported).
Benchmarks published on the Hugging Face model pages and documented in NVIDIA’s NeMo Gym show BF16 vs NVFP4 comparisons on multiple suites. Examples (reported values):
- MMLU Pro: BF16 81.94 / NVFP4 81.62
- GPQA Diamond: BF16 75.44 / NVFP4 75.57
- SWE‑bench Verified: BF16 51.56 / NVFP4 52.80
- PinchBench: BF16 85.37 / NVFP4 83.43
- IFBench: BF16 71.88 / NVFP4 72.88
- AA‑LCR: BF16 52.00 / NVFP4 49.19
Those numbers come from vendor‑published evaluation harnesses and the NeMo Gym reproducibility recipes. They are useful signals, but they were measured on specific hardware and inference stacks. Validate them with your own traces and hardware before planning migrations.
How to get and deploy Lightning on SageMaker JumpStart
AWS added Lightning to SageMaker JumpStart to simplify deployment. The high‑level flow in SageMaker Studio is: search JumpStart → open the Lightning model card → Deploy → choose instance → wait for the endpoint to reach InService. AWS contributors to the announcement include Venu Kanamatareddy, Evan Kravitz, Vivek Gangasani, and Naidile Murali.
JumpStart model identifiers you can use when automating deployments:
- NVFP4: huggingface-reasoning-nemotron-3-5-lightning-30b-a3b-nvfp4
- BF16: huggingface-reasoning-nemotron-3-5-lightning-30b-a3b-bf16
Two important AWS notes (verbatim):
“The SageMaker JumpStart model card for this launch does not expose JumpStart customization.”
“Deploying this model creates a SageMaker AI endpoint that incurs charges while running.”
Practically, make sure you have an AWS account, the correct SageMaker permissions, and enough GPU service quota for your chosen instance types. AWS examples mention instances such as ml.g6e.12xlarge, ml.p4d.24xlarge, and ml.p5.48xlarge. Those are examples and region availability will vary.
Practical deployment considerations for engineering and infra teams
- MoE serving complexity: MoE improves throughput by routing tokens to experts, but it adds routing overhead, expert placement decisions, and memory layout challenges across GPUs. Instrument per‑expert utilization and tail latency during trials.
- Variant tradeoffs: BF16 tends to give higher fidelity. NVFP4 reduces memory use and can be faster on quantization‑friendly stacks. Benchmark both variants on your inference stack (vLLM, Triton, NeMo) and hardware.
- Single‑GPU runs: Quick‑start recipes show single‑H100 runs for BF16. Support and performance on other single GPUs depend on quantization and the inference toolchain, so validate with your hardware.
- Cold start and warm pools: Large weight files increase cold‑start time. Use warm endpoint pools or pre‑warmed containers to meet low‑latency SLAs.
- Quantization pitfalls: Aggressive quantization can break strict structured outputs, producing broken JSON or schema failures. Add schema validation gates and fallback rules to a denser model if needed.
- Costs: SageMaker endpoints bill while active. Model and instance choices make the biggest difference. Separate serverless token pricing (for example vendors like DeepInfra publish token‑based day‑zero pricing) from dedicated endpoint costs, then convert both to $/task for apples‑to‑apples comparisons.
- Governance & licensing: Lightning is released under OpenMDW‑1.1 and NVIDIA markets it as post‑trainable. That gives control, but it also means you must enforce data governance, PII handling, and audit trails for any fine‑tuning.
What to validate before you flip the switch
Run a short, repeatable experiment that mirrors your agentic workload. Don’t chase vendor headlines. Measure these concrete metrics on your target instance types and on both BF16 and NVFP4 variants:
- P50/P95 per‑step latency (ms)
- End‑to‑end task completion time and variance
- Cost per completed task (USD/task) at expected QPS
- Structured‑output fidelity: schema‑validated success rate (%)
- Failover rate: how often the step needs escalation to a frontier model and the added cost
Suggested experiment outline:
- Run 1, 000 representative traces (your real agent traces) against Lightning on the target instance and against your current baseline.
- Measure P95 latency, schema pass rate, tokens per request, and compute $/task for each configuration.
- Reproduce vendor NeMo Gym recipes to sanity‑check benchmark alignment, then run your traces to measure real production delta.
Decision checklist for leaders (5 quick items)
- Do we have repeatable, high‑volume agent steps where throughput matters? If not, Lightning may not move the needle.
- Can we afford the operational complexity, including routing, expert monitoring, and warm pools, required for MoE serving?
- Have we budgeted endpoint or serverless token costs and computed $/task at expected QPS?
- Do we have governance controls for post‑training, including data lineage, PII handling, and audit trails?
- Can we run the 1, 000‑trace experiment within two weeks to validate vendor claims on our workload?
Final practical advice
Lightning is worth evaluating if your agents do predictable, repeated, structured work at scale, such as customer triage, fraud checks, telemetry parsing, or routine security investigations. It can cut latency and cost for those steps while keeping heavier models for complex reasoning. Vendor throughput and task‑completion claims were measured under specific harnesses; treat them as a starting point, not a guarantee. Run controlled experiments on your traces, compare BF16 vs NVFP4, and build governance around post‑training and endpoint lifecycle so you capture benefits safely.
Key takeaways, questions a curious reader will ask
- What problem is Lightning designed to solve?
Lightning targets high‑volume, repetitive agentic steps, those pieces of an agent’s workflow where throughput and lower latency matter more than maximal general reasoning capability (per NVIDIA and the Hugging Face model card).
- How does Lightning reduce compute cost per token?
Its hybrid Mixture‑of‑Experts architecture keeps 30B total parameters but activates roughly 3B per forward pass, lowering per‑token compute while preserving large‑model capacity (vendor documentation).
- Are NVIDIA’s “up to 4x throughput” and “up to 30% faster” claims reliable for planning?
Those are vendor‑reported numbers measured under NVIDIA/partner harnesses; they provide direction but must be validated on your hardware and traces using NeMo Gym recipes or equivalent reproducibility steps.
- Can I customize the model from SageMaker JumpStart?
Per AWS: “The SageMaker JumpStart model card for this launch does not expose JumpStart customization.” Customers who need post‑training should use NVIDIA NeMo workflows and retain weights under the OpenMDW‑1.1 license.
- How should I pick between BF16 and NVFP4?
BF16 offers higher numerical fidelity; NVFP4 reduces memory and can be faster on quantization‑friendly toolchains. Benchmark both variants on your intended instance and workload before choosing.
Further reading
One concise reference that complements the deployment and cost notes above: