IBM Granite 4.2: Apache‑2.0 LLMs with Controllable Chain‑of‑Thought, 128K Context, and Agentic RL

TL;DR

Apache‑2.0 LLM family (Granite 4.2) in 3B / 8B / 30B sizes with explicit chain‑of‑thought controls, long‑context engineering, and agentic RL applied to the 8B and 30B models. Granite Speech 5.0 Turbo CTC is a 470M‑param, high‑throughput CTC ASR family. All architecture, benchmark and throughput numbers below are vendor‑reported by IBM (Hugging Face model cards / IBM write‑up) and reported coverage; independent reproduction is still needed for key performance claims.

What Granite 4.2 is, the quick read

Granite 4.2 is IBM’s reasoning‑first family of dense, decoder‑only transformers: three LLM checkpoints at 3B, 8B and 30B parameters plus two 470M‑parameter Granite Speech 5.0 Turbo CTC models. The release focuses on controllable reasoning (models can emit chain‑of‑thought and expose runtime flags like “enable_thinking”, “low_effort”, and “truncate_history_thinking=True”) and on‑prem deployability under Apache‑2.0 licensing.

Core vendor‑reported architecture and training facts

  • Pretraining: roughly 15 trillion tokens (IBM reported).
  • Model family: dense, decoder‑only transformers (not MoE/hybrid).
  • Architectural highlights: Grouped Query Attention (8 KV heads), RoPE with θ ≈ 10, 000, 000, SwiGLU MLPs, RMSNorm (ε = 1e‑5), untied input/output embeddings, bfloat16 precision (IBM reported).
  • Per‑size specs (IBM reported):
    • 3B, 40 layers, embedding size 2560.
    • 8B, 40 layers, embedding size 4096.
    • 30B, 64 layers, MLP hidden size 32, 768.
  • Context engineering: architecture table lists a 131, 072‑token (128K) context window for inference. IBM also reports a pretraining curriculum that exposed models to sequences up to 512K tokens (the 512K exposure is a training curriculum detail and should not be read as an unconditional inference guarantee for all sizes).
  • Supervised fine‑tuning (SFT): about 7.2 million labeled samples (≈100B tokens). IBM reports roughly ~65B tokens used for trainable SFT updates and that ~31.6% of the SFT examples were agentic in nature (software engineering examples make up approximately 69% of that agentic subset).
  • Post‑training: staged reinforcement learning using a GRPO‑style pipeline with warm‑started checkpoints and a mix of verifiable, targeted, agentic and alignment stages (stage names include RLVR, Boosters, SWE, Terminal, Search, and RLHF, IBM reported).
  • Agentic RL: sandboxed tool interaction (code editing, terminal, web search) was applied to the 8B and 30B models only. The 3B model did not receive the agentic stages (IBM reported).
  • Infrastructure and tooling: IBM ran NeMo‑RL and NeMo‑Gym on an NVIDIA GB200 NVL72 cluster hosted by CoreWeave and used engineering inputs like ~1 trillion synthetic code tokens from the CodeAlchemy pipeline (IBM reported).
  • Serving: Granite 4.2 emits OpenAI‑format function calls over an OpenAI‑compatible endpoint to ease integration with existing agent harnesses (IBM reported).

What GRPO and the staged pipeline mean (short gloss)

GRPO is the asynchronous policy‑optimization variant IBM uses for post‑training RL. The pipeline stages play different roles:

  • SFT (Supervised), labeled examples to shape base behavior.
  • RLVR (Verifiable), verifiable scoring on long sequences (IBM runs these with long max‑seq budgets, e.g., up to 64K tokens).
  • Boosters, targeted RL tweaks for niche skills (small KL penalties for code improvement tasks).
  • SWE / Terminal / Search (Agentic), sandboxed tool interactions to generate multi‑step agentic trajectories (long rollout horizons, larger max‑seq budgets; applied to 8B and 30B only).
  • RLHF (Alignment), human or LLM‑judged preference tuning with reasoning‑length penalties to reduce verbosity.

Vendor‑reported benchmark signals (select highlights)

IBM published a wide set of internal benchmark numbers across code, reasoning and long‑context tests. These are vendor‑reported indicators, useful signals, not independent third‑party validations. Selected IBM figures (3B / 8B / 30B where available):

  • Reasoning: AIME25-78.33 / 86.67 / 89.17; HMMT Feb25-66.67 / 78.33 / 89.17.
  • Code: SWE‑Bench Verified, N/A / 47.67 / 57.00 (30B headline: 57.00).
  • Terminal automation: Terminal‑Bench 2.1, N/A / 20.56 / 29.24 (30B headline: 29.24).
  • Long‑context: RULER 64K, 67.52 / 80.99 / 89.96; RULER 128K, 55.30 / 71.41 / 81.38.

Benchmarks are affected by evaluation settings (temperature, prompting, judge choice). Treat IBM’s numbers as vendor‑run baselines. Reproduce the ones that matter for your use case before committing.

Granite Speech 5.0 Turbo CTC, what’s different

  • Two 470M‑parameter CTC speech checkpoints were released (IBM reported); architecturally these are Conformer encoders trained with Connectionist Temporal Classification (CTC), explicitly without an autoregressive LLM decoder.
  • IBM reports a headline throughput of RTFx ≈ 12, 600 on a single H200 for the Turbo CTC model, compared to ~6, 000 RTFx cited as current speed leaders on the Open ASR leaderboard (IBM reported). Independent explainers such as OrcaRouter flag the throughput figure as vendor‑reported and not yet reproduced by third parties. Validate on your hardware and audio together with WER before assuming parity.
  • Tradeoff: the CTC design yields much higher runtime throughput (greedy decoding, lighter runtime), but it sacrifices decoder‑side features like flexible autoregressive rescoring, some decoder‑based biasing and translation use cases.

“RTFx near 12, 600 on a single H200”, IBM

Who should run which size (deployment guidance)

  • 3B, intended for solo developers and startups: runs locally (Ollama, LM Studio) and is available in GGUF quant formats (Q4_K_M quantization reported).
  • 8B, positioned for mid‑market teams on a single modern GPU.
  • 30B, targets enterprises with A100/H100 capacity or those using FP8/NVFP4 serving stacks (vLLM suggested by IBM for efficient serving).
  • License: all three Granite 4.2 models ship under Apache‑2.0 (IBM reported), which permits commercial use and on‑prem deployment, though organizations should still run legal and export‑control reviews for regulated environments.

Why the agentic stages matter, capability and risk

Agentic RL (models learning by interacting with sandboxed tools) changes behavior qualitatively. Models learn multi‑step tool chains, stateful workflows and orchestration patterns. IBM applied the agentic curriculum to 8B and 30B to emphasize software engineering and terminal automation skills.

That capability is useful, but it increases your operational surface area. Before you enable agentic modes in production, require:

  • Permissioned tool access and strict RBAC for any API that executes actions.
  • Sandboxing with robust input/output capture and immutable audit trails.
  • Human‑in‑the‑loop gates for any write/execute actions and canary rollouts for behavior changes.
  • Continuous red‑team testing targeted at prompt injection, privilege escalation, and adversarial tool misuse.

Operational red‑team & validation checklist (practical tests)

  • Prompt injection & context‑leak tests: probe chain‑of‑thought outputs for exposure of secrets or sensitive context.
  • Privileged command simulation: simulate the model composing and executing terminal commands against a sandboxed host. Verify permission checks and dry‑run protections.
  • Code‑editing audits: run the model on a codebase and verify diff quality, test coverage impact, and whether it introduces insecure patterns.
  • Chain‑of‑thought leakage tests: confirm that enabling thinking mode does not inadvertently reveal private prompts, hidden data or usage logs.
  • Canary & rollback triggers: implement automated rollback conditions when hallucination, error or risky actions exceed thresholds.
  • Audit log completeness: ensure every tool call, user prompt, model response and chain‑of‑thought is timestamped and traceable to a user session ID.

What’s missing or unverified right now

  • Aggregate pretraining compute (total GPU‑hours / dollar cost) for the 15T token run and the staged GRPO pipeline is not published by IBM.
  • Benchmarks and throughput numbers are vendor‑reported; independent third‑party reproductions and detailed safety metrics (hallucination rates, adversarial robustness) are not yet public.
  • Exact quantized artifact availability for 8B/30B (beyond the 3B GGUF claim) should be checked on Hugging Face and the IBM GitHub repo before operational planning.
  • Whether the 512K exposure during pretraining corresponds to supported inference at 512K for all sizes is unclear, IBM’s architecture table documents 128K for inference and the 512K mention refers to curriculum exposure during some pretraining phases.

Reproducibility sprint, practical steps to validate the claims that matter

Run a focused 2‑ to 4‑week sprint to validate capabilities important to your stack. Minimum protocol:

  1. Reproduce a long‑context benchmark (e.g., RULER 64K) with fixed seed, temperature and prompt formatting. Compare latency, memory use and score to IBM’s reported numbers.
  2. Run a factuality/hallucination suite covering your domain prompts (at least 500 prompts) and measure incorrect answers, confidence calibration, and hallucination patterns.
  3. Measure inference latency and peak memory for the context lengths you plan to use (16K, 128K) under your quantization/serving stack (vLLM, FP8/NVFP4 if planned).
  4. For speech: run end‑to‑end transcription on a representative 10-20 hour audio sample and measure real‑time factor (RTFx) and WER on your data to validate or refute IBM’s reported RTFx ≈ 12, 600 on H200.
  5. If enabling agentic features, run the command/control red‑team tests in a fully isolated sandbox and validate rollback, audit logging, and permission enforcement.

Key takeaways, quick questions you’ll want answered

  • Can I use Granite 4.2 commercially and run it on‑prem?

    Yes, IBM ships all three Granite 4.2 language models under Apache‑2.0, which permits commercial and on‑prem use. Still perform legal and export‑control reviews for regulated deployments.

  • Which model size should my team start with?

    Start with 3B for local experimentation (Ollama/LM Studio, GGUF quant options) to validate prompts and tooling. Move to 8B for single‑GPU production prototypes and 30B for fleet or latency‑tolerant enterprise deployments with A100/H100 or FP8 serving.

  • Are agentic tool‑use capabilities present in all sizes?

    No, agentic RL stages were applied only to the 8B and 30B models; the 3B model did not receive those agentic stages (IBM reported).

  • Is the speech throughput claim (RTFx ≈ 12, 600 on H200) independently verified?

    That is a vendor‑reported figure from IBM. Independent explainers (e.g., OrcaRouter) flag it as unreproduced; validate on your hardware and audio before assuming it.

  • Does enabling chain‑of‑thought make the model safer or riskier?

    Visible chain‑of‑thought improves transparency and debugging but can expose private context and increase opportunities for misleading rationales. IBM exposes runtime thinking flags and applies a reasoning‑length penalty in RLHF, but evaluate this tradeoff in your domain with targeted tests.

Bottom line for business leaders

Granite 4.2 is a practical release for organizations that need on‑prem weights, long‑context engineering and controllable reasoning traces. The agentic RL curriculum in the 8B and 30B models is the distinctive capability here. It can accelerate developer automation and terminal tooling, but it also requires careful operational controls before you hand it real privileges.

If you’re evaluating Granite for production, run a short reproducibility sprint that validates latency, memory, hallucination rates, and ASR throughput on your data. Prepare governance for agentic actions (sandboxing, audit trails, human gates). Ask IBM (or check the GitHub/Hugging Face model cards) for the latest quantized artifacts, license details and any additional safety metrics you require.

“Granite 4.2 emits OpenAI-format function calls over an OpenAI-compatible endpoint, so it drops into agent harnesses without adapters.”, IBM

“All three Granite 4.2 language models ship under Apache 2.0”, IBM

“Okay, let’s see. The problem is to find how many ‘r’s are in ‘strawberry’.\nWrite it out: s t r a w b e r r y.\nPosition 3 is r. Position 8 is r. Position 9 is r.\nTotal r’s = 3.”, example chain‑of‑thought shown by IBM

2