Needle 2: 14MB on-device model turning voice commands into precise API calls using 28MB RAM

A 14MB binary that turns messy voice commands into precise API calls, and runs in 28MB of RAM

Needle 2 is a compact, purpose-built translator: natural language in, typed function call out. Cactus Compute describes it as a 45M-parameter model that ships as a single 14MB binary and runs a full session in about 28MB of RAM. The engineering tradeoffs are deliberate. Needle 2 is specialized for tool-calling on CPU-only, memory-constrained devices rather than open-ended text generation.

Executive summary

Cactus reports Needle 2 is 45M parameters, ships as a 14MB binary, and runs a session in ~28MB RAM using CQ2-bit weights and a sealed C++ runtime.
Reported decode throughput: Raspberry Pi 5 ~500 tokens/sec; Meta Quest 3S and Apple Vision Pro 400-1, 500 tokens/sec; sub‑$200 phones 300-700 tokens/sec (Cactus reports these figures; see methodology caveats below).
Design focus: strict, schema-constrained function calls for device actions, low latency, local privacy, and tiny memory footprint at the cost of generality.
Before production use, teams should validate licensing, runtime signing, benchmark reproducibility, power/battery impact, and security for on-device tool invocation.

What Needle 2 actually is

Cactus reports Needle 2 is an attention-only model family called a “Simple Attention Network”: 27 layers, width 512, 45M parameters. The model is trained and deployed using Cactus Quants at a CQ2-bit quantization level and executed inside Cactus’s C++ engine without an external runtime at inference.

Important vendor-attributed claims that product teams should verify independently:

Weights and memory: Cactus reports “weights never decompress into RAM”. They say 2-bit codes expand on-the-fly in vector registers and fuse into integer dot products so arithmetic proceeds along int8 paths. That wording suggests no full, dequantized weight snapshot sits in DRAM at once, but independent memory profiling is needed to confirm peak working set and temporary activation behavior.
Quantization: CQ2-bit is the named deployment format. The company has not published a complete numeric spec in the material reviewed here, so ask for format details and cross-tool compatibility before committing.

Why this tiny footprint matters

On-device tool-calling shifts three practical tradeoffs: latency, privacy, and availability. If a watch, kiosk, or entry-level phone can map a voice prompt directly to an API call without a cloud round trip, you get lower cost, personal data stays local, and the feature works where connectivity is poor.

Cactus reports 28MB session RAM is maintained regardless of conversation length because Needle uses a 256-token sliding attention window and pins system/tool declarations as key-value sinks. That limits peak memory growth from arbitrarily long histories, but teams should validate how activations and temporary buffers behave in their real workloads.

The engineering that makes it small, and why those choices matter

Attention-only Simple Attention Network: replaces transformer feed-forward networks with lighter constructs (the source references Hadamard MLPs and a GQA attention variant). Why it matters: fewer dense matrix ops reduce compute and parameter overhead for function-calling tasks that need structure, not encyclopedic knowledge.
CQ2-bit quantization and 2-bit codes: Cactus reports weights are stored as CQ2-bit codes that expand on-the-fly in vector registers for integer fused kernels. Why it matters: tiny binary size and lower RAM pressure, but ask for the numeric encoding, overflow and underflow behavior, and any accuracy regressions in edge cases.
Kernel auto-selection: the engine probes the CPU at startup and picks among SDOT, NEON, AVX2, RISC‑V vectors, wasm SIMD, or scalar kernels. Why it matters: devices use their best instruction set automatically, improving real-world throughput.
Byte-level grammar from JSON schemas: the matcher knows legal tokens for structural outputs before logits are computed, allowing the engine to skip up to 98% of the vocabulary projection on structural tokens (Cactus reports this). Why it matters: constrained decoding reduces compute and helps guarantee well-formed API calls.
Retrieval gating for many tools: when more than five tools are declared, a contrastive retrieval head embeds and scores schemas once per turn and only admits the top five. Why it matters: this limits runtime surface and reduces decoding complexity in multi-tool apps.
Key-value engram memory: hashed n-gram tables act as a compact memory mechanism to keep mappings stable without growing model size.

How tools, confidence, and safety are handled

Tool gating: integrators declare tools. Up to five render directly. If more are present, only the top five after per-turn retrieval are reachable.
Confidence and fail-safes: every response carries a confidence value. Cactus describes it as the minimum of a calibrated post-hoc head and the decoding probability of the call tokens. The runtime returns the empty call [] for off-topic requests. The team’s guidance is: “act above it, re-ask or escalate below it.”
Contractual behavior: constrained output makes the model less likely to hallucinate free-form text, but teams should stress-test schema handling and adversarial inputs to ensure the gating behaves safely under malformed or malicious requests.

“The design premise is narrow and stated plainly by the team: mapping a messy sentence onto a typed function signature needs no world knowledge and no open-ended prose.”

Benchmarks, performance figures, and methodology caveats

Cactus evaluated Needle 2 end-to-end in its shipped engine at CQ2-bit with retrieval enabled across five public function-calling benchmarks using an ordered strict exact match: names, order, and every argument must match. Needle 2 leads both Seal‑Tools splits and posts 98.3% function-name accuracy on Mobile Actions; a well-formed output rate across 3, 641 BFCL rows is 93.4%. The model trails on BFCL v4, which Cactus attributes to distribution mismatch (Needle’s corpus is consumer device actions).

Reported decode throughput (vendor-attributed):

Raspberry Pi 5: 500 tokens/sec (Cactus reports this number; methodology details such as kernel used, CPU frequency and thermal state, OS, decoding algorithm, sequence length, and batch size were not published, request these when validating).
Meta Quest 3S and Apple Vision Pro: 400-1, 500 tokens/sec (reported range; ask for device model, clock and thermal knobs, and whether measurements include audio preprocessing or just text decode).
Sub-$200 phones: 300-700 tokens/sec (Cactus reports this range; confirm Android variant, thermal throttling policy, and single-thread vs multi-threaded kernel choice).

Cactus reports theoretical and measured compute per token: Needle 2 spends about 70 MFLOPs/token with 35M of 45M parameters matmul-active. For context, the company contrasts other models: LFM2.5-230M ~460 MFLOPs/token, FunctionGemma 270M ~540, Apple FM near 6, 000. Ask whether MFLOPs are theoretical FLOP counts calculated from layer math or measured on-device; both approaches matter but answer different questions.

Key methodological questions to request when evaluating the numbers yourself:

Were the benchmarks run at the same quantization and runtime settings for baselines? (Cactus notes baselines ran f16 under vLLM, which favors f16 setups.)
Is throughput measured under greedy decoding, beam search, or temperature sampling?
What sequence lengths and attention windows were used for MFLOPs/token and tokens/sec figures?
Were devices thermally stabilized or measured under battery constraints? Power draw was not reported and should be measured for wearables and phones.

Platforms, examples, and a small illustrative mapping

Cactus reports prebuilt binaries and a static library for macOS, Linux (x86-64, ARM64, ARMv7, RISC‑V, MIPS32el), Windows, Android, iOS/watchOS/tvOS, and WebAssembly. Pebble is a cited integrator that runs Needle locally in the Index 01 app to enable offline voice actions on a watch.

Example (illustrative): a consumer voice prompt and a compact typed function signature the model might produce. This is an illustrative mapping to show the structural form, not a verbatim Needle 2 transcript.

User: “Set a quiet reminder for Tuesday morning at 9.”
Illustrative function call: reminder.create({ title: “Quiet reminder”, datetime: “2026-08-18T09:00:00”, priority: “low”, notify: true })

That example shows the property the model targets: a precise function name, typed arguments, and normalized values. Ordered strict exact match evaluation would require name, order, and all arguments to match the ground truth exactly.

Business impact, where Needle 2 helps and where it doesn’t

Where it helps: privacy-sensitive automation (personal data stays local), lower operational cost and latency (fewer cloud calls), broader reach for AI agents on constrained hardware (wearables, kiosks, industrial terminals).
Where it doesn’t: open-ended reasoning, multi-step planning, or tasks that require substantial world knowledge, those still belong to larger cloud LLMs or hybrid flows where the tiny on-device model handles structured calls and delegates complex work to the cloud.

Open questions and a pre-adoption checklist for product teams

Cactus provides strong engineering signals, but teams should validate practical, security, and legal assumptions before adopting Needle 2 in production.

License and redistribution: Confirm the exact license text and any redistribution or commercial use restrictions. “Open” was stated, but license terms were not published in the materials reviewed.
Runtime signing and attestation: Request cryptographic signing of binaries, reproducible builds, and any attestation mechanisms to guard against tampering on-device.
Data provenance: Ask for a breakdown of the 115B pretraining tokens and 38B post-training tokens: public vs proprietary sources, filtering, and any sensitive data handling policies.
Benchmark reproducibility: Get kernel, OS, decoding mode, sequence length, batch size, and thermal conditions used for published throughput and MFLOPs numbers; reproduce tests on representative hardware.
Power and latency testing: Measure device-level energy consumption and end-to-end latency (audio capture → ASR → Needle decode → API call execution) in real scenarios and under battery constraints.
Security and adversarial testing: Fuzz schema inputs, malformed JSON, and adversarial prompts to verify the tool-gating and empty-call behavior are robust.
Customization and lifecycle: Ask about tooling for fine-tuning or adding private tools, update paths for binaries, and support for enterprise lifecycle management.

Key takeaways, questions a curious leader would ask

Can Needle 2 run on tiny devices without a GPU?

Cactus reports Needle 2 ships as a 14MB binary and runs a full session in about 28MB of RAM, with kernels tuned for many CPU platforms including ARM and RISC‑V. Independent memory and performance profiling on your target device is still essential.
How accurate is it at producing exact API calls?

Cactus reports strong results on device-action benchmarks: Needle 2 leads Seal‑Tools splits, posts 98.3% function-name accuracy on Mobile Actions, and has a 93.4% well‑formed output rate across 3, 641 BFCL rows. These figures reflect strict exact-match metrics; confirm dataset alignment and evaluation conditions when comparing to your targets.
Is it suitable for general NLP tasks like summarization or long-form chat?

No, Needle 2 is intentionally specialized for schema-constrained tool-calling and trades open-ended generation and extensive world knowledge for compactness and precision.
How does it limit incorrect or unsafe calls?

Needle 2 uses a byte-level grammar compiled from JSON schemas to constrain outputs, a contrastive retrieval head when many tools are declared, and a calibrated confidence head. Off-topic inputs return the empty call []. Still, validate behavior with adversarial and malformed inputs in your environment.
What should my team verify before adopting it?

Confirm licensing and redistribution terms, binary signing and attestation, provenance for training data, benchmark reproducibility (kernel/OS/decoding settings), power and end-to-end latency on target devices, and robustness against malformed inputs.

Needle 2 sketches a useful division of labor: tiny, specialist models doing crisp, structured work at the edge while larger models handle messy, creative, or knowledge-heavy tasks in the cloud. For product leaders building AI-enabled devices, that hybrid approach can deliver always-on automation with better privacy and lower operational cost, provided you do the due diligence on licensing, security, and performance in your own environment.