Sixteen point nine megabytes. That is the whole weight file for Whistle, Cactus Compute’s speech-to-text model, built to run on a CPU beside Needle, the company’s tool-calling runtime.
What that file does not prove is that you can retire Whisper, hand it a customer recording, or let it fire business tools.
If you pilot it, keep the job small. Short clips. Language forced when you already know it. A probability gate in front of any human review. Nothing past 30 seconds until a build you have hashed either windows long audio or refuses it out loud. And do not let a tool router sitting on these transcripts touch a customer, a ledger, or a CRM. The reasons below come from one notebook on an Apple M4, not from a multi-lab audit.
What the file actually promises
Cactus’s model card describes a single 16.9 MB model that transcribes 16 kHz mono audio in English, German, French, Spanish, Italian, Dutch, and Polish. Language is detected unless you force it. Each word comes back with a start, an end, and a probability. It can lean toward a keyword list, and it can stream, committing a word once two passes agree. Silence and steady noise are supposed to come back as empty text. Music and other sounds can still be written out as words. The card suggests a voice-activity check or a wake word, and this test did not try music.
The same engine family can load Whistle and Needle together, with no GPU required. Decoder depth is a dial: anything from two layers up can be deployed. The notebook never says which depth needle.Whistle() loaded. If quality moves with that choice, a pilot that does not record the depth is not reproducing this test.
The vendor chart puts Whistle next to other speech systems that are not the same kind of machine. Different precision, different runtimes, different decoders. A shared error number is not a controlled experiment. The readable card also does not state a LibriSpeech win over Whisper base. Whatever a comparison graphic implies, those figures are not in the prose.
The card does say its own word error rate was measured across 86, 174 utterances, on a wider set than this notebook touched, including conversational and lecture audio. It does not print that overall score in readable text. This test did not rerun those sets.
One notebook, with the rule written down first
A Marktechpost coding guide tried to answer the comparison people will actually ask: does this hold up against OpenAI Whisper tiny and base? Read this as that notebook’s rules on one Apple M4 run. It is not an independent audit of the published benchmarks.
Word error rate, before any score, is the share of words missed, added, or wrong. Ten percent means about one word in ten is not what was said.
The decoder is half the argument. Greedy decoding takes the likeliest next word and moves on. Beam search keeps several candidate sentences alive and picks later. The guide notes that the LibriSpeech figures in the Whisper paper come from beam search, while the Whisper package most people install defaults to greedy decoding, with a temperature fallback if it gets stuck. Whistle searches with five beams and length normalization. Every gap here is a system gap: model, bits, runtime, and search together. A “win over Whisper” can be a win over a weaker search setting.
The pass/fail rule was fixed before the final run. A lead counted only if a check that resamples speakers, rather than individual clips, kept the gap away from zero, and if that gap still held after a penalty for testing several claims at once. Exploratory probes had already touched some of the same clips before those rules were frozen, which softens how confirmatory the calls can be. Everything that follows is one run.
English draws used seed 2026 and 300 rows per LibriSpeech split, dropping anything longer than the 30-second single pass. That left 299 clips in test-clean and 299 in test-other. Head-to-head Whisper runs used the first 100 test-other clips and 60 test-clean clips. LibriSpeech is read speech, closer to an audiobook than to a sales call in a car. Whisper tiny was run and treated as descriptive, not as a pass/fail claim. The write-up does not publish tiny’s word error rate. It does say tiny accounted for 18.4 minutes of transcription time, against 68.7 minutes for base.
One lead that held, and several that did not
Keep the samples separate. They are not three scores for the same test.
On the 299-clip test-other sample, Whistle scored 10.47 percent. The guide quotes a full-split Whistle figure of 10.49 percent on test-other, which it did not remeasure and which is not printed in the readable model card. Near, not identical, and not a second audit.
On the paired subset, the first 100 test-other clips, Whistle scored 10.27 percent, against 12.46 percent for greedy Whisper base and 10.83 percent for beam search. The 2.19-point lead over greedy base is the one that survived the pre-declared check. The beam-search gap did not.
Test-clean did not clear the bar. Neither did a pooled multilingual sample: 72 clips, 22 speakers, six languages, twelve clips each. That comparison forced the language. Whistle scored 26.40 percent, greedy Whisper base 24.06 percent. The difference favors Whisper and is not significant. Language detection was a separate check, and it was correct on all 72 clips for both systems. Per-language rows are too thin to rank.
So the careful claim is narrow. On this seeded design, Whistle is competitive on short read English, and clearly ahead only of greedy Whisper base on the harder LibriSpeech split. It is not a settled replacement for Whisper base.
Budget the memory, not a fantasy throughput
Speed was measured in fresh processes, on identical float32 audio, as medians after warm-up. In the guide’s prose, Whisper’s time falls from 7.7 times Whistle’s at 5 seconds to 2.8 times at 30 seconds. A separate line in the same write-up says Whistle ran 5.7 times faster than greedy base on a 10-second clip, which sits between those two. Peak resident memory was 124 MB for Whistle and 823 MB for Whisper base.
Hold the ratios loosely. Per-clip time differed nearly threefold between the two English splits, and clip length does not explain it, so the guide itself calls the speed ratios approximate. Do not staple them to the vendor’s timing note. That comparison is 10 seconds on an Apple M4 Pro, and the card gives no numeric speedup in prose. Different chip. No number to merge.
The figure an infrastructure lead can actually use is the memory gap. In this run, one Whisper base process held about six and a half times the peak memory of Whistle. That is single-process peak memory, not a concurrency test. Nobody loaded six copies and proved they fit. Still, 124 MB is a background process on a branch desktop or a sales laptop. 823 MB is a guest you have to plan for.
The speed gap points the same way, and then shrinks. A 5-second clip is where Whistle looks several times quicker. A 30-second customer turn, the longest audio this model is clearly willing to accept, was only 2.8 times quicker here. The economic case is a high volume of short utterances: counter questions, voicemail snippets, after-call notes. It is not an argument for chewing hour-long recordings on a CPU you already own. A single 3.5-second clip finishing in 33 milliseconds is a nice anecdote, not that capacity plan.
Long audio, on one file
The documented contract is still the line a buyer will rely on:
Audio of any length, one pass up to 30 seconds and windowed beyond it.
On one longer same-speaker clip, that contract failed, and the two engines failed differently. The Python package was pinned at cactus-needle 3.2.0. The engine that package loaded identified itself as 3.3.0. That build returned fewer than half the words and raised no error. An engine identifying as 3.2.0, taken from the same repository snapshot, refused the file with a runtime error instead. Package version and engine version are not the same label here. The card does not say which behavior is intended.
This is one probe, not proof that windowing never works. It is enough to keep long recordings off a pilot. Silent omission is the dangerous case. A refused file is an incident. A transcript that skips the middle of a customer call looks finished.
A hand-built workaround on that one constructed clip did better when splits followed quiet points: 3.75 percent word error rate, against 6.74 percent for fixed windows, with joins padded by 0.4 seconds of silence. One clip cannot choose a segmentation method. Until a hashed build either windows long audio or fails loudly, treat anything past half a minute as unsupported.
Pin the hash if you try this at all. The repositories replace files in place. Another public snapshot already shows the weights and the card being swapped. Which snapshot is newer is not settled from the dates on hand. An unpinned download is not a promise that you are running the build someone else timed.
Probability is the review queue
Word probability was a much stronger error signal than word duration. Words scored below one half were wrong 69.2 percent of the time. Words at 0.9 or above were wrong 2.5 percent of the time. The notebook does not say how many words those rates cover, so treat them as this run’s split, not a universal calibration.
Sorted by how unsure the model was, wrong words separated from right ones with an AUROC of 0.916. Sorted by how short the word was, the score was 0.509, barely better than a coin flip. If a person is going to check anything, send them the low-probability words. Do not send them the brief ones.
Timestamps, keyword lists, and a slow commit
Timestamps were monotonic on every clip checked. Prepending 2.000 seconds of silence to 20 clips shifted the first word by a median of 2.000 seconds, which is what you want from a start time. Cutting the audio at a reported word end reproduced the expected prefix on only one fifth of those 20 clips. That is a stability probe, not proof the end times are accurate. There is no gold alignment in the test.
Keyword biasing has a ceiling and a trap. On 185 test-other clips that already contained at least one Whistle error, an oracle list (the actual missed reference words, which means the test used the answer) cut word error rate from 14.74 percent to 7.62 percent. A matched list of wrong keywords, words from another clip that were not spoken here, left the rate at 14.82 percent, a non-significant change, and sat 7.20 points worse than the oracle after the multiple-comparison check. Eight frequent words that were not in the clip raised error by 2.05 points. A bias list of product names can help when those words were said. It can also shove the transcript toward words nobody spoke. The oracle is not an operating mode.
Streaming, on eight artificial same-speaker sessions at one-second chunks, scored 10.05 percent against 10.41 percent utterance by utterance. Not significant. Median commit latency was 1.68 seconds after a word ended. On two sessions, that median rose from 0.78 seconds with half-second chunks to 3.44 seconds with two-second chunks.
That latency is the live-agent constraint, and it is stricter than the throughput story. A caller finishes saying “cancel the transfer, ” and on one-second chunks the word is not committed until about a second and a half later. A note-taker can live with that. An assistant that is supposed to interrupt politely cannot. Smaller chunks buy some of the time back, on a sample of two sessions, and they do not make this a barge-in engine.
Non-speech probes (digital silence, two levels of white noise, and a pure tone) returned empty text, matching the card. A real sentence scaled to −30 dBFS was transcribed word for word by Whistle. Whisper base wrote “crazed” where the reference said “traced.” Under the harshest white noise, on 20 test-other clips, Whistle’s error rose from 12.4 percent to 37.1 percent and Whisper base’s from 15.2 percent to 39.9 percent. No difference you can separate from zero. The guide’s explanation is that the front ends do not hear the same noise, roughly 250-3500 Hz for Whistle against 0-8 kHz for Whisper. The card does not make that claim.
The tool call fails on refusal, not on the transcript
The experiment that matters for AI agents is not the audiobook score. Telephone audio from PolyAI’s MInDS-14 English-US training set, originally 8 kHz and resampled to 16 kHz, with 200 rows sampled and clips over 30 seconds dropped, gave Whistle a 31.9 percent word error rate against casual references. That is a different sport from LibriSpeech.
Needle 3 then routed those transcripts among 14 banking-intent tools. The tools and their descriptions were written by the people running the test. Accuracy was 50.3 percent from Whistle transcripts and 51.8 percent from the reference text. Not a significant gap, so the transcript was not the bottleneck. Read that 50 percent carefully. If a router must name one of 14 tools, chance is about 7 percent, so 50 percent is not “it is guessing among the menu.” The write-up also does not fully define the metric: 14-way intent, intent plus a refusal class, or balanced classes. It is a lab sketch of a homemade router, not a verdict on Needle as a product, and not a service level. Overlap with public training audio cannot be ruled out on this set.
The result that should stop a rollout is the off-topic test. Of 40 unrelated LibriSpeech sentences, 17 were refused and 23 still received a banking tool call. A router that assigns a banking function to unrelated speech more often than it declines is a confident intern. For anything that logs a next step, opens a ticket, or moves money, refusal has to be the default. Here it was the exception, and a cleaner transcript would not have fixed it.
Cheap, and not yet honest enough
The direction is right. A small speech model that shares an engine with a tool-caller is what on-device automation actually looks like: notes on a laptop you already own, a store counter, a branch desktop, without a GPU rental attached to every utterance. The memory gap in this run is real. The accuracy story, on the only method we can inspect closely, is competitive on short read English, and only clearly ahead of greedy Whisper base on the harder split.
I would pilot it on short, forced-language audio, with probability thresholds in front of any review, and with keyword lists measured for the damage they do when the word was never said. I would not let it own a recording past 30 seconds. I would not let this style of tool router touch a customer until unrelated speech is declined more often than it is acted on. Small models earn their place by being cheap and honest about what they missed. This one is cheap. On long audio, and on off-topic speech, it is not yet honest enough.
Questions worth asking before a pilot
- Does this 16.9 MB model beat Whisper base?
Not as a general claim, and not on this evidence. The only clear win was a paired LibriSpeech test-other sample against greedy Whisper base. Beam search, test-clean, and a small forced-language multilingual sample did not show a significant lead. One notebook, one run.
- How much lighter is the CPU bill?
On this Apple M4 run, peak memory was 124 MB against 823 MB for Whisper base. Short clips were several times faster, and the gap narrowed to 2.8 times at 30 seconds. Those speed ratios are approximate, and they should not be merged with the vendor’s M4 Pro note, which states no number.
- Can it take audio longer than 30 seconds?
Not on the evidence here. The card promises windowing. One long clip came back missing more than half its words, with no error, on the engine that identified as 3.3.0. A 3.2.0 engine refused the file. Treat long audio as unsupported until a hashed build windows it or fails loudly.
- Can you trust the timestamps and the confidence scores?
Confidence, as an error flag, yes in this run: words under 0.5 were usually wrong, and words at 0.9 or above rarely were. End times are less settled. On a 20-clip stability probe with no gold alignment, cutting at a reported word end reproduced the expected prefix on only one fifth of clips.
- Is it ready to drive tool calls for an agent?
No. In a tester-written banking router, 23 of 40 off-topic sentences still triggered a tool, and a cleaner transcript would not have fixed that. Near-50 percent intent accuracy is not proof the router is guessing among 14 tools, but it is not a service level either.
- Will a fresh download match this test?
Do not assume so. Weights and the card are replaced in place, so pin the hash. Also record the decoder depth. This run never stated which one loaded.