Together AI timed one agent-written top-k kernel, a small GPU program that chooses which expert blocks to run, at 20 microseconds on random inputs and up to 62 milliseconds on real traffic with 70, 000 tokens of context. Those times are kernel runtime only. The wait for a full answer is a separate clock, and the real-traffic figure is up to about 3, 100 times slower. Random inputs also activated 13 to 38 percent more experts than production text. Same box. The test graded a different workload.
Together AI serves models on its own cloud. The system it describes for this gap is Together Compiler, also called TKCC. The pitch is a little rude to last quarter’s tuning project:
an engine tuned once goes stale, so tuning should never stop.
If you are running AI agents in sales follow-up, support, or other AI automation, expect the history to fatten. Agents hold a session, spawn side tasks, and come back with a longer transcript. A serving stack frozen on launch day ages. You also cannot flip this on for your own stack on Monday. The idea worth taking, whether or not you ever use the compiler, is that inference optimization has become an operations problem.
The tune expires
Mixture-of-experts models do not run every parameter on every token. They route each token to a subset of specialist blocks, the experts. Which experts wake up depends on the actual text, not on a length you typed into a spreadsheet. A kernel that looks brilliant on synthetic junk can stall when real prompts show up with their long, repetitive habits.
Together reports the drift in small, sharp numbers. Prompt lengths in one country ran about 97, 000 tokens against about 70, 000 elsewhere. One customer’s 90th-percentile input grew from 18, 000 to 33, 000 tokens in three days. Nobody rewrote the model. The workload did.
A compiler that stays on the clock
Together Compiler stays in a loop. You do not tune it once and shelve it. You hand it a server launch command, a traffic profile, and a hardware configuration. Coding agents rewrite the model’s forward pass, the code path that actually produces the next tokens. A persistent engine measures the candidates. Winning code ships into the serving engine. When traffic drifts, the loop runs again.
Together says this already runs on several of its serverless endpoints. It does not name them, and it does not publish what those endpoints gained in production rather than on a benchmark. Customer access is a limited beta for Together customers. Running it on Together’s own servers and handing it to you are different claims.
The models named as test targets are Kimi K3, GLM 5.3, MiniMax M3.1 Flash, DeepSeek v4.1 Flash, and Llama 1B. The hardware list runs through NVIDIA B200, B300, GB300, and H100, plus AMD MI355X and AWS Trainium2, with at most two nodes per model. Results are not reported for every pairing. B300, H100, and Trainium2 show up as run targets without token-speed numbers attached.
The wait was the bottleneck
Most agent loops restart vLLM, SGLang, or TensorRT-LLM for every experiment. Together measured a cold start of 761 seconds for MiniMax M3.1 Flash on TensorRT-LLM. That is more than twelve minutes before a candidate is even timed. TensorRT-LLM compiles an engine for a specific GPU type and count, so a long start is plausible. Treat 761 seconds as Together’s measurement on that model. Do not treat it as a constant.
Together Compiler starts the engine once. Agents submit a candidate forward pass over a remote call. The server swaps the forward-function pointer and re-captures CUDA graphs, an NVIDIA technique for replaying captured GPU work with less launch overhead. The hardware list also includes AMD and Trainium2. Together never describes an equivalent recapture path on those chips, and it does not explain the MI355X result. That row is a different model on a different board. Missing CUDA graphs are not shown to be the cause of the small gain.
Measurement is split into two gates. A fast tier serves 12 layers, enough to throw out duds. A slow tier serves 60 layers and checks the survivor on real distributions. Together says the two-tier setup tested 1, 354 candidates against 168 for a single slow tier, about 8.1 times more shots, and “found a solution 7.2 percent faster.” It does not say whether that 7.2 percent is a faster candidate or a search that finished sooner. The trade is plain enough: about eight times the trials bought a single-digit edge.
The 252x figure is easy to misuse. Together reports 252 times higher agent experiment throughput than a loop that restarts the engine. That counts how many more tries the agents got. User latency did not fall by 252x, and token output did not rise by 252x.
Read the rows, not the headline
The comparison Together invokes is SemiAnalysis InferenceX’s AgentX workload. On SemiAnalysis’s own description, AgentX keeps whole session trees alive: a root coding conversation can spawn children, wait, and resume. Prompt text is synthesized. What gets replayed is recorded shape, including lengths, prefix sharing, subagent topology, and timing between turns. A comparable run forces full requested outputs rather than stopping early, and it wants a profiling window of at least 900 seconds. Together’s “under 24 hours” is a search budget. It is a different clock from that window.
Batch size 1 and one live session tree sit on different dials. Together never says which corpus date it pinned, what concurrency it used, or whether a run passed SemiAnalysis’s validity checks. It also never says whether the winning forward passes were chosen on production traffic, on AgentX traces, or on both. Given Together’s own warning that token content changes which experts fire, a benchmark that invents the words is a cousin of production at best.
Hold that next to the banner. In a Marktechpost account dated 5 November 2026, relaying Together and labeled self-reported, Together Compiler beat the best published open InferenceX configurations by up to 2.6x tokens per second. SemiAnalysis’s public AgentX page does not publish these baselines and does not mention Together Compiler, so the matchup has not been checked from the other side.
The batch-size-1, 90th-percentile tokens-per-second-per-user pairs Together actually prints are a narrower claim. The metric is interactive and per user. Tokens from a packed cluster, with many jobs sharing the GPUs, are a different number.
- MiniMax M3.1 Flash on NVIDIA B200: 1, 111 versus 633, about 1.8x. This is the strongest printed row.
- DeepSeek v4.1 Flash on GB300: 1, 205 versus 943, about 1.3x.
- GLM 5.3 on B200: 417 versus 329, about 1.3x.
- Kimi K3 on GB300: 450 versus 423, about 6 percent.
- GLM 5.3 on AMD MI355X: 480 versus 463, about 4 percent.
None of those rows is 2.6x. The largest shown InferenceX gain is about 1.8x. If a 2.6x setting exists, it is not in this comparison.
The layout choice can change the bill
Before the agents write code, a cost model ranks tensor, expert, data, and context parallel layouts. It uses spec sheets, microbenchmarks, measured forward passes, expert imbalance, and uneven request lengths. On Kimi K3 it found a layout with up to 24 percent higher tokens per second. Keep that separate from the InferenceX Kimi row, which moved only from 423 to 450.
On MiniMax, the model picked tensor parallelism of 8 over the baseline’s 4. Together says that halves each GPU’s weight traffic in the memory-bound decode phase. Per-user speed and per-GPU efficiency are different questions, and Together does not say whether that choice stayed inside the same GPU budget. Two nodes is a cap, not a constant bill.
The one-line instruction beat the playbook
Together told some agents only to make the forward pass “as fast as possible.” Those naive agents scored higher than expert-assigned strategies on Together’s unpublished ensemble metric, 6.2x versus 4.9x after 10 trials. Together never says what those scores are relative to. They dwarf the InferenceX ratios, which rules them out as that benchmark in a louder voice.
An expert-curated knowledge base did worse still: 25 to 31 percent less speedup after 12 hours. Custom logging structures ate up to 43 percent of tool calls. The agents were spending a large share of their effort describing the work instead of doing it. Together’s answer was to stop handing them an intermediate representation and give them the raw engine source.
That result comes from Together’s trial. It has not been replicated as a law of AI agents. It is still a useful warning for AI automation programs. Process that feels like rigor can become the workload. If your coding agents are filing more status than patches, you are looking at a measurement failure, and adding knowledge did not help in this trial.
Don’t put these multipliers on the same slide
Other agentic optimizers use similar adjectives and do different jobs. A Cursor and NVIDIA multi-agent effort optimized 235 standalone CUDA kernels on SOL-ExecBench, using 27 B200 GPUs over three weeks, and reported a 1.38x geometric mean with wins on 149 of 235 kernels. Together’s cap was two nodes and under 24 hours, on a full forward pass inside a serving engine. AMD’s GEAK, a Triton kernel agent framework, reports up to 2.59x, close enough to 2.6x to cause a slide accident. That number is on MI300X rather than MI355X, and correctness in that result reached only 63.33 percent. Apple’s RSI-Inference Bench scores agents on 37 tasks that edit vLLM or SGLang, up to eight hours a task, on H100 and B200. It is a scoreboard. It stops short of crowning a speed winner.
Agent sessions often share a long stem, which is exactly where prefix-cache reuse helps or fails. Together never reports a prefix-cache hit rate. A single multiplier is a bad procurement document.
If you pay the token bill
Traffic shape moves in days. Expert routing cares about words, not just lengths. Restarting a serving engine to test an idea can cost longer than the idea deserved. A human playbook can slow the search it was meant to guide.
The risk cut is just as plain. These speed claims are Together’s, self-reported, and uneven, from about 4 percent to a headline the printed rows do not contain. Separate from that table, Together also reports up to 1.7x on Llama 1B at batch size 1 against Hazy Research megakernels, Mirage, and CUDA-graph baselines, and 1.5x to 6.4x on eight unnamed scientific models with bit-exact outputs, meaning the numbers matched exactly. That scientific range can clear 2.6x, and it sits outside the language-model comparison. Bit-exact is not claimed for the models behind the token figures. Numerical tolerances, failure rates, sampling differences, and KV-cache safety for those language models are not published. A faster forward pass that quietly corrupts a long agent session is an incident with a benchmark attached. Do not book it as a savings.
If a vendor, including Together in that limited beta, offers you a continual compiler, ask for the traffic profile it optimized against, the GPU count behind any parallelism change, the correctness bar for the model you actually serve, and the gain on your prompts after the context grows. Then ask again in a week. The loop’s whole point is that last week’s winner is already aging.
Questions worth asking before you budget for this
- Does a tuned inference engine stay fast? Not reliably. Together reports country-level prompts around 97, 000 versus 70, 000 tokens, and one customer’s 90th-percentile input rising from 18, 000 to 33, 000 tokens in three days.
- Is Together Compiler 2.6x faster? That is Together’s headline against the best published InferenceX configurations, self-reported via Marktechpost on 5 November 2026. The five printed rows run from about 1.8x on MiniMax M3.1 Flash on B200 down to about 4 percent on GLM 5.3 on MI355X, and none of them is 2.6x.
- How did random inputs distort the test? Together reports the symptom. A full mechanism is not in the writeup. Random inputs activated 13 to 38 percent more experts than real traffic, and one top-k kernel that took 20 microseconds on random data took up to 62 milliseconds of kernel runtime on real data at 70, 000 tokens of context.
- Did expert guidance help the coding agents? Not in Together’s own trials. A one-line speed goal beat expert strategies on an unpublished ensemble metric, 6.2x versus 4.9x after 10 trials. That score is separate from the InferenceX result. A curated knowledge base cut speedup by 25 to 31 percent after 12 hours.
- What should you demand before you budget for this? The traffic profile it optimized against, the GPU count behind any parallelism change, the correctness bar for the model you serve, and the gain on your prompts after context grows. Then ask again in a week. Dollar cost, rerun frequency, and production gains are not published.
Serving used to be a launch task: pick an engine, tune the kernels, move on to the product. Agent workloads are making that look like painting a ship that is still taking on cargo. Measure on the tokens you actually serve, separate per-user speed from the GPU bill, and refuse any multiplier that cannot point at the row it came from.