GPU Embeddings: How Perplexity’s Ivy, Tulip and ROSE Cut Latency and Cost

Perplexity Details Its GPU Embedding Stack: How Ivy, Tulip and ROSE Serve pplx-embed

“Retrieval quality in an AI search product is bounded by two things: how good the embedding model is, and how cheaply you can run it across an index.”, Perplexity Engineering.

TL;DR: Perplexity’s “Fast Embeddings on GPUs” (Perplexity Engineering, Sep 4, 2026) argues that most practical wins for embedding inference come from runtime and harness engineering, like whole-model CUDA graphs, lazy capture, and overlapping CPU/GPU work, rather than from new model architectures. They assemble these into a three-service stack (Ivy → Tulip → ROSE) that prioritizes low launch overhead and high GPU utilization for both batch and online embedding workloads.

Why this matters

Embeddings are easy to design mentally and costly to run at scale. For teams shipping semantic search, reranking, or real‑time personalization, the cost per embedding and tail latency decide whether a feature is profitable and usable.

Perplexity’s practical answer is simple: reuse the LLM prefill/decode kernels you already have, then focus on how you drive the accelerator. Those runtime optimizations, not a new model, are where money and user experience are won or lost.

The three-player stack: Ivy → Tulip → ROSE (and why each exists)

  • Ivy, Rust HTTP gateway. Parses JSON, tokenizes (in‑house unigram tokenizer), applies input templating, splits large batches, translates to the internal gRPC protocol, and load‑balances chunks across replicas. So what: Ivy is where you should measure CPU cost, tokenization latency, and request shaping before anything touches the GPU.
  • Tulip, Rust gRPC scheduler/batcher. Built on tokio + tonic. It accumulates requests, picks sequences first‑come, first‑served, and packs them into batches for the accelerator. So what: Tulip’s simplicity is deliberate. FCFS packing is often sufficient when token cost dominates, but heavier concurrency mixes may need smarter schedulers.
  • ROSE, Runtime‑Optimized Serving Engine (Python runtime orchestrating compiled CUDA kernels). Implements model kernels and layers, manages CUDA graphs, and exposes a step() API that returns a handle for GPU computation. ROSE reuses prefill/decode kernels but runs embeddings as one‑shot forward passes (no KV cache allocation). So what: ROSE is the performance layer where kernel selection, graph capture strategy, and overlap primitives live.

Workloads and an important empirical regime note

  • Batch embedding: Throughput‑oriented jobs such as re‑indexing a vector DB.
  • Online embedding: Low‑latency, query‑time single‑shot embeddings.
  • Scoring: Post‑retrieval ranking of document batches, an intermediate mix of latency and throughput concerns.

Perplexity reports that for the small embedding models they serve, latency tracked total token count more closely than sequence count. They interpret this to mean pointwise MLP and dense layers, which scale with total tokens, dominated compute. After a saturation point (they observed saturation around ~512 tokens on a sub‑1B model), packing more sequences yielded diminishing returns.

Important caveat: this is a regime‑specific observation. For larger models, much longer sequences, or different GPU SKUs, attention’s O(L²) behavior or other bottlenecks can change the outcome. Treat the ~512‑token note as a testable starting point, not a universal constant.

Technical levers that actually move the needle, and how to use them

Perplexity’s stack focuses on a handful of concrete techniques. Each is practical but has operational nuance you must plan for.

  • Whole‑model CUDA graphs. Capture the forward pass into a CUDA graph to collapse many small kernel launches into a single replayable call, which cuts CPU driver overhead sharply. Implementation hint: graphs are shape and configuration specific, so you’ll capture multiple graphs per padded‑token bucket.
  • Lazy capture. Instead of capturing every graph up front, which can take minutes and stall startup, ROSE performs an eager warmup and captures a graph on the second occurrence of a configuration. Implementation hint: prewarm the buckets you expect to see in production, otherwise expect p99 cold‑hit latency for the first few requests to uncaptured buckets.
  • LazyTensor abstraction. A runtime primitive that pairs a page‑locked host buffer with an async cudaMemcpy and a CUDA event. ROSE’s step() returns a LazyTensor handle so Tulip can prepare the next host batch while the GPU is still working. Implementation hint: monitor pinned (page‑locked) host memory closely. Set caps and eviction policies to prevent host OOM under high concurrency.
  • Ragged attention and no KV cache for embeddings. Embeddings are produced in one forward pass, so ROSE skips KV cache allocation and uses ragged attention kernels to avoid padding waste. Implementation hint: ragged kernels reduce wasted compute but require kernel support and careful batching logic.
  • Backend selection for attention kernels. ROSE supports FlashInfer v2/v3 and FlashAttention v4. Perplexity reports FlashAttention v4 is generally faster, but FlashInfer v3 can win for some long‑sequence Qwen runs. Implementation hint: benchmark kernel backends against your exact model and sequence lengths, and don’t assume one backend dominates.
  • Simple scheduling. Tulip’s FCFS packing is pragmatic and performs well under Perplexity’s token‑dominant regime. Implementation hint: if you mix latency‑sensitive single requests with large background batches, try deadline‑aware or priority scheduling to protect the tail latency of interactive queries.

Benchmarks, methodology and the data you’ll need to trust the numbers

What Perplexity reports: they benchmarked their stack against vLLM v0.22.0 using BF16 weights and eval‑derived inputs. Warmup verification checked cosine similarity divergence within 0.1% before running four suites:

  • Low‑latency embeddings: batch size 1 at 128 / 512 / 4096 tokens.
  • Low‑latency scoring: batch sizes 5 / 25 / 50 at 512 tokens.
  • High‑throughput embeddings: batch size 100 with four concurrent processes.
  • High‑concurrency embeddings: 1-16 concurrent requests, including Ivy tokenization and network overhead.

What’s missing, and why it matters: the public summary does not publish raw latency and throughput numbers (p50/p95/p99), exact GPU SKUs, CUDA and driver versions, or the vLLM configuration details used as the baseline. Those artifacts, including hardware SKUs, driver and CUDA versions, benchmark scripts, and p50/p95/p99 data, are necessary to translate Perplexity’s engineering conclusions into production capacity and cost estimates for your workload. Ask for them or reproduce the suites on your infra before making capacity decisions.

Operational risks and runbook recommendations

  • Graph capture cardinality. Thousands of graphs and “multiple minutes” of capture per model (Perplexity’s phrasing) are plausible. That implies artifact storage, versioning, and CI/CD complexity. Runbook item: persist graphs alongside model artifacts and include graph‑replay tests in CI targeted at the cluster’s driver and CUDA versions.
  • Cold‑hit p99s from lazy capture. Lazy capture spreads capture work but produces p99 startup spikes on first uncaptured buckets. Runbook item: schedule a controlled warmup job during rollout that triggers targeted captures for expected shapes.
  • Pinned host memory pressure. LazyTensor uses page‑locked buffers, and under high concurrency these can exhaust host memory. Runbook item: instrument pinned memory, enforce per‑worker caps, and implement backpressure on request ingress when host memory is high.
  • Driver/CUDA/GPU compatibility. CUDA graphs and replayable artifacts are sensitive to driver and microarchitecture changes. Runbook item: add a compatibility gate in deployment pipelines that runs graph replay smoke tests on target nodes before routing live traffic.
  • Kernel variability. FlashInfer versus FlashAttention performance depends on model and length. Runbook item: benchmark backend A/Bs for each model and expected max sequence length, and include the winning backend in production configuration metadata.

A practical checklist for engineering and product teams

  1. Run a saturation curve: measure latency versus total tokens and number of sequences across 2-3 representative GPU SKUs to find your saturation point (use the 512‑token note as a starting target).
  2. Prototype CUDA graph capture with lazy capture: instrument capture time and cold‑hit p99s. Design a prewarm job for hot buckets and persist captured graphs with the model artifact.
  3. Implement LazyTensor‑style overlap: add async host→device copies and event signaling to overlap CPU batch prep and GPU compute. Monitor pinned host memory closely.
  4. Benchmark attention backends: run FlashInfer v3 versus FlashAttention v4 on your models at expected sequence lengths, and pick per‑model defaults rather than a global assumption.
  5. Add CI checks for graph replay on target drivers and GPU SKUs, and include graph‑replay smoke tests in deployment pipelines.
  6. Measure cost impact: translate latency and throughput gains into GPU‑hours saved and cost per 1M embeddings for your billing model before migrating.
  7. Prepare an operational runbook: include graph capture procedures, prewarm jobs, pinned memory thresholds, failure and recovery flows, and rollback steps for driver or model upgrades.

What Perplexity explicitly concludes (preserve their framing)

  • They reuse LLM prefill/decode kernels for embedding inference rather than running a separate embedding engine.
  • For their small embedding models at the reported sequence lengths, latency tracked token count more than sequence count; roughly ~512 tokens saturated a sub‑1B model.
  • Whole‑model CUDA graphs plus lazy capture cut launch overhead without minutes‑long startup pains.
  • LazyTensor lets the CPU prepare subsequent batches while the GPU is in flight instead of synchronously waiting, improving overlap.
  • Ivy, Tulip and ROSE are internal components; pplx‑embed is reachable via Perplexity’s Embeddings API.

Key questions, and short answers

  • How much of the speedup comes from model changes versus runtime engineering?

    Perplexity attributes the largest gains to runtime and harness engineering, CUDA graphs, lazy capture, and host/GPU overlap, while reusing existing LLM prefill/decode kernels rather than changing model architectures.

  • Does latency scale with number of sequences or total tokens?

    In Perplexity’s measured regime (small embedding models and the tested sequence lengths), latency tracked total token count more closely than sequence count because MLP/dense layers dominated compute. This is empirical and may change with larger models, longer sequences, or different GPUs.

  • What are whole‑model CUDA graphs and why use lazy capture?

    Whole‑model CUDA graphs record the forward pass into a reusable GPU graph to reduce per‑kernel CPU launch overhead. Lazy capture delays full upfront capture by warming up and capturing on the second occurrence of a shape, spreading capture cost but producing p99 cold hits unless you prewarm.

  • Which attention backend should I pick?

    ROSE supports FlashInfer v2/v3 and FlashAttention v4. FlashAttention v4 is generally faster in Perplexity’s tests, yet FlashInfer v3 outperformed it for some Qwen long‑sequence runs, benchmark per model and length to choose.

  • Can I reproduce Perplexity’s improvements on my stack?

    The architectural techniques are broadly applicable (CUDA graphs, async memcpy overlap, ragged attention). Exact gains depend on GPU SKU, model size, and workload mix, reproduce their benchmark suites (or request their artifacts: hardware SKUs, p50/p95/p99 data, configs) before making capacity decisions.

Final thought

Model quality still sets the upper bound for retrieval, but when you run this at scale the runtime decides whether that retrieval is affordable and fast. Perplexity’s write‑up is a clear reminder that focused engineering, capture your graphs, overlap CPU and GPU, and test kernel backends, compounds into large operational wins. Start by measuring token and host‑memory costs, run attention‑backend A/Bs on your models, and bake a warmup and capture plan into your deploy pipeline so users see steady performance instead of the cold‑start surprises engineers dread.

Reference: Perplexity Engineering, “Fast Embeddings on GPUs” (engineering blog post, Sep 4, 2026).

0