Baseten Inference Stack

Multimodal serving, in three moves

How we went from "the LLM engine does everything" to a disaggregated encoder with hierarchical response caching. Click the scenarios in each diagram.

Background

How the vision path works

MoonViT + mm_proj turn pixels into a tensor of LLM-space embeddings. That's the whole interface — which is what makes everything after this slide possible.

image / video preprocess pixel_values · grid_thws MoonViT tower 416M-param ViT mm_proj 2-layer MLP mm_embeds N × d_model, bf16 mm_hash · content hash routing + cache key prompt tokens ← one ⟨media⟩ placeholder token engine input mm_positions · mm_lengths
output is just a tensor — no engine state attached mm_hash identifies content, not URLs video = same path, chunked frames

Why this matters. The vision tower's only contract with the LLM is "hand me N embeddings and where to put them." Anything with that contract can run anywhere — and anything content-addressed can be cached.

Stage 1

Everything in one engine

Vision tower, prefill, and decode share one process and one set of GPUs. The engine's native multimodal path (engine_native_passthrough) fetches raw URLs and encodes in the same loop that generates tokens.

User / client Frontend template + tokenize, raw URLs pass through LLM ENGINE — ONE PROCESS · B200 ×8 Vision tower fetch + encode in-loop Prefill Decode req A · image encode ~189 ms prefill decode req B · text blocked behind encode prefill decode time →
416M ViT hits 92% SM util at batch 1 encode is ~189 ms of 251 ms for a 4K image tiny model, huge GPUs — worst host for it

The problem. The vision tower is compute-bound and saturates whatever GPU it lands on — so every image steals SM time from token generation, and the encoder can only scale by scaling the whole B200×8 engine with it.

Stage 2

Disaggregate the encoder — the "E" in EPD

The vision tower + mm_proj move into their own service, multimodal-encoder, on small cheap GPUs. The frontend calls it once per media item and splices the returned embeddings into the token stream; the LLM engine never sees a pixel.

User / client Frontend chat processor · one encoder call per media item MULTIMODAL-ENCODER · 1× H100 / L4 MoonViT + mm_proj ~20 GB of a 1 TB checkpoint in-process LRU mm_hash + URL alias KV router (tokens, [mm_hash]) → worker LLM ENGINE · B200 ×8 Prefill inputs_embeds Decode text-only from here
wire format: {mm_hash, mm_kwargs: mm_embeds · grid_thws · length} HTTP or Dynamo RPC transport E and P/D scale independently validation + SSRF checks off the hot path

What we bought. Encode capacity is now a knob: a fleet of single-GPU encoders in front of the big engines, each loading only the vision + embedding shards. mm_hash flows with the tokens into KV-aware routing, so repeated images also land on workers that already hold their KV blocks.

Stage 3

Cache the encoder's answers

Encoding is deterministic and responses are big (~50 MB for a 4K image) — ideal cache material. response-cache-nginx (OpenResty + Lua) is a transparent response cache deployed at two levels: one in the cluster where the model runs, one shared next to the encoder.

MODEL CLUSTER — WHERE THE REQUEST LANDS Frontend · MultiModalCaller route header names the cache chain nginx cache — in-cluster tmpfs · one hop from the model ENCODER SIDE — SHARED ACROSS CLUSTERS nginx cache — shared one for all clusters, next to the encoder multimodal-encoder LRU → GPU forward
key: client header, else md5(request body) — same key at every level proxy_cache_lock coalesces identical concurrent encodes entries valid 24 h, only 200s cached

The full hierarchy a repeated image can hit before any GPU work:

in-cluster cachetmpfs, by the model→ shared cacheby the encoder→ URL-alias LRUskips download→ mm_hash LRUskips preprocess→ GPU forwardthe thing we're avoiding

Under the hood. Each cache level can scale out: a stateless router consistent-hashes the key over sticky tmpfs shards, so a 50 MB entry lives on exactly one shard, the same key always lands there, and adding a replica only remaps ~1/n of keys.

End state. One deterministic function (pixels → embeddings), pulled out of the engine, put behind content-addressed caches. The engine decodes tokens; the encoder fleet encodes; nginx makes sure each unique image is encoded roughly once.