Baseten Inference Stack

Hierarchical caching for multimodal responses

Click through in order to watch one image warm the hierarchy. Each plane runs the full EPD path — encoder artifact flows through local cache into prefill, then decode. Pre-warm is a separate upload-time optimization.

Hierarchical multimodal caching architecture User requests enter a global entrypoint that routes to one of two GB300 workload planes. Each plane contains a local distributed cache, a prefill stage, and a decode stage. A shared L4 workload plane C at the bottom contains the central cache and L4 encoder, which back cache misses. GB300 WORKLOAD PLANE A GB300 WORKLOAD PLANE B L4 WORKLOAD PLANE C User / client Global entrypoint KV-aware routing Local cache Prefill Decode rendezvous hashing → shard routing Local cache Prefill Decode rendezvous hashing → shard routing Central cache Cross-cluster reuse L4 encoder Multimodal embeds
routing path cache miss path

The story. A new image lands on plane A — everything is cold, so the request walks the full hierarchy and pays the L4 encode cost before prefill and decode run. The same image later arrives on plane B: its local cache is empty, but central has it from plane A's compute — central hit, then prefill, then decode. A follow-up turn on plane B finds it locally and runs prefill/decode immediately. Pre-warm fires an encode on upload so central is populated before the prompt is even sent.