Baseten Inference Stack

The encoder, made fast

Six changes to the multimodal encoder and its request path. Hit ▶ Play on each one to watch the before, the after, and what it bought.

2.0×encoder end-to-end
1040 → 510 ms / 4K image
−41%payload per response
base64 gone from every hop
+10–20%throughput
pack overlaps the next forward
566 req/son cache hits
at 1.8 ms each
0.9994new-model parity
GLM-5.2-Vision, live in prod

First — where the time actually goes

282 ms per image, profiled honestly

full-HD image CPU · decode + resize ~86 ms GPU · ViT forward ~190 ms pack + respond msgpack ≈282 ms per image one image CPU 86 ms GPU forward 190 ms TRIED · MEASURED · REJECTED ① normalize on GPU transfer is ~2 ms of the 190 ② batch the forward full-HD already fills the GPU ③ concurrency 1 → 2 one process — the GIL wins

Change 1 · #23973 · merged

The encoder becomes its own package

BEFORE — ONE PACKAGE dynamo · frontend / router / worker encoder — nested inside every change drags all of it AFTER dynamo untouched MM_ENCODER standalone own image · own CI own releases WHAT ONE ENCODER CHANGE COSTS before full CI · GPU gate · every suite after encoder checks only

Change 2 · #21313 · merged

Double the encoder itself

0.0.9 · vLLM 0.15 CPU resize · 98 ms forward 412 ms — 27 GPU→CPU syncs fetch + serialize + HTTP 1040 ms 0.0.11 · vLLM 0.24 GPU resize forward 194 ms · no syncs rest saved ~530 ms 510 ms THE FOUR PATCHES SDPA no-sync kills 27 per-layer GPU→CPU syncs plain nn.Linear replaces 108 wrappers less dispatch pinned async copy side CUDA stream overlaps D2H GPU resize 98 ms of PIL work → F.interpolate

Change 3 · #23921 · #23964 · #24400 · merged

Stop paying the base64 tax

encoder ~40 MB of embeddings nginx cache stores whatever you send frontend splices by length worker bytes on the wire PAYLOAD PER RESPONSE b64 + JSON 7.91 MB · 189 req/s raw bytes 5.94 MB · 315 req/s raw + zstd 4.69 MB · 566 req/s on cache hits

Change 4 · #24032 · merged

Pack while the next image runs

BEFORE — ONE EXECUTOR DOES BOTH GPU executor forward A pack A forward B pack B AFTER — PACK MOVES TO A POSTPROCESS POOL GPU executor forward A forward B forward C postprocess pool pack A pack B

Change 5 · #24403 · open — needs a merge

Free the blob the moment it's dead

dispatch to engine embeds ingested here decode stream seconds to minutes of tokens mm_kwargs · up to hundreds of MB

Change 6 · #24454 · merged

Recount the tokens after the splice

compute_remaining_tokens enforces limit · caps max_tokens prompt · one ⟨media⟩ placeholder after the splice — grew by thousands + sum(mm_lengths) recheck · the fix engine where overflows used to surface

The payoff · #24242 · merged

New model in 34 lines: GLM-5.2-Vision

image any GLM 5.2 request MoonViT tower frozen — shared with Kimi GLM projector the only new weights N × 6144 GLM embedding space ONE ENCODER ANSWERS TO EVERY SERVED NAME zai-org/GLM-5.2 GLM-5.2-Fast GLM-5.2-1M billing alias → all 200

Still open: #24403 (free blobs at dispatch) and #24540 (GLM encoder packaging). Everything else above is merged — states verified Jul 29.