Baseten Inference Stack

B10 Cache Transceiver

transceiver_runtime: B10 · opt-in

B10 is an opt-in Python cache transceiver with a UCXX data plane for disaggregated KV transfer. It plugs into the existing V2 session machinery as a transfer agent: V2 keeps request orchestration, KV slicing, cancellation, and status flow, while B10 owns only how bytes move between ranks — fault-tolerance-first, through pinned host staging.

1. Where B10 sits

Two message planes coexist: the native plane (ZMQ) belongs to the shared V2 transceiver and carries request orchestration — peer registration, receive-request registration, agent results, session cancellation. The B10 plane (UCXX) carries the transfer protocol and the payload bytes. B10 inherits from the V2 transceiver and overrides exactly one thing — the transfer agent, the box NIXL previously filled — so which requests transfer, when, to whom, timeouts, and cross-rank consensus all stay in V2, untouched.

Where B10 sits in the V2 transceiver stack Prefill and decode rank stacks. V2 components sit above the B10 agent on each side. A dashed ZMQ plane connects the native sender and receiver; a solid UCXX plane connects the two B10 agents. PREFILL RANK (CTX) — SENDER DECODE RANK (GEN) — RECEIVER PyExecutor _send_kv_async B10CacheTransceiver respond_and_send_async TxSession.send native/transfer.py Sender worker threads _deliver_kv_to_agent B10 agent — SendPipeline asyncio loop · _submit_write PyExecutor request_and_receive_async RxSession.receive registers dst descriptors decode KV pool paged KV blocks (VRAM) B10 agent — RecvPipeline UCXX listener · _handle_incoming_write scatter into KV NATIVE PLANE · ZMQ REQUEST_DATA · KV_AGENT_RESULT CANCEL_SESSION B10 PLANE · UCXX control / READY / DATA×N / RESULT + payload B10 replaces only the transfer-agent box NIXL previously filled — everything above it is V2, unchanged.
V2 / native (unchanged) B10-owned ZMQ orchestration UCXX protocol + payload

2. Anatomy of a WRITE

V2 hands the agent size-matched lists of raw source and destination spans; B10's entire job is to make the destination bytes equal the source bytes. Descriptors are coalesced by source pointer into large DATA chunks (512 MiB staging buffers by default, pool count derived from the transfer token budget), and the chunks are pipelined — bounded by max_in_flight_ops — so chunk i copies into staging while chunk i−1 is on the wire. RESULT is sent only after every destination copy event has completed: success means the KV physically landed.

Anatomy of a B10 WRITE transfer Animated sequence: control message with packed descriptors, READY back, then pipelined DATA chunks gathered from prefill KV into pinned host staging, sent over UCXX, scattered from decode staging into destination KV, followed by RESULT after copy events complete. PREFILL (CTX) DECODE (GEN) KV cache (VRAM) fragmented source spans pinned host staging 512 MiB buffers (DRAM) NIC · UCXX endpoint destination KV (VRAM) fragmented destination spans pinned host staging 512 MiB buffers (DRAM) NIC · UCXX endpoint control · READY · RESULT (per-transfer tags) DATA×N payload · UCXX (per-chunk tags) gather kernel · D2H scatter kernel · H2D control READY RESULT C1 C2 C3 ✓ copy events landed
control message (packed descriptors) READY / RESULT coalesced DATA chunk

The key insight. The pipeline overlaps stages: while C1 rides UCXX, C2 is already gathering into pinned staging, and C3 follows — so the D2H copy stream, the NIC, and the H2D scatter all stay busy at once instead of taking turns.

3. Quarantine on uncertainty

The doctrine: on any uncertainty — timeout, peer death, late completion — quarantine the resource and retire the identity; never reuse anything UCXX might still touch. The failure funnel sweeps the staging views, tags, transfer id, and endpoint generation into a TTL quarantine (120 s) instead of returning them to circulation, and expired quarantine drops staging rather than recycling it. Because every receive lands in pinned host memory first, the blast radius of a corrupt or late-arriving transfer is a DRAM staging buffer — never the live KV cache.

Quarantine on uncertainty A transfer fails mid-flight. The staging buffer, tags, transfer id, and endpoint generation move into a TTL quarantine pen. A late peer write hits only the quarantined DRAM buffer; the live KV cache is unreachable. After the TTL the buffer is dropped and the pool refills. from wire PINNED STAGING POOL (DECODE) buffer — free buffer — free tag registry · transfer ids · endpoint gen QUARANTINE · TTL 120 S LIVE KV CACHE no copy is ever issued from quarantined staging ✕ deadline staging buffer (view) tags transfer id endpoint gen late write ✕never fresh buffer
quarantine (TTL 120 s) late peer write live KV — protected

Blast radius. A failed or suspect transfer forfeits its resources instead of releasing them; a late peer write can tag-match only the quarantined DRAM buffer, never the KV cache. Failures stay request-scoped — cleanup never marks the worker unhealthy.

4. v1 UCX vs B10 — reliability contracts

Both designs stage payloads and scatter from staging into KV blocks — the reliability differences are contracts, not topology. The worst failure mode is silent KV corruption: a late or replayed message tag-matching into another transfer's staging and getting scattered into a live request. B10 has not had fewer bugs than v1 — it has had bounded, visible, per-request ones.

v1 UCX transceiver (C++) B10
Failure doctrine Quarantines staged payload buffers on failure (fork hardening), but identities — tags and request ids — carry no registry or reuse exclusion and return to circulation immediately. Quarantine on uncertainty, uniformly — staging views, tags, transfer IDs, and endpoint generations all sit out a TTL (120 s).
Staging location Pinned host DRAM payload staging in this fork (on by default); upstream defaults to device staging. A transport optimization with containment attached to one buffer pool. Pinned host DRAM unconditionally — the entire failure contract is built on staging being cheap DRAM, so quarantine fires on every uncertain failure.
Identity lifecycle Tags derived from identifiers that can recur; no reuse protection. Tags hashed over (pair domain, endpoint generation, transfer id, chunk, kind); every tag registered before receives post (collision → hard failure); failed transfers' tags and ids quarantined; endpoint generations never reused.
Completion semantics Send-complete — "success" means handed to the NIC. RESULT sent only after CUDA copy events prove the bytes landed in destination KV; completion decided locally on the receiver, not via a cross-plane notification.
Time bounds Ad hoc timeout behavior spread across implicit state. Every await is fenced by the per-transfer deadline — remaining_s() raises at expiry — so failures exit through one funnel per pipeline into quarantine.
Failure posture Quiet wrongness — the signature failure was silent. Loud, per-request failure; the process keeps serving.