Hackathon · KVBM G2PB + G3

G3 on the peer: spill the host cache to disk instead of dropping it

G2PB peers keep KV blocks in a fixed pinned host pool. Until today, filling that pool meant silently deleting the LRU tail. Now the peer demotes evicted blocks to a local NVMe-backed G3 pool and promotes them back into host memory on fetch — same protocol, same RDMA fast path, validated end to end on a live cluster.

9×
peer cache capacity in the demo (8 host + 64 disk blocks)
37 ms
to demote 4 evicted blocks to disk (observed, in-band)
44 ms
to promote 4 disk blocks back + re-verify xxh3 checksums
0
payload mismatches — every fetched block verified byte-for-byte

1. What changes inside the peer

The service keeps its pinned host pool and access-ordered LRU. New: a second, model-shaped pool backed by an unlinked disk file and registered with NIXL’s POSIX plugin. Eviction becomes demotion; fetch of a disk-resident block becomes promotion. Remote readers still only ever RDMA-read registered host memory.

G2PB peer disk spill tiers A G2PB peer with a pinned host pool and a disk spill pool. Stores land in host memory; evictions demote blocks to disk; fetches promote them back before serving RDMA descriptors. G2PB PEER (CPU NODE, NO GPU) G2 HOST POOL · PINNED access-ordered LRU · RDMA-readable G3 DISK SPILL · NEW unlinked NVMe file 64 blocks · NIXL POSIX plugin BEFORE G3 deleted on evict next request: recompute Staging allocator needs free host blocks; pops the unleased LRU tail SERVING WORKERS / GPU CLIENTS store & fetch by sequence_hash request plane + NIXL/UCX RDMA never touch the peer’s disk directly
KV block payload copy old delete-on-evict path

2. Live proof: three smoke runs against one peer

Deployed on the apse7-mpdev cluster: the peer service (CPU-only image, 8 host + 64 disk blocks) on a CPU node, the smoke client on a GB200 node with one GPU, talking over etcd discovery and NIXL/UCX across nodes. Log lines below are unedited from the live pods.

Run 1 — baseline: store set A, fetch it back

Six blocks staged, RDMA-pulled by the peer, committed, queried, fetched back, payload-verified, and onboarded into the client’s GPU pool. The normal G2PB path is untouched.

instance 7587896719124369319 offer accepted hashes: [17227776552639611783, …] (6) uploaded accepted blocks to instance 7587896719124369319 via staged NIXL transfer onboarded 6 fetched blocks into local device pool

Run 2 — overflow: store set B, host pool evicts set A… to disk

Staging six more blocks into an 8-block host pool forces the LRU tail out. Before today those four blocks were deleted. Now:

G2PB demoted evicted host blocks to disk demoted_blocks=4 elapsed_ms=37 G2PB service reclaimed committed blocks for staging capacity evicted_blocks=4 demoted_blocks=4 pool_blocks=2 remaining_committed_blocks=2 G2PB NIXL offer classified … resident_committed_blocks=8 resident_disk_blocks=4

Run 3 — the payoff: ask for set A again

The offer is rejected (the peer already has every block — four of them on disk). The fetch demotes four other blocks to make room, promotes set A back into pinned memory, re-verifies checksums, and serves RDMA descriptors. The client checks every byte against the original payloads.

instance 7587896719124369319 offer accepted hashes: [] G2PB demoted evicted host blocks to disk demoted_blocks=4 elapsed_ms=27 G2PB promoted disk blocks into host memory promoted_blocks=4 elapsed_ms=44 onboarded 6 fetched blocks into local device pool — payloads byte-identical

3. How it is built

One flag on the service: --disk-blocks N. No protocol changes — offer, query, stage_put, fetch all keep their contracts, and a failed spill degrades to today’s delete-and-miss behavior.

Disk is never remotely addressableFetch leases still expose only registered host memory. Disk-resident blocks are promoted first, then read over RDMA as usual.
Checksums end to endEvery promoted block is re-validated against its stored xxh3_64 before it becomes fetchable. A corrupt disk copy degrades to a cache miss, never bad KV.
Same LRU discipline, two tiersDemotion pops the unleased host LRU tail; a full disk pool drops its own LRU tail for real. Fetch-leased blocks are never recycled.
CPU-only peers stay CPU-onlyThe spill path runs on NIXL’s POSIX plugin with plain file I/O — no GPU, no CUDA driver on the peer node.
Found along the wayThe runtime image was missing the libaio runtime (POSIX plugin silently absent), and CPU/CUDA builds were poisoning each other’s shared cargo cache with incompatible-glibc build scripts. Both fixed.
What’s nextBackground demotion off the staging path, keeping promoted blocks also on disk to avoid re-copying hot data, and O_DIRECT alignment for production disks.

The story. A G2PB peer’s value is how much reusable KV it can hold. Pinned host memory is the expensive, RDMA-fast tier; NVMe is two orders of magnitude cheaper. With G3 spill, filling the host pool no longer destroys cache — the LRU tail slides down to disk and slides back up on demand, invisible to serving workers except that what used to be a recompute is now a ~40 ms promotion. Branch: halex/g2pb-peer-disk-offload.