2026-08-19 · Engineering
Sharding multipart uploads across storage nodes
A 100 GB upload is not one object — it is ten thousand parts asking to be spread across racks. Here is how the gateway decides placement, and why we keep the part table in the metadata service rather than on the storage node.
Why placement matters
When a client uploads a large object with the multipart API, each part is an independent PUT. If every part lands on the same rack, a single switch failure invalidates hours of transfer. Our gateway therefore spreads parts across failure domains as they arrive, using a hash of upload_id plus part number, then records the resulting layout in the metadata service.
A part becomes durable the moment it validates: signature, length, checksum. Nothing waits for the whole object, so a client that dies at part 9 000 can resume at part 9 001 with no penalty.
The part table
Storage nodes are intentionally dumb: they hold immutable, checksummed blobs and know nothing about the object they belong to. The part table — which upload owns which parts, and where — lives in the metadata service, which is versioned and replicated across the region. This keeps nodes swappable and makes a failed node a metadata event, not a data-loss event.
Reaping incomplete uploads
Incomplete uploads are reaped after seven days. Clients can extend the window per upload, and the console lists open uploads with bytes already stored so nobody pays for an abandoned transfer they forgot about.