Multipart S3 upload silently drops whole 16-MiB parts at scale — objects are stored short (metadata full), breaking training reads

[gaikwadabhishek]

When s3dlio writes large objects to the S3 backend (NVIDIA AIStore) using its
MultipartUploadWriter (the default path for objects ≥ S3DLIO_MULTIPART_THRESHOLD_MB,
16 MiB), a small fraction of objects come back truncated: the object's recorded
size metadata is the full intended size, but the actual stored/retrievable data is
short by one or more whole 16-MiB parts. The completed multipart object is missing
parts, yet reports the full size.

Reads of these objects then fail — the S3 GET sends Content-Length = full size but
the body ends early:

RuntimeError: S3 GET for s3://.../img_09401_of_56000.npz failed:
dispatch failure (I/O error: error decoding response body: error reading a body from connection)

During an actual training run this surfaces as end of file before message length reached and aborts the run — a single unreadable sample is fatal.

I strongly belive there is a race somewhere in client code which is causing this issue.

Severity

Data-integrity / silent corruption. Rate is low (~0.016%) but the failure is
persistent (retries fail) and fatal to a run (any one bad sample kills the
benchmark). It only manifests at scale/concurrency, so it can pass small smoke tests
and then fail a full-size run.

Environment

• s3dlio 0.9.102 (MultipartUploadWriter, 16-MiB parts)
• mlpstorage 3.0.25, DLIO mlcommons/DLIO_local_changes rev 252a54b
• Backend: NVIDIA AIStore S3 endpoint (proxy :51080, 307-redirect to targets :51081)
• 56000-object unet3d dataset, generated with NP=32 writer processes, single host

Reproduction

1. Generate a large dataset with default multipart (do not set
   S3DLIO_MULTIPART_THRESHOLD_MB), high writer concurrency:

   mlpstorage open training unet3d datagen object \
     --num-processes 32 --data-dir data/mp-dg \
     --params dataset.num_files_train=56000 \
     --hosts <host> --dlio-bin-path <venv>/bin \
     --systemname <name> --results-dir <init'd-dir> --skip-validation

2. Verify every object — compare s3dlio.stat (S3 HEAD) vs a full s3dlio.get:

   head = s3dlio.stat(uri).size          # full intended size
   got  = len(s3dlio.get(uri))           # raises on the broken objects

3. For each failure, cross-check the stored bytes natively (bypasses the S3 GET path):

   ais get ais://<bucket>/<key> /tmp/o.bin   # then stat -c %s /tmp/o.bin
   ais ls  ais://<bucket>/<key> --props size # AIStore's own recorded size

Observed (this run): 9 of 56000 objects truncated

For every failing object, AIStore's own metadata size > its own native ais get
byte count, and the stored byte count is an exact multiple of 16 MiB (the
multipart part size):

object	ais ls size (metadata)	native ais get bytes	= 16-MiB parts
img_09401	189.53 MiB	150994944 (144 MiB)	9
img_11156	207.68 MiB	83886080 (80 MiB)	5
img_16246	65.11 MiB	33554432 (32 MiB)	2
img_16935	251.86 MiB	50331648 (48 MiB)	3
img_20780	107.06 MiB	67108864 (64 MiB)	4
img_34725	155.97 MiB	33554432 (32 MiB)	2
img_35032	152.59 MiB	134217728 (128 MiB)	8
img_37012	113.08 MiB	33554432 (32 MiB)	2
img_39774	130.28 MiB	33554432 (32 MiB)	2

The stored data is always a whole number of 16-MiB parts, fewer than the object
needs → one or more multipart parts were lost, but CompleteMultipartUpload
recorded the full size.

Analysis — which layer?

The 16-MiB-part alignment is the decisive clue: this is multipart part loss, not
arbitrary byte truncation. Two narrowing facts:

• Not the client buffer handling, not low-scale transport. A standalone s3dlio
  stress test — 5000 × 140-MiB objects via the exact
  with MultipartUploadWriter.from_uri(uri) as w: w.write(buf.getbuffer()) pattern,
  in both fresh-buffer and reused-BytesIO+zero-copy-memoryview modes (the
  latter mirrors DLIO's obj_store_lib.py:630/641) — was 100% clean (0 mismatch,
  0 errors). So it is scale/concurrency-dependent, not a deterministic write-path
  defect.
• AIStore's own metadata is internally inconsistent with its own stored data
  (ais ls size > ais get bytes), in whole-part increments.

That leaves two candidates, to be separated by capturing the S3 wire exchange
(UploadPart count + the CompleteMultipartUpload part list vs the bytes AIStore
actually persists):

1. s3dlio issues CompleteMultipartUpload listing a part whose UploadPart did
   not durably succeed (e.g. a dropped/under-retried part under high concurrency).
2. AIStore acks an UploadPart or accepts CompleteMultipartUpload without
   durably storing every part under concurrent load.

Either way the result is an object whose recorded size exceeds its stored data.

Workaround (verified clean)

Force single-PUT uploads by raising the multipart threshold above the largest
object, disabling multipart entirely:

S3DLIO_MULTIPART_THRESHOLD_MB=4096   # > max object size → s3dlio.put_bytes(), no multipart

Regenerating the same 56000-object dataset this way produced 0 truncated objects,
and the training run then read cleanly at NIC line rate.

Question for the WG — is single-PUT permitted for CLOSED?

S3DLIO_MULTIPART_THRESHOLD_MB is a storage-library environment variable, not a
DLIO workload parameter. The mlpstorage rules engine
(CLOSED_ALLOWED_PARAMS, mlpstorage_py/rules/run_checkers/training.py) only governs
DLIO --params overrides, so it neither tracks nor blocks this setting today.

• Training: the dataset is produced in a separate datagen step; the upload
  method there does not affect the measured (read) benchmark, so single-PUT datagen
  appears unobjectionable.
• Checkpointing: the write path is the measured metric, and multipart vs
  single-PUT materially changes write behavior/throughput. Is forcing single-PUT
  (non-multipart) writes permissible in a CLOSED checkpointing submission?

Related

• mlpstorage_py.../storage/obj_store_lib.py:621-644 — the put_data multipart path
  (_MULTIPART_THRESHOLD, MultipartUploadWriter.from_uri(...).write(payload)).
• Datagen to S3 fails using s3dlio #392 / fix(dlio): strip URI scheme from storage.storage_root (#392) #459 — earlier s3dlio↔AIStore S3-compat fixes (URI scheme); different issue.

[russfellows]

Reviewing. May take a day or two to resolve this issue along with #592

[russfellows]

Working on a solution. As feared, this bug and a proper solution requires fixing s3dlio library, DLIO_benchmark code and MLC storage code. So, the fix has to be incrementally created from s3dlio upward, and unit tested at each level. I think I will have a solution pushed today.

[russfellows]

Actually, after more investigation, I realize this is likely an AIstore bug. That is, AIstore (or ANY object storage) should not / MUST not issue a "200 OK" unless the object was completely received correctly. The fact that it is issuing an OK response points to a potential bug in AIstore, not in DLIO or s3dlio.

I am modifying the s3dlio library AGAIN and backing out the default retry operations. I am going to make a head check an OPT in thing, rather than default. No other library does a head check after receiving a 200 ok response, so s3dlio shouldn't either. I will make it an opt in setting, so it is there if desired, but it should not be default.

Again, I think this points to a potential bug in AIstore, more than anything else. But I will continue with s3dlio, DLIO and mlp-storage changes to accomodate.

[russfellows]

OK, after reviewing and thinking about it, I am NOT convinced this was ever a bug in the s3dlio library. There were perhaps some enhancements that are helpful now in DLIO_local_changes, and also in s3dlio. However, I believe it is more likely that the storage target (NVIDIA AIstore in this case) was the actual cause of the issue. A storage system should NOT issue a "200 OK" message if they have not successfully received and stored all portions of a multi-part upload. It is possible the error is outside of AIstore, but I do not see how.

However, s3dlio is updated with version 0.9.106 to provide the opt in verify after put checks if desired. There is also built-in retry code to handle this in DLIO now as of this PR being merged:
mlcommons/DLIO_local_changes#41

Regardless, there are now many mechanisms to do checks and force retries if desired, but these are all opt-in, as no other storage library even provides these mechanisms, much less makes them the default. To make them the default in s3dlio would have meant every put or multi-part upload incurrs a performance penalty. That is not how things should operate, hence it is opt in if desired.

Again, closing until it can be proven that DLIO or s3dlio is causing the issue, rather than a rogue message or redirect from AIstore.

[gaikwadabhishek]

Thanks @russfellows for looking into it and adding the patches. I will check on the aistore side.