Skip to content

fix(data): order GPU representation reads after writers - #182

Open
kevkrist wants to merge 1 commit into
NVIDIA:mainfrom
kevkrist:agent/hybrid-zero-copy-hash-partition
Open

kevkrist wants to merge 1 commit into
NVIDIA:mainfrom
kevkrist:agent/hybrid-zero-copy-hash-partition

Conversation

@kevkrist

@kevkrist kevkrist commented Aug 12, 2026

Copy link
Copy Markdown

Summary

This change orders GPU-table reads performed by clone() and release_table() after the
representation's recorded writer work, without host-synchronizing the consumer stream.

  • Owned and view-backed GPU representations record a writer event during construction, including
    when the writer is the CUDA default stream.
  • clone() waits for the source writer event on the clone stream before enqueueing its deep copy.
    The cloned representation records its own writer event and clone() may return while that stream
    is still pending.
  • release_table() waits for the source writer event on the release stream before an owned table is
    handed off or a view-backed table is materialized. It may likewise return while the release stream
    is still pending.
  • The API documentation now states the required device, stream, and source-lifetime preconditions.
  • Regression tests cover default-stream event creation and gated producer ordering for both
    clone() and release_table().

Failure example: clone()

Assume a source allocation initially contains zeroes:

  1. Producer stream P is paused, with a write of 0x5a and source writer event Wsrc queued
    behind the pause.
  2. Clone stream C calls clone() while P is still paused.
  3. Before this fix, C could enqueue its copy without waiting for Wsrc, so the copy was free
    to read the old zeroes instead of 0x5a.
  4. With this fix, C first waits for Wsrc, then enqueues the copy and records the clone's own
    writer event Wclone.

The resulting dependency chain is:

P: source write -> Wsrc
                    |
                    v
C:              source-to-clone copy -> Wclone

clone() remains asynchronous: it can return while C is pending. The caller must keep the
source alive and unmodified until the clone-stream work completes. A consumer of the clone must
wait for Wclone before reading it from another stream.

Failure example: release_table()

Before this fix, release_table() could hand an owned table to another stream, or begin
materializing a view-backed table there, without ordering that stream after the producer. The
release stream could therefore observe stale or partially written source data.

The fixed path inserts a wait for the representation's writer event on the release stream before
the table is handed off or materialized. The operation remains asynchronous and does not
synchronize the release stream. For a view-backed representation, the external owner must continue
to keep the viewed allocations alive and unmodified until the queued materialization completes.

Scope

This patch does not register clone work as an asynchronous data_batch reader, add a
wait_until_ready() API, change the mutable clone/clone_to() APIs, synchronize clone or release
streams, or add exception-path stream draining. Those lifetimes remain governed by the documented
caller preconditions.

@copy-pr-bot

copy-pr-bot Bot commented Aug 12, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@kevkrist
kevkrist marked this pull request as ready for review August 13, 2026 14:48
@kevkrist kevkrist changed the title Order GPU representation reads after writers fix(data): order GPU representation reads after writers Aug 14, 2026
@mbrobbel

This comment was marked as resolved.

@kevkrist
kevkrist force-pushed the agent/hybrid-zero-copy-hash-partition branch from 410d697 to 24496d8 Compare September 17, 2026 16:28
@mbrobbel

Copy link
Copy Markdown
Member

/ok to test 24496d8

@kevkrist
kevkrist force-pushed the agent/hybrid-zero-copy-hash-partition branch from 24496d8 to fd85240 Compare September 17, 2026 18:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants