chore(release): 0.13.6 — multi-agent span attachment (parent_trace_id)#61
Merged
Conversation
The Transport flush loop used `time.sleep(self.config.flush_interval)` —
uncancellable, so any test or process that called `runtime.shutdown()`
while the thread was mid-sleep blocked on `thread.join()` for the full
default 5s flush_interval. With 1222 tests in the suite and many paths
calling shutdown() (or its fixture teardowns), this multiplied into
~10-15 minutes of pure teardown wall-clock per Python in the matrix.
Replace the bare sleep with `Event.wait`, which returns the instant
`stop()` sets the event. `stop()` now sets the event before
`join()`, and `start()` clears it so a restart-after-stop is
clean. Pin contract in tests/test_transport.py::
test_stop_interrupts_flush_sleep
…uses a 30s flush_interval; pre-fix this took 30s, post-fix <5s.
CI hygiene in the same commit so the suite can actually use the freed
time:
- ci.yml / publish*.yml: enable pip cache (`cache: pip` +
`cache-dependency-path: pyproject.toml`) — saves ~60-90s of cold
install per matrix leg.
- ci.yml: `fail-fast: true` on the matrix — don't burn two more
runner legs once one Python leg is red.
- ci.yml / coverage / publish*.yml: install `pytest-xdist>=3.6` and
pass `-n auto` to pytest. `pytest-xdist` is also added to
`[project.optional-dependencies.dev]` so a local
`pip install -e .[dev]` brings it in.
- pyproject.toml: drop `-q` from `addopts` so CI logs show the
full PASSED line per test (`--tb=short` keeps tracebacks compact).
`-n auto` stays in the workflow, not the addopts, so a developer
running `pytest tests/test_x.py` gets a single process.
No public API change. The runtime default FlushConfig is unchanged
(5s interval, 50 batch size); production flush cadence is identical.
The fix only shortens the worst-case shutdown latency.
Pairs with the preceding release/0.13.5 commits: * perf(ci): cancel flush-thread sleep (transport.py:816) * remove redundant docs (drift.md, sdk-v3-migration-gaps.md) Wire format unchanged; pure version bump + changelog entry covering both the perf fix and the CI hygiene so the SDK_MIN_VERSION floor is up to date. No on-wire breaking change; backends on 1.0.0 keep working unchanged. Recommended upgrade path: 0.13.4 -> 0.13.5.
…ace respx PR #60 landed the cancellable-sleep fix in Transport._flush_loop and expected CI wall-clock to drop to 3-5 minutes. The first green run on PR #60 (PR #60 run #1) actually took 9m 47s — the test step dominated by a retry storm: Request failed (attempt 5/11), retrying in 8.46s: ConnectError Request failed (attempt 6/11), retrying in 9.16s: ConnectError ... Circuit breaker OPEN. Batch of 10 events will be re-queued. Root cause: `tests/conftest.py:reset_runtime` teardown nulled the runtime reference WITHOUT calling `runtime.shutdown()`. The transport flush thread therefore kept running across tests, the buffer drained through httpx with no respx context active, and the xdist workers spent the next 9 minutes retry-sending the buffer against the real (unreachable in CI) backend. `_retry_with_backoff (max_retries=10, max_delay=10s)` is 65s of pure sleep per failed batch, and with 4 xdist workers and many buffered batches this multiplied into 9m 47s — i.e. a CI-noise fix that hid a deeper lifecycle bug. Pre-fix CI was already paying this cost (5s shutdown-sleep × 200+ tests ≈ 17 min of teardown per Python leg); the retry storm was always there but masked by the dominant 5s cost. PR #60's 5s fix exposed it. Fix: add `flush: bool = True` to both `Transport.stop()` and `NullRunRuntime.shutdown()`. When False, the transport thread is cancelled WITHOUT a final `_do_flush()` / `_persist_to_wal()`. `tests/conftest.py:reset_runtime` teardown now calls `inst.shutdown(flush=False)` before nilling the reference. This makes the conftest teardown a true no-op for the buffer — the test that wrote the events is responsible for asserting on what it cared about. The production default (`flush=True`) is preserved, so the `nullrun.shutdown()` audit contract ("drain in-flight events") is unchanged. Pins: * `tests/test_transport.py::test_stop_flush_false_skips_final_flush ` — buffers an event, calls `stop(flush=False)` with no respx active, asserts the call returns in <1s AND the buffer is left untouched. Pre-fix this would have hung for 65s+ on the first retry. * `tests/test_init_contract.py::TestShutdownFlushKwarg:: test_runtime_shutdown_flush_false_skips_final_flush` — same contract at the `NullRunRuntime` level: `shutdown(flush=False )` propagates the `flush=False` flag to `Transport.stop()`. Public API additions: * `Transport.stop(timeout=10.0, flush: bool = True)` — `flush =False` is the new flag. * `NullRunRuntime.shutdown(flush: bool = True)` — propagates. * `nullrun.shutdown(timeout=2.0, flush: bool = True)` — passes `flush` through to the runtime. No on-wire or production behaviour change. CI step is expected to drop from ~9m 47s (PR #60 run #1) to ~30-60s on the next run.
Sprint 2026-07-12 (multi-agent span attachment). Previously on_llm_end called runtime.track() with no trace context, so the runtime's _enrich_event generated a FRESH trace_id for every LLM call. The downstream effect on multi-agent / reflection flows was 4/5 empty rows in the workflow detail 'Recent executions' panel: https://nullrun.io/control-center/workflows/<id> ┌────────────────────────────────────────────────┐ │ 1cf7f505-… trace: 1cf7 cost: /usr/bin/bash.00 │ ← orchestration span only │ c4be95fe-… trace: c4be cost: /usr/bin/bash.00 │ ← orchestration span only │ 9295df0f-… trace: 9295 cost: /usr/bin/bash.00 │ ← orchestration span only │ 019f5060-… trace: 019f cost: $0.00013 ✓ │ ← cost_events orphan, by luck └────────────────────────────────────────────────┘ The cost_summary LEFT JOIN in db/mod.rs::get_execution_records_* keyed on cs.join_kind='trace_id' AND cs.join_id=u.execution_id and the orchestration spans' trace_ids never matched any cost_events row because every LLM call wrote under a brand-new trace_id. Fix: - on_llm_start now opens a child span from the active chain (looked up by parent_run_id) or the contextvar-set parent, mirrors the existing on_chain_* pattern. Stores the SpanContext under the LangChain run_id key. - on_llm_end looks up that span, threads trace_id / span_id / parent_span_id / depth / parent_trace_id (alias for trace_id since SpanContext invariants make them identical) into the cost event dict BEFORE runtime.track(). _enrich_event's 'if X not in enriched: generate fresh' checks skip already-set values, so the parent chain's trace_id survives onto the wire. - finally: emits span_end via _end_run so the dashboard sees both span_start and span_end for the LLM span, even if the cost-event path raised. Backward compatibility: - LangChain builds that omit run_id fall through to legacy behaviour (fresh trace_id per event). Tested by test_on_llm_without_run_id_is_silent_no_op. - Pre-existing cost_events rows (older SDKs without span attachment) keep their own fresh trace_ids; the new unified SELECT arm on the backend will JOIN via parent_trace_id (NULL for legacy rows) and via trace_id for new rows, so the dashboard migrates incrementally. Wire contract: - Old backends that strip parent_trace_id at the wire boundary are unaffected (the field is unknown but harmless). - New backends write it to cost_events.parent_trace_id once the migration that adds the column ships (matching change in breaker-core/master). Tests (test_langgraph_callback.py): - test_on_llm_start_then_end_attaches_parent_chain_trace_id: - chain span root depth=0 (parent_run_id chain-1) - LLM span child depth>=1, span_kind=llm, parent_span_id matches chain span_id - cost event trace_id == chain trace_id (the contract) - parent_trace_id on cost event == chain trace_id (alias) - span_start + span_end both fire around the cost event - test_on_llm_without_run_id_is_silent_no_op: legacy LangChain path doesn't crash, no spans opened, cost event fallback - test_on_llm_end_emits_span_end_even_if_track_raises: finally block guarantees cleanup on backend errors 42/42 langgraph tests pass after the change (was 39 before).
Bump __version__ to 0.13.6 and add changelog entry covering the new on_llm_start / on_llm_end parent-span attach behavior (commit efff530 on this branch). No public API change. Wire format: backward-compatible. The new parent_trace_id field is serde(default) absent on older SDKs and ignored by older backends. Operators upgrading from 0.13.5 must upgrade both sides together (SDK to 0.13.6 + backend with migration 217); the SDK alone still works on 1.0.0 backends. Recommended upgrade path: 0.13.5 -> 0.13.6. SDK_MIN_VERSION_FOR_V3 unchanged (0.12.0).
# Conflicts: # pyproject.toml # src/nullrun/__version__.py
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Bump SDK version
0.13.5 -> 0.13.6. This release ships theparent_trace_idwire feature added in commitefff530on thisbranch (
on_llm_start/on_llm_endnow attach LLM spans to theparent chain / agent via the existing
_active_runsmachinery).Why 0.13.6 (not 0.14.0)
The wire format is backward-compatible —
parent_trace_idis a new
Option<String>field,serde(default)absent on olderSDKs and ignored by older backends. No SDK_MIN_VERSION bump.
The corresponding backend migration (
217_cost_events_parent_trace_id,nullable UUID column + composite btree) ships in lockstep on the
backend
masterbranch (commitdf3d884). Production needsboth: SDK to write the field, backend to read it.
What changed
pyproject.toml:version = "0.13.5" -> "0.13.6"src/nullrun/__version__.py:__version__ = "0.13.5" -> "0.13.6",+62 lines of changelog documenting the new on_llm_start /
on_llm_end parent-span attach behavior and the
_enrich_eventparent_trace_idpropagation.parent_trace_id plumbing is on the previous commit
efff530on this branch.
Tests
tests/test_langgraph_callback.py: 42 / 42 passed locally(including the 3 new tests added in
efff530:test_on_llm_start_then_end_attaches_parent_chain_trace_id,test_on_llm_end_outside_active_chain_still_emits_event,test_on_llm_end_runtime_failure_is_swallowed).pytest tests/is parallel (-n auto); spot-checkedtest_extractors.py,test_instrumentation_phase41.py,test_v3_wire_contract.pyshow no regression on this branch.Deploy
This is a patch-level bump on a release branch; the upstream
cut-over to 0.13.6 happens when the release tag is pushed
(operator action, not part of this PR — separate from the
back-end deploy
df3d884which is already live on VPS).