feat(python): expose TCP client configuration by ethanlin01x · Pull Request #3776 · apache/iggy

ethanlin01x · 2026-07-29T17:58:35Z

Which issue does this PR address?

Rationale

The Python binding accepts only a bare server address, so reconnection and auto-login cannot be configured from Python. The SDK's session recovery is therefore unreachable: a server restart surfaces as Unauthenticated on the next call.

What changed?

IggyClient(...) took only host:port, with AutoLogin::Disabled hardcoded and the reconnection policy untunable.

It now also accepts a keyword-only TcpConfig mirroring the Rust TcpClientConfig (auto_login, reconnection, heartbeat_interval, TLS, nodelay). Unset fields fall back to the Rust defaults, and durations are validated datetime.timedelta. The bare-address constructor and from_connection_string are unchanged.

One behavior change: a negative timedelta on create_topic / update_topic (message_expiry), the consumer(...) intervals, or AutoCommit.Interval(...) became a near-u64::MAX duration and now raises ValueError.

Local Execution

Passed
Pre-commit hooks ran

AI Usage

Which tools? Claude
Scope of usage? help generate and review this PR
How did you verify the generated code works correctly? Integration tests against a real server prove credentials are replayed on connect without a manual login_user().
Can you explain every line of the code if asked? Yes, all the changes are checked by the human.

IggyClient accepted only a server address, so auto-login and reconnection tuning were unreachable from Python. Without credentials to replay, the SDK's own session recovery never fires and a dropped session surfaces as Unauthenticated on the next call, leaving the application to hand-roll a connect/login/probe loop. TcpConfig mirrors TcpClientConfig field for field and is accepted by the IggyClient constructor alongside the existing address string. AutoLogin carries the credentials without exposing them back to Python, and TcpReconnectionConfig carries the retry policy. Credentials is re-exported from the SDK prelude because AutoLogin::Enabled cannot be constructed without naming it. Closes apache#3742

Round-trip every field through the getters so a default that drifts from the Rust SDK is caught, and assert that neither the password nor a personal access token comes back out of repr. The auto-login tests are the point of the configuration: a privileged call succeeds without a manual login_user() when credentials are configured, and fails without them.

The existing examples all reach for a connection string, which leaves the new config types undiscoverable. This one configures auto-login and reconnection directly and never calls login_user, so the recovery the credentials unlock is visible: restart the server while it runs and the client picks up where it left off.

The README pointed only at the examples directory, so the configuration surface stayed invisible to anyone reading the package page on PyPI.

A negative timedelta normalizes to negative days plus positive seconds, so the old conversion summed to a negative i32 and cast it to u64, turning interval=timedelta(seconds=-1) into u64::MAX seconds: the config constructed fine and the client then slept forever on reconnect. Days arithmetic also overflowed i32 beyond ~68 years, and the reverse conversion stuffed everything into the seconds argument so such values could not read back. Conversion is now fallible, rejects negative input with ValueError at construction, computes in i64, and splits days on the way out. The AutoCommit conversion becomes TryFrom to carry the error. The boolean constructor defaults were literals in the pyo3 signature, so a change to a Rust default would silently not propagate. They are now Option arguments that fall back to TcpClientConfig::default(), the same way the durations already did.

codecov · 2026-07-29T18:26:47Z

Codecov Report

❌ Patch coverage is 96.69118% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.99%. Comparing base (70903b8) to head (5be29a8).

Files with missing lines	Patch %	Lines
foreign/python/src/config.rs	97.07%	6 Missing ⚠️
foreign/python/src/topic.rs	66.66%	2 Missing ⚠️
foreign/python/src/duration.rs	96.55%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #3776      +/-   ##
============================================
- Coverage     76.59%   75.99%   -0.60%     
  Complexity     1046     1046              
============================================
  Files          1346     1365      +19     
  Lines        170846   172773    +1927     
  Branches     142405   142686     +281     
============================================
+ Hits         130860   131301     +441     
- Misses        36177    37566    +1389     
- Partials       3809     3906      +97

Components	Coverage Δ
Rust Core	`75.81% <ø> (+0.01%)`	⬆️
Java SDK	`63.67% <ø> (ø)`
C# SDK	`58.99% <ø> (-13.11%)`	⬇️
Python SDK	`89.98% <96.69%> (+1.28%)`	⬆️
PHP SDK	`82.97% <ø> (ø)`
Node SDK	`96.36% <ø> (+0.08%)`	⬆️
Go SDK	`69.18% <ø> (ø)`

Files with missing lines	Coverage Δ
foreign/python/src/client.rs	`99.84% <100.00%> (+0.62%)`	⬆️
foreign/python/src/consumer.rs	`82.21% <100.00%> (+1.10%)`	⬆️
foreign/python/src/lib.rs	`100.00% <100.00%> (ø)`
foreign/python/src/receive_message.rs	`88.37% <ø> (ø)`
foreign/python/src/send_message.rs	`87.67% <ø> (ø)`
foreign/python/src/duration.rs	`96.55% <96.55%> (ø)`
foreign/python/src/topic.rs	`82.27% <66.66%> (-1.06%)`	⬇️
foreign/python/src/config.rs	`97.07% <97.07%> (ø)`

... and 101 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

conftest auto-marked every module as integration, so tests explicitly marked unit could not be selected with -m "not integration" even though they need no server. The auto-mark now skips them. New cases pin the duration boundaries (negative rejected, zero legal, beyond the i32 seconds range round-trips) and the README claim that a connection string and TcpConfig reach the same behavior.

The snippet ended with a top-level await; every other sample in the repo wraps in asyncio.run, so paste-and-run failed on the only snippet a PyPI reader sees first.

ethanlin01x · 2026-07-29T19:01:43Z

/ready

slbotbm

Looks good. Mostly cosmetic changes. One thing though: in the docs, you are declaring the thrown errors as PyValueError and similar types. These are rust types which the python user will not see. Also, there are references to the rust sdk in public docs. Please remove them. Our modelled users are python users, who would not know anything about rust.

A separate example for the new config is not needed. The getting-started producer and consumer now build a TcpConfig with auto-login and reconnection instead of a connection string.

The TLS and nodelay options appear as commented-out fields in the snippet instead of prose, and the auto_login and from_connection_string notes are dropped.

Python users see ValueError and RuntimeError rather than the PyO3 exception names, and the wrapped Rust types are an implementation detail.

Docstrings for methods returning Awaitable[None] said they return Ok(()), which does not exist for a Python caller. State the raised exception instead.

IggyDuration::as_micros() truncates the count to u64, so a duration near timedelta.max wrapped to a wrong value instead of surviving the round trip, and the OverflowError guard below could never fire. Read the std Duration directly to keep the u128.

The negative-duration rejection also changed methods that shipped before this branch, such as create_topic's message_expiry, but only the new config classes had coverage.

The tls_validate_certificate docstring was neutral for a flag that accepts any certificate the server presents.

ethanlin01x · 2026-08-01T05:41:26Z

Hi @slbotbm
Thanks for the review. Renamed to the Python exception names in 1e5561d, and 42ad9ab clears the remaining Rust the module docstring. The RuntimeError references will need another pass once #3696 lands and Iggy failures become IggyError

hubcio · 2026-08-05T19:58:32Z

@ethanlin01x you can fix them without creation of issue, just mention that it was found in #3776.

The six synchronous getters on IggyConsumer took the consumer mutex with blocking_lock() while holding the GIL. consume_messages holds that mutex for the whole consumption run, so reading an attribute during consumption hung the interpreter, and doing it from a callback panicked inside the Tokio runtime. None of those getters need exclusive access: the name, stream and topic are fixed at construction, and the partition id and offsets live behind Arcs in the Rust SDK. IggyConsumerState exposes the latter as a cloneable view, so the Python wrapper can keep its own copies and never touch the lock. Found while reviewing apache#3776.

The synchronous getters on IggyConsumer took the consumer mutex with blocking_lock() while holding the GIL, and consume_messages holds that mutex for the whole consumption run. Reading an attribute during consumption hung the interpreter; reading one from a callback panicked inside the Tokio runtime. None of those getters need exclusive access. The name, stream and topic are fixed at construction, and the partition id and offsets live behind Arcs that IggyConsumerState now exposes as a cloneable view. IggyConsumer owns that state and delegates to it, so the Python wrapper reads metadata without the lock. Found while reviewing apache#3776.

`init_retry_interval = 0` reached `time::interval`, which asserts on a zero period; the timer is built unconditionally, so it panicked even when the stream and topic already existed, surfacing in bindings as an unnamed panic. `polling_retry_interval = 0` became the poll retry sleep, whose loop body makes no syscall, so it burned a core, and with auto-join disabled the join flag never flips and the spin never ends. `init()` already returns `Result` and must run before polling, so it is the choke point for both, closing the hole for every binding rather than each one guarding its own constructor. Found during review of apache#3776.

A zero heartbeat interval pings without pause, and a zero reconnection interval with unlimited retries spins on connect. Found during review of apache#3776.

The synchronous getters on IggyConsumer took the consumer mutex with blocking_lock() while holding the GIL, and consume_messages holds that mutex for the whole consumption run. Reading an attribute during consumption hung the interpreter; reading one from a callback panicked inside the Tokio runtime. None of those getters need exclusive access. The name, stream and topic are fixed at construction, and the partition id and offsets live behind Arcs that IggyConsumerState now exposes as a cloneable view. IggyConsumer owns that state and delegates to it, so the Python wrapper reads metadata without the lock. Found while reviewing apache#3776.

ethanlin01x added 5 commits July 30, 2026 01:48

docs(python): document client configuration in the SDK README

76ab557

The README pointed only at the examples directory, so the configuration surface stayed invisible to anyone reading the package page on PyPI.

ethanlin01x marked this pull request as ready for review July 29, 2026 18:32

github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Jul 29, 2026

ethanlin01x marked this pull request as draft July 29, 2026 18:35

github-actions Bot removed the S-waiting-on-review PR is waiting on a reviewer label Jul 29, 2026

ethanlin01x added 2 commits July 30, 2026 02:39

docs(python): make the README configuration snippet runnable

b3190f4

The snippet ended with a top-level await; every other sample in the repo wraps in asyncio.run, so paste-and-run failed on the only snippet a PyPI reader sees first.

ethanlin01x force-pushed the feat/python-tcp-config branch from 2b6c6ee to b3190f4 Compare July 29, 2026 18:39

ethanlin01x marked this pull request as ready for review July 29, 2026 19:01

github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Jul 29, 2026

slbotbm requested changes Jul 29, 2026

View reviewed changes

Comment thread examples/python/client-configuration/main.py Outdated

Comment thread examples/python/README.md

Comment thread foreign/python/README.md Outdated

Comment thread foreign/python/README.md Outdated

github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Jul 29, 2026

numinnex reviewed Jul 31, 2026

View reviewed changes

Comment thread foreign/python/src/duration.rs Outdated

numinnex reviewed Jul 31, 2026

View reviewed changes

Comment thread foreign/python/tests/test_client_config.py

numinnex reviewed Jul 31, 2026

View reviewed changes

Comment thread foreign/python/src/config.rs Outdated

ethanlin01x added 7 commits August 1, 2026 13:10

docs(python): fold the configuration example into getting-started

4fb0ff3

A separate example for the new config is not needed. The getting-started producer and consumer now build a TcpConfig with auto-login and reconnection instead of a connection string.

docs(python): show the optional TcpConfig fields inline in the README

12da3bc

The TLS and nodelay options appear as commented-out fields in the snippet instead of prose, and the auto_login and from_connection_string notes are dropped.

docs(python): name Python exceptions and drop Rust types from docs

1e5561d

Python users see ValueError and RuntimeError rather than the PyO3 exception names, and the wrapped Rust types are an implementation detail.

docs(python): describe results in Python terms instead of Ok(())

42ad9ab

Docstrings for methods returning Awaitable[None] said they return Ok(()), which does not exist for a Python caller. State the raised exception instead.

test(python): cover negative durations on the pre-existing surface

dc180e5

The negative-duration rejection also changed methods that shipped before this branch, such as create_topic's message_expiry, but only the new config classes had coverage.

docs(python): state the cost of disabling certificate validation

ab97c0b

The tls_validate_certificate docstring was neutral for a flag that accepts any certificate the server presents.

ethanlin01x and others added 6 commits August 7, 2026 07:12

Merge branch 'master' into feat/python-tcp-config

f80d1f0

Merge branch 'master' into feat/python-tcp-config

752a4b9

Merge branch 'master' into feat/python-tcp-config

b4c3584

Merge branch 'master' into feat/python-tcp-config

b63f701

Merge branch 'master' into feat/python-tcp-config

9ee182d

Merge branch 'master' into feat/python-tcp-config

d7cb75f

hubcio approved these changes Aug 10, 2026

View reviewed changes

Merge branch 'master' into feat/python-tcp-config

5be29a8

slbotbm approved these changes Aug 10, 2026

View reviewed changes

mmodzelewski merged commit 1ac819e into apache:master Aug 10, 2026
100 checks passed

github-actions Bot removed the S-waiting-on-review PR is waiting on a reviewer label Aug 10, 2026

ethanlin01x mentioned this pull request Aug 15, 2026

fix(python): read consumer metadata without the consumer lock #3888

Merged

ethanlin01x mentioned this pull request Aug 16, 2026

refactor(sdk): introduce NonZeroIggyDuration for retries and heartbeats #3891

Merged

chengxilo mentioned this pull request Aug 21, 2026

ci(rust): don't fail when a DAG scope matches no tests #3943

Merged

This was referenced Aug 29, 2026

feat(python): add QuicConfig transport configuration #3991

Open

feat(python): add HttpConfig transport configuration #3992

Open

justinmclean mentioned this pull request Sep 1, 2026

Python SDK: expose create_partitions and delete_partitions #4014

Open

2 tasks

Navigation Menu

Uh oh!

feat(python): expose TCP client configuration - #3776

feat(python): expose TCP client configuration#3776
mmodzelewski merged 31 commits into
apache:masterfrom
ethanlin01x:feat/python-tcp-config

ethanlin01x commented Jul 29, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Jul 29, 2026 •

edited

Loading

Uh oh!

ethanlin01x commented Jul 29, 2026

Uh oh!

slbotbm left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ethanlin01x commented Aug 1, 2026

Uh oh!

hubcio commented Aug 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

ethanlin01x commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR address?

Rationale

What changed?

Local Execution

AI Usage

Uh oh!

codecov Bot commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ethanlin01x commented Jul 29, 2026

Uh oh!

slbotbm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ethanlin01x commented Aug 1, 2026

Uh oh!

hubcio commented Aug 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ethanlin01x commented Jul 29, 2026 •

edited

Loading

codecov Bot commented Jul 29, 2026 •

edited

Loading