zenoh-flat transition (integration branch)#482
Draft
milyin wants to merge 8 commits into
Draft
Conversation
Tracking document for the rebuild of zenoh-java on the generated zenoh-flat-jni bindings. Constituent PRs target this branch; it merges to main as a whole when the transition completes (removing this file).
* initial code generation
* templates in separate files in prebindgen-ext
* mangle harness
* dependency on zenoh-flat-jni added
* switch to keyexprValidate, package support
* zenoh-jni reexported by zenoh-flat-jni
* No load generator
* output only declared in prebindgen-ext
* better error reporting
* reporting fix
* report unresolved descendants from required roots
BFS from each required-but-unresolved type through struct fields, enum
variants, generic args, and impl Fn args, surfacing transitively
unresolved entries that propagate_required cannot reach (because subs
edges don't exist past an unresolved parent). Without this, a missing
declaration in build.rs for a type only referenced as a field of another
unresolved struct went unreported.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactored, test added
* ptr class added
* qualify non-fqn types
* filter undeclared fns out of JNINative/JNIOrphaned emit
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* drop JNIOrphaned bucket; reuse declared_functions trait method
.method() is a namespace declaration — every #[prebindgen] fn must
be assigned to a typed handle, data class, enum, or kotlin_package.
There is no third "orphan" bucket, so JNIOrphaned was always emitted
empty and the duplicate declared_function_names helper just repeated
the PrebindgenExt::declared_functions trait method.
- delete write_jni_orphaned, jni_orphaned_class_name, the stale
generated JNIOrphaned.kt file
- rename render_jni_orphaned_source -> render_jni_package_source,
drop the now-redundant include_only_promoted / wrap_in_object /
declared / class_name params
- drop declared_function_names; render_jni_native_source now takes
&HashSet<syn::Ident> from declared_functions() directly
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* missing functions added to autogeneration
* into in build.rs
* universal post-emit qualification via VisitMut visitor
Replace the five scattered self.qualify_emitted_type() calls (build_input_fn,
build_output_fn, opaque_handle_input, build_handle_destructor_items,
option_input) with a single AST visitor pass applied once at write time.
The previous design qualified only at specific emit sites — each new emit
site had to remember the call. emit_into_dispatcher missed it, so the
impl-Into dispatcher body emitted bare KeyExpr/ZKeyExpr references that
failed to resolve in the binding crate's scope.
Architecture:
- New PrebindgenExt::post_process_item(item: &mut syn::Item) trait hook
(default no-op), invoked by write_rust on every emitted item.
- JniExt implements it via qualify_item, which runs QualifyEmittedTypes —
a syn::visit_mut::VisitMut that walks the full AST (signatures, generic
args, type ascriptions in let bindings, *mut/*const in cast expressions)
and prefixes every bare single-segment Type::Path matching a registered
source-module name with source_module.
- option_input no longer takes &JniExt; the body uses the bare t1 token
and the post-emit visitor qualifies it later.
Also fixes the pre-existing TypeEntry field-name drift in
final_invariant_reports_unresolved_field_of_unresolved_struct.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* transition to zenoh-flat keyexpr for zenoh-java
* ZKeyExpr improved
* closeable data classes
* universa close for data classes
* throwable data class
* config in zenoh-flat
* scouting api
* JNIScout removed
* whatami from jni
* value type
* zenoh id is a wrapper
* &str as input parameter
* logging api
* structured methods
* buildrs api renamr
* scouting package
* packaging fixes
* set intersection level
* zenoh-flat as mods
* qos enums ported
* enum validation in prebindgen
* wrap enums
* bytearray in zenoh-flat
* zenoh-java zbytes
* jni synchronized
* enum return fix
* take and ptr wraper in callback
* manually implemented serialization
* publisher ported
* attachment into
* encoding in zenoh-flat
* encoding constants
* renaming, fromstring added
* zencoding
* subscriber ported
* querier/queriable ported
* query and reply methods ported
* query structure
* renaming
* get and corresponding structures
* full timestamp representation in zenoh-flat
* old log init removed
* liveliness migrated, codegenerator fixed
* value type return fix
* sesstion migrated
* valur type handling fix
* niche fix
* wrappers directly in Session and. Liveliness
* bytes form zenoh_id to string added
* zenoh-jni removed
* jni generator and zenoh-flat moved out
* wrapper lock function
* blob type
* vlaue blob zzenohid
* missing api surface added
* switch to wrapped functions
* single into
* gtom struct from callback
* recursive call
* flatten callbacks
* create java object directly
* flatten input
* no array for locks
* removed expanded code
* expand constructor support
* types expanded
* use accessors
* sample attachment accessor
* nested output
* vec unfold
* comment added
* naming shortened
* converter, renamings
* process error as normal type
* zerror declared
* keyexpr default constructor
* accessor_fun
* full expand
* recursive expamd demo
* new api migration
* zbytes support callback error interface
* expand callbacks
* output kotlin one file per package
* flattened structure
* named callback parameters
* regenrate kotlin code
* generated files reformatted
* cargo fmt
* callback formatting
* no throws in generated code
* hello and query unwrap
* tests fixed
* query close in reply operations
* reply unwrap
* multiline long calls
* sample attributes added
* typed interface callback
* generated code insignificant change
* derfaults in error
* proxy interface
* accelerated onerror handler
* unfold encoding for acceleration
* renamings in builder
* refactor: Remove local zenoh-flat-jni subproject, consume as Maven dependency
- Remove :zenoh-flat-jni from gradle subprojects (moved to separate repo)
- Replace local source dependency with Maven: org.eclipse.zenoh:zenoh-flat-jni:1.9.0
- Remove Cargo build orchestration (buildZenohFlatJni, buildZenohFlatJNI, BuildMode)
- Remove build paths for zenoh-flat-jni libraries (now bundled in Maven JAR)
- Remove Cargo-related Gradle plugins and tasks
- Native libraries are now obtained from the zenoh-flat-jni Maven artifact
- Kotlin sources are now imported from the Maven JAR as a dependency
- Supports optional local development via: includeBuild("../zenoh-flat-jni")
- Simplifies zenoh-java build: Gradle now only compiles Kotlin, not Rust
Migration notes:
- zenoh-flat-jni is now at https://github.com/ZettaScaleLabs/zenoh-flat-jni
- Versions will be synchronized with Zenoh releases (1.9.0, etc.)
- For local development, clone both repos and use composite builds if needed
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* build: Enable Gradle composite build for local zenoh-flat-jni development
- Uncomment includeBuild for ../zenoh-flat-jni since it's now in the PREBINDGEN workspace
- zenoh-flat-jni is at /Users/milyin/PREBINDGEN/zenoh-flat-jni
- Gradle will prefer local build over Maven artifact when composite build is enabled
- Allows coordinated development of zenoh-java and zenoh-flat-jni
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* chore: Remove zenoh-flat-jni directory from zenoh-java
The zenoh-flat-jni project has been moved to:
- GitHub: https://github.com/ZettaScaleLabs/zenoh-flat-jni
- Local workspace: /Users/milyin/PREBINDGEN/zenoh-flat-jni
zenoh-java now consumes zenoh-flat-jni via:
1. Maven artifact (org.eclipse.zenoh:zenoh-flat-jni:1.9.0) in production
2. Gradle composite build (../zenoh-flat-jni) for local development
Removing the local copy avoids duplication and confusion.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* api change ported
* serialize fixes
* loading in zenoh-flat-jni
* import predefined encodings as consts
* Use typed zenoh-flat-jni key expression overloads
* Install CI lint components for the pinned Rust toolchain
* Run flat JNI clippy without a tracked lockfile
* Preserve consuming Rust API semantics
* Limit rustfmt CI to the JNI crate
* Remove stale local task lock
* Use generated JNI receiver methods
* Pin receiver-style JNI bindings in CI
* Track de-prefixed zenoh-flat type names in JNI bindings
zenoh-flat's consolidated API dropped the Z prefix from its Rust types
(ZSample -> Sample, ...), and the generated callback interface names follow
the type name mechanically: ZSampleCallback -> SampleCallback etc. Rename the
four callback references in FlatCallbacks.kt (signatures are unchanged) and
follow the encoding id accessor rename (id -> getId).
Pin CI to the updated zenoh-flat-jni (receiver-style bindings regenerated
against prebindgen main and zenoh-flat main) and to zenoh-flat main.
Verified locally: jvmTest 101 tests, 0 failures against the composite build.
* Use derived JNI factory and getter names
zenoh-flat-jni now derives all regular method names from the flat crate's
identifiers (explicit .name() only where a clash forces it), so the internal
calls follow: JniKeyExpr.newTryFrom/newAutocanonize/newJoin/newConcat,
JniConfig.newFrom*, JniEncoding.newFromString/newWithSchema,
JniZBytes.newFromVec, zSession.getPeersZid/getRoutersZid. SDK public API
unchanged. CI pin bumped to the matching zenoh-flat-jni commit.
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…put override (#484) * Make Encoding and ZenohId.toString pure JVM values An encoding IS its (id, schema) pair — Zenoh's own representation; the textual form is derived from a fixed table. With the id<->name table already JVM-side (the generated ENCODING_* constants) and the companion zenoh-flat-jni change crossing Encoding by value in both directions, the SDK Encoding becomes a plain immutable value: no native handle, no native calls, no lazy caches, no locking, nothing to close. This also removes a native memory leak: received Sample/Query/ReplyError previously retained a cloned Encoding handle with no close path. - Encoding.kt: immutable (id, schema); from()/toString()/withSchema() implement Zenoh's exact conversion rules (incl. the custom-encoding name-preserving withSchema and the render-only CUSTOM<->"" table row); equality on (id, schema), matching Zenoh core. - ZenohId.toString: the little-endian lowercase-hex rule in pure Kotlin. - Callback/fromParts leaves follow the new (encId, encSchema) decomposition; outbound sites pass .id/.schema directly. - Correspondence tests (EncodingCorrespondenceTest, ZenohIdCorrespondenceTest) verify the pure implementations against the native ones across the whole predefined id range, parse/render edge shapes, and random ids — the contract for any JVM-side reimplementation of zenoh-flat API. One test caught and fixed a real divergence (custom-encoding withSchema). Pairs with ZettaScaleLabs/zenoh-flat-jni#4 (CI pin bumped). jvmTest: 105 tests, 0 failures. * Publisher default encoding at declare + PinnedEncoding per-put override Two encoding fast paths over the dual-arm expansion (prebindgen#80, zenoh-flat#3, zenoh-flat-jni#4): - The publisher's default encoding is set NATIVELY once at declare time (PublisherOptions.encoding -> declarePublisher's encoding arm); plain put(payload) passes absent (sel -1), so no encoding data crosses per call. Previously the JVM tier re-sent (id, schema) on every put. - PinnedEncoding (Encoding.pinned()) preallocates the native form; passing it in PutOptions/ReplyOptions/GetOptions crosses only a borrowed handle (native clone = Arc bump) — no schema-string traffic in hot publish loops. close() releases the handle (falls back to the plain (id, schema) arm); a finalizer backstops leaks. A PinnedEncoding equals its plain counterpart. - All outbound sites route through one Encoding?.forWire() helper emitting the selector tuple (-1 absent / 0 value / 1 pinned handle). Tests: PinnedEncodingTest (publisher-default applies natively end-to-end; pinned override reused across puts; post-close fallback), correspondence tests updated to the dual-arm newWithSchema. jvmTest: 107 tests, 0 failures. CI pins: zenoh-flat-jni cbbca40, zenoh-flat d792547. * Move encoding/zid logic to the shared bindings tier; drop pinning zenoh-flat-jni is the shared tier reused by both zenoh-java and zenoh-kotlin, so the pure-JVM conversion logic moves there (EncodingCodec, ZenohId.zidString — next to the generated constants they are built from). io.zenoh.bytes.Encoding shrinks to a thin facade: the established constants and API delegating to the shared codec, zero logic. Drop PinnedEncoding and the selector-tuple wire helper: with the publisher's default encoding set natively at declare time, the hot publish loop crosses no encoding data at all, and per-call overrides crossing (id, schema) cost only a short string decode — not worth a resource class in the public API. Encoding params revert to the plain (present, id, schema) crossing. jvmTest: 106 tests, 0 failures (correspondence suites now exercise the shared codec; PublisherEncodingTest verifies the native declare-time default + per-put override end-to-end). CI pins: zenoh-flat-jni 9bc155b, zenoh-flat main (6d22091, #3 merged).
…485) * Encoding: native handle where it is born, value where it is cheapest Minimize JNI crossings for every encoding flow (consumes zenoh-flat-jni PR #5): - Predefined constants stay VALUE-ONLY: a send carries just their id inside the send call itself — no native handle ever exists for them, no extra crossing. - Custom (schema-carrying) encodings create their native handle at construction — the one natural crossing — and every send after passes a bare jlong (borrowed and cloned natively, reusable forever). - Received encodings (sample/query/reply) arrive send-ready: the delivery decomposition now includes the owned handle, so the save-and-republish scenario never rebuilds the native value from its schema string. Handle release is GC-managed (EncodingCleaner in the shared tier); Encoding stays a plain non-closeable value with unchanged (id, schema) equality/rendering. Send call sites drive the generated encoding selector block through the internal Encoding?.jniSel/jniId/jniSchema/jniHandle helpers — one flat call, zero extra crossings in every case. New EncodingHandleTest covers the ownership model end-to-end; 110 jvm tests green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Lifecycle backstops moved into the gc_managed JNI tier zenoh-flat-jni's Encoding, Session, Publisher, Subscriber, Queryable, Querier, Scout, LivelinessToken and Query handles are now gc_managed (milyin/prebindgen#82): a shared Cleaner frees an unreachable handle whose owner never released it, with explicit close/undeclare settling the release ticket first. Delete the 8 deprecated-for-removal finalize() nets (JEP 421) — the JNI handle is its own backstop now — and drop Encoding's EncodingCleaner registration (superseded by the gc_managed class; the hand-written helper is deleted in zenoh-flat-jni). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * CI: bump zenoh-flat-jni pin to the merged encoding-handle commit The encoding selector wire shape and gc_managed lifecycles this branch consumes landed in ZettaScaleLabs/zenoh-flat-jni#5 (e802317); the old pin predates it, so CI compiled against the previous (present, id, schema) encoding signatures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Received preset encodings are value-only end to end zenoh-flat-jni#6 makes the delivered encoding handle CONDITIONAL on schema presence (binding-local encoding_if_schema behind prebindgen#84's field! leaf): a received preset re-sends through the id arm for free, so no per-message native handle — clone + Box + JVM wrapper + gc-Cleaner registration + Cleaner free — is ever materialized for it. Schema- carrying encodings keep the send-ready handle. Adapt the delivery plumbing: Sample.fromParts encH becomes nullable (Query's already was), the reply callback drops its non-null assertion, and EncodingHandleTest's preset round-trip flips to assert the received copy has NO handle — the new contract. CI pin bumped to the flat-jni#6 tip. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * CI: bump zenoh-flat-jni pin to the #6 merge commit The previous pin was a branch commit deleted on merge; 1e040c9 is ZettaScaleLabs/zenoh-flat-jni#6 on main (conditional encoding handle, final fun!+sig! syntax). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…Encoding always (id, schema) (#489) * Received preset encodings are value-only end to end zenoh-flat-jni#6 makes the delivered encoding handle CONDITIONAL on schema presence (binding-local encoding_if_schema behind prebindgen#84's field! leaf): a received preset re-sends through the id arm for free, so no per-message native handle — clone + Box + JVM wrapper + gc-Cleaner registration + Cleaner free — is ever materialized for it. Schema- carrying encodings keep the send-ready handle. Adapt the delivery plumbing: Sample.fromParts encH becomes nullable (Query's already was), the reply callback drops its non-null assertion, and EncodingHandleTest's preset round-trip flips to assert the received copy has NO handle — the new contract. CI pin bumped to the flat-jni#6 tip. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * CI: bump zenoh-flat-jni pin to the #6 merge commit The previous pin was a branch commit deleted on merge; 1e040c9 is ZettaScaleLabs/zenoh-flat-jni#6 on main (conditional encoding handle, final fun!+sig! syntax). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * KeyExpr: string-backed except declared Consumes zenoh-flat-jni's string-delivering KeyExpr output. Ownership model (the Encoding model transplanted): a native handle exists ONLY behind Session.declareKeyExpr — the single case zenoh attaches a wire declaration (the only state beyond the string, paying off only on send through the declaring session). Everything else is a plain string value: - tryFrom validates natively once and releases the probe handle; autocanonize reads the canonized string and releases; - RECEIVED keyexprs (sample/query/reply) arrive as ONE eager string — no native allocation, nothing to free; the old per-message handle never carried a declaration (zenoh's RX path builds declaration-less) so it bought nothing on re-send while costing an unfreeable alloc and a second getStr crossing for every string-reading consumer; - sends pick the string or handle selector arm via KeyExpr.jniSel/jniStr/jniHandle (cloneHandle for consuming params); - algebra ops use the declared handle or a transient probe; - undeclare demotes to string-backed; close() is meaningful only for declared instances (doc matches upstream again). 113 jvmTest green incl. the new KeyExprHandleTest (constructed/declared/ received ownership + saved-keyexpr re-publish). CI pin bumped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * CI: bump flat-jni pin to the gc_managed-KeyExpr commit Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * examples: ping-pong exercises the receive-then-resend encoding path ZPing puts carry a custom schema-carrying encoding and verify (after warmup) that the echoed sample brings it back; ZPong retransmits the RECEIVED payload AND encoding — a pong that drops metadata is not a faithful echo, and the round trip now measures the real receive-then-resend path the send-ready Encoding handle was built for. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Encoding: received encodings are value-only (id, schema) Adapts to zenoh-flat-jni aecffac, which dropped the receive-side conditional handle leaf: the ping-pong A/B (ZPing/ZPong with a custom schema-carrying encoding, real payload+encoding retransmission) showed the per-receive handle lifecycle costs more than the schema re-decode it saves on resend (flat medians 68/68us handle vs 58/67us value-only at 8B/1KB, value-only never slower). Custom encodings still own a construction-born handle and send as a bare jlong; a saved received encoding re-sends through the (id, schema) value arm. Callback lambdas and Sample/Query.fromParts lose the encH leaf; EncodingHandleTest asserts value-only delivery; CI pin bumped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * ci: bump zenoh-flat-jni pin to e4a8ed7 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * examples: restore upstream ZPing/ZPong The custom-encoding echo instrumentation was measurement scaffolding for the Encoding handle A/B (verdict shipped: received encodings are value-only); the examples keep their upstream form. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Every handle-owning class except ZBytes is now GC-backstopped (Config joined in zenoh-flat-jni PR#8), and the docs now say exactly what happens to forgotten instances: - Session: close() KDoc contradicted the code — close undeclares every live declaration, it does not leave them alive; class KDoc notes the GC backstop. - ZBytes: new lifecycle section — created ZBytes are plain values; received ones free their native buffer on first read; a received payload/attachment that is never read keeps it (deliberately NOT GC-managed: per-message Cleaner cost measured -23% thr @8B). - Publisher/Querier: session reference is weak — forgotten instances are undeclared by the GC backstop, not at session close only. - Subscriber/Queryable: session reference is strong — dropping the reference does NOT stop them; active until undeclare or session close. - Scout/LivelinessToken: 'dropped' now spelled out as GC-collected, non-deterministic; close for prompt effect. - KeyExpr (declared) / Config: backstop notes. CI pin bumped to flat-jni dbb1f8c (Config gc_managed, PR#8). Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 task
…ls (#491) * Session.undeclare: detach the key-expr handle even when undeclare fails The generated wrapper consumes the handle even when the native undeclare errors (the Rust side takes it by value). Clearing keyExpr.handle only after a successful call left a dead handle attached on failure: every later operation selected the closed handle and failed instead of degrading to the string form. Detach in a finally; regression-tested by undeclaring through the wrong session. Found while porting zenoh-kotlin (eclipse-zenoh/zenoh-kotlin#668), which had inherited the same pattern. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Record PR #491 in the transition table Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Umbrella PR for rebuilding zenoh-java on top of the generated JNI/Kotlin bindings (
zenoh-flat-jni, produced by prebindgen from zenoh-flat), replacing the hand-writtenzenoh-jnilayer.This PR stays draft while the transition is in progress. Individual changes land as reviewable PRs targeting the
zenoh-flat-transitionbranch; this PR accumulates them and merges tomainas a whole when the transition is complete. SeeZENOH_FLAT_TRANSITION.mdon the branch for the architecture and the living PR list.Constituent PRs
Merge order
For each constituent PR: upstream companion PRs (prebindgen → zenoh-flat → zenoh-flat-jni) merge first; the zenoh-java PR pins the exact upstream commits in CI and merges here after its checks are green.