Skip to content

fix(compile): PerryTS/storekit#1 — route FFI imports through manifest path - #1085

Merged
proggeramlug merged 1 commit into
mainfrom
fix/storekit-1-ffi-imports
May 19, 2026
Merged

fix(compile): PerryTS/storekit#1 — route FFI imports through manifest path#1085
proggeramlug merged 1 commit into
mainfrom
fix/storekit-1-ffi-imports

Conversation

@proggeramlug

Copy link
Copy Markdown
Contributor

Summary

Closes PerryTS/storekit#1. Imports from a package declaring perry.nativeLibrary (e.g. @perryts/storekit) carry ambient export declare function signatures only — the real implementation lives in the linked static library. The compile-side import loop was unconditionally registering every named/default specifier in import_function_prefixes, which made lower_call.rs emit a call against perry_fn_<src>__<name> for symbols whose source module never emits that wrapper (no body to compile). Result: link failure (Undefined symbols: _perry_fn_..._js_storekit_start_listener), or — on older Perry builds where the codegen path differed — an LLVM IR verifier error reporting use of undefined value '@js_storekit_*' because the call was emitted with no matching declare.

Fix: when the import source matches a perry.nativeLibrary package and the specifier's exported name is listed in perry.nativeLibrary.functions, skip the import_function_prefixes registration. lower_call.rs then falls through to the FFI-manifest path (ctx.ffi_signatures), which emits a direct call to the declared FFI symbol along with a matching declare external at the top of the caller IR. The native-library lookup is pre-computed once per import (outside the per-specifier loop) so the check is constant-time.

Before / after

main.ts:

import { js_storekit_start_listener } from \"@perryts/storekit\";
js_storekit_start_listener();

Before:

Undefined symbols for architecture arm64:
  \"_perry_fn_node_modules__perryts_storekit_src_index_ts__js_storekit_start_listener\", referenced from:
      _main in main_ts.o

After (caller IR):

declare void @js_storekit_start_listener()
...
call void @js_storekit_start_listener()

For the 6-symbol smoke test (all params/returns shapes in storekit's manifest):

declare void @js_storekit_start_listener()
declare double @js_storekit_load_products(ptr)
declare double @js_storekit_purchase(ptr)
declare double @js_storekit_restore()
declare double @js_storekit_has_subscription()
declare double @js_storekit_get_jws()

All declares match the calls — types come from the FFI-manifest path in lower_call.rs (which is what ffi_signatures-driven calls were always supposed to use).

Test plan

  • cargo test --release -p perry --bin perry → 332 passed, 0 failed
  • Reproduce from FFI symbols emitted as undefined values in caller IR (compile fails for any importer) storekit#1 (import { js_storekit_start_listener } from \"@perryts/storekit\"; js_storekit_start_listener();) — compiles and runs (exits 0) instead of failing at link
  • Inspect generated IR — every FFI symbol has a matching declare external next to its call
  • Smoke against an unscoped non-manifest dep (e.g. perry-searchbird-apple-auth) to confirm no regression on the path that already worked

… path

Imports from a package declaring `perry.nativeLibrary` (e.g. `@perryts/
storekit`) carry ambient `export declare function` signatures only — the
real implementation lives in the linked static library. The compile-side
import loop was unconditionally registering every named/default specifier
in `import_function_prefixes`, which made `lower_call.rs` emit a call
against `perry_fn_<src>__<name>` for symbols whose source module never
emits that wrapper (no body to compile). The result was a clean compile
followed by a link failure (`Undefined symbols: _perry_fn_..._js_storekit_
start_listener`), or — on older Perry builds — an LLVM IR verifier error
when the call was emitted without a matching declaration.

Skip the `import_function_prefixes` registration when the specifier's
exported name is listed in the source package's `perry.nativeLibrary.
functions` manifest. `lower_call.rs` then falls through to the FFI-
manifest path (`ctx.ffi_signatures`), which emits a direct call to the
declared FFI symbol along with a matching `declare external` at the top
of the caller IR.

Pre-computes the native-library lookup once per import (outside the
per-specifier loop) so the check is constant-time.
@proggeramlug
proggeramlug merged commit 437018a into main May 19, 2026
9 checks passed
@proggeramlug
proggeramlug deleted the fix/storekit-1-ffi-imports branch May 19, 2026 08:25
proggeramlug added a commit that referenced this pull request May 19, 2026
…llection; perry-ffi to crates.io

v0.5.1010 — two issues, same root: shipping `@perryts/storekit` and
similar scoped npm wrappers without a sibling Perry checkout.

collection. `#1085` skipped the wrapper-symbol registration for direct
imports of FFI functions, but the manifest was only added to
`ctx.native_libraries` in `collect_modules.rs`'s import-walk —
never in the re-export-walk at line ~954. Programs that reach the
FFI-bearing package only through a re-export chain
(`export { js_foo } from "@perryts/storekit"` in a wrapper module)
ended up with an empty `ffi_signatures` map in every codegen pass,
fell through to direct-calling the FFI symbol without emitting a
matching `declare external`, and clang rejected the IR with
`use of undefined value '@js_storekit_get_jws'`.

Fix: mirror the per-kind manifest collection inside the
re-export-walk loop. Also defensively force the FFI-manifest path
in `lower_call.rs` whenever `ffi_signatures` knows the name —
guards against future re-export shapes leaking entries past the
per-specifier skip in `#1085`.

moves from optional dep to path-only dev-dependency (its only
references in `crates/perry-ffi/src/` were `#[cfg(test)]` layout
assertions). `runtime-link` feature kept as empty stub for the
existing `[dev-dependencies] features = ["runtime-link"]` lines in
every in-tree `perry-ext-*`. Adds keywords/categories/readme +
crates.io-version metadata on the workspace `perry-runtime` and
`perry-ffi` dep lines. `cargo publish --dry-run -p perry-ffi`
now reaches the upload step cleanly.

Adds `crates/perry-ffi/README.md` for the crates.io rendering.

EventEmitter needs hyper accept-loop wiring (substantial); the
GC-thrash regression bisect surfaced fastify non-blocking listen
(`634e1f58`) as the most likely candidate but needs the user's
actual shop-admin repro to land cleanly. Both documented in
CHANGELOG.
proggeramlug added a commit that referenced this pull request May 19, 2026
…llection; perry-ffi to crates.io

v0.5.1010 — two issues, same root: shipping `@perryts/storekit` and
similar scoped npm wrappers without a sibling Perry checkout.

collection. `#1085` skipped the wrapper-symbol registration for direct
imports of FFI functions, but the manifest was only added to
`ctx.native_libraries` in `collect_modules.rs`'s import-walk —
never in the re-export-walk at line ~954. Programs that reach the
FFI-bearing package only through a re-export chain
(`export { js_foo } from "@perryts/storekit"` in a wrapper module)
ended up with an empty `ffi_signatures` map in every codegen pass,
fell through to direct-calling the FFI symbol without emitting a
matching `declare external`, and clang rejected the IR with
`use of undefined value '@js_storekit_get_jws'`.

Fix: mirror the per-kind manifest collection inside the
re-export-walk loop. Also defensively force the FFI-manifest path
in `lower_call.rs` whenever `ffi_signatures` knows the name —
guards against future re-export shapes leaking entries past the
per-specifier skip in `#1085`.

moves from optional dep to path-only dev-dependency (its only
references in `crates/perry-ffi/src/` were `#[cfg(test)]` layout
assertions). `runtime-link` feature kept as empty stub for the
existing `[dev-dependencies] features = ["runtime-link"]` lines in
every in-tree `perry-ext-*`. Adds keywords/categories/readme +
crates.io-version metadata on the workspace `perry-runtime` and
`perry-ffi` dep lines. `cargo publish --dry-run -p perry-ffi`
now reaches the upload step cleanly.

Adds `crates/perry-ffi/README.md` for the crates.io rendering.

EventEmitter needs hyper accept-loop wiring (substantial); the
GC-thrash regression bisect surfaced fastify non-blocking listen
(`634e1f58`) as the most likely candidate but needs the user's
actual shop-admin repro to land cleanly. Both documented in
CHANGELOG.
proggeramlug added a commit that referenced this pull request May 19, 2026
…llection; perry-ffi to crates.io (#1118)

* fix(compile) #1110 + chore(ffi) #1112: re-export-only FFI manifest collection; perry-ffi to crates.io

v0.5.1010 — two issues, same root: shipping `@perryts/storekit` and
similar scoped npm wrappers without a sibling Perry checkout.

collection. `#1085` skipped the wrapper-symbol registration for direct
imports of FFI functions, but the manifest was only added to
`ctx.native_libraries` in `collect_modules.rs`'s import-walk —
never in the re-export-walk at line ~954. Programs that reach the
FFI-bearing package only through a re-export chain
(`export { js_foo } from "@perryts/storekit"` in a wrapper module)
ended up with an empty `ffi_signatures` map in every codegen pass,
fell through to direct-calling the FFI symbol without emitting a
matching `declare external`, and clang rejected the IR with
`use of undefined value '@js_storekit_get_jws'`.

Fix: mirror the per-kind manifest collection inside the
re-export-walk loop. Also defensively force the FFI-manifest path
in `lower_call.rs` whenever `ffi_signatures` knows the name —
guards against future re-export shapes leaking entries past the
per-specifier skip in `#1085`.

moves from optional dep to path-only dev-dependency (its only
references in `crates/perry-ffi/src/` were `#[cfg(test)]` layout
assertions). `runtime-link` feature kept as empty stub for the
existing `[dev-dependencies] features = ["runtime-link"]` lines in
every in-tree `perry-ext-*`. Adds keywords/categories/readme +
crates.io-version metadata on the workspace `perry-runtime` and
`perry-ffi` dep lines. `cargo publish --dry-run -p perry-ffi`
now reaches the upload step cleanly.

Adds `crates/perry-ffi/README.md` for the crates.io rendering.

EventEmitter needs hyper accept-loop wiring (substantial); the
GC-thrash regression bisect surfaced fastify non-blocking listen
(`634e1f58`) as the most likely candidate but needs the user's
actual shop-admin repro to land cleanly. Both documented in
CHANGELOG.

* fix(compile) #1110 (also): force needs_stdlib when a nativeLibrary is loaded

User's actual /tmp/repro1110_real reduction (direct `import { ... }
from "@perryts/storekit"`) compiled clean — `main_ts.o` referenced
`_js_storekit_get_jws` correctly — but the LINK then failed:

  Undefined symbols for architecture arm64:
    "_perry_ffi_promise_new", referenced from:
        perry_ffi::async_runtime::JsPromise::new::h... in libperry_storekit.a[7](...)
    "_perry_ffi_promise_resolve_bits", referenced from:
        perry_ffi::async_runtime::JsPromise::resolve_string::h... in libperry_storekit.a[7](...)

The link command was `Linking (runtime-only)...` — no
libperry_stdlib.a passed to ld. The user's TS source didn't touch
anything that triggered `ctx.needs_stdlib = true` (console.log alone
doesn't). But every `returns: "promise"` manifest entry in a
nativeLibrary package compiles to a perry-ffi JsPromise::new() call
in the wrapper crate; those C-ABI shims (`perry_ffi_promise_*`,
`perry_ffi_spawn_*`) are *declared* in perry-ffi and *defined* in
`perry-stdlib::perry_ffi_async`. Without stdlib linked, the
references stayed undefined.

Force `ctx.needs_stdlib = true` whenever any `perry.nativeLibrary`
manifest is loaded. One-line change in compile.rs right after
`ffi_functions` is materialized.

Verified end-to-end against /tmp/repro1110_real — both the direct
import form (user's filing) and the re-export-chain shape from the
codegen half of #1110 now reach `Wrote executable: repro` and the
binary runs.

* fix(fastify) #1113 + chore(ffi) #1112: app.server EventEmitter (boot fix); publish script

FastifyApp handle pointer-tagged so `typeof app.server === "object"`,
and `.on(event, cb)` dispatches through a new `js_fastify_app_on`
extern that stores upgrade callbacks in
`FastifyApp::upgrade_handlers`. GC scanner pins the closures across
cycles. Hyper accept loop emits a 501 + diagnostic when an `Upgrade:`
request arrives against a registered handler. Bidirectional WS
upgrade through hyper (the second half of the user's pattern with
`wss.handleUpgrade(req, socket, head, …)`) is documented as the
tracked follow-up — needs `hyper::upgrade::on(req)` wiring plus a
Node-compatible (req, socket, head) triple back to TS.

Codegen path: `Call { callee: PropertyGet { object: NativeMethodCall
{ module, … }, property: P }, args }` chains are forwarded into the
NATIVE_MODULE_TABLE arm for `(module, P)` whenever the table
recognises `P` as a method of `module`. Pre-PR the property read
returned undefined and `(undefined)(…)` returned NaN — no exception,
no callback registered. Scoped narrowly; falls back to the generic
Call lowering on a miss.

Mirror changes landed in both perry-stdlib's `crate::fastify`
(bundled-fastify path) and perry-ext-fastify (auto-optimize-flipped
path) since either can be live depending on whether the
well-known flip routed `import 'fastify'`.

maintainer to run `cargo publish -p perry-ffi --really-publish`
once they `cargo login`. Dry-run cleanly reaches the Uploading step.

cleanly. Bisect candidates documented in CHANGELOG.

* chore(repro) #1114: bisect harness + investigation notes

Built two synthetic repros for the user's setInterval+MySQL CPU
wedge: a timer-only fastify shape (no DB) and a real-MySQL variant
against a local server (createPool + FOR UPDATE SKIP LOCKED).
Neither triggered the wedge — CPU stayed at 0%, healthz responded
in milliseconds. shop-admin's ~68-server-file shape is required to
trigger it; the minimal reductions don't.

Surveyed the diff between v0.5.1008 and v0.5.1009. The CHANGELOG
only credited the misaligned-pointer Object.assign fix, but
`git log --first-parent 0a90839..c71c780` shows 9 other on-main
commits in that range, all collapsing into the one v0.5.1009 release.
The fastify-flavoured candidates ranked by suspicion: 634e1f5
(non-blocking listen + main-thread pump) > 3856caa (transform
async early-return rewrite) > 5284700 (jsruntime V8-fallback CJS
require cycles) > 7e3bd5a (codegen Effect.succeed narrowing).
634e1f5 is the leading hypothesis given the symptom shape (madvise
hot suggests per-tick allocations; the pump now walks every
FastifyServerHandle on every main-pump tick via
iter_handle_ids_of).

Ships scripts/bisect_1114.sh — walks first-parent commits in the
range, rebuilds runtime+stdlib+perry at each step, runs the user's
binary ($PERRY_REPRO_CMD), and reports the first commit where CPU
exceeds the configured limit. The user can run this against their
actual shop-admin binary to localise the regression in ~15 minutes.

Notes in CHANGELOG. Actual fix needs the bad commit to be confirmed
against the real repro first.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FFI symbols emitted as undefined values in caller IR (compile fails for any importer)

1 participant