Openfn compile for unit tests

[mtuchi]

Short Description

Update openfn compile to compiles workflows job expressions to standard JavaScript, writing output to compiled/ by default.

Fixes #1424

Implementation Details

• openfn compile now compiles all workflows in the project to compiled/ by default (no flag needed)

• openfn compile <workflow-name> looks up a workflow by name in the project and compiles it to compiled/ use -O for stdout or -o <dir> to redirect output

• openfn compile <file.js> and openfn compile <workflow.json> retain their existing behaviour (stdout by default)

• Added new exports-only transformer (src/transforms/exports-only.ts) with a single responsibility: keep imports, named export declarations, and their transitive non-exported dependencies; drop everything else (operations, export default, bare declarations). Runs at order: 0 (before
  all other transformers). Is a no-op unless explicitly enabled.

• Added --exports-only flag: strips adaptor operation calls and keeps only exported declarations — intended for unit testing job code

QA Notes

Setup

Check out the 1424-unit-tests branch and install openfnx:

pnpm install:openfnx

Verify the correct version is installed — openfnx --version should output branch/1424-unit-tests.

Testing openfn compile

Navigate to a local OpenFn project containing an openfn.yaml file and test the new openfn compile command. Available options:

# Compile all workflows in the project (full compilation, preserves operations)
openfn compile

# Compile all workflows, stripping operation calls (useful for unit testing)
openfn compile --exports-only

# Compile a single workflow by name
openfn compile my-workflow

# Compile a single workflow to a custom directory
openfn compile my-workflow -o tests/

# Compile a single job expression (prints to stdout)
openfn compile workflows/my-workflow/step-a.js --exports-only

# Watch mode: recompile whenever source files change
openfn compile --exports-only --watch

AI Usage

Please disclose whether you've used AI anywhere in this PR (it's cool, we just
want to know!):

• I have used Claude Code
• I have used another model
• I have not used AI

You can read more details in our
Responsible AI Policy

[josephjclark]

@mtuchi I'm not liking the --tests argument. What we really want here is just to easily compile some or all of a project to disk. Stripping the operations is an optional step. As mentioned in the issue I'd call that compile --exports-only.

When you do openfn compile workflow.json today, it'll compile that workflow and print the result to stdout.

When you do openfn compile my-workflow, what will happen? It should lookup the workflow called my-workflow in the project, compile that, and print it to stdout. It doesn't, but that would be consistent.

When you do openfn compile it should compile the whole project. It can't print that to stdout because it'll compiling multiple files. But it can compile to dist and write all the files there. That is what I expect the default behaviour to be.

I'd also suggest that openfn compile my-workflow should compile to disk by default (the user can pass -O to write to stdout). That's a little inconsistent with openfn compile workflow.yaml but it IS consistent with compiling through the project. So I'm OK with it. Maybe later, in CLI 2.0, we'll always compile to disk for consistency. But since this stuff is very rarely used I wouldn't worry too much.

So drop the --tests flag please and just make openfn compile compile a project to a folder on disk.

[mtuchi]

QA walk through video for @hunterachieng
https://www.loom.com/share/033dc7f75c5642a1b5e8edb8a9a35f46

[mtuchi]

Unit test are failing, even on main. I am not sure what's causing it, i will investigate more tomorrow

[josephjclark]

@mtuchi the tests failing here in CI are probably just flaky ones. I'll re-run them now. Locally it's probably most important that your compiler tests are passing. Sometimes my CLI tests fail locally because of caches stuff (they take so long to test I don't tend to run the whole suite locally - just the bit I'm working on)

[josephjclark]

sorry @mtuchi I don't think I'm going to get to this this week. I've not forgotten, just couldn't get to it yet. It'll be high priority early next week.

[josephjclark]

Hi @mtuchi, I'm so sorry for the slow reaction on this.

I've been through and tested and it all works great. I've picked out a few things that need addressing - mostly style refactors but also a couple of important details.

If you don't have time to resolve this I may be able to pick it up at the end of the week.

[josephjclark]

We need to add the workspace command now: this sets the working directory for the current project. --workspace is used by openfn execute and openfn project.

Very very useful for local testing!

[josephjclark]

The logic here feels weird. workflowName feels too important, and compileProject(options, logger) is duplicated, which is a smell.

A cleaner entrypoint for this stuff is good. Basically if the path does not end with a file extension, we should compileProject

[josephjclark]

I would rather compile to dist than compiled. That's more likely to be tracked by a gitignore file too.

[josephjclark]

also wsConfig.dirs.compiled feels like a new key in workspace config. It should be defined in lexicon/core.d.ts:WorkspaceConfig

[josephjclark]

It's always bothered me that job.js and job.compiled.js have the same file extension. For years I'd wondered if we should be using job.ojs for job expressions - but that ship has long since sailed.

As a compromise, can we output compiled files to job.mjs? That's more standard for modern Javascript esm modules, and it helps to distunguish a bit the file types

[josephjclark]

This would break with export { x, y, z } no?

Not a fan of this. What purpose does it solve? Feels like we're over-optimising

[josephjclark]

This comment has no value. AI is terrible fof this sort of thing - please try and be strict on it. Only keep stuff with value

[josephjclark]

I don't like logic inside an assertion. A better pattern is:

const before = `fn();
export default [];`;
const after = compile(before)
  t.is(after, before);

[josephjclark]

In terms of storytelling through tests, rather than testing the inverse case first, I'd rather test basic functionality:

strips a non exported value
is a no-op if exports-only is false

Where the "before" code in both tests is exactly the same

[josephjclark]

Actually come to think of it, I don't like the use of compile here

The other transformer functions just test the transform functoin with visitors passed in.

This calls the top level compile function, so it's testing all sorts of logic before even getting to the transformer.

Ask the AI to rewrite these tests against the transform function, not the compile function. use the parse function with source code code, like in lazy-state.test.ts, rather than the the AST builders in add-imports.test.ts