Give the worker the same NDJSON frame limit as the main process

[webard]

A job chunk larger than 64 KB is dropped on arrival at the worker. The run then reports success, having analysed almost nothing:

$ vendor/bin/rector process --dry-run   # withParallel(jobSize: 600)
 [OK] Rector is done!
$ echo $?
0

Nine seconds, empty stderr, exit code 0 - and 6 files analysed out of 12 492. Counted by the entries written to the cache directory, because the console output gives no way to tell.

Root cause

The two ends of the connection disagree about how large a frame may be.

src/Parallel/Application/ParallelFileProcessor.php (main process) asks for 4 MB:

$inDecoder = new Decoder($connection, true, 512, 0, 4 * 1024 * 1024);

src/Console/Command/WorkerCommand.php (worker) omits $maxlength and gets the clue/ndjson-react default of 64 KB:

$inDecoder = new Decoder($connection, true, 512, JSON_INVALID_UTF8_IGNORE);

A job frame carries one chunk of absolute file paths, so its size follows path length, not file count. The ceiling is therefore project-specific and moves on its own: a deeper directory tree, a longer checkout prefix (a git worktree sits some forty characters below the main clone) or simply more files per directory all lower it.

Reproduction

12 762 files, 14 cores, PHP 8.5.10, average absolute path length 115 characters. Each timing repeated in both orders to rule out page-cache effects.

jobSize	wall time	files actually analysed
16 (default)	461 s / 468 s	12 492
150	124 s / 134 s	12 494
300	120 s / 116 s	12 492
600	9 s, [OK] Rector is done!, exit 0	6

The cliff is arithmetic, not load-dependent. The worst 600-path window in this project encodes to 64 868 B against the 65 536 B default - 1.01×, right at the edge, which is why 300 passes and 600 does not. Raising timeoutSeconds to 900 changes nothing (still 6 files), so it is not the job timeout.

To reproduce on any project: pick jobSize such that jobSize × (average absolute path length + 3) exceeds 65 536, then compare the number of cache entries written against the file count.

Why this is not just "pick a smaller jobSize"

Two reasons.

First, there is no signal. The run is green, stderr is empty, and the progress bar reaches 100%. #8894 reported this same symptom in 2024 - "Rectorphp silently exits before completing the run", exit code 0, tied by the reporter to jobSize - and was closed the same day for lack of a reproducible case. The concern raised there still stands: a CI pipeline treats that exit code as proof the code was refactored.

Second, the default jobSize of 16 leaves a lot on the table - 461 s against 124 s in the table above, for identical work - so raising it is a reasonable thing for a user to try, and today that is a loaded gun.

Precedent

Rector's parallel scheduling is derived from PHPStan (ScheduleFactory says so). PHPStan sets the limit on both ends from a single configurable value:

• ParallelAnalyser (main): maxlength: $this->decoderBufferSize
• WorkerRunner (worker): maxlength: $this->decoderBufferSize
• conf/config.neon: parallel: buffer: 134217728 # 128 MB

Rector's worker limit is currently 64× smaller than its own main process and 2048× smaller than PHPStan's default.

[samsonasik]

Looks good to me 👍

[TomasVotruba]

I'm not sure I understand this change.
Could you put in more simpler words?