fs: improve performance of recursive directory read

[avivkeller]

Currently, when recursively reading a directory, we do a few things which can be considered slow:

1. We round-trip between CPP and JS for each entry, meaning that a 5.4k-entry large directory needs to round-trip 5.4k times, after this PR, that's only one round trip.
2. We stat'd every file previously, and now, we only stat when needed, using the native lstat over the JS version.

Benchmarks:

                                                                                             confidence improvement accuracy (*)    (**)   (***)
fs/bench-readdir-recursive.js withFileTypes='false' mode='callback' dir='lib' n=10                  ***    248.34 %      ±22.14% ±29.62% ±38.87%
fs/bench-readdir-recursive.js withFileTypes='false' mode='callback' dir='test/parallel' n=10        ***    471.49 %      ±45.94% ±61.85% ±81.99%
fs/bench-readdir-recursive.js withFileTypes='false' mode='promise' dir='lib' n=10                   ***    243.07 %      ±25.02% ±33.53% ±44.11%
fs/bench-readdir-recursive.js withFileTypes='false' mode='promise' dir='test/parallel' n=10         ***    485.55 %      ±37.61% ±50.61% ±67.05%
fs/bench-readdir-recursive.js withFileTypes='false' mode='sync' dir='lib' n=10                      ***    188.53 %      ±21.79% ±29.19% ±38.39%
fs/bench-readdir-recursive.js withFileTypes='false' mode='sync' dir='test/parallel' n=10            ***    497.94 %      ±31.68% ±42.61% ±56.39%
fs/bench-readdir-recursive.js withFileTypes='true' mode='callback' dir='lib' n=10                   ***    118.38 %      ±21.14% ±28.27% ±37.08%
fs/bench-readdir-recursive.js withFileTypes='true' mode='callback' dir='test/parallel' n=10         ***    290.39 %      ±21.57% ±28.89% ±37.99%
fs/bench-readdir-recursive.js withFileTypes='true' mode='promise' dir='lib' n=10                    ***     53.59 %      ±14.66% ±19.53% ±25.47%
fs/bench-readdir-recursive.js withFileTypes='true' mode='promise' dir='test/parallel' n=10                   8.72 %       ±9.27% ±12.33% ±16.06%
fs/bench-readdir-recursive.js withFileTypes='true' mode='sync' dir='lib' n=10                       ***     71.40 %      ±14.63% ±19.52% ±25.51%
fs/bench-readdir-recursive.js withFileTypes='true' mode='sync' dir='test/parallel' n=10             ***    269.65 %      ±26.21% ±35.19% ±46.45%

Be aware that when doing many comparisons the risk of a false-positive
result increases. In this case, there are 12 comparisons, you can thus
expect the following amount of false-positive results:
  0.60 false positives, when considering a   5% risk acceptance (*, **, ***),
  0.12 false positives, when considering a   1% risk acceptance (**, ***),
  0.01 false positives, when considering a 0.1% risk acceptance (***)

Also,
Fixes #58892

[nodejs-github-bot]

Review requested:

• @nodejs/performance

[avivkeller]

cc @nodejs/fs

[codecov]

Codecov Report

❌ Patch coverage is 75.05938% with 105 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.01%. Comparing base (05a8e91) to head (cb1f218).
⚠️ Report is 6 commits behind head on main.

Files with missing lines	Patch %	Lines
src/node_file.cc	79.45%	27 Missing and 26 partials ⚠️
lib/internal/fs/utils.js	67.46%	25 Missing and 2 partials ⚠️
lib/internal/fs/promises.js	36.36%	21 Missing ⚠️
lib/fs.js	91.48%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #65524      +/-   ##
==========================================
- Coverage   90.06%   90.01%   -0.06%     
==========================================
  Files         751      751              
  Lines      254919   255226     +307     
  Branches    48124    48171      +47     
==========================================
+ Hits       229603   229748     +145     
- Misses      16492    16623     +131     
- Partials     8824     8855      +31     

Files with missing lines	Coverage Δ
lib/fs.js	97.33% <91.48%> (-1.10%)	⬇️
lib/internal/fs/promises.js	91.32% <36.36%> (-0.98%)	⬇️
lib/internal/fs/utils.js	95.88% <67.46%> (-2.04%)	⬇️
src/node_file.cc	74.53% <79.45%> (+0.39%)	⬆️

... and 43 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[codebytere]

thanks for picking this up! fyi the table in the description is against main before #65487 landed, which already dropped the per-entry stat() and went to one binding call per directory, so most of that delta is gone. against current main (linux x64, means of 5 runs of this PR's benchmark, plus dir=test for a nested tree with 860 dirs / 13k entries) i get:

	main	this PR
promise, lib	362 ops/s	895 (+147 %)
promise, test	25.0	59.9 (+140 %)
sync, lib	850	1561 (+84 %)
sync, test	57.5	65.7 (+14 %)
sync + withFileTypes, lib	930	801 (−14 %)
sync + withFileTypes, test	55.4	56.8 (n.s.)
any mode, test/parallel (flat)		n.s.

so the promises path is the clear win (it awaited one thread pool round trip per directory), sync is a modest win, and withFileTypes sync regresses on small trees, which i think is the marshalling (comment below). could you rerun against current main and update the description please?