Skip to content

Extract the module graphs through a staging directory - #5

Merged
traviswheeler merged 3 commits into
mainfrom
fix_graph_extraction_race
Aug 13, 2026
Merged

Extract the module graphs through a staging directory#5
traviswheeler merged 3 commits into
mainfrom
fix_graph_extraction_race

Conversation

@traviswheeler

Copy link
Copy Markdown
Member

Concurrent BLIMMP processes could not extract the module graphs at the same time. On a
9-genome Nextflow run, 4 of 8 tasks died before any analysis ran:

[FATAL] Failed to extract KEGG_Graphs_Generated_March26.zip. Tried ... and
/tmp/blimmp-cache-62226/KEGG_Graphs_Generated_March26.

Nextflow dispatches one BLIMMP task per genome, so the tasks landed on one node together
and all extracted into the same directory. _extract_module_graphs() extracted in place
and then mutated the result, removing __MACOSX and flattening the nested folder. One
process would delete or move files while another was still reading them. The OSError
that followed looked to the caller like an unwritable destination, so it tried the next
candidate and then reported that every candidate had failed.

Extraction now happens in a private staging directory that is renamed into position. A
rename onto a missing or empty directory is atomic on POSIX, so the first process to
finish wins and the others use its copy instead of failing.

Commits

The first commit adds the test without the fix, so CI records the failure before the fix
lands:

commit Concurrent extraction step
3401ada test only fails, 6 of 8 workers dead with the same FATAL
1efc959 fix all workers pass; a bug in the test's own assertion kept it red
bc68eda assertion passes, 8 workers, 340 graphs, no leftovers

Scope

0.1.4 makes this unreachable in the published image, because the graphs ship pre-extracted
and the extraction path is never entered. This matters for source and wheel installs, and
for any future image that stops pre-extracting. No release is needed to keep the container
behaving correctly.

The existing smoke test runs one process and stayed green through all of it, which is how
the bug shipped. tests/concurrency_test.py runs 8 workers against one directory and
checks that they all exit cleanly, agree on the result, and leave no staging directories
behind.

Nextflow dispatches one BLIMMP task per genome, so several processes
extract the module graphs into the same directory at once. On a
9-genome run, 4 of 8 tasks died with

  [FATAL] Failed to extract KEGG_Graphs_Generated_March26.zip

before running any analysis.

This commit adds the test without the fix, so CI records the failure
first. The existing smoke test runs a single process and stays green
throughout, which is why the bug shipped.
Nextflow dispatches one BLIMMP task per genome, so several processes
extract the module graphs into the same directory at once. Extracting
straight into that directory let them destroy each other's work: one
process would rmtree __MACOSX or flatten the nested folder while
another was still reading from it. The OSError that followed looked to
the caller like an unwritable destination, so it moved on to the next
candidate and then reported that all of them had failed.

Extraction now happens in a private staging directory that is renamed
into position. A rename onto a missing or empty directory is atomic on
POSIX, so the first process to finish wins and the others adopt its
copy instead of failing.

The test added in the previous commit went from 6 of 8 workers dead to
all 8 agreeing on one directory with 340 graphs and no staging
directories left behind.

0.1.4 makes this unreachable in the published image, since the graphs
ship pre-extracted and the extraction path is never entered. It still
runs for anyone installing from source or from a wheel.

Implemented with assistance from Claude (Opus 5)
ensure_module_graphs prints a progress line when it does the extracting,
so the worker that wins the race emits two lines and the others emit
one. Comparing whole stdout blobs made the workers look like they
disagreed when they had all returned the same directory.
@traviswheeler
traviswheeler merged commit 47a2540 into main Aug 13, 2026
2 checks passed
@traviswheeler
traviswheeler deleted the fix_graph_extraction_race branch August 13, 2026 16:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant