Read BrainVision data in cache-sized blocks - #14241
Merged
larsoner merged 2 commits intoAug 28, 2026
Merged
Conversation
_read_segments_file read up to 100 MB at a time. Letting a caller choose a smaller budget and having BrainVision ask for 8 MiB keeps the working set in cache: preloading a 101 MB .eeg (32 channels, 404 MB decoded) goes from 61.5-63.4 ms to 54.0-55.3 ms, min of 5 per process over 3 interleaved A/B rounds against a pristine main worktree. The default is unchanged, so the other eight callers of _read_segments_file read exactly as before. Output is bit-identical on all 13 BrainVision fixtures plus the 101 MB file. block_size is also floored at one channel frame, since a budget smaller than a frame would otherwise leave it at zero and the read would not advance.
bruAristimunha
requested review from
agramfort,
drammock,
larsoner and
sappelhoff
as code owners
August 28, 2026 13:28
larsoner
approved these changes
Aug 28, 2026
larsoner
left a comment
Member
There was a problem hiding this comment.
Looks great, thanks @bruAristimunha !
Member
|
(also would be happy to see follow-up PRs for other formats, too, especially if they are data formats you have multiple instances of where you can check across files etc.) |
larsoner
enabled auto-merge (squash)
August 28, 2026 14:14
This was referenced Aug 28, 2026
bruAristimunha
added a commit
to bruAristimunha/mne-python
that referenced
this pull request
Aug 30, 2026
The reader asked for up to 100 MB per block, then cast the whole block to
float64 and scaled it in place, so the working set was several hundred MB and
none of it stayed in cache.
raw.get_data(), interleaved cross-process medians:
136 MB file (161ch x 425000) 258.5 -> 145.6 ms 1.78x
A sweep over 32/16/8/4/2/1 MiB puts the optimum at 2 MiB (32 MiB is only 1.14x,
8 MiB 1.38x, 4 MiB 1.66x, 1 MiB 1.60x).
No shipped fixture regresses; two of them improve because they were already
being split differently:
ArtificalSignalData_Yokogawa_1khz.con 2.87 -> 2.26 ms 1.27x
ArtificalSignalData_RICOH_1khz.con 3.06 -> 2.77 ms 1.10x
Example_PQA160C_1001-export.con 0.99x
010409_Motor_task_coregist-export.con 0.98x
Output is bit-identical on every readable KIT fixture. The large file was
synthesised by tiling the raw-data section of data_berlin.con -- it is the last
section in the file -- and patching n_samples in the acquisition header.
Same treatment as mne-tools#14241 and mne-tools#14246; the constant differs because the optimum
tracks time points per block, so it moves with the channel count.
larsoner
added a commit
to larsoner/mne-python
that referenced
this pull request
Sep 2, 2026
* upstream/main: (22 commits) Avoid copying all epochs data in GetEpochsMixin._getitem (mne-tools#14262) Add Report.save(only_if_changed=True) (mne-tools#14261) Add jamica to related software [ci skip] (mne-tools#14260) Remove debugging cruft (mne-tools#14259) ENH: add Raw annotation span conversion (mne-tools#14240) Interactive dipole fitting: add STC mesh controls (mne-tools#14256) Document code principles in AGENTS.md (mne-tools#14239) MAINT: Update dependency specifiers (mne-tools#14257) [dependabot]: Bump the actions group with 2 updates (mne-tools#14258) ENH: Add JAMICA as an ICA method (mne-tools#14247) Reuse the MEF session across reads (mne-tools#14254) Read KIT data in cache-sized blocks (mne-tools#14255) Read EGI simple-binary event channels in blocks (mne-tools#14250) Decode Persyst and Nihon Kohden data in cache-sized blocks (mne-tools#14251) Normalize byte order before calibrating strided integer buffers (mne-tools#14252) Speed up EDF and BDF reading (mne-tools#14237) ENH: Add Forward-based projection reconstruction (mne-tools#14235) Read Artemis123, Curry, EEGLAB, Eximia, FIL and NSx in cache-sized blocks (mne-tools#14246) Remove rotating dipoles capability from interactive dipole fitting GUI (mne-tools#14243) Read BrainVision data in cache-sized blocks [ci skip] (mne-tools#14241) ...
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reference issue (if any)
None. Split out of #14237, which was too large to review as one change.
What does this implement/fix?
_read_segments_fileread up to 100 MB of a binary file at a time. Letting acaller pick a smaller budget, and having BrainVision ask for 8 MiB, keeps the
working set in cache.
Preloading a 101 MB
.eeg(32 channels, 404 MB decoded, built by repeating thetest.eegpayload), warm page cache, min of 5 per process, 3 interleaved A/Brounds against a pristine
mainworktree on the same machine:That is ~1.15x, and the rounds do not overlap.
The default stays
int(100e6), so the other eight callers of_read_segments_file(EGI, Curry, Artemis123, Nicolet, Eximia, EEGLAB, FIL,NSx) read exactly as before — only BrainVision opts in.
block_sizeis also floored at one complete channel frame. A budget smallerthan a frame would otherwise round it to zero and the read loop would not
advance.
Correctness
Output is bit-identical to
mainon all 13 BrainVision fixtures in the repoplus the 101 MB file, compared with
np.array_equal.The new test reads the same file twice with very different block budgets and
asserts the data matches. Both budgets are deliberately not whole numbers of
channel frames, so a block size that stopped landing on frame boundaries is
caught; the sub-frame case pins the floor described above. Mutation-tested:
making blocks frame-unaligned and adding an off-by-one to the block size are
both caught.
Additional information
AI disclosure: I directed the work and reviewed and tested everything; Claude
Code (Claude Opus 5) made the code edits, built the benchmark fixture and ran
the A/B measurements under my direction.