Segfault and memory leak on large data creation

[ThomasMalletCodra]

Windows crash in long-running test sessions due to GDI handle exhaustion (suspected object lifetime/cache issue)

Describe the bug

On Windows, PythonQwt may crash with an access violation during long-running test sessions.

The failure appears after extended execution (on a large test run on Windows), which is consistent with Windows GDI handle exhaustion. Windows limits GDI handles to roughly 10,000 per process, and the process seems to exceed that limit over time.

Current analysis suggests the issue is linked to object lifecycle and cache management in PythonQwt, especially for internal private data objects backed by Qt/C++ resources.

Regression might be introduced after optimization : #93

To Reproduce

1. Run a large PythonQwt-dependent test suite on Windows in a single process with large data creation
2. Monitor process GDI handle count over time (for example with Task Manager or Process Explorer).
3. Continue until late-stage execution.
4. Observe continuous handle growth and eventual crash/access violation near the GDI limit.

Ideas

• Reintroduce QObject inheritance for the three affected PrivateData classes to ensure proper destruction and Qt-managed lifetime behavior.
• Remove slots in these classes (suspected to interact poorly with asynchronous deallocation/lifetime expectations).
• Add a defensive cache size limit.

[ThomasMalletCodra]

@dappham-CODRA

I think the right direction is to derive a dedicated PythonQwt non-regression test from the scenario that exposed the issue in DataLab, without keeping any DataLab-specific dependency.

Reference scenario

The DataLab test that revealed the problem is based on a very simple idea: create a large synthetic dataset, turn it into many logical signals, load them through the real GUI path, and let the rendering layer deal with a large amount of object creation and painting work.

In other words, the interesting part is not the CSV import itself, nor the DataLab object model, but the pressure created by:

• large in-memory signal data,
• many graphical objects created from that data,
• real Qt show / paint / event cycles,
• repeated lifecycle operations on widgets/items.

What should be kept vs removed

For a PythonQwt regression test, we should keep the stress pattern but remove everything that belongs to DataLab:

Keep:

• large synthetic datasets,
• many objects created from those datasets,
• real rendering through Qt,
• repeated create/show/processEvents/close/destroy cycles.

Remove:

• CSV generation,
• pandas I/O,
• DataLab SignalPanel,
• DataLab SignalObj / application workflow,
• save/load application logic.

Why the existing PythonQwt loadtests are not enough

We already have useful performance-oriented tools:

• qwt/tests/test_loadtest.py
• scripts/bench_plotpy_loadtest.py
• scripts/lineprofile_loadtest.py

These are very helpful for benchmark/profiling work, especially around issue #93, but they are not really aimed at resource-lifetime regression detection.

Their main question is essentially:

• “how long does construction/rendering take?”

For issue #107, the question is different:

• “does repeated creation/render/destruction stay stable over time on Windows, or do GDI handles / memory usage drift until failure?”

Proposed dedicated regression test

I would suggest adding a dedicated pytest test focused on stability rather than speed.

The test idea would be:

1. Generate signal data directly with NumPy.
2. Create many curves / labels / plot-related objects in PythonQwt.
3. Attach them to real widgets/plots.
4. Call show() and QApplication.processEvents() to force actual rendering.
5. Close and destroy everything.
6. Repeat this cycle enough times to expose leaks or lifetime issues.

This keeps the scenario close to what exposed the bug originally, while making it fully PythonQwt-native.

So I would consider such a script optional, not the primary deliverable.

Optional VS Code task

It may also make sense to add a dedicated VS Code task for manual non-regression runs, for example to launch:

• the dedicated leak/stability test,
• or a longer stress/benchmark script for Windows validation.

I would keep that as a manual or opt-in task, not part of the default fast CI path, because:

• these tests may be slower,
• they may be more environment-sensitive,
• and benchmark-like checks tend to be noisy on shared runners.

Summary

So the proposal would be:

• add a dedicated PythonQwt non-regression test inspired by the DataLab large-signal scenario,
• keep it independent from DataLab,
• focus on repeated object creation / rendering / destruction,
• keep any extra stress script optional,
• and optionally expose it through a manual VS Code task for Windows validation.

This would complement the current performance loadtests with an actual regression guard for the segfault / memory leak / GDI exhaustion class of issues.

[PierreRaybaut]

Problem statement

On Windows, long-running, GUI-heavy sessions could crash with an access violation (0xC0000005). The pattern was consistent with GDI/USER handle exhaustion (Windows caps GDI handles at ~10,000 per process), suspected to come from object lifetime / cache management of PythonQwt's internal *_PrivateData objects backed by Qt/C++ resources — possibly a regression after the optimization in #93.

The bug originally surfaced through a DataLab scenario (large synthetic signals → many graphical objects → real Qt show/paint cycles → repeated create/destroy).

Two distinct phenomena (important)

The DataLab investigation that fed this issue actually separated two problems that were easy to conflate:

1. A DataLab-side native teardown crash (test_import_wizard). This was a deterministic-teardown bug in DataLab, not in PythonQwt:

   • It always crashed at the same test, in native (on-screen) mode only, with no faulthandler traceback (pure native C++ teardown).
   • It was version-independent: a controlled native matrix showed PythonQwt v0.15.0 ≡ 0.16.2 (same sporadic crash rate, GDI bounded ~88–251). The "regression introduced between 0.15.0 and 0.16.0" hypothesis was therefore not confirmed for this crash.
   • Root cause: several TextImportWizard instances (each embedding a heavy PlotPy preview) were destroyed lazily by the GC, racing in Qt/PlotPy teardown.
   • Fixed on the DataLab side by destroying each wizard deterministically (close() + deleteLater() + processEvents(), plus a cleanup()/closeEvent releasing the plot items and disconnecting signals). Controlled repro: 4/10 crashes before → 0/6 after at identical load.

2. The GDI/USER handle-lifetime class addressed by this issue (Segfault and memory leak on large data creation #107). This is the genuinely PythonQwt-side concern: when many widgets/items are created/shown/destroyed rapidly, Windows handles must be released promptly. In a deliberately over-stressed harness (hundreds of top-level plots shown in a tight loop) the process did hit CreateWindowEx failed (handle-cap exhaustion). PythonQwt 0.16.2 already includes the weakref / font-cache GDI fixes; Fix/segment fault #108 targets the remaining *_PrivateData lifetime path.

Take-away for #107: the PythonQwt fix and its regression test should focus on resource-lifetime stability (handles/memory staying flat over repeated create/render/destroy), independently of the DataLab-specific crash, which is already resolved upstream of PythonQwt.

Fix — PR #108 (merged)

The object-lifetime / cache fix for the affected *_PrivateData classes landed in PR #108 (merged), shipped in PythonQwt 0.16.2 together with the earlier weakref / font-cache GDI fixes. No further code change is needed here.

Dedicated regression test — decision: deferred

Decision (2026-06-21): we will NOT add a dedicated load/stability test now. It will be created only if the problem reproduces. Rationale: the DataLab-specific crash is already fixed upstream of PythonQwt; in the DataLab matrix the GDI count stayed bounded (no drift was observed against 0.16.2, which equals 0.15.0); and benchmark-like stability checks are slow and noisy. Adding a guard with no reproducible failing case would mostly add CI cost and flakiness.

If #107 (or a similar GDI/segfault report) does reproduce, the reference design below should be used to build a PythonQwt-native stability test (stability over time, not speed — the existing test_loadtest.py, scripts/bench_plotpy_loadtest.py, scripts/lineprofile_loadtest.py answer "how fast?", not "does it drift?").

Reference design (only if the problem reproduces; pytest, opt-in / slow-marked, Windows-relevant):

1. Generate signal data directly with NumPy (no CSV, no pandas, no DataLab).
2. Create many curves / labels / plot items in PythonQwt and attach them to real plots/widgets.
3. show() + QApplication.processEvents() to force real rendering.
4. Close and destroy everything deterministically.
5. Repeat the cycle enough times to expose drift.
6. Assert stability: sample the GDI handle count (Windows, via GetGuiResources) and/or process working set across cycles and assert it stays bounded (no monotonic growth beyond a tolerance). On non-Windows, fall back to a memory/object-count check and skip the GDI assertion.

Notes:

• Keep it independent from DataLab (no SignalObj, SignalPanel, app workflow).
• Keep it out of the default fast CI (slower, environment-sensitive, noisy on shared runners); expose it via a manual / opt-in VS Code task for Windows validation.
• A handy reference implementation for the GDI sampling already exists on the DataLab side (scripts/pytest_gdi_probe.py, GetGuiResources(GR_GDIOBJECTS)), which can be mirrored as a small PythonQwt-native helper.

Verification checklist before closing

• PR Fix/segment fault #108 merged (object-lifetime / cache fix).
• CHANGELOG updated; release note mentions the GDI/lifetime fix.
• Dedicated load/stability test deferred — to be added only if the problem reproduces (see decision above).
• (If it reproduces later) add the PythonQwt-native stability test + optional manual VS Code task, and confirm it fails on pre-fix code.

Cross-references

• DataLab-side teardown fix (separate, already merged): deterministic destruction of the Text Import Wizard's embedded PlotPy preview.
• Native matrix result exonerating the 0.15.0→0.16.0 regression hypothesis for the DataLab crash (v0.15.0 ≡ 0.16.2).
• Related optimization: Performance degradation with Qt6 #93. Closing PR: Fix/segment fault #108.