Skip to content

馃悰 fix(sqlite): isolate forked connections - #654

Closed
gaborbernat wants to merge 1 commit into
tox-dev:fix/634-fork-ownershipfrom
gaborbernat:fix/635-sqlite-fork
Closed

gaborbernat wants to merge 1 commit into
tox-dev:fix/634-fork-ownershipfrom
gaborbernat:fix/635-sqlite-fork

Conversation

@gaborbernat

@gaborbernat gaborbernat commented Jul 14, 2026 •

Copy link
Copy Markdown
Member

SQLite forbids a process from using or closing a database connection inherited through fork(). Read-write locks kept
one connection for the object's lifetime, so a child could close its parent's handle during interpreter shutdown and
corrupt the database. Each outer acquisition now owns its connection, and a forked child abandons inherited handles
according to
SQLite's fork guidance.
Closes #635.

The child rejects an active database by canonical path or inode until exec(). CPython 3.10 and 3.11 keep inherited
base connections alive through a raw-reference escrow; later CPython versions use the guarded subclass finalizer. The
implementation follows the
CPython 3.10 _sqlite lifecycle and
current CPython source. A child forked from
a SQLite callback waits for the call to leave SQLite, or exits with status 70 if an earlier audit hook prevents
coordination. Fork registration uses a reentrant gate because connection finalizers can register another lock while the
current thread owns that gate.

PyPy can retain unsafe process-wide SQLite state after tracked connections close. A PyPy child rejects read-write locks
when the parent used SQLite after importing filelock; importing filelock in the child cannot detect earlier parent use.
The public acquisition interfaces stay unchanged. Final release closes the per-acquisition connection. A local
2,000-acquisition benchmark measured 5.7 microseconds for the former persistent connection and 45.6 microseconds for the
per-acquisition design.

Open SQLite connections for each outer acquisition and abandon inherited handles
without calling SQLite in the child. Reset caches and executors after fork,
preserve parent ownership, and reject child reuse until exec.

Use a reentrant fork-transition gate because connection finalizers can re-enter
registration while the current thread owns the gate.

Refs tox-dev#635
@gaborbernat
gaborbernat force-pushed the fix/635-sqlite-fork branch from c877a37 to acb5754 Compare July 14, 2026 10:40
@gaborbernat
gaborbernat deleted the branch tox-dev:fix/634-fork-ownership July 14, 2026 13:43
gaborbernat added a commit that referenced this pull request Jul 14, 2026
SQLite forbids a process from using or closing a database connection
inherited through `fork()`. Read-write locks kept one connection for the
object's lifetime, so a child could close its parent's handle during
interpreter shutdown and corrupt the database. Each outer acquisition
now owns its connection, and a forked child abandons inherited handles
according to [SQLite's fork
guidance](https://sqlite.org/howtocorrupt.html#_carrying_an_open_database_connection_across_a_fork_).
Closes #635.

The child rejects an active database by canonical path or inode until
`exec()`. CPython 3.10 and 3.11 keep inherited base connections alive
through a raw-reference escrow; later CPython versions use the guarded
subclass finalizer. The implementation follows the [CPython 3.10
`_sqlite`
lifecycle](https://github.com/python/cpython/blob/v3.10.20/Modules/_sqlite/connection.c)
and [current CPython
source](https://github.com/python/cpython/blob/main/Modules/_sqlite/connection.c).
A child forked from a SQLite callback waits for the call to leave
SQLite, or exits with status 70 if an earlier audit hook prevents
coordination. Fork registration uses a reentrant gate because connection
finalizers can register another lock while the current thread owns that
gate.

PyPy can retain unsafe process-wide SQLite state after tracked
connections close. A PyPy child rejects read-write locks when the parent
used SQLite after importing filelock; importing filelock in the child
cannot detect earlier parent use. The public acquisition interfaces stay
unchanged. Final release closes the per-acquisition connection. A local
2,000-acquisition benchmark measured 5.7 microseconds for the former
persistent connection and 45.6 microseconds for the per-acquisition
design.

This replaces #654, which GitHub closed after merging #653 deleted its
base branch `fix/634-fork-ownership`. The branch now sits on `main` and
carries the same work.

It also carries the `close_failure` fixture change from #656, without
which the `pypy3.11` job fails here. Landing #656 first drops that
commit from this branch on the next rebase.
gaborbernat added a commit that referenced this pull request Jul 14, 2026
The `close_failure` fixture patches `filelock._api.os.close`, and
`filelock._api.os` is the `os` module, so mocking it swaps `os.close`
for the whole process. While a test holds the mock, any `os.close` in
the process raises `OSError`, including the ones a garbage collector
runs inside a finalizer.

CPython hides this. Refcounting reclaims lock objects as soon as a test
drops them, outside the window the mock covers. PyPy reclaims them
whenever a cycle runs, so a collection can land inside that window.
`BaseFileLock.__del__` then force-releases a lock an earlier test left
held, its close hits the mock, and the `OSError` escapes the finalizer
as an unraisable exception. Pytest reports it against whichever test
happens to be running, which is how the `pypy3.11` job on #654 failed
inside `test_context_group_handles_deep_body_context`, a test that
touches none of this.

The mock now raises for the descriptor the test captured through
`on_acquired` and hands every other descriptor to the real `os.close`.
The lock under test still fails to close, so the `close_error_policy`
cases keep their meaning, and a collector running a finalizer mid-test
stops tripping over an unrelated fd.

This also drops `docs/changelog/coverage-path-remapping.bugfix.rst`.
#651 changed how CI maps coverage paths across tox environments, which
no user of the library can observe, and the slug filename does not
render through the `:pr:` role that `issue_format` applies to every
fragment.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant