FEAT: Add remove_seeds_from_memory_async to memory by blahdeblahde · Pull Request #2243 · microsoft/PyRIT

blahdeblahde · 2026-07-21T22:22:57Z

Description

PyRIT memory can add seed prompt datasets (via add_seed_datasets_to_memory_async / add_seeds_to_memory_async) and query them with get_seeds, but there is no counterpart for removing seeds. Today users have to drop into raw SQL, which is error-prone and differs across the supported backends (SQLite, Azure SQL).

This PR adds two removal methods that accept the same filter parameters as get_seeds (dataset_name, dataset_name_pattern, added_by, harm_categories, authors, groups, source, value_sha256, data_types, seed_type, parameters, metadata, prompt_group_ids), plus a value_exact parameter for safe exact-match deletion:

remove_seeds_from_memory — removes the individual seeds that match the filters and returns the count removed.
remove_seed_groups_from_memory — removes entire seed groups: any match is expanded to its whole prompt_group_id so partial groups are never left behind (e.g. filtering by one modality removes the whole multimodal group).

Recommended workflow — preview, then delete with the same filters:

# See what matches before deleting
to_delete = memory.get_seeds(dataset_name="illegal")

# Remove them; returns the count deleted
removed = memory.remove_seeds_from_memory(dataset_name="illegal")

# Or remove whole groups (preserves group integrity)
removed_groups = memory.remove_seed_groups_from_memory(dataset_name="illegal")

Implementation

Extracted the filter-building logic out of get_seeds into a shared private helper (_build_seed_filter_conditions) so query and removal build conditions from a single source and cannot drift.
Both methods delete in a single transaction using the SQLAlchemy ORM (rollback on error), so they behave consistently across SQLite and Azure SQL.
remove_seed_groups_from_memory expands matches to whole groups with a server-side subquery (prompt_group_id.in_(select(...))) rather than materializing the group IDs into Python and sending them back as an IN (...) list. This avoids exceeding the backend bound-parameter limit (e.g. Azure SQL) on broad filters. Seeds with a NULL prompt_group_id (e.g. added individually rather than as a group) are excluded and skipped — use remove_seeds_from_memory for those.
The methods are synchronous, mirroring get_seeds: they perform only database work and never await I/O, so making them async would just wrap blocking calls in a coroutine and stall the event loop.

Safety

At least one filter must be provided; a no-filter call raises ValueError to prevent accidentally wiping the entire seed database.
value matches by substring, which is a footgun for deletion (a short/common value can over-match). Use the new value_exact parameter for full-string equality when deleting by value; reserve substring value for previewing with get_seeds. The docstrings and docs call this out, along with the group-integrity consequences of deleting individual seeds.
Only database records are removed. For file-backed seeds (image_path, audio_path, video_path) the serialized file on disk is left in place; this is documented so callers can delete those files separately if needed.

Tests and Documentation

Tests — tests/unit/memory/memory_interface/test_interface_remove_seeds.py covers, for remove_seeds_from_memory: every individual filter (dataset_name, dataset_name_pattern, added_by, source, harm_categories, authors, groups, parameters, metadata, data_types, seed_type, value_sha256, prompt_group_ids), broad value substring vs narrow value_exact, multi-filter narrowing, no-filter ValueError, zero-match, and rollback-on-error; and for remove_seed_groups_from_memory: whole-group removal, groups spanning datasets, non-matching groups preserved, ungrouped (NULL group id) matches skipped, no-filter ValueError, zero-match, and rollback-on-error. File-backed deletion semantics are documented rather than unit-tested (testing real serialized-file deletion is brittle); happy to add an explicit file-cleanup implementation + test if preferred.

Documentation — added a "Removing Seeds from the Database" section to doc/code/memory/8_seed_database.py (and the synced .ipynb) demonstrating the preview-then-remove workflow, whole-group removal, the value vs value_exact distinction, the ungrouped-seed skip, and the file-backed caveat.

Copilot

Pull request overview

This PR adds first-class support for deleting seed prompts from PyRIT memory by introducing remove_seeds_from_memory_async, mirroring the existing get_seeds filter surface so users no longer need backend-specific SQL for seed cleanup.

Changes:

Extracts seed filter construction into a shared private helper (_build_seed_filter_conditions) used by both seed querying and deletion.
Adds remove_seeds_from_memory_async to delete matching SeedEntry rows transactionally and return the number removed (with a no-filter safety guard).
Adds unit tests and documentation demonstrating a “preview via get_seeds, then delete via remove_seeds_from_memory_async” workflow.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 13 comments.

File	Description
`pyrit/memory/memory_interface.py`	Adds shared seed-filter builder and new async seed removal API implemented via SQLAlchemy.
`tests/unit/memory/memory_interface/test_interface_remove_seeds.py`	Adds unit coverage for common filter cases and safety behavior.
`doc/code/memory/8_seed_database.py`	Documents seed removal usage and recommended preview-then-delete workflow.
`doc/code/memory/8_seed_database.ipynb`	Syncs the same documentation updates into the notebook format.

+@pytest.mark.asyncio
+async def test_remove_seeds_by_dataset_name_pattern(sqlite_instance: MemoryInterface):


+@pytest.mark.asyncio
+async def test_remove_seeds_by_added_by(sqlite_instance: MemoryInterface):


+@pytest.mark.asyncio
+async def test_remove_seeds_multi_filter_narrowing(sqlite_instance: MemoryInterface):


+@pytest.mark.asyncio
+async def test_remove_seeds_no_filter_raises_value_error(sqlite_instance: MemoryInterface):


+        remove_seeds_from_memory_async stay in sync and cannot drift.

        Args:
            value (str): The value to match by substring. If None, all values are returned.


+
+        Args:
+            value (str): The value to match by substring. If None, all values are returned.
+            value_sha256 (str): The SHA256 hash of the value to match. If None, all values are returned.


+            prompt_group_ids (Sequence[uuid.UUID]): A list of prompt group IDs to filter by.
+
+        Returns:
+            Sequence[SeedPrompt]: A list of prompts matching the criteria.


+
+        Args:
+            value (str): The value to match by substring. If None, all values are considered.
+            value_sha256 (str): The SHA256 hash of the value to match. If None, all values are considered.


+                query = session.query(SeedEntry).filter(and_(*conditions))
+                count = query.delete(synchronize_session="fetch")
+                session.commit()


blahdeblahde · 2026-07-21T22:38:57Z

Thanks for the review! Addressed all points in the latest push:

Removed the @pytest.mark.asyncio decorators from the new tests (asyncio_mode = auto is set globally).
Fixed the value_sha256 docstring type to Sequence[str] | None in the shared helper, get_seeds, and remove_seeds_from_memory_async.
Corrected the get_seeds return docstring to Sequence[Seed] (not just SeedPrompt).
Changed the delete to synchronize_session=False to avoid the extra SELECT on a short-lived session.

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

blahdeblahde · 2026-07-21T22:47:17Z

@microsoft-github-policy-service agree company="Microsoft"

hannahwestra25 · 2026-07-22T18:24:54Z

+        with closing(self.get_session()) as session:
+            try:
+                query = session.query(SeedEntry).filter(and_(*conditions))
+                count = query.delete(synchronize_session=False)


What are we doing to file-backed image, audio, and video seeds? This removes the database row but leaves the serialized file behind. We should either clean up owned artifacts or clearly document that this only removes database records.

blahdeblahde · 2026-07-29T16:35:06Z

@hannahwestra25 Thanks again for the review — I've pushed updates addressing all your comments:

DuckDB wording / helper return type — fixed.
Async vs sync — converted remove_seeds_from_memory to synchronous (it only does DB work, no awaited I/O, so async would just stall the event loop); rationale in the thread above.
Whole-group removal (Add poetry attributes #5) — added remove_seed_groups_from_memory, which expands any filter match to its full prompt_group_id so partial groups are never left behind. Unit tests included.
value substring deletion danger (Upgrade transformers to >=4.36.0 to address dependabot alert #4) — documented the over-match risk with preview-first guidance, and proposed a safe exact-match deletion mode (value_exact=) as a separate follow-up ticket, since value matching lives in the shared filter helper used by get_seeds/get_seed_groups.
Deletion footguns (Fixed broken link to notebook #7) — documented in the "Removing entire groups" section of the seed-database notebook.

The PR description has been updated with the new method and a "Follow-up work" section. Let me know if you'd like the exact-match mode folded into this PR instead of a follow-up.

Adds an API to remove seed prompts from memory using the same filter parameters as get_seeds (dataset_name, dataset_name_pattern, added_by, harm_categories, authors, groups, source, value_sha256, etc.). Implementation: - Extract filter-building logic from get_seeds into shared _build_seed_filter_conditions helper to prevent drift - Add remove_seeds_from_memory_async that uses the shared helper - Require at least one filter (raises ValueError if none provided) - Single transaction with rollback on failure - Works across DuckDB, SQLite, Azure SQL (uses SQLAlchemy ORM) Unit tests cover: exact name filter, LIKE pattern, multi-filter narrowing, no-filter ValueError, zero-match return, harm_categories, and source filter. Docs: add a 'Removing Seeds from the Database' section to the seed database notebook (8_seed_database.py/.ipynb) demonstrating the preview-then-remove workflow. Resolves AB#5688 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: f9e3cc10-5c77-4384-92d9-d27dad9baea2

hannahwestra25 · 2026-07-31T14:59:39Z

        if value:
            conditions.append(SeedEntry.value.contains(value))
+        if value_exact:
+            conditions.append(SeedEntry.value == value_exact)


imo we should have a boolean variable for this. I can't think of a case where users would input value and value_exact which should be treated differently. if value_exact is true, we use == value otherwise contains(value). I'm thinking we should maintain the get_seeds behavior so value_exact would default to false for that one but delete_seeds would be true (althougth we should be explicit in docs since that is maybe unexpected inconsistency between the two functions).

note: that i'm assuming get_seeds and delete_seeds share the same parameters (you have a comment above which says so but the current implementation contradicts that) which i think makes sense because exact matching could be useful for getting seeds

hannahwestra25 · 2026-07-31T15:00:11Z

-            value_sha256 (str): The SHA256 hash of the value to match. If None, all values are returned.
+            value_exact (str): The value to match exactly (full-string equality). Unlike ``value``,
+                this does not substring-match, so it is the safer choice when deleting by value.
+                If None, no exact-value filter is applied.


nit: to match other descriptions:

Suggested change

If None, no exact-value filter is applied.

If None, all values are returned.

hannahwestra25 · 2026-07-31T15:01:39Z

+# %% [markdown]
+# ## Removing Seeds from the Database
+#
+# Just as you can add and query seeds, you can remove them using `remove_seeds_from_memory`. It accepts the same filter parameters as `get_seeds`, so the recommended workflow is to preview the matching seeds with `get_seeds(...)` first, then remove them with the same filters. The method returns the number of seeds removed.


in the current implementation the filter parameters aren't the same

# Just as you can add and query seeds, you can remove them using `remove_seeds_from_memory`. It accepts the same filter parameters as `get_seeds`, so the recommended workflow is to preview the matching seeds with `get_seeds(...)` first, then remove them with the same filters. The method returns the number of seeds removed.

hannahwestra25 · 2026-07-31T15:03:41Z

+        Remove seeds matching the specified filters from memory.
+
+        Accepts the same filter parameters as get_seeds. It is recommended to call get_seeds with the same
+        filters first to preview which seeds will be removed. At least one filter must be provided to
+        prevent accidental deletion of all seeds. The deletion runs in a single transaction, so it works
+        consistently across SQLite and Azure SQL.
+
+        Only database records are removed. For file-backed seeds (image_path, audio_path, video_path) the
+        serialized file on disk is left in place; delete those files separately if they are no longer needed.


nit: this is pretty verbose, i think we could just say, "delete a list of seed prompts based on the specified filters." like get seeds

hannahwestra25 · 2026-07-31T15:05:12Z

+        Remove entire seed groups matching the specified filters from memory.
+
+        Unlike remove_seeds_from_memory, which deletes only the individual seeds that match, this
+        removes every seed that shares a prompt_group_id with any matching seed. This preserves group
+        integrity: filtering by a single modality (e.g. data_types=["image_path"]) or attribute removes
+        the whole group rather than leaving a partial group behind. It accepts the same filter parameters
+        as get_seeds. It is recommended to call get_seed_groups with the same filters first to preview
+        which groups will be removed. At least one filter must be provided to prevent accidental deletion
+        of all seeds. The deletion runs in a single transaction, so it works consistently across SQLite
+        and Azure SQL.
+
+        Only seeds that belong to a group are affected: a matching seed with no prompt_group_id (for example,
+        one added individually via add_seeds_to_memory_async rather than as part of a group) is skipped, since
+        it has no group to expand. Use remove_seeds_from_memory to delete ungrouped seeds.
+
+        Only database records are removed. For file-backed seeds (image_path, audio_path, video_path) the
+        serialized file on disk is left in place; delete those files separately if they are no longer needed.


could also be trimmed down to "delete groups of seed prompts based on the provided filtering criteria."

Copilot AI review requested due to automatic review settings July 21, 2026 22:22

Copilot started reviewing on behalf of blahdeblahde July 21, 2026 22:23 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 21, 2026 22:38

blahdeblahde force-pushed the blahdeblahde-remove-seeds-from-memory-api branch from 2d45b57 to 0e3ae02 Compare July 21, 2026 22:38

Copilot started reviewing on behalf of blahdeblahde July 21, 2026 22:39 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Comment thread pyrit/memory/memory_interface.py