Skip to content

FEAT: Add Local File Dataset Configuration for Dataset Tinkering - #2285

Open
ValbuenaVC wants to merge 7 commits into
microsoft:mainfrom
ValbuenaVC:vvalbuena-microsoft-dataset-tinkering
Open

FEAT: Add Local File Dataset Configuration for Dataset Tinkering#2285
ValbuenaVC wants to merge 7 commits into
microsoft:mainfrom
ValbuenaVC:vvalbuena-microsoft-dataset-tinkering

Conversation

@ValbuenaVC

@ValbuenaVC ValbuenaVC commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Description

RapidResponse operators need a short local iteration loop: ask Copilot to create or edit YAML seeds, reinitialize the scenario, and run against the latest file contents without synchronizing those seeds to a database.

This change adds DatasetAttackConfiguration.from_yaml_file(...), a file-backed inline source that:

  • Rereads local YAML during each dataset resolution, normally once per scenario initialization.
  • Preserves YAML dataset names and prompt grouping for scenario identity and result display.
  • Reuses existing validation and sampling without querying or writing PyRIT seed memory.
  • Performs blocking YAML I/O off the async event loop.
  • Rejects subclass calls that would silently discard subclass grouping behavior.
  • Converts read, parse, and grouping failures into actionable DatasetConstraintError messages.

Because disk remains authoritative, successful local loads now warn users at every local provider boundary. These warnings explain that in-process seed mutations are not written back automatically and will be lost unless saved to disk. Failed loads do not emit false success warnings.

The standard RapidResponse example remains unchanged. A separate recipe demonstrates the Copilot edit/rerun workflow using ~/.pyrit/rapid_response_local.prompt, a fresh scenario instance per iteration, and visible persistence warnings.

This intentionally does not add file watchers, dataset identifiers, revisioning, synchronization commands, or database schema changes.

Tests and Documentation

Developed in red-green TDD cycles covering:

  • Rereading edited and removed seeds.
  • Dataset-name and prompt-group preservation.
  • No seed-memory access for file-backed scenarios.
  • Validation before sampling and malformed-group errors.
  • Missing, deleted, and invalid files.
  • Subclass factory misuse.
  • Named-inline validator semantics.
  • Persisted scenario identity and RapidResponse grouping.
  • Success-only warnings for all local providers and file-backed resolution.
  • Synchronized notebook documentation and visible warning admonitions.

Validation completed:

  • 105 passed across local provider and dataset-configuration suites.
  • 46 passed in the RapidResponse suite.
  • Ruff lint and formatting passed.
  • ty passed.
  • JupyText pairs match and existing outputs are preserved.
  • git diff --check passed.
  • Independent Copilot and Opus reviews found no remaining blockers.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 34c04c70-cd29-4216-9321-65a3beedc523

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds file-backed YAML dataset configuration for rapid local scenario iteration without memory synchronization.

Changes:

  • Adds asynchronous file-backed dataset resolution with validation and sampling.
  • Adds unit coverage for reloads, grouping, validation, and memory isolation.
  • Updates the RapidResponse notebook workflow.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File Description
pyrit/scenario/core/dataset_configuration.py Implements local YAML-backed configuration.
tests/unit/scenario/core/test_dataset_configuration.py Tests file-backed behavior.
doc/scanner/airt.py Documents the local iteration workflow.
doc/scanner/airt.ipynb Synchronizes notebook documentation.

@ValbuenaVC
ValbuenaVC marked this pull request as ready for review July 29, 2026 19:43
@ValbuenaVC ValbuenaVC changed the title [DRAFT] FEAT: Add Local File Dataset Configuration for Dataset Tinkering FEAT: Add Local File Dataset Configuration for Dataset Tinkering Jul 29, 2026
@ValbuenaVC
ValbuenaVC requested a review from Copilot July 29, 2026 19:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

tests/unit/scenario/airt/test_rapid_response.py:257

  • This new test omits explicit types for both mock fixtures and its return value. The repository requires every function parameter and return type to be annotated (.github/instructions/style-guide.instructions.md:81-84); please use MagicMock for these fixtures and None for the return.
        mock_objective_target,
        mock_objective_scorer,
    ):

Victor Valbuena and others added 3 commits July 29, 2026 13:29
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 34c04c70-cd29-4216-9321-65a3beedc523
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants