Skip to content

redteam suite: non-blocking follow-ups from #29 review #37

Description

@stefanoferi

Non-blocking follow-ups from the review of #29 (admina-redteam detection-efficacy suite). These do not block the merge of #29; tracking them here so they are not lost.

  • Corpus size / reporting — corpora are small (37/26/11 positives) and per-lang/per-tag cells are 1–3 samples. Show numerator/denominator per cell and mark the output indicative; grow the corpora (especially non-EN) over time.
  • Label provenance — labels are author-assigned in-repo. Add a one-line note that they are not third-party adjudicated.
  • CLI not shipped — scripts/redteam.py is not in [project.scripts], so a pip-installed user cannot run the CLI the docs reference. Add an entry point (e.g. admina redteam) or adjust the docs.
  • Gate fp invariant — the gate compares raw fp counts, sound only because the corpora are sealed. Add assert cur["fp_samples"] == base["fp_samples"], or compare rates.
  • Corpus edge cases — loop-en-009 (counter-reset labelled loop), non-Luhn test cards, and no hard negatives in the PII FPR set.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions