feat: Add BijectionConverter and BijectionAttack

[sajithamathi19]

Summary

Implements the Bijection Attack from arXiv:2410.01294 (Haize Labs) into PyRIT.

The attack works by teaching a target LLM a secret character mapping through demonstration shots, then sending harmful prompts encoded in that mapping to bypass safety filters. Responses are decoded using the inverse mapping.

Changes

New Files

• pyrit/converter/bijection_converter.py - abstract base plus LetterBijectionConverter, DigitBijectionConverter, TokenBijectionConverter
• pyrit/executor/attack/single_turn/bijection_attack.py - runs full bijection attack with teaching phase
• tests/unit/converter/test_bijection_converter.py
• tests/unit/executor/test_bijection_attack.py
• doc/code/executor/attack/bijection_attack.ipynb - usage notebook

Modified Files

• pyrit/converter/__init__.py - registered converters
• pyrit/executor/attack/single_turn/__init__.py - registered BijectionAttack

How It Works

• BijectionConverter generates a random secret mapping (letter, digit, or tokenizer-vocab based)
• BijectionAttack sends teaching messages to the target to teach the mapping
• Harmful prompt is encoded and sent as the task
• Response is decoded using the inverse mapping and scored by the judge

Pattern Followed

• BijectionConverter follows the FlipConverter pattern
• BijectionAttack follows the FlipAttack pattern

Reference

• Haize Labs implementation: https://github.com/haizelabs/bijection-learning
• Paper: arXiv:2410.01294

Bugs fixed in this resubmission

• TokenBijectionConverter - the mode @romanlutz originally flagged as producing garbage. Root cause was three separate bugs: no delimiter between encoded units, tokenizer word-initial markers (Ġ/▁) not handled, and teaching shots using hardcoded letter-cipher instructions even in token mode. All fixed; round-trips correctly now.
• DigitBijectionConverter - found while re-verifying: digit tokens have no upper/lowercase, so capitalized input was silently lowercased on decode. Added a case-marker prefix so capitalization round-trips correctly.
• Teaching-example coverage - per @romanlutz's review on feat: Add BijectionConverter and BijectionAttack (#1903) #1942, the original 5-sentence pool only demonstrated 14 of 26 letters. Expanded to 13 sentences covering the full alphabet.

Empirical validation against real models

Tested the full teaching protocol (not just converter round-trip logic) against real targets, since converter correctness and actual attack reliability are two different questions:

Target	Coherent decode rate
Gemini 2.5 Flash Lite (small/fast)	2/9
Gemini 2.5 Pro	8/9
GPT-4o	6/9

Capable models now succeed the majority of the time (up from ~1/6 before the alphabet-coverage fix). It's not perfectly reliable - the small/fast model still mostly fails, and some runs on capable models still truncate or have letter-level errors - which tracks with the paper's own "scale-adaptive" framing (attack success is expected to depend on target capability, not be a fixed property of the implementation).

Status

• CLA signed
• Rebased against main (picked up the pyrit.prompt_converter -> pyrit.converter rename and other upstream changes); full converter + executor suite (2176 tests) passing
• Resubmits #1942, which was auto-closed when the source account was accidentally deleted. All prior commit history and review context carries over.

[sajithamathi19]

Hi @romanlutz — resubmitting after my old account was accidentally deleted (all the review history and commits from #1942 are preserved here).

Before pinging you again I wanted to actually re-test this properly rather than just say "tests pass." Two things:

Fixed two real bugs while re-verifying converter correctness against the actual installed package (not just unit-test mocks):

• TokenBijectionConverter — the one you originally flagged. Three root causes: no delimiter between encoded units, tokenizer word-initial markers (Ġ/▁) not handled, hardcoded letter-cipher teaching instructions even in token mode. All fixed.
• DigitBijectionConverter — found this one while re-checking: digit tokens have no case, so capitalized input was silently lowercased. Fixed with a case marker.

Your actual bar — "a working bijection" from a real model response — is still not met. I tested the full teaching protocol against real targets instead of assuming the code fixes were enough:

• Claude refuses the setup outright before any cipher behavior can occur.
• Gemini 2.5 Flash Lite: 1/6 trials scored "valid" by the code's own heuristic, and that one was a degenerate repeated-nonsense loop, not real cipher-following. The rest dropped the cipher, looped, or produced undecodable text.
• Tried reducing complexity (fixed_size) and a stronger model (GPT-4o) — better but still inconsistent, not reliable.

This tracks with the paper describing Bijection Learning as scale-adaptive (success depends on target capability), so I don't think this is something I can fix further in the wrapper code alone. Marked this as a draft rather than claim it's ready.

Would rather have your read on this now than dress it up: is this worth merging with clear documentation that reliability varies by target, or should it wait until it's consistent against some reference model? Full breakdown is in the updated PR description. Thanks for your patience through this one.

[sajithamathi19]

Re-tested against real models with the alphabet-coverage fix now actually in the code (not just the converter round-trip logic — the full teaching protocol as it runs today, default num_teaching_shots=5, no other changes):

Target	Coherent decode rate
Gemini 2.5 Flash Lite (small/fast)	2/9
Gemini 2.5 Pro	8/9
GPT-4o	6/9

This is a real improvement — capable models now produce a coherent decoded response the majority of the time, up from roughly 1/6 before this fix. It's not fully reliable (some runs still truncate or have letter-level errors, and the small/fast model still mostly fails), which tracks with the paper's own "scale-adaptive" framing — but this is genuine, measured progress on the exact thing you flagged, not just a claim.

Converted this back to ready for review. Let me know if you want the raw transcripts from these runs, or if you'd rather see this validated against a different reference target before moving forward.

[sajithamathi19]

@microsoft-github-policy-service agree