Skip to content

fix(buffs): use prompt setter in paraphrase buffs to update conversations - #2200

Merged
jmartin-tech merged 1 commit into
NVIDIA:mainfrom
bodapatisaikrishna:fix/paraphrase-prompt-setter
Sep 16, 2026
Merged

jmartin-tech merged 1 commit into
NVIDIA:mainfrom
bodapatisaikrishna:fix/paraphrase-prompt-setter

Conversation

@bodapatisaikrishna

Copy link
Copy Markdown
Contributor

Description

In garak/buffs/paraphrase.py, PegasusT5.transform() and Fast.transform() previously assigned paraphrased messages directly to the private attribute paraphrased_attempt._prompt:

delattr(paraphrased_attempt, "_prompt")  # hack to allow prompt set
paraphrased_attempt._prompt = garak.attempt.Message(
    text=paraphrase, lang=last_message.lang
)

Directly assigning _prompt = Message(...) bypassed the Attempt.prompt property setter. This led to several issues:

  1. Unupdated conversations: The prompt setter is responsible for updating self.conversations. Bypassing it meant paraphrased_attempt.conversations was never updated with the paraphrased text and retained the original prompt. Downstream generator executions or reports reading conversations received the unparaphrased text.
  2. Incorrect prompt type: Attempt.prompt returned <class 'garak.attempt.Message'> instead of <class 'garak.attempt.Conversation'>.
  3. AttributeError on attempt.lang: Attempt.lang expects self.prompt to be a Conversation and accesses self.prompt.turns[-1].content.lang. Calling .lang on any attempt transformed by PegasusT5 or Fast raised AttributeError: 'Message' object has no attribute 'turns'.

Fix

Update both PegasusT5.transform() and Fast.transform() to use the public property setter:

paraphrased_attempt.prompt = garak.attempt.Message(
    text=paraphrase, lang=last_message.lang
)

This matches the correct pattern used in other buffs (garak/buffs/lowercase.py, garak/buffs/encoding.py, garak/buffs/low_resource_languages.py).

Testing

  • Added validation in test_buff_load_and_transform in tests/buffs/test_buffs.py verifying that every transformed attempt's prompt is a Conversation and that .lang is accessible without error.
  • Added dedicated unit test test_paraphrase_transform_conversation_and_lang in tests/buffs/test_buffs.py.

Verification

  • Run unit tests and ensure they pass:
    pytest tests/buffs/test_buffs.py -k test_paraphrase_transform_conversation_and_lang
    Result: 1 passed.
  • Run full buff test suite:
    pytest tests/buffs/test_buffs.py tests/buffs/test_buff_config.py
    Result: 14 passed, 1 skipped.
  • Run linter and formatting:
    ruff check garak/buffs/paraphrase.py tests/buffs/test_buffs.py
    ruff format --check garak/buffs/paraphrase.py tests/buffs/test_buffs.py
    Result: All checks passed.
  • Verify the thing does what it should: paraphrased_attempt.prompt is a Conversation, .lang returns the language string, and paraphrased_attempt.conversations reflects the new paraphrase text.
  • Verify the thing does not do what it should not: No regressions in existing buff transform behavior.

…ions and maintain Conversation type

In PegasusT5.transform() and Fast.transform(), paraphrased_attempt._prompt was being assigned directly to a garak.attempt.Message object rather than using the paraphrased_attempt.prompt property setter.

Directly setting the private _prompt attribute caused:
1. paraphrased_attempt.conversations was not updated, leaving the original unparaphrased prompt in conversations.
2. paraphrased_attempt.prompt returned a Message instead of a Conversation.
3. Calling paraphrased_attempt.lang raised AttributeError: 'Message' object has no attribute 'turns'.

By assigning via paraphrased_attempt.prompt = ... (matching lowercase.py, encoding.py, and low_resource_languages.py), the setter creates the proper Conversation turn structure and updates conversations.

Added regression tests verifying that transformed attempts have a Conversation prompt, valid lang access, and updated conversations history.

Signed-off-by: bodapatisaikrishna <saikrishnabodapati@gmail.com>

@jmartin-tech jmartin-tech left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch this is a clear error from the Conversation support refactor.

@jmartin-tech jmartin-tech self-assigned this Sep 16, 2026
@jmartin-tech
jmartin-tech merged commit b7f39fd into NVIDIA:main Sep 16, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants