The PR opened by this comment, feels too shallow in its validation and reasoning.
The automation fixed the issue, but ran the test once and called it a day. It also failed to include the reasoning I provided in the prompt, such as the rationale behind the change and the supporting documentation.
I would expect the automation to be more proactive and thorough. At the very least, it should run the test multiple times, consider potential side effects, reason about the impact of the changes, include relevant context, and evaluate how they could break things. That is the standard I follow when opening any PR.
I have been thinking that perhaps we should introduce a general action for creating pull requests that encompasses all of these points, so the more generic Mention on Issue automation can use it when generating PRs. I'm not sure if this already exists.
The PR opened by this comment, feels too shallow in its validation and reasoning.
The automation fixed the issue, but ran the test once and called it a day. It also failed to include the reasoning I provided in the prompt, such as the rationale behind the change and the supporting documentation.
I would expect the automation to be more proactive and thorough. At the very least, it should run the test multiple times, consider potential side effects, reason about the impact of the changes, include relevant context, and evaluate how they could break things. That is the standard I follow when opening any PR.
I have been thinking that perhaps we should introduce a general action for creating pull requests that encompasses all of these points, so the more generic Mention on Issue automation can use it when generating PRs. I'm not sure if this already exists.