Add Pydantic AI shared engine definition and smoke test by Copilot · Pull Request #51161 · github/gh-aw

Copilot · 2026-08-07T18:51:35Z

Pydantic AI wants to dogfood their own stack as a gh-aw engine. This adds the shared definition and smoke test to support engine: id: pydantic-ai as a first-class importable engine.

`shared/pydantic.md`

Shared engine definition following the aider.md pre-agent-steps pattern (used because behaviors.installation.package-manager: uv silently emits no install step):

engine:
  id: pydantic-ai
model: copilot/claude-sonnet-4-5
imports:
  - shared/pydantic.md

Installs pydantic-ai-harness via pip in pre-agent-steps; version pinned via engine.version
secret-strategy: universal-llm-consumer — routes through AWF proxy
MCP enabled; reads server config from GH_AW_MCP_CONFIG so safe outputs flow through safeoutputs automatically
Includes a log-parser function that normalizes Pydantic AI JSONL output into conversation events, enabling step summaries and token tracking in gh aw logs / gh aw audit

`smoke-pydantic.md`

Smoke test workflow mirroring smoke-aider.md: validates file writing, bash execution, and repo access, then emits results as an issue + PR comment via $GH_AW_SAFE_OUTPUTS.

Fixes Pydantic AI as an agentic engine #50971

Generated by 👨‍🍳 PR Sous Chef · gpt54 · 7.54 AIC · ⊞ 8.5K · ◷
_{Comment /souschef to run again}

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

pelikhan · 2026-08-07T19:13:34Z

@copilot add pydantic-ai to engines.json

Copilot

Pull request overview

Adds a first-class Pydantic AI engine definition and smoke workflow.

Changes:

Defines installation, execution, MCP, networking, and log parsing.
Adds a smoke workflow and slash-command routing.
Generates the corresponding workflow lock file.

Show a summary per file

File	Description
`.github/workflows/shared/pydantic.md`	Defines the Pydantic AI engine.
`.github/workflows/smoke-pydantic.md`	Adds engine smoke testing.
`.github/workflows/smoke-pydantic.lock.yml`	Compiles the smoke workflow.
`.github/workflows/agentic_commands.yml`	Registers `/smoke-pydantic`.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 3/4 changed files
Comments generated: 1
Review effort level: Balanced

+1. **File Writing Testing**: Create a test file `/tmp/gh-aw/agent/smoke-test-pydantic-${{ github.run_id }}.txt` with content "Smoke test passed for Pydantic AI" (create the directory if it doesn't exist)
+2. **Bash Tool Testing**: Execute bash commands to verify file creation was successful (use `cat` to read the file back)
+3. **Repository Access Testing**: Run `git log --oneline -1` in the repository checkout and confirm a commit is reported


pelikhan · 2026-08-07T19:20:58Z

+        - .pydantic-ai/
+    network:
+      defaults:
+        - host.docker.internal


@copilot none of these should be needed since pydantic is already pre-installed. keep network as "defaults"

Updated in d36df30. I removed the Pydantic install pre-step and switched the shared engine networking back to defaults (dropping the extra PyPI domains).

github-actions · 2026-08-07T19:21:55Z

⚠️ Security scanning failed for Test Quality Sentinel. Review the logs for details.

No test files were added or modified in this PR. Test Quality Sentinel skipped. PR contains only workflow definitions (.md, .yml, .lock.yml) and no behavioral tests (Go *_test.go or JavaScript .test.cjs/.test.js).

github-actions · 2026-08-07T19:21:56Z

⚠️ Security scanning failed for Design Decision Gate 🏗️. Review the logs for details.

No ADR enforcement needed: PR does not have the 'implementation' label and has ≤100 new lines of code in business logic directories (0 additions detected in default business logic paths).

github-actions · 2026-08-07T19:21:59Z

⚠️ PR Code Quality Reviewer failed during code quality review.

Warning

Threat Detection Engine Failure — The analysis engine could not complete. This is a tooling failure, not a security finding.

What happened

The threat detection engine failed to produce results.

Review the workflow run logs for details.

github-actions · 2026-08-07T19:22:08Z

⚠️ Security scanning failed for Matt Pocock Skills Reviewer. Review the logs for details.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-08-07T19:27:28Z

@copilot add pydantic-ai to engines.json

Added in 4d4987c: .github/aw/engines.json now includes pydantic-ai mapped to github/gh-aw/.github/workflows/shared/pydantic.md.

github-actions

Skills-Based Review 🧠

Applied /codebase-design and /tdd — commenting (no blocking issues, but a few items worth addressing before promoting from experimental).

📋 Key Themes & Highlights

Key Themes

Runtime network policy: smoke-pydantic.md sets network: allowed: [] which may conflict with the engine's need to reach the LLM proxy. Needs verification against smoke-aider.md baseline.
Log-parser token extraction: Silent zero-token totals when Pydantic AI emits usage under a nested key (e.g. usage.input_tokens). A small fixture-based test would prevent regressions.
Empty tool_result content: msg can be empty string; downstream schema validators may reject it.
Network hosts at runtime vs install-time: PyPI entries in the runtime allowlist are only needed during pre-agent-steps.
Version override discoverability: Override pattern documented only in an HTML comment; a YAML-adjacent comment would be more visible.

Positive Highlights

✅ Clean log-parser structure mirroring the aider pattern
✅ secret-strategy: universal-llm-consumer correctly routes through AWF proxy
✅ MCP config via GH_AW_MCP_CONFIG is consistent with the existing engine contract
✅ Smoke test structure mirrors smoke-aider.md — easy to compare and audit

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · sonnet46 · 38.3 AIC · ⊞ 7.1K
_{Comment /matt to run again}

github-actions · 2026-08-07T19:28:29Z

+        - .pydantic-ai/
+    network:
+      defaults:
+        - host.docker.internal


[/codebase-design] Network defaults includes pypi.org and files.pythonhosted.org — these are only needed during pre-agent-steps installation, not agent runtime.

💡 Suggestion

PyPI hosts are consumed by pip install in pre-agent-steps, which runs before the agent sandbox. If the runtime network policy cannot be scoped per-phase, document why these hosts must remain open during agent execution. Also, the PR comment from @pelikhan suggests pydantic-ai-harness may already be pre-installed, which would make the pip install step and these entries unnecessary entirely.

@copilot please address this.

github-actions · 2026-08-07T19:28:29Z

+            if (parsed.input_tokens) inputTokens += parsed.input_tokens;
+            if (parsed.output_tokens) outputTokens += parsed.output_tokens;
+            const entryType = parsed.type != null ? String(parsed.type) : "log";
+            const msg = parsed.msg || parsed.message || parsed.content || "";


[/tdd] The log-parser only accumulates input_tokens / output_tokens from JSON lines that have those exact keys — if Pydantic AI emits them under a different key (e.g. usage.input_tokens) the totals silently remain zero.

💡 Suggestion

Add a test fixture (a small JSONL sample) and a unit test that asserts:

known token-bearing lines are counted correctly

lines with a nested usage object are also handled

the final result entry reflects the correct totals

Without this, a Pydantic AI output format change will produce invisible zero-token audit entries.

@copilot please address this.

github-actions · 2026-08-07T19:28:29Z

+              const toolName = parsed.tool || parsed.name || entryType;
+              logEntries.push({ type: "assistant", message: { content: [{ type: "tool_use", id: toolId, name: toolName, input: {} }] } });
+              logEntries.push({ type: "user", message: { content: [{ type: "tool_result", tool_use_id: toolId, content: msg }] } });
+            } else if (msg) {


[/codebase-design] Tool-result entries always set content: msg where msg may be an empty string — the harness may reject a tool_result with an empty content array.

💡 Suggestion

Use a fallback: content: msg || '(no output)' to guarantee a non-empty string. Downstream log consumers that validate against the Anthropic message schema will reject empty content.

@copilot please address this.

github-actions · 2026-08-07T19:28:29Z

+  id: pydantic-ai
+strict: true
+imports:
+  - shared/pydantic.md


[/codebase-design] network: allowed: [] blocks all outbound network access, but the smoke test calls git log which may need to resolve remote refs depending on the checkout strategy. More critically, pai needs to reach the LLM proxy at api.githubcopilot.com.

💡 Suggestion

Compare with smoke-aider.md to confirm whether the smoke test inherits the shared engine network policy or fully overrides it. If allowed: [] is an override, the agent will be unable to reach any LLM endpoint and the smoke test will fail at the engine invocation step.

@copilot please address this.

github-actions · 2026-08-07T19:28:29Z

+    version: "3.12"
+pre-agent-steps:
+  - name: Install Pydantic AI CLI
+    run: |


[/codebase-design] engine.version: "0.1.0" is hard-coded in the shared definition. When the shared file is imported, users who want to override the version must redeclare engine.version in their own workflow — but the precedence of that override is not documented.

💡 Suggestion

The PR description explains version override syntax, but that comment lives only in the HTML comment block at the bottom of the file. Move a one-line note into the visible YAML, or add a comment adjacent to the version key:

engine: id: pydantic-ai version: "0.1.0" # override in your workflow: engine: { version: "x.y.z" }

This keeps the contract discoverable without reading the HTML comment.

@copilot please address this.

github-actions

Review: Add Pydantic AI shared engine definition and smoke test

The shared engine definition is well-structured and consistent with existing engines (aider, goose). Two existing review threads cover MCP smoke-test coverage and unnecessary pip installs. One new issue found:

Log-parser: MCP failure lines leak into the assistant transcript — lines matching the MCP failure regex are added to mcpFailures but not continued, so they also fall through to pendingText.push(line.trim()) and appear as synthetic assistant text. See inline comment on line 103 of shared/pydantic.md.

Overall the PR is directionally sound. Blocking the MCP failure line leak before merge is recommended.

🧵 Reviewed using Impeccable skills by Impeccable Skills Reviewer · sonnet46 · 62.2 AIC · ⊞ 5.5K

github-actions · 2026-08-07T19:30:41Z

+              pendingText.push(msg);
+            }
+          } else {
+            pendingText.push(line.trim());


MCP failure lines match the mcpFailures push above but are not skipped afterward — they fall through to pendingText.push(line.trim()) and are emitted as synthetic assistant text in the log transcript. Every MCP error message ends up double-counted: once in mcpFailures and once as an assistant turn.

Fix by adding continue after the mcpFailures.push(...) call:

if (/MCP server .* failed|MCP.*connection.*error|Failed to connect to MCP/i.test(line)) { const serverMatch = line.match(/MCP server ['"x]?([^\s'"]+)['"x]?/i); mcpFailures.push(serverMatch ? serverMatch[1] : line.trim()); continue; // prevent double-emit as transcript text }

@copilot please address this.

dsfaccini · 2026-08-07T19:32:06Z

fyi, this will be valuable to have in place already, but we have some more work to do on the pydantic-ai and pydantic-ai-harness side of things

edit: some more info

this is the draft PR for the implementation in our harness Add a compile-tested gh-aw engine definition for a pydantic-ai behavior-defined engine pydantic/pydantic-ai-harness#569
this is the draft PR for an underlying claude code json stream adapter in pydantic ai core Add ClaudeCodeEventStream emitting Claude Code CLI stream-json from agent run events pydantic/pydantic-ai#7276

gh-aw-bot · 2026-08-07T19:35:16Z

@copilot please refresh the branch if needed and run the pr-finisher skill to address any remaining review or validation items.

Generated by 👨‍🍳 PR Sous Chef · gpt54 · 7.54 AIC · ⊞ 8.5K · ◷
_{Comment /souschef to run again}

pelikhan · 2026-08-07T19:40:54Z

@dsfaccini we don't plan to own this long term but it is easier to iterate here to flush out bugs in gh-aw, then glabdly transfer and keep a copy.
@copilot see pydantic/pydantic-ai-harness#569 to compare notes. use "uv".

pelikhan · 2026-08-07T19:41:30Z

(claude code compatibility is smart because a lot of tooling will "just work")

pelikhan · 2026-08-07T19:41:57Z

This PR will be merged probably a bit broken because I need a "smoke-pydantic" action in main in order to run it.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-08-07T19:57:12Z

@dsfaccini we don't plan to own this long term but it is easier to iterate here to flush out bugs in gh-aw, then glabdly transfer and keep a copy.
@copilot see https://github.com/pydantic/pydantic-ai...

Addressed in 9ca8622. The shared Pydantic engine now runs through uv (uv run pai run ...), and the lockfile was recompiled accordingly.

@copilot please refresh the branch if needed and run the pr-finisher skill to address any remaining review or validation items.

Generated by [👨‍🍳 PR Sous Chef](https://github.com/github/g...

Done in 9ca8622. I refreshed the branch context, addressed the remaining actionable item (use uv), recompiled, and ran local validation + review/security checks.

pelikhan · 2026-08-07T19:59:07Z

@copilot add a step using uv api version to predownload pai

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-08-07T20:48:13Z

@copilot add a step using uv api version to predownload pai

Added uv run pai --version as a pre-agent preload step and regenerated the lockfile in 382bfaf.

github-actions · 2026-08-07T21:52:31Z

🎉 This pull request is included in a new release.

Release: v0.86.1

Initial plan

45a2c29

Copilot AI assigned Copilot and pelikhan Aug 7, 2026

Copilot AI linked an issue Aug 7, 2026 that may be closed by this pull request

Pydantic AI as an agentic engine #50971

Closed

Copilot started work on behalf of pelikhan August 7, 2026 18:52 View session

feat: add Pydantic AI shared engine definition and smoke test workflow

447f88f

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Add Pydantic AI as an agentic engine~~ Add Pydantic AI shared engine definition and smoke test Aug 7, 2026

pelikhan marked this pull request as ready for review August 7, 2026 19:13

Copilot AI balanced review requested due to automatic review settings August 7, 2026 19:13

Copilot started reviewing on behalf of pelikhan August 7, 2026 19:13 View session

Copilot AI requested a review from pelikhan August 7, 2026 19:16

Copilot finished work on behalf of pelikhan August 7, 2026 19:16

Copilot AI reviewed Aug 7, 2026

View reviewed changes

Copilot started work on behalf of pelikhan August 7, 2026 19:20 View session

pelikhan reviewed Aug 7, 2026

View reviewed changes

Register pydantic-ai in engines registry

4d4987c

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI requested a review from pelikhan August 7, 2026 19:27

Copilot finished work on behalf of pelikhan August 7, 2026 19:27

Copilot started work on behalf of pelikhan August 7, 2026 19:28 View session

github-actions Bot reviewed Aug 7, 2026

View reviewed changes

github-actions Bot added the needs-review label Aug 7, 2026

Merge branch 'main' into copilot/pydantic-ai-agentic-engine

ad1e4f5

github-actions Bot mentioned this pull request Aug 7, 2026

[pr-sous-chef] PR Sous Chef run report #50950

Closed

Remove redundant pydantic install and use default network

d36df30

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan August 7, 2026 19:43

Copilot started work on behalf of pelikhan August 7, 2026 19:43 View session

Run pydantic engine via uv runtime

9ca8622

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan August 7, 2026 20:11

Copilot started work on behalf of pelikhan August 7, 2026 20:11 View session

Preload Pydantic AI CLI with uv

382bfaf

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan August 7, 2026 20:48

Merge branch 'main' into copilot/pydantic-ai-agentic-engine

0957f06

pelikhan merged commit 0e301e6 into main Aug 7, 2026
1 check failed

pelikhan deleted the copilot/pydantic-ai-agentic-engine branch August 7, 2026 20:57

github-actions Bot mentioned this pull request Aug 7, 2026

Smoke Test: MAI-Code-1-Flash - 31220061988 #51200

Closed

This was referenced Aug 7, 2026

Smoke Test: MAI-Code-1-Flash - 31222506585 #51219

Closed

Smoke Test: MAI-Code-1-Flash - 31231539109 #51260

Closed

Uh oh!

Conversation

Copilot AI commented Aug 7, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

shared/pydantic.md

smoke-pydantic.md

Uh oh!

pelikhan commented Aug 7, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

pelikhan Aug 7, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 7, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI commented Aug 7, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Skills-Based Review 🧠

Key Themes

Positive Highlights

Uh oh!

github-actions Bot Aug 7, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Aug 7, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Aug 7, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Aug 7, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Aug 7, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Review: Add Pydantic AI shared engine definition and smoke test

Uh oh!

github-actions Bot Aug 7, 2026

Choose a reason for hiding this comment

Uh oh!

dsfaccini commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gh-aw-bot commented Aug 7, 2026

Uh oh!

pelikhan commented Aug 7, 2026

Uh oh!

pelikhan commented Aug 7, 2026

Uh oh!

pelikhan commented Aug 7, 2026

Uh oh!

Copilot AI commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pelikhan commented Aug 7, 2026

Uh oh!

Copilot AI commented Aug 7, 2026

Uh oh!

Copilot AI commented Aug 7, 2026 •

edited by github-actions Bot

Loading

`shared/pydantic.md`

`smoke-pydantic.md`

github-actions Bot commented Aug 7, 2026 •

edited

Loading

github-actions Bot commented Aug 7, 2026 •

edited

Loading

github-actions Bot commented Aug 7, 2026 •

edited

Loading

github-actions Bot commented Aug 7, 2026 •

edited

Loading

dsfaccini commented Aug 7, 2026 •

edited

Loading

Copilot AI commented Aug 7, 2026 •

edited

Loading