V0.9.0/finalization - #38
Open
gimlichael wants to merge 66 commits into
Open
Conversation
Reorganized dotnet-remote-testing skill documentation and evaluation definitions for better clarity and maintainability. Updated SKILL.md, FORMS.md, and reference materials to align with current implementation capabilities. Defined comprehensive evaluation suite with test fixtures to validate skill behavior across diverse environments.
Added test harness (test-remote-testing.ps1) to exercise skill functionality across real and simulated Docker environments. Enhanced validate-skill.ps1 with more robust checks for deterministic validation and error recovery. Both scripts integrate with the skill's evaluation framework to catch regressions early.
Significantly improved remote-test.cs implementation with better Docker image discovery logic and multi-SDK runner selection. Added intelligent fallback mechanisms for offline environments and improved error messages for troubleshooting. New MultiSdkRunner and MultiSdkRunnerStore classes provide tighter image selection, reducing redundant SDK layers in test environments. Enhanced caching strategy to minimize network calls and improve deterministic behavior.
Synced README.md with latest dotnet-remote-testing skill enhancements, including improved feature descriptions and updated capability documentation to reflect the expanded testing infrastructure and enhanced Docker environment discovery.
The runner now probes images for required build tooling (git) and conditionally provisions it in a cached preparation layer. Update SKILL.md constraints and guidance to explain that the runner owns this behavior outside the repository, with docker-execution.md documenting the image preparation sequence, caching, and best-effort fallback when tooling cannot be added.
Implement ImageProvisioner to handle missing build tooling in container images. The runner probes the resolved image for git (required by MinVer, Nerdbank.GitVersioning, GitInfo, and SourceLink during dotnet build), and when absent, provisions a cached preparation layer using the appropriate package manager (apt-get, apk, microdnf). Add NuGetPackagesPath helper to ensure SourceLink receives SourceRoot paths with required trailing separator, ErrorLines helper for diagnostic output, FirstNonEmpty string utility, and ImagePreparationTests covering tag derivation, probing, dockerfile generation, and package-manager detection.
Expand the v0.9.0 changelog entry for dotnet-remote-testing to document the build-tooling preparation feature: the runner probes images for git, provisions a cached preparation layer when missing, and reports its status transparently without treating it as a repository problem or reason to author container plumbing.
Implement deterministic skill synchronization script that copies whole skill trees from repository source to three local installs (Claude, global agents, Gemini Antigravity), compares SHA-256 hashes across all four locations, and exits non-zero on any drift. Script supports -Skill parameter to sync a single skill, -VerifyOnly to check without copying, and -Prune to delete install-only files. Excludes generated build output (bin/, obj/) which is regenerated per location. This is the mechanized equivalent of the previous manual sync guidance.
Restructure Local Install Sync section to reference the deterministic sync-skill-install.ps1 script, emphasizing that mechanical hash comparison is the required verification gate rather than manual file copying. Add new Blocking Completion Gates section that identifies required script executions and validators as hard requirements before completion; specifically mark sync-skill-install.ps1 as the final blocking gate since every prior step can still change files. This updates governance to enforce the mechanized sync process as repository policy.
Expand the skill synchronization paragraph to explicitly document the sync-skill-install.ps1 script command and explain that mechanical hash-based verification is required, not just a remembered list of copied files. Clarify that a sync claim must be backed by actual command output in the same response that makes it, and that an earlier run does not satisfy the requirement. This aligns README user-facing documentation with the mechanized sync governance.
Add git metadata staging to the workspace so version stamping, SourceLink, and repository-root detection behave as they do on the host. Introduce --no-git-metadata to opt out when .git dominates staging cost. Improve result reporting to break down per test assembly and target framework, with full failure detail (assertion message, stack trace, test output) so failures are actionable without a rerun. Add --show-log for diagnostics.
Document git metadata staging and per-assembly result reporting enhancements for dotnet-remote-testing.
Greptile SummaryThis PR expands the repository’s skill authoring, portable evaluation, synchronization, and .NET testing workflows.
Confidence Score: 4/5The PR is not yet safe to merge because the mandatory synchronization gate can still succeed when expected host installation roots are absent. The current sync script skips every missing host root without adding drift, so a machine with no recognized local host installation can receive a successful “verified, 0 drift” result without synchronizing or verifying any copy. Files Needing Attention: scripts/sync-skill-install.ps1 Important Files Changed
Reviews (9): Last reviewed commit: "🛠️ make bind mounts writable for contai..." | Re-trigger Greptile |
Separate host-tool root detection from skill installation path validation. This prevents false positive drift reports when a tool host exists but contains no skill installation yet; previously, both states were treated identically as 'not installed'. The refactor also creates missing skill directories on sync and correctly reports them as drift during verify-only mode.
Add ResolveUserAsync to extract the configured USER from a docker image, then restore it in the prepared-image Dockerfile after package installation. This ensures a base image configured to run as a non-root user continues to do so in the prepared image, preserving file ownership and permission-sensitive test behavior. Update docker-execution.md to explain the identity-preserving guarantee and the linked-worktree git-directory handling.
Expands the skill's README entry with additional constraints around immediate action and clarifies that the skill does not ask permission before proceeding.
Restructures instructions with a new 'Start Here' section clarifying that the skill acts immediately without seeking permission. Adds explicit 'Do not' guidance around permission-seeking questions and adds test case 19 to verify this behavior.
Restructures instructions with a new 'Start Here' section clarifying that the skill acts immediately without seeking permission. Adds explicit 'Do not' guidance around permission-seeking questions and adds test case 19 to verify this behavior.
Adds input validation to prevent path traversal attacks via skill name parameter. Ensures skill names follow kebab-case convention (alphanumeric + hyphens only) before constructing file paths in sync operations.
Add explicit assertions for immediate-action behavior and evidence-first contracts. These validators lock down the regression where a bare invocation produced a capability menu instead of running the first action. Also adds assertions for managed-fixture floor version and async-disposal defense-code guidance.
Restructure SKILL.md to lock down immediate-action behavior: a bare invocation must act on inspector evidence instead of presenting a capability menu. Add form fallback documentation and tighten managed-fixture floor version checks. Harden eval scenarios to cover both paired role selections and the regression case. Update reference docs with async-disposal guidance for test hosts and add fixture version assertions in inspection scripts.
Add dotnet-test skill to the Available Skills table and the project scaffold examples. Include both installation and discovery sections to help users locate the skill. Clarifies the skill's role in transitioning WebApplicationFactory-based test projects to Codebelt's entrypoint-owned test host pattern.
…t/agentic into v0.9.0/finalization
The skill contract now documents the xunit anchor mechanism that prevents xunit* packages from outrunning the Codebelt release the skill targets. The resolver uses the Codebelt xUnit package's published nuspec to determine which xunit* ids pin 1:1 to declared versions and which cap at the anchor's major version. Updated SKILL.md with constraints documentation and xunit-v3-modernization.md with version distinction between package name and actual version numbers.
The resolver now anchors every xunit* package to the Codebelt xUnit release (Codebelt.Extensions.Xunit or Codebelt.Extensions.Xunit.App by role). It queries the anchor's published nuspec dependencies to determine which xunit ids resolve 1:1 to the declared version and which cap at the anchor's major version. New functions: Resolve-XunitAnchor reads nuspec and returns major/dependencies, Select-AnchoredCandidate filters candidates by anchored major, Get-AnchorConstraint reports the constraint for each resolved package. The resolver caches and reports anchoring evidence.
Test harness now covers xunit anchor anchoring logic: mocks for nuspec queries, anchor resolution from Codebelt xUnit packages, candidate selection filtered by anchored major, and constraint reporting. Validates that candidates above the anchored major are excluded, that no candidate above the anchored major may reach a restore, and that the anchor evidence is correctly returned in the resolver output.
Introduce prepare-skill-evals.ps1 as the primary entry point for the portable eval handoff workflow. Generates a self-contained evaluation package outside the repository: for each eval, writes with-skill.prompt.md with inlined skill instructions, without-skill.prompt.md with the identical task and no skill, eval-metadata.json with expected output and assertions, fixture files, and result stubs. Supports -CollectResults to validate returned results and write a deterministic comparison without executing any prompt or calling a model.
Reorganized the portable eval handoff section with three subsections: 'Asking for an eval' documents trigger phrases and -Changed mode behavior, 'Eval preparation is a completion gate' specifies that skill changes require auto-running the eval preparation script, and 'Manual execution boundary' clarifies the handoff point where repository stops and user control takes over. Updated blocking completion gates to sequence eval preparation before sync-skill-install since the latter must observe final state.
Added release notes for -Changed mode and eval preparation as a blocking completion gate, clarifying that adding or modifying a skill triggers automatic evaluation. Updated README to explain that eval preparation is a completion gate an agent cannot skip, and documented how to use -Changed to resolve every touched skill or -Skill to prepare one on demand.
Updated contributor guidelines to clarify that eval preparation is a blocking completion gate that runs after the last skill edit and before sync-skill-install. Documented the -Changed mode for resolving every touched skill and added the requirement to report prepared prompt paths before considering work complete.
Fixed a bug in Invoke-ChangedMode where an empty pipeline result would assign instead of an empty array by wrapping the result in @(). Added assertions in validate-skill-templates.ps1 to verify that new documentation about eval triggers, completion gates, and -Changed mode is present. Added integration test for -Changed mode to ensure it resolves touched skills and reports scope correctly.
Extend prepare-skill-evals.ps1 to generate RUN-THIS.prompt.md, a single portable prompt that hands the entire eval package to an external agent in one paste. Update validate-skill-templates.ps1 to verify the handoff prompt exists, contains required guidance, and does not leak the grading key.
Explain the new RUN-THIS.prompt.md eval handoff in AGENTS.md, README.md, and CONTRIBUTING.md. Update guidance to show that eval packages now include a single portable prompt that consolidates the entire with-skill and without-skill package, reducing handoff complexity from twenty-six pastes to one.
Distinguish the agent that prepares eval packages from the harness that executes them. Establish .bot/<skill>-workspace/ as the default eval location for harnesses that must work inside the repository, backed by git ignore enforcement. Clarify when model execution is prohibited (scripts, gates, automation fan-out) versus authorized (explicit user handoff). Add executor authorization section and rules for executing handed packages.
Default eval packages to .bot/<skill>-workspace/, which git already ignores, allowing harnesses that must run inside the repository to work without polluting the working tree. Add Assert-WorkspaceLocation function to validate packages use .bot/ (when inside repo) or live outside entirely, with git ignore status check. Enforce these constraints when preparing packages. Update RUN-THIS.prompt.md to clarify executor authorization and scope boundaries. Enhance validation to check .bot/ usage and git ignore coverage.
Add CHANGELOG entries for RUN-THIS.prompt.md, explicit executor role separation, and .bot/ default location. Update CONTRIBUTING.md to reflect .bot/ as the default eval package home and clarify git ignore coverage. Update README.md to mention .bot/ location so packages remain invisible to git even when inside the repository.
Expand the RUN-THIS.prompt.md runner instructions with detailed guidance on executor roles, context freshness (including memory isolation), result capture, and handoff procedures. Clarify that a fresh context must exclude carried memory as well as transcript, that partial packages collect like complete ones, that the executor role cannot read the grading key, and that single-context harnesses should decline running when they have seen these instructions already. Add optional duration_seconds, total_tokens, and tool_calls fields to result shapes for finer performance analysis across runs.
Expand the Portable Eval Handoff section in AGENTS.md with deeper guidance on executor role separation, context isolation including memory boundaries, result handoff formats and contracts, the distinction between preparation and execution work, and how single-context harnesses fit into the workflow. Clarify that knowing the grading key is the only disqualifier for the preparer, that capability to spawn subagents is not a disqualifier for the executor, and that the runner file should be handed as an absolute path rather than pasted as text to preserve internal path resolution.
Update v0.9.0 release notes to document the portable eval handoff workflow, including the executor role separation, result handoff formats with optional performance metrics, guidance for single-context harnesses, and clarifications around context isolation and memory freshness requirements. These release notes accompany the new eval preparation and execution workflow introduced in this version.
Document the Portable Eval Handoff pattern that prepares paired candidate and baseline inputs as a portable package outside the repository. Emphasize the updated execution orchestration model where a user-selected harness creates isolated workers, the executor role separation that prevents repository scripts from running prompts, the Priority 1 AI/LLM Evaluation Automation Prohibition, blind worker prompts that omit eval metadata, and the optional metrics collection for duration, tokens, tool calls, and transcript. Update CHANGELOG.md release notes, CONTRIBUTING.md eval instructions, README.md eval workflow section, and AGENTS.md governance rules.
Implement Portable Eval Handoff framework in prepare-skill-evals.ps1 with blind worker prompts that omit eval headers, configuration labels, and grading criteria. Refactor prompt generation to create self-contained task descriptions without metadata that would signal to the model that it is under evaluation. Add optional metrics fields (transcript, duration_seconds, total_tokens, tool_calls) to result stubs. Implement manifest.json that lists prompt and result file paths for each eval. Refactor runner prompt to position the selected agent as an orchestrator that creates isolated workers and never executes eval prompts in its own context. Update -CollectResults to report available run metrics in a new metrics table. Change default output root from temp directory to gitignored .bot/<skill>-workspace/ storage.
Update AGENTS.md, CONTRIBUTING.md, and README.md to reflect the new eval infrastructure design where evaluation packages contain isolated run directories (with_skill/ and without_skill/) that workers execute from. This clarifies hermetic execution boundaries, fixture staging, and how the grading key remains outside worker access.
Update prepare-skill-evals.ps1 preparation logic and validate-skill-templates.ps1 validator to support the new run directory structure with schema version 2. Add validation for workspace configuration and isolation requirements (fresh_context_required, isolated_home_required, isolated_cwd_required) to ensure harnesses execute workers hermetically.
Add evaluation case for dotnet-remote-testing skill that tests the multi-targeted failing scenario. Includes fixture project structure and evals.json configuration to validate skill performance on complex test scenarios with multiple target frameworks and failures.
Added generate-eval-report.ps1 to produce self-contained HTML review and JSON benchmark reports for eval packages. The script reads manifest, eval metadata, and result files from a completed iteration and generates report.html plus benchmark.json without invoking any model. This enables the portable eval handoff workflow where external evaluators run paired configurations, record and grade results, then use this tool to generate the formal comparison report. Integrated into prepare-skill-evals.ps1 -CollectResults and updated validation to ensure report artifacts are present.
Updated CHANGELOG.md with v0.9.0 release notes documenting the portable eval handoff workflow enhancements, including the new self-contained report generation tool and clarified evaluation methodology. Release represents the finalization of the portable eval package design where external evaluators run paired configurations, record results with grading, and generate formal comparison reports deterministically without repository-side model invocation.
Updated AGENTS.md with clarified portable eval handoff workflow, emphasizing the separation of preparation, execution, and reporting phases. Specified that external evaluators read package instructions, create isolated workers, record results with grading, and generate reports deterministically. Reinforced that repository-side automation remains non-model-invoking and deterministic. Updated CONTRIBUTING.md and README.md to reflect current skills inventory, repository conventions, and governance rules for agent customization and skill development.
Update AGENTS.md, CONTRIBUTING.md, and README.md to document how eval packages now carry the exact Anthropic skill-creator assets (grader, aggregator, and eval-viewer). Clarify that the external evaluator invokes the packaged skill-creator tools instead of a standalone report generator, and update contribution guidance for skill authors.
Document the patch release that integrates Anthropic skill-creator aggregator and eval-viewer into the portable eval package workflow. The external evaluator now produces report.html, benchmark.json, and benchmark.md using the exact upstream skill-creator tools instead of a standalone generator.
Refactor eval package preparation and reporting to use Anthropic's upstream skill-creator assets: Resolve and stage the exact grader, aggregator, and eval-viewer assets into prepared packages. Update generate-eval-report.ps1 adapter to stage portable results into skill-creator's workspace contract and invoke aggregate_benchmark.py and eval-viewer/generate_review.py --static. Add Resolve-SkillCreatorSourcePath and Copy-SkillCreatorEvalTools functions to locate and stage skill-creator assets. Update prepare-skill-evals manifest to include skill_creator tool paths and new benchmark_markdown output. Update validate-skill-templates to verify presence of all required skill-creator artifacts in prepared packages.
Update validate-skill-templates workflow to fetch Anthropic skill-creator eval assets from the upstream skills repository at a known commit, extract the skill-creator path, and expose it via SKILL_CREATOR_PATH environment variable for the validation script.
Minor wording refinements in AGENTS.md, CONTRIBUTING.md, and README.md to clarify the dual-report approach: the adapter produces both a first-party paired comparison report and the upstream skill-creator viewer report for compatibility and inspection.
Enhance generate-eval-report.ps1 to produce a first-party side-by-side comparison report alongside the upstream skill-creator viewer. The new report displays paired with_skill and without_skill outputs, formal grades with evidence, optional run telemetry (duration, tokens, tool calls), transcripts, isolation evidence, and human feedback collection. Add eval-report-template.html with dual-pane layout for paired result comparison. Add helper functions for handling output files (text, image, binary with data URIs) and grading display. Update manifest in prepare-skill-evals to include the new benchmark_markdown output artifact. Update validate-skill-templates to confirm presence of all skill-creator files and validate the dual-report contract in prepared packages.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request updates the documentation to clarify and strengthen the requirements for synchronizing repo-managed skills across local installs. The changes emphasize that the repository is the source of truth, and introduce a PowerShell script (
scripts/sync-skill-install.ps1) as the authoritative way to sync and verify skill installations. The documentation now treats running this script as a mandatory, blocking completion gate, and describes how to use it, why partial or remembered file lists are insufficient, and what counts as proof of sync.Key documentation improvements:
Skill synchronization process:
pwsh -NoProfile -File ./scripts/sync-skill-install.ps1 -Skill <name>, which syncs the entire skill tree to all local installs and verifies SHA-256 hashes across all locations. The script also supports verification-only and pruning of orphaned files.Blocking completion gate:
Changelog update: