Complete OpenAI plugin review fixtures - #797
Merged
Merged
Conversation
|
I hit an error while handling your request (Model unavailable on AI Gateway free tier: Free tier users do not have access to this model. Upgrade to paid credits at https://vercel.com/d?to=%2F%5Bteam%5D%2F%7E%2Fai%3Fmodal%3Dtop-up for unrestricted…). Please try again, rephrase, or reach out if it keeps failing. Error id: 2a946342-7089-48a2-84e9-670144810dcc |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
OpenAI's public plugin review requires positive cases to include reproducible fixtures and expected result shapes, and negative cases to explain why the plugin should refuse, clarify, or fall back. The existing cross-skill suite had prompts and expected behavior but omitted those reviewer fields.
This follow-up adds the missing material for all cases, includes an initial-submission release-note draft, and makes the validator enforce the required positive and negative fields.
Validation:
bun scripts/validate-skills.tsclaude plugin validate . --strictgit diff --checkThis prepares review materials only. It does not submit or publish the plugin.
Note
Low Risk
Changes are limited to evaluation JSON, release-note copy, and package validation rules; no skill runtime or MCP behavior is modified.
Overview
Aligns the cross-skill publishing eval suite with OpenAI plugin review expectations by adding reviewer-ready metadata on every case and locking it in with validation.
Each positive case in
publishing-cases.jsonnow includesexpected_result_shapeandrequired_fixture(reproducible setup). Each negative case includeswhy_notexplaining why the plugin must refuse or fall back. A new negative case (furniture-candidate-door-borrowing-negative) covers refusing to treatverify_sceneevidence as candidate-specific door access.scripts/validate-skills.tsfails the build if those fields are missing for the matchingkind.Adds
plugin-evals/release-notes.mdas a 0.1.5 skills-only submission draft (scope, MCP connection expectations, what is not bundled).plugin-evals/README.mdandskills/VALIDATION.mdare updated to describe the richer suite and the stored release-note draft.Reviewed by Cursor Bugbot for commit 0fe434e. Bugbot is set up for automated code reviews on this repo. Configure here.