Skip to content

Complete OpenAI plugin review fixtures - #797

Merged
Aymericr merged 1 commit into
mainfrom
chore/openai-review-materials-20260909
Sep 9, 2026
Merged

Aymericr merged 1 commit into
mainfrom
chore/openai-review-materials-20260909

Conversation

@Aymericr

@Aymericr Aymericr commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

OpenAI's public plugin review requires positive cases to include reproducible fixtures and expected result shapes, and negative cases to explain why the plugin should refuse, clarify, or fall back. The existing cross-skill suite had prompts and expected behavior but omitted those reviewer fields.

This follow-up adds the missing material for all cases, includes an initial-submission release-note draft, and makes the validator enforce the required positive and negative fields.

Validation:

  • bun scripts/validate-skills.ts
  • claude plugin validate . --strict
  • git diff --check

This prepares review materials only. It does not submit or publish the plugin.


Note

Low Risk
Changes are limited to evaluation JSON, release-note copy, and package validation rules; no skill runtime or MCP behavior is modified.

Overview
Aligns the cross-skill publishing eval suite with OpenAI plugin review expectations by adding reviewer-ready metadata on every case and locking it in with validation.

Each positive case in publishing-cases.json now includes expected_result_shape and required_fixture (reproducible setup). Each negative case includes why_not explaining why the plugin must refuse or fall back. A new negative case (furniture-candidate-door-borrowing-negative) covers refusing to treat verify_scene evidence as candidate-specific door access. scripts/validate-skills.ts fails the build if those fields are missing for the matching kind.

Adds plugin-evals/release-notes.md as a 0.1.5 skills-only submission draft (scope, MCP connection expectations, what is not bundled). plugin-evals/README.md and skills/VALIDATION.md are updated to describe the richer suite and the stored release-note draft.

Reviewed by Cursor Bugbot for commit 0fe434e. Bugbot is set up for automated code reviews on this repo. Configure here.

@pascal

pascal Bot commented Sep 9, 2026

Copy link
Copy Markdown

I hit an error while handling your request (Model unavailable on AI Gateway free tier: Free tier users do not have access to this model. Upgrade to paid credits at https://vercel.com/d?to=%2F%5Bteam%5D%2F%7E%2Fai%3Fmodal%3Dtop-up for unrestricted…).

Please try again, rephrase, or reach out if it keeps failing.

Error id: 2a946342-7089-48a2-84e9-670144810dcc

@Aymericr
Aymericr merged commit 7163fce into main Sep 9, 2026
3 checks passed
@Aymericr
Aymericr deleted the chore/openai-review-materials-20260909 branch September 9, 2026 16:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant