Skip to content

feat: add labeled evaluation and observation hooks - #16

Open
gudcks0305 wants to merge 1 commit into
mainfrom
feat/evaluation-observation
Open

gudcks0305 wants to merge 1 commit into
mainfrom
feat/evaluation-observation

Conversation

@gudcks0305

Copy link
Copy Markdown
Owner

Add opt-in completion metadata on provider builders and a labeled evaluation example for choosing probability thresholds.

  • .observer(...) reports one logical evaluation's duration, requested/returned model, question count, usage, outcome, and bounded error kind/status. Events exclude keys, headers, endpoints, state, questions, answers, raw responses, and exception messages. Observer failures do not alter inference or cancellation; transport retries remain one observation.
  • LabeledEvaluationExample defaults to eight synthetic offline rows without credentials or network calls. Explicit --live mode sends only state, keeps gold labels local, and supports labeled JSONL files. Threshold reports distinguish errors and abstentions, compute coverage over all rows, and report accepted accuracy as N/A when nothing is accepted.
  • Add usage/lifecycle/metric documentation and mark these features unreleased. No logging framework, Micrometer binder, provider module, or default model fallback is introduced.

Validation:

  • ./mvnw -q verify: 127 tests passed, no failures/errors/skips.
  • Offline example ran successfully; its figures are synthetic demonstrations, not measured model quality.
  • Observation guide Java example compiled with javac --release 17; local documentation links checked.
  • Independent read-only review found no confirmed defects.

The compatibility documentation and repository AGENTS.md were merged separately in #15. This PR does not claim successful live inference on additional gateways.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant