feat(evals): Added verdict categorization system

[Ayush8923]

Issue

Closes #1092

Summary

• Adds a per-metric verdict band to each row of an Evaluation v2 (judge) run, so results are scannable at a glance (pass / borderline / fail) instead of forcing users to interpret raw 0–1 scores.
• Each numeric judge-metric score in a row's scores (Adherence to Ground Truth / Prompt / Knowledge Base) now carries a verdict derived from that score:

1. 0 ≤ score < 0.3 --> Needs Improvement
2. 0.3 ≤ score < 0.6 --> Needs Refinement
3. 0.6 ≤ score ≤ 1 --> Good

Checklist

Before submitting a pull request, please ensure that you mark these task.

• Ran fastapi run --reload app/main.py or docker compose up in the repository root and test.
• If you've fixed a bug or added code that is tested and has test cases.

[coderabbitai]

📝 Walkthrough

Walkthrough

Numeric v2 judge metric trace scores now include verdict bands derived from rounded scores. The score module defines thresholds and enum values, fast evaluation populates verdicts, tests cover boundaries and exclusions, and evaluation documentation describes the contract.

Changes

Evaluation verdict bands

Layer / File(s)	Summary
Verdict mapping and trace schema backend/app/crud/evaluations/score.py, backend/app/tests/crud/evaluations/test_score.py	Defines VerdictEnum, 0.3/0.6 thresholds, score mapping behavior, and an optional TraceScore.verdict field with boundary and serialization tests.
Trace verdict population and validation backend/app/crud/evaluations/fast.py, backend/app/tests/crud/evaluations/test_fast_judge.py, docs/wiki/modules/evaluations.md	Adds verdicts to numeric v2 trace scores, verifies exclusions for cosine and N/A entries, and documents the resulting trace behavior.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• ProjectTech4DevAI/kaapi-backend#1004: Adds native LLM judge numeric scores to fast evaluation trace scores, which this change extends with verdict bands.
• ProjectTech4DevAI/kaapi-backend#1058: Adds the prompt-adherence judge metric in the same fast evaluation scoring pipeline.

Suggested reviewers: akhileshnegi

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The changes add verdict mapping, surface the field on v2 judge trace scores, and update tests/docs to match the requested thresholds.
Out of Scope Changes check	✅ Passed	The PR stays focused on verdict support for evaluation rows; the tests and docs changes are directly related.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly reflects the new verdict categorization added for evaluation scores.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/evals-verdict-field

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

OpenAPI changes   ⚪ No API surface changes

Note

This PR does not modify the API contract.

_{main ↔ 8c9df40d · generated by oasdiff}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

backend/app/crud/evaluations/score.py (1)

14-14: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Use StrEnum to satisfy Ruff UP042.

Line 14 triggers the configured Ruff rule; StrEnum preserves the existing string serialization and is available with the required Python 3.11+ baseline.

Proposed fix

-from enum import Enum
+from enum import StrEnum

-class VerdictEnum(str, Enum):
+class VerdictEnum(StrEnum):

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/app/crud/evaluations/score.py` at line 14, Update VerdictEnum to
inherit from the standard library StrEnum instead of combining str with Enum,
preserving its existing members and string serialization behavior while
satisfying Ruff UP042.

Sources: Coding guidelines, Linters/SAST tools

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@backend/app/tests/crud/evaluations/test_fast_judge.py`:
- Around line 1091-1185: Add return annotations to the three test methods in
backend/app/tests/crud/evaluations/test_fast_judge.py lines 1091-1185, and
annotate the nested _judge(params) callback parameter and return type. In
backend/app/tests/crud/evaluations/test_score.py lines 25-42, add self:
TestVerdictFromScore to each TestVerdictFromScore test method.

---

Nitpick comments:
In `@backend/app/crud/evaluations/score.py`:
- Line 14: Update VerdictEnum to inherit from the standard library StrEnum
instead of combining str with Enum, preserving its existing members and string
serialization behavior while satisfying Ruff UP042.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 16cc1f72-0090-4fdf-b809-aab74f6fb049

📥 Commits

Reviewing files that changed from the base of the PR and between 89132d7 and 6cd310b.

📒 Files selected for processing (5)

• backend/app/crud/evaluations/fast.py
• backend/app/crud/evaluations/score.py
• backend/app/tests/crud/evaluations/test_fast_judge.py
• backend/app/tests/crud/evaluations/test_score.py
• docs/wiki/modules/evaluations.md

[codecov]

Codecov Report

❌ Patch coverage is 98.36066% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...kend/app/tests/crud/evaluations/test_fast_judge.py	97.14%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!