feat(evaluation): Improve summary readability by Ayush8923 · Pull Request #1095 · ProjectTech4DevAI/kaapi-backend

Ayush8923 · 2026-07-30T03:09:32Z

Issue

Summary

In this PR,

Added weighted overall scores for judge evaluations, including per-area scores, comparisons, and quality verdicts.
Added AI-generated summaries for judge runs, with graceful handling when summaries are unavailable.

API response of overall:

{
    "verdict": "Needs Refinement",
    "breakdown": [
        {
            "key": "ground_truth",
            "name": "Adherence to Ground Truth",
            "delta": -0.01,
            "score": 0.57,
            "weight": 0.5,
            "verdict": "Needs Refinement"
        },
        {
            "key": "prompt",
            "name": "Adherence to Prompt",
            "delta": -0.14,
            "score": 0.44,
            "weight": 0.2,
            "verdict": "Needs Refinement"
        },
        {
            "key": "knowledge_base",
            "name": "Adherence to Knowledge Base",
            "delta": 0.11,
            "score": 0.69,
            "weight": 0.3,
            "verdict": "Good"
        }
    ],
    "ai_summary": "Needs refinement overall, with the assistant showing uneven performance. It was generally backed by the source material but the answers varied, and it did not consistently match the expected answers or follow the requested tone and instructions. Each question was asked once.",
    "overall_score": 0.58
}

Checklist

Before submitting a pull request, please ensure that you mark these task.

Ran fastapi run --reload app/main.py or docker compose up in the repository root and test.
If you've fixed a bug or added code that is tested and has test cases.

coderabbitai · 2026-07-30T03:09:41Z

📝 Walkthrough

Walkthrough

V2 judge evaluations now calculate weighted overall scores, assign qualitative verdicts to metric traces, generate optional AI summaries, persist these fields, and preserve them during cached or resynchronized score reconstruction.

Changes

Evaluation scoring and summary flow

Layer / File(s)	Summary
Scoring contracts and weighted aggregation `backend/app/crud/evaluations/judge.py`, `backend/app/crud/evaluations/score.py`, `backend/app/tests/crud/evaluations/test_overall_summary.py`	Judge metrics define weights; score types and helpers implement verdict bands, weighted overall summaries, renormalized dimensions, and boundary tests.
Qualitative AI summary generation `backend/app/core/config.py`, `backend/app/crud/evaluations/summary.py`, `backend/app/tests/crud/evaluations/test_run_ai_summary.py`	Adds the configurable summary model and generates trimmed, best-effort summaries with non-fatal handling for provider, empty-response, and malformed-payload failures.
Fast judge pipeline and score persistence `backend/app/crud/evaluations/fast.py`, `backend/app/crud/evaluations/core.py`, `backend/app/tests/crud/evaluations/test_fast_judge.py`	Fast judge runs resolve duplication metadata, attach metric verdicts, compute and summarize overall scores, return them, and persist them with completed runs and uploaded traces.
Score reconstruction and preservation `backend/app/services/evaluations/evaluation.py`, `backend/app/tests/services/evaluations/test_evaluation_service_s3.py`	Cached and resynchronized trace reconstruction retains the stored overall block, with coverage for both read paths.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant FastEvaluation
  participant SummaryGenerator
  participant EvaluationRun
  participant ScoreStorage
  participant EvaluationService
  FastEvaluation->>SummaryGenerator: generate AI summary from overall scores
  SummaryGenerator-->>FastEvaluation: summary text or None
  FastEvaluation->>EvaluationRun: persist summary_scores and overall
  FastEvaluation->>ScoreStorage: persist trace scores and overall
  EvaluationService->>ScoreStorage: reconstruct cached or merged traces
  ScoreStorage-->>EvaluationService: traces with preserved overall

Possibly related PRs

ProjectTech4DevAI/kaapi-backend#1094: Adds overlapping verdict fields in the v2 judge scoring pipeline.
ProjectTech4DevAI/kaapi-backend#1058: Introduces the prompt metric that this change weights for overall scoring.
ProjectTech4DevAI/kaapi-backend#746: Modifies the same score persistence flow extended here for overall data.

Suggested labels: enhancement, released on @main``

Suggested reviewers: akhileshnegi, vprashrex, prajna1999

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 20.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The changes implement the requested human-readable summary, weighted overall score, breakdown, badge bands, and score-band legend.
Out of Scope Changes check	✅ Passed	The added config, scoring, persistence, and tests all support the evaluation summary feature and appear in scope.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title relates to the new AI-generated evaluation summaries but does not mention the primary weighted overall scoring changes.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch feat/evals-human-readable-summary-generation

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-07-30T03:10:56Z

OpenAPI changes ⚪ No API surface changes

Note

This PR does not modify the API contract.

_{main ↔ d83527ce · generated by oasdiff}

codecov · 2026-07-30T03:32:28Z

Codecov Report

❌ Patch coverage is 99.42857% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
backend/app/crud/evaluations/core.py	50.00%	1 Missing ⚠️
backend/app/crud/evaluations/summary.py	98.27%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

coderabbitai

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@backend/app/crud/evaluations/fast.py`:
- Around line 910-926: Update _resolve_duplication_factor to defensively handle
invalid DATASET_META_DUPLICATION_FACTOR metadata, including None and non-numeric
values, by falling back to 1 while preserving the minimum value of 1 for valid
inputs. Ensure malformed metadata cannot propagate an exception or fail the
evaluation run.

In `@backend/app/crud/evaluations/summary.py`:
- Around line 126-176: Move the `extract_response_text(response).strip()` and
empty-summary handling inside the existing `try` block in
`generate_run_ai_summary`, so malformed Responses payloads are caught by the
broad exception handler and return `None`. Preserve the current logging and
successful non-empty summary behavior.

In `@docs/wiki/modules/evaluations.md`:
- Line 25: Update the EvaluationRun documentation to include the run-level
score["overall"] contract: weighted score, verdict, breakdown, and optional AI
summary. Document EVAL_JUDGE_MODEL and EVAL_SUMMARY_MODEL as configurable
default model settings rather than stating that judging always uses the fixed
gpt-5-mini fallback, while preserving the existing trace-level verdict details.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 164d28d5-1eb1-4208-b6b2-ae82abfae480

📥 Commits

Reviewing files that changed from the base of the PR and between 89132d7 and 6900d8a.

📒 Files selected for processing (13)

backend/app/core/config.py
backend/app/crud/evaluations/core.py
backend/app/crud/evaluations/fast.py
backend/app/crud/evaluations/judge.py
backend/app/crud/evaluations/score.py
backend/app/crud/evaluations/summary.py
backend/app/services/evaluations/evaluation.py
backend/app/tests/crud/evaluations/test_fast_judge.py
backend/app/tests/crud/evaluations/test_overall_summary.py
backend/app/tests/crud/evaluations/test_run_ai_summary.py
backend/app/tests/crud/evaluations/test_score.py
backend/app/tests/services/evaluations/test_evaluation_service_s3.py
docs/wiki/modules/evaluations.md

coderabbitai

Caution

Inline review comments failed to post. This is likely due to GitHub's internal server error or limits when posting large numbers of comments. If you are seeing this consistently it is likely a permissions issue. Please check "Moderation" -> "Code review limits" under your organization settings.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@backend/app/crud/evaluations/fast.py`:
- Around line 910-926: Update _resolve_duplication_factor to defensively handle
invalid DATASET_META_DUPLICATION_FACTOR metadata, including None and non-numeric
values, by falling back to 1 while preserving the minimum value of 1 for valid
inputs. Ensure malformed metadata cannot propagate an exception or fail the
evaluation run.

In `@backend/app/crud/evaluations/summary.py`:
- Around line 126-176: Move the `extract_response_text(response).strip()` and
empty-summary handling inside the existing `try` block in
`generate_run_ai_summary`, so malformed Responses payloads are caught by the
broad exception handler and return `None`. Preserve the current logging and
successful non-empty summary behavior.

In `@docs/wiki/modules/evaluations.md`:
- Line 25: Update the EvaluationRun documentation to include the run-level
score["overall"] contract: weighted score, verdict, breakdown, and optional AI
summary. Document EVAL_JUDGE_MODEL and EVAL_SUMMARY_MODEL as configurable
default model settings rather than stating that judging always uses the fixed
gpt-5-mini fallback, while preserving the existing trace-level verdict details.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 164d28d5-1eb1-4208-b6b2-ae82abfae480

📥 Commits

Reviewing files that changed from the base of the PR and between 89132d7 and 6900d8a.

📒 Files selected for processing (13)

backend/app/core/config.py
backend/app/crud/evaluations/core.py
backend/app/crud/evaluations/fast.py
backend/app/crud/evaluations/judge.py
backend/app/crud/evaluations/score.py
backend/app/crud/evaluations/summary.py
backend/app/services/evaluations/evaluation.py
backend/app/tests/crud/evaluations/test_fast_judge.py
backend/app/tests/crud/evaluations/test_overall_summary.py
backend/app/tests/crud/evaluations/test_run_ai_summary.py
backend/app/tests/crud/evaluations/test_score.py
backend/app/tests/services/evaluations/test_evaluation_service_s3.py
docs/wiki/modules/evaluations.md

🛑 Comments failed to post (1)

docs/wiki/modules/evaluations.md (1)
25-25: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Document the run-level overall contract and configurable models.

This only documents trace verdicts. Add score["overall"] (weighted score, verdict, breakdown, optional AI summary) and describe EVAL_JUDGE_MODEL / EVAL_SUMMARY_MODEL as configurable defaults, not an always-fixed fallback model.

As per coding guidelines, “When changing a module's routes, tables, models, or services, update that module's wiki page in the same pull request.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/wiki/modules/evaluations.md` at line 25, Update the EvaluationRun
documentation to include the run-level score["overall"] contract: weighted
score, verdict, breakdown, and optional AI summary. Document EVAL_JUDGE_MODEL
and EVAL_SUMMARY_MODEL as configurable default model settings rather than
stating that judging always uses the fixed gpt-5-mini fallback, while preserving
the existing trace-level verdict details.
Source: Coding guidelines

Ayush8923 added 2 commits July 29, 2026 23:33

feat(*): implement the evals human readable summary generation

194c8f5

fix(*): update the ai_summary code

6900d8a

Ayush8923 self-assigned this Jul 30, 2026

Ayush8923 added the ready-for-review label Jul 30, 2026

github-actions Bot changed the title ~~feat(evaluation): Human-readable summary generation~~ feat(evaluation): Improve summary readability Jul 30, 2026

Ayush8923 changed the base branch from main to feat/evals-verdict-field July 30, 2026 03:10

coderabbitai Bot reviewed Jul 30, 2026

View reviewed changes

Comment thread backend/app/crud/evaluations/fast.py

Comment thread backend/app/crud/evaluations/summary.py

coderabbitai Bot reviewed Jul 30, 2026

View reviewed changes

fix(evals): clenaups

5b1074b

Ayush8923 changed the base branch from feat/evals-verdict-field to main July 30, 2026 08:59

Ayush8923 changed the base branch from main to feat/evals-verdict-field July 30, 2026 08:59

fix(evals): clenaups

28e3b72

Ayush8923 requested a review from AkhileshNegi July 30, 2026 13:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(evaluation): Improve summary readability - #1095

feat(evaluation): Improve summary readability#1095
Ayush8923 wants to merge 4 commits into
feat/evals-verdict-fieldfrom
feat/evals-human-readable-summary-generation

Ayush8923 commented Jul 30, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Jul 30, 2026 •

edited

Loading

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

github-actions Bot commented Jul 30, 2026

Uh oh!

codecov Bot commented Jul 30, 2026

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

Ayush8923 commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue

Summary

Checklist

Uh oh!

coderabbitai Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

github-actions Bot commented Jul 30, 2026

OpenAPI changes ⚪ No API surface changes

Uh oh!

codecov Bot commented Jul 30, 2026

Codecov Report

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Ayush8923 commented Jul 30, 2026 •

edited

Loading

coderabbitai Bot commented Jul 30, 2026 •

edited

Loading