Is your feature request related to a problem?
Fast evaluation is currently limited to small datasets, which restricts the ability to effectively utilize direct LLM calls for evaluations, causing inefficiencies.
Describe the solution you'd like
- Increase
EVAL_FAST_MAX_UNIQUE_ROWS from 10 to 100
- Allow up to ~500 total items per fast run with a default duplication factor of 5
- Maintain eligibility checks based on unique rows
- Monitor API concurrency and memory usage during testing
- Confirm failure threshold is appropriate at the new scale
- Follow rollout plan to test and deploy the changes.
Is your feature request related to a problem?
Fast evaluation is currently limited to small datasets, which restricts the ability to effectively utilize direct LLM calls for evaluations, causing inefficiencies.
Describe the solution you'd like
EVAL_FAST_MAX_UNIQUE_ROWSfrom10to100