Skip to content

fix(planner): pass deployment accuracy into Pass 1 - #551

Closed
zzylol wants to merge 1 commit into
stack/528-08-cleanupfrom
stack/528-09-accuracy
Closed

zzylol wants to merge 1 commit into
stack/528-08-cleanupfrom
stack/528-09-accuracy

Conversation

@zzylol

@zzylol zzylol commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Problem: Pass 1 ignores the deployment's accuracy model, so it drops candidates that model would certify

#509 makes accuracy a deployment input. §Goal says the deployment supplies "its empirical cost model, empirical accuracy model and capabilities", and Assumption 5 says ASAPPlanner "uses the deployment's cost and accuracy models, or built-in defaults when the deployment supplies none". §Pass 1: Local candidate generation keeps "every candidate that is not provably unable to meet its accuracy requirement". §Scenarios, "Adding a better cost or accuracy estimation", allows the model to change the candidate set "where a model also changes sizing or admits a summary that has no built-in guarantee". #509 Example 2 is that case: one UnivMon state answers Distinct, Entropy and L2.

Before this PR, MajorPass used two different accuracy models:

MajorPass::optimize
├── Pass 1 strategies   default_strategies_with_evidence(cost, evidence)
│                       └── ASAPStrategies / HydraGroupingStrategy use &DEFAULT_ACCURACY_MODEL  ← built-in
└── root target check   search_workload_with_targets(roots, &strategies, models.accuracy)   ← deployment

PlanningModels::with_accuracy therefore reached only the final root check. A candidate that the built-in model cannot certify is already gone by then.

The built-in model gives UnivMon a guarantee for one statistic only (accuracy/estimators/univmon.rs):

pub(super) fn guarantee(query: &SketchStatistic) -> Option<ResultGuarantee> {
    matches!(query, SketchStatistic::PointCount { value: None, .. })
        .then(|| ResultGuarantee::exact("univmon_unit_update_total"))
}
Query (ε = 0.02) UnivMon statistic Built-in guarantee Pass 1 result before this PR
distinct_over_time(m[5m]) Cardinality None UnivMon candidate rejected
entropy_over_time(m[5m]) FrequencyEntropy None UnivMon candidate rejected
l2_over_time(m[5m]) FrequencyL2 None UnivMon candidate rejected

A deployment model that does certify these statistics could not change this. The shared-UnivMon test from #519 had to rebuild MajorPass by hand (lower, search_workload_with_targets, global_selection_with_summary_maintenance_lifecycles, assemble, share_common_sub_dags) so it could pass its own model to ASAPStrategies.

Scope. This PR covers the #509 requirement that Pass 1 uses the deployment accuracy model. It does not add a calibrated UnivMon error model, does not change sizing rules, and does not change plan selection. Native execution of the UnivMon readouts is #552.

Proposed method

  1. Add default_strategies_with_models(cost, accuracy, evidence). It builds the same five default strategies as before. ASAPStrategies and HydraGroupingStrategy now receive the caller's AccuracyModel instead of &DEFAULT_ACCURACY_MODEL.
  2. Keep default_strategies_with_evidence(cost, evidence) with the same signature. It now calls default_strategies_with_models with &DEFAULT_ACCURACY_MODEL, so callers that do not supply a model keep the built-in behavior.
  3. MajorPass::optimize (Pass 1 entry, pass/major.rs) builds its strategies with models.accuracy. Pass 1 candidate construction and the final root check now use the same model.

No new types. The allocator stays DEFAULT_ALLOCATOR (EqualSplitAllocator).

Pipeline position: Pass 1 (logical candidate generation). Pass 2 sharing, lifecycle selection and assembly are unchanged.

Key code interfaces

crates/asap-aware-mapping/src/replacement.rs:

/// Pass 1 must use the same accuracy algebra as final root validation
/// or it can discard candidates the deployment can certify.
pub fn default_strategies_with_models<'a>(
    cost_model: &'a dyn CostModel,
    accuracy_model: &'a dyn AccuracyModel,
    evidence: &'a dyn AccuracyEvidenceProvider,
) -> Vec<Box<dyn ReplacementStrategy + 'a>>;

// Unchanged signature; now a wrapper.
pub fn default_strategies_with_evidence<'a>(
    cost_model: &'a dyn CostModel,
    evidence: &'a dyn AccuracyEvidenceProvider,
) -> Vec<Box<dyn ReplacementStrategy + 'a>> {
    default_strategies_with_models(cost_model, &DEFAULT_ACCURACY_MODEL, evidence)
}

crates/asap-aware-mapping/src/pass/major.rs, in MajorPass::optimize:

let strategies =
    default_strategies_with_models(models.cost, models.accuracy, models.evidence);
// ...
let space = search_workload_with_targets(roots, &strategies, models.accuracy);

The models come from the existing PlanningModels (pass/mod.rs, unchanged):

pub struct PlanningModels<'a> {
    pub cost: &'a dyn CostModel,
    pub accuracy: &'a dyn AccuracyModel,
    pub evidence: &'a dyn AccuracyEvidenceProvider,
}

Usage, as in the new test:

PlanningModels::builtin()
    .with_cost(&CHEAP_SUMMARY)
    .with_accuracy(&UnivMonEvidence)

Fields

default_strategies_with_models:

Parameter / output Type Meaning
cost_model &'a dyn CostModel Ranks and binds candidates. Passed to ASAPStrategies, HydraGroupingStrategy and ExactCompositionStrategy.
accuracy_model &'a dyn AccuracyModel Gives local guarantees (local_guarantee), composes them (propagate) and checks targets (satisfies) while Pass 1 builds summary candidates. Passed to ASAPStrategies and HydraGroupingStrategy. MajorPass passes models.accuracy, the same model it passes to the root check.
evidence &'a dyn AccuracyEvidenceProvider Typed planning-time evidence (for example TopK membership, Hydra shared-grid composition). Passed to ASAPStrategies and HydraGroupingStrategy.
return Vec<Box<dyn ReplacementStrategy + 'a>> In order: ASAPStrategies, HydraGroupingStrategy, SharedSubDAGStrategy, AvgToSumOverCountStrategy, ExactCompositionStrategy. Same list as default_strategies_with_evidence returned before.

default_strategies_with_evidence: cost_model and evidence as above. The accuracy model is fixed to DefaultAccuracyModel. Behavior is unchanged for its callers.

PlanningModels (unchanged; listed because this PR changes where accuracy is used):

Field Type Meaning Set by
cost &dyn CostModel Deployment cost model. PlanningModels::new, with_cost; builtin() uses DefaultCostModel.
accuracy &dyn AccuracyModel Deployment accuracy model. Now used by Pass 1 and by the root check. new, with_accuracy; builtin() uses DefaultAccuracyModel.
evidence &dyn AccuracyEvidenceProvider Deployment accuracy evidence. new, with_evidence; builtin() uses NoAccuracyEvidence.

Examples

#509 Example 2 through e2e_plan

Test: e2e_certified_frequency_evaluations_share_one_univmon_state in crates/planner/tests/summary_sharing.rs. It replaces the hand-built certified_frequency_evaluations_share_one_univmon_state.

Input. Three PromQL queries, each with AccuracyTarget::Epsilon(0.02):

distinct_over_time(m[5m])
entropy_over_time(m[5m])
l2_over_time(m[5m])

Models: PlanningModels::builtin().with_cost(&CHEAP_SUMMARY).with_accuracy(&UnivMonEvidence). UnivMonEvidence is a synthetic test model. For any UnivMon family it returns a guarantee with metric RelativeValue, bound 0.01 and failure probability 0.01. For every other family, and for propagate and satisfies, it defers to DefaultAccuracyModel. It exercises sharing, not runtime accuracy.

What this PR changes. MajorPass passes UnivMonEvidence into ASAPStrategies. The UnivMon candidate for each query now gets a guarantee in Pass 1 and stays in the candidate set. Selection and Pass 2 sharing then work as before.

Output. The test checks:

  • output.plans.len() == 3.
  • unique_deployments(&output) == 1: one deployed state across the workload, by pointer.
  • Each plan deploys exactly one state, and the plans' states are the same Rc.
  • Each plan root is ASAPOp::SummaryEstimate whose summary_input is ASAPOp::SummaryAgg with family Sketch(UnivMon, _).
  • Each root guarantee is Some and satisfies Epsilon(0.02) under DefaultAccuracyModel.satisfies.
SummaryEstimate(Cardinality) ─┐
SummaryEstimate(Entropy)     ─┼─> SummaryAgg(UnivMon)   (one Rc, one deployment)
SummaryEstimate(L2)          ─┘

Which model each step uses

Caller Pass 1 strategies use Root check uses
MajorPass before this PR DefaultAccuracyModel models.accuracy
MajorPass after this PR models.accuracy models.accuracy
default_strategies_with_evidence(cost, evidence) DefaultAccuracyModel (unchanged) caller's choice
default_strategies_with_models(cost, acc, evidence) acc caller's choice

Out of scope

Stack and validation

Stack 9 · Base: #543 · Next: #552 · Closes #523. Continues the #528 review stack after #543.

Validation: cargo test --locked -p asap-planner --test summary_sharing e2e_certified_frequency_evaluations_share_one_univmon_state.

🤖 Generated with Claude Code

@zzylol

zzylol commented Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

Closing: this passes the deployment's accuracy model into Pass 1, which conflicts with the agreed #572 decision (Q22): Stage 1 uses only the planner's built-in analytical rules; the deployment's accuracy model is used in Stage 3 (plan-selection) only, and Pass 1 keeps candidates without an analytical rule with guarantee = None instead of pruning them. See #572 "Accuracy placement".

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant