diff --git a/README.md b/README.md index 2fc779765..d2b3bf43e 100644 --- a/README.md +++ b/README.md @@ -5,6 +5,9 @@ The query backend of the **ASAP** observability system. +Design documents, developer workflows, and user guides are indexed in the +[ASAPQuery-backend documentation](docs/README.md). + ASAPQuery-backend exposes a **single PromQL HTTP surface** that internally dispatches to one of two engines based on the control plane's plan and the query's shape: @@ -233,7 +236,7 @@ that produced the current architecture: controller's L3 `intent_algebra` + L4 `sketch_algebra` - **JSONL cold-fallback path** — deleted; the configured exact backend is the explicit fallback described by - [`query-execution.md`](data_plane/docs/design_docs/query-execution.md). + [query-engine implementation design](docs/developer_docs/query-engine/design.md). - **`StorageBackend::ColdJsonlFallback`** enum variant — removed - **Backend-local cost-model line item for cold-tier scan bytes** — removed (controller's tier-spanning cost model is the source of diff --git a/control_plane/docs/README.md b/control_plane/docs/README.md deleted file mode 100644 index 6b33fc8d0..000000000 --- a/control_plane/docs/README.md +++ /dev/null @@ -1,37 +0,0 @@ -# ASAPQuery-backend control-plane design - -> Status: active - -## TL;DR - -These documents describe only the design owned by ASAPQuery-backend: its -integration boundary with ASAPPlanner, physical compilation for the collector -and backend, and the plan consumed by the ASAPQuery data plane. - -ASAPPlanner owns PromQL parsing, pre-ASAP and post-ASAP IR, query-to-summary -mapping, accuracy reasoning, workload sharing, candidate generation, and -logical plan selection. Refer to the -[ASAPPlanner repository](https://github.com/ProjectASAP/ASAPPlanner) for those -designs; they are intentionally not repeated here. - -## Documents - -| Document | ASAPQuery-backend-owned scope | -| --- | --- | -| [ASAPPlanner integration](asapplanner-integration.md) | Ownership boundary and integration contract for consuming a selected logical workload plan. | -| [Physical planning](physical-planning.md) | Placement, windows, representation, transmission, and compilation into matching collector and backend plans. | -| [BackendPlan](backend-plan.md) | Versioned contract installed and executed by the ASAPQuery data plane. | - -## Developer documentation - -- [Planner adapter and physical compiler](developer_docs/planner-and-physical-compiler.md) -- [Runtime plan publication](developer_docs/runtime-plan-publication.md) - -The corresponding collector-facing plan is documented by -[ASAPCollector](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/developer_docs/opamp-config-push.md). - -## Scope rule - -A design belongs here only when ASAPQuery-backend owns the decision or must -enforce the contract at runtime. If ASAPPlanner owns the decision, this -directory links to Planner instead of maintaining a second description. diff --git a/control_plane/docs/asapplanner-integration.md b/control_plane/docs/asapplanner-integration.md deleted file mode 100644 index 087760ad6..000000000 --- a/control_plane/docs/asapplanner-integration.md +++ /dev/null @@ -1,179 +0,0 @@ -# ASAPQuery-backend integration with ASAPPlanner - -> Status: proposed -> -> MVP relation: required for query planning and control-plane decisions. - -Developer guide: [Planner adapter and physical compiler](developer_docs/planner-and-physical-compiler.md). - -## TL;DR - -ASAPPlanner chooses a logical plan for a query workload. ASAPQuery-backend -turns that selected plan into deployable collector and backend plans, installs -them, and routes queries to the resulting state. - -ASAPQuery-backend does not maintain a second design for parsing PromQL, -building Planner IR, mapping queries to summaries, reasoning about accuracy, -sharing work across queries, or ranking logical candidates. Those designs are -owned by [ASAPPlanner](https://github.com/ProjectASAP/ASAPPlanner). - -```text -PromQL workload - | - v -ASAPPlanner -selected logical workload plan - | - v -ASAPQuery-backend control plane -physical compilation and activation - | - +-------------------------+ - | | - v v -CollectorPlan BackendPlan -ASAPCollector ASAPQuery data plane -``` - -## Ownership boundary - -### ASAPPlanner - -Planner is the source of truth for: - -- query parsing and semantic IR; -- logical exact and summary-based alternatives; -- summary family, parameters, grouping, and readout semantics; -- accuracy constraints and logical result guarantees; -- workload-wide reuse and common subexpressions; -- logical cost comparison and candidate selection; and -- logical rejection and explanation. - -See ASAPPlanner's own design documents for those semantics. Their types and -rules must be consumed from the pinned Planner revision, not copied into this -repository. - -### ASAPQuery-backend control plane - -The control plane owns: - -- supplying workload context, runtime statistics, and executor capabilities; -- invoking the pinned Planner revision and consuming one selected workload - plan; -- choosing collector/backend placement, physical panes, transmission mode, - and storage routes; -- compiling matching CollectorPlan and BackendPlan artifacts; -- staging, activating, retiring, and rolling back plan versions; and -- exposing planning and activation status to operators. - -These decisions are described in [physical planning](physical-planning.md). - -### ASAPQuery data plane - -The data plane owns: - -- installing [BackendPlan](backend-plan.md); -- validating, ingesting, storing, and merging summary state; -- applying the selected readout and remaining backend-side operators; -- enforcing readiness, freshness, and plan identity; and -- executing an explicit exact fallback when the selected plan requires one. - -### ASAPCollector - -ASAPCollector owns validation and execution of CollectorPlan: observing the -selected metrics, maintaining the requested state, transmitting raw data or -full/delta summaries, and reporting which plan is actually active. Its -interface is documented in the -[ASAPCollector repository](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/developer_docs/opamp-config-push.md). - -## Integration contracts - -### Workload request - -ASAPQuery-backend passes the complete workload to Planner so that sharing and -global selection remain possible. A request includes the query expressions, -evaluation shape, requested accuracy, and relevant schema or cost context. -Protocol-specific scheduling and alert state remain outside Planner. - -For example, two dashboard queries that read the same five-minute latency -distribution are planned together. ASAPQuery-backend must not reduce them to -two independent `(metric, sketch)` requests before Planner can identify shared -state. - -### Selected logical plan - -The returned boundary is one selected post-ASAP workload plan from the pinned -Planner API. Shared logical nodes remain shared. Exact fallback is an explicit -part of that plan; absence of a supported summary is not permission for the -backend to invent one. - -For example, Planner may select one DDSketch producer shared by: - -```promql -quantile_over_time(0.50, request_duration_seconds[5m]) -quantile_over_time(0.95, request_duration_seconds[5m]) -``` - -The backend consumes the shared producer and two readouts. It must not select -DDSketch again from the query strings. - -### Physical plans - -One physical compile produces CollectorPlan and BackendPlan from the same -selected logical plan. Both artifacts carry the same plan version and -materialization identities. Independent compilation is invalid because it can -make the collector and data plane disagree about family, parameters, grouping, -windows, or state representation. - -### Runtime evidence - -Delivery acknowledgement is not activation evidence. Query routing changes -only after the backend has installed BackendPlan and each required collector -has reported that it applied the matching CollectorPlan. Emitted state must -also carry identities that the backend can validate. - -## Planner version policy - -All Planner crates used by one backend build are pinned to one immutable -revision. The revision is recorded in compiled plans and diagnostic output. - -An upstream Planner change is adopted only after the backend has deliberately -handled any new IR variant, guarantee, capability, or cost-model requirement. -Unknown variants fail as unsupported; the backend must not pre-copy proposed -Planner types or infer semantics from explain/viewer output. - -Open Planner pull requests are compatibility inputs, not backend design. Their -details stay in Planner and in the implementation change that updates the pin, -instead of becoming a second long-lived specification here. - -## Failure behavior - -Planning or activation fails closed when: - -- the selected logical operation is unsupported by its assigned executor; -- collector and backend plans disagree on any materialization contract; -- a required accuracy guarantee is missing or insufficient; -- full/delta state semantics are incompatible; -- a plan is stale, expired, partially applied, or from another run; or -- exact fallback is required but unavailable. - -The system must not silently choose another summary, loosen an accuracy -requirement, remove grouping, or serve state from a different plan. - -## MVP requirements - -The integration is MVP-ready when one end-to-end run demonstrates that: - -- the submitted PromQL workload reaches Planner as one workload; -- the selected logical plan is preserved through physical compilation; -- matching collector and backend plans are captured as artifacts; -- ASAPCollector reports semantic application of the collector plan; -- the data plane serves supported summary-backed queries from BackendPlan; -- unsupported queries are rejected or use the selected exact fallback; and -- missing, stale, or incompatible plan evidence makes the run fail. - -## Non-goals - -This document does not define Planner IR, query-to-summary mappings, accuracy -algebra, summary algorithms, collector configuration fields, state byte -encoding, or query-engine implementation details. diff --git a/control_plane/docs/backend-plan.md b/control_plane/docs/backend-plan.md deleted file mode 100644 index a7b36b617..000000000 --- a/control_plane/docs/backend-plan.md +++ /dev/null @@ -1,317 +0,0 @@ -# BackendPlan: control-plane to data-plane contract - -> Status: proposed -> -> MVP relation: required for the ASAPQuery data plane to ingest and query the -> state selected by the control plane. -> -> Scope: the typed runtime contract by which ASAPQuery-backend's control -> plane tells its data plane what summary materializations exist, how they -> are ingested, and which query capabilities they serve. - -Developer guides: -[runtime plan publication](developer_docs/runtime-plan-publication.md) and -[BackendPlan installation](../../data_plane/docs/developer_docs/backend-plan-runtime.md). - -## TL;DR - -Planning chooses once; serving reuses that exact decision. - -`BackendPlan` is the backend half of a `CompiledPlan`. It describes the state -that matching CollectorPlans produce and the readout/routing decisions the -data plane must apply. It prevents the data plane from independently guessing -summary family, parameters, grouping, windows, accuracy, or storage. - -```text -selected Planner DAG - | - v -physical compile - / \ -CollectorPlan BackendPlan - | | -summary state ingest + route + readout -``` - -## 1. Why BackendPlan exists - -Control-plane planning and data-plane serving happen in different processes -and at different times. They must nevertheless agree on: - -- which materializations should exist; -- their exact summary state types; -- where their payloads come from; -- which windows and groups they represent; -- which queries/readouts they can answer; -- which guarantees apply; and -- which plan version is active. - -Without a typed plan, serving tends to reconstruct decisions from query text, -hard-coded defaults, or observed storage metadata. That can silently select a -different algorithm or parameters, miss valid state, merge incompatible -state, or serve under the wrong accuracy contract. - -BackendPlan makes the control-plane decision authoritative. - -## 2. Relationship to other plans - -ASAPPlanner supplies the selected logical post-ASAP DAG. ASAPQuery's physical -compiler creates: - -- CollectorPlan, which constructs and transmits materializations; and -- BackendPlan, which validates, stores, routes, merges, and reads them. - -The two are defined together in -[`physical-planning.md`](physical-planning.md). - -BackendPlan is not: - -- a copy of ASAPPlanner's internal DAG; -- a query-language AST; -- a collector configuration; -- a storage inventory discovered after planning; or -- an explain/viewer artifact. - -## 3. Plan envelope - -Every BackendPlan contains: - -| Field | Meaning | -| --- | --- | -| `plan_id` | Identity shared with every CollectorPlan compiled from the same decision. | -| `plan_version` | Ordered version within that plan identity. | -| `activation` | Earliest time this plan may serve queries. | -| `expiry` | Optional time after which this plan is invalid. | -| `backend_compat` | Backend-plan and summary-state schema compatibility identity. | -| `generated_at` | Time the control plane emitted the plan. | -| `planner_revision` | Immutable Planner revision that produced the logical selection. | - -The data plane rejects stale, conflicting, premature, expired, or -incompatible plans. Reapplying the same plan/version/content is idempotent. - -## 4. Materializations - -A materialization describes one logical summary state expected by the -backend. It contains: - -- content-addressed materialization identity; -- bound metric/source and canonical filters; -- summarized value or item label; -- physical/logical window contract; -- reduction and grouping layout; -- exact `SummaryFamilyType`, including concrete algorithm and parameters; -- collector producer references; -- state encoding/schema compatibility; -- storage destination and retention; and -- selected result guarantee when supplied by Planner. - -Exact accumulators and approximate summaries use the same materialization -concept. The family discriminator determines which state and parameters are -valid. An algorithm/parameter mismatch is a decoding or validation failure, -not an untyped configuration bag. - -The materialization identity includes every semantic property required for -safe reuse and merge. Two states with the same metric name but different -filters, reductions, grouping, windows, families, parameters, or encoding are -different materializations. - -## 5. Routing and readout - -Materializations describe what state exists. Routing entries describe what -queries that state can answer. These are separate because one materialization -may serve several readouts or queries. - -For example, one sufficiently accurate quantile materialization may support -p50, p95, and p99 readouts. It remains one maintained state with several -routing entries. - -A routing entry identifies: - -- the query capability/readout it satisfies; -- the materialization used; -- any remaining backend-side operation; -- required grouping/window compatibility; -- the storage tier; and -- exact fallback behavior on a miss. - -Routing never changes the materialization's summary semantics. A capability -match can choose among already valid planned routes; it cannot reinterpret -stored state as another family or parameterization. - -## 6. Two levels of matching - -Serving needs two different matching strengths. - -### Capability matching - -Capability matching answers: - -> Is there a planned materialization that can answer this logical query shape? - -It may allow a compatible family-level or readout-level relationship, such as -using one quantile summary for several ranks or rolling up a mergeable finer -grouping when the plan explicitly permits it. - -### State compatibility matching - -State compatibility answers: - -> Can these exact payloads be decoded, merged, and read under this -> materialization contract? - -This comparison is strict. Family, algorithm, parameters, grouping layout, -window, encoding, plan identity, and materialization identity must agree. - -Capability compatibility never implies state compatibility. The backend may -route a query to a compatible planned materialization, but it may merge only -strictly compatible states. - -## 7. Accuracy guarantees - -`AccuracyTarget` is the requested constraint used during planning. When the -pinned Planner revision provides `ResultGuarantee`, BackendPlan preserves the -selected result's: - -- error metric; -- bound expression; -- failure probability; -- provenance and budget allocation; and -- any unavailable statistic that kept the guarantee unknown. - -The data plane does not recompute or weaken that guarantee. Unknown is not -zero and not exact. A route that requires a guarantee fails when the stored -plan lacks a sufficient one. - -For an explicitly permitted approximate realization of an Exact-requested -TopK query, BackendPlan records the effective approximation target and -guarantee. It must never label that realization exact. - -## 8. Source and payload validation - -Each ingested summary payload identifies: - -- plan ID and version; -- backend compatibility identity; -- materialization ID; -- collector/producer ID; -- window identity; -- state schema version; and -- full/delta sequencing and checkpoint identity when applicable. - -The backend rejects payloads that are unknown, stale, incompatible, out of -sequence, or addressed to another active plan. It does not place them into a -best-effort metric-name bucket. - -A plan may reference several collector producers for one sharded -materialization. The backend merges them only if the materialization contract -and family algebra allow it. - -## 9. Installation and lifecycle - -BackendPlan installation is staged and atomic: - -1. Decode and validate the complete plan. -2. Validate every family, readout, route, source, and storage capability. -3. Build a new routing/index view without mutating the active view. -4. Confirm compatibility with the matching collector subplan. -5. Mark the plan staged until its activation time and collector application - evidence are available. -6. Atomically switch query routing to the new view. -7. Retain the previous view until in-flight readers and the configured drain - horizon finish. - -An invalid update never partially changes routing. The previous unexpired -plan remains available for rollback. - -## 10. Query-time behavior - -At query time, the data plane: - -1. parses/canonicalizes the query only as needed to identify its planned - logical shape; -2. looks up routes installed from BackendPlan; -3. verifies readiness, freshness, grouping, window, and guarantee; -4. fetches strictly compatible states; -5. merges and reads them according to the planned operation; and -6. returns the result or takes the explicit exact fallback. - -It does not invoke Planner candidate search, run a planning cost model, resize -a sketch, or infer a missing parameter. - -## 11. Example - -Suppose the workload contains: - -```promql -quantile_over_time(0.95, request_duration_seconds[5m]) -``` - -The selected and compiled plan may contain one DDSketch materialization with -one-minute panes and a p95 routing/readout entry. BackendPlan records the -DDSketch parameter, per-entity reduction, independent grouping layout, -producer collectors, storage route, window composition, and selected -guarantee. - -When the query arrives, the data plane finds that route, fetches five -compatible panes, merges DDSketch state, and reads p95. It does not run a cost -model to reconsider KLL or choose a new DDSketch parameter. - -## 12. Wire-format principles - -BackendPlan is a typed protobuf contract. - -The schema follows these principles: - -- family-specific values use typed discriminated variants; -- invalid family/parameter combinations are unrepresentable or rejected; -- additive optional fields support controlled rollout; -- unknown required variants fail closed; -- compatibility is explicit through `backend_compat`; -- materialization identity is stable and content-addressed; and -- debug/explain fields are not runtime identity. - -Exact protobuf field numbers and generated-language types belong in the -protocol definition and implementation review, not this design document. - -## 13. Fail-closed behavior - -The plan or query fails when: - -- the plan envelope is stale or incompatible; -- a materialization family/parameter/grouping/window is unsupported; -- a collector source does not match the expected contract; -- state payload identities or sequences are invalid; -- a requested route has no valid materialization; -- a required guarantee is absent or insufficient; -- required state is empty, stale, or incomplete; or -- exact fallback is required but unavailable. - -The backend must not return a plausible summary result from mismatched state, -silently drop missing series, treat missing data as zero, or hide a failure as -a routing miss. - -## 14. Non-goals - -This document does not define: - -- Planner candidate generation or ranking; -- CollectorPlan; -- summary-state byte encoding; -- storage-engine implementation; -- query parser implementation; -- routing-index data structures or performance optimizations; or -- protobuf field numbering. - -## 15. Definition of done - -The BackendPlan design is satisfied when: - -- control plane and data plane share one typed contract; -- every expected collector materialization has an exact backend declaration; -- one materialization can serve several explicit routes/readouts; -- state merge uses strict compatibility; -- guarantees remain Planner-derived and fail closed; -- plan installation and routing switch atomically; -- query serving performs no independent summary planning; -- stale or mismatched payloads are rejected; and -- end-to-end tests prove matching plans serve and mismatched plans fail. diff --git a/control_plane/docs/physical-planning.md b/control_plane/docs/physical-planning.md deleted file mode 100644 index d4a55b7dc..000000000 --- a/control_plane/docs/physical-planning.md +++ /dev/null @@ -1,343 +0,0 @@ -# Physical planning for collector and backend execution - -> Status: proposed -> -> MVP relation: required to turn one Planner selection into matching collector -> and backend runtime plans. -> -> Scope: the ASAPQuery-backend physical-planning step between ASAPPlanner's -> selected post-ASAP workload DAG and the two runtime executors: -> ASAPCollector and the ASAPQuery data plane. - -Developer guides: -[Planner adapter and physical compiler](developer_docs/planner-and-physical-compiler.md) -and [runtime plan publication](developer_docs/runtime-plan-publication.md). - -## TL;DR - -ASAPPlanner selects a logical plan. That plan says which summaries and exact -operations answer a workload, but it intentionally does not choose machines, -shards, runtime windows, transport modes, or storage routes. - -ASAPQuery-backend performs one physical compile that produces both runtime -views of the decision: - -```text -selected post-ASAP workload DAG - | - v - physical compilation - / \ - v v -CollectorSubplan BackendSubplan -CollectorPlan(s) BackendPlan -``` - -The two subplans share the same plan and materialization identities. They are -never derived independently and are never considered active unless both sides -confirm a compatible decision. - -## 1. Why this layer exists - -ASAPPlanner owns logical choices such as: - -- exact accumulator versus approximate summary; -- summary family, algorithm, and parameters; -- reduction and grouping strategy; -- logical sharing and composition; -- summary readout; and -- exact fallback. - -The runtime still needs deployment decisions that do not belong in Planner: - -- which collector or backend stage runs each operation; -- how logical state is sharded or shared; -- which streaming panes materialize a query time range; -- whether state is sent as raw observations, full summaries, or deltas; -- where state is stored and queried; -- when a new plan becomes active; and -- how incompatible or failed updates are rolled back. - -Skipping this layer causes two common failures: - -1. treating a logical `SummaryAgg` as though it already names a collector - process and runtime configuration; and -2. deriving collector and backend plans separately, allowing family, - parameters, grouping, windows, or identities to drift. - -The physical compiler closes both gaps in one operation. - -## 2. Input contract - -The compiler receives: - -- the selected post-ASAP DAG for the whole workload; -- stable workload/query correlation information; -- collector and backend capability snapshots; -- deployment topology and stage boundaries; -- workload statistics and resource constraints; -- runtime window, freshness, and retention policy; and -- transmission and storage policy. - -Shared logical nodes remain shared at this boundary. The compiler must not -first flatten the workload into independent per-query or per-metric rows. - -The selected DAG may contain summary producers, estimates, merges, -subtractions, deletes, joins, exact operations, shared sub-DAGs, and -`KeepPreAsap` fallback. A pinned Planner revision may also carry explicit -execution phases and result guarantees. The compiler consumes those canonical -values rather than defining local equivalents. - -## 3. Output contract - -One compile returns one `CompiledPlan` with: - -- a collector subplan containing one `CollectorPlan` for every targeted - collector; -- a backend subplan containing one `BackendPlan` for the ASAPQuery data - plane; -- a shared plan envelope; -- content-addressed materialization identities; and -- a validation record showing that both subplans were produced from the same - selected DAG and capability snapshots. - -### Shared plan envelope - -| Field | Meaning | -| --- | --- | -| `plan_id` | Content-addressed identity of the logical selection, topology identity, and semantic constraints. | -| `plan_version` | Ordered update within the same plan identity, such as changed sizing or lifecycle policy. | -| `activation` | Earliest time at which both subplans may become authoritative. | -| `expiry` | Optional time after which the plan may no longer produce or serve state. | -| `backend_compat` | Compatibility identity for BackendPlan and emitted summary-state schemas. | -| `planner_revision` | Immutable ASAPPlanner revision used for the selection. | - -Mutable lifecycle or sizing fields do not silently change the identity of an -existing version. Reusing the same `(plan_id, plan_version)` for different -content is invalid. - -### Materialization identity - -A materialization is the physical state built for one selected logical -summary producer. Its identity includes every property required for safe -reuse and merge, including: - -- bound source and filters; -- summarized value/item; -- summary family, algorithm, and parameters; -- reduction and grouping layout; -- logical/physical window contract; and -- state schema compatibility. - -Placement, transport cadence, or storage location may change without -pretending a semantically different summary is the same materialization. - -Explain or viewer node IDs are traceability metadata, not materialization -identity. - -## 4. Partitioning the selected DAG - -The baseline partition follows data availability: - -- operations that consume new observations and construct maintained state - belong on the update side; -- operations that read, merge, or transform maintained state into query - answers belong on the readout side; and -- `KeepPreAsap` remains exact backend/archive execution. - -In the current Planner vocabulary, `SummaryAgg` is the principal update-side -boundary and `SummaryEstimate` is a readout. Summary merge and other state -composition are placed according to topology, capabilities, and transport -cost without changing their logical semantics. - -If the pinned Planner revision provides explicit execution availability or -phase assignments, those validated phases are authoritative. The compiler -must not infer a conflicting phase from node names. - -An operation may be assigned only to an executor that advertises its full -semantics. A nameable Planner alternative is not automatically deployable. - -## 5. Collector subplan - -The collector subplan follows ASAPCollector's -[collection-plan interface](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/developer_docs/opamp-config-push.md). - -For each assigned collector, it specifies: - -- target collector and capability snapshot; -- the shared plan envelope; -- source metrics and matchers; -- materialization identities; -- summary family, algorithm, parameters, and accuracy requirement; -- reduction and grouping semantics; -- concrete streaming windows and lateness; -- local sharding; -- raw, full, or delta transmission; and -- backend endpoint/schema compatibility. - -OpAMP is the delivery transport. The payload is the versioned -`asap-collector-plan.yaml` document, not ASAPPlanner's Rust IR and not the -collector's complete bootstrap configuration. - -## 6. Backend subplan - -The backend subplan follows -[`backend-plan.md`](backend-plan.md). - -It specifies: - -- the same plan envelope; -- every expected materialization; -- the collector assignments producing its state; -- exact summary family, algorithm, parameters, grouping, and windows; -- ingestion/storage destination; -- query capabilities and readouts satisfied by the materialization; -- remaining backend-side operators; and -- selected result guarantees when provided by Planner. - -The backend does not reconstruct the chosen summary from query text or stored -state. It installs and executes the control plane's exact decision. - -## 7. Sharing, sharding, and merge - -One shared logical producer remains one logical materialization even when it -serves several queries. The backend may associate several routing/readout -entries with that one materialization. - -A materialization may have several physical producers when sharded across -collectors. The compiler records all producers and inserts a compatible merge -at the selected boundary. A merge is legal only when every input agrees on -the materialization contract and the selected family supports merge. - -Sharding does not create several unrelated logical summaries, and sharing -does not allow consumers with incompatible filters, reductions, grouping, -windows, parameters, or guarantees to reuse state. - -## 8. Window and transmission decisions - -Planner time ranges express query semantics. The physical compiler chooses -streaming panes capable of answering those ranges. - -For the MVP: - -- panes are anchored and tumbling; -- pane composition must exactly cover each claimed query range; -- allowed lateness and watermark behavior are explicit; -- incompatible alignment is rejected; and -- freshness policy is shared with the backend readiness check. - -Transmission is independent of logical summary choice: - -- `raw` forwards selected observations for exact/backend execution; -- `full` sends complete summary state; and -- `delta` sends ordered state changes plus periodic full checkpoints. - -Delta is legal only when collector and backend advertise the same state, -sequence, and checkpoint semantics. Every payload identifies its plan, -materialization, producer, window, sequence, and base/checkpoint. - -### Aggregation placement - -Planner's reduction and grouping are logical requirements. The physical -compiler decides where that reduction runs without changing them. For example, -`sum by (region) (rate(http_requests_total[5m]))` may maintain one summary per -`region` at collectors, while a query with no grouping may use one -whole-workload materialization. A per-series or per-group result must never be -silently collapsed into a global result. - -### State representation - -Dense versus sparse state is a physical representation choice, not a new -summary choice. For example, an HLL materialization may use sparse state for -low-cardinality groups and promote to dense state as cardinality grows, but -both representations must preserve the same HLL parameters, merge semantics, -wire compatibility, and accuracy contract. - -The compiler may select a representation only when both producer and consumer -advertise compatible support. Otherwise it uses the declared fallback or -rejects the plan. Representation details such as collector configuration field -names belong in the collector interface, not in this design. - -## 9. Compile and activation sequence - -1. Validate the selected DAG against both capability snapshots. -2. Allocate update/readout operators and physical producers. -3. Choose compatible windows, transmission, and storage routes. -4. Construct materialization identities. -5. Emit both subplans from the same in-memory decision. -6. Validate cross-subplan equality for all shared contracts. -7. Stage both subplans before `activation`. -8. Require backend installation and collector semantic application reports. -9. Route queries to the new plan only after both sides report compatible - active identities. -10. Retire old state after its readers and lateness horizon drain. - -If any step fails, the previous unexpired plan remains authoritative. A -partial push, OpAMP delivery acknowledgement, file write, or process restart -does not constitute plan activation. - -## 10. Fail-closed rules - -The compiler or runtime rejects the plan when: - -- an assigned executor lacks a required family, algorithm, grouping, phase, - readout, window, or transmission capability; -- collector and backend materialization contracts differ; -- a required accuracy guarantee is unknown or insufficient; -- a merge combines incompatible state; -- delta sequencing/checkpoint semantics do not match; -- plan versions conflict or lifecycle conditions disallow activation; or -- semantic application evidence is missing. - -It must never substitute another family, parameter, grouping layout, -accuracy target, or transmission semantics to make an invalid plan appear -deployable. - -## 11. Example - -For: - -```promql -quantile_over_time(0.95, request_duration_seconds{region="us-east"}[5m]) -``` - -Planner may select a per-entity DDSketch summary and a p95 readout. The -physical compiler may then: - -- assign DDSketch construction to selected collectors; -- choose one-minute panes that compose into the five-minute query range; -- transmit deltas every ten seconds with periodic full checkpoints; -- declare one shared DDSketch materialization in BackendPlan; and -- route the p95 query readout to that materialization. - -The compiler does not change DDSketch to KLL, change the accuracy parameter, -or aggregate series together merely because another physical layout would be -cheaper. Such a change requires selection of a different valid Planner -candidate. - -## 12. Non-goals - -This document does not define: - -- query parsing or summary selection; -- internal Planner serialization; -- exact protobuf field numbers; -- summary-state byte encoding; -- collector bootstrap configuration; -- storage-engine implementation; or -- query-engine implementation details. - -## 13. Definition of done - -The compiled-plan design is satisfied when: - -- one compile produces both subplans; -- all shared identities and semantic fields match; -- shared Planner nodes remain shared materializations; -- sharded producers merge only under a valid contract; -- unsupported shapes fail before activation; -- both plans stage and activate atomically from the user's perspective; -- emitted state carries the active identities; -- the data plane serves without replanning; and -- a deliberately mismatched or partially applied plan is rejected in an - end-to-end test. diff --git a/data_plane/docs/README.md b/data_plane/docs/README.md deleted file mode 100644 index 2a5bc459a..000000000 --- a/data_plane/docs/README.md +++ /dev/null @@ -1,48 +0,0 @@ -# ASAPQuery data-plane documentation - -The data plane ingests state produced under an active BackendPlan, stores that -state, answers supported PromQL queries, and uses an explicit exact fallback -for unsupported queries. It executes plans; it does not choose summary families -or re-plan queries. - -## Design documents - -- [Plan-aware query execution](design_docs/query-execution.md) — ingestion, - routing, readiness, summary readout, and exact fallback contracts. -- [Repository-wide summary storage](../../docs/design_docs/summary-storage.md) — - materialization state, lifecycle, and query consistency. -- [BackendPlan](../../control_plane/docs/backend-plan.md) — the control-plane - contract installed by the data plane. - -## Developer documentation - -- [Extension boundaries](developer_docs/extension-points.md) — responsibilities - of protocol servers, adapters, and fallback clients. -- [BackendPlan runtime](developer_docs/backend-plan-runtime.md) — validation, - staging, atomic installation, and snapshots. -- [OTLP summary ingestion](developer_docs/otlp-summary-ingestion.md) — decoding, - plan validation, SID resolution, and full/delta handling. -- [Query routing and readout](developer_docs/query-routing-and-readout.md) — - readiness, summary execution, and exact fallback. -- [Summary storage and series identity](developer_docs/summary-storage-and-series-identity.md) - — store boundaries, identity hierarchy, lifecycle, and concurrency. -- [Adding a summary family](../../docs/developer_docs/adding-summary-family.md) — - cross-repository prerequisites and backend validation. - -## User guide - -- [Querying ASAP](user_guide/querying-asap.md) — PromQL behavior, planned - summary execution, exact fallback, freshness, and errors. - -## Ownership - -- [ASAPPlanner](https://github.com/ProjectASAP/ASAPPlanner) owns logical query - planning, query-to-summary mapping, and accuracy reasoning. -- The ASAPQuery control plane owns physical compilation and BackendPlan. -- The data plane owns ingestion, storage, readout, query execution, and exact - fallback under the installed plan. -- [ASAPCollector](https://github.com/ProjectASAP/ASAPCollector) owns summary - construction and transmission at the edge. - -Historical ingestion paths, file-by-file migrations, and configuration -walkthroughs are not data-plane design contracts. diff --git a/data_plane/docs/design_docs/query-execution.md b/data_plane/docs/design_docs/query-execution.md deleted file mode 100644 index 484688c8b..000000000 --- a/data_plane/docs/design_docs/query-execution.md +++ /dev/null @@ -1,127 +0,0 @@ -# Plan-aware query execution - -> Status: proposed -> -> MVP relation: required for every summary-backed query and exact fallback. - -Developer guides: -[BackendPlan runtime](../developer_docs/backend-plan-runtime.md), -[OTLP summary ingestion](../developer_docs/otlp-summary-ingestion.md), and -[query routing/readout](../developer_docs/query-routing-and-readout.md). - -## TL;DR - -The data plane accepts only state compatible with its active BackendPlan. At -query time it matches the PromQL request to a planned readout, checks coverage -and freshness, reads the matching materialization, and returns a -Prometheus-compatible result. A query that cannot be served safely is rejected -or sent to the exact fallback selected by the plan. - -## Data flow - -```text -ASAPCollector -- OTLP summary state --> ingest validation --> SummaryStore - | -PromQL request --> protocol adapter --> planned routing + readiness - | - summary readout <----------+ - | - Prometheus-compatible response - -Unsupported/planned-exact query --> exact fallback backend -``` - -The data plane never infers a summary family from a metric name or query text. -It uses the materialization and readout declared by BackendPlan. - -## Ingestion contract - -Before accepting a payload, the data plane validates: - -- plan and plan-version identity; -- materialization, producer, and tenant identity; -- summary family, parameters, grouping, window, and representation; -- full/delta sequence and checkpoint requirements; and -- lifecycle and schema compatibility. - -Unknown, expired, reordered, or incompatible state is rejected and surfaced. -Receiving bytes is not evidence that the corresponding window is queryable. - -## Query routing - -Routing has three outcomes: - -1. **Summary readout** when BackendPlan contains a compatible route and the - required windows are fresh and complete. -2. **Exact fallback** when BackendPlan explicitly routes the query shape to an - exact backend. -3. **Explicit failure** when neither route is valid. - -A store miss, stale window, or incompatible payload must not be converted into -an empty or plausible approximate result. - -## Supported aggregation shapes - -The MVP exercises these shapes without defining Planner's query-to-summary -rules here: - -- within one series over time: - - ```promql - quantile_over_time(0.95, request_duration_seconds[5m]) - ``` - -- across label groups at an evaluation timestamp: - - ```promql - sum by (region) (http_requests_total) - ``` - -- across both a time range and label groups: - - ```promql - sum by (region) (rate(http_requests_total[5m])) - ``` - -Whether a particular expression is exact, summary-backed, or unsupported is -the selected plan's decision. The data plane only executes that decision. - -## Readiness and freshness - -A readout is ready only when all materializations required by its route: - -- belong to the active plan version; -- cover the requested logical interval; -- satisfy watermark and allowed-lateness policy; -- have no unresolved delta gap; and -- meet any declared source-completeness requirement. - -For example, a query at `12:05` over `[5m]` cannot reuse complete panes from an -earlier run merely because their labels match. The plan identity and logical -window must also match. - -## Result semantics - -The response preserves Prometheus labels, timestamps, result type, and error -behavior. Summary error guarantees come from the selected Planner result and -are carried by BackendPlan; the data plane neither tightens nor loosens them. - -When several physical shards contribute to one result, they may be merged only -if their materialization contracts match and the chosen summary supports the -declared merge. - -## Exact fallback - -Fallback is a correctness path, not a silent catch-all. BackendPlan identifies -the backend and query scope eligible for fallback. Transport failures and exact -query errors remain visible to the caller. - -Examples that may require exact fallback include an unsupported PromQL -operator, a request outside retained summary coverage, or a query whose exact -accuracy requirement has no compatible maintained state. - -## Non-goals - -This document does not define PromQL parsing, summary selection, Planner IR, -summary algorithms, state byte encoding, storage-engine implementation, or -protocol-specific server code. diff --git a/docs/01-getting-started/architecture.md b/docs/01-getting-started/architecture.md index 31f4ce2a2..3f6710c16 100644 --- a/docs/01-getting-started/architecture.md +++ b/docs/01-getting-started/architecture.md @@ -38,7 +38,7 @@ owns logical parsing, summary alternatives, accuracy reasoning, and selection. The ASAPQuery control plane owns deployment capabilities, physical placement, windows, transmission, matching runtime plans, and activation. -See the [control-plane design](../../control_plane/docs/README.md). +See the [control-plane physical-planning design](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/design_docs/control-plane/physical-planning.md). ## Ingestion path @@ -60,19 +60,32 @@ coverage, freshness, and state compatibility before execution. If the selected logical plan requires exact execution, the request is sent to the configured exact backend. -See [plan-aware query execution](../../data_plane/docs/design_docs/query-execution.md). +See [plan-aware query execution](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/design_docs/asapquery-backend/query-query-engine.md). ## State and identity -Three identities remain distinct: +The three levels are related but must not be collapsed: - a **plan identity** versions one deployed physical decision; -- a **materialization identity** describes maintained summary semantics; and -- a **series identity (`sid`)** identifies one canonical metric series. +- a **materialization identity** is the stable descriptor/fingerprint for the + maintained summary semantics: bound source/filter, summarized value, + reduction and retained grouping keys, aggregation family/algorithm and + parameters, accuracy contract, window semantics, and compatible state schema; + and +- a **summary series identity (`sid`)** identifies one canonical materialized + metric series under that descriptor, including the canonical metric and the + concrete retained label values. A raw materialization may likewise allocate + a distinct SID for its raw sample series. + +For example, “DDSketch over `request_duration_seconds`, grouped by `service`, +alpha 0.01, one-minute panes” is one materialization definition. Its +`service=checkout` and `service=payments` outputs are two materialized series +and therefore have different SIDs. Changing alpha or retained grouping keys +creates a different materialization definition and cannot reuse either SID. Conflating them can cause incompatible state reuse. The storage and identity -contracts are described in [summary storage](../design_docs/summary-storage.md) -and [series identity](../design_docs/series-identity.md). +contracts are described in [summary storage](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/design_docs/asapquery-backend/query-summary-store-engine.md) +and [series identity](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/design_docs/cross-cutting/summary-series-id.md). ## Component failure behavior diff --git a/docs/README.md b/docs/README.md index 3f3b30597..5c635929a 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,43 +1,32 @@ # ASAPQuery-backend documentation +## Canonical system design + +The end-to-end architecture and shared contracts are maintained centrally in +[ASAPCollector design docs](https://github.com/ProjectASAP/ASAPCollector/tree/main/docs/design_docs). +That documentation owns collection, transmission, backend ingest/storage/query, +SID, physical plans, workload inputs, and runtime feedback. + +This repository keeps only code-owned implementation guidance and user-facing +backend operation material. + ## Getting started - [Overview](01-getting-started/overview.md) -- [System architecture](01-getting-started/architecture.md) - [Local setup](01-getting-started/local-setup.md) +- [Architecture orientation](01-getting-started/architecture.md) + +## Backend implementation + +- [Developer guide](developer_docs/README.md) +- [Summary store engine](developer_docs/summary-store-engine/implementation.md) +- [Query engine](developer_docs/query-engine/query-engine.md) +- [Ingest engine](developer_docs/ingest-engine/ingest-engine.md) +- [Summary series ID resolver](developer_docs/summary-series-id/resolver.md) +- [Control plane](developer_docs/control-plane/physical-compiler.md) + +## User guide -## Component design - -- [Control plane](../control_plane/docs/README.md) -- [Data plane](../data_plane/docs/README.md) -- [Summary storage](design_docs/summary-storage.md) -- [Series identity](design_docs/series-identity.md) -- [Future storage and compression](design_docs/future-storage-and-compression.md) - -## Developer guides - -- [Adding a summary family](developer_docs/adding-summary-family.md) -- [Data-plane extension boundaries](../data_plane/docs/developer_docs/extension-points.md) -- [Planner adapter and physical compiler](../control_plane/docs/developer_docs/planner-and-physical-compiler.md) -- [Runtime plan publication](../control_plane/docs/developer_docs/runtime-plan-publication.md) -- [BackendPlan runtime](../data_plane/docs/developer_docs/backend-plan-runtime.md) -- [OTLP summary ingestion](../data_plane/docs/developer_docs/otlp-summary-ingestion.md) -- [Query routing and readout](../data_plane/docs/developer_docs/query-routing-and-readout.md) -- [Summary storage and series identity](../data_plane/docs/developer_docs/summary-storage-and-series-identity.md) -- [Querying ASAP](../data_plane/docs/user_guide/querying-asap.md) - -## Documentation ownership - -This repository documents ASAPQuery-backend-specific physical planning, -runtime query execution, and storage contracts. - -- Logical query planning and query-to-summary mapping belong to - [ASAPPlanner](https://github.com/ProjectASAP/ASAPPlanner). -- Collector processing and CollectorPlan application belong to - [ASAPCollector](https://github.com/ProjectASAP/ASAPCollector). -- Summary algorithm implementation and mathematical guarantees belong to the - corresponding summary library. - -Design documents link to those owners instead of maintaining parallel copies. -Benchmark outputs, migration histories, file-by-file implementation plans, and -future roadmaps are not active design specifications. +- [User guide index](user_guide/README.md) +- [Run and verify the backend](user_guide/running-and-verifying.md) +- [Querying ASAP](user_guide/querying-asap.md) diff --git a/docs/design_docs/README.md b/docs/design_docs/README.md index ed8e8bf02..a48d08dac 100644 --- a/docs/design_docs/README.md +++ b/docs/design_docs/README.md @@ -1,20 +1,9 @@ -# ASAPQuery-backend design documents +# System design location -> Status: active +ASAPQuery-backend does not maintain a second copy of the system design. +The canonical component design is in +[ASAPCollector/docs/design_docs](https://github.com/ProjectASAP/ASAPCollector/tree/main/docs/design_docs). -## TL;DR - -These documents cover backend-owned storage and identity decisions. Planning -design lives in [control-plane docs](../../control_plane/docs/README.md), and -query serving lives in [data-plane docs](../../data_plane/docs/README.md). - -| Document | Scope | Status | -| --- | --- | --- | -| [Summary storage](summary-storage.md) | Materialization state, windows, ingest/query consistency, and lifecycle. | Active MVP design | -| [Series identity](series-identity.md) | Canonical metric-series identity and recovery behavior. | Active MVP design | -| [Future storage and compression](future-storage-and-compression.md) | Persistence tiers, compaction, pluggable service mode, backfill, compression, and profiling. | Dormant/future | - -Logical planning and summary accuracy algebra are owned by -[ASAPPlanner](https://github.com/ProjectASAP/ASAPPlanner) and the relevant -summary libraries. Benchmark measurements belong in reproducible run artifacts, -not in these design documents. +Backend-specific implementation design notes are organized by component under +[`../developer_docs`](../developer_docs/README.md). They explain current Rust +internals and are subordinate to the shared system contracts. diff --git a/docs/design_docs/future-storage-and-compression.md b/docs/design_docs/future-storage-and-compression.md deleted file mode 100644 index 7ca5a4bce..000000000 --- a/docs/design_docs/future-storage-and-compression.md +++ /dev/null @@ -1,96 +0,0 @@ -# Future storage and compression - -> Status: dormant -> -> MVP relation: not required for the OTel summary-pipeline MVP. - -## TL;DR - -This document records future backend storage and representation scopes without -turning unimplemented proposals, analytical estimates, or benchmark snapshots -into current architecture. Each scope requires its own acceptance criteria and -implementation proposal before becoming active. - -## Durable summary storage - -A future tier may flush immutable summary parts to disk or object storage and -recover them after restart. It must preserve the active storage contract: -materialization identity, logical windows, completeness, delta/checkpoint -lineage, and plan lifecycle. - -Example query: - -```promql -quantile_over_time(0.99, request_duration_seconds[24h]) -``` - -Serving older persisted panes is valid only when they are compatible and fully -cover the requested interval. - -## Semantic compaction - -Compaction may merge adjacent panes to reduce objects or read work. It is legal -only for a mergeable summary and must not cross incompatible materialization, -accuracy, grouping, or representation boundaries. - -Example: sixty compatible one-minute quantile-summary panes may compact into -one one-hour pane for: - -```promql -quantile_over_time(0.95, request_duration_seconds[1h]) -``` - -The actual PromQL mapping remains Planner-owned; this example describes only -the storage operation after a valid plan exists. - -## Backfill and refresh - -Backfill may build missing summary windows from an exact retained source. It -must be deterministic for the same source snapshot and materialization -contract, isolated from live ingestion, and atomically publish coverage. - -## Pluggable service mode - -Summary storage may eventually run as an in-process library or a separate -service. Both deployment shapes must expose equivalent semantic validation, -readiness, and error behavior. Network transport must not become a second -planning or identity authority. - -## Compression and representation - -Future work may include sparse summary state, delta checkpoints, compressed -raw fallback blocks, shared timestamp columns, or family-specific encodings. -Every representation must declare compatibility, recovery, and accuracy -effects. Lossy compression cannot be presented as exact. - -Example workload: - -```promql -sum by (service) (rate(http_requests_total[1h])) -``` - -Compression is evaluated on the state selected for this workload; it does not -change the logical query or choose a different summary. - -## Sampling and learned summaries - -Sampling, wavelets, anomaly models, or learned summaries require Planner-owned -logical semantics and guarantees before backend support. The backend may store -and execute an accepted family but must not define its query mapping locally. - -## Profiling and cost inputs - -A profiler may measure update cost, merge cost, readout latency, memory, and -encoded size for Planner's cost model. Measurements must identify the summary -implementation, parameters, workload, and hardware. Checked-in estimates or -one-off benchmark results are not substitutes for reproducible artifacts. - -## Activation rule - -A future scope becomes active only when it has: - -- an owning component and stable semantic interface; -- predeclared correctness and performance criteria; -- failure and recovery behavior; -- compatibility with BackendPlan and CollectorPlan; and -- reproducible end-to-end validation. diff --git a/docs/design_docs/series-identity.md b/docs/design_docs/series-identity.md deleted file mode 100644 index b786b063b..000000000 --- a/docs/design_docs/series-identity.md +++ /dev/null @@ -1,89 +0,0 @@ -# Series identity - -> Status: active -> -> MVP relation: provides stable identity for ingestion, grouping, and result -> labels across collector and backend boundaries. - -Developer guide: -[Summary storage and series identity](../../data_plane/docs/developer_docs/summary-storage-and-series-identity.md). - -## TL;DR - -A series ID (`sid`) names one canonical metric series within a tenant and -identity namespace. The backend registry assigns or validates this mapping; -collectors may cache it, but payload labels remain the recovery evidence needed -to detect stale or unknown IDs. - -Formally, SID is the pair `(namespace, numeric_value)`, not a globally meaningful -integer. The namespace contains the tenant/isolation domain and a registry -version. Its public data structures and registry interfaces are defined in the -[developer guide](../../data_plane/docs/developer_docs/summary-storage-and-series-identity.md#sid-definition). - -## Identity contract - -The canonical series key consists of: - -- tenant or isolation domain; -- metric name; and -- a deterministically ordered set of identifying labels. - -Two observations with the same canonical key resolve to the same `sid` within -one namespace. Different canonical keys must not share a `sid`. Aggregation -group labels and summary parameters are not silently folded into series -identity; they belong to the materialization/group contract. - -For example, these are different series: - -```text -http_requests_total{job="api",region="us-east"} -http_requests_total{job="api",region="eu-west"} -``` - -but reordering the two labels does not create a third identity. - -## Relationship to plan identity - -`sid`, materialization identity, and plan identity serve different purposes. A -series can participate in several materializations and plan versions. Reusing a -`sid` does not authorize reuse of summary state with different family, -grouping, parameters, or windows. - -## Resolution and caching - -The backend registry is authoritative for the namespace. A collector may cache -resolved IDs to reduce coordination, provided it also carries enough canonical -identity evidence for the backend to validate or recover the mapping. - -Resolution is idempotent: retrying the same canonical key returns the same -mapping. Registering an ID without receiving state is allowed and must not make -a query appear complete. - -## Recovery - -When the backend does not recognize a sender-provided `sid`, or finds that it -maps to different labels, it rejects the numeric shortcut and resolves from the -canonical key. A stale cache cannot overwrite an existing authoritative -mapping. - -Backend restart behavior depends on registry durability: - -- with durable identity state, mappings are restored before dependent payloads - become queryable; -- without durable state, collectors re-resolve from canonical labels under a - new namespace/version. - -In both cases ambiguity fails closed. - -## Distributed backend - -Distributed allocation, shard ownership, rebalancing, and high availability are -future deployment concerns. Any scheme must preserve deterministic lookup, -namespace/version evidence, and conflict detection. Numeric partitioning alone -must not weaken the canonical-key contract. - -## Non-goals - -This document does not prescribe RPC messages, integer width, database tables, -cache files, sharding algorithms, or a migration sequence from older `agg_id` -names. diff --git a/docs/design_docs/summary-storage.md b/docs/design_docs/summary-storage.md deleted file mode 100644 index 924a320df..000000000 --- a/docs/design_docs/summary-storage.md +++ /dev/null @@ -1,99 +0,0 @@ -# Summary storage - -> Status: active -> -> MVP relation: stores the state needed for summary-backed query execution. - -Developer guide: -[Summary storage and series identity](../../data_plane/docs/developer_docs/summary-storage-and-series-identity.md). - -## TL;DR - -Summary storage is a plan-aware materialized-state store. It accepts only state -compatible with an active BackendPlan, indexes it by semantic materialization, -series/group, and logical window, and exposes complete state to planned -readouts. It is not a general raw time-series database and does not choose -summaries. - -## Stored object - -Each stored state belongs to: - -- one tenant and source; -- one active plan version; -- one materialization identity; -- one canonical series or aggregation group; -- one logical window; -- one summary family, parameter set, and representation version; and -- one producer/checkpoint lineage when full or delta state is used. - -The materialization identity includes the query-relevant semantics required for -safe reuse: source matchers, summarized value, grouping, family, parameters, -accuracy contract, and window definition. Storage location and delivery cadence -do not create a different logical materialization. - -## Ingestion - -Ingestion validates payload metadata against BackendPlan before changing -queryable state. Full state replaces a declared checkpoint. Delta state applies -only to its expected base and sequence. Duplicate delivery is idempotent; -missing or conflicting sequences create a visible gap rather than guessed -state. - -A window becomes queryable only after its completeness and freshness conditions -are satisfied. Payload receipt alone is insufficient. - -## Query lookup - -The data plane resolves a BackendPlan route to one or more materializations and -logical windows. Lookup returns either: - -- the complete compatible state required by the readout; or -- an explicit reason it is unavailable, such as missing coverage, stale state, - plan mismatch, delta gap, or unsupported merge. - -For example: - -```promql -quantile_over_time(0.95, request_duration_seconds[5m]) -``` - -may read five compatible one-minute DDSketch panes. The store may compose them -only when they exactly cover the requested interval and share the same -materialization contract. - -## Reconfiguration and lifecycle - -New and old plan versions may coexist during warm-up and drain, but state is -never mixed across incompatible materializations. Activation makes one version -authoritative for its declared interval. Retirement waits until readers, -lateness, and rollback policy no longer require the old version. - -State lifecycle includes staged, active, draining, expired, and rejected -conditions. “Present in storage” is not equivalent to “eligible for query.” - -## Memory and persistence - -The MVP may use bounded in-memory state, but it must report memory use and fail -visibly when limits prevent correct service. Evicting required state without -changing routing/readiness would violate correctness. - -Disk tiers, background flush, compaction, backfill, and standalone storage -service deployment are future extensions described in -[future storage and compression](future-storage-and-compression.md). - -## Guarantees - -- Incompatible summary states never merge. -- Missing series, groups, or windows are not treated as zero. -- Query responses never combine stale-run and current-run state. -- Accuracy metadata follows the selected logical result; storage does not - invent a new bound. -- Every accepted payload and served readout is traceable to plan and - materialization identity. - -## Non-goals - -This document does not define summary algorithms, Planner candidate selection, -Rust storage types, database schemas, file layouts, cache implementation, or -benchmark results. diff --git a/docs/developer_docs/README.md b/docs/developer_docs/README.md new file mode 100644 index 000000000..bf6fe2607 --- /dev/null +++ b/docs/developer_docs/README.md @@ -0,0 +1,34 @@ +# ASAPQuery-backend developer documents + +These pages are grouped by code component. Canonical system behavior and +cross-repository interfaces live in +[ASAPCollector/docs](https://github.com/ProjectASAP/ASAPCollector/tree/main/docs). + +## Summary store engine + +- [Storage implementation and SID internals](summary-store-engine/implementation.md) + +## Query engine + +- [Routing, readout, and query-engine implementation](query-engine/query-engine.md) +- [BackendPlan runtime](query-engine/backend-plan-runtime.md) +- [Extension points](query-engine/extension-points.md) + +## Ingest engine + +- [OTLP ingest-engine implementation](ingest-engine/ingest-engine.md) + +## Summary series ID + +- [Resolver guide](summary-series-id/resolver.md) + +## Control plane + +- [Physical compiler](control-plane/physical-compiler.md) +- [Plan publication](control-plane/plan-publication.md) +- [Workload inputs](control-plane/workload-inputs.md) +- [Runtime accuracy feedback](control-plane/runtime-accuracy-feedback.md) + +## Cross-component + +- [Adding a summary family](cross-component-adding-summary-family.md) diff --git a/control_plane/docs/developer_docs/planner-and-physical-compiler.md b/docs/developer_docs/control-plane/physical-compiler.md similarity index 55% rename from control_plane/docs/developer_docs/planner-and-physical-compiler.md rename to docs/developer_docs/control-plane/physical-compiler.md index cad96f756..cb698f2e8 100644 --- a/control_plane/docs/developer_docs/planner-and-physical-compiler.md +++ b/docs/developer_docs/control-plane/physical-compiler.md @@ -29,7 +29,7 @@ PhysicalCompiler -------> CompiledPlanBundle Logical query parsing, summary alternatives, guarantees, and candidate search remain public ASAPPlanner interfaces. Runtime publication is documented in -[Runtime plan publication](runtime-plan-publication.md). +[Runtime plan publication](plan-publication.md). ## 2. Public interfaces and definitions @@ -143,6 +143,102 @@ Output definitions: | `collector_plans` | One plan per targeted collector, following ASAPCollector's public CollectorPlan schema. | | `backend_plan` | Matching data-plane materialization and routing contract. | +### What the compiler puts in CollectorPlan + +Each targeted Collector receives a complete executable projection of the +physical decision. It is not a Planner DAG and not a pointer requiring the +Collector to recover missing semantics. + +| CollectorPlan section | Required content | Consuming ASAPCollector component | +| --- | --- | --- | +| envelope | Schema version, kind, shared plan ID/version, generation/activation/expiry, backend compatibility, Planner revision, candidate/query traceability | OpAMP receiver and plan validator | +| target | Collector instance UID, edge assignment, capability snapshot hash | Control-plane agent and capability validator | +| materialization reference | Content fingerprint describing maintained semantics plus logical-node traceability | Plan applier and materialization registry | +| input | Source metric, canonical matchers, and whether the summarized input is sample value or a named label | OTLP/OTAP metric adapter and observation router | +| summary | Raw, exact-aggregation, or sketch kind; concrete algorithm; typed parameters; requested accuracy | Precompute factory and accumulator/sketch implementation | +| reduction/grouping | Per-entity or reduction semantics, retained/excluded labels, and independent or shared grouping layout | Series-key builder and aggregation state manager | +| window | Kind, size, slide/anchor, allowed lateness, flush behavior | Window manager | +| placement | Collector stage and local shard assignment | Processor topology/runtime | +| transmission | Raw/full/delta mode, encoding and schema version, emission cadence, checkpoint interval and sequence scope | Envelope encoder and delta sender | +| output/lifecycle | Backend endpoint reference, unsupported behavior, drain/retirement policy | Exporter, plan lifecycle manager, and application reporter | + +For a Collector-side DDSketch materialization, the relevant projection is +conceptually: + +```yaml +materializations: + - id: mat:sha256:98f1... + input: + metric: request_duration_seconds + matchers: [{label: region, op: eq, value: us-east}] + value: sample_value + summary: + family: sketch + algorithm: ddsketch + parameters: {alpha: 0.01} + accuracy: {kind: epsilon, epsilon: 0.01} + reduction: {kind: reduce, by: [service], without: false} + grouping: {kind: per_subpopulation_instance} + window: {kind: tumbling, size: 1m, slide: 1m, allowed_lateness: 10s} + placement: {stage: collector, shards: 1} + transmission: + mode: delta + encoding: ddsketch-v1 + schema_version: 1 + emit_every: 10s + full_checkpoint_every: 1m + sequence_scope: materialization_window_producer + output: {endpoint_ref: asapquery-primary} +``` + +The materialization `id` above identifies the maintained summary definition. +It is not the SID. During collection and ingest, each concrete canonical label +set under that definition resolves to its own SID. + +### What the compiler puts in BackendPlan + +The matching BackendPlan is the consumer and query-serving projection of the +same decision: + +| BackendPlan section | Required content | Consuming ASAPQuery-backend component | +| --- | --- | --- | +| envelope | The same plan ID/version, activation/expiry, compatibility identity and Planner revision | Backend plan manager | +| materialization descriptor | Same materialization fingerprint, canonical metric, retained grouping keys, capability, aggregation kind/parameters, accuracy, policy fingerprint and lifecycle | SID resolver and instance-metadata registry | +| producers/ingest | Expected Collector/producer IDs, input payload kind, state schema, full/delta lineage and optional backend-precompute placement | OTLP ingest engine and precompute engine | +| window/coverage | Pane/window compatibility, lateness, expected coverage and merge rules | Ingest validation and summary store | +| storage | Warm/archive/remote route, retention, retirement and expiry | Summary store engine and archive adapter | +| query routes | Query IDs or canonical capability match, materialization reference, readout/operator, grouping/window composition and remaining backend operators | Query classifier, router and readout engine | +| guarantee/fallback | Selected guarantee and explicit exact/archive fallback behavior | Query engine and response metadata | + +The backend declaration does not contain a pre-enumerated SID for every label +value. It installs immutable metadata for the materialization; the ingest/SID +resolver then binds each canonical materialized series to an SID and the store +indexes its windows under that SID. + +### One decision, two component graphs + +```text +CollectorPlan + -> plan validator + -> source matcher / collection router + -> windowed raw, exact-aggregation, or sketch state + -> full/delta encoder + SID dictionary + -> exporter + +BackendPlan + -> plan installer + -> ingest validator / optional backend precompute + -> SID resolver + instance metadata + -> summary store + -> query route + readout / explicit exact fallback +``` + +Cross-plan validation proves that every Collector-produced materialization has +one compatible backend ingest/storage declaration and that every planned query +route references a declared materialization. A Backend-only precompute has a +BackendPlan producer but no Collector materialization; a raw pass-through has +matching raw transmission and ingest/archive declarations. + The compiler error must identify an unsupported capability, invalid placement, window incompatibility, identity conflict, or invalid selected guarantee. It must not silently substitute another logical summary. diff --git a/control_plane/docs/developer_docs/runtime-plan-publication.md b/docs/developer_docs/control-plane/plan-publication.md similarity index 54% rename from control_plane/docs/developer_docs/runtime-plan-publication.md rename to docs/developer_docs/control-plane/plan-publication.md index 701d2966b..6d62ec483 100644 --- a/control_plane/docs/developer_docs/runtime-plan-publication.md +++ b/docs/developer_docs/control-plane/plan-publication.md @@ -25,9 +25,13 @@ CollectorReport BackendPlanReport `CollectorClient` is the ASAPQuery-side counterpart of ASAPCollector's authoritative -[`opamp-config-push.md`](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/developer_docs/opamp-config-push.md). +[`opamp-config-push.md`](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/developer_docs/asapcollector/opamp-config-push.md). This repository does not redefine CollectorPlan fields. +Publication also does not reinterpret either plan. The physical compiler owns +their contents; the publisher owns delivery, staging, activation barriers, +report correlation, and rollback. + ## 2. Public interfaces and definitions ### Runtime clients @@ -41,6 +45,20 @@ pub trait CollectorPlanClient { target: CollectorTarget, plan: CollectorPlan, ) -> Result; + + async fn activate( + &self, + target: CollectorTarget, + plan_id: &str, + plan_version: u64, + ) -> Result; + + async fn abort( + &self, + target: CollectorTarget, + plan_id: &str, + plan_version: u64, + ) -> Result; } pub trait BackendPlanClient { @@ -51,6 +69,20 @@ pub trait BackendPlanClient { target: BackendTarget, plan: BackendPlan, ) -> Result; + + async fn activate( + &self, + target: BackendTarget, + plan_id: &str, + plan_version: u64, + ) -> Result; + + async fn abort( + &self, + target: BackendTarget, + plan_id: &str, + plan_version: u64, + ) -> Result; } ``` @@ -64,6 +96,20 @@ ASAPCollector interface: - `RemoteConfigStatus.APPLIED` is delivery/application evidence, not semantic activation evidence. +The payload delivered to each target is different even though it shares one +bundle envelope: + +| Target | Published artifact | Components that must stage it | +| --- | --- | --- | +| Each ASAPCollector | That target's `CollectorPlan` YAML in `asap-collector-plan.yaml` | OpAMP receiver, plan validator, collection/precompute runtime, window manager, transmission/export pipeline | +| ASAPQuery data plane | One typed `BackendPlan` | Plan manager, ingest/precompute engine, SID/metadata registry, summary store, query router/readout and fallback adapter | + +The Collector report proves the expected materialization definitions are +installed and executable. It does not enumerate future per-label SIDs. The +Backend report proves matching descriptors, ingest/storage routes, and query +routes are staged. SIDs are allocated or resolved later as concrete +materialized series arrive. + ### Application reports ```rust @@ -144,6 +190,41 @@ pub struct ActivationResult { plan/version/compatibility and expected materializations. Partial staging is an error result and keeps the prior valid plan authoritative. +### Publication and activation sequence + +```text +1. validate complete CompiledPlanBundle +2. stage BackendPlan (ingest/store/query routes not yet authoritative) +3. stage every CollectorPlan (collection/export not yet authoritative) +4. correlate reports with envelope, targets, capabilities, and expected materializations +5. wait for the common activation time and all readiness conditions +6. issue activation for the common time; install backend routing before allowing Collector export +7. observe emitted plan/materialization IDs and retain the previous version for drain/rollback +``` + +Backend staging precedes Collector staging so the consumer can validate the +contract before new producers are allowed to emit. Staging order alone is not +activation: both sides remain gated by the shared activation time and the +publisher's readiness decision. If any required target rejects or times out, +the publisher aborts the new version on every staged target; the previous +unexpired bundle remains authoritative. + +`stage`, `activate`, and `abort` are distinct semantic operations even when a +transport implements them as versioned remote-config updates. An `APPLIED` +transport acknowledgement for staged bytes must not be mapped directly to +`Active`. Distributed activation cannot be literally instantaneous, so the +backend installs the accepting ingest view first, both sides use the common +activation timestamp, and query routing becomes authoritative only after the +required active reports correlate. During that bounded transition the backend +may accept new-version payloads without serving queries from an incomplete +new-version view. + +For replacement, Collector state created under the old materialization drains +according to its lifecycle policy, while the backend retains the corresponding +SID metadata and query route for the declared drain horizon. A changed summary +semantic produces a new materialization identity and new SIDs; publication +must not relabel old state into the new definition. + ## 3. Adding and verifying functionality ### Add another collector transport @@ -177,4 +258,11 @@ Interpretation: a successful `stage` report is not global activation; only - stale/expired/conflicting versions fail; - collector-only or backend-only success never returns `Active`; - report artifacts are machine-readable by the MVP harness; and -- post-activation emitted state carries the activated identities. +- post-activation emitted state carries the activated identities; +- the BackendPlan declares ingest/storage for every CollectorPlan output; +- every backend query route references a staged materialization descriptor; +- Collector reports refer to materialization definitions, not runtime SIDs; +- no Collector begins new-version export before compatible backend readiness; + and +- failed activation sends an abort/rollback action to every target that staged + the candidate version. diff --git a/docs/developer_docs/control-plane/runtime-accuracy-feedback.md b/docs/developer_docs/control-plane/runtime-accuracy-feedback.md new file mode 100644 index 000000000..f619c4d84 --- /dev/null +++ b/docs/developer_docs/control-plane/runtime-accuracy-feedback.md @@ -0,0 +1,25 @@ +# Developing runtime accuracy feedback + +## Architecture + +Runtime samples and monitoring collect Collector/Backend application evidence, +query accuracy/freshness/latency, and cost signals. The replanner validates and +keys evidence to the exact plan/materialization/family/runtime versions before +building a new workload/cost input. + +## Interfaces + +Feedback records need plan and materialization identity, SID lineage, family and +parameters, query shape, labels/timestamps alignment summary, sample count, +observed error/freshness/latency/resource values, collection time, and producer +version. A violation event references the declared threshold and observed value. + +## Extension and verification + +New evidence requires a unit, aggregation rule, freshness rule, cardinality, +privacy classification, and Planner consumer. Tests reject unattributed, +stale, under-sampled, wrong-version, or incomparable evidence. Replanning tests +verify a new plan version and fail-safe publication/rollback, not in-place +mutation of an active SID. + +See [runtime feedback design](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/design_docs/control-plane/runtime-accuracy-feedback.md). diff --git a/docs/developer_docs/control-plane/workload-inputs.md b/docs/developer_docs/control-plane/workload-inputs.md new file mode 100644 index 000000000..e51f7de82 --- /dev/null +++ b/docs/developer_docs/control-plane/workload-inputs.md @@ -0,0 +1,24 @@ +# Developing query and data workload inputs + +## Architecture + +The control-plane workload registry and analyzer normalize query registrations, +runtime query demand, data statistics, deployment inventory, and planning +horizon into ASAPPlanner input. Planner's workload model at commit `d31a566` is +the semantic source; local code adapts transport and runtime evidence. + +## Interfaces + +An input snapshot includes query expression/normalized identity, recurrence, +time scope, accuracy/freshness/latency requirements, data arrival/cardinality/ +distribution evidence, horizon, timestamp, and evidence version. Unknown fields +remain explicit. + +## Extension and verification + +Adding an input requires provenance, freshness, tenant scope, serialization, +and deterministic normalization. Tests cover one-time/recurring demand, +historical/live data, unknown versus zero, stale evidence, duplicate +registrations, and fixed-snapshot reproducibility. + +See [workload input design](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/design_docs/control-plane/workload-inputs.md). diff --git a/docs/developer_docs/adding-summary-family.md b/docs/developer_docs/cross-component-adding-summary-family.md similarity index 100% rename from docs/developer_docs/adding-summary-family.md rename to docs/developer_docs/cross-component-adding-summary-family.md diff --git a/data_plane/docs/developer_docs/otlp-summary-ingestion.md b/docs/developer_docs/ingest-engine/ingest-engine.md similarity index 93% rename from data_plane/docs/developer_docs/otlp-summary-ingestion.md rename to docs/developer_docs/ingest-engine/ingest-engine.md index ebc1c2cb1..8cf87ac5e 100644 --- a/data_plane/docs/developer_docs/otlp-summary-ingestion.md +++ b/docs/developer_docs/ingest-engine/ingest-engine.md @@ -72,14 +72,14 @@ pub enum SummaryFrame { ```rust pub trait SeriesIdentityResolver { type Error; - fn resolve(&self, key: CanonicalSeriesKey) + fn resolve(&self, key: CanonicalMaterializedSeriesKey) -> Result; } ``` -`SeriesId`, `SeriesIdNamespace`, `CanonicalSeriesKey`, and `ResolvedSeries` have +`SeriesId`, `SeriesIdNamespace`, `CanonicalMaterializedSeriesKey`, and `ResolvedSeries` have one public definition in -[Summary storage and series identity](summary-storage-and-series-identity.md#sid-definition). +[Summary storage and series identity](../summary-store-engine/implementation.md#sid-definition). ```rust pub trait SummaryValidator { @@ -160,7 +160,7 @@ recovery full state. Verify `AppliedDelta`, `Duplicate`, and ### Add series identity behavior -Add canonical input fields to `CanonicalSeriesKey`, never to `SeriesId.value` +Add canonical input fields to `CanonicalMaterializedSeriesKey`, never to `SeriesId.value` alone. Verify label-order independence, tenant isolation, cached-ID conflict recovery, and stable namespace reporting. diff --git a/data_plane/docs/developer_docs/backend-plan-runtime.md b/docs/developer_docs/query-engine/backend-plan-runtime.md similarity index 100% rename from data_plane/docs/developer_docs/backend-plan-runtime.md rename to docs/developer_docs/query-engine/backend-plan-runtime.md diff --git a/data_plane/docs/developer_docs/extension-points.md b/docs/developer_docs/query-engine/extension-points.md similarity index 97% rename from data_plane/docs/developer_docs/extension-points.md rename to docs/developer_docs/query-engine/extension-points.md index 2829644cd..98e066dc0 100644 --- a/data_plane/docs/developer_docs/extension-points.md +++ b/docs/developer_docs/query-engine/extension-points.md @@ -64,7 +64,7 @@ pub struct ExactBackendCapabilities { ``` `QueryRequest` and `QueryResponse` are defined in -[Query routing and readout](query-routing-and-readout.md). They preserve tenant, +[Query routing and readout](query-engine.md). They preserve tenant, query language/expression, logical evaluation range, requested accuracy, result labels/timestamps/type, source, guarantee, and coverage. diff --git a/data_plane/docs/developer_docs/query-routing-and-readout.md b/docs/developer_docs/query-engine/query-engine.md similarity index 100% rename from data_plane/docs/developer_docs/query-routing-and-readout.md rename to docs/developer_docs/query-engine/query-engine.md diff --git a/docs/developer_docs/summary-series-id/resolver.md b/docs/developer_docs/summary-series-id/resolver.md new file mode 100644 index 000000000..822b8e9dd --- /dev/null +++ b/docs/developer_docs/summary-series-id/resolver.md @@ -0,0 +1,39 @@ +# Developing summary-series identity + +## Architecture + +The Backend is the SID authority. `drivers/ingest/series_resolver.rs` maps: + +```text +(canonical metric, retained-attributes fingerprint, + materialization/agg-kind canonical string) -> u64 +``` + +The OTLP receiver handles attribute-bearing registration, ID-only lookup, +assignment responses, and unknown-ID feedback. `SketchStore` owns the reverse +metadata needed after resolution. Collector owns the assignment cache. + +## Contract + +SID identifies one stored summary/exact/raw materialization series, not +necessarily one original input time series. SID zero is unassigned. A supplied +ID never overrides conflicting labels/kind. Different family, parameters, +filter, or retained label values require different SIDs; window/full/delta does +not. + +`FilePersistence` WAL recovery can preserve assignments across Backend restart. +Without it, unknown-ID feedback makes Collector evict and re-register. + +## Safe changes + +Canonicalization changes are cross-repository wire changes. Version them or +prove old/new equivalence with shared vectors. When adding tenant/distributed +namespaces, make namespace evidence explicit before allowing the same numeric +value in multiple authorities. + +## Verification + +Test label-order stability, kind/config separation, concurrent idempotent +resolve, conflict handling, ID-only unknown rejection, WAL replay/torn tail, +Collector eviction/relearning, and query label reconstruction. See the +[cross-repository design](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/design_docs/cross-cutting/summary-series-id.md). diff --git a/data_plane/docs/developer_docs/summary-storage-and-series-identity.md b/docs/developer_docs/summary-store-engine/implementation.md similarity index 78% rename from data_plane/docs/developer_docs/summary-storage-and-series-identity.md rename to docs/developer_docs/summary-store-engine/implementation.md index e608803b0..42e4de10a 100644 --- a/data_plane/docs/developer_docs/summary-storage-and-series-identity.md +++ b/docs/developer_docs/summary-store-engine/implementation.md @@ -6,18 +6,20 @@ ## 1. Code architecture ```text -CanonicalSeriesKey -> SeriesRegistry -> SeriesId - | -ValidatedMaterialization ----------------+ +CanonicalMaterializedSeriesKey -> SeriesRegistry -> SeriesId + ^ | +ValidatedMaterialization ----------------------------+ | v SummaryStore write / coverage / read / retire ``` -The series registry owns only canonical metric-series identity. The summary -store owns materialization/group/window state. Plan, materialization, and SID -identities remain distinct. +The series registry owns canonical **materialized-series** identity. Its key +combines the materialization definition with the canonical metric and concrete +retained label values. The summary store owns the windows and payload state +under the resulting SID. Plan, materialization, and SID identities remain +distinct but related. ## 2. Public interfaces and definitions @@ -32,21 +34,22 @@ pub struct SeriesIdNamespace { pub version: String, } -pub struct CanonicalSeriesKey { +pub struct CanonicalMaterializedSeriesKey { pub tenant: String, + pub materialization_fingerprint: String, pub metric_name: String, pub identifying_labels: BTreeMap, } pub struct ResolvedSeries { pub id: SeriesId, - pub canonical_key: CanonicalSeriesKey, + pub canonical_key: CanonicalMaterializedSeriesKey, } pub trait SeriesRegistry: Send + Sync { type Error; - fn resolve(&self, key: CanonicalSeriesKey) + fn resolve(&self, key: CanonicalMaterializedSeriesKey) -> Result; fn lookup(&self, id: &SeriesId) @@ -56,13 +59,22 @@ pub trait SeriesRegistry: Send + Sync { ### SID definition +The interfaces below are the target public namespace model. The current +implementation uses a backend-allocated `u64` keyed by `(canonical metric, +canonical stored labels, agg_kind_canonical)`. It therefore identifies a stored +summary or exact-aggregate series. The same target model can identify a raw +sample series under a distinct raw materialization kind, but current raw samples +are served through the configured archive/pass-through path rather than a +`SketchStore` raw payload variant. See the +[cross-repository identity design](../summary-series-id/resolver.md). + `SeriesId` (`sid`) is an opaque numeric identifier scoped by exactly one `SeriesIdNamespace`. The namespace contains the tenant/isolation domain and a version that changes whenever the authoritative registry is rebuilt without preserving its previous assignments. ```text -(SeriesIdNamespace, SeriesId.value) <-> CanonicalSeriesKey +(SeriesIdNamespace, SeriesId.value) <-> CanonicalMaterializedSeriesKey ``` Within one namespace this mapping is one-to-one: @@ -159,9 +171,9 @@ pub enum CoverageResult { Supporting types `ValidatedMaterialization`, `ValidatedSummary`, and `IngestDisposition` are defined by -[OTLP summary ingestion](otlp-summary-ingestion.md). `EvaluationRange`, +[OTLP summary ingestion](../ingest-engine/ingest-engine.md). `EvaluationRange`, `SummaryRoute`, and `LogicalCoverage` are defined by -[Query routing and readout](query-routing-and-readout.md). +[Query routing and readout](../query-engine/query-engine.md). Why these interfaces exist: callers receive typed completeness/failure rather than interpreting an empty collection as “no data,” and storage cannot accept diff --git a/docs/user_guide/README.md b/docs/user_guide/README.md new file mode 100644 index 000000000..1d32717bc --- /dev/null +++ b/docs/user_guide/README.md @@ -0,0 +1,14 @@ +# ASAPQuery-backend user guide + +This guide is for operators and API users. It covers startup checks and the +PromQL-compatible query surface, not private Rust interfaces. + +## Start here + +- [Run and verify the backend](running-and-verifying.md) — build prerequisites, + component startup, health checks, and troubleshooting. +- [Querying ASAP](../../docs/user_guide/querying-asap.md) — supported + query behavior, accuracy, freshness, fallback, and errors. + +For the complete multi-node demonstration, use +[ASAPCollector's MVP demo runbook](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/user_guide/mvp-demo-runbook.md). diff --git a/data_plane/docs/user_guide/querying-asap.md b/docs/user_guide/querying-asap.md similarity index 90% rename from data_plane/docs/user_guide/querying-asap.md rename to docs/user_guide/querying-asap.md index 51ace1afa..79b0c0c07 100644 --- a/data_plane/docs/user_guide/querying-asap.md +++ b/docs/user_guide/querying-asap.md @@ -78,6 +78,6 @@ incompatible materialization, or exact-backend failure. ## Related documentation -- [Data-plane design](../design_docs/query-execution.md) -- [Summary storage](../../../docs/design_docs/summary-storage.md) +- [Data-plane design](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/design_docs/asapquery-backend/query-query-engine.md) +- [Summary storage](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/design_docs/asapquery-backend/query-summary-store-engine.md) - [ASAPCollector MVP demo runbook](https://github.com/ProjectASAP/ASAPCollector/blob/main/docs/user_guide/mvp-demo-runbook.md) diff --git a/docs/user_guide/running-and-verifying.md b/docs/user_guide/running-and-verifying.md new file mode 100644 index 000000000..b4ec0ec6b --- /dev/null +++ b/docs/user_guide/running-and-verifying.md @@ -0,0 +1,95 @@ +# Run and verify the backend + +## Prerequisites + +Clone `ASAPCollector`, `ASAPQuery-backend`, and `asap_sketchlib` as sibling +directories. Install the Rust toolchain required by the workspace, then build: + +```bash +cargo build --workspace +``` + +For a fully configured system, prefer the ASAPCollector deployment harness. The +commands below are useful for component development and diagnostics. + +## Inspect runtime options + +```bash +cargo run -p data_plane -- --help +``` + +The data plane requires a streaming configuration file. A minimal development +launch follows this shape: + +```bash +cargo run -p data_plane -- \ + --streaming-config PATH_TO_STREAMING_CONFIG \ + --http-port 8088 \ + --enable-otel-ingest \ + --otel-grpc-port 4317 \ + --otel-http-port 4318 +``` + +Use a checked-in or controller-emitted streaming configuration appropriate to +your workload; do not invent aggregation identities by hand for production. + +Run the control plane with its defaults: + +```bash +cargo run -p control_plane +``` + +Important default listeners are: + +| Component | Endpoint | Purpose | +| --- | --- | --- | +| Control plane | `http://localhost:8080` | Planning and configuration API | +| Control plane | `ws://localhost:4320/v1/opamp` | Collector OpAMP connection | +| Data plane | `http://localhost:8088` | PromQL-compatible query and diagnostics | +| Data plane | `localhost:4317` | OTLP gRPC ingest when enabled | +| Data plane | `localhost:4318/v1/metrics` | OTLP HTTP ingest when enabled | + +Deployment configuration can override these values. + +## Verify readiness + +Check data-plane health: + +```bash +curl -fsS http://localhost:8088/api/v1/health +``` + +Inspect installed runtime state when diagnosing plan or routing problems: + +```bash +curl -fsS http://localhost:8088/api/v1/streaming-config +curl -fsS http://localhost:8088/api/v1/backend-plan +curl -fsS http://localhost:8088/api/v1/storage_routing +``` + +Run an instant query: + +```bash +curl -fsS --get http://localhost:8088/api/v1/query \ + --data-urlencode 'query=up' +``` + +A ready result has `status: success` and the expected `infos` annotations for +data source and accuracy. Health alone does not prove that plans, summary data, +or exact fallback are available. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Build cannot resolve a path dependency | Verify the three sibling repository names and locations | +| Data plane exits immediately | Supply `--streaming-config` and check YAML errors and port conflicts | +| OTLP connection is refused | Start with `--enable-otel-ingest` and verify ports 4317/4318 | +| Query returns no compatible summary | Inspect streaming config, BackendPlan, storage routing, metric labels, and window | +| Unsupported query fails | Configure an exact backend or use a supported planned query | +| Result comes from an unexpected tier | Inspect `infos` provenance and the installed storage routing table | +| Results are stale | Check collector export, OTLP ingest, active-window state, and source timestamps | + +Capture component commits, effective configuration, relevant logs, diagnostic +endpoint output, and one complete query response when reporting a problem. +Remove credentials and environment secrets first.