[FEATURE] Unified query API layer

[dai-chen]

Is your feature request related to a problem?

The unified query API layer is the cornerstone of the broader PPL unification effort across engines (see opensearch-project/opensearch-spark#1136). However, PPL parsing, validation, planning, and execution are tightly coupled to the current PPL implementation inside OpenSearch. This makes it hard for other engines and tools to integrate in a consistent way.

We’ve started to address this with the new unified-query-api module introduced in #3734, which exposes a basic query planning API that returns a RelNode as IR. This initial API surface is sufficient to unblock the first phase of Spark SQL integration, additional APIs are required for deeper integration and for other consumers such as the CLI and other engines.

What solution would you like?

As an immediate solution, we propose to evolve the unified-query-api module into a stable, engine-agnostic “front door” for PPL. This layer will expose a small set of well-defined operations over PPL queries and functions, so that all engines integrate with PPL through the same API instead of talking directly to OpenSearch/Calcite internals.

Unified Language Specification

1. parse() API – syntax & custom validation: Provide a parse() API for consumers that only need syntax/semantic checks (possibly with their own rules). The API should return an AST/parse result together with diagnostics and support pluggable custom visitors for additional validation or query-shape checks.

2. transpile() API – PPL → engine-specific query text: Provide a transpile() API for engines like Spark that want to run PPL by translating it to their native language. This API should translate PPL into a target SQL dialect (for example, Spark SQL), and preserve PPL semantics and function behavior as much as possible.

Unified Execution Runtime

1. Reference Implementation: Provide an evaluate() API for consumers such as the CLI or OpenSearch PPL that need a reference runtime. This API should execute a given query using a Calcite-based in-memory evaluator, and serve use cases like CLI execution, testing/compliance, and acting as a behavioral reference for other engines.

2. Unified Function Library: Provide a unified function library with a resolve / load function API, so engines can then adapt these unified function definitions into their native function mechanisms (such as Spark Catalyst expressions), ensuring a single source of truth for PPL function signatures and semantics.

// Validation only
UnifiedQuery
  .lang(PPL)
  .visitor(new QueryValidation())
  .parse("source = opensearch.test");


// Spark – transpile PPL → Spark SQL
val sqlText =
  UnifiedQuery
    .lang(PPL)
    .catalog("opensearch", schema)
    .defaultNamespace("opensearch")
    .transpile("source = opensearch.test", Dialect.Spark);

spark.sql(sqlText);


// CLI / reference runtime – evaluate directly
UnifiedQuery
  .lang(PPL)
  .catalog("opensearch", schema)
  .defaultNamespace("opensearch")
  .eval("source = opensearch.test");

What alternatives have you considered?

N/A

Do you have any additional context?

• Initial unified query planner PR: Add unified query API for external integration #3783
• API module documentation: https://github.com/opensearch-project/sql/tree/main/api
• Short-term solution: leverage this thin Unified Query Layer atop existing modules, exposing a single facade API, pluggable SPIs, and a lightweight conformance suite. This gives downstream consumers a clean entry point to integrate and adopt PPL with minimal changes to the current codebase.

• Longer-term solution: Evolve PPL into a layered stack with a well-defined language specification and a reference (standard) runtime. Specifically, extract a Unified Language Specification from the current ppl/core modules and a Unified Execution Runtime from core, with clear API boundaries; in parallel, abstract specific data-source access into pluggable adapters for OpenSearch, Spark, and Flint indexes, etc.