feat: migrate ClickHouse adapter to utopia-php/query 0.3.x builder#120
feat: migrate ClickHouse adapter to utopia-php/query 0.3.x builder#120lohanidamodar wants to merge 26 commits into
Conversation
Pins to dev-feat/clickhouse-insert-delete-settings-mv as 0.3.2 — the branch on utopia-php/query (PR #11) that adds the three ClickHouse builder extras audit needs: - Builder\ClickHouse::insertFormat(...) for INSERT ... FORMAT JSONEachRow - Trailing SETTINGS clause on DELETE (for async cleanup) - Schema\ClickHouse::create/dropMaterializedView (not used here) TODO: flip to ^0.3.2 once PR #11 lands on utopia-php/query main and a 0.3.2 tag exists.
The 0.3.x utopia-php/query base class replaced the legacy `TYPE_*` string constants with a `Method` enum. Audit's adapter switches and tests all still compare against the string constants, so they're re-declared here mapped to the same string values (`equal`, `lessThan`, ...). Tests that compared `$query->getMethod()` (now returns a `Method` enum) against the constants are updated to compare against `getMethod()->value`.
utopia-php/query 0.3.x uses PHP 8.4 asymmetric visibility (`public protected(set)`) on schema and builder properties. PHPStan 1.12's bundled PhpParser can't tokenize that syntax — `composer check` crashes with a Lexer internal error on any audit source file that imports a query schema/builder class. PHPStan 2.x ships an updated PhpParser that handles asymmetric visibility, so bumping the dev dependency is the smallest change that restores the static-analysis gate.
…sites utopia-php/query 0.3.x's `Query::getMethod()` returns a `Method` enum instead of a string. The ClickHouse adapter's parseQueries switch, and the Database adapter's count-time filter, both compare against the legacy string `TYPE_*` constants. Switch from `getMethod()` to `getMethod()->value` at the two call sites so the existing comparisons keep matching.
Bumping PHPStan to 2.x to handle PHP 8.4 syntax surfaces a handful of pre-existing diagnostics in code paths the migration does not otherwise touch: - Database adapter: redundant `instanceof Utopia\Audit\Query` guards inside a method whose signature already constrains the parameter. Kept as runtime defense (real callers occasionally pass mixed arrays); annotated with @phpstan-ignore-next-line. - Log::getData(): PHPStan can't widen the ArrayObject return without a local @var hint. - AuditBase batch test: removed a duplicate `applyRequiredAttributesToBatch` call (typo; the second invocation already re-merged the same row).
Replace the hand-built CREATE TABLE in `setup()` with a `Schema\ClickHouse` table builder. Engine, ORDER BY tuple, PARTITION BY expression, table settings, columns, and data-skipping (bloom_filter) indexes are now declared through the schema API; the resulting Statement is executed verbatim through the existing HTTP layer. Column definitions follow the audit attribute descriptors: - `id String` (primary) - `time DateTime64(3)` (NOT NULL — partition key) - `<id> [Nullable(]String[)]` for every other attribute, nullability driven by the descriptor's `required` flag - `tenant Nullable(UInt64)` when sharedTables is on CREATE DATABASE IF NOT EXISTS still goes through a raw string — the schema's `createDatabase()` helper has no IF NOT EXISTS form and we'd otherwise break the second `setup()` call. Also folds in a few PHPStan-driven cleanups in this file (collapsed `!empty($inParams)` guards into unconditional emits since the upstream VALUE_REQUIRED_METHODS guard already rejects empty value lists; dropped the redundant is_array($row) inside parseJsonResults whose row type was already array<string,mixed>).
…eBatch Build the JSONEachRow INSERT statement through `Builder\ClickHouse::insertFormat` instead of a hand-rolled string. Column list is derived from the audit schema (id, time, the remaining attributes, optional tenant) in the same insertion order as the row maps so the JSON keys line up against ClickHouse's declared columns. The JSONEachRow body itself still serializes in the adapter — that's an HTTP-payload concern that stays in the runtime layer.
Compile the cleanup DELETE through `Builder\ClickHouse::delete()` with a
trailing SETTINGS clause emitted by the builder when async cleanup is
enabled. The DELETE WHERE expression is supplied via `whereRaw()` so the
ClickHouse-typed parameter syntax (`{datetime:DateTime64(3)}`) and the
existing tenant filter stay in the runtime layer — the builder still
emits generic `?` placeholders today (dry-run gap #7).
Behaviour note: the ClickHouse builder compiles DELETE as
`ALTER TABLE ... DELETE` (mutation) instead of the lightweight
`DELETE FROM ...` the previous string emitted. The async setting tracks
the same shift: `mutations_sync = 0` for mutations replaces the
previous `lightweight_deletes_sync = 0`. End-state row visibility is
the same; the storage path is different (mutations rewrite parts,
lightweight deletes mask rows).
Route the SELECT skeleton (FROM, raw projection, WHERE conjunction,
ORDER BY) for `find()`, `count()`, and `getById()` through
`Builder\ClickHouse`. Filter expressions emitted by `parseQueries()`
still carry ClickHouse-typed parameter hints (`{name:Type}`) and the
audit-side associative params map; both ride along via `whereRaw()` /
`orderByRaw()` so the typed-binding HTTP layer keeps working.
`LIMIT` / `OFFSET` and the trailing `FORMAT JSON` / `FORMAT TabSeparated`
are appended on the compiled SQL string — the base builder emits
positional `?` placeholders for limit/offset, which would collide with
the ClickHouse `{name:UInt64}` placeholders the runtime layer binds.
Cleaning that up belongs with the gap #7 (ClickHouse param hint) work
on utopia-php/query.
Cursor pagination still builds tuple WHERE fragments through the
existing `buildCursorWhere()` helper and feeds them in via `whereRaw()`
(gap #8, deferred).
Pin the SQL emitted by `Schema\ClickHouse` and `Builder\ClickHouse` through the audit configurations the adapter relies on: - `setup()` CREATE TABLE — engine, columns, indexes, ORDER BY, PARTITION BY, SETTINGS - `createBatch()` `INSERT ... FORMAT JSONEachRow` with the audit column list - `cleanup()` async DELETE with trailing SETTINGS clause - `cleanup()` sync DELETE with no SETTINGS - `find()` SELECT with whereRaw filters, cursor tuple comparison, ORDER BY tiebreaker, LIMIT and FORMAT JSON tail These do not require a live ClickHouse so they run as fast unit tests and prevent silent SQL drift from a query-lib upgrade.
Greptile SummaryThe PR migrates ClickHouse SQL generation to the utopia-php/query 0.3 builder while preserving existing adapter behavior.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains; the previously reported cleanup semantics, malformed-row guard, multi-value exclusion coverage, and cursor predicate grouping are corrected at current HEAD. Important Files Changed
Reviews (11): Last reviewed commit: "Merge origin/main into feat/utopia-query..." | Re-trigger Greptile |
Bump utopia-php/query to the upstream branch HEAD that ships named-typed
{name:Type} placeholder support and the lightweight DELETE form, then add
two helpers to the adapter:
- getColumnTypeMap() derives a column → ClickHouse-type map from the
schema attributes (DateTime → DateTime64(3), all other columns →
String) plus id, tenant (when sharedTables is enabled) and the
limit/offset/max pseudo-columns used by the count/find SQL wrappers.
- newBuilder() returns a Builder\ClickHouse with useNamedBindings() and
withParamTypes() pre-applied, so positional `?` bindings flow through
the typed-binding rewriter at Statement-emission time.
Every existing adapter call site is rerouted from `new ClickHouseBuilder()`
to `$this->newBuilder()`. The current call sites all hand-construct their
{name:Type} placeholders inside whereRaw fragments with zero `?` bindings,
so this change is a no-op for the existing SQL shape — it just preps the
infra that the find()/count()/getById() reads will switch to next.
…ter API
Drop every hand-built WHERE / ORDER BY fragment in find(), count() and
getById() and feed Query value objects straight to Builder\ClickHouse via
filter(), sortAsc/sortDesc/sortRandom, limit() and offset(). Positional
`?` bindings are rewritten to `{paramN:Type}` placeholders by the
typed-binding registration installed in newBuilder(), so HTTP params now
flow through $statement->namedBindings instead of a hand-maintained
$paramCounter dict.
parseQueries() is reduced to a list of Query objects plus auxiliary
metadata (orderAttributes, limit, offset, cursor, select). Two
audit-specific rewrites stay in this layer:
- Contains / NotContains are remapped to Equal / NotEqual so they keep
audit's historical IN / NOT IN semantics. The base builder compiles
Contains to substring-match `position(x, ?) > 0`, which is not what
callers like Audit::getByUserAndEvents() expect.
- `time`-column DateTime values are stringified at parse time so they
appear in namedBindings as `Y-m-d H:i:s.v` literals rather than raw
objects the HTTP layer can't serialise.
Cursor pagination keeps its whereRaw escape hatch with explicit
{name:Type} placeholders — Builder\ClickHouse still has no tuple-compare
helper, so the existing buildCursorWhere() output is appended to the
builder and its params are merged into the final HTTP request alongside
$statement->namedBindings. The dead buildOrderBySql() helper is removed;
applyOrderBy() drives the builder directly.
Builder\ClickHouse now defaults to DELETE_MODE_LIGHTWEIGHT, so cleanup() emits `DELETE FROM t WHERE …` again — matching audit's pre-migration baseline. The previous mutation form (`ALTER TABLE t DELETE WHERE …`) was a workaround forced by the older builder API and changed the storage-path semantics: lightweight marks rows deleted via a mask and is the right tool for row-level cleanup, while mutations rewrite parts on disk and are heavier. The async SETTINGS knob switches from `mutations_sync = 0` to `lightweight_deletes_sync = 0` to match the new DELETE form. The public setAsyncCleanup() docblock already referenced lightweight_deletes_sync, so no docs change is needed.
…LETE
Pin the new SQL shape on the adapter's hot paths:
- cleanup() emits `DELETE FROM t WHERE …` with an optional trailing
`SETTINGS lightweight_deletes_sync=0` for the async path.
- find() / count() filter via Builder\ClickHouse::filter() and
sortAsc/sortDesc/limit, producing typed `{paramN:Type}` placeholders
and a populated `namedBindings` map on the Statement.
- The cursor whereRaw fragment with explicit `{name:Type}` placeholders
composes cleanly with the typed positional bindings — both end up in
the final HTTP params dict by the adapter merging them.
- count(max) wraps the inner Statement in `SELECT COUNT(*) FROM (… LIMIT ?) sub`
and reuses the inner builder's namedBindings unchanged.
A small newAuditBuilder() / auditTypeMap() helper mirrors the adapter's
own column → ClickHouse-type registration so the snapshot tests exercise
the same typed-binding flow callers will see in production.
The Dockerfile pinned php:8.3.3-cli-alpine3.19, but composer.json requires php >=8.4. Once utopia-php/query: ^0.3.0 was adopted, the CI Tests check failed because the library uses 8.4-only asymmetric visibility syntax (public protected(set)). Bump the test image to php:8.4.21-cli-alpine3.23 so the CI environment matches the package requirement.
ClickHouse JSON responses come from json_decode and aren't statically guaranteed to satisfy any inner shape. The pre-migration code skipped non-array rows defensively; restore that guard and loosen the @var on \$data to array<int, mixed> so PHPStan max accepts the runtime check instead of pruning it as already-narrowed.
Adds a snapshot test that locks the SQL emitted when TYPE_NOT_CONTAINS
is mapped to Method::NotEqual with multiple values. The base builder
compiles NotEqual with 2+ non-null values via compileNotIn(), producing
`attr NOT IN (?, ?)`; with named typed bindings on Builder\ClickHouse
this becomes `attr NOT IN ({param0:Type}, {param1:Type})`. Pinning the
shape guards against a query-lib drift silently degrading multi-value
notContains() to a single-value `!=` filter.
| if ($this->asyncCleanup) { | ||
| $builder->settings(['lightweight_deletes_sync' => '0']); | ||
| } |
There was a problem hiding this comment.
The
cleanup() builder uses lightweight_deletes_sync here, but the PR description states the setting was changed to mutations_sync = 0. DELETE FROM is ClickHouse's lightweight-delete syntax, and lightweight_deletes_sync is the correct setting for it — so the code and snapshot test are internally consistent. However, the PR description is factually wrong when it says this was switched to mutations_sync, which could mislead future maintainers into thinking the mutation semantics changed.
| if ($this->asyncCleanup) { | |
| $builder->settings(['lightweight_deletes_sync' => '0']); | |
| } | |
| if ($this->asyncCleanup) { | |
| $builder->settings(['mutations_sync' => '0']); | |
| } |
Brings in actor-terminology rename (PR #122), location-column drop / N-part resource parser (PR #118), and country / userAgent test refactors. Conflict resolution (src/Audit/Adapter/ClickHouse.php parseQueries): - Kept main's call to translateAttribute() right after reading $query->getAttribute() so callers passing legacy 'userId' / 'userType' / 'userInternalId' Query filters are rewritten to the renamed 'actorId' / 'actorType' / 'actorInternalId' columns before the typed-bindings layer compiles them. - The column type map registered via withParamTypes() is derived from getAttributes() and already uses the actor* names, so once the attribute is translated the map binds it as String correctly. - Public Audit API (Audit::log, getLogsByUser, countLogsByUser, getLogsByUserAndEvents, countLogsByUserAndEvents) is unchanged -- the ClickHouse adapter's getByUser / countByUser implementations already build internal Query::equal('actorId', $userId) filters. ClickHouseSqlSnapshotTest: - Updated the synthetic audit type map and DDL snapshot to reflect actorId / actorType / actorInternalId columns, idx_actorId_event, and the dropped location column. Find / count / cursor snapshots now filter on actorId.
utopia-php/query 0.3.2 is now tagged and published with all ClickHouse builder/schema features this PR depends on. Switch from the temporary dev-branch alias to the stable semver constraint.
Re-introduces the dev-branch pin (b26baf9) for the duration of utopia-php/query PR #13's lifecycle so audit can adopt the new typed Builder\ClickHouse::bulkInsert(Format, rows, columns) bulk-ingest entry point ahead of the 0.3.3 tag. Relaxes composer's minimum-stability to dev with prefer-stable=true so the alias resolves. TODO: flip composer.json back to "utopia-php/query": "^0.3.3" and minimum-stability: "stable" once 0.3.3 is tagged.
…nual JSONEachRow body assembly
Switches createBatch() to the typed Builder\ClickHouse::bulkInsert(
Format::JSONEachRow, $rows, $columns) entry point so the INSERT envelope
and the serialized row body are produced together. The HTTP helper now
takes a pre-serialized $rawBody string instead of a $jsonRows array,
deleting the inline array_map(json_encode, ...) + implode("\n", ...)
body assembly loop — that serialization now lives in Format::serialize()
on the builder side, with the JSON flags (JSON_THROW_ON_ERROR,
JSON_UNESCAPED_SLASHES, JSON_UNESCAPED_UNICODE) baked into Format::JSONEachRow.
The SQL envelope shape is unchanged (`INSERT INTO ... (cols) FORMAT
JSONEachRow`); the snapshot test is updated to drive bulkInsert and now
pins the JSONEachRow body serialization with a fixture row, so an
upstream regression in Format::serialize() would be caught by CI.
Adopt utopia-php/database ^6.0.0 from main and express main's shared-table sort key (tenant, time, id), allow_nullable_key and LowCardinality columns through the Schema builder. LowCardinality is applied to required columns only because the query builder emits Nullable(LowCardinality(String)) for nullable columns, which ClickHouse rejects.
… utopia-php/client Swap the Fetch client for the PSR-18 utopia-php/client cURL adapter with connection reuse so the TCP handshake is paid once across queries, matching the transport used by utopia-php/usage. The constructor accepts an optional injected ClientInterface for custom transports.
…inality on country Query 0.3.3 compiles lowCardinality()->nullable() to LowCardinality(Nullable(T)), so the nullable country column again matches main's DDL.
Syncs the query 0.3.x migration onto audit main (post-2.9.1), keeping everything main added since the previous sync and re-expressing the ClickHouse-adapter parts through the utopia-php/query 0.3 builder. Dependencies - utopia-php/query 0.1.* -> 0.3.* (locked 0.3.3, stable; no dev pins) - php >=8.5 and utopia-php/validators 0.3.* from main - keeps main's utopia-php/fetch ^1.1 HTTP transport - minimum-stability back to "stable" Carried over from main and routed through the 0.3 builder - retention TTL (MODIFY TTL / REMOVE TTL) in setup() - sdk / sdkVersion columns and the _key_sdk index - premium geo columns (city, continentCode, subdivisions, isp, autonomousSystemNumber, autonomousSystemOrganization, connectionType, connectionUsageType, connectionOrganization) - parsed user-agent OS / client / device columns - LowCardinality column set, emitted via Schema\ClickHouse::lowCardinality() - contains/notContains substring semantics, now compiled by the builder to position(col, ?) > 0 / = 0 instead of hand-written LIKE '%needle%' - cleanup() DateTime64(3) parameter type - readonly promoted constructor properties and typed class constants Query 0.3 API - Query::getMethod() returns the Utopia\Query\Method enum; all method comparisons in ClickHouse and Database adapters use Method:: cases - Utopia\Audit\Query still exposes the legacy TYPE_* string constants
c7ad06b to
c1aefab
Compare
|
Moving this to the monorepo: utopia-php/monorepo#91
The |
Summary
Migrates the ClickHouse audit adapter off the legacy
utopia-php/query0.1.xcontract onto the 0.3.x
Builder\ClickHouse/Schema\ClickHouseAPI.Every SQL and DDL string the adapter emits now comes from the query library.
The audit side keeps only the HTTP transport, the column → ClickHouse type map
that turns positional
?bindings into typed{paramN:Type}placeholders, andthe tuple-cursor compilation.
This branch is a strict superset of
main.origin/mainis an ancestor ofthis branch (0 commits behind), so everything shipped up to and past 2.9.1 is
included and re-expressed through the 0.3 builder where it touches ClickHouse.
Dependencies
The only line that differs from
mainisutopia-php/query:0.1.*→0.3.*.utopia-php/query 0.3.3is tagged and stable — no dev-branch pinsanywhere,
minimum-stabilityis back tostable,stability-flagsis empty.php >=8.5andutopia-php/validators 0.3.*are taken from main.utopia-php/fetch ^1.1stays as the HTTP transport, matching main. (Anearlier revision of this branch swapped in PSR-18
utopia-php/client; thathas been reverted so the adapter constructor signature is byte-identical to
main's and cloud call sites need no change.)
utopia-php/query 0.3.3,utopia-php/validators 0.3.1,utopia-php/database 6.4.5,utopia-php/fetch 1.1.2.What is compiled by the query library now
setup()Schema\ClickHouse— columns,LowCardinality(...),Nullable(...), bloom-filter indexes,ENGINE,ORDER BY,PARTITION BY,SETTINGScreateBatch()Builder\ClickHouse::bulkInsert(Format::JSONEachRow, $rows, $columns)— emits theINSERT … FORMAT JSONEachRowenvelope and serializes the bodyfind()Builder\ClickHouse—filter(),sortAsc/sortDesc,sortRandom(),limit(),offset()count()Builder\ClickHouse, with the$maxvariant wrapping an innerSELECT 1 … LIMITgetById()Builder\ClickHouse::filter()+limit(1)cleanup()Builder\ClickHouse::delete()with optional trailingSETTINGS lightweight_deletes_sync = 0Typed placeholders come from
useNamedBindings()+withParamTypes(), fed by acolumn → type map derived from
getAttributes()(DateTime columns getDateTime64(3),tenant/limit/offset/maxgetUInt64, the restString).Main features carried over and routed through the builder
Everything
mainadded since this branch last synced now goes throughSchema\ClickHouse/Builder\ClickHouse:setRetention()/getRetention()) —MODIFY TTL … SETTINGS materialize_ttl_after_modify = 0, and the idempotentREMOVE TTLteardownthat tolerates tables with no TTL.
sdk/sdkVersioncolumns and the_key_sdkbloom-filter index.city,continentCode,subdivisions,isp,autonomousSystemNumber,autonomousSystemOrganization,connectionType,connectionUsageType,connectionOrganization.osCode,osName,osVersion,clientType,clientCode,clientName,clientVersion,clientEngine,clientEngineVersion,deviceName,deviceBrand,deviceModel.LOW_CARDINALITY_COLUMNSset, emitted via the schemabuilder's
lowCardinality(); nullable ones compile toLowCardinality(Nullable(String)).contains/notContainssubstring semantics. Main replaced the oldIN/NOT INbehaviour withLIKE '%needle%'/NOT LIKE '%needle%'plusmanual wildcard escaping. The 0.3 ClickHouse builder compiles these natively
to
position(col, ?) > 0andposition(col, ?) = 0— same results, sameOR/AND combination across multiple needles, and no wildcard escaping needed,
so
escapeLikeWildcards()is gone.cleanup()DateTime64(3)parameter type, readonly promoted constructorproperties, and typed class constants.
Query 0.3 API notes
Query::getMethod()returns theUtopia\Query\Methodenum. All methodcomparisons in
Adapter\ClickHouseandAdapter\Databaseare nowMethod::case comparisons, andVALUE_REQUIRED_METHODSholds enum cases.Utopia\Audit\Querystill exposes the legacyTYPE_*string constantsmapping to the same string values, so external call sites keep working.
Audit::log()andAudit::logBatch()signaturesare byte-identical to main, and the
ClickHouseconstructor is unchanged.Deferred / known gaps
(a > A) OR (a = A AND b > B) …cascade in the adapter and feeds it throughwhereRaw()with hand-typed placeholders; the base builder does not yet emitClickHouse-flavoured tuple comparisons. Those raw bindings are merged with the
builder's
namedBindingsat the call site.whereRaw().Utopia\Query\Query::contains()is deprecated upstream in 0.3.3 in favour ofcontainsString()/containsAny().Utopia\Audit\Queryinherits it, socallers using
Query::contains()will see a deprecation notice. The libraryitself no longer calls it (main already moved
findByUserAndEvents()andfriends to
Query::equal()). Renaming the audit-side factory is out of scopefor this PR.
Test plan
Run on PHP 8.5.8. Docker is unavailable on the machine used for this sync,
so the ClickHouse/MariaDB-backed suite could not be executed — see the
unverified section below.
Verified:
composer validate—./composer.json is valid.composer updateresolves cleanly to stableutopia-php/query 0.3.3with no dev pins.
pint.json,perpreset) —{"tool":"pint","result":"passed"}over
srcandtests.[OK] No errors.src/is completely clean. Three findings remain,all in test files and all pre-existing on
main(
ClickHouseTest.php:630offsetAccess.invalidOffset, andAuditBase.php:818return.typein two class contexts). This branch alsoremoves two
argument.typefindings main has atAuditBase.php:251.phpunit --testsuite unit— 22 tests, 106 assertions, 0 failures(3 deprecations, all the upstream
Query::contains()notice above).ClickHouseSqlSnapshotTestis now part of theunitsuite since itneeds no server, and is excluded from
e2eto avoid running twice.stub, covering
setup()(both retention branches, shared-tables on/off),find()(multi-value equal, contains, notContains, DateTime comparison,select projection, orderRandom, cursorBefore direction flip),
count()with and without$max,getById(),cleanup()async, andcreateBatch(). The emittedCREATE TABLEcontains every column mainadded, with the correct
LowCardinality(Nullable(String))wrapping, all12 bloom-filter indexes including
_key_sdk,ORDER BY (tenant, time, id)and
allow_nullable_key = 1under shared tables. All of main's errormessages are reproduced verbatim.
Not verified
phpunit --testsuite e2edid not run — no Docker daemon available onthis machine, and no ClickHouse or MariaDB was reachable on any of the
compose ports. Unrun classes:
tests/Audit/Adapter/ClickHouseTest.phpandtests/Audit/Adapter/DatabaseTest.php. These needdocker compose up(ClickHouse 25.11 + MariaDB 10.11) before merge.Note on repository layout
mainhas since been absorbed intoutopia-php/monorepoand this repository isnow a mirror (
.github/workflows/mirror.yml,phpstan.neonincluding../../phpstan.neon, norequire-dev, nopint.json, noDockerfile). Thisbranch follows main's layout rather than restoring the removed files; the
tooling above was run with the monorepo's own
pint.jsonand PHPStan level.