Skip to content
BranchesTags

Latest commit

 

History

13 Commits

Folders and files

NameName
Last commit message
Last commit date
examples/v1beta2
examples/v1beta2
 
 
AGENTS.md
AGENTS.md
 
 
LICENSE
LICENSE
 
 
README.md
README.md
 
 
README_SQL_TESTS.md
README_SQL_TESTS.md
 
 

Repository files navigation

Coalesce Quality CLI (synqcli)

Command-line tool for managing data quality tests, monitors and deployment rules on Coalesce Quality.

AGENTS.md, shipped beside the binary, is the operating guide — the deploy loop, what a reconcile deletes, how to confirm which workspace you are pointed at, and what each command costs. It is written for a coding agent driving the tool, and is also published at docs.synq.io/monitors/agent-workflow. This README covers what the tool is and how a config is structured.

Features

  • Deploy - Deploy data quality tests, monitors and deployment rules from YAML configuration files
  • Advisor - Get AI-powered suggestions for data quality tests based on your schema
  • Export - Export existing monitors, tests and deployment rules to YAML format

Reference documentation:

Installation

macOS — Homebrew

brew install getsynq/tap/synqcli

brew upgrade synqcli from then on. Homebrew owns the binary once it installs it, so synqcli upgrade will point you back here rather than replace it.

macOS and Linux — release archive

The archive filename carries the version, so the version has to be resolved first. GitHub redirects /releases/latest to the newest release's tag, which needs no API token and no login:

VERSION=$(curl -fsSLI -o /dev/null -w '%{url_effective}' \
  https://github.com/getsynq/synqcli/releases/latest | sed 's#.*/v##')
OS=$(uname -s | tr '[:upper:]' '[:lower:]')          # darwin or linux
ARCH=$(uname -m | sed 's/x86_64/amd64/; s/aarch64/arm64/')

curl -fL "https://github.com/getsynq/synqcli/releases/download/v${VERSION}/synqcli_${VERSION}_${OS}_${ARCH}.tar.gz" \
  | tar -xz
sudo mv synqcli /usr/local/bin/

To pin a version instead, set VERSION by hand from the releases page.

Builds are published for macOS and Linux on both amd64 and arm64, and every release ships a checksums.txt (sha256sum -c checksums.txt --ignore-missing).

Windows

Download synqcli_<version>_windows_amd64.zip (or _arm64) from the releases page and extract synqcli.exe to your PATH.

Upgrading

synqcli upgrade --check      # what it would do, without doing it
synqcli upgrade

upgrade resolves the latest release, downloads the archive for this platform, verifies it against the release's checksums.txt, and runs the new binary once to prove it works on this machine before replacing anything. If the binary lives somewhere you cannot write — /usr/local/bin usually is not — it says so and changes nothing; re-run it with sudo. A binary installed by a package manager is left to that package manager — a Homebrew install is upgraded with brew upgrade synqcli, which upgrade will tell you.

synqcli also mentions a newer release on stderr, at most once a day. That check reads a tag from a public GitHub URL and sends nothing but the tool name and version — no credentials, no workspace, no identity. It never delays the command it runs beside and never reports its own failure, so a machine with no route to the internet behaves exactly like one that is up to date. It is already silent in CI, when output is not a terminal, and inside a container or a Kubernetes pod. To switch it off everywhere:

export QUALITY_NO_UPDATE_CHECK=1     # DO_NOT_TRACK=1 has the same effect

Configuration

API Credentials

For interactive use, log in through the browser once:

synqcli auth login             # EU, the default deployment
synqcli auth login --region us # or au
synqcli auth status

The credential is cached under ~/.synq/oauth/ and shared with the other Coalesce Quality CLIs, so later commands need no flag — the login records which deployment it authenticated against.

For CI, set client credentials via environment variables:

export QUALITY_CLIENT_ID="your-client-id"
export QUALITY_CLIENT_SECRET="your-client-secret"
export QUALITY_REGION="eu"     # eu (default), us, or au

Or create a .env file in your project root:

QUALITY_CLIENT_ID=your-client-id
QUALITY_CLIENT_SECRET=your-client-secret
QUALITY_REGION=eu

Or use command-line flags (highest priority):

synqcli deploy --client-id="your-id" --client-secret="your-secret" --region=eu

Pass --endpoint (or QUALITY_API_ENDPOINT) instead of --region to reach a staging or self-hosted deployment.

Priority order: client credentials (flags > environment variables > .env) > a pre-issued QUALITY_TOKEN > the cached browser login. The CI paths deliberately win, so adding a browser login cannot change what an existing pipeline authenticates as.

Advisor Credentials

For the advisor command, you need an OpenAI-compatible API key or AWS Bedrock credentials:

OpenAI (default):

export OPENAI_API_KEY="your-api-key"

Custom endpoint (LiteLLM, Azure, etc.):

export OPENAI_API_KEY="your-api-key"
export OPENAI_BASE_URL="https://your-endpoint.com/v1"

AWS Bedrock (direct):

To use Claude models hosted on AWS Bedrock directly:

# Set the Bedrock model ID (required for Bedrock)
export AWS_BEDROCK_MODEL_ID="anthropic.claude-sonnet-4-20250514-v1:0"

# Set the AWS region (optional, defaults to us-east-1)
export AWS_REGION="us-east-1"

# AWS credentials are loaded from the standard AWS credential chain:
# - Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN)
# - Shared credentials file (~/.aws/credentials)
# - IAM roles (when running on AWS infrastructure)

Example usage:

AWS_BEDROCK_MODEL_ID="anthropic.claude-sonnet-4-20250514-v1:0" \
AWS_REGION="us-east-1" \
  synqcli advisor \
  --entity-id "postgres::public::users" \
  --instructions "Suggest data quality tests"

Available Bedrock Claude models:

  • anthropic.claude-sonnet-4-20250514-v1:0 (Claude Sonnet 4)
  • anthropic.claude-3-5-sonnet-20241022-v2:0 (Claude 3.5 Sonnet v2)
  • anthropic.claude-3-5-sonnet-20240620-v1:0 (Claude 3.5 Sonnet)
  • anthropic.claude-3-haiku-20240307-v1:0 (Claude 3 Haiku)

DWH Connection (Optional)

For enhanced test suggestions with data profiling, configure a database connection:

Via environment variables:

export DWH_TYPE="postgres"           # postgres, mysql, bigquery, snowflake, clickhouse, redshift, databricks
export DWH_HOST="localhost"
export DWH_PORT="5432"
export DWH_DATABASE="mydb"
export DWH_USERNAME="user"
export DWH_PASSWORD="pass"

Via connections file:

# connections.yaml
- id: my-postgres
  type: postgres
  host: localhost
  port: 5432
  database: mydb
  username: user
  password: pass

Snowflake Configuration

Snowflake supports multiple authentication methods:

Password authentication:

# connections.yaml
- id: my-snowflake
  type: snowflake
  account: myaccount.us-east-1     # Account identifier (with region if needed)
  warehouse: COMPUTE_WH
  role: ANALYST
  username: myuser
  password: mypassword
  databases: ["PROD", "DEV"]       # Optional: limit to specific databases
  use_get_ddl: true                # Optional: use GET_DDL for view definitions

Private key authentication:

# connections.yaml
- id: my-snowflake-key
  type: snowflake
  account: myaccount.us-east-1
  warehouse: COMPUTE_WH
  role: ANALYST
  username: myuser
  private_key_file: /path/to/rsa_key.p8
  private_key_passphrase: optional-passphrase  # If key is encrypted
  databases: ["PROD"]

SSO/Browser authentication (externalbrowser):

For organizations using SSO (Okta, Azure AD, etc.), use browser-based authentication:

# connections.yaml
- id: my-snowflake-sso
  type: snowflake
  account: myaccount.us-east-1
  warehouse: COMPUTE_WH
  role: ANALYST
  username: myuser@company.com     # Your SSO username/email
  auth_type: externalbrowser       # Triggers browser-based SSO
  databases: ["PROD"]

Or via environment variables:

export DWH_TYPE="snowflake"
export DWH_ACCOUNT="myaccount.us-east-1"
export DWH_WAREHOUSE="COMPUTE_WH"
export DWH_ROLE="ANALYST"
export DWH_USERNAME="myuser@company.com"
export DWH_AUTH_TYPE="externalbrowser"

How SSO authentication works:

  1. First connection opens your default browser for SSO login
  2. After successful login, the ID token is cached in your OS credential manager:
    • macOS: Keychain
    • Windows: Credential Manager
    • Linux: File-based (requires explicit opt-in)
  3. Subsequent connections reuse the cached token (valid for ~4 hours)
  4. When token expires, browser opens again for re-authentication

Requirements for SSO:

  • Your Snowflake account must have ID token caching enabled: 
    ALTER ACCOUNT SET ALLOW_ID_TOKEN = TRUE;
  • Your organization's IdP must be configured in Snowflake

Example usage with SSO:

synqcli advisor \
  --entity-id "snowflake::PROD::ANALYTICS::ORDERS" \
  --instructions "Suggest data quality tests" \
  --connections ./connections.yaml

Commands

Deploy

Deploy data quality tests and monitors from YAML configuration files.

synqcli deploy [FILES...] [flags]

How It Works

  1. File Discovery - If no files specified, discovers all .yaml files in current directory
  2. Parse - Parses YAML files and converts to API format
  3. Resolve - Resolves the short entity ids in the YAML to full asset paths
  4. Preview - Shows configuration changes and delta (creates, updates, deletes)
  5. Confirm - Asks for confirmation (unless --auto-confirm is used)
  6. Deploy - Applies the configuration changes

Examples

# Deploy specific files
synqcli deploy tests.yaml monitors.yaml

# Deploy all YAML files in current directory
synqcli deploy

# Deploy all YAML files recursively
synqcli deploy **/*.yaml

# Preview changes without deploying (dry run)
synqcli deploy --dry-run

# Deploy with auto-confirmation (for CI/CD)
synqcli deploy --auto-confirm

# Deploy only specific namespaces
synqcli deploy --namespace=data-team-pipeline

# Deploy with debug output
synqcli deploy -p  # prints protobuf messages in JSON format

Flags

synqcli deploy --help, or the CLI reference.

Advisor

Get AI-powered suggestions for data quality tests based on your table schema.

synqcli advisor [flags]

How It Works

  1. Fetch Context - Retrieves table schema, existing checks, and code from Coalesce Quality
  2. Profile Data (optional) - If DWH connection is configured, profiles columns to discover actual values, min/max bounds, and null rates
  3. Analyze - AI analyzes the schema (and profiling results) to generate appropriate test suggestions
  4. Output - Returns JSON (default) or writes YAML files to specified directory
  5. Deploy - Optionally deploys generated tests immediately

Examples

# Get suggestions for a single entity (outputs JSON)
synqcli advisor \
  --entity-id "postgres::public::users" \
  --instructions "Suggest basic data quality tests"

# Generate YAML files for multiple entities
synqcli advisor \
  --entity-id "postgres::public::users" \
  --entity-id "postgres::public::orders" \
  --entity-id "postgres::public::products" \
  --instructions "Suggest comprehensive tests for e-commerce tables" \
  --output ./generated-tests

# Use instructions from a file
synqcli advisor \
  --entity-id "snowflake::analytics::customers" \
  --instructions-file ./test-instructions.txt \
  --output ./tests

# Generate and deploy in one step
synqcli advisor \
  --entity-id "bigquery::dataset::events" \
  --instructions "Suggest freshness and volume monitors" \
  --output ./tests \
  --deploy \
  --auto-confirm

# Customize namespace and severity
synqcli advisor \
  --entity-id "postgres::public::transactions" \
  --instructions "Suggest tests for financial data" \
  --output ./tests \
  --namespace "finance-team" \
  --severity "ERROR"

# Force overwrite existing files
synqcli advisor \
  --entity-id "postgres::public::users" \
  --instructions "Suggest tests" \
  --output ./tests \
  --force

# With DWH connection for data profiling (discovers actual values)
synqcli advisor \
  --entity-id "postgres::public::users" \
  --instructions "Suggest accepted_values tests for enum-like columns" \
  --connections ./connections.yaml \
  --output ./tests

# DWH connection via environment variables
DWH_TYPE=postgres DWH_HOST=localhost DWH_DATABASE=mydb \
  synqcli advisor \
  --entity-id "postgres::public::users" \
  --instructions "Suggest min/max tests based on actual data ranges"

# Verbose mode to see AI reasoning and tool calls
synqcli advisor \
  --entity-id "postgres::public::users" \
  --instructions "Suggest tests" \
  --connections ./connections.yaml \
  --verbose

# Filter suggestions to specific columns (comma-separated)
synqcli advisor \
  --entity-id "postgres::public::users" \
  --columns "status,email,role" \
  --instructions "Suggest accepted_values tests for these columns"

# Filter to specific columns (multiple flags)
synqcli advisor \
  --entity-id "postgres::public::orders" \
  --columns status \
  --columns priority \
  --columns region \
  --instructions "Suggest tests for these enum-like columns" \
  --output ./tests

Flags

synqcli advisor --help, or the CLI reference.

Export

Export existing monitors to YAML format.

synqcli export [flags] <output-file>

Examples

# Export all app-created monitors
synqcli export --namespace=exported-monitors output.yaml

# Export monitors for a specific table
synqcli export \
  --namespace=orders-monitors \
  --monitored="bq-prod.dataset.orders" \
  output.yaml

# Export monitors from multiple tables
synqcli export \
  --namespace=sales-monitors \
  --monitored="bq-prod.dataset.orders" \
  --monitored="bq-prod.dataset.customers" \
  output.yaml

# Export all monitors (including API-created)
synqcli export --namespace=all-monitors --source=all output.yaml

# Export monitors from a specific integration
synqcli export \
  --namespace=dbt-monitors \
  --integration="dbt-cloud-prod" \
  output.yaml

Selective export by resource type

By default export writes all three resource types (custom monitors, SQL tests, deployment rules). Use --type (repeatable) to narrow, or pass an ID-scoped flag and the type is inferred automatically.

# Only SQL tests
synqcli export --type=sql-tests generated/tests.yaml

# SQL tests + deployment rules, no custom monitors
synqcli export --type=sql-tests --type=deployment-rules generated/tests_and_rules.yaml

# One specific test (auto-narrows to --type=sql-tests)
synqcli export --sql-test=<test-uuid> generated/one_test.yaml

# All monitors plus one specific test (union — explicit --type widens, doesn't restrict)
synqcli export --type=monitors --sql-test=<test-uuid> generated/mix.yaml

Query-based deployment rules are authoring-only and are never exported; export writes the single-asset rules only.

Flags

synqcli export --help, or the CLI reference.

YAML Configuration Format

synqcli uses the v1beta2 YAML format for defining tests and monitors.

Basic Structure

version: v1beta2
namespace: my-project

defaults:
  severity: WARNING

entities:
  - id: postgres::public::users
    tests:
      - type: not_null
        description: Ensure critical user identifiers are always present
        columns: [user_id, email]
    monitors:
      - type: automated
        metrics: [ROW_COUNT, DELAY]

Complete Example

# yaml-language-server: $schema=https://schemas.synq.io/synq-monitors/v1/config.schema.json
version: v1beta2
namespace: data-team-pipeline

defaults:
  severity: ERROR
  schedule:
    type: daily
    query_delay: 2h
  mode:
    anomaly_engine:
      sensitivity: BALANCED

entities:
  - id: bq-prod.dataset.orders
    time_partitioning_column: created_at

    tests:
      # Ensure critical columns are never null
      - type: not_null
        description: Order ID, customer ID and total amount are required for all orders
        columns:
          - order_id
          - customer_id
          - total_amount

      # Ensure order_id is unique
      - type: unique
        description: Each order must have a unique identifier
        columns: [order_id]

      # Validate status values
      - type: accepted_values
        description: Order status must be one of the valid workflow states
        column: status
        values: [pending, processing, shipped, delivered, cancelled]

      # Ensure amounts are positive
      - type: min_value
        description: Order amounts cannot be negative
        column: total_amount
        min_value: 0

      # Business rule: ship_date must be after order_date
      - type: relative_time
        description: Ship date must be on or after order date
        column: ship_date
        relative_column: order_date

    monitors:
      # Automated monitoring for volume, freshness, and delays
      - type: automated
        metrics: [ROW_COUNT, DELAY, VOLUME_CHANGE_DELAY]
        severity: ERROR
        sensitivity: BALANCED

      # Volume monitoring segmented by region
      - id: orders_by_region
        type: volume
        segmentation:
          expression: region
        filter: "region IN ('US', 'EU', 'APAC')"

      # Field statistics monitoring
      - id: order_stats
        type: field_stats
        columns:
          - total_amount
          - discount_amount

  - id: bq-prod.dataset.customers
    tests:
      - type: not_null
        description: Customer ID and email are required for all customers
        columns: [customer_id, email]

      - type: unique
        description: Email addresses must be unique across all customers
        columns: [email]

      - type: business_rule
        description: Updated timestamp must be on or after creation timestamp
        sql_expression: "created_at <= updated_at"

Schema Reference

Reference the JSON schema in your YAML files for IDE autocompletion and validation:

# yaml-language-server: $schema=https://schemas.synq.io/synq-monitors/v1/config.schema.json
version: v1beta2

The published schema always describes the current release. To pin the schema to the CLI version you deploy with, write it out and reference the local file instead:

synqcli schema > schema.json
# yaml-language-server: $schema=./schema.json
version: v1beta2

A rendered, browsable version of the same schema is at https://schemas.synq.io/synq-monitors/v1/config.html.

Query-based deployment rules

A monitor under an entities[].id targets a single asset. To cover many assets by a rule instead of listing each one, author query-based deployment rules at the top level with a ResolverQL selection string — the same selection the app and the API expose. New assets that match are covered automatically, with no YAML edit.

# yaml-language-server: $schema=https://schemas.synq.io/synq-monitors/v1/config.schema.json
version: v1beta2
namespace: "data-team-pipeline"

# Inclusion: deploy monitors to every matching asset (full config).
deployment_rules:
  - name: snowflake tables row-count and delay
    type: table_stats
    resolver_ql: with_type("table", filter=with_platform("snowflake"))
    severity: ERROR
    sensitivity: RELAXED
    metrics:
      - ROW_COUNT
      - DELAY

# Exclusion: carve matching assets OUT of coverage. Only a selection — no
# metrics/severity/sensitivity (those describe how to monitor, not what to skip).
deployment_exclusions:
  - name: exclude staging tables
    type: table_stats
    resolver_ql: with_type("table", filter=with_tag("staging"))

Notes:

  • name is required on every rule and exclusion; it labels the rule in the deploy preview. It does not affect rule identity (that is derived from the selection), so renaming a rule does not create a duplicate.
  • resolver_ql is the only selection form supported here. The string is forwarded verbatim; the backend compiles and validates it, so an invalid query fails at deploy time.
  • A query rule authored here and the same resolver_ql authored via the API resolve to the same rule, so the two paths converge rather than creating duplicates.
  • Query rules are authoring-only: export does not emit them (it writes single-asset entities rules). A full example is examples/v1beta2/query_deployment_rules.yaml.
  • The deploy preview (before confirm, and under --dry-run) shows each query rule's downstream effect — how many monitors it will create / delete / change, plus skipped assets. The asset lists are capped; pass --verbose to list every affected asset.

SQL Tests Reference

SQL tests are data quality validation rules that run SQL queries to check your data. README_SQL_TESTS.md covers the parts this summary leaves out: business_query evaluators, save_failures, and exactly which edits reset a test.

Test Types

not_null

Ensures specified columns do not contain null values.

- type: not_null
  description: Critical user fields must always have values
  columns:
    - user_id
    - email
    - created_at

empty

Ensures specified columns are not empty strings.

- type: empty
  description: Description and notes should contain meaningful content when present
  columns:
    - description
    - notes

unique

Ensures column values are unique, optionally within a time window.

# Simple unique check
- type: unique
  description: Order ID must be unique across all orders
  columns: [order_id]

# Composite unique key
- type: unique
  description: Customer can only have one order per day
  columns:
    - customer_id
    - order_date

# Unique within time window (e.g., last 30 days)
- type: unique
  description: Transaction IDs must be unique within rolling 30-day window
  columns: [transaction_id]
  time_partition_column: created_at
  time_window_seconds: 2592000  # 30 days

accepted_values

Ensures column values are within a predefined list of acceptable values.

# String values
- type: accepted_values
  description: Account status must be a valid lifecycle state
  column: status
  values:
    - active
    - inactive
    - pending

# Numeric values
- type: accepted_values
  description: Priority must be between 1 (highest) and 5 (lowest)
  column: priority
  values: [1, 2, 3, 4, 5]

rejected_values

Ensures column values are NOT in a predefined list of blocked values.

- type: rejected_values
  description: Error codes must not contain placeholder or invalid values
  column: error_code
  values:
    - -1
    - 0
    - 999

min_value

Ensures column values are greater than or equal to a minimum value.

# Numeric minimum
- type: min_value
  description: Users must be at least 18 years old
  column: age
  min_value: 18

# Strict comparison (greater than, not equal)
- type: min_value
  description: Quantity must be positive (greater than zero)
  column: quantity
  min_value: 0
  strictly: true

# Date minimum
- type: min_value
  description: Start date must be in 2024 or later
  column: start_date
  min_value: "2024-01-01"

max_value

Ensures column values are less than or equal to a maximum value.

# Numeric maximum
- type: max_value
  description: Product price cannot exceed maximum allowed price
  column: price
  max_value: 1000.99

# Use SQL expression (e.g., no future dates)
- type: max_value
  description: Created timestamp cannot be in the future
  column: created_at
  max_value:
    type: expression
    value: NOW()
  strictly: true

min_max

Ensures column values fall within a specified range.

# Numeric range
- type: min_max
  description: Percentage values must be between 0 and 100
  column: percentage
  min_value: 0
  max_value: 100

# Date range
- type: min_max
  description: Event dates must fall within the 2024 calendar year
  column: event_date
  min_value: "2024-01-01"
  max_value: "2024-12-31"

# Temperature range
- type: min_max
  description: Temperature readings must be within valid sensor range
  column: temperature
  min_value: -40
  max_value: 120

freshness

Ensures data is updated within a specified time window.

- type: freshness
  description: Table should be updated at least every 2 hours
  time_partition_column: updated_at
  time_window_seconds: 7200  # 2 hours

relative_time

Ensures temporal relationships between columns (e.g., end_date >= start_date).

- type: relative_time
  description: Ship date must be on or after order date
  column: ship_date
  relative_column: order_date

- type: relative_time
  description: End time must be after start time
  column: end_time
  relative_column: start_time

business_rule

Validates custom SQL expressions that represent business logic. The expression should return TRUE for invalid rows.

# Accounting equation must balance
- type: business_rule
  description: Assets must equal liabilities plus equity (accounting equation)
  sql_expression: "assets = liabilities + equity"

# Discount cannot exceed total
- type: business_rule
  description: Discount amount cannot exceed order total
  sql_expression: "discount_amount <= total_amount"

# Complex validation
- type: business_rule
  description: Shipped orders must have a ship date
  sql_expression: "status = 'shipped' AND ship_date IS NOT NULL OR status != 'shipped'"

Monitors Reference

Monitors continuously track metrics and detect anomalies in your data.

Monitor Types

automated

The simplest way to monitor table health. Tracks volume, freshness, and change delays automatically.

- type: automated
  severity: ERROR
  sensitivity: BALANCED
  metrics:
    - ROW_COUNT          # Monitor row count changes
    - DELAY              # Monitor data freshness
    - VOLUME_CHANGE_DELAY # Monitor when data typically changes

volume

Monitors row count with optional segmentation and filtering.

# Basic volume monitoring
- type: volume

# Volume with segmentation (creates separate time series per segment)
- id: orders_by_region
  type: volume
  segmentation:
    expression: region
    include_values:
      - US
      - EU

# Volume with filter
- id: high_value_orders
  type: volume
  filter: "total_amount > 1000"

freshness

Monitors data freshness based on a timestamp column.

- id: orders_freshness
  type: freshness
  expression: created_at

field_stats

Monitors column-level statistics including null rates, distinct values, and min/max values.

- id: customer_stats
  type: field_stats
  columns:
    - email
    - status
    - created_at
  mode:
    anomaly_engine:
      sensitivity: BALANCED

custom_numeric

Monitors custom SQL aggregations.

# Monitor active user count
- id: active_users
  type: custom_numeric
  metric_aggregation: "COUNT(DISTINCT user_id)"
  mode:
    fixed_thresholds:
      min: 100
      max: 100000

# Monitor average order value
- id: avg_order_value
  type: custom_numeric
  metric_aggregation: "AVG(total_amount)"
  mode:
    anomaly_engine:
      sensitivity: HIGH

# Monitor with segmentation
- id: revenue_by_country
  type: custom_numeric
  metric_aggregation: "SUM(revenue)"
  segmentation:
    expression: country

Monitor Options

Severity

severity: INFO | WARNING | ERROR

Schedule

# Daily schedule
schedule:
  type: daily
  query_delay: 2h  # Wait 2 hours after midnight before running

# Hourly schedule
schedule:
  type: hourly
  query_delay: 15m

Time segmentation

time_partitioning_column splits the asset into time segments (one data point per day or hour) and is set on the monitor, the entity, or in defaults:

entities:
  - id: bq-prod.dataset.orders
    time_partitioning_column: created_at
    monitors:
      - type: volume
        id: orders_volume

Omit it to monitor the asset without time segmentation — the metric is computed over the whole asset once per run, and there is no historical backfill on the first run:

entities:
  - id: bq-prod.dataset.reference_data
    monitors:
      - type: volume
        id: reference_data_row_count

time_partitioning_interval (ondemand schedules) requires a time_partitioning_column.

Without time segmentation there are no segments to skip, so ignore_last has no effect.

Mode

# Anomaly detection
mode:
  anomaly_engine:
    sensitivity: LOW | BALANCED | HIGH

# Fixed thresholds
mode:
  fixed_thresholds:
    min: 0
    max: 1000

Categories (category, governance_category)

A monitor can declare its own categories. category is the technical dimension — what kind of check this is mechanically, e.g. volume or freshness. governance_category is what the check is for, the data quality dimension governance reports on, e.g. timeliness. Both are free-form strings — use whatever vocabulary your categorisation rules already use — and they resolve independently, so a monitor may set either, both, or neither:

entities:
  - id: bq-prod.dataset.orders
    time_partitioning_column: created_at
    monitors:
      - type: volume
        id: orders_volume
        category: volume
        governance_category: timeliness

      - type: freshness
        id: orders_freshness
        expression: created_at
        category: freshness
        # governance category left to the categorisation rules

What a monitor declares here takes precedence over your workspace's categorisation rules for that monitor. Leave a category out and the rules decide it as before; delete one from the file and the next deploy hands that dimension back to the rules. Values are shown with underscores as spaces, so snake_case reads well.

A segmented monitor's segments take the monitor's categories. A segment is the same monitor sliced by a column value, so it cannot declare categories of its own.

There is deliberately no defaults: entry for either field. A default would categorise every monitor in the file, and because a declared category outranks the rules, that would switch the rules off for all of them rather than fill a gap.

Changing a category does not reset a monitor's learned baseline.

Test Lifecycle

UUID Generation

Tests and monitors use deterministic UUID generation based on their configuration:

  • Same configuration = Same UUID: Redeploying with identical configuration updates the existing test
  • Changed configuration = New UUID: Changing critical fields creates a new test

Test Reset Behavior

Tests are reset (re-triggered) when these fields change:

  • Schedule/recurrence
  • Severity
  • Test type
  • Template configuration (columns, values, expressions, etc.)

Tests are NOT reset when only metadata changes (name, description).

Production Deployment

The recommended workflow for production environments separates test generation from deployment:

  1. Local Development: Use advisor to generate YAML files locally
  2. Code Review: Commit and review generated files via pull request
  3. Automated Deployment: CI/CD automatically deploys on merge to main

Recommended Project Structure

my-data-project/
├── data-quality/
│   ├── orders.yaml           # Tests for orders table
│   ├── customers.yaml        # Tests for customers table
│   └── products.yaml         # Tests for products table
├── .github/
│   └── workflows/
│       └── deploy-data-quality.yml
└── README.md

Local Workflow

# 1. Generate tests using advisor
synqcli advisor \
  --entity-id "bq-prod.dataset.orders" \
  --entity-id "bq-prod.dataset.customers" \
  --instructions "Suggest comprehensive data quality tests" \
  --output ./data-quality \
  --namespace "production-tests"

# 2. Review generated files
cat data-quality/*.yaml

# 3. Make any manual adjustments if needed
# Edit files as necessary

# 4. Commit and push
git add data-quality/
git commit -m "Add data quality tests for orders and customers"
git push origin feature/add-dq-tests

# 5. Create PR for review
# After approval and merge, CI/CD handles deployment

GitHub Actions Workflow

Create .github/workflows/deploy-data-quality.yml:

name: Deploy Data Quality Tests

on:
  push:
    branches: [main]
    paths:
      - 'data-quality/**/*.yaml'
  pull_request:
    branches: [main]
    paths:
      - 'data-quality/**/*.yaml'

jobs:
  validate:
    name: Validate Configuration
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install synqcli
        run: |
          # The archive filename carries the version, so resolve it first. Set
          # VERSION to a literal instead to pin the pipeline to a known release.
          VERSION=$(curl -fsSLI -o /dev/null -w '%{url_effective}' \
            https://github.com/getsynq/synqcli/releases/latest | sed 's#.*/v##')
          curl -fL "https://github.com/getsynq/synqcli/releases/download/v${VERSION}/synqcli_${VERSION}_linux_amd64.tar.gz" | tar -xz
          sudo mv synqcli /usr/local/bin/
          synqcli --version

      - name: Validate YAML files
        env:
          QUALITY_CLIENT_ID: ${{ secrets.QUALITY_CLIENT_ID }}
          QUALITY_CLIENT_SECRET: ${{ secrets.QUALITY_CLIENT_SECRET }}
          QUALITY_API_ENDPOINT: https://developer.synq.io
        run: |
          synqcli deploy data-quality/**/*.yaml --dry-run

  deploy:
    name: Deploy to Coalesce Quality
    runs-on: ubuntu-latest
    needs: validate
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    steps:
      - uses: actions/checkout@v4

      - name: Install synqcli
        run: |
          # The archive filename carries the version, so resolve it first. Set
          # VERSION to a literal instead to pin the pipeline to a known release.
          VERSION=$(curl -fsSLI -o /dev/null -w '%{url_effective}' \
            https://github.com/getsynq/synqcli/releases/latest | sed 's#.*/v##')
          curl -fL "https://github.com/getsynq/synqcli/releases/download/v${VERSION}/synqcli_${VERSION}_linux_amd64.tar.gz" | tar -xz
          sudo mv synqcli /usr/local/bin/
          synqcli --version

      - name: Deploy tests and monitors
        env:
          QUALITY_CLIENT_ID: ${{ secrets.QUALITY_CLIENT_ID }}
          QUALITY_CLIENT_SECRET: ${{ secrets.QUALITY_CLIENT_SECRET }}
          QUALITY_API_ENDPOINT: https://developer.synq.io
        run: |
          synqcli deploy data-quality/**/*.yaml --auto-confirm

This workflow:

  • On Pull Request: Validates YAML files with --dry-run (no actual deployment)
  • On Merge to Main: Deploys tests and monitors to Coalesce Quality

GitLab CI

Create .gitlab-ci.yml:

stages:
  - validate
  - deploy

validate-data-quality:
  stage: validate
  image: alpine:latest
  script:
    - apk add --no-cache curl
    # The archive filename carries the version, so resolve it first. Set VERSION
    # to a literal instead to pin the pipeline to a known release.
    - VERSION=$(curl -fsSLI -o /dev/null -w '%{url_effective}' https://github.com/getsynq/synqcli/releases/latest | sed 's#.*/v##')
    - curl -fL "https://github.com/getsynq/synqcli/releases/download/v${VERSION}/synqcli_${VERSION}_linux_amd64.tar.gz" | tar -xz
    - mv synqcli /usr/local/bin/
    - synqcli --version
    - synqcli deploy data-quality/**/*.yaml --dry-run
  variables:
    QUALITY_CLIENT_ID: $QUALITY_CLIENT_ID
    QUALITY_CLIENT_SECRET: $QUALITY_CLIENT_SECRET
    QUALITY_API_ENDPOINT: https://developer.synq.io
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      changes:
        - data-quality/**/*.yaml

deploy-data-quality:
  stage: deploy
  image: alpine:latest
  script:
    - apk add --no-cache curl
    # The archive filename carries the version, so resolve it first. Set VERSION
    # to a literal instead to pin the pipeline to a known release.
    - VERSION=$(curl -fsSLI -o /dev/null -w '%{url_effective}' https://github.com/getsynq/synqcli/releases/latest | sed 's#.*/v##')
    - curl -fL "https://github.com/getsynq/synqcli/releases/download/v${VERSION}/synqcli_${VERSION}_linux_amd64.tar.gz" | tar -xz
    - mv synqcli /usr/local/bin/
    - synqcli --version
    - synqcli deploy data-quality/**/*.yaml --auto-confirm
  variables:
    QUALITY_CLIENT_ID: $QUALITY_CLIENT_ID
    QUALITY_CLIENT_SECRET: $QUALITY_CLIENT_SECRET
    QUALITY_API_ENDPOINT: https://developer.synq.io
  rules:
    - if: $CI_COMMIT_BRANCH == "main"
      changes:
        - data-quality/**/*.yaml

Required Secrets

Add these secrets to your CI/CD environment:

Secret Description
QUALITY_CLIENT_ID Your Coalesce Quality API client ID
QUALITY_CLIENT_SECRET Your Coalesce Quality API client secret

For GitHub: Settings → Secrets and variables → Actions → New repository secret

For GitLab: Settings → CI/CD → Variables

Troubleshooting

Common Issues

Authentication errors:

Error: failed to connect to Coalesce Quality API: authentication failed
  • Verify QUALITY_CLIENT_ID and QUALITY_CLIENT_SECRET are correct
  • Check you're using the correct QUALITY_API_ENDPOINT for your region

Entity not found:

Error: failed to resolve entity: postgres::public::users
  • Verify the entity ID matches exactly what's shown in the app
  • Check the entity exists and is synced to Coalesce Quality

Invalid YAML:

Error: failed to parse YAML: ...
  • Validate your YAML syntax
  • Reference the JSON schema for field names and types

Debug Mode

Use -p flag to print detailed protobuf messages:

synqcli deploy tests.yaml -p

Support

Every Coalesce Quality customer has a shared Slack channel with a Technical Account Manager. Ask there for anything — getting a configuration deployed, a platform you want supported, or something that looks wrong. Support has the details, and docs.synq.io covers the rest of the platform.

About

Coalesce Quality CLI for monitors, tests and deployment rules: release downloads and documentation

docs.synq.io/monitors/cli

Resources

Readme
Apache-2.0 license
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Used by

Contributors