Skip to content

fix(selfhost): make the Postgres connection pool size operator-tunable #2536

Description

@JSONbored

Parent: #1936

Problem

The self-host Postgres connection pool is created with no max option, so it silently uses the driver's default of 10 connections (src/server.ts). This single pool is shared by every HTTP request handler AND every queue worker's database operations, including fan-out jobs that intentionally run several database writes concurrently as part of one job. A realistic burst — a handful of concurrent job workers each running a fan-out step, plus normal HTTP/webhook traffic — can plausibly want more concurrent connections than the pool provides, well before Postgres's own connection ceiling or the existing Postgres-side alerting would trip. There is currently no environment variable to raise this without a code change.

Requirements

  • Expose the pool's max connection count as a documented environment variable with a sensible default (matching or close to the current implicit default, so this is a pure opt-in tuning knob, not a behavior change by default).
  • Document the tradeoff (higher pool size vs. Postgres's own max_connections, and interaction with the optional pgbouncer profile) in the self-hosting docs.
  • Add a note/warning path if the configured value would exceed a sane fraction of Postgres's max_connections, if that's cheaply detectable at boot.

Deliverables

  • A new env var (e.g. PGPOOL_MAX) wired into the pool construction in src/server.ts, with a fallback to the current default when unset.
  • .env.example entry with guidance on when/how to raise it.
  • A test covering that the env var is actually respected by the constructed pool.

Acceptance criteria

  • Setting the env var changes the pool size; leaving it unset preserves current behavior exactly.
  • Documentation clearly explains when an operator should raise this (registering many repos, high job concurrency) and what to weigh against.

Expected outcome

An operator running this at a larger scale than the reference deployment can tune database concurrency without patching the source.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.roadmapOn the Wave-2 agent-layer roadmap board (project 9)

Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions