Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 13 additions & 11 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -49,19 +49,21 @@ MINIO_HOST_BIND=127.0.0.1
# MINIO_API_HOST_PORT=9000
# MINIO_CONSOLE_HOST_PORT=9001

# API (optional defaults; API_SHARED_SECRET required to accept ingest calls)
WEBHOOK_INGEST_HOST=0.0.0.0
# Host-run web/API processes launched through `./scripts/dev.sh web` ignore this
# `.env` value and compute a deterministic per-worktree port unless you export
# WEBHOOK_INGEST_PORT in your shell. compose.yaml pins the API container
# to 8090 internally; compose.local.yaml varies only the published host port.
WEBHOOK_INGEST_PORT=8090
WEBHOOK_INGEST_HOST_BIND=127.0.0.1
# Optional: expose ingest API on a fixed host port for local debugging.
# Web/API service (optional defaults; API_SHARED_SECRET required for protected API calls)
WEB_HOST=0.0.0.0
# Direct host-run web/API processes use WEB_PORT. Compose pins the container's
# internal listen port to 8090 and uses WEB_HOST_PORT for the published host port.
# `./scripts/dev.sh web` ignores this `.env` value and computes a deterministic
# per-worktree port unless you export WEB_PORT in your shell.
WEB_PORT=8090
WEB_HOST_BIND=127.0.0.1
# Optional: expose the web/API service on a fixed host port for Cloudflare Tunnel/local debugging.
# Leave unset to let ./scripts/docker-compose.sh compute a deterministic port.
# WEBHOOK_INGEST_HOST_PORT=8090
# WEB_HOST_PORT=8090
# Host-run web/API processes launched through `./scripts/dev.sh web` override
# `WEBHOOK_INGEST_PORT` with a deterministic per-worktree port automatically.
# `WEB_PORT` with a deterministic per-worktree port automatically.
# Deprecated fallback names still work: WEBHOOK_INGEST_HOST, WEBHOOK_INGEST_PORT,
# WEBHOOK_INGEST_HOST_BIND, WEBHOOK_INGEST_HOST_PORT.
# Required: ingest requests are rejected when unset
API_SHARED_SECRET=
# Authentik admin API (required for /create-sso-user and /create-user-accounts)
Expand Down
35 changes: 19 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,15 @@ This repository follows a service-oriented monorepo layout:
├── apps/
│ ├── discord_bot/ # Discord gateway process
│ │ └── src/five08/discord_bot/
│ ├── api/ # Backend API + dashboard service
│ ├── api/ # Backend API + dashboard code
│ │ └── src/five08/backend/
│ └── worker/ # Async queue worker
│ └── src/five08/worker/
├── packages/
│ └── shared/
│ └── src/five08/ # Shared settings, queue helpers, shared clients
├── compose.yaml # canonical Coolify/base container stack
├── compose.local.yaml # local host port publishing override
├── compose.local.yaml # local infra host port publishing override
├── docker-compose.yml # compatibility wrapper including compose.yaml
├── tests/ # Unit and integration tests
└── pyproject.toml # uv workspace root
Expand All @@ -28,7 +28,7 @@ This repository follows a service-oriented monorepo layout:
## Services

- `discord_bot`: Discord gateway process.
- `api`: FastAPI dashboard + ingest service that validates and enqueues jobs.
- `web`: FastAPI dashboard + ingest service that validates and enqueues jobs.
- `worker`: Dramatiq worker that executes jobs from Redis queue.
- `redis`: queue transport between API and worker.
- `postgres`: job state persistence, retries, idempotency.
Expand All @@ -37,7 +37,7 @@ This repository follows a service-oriented monorepo layout:
Migrations:

- `apps/worker/src/five08/worker/migrations` (Alembic)
- `api` runs `run_job_migrations()` during startup to keep DB schema current.
- `web` runs `run_job_migrations()` during startup to keep DB schema current.

### Job model

Expand Down Expand Up @@ -150,10 +150,12 @@ For local full-container runs, including deterministic localhost ports:

Coolify should use `/compose.yaml` as the base Compose file. A small
`docker-compose.yml` compatibility wrapper includes it for tools still configured
to read the older filename. The base file intentionally
does not publish Redis, Postgres, MinIO, or API host ports; Coolify should expose
only the services/domains it manages. The app services also attach to the
shared infra network named by `INFRA_DOCKER_NETWORK` so they can reach
to read the older filename. The `web` service publishes container port `8090`
to `${WEB_HOST_BIND:-127.0.0.1}:${WEB_HOST_PORT:-8090}`
so a host-side Cloudflare Tunnel can target the dashboard/API at localhost.
The base file does not publish Redis, Postgres, or MinIO host ports. The app
services also attach to the shared infra network named by `INFRA_DOCKER_NETWORK`
so they can reach
Portainer-managed Bifrost and Langfuse by Docker DNS. The network is declared
as external, so pre-create it before running Compose if it does not already
exist.
Expand Down Expand Up @@ -236,13 +238,14 @@ Use `.env.example` as the source of truth for defaults.
- Note: `MINIO_ACCESS_KEY` / `MINIO_SECRET_KEY` are `SharedSettings` alias properties (`minio_access_key`, `minio_secret_key`) and are not env-loaded fields.
- Note: use `MINIO_ROOT_USER` and `MINIO_ROOT_PASSWORD` as the actual env vars.

### Backend API Ingest
### Web/API Service

- `Required` for non-dashboard protected endpoints: `API_SHARED_SECRET` (ingest requests are rejected when unset)
- `Optional`: `WEBHOOK_INGEST_HOST` (default: `0.0.0.0`)
- `Optional`: `WEBHOOK_INGEST_HOST_BIND` (default: `127.0.0.1`; Compose host bind for local exposure)
- `Optional`: `WEBHOOK_INGEST_PORT` (host-run `./scripts/dev.sh` ignores `.env` for this key and defaults to a deterministic per-worktree value near `18080 + WORKTREE_ENV_SLOT`; export it in your shell only when you intentionally want a fixed port, and avoid browser-unsafe ports such as `5060`)
- `Optional`: `WEBHOOK_INGEST_HOST_PORT` (default: `8090` when running `docker compose` directly; `./scripts/docker-compose.sh` computes a deterministic per-worktree value when unset, and pinned values must avoid browser-unsafe ports such as `5060`; see `./scripts/docker-compose.sh print-ports`)
- `Required` for non-dashboard protected endpoints: `API_SHARED_SECRET` (protected API requests are rejected when unset)
- `Optional`: `WEB_HOST` (default: `0.0.0.0`; direct process bind host, while Compose pins the container bind host to `0.0.0.0`)
- `Optional`: `WEB_HOST_BIND` (default: `127.0.0.1`; Compose host bind for Cloudflare Tunnel/local exposure)
- `Optional`: `WEB_PORT` (direct process listen port; Compose pins the container's internal listen port to `8090`; host-run `./scripts/dev.sh` ignores `.env` for this key and defaults to a deterministic per-worktree value near `18080 + WORKTREE_ENV_SLOT`)
- `Optional`: `WEB_HOST_PORT` (published host port for Docker/Cloudflare Tunnel; default `8090` when running `docker compose` directly; `./scripts/docker-compose.sh` computes a deterministic per-worktree value when unset, and pinned values must avoid browser-unsafe ports such as `5060`; see `./scripts/docker-compose.sh print-ports`)
- Deprecated fallback names still work for now: `WEBHOOK_INGEST_HOST`, `WEBHOOK_INGEST_PORT`, `WEBHOOK_INGEST_HOST_BIND`, `WEBHOOK_INGEST_HOST_PORT`.

### Backend API OIDC Session Auth

Expand Down Expand Up @@ -312,7 +315,7 @@ Use `.env.example` as the source of truth for defaults.
### Discord Bot Core

- `Required`: `DISCORD_BOT_TOKEN`
- `Optional`: `BACKEND_API_BASE_URL` (default: `http://127.0.0.1:8090`; `./scripts/dev.sh` overrides it to the worktree web/API port, Compose injects `http://api:8090`)
- `Optional`: `BACKEND_API_BASE_URL` (default: `http://127.0.0.1:8090`; `./scripts/dev.sh` overrides it to the worktree web/API port, Compose injects `http://web:8090`)
- `Optional`: `HEALTHCHECK_PORT` (host-run `./scripts/dev.sh` ignores `.env` for this key and defaults to a deterministic per-worktree value near `30000 + WORKTREE_ENV_SLOT`; export it in your shell only when you intentionally want a fixed port, and avoid browser-unsafe ports such as `5060`)
- Note: bot message chunking uses Discord's 2000 character limit in code.

Expand Down Expand Up @@ -415,4 +418,4 @@ Deploy as a single Compose application.
MinIO is used as the internal transfer mechanism so file handoffs stay inside the stack.
External object storage adapters can be added later for multi-cloud or vendor-specific routing.

This keeps one stack and one shared env set while still allowing independent service scaling/restarts (`discord_bot`, `api`, `worker`).
This keeps one stack and one shared env set while still allowing independent service scaling/restarts (`discord_bot`, `web`, `worker`).
4 changes: 2 additions & 2 deletions apps/api/src/five08/backend/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -3900,8 +3900,8 @@ def run() -> None:
)
uvicorn.run(
create_app(),
host=settings.webhook_ingest_host,
port=settings.webhook_ingest_port,
host=settings.web_host,
port=settings.web_port,
log_level=settings.log_level.lower(),
)

Expand Down
4 changes: 2 additions & 2 deletions apps/worker/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Defaults:

- Base URL:
- Outside Docker: `http://localhost:8090`
- Inside Docker: `http://api:8090`
- Inside Docker: `http://web:8090`
- Override: `$WORKER_API_BASE_URL`
- API secret: `$API_SHARED_SECRET` (sent as `X-API-Secret`)
- Timeout: `10.0` seconds
Expand Down Expand Up @@ -87,7 +87,7 @@ DiscordWebhookLogger(
"color": 15158332,
"fields": [
{"name": "Environment", "value": "production", "inline": True},
{"name": "Service", "value": "api", "inline": True},
{"name": "Service", "value": "web", "inline": True},
],
}
],
Expand Down
2 changes: 1 addition & 1 deletion apps/worker/src/five08/jobcli.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
from five08.queue import JobStatus

DEFAULT_API_URL = "http://localhost:8090"
DEFAULT_DOCKER_API_URL = "http://api:8090"
DEFAULT_DOCKER_API_URL = "http://web:8090"
DEFAULT_TIMEOUT_SECONDS = 10.0
API_SECRET_ENV_VAR = "API_SHARED_SECRET"
JOB_STATUSES = [status.value for status in JobStatus]
Expand Down
4 changes: 0 additions & 4 deletions compose.local.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,3 @@ services:
ports:
- "${MINIO_HOST_BIND:-127.0.0.1}:${MINIO_API_HOST_PORT:-9000}:9000"
- "${MINIO_HOST_BIND:-127.0.0.1}:${MINIO_CONSOLE_HOST_PORT:-9001}:9001"

api:
ports:
- "${WEBHOOK_INGEST_HOST_BIND:-127.0.0.1}:${WEBHOOK_INGEST_HOST_PORT:-8090}:8090"
16 changes: 10 additions & 6 deletions compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ services:
environment:
REDIS_URL: redis://redis:6379/0
REDIS_QUEUE_NAME: ${REDIS_QUEUE_NAME:-jobs.default}
BACKEND_API_BASE_URL: http://api:8090
BACKEND_API_BASE_URL: http://web:8090
restart: unless-stopped
networks:
- default
Expand All @@ -77,7 +77,7 @@ services:
redis:
condition: service_healthy

api:
web:
build:
context: .
dockerfile: apps/api/Dockerfile
Expand All @@ -98,8 +98,12 @@ services:
MINIO_ROOT_USER: ${MINIO_ROOT_USER:-internal}
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:-change-me}
MINIO_INTERNAL_BUCKET: ${MINIO_INTERNAL_BUCKET:-internal-transfers}
WEBHOOK_INGEST_HOST: 0.0.0.0
WEBHOOK_INGEST_PORT: 8090
# Keep the container's internal listen port fixed; vary only the published
# host port for tunnels/local multi-worktree runs.
WEB_HOST: 0.0.0.0
WEB_PORT: 8090
ports:
- "${WEB_HOST_BIND:-${WEBHOOK_INGEST_HOST_BIND:-127.0.0.1}}:${WEB_HOST_PORT:-${WEBHOOK_INGEST_HOST_PORT:-8090}}:8090"
restart: unless-stopped
networks:
- default
Expand Down Expand Up @@ -127,7 +131,7 @@ services:
environment:
REDIS_URL: redis://redis:6379/0
REDIS_QUEUE_NAME: ${REDIS_QUEUE_NAME:-jobs.default}
WORKER_API_BASE_URL: http://api:8090
WORKER_API_BASE_URL: http://web:8090
WORKER_QUEUE_NAMES: ${WORKER_QUEUE_NAMES:-jobs.default}
POSTGRES_USER: ${POSTGRES_USER:-postgres}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-postgres}
Expand All @@ -146,7 +150,7 @@ services:
- default
- infra
depends_on:
api:
web:
condition: service_started
redis:
condition: service_healthy
Expand Down
18 changes: 14 additions & 4 deletions packages/shared/src/five08/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
import os
import sys

from pydantic import field_validator, model_validator
from pydantic import AliasChoices, Field, field_validator, model_validator
from pydantic_settings import BaseSettings, SettingsConfigDict


Expand Down Expand Up @@ -43,8 +43,14 @@ class SharedSettings(BaseSettings):
minio_root_password: str = ""
minio_internal_bucket: str = "internal-transfers"

webhook_ingest_host: str = "0.0.0.0"
webhook_ingest_port: int = 8090
web_host: str = Field(
default="0.0.0.0",
validation_alias=AliasChoices("WEB_HOST", "WEBHOOK_INGEST_HOST"),
)
web_port: int = Field(
default=8090,
validation_alias=AliasChoices("WEB_PORT", "WEBHOOK_INGEST_PORT"),
)
api_shared_secret: str | None = None
discord_logs_webhook_url: str | None = None
discord_logs_webhook_wait: bool = True
Expand All @@ -68,7 +74,11 @@ class SharedSettings(BaseSettings):
outline_api_key: str | None = None
outline_api_timeout_seconds: float = 20.0

model_config = SettingsConfigDict(env_file=".env", extra="ignore")
model_config = SettingsConfigDict(
env_file=".env",
extra="ignore",
populate_by_name=True,
)

@field_validator("docuseal_member_agreement_template_id", mode="before")
@classmethod
Expand Down
14 changes: 7 additions & 7 deletions scripts/dev.sh
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ worktree_env_load "$script_dir"
export REDIS_URL="redis://127.0.0.1:${REDIS_HOST_PORT}/0"
export POSTGRES_URL="postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@127.0.0.1:${POSTGRES_HOST_PORT}/${POSTGRES_DB}"
export MINIO_ENDPOINT="http://127.0.0.1:${MINIO_API_HOST_PORT}"
export BACKEND_API_BASE_URL="http://127.0.0.1:${WEBHOOK_INGEST_PORT}"
export WORKER_API_BASE_URL="http://127.0.0.1:${WEBHOOK_INGEST_PORT}"
export BACKEND_API_BASE_URL="http://127.0.0.1:${WEB_PORT}"
export WORKER_API_BASE_URL="http://127.0.0.1:${WEB_PORT}"
export DISCORD_BOT_INTERNAL_BASE_URL="http://127.0.0.1:${HEALTHCHECK_PORT}"

shell_quote() {
Expand Down Expand Up @@ -42,7 +42,7 @@ Infrastructure is running in Docker on localhost:
Console: 127.0.0.1:${MINIO_CONSOLE_HOST_PORT}

Host-run app ports for this worktree:
Web/API: 127.0.0.1:${WEBHOOK_INGEST_PORT} (hot reload)
Web/API: 127.0.0.1:${WEB_PORT} (hot reload)
Bot: 127.0.0.1:${HEALTHCHECK_PORT}

Run app services on the host with:
Expand All @@ -65,7 +65,7 @@ REDIS_HOST_PORT=$REDIS_HOST_PORT
POSTGRES_HOST_PORT=$POSTGRES_HOST_PORT
MINIO_API_HOST_PORT=$MINIO_API_HOST_PORT
MINIO_CONSOLE_HOST_PORT=$MINIO_CONSOLE_HOST_PORT
WEBHOOK_INGEST_PORT=$WEBHOOK_INGEST_PORT
WEB_PORT=$WEB_PORT
HEALTHCHECK_PORT=$HEALTHCHECK_PORT
EOF
;;
Expand All @@ -76,7 +76,7 @@ EOF
emit_export POSTGRES_DB "$POSTGRES_DB"
emit_export MINIO_API_HOST_PORT "$MINIO_API_HOST_PORT"
emit_export MINIO_CONSOLE_HOST_PORT "$MINIO_CONSOLE_HOST_PORT"
emit_export WEBHOOK_INGEST_PORT "$WEBHOOK_INGEST_PORT"
emit_export WEB_PORT "$WEB_PORT"
emit_export HEALTHCHECK_PORT "$HEALTHCHECK_PORT"
emit_export REDIS_URL "$REDIS_URL"
emit_export MINIO_ENDPOINT "$MINIO_ENDPOINT"
Expand All @@ -91,8 +91,8 @@ EOF
web|api)
exec uv run --package api uvicorn five08.backend.api:create_app \
--factory \
--host "${WEBHOOK_INGEST_HOST:-0.0.0.0}" \
--port "$WEBHOOK_INGEST_PORT" \
--host "${WEB_HOST:-${WEBHOOK_INGEST_HOST:-0.0.0.0}}" \
--port "$WEB_PORT" \
--reload \
--reload-dir apps/api/src \
--reload-dir apps/worker/src \
Expand Down
4 changes: 2 additions & 2 deletions scripts/dev_mux.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,9 @@ def _service_commands(env: dict[str, str]) -> list[tuple[str, list[str]]]:
"five08.backend.api:create_app",
"--factory",
"--host",
env.get("WEBHOOK_INGEST_HOST", "0.0.0.0"),
env.get("WEB_HOST", env.get("WEBHOOK_INGEST_HOST", "0.0.0.0")),
"--port",
env["WEBHOOK_INGEST_PORT"],
env.get("WEB_PORT", env.get("WEBHOOK_INGEST_PORT", "8090")),
"--reload",
"--reload-dir",
"apps/api/src",
Expand Down
6 changes: 4 additions & 2 deletions scripts/docker-compose.sh
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,14 @@ repo_root=$WORKTREE_ENV_REPO_ROOT
export COMPOSE_PROJECT_NAME
export REDIS_HOST_PORT
export POSTGRES_HOST_PORT
export WEB_HOST_PORT
export WEBHOOK_INGEST_HOST_PORT
export MINIO_API_HOST_PORT
export MINIO_CONSOLE_HOST_PORT

# Host-run-only app ports must not leak into Compose interpolation, or the API
# Host-run-only app ports must not leak into Compose interpolation, or the web
# container can start on a high worktree port while peers still target :8090.
unset WEB_PORT
unset WEBHOOK_INGEST_PORT
unset HEALTHCHECK_PORT

Expand All @@ -24,7 +26,7 @@ WORKTREE=$repo_root
COMPOSE_PROJECT_NAME=$COMPOSE_PROJECT_NAME
REDIS_HOST_PORT=$REDIS_HOST_PORT
POSTGRES_HOST_PORT=$POSTGRES_HOST_PORT
WEBHOOK_INGEST_HOST_PORT=$WEBHOOK_INGEST_HOST_PORT
WEB_HOST_PORT=$WEB_HOST_PORT
MINIO_API_HOST_PORT=$MINIO_API_HOST_PORT
MINIO_CONSOLE_HOST_PORT=$MINIO_CONSOLE_HOST_PORT
EOF
Expand Down
Loading