Skip to content

docs(selfhost): document measured resource profiles and trim image bloat - #3198

Merged
JSONbored merged 2 commits into
mainfrom
selfhost-resource-profile-matrix-1828
Jul 4, 2026
Merged

docs(selfhost): document measured resource profiles and trim image bloat#3198
JSONbored merged 2 commits into
mainfrom
selfhost-resource-profile-matrix-1828

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

  • Advances perf(selfhost): profile stack resources and tune defaults #1828 (profile stack resources and tune defaults). Adds a Resource profiles section to the self-host operations docs with a measured-vs-estimated CPU/memory matrix per docker-compose.yml profile, plus a disk breakdown, sourced from real docker stats / docker system df snapshots on a production instance running the full profile set (qdrant + redis + observability + backup + postgres + ollama).
  • Audits every deploy.resources.limits.memory default against that measured data and documents the conclusion: none look miscalibrated enough to change right now (closest is Grafana at ~60% of its 512MiB limit, which still has real headroom). This is a no-op on docker-compose.yml limits, made explicit in the docs rather than guessed at.
  • Trims the self-host runtime image by 188MB (942MB → 754MB, measured before/after with docker build/docker images): npm install -g for the bundled Claude Code / Codex CLIs was leaving ~180MB of npm download cache (~/.npm/_cacache) baked into the image with zero runtime use; npm cache clean --force after the install removes it. Also excludes the unused apps/ (gittensory-ui) and test/ directories from the Docker build context via .dockerignore — the self-host bundle's only entry point is src/server.ts and npm ci only ever sees the root package*.json (copied before the rest of the tree), so neither directory is ever read during the image build.

What this PR does NOT do (remaining #1828 scope)

  • No compose default changes — the audit concluded none were justified by the data in hand; this is stated explicitly in the docs rather than left implicit.
  • --profile runners still ships with no CPU/memory limit (documented as a known, deliberate gap — see docker-compose.override.yml.example — rather than something this PR fixes, since the right ceiling is host-specific).
  • No load-testing or synthetic benchmarking of Redis maxmemory policy, Prometheus/Loki/Tempo retention, or queue concurrency defaults under stress — the issue's "Review Redis maxmemory policy, Prometheus/Loki/Tempo retention..." requirement is not addressed here; this PR is scoped to the resource-profile/image-size half of the issue using data that was already available.
  • No dashboard changes for inspecting resource pressure (the issue's "Dashboard or docs updates showing how to inspect resource pressure" deliverable) — Grafana/Prometheus dashboards already exist per the current operations docs; no new panel was added in this PR.

Measured vs. estimated, explicitly

  • Measured (real production docker stats/docker system df snapshot, full profile set, steady state): app 2.6% CPU / 365MiB of 2GiB; postgres 14.24% CPU / 196MiB of 2GiB (highest CPU); grafana 305MiB of 512MiB (~60%); tempo 209MiB of 1GiB (~20%); disk 48GB/151GB (32%) used, with docker system df Images/Volumes/Build-cache reclaimable percentages as reported.
  • Estimated (reasoned from the measured baseline + each service's declared limit/image size, not measured directly): minimal (app+redis only) profile footprint, Ollama under active inference, and the --profile runners risk — each is called out in the docs table as estimated, with the reasoning shown.
  • Image-bloat measurement: built the image locally (docker build --target runtime) before and after the fix, confirmed the size delta and inspected /home/node/.npm/_cacache (180M) and /home/node/.npm-global (465M) inside the built container directly.

Scope

  • The PR title follows type(scope): short summary Conventional Commit format, for example fix(api): restore profile access checks.
  • This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes. (Docs + Dockerfile/.dockerignore are both part of the same perf(selfhost): profile stack resources and tune defaults #1828 profiling/tuning work.)
  • This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
  • I linked an issue (perf(selfhost): profile stack resources and tune defaults #1828); see "What this PR does NOT do" above for exactly what remains of that issue's scope.

Validation

  • git diff --check — ran, clean.
  • npm run actionlint — not run; no .github/workflows/** changes in this PR.
  • npm run typecheck — clean.
  • npm run test:coverage locally — not run; this PR has zero src/** changes (docs + Dockerfile + .dockerignore only), so there is no Codecov patch obligation.
  • npm run test:workers — not run for the same reason (no src/**/worker changes).
  • npm run build:mcp / npm run test:mcp-pack — not run; no MCP package changes.
  • npm run ui:openapi:check — not run; no API/OpenAPI schema changes.
  • npm run ui:lint — clean (0 errors; only the same pre-existing react-refresh/only-export-components warning every other route file in this directory already has).
  • npm run ui:typecheck — clean.
  • npm run ui:build — clean, docs.self-hosting-operations bundle built successfully.
  • npm audit --audit-level=moderate — not run for this docs/Dockerfile-only change; no dependency changes.
  • New or changed behavior has unit/integration tests for new branches, fallback paths, and sanitizer boundaries — N/A, no src/** logic changed.

Additionally validated outside the standard checklist (Docker image change, not covered by the JS test suite):

  • Built the runtime image locally with docker build --target runtime before and after the Dockerfile change; confirmed the final image shrank from 942MB to 754MB.
  • Ran the built image's claude --version and codex --version inside the container to confirm the AI CLIs still work correctly after npm cache clean --force.
  • Ran a full docker build --no-cache after the .dockerignore change to confirm npm ci and scripts/build-selfhost.mjs still succeed with apps/ and test/ excluded from the build context.
  • Booted the built image standalone and confirmed it fails on the expected self-host preflight validation (missing SELFHOST_SETUP_TOKEN/PUBLIC_API_ORIGIN), i.e. the bundle executes correctly end-to-end.

If any required check was skipped, explain why:

  • This PR touches zero files under src/** (only apps/gittensory-ui/src/routes/** docs, Dockerfile, and .dockerignore), so the Codecov-gated JS checks (test:coverage, test:workers, build:mcp, test:mcp-pack, ui:openapi:check, npm audit) don't apply — there is no changed src/** line for them to cover, and no dependency or OpenAPI change to check. actionlint doesn't apply because no workflow files changed.

Safety

  • No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed. (The production measurements included are aggregate resource numbers only — no repo names, tokens, or identifying data.)
  • Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
  • Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests. — N/A, no such changes.
  • API/OpenAPI/MCP behavior is updated and tested where needed. — N/A, no such changes.
  • UI changes use live API data or real empty/error/loading states, not production mock/demo fallbacks. (This is a static docs page; no data fetching involved.)
  • Visible UI changes include a UI Evidence section below.
  • Public docs/changelogs are updated where needed; changelogs are only edited for release-prep PRs. (CHANGELOG.md untouched.)

UI Evidence

This is a docs-only content addition to an existing, unstyled prose page (new <h2>/<h3>/<table> content using the same DocsPage/FeatureRow primitives and border-hairline/divide-hairline design tokens already used elsewhere on this exact page) — no new components, layout, or visual design were introduced.

A full-page screenshot of the new section deep in the page could not be captured in this sandboxed environment: the preview tool's screenshot capture reproducibly returns a blank/black image at any non-zero scroll position on this route, and this reproduces identically on an unmodified sibling docs page (self-hosting-backup-scaling), confirming it is a pre-existing environment/tooling limitation and not caused by this change.

What was verified instead:

  • The rendered SSR HTML (fetched directly from a local dev server running this exact branch) contains the full new section, correctly structured, with the table and all rows rendering as expected HTML (<table class="w-full border-collapse ..."> etc.).
  • The page's accessibility-tree snapshot and the right-hand "on this page" table of contents both show the new "Resource profiles" entry with its "Disk" and "When a compose default might need to change" sub-sections correctly nested, confirming the heading hierarchy renders correctly.
  • npm run ui:build succeeds and produces the docs.self-hosting-operations route bundle with the new content included.
State / title Evidence
Top of page, loaded state (own local dev server, this branch) Verified visually via preview tool at page top; TOC sidebar shows the new "Resource profiles" entry with nested "Disk" / "When a compose default might need to change" sub-items — screenshot capture below scroll position 0 is blocked by the environment limitation described above, not by this change.

Notes

  • The npm-cache and .dockerignore findings were discovered by actually building the image locally (docker build --target runtime -t ... .) and inspecting the resulting layers with docker history --no-trunc and docker run ... du -sh, rather than guessed — see the Validation section above for the exact before/after numbers.
  • A sibling PR (docs(selfhost): align release upgrade/rollback docs with deploy scripts #3191, already merged to main) covers the update/rollback documentation gap called out in this issue's parent thread; this PR does not touch that file (docs.self-hosting-releases.tsx) and there is no overlap.

JSONbored added 2 commits July 4, 2026 12:34
…oat (#1828)

Add a measured-vs-estimated resource profile matrix (CPU/memory per compose
profile, plus disk) to the self-host operations docs, sourced from a real
production instance running the full profile set. Audit docker-compose.yml's
memory limits against that data: none look miscalibrated enough to change.

Also trim the runtime image by ~188MB (942MB -> 754MB, measured): the AI CLI
install left ~180MB of npm download cache baked into the image with zero
runtime use, and the Docker build context carried the unused gittensory-ui
app and test suite (never read by the self-host bundle's entry point).
The prior commit's edit accidentally dropped the "Docker resource hygiene"
h2 and its lead paragraph while inserting the new resource-profile matrix
ahead of it, leaving the section's install/prune instructions orphaned
under no heading. Restore the heading and intro paragraph in place.
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
gittensory-ui 09e7d7b Commit Preview URL

Branch Preview URL
Jul 04 2026, 07:37 PM

@JSONbored JSONbored self-assigned this Jul 4, 2026
@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 4, 2026
@loopover-orb

loopover-orb Bot commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Important

🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪🟪

🔍 Gittensory is reviewing…

AI analysis is in progress. This comment will update when the review is complete.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed · 🟪 Reviewing

@JSONbored
JSONbored merged commit c283373 into main Jul 4, 2026
11 checks passed
@JSONbored
JSONbored deleted the selfhost-resource-profile-matrix-1828 branch July 4, 2026 19:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Development

Successfully merging this pull request may close these issues.

1 participant