Skip to content

fix(examples): manter o container para o erro de startup sobreviver (core#135, item 1) - #136

Merged
fabianocruz merged 2 commits into
mainfrom
fix/core135-keep-container-logs
Sep 6, 2026
Merged

fix(examples): manter o container para o erro de startup sobreviver (core#135, item 1)#136
fabianocruz merged 2 commits into
mainfrom
fix/core135-keep-container-logs

Conversation

@fabianocruz

Copy link
Copy Markdown
Member

Item 1 da core#135. Nao conserta os quatro jobs vermelhos — faz o proximo vermelho se explicar sozinho.

O que estava acontecendo

validate.sh subia o runtime com docker run -d --rm. Quando o container morre no startup, o Docker remove na hora, entao o docker logs do caminho de timeout roda em cima de um container que ja nao existe:

validate.sh: runtime did not become healthy in 30s
--- last 40 lines of container log ---
Error response from daemon: No such container: codespar-example-<demo>-<N>

O runtime imprime um diagnostico preciso no startup — nomeia o caminho que falhou, o uid sob o qual roda e o comando de conserto. Nada disso chegava a lugar nenhum. Quatro jobs ficaram tres dias sem explicacao por causa de uma flag.

E $RUNTIME_LOG nunca era impresso: no caminho de falha do docker run a mensagem manda "check $RUNTIME_LOG", que e um arquivo dentro de um runner efemero, e no caminho de timeout ele nem e mencionado.

Medicao, antes e depois

Mesma imagem, mesma falha de startup, so a flag mudando:

############ ANTES: docker run -d --rm ############
--- last 40 lines of container log ---
    Error response from daemon: No such container: cs-probe-old

############ DEPOIS: docker run -d ############
--- last 40 lines of container log ---
    Error: CODESPAR_PLUGINS: failed to import "/example/demo-plugin.mjs":
           Cannot find module '/example/demo-plugin.mjs' imported from
           /app/packages/core/dist/plugins/startup-loader.js
    Node.js v22.23.2
--- container state ---
    status=exited exit=1 oom=false

A causa passa de invisivel a impressa no log do job.

O que mudou, nos quatro scripts

Os quatro blocos eram identicos exceto pelo nome do container, e receberam a mesma alteracao (os quatro diffs tem hash identico):

  • docker run -d --rm vira docker run -d, com o motivo no comentario.
  • cleanup_docker passa a remover o container explicitamente, ja que o --rm saiu. O container sobrevive a propria morte so ate o dump de log; o trap EXIT INT TERM reapa depois.
  • docker rm -f defensivo antes do docker run, para um container orfao de uma execucao morta com SIGKILL nao colidir com o nome (que usa $$).
  • Caminho de timeout: alem do docker logs, agora imprime docker inspect (status, exit code, OOM) e o conteudo de $RUNTIME_LOG.
  • Caminho de falha do docker run: imprime o arquivo em vez de pedir para alguem abrir.

Ordem importa e esta certa: o dump acontece antes do exit 3, e so entao o trap remove o container.

Verificacao

bash -n limpo nos quatro. A prova antes/depois acima foi feita com a imagem real (ghcr.io/codespar/codespar:main) e uma falha real de startup.

Nao consegui rodar o validate.sh inteiro localmente: o macOS traz bash 3.2, que sob set -u engole o multibyte logo depois de $AIMOCK_PORT e aborta na linha 74. O CI roda ubuntu com bash 5, onde o script funciona ha meses. Entao o teste ponta a ponta desta mudanca e o proprio CI deste PR: os quatro jobs devem continuar vermelhos, e o log deles deve passar a mostrar o erro do runtime em vez de No such container. Comento aqui com o resultado.

Escopo

Mexi so nos quatro validate.sh dos jobs vermelhos, como acordado. Os outros tres exemplos (pix-nfse-skeleton, nfse-from-natural-language, whatsapp-installment-negotiation) tem exatamente o mesmo docker run -d --rm e a mesma armadilha; estao verdes hoje so porque usam a tag :latest. Ficam para uma decisao separada, registrada na core#135.

Os itens 2 e 3 da core#135 (dar ao container um diretorio onde escrever, e fazer CODESPAR_STATE_DIR valer no IdentityStore) continuam abertos com dono a definir.

fabianocruz and others added 2 commits September 6, 2026 19:36
…es (core#135)

The four meta-tool demo jobs have been red on main since 03/09 and nobody
could say why, because the harness destroys the answer.

validate.sh started the runtime with `docker run -d --rm`. When the container
dies during startup, Docker removes it immediately, so the `docker logs` on
the health-timeout path runs against a container that no longer exists and
prints "No such container" instead of the reason. The runtime's own startup
errors are precise — they name the failing path, the uid it runs as, and the
command to fix it — and all of it was being thrown away.

Measured, same image and same startup failure, only the flag differing:

  with --rm:  --- last 40 lines of container log ---
              Error response from daemon: No such container: cs-probe-old

  without:    --- last 40 lines of container log ---
              Error: CODESPAR_PLUGINS: failed to import ... Cannot find module
              ...
              Node.js v22.23.2

Since --rm is gone, cleanup_docker now removes the container explicitly, and
a stale container from a killed run is cleared before docker run. The timeout
path also prints `docker inspect` (status, exit code, OOM) and the contents of
$RUNTIME_LOG, and the docker-run-failed path prints that file instead of
telling a CI log reader to go open it.

This does not fix the four jobs. It makes the next red run explain itself.

Refs core#135.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ogress

The first CI run of this branch proved the change works and showed the
$RUNTIME_LOG dump is 60 lines of `docker pull` layer progress before
anything useful. Docker-level errors land at the end of that file, so
tail -n 20 keeps what the dump is for and drops the chatter.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@fabianocruz

Copy link
Copy Markdown
Member Author

Verificado no ambiente que importa: o CI deste PR

Como avisei no corpo, o macOS traz bash 3.2 e nao roda o validate.sh inteiro, entao o teste ponta a ponta era o proprio CI. Rodou. Os quatro jobs continuam vermelhos, como esperado — a causa nao foi consertada aqui — mas o log deles mudou de categoria.

Antes, no main:

validate.sh: runtime did not become healthy in 30s
--- last 40 lines of container log ---
Error response from daemon: No such container: codespar-example-payment-failure-triage-2581

Agora, neste PR:

validate.sh: runtime did not become healthy in 30s
--- last 40 lines of container log ---
  code<spar>  v0.5.0 (server)
  ─────────────────────────────

[server] Cannot write to the state directory: /example/.codespar
[server] EACCES: permission denied, mkdir '/example/.codespar'

[server] This usually means the directory was created by a different
[server] user than the one this process runs as. In Docker the process
[server] runs as `node` (uid 1000), and a volume written by an earlier
[server] release that ran as root stays owned by root across the upgrade.

[server] Fix it by giving the volume to the runtime user, once:
[server]   docker compose run --rm --user root core chown -R node:node /app/.codespar

[server] Or point the runtime somewhere it can write, with CODESPAR_STATE_DIR.

[server] DATABASE_URL is not set, so this directory IS the datastore. Stopping.
--- container state ---
status=exited exit=1 oom=false

Causa, uid, comando de conserto e exit code, no log do job, sem ninguem precisar investigar.

Isso fecha a inferencia que estava aberta na core#135

La eu separei o que estava medido do que era hipotese: "medi a classe do erro, nao o uid", porque no macOS o compartilhamento de arquivos do Docker aplica a permissao do host independente do uid do container e ate --user root recebia EACCES.

O CI respondeu. EACCES: permission denied, mkdir '/example/.codespar', com o runtime declarando que roda como node (uid 1000). A hipotese do uid esta confirmada no ambiente real, e a core#135 pode tratar isso como medido. Nao precisa mais da maquina Linux que eu tinha deixado como pendencia.

Um ajuste vindo do proprio run

O dump do $RUNTIME_LOG saiu com ~60 linhas de progresso de docker pull antes de qualquer coisa util. Trocado por tail -n 20, que preserva o que o dump existe para mostrar (erros de nivel Docker caem no fim do arquivo) e corta o ruido. Segundo commit deste PR.

@fabianocruz
fabianocruz merged commit d2b82e5 into main Sep 6, 2026
5 of 9 checks passed
fabianocruz added a commit that referenced this pull request Sep 6, 2026
…test (#137)

Preventive, not corrective. These three jobs are green today, and the reason
is contingency rather than structure: they pull ghcr.io/codespar/codespar:latest,
built 2026-05-31, which runs as root and predates the change that made the
runtime require a writable state directory. The four jobs pinned to :main
(built 2026-08-22, runs as `node` uid 1000) went red on 03/09 for exactly that
reason.

They carried the identical `docker run -d --rm`, so the day anyone moves them
to the current image they fail the same way AND with the same missing
diagnostic, and whoever picks it up rediscovers core#135 from scratch.

Same change already verified on the other four in #136: no --rm, explicit
removal in cleanup_docker, a defensive rm before docker run, and a timeout
path that prints docker inspect plus the tail of $RUNTIME_LOG. The docker
lines are now byte-identical across all seven validate.sh, and the old
pattern no longer appears anywhere in the repo.

Control for a preventive change is the inverse of #136's: the happy path must
stay happy, and for the right reason. All three stayed green with the
container actually starting — "runtime up after 2s" in each.

Refs core#135.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant