Native BYOM worker exits 1 during clean systemd shutdown

[danny-avila]

Summary

A connected native BYOM worker using the Anthropic SRT process-isolated executor reports Native executor is unavailable and exits with status 1 when systemd stops or restarts it, even when no workspace command is executing. The worker starts again and reconnects normally, but an ordinary administrative shutdown is recorded as a service failure.

This reproduced twice on the current main source used by the worker.

Environment

• Code Interpreter commit: 118eb9b3ad704e752acc6ffa4b50fb9f1b3bb0ee
• Package: @librechat/code@0.1.0, built and installed from that commit
• OS: Ubuntu 24.04, x86_64
• Node.js: v22.23.2
• Command sandbox: Anthropic SRT with trusted-vm policy
• Service manager: systemd
• Unit properties relevant to shutdown:

[Service]
Type=simple
User=ubuntu
Group=ubuntu
WorkingDirectory=/home/ubuntu/src
Environment=HOME=/home/ubuntu
Environment=NODE_ENV=production
Environment=PATH=/opt/node/bin:/usr/local/bin:/usr/bin:/bin
ExecStart=/usr/local/bin/librechat-code run \
  --worker-dir /home/ubuntu/src \
  --workspace-id primary \
  --workspace-name "Skynet Projects" \
  --allow-workspace-writes \
  --allow-workspace-commands \
  --command-policy-preset trusted-vm
Restart=always
RestartSec=5s
TimeoutStopSec=35s
KillMode=control-group
UMask=0077

Reproduction

1. Start the unit and wait until the bridge reports the worker online and ready.
2. Ensure no workspace command is running.
3. Run sudo systemctl restart librechat-code-skynet.service or stop the unit.
4. Inspect systemctl status or the journal.

Observed twice:

systemd[1]: Stopping librechat-code-skynet.service - LibreChat BYOM worker for Danny Skynet...
librechat-code[15259]: librechat-code: Native executor is unavailable
systemd[1]: librechat-code-skynet.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: librechat-code-skynet.service: Failed with result 'exit-code'.
systemd[1]: Stopped librechat-code-skynet.service - LibreChat BYOM worker for Danny Skynet.
systemd[1]: Started librechat-code-skynet.service - LibreChat BYOM worker for Danny Skynet.

The same sequence occurred at 07:25:19 UTC and 07:31:25 UTC on 2026-09-13. After each restart, the worker reconnected and resumed normal operation.

Expected behavior

An idle worker receiving SIGTERM should:

1. stop leasing work,
2. close or reap its native executor,
3. stop relay resources,
4. exit with status 0 before TimeoutStopSec.

If a mutation is active or its outcome is ambiguous, existing fail-closed quarantine behavior must remain intact.

Actual behavior

The parent CLI prints Native executor is unavailable, exits 1, and makes a normal deploy or service restart appear unhealthy. The service can subsequently start and reconnect, so this is a shutdown lifecycle failure rather than a persistent startup failure.

Likely race to investigate

This is a hypothesis, not a confirmed root cause.

cli.ts installs SIGTERM handling by aborting the worker controller, then closes nativeCommandSandbox in finally. NativeProcessWorkspaceCommandSandbox.close() normally asks its forked child to close over IPC.

With KillMode=control-group, systemd sends SIGTERM to both the parent and the native executor child. The child handles SIGTERM by closing its SRT sandbox and calling process.exit(1). Its exit/disconnect events mark the parent-side sandbox as failed and can reject pending work with Native executor is unavailable. That ordering may race the parent's normal close RPC and leak the executor error through main().catch().

Relevant source:

• packages/code/src/cli.ts: SIGTERM abort and nativeCommandSandbox.close() in finally
• packages/code/src/native-process.ts: child loss marks the executor failed; close() and stop() lifecycle
• packages/code/src/native-process-child.ts: SIGTERM shutdown ends with process.exit(1)

Suggested acceptance coverage

• Simulate shutdown where the parent begins closing before the child exits.
• Simulate systemd-style ordering where the child receives SIGTERM or disconnects first.
• Assert an idle, fully prepared worker shutdown exits cleanly.
• Assert active commands are cancelled and reaped within a bound.
• Preserve quarantine and a non-zero exit for mutations whose completion or settlement is genuinely ambiguous.
• Ensure repeated stop/start cycles do not leave executor children behind or affect reconnect behavior.