bug: GitHub-template agent deploy — race intermittently leaves a silent empty agent (clone into non-empty /home/developer, exit 128, not surfaced)

[webmixgamer]

Summary

Creating an agent from a GitHub template (create_agent(template: "github:<owner>/<repo>"))
intermittently produces a silent empty agent: the container comes up running and
get_agent_health returns healthy, but the workspace was never populated — no .git,
no CLAUDE.md, no template.yaml, no .claude/skills/. Two linked defects: a race
condition in the clone step (root cause) and a silent-failure path that never
surfaces it (compounding).

Context

docker/base-image/startup.sh (git-sync clone path) empties the agent home and then
clones into that same directory:

• ~line 66: rm -rf /home/developer/* /home/developer/.[!.]* (empty the home dir)
• ~line 78: git clone <url> /home/developer (clone into it)

A concurrent process populates /home/developer in the window between the rm and git's
empty-directory precondition check, so the clone intermittently aborts:

fatal: destination path '/home/developer' already exists and is not an empty directory.
Exit code: 128

Evidence it is a race (not deterministic, not a stale image, not a PAT issue):

• Flaky on identical inputs: 1 failure in 3 runs against the same base image + same repo.
• Timing-flip: a base image identical except for a passive ls inserted right before the
  clone succeeded every time — a read-only op that changes only timing can only affect a race.
• Never fails in isolation: running startup.sh under docker run (no entrypoint/backend
  concurrency) always empties cleanly and clones OK.

Leading suspect for the concurrent writer: the runtime's asynchronous named-volume population
(copying the base image's sizeable /home/developer into the fresh volume) still completing as
the entrypoint runs. The fix below is robust regardless of the exact writer.

Silent-failure path (compounding): on clone failure, startup.sh writes
/home/developer/.git-clone-status and continues to "Agent ready" (around lines 137–155) —
no abort. The backend never reads .git-clone-status, and health/monitoring never check
whether the identity clone succeeded. Net: status: running, aggregate_status: healthy, empty
workspace. Because the clone failure is flaky, GitHub deploys occasionally yield a healthy-looking,
identity-less agent that is hard to notice and non-reproducible.

Related prior work (same "silent empty agent" family, different paths): #218
(clone-not-executed), #950 (deploy_local silently creates empty agents).

Steps to Reproduce

1. On an instance whose global GitHub PAT can read a repo containing a valid template.yaml
   • .claude/skills/ (private is fine), call
     create_agent(name, template: "github:<owner>/<repo>", source_branch: "main").
2. Repeat several times. Some runs fail: get_agent_logs shows the
   exit 128 … 'already exists and is not an empty directory' block; the workspace has no
   .git/CLAUDE.md; get_agent_health still returns healthy. Other runs succeed.
   (~1 in 3 failed during investigation.)

Acceptance Criteria

• GitHub-template deploy is not sensitive to clone-vs-writer timing — repeated deploys
  never leave an empty workspace (clone no longer runs into a non-empty /home/developer).
• A genuine clone failure is terminal and surfaced: the agent reports failed/unhealthy
  (not running/healthy), and the git error is visible via get_agent / get_agent_health
  (backend consumes .git-clone-status).
• Regression coverage that a GitHub-template deploy reliably populates the workspace
  (.git, template.yaml, .claude/skills/).

Technical Notes / Suggested Fix

1. (fixes the race) Never git clone into the live home dir. Either clone into a temp dir and
   move contents into /home/developer, or initialize in place:
   git init && git remote add origin <url> && git fetch origin <branch> && git checkout -f <branch>.
   Eliminates the empty-dir precondition, so a concurrent writer is harmless.
2. (fixes the silent failure) Make identity-clone failure terminal — surface the git error
   through get_agent/get_agent_health; have the backend consume .git-clone-status.
3. (minor, separate follow-up) base_image_version in the create response is set from
   get_platform_version() (src/backend/services/agent_service/crud.py, ~line 788), not the
   actual base-image label — so agents report the platform version. Cosmetic/observability;
   caused a red-herring "0.7.0 vs 0.9.0" confusion during investigation.

Verified on dev @ abc8e5cd, base image rebuilt from dev.

[webmixgamer]

/claim

[github-actions]

Claimed by @webmixgamer. Label status-in-progress added.

Next steps:

1. Create branch: feature/1439-<short-slug>
2. Implement the feature
3. Open a PR with Fixes #1439