Skip to content

feat: display Pi image artifacts inline - #36

Merged
sambitcreate merged 35 commits into
mainfrom
feature/pi-inline-image-artifacts
Aug 25, 2026
Merged

sambitcreate merged 35 commits into
mainfrom
feature/pi-inline-image-artifacts

Conversation

@sambitcreate

Copy link
Copy Markdown
Owner

Summary

  • add an extensible, chat-scoped Pi GUI artifact protocol and first-party display_image tool
  • render bounded PNG, JPEG, static GIF, static WebP, and BMP images inline with an accessible preview and fallback card
  • durably stage non-replayable image effects, recover interrupted responses after restart, and preserve image-only model continuity
  • gate the tool to attended workspace chats and document the extension contract

Safety

  • require workspace-relative paths and pin the authorized canonical root
  • reject descendant symlinks, path escapes, malformed or animated rasters, dangerous dimensions, and quota overflow
  • validate versioned artifact IPC before React state and never expose local paths or file URLs
  • block sends, copies, and exports while crash recovery remains pending

Validation

  • npm run type-check
  • focused ESLint across all changed TypeScript and TSX files
  • npm run test:display-image (62 passing)
  • npm run test:slash-commands (324 passing)
  • npm run test:onboarding (18 passing)
  • npm run test:compaction (184 passing)
  • npm run build
  • React Doctor changed scope: 84/100, only the pre-existing ChatPane state-count warning

Review

  • two independent subagent reviews completed and all actionable findings were fixed
  • 33 commits, each containing exactly one file as requested

@sambitcreate
sambitcreate marked this pull request as ready for review August 25, 2026 14:38
@sambitcreate
sambitcreate merged commit 6ebecd7 into main Aug 25, 2026
3 checks passed

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ No critical issues — two minor suggestions and one robustness note inline.

Reviewed changes

  • Artifact protocol + display_image tool — adds a versioned ChatArtifactV1 / ChatArtifactEventV1 contract and a chat-scoped PiAgentRuntimeExtension that exposes display_image only to attended foreground workspace chats.
  • Durable staging + crash recoveryDisplayImageArtifactStore stages normalized bytes before announcement, recovers interrupted responses into image-only interrupted assistant messages at startup, and blocks sends/copies/exports while recovery is pending.
  • Inline renderingMessageAttachments renders bounded PNG/JPEG/static GIF/WebP/BMP inline with a preview dialog and file-card fallback; live artifacts dedupe against persisted history.
  • Safety validation — workspace-relative path pinning, descendant-symlink rejection, raster structure/animation/dimension checks, quota enforcement, and versioned IPC parsed before React state.
  • Continuity & onboarding — empty assistant content is rewritten to an image-count placeholder for Pi history continuity; onboarding bento copy advertises the feature.

The hand-rolled GIF/PNG/WebP/JPEG/BMP header parsers and the symlink/identity checks in readPickedAttachments (O_NOFOLLOW + lstat walk) hold up under review; the pre-decode dimension checks are defense-in-depth over Chromium's actual decode with a renderer-side card fallback.

ℹ️ Startup image recovery has no failure path

displayImageArtifactStore.initialize() and recover() run inside app.whenReady() with no try/catch, so any throw — a corrupt/unsafe staging file, or recover()'s "would exceed chat limits" guard — aborts the ready handler and Aiden never finishes launching. initialize()'s corrupt-file throw matches the existing fail-fast pattern (piRuntimeEffectStore), but the recover() limit guard is a new, data-state failure mode whose only recovery is hand-editing display-image-artifacts.json.

Technical details
# Startup image recovery has no failure path

## Affected sites
- main/index.ts:1475-1507 — `initialize()` and `recover()` called with no guard; a throw propagates out of `whenReady`.
- main/services/display-image-artifact-store.ts:1003-1064 — `recover()` throws `Recovered image artifacts would exceed chat ... limits.` when persisted + pending usage exceeds per-chat caps.

## Required outcome
- A failed recovery/initialization must not prevent the app from launching. Log and degrade (leave the stage pending and keep the composer blocked for that chat), rather than throwing out of the ready handler.
- Optional: a test asserting startup tolerates a throwing `recover`.

## Open questions for the human
- Is the fail-fast-on-corrupt-store behavior intentional here, or should only the corrupt/unsafe case be fatal while the limit-exceeded case degrades?

Pullfrog  | Fix all ➔Fix 👍s ➔View workflow run | Using DeepSeek Pro𝕏


async pending(): Promise<readonly StagedDisplayImageArtifact[]> {
this.requireInitialized();
return structuredClone((await this.data.load()).records);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pending() deep-clones every staged artifact — including the base64 image payloads — on each call, and both hasPending() and usageByChat() route through it. hasPending() runs on every chats:get, append, copy, export, and delete, so while a recovery is pending (stages still present) each chat read clones up to megabytes of image bytes just to learn count > 0.

Technical details
# Avoid deep-cloning staged payloads in usage queries

## Affected sites
- main/services/display-image-artifact-store.ts:304 — `structuredClone((await this.data.load()).records)` clones `data` for every caller.
- main/services/display-image-artifact-store.ts:311-320 — `usageByChat` / `hasPending` call `pending()`.

## Required outcome
- A non-cloning usage projection that sums `size`, `count`, and `pixels` from the cached records (no base64 copy), used by `usageByChat`/`hasPending`.

@sambitcreate
sambitcreate deleted the feature/pi-inline-image-artifacts branch August 25, 2026 17:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant