feat(aio): Add prompt config editing UI - #75423
Conversation
|
😎 Merged manually by @jurajmajerik - details. |
🤖 CI report
|
| File | Size | Δ vs base |
|---|---|---|
posthog-app/_parent/products/ai_observability/frontend/prompts/LLMPromptScene.js |
45.3 KiB | 🔺 +3.2 KiB (+7.7%) |
render-query/src/render-query/render-query.js |
21.26 MiB | 🔺 +1.4 KiB (+0.0%) |
Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report
✅ Eager graph — within budget
How much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy import() / React.lazy chunks are not counted.
| Root | Eager (shipped) | Δ vs base | Budget |
|---|---|---|---|
entry (logged-out pages, app bootstrap)src/index.tsx |
1.24 MiB · 22 files | no change | ███░░░░░░░ 27.6% of 4.51 MiB |
authenticated shell (every logged-in page)src/scenes/AuthenticatedShell.tsx |
8.11 MiB · 3,020 files | 🔺 +38 B (+0.0%) | ████████░░ 83.5% of 9.71 MiB |
🟢 node_modules/monaco-editor/ stays out of src/index.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 node_modules/monaco-editor/ stays out of src/scenes/AuthenticatedShell.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
Largest files eagerly shipped from src/index.tsx
| Size | File |
|---|---|
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 24.6 KiB | ../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js |
| 6.3 KiB | ../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js |
| 4.5 KiB | ../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js |
| 3.9 KiB | ../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js |
| 1.4 KiB | ../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js |
| 1.3 KiB | src/RootErrorBoundary.tsx |
| 912 B | ../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js |
| 789 B | src/scenes/ChunkLoadErrorBoundary.tsx |
| 762 B | src/index.tsx |
Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx
| Size | File |
|---|---|
| 285.3 KiB | ../node_modules/.pnpm/posthog-js@1.408.2/node_modules/posthog-js/dist/rrweb.js |
| 267.7 KiB | ../node_modules/.pnpm/@posthog+icons@0.38.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js |
| 235.5 KiB | src/taxonomy/core-filter-definitions-by-group.json |
| 226.5 KiB | ../node_modules/.pnpm/posthog-js@1.408.2/node_modules/posthog-js/dist/module.js |
| 154.3 KiB | ../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js |
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 105.2 KiB | src/lib/api.ts |
| 94.7 KiB | ../packages/quill/packages/quill/dist/index.js |
| 93.3 KiB | ../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js |
| 90.6 KiB | ../node_modules/.pnpm/@tiptap+core@3.20.6_@tiptap+pm@3.20.6/node_modules/@tiptap/core/dist/index.js |
Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479
✅ Toolbar bundle — eager 2.18 MiB within budget
What the toolbar ships to customer pages, measured from the esbuild output (minified, post-tree-shake). The eager set is the entry plus everything statically imported from it — fetched before any feature runs; deferred chunks load lazily. The eager guardrail is 5.72 MiB. Each output file must also stay below 10 MB, where CloudFront stops compressing it. The module boundary is enforced separately by check-toolbar-graph.
| Metric | Size | Δ vs base | Budget |
|---|---|---|---|
| Eager (shipped) entry + static imports |
2.18 MiB · 17 files | 🔺 +38 B (+0.0%) | ████░░░░░░ 38.2% of 5.72 MiB |
| Deferred (lazy) | 2.07 MiB · 33 files | no change | n/a — loads on demand |
Loader dist/toolbar.js |
1.1 KiB | no change | █░░░░░░░░░ 5.8% of 19.5 KiB |
Largest eagerly-shipped chunks
| Size | File |
|---|---|
| 717.7 KiB | dist/toolbar/toolbar-app-UQPPHLFT.css |
| 546.1 KiB | dist/toolbar/chunk-chunk-JC4MEOXU.js |
| 484.4 KiB | dist/toolbar/chunk-chunk-F4F6H6OV.js |
| 133.6 KiB | dist/toolbar/chunk-chunk-UXUOCKXC.js |
| 131.8 KiB | dist/toolbar/chunk-chunk-T5KY5WYR.js |
| 71.0 KiB | dist/toolbar/toolbar-app-PKB4XMNF.js |
| 69.0 KiB | dist/toolbar/chunk-chunk-27JL52RE.js |
| 35.6 KiB | dist/toolbar/chunk-chunk-XJ2KWEQI.js |
| 20.9 KiB | dist/toolbar/chunk-chunk-4OUFGAAK.js |
| 12.2 KiB | dist/toolbar/chunk-chunk-PIK3PADE.js |
Posted automatically by check-toolbar-size · sizes are toolbar output bytes (shipped, post-tree-shake) from the esbuild metafile
✅ Dist folder size — 🔺 +39.7 KiB (+0.0%)
Total size of the built frontend/dist folder (all assets), compared against the base branch.
Total: 1368.99 MiB · 🔺 +39.7 KiB (+0.0%)
⚠️ Playwright — 1 flaky
🎭 Playwright report · View test results →
- Split a person with multiple distinct IDs (chromium)
These issues are not necessarily caused by your changes.
Annoyed by this section? Help fix flakies and failures and it will go green!
⚠️ Backend snapshots — 13 updated (13 modified, 0 added, 0 deleted)
Query snapshots: Backend query snapshots updated
Changes: 13 snapshots (13 modified, 0 added, 0 deleted)
What this means:
- Query snapshots have been automatically updated to match current output
- These changes reflect modifications to database queries or schema
Next steps:
- Review the query changes to ensure they're intentional
- If unexpected, investigate what caused the query to change
🦔 Hogbox preview · ✅ ready▶ Open the preview
commit |
| if (parsed.error) { | ||
| return true | ||
| } | ||
| return JSON.stringify(parsed.config) !== JSON.stringify(prompt.config ?? null) |
There was a problem hiding this comment.
Reordering object keys or changing whitespace marks an otherwise equivalent configuration as changed because isConfigChanged compares serialized key order while dirty checking compares raw editor text. This enables redundant publishing, shows a misleading configuration diff, and triggers unnecessary unsaved-change warnings.
Prompt To Fix With AI
This is a comment left during a code review.
Path: products/ai_observability/frontend/prompts/llmPromptLogic.ts
Line: 826
Comment:
**Compare configuration semantically**
Reordering object keys or changing whitespace marks an otherwise equivalent configuration as changed because `isConfigChanged` compares serialized key order while dirty checking compares raw editor text. This enables redundant publishing, shows a misleading configuration diff, and triggers unnecessary unsaved-change warnings.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.There was a problem hiding this comment.
Partly taken, in 6a54088. isConfigChanged now compares with recursively sorted keys, so reordering keys no longer shows a config diff in the review modal. The stronger reason than cosmetics: config is stored as Postgres jsonb, which does not preserve key order, so the old comparison advertised a change the server would not actually store. Added a test that a key reorder reads as unchanged and a value change still reads as changed.
Kept the dirty check on raw editor text on purpose. That is how the prompt text field behaves too, and warning about unsaved edits after the user edited the text (even just formatting) is standard text-field behavior.
There was a problem hiding this comment.
The jsonb angle is the right reason to fix this — the old comparison would have advertised a change the server would silently discard. Good call.
The raw-text dirty check rationale holds. Prompt text behaves the same way, and warning on any edit (including whitespace) is the expected contract for a text editor. No objection there.
Prompt To Fix All With AI### Issue 1
products/ai_observability/frontend/prompts/llmPromptLogic.ts:826
**Compare configuration semantically**
Reordering object keys or changing whitespace marks an otherwise equivalent configuration as changed because `isConfigChanged` compares serialized key order while dirty checking compares raw editor text. This enables redundant publishing, shows a misleading configuration diff, and triggers unnecessary unsaved-change warnings.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Reviews (1): Last reviewed commit: "fix(aio): vendor-neutral model name in c..." | Re-trigger Greptile |
Problem
Prompt versions can store a
configobject since #74264: model parameters or any settings an app wants to change without a deploy. The API, SDKs, and MCP tools can read and write it, but the UI can't. Anyone using the prompt editor can't see that a config exists, let alone change it.Changes
Everything is behind the
llm-prompt-configfeature flag.View mode shows the config as a JSON block under the prompt, only when the version has one.
Edit mode gets a Configuration section with a JSON editor. It validates that the text is a JSON object, same as the API. A Remove button clears it. Prompts without a config show an "Add configuration" button instead, which seeds a small starter example.
The publish review modal shows a config diff with a "Changed" tag when the config differs, so a config change gets the same look-before-you-publish treatment as a prompt change.
Smaller pieces:
configexplicitly: the parsed object, or null when cleared. Leaving the key out would make the API carry the old config forward, which is right for API users editing text but wrong for a form that owns the whole valueHow did you test this code?
Two new jest tests in
llmPromptLogic.test.ts: one pins that publish sends the parsed config and sends null when the editor is cleared (leaving the key out would silently keep the old config), one pins that invalid config goes to a field error instead of the review modal. One existing publish test now asserts the full request body includingconfig: null. Backend suite (132 tests) still green.I also drove the real app with the flag on and checked every surface above: view, edit, add, remove, config-only publish, the modal diff, and the History line. The screenshots are from that run.
Automatic notifications
Docs update
Docs land together with the SDK releases (PostHog/posthog-python#801, PostHog/posthog-js#4329), after this rolls out.
🤖 Agent context
Autonomy: Human-driven (agent-assisted)
I directed Claude Code through this and reviewed it in the running app. Skills invoked: /writing-user-facing-copy, /writing-tests, /writing-kea-logics conventions via frontend AGENTS.md. Decisions along the way: Monaco for the editor (already lazy-loaded by this scene for the compare view, so no bundle cost), a plain JSON editor over typed fields for v1 (config is schemaless by design), and the Remove button plus starter example came out of manual testing rather than the original plan.