sync: knowledge-base images, resumable indexing, readable tool results, job progress - #115
Merged
Conversation
added 2 commits
August 18, 2026 20:46
…s, job progress Derived from the upstream tree with `scripts/build_ce.py`; all release gates pass (brand scan, required-runtime manifest, LICENSE, and the CE import/OpenAPI/ORM contract self-check). **Knowledge base learns to see.** Images are first-class now: the uploader stops rejecting them, figures cut out of document layouts are stored as assets with a vision-generated caption, and retrieval returns them as structured `images` (url + caption + locator) instead of a bare filename. Both the tool-result card and the citation popover render thumbnails, and a document that is nothing but a picture no longer reports "no content". New migration `ce_0007_kb_assets`. **Indexing survives a restart.** Parse → chunk → embed used to run on FastAPI BackgroundTasks, which competes for Starlette's shared request threadpool — a bulk upload saturated it, every endpoint queued before reaching its handler, and the work was lost on restart with documents stuck at "indexing" forever. It now runs on a persistent queue with a dedicated worker that adopts whatever the previous process left behind. **Tool results are readable.** Raw JSON walls are replaced by a structured data view: long lists page in, long text is windowed rather than truncated to a grey brick, and opening a citation scrolls to the evidence instead of always showing the first 500 characters. **Background jobs report honestly.** Plan progress is persisted server side rather than living only in the prompt and being stamped by the browser, internal wake-up prompts no longer leak into the transcript, a batch job that produced nothing stops claiming success, and the delivery turn — artifact cards and plan close-out included — is pulled back into the page for users who walked away while the job ran. **Smaller first paint.** Static assets ship pre-compressed at maximum level (`gzip_static`; nginx's implicit `gzip_comp_level 1` was costing ~250KB on the entry bundle alone), the 76 unused Univer hyphenation tables are dropped from the build, and the frontend image tolerates macOS AppleDouble sidecar files. Also: MCP servers published by an administrator on an internal address are now installable by users.
…ut renderer The renderer referenced DataView without importing it, so tsc resolved the name to the built-in DataView type and the desktop cold-start build failed with TS2786 on all four platforms. Verified with a full npm run build.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Community-edition sync derived from the upstream tree with
scripts/build_ce.py.74 files changed (+6513 / −242), no deletions.
Knowledge base learns to see
Images are first-class now. The uploader stops rejecting them (the backend had
always accepted them — the file picker was the gate), figures cut out of document
layouts are stored as assets with a vision-generated caption, and retrieval
returns them as structured
images(url + caption + locator) rather than a barefilename. Both the tool-result card and the citation popover render thumbnails,
and a document that is nothing but a picture no longer reports "no content".
Listing datasets deliberately does not carry images — that is a directory call,
and stuffing every library's pictures into it only burns prompt budget. Anything
you want to show has to be retrieved first.
New migration:
ce_0007_kb_assets. New settings are documented indocument/{zh-CN,en}/deployment/environment-variables.md(KB_MULTIMODAL_INDEXING,KB_ASSET_CAPTION_MAX_PER_DOC,KB_ASSET_CAPTION_BATCH,KB_ASSET_TEXT_MAX_CHARS,KB_ASSET_MIN_BYTES/KB_ASSET_MIN_EDGE_PX,KB_ASSET_URL_PREFIX).Indexing survives a restart
Parse → chunk → embed used to run on FastAPI
BackgroundTasks, which competes forStarlette's shared request threadpool. A bulk upload saturated it, so every
endpoint queued before reaching its handler — the UI spun and the gateway
returned 502s — and because the work was in-process it was lost on restart, leaving
documents stuck at "indexing" forever.
It now runs on a persistent queue with a dedicated worker (
core/kb/index_queue.py,core/kb/index_worker.py) that adopts whatever the previous process left behind.Tool results are readable
Raw JSON walls are replaced by a structured data view: long lists page in, long
text is windowed instead of being truncated into a grey brick, and opening a
citation scrolls to the evidence rather than always showing the first 500
characters of the tool output.
Background jobs report honestly
being stamped by the browser — the plan strip no longer freezes at "2/5".
into the page for users who walked away while the job ran. This works off the
conversation's
updated_at, not SSE, so it survives a closed tab.Smaller first paint
gzip_static. nginx's implicitgzip_comp_level 1was costing roughly 250KB onthe entry bundle and main stylesheet alone — close to a second on a few-Mbps
link.
Also
MCP servers published by an administrator on an internal address are now
installable by users.
Verification
python3 scripts/build_ce.py --import-check— all release gates pass: brand scan(0 text hits, full path scan), required-runtime manifest (66 entries), forbidden
artifacts, binary asset allowlist, LICENSE gate, and the CE import / OpenAPI / ORM
contract self-check (no enterprise modules, routers, ORM tables or schema
properties reachable from the derived tree).