Skip to content

sync: knowledge-base images, resumable indexing, readable tool results, job progress - #115

Merged
Luhaozhu merged 2 commits into
mainfrom
sync/ce-2026-08-19
Aug 20, 2026
Merged

sync: knowledge-base images, resumable indexing, readable tool results, job progress#115
Luhaozhu merged 2 commits into
mainfrom
sync/ce-2026-08-19

Conversation

@Luhaozhu

Copy link
Copy Markdown
Contributor

Community-edition sync derived from the upstream tree with scripts/build_ce.py.
74 files changed (+6513 / −242), no deletions.

Knowledge base learns to see

Images are first-class now. The uploader stops rejecting them (the backend had
always accepted them — the file picker was the gate), figures cut out of document
layouts are stored as assets with a vision-generated caption, and retrieval
returns them as structured images (url + caption + locator) rather than a bare
filename. Both the tool-result card and the citation popover render thumbnails,
and a document that is nothing but a picture no longer reports "no content".

Listing datasets deliberately does not carry images — that is a directory call,
and stuffing every library's pictures into it only burns prompt budget. Anything
you want to show has to be retrieved first.

New migration: ce_0007_kb_assets. New settings are documented in
document/{zh-CN,en}/deployment/environment-variables.md (KB_MULTIMODAL_INDEXING,
KB_ASSET_CAPTION_MAX_PER_DOC, KB_ASSET_CAPTION_BATCH, KB_ASSET_TEXT_MAX_CHARS,
KB_ASSET_MIN_BYTES / KB_ASSET_MIN_EDGE_PX, KB_ASSET_URL_PREFIX).

Indexing survives a restart

Parse → chunk → embed used to run on FastAPI BackgroundTasks, which competes for
Starlette's shared request threadpool. A bulk upload saturated it, so every
endpoint queued before reaching its handler — the UI spun and the gateway
returned 502s — and because the work was in-process it was lost on restart, leaving
documents stuck at "indexing" forever.

It now runs on a persistent queue with a dedicated worker (core/kb/index_queue.py,
core/kb/index_worker.py) that adopts whatever the previous process left behind.

Tool results are readable

Raw JSON walls are replaced by a structured data view: long lists page in, long
text is windowed instead of being truncated into a grey brick, and opening a
citation scrolls to the evidence rather than always showing the first 500
characters of the tool output.

Background jobs report honestly

  • Plan progress is persisted server side instead of living only in the prompt and
    being stamped by the browser — the plan strip no longer freezes at "2/5".
  • Internal wake-up prompts stop leaking into the visible transcript.
  • A batch job that produced nothing stops claiming success.
  • The delivery turn — artifact cards and plan close-out included — is pulled back
    into the page for users who walked away while the job ran. This works off the
    conversation's updated_at, not SSE, so it survives a closed tab.

Smaller first paint

  • Static assets ship pre-compressed at maximum level and are served with
    gzip_static. nginx's implicit gzip_comp_level 1 was costing roughly 250KB on
    the entry bundle and main stylesheet alone — close to a second on a few-Mbps
    link.
  • 76 unused Univer hyphenation tables are dropped from the build.
  • The frontend image tolerates macOS AppleDouble sidecar files.

Also

MCP servers published by an administrator on an internal address are now
installable by users.

Verification

python3 scripts/build_ce.py --import-check — all release gates pass: brand scan
(0 text hits, full path scan), required-runtime manifest (66 entries), forbidden
artifacts, binary asset allowlist, LICENSE gate, and the CE import / OpenAPI / ORM
contract self-check (no enterprise modules, routers, ORM tables or schema
properties reachable from the derived tree).

Luhaozhu added 2 commits August 18, 2026 20:46
…s, job progress

Derived from the upstream tree with `scripts/build_ce.py`; all release gates pass
(brand scan, required-runtime manifest, LICENSE, and the CE import/OpenAPI/ORM
contract self-check).

**Knowledge base learns to see.** Images are first-class now: the uploader stops
rejecting them, figures cut out of document layouts are stored as assets with a
vision-generated caption, and retrieval returns them as structured `images`
(url + caption + locator) instead of a bare filename. Both the tool-result card
and the citation popover render thumbnails, and a document that is nothing but a
picture no longer reports "no content". New migration `ce_0007_kb_assets`.

**Indexing survives a restart.** Parse → chunk → embed used to run on FastAPI
BackgroundTasks, which competes for Starlette's shared request threadpool — a
bulk upload saturated it, every endpoint queued before reaching its handler, and
the work was lost on restart with documents stuck at "indexing" forever. It now
runs on a persistent queue with a dedicated worker that adopts whatever the
previous process left behind.

**Tool results are readable.** Raw JSON walls are replaced by a structured data
view: long lists page in, long text is windowed rather than truncated to a grey
brick, and opening a citation scrolls to the evidence instead of always showing
the first 500 characters.

**Background jobs report honestly.** Plan progress is persisted server side
rather than living only in the prompt and being stamped by the browser, internal
wake-up prompts no longer leak into the transcript, a batch job that produced
nothing stops claiming success, and the delivery turn — artifact cards and plan
close-out included — is pulled back into the page for users who walked away while
the job ran.

**Smaller first paint.** Static assets ship pre-compressed at maximum level
(`gzip_static`; nginx's implicit `gzip_comp_level 1` was costing ~250KB on the
entry bundle alone), the 76 unused Univer hyphenation tables are dropped from the
build, and the frontend image tolerates macOS AppleDouble sidecar files.

Also: MCP servers published by an administrator on an internal address are now
installable by users.
…ut renderer

The renderer referenced DataView without importing it, so tsc resolved the name
to the built-in DataView type and the desktop cold-start build failed with
TS2786 on all four platforms. Verified with a full npm run build.
@Luhaozhu
Luhaozhu merged commit 5e492d9 into main Aug 20, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant