Skip to content

feat: local image generation - #79

Merged
MegalithOfficial merged 60 commits into
mainfrom
feat/local-image-generation
Jul 25, 2026
Merged

feat: local image generation#79
MegalithOfficial merged 60 commits into
mainfrom
feat/local-image-generation

Conversation

@MegalithOfficial

Copy link
Copy Markdown
Contributor

Local Image Generation: stable-diffusion.cpp engine, Playground, LoRA ecosystem, and guided tours

This PR ships fully local, on-device image generation end to end: a managed stable-diffusion.cpp runtime, a curated hardware-aware model catalog, a dedicated Playground for freeform generation, a LoRA library with an integrated CivitAI browser, deep integration into the existing avatar/scene flows, guided tours across every new surface, and full localization. Local generation is desktop-only by design; mobile keeps the existing provider-based flows and never touches sd.cpp.

Stable Diffusion Engine

  • Managed sidecar: stable-diffusion.cpp runs as a managed sd-server sidecar with a native JSON API, forwarded logs, and lifecycle handling.
  • Versioned engine manager: install, switch between, and remove multiple engine builds. The build catalog is resolved at runtime; the best variant for the detected GPU (Vulkan/CUDA/CPU) is preselected automatically, with the detection result shown inline.
  • GPU-aware runtime manager: hardware is probed once and reused for build selection, model fit checks, and compute policy decisions.
  • Upscaler: one-tap install of the RealESRGAN model that powers the Upscale action and the high-resolution pass.

Local Model Catalog

  • Curated, hardware-checked model families: Z-Image (Turbo + base), FLUX.2 Klein (4B, 9B, 9B Base), Krea 2 (Turbo + RAW), and Qwen Image Edit 2511, each with per-variant quantizations, licenses, and sensible generation defaults.
  • Verified bundle installer: Select & Download resolves the full component set (diffusion GGUF, text encoders, VAE) from Hugging Face, with stall guards instead of brittle total-request timeouts, and per-role type badges on installed files.
  • Custom models: local image models are keyed by diffusion GGUF path; component files can be hand-picked from disk or from already-downloaded files, with auto-detection in the model editor.
  • Runtime feature work: generation progress events, cancellation, inpainting, upscaling, step caching, expanded sampler/scheduler/CFG/size controlompatibility checks.
  • The HF token moved to Local Runtime Defaults with guidance for gated model access.

Playground

A new desktop-only /playground route reachable from Settings → Image Generation:

  • Three-pane layout: prompt pane (prompt, negative prompt, img2img init image), fullscreen feed carousel in the center, and a settings pane (meps, CFG, sampler, scheduler, seed, hires fix, batch, LoRA stack). Side panes are drag-resizable; on narrow windows the panes become bottom sheets.
  • Write-ahead generation history: every generation is persisted with its full parameters and materialized seed, so history survives crashes and restarts.
  • Feed carousel: deterministic wheel paging, snap scrolling, measured skeletons at the target aspect ratio (including batch grids), and lazy older-page loading.
  • Remix actions on every card: copy prompt, reuse seed, regenerate, upscale, send to img2img, expandable prompt, per-image overlay actions for batches, and delete with optional file cleanup.
  • Init image sources: pick from disk or from any previous generation.

LoRA Library + CivitAI

  • New LoRA Library page with two tabs: Library (installed files with size, architecture badge, trigger keywords, keyword editing, CivitAI keyword discovery, delete) and Browse CivitAI (search, sort/period, base-model filter chips, NSFW-safe Pure-mode filtering, version/file pickers, queued downloads that land
    directly in the library).
  • CivitAI backend: search, model detail, auth (token save/validate/clear stored in the local DB), and download queueing through the shared download pipeline.
  • Only officially supported models are surfaced: search results, filter chips, and detail versions are all gated by the engine catalog itself (derived from the profile → LoRA architecture mapping, so new catalog families become browsable automatically). SD 1.x/2/XL, Pony, Illustrious, NoobAI, and Flux.1 content no longer appears.
  • Full result pages: the search command now pages CivitAI (up to 5 fetches of 100) until the requested count of supported results is met, so the first page arrives full instead of shrinking to a handful after filtering.
  • Missing-token warning: the browse tab checks token status and shows an inline warning (missing or expired) with an Add/Replace action; public downloads still work without one.
  • Always-active LoRAs: LoRAs without trigger keywords are supported and labeled as always active.
  • Flux LoRA application fixes and architecture detection from safetensors metadata with a compatibility cache.

Chat / Avatar / Scene Integration

  • Scene prompts are tailored for local image models, with split image protocols per model type.
  • Avatar generation supports local models (AI-editing disabled where the model can't do it).
  • Deleted local image models no longer re-register themselves on listing.
  • stable-diffusion.cpp is branded and kept selectable as a provider.

Settings & Discoverability UX

  • Image Generation page: Playground and Stable Diffusion Engine now share a compact side-by-side tools row at the top (desktop only), so Generation/Prompting config is no longer pushed below the fold.
  • Models page: the Model Browser / Local Runtime Defaults / LoRA Library shortcut row switched from dashed "empty-state" styling to the app's solid nav-card idiom with one-line descriptions.
  • LoRA Library: tabs, installed count, and Import file merged into a single header row.

Guided Tours

  • Five new tours on the existing GuidedTour engine: Image Generation settings, Playground, LoRA Library, CivitAI Browse (fires on first visit to the Browse tab), and Stable Diffusion Engine, plus a new CivitAI-token step in the existing Runtime Defaults tour.
  • Engine upgrades: per-step desktopOnly flag (steps are filtered on mobile instead of stalling on missing targets) and optional flag (condip in ~400ms instead of 4s).
  • Demo card: if the Playground feed is empty when the tour reaches the feed step, a realistic example card (image, prompt, model/size/seed chips, "Example" badge) renders in place of the empty state. The demo asset is dynamically imported only when that step is active and disappears with the tour.
  • All tours replayable via window.__debug.resetTour(id) / resetAllTours().
  • Demo card: if the Playground feed is empty when the tour reaches the feed step, a realistic example card (image, prompt, model/size/seed chips, "Example" badge) renders in place of the empty state. The demo asset is dynamically imported only when that step is active and disappears with the tour.
  • All tours replayable via window.__debug.resetTour(id) / resetAllTours().

Desktop-only Gating

  • The Playground card and SD Engine section are hidden on mobile; /playground and the LoRA Library render a desktop-only fallback if reached directly; sd.cpp commands are never invoked from mobile code paths.

i18n

  • All new UI strings (tours, token warnings, playground labels, LoRA library, engine manager) added to en.ts and translated across all 21 locales in each file's established register and terminology; zh-Hans/zh-Hant written independently. Brand terms (CivitAI, LoRA, Playground, Stable Diffusion, ESRGAN) untranslated.

Verification

  • bun run check (tsc + cargo check) green throughout.
  • Tours exercised via the debug reset helpers on populated and empty states; missing-token banner verified against saved/missing/expired token states; mobile gating verified via the platform fallback paths.

Add a desktop-only local image runtime that discovers upstream releases dynamically, lets users select compatible CPU and accelerator builds, verifies model and runtime artifacts, and registers curated image models without fixed database IDs.

Coordinate the sidecar with llama.cpp so only one local workload owns the GPU, support multi-reference and structured LoRA requests, and replace guessed VRAM sizing with an exact one-step runtime probe.

Expose the curated installer and download state in image-generation settings, localize the new controls across all supported locales, and hide synced stable-diffusion.cpp models and providers on mobile.
Detect compatible stable-diffusion.cpp builds and recommend a GPU-specific runtime variant while keeping manual variant selection available.

Add installation, switching, deletion, and persistent active-version handling through a dedicated settings page. Reuse the Hugging Face download card and seed expected asset sizes so GitHub runtime downloads report progress correctly.
Add Stable Diffusion model variant installation and removal through the shared download queue, including live progress and runtime pairing.

Estimate sd.cpp component placement before download from the selected engine's devices and live memory budgets, then retain real execution probes for installed models. Surface GPU, CPU-offload, RAM, and component reserve details in a bottom menu so users can compare variants before installing.
Manage versioned stable-diffusion.cpp runtimes and variants, select compatible backends from detected hardware, and persist per-device multi-GPU budgets and split policies for planning and launch.

Gate model downloads on an installed engine, recalculate runnability after hardware refresh, and expose fast non-blocking SHA verification so completed transfers no longer appear stuck.
Add larger FLUX.2 Klein variants and the Krea 2 series to the local model catalog with component paths and hardware-aware requirements.

Persist model-level LoRAs in the model editor, merge them with request adapters in stable-diffusion.cpp, load them authoritatively from the database, and expose filtered runtime application diagnostics. Local avatar generation now sends concise prompts directly so adapter trigger words are not diluted by template prose.

Add the associated model fields, commands, parameter descriptions, and localized copy across every supported locale.
Replace hardcoded sd.cpp sampling defaults with persisted model-aware sampler, scheduler, guidance, VAE tiling, reference, and high-resolution controls.

Expose model-level pre-prompts, apply them once before provider dispatch, add payload regression tests, and translate the new UI across every supported locale.
…ecks

Persist discovered and manually edited LoRA keywords, file identity, and architecture in SQLite so every model and generation request uses the shared library record.

Detect architecture from SafeTensors metadata or exact-hash Civitai data, warn on unknown compatibility, block known mismatches, and repair partial schemas through migrations v80 and v81.

Add the localized bottom-sheet keyword editor, required-keyword validation, prompt activation, and identical-file import reuse across all supported locales.
Split the scene writer template by image backend so existing remote workflows keep their reference-driven format while local models receive concise LoRA-aware instructions.

Resolve saved character and persona LoRAs with trigger keywords, generate local scenes without appearance prose or reference images, and avoid duplicating trigger keywords in the final image prompt.
… files in picker, add unfiltered browser search
Avatar editing remained exposed whenever an avatar generation model was available, including when the selected model used the local sdcpp provider.

Hide local edit and refine entry points, guard stale edit actions, and explain the restriction in every supported locale while preserving generation, regeneration, repositioning, and upscaling.
Convert Flux/Klein LoRA tensor aliases before stable-diffusion.cpp inference so all compatible tensors are applied, while preserving manually keywordless LoRAs as always-active styles. Keep transient WIDTHxHEIGHT input text during editing so vertical image sizes can be saved.
MegalithOfficial and others added 29 commits July 20, 2026 01:27
@MegalithOfficial
MegalithOfficial merged commit c3e4f23 into main Jul 25, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant