Pre-warm cache for user templates typically generated when first logging in

[philayres]

The first time a user logs in, a fairly complete set of templates are created and downloaded. Although this doesn't have to be repeated often, the initial load is offputting for users. It would be good to pre-warm the cache for users that have been shown to login to the system within the last month (configurable time), for each of the apps they have access to.

[philayres]

Implementation plan

Background: two independent caching layers

Layer 1 — Rails.cache fragment (out of scope here). masters/_cache_search_results_template.html.erb wraps the render in Rails.cache.fetch(partial_cache_key(...)). That key includes current_sign_in_at, so the fragment is cold on every login and cannot be pre-warmed ahead of one. This is deliberate (#1270 / #1287 browser cache poisoning) and belongs with the cache-key audit in #1289. The warmer is structured so switching to it later is a one-line change.

Layer 2 — on-disk compiled Handlebars (the target). handlebars_cache_key deliberately excludes current_sign_in_at, so these artifacts survive logins. They go cold when a dynamic definition changes or the server restarts — and HandlebarsPrecompiler.cleanup_public_dir wipes them on every Rails process boot, which is why the first login after a deploy is slow.

Confirmed: any dynamic definition save rotates the key for everyone, via both handlebars_item_updates_key and the Rails.cache.clear in AdminHandler#invalidate_cache dropping server_cache_version.

Phases

1. Offline render harness

• PrewarmController < ApplicationController overriding current_user to return an injected user via assigns:. ActionController::Renderer doesn't run before_action, so authentication never fires.
• Prewarm::MasterTemplates renders the inner partial (masters/search_results_template), avoiding a layer-1 fragment written under a key that can never be hit today. Fresh User instance per combination, app_type_id set in memory only — never saved.
• Warm marker written by whichever path warms a combination — the warmer and ordinary user requests — keyed on (user_id, app_type_id, access_control_version), all known before rendering. Without this the warmer re-renders the full ERB for every user who logged in before it reached them.

2. Per-artifact lock

The lock is a work-duplication optimisation, not a correctness mechanism — atomic writes plus deterministic filenames make concurrent producers safe. Two non-contending lock types: artifact locks held for seconds, and the warmer lock held across a pass but never touched by any request path. A user request therefore never blocks: short bounded wait, then it does the work itself (on_contention: :proceed). The warmer skips contended artifacts (:skip). Warm passes can't stall the system.

Also makes artifact writes atomic (.tmp + File.rename), which is what makes the lock optional rather than load-bearing, and stops any reader being served a half-written bundle.

3. Candidate selection + settings — non-disabled, not_template users with current_sign_in_at in a configurable window (default 30 days) × Admin::AppType.all_available_to(user).

4. In-process, per-instance warmer

delayed_job is deliberately not used: its queue is DB-backed and shared, so a queued job warms one instance while the rest stay cold, and one worker per instance can't be guaranteed long term.

• Prewarm::Warmer — one background thread, started from an initializer in web/server processes only.
• Single runner per instance via a non-blocking flock; other puma workers skip.
• Generation-poll loop replaces both a schedule and a signal: each tick, if the generation key differs from the last warmed one and has been stable for a quiet period, run a pass then sweep. Self-synchronising across processes and instances, and it debounces config bursts and app imports for free. Admin::AppTypeImport.import_in_progress? is also checked, as a cheap same-process optimisation.
• Throttled, low priority, with_connection, stop flag, per-combination rescue.
• prewarm:templates rake task for manual/deploy-hook use.

5. Disk layout and sweeping — no retention

Superseded artifacts must not accumulate. Bounded structurally, not by best-effort filename matching:

• Move templates/ + partials/ out of public/ into tmp/ — handlebars_template_tag has its javascript_include_tag commented out, so the browser only ever fetches the multi/ bundle; these are a purely server-side compile cache.
• Generation directories gen-<global_key>/ keyed only on non-user-specific inputs. Sweeping is one glob and an rm_rf.
• Count-based retention: current plus up to N previous (default 2), ordered by mtime, with a short safety window — a worker that hasn't observed the rotation still reads the old generation.
• Content-addressed individual files: name by source digest instead of the per-user cache key, collapsing templates × users × app types to templates × distinct variants. Strictly more correct than the current key, which over-partitions.
• Guard the boot-time cleanup_public_dir to web/server processes — bin/delayed_job shares /var/app/current and would otherwise wipe the web server's templates on restart.

6. Specs

Key guarantees: a user request writes the marker and the warmer then skips without entering the render path; a user request is never blocked by an in-progress warm pass; two concurrent renders compile exactly once; after N successive definition changes at most KeepGenerations + 1 directories remain and file count doesn't grow with N; and the payoff — pre-warm, sign in, and GET /pages/:id/template invokes no node CLI, with a paired negative spec proving it does when unwarmed.

Open items

1. App type scope — all app types available per user (complete) vs current app_type_id only (cheaper at scale).
2. CPU budget — warming now shares the box with request serving; whether to pause under high load rather than throttle blindly.
3. Tuning defaults — quiet period, tick interval, generations retained, bounded lock wait.

[philayres]

Plan update: split into two independently shippable stages

Revising the plan above. Most of it does not depend on pre-warming and delivers value on its own — and one part of it may resolve most of this issue without a warmer at all.

The insight

Compiled files are named by handlebars_compiled_filename, which uses handlebars_cache_key — and that key includes the user id.

So user A logs in and compiles ~N templates into files named with A's key. User B logs in needing the same N templates, byte-identical, but named with B's key — so every one is recompiled from scratch. That means every user pays the full node-CLI compile cost on their first login after each rotation, not just the first user.

With content-addressed naming (digest of the preprocessed source), only the first user after a rotation pays. Everyone else hits the existing File.exist? short-circuit in write_handlebars_template and skips both the temp write and the node subprocess. They still assemble their own multi bundle, but that's file reads plus one write — no subprocess.

"Every user's first login is slow" becomes "the first login after a deploy is slow." That is likely most of this issue, with no warmer involved.

Stage 1 — Handlebars compile efficiency & correctness (no warmer)

Phase 2 (lock + atomic writes) and Phase 5 (content addressing, generation directories, sweeper, move intermediates out of public/, boot-cleanup guard), with the corresponding specs.

Beyond the redundant-compile fix, this addresses four problems that are live today, independent of warming:

• Readers can be handed a half-written bundle — output files are written with File.write straight into public/, non-atomically.
• Thundering-herd recompiles when several people log in at once after a deploy, each shelling out to the node CLI for the same artifacts.
• Orphaned generations accumulate on every config change; nothing is deleted until the next process boot.
• A delayed_job restart wipes the web server's compiled templates and rotates server_cache_version mid-day, because cleanup_public_dir runs in every Rails process and bin/delayed_job shares /var/app/current.

Stage 2 — Pre-warming

Phase 1 (offline render harness + warm marker), Phase 3 (candidate selection), Phase 4 (in-process per-instance warmer), with their specs.

After Stage 1 this covers only the single remaining cold login after a deploy, plus brand-new users.

Measurement gate between the stages

Benchmark GET /pages/:id/template for the 1st, 2nd and 3rd distinct user after a restart, before and after Stage 1. Expect users 2+ to drop sharply once content addressing lands.

If they do, Stage 2's remaining value is narrow, and it should be re-justified before building a long-lived background thread, a poll loop and an offline render harness into every web process. Stage 1 is contained entirely within handlebars_precompiler_helper.rb, its initializer and their specs, with no new runtime moving parts — much the lower-risk half.

spec/helpers/handlebars_precompiler_helper_spec.rb gains the Stage 1 payoff assertion: two different users with identical template source share one compiled file, and Utilities::ProcessPipes.pipe_in_out is invoked exactly once. That spec is the evidence for whether Stage 2 is still needed.

Caveat

Content addressing changes the compiled-filename contract and touches several existing specs that assert "<id>-<cache_key>.js" naming, so the diff is larger than it sounds. The one wrinkle — write_multiple_handlebars_templates currently recomputes the filename from the template id via read_handlebars_template — is already solved: handlebars_template_tag records compiled_file_path: in @requested_handlebars_templates, so the bundle writer uses the recorded path instead.

[philayres]

Stage 2 plan — diverges from the original phases 1, 3 and 4

Stage 1 has shipped (#1377), and reviewing Stage 2 against what actually landed changes its shape enough that it is worth recording before any code is written. Two of the three original open items are resolved below; the third (CPU budget) is dropped as no longer relevant.

What Stage 1 left behind

Individual compiled templates and partials are now content-addressed — handlebars_compiled_filename(template_id, source) digests the preprocessed source — so they are already shared by every user whose rendered content is identical, and they live under tmp/handlebars-<env>/gen-<generation_key>/.

That leaves exactly one per-user artifact: the multi bundle, requested-templates-<user_id>-<app_type_id>-<req_digest>-<access_control_version>.js. Its content is a function only of the requested template set and the compiled files it concatenates — not of user identity. Assembling it is a handful of file reads plus one atomic write, with no node subprocess.

That single fact drives most of the divergence below.

Divergence 1 — warm one representative per access variant, not every user × app type

Original phase 3 selected every non-disabled, non-template user with current_sign_in_at in a window, crossed with Admin::AppType.all_available_to(user). Post-Stage-1 that is largely wasted work: users sharing roles and access produce byte-identical compiled artifacts, which are now shared by construction.

Instead, candidates are grouped by a variant key over (app_type_id, sorted active Admin::UserRole role_names, that user's own Admin::UserAccessControl rows), and one representative per variant is warmed. That populates every expensive artifact. Real users still assemble their own bundle on first request, which is cheap.

The variant key is deliberately a coverage heuristic, not a correctness mechanism. Because artifacts are content-addressed, an over-sharing key can only cause a missed warm — which a real request then compiles as it does today — and never a wrong or cross-contaminated artifact. This is what makes the cheap grouping safe to reason about.

This resolves open item 1 (app type scope): all app types available to each representative, since the number of representatives is now small.

Divergence 2 — shelled rake task on server start, not an in-process warmer thread

Original phase 4 specified a long-lived background thread started from an initializer, a generation-poll loop with a quiet period, an Admin::AppTypeImport.import_in_progress? check, and a single-runner flock. All of that is dropped.

Instead: rake prewarm:templates is spawned as a detached child process via IO.popen from the web process at server start/restart, guarded to web/server processes only and to a single spawn.

Rationale: with the warmer running only at start/restart, the poll loop and quiet-period debounce exist solely to catch mid-day generation rotations from config changes. That cost is one cold request, which is what happens today anyway. Trading it away removes a long-lived thread, a poll loop and a debounce from every web process — the exact risk the Stage 1/Stage 2 split was created to avoid taking on lightly.

Accepted trade-off, stated plainly: a config change that rotates the generation key mid-day is not pre-warmed. The next user request pays the compile cost, unchanged from current behaviour.

This also makes open item 2 (CPU budget, pausing under load) moot — there is no long-running in-process work to pause. A throttle between renders in the child process is sufficient.

The ordering constraint that governs the implementation

Application.server_cache_version is a Rails.cache.fetch against shared memcached and feeds HandlebarsPrecompiler.generation_key. startup_cleanup! deletes that key on web boot.

If the child process boots first, it sets a value, the web process then deletes it and resolves a different one — and every warmed artifact lands in an orphaned generation directory, silently doing nothing. So the spawn must happen from inside the existing after_initialize block in config/initializers/handlebars_precompiler.rb, after startup_cleanup!, rather than from a separate initializer whose ordering is incidental.

The same existing rake guard (next if rake_tasks.any? && rake_tasks.none? { |t| %w[server s].include?(t.to_s) }) is what stops the child running startup_cleanup! and wiping the web server's directories, and what stops it recursively spawning itself. Both get a spec.

Phases

0. Measurement gate — blocks everything below. Unchanged from the Stage 1/Stage 2 split: benchmark GET /pages/:id/template for the 1st, 2nd and 3rd distinct user after a restart, on develop and on the parent of the #1377 merge. Primary metric is the count of Utilities::ProcessPipes.pipe_in_out invocations per request, which is deterministic, with wall-clock as secondary. Results posted here for an explicit go/no-go before phase 1.

1. Offline render harness. PrewarmController overriding current_user to an injected user and current_admin to nil; Prewarm::MasterTemplates renders masters/search_results_template — the inner partial — via ActionController::Renderer, with app_type_id assigned in memory and never saved. That partial calls retrieve_requested_handlebars_templates itself, so rendering it alone drives compile and bundle assembly. Rendering the inner partial also avoids writing a layer-1 fragment under a key including current_sign_in_at that can never be hit.

2. Candidate selection. Prewarm::Candidates — sign-in window, active/not_template, variant grouping as above, capped.

3. prewarm:templates rake task. Calls setup_directories explicitly (the after_initialize body is skipped for non-server rake tasks), wraps the pass in FileLock.acquire('prewarm-pass', wait: 0, on_contention: :skip) so a second invocation is a no-op and no user request can ever be blocked, throttles between renders, rescues per variant, honours a warm marker.

4. Spawn on server start. Prewarm::Spawner — IO.popen + Process.detach, output drained to the log, no-op for delayed_job and console/runner processes, single-spawn lock.

5. Settings. PrewarmTemplatesEnabled (FPHS_PREWARM_TEMPLATES, default off — opt-in), PrewarmSignInWindowDays (30), PrewarmMaxVariants (50), PrewarmThrottleSeconds (0.5). This is open item 3.

6. Specs. Candidate variant grouping including the case where a user-specific access-control row splits a variant; harness renders with no signed-in user and never persists the injected app_type_id; spawner guards; task no-op on contention and on warm marker. The payoff spec is the one that matters: pre-warm, sign in, GET /pages/:id/template, assert pipe_in_out is never invoked, paired with a negative example proving it is invoked when unwarmed.

Still out of scope

Layer-1 Rails.cache fragment warming — the key includes current_sign_in_at and is cold on every login by design (#1270 / #1287), and belongs with the cache-key audit in #1289.

[philayres]

masters/_cache_search_results_template.html.erb cache was originally tied to the user login to allow a mechanism for failures during handlebars compilation to be recovered from, without having to restart the server for everybody. It also provided a cheap way of ensuring that config changes could easily be picked up if the last update on dynamic defs, etc weren't correctly produced.

Since handlebars compilation is now more reilient to failure, we should revisit this assumption. As part of this, we need to be very careful to ensure that failures can't leave a user's template cache half created.