Feat/spreadsheet import

[wilbert]

PR 5: Asynchronous spreadsheet import with live progress

Admins upload a .csv or .xlsx of users; Solid Queue processes it in the
background and the progress bar updates over Solid Cable.

Resumability via Active Job Continuations

ProcessImportJob uses the Rails 8.1 ActiveJob::Continuable API. The job
declares three steps (count_rows, import_rows, finalize) and checkpoints
a row cursor every 100 rows. On graceful shutdown or interruption, Solid Queue
re-enqueues and the job resumes at the cursor rather than restarting at row 1.

Checkpointing every 100 rows rather than every row keeps the checkpoint writes
off the hot path; an interruption replays at most 99 rows, and those replays
are no-ops (see idempotency below).

Partial success, not all-or-nothing

There is no wrapping transaction. One malformed email on row 4,000 must not
discard 3,999 valid rows. Each row is validated independently and the outcome
tallied as created / skipped / failed, with rejected rows collected into
error_report.

The tradeoff: a failed import leaves partial data behind. That is why the
Import record carries a full accounting of what happened rather than just a
status flag. The opposite choice (atomic, reject the whole file) is defensible
for a finance-grade import; for user onboarding, partial success is friendlier.

error_report is capped at 500 entries with failed_count tracking the real
total, so a 50k-row file of garbage cannot write a 50k-element JSONB column
that then has to be serialized on every page render.

Idempotency does not depend on the cursor

A checkpoint can be lost between step.set! and a crash, so row application is
idempotent on its own: find_or_initialize_by(email_address:) skips users that
already exist. Rerunning any import is a no-op. ActiveRecord::RecordNotUnique
is rescued as a skip, since find-then-save is check-then-act and loses races
against concurrent signups; the unique index is the real guarantee.

Consequence: this import cannot bulk-edit existing users. Intentional.

Memory

CSV.foreach and Roo#each_row_streaming, never read or parse.
import.file.open streams the blob to a tempfile rather than
download into a String. A 50k-row upload must not be a memory event on a
512 MB Fargate task.

Broadcast suppression

Without a guard, a 10k-row import fires 10k dashboard broadcasts through the
PR 4 after_commit hook. DashboardBroadcasts.suppress wraps the row loop
using ActiveSupport::IsolatedExecutionState (fiber-aware, unlike
Thread.current), and finalize fires exactly one dashboard broadcast.
Import progress broadcasts are batched at the same 100-row cadence.

Progress broadcasts carry a signal, not a payload, for the same authorization
reason as PR 4.

Security notes

• Cell values have leading = + - @ stripped on ingest. A cell containing
  =HYPERLINK(...) that round-trips back out to a spreadsheet is a live
  formula injection; stripping on the way in beats remembering to escape on
  every export path.
• Zip.setup { validate_entry_sizes = true } because xlsx is a zip archive and
  decompression bombs are an upload vector.
• Content type and 10 MB size validated before the job is queued.
• Imported users get a SecureRandom password and cannot sign in until they
  run a password reset. Production would send an invitation token instead.

Why row-by-row and not insert_all

Roughly 50x slower, and chosen anyway: insert_all bypasses validations,
has_secure_password, normalizes, and Active Record encryption on
email_address. Encrypted columns cannot be batch-inserted without hand-rolling
ciphertext. Measured at ~2,100 rows/sec locally, which is comfortably inside
the resumable-job envelope.

Infrastructure

Imports run on a dedicated single-threaded imports queue. Sharing default
means a large import starves password reset mails behind it.

Test plan

• bundle exec rspec spec/imports spec/jobs spec/channels/import_channel_spec.rb spec/requests/admin/imports_spec.rb
• Manual: upload a 5k-row CSV, watch the bar advance, kill the Solid Queue
  worker mid-run, restart it, confirm it resumes and the final count is correct.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: f8b27046-8a22-443d-8504-6eb335f2a84f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}