Skip to content

Allow REDCap pulls to continue when an individual record's save trigger raises #1383

Description

@philayres

Summary

Currently, Redcap::DataRecords#store processes retrieved records in a loop
(app/models/redcap/data_records.rb):

subset.each do |record|
  res = create_or_update record
  upserts << res if res
end

create_or_update calls new_record.save / existing_record.save with no rescue, and this loop has
no rescue either. Any unhandled exception raised while persisting or triggering a single record —
whether from a before_save-phase save trigger (row not yet persisted) or an after_commit-phase
save trigger (row already persisted, e.g. create_reference, add_tracker, generate_document) —
propagates all the way up through store to retrieve_validate_store, which logs the error to the
job request and re-raises, aborting the entire pull run.

This means a single bad/edge-case REDCap record (e.g. one that fails a downstream save-trigger
action such as create_reference due to unexpected data, or simply fails field validations) currently blocks every other record in
that pull from being processed, rather than being skipped/reported individually.

Requirement

Add an option (opt-in, not a change of default behavior) that allows individual record
exceptions during a REDCap pull's store phase to be caught and recorded, so the pull continues
processing the remaining records instead of aborting the whole run.

Proposed Behavior

  • Add a project-admin-configurable option (e.g. on Redcap::ProjectAdmin /
    records_request_options, following existing patterns like verify_file_fields) to enable
    per-record exception isolation during store.
  • When enabled, wrap the per-record create_or_update call in store's loop in a rescue that:
    • Catches the exception for that record only.
    • Records it in errors (consistent with the existing { id:, errors:, action: } shape used
      elsewhere in this class) so it's visible in the job request result.
    • Allows the loop to continue to the next record.
  • When disabled (or by default, to preserve current behavior for callers relying on
    fail-fast semantics), retain today's behavior: an unhandled exception aborts the whole pull.
  • Clarify and document, for a record that fails via a before_save trigger exception, that its
    row is not persisted (save aborted), vs. failing via an after_commit trigger exception, where
    the row is already committed but subsequent trigger actions for that record did not complete —
    both cases should be caught and reported the same way under this option.
  • Ensure capture_files' existing per-field rescue behavior is unaffected (it already isolates
    failures without this option).

Acceptance Criteria (tests)

  • Spec demonstrating that with the option disabled (default), a single record whose save
    trigger raises still aborts the whole store run (current/documented behavior, regression guard).
  • Spec demonstrating that with the option enabled, a record whose save trigger raises is
    recorded in errors and other records in the same pull are still created/updated.
  • Spec covering both a before_save-phase trigger failure and an after_commit-phase trigger
    failure under the new option.
  • Follow Rspec project coding standards; add a
    comment at the top of the new/modified spec file(s) explaining their purpose.

Related context

Related issue: #1382 "Add test coverage for save_trigger before_save execution and exception semantics".
Both stem from investigating whether save trigger failures during REDCap pulls roll back the
individual record or the entire pull (they currently do neither correctly — the record's own
persistence is unaffected by after_commit-phase failures, but the entire pull is aborted
regardless of which phase raised).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions