Skip to content

Added continue_on_record_error option for REDCap pulls - fixes #1383 - #1398

Merged
philayres merged 1 commit into
consected:developfrom
hmsrc:continue-on-record-error-1383
Sep 3, 2026
Merged

Added continue_on_record_error option for REDCap pulls - fixes #1383#1398
philayres merged 1 commit into
consected:developfrom
hmsrc:continue-on-record-error-1383

Conversation

@philayres

@philayres philayres commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a new continue_on_record_error data option to Redcap::ProjectAdmin (issue #1383). When enabled, an exception raised while persisting or triggering a single record during a REDCap pull (for example a before_save or after_commit save trigger) is caught and recorded in errors, allowing the pull to continue processing the remaining records instead of aborting the entire run. Default behaviour (option absent/disabled) is unchanged: any unhandled exception aborts the entire pull (fail-fast).

Also adds a distinct "completed with errors" status so a run that completes but recorded errors is no longer indistinguishable from a fully clean run.

Details

  • Redcap::DataRecords#store now rescues StandardError per record when continue_on_record_error is enabled (via a shared #create_or_update_continuing wrapper, also used by #disable_deleted_records so a failing trigger while disabling a deleted record is handled the same way), recording the failure via #record_store_error and continuing to the next record.
  • #record_store_error logs the full exception/backtrace via Rails.logger, but only persists a truncated class+message string in errors (MaxRecordErrorMessageLength), since raw exception text can include bound SQL parameter values. Once MaxRecordErrorsRecorded (100) is reached, further per-record failures are only counted (tracked independently of unrelated pre-existing error sources) to bound the size of the job request result; a summary entry is appended once storing completes.
  • #create_or_update is split into update_existing_record/create_new_record/persist_record/track_upsert to keep the rescue handling within Rubocop's Metrics/MethodLength/Metrics/BlockNesting limits. Whether a failed record is still counted as an upsert depends on #saved_changes?:
    • A before_save-phase trigger failure rolls back the record's transaction, so it is correctly excluded.
    • An after_commit-phase trigger failure means the record was already committed, so it is still counted even though the failing trigger's own action did not complete.
  • #safe_record_identifiers falls back to the raw record_id_field/extra identifier fields if #record_identifiers itself raises (e.g. a misconfigured data dictionary), so the failing record can still be identified in errors; guaranteed not to raise even if the fallback fields themselves are unavailable.
  • continue_on_record_error is normalized to a strict boolean at read time so a blank/'' value (a valid, disabled configuration per ValidContinueOnRecordErrorValues) is never mistaken for "enabled".
  • Added ValidContinueOnRecordErrorValues validation (nil, '', true, false) on Redcap::ProjectAdmin#data_options.
  • Added manual_run_completed_with_errors/scheduled_run_completed_with_errors statuses and Redcap::ProjectAdmin.completed_status, used by CaptureRecordsJob/RecurringPullTask to distinguish a clean run from one where Redcap::DataRecords#errors was non-empty (from any source, not just this option). Not treated as a failed? status, so it does not trigger the admin failure-alerts panel.
  • Documented the new option in docs/admin_reference/project_admins/detailed_options.md and in the Redcap::ProjectAdmin data_options doc comment.

Testing

  • spec/models/redcap/data_records_continue_on_error_spec.rb (13 examples): default fail-fast behaviour, blank-value handling, option validation, error recording with failing record id, error count capping (including that unrelated pre-existing errors don't consume the cap), safe_record_identifiers fallback, before_save/after_commit trigger failures for both record creation and updates, and continuation when disabling a deleted record raises.
  • spec/models/redcap/project_admin_spec.rb: new tests for completed_with_errors?/completed_status.
  • spec/jobs/redcap/capture_records_job_spec.rb: new integration tests verifying the job sets the correct status based on recorded errors.
  • spec/models/redcap/ full suite (205 examples) passes.

Fixes #1383

@philayres
philayres force-pushed the continue-on-record-error-1383 branch from 7802a92 to a1da364 Compare September 3, 2026 09:42
@philayres
philayres merged commit b752915 into consected:develop Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow REDCap pulls to continue when an individual record's save trigger raises

1 participant