Added continue_on_record_error option for REDCap pulls - fixes #1383 - #1398
Merged
Merged
Conversation
philayres
force-pushed
the
continue-on-record-error-1383
branch
from
September 3, 2026 09:42
7802a92 to
a1da364
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new
continue_on_record_errordata option toRedcap::ProjectAdmin(issue #1383). When enabled, an exception raised while persisting or triggering a single record during a REDCap pull (for example abefore_saveorafter_commitsave trigger) is caught and recorded inerrors, allowing the pull to continue processing the remaining records instead of aborting the entire run. Default behaviour (option absent/disabled) is unchanged: any unhandled exception aborts the entire pull (fail-fast).Also adds a distinct "completed with errors" status so a run that completes but recorded errors is no longer indistinguishable from a fully clean run.
Details
Redcap::DataRecords#storenow rescuesStandardErrorper record whencontinue_on_record_erroris enabled (via a shared#create_or_update_continuingwrapper, also used by#disable_deleted_recordsso a failing trigger while disabling a deleted record is handled the same way), recording the failure via#record_store_errorand continuing to the next record.#record_store_errorlogs the full exception/backtrace viaRails.logger, but only persists a truncated class+message string inerrors(MaxRecordErrorMessageLength), since raw exception text can include bound SQL parameter values. OnceMaxRecordErrorsRecorded(100) is reached, further per-record failures are only counted (tracked independently of unrelated pre-existing error sources) to bound the size of the job request result; a summary entry is appended once storing completes.#create_or_updateis split intoupdate_existing_record/create_new_record/persist_record/track_upsertto keep the rescue handling within Rubocop'sMetrics/MethodLength/Metrics/BlockNestinglimits. Whether a failed record is still counted as an upsert depends on#saved_changes?:before_save-phase trigger failure rolls back the record's transaction, so it is correctly excluded.after_commit-phase trigger failure means the record was already committed, so it is still counted even though the failing trigger's own action did not complete.#safe_record_identifiersfalls back to the rawrecord_id_field/extra identifier fields if#record_identifiersitself raises (e.g. a misconfigured data dictionary), so the failing record can still be identified inerrors; guaranteed not to raise even if the fallback fields themselves are unavailable.continue_on_record_erroris normalized to a strict boolean at read time so a blank/''value (a valid, disabled configuration perValidContinueOnRecordErrorValues) is never mistaken for "enabled".ValidContinueOnRecordErrorValuesvalidation (nil,'',true,false) onRedcap::ProjectAdmin#data_options.manual_run_completed_with_errors/scheduled_run_completed_with_errorsstatuses andRedcap::ProjectAdmin.completed_status, used byCaptureRecordsJob/RecurringPullTaskto distinguish a clean run from one whereRedcap::DataRecords#errorswas non-empty (from any source, not just this option). Not treated as afailed?status, so it does not trigger the admin failure-alerts panel.docs/admin_reference/project_admins/detailed_options.mdand in theRedcap::ProjectAdmindata_options doc comment.Testing
spec/models/redcap/data_records_continue_on_error_spec.rb(13 examples): default fail-fast behaviour, blank-value handling, option validation, error recording with failing record id, error count capping (including that unrelated pre-existing errors don't consume the cap),safe_record_identifiersfallback,before_save/after_committrigger failures for both record creation and updates, and continuation when disabling a deleted record raises.spec/models/redcap/project_admin_spec.rb: new tests forcompleted_with_errors?/completed_status.spec/jobs/redcap/capture_records_job_spec.rb: new integration tests verifying the job sets the correct status based on recorded errors.spec/models/redcap/full suite (205 examples) passes.Fixes #1383