Skip to content

sync: production diff/push after 'sync pull --branch' flags the whole orphaned main/ tree as added (mass-duplicate risk) #649

Description

@padak

Summary

After sync pull --branch <dev> re-targets manifest.configurations to the dev branch, the default-branch tree (main/) becomes orphaned on disk. A subsequent production sync diff / sync push then classifies every config in main/ as added with an empty config_id — one sync push away from duplicating the entire production project.

This is the same failure class as #482, but on the axis #482's fix does not cover: #482 scoped the untracked-file scan to the source branch subtree, which protects a dev-branch push from orphaned trees. The production push reads main/ as its legitimate source — and main/ is exactly the tree that pull --branch just orphaned.

Reproduced live (v0.88.0, project 5946)

sync init + sync pull                  # 14 configs tracked on main
branch create "issue644-b"             # -> id 51406
sync pull --branch 51406               # materializes issue644-b/ subtree (17 configs)
                                       # AND re-targets ALL manifest entries to branch 51406
branch reset                           # back to production
sync diff                              # target: production

Result:

summary: {'added': 19, 'modified': 0, ..., 'unchanged': 15, 'remote_only': 0}
  added | 01m0nnjyyv1szn29yjmsnh06y1 | ZZ ... Exp C      <- dev-only config, tracked on dev, diffed against production
  added | 01m0nnnx4efmnqq3ktdk064901 | ZZ ... Exp B      <- dev-only config, tracked on dev, diffed against production
  added | (empty) | e2e-pr1-e2e-1784722173               <- ordinary production config, file in main/ WITH correct id
  added | (empty) | ... (14 more, the whole main/ tree)

sync push --dry-run confirms these would all be POSTed as new configurations. 34-duplicate incidents of the #644 shape are one plausible outcome; this one is worse — it needs zero scaffolds, just the documented multi-branch pull.

Root cause (two independent legs)

  1. Untracked-scan leg. The orphaned main/ files carry correct _keboola.config_id values, and those ids resolve in the production remote — but the adopt-by-id guard (issue sync push creates duplicate configs for every config a dev branch inherits from main (even with zero local changes) #482, sync_service.py diff()) requires untracked_key not in tracked_keys. tracked_keys is keyed "{component_id}/{config_id}" with no branch dimension, and after the re-target every one of those ids is "claimed" by a manifest entry sitting on the dev branch. The guard concludes fork-by-copy and keeps the create: added with empty id.

  2. Tracked-entry leg. diff() builds the local side from all manifest.configurations regardless of the target branch. Entries tracked on the dev branch whose configs exist only there (created on the branch) diff against the production remote as added with an id — push would then create production copies of dev-only configs.

Suggested fix (surgical, preserves #482/#497 and KFR-07 semantics)

  • Untracked scan: when an untracked file's _keboola.config_id is claimed by a manifest entry whose branch_id differs from the diff/push target branch, classify it as an orphan of the other branch: exclude it from added and surface it in a new orphaned warning bucket (summary counter + details) telling the user to sync pull or delete the stale tree. A claim by a same-branch entry keeps today's fork-by-copy behaviour (still added).
  • Tracked entries: an entry whose branch_id differs from the target branch participates in the diff only if its id resolves on the target branch (this keeps the KFR-07 "promote default tree to dev" flow and the inherited-config comparisons working). If the id does not resolve on the target, exclude it with a warning ("tracked on branch X; use branch merge to promote") instead of reporting added.
  • push must honour the same classification (it consumes the diff's change list).

Acceptance

Replaying the repro above: production sync diff after sync pull --branch reports added: 0, unchanged: 14, plus warnings for the orphaned/dev-only entries; production sync push --dry-run plans no creates. Dev-branch diff/push behaviour unchanged. Existing #482/#497 regression tests stay green.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions