Skip to content

Fix duplicate Needs Review rows for equivalent URL spellings - #19

Merged
rootdr-backup merged 2 commits into
rootdr-backup:mainfrom
Boreas37:fix/url-normalization-dedup
Sep 9, 2026
Merged

Fix duplicate Needs Review rows for equivalent URL spellings#19
rootdr-backup merged 2 commits into
rootdr-backup:mainfrom
Boreas37:fix/url-normalization-dedup

Conversation

@Boreas37

@Boreas37 Boreas37 commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Problem\nNeeds Review could project multiple rows when the same candidate was observed through transport-equivalent URL spellings, such as host case differences, an explicit default port, fragment changes, or different values for the same query keys.\n\n## Fix\n- Normalize scheme and hostname case.\n- Strip only the scheme's default port and drop fragments.\n- Keep the first observed raw URL as the actionable evidence URL.\n- Reload the authoritative candidate after fingerprint upsert so repeated spellings project through the same candidate identity.\n- Preserve security-relevant boundaries: www vs apex, trailing slash, query-key case, scheme, subdomain, and non-default port remain distinct unless future response evidence proves equivalence.\n\n## Verification\n- Regression coverage for equivalent host/default-port/fragment spellings.\n- Regression coverage preventing false merges across www/apex, trailing slashes, query-key case, schemes, subdomains, and non-default ports.\n- go test ./internal/scanner ./internal/api ./internal/database\n- go vet ./internal/scanner ./internal/api ./internal/database\n- targeted race tests pass.\n\nExisting legacy duplicate rows are not destructively backfilled; rescans use the corrected identity path.

NormalizeURL now lowercases host, strips default ports (:443 for https, :80 for http), folds a single www prefix, and trims trailing slash and fragment noise. Candidate fingerprints and correlation keys match for these variants.

The vuln_findings projection also stores the canonical URL so the unique constraint on (target, type, url, parameter) dedups instead of creating a second row. Raw evidence is untouched.
@rootdr-backup
rootdr-backup force-pushed the fix/url-normalization-dedup branch from 03d27a5 to e393273 Compare September 9, 2026 03:03
@rootdr-backup rootdr-backup changed the title Fix duplicate Needs Review rows for same URL with www or default port Fix duplicate Needs Review rows for equivalent URL spellings Sep 9, 2026
@rootdr-backup

Copy link
Copy Markdown
Owner

Thanks for tracing the duplicate projection path. I rebased this on current main and tightened the normalization before merge: default ports, host/scheme case and fragments are canonicalized, while www/apex, trailing slash and query-key case remain separate because they can represent different origins or handlers. Projection now reuses the authoritative candidate URL, so dedup no longer turns evidence URLs into non-actionable endpoint templates. Full scanner/API/database tests, vet and targeted race tests pass.

Deduplicate transport-equivalent endpoint spellings through candidate identity while keeping actionable evidence URLs. Preserve www/apex, trailing-slash, query-key case, scheme, and non-default-port boundaries to avoid false merges.
@rootdr-backup
rootdr-backup merged commit d76edb9 into rootdr-backup:main Sep 9, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants