Fix duplicate Needs Review rows for equivalent URL spellings - #19
Merged
rootdr-backup merged 2 commits intoSep 9, 2026
Merged
Conversation
NormalizeURL now lowercases host, strips default ports (:443 for https, :80 for http), folds a single www prefix, and trims trailing slash and fragment noise. Candidate fingerprints and correlation keys match for these variants. The vuln_findings projection also stores the canonical URL so the unique constraint on (target, type, url, parameter) dedups instead of creating a second row. Raw evidence is untouched.
rootdr-backup
force-pushed
the
fix/url-normalization-dedup
branch
from
September 9, 2026 03:03
03d27a5 to
e393273
Compare
Owner
|
Thanks for tracing the duplicate projection path. I rebased this on current main and tightened the normalization before merge: default ports, host/scheme case and fragments are canonicalized, while www/apex, trailing slash and query-key case remain separate because they can represent different origins or handlers. Projection now reuses the authoritative candidate URL, so dedup no longer turns evidence URLs into non-actionable endpoint templates. Full scanner/API/database tests, vet and targeted race tests pass. |
Deduplicate transport-equivalent endpoint spellings through candidate identity while keeping actionable evidence URLs. Preserve www/apex, trailing-slash, query-key case, scheme, and non-default-port boundaries to avoid false merges.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem\nNeeds Review could project multiple rows when the same candidate was observed through transport-equivalent URL spellings, such as host case differences, an explicit default port, fragment changes, or different values for the same query keys.\n\n## Fix\n- Normalize scheme and hostname case.\n- Strip only the scheme's default port and drop fragments.\n- Keep the first observed raw URL as the actionable evidence URL.\n- Reload the authoritative candidate after fingerprint upsert so repeated spellings project through the same candidate identity.\n- Preserve security-relevant boundaries: www vs apex, trailing slash, query-key case, scheme, subdomain, and non-default port remain distinct unless future response evidence proves equivalence.\n\n## Verification\n- Regression coverage for equivalent host/default-port/fragment spellings.\n- Regression coverage preventing false merges across www/apex, trailing slashes, query-key case, schemes, subdomains, and non-default ports.\n- go test ./internal/scanner ./internal/api ./internal/database\n- go vet ./internal/scanner ./internal/api ./internal/database\n- targeted race tests pass.\n\nExisting legacy duplicate rows are not destructively backfilled; rescans use the corrected identity path.