fix(dadosgov): read the total size from Content-Range on ranged GETs - #352
Merged
luabida merged 1 commit intoOct 2, 2026
Merged
Conversation
All three size probes fall back to `GET` with `Range: bytes=0-0` when a server rejects HEAD. That response is a 206 Partial Content: its `Content-Length` describes the single requested byte, so it is always `1`. The real total is only reported by `Content-Range: bytes 0-0/<total>`, which the code never looked at. Consequence: `Recurso.get_size`, `File.fetch_metadata` and `File.fetch_size` all returned 1 for every resource served by a host that rejects HEAD, and the catalog then persisted size=1. Since the sync path compares that value to decide whether to re-download, every subsequent run saw a size change and re-fetched unchanged files indefinitely. Added a shared `_remote_size` helper that prefers the total from Content-Range, falls back to Content-Length when there is no Content-Range (plain HEAD), and handles an unsatisfiable `bytes 0-0/*` by falling back rather than raising. All three call sites use it.
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #352 +/- ##
=======================================
Coverage ? 97.22%
=======================================
Files ? 180
Lines ? 22771
Branches ? 0
=======================================
Hits ? 22139
Misses ? 632
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
🎉 This PR is included in version 2.11.6 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
All three remote-size probes in the dados.gov.br client fall back to a ranged GET when a server rejects HEAD:
That response is a
206 Partial Content, whoseContent-Lengthdescribes only the single requested byte — it is always1. The real total is reported byContent-Range: bytes 0-0/<total>, which was never read.Impact
Recurso.get_size,File.fetch_metadataandFile.fetch_sizeall returned1for every resource on a host that rejects HEAD, and the catalog persistedsize=1. Since the sync engine compares that stored value to decide whether to re-download, every subsequent run saw a size change and re-fetched unchanged files indefinitely — the sync never converged.Fix
A shared
_remote_size(headers)helper now:Content-Range(bytes 0-0/12345→12345)Content-Lengthwhen there is noContent-Range(a plain HEAD)bytes 0-0/*by falling back instead of raisingAll three call sites route through it, and the docstrings now state that the ranged response's total comes from
Content-Range.Tests
6 tests added across
test_client.pyandtest_models.py:ranged GET with
Content-Range→ total (the regression: previously1)HEAD carrying a
Content-Range→ total wins overContent-Lengthunsatisfiable range
bytes 0-0/*→ falls back toContent-Lengthno headers at all →
0the same for
fetch_metadataandfetch_size, assertingrecord.api_sizeis updated too1739 passed, 6 skippedblackandisortclean