Skip to content

fix(dadosgov): read the total size from Content-Range on ranged GETs - #352

Merged
luabida merged 1 commit into
AlertaDengue:mainfrom
devgtv:fix/dadosgov-size-range-content-length
Oct 2, 2026
Merged

luabida merged 1 commit into
AlertaDengue:mainfrom
devgtv:fix/dadosgov-size-range-content-length

Conversation

@devgtv

@devgtv devgtv commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

Summary

All three remote-size probes in the dados.gov.br client fall back to a ranged GET when a server rejects HEAD:

response = await client.head(self.url)
if response.status_code == 405:
    response = await client.get(self.url, headers={"Range": "bytes=0-0"})
size = response.headers.get("Content-Length")
return int(size) if size else 0

That response is a 206 Partial Content, whose Content-Length describes only the single requested byte — it is always 1. The real total is reported by Content-Range: bytes 0-0/<total>, which was never read.

Impact

Recurso.get_size, File.fetch_metadata and File.fetch_size all returned 1 for every resource on a host that rejects HEAD, and the catalog persisted size=1. Since the sync engine compares that stored value to decide whether to re-download, every subsequent run saw a size change and re-fetched unchanged files indefinitely — the sync never converged.

Fix

A shared _remote_size(headers) helper now:

  • prefers the total from Content-Range (bytes 0-0/12345 → 12345)
  • falls back to Content-Length when there is no Content-Range (a plain HEAD)
  • handles an unsatisfiable bytes 0-0/* by falling back instead of raising

All three call sites route through it, and the docstrings now state that the ranged response's total comes from Content-Range.

Tests

6 tests added across test_client.py and test_models.py:

  • ranged GET with Content-Range → total (the regression: previously 1)

  • HEAD carrying a Content-Range → total wins over Content-Length

  • unsatisfiable range bytes 0-0/* → falls back to Content-Length

  • no headers at all → 0

  • the same for fetch_metadata and fetch_size, asserting record.api_size is updated too

  • 1739 passed, 6 skipped

  • black and isort clean

All three size probes fall back to `GET` with `Range: bytes=0-0` when a
server rejects HEAD. That response is a 206 Partial Content: its
`Content-Length` describes the single requested byte, so it is always
`1`. The real total is only reported by `Content-Range: bytes 0-0/<total>`,
which the code never looked at.

Consequence: `Recurso.get_size`, `File.fetch_metadata` and
`File.fetch_size` all returned 1 for every resource served by a host that
rejects HEAD, and the catalog then persisted size=1. Since the sync path
compares that value to decide whether to re-download, every subsequent
run saw a size change and re-fetched unchanged files indefinitely.

Added a shared `_remote_size` helper that prefers the total from
Content-Range, falls back to Content-Length when there is no Content-Range
(plain HEAD), and handles an unsatisfiable `bytes 0-0/*` by falling back
rather than raising. All three call sites use it.
@codecov-commenter

codecov-commenter commented Oct 2, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@24f736b). Learn more about missing BASE report.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #352   +/-   ##
=======================================
  Coverage        ?   97.22%           
=======================================
  Files           ?      180           
  Lines           ?    22771           
  Branches        ?        0           
=======================================
  Hits            ?    22139           
  Misses          ?      632           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@luabida
luabida merged commit 0f8eff3 into AlertaDengue:main Oct 2, 2026
15 checks passed
@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown

🎉 This PR is included in version 2.11.6 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants