Skip to content

fix(cli): use the resolved token and the real API in pysus dadosgov download - #355

Open
devgtv wants to merge 1 commit into
AlertaDengue:mainfrom
devgtv:fix/cli-dadosgov-token-and-list
Open

devgtv wants to merge 1 commit into
AlertaDengue:mainfrom
devgtv:fix/cli-dadosgov-token-and-list

Conversation

@devgtv

@devgtv devgtv commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

Summary

pysus dadosgov download failed on every invocation, for three separate reasons.

1. The resolved token was discarded

_get_token(token)               # resolves --token or DADOSGOV_TOKEN
...
await dg.connect(token=token)   # still the raw option, i.e. usually None

connect() falls back to a previously stored token and raises AuthenticationError when there is none. The documented workflow therefore died with A token is required to connect to DadosGov even though the variable was set.

Verified directly against the old code — with DADOSGOV_TOKEN=env_token it called connect(token=None).

2. Dataset.get_files() does not exist

Dataset is a BaseRemoteDataset (pysus/api/dadosgov/models.py:359); its files are reached through the content/search contract that recurses into groups. get_files appears nowhere in the codebase, so the command died with AttributeError: 'SINAN' object has no attribute 'get_files'. Replaced with await target.search().

3. download() was handed the output directory

await dg.download(f, out_dir)   # client does open(output, "wb")

The client opens its second argument as a file, so this raises IsADirectoryError as soon as a real transfer starts. The path is now out_dir / basename, taken with PurePosixPath since dados.gov.br paths always use forward slashes.

Filters also compare against str(f.path), matching the FTP CLI.

Impact

The command is documented in the CLI help and the README but has never worked: it either raises AuthenticationError, AttributeError, or — once both are fixed — IsADirectoryError.

Tests

The existing test_download_bad_slug set the env var but asserted only on the slug lookup, so execution never reached the client. Added 5 tests with a fake async client:

  • DADOSGOV_TOKEN is forwarded to connect() (token='env_token')
  • --token takes precedence over the env var
  • transfers target real files inside out_dir, not the directory itself
  • the --year filter reaches the download stage
  • no-match reports "No files match", skips transfers, and still closes the client

All 5 fail against the previous code. Confirmed the token defect independently by driving the old command with a stubbed client: connect(token=None) before, connect(token='env_token') after.

  • 1738 passed, 6 skipped
  • black and isort clean

…download`

Three defects made the command fail on every invocation.

The resolved token was thrown away:

    _get_token(token)          # resolves --token or DADOSGOV_TOKEN
    ...
    await dg.connect(token=token)   # still the raw option, usually None

`connect()` falls back to a previously stored token and raises
`AuthenticationError` when there is none, so the documented
`export DADOSGOV_TOKEN=...` workflow died with "A token is required to
connect to DadosGov" even though the variable was set. Verified: with
DADOSGOV_TOKEN=env_token the old code called `connect(token=None)`.

`Dataset.get_files()` does not exist. `Dataset` is a
`BaseRemoteDataset`, whose files are reached through the
`content`/`search` contract that recurses into groups; `get_files`
appears nowhere in the codebase. Replaced with `await target.search()`.

`download()` was handed the output *directory*:

    await dg.download(f, out_dir)

The client opens its second argument with `open(output, "wb")`, so this
only fails once a file is actually transferred (`IsADirectoryError`).
The path is now `out_dir / basename`, taken from the URL with
`PurePosixPath` since dados.gov.br paths always use forward slashes.

Filters also compare against `str(f.path)`, matching the FTP CLI.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant