Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@ With DLoad, you can:
- [PHAR Tools Management](#phar-tools-management)
- [Frontend Asset Distribution](#frontend-asset-distribution)
- [API Rate Limits](#api-rate-limits)
- [Failure Reporting](#failure-reporting)
- [Gitlab CI configuration](#gitlab-ci-configuration)
- [Contributing](#contributing)

Expand Down Expand Up @@ -574,6 +575,31 @@ GITLAB_TOKEN=your_token_here ./vendor/bin/dload get

Add to CI/CD environment variables for automated downloads.

> [!NOTE]
> In GitHub Actions, `secrets.GITHUB_TOKEN` is scoped to the current repository and shares a limit of
> 1,000 requests per hour across all jobs of the repository. With a large job matrix the limit may run out,
> and downloads from other repositories may be rejected. Use a personal access token if that happens.

## Failure Reporting

`dload get` exits with a non-zero code when at least one requested package was not installed, and prints
the reason for every failed download: the API error (invalid token, exhausted rate limit, missing repository),
the number of matched releases, the assets each checked release contains, and the filters that rejected them.

```
Failed to download `rr`
Requested: version `any`, OS `linux`, architecture `amd64`, minimum stability `stable`, asset type `any`.
Tried 1 repository(ies):
1) github `roadrunner-server/roadrunner`
GitHub API rejected the credentials (HTTP 401: Bad credentials).
The API token from the GITHUB_TOKEN environment variable is invalid, expired or revoked. Provide a valid
token or unset the variable to use anonymous access.

1 of 1 download(s) failed.
```

Run with `-vvv` to also get stack traces and the full request log.

## Gitlab CI configuration

When you make a release in Gitlab, make sure to upload your assets to the release page via
Expand Down
4 changes: 4 additions & 0 deletions psalm-baseline.xml
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,8 @@
</file>
<file src="src/Module/Downloader/Downloader.php">
<InternalMethod>
<code><![CDATA[$context->release->getAssets()]]></code>
<code><![CDATA[$releases]]></code>
<code><![CDATA[limit]]></code>
<code><![CDATA[limit]]></code>
<code><![CDATA[toArray]]></code>
Expand All @@ -220,7 +222,9 @@
<code><![CDATA[$asset]]></code>
<code><![CDATA[$file]]></code>
<code><![CDATA[$release]]></code>
<code><![CDATA[$releaseAttempt]]></code>
<code><![CDATA[$repoConfig]]></code>
<code><![CDATA[$repositoryAttempt]]></code>
</PropertyNotSetInConstructor>
</file>
<file src="src/Module/Downloader/SoftwareCollection.php">
Expand Down
64 changes: 60 additions & 4 deletions src/Command/Get.php
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,12 @@
use Internal\DLoad\Module\Common\Stability;
use Internal\DLoad\Module\Config\Schema\Action\Download as DownloadConfig;
use Internal\DLoad\Module\Config\Schema\Actions;
use Internal\DLoad\Module\Downloader\Exception\DownloadFailed;
use Internal\DLoad\Service\Container;
use Symfony\Component\Console\Attribute\AsCommand;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Exception\InvalidArgumentException;
use Symfony\Component\Console\Formatter\OutputFormatter;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Input\InputOption;
Expand Down Expand Up @@ -81,7 +83,8 @@
* @param InputInterface $input Command input
* @param OutputInterface $output Command output
*
* @return int Command result code
* @return int `Command::SUCCESS` when every requested package is in place,
* `Command::FAILURE` when at least one download failed
*
* @throws \RuntimeException When no software is specified to download
*/
Expand All @@ -104,14 +107,43 @@

/** @var DLoad $dload */
$dload = $container->get(DLoad::class);
$forceDownload = $input->getOption('force');
$forceDownload = (bool) $input->getOption('force');

/** @var list<array{non-empty-string, \Throwable}> $failures */
$failures = [];
foreach ($actions as $action) {
$dload->addTask($action, $forceDownload);
try {
$dload->addTask($action, $forceDownload)->then(
null,
static function (\Throwable $e) use (&$failures, $action): void {
$failures[] = [$action->software, $e];
},
);
} catch (\Throwable $e) {
// A task may fail before it is even scheduled, e.g. when the software is unknown
$failures[] = [$action->software, $e];
}
}
$dload->run();

return Command::SUCCESS;
return $failures === []
? Command::SUCCESS
: $this->reportFailures($output, $failures, \count($actions));
}

/**
* Builds a user-facing explanation of a failure.
*/
private static function describeFailure(\Throwable $error): string
{
$message = $error->getMessage();

return match (true) {
// The report already describes the whole context of the failure
$error instanceof DownloadFailed => $error->report,
$message === '' => \sprintf('Unexpected %s without a message.', $error::class),
default => $message,
};
}

/**
Expand All @@ -138,12 +170,12 @@
$toDownload[$action->software] = $action;
}

$destinationPath = $input->getOption('path');

Check failure on line 173 in src/Command/Get.php

View workflow job for this annotation

GitHub Actions / psalm (ubuntu-latest, 8.2, locked)

MixedAssignment

src/Command/Get.php:173:9: MixedAssignment: Unable to determine the type that $destinationPath is being assigned to (see https://psalm.dev/032)

Check failure on line 173 in src/Command/Get.php

View workflow job for this annotation

GitHub Actions / psalm (ubuntu-latest, 8.2, locked)

MixedAssignment

src/Command/Get.php:173:9: MixedAssignment: Unable to determine the type that $destinationPath is being assigned to (see https://psalm.dev/032)

return \array_map(

Check failure on line 175 in src/Command/Get.php

View workflow job for this annotation

GitHub Actions / psalm (ubuntu-latest, 8.2, locked)

LessSpecificReturnStatement

src/Command/Get.php:175:16: LessSpecificReturnStatement: The type 'array<array-key, Internal\DLoad\Module\Config\Schema\Action\Download>' is more general than the declared return type 'list<Internal\DLoad\Module\Config\Schema\Action\Download>' for Internal\DLoad\Command\Get::getDownloadActions (see https://psalm.dev/129)

Check failure on line 175 in src/Command/Get.php

View workflow job for this annotation

GitHub Actions / psalm (ubuntu-latest, 8.2, locked)

LessSpecificReturnStatement

src/Command/Get.php:175:16: LessSpecificReturnStatement: The type 'array<array-key, Internal\DLoad\Module\Config\Schema\Action\Download>' is more general than the declared return type 'list<Internal\DLoad\Module\Config\Schema\Action\Download>' for Internal\DLoad\Command\Get::getDownloadActions (see https://psalm.dev/129)
static fn(string $software): DownloadConfig => $toDownload[$software] ?? self::parseSoftware(
$software,

Check failure on line 177 in src/Command/Get.php

View workflow job for this annotation

GitHub Actions / psalm (ubuntu-latest, 8.2, locked)

ArgumentTypeCoercion

src/Command/Get.php:177:17: ArgumentTypeCoercion: Argument 1 of Internal\DLoad\Command\Get::parseSoftware expects non-empty-string, but parent type string provided (see https://psalm.dev/193)

Check failure on line 177 in src/Command/Get.php

View workflow job for this annotation

GitHub Actions / psalm (ubuntu-latest, 8.2, locked)

ArgumentTypeCoercion

src/Command/Get.php:177:17: ArgumentTypeCoercion: Argument 1 of Internal\DLoad\Command\Get::parseSoftware expects non-empty-string, but parent type string provided (see https://psalm.dev/193)
$destinationPath,

Check failure on line 178 in src/Command/Get.php

View workflow job for this annotation

GitHub Actions / psalm (ubuntu-latest, 8.2, locked)

MixedArgument

src/Command/Get.php:178:17: MixedArgument: Argument 2 of Internal\DLoad\Command\Get::parseSoftware cannot be mixed, expecting null|string (see https://psalm.dev/030)

Check failure on line 178 in src/Command/Get.php

View workflow job for this annotation

GitHub Actions / psalm (ubuntu-latest, 8.2, locked)

MixedArgument

src/Command/Get.php:178:17: MixedArgument: Argument 2 of Internal\DLoad\Command\Get::parseSoftware cannot be mixed, expecting null|string (see https://psalm.dev/030)
),
(array) $input->getArgument(self::ARG_SOFTWARE),
);
Expand Down Expand Up @@ -201,4 +233,28 @@
"Unknown architecture: {$arch}.",
));
}

/**
* Prints the reason of every failed download and returns a failure exit code.
*
* @param list<array{non-empty-string, \Throwable}> $failures Software identifier with its failure
* @param int<1, max> $total Total number of requested downloads
*/
private function reportFailures(OutputInterface $output, array $failures, int $total): int
{
foreach ($failures as [$software, $error]) {
$output->writeln('');
$output->writeln(\sprintf('<error> Failed to download `%s` </error>', OutputFormatter::escape($software)));
$output->writeln(OutputFormatter::escape(self::describeFailure($error)));

$output->isVeryVerbose() and $this->logger->exception($error, important: true);
}

$output->writeln('');
$output->writeln(
\sprintf('<comment>%d of %d download(s) failed.</comment>', \count($failures), $total),
);

return Command::FAILURE;
}
}
29 changes: 29 additions & 0 deletions src/DLoad.php
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
use Internal\DLoad\Module\Config\Schema\Embed\File;
use Internal\DLoad\Module\Config\Schema\Embed\Software;
use Internal\DLoad\Module\Downloader\Downloader;
use Internal\DLoad\Module\Downloader\Exception\NothingExtracted;
use Internal\DLoad\Module\Downloader\SoftwareCollection;
use Internal\DLoad\Module\Downloader\Task\DownloadResult;
use Internal\DLoad\Module\Downloader\Task\DownloadTask;
Expand Down Expand Up @@ -234,11 +235,14 @@ private function prepareExtractTask(
$extractor = $archive->extract();
$this->logger->info('Extracting %s', $fileInfo->getFilename());
$binaryPattern = $this->generateBinaryExtractionConfig($software->binary);
$extractionRules = $this->describeExtractionRules($software, $binaryPattern);
$archiveFiles = [];

while ($extractor->valid()) {
$to = $rule = null;
$file = $extractor->current();
\assert($file instanceof \SplFileInfo);
$archiveFiles[] = $file->getFilename();

# Check if it's binary and should be extracted
$isBinary = false;
Expand Down Expand Up @@ -283,6 +287,13 @@ private function prepareExtractTask(
}
}

# A downloaded asset without a single matching file means nothing was installed
$resultFiles === [] and throw new NothingExtracted(
assetName: $fileInfo->getFilename(),
rules: $extractionRules,
files: $archiveFiles,
);

return new DloadResult($resultFiles, $resultBinary);
} finally {
// Cleanup: Delete the temporary downloaded file
Expand Down Expand Up @@ -336,6 +347,24 @@ private function getDestinationPath(DownloadConfig $action): Path
return Path::create($this->configDestination->path ?? $action->extractPath ?? (string) \getcwd());
}

/**
* Lists the patterns applied to archive entries, to explain why nothing was extracted.
*
* @param File|null $binaryPattern Generated binary extraction rule
* @return list<string>
*/
private function describeExtractionRules(Software $software, ?File $binaryPattern): array
{
$rules = [];
$binaryPattern === null or $rules[] = \sprintf('binary `%s`', $binaryPattern->pattern);

foreach ($software->files as $file) {
$rules[] = \sprintf('file `%s`', $file->pattern);
}

return $rules;
}

/**
* Generates a binary extraction configuration based on the provided binary configuration.
*
Expand Down
Loading
Loading