GitHub target: reduce memory consumption by streaming uploads/downloads

[rhcarvalho]

The GitHub target should never need to hold full release assets (nor any other potentially large files) in memory.

This issue tracks replacing code that reads full files from disk, downloads arbitrary files from the internet and calculate checksums. All of those steps can be performed with fixed size buffers, consuming less resources and eliminating a failure mode of running out of memory.

See discussion in #328 (comment).

Note: this might require changes to how we use the Octokit library.

---

Some of the candidates

Reading full files of arbitrary size to memory where we could possibly be stream uploading / calculating checksum:

craft/src/targets/github.ts

Line 365 in 244efd8

const file = readFileSync(path);

Calculating MD5 hashes from in-memory bytes where we could be updating the checksum chunk-by-chunk:

craft/src/targets/github.ts

Lines 451 to 453 in 244efd8

	private checksumFromData(data: BinaryLike): string {
	return createHash('md5').update(data).digest('hex');
	}

Downloading arbitrarily large assets from the Internet (into response.data) where we could read chunks and update a checksum:

craft/src/targets/github.ts

Line 438 in 244efd8

response = await this.github.request(`GET ${url}`, {

[rhcarvalho]

I've put down in the issue description some of the lines of code that I've recently noticed implicitly assuming small release assets.

A more detailed review of the GitHub target might reveal more places where the assumption is made.

A similar problem might exist in other targets.

As far as I know, this problem has not manifested yet in any release of our projects.

[rhcarvalho]

See also

• Downloading a tarball of a migration archive octokit/octokit.js#916 (comment)
• How to use streaming in createfile and updatefile octokit/octokit.js#887 (comment)
• And don't forget the conversation in feat(target-gh): Check asset MD5 hash by downloading the asset #328 (comment)