feat!: cache tf provider mirror via ctx.download

[dominics]

Background

The mirror was built by shelling out to TF_PLUGIN_CACHE_DIR=... terraform init inside the toolchain repo rule (or previously by using terraform providers mirror). Those downloads flow through the terraform binary, not ctx.download*, so --repository_cache structurally cannot cache them. On an ephemeral runner where the external repo dir isn't persisted, every job re-downloads the full provider set from the registry. In core-infra that is 336 MiB across 11 providers on each of ~62 parallel plan jobs: roughly 20 GiB of registry traffic and 682 separate provider installs per plan run, every one of them a chance for a transient reset to fail a job.

There is a second problem underneath that one. main resolves nothing: mirror entries must be exact pins, because the only thing that could resolve a constraint was the terraform init subprocess. And because that subprocess ran inside a repo rule, whatever it worked out was invisible to everything outside. Terraform's own answer to this is .terraform.lock.hcl. This ruleset had no equivalent.

What

Populate the provider mirror via ctx.download instead of a terraform init subprocess, so provider bytes are content-addressed and served by Bazel's --repository_cache across runs.

Resolve the manifest in the module extension rather than in the download repo, so that what it resolved to is recorded in MODULE.bazel.lock and mirror entries can be version constraints rather than only exact pins.

Replacing terraform's provider installer means reimplementing the parts of it we depend on: service discovery, private registry auth, constraint resolution, and signature-derived hash verification. The first commit does the swap; the rest restore what the swap dropped.

Measured impact

Rolled out to core-infra in rillanetwork/core-infra#141: ~62 parallel plan jobs, 11 providers, 336 MiB of packages per platform, against the warm --repository_cache that setup-bazel's repository-cache: true restores. Cache hits were confirmed on the exact key rather than the prefix fallback.

	Terraform init median	job median
this branch, exact-key cache hit (2 runs)	47.0s, 47.0s	85s, 85s
main at v1.0.1 (6 runs)	49.0 - 56.5s	86 - 91s

Both warm runs reproduce 47.0s and fall below the whole baseline band; means are 49.9s and 51.1s against a baseline 54.4 - 59.0s. That is about 10% off the init step, under 5% off the job, and roughly 4 minutes of aggregate runner time per plan run. Wall clock does not move: the matrix is fully parallel, so run duration is set by the slowest job rather than the sum.

The speed change is real but minor, and it is not the reason to do this. The reason is that 682 public-registry fetches per plan run collapse into a single ~5s intra-datacentre cache restore, and the transient-reset exposure goes with them. Read the caching as a reliability change that happens to be slightly faster.

The cache cost is modest. In core-infra the repository-cache entry measures 342 MiB, and the repo sits at 0.92 GiB of GitHub's 10 GB per-repo ceiling with two such entries live, so eviction pressure is not a practical concern: one entry exists per distinct MODULE.bazel hash, and unused entries age out after 7 days long before ~29 of them could accumulate. Consumers with a much busier cache should still check, since an evicted entry reverts the gain to the cold path.

How

For each (source, version), in the module extension:

1. Locate the host's provider API. The two default hosts come from a built-in table; any other is resolved through https://<host>/.well-known/terraform.json.
2. For a constraint, GET the published version list and select the highest match.
3. GET the registry download endpoint, which returns download_url and sha256 shasum in one call.

The resulting concrete coordinates are passed to the download repo as the providers_json attribute. The repo rule then does only step 4, and reaches no registry at all:

4. ctx.download(url, sha256=...) then ctx.extract into the unpacked filesystem-mirror layout mirror/<host>/<ns>/<type>/<version>/<os>_<arch> that downstream terraform init -plugin-dir=<mirror> already consumes.

Steps 2, 3 and 4 are each batched across the whole manifest before anything is awaited, so N providers cost two rounds of latency rather than 2N. (Only ctx.download accepts block = False; download_and_extract does not, hence the separate extract. The zip is staged in the repo dir and deleted as soon as it is unpacked, and sha256 is still passed so the bytes remain cache-served.)

Downstream init runs fully offline; no downstream change needed. Terraform resolves from registry.terraform.io, tofu from registry.opentofu.org.

What lands in MODULE.bazel.lock

Doing resolution in the extension rather than the repo rule is what makes the result visible to bzlmod, and it is the point of the design. No new lock file is introduced; this is the lockfile every Bazel repo already has.

The concrete coordinates, as attributes of the generated download repo, so generatedRepoSpecs states outright which version each constraint selected:

"attributes": {
  "version": "1.9.5", "os": "linux", "arch": "amd64",
  "providers_json": "[{\"source\":\"hashicorp/random\",\"version\":\"3.1.3\", ...}]"
}

The registry's answers, as extension facts:

"facts": {
  "@@rules_tf+//tf:extensions.bzl%tf_repositories": {
    "resolve/registry.terraform.io/hashicorp/random/~> 3.1.0": {"version": "3.1.3"},
    "package/registry.terraform.io/hashicorp/random/3.1.3/linux_amd64": {
      "download_url": "https://releases.hashicorp.com/...linux_amd64.zip",
      "sha256": "bcf7806b..."
    },
    "verified/registry.terraform.io/hashicorp/random/3.1.3": {"zh": ["bcf7806b...", "..."]}
  }
}

A resolve/ fact answers "which version does this constraint select"; a package/ fact answers "where does this version live and what does it hash to"; a verified/ fact answers "which hashes for this version survived a signature check", and is written by the lock target below rather than by resolution. Together they are everything the extension would otherwise ask the registry, so it declares reproducible = True: a second evaluation defines exactly the same repos with no network access, and a warm --repository_cache then serves the whole mirror offline. Being reproducible, the extension does not appear in the lockfile's moduleExtensions section at all; facts are persisted separately.

Two module_ctx constraints shaped the implementation:

• module_ctx.facts is a lookup exposing only get, not an iterable, and facts are keyed per extension rather than per platform. Coordinates for the other platforms are therefore carried forward by asking for each key by name, over the four platforms in MIRROR_PLATFORMS. Building on each host in turn accumulates one lockfile covering them all, which is the property terraform gets from a multi-platform .terraform.lock.hcl.
• module_ctx has no delete(), so the small metadata JSONs are left in the extension's working directory rather than in any repo.

Version constraints

Mirror entries currently must be exact pins. They may now also be a constraint (~> 3.1.0, >= 4.0.0, < 4.0.5, comma-joined as AND), resolved against the registry's published version list, selecting the highest match. Exact pins gain full semver, so -prerelease and +build suffixes are accepted. Prereleases are never selected by a constraint, matching terraform; mirroring one needs an exact pin.

A constraint does not let mirror contents drift between builds, because the resolve/ fact holds the selection. Re-resolve deliberately with bazel mod deps --lockfile_mode=refresh. Pinning is still the clearer form, since the manifest then says outright what the mirror holds.

Breaking change: requires Bazel 9

module_ctx.facts does not exist on Bazel 8 (verified against 8.4.2, which fails with 'module_ctx' value has no field or method 'facts'). 9.0.0 supports the full cycle: facts are written to MODULE.bazel.lock and read back on a cold output base. So bazel_compatibility = [">=9.0.0"], the README says Bazel 9, and the CI matrix drops its 8.x row. The repo's own .bazelversion stays at 9.1.0, unchanged from main.

Fact keys deliberately carry no schema version. module_extension(facts_version = ...) would supply one, but it requires Bazel 9.2 and nothing else here needs a release that recent. Changing the shape of a key or its value later means adopting facts_version and raising the floor to 9.2 at that point; there is a comment next to the key builders so that cost is visible where the change would be made.

mirror, mirror_json and every existing manifest are otherwise unaffected: anything that worked on main still works unchanged.

Signature verification (opt-in)

terraform init verified the registry's SHA256SUMS signature against public keys compiled into the terraform binary. The binary carries both "HashiCorp Security" and "HashiCorp Security (Terraform Partner Signing)", so for hashicorp/* and partner providers that verification had a trust root independent of the registry's response. Fetching via ctx.download against a registry-reported shasum does not: an attacker who could compromise the registry or forge TLS to it could substitute a package unnoticed. (Community providers were always registry-anchored either way, since terraform takes their signing key from the registry response.) Losing this is the one real regression in swapping out the subprocess, and it is worth being explicit about.

Starlark has no crypto primitives, so the check cannot be reproduced here. It can be imported. terraform providers lock performs the full verification and records the resulting package hashes as zh: entries, and a zh: value is exactly the sha256 of a release zip. Check every package against those and the hash it is admitted on traces back to a signature rather than to the registry's word. There are two ways to supply them.

tf_providers_lock, a bazel run target, runs the lock command over the resolved manifest and merges what it verified into the facts section of MODULE.bazel.lock, keyed verified/<host>/<ns>/<type>/<version>:

tf_providers_lock(name = "providers_lock")

bazel run //:providers_lock              # re-run when the manifest changes, commit the diff
bazel run //:providers_lock -- --check   # CI gate: fails if the recorded hashes are not current

The hashes land in the same file the coordinates already live in, so nothing new has to be maintained alongside the manifest. The extension reads them back through ctx.facts and checks every package before a byte is fetched. Two details of the design:

• The manifest comes from the resolved toolchain, not from the tag, so a constraint is locked against the version the extension selected rather than against the text it was written as.
• Facts must be re-emitted by name to survive, since an extension's facts are replaced wholesale by what it returns. That is also what prunes them: a provider dropped from the manifest takes its hashes with it.

provider_locks still takes .terraform.lock.hcl files directly, for a repository that already generates one for its own terraform workflow. The two sources are merged, so they can be combined.

provider_locks_strict = True now means "every mirror entry must have a verified hash", from either source; uncovered entries warn otherwise.

Under use_tofu = True the target runs tofu providers lock against registry.opentofu.org. Not interchangeable: verified that for random@3.3.2 the two registries' zh: sets have zero overlap, since OpenTofu repackages and re-signs.

One run of the lock command covers every platform. zh: hashes come from a single signed SHA256SUMS document, so -platform only decides which packages are downloaded for the h1: hashes this ignores -- confirmed by observing that one platform still yields all 12 zh: hashes. An earlier draft passed all four platforms and paid 4x the downloads for nothing.

Caveats, documented in docs/mirror.md:

• zh: hashes cover every platform and the lock does not record which is which, so the check is set membership. A registry-level attacker could still redirect one platform's URL to another platform's genuinely signed package. That is a broken build, not an avenue for unsigned code.
• A dependency lock holds one version per provider while a mirror may stock several, so the target runs the lock command once per version set (and provider_locks, used by hand, needs one file per set).
• terraform providers lock authenticates the way terraform does, which is a superset of what the extension can read (HCL .tfrc, credential helpers). A token only terraform can find will lock hashes for a mirror the extension then cannot fetch.

Known rough edges

• Enforcement depends on a CI flag. Under the default --lockfile_mode=update Bazel re-resolves and rewrites the lockfile when the manifest changes; only --lockfile_mode=error makes an unexpected change fail. A standalone lock file read through ctx.read() would have been enforced unconditionally. This is documented, but it is a convention consumers have to adopt rather than something the ruleset can guarantee.
• MODULE.bazel.lock is machine-managed, so a pin cannot be hand-edited or commented the way a dedicated lock file could be.
• Verified hashes are checked whenever a package is resolved, which is the case that matters, but editing the recorded hashes does not by itself re-trigger the check: facts are not an input Bazel invalidates the extension on. Confirmed by injection -- a tampered hash is caught on a cold resolution (a fresh checkout, or CI) and not before.
• Bootstrapping provider_locks_strict = True is ordered: with no hashes recorded yet the extension fails before the lock target can run, so hashes have to be recorded first. Recovering from a genuine mismatch means dropping the offending verified/ fact, re-running the target, and comparing what it writes.
• Only the host platform is resolved per run. Unchanged from the TF_PLUGIN_CACHE_DIR behaviour this replaces, and facts now accumulate across hosts.
• No .terraform.lock.hcl is produced for consumers; the mirror is host-scoped and verified per package.

Test plan

Five workspaces, all wired into CI:

• tests/bcr covers exact pins, a prerelease pin, two versions of one source, a bounded range resolving to tls@4.0.4, a pessimistic ~> 3.1.0 resolving to random@3.1.3 rather than 3.6.0, and a second constraint on an already-constrained source (which caught a real bug: two constraints on one provider raced two concurrent downloads onto the same output path, now fixed by fetching each version listing once).

• tests/bcr_mirror_json runs provider_locks_strict = True against a genuinely generated lock. Verified out of band that corrupting a single zh: hash fails the build naming the provider, and that removing a provider block fails the coverage check.

• tests/bcr_facts gives every entry a registry host that does not resolve, and makes one of them a constraint. It passes only because the facts committed in its MODULE.bazel.lock answer both the version question and the coordinates question, so discovery and metadata never happen. Removing the facts fails on service discovery, confirming the test is not passing by accident. Its lockfile is committed for exactly this reason, and the workspace pins its own .bazelversion: 9.1 and 9.2 disagree on how facts are represented on disk and each rejects the other's, so the fixture is only read by the release that wrote it.

• tests/bcr_tofu runs the same features against the tofu rule and registry.opentofu.org, including a constraint, and provider_locks_strict = True over its whole manifest against locks generated by tofu providers lock (two files, since the manifest stocks random at two versions). Previously nothing exercised use_tofu = True. This is not redundant with bcr_mirror_json: registry.opentofu.org serves artifacts repackaged from github.com/opentofu/terraform-provider-* and re-signed under OpenTofu's key, so for a given provider version its zh: hashes are disjoint from the terraform registry's -- a lock generated by the other tool would cover nothing the mirror fetched. Verified the same two ways as the terraform suite: corrupting a zh: hash fails naming the provider, and dropping the second lock file fails the coverage check on random@3.1.3.

• tests/bcr_verified is provider_locks_strict = True with no lock file anywhere: it passes only on the verified/ facts committed in its MODULE.bazel.lock, which //tf:providers_lock put there. Verified out of band, on a cold output base, that corrupting one zh: hash fails naming the provider and that deleting an entry fails the coverage check. CI runs the target with --check there, so the committed hashes must be the ones the lock command reproduces -- an admitted hash is one a signature was checked against, not one that was pasted in. Its lockfile is committed and its .bazelversion pinned for the same reason bcr_facts does both. CI also runs the target for real in tests/bcr_tofu, covering the tofu binary and its registry. The registry-free logic -- manifest parsing, the split into version sets, the merge -- has unit tests in //tf/rules:providers_lock_test.

Also verified: a cold resolution with MODULE.bazel.lock deleted regenerates the facts and factsVersions correctly; the extension is absent from moduleExtensions, as reproducible = True intends; the on-disk layout matches the old TF_PLUGIN_CACHE_DIR output (unpacked, binary directly in <os>_<arch>/); no .terraform, zip, or metadata artifacts leak into the repo; and service discovery works against the real .well-known document (checked by temporarily emptying the seeded host table).

[dominics]

Open for review, but still in the middle of testing this in other repos