Enable one-click crates.io release for mxc-sdk by dhoehna · Pull Request #647 · microsoft/mxc

dhoehna · 2026-07-14T19:04:51Z

Summary

Adds a one-click Azure DevOps pipeline that publishes the mxc-sdk crate and its first-party dependency closure to crates.io, plus the crate-metadata changes required to make that closure publishable.

This branch contains two logical commits:

Make the mxc-sdk crate closure publishable (11 Cargo.toml files) — registry-agnostic metadata: workspace repository, per-crate description/license/repository, dual { path, version } on internal path deps, and making sandbox_spec publishable. Required to publish from anywhere.
Add the release pipeline (2 files) — .azure-pipelines/1ES.Crate.Release.yml and .azure-pipelines/scripts/publish_crates_to_cratesio.py.

How it works

Run pipeline with dryRun = true (the default) previews every crate via cargo package --list and publishes nothing.
With dryRun = false, a Publish stage waits for manual approval on the MXC-CratesIo-Production environment, then publishes leaf-first, waiting for index propagation between crates. Re-runs are idempotent (a version already on crates.io is skipped).

Secrets stay out of the repo

CARGO_REGISTRY_TOKEN comes from the ADO variable group MXC-CratesIo-Publish and is mapped to the publish step's env only. Nothing secret is committed.

One-time setup (not codeable in the repo)

Variable group, environment + approval, crates.io egress allowlist, crate-name reservation / OSPO sign-off, and registering the pipeline. See the checklist at the bottom of 1ES.Crate.Release.yml.

Microsoft Reviewers: Open in CodeFlow

Add crates.io-required metadata so mxc-sdk and its first-party dependency closure (10 crates) can be published: - workspace: add repository URL; give the 8 first-party path-deps in [workspace.dependencies] explicit versions (cargo publish requires a version on every path dependency). - wxc_common: version its optional `nanvix_common` path-dep. Although it is gated by the off-by-default `microvm` feature, cargo requires every dependency in a published manifest -- optional ones included -- to carry a version and exist on crates.io, so nanvix_common must ship too. It is a leaf (serde/serde_json only). - add description to wxc_common, mxc_pty, and the appcontainer / bubblewrap / lxc / seatbelt common crates. - add license + repository to mxc_telemetry; repository to nanvix_common. - inherit repository on each published crate. - make sandbox_spec publishable (remove publish = false). Validated with `cargo metadata` and per-crate `cargo package --list` (plus `cargo publish --dry-run --no-verify` for leaf crates) against real crates.io. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e5d2aa5b-7f04-4e4d-83d3-a02efe7020ab

New ADO pipeline publishes the mxc-sdk crate and its 10-crate first-party closure to crates.io from a single "Run pipeline" click, with a dryRun parameter (default true) that previews every crate before anything uploads. - .azure-pipelines/scripts/publish_crates_to_cratesio.py: leaf-first ordered publish (mxc_telemetry, nanvix_common, mxc_pty, sandbox_spec, wxc_common, lxc_common, seatbelt_common, appcontainer_common, bwrap_common, mxc-sdk), per-crate version resolved via `cargo metadata`, idempotent version-exists skip via the crates.io sparse index, `cargo publish --no-verify` with a post-publish index-propagation wait, and a --dry-run mode that runs `cargo package --list` (validates each manifest without needing first-party deps on the index yet). Modeled on scripts/seed_feed.py conventions. - .azure-pipelines/1ES.Crate.Release.yml: trigger:none, extends the 1ES Official template; DryRun stage always previews; Publish stage is compiled in only when dryRun=false and gates on manual approval via the MXC-CratesIo-Production environment. Reuses Rust.Toolchain.Public.yml and deliberately skips Cargo.Setup.* so cargo talks to the real crates.io. Secrets stay out of the repo: CARGO_REGISTRY_TOKEN comes from the MXC-CratesIo-Publish variable group and is mapped to env only in the publish step. One-time ADO/crates.io setup is documented in the pipeline header. Validated locally against real crates.io: the full --dry-run passes for all 10 crates; all 10 target names are currently unregistered on crates.io. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e5d2aa5b-7f04-4e4d-83d3-a02efe7020ab

Copilot

Pull request overview

This PR enables releasing the Rust mxc-sdk crate to crates.io by (1) making the crate’s first-party dependency closure publishable via Cargo metadata/manifest adjustments, and (2) adding an Azure DevOps “one-click” pipeline plus a helper script to publish the closure leaf-first.

Changes:

Add crates.io publish metadata to mxc-sdk and its first-party dependency closure (license/repository/description) and make sandbox_spec publishable.
Convert internal path dependencies used by the publishable closure to dual { path, version } so cargo publish can strip path and publish clean manifests.
Add an ADO pipeline (1ES.Crate.Release.yml) and a Python script to dry-run/package or publish the crate closure with index-propagation waits.

Reviewed changes

Copilot reviewed 12 out of 13 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`src/Cargo.toml`	Adds workspace `repository` and sets `{ path, version }` for publishable internal workspace deps.
`src/core/wxc_common/Cargo.toml`	Adds publish metadata and ensures optional `nanvix_common` path dep includes a version for publishing.
`src/core/mxc-sdk/Cargo.toml`	Adds workspace `repository` metadata needed for crates.io publishing.
`src/core/mxc_pty/Cargo.toml`	Adds publish metadata (repository/description) for the closure.
`src/mxc_telemetry/Cargo.toml`	Adds workspace license/repository metadata for publishing.
`src/core/generated/base_container_specification/Cargo.toml`	Removes `publish = false` by making `sandbox_spec` publishable and adds repository metadata.
`src/backends/appcontainer/common/Cargo.toml`	Adds publish metadata for publishing `appcontainer_common`.
`src/backends/bubblewrap/common/Cargo.toml`	Adds publish metadata for publishing `bwrap_common`.
`src/backends/lxc/common/Cargo.toml`	Adds publish metadata for publishing `lxc_common`.
`src/backends/seatbelt/common/Cargo.toml`	Adds publish metadata for publishing `seatbelt_common`.
`src/backends/nanvix/common/Cargo.toml`	Adds workspace `repository` metadata so `nanvix_common` is publishable.
`.azure-pipelines/1ES.Crate.Release.yml`	New ADO pipeline that runs dry-run preview and gated publish to crates.io.
`.azure-pipelines/scripts/publish_crates_to_cratesio.py`	New helper script to dry-run package or publish crates leaf-first and wait for sparse-index propagation.

+def _publish_one(crate: str, version: str, manifest_path: str, dry_run: bool) -> bool:
+    if version in _published_versions(crate):
+        print(f"SKIP  {crate} {version} - already on crates.io")
+        return True
+
+    if dry_run:
+        # `cargo package --list` validates the manifest (it rejects a path dep
+        # that lacks a version, exactly as publish does) and prints the packaged
+        # file set, WITHOUT resolving first-party deps against the crates.io
+        # index. `cargo publish --dry-run` cannot be used for a whole-closure
+        # preview: it resolves dependencies against the live index, so every
+        # non-leaf crate would fail until its first-party deps are actually
+        # published. --allow-dirty keeps the preview frictionless if CI leaves
+        # the tree in a state cargo considers dirty; the real publish never
+        # allows it.
+        rc = _run_cargo(
+            ["cargo", "package", "-p", crate, "--no-verify", "--allow-dirty",
+             "--manifest-path", manifest_path, "--list"]
+        )
+        ok = rc == 0
+        print(f"{'OK   ' if ok else 'FAIL '} dry-run {crate} {version}")
+        return ok


+      # ----------------------------------------------------------------------
+      # Preview: always runs, publishes nothing. Gives the operator the full
+      # `cargo publish --dry-run` + packaged-file list for every crate before
+      # anything reaches crates.io.
+      # ----------------------------------------------------------------------
+      - stage: DryRun
+        displayName: "Preview crate publish (dry run)"
+        jobs:
+          - job: DryRunJob
+            displayName: "cargo publish --dry-run"


bbonaby · 2026-07-15T17:15:29Z

+# only cargo config in effect is the repo-root .cargo/config.toml (Windows
+# rustflags only -- no source replacement, and inert on Linux).
+
+trigger: none


issue: To publish to crates.io using the official Microsoft open source account we will need to publish it through the ESRP release system. See: https://eng.ms/docs/microsoft-security/identity/trust-and-security-services/tss-release-distribute/tss-release-esrp-parent/oss-publishing/releasing-open-source/cratesio . You will need to follow that and do something similar to what we do in the 1ES.Release.yml. To be honest, we should be able to reuse that same yaml file to release to either Npm, Crates.io or both.

At the very least that one should be the entry point to all releases, but we can still have template yml files that we point to to keep it clean. Everything in terms of set up on the back end for ESRP should already be set up for us. So, let me know if you have any issues with that.

In the ESRP docs they say the azure rust sdk folks have this working, where they publish all their crate dependencies first then publish their main crate. I believe the yml file they do this in is this one https://github.com/Azure/azure-sdk-for-rust/blob/95eb8c3878cb2c7d3239b5f584b8179ee0483c31/eng/pipelines/templates/stages/archetype-rust-release.yml . We can probably do something similar.

Thanks Brandon -- followed this. Reworked the crate release to go through ESRP and folded it into 1ES.Release.yml so one pipeline releases npm, crates.io, or both:

1ES.Release.yml now takes publishNpm / publishCrates boolean params. The existing NPM stage is gated on publishNpm, and a new Publish_to_CratesIo stage runs the ESRP path.

Following the azure-sdk-for-rust approach, the official build packages the closure leaf-first in a single cargo package invocation (all --package flags together, so intra-closure deps resolve locally) into the mxc-crates-package artifact. The release job then runs one EsrpRelease@12 task per crate (contenttype: Rust, account microsoft-oss-releases), leaf-first, verifying each on the crates.io sparse index before the next.

Deleted the old token-based 1ES.Crate.Release.yml + publish_crates_to_cratesio.py -- no more CARGO_REGISTRY_TOKEN. New templates: templates/Package.Crates.Job.yml (packaging) and templates/Publish.CratesIo.Job.yml (ESRP release loop).

This also obsoletes the two Copilot comments above (they were on the now-deleted script/pipeline).

One heads-up before we flip publishing on -- publishCrates defaults to false for now. Packaging the true --all-features closure of mxc-sdk surfaced that it now routes through the mxc_engine hub crate and expands to 17 first-party crates. Before a first real publish we still need to: (1) add a version to the 7 path deps in src/Cargo.toml that currently only carry path; (2) resolve isolation_session_bindings being publish = false; and (3) do the OSPO OSS-release registration + reserve the crate names. Happy to take that as a follow-up -- let me know if you'd rather have it in this PR or a separate one.

Resolved Cargo.toml conflicts: - src/Cargo.toml [workspace.dependencies]: kept version pins on wxc_common and appcontainer_common; kept upstream's new mxc_engine and mxc-sdk path deps (left unversioned - not part of the current publish closure). - src/mxc_telemetry/Cargo.toml: kept repository.workspace = true. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e5d2aa5b-7f04-4e4d-83d3-a02efe7020ab

Rework the crates.io release per PR microsoft#647 review feedback (Brandon Bonaby): publish through the ESRP OSS release system instead of a bespoke CARGO_REGISTRY_TOKEN, folded into the existing 1ES.Release.yml so one pipeline releases npm, crates.io, or both (modeled on azure-sdk-for-rust). - scripts/crates_release.py: package/verify-order/stage/wait subcommands; cargo-packages the closure leaf-first, no token, no publish endpoint. - templates/Package.Crates.Job.yml: official-build job -> mxc-crates-package. - templates/Publish.CratesIo.Job.yml: releaseJob, EsrpRelease@12 contenttype Rust, one task per crate leaf-first, verify each on the crates.io index before the next. - 1ES.Release.yml: publishNpm/publishCrates params; gate NPM stage; add Publish_to_CratesIo stage; ESRP one-time-setup notes in the header. - 1ES.Build.Stages.yml: add Package_Crates stage. - Delete 1ES.Crate.Release.yml + publish_crates_to_cratesio.py (token approach). publishCrates defaults to false: enabling actual publishing still needs the crate closure metadata reconciled (7 path deps need versions; isolation_session_bindings is publish=false) plus OSPO OSS-release setup. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 54ab05fd-1ef7-4fa4-b130-ea17a9ef8a47

cmd_package looped `cargo package -p <crate>` one crate at a time, which fails on the first non-leaf crate: packaging a crate alone strips the path from each dependency and cargo then looks for the sibling on the crates.io index, where it is not published yet (e.g. wxc_common -> mxc_telemetry -> "no matching package named `mxc_telemetry` found"). Package the entire closure in a single `cargo package` invocation with every crate passed via -p, so cargo resolves intra-closure deps from target/package/ instead of crates.io (the azure-sdk-for-rust Pack-Crates.ps1 approach). Verified locally: all 9 packageable crates now tar in one run; the only remaining failure is the known mxc-sdk -> mxc_engine missing-version blocker. Also corrected two docstring claims that wrongly implied per-crate `--no-verify` packaging works. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 54ab05fd-1ef7-4fa4-b130-ea17a9ef8a47

dhoehna and others added 2 commits July 13, 2026 20:41

Copilot AI review requested due to automatic review settings July 14, 2026 19:04

dhoehna requested a review from a team as a code owner July 14, 2026 19:04

Copilot started reviewing on behalf of dhoehna July 14, 2026 19:05 View session

Copilot AI reviewed Jul 14, 2026

View reviewed changes

bbonaby requested changes Jul 15, 2026

View reviewed changes

microsoft-github-policy-service Bot assigned dhoehna Jul 15, 2026

microsoft-github-policy-service Bot added the Needs-Author-Feedback Issue needs attention from issue or PR author label Jul 15, 2026

dhoehna and others added 2 commits July 15, 2026 12:53

microsoft-github-policy-service Bot added Needs-Attention Issue needs attention from Microsoft and removed Needs-Author-Feedback Issue needs attention from issue or PR author labels Jul 20, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Enable one-click crates.io release for mxc-sdk#647

Enable one-click crates.io release for mxc-sdk#647
dhoehna wants to merge 5 commits into
microsoft:mainfrom
dhoehna:user/dahoehna/crate-release-pipeline

dhoehna commented Jul 14, 2026 •

edited by microsoft-github-policy-service Bot

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

bbonaby Jul 15, 2026

Uh oh!

dhoehna Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

dhoehna commented Jul 14, 2026 • edited by microsoft-github-policy-service Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How it works

Secrets stay out of the repo

One-time setup (not codeable in the repo)

Microsoft Reviewers: Open in CodeFlow

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

bbonaby Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

dhoehna Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dhoehna commented Jul 14, 2026 •

edited by microsoft-github-policy-service Bot

Loading