Skip to content

test(e2e): add API and CLI conformance profiles#2182

Open
elezar wants to merge 7 commits into
mainfrom
feat/conformance-suite-skeleton
Open

test(e2e): add API and CLI conformance profiles#2182
elezar wants to merge 7 commits into
mainfrom
feat/conformance-suite-skeleton

Conversation

@elezar

@elezar elezar commented Jul 8, 2026

Copy link
Copy Markdown
Member

Summary

Adds explicit API and CLI conformance profiles to the standalone Rust e2e package without changing production code.

The API profile validates driver-independent gateway and sandbox semantics directly over gRPC. The CLI profile identifies portable user workflows that can run against any compatible gateway while remaining separate from driver-specific infrastructure coverage. Gateway restart tests now share an application-level resume scenario but remain outside baseline conformance because they require provisioner control.

Related Issue

Changes

  • add an internal openshell-conformance binary under e2e/rust
  • add a shared API scenario engine used directly by the Rust integration test
  • implement API scenarios for lifecycle, not-found behavior, idempotent deletion, validation, concurrent creation, labels, streamed command execution, and process hardening
  • add the e2e-api-conformance profile and run it against Docker, Podman, Kubernetes, and VM gateway lanes
  • add the e2e-cli-conformance profile for portable smoke, sandbox lifecycle, labels, port forwarding, sync, and upload-on-create workflows
  • verify both CLI command paths: sandbox create -- <command> and sandbox exec
  • run CLI conformance in the canonical Docker-backed Rust e2e suite
  • extract VM overlay, writable-workspace, and TLS-key permission assertions into a VM-specific test
  • share the gateway restart/resume contract across Docker, Podman, and VM tests while retaining driver-specific observation hooks
  • keep gateway provisioning separate from scenario execution through an explicit endpoint and mTLS environment contract
  • use local tonic transport and output formatting so the suite does not depend on the full openshell-cli crate
  • keep the conformance binary internal to tests and out of release packaging

The capabilities scenario remains deferred until the gateway exposes driver capabilities through its public API. Additional CLI, security/network, and external-image conformance candidates are intentionally left for follow-up work.

Testing

  • locked compilation of all affected API, CLI, VM, and gateway-resume targets
  • focused Clippy with warnings denied
  • 22 e2e library unit tests
  • 2 openshell-conformance binary tests
  • openshell-conformance list --output json reporting all eight API scenarios
  • Rust formatting checks for modified files
  • shell syntax checks for modified e2e scripts
  • git diff --check
  • live Docker, Podman, Kubernetes, or VM conformance run
  • mise run pre-commit (mise is unavailable in the current environment)

Checklist

  • Changes are limited to e2e test code, test provisioners, and task configuration
  • No production code or production dependencies changed
  • API and CLI conformance remain explicitly separated
  • All commits are signed off and follow Conventional Commits
  • Documentation updated

@copy-pr-bot

copy-pr-bot Bot commented Jul 8, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@elezar
elezar force-pushed the feat/conformance-suite-skeleton branch from 8f15a6c to a0bb1ff Compare July 16, 2026 14:19
@elezar elezar changed the title feat(cli): add openshell conformance subcommand skeleton test(e2e): add API and CLI conformance profiles Jul 16, 2026
elezar added 2 commits July 17, 2026 15:58
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
@elezar
elezar force-pushed the feat/conformance-suite-skeleton branch from de26f37 to 9493e29 Compare July 17, 2026 14:00
@elezar
elezar marked this pull request as ready for review July 17, 2026 14:03
@elezar
elezar force-pushed the feat/conformance-suite-skeleton branch from 9493e29 to e213a39 Compare July 17, 2026 14:15
elezar added 5 commits July 17, 2026 16:21
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
@elezar
elezar force-pushed the feat/conformance-suite-skeleton branch from e213a39 to 6fe2c16 Compare July 17, 2026 14:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant