Isolate the VS Code test harness from the user's extension install - #218
Open
Chiara Mooney (chiaramooney) wants to merge 1 commit into
Open
Isolate the VS Code test harness from the user's extension install#218Chiara Mooney (chiaramooney) wants to merge 1 commit into
Chiara Mooney (chiaramooney) wants to merge 1 commit into
Conversation
The harness installed the locally built VSIX into the developer's real
VS Code extensions directory and launched VS Code against it. That let a
stale extension metadata cache survive between runs, so the harness could
silently drive a different build than the one it had just built.
Pin both the install and the launch to the harness-local
.drive-extensions directory, which the skill's .gitignore, README, and
AGENTS.md already describe and which vscode-drive.psm1 already prefers.
Uninstall the registered extension before reinstalling so VS Code cannot
reuse stale metadata, and assert that the version it reports back matches
the VSIX just built -- failing loudly instead of warning, since a silent
version mismatch invalidates every result the harness produces.
Also surface the diagnostic code from queryDiagnostics, normalizing the
{value, target} object form to a scalar, so scripted steps can assert on
diagnostic codes rather than matching message text.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0a32116a-78ec-415a-aa5e-f98a06c74e9d
7 tasks
Collaborator
Author
|
Extracted from #50, which no longer touches |
VSIX Build
Updated 2026-09-02 17:20:08 UTC · commit |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The
vsce-testingharness installed the locally built VSIX into the developer's real VS Code extensions directory and then launched VS Code against that same directory. Because VS Code keeps an extension metadata cache there, stale metadata could survive between runs — so the harness could silently drive a different build than the one it had just built. Any run that hit that case produced results that looked fine but were untrustworthy.What changed
scripts/install-extension.ps1andscripts/drive-extension.ps1now pin--extensions-dirto the harness-local.drive-extensionsdirectory, so the harness never touches the developer's own VS Code install.install-extension.ps1uninstalls the registeredmicrosoft-winappcli.winappextension before reinstalling, so VS Code cannot reuse stale metadata.Behaviour change worth reviewing
The post-install check now throws where it previously warned — both when the extension is missing from
code --list-extensionsand when the reported version disagrees with the built VSIX. This is a deliberate fail-fast: a silent version mismatch invalidates every result the harness produces, so it should stop the run rather than scroll by. A previously noisy-but-passing run can now hard-fail.Consistency note
.drive-extensionswas not a new concept: it was already gitignored, already documented in the skill'sREADME.mdandAGENTS.md, andscripts/vscode-drive.psm1:184already auto-added the flag. This just brings the two remaining scripts in line with what the rest of the harness already assumed.Also included
queryDiagnosticsin the driver extension now returns the diagnosticcode, normalized from VS Code's{ value, target }object form down to a scalar. Scripted steps can now assert on diagnostic codes instead of pattern-matching message text.Note
Extracted from #50, where these three files landed by accident alongside the WinUI XAML language-service work. Diff is exactly 3 files, +31/-4.