This post is about using Jest to define and run tests for a VSCode extension using the VSCode Extension Host, which is a live instance of VSCode that actions can be automated against. I am developer of the VSCode extension Stryker Runner (blog, Github) and I wanted to add more in depth testing to simplify and strengthen the release process.

What is VS Code?

Visual Studio Code (VS Code) is free and widely used IDE developed by Microsoft. VS Code has access to an official marketplace of extensions that are used to add additional functionality, integrate libraries like Stryker and even add additional language support. There are also unofficial marketplaces that can be used instead of, or as well as, the official one, e.g. the open source marketplace Open VSX.

Why bother testing with the Extension Host?

Stryker Runner has been available for about a year now and has recently received its first third party contribution (exciting!) and been used as base for a similar extension to cater to .NET mutation testing in VSCode (exciting!).

Stryker Runner has always been fully unit and fully mutation tested, but I have been doing manual testing prior to any release of a new version. I’m human and I don’t like doing this. Luckily, the manual tests are pretty simple and good candidates for automation.

Why bother using Jest?

The out of the box configuration is using Mocha (more info) so that would be easier. However, when I first started developing Stryker Runner these kinds of tests were not a priority and, being very familiar with Jest, used it for all the existing tests in the project.

To keep development and maintenance simpler, using Jest everywhere feels like the best way forward.

How can I use Jest to define and run VSCode Extension Host tests?

It is more difficult to run (any) test runner using the VSCode Extension Host than usual because we are resposible for getting the test runner initialised and pointed at the test suite once the Extension Host has been spun up. The default test setup makes it easier with Mocha.

Using Jest is possible but I struggled to find a definitive solution out there to get set up easily. I had to put together information and examples from a few different places to get it working, so I wanted to put it here all in once place. The following things must be possible:

  1. Any custom test running script must conform to a vscode-test provided interface - more below
  2. We must be able to programmatically run Jest
  3. We must be able to surface feedback from Jest both in the terminal and to the Extension Host process
  4. We must ensure the vscode API is available in our tests (the Jest environment is complex)

Custom test running script

To use a custom test runner with VSCode extension tests you must provide a an object or module that conforms to a simple interface. It currently looks like like this:

interface INewTestRunner {
	run(): Promise<void>;
}

Pretty simple. You will point the VSCode extension tests script at a module that exports something conforming to this interface as the default export. In the case of my extension, Stryker Runner, it looks like this.

import { runCLI } from 'jest';

export async function run() {}

Once you have this implemented the extensionTestsPath part of the config shown in the docs should point to it. You don’t need to implement any functionality in the run method to prove Extension Host testing process will work at this point, as long as the method does not reject the promise the test process will run happily (but with no tests yet).

import { runTests } from '@vscode/test-electron';
import * as path from 'path';

async function main() {
  try {
    // The folder containing the Extension Manifest package.json
    // Passed to `--extensionDevelopmentPath`
    const extensionDevelopmentPath = path.resolve(__dirname, '../../');

    // The path to the extension test runner script
    // Passed to --extensionTestsPath
    const extensionTestsPath = path.resolve(__dirname, './suite/index');

    await runTests({ extensionDevelopmentPath, extensionTestsPath });
  } catch (err) {
    console.error('Failed to run tests');
    process.exit(1);
  }
}

Programmatically run Jest

Now we need to invoke Jest programmatically within this method and get our actual tests running. There is no official way to do this but there is an open issue looking to make it possible (from one of the StrykerJS maintainers no less). That being said, Jest does export a runCLI method that will allow us to run Jest programmatically, albeit unofficially.

In the case of my extension, Stryker Runner, it is being used like this. The open issue above is proposing passing dynamic, defined in code config to Jest when called programmatically, for now using runCLI we must pass arguments as the CLI would want them, which means a static pointer to a static config. That works, and it may be prudent to use a separate jest config for these tests than your unit tests (mine is here).

import { runCLI } from 'jest';
import path from 'path';

export async function run() {
  const repoRoot = path.join(__dirname, '..', '..', '..');
  const config = path.join(repoRoot, 'jest.e2e.config.js');

  try {
    const results = await runCLI({ config } as any, [repoRoot]);
  } catch (error) {
    console.error((error as Error).message);
    process.exit(1);
  }
}

Surface test feedback - success/failure

The runCLI method we are using from Jest will always succeed and resolve its Promise unless Jest itself cannot get off the ground, this means that it resolves even when there are failing tests. The knock on effect is that the Extension Host test process will also finish quite happily, which prevents it from being a blocking process in an environment like CI.

The result of a resolved runCLI Promise contains the data required to understand if, and what, tests failed. In the case of my extension, Stryker Runner, I am only inspecting the following property: results.results.numFailedTests. If this number is greater than 0 I will throw an Error and force the the Extension Host process to exit with non-zero code.

if (results.results.numFailedTests > 0) {
    throw new Error('Some tests failed.');
}

Now we can reliably know if some tests failed, but how do we know which tests failed? Well some of the information I found out there when putting this together suggested workarounds to manually output the successes/failures to the terminal but I didn’t find that was necessary as stdout from the Extension Host is piped in the terminal that initiated it anyway. For my use, I did nothing. But if you want to look at other approaches that may or may not still be required in certain circumstances, check here and here.

Ensure the real vscode API is available in the Extension Host

Jest does some funky stuff to provide its own environment when running tests, all (or some) of which results in the vscode API not being available when running the Extension Host tests. For Stryker Runner, to rectify that and ensure that it is included in the environment I am supplying my own TestEnvironment (here) that inherits from the base Jest one, its only job it set the vscode API on the global object as the extension expects it to be there. I am re-exporting the API in a local module to grab a hold of it. My TestEnvironment must then be pointed to using the testEnvironment (docs) property of the Jest configuration. I am, of course, standing on the shoulders of giants here.

Summary

To see this stuff in action feel free to clone the Stryker Runner source and run yarn test:integration, it runs cross platform and will allow you to play around with anything that was documented above.

Go forth and test you VSCode extensions using the Extension Host and Jest.

Get in contact

If you have comments, questions or better ways to do anything that I have discussed in this post then please get in contact.