Implementing a Custom JUnit5 Test Engine

With the release of JUnit5, a new way of running tests was introduced. Previously, all of our unit tests had to be in the form of Java classes (this type of test is now handled in the JUnit Vintage (legacy) and JUnit Jupiter (current) test engines). This is fine for many (or even most) cases, but sometimes you might prefer to structure your tests in a file-based manner, e.g. because they are generated or downloaded during test runtime.

Thanks to the platform-like structure of JUnit 5 and it’s first-class support for resource-based testing, we now can implement our own custom test engines that specifically fit our use case.

This post will be a short explanation of how to accomplish such a thing, because documentation on this topic is hard to find (notable exception is this great talk which unfortunately is only available in german).

Understanding the role of a Test Engine

Before we get into the code, let’s quickly look at what a test engine actually is and how the JUnit platform will interact with it. Looking into the javadoc of the TestEngine interface, we find:

A TestEngine facilitates discovery and execution of tests for a particular programming model.

In other words, a test engine is a Java class that JUnit directly interacts with, which is responsible for discovering (i.e. telling the platform which tests can be run) and executing tests. We already saw two concrete implementations: JUnit Vintage, which can discover and run tests written in the “old” JUnit 3 and 4 style and JUnit Jupiter which supports the new programming model introduced in JUnit 5.

The TestEngine interface defines two notable methods:

• discover(EngineDiscoveryRequest discoveryRequest, UniqueId uniqueId): This method is called whenever tests should be discovered. The discoveryRequest contains information on how the discovery should be carried out. In particular it consists of selectors and filters, which tell the test engine which resources (i.e. files or directories, classes, methods, packages, etc.) to scan and how to filter these resources respectively. The method should return a TestDescriptor object, which essentially is a container for the collection of tests that were discovered (note that tests can also be nested further).
• execute(ExecutionRequest request): This method will be called when the test engine should actually run tests. The request parameter contains the exact TestDescriptor object that was previously returned during discovery and can therefore be used to quickly access the resources required for test execution. During execution, the engine must notify the platform using the EngineExecutionListener object contained in the request of any failures or successes of tests.

A Basic Custom Test Engine

As you can see, a custom test engine is fairly straight-forward to implement, but provides a powerful way to create tests which exactly fit your business domain. Let’s implement our custom test engine that can discover and “execute” file-based tests.

To get started, create a new class implementing the TestEngine interface we saw before as follows:

public class MyCustomTestEngine implements TestEngine
{
    @Override
    public String getId() {
      // give your test engine a nice id
      return "my-custom-test-engine";
    }

    @Override
    public TestDescriptor discover(EngineDiscoveryRequest discoveryRequest, UniqueId uniqueId) {
      // TODO
    }

    @Override
    public void execute(ExecutionRequest request) {
      // TODO
    }
}

Discover

For the discover method, let’s start by creating an EngineDescriptor object which represents a container for tests discovered by our test engine:

EngineDescriptor engineDescriptor = new EngineDescriptor(uniqueId, "My Custom Test Engine");
// TODO
return engineDescriptor;

Using this engineDescriptor, we can now add additional test descriptors using the addChild method. However, before doing that, let’s make sure we can actually extract and store all information we will later need for execution.

Create a new MyCustomTestDescriptor class extending AbstractTestDescriptor as follows:

public class MyCustomTestDescriptor extends AbstractTestDescriptor {

    protected MyCustomTestDescriptor(UniqueId engineId, String displayName, List<String> fileContent) {
        super(engineId.append("my-custom-test", displayName), displayName);
        this.fileContent = fileContent;
    }

    private final List<String> fileContent;

    @Override
    public Type getType() {
      // either TEST or CONTAINER
        return Type.TEST;
    }

    public List<String> getFileContent() {
        return fileContent;
    }

}

As you can see, we will be able to store the entire file content in the test descriptor.

Note that this is probably a really bad idea because these files might be really big. It is better to just store the file path and then load the file during execution.

Using our custom test descriptor let’s go back to our discover method in TestEngine and add files:

discoveryRequest.getSelectorsByType(FileSelector.class).forEach(selector -> {
    try {
        engineDescriptor.addChild(new MyCustomTestDescriptor(engineDescriptor.getUniqueId(), selector.getFile().getName(), Files.readAllLines(selector.getFile().toPath())));
    } catch (IOException e) {
        // handle exception
    }
});

First, we get all FileSelector objects in the request. Then, for each selector we take the file it refers to and create a test descriptor from it while also extracting the file name and content.

We are now done with the discover method. Of course, this is an extremely limited test engine, because it only supports FileSelector. You might also look into DirectorySelector and ClassSelector for your implementation, but the underlying approach to handle these will be similar.

Execute

For the execute method, we can start by first extracting some of the objects from the request that we will need:

@Override
public void execute(ExecutionRequest request) {
    TestDescriptor engineDescriptor = request.getRootTestDescriptor();
    EngineExecutionListener listener = request.getEngineExecutionListener();
    // TODO
}

Then, we can add calls to the listener class to notify of execution start and finish:

listener.executionStarted(engineDescriptor);
// TODO
listener.executionFinished(engineDescriptor, TestExecutionResult.successful());

Note, that you might not always to return a successful result for the engine descriptor. For example, there might be cases where a single test failure means that the entire container of tests has failed.

Now, for the core of our implementation, we iterate through all test descriptors in the engineDescriptor object (remember: we actually added these ourselves in the discover method):

for (TestDescriptor testDescriptor : engineDescriptor.getChildren()) {
    // cast it to our own class
    MyCustomTestDescriptor descriptor = (MyCustomTestDescriptor) testDescriptor;
    listener.executionStarted(testDescriptor);
    // here you would add your super-complicated logic of how to actually run the test
    if (descriptor.getFileContent().get(0).equals("true")) {
        listener.executionFinished(testDescriptor, TestExecutionResult.successful());
    } else {
        listener.executionFinished(testDescriptor, TestExecutionResult.failed(new AssertionError("File content was incorrect.")));
    }
}

This is a really basic implementation, as it will only mark the test as successful if the first line of the test file is “true”, but it demonstrates the approach fairly well.

Test Engine Registration

With this, our custom test engine is actually already done. However, one question remains: How does JUnit 5 actually know that our test engine exists? For that, create a new file in your test engine project under src/main/resources/META-INF/services/org.junit.platform.engine.TestEngine and add the name of your implementation. For example, if your package is com.custom.engine and your custom test engine class is called MyCustomTestEngine, the file should contain the text:

com.custom.engine.MyCustomTestEngine

Now, you can build your test engine, publish it to a maven repo and add it as a dependency to any project using JUnit 5 and it should just work.

Running tests

Unfortunately, while JUnit 5 has full support for file-based testing, many build tools and IDEs do not. For example, Gradle only supports the “normal” way of invoking the test engine class- or package-selectors, but the team is working on extending it.

In the meanwhile, there are some workarounds you can use:

Console Launcher

Using the console launcher which is provided by the JUnit 5 team, it is very easy to run tests (including file-based with the -f console option). I won’t go into details here, but the team also provides an example of how to integrate this into a gradle project.

One downside of this approach is, that you can not use the nice integration your IDE probably has with gradle. Because you rely on an external tool, the results of the execution will not be propagated to gradle and are therefore also not available in your IDE. You might be able to extract results from stdout or some generated xml reports but that’s not really an elegant solution. So what else can you do?

Running resource-based tests using a ClassSelector

I developed this workaround following a comment by johanneslink on my question on StackOverflow. It essentially abstracts away the fact that you are doing resource-based testing and acts as if you are actually testing classes. To get started, create a new class:

public class TestInitializer {
}

Then, in your test engine implementation discover method, add the following snippet:

discoveryRequest.getSelectorsByType(ClassSelector.class).forEach(selector -> {
      if (selector.getClassName().equals(TestInitializer.class.getName())) {
          discoverTests(new File(System.getenv("test_dir")), engineDescriptor);
      }
  });

This will look for a ClassSelector in the request which has the TestInitializer class as the target. Then it will read the environment variable test_dir which will be set by the invoking tool to tell the test engine where the test file/directory is actually located. Of course, you might have more than one such initializer, depending on which selector you want to support (i.e. one FileTestInitializer for files and another one for directories).

To invoke this, you can then add the following snippet to your maven pom.xml (note that I’m using Surefire here):

<build>
  <plugins>
      <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-surefire-plugin</artifactId>
          <version>2.22.2</version>
          <configuration>
              <dependenciesToScan>
                  <dependency>
                      my_custom_test_engine:custom_test_engine
                  </dependency>
              </dependenciesToScan>
              <includes>
                  <include>**/TestInitializer*
                  </include>
              </includes>
              <environmentVariables>
                  <test_dir>target/test-classes
                  </test_dir>
              </environmentVariables>
          </configuration>
      </plugin>
  </plugins>
</build>

Replace my_custom_test_engine:custom_test_engine with the dependency containing your initializer classes and target/test-classes with the test file or directory.

Similarly, you can use the following snippet for Gradle:

task surefireEmulatorTest(type: Test) {
    useJUnitPlatform {
    }
    testClassesDirs = sourceSets["main"].runtimeClasspath
    include '**/TestInitializer*'
    environment "test_dir", "build/resources/test"
}

And just like that, you can discover and run resource-based tests in your favorite IDE or build tool while still enjoying the full support for class-based testing.

Conclusion

This post turned out to be longer than I expected, but I think the topic is worth exploring as it gives you the opportunity to write tests that really fit your business domain.

I hope we will see some general-purpose test engine implementations in the future. For example, one might define tests in a table-like structure (similar to FitNesse) and use a test engine to run them. This could even be used outside the world of unit- or even software-testing, because it brings the power of popular tools like Surefire to completely new areas.