Maven – Developers centre - Testing Plugins Strategies

Currently, Maven only supports unit testing out of the box. This document is intended to help Maven Developers to test Plugins with Unit Tests, Integration Tests or Functional tests.

Note: There are a lot of different ways to test a Maven plugin. For a review of different strategies and tools, please refer to Review of Plugin Testing Strategies

Testing Styles: Unit Testing vs. Functional/Integration Testing

A unit test attempts to verify a mojo as an isolated unit, by mocking out the rest of the Maven environment. A mojo unit test does not attempt to run your plugin in the context of a real Maven build. Unit tests are designed to be fast.

A functional/integration test attempts to use a mojo in a real Maven build, by launching a real instance of Maven in a real project. Normally this requires you to construct special dummy Maven projects with real POM files. Often this requires you to have already installed your plugin into your local repository so it can be used in a real Maven build. Functional tests run much more slowly than unit tests, but they can catch bugs that you may not catch with unit tests.

The general wisdom is that your code should be mostly tested with unit tests, but should also have some functional tests.

Unit Tests

Using JUnit alone

In principle, you can write a unit test of a plugin Mojo the same way you'd write any other JUnit test case, by writing a class that extends TestCase.

However, most mojos need more information to work properly. For example, you'll probably need to inject a reference to a MavenProject, so your mojo can query project variables.

Using PlexusTestCase

Mojo variables are injected using Plexus, and many Mojos are written to take specific advantage of the Plexus container (by executing a lifecycle or having various injected dependencies).

If you all you need is Plexus container services, you can write your class with extends PlexusTestCase instead of TestCase.

With that said, if you need to inject Maven objects into your mojo, you'll probably prefer to use the maven-plugin-testing-harness.

maven-plugin-testing-harness

The maven-plugin-testing-harness is explicitly intended to test the org.apache.maven.reporting.AbstractMavenReport#execute() implementation.

In general, you need to include maven-plugin-testing-harness as dependency, and create a *MojoTest (by convention) class which extends AbstractMojoTestCase.

...
  <dependencies>
    ...
    <dependency>
      <groupId>org.apache.maven.plugin-testing</groupId>
      <artifactId>maven-plugin-testing-harness</artifactId>
      <version>3.3.0</version>
      <scope>test</scope>
    </dependency>
    ...
  </dependencies>
...

public class YourMojoTest
    extends AbstractMojoTestCase
{
    /**
     * @see junit.framework.TestCase#setUp()
     */
    protected void setUp() throws Exception
    {
        // required for mojo lookups to work
        super.setUp();
    }

    /**
     * @throws Exception
     */
    public void testMojoGoal() throws Exception
    {
        File testPom = new File( getBasedir(),
          "src/test/resources/unit/basic-test/basic-test-plugin-config.xml" );

        YourMojo mojo = (YourMojo) lookupMojo( "yourGoal", testPom );

        assertNotNull( mojo );
    }
}

For more information, please refer to Maven Plugin Harness Wiki

Integration/Functional testing

maven-verifier

maven-verifier tests are run using JUnit or TestNG, and provide a simple class allowing you to launch Maven and assert on its log file and built artifacts. It also provides a ResourceExtractor, which extracts a Maven project from your src/test/resources directory into a temporary working directory where you can do tricky stuff with it.

Maven itself uses maven-verifier to run its core integration tests. For more information, please refer to Creating a Maven Integration Test.

public class TrivialMavenVerifierTest extends TestCase
{
    public void testMyPlugin()
        throws Exception
    {
        // Check in your dummy Maven project in /src/test/resources/...
        // The testdir is computed from the location of this
        // file.
        File testDir = ResourceExtractor.simpleExtractResources( getClass(), "/my-dummy-maven-project" );

        Verifier verifier;

        /*
         * We must first make sure that any artifact created
         * by this test has been removed from the local
         * repository. Failing to do this could cause
         * unstable test results. Fortunately, the verifier
         * makes it easy to do this.
         */
        verifier = new Verifier( testDir.getAbsolutePath() );
        verifier.deleteArtifact( "org.apache.maven.its.itsample", "parent", "1.0", "pom" );
        verifier.deleteArtifact( "org.apache.maven.its.itsample", "checkstyle-test", "1.0", "jar" );
        verifier.deleteArtifact( "org.apache.maven.its.itsample", "checkstyle-assembly", "1.0", "jar" );

        /*
         * The Command Line Options (CLI) are passed to the
         * verifier as a list. This is handy for things like
         * redefining the local repository if needed. In
         * this case, we use the -N flag so that Maven won't
         * recurse. We are only installing the parent pom to
         * the local repo here.
         */
        List cliOptions = new ArrayList();
        cliOptions.add( "-N" );
        verifier.executeGoal( "install" );

        /*
         * This is the simplest way to check a build
         * succeeded. It is also the simplest way to create
         * an IT test: make the build pass when the test
         * should pass, and make the build fail when the
         * test should fail. There are other methods
         * supported by the verifier. They can be seen here:
         * http://maven.apache.org/shared/maven-verifier/apidocs/index.html
         */
        verifier.verifyErrorFreeLog();

        /*
         * Reset the streams before executing the verifier
         * again.
         */
        verifier.resetStreams();

        /*
         * The verifier also supports beanshell scripts for
         * verification of more complex scenarios. There are
         * plenty of examples in the core-it tests here:
         * http://svn.apache.org/repos/asf/maven/core-integration-testing/trunk
         */
    }

Note: maven-verifier and maven-verifier-plugin sound similar, but are totally different unrelated pieces of code. maven-verifier-plugin simply verifies the existence/absence of files on the filesystem. You could use it for functional testing, but you may need more features than maven-verifier-plugin provides.

maven-invoker-plugin

You can use maven-invoker-plugin to invoke Maven and to provide some BeanShell tests. Tests written in this way don't run under JUnit/TestNG; instead, they're run by Maven itself.

<project
  xmlns="http://maven.apache.org/POM/4.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
                      http://maven.apache.org/xsd/maven-4.0.0.xsd">
  ...
  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-invoker-plugin</artifactId>
        <version>1.9</version>
        <configuration>
          <projectsDirectory>src/it</projectsDirectory>
          <pomIncludes>
            <pomInclude>**/pom.xml</pomInclude>
          </pomIncludes>
          <postBuildHookScript>verify</postBuildHookScript>
        </configuration>
        <executions>
          <execution>
            <goals>
              <goal>run</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
      ...
    </plugins>
  </build>
  ...
</project>

shitty-maven-plugin

The shitty-maven-plugin (Super Helpful Integration Testing ThingY) provides a simple way to run integration tests for a project (single module or multi-module).

shitty-maven-plugin does many of the same things as the maven-invoker-plugin (it supports Groovy tests instead of BeanShell tests), though it has some features that aren't available in maven-invoker-plugin. Notably, it provides some advanced setup steps to automatically install your plugin using a special version name ("TESTING"), so your dummy projects can depend on that version explicitly.

maven-it-plugin

Note: maven-it-plugin is not at 1.0 yet. Use it at your own risk.

The maven-it-plugin is used directly in the integration-test phase.

Note: this it plugin can not be used to test a plugin that is being built for the first time (i.e. with no release). In this case, you could, for instance, defined an it-snapshot of the plugin with maven-install-plugin. during the pre-integration-test phase.

<project
  xmlns="http://maven.apache.org/POM/4.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
                      http://maven.apache.org/xsd/maven-4.0.0.xsd">
  ...
  <artifactId>maven-XXX-plugin</artifactId>
  <packaging>maven-plugin</packaging>
  <version>1.0-SNAPSHOT</version>
  <description>Test Report plugin in the site phase</description>
  ...
  <build>
    <plugins>
      ...
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-it-plugin</artifactId>
        <version>1.0-alpha-1-SNAPSHOT</version>
        <configuration>
          <integrationTestsDirectory>${project.basedir}/src/it</integrationTestsDirectory>
          <includes>
            <include>**/pom.xml</include>
          </includes>
          <goals>site</goals>
        </configuration>
        <executions>
          <execution>
            <phase>integration-test</phase>
            <goals>
              <goal>test</goal>
            </goals>
          </execution>
        </executions>
      </plugin>

      <!-- Need to install IT snapshot of maven-XXX-plugin -->
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-install-plugin</artifactId>
        <version>2.2-SNAPSHOT</version>
        <executions>
          <execution>
            <id>it-test</id>
            <phase>pre-integration-test</phase>
            <goals>
              <goal>install-file</goal>
            </goals>
            <configuration>
              <file>${project.build.directory}/maven-XXX-plugin-1.0-SNAPSHOT.jar</file>
              <groupId>org.apache.maven.plugins</groupId>
              <artifactId>maven-XXX-plugin</artifactId>
              <version>1.0-it-SNAPSHOT</version> <!-- IT SNAPSHOT -->
              <packaging>maven-plugin</packaging>
              <pomFile>${project.basedir}/pom.xml</pomFile>
            </configuration>
          </execution>
        </executions>
      </plugin>

      <!-- Testing the result of the it pom.xml -->
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.4.2</version>
        <executions>
          <execution>
            <id>it-test</id>
            <phase>post-integration-test</phase>
            <goals>
              <goal>test</goal>
            </goals>
            <configuration>
              <includes>
                <include>**/*TestIt.class</include>
              </includes>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
  ...
</project>

The it pom should use the it snapshot:

<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
                      http://maven.apache.org/xsd/maven-4.0.0.xsd">
...
  <reporting>
    <outputDirectory>
      ${project.basedir}/../../../../../target/it/it1/target/site
    </outputDirectory>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-XXX-plugin</artifactId>
        <version>1.0-it-SNAPSHOT</version>
      </plugin>
    </plugins>
  </reporting>
...
</project>

maven-plugin-management-plugin

The maven-plugin-management-plugin is to stage/unstage a plugin into the local repository for pre/post-integration-test.

You need to configure the maven-plugin-test-plugin and the maven-surefire-plugin:

<project
  xmlns="http://maven.apache.org/POM/4.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
                      http://maven.apache.org/xsd/maven-4.0.0.xsd">
  ...
  <artifactId>maven-XXX-plugin</artifactId>
  <packaging>maven-plugin</packaging>
  <version>1.0-SNAPSHOT</version>
  <description>Test Report plugin in the site phase</description>
  <prerequisites>
    <maven>2.0.4</maven>
  </prerequisites>
  ...
  <dependencies>
    ...
    <!-- Due to the Maven 2.0.4 prerequisites.
    By default, maven-plugin-test-plugin uses 2.0.1 -->
    <dependency>
      <groupId>org.apache.maven</groupId>
      <artifactId>maven-embedder</artifactId>
      <version>2.0.4</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.apache.maven</groupId>
      <artifactId>maven-core</artifactId>
      <version>2.0.4</version>
      <scope>test</scope>
    </dependency>
  </dependencies>
  <build>
    <plugins>
      ...
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-plugin-management-plugin</artifactId>
        <version>1.0-SNAPSHOT</version>
        <executions>
          <execution>
            <id>pre-it-test</id>
            <phase>pre-integration-test</phase>
            <goals>
              <goal>stage</goal>
            </goals>
          </execution>
          <execution>
            <id>post-it-test</id>
            <phase>post-integration-test</phase>
            <goals>
              <goal>unstage</goal>
            </goals>
          </execution>
        </executions>
      </plugin>

      <!-- Testing the result of the it pom.xml -->
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.4.2</version>
        <executions>
          <execution>
            <id>it-test</id>
            <phase>integration-test</phase>
            <goals>
              <goal>test</goal>
            </goals>
            <configuration>
              <includes>
                <include>**/*TestIt.class</include>
              </includes>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
  ...
</project>

Introduction