groovy on Solid Soft

More compact parameterized tests reporting in Spock 2.1

Marcin Zajączkowski — Tue, 05 Apr 2022 10:00:00 +0200

Get even more compact reporting for parameterized tests with Spock 2.1.

Historical background

TL;TR. To just get a recipe, jump to the “More compact reporting” section.

Parameterized tests, a place where Spock excels. A beautiful and powerful mechanism to reduce duplication across tests with a very readable table-like data representation (and powerful data providers if sophisticated custom logic is required).

When I started using Spock years ago (0.6?), one question perpetually dinned in somebody’s ears. Why unrolling was not enabled by default? Adding @Unroll (even at the test class level) was for me redundant, error prone and… just boring. Due to that, I created a small spock-global-unroll extension which was dead simple. It was enough to just put it on a classpath (i.e. add as a dependency in Gradle or Maven) to have all the parameterized tests automatically unrolled in the whole project. Zero configuration, it just worked. You could remove all @Unroll annotation (without parameters) in your project. I really liked it and used it in every project I had my fingers in. Have you ever heard about it?

It was a rhetorical question. The vast majority of you did not. Why? Because it was an extension and people start looking for extensions when something does not work. For the extensions that improve (even dramatically) some aspect of your tests you would need to hear about them before (and I am hopeless at self promotion :-/).

That was a reason why I pushed hard to have it available and enabled by default in the core of Spock 2.0. With a comprehensive implementation by Björn Kautler in landed in 2.0-M3 to stay there for good. However, preparing some time ago my presentation “What you can expect from Spock 2?" , I’ve noticed that it could be improved even more.

Default (verbose) reporting

Performance tip. For Gradle projects, I prefer to execute unit and integration tests directly by IDEA, not by IDEA using Gradle (which - unfortunately - has been a default option since 2019 or so). First of all, it is much faster - a crucioal thing for TDD - and in the majority of cases it works perfectly fine. The examples below use that mode. You can read how to speed up your unit tests execution in my another blog post.

Let’s assume we have a simple parameterized test:

class UnrollSpec extends Specification {

    void "should calculate number of characters in given animal name"() {
        expect:
            animalName.length() == expectedLength
        where:
            animalName     || expectedLength
            "dog"          || 3
            "wolf"         || 4
            "hippopotamus" || 12
    }
}

Spock 2 unrolls parameterized tests by default displaying also all the used parameters. However, the output in IDEA is hard to read:

To see the parameters it is usually needed to scroll the result window horizontally (if luckily your fancy mouse supports that :-) ) or reduce space available for the test standard output.

The crazy idea sprang in my mind - could the (repeated) feature name be omitted in the iterations name?

To remove it for a particular test it is easy:

@Unroll("animaName: #animalName, expectedLength: #expectedLength")    //inconvenient - just for one test
void "should calculate number of characters in given animal name"() {
    //...
}

However, we have been just celebrating the removal of @Unroll, not to be forced to put it back everywhere… Luckily, Spock has a global configuration mechanism, could not it be used?

After some minutes when I was analyzing the UnrollExtension code, it turned out that no. SpockConfig.groovy allows to define a custom defaultPattern, but there is no way to display just the parameters. It is a FOSS project, what should be done in that case? :-)

More compact reporting

The idea described in the previous section resulted in the improvements in Spock 2.1. There is a new configuration parameter which needs to be placed inside the unroll {} closure in the SpockConfig.groovy file (located for example in src/test/resources - NOT together with the test classes):

//in SpockConfig.groovy
unroll {
    includeFeatureNameForIterations false
}

Much shorter, still meaningful output. And for all tests in the project!

Note. If for some reason, you want to use it just for individual features @Unroll('#dataVariablesWithIndex') can be used as well. Please refer the official documentation for other use cases.

Keep in mind that SpockConfig.groovy can be placed in various places, including your home directory - to more global impact.

Limitations

Why that marvelous feature was not enabled by default in Spock 2.1? We were thinking about that, but there two arguments against:

Backward compatibility.
Poor support in some reporting tools.

The second point was more important. While the IDEA represents the test results with a very nice tree-like view, some other tools are not. The test report generated by Gradle or Maven’s Surefire displays them as a flat list, without the (parent) test name. As a result, not having the feature name in every iteration could be confusing.

As a result you might want to enable that feature conditionally only if executed in IDEA, for example:

//in SpockConfig.groovy
static boolean isExecutedFromIdea() {
    return System.getProperty("java.class.path").contains("idea_rt.jar")    //any more reliable way?
}

unroll {
    includeFeatureNameForIterations !isExecutedFromIdea()
}

That check is somehow fragile (e.g. doesn’t work if IDEA executes tests not directly, but using Gradle), but I wasn’t able to find anything better. Feel free to put your proposal in the comments.

Summary

I hope that you also found includeFeatureNameForIterations false useful. Hopefully, having better support in the reporting tools there will be no obstacles to make it enabled by default in Spock 3.0.

Btw, in the middle of May, I will be talking about that and other new and noteworthy in Spock 2 at the GeeCON conference . This time, finally, on-site, in Kraków, Poland (but the whole conference is in English).

Lead photo by Paulina101, published in Pixabay, Pixabay License.

Running Spock with unsupported Groovy version (Gradle + Maven)

Marcin Zajączkowski — Fri, 19 Nov 2021 00:10:00 +0200

Stuck with the ‘The Spock compiler plugin cannot execute because Spock 1.3-groovy-2.5 (3.0) is not compatible with Groovy 3.0 (4.0)’ error message? Find out what can be done about that (using Gradle and Maven).

Historical background

In the dim and distance past, when Groovy 1.x occasionally had some unpleasant (also runtime) backward compatibility issues, Spock had started to be built for the particular Groovy version individually. There were spock-0.5-groovy-1.6, spock-0.5-groovy-1.7 and spock-0.5-groovy-1.8 for instance. To help people choose the right Spock and the Groovy versions, the runtime check was added to fail the compilation (at the AST transformation level) with the meaningful error message, in the situation Spock was compiled with the different Groovy version that is available at runtime, e.g.:

The Spock compiler plugin refused to run because Spock 1.3-groovy-2.5 is not compatible with Groovy 3.0.4. For more information,
see http://versioncheck.spockframework.org Spock location: .../spock-1.3-groovy-2.5.jar, Groovy location: .../groovy-3.0.4.jar

The error message is very clear. The reasoning behind this decision is good, and definitely it can help developers to detect problems with the dependencies. However, it can also greatly complicate development, especially taking into account rather rarely released new Spock versions.

If you are here, because of that error visible in your build, jump to the proper section to see what can be done about the problem.

Possible options

Spock 1.3 and Groovy 3.0

Unfortunately, you are out of luck. If you stuck with Spock 1.3 (e.g. migration to Spock 2.0 is blocked for some reasons) there are two (mediocre) options:

Switch to Spock 1.3-SNAPSHOT. The version compatibility check is relaxed in the SNAPSHOT builds of Spock 1.3. What’s more, there were multiple reports that (in the majority of cases - some corner cases remains) Spock 1.3-groovy-2.5-SNAPSHOT works fine with Groovy 3.0.x.
There are no plans to release any new maintenance release of Spock 1.x. However, you can still create a new issue on GitHub and try to convince The Spock Developers to make 1.3.1 with that change :-).

Spock 2.0+ and Groovy 3.0

The Spock compiler plugin refused to run because Spock 2.0-groovy-2.5 is not compatible with Groovy 3.0.9.
For more information (including enforce mode), see http://docs.spockframework.org (section 'Known Issues'). 
Spock location: .../spock-2.0-groovy-2.5.jar
Groovy location: .../groovy-3.0.9.jar

That a trivial case. Just change Spock version from 2.0-groovy-2.5 to 2.0-groovy-3.0, and you are set.

Spock 2.0/2.1 and Groovy 4.0

The Spock compiler plugin refused to run because Spock 2.0-groovy-3.0 is not compatible with Groovy 4.0.0-beta-2.
For more information (including enforce mode), see http://docs.spockframework.org (section 'Known Issues'). 
Spock location: .../spock-2.0-groovy-3.0.jar
Groovy location: .../groovy-4.0.0-beta-2.jar

This variant assumes that you use Groovy 4 (SNAPSHOT, beta or a regular version) and Spock 2.0-groovy-3.0 (or 2.1). The error message is even more precise than in Spock 1.3, suggesting the ‘Known Issues’ section in the Spock documentation. We can leverage a brand-new system property spock.iKnowWhatImDoing.disableGroovyVersionCheck, introduced right in Spock 2.0-M3, which allows to disable the version check - of course, at your own risk :-).

Btw, if you in a hurry (e.g. just to test something), it is also possible just to switch Spock to SNAPSHOT version which accepts (almost) any Groovy version.

Btw2, if you prefer to just jump into code, there is a code repository with working examples for Gradle and Maven.

Gradle

As Groovy 4 changed its groupId from org.codehaus.groovy to org.apache.groovy it is required to add implicit exclusion for spock-2.0-groovy-3.0 transitive dependencies:

testImplementation("org.spockframework:spock-core:2.0-groovy-3.0") {
    exclude group: 'org.codehaus.groovy'
}

to avoid conflicting Groovy versions on a classpath (it’s nice that Gradle is able to detect that!):

> Task :compileTestGroovy FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':compileTestGroovy'.
> Could not resolve all files for configuration ':testCompileClasspath'.
   > Could not resolve org.apache.groovy:groovy:4.0.0-beta-2.
     Required by:
         project :
         project : > org.apache.groovy:groovy:4.0.0-beta-2 > org.apache.groovy:groovy-bom:4.0.0-beta-2
      > Module 'org.apache.groovy:groovy' has been rejected:
           Cannot select module with conflict on capability 'org.codehaus.groovy:groovy:4.0.0-beta-2' also provided by [org.codehaus.groovy:groovy:3.0.8(compile)]
   > Could not resolve org.codehaus.groovy:groovy:3.0.8.
     Required by:
         project : > org.spockframework:spock-core:2.0-groovy-3.0
      > Module 'org.codehaus.groovy:groovy' has been rejected:
           Cannot select module with conflict on capability 'org.codehaus.groovy:groovy:3.0.8' also provided by [org.apache.groovy:groovy:4.0.0-beta-2(groovyApiElements)]

It won’t be required for Spock 2.0-groovy-4.0, once available.

To be able use Groovy 4, Gradle 6.7+ is required.

With earlier Gradle version you might got some misleading errors about not resolvable dependency to org.codehaus.groovy:groovy:4.0.0-beta-2, even though you configured org.apache.groovy:groovy:4.0.0-beta-2 with the correct, changed group id.

Execution failed for task ':compileTestGroovy'.
> Could not resolve all files for configuration ':detachedConfiguration1'.
   > Could not find org.codehaus.groovy:groovy:4.0.0-beta-2.

The obvious way to add system properties to the Gradle execution itself is:

./gradlew check -Dspock.iKnowWhatImDoing.disableGroovyVersionCheck=true     //wrong way

Unfortunately it doesn’t work:

> Task :compileTestGroovy FAILED
startup failed:
Could not instantiate global transform class org.spockframework.compiler.SpockTransform specified at jar:file:/home/foobar/.m2/repository/org/spockframework/spock-core/2.0-groovy-3.0/spock-core-2.0-groovy-3.0.jar!/META-INF/services/org.codehaus.groovy.transform.ASTTransformation  because of exception org.spockframework.util.IncompatibleGroovyVersionException: The Spock compiler plugin cannot execute because Spock 2.0-groovy-3.0 is not compatible with Groovy 4.0.0-beta-2. For more information (including enforce mode), see http://docs.spockframework.org (section 'Known Issues').
Spock artifact: file:/home/foobar/.m2/repository/org/spockframework/spock-core/2.0-groovy-3.0/.../spock-core-2.0-groovy-3.0.jar
Groovy artifact: file:/home/foobar/.gradle/caches/modules-2/files-2.1/org.apache.groovy/groovy/4.0.0-beta-2/.../groovy-4.0.0-beta-2.jar

1 error


FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':compileTestGroovy'.
> Compilation failed; see the compiler error output for details.

After some searching on the Internet, we can find out that tests in Gradle are (by default) executed in a separate JVM, to it is required to pass the aforementioned system property to the test task:

1
2
3

test {
    systemProperty "spock.iKnowWhatImDoing.disableGroovyVersionCheck", "true"   //doesn't work
}

However, it still fails:

> Task :compileTestGroovy FAILED
startup failed:
Could not instantiate global transform class org.spockframework.compiler.SpockTransform specified at ...

Hmm, maybe it would be better to refer the Gradle test task by type?

1
2
3

tasks.withType(Test).configureEach {
    systemProperty "spock.iKnowWhatImDoing.disableGroovyVersionCheck", "true"   //doesn't work either
}

It fails with the same error message :-/. What is wrong?

The key difference - quite easy to miss - is the fact the exception is thrown not during the test execution, but during the compilation phrase. Then, the main Spock AST transformation checks if it could be applied:

> Task :compileTestGroovy FAILED   <--------- :compileTestGroovy not :test !
startup failed:
...

It is definitely less popular case, and I had a problem to find any sensible source of wisdom proposing a complete solution. One of the possible option (which I talk about deeper in the article about enabling preview features of the new Java versions) is CompileOptions (e.g. options.compilerArgs.add('--enable-preview') in the GroovyCompile task. The second might be GroovyCompileOptions also in GroovyCompile. However, it does not bring expected result. After some digging, I was able to find out, that as the Groovy compilation is performed in the forked JVM and the right way to achieve the goal is:

1
2
3

tasks.withType(GroovyCompile).configureEach {
    options.forkOptions.jvmArgs << '-Dspock.iKnowWhatImDoing.disableGroovyVersionCheck=true'    //it works! But be aware that "options" not "groovyOptions"!
}

And it works as expected:

> Task :compileTestGroovy
Executing Spock 2.0-groovy-3.0 with NOT compatible Groovy version 4.0.0-beta-2 due to set spock.iKnowWhatImDoing.disableGroovyVersionCheck system property set. This is unsupported and may result in weird runtime errors!

> Task :test
...

displaying a nice disclaimer :). Spock 2.0-groovy-3.0 is used together with Groovy 4.

Btw, if you see just not very helpful error message:

Could not instantiate global transform class org.spockframework.compiler.SpockTransform specified at
jar:file:/.../spock-core-.../org.codehaus.groovy.transform.ASTTransformation
because of exception java.lang.reflect.InvocationTargetException

it is caused by GROOVY-9469 which affects Groovy 3 and was fixed in 3.0.3. Upgrade Groovy to see more verbose error message.

Maven

As Groovy 4 changed its groupId from org.codehaus.groovy to org.apache.groovy it is required to add implicit exclusion for spock-2.0-groovy-3.0 transitive dependencies:

<dependencies>
    <dependency>
        <groupId>org.spockframework</groupId>
        <artifactId>spock-core</artifactId>
        <version>${spock.version}</version>
        <scope>test</scope>
        <exclusions>
            <!-- to do not have conflicting Groovy versions - Groovy 4 changed groupId from "org.codehaus.groovy" to "org.apache.groovy" -->
            <exclusion>
                <groupId>org.codehaus.groovy</groupId>
                <artifactId>*</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.apache.groovy</groupId>
        <artifactId>groovy</artifactId>
        <version>4.0.0-beta-2</version>
    </dependency>
</dependencies>

to have your tests executed with Groovy 4 instead of (silently) with Groovy 3, taken from a transitive dependency of Spock 2.0-groovy-3.0!

[INFO] --- gmavenplus-plugin:1.13.0:compileTests (default) @ spock2-groovy4-maven ---
[INFO] Using isolated classloader, without GMavenPlus classpath.
[INFO] Using Groovy 3.0.8 to perform compileTests.     <---- Without exclusion, it is executed "silently" with old Groovy! 
[INFO] Parallel parsing disabled.
[INFO] Compiled 1 file.

It won’t be required for Spock 2.0-groovy-4.0, once available.

Writing my Spock, Groovy and modern Java related tutorials, I try to cover also the Maven users, which still is quite popular build tool :-). Usually, it takes much more (XML) lines to achieve the same in Maven in comparison to Gradle, however, sometimes it just works flawlessly without any additional effort. Unfortunately, it was not the case this time.

The build with Spock-2.0-groovy-3.0 and Groovy 4.0.0 by default fails with:

[ERROR] Failed to execute goal org.codehaus.gmavenplus:gmavenplus-plugin:1.8.1:compileTests (default) on project spock2-migration-maven: Error occurred while calling a method on a Groovy class from classpath. InvocationTargetException: startup failed:
[ERROR] Could not instantiate global transform class org.spockframework.compiler.SpockTransform specified at jar:file:/home/foobar/.m2/repository/org/spockframework/spock-core/2.0-groovy-3.0/spock-core-2.0-groovy-3.0.jar!/META-INF/services/org.codehaus.groovy.transform.ASTTransformation  because of exception org.spockframework.util.IncompatibleGroovyVersionException: The Spock compiler plugin cannot execute because Spock 2.0-groovy-3.0 is not compatible with Groovy 4.0.0-beta-2. For more information (including enforce mode), see http://docs.spockframework.org (section 'Known Issues').
[ERROR] Spock artifact: file:/home/foobar/.m2/repository/org/spockframework/spock-core/2.0-groovy-3.0/spock-core-2.0-groovy-3.0.jar
[ERROR] Groovy artifact: file:/home/foobar/.m2/repository/org/apache/groovy/groovy/4.0.0-beta-2/groovy-4.0.0-beta-2.jar
[ERROR] 
[ERROR] 1 error
[ERROR] -> [Help 1]

To make the long story short, I started with the plain Maven properties:

<project>
...
  <properties>
     <!-- doesn't work -->
     <spock.iKnowWhatImDoing.disableGroovyVersionCheck>true</spock.iKnowWhatImDoing.disableGroovyVersionCheck>
  </properties>
...
</project>

It fails. Next, I added the following to the compiler parameters:

    <!-- ... -->
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.1</version>
                <configuration>
                    <!-- doesn't work -->
                    <compilerArgs>
                        <arg>-v</arg>
                        <arg>-Dspock.iKnowWhatImDoing.disableGroovyVersionCheck=true</arg>
                    </compilerArgs>
                </configuration>
            </plugin>
            <!-- ... -->
        </plugins>
    </build>
    <!-- ... -->

however, it doesn’t affect Groovy compilation itself. I even took a look at the gmavenplus plugin parameters and Groovy’s CompilerConfiguration , but without any effect. In the end, I was able to achieve that with another plugin properties-maven:

    <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>properties-maven-plugin</artifactId>
        <version>1.0.0</version>
        <executions>
            <execution>
                <goals>
                    <goal>set-system-properties</goal>
                </goals>
                <configuration>
                    <properties>
                        <!-- works fine -->
                        <property>
                            <name>spock.iKnowWhatImDoing.disableGroovyVersionCheck</name>
                            <value>true</value>
                        </property>
                    </properties>
                </configuration>
            </execution>
        </executions>
    </plugin>

resulting in:

[INFO] --- gmavenplus-plugin:1.8.1:compileTests (default) @ spock2-migration-maven ---
[INFO] Using isolated classloader, without GMavenPlus classpath.
[INFO] Using Groovy 4.0.0-beta-2 to perform compileTests.
Executing Spock 2.0-groovy-3.0 with NOT compatible Groovy version 4.0.0-beta-2 due to set spock.iKnowWhatImDoing.disableGroovyVersionCheck system property set. This is unsupported and may result in weird runtime errors!
[INFO] Compiled 10 files.
[INFO] 
[INFO] --- maven-surefire-plugin:3.0.0-M4:test (default-test) @ spock2-migration-maven ---
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
...

It turned out to be quite easy (if you know what to use :-) ), but - unless I missed something - it is somehow not much covered in the existing tutorials or documentation.

Potential issue in IntelliJ IDEA

Even though, having your up-and-running Maven project configured as above, you might still encounter the incompatibility error after importing it in Idea:

Groovyc: While compiling [tests of spock2-groovy4-gradle.test]: Could not instantiate global transform class org.spockframework.compiler.SpockTransform specified at jar:file:/home/…/spock-core-2.0-groovy-3.0.jar!/META-INF/services/org.codehaus.groovy.transform.ASTTransformation because of exception java.lang.NullPointerException: Cannot invoke “java.net.URL.toString()” because the return value of “java.security.CodeSource.getLocation()” is null

~~The problem is caused by a limitation of Idea (IDEA-287642 ), which does not allow providing custom system properties for groovyc compilation (just for javac). You might want to vote for that issue.~~ In fact, as Paul King pointed out, there is a way to set that property for all the compilers used in the project with Settings → Compiler → Shared build process VM options. In the context of Spock, it should be enough.

The error itself is not very meaningful (NullPointerException) which in turn is caused by an unexpected behavior of Idea, reported as IDEA-287643 .

It is worth to mention, the same problem might occur also with Gradle, once “Build and run” is manually set to “Idea” (instead of “Gradle” - default behavior, but not the fastest one).

Summary

The main reason, I implemented that feature in Spock was development of Groovy 4. I didn’t feel comfortable seeing The Groovy Team to be forced to use the Spock snapshot version in the master (Groovy 4.x) builds. With Spock-2.0-M3 it was simplified . The same construction can be used in your project while playing with Groovy 4 (in beta as of XI 2021), in the way I precised for Gradle and Maven (which in turn can be really confusing on their own way which motivated me to write this short post :-) ). There is a plan to add the official support for Groovy 4 in Spock, once it is closer to Groovy 4.0.0-final.

Btw, there is a code repository with working examples for Gradle and Maven.

P.S. If you bumped into that topic during Spock 1.3 -> 2.0 migration, there is a dedicated migration guide article .

Updated 2021-01-31. Added section about potential issue with IntelliJ IDEA.
Updated 2021-02-02. Add workaround suggested by Paul King about the potential issue with IntelliJ IDEA.

Lead photo by geralt, published in Pixabay, Pixabay License.

What happened to Groovy dependencies in Spock 2.0?

Marcin Zajączkowski — Tue, 16 Jun 2020 10:00:00 +0200

Spock 2.0(-M3) reduced number of provided Groovy dependencies. If you started to get various Groovy-related ClassNotFoundExceptions after migration and wonder why and what to do, read on.

Historical background

Originally Groovy was just one jar to rule them all. However, in process of time, that jar became bigger and bigger. Groovy 2 was the first version promoting the usage of (several at that time) Groovy modules, as a supplemental to groovy.jar. However the adoption of that approach was slowly and groovy-all remained prevalent in pom.xml and build.gradle files.

What even worse, having some projects use groovy.jar plus some modules (e.g. groovy-json.jar or groovy-xml.jar) while some other preferring good old groovy-all.jar could lead to version clashes as for Maven or Gradle those were independent packages. To avoid that it was quite common to see the defensive Groovy exclusions:

<dependencies>
    <dependency>
        <groupId>org.codehaus.gmaven.runtime</groupId>
        <artifactId>YYY</artifactId>
        <version>1.2</version>
        <exclusions>
            <exclusion>
                <groupId>org.codehaus.groovy</groupId>
                <artifactId>*</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.codehaus.groovy</groupId>
        <artifactId>groovy-all</artifactId>
        <version>${groovy.version}</version>
    </dependency>
</dependencies>

and declare all required Groovy dependencies on their own.

Groovy 2.5 switched to the model of Maven’s BOM (Bill Of Materials) where groovy-all isn’t a binary jar itself, but rather has the official Groovy modules as dependencies. Thanks to that, Maven and Gradle have easier to keep the versioning of the particular modules consistent. This holds for Groovy 3 which still provides the groovy-all BOM.

Groovy dependencies in Spock 1.3

Spock 1.2 switched from groovy-all.jar to groovy.jar + modules. Unfortunately due to historical reasons it was a lot of modules:

testRuntimeClasspath - Runtime classpath of source set 'test'.
+--- org.spockframework:spock-core:2.0-M2-groovy-3.0
|    +--- org.codehaus.groovy:groovy:3.0.0
|    +--- org.codehaus.groovy:groovy-json:3.0.0
|    |    \--- org.codehaus.groovy:groovy:3.0.0
|    +--- org.codehaus.groovy:groovy-nio:3.0.0
|    |    \--- org.codehaus.groovy:groovy:3.0.0
|    +--- org.codehaus.groovy:groovy-macro:3.0.0
|    |    \--- org.codehaus.groovy:groovy:3.0.0
|    +--- org.codehaus.groovy:groovy-templates:3.0.0
|    |    +--- org.codehaus.groovy:groovy-xml:3.0.0
|    |    |    \--- org.codehaus.groovy:groovy:3.0.0
|    |    \--- org.codehaus.groovy:groovy:3.0.0
|    +--- org.codehaus.groovy:groovy-test:3.0.0
|    |    \--- org.codehaus.groovy:groovy:3.0.0
|    +--- org.codehaus.groovy:groovy-sql:3.0.0
|    |    \--- org.codehaus.groovy:groovy:3.0.0
|    +--- org.codehaus.groovy:groovy-xml:3.0.0 (*)
|    \--- org.junit.platform:junit-platform-engine:1.5.2
|         ...
...

As a result the following was quite common:

test('org.spockframework:spock-core:1.3-groovy-2.5') {
    exclude group: 'org.codehaus.groovy'    //get rid of all dependant groovy modules
}
test 'org.codehaus.groovy:groovy:2.5.11'    //apply only those needed
test 'org.codehaus.groovy:groovy-json:2.5.11'
test 'org.codehaus.groovy:groovy-templates:2.5.11'

However, it turned out that in fact all those extra modules are not needed internally for spock-core and they were rather provided as a convenience for end users.

Groovy dependencies in Spock 2.0+

Starting with Spock 2.0-M3 the Groovy dependencies were simplified to just groovy.jar. Thanks to that you safe some bandwidth on the initial build (what is well-timed ;-) ) and the CI builds without proper caching take slightly shorter.

testCompileClasspath - Compile classpath for source set 'test'.
+--- org.spockframework:spock-core:2.0-M3-groovy-3.0
|    +--- org.codehaus.groovy:groovy:3.0.4
|    +--- org.junit.platform:junit-platform-engine:1.7.1-M1
|         ...
...

As a downside, after the migration to Spock-2.0-M3+, in rare cases you might get ClassNotFoundException: groovy.sql.Sql, ClassNotFoundException: groovy.json.JsonSlurper or ClassNotFoundException: groovy.util.XmlParser. That error suggest that you use those classes somewhere in your tests, but the corresponding dependency is not declared implicitly in your build configuration (and it’s not accidentally provided by spock-core any longer). The quick fix is to add org.codehaus.groovy:groovy-sql:3.0.X, org.codehaus.groovy:groovy-json:3.0.X or similar explicitly.

P.S. If you bumped into that topic during Spock 1.3 -> 2.0 migration, there is a dedicated migration guide article .

Lead photo by Tumisu, published in Pixabay, Pixabay License.

Spock vs JUnit 5 - the ultimate feature comparison

Marcin Zajączkowski — Wed, 15 Apr 2020 10:00:00 +0200

What is the best testing framework for Java code nowadays? Spock or JUnit 5? Check it out in this ultimate feature comparison.

Spock was a game changer for all the people struggling with unit testing in JUnit 4. Compact syntax, parameterized tests or flexibility to mention just a few advantages. Over 10 years after JUnit 4.0, the brand new, written from scratch, Java 8 optimized Junit 5 has been released. And time has not stopped then.

In this blog post, I will compare selected areas of Spock and JUnit 5 to give you an overview how the situation looks like nowadays. I will try to answer the question if its time for Spock to fade into oblivion or maybe quite the opposite it is still light years ahead of JUnit 5.

This blog post is a written version of my presentation titled “Spock vs JUnit 5 - Clash of the Titans”. If you prefer, feel free to see the slides or watch the video instead.

Historical background

As a warm-up, let’s take a look at the important dates in the automatic code testing in Java.

2000 - JUnit - first xUnit for Java
2004 - TestNG - Java 5 (annotations) leveraged in tests
- with some unique (at the time) features (such as parameterized tests, test repetition or test dependency)
2006 - JUnit 4 - Java 5 support
- catching up TestNG features over the next few years
- de facto standard (the world chose :-/ ), steady evolution rather than revolution
2009 - Spock - revamped test creation
- fresh ideas and power of Groovy under the hood
2017 - JUnit 5 - redesigned and rewritten from scratch
- new king of the hill? - let’s check it out!

This comparison covers the latest released stable versions (as of April 2020) of JUnit (5.6) and Spock (1.3). I am fully aware of in-development Spock 2.0 and I plan to update this comparison (or write a supplement post) once 2.0-final is available.

Development comparison

Let’s start with a comparison of some biased aspects of development of JUnit 5 and Spock.

	JUnit 5 (as of 5.6)	Spock (as of 1.3)
inception year	2015	2009
number of GitHub stars	3,7K	~2,6K
number of commits	~6K	~2,5K
development activity	high (young project)	medium (mature project)
number of active committers	3	1 + 3
number of contributors (ever)	~130	~80

The first thing to notice is the fact that JUnit 5 is much younger project with very high development activity. There are 3 active committers with a bunch of external contributors.

One the other hand, Spock is much more mature with lower development activity. However, many of the features developed in JUnit 5 recently were already available in Spock. Therefor, the need for lots of commit in Spock is lower. Spock has currently one main maintainer and 2-3 people regularly contributing to the project (of course also with a pack of external contributors).

Partial verdict

Development: JUnit 5 wins

Reason: Mainly thanks to more active development activity in recent years.

Tool support

The next thing to cover is Java 11 compatibility and support in tools.

	JUnit 5	Spock
Java 11+	very good	good (with modern Groovy 2.5.x)
IntelliJ IDEA	built-in	built-in (some Groovy limitations)
Eclipse	build-in	built-in (some Groovy limitations)
Netbeans	10.0+ (Maven only)	unknown
Maven	plugin (official, Surefire)	plugin (GMavenPlus for Groovy)
Gradle	built-in	built-in
SonarQube	built-in	plugin (official for Groovy)
PIT - mutation testing	plugin (official)	build-in

Both of the frameworks are currently very well supported in the Java ecosystem. However, it is worth to mention that tests (specifications) in Spock are written in Groovy (which notabene is the main power of Spock). As a downside, due to much more dynamic nature of Groovy as a language, IDEs have much harder time to provide as good support for it as for Java (e.g. refactoring or compile time error reporting).

Regarding Java 11 and Spock, with modern Groovy 2.5.x it should work flawlessly. In general, most of the things should work fine also with Java 12, 13 and (with Groovy 2.5.10+) also 14. The official Java 14+ support should be available in Spock 2.0 (still in development) with Groovy 3.0.

On the other hand, JUnit 5 is a role model on that field. It is continuously tested on a CI server with the recent Java versions (including the early access builds, such as OpenJDK 15 EA).

Partial verdict

Tool support: JUnit 5 wins

Reason: Great IDE support for Java code and continuous testing with the bleeding edge versions of Java.

Test structure

Let’s move on to less technical aspects of test development - test structure.

As die-hard followers of my blog may know, I am a big propagator of well structured automatic tests with the BDD-originated given-when-then concept. In short, it unifies test creation, improves their readability and makes it easier to write, especially for less experienced people. To learn more, please take a loot at a separate blog post that I wrote some time ago.

JUnit 5

class SimpleCalculatorTest {
    @Test
    void shouldAddTwoNumbers() {
        //given
        Calculator calculator = new Calculator();
        //when
        int result = calculator.add(1, 2);
        //then
        assertEquals(3, result);
    }
}

The given-when-then sections are marked just by plain comments in code. It is not perfect. They might be forgotten and are easy to lost/misplace during refactoring. Could it be done better?

Spock

class SimpleCalculatorSpec extends Specification {
    def "should add two numbers"() {
        given:
            Calculator calculator = new Calculator()
        when:
            int result = calculator.add(1, 2)
        then:
            result == 3
    }
}

Well, in Spock the (almost not used in Java) label construction is leveraged. It is crucial to highlight that they are not just comments in code - they are a part of the specification. For example, having when: removed in the example above, generates real compilation errors. Nice.

Btw, of course, in Spock it is possible to write just one-liners, if feasible. However, developers have to do it intently.

Partial verdict

Test structure: Spock wins

Reason: Spock cares more about well structured (and more readable) tests.

Btw, with both JUnit 5 and Spock it is worth to use code templates to generate test skeleton to fill instead of writing all those given-when-then by hand :-).

Exception testing

Another topic I want to cover is exception testing. In general, to test corner cases (alternative scenarios) it is needed to verify that an exception with a given type was thrown. Usually, it is followed by some extra assertions such as an exception message and a cause.

JUnit 5

In JUnit 5, it is no longer possible to use @Test(expected = NullPointerException.class), useful for one-liners, but potentially risky with more complicated test code. Instead of, we have the assertThrows() construction, similar to catchThrowable() from AssertJ.

@Test
void shouldThrowBusinessExceptionOnCommunicationProblem() {
    //when
    Executable e = () -> client.sendPing(TEST_REQUEST_ID)
    //then
    CommunicationException thrown = assertThrows(CommunicationException.class, e);
    assertEquals("Communication problem when sending request with id: " + TEST_REQUEST_ID,
                 thrown.getMessage());
    assertEquals(TEST_REQUEST_ID, thrown.getRequestId());
}

The syntax is very compact (thanks to the lambda expression which is used to catch an exception). In addition, it is easy to perform any required further assertion(s) on the returned exception instance.

Spock

Spock, on the other hand, still provides @FailsWith(NullPointerException) for one-liners. However, it is almost completely not used on a daily basis. All thanks to the thrown() construction.

def "should capture exception"() {
    when:
        client.sendPing(TEST_REQUEST_ID)
    then:
        CommunicationException e = thrown()
        e.message == "Communication problem when sending request with id: $TEST_REQUEST_ID"
        e.requestId == TEST_REQUEST_ID
}

It is worth to notice that there is no lambda expression to catch an exception. To implement that Spock leverages Groovy AST transformations which under the hood transparently add required code. As a result, the exception thrown in the when block is caught and returned from the thrown() method. Really smart.

In addition, it is nice that in Spock, there is a smart type inference. It is not needed to precise the exception type twice. Based on the variable type on the left side, Spock is able to create the proper try-catch block.

As a bonus, there as a separate utility class Exceptions in Spock which provides extra assertions to play with the cause chain.

Partial verdict

Exception testing: Spock wins

Reason: Even shorter and less intrusive mechanism for exception testing.

Conditional test execution

In multiple situation it is required to (not) execute a given test only if given condition is (not) met.

The most common cases include:

particular Java version
- reflection hack to use newer version features
- compatibility testing for lower version
specific operating system
- notifications about changed files on Mac are much delayed
- testing symlinks on Windows makes no sense
tests executed only on CI server, stage environment, …

JUnit 5

JUnit 5 introduced brand new support for annotation-based @Enabled* and @Disabled* conditions.

There is a predefined set of conditions for:

JVM version
operating system
system property
environment variable

For example:

@Test
@DisabledOnOs(OS.WINDOWS)
void shouldTestSymlinksBasedLogic() {
    ...
}

@Test
@EnabledIfSystemProperty(named = "os.arch", matches = ".*32.*")
void shouldBeRunOn32BitSystems() {
    ...
}

The JVM version and operating system conditions are enums and thanks to that, there is a nice code completion provided by an IDE. In addition the conditions can be used as meta-annotations to make more complex scenarios easier to use in multiple places.

@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
@Test
@EnabledOnOs(WINDOWS)
@EnabledIfSystemProperty(named = "os.arch", matches = ".*32.*")
@interface TestOn32BitWindows {
}

With the following usage:

@TestOn32BitWindows
void shouldDoStrangeThings1OnLegacyWindows() { ... }

@TestOn32BitWindows
void shouldDoStrangeThings2OnLegacyWindows() { ... }

Originally, there was also support for custom logic in script-based inline conditions. However, due to limited usability (code written in String) and deprecation of Nashorn in Java 11 it has been removed in JUnit 5.6. Writing own condition logic currently requires a custom implementation of ExecutionCondition.

Spock

Since time immemorial, Spock has provided support for annotation based conditions with @Requires and @IgnoreIf. Out of box it is possible to check:

JVM version
operating system
system property
environment variable

The same set implemented later on in JUnit 5. However, the usage is slightly different.

@IgnoreIf({ !jvm.java8Compatible })
def "should return empty Optional by default for unstubbed methods with Java 8+"() { ... }

@Requires({ sys["targetEnvironment"] != "prod" })
def "should execute smoke testing on non production environment"() { ... }

Instead of enums, the logic is written in Groovy Closures leveraging delegation to the jvm/os/sys/... objects providing a dedicated methods (such as isJava11Compatible() or isLinux()). Unfortunately there is no code completion available by default, but it can be enabled with a small trick (Update. Just please be aware that the linked slides are from 2016 - Spock 1.0/1.1 - and in some places are outdated).

Using Closures provides one more important benefit - custom logic can implemented inline with pure Groovy.

1
2

@Requires({ isStrongCryptographyEnabled() })    //custom static method
def "should test strong cryptography-based features"() { ... }

Partial verdict

Conditional test execution: Spock wins

Reason: Higher flexibility with non standard conditions.

Mocking

Mocking is a crucial part of automatic code testing. It is especially beneficial in unit testing, but can be also usable in integration testing (e.g. with Spring Framework).

JUnit 5

In JUnit 5 tests, Mockito is the first port of call when it comes to mocking.

@Test
public void should_not_call_remote_service_if_found_in_cache() {
    //given
    given(cacheMock.getCachedOperator(CACHED_MOBILE_NUMBER)).willReturn(Optional.of(PLUS));
    //when
    service.checkOperator(CACHED_MOBILE_NUMBER);
    //then
    then(webserviceMock).should(never()).checkOperator(CACHED_MOBILE_NUMBER);
//   verify(webserviceMock, never()).checkOperator(CACHED_MOBILE_NUMBER);   //alternative syntax
}

Mockito is an industrial standard for mocking in Java. It is very mature, well known with a rich set of features. In addition, Spring Boot provides the official support for injecting Mockito’s mocks and spies into its context, which can be very useful in integration tests.

Spock

Spock on the other hand contains a built-in mocking subsystem.

def "should not hit remote service if found in cache"() {
    given:
        cacheMock.getCachedOperator(CACHED_MOBILE_NUMBER) >> Optional.of(PLUS)
    when:
        service.checkOperator(CACHED_MOBILE_NUMBER)
    then:
        0 * webserviceMock.checkOperator(CACHED_MOBILE_NUMBER)
}

While the syntax in Mockito is pretty readable, there are all those given/thenReturn/thenAnswer/... to write. Spock, however, leverages the power of operator overloading in Groovy (and of course its AST transformations) to make stubbing and verification as simple as possible.

given(cacheMock.getCachedOperator(CACHED_MOBILE_NUMBER)).willReturn(Optional.of(PLUS)); //Mockito
// vs
cacheMock.getCachedOperator(CACHED_MOBILE_NUMBER) >> Optional.of(PLUS) //Spock

verify(webserviceMock, never()).checkOperator(CACHED_MOBILE_NUMBER); //Mockito
// vs 
0 * webserviceMock.checkOperator(CACHED_MOBILE_NUMBER) //Spock

In a basic case, it’s enough to write a method, use an operator and provide a returned value. In verification it’s just a method call with requested cardinality. Most likely, it will not be possible to achieve that in Java in the predictable future.

Historically, the main limitation with Spock’s mocks was their tight coupling with the specification (the test itself). As a result it was not possible to use Spock’s stubs, mocks and spies in the Spring context for integration tests. However, starting with Spock 1.1 it was relaxed and Spock 1.2 introduced the first class support for mocking and spying with Spring. As a bonus, it is available also for pure Spring context (Spring Boot is not required as it takes place with Mockito’s mocks).

To be fair, it is required to admit that more magic in Spock generates some quirks with some corner cases. Nevertheless, once knowing all of them the readability and compactness definitely pay off :-).

One more thing. It is worth to remember that with Spock used for testing, we are not forced to use it also for mocking. It is perfectly fine to use Mockito if wanted (or needed).

Partial verdict

Mocking: Spock wins

Reason: The syntax and really readable and compact. And with Spring support in Spock 1.2+ the last big limitation of the mocking subsystem in Spock was removed.

Parameterized tests

Parameterized tests in general provide a way to call one test with different set of input (and expected) parameters. It can dramatically reduce duplication in specific cases (e.g. in testing with the user defined acceptance data set).

A word of warning here. In some scenarios, using parameterized tests can hide specific business use cases which would be clearly exposed in our non-parameterized tests otherwise. There are techniques to avoid that, but it is a topic for another article.

JUnit 5

Parameterized tests were one of the key features of TestNG from its early days. In JUnit 4 the feature has been not available for year and even once implemented, the design made it very unpractical to use in most of the cases. To improve the situation the 3rd-party runners were available, but it was still sad (for JUnit 4).

Luckily, JUnit 5 is the first version of JUnit with sane built-in parameterized tests support.

@ParameterizedTest
@CsvSource({ "dog, 3", "wolf, 4", "hippopotamus , 12" })
void testStringLengthWithParameterizedTestInJUnit5(String word, int expectedNumberOfLetters) {
    assertEquals(word.length(), expectedNumberOfLetters);
}

For simple scenario with inline values (provided as CSV strings), there is a nice implicit argument conversion from String for various types (e.g. dates, files, path or UUID). It just requires to add @ParameterizedTest and @CsvSource (or @ValueSource, @EnumSource, etc.).

However, not everything can be put into string. The commonly encountered scenario is an object creation or calling a method defined in a test fixture. To support that JUnit 5 gives an ability to call a local method (or the one from external object) to return required input/output arguments. They should be provided as Stream (or List) of Arguments.

@ParameterizedTest(name = "value to pay for invoice {0} should be {1}")
@MethodSource("invoiceProvider")
void shouldCalculateToPayValueForInvoice(Invoice invoice, BigDecimal expectedValueToPay) {
    //when
    int valueToPay = invoice.toPayValue();
    //expect
    assertEquals(expectedValueToPay, valueToPay);
}

private static Stream<Arguments> invoiceProvider() {
    return Stream.of(
            Arguments.of(regularInvoice(), 54),
            Arguments.of(overduedInvoice(), 81),
            Arguments.of(paidInvoice(), 0)
    );
}

It is just required to use @MethodSource to point the method (or use a naming convention to resolve the correct method name automatically).

The thing I don’t like is a lack of the arguments type check. In tests with multiple parameters it can be quite confusing to determine which is which.

As a bonus it is also possible to customize a test name which will be displayed in the report (in IDE, HTML report, etc.).

Spock

In Spock, first let’s take a longer look at the simple parameterized test.

@Unroll
def "should sum two integers (#x + #y = #expectedResult)"() {
    when:
        int result = calculator.add(x, y)
    then:
        result == expectedResult
    where:
         x |  y || expectedResult
         1 |  2 ||  3
        -2 |  3 ||  1
        -1 | -2 || -3
}

The words that come to my mind are “state of the art”. This is a place where Spocks excels.

With table-like formatting after the where keyword, input data looks very natural and is easy to read. What is worth mentioning, there is no need to define x, y, expectedResult as method arguments in a test (as it takes place in JUnit 5). The variables are added implicitly with full visibility and type inference in IDE.

The method name can be parameterized inline with the #var syntax (it’s Groovy - methods can have spaces and other “strange” characters).

In Spock it is also not a problem to use table-like syntax in a situation where constructor or methods are needed to be called (which in JUnit 5 requires using a separate provider method).

@Unroll("Value to pay for invoice #invoice should be #expectedValueToPay")
def "should calculate value to pay for invoice"() {
    expect:
        invoice.valueToPay() == expectedValueToPay
    where:
        invoice           || expectedValueToPay
        regularInvoice()  || 54
        overduedInvoice() || 81
        paidInvoice()     || 0
}

While working with acceptance testing where results are presented to less technical users, it is also possible to distinguish a test name for developers with a scenario name for QA engineers (or product owners) by writing it in the @Unroll annotation.

Of course, for more complex/advanced use cases (e.g. generating input parameters dynamically based on exterenal source), data pipes and data providers can be used. They are not as natural as the aforementioned table, but are very powerful.

@Unroll("#pesel is valid (#dbId)")
def "should validate PESEL correctness (CSV)"() {
    expect:
        sut.validate(pesel)
    where:
        [dbId, _, _, pesel] << readValidPeopleFromCSVFile()
                                   .readLines().collect { it.split(',') }
                                   //ugly way to read CSV - don't do this :-)
}

Partial verdict

Parameterized tests: Spock wins

Reason: It is enough to visually compare parameterized tests in those two frameworks.

Migration

The next point is very important for people currently using JUnit 4.

JUnit 4 → JUnit 5

The JUnit 5 tests can co-exist with the JUnit 4 tests. There is a dedicated module junit5-vintage which executes old JUnit 4 tests on the new JUnit Platform (part of JUnit 5).

Writing new tests with JUnit 5 is also easy. The test structure is very similar. We just need to learn some new keywords, annotations and assertions (and remember to use @Test from the new org.junit.jupiter.api package :-) ).

JUnit 4 → Spock

The Spock tests can also co-exists with the JUnit 4 tests. In fact Spock 1.x is a (sophisticated, but still) JUnit 4 runner. However, besides that there is a completely new test structure which we need to get familiar with. It is easy to grasp and, in the end, it is more readable, but still, it is something new to learn.

Partial verdict

Tool support: JUnit 5 wins

Reason: The JUnit 5 tests are more similar to the JUnit 4 tests and therefor, that migration path should be a little bit easier.

One more important migration-related thing here. During my training or consulting , I do not recommend rewriting existing hundreds and thousands of tests from JUnit 4 to the new technology, just for fun. Most of the existing tests will stay untouched for the foreseeable future. It is much better to write the new tests in the new technology and rewrite existing tests only along the way of doing some production logic related changes in them.

Summary - final verdict

Inarguably, JUnit 5 is a great progress over JUnit 4. Many features (previously) unique to Spock now are available in JUnit 5. However, Spock still excels in some areas. The best choice is no longer obvious.

Here, an anecdote, proposed my a colleague of mine. There are BMW and Audi cars. Both brands are very good and the choice usually depends on our personal preferences.

There same with Spock and JUnit 5. First, let’s take a look at the features comparison of the presented testing frameworks.

	JUnit 5 (as of 5.6)	Spock (as of 1.3)
development	very good	good (new features and bugfixes)
learning curve	very good (similar to 4)	good (Groovy to grasp)
tool support	very good (trending up)	good (weaker compile time checks)
test structure	good	very good (BDD by default)
exception testing	good	very good
conditional test execution	good	very good (custom logic)
mocking	good (Mockito)	good+ (very compact, some quirks)
parameterized tests	good	very good (exceptional!)
migration from JUnit 4	very good	good

As you see, they both have their strengths and weaknesses. In addition to covered aspects, you should add to that list other aspects which are important for you (and for your team) and see which framework fits you more.

For instance.

If you prefer:

old good Java as the only language
stability and being mainstream
stronger compile time error checking

you should choose JUnit 5.

On the other hand,

if - like me - you favor:

simplicity and readability
power of Groovy under the hood
beautiful parameterized tests and exception testing

then Spock should be better for you.

One final reflection. Preparing my presentation (and later on that blog post), I realized it would be great to have only that kind of dilemma while crafting software, choose between two pieces of good software :-).

A note about Spock logo. As of earthdate 2020-04, Spock doesn’t have the official logo. The one used in the lead photo is a proposal by Søren Berg Glasius made back in 2014. There are also some other proposals, but if you have a good idea, feel free to propose it there (or vote for existing candidates).

Places where you REALLY NEED to use 'assert' keyword in Spock assertions

Marcin Zajączkowski — Wed, 18 Mar 2020 10:00:00 +0200

Make sure you know all the places where putting the ‘assert’ keyword in Spock is necessary to avoid false sense of security and tests which - effectively - do nothing.

Introduction

Spock greatly simplifies test creation. One of the improvements is an ability to skip the assert keyword in the verification block (then or expect). It is very convenient, but Spock newcomers (and based on my code testing training also even more experienced developers) might not be aware that under some circumstances that keyword is required, making - as a result - that particular test useless. After lecture of this blog post you will know the places (situations) where the assert keyboard is mandatory.

Let’s start from the beginning.

Power assertions

Power assertions is an advanced assertion mechanism which renders the failed assertion in a very meaningful way. For example:

assert (2 + 3) * 4 != (2 * 4) + (3 * 4)

results in:

Assertion failed:

assert (2 + 3) * 4 != (2 * 4) + (3 * 4)
          |    |   |     |    |    |
          5    20  false 8    20   12

This is a great step forward over the classic assertEquals() assertion with just Assertion failed in JUnit.

What is nice, it works also for objects and method calls:

Assertion failed: 

assert ann.name == bob.name && ann.age == bob.age
       |   |    |  |   |    |
       |   Ann  |  |   Bob  false
       |        |  Person(name:Bob, age:7)
       |        false
       Person(name:Ann, age:4)

Power assertions have been developed initially for Spock, but later on they were backported to Groovy 1.7 (2009). Currently, they are one of the killer features of Groovy in the context of testing (in general, not only with Spock).

Optional ‘assert’

As I already mentioned in the introduction, Spock using the Groovy magic (primarily AST transformations and operator overloading) generates and automatically handles different aspect of tests to make its creation as simple as possible. Among others, it is visible in exception testing - a hidden try..catch around the when block or implicit method arguments in parameterized tests.

Another case for simplification in an optional assert keyword with assertions. For example:

    ...
expect:
    assert numberOfItems == 2

can be simplified to just:

    ...
expect:
    numberOfItems == 2

The assert keyword is automatically added by the Spock AST transformation in the then or expect blocks, if there is a boolean value, but also any non-void statement which can be evaluated with The Groovy Truth .

This opens different possibilities:

    ... 
when:
    List<Invoiced> overdueInvoices = userAccount.getOverdueIncoives()
then:
    overdueInvoices   //collection cannot be null or empty to not to fail

Under the hood, the Spock AST transformation detects all those places to enhance them with assert, but there are some limitations.

Required ‘assert’

There are places (situations) where assert cannot be omitted. It is especially crucial to know them as there is no error displayed. The assertions just do not work which makes them useless. Or event worse, it gives a false sense of security - those tests do not test anything.

Guard assertion

The first case is so-called guard assertion. Occasionally, mostly in integration/acceptance testing there is a need to check the initial state (e.g. number of records in a database to ensure than some previously added new - badly written - test did not break our test data).

Spock cares only about then and expect blocks in the context of the implicit assert. The assert keyword is not automatically added in the given block and has to be placed manually.

given:
    List<Invoice> initialInvoices = legacyRepository.getUserInvoices(USER_ID)  
    assert initialInvoices.size() == 0  //'assert' is mandatory - 'initialInvoices.size() == 0' does nothing!
when:
    issueInvoice(...)
then:
    legacyRepository.getUserInvoices(USER_ID).size() == 1   //'assert' is not needed in 'then'

Assertion in method execution

The another situation refers to the then (or expect) block. Sometimes there is a set of assertions that is used in more than one tests. It might be handy to extract them to a separate method (placed locally in the test class, in a super class, or in a trait):

    when:
        ...
    then:
        assertCompatibilityWithArticle17OfEUDirective(socialMediaSubmission)
}

private void assertCompatibilityWithArticle17OfEUDirective(SocialMediaSubmission submission) {
    assert submission.field1 == ...  //'assert' is mandatory 
    assert submission.field2 == ...
}

For Spock, method call is treated as a regular value to be evaluated (if method is non-void). Therefore, if a method returns false or 0 or an empty collection the test fails. Otherwise, also if a method has the void return type, the test passes.

It is requires to care about the assertions manually, but placing the assert keyword explicitly withing the method.

Assertion in Closure execution

A similar class of problems poses assertions within closures. They might be not so often used as in methods, but it makes them even more at risk of incorrect usage.

    ...
expect:
    GParsPool.withPool {
        ["1"].eachParallel CorrelationIdUpdater.wrapClosureWithId {
            assert it == "1"    //mandatory "assert"
        }
    }

With Groovy 3 (and Spock 2.0-M2+ ) the same applies to assertions in lambda expressions.

Other cases

Quoting the Spock documentation :

Conditions are an essential ingredient of then blocks and expect blocks. Except for calls to void methods and expressions classified as interactions, all top-level expressions in these blocks are implicitly treated as conditions.

So, in addition to the 3 aforementioned cases, anything not “top-level” (or not placed in the then and expect blocks) should bring our attention. That includes - less popular - conditional statements (if/else), loops, and the catch blocks.

Updated 2021-12-26. Added the “Other cases” section, based on a constructive comment by Hari Krishna Dara .

Summary

Spock and Groovy (as every tool or language) can be used in more or less “messy way”. Groovy being a very flexible and optionally dynamic language makes it even easier. “With great power comes great responsibility”, a proverb popularized by the Spider-Man franchise, can be applied also to Spock. Users should be aware of the limitations and corner cases to avoid mistakes.

Btw, in the context of assertions those are corner cases where assertion(-like) expression is just ignored, which is much worse than just an error. However, it is not the only place. A few years ago I made a 50-minutes long presentations just about “Interesting nooks and crannies of Spock” (slides and videos ). And definitely I didn’t exhausted a topic :-).

Lead photo based on unofficial Spock logo proposal from 2014 by Søren Berg Glasius and work safety signs by Clker-Free-Vector-Images, published in Pixabay.

Simplify integration testing of legacy applications with Spock 1.2

Marcin Zajączkowski — Fri, 14 Sep 2018 10:00:00 +0200

Learn how leverage Spock 1.2 to slice a Spring context of a legacy application writing integration tests.

Have you ever wanted, having some legacy application which you were starting to work on, to write some tests to get know what is going on and possibly be notified about regressions? That feeling when you want to instantiate a single class and it fails with NullPointerException. 6 replaced (with difficulty) dependencies later there are still some errors from the classes that you haven’t heard about before. Sounds familiar?

There are different techniques to deal with hidden dependencies. There is the whole dedicated book about that (and probably a few other that I haven’t read). Occasionally, it may be feasible to start with the integration tests and run through some process. It may be even more “entertaining” to see what exotic components are required to just setup the context, even if they are completely not needed in our case. Thank you (too wide and carelessly used) @ComponentScan :).

Injecting stubs/mocks inside the test context is a way to go as an emergency assistance (see the last paragraph, there are better, yet harder approaches). It can be achieved “manually” with an extra bean definition with the @Primary annotation (usually a reason to think twice before doing that) for every dependency at which level we want to make a cut of (or for every unneeded bean which is instantiated by the way). @MockBean placed on a field in a test is more handy, but still, it is needed to define a field in our tests and put the annotation on it (5? 10? 15 beans?). Spock 1.2 introduces somehow less know feature which may be useful here - @StubBeans.

It can be used to simply provide a list of classes which (possible) instances should be replaced with stubs in the Spring test context. Of course before the real objects are being instantiated (to prevent for example NPE in a constructor). Thanks to that up to several lines of stubbing/mock injections:

@RunWith(SpringRunner.class) //Spring Boot + Mockito
@SpringBootTest //possibly some Spring configuration with @ComponentScan is imported in this legacy application
public class BasicPathReportGeneratorInLegacyApplicationITTest { //usual approach

    @MockBean
    private KafkaClient kafkaClientMock;

    @MockBean
    private FancySelfieEnhancer fancySelfieEnhancerMock;

    @MockBean
    private FastTwitterSubscriber fastTwitterSubscriberMock;

    @MockBean
    private WaterCoolerWaterLevelAterter waterCoolerWaterLevelAterterMock;

    @MockBean
    private NsaSilentNotifier nsaSilentNotifierMock;

    //a few more - remember, this is legacy application, genuine since 1999 ;)
    //...

    @Autowired
    private ReportGenerator reportGenerator;

    @Test
    public void shouldGenerateEmptyReportForEmptyInputData() {
        ...
    }
}

can be replaced with just one (long) line:

@SpringBootTest //possibly some Spring configuration with @ComponentScan is imported
@StubBeans([KafkaClient, FancySelfieEnhancer, FastTwitterSubscriber, WaterCoolerWaterLevelAterter, NsaSilentNotifier/(, ... */])
  //all classes of real beans which should be replaced with stubs
class BasicPathReportGeneratorInLegacyApplicationITSpec extends Specification {

    @Autowired
    private ReportGenerator reportGenerator

    def "should generate empty report for empty input data"() {
        ....
    }
}

(tested with Spock 1.2-RC2)

It’s worth to mention that @StubBeans is intended just to provide placeholders. In a situation it is required to provide stubbing and/or an invocation verification @SpringBean or @SpringSpy (also introduced in Spock 1.2) are better. I wrote more about it in my previous blog post .

There is one important aspect to emphasize. @StubBeans are handy to be used in a situation when we have some “legacy” project and want to start writing integration regression tests quickly to see the results. However, as a colleague of mine Darek Kaczyński brightly summarized, blindly replacing beans which “explode” in tests is just “sweeping problems under carpet”. After the initial phase, when we are starting to understand what is going on, it is a good moment to rethink the way the context - both in production and in tests - is created. The already mentioned too wide @ComponentScan is very often the root of all evil. An ability to setup a partial context and put it together (if needed) is a good place to start. Using @Profile or conditional beans are the very powerful mechanisms in tests (and not only there). @TestConfiguration and proper bean selection to improve context caching are something worth to keep in your mind. However, I started this article to present the new mechanism in Spock which might be useful in some cases and I want to keep it short. There could be an another, more generic blog post just about managing the Spring context in the integration tests. I have to seriously thing about it :).