junit5 on Solid Soft

Spock vs JUnit 5 - the ultimate feature comparison

Marcin Zajączkowski — Wed, 15 Apr 2020 10:00:00 +0200

What is the best testing framework for Java code nowadays? Spock or JUnit 5? Check it out in this ultimate feature comparison.

Spock was a game changer for all the people struggling with unit testing in JUnit 4. Compact syntax, parameterized tests or flexibility to mention just a few advantages. Over 10 years after JUnit 4.0, the brand new, written from scratch, Java 8 optimized Junit 5 has been released. And time has not stopped then.

In this blog post, I will compare selected areas of Spock and JUnit 5 to give you an overview how the situation looks like nowadays. I will try to answer the question if its time for Spock to fade into oblivion or maybe quite the opposite it is still light years ahead of JUnit 5.

This blog post is a written version of my presentation titled “Spock vs JUnit 5 - Clash of the Titans”. If you prefer, feel free to see the slides or watch the video instead.

Historical background

As a warm-up, let’s take a look at the important dates in the automatic code testing in Java.

2000 - JUnit - first xUnit for Java
2004 - TestNG - Java 5 (annotations) leveraged in tests
- with some unique (at the time) features (such as parameterized tests, test repetition or test dependency)
2006 - JUnit 4 - Java 5 support
- catching up TestNG features over the next few years
- de facto standard (the world chose :-/ ), steady evolution rather than revolution
2009 - Spock - revamped test creation
- fresh ideas and power of Groovy under the hood
2017 - JUnit 5 - redesigned and rewritten from scratch
- new king of the hill? - let’s check it out!

This comparison covers the latest released stable versions (as of April 2020) of JUnit (5.6) and Spock (1.3). I am fully aware of in-development Spock 2.0 and I plan to update this comparison (or write a supplement post) once 2.0-final is available.

Development comparison

Let’s start with a comparison of some biased aspects of development of JUnit 5 and Spock.

	JUnit 5 (as of 5.6)	Spock (as of 1.3)
inception year	2015	2009
number of GitHub stars	3,7K	~2,6K
number of commits	~6K	~2,5K
development activity	high (young project)	medium (mature project)
number of active committers	3	1 + 3
number of contributors (ever)	~130	~80

The first thing to notice is the fact that JUnit 5 is much younger project with very high development activity. There are 3 active committers with a bunch of external contributors.

One the other hand, Spock is much more mature with lower development activity. However, many of the features developed in JUnit 5 recently were already available in Spock. Therefor, the need for lots of commit in Spock is lower. Spock has currently one main maintainer and 2-3 people regularly contributing to the project (of course also with a pack of external contributors).

Partial verdict

Development: JUnit 5 wins

Reason: Mainly thanks to more active development activity in recent years.

Tool support

The next thing to cover is Java 11 compatibility and support in tools.

	JUnit 5	Spock
Java 11+	very good	good (with modern Groovy 2.5.x)
IntelliJ IDEA	built-in	built-in (some Groovy limitations)
Eclipse	build-in	built-in (some Groovy limitations)
Netbeans	10.0+ (Maven only)	unknown
Maven	plugin (official, Surefire)	plugin (GMavenPlus for Groovy)
Gradle	built-in	built-in
SonarQube	built-in	plugin (official for Groovy)
PIT - mutation testing	plugin (official)	build-in

Both of the frameworks are currently very well supported in the Java ecosystem. However, it is worth to mention that tests (specifications) in Spock are written in Groovy (which notabene is the main power of Spock). As a downside, due to much more dynamic nature of Groovy as a language, IDEs have much harder time to provide as good support for it as for Java (e.g. refactoring or compile time error reporting).

Regarding Java 11 and Spock, with modern Groovy 2.5.x it should work flawlessly. In general, most of the things should work fine also with Java 12, 13 and (with Groovy 2.5.10+) also 14. The official Java 14+ support should be available in Spock 2.0 (still in development) with Groovy 3.0.

On the other hand, JUnit 5 is a role model on that field. It is continuously tested on a CI server with the recent Java versions (including the early access builds, such as OpenJDK 15 EA).

Partial verdict

Tool support: JUnit 5 wins

Reason: Great IDE support for Java code and continuous testing with the bleeding edge versions of Java.

Test structure

Let’s move on to less technical aspects of test development - test structure.

As die-hard followers of my blog may know, I am a big propagator of well structured automatic tests with the BDD-originated given-when-then concept. In short, it unifies test creation, improves their readability and makes it easier to write, especially for less experienced people. To learn more, please take a loot at a separate blog post that I wrote some time ago.

JUnit 5

class SimpleCalculatorTest {
    @Test
    void shouldAddTwoNumbers() {
        //given
        Calculator calculator = new Calculator();
        //when
        int result = calculator.add(1, 2);
        //then
        assertEquals(3, result);
    }
}

The given-when-then sections are marked just by plain comments in code. It is not perfect. They might be forgotten and are easy to lost/misplace during refactoring. Could it be done better?

Spock

class SimpleCalculatorSpec extends Specification {
    def "should add two numbers"() {
        given:
            Calculator calculator = new Calculator()
        when:
            int result = calculator.add(1, 2)
        then:
            result == 3
    }
}

Well, in Spock the (almost not used in Java) label construction is leveraged. It is crucial to highlight that they are not just comments in code - they are a part of the specification. For example, having when: removed in the example above, generates real compilation errors. Nice.

Btw, of course, in Spock it is possible to write just one-liners, if feasible. However, developers have to do it intently.

Partial verdict

Test structure: Spock wins

Reason: Spock cares more about well structured (and more readable) tests.

Btw, with both JUnit 5 and Spock it is worth to use code templates to generate test skeleton to fill instead of writing all those given-when-then by hand :-).

Exception testing

Another topic I want to cover is exception testing. In general, to test corner cases (alternative scenarios) it is needed to verify that an exception with a given type was thrown. Usually, it is followed by some extra assertions such as an exception message and a cause.

JUnit 5

In JUnit 5, it is no longer possible to use @Test(expected = NullPointerException.class), useful for one-liners, but potentially risky with more complicated test code. Instead of, we have the assertThrows() construction, similar to catchThrowable() from AssertJ.

@Test
void shouldThrowBusinessExceptionOnCommunicationProblem() {
    //when
    Executable e = () -> client.sendPing(TEST_REQUEST_ID)
    //then
    CommunicationException thrown = assertThrows(CommunicationException.class, e);
    assertEquals("Communication problem when sending request with id: " + TEST_REQUEST_ID,
                 thrown.getMessage());
    assertEquals(TEST_REQUEST_ID, thrown.getRequestId());
}

The syntax is very compact (thanks to the lambda expression which is used to catch an exception). In addition, it is easy to perform any required further assertion(s) on the returned exception instance.

Spock

Spock, on the other hand, still provides @FailsWith(NullPointerException) for one-liners. However, it is almost completely not used on a daily basis. All thanks to the thrown() construction.

def "should capture exception"() {
    when:
        client.sendPing(TEST_REQUEST_ID)
    then:
        CommunicationException e = thrown()
        e.message == "Communication problem when sending request with id: $TEST_REQUEST_ID"
        e.requestId == TEST_REQUEST_ID
}

It is worth to notice that there is no lambda expression to catch an exception. To implement that Spock leverages Groovy AST transformations which under the hood transparently add required code. As a result, the exception thrown in the when block is caught and returned from the thrown() method. Really smart.

In addition, it is nice that in Spock, there is a smart type inference. It is not needed to precise the exception type twice. Based on the variable type on the left side, Spock is able to create the proper try-catch block.

As a bonus, there as a separate utility class Exceptions in Spock which provides extra assertions to play with the cause chain.

Partial verdict

Exception testing: Spock wins

Reason: Even shorter and less intrusive mechanism for exception testing.

Conditional test execution

In multiple situation it is required to (not) execute a given test only if given condition is (not) met.

The most common cases include:

particular Java version
- reflection hack to use newer version features
- compatibility testing for lower version
specific operating system
- notifications about changed files on Mac are much delayed
- testing symlinks on Windows makes no sense
tests executed only on CI server, stage environment, …

JUnit 5

JUnit 5 introduced brand new support for annotation-based @Enabled* and @Disabled* conditions.

There is a predefined set of conditions for:

JVM version
operating system
system property
environment variable

For example:

@Test
@DisabledOnOs(OS.WINDOWS)
void shouldTestSymlinksBasedLogic() {
    ...
}

@Test
@EnabledIfSystemProperty(named = "os.arch", matches = ".*32.*")
void shouldBeRunOn32BitSystems() {
    ...
}

The JVM version and operating system conditions are enums and thanks to that, there is a nice code completion provided by an IDE. In addition the conditions can be used as meta-annotations to make more complex scenarios easier to use in multiple places.

@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
@Test
@EnabledOnOs(WINDOWS)
@EnabledIfSystemProperty(named = "os.arch", matches = ".*32.*")
@interface TestOn32BitWindows {
}

With the following usage:

@TestOn32BitWindows
void shouldDoStrangeThings1OnLegacyWindows() { ... }

@TestOn32BitWindows
void shouldDoStrangeThings2OnLegacyWindows() { ... }

Originally, there was also support for custom logic in script-based inline conditions. However, due to limited usability (code written in String) and deprecation of Nashorn in Java 11 it has been removed in JUnit 5.6. Writing own condition logic currently requires a custom implementation of ExecutionCondition.

Spock

Since time immemorial, Spock has provided support for annotation based conditions with @Requires and @IgnoreIf. Out of box it is possible to check:

JVM version
operating system
system property
environment variable

The same set implemented later on in JUnit 5. However, the usage is slightly different.

@IgnoreIf({ !jvm.java8Compatible })
def "should return empty Optional by default for unstubbed methods with Java 8+"() { ... }

@Requires({ sys["targetEnvironment"] != "prod" })
def "should execute smoke testing on non production environment"() { ... }

Instead of enums, the logic is written in Groovy Closures leveraging delegation to the jvm/os/sys/... objects providing a dedicated methods (such as isJava11Compatible() or isLinux()). Unfortunately there is no code completion available by default, but it can be enabled with a small trick (Update. Just please be aware that the linked slides are from 2016 - Spock 1.0/1.1 - and in some places are outdated).

Using Closures provides one more important benefit - custom logic can implemented inline with pure Groovy.

1
2

@Requires({ isStrongCryptographyEnabled() })    //custom static method
def "should test strong cryptography-based features"() { ... }

Partial verdict

Conditional test execution: Spock wins

Reason: Higher flexibility with non standard conditions.

Mocking

Mocking is a crucial part of automatic code testing. It is especially beneficial in unit testing, but can be also usable in integration testing (e.g. with Spring Framework).

JUnit 5

In JUnit 5 tests, Mockito is the first port of call when it comes to mocking.

@Test
public void should_not_call_remote_service_if_found_in_cache() {
    //given
    given(cacheMock.getCachedOperator(CACHED_MOBILE_NUMBER)).willReturn(Optional.of(PLUS));
    //when
    service.checkOperator(CACHED_MOBILE_NUMBER);
    //then
    then(webserviceMock).should(never()).checkOperator(CACHED_MOBILE_NUMBER);
//   verify(webserviceMock, never()).checkOperator(CACHED_MOBILE_NUMBER);   //alternative syntax
}

Mockito is an industrial standard for mocking in Java. It is very mature, well known with a rich set of features. In addition, Spring Boot provides the official support for injecting Mockito’s mocks and spies into its context, which can be very useful in integration tests.

Spock

Spock on the other hand contains a built-in mocking subsystem.

def "should not hit remote service if found in cache"() {
    given:
        cacheMock.getCachedOperator(CACHED_MOBILE_NUMBER) >> Optional.of(PLUS)
    when:
        service.checkOperator(CACHED_MOBILE_NUMBER)
    then:
        0 * webserviceMock.checkOperator(CACHED_MOBILE_NUMBER)
}

While the syntax in Mockito is pretty readable, there are all those given/thenReturn/thenAnswer/... to write. Spock, however, leverages the power of operator overloading in Groovy (and of course its AST transformations) to make stubbing and verification as simple as possible.

given(cacheMock.getCachedOperator(CACHED_MOBILE_NUMBER)).willReturn(Optional.of(PLUS)); //Mockito
// vs
cacheMock.getCachedOperator(CACHED_MOBILE_NUMBER) >> Optional.of(PLUS) //Spock

verify(webserviceMock, never()).checkOperator(CACHED_MOBILE_NUMBER); //Mockito
// vs 
0 * webserviceMock.checkOperator(CACHED_MOBILE_NUMBER) //Spock

In a basic case, it’s enough to write a method, use an operator and provide a returned value. In verification it’s just a method call with requested cardinality. Most likely, it will not be possible to achieve that in Java in the predictable future.

Historically, the main limitation with Spock’s mocks was their tight coupling with the specification (the test itself). As a result it was not possible to use Spock’s stubs, mocks and spies in the Spring context for integration tests. However, starting with Spock 1.1 it was relaxed and Spock 1.2 introduced the first class support for mocking and spying with Spring. As a bonus, it is available also for pure Spring context (Spring Boot is not required as it takes place with Mockito’s mocks).

To be fair, it is required to admit that more magic in Spock generates some quirks with some corner cases. Nevertheless, once knowing all of them the readability and compactness definitely pay off :-).

One more thing. It is worth to remember that with Spock used for testing, we are not forced to use it also for mocking. It is perfectly fine to use Mockito if wanted (or needed).

Partial verdict

Mocking: Spock wins

Reason: The syntax and really readable and compact. And with Spring support in Spock 1.2+ the last big limitation of the mocking subsystem in Spock was removed.

Parameterized tests

Parameterized tests in general provide a way to call one test with different set of input (and expected) parameters. It can dramatically reduce duplication in specific cases (e.g. in testing with the user defined acceptance data set).

A word of warning here. In some scenarios, using parameterized tests can hide specific business use cases which would be clearly exposed in our non-parameterized tests otherwise. There are techniques to avoid that, but it is a topic for another article.

JUnit 5

Parameterized tests were one of the key features of TestNG from its early days. In JUnit 4 the feature has been not available for year and even once implemented, the design made it very unpractical to use in most of the cases. To improve the situation the 3rd-party runners were available, but it was still sad (for JUnit 4).

Luckily, JUnit 5 is the first version of JUnit with sane built-in parameterized tests support.

@ParameterizedTest
@CsvSource({ "dog, 3", "wolf, 4", "hippopotamus , 12" })
void testStringLengthWithParameterizedTestInJUnit5(String word, int expectedNumberOfLetters) {
    assertEquals(word.length(), expectedNumberOfLetters);
}

For simple scenario with inline values (provided as CSV strings), there is a nice implicit argument conversion from String for various types (e.g. dates, files, path or UUID). It just requires to add @ParameterizedTest and @CsvSource (or @ValueSource, @EnumSource, etc.).

However, not everything can be put into string. The commonly encountered scenario is an object creation or calling a method defined in a test fixture. To support that JUnit 5 gives an ability to call a local method (or the one from external object) to return required input/output arguments. They should be provided as Stream (or List) of Arguments.

@ParameterizedTest(name = "value to pay for invoice {0} should be {1}")
@MethodSource("invoiceProvider")
void shouldCalculateToPayValueForInvoice(Invoice invoice, BigDecimal expectedValueToPay) {
    //when
    int valueToPay = invoice.toPayValue();
    //expect
    assertEquals(expectedValueToPay, valueToPay);
}

private static Stream<Arguments> invoiceProvider() {
    return Stream.of(
            Arguments.of(regularInvoice(), 54),
            Arguments.of(overduedInvoice(), 81),
            Arguments.of(paidInvoice(), 0)
    );
}

It is just required to use @MethodSource to point the method (or use a naming convention to resolve the correct method name automatically).

The thing I don’t like is a lack of the arguments type check. In tests with multiple parameters it can be quite confusing to determine which is which.

As a bonus it is also possible to customize a test name which will be displayed in the report (in IDE, HTML report, etc.).

Spock

In Spock, first let’s take a longer look at the simple parameterized test.

@Unroll
def "should sum two integers (#x + #y = #expectedResult)"() {
    when:
        int result = calculator.add(x, y)
    then:
        result == expectedResult
    where:
         x |  y || expectedResult
         1 |  2 ||  3
        -2 |  3 ||  1
        -1 | -2 || -3
}

The words that come to my mind are “state of the art”. This is a place where Spocks excels.

With table-like formatting after the where keyword, input data looks very natural and is easy to read. What is worth mentioning, there is no need to define x, y, expectedResult as method arguments in a test (as it takes place in JUnit 5). The variables are added implicitly with full visibility and type inference in IDE.

The method name can be parameterized inline with the #var syntax (it’s Groovy - methods can have spaces and other “strange” characters).

In Spock it is also not a problem to use table-like syntax in a situation where constructor or methods are needed to be called (which in JUnit 5 requires using a separate provider method).

@Unroll("Value to pay for invoice #invoice should be #expectedValueToPay")
def "should calculate value to pay for invoice"() {
    expect:
        invoice.valueToPay() == expectedValueToPay
    where:
        invoice           || expectedValueToPay
        regularInvoice()  || 54
        overduedInvoice() || 81
        paidInvoice()     || 0
}

While working with acceptance testing where results are presented to less technical users, it is also possible to distinguish a test name for developers with a scenario name for QA engineers (or product owners) by writing it in the @Unroll annotation.

Of course, for more complex/advanced use cases (e.g. generating input parameters dynamically based on exterenal source), data pipes and data providers can be used. They are not as natural as the aforementioned table, but are very powerful.

@Unroll("#pesel is valid (#dbId)")
def "should validate PESEL correctness (CSV)"() {
    expect:
        sut.validate(pesel)
    where:
        [dbId, _, _, pesel] << readValidPeopleFromCSVFile()
                                   .readLines().collect { it.split(',') }
                                   //ugly way to read CSV - don't do this :-)
}

Partial verdict

Parameterized tests: Spock wins

Reason: It is enough to visually compare parameterized tests in those two frameworks.

Migration

The next point is very important for people currently using JUnit 4.

JUnit 4 → JUnit 5

The JUnit 5 tests can co-exist with the JUnit 4 tests. There is a dedicated module junit5-vintage which executes old JUnit 4 tests on the new JUnit Platform (part of JUnit 5).

Writing new tests with JUnit 5 is also easy. The test structure is very similar. We just need to learn some new keywords, annotations and assertions (and remember to use @Test from the new org.junit.jupiter.api package :-) ).

JUnit 4 → Spock

The Spock tests can also co-exists with the JUnit 4 tests. In fact Spock 1.x is a (sophisticated, but still) JUnit 4 runner. However, besides that there is a completely new test structure which we need to get familiar with. It is easy to grasp and, in the end, it is more readable, but still, it is something new to learn.

Partial verdict

Tool support: JUnit 5 wins

Reason: The JUnit 5 tests are more similar to the JUnit 4 tests and therefor, that migration path should be a little bit easier.

One more important migration-related thing here. During my training or consulting , I do not recommend rewriting existing hundreds and thousands of tests from JUnit 4 to the new technology, just for fun. Most of the existing tests will stay untouched for the foreseeable future. It is much better to write the new tests in the new technology and rewrite existing tests only along the way of doing some production logic related changes in them.

Summary - final verdict

Inarguably, JUnit 5 is a great progress over JUnit 4. Many features (previously) unique to Spock now are available in JUnit 5. However, Spock still excels in some areas. The best choice is no longer obvious.

Here, an anecdote, proposed my a colleague of mine. There are BMW and Audi cars. Both brands are very good and the choice usually depends on our personal preferences.

There same with Spock and JUnit 5. First, let’s take a look at the features comparison of the presented testing frameworks.

	JUnit 5 (as of 5.6)	Spock (as of 1.3)
development	very good	good (new features and bugfixes)
learning curve	very good (similar to 4)	good (Groovy to grasp)
tool support	very good (trending up)	good (weaker compile time checks)
test structure	good	very good (BDD by default)
exception testing	good	very good
conditional test execution	good	very good (custom logic)
mocking	good (Mockito)	good+ (very compact, some quirks)
parameterized tests	good	very good (exceptional!)
migration from JUnit 4	very good	good

As you see, they both have their strengths and weaknesses. In addition to covered aspects, you should add to that list other aspects which are important for you (and for your team) and see which framework fits you more.

For instance.

If you prefer:

old good Java as the only language
stability and being mainstream
stronger compile time error checking

you should choose JUnit 5.

On the other hand,

if - like me - you favor:

simplicity and readability
power of Groovy under the hood
beautiful parameterized tests and exception testing

then Spock should be better for you.

One final reflection. Preparing my presentation (and later on that blog post), I realized it would be great to have only that kind of dilemma while crafting software, choose between two pieces of good software :-).

A note about Spock logo. As of earthdate 2020-04, Spock doesn’t have the official logo. The one used in the lead photo is a proposal by Søren Berg Glasius made back in 2014. There are also some other proposals, but if you have a good idea, feel free to propose it there (or vote for existing candidates).

PIT, JUnit 5 and Gradle - with just one extra line of configuration

Marcin Zajączkowski — Thu, 27 Feb 2020 00:00:00 +0000

Discover dead simple, improved PIT and JUnit 5 configuration in Gradle (with gradle-pitest-plugin 1.4.7+).

JUnit 5 is undeniably more and more popular nowadays. While there is a dedicated plugin for PIT for JUnit 5 and it has been supported by gradle-pitest-plugin for years, it was required to add a few lines of boilerplate code to achieve that. Recently, I’ve got a question if it could be simplified. I liked it. Challenge accepted :-).

Generic approach with ‘buildscript {}’

First, take a look at the generic approach with the buildscrip {} code block, which remembers times of Gradle 0.x:

buildscript {
   repositories {
       mavenCentral()
       gradlePluginPortal() //optionally, if any plugin is not available in Maven Central
   }
   configurations.maybeCreate('pitest')
   dependencies {
       classpath 'info.solidsoft.gradle.pitest:gradle-pitest-plugin:1.4.6'
       pitest 'org.pitest:pitest-junit5-plugin:0.12'
   }
}

apply plugin: 'java'
apply plugin: 'info.solidsoft.pitest'

pitest {
    testPlugin = 'junit5'
}

Just 3 extra lines. Acceptable, however buildscript {} for plugin configuration is somehow verbose by itself.

Modern approach with ‘plugins {}‘
(with older gradle-pitest-plugin)

The modern variant with plugins {} should be shorter:

buildscript {
   repositories {
       mavenCentral()
   }
   configurations.maybeCreate('pitest')
   dependencies {
       pitest 'org.pitest:pitest-junit5-plugin:0.12'
   }
}

plugins {
    id 'java'
    id 'info.solidsoft.pitest' version '1.4.6'
}

pitest {
    testPlugin = 'junit5'
}

Unfortunately, the compact syntax of the plugin {} block is wasted by a need to add an extra dependency pitest-junit5-plugin used by gradle-pitest-plugin in runtime in the buildscript {} block - 10 lines extra. Highly disappointing ;-).

Modern improved approach with ‘plugins {}‘
and gradle-pitest-plugin 1.4.7+

With just released gradle-pitest-plugin 1.4.7 we can forget about all the boilerplate code:

plugins {
    id 'java'
    id 'info.solidsoft.pitest' version '1.4.7'
}

pitest {
    //adds dependency to org.pitest:pitest-junit5-plugin and sets "testPlugin" to "junit5"
    junit5PluginVersion = '0.12'
}

Just one line junit5PluginVersion = '0.12' which under the hood adds a pitest-junit5-plugin dependency in the required versions and activates junit5 as testPlugin used by PIT. Doesn’t it look nice? :-)

Summary

In this short blog post I presented how the configuration of the PIT, JUnit 5 and Gradle(-pitest-plugin) could be simplified with just a few changes in the plugin itself. All thanks to the question by John Scancella and the idea that sprang into my mind how to implement it in the smart way.

Therefore, I encourage you to report (sensible) ideas for improvements and things that limit you in the projects you use (or even better a pull/merge request after the initial discussion with the maintainer). Maybe it will implemented (accepted) for common good :-).

Migrating Spock 1.3 tests to Spock 2.0

Marcin Zajączkowski — Thu, 02 Jan 2020 10:00:00 +0200

Get know what you can expect from Spock 2.0 M2 (based on JUnit 5), how to migrate to it in Gradle and Maven, and why it is important to report spotted problems :).

Important note. I definitely do not encourage you to migrate your serious real-life project to Spock 2.0 M1 or M2 for good! This is the first (pre-)release of 2.x with not finalized API, intended to gather user feedback related to internal Spock migration to JUnit Platform.

This blog post arose to try to encourage you to make a test migration of your projects to Spock 2.0, see what started to fail, fix it (if caused by your tests) or report it (if it is a regression in Spock itself). As a result - at the Spock side - it will be possible to improve the code base before Milestone 2. The benefit for you - in addition to contribution to the FOSS project :-) - will be awareness of required changes (kept in a side branch) and readiness to migration once Spock 2.0 is more mature.

I plan to update this blog post when the next Spock 2 versions are available.

Updated 2020-02-10 to cover Spock 2.0 M2 with a dedicated Groovy 3 support.
Updated 2020-03-05. New point about dealing with NoClassDefFoundError: LambdaExpression.
Updated 2020-03-11. Detecting “no tests” issue with multiple test tasks.

Powered by JUnit Platform

The main change in Spock 2.0 M1 is migration to JUnit 5 (precisely speaking to execute tests with JUnit Platform 1.5, part of JUnit 5 instead of the JUnit 4 runner API). This is very convenient as Spock tests should be automatically recognized and executed everywhere the JUnit Platform is supported (IDEs, build tools, quality tools, etc.). In addition, the features provided by the platform itself (such as parallel test execution) should be (eventually) available also for Spock.

Gradle

To bring Spock 2 to a Gradle project it is needed to bump the Spock version:

testImplementation('org.spockframework:spock-core:2.0-M2-groovy-2.5')

and activate tests execution by JUnit Platform:

test {
    useJUnitPlatform()
}

Update 20200218. It is enough, but as Tomek Przybysz reminded in his comment, Gradle by default doesn’t fail if not tests are found. It may lead to a situation when after making that switch a build finished successfully, giving a false sense of security, while there are no tests executed at all.
It is a known issue in Gradle, not only limited to Spock. As a workaround the aforementioned configuration might to extended to:

test {
    useJUnitPlatform()

    afterSuite { desc, result ->
        if (!desc.parent) {
            if (result.testCount == 0) {
                throw new IllegalStateException("No tests were found. Failing the build")
            }
        }
    }
}

Or the more sophisticated version for multiple test tasks (e.g. unit/integration and functional):

tasks.withType(Test).configureEach { testTask ->
    testTask.configure {
        useJUnitPlatform()

        afterSuite { desc, result ->
            if (!desc.parent) {
                if (result.testCount == 0) {
                    throw new IllegalStateException("No tests were found. Failing the build")
                }
            }
        }
    }
}

Maven

With Maven on the other hand, it is still required to switch to a never Spock version:

<dependency>
  <groupId>org.spockframework</groupId>
  <artifactId>spock-core</artifactId>
  <version>2.0-M2-groovy-2.5</version>
  <scope>test</scope>
</dependency>

but that is all. The Surefire plugin (if you use version 3.0.0+) executes JUnit Platform tests by default, if junit-platform-engine (a transitive dependency of Spock 2) is found.

The minimal working project for Gradle i Maven is available from GitHub .

Other changes

Having such big change as migration to JUnit Platform, number of other changes in Spock 2.0 M1 is limited, to make finding a reason of potential regressions a little bit easier. As a side effects of the migration itself, the required Java version is 8.

In addition, all parameterized tests are (finally) “unrolled” automatically. That is great, however, currently there is no way to “roll” particular tests, as known from spock-global-unroll for Spock 1.x.

Some other changes (such as temporarily disabled SpockReportingExtension) can be found in the release notes .

More (possibly breaking) changes are expected to be merged into Milestone 2.

Issue with JUnit 4 rules

The tests using JUnit 4 @Rules (or @ClassRules) are expected to fail with an error message suggesting that requested objects were not created/initialized before a test (e.g. NullPointerException or IllegalStateException: the temporary folder has not yet been created) or were not verified/cleaned up after it (e.g. soft assertions from AssertJ). The Rules API is no longer supported by JUnit Platform. However, to make the migration easier (@TemporaryFolder is probably very often used in Spock-based integration tests), there is a dedicated spock-junit4 which internally wraps JUnit 4 rules into the Spock extensions and executes it in the Spock’s lifecycle. As it is implemented as a global extension, the only required thing to add is another dependency. In Gradle:

testImplementation 'org.spockframework:spock-junit4:2.0-M2-groovy-2.5'

or in Maven:

<dependency>
    <groupId>org.spockframework</groupId>
    <artifactId>spock-junit4</artifactId>
    <version>2.0-M2-groovy-2.5</version>
    <scope>test</scope>
</dependency>

That makes migration easier, but it is good to think about a switch to native Spock counterpart, if available/feasible.

Groovy 3 support

Updated 2020-02-10. The whole Groovy 3 section was added to cover changes in Spock 2.0 Milestone 2.

After my complains about runtime failure when Spock 2.0 M1 is used with Groovy 3.0, I rolled up my sleeves to check how hard would it be to provide that support. It took me some time, however, after after 23 (multiple times rebased) commits , constructive feedback from other Spock contributors and fruitful discussion with the Groovy developers, the support for Groovy 3 has been merged and is available as the main feature of Spock 2.0 M2, released right after Groovy 3.0.0 final.

To use Spock 2.0 M2 with Groovy 3 it is enough to just use the spock-*-2.0-M1 artifacts with the -groovy-3.0 suffix.

It is worth noting that Groovy 3 is not backward compatible with Groovy 2. To keep one Spock codebase, there is a layer of abstraction in Spock to allow to build (and use) the project with both Groovy 2 and 3. As a result, an extra artifact spock-groovy2-compat is (automatically) used in projects with Groovy 2. It is very important to do not mix the spock-*-2.x-groovy-2.5 artifacts with the groovy-*-3.x artifacts on a classpath. This may result in weird runtime errors.

I’m really happy that developers can immediately start testing the new Groovy 3.0.0 with Spock 2.0-M2 in their projects. In addition, as Spock is quite important (and low level) project in the Groovy ecosystem, it was nice to confirm that Groovy 3 works properly with it (and to report - along the way - a few minor detected issues to make Groovy even better ;-) ).

Other issues and limitations

NoClassDefFoundError: LambdaExpression with Groovy 2.5

Using Spock 2.0-M2 with Groovy 2.5 it is possible to get the following errors:

java.lang.NoClassDefFoundError: org/codehaus/groovy/ast/expr/LambdaExpression
java.lang.NoClassDefFoundError: org/codehaus/groovy/ast/expr/MethodReferenceExpression

As Groovy 2 and 3 are not backward compatible, there is an abstraction layer which allows to build (and use) Spock with two different Groovy versions. However, to achieve that it was necessary to add dummy LambdaExpression and MethodReferenceExpression in an extra Spock dependency spock-groovy2-compat used in the -groovy-2.5 variant.

It is automatically fetched by Maven/Gradle for spock-core, but in the situation you disabled transitive dependency for Spock:

dependencies {
    testImplementation('org.spockframework:spock-core:2.0-M2-groovy-2.5') {
        transitive = false
    }
    ...
}

@Grab('org.spockframework:spock-core:2.0-M2-groovy-2.5;transitive=false')

they are missing. In that case you might want to add spock-groovy2-compat dependency explicitly, e.g.:

dependencies {
    testImplementation('org.spockframework:spock-core:2.0-M2-groovy-2.5') {
        transitive = false
    }
    testImplementation('org.spockframework:spock-groovy2-compat:2.0-M2-groovy-2.5')
}

or exclude just Groovy dependencies (not all of them):

dependencies {
    testImplementation('org.spockframework:spock-core:2.0-M2-groovy-2.5') {
        exclude group: 'org.codehaus.groovy'
    }
}

Thanks to Paul King for suggesting that point.

Missing Groovy 3 support

Updated 2020-02-10. This section originally refereed to limitations of Spock 2.0 M1. Milestone 2 supports Groovy 3.

Spock 2.0 M1 is compiled and tested only with Groovy 2.5.8. As of M1, execution with Groovy 3.0 is currently blocked at runtime. Unfortunately, instead of a clear error message about incompatible Groovy version there is only a very cryptic error message:

Could not instantiate global transform class org.spockframework.compiler.SpockTransform specified at
jar:file:/.../spock-core-2.0-M1-groovy-2.5.jar!/META-INF/services/org.codehaus.groovy.transform.ASTTransformation
because of exception java.lang.reflect.InvocationTargetException

It is already reported and should be enhanced by M2.

Sadly, the limitation to Groovy 2.5 only, reduces potential feedback from people experimenting with Groovy 3 which is pretty close to a stable version (RC2 as of 2019/2020). It’s especially inconvenient as many Spock tests just work with Groovy 3 (of course there are some corner cases). There is a chance that Spock 2 before getting final will be adjusted to changes in Groovy 3 or at least the aforementioned hard limitation will be lifted. In the meantime, it is required to test Groovy 3 support with the snapshot version - 2.0-groovy-2.5-SNAPSHOT (which has that check disabled).

Summary

The action to do after reading this post is simple. Try to temporarily play with Spock 2.0 M2 in your projects and report any spotted issues, to help make Spock 2.0 even better :).

Convenient mocking in Mockito with JUnit 5 - the official way

Marcin Zajączkowski — Tue, 27 Mar 2018 10:00:00 +0200

Starting with the version 2.17.0 Mockito provides the official (built-in) support for managing a mocking life cycle if JUnit 5 is used.

Getting started

To take advantage of the integration, the Mockito’s mockito-junit-jupiter dependency is required to be added next to the JUnit 5’s junit-platform-engine one (see below for details).

After that, the new Mockito extension MockitoException for JUnit 5 has to be enabled. And that’s enough. All the Mockito annotation should automatically start to work.

import org.junit.jupiter.api.Test;  //do not confuse with 'org.junit.Test'!
//other imports
import org.mockito.junit.jupiter.MockitoExtension;

@ExtendWith(MockitoExtension.class)
class SpaceShipJUnit5Test {

    @InjectMocks
    private SpaceShip spaceShip;

    @Mock
    private TacticalStation tacticalStation;

    @Mock
    private OperationsStation operationsStation;

    @Test
    void shouldInjectMocks() {
        assertThat(spaceShip).isNotNull();
        assertThat(tacticalStation).isNotNull();
        assertThat(operationsStation).isNotNull();
        assertThat(spaceShip.getTacticalStation()).isSameAs(tacticalStation);
        assertThat(spaceShip.getOperationsStation()).isSameAs(operationsStation);
    }

It’s nice that both a test class and test methods don’t need to be public anymore.

Please note. Having also JUnit 4 on a classpath (e.g. via junit-vintage-engine) for the “legacy” part of tests it is important to do not confuse org.junit.jupiter.api.Test with the old one org.junit.Test. It will not work.

Stubbing and verification

If for some reasons you are not a fan of AssertJ (although, I encourage you to at least give it a try) JUnit 5 provides a native assertion assertThrows (which is very similar to assertThatThrownBy() from AssertJ). It provides a meaningful error message in a case of an assertion failure.

@Test
void shouldMockSomething() {
    //given
    willThrow(SelfCheckException.class).given(tacticalStation).doSelfCheck();   //void method "given..will" not "when..then" cannot be used
    //when
    Executable e = () -> spaceShip.doSelfCheck();
    //then
    assertThrows(SelfCheckException.class, e);
}

I were not myself if I would not mention here that leveraging support for default methods in interfaces available in AssertJ and mockito-java8 a lot of static imports can be made redundant.

@ExtendWith(MockitoExtension.class)
class SpaceShipJUnit5Test implements WithAssertions, WithBDDMockito {
    ...
}

Tweaking default behavior

It also worth to point our that using the JUnit 5 extension Mockito by default works in the “strict mode” . It means that - for example - unneeded stubbing will fail the test. While very often it is a code smell, there are some cases where that test construction is desired. To change the default behavior an @MockitoSettings annotation can be used.

@ExtendWith(MockitoExtension.class)
@MockitoSettings(strictness = Strictness.WARN)
class SpaceShipJUnitAdvTest implements WithAssertions, WithBDDMockito {
    ....
}

Dependencies

As I already mentioned, to start using it is required to add a Mockito’s mockito-junit-jupiter dependency next to a JUnit 5’s junit-platform-engine one. In a Gradle build it could look like:

dependencies {
    testCompile 'org.junit.vintage:junit-platform-engine:5.1.0'
    testCompile 'org.mockito:mockito-junit-jupiter:2.17.2'  //mockito-core is implicitly added

    testCompile 'org.junit.vintage:junit-vintage-engine:5.1.0'  //for JUnit 4.12 test execution, if needed
    testCompile 'org.assertj:assertj-core:3.9.1'    //if you like it (you should ;) )
}

Please note. Due to a bug with injecting mocks via constructor into final fields that I have found writing this blog post, it is recommended to use at least version 2.17.2 instead of 2.17.0. That “development” version is not available in the Maven Central and the extra Bintray repository has to be added.

repositories {
    mavenCentral()
    maven { url "https://dl.bintray.com/mockito/maven" }    //for development versions of Mockito
}

In addition, it would be a waste to do not use a brand new native support for JUnit 5 test execution in Gradle 4.6+ .

test {
    useJUnitPlatform()
}

IntelliJ IDEA has provided JUnit support since 2016.2 (JUnit 5 Milestone 2 at that time). Eclipse Oxygen also seems to add support for JUnit 5 recently.

Summary

It is really nice to have a native support for JUnit 5 in Mockito. Not getting ahead there is ongoing work to make it even better.
The feature has been implemented by Christian Schwarz and polished by Tim van der Lippe with great assist from a few other people .

The source code is available from GitHub .

Btw, are you wondering how JUnit 5 compares with Spock? I will be talking about that at GeeCON 2018 .

junit5 on Solid Soft

Spock vs JUnit 5 - the ultimate feature comparison

Historical background

Development comparison

Partial verdict

Tool support

Partial verdict

Test structure

JUnit 5

Spock

Partial verdict

Exception testing

JUnit 5

Spock

Partial verdict

Conditional test execution

JUnit 5

Spock

Partial verdict

Mocking

JUnit 5

Spock

Partial verdict

Parameterized tests

JUnit 5

Spock

Partial verdict

Migration

JUnit 4 → JUnit 5

JUnit 4 → Spock

Partial verdict

Summary - final verdict

PIT, JUnit 5 and Gradle - with just one extra line of configuration

Generic approach with ‘buildscript {}’

Modern approach with ‘plugins {}‘ (with older gradle-pitest-plugin)

Modern improved approach with ‘plugins {}‘ and gradle-pitest-plugin 1.4.7+

Summary

Migrating Spock 1.3 tests to Spock 2.0

Powered by JUnit Platform

Gradle

Maven

Other changes

Issue with JUnit 4 rules

Groovy 3 support

Other issues and limitations

NoClassDefFoundError: LambdaExpression with Groovy 2.5

Missing Groovy 3 support

Summary

Convenient mocking in Mockito with JUnit 5 - the official way

Getting started

Stubbing and verification

Tweaking default behavior

Dependencies

Summary

Modern approach with ‘plugins {}‘
(with older gradle-pitest-plugin)

Modern improved approach with ‘plugins {}‘
and gradle-pitest-plugin 1.4.7+