What is the best testing framework for Java code nowadays? Spock or JUnit 5? Check it out in this ultimate feature comparison.
Spock was a game changer for all the people struggling with unit testing in JUnit 4. Compact syntax, parameterized tests or flexibility to mention just a few advantages. Over 10 years after JUnit 4.0, the brand new, written from scratch, Java 8 optimized Junit 5 has been released. And time has not stopped then.
In this blog post, I will compare selected areas of Spock and JUnit 5 to give you an overview how the situation looks like nowadays. I will try to answer the question if its time for Spock to fade into oblivion or maybe quite the opposite it is still light years ahead of JUnit 5.
As a warm-up, let’s take a look at the important dates in the automatic code testing in Java.
- 2000 - JUnit - first xUnit for Java
- 2004 - TestNG - Java 5 (annotations) leveraged in tests
- with some unique (at the time) features (such as parameterized tests, test repetition or test dependency)
- 2006 - JUnit 4 - Java 5 support
- catching up TestNG features over the next few years
- de facto standard (the world chose :-/ ), steady evolution rather than revolution
- 2009 - Spock - revamped test creation
- fresh ideas and power of Groovy under the hood
- 2017 - JUnit 5 - redesigned and rewritten from scratch
- new king of the hill? - let’s check it out!
Let’s start with a comparison of some biased aspects of development of JUnit 5 and Spock.
|JUnit 5 (as of 5.6)||Spock (as of 1.3)|
|number of GitHub stars||3,7K||~2,6K|
|number of commits||~6K||~2,5K|
|development activity||high (young project)||medium (mature project)|
|number of active committers||3||1 + 3|
|number of contributors (ever)||~130||~80|
The first thing to notice is the fact that JUnit 5 is much younger project with very high development activity. There are 3 active committers with a bunch of external contributors.
One the other hand, Spock is much more mature with lower development activity. However, many of the features developed in JUnit 5 recently were already available in Spock. Therefor, the need for lots of commit in Spock is lower. Spock has currently one main maintainer and 2-3 people regularly contributing to the project (of course also with a pack of external contributors).
Development: JUnit 5 wins
Reason: Mainly thanks to more active development activity in recent years.
The next thing to cover is Java 11 compatibility and support in tools.
|Java 11+||very good||good (with modern Groovy 2.5.x)|
|IntelliJ IDEA||built-in||built-in (some Groovy limitations)|
|Eclipse||build-in||built-in (some Groovy limitations)|
|Netbeans||10.0+ (Maven only)||unknown|
|Maven||plugin (official, Surefire)||plugin (GMavenPlus for Groovy)|
|SonarQube||built-in||plugin (official for Groovy)|
|PIT - mutation testing||plugin (official)||build-in|
Both of the frameworks are currently very well supported in the Java ecosystem. However, it is worth to mention that tests (specifications) in Spock are written in Groovy (which notabene is the main power of Spock). As a downside, due to much more dynamic nature of Groovy as a language, IDEs have much harder time to provide as good support for it as for Java (e.g. refactoring or compile time error reporting).
Regarding Java 11 and Spock, with modern Groovy 2.5.x it should work flawlessly. In general, most of the things should work fine also with Java 12, 13 and (with Groovy 2.5.10+) also 14. The official Java 14+ support should be available in Spock 2.0 (still in development) with Groovy 3.0.
On the other hand, JUnit 5 is a role model on that field. It is continuously tested on a CI server with the recent Java versions (including the early access builds, such as OpenJDK 15 EA).
Tool support: JUnit 5 wins
Reason: Great IDE support for Java code and continuous testing with the bleeding edge versions of Java.
Let’s move on to less technical aspects of test development - test structure.
As die-hard followers of my blog may know, I am a big propagator of well structured automatic tests with the BDD-originated given-when-then concept. In short, it unifies test creation, improves their readability and makes it easier to write, especially for less experienced people. To learn more, please take a loot at a separate blog post that I wrote some time ago.
The given-when-then sections are marked just by plain comments in code. It is not perfect. They might be forgotten and are easy to lost/misplace during refactoring. Could it be done better?
Well, in Spock the (almost not used in Java)
label construction is leveraged. It is crucial to highlight that they are not just comments in code - they are a part of the specification. For example, having
when: removed in the example above, generates real compilation errors. Nice.
Btw, of course, in Spock it is possible to write just one-liners, if feasible. However, developers have to do it intently.
Test structure: Spock wins
Reason: Spock cares more about well structured (and more readable) tests.
Btw, with both JUnit 5 and Spock it is worth to use code templates to generate test skeleton to fill instead of writing all those given-when-then by hand :-).
Another topic I want to cover is exception testing. In general, to test corner cases (alternative scenarios) it is needed to verify that an exception with a given type was thrown. Usually, it is followed by some extra assertions such as an exception message and a cause.
In JUnit 5, it is no longer possible to use
@Test(expected = NullPointerException.class), useful for one-liners, but potentially risky with more complicated test code. Instead of, we have the
assertThrows() construction, similar to
catchThrowable() from AssertJ.
The syntax is very compact (thanks to the lambda expression which is used to catch an exception). In addition, it is easy to perform any required further assertion(s) on the returned exception instance.
Spock, on the other hand, still provides
@FailsWith(NullPointerException) for one-liners. However, it is almost completely not used on a daily basis. All thanks to the
It is worth to notice that there is no lambda expression to catch an exception. To implement that Spock leverages Groovy AST transformations which under the hood transparently add required code. As a result, the exception thrown in the
when block is caught and returned from the
thrown() method. Really smart.
In addition, it is nice that in Spock, there is a smart type inference. It is not needed to precise the exception type twice. Based on the variable type on the left side, Spock is able to create the proper try-catch block.
As a bonus, there as a separate utility class
Exceptions in Spock which provides extra assertions to play with the cause chain.
Exception testing: Spock wins
Reason: Even shorter and less intrusive mechanism for exception testing.
Conditional test execution
In multiple situation it is required to (not) execute a given test only if given condition is (not) met.
The most common cases include:
- particular Java version
- reflection hack to use newer version features
- compatibility testing for lower version
- specific operating system
- notifications about changed files on Mac are much delayed
- testing symlinks on Windows makes no sense
- tests executed only on CI server, stage environment, …
JUnit 5 introduced brand new support for annotation-based
There is a predefined set of conditions for:
- JVM version
- operating system
- system property
- environment variable
The JVM version and operating system conditions are enums and thanks to that, there is a nice code completion provided by an IDE. In addition the conditions can be used as meta-annotations to make more complex scenarios easier to use in multiple places.
With the following usage:
Originally, there was also support for custom logic in script-based inline conditions. However, due to limited usability (code written in
String) and deprecation of Nashorn in Java 11 it has been removed in JUnit 5.6. Writing own condition logic currently requires a custom implementation of
Since time immemorial, Spock has provided support for annotation based conditions with
@IgnoreIf. Out of box it is possible to check:
- JVM version
- operating system
- system property
- environment variable
The same set implemented later on in JUnit 5. However, the usage is slightly different.
Instead of enums, the logic is written in Groovy Closures leveraging delegation to the
jvm/os/sys/... objects providing a dedicated methods (such as
isLinux()). Unfortunately there is no code completion available by default, but it can be enabled with a small trick
(Update. Just please be aware that the linked slides are from 2016 - Spock 1.0/1.1 - and in some places are outdated).
Using Closures provides one more important benefit - custom logic can implemented inline with pure Groovy.
Conditional test execution: Spock wins
Reason: Higher flexibility with non standard conditions.
Mocking is a crucial part of automatic code testing. It is especially beneficial in unit testing, but can be also usable in integration testing (e.g. with Spring Framework).
In JUnit 5 tests, Mockito is the first port of call when it comes to mocking.
Mockito is an industrial standard for mocking in Java. It is very mature, well known with a rich set of features. In addition, Spring Boot provides the official support for injecting Mockito’s mocks and spies into its context, which can be very useful in integration tests.
Spock on the other hand contains a built-in mocking subsystem.
While the syntax in Mockito is pretty readable, there are all those
given/thenReturn/thenAnswer/... to write. Spock, however, leverages the power of operator overloading in Groovy (and of course its AST transformations) to make stubbing and verification as simple as possible.
In a basic case, it’s enough to write a method, use an operator and provide a returned value. In verification it’s just a method call with requested cardinality. Most likely, it will not be possible to achieve that in Java in the predictable future.
Historically, the main limitation with Spock’s mocks was their tight coupling with the specification (the test itself). As a result it was not possible to use Spock’s stubs, mocks and spies in the Spring context for integration tests. However, starting with Spock 1.1 it was relaxed and Spock 1.2 introduced the first class support for mocking and spying with Spring. As a bonus, it is available also for pure Spring context (Spring Boot is not required as it takes place with Mockito’s mocks).
To be fair, it is required to admit that more magic in Spock generates some quirks with some corner cases. Nevertheless, once knowing all of them the readability and compactness definitely pay off :-).
One more thing. It is worth to remember that with Spock used for testing, we are not forced to use it also for mocking. It is perfectly fine to use Mockito if wanted (or needed).
Mocking: Spock wins
Reason: The syntax and really readable and compact. And with Spring support in Spock 1.2+ the last big limitation of the mocking subsystem in Spock was removed.
Parameterized tests in general provide a way to call one test with different set of input (and expected) parameters. It can dramatically reduce duplication in specific cases (e.g. in testing with the user defined acceptance data set).
A word of warning here. In some scenarios, using parameterized tests can hide specific business use cases which would be clearly exposed in our non-parameterized tests otherwise. There are techniques to avoid that, but it is a topic for another article.
Parameterized tests were one of the key features of TestNG from its early days. In JUnit 4 the feature has been not available for year and even once implemented, the design made it very unpractical to use in most of the cases. To improve the situation the 3rd-party runners were available, but it was still sad (for JUnit 4).
Luckily, JUnit 5 is the first version of JUnit with sane built-in parameterized tests support.
For simple scenario with inline values (provided as CSV strings), there is a nice implicit argument conversion from String for various types (e.g. dates, files, path or UUID). It just requires to add
However, not everything can be put into string. The commonly encountered scenario is an object creation or calling a method defined in a test fixture. To support that JUnit 5 gives an ability to call a local method (or the one from external object) to return required input/output arguments. They should be provided as
It is just required to use
@MethodSource to point the method (or use a naming convention to resolve the correct method name automatically).
The thing I don’t like is a lack of the arguments type check. In tests with multiple parameters it can be quite confusing to determine which is which.
As a bonus it is also possible to customize a test name which will be displayed in the report (in IDE, HTML report, etc.).
In Spock, first let’s take a longer look at the simple parameterized test.
The words that come to my mind are “state of the art”. This is a place where Spocks excels.
With table-like formatting after the
where keyword, input data looks very natural and is easy to read. What is worth mentioning, there is no need to define
x, y, expectedResult as method arguments in a test (as it takes place in JUnit 5). The variables are added implicitly with full visibility and type inference in IDE.
The method name can be parameterized inline with the
#var syntax (it’s Groovy - methods can have spaces and other “strange” characters).
In Spock it is also not a problem to use table-like syntax in a situation where constructor or methods are needed to be called (which in JUnit 5 requires using a separate provider method).
While working with acceptance testing where results are presented to less technical users, it is also possible to distinguish a test name for developers with a scenario name for QA engineers (or product owners) by writing it in the
Of course, for more complex/advanced use cases (e.g. generating input parameters dynamically based on exterenal source), data pipes and data providers can be used. They are not as natural as the aforementioned table, but are very powerful.
Parameterized tests: Spock wins
Reason: It is enough to visually compare parameterized tests in those two frameworks.
The next point is very important for people currently using JUnit 4.
JUnit 4 → JUnit 5
The JUnit 5 tests can co-exist with the JUnit 4 tests. There is a dedicated module
junit5-vintage which executes old JUnit 4 tests on the new JUnit Platform (part of JUnit 5).
Writing new tests with JUnit 5 is also easy. The test structure is very similar. We just need to learn some new keywords, annotations and assertions (and remember to use
@Test from the new
org.junit.jupiter.api package :-) ).
JUnit 4 → Spock
The Spock tests can also co-exists with the JUnit 4 tests. In fact Spock 1.x is a (sophisticated, but still) JUnit 4 runner. However, besides that there is a completely new test structure which we need to get familiar with. It is easy to grasp and, in the end, it is more readable, but still, it is something new to learn.
Tool support: JUnit 5 wins
Reason: The JUnit 5 tests are more similar to the JUnit 4 tests and therefor, that migration path should be a little bit easier.
One more important migration-related thing here. During my training or consulting , I do not recommend rewriting existing hundreds and thousands of tests from JUnit 4 to the new technology, just for fun. Most of the existing tests will stay untouched for the foreseeable future. It is much better to write the new tests in the new technology and rewrite existing tests only along the way of doing some production logic related changes in them.
Summary - final verdict
Inarguably, JUnit 5 is a great progress over JUnit 4. Many features (previously) unique to Spock now are available in JUnit 5. However, Spock still excels in some areas. The best choice is no longer obvious.
Here, an anecdote, proposed my a colleague of mine. There are BMW and Audi cars. Both brands are very good and the choice usually depends on our personal preferences.
There same with Spock and JUnit 5. First, let’s take a look at the features comparison of the presented testing frameworks.
|JUnit 5 (as of 5.6)||Spock (as of 1.3)|
|development||very good||good (new features and bugfixes)|
|learning curve||very good (similar to 4)||good (Groovy to grasp)|
|tool support||very good (trending up)||good (weaker compile time checks)|
|test structure||good||very good (BDD by default)|
|exception testing||good||very good|
|conditional test execution||good||very good (custom logic)|
|mocking||good (Mockito)||good+ (very compact, some quirks)|
|parameterized tests||good||very good (exceptional!)|
|migration from JUnit 4||very good||good|
As you see, they both have their strengths and weaknesses. In addition to covered aspects, you should add to that list other aspects which are important for you (and for your team) and see which framework fits you more.
If you prefer:
- old good Java as the only language
- stability and being mainstream
- stronger compile time error checking
you should choose JUnit 5.
On the other hand,
if - like me - you favor:
- simplicity and readability
- power of Groovy under the hood
- beautiful parameterized tests and exception testing
then Spock should be better for you.
One final reflection. Preparing my presentation (and later on that blog post), I realized it would be great to have only that kind of dilemma while crafting software, choose between two pieces of good software :-).A note about Spock logo. As of earthdate 2020-04, Spock doesn’t have the official logo. The one used in the lead photo is a proposal by Søren Berg Glasius made back in 2014. There are also some other proposals, but if you have a good idea, feel free to propose it there (or vote for existing candidates).