gradle on Solid Soft

Running Spock with unsupported Groovy version (Gradle + Maven)

Marcin Zajączkowski — Fri, 19 Nov 2021 00:10:00 +0200

Stuck with the ‘The Spock compiler plugin cannot execute because Spock 1.3-groovy-2.5 (3.0) is not compatible with Groovy 3.0 (4.0)’ error message? Find out what can be done about that (using Gradle and Maven).

Historical background

In the dim and distance past, when Groovy 1.x occasionally had some unpleasant (also runtime) backward compatibility issues, Spock had started to be built for the particular Groovy version individually. There were spock-0.5-groovy-1.6, spock-0.5-groovy-1.7 and spock-0.5-groovy-1.8 for instance. To help people choose the right Spock and the Groovy versions, the runtime check was added to fail the compilation (at the AST transformation level) with the meaningful error message, in the situation Spock was compiled with the different Groovy version that is available at runtime, e.g.:

The Spock compiler plugin refused to run because Spock 1.3-groovy-2.5 is not compatible with Groovy 3.0.4. For more information,
see http://versioncheck.spockframework.org Spock location: .../spock-1.3-groovy-2.5.jar, Groovy location: .../groovy-3.0.4.jar

The error message is very clear. The reasoning behind this decision is good, and definitely it can help developers to detect problems with the dependencies. However, it can also greatly complicate development, especially taking into account rather rarely released new Spock versions.

If you are here, because of that error visible in your build, jump to the proper section to see what can be done about the problem.

Possible options

Spock 1.3 and Groovy 3.0

Unfortunately, you are out of luck. If you stuck with Spock 1.3 (e.g. migration to Spock 2.0 is blocked for some reasons) there are two (mediocre) options:

Switch to Spock 1.3-SNAPSHOT. The version compatibility check is relaxed in the SNAPSHOT builds of Spock 1.3. What’s more, there were multiple reports that (in the majority of cases - some corner cases remains) Spock 1.3-groovy-2.5-SNAPSHOT works fine with Groovy 3.0.x.
There are no plans to release any new maintenance release of Spock 1.x. However, you can still create a new issue on GitHub and try to convince The Spock Developers to make 1.3.1 with that change :-).

Spock 2.0+ and Groovy 3.0

The Spock compiler plugin refused to run because Spock 2.0-groovy-2.5 is not compatible with Groovy 3.0.9.
For more information (including enforce mode), see http://docs.spockframework.org (section 'Known Issues'). 
Spock location: .../spock-2.0-groovy-2.5.jar
Groovy location: .../groovy-3.0.9.jar

That a trivial case. Just change Spock version from 2.0-groovy-2.5 to 2.0-groovy-3.0, and you are set.

Spock 2.0/2.1 and Groovy 4.0

The Spock compiler plugin refused to run because Spock 2.0-groovy-3.0 is not compatible with Groovy 4.0.0-beta-2.
For more information (including enforce mode), see http://docs.spockframework.org (section 'Known Issues'). 
Spock location: .../spock-2.0-groovy-3.0.jar
Groovy location: .../groovy-4.0.0-beta-2.jar

This variant assumes that you use Groovy 4 (SNAPSHOT, beta or a regular version) and Spock 2.0-groovy-3.0 (or 2.1). The error message is even more precise than in Spock 1.3, suggesting the ‘Known Issues’ section in the Spock documentation. We can leverage a brand-new system property spock.iKnowWhatImDoing.disableGroovyVersionCheck, introduced right in Spock 2.0-M3, which allows to disable the version check - of course, at your own risk :-).

Btw, if you in a hurry (e.g. just to test something), it is also possible just to switch Spock to SNAPSHOT version which accepts (almost) any Groovy version.

Btw2, if you prefer to just jump into code, there is a code repository with working examples for Gradle and Maven.

Gradle

As Groovy 4 changed its groupId from org.codehaus.groovy to org.apache.groovy it is required to add implicit exclusion for spock-2.0-groovy-3.0 transitive dependencies:

testImplementation("org.spockframework:spock-core:2.0-groovy-3.0") {
    exclude group: 'org.codehaus.groovy'
}

to avoid conflicting Groovy versions on a classpath (it’s nice that Gradle is able to detect that!):

> Task :compileTestGroovy FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':compileTestGroovy'.
> Could not resolve all files for configuration ':testCompileClasspath'.
   > Could not resolve org.apache.groovy:groovy:4.0.0-beta-2.
     Required by:
         project :
         project : > org.apache.groovy:groovy:4.0.0-beta-2 > org.apache.groovy:groovy-bom:4.0.0-beta-2
      > Module 'org.apache.groovy:groovy' has been rejected:
           Cannot select module with conflict on capability 'org.codehaus.groovy:groovy:4.0.0-beta-2' also provided by [org.codehaus.groovy:groovy:3.0.8(compile)]
   > Could not resolve org.codehaus.groovy:groovy:3.0.8.
     Required by:
         project : > org.spockframework:spock-core:2.0-groovy-3.0
      > Module 'org.codehaus.groovy:groovy' has been rejected:
           Cannot select module with conflict on capability 'org.codehaus.groovy:groovy:3.0.8' also provided by [org.apache.groovy:groovy:4.0.0-beta-2(groovyApiElements)]

It won’t be required for Spock 2.0-groovy-4.0, once available.

To be able use Groovy 4, Gradle 6.7+ is required.

With earlier Gradle version you might got some misleading errors about not resolvable dependency to org.codehaus.groovy:groovy:4.0.0-beta-2, even though you configured org.apache.groovy:groovy:4.0.0-beta-2 with the correct, changed group id.

Execution failed for task ':compileTestGroovy'.
> Could not resolve all files for configuration ':detachedConfiguration1'.
   > Could not find org.codehaus.groovy:groovy:4.0.0-beta-2.

The obvious way to add system properties to the Gradle execution itself is:

./gradlew check -Dspock.iKnowWhatImDoing.disableGroovyVersionCheck=true     //wrong way

Unfortunately it doesn’t work:

> Task :compileTestGroovy FAILED
startup failed:
Could not instantiate global transform class org.spockframework.compiler.SpockTransform specified at jar:file:/home/foobar/.m2/repository/org/spockframework/spock-core/2.0-groovy-3.0/spock-core-2.0-groovy-3.0.jar!/META-INF/services/org.codehaus.groovy.transform.ASTTransformation  because of exception org.spockframework.util.IncompatibleGroovyVersionException: The Spock compiler plugin cannot execute because Spock 2.0-groovy-3.0 is not compatible with Groovy 4.0.0-beta-2. For more information (including enforce mode), see http://docs.spockframework.org (section 'Known Issues').
Spock artifact: file:/home/foobar/.m2/repository/org/spockframework/spock-core/2.0-groovy-3.0/.../spock-core-2.0-groovy-3.0.jar
Groovy artifact: file:/home/foobar/.gradle/caches/modules-2/files-2.1/org.apache.groovy/groovy/4.0.0-beta-2/.../groovy-4.0.0-beta-2.jar

1 error


FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':compileTestGroovy'.
> Compilation failed; see the compiler error output for details.

After some searching on the Internet, we can find out that tests in Gradle are (by default) executed in a separate JVM, to it is required to pass the aforementioned system property to the test task:

1
2
3

test {
    systemProperty "spock.iKnowWhatImDoing.disableGroovyVersionCheck", "true"   //doesn't work
}

However, it still fails:

> Task :compileTestGroovy FAILED
startup failed:
Could not instantiate global transform class org.spockframework.compiler.SpockTransform specified at ...

Hmm, maybe it would be better to refer the Gradle test task by type?

1
2
3

tasks.withType(Test).configureEach {
    systemProperty "spock.iKnowWhatImDoing.disableGroovyVersionCheck", "true"   //doesn't work either
}

It fails with the same error message :-/. What is wrong?

The key difference - quite easy to miss - is the fact the exception is thrown not during the test execution, but during the compilation phrase. Then, the main Spock AST transformation checks if it could be applied:

> Task :compileTestGroovy FAILED   <--------- :compileTestGroovy not :test !
startup failed:
...

It is definitely less popular case, and I had a problem to find any sensible source of wisdom proposing a complete solution. One of the possible option (which I talk about deeper in the article about enabling preview features of the new Java versions) is CompileOptions (e.g. options.compilerArgs.add('--enable-preview') in the GroovyCompile task. The second might be GroovyCompileOptions also in GroovyCompile. However, it does not bring expected result. After some digging, I was able to find out, that as the Groovy compilation is performed in the forked JVM and the right way to achieve the goal is:

1
2
3

tasks.withType(GroovyCompile).configureEach {
    options.forkOptions.jvmArgs << '-Dspock.iKnowWhatImDoing.disableGroovyVersionCheck=true'    //it works! But be aware that "options" not "groovyOptions"!
}

And it works as expected:

> Task :compileTestGroovy
Executing Spock 2.0-groovy-3.0 with NOT compatible Groovy version 4.0.0-beta-2 due to set spock.iKnowWhatImDoing.disableGroovyVersionCheck system property set. This is unsupported and may result in weird runtime errors!

> Task :test
...

displaying a nice disclaimer :). Spock 2.0-groovy-3.0 is used together with Groovy 4.

Btw, if you see just not very helpful error message:

Could not instantiate global transform class org.spockframework.compiler.SpockTransform specified at
jar:file:/.../spock-core-.../org.codehaus.groovy.transform.ASTTransformation
because of exception java.lang.reflect.InvocationTargetException

it is caused by GROOVY-9469 which affects Groovy 3 and was fixed in 3.0.3. Upgrade Groovy to see more verbose error message.

Maven

As Groovy 4 changed its groupId from org.codehaus.groovy to org.apache.groovy it is required to add implicit exclusion for spock-2.0-groovy-3.0 transitive dependencies:

<dependencies>
    <dependency>
        <groupId>org.spockframework</groupId>
        <artifactId>spock-core</artifactId>
        <version>${spock.version}</version>
        <scope>test</scope>
        <exclusions>
            <!-- to do not have conflicting Groovy versions - Groovy 4 changed groupId from "org.codehaus.groovy" to "org.apache.groovy" -->
            <exclusion>
                <groupId>org.codehaus.groovy</groupId>
                <artifactId>*</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.apache.groovy</groupId>
        <artifactId>groovy</artifactId>
        <version>4.0.0-beta-2</version>
    </dependency>
</dependencies>

to have your tests executed with Groovy 4 instead of (silently) with Groovy 3, taken from a transitive dependency of Spock 2.0-groovy-3.0!

[INFO] --- gmavenplus-plugin:1.13.0:compileTests (default) @ spock2-groovy4-maven ---
[INFO] Using isolated classloader, without GMavenPlus classpath.
[INFO] Using Groovy 3.0.8 to perform compileTests.     <---- Without exclusion, it is executed "silently" with old Groovy! 
[INFO] Parallel parsing disabled.
[INFO] Compiled 1 file.

It won’t be required for Spock 2.0-groovy-4.0, once available.

Writing my Spock, Groovy and modern Java related tutorials, I try to cover also the Maven users, which still is quite popular build tool :-). Usually, it takes much more (XML) lines to achieve the same in Maven in comparison to Gradle, however, sometimes it just works flawlessly without any additional effort. Unfortunately, it was not the case this time.

The build with Spock-2.0-groovy-3.0 and Groovy 4.0.0 by default fails with:

[ERROR] Failed to execute goal org.codehaus.gmavenplus:gmavenplus-plugin:1.8.1:compileTests (default) on project spock2-migration-maven: Error occurred while calling a method on a Groovy class from classpath. InvocationTargetException: startup failed:
[ERROR] Could not instantiate global transform class org.spockframework.compiler.SpockTransform specified at jar:file:/home/foobar/.m2/repository/org/spockframework/spock-core/2.0-groovy-3.0/spock-core-2.0-groovy-3.0.jar!/META-INF/services/org.codehaus.groovy.transform.ASTTransformation  because of exception org.spockframework.util.IncompatibleGroovyVersionException: The Spock compiler plugin cannot execute because Spock 2.0-groovy-3.0 is not compatible with Groovy 4.0.0-beta-2. For more information (including enforce mode), see http://docs.spockframework.org (section 'Known Issues').
[ERROR] Spock artifact: file:/home/foobar/.m2/repository/org/spockframework/spock-core/2.0-groovy-3.0/spock-core-2.0-groovy-3.0.jar
[ERROR] Groovy artifact: file:/home/foobar/.m2/repository/org/apache/groovy/groovy/4.0.0-beta-2/groovy-4.0.0-beta-2.jar
[ERROR] 
[ERROR] 1 error
[ERROR] -> [Help 1]

To make the long story short, I started with the plain Maven properties:

<project>
...
  <properties>
     <!-- doesn't work -->
     <spock.iKnowWhatImDoing.disableGroovyVersionCheck>true</spock.iKnowWhatImDoing.disableGroovyVersionCheck>
  </properties>
...
</project>

It fails. Next, I added the following to the compiler parameters:

    <!-- ... -->
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.1</version>
                <configuration>
                    <!-- doesn't work -->
                    <compilerArgs>
                        <arg>-v</arg>
                        <arg>-Dspock.iKnowWhatImDoing.disableGroovyVersionCheck=true</arg>
                    </compilerArgs>
                </configuration>
            </plugin>
            <!-- ... -->
        </plugins>
    </build>
    <!-- ... -->

however, it doesn’t affect Groovy compilation itself. I even took a look at the gmavenplus plugin parameters and Groovy’s CompilerConfiguration , but without any effect. In the end, I was able to achieve that with another plugin properties-maven:

    <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>properties-maven-plugin</artifactId>
        <version>1.0.0</version>
        <executions>
            <execution>
                <goals>
                    <goal>set-system-properties</goal>
                </goals>
                <configuration>
                    <properties>
                        <!-- works fine -->
                        <property>
                            <name>spock.iKnowWhatImDoing.disableGroovyVersionCheck</name>
                            <value>true</value>
                        </property>
                    </properties>
                </configuration>
            </execution>
        </executions>
    </plugin>

resulting in:

[INFO] --- gmavenplus-plugin:1.8.1:compileTests (default) @ spock2-migration-maven ---
[INFO] Using isolated classloader, without GMavenPlus classpath.
[INFO] Using Groovy 4.0.0-beta-2 to perform compileTests.
Executing Spock 2.0-groovy-3.0 with NOT compatible Groovy version 4.0.0-beta-2 due to set spock.iKnowWhatImDoing.disableGroovyVersionCheck system property set. This is unsupported and may result in weird runtime errors!
[INFO] Compiled 10 files.
[INFO] 
[INFO] --- maven-surefire-plugin:3.0.0-M4:test (default-test) @ spock2-migration-maven ---
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
...

It turned out to be quite easy (if you know what to use :-) ), but - unless I missed something - it is somehow not much covered in the existing tutorials or documentation.

Potential issue in IntelliJ IDEA

Even though, having your up-and-running Maven project configured as above, you might still encounter the incompatibility error after importing it in Idea:

Groovyc: While compiling [tests of spock2-groovy4-gradle.test]: Could not instantiate global transform class org.spockframework.compiler.SpockTransform specified at jar:file:/home/…/spock-core-2.0-groovy-3.0.jar!/META-INF/services/org.codehaus.groovy.transform.ASTTransformation because of exception java.lang.NullPointerException: Cannot invoke “java.net.URL.toString()” because the return value of “java.security.CodeSource.getLocation()” is null

~~The problem is caused by a limitation of Idea (IDEA-287642 ), which does not allow providing custom system properties for groovyc compilation (just for javac). You might want to vote for that issue.~~ In fact, as Paul King pointed out, there is a way to set that property for all the compilers used in the project with Settings → Compiler → Shared build process VM options. In the context of Spock, it should be enough.

The error itself is not very meaningful (NullPointerException) which in turn is caused by an unexpected behavior of Idea, reported as IDEA-287643 .

It is worth to mention, the same problem might occur also with Gradle, once “Build and run” is manually set to “Idea” (instead of “Gradle” - default behavior, but not the fastest one).

Summary

The main reason, I implemented that feature in Spock was development of Groovy 4. I didn’t feel comfortable seeing The Groovy Team to be forced to use the Spock snapshot version in the master (Groovy 4.x) builds. With Spock-2.0-M3 it was simplified . The same construction can be used in your project while playing with Groovy 4 (in beta as of XI 2021), in the way I precised for Gradle and Maven (which in turn can be really confusing on their own way which motivated me to write this short post :-) ). There is a plan to add the official support for Groovy 4 in Spock, once it is closer to Groovy 4.0.0-final.

Btw, there is a code repository with working examples for Gradle and Maven.

P.S. If you bumped into that topic during Spock 1.3 -> 2.0 migration, there is a dedicated migration guide article .

Updated 2021-01-31. Added section about potential issue with IntelliJ IDEA.
Updated 2021-02-02. Add workaround suggested by Paul King about the potential issue with IntelliJ IDEA.

Lead photo by geralt, published in Pixabay, Pixabay License.

Unified Gradle projects releasing to Maven Central in 2021 + migration guide

Marcin Zajączkowski — Fri, 26 Feb 2021 10:00:00 +0200

As a counterbalance to an imminent JCenter shutdown, get know the unified (and united) power of releasing to Maven Central with a brand new Gradle Nexus Publish Plugin. A migration guide from the old plugins is included.

TL;TR. A brand new Gradle Nexus Publish Plugin is currently the best solution for releasing projects to Maven Central.

2 big things in Q1 2021

The beginning of 2021 electrified the JVM ecosystem by an unexpected decision of the fast upcoming shutdown of JCenter , a quite popular place to keep FOSS projects’ binary artifacts. Business is for making money, not for supporting the community that helped the company got to the successful IPO a few months earlier, so the need “to streamline the productivity of the JFrog Platform” has to be understood.

As a (positive) side effect, I and Marc Phillip streamlined our focus to release a stable version of our unified Gradle plugin for publishing projects to Maven Central. 1.0.0 hit the repository soon after. That version provides an initial set of functionality which should allow the majority of the Gradle projects to publish artifacts to Maven Central (aka The Central Repository).

Please note, this is not an article about migration from JCenter to Maven Central, only about migration from the Gradle Nexus Staging Plugin and Nexus Publish Plugin duo to the new, unified Gradle Nexus Publish Plugin. People switching their projects from JCenter definitely can use the new plugin, but I plan to cover the migration from JCenter in a separate post, but in the meantime you can take a look at this sample project.

A story of three plugins

In 2015, I created gradle-nexus-staging-plugin which was providing - not available earlier for the Gradle users - an ability to close and release staging repositories in Nexus repository manager without touching Nexus UI. It opened an opportunity to manage releasing Gradle projects to Maven Central completely from code (also using a CI server). Over the years, it has been adopted by the various projects and companies across the globe , however there was a small problem. Due to technical limitations in the publishing process in Gradle, it was required to use heuristics to track implicitly created staging repositories, what often failed for multiple repositories in a given state. The situation became even worse when Travis changed its network architecture in late 2019 and the majority of releases started to fail.
Here, Marc Philipp entered the stage who created Nexus Publish Plugin which was enriching the publishing mechanism in Gradle to explicitly create staging repositories and publish (upload) artifacts directly to it.

Those two plugins nicely worked together, providing a reliable way to handle publishing artifacts to Maven Central (and to other Nexus instances in general). However, the need of using two plugins was very often confusing for users. As a result, an idea to create one plugin mixing the aforementioned capabilities emerged. It took us some time, but sunsetting JCenter motivated us to intensify work and Gradle Nexus Publish Plugin 1.0.0 has been released in February 2021 as joint effort.

Migration guide

The new plugin was intended as a natural successor of the aforementioned two plugins. We tried to keep the syntax unchanged where feasible. As a result a migration guide - in the majority of cases - is straightforward.

Plugins

Let’s start with the plugins declaration.

plugins {   //old
    id "io.codearte.nexus-staging" version "0.22.0"
    id "de.marcphilipp.nexus-publish" version "0.4.0"
}

The old two has to be replaced with the new one:

1
2
3

plugins {
    id "io.github.gradle-nexus.publish-plugin" version "1.0.0"
}

Configuration - simple case

If only default plugins settings were used:

nexusStaging {   //old
    packageGroup = "com.example.mycompany.myproject"
    stagingProfileId = "yourStagingProfileId"
}

nexusPublishing {
    repositories {
        sonatype()
    }
}

packageGroup and stagingProfileId have to be moved to the nexusPublishing section:

nexusPublishing {
    packageGroup = "com.example.mycompany.myproject"  //defaults to 'project.group'
    repositories {
        sonatype {   //custom repository name - 'sonatype' is pre-configured
                     //for Sonatype Nexus (OSSRH) which is used for The Central Repository
            stagingProfileId = "yourStagingProfileId" //can reduce execution time by even 10 seconds
        }
    }
}

Due to the improved support for releasing to multiple Nexus instances in one project, the credentials configuration has changed in non-backward compatible way. gradle-nexus-staging-plugin by default was using nexusUsername and nexusPassword project properties which could be reused by nexus-publish-plugin. With the new plugin it has been unified to sonatypeUsername and sonatypePassword, where a prefix reflects a name of a configured repository (here sonatype).

Configuration - more customizations

Having more customizations in place:

//majority of the properties is optional, you might just don't have them in your project       //old
nexusStaging {
    packageGroup = "com.example.mycompany.myproject"
    stagingProfileId = "yourStagingProfileId"

    //by default project properties 'nexusUsername' and 'nexusPassword' are used
    username = project.findProperty("...")   
    password = project.findProperty("...")
    numberOfRetries = 40
    delayBetweenRetriesInMillis = 3000
}

nexusPublishing {
    repositories {
        sonatype {
            //by default value from above 'nexusStaging' closure is used (alternative project properties 'sonatypeUsername' and 'sonatypePassword')
            username = project.findProperty("...")  
            password = project.findProperty("...")
        }
    }

    clientTimeout = Duration.ofSeconds(300)
    connectTimeout = Duration.ofSeconds(60)

}

they also have to be moved to the nexusPublishing closure:

nexusPublishing {
    packageGroup = "com.example.mycompany.myproject"  //defaults to 'project.group'

    repositories {
        sonatype {   //or custom repository name
            stagingProfileId = "yourStagingProfileId" //can reduce execution time by even 10 seconds

            //defaults to project properties 'sonatypeUsername' and 'sonatypePassword', where 'sonatype' is name of configured repository
            username = project.findProperty("...")  
            password = project.findProperty("...")
        }
    }

    clientTimeout = Duration.ofSeconds(300)
    connectTimeout = Duration.ofSeconds(60)

    transitionCheckOptions {
        maxRetries.set(40)
        delayBetween.set(java.time.Duration.ofMillis(3000))
    }
}

It is worth to mention - in addition to changed default project properties to read credentials from - that the configuration of retries during the close and release attempts has been renamed to transitionCheckOptions with more modern duration types.

Tasks - simple case

At the task level - due to support for multiple repositories - the following common execution:

./gradlew publish closeAndReleaseRepository        //old

has to be enhanced with a repository name:

./gradlew publishToSonatype closeAndReleaseSonatypeStagingRepository

To short the second task name slightly, there is still a simplified variant closeAndReleaseStagingRepository which closes and releases the staging repositories in all configured Nexus instances (one by default).

Summary

For the new projects, using gradle-nexus-publish-plugin is the only logical choice. The main authors’ effort will be put in its development. The whole process of onboarding to Soantype Nexus (together with a sample project) is nicely described by Rob Winch .

For the projects migrating from releasing from JCenter to Maven Central - I plan to cover it in a separate blog post - it also holds.

I hope this short blog post clearly showed that migration - in the majority of cases - from the old due-solution should be trouble-free. All the projects should - eventually - migrate to the new plugin.

For existing projects, temporary staying with the already working (old) duo-solution also is an acceptable choice. The migration might wait for the next big configuration change in the project (e.g. a migration to Gradle 7). The new plugin is less mature and there might be some features missing. In that case, however, please let us know !

Lead photo by TheDigitalArtist, published in Pixabay, Pixabay License.

Simpler and safer artifact signing on CI server with Gradle

Marcin Zajączkowski — Wed, 03 Jun 2020 10:00:00 +0200

Check out how to simplify your Gradle configuration for artifact signing (and releasing) on a CI server (and along the way avoid CVE-2020-13165 in Gradle <6.5).

TL;TR. To get just the short migration recipe, jump to the second to last section.

Releasing from a CI server

Testing, build and releasing (deploying) from CI servers has been more and more popular in the recent years. In many organizations this is something natural. Also multiple FOSS project has started to publish their artifacts to Maven Central (aka The Central Repository) directly from a CI server. This is very nice, as it reduces time to market (or time to release), makes the process less error-prone and removes all the release-related legwork (git commit -m "Trigger release" -m "[#DO_RELEASE]" --allow-empty && git push in many cases can be just enough!).

However, there are also drawbacks of that approach. The CI server - especially in non enterprise environments - is very often a place which has access to all secrets needed to sign and release the software (e.g. the private signing PGP key or the artifacts repository credentials). A data breach from one of the popular SaaS CI servers could open an opportunity to inject malicious code into various popular FOSS projects… Luckily, usually it works fine (or is not actively exploited), however, I cannot resist to mention the vulnerability I have found in Travis back in 2016 which was causing - in some circumstances - to have the build secrets available to anybody in plain text :-/.

There are also some other possible vectors of attack, however, in that article I would like to focus on mitigating the risk of revealing secrets in the system logs, which very often - especially in the FOSS projects - are publicly available.

Gradle and security

Recently, it is pretty visible that the Gradle team takes security very seriously. In January 2020, the HTTP access for Gradle services has been disabled. In February, the verification of Gradle Wrapper was added as an official action for GitHub Actions. A few weeks later, Gradle 6.2 introduced a built-in ability to verify the checksums (and signatures) of the artifacts used in the build process. Starting with Gradle 6.4, using the DEBUG log statement clearly states that it may reveal some secrets and shouldn’t be used in non private builds.

Just released Gradle 6.5 fixes another vulnerability (CVE-2020-13165 ) which I discovered and reported to the Gradle team recently. The earlier Gradle versions might reveal the passphrase for a PGP key used to sign published artifacts, if gpg2 was used together with the INFO log level. This mostly affects CI servers, especially those with the limited (and faulty) secret masking.

I will use CVE-2020-13165 as an excuse to present the simpler artifact signing in Gradle, designed especially for CI servers, which unfortunately is very little-known (an order of magnitude less results on GitHub that a regular useGpgCmd() for GnuPG 2).

PGP artifact signing in Gradle

Originally, Gradle was using the Bouncy Castle library to deal with keys stored in the GnuPG 1 format. However, GnuPG 2 changed the way how keys are kept and their handling has been delegated to a gpg2 process spawn during the Gradle build (which also made it possible to use a system defined gpg-agent program). It was not very convenient on a CI server, where the key usually had to be kept in the encrypted way in the code repository, decrypted with some shell magic during a build using the provided secret and imported into the locally stoped keyring. The procedure was not fully CI server agnostic. In addition, that file could persist on a file system, which especially combined with not using a passphrase could generate some security risk.

Simpler and safer artifact signing

In April 2019, Gradle 5.4 introduced a very nice feature which unfortunately was not even mentioned in the release notes :-(. Marc Philipp (who you may know from his work in JUnit 4/5) implemented a PGP artifact signing optimized for CI servers. Having that in place remote artifact signing is a piece of cake ;-).

Export the private OpenPGP key intended for the CI environment to the ascii-armored form:

$ gpg --export-secret-keys --armor KEY_ID > private-ci.key

Create two secret environment variables on a CI server:

ORG_GRADLE_PROJECT_signingKey = <content of exported private key file>
ORG_GRADLE_PROJECT_signingPassword = <key passphrase>

Enable in-memory signatory provider:

signing {
    useInMemoryPgpKeys(findProperty("signingKey"), findProperty("signingPassword"))
    sign ...
}

And that’s all. Isn’t is simple?

Btw, that mechanism supports also using subkeys (which provide some benefits ).

Summary

As a bonus to simplicity of using the in-memory signatory provider, that method prevents aforementioned passphrase revealing, even in Gradle <6.5. Therefore, it hard to find a good argument against migration :).

Lead photo by Steve Buissinne, published in Pixabay, Pixabay License.

PIT, JUnit 5 and Gradle - with just one extra line of configuration

Marcin Zajączkowski — Thu, 27 Feb 2020 00:00:00 +0000

Discover dead simple, improved PIT and JUnit 5 configuration in Gradle (with gradle-pitest-plugin 1.4.7+).

JUnit 5 is undeniably more and more popular nowadays. While there is a dedicated plugin for PIT for JUnit 5 and it has been supported by gradle-pitest-plugin for years, it was required to add a few lines of boilerplate code to achieve that. Recently, I’ve got a question if it could be simplified. I liked it. Challenge accepted :-).

Generic approach with ‘buildscript {}’

First, take a look at the generic approach with the buildscrip {} code block, which remembers times of Gradle 0.x:

buildscript {
   repositories {
       mavenCentral()
       gradlePluginPortal() //optionally, if any plugin is not available in Maven Central
   }
   configurations.maybeCreate('pitest')
   dependencies {
       classpath 'info.solidsoft.gradle.pitest:gradle-pitest-plugin:1.4.6'
       pitest 'org.pitest:pitest-junit5-plugin:0.12'
   }
}

apply plugin: 'java'
apply plugin: 'info.solidsoft.pitest'

pitest {
    testPlugin = 'junit5'
}

Just 3 extra lines. Acceptable, however buildscript {} for plugin configuration is somehow verbose by itself.

Modern approach with ‘plugins {}‘
(with older gradle-pitest-plugin)

The modern variant with plugins {} should be shorter:

buildscript {
   repositories {
       mavenCentral()
   }
   configurations.maybeCreate('pitest')
   dependencies {
       pitest 'org.pitest:pitest-junit5-plugin:0.12'
   }
}

plugins {
    id 'java'
    id 'info.solidsoft.pitest' version '1.4.6'
}

pitest {
    testPlugin = 'junit5'
}

Unfortunately, the compact syntax of the plugin {} block is wasted by a need to add an extra dependency pitest-junit5-plugin used by gradle-pitest-plugin in runtime in the buildscript {} block - 10 lines extra. Highly disappointing ;-).

Modern improved approach with ‘plugins {}‘
and gradle-pitest-plugin 1.4.7+

With just released gradle-pitest-plugin 1.4.7 we can forget about all the boilerplate code:

plugins {
    id 'java'
    id 'info.solidsoft.pitest' version '1.4.7'
}

pitest {
    //adds dependency to org.pitest:pitest-junit5-plugin and sets "testPlugin" to "junit5"
    junit5PluginVersion = '0.12'
}

Just one line junit5PluginVersion = '0.12' which under the hood adds a pitest-junit5-plugin dependency in the required versions and activates junit5 as testPlugin used by PIT. Doesn’t it look nice? :-)

Summary

In this short blog post I presented how the configuration of the PIT, JUnit 5 and Gradle(-pitest-plugin) could be simplified with just a few changes in the plugin itself. All thanks to the question by John Scancella and the idea that sprang into my mind how to implement it in the smart way.

Therefore, I encourage you to report (sensible) ideas for improvements and things that limit you in the projects you use (or even better a pull/merge request after the initial discussion with the maintainer). Maybe it will implemented (accepted) for common good :-).

Migrating Spock 1.3 tests to Spock 2.0

Marcin Zajączkowski — Thu, 02 Jan 2020 10:00:00 +0200

Get know what you can expect from Spock 2.0 M2 (based on JUnit 5), how to migrate to it in Gradle and Maven, and why it is important to report spotted problems :).

Important note. I definitely do not encourage you to migrate your serious real-life project to Spock 2.0 M1 or M2 for good! This is the first (pre-)release of 2.x with not finalized API, intended to gather user feedback related to internal Spock migration to JUnit Platform.

This blog post arose to try to encourage you to make a test migration of your projects to Spock 2.0, see what started to fail, fix it (if caused by your tests) or report it (if it is a regression in Spock itself). As a result - at the Spock side - it will be possible to improve the code base before Milestone 2. The benefit for you - in addition to contribution to the FOSS project :-) - will be awareness of required changes (kept in a side branch) and readiness to migration once Spock 2.0 is more mature.

I plan to update this blog post when the next Spock 2 versions are available.

Updated 2020-02-10 to cover Spock 2.0 M2 with a dedicated Groovy 3 support.
Updated 2020-03-05. New point about dealing with NoClassDefFoundError: LambdaExpression.
Updated 2020-03-11. Detecting “no tests” issue with multiple test tasks.

Powered by JUnit Platform

The main change in Spock 2.0 M1 is migration to JUnit 5 (precisely speaking to execute tests with JUnit Platform 1.5, part of JUnit 5 instead of the JUnit 4 runner API). This is very convenient as Spock tests should be automatically recognized and executed everywhere the JUnit Platform is supported (IDEs, build tools, quality tools, etc.). In addition, the features provided by the platform itself (such as parallel test execution) should be (eventually) available also for Spock.

Gradle

To bring Spock 2 to a Gradle project it is needed to bump the Spock version:

testImplementation('org.spockframework:spock-core:2.0-M2-groovy-2.5')

and activate tests execution by JUnit Platform:

test {
    useJUnitPlatform()
}

Update 20200218. It is enough, but as Tomek Przybysz reminded in his comment, Gradle by default doesn’t fail if not tests are found. It may lead to a situation when after making that switch a build finished successfully, giving a false sense of security, while there are no tests executed at all.
It is a known issue in Gradle, not only limited to Spock. As a workaround the aforementioned configuration might to extended to:

test {
    useJUnitPlatform()

    afterSuite { desc, result ->
        if (!desc.parent) {
            if (result.testCount == 0) {
                throw new IllegalStateException("No tests were found. Failing the build")
            }
        }
    }
}

Or the more sophisticated version for multiple test tasks (e.g. unit/integration and functional):

tasks.withType(Test).configureEach { testTask ->
    testTask.configure {
        useJUnitPlatform()

        afterSuite { desc, result ->
            if (!desc.parent) {
                if (result.testCount == 0) {
                    throw new IllegalStateException("No tests were found. Failing the build")
                }
            }
        }
    }
}

Maven

With Maven on the other hand, it is still required to switch to a never Spock version:

<dependency>
  <groupId>org.spockframework</groupId>
  <artifactId>spock-core</artifactId>
  <version>2.0-M2-groovy-2.5</version>
  <scope>test</scope>
</dependency>

but that is all. The Surefire plugin (if you use version 3.0.0+) executes JUnit Platform tests by default, if junit-platform-engine (a transitive dependency of Spock 2) is found.

The minimal working project for Gradle i Maven is available from GitHub .

Other changes

Having such big change as migration to JUnit Platform, number of other changes in Spock 2.0 M1 is limited, to make finding a reason of potential regressions a little bit easier. As a side effects of the migration itself, the required Java version is 8.

In addition, all parameterized tests are (finally) “unrolled” automatically. That is great, however, currently there is no way to “roll” particular tests, as known from spock-global-unroll for Spock 1.x.

Some other changes (such as temporarily disabled SpockReportingExtension) can be found in the release notes .

More (possibly breaking) changes are expected to be merged into Milestone 2.

Issue with JUnit 4 rules

The tests using JUnit 4 @Rules (or @ClassRules) are expected to fail with an error message suggesting that requested objects were not created/initialized before a test (e.g. NullPointerException or IllegalStateException: the temporary folder has not yet been created) or were not verified/cleaned up after it (e.g. soft assertions from AssertJ). The Rules API is no longer supported by JUnit Platform. However, to make the migration easier (@TemporaryFolder is probably very often used in Spock-based integration tests), there is a dedicated spock-junit4 which internally wraps JUnit 4 rules into the Spock extensions and executes it in the Spock’s lifecycle. As it is implemented as a global extension, the only required thing to add is another dependency. In Gradle:

testImplementation 'org.spockframework:spock-junit4:2.0-M2-groovy-2.5'

or in Maven:

<dependency>
    <groupId>org.spockframework</groupId>
    <artifactId>spock-junit4</artifactId>
    <version>2.0-M2-groovy-2.5</version>
    <scope>test</scope>
</dependency>

That makes migration easier, but it is good to think about a switch to native Spock counterpart, if available/feasible.

Groovy 3 support

Updated 2020-02-10. The whole Groovy 3 section was added to cover changes in Spock 2.0 Milestone 2.

After my complains about runtime failure when Spock 2.0 M1 is used with Groovy 3.0, I rolled up my sleeves to check how hard would it be to provide that support. It took me some time, however, after after 23 (multiple times rebased) commits , constructive feedback from other Spock contributors and fruitful discussion with the Groovy developers, the support for Groovy 3 has been merged and is available as the main feature of Spock 2.0 M2, released right after Groovy 3.0.0 final.

To use Spock 2.0 M2 with Groovy 3 it is enough to just use the spock-*-2.0-M1 artifacts with the -groovy-3.0 suffix.

It is worth noting that Groovy 3 is not backward compatible with Groovy 2. To keep one Spock codebase, there is a layer of abstraction in Spock to allow to build (and use) the project with both Groovy 2 and 3. As a result, an extra artifact spock-groovy2-compat is (automatically) used in projects with Groovy 2. It is very important to do not mix the spock-*-2.x-groovy-2.5 artifacts with the groovy-*-3.x artifacts on a classpath. This may result in weird runtime errors.

I’m really happy that developers can immediately start testing the new Groovy 3.0.0 with Spock 2.0-M2 in their projects. In addition, as Spock is quite important (and low level) project in the Groovy ecosystem, it was nice to confirm that Groovy 3 works properly with it (and to report - along the way - a few minor detected issues to make Groovy even better ;-) ).

Other issues and limitations

NoClassDefFoundError: LambdaExpression with Groovy 2.5

Using Spock 2.0-M2 with Groovy 2.5 it is possible to get the following errors:

java.lang.NoClassDefFoundError: org/codehaus/groovy/ast/expr/LambdaExpression
java.lang.NoClassDefFoundError: org/codehaus/groovy/ast/expr/MethodReferenceExpression

As Groovy 2 and 3 are not backward compatible, there is an abstraction layer which allows to build (and use) Spock with two different Groovy versions. However, to achieve that it was necessary to add dummy LambdaExpression and MethodReferenceExpression in an extra Spock dependency spock-groovy2-compat used in the -groovy-2.5 variant.

It is automatically fetched by Maven/Gradle for spock-core, but in the situation you disabled transitive dependency for Spock:

dependencies {
    testImplementation('org.spockframework:spock-core:2.0-M2-groovy-2.5') {
        transitive = false
    }
    ...
}

@Grab('org.spockframework:spock-core:2.0-M2-groovy-2.5;transitive=false')

they are missing. In that case you might want to add spock-groovy2-compat dependency explicitly, e.g.:

dependencies {
    testImplementation('org.spockframework:spock-core:2.0-M2-groovy-2.5') {
        transitive = false
    }
    testImplementation('org.spockframework:spock-groovy2-compat:2.0-M2-groovy-2.5')
}

or exclude just Groovy dependencies (not all of them):

dependencies {
    testImplementation('org.spockframework:spock-core:2.0-M2-groovy-2.5') {
        exclude group: 'org.codehaus.groovy'
    }
}

Thanks to Paul King for suggesting that point.

Missing Groovy 3 support

Updated 2020-02-10. This section originally refereed to limitations of Spock 2.0 M1. Milestone 2 supports Groovy 3.

Spock 2.0 M1 is compiled and tested only with Groovy 2.5.8. As of M1, execution with Groovy 3.0 is currently blocked at runtime. Unfortunately, instead of a clear error message about incompatible Groovy version there is only a very cryptic error message:

Could not instantiate global transform class org.spockframework.compiler.SpockTransform specified at
jar:file:/.../spock-core-2.0-M1-groovy-2.5.jar!/META-INF/services/org.codehaus.groovy.transform.ASTTransformation
because of exception java.lang.reflect.InvocationTargetException

It is already reported and should be enhanced by M2.

Sadly, the limitation to Groovy 2.5 only, reduces potential feedback from people experimenting with Groovy 3 which is pretty close to a stable version (RC2 as of 2019/2020). It’s especially inconvenient as many Spock tests just work with Groovy 3 (of course there are some corner cases). There is a chance that Spock 2 before getting final will be adjusted to changes in Groovy 3 or at least the aforementioned hard limitation will be lifted. In the meantime, it is required to test Groovy 3 support with the snapshot version - 2.0-groovy-2.5-SNAPSHOT (which has that check disabled).

Summary

The action to do after reading this post is simple. Try to temporarily play with Spock 2.0 M2 in your projects and report any spotted issues, to help make Spock 2.0 even better :).

Deploy to Maven Central using API key (aka auth token)

Marcin Zajączkowski — Tue, 08 Sep 2015 10:00:00 +0200

How to communicate with Maven Central/Nexus without using the password kept locally unencrypted (especially with Gradle, but not limited to it).

Rationale

Unfortunately, Gradle (and many other build tools) does not provide any mechanism to locally keep passwords encrypted (or at least encoded). Without that even such a simple activity like showing your global Gradle configuration (~/.gradle/gradle.properties) to a colleague is uncomfortable, not to mention more serious risks associated with storing passwords on a disk in a plain-text form (see among others Sony Pictures Entertainment hack ). It is Gradle, so with all Groovy magic under the hood it would be possible to implement an integration with a system keyring on Linux to fetch a password, but I’m not aware of any existing plugin/mechanism to do that and I would rather prefer not to write it.

Another issue is that nowadays, in the world of ubiquitous automation and cloud environments it is common to use API keys which allow to perform given operation(s). However, its lost doesn’t provide an attacker a possibility to hijack the account (e.g. token cannot be used neither to log into an administration panel nor to change of email or password which requires additional authentication).

It is very important if you need to keep valid credentials on a CI server to make automatic or even continuous releases. Thanks to my ~~gradle-nexus-staging-plugin~~ (Update 2021. Check out my new plugin gradle-nexus-publish-plugin ) there is no need to do any manual steps in Nexus GUI to promote artifacts to Maven Central, so this was the next issue I wanted to deal with for my private and our FOSS projects in Codearte .

Nexus API key generation

Internet search for “maven central api key” wasn’t helpful, so I started digging into Nexus REST API documentation and I’ve found that in fact there is a (non widely known) way to generate and use an API key (aka an auth token).

Log into Nexus hosting Sonatype OSS Repository Hosting (or your own instance of Nexus).
Click on your login name in right-upper corner and choose “Profile”.
From the drop-down list with “Summary” text select “User Token”.
Click “Access User Token”.

Enter your password
Copy and paste your API username and API key (into your ~/.gradle/gradle.properties file or a CI server configuration).
Work as usual with a little safer way.

Summary

It is good that using API keys is possible to deploy artifacts to Maven Central/Nexus and it is very easy to set it up. Someone could argue that the permission policy is coarse-grained (nothing or all operations except password/email change), but in my opinion it seems to be enough for the artifact repository system class. In addition, such an approach should work also with Sbt, Ivy, Leiningen and everything else that tries to upload artifacts into Maven Central (including Maven itself by removing limitations of the master password encryption with settings-security.xml). Hopefully, that post will make it widely known.

gradle on Solid Soft

Running Spock with unsupported Groovy version (Gradle + Maven)

Historical background

Possible options

Spock 1.3 and Groovy 3.0

Spock 2.0+ and Groovy 3.0

Spock 2.0/2.1 and Groovy 4.0

Gradle

Maven

Potential issue in IntelliJ IDEA

Summary

Unified Gradle projects releasing to Maven Central in 2021 + migration guide

2 big things in Q1 2021

A story of three plugins

Migration guide

Plugins

Configuration - simple case

Configuration - more customizations

Tasks - simple case

Summary

Simpler and safer artifact signing on CI server with Gradle

Releasing from a CI server

Gradle and security

PGP artifact signing in Gradle

Simpler and safer artifact signing

Summary

PIT, JUnit 5 and Gradle - with just one extra line of configuration

Generic approach with ‘buildscript {}’

Modern approach with ‘plugins {}‘ (with older gradle-pitest-plugin)

Modern improved approach with ‘plugins {}‘ and gradle-pitest-plugin 1.4.7+

Summary

Migrating Spock 1.3 tests to Spock 2.0

Powered by JUnit Platform

Gradle

Maven

Other changes

Issue with JUnit 4 rules

Groovy 3 support

Other issues and limitations

NoClassDefFoundError: LambdaExpression with Groovy 2.5

Missing Groovy 3 support

Summary

Deploy to Maven Central using API key (aka auth token)

Rationale

Nexus API key generation

Summary

Modern approach with ‘plugins {}‘
(with older gradle-pitest-plugin)

Modern improved approach with ‘plugins {}‘
and gradle-pitest-plugin 1.4.7+