Develocity integration

[ribafish]

As one of the impactful Eclipse projects, we (the Develocity Solutions team) would like to invite you to be a part of the Eclipse Develocity evaluation initiative.

Description

This improvement will enhance the functionality of the Eclipse PDE build by publishing a build scan for every CI build and for every local build from an authenticated Eclipse committer. The build will not fail if publishing fails. The build scans of the Eclipse PDE project are published to the Develocity instance at develocity-staging.eclipse.org, hosted by the Eclipse Foundation and run in partnership between the Eclipse and Gradle. This Develocity instance has all features and extensions enabled and is freely available for use by the Eclipse PDE project and all other Eclipse projects.

On this Develocity instance, the Eclipse PDE project will have access not only to all of the published build scans but also to other aggregate data features such as:

• Dashboards to view all historical build scans, along with performance trends over time. For example, look at the Quarkus instance trends dashboard with the ci filter applied.
• Build failure analytics for enhanced investigation and diagnosis of build failures. You can also explore the Quarkus instance for example of a different OSS projects Develocity instance.
• Test failure analytics to better understand trends and causes around slow, failing, and flaky tests. Going to the Quarkus instance again, this will probably give you a better representation of the type of data available. You can also order it by flakiness, which is a good starting point when deciding which tests need fixing the most.

This will also enable you to (optionally) use build time optimization features, such as (remote) build caching and Predictive Test Selection. In this PR, build caching is enabled, with the local cache being disabled on CI and only allowing CI to write to the remote cache, as per the general recommendations.

I ran some tests about build speed improvements when caching is configured with the following results. Here are the best-case (no code change) savings:

Invocation	Build time without cache	Build time with cache	Speedup
install -DskipTests=true	6m 48.457s	1m 51.763s	3.67x (4m 56.694s savings
install	15m 15.283s	4m 19.865s	3.52x (10m 55.418s savings
verify -Ptck -Dscan.tag.Ptck	17m 55.511s	4m 36.046s	3.89x (13m 19.465s savings

Note: Oddly enough, I'd expect the 2nd run of the same build to already get the full benefits, but there appears to be some differences in the org.eclipse.pde.ui.tests-3.13.100-SNAPSHOT.jar file, causing cache misses for Tycho tests. The 3rd run does not have these differences, as it's being fed by the 2nd run. You can see the diff between 1st and 2nd run below - there are certain dummy classes that appear in the 2nd run. I'd be happy to help out optimize this, but I have to admit I lack some build and test context that causes this in the Eclipse PDE project.

Caching can be tested without setting up the Develocity credentials - simply run 2 or more builds (the first one will fill the local cache, next ones can use it).

More information can be read in the Eclipse announcement. Here is also a blog post abut Develocity from Eclipse Foundation, as well as a Develocity presentation, made in collaboration with Eclipse Foundation (and Gradle).

Please let me know if there are any questions about the value of Develocity or the changes in this pull request and I’d be happy to address them.

IMPORTANT

To get scans publishing on CI, a helpdesk ticket needs to be opened in order for those credentials to be created, as explained in the initiative documentation. If this PR is merged without the credentials vault setup, your CI will not work because of the missing vault! This won't be visible in the PR builds, as they use the current Jenkins setup.

[laeubi]

This currently fails the build:

17:42:05  Caused by: java.lang.UnsupportedOperationException: the build cache was not yet initialized.
17:42:05      at com.gradle.maven.cache.extension.a.f$b.a (SourceFile:317)
17:42:05      at com.gradle.maven.cache.extension.a.f$a.a (SourceFile:292)
17:42:05      at com.gradle.maven.cache.extension.c.b.a (SourceFile:85)
17:42:05      at com.gradle.maven.cache.extension.c.b.b (SourceFile:80)
17:42:05      at com.gradle.maven.cache.extension.c.b.a (SourceFile:50)
17:42:05      at com.gradle.maven.cache.extension.c.g.a (SourceFile:27)
17:42:05      at com.gradle.maven.cache.extension.c.a.a (SourceFile:46)
17:42:05      at com.gradle.maven.cache.extension.c.o.a (SourceFile:18)
17:42:05      at com.gradle.maven.cache.extension.c.a.a (SourceFile:46)
17:42:05      at com.gradle.maven.cache.extension.c.c.a (SourceFile:26)
17:42:05      at com.gradle.maven.cache.extension.c.h$1.run (SourceFile:35)
17:42:05      at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute (SourceFile:29)
17:42:05      at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute (SourceFile:26)
17:42:05      at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute (SourceFile:66)
17:42:05      at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute (SourceFile:59)
17:42:05      at org.gradle.internal.operations.DefaultBuildOperationRunner.execute (SourceFile:166)
17:42:05      at org.gradle.internal.operations.DefaultBuildOperationRunner.execute (SourceFile:59)
17:42:05      at org.gradle.internal.operations.DefaultBuildOperationRunner.run (SourceFile:47)
17:42:05      at com.gradle.maven.cache.extension.c.h.a (SourceFile:31)
17:42:05      at com.gradle.maven.cache.extension.c.m.a (SourceFile:80)
17:42:05      at com.gradle.maven.cache.extension.g.b.lambda$createProxy$0 (SourceFile:76)
17:42:05      at jdk.proxy10.$Proxy84.execute (Unknown Source)
17:42:05      at com.gradle.maven.scan.extension.internal.d.b.executeMojo (SourceFile:116)
17:42:05      at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
17:42:05      at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
17:42:05      at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
17:42:05      at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
17:42:05      at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
17:42:05      at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
17:42:05      at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
17:42:05      at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
17:42:05      at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
17:42:05      at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
17:42:05      at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
17:42:05      at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
17:42:05      at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
17:42:05      at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
17:42:05      at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
17:42:05      at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
17:42:05      at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
17:42:05      at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
17:42:05      at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
17:42:05      at java.lang.reflect.Method.invoke (Method.java:580)
17:42:05      at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:255)
17:42:05      at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:201)
17:42:05      at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:361)
17:42:05      at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:314)

due to past experience in other context I'm also a bit concerned to enable closed-source extensions (that are even obfuscated) in our builds. If anything goes wrong there no way for us to analyze the build failures.

[ribafish]

Hi @laeubi, the underlying cause is actually:

11:42:01  java.lang.IllegalStateException: Failure creating Develocity storage directory
...
11:42:01  Caused by: java.nio.file.AccessDeniedException: /home/jenkins/.m2/.develocity
...

This happens because you're using a custom agent that prevents write access to /home/jenkins - I missed that, as you're not defining it in the Jenkinsfile. How to fix this is highlighted in the Eclipse Develocity documentation about what to do in this case. I'm not sure whether that's feasable here, since you're using a prepackaged image, so I've included a workaround fix (83795338379533), similar to what you're already setting for the maven home (-Dmaven.repo.local).

Also note: To get scans publishing on CI, a helpdesk ticket needs to be opened in order for those credentials to be created, as explained in the initiative documentation. If this PR is merged without the credentials vault setup, your CI will not work because of the missing vault! This won't be visible in the PR builds, as they use the current Jenkins setup.

[laeubi]

Also note: To get scans publishing on CI, a helpdesk ticket needs to be opened in order for those credentials to be created, as explained in the initiative documentation. If this PR is merged without the credentials vault setup, your CI will not work because of the missing vault!

This looks a bit like chicken-egg problem to me... We first need to enable this to see how its working but it really feels odd to enable it without seeing it live in action(e.g. a working build).

[ribafish]

This looks a bit like chicken-egg problem to me... We first need to enable this to see how its working but it really feels odd to enable it without seeing it live in action(e.g. a working build).

You can create local builds without any additional configuration - (local) caching will work, but build scans will not be published until you authenticate with Develocity, which you can do by running mvn develocity:provision-access-key.

To verify the CI setup, you will need to get the secrets set up by the EF infra team, but you should be able to test it out afterwards, before merging the PR, by opening the PR yourself or opening it without a fork, depending on your CI setup. The issue here is that, for security reasons, Jenkinsfile changes are not accepted for external contributors' forks. Other projects in the Eclipse Foundation have done the same to verify the CI config changes.

[HannesWell]

Thank you for your offer, but with the current unstable state of the infrastructure, already existing undesired caches and migration tasks we have due to infrastructure changes, I'm currently not in favor of adding another layer of caching and a potential (for us) opaque layers to the build. The reductions in build times are nice to have, but it's not clear to me how e.g. how the set of skipped tests is selected and as things are often convoluted in PDE, thus leading to unexpected test-failures. Therefore I see some risk of false positive test results with this (i.e. false absent of failures). But in general a stable build pipeline is more important to me than a fast one. Unstable builds often cause so much more work and annoyance.

I'm sorry, but I don't have the capacity at the moment to work on this and supervise the build when this is applied and I'm not aware of another committer, who is willing to do it instead. If somebody steps up to take care of this entirely (not only the implementation, but also handling all consequences that might arise), we can think about this again, but until then I'm afraid this probably won't fly, sorry. Maybe we'll reevaluate it when more Eclipse top-level projects are using it successfully.

[ribafish]

@HannesWell, Thank you for the consideration, I completely understand.

To give a brief answer, the Develocity build cache works on a per-goal level, which means that goal outputs are taken from the cache if the cache key is present in the (local or remote) build cache. The cache key is calculated from hashing the inputs, which are custom to every goal - in general, those might include things like source files, dependencies, different properties and the env that's running the goal (for example Java version). If there are other things that affect a goal output, those can also be added to the tracked input, custom for a specific project. If you're talking about Predictive Test Selection - that's out of scope for this PR, but I'd be happy to discuss more if you're interested (now or in the future).

A lot of value Develocity brings is also its monitoring and debugging capabilities, especially for large and complex projects. Here is a list of builds I did when testing out support with the PDE project (using one of our Develocity intances, which is functionally the same as the EF one). Checking one of the failed builds you can see for example it's very easy to see the failure, check what dependencies and their versions were used in this build, how long it took, how long each of the goals took, etc. You can also see the trends, list of failures, and how often they occur, tests history, etc (this data becomes more interesting the more builds you have). For more info you can also check out the presentation I did together with Eclipse Foundation, the Develocity demo starts around 10m mark.

[merks]

Keeping the builds green is enough challenge currently. Sorry.