jar-with-deps don't like META-INF/services

Recently, I was preparing a connection checker for Deployit's powerful remote execution framework Overthere. To make the checker, as compact as possible, I put together a jar-with-deps1 for distribution.
Tests and trial runs from the IDE worked, so I was expected the dry-run of the distribution to be a quick formality. Instead: boom!
Turns out that one of the libraries used by Overthere, TrueZIP - or indeed any code that utilizes Java's SPI mechanism2 - doesn't play well with the jar-with-deps idea.

Seeing Double

TrueZIP uses a de.schlichtherle.truezip.fs.spi.FsDriverService provider configuration file in META-INF/services to register drivers for various archive formats, and the checker was failing because one of the required drivers wasn't being loaded, even though the code was correctly included in the jar-with-deps.

The duplicate option of Ant's Jar task3 and Gradle issue 1050, which talks about merging files during archive creation, hint at what the problem is: once packaged up in a single jar-with-deps archive, only one of TrueZIP's provider configuration files was being found.

Maven, Gradle and Java

Once I examined the jar-with-deps constructed by Maven4, it was pretty clear why: the JAR only contains one of the configuration files - looks like the first one encountered in my case, but that may be non-deterministic. Presumably, this happens because Maven uses a temporary directory to prepare the archive contents, which of course can't hold multiple files of the same name.

With Gradle5, things are a bit more interesting because the archive does indeed contain all the configuration files - the ZIP format supports duplicate entries6. However, ClassLoader.getResources doesn't play along, so again only one entry is found.

There Can Be Only One

Basically, it seems that merging the affected files is the only feasible way around this. Here's a possible Gradle snippet7:

task jarWithDeps(type: Jar, dependsOn: classes) {
	mergeDir = "${buildDir}/merge"
	// might need a lazy var in a multi-module project where deps are inherited
	runtimeDeps = configurations.runtime.collect { zipTree(it) }

	doFirst {
		new File(mergeDir).delete()
		mergeFiles(mergeDir, runtimeDeps, file-to-merge)
		// ... possibly other files too
	}

	// this project's classes and all deps
	from sourceSets*.classesDir
	from(runtimeDeps) {
		exclude file-to-merge
	}
	from mergeDir
}

private def mergeFiles(targetDir, fileTrees, relativePath) {
  // prepare the merge
  mergedFile = new File("${targetDir}/${relativePath}")
  new File(mergedFile.parent).mkdirs()

  fileTrees*.matching({ include "**/${relativePath}" })*.each {
    mergedFile << it.bytes
  }
}

Unfortunately, I haven't been able to find a similar option for the Maven Archiver, but I guess there must be something out there in the Maven ecosystem 😉

Edited 21/07/2011 to add: The sample code now includes possible solutions suggested in comments. Run gradle clean jarWithDeps2, mvn clean package -Pwith-shade-plugin or mvn clean package -Pwith-keystone-plugin8 to try them out.
Edited 25/07/2011 to add: You can now also run mvn clean package -Pwith-services-handler to see how the Assembly plugin's <containerDescriptorHandlers> can take care of this too.

Footnotes

  1. A.k.a. "farJar". For an equivalent Gradle task definition, see here.
  2. See MultiSPI for an approach that provides more flexibility and "modern alternatives" for service providers.
  3. Note that the jar utility itself doesn't provide a similar option.
  4. run mvn clean package
  5. run gradle clean jarWithDeps
  6. although the ZipOutputStream evidently wasn't too happy with them and, judging by the 1.6.0_20 implementation, will still throw a ZipException
  7. Would be interesting to try to do the merging at JAR construction time, perhaps by keeping "merge-so-far(s)" in the mergeDir and using eachFile to replace the unmerged files and update the "merge-so-far(s)" at the same time.
  8. required two fixes to the code to get it work correctly on my system

Comments (9)

  1. Jerome - Reply

    July 20, 2011 at 3:14 pm

    Hi, i've developped my own maven plugin for doing that.
    Keystone projet package for you all your dep, and launch your main class after class loader initialization.
    Here a link: http://intelligents-ia.com/index.php/post/2011/04/29/Keystone-1.1-est-sortie-%21 (sorry for french)
    and code is on google code: https://code.google.com/p/blog-intelligents-ia/
    I you need, i can translate it in english, i will be happy to known that my little code is used by other people 🙂

  2. Andrew Phillips - Reply

    July 20, 2011 at 3:25 pm

    @Jerome: Merci bien! 😉

  3. Larry - Reply

    July 20, 2011 at 4:16 pm

    The Maven Shade Plugin will help...

    http://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html

    I use it to concatenate META-INF/spring.handlers and META-INF/spring.schemas.

  4. Andrew Phillips - Reply

    July 21, 2011 at 12:39 am

    @Larry: Thanks, now included in the sample code (run using mvn clean package -Pwith-shade-plugin)

    @Jerome: The sample code also includes the keystone plugin, but I had to fix two issues to get it to work correctly (related to loading properties and including dependencies in the boot JAR). The diff is in Pastebin.
    Also, the example in your blog posts uses <main-class> as a configuration attribute, but that doesn't work - <mainClass> does (the "expression" attribute defines the command-line parameter Maven uses, not the name of the <configuration> tag).

  5. Christian Schlichtherle - Reply

    July 21, 2011 at 3:51 pm

    While a JAR file can physically contain multiple archive entries with an equal name, there is only one central directory and it can contain only one archive entry for any name. Hence you need to concatenate the contents of the service provider files as you figured. You can use the maven-assembly-plugin or maven-shade-plugin to achieve this. I use the maven-assembly-plugin to build the jar-with-dependencies classifier for TrueZIP.

  6. Andrew Phillips - Reply

    July 22, 2011 at 12:41 am

    @Christian: Was it the mention of TrueZIP that triggered your visit, I wonder? 😉

    Yes, merging the contents is clearly the way to go about this...or abandon a flat jar-with-deps and use a classloading strategy that can cope with bundled JARs, as the Keystone plugin tries to.
    As far as I can figure out, there is no easy way to get the maven-assembly-plugin to support this merging, though - if you run mvn clean package on the sample project, the resulting JAR will contain one file which is the file from one of the dependent JARs - no merging.

    Some more searching has thrown up the practically undocumented MetaInfServicesHandler, which is one of a number of built-in containerDescriptorHandlers and looks like it might do the trick.

    It's not part of the default jar-with-dependencies descriptorRef, though.

  7. Andrew Phillips - Reply

    July 25, 2011 at 9:26 pm

    The MetaInfServicesHandler solution actually works, which is great. Note that you will need a custom assembly descriptor, as jar-with-dependencies does not define the required handlers.

    You can try it by running mvn clean package -Pwith-services-handler in the sample project.

  8. Jim Shingler - Reply

    January 4, 2012 at 7:27 pm

    runtimeDeps needs to be a closure or it will be empty

    see: http://issues.gradle.org/browse/GRADLE-1361

  9. Andrew Phillips - Reply

    January 4, 2012 at 7:45 pm

    @Jim: You're right, in a multi-project build (as is the case in the issue you linked to) this will indeed be required.

    That's what the might need a lazy var in a multi-module project where deps are inherited comment in the code is trying to indicate - s/lazy var/closure/ in this case 😉

    Thanks for making that point again!

Add a Comment