Recently, I was preparing a connection checker for Deployit's powerful remote execution framework Overthere. To make the checker, as compact as possible, I put together a jar-with-deps1 for distribution.
Tests and trial runs from the IDE worked, so I was expected the dry-run of the distribution to be a quick formality. Instead: boom!
Turns out that one of the libraries used by Overthere, TrueZIP - or indeed any code that utilizes Java's SPI mechanism2 - doesn't play well with the jar-with-deps idea.

Seeing Double

TrueZIP uses a de.schlichtherle.truezip.fs.spi.FsDriverService provider configuration file in META-INF/services to register drivers for various archive formats, and the checker was failing because one of the required drivers wasn't being loaded, even though the code was correctly included in the jar-with-deps.

The duplicate option of Ant's Jar task3 and Gradle issue 1050, which talks about merging files during archive creation, hint at what the problem is: once packaged up in a single jar-with-deps archive, only one of TrueZIP's provider configuration files was being found.

Maven, Gradle and Java

Once I examined the jar-with-deps constructed by Maven4, it was pretty clear why: the JAR only contains one of the configuration files - looks like the first one encountered in my case, but that may be non-deterministic. Presumably, this happens because Maven uses a temporary directory to prepare the archive contents, which of course can't hold multiple files of the same name.

With Gradle5, things are a bit more interesting because the archive does indeed contain all the configuration files - the ZIP format supports duplicate entries6. However, ClassLoader.getResources doesn't play along, so again only one entry is found.

There Can Be Only One

Basically, it seems that merging the affected files is the only feasible way around this. Here's a possible Gradle snippet7:

task jarWithDeps(type: Jar, dependsOn: classes) {
	mergeDir = "${buildDir}/merge"
	// might need a lazy var in a multi-module project where deps are inherited
	runtimeDeps = configurations.runtime.collect { zipTree(it) }

	doFirst {
		new File(mergeDir).delete()
		mergeFiles(mergeDir, runtimeDeps, file-to-merge)
		// ... possibly other files too
	}

	// this project's classes and all deps
	from sourceSets*.classesDir
	from(runtimeDeps) {
		exclude file-to-merge
	}
	from mergeDir
}

private def mergeFiles(targetDir, fileTrees, relativePath) {
  // prepare the merge
  mergedFile = new File("${targetDir}/${relativePath}")
  new File(mergedFile.parent).mkdirs()

  fileTrees*.matching({ include "**/${relativePath}" })*.each {
    mergedFile << it.bytes
  }
}

Unfortunately, I haven't been able to find a similar option for the Maven Archiver, but I guess there must be something out there in the Maven ecosystem πŸ˜‰

Edited 21/07/2011 to add: The sample code now includes possible solutions suggested in comments. Run gradle clean jarWithDeps2, mvn clean package -Pwith-shade-plugin or mvn clean package -Pwith-keystone-plugin8 to try them out.
Edited 25/07/2011 to add: You can now also run mvn clean package -Pwith-services-handler to see how the Assembly plugin's <containerDescriptorHandlers> can take care of this too.

Footnotes

  1. A.k.a. "farJar". For an equivalent Gradle task definition, see here.
  2. See MultiSPI for an approach that provides more flexibility and "modern alternatives" for service providers.
  3. Note that the jar utility itself doesn't provide a similar option.
  4. run mvn clean package
  5. run gradle clean jarWithDeps
  6. although the ZipOutputStream evidently wasn't too happy with them and, judging by the 1.6.0_20 implementation, will still throw a ZipException
  7. Would be interesting to try to do the merging at JAR construction time, perhaps by keeping "merge-so-far(s)" in the mergeDir and using eachFile to replace the unmerged files and update the "merge-so-far(s)" at the same time.
  8. required two fixes to the code to get it work correctly on my system