How Sonatype Nexus 1.9 ruined my day

Barend Garvelink

Update, 26-02: Brian Demers from Sonatype pointed out in the comments that Maven 2.0.10 and later are forwards-compatible with changes in the metadata format. If your Maven 2 version is one of the recommended versions on the download page, you will not have this problem.

Two days, in fact. Yesterday evening, after my colleagues went home, I brought down our Nexus 1.8.0.1 instance to upgrade it to 1.9 without interrupting their work. The download page for Nexus 1.9 contains the following instruction:

Sonatype has changed how the lucene indexes are stored on disk, it is required that users reindex all repositories in their nexus server to start benefitting from the changes (and for search to work properly).

Inconspicuous enough. Furthermore, clicking through from the change overview to the full change log reveals:

[NEXUS-3849] - Add full support for the new maven 3 snapshot metadata

What it doesn't reveal is that the rebuild metadata command in the repository administration screen, which would appear to be proper housekeeping at a time when you're reindexing the repositories, now generates Maven 3 style metadata and inadvertently breaks compatibility with Maven 2 (update: older versions). This is where the fun begins.

The fun

Having upgraded Nexus to 1.9 and having rebuilt metadata en reindexed repositories, my Jenkins builds started failing:

[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Error retrieving previous build number for artifact 'myGroup:myArtifact:pom': Cannot read metadata from \
    '/home/maven/maven-repository/myGroup/myArtifact/1.8.0-SNAPSHOT/maven-metadata-nexus-snapshots.xml': \
    expected START_TAG or END_TAG not TEXT (position: TEXT seen ...<extension>pom</... @14:25) 

Wouldn't you know it, Maven 2 builds cannot handle the new metadata format.

(I feel it's worth mentioning that when Jenkins releases a new version, everything Just Works (tm). They have their act together in a fantastic way.)

Attempted fix: downgrade Nexus

My previous Nexus version is still on the file system. To downgrade, all I had to do is stop Nexus, revert symlink, start Nexus.

jvm 1    | 2011-02-24 18:30:28 ERROR [er_start_runner] - o.s.n.DefaultNexus            - Could not start Nexus, user configuration exception!
jvm 1    | org.sonatype.configuration.upgrade.UnsupportedConfigurationVersionException: Unsupported configuration file in /home/maven/nexus-oss-webapp-1.8.0.1/./../sonatype-work/nexus/conf/nexus.xml with version: 1.4.4. Cannot upgrade.

Oh crap. Well at least the error message is perfectly clear. Even better, there turns out to be a nexus.xml.bak in the conf directory. For the first time that evening, I mutter something vaguely grateful about Sonatype. Now, let me just copy the entire conf directory to a safe location, restore that backup config file and try again.

jvm 1    | 2011-02-24 18:39:21 INFO  [er_start_runner] - o.s.n.c.a.DefaultNe~          - Loading Nexus Configuration...
jvm 1    | 2011-02-24 18:39:21 INFO  [er_start_runner] - o.s.n.c.s.StaticCon~          - Configuration loaded succesfully.

...so far so good...

jvm 1    | 2011-02-24 18:39:22 ERROR [er_start_runner] - o.s.g.b.r.NamedClass          - Error injecting: org.sonatype.nexus.DefaultNexus
jvm 1    | com.google.inject.ProvisionException: Guice provision errors:
jvm 1    |
jvm 1    | 1) Error starting: class org.sonatype.nexus.DefaultNexus
jvm 1    |   while locating org.sonatype.nexus.DefaultNexus

(truncated)

jvm 1    | Caused by: java.lang.NullPointerException
jvm 1    |      at org.sonatype.security.configuration.DefaultSecurityConfigurationManager.getRealms(DefaultSecurityConfigurationManager.java:104)
jvm 1    |      at org.sonatype.security.DefaultSecuritySystem.getRealmsFromConfigSource(DefaultSecuritySystem.java:202)
jvm 1    |      at org.sonatype.security.DefaultSecuritySystem.start(DefaultSecuritySystem.java:846)
jvm 1    |      at org.sonatype.nexus.DefaultNexus.startService(DefaultNexus.java:682)
jvm 1    |      at org.sonatype.nexus.DefaultNexus.start(DefaultNexus.java:647)

... oh bugger.

Restore the configuration for Nexus 1.9 and start that. I'm now in a stable, but useless situation where Nexus 1.9 is running and my Maven 2 builds are failing. It's 19:20, I'll return tomorrow.

Attempted fix: -Dmaven.metadata.legacy=true

Researching this issue, I came across NEXUS-3806, which mentions a maven.metadata.legacy system property. This looks hopeful, assuming that Nexus and Maven 2/3 share parts of their code base. I brought down Nexus, added -Dmaven.metadata.legacy=true to wrapper.conf. Restart nexus, rebuild metadata for my snapshots repository. Fingers crossed...

...but no luck, the metadata is still in Maven 3 format. It is now noon the next day and my Maven builds are still broken.

As I'm writing this, I realize that I didn't try deleting the existing metadata files before running the Rebuild Metadata command. After going through all this, I didn't feel like finding out, so if you're in a position to verify this, please do so and leave a note in the comments.

Attempted fix: making Maven 2 understand the Maven 3 metadata format

I think MNG-4452 prompted the change in metadata XML format, applied to maven-artifact version 3.0. I know enough about Maven's internals from earlier troubleshooting to I quickly abandon any hope of backporting this to Maven 2.

Back to getting Nexus 1.8.0.1 to work

I decided to pick up where I left off last night. I stopped Nexus, renamed my sonatype-work directory and restarted Nexus, emulating a clean install. This gave me a fresh, newly-generated sonatype-work directory to compare to my existing one. I eventually traced the failed startup to the security-configuration.xml file. Here's the original one:

<?xml version="1.0"?>
<security-configuration>
	<version>2.0.3</version>
	<enabled>true</enabled>
	<anonymousAccessEnabled>true</anonymousAccessEnabled>
	<anonymousUsername><!-- hidden --></anonymousUsername>
	<anonymousPassword><!-- hidden --></anonymousPassword>
	<realms>
		<realm>NexusLdapAuthenticationRealm</realm>
		<realm>XmlAuthenticatingRealm</realm>
		<realm>XmlAuthorizingRealm</realm>
	</realms>
	<securityManager>web</securityManager>
</security-configuration>

The <securityManager> element and the NexusLdapAuthenticationRealm were absent in the fresh config. I restored my original sonatype-work directory and removed these two lines. Nexus 1.8.0.1 now starts up successfully! It is now three in the afternoon, I needed some good news by now.

I went into the web interface and re-enabled LDAP on the Administration/Server page. This restored the NexusLdapAuthenticationRealm in my XML file, meaning that the <securityManager> element is what prevented my downgraded Nexus from starting. The very first Nexus version I installed on our build server was 1.7.1. The securityManager element must be a relic of that version, although this doesn't explain why this didn't previously prevent Nexus from starting up.

Did that fix my Maven builds?

Not at first sight. The Rebuild Metadata task in Nexus 1.8.0.1 skips any existing metadata files. Once I figured that out I nuked these on the filesytem with a quick find /home/maven/sonatype-work/nexus/storage/snapshots -name "maven-metadata.xm*" -delete. After another Rebuild Metadata, I finally have my repository metadata in Maven 2 format.

Did that fix my Maven builds? Nope, because the Maven 3-style metadata files were still in my local repository. This is the home stretch; removing them from the local repository with another quick find command got my Maven 2 builds working again.

Getting some satisfaction

rm -rf nexus-oss-webapp-1.9

Suck on that, you empty-headed animal food trough wiper! I fart in your general direction*!

Lessons learned

With Maven 3 around, installing stuff from Sonatype when you're still on (update: an outdated) Maven 2 version is a dangerous proposition. Nexus 1.9 has proven to be incompatible with Maven 2, unintentionally I'm sure. Not upgrading Nexus is easier than downgrading it. I refuse to run a DTAP chain for my development support tooling, so I'll just refrain from updating Nexus altogether from here on.

I'll use Jenkins to satisfy my keeping-stuff-up-to-date compulsions. 😉

Comments (14)

  1. [...] This post was mentioned on Twitter by Xebia Nederland, Jan Plas. Jan Plas said: RT @Xebia: New blog post: : How Sonatype Nexus 1.9 ruined my day http://blog.xebia.com/2011/02/25/how-nexus-1-9-ruined-my-day/ [...]

  2. Brian Demers - Reply

    February 25, 2011 at 8:03 pm

    You must be using Maven 2.0.9 or less ?
    You should upgrade to 2.0.11 (or 2.2.1)
    See: https://cwiki.apache.org/MAVEN/maven-3x-compatibility-notes.html

    You may run into problem with other repositories as well using maven 2.0.9.

    Take a look at: http://maven.apache.org/download.html to see which version of maven version would work best for you.

  3. Edoardo - Reply

    February 25, 2011 at 8:46 pm

    Hi Barend,

    I kind of understand your pain... Maven is a very good idea - in principle, but it takes a very hefty amount of effort to tame it's numerous bugs and a certain recklessness in its dev community 😉

    Oh well: backups, backups... backups. Much cheaper than hoisting a DTAP chain and if you happen to run it on a VM, they're really just a shapshot away 😉

    Glad you fixed it though...

  4. Fred Simon - Reply

    February 26, 2011 at 12:35 am

    I think it's time you use Artifactory!
    Our Jenkins plugins will remove Maven from the deployment equation and Artifactory will calculate and provide the correct maven-metadata.xml to any of your build tool: Maven 2/3, Gradle, Ivy.
    Keep your freedom drop Nexus...

  5. Alexis Kinsella - Reply

    February 26, 2011 at 2:00 am

    Hi Barend,

    I migrated some nexus repository from version 1.8.0.1 to 1.9 some days ago. I didn't noticed a such issue. Nexus repository still works with Nexus in version 1.9 and Maven 2.

    You got the repository broken because of the rebuild of metadata, or it was simply broken after restart ?

    Do I risk some similar crash in case I run a such command (Rebuild of metadata) on nexus repository?

    Thank you for that blog post, I guess it will help many people to save lot of time 😉

    Alexis

  6. Arnaud Héritier - Reply

    February 26, 2011 at 5:20 am

    As Brian said, it has probably more to have with the version of Maven 2 used than with the repo manager.
    Fred you could test it with artifactory but I think you won't serve different metadata depending of the maven version used ?
    The problem was that in maven <= 2.0.9, Maven failed if metadata were wrong (or had unknown elements if I remember well). Nowadays it ignores unknow elements (to allow us to add new things in new versions) and does like if metadata are here if they are unreadable.
    About upgrade process for such soft, I agree that having a validation environment is annoying to manage because you need to have all the stack (repo, ci ...) and to reconfigure everything all together. I agree with Edoardo, before each upgrade I create a backup of all data. And if possible I do that when teams are off.
    I'm using 1.9-M1 for months with Maven 3.x and 2.2.1 jobs without having encountered this issue. This WE I'm trying to upgrade to 1.9.0.1.

    Cheers

    Arnaud

  7. Tony - Reply

    February 26, 2011 at 9:45 pm

    Maven 2.0.9 was released 3 years ago. Wow. Long past time to upgrade.

  8. Leon - Reply

    February 27, 2011 at 10:40 am

    Hi Barend,

    It does seem to be in the very best traditions of Maven and maven plugin updates. Innocent-looking minor version upgrades breaking the build are not really unheard of 🙁

    Next time, I hope you take the path of wisdom and backup the entire intra- and internet before you touch sonatype stuff. Or have a time machine handy (aren't black holes rumored to provide that capability?)

    I am getting curious about the amount of work required to upgrade maven to 2.2x. I have the funny feeling that that would require manually porting all the pom files and recompile and test everything again. Do you think such a step would be worthwile? Or would switching to artifactory be a better option, potentially blacklisting maven plugin updates?

  9. Yoav Landman - Reply

    February 28, 2011 at 3:49 pm

    Artifactory will return variable metadata depending on the maven client version from the next release (https://issues.jfrog.org/jira/browse/RTFACT-3794). Until then, we keep generating on the Artifactory side metadata that is fully compliant with the older format so we don't ruin your day.
    For Jenkins, you are also welcome to use the Jenkins integration, to make sure you don't end up with partially deployed modules if your build breaks in the middle and to have traceable artifacts and dependencies in your repo http://wiki.jenkins-ci.org/display/JENKINS/Artifactory+Plugin

  10. Leon - Reply

    April 16, 2011 at 2:59 pm

    I was just googling for maven repo ("liquid LAF swing maven repo" to be exact) and this is my first hit. Well done 🙂

  11. witek - Reply

    October 8, 2012 at 1:44 pm

    just saved about 2 hours on googling / trying / fixing / failing - thanks man!

  12. Zardoz - Reply

    April 15, 2015 at 10:06 am

    I verified that using "-Dmaven.metadata.legacy=true" with Nexus 2.11 and maven 2.2.1 works fine. Also, I deleted the old metadata to enforce nexus to rebuild it.

    Thanks for the post. It avoid me a lot of headhaches.

Add a Comment