Update, 26-02: Brian Demers from Sonatype pointed out in the comments that Maven 2.0.10 and later are forwards-compatible with changes in the metadata format. If your Maven 2 version is one of the recommended versions on the download page, you will not have this problem.

Two days, in fact. Yesterday evening, after my colleagues went home, I brought down our Nexus instance to upgrade it to 1.9 without interrupting their work. The download page for Nexus 1.9 contains the following instruction:

Sonatype has changed how the lucene indexes are stored on disk, it is required that users reindex all repositories in their nexus server to start benefitting from the changes (and for search to work properly).

Inconspicuous enough. Furthermore, clicking through from the change overview to the full change log reveals:

[NEXUS-3849] - Add full support for the new maven 3 snapshot metadata

What it doesn't reveal is that the rebuild metadata command in the repository administration screen, which would appear to be proper housekeeping at a time when you're reindexing the repositories, now generates Maven 3 style metadata and inadvertently breaks compatibility with Maven 2 (update: older versions). This is where the fun begins.

The fun

Having upgraded Nexus to 1.9 and having rebuilt metadata en reindexed repositories, my Jenkins builds started failing:

[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Error retrieving previous build number for artifact 'myGroup:myArtifact:pom': Cannot read metadata from \
    '/home/maven/maven-repository/myGroup/myArtifact/1.8.0-SNAPSHOT/maven-metadata-nexus-snapshots.xml': \
    expected START_TAG or END_TAG not TEXT (position: TEXT seen ...<extension>pom</... @14:25) 

Wouldn't you know it, Maven 2 builds cannot handle the new metadata format.

(I feel it's worth mentioning that when Jenkins releases a new version, everything Just Works (tm). They have their act together in a fantastic way.)

Attempted fix: downgrade Nexus

My previous Nexus version is still on the file system. To downgrade, all I had to do is stop Nexus, revert symlink, start Nexus.

jvm 1    | 2011-02-24 18:30:28 ERROR [er_start_runner] - o.s.n.DefaultNexus            - Could not start Nexus, user configuration exception!
jvm 1    | org.sonatype.configuration.upgrade.UnsupportedConfigurationVersionException: Unsupported configuration file in /home/maven/nexus-oss-webapp- with version: 1.4.4. Cannot upgrade.

Oh crap. Well at least the error message is perfectly clear. Even better, there turns out to be a nexus.xml.bak in the conf directory. For the first time that evening, I mutter something vaguely grateful about Sonatype. Now, let me just copy the entire conf directory to a safe location, restore that backup config file and try again.

jvm 1    | 2011-02-24 18:39:21 INFO  [er_start_runner] - o.s.n.c.a.DefaultNe~          - Loading Nexus Configuration...
jvm 1    | 2011-02-24 18:39:21 INFO  [er_start_runner] - o.s.n.c.s.StaticCon~          - Configuration loaded succesfully.

...so far so good...

jvm 1    | 2011-02-24 18:39:22 ERROR [er_start_runner] - o.s.g.b.r.NamedClass          - Error injecting: org.sonatype.nexus.DefaultNexus
jvm 1    | com.google.inject.ProvisionException: Guice provision errors:
jvm 1    |
jvm 1    | 1) Error starting: class org.sonatype.nexus.DefaultNexus
jvm 1    |   while locating org.sonatype.nexus.DefaultNexus


jvm 1    | Caused by: java.lang.NullPointerException
jvm 1    |      at org.sonatype.security.configuration.DefaultSecurityConfigurationManager.getRealms(DefaultSecurityConfigurationManager.java:104)
jvm 1    |      at org.sonatype.security.DefaultSecuritySystem.getRealmsFromConfigSource(DefaultSecuritySystem.java:202)
jvm 1    |      at org.sonatype.security.DefaultSecuritySystem.start(DefaultSecuritySystem.java:846)
jvm 1    |      at org.sonatype.nexus.DefaultNexus.startService(DefaultNexus.java:682)
jvm 1    |      at org.sonatype.nexus.DefaultNexus.start(DefaultNexus.java:647)

... oh bugger.

Restore the configuration for Nexus 1.9 and start that. I'm now in a stable, but useless situation where Nexus 1.9 is running and my Maven 2 builds are failing. It's 19:20, I'll return tomorrow.

Attempted fix: -Dmaven.metadata.legacy=true

Researching this issue, I came across NEXUS-3806, which mentions a maven.metadata.legacy system property. This looks hopeful, assuming that Nexus and Maven 2/3 share parts of their code base. I brought down Nexus, added -Dmaven.metadata.legacy=true to wrapper.conf. Restart nexus, rebuild metadata for my snapshots repository. Fingers crossed...

...but no luck, the metadata is still in Maven 3 format. It is now noon the next day and my Maven builds are still broken.

As I'm writing this, I realize that I didn't try deleting the existing metadata files before running the Rebuild Metadata command. After going through all this, I didn't feel like finding out, so if you're in a position to verify this, please do so and leave a note in the comments.

Attempted fix: making Maven 2 understand the Maven 3 metadata format

I think MNG-4452 prompted the change in metadata XML format, applied to maven-artifact version 3.0. I know enough about Maven's internals from earlier troubleshooting to I quickly abandon any hope of backporting this to Maven 2.

Back to getting Nexus to work

I decided to pick up where I left off last night. I stopped Nexus, renamed my sonatype-work directory and restarted Nexus, emulating a clean install. This gave me a fresh, newly-generated sonatype-work directory to compare to my existing one. I eventually traced the failed startup to the security-configuration.xml file. Here's the original one:

<?xml version="1.0"?>
	<anonymousUsername><!-- hidden --></anonymousUsername>
	<anonymousPassword><!-- hidden --></anonymousPassword>

The <securityManager> element and the NexusLdapAuthenticationRealm were absent in the fresh config. I restored my original sonatype-work directory and removed these two lines. Nexus now starts up successfully! It is now three in the afternoon, I needed some good news by now.

I went into the web interface and re-enabled LDAP on the Administration/Server page. This restored the NexusLdapAuthenticationRealm in my XML file, meaning that the <securityManager> element is what prevented my downgraded Nexus from starting. The very first Nexus version I installed on our build server was 1.7.1. The securityManager element must be a relic of that version, although this doesn't explain why this didn't previously prevent Nexus from starting up.

Did that fix my Maven builds?

Not at first sight. The Rebuild Metadata task in Nexus skips any existing metadata files. Once I figured that out I nuked these on the filesytem with a quick find /home/maven/sonatype-work/nexus/storage/snapshots -name "maven-metadata.xm*" -delete. After another Rebuild Metadata, I finally have my repository metadata in Maven 2 format.

Did that fix my Maven builds? Nope, because the Maven 3-style metadata files were still in my local repository. This is the home stretch; removing them from the local repository with another quick find command got my Maven 2 builds working again.

Getting some satisfaction

rm -rf nexus-oss-webapp-1.9

Suck on that, you empty-headed animal food trough wiper! I fart in your general direction*!

Lessons learned

With Maven 3 around, installing stuff from Sonatype when you're still on (update: an outdated) Maven 2 version is a dangerous proposition. Nexus 1.9 has proven to be incompatible with Maven 2, unintentionally I'm sure. Not upgrading Nexus is easier than downgrading it. I refuse to run a DTAP chain for my development support tooling, so I'll just refrain from updating Nexus altogether from here on.

I'll use Jenkins to satisfy my keeping-stuff-up-to-date compulsions. 😉