How Sonatype Nexus 1.9 ruined my day
Update, 26-02: Brian Demers from Sonatype pointed out in the comments that Maven 2.0.10 and later are forwards-compatible with changes in the metadata format. If your Maven 2 version is one of the recommended versions on the download page, you will not have this problem.
Two days, in fact. Yesterday evening, after my colleagues went home, I brought down our Nexus 184.108.40.206 instance to upgrade it to 1.9 without interrupting their work. The download page for Nexus 1.9 contains the following instruction:
Sonatype has changed how the lucene indexes are stored on disk, it is required that users reindex all repositories in their nexus server to start benefitting from the changes (and for search to work properly).
[NEXUS-3849] - Add full support for the new maven 3 snapshot metadata
What it doesn't reveal is that the rebuild metadata command in the repository administration screen, which would appear to be proper housekeeping at a time when you're reindexing the repositories, now generates Maven 3 style metadata and inadvertently breaks compatibility with Maven 2 (update: older versions). This is where the fun begins.
Having upgraded Nexus to 1.9 and having rebuilt metadata en reindexed repositories, my Jenkins builds started failing:
[INFO] ------------------------------------------------------------------------ [ERROR] BUILD ERROR [INFO] ------------------------------------------------------------------------ [INFO] Error retrieving previous build number for artifact 'myGroup:myArtifact:pom': Cannot read metadata from \ '/home/maven/maven-repository/myGroup/myArtifact/1.8.0-SNAPSHOT/maven-metadata-nexus-snapshots.xml': \ expected START_TAG or END_TAG not TEXT (position: TEXT seen ...<extension>pom</... @14:25)
Wouldn't you know it, Maven 2 builds cannot handle the new metadata format.
(I feel it's worth mentioning that when Jenkins releases a new version, everything Just Works (tm). They have their act together in a fantastic way.)
Attempted fix: downgrade Nexus
My previous Nexus version is still on the file system. To downgrade, all I had to do is stop Nexus, revert symlink, start Nexus.
jvm 1 | 2011-02-24 18:30:28 ERROR [er_start_runner] - o.s.n.DefaultNexus - Could not start Nexus, user configuration exception! jvm 1 | org.sonatype.configuration.upgrade.UnsupportedConfigurationVersionException: Unsupported configuration file in /home/maven/nexus-oss-webapp-220.127.116.11/./../sonatype-work/nexus/conf/nexus.xml with version: 1.4.4. Cannot upgrade.
Oh crap. Well at least the error message is perfectly clear. Even better, there turns out to be a nexus.xml.bak in the conf directory. For the first time that evening, I mutter something vaguely grateful about Sonatype. Now, let me just copy the entire conf directory to a safe location, restore that backup config file and try again.
jvm 1 | 2011-02-24 18:39:21 INFO [er_start_runner] - o.s.n.c.a.DefaultNe~ - Loading Nexus Configuration... jvm 1 | 2011-02-24 18:39:21 INFO [er_start_runner] - o.s.n.c.s.StaticCon~ - Configuration loaded succesfully.
...so far so good...
jvm 1 | 2011-02-24 18:39:22 ERROR [er_start_runner] - o.s.g.b.r.NamedClass - Error injecting: org.sonatype.nexus.DefaultNexus jvm 1 | com.google.inject.ProvisionException: Guice provision errors: jvm 1 | jvm 1 | 1) Error starting: class org.sonatype.nexus.DefaultNexus jvm 1 | while locating org.sonatype.nexus.DefaultNexus (truncated) jvm 1 | Caused by: java.lang.NullPointerException jvm 1 | at org.sonatype.security.configuration.DefaultSecurityConfigurationManager.getRealms(DefaultSecurityConfigurationManager.java:104) jvm 1 | at org.sonatype.security.DefaultSecuritySystem.getRealmsFromConfigSource(DefaultSecuritySystem.java:202) jvm 1 | at org.sonatype.security.DefaultSecuritySystem.start(DefaultSecuritySystem.java:846) jvm 1 | at org.sonatype.nexus.DefaultNexus.startService(DefaultNexus.java:682) jvm 1 | at org.sonatype.nexus.DefaultNexus.start(DefaultNexus.java:647)
... oh bugger.
Restore the configuration for Nexus 1.9 and start that. I'm now in a stable, but useless situation where Nexus 1.9 is running and my Maven 2 builds are failing. It's 19:20, I'll return tomorrow.
Attempted fix: -Dmaven.metadata.legacy=true
Researching this issue, I came across NEXUS-3806, which mentions a maven.metadata.legacy system property. This looks hopeful, assuming that Nexus and Maven 2/3 share parts of their code base. I brought down Nexus, added -Dmaven.metadata.legacy=true to wrapper.conf. Restart nexus, rebuild metadata for my snapshots repository. Fingers crossed...
...but no luck, the metadata is still in Maven 3 format. It is now noon the next day and my Maven builds are still broken.
As I'm writing this, I realize that I didn't try deleting the existing metadata files before running the Rebuild Metadata command. After going through all this, I didn't feel like finding out, so if you're in a position to verify this, please do so and leave a note in the comments.
Attempted fix: making Maven 2 understand the Maven 3 metadata format
I think MNG-4452 prompted the change in metadata XML format, applied to maven-artifact version 3.0. I know enough about Maven's internals from earlier troubleshooting to I quickly abandon any hope of backporting this to Maven 2.
Back to getting Nexus 18.104.22.168 to work
I decided to pick up where I left off last night. I stopped Nexus, renamed my sonatype-work directory and restarted Nexus, emulating a clean install. This gave me a fresh, newly-generated sonatype-work directory to compare to my existing one. I eventually traced the failed startup to the security-configuration.xml file. Here's the original one:
<?xml version="1.0"?> <security-configuration> <version>2.0.3</version> <enabled>true</enabled> <anonymousAccessEnabled>true</anonymousAccessEnabled> <anonymousUsername><!-- hidden --></anonymousUsername> <anonymousPassword><!-- hidden --></anonymousPassword> <realms> <realm>NexusLdapAuthenticationRealm</realm> <realm>XmlAuthenticatingRealm</realm> <realm>XmlAuthorizingRealm</realm> </realms> <securityManager>web</securityManager> </security-configuration>
The <securityManager> element and the NexusLdapAuthenticationRealm were absent in the fresh config. I restored my original sonatype-work directory and removed these two lines. Nexus 22.214.171.124 now starts up successfully! It is now three in the afternoon, I needed some good news by now.
I went into the web interface and re-enabled LDAP on the Administration/Server page. This restored the NexusLdapAuthenticationRealm in my XML file, meaning that the <securityManager> element is what prevented my downgraded Nexus from starting. The very first Nexus version I installed on our build server was 1.7.1. The securityManager element must be a relic of that version, although this doesn't explain why this didn't previously prevent Nexus from starting up.
Did that fix my Maven builds?
Not at first sight. The Rebuild Metadata task in Nexus 126.96.36.199 skips any existing metadata files. Once I figured that out I nuked these on the filesytem with a quick find /home/maven/sonatype-work/nexus/storage/snapshots -name "maven-metadata.xm*" -delete. After another Rebuild Metadata, I finally have my repository metadata in Maven 2 format.
Did that fix my Maven builds? Nope, because the Maven 3-style metadata files were still in my local repository. This is the home stretch; removing them from the local repository with another quick find command got my Maven 2 builds working again.
Getting some satisfaction
rm -rf nexus-oss-webapp-1.9
Suck on that, you empty-headed animal food trough wiper! I fart in your general direction*!
With Maven 3 around, installing stuff from Sonatype when you're still on (update: an outdated) Maven 2 version is a dangerous proposition. Nexus 1.9 has proven to be incompatible with Maven 2, unintentionally I'm sure. Not upgrading Nexus is easier than downgrading it. I refuse to run a DTAP chain for my development support tooling, so I'll just refrain from updating Nexus altogether from here on.
I'll use Jenkins to satisfy my keeping-stuff-up-to-date compulsions. 😉