Installing a nodejs application without your good old internet

Kris Geusebroek

While we were building a little server to enable auditlogging on our hadoop cluster (more on that in a future blogpost) we needed a way to distribute our application.
This blog is about the packaging of this application. The application is build with nodejs and packaging and dependency management is mostly done with npm (the node package manager).

Of course installing this application in the production environment should have been as easy as the setup on our own laptop's right? Wrong! On our laptops it was a easy git clone followed by a npm install and voila we have a running application. So how hard could it be to do this on a server at the client. Let me tell you....

This server is not connected to the internet, so the git clone wouldn't work in the first place. Not really a problem because it's a small app and we could just make a tarball and ship it to the server.
Next thing was all our dependencies. We used a few modules which were mentioned as dependencies in our package.json file so npm with the install command would do it's magic.

The npm install magic consists, among other things, of getting the modules from the npm registry and that fails if you're not connected to the internet. Searching for a way to do this differently I figured out that npm had a cache directory and thought I could get the stuff from there. This might have worked, but with that solution I would miss the dependencies where these modules depended on. And it would be a messy kind of script that I needed to make.

Browsing the internet didn't provide me with the right answer but it led me on the path to the npm pack function. This is used to pack your module together with all dependencies. The only thing you need to configure it correctly, is a separate array containing the dependencies to bundle with your app.

So far so good, so I went on and added a bundleDependencies section in my package.json and ran npm pack.
The result was a nice .tgz file containing all the files needed for the application together with all the modules it depended on. At least the main modules it depended on. My fellow programmers in crime from which I got these modules hadn't bothered to add this extra section, so npm had no notice of the modules they depended on.

This was easy to solve. Just add a correct section of bundleDependencies to all package.json files.
Sounded like a boring task to do this manually and because I love my programming job, I decided to write a program for it.

My obvious choice of programming languages was: awk, grep and sed. Why? Because I can.
Without further ado, here it is:

    awk '/dependencies/,/]|}/' $file |
    grep -o '\".*\".*:' |
    sed 's/^.*{//g' |
    sed 's/\"dependencies.*\://g' |
    grep -v -e '^$' |
    uniq |
    sed 's/\"[ ^I]\:/\",/g' |
    sed 's/\"\:/\",/g' |
    sed '$ s/,/ ]/' |
    sed '1 s/\"/\"bundleDependencies\" \: [ \"/' |
    sed 's/\"/\\"/g' |
    tr -d '\n'

What it does, you ask?
I'll explain line by line:

line1: get the dependencies part (an array or json object) from the file (the package.json file)
ex: "dependencies" : { "express": "0.2.2", "findit" : "0.0.1" }

line2: get only the stuff from that object between the double quotes before the colon (removing the version number part of the dependency)
ex: "express":
"findit":

line3: remove anything preceding the {
line4: remove the original dependencies text
line5: remove any empty lines
line6: remove duplicates

line7 and line8: replace the ": by ", so we can create an array from it
ex: "express",
"findit",

line9: replace the last , by an ] to close the array
ex: "express",
"findit"]

line10: replace the first " by "bundleDependencies" : [ "
ex: "bundleDependencies" : [ "express",
"findit"]

line11: precede all quotes by a backslash so it can be safely used in the sed command to add it to the file

line12: remove all newlines
ex: "bundleDependencies" : [ "express", "findit"]

This is added to the package.json file we are currently processing and if we have finished doing this for all package.json files we can use npm pack to create our tarball.
Works pretty well I might say. But I'm the first to admit this isn't the most readable program ever written.

Of course when building a node application you might have node around to help you do this so I also created a javascript version to do this:

var fs=require('fs')
var findit=require('findit')

findit.find('.', function(name) {
  if (endsWith(name,'package.json')) {
    handleFile(name)
  }
}).on('end', bundleApp)

function handleFile(file) {
  var data = fs.readFile(file, function(err, data) {
    if (err) {
      console.log('Not processesed '+file+' bo the following error: '+err)
    } else {
      var arr = []
      var packageFile = JSON.parse(data)
      if (packageFile.bundleDependencies) {
        console.log('Bundledeps already present. Skipping')
      } else {
        for (var d in packageFile.dependencies) {
          arr.push(d+"")
        }
        if (arr.length > 0) {
          packageFile['bundleDependencies'] = arr
          fs.writeFile(file, JSON.stringify(packageFile, null, 4))
        }
      }
    }
  })
}

function bundleApp() {
  console.log('Finished. preparing package.json files for packaging. Now run npm pack to create the fullblown tarball')
  //exercise left for the reader to require('npm') and run the pack command
}

function endsWith(str, suffix) {
  return str.indexOf(suffix, str.length - suffix.length) !== -1;
}

Hope somebody can benefit from this in the future.

Comments (0)

    Add a Comment