Vagrant, Puppet, and Chef, oh my...

My first attempt at build automation was to use the popular Vagrant software from Hashicorp. Software developers have a way of beating on their machines pretty hard, which means our computers are often different from each other and our production servers in configuration. It can take a lot of work to simply stack up all of the pieces needed to get our work done, but variations between our dev machines can be a big source of stress when something "works for me" but not for somebody else, leading to errors which are difficult to reproduce.

I started usimg using Vagrant to build development environments, so using it to build machine images loaded with data was workable, despite the fact that Vagrant wasn't really designed for building machine images. (Hashicorp provides something called Packer for that.) The existence of two different tools was my first bit of disagreement with hashicorp; if I need a different configuration file to build both a development and a production environment, aren't we just going back to the bad old days where things were normally out of sync?

Over time, I came to see Vagrant more as part of the problem rather than part of the solution. For one thing, it had a number of features, such as virtual networking, that didn't work with AWS and that I didn't use anyway. I was doing almost all of my provisioning with the bash shell and not using the many built-in plug-ins to support Puppet, Chef, Salt, Ansible and many other configuration management systems that, seemed to me to be just another complex thing to learn. The final issue was that I found the Ruby-based internal DSL awkward, having to write obscure code like:

Vagrant.configure("2") do |config|
      config.vm.synced_folder ".", "/vagrant", disabled: true
      config.vm.provider :aws do |aws, override|
      aws.instance_type="c3.xlarge"
   end

  config.vm.provision "shell", inline: "echo 'source .bash.d/*.sh' >> .bashrc", privileged: true
  config.vm.provision "shell", inline: "mkdir .bash.d", privileged: false
  config.vm.provision "file", source: "~/.netrc", destination: ".netrc"
  config.vm.provision "shell", inline: "apt-get install -y python-dateutil"
  config.vm.provision "shell", inline: "apt-get install -y axel"
  ...
  config.vm.provision "shell", path: "install-virtuoso.sh", privileged: true
  config.vm.provision "shell", path: "add-virtuoso-service.sh", privileged: false
  ...
end

when all I really wanted was a few lines of shell script. Small problems became really irksome because it would take five minutes for a server to start and scripts to run to discover that a file I'd copied into the system was rejected by the Unix server because it had Windows line endings. (A problem that the Vagrant developers think should be addressed by adding more compleity to your Vagrantfiles.) Up until Spring 2016 I was still using Vagrant to build machine images for the LEI Demo, but with new products in the AWS marketplace about to be developed, I saw that I didn't have time to waste on a build automation framework that didn't value my time.

A changing environment

Just as I was getting more serious about automation, bigger changes were happening in the world of generic databases. Google, which had bought Freebase, made the decision to shut it down, after incorporating Freebase data and technology into the proprietary Google Knowledge Graph. In the meantime, some interest has shifted to Wikidata, and DBpedia has been steadily improving. In October 2015 I also attended the first DBpedia meetup to be held in the U.S., which was at Stanford University.

As I was learning about popular interests in databases like DBpedia and Freebase, I was simultaneously (i) focused on exploiting traditional data sources with RDF tools and (ii) using data from sources such as DBpedia and Freebase to support that. Excited by the improvements in DBpedia 2015-10, I began the process of rolling out a new generation of AWS marketplace products based on Real Semantics packaging technology.

henson, named after the illustrious puppeteer, is the Real Semantics module that handles cloud packaging. Starting from a clean slate, henson is much faster and more reliable than our old system that used Vagrant for the kind of work that we do. Here is a diagram of the build process:

This process has the following steps:

1. Prior to the product build, most of the files that support the build process, including several gigabytes of DBpedia data files have been transferred to and stored in Amazon S3. This means the files are quickly available to the build server, and the process is not affected by the relatively slow internet connection between the henson server and AWS.
2. Working from an RDF job description, the henson server compiles a cloud-init script. This is primarily a bash shell script that is assembled from bits and pieces of bash scripts in henson's library, united with a standard mechanism for passing configuration information from the henson server to the build server. In most cases, henson can embed all the required configuration in the user data. If necessary, however, henson will upload additional configuration data for this instance to S3.
1. Just before launching the build server, henson also creates an Amazon SQS message queue to receive communications returning from the build service. This way we can monitor a wide range of events on any number of servers with a single polling loop and a single strategy for handling events.
2. henson embeds hooks in the cloud-init shell script that report on the progress of the script and to report errors in a standardized way using the message queue. henson knows with certainty if the script succeeded or failed in terms of shell errors and at which stage the failure occured.
3. The cloud-init script is designed to function without further interaction with the henson server (other than progress tracking) until the database build is complete. This addresses a problem seen with Vagrant, which is that if a process takes several hours to complete, there is a good chance that the internet connection between the control server and the build server will glitch out. Transient internet outages create real and imagined errors in the build process which undermined the reliability and determinism of the build; minimizing round-trip communications dramatically improves reliability and performance.
4. Henson assigns an IAM Instance Profile to the build server. This delegates authority to the build server in a secure way so that the build server can access S3, SQS, and any other resources in AWS. This supports the independence of the build server, since the build server can provision any resoures it needs in the cloud for itself, such as additional disk volumes.
5. henson creates a directory in a designated location in which information such server configuration, process status, timing information, captured server logs and other information about the build is stored
3. The cloud-init script downloads supporting files from S3 to do its job; if possible, software packages are installed using apt-get but if necessary, specialized software packages are compiled and installed with maven or with the GNU toolchain. To save time and improved determinism, we will often make a machine image with software installed, then start from this image when we load the database.
4. The cloud-init script next scans S3 to measure the total size of the RDF data files to be loaded. The script then creates an EBS volume large enough to handle the input data files, attaches it, formats the volume, and copies the files to the volume.
5. The cloud-init script examines the resources available on the machine, such as RAM and direct attached disks and updates the configuration file for the database with tuning parameters ideal for the environment. Preferably we use a build server (such as R3 series server) which has an internal SSD for high throughput and low latency I/O.
6. The cloud-init script then configures and starts the database bulk loader. In the case of OpenLink Virtuoso, this involves the following stages:
   1. Inserting rows into a relational table that tell Virtuoso where to find the files
   2. Running one or more instances of a stored procedure that starts the bulk load process
   3. Polling a relational table to detect completion of the load
   This series of steps converts what would otherwise be an asynchronous operation into a synchronous operation that is easily composable with the steps that preceed and follow it.
7. The database contents are flushed to disk, and the database shut down
8. A new EBS volume is created, large enough to hold the database files plus an additional margin for temporary files. Database files are copied to the EBS volume and the database configuration files are edited to find the database in the new location.
9. The cloud-init script completes, sending a message to the SQS queue notifying the henson server that the load is complete. Had the shell script (or a command run by the script) unexpectedly completed with a non-zero exit value, an SQS message indicating failure would be sent to the henson server
10. The henson server uses the SFTP protocol to download diagnostic files from the server such as cloud-init.log and cloud-init-output.log. henson applies both general-purpose and task-specific rules to determine if the build was successful before moving on the the image creation procedure; if the build fails, the ruleset extracts and display a small set of lines immediately around the failure point to help the operator quicjly diagnose the problem.
11. A final procedure erases any security sensitive information on the build server, such as security keys, and clears the database password stored on the instance. When a machine boots off the image, the user's public key will be installed and the system will generate a new database password. This satisfies the security requirements for a product in the AWS marketplace. This script then shuts down the sever cleanly.
12. henson then commands the EC2 control plane to make an machine image from the build server
13. When the machine image is ready, henson destroys the build server and all of the associated EC2 resources
14. Finally henson displays a report including the identity of the created image and (if it is running on a machine that supports sound with JavaFX) plays a pleasant chime to indicate success. Different sounds are played in case of warning or failure.

Note that we can set henson up to do a different combinations of functions in different cases. If we were interested in using the server directly, for instance, instead of creating an image, that is straightfoward. henson can be used to create development servers and then later be used to create a product image. Unlike Vagrant and some other tools, henson puts developer productivity first -- by minimizing the possibility of making a mistake with a build, running the build as fast as possible, automatically diagnosing problems with the build, and notifying the operator of the status of the build immediately so that person's attention can be put completely on something else while waiting.

Paths not taken (at least so far)