Development and DevOps

Use SSH keys with Terraform on Azure

Wed, 24 Feb 2016 10:33:17 +1200

Recently I’ve been using an awesome tool called Terraform for provisioning temporary cloud infrastructure for performance and scalability testing some software.

I initially started with AWS and then moved on to repeating the same basic infrastructure on Azure. Azure is a bit different from AWS in terms of its resource types but generally maps quite well. One thing that caused an hour of head scratching and frustration was trying to configure the virtual machines I was creating to use SSH keys.

When you create a Linux virtual machine on Azure via the Portal you are given the opportunity to provide a public SSH key to use for authentication. I expected Terraform to provide an ssh_key parameter but instead found an ssh_key_thumbprint parameter. After some digging it turns out you can associate certificates with a “hosted service” in Azure and then reference those when creating virtual machines.

Terraform doesn’t provide support for uploading SSH keys so I came up with a simple solution. Note this requires that you have the Azure CLI tool installed (and that you’ve logged in) as well as OpenSSH.

In your Terraform template take two arguments:

azure_ssh_key_path - The path to the SSH key you want to use for your virtual machines. Note it must be in der format.
azure_ssh_key_fingerprint - The fingerprint of the SSH key.

Then, when creating hosted services, use a local provisioner to upload the key:

variable "azure_ssh_key_path" {}
variable "azure_ssh_key_fingerprint" {}

resource "azure_hosted_service" "hosted_service" {
  name = "my-hosted-service"
  location = "West US"
  ephemeral_contents = true
  provisioner "local-exec" {
    command = "azure service cert create ${self.name} ${var.azure_ssh_key_path}"
  }
}

And finally, when creating a virtual machine inside the hosted service, use the thumbprint:

resource "azure_instance" "client" {
  name = "my-virtual-machine"
  image = "${var.azure_image_name}"
  size = "Standard_G5"
  hosted_service_name = "${azure_hosted_service.hosted_service.name}"
  storage_service_name = "..."
  location = "West US"
  username = "my-user"
  ssh_key_thumbprint = "${var.azure_ssh_key_fingerprint}"
  endpoint {
    name = "SSH"
    protocol = "tcp"
    public_port = 22
    private_port = 22
  }
}

If you need help getting your key in to the right format for Azure you can use this bash script:

SSH_KEY_PATH="~/.ssh/id_rsa"
AZURE_SSH_KEY_PATH="/tmp/azure-deployer.pfx"
openssl req -x509 \
  -key $SSH_KEY_PATH \
  -nodes \
  -days 365 -newkey rsa:2048 \
  -out /tmp/azure-deployer.pem \
  -subj '/CN=www.mydomain.com/O=MyCompany./C=US'
openssl x509 \
  -outform der \
  -in /tmp/azure-deployer.pem \
  -out $AZURE_SSH_KEY_PATH
fingerprint=$(openssl x509 -fingerprint -inform der -in $AZURE_SSH_KEY_PATH | grep "SHA1 Fingerprint")
fingerprint="${fingerprint#*=}"
fingerprint="${fingerprint//:/}"
AZURE_SSH_KEY_FINGERPRINT=$fingerprint

Then pass the AZURE_SSH_KEY_PATH and AZURE_SSH_KEY_FINGERPRINT variables to Terraform:

terraform apply -var azure_ssh_key_path=$AZURE_SSH_KEY_PATH -var azure_ssh_key_fingerprint=$AZURE_SSH_KEY_FINGERPRINT

Hosting a YUM repository on Amazon S3

Sat, 09 Jan 2016 10:33:17 +1200

Up until recently at StorReduce we had been building and distributing our software as an RPM but we didn’t actually operate a YUM repository. This meant that while we could give a customer a new RPM to install manually, they couldn’t just run yum update to pull down the latest version of StorReduce.

We decided to start operating a YUM repository to make things easier, which meant finding a suitable place to host it, as well as updating our build processes to publish RPMs to it as part of our CI process, and to pull RPMs from it when we built our appliances.

To avoid operating our own web server we turned to Amazon S3 to host the repository. By leveraging S3’s ability to serve static content over HTTP(S) and a few simple lines of shell script we were able to come up with a scalable and secure solution that requires zero maintenance. In addition to this we created a simple way to support development and production repositories, with an easy way to “promote” a specific RPM from development to production, thereby exposing it to our end users for use. Additional stages could be added to the workflow (e.g. perhaps introducing a QA repo in between dev and prod, but that will be specific to the situation).

Below is a guide for reproducing this setup. I recommend running these scripts via an automated CI tool, which will give you visibility and auditability over what was pushed to production and when.

1. Create and configure the buckets

In the S3 web console create two buckets, one for development and one for production:

production-yum\
development-yum\

Next we need to ensure that the buckets are publicly accessable, so update each buckets bucket policy to enable read-only access for everyone, making sure to substitute in the correct bucket name:

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "AddPerm",
			"Effect": "Allow",
			"Principal": "*",
			"Action": "s3:GetObject",
			"Resource": "arn:aws:s3:::production-yum/*"
		}
	]
}

2. Create your repo definition file (and optional GPG key)

Now we need to create the .repo file that can be imported in to end users systems to allow them to install packages from our repo.

This is the time to create a GPG key if you haven’t already, I recommend following the instructions on the Fedora wiki, specifically the Creating GPG Keys Using the Command Line and Exporting a GPG Key Using the Command Line sections.

Once you have a GPG key upload it to the root of each bucket. You can skip using GPG keys if you’d like, although it isn’t recommended.

For each of the buckets created in step 1, create a repo-name.repo file in the root of the bucket with the following contents, again making sure to substitute in the name of your bucket/repo:

[production]
name=Production Packages  - $basearch
baseurl=https://production-yum.s3-us-west-2.amazonaws.com/$basearch/
enabled=1
gpgkey=https://production-yum.s3-us-west-2.amazonaws.com/my-gpg-key
gpgcheck=1

NOTE: If you’ve decided not to use GPG keys then delete the gpgkey line above and set gpgcheck=0.

Your buckets should now look like this:

production-yum\
  - production.repo
  - my-gpg-key

development-yum\
  - development.repo
  - my-gpg-key

3. Publish your RPMs

In this step I’ll assume you are already producing RPMs via a tool like rpmbuild. If you want to learn how to create RPMs then the Fedora Wiki has a detailed (if a little dense) article here. If you’ve opted to use GPG keys make sure you sign the RPMs when you build them.

Once you have built your RPM the next step is to publish it to the repo(s) hosted on S3. The basic idea is that you pull down the contents of the repo to your local machine, add the new RPM, then push the updated repo contents back to S3.

The trick is to use the aws s3 sync command to ensure only new/modified files are moving back and forth when pulling/pushing to S3. To make this work you should always use a consistent folder name for the work directory. I chose to use /tmp/$BUCKET_NAME which means that for the uptime of our build server we will use a cached copy of the repo, making RPM publishing nice and quick.

Here is the script that we use, it expects to be given a source directory that contains the output of rpmbuild and a target bucket that contains the YUM repo that it will publish the RPM to. It works regardless of if the repo has been created before or not (by omitting the --update arg).

I recommend only using this script to publish to the development repo you created in step 1, we will use a different script in step 4. to move RPMs from this development repo to the production repo.

publish-rpm.sh:

./publish-rpm.sh -s /tmp/rpmbuild -t development-yum

#!/bin/bash
# Publishes built RPMs to an s3-backed RPM repo.
set -e
if [ ! -z "${DEBUG}" ]; then
  set -x
fi

SCRIPT_DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
SRC_BASE="${SCRIPT_DIR}/../.."

DEPENDENCIES=("aws" "createrepo")
REGION="us-west-2"
SOURCE_DIR=""
TARGET_BUCKET=""

for dep in "${DEPENDENCIES[@]}"
do
  if [ ! $(which ${dep}) ]; then
      echo "${dep} must be available."
      exit 1
  fi
done

while getopts "s:t:" opt; do
  case $opt in
    s) SOURCE_DIR=$OPTARG ;;
    t) TARGET_BUCKET=$OPTARG ;;
    \?)
      echo "Invalid option: -$OPTARG" >&2
      exit 1
      ;;
  esac
done

if [ -z "${SOURCE_DIR}" ]; then
  echo "Source directory must be specified."
  exit 1
fi

if [ -z "${TARGET_BUCKET}" ]; then
  echo "Target bucket must be specified."
  exit 1
fi

TARGET_DIR="/tmp/${TARGET_BUCKET}"

# make sure we're operating on the latest data in the target bucket
mkdir -p $TARGET_DIR
aws --region "${REGION}" s3 sync "s3://${TARGET_BUCKET}" $TARGET_DIR

# copy the RPM in and update the repo
mkdir -pv $TARGET_DIR/x86_64/
cp -rv $SOURCE_DIR/RPMS/* $TARGET_DIR
UPDATE=""
if [ -e "$TARGET_DIR/x86_64/repodata/repomd.xml" ]; then
  UPDATE="--update "
fi
for a in $TARGET_DIR/x86_64 ; do createrepo -v $UPDATE --deltas $a/ ; done

# sync the repo state back to s3
aws --region "${REGION}" s3 sync $TARGET_DIR s3://$TARGET_BUCKET

4. Promote RPMs from development to production

Now that RPMs can be published to the development repo you have an opportunity to test these builds internally. Once a release of a new version of your software is tagged you will want to push the corresponding RPM to the production repo so that your end users can access it via yum update.

Here is the script that we use to do that. It takes the name of the RPM, the name of the bucket to copy it from and the name of the bucket to copy it to.

This script is quite similar to the previous script, except that it knows to fetch the source RPM from the source bucket/repo instead of the local file system.

promote-rpm.sh:

./promote-rpm.sh -r my-rpm-1.0.1 -s development-yum -t production-yum

#!/bin/bash
# Promotes an RPM from one repo to another (e.g. dev -> prod)
set -e
if [ ! -z "${DEBUG}" ]; then
  set -x
fi

SCRIPT_DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
SRC_BASE="${SCRIPT_DIR}/../.."

DEPENDENCIES=("aws" "createrepo")
REGION="us-west-2"

while getopts "s:t:r:" opt; do
  case $opt in
    r) RPM_MATCH=$OPTARG ;;
    s) SOURCE_BUCKET=$OPTARG ;;
    t) TARGET_BUCKET=$OPTARG ;;
    \?)
      echo "Invalid option: -$OPTARG" >&2
      exit 1
      ;;
  esac
done

for dep in "${DEPENDENCIES[@]}"
do
  if [ ! $(which ${dep}) ]; then
      echo "${dep} must be available."
      exit 1
  fi
done

if [ -z "${RPM_MATCH}" ]; then
  echo "RPM match string must be specified."
  exit 1
fi

if [ -z "${SOURCE_BUCKET}" ]; then
  echo "Source bucket must be specified."
  exit 1
fi

if [ -z "${TARGET_BUCKET}" ]; then
  echo "Target bucket must be specified."
  exit 1
fi

SOURCE_DIR="/tmp/${SOURCE_BUCKET}"
TARGET_DIR="/tmp/${TARGET_BUCKET}"

# make sure we're operating on the latest data in the source bucket
mkdir -p $SOURCE_DIR
aws --region "${REGION}" s3 sync "s3://${SOURCE_BUCKET}" $SOURCE_DIR

# make sure we're operating on the latest data in the target bucket
mkdir -p $TARGET_DIR
aws --region "${REGION}" s3 sync "s3://${TARGET_BUCKET}" $TARGET_DIR

# copy the RPM in and update the repo
mkdir -pv $TARGET_DIR/x86_64/
cp -rv $SOURCE_DIR/x86_64/$RPM_MATCH-1.x86_64.rpm $TARGET_DIR/x86_64/
UPDATE=""
if [ -e "${TARGET_DIR}/noarch/repodata/repomd.xml" ]; then
  UPDATE="--update "
fi
for a in $TARGET_DIR/x86_64 ; do createrepo -v $UPDATE --deltas $a/ ; done

# sync the repo state back to s3
aws --region "${REGION}" s3 sync $TARGET_DIR s3://$TARGET_BUCKET

5. Test the new repos

This step is nice and simple and is exactly what end users will need to do to install or update RPMs from the new repo.

Install the new repo on a client machine:

sudo wget https://production-yum.s3-us-west-2.amazonaws.com/production.repo -O /etc/yum.repos.d/production.repo

Then start installing or updating packages from the repo:

sudo yum install <my-package-name>

There you have it, a secure and scalable YUM repo hosted on S3 that requires no servers or maintenance.

Build VMware Appliances with Packer on Linux

Sun, 20 Sep 2015 11:11:06 +1200

Recently I set up automated VMware virtual appliance builds using Packer and Ansible. I had manually tested and pieced together the various steps on OS X, and thought everything would Just Work™ when I transferred it over to Linux to run on our automated build infrastructure. I was wrong…

It turns out building on Linux can be a painfully cryptic experience. To begin with, it’s not well documented what you actually need to install to get a functioning set up, and even once you have a set up you expect to work the various VMware components can often return generic error messages which Packer unhelpfully parrots. You frequently have to resort to installing a desktop environment to get the real error message via a dialog in the VMware Workstation UI. Below is a brain dump to help anyone else out that finds themselves with a similar need. It assumes you’re already familiar with Packer and have a VMware based build ready to run on Linux.

Requirements

Firstly, I never got VMware Player to work on Linux, despite the odd scrap of Packer documentation that might suggest otherwise. I found various GitHub issues where others had the same experience. So the first requirement is to use VMware Workstation (specifically I use version 10).

Secondly, you will need a valid license key for VMware Workstation.

Next up, you need to install the VMware Vix component. You need the version that corresponds to the version of Workstation you’re using. I used version 1.14.2 which corresponds to VMware Workstation 10.

Finally, if you want to distribute a virtual appliance that can be loaded on any version of the desktop apps (Workstation, Player, Fusion etc.) as well as the server offerings (vSphere, ESXi etc.) then you will want to produce an OVA (single-file OVF, effectively) instead of a VMX file, so you need to install OVF Tool. You can grab the latest version of this, independently of what Workstation version you use.

Automation

If you’re like me you want to automate the provisioning of build servers. Here is the Ansible role I use to install VMware on Ubuntu 14.04 ready for appliance builds. Even if you’re not interested in Ansible you can follow the steps manually in your shell:

vars/main.yml:

vmware:
  workstation_key: "VALID LICENSE KEY HERE"
  workstation_url: "https://<your file server here>/VMware-Workstation-Full-10.0.6-2700073.x86_64.bundle"
  workstation_download_path: "/tmp/workstation.bundle"
  vix_url: "https://<your file server here>/VMware-VIX-1.14.2-2780323.x86_64.bundle"
  vix_download_path: "/tmp/vix.bundle"
  ovftool_url: "https://<your file server here>/VMware-ovftool-4.1.0-2459827-lin.x86_64.bundle"
  ovftool_download_path: "/tmp/ovftool.bundle"

tasks/main.yml:

---
- name: Install Packages
  apt: pkg={{item}} state=installed
  with_items:
    - git
    - qemu-utils
    - build-essential
    - libxtst6
    - libxss1
    - libxcursor1
    - libxinerama1
    - libxi6

- name: Download VMware Workstation
  get_url: url={{ vmware.workstation_url }} dest={{ vmware.workstation_download_path }} mode=755

- name: Download VMware VIX
  get_url: url={{ vmware.vix_url }} dest={{ vmware.vix_download_path }} mode=755

- name: Download VMware OVFTool
  get_url: url={{ vmware.ovftool_url }} dest={{ vmware.ovftool_download_path }} mode=755

- name: Install VMware Workstation
  shell: "{{ vmware.workstation_download_path }} --eulas-agreed --required"

- name: Licence VMware Workstation
  shell: /usr/lib/vmware/bin/vmware-vmx --new-sn {{ vmware.workstation_key }}

- name: Install VMware VIX
  shell: "{{ vmware.vix_download_path }} --eulas-agreed --required"

- name: Install VMware OVFTool
  shell: "{{ vmware.ovftool_download_path }} --eulas-agreed --required"

Building OVAs

By default Packer will produce a VMX based virtual machine (via the vmware-vmx builder). If you want to package this in to a single file for easy distribution, or you want to make sure it can be used on vSphere, then you need to convert it to an OVA.

There is a community provided post-processor for Packer here that you can then invoke from your packer.json file. I opted not to use this, as I didn’t a) want to install golang on my build servers just to compile/install this or b) didn’t want to manage my own pre-compiled binaries for it. Instead I chose to use a shell script to call OVF Tool after Packer has finished.

Here’s an example of that script - I’ve left some of the skeleton of our main build-vm.sh script in place, so that you can see how we drive Packer and convert the VMX to an OVA if appropriate.

build-vm.sh:

#!/bin/bash

# Do your domain-specific steps here (e.g. accept args for different infrastructures
# and set certain environment variables for Packer to use.)

# ...

# infrastructure specific options and steps
case "$INFRASTRUCTURE" in
  "$INFRA_VMWARE")
    export PACKER_BUILDER="vmware-vmx"
    # ...
  ;;
  "$INFRA_AMAZON")
    export PACKER_BUILDER="amazon-ebs"
    # ...
  ;;
  # ...
esac

# build the vm via packer
cd "${SRC_BASE}/build/packer-storreduce" && packer build -only $PACKER_BUILDER storreduce.json

# special case for vmware builds - convert to ova AFTER packer has run
if [[ "${INFRASTRUCTURE}" == "${INFRA_VMWARE}" ]]; then
  cd "${SCRIPT_DIR}" && ./vmx-to-ova.sh -s "${PACKER_OUTPUT_DIRECTORY}"
fi

vmx-to-ova.sh

#!/bin/bash

set -e
if [ ! -z "${DEBUG}" ]; then
  set -x
fi

DEPENDENCIES=("ovftool")
for dep in "${DEPENDENCIES[@]}"
do
  if [ ! $(which ${dep}) ]; then
      echo "${dep} must be available."
      exit 1
  fi
done

print_usage () {
  echo "vmx-to-ova.sh - Converts a VMX to an OVA and deletes the VMX if successful."
  echo "-s=<source_vmx_folder>    The directory that contains the VMX."
}

while getopts "s:" opt; do
  case $opt in
    s) SOURCE_DIR=$OPTARG ;;
    \?)
      echo "Invalid option: -$OPTARG" >&2
      print_usage
      exit 1
      ;;
  esac
done

if [[ -z "${SOURCE_DIR}" ]]; then
  echo "Source VMX folder must be specified"
  print_usage
  exit 1
fi

for vmx in "$SOURCE_DIR"/*.vmx; do
  name=$(basename "${vmx}" .vmx)
  ovftool -dm=thin --compress=1 "${vmx}" "${SOURCE_DIR}/${name}.ova"
done

cd "${SOURCE_DIR}" && rm *.vmdk *.vmx *.vmxf *.vmsd *.nvram

Troubleshooting

The virtual machine has exited.

At some point you’ll probably see this horrifically unhelpful error message in the Packer logs:

The virtual machine has exited.

This comes from VMware, and it can mean a wide variety of things. These are some of the things to check before falling back to the last resort:

Did you license VMware Workstation with a valid license key?
Does the host machine you’re building on have enough memory to support the VM you’re trying to start?
Does the host machine you’re building on have enough CPU to support the VM you’re trying to start?
Does the machine you’re building on have VT-X enabled? e.g. If you’re building inside a nested virtual machine you will need to enable nested virtualization. Don’t even bother to try on public clouds like AWS or GCE etc. as they don’t enable this.

Can’t think of anything you’ve done wrong? Then sadly the last resort is to install a desktop environment and attempt to manually start the VM through the VMware Workstation UI - you’ll get the real error message in a pop up dialog.

Failed to build vmnet. Failed to execute the build command.

This one only recently started happening to me with the release of kernel 3.19. I initially received the generic error message above and after falling back to installing a desktop environment and starting the Workstation UI was prompted to recompile VMware kernel modules for the new kernel. After clicking OK and waiting a while the process failed with this message:

Failed to build vmnet. Failed to execute the build command.

Some Googling lead me to this helpful thread. I followed the instructions in this answer (after making sure I was happy with the content of the patch) and that brought VMware back to life.

Adding this step to my Ansible role is currently on my TODO list.

SSH Keys with Docker and TeamCity

Sat, 19 Sep 2015 10:33:17 +1200

Lately I’ve been configuring TeamCity to run builds inside of Docker containers. One of the classes of builds I’ve been configuring is nightly integration testing. These tests use Ansible to provision a variety of cloud resources including servers, storage backends and load balancers, and then install and configure our software and test it behaves as expected in real-world scenarios (automated cluster failover testing, for example). Due to the nature of these tests they need to be able to SSH in to the provisioned servers - which means they need SSH key access. TeamCity has this covered.

TeamCity has SSH key management built in - you simply upload your key to TeamCity via the web interface, and then on a per-build basis you can tell it to push the key to the build agent, and have it made available via the local SSH agent for the duration of the build. This is exactly what I needed, except out of the box it doesn’t work with the Docker plugin. Luckily for us the underlying mechanism TeamCity uses to load the key is the local SSH agent, so we just need to make sure the host’s SSH agent is available within the build’s Docker container.

This turned out to be incredibly simple. The trick is to mount the local SSH agent within the Docker container via the -v flag, then to expose the mounted SSH agent within the Docker container via the SSH_AUTH_SOCK environment variable.

TeamCity Docker Step Config:

-v "$SSH_AUTH_SOCK:/tmp/ssh_auth_sock"
-e "SSH_AUTH_SOCK=/tmp/ssh_auth_sock"

After making this tiny configuration change you can then use the SSH Agent build feature just as you normally would, and your Dockerized builds will inherit the SSH key access!

About

Fri, 18 Sep 2015 18:23:05 +1200

Hello I’m Tyler. I’m an entrepreneur and software engineer. I founded Appsecute which was acquired by ActiveState, then worked on Stackato through to its acquisition by HP and now I’m helping to disrupt Cloud Storage at StorReduce.

Follow me on Twitter.

Connect with me on LinkedIn.

CI with Docker and TeamCity

Fri, 18 Sep 2015 18:23:05 +1200

Recently I’ve been working on setting up the build infrastructure for StorReduce. We have a diverse range of test, build and deployment scenarios including unit testing, integration testing, producing binaries and RPM’s, producing virtual appliances for various infrastructures and deploying our website, and I expect this to grow in the future.

I decided to use my go-to Continuous Integration (CI) solution - TeamCity. I’ve only ever had good experiences with TeamCity - it’s incredibly mature and it continues to get better with age, it’s reliable and highly configurable and it Just Works™.

Given the diverse range of builds we want to push through TeamCity it was desirable to be able to compartmentalize the software on the build servers. It can quickly get out of hand having to ensure a growing list of requirements (golang and tooling, hugo, aws cli, rpmbuild, test utils, debian packaging tools, ansible, python… etc.) are configured and installed correctly on each build server in a CI pipeline. That’s where Docker comes in.

Docker and CI are a match made in heaven. Using Docker, each build in the pipeline can be run in a tailor-made, isolated and consistent environment every single time. Each build server can be configured to have the bare minimum requirements installed (OS, build agent software and Docker) and the rest can be pulled in via a Docker image based on the build that is running. This makes the build infrastructure much easier to provision and maintain.

Unfortunately, Docker is not natively supported in TeamCity, however, there is a brilliant open source plugin called TeamCity.Virtual that adds a Docker target to TeamCity builds. To install the plugin follow the simple instructions here. Additionally, you will need to ensure Docker is installed on each of your build servers.

Once the TeamCity.Virtual plugin is installed TeamCity will start showing a new build step:

And after selecting it you can specify which Docker image to use, as well as the command to run:

For the build above, I mount the TeamCity home directory in to the Docker container via the -v flag, as our build script needs to run git commands to determine the version of our software (our versioning is managed via git tags), and in TeamCity the git checkout lives in the home directory.

The final link in the chain is to ensure you have a Docker image suited to the build you’re configuring. The build pictured is our RPM build, it compiles our binaries and produces an RPM. To do this we need to ensure rpmbuild is installed, as well as golang and supporting tools. We then push our built Docker image to the Docker Hub for access during the build. One of the nice things about the TeamCity.Virtual plugin is that it does a docker pull at the beginning of the build, so it will always pull down the latest version of your Docker images.

Here is the Docker file used for our RPM build - everything in here is something we no longer have to set up on each build server thanks to Docker:

# This docker image is used on CI for building RPMs
# awscli is installed for uploading to S3 after the build, as well as godep
# and rpm for the rpm build.

FROM centos:6
MAINTAINER StorReduce

# install EPEL repos
RUN yum -y install wget && \
    wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm && \
    yum -y install epel-release-6-8.noarch.rpm && \
    rm epel-release-6-8.noarch.rpm

# install dev packages
RUN yum -y groupinstall 'Development Tools'

# install our dependencies
RUN yum -y install which && \
    yum -y install tar && \
    yum -y install hg && \
    yum -y install git && \
    yum -y install python-pip && \
    yum -y clean all && \
    pip install awscli && \
    mkdir /root/.aws

# install go
RUN wget https://storage.googleapis.com/golang/go1.4.2.linux-amd64.tar.gz && \
    tar -C /usr/local -xzf go1.4.2.linux-amd64.tar.gz && \
    rm go1.4.2.linux-amd64.tar.gz

# configure go
RUN mkdir -p /go/src /go/bin && chmod -R 777 /go
ENV GOPATH /go
ENV PATH /usr/local/go/bin:/go/bin:$PATH
WORKDIR /go

# install godep
RUN go get github.com/tools/godep

Finally, when the builds runs you get the same great logging output from TeamCity, here you can see the build executing inside the Docker container:

Linear and Exponential Back-Off for Go

Sun, 19 Apr 2015 15:25:22 +1200

I’ve recently been working on extending my side project Connectrix to support receiving event data from Twitter. The natural fit was to use the Twitter Stream API to receive events via a push with low latency and low overhead. I looked around and found a couple of Go libraries for consuming the Twitter Stream, but one of them looked very hacky and far from idiomatic Go code, while the second was very large and made a lot of assumptions about the use case. So, I started to write my own low-level, hopefully idiomatic, library for connecting to these streams, the result is go-http-stream-reader.

One of the interesting things about the Twitter Stream API is that is enforces a rate limit, which needs to be managed by adhering to their published back off rules - if you don’t back off when told to then they will punish you for it by making you wait even longer.

From the Twitter Streaming API docs:

Once an established connection drops, attempt to reconnect immediately. If the reconnect fails, slow down your reconnect attempts according to the type of error experienced:

Back off linearly for TCP/IP level network errors. These problems are generally temporary and tend to clear quickly. Increase the delay in reconnects by 250ms each attempt, up to 16 seconds.

Back off exponentially for HTTP errors for which reconnecting would be appropriate. Start with a 5 second wait, doubling each attempt, up to 320 seconds.

Back off exponentially for HTTP 420 errors. Start with a 1 minute wait and double each attempt. Note that every HTTP 420 received increases the time you must wait until rate limiting will no longer will be in effect for your account.

To manage these back off rules I needed to implement an exponential back off algorithm. This algorithm is very trivial but it comes up frequently so I thought it would be good to create a little Go library to take care of it. The result of that is go-backoff. It implements linear, exponential and exponential with full jitter back off algorithms and makes it trivial to use them. For a more detailed look at the use cases for exponential back off algorithms and their behaviour then take a look at this great post from the Amazon AWS blog.

To use go-backoff it’s as simple as instantiating a backoff instance with the desired behaviour:

// Back off linearly, starting at 250ms, capping at 16 seconds
linear = backoff.NewLinear(250*time.Millisecond, 16*time.Second)

// Back off exponentially, starting at 5 seconds, capping at 320 seconds
exp = backoff.NewExponential(5*time.Second, 320*time.Second)

// Back off exponentially, starting at 1 minute, with no cap
expt = backoff.NewExponential(time.Minute, 0)

// Back off between 0 and exponentially, starting at 30 seconds, capping at 10 minutes
expj = backoff.NewExponentialFullJitter(30*time.Second, 10*time.Minute)

for {
  err := tryDoThing()
  if err != nil {
    glog.Debugf("Backing off %d second(s)", exp.NextDuration/time.Second)
    exp.Backoff()
  } else {
    exp.Reset()
    break
  }
}

You can find go-backoff on my GitHub here: go-backoff