W3C SVG

What is your container strategy?

Do any of these statements cause you anxiety?

The queue is backed up with work, can you add more servers?

One or more of your Amazon EC2 instances are scheduled for maintenance.

Alert: production-api-3 is at 90% disk space.

We need to upgrade the version of Ruby on the production system.

Can I borrow the staging environment for a few days?

We need to rollback the latest code deployment.

Anxiety around these statements usually indicates that your “container” strategy needs to be addressed, and I’m not just talking about your migration to Kubernetes or Docker. In the technical sense of a word, a container is: “an object that can be used to hold or transport something.” From a DevOps perspective, we’re going to look at the four layers of containerization:

  1. Code Artifacts
  2. Server Images
  3. Infrastructure as Code
  4. Configuration Management

No Containerization Strategy

When things go wrong, you’re in trouble. Turning on a new server, or recreating an old server is a daunting task. A mid-deployment failure can be catastrophic to you. Documentation is updated manually and is likely out of date. Not many people understand the deployment strategy.

If you answer yes to any of these questions, you need to address your strategy immediately:

  • Our deployment process includes commands like scp or rsync to move files onto the server
  • We transfer files directly to our server using applications like FileZilla.
  • We don’t use a version control system (Git, TFS, etc.)
  • We don’t have a CI system that builds our code

Code Artifacts

Does this sound like your workflow?

  • You have a build system that generates a .zip file (or tarball) of your code changes (Jenkins, CircleCI, AWS CodeBuild)
  • You have a ./build.sh script in your repo that generates an artifact manually
  • You deploy code by using capistrano, or some other form of git deployment workflow
  • Updating code on the server is a simple git checkout or git pull command
  • You can describe the steps to deploy new code changes in three sentences or less

Congratulations! You’ve reached the first level of containerization! A code container, or more commonly called an artifact, can be any packaged or compressed file that contains your complete code base.

It can be a complex installation script, like a .msi file, or it can simply be a .zip or .tar.gz file of your git repo. In these scenarios, we can define your “container” as the code artifact generated, for instance,build-85.tar.gz. When you need to update new code or rollback old code, you can simply select a new artifact and begin the deployment process. The artifact could be a compressed tar/zip file, or it could be a git commit. Either way, you’ve earned your first DevOps container badge. As long as you have a copy of your code, with a version number in your filename, you’re in good shape.

In an ideal world, these files should be generated by some form of continuous integration systems like CircleCI or Jenkins. It’s not strictly necessary. I can create a versioned copy of a .zip file faster than you can set up Jenkins:

aws s3 cp build.tar.gz s3://my-artifact-bucket/project/build-$(date +%Y-%m-%d-%H%M).tar.gz

Server Images (AMI, VMD, ISO)

Can you re-create an AWS EC2 image by deploying a custom image? Do you have a template that has all of the necessary dependencies to run your application? Does it come pre-installed with your preferred versions of NodeJS, Ruby, Java, and/or Python? If so, you’re well all your way to server images.

Caveat: This only counts if your server images are defined in code, and created using a build tool (Ansible, Packer, Puppet, Chef, etc.) If you’ve simply taken a “snapshot copy” of an existing server, and use that as your image, you haven’t really unlocked the “Server Image” achievement yet.

You’ll identify with this containerization strategy if:

  • All of your server dependencies like Ruby, NodeJS, or Nginx are defined in JSON, Yaml, or other scripting files
  • You have “base images” that you apply additional changes to
  • New employees add their SSH Key to a code repo, which is applied to all servers

When it comes time to upgrade to a new version of Ruby, you’re not afraid. You can branch your existing code repository, update the Ruby version, and spin up a test server in record time. You might still be deploying your code onto this system manually, but when you’re happy with the changes your new “blessed” server image is up to date and ready to go.

Infrastructure as Code

You use Terraform or CloudFormation, and your team is proud of it. Your team complains that the infrastructure code isn’t DRY, or argues over the use of Terraform Modules.

Congratulations, when you need a new load balancer you can write out a definition in a code file, merge the branch, and your load balancer comes online, perfectly configured with the right TLS certificates. AWS Security Groups are managed in code, ingress IP addresses are listed with code comments about why they exist.

You’ve probably also felt the pain of your automation system deleting a crucial production system that you needed (but it thought you didn’t want anymore.)

A word of caution, if you haven’t achieved either Code or OS containerization, you’re chasing the wrong problem. Ask your team:

  • When was the last time a load balancer’s configuration was changed by mistake?
  • How much time do you spend managing DNS records, compared to how much time you spend launching new servers?

In a healthy company, you’ll find much more return on investment in code artifacts and server images than you will in infrastructure as code. That’s not to say that this section isn’t important, it’s just not as important as the others.

Configuration

Where do you store your configuration files? Do they exist anywhere other than the server? Are you confident they are up to date? (Seriously, have you checked the delta between what’s on the server and what the config file is?)

A health configuration management system will let you quickly move secrets out of your codebase and into encrypted storage. It will also let you set multiple configuration settings for each dev/test/qa/production environment you have. Developers should know what this system is, and fully understand the process to add/update values within it.

  • Is it easy for you to roll new API Tokens or Database credentials?
  • Do you have hard-coded security tokens in your codebase?
  • Where do you store your configuration settings?
  • How do you switch configuration settings between environments?

If you are already using a configuration management tool, go double-check your configuration files to make sure they haven’t diverged. You might be surprised.

Team Health Check

Ask your team to answer these simple questions:

  • Is our code stored in a version management system?
  • Do we generate code artifact files that are versioned?
  • Is our code automatically built and versioned when we commit new code?
  • Do we track server dependencies like required apt packages in code anywhere?
  • Is anything on our servers configured manually?
  • Are our operating system images built automatically?
  • Are we managing our infrastructure as code?
  • Do we have a configuration management system in place?
  • Does our configuration management system support local development, QA environments, as well as production settings?
  • Do we store any secrets in our codebase?
  • Are we storing secrets directly on servers?

You’ll want to start at the first “no” answer. Want some help? I’m available for consulting projects. Contact Me.