Continuous infrastructure: The other CI

Want to boost the efficiency of your DevOps operations? Make infrastructure a first-class citizen of the CI pipeline.
517 readers like this.
diagram of planning a cloud

Opensource.com

Continuous delivery (CD) and continuous integration (CI) are two well-known aspects of DevOps. But the CI in vogue today is missing a critical “I:” infrastructure.

There was a time when "infrastructure" meant headless black boxes, enormous server rooms, and towering racks—not to mention procurement processes that stretched for months and load estimates that erred on the side of surplus. Then came the virtual machine revolution, which made the infrastructure, well, virtual—and the world has never been the same. We no longer need to manage brick-and-mortar boxes. We can create and destroy, start and stop, upgrade and downgrade machines with just a few clicks.

There's a popular story about a bank that went digital and introduced online forms, which customers needed to fill out manually, print, and snail-mail to the bank. That's where we are today with infrastructure: using new technology to do things the same old way.

In this article, we'll look at progressive infrastructure management, treating infrastructure as a versioned artifact and exploring the concept of immutable servers. In a subsequent post, we’ll look at how open source tools can be used to achieve continuous infrastructure.

continuous infrastructure pipeline

The in-practice continuous infrastructure pipeline

This is the familiar CI, release-early, release-often cycle pipeline. This pipeline is missing a key component: infrastructure.

Pop quiz:

  • How do you create and upgrade your infrastructure?
  • How do you control and track changes to your infrastructure?
  • How does your infrastructure scale with your business?
  • How do you ensure tests on the right infrastructure configuration?

To answer these questions, introduce continuous infrastructure. Split the CI build pipeline into continuous integration code (CIc) and continuous integration infrastructure (CIi) to develop and build code and infrastructure in parallel, converging the two for unified test and release. Make infrastructure a first-class citizen of the CI pipeline.

pipeline with infrastructure

CI pipeline with infrastructure

The defining aspects of CIi include:

1. Code

Create infrastructure by code, not by installation. Infrastructure as code (IaC) is the contemporary method used to develop infrastructure from configuration scripts. These scripts follow the typical development life cycle of coding and unit tests (see the Terraform script below for an example).

2. Version

The IaC artifacts are versioned in the source repository. This brings all the advantages of version control to the infrastructure: consistency, traceability, branching, and tagging.

3. Manage

With coded and versioned infrastructure, you can apply the familiar test and release processes to manage infrastructure development.

CIi offers the following advantages:

1. Consistency

Versioned and tagged infrastructure means you unambiguously know the components and configuration of the system you are using. This establishes an excellent DevOps practice to identify and manage infrastructure consistently.

2. Reproducibility

With infrastructure tagged and baselined, recreating infrastructure is easy. Think of how often you've heard this: “But it works on my machine!” Now you can reproduce a production-like environment quickly in a local test bench to remove environment as a variable of your debug cycle.

3. Traceability

How many times have you gone through history to find out who changed the permissions of a folder, or who upgraded the ssh package? Coded, versioned, released infrastructure eliminates ad hoc changes, bringing easy traceability and predictability to infrastructure management.

4. Automation

With scriptable infrastructure, automation is the next logical step. Automation lets you create infrastructure on demand and destroy it when you're done, so you can focus your valuable time and energy on more productive tasks.

5. Immutability

CIi brings innovations such as immutable infrastructure. Instead of upgrading, you can simply create new infrastructure components (see the note on immutable infrastructure below).

Continuous infrastructure is about evolving run-environments with run-artifacts. Treat infrastructure like code, and take it through proven DevOps processes. The traditional CI is redefined to include that missing "i," leading to a coherent CD.

(CIc + CIi) = CI → CD

Infrastructure as code (IaC)

A key enabler for CIi pipeline is infrastructure as code (IaC). IaC is the mechanism for creating and upgrading infrastructure with configuration files. These configuration files are developed like code and versioned in version control system. The files follow the usual code development life cycle: unit test, commit, build, and release. IaC process brings all advantages of version control for infrastructure development, such as tagging, versioning consistency, and change traceability.

Here's a sample Terraform script to create a two-tier infrastructure on AWS, consisting of a virtual private cloud (VPC), an elastic load balancer (ELB), security groups, and an NGINX server. Terraform is an open source tool to create and change infrastructure through scripts.

terraform script

Sample Terraform script to create two-tier infrastructure on AWS.

The complete script is available on GitHub.

Immutable infrastructure

You have several VMs running and need to apply a security patch. A common approach is to update all systems individually using a remote push script.

Instead of updating the old systems, how about throwing them away and deploying new systems with a security patch installed? This is immutable infrastructure. Since the previous version of infrastructure is versioned and tagged, installing the patch is simply a matter of updating the script and pushing it through the release pipeline.

Now do you see why infrastructure should be a first-class citizen of the CI pipeline?

Tags
User profile image.
Girish has over 20 years’ experience in technology and software at a global IT Services organization based in India. Girish is architect of "I Got" cloud platform to uplift the bottom of the pyramid built with open source stack and contemporary architectural patterns such as microservices, containerisation and multi tenancy. Girish writes on open source and tech topics.

6 Comments

Hi, great post! Would you know some open source projects that use such practices that you talked about?

I think this is what part 2 is for....

"In Part 2, we’ll look at open source tools that can help you implement a continuous infrastructure."

In reply to by Daniel Alencar… (not verified)

Thanks Daniel. Don't know about open source projects that practice CIi, but we certainly use these at our organisation.

In reply to by Daniel Alencar… (not verified)

Great Post Girish!!, we use terraform extensively as part of our DevOps. I think packer could be a great addition to your stack

It would be great to integrate security across all your pipeline and move to DevSecOps.

Creative Commons LicenseThis work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License.