A beginner's guide to building DevOps pipelines with open source tools

If you're new to DevOps, check out this five-step process for building your first pipeline.
382 readers like this.
Shaking hands, networking

DevOps has become the default answer to fixing software development processes that are slow, siloed, or otherwise dysfunctional. But that doesn't mean very much when you're new to DevOps and aren't sure where to begin. This article explores what a DevOps pipeline is and offers a five-step process to create one. While this tutorial is not comprehensive, it should give you a foundation to start on and expand later. But first, a story.

My DevOps journey

I used to work for the cloud team at Citi Group, developing an Infrastructure-as-a-Service (IaaS) web application to manage Citi's cloud infrastructure, but I was always interested in figuring out ways to make the development pipeline more efficient and bring positive cultural change to the development team. I found my answer in a book recommended by Greg Lavender, who was the CTO of Citi's cloud architecture and infrastructure engineering, called The Phoenix Project. The book reads like a novel while it explains DevOps principles.

A table at the back of the book shows how often different companies deploy to the release environment:

Company Deployment Frequency
Amazon 23,000 per day
Google 5,500 per day
Netflix 500 per day
Facebook 1 per day
Twitter 3 per week
Typical enterprise 1 every 9 months

How are the frequency rates of Amazon, Google, and Netflix even possible? It's because these companies have figured out how to make a nearly perfect DevOps pipeline.

This definitely wasn't the case before we implemented DevOps at Citi. Back then, my team had different staged environments, but deployments to the development server were very manual. All developers had access to just one development server based on IBM WebSphere Application Server Community Edition. The problem was the server went down whenever multiple users simultaneously tried to make deployments, so the developers had to let each other know whenever they were about to make a deployment, which was quite a pain. In addition, there were problems with low code test coverages, cumbersome manual deployment processes, and no way to track code deployments with a defined task or a user story.

I realized something had to be done, and I found a colleague who felt the same way. We decided to collaborate to build an initial DevOps pipeline—he set up a virtual machine and a Tomcat application server while I worked on Jenkins, integrating with Atlassian Jira and BitBucket, and code testing coverages. This side project was hugely successful: we almost fully automated the development pipeline, we achieved nearly 100% uptime on our development server, we could track and improve code testing coverage, and the Git branch could be associated with the deployment and Jira task. And most of the tools we used to construct our DevOps pipeline were open source.

I now realize how rudimentary our DevOps pipeline was, as we didn't take advantage of advanced configurations like Jenkins files or Ansible. However, this simple process worked well, maybe due to the Pareto principle (also known as the 80/20 rule).

A brief introduction to DevOps and the CI/CD pipeline

If you ask several people, "What is DevOps? you'll probably get several different answers. DevOps, like agile, has evolved to encompass many different disciplines, but most people will agree on a few things: DevOps is a software development practice or a software development lifecycle (SDLC) and its central tenet is cultural change, where developers and non-developers all breathe in an environment where formerly manual things are automated; everyone does what they are best at; the number of deployments per period increases; throughput increases; and flexibility improves.

[Read next: A primer on DevOps pipeline: Continuous Integration & Continuous Delivery (CI/CD)]

While having the right software tools is not the only thing you need to achieve a DevOps environment, some tools are necessary. A key one is continuous integration and continuous deployment (CI/CD). This pipeline is where the environments have different stages (e.g., DEV, INT, TST, QA, UAT, STG, PROD), manual things are automated, and developers can achieve high-quality code, flexibility, and numerous deployments.

This article describes a five-step approach to creating a DevOps pipeline, like the one in the following diagram, using open source tools.

Complete DevOps pipeline

Without further ado, let's get started.

Step 1: CI/CD framework

The first thing you need is a CI/CD tool. Jenkins, an open source, Java-based CI/CD tool based on the MIT License, is the tool that popularized the DevOps movement and has become the de facto standard.

So, what is Jenkins? Imagine it as some sort of a magical universal remote control that can talk to many many different services and tools and orchestrate them. On its own, a CI/CD tool like Jenkins is useless, but it becomes more powerful as it plugs into different tools and services.

Jenkins is just one of many open source CI/CD tools that you can leverage to build a DevOps pipeline.

Name License
Jenkins Creative Commons and MIT
Travis CI MIT
CruiseControl BSD
Buildbot GPL
Apache Gump Apache 2.0
Cabie GNU

Here's what a DevOps process looks like with a CI/CD tool.

CI/CD tool

You have a CI/CD tool running in your localhost, but there is not much you can do at the moment. Let's follow the next step of DevOps journey.

Step 2: Source control management

The best (and probably the easiest) way to verify that your CI/CD tool can perform some magic is by integrating with a source control management (SCM) tool. Why do you need source control? Suppose you are developing an application. Whenever you build an application, you are programming—whether you are using Java, Python, C++, Go, Ruby, JavaScript, or any of the gazillion programming languages out there. The programming codes you write are called source codes. In the beginning, especially when you are working alone, it's probably OK to put everything in your local directory. But when the project gets bigger and you invite others to collaborate, you need a way to avoid merge conflicts while effectively sharing the code modifications. You also need a way to recover a previous version—and the process of making a backup and copying-and-pasting gets old. You (and your teammates) want something better.

This is where SCM becomes almost a necessity. A SCM tool helps by storing your code in repositories, versioning your code, and coordinating among project members.

Although there are many SCM tools out there, Git is the standard and rightly so. I highly recommend using Git, but there are other open source options if you prefer.

Name License
Git GPLv2 & LGPL v2.1
Subversion Apache 2.0
Concurrent Versions System (CVS) GNU
Vesta LGPL
Mercurial GNU GPL v2+

Here's what the DevOps pipeline looks like with the addition of SCM.

Source control management

The CI/CD tool can automate the tasks of checking in and checking out source code and collaborating across members. Not bad? But how can you make this into a working application so billions of people can use and appreciate it?

Step 3: Build automation tool

Excellent! You can check out the code and commit your changes to the source control, and you can invite your friends to collaborate on the source control development. But you haven't yet built an application. To make it a web application, it has to be compiled and put into a deployable package format or run as an executable. (Note that an interpreted programming language like JavaScript or PHP doesn't need to be compiled.)

Enter the build automation tool. No matter which build tool you decide to use, all build automation tools have a shared goal: to build the source code into some desired format and to automate the task of cleaning, compiling, testing, and deploying to a certain location. The build tools will differ depending on your programming language, but here are some common open source options to consider.

Name License Programming Language
Maven Apache 2.0 Java
Ant Apache 2.0 Java
Gradle Apache 2.0 Java
Bazel Apache 2.0 Java
Make GNU N/A
Grunt MIT JavaScript
Gulp MIT JavaScript
Buildr Apache Ruby
Rake MIT Ruby
A-A-P GNU Python
SCons MIT Python
BitBake GPLv2 Python
Cake MIT C#
ASDF Expat (MIT) LISP
Cabal BSD Haskell

Awesome! You can put your build automation tool configuration files into your source control management and let your CI/CD tool build it.

Build automation tool

Everything is good, right? But where can you deploy it?

Step 4: Web application server

So far, you have a packaged file that might be executable or deployable. For any application to be truly useful, it has to provide some kind of a service or an interface, but you need a vessel to host your application.

For a web application, a web application server is that vessel. An application server offers an environment where the programming logic inside the deployable package can be detected, render the interface, and offer the web services by opening sockets to the outside world. You need an HTTP server as well as some other environment (like a virtual machine) to install your application server. For now, let's assume you will learn about this along the way (although I will discuss containers below).

There are a number of open source web application servers available.

Name License Programming Language
Tomcat Apache 2.0 Java
Jetty Apache 2.0 Java
WildFly GNU Lesser Public Java
GlassFish CDDL & GNU Less Public Java
Django 3-Clause BSD Python
Tornado Apache 2.0 Python
Gunicorn MIT Python
Python Paste MIT Python
Rails MIT Ruby
Node.js MIT Javascript

Now the DevOps pipeline is almost usable. Good job!

Web application server

Although it's possible to stop here and integrate further on your own, code quality is an important thing for an application developer to be concerned about.

Step 5: Code testing coverage

Implementing code test pieces can be another cumbersome requirement, but developers need to catch any errors in an application early on and improve the code quality to ensure end users are satisfied. Luckily, there are many open source tools available to test your code and suggest ways to improve its quality. Even better, most CI/CD tools can plug into these tools and automate the process.

There are two parts to code testing: code testing frameworks that help write and run the tests, and code quality suggestion tools that help improve code quality.

Code test frameworks

Name License Programming Language
JUnit Eclipse Public License Java
EasyMock Apache Java
Mockito MIT Java
PowerMock Apache 2.0 Java
Pytest MIT Python
Hypothesis Mozilla Python
Tox MIT Python

Code quality suggestion tools

Name License Programming Language
Cobertura GNU Java
CodeCover Eclipse Public (EPL) Java
Coverage.py Apache 2.0 Python
Emma Common Public License Java
JaCoCo Eclipse Public License Java
Hypothesis Mozilla Python
Tox MIT Python
Jasmine MIT JavaScript
Karma MIT JavaScript
Mocha MIT JavaScript
Jest MIT JavaScript

Note that most of the tools and frameworks mentioned above are written for Java, Python, and JavaScript, since C++ and C# are proprietary programming languages (although GCC is open source).

Now that you've implemented code testing coverage tools, your DevOps pipeline should resemble the DevOps pipeline diagram shown at the beginning of this tutorial.

Optional steps

Containers

As I mentioned above, you can host your application server on a virtual machine or a server, but containers are a popular solution.

What are containers? The short explanation is that a VM needs the huge footprint of an operating system, which overwhelms the application size, while a container just needs a few libraries and configurations to run the application. There are clearly still important uses for a VM, but a container is a lightweight solution for hosting an application, including an application server.

Although there are other options for containers, Docker and Kubernetes are the most popular.

Name License
Docker Apache 2.0
Kubernetes Apache 2.0

To learn more, check out these other Opensource.com articles about Docker and Kubernetes:

Middleware automation tools

Our DevOps pipeline mostly focused on collaboratively building and deploying an application, but there are many other things you can do with DevOps tools. One of them is leveraging Infrastructure as Code (IaC) tools, which are also known as middleware automation tools. These tools help automate the installation, management, and other tasks for middleware software. For example, an automation tool can pull applications, like a web application server, database, and monitoring tool, with the right configurations and deploy them to the application server.

Here are several open source middleware automation tools to consider:

Name License
Ansible GNU Public
SaltStack Apache 2.0
Chef Apache 2.0
Puppet Apache or GPL

For more on middleware automation tools, check out these other Opensource.com articles:

Where can you go from here?

This is just the tip of the iceberg for what a complete DevOps pipeline can look like. Start with a CI/CD tool and explore what else you can automate to make your team's job easier. Also, look into open source communication tools that can help your team work better together.

For more insight, here are some very good introductory articles about DevOps:

Integrating DevOps with open source agile tools is also a good idea:

Tags
User profile image.
Bryant Jimin Son is an Octocat, which not official title but likes to be called that way, at GitHub, a company widely known for hosting most open source projects in the world. At work, he is exploring different git technology, GitHub Actions, GitHub security, etc. Previously, he was a Senior Consultant at Red Hat, a technology company known for its Linux server and opensource contributions.

14 Comments

Really liked the article. I just need to ask: C++ is a proprietary language? I tought was open as C. C# I know is proprietary.

C++ itself is a language, not a specific implementation, so there's no source code available for the standard/language itself. Some C++ implementations are open source (e.g., Gnu and Clang).

In reply to by Marcos Alano (not verified)

Nice article Bryant, but I think Travis is SAAS based service not an open source tool (with MIT license.)

You forgot about GitLab. IMO it's the best of all the CI/CD and code hosting platforms (but that depends on your use case), and it's all in a single application. You don't have to integrate 5 different applications to get a good DevOps workflow, you just need to write a simple .gitlab-ci.yml script in the repository root and that's it - the rest is GitLab magic. You guys should really try it - https://about.gitlab.com

Why Gitlab CE is not even mentioned as a CI/CD alternative?

Creative Commons LicenseThis work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License.