What is a CI/CD pipeline?

How you define a continuous integration/continuous deployment pipeline depends on your organization's requirements.
85 readers like this.
Plumbing tubes in many directions

A continuous integration/continuous deployment (CI/CD) pipeline is an anchor for every DevOps initiative. The CI/CD pipeline breaks down traditional silos and enables development and operations teams to collaborate throughout the entire software development lifecycle.

Better yet, moving to DevOps and a CI/CD pipeline can help your organization deliver software more securely at a higher velocity.

Breaking down the CI/CD pipeline

There are many definitions of CI/CD pipelines out there, so I always advise organizations to define their own version of a CI/CD pipeline and other DevOps concepts rather than using someone else's. Open source CI/CD tools give you the freedom and options to construct a CI/CD pipeline that meets your organization's requirements.

The stages that form a CI/CD pipeline are distinct subsets of tasks grouped into pipeline stages. Typical pipeline stages include:

  • Build: Developers compile the application code. 
  • Test: The quality assurance (QA) team tests the application code using automated testing tools and strategies. 
  • Release: The development team delivers the application code to the code repository.
  • Deploy: The DevOps team stages application code to production. 
  • Security and compliance: The QA team validates a build based on the project's requirements. This is the stage where organizations deploy container-scanning tools that check the quality of images against Common Vulnerabilities and Exposures (CVEs). 

These are standard stages for a CI/CD pipeline, yet some organizations adapt the CI/CD pipeline model to fit their requirements. For example, an organization building applications for the healthcare market, with its stringent compliance standards, may distribute the test, validation, and compliance gates throughout their toolchain.

Other examples might be an organization that depends on a complex software supply chain with open source software (OSS). Commercial components may institute a gate where development team members generate a software bill of materials (SBOM) for OSS packages or the outside commercial software vendor must deliver an SBOM as part of their contract deliverable.

Barriers to CI/CD pipelines

Implementing a CI/CD pipeline changes a team's processes and culture. While many developers are receptive to automation of some tasks and testing, people can be a barrier to CI/CD adoption.

Moving from a waterfall process to CI/CD can shake up the fundamental and implied power structure in some organizations. Since a CI/CD pipeline increases software delivery velocity, the "gatekeepers" of your old manual process may feel threatened by this change.

Integration opportunities

The open source roots of the tools comprising a CI/CD toolchain create opportunities for some exciting integrations as you achieve greater DevOps maturity levels in your culture, processes, and tooling.

Analyst firm Forrester predicted in 2020 that just-in-time learning will join the CI/CD pipeline. It makes sense if you think about it. It makes even more sense in the current era of remote work and even for remote onboarding of new employees. For instance, an organization could integrate a documentation wiki into its pipeline with documentation of internal processes.

A more ambitious organization can integrate a learning management system (LMS) such as Moodle into its CI/CD pipeline. It could use the LMS to publish short videos about new DevOps toolchain features that developers need to learn as they onboard or when tools are updated across the pipeline.

Some organizations are integrating group chat and other collaboration tools directly into their CI/CD pipelines. The chat platform serves alerts and enables collaboration and communication among teams. Integrating Mattermost, Rocket.Chat, or another enterprise chat platform into your CI/CD pipeline requires upfront planning and analysis to ensure that pipeline users won't drown in alerts.

Another integration opportunity to explore is building analytics and advanced reporting into your CI/CD pipeline. This helps you harness the data that courses through your pipeline.

Final thoughts

The CI/CD pipeline is foundational to DevOps. And open source makes it adaptable and flexible to new requirements resulting from operational changes you implement during your DevOps journey.

I hope to see an open source response to the unified DevOps platform trend, in which organizations seek an end-to-end CI/CD solution. The makings of such a solution are out there. After all, GitLab and GitHub trace their platforms back to open source roots.

Lastly, don't forget the education and outreach underlying every successful CI/CD toolchain. Documenting your toolchains and accompanying processes will improve developer onboarding and ongoing DevOps team training.

How do you and your organization define your CI/CD toolchain? Please share your feedback in the comments.

What to read next

What is CI/CD?

Continuous integration (CI) and continuous delivery (CD) are extremely common terms in software production. But do you know what they really mean?

User profile image.
Will Kelly is a product marketer and writer. His career has been spent writing bylined articles, white papers, marketing collateral, and technical content about the cloud and DevOps. Opensource.com, TechTarget, InfoQ, and others have published his articles about DevOps and the cloud. He lives and works in the Northern Virginia area. Follow him on Twitter:@willkelly.


Perhaps also interesting is that there is no strong consensus on what the term "pipeline" means between different CI/CD systems. The oldest explicit use of "pipeline" I'm aware of by a CI/CD system is in Zuul, which has been using it since its first release 9 years ago: https://zuul-ci.org/docs/zuul/reference/glossary.html#term-pipeline

In short, because it was designed to be a queuing/sequencing "project gating" system, it used "pipeline" to refer to a collection of queues which provide context for a particular outcome (so from your example testing, building, and deploying would each be different pipelines configured in Zuul). The primary example is dependent queues built up from change approvals, where each change enters a "gate" pipeline awaiting the predicted merge state for the changes on which it depends or those approved ahead of it for related projects to reflect reality.

The Interoperability Special Interest Group within the Continuous Delivery Foundation (a fund of the Linux Foundation) has been attempting to survey the discrepancies between this and other related terminology here: https://github.com/cdfoundation/sig-interoperability/blob/master/docs/vocabulary.md

I should have also pointed out that Zuul's use of "pipeline" (much like many of its other terms of art) are drawn from processor design. Zuul pipelines changes in order to perform predictive parallelization, much like how a CPU pipelines operations to achieve speculative execution.

In reply to by fungi

Creative Commons LicenseThis work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License.