The next generation of continuous integration

No readers like this yet.
Open innovation

Traditional continuous integration (CI) systems are designed as a pipeline of jobs. You have a peer review, then the build job, then the unit tests job, then the integration tests job, then the performance tests jobs, and so on.

Each job is triggered by the successful completion of the previous job, and the first job is triggered by a change in the version control source code files. Of course, it can be more complex if you are targeting multiple binary platforms or if you are building a set of components to be able to test your full application.

Continuous Integration flowchart

But what happens when a job fails? According to Jez Humble and David Farley in the reference book Continuous Delivery, you have to first follow this rule: "Don't Check in on a Broken Build." In other words, don't make it worse by pushing other changes. If you do, you won't be able to know the cause of the breakage. Humble and Farley suggest using one of these two strategies to handle breakages:

  1. "Never go home on a broken build," meaning everyone on the team needs to stop working on their current tasks and switch to fixing the problem.
  2. "Always be prepared to revert to the previous revision." Reverting can also be a strategy to avoid blocking the whole team.

Of course, it could also be a mix: timebox the attempt to fix and then revert if outside of the allowed time.

Another way to mitigate this issue is to use an integration branch to merge the modifications only once the integration branch is green (all the tests are OK). With this tactic you have the same problem on the integration branch, but your master branch is always in a usable state.

This could work in small team when you can stop all the teammates from committing but even in small teams, this process often results in a CI that is red for long periods of time. You need to enforce a good discipline to succeed with this kind of CI or you can switch to the new way to do CI.

Next-generation CI

Current-generation CI is centered around the CI server. Changes are detected by the CI server and triggering jobs.

In next-generation CI, the system is centered around the code review system to be able to do actions before a modification is merged into the versioning system.

The decision of whether or not to implement a teammate code review process is up to you. I really recommend doing it to improve the quality of your code, but that's orthogonal to the CI system. Anyway, the important thing to understand is that the triggering of builds and tests are done by a new submission to the code review system. And once all the tests are green, the merge of the code to the master branch happens. This way, your master branch is always green and your developers can work in parallel on their modifications. This new CI system allows you to scale your automation effort and be fluid, as there are no longer blocking issues.

OpenStack workflow

This new approach to CI has been implemented at scale in the OpenStack project to manage the CI of all the different sub-projects. To give you an idea of the scale, every day OpenStack handles 1,000 proposed patch sets, 7,500 posted comments and votes on Gerrit, 16,000 test environments spawned, and 250 changes merged (source).

To implement this next-generation CI system, the OpenStack project uses the following components:

  • Gerrit, the code review and git repository manager.
  • Zuul, the git source repositories gating system.
  • Jenkins, the continuous integration server.
  • Nodepool, intelligent Jenkins slave provisioning on OpenStack clouds.

These tools allow for parallel tests of the same project by using speculative merging strategies. If multiple reviews for the same project arise at the same time, Zuul is able to to test them in parallel by stacking them in a speculative manner. For example, if the reviews were named A, B, and C, Zuul will test A, A+B, and A+B+C in parallel. If they all succeed, it will have been the same as if A was tested and merged, then B was tested on top of the branch (A) and then merged, and the same for C on top of A+B. This speeds up the process a lot when you have multiple contributors to the same project.

Zuul is also able to manage cross-project dependencies, allowing the merging of reviews between repositories. That's key in a git world where your components live in different git repositories.

Try it yourself

For big teams or even for small teams, you can benefit from this workflow for your project by configuring the previously described components. There are puppet modules that can be used to configure these services easily.

Another way is to use our own integration of these services called Software Factory. You will get the following features:

  • A nice integration under a single web menu.
  • A single sign-on between all the services easily and external authentication on LDAP, GitHub or launchpad (cauth).
  • A bug tracking system (Redmine).
  • Collaboration tools:
    • Paste for sharing output or code extracts.
    • Etherpad for collaborative editing.
  • Managed upgrades from previous versions in an easy way.

Because Software Factory is self-hosted (we use Software Factory to develop Software Factory), you can see it in action at

If you want to test drive Software Factory, just follow the documentation at

Keep us posted on what you have achieved using this new way of doing CI!

User profile image.
Frédéric Lepied is a Senior Manager at Red Hat. He has been involved with the Open Source movement since 1996. Contributing to Debian and XFree86, then becoming CTO for MandrakeSoft/Mandriva. He has always written and coordinated projects using Python (rpmlint, lads, msec, pxemngr, edeploy). At Red Hat, his team is contributing to OpenStack and building advanced CI/CD solutions.


Great post, Frédéric. The traditional method is really more of a continuous nightmare. So much better to merge commits to master *after* they are tested, so that master is always deployable.

My complaint with most CI approaches is the lack of inclusion of any manual QA process, UAT or other human review of the changes. Most focus only on automated tests and you can also do code review as you say. But in the real world you need to get feedback on work from stakeholders, and you don't want that testing and feedback to happen after merging to a staging branch or master for the same reasons that you don't want to merge the code before you do automated tests.

To support that workflow we built which is now in Beta. We'd love to get your feedback.

I completely agree with the sentiment here. We've seen a huge adoption of the so-called 'Github Workflow' in the startup community. This workflow is essentially your proposed code review-centric approach and it's something I've written about before… though I've never encountered speculative merging like you describe with Zuul

Thanks for the article Frederic. Rare, I find myself clicking through so many links.

Zuul was the one that really caught my attention. Do you experience many false positives when using this tool? Thanks in advance.

Creative Commons LicenseThis work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License.