Screwdriver: A scalable build platform for continuous delivery

In 2012, after experiencing performance and scaling limitations with the CI/CD solution we were using, my team at Yahoo started searching for another platform. We didn’t find a solution that met our rapidly growing developer needs, so we built Screwdriver. It is the product of our CI/CD journey at scale, lessons gathered along the way, and enormous amounts of feedback from our internal developers. After seeing the value Screwdriver added, we decided to open source it in 2016 to share our solution with the global DevOps community.

Screwdriver is a build platform designed for continuous delivery at scale. It supports an expanding list of source code services, execution engines, and databases since it is not tied to any specific compute platform. Screwdriver has a fully documented API and a growing open source community. A Screwdriver demo is available here.

Highly customizable

Screwdriver’s architecture allows you to use pluggable components under the hood to swap out pieces that suit your infrastructure. A Screwdriver cluster admin can swap Postgres or MySQL for the datastore or use GitHub, GitLab, Bitbucket, or a combination of them for source control management (SCM). You can also dynamically select an executor engine based on the needs of each build pipeline. As an example, you can send Golang builds to the Kubernetes executor while your iOS builds run in a Jenkins execution farm.

Pipeline as code

With Screwdriver, you define your delivery workflow in a simple YAML file. Screwdriver supports advanced workflow features such as:

• Branch filtering: Creates workflows with branch-specific jobs.
• Parallel and join: Workflow where jobs fan out of a single job and then fan into another job. Creates multiple parallel workflows for a pipeline.
• Remote triggers: Builds interconnected pipelines. Triggers a workflow for building an application when one of its dependent libraries has been built.
• Blocked by: Prevents builds from running concurrently. Blocks a deploy job from running when a test job is already in progress.
• Detached jobs and pipelines: Defines jobs that are not tied to any SCM changes. These can be started manually and will use a meta set from a past event.

Sharing best practices via templates and commands

Screwdriver enables expert users to define templates, which define reusable jobs. For example, if you are passionate about how Golang applications should be built, tested, and deployed, you can create a template for Golang that all users of the Screwdriver cluster can include in their pipeline YAML config.

Templates allow an organization to encapsulate the best behavior on how an application should be built and deployed. Example templates are available to browse in the open source Screwdriver instance.

A template is to a job what a shared command is to a step. Templates allow users to share common job configurations, whereas shared commands allow users to share common build steps. For example, a step to build an RPM package or to set up Docker in your build environment can be packaged as a shared command. Example commands are available to browse in the open source Screwdriver instance.

Security

Pluggable build steps allow cluster admins to set up a build environment with authorization credentials such as SSH certificate authority. Built-in secret management allows users to store build secrets securely. Screwdriver also provides a VM-based executor to provide further build isolation. Access tokens allow user- or pipeline-level scoped programmatic interactions with the Screwdriver API.

Getting started

Start exploring Screwdriver today by running the command below in your terminal to bring up a cluster locally.

python <(curl -L https://git.io/screwdriver-box)

This command will run a script that will create a Docker Compose file locally, complete with OAuth credentials using a generated JSON Web Token and a user-provided OAuth Client ID and secret. If you choose to do so, Docker will then pull the Screwdriver API, UI, and log store images to bring up an entire Screwdriver instance locally for you to play with. All data written to a database will be stored in a /data directory.

For more configuration details, explore the SD-in-a-box documentation. To set up a production environment, please visit the documentation for cluster management.

Join the Screwdriver community

If you’d like to learn more about Screwdriver and contribute to CI/CD at scale, I invite you to explore the project on GitHub and chat with us on Slack.

Jithin Emmanuel will present CI/CD with Screwdriver at the 17th annual Southern California Linux Expo (SCaLE 17x) March 7-10 in Pasadena, Calif.