What if you treated all of the configuration files and scripts which set up and control your infrastructure like an open source project in their own right? What if you put the code out there for review and allowed developers and operators to work in tandem to ensure a smooth roll out of code to production? That's how the OpenStack Infrastructure team at HP work.
Elizabeth K. Joseph is an Automation and Tools Engineer at HP, who works on the OpenStack Infrastructure team running a totally open source infrastructure built for OpenStack development. In her upcoming talk for All Things Open, she outlines the tools her team uses, how doing their work in the open affects their operations, and what other organizations seeking to make their workflow more open can learn from HP's experience.
Tell us a little about yourself. How did you get started working in systems administration?
I've been using Linux on my desktop since 2002 and pretty much from the beginning I was going to Linux Users Groups and collaborating with other users. I was drawn to Linux and Open Source first by the ability to customize (you had the source code!) and then the amazing community where so many people are passionate about sharing their work.
Around 2006 I started doing work for a local (to Philadelphia) tech services provider doing racking of servers and on site Linux installs. Eventually I was hired full time as a junior systems administrator for them. We were focused on Debian-based deployments, but were often also a go-to source for network debugging and other troublesome tech problems our customers brought to us. As my work progressed there I started working on bigger deployments and Highly Available clusters using pacemaker and heartbeat, along with a lot of virtualization tools. This is really where I grew my passion for systems administration, it's exciting to build things from open source tools and have so many options at your fingertips - plus my employer was enthusiastic about us contributing back upstream so I was able to make contributions to Debian during my tenure there.
Over the years I got involved in several open source projects, most notably Ubuntu, which lead to being awarded an O'Reilly Open Source Award in 2012 and co-authorship in the latest (8th) edition of The Official Ubuntu Book which came out in July.
The move into working for HP on the OpenStack Infrastructure team to support OpenStack developers came for a desire for a path that would combine my passion for open source with my systems administration work, so here I am!
It seems like there are a lot of related tools which perform similar tasks when consuming OpenStack. How does the Infrastructure team choose which is the right tool, and how do they support downstreams who make different choices?
The OpenStack Infrastructure team combines several open source technologies like Git, Gerrit and Jenkins, along with some home brewed open source projects to provide a robust Continuous Integration system for the high velocity environment within the OpenStack Project. From there, the team consumes donated cloud VMs from multiple OpenStack providers (currently totally about 800 machines across 2 public clouds, plus some bare metal testing on 2 private OpenStack clouds). Since our tools are connecting with multiple clouds with slightly different configurations, we try to stick to the basic OpenStack commands in our tools so we are sure to support most OpenStack-based clouds that adhere to best practices.
We also use a variety of other popular open source tools to support the developer community, from MediaWiki for our wiki to Cacti for keeping track of how our servers are doing over time. These are generally selected by one or a small team of project members who understand the project needs and then evaluate the options available. For instance, last summer I went through a pretty thorough review of git web front ends before we had a meeting about my findings and settled on the use of cgit.
When it comes to configuration management, we use a mix of Puppet and Ansible to drive our infrastructure. While it has treated us pretty well, Puppet was largely an arbitrary choice when the project really came together and it just as well could have been Chef at the time.
As our downstream community grows, specifically those using our Continuous Integration system, we're working this cycle to make this less OpenStack-specific and split out segments of our infrastructure to make it easier to consume. Our mandate is still support of the OpenStack project though, so our energies are primarily focused there. We only really provide our source and ask down-streams to submit patches, for instance, we don't use LDAP or extensive proxying in our infrastructure, but we have down-streams who do and so we're now carrying their patches to support those things.
What's exciting to you about working with the OpenStack infrastructure team?
The blend of two of my passions, open source and systems administration, has made this a wonderful job for me. I also have the opportunity thanks to HP to travel to conferences and share the work that we're doing, from talking about how we open sourced our entire Puppet configuration to lessons we've learned as we run a fully open source infrastructure out on the Internet. It's also been great to connect with contributors to other open source projects who have also open sourced all or part of their infrastructures, including Debian, Fedora, Mozilla and Jenkins.
There's been a lot of buzz around DevOps changing the way code is managed and deployed, but there are still a lot of organizations still struggling to adopt a more DeveOps approach. What do you think is holding them back?
I think there are two major barriers. The first is that it requires a lot of learning on the systems administration side, I've never had to write so much code in my life! That's a big change, we're no longer simply putting together the pieces and taking what's given to us to deploy, we're playing an important role in developing the pieces and that can be a tough switch.
Then you have management buy in. DevOps requires trust and freedom to your Ops team, and that can be tough to give, particularly in a larger organization. DevOps moves fast and so having something like a huge change review process that's put in place by well-intentioned people can throw a real wrench in things and make the switch pretty much impossible.
"The Phoenix Project" is pretty much the DevOps novel that dives into more of these challenges and how a fictional "old guard" systems team turned their organization around. Highly recommended.
Without giving too much away, tell us a little bit about your upcoming talk at All Things Open. What can attendees expect to learn?
I'll be talking about Open Source Systems Administration. In my talk I'll be covering the infrastructure we have in the OpenStack project to give some background, explain our usage of a code review system for our systems administration changes so we can safely get contributions from across the OpenStack community, and how we've managed to build a cohesive team of remote systems administrators by doing all of our work online, in the open.
Our case is an extreme one as an open source project, but the lessons we've learned are also valuable to companies who are considering opening up their systems administration repositories to others in the company. What if developers at your company could propose changes to your CI system? Or how the internal wiki is configured? You all of a sudden have a lot more people who can help out and solve pain points themselves and who don't have to wait for you to get to their ticket. It's really empowering for the whole organization.
Why does open source matter to you? Why is it important that administrators, and not just developers, contribute code back to the community?
I live in San Francisco now and am on the board of a non-profit that puts computers into schools that lack them (Imagine that! Just miles from Silicon Valley!), and in 2012 I spent a couple weeks in Ghana deploying Ubuntu-based desktops to schools there. This work would not be possible without open source. Both of these initiatives are run by relatively small teams at non-profits that only have limited (or no) grant money and donated computers, if we had to pay for proprietary software every year to support these kids it would be impossible. Beyond the work I do in schools, OpenStack has really taken off in Asia where the lack of licensing costs associated with it has created a boon of developer power from that part of the world with major companies investing in development. I believe open source software really has the power to equalize and provide opportunities to those who may not have had the resources otherwise.
As for contributors, administrators use the software so it's continually important to have their collaboration and feedback so that developers can make the product they want to use. The OpenStack Summit has done a really clever thing of putting both the User Conference and Developer Summit at the same time and location. Although sometimes a controversial decision due to the week long limit, it gives an opportunity for developers and administrators of the software they build to be in the same place at the same time and learn more about usage and needs.
See the full series of All Things Open 2014 speaker interviews.
Comments are closed.