I was stationed at a small base outside of the Hanscom U.S. Air Force Base in Massachusetts from 2008 to 2012, where I was responsible for the majority of the Air Force's software programs. My job was to take the existing command center, a behemoth known as the Air and Space Operations Center (AOC), and modernize it.
For some context, the AOC is a combined data and operations center. There are about a dozen of these major centers scattered around the world. Each is responsible for planning the Air Force operations in a specific area of the world.
The AOC grew up out of a pick-up game of sorts between the users (Air Force personnel) and the vendors (commercial and government programs). The users simply bought what they wanted, and the vendors happily took their money and installed the systems. The result was a collection of standalone systems; each came installed with its own hardware and software, and there was very little sharing of resources between them. While the team made it work through sheer willpower, it was horribly inefficient, a maintenance nightmare, not user friendly, and agility was measured in decades.
Our job was to take that mess and fix it. The idea was to build a standard hardware and software platform, then migrate the legacy applications to the new platforms. At the same time, we needed to define those platforms as the target platforms for any new software projects. So we started looking at various options, including JBoss, WebSphere, WebLogic, GlassFish, and others. At a minimum, we wanted a standard application server and enterprise service bus.
This was my first trial-by-fire experience that showed the true resistance within the Air Force and Department of Defense (DoD) to open source software.
It is important to understand that at this point in time, open source software was not popular in the U.S. government, and it was often misunderstood by key decision makers. The DoD has a policy that everything purchased must be supportable, which is a good thing. For software, this means there must be a way to fix bugs and other issues—especially in regards to security—in a timely manner. Traditionally, this meant using proprietary software that had expensive annual maintenance contracts. This is the model that I was trying to change.
I wasn't alone. There were several outstanding engineers and program managers at Hanscom who were also attempting to incorporate open source into their programs. It was clear to us that open source was the right way to go. We just didn't like the idea of being locked to a vendor that could charge whatever they wanted in the future, or worse, abandon their product on a whim and leave us without support. Our software was used for critical operations, and we thought the risks of proprietary software were too high given that there were open source alternatives.
I wanted an open source solution and faced a fair amount of resistance from our lawyers, management, users, and proprietary vendors. It was a difficult struggle at times, and it wasn't until the DoD published their first official guidance on the use of open source software that we started to gain traction. Finally, in the middle of all of the drama, the DoD leadership issued a policy update explicitly stating that open source software was acceptable as long as there was support for it, and that the support could come in the form of government programmers, if necessary.
This memo was a game changer, but it took more than just a policy update to get momentum to shift toward open source.
What were the arguments against open source?
Lawyers: "Department policy requires that software has technical support."
Before the DoD's clarifying guidance on open source, there was a great deal of debate over whether open source could be considered commercial off-the-shelf software for procurement purposes. To avoid any perceived risk, the legal department required that all software purchased came with technical support. Further, the lawyers wanted the company behind the software to offer this support. We had to explain that not only were there companies behind open source products, but that we could do the support ourselves if needed—something we couldn't do if a proprietary software company went out of business.
Management: "There's too much risk, even if I can't explain why."
We explained that all software has some risk associated with it, but we'd be able to pick open source software with less risk than a commercial equivalent. We were able to argue that open source software may have less risk because many more programmers look at the code, and that the quality can be easily seen. Many of the largest commercial companies have relied successfully on open source software for their most critical needs, and we could too. It wasn't about choosing open source software for philosophical reasons—it had to do the job and do it well.
Users: "We are comfortable with the vendors we have."
It is perfectly reasonable to stay with what is working. However, when you simply don't have the budget to continue to buy and support all of the proprietary software on board, it's time to consider open source software. Open source solutions may not always have all the bells and whistles of their proprietary equivalents, but the software works just fine for how the users are actually using it.
Vendors: "Ours is better!"
In some cases, that was true. However, the real question to ask in many cases is, "Which software solutions are good enough?", not, "Which are the best?". The very best options are often too expensive to be used throughout an enterprise. We strived to maintain the rule that if a proprietary piece of software met the requirements and there was not an open source software counterpart, we would be willing to pay the bill and go with the proprietary solution. However, many had similar open source options.
Why did I fight so hard?
First of all, I believe that when government pays for something, we owe something back to the public. Investing in open source invests in something that the public can use without paying twice for. I can't in good conscience pay a proprietary vendor to modify their system for us so they can sell that improvement to others, although I've had no choice but to do exactly this in my career.
It is often a matter of maintaining operational capability. I could not take the chance that our AOCs would have to shut down operations because one of its licenses expired. I had one software vendor demand that we stop operations because of a perceived license violation. (We were in compliance.) We had another vendor whose licenses were so expensive that we kept a pool for all of the AOCs. We had to decrease at one AOC to increase at another, so we had to know ahead of time when a crisis would break out and where in order to reallocate the licenses (which took a minimum of three to four days).
My reason for fighting for open source wasn't just about licensing. If a commercial software solution did not work like we wanted it to, we had to work with the vendor to update the software. This assumed that we had sufficient clout to get a change made, which often was not the case. With open source software, we had the ability to fork the code and make the necessary changes if it didn't work.
I fought for open source because we needed to create an environment of innovation and agility. This was where most of my passion came from.
We have over 46 major applications in the AOC and hundreds of smaller, specialized applications. Traditionally, it could take up to five years after you finished developing your software for it to be installed in the AOC. That is a lifetime in software, and I wanted to bring that down to something actually useful for the operators.
My plan was to pass out the AOC's Platform-as-a-Service (PaaS) to potential developers along with what would effectively be an AOC development kit. It would include APIs and test stub services so others could virtually integrate with software they didn't have on hand. In fact, I wanted to host this on the internet and let companies download it. My hope was that before a company spent the business development dollars flying up to meet me and knock on my door, they would be able to see for themselves if their software would work in our environment.
So how do you do this if your PaaS is proprietary? I could pay for an enterprise license, I could use an open source solution, or I could force all of these companies to buy the proprietary PaaS (not easy for the small software shops I wanted to target). So, the easy answer was open source. (And even that had some challenges.) Once a company could show that their software worked with our reference platform, we would have a serious discussion on what it would take to formally integrate them into the program. After the software passed that first hurdle, the fielding time was reduced from years to months.
Higher-level politics and funding issues derailed my efforts, and the entire modernization program was delayed. Thankfully, the open source PaaS program we started survived and is now the basis for several different application developments that will eventually replace major AOC applications. Had I started this effort today, I would be pushing OpenStack, OpenShift, and Docker instead of a semi-customized solution.
8 Comments