As software professionals or enthusiasts, we're overwhelmed by choice when it comes to technology components. Just looking at the message queue space, you can choose from Kafka, RabbitMQ, ActiveMQ, HornetQ, and more, or use a managed service from one of the large cloud providers. With all these options, how do we begin to know which will be best for our use cases?
This is a topic that has been itching at me for a while, and it bubbled up during an Apache Kafka "Birds of a Feather" session I participated in at a recent PyCon. It's also a topic I've been able to experiment with and refine my thinking on in my career as a professional developer. I even addressed it as part of a talk on how to make the right choices for your current scale without sacrificing your dream of being the next Google.
Here are high-level criteria to consider when evaluating open source software components. After I cover open source, I'll add a couple of more points specifically about what it means to pay someone else for a managed or software-as-a-service solution.
Open source evaluation criteria
1. Project age and maturity
How established is the project you're considering? Is it still in beta (or alpha?), a decade old and getting ready for its third major version release (Hadoop!), or somewhere in the middle?
The weight you should give to this criterion varies with how critical a role this project will play in your architecture and how much code is involved. For example, I'd be extremely hesitant if someone suggested a brand-new database technology for the primary user storage backend. For a JavaScript library to render an image carousel on an A/B-testing variant of one page, something newer is probably okay to experiment with.
2. Maintainership
Almost hand-in-hand with a project's maturity, current maintainership is pretty much non-negotiable. Why adopt something you know won't be seeing reliable updates?
In most cases, you want an open source project that has grown beyond the original company or individual that pioneered it and has established support from a wider community. Large enough projects will eventually evolve their own foundations to support them—the Django Software Foundation, to name one—but if this isn't the case, look for the support of groups like Apache, the Linux Foundation, or the Software Freedom Conservancy.
3. Release velocity and stability
While on the subject of maintainership, consider how quickly a project is moving. How frequent are releases? Is there a regular cadence? Maintaining a fork on your internal package server for months while waiting for upstream to integrate bug fixes and do an official release can add up to a lot of overhead.
When releases happen, is it easy to tell what's changed? Does the project use semantic versioning or a similar mechanism to indicate whether breaking changes are present? Are long-term support (LTS) branches available that receive backported security and stability patches from the mainline development releases?
4. Security
At a bare minimum, search the Common Vulnerabilities and Exposures (CVE) database for the projects you're considering. Questions to ask: How many vulnerabilities have been reported? Have they been resolved, and how long did it take? How difficult will it be to implement workarounds while waiting for patches? When those patches arrive, how disruptive will it be to apply them (e.g., uploading a new asset bundle to your CDN or rebooting your entire database tier)?
If, like a lot of us these days, you work on web services, make sure to become familiar with attacks and mitigations documented by the Open Web Application Security Project and understand how adopting new code or systems into your environment changes your threat model.
5. Standards and project ecosystem
Does this project adhere to well-known standards for its conceptual space, if they exist? It'll be easier for you and your team to integrate a new search backend if it speaks HTTPS and JSON instead of a bespoke binary protocol.
Are libraries available in your implementation language(s) of choice, or would you be writing "glue" code by hand? Keep in mind that if a library for your language doesn't exist, this can definitely be worth the tradeoff for a project that meets all of your other requirements. It is also a great way to get involved and contribute back to a community!
Are developers aware of and familiar with this project? If you need help, will it be easy to find people who know about your chosen technologies and can hit the ground running?
6. "Out-of-the-boxiness"
This is a wonderful phrase that my friend and fellow conference organizer Josh Simmons used in a great talk on why a new developer might choose Django. It means, roughly, what your first half-hour with the software is like.
How hard is it to install on your laptop and run the sample project? Does it have persnickety requirements that will make it difficult to run in your continuous integration environment? What about putting it into production—are the default configurations reliable and secure, or are you going to spend weeks tuning variables?
7. Documentation
Good documentation is definitely an art, but there are basic requirements for any project:
- First of all, documentation should exist.
- It should be searchable and discoverable.
- Docs should be kept up-to-date, and outdated/deprecated versions should be obviously flagged as such.
8. Support and contractors
For major components of any large endeavor, look into whether a consulting and support ecosystem exists. If you're a small shop that can't afford a full-time database administrator (DBA), having a contract with a DBA firm might make all the difference when your relational database dies at 2 a.m., and you discover all your backups from the last month are corrupted.
9. Licensing
Finally, you should be aware of the license(s) a project is released under and whether they're compatible with your own. Last year, Facebook relicensed React in response to community concerns over the compatibility of its licenses with other open source projects. Developing for iOS or MacOS? Keep in mind that GPL-licensed software is not compatible with the terms of the Apple App Store, so anything you use will need to be under another license.
Software-as-a-Service evaluation criteria
Many of the above concerns for open source software also apply to using managed or hosted solutions. There are a few more items to think about specifically for these cases, though.
10. Cost tradeoffs
The obvious component of the cost of a subscription-based software package is the monetary piece. Analyzing this is an article unto itself, but the most important element is to find a standard unit of cost (e.g., cost per request processed) to compare offerings that may have different pricing models.
Beyond the immediate sticker price, though, are other costs:
- How much time would it cost your team to build the functionality in-house or piece together existing solutions and learn how to operate them?
- What do you lose from your feature-development roadmap while your team is busy working on this, and can you afford not to have it tomorrow?
11. Spontaneous Massive (Vendor) Existence Failure
This is pretty much what it says on the tin—if your project relies on a third-party service, what is your impact if the vendor that provides that service suddenly goes out of business?
How to use these criteria
These criteria are by no means an exhaustive list of what to think about when considering software components. I encourage you to pick the ones that make sense for you and your project and to add others specific to your situation.
For choices that affect a small scope of work, it's fine to think about them holistically and keep them in the back of your head when checking out a new piece of software, but for larger decisions it's worthwhile to keep track of the criteria in a spreadsheet and rate candidate implementations numerically to derive overall scores before making a decision.
Happy hunting!
There are myriad open source projects available for just about every component of a modern software stack—the array of choices can be dizzying, especially when starting from scratch or making many choices at once. With the above criteria in mind, however, you should be better equipped to think rationally about your needs and how each of your options might or might not suit them. Happy hunting!
2 Comments