We are all obsessed with the numbers and statistics we can measure in our lives. We are concerned about our health, so we monitor our weight, blood pressure, and calorie intake. We also observe ourselves and our work environments to evaluate our efficiency and team dynamics. This mindset of focusing on the numbers carries over to how we evaluate open source communities.
Why are metrics important?
Open source communities, like the human body, are complex organizations with commonalities as well as unique operational characteristics and dynamics. By their nature, open source projects make a lot of data available, not just related to the source code, but also about contributors' processes and actions. This information gives us a better picture of a project's ecosystem and how it changes over time.
When evaluating community health and progress, communities typically look at metrics on contributions, diversity, and adoption of the artifacts they produce. Metrics can also be very helpful to find bottlenecks and identify changes in the balance of the ecosystem. Metrics can provide insights into community health, growth, and overall dynamics—but only if we use them wisely.
Why looking beyond the numbers is crucial
Although metrics are widely used and essential to understanding the community, being careful about how we use the numbers is important. There are no magic "healthy" numbers in open source community metrics. In fact, numbers can be misleading unless you look further into the details and context. You could get an incomplete picture, for example, if you only count code contributions and overlook valuable documentation and tests in other parts of repositories.
Finally, repeatedly collecting and publishing the same metrics can cause people to try to game the system, resulting in unhealthy community behavior. Judging a community's health purely on numbers could lead to false conclusions and inappropriate follow-up actions, so how can we do better?Case study: Code reviews
Code reviews are highly encouraged in both corporate environments and open source projects to identify and fix problems before they go live. Code reviewers learn the most about the code and changes in the software, and project maintainers rely on steady contributors' opinions before merging new changes. So how do metrics come into the picture?
Measuring the number of positive and negative code reviews over a specific period (e.g., a quarter of a year or per release cycle) is easy. Many open source projects publish these activity metrics with options to filter results on things such as data about one contributor or all contributors working for the same company.
Although the tools used by open source projects are accessible by anyone (which means anyone can extract the numbers), publishing these metrics on a dashboard may lead to gaming them over time. For example, people may try to have the most reviews, thinking it will speed up their acceptance to the community, or companies may encourage employees to generate higher numbers to improve their reputation with customers.
The unfortunate consequence of trying to increase these numbers quickly is that the quality of the code reviews drops. One example is a negative review in which the reviewer just repeats what the automated testing system pointed out. Another is a reviewer simply saying he or she agrees with previous reviewers, which adds nothing to the discussion. Or even less helpful, the reviewer merely adds a "+1" mark (which means the change looks good) on as many open changes as possible with no meaningful comment.
There are multiple problems with these behaviors. These meaningless reviews are disturbing for active contributors trying to help code authors get the highest quality changes merged. Not to mention that people who aren't trying to help maintain the project, rather trying only to boost their statistics in the open dashboards, are annoying to regular contributors. Also, people who abuse the system like this are easy to recognize, and often their reputation goes down once they are identified.
How to use metrics better
Education is important to address these challenges. The success of an open source project depends upon a group of people who care about the topic and the technology working to maintain the source code, tests, and documentation. Metrics are important to get an overall picture of the balance of the ecosystem, which is not driven by any single metric, rather a combination of multiple key performance indicators (KPIs).
When we look at metrics, like the number of code reviews, we must always look beyond the number itself and understand how we can use the data for growth and reflection on whether we are moving in the right direction.
We need to ask key questions to identify which metrics we should look into and how to combine them to gain meaningful information. For example:
- Why is a data point important for us (or our managers)?
- What does it mean to have a higher or lower number?
- And what do changes over time say?
Or to look back at an earlier example:
- What does the ration of negative and positive reviews mean?
Measuring only a single set of metrics and thinking only the numbers matter is a bad idea. Instead, dig deeper and look behind the numbers.
Ildiko Vancsa (ecosystem technical lead at OpenStack Foundation) will give a talk with Ray Paik (operations manager at Linux Foundation) called What We Learned from Metrics as Community Managers at the Open Source Leadership Summit in Sonoma, California, March 6-9, 2018.
Comments are closed.