How we develop success metrics for open source events

The CHAOSS App Ecosystem Working Group is establishing ways to measure how well virtual and in-person events meet their goals.
55 readers like this.
3 warning flags of DevOps metrics

As many open source communities grow larger and older, they may face problems managing members' engagement. People turn towards metrics to understand large systems and prioritize resources, but there has not been a consensus set of metrics for understanding open source communities.

The Community Health Analytics Open Source Software (CHAOSS) project, a Linux Foundation-sponsored community of industry professionals and academics, is working to solve this problem by defining metrics for open source projects. The CHAOSS project focused first on developing metrics for open source event organizers because open source communities often depend on events like hackathons, meetups, conferences, and user group meetings to grow their communities and work on important project updates.

The CHAOSS App Ecosystem Working Group, which developed the metrics for events and worked together on this article, is comprised of members who have organized and participated in many open source events. While we developed these metrics to help the GNOME and KDE communities, we believe they can help other open source event organizers create their own metrics strategy. We also hope that more people will be inspired by this work and join us to improve it.

How we began

The CHAOSS App Ecosystem Working Group emerged from the SCALE 18x conference in March 2020, when CHAOSS and GNOME members began talking about how to measure the Linux App Summit (LAS) community, an event co-hosted by GNOME and KDE. 

We realized that we did not have sufficient data to measure our progress towards LAS's goals, and that hindered our ability to make important decisions to support those goals. For LAS (and the whole Linux app ecosystem) to thrive, we recognized the need to be more data-driven in our approach. We also realized the importance of extending this mindset to other similarly organized communities.

From our experience at CHAOSScon, we found opportunities to partner with the CHAOSS community around open source metrics. Therefore, we organized ourselves under the CHAOSS project, which provides a platform for working on metrics.

We chose the name App Ecosystem Working Group because one of the GNOME and KDE communities' primary goals is to grow the Linux app ecosystem. Although we had the GNOME and KDE communities in mind at the start, we are creating the metrics to help everyone involved in the effort to grow the Linux app ecosystem.

We chose to begin with the event organizer use case because of the nature of events: Events tend to have clearly defined boundaries, with a set start and end date, a list of registered attendees, and activities like talks, presentations, and workshops. This clarity does not generally exist in open source projects and communities, where the various projects, contributors, and users are less often strictly defined.

The GNOME and KDE communities both had events coming up in 2020, so we believed we could iterate faster using feedback gathered from the events we organized, separately and together.

Approach to data gathering

The CHAOSS project uses the goal-question-metric (GQM) approach to gathering data. One problem with metrics is that often we try to measure everything without understanding how the data will be helpful. But data should help inform decisions. Therefore, the GQM approach starts with a list of high-level goals that are important to a particular persona.

The goals are the community's strategic objectives. For each goal, we look at questions that help determine whether we are achieving the goal. Metrics then provide quantitative information that helps answer those questions.

Before diving into metrics, the App Ecosystem Working Group brainstormed goals and the personas that those goals affect. We collected 17 different goals, spread across seven stakeholder personas, that a community or project ecosystem might have.

We collected and curated these goals from community members' experience in major open source projects like GNOME and KDE. Although each community is different, we think most of these goals are relevant to and will help a wide variety of open source projects.

Metrics for event organizers

The full list of metrics for event organizers focus on three goals that event organizers care about:

  • Attracting and retaining contributors
  • Understanding company contributions to an event
  • Ensuring events contribute towards diversity goals and closing skill gaps in the community

We came up with 19 questions that event organizers have about how well they are doing in reaching these goals. The metrics are directly linked to these questions to give the event organizers actionable insights.

Privacy implications

While this work was developed by and for the GNOME and KDE communities, we hope that the metrics will help other open source communities.

While GNOME and KDE have not implemented the metrics yet (and do not yet have a timeline to do so), one known limitation is privacy concerns around data collection. Every geographic location has laws that may limit data collection, which open source and free software projects must take into account.

Once GNOME and KDE implement these metrics in their communities, they plan to share the legal considerations they took into account.

Join us

We are hopeful that these metrics will be useful to open source event organizers interested in integrating virtual and in-person events into their community-building and engagement strategies.

If you'd like to participate in developing these metrics, we welcome your experience, insight, and voice! The CHAOSS App Ecosystem Working Group meets biweekly; please join us.

You can also communicate with us asynchronously through our GitHub repository. For more information on other ways to contribute or connect with us, we encourage you to visit the CHAOSS community participation page.

What to read next
User profile image.
Georg Link is an Open Source Strategist. Georg’s mission is to make open source more professional in its use of community metrics and analytics. Georg co-founded the Linux Foundation CHAOSS Project to advance analytics and metrics for open source project health. Georg is an active contributor to several open source projects and has presented on open source topics on many occasions.
User profile image.
Nuritzi is currently the Sr. Open Source Program Manager at GitLab. Previously, she was a founding team member of Endless, a company making Linux-based computers for users with little to no internet access, and she is a former President and Chairperson of the Board at the GNOME Foundation.
User profile image.
Community and Engagement Team member for the GNOME Project. Community manager for oneAPI.
User profile image.
I enjoy connecting the dots between teams and projects, products and communities. I thrive working in nimble teams, environments with diversity and projects developed in a forward-looking mindset. I am a long-time contributor to international Open Source software projects, and a founder of Open Data/Government initiatives in my home-country, Cyprus.
User profile image.
Shaun McCance has a long history of doing open source development, documentation, and community leadership. He currently works for Red Hat as the CentOS community architect, and tries to find time to keep working on GNOME.

Comments are closed.

Creative Commons LicenseThis work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License.