Events are an essential component of open source community health. A positive event experience can inspire current contributors and encourage new ones. But how can you tell whether your events are successful?
We at the CHAOSS (Community Health Analytics Open Source Software) App Ecosystem Working Group have considered this question for our events to maintain the health of the communities involved. The CHAOSS App Ecosystem includes several projects focused on developing applications for the Linux platform. While currently dominated by the GNOME and KDE communities, it is not defined by them. The app ecosystem, as it stands today, is primarily driven by volunteers with altruistic goals organized around free software principles. The work we share in this article is an update from our November 2020 article about success metrics for open source events.
We follow the goal-question-metric approach. We first identify several goals that a person or organization might have. We then identify questions we need to answer to achieve those goals. Finally, for each question, we consider quantifiable metrics that provide answers. For event organizers, we established four goals.
Goal 1: Retain and attract contributors
Contributors are the lifeblood of any open source project or ecosystem, and events are a core part of the contributor experience. They are an opportunity to attract new contributors, and they can build and reaffirm relationships that help retain existing contributors. Events should be geared toward creating a harmonious and enthusiastic atmosphere that builds relationships with the project, ecosystem, and community.
One question to consider, then, is how long people who attend events stay with the community. If the value of events is to strengthen the relationships that retain contributors, we should be able to measure that. Answering this question requires long-term data on contributions and event attendance. Fortunately, many projects will already have this historical data.
Another question is what roles event attendees have in our open source projects. Events can often be highly technical and may not attract people who want to contribute in non-coding roles, such as design, documentation, and marketing. If projects have the means to track non-code contributions, which is often tricky, you can correlate with event attendee lists. This information can help event organizers create events that attract a wider audience.
[ Also read Attract contributors to your open source project with authenticity ]
Goal 2: Have engaging events
Events are more than presentations. They create a space for community members to come together. If they don't engage with each other, the event is a lost opportunity. The importance of interactions between members is also evident in the so-called hallway track, the name given to engagement that occurs in the hallways and outside the regular event schedule. Some conferences even have discounted tickets for people who don't plan to attend any sessions but want to engage with others in the hallway track. The question for us is how to measure an event's success in terms of how it fosters engagements between event participants.
To measure engagement during regularly scheduled sessions, we could count the number of questions asked during a Q&A session. While the individual speakers directly influence this metric, there are things event organizers can do to encourage Q&A participation. For example, event organizers can use their morning keynote to encourage everyone to ask questions as a form of showing appreciation.
We also considered engagement during the hallway track. One way to measure this is observing how people behave when not in sessions. Another way to measure this is to ask participants in a post-event survey about their experience outside of scheduled sessions.
Finally, we considered engagement in virtual spaces. One option for measuring this is through the number of social media messages that use the event's hashtag or from people we know are at the event. Another possibility for online or even in-person sessions is to have an emoji that could be collected as an easy one-click reaction to sessions, keynotes, and general conference experience.
Goal 3: Understand company contributions
Companies and other organizations are essential contributors to events, even community events. They may provide sponsorship, sponsor employees to help organize, or simply send employees to the event. It's important to ensure companies find value in their contributions, but we must do this in a way that doesn't alienate community contributions. We looked at what companies expect from community events and what we can measure to improve company contributions.
Some open source events have a reputation for being more corporate than others, so one question you might ask is what percentage of attendees are sent by their employer rather than attending as volunteers. Sometimes, the difference is built into ticket pricing. Otherwise, it's an easy question to ask on a registration survey. Although this information is helpful, it is an imperfect metric because paid open source contributors often attend events outside their job responsibilities.
We also considered several metrics around which companies are attending events, what companies are doing apart from sending employees (for example, financially sponsoring the event), and whether companies repeat their involvement. It's important to recognize this to cultivate long-term relationships.
[ Related read 8 ways your company can support and sustain open source ]
Finally, we considered how competitive the landscape of similarly scoped events is. Companies only have so much budget to spend on events. As much as we enjoy similar events, they draw from the same pool of potential sponsors. Understanding which other events are seeking sponsorship can help organizers better differentiate their own events.
Goal 4: Address diversity and skill gaps
The international communities built and developed across the globe, consisting of people from diverse backgrounds, are an essential component of open source projects. These communities contribute their ideas, collaborate on common goals, and help the project expand and scale toward new directions.
In-person events are a unique chance for projects and organizations to bring their contributors together and allow them to interact and exchange diverse experiences, boosting innovation and accelerating growth from different perspectives. They are also an opportunity to cultivate and reinforce a common culture, focusing on increasing diversity and inclusion. These metrics should measure how well an event contributes to fostering this diversity and closing skill gaps in the community.
We started by considering which skill programs we have at our events and how wide a breadth of skills these programs represent. Skill programs could include tutorials, hands-on workshops, hackathons, or many other formats. Do we have skill programs around skills other than coding? There are many skills and roles valuable to open source projects, such as the skills required to organize events.
It's then helpful to look at which skills we need in our community and which skills are lacking. Project leads and people involved in onboarding are often a good source of this information. By comparing the skills programs we have with the skills our community needs, we can better design the programs for future events.
Get involved
The CHAOSS App Ecosystem working group is interested in working with event organizers to continually refine and implement metrics. The KDE and GNOME event organizers have already discussed changing their events to better capture some of these metrics. Our work for event organizers is also available as a PDF.
The CHAOSS App Ecosystem working group is also taking up the challenge of defining metrics for the marketing and communications functions within OSS App Ecosystems. The first step towards this goal was a conversation with folks from KDE and GNOME fulfilling this role. The conversation is available as CHAOSScast Episode #31: Marketing Metrics for OSS Foundations and Projects. The learnings from this conversation are next translated into goals-questions-metrics, as described above for the event organizer metrics.
After our work on promotions and communications teams, we plan to address metrics for finance teams, community managers, release managers, cross-project coordinators, and mentors. Our work has only begun, and we welcome feedback and new contributors.
You are invited to join the work of the CHAOSS App Ecosystem WG. Find more information on our GitHub page.
Comments are closed.