As I looked around the 2014 Grace Hopper Celebration of Women in Computing career fair (PDF) floor, I stopped by the Pinterest booth and learned that open source software plays a big role at the company. And even better, Pinterest now plays a big role in the world of open source software, too.
After the career fair, I followed up with the technical lead on the Pinterest Growth engineering team, Ludo Antonov. Ludo says that the Growth team builds the features, core products, and systems that directly help sustainably grow Pinterest to billions of users worldwide. In this interview, he explains the roles open source software plays at the company.
Which open source technologies does Pinterest rely on, and how are these used to power the site?
Open source technologies are at the core of many critical systems powering Pinterest. To name a few, our data reporting pipeline is powered by Hadoop/Hive, we heavily rely on MySql and HBase for data storage, our caching is reliant on Memcache and Redis, and our log collection is done via Kafka. Additionally, our web platform build system runs on Node and we are using Twitter's Finagle. That's but a portion of the complete list of open source software we use.
I hear that Pinterest is releasing some software as open source. What are the specifics of the code you are opening up?
Our open source program is relatively new. We've been very lucky to be able to learn from companies that have a rich history with open source software. We formed an open source committee and set up the framework for open sourcing projects at Pinterest only four or five short months ago, although our contributions date back a couple of years. Since then, we've been able to release two very interesting projects, Secor and Bender, both of which sit on our public GitHub repository GitHub.com/Pinterest.
We open source projects that we ourselves internally have heavily adopted and use in production. This helps us guarantee that the projects we put our engineering brand behind have high quality and that there will be continued support for them after their initial release. Secor (GitHub.com/Pinterest/Secor) solves a very important problem for Kafka users, by allowing logs to be persisted to S3. It powers our monetization pipeline, one of the most critical pieces of the Pinterest stack. Bender (GitHub.com/Pinterest/Bender) is a load testing library that makes it easy to write flexible load-testing tools. It has been heavily used by the Search team to discover severe performance issues, the kind of bugs that are extremely hard to isolate outside of a production environment. What is unique about Bender is that it is written in Go.
What inspired Pinterest to release the code under and open source license?
Contributing back to the community is essential for innovation. Without open source software, Pinterest would probably not be where we are today. We feel that successful companies born with the help of open source software are essential in moving the community forward. The desire to help other great companies rise, take advantage of, and build upon tools and services we've built is behind every contribution and project we undertake. We're releasing new projects primarily under the Apache 2.0 license. We feel that it provides us with the best mix of permissiveness vs definition for derivative work that might come out of a project. This is particularly important for companies, and it is what makes Apache 2.0 stand out from other great license options.
Are other organizations/developers working with you on the code?
Our recently open sourced project Secor has been adopted by Airbnb, and they've had many great contributions that have made their way back into our code base and onto our production environment. Uber is also in the process of making a very significant contribution to the project. A complete list of contributors can be found at: https://github.com/pinterest/secor/graphs/contributors. A few of the upcoming projects that we are working on are things that other companies that we are in touch with are very interested in adopting. We also continually contribute to projects we internally utilize.
Conclusion
Code isn't the only thing Pinterest is contributing to open source. Tracy Chou and fellow Pinterest engineers helped lead a year-long discussion (Where are the numbers?) around tech workplace diversity, and they recently released their findings on GitHub. And you thought Pinterest was just a website full of cupcake recipes and craft ideas.
See more reports from Rikki at Grace Hopper Celebration '14 on the Open Source Community blog: GHC '14 Opens Up with Open Source, GHC '14: OpenHatch Open Source Workshop & Getting Into GitHub, and GHC '14: Advancing the Work from Home Option.
Comments are closed.