Get the highlights in your inbox every week.
Building cloud storage with Minio
Minimal object storage with Minio
In 2007, GlusterFS revolutionized the way large, distributed storage solutions were implemented across organizations—anyone could now use commodity hardware to create distributed storage solutions for media streaming, data analysis and other data intensive tasks using GlusterFS. Red Hat acquired GlusterFS in 2011. Recently one of the founders of GlusterFS, Anand Babu Periasamy, announced a new open source project, Minio. Minio is designed to enable application developers to build their own cloud storage. I recently interviewed Anand Babu to learn more about Minio.
Minio is another take on cloud storage. The goal of Minio is to enable application developers to build their own storage clouds, just like how Amazon, Google, and Facebook developed their own proprietary cloud storage.
Minio is based on a client-server model. The object storage client utility mc and libraries (Node.js, Go & Java) are ready for production use. Compatible with Amazon S3 cloud storage service, Minio will enable users to move large quantities of data in and out of cloud storage. mc provides Unix ls-, cp-, and rsync-like commands, which works for both local disk and cloud storage.
A standalone version of the server is currently available for download. It supports in-memory, file system, and Donut (erasure storage) back ends. A hosted version of the server is also available, intended for testing and development.
A distributed version of the server and operations and user consoles are in development, and in about 6 to 12 months, the stack should be completed.
Going with Golang
Minio uses Go, which is similiar to Java, but without the JVM dependency. Go's channels and goroutines make it suited for writing scalable high-performance servers. One touted part of Go is eliminating the need for package management. Go code compiles to a single static binary; just copy it anywhere and run.
Go is opinionated, and influenced heavily by Google's use case, but Minio doesn't mind inheriting Google's way of building infrastructure software. Go is not as mature as Java yet, and it still has some rough edges when it comes to tooling and garbage collection; however, Go is ready for developing production-grade software. The next release of Go 1.5 implements efficient concurrent garbage collection and address the tooling issues.
A number of options were considered before adopting Golang:
Java: Java is mature and fast. Everything about Java is great, except for its bloated JVM dependency.
Haskell: Haskell would have been the first choice for Minio, if it were written for internal use. Unfortunately, Haskell has not received enough attention from the developer community.
Rust: Rust is a powerful modern language and would have been a second choice if Golang wasn't around. Rust is ideal for projects such as database engines, transaction servers, and Internet browsers. Implementing object storage is relatively simple and does not require Rust as much.
C and Python: C is powerful and Python is fun to write code, but C is primordial and Python is better suited for DevOps.
C and GNU Guile: This option was used for GNU FreeIPMI and GNU Freetalk; Scheme is awesome, but it has limited community adoption.
Contribute to Minio
Minio aims to finish the complete stack development in 6-12 months time, so now is a good time to join the community and influence their direction. Here are the ways you can contribute to Minio:
Gitter: Minio community hangs out on Gitter, which is similar to IRC.
GitHub: All of Minio's code is openly developed on GitHub.
Contribution guide: Follow this guide to set up a development environment. Because Go code looks much like C or Java, getting started is easy.