Minimal object storage with Minio

Image by:

Scott Meyers. Modified by Opensource.com. CC BY-SA 2.0.

In 2007, GlusterFS revolutionized the way large, distributed storage solutions were implemented across organizations—anyone could now use commodity hardware to create distributed storage solutions for media streaming, data analysis and other data intensive tasks using GlusterFS. Red Hat acquired GlusterFS in 2011. Recently one of the founders of GlusterFS, Anand Babu Periasamy, announced a new open source project, Minio. Minio is designed to enable application developers to build their own cloud storage. I recently interviewed Anand Babu to learn more about Minio.

Minio roadmap

Minio is another take on cloud storage. The goal of Minio is to enable application developers to build their own storage clouds, just like how Amazon, Google, and Facebook developed their own proprietary cloud storage.

Modern applications have to deal with large quantities of data. Databases are designed for storing and querying metadata. When it comes to storing data itself, cloud storage has emerged as the preferred choice for application developers. There is MongoDB for document store, Node.js for JavaScript runtime, Angular for HTML framework, and Minio is designed to fill the gap for object storage.

Minio is based on a client-server model. The object storage client utility mc and libraries (Node.js, Go & Java) are ready for production use. Compatible with Amazon S3 cloud storage service, Minio will enable users to move large quantities of data in and out of cloud storage. mc provides Unix ls-, cp-, and rsync-like commands, which works for both local disk and cloud storage.

A standalone version of the server is currently available for download. It supports in-memory, file system, and Donut (erasure storage) back ends. A hosted version of the server is also available, intended for testing and development.

A distributed version of the server and operations and user consoles are in development, and in about 6 to 12 months, the stack should be completed.

Going with Golang

Minio uses Go, which is similiar to Java, but without the JVM dependency. Go's channels and goroutines make it suited for writing scalable high-performance servers. One touted part of Go is eliminating the need for package management. Go code compiles to a single static binary; just copy it anywhere and run.

Go is opinionated, and influenced heavily by Google's use case, but Minio doesn't mind inheriting Google's way of building infrastructure software. Go is not as mature as Java yet, and it still has some rough edges when it comes to tooling and garbage collection; however, Go is ready for developing production-grade software. The next release of Go 1.5 implements efficient concurrent garbage collection and address the tooling issues.

A number of options were considered before adopting Golang:

C++ and Google V8 JavaScript: Minio was prototyped with C++ and V8. JavaScript brings the ability to send code over the wire and patch a live system on a continuous basis. It was ditched because the V8 C++ API is unnecessarily complex. The Minio team also did not want to inherit JavaScript’s limitations.

C++ and Mozilla SpiderMonkey: Compared to Google V8, SpiderMonkey's performance is neck to neck, and its API is also much cleaner; however, SpiderMonkey was dismissed because of the JavaScript limitations.

Java: Java is mature and fast. Everything about Java is great, except for its bloated JVM dependency.

Haskell: Haskell would have been the first choice for Minio, if it were written for internal use. Unfortunately, Haskell has not received enough attention from the developer community.

Rust: Rust is a powerful modern language and would have been a second choice if Golang wasn't around. Rust is ideal for projects such as database engines, transaction servers, and Internet browsers. Implementing object storage is relatively simple and does not require Rust as much.

C and Python: C is powerful and Python is fun to write code, but C is primordial and Python is better suited for DevOps.

C and GNU Guile: This option was used for GNU FreeIPMI and GNU Freetalk; Scheme is awesome, but it has limited community adoption.

Contribute to Minio

Minio aims to finish the complete stack development in 6-12 months time, so now is a good time to join the community and influence their direction. Here are the ways you can contribute to Minio: