An open source, infinitely scalable Relational Database Management System (RDBMS) | Opensource.com

An open source, infinitely scalable Relational Database Management System (RDBMS)

Posted 02 Dec 2013 by 

Rating: 
(11 votes)
Open wires
Image by : 

opensource.com

submit to reddit

InfiniSQL is a massively scalable relational database system (RDBMS), composed entirely from scratch (not built upon some other technology). There is reproducible benchmark data described on InfiniSQL's blog proving that it can perform over 500,000 complex, multi-node transactions per second with over 100,000 simultaneous transactions—all on only 12 small server nodes.

The limitation of 12 nodes was budgetary: this is an open source project entirely funded out of pocket, and not part of an institution. If I had access to more servers, I'm positive that scalability would grow much higher. But those kinds of details are on the blog. The bottomline is that this is a very high performance system, and in its infancy.

The only open source software which is remotely similar to InfiniSQL is MySQL Cluster. I believe that InfiniSQL scales and performs better than MySQL Cluster for many operations, especially transactions that contain records on multiple nodes. But, I have done no direct apples to apples comparisons between InfiniSQL and MySQL Cluster and welcome any data to that effect. I may do such benchmarking in the near future, as time allows. The purpose of this article is not to disparage MySQL Cluster (or any other product), but rather to explain the advantages of InfiniSQL, and what sets it apart.

I am a long time Linux user. My first distribution came on 30+ 1.44MB floppy disks from a company called SoftLanding Systems. The Linux kernel was version 0.99 something, and I loaded it on my beige box 80386 system with 4MB of RAM. I had to fiddle with /usr/X11R6/etc/Xconfig to get my monitor to provide a graphical interface (and, I nearly fried it). That's how old I am. I have worked for many years as a sysadmin. Most recently, at Visa Corporation.

There, I was the capacity planner and performance engineer for their online division called CyberSource. I've architected, configured, maintained, and troubleshot very high volume, 24x7, mission-critical environments, where every second of downtime meant many thousands of dollars of lost revenue for our customers. As anybody familiar with complex environments knows, the biggest pain points often come from the backend database systems that handle the transaction and reporting workloads. Traditional database systems have very serious architectural flaws which make them nearly impossible to scale beyond a single server; it's very difficult and expensive to make them continuously available.

I felt that I could do better.

I began to design InfiniSQL as an open source project because I believe it will get the most users, will acquire the best developers, and will grow in time to have the capabilities and features to be the RDBMS of choice for people who only currently use large, expensive, and proprietary systems. That's how Linux grew to displace proprietary operating systems running on expensive servers.

Eventually had to choose between my full-time, comfortable job or following my passion, and in March of 2013 I quit my job at Visa. I just recently began telling people outside of my closest circle of friends a few days ago, because until now the project wasn't in a suitable state for consumption (but it is still in alpha). As Eric Raymond said, "release early, release often." 

There are a handful of closed source companies that have features similar to InfiniSQL, so it is comparable in many respects yet superior in other ways to these offerings. InfiniSQL is also not as far along in development. Similar closed source offerings have already received hundreds of millions in funding, enabling them a higher capacity to hire. InfiniSQL's use cases are mainly for high volume data collection, analysis, and transaction processing. If your environment has a mix of SQL and NoSQL, it's very likely that InfiniSQL can be used for both type of workloads. This is because InfiniSQL performs like many fast NoSQL systems, but still speaks SQL and performs transaction processing, unlike NoSQL. In fact, InfiniSQL speaks the PostgreSQL Frontent/Backend Protocol (PostgreSQL client drivers will work with InfiniSQL) because the beauty of open source allowed me to use their open protocol in InfiniSQL. This has saved me time in writing functional, mature, and cross-platform client drivers.

My goal now is to tell as many people as possible about InfiniSQL and invite them to learn and participate in any way they wish. I am looking for co-developers (C++ and Python) and people willing to download and test it. There is over 100 pages of documentation, including a guide on how to configure a Linux x86-64 environment and build, run, and test InfiniSQL. InfiniSQL is also entirely written from the ground up, and it is not based on another product, either open or proprietary. There's already 35,000+ lines of code, and that's just the beginning.

 

submit to reddit

7 Comments

wanderson
Community Member

The article writer needs to better explain "exactly" what he means with statement "The only open source software which is remotely similar to InfiniSQL is MySQL Cluster." - particularly in regard to use of word "similar".

If he is referring to reliability and scalability of SQL database clusters, PostGreSQL is a world renown and regarded Free/Open Source Software (FOSS) RDBMS that has performed "record" transaction scalability, in documented reports from Yahoo, Fujitsu, NASA, Verisign, Netflix and many other enterprise entities. In certain controlled tests, PostGreSQL has recorded outperforming recent Oracle Database 11 in identical operations and workloads fora Fortune 100 Corporation.

Did the writer even "extensively" evaluate and test PostGreSQL against InfiniSQL and/or MySQL?

While such comparative evaluation project does not necessarily detract from any great features and value of InfiniSQL, it brings into question the overall "expert" technological judgment and know-how (or lack thereof) of the writer in such area of Clustered, Scalable RDBMS or any high end database technology.

Vote up!
4
Vote down!
0
Mark Travis

PostgreSQL does not do horizontal scaling. The first paragraph of the article described what InfiniSQL can accomplish in this regard: "There is reproducible benchmark data described on InfiniSQL's blog proving that it can perform over 500,000 complex, multi-node transactions per second with over 100,000 simultaneous transactions—all on only 12 small server nodes."

If you want explanation of what these terms mean, I can clarify further. Please contact me through the form on the http://www.infinisql.org/community/ web page.

I think that PostgreSQL is a great project and I admire their engineers. They just don't have a product that scales horizontally.

Vote up!
4
Vote down!
0
Peter Mengaziol

Perhaps you could/should make iDB a contributed module that will make pg scale horizontally because there is a greater potential for a 'killer' product with tens of thousands of fans/afficionados/consultants. Unless in doing 'your own thing' you have captured all the other features besides scalability, it will be a long time in building a better pg.

Vote up!
0
Vote down!
0
Mark Travis

Hi, Peter. I've considered what you suggest, and describe my opinion in a FAQ: http://www.infinisql.org/docs/faq/
I welcome any contributions to InfiniSQL!

Vote up!
0
Vote down!
0
veggen

How is persistency handled? What are the plans there? Going for occasinal flushes like Redis, or for a fully persistant system? What about the rest of ACID?

Vote up!
2
Vote down!
0
Unidentified

The answer to this is in the documentation section. It's very well written, not overwhelmingly long, and actually pleasurable to read.

Vote up!
2
Vote down!
0
Mark Travis

Thank you, Unidentified, for the kind words. The section to which you refer is in http://www.infinisql.org/docs/overview/ under the "Data Storage" heading.

Vote up!
0
Vote down!
0

Comment now

I am a long time Linux user and IT professional. My career is largely a result of what I learned about computing by using Linux as a hobby many years ago. I have spent much of my career in very high volume, mission critical transaction processing environments. I want to help continue the tradition of bringing open source software into places that currently rely on closed systems.

I live in Silicon Valley with a beautiful wife who owns her own business, and two little dogs that don't

Open source project management

Holiday gift guide promo