695k TPS with Node.js and VoltDB
voltdb.com
voltdb.com
As far as open source: I was using the .NET client, and fixed a few issues with it. The folks at VoltDB were really responsive about working with me and accepting my patches. They also took time to explain the inner workings, sorta give me a launching point to investigating the core code. Pretty good experience for a commercial open source project.
would you describe what telecom application this was ? was it billing, ocs, pcrf or something else ? thanks !
TITLE DISCLAIMER: 695k using 20 fairly large machines
Just for an idea of how big of a machine we're talking for each of these instances:
High-Memory Quadruple Extra Large Instance
68.4 GB of memory
26 EC2 Compute Units (8 virtual cores with 3.25 EC2 Compute Units each)
1690 GB of instance storage
64-bit platform
I/O Performance: High
API name: m2.4xlarge
$1.800 an hour running Linux on demand
$864.00 a day baseline for 20 of these
The results are definitely impressive for the right target audience.
> On the database side, I configured VoltDB to run at K=0,
> which means the database was running with HA switched off.
> Most applications would not run that way in a production
> environment, but my goal was to test client-side throughput
> and scaling; I wasn’t really focused the database side of
> the tests.
These benchmarks don't mean much to me -- what matters is performance for your application.
>These benchmarks don't mean much to me -- what matters is performance for your application.
Very true. Think about how you would implement an application like Voter in Redis. Voter is an event counting app that applies business logic server side to do what is basically fraud detection.
Let's take a look at what each of these transactions is. https://github.com/VoltDB/voltdb/blob/master/examples/voter/... 3 statements and an entire round trip to validate data is kosher. 1 insert/update statement that is actually updating two different summary views
You would need to implement your own locking to get the isolation necessary to implement the business logic and there would be at least two round trips for each vote. You would also have to write your own code to maintain the materialized view used to print the summary. Sure you could implement the validation logic client side, but then you lose the ability to transactionally update said logic and the state it depends on.
With the new scripting functionality in Redis you would be able to do it in one round trip but you would still end up writing your own code for the view and rollback. How you will attach a debugger to debug your server side logic is also worth considering. I mistakenly wrote that Redis doesn't cache Lua scripts, but it actually does.
Now that you have server side logic you will also probably want to start maintaining replicated state to make some of your transactions single shard transactions. With Redis you are on your own for updating replicated state in a way that stays consistent across the cluster even when restoring different shards.
Also, no matter how fast Redis is you may reach a point where you have run out of disk/network IO, or even CPU. Now you are looking at sharding Redis. You are probably going to want to maintain replicated state to make more of your transactions single shard transactions. With Redis you are on your own for updating replicated state in a way that stays consistent across the cluster even when restoring different shards.
Don't forget re-sharding and backups which are now your problem. You definitely want to be able to roll back to before that last application update that corrupted your data. Don't forget that you will need to bring the replicas up in the correct state as well. Preferably in parallel with bringing up the master since you are down right now, and it would be nice if it resharded at the same time since you took this opportunity to add capacity. Redis cluster is coming, but it isn't here today and depending on the application you may not want to risk it not being ready when the time comes.
You should at least double the number of ops (where an op is roundtrip to the DB) done by Volt if not more. That said, doing half as much work inside the procedure doesn't make it twice as fast and doing twice as much doesn't make it twice as slow. The dominant cost for such a trivial operation is not the operation itself. Redis is much better at this because it is single threaded and written in C. My bet is that the number of instructions and cache misses to accept and execute a command in Redis is much smaller because the execution engine isn't dealing with stored procedures, schema, undo logging, SQL, and code expecting to deal with distributed transactions even if none are in play.
>I'm not sure if I am supposed to be impressed or not.
If you haven't tl;dr-ed already I'd call it good.
At any rate, as I mention in my longer report on the benchmark [1] the high total translates into very LOW hardware requirements for lesser needs. The number to go by may rather be "10k+ TPS per core for linear scaling ACID transactions." I think that's the number that matters. Linear 10k+/core ACID TPS.
[1] http://community.voltdb.com/sites/default/files/NodejsBenchm...
edit: ah, MichaelGG has explained it much better in another part of the comment thread!
VoltDB sound like a potentially great thing for an application that's really (and only) OLTP. But it's definitely not ready to replace Oracle, PostgreSQL, etc. for more general-purpose needs.
VoltDB also competes indirectly with NoSQL products like Cassandra, which achieve performance and scale primarily by sacrificing SQL and ACID transactions (Volt preserves both).
As an in-memory database, VoltDB offers durability through command logging and disaster recovery via a recently-introduced feature called database (WAN) replication.
Hope this helps.
VoltDB is often compared to other in-memory NoSQL systems, although I'm not aware of any that actually offer cross-server transactions. VoltDB is best when you have a high volume OLTP workload and need to really scale beyond what traditional RDBMS can do.
Edit: Here's a benchmark using VoltDB as a key-value store: http://voltdb.com/company/blog/key-value-benchmarking
Oh sure, except that the probability of losing your hard drive when you lose power is about 1 in a very big number. Whereas, the probability of losing the contents of your RAM when you lose power is 100%.
Now, MySQL (and in fact, most RDBMS outfits) provide tools for fixing corrupt data. But you actually need corrupt data to fix. That's the entire point of a hard drive.
I'm sure VoltDB would be perfect for a chat application or something of that nature. It would be terrible for anything that involved the transfer of $$$ (banking, ecommerce, etc).
In practice with VoltDB, you'll replicate to multiple machines synchronously (so you'd need to lose power to separate servers to lose data). VoltDB enterprise has command logging.
For a chat messages, I'm not sure VoltDB would be the _best_ choice, perf wise. Chat messages are usually more ephemeral - why go through the overhead of ACID for those?
For real $$$ transfers, you'll probably want synchronous WAN replication, to really ensure you never lose transactions.
The Enterprise Edition adds a synchronous or asynchronous command log (a logical journal) as well as asynchronous multi-cluster replication.
The VoltDB blog has a series of posts covering command logging in detail: http://voltdb.com/company/blog/intro-voltdb-command-logging http://voltdb.com/company/blog/voltdb-command-logging-replay http://voltdb.com/company/blog/voltdb-command-logging-option...
In short, VoltDB offers tunable durability from single-node purely in-memory, all the way up to synchronous disk storage on multiple local nodes and multiple asynchronous copies in another data center. All options are fully transactional. All options are extremely performant.
We've worked very hard to make durability and performance not mutually exclusive. We have internally achieved six-figure TPS numbers using synchronous writes (requires a disk controller with a BBU, or a decent SSD).
It's like node but isn't single threaded so scales over multiple cores without having to fork. Also it's polyglot so you can don't have to use JS if you don't want.
I'm hoping to publish some performance results vs node.js before our 1.0.final release in the next few weeks.
On the other hand, a big chunk of the web is run from AWS. Demonstrating performance there implies performance in many places. It's also an easily reproducible environment, which is helpful for benchmarking.
If you look at the report details, you'll see some of the lumpiness your question suggests, particularly related to the network. The Node instances were beginning to starve when database server CPU utilization was still pretty low.
We would certainly have gotten better numbers if we ran these benchmarks on bare metal or a closely-tended cloud infrastructure. But we're a pretty low-BS group, so we'll stand behind what EC2 gave us.
- I assume you did the tests with direct SQL statements without any ORM? How would an ORM affect the test's performance (with an Node.JS-ORM of your choice, e.g. sequelize)?
- How does Node+VoltDB compare to i.e. Node+PostgreSQL or Node+MySQL
- What does in memory SQL exactly mean? How is data made persistent?
"Third is NewSQL, which obtains better performance and scale in the only way possible – with a totally new architecture. NewSQL delivers high performance and scalability while preserving the traditional ACID functions needed for OLTP. NewSQL systems like VoltDB deliver better throughput than NoSQL without requiring transactions to be supported in user code. NewSQL systems also preserve the high-level language capabilities of SQL."
How does this work?
Basically, by staying in memory and eliminating locking, you get awesome performance. However, why should you have to give up SQL, tables, and transactions? Those things make developing apps easy. So why not get "NoSQL-like" performance, but keeping the good parts of "SQL"?
There are concurrency primitives around networking/messaging and transaction scheduling, so there aren't truly ZERO locks in the system as a whole.
The big difference is that VoltDB has zero system-locks (mutexes,etc) around table data and indexes. It also has no logical locks/latches arounds indexes and rows like other systems. This makes executing SQL very fast. Because it doesn't wait on disk or on mutexes, SQL execution is typically bottlenecked on memory bandwidth.
VoltDB smokes other RDBMS (Postgres, MySQL, MSSQL, etc.) for certain OLTP workloads because it stays purely in-memory, but removes all locking. Normal databases spend a huge amount of time dealing with locks, even if the memory is in RAM. VoltDB was based on the H-Store project; you can read more about the ideas at [1].
Making transactions execute serially[2] adds latency to an individual transaction, but the point with VoltDB is to send tons of transactions so that there's always work queued up. You should be able to get average latency of, say, 10ms for the type of app mentioned here. I'm surprised it's not mentioned in the blog post. (Maybe on EC2, the latency numbers were less-than-ideal, but throughput was fine.)
SQL and ACID have nothing to do with the backing store. Using RAM is just as valid as using disk. Practically, with VoltDB you're going to set the "k factor" - number of replicas you want. This allows you to lose k servers without suffering data loss (put batteries on the servers; that should be pretty good). There's a snapshot backup feature, and the enterprise edition has a disk logger to further minimize the loss window in case of catastrophic failure.
1: http://hstore.cs.brown.edu/publications/ 2: More or less. Servers are sync'd via NTP, allowing one to know when it's safe to run a specific transaction. There's also some ways Volt might know when one tx is independent and safe to run before another.
Here's the Epic in JIRA: https://issues.voltdb.com/secure/IssueNavigator.jspa?reset=t...
Typically, applications are not 100% ad-hoc on VoltDB, but we're starting to see more hybrid use cases. Stored procedures can be used for extreme throughput or for transactional requirements, but other parts of the app can be more flexible.
Yes, the tests were run without an ORM. Although VoltDB is accessible via JDBC, best throughput is achieved with parameterized SQL embedded in Java stored procedures. That's how the Node.js benchmark app (Voter) was implemented.
Node+Volt will be much faster than Node+Postgres or Node+MySQL for many OLTP-style workloads. So it's a great fit for apps like financial trading, digital advertising, online gaming and network monitoring - each of which requires super-high throughput writes (at single ms latencies) and fast, simple analytics. Node+Postgres or Node+MySQL would be a better fit for general purpose database workloads (e.g., with large-grained reporting).
In-memory means that Volt stores all data in main memory. Durability is achieved through the use of a highly innovative command log, which logs all transaction invocations to persistent storage. You can tune this feature to log synchronously (100% guaranteed durability at reduced transaction latency) or asynchronously (slightly more "lossy", but less of a latency impact). VoltDB also recently released a full database replication feature that supports WAN replication for disaster recovery.
I hope these responses are in some way helpful.
I am able to get sub-millisecond offsets (as reported by ntpq -p) with m1.large instances, but I have never left them running long. I am not sure what Henning got when he ran. I pointed him to my blog post on the topic http://www.afewmoreamps.com/2011/07/configuring-ntp-for-volt...
On bare metal ntpdate reports offsets that are sub microsecond.
I find that nodes sync up quickly if only one of them polls the public pool.
For time sync, I got sub-millisec syncs after using Ariel's tip at http://www.afewmoreamps.com/2011/07/configuring-ntp-for-volt... . Essentially, switching NTP off and manually running ntpdate -p 8 <server name> repeatedly can give you that.
Just do it ten times in a row - I don't know why the normal operation of NTP doesn't achieve the same.
This tight sync seemed to increase throughput by 2%. Because that was not so significant I did not further investigate the consequences of good or bad time syncing.
The time sync that EC2 instances come with out of the box seemed to sometimes be good enough to run multiple VoltDB hosts without any additional syncing, sometimes not, and it seemed to vary quite a bit over time.
So really you must deal with NTP in EC2, but of course you need NOT get the sync to sub-millisecond values.
That's not very accurate (unless you run multiple Node.js processes). See http://stackoverflow.com/questions/2387724/node-js-on-multi-...