Hacker News Reader

Hacker News

Comment by vessenes | Hacker News Reader

Full thread

vessenes

This company was probably an extremely easy investment decision:

1) Facebookers: Check

2) Data Scaling experience: Check

3) In-Memory with SQL semantics: Check

4) New-York based software that can be sold to quant funds: Check

I'm excited. It's downloading now.

View on HN

Is it just me or does this entire product smell like it was designed for the sole purpose of extracting money from less than tech-savvy investors and clueless institutions?

The bullshit-bingo-lingo on their homepage is mindnumbing.

Meanwhile their actual software seems rather underwhelming, bordering on SnakeOil.

If you've ever tried to use SQL on a large, but fittable-in-memory dataset on existing row databases, then you would not be underwhelmed. Existing non-finance-industry solutions suck for things like mid-size trading queries, or (in coinlab's case) work setups that require periodic, frequent table scans.

You can buy something like kdb, but that costs $70k a year and requires that your engineers learn some extremely new semantics if they're only used to SQL and (choose any popular language here, Python to C++).

So, MemSQL is a ColumnStore now? The docs didn't make that clear for me.

Anyway, why is there not a single benchmark available to support the sales-pitch?

(In your favor I'll just pretend you didn't mention kdb here...)

MemSQL is a row store. KDB is single threaded. More benchmarks are coming!

http://kx.com/faq.php seems pretty clear that kdb is multi-threaded. "Does kdb+ support multiple cores and/or multi-threading? Yes, both are built right in to the system and this makes kdb+ extraordinarily fast when compared with traditional applications, as it can make full use of all available cores. "

This ties with my own experience using it.

I'm sorry, I have to clarify this. It's a single writer database...

Any chance you could publish full info on the benchmarks?

For example, where can we find your configurations for the MySQL vs. MemSQL benchmark you show in your video? Or how big the dataset was, etc...?

In your favor I'll just pretend you didn't mention kdb here

Please explain. What's bad about kdb, besides how nonstandard it is?

There is nothing bad about it (other than the price). It just seems an outrageous claim that MemSQL, with all its constraints, is even in the same game.

At the least they should come up with some seriously impressive benchmarks before dropping names like that.

I have had the misfortune of working with Q. It's a very clever little language that could be a great platform for numerical prototyping. Unfortunately, it's hobbled by a terrible programming culture. The interpreter gives cryptic single-character error messages and the C interface is written in a nightmarish soup of macros.

Fast, in-memory data stores already exists with support and ease of use/deployment...I was able to set up hazelcast for my distributed cluster in minutes...I don't see the advantage of this over nosql or in-memory data storage.

Really? I've done many, many queries on large, but fittable-in-memory datasets and with a few little tricks even MySQL can be fast (make everything a temporary table with an append only transaction table to rebuild the temporary tables if lost, don't use keys, use indexes, avoid bad practices (filtering after a join, using subselects, or needlessly avoiding HAVING clauses).

Did you mean large, fittable-in-memory, and (distributed across multiple computers and/or very reliable)? Because that is much harder.

Can you explain this further? How is using a temporary table faster? Explicit temporary tables are simply regular tables that only last your current session. Do you mean the MEMORY storage engine? What do you mean by "don't use keys, use indexes?" The term KEY and INDEX in MySQL are exact synonyms.

Temporary tables default to being mapped in memory and for more complex querying (where you have multiple queries and conditionally select further queries depending on results) are faster since the table that they are based off of could have changed, but they will not, so caching is much more likely. Keys and Indexes are not exact synonyms. Give it a try some time, do a where clause with mod(id, 5) = 0 on a keyed table and on an indexed table. The indexed table will be non-trivially faster (I've seen 3x faster).

You are clearly not talking about MySQL. CREATE TEMPORARY TABLE simply creates a physical table on disk that is destroyed at the end of the session, which can be created with any storage engine. Explicit temporary tables are handled just like "regular" tables and is dependent upon the storage engine (w/ MyISAM, indexes are stored in the key buffer, InnoDB keeps both recently accessed data and indexes in the InnoDB buffer pool).

KEY and INDEX are literally the exact same thing and are just there to support different syntaxes: http://dev.mysql.com/doc/refman/5.1/en/alter-table.html

Perhaps you are referring to primary keys vs secondary indexes, for which there is a difference in how they are physically stored on disk and in memory, particularly with InnoDB. There is a substantial performance gain to be had by having a relevant primary key that is referenced in the where clause as the data is stored in the same page as the primary key, both on disk and in the buffer pool. Secondary indexes reference the primary key, thus require two lookups.

Another key point is that temporary tables are horrible for concurrency. Obviously only one session can access them, but more importantly, they end up causing a great deal of disk activity. At the end of every session, the table is physically dropped. In ext3, this can take quite a while for large tables (not so bad in filesystems like xfs). It also can cause substantial stalls in InnoDB as the buffer pool is scanned for matching pages and are flushed out. Percona Server has innodb_lazy_drop_table to mitigate this issue to some degree, but dropping tables is still always an expensive operation. This is mostly a concern for OLTP, not necessarily OLAP servers.

Actually, what you say about the price for kdb+ is true, but Q (the new language to query kdb+, essentially the latest version of K which is now deprecated) is very much like SQL, syntaxwise.

The way I see it, this is very much targeted at financial firms that use kdb+ or products like it that perform really well for things like writing lots of tick data from dozens of exchanges at a time and querying across them quickly in a familiar SQL style.

It's not just you. But I think that's the whole idea. It's about fooling people. And I think some of the profits will be flowing back to Oracle, though that is not stated upfront. The marketing power of the acronym "SQL" should not be underestimated.

But that's only a query language. Not a db architecture.

Let's see the source code. That's the only way to verify the approach that is being taken.

Given the source the first thing I'd do is search for "mmap".

They already get one demerit for using a row store.

And another for disk access.

SnakeOilSQL.

We do use mmap, but we do not use memory mapped files. We use mmap to allocate big chunks of memory from the OS, which we later distribute to our custom memory allocators.

It's not a column store. We are thinking about it, maybe some time in the future as we add more analytical feature. It's a row store.

We do write back to dist. We have a transaction log, which we truncate once it grows over a certain threshold, by taking a snapshot.

paul buchheit and max levchin seem pretty tech-savvy to me

All of the arguments for using this are contrived. How big is the total addressable market of projects where there's a huge budget and engineers who ultimately have no idea of what they're doing?

Thanks! We're super-excited to have the opportunity to build this. When we started we asked ourselves - what would a database look like if it were designed today?

We see a future where OLTP databases live in memory, and where you have hundreds of terabytes of memory and hundreds to thousands of CPUs at your disposal.

We built MemSQL to make it easy to go directly into memory and get the speed improvements we all need. What we're releasing aims to fulfills that vision.

p.s. Eric, two annoyances about your site: once I've been dumped on to the developers subsite, it's non-trivial to get back to the main company homepage.

Also, pricing. Come on, man! Pricing!!

Thanks! We'll add a link back to memsql.com =) What did you think of the video and diagram on developers?

I think none of that really matters until we see the price. There's a trial version and a free version of Microsoft SQL, but their full version can quickly top $20k without much effort - knowing how bad your "ouch" will be is key before I'd spend any time on the "ahhhh."

The diagram was fine; I didn't watch the video. I'll hit it up later.

p.s. Isn't a totally reasonable business model for a truly faster DB engine "free, everywhere, bought by Oracle?"

We think a lot about it. And even though this model definitely makes sense, we would like to build a business that can generate money and pay for itself.

Reply on news.ycombinator.com