Hacker News Reader

Comment by makmanalp | Hacker News Reader

Hmm, I wonder how many potential customers they have.

Far more than you might imagine, at least if they develop a more sophisticated geometry model. The real limitation may be the pure in-memory model. Many of the high-value applications have enormous working sets, much larger than what will fit in a small RAM-based cluster. Hundreds of terabytes is where the applications just start to become interesting. In this sense, focusing on game worlds is probably a good idea because it is one of the use cases that will fit within their immediate scaling targets.

They are correct that traditional database spatial indexes are slow and scale terribly, being designed for relatively small and static data sets. It does not sound like they are pushing the state-of-the-art, just meeting an under-served need in the gaming space, a market which I can validate as existing but with somewhat limited revenue potential even if you sign the major game companies. It is a good "base hit" startup opportunity if they can execute it well. (I have designed massive-scale real-time spatial database engines for a number of years; we passed up the gaming market because the size of the market was too small relative to other markets for this technology.)

Forgive the basic question: Is cost the only limiting factor for pure in-memory with large datasets?

Cost is a big factor but it is not the only consideration. Even if cost was no object and you could just throw a lot of machines at the problem, designing distributed algorithms that will scale to thousands of machines is a different kind of problem than what you can get away with on several dozen machines especially when talking spatial access methods (which have their own unique nuances).

This is surmountable but few people know how to design the data structures and algorithms required to make non-trivial spatial data models scale to that level. There are significant gaps in the published literature.

Actually, we're very much pushing state-of-the-art :)

It's just that we're tackling a different problem - low latency applications - and when I say low latency I mean in the microseconds range. Big spatial data is a very interesting problem, and tracking a large number (though less than billions) of moving objects is another - though very different - interesting problem.

For working sets that don't fit in one machine's RAM we offer a cluster.

While I do not currently work on ultra-low latency spatial databases (more like milliseconds), I have in the past and so have some idea of what is out there. :-) I am not all that familiar with the design of your system so I was mostly working off the scale numbers offered.

The best example I can think of is an ultra-low latency in-memory prototype I designed in 2009 on a parallel cluster. The working set was several billion irregular 3-cubes ranging in (metaphorical) size from birds to hurricanes. The average CPU cost of an access operation was sub-microsecond so the latency was mostly interconnect related (which was a slow but proper low-latency supercomputing fabric). The current work I do uses complex geodetic polygons geometries so the computational cost of operations is quite a bit more but the actual computational cost of the access method is below the noise floor of the network fabric.

You are correct though that if you are mostly dealing with tracking points or cubes then in-memory is sufficient to hold many applications. It is the sensing data that really kills you... :-)

Having talked to Ron's cofounder, it sounds like there are a surprising number of applications for SpaceBase outside of gaming. The TC article seems to focus on gaming but I have a hunch it's mostly that they're going after one vertical at a time.

The company I work for is a potential customer. We store thousands of locations per minute in one of our Postgres databases.

Location based services could be big. Asking "what's nearby meeting criteria A,B,C" requires a non traditional backend.

Not really, Postgres works fine for most of this stuff, and as of 9.1 without PostGIS

'nearby' means ordering your query using the <-> operator.

'meeting criteria' goes in the where clause.

For example to find interesting things nearby in postgres:

  SELECT * from interesting_things as it where (it.A ='A' AND it.B ='B' AND it.C = 'C') 
  order by it.location <-> point '(current_lat,current_lon)' LIMIT 10

http://wiki.postgresql.org/wiki/Whats_new_in_PostgreSQL_9.1#...

Founder of Parallel Universe here.

Sure, Postgres works great and it's not our competition. The idea is that you'll store all of your static stuff in Postgres, and all of your dynamic objects in SpaceBase. We've been approached by companies that need to track thousands of vehicles, each updating its location every second or so, and Postgres just couldn't handle it. For LBSs, SpaceBase complements Postgres.

Yeah postgres would fall down pretty fast on that workload without sharding, and it's the kind of ephemeral data that doesn't need ACID semantics.

Do you have a trial? Or other demo? I may be encountering those kinds of problems in the near future and would like to see whats out there. Right now Postgres is perfect for our workload but we've crafted the problem we're solving to avoid the realtime issue.

E-mail us and we'll gladly send you a trial version.

Some of the data sources of interest for location-based applications generate tens of millions of updates per second. I love PostgreSQL but it does not operate at that scale. The index locking alone would kill it even if you weren't touching disk.

Extremely high spatial ingest rates concurrent with queries requires different architecture and algorithms.

It doesn't necessarily require a non-traditional backend, as you could design your backend using plain old rdbms. What does require a non-traditional design would be your app, however.

Reply on news.ycombinator.com