In particular, to compute where a user drops off in a funnel, I need to scan one array from left to right, and I don't need to do any joins. This shards very well, since all of a user's data lives on one shard, and most of the queries are aggregations, which are simple to reassemble from subqueries.
Because even a shitty RDMBS is more robust, more secure and faster than any NoSQL wankery for this kind of use case.
Postgres is robust, feature-filled, and scalable. You dont have to want all three to extract value from it.
There are even cases where performance has been found to be better than e.g: mongo: source: http://obartunov.livejournal.com/175235.html
There is a lot of momentum behind the improvements/additions. We will be seeing much more NoSql in 9.4, 9.5, and beyond.
And yet appending a single event (you dedupe for each event, right?) takes half a second. That's an eternity!
Even so, there are a few factors to consider:
- Half second dedupes are for users with 100k+ events, which is <<1% of them.
- We batch events for ~5s before adding them to the cluster, so we aren't deduping for every event -- only once per ~5s of events per user.
If this becomes an issue, we can remove the deduping from normal operation and only call it when we're backfilling / updating events. Even so, we still need this function to exist, and the 100x performance improvement is very helpful.