Why use an array instead of a separate table? Are you trying to avoid a JOIN?
In particular, to compute where a user drops off in a funnel, I need to scan one array from left to right, and I don't need to do any joins. This shards very well, since all of a user's data lives on one shard, and most of the queries are aggregations, which are simple to reassemble from subqueries.
Postgres is robust, feature-filled, and scalable. You dont have to want all three to extract value from it.
Because even a shitty RDMBS is more robust, more secure and faster than any NoSQL wankery for this kind of use case.
There are even cases where performance has been found to be better than e.g: mongo: source: http://obartunov.livejournal.com/175235.html
There is a lot of momentum behind the improvements/additions. We will be seeing much more NoSql in 9.4, 9.5, and beyond.
And yet appending a single event (you dedupe for each event, right?) takes half a second. That's an eternity!
Even so, there are a few factors to consider:
- Half second dedupes are for users with 100k+ events, which is <<1% of them.
- We batch events for ~5s before adding them to the cluster, so we aren't deduping for every event -- only once per ~5s of events per user.
If this becomes an issue, we can remove the deduping from normal operation and only call it when we're backfilling / updating events. Even so, we still need this function to exist, and the 100x performance improvement is very helpful.