Under the hood we're using NSQ as a queuing layer, S3 for storage and batched uploads, Amazon Aurora (for S3 indexing), DynamoDB for billing and metadata storage, and several distinct Go services that handle batching, transformation, schema updating, deduplication and internal consistency checking.
It's been in beta for several months and we're loading about 10,000 events per second into customers' databases today.
Instead, when we put a new object, we update a table in Aurora which tracks all of the relevant objects. That way, we can query information like "what objects were uploaded in a certain time range" very quickly.
That is, what happens if a db write fails? How are you handling concurrent updates? Is there a reconcile process that runs periodically?
(those details would be an awesome engineering blog post :) )
Edited to add:
Any reason that AWS Lambda didn't work out? Was it due to the public endpoint requirements?
Double Edited to add:
I totally geek out on composing AWS Services, and this is fascinating to me.
We are eagerly waiting for Lambda VPC support!
I'm happy you're interested in this sort of stuff; shoot me an email, let's chat :) michael@segment.com