724 karma · joined October 8, 2014
1. It's tricky to scale out a Redis node when it gets too big. Because RDB files are just a single dump of all data, it's not easy to make a specific partitioning of the dataset. This was a very important requirement for us in order to ease scaling (redis-cluster wasn't ready yet -- we've been following that carefully).
2. When you store hundreds of GB of persistent data in Redis, the startup process can be very slow (restoring from RDB/AOF). Since it can't serve reads or writes during this time, you're unavailable (setting up a slave worsens the following problem).
3. The per-key overhead in Redis (http://stackoverflow.com/questions/10004565/redis-10x-more-m...). We have many billions of sets that are often only a few elements in size -- think of slicing data by city or device type -- which means that the resulting overhead can be larger than the dataset itself.
If you think about these problems upfront, they're not too difficult to solve for a specific use case (partition data on disk, allow reads from disk on startup), but Redis has to be generic and so can't leverage the optimizations we made.
That said, we have looked at Druid, which is also a good example of using lambda architecture in practice (http://druid.io/docs/0.8.0/design/design.html -- note the historical vs realtime distinction). They use many of the same design principles as us, and one of our sub-systems is very similar to it. We still believe the pre-aggregation approach is critical for performance in our use case, though. Lastly, when we started building the architecture (mid-2014), Druid was very new, and I'm generally wary of designing everything around a new and potentially unstable piece of software.
To give a little more context around how web sessions work, the first event creates the session and each subsequent event triggered within 30 min of the previous one will be considered in the same session. The session length is calculated as the time between the first event of the session and the last.
We've been thinking a lot about this exact issue over the last few years and have come to the same conclusion: when dealing with a platform that provides a ton of value out of the box, you need to be somewhat opinionated, and you can't expect to cover every use case. Different customers can have very different questions about their data and unique insights that they want to discover.
To address that, we offer direct SQL access to the raw data in Amazon Redshift. While Redshift isn't the best solution for powering high-performance, realtime dashboards, its main benefit is that it supports most of the SQL standard, which means that you can answer close to any question about the data. This duality of the fast, easy-to-use dashboards that provide immediate value and the flexibility of Amazon Redshift for deeper, customized queries has helped a lot of our customers overcome the problems you describe.
Hope this makes sense, and we'd love to hear your feedback!
We're hiring backend, frontend, devops, mobile, and test engineers who are looking to work at an exciting early stage startup (20 people) that has grown very quickly over the last 6 months and is about to go through a ton more growth over the next year. We're very generous with salary and equity because we have a high bar for hiring (https://amplitude.com/team).
Amplitude is in the mobile app analytics space, and we're bringing the next level of analytics to app developers out there. Instead of just showing people numbers about their app, we're helping them find real insights and take action based on those findings. Check out our blog (http://blog.amplitude.com/) to learn more about how we think about analytics.
We don't hire for specific skills since we believe any good engineer will pick them up, but some of the technologies we use include: Python, AngularJS, Java, PostgreSQL, Redis, Amazon S3. We spend a lot of time thinking about distributed systems for data processing and querying as well as the user interaction/experience of dealing with a very complex and powerful analytics product. If any of this sounds interesting to you, check out our careers page (https://amplitude.com/careers).
Apply to: careers@amplitude.com
We're hiring backend, frontend, devops, mobile, and test engineers who are looking to work at an exciting early stage startup (11 people) that has grown very quickly over the last 6 months and is about to go through a ton more growth over the next year. We're very generous with salary and equity because we have a high bar for hiring (https://amplitude.com/team).
Amplitude is in the mobile/web app analytics space, and we're bringing the next level of analytics to app developers out there. Instead of just showing people numbers about their app, we're helping them find real insights and take action based on those findings. Check out our blog (http://blog.amplitude.com/) to learn more about how we think about analytics.
We don't hire for specific skills since we believe any good engineer will pick them up, but some of the technologies we use include: Python, AngularJS, Java, PostgreSQL, Redis, Amazon S3. We spend a lot of time thinking about distributed systems for data processing and querying as well as the user interaction/experience of dealing with a very complex and powerful analytics product. If any of this sounds interesting to you, check out our careers page (https://amplitude.com/careers).
Apply to: careers@amplitude.com