Snappy Dashboards with Redis
blog.togo.io
blog.togo.io
Introducing new services/technologies just complicates things on both the application and ops layer (even though the author did say he was using RedisToGo).
Make sense?
I don't know how well Mongo's map-reduce works, but in Postgres, COUNT(star) [1] does not perform well for very large tables (e.g. 100 million rows). You wouldn't want to be doing a COUNT(star) once per minute for each customer that had their dashboard open on a plasma screen.
Of course, there are other solutions to that problem: generate the counts on some more feasible schedule and cache them; have a read replica used for analytics queries; shard by customer and have no large customers.
I don't know whether their scale strictly requires the Redis solution, but in any case there are situations where it's not as simple as "throw it in Postgres and use an aggregate function".
[1] "star" instead of an asterisk to avoid HN thinking I'm trying to write in italics.
It does indeed sound like they're storing every click: that's precisely why using aggregate functions would be expensive.
samstokes: I said you can't do it with COUNT()
guywithabike: You absolutely can do it without COUNT
Then we agree.
Appreciate the time and thoughtful response.
Our API is a Rails project and though we do a lot of work that's "off the rails," so to speak, we do try to follow a convention where possible. A convention that's worked well for us is to use Postgres only for database-backed models in the app. So when it came time to solve this problem, Redis seemed like a good choice.
Map/reduce isn't an option here. First, it's too slow. Second, it's asynchronous by nature, which doesn't work well when trying to load a webpage. We did consider using it in the background, however, to generate the counter-caches.
Thanks for reading!
Just like this also has nothing intrinsically to do with Redis and could be any regular ole key-value store.
1) They're so easy to move out to something else
2) You main DB probably runs on very expensive hardware, unneccessary expensive for things like this. This probably does not need that expensive san backing it. Not all data is created equal.
3) Redis does this faster
4) Redis is simpler
Of course there are some advantages to storing everything in the same DB, but to me, this seems like a good example of "use the right tool for the job". But, I haven't actually tried this in Redis, so what do I know..
> If Redis isn’t available, for whatever reason, we could rebuild the gaps from the canonical data in Mongo. We’ve never had to do this.
I'm not sure this is so straight-forward.
So, this is a guest post by Sqoot hosted by togo.io, who owns the redistogo.com. Can anyone explain redistogo.com's pricing? I understand the value in not running your own service dependencies, but the redistogo.com prices seem really high to me. I was under the impression redis is fairly easy to manage. What kinds of operational tasks does redistogo.com perform for a redis instance that would warrant such high prices?
Speaking for ourselves, we've historically preferred to have someone else manage our infrastructure. Setting up Redis is fairly trivial, I agree. However, setting up a secure machine out on the internet and making sure it's always there is less trivial. Rather than juggle an ops/engineer role, we just double down on engineering. I'm sure at some point this may need to change. Hopefully, when that day comes we can afford a sysadmin!
On a related note, it feels good to know that everything our app needs to run is in the code base. Back in my C# days, I remember relying on stored procedures that were configured manually at the database level. Rails fights that with migrations and the callback chain too, so I guess that thinking has sunk in for me.
Thanks for the comment.
What does hosting with Heroku have to do with using a trigger?
> On a related note, it feels good to know that everything our app needs to run is in the code base.
Except the database schema and any additional indexes you need to make it not perform terribly. All basic setup, just like creating triggers.
> Rails fights that with migrations and the callback chain too, so I guess that thinking has sunk in for me.
The problem with doing stuff like this in a callback is that an additional query is sent to the database which can be a big performance problem if the insert load is high. I generally agree that complex logic should be avoided in triggers and is better left in the application code in most cases but incrementing a counter is about as simplistic as you can get.
save 1900 1
save 1300 10
save 160 10000
I should add, I also use redis to handle some pretty large calculations... zrange's for distance calculations intersections and so on... I did some benchmarking and found that at least in ruby (1.9.3), this was more efficient to load up redis for the sets and sorting, fewer GC hits and faster sorting intersecting... I'm thinking it might be good if for these one off frequent compute tasks, I run on a redis instance that has no save, especially considering I expire/delete the keys immediately after running my calculations.
Also, if you don't do this already, look into using bitsets to track users. As long as userID's are integers, it's real easy it saves a lot of space.