Hacker News Reader

Comment by blizkreeg | Hacker News Reader

Does anyone here use compression on all of their cached content? Our Redis cache store has ballooned up to about 8G (we store a lot of html/json fragments) and is unwieldy to ship around when we want to debug bad data bugs on our dev machines. We are experimenting with lz4 compression now and the speed-compression ratio tradeoff looks pretty good with it.

What has been your experience with Rails caching + compression?

View on HN

If you're using Readthis[0] for Redis caching you can use an alternate marshaller to avoid the size and performance overhead of marshalling all objects through Ruby. If you aren't using Readthis you really should, it's faster than RedisStore, more customizable, and actually maintained!

Mandatory disclaimer, I wrote the gem.

0: https://github.com/sorentwo/readthis

Does marshaling pose a problem (leading to cache flushes) when you upgrade to a new Ruby version?

No, that hasn't posed a problem in my experience. Note that all of the stores rely on Ruby marshalling underneath. To my knowledge Readthis is the only cache that lets you choose something else like pass through, JSON, Oj, etc.

You will definitely have to flush before switching from no compression to compression or changing marshallers though.

I wrote a small snippet to share cache (particularly session) between a php webapp and rails using Dalli long back. It might be completely broken by now, but worked brilliantly back then.

it was based roughly on this https://gist.github.com/watsonbox/3170415

That sounds like a job for phuby! https://github.com/tenderlove/phuby

No, not really.

Oh man, this is great! I'll try to add you in to the Redis benchmarks in this post, or at least talk about using you instead of redis-store. I was very frustrated with the state of redis-store when writing this post.

If you have problem mem space you can use to Model cache instead of the view cache. In many use cases the view render is very fast , except if you have to many logic inside it . You can save memory without save html tags , considering that many cached views have the some models/record in input

My experience has been the exact opposite (even with simple views); the view layer always incurs the most CPU time. With the proper indexes and eager loading, data retrieval is incredibly quick.

Just curious, what are you using: ERB, Haml, Slim? According to this gem Haml could be a lot faster, but I haven't had the chance yet to check it out: https://github.com/k0kubun/hamlit

Out of interest, what database do you use? And how many joins do you normally do per view?

We use postgres and once you get to around 1m records, even COUNT is slow as it does a full table scan, even with indexes.

Same experience here. View renders have been the dog, the db queries with the right indexes and done smartly, have been fast.

Memcached is probably more suited for storing html/json fragments than Redis. The Dalli[0] memcached driver does compression for you.

0 - https://github.com/mperham/dalli

Redis is perfectly well suited to storing html/json fragments. It is as fast or faster, has built in clustering as of 3.0, has customizable expiration behavior, and is probably already used by your infrastructure. Most of the ActiveSupport caching compliant libraries will support optional gzip compression., Readthis certainly does.

There are benchmarks on the project that illustrate the performance edge it has over Dalli as well.

The bigger problem is that because of how awesome Redis is, it tends to get used for lots of things and RAM is finite. There's lots of configurations for Redis for how it handles running out of memory and nobody looks at these until they've run out of RAM and important things start disappearing. I can't count how many Rails apps I've encountered that use Redis for all or a mix of caching, background queues, pubsub, and list operations.

RAM is finite whether you're using it with Redis or Memcached.

Redis provides multiple databases (0-16) for just this situation. Use one db for caching, another for Sidekiq, and another for ad-hoc tasks. Redis can also be configured to evict entries with an expiration first, which means cached data will be cycled through while the important bits linger safely.

Not a good idea, I'm afraid. Your ad-hoc will cause the caching to stutter and the job queues to stagger. Redis is a single-threaded server and shared databases, well, share the same process. The common practice is to use dedicated Redis servers, one for each database. This also has the nice side benefit of allowing you config each as needed (e.g. LRU eviction for cache, persistency for jobs, etc...)

Reply on news.ycombinator.com