K/V Benchmark: PostgreSQL vs. Redis vs. Memcached | Hacker News Reader

K/V Benchmark: PostgreSQL vs. Redis vs. Memcached

cybertec-postgresql.com

40 pointsklaussilveira13 comments

I wonder why such an old version of Redis was used in these benchmarks? Redis 3.0.7 was released back in January 2016. The current stable version as of this writing is 6.2.5.

EDIT: Additionally the version of memcached used in the benchmarks was from July of 2012 -- even older!

Well the site does seem to be some sort of PostgreSQL consultancy.

In the "Other thoughts/notes" he acknowledges all of the tested software was old. (And it being I/O bound). The whole thing is more of "this is kind of interesting, you should do this test", than hard facts. I'd do it myself today but I have a busy one.

That doesn't answer the why though. It just makes it seem suspicious. How hard could it have been to use the latest versions of the non-Postgres software? Even as a "this is kind of interesting" experiment it seems like you'd have to go out of your way to install such old versions.

I could be wrong but it seems like on any recent Linux distro it would be harder to install these older versions… doubling the suspicion factor for me at least

Neither this article nor the referenced DZone one actually benchmarked Redis and Memcached. The numbers are taken from the 2016 paper: https://www.researchgate.net/figure/The-calculated-time-to-w...

Useless numbers. The fact they even tried to do this on the same machine shows they haven't a clue about benchmarking.

> the results were pretty horrifying indeed and one should rather not use Python for benchmarking databases – a lot of CPU time just disappeared somewhere!

The measurement will be dominated by network IO time. Does the driver wait for acknowledgement when writing? Does it wait for each read before transmitting the next query? These details matter.

Postgres reads might be so fast because the test script reads keys in the order they were written. Postgres benefits from cache locality, as the table file maintains that order.

also the value data structure is way to beneficial for postgres. the value should be a string and change in size and if the value changes more table bloat is occured so that a fillfactor might be a good idea. also most cache at least support a ttl which might be even worse for postgres on the query side since it needs to evaluate the expiration date on the client side while redis supports that directly.

> also most cache at least support a ttl which might be even worse for postgres on the query side since it needs to evaluate the expiration date on the client side while redis supports that directly.

Afaik redis will check TTLs on access as well as in a background process, so postgres isn't really behind on this one.

I wonder how Sqlite would fair, especially for the read test.

I would note that these rows have almost no content in them.

In my experience, your cache entries are usually gonna be a ton of pre-serialized bytes or some json blobs -- which could be on the order of kbs or mbs. That'll be where redis/memcached start to really shine.

I was thinking the same. If you have wide rows in Postgres then read performance will drop quickly. Of course if you can keep it in cache then that is great but for a mixed workload you aren't likely to see a hot cache all the time

Reply on news.ycombinator.com