Redis: the AK-47 of databases | Hacker News Reader

Redis: the AK-47 of databases

flazz.me

158 pointsflazzarino77 comments

The title randomly reminded me of the AK-47 quote from Lord of War (an actually decent Cage film, and definitely a good bit of monologue):

Of all the weapons in the vast soviet arsenal, nothing was more profitable than Avtomat Kalashnikova model of 1947. More commonly known as the AK-47, or Kalashnikov. It's the world's most popular assault rifle. A weapon all fighters love. An elegantly simple 9 pound amalgamation of forged steel and plywood. It doesn't break, jam, or overheat. It'll shoot whether it's covered in mud or filled with sand. It's so easy, even a child can use it; and they do. The Soviets put the gun on a coin. Mozambique put it on their flag. Since the end of the Cold War, the Kalashnikov has become the Russian people's greatest export. After that comes vodka, caviar, and suicidal novelists. One thing is for sure, no one was lining up to buy their cars.

Only tangentially related :) but I thought worth sharing. Anyone care to write a redis version?

It reminded me of the AK-47 quote from Jackie Brown:

AK-47. The very best there is. When you absolutely, positively got to kill every motherf@#$%r in the room, accept no substitutes.

I'll only be pedantic about them rarely being forged (mostly stamped) and definitely still having problems of overheating. Try firing one on full auto for more than one clip - better have gloves...

They are great weapons though. I've never cleaned mine or had a jam. I hope this database is that good!

If we're being pedantic, they use magazines, not clips :)

Actually, the words are interchangeable. You've heard of "Banana Clip" I'm sure. Clip is just slang for magazines, but the short word is for some reason associated with small magazines. The manuals all call them magazines - rifles or pistols.

My understanding of it (and wikipedia seems to back me up) is that the term 'clip' when used properly refers to the device used to assist in loading a magazine.

For example, you use a clip to load ammunition into the (internal) magazine of an M1, and you could use a clip to load ammunition into a "banana clip", which actually refers to a style of magazine.

You're referring to "stripper clips" that were invented before guns had removable springed magazines. Those are a type of clip (and lead to the term), however the slang term clip doesn't refer to pistol magazines - it's merely slang for a magazine. It's a common misconception and what I was responding to.

In fact, the internal pistol magazine of a C96 pistol is still a magazine, but it loaded by a stripper clip, and some guns like the Japanese Type 3 are fed directly from stripper type clips - but those aren't "speedloaders" as the article suggests.(To juxtapose the concepts completely...)

http://en.wikipedia.org/wiki/Stripper_clip http://en.wikipedia.org/wiki/C96_pistol http://en.wikipedia.org/wiki/Type_3_Heavy_Machine_Gun

I'm refering to the wikipedia article on ammunition clips in general: "A clip is a device that is used to store multiple rounds of ammunition together as a unit, ready for insertion into the magazine or cylinder of a firearm. This speeds up the process of loading and reloading the firearm as several rounds can be loaded at once, rather than one round being loaded at a time."

https://secure.wikimedia.org/wikipedia/en/wiki/Clip_%28ammun...

Using the term "clip" to mean "magazine" is both slang, and incorrect.

All I'm saying is "clip" doesn't mean "magazine for a pistol" - which is the misconception. Clip, as used commonly in slang, is for any magazine. Rarely would one refer to stripper clips as just "clips" - usually it's "stripper" or "strip" to avoid confusion.

No, they are not interchangeable, and no, they do not mean the same thing.

Some nice illustrations I found: http://i.min.us/ie52d2.jpg http://i.min.us/icklUo.jpg

To any firearms enthusiast, hearing clip and magazine being confused is like hearing nails on a chalk board. Imagine someone at your work insisting that Java and Javascript are the same language.

"Imagine someone at your work insisting that Java and Javascript are the same language."

This is a rather poor analogy. Someone confusing Java and Javascript might believe they can paste their Javascript code into a Java program and have it work. Nothing of this nature will result from confusing clips and magazines. A shooter isn't going to expect to be able to use their Mosin-Nagant clips with their PSL, any more than they would expect for the PSL magazines to be valid replacements for their Winchester 1895 magazines.

It is incorrect, but it's not all that bad. 95% of the time, it is clear from context; the remaining cases primarily concern weapons with a fixed magazine, for which the user probably does know the difference.

No true Scotsman... Really - this is a matter of taste (what slang you prefer). As a former arms dealer / in a family of government weapons contractors and servicemen, we all use "clip" as slang.

It's like everyone calling car wheels "rims"

Really? I am in the Canadian Forces infantry, and if anyone ever used the term "clip" in reference to a magazine, and was actually serious, I'm pretty sure they would get beat up. May I ask what country you are affiliated with and in what branch of the military those servicemen are? It is interesting to see such a basic difference in terminology.

USAF and small weapons / ammo manufactures. You're right - military nomenclature (US) uses very precise (and verbose) terminology: "magazine" is always used strictly. Though the words "gun" and "rifle" are dropped in favor of "weapon" in some cases. What I find is that the longer you're out, the more the language becomes civilian - unless you're a hardcore / die-hard kind of person.

You actually own a AK-47? Out of curiosity, why?

Why do people own TVs?

They're fun.

Well, few people own true AK47's, most own copies of the Russian AKM which is made from stamped parts (as opposed to machined on the true AK47 [1]. I own a semi-auto Romanian WASR 10 - which is very common in the US (PA). They go for ~$300 at a sporting goods store.

It's a lot of fun to shoot, the AK is cheap reliable, historical and can be used for fun, hunting (yes) and in survival situations. I actually do a "StartupGuns" [2] event in Pittsburgh where a bunch of us go shooting like OpenCoffee or similar.

[1] http://en.wikipedia.org/wiki/Akm ("Features") [2] http://www.facebook.com/group.php?gid=207534415228

Supposedly the Ak 47s used in the film were real, as real Ak 47s are less expensive than fake ones. But I'm not sure where I heard this, or if it's even true.

Well, have I got the definite resource for you! http://www.imfdb.org/index.php/Lord_of_War

"According to Director Andrew Niccol in the DVD commentary, the guns were real guns rented from a real arms dealer, as it was cheaper for the production to rent 3,000 real guns than to rent 3,000 blank converted props. "

Didn't Brandon Lee die because someone mistook a real gun and bullets for prop ones?

Not quite.

http://en.wikipedia.org/wiki/Brandon_Lee#Death

> (...) behind schedule (...) they decided to make dummy cartridges from real cartridges by pulling out the bullets, dumping out the propellant and reinserting the bullets. However, the team neglected to remove the primers, which, if fired, could still produce just enough force to push the bullet out of the cartridge and into the barrel. At some point prior to the fatal scene, the live primer in one of the improperly constructed dummy rounds was discharged by an unknown person while in the pistol, leaving the bullet stuck in the barrel.

> (...) the same gun was later reloaded with blank cartridges and used in the scene in which Lee was shot. When the first blank cartridge was fired, the stuck bullet was propelled out of the barrel and struck Lee in the abdomen, lodging in his spine.

I think most movies use the Chinese Type 56 assault rifle, which is a Kalashnikov knock-off and is far more common in the US than the real thing. Most people can't tell the difference between the two.

BTW, I highly recommend "The Gun" by C.J. Chivers. It's a fascinating historical treatise about this weapon.

How do the Chinese rifles compare? Are they similarly simple & resilient?

Yeah, pretty much. It's an extremely simple (albeit clever) design so there's only so much you can get wrong when you're copying it. Note that the Type 56 like most other AK knock-offs are really copies of the later AKM "improved" model, which was cheaper to manufacture. So comparing it to the original AK-47 is probably not fair. Then again there aren't many copies of the original model around anymore.

If I recall, the AK-47 had an interesting role in a trilogy of time traveling books.

http://en.wikipedia.org/wiki/Weapons_of_choice

In the books, a near future (near our present) naval task force accidentally travels back in time to WWII. The U.S. eager to get a leg up in the war asks the chrononauts to give them future tech. The only problem is that WWII American doesn't have the manufacturing technology to produce any of the current equipment. So the chrononauts dig through their archives and find plans for the AK-47, which, as it turns out, is manufacturable in early 1940's U.S. thereby giving the Americans a weapon from the future, but still old enough that they could be produced in bulk.

Actually the Chinese copies are considered pretty good, but are pretty rare because the US embargoes "assault" weapons from there for civilian use with State Dept's permission (never).

Most of the cheap ones in the 90's were Maadi (Egyptian), and now they're usually Romanian.

Well, not all knock-offs are bad :) My experience with both the older (state-built) and newer (Norinco) Type 56's has been generally good.

I doubt the actual weapon was profitable. They literally made like 75 million of the things. The Soviets may have made a few bucks setting up factories in the 3rd world for licensed production.

Colonel Kalashnikov was paid by a west german company that owned the marketing rights to the name - I think they made aftershave!

The largest chunk of export from Russia is oil and gas, not weapons.

Hollywood doesn't really care.

This is somewhat related:

http://i.imgur.com/Tz0m3.png

  > In the spirit of full disclosure, I’m a newb to Redis. My
  > knowledge is basically the contents of this post (at the
  > time of writing). I don’t use it in production (yet).
  > Likewise I’m no expert in AK-47s (I’ve never even fired
  > one) or guns in general. I’m aware via notoriety alone.

Fantastic quote. So you're hardly familiar with either component of your simile, but you'll post it anyway? I really don't expect New Yorker-esque editing from bloggers, but that's simply not very substantial writing.

No wonder people make fun of the so-called "NoSQL movement."

Expert knowledge of an AK-47 wouldn't have improved the post and he managed to expose me to some bits of Redis that I wasn't previously aware of. That makes it more substantial than half the shit that ends up on Hacker News. :)

I thought it was a well written article that was a good read. Possibly the author was demonstrating humility rather than ego.

Consider the blog post a conversation starter. You don't have to be an expert in a topic before you can start a useful online discussion about it.

Yes, bloggers talk about things they just learned about rather than something they're 360-degree experts about.

But that's good. It helps them -- it reinforces their learning and they certainly find out if they got something wrong -- and it helps us -- readers see a lot of articles pop up right when people are interested in and passionate about learning something, rather than months later when they know it all and have moved on.

Early blogging is quick iteration of knowledge. Blog about something you learned today, not years ago.

Your last sentence is pretty flawed/ad-hominem. That being said, I do agree in the sense that the comment at the end of the post annoyed me and degraded the value of the article. Certainly, blog about whatever you want, but if you just picked up some new technology, don't write a link bait title and post it on HN. HN needs more breathless hype articles like I need a bullet in the head.

It still has value according to the two-step HN algorithm of fact-finding.

step 1. Someone posts something (anything will do) on a blog

step 2. HN tears it apart and reconstitutes it with some decent approximation of truth and usually a fair bit of insight.

Basically, the source content is just the jumping off point. The blogs serve as a focal point for directing the HN hive's attention for a while.

You could post a story titled "Sarah Palin is more influential than the Pope but less influential than the iPhone" and fill it with lolcat pictures and still get an interesting discussion here.

Why? Because smart people are awesome.

Very true, thats why most of the time I spent time reading the hacker news discussion rather than the original link.

"According to AK-47 legend, assault rifles were not popular because of their tendency to consume large amounts of ammo. The Soviets embraced the idea and simply supplied their troops with more ammo. Kinda like Redis and RAM."

That's obviously just a legend. This debate was actually carried out in the 2nd half of the 19th century, starting with the Civil War. (Many repeating rifle designs were offered to the Union. Quartermasters repeatedly shut them down on account of expected difficulties with ammo supply. After much time and effort, a few repeating rifles made it through, particularly to the cavalry, where they proved quite decisive in the last year of the war. See Five Forks, for instance.)

The British army only just relented - the new SA80 is the first fully automatic rifle to be standard issue.

That's not exactly true. The SA80 isn't exactly new. It was in use during the Falklands for example. And the British Army used a version of FN FALs before the SA80s were adopted, and those were also automatic rifles.

Now, I'm pretty sure 'automatic' fire was burst limited. But that's pretty much nearly universally accepted as a good idea in main infantry rifles.

It was standard issue from about 1985/1986 - before that they used self loading rifles ie. one trigger - one round.

The LSW variant has always been fully auto and is 25 years old, and the L85 is to this day semi-automatic and is the main rifle in use in the British army. I don't see how they've "only just relented". Full auto isn't that useful anyway! The ability to empty your magazine in a second comes in handy less often than you'd think.

In fact the US Army's M16 was changed from automatic to semi-automatic, so it seems there is a movement in the other direction.

Being the AK-47 of databases is not a bad way to be known.

That said I hope the author will revisit the topic when he has more knowledge of the subject matter, such as when using it in production for a while on a fair sized project with a lot of users. That's where the rubber meets the road, first impressions tend to gloss over the areas where the meat is.

> That's where the rubber meets the road, first impressions tend to gloss over the areas where the meat is.

One hopes the meat is not between the rubber and the road. That would be unfortunate.

Eek. Locking the entire data set to perform transactions? Isn't that so... MyISAM? Does rolling the transaction back work? The optimistic lock is not a great solution either: instead of deadlocking while waiting for a lock to be released, you are now stuck in a while (!saved) loop. Imagine two web pages trying to update the same resource at the same time.

Also this part of the config:

  save 900 1
  save 300 10
  save 60 10000

is not at all obvious. Maybe it's better documented in the sample config.

According to the first comment, the part about locking the whole data set for MULTI is not accurate. Only the connection calling that is stalled, which makes perfect sense.

Ah: http://redis.io/commands/multi. So the problem with this approach is that there are no consistent reads. If I say:

  MULTI
  get foo # Returns 'abc'
  set foo 'abcd' # (as in application code said foo += 'd')
  EXEC

while another process sets foo to 'xyz' between my get and set, I just overwrote that value. No serializable transactions.

im not exactly sure about that but i know that at least for integers there are atomic update commands and that for strings there are atomic append commands.

so for this you would really just want to do: APPEND foo d

or for more complex situations you could do use WATCH instead of MULTI so that you can get the intermediate results. so for your example:

WATCH foo GET foo ->value //calculate new_value MULTI SET foo new_value EXEC

now if at any other operation (not part of this connection) modifies foo then the transaction will be aborted.

GET foo -> value //application logic to update value GETSET foo new_value ->prev

then prev will contain the value of foo at the time it was set, so you can then use application logic to roll back or redo the transaction

Redis is single-thread one-process program. Since the execution only happens when EXEC command received, such scenario cannot exist.

MULTI/EXEC is completely atomic and transactional.

You can perform just writes inside the block. If you want atomicity with reads in the middle, you need to use WATCH/MULTI/EXEC that uses an optimistic locking algorithm.

Ah, that makes sense. I imagine allowing atomic reads without WATCH would complicate the code quite a bit.

Indeed, not very Redis-ish :)

With WATCH is very easy. WATCH this keys. Perform your reads. Prepare to write in a transaction. If at least one key was modified in the meantime the transaction will fail.

MyISAM doesn't support transactions.

My point exactly. MyISAM supports locking the entire table, performing the actions you want to perform and releasing the lock. And there is no rollback, if something goes wrong. The only thing this guarantees is that you have exclusive access to the data. Seems to me that this is what MULTI does as well.

EDIT: looks like there is rollback support: http://redis.io/commands/discard

It's called optimistic for a reason. You assume there is low contention and configure your client to have random back off.

Maybe not, but I found the docs in about 10 seconds.

http://redis.io/topics/persistence

Warning, the documentation was not updated with the current behavior!

Now I clearly state in the doc that transactions are reflected also in the append only file, so there is no chance of partial reply.

Transactions are also applied to the master -> slave link, so also slaves will either have the full transaction or nothing.

Thanks to @garybernhardt on twitter for noticing this problem.

The configuration options you have quoted are made extremely obvious in the sample configuration and are explained very well. See:

  # Save the DB on disk:
  #
  #   save <seconds> <changes>
  #
  #   Will save the DB if both the given number of seconds and the given
  #   number of write operations against the DB occurred.
  #
  #   In the example below the behaviour will be to save:
  #   after 900 sec (15 min) if at least 1 key changed
  #   after 300 sec (5 min) if at least 10 keys changed
  #   after 60 sec if at least 10000 keys changed

My point is the OP talks about how all the options are self-evident.

Factual incorrectness aside, and even speaking as a person that doesn't have much to do with anything other than SQL RDBMSes: this style of presentation piqued my interest in Redis. Seeing how fast and easy it was to get up and running, seeing how simple the protocol was, and getting a smattering of its features in this very terse format was effective on me.

The author needs to know more about databases before claiming something is the AK-47 of the subject. It kinds of reminds me of those guys who think they are gurus just because they know some rarely used Linux commands, while in fact they don't have good understanding of OS. :)

What do you think he's missing? The comparison seems rather apt to me - Redis is shockingly simple and surprisingly effective compared to other persistence options.

The information about transactions in this article is wrong. Read the redis docs he actually points to for an accurate description.

This article should not get upvoted.

and the title?

"The entire dataset is in RAM."

How can they guarantee that, when RAM is little more than a disk cache on modern operating systems? Is there a syscall that prevents specified pages of virtual memory from being paged out to disk? Or is Redis implemented as a kernel module?

Your swap should be off, and there is mlockall(2).

The kernel does let userspace have a /little/ fun on its own.

Redis uses a memory layout that, unless the instance is completely idle, will make it very easy to touch every page even if there are just a few queries per second.

So fare in two years we never saw an instance where there was a latency problem because part of the dataset was swapped on disk by the OS. This is why we currently don't use mlockall().

In another comment, someone compared the PUB/SUB feature of Redis to ZeroMQ. Can the PUB/SUB feature of Redis function as a queue system for many common uses, or is there another significant difference that sets a full queue implementation apart? I ask because I am not particularly familiar with Redis, my queue usage has been fairly simple so far and I would enjoy simplifying my technology stack by combining the two if it makes sense.

Redis is great for small things. Massive scale players in the write-heavy owrld can rarely consider it in its current form.

"Pub/Sub is a unique and useful feature."

Pub/sub exists in a multitude of different pieces of software. ZeroMQ has a pub/sub feature. PostgreSQL has the NOTIFY/LISTEN command which is also a Pub/Sub channel allowing all those listening to be notified of certain events.

This is by no means unique to Redis. It is very useful, so I'll agree with that!

I served in the bulgarian army with this weapon, and it was very easy to handle. That's all I can say

But every weapon was a bit off, so you had to get used to it (I guess that might be the same for all weapons).

The total number of the AK-type rifles made worldwide during the last 50 years is estimated at 90+ millions. Does the title implies that Redis will handle this amount of values too?

This one of very few google hits for the phrase "get the juxt".

AK of DBs should be used for some Russian DB! or Chinese...

okay okay. i hereby patent the definition 'an iPhone of databases' :)

Reply on news.ycombinator.com