Redis-dump: Backup and restore Redis with JSON | Hacker News Reader

Redis-dump: Backup and restore Redis with JSON

github.com

33 pointsdelano8 comments

It's a good effort, but it is not the right approach IMHO. Redis is very good at producing point-in-time snapshots via RDB files, so I think the right thing to do is to create a tool, or add this feature to redis-cli, so that it is able to translate RDB files into JSON and the other way around.

Also I think that a "generic" JSON is not a good idea, since a single JSON value can be a multi million elements list or alike. The best thing to do is to create a JSON that is also specifically conceived to be both valid but to also have a single element per line, so that can be parsed much faster and with minimal state without a real JSON parser if needed.

Salvatore, thanks for taking a look.

or add this feature to redis-cli, so that it is able to translate RDB files into JSON and the other way around.

Absolutely, that would be fantastic. I'm not a C guy so I wrote something quickly in Ruby to solve my problem.

It does do two things that aren't possible with RDB though: it can backup by database and/or a filter on the keys and it can output to STDOUT so it can be piped to other processes (I use it to encrypt the data without an intermediary file).

I use it in production but it's a proof-of-concept more than anything.

Thanks for the reply delano, actually it is possible to build both features in the RDB -> JSON dump (filter by key pattern, type, single DB, and so forth). But that would be much faster.

Your approach is also interesting but IMHO the main limit is that with the current Redis API there is no robust way to iterate keys in a good way if the key space is very big, without blocking the server and using a lot of additional memory, and it is also not possible to easily get big sets in ranges. Also what happens if somebody is writing to the DB while you need the dump? It is no longer point-in-time.

Btw since this is an interesting feature to have I'm adding a feature request in the github issues for a redis-cli feature able to do this work.

EDIT: feature request added -> https://github.com/antirez/redis/issues/288

actually it is possible to build both features in the RDB -> JSON dump (filter by key pattern, type, single DB, and so forth). But that would be much faster.

Absolutely, that would be awesome and definitely a lot faster.

feature request added

Thanks for doing that. I'll continue my reply there.

How about using something like BSON to reduce overhead? (and writing it in C would help). http://bsonspec.org/

    def dump filter=nil
      filter ||= '*'
      entries = []
      each_database do |redis|
        chunk_entries = []
        dump_keys = redis.keys(filter)
        . . .

http://redis.io/commands/keys

Warning: consider KEYS as a command that should only be used in production environments with extreme care. It may ruin performance when it is executed against large databases. This command is intended for debugging and special operations, such as changing your keyspace layout. Don't use KEYS in your regular application code.

Yep, good eye. It's not an ideal approach but using KEYS is the only option for a non-specific tool.

The right solution is to have something similar build into redis-cli (see antirez's link above).

Why not just copy the rdb file?

Reply on news.ycombinator.com