Redis Loadable Modules System | Hacker News Reader

Redis Loadable Modules System

antirez.com

297 pointsdjanowski60 comments

The docs explain the module system (https://github.com/antirez/redis/blob/unstable/src/modules/I...) but it doesn't quiet explain what the system is for. I couldn't quite get that from reading the blog post or the doc page. Specifically:

What kind of modules did you have in mind when designing this? Are any of the below possible?

* ldap backend login system

* multi-tenant redis

* alternate db format/file system.

* implement a new redis data type. say built-in date support

* can i implement new commands and operators on said data types

An of course, thanks for this fantastic software. We use it everyday. Will read in more detail about this over the weekend :-)

Actually the post explains the motivation well. The basic idea is to add capabilities to redis at native speeds, without having to either convince antirez that they're necessary or forking redis.

I work for Redis Labs, and over the course of the past few weeks we've experimented with some modules using the API. Some of the things we did that might answer some of your questions:

a PAM based authentication module, which IIRC can plug into LDAP potentially.

A bloom filter implementation on top of bitmaps.

An inverted index that doesn't use existing data types but rather creates compressed indexes directly using redis strings (I wrote that one personally BTW)

An API to store images on redis and retrieve them resized or cropped.

So right now the only way to create new data types is to piggyback strings as raw memory, but the final version of the API will include actually registering new data types.

You can see the modules we've released here: https://github.com/RedisLabsModules/

> a PAM based authentication module, which IIRC can plug into LDAP potentially.

Yes, pam-ldap / pam-sssd / pam-winbind plug into various LDAP implementations (plain LDAP / ActiveDirectory) and work transparently with all PAM clients.

Maybe you can relay the information to the appropriate person, but on http://redismodules.com, I can neither right click "open in new tab" or ctrl+click to open in a new tab.

This breaks my normal browsing habit of opening new tabs as I see something that interests me, but waiting until I finish skimming the list to read.

Thanks, will let the relevant person know.

I'm not related to redis devel, but I happened to be at the conference where antirez announced the modules.

My understanding from the talk was:

* already available: strings DMA, that you can use for new data types

* in progress but soon (like, a few weeks): commands pre-hooks, that you can use e.g. for auth

* I assume soon (but I don't recall, it could be already available): operations on new data types

* I don't think I've heard anything about alternate db format, but maybe you can think of creative ways of mixing pre-hooks and strings DMA

* I'm not sure what you have in mind about multi-tenant, but I guess you can implement some primitives in a way similar to auth

What I meant by multi-tenant is a multi-user redis (just like multi-user mysql/postgresql. (ie) i can create users and have them use redis completely separately on a single instances on redis.

There's been a lot of unusual features I've been hoping Redis will support but obviously given their edge-case nature it's unlikely that they'd ever be implemented.

This is great to see. Instead of cluttering up the core engine with junk people can mix in their own functionality.

This looks interesting. I've only glanced at the docs, but it looks like this module system lets you define new commands but not necessarily new data structures (since you'd have to implement RDB/AOF serialization for those).

Are there plans for that? It's probably possible to do most of that with this module system but you'd have to serialize the data structure into a string and "type X" wouldn't return your custom data structure type.

Definitely plans to register new data types with serialize/deserialize callbacks! Thanks.

He mentioned in the keynote being able to do your own datastructures by raw access to the data and primitives like seeks and next/prev . Sounds like it's at least planned, if not roughed in.

This is great and long awaited, please consider some kind of a namespacing scheme for module-defined commands though, or things can easily become messy. In the module example the "hello." prefix is specified for all defined commands - it would probably be better to specify it once on module initialization and have all commands defined by the module prefixed with it automatically, the module registries could then manage the prefix registration/allocation to module authors.

Since Redis is BSD licensed I wouldn't be surprised if this is the start of popular paid-for proprietary third party modules.

Even if redis was GPL licensed you still could do that. Linux is GPL and there is proprietary driver modules.

There are proprietary modules for linux, and they are distributed and used, but there isn't really a full consensus on whether they are legal. You can certainly do it and get away with it though.

http://yarchive.net/comp/linux/gpl_modules.html

I once built something like this for Redis but never really thought he would do it!

https://github.com/spullara/redis/tree/plugins

I've written something similar too. Adding plugins/modules was like a rite of passage for anyone digging into the redis source. I'm definitely happy to see modules getting official support now though.

I was blown away with how easy it was and with the code quality.

If these modules are going to be as fast as the native commands, is there any reason to not lift the entire native command set out into a set of "base" modules?

They're not going to be as fast, but almost as fast. You could do that in theory but IMHO there's not much point

How people can code on airplanes amazes me. I have tried but the cramped space, the constant interruptions.. I never get much done.

Agree, I sometimes have problems even fully opening my laptop.

It's not the constant interruptions (headphones do a good job of dealing with that), its the lack of internet access that is the showstopper - I don't have all the APIs + docs in my head.

Use https://kapeli.com/dash on Mac or the open source equivalents on other platforms.

Thank you very much!

http://devdocs.io/offline works well on Linux and Windows.

I use https://kapeli.com/dash for offline documentation for when I'm on flights. Also, WiFi has been available on more and more flights. I actually have found long flights good for solid focused coding sessions (especially when I already have thought through what I want to build).

edit: woops, looks like someone submitted this at the same time. In any case, working offline also forces you to reign in your app dependencies.

Wow, it even has Hackage support! Too bad it's OS X-only.

Check out Zeal for Linux. It uses the same docsets as Dash.

Perfect - thanks!

I have all the APIs in my IDE.

As a father of three toddlers, I can tell you that sitting in an airplane for hours is pure heaven for concentration and lack of distractions.

I love working on the plane too. Noise-cancelling headphones and a 13" MacBook Pro. I usually code for around 40% of the SF to London flight. Great opportunity for uninterrupted flow.

I get some of my best work done on a plane. I flew out of LA a few months back and a family of five (mom, dad, daughter, and two young boys) had the rows behind me. They made a lot of noise and were just being kids. The dad apologized and offered to buy me a drink. I told him no need, because that activity allowed me to focus on what I was doing. I think that I added a new feature that I was mulling over during that flight.

I love coding in public spaces. Life acts as a very good white noise

This is how I try to justify flying up front (if the price works out right). I take the extra cost and attempt to "pay it back" by how many hours of extra work I can get done (or, if an overnight flight, extra work I can do the next day after sleeping the whole way instead of feeling like a zombie).

One of the companies I started a few years back was born in the back of the highway 17 Express, a bus with cramped seats that blazes down a really windy road at 60 mph. It even has wifi (for some parts of the ride). After that, working on airplanes was easy. You just need to fight a bit for elbow room and hope you aren't seated between two obese people. If you're really lucky, the person in front of you will be too concerned with others to lean their chair back...but even if they do, it's workable.

Probably helps that I'm fairly scrawny though.

@antirez, is Redis 4.0 going to be part of the more rapid release schedule (will we see it in 3, 6 months) or are major upgrades still annual releases (we'll see 4.0 in 12 months)?

Definitely part of the short release cycle, we'll probably focus on getting modules done and the features already pending into "unstable", and a few more minor stuff, and release it.

https://github.com/RedisLabsModules/redex

Shows some sample modules from Redis Labs called RedisEx, pretty useful stuff (although most of these should really be in Redis by now but at least it's an option to include as modules).

> most of these should really be in Redis by now

This exactly the point of the module system. You don't need to either convince antirez to accept/implement a new feature, or fork redis if he doesn't agree. It will make the dynamics of redis much more interesting IMO

they also announced a couple of "hubs" for hosting/discovering these plugins, plus a vetted one by redislabs

Indeed - http://modules.redis.io will be the open source place and the Redis Labs' Module Hub is at https://redismodules.com

So who is adding node.js support?!

Someone should really implement an interval module.

This custom implementation was pretty good: http://blog.togo.io/how-to/adding-interval-sets-to-redis/

I bet a package manager will follow it. redis install moduleX and then we'll have lib and target dependencies.

But everything will depend on the padding module.

and then the author will remove it one day. :)

Super excited to hear about this at Redis Conf, looking forward to implementing a few ideas.

Sounds exciting. Hopefully it won't be as under-appreciated as Apache Traffic Server's plugin system.

databases (redis included) are awesome. It's always surprising to me that we don't run web apps directly on the database; they're high-performance, durable and parallel, and for some apps are already the perf bottleneck.

I think the only reason we don't is that deployment and testing isn't straightforward. Fingers crossed antirez will get this right.

You might find CouchDB pretty interesting, then! [0]

[0] https://couchdb.apache.org/

I know them for data sync. is there a stored proc deployment feature as well?

I'm hoping there's an easy way to turn this off for the foreseeable future, to avoid all the obvious security holes this is likely to introduce.

You don't have to use it, and you should not use modules you don't trust.

In an ops-type capacity, I support developers. I keep up with news like this specifically to avoid winding up with modules somebody else trusts, but didn't inspect.

What would be a must and should probably be a default, is not allowing to load modules using the network API, only via config file.

I tried to read the article but I couldn't. It's all one big <pre> block. The word wrapping is wrong on input! Individual paragraphs are groups of individual lines and extra long lines that the browser wraps for you. So you end up with lines in the middle of paragraphs that only have one or two words.

Net result is it's very hard to read.

I tried Safari's reader mode, but that was even worse. Because all formatting comes from happenstance in the <pre> tag reader mode loses all formatting. The entire article (and code!) becomes one long block of text.

I'm continually amazed at people's attempts to reinvent text layout in the browser.

> I'm continually amazed at people's attempts to reinvent text layout in the browser.

Agreed.

I don't recommend you try to read pg's essays, they will make you very angry.

Why is this downvoted? It's true that this blog has bad formatting considering how simple it is, there's no reason to use <pre> for the entire text.

Just because the author created Redis doesn't mean the site isn't badly designed.

I make these kind of comments every once in a while. They're usually downvoted, I assume, for being off-topic or unnecessarily harsh ('We might have cured cancer!' vs 'I can't read the article because of the video behind the text.').

But I hope some people see them and work on the issue.

Sometimes, if enough other people agree, these kind of comments get voted up. I guess this issue didn't annoy most people enough to get the up votes.

Reply on news.ycombinator.com