Hacker News Reader

Comment by nattaylor | Hacker News Reader

A friend built a system where the ad campaigns with all their targeting rules were Lua scripts and it was also fast and simple in a glorious way!

View on HN

I tried a similar approach with a team I was working with. We were building it on top of redis and after some basic benchmarks gave up the idea and figured out from the docs that that eval script is blocking.

When I did that on a project I solved that problem by replicating redis to the same container that the ads were being served from. Replication was very fast, and all that was blocked was the local redis. I worked to make the matching rules in the script efficient, and the blockage was truly not a problem.

Doesn’t local redis kind of miss the point of using redis in the first place? On the surface, that’s a big chunk of additional complexity for something that could be done internal to an application. Was this meant to be an incremental step in a larger refactor?

well, redis manages the replication without fuss. Unless the framework / language one uses supports it out of the box, there is no point in re implementing replication.

https://www.erlang.org/docs/17/reference_manual/distributed

Oh so this would effectively serve the same purpose as a centralized cluster.

Yeah with the added advantage of reduced latency. The system I worked on had a CRUD application which was in Django , ultimately the rules was stored in a Redis.. The ad serving application was in golang that connected to this local redis instances that were replicas of the central redis..

https://redis.io/docs/latest/commands/replicaof/

Exactly the same for the one that I did, except that the ad serving was in Python.

But golang is probably the better choice.

Lua in Redis is not useful for serving as a runtime for running Lua application code; it's valuable for allowing you to perform a series of steps atomically. You can read and write data without worrying about something else changing: only one piece of code can write at a time. We use it for rate limiting, which requires reading and writing atomically, for instance.

Why wasn't Python used? I imagine for such a use case it wouldn't be that much heavier to use Python vs Lua.

Speed, probably. Lua is much faster.

Yes, speed. 1,000,000+ QPS and 1,000s of ads to evaluate is common. At that scale it's very distributed, so replication is another challenge.

In a way, its a search problem where bid≈relevance and targeting_match≈recall, so I've seen Solr used here too.

Integration complexity could be a factor. I’m making an assumption about the architecture, but embedding Lua in a program is dead simple and you can do so without introducing external dependencies. Python IIRC requires you to ship and package the standard library or have it already installed, the Lua interpreter can be statically embedded in a program.

Reply on news.ycombinator.com