Hacker News Reader

Comment by abledon | Hacker News Reader

Has anyone used gen stage / flow for their jobs / small - medium projects ? What are some examples where this could be used ? Game server ? I found an example on YouTube about using it in an automation warehouse but that seems too niche to implement

View on HN

We use Elixir at Tinfoil Security for our API Security Scanner, and it uses GenStage quite a bit under the hood. It's a big simplification, but we basically have a producer that emits a stream of "scan tasks" to be performed (things like "scan this endpoint for SQL injection"), and then those are consumed by individual worker processes.

It lets us super easily manage things like rate limiting and throttling, while providing backpressure so that an influx of scans (or scans on large APIs) won't overload our infrastructure.

Flow - used it at work for an odd use case where had to merge data from 3 database sources (given to me in csv format) against other data to migrate it. It involved 100s of millions of rows and lots of unknowns so flow was perfect for processing and adaptability whereas spark or many similar solutions felt a bit heavy for the use case.

GenStage specifically: we needed an internal load tester and GenStage plus easy concurrency allowed to quickly build an effective and reliable tool with rate control

https://www.youtube.com/watch?v=Aa--NDjL9SI

In the talk by Chris Bell, (29:15) they use Flow to migrate millions of records.

Peter Hastie, of sports-news site Bleacher Report, gave a talk at Erlang & Elixir Factory (2017) about GenStage (and Flow) -- there's a description of the talk and an embedded YouTube video at the bottom:

http://www.erlang-factory.com/sfbay2017/peter-hastie.html

Reply on news.ycombinator.com