Comment by sergiotapia | Hacker News Reader

Comment by sergiotapia | Hacker News Reader

Hope you guys like this and use it! If it's hard to use, tell me. My goal is to make this simple to run anywhere. Part of that will be to use Distillery to create compiled executables.

I rewrote this after an initial version that used Redis as a queue to process. Asking end users to install Redis was a step too far I think, so I stepped back and thought about how to solve this without external dependencies.

I ended up with a simple Elixir and Erlang queue implementation. It works much simpler, just as fast, and no freaky deps. GenServer starts up a worker for each crawler, and it schedules work by itself. It's really strange to use a language so complete, it kind of feels like cheating. Programming in easy-mode.

Pull requests very welcome to the crawler folder, it's super easy to write a crawler for your favorite site!

https://github.com/sergiotapia/magnetissimo/tree/master/lib/...

you could make it a nerves project too, maybe after you get rid of some of those database dependencies.

It'll make you a ~100Mb linux distro that will boot straight into your app in ~10 seconds on a raspberri pi.

http://nerves-project.org/

very useful project, thanks for sharing this!

If the goal is to make it simple to run, why not support SQLite?

Seems like the goal is avoiding dependencies and he's got 3 built in database choices for that already. DETS or Mnesia would probably work well for this.

Thank you, I'll take a look at what those are. If there's a way to remove Postgres from the equation, I'm all for it.

DETS (and ETS, the in-memory equivalent) are quite simple for basic K/V-ish storage needs. If you need a relational database, Mnesia should do the trick.

Mnesia is probably what you want in that case, since it turns (D)ETS into a full-fledged database rather than a simple key-value store (think SQL v. Redis). Both are part of Erlang/OTP, though, so you get them for free just by using Erlang/Elixir/LFE/whatever.

There's a hex.pm package called "Amnesia" that wraps Mnesia in a more Elixiry format.

That sounds fantastic. I'm all for trimming the fat if we don't need it. I'll research this, thanks for the suggestion.

If you're using Phoenix or Ecto, look into ecto_mnesia (https://github.com/Nebo15/ecto_mnesia). It wraps up mnesia into a nice ecto wrapper. It's limited on what your primary keys can be.

Reply on news.ycombinator.com