Building a Reddit-like site after 3-4 months learning Elixir | Hacker News Reader

Building a Reddit-like site after 3-4 months learning Elixir

reddit.com

175 pointsstanislavb70 comments

My company replaced the integration server I had originally written in Python/Django/Celery/RabbitMQ with one built in Elixir/OTP. The results were impressive.

Our task load was getting out of hand, and during peak load throwing memory and CPU at it was not helping. Task monitoring was also a pain. It needed a complete rewrite, because the original design, while robust for what it targeted at the time, could not keep up with our growth.

The integration developer I hired to take over the project looked into using Go at first, but then discovered Elixir. I started learning it with him (he quickly surpassed me), and after a few trial task implementations, I gave him the go-ahead. We couldn't be happier with the result. Thanks to OTP and the existing tools built around it, we have great monitoring, and the ability to easily and quickly scale up workers at peak times.

Are you sure this isn't more of a case of "a motivated developer writes great software" than "a silver bullet language"?

Did he say "silver bullet language"? It seemed like a good fit for his use case; I don't see him claiming anything more than that.

Correct. Not saying anything more than that. The BeamVM/OTP is a better fit for long-running high-concurrency fault-tollerant systems. That's what it was designed for.

I have not switched to Elixir/Phoenix for my other web development. I still use Django for that.

Anything specific you liked about Elixir more than Go?

As someone who writes both, it's even difficult to compare these as they are apples and oranges.

Elixir (due to Erlang base) is great for long-running or distributed services and includes tooling to, for example, easily introspect the state/memory usage of your services down to a single mutable data structure, to heal from failures, and to hot-swap upgrade/code while the server is still running, and to perform node-to-node RPC.

Go would be a replacement for something that was previously written in C/C++, that may need concurrency, and that should compile to native code. You get very fine-grained control of everything from network to file operations and excellent concurrency constructs. A web crawler, a device driver, or writing an Nginx/HAProxy-type daemon, all may be great example use cases.

One could write their next social network/app backend using Go (just as one could do it in C), but it would probably take more work than just using a higher-level language.

Go is much more suited for web apps than any driver or real time performance application; although, I will agree that using Go for command-line tools is great. I haven't used Elixir but as somebody who uses Go, I highly recommend not replacing C/C++ with it unless it isn't performance critical.

PS: Go's memory usage is pretty up there some times, another reason you should probably stick with C/C++ in the context mentioned above.

I'm the wrong person to answer that, perhaps, but with OTP you get a task broker / monitor for free. With Go we would be back to piecing together a system using other tools such as AMPQ or similar. It also isn't as easy to write it for high-concurrency monitored tasks. But I have little experience with Go, so correct me if I am wrong.

It sounds like you hired a great engineer to be honest. Elixir, and the OTP environment is well designed. But it shouldn't downplay the achievement of the guy who re-implemented it (or yours, for letting him do so and provide him with the adequate mentorship). Props to you both, it's an impressive feat.

Despite being a $120M company, we still have and foster an entrepreneurial spirit. I love working here.

Are you hiring?

I had a simple rabbitmq -> elasticsearch logger I wrote in scala, which had to handle peaks of about 8000 messages/s on a single t2.large instance (with elasticsearch on the same box).

It took about a week and a half to write, and would have taken a bunch more time to fix to apply backpressure properly, so it suffered from getting OOMed. Even it's base memory usage was around 300MB. I rewrote it in elixir + genstage in about 8 hours, now it takes 60MB of ram for basically as many messages as I can throw at it, because 1) beam uses a lot less base memory 2) backpressure concerns are sorted out.

In scala, I would have to use akka streams or similar, and it would take me weeks and still I don't think I could get it to fit in such a small footprint.

I plan to add websockets to view a live stream, but I have a little trouble finding the right genstage patterns (as it's still new).

If I were doing high frequency numerical calculations, maybe scala would be a better fit, but for most high speed messaging elixir is pretty amazing.

Happy to see more and more non-trivial platform-based implementations in Elixir. We're using the Elixir/OTP+Phoenix stack for our own projects, handling networks of radio-connected sensors. It has been a breeze developing with Elixir, as not only the language is maturing, but the ecosystem is steadily growing as well - and there are a number of great libraries, from Ecto to Phoenix itself.

The OTP foundation of Erlang and hence Elixir just screams "platform" or at least "network", as it was designed to handle parallel flows of data. Great to see yet another project utilizing this aspect of the language.

May I hear more about the nature of your sensor project?

Not sure what you mean by "parallel flows of data", but Erlang/Elixir isn't particularly good at handling multiple streams of data between distributed nodes if you're relying on the built-in distribution subsystem (net_kernel).

Only a single TCP connection is opened between any two nodes and there's a head-of-line blocking issue with message delivery (intentionally for backward-compatibility reasons).

Certainly you can build your own multi-connection subsystem if you want to, but that's true of almost anything. I've had to do this for Erlang several times.

Instead of rolling your own connection subsystem, there is also this https://github.com/lasp-lang/partisan Although they call it still experimental, it seems to be pretty actively developed.

I don't appreciate the "Elixir is growing" commentary in the title. That's not justified because a person wrote a program in it.

Just grew by 1

And somewhere else someone stopped using it. One person doesn't make a trend as the title claims.

Arguably this post will lead to net-positive growth. I'm certainly going to try it after having read the top comments.

I stopped using it just to throw a wrench in your gears.

Grind this thing to a halt!

(...no, actually Elixir is pretty cool.)

Yeah, the post has nothing to do with Elixir's growth. Simply "Building a Reddit clone in Elixir" is a much better title.

Agreed that that is not a logical conclusion given that evidence.

It is, however, likely true based on the evidence that each ElixirConf so far has doubled the number of attendees of the previous year.

Search trends for the language seem to be growing, although not seemingly strongly: https://trends.google.com/trends/explore?q=%2Fm%2F0pl075p

I write a lot of server processes that communicate with remote devices.

Elixir has become my go-to language for whipping up new communication utilities. The syntax is comfortable, although I had to get used to a Functional Programming style at first.

What I like most is the language and runtime's features that make parallelization so easy. Spawning threads that are monitored, have their own protected state, that are managed if they crash, and are trivially spread across different machines on the network made it straightforward to write fairly highly available code right from the beginning without a lot of planning.

Thanks to Erlang

Thanks to BEAM, no?

Which is "Bogdan/Björn's Erlang Abstract Machine". So erlang.

Both, a lot of the cool features come from Erlang's OTP library.

Can you go into more detail about the type of highly available code that you write?

For the lazy who are unsure what Elixir is:

http://elixir-lang.org/

(I assumed it was a front-end Javascript framework or something like that)

From what I gathered, it's popular for Rails developers who want something that feels like Rails, but with (much) better performances and concurrency.

For me coming from Scala, I feel like the lack of typesystem would be a step back.

It has a type system, just a differently structured one to support its goals. In a naturally distributed language there are a number of headaches that are created by overly strict typing (think REST vs WSDL as an example). It's an entirely different way of thinking about the problem that enables you to accomplish things that you can't effectively do outside of the BEAM.

https://blog.codeship.com/understanding-elixir-types/

[Plain Elixir has one data type.](http://www.theorangeduck.com/page/defence-unitype)

There are (optional) type annotations. https://hexdocs.pm/elixir/typespecs.html

Typespecs are mostly for static analysis. They describe the types of inputs and expected outputs.

Phoenix is the framework that resembles Rails and is built on Elixir the language, which runs on the Erlang VM.

Phoenix (elixir-based web framework) is popular for RoR developers, yes. Elixir, has much broader application field.

I came to Elixir from Scala, and for me the dynamic typing is a big step forward, because now I don't have to deal with the half of the Scala community that tries to make Scala into Haskell.

Between Dialyzer (a static analyzer that finds type errors) and runtime pattern-matching (extremely powerful destructuring, sort of borrowed from Prolog initially), I don't find myself wanting to go back to strong typing on the JVM.

> because now I don't have to deal with the half of the Scala community that tries to make Scala into Haskell

side-effecting shots fired!

Oh, screw side-effects -- Elixir is much more hardline about immutability than Scala. I'm talking about jamming half your program logic into the type system, then braying to anyone who stays still long enough about how that's the right way to do things.

Holy hell, you just described my daily problem more than I previously knew.

After dealing with Elixir's no-mutation-possible-anywhere design, I don't want to go back to any language that permits it

I'm currently working with Scala and recently dabbling with Elixir and Phoenix. It's a good language and great platform, however, I still prefer static typing.

I'm still getting used to dynamic typing in Elixir. Most of the time I feel like just matching against Map data structure or records.

While Dialyzer is great, writing the typespecs is a bit of a maintenance overhead. I suppose it's a tradeoff...

> I came to Elixir from Scala, [..] now I don't have to deal with the half of the Scala community that tries to make Scala into Haskell.

IMO Scala will continue to be a multi-paradigm lang. On pure-FP and libs, it's a choice and really depends on the problem domain. It might not be a good solution to everyone, but I'm glad it exists.

I think of Elixir typespecs as documentation more than as overhead. Given that I'm writing comments that tell the shape of the inputs and outputs of every function anyhow, I might as well do it in a way that lends itself to static analysis. :)

Google trends shows that Elixir is in fact growing: https://trends.google.co.in/trends/explore?q=%2Fm%2F0pl075p,...

Strange - that graph set shows Elixir on a slight growth path, but Erlang on a slight downward path? Isn't Elixir simply the framework around Erlang??

It would be a bit like showing Rails on the uptake, but Ruby going down, isn't it?

No, Elixir is an entirely different language running on the same virtual machine as Erlang.

It's as seeing a graph with Java usage going down, while Kotlin or Scala up.

Thanks @unscaled and @LukaD for clarifying this. I hadn't used the language, so didn't understand that Elixir wasn't just the name of the framework.

Not sure why I got the downvotes for asking a question, but there you go...

Because literally five seconds of googling would have gotten you the answer. Your comment didn't add anything for anyone but yourself. That's why you got downvoted.

I was wondering the same thing, so I found cyberferret's comment (and the answers) useful. It might be helpful to anyone who's skimming through here.

Elixir is not a framework around Erlang. It's a language running on the Erlang VM (BEAM) with painless erlang interoperability. It's the most popular alternative language to erlang on BEAM.

Opened that link and "Hmm, your search doesn't have enough data to show here."

weird?

I have been using Elixir extensively for my services for a year now and it's been a really great experience overall.

p.e. why "Elixir is growing" in the title - because more and more people are building sophisticated and production ready services using Elixir.

Elixir isn't in the Top 50 programming languages yet, but I reckon it will be there in not more than 3 months... and that is respectful given its age.

Exactly what I See in elixir. It Keeps growing each and every day

Says you? It's hard/to early to tell ofcourse, but currently it looks like it's stagnating. https://insights.stackoverflow.com/trends?tags=elixir

A lot of Elixir Q&A actually happens on https://elixirforum.com/ and the slack channel https://elixir-lang.slack.com/ . So stackoverflow trends may not be the best thing to judge this.

As an avid Elixir user, your comment got me thinking about how and where I look for help. I realized that it wouldn't even really occur to me to use Stack Overflow for answers regarding Elixir, as the other platforms you mentioned tend to be so active and well suited for that purpose.

This isn't to say that I wouldn't use SO for other purposes, as I actively visit the site for JS/PHP content. Anecdotal evidence to support your insight by all means, of course.

Absolutely. The elixir community on the forum are amazing.

I haven't used it, but the prospect of having to use a forum rather than Stack Overflow for Elixir help would worry me

Discourse (the platform used by ElixirForum) is built by Jeff Atwood one of the founders of StackOverflow. It is a really nice app to use.

Discourse is garbage but that's an ancillary point I think. Developers are on StackOverflow. If you have a C#, C++, JavaScript, Java, Python, Ruby, or Haskell question and you search for it online chances are it's already been answered on StackOverflow. If it hasn't there is still probably enough information to guide your future search.

"Go to the Slack channel" is not really an acceptable substitute for that level of independence in research.

Agree with this. I've had a couple of questions on Elixir & while I use SO for looking up answers to other languages, for some reason it never even occurred to me to ask Elixir questions there. Possibly because the folks on Elixir Forum & the Slack channel are incredibly welcoming. They also have areas specifically for those questions you know/feel are very beginner noobish levels.

SO shouldn't be used as a gauge of popularity of a language or framework. Well established languages/frameworks will not see a lot of new questions being posted there since they have an established set of answers. Newer frameworks/languages will see a lot more activity there building up a base set of questions/answers.

The other angle that SO can misrepresent is that inexperienced developers are using the site more than experienced devs. If you know your language/framework/libraries pretty well, then you are mostly going other places for help like API docs or maybe directly to the Github repo for a library and asking committers since you are much deeper into the language.

Another reason could be that SO itself is trending downward. I know that there are people, like myself, who don't often use it or who just use it as part of a set of google results for the topic they are researching.

I prefer LFE (Lisp Flavored Erlang) because I prefer Lisp syntax to the Ruby-like Elixir syntax. LFE compiles down to core Erlang, while Elixir compiles down to vanilla Erlang. True macros can be done in LFE, but as I understand it, macros are done after parsing in Elixir. LFE was created by one of Erlang's co-creators.

I'm curious to hear what you've built with LFE, and what the experience was like? What would you say were the biggest stumbling blocks you ran into?

I'm also in your camp (not liking Elixir or Erlang's syntax, but really liking BEAM). I like Clojure a whole lot, but would prefer it on BEAM for a whole variety of reasons. LFE seemed like a step back from Clojure in many ways when I looked at it.

Perhaps the 'step back' is that LFE is a Lisp-2 like Common Lisp whereas Clojure is a Lisp-1 like Scheme.

You might find Joxa interesting, it predates Elixir, but came after LFE and is patterned on Clojure.

http://joxa.org

um.. this is incorrect. Elixir Source Code -> -> Elixir Macro Expansion -> Erlang Abstract Format --> Core Erlang --> BEAM bytecode

Erlang does something very similar.

Erlang Source Code -> Erlang Abstract Format --> Core Erlang --> BEAM bytecode

I'm pretty sure that LFE doesn't skip the Erlang Abstract Format step

It's fine to be a fan of LFE, but not so fine to spread misinformation about Elixir macros.

Macros in Elixir are "true" compile-time macros... They take the AST as an argument, and are expected to return an AST. Possibly the first, or one of the first, implementations of "true" macros in a non-homoiconic language, which was previously a strong selling point of homoiconic (Lisplike, syntax-less) languages.

Reply on news.ycombinator.com