Build a real-time Twitter clone with LiveView and Phoenix 1.5

Yup, launching soon. Probably will open source it too.

So as someone who worked in Erlang (and is familiar with Elixir), I can answer that a little bit.

They hit a sweet spot. Enough rigor, enough limitation, at the language level, to force you to do things "the right way" for a broad swath of problems (there are certain problems they're poorly suited for), but enough simplicity and freedom to keep a minimal learning curve, and minimal runtime complexity to figure out issues.

The focus on fault tolerance also means once you learn to use the app structure, with supervisors, you mostly have just one concern for reliability ("how do I start this into a known, good, consistent state?"), rather than hundreds ("what happens if -this- fails? Or that? Or that other thing?").

Haskell and Scala both have a huge surface area to learn before you're productive, let alone anything approaching fault tolerant, and even after years of use, there's still a lot of hidden gotchas with them. I've seen teams take Erlang, and their very first project just...worked. None of their lessons learned caused production issues, and no weirdness; the three issues that made it to prod I can even think of were, variously, one that wasn't user facing (just a log entry indicating something wasn't handled right; supervisor took care of it), one where performance started to slow (and it was due to having written an O(N^2) algorithm accidentally, and not testing at higher loads than production), and one where a low level C driver made an unnecessary reverse DNS lookup that, when the caches flushed, caused things to hang, which became an issue when load increased, and which should have been circuit broken (but which instead caused a failover to another node).

I've never had that experience with another language. The ramp up time to productivity was longer, the production issues caused by us missing something about the language were more frequent, the production issues caused by us failing to expect or handle a failure condition were more frequent, etc.

That said, Elixir and Phoenix do raise the bar a little. Elixir has a larger surface than Erlang (only a few concepts, but a lot of ramifications when it comes to macros if you haven't used them in other languages), and Phoenix is reliant on a fair bit of magic so you need to read the docs and it may take a little bit of work to feel happy with things. But even when I knew nothing about it, picking it was easy because of the experience I had with Erlang.

Ok, I've heard of Haskell and I've been to many Haskell meetups too. This was before I found Elixir. For me, the biggest problem with Haskell is that it will turn cryptic if you don't touch the codebase for say, 6 months. Because, you need to essentially re-learn the special syntax of the language if you're touching Haskell code after 6 months.

In contrast, Elixir is super simple. I can describe the language in a single HN comment. Everything is just a function and they're inside modules. That's pretty much it. When I was picking up Elixir, I started off writing an E-Commerce platform from scratch with Elixir. When I was about to finish it, I was thrown into a 8 month long PHP project. When I was finally done with it and returned back to work on my E-Commerce project, I was able to just open up the code in a text editor and immediately understand what was happening - without having to re-learn the language. That is a the "Joy" part for me.

One more "Joy" of writing in Elixir is you don't need to worry about memory usage or performance that much. It's pretty efficient unlike Ruby or PHP even. So, I can just throw this entire application in a $5 Digital Ocean droplet and watch it handle insane amounts of traffic. And this is without optimizing anything. You will probably able to squeeze more with caching, optimizations, etc.

As for re-thinking the way you write code, it's mostly to do with pipe operations. Piping was a new concept for me jumping from Ruby. It's absolutely powerful, concise and productive. In Elixir, if you don't write good code, the compiler will warn you AND show you examples of how (and sometimes why) it should be re-written. Eg. If you write a nested case statement which many people subconsciously may do coming from OOP backgrounds, the compiler will ask you to re-think your code. And stuff like this really challenges you, but actually doesn't do anything destructive to your code - your code will still compile and function, but you will feel that tingling inside - "Hmm, maybe I should consider re-writing that?"

THAT is the balance. Before Elixir, I also used Scala. Took me 3 months to fully learn the language from the 700+ pages book on Scala. I absolutely love Scala and the JVM. It's powerful, but there's just too many ways to do the same thing in Scala to keep track of. So, it goes back to my first point on looking at the language after 6 months without the need to re-learn it. I would still try Scala at some point because JVM is very powerful. But, will it replace Elixir? Absolutely not.

The Elixir video courses at https://pragmaticstudio.com/ are the best available, in my opinion. There are some free tutorials available and some other good paid courses, to be sure, but I think Mike and Nicole over at Pragmatic Studio offer the most complete and easy-to-follow courses available. The cost-to-value ratio of their courses is really good, I think.

I'm not affiliated with them in any way. I'm just a happy customer.

> any good tutorials?

https://elixir-lang.org/learning.html - Start here. It starts off kind of abstract, and then works its way into the concrete (from my experience) -- but is very easy to pick up.

> What major sites?

Last I heard, Discord was serving ~5 million concurrent users, and that was well before compelled isolation.

Bleacher Report is also using Elixir, and serving something like 100k requests per minute just to mobile clients.

Bet365 handles as much as 100k monetary transactions during a big sporting event like a Champions League final.

Discord and Pinterest use Elixir at a massive scale. Discord in particular has been writing a great technical blog about the process.

On a smaller scale Feedback Panda very successfully bootstrapped to a 55k MRR SaaS and was aquired in under 2 years: https://youtu.be/vaXp81OxxK0

Edit: Not to be too self-promotional, but AFIK I've created more free Elixir-learning screencasts than anyone on YouTube. I cover a pretty broad range of topics from total beginner to some fairly niche things: https://youtu.be/z1nKbzZiRtY

> What major sites are currently using Elixir?

Besides what was listed already, if you're looking for info on people running Elixir / Phoenix in production there's this podcast: https://runninginproduction.com/tags/phoenix

One guy was pumping over 7 billion log events a month through it.

Another one (remote.com episode) serves 100k+ daily requests.

Each podcast episode goes through how the app was built and how it's deployed at a high level, but still has enough details to apply those things back to your own projects.

With that said, if anyone is running Elixir or Phoenix in production and sees this and wants to be on the show, just head over to the above link and click "become a guest" on the top right. I'd love to have you on, both big and small sites are welcome.

I heard Divvy, Brex, Slab, and PagerDuty use elixir, so if you're a bay area startup, there's a good chance you're already using elixir as part of your business stack.

If you are interested in a React like component library built on top of LiveView you should check out Surface http://surface-demo.msaraiva.io/

Hey, just a friendly Elixir user here! Could you explain why you think that maps vs. structs is a pain point? I found this interesting because I personally really like how they are so similar, because you can almost always manipulate a struct like a map, just like you'd expect. It lets you use standard operations for working with structs.

Static typing is far from a panacea. Don't forget that the rise of Perl and PHP, and then later Ruby and Python was in response to statically typed languages like Java.

There are trade-offs.

> Are many of the other languages you've used strongly typed? That's the thing that gives me pause - I feel like I rely on the compiler a lot in languages where it can do a lot for me, and that I'd be really frustrated managing a large project without it.

As someone who has been writing Elixir professionally for 4-ish years, and who is otherwise about as big of a proponent of static types as is possible, this is simultaneously my biggest critique of Elixir, and also a critique that has ended up being more theoretical than not. I wish that it were statically typed, but in practice that hasn't been slowing me down.

Part of the reason for that is that while Elixir is dynamically typed, I've found that it's possible to pretend it is a statically typed language if you squint just right. By that, I mean making judicious use of typespecs [0], dialyzer [1], Norm [2], and just generally constraining the way you write code to mirror the sorts of code that you'd write in a language that offers algebraic data-types, like Haskell. I've been meaning to put together a blog post on what I mean by this, because I often hear people talking about how unusable Dialyzer is, so I feel like my team is a rare example of a large production app that requires that Dialyzer typechecks on every build. This makes large-scale refactoring of our app almost (but not quite) as easy as it would be in Haskell.

The macro system in Elixir also means that it's possible to write libraries like typeclass [3], that offer compile-time guarantees (using compile-time property tests!) that your implementations correctly implement a given interface. I think that as the language evolves, we'll be seeing a lot more examples of macros that move runtime logic into the compilation step, to offer compile-time guarantees and safety. For example, a few weeks ago, I prototyped an experimental library in a few hours that added support for compile-time OCaml style parameterized-modules. [4]

Obviously, all of these techniques don't get you quite as far as proper static typing would (though the Whatsapp team is working on static types for Erlang! [5]), but the rest of the language and the ecosystem is just so well thought out, that I've been okay with that!

[0] https://hexdocs.pm/elixir/typespecs.html

[1] https://github.com/jeremyjh/dialyxir

[2] https://github.com/keathley/norm/

[3] https://github.com/witchcrafters/type_class

[4] https://github.com/QuinnWilton/daat/

[5] https://www.facebook.com/careers/jobs/229713254864749/

Before Elixir, I considered Scala because static typing was a strict requirement for me. After I tried Elixir, I found out that good, reliable and maintainable code doesn't have to be necessarily static typed.

I've been programming Elixir professionally for three years. I wish Elixir was statically typed, but there isn't a statically typed language that I really like better all around. I've done a lot of Haskell development but it seems like I always spent too much time playing type tetris and polishing abstractions instead of getting work done. Elixir feels more static than most dynamic languages; you'll get compile errors or warnings if function arity doesn't match or if you use names that are undefined. There is a static analyzer called Dialyzer that can detect a lot of errors you'd catch with static language compiler such as passing the wrong type to a function (often, even if there are no type specs defined). Yet at the same time you get the benefits of a dynamic language with excellent meta-programming capabilities.

Over time I find my Elixir is more and more "statically typed". Since you get access to basically the full language at compile time, with a bit of hand-coding you can opt-in to features that statically type critical sections of your code at compile time. Critically important things can be guarded by relatively advanced checking, if you want to get really clever you can yank those checks for performance in prod deployments. I have been doing this with configurations, so that configs are checked at compile time instead of at deploy time, and I'm pretty happy with it.

I was excited to the other day to see https://gleam.run/ on HN, which is statically typed and will interop with elixir... but still early days for the language.

I loved his complaints about genservers (and his complaints about nomenclature), and it made me settle on my own genserver structure. I think the hard thing about dave thomas' component is that it just doesn't have enough buyin in the community to make it "its own thing". I wish it did though. But on the other hand, if elixir had diverged too much from erlang it would have made people angry...

Unlike a Promise, I hope you can cancel processes in Elixir

Why is that? The GenServer API very similar to Erlang version.

You can add your own abstractions with macros on top of that if you like.

I agree. I’ve tried learning a ton of times but I always hit something I don’t understand and struggle to find help except asking. I think because it’s such a niche language still. With Rails there aren’t a lot of questions left to ask. They’ve already been asked and answered. That’s not the case with Elixir and especially Phoenix.

It sounds like you're mostly backend and struggle with the blending of backend/frontend with liveview?

LiveView is much much easier to understand if you have a solid understanding of how Elixir works in a non-webdev setting, because Phoenix hides a lot of the process architecture that it's leveraging to serve pages, and it's front-and center in LiveView.

And also, it's easy to find help for Flask and Rails by googling because... It's been around longer. LiveView has been GA for less than a year, and even some major shifts have come around (like mount/2 suddenly becoming mount/3 in one of the last releases).

It seems since Elixir is purely functional you don't mutate things. So basically you use tail call optimised recursion for everything. Neat.

Recursion lol.

Erlang does it too.

If you want to have state you do recursion and call itself with the updated value.

fn me(iter) -> me(iter+1) end

It's not OO per say, but it is an imperative concept, OO is almost always imperative.

What's the problem there? A map has unique keys, a keyword list is just a list of {atom, term}, so there might be duplication.

They are similar but they are not the same thing and are not to be used in the same places.

As with anything benchmarking is hard and "performance" is not a 1 dimensional thing.

That said, there are some ways in which Phoenix/LiveView is extremely "scalable".

For example, pushing messages to 2 million simultaneous websocket connections: https://www.phoenixframework.org/blog/the-road-to-2-million-...

... and running on the BEAM gives all sorts of exciting horizontal scaling possibilities on top of that.

I think calculations are it's weak point, but for the typical message passing application like Twitter it's fast and incredibly scalable.

No - video stream is webrtc, pdf viewer is pdf.js

We use liveview + hooks(where necessary) for all of the interactive parts on the site, switching pages, changing books etc...

Yes, there are two Blazor variants. Serverside Blazor is basically the same thing as Phoenix Live View, client side Blazor is providing some optimized Mono runtime as webassembly.

you are correct, you can't escape the laws of physics latency wise, but most web things we are all building, including 100% JavaScript apps require a round trip to the server. For example I can't post tweet without a round trip to twitter. On the LiveView side UX wise, we apply css loading states to the interacted elements, so the user is provided feedback while awaiting an acknowledgement. On the server, our templates are precompiled and do change tracking. So we only execute the elixir code that is necessary to render the parts of the template that changed. The rest of the calls are noops. From the top down, it appears like an extremely heavy approach, but with our change tracking and diffing, it's actually incredibly optimized. Also note the Erlang VM was made to handle multiple users, as we showed in our "The road to 2M connections with Phoenix" blog post a couple years ago. More connected users will mean more memory, but Memory is cheap, and the platform scales well enough that opening a websocket on all pages as a matter of course is completely viable. Hope that helps!

LiveView does not re-render the template on every interaction. LiveView actually sends patches over the wire, which is typically smaller than a hand-written JSON response. The screencast linked above has a good example of this, where clicking the "retweet" button sends a minimal payload, since we know the exact position on the page.

LiveView also uses a long-running WebSocket connection and that reduces the amount of data sent over the wire compared to regular requests/responses, as you don't need to encode headers, cookies, etc.

Finally, if you are worried about latency, you can call "liveView.enableLatencySim(200)" in your browser console, and you will be able to emulate how your application behaves over high latencies.

> People who don't enjoy videos for various reasons would probably like a text version too.

Yes this. I went to the video eagerly looking for a repo link and was disappointed not to find one.

Hi miki123211, I wrote up a transcription for you. It details the code he's talking about, along with the context of what's going on, and why.

https://pastebin.com/raw/WxH0uAsf

If Pastebin doesn't work for you, let me know and I can get it hosted at a different location.

My apologies if there are any mistakes in this. I'm not familiar with Phoenix, but found this video very interesting, so wanted to make sure it could be shared.

"offline support" is strictly on the side of needing to run a lot of code on the client, so this is where we draw a line and say you need a js framework, of which there are number of great options. We have javascript interop via phx-hook, so it could be made to work, but you'd necessarily have to have a lot of js running on the client, and any feature you wanted to work online & offline would not share a code path or need js, thus client side rendering and templating gets involved and suddenly we're writing a SPA. If we're strictly talking allowing the edit of a textual input and syncing on reconnect, it would be straight forward to do with LV and some small js hooks, but the moment you start adding offline UI features you'll need a js library.

The funny thing is last I tried to use Google Docs offline, it locked the page in read-only mode, so offline mode even in the SPA space isn't just a given – you're definitely opting in to some necessary complexity.

Confirm. LiveView will apply css loading state classes, which are user defined, but we generated css by default in new apps to lock the UI with loading cursor, and show a top loading bar. On reconnect, which happens automatically, we rematch the DOM with any changes, and we also send up any user form input beforehand so client state is lost that was in progress. We already have reports of users pushing bugs to their systems and the UI seemingly being unaffected because the LiveViews automatically recover, wether from a disconnect or a crash on the server.

As always, I can make no guarantees before 1.0, but we have really focused on refining the programming model and APIs for the last several months, so we're in a good state currently.

Components IDs are unique. So once you try to render the same component, we know you simply want to update the current one.

The same happens on the client. If the server emits the same DOM ID, the client just updates it in place too.

Oh, yeah, here we go:

1. Tooling, yeah, you gotta have node installed, but that's pretty much the last time you have to touch it. I'm not anti-node or anti-js by any means, but the tooling around them often turns me off. Getting to do front-end work without _having_ to think about 'em is a breath of fresh air.

2. Less duplication of effort. I don't need to define schemas twice... once in the "backend" and once in the "frontend" application. You don't think this is bad until you spend two hours adding a adding, removing, or changing a simple property to your application because you have to update the backend, the backend's api docs, its tests. Then on the frontend, adjust the schema, adjust the frontend tests, and finally see if it's working.

3. Faster iteration. I prepare a new context, and can immediately consume it in a view. Views are reloaded instantly when changes are made during development reducing the time it takes to get feedback and increasing how fast I get shit done. Also less duplication of effort REALLY helps here...

4. We (at least me) got S(F)PA idea totally fucking backwards. The idea isn't to make you're entire web site a Single Fucking Page Application, but to make individual (single) pages which NEED TO function like applications do so... Quintessential examples that do _NOT_ need an SPA: a fucking FAQ, about, or contact pages. In fact, having an "application" on top of them is more burden than not.

5. CSS is really powerful these days and provides a lot of what the 2008-2012 JS movement was trying to do. If I combine that, with being able to re-render only part of the page, I've got almost the entire flexibility of what SPAs offer with almost none of the complexity.

6. Going to a new page, and causing a page refresh, but totally knowing the next page isn't going to be broken because of the previous page's state. Page refreshes are awesome. Growing up on 28.8k that wasn't the case, but ever since I got above 10Mbps its really not something I ever even notice (though I do notice when pages are broken that shouldn't be because they're SPAs and not just fucking content).

I could probably keep rambling but I'll stop here...

Seriously. I would love to work at this level of abstraction in Clojure, but to do something that is trivial in Rails with something like Devise is a chore in Clojure, and you have to put it all together yourself. Of course this makes it excellent for learning, but not so great for deadlines. More and more I think I might've made a mistake trying to do basic CRUD web stuff in Clojure for my current project.

That's the same risk you have with any open source project. Elixir and Phoenix have similar risks. I think there's a lot to be said for getting default/easy functionality for free so you can focus on building your unique amazing things vs working on plumbing.

I'm not the person you asked this, but the BEAM (and phoenix by extension) excels in highly parallel contexts.

basically if you got an application which wants to connect everyone to each other in real time, thats a perfect usecase for the BEAM. so chat rooms, live-feeds, comments, video/audio conferencing, etc.

and thats also exactly where phoenix excels in my opinion as a occasional user. (never professionally though) quick and seamless communication between silly amounts of people with almost negligible amount of resources while being error tolerant to an extreme (functional, so almost no state)

just try it out on a weekend. its definitely worth it just to see how web-development couldve been. you'll probably go back to the old way, because thats just how you earn your money and where you already have your expertise... but its definitely amazing to see how easy it can be

The common retort from SPA devs is "what about offline?" Since LiveView keeps state on the server, it doesn't really work offline.

However, 1) you can add custom JS using LiveView's hooks, which might be enough for very simple offline behavior and 2) many SPAs don't work offline either.

If offline support is a major part of your app's design, LiveView isn't a good fit. But you could still use Phoenix Channels (the building block underneath LiveView) to provide fast push updates to your client. See the channels docs for an idea of how they work - https://hexdocs.pm/phoenix/channels.html

I'd say I sometimes miss Rails' ecosystem, but the many advantages of Phoenix and especially LiveView are worth that small downside.

To someone who uses Rails/Django with React, I'd describe LiveView as something that offers pretty much everything you're used to, but with the React part consolidated with the server-side of things.

I've been playing around with LiveView since it came out, and have been using it for 'serious' work for the past year or so, and I still regularly discover ways in which this setup simplifies things, makes development faster and more fun, and saves me from various potential security flaws or 'busiwork'.

Imagine that your store (Redux, whatever) and view (React) ran in the same place as your backend.

Direct access to the database and the rest of the backend, no need to carefully consider whether it's really worth it to increase your js payload with library x, no constant context switching, no building and maintaining of, essentially, two completely separate applications, no need to make sure your API endpoints are properly versioned, or that they don't expose data that shouldn't be exposed, or that they send only the data you need. Little to no need for rube-goldberg js pipelines, and significant less time spent dealing with synchronizing state between server and client.

I'm probably forgetting some other advantages.

There are some things where LiveView doesn't seem ideal, but I've found that even in those cases I prefer using LiveView wherever I can anyways, and then I do the specific javascripty bits wherever I need them.

Go may be popular, but is also a lot less complete.

Some other things to consider are that Go's concurrency model is quite bloated when compared to Elixir's, Go uses public memory for its goroutines vs. Elixir's private memory that protects processes, Go uses cooperative multitasking vs. Elixir's preemptive model, and Go's dependency handling is absolute rubbish. Elixir's process supervision is amazing and Elixir's pattern-matching allows you to write easier-to-read and more maintainable code, in my opinion.

While both are good languages, I've personally found Elixir > Go.

I’ve invested a lot of time into both Go and Elixir. I like both languages a lot.

My anecdotal view of the marketplace is that Elixir recruitment emails tend to come from startups looking for their 2nd or 3rd dev, usually full stack, and the salaries are nothing special. Go recruitment emails tend to come from bigger companies looking for backend devs, often hoping for kubernetes experience too, and the pay is higher.

I built an internal service with LiveView at work and I loved it. I gave a presentation about how it all works and was hoping it would catch on. It didn’t. The front end team would probably mutiny if we told them to give up React and level up their LiveView skills.

If I were running a web business as a solo founder I would 100% want to use LiveView. As a dev on a larger team that already has specialization it’s a tougher sell.

I agree. Luckily, sometimes the fix is easy. For example, I packaged my invoice PDF library[0] as a Docker container and simply created a small Elixir module that calls it.

[0] https://github.com/strzibny/invoice_printer

I'm of a similar view to them, this felt like lots of magic or at least lots of gotchas I can see myself hitting. I saw two function definition types (def and defp), pattern matching input then requiring a caret for the variable name (?), passing in "assigns" but then referring directly to other variable names, pipes, etc. There's a lot there compared to when I've worked in Erlang.

> The only modern web problem that Elixir isn't ideally suited for is heavy number crunching.

I would caveat that in a couple of ways.

First, suppose you have a web app where some requests involve heavy number crunching and others don't. In web frameworks where 1 request ties up 1 OS thread, a burst of heavy requests could gobble up all your available connections. Phoenix would use one cheap BEAM process per request, and the BEAM's preemptive scheduler would ensure that other requests are answered in a timely way and that all the heavy ones continue to make steady progress. So although the heavy requests might be completed more slowly than in another language, the overall system would remain more responsive.

Second, if you have need for heavy computation or data structures that work better with mutability, it's possible to (eg) use Rustler (https://github.com/rusterlium/rustler) to implement that part in Rust. See https://github.com/rusterlium/rustler for a story about doing this.

correct. I was just wondering if it also has inbuilt http server as well to process things like static files etc or is that completely on 3rd party tools like nginx (which I love btw). Kinda like in Golang where you could technically use the static file server but I always use nginx for those things.

Coming from Python as well I'm really impressed with the tooling. `mix new` is just amazing.

But without requiring all of the ActiveSupport patches to the base language!

Yeah, really looking forward to see this project keeps growing! I really hope we can use it in projects soon.

Implementing runtime for Elixir in JavaScript is quite challenging IMO. Kudos for Dockyard trying to tackle this.

Excuse my lack of knowledge, but I was always under the impression that websockets are not that scalable since you are opening 1:1 connections.

I'm assuming the parent meant how would this specific configuration of elixir + phoenix + liveview scale.