Elixir web development 101: collaborative todolist with realtime updates
blog.openbloc.fr
blog.openbloc.fr
While the concept is not new to me, I have recently become totally infatuated with the concept of green threads, actor models, etc... for dealing with concurrent processing. This is literally what Erlang was designed to do, and Elixir makes it user friendly.
The real beauty of Elixir/Erlang is being able to run hundreds of thousands of concurrent "threads" (co-operative in userspace, not OS-level).
Anyway I do not mean to detract from the post here -- just feel like a todo list is not a shining-star example of Elixir and would love to see some more hackers post their concurrency work here.
Plus, when you really start thinking about how the entire platform is designed specifically for passing small messages around between process, websockets become an even better example since that’s exactly what they do with the browser. You enable passing small messages between millions of connected clients to a server side environment built for passing millions of small messages between clustered and horizontally scalable servers.
I would love to see more examples that get away from web dev with Elixir. While it's great with the Phoenix framework there is so much more to Elixir than that. Trying to deploy an elixir daemon has been a pain and I haven't found much around that. There is escript and distillery and while I haven't tried distillery for non-phoenix apps, escript works but deploying is still a pain.
We spent some time with friend doing research and trying to connect all pieces to make it work.
Since the project I'm working on is open source, you can see how I've done it. It's still far from perfect but maybe it could be a place to start.
Elixir app is on https://github.com/hackerexperience/helix. You'd probably be interested on `lib/release.ex` and `rel/`.
For the ci, our infrastructure playbooks are at https://github.com/hackerexperience/infrastructure, and you probably want to look at `roles/{helix, erlang, elixir}`.
Hit me up if you have questions our would like to discuss further. Email is on profile.
Btw, here's a great presentation about the topic: https://youtube.com/watch?v=H686MDn4Lo8
[1]Because insight at the system level is required to do effective scheduling and because the transition between the userspace's concurrency model and the system's typically introduces unpleasant limitations and sometimes bugs.
If you're saying that a thread/actor/CSP model of concurrency is better than "async" (callbacks) as used with JavaScript, then of course -- 100% agree on that.
One of the biggest strengths of Erlang (and Elixir by extension) is that they take the approach of finding practical solutions to real problems.
Sure, theoretically there might be a better solution than green threads, but like you said, we're still waiting for that better solution 30 years later. In the meantime, Erlang has been powering highly reliable and concurrent systems for ~30 years now.
Which I don't believe. Throwing away thread and managing it yourself and playing the game of will this code block the event loop is ludicrous.
BEAM VM does a very good jobs. It serves financial, large telecom, messaging, web and many other services. It is one of the marvels of software engineering. Not only does it provide this N:M scheduling well, it also provides process heap isolation between concurrency units (processes).
Heap isolation is very important, it increases safety, fault tolerance and provides guarantees that are just not there with shared memory model. Heap isolation is also other CSP/Erlang-like-clones ignore when they say they provide an Erlang-like environment. The only equivalent is really something like forking hundreds of thousands of OS processes [+]
> Once that deficiency is fixed (yes...we're still waiting after 30 years) green threads can go away.
I am not sure when it will be fixed. We'd need to be able to run one million OS processes effectively with the same latency and memory usage as what Erlang's processes currently have. We haven't gotten there in 30 years as you said, I don't think we are getting any closer. Sure there are machines that could probably run a lot more OS processes than before, but then putting an Erlang VM on the same hardware would allow running even more of its lightweight processes and so on.
[+] Another interesting exception could be Rust. If say threads were very cheap to create and could have millions of them, then could provide the same memory safety guarantees at compile time.
Erlang's units of concurrency are processes, not (green) threads. Unlike threads, they are completely isolated from each other.
> (co-operative in userspace, not OS-level).
Erlang's scheduler is preemptive, which means it has the ability to context switch between tasks without cooperation at any time.
> etc... for dealing with concurrent processing. This is literally what Erlang was designed to do, and Elixir makes it user friendly.
What exactly is user unfriendly in Erlang's concurrency model? You only need to understand 3 concepts, spawn, receive and send in order to perform any concurrent task. I'm not familiar with Elixir, what does it do differently?
Check out Jose's talk on it: https://www.youtube.com/watch?v=XPlXNUXmcgE
If you're looking for cool Elixir applications that aren't web-oriented, look into the Nerves project (http://nerves-project.org/).
> channel.push('delete:todo' ...
> Have a CRUD interface over websocket
REST over websockets. What's the advantage over HTTP? I know websockets allow for pushing data to clients, but this is pull so apparently there should be no advantage, only more code to write.
Edit: maybe you're sharing updates among all the users connected to the server. Still, for sending requests HTTP is enough. Any performance advantage by using websockets?
Once the connection is established, you can send requests starting at 1 byte size, whereas HTTP needs the full headers on every request.
HTTPS also has a lot of handshake overhead, where you need 2-3 round trips per request instead of a single one with websocket. This can make a world of difference in a 150+ ping situation, where HTTP feels sluggish and websocket feels instant.
In practice I've always observed HTTPS requests to take at least twice as long compared to a back and forth through a websocket, even when SSR and keepalive are enabled. I don't know if having both theorically allows for single-roundtrip HTTPS requests.
The tradeoff is dealing with half-open TCP connections (need an in-process ping to ensure the connection is still alive) and designing the app for state recovery in the case of missed messages.
This is not something obvious because we have helpers to make http requests, but there are many things going on while issuing a http request (building headers, encoding body, etc) [1]. A socket (be it a websocket or a classic socket) has this advantage that all the negotiating part is already done and you just have to send the message, in whatever format you want. Decoding the request on server side is way easier too.
[1] https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol#Me...
Excellent. How about raw TCP then?
The cool thing about Phoenix specifically is that they provide a JS client to integrate with a Phoenix Channels [1] backend that will automatically fail-over to long-polling if window.WebSocket is not available [2]. These Phoenix Channels are set up to be transport agnostic, so you don't have to write any special backend or client code to handle one way or the other, it "just works" for the most part.
[0] https://caniuse.com/#search=Websockets
[1] https://hexdocs.pm/phoenix/channels.html
[2] https://github.com/phoenixframework/phoenix/blob/master/asse...
But Phoenix contains these workarounds and it works great. So with Phoenix Channels you can really just act like It Just Works.
(phoenix.js is one of the best parts of the library IMHO - its design beats eg socket.io dead out of the water)
[EDIT] I see another commenter in this thread mentions phoenix.js beating socket.io. Thanks, I will have to look into this!
Phoenix channels make it super easy to setup and use websockets, but you're sort of on your own as far as figuring out how to work REST conventions into it since all the tooling that exists for REST generally assumes and requires HTTP.
It's great to be able to write the code for the API once and support it across a broad variety of transports.
They both justified using elixir because it was "closer to ruby" (???), and they both seemed to live a pretty difficult time with that language. Which makes me wonder : why would you ever want to use such a special language (purely functional isn't for everyone) if you don't have any scalability issue, and don't even use OTP ?
I recommend the book “Programming Elixir”. Even if you end up not picking the language, it makes a good point about how OOP is a natural enemy to concurrency (because of state) and about how transforming data (I.e functional) is a better match. The language runs atop the Erlang VM, which successfully navigated the troubles of concurrency in ISP nodes a long time ago.
Again, worth a read.
Also - the performance can really make a perceptible difference in user experience even if you don't "need" the scalability.
The thing is, going from Ruby to Elixir involves two learning curves: groking functional programming, and then groking processes. They really do need to be learned in that order since, within any given process, you're just operating in a relatively normal functional world.
HTTP requests are very easily envisioned in a simple functional way. You've got request data in, and you're gonna return response data. The interaction you have along the way with OTP based database pools and so forth is as a client, you aren't really having to manage the lifecycle of any of that stuff. In other words to get going with basic web stuff it's more critical to just become familiar with functional programming, and less critical to understand how OTP works. This is totally fine from a learning curve perspective.
What's cool about Phoenix channels though is that each user's connection, and each channel that they run is a GenServer. It provides a great opportunity to get some hands on interaction with the concept in a way that naturally fits into with the web application they're building.
TL;DR:You can be a client / user of OTP processes without knowing the details of how they work, and that's generally enough for HTTP.
aka : you should get to understand erlang design decisions prior to jumping into the tech. Which probably means having more experience than building the typical low load website made with RoR.
After 10 years of Rails, all things considered Elixir and Phoenix are pretty good.
The best feature of the language so far is pattern matching. It can be used in the arguments of function definitions. It makes all the "if"s go away. It seems strange after 30 years of programming, but a program without conditionals is easier to read.
Not having to use an external background job framework is great news too. Just spawn your processes to send a mail, process something asynchronously and store the result in the database. No Sidekiq (or Celery if you're into Python.)
On the awful side, Ecto is too low level and it feels like a self inflicted pain. I understand the reasons behind its design decisions, but the typical web application or mobile backend don't need all of that. Actually the team working with me on this project wrote a lot of code to build queries and wrap them into another layer to return {:ok, result} / {:error, reason} tuples. We're calling them from controllers more or less as Model.get_something(args). I checked it now and we have zero occurrences of " from " in controller code. All the queries are confined inside a model. If we were using Rails ActiveRecord would have written all that code for us. Instead we had to use a more verbose query syntax, write our functions and tests. This is a net loss of productivity.
There are some modules that can be mounted on the top of Ecto to make it look like ActiveRecord or other ORMs for other languages. I'm looking forward to using Ecto.Rut in my next project. I hope I'll never have to use Ecto directly.
https://github.com/sheharyarn/ecto_rut
We almost didn't use the supervisor layer of the language (somewhat like systemd/init for who's not familiar with it.) I used it in smaller projects and I feel like it's a little too complicated. Same for defining a GenServer, more complex and verbose than exposing the same functionality with methods from a object. I know we have to split the code between what runs in the client process and what runs on the server process, but I feel there should be a vanilla GenServer that handles the most straightfoward case inside the behavior, which is no code in the client with the exception of passing the arguments to the server. Maybe use the name, sigils or module attributes to declare if there will be a synchronous return value.
Anyway, Elixir is still young and there is time to smooth the rough edges. So far it's an acceptable contender and definitely better than the popular scripting languages when concurrency is important.
> We're calling them from controllers more or less as Model.get_something(args). I checked it now and we have zero occurrences of " from " in controller code. All the queries are confined inside a model.
Yeah, that's almost a best practice, except you're regarding the "model" (the terminology is 'schema' in Ecto) as primary instead of building it as a separate context or application (depending on your preferred approach to modularity within Elixir/Phoenix). For that matter, I fail to see the advantages of `Model.get` over `Model |> Repo.get(id)`.
> If we were using Rails ActiveRecord would have written all that code for us.
There's not a substantial difference between the kind of AR usage that's on the happy path and the equivalent query in Ecto.
> I know we have to split the code between what runs in the client process and what runs on the server process, but I feel there should be a vanilla GenServer that handles the most straightfoward case inside the behavior, which is no code in the client with the exception of passing the arguments to the server
You mean like an Agent[1]?
I don't think that AR is correct, I think that AR is more convenient than Ecto in many cases we run into in web development. With AR I don't have to write my own queries and encapsulate them into functions (but it's kind of what we do with scopes) and it has a more compact syntax. Ecto is more flexible, but that flexibility is not needed in most cases. Anyway when we must really be flexible we write queries in SQL and manually unmarshal resultsets. ORMs/Data Mappers are for simple and medium use cases.
I still didn't run into problems with the AR approach. Maybe it's because all of my projects were medium or small sized. I found AR to be perfect for them to the point that I want to disguise Ecto as AR using Ecto.Rut.
> you're regarding the "model" (the terminology is 'schema' in Ecto) as primary instead of building it as a separate context or application
This sounds interesting but I fail to understand what you mean. Would you mind explaining or posting a link? Thanks.
> I fail to see the advantages of `Model.get` over `Model |> Repo.get(id)`
It's shorter but not by much. One reason is that Repo doesn't mean much to me, so it could be hidden. What I care about is Model. But getting values out of the db is not such a pain. Inserting and updating is, because they are more verbose. I quote Ecto.Rut for the insert
Post.insert(title: "Awesome Post", slug: "awesome-post", category_id: 3)
# instead of:
# changeset = Post.changeset(%Post{}, %{title: "Awesome Post", slug: "awesome-post", category_id: 3})
# YourApp.Repo.insert(changeset)
Regardless of this discussion, IMHO a thing that Ecto should fix is requiring developers to write both the migration and the schema. It's either the AR way, migration first and auto generated model, or the Python way, model first and auto generated migration. We got some bugs because we didn't write the same things inside the migration and the schema. Mistakes happens and the tools we use should help us not the make them. Ecto is not DRY but it should.
Btw, the compactness argument applies more or less to all of Ruby vs Elixir because of the object.oriented.notation.is.shorter than the Module1.functional |> Module2.way |> Module3.of |> Module4.composing. Aliases help to some degree but they add clutter to the top of the file. It's a little nuisance but pattern matching more than evens it.
I have to look more into Agent (thanks) but it seems exactly the opposite of what I want: write in the module only the code that should run in the server. Probably what I'm looking for is a macro that writes a vanilla GenServer for me hiding all the functions run on the client.
I wouldn't say it is to use more repositories in the future (I also hate future-proofing code) but rather to make it explicit what is happening on the database side. It aligns well with other ideas in Ecto, such as letting the database do uniqueness checks.
Most of the problematic Rails projects I worked with were because of this coupling between business logic and database that ActiveRecord encourages. But this is nothing new, it is one of the top 3 complaints about Rails.
Repo.insert! %Post{title: "Awesome Post", slug: "awesome-post", category_id: 3}
%Post{user_id: current_user.id}
|> Post.changeset(attributes)
|> Repo.insert
Post.create(post_params.merge(user_id: current_user.id))
def post_params
params.require(:post).permit(:title, :slug, : category_id)
end> I still didn't run into problems with the AR approach. Maybe it's because all of my projects were medium or small sized. I found AR to be perfect for them to the point that I want to disguise Ecto as AR using Ecto.Rut.
Perhaps its a stylistic thing, but recent versions of Ecto feel quite natural to me. I frequently use Ecto without a database by using it for embedded schemas (which I don't embed)—they're basically just structs that I can use with changesets a little more cleanly.
As someone else noted, you don't have to use changesets for the happy path interactions like simple inserts. It really just comes down to whether you can live with the Repo as the first thing you type instead of the schema.
> This sounds interesting but I fail to understand what you mean. Would you mind explaining or posting a link? Thanks.
Rails teaches us to look at the model as the primary point of business logic. In my view this is putting the cart before the horse. By putting the Model (big M, not little m) we limit our thinking around abstractions to what we can represent structurally in the database. A well-designed, thought-out application model (little m, not big M) encompasses much more than database tables. With AR, even if you try to create behaviors on fat models, ultimately the only vocabulary you have to work with are those nouns the database allows you.
Ecto does something subtle but important: it demotes your data schema to just that—data. Behavior is modelled in the messages you pass between processes, which is why you see so many people in the elixir community jumping into architectural techniques like eventsourcing. While things like Ecto.Rut add some conveniences, they also encourage promotion of the data to the central artifact of the system. After building Rails applications from small to very large over 10 years, I can say for my part I want to stay as far away from that as possible. Its convenient until its not, and when you can start to feel the pain of it, its very difficult to unwind its effects throughout your system.
re: agents, maybe what you're looking for is ExActor: https://github.com/sasa1977/exactor
Worth noting, especially if you're in the first 12-18 months of using Elixir: OTP is incredible, but it takes time to wrap your head around all its pieces. I usually recommend new users coming from an MVC framework just try to muddle along using Phoenix as they would Rails until they start getting comfortable with things like supervision trees and GenServers... and then the fun really starts. All the rest of it, like Ecto's hands-off approach to the database, really starts to make sense around that time.
After so many years of software development I don't like unnecessary complexity, that's why I'm keen to shave off features from Ecto and GenServer (and a lot of other tools, not only in Elixir) and settle for a subset with an easier API.
That said, you have a point when you write that AR's approach is "convenient until its not, and when you can start to feel the pain of it, its very difficult to unwind its effects". The project I'm working on is an MVP. Who knows where my customer is going to be in 12 month. What I know for sure is that using Ecto cost them some extra time to deliver because of all that boilerplate we had to write. Would I add an extra layer between the db and the logic in a Rails application if its requirements imply complicated logic and interactions? Probably yes. We can put any kind of objects in the models (or lib) directory of Rails, not only ones derived from AR.
> After so many years of software development I don't like unnecessary complexity, that's why I'm keen to shave off features from Ecto and GenServer (and a lot of other tools, not only in Elixir) and settle for a subset with an easier API.
This is what I'm missing—there's essential and accidental complexity, and when I look at GenServer, I see an API that's been shaved down by decades of practice in Erlang to its most essential complexity. Even ExActor is just a set of macros for generating those essential parts—it basically just saves on typing, not skipping functionality. Ecto hasn't had the years of legacy that OTP has, so its API has fluxed a bit in the last few years, but its still something I think is cut down to the bare minimum for healthy database interaction.
This kind of discussion is best handled by email, i think, and mine is in my profile—feel free to send a gist of something that illustrates what you're talking about, and maybe I'll have a better understanding of what aspects of the API are headaches. I'm really curious what I'm missing about your perspective here.
user = User |> User.actual |> User.by_email(email) |> ....
user = User.actual.by_email(email)... Repo.get_by(Post, title: "My post")
(Rails/AR deprecated find_by_xxx methods, promotes find_by() instead - for performance reasons)
I'm very confident in his stewardship.
For a project where scalability is a concern, I'd have no qualms about moving it over. At the very least I'd carve out the slices which most impact performance and implement them in Elixir and deal with the PITA of having two app servers. In a real world project with these kinds of issues I've seen some pretty huge improvements by doing this piece by piece.
That seems to be the case with the majority of vocal (i.e hosting or being on podcasts) people who use Elixir. In a sense, I'm lucky since I came from a C# background and don't have to worry about doing things the "Ruby Way" instead of the "Elixir way".
>why would you ever want to use such a special language (purely functional isn't for everyone) if you don't have any scalability issue, and don't even use OTP ?
There are a few reasons. One being the hype train that has surrounded the language over the past year+ (at least that's when I noticed it gaining steam). Another reason is that they just don't know enough to get into OTP. Elixir is still young and there is a lot of OTP functionality that isn't represented in Elixir that you need to drop down to Erlang to use. Erlang is not the prettiest of languages and has its own learning curve as well. I think eventually the teams that you heard of will get to OTP as get more familiar with the language.
There are a lot of other great reasons to use Elixir though.
On free tier heroku dyno and the $7 postgres hobby dev, performance was around 27,000 messages sent per second before POSTGRES started complaining. It wasn't even the app itself that was complaining, it was postgres. This was sent via websocket using phoenix channels and persisted to the database.
If you're a backend guy, add this tool to your repertoire - you'll become a wizard and just have tighter code with less headaches.
Mainly convert the .ogv to .mp4 with ffmpeg (don't have the command here), create a png palette from the video so your gif colors are more accurate and use ffmpeg to create the gif:
ffmpeg -y -i video.mp4 -vf fps=10,scale=800:-1:flags=lanczos,palettegen palette.png
ffmpeg -i video.mp4 -i palette.png -filter_complex 'fps=1,scale=800:-1:flags=lanczos,setpts=0.25*PTS[x];[x][1:v]paletteuse' todoApp.gif
In Clojure you can still stick something in an atom or volatile and easily update it in place... doing that is a lot more involved in Elixir.