Elixir and Phoenix can do it all
fly.io
fly.io
Elixir and the BEAM make it so easy and pleasant to run extremely complicated infrastructure in the same codebase and in the same application context. Hard problems made easy.
I miss it.
If I didn't have context clues, it'd be difficult to know which one you mean.
Edit: not native but I've always understood it as the first def here https://idioms.thefreedictionary.com/all+but
This stems from "all but" = "almost"[0]
Compare to a disaster described as "the destruction is all-but-total".
That phrase does not mean "you could describe the destruction by any label except total." For example, whatever just happened it definitely not "tiny" or "moderate".
Like “all but dead”. Everything except dead, as in as close to death as possible without actually being dead (and certainly not the opposite of dead, as in perfect health).
I am surprised, both in terms of the difficulty you faced and that you would want to; I have beef with the fact that using Celery with Django runs "worker" process on the application server by default.
It's much better than having to worry about extra infrastructure, just deploy your app and because it's code you control you can customise what the queue is going in really bespoke and application specific ways.
Wow, that's pretty cool
:sigh:
It is common to have:
frontend <-JSON-> backend <-db/queue-> background jobs
Each boundary introduces complexity (marshalling / impedance mismatch / but also different HR/recruiting needs).
In Elixir you can more or less do:
front/back/background
with no boundaries or reduced boundaries (e.g. using LiveView & OBAN in the same process, even if going through Postgres as a queue).
Someone coined the term "deepstack engineer" as well recently I think, which I could draw as:
frontend <-> backend <-> background <-> machine learning
Now with Elixir you can do everything in the same process (but with isolation as required):
front/back/background/ml
And this is something I find really interesting about this stack.
So I can relate :-)
See:
- https://www.youtube.com/watch?v=g3oyh3g1AtQ - https://www.youtube.com/watch?v=HK38-HIK6NA
I'm sure you're right that the nature of Elixir makes wrangling all the different services easier but is there something special it does to make embedding easier as well?
This is usually things around either ETS/mnesia for state storage or doing hot-releases, both of which I find most people in the community recommend _against_.
A few examples:
- Server-wide state - Persistable data - Background Jobs
Having worked in a heavy Erlang codebase and enjoyed some of the features of it, but realized some of the shortcomings where when reliability is a primary concern (medical/healthcare), a lot of the advice changes.
Though I haven't written any F# in a while, I was always curious to see how F# + https://learn.microsoft.com/en-us/dotnet/orleans/overview (virtual actors) would feel comparatively.
Specifically, hot loading allows for tremendous agility in rolling out code: for better and worse. There's sharp edges, but it's worth it IMHO (of course, it should be noted that AFAIK, hot loading is either unused or very rarely used at WhatsApp now. I believe they now use fairly conventional rollouts where new instances are started with the new code, load balancers shift the load and old instances are shut down eventually... back in my day, we didn't have load balancers or easy access to spare machines to rotate through, etc)
Mnesia has lots of sharp edges too, but if you are willing to learn them as you grow, and maybe do a bit of work here and there, disc_copies and ram_copies tables scale pretty good (afaik, we only used disc_only_copies for the schema table, and that's a small table, so it works; I don't think that table type works well with large tables). And having data and application colocated makes a lot of things pretty nice. IMHO, if you are doing the volume of data and accesses we were doing with any other database, you'd need in house engineering on the database as well, so I don't see it as a strong negative that it's needed for mnesia. If you have a smaller need, maybe it works out of the box. IMHO, mnesia works best with stable nodes and stable network though: the original use case was for two nodes in the same chassis, and the farther you stray from that, the more work you may have to do. But we had replication setup between Reston, VA and Dallas, TX and it worked fine as long as the backbone between the two wasn't being extra flakey.
Again, I think post-acquisition, mnesia was phased out, but FB doesn't value stability of nodes or networks.
From the first slide in the linked article, I would almost never use Erlang to store data. I'm always going to reach for Postgres there unless I've got some very specific use case.
I do think having most everything else "under the same roof" is nice for a smaller company that doesn't want to run a whole bunch of different services, though. Having an internal cron thing, and an internal job thing are probably fine for a lot of people getting something off the ground and easier to deal with than having something external to deal with. I think in particular this is often nice when trying to replicate the production environment on both a staging and local development system. If the infrastructure is complicated, that's harder to do. It's nice to be running substantially similar systems everywhere.
I'm not 100% sold that it's that big a win compared to having more libraries and such available with, say, Rails or PHP or Python or something. But it's definitely nice to have.
https://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.JS.htm...
It might be hard if you never used JS before but it not as hard as you think once you build a component.
Also Phoenix comes with common components like modals.
What I ended up doing was loading react components using LiveView hooks and syncing state using pushed events. Saved me a ton of time because libraries were all there, and worked seamlessly alongside my LiveView app.
https://github.com/petalframework/petal_components
https://github.com/coingaming/moon
I personally wouldn't use React for components in LiveView, you are just adding more complexity to your application for no reason. It really not that hard to build components using JS and LiveView. In the end you get less complex components than what you would get with react.
Simple components are taken care of, but I think that's not where the discussion is very relevant.
I'm sorry to disagree here, but I think it's disingenuous to act like the problem is solved when there's really quite a large gap.
(Btw, of course I built my components with pure LiveView when it made sense.)
Depending on what you're building you could also say: "I want to spend time building my business, not writing an API for myself just so I can communicate with my own backend." Which is not even to mention how much simpler concurrency features are. But again, it depends on the application.
To be clear, I heartily endorse LiveView. Not using it now due to pretty heavy chrome extension needs, but it was solid. It's just rough in the "just works JS integrations" department. I'd argue you have to be really good at JS to handle any heavier library integrations.
You aren't wrong there! LiveView was initially developed to make a certain class of web-app without needing JS. Of course, people went and pushed it further and I see it as a "only write the JS that is necessary" type of framework. I'm very fullstack-minded so this doesn't bother me in the least. I love writing JS for DOM manipulations and whatnot, I just don't want to write any business or server-side logic with it, so LiveView is a great fit for me. But even as an advocate, if your application needs to be very JS-heavy, I wouldn't necessarily recommend it. No frameworks are one-size-fits all, but of course most people want to work that way.
That leads to the market responding much more slowly to newer options, even when they have clear advantages. Smaller startups and indie hackers don’t have the same switching costs though, so you’ll see more of them making a bet on a less popular technology they believe has a productivity edge for their use case.
Plus since these solutions already exist, there is less incentive to switch things over to the newcomer when you already have something that is working.
I know from Elm that it does change how things feel. I could see an upside. But coming from a dynamic life I can't quite grok considering it a requirement.
It really depends on your definition of your massive. Discord seems to have a few hundreds to a thousand employees in total. I am from a typical tech company. So I could estimate that their total engineers count is probably just a couple of hundreds, which is about our director/senior director's team size.
In our definition of massive, we could have even hundreds of ppl to touch the same portion of the code base frequently. We regularly have to deal with code which are about 10 years old and it's a moving target. You have to deliver quickly with migration requirement in mind. Also we frequently collaborate with the other teams and you have to be able to understand others' code quickly.
Typing is critical as you don't have time to deal with the constant stream of the new projects to dig deeper in to the objects. And you really want to have some understanding of the code in the first glance. We also want to offload lots of the things to compiler and ide as much as possible.
It doesn't have to be full on static typing like Java, type annotation in python/ruby already helped a lot
massive as in the top 10% of tech companies in the entire globe
> Discord seems to have a few hundreds to a thousand employees in total
quote: WhatsApp scaled to 1B users with only 50 engineers [1]
But are you talking about the size or the impact of the project? I am in the fintech space and the complexity and cruft is just incredible thick. We just needs so many ppl to do the dirty work. There is no way for you to shrink the code base fundamentally
> quote: WhatsApp scaled to 1B users with only 50 engineers [1]
Again, are we talking about the size or the impact? Are you saying all the projects for in faang can be handled like this? Meta currently have about 70k engineers. What do you think that all those people are doing?
justifying the salary of middle managers.
at my current job the number of employees has increased 30x in just 5 years.
there's no important single task we couldn't do when we were 1/30th of now, 2.5x would have been the perfect number to make all of us work no more than 3 days/week, now we are just working on more tasks that are basically DOA or scratch the itches of higher ups who want their name on something when they meet with other higher ups but are useless. Moreover, more often than not, they are a minor variation of something that already existed for a long time, works better, serves a real business purpose and it's written in legacy languages because it just works (I'm sure many of us have attended the and now with AI! BS meeting).
We also know Facebook's 70k employees are there so that many of them won't end up working somewhere else and help the competition (see the Silicon Valley No-poaching Case).
I'm an advocate for not burning up your employees and hire more than you need so everyone can relax, I'm a union representative (on of the) in the company I work for (yes, my country is highly unionized and thanks god it is) so I deeply support the needs of the workers against the absurd demands of the companies, but at some point too many people becomes a burden and severely hinders communication, it basically creates a layers of bureaucracy that resembles much more politics than the job you were hired for, ending up stressing people more, creating harmful frictions, complicated workflows, absurd chains of indirection and countless authorization requests that in the end reduce everyone's output, regardless of the commitment of the single employee.
Also, larger companies get a lot more slack taxation wise and receives a lot more cuts from the government, because unemployment makes us look bad.
That's just ridiculous if you think this applies to all projects. We have incredible amount legal, tax, sanction etc in the code. Have you seen how complex real world financial and legal system can be, do you know how much work to translate those into code? And keeps updating it?
And to just to interface with other financial system worldwide, we have to keep the old system in place, at same time upgrade as the industry move forward. It's a also a big challenge. Despite the above and we had to ship fast, we are actually still SHORT on people. Our engineer to manager ratio is also high. The world is a complex place, sometimes you just have to deal with it.
Again, why do you think 50 man team applies to all projects? What is it so hard to believe that some complexity cannot be reduced?
> We also know Facebook's 70k employees are there so that many of them won't end up working somewhere else and help the competition
You get this from the TV show. Have you actually worked in the companies at Meta caliber? Do you really think most faang employees just do nothing everyday?
you're taking it personally.
I work for a > 30k employees company, not based in the US.
All I'm saying is companies optimize for what's better for them, so at some point you'll have on payroll thousands of people you don't need for your core business because they solve a lot of non-core functions. such as dealing with authorities or devise a strategy to reduce the pool of talents competition can pick from. HR, accounting and legal departments will also increase in size a lot consequentially. so their massive size doesn't says much about the complexity of the tech problem they are solving.
the fact that Discord runs on Elixir at that scale is remarkable, regardless of how many people work at Discord.
YMMV very large project have been written in Erlang/Elixir without static typing and they've been wildly successfull
i.e WhatsApp (based on Ejabberd), Ericsson network switches, RabbitMQ, CouchDB and a lot more, you can read about some of them here https://www.erlang-solutions.com/blog/which-companies-are-us...
as a less knows example, an entire insurance company in my country with > 1 million customers
Glad to see that some people are doing a more serious job than me :)
(also, it's a small world – I know Giuseppe Castagna, wasn't aware that he was working on typing Elixir)
I will say that editor support for static languages is categorically better due to the language server being able to more completely reason about the state of your code. That’s the one drawback.
For example doing "1" + 2 would give you an error.
OCaml takes it a step further with numbers where `+` only works on integers. You need to do `+.` to add floats. This lets it be statically typed without having to actually specify any types.
Again, a little out of my depth in terms of rock solid explanations.
I wasn't meaning to imply everything is perfect, though. For example equality operators take anything. I believe there may be changes coming around there.
https://thinkingelixir.com/elixir-in-the-type-system-quadran....
You omitted the caveat: the program will behave in unexpected ways sometimes. What else do you do to ensure correctness?
ie The minimum bar for modern software is not "it compiles, ship it".
Elixir has great support for pattern matching, guards, documentation, testing and being a functional language all help negate the problems of a dynamic language.
I think the fact that most functions are "pure" / "static" is really helping with refactorings.
Really they are a perfectly regular form of type hinting--as is pattern matching in function heads--when being used for no other purpose than to ensure the correct type is passed at runtime.
This topic has come up quite a few times on HN, but it seems like it’s generally people who haven’t ever used Clojure or another similar immutable functional language that worry about this.
IMO, Elixir and Clojure hit the sweet spot between a language like Ruby and one with an ML-inspired type system.
The "niche" label cast at Elixir and its ecosystem hasn't been relevant for years; it can do what you need it to and with massive reduction in complexity.
If or when you do need to write your own library: you'll find that straightforward as well.
Have a proposal here: https://docs.google.com/document/d/1adXQGiDoqyJ37r8XWYthFFZf...
[1] https://github.com/hbcondo/last10k_liveview
I also did Rob Conery's Elixir tutorial which was fun - https://robconery.com/course/take-off-with-elixir/#/ Absolutely loved the way he did this one as a story but it also didn't feel like I was sitting there forever waiting for him to get to the lesson either. Looks like it's out dated though.
I've been brainstorming writing my own book that covers elixir pheonix crud, auth, background jobs, live view, (+ more?) using a twitter-clone example, which feels sufficiently complex. Basically writing a book that I wish existed.
I can cite some easily available public stats: Scholar, which is compared to numpy, has like 0.001% of its popularity. (While there are rumors people buy likes on github I doubt this is something affecting the numpy project).
Maybe the reason believers in elixir/erlang are more passionate is precisely because there is no obvious reason there isn't major adoption?
My best guess is that the missing ingredient are major corporate interests that directly or indirectly are promoting the technology. E.g the usual big tech suspects seem to be absent. For better or worse, they are the ones dangling lucrative jobs prospects etc.
But if its a technology delivering the goods people would gravitate to it sooner or later. Would be interested if more informed people can add their insights
The ruby community it aimed at initially already moved to go and JavaScript by the time phoenix became production ready.
Erlang is an arcane, hard to understand language which many of the base packages of the elixir ecosystem uses.
I think that’s why it’s not as popular as it should be.
It’s great though. My go to stack is phoenix nowadays.
ML isn’t even close to the primary use case of Elixir either. This is just a bizarre comparison.
Also, Scholar would be equivalent to Sklearn, not Numpy. Nx would be the Numpy equivalent. Scholar has literally only existed for 2 years.
You shouldn't have any issues integrating it into Absinthe, because it doesn't have any ecto bindings out of the box. (There's a helper library, but even that is fairly small.)
I have heard of people using mnesia, but it's not really that common. I'd personally avoid it unless I had a good reason not to.
[1] https://elixirforum.com/t/elixir-ls-error-undefinedfunctione...
[2] https://hexdocs.pm/credo/Credo.Check.Readability.AliasOrder....
Plus, these days there are many alternate LSP implementations besides Elixir LS: