Hacker News Reader

Comment by shijie | Hacker News Reader

We’re experiencing this at my place of work. Our backend stack is Python, and coming from an Elixir/Phoenix background, adding a durable queue to our infrastructure should be trivial but because of the nature of Python, it is all but impossible to run the queue in the same codebase and application server.

Elixir and the BEAM make it so easy and pleasant to run extremely complicated infrastructure in the same codebase and in the same application context. Hard problems made easy.

I miss it.

View on HN

Something I often "draw" to other developers when showing Elixir is the difference in terms of "boundaries".

It is common to have:

frontend <-JSON-> backend <-db/queue-> background jobs

Each boundary introduces complexity (marshalling / impedance mismatch / but also different HR/recruiting needs).

In Elixir you can more or less do:

front/back/background

with no boundaries or reduced boundaries (e.g. using LiveView & OBAN in the same process, even if going through Postgres as a queue).

Someone coined the term "deepstack engineer" as well recently I think, which I could draw as:

frontend <-> backend <-> background <-> machine learning

Now with Elixir you can do everything in the same process (but with isolation as required):

front/back/background/ml

And this is something I find really interesting about this stack.

Interesting, most of my work has been 'across the stack' but for desktop applications. This looks like a better match for me than most other approaches to web applications.

It is a very good point: I did a lot of desktop apps at one point (the largest being a mixture of C# and legacy C++ wrappers), and the first time I tried LiveView it somehow reminded me of WinForms!

So I can relate :-)

I'm curious, how do you run ML in the same process?

It is just the way it is with Elixir - everything can run in the same process (or in different connected nodes):

See:

- https://www.youtube.com/watch?v=g3oyh3g1AtQ - https://www.youtube.com/watch?v=HK38-HIK6NA

Every time I see microservices, or Kafka, or Redis, ... I can't help being disappointed because we seem to be stuck reinventing BEAM/Erlang (and Elixir) 30 years after the fact, except with technologies that are slower, more fragile, more complicated to use and harder to deploy.

:sigh:

Someone in the Erlang community named our processes “nanoservices” as a joke. I thought that was pretty apt though, a gen_server is pretty close. And you can have 3M of those on one node!

I think it's more like Erlang processes are microservices and microservices as we know them today are usually "regularly-sized services" :D

> it is all but impossible to run the queue in the same codebase and application server.

I am surprised, both in terms of the difficulty you faced and that you would want to; I have beef with the fact that using Celery with Django runs "worker" process on the application server by default.

It sounds mad but because of the process isolation within the language you can trust running services within your application and message passing can be local or to a different machine in the BEAM cluster. It opens up a whole new way to think about a runtime once everything is an actor model process with a mailbox you can send messages to.

It's much better than having to worry about extra infrastructure, just deploy your app and because it's code you control you can customise what the queue is going in really bespoke and application specific ways.

> to a different machine in the BEAM cluster.

Wow, that's pretty cool

Wait until you find out you can connect an interactive shell to production and run functions in your application to debug things live…

But why? I have one project that is a single Python codebase that embeds the whole http server, task scheduler, a durable queue implementation, and job processing implementation using those queues all in a single process.

I'm sure you're right that the nature of Elixir makes wrangling all the different services easier but is there something special it does to make embedding easier as well?

Your performance is going to be comparatively terrible because of the way those tasks are scheduled in Python

How do you dispatch to multiple cores in a single process in Python?

Care to share the code of this project? I am looking to build something similar.

Are there any examples online of doing these things with simple (or just not as complex) elixir code compared to using dedicated libraries in other languages

FYI "all but impossible" usually means the opposite of what you think it means - "all but" meaning "everything except". Seen this usage a lot and it bothers me, because both usages are common and theyre the exact opposites.

If I didn't have context clues, it'd be difficult to know which one you mean.

Edit: not native but I've always understood it as the first def here https://idioms.thefreedictionary.com/all+but

It's an idiom and means "nearly impossible". There is no other usage.

In this context I meant “all but” == “everything except” i.e it is almost impossible. Are you saying it means the opposite of how I used it? I’m not certain what you mean.

100% of the usage I've encountered as a native speaker is "all but impossible" = "almost impossible".

This stems from "all but" = "almost"[0]

[0]https://www.merriam-webster.com/dictionary/all%20but

Agreed, to me the meaning unpacks like: "it meets or exceeds every category of difficulty that is less severe than Impossible."

Compare to a disaster described as "the destruction is all-but-total".

That phrase does not mean "you could describe the destruction by any label except total." For example, whatever just happened it definitely not "tiny" or "moderate".

I'm not sure how it makes sense to claim that an action is "everything except impossible." This is more in the classic sense of "the battle was all but lost" for a battle was practically but not actually lost. You could implement a queue in a Python server, but it would be technically awful to do.

“All but” = “almost” (which presumably historically was “all most”).

Like “all but dead”. Everything except dead, as in as close to death as possible without actually being dead (and certainly not the opposite of dead, as in perfect health).

Reply on news.ycombinator.com