Hacker News Reader

Comment by DuLR10 | Hacker News Reader

We haven't touched the distributed part of the game, but our understanding is that when that time comes, it will be easier to use the BEAM approach given that it was made for this purpose.

Given the experience so far, it seems that using Erlang was the correct choice, not only because of the above, but also because Erlang made the server implementation way easier than we thought.

View on HN

I see now, you are sending messages directly to Erlang server so you don't have to worry about network sockets.

In my experience the issues with Erlang come with working with data structures, records are not flexible and there is not much one can do to abstract the boilerplate.

We are using maps all over the place in the server, and so far nothing has been annoying.

I gotta say though that lack of infix custom operators for the monadic bind is a pain.

Take a look at Erlang's "parse transforms" which would allow you to implement some syntactic sugar. That's what is used to implement qlc, the query language for ETS/Mnesia - that one adds new semantics to list comprehensions, but you can modify any part of the syntax you want.

Also, Elixir supports macros and infix operator overloading - have you considered using it? If you know Erlang's stdlib and BEAM, switching to Elixir (and back) is almost painless. Not sure which monads you needed, but `with` macro is built-in, and it's a monadic bind for Option types (well, more or less). Adapting it for other monads shouldn't be hard.

Thanks, I will for sure take a look!

For reference (for the "parse transform" approach in Erlang): https://github.com/rabbitmq/erlando - it doesn't look maintained, but it's probably still usable; otherwise, you might get some inspiration from the code :) This also (ab)uses list comprehension syntax:

    write_file(Path, Data, Modes) ->
        Modes1 = [binary, write | (Modes -- [binary, write])],
        do([error_m ||
            Bin <- make_binary(Data),
            Hdl <- file:open(Path, Modes1),
            Result <- return(do([error_m ||
                                 file:write(Hdl, Bin),
                                 file:sync(Hdl)])),
            file:close(Hdl),
            Result]).

(if one of the calls returns `{error, Reason}` tuple, the execution terminates and the tuple is returned; otherwise, `{ok, Value}` tuple is unpacked)

Having a potentially untrusted client connect to the erlang node as a c_node (which seems to be what zerl does) is not a good idea generally, as connecting that way essentially allows the client to execute arbitrary code on the server.

Please correct me if I say anything wrong.

As far as I can tell, this is not possible at all; the serialization layer (Zerl) cannot send arbitrary code to another node. Now, assuming we implement this, I also think this is not possible due to how the server is designed; based on supervisors and child processes for user sessions.

We recently became aware that you can indeed send tuples that have fixed effects when using the supervisor behavior, so it may be totally possible and probable that one could exploit this vulnerability to some degree in our server. We plan to investigate more about it as we continue to learn more about OTP and the BEAM.

If you're using zerl on the client and plain dist on the server; the question isn't what Zerl can serialize, but what the server will process.

With stock OTP dist, there is no barrier between nodes. Stock OTP runs an rpc server that you can use to send function calls to run, which can include BEAM code to load (or file I/O to do); and even if that's disabled, you can spawn a process to run a function with arguments on a remote node without needing an rpc server at all.

How can one protect the server then? Do we need some special behavior and/or library?

I'm not aware of anyone running a limited dist to allow for untrusted dist clients. But here's an OTP response to a proposal that's pretty clearly a no [1].

It'd be much simpler to put together a custom protocol to communicate between the client and server. You could use Erlang's External Term Format to exchange data if you want, in which case you'd want to do binary_to_term(Binary, [safe]) to prevent creation of new atoms and new function references which can fill up tables and also consider that just because deserializing is safe for the runtime doesn't mean you can trust the client.

Erlang makes it pretty easy to parse sensible things off of network sockets, if you want to go more custom, too. Binary pattern matching is lovely.

[1] https://erlangforums.com/t/rfc-erlang-dist-security-filterin...

Thanks for the ideas and references! I gotta say, though, that I will be pretty sad if having to write a custom protocol turns out to be the final solution. So much more convenient to use OTP (especially now that we finally have an infinitely extensible serialization library for it; Zerl). I'm shocked such an oversight would exist in a real commercial solution which is the BEAM.

The original application of dist clustering was dual computers in a single telecom switch. There's not really a need for a security barrier in that case; anyone with access to one computer would be expected to have access to the other.

Additional applications for dist have been explored over the years, but most of them involve clustering servers; where a security barrier isn't necessary; although it might be desirable --- I've used dist clusters where some people had access to only certain types of nodes; bypassing access control using dist clustering was certainly a possibility. Bolting security onto something designed without it often is pretty challenging. Especially if you want to keep all the existing applications working.

There's a good (new) library in Elixir that may work for this use case called Zigler https://hexdocs.pm/zigler/Zig.html

As another commenter said, OTP messages are meant to be between processes in the same privilege zone. That said, using a custom protcol via a good library can actually bring benefits relative to core OTP stuff.

For example, several of the gRPC libs I've used for Erlang/Elixir are pretty low-cognitive-overhead to use, and they come with all the added gRPC goodies: RPC semantics are described in one place rather than ad-hoc throughout code, protobufs have at least a documented (if not actually good) process for upgrades and backwards compatibility, multilanguage gets easier (even if your second language is just a tiny sliver of "dump protobufs into a database/Jupyter notebook/Rust program occasionally for offline reporting").

To be clear, this isn't a paean to gRPC; most of those features are table stakes for an IDL-driven protocol definition. Just saying that you do get some things in return for giving up the convenience of OTP, if you pick the right tools.

I think this behavior can be fixed by properly using something like `lib_chan`, but we needed something that worked first for our Func Prog Sweden demo.

Indeed a malicious client can craft an brutal kill message as long as it knows the PID a process (either a worker or a supervisor) for instance.

Reply on news.ycombinator.com