Comment by JonChesterfield | Hacker News Reader

Comment by JonChesterfield | Hacker News Reader

Parent Full thread

JonChesterfield

This might be an interesting project. "Type safe RPC" is not a great description to lead with as remote procedure calls have a large amount of negative associations that have little to nothing to do with static type systems.

I think it's a distributed language runtime where you write functions in typescript and the runtime deals with passing data between machines.

The website is quite good at presenting the system as a nice thing that does nice things and you should like it and it's all very nice. It does a very poor job of convincing me that it works properly. I'd want to know how the scheduler works and how it handles mutable state in the presence of network partitions.

Also just found a caveat which amounts to they've got the implementation wrong:

> Arguments must be JSON serializable. This means that you can pass strings, numbers, booleans, arrays, objects, and null. You cannot pass functions, promises, or other non-serializable objects

Author here.

> It does a very poor job of convincing me that it works properly. I'd want to know how the scheduler works and how it handles mutable state in the presence of network partitions.

Thank you. This is valuable feedback, and will definitely work this in to the docs.

It's probably an audience thing.

The current pages are a strong sales pitch to semi-technical management. Look at all the things you don't have to implement!

I read this looking for indications that you've built the complicated thing correctly. Do you understand the maths involved, have you implemented it in a fashion that inspires confidence.

A distributed language runtime that works properly is a valuable thing. One that kind-of works when held carefully is a liability that people would regret trying to build on.

What do you mean by “they’ve got the implementation wrong”?

> You cannot pass functions, promises, or other non-serializable objects

Functions are serializable. And deserializable. You can absolutely take one defined on one machine, kick it over the wire, then execute it on another one. Likewise closures, you serialise the associated state as well.

Promises are the same thing - a distributed language runtime should be exactly the sort of graph execution system which lets you pass promises between machines and have things work out.

Further, they don't say what other "non-serializable" things are. Is a DAG serialisable without spuriously unrolling it into a tree and failing to fold back into a DAG at the other end? How about a graph? Given that all things are serializable it is difficult to guess what things they haven't implemented.

So by "they've got the implementation wrong", I'm saying they haven't implemented the tricky part of serialisation, only the copy ints over wires part. And by "wrong", I'm saying that a language runtime dedicated to transparently running typescript functions on other machines should be able to run those functions on other machines even if they have closed over state.

Author here. Absolutely agree with what you're saying.

Maybe the line could be written better. It is supposed to say: You can't pass functions and promises. And you can't pass in other data types that are not serialisable.

I definitely did not intend to say that functions or promises are not serialisable, and it is something we're looking into. The complexity is with serialising the closure state.

The other thing we're implementing is constraining the data types via the type system itself.

edit: punctuation

All data is serialisable. I'd say that's axiomatic. Data is information with a concrete representation.

At present the docs say it can serialise things that it can serialise with a couple of examples on each side.

My best guess is there's some limitations caused by the stack under your implementation that you haven't worked around, but I don't know what those limitations are or which ones you plan to resolve.

In addition to what can be sent there's a load of design choices about what is sent.

Is aliasing within the original preserved?

Is potential aliasing introduced?

Is it compressed in transit?

Is it delta encoded with respect to information that was sent previously?

Is there some initial blob of information every instance starts up with that messages are deduplicated against?

And that's before the behaviour on mutation - when one machine changes the data, how and when is that change reflected on other machines?

> All data is serialisable. I'd say that's axiomatic. Data is information with a concrete representation.

I Do not disagree. I think the problem comes from us using the term serializable to liberally.

> My best guess is there's some limitations caused by the stack under your implementation that you haven't worked around, but I don't know what those limitations are or which ones you plan to resolve.

Agree. Feedback received, and thanks. This will be worked in.

But I'll attempt to answer your questions here:

> Is aliasing within the original preserved?

No. The data being sent over the network is copied by value, not reference - if that makes sense.

> Is it compressed in transit?

Yes. With Message Pack.

> Is it delta encoded with respect to information that was sent previously?

I'm not sure I follow, but all calls are independent of each other.

> Is there some initial blob of information every instance starts up with that messages are deduplicated against?

> And that's before the behaviour on mutation - when one machine changes the data, how and when is that change reflected on other machines?

Not sure if this helps, but this [1] talks about the architecture. There's a C&C. The control-plane is the orchestrator for all machines.

[1] https://docs.differential.dev/advanced/architecture/

I'm guessing, but it should be possible to pass anything remotely as a proxy with an ID: calls to a closure or the resolution of a promise cause further RPC, and sending it back recovers the original value.

Not sure, but functions are serializable.

It’s technically correct but really we all know what was meant. They’re serialisable but not really deserialisable, even if it’s a pure function it might not even be serialised in a version of JS that the receiver understands.

> serialised in a version of JS that the receiver understands

I don't think the use case for this is as general as most RPC systems.

The linked site envisions a situation where you had an app running on one server, now you want to break that workload up while making the absolute minimum number of code changes.

All the clients and code would still be controlled by the same team.

that's still not something I'd want to inflict to myself. That means you need to somehow ensure the functions you pass are pure, and that upgrading Node or changing the compilation target could break the app if the release rollout is progressive.

Using function serialisation for applicative purposes is always going to be a hack that will come back to bite the team at one time or another.

Reply on news.ycombinator.com