Hacker News Reader

Comment by kccqzy | Hacker News Reader

This is some very valuable perspective. Personally, I previously also struggled to understand why. For me, the thing that clicked was to understand protobuf and Cap'n proto as serialization formats that need to work across API boundaries and need to work with different versions of their schema in a backwards- and forwards-compatible way; do not treat them as in-memory data structures that represent the world from the perspective of a single process running a single version without no compatibility concerns. Thus, the widely repeated mantra of "making illegal states unrepresentable" does not apply.

View on HN

Rich Hickey (creator of the Clojure language) has a good talk "Maybe Not" that touches on these issues, with a nice way of explaining it

https://www.youtube.com/watch?v=YR5WdGrpoug

The capnproto link explains it concretely in terms of a message bus example, which is useful.

But more abstractly you can think of the shape of data (aka schema, names and types) and field presence (optional/required) as separate things

https://lobste.rs/s/zdvg9y/maybe_not_rich_hickey

First, being valid or invalid with respect to a static type system is a GLOBAL property of program -- writing a type checker will convince you of that. And big distributed systems don't have such global properties: https://news.ycombinator.com/item?id=36590799

If they did, they'd be small :) Namely you could just reboot the whole thing at once. You can't reboot say the entire Internet at once, and this also holds for smaller systems, like the ones at say Google (and I'm sure Cloudflare, etc.).

So the idea is that the shape/schema is a GLOBAL property -- you never want two messages called foo.RequestX or two fields called "num_bar" with different types -- ever, anywhere.

But optional/required is LOCAL property. It depends on what version of a schema is deployed in a particular binary. Inherently, you need to be able to handle a mix of inconsistent versions running simultaneously.

---

To be pedantic, I woulds say "making illegal states unrepresentable" DOES apply, but you can't do it in a STATIC type system. [1] Your Maybe<T> type is not useful for data that crosses process boundaries.

A distributed system isn't a state machine.

1. Lamport showed us one important reason why: the relative order of messages means that there is no globally coherent state. You need something like Paxos to turn a distributed system back into a state machine (and this is very expensive in general)

2. The second reason is probably a consequence of the first. You can think of deploying a binary to a node as a message to that node. So you don't have a completely consistent state -- you always have an in-between state, a mix of versions. And presumably you want your system to keep working during this time period :)

And that coarse-grained problem (code versioning and deployment) implies the fine-grained problem (whether a specific message in a field is present). This is because protobufs generate parsers with validation for you -- or they used to!

---

tl;dr Think of the shape of data (aka schema) and field presence (optional/required) as different dimensions of data modeling. Maybe<T> mixes those up, which is fine in a single process, but doesn't work across processes.

---

[1] A very specific example of making illegal states unrepresentable without static types - my Oils project uses a DSL for algebraic data types, borrowed from CPython. The funny thing is that in CPython, it generates C code, which doesn't have any static notion of Maybe<T>. It has tagged unions.

And in Oils we first generated dynamically typed Python at first. Somewhat surprisingly, algebraic data types are STILL useful there.

Now the generated code is statically typed with MyPy (and with C++), and we do pleasant type-driven refactorings. But using algebraic data types were still extremely useful before static typing. They made illegal states unrepresentable -- but you would get the error at runtime.

I wonder about how to make this play nicely with systems that have different perspectives. Yes, a message bus is written to deal with any possible message and it can do that because it doesn't care what's in the message. Incomplete messages are useful to have, too.

This is sort of like the difference between a text editor and a compiler. An editor has to deal with code that doesn't even parse, which is easiest if just treats it as a plain text file, but then you're missing a lot of language-specific features that we take for granted these days. Meanwhile, a compiler can require all errors to be fixed before it emits a binary, but it has to be good at reporting what the errors are, because they will certainly happen.

It's unclear to me how the type of the field can be a global property in a large system. From a text editor's point of view, you can just edit the type. How can anyone guarantee that a type is always the same?

Also, SQL tables actually do make closed-world assumptions; every record meets the schema, or it can't be stored there. If you change the schema, there is a migration step where all the old rows in the production database gets upgraded. This doesn't seem unrealistic?

I guess it's unrealistic that you only have one production database, and not also a staging database, and every developer having their own database? And they will be on different versions. As soon as you have lots of databases, things get complicated.

Yes databases and persisted data are an even bigger problem -- it's not enough to "reboot the Internet", you would also have to migrate all the data it stores to a different format!

I started using the terminology "interior vs. exterior" for these issues, and I find it pretty helpful - https://www.oilshell.org/blog/2023/06/ysh-design.html

e.g. type systems are interior, network protocols and persisted data are exterior.

SWEs tend to reason about the interior; SREs tend to reason about the exterior. Every problem a SRE deals with has passed type checks.

I see many fallacies where programmers want to think about the interior ONLY. They want the type system to ensure correctness. But that can be taken too far -- there are some things the interior view can't (or doesn't currently) handle, like mixed versions of binaries, schema migrations, etc.

The key point with databases is that your schema is LITERALLY dynamic -- it lives in a process outside your program, outside your type system (unless your program consists entirely of stored procedures, etc.)

Of course most people have some kind of synchronization or ORM (with varying degrees of success). But the point is that the exterior reality is the thing that matters; the interior is just a guide. "When models and reality collide, reality wins"

---

On the other hand, I think there can be more static tools in the future -- if they are built to understand more of the system; if they're not so parochially limited to a single process.

But I think these issues are never going away -- quite the contrary they will increase, because software tends to get more heterogeneous as it gets bigger. It's tempting to think that someday there's going to be a "unified platform" that solves it all, but I think the trend is actually the opposite.

The other issue is that while type systems can get better, they're mostly applicable when you control both sides of the wire.

Most orgs writing software do not control both sides of the wire -- they are reusing services like Google Maps or Stripe. In the last 10 years it seems like every org is now relying on 50 different third party integrations (which btw makes the web super slow ...)

As I mentioned in the previous comment, even if you can somehow go into the repo of Google Maps or Stripe, download their schema, and compile it into your binary, that STILL doesn't give you any guarantees.

They can upgrade or roll back their binaries whenever. They might switch from free-form JSON to JSON schema to Swagger, etc. You don't control that decision.

The people on the other side of the wire may have started using protobufs 10 years ago, and don't feel like switching to whatever you think is great right now. There's a lot of heterogeneity across time too, not just across different tech right now.

So fundamentally the problems are with "the world" the type system is modeling, not about or within the type system itself!

It seems like protobufs are sort of a mixed interior / exterior model? In an organization that uses centralized protobuf schemas and generates serialization code from that, you can assume every protobuf record of that type came from a serializer that obeys some version of the schema.

Why can this guarantee a field's type but not whether it's required? Because fields have a lifecycle. They get added to the schema, used for a while, and later become obsolete and are deleted.

The maintainers of a protobuf schema can guarantee that a field number is always associated with a certain type, and that when the field is deleted, that field number is never reused. This makes (field number, type) tuples timeless. But they can't say whether a field will be needed forever, and they can't control which versions of the schema are still being used.

Effectively, "required" means "possibly required forever." As long as there is some parser out there that treats the field as required, deserialization will break if you leave out the field.

Changing a field's type is easier, because you don't really do it. You add a new field number with a new type and maybe eventually stop emitting the old field.

This suggests a strategy: fields can be required as long as they're not top-level. You can define coordinates(42) to always be (x,y) where both x and y are required. If you change it to be a 3D coordinate, then that will be coordinates(43) = (x,y,z), and you never reuse field number 42 in that schema.

The strategy illustrated at the end still ignores the learning of the FAQ. What if, let's say, one microservice is responsible for calculating the x coordinate but another microservice is responsible for calculating the y coordinate? The former microservice could rightly fill in only the x coordinate and ask the latter microservice to fill in the y coordinate.

Then you start to make compromises such as defining an IncompleteCoordinate message that requires x. But what if later you calculate y first and use that to calculate x?

Etc. Etc. I find it more convenient to make everything optional at the wire format level and explicitly validate what is required at the point of use, not at the point of parse.

It seems like an edge case and you could define another message altogether for the microservice traffic.

But yeah, for larger child messages, copying fields would be annoying. Often you want to be able to create child messages without creating the parent.

Reply on news.ycombinator.com