Hacker News Reader

Comment by wrs | Hacker News Reader

Other needed topics in this series: “You have built a database”, “You have built an orchestrator”, “You have built an RPC layer”, “You have built a build system”…

View on HN

And of course the classic "you have re-invented TCP"

...which often happens when somebody creates a UDP-based protocol and then adds their own reliability and robustness on top of it ;)

To be fair to people reinventing TCP, the two transport protocols that will reliably traverse the Internet are often not great fits for most applications. Most people either want to process multiple streams or datagrams independently[0] (which TCP can't do) while having some amount of reliability guarantees[1] (which UDP can't provide).

And if you don't have that problem then you're probably also reinventing HTTP as well as TCP.

[0] without head-of-line blocking

[1] ideally with controls over how reliable we want our messages to be. Real-time usecases like videoconferencing or multiplayer games tend to fail horribly over TCP.

This is why QUIC/http3 is happening right?

Yes.

The last "You have built a browser" lead to Ladybird and is now sponsored. (I've heard or read its creator say that its web engine was initially meant to display help pages in Serenity, or something like this)

And Greenspun's 10th rule I suppose.

For those that needed to Google like me:

"Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp."

In the age of horizontal scalability, what you've built is likely more like a nightmare version of Erlang, rather than Common Lisp.

It's the Cassandra curse of computer scientists. Business people think they don't have time or resources for correctness, and end up forcing incorrect systems to be held by hand. Which, despite the high cost of this labor, is a decent trade-off because thanks to the internet business models have very low marginal cost per costumer. I think that the fact that Netflix has a very high marginal cost per costumer and actually hires very highly qualified programmers and pioneers systems designs illustrates my point.

You can see that with so called "Big Data" tools. Those which originated as databases (Mongo), ended up adding "Map-reduce" feature. Those which started as map-reduce tools evolved to support SQL (Hadoop->Spark). Those which started as SQL engines (Spark) added support for streaming, while those who started as steaming platforms (Kafka) added SQL support (KSQLDB). Traditional DB engines evolve to allow document data (Postgres with JSON column type). One more decade until one-tool-to-rule-them-all emerges :)

Would you consider orchestrator and scheduler as two separate things or variations on a common theme?

To me, an orchestrator would certainly have a scheduler in it, but also handle higher layer things like data flow DAGs, event triggers, retries, temporary storage, caching, error propagation, etc.

here! https://dx.tips/oops-database

lol I have accidentally written databases for embedded systems many times. Usually the team lead or architect of the project doesn't see the need for it until we already have too much data and we're close to releasing the product, at which point it's impossible to retrofit.

Reply on news.ycombinator.com