Comment by strommen | Hacker News Reader

Comment by strommen | Hacker News Reader

In 2006 I worked briefly on an air-traffic control system (ERAM) that was eventually delivered 5 years late and $400M over budget. I was a small cog in a giant government-contract-software machine, but it was easy to see the project was going to be a disaster.

There are some pretty basic strategies to make building "large" software easier. All of them were attempted, but in an entirely broken way:

- Break the architecture down into smaller components that can be tested independently.

- Break the functionality down into smaller features that can be released independently.

- Develop automated tests at the system level (i.e. integration tests, not unit tests).

The software took 2-4 minutes to launch in development, and very few of us knew how to meaningfully test it. This meant that any change that compiled (and passed code review) would ship.

Oh yeah - and the system used binary structures in shared memory to communicate across processes, which were running code written in two entirely different languages.

> the system used binary structures in shared memory to communicate across processes, which were running code written in two entirely different languages.

What could go wrong

"We should have just used XML for everything!" - said every Java developer ever.

"We should just have used binary for everything!" - said every C/C++ developer ever.

"We should just have used Json for everything!" - said everyone currently.

Any reason not to use JSON? Apart from needing lightweight traffic, in which case MQTT would be fine, right?

* You need strict typing * You need well defined numeric formats * Message volume is high enough that the parsing costs of messages outweigh the savings on implementation time * You cannot guarantee plain text messages won't be managed in transit.

You can get around the strict typing by adopting a schema format, but then you have a tooling problem that all the languages you need to support may not support your chosen JSON schema format.

Other solutions such as protobufs exist and are far better for transmitting large amounts of binary data.

Dates are a thing.

> "We should just have used Json for everything!" - said everyone currently.

Protobufs FTW.

1. Not human readable in serialized form. Unless you have concerns about traffic or storage costs, human readable formats win.

2. Requires code generation. Not a big hassle especially if your IDE hides it, but a hassle nevertheless.

> Requires code generation. Not a big hassle especially if your IDE hides it, but a hassle nevertheless.

I've never understood the objection to code generation (or IDLs). If you actually care about what the serialized form looks like (and compatbility, etc.) this is by far the sanest way to do it.

It's also incredibly simple (i.e. transparent to inspection), it's a good way to do cross-platform/language marshalling.

reproducability is the main issue and IMO determines whether generation is a hassle or not.

Unless you have full control over the delivery tool-chain, then generation is not much of a problem, even less so if your stuff is open source.

But what if you're building software that generates code for long running projects/code bases (e.g. aircrafts, defence, power-plants, telecom equipment, medical devices, mobile radio base stations). Selecting some closed source tool to generate foo can cost you dearly.

You have to version these tools (e.g. the binaries if you don't have the source) to the generaters in your VCS to provide reproducability. Ok you can also ignore all this and simply version what was generated and lose control over the generator but that is a horrid idea and you would never do that of course.

> But what if you're building software that generates code for long running projects/code bases (e.g. aircrafts, defence, power-plants, telecom equipment, medical devices, mobile radio base stations). Selecting some closed source tool to generate foo can cost you dearly.

Yes, that would be insane. So don't do it? If you willfully chooses badly suited tools, then you get what you deserve.

> You have to version these tools (e.g. the binaries if you don't have the source) to the generaters in your VCS to provide reproducability.

You can vendor source code generators. If you want to be extra-certain that generator output remains the same across upgrades, you can also check-in the generated code and manually inspect any changes that happen. (I'm assuming relatively sane source code generators that don't intentionally obfuscate things.)

So, yes, if you choose a terrible code generator then you'll be in trouble.

What alternative approaches would you suggest, incidentally, where you're not also just as much at the mercy of the whims of the tools you're using?

I mean, you could argue the network stack is a "binary structure in shared memory to communicate across processes". Really it depends on how it's done: the interface is everything.

There is nothing wrong with binary protocols.

Sharing c structs across languages is brave but it is doable as long as padding/offsets/lengths are expertly crafted and very strictly controlled.

but on the other hand sharing pointers would be madness.

Clay Johnson had some relevant comments during the flawed healthcare.gov launch. Many critiquing the procurement process itself. As you stated, the process generally favors the "government-contract-software machine":

http://blog.dobt.co/2013/10/07/the-healthcare-gov-fiasco/

Reply on news.ycombinator.com