Glamorize your problem domain (2022)

There are useful mental analogies between pedestrian line-of-business CRUD apps and a compilers. I don't disagree with that thesis at all. But I have seen it taken too far. Many engineers abstract and generalize business requirements until they find them interesting, and then implement against those requirements.

Programming languages are typically designed to be relatively simple and easy to compile. Line-of-business software tends to be as complicated and arbitrary as the organization requires. Trying too hard to make your mundane app into a compiler can push the software in a direction that makes it difficult to change in the messy ways you'll need to later on.

Note, for instance, that the author worked on a system that generates SDKs in multiple languages. This is pretty obviously a sort of compiler. Are your requirements really so fancy? Or are you making a CRUD app hooked up to a couple external integration APIs?

It's useful to see the analogies between one's domain and compilers, but see if you can't keep that analogy in your head and out of the code whenever possible.

Funny. This is happening at work. Like, technically, we're just sysadmins with a lot of systems and different services running on those. Kinda boring, just like a crud service. But if you squint a bit, we're seeing ideas of formal verification, algorithmic correctness proofs and empiric experiments creeping and seeping into gaps.

Just today, me and a coworker kinda laughed about the fact that this was the second or third time we kinda ended up with an induction proof about the overall correctness of our backup retention handling for off-boarded customers, correctness being dictated by the contractual regulations of our customers. When we stopped laughing, our team lead pointed out a logical problem in one of the steps, and we were eventually able to construct a counter-example which would violate the contracts. Dang it.

Call me silly, but at this point I'm entirely ready to take a day or three off of other work to pick up a formal verification tool to validate our backup retention strategies against the different classes of contracts we have.

In a similar vein, we're starting to work towards certifications like an ISO27001. And there is a staggering amount of similarity between policies and invariants, if you look at them on a systemic level. And from there you can enter the ideas of different algebraic constructs - just like you can have broader and narrower clasess of servers with different amounts of invariants holding or not holding for them.

On top, we've started to pull data from large numbers of system to jam them through jupyter notebooks so we can use data visualization tools to get a stronger impression of what reality is like. It's pretty interesting if you can use data visualization tools like scatter plots and such to deduce the existence of a default value for a config setting. Or, to pull data and to realize: "hello. This is reality calling. Two out of three assumptions you made about systems are incorrect."

I find this applies to media production too. When you use something like LaTeX to write a report, it seems kinda obvious that the .tex file is source code and the generated .pdf is the binary built from that source. You use markdown to write blog posts and then use jekyll, hugo or similar to "compile" it into a static site. But it applies broader too, to images and video for example. If I import an image file into my app, but I don't have access to or control over the source image file that it was scaled and exported from, then I'm essentially building a dependency on a compiled binary that I don't have the source code for. If Apple decides that all iOS apps should now have a @4x high rez version of all images, then I'm not able to create one.

There's a lot of lessons from software engineering that can be expanded to other fields with that insight. Best practices for version control of source code applies to anything that can be called a kind of "source". And you need not just to maintain the source, but also the tools with which the source is edited, and the pipeline with which that source is transformed into a shippable product.

There's value even in the basic idea that any given "thing" can have multiple distinct representations, with one optimized for editing, versioning and collaboration, while another representation might be optimized for distribution and/or consumption (and you might have intermediate representations optimized for other steps in the pipeline toward distribution). It might be overkill for many use cases, but it's helpful simply to know that it's a sane model to evolve toward as a project grows.

Interpreters/compilers are one of the most fundamental patterns of computer science. They recur everywhere, but you need the understanding and the eyes to see them. Ignore them at your peril.

There are so many analogous concepts in the field of computer science and beyond (what might be called 'duality' in a mathematical setting) that if you actively look for ways to apply the wisdom and theory of another domain you will probably succeed. What an inspiring blog post - pardon me, I'm off to brush up on my Yacc skills!

Not sure about "compilers" specifically, but every software may be considered a domain-specific interpreter, if you consider it ingests instructions and mutates memory as a result. Many powerful software tend to be built around this idea: Emacs, AutoCAD, Postscript, OpenGL...

Oh no. I wrote a mini-language with a PEG parser and an AST. I guess I'm a compiler engineer now.

To me this reads like satire. But no one in comments is reading it that way so maybe not?

Glamorize your problem domain

by ftxbro - March 27, 2023

I gave a conference talk last year, tldr: “we build apps. Here are all the lessons we’ve learned from framing this as a blockchain problem.”

My introductory joke was “how to become a blockchain expert in three easy steps”.

    Work in software
    Work on a project
    Decide that whatever you’re working on is a blockchain

But was I truly joking? I genuinely believe teams should try really really hard to draw analogies between what they are building and “classic”, well-understood, formally studied systems and ideas in computing. Is your web API a blockchain if you squint? Is your authentication system really a smart contract if you squint? Squint, I tell you, squint!

The reason is not so you will somehow be able to apply advanced techniques from research journals to your web apps. You won’t. But there are other advantages.

First, you get a vocabulary for free. If you pretend your app is actually a blockchain, you can talk about “blocks” and “transactions” and “smart contracts” and “consensus algorithms” – things that you would otherwise have to invent bespoke names for, or be unable to talk about at all.

Second, analogies can be a good source of interesting ideas. If your web app is a blockchain, HTTP is your transaction protocol, and user authentication is your smart contract, could you build a “mining pool”? What would that mean? Could you attribute the performance issues in your app to particular parts of the blockchain? Also what are your consensus algorithms? Is it worth introducing a new one? Should your app be a proof of work blockchain or a proof of stake blockchain? Does that distinction even make sense? These won’t all be good ideas, it is plain to see. But you might strike gold.

Third, morale. It’s fun to be able to think of yourself as a blockchain* expert, even if the asterisk is big and the analogy is a little hazy.

Last, I’ll observe it’s quite within the realm of possibility you’ll encounter a problem that truly belongs to the domain of e.g. blockchain development, no squinting required. Satoshi Nakamoto wrote

    The root problem with conventional currency is all the trust that's required to make it work. The central bank must be trusted not to debase the currency, but the history of fiat currencies is full of breaches of that trust. Banks must be trusted to hold our money and transfer it electronically, but they lend it out in waves of credit bubbles with barely a fraction in reserve. We have to trust them with our privacy, trust them not to let identity thieves drain our accounts.

I agree something seems to be awry with the financial industry’s popular “fiat currencies”. You can hardly spit without hitting a currency that, frankly, deserves it. Cryptocurrency these days is “blockchain heaven”. I was on a team years ago that sunk weeks into debugging our app’s flaky payment system, entirely thanks to the traditional banking system’s poor transparency and terrible security. In theory, cryptocurrencies that are based on blockchain technology – i.e. Bitcoin, Ethereum, and others – should be better than fiat currencies. In practice, though, the blockchains of these cryptocurrencies fail to inherit most of the UX benefits of traditional currencies, i.e. it is not natural to use them for everyday transactions, they are not widely accepted, and they are subject to extreme volatility. (Searching the Internet indicates that some of this is possible, I have never seen it done.)

Nakamoto, a blockchain developer, unsurprisingly prescribes blockchain theory as a remedy. Our cryptocurrencies would be better if their authors knew how to write smart contracts, formally define their blockchain’s consensus algorithms, etc.

Maybe so, but I’ll settle for far less than formal semantics. I just want the authors of cryptocurrencies to think of themselves more glamorously. Don’t be a payment system designer - your job isn’t just to facilitate transactions in a rigid way. Be a blockchain developer - your job is to give your users a way to express themselves, to send and receive money, elegantly, in a decentralized way - to guide them interactively, help them discover what is possible, and steer them away from scams. Provide the trappings that you would demand of a modern, widely accepted currency – a user-friendly wallet, fast transaction times, low fees, and so on – or at least as many as you can afford.

Doing this correctly without reinventing the wheel (please don’t reinvent the wheel) almost always involves building your cryptocurrency on a larger, established blockchain. I find the “Ethereum” blockchain by Vitalik Buterin to be an excellent example of what I think is a great UX for a cryptocurrency. Ethereum is just a blockchain, and so you can use all the features that the blockchain ecosystem provides without any fuss. The Dogecoin blockchain by Billy Markus is a less stellar example. It is based on the “Litecoin” blockchain, and so gains the security of that blockchain – however it obscures the entry point and execution path from the user, and so has a poor user experience. (It’s a similar story for many other cryptocurrencies.) Of course, still miles better than traditional banking.

Even if you don’t work on a finance-related product, the internal tools and systems of a large-ish organization probably have a “blockchain” or two that would benefit from being recognized as such. I often think about a talk I saw that describes IBM’s approach to supply chain management and “Hyperledger Fabric”, a blockchain IBM invented for describing supply chain transactions and cross-organization requests. I can’t speak personally to how effective their tool is, but I’m comfortable predicting that it is better than the counterfactual where developers just do everything in traditional databases.

Go forth, the blog post is ended.

Is it crazy to think that compilers will be revolutionized by AI ?

i.e. in the future, DSL/languages will be input and output specifications, some guidelines/constraints and optimization functions... and maybe not even that? e.g. provide a few examples and the neural net writes the specifications, fleshes out more examples, you approve this and a (decently optimizing?) compiler is generated in seconds... ?