Architecting large software projects [video]
youtube.com
youtube.com
1. Everything is exposed as an API with little no insight to inner workings (black box)
2. Everything should be broken down into modules at a level where one person works on that module
3. He shows a video of his video editor, saying it supports multiple inputs (like two keyboards or two mice), then says no platform supports it, but if they ever do, it will work
4. Don't implement 'good enough apis'
I hope that anybody who has ever worked on software understands that there are virtues to doing exactly the opposite of what is described in each of these points. Even if you can make an argument for any of these, you would have to qualify them with so many exceptions that you would effectively negate the entire argument.
I spent a lot of evenings early in my career watching similar videos, hoping to find some magic bullet in how people better than me do what I do. People make a living doing this on the conference circuit. Maybe it is a fools errand to try to distill something as complex and situationally dependent as software into a video, but I'm having a hard time finding any major insights in all of the videos I watched.
I’m still amazed it isn’t obvious: every piece of software should be a black box with a pin-hole for input and an even tinier pin-hole for output. The best code I’ve ever touched worked exactly like that and maintaining it was a pleasure, everything else was garbage. I push this rule in every project I touch.
[0] https://www.destroyallsoftware.com/screencasts/catalog/funct...
None of these projects reinvented the concept of FCIS.
These are just projects that had functional elements, or were outright functional languages. This is like saying jsx reinvented markup languages.
> We love reinventing the wheel.
His input layer is good because it helped him emulate local multiplayer with keyboard and mouse. It solved _his problem_.
The graphics layer is good because it is so much easier to work with than OpenGL _for him_.
The library wrappers are good for him because they solve _his desire_ to run portably while maintaining the smallest possible interface.
This stuff matters to Eskil because he’s: - just one person - has deep expertise in what he’s wrapping (win32, OpenGL) - wants to make an impressive program solo
I think his expertise, historical perspective, and culture make it feel as if this is the only way to do this very hard task, so he wants to share it. It helps him believe his way is right that many great, and reliable projects are done in C89.
I think the truth at this point is that folks still using old versions of C have, on average, more experience than everyone else. It’s not just the language that’s making them strong, either. It’s having evolved with the platforms.
Now the only question that leaves is whether it makes a huge difference to really stick with one language over the decades. I know we’ve all heard both sides of that quandary.
Only if static and dynamic analysers are part of the development cycle.
Most people don't work in that world.
You’re either making a productivity app where CRUD and UX are pretty well known patterns.
Or a scalable web system - also very well tried territory.
Or analytics and data processing - again well trod.
If you’re not a good pattern matcher you might think every UI framework, or your next API abstraction, is some next general theory of relativity.
But otherwise the major novelty in most software project pursuits is going to be the context, people, and industry your building it into not the tech
I haven't watched many of these videos but the two that stand out for me as being genuinely helpful is Jonathan Blow's talk about software quality [0-4] and Daniel Sockwell's "Imagining the Ideal Language for Writing Free Software" [5].
From Blow's talk, my takeaways are:
* Large organizations don't necessarily know how to make software and small teams can make quality, large scale projects
* Abstraction is its own form of optimization and should be used sparingly. Stripping away a level of abstraction is sometimes desirable.
* Design should be data centric. That is, focus on how to pass data between different domains of concern. Don't be shy about exposing data and don't hide data. This differs slightly from "API-centric" approaches as the contract is in the data, not in the function call interface.
* Keep inheritance to a maximum of 2-3 levels deep, with a preference of 1 or none. Not all language features should be used. Use language features sparingly, and this is one of them.
* Duplicate code (e.g. "cut-and-paste") for clarity. Dispensing with code-reuse isn't a sin and is sometimes desirable. DRY is fine but sometimes having a common function increases cognitive load to understand what code is doing and often portions of code that look very similar are different in substantial but subtle ways that make de-duplication overly confusing to handle all cases. When reviewing a function, having all code available locally, without having to page out to different areas of the code base to see what a common function is doing, often helps with readability and isn't as detrimental to reliability as people make it out to be.
From Sockwell's talk:
It's a weird premise (how can software be ideal for writing FOSS?) but the point is that many software architectural concerns stem from what economic environment they're being developed in. FOSS tends to be developed by a single or small team. The focus is on "master-craft" tools that allow a single contributor to be more effective at the cost of a high learning curve. Since corporate environments tend to have high turn over rate with software developers, the focus is on stable "commodity" tools and methodologies that can better take advantage of the fungible nature of their software developer labor.
The talk itself isn't focused on large software projects architecture directly, but, from my perspective, it gives some very actionable items for tackling larger projects on an individual or small teams basis:
* Focus on expressibility. That is, choose languages that allow mastery over being "safe/boring/simple". Don't worry about making code readable to the lowest common denominator, make it readable for someone who's at your skill level but is trying to find out what the code does (aka "future you"). Allow "footguns" if you trust you can avoid them. Don't worry about language popularity. Enjoy using a "dialects" if it helps empower you.
* Focus on composability. Composability is a broad term but focusing on data-centric design is one way to make things composable as the design can be made around the data being transported.
For me, the major insight is to understand the context of what software is being developed. A corporate environment will have different concerns, for both the employer and employee, than a small project with one or a few developers.
For example, a corporate environment might favor a language that encourages a type of code structure, testing suite, code review process and idiomatic way of solving problems. This makes labor more fungible but means that increased productivity of any individual developer will be met with resistance if it deviates from corporate convention. It also means employees will be motivated to recommend projects using technology that's perceived to be corporate friendly to make sure they're more marketable.
Small projects (FOSS, say) might focus on niche languages, dialects, idioms or other non-standard conventions that incorporate well with the skills of the primary developer. This means the individual developer is more productive but makes it harder for others to contribute.
[0] https://www.youtube.com/watch?v=oJ4GcZs7y6g
[1] https://www.youtube.com/watch?v=FvBySbGueww
[2] https://www.youtube.com/watch?v=URSOH4F3vbU
[3] https://www.youtube.com/watch?v=MlIZ2Q3Rvc8
[4] https://www.youtube.com/watch?v=xK6YZ3NFjuE
[5] https://ftp.belnet.be/mirror/FOSDEM/video/2021/D.perl/progra...
This is how I build large scale biz software (30+ years of experience). And I have never seen a case where it wasn't a good idea.
For example, the largest software system in the world (the internet) operate this way.
However I am always open to learn something new?
I’m eager to find someone who has managed to abstract expertise.
I’m 25 years into repeatedly building large software systems. I was thinking, look here maybe someone’s managed to codify what I do.
Nope.
To your point, everything is contextual. If you try to apply a suite of rules the systems (which extend far outside of the bounds of a software subsystem deployment) will mock you by burying your efforts leaving no intentional outcome.
Everything is systems thinking and every seat & context is different. You have to navigate mostly with realtime navigations based on long- and hard- earned muscle memory.
Agree in theory, in practice this is impossible. Even if you're an absolute domain expert in whatever you're doing, software and requirements will evolve and you will end up needing to implement something for which your current API is not suitable. Just ask S3:ListObjectsV2 or golangs' `encoding/json/v2` etc.
I push back hard on this one because a lot of developers will try to be "clever" and basically turn their api into
def my_api(parameters: dict[str, str]) -> dict[str, str]:
It's better to accept that your API will change at some point and have a versioning and deprecation strategy in place. With well-architected software this usually isn't hard.
I’d argue it’s your job to strike some kind of balance there. If you know you’re working with something stable, why settle for good-enough? Well, because the task of assessing what is stable requires years of mistakes, and good-enough varies drastically between languages. I think I see a point here for using primitive C; there’s hardly a type-system to waste time with, but you can cut yourself with macros. This is why I use Odin.
Don't ever implement good-enough-for-now APIs without a plan to come back and fix it and a deadline to fix it
Most of the time "good-enough-for-now" really is just "good-enough-forever".
By all means, shoot for absolute perfection on your own projects. I work very, VERY differently on my own code than I do on work code, and I get different types of satisfaction out of both (though of course massively prefer working solo).
And the new team might not even be allowed budget to fix those issues, and better not spend hours that do not fit tickets on the Excel budget plan.
Requirements _may_ change, but it’s much harder to have consumers move to a new API. Once it’s published, it’s a contract you don’t want to break.
That doesn’t mean you need to design an extremely flexible and future-proof API to the point it stops making sense — it’s a false dichotomy.
What you can do is take your time designing an API that still makes sense as long as possible, until your understanding of the domain has changed so much that it’s a different shape altogether.
Throwing your hands up and saying “it’s impossible” is easy, the art is in figuring out how much has to change until you have to say that.
Design is a game of push-pull between context and form, and you can add tolerances both ways.
But even that's too far really. Like it or not, "shiny fad" languages like Python & Javascript have been around forever, support for them isn't going away, this should be a non-concern at the architectural level. (Bigger language concerns: is it performant enough for my expected use? Does it have strong types to help with correctness? Can I hire programmers who know it? etc)
??? Python 2 went out of support five years ago.
I meant support in terms of there's an active community of people using the language and building things with it. It's not going to die as a language like Algol 68 or Pascal.
In addition to this, its existence and validity is still relied on by basically every other language via standard library, transient dependency, or calling convention. Heck, practically every C++ project probably depends on it.
The Linux Kernel, ffmpeg, SQLite, curl and many others chose C89 and often consider using C99, but most do not. Each of those projects also write at-length via newsletter or blog as to why they’re still not quite ready to update (and sometimes why they are.)
Both projects made this change in the last couple years. ffmpeg still requires c99 compatibility for their headers, and may indefinitely.
Should everyone? No, probably not.
I've worked on large software projects. The only ones I've met that weren't dreadful to work on had engineering and management working together with a shared, coherent vision.
One thing I often do, in terms of modularizing, especially for re-share or external use is sit down and think about the ergonomics of what I'm creating and how I would like to be able to use/consume said module... often writing documentation before writing the code.
His name has been popping up a lot more recently. I would be worried about the impact he might be having on young programmers, but honestly I doubt his worst advice would survive in industry anyway. I didn’t watch this particular video but his recent talk at Better Software Conference was illuminating, in a bad way.
The title is the only good thing about this talk, his whole ideology about building a WRAPPER around everything and maintaining hundreds of these wrappers is going to make you NEVER finish a single project. Also it just feels like this entire talk is an advert for C89, I do enjoy playing with the language sometimes especially when I want to scratch my software architecture itch (I use it kind of like pseudocode to develop a data model for my application), but it's simply not a silver bullet.
I've been watching his screencasts on and off for a couple of years and am always slightly aghast at how he can be really spot on about some very specific things and be wildly off the mark about most of the things regarding software design/architecture.
Anyway, people should finish projects, that's the most important part, my 50 weekend projects and 12 registered domains have been screaming this at me for years.
But I got to the "my wrapper around SDL supports multiple simultaneous mouse inputs, even though no operating system does" and noped out. YAGNI, even in a large project
I didn’t find much fault at all with what he’s saying about SDL. It’s just an example of the “layered design” he’s advocating for. You may have drawn your conclusion a little early; he immediately follows up with his API for graphics, which is actually a very practical example. He’s really just saying people should consider writing their own APIs even if it’s implemented with a library, because you can select exactly how complex the API needs to be for your app’s requirements. This makes the task of replacing a dependency much simpler.
He’s actually trying to prevent relying on things you don’t need. YAGNI.
“You Aren’t Gonna Need It”
Still, he gives a lot of good advice and fundamental engineering rules.
As someone still figuring out my footing, I often feel pretty torn: should I train myself to always push for the cleanest (or near-perfect) design I can manage (as books like "The Pragmatic Programmer" and talks like this suggest), or is that just setting myself up for frustration in industry where pragmatism and deadlines usually win?
What worries me is exactly what some people here are saying – that "temporary" APIs usually last forever. I’ve already seen that on my team. So part of me wants to internalize Eskil's idealism early, even if it means clashing with more pragmatic coworkers.
For those of you with decades of experience: if you were mentoring someone at my stage, would you tell them to lean toward ideals (aiming high, then compromising when needed), or to embrace trade-offs and "good enough" thinking from the start?
7:05> "It's faster to write five lines of code today, than to write one line today, and have to edit it in the future"
This is my experience writing C as well, which is why I now avoid it.
There are plenty of times where writing something in C seems like it would be simpler and more reliable than in other languages. But I found I couldn't edit it. Too many of my assumptions get baked into the code and I can't see them later.
His ideas about abstraction of the platform made a lot of sense in the past decades, but IMHO are not that applicable nowadays: It is well understood to separate domain logic from UI and storage, and a desktop application has a very different UI from a mobile app or even a web app.
My next critic point is, that Eskil assumes all domain knowledge is there from the beginning and will not change over time (his idea of having perfect APIs from the beginning).
No mention about error handling, reliability, response times, atomic operations etc.
To make it very clear: I believe Eskil gives his advice in good faith and his advice served him well in his own projects. In my opinion most of his advice is just not good advice in general.
(Besides having modules as black box building blocks if possible).
Large software projects could include AAA console videogame, or banking mobile app, or embedded vehicle entertainment system, or airport baggage handling system, or hundreds of other totally different domains.
The only good advice I can think of is given a large software project try to hire someone (or a whole team) that have built something pretty similar before.
His ideas feel, frankly, like catnip for developers. Each dev writes there own modules, so you get to do whatever you want without having to deal with other people's coding styles or philosophies (or better ideas), and no meetings, and you also spend the "right" amount of time building everything.
But it doesn't sound like it would work on "LARGE" (his capitals) software projects, at least none that I've worked on. In those, creating modules small enough to be built once by one dev would just push all discussion (and dreaded meetings) into the API layer, with double the meetings because you should get it right first time, actually double them again because the strategy if someone leaves and their modules aren't right is to just reimplement them so you have to be even more sure they are thoughtfully designed.
It actually sounds like a pitch for agentic coding: if you define your APIs small enough and correctly enough, you don't need to worry about implementation quality, so just let the api do it. I think it's a good way of thinking about it, and thinking about all the ways in which you might not be comfortable with that.
Anyway, what do I know? Maybe he's a super expert on this stuff. I would gentle suggest, if he redid the talk, to open with a) what he means by LARGE software projects, and b) an example where this strategy he is about to explain has been implemented.
Well, Agile, Scrum, and backlogs took care of that. /s
Anyway, "you sit down and think before you build" is indeed what you want and the word for that is "strategy".
In construction, architects and designers do different jobs. The software equivalents of those roles map reasonably well to the construction equivalents.