“Simplicity is a great virtue but it requires hard work to achieve it and education to appreciate it. And to make matters worse: complexity sells better.”
– Edsger W. Dijkstra
There are problem domains where there is simply a minimal complexity required implicitly. You can’t write an insanely simple program that will render vector fonts, simply because the problem inherently has to have a fixed amount of complexity. Anything more (accidental complexity) is of course bad, and “clean” code or whatever should try to minimize its amount, but it is harmful in my view if we keep believing that it could be made simpler. One really bad example is this list: http://harmful.cat-v.org/software/
Their primary challenge is making money, and that part probably is better as a web service, but it makes it worse from technical point of view. Just like with hypothetical FaaS - worse technically, but you can have adds.
The sentiment behind that list is okay but it does demonstrate a lot of ignorance The biggest joke is simple tables or file systems as a SQL alternative. If you do that for anything that needs more than just a map you will eventually end up with a badly implemented buggy slow relational database.
I feel like I am turning into a bot for posting the Out of the Tar Pit paper:
http://curtclifton.net/papers/MoseleyMarks06a.pdf
This changed my understanding of computer science and our product virtually overnight. We are using a hybrid model of Functional Relational Programming (see section 9 in the paper). This is in production right now and its clearly the right answer for managing complexity in our product.
If you think "eww i have to learn some FRP language" - No. You just need to tack SQLite onto whatever preferred language you use today and model all your business logic as SQL queries over properly-normalized tables. If you can achieve this with 6NF, you have an infinitely-extensible domain model. If you don't know where to start, 3NF is the safest place. Most humans tend to think in terms of 3NF when referring to the business entities.
Let's say you have billions of rows of event data you want to perform summary counts for by a few different key columns.
Doing this up front as the events are ingested is going to allow for much more efficient querying on an already grouped table than having to group on your billions of events in each SELECT query.
I'm not saying don't normalize. But normalizing creates its own problems too you may need to think about.
If it were up to me (and it almost never is, because I'm not a DB expert), I would ALWAYS normalize as much as possible when designing a database. I really can't even wrap my brain around why large (and old) databases ever get into this state. But I've seen it at two different employers and it's very painful (and costly) to deal with. My take was that these databases were probably originally set up by people who had no idea what they were doing, and ended up locking the company into a shitty implementation that kept the company crippled 15 years later. But that's just me.
> Let's say you have billions of rows of event data you with to perform summary counts for by a few different key columns.
Guess what? 99% of people on here don't have billions of rows of event data they need to regularly aggregate. Problem solved.
Such a terrible tradition in our industry in focusing on outlier cases, or on what FB/Google/... might need when making technical decisions.
All sorts of different patterns and architectures have to be brought together to make sense of it.
But the technique described above still has its place within such a system. For example - you boil billions of data points down to a handful (several thousand) facts, such as events that occurred or anomalies. Then those several thousand facts can be analysed standalone (as described above) without going back to the source. Etc.
having built a real time analytics solution a few years ago because we wanted one we controlled for our startup, it doesn't take much to get to billions of rows of event data in the modern web.
Analytics on large-ish data is definitely a common challenge. And sure, CRUD is even more common. But knowing what your options are for each challenge makes sense.
While I'm not necessarily arguing in favour of using Sqlite for this, it most certainly can do this up front during ingestion.
So yes, sometimes you do end up sacrificing normalisation, but often that means keeping a normalised form, and having code that can re-generate summary data from the normalised form.
Absolutely agree. That is why we use in-memory SQLite projections of business state that are scoped per-user-session so that row counts never exceed double digits in any given table.
For us, adding indexes to these special database instances would actually make things go slower.
> You just need to tack SQLite onto whatever preferred language you use today and model all your business logic as SQL queries over properly-normalized tables.
was to think "well that can't work for _all_ my business logic". But I'd like to see how this idea works in practice before I jump to that conclusion.
My first thought was the same as yours (no way that works for everything).
It seems like a neat idea. But I can't imagine that anyone has ever done this for anything non-trivial. I want some sort of compelling argument that this works at all before I'm willing to accept this as anything else than a pipe dream.
For us, the more complex the business gets, the more justified this path becomes. Our business fails to scale if we have to write custom code for every customer. Writing custom SQL per customer (against a common schema) is far more tenable.
Are you thinking of some specific logical determination that would be infeasible in SQL vs in code? Or, is this more about webscale arguments?
Finally, I'm not going to venture in (let alone actually live in it myself) without someone sitting me down and carefully explaining the structural integrity principles (and other logistics) that make the cardboard skyscraper possible.
Hearing business rules in sql works for some unknown entity doesn't really give any information about the viability for anything else. Maybe the initial modeling is really hard, but all your customers have near identical concerns, so you only pay that cost once. Maybe having multiple customers is actually the key because if one customer's needs violates some assumption that makes the whole thing work, you can tell them go to someplace else. Maybe your customers only have five business logic rules apiece that are all well spec'ed.
What's the domain? The industry? What problems are actually being solved? Can we have an example (even a contrived one)? Is there an open source example? How about a mathematical proof or even principle that argues for why this can be expected to always (or even nearly always) work? How about a youtube video with someone describing the approach?
EDIT: How about this. Show me a partial parser using this approach (grammar rules can be considered business logic, right). Just parse function declarations in C or something. I'll be able to project such an example onto what the full solution would look like to see if I believe it could actually work in general.
I'm disappointed to see there wasn't a response because I was pretty willing to entertain the notion of business-logic-as-normalized-tables. But I needed, like, any evidence.
The translation of that into requirements is by itself one of the largest sources of accidental complexity.
And even the value statement sometimes is wrong and leads to unnecessary complexity too.
Good point!
If a system is hard to interface with, the complexity is still accidental, it is just outside your own system and maybe out of your control.
Even if you’re a “middleware” company that connects multiple shitty systems together, you’re still adding zero value to anything but your own pocket. The complexity is still there.
We still have the right to call a spade a spade. In the past I had to say “no” to managers about interfacing with shitty systems. Sometimes you CAN make a difference.
The trope I see all the time is enterprise tooling in small projects. Small teams basically moon-lighting as devops 50% of their time and reducing their capacity to solve the actual domain problems significantly.
The domain complexity is something you can't remove, but complexity you've chosen to introduce through tooling should be hard thought about, and the less time you spend thinking about stuff outside the problem you're actually trying to solve the better.
YAGNI is so hard for people to really grok.
Get the revenue model right and suddenly every feature is cheap; get it wrong and every decision is infinitely expensive.
Yeah. This is both true and pointless.
Nobody codes things too complex on-purpose, at least not normal people. So it's not the difference in types of complexity, it's the difference in our ability to understand accidental and essential complexity.
I find that I do a really poor job at this, and I'm the first to stand on the soapbox and rail against systems being too complex. Just like programmers naturally introduce bugs into code without realizing it, programmers naturally introduce complexity into code without realizing it.
We may theoretically be able to talk about the differences in type, or how to manage each, but our real problem begins in our conceptual models inside our brains or various types of problems, and this happens a long time before any code is ever written.
In my personal practice I've come up with a few gimmicks that help me refactor my preconceptions. It is very difficult, however, for people to adopt practices that continue to inform them that they make lots of mistakes. Everybody wants to code as if they're conquering the boss-level at the end of a game. We sell coding, tooling, frameworks, and practices to other coders under the assumption that they're going to have a blast, not that they're going to continuously be reminded that they're screwing things up.
So yes, I agree, but without moving past that statement into something more tractionable, it's more of a truism than a starting point.
This resulted in them coding things too complex unintentionally, so while not on-purpose, it was absolutely a useful conversation to have. It's a good starting point.
I don't have a good single rule for a "second step," though. It's going to depend on the details of your project, by and large, though I think some principles to try to strive for include composition, encapsulation of details, and so on - nothing new or startling, but just adding the "is this essential or accidental" lens to the decision-making process.
However, if you bring up the topic, then take a look at some code or architecture? Then suddenly we're talking about all the ways we might have done it better/less-complex if things had been different.
Most every coder I know understands the topic, and most are even willing to go on at length about how important it is, including me! (grin). It's the actual application where things fall apart.
>> I don't have a good single rule for a "second step," though.
I do. It was bugging me so I spent a couple of years coming up with one. Seems to work great for me. YMMV. I do not believe it is as context-dependent as most in our industry seem to think. (It's very much problem dependent, though. It's just vastly more related to the business problem than the technical considerations. [Discussion goes here about tech-related problems not related to the solution foisted on the solutions team])
It's not so simple — I've seen people create over-architected monsters because they thought everyone coded like some consultant's magazine article/talk/book claimed, because they were trying to be taken seriously and didn't think they could push back, because they had talked their way into a job they weren't qualified for and tossing around buzzwords was a way to obscure the fact that they had no idea what they were doing, because the client insisted that all real projects must run in the way the consultants in expensive suits assured them everyone serious runs them (ignoring massive differences in scope and resources or long term success), or because the institutional incentives made it easier to say their project would handle everything anyone had ever thought might be useful into one application rather than working with the actual users to build multiple smaller, simpler tools.
One way to think about that is that it's neither accidental nor essential complexity but environmental. Fixing that can be quite hard but it's often work the political capital if you have it because the alternatives might not be survivable.
If simplicity is always the right solution, then why do so many programming languages have dedicated parsing libraries for .csv, which has to be the world’s most simplest data format ever?
The basic rule for CSV is: Don’t. Or at least use a library which emits RFC 4180-compatible results. If you need to parse some co-called “CSV” non-RFC-compatible monstrosity, do whatever you need to parse it, but don’t have any illusions of it being in any way “standard”.
This is acutely summarized by Larry Wall's "Waterbed Theory of Complexity" which is basically Einstein's "Make things simple as they have to be but not any simpler than that". i.e if complexity is like water -- not very compressible. If you compress that complexity (a part of the bed) -- it will manifest elsewhere in the bed as more awkward/expanded because it's all connected.
Many discussions on complexity do make this distinction.
To me it’s sort of implied that it’s unneeded complexity when we talk about it as something that should not be there and it’s making our life hard.
That's indeed a pretty bad example, regrettably of a website often shared in programming circles. The author simply listed his beliefs about stuff he doesn't like vs stuff he likes, and never bothered to justify them. In many places the list is obsolete, too.
Also, UTF-32 seems way simpler conceptually than UTF-8, so I don't understand how UTF-8 ended up "less harmful" and UTF-32 ended up "harmful".
Same sort of issue makes people use microservices not because of their true advantages but simply to enforce boundaries between features because so many people lack discipline to make a well-structured (distributed) monolith.
This is coupled with organizational dynamics where to get resources you have to prove value. Proving value is a lot harder than scaling/rewriting a software system, because if you need to scale a software system, you have rewards and money attached which are good business drivers. If you need to prove business value, no one cares (and you won't get resources) until you do.
It's a fine line to walk for sure.
Well said Dijkstra. Some of the my favorite algorithms are beautifully simple, mostly in the way of being naturally recursive. My compiler course in university involved a lot of recursion, but everything just flowed. Sometimes I'd mindlessly write some code and just assume I'd recurse down the AST, and everything just worked.
Obviously simplicity isn't always achievable in large software, but hopefully there are many simple pieces involved that come together in a clean and concise way.
"That's been one of my mantras - focus and simplicity. Simple can be harder than complex: you have to work hard to get your thinking clean to make it simple. But it's worth it in the end because once you get there, you can move mountains."
Also true with personal finance:
* https://canadiancouchpotato.com/2016/01/25/why-simple-is-sti...