Solve Less General Problems | Hacker News Reader

Solve Less General Problems

hacksoflife.blogspot.com

193 pointskiyanwang82 comments

The interesting thing is that the programmer's tendency to abstract and future-proof their code mirrors the late 19th and early 20th centuries' high modernism movements, as James C. Scott analyzes in Seeing Like A State and Michel Foucault discusses in depth in Discipline and Punish and others.

"Modernism" is basically characterized by attempts to predict and control the future, to rationalize and systematize chaos. In the past this was done with things like top-down centrally planned economies and from-scratch redesigns of entire cities (e.g. Le Corbusier's Radiant City), even rearchitectures of time itself (the French Revolution decimal time). The same kinds of "grand plans" are repeated in today's ambitious software engineering abstractions, which almost never survive contact with reality.

Amazing parallel

https://www.deconstructconf.com/2017/brian-marick-patterns-f... is an interesting talk on this topic, which is where I learned about Seeing Like a State. I've just finished reading the book and found the parallels between software engineering and the high modernist statecraft and agroeconomic policy discussed in the book to be quite striking. I thought the middle chapters were a tad repetitive, but overall it was a fascinating read and I would highly recommend it.

Actually, I've thought, for a few years now, that software engineering principles, like loose coupling, have a natural application to just the sorts of problems _Seeing Like a State_ talks about.

And if these abstractions do survive contact with reality, it's often reality that has to change to adapt to the abstraction. You find yourself saying, "the computer can't do that", not for any fundamental reason, but because the abstraction wasn't designed with that in mind. The unfortunate users have to change the way they work to match the abstraction.

Seeing Like a State describes a similar situation, where people had to give themselves last names so they could be taxed reliably. I wonder how many people have had to enter their Facebook name differently for it to be considered a "real name"? Software that "sees" the world in a particular way is a lot like a state bureaucracy that "sees" the world in a particular way, especially when this software is given power.

Don't forget the planned city that actually got built -- Brasilia. Niemeyer was a disciple of Le Corbusier.

> The same kinds of "grand plans" are repeated in today's ambitious software engineering abstractions, which almost never survive contact with reality.

Yonks ago, i was reading an introduction to metaphysics at the same time as an introduction to Smalltalk, and it struck me that the metaphysicians' pursuit of ontology was quite similar to the Smalltalker's construction of their class hierarchy. The crucial difference, it seemed to me, was that the metaphysicians thought they were pursuing a truth that existed beyond them, whereas the Smalltalkers were aware that they were creating something from within them.

It's very likely that my understanding of one or both sides was wrong. But ever since then, i've always seen the process of abstraction in software as creating something useful rather than discovering something true. A consequence of that is being much more willing to throw away abstractions that aren't working, but also to accept that it's okay for an abstraction to be imperfect, if it's still useful.

I, probably arrogantly, speculate that metaphysicians would benefit from adopting the Smalltalkers' worldview.

I, perhaps incorrectly, think that the ontologists' delusion is endemic in the academic functional programming [1] community.

[1] Haskell

> But ever since then, i've always seen the process of abstraction in software as creating something useful rather than discovering something true.

Software, in my personal experience, is closest to the study of mathematics: there is an arbitrary part (the choice of axioms)—but, once that part is in place, you must obey those axioms and discover what facts are true within them.

If you don't obey your own chosen axioms, the system you will create will be incoherent. In math, this means it just fails to prove anything. In software, this means that it might still be useful, but it fails to obey the (stronger) Principle of Least Surprise.

The regular PoLS is just about familiarity, effectively.

The strong PoLS is more interesting. It goes something like: "you should be able to predict what the entire system will look like by learning any arbitrary subset of it."

The nice thing that obeying the strong PoLS gets you, is that anyone can come in, learn the guiding axioms of the system from exposure, and then, when they add new components to the system, those components will also fit naturally into the system, because they'll be relying on the same axioms.

> there is an arbitrary part (the choice of axioms)—but, once that part is in place, you must obey those axioms and discover what facts are true within them.

However, that's almost never the way math is originally developed. As a student one gets this impression, but that is usually on a topic that has been distilled and iterated over again and again, with people spending a lot of time on how to line out the "storyline" of a subfield.

More commonly, some special case is first encountered and then someone tries to isolate the most difficult core problem involved, stripping down the irrelevant distractions. The axioms don't come out of the blue. If a certain set of axioms don't pan out as expected (don't produce the desired behavior that you want to model with them, but for example "collapse" to a trivial and boring theory), then the axioms are tweaked. Indeed, most math was first developed without having (or feeling the need for) very precise axioms, including geometry, calculus and linear algebra.

I don't address this to you specifically, but I see that similar views of math in education make people believe it's some mechanistic rule-following and a very rigid activity. I think it would help if more of this "design" aspect of math was also shown.

Even when mathematicians feel they are discovering something, they rarely feel like discovering axioms, more like types of "complexity" or interesting behavior of abstract systems, where this complexity still has to be deliberately expressed as formal axioms and theory, but I'd say that's more like design or engineering or the exact word choice for a writer vs. the plot or the overall story.

> the French Revolution decimal time

The French were 99% successful with their decimals everywhere (everything except time) in 99% of all countries (everywhere except the US and some freak dictatorships).

> The same kinds of "grand plans" are repeated in today's ambitious software engineering abstractions, which almost never survive contact with reality.

Unless these "grand plans" just standardize and compromise the right things. Then they conquer the world (PC, USB, TCP, HTML, ...)

Seeing Like A State actually goes into a tremendous amount of detail about what a monumentous effort it was to introduce decimal measures in France. Progress was slow and took generations. It is only in hindsight that it was successful.

> The state could insist on the exclusive use of its units in the courts, in the state school system, and in such documents as property deeds, legal contracts, and tax codes. Outside these official spheres, the metric system made its way only very slowly. In spite of a decree for confiscating toise sticks in shops and replacing them with meter sticks, the populace continued to use the older system, often marking their meter sticks with the old measures. Even as late as 1828 the new measures were more a part of le pays legal than of le pays reel. As Chateaubriand remarked, "Whenever you meet a fellow who, instead of talking arpents, toises, and pieds, refers to hectares, meters, and centimeters, rest assured, the man is a prefect."

I would imagine likewise for the other standards that you mentioned. And in a lot of ways they succeeded exactly because they weren't (thinking specifically about how HTTP/HTML succeeded because it was less comprehensive than, say, Xanadu).

I don't disagree.

But calling the "decimal time" a failure is just unfair, given that they rest of the program was a huge success getting rid of what we would call today "legacy" (of course in hindsight. When else?)

Was it a success though?

Given how long it took to transition even in France, it sounds pretty expensive. Given that the rest of Europe wouldn't have bothered with it if the French hadn't gone on a crazy conquering spree, it sounds pretty expensive.

Given that the imperial system actual works pretty fine for those who use it, and given that we are stuck with lots of discordant units of measure anyway, and new ones are being invented every day, it doesn't look like the gain was with the cost.

Other examples of us trying to rationalize and systematize chaos: "the heliocentric model of the solar system", "physics", "economics", and "science."

I'm glad we try to understand the world and bend it to our will.

Feels different when "the world" is other people with their own lives, hopes, and dreams. In that case be ding it to your will called authoritarianism.

From a lifetime of coding I indeed came to a similar conclusion: "Micro-abstractions" are more useful in the longer run than large-scale abstractions. Nobody can predict the future accurately: I've seen many Grand Abstractions byte the dust face first. BUT, with micro-abstractions you can often re-use a decent portion of them when the future doesn't go as planned. Unlike large abstractions, you don't have to marry micro-abstractions: you just date them. If any one doesn't work out, you can ignore and replace it without much penalty: no messy divorce. I've built up a library of "handy" functions and mini-classes that I can borrow as needed per project. I've translated the commonly re-used ones into different programming languages.

Further, one can experiment with different micro-abstractions without investing too much time. They are micro-experiments. If one fails to provide re-use or flexibility, so be it: just stop using it. It's like picking stocks: some stocks will wilt, but if you buy enough, you'll roughly approximate the general market at about 10% annual growth.

I do find dynamic languages make such mixing and matching easier, though, because they are less hard-wired to a specific type or iteration interface: less converting needed to re-glue. I've yet to get as much re-use in type-heavy languages, but not ruling it out. I just haven't cracked that nut yet. Maybe smarter people can.

Beautifully put. It resonated with me about not over abstracting too early.

Abstractions usually come earlier in the design cycle than optimization for me and the wrong one makes that later stage harder.

> I do find dynamic languages make such mixing and matching easier, though, because they are less hard-wired to a specific type or iteration interface: less converting needed to re-glue. I've yet to get as much re-use in type-heavy languages, but not ruling it out.

I go back and forth. I mostly develop in a mix of Python and Java. Python may give me a lot of room to explore ideas, but I find refactoring a fragile nightmare. Java's type system is fairly simple and lets me quickly pull up all use sites and trivaly catches misspellings.

It lets me back out of and refector the wrong abstraction much more simply.

I don't know why there are not more code analysis and refactoring engines for dynamic languages. Sure, it takes more guessing on part of the tool, but I see no reason why they can't identify and help with a large portion of such tasks. It shouldn't be an all-or-nothing thing. Sure, the more "meta" your programming, the harder it is to automate analysis, but only a portion of an application typically requires high meta.

Because the refactorings are generally impossible. At best the automated tool could tell you everywhere it thinks a change might be necessary, but has to ask you want you want to do there. And it would likely greatly over or underestimate the number of changes needed. It's rather akin to saying "why can't the system implement my functions for based on their signatures?"

I’ve been saying and doing the same since forever. What are some of your commonly used functions?

I'll pick a couple that are easy to describe. First a simple string function: "softEquals(a,b)" that compares ignoring case and leading and trailing white-space, similar to "return trim(upper(a))==trim(upper(b)). The second is to "draw" an HTML button and/or hyperlink because often the user or designer change their mind about whether it's a link, button, or icon: link(title, url, httpType=[hyperlink,submit,get,reset], dispalyType=[hyperlink,button,icon],target="",classes="",otherAttribs="",buttonImg=""); Those with "=" are optional named parameters or optional object values, depending on the language of implementation. I'll usually tune it to app or shop conventions, which can shorten the range of variations or likely defaults, and thus simplify the interface. But the implementation is mostly the same: just the interface needs trimming/tuning. Or just leave the fuller one intact for edge cases, but create short-cut versions better fitting local conventions.

Interesting.

Could we say the much-debated ORM is (usually) a large-scale abstraction?

Yes, and ORM's are often a painful failure point when they don't work as expected unless you really master their guts. In my opinion it's best to code your ORM queries such that if the ORM is not doing something right, you leave your options open such as using direct SQL, stored procedures, or views. In other words, don't overly rely on ORM's both for the total application stack and individual queries. Have and test a Plan B.

I've built up a series of "SQL helper" functions to do a lot of the SQL syntactic and Prepared Statement grunt-work to avoid ORM's. They are NOT intended to entirely wrap or hide the RDBMS, but rather help with the app's interaction with the RDBMS. Helpers rather than wrappers. If the helpers don't work in a particular case, you can ignore them with minimal code rework. If you depend heavily on an ORM, such is often not the case.

I think that's fair.

On the more general point, I think that the larger scale the abstraction is, the larger the number of actually encountered use cases needs to be. You shouldn't write an ORM because you feel like existing ones don't serve your specific needs. You should write SQL, and then replace it with an ORM once it's clear you will benefit from the extra abstraction.

There seems to be a fairly consistent trade-off between the scale of the abstraction, the degree of over-engineering, and the leakiness of the abstraction. So if you're going for a large scale abstraction, then I'd want lots of proof that it's not a leaky one.

I'm kind of bothered by the choice of direct SQL versus ORM. I'd prefer something in between that assists with SQL but doesn't try to completely hide you from the RDBMS: helper API's. I know many developers "hate" the RDBMS (perhaps because it's unfamiliar), but shifting the burden from potentially tricky RDBMS's to potentially tricky ORM's is not problem solving, just problem shifting. But maybe the ORM debate is another topic in itself.

I think I phrased my opinion badly. I'm generally in favor of using ORM's because they simplify 99% of the stuff you need to do in the database. There will always be stuff that doesn't fit your choice of abstraction well... that's just a fact of life.

I'm mostly wailing against writing your own ORM (and also other forms of "not invented here" syndrome), which is something I've seen done way, way too many times, because you have to reinvent a ton of things you probably don't have much experience with, will probably get wrong in ways you don't anticipate, and will miss scaling requirements you don't know about yet. Because of this, I think there should be a heavy burden of experience on writing "core" libraries.

ORM's "simplify" things until something goes wrong, then they can be a pain to troubleshoot. A "light" set of utilities can also simplify a good portion of the database interaction. However, they don't give you a false sense of power such that you tie yourself into a knot that bites you 3 years down the road. But, it depends on the shop: If your shop always has a couple of good ORM experts around, then they can deal with and/or avoid such snafus as they happen. But, it's hard to "casually" use ORM's right.

I suspect our experiences differ, or maybe it's our philosophies. I'm of the opinion that if your self-tied knot bites you 3 years down the road, that's better than the status quo. :P

That's situational, of course. My current projects are tech giant things, so some choices actually do have a >3 year window, but then there's a tech giant sized stack of use cases to design against.

When my projects were startup problems though, I'd be happy if that component still worked well after 3 years of growing and pivoting. In that kind of environment, getting more for free is by itself valuable for your ability to pivot quickly.

The environment does indeed matter. With startups, the short term is critical: if you don't grow fast enough, there will be NO "3 years from now" to fix. A big bank will care about glitches 3 years from now.

I'm not sure what you mean in the middle paragraph, per "stack of use cases to design against". You mean getting it done "now" overrides problems 3 years from now? Your org making that trade-off is fine as long as it's clear to everyone and perhaps documented.

We start to see ORMs like Diesel (https://diesel.rs) or GRDB (http://github.com/groue/GRDB.swift) that radically simplify their domain by removing traditional ORM features like implicit uniquing, lazy-loading, or auto-updates. This of course gives more work to do on the client side. But this work is positive and often simpler than expected. With traditional ORMs, there is still some work to do, which often looks like fights: against misunderstandings, against leaky abstractions, against configuration levels that don't match the needs of application, and generally speaking against the framework.

ORMs are not done yet :-)

This resonates with me. For my long-running Python projects I usually have a file called small.py which has these, as you put it, micro-abstractions. Over time some stuff from the file gets into the larger projects ... and some stuff doesn't. But I like the comfort of "cognitive offloading" this file affords me: I don't have to keep worrying about increasing the scope of my abstractions.

> I do find dynamic languages make such mixing and matching easier

Yes! I think all of the people I've met who advocate the article's position were using static languages that make it harder to re-use what they've written. This article sounds like it's about a C++ project, which doesn't surprise me.

Where are all the articles about this happening to dynamic data-driven projects? In contrast, people seem to go to incredible lengths to keep old Lisp code running. A lot of people (even non-Lispers!) would apparently rather write a new Lisp interpreter than port their Lisp code to another language -- because the Lisp code is largely a declarative description of the problem and solution.

Data itself is already a Grand Abstraction that has survived the test of time. I don't need a library of mini-classes.

Could be selection bias. People who use lisp today are people who refused to move on over the past decades when almost everyone moved on.

The boundary between data and behavior is fuzzy, and one can often be remapped into the other. A common example is a "field", such as a screen field. We can represent it as a structure similar to: field(dbName="socSecNo", screenName="Social Security No.", type="string", mask="999-99-9999", required=true...); Whether that's a method call or "data" doesn't make much difference. Ideally we want to define it once, and that's our abstraction.

Lisp does make "swapping" between data and behavior easier, but many say Lisp code is just hard to read in practice. I've been in Yuuuge debates about Lisp, but my summary conclusion is that many just find existing Lisp code hard to read. Perhaps its too much abstraction. Short of eugenics, we are stuck with that problem. Lisp has had 50+ years to try mainstreaming; time to give up.

Definitely agree. In ~25 years of professional coding, I've witnessed more money wasted by over-engineering, over-generalizing, and solving unnecessarily hard problems, than I have of not being general enough.

I've seen that often enough that I'm personally now starting to make the mistake of not being general enough regularly. :) But it seems like a good thing if I'm 50/50 between not general enough and too general.

Running a startup was good practice, you just can't afford to work on problems you don't really have. If multiple customers aren't screaming for it right now, it can probably wait.

Ironically, I think I learned this lesson more thoroughly with writing than with computer science. Using abstract language, using categorical and abstract words rather than specific examples, makes for super boring reading. Communicating a pattern is even sometimes even more accurate when you list three specific examples than when you choose the most accurate abstract word; the reader will see the pattern.

Just googling for examples, this one seems decent. Lots of writing advice online leans in favor of being specific. https://www.unl.edu/gradstudies/current/news/use-definite-sp...

"In proportion as the manners, customs, and amusements of a nation are cruel and barbarous, the regulations of its penal code will be severe."

vs "In proportion as men delight in battles, bullfights, and combats of gladiators, will they punish by hanging, burning, and the rack."

I see the author's point, and it is valid, but I'd like to give a counter example.

I was asked if I could make a program that could 'put a photo on top of another one'-- the idea being they wanted a program where you could load a template image, maybe one that looks like a birthday card, and it would have a spot where someone's picture could be put on top of the birthday card image. Simple enough, but I decided to build a generalized templating system with a built in editor so new templates could be created. This way it could make birthday cards for one person, or make a "Good Job team!" cards for multiple people. It was completely agnostic to what kind of templates you could make with it. It supported layers, conditions, database integration.

Sure enough, over the course of months, I was asked to make the program do more things than it was originally asked to do, and I was able, with few exceptions, to accommodate these requests without needing to modify the source code. Even when I did need to modify source code, it was to extend functionality, not change the basic template abstraction model. It found its way into other uses as well (digital signage, ads, etc).

If I had only solved the least general problem first time, I would have been back at the drawing board rewriting most of the app every time something new was requested.

Maybe this is a rare counter example, but sometimes pursuing the general solution pays off.

I actually agree with both of you, _as long as it's up to the implementer_. I see requirements specifications that deliberately try to remain vague in the hopes that the developer will produce a "general" abstraction that can be reused - and always end up shooting themselves in the foot by never saying exactly what they want done. Or the "disruptive scrappy startup founders" who want you to build "legos".

I think another way of putting the title is "Right-size your abstractions." My rule of thumb is to solve the problem at hand while accommodating (but not implementing) known tasks within the next 3-6 months. If you're not designing your abstractions to deal with changes you have very good reason to believe will be coming soon, you'll be stuck rewriting all your code over and over again. At the same time, it's very difficult to anticipate requirements more than 6 months out unless you're very experienced with the problem domain and application you're working on.

But you did not go and try to implement PhotoShop or a node-editor graphics processing tool, which would have been even more general - and likely too far.

I've seen the other side of the coin, where the desire to solve a concrete problem forces a solution to be unnecessarily brittle and actually harder to implement than a more generic solution.

I'll give an example: "we're never gonna have more than three devices, so this array should have a hardcoded size of 3." It sounds pretty extreme, but believe me, some variation of this theme comes up more often than I would like, especially from people who are not software engineers. It seems like some people think that every abstraction (and potential genericity) is expensive, so they tend to put constraints in places where none need to exist. Very soon you'd need to make a small modification to your code, or extend it slightly, and those unnecessary constraints are going to make things difficult.

Personally, I'd like to approach a problem from the bottom-up. Start with solving a concrete case, and see if any patterns emerge in the process. And if they do, then I generalize those patterns. Frequent refactoring is very helpful during this process. Sure, mistakes and over-generalizations can still occur. But they are usually not very expensive if noticed quickly.

In this specific case I would say that the abstraction rises as soon as the number of devices gets >= 2.

Isn't the "0, 1, n" rule a thing?

I think this conflates requirements and implementation choices. "we're never gonna have more than three devices" is a statement about requirements, and implies the absence of a requirement to support more than 3 devices. It does not imply a requirement to fail to support more than 3 devices though, and leaves an implementation choice how many devices to support, as long as it's at least 3.

On the implementation side, the "0, 1, n" rules already mentioned by muxator dictates that this specific aspect of the implementation should not treat 3 different from 5, and implement support for n devices. (By "This specific aspect" I mean that other factors will surely prevent supporting n = 1 trillion, but we don't have to care about that since it doesn't contradict the requirements).

this is why I think one-way specifications are a massive anti-pattern in the software industry. Often times the developer, who has the front-line view of the existing abstractions, could turn a 4 week project into a 1 week one just by adjusting the spec, while still solving the actual problem.

This is one if those pesky differences between “computer science” as it’s taught in college, and actual software engineering.

I believe that bigger parts of our education should revolve around reading and analyzing professionally written code as opposed to solving completely artificial problems.

Agreed. What really needs to happen is the splitting of software development and computer science into two related but distinct education tracks.

One could say that bootcamps were the first to pave the software development track.

Hardly... applied programming education has been alive and well in community colleges (and some higher ed. institutions) for decades. Not every program, everywhere, is theory-heavy!

Bootcamps tend to be at the other end of the spectrum. In my experience they often produce developers that have a solid understanding of specific technologies, but have a harder time solving more general problems and/or moving up and down the tech stack.

I believe that what we need is something in the middle - a solid CS education combined not with in-depth dives into specific technologies, but with a wide range of samples of solid engineering in various real world projects.

Isn’t this the difference between computer science and software engineering courses?

At least in the US university I went to there wasn't a difference.

I graduated ~6 years ago and there was no opportunity for specifically software engineering classes; there were only computer science classes.

I highly recommend following a data oriented approach to software organization. OO loves to abstract data, but why? Data is the main character of the story. I also hate the take on Knuth's "premature optimization". Don't sweat the small stuff, but do architect for performance, it is a feature. If you focus on performance and not hiding your data, the code will fall into the right level of abstraction.

Solving general problems happens when you start thinking about data you don't have, but might, and throw performance away for premature organization.

I'd formulate the "don't overgeneralize too early" as "make concious decisions about where you solve specific and general problems". It's fine to solve a specific problem knowingly, and while considering the cost of solving a general problem.

Saying "let's cross that bridge when we come to it" is the idea (rather than "whoa we never saw that bridge we should/shouldn't have crossed")

There is also a human perspective to this. Solving general problems is FUN, (at least I tend to think so). The generalization is the fun part of the problem solving. Some people think seeing the working product is cool, no matter how elegant or general. I don't mind if everything is half-finished forever so long as the half-finished product is elegant and general (exaggerating but you get my point about the different personalities). This is also why you need different perspectives (and probably people) on a team.

Build one. Build another. Now you have built two. Hopefully you learnt something and now you know where things are headed. Now you can write an abstraction. Now you can rebuild the first two using the abstraction along with the next. If you’ve done the abstraction right all three now share and use a commonality that helps clear up and simplify.

https://en.wikipedia.org/wiki/Rule_of_three_(computer_progra... lines up better with my experience: postpone writing the abstraction until you are actually implementing the third thing. With just two cases, there will generally be a bunch of incidental commonalities that are better left out of the abstraction, and it's frequently not obvious which ones they are.

Nice to see it codified. Sometimes you don’t realise you need or decide you want to abstract until you’ve hit 5 or more.

The true meaning of a General solution does not mean providing support for every kind of scenario your data may encounter, the true meaning from a functional perspective is to build composable structures/mechanisms to produce the correct shape of data the problem requires. So for a near sighted programmer YAGNI would work, but for a software or machine to be really durable, it needs to be composable over a period of time. An example of this is emacs, it has kept with the passage of time because of its composable nature. Vim on the other hand is undergoing a massive rewrite project known as Neovim. This is the difference between a true general solution and a specific YAGNI based solution. The only differentiator between the two is Durability, and this durability comes at no extra cost, just the wise decision of writing software in a much more composable language solves half the durability problems.

This is my favorite (and least useful) quote from the article:

> I made the right decision, after trying all of the other ones first - very American.

It's not my favorite because it's snarky, or because it makes a generalization I think is accurate. It's my favorite because it has an element of the person behind the technical article.

Too often, people writing about technical subjects get lost in the topic, and forget that the act of writing for an audience is a human conversation.

Richard Hamming or John Tukey said "It is better to have an approximate solution to the correct problem than the perfect solution to a slightly wrong problem."

It might apply to this case here.

Jonathan Blow discusses harmful abstractions in several youtube videos, while discussing the Jai Programming language. That language isn't out yet, but an overview is available at: https://github.com/BSVino/JaiPrimer/blob/master/JaiPrimer.md

Hmm... it's not quite addressed here (they address generality as the size of the scope of the solution), but I have a slightly different take:

I love generalizing things. If I need to remove a couple of prefixes from a list of strings, I don't hard code those strings in a function. I make a general string prefix removal and pass those prefixes in. I find that I write more correct and stable code by switching from special cased code to generalized code.

One can take this to extremes of course. I think the article is against generalizing solutions that shoehorn things together that normally don't go together. E.g. hal for handling disparate hardware abstractions. Its a forced abstraction to gain certain things.

I guess it's abstract vs generalize. Generalize is good, if it doesn't introduce abstractions... that's the thought process I just had anyway.

Suppose I already had two function functions like:

   def strip_prefix(prefix, str):
     ...


   def name_from_fooid(fooid):
      return strip_prefix("foo", id)

Now is it better to call `name_from_fooid` or to call `strip_prefix` directly? It depends on the high-level semantics we are trying to express. But assuming we really are extracting a Name from a FooId, then the former is probably better.

But then if you want a name_from_fooid function, it's just an implementation detail that you happen to implement it on top of a strip_prefix utility.

Great abstractions exist, that doesn't mean every abstraction is great. Most are bad.

This is a fantastic description. As smart, analytic thinkers it is so easy to generalise. My experience is that we should generalise when we are creating conceptual designs, but be very lean and specific in the implementation.

A good abstraction leverages the shared knowledge of other previously unconnected pieces of information. It has more to do honestly with psychology than computer science; I don’t need to learn anything about tree or graph data structures when I can use HTML in the same way as I am used to organizing information in bulleteted lists.

I'm going to risk being contrarian and say that I don't see the HAL as the problem. The problem I see is "the product was canceled, I was moved to a different group, and the HAL was never used again". When the product is cancelled, it doesn't matter what you built, or how good or bad its architecture was. It's dead.

Our industry has an organizational structure for shipping software today that makes every line of code have zero value until it's in a consumer product that ships, and maybe even then it'll be cancelled next month and its value drops to zero anyway. We've long known that most software projects fail. What's wrong with this picture? Why do we continue to work in a system where the most likely result of any day's contribution is literally a useless waste of time?

It seems as though the problem was short-sightedness, and the proposed cure is to be even more short-sighted. (You're looking down as you walk, and you occasionally glance up. While looking up once, you tripped over a stone. The solution is to focus even harder at the 6 inches in front of your shoes -- then you'll never run into trouble!) Is it any wonder that most software projects fail?

We joke about "resume-driven development" but economically that's the only certain gain from any new software project. I'd be foolish to optimize for the lottery ticket of "successful product", instead of the solid long-term investment of "increased personal knowledge".

The Gossamer Albatross succeeded where others had failed when they realized that the actual problem was fixing the process first [1]. They won big by solving the more general problem. What if we fixed the software development process so that everything we built could be useful?

[1]: http://www.azarask.in/blog/post/the-wrong-problem/

Sure, some code is just bad, and the trash can is a useful tool, but a DV HAL sounds like a useful tool, too, and I don't think anyone is better off that it got built and then discarded. Could there have been a secondary market for commercial libraries whose products were cancelled? Could it have been open-sourced? Could it have been built as part of an alliance, so other teams could have helped drive the design, and reap the benefit even if one team dropped out? I don't have The Answer but it seems awfully unlikely to me that optimizing one's architecture for the next day or week is an optimal process on any larger timescale. That's basically admitting that we're still at the hunter-gatherer stage of software development. You can't plan for next year's harvest, if you haven't invented agriculture, and don't know where you're going to be in 6 months.

Bonus: "At the time templates tended to crash the compiler, so going fully templated was really expensive." Yes, we need to reduce our scope, because this other program we're using reduced theirs! The limit of this function is a catastrophe. If I were to get a batch of bad steel from my supplier, I wouldn't try to compensate by using twice as many bolts.

playing devil's advocate: if the author's manager had communicated the tech improvements correctcly ("we can now support any hardware"), maybe others teams knowing about that would have moved the company in a different direction instead of "cancelling everything after firewire support". maybe they even got to that decision because one new input source took so much time in the first place.

Can someone better explain the leaky abstraction concept? I take it to mean an abstraction that really isn’t.

It's basically an abstraction that fails to properly abstract away the (complex) reality behind it, which leads to things being more, instead of less, difficult to grasp.

https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a...

A leaky abstraction colors your approach to solving problems in ways above and beyond what is minimally needed to solve the problem at hand.

For instance, when the author created an HAL instead of a specific problem specific driver, the HAL meant that he had to solve 2 problems instead of 1 The end product had to work, and had to be compatible with the HAL as implemented. The abstraction leaked.

Just making the driver would have yielded a solution to the problem that would"t have required as much refactoring of old code. It's a way of saying he made more work for himself than was necessary.

A very YMMV observation.

All abstractions leak just in different degrees. Perf/memory is the most common but there are others as well.

Your ivory tower parser/data format/framework is great until you need to fit it in 1/4 the memory or MIPS. That's when you start pulling out the abstractions to get back to where the product actually needs to be.

One of the biggest industrial-scale examples of abstraction failure is OOP GUI's. They were an abstract interface, meaning in theory they could adapt to ANY future UI interface or standard. But the "the Web" came and took a shot-gun to it by not preserving program state, and by mostly forcing one to let the client determine the final layout (positioning). The existing OOP GUI engines ASSUMED the engine controlled both state and final positioning, because it had usually been that way in the past. But Web-land said, "Sorry, Dave, I cannot do that.", and so the engines were largely abandoned in back dumpsters. (Web UI can turn one prematurely grey, but that's another story.)

A leaky abstraction is one that mostly works, but details and assumptions leak through.

For example, many languages attempt to provide a cross-platform API for filesystems; they give you an algebra that lets you construct and manipulate paths.

So if you have a Path, the `/` abstract operator might join path parts, and "basename" and "dirname" are functions that extract parts of the pathname.

Typically, the API designer makes some intentional (and some unintentional) design decisions to keep the API comprehensible but at the expense of being inconsistent with the underlying reality being abstracted.

As an example, some filesystems are case-insensitive, others aren't, so to know if two paths refer to the same file, e.g. if you want to use paths as keys, you need to determine what volume each is on. This gets complicated in Unix due to symlinks (which can be any component of a path) and mounts that can be anywhere in the tree. You could resolve a path to an inode, but then other filesystems might not provide an inode.

By this logic though, aren't all abstractions leaky in the end?

edit: not arguing, just curious. I haven't thought much about abstraction but it seems like it could be a truism that every conceivable abstraction could succumb to this problem in some way.

You can only answer that with respect to a specific usage of the abstraction. An abstraction that should abstract from different implementations A, B, C is leaky with respect to use case X, if the implementation for X still needs to know about differences between A/B/C.

A good abstraction is not leaky for most use cases. A bad abstraction is leaky for many use cases. The main issue is that you can't foresee the use cases, so premature abstractions usually turn out to be leaky.

>By this logic though, aren't all abstractions leaky in the end?

Almost all. the question is just one of degree. The fact that they are almost all leaky is why you should be hesitant to abstract away in the first place. The effort you think you are saving may be taken up later when the leaky details bite you. "I'll use this game engine instead of raw vulkan" -> "oh no I can't render all my widgets fast enough on mobile" or "I'll use this easy UI library" -> "oh no I can't actually do a list of text boxes that is scrollable with this"

Whether perfect abstractions "could" be made is mostly moot because in practice they are not. The reasons include but are not limited to:

1) Humans make mistakes and are often ambiguous.

2) Perfection is often not economical. 98% good enough may require 1/10 the code of 99.9% perfect and customers don't want to pay for 10x the coding. See: https://en.wikipedia.org/wiki/Worse_is_better

3) New standards or technology comes along that challenge the original assumptions made. (I gave an example of OOP GUI libraries versus Web elsewhere.)

4) Performance: heavy-handed abstractions are often too slow because they have more conditionals and converters/fixers to adjust the translations or interfaces.

While I agree with the author with respect to his main point, I think the term leaky abstraction side-tracks the discussion. It is simply about wrong abstractions.

A good example are SQL queries. You can have two semantically equivalent queries i.e, two queries that will always return the same results.

However, runtime may be dramatically different. One may take a second, and the other one days.

That's because SQL is an imperfect abstraction. It isolates you from the underlying details on how to compute the results of a query, but only to some extent. On many occasions you need to actually understand how the declarative query is translated into an execution plan.

So sorting is a leaky abstraction? Because merge sort and bubble sort can both have the same interface, isolating you from underlying details...

Yes, exactly. The slowness of the bubble sort "leaks" the fact that it's a bad/naive implementation of sorting.

Abstraction is often defined similar to "hiding irrelevant details" so that one can focus on the forest instead of the trees. However, sometimes tree-level issues affect the forest view also. An unexpectedly slow sort operation may be such a case. Normally you cay "ORDER BY x" in SQL and not worry about HOW the RDBMS sorts on "x": "How" is a "grunt-level detail" to you. But if the sort is slow or "wrong" (such as a collating sequence not expected by the customer), THEN you have to dig into the details, stepping out of the abstraction forest to visit and study individual trees.

that's a whole lot of blathering to say pre-optimization is the root of all evil... @example: the Intel meltdown &Spectre bug.

Reply on news.ycombinator.com