Hacker News Reader

>I have a strong suspicion that people smart enough to do advanced meta programming on their own are not smart enough to collaborate with equally skilled coworkers on code that uses all those abstractions to the max. Even collaborating with your own former self is more difficult that writing new code.

I'm an unabashed Clojure fanboy and will talk the language up to anyone who will listen, but one thing that I've been thinking about lately is that good judgment on abstraction use is necessary for using such powerful tools. I don't think the place to tackle it is by being "smart enough to understand," but rather by being "disciplined enough to not abstract sometimes." The exciting thing is that discipline doesn't require you to be super smart, just to think it's worth caring about.

Also half-baked, but I think it's something.

The easy solution to this is to create a culture of using powerful tools, but requiring REALLY good reasons to go far to their edges in terms of metaprogramming.

It's not that much harder to enforce than it is to enforce conventions in less powerful languages, and the convenience of working in a well-designed language is so nice.

Java is abstraction over JVM which is abstraction over machine code. But I don't need to know anything about JVM bytecode or machine code to write typical Java code. Good abstraction does not require decompression. You're not going down to assembly to understand some Java code. You think on Java code level. That is good abstraction. If you need to decompress what you're reading, then it's a leaky abstraction. It's okay to use leaky abstractions to reduce repetitions, for example. But ideally your abstractions should not require one to learn its implementations in typical cases. For example if your function name is good enough, reader could trust you to do the proper thing and not read its implementation.

You just explained cornerstone of good design

Abstractions are a spectrum. There's always a degree of benefit to be had from peeking under the hood. I'd argue that it's rare that you find an abstraction that can be used well without having to understand, or "decompress", some of it.

Consider JVM and Java: that you can write Java seemingly without any knowledge about JVM is an illusion - you already learned a lot about how things like JVM work under the hood when first learning programming[0]. Learning more about JVM itself lets you write more efficient code and debug better.

Consider functions: it's near-impossible to contain all the information necessary to use a non-trivial function in its header[1]. That's why good code contains comments and other forms of documentation describing the abstraction in more details. Even then, it's not always enough - sometimes it's really much easier to understand what you need by reading the source[2].

Consider any appliance - be it a car, or a radio, or a dishwasher. You can use it to it interface, to some extent at least. But knowing what's going on under the literal hood really does help with use, and especially helps when something goes wrong. If you don't know what's hidden under the abstraction layer, any failure will likely be incomprehensible to you and leave you helpless.

--

[0] - In theory, it might be possible to learn some programming without learning anything about hardware or hardware-emulating abstractions. In reality, I've never seen it or heard of it, and I suspect that the simplest mental model of code execution is isomorphic to somewhat simplified computer or virtual machine.

[1] - Function name + name of its arguments + types of its arguments and return value, if available.

[2] - Then again, implementation code may not capture the entire abstraction either. That's why good code often features comments inside the implementation, explaining the rationale behind some of the less obvious code parts.

>Good abstraction does not require decompression

I think there's a misunderstanding. What I'm calling decompression has nothing to do with digging into the implementation of a particular abstraction. What I mean is merely tracking down all the indirections in the public interface and breaking them down to their concrete meaning in a specific case.

Some languages allow for a lot more moving parts than others. For instance, in Java obj.otherName means that otherName is an instance variable (or theoretically a class variable) and access time is guaranteed to be constant (tbd: caveats). It's not redefinable as it is in C#, Python or Swift. So there's one less thing to track down but also less flexibility.

Or take an extreme example like Go's range loop. It's defined only for a handful of builtin data structures. When you see a range loop, you know what it does without following any further indirections. It can be extremely annoying and create a lot of friction when you have to work with custom data structures. But it can make other people's code easier to read than in other languages.

Or take something like this:

  a == b

This simple expression has a far greater number of possible meanings in a language that supports operator overloading and generics than in a language that doesn't.

So transforming an expression's possible meanings into its concrete meaning is what I call decompression. There may not be a need for looking anything other than public interfaces.

Also, you're talking about abstractions purely from the perspective of users. But we are often creators and maintainers of abstractions as well when we model a particular problem.

I think the better word you're looking for might be "reify" rather than "decompress". But there is something here that I can relate to. A long time ago I also made a comparison between various languages of "what do you have to understand to fully understand a Hello World program?" An interesting followup to that would be how much you have to understand on top of that to understand a more useful program.

You might be interested in this book, Patterns of Software, which brings up the issue that inheritance in OOP isn't about reuse but about compression and how the implications of that are why it's not had that much success in following through on the "reuse" promise. http://dreamsongs.com/Files/PatternsOfSoftware.pdf

Because no large effect of programming language choice has been observed, and because no theory explains why there would even be one[1], speculating on the causes of a non-existent phenomenon is not very helpful.

[1]: Once a certain point has been reached; theory does predict diminishing returns.

> Because no large effect of programming language choice has been observed

Do you mind sharing a source for this? It seems trivial to generate a programming language (Brainf*ck) that performs very poorly.

A more likely statement: all commonly used languages are more or less on par.

I meant the latter. I just can't repeat all caveats every time I bring this subject up :) https://news.ycombinator.com/item?id=20074684

It's near impossible to objectively measure anything when it comes to programming as projects are neither repeatable nor comparable and test subjects are never blank slates.

I have read quite a few studies, and I always come away thinking that the methods used are inadequate to support any conclusions at all.

But does that mean there is no causal effect of programming language choice? I don't know and I'm not making any claims either way.

I was talking about abstraction in one person's mind vs abstraction shared between a group a people. And what I said could actually be the reason why the choice of programming language has little effect (if that is the case)

> But does that mean there is no causal effect of programming language choice? I don't know and I'm not making any claims either way.

I'm also not making definitive claims, but we can't ignore observations, either. While small effects can appear or disappear depending on methodology, big effects are easy to find and hard to hide, especially in an environment with strong selection pressures. Therefore, the most likely explanation to why no big effect has been detected -- either in studies or in industry -- is that there isn't one, and that should at least be enough to make it the working hypothesis. So before explaining effects that have yet to be detected we should establish that they exist at all, something we have so far failed to do.

There may well be a small effect, and there could still be a large effect, although the chance of that is small unless we have a really good explanation to why we haven't found it.

Here are some reasons to doubt this working hypothesis:

1) It is inconsistent with the personal experience of most developers. Few would claim that C is just as productive and safe as some garbage collected language (provided that both are suitable for the task at hand), or that a SQL group by statement is not more productive to write than the equivalent procedural code.

2) The methods used to study the subject are completely unfit for purpose.

3) It is inconsistent with the few results that seem at least somewhat well supported because they are simple enough to measure, such as the constant bug count per line of code.

> It is inconsistent with the personal experience of most developers.

I disagree. I've been programming for about 30 years now, and most reports I've received -- both from programmers and managers -- are that language doesn't matter; certainly not much. Carefully selected personal "feeling" report are worthless (especially in biased forums), as that would produce even more evidence that homeopathy is effective than that programming languages make a difference.

> Few would claim that C is just as productive and safe as some garbage collected language

I agree, but that's where the many caveats come in. First, my argument is more one of diminishing returns (following Brooks's theory). I.e., languages make an increasingly small difference. C may have been a big improvement over assemby, and Java may have been a smaller but still quite big improvement over C, but now differences are quite small.

Indeed, the effect of GC and memory safety vs C was one that was almost immediately detected by industry (once mature, performant, etc.) and triggered a huge shift. We do not see such shifts now. Companies move among mainstream languages, among non-mainstream languages, and between the two classes in a way that does not suggest a big effect at all.

> or that a SQL group by statement is not more productive to write than the equivalent procedural code.

That's a completely different matter. I don't think anyone believes that writing a 1MLOC program in SQL is easier than writing a similar one in Java or Python, even if the Python/Java one is 1MLOC, and the SQL one is 200KLOC. When the program is small, there can be large differences, but that's a completely different problem (and also explained by theory).

> The methods used to study the subject are completely unfit for purpose.

I disagree, but it doesn't matter. Big effects are easy to find and hard to hide. And in an environment with strong selective pressures, they're found even when no studies at all are conducted.

> It is inconsistent with the few results that seem at least somewhat well supported because they are simple enough to measure, such as the constant bug count per line of code.

It is not. First, the difference in LOC is not as big as PL fans claim (it can be big for small programs, not large ones). Second, you may want to review those results.

>I agree, but that's where the many caveats come in. First, my argument is more one of diminishing returns (following Brooks's theory). I.e., languages make an increasingly small difference. C may have been a big improvement over assemby, and Java may have been a smaller but still quite big improvement over C, but now differences are quite small.

I don't necessarily disagree, but I'm a bit skeptical about the way in which you use the concept of diminishing returns. If I'm not mistaken then Brooks used diminishing returns in relation to headcount. And this is in fact how economists use the term as well - adding quantitatively more of one production factor.

But that doesn't apply to qualitative changes like introducing new production methods and tools, which is what programming languages are. New programming languages are not simply "more programming language".

If we agree that there have been observable effects of new programming languages in the past, then we cannot exclude the possibility of the same happening in the future.

>First, the difference in LOC is not as big as PL fans claim

What PL fans claim is a bit of a fluffy benchmark, but there are undoubtedly significant differences in LOC, certainly significant enough to make an economic difference.

>I've been programming for about 30 years now...

Shockingly, me too :)

> If I'm not mistaken then Brooks used diminishing returns in relation to headcount.

Oh, I'm referring to "No Silver Bullet" and diminishing returns in the sense that languages at best help with "accidental complexity", so the less of it there is, the less languages can help.

> New programming languages are not simply "more programming language".

But there are inflexible theoretical limitations on their utility (even without Brooks). We know the effect of different programming constructs on expressive compression and reasoning costs. The closer we get to the limit, the less benefit we can have.

> then we cannot exclude the possibility of the same happening in the future.

I'm mostly pointing out that it's not happening at present. I'm less sanguine about future advances because of various limitations (and, BTW, Brooks's predictions were called overly pessimistic by PL fans at the time and it turned out they were too optimistic), but I won't rule out another major breakthrough, or possibly two.

> but there are undoubtedly significant differences in LOC, certainly significant enough to make an economic difference.

I don't think I agree with your first assertion (although that depends on what we mean by "significant" here), and I certainly disagree with your second. We simply have not been able to detect or induce such an effect. Here, too, there are theoretical results showing that increased expressiveness cannot lead to cheaper reasoning, which is not an obvious result even in the worst case (because there are far fewer "compressible" programs than non compressible ones).

>We know the effect of different programming constructs on expressive compression and reasoning costs. The closer we get to the limit, the less benefit we can have.

>Here, too, there are theoretical results showing that increased expressiveness cannot lead to cheaper reasoning, which is not an obvious result even in the worst case (because there are far fewer "compressible" programs than non compressible ones).

I find that very interesting. Do you have a source for it?

Yes. Some interesting papers are:

* http://www.lsv.fr/Publis/PAPERS/PDF/Sch-aiml02.pdf

* http://www.lsv.fr/Publis/PAPERS/PDF/DLS-jcss-param.pdf

* https://www.cis.upenn.edu/~alur/Zohar03.pdf

What's important to put those results in context for those who are not familiar with the subject is that, in the context of complexity theory, the "model checking problem" does not refer to the complexity of a particular model checker algorithm, but to the inherent complexity of the problem of deciding whether a program M satisfies some property 𝜑 (i.e. whether M is a model of 𝜑 in the formal logic sense). The "model checking problem" is the closest to a mathematical description of what we usually mean when we talk of "reasoning about a program."

I've summarized these results and more here: https://pron.github.io/posts/correctness-and-complexity

Thanks!