Hacker News Reader

Comment by recursivedoubts | Hacker News Reader

This is one vector for complexity, to be sure. Saying "no" to a feature that is unnecessary, foists a lot of complexity on a system, or has a low power to weight ratio is one of the best skills a senior developer can develop.

One of my favorite real world examples is method overloading in Java. It's not a particularly useful feature (especially given alternative, less complicated features like default parameter values), interacts poorly with other language features (e.g. varargs) and ends up making all sorts of things far more complex than necessary: bytecode method invocation now needs to encode the entire type signature of a method, method resolution during compilation requires complex scoring, etc. The JVM language I worked on probably had about 10% of its total complexity dedicated to dealing with this "feature" of dubious value to end users.

Another vector, more dangerous for senior developers, is thinking that abstraction will necessarily work when dealing with complexity. I have seen OSGi projects achieve negative complexity savings, while chewing up decades of senior man-years, for example.

View on HN

Well what I mostly experienced in my years in the field was that developers, wether senior or not, feel obliged to create abstract solutions.

Somehow people feel that if they won't do a generic solution for a problem at hand they failed.

In reality the opposite is often true, when people try to make generic solution they fail to make something simple, quick and easy to understand for others. Let alone the idea that abstraction will make system flexible and easier to change in the future. Where they don't know the future and then always comes a plot twist which does not fit into "perfect architecture". So I agree with that idea that abstraction is not always best response to complex system. Sometimes making copy, paste and change is better approach.

Kevlin Henney makes interesting points on this. We often assume that abstractions make things harder to understand, at the benefit of making the architecture more flexible. When inserting an abstraction, it is supposed to do both, if at all possible. Abstracting should not only help the architecture, but also the understanding of the code itself. If it doesn't do the latter, then you should immediately question whether it is necessary, or if a better abstraction exists.

The take-away I took from it is that as developers, we love to solve problems using technical solutions. Sometimes, the real problem is one of narration. As we evolve our languages, better technical abstractions become available. But that's not going to prevent 'enterprisey' code from making things look and sound more difficult. Just look at any other field where abstractions aren't limited by technicalities: the same overcomplicated mess forms. Bad narrators narrate poorly, even when they are not limited.

I think we forget that while “engineering” is about maximizing the gain for a given investment of resources, it can be stated another way as building the least amount of bridge that (reliably) satisfies the requirements.

Abstraction can be used to evoke category theoretical things, but more often it’s used to avoid making a decision. It’s fear based. It’s overbuilding the bridge to avoid problems we don’t know or understand. And that is not Engineering.

I find sometimes that it helps me to think of it as a fighter or martial artist might. It is only necessary to not be where the strike is, when the strike happens. Anything more might take you farther from your goal. Cut off your options.

Or a horticulturist: here is a list of things this plant requires, and they cannot all happen at once, so we will do these two now, and assuming plans don’t change, we will do the rest next year. But plans always change, and sometimes for the better.

In Chess, in Go, in Jazz, in coaching gymnastics, even in gun safety, there are things that could happen. You have to be in the right spot in case they do, but you hope they don’t. And if they don’t, you still did the right thing. Just enough, but not too much.

What is hard for outsiders to see is all the preparation for actions that never happen. We talk about mindfulness as if it hasn’t been there all along, in every skilled trade, taking up the space that looks like waste. People learn about the preparation and follow through, instead of waiting. Waiting doesn’t look impressive.

Your analogy with Chess and Go is flawed though. In these games you try to predict the best opponent move responding to yours, the worst case basically, and then try to find the best response you have to it, and so on, until you cannot spend any more time on that line or reach your horizon. You are not "hoping things do not happen". If you did that, you would be a bad chess or go player. You make sure things do not happen.

So that analogy works against your point.

I disagree. Especially in teaching games, and everyone writing complex software is still learning.

In Go there are patterns that are probably safe, but that safety only comes if you know the counters. In a handicapped game, it’s not at all uncommon for white to probe live or mostly live groups to see if the student knows their sequences. You see the same in games between beginners.

Professional players don’t do this to each other. They can and will “come sideways” at a problem (aji) if it can still be turned into a different one, but they don’t probe when the outcome is clear. In a tournament it inflicts an opportunity cost on the eventual winner, and it is considered rude or petty. They concede when hope is lost.

They still invested the energy, but now it comes mostly from rote memorization.

And how does that contradict my point to always expect the best opponent move and think about the best thing to do in return, instead of simply hoping the worst will not happen? I think you are actually even supporting my point here.

I think the thing that comes with ~~seniority~~ experience is being better able to predict where abstraction is likely to be valuable by becoming: more familiar with and able to recognize common classes of problems; better able to seek/utilize domain knowledge to match (and anticipate) domain problems with engineering problem classes.

I’m self taught so the former has been more challenging than it might be if I’d gone through a rigorous CS program, but I’ve benefited from learning among peers who had that talent in spades. The latter talent is one I find unfortunately lacking in many engineers regardless of their experience.

I’m also coming from a perspective where I started frontend and moved full stack til I was basically backend, but I never lost touch with my instinct to put user intent front and center. When designing a system, it’s been indispensable for anticipating abstraction opportunities.

I’m not saying it’s a perfect recipe, I certainly get astronaut credits from time to time, but more often than not I have a good instinct for “this should be generalized” vs “this should be domain specific and direct” because I make a point to know where the domain has common patterns and I make a point to go learn the fundamentals if I haven’t already.

I agree that premature abstraction is bad. Except when using a mature off-the-shelf tool, e.g. Keycloak. Sometimes if you know that you need to implement a standard and are not willing to put in the effort for an in-house solution, that level of complexity just comes with the territory, and you can choose to only use a subset of the mature tool's functionality.

I also have a lot of experience starting with very lo-fi and manual scripting prototypes to validate user needs and run a process like release management or db admin, which would then need to be wrapped in some light abstractions to hide some of the messy details to share with non-maintainers.

Problem is, I've noticed that more junior developers tend to look at a complex prototype that hits all the user cases, and see it as being complicated. Then they go shopping for some shiny toy that can only support a fraction of the necessary cases, and then I have to spend an inordinate amount of time explaining why it's not sufficient and that all the past work should be leverages with a little bit of abstraction if they don't like the number of steps in the prototype.

So, not-generic can also end up failing from a team dynamic perspective. Unless everyone can understand the complexity, somebody is going to come along and massively oversimplify the problem, which is a siren song. Queue the tech debt and rewrite circle of life.

Sure over abstraction is a problem. And sometimes duplication is better than depend y hell. But other times more as traction is better.

In true it’s an optimisation problem where both under abs over abstracting, or choosing a the wrong abstractions lead to less optimal outcomes.

To get more optimal out comes it helps to know what your optimisation targets are: less code, faster compilation, lower maintenance costs, performance, ease of code review, adapting quirky to market demands, passing legally required risk evaluations, or any number of others.

So understand your target, and choose your abstractions with your eyes open.

I’ve dealt with copy paste hell and inheritance hell. Better is the middle way.

I would like to be able to upvote this answer 10 times.

I often remember that old joke:

When asked to pass you the salt, 1% of developers will actually give it to you, 70% will build a machine to pass you a small object (with a XML configuration file to request the salt), and the rest will build a machine to generate machines that can pass any small object from voice command - the latter being bootstrapped by passing itself to other machines.

Also makes me remember the old saying

- junior programmer find complex solutions to simple problems

- senior programmers find simple solutions to simple problems, and complex solutions to complex problems

- great programmers find simple solutions to complex problems

To refocus on the original question, I often find the following misconceptions/traps in even senior programmers architecture:

1) a complex problem can be solved with a declarative form of the problem + a solving engine (i.e. a framework approach). People think that complexity can be hidden in the engine, while the simple declarative DSL/configuration that the user will input will keep things apparently simple.

End result:

The system becomes opaque for the user which has no way to understand how things work.

The abstraction quickly leaks in the worst possible way, the configuration file soon requires 100 obscure parameters, the DSL becomes a Turing complete language.

2) We have to plan for future use cases, and abstract general concepts in the implementation.

End result:

The abstraction cost is not worth it. You are dealing with a complex implementation for no reason since the potential future use cases of the system are not implemented yet.

3) We should factor out as much code as possible to avoid duplication.

End result:

Overly factored code is very hard to read and follow. There is a sane threshold that should not be reached in the amount of factorization. Otherwise the system becomes so spaghetti that understanding a small part requires untangling dozens and dozens of 3 lines functions.

---

When I have to argue about these topics with other developers, I often make them remember the worst codebase they had to work on.

Most of the time, if you work on a codebase that is _too_ simplistic and you need to add a new feature to it, it's a breeze.

The hard part, is when you have an already complex system and you need to make a new feature fit in there.

I'd rather work on a codebase that's too simple rather that too complex.

I like what you are saying here! My observations below.

1) When gradually most of your implementation is happening in a DSL / graph based system all of your best tools for debugging and optimizing are useless.

2) So often I've seen people make an 'engine' before they make anything that uses the engine and in practice the design suffers from needless complexity and is difficult to use because of practical matters not considered or forseen during the engine creation. Usually much work has been spent on tackling problems that are never encountered but add needless complexity and difficulty in debugging. Please design with debugging having an equal seat at the table!

3) Overly factored code is almost indistinguishable from assembly language. - Galloway

> Another vector, more dangerous for senior developers, is thinking that abstraction will necessarily work when dealing with complexity.

I'm pretty good at fighting off features that add too much complexity, but the abstraction trap has gotten me more than once. Usually, a moderate amount of abstraction works great. I've even done well with some really clever abstractions.

Abstraction can be seductive, because it can have a big payoff in reducing complexity. So it's often hard to draw the line, particularly when working in a language with a type of abstraction I've not worked much with before.

Often the danger point comes when you understand how to use an abstraction competently, but you don't yet have the experience needed to be an expert at it.

Yes, but remember Sanchez's Law of Abstraction[0]: abstraction doesn't actually remove complexity, it just puts off having to deal with it.

This may be a price worth paying: transformations that actually reduce complexity are much easier to perform on an abstraction of a program than on the whole mess with all its gory details. It's just something to keep in mind.

[0] https://news.ycombinator.com/item?id=22601623

You're echoing my comment: abstraction can be very useful, but you have to take care.

Also, it's pointless to claim that transformations on an abstraction can reduce complexity, but abstractions themselves can not. Abstractions are required for that reduction in complexity.

As a Java end user I'm really glad that method overloading exists. The two largest libraries I ever built would have been huge messes without overloading. But I take your point that method overloading might be a net negative for the Java platform as a whole.

Yes, java would be a mess without overloading (particularly for telescoping args), but that's because it doesn't include other, simpler features that address the same problems. Namely:

- default parameter values

- union types

- names arguments

I would also throw in list and map literals, to do away with something like varargs.

All of these are much simpler, implementation-wise, than method overloading. None would require anywhere near the compiler or bytecode-level support that method overloading does. It just has a very low power to weight ratio, when other langauge features are considered. And, unfortunately, it makes implementing all those other features (useful on their own) extremely difficult.

I don‘t really see a significant difference between overloaded methods and a single method with a sum type (where, in the general case, the sum is over the parameter-list tuple types of the overloaded methods). One can interpret the former as syntactic sugar for the latter.

Static vs dynamic dispatch, to start with.

You need default values (and ideally named parameters) to really make them equivalent, but yes, both address a similar set of developer needs.

The java implementation, where it bleeds into bytecode and method scoring in insane ways, is particularly unfortunate.

But it’s not actually syntactic sugar in java. There is a whole concept of type-signature which is part of method look up.

Without overloading lookup would just be via name.

On the other hand languages get by completely without method overloading and use for example default parameter values.

I don not think that method overloading is necessarily required to not make a mess.

Overloading is very valuable, especially in conjunction with generic code.

But unfortunately, it is way overused. It takes a long time to develop good judgement. I'm still working on it.

> This is one vector for complexity, to be sure. Saying "no" to a feature that is unnecessary, foists a lot of complexity on a system, or has a low power to weight ratio is one of the best skills a senior developer can develop.

I don't consider myself to be an exceptional developer, but this alone has launched my career much faster than it would if I was purely, technically competent. Ultimately, this is a sense of business understanding. The more senior/ranking you are at a company, the more important it is for you to have this tune in well.

It can be really, really hard to say no at first, but over time the people ask you to build things adapt. Features become smaller, use cases become stronger, and teams generally operate happier. It's much better to build one really strong feature and fill in the gaps with small enhancements than it is to build everything. Eventually, you might build "everything", but you certainly don't need it now. If your product can't exist without "everything", you don't have a strong enough business proposition.

----

Note: No, doesn't mean literally "I'm/we're not building this". It can mean two things:

* Forcing a priority. This is the easiest way to say no and people won't even notice it. Force a priority for your next sprint. Build a bunch of stuff in a sprint. Force a priority for another sprint. Almost inevitably, new features will be prioritized over the unimportant left overs. On a 9 month project, I have a 3 month backlog of things that simply became less of a priority. We may build them, but there's a good chance nobody is missing them. Even if we build half of them, that still puts my team 1.5 months head. For a full year, that's almost like getting 2 additional months of build time.

* Suggesting an easier alternative. Designers have good hearts and intentions, but don't always know how technically difficult something will be. I'm very aggressive about proposing 80/20 features - aka, we can accomplish almost this in a much cheaper way. Do this on 1 to 3 features a sprint and suddenly, you're churning out noticeably more value.

I have seen OSGi projects achieve negative complexity savings, while chewing up decades of senior man-years, for example.

I'm not surprised; that and a lot of the Java "culture" in general seems to revolve around creating solutions to problems which are either self-inflicted or don't actually exist in practice, ultimately being oriented towards extracting the most (personal) profit for a given task. In other words: why make simple solutions when more complex ones will allow developers to spend more time and thus be paid more for the solution? When questioned, they can always point to an answer laced with popular buzzwords like "maintainability", "reusability", "extensibility", etc.

I always found it surprising that Java implemented method overloading, but not operator overloading for arithmetic/logical operators. It's such a useful feature for a lot of types and really cleans up code, and the only real reason it's hard to do is because it relies on method overloading. But once you have that, why not just sugar "a + b" into "a.__add(b)" (or whatever).

You don't have to go all C++ insane with it and allow overloading of everything, but just being able to do arithmetic with non-primitive types would be very nice.

Operator overloading is deliberately more limited in D, with an eye towards discouraging its use for anything other than arithmetic.

A couple of operator overloading abominations in C++ are iostreams, and a regex DSL (where operators are re-purposed to kinda sorta look like regex expressions).

Yeah.

One in-between option I have kicked around w/ people is offering interfaces that allow you to implement operator overloading (e.g. Numeric). Then you wouldn't have one-off or cutesy operator overloading, but would rather need to meet some minimum for the feature. (Or at least feel bad that you have a lot of UnsupportedOperationExceptions)

Java had/has a ton of potential, but they kept/keep adding features that make no sense to me, making the language much more complex without some obvious day-to-day stuff like list literals, map literals, map access syntax, etc.

Oh well.

C++ did set a bad example with operator overloading.

    cout << "Hello World";

What the heck is that? Shift cout by "Hello World"?

It looks like they did that just because they can.

> It looks like they did that just because they can.

The thing is, people thought it was a good idea at the time, and had good reasons. It took maybe a decade before experience showed otherwise. Even today it takes a while to convince newer programmers that it is indeed a bad idea, and even then they don't really believe it.

An even more perniciously bad language feature is macros. I'm pretty sure that once I step down as D's BDFL or am forcibly removed D will get macros :-/

Does this actually confuse anyone? There are plenty of problems with iostreams, but I have never been encountered an issue caused by the use of << for streams.

Optionals help, but what's really missing is union types for arguments.

And named arguments.

All three features, useful on their own, could be added to java at maybe 10% of the complexity of method overloading. With method overloading, they are all exponentially more complicated.

It's crazy how many places method overloading ends up rearing its ugly head if you are dealing with the JVM.

I wish all the args always just came as one structured object with some convenient syntax for accessing destructured parts. That object can have named parts as well as unions, optional, collections, and combinations of them.

What is method scoring? I’ve never heard that term in VM/compilers and my Google-fu is failing me.

I imagine it refers to the comparison of argument types to find the best match among overloaded methods with the same arity. e.g. when the machine has got "void foo(java.lang.Object)" and "void foo(java.lang.Number)" to choose from.

Yep, exactly.

Gets to be lots of fun when you throw autoboxing and, especially, generics, into the mix.

Your example is funny. Default parameters are far more complicated than method overloading.

That said, I fully agree on the OSGi point. Makes me worried about a lot of the new features I hear are on the way. :(

> Default parameters are far more complicated than method overloading.

I've implemented both, I disagree. Unless you are talking about default arguments in the presence of method overloading, which is insane, and which I have also implemented.

The important thing with default parameters, if you have separate compilation, is that they are fixed in the callee and not in the caller. Otherwise, when the default value is changed, previously compiled callers will use a different value than declared by the callee. Compiling the default value into the caller is a problem in C++, and (IIRC) in Swift. In other words, (positional) default parameters should act like syntactic sugar for equivalent method overloads. And then actual overloads provide more flexibility for the callee implementation because it doesn’t have to represent the default case as a special in-band value.

I can see that as an implementation detail, but it need not be one. There isn't a reason that a runtime couldn't use a flag value for absent args and let the callee code set the value up on their side.

I mean, the java runtime has all sorts of crazy stuff going on w/ a security manager, to almost zero benefit. Seems easy enough to make something like this work.

I'm taking about in use, mainly. Python, as an example, really screws it up by not giving you a way to know if you have the default or a value that equals it. Then, the more default values you pile in, the more this fun facet piles up.

So, if you skip on that part of it, and you can while still getting far, it is easier. With it, though, is stupid complicated for little benefit.

Common Lisp has a pretty simple fix for this: when you declare a parameter with a default value, you can also declare a variable that will be true iff the parameter was actually passed.

The function definition looks like this:

  (defun foo (&optional (a 1 a-passed))
    (if a-passed (print a) 
       (print "a not passed"))

  > (foo 10)
  10
  > (foo 1)
  1
  > (foo)
  a not passed

This is still relatively easy to implement, and very easy to use in my opinion. Of course, combining this with named arguments is even better, and that is supported as well (just replace &optional with &key, and then specify the name when calling the function - (foo :a 1)).

Why would you need to know if a value is the default or a value the equals the default? I can't think of a reason for that as I've never faced this problem in my career.

It is niche, but it does come up. Usually with boolean parameters, but also on migrations. Makes it possible to change defaults in a controlled way rather easily, since you can very easily log all of the places you used the default.

Nah, you should be using another value if you care about that. For example in C# a nullable bool would be a fine choice (three values: true, false, null), or alternatively Option types.

This feels like classic blaming the user. :)

You can also have better behavior between multiple defaults if you know whether someone gave a value or not.

Basically, it is another tool compared to overloading. It should be no surprise that there can be preference on how to use between them.

one example is `namedtuple`'s `._replace`:

  >>> x = Foo(a=1, b=2, c=3)
  >>> x._replace(a=1000, c=3000)
  Foo(a=1000, b=2, c=3000)

in this case, you really need to know if the user passed a replacement or not.

> Python, as an example, really screws it up by not giving you a way to know if you have the default or a value that equals it.

It does: set a unique `object()` instance as the default. It will only compare equal to itself, but it's even clearer to read if you check for identity with `is`. You'll need to define it outside of the function definition.

It's not the most concise, but it works, and it's an established pattern used in many high quality open source projects.

Thanks. This is kind of reinforcing my "not actually simple to use" point, though. :)

Reply on news.ycombinator.com