Comment by dvt | Hacker News Reader

Comment by dvt | Hacker News Reader

> I like the concept of “simplistic programming” by which I mean “programming that is so simple that people will criticize you for it”. At first, that sounds strange… can we really get criticized for being “too simple”? Of course, we do.

Oh man, this hits home.

I've attempted to be a voice of reason on this front so many times, I've practically given up on it. I've come to the conclusion that most programmers suffer from an intellectual Napoleon complex (which I've seen in academic Philosophy, as well). I've argued for the use of simple "for loops" in lieu of complex map-reduce logic, argued against overly-complex threading logic where threading wasn't necessary, argued against using Spark where the dataset was so tiny, it could be handled by my iPhone. All my suggestions often fall on deaf ears. The counter-argument tends to be some contrived edge-case which (by that point in the discussion), I'm simply too fatigued to argue against. I've lost this war of attrition many (many) times.

The irony is that most OSS code tends to be "so simple it's dumb" -- why? Because simple code is easy to contribute to! I think if more professional programmers actually looked at real code outside of their corporate Bitbucket, working as a programmer would be much more enjoyable.

Is map-reduce really less simple than a for-loop? I agree with your other examples, but I see these too things as totally equivalent (unless you're in a language that make a dog's dinner of this type of thing, like old C++).

I think so, consider[1]:

     var r = array.map(x => x*x).reduce((total, num) => total + num, 0);

vs its for-loop counterpart:

    var r = 0.0;
    for (var j = 0; j < array.length;j++){
        var x = array[j];
        r += x*x;
    }

Maybe I'm just not as smart as everyone else, but I only have to keep like two things in my mind when looking at the for loop (easy), whereas in the map-reduce case, I need to have a very "holistic" view of what's going on (hard).

[1] https://stackoverflow.com/questions/17546450/for-array-is-it...

C++ guy but anecdotally, the first one really does read better.

I think your giving a bad map reduce example. It is written in a very immature functional programming style.

It should be written like this, IMO:

  var r = array
          .map(square)   
          .reduce(sum, 0);

Differences:

1. Separating out actions by line

2. Pass named functions that describe what they do

Eliminating the map, named functions or not, makes it pretty clear:

var r = array.reduce((total, num) => total + (num * num), 0);

Except the over head of that loop is orders of magnitude higher than the reduction. Did you get the bounds right; what happens if you accidently start at 1 or end at <=; what if you're nested; did you use i and j in the right place? The nooks for bugs in your "simple" loop are so profound that, in a language with expressive maps, I wouldn't accept that code in a review.

"Simple" is great until you're debugging off by one errors or any other errors between chair and keyboard. Complexity, in the sense that this guy rails against, is expressiveness. And, expressive code beats "simple" code every day of the week.

--Quick edit Imagine that loop in assembly. It's objectively "less complex" yet you didn't express it that way. Why? I think the answer to that question is the heart of the positive argument for programming abstractions (and the "complexity" they bring).

> Except the over head of that loop is orders of magnitude higher than the reduction.

Except that it's not. Read the SO answer I linked.

> The nooks for bugs in your "simple" loop are so profound that, in a language with expressive maps, I wouldn't accept that code in a review.

Yep, I've definitely worked with people like you. With all due respect, I just think you're fundamentally wrong. Off-by-1 errors will happen no matter what, and even a rudimentary test suite can catch those.

Off by one errors can't happen in the functional version. There's no reason to take a fatalist stance here. Just saying "well those programmers should write more tests" is silly when you could say, well why not tests and write code that is naturally hardened against foot guns.

I was referring to mental overhead; though in a more modern language like rust the performance implications would be far less profound than the linked SO discussion.

> Just saying "well those programmers should write more tests" is silly when you could say, well why not tests and write code that is naturally hardened against foot guns.

I don't think off-by-1 errors are that big of a foot gun, compared to, like, having a null as part of the language, or pointers, or about a million other things.

> I was referring to mental overhead

Gotcha. I still think that "linear step-by-step" thinking like "you take stuff out of a bucket, and then do stuff to it" is easier to parse than map-reduce, which requires you to have a "big picture" view of the whole process.

> Off by one errors can't happen in the functional version.

No? Not only can they happen, but I'd argue that it's easier for them to occur when using `reduce`, because you have to remember the initializer. Omitting it has no effect in the given example, but the given example is contrived. It's not rare to be dealing with something more complex than a list of numbers, so let's try that instead.

Let's say your items are a list of nodes. You want to sum their payloads to determine their aggregate cost. And, like kgwxd[1], you see `map` as superfluous. You've got this. You write:

    nodes.reduce((sum, x) => sum + x.value)

Uh-oh. Now the fact that there is no initializer does affect the result.

1. https://news.ycombinator.com/item?id=15867033

> No? Not only can they happen, but I'd argue that it's easier for them to occur when using `reduce`, because you have to remember the initializer.

I would say that's not an off by one error. Its a similar scale and type of error, but not the same.

It's much more insidious than an off-by-one error in a for-loop - the initializer is hidden!

Unless you use a language that actually enforces you to pass required parameters to a function call.

... which still wouldn't fix this problem, because the initializer is not a required parameter. It's allowed to be omitted by design, to signify that the first element shall be skipped and never passed as the second arg to the callback—or to put it another way, instead of being called once for each element, it's meant to handle the gaps between elements.

Er, the single-parameter version of reduce differs from the two-parameter version in that the semantics are changed to literally skip the callback invocation where the first element gets passed as x. Instead of being called n times, it gets called n-1 times. It's one of the most clear-cut manifestations of fences vs posts—the quintessential off-by-one problem.

Why have a compiler at all if you want to defer all error-checking to runtime?

Why does a programmer need to know an index counter exists if the index counter isn't used inside the for loop?

Might look nicer to write the first version as:

  let square = (n)   => n*n
  let sum    = (s,n) => s+n
  
  let r = array.map (square).reduce (sum,0)

I dunno, I find the second one harder. Because now you have to know about array indices and things like that. The for loop forces you to deal with petty details. The map reduce lets someone else deal with that.

I tend to think there's some additional overhead from Javascript's particular object oriented implementation of mapping and reducing when compared with haskell's i.e.

     f xs = foldr + 0 (map (\x -> x * x) xs)

seems a bit easier to me.

Is it habit? Probably. But I really find it hard to understand why maps and folds are simpler than manual iterative loops.

since a map and a sum are both folds, you can simplify it further:

f = foldr (\x total -> total + (x * x)) 0

but this is actually a counter argument. look at all these ways of doing this simple operation with higher order functions. plenty of replies from intellectual napoleons like me, too. whereas with the for loop, there's nothing much to say, which seems like a good thing.

i genuinely don't know how to feel about this.

I agree, but I think it depends a lot on your background as well as the background of the people who are likely to be looking at and modifying the code. Virtually any programmer is going to be able to eventually read and understand the second version while requiring little extra research, but for a lot of programmers they will have no idea how to reason about the first and no idea how to use/modify/extend it, so it is just some weird black-box.

Yes, obviously they can learn, and I don't think learning a bad thing (It's absolutely a necessary skill for a programmer, or anybody), but if the only place in the code that ever uses map/reduce is that one location, then programmers that come by it are likely not going to get a lot out of the time they're going to sink trying to figure it out. If it was just a basic loop, then overall people will likely spend less time figuring it out. For one piece of code it's not that big of a deal, but if your code-base is just full of fun little one-liners everywhere, it quickly becomes a mess to understand.

But with that, if your code is already full of stuff like map/reduce, then I think using map/reduce instead of the basic loop may very-well be preferred for readability (Though of course, your citation brings up performance concerns, which should also be taken into consideration). I think most important is just keeping the code-base consistent.

This is interesting to me because the map reduce has way less cognitive load for me.

With map reduce I see the following:

- We've got an array

- We're going to loop through it and square everything

- We're going to take that and add it all together

- The result is r

When I read the for-loop counterpart:

- We have a variable called r that we're initializing to 0

- We're starting a loop with j as the index initialized to 0 iterating through an array

- We've got a variable x on each iteration that is the current position in the array

- We're adding the square of x to r

Maybe it's a personal quirk. for-loops make me feel stupid.

The biggest issue I have is with this step:

> - We're going to take that and add it all together

Here, we have an imaginary variable: namely, "that." Something that we need to keep track of and is not explicitly referenced anywhere. It's relatively easy when you only have one of these things, but with complex map-reducer logic, you can end up having a whole bunch of these.

In the for-loop, everything I use is explicitly referenced. There's no "that" -- everything is "r" or "x" or "j."

The main advantage that I’ve seen when writing and reading functional instead of procedural/imperative code, is that the intention is usually clear.

We have a list of numbers that we want to square and add up, instead of the more mechanical steps with the loop approach.

The for loop is definitely easier to debug if that is necessary. Map reduce is more concise of course.

What about:

(loop for x in a summing ( * x x ))

Both map-reduce and for-loop are idioms. That is, they are things that one fluent in the language should be able to read at a glance. I suspect that dvt has more experience with languages where the for-loop is the standard idiom.

And in fact, many languages have only one or the other as a standard idiom. So it's really pretty simple: Use the one that's a standard idiom in the language that you're using.

Your language offers both? It's still pretty simple: Use the one that's better understood by the people that you're working with.

They understand both? The project still probably leans one way or the other. Prefer the one that is more commonly used in the project. Projects wind up using subsets of languages; using things outside the subset, while valid, looks "odd" within the project.

It's all about minimizing the cognitive load on the reader. It's not about what the writer prefers.

I agree: one of these things is not like the other. Map / filter / collect is not better because it is "fancy" but because those operations match the semantic intent of most operations on collections while being constrained to do a single thing. Conversely, for loops are unconstrained and reflect no intent in particular.

I mean, it can be, right? Take todays Advent of Code problem, where you want to grab the index of the largest element in a sequence breaking ties toward early elements, then re-distribute that element one at a time over other positions starting from the largest element (looping around as required).

I can write that using various maps, folds, flat_maps, and reduces, but no one else would understand what is going on in the code. Maybe someone will post the right idiomatic functional way to do that, but nothing elegant comes to mind.

People are constantly quizzed on map-reduce, spark type of stuff in interviews.

Resume 1:

> Did simplistic programming using for loops.

Resume 2:

> Map reduce using spark.

Guess which resume gets picked up by filters? Noone wants to get caught with their pants down when they have to look for their next job.

Pressure to "keep up" is unfortunate reality of our profession. Resume Driven Development is a thing.

I can't... agree with you more. Just lost one recently. There was no need to go "microservice" but we did (I have nothing against that structure). Argued against it, got shot down and they went around and are implementing it now. Happened so many times and it's.. dead tiring.

suffer from an intellectual Napoleon complex

I call it the low talent to ego ratio.

While I disagree on the map -> reduce I agree on the other positions you articulated.

Most corporate problems are most likely solved by pretty simple code. The problem is that you are not building your resume that way and your future job searches will suffer. To stay up to date I often squeeze things into my work that are not strictly necessary.

what's funny is that the code compiles the small map-reduce and the for loop to exactly the same bytecode probably. :)

Not my area of expertise, so I want to ask this question: Do compiler optimizations exist to automatically parallelize a for loop, which can be parallelized, no matter how to programmer wrote their for loop?

If not, then map / reduce offers non-trivial performance benefits over a for loop. Mapping, as far as I'm aware, is "embarrassingly parallel", assuming the function that's mapped is pure? I think?

in perl6 this is done with the hyper operator, but for a small data set a for loop (non parallel) will almost always just run faster.

No, map reduce can be run in parallel, at least in some cases.

Reply on news.ycombinator.com