Magic is often used to describe confusing code | Hacker News Reader

Magic is often used to describe confusing code

quii.co.uk

76 pointsquii56 comments

I tend to use the word magic for those situations when you can not easily examine the source code and follow the control flow. For instance, you are in some part of the code and you have a User object and you see the code calls the .babble() method on the object. You go to the User definition, and there's no babble method. You follow up the superclasses, and there's no babble method. Where did the babble method come from? Where is it going? Who created it? Where can I find the documentation for it (if any)? Even if I do find something that seems to implement it, can I be sure that some other code didn't fiddle with the babble method in another way?

Languages permit varying levels of this sort of magic, ranging from the Ruby extreme of dynamically rewriting classes and inheritance hierarchies on the fly, to the pretty-much-no-magic of C. (C qua C anyhow. Preprocessor can get fun.)

I think this is one of the worst sorts of magic, and I've generally tried to arrange my code so that even when I'm using a dash of magic, the pathway back to where everything came from is still fairly clear. (Usually this can be done just by dealing with the fact you may need to use a few more keystrokes to leave the breadcrumb trail back to the implementation. Sometimes simplifying a bit of functionality down to zero bytes in the source code is going a step too far.)

I'll also observe that having used a number of both high-magic and low-magic languages, in terms of costs/benefits, I am generally underwhelmed by the benefits of the magic, and find the cost generally go up more quickly than most people realize as source code size goes up. I generally prefer low-magic languages. With just a bit of thought, you can still get a huge amount of power without the costs. (For instance, believe it or not, Haskell is a nearly-zero-magic language by this definition. For all the crazy stuff it seems to be able to do, you can always see what's going on just by following back the symbol definitions, with the exception of just a finite bit of syntax sugar implemented by the compiler, the most important of which is generally tutorial-level.)

the problem is when you use magic, you're deliberately being clever, and the only proper way to be clever is if you thoroughly document your cleverness with comments and diagrams and examples. the person you befuddle with how clever you are is probably yourself, 2 years from now.

Eh, try a month.

Later today: Wait, what was I doing again?

What is this in reply to?

I think they were raising your "try a month" to "try 6 hours".

Correct. Thanks!

Yes, and I was re-raising it to "five minutes" :( Apparently the joke wasn't very obvious.

see - Lions' Commentary on UNIX 6th Edition, with Source Code [1]

[1] https://en.wikipedia.org/wiki/Lions%27_Commentary_on_UNIX_6t...

        /*
         * If the new process paused because it was
         * swapped out, set the stack level to the last call
         * to savu(u_ssav).  This means that the return
         * which is executed immediately after the call to aretu
         * actually returns from the last routine which did
         * the savu.
         *
         * You are not expected to understand this.
         */
        if(rp->p_flag&SSWAP) {
                rp->p_flag =& ~SSWAP;
                aretu(u.u_ssav);
        }

> you see the code calls the .babble() method on the object... Where did the babble method come from?

I call it "grep-resistant programming"

Smalltalk would help with this kind of magic. The Smalltalk Method finder is (British-English School-yard definition of) Magic ;)

> I tend to use the word magic for those situations when you can not easily examine the source code and follow the control flow.

Reminds me of the quote “In Smalltalk, everything happens somewhere else “—Adele Goldberg.

I call these kinds of things, “action-at-a-distance,” or if I’m feeling snide, “COMEFROM code.” The thing is, action-at-a-distance is exactly what I want when I’m trying to separate concerns.

In Rails, for example, you can decorate a controller method using `before` at the top of a controller. To the person unfamiliar with the spell, this makes perfect sense:

  def index
    authenticate_current_user
    
    render something_or_other
  end

But this is magic:

  before authenticate_current_user, only: [:index]

  # ...lots of code...

  def index
    render something_or_other
  end

That is perfectly grep-able if you know to grep a method name, and if you have a small controller class it ought to stand out, &c. &c., so it rarely offends. But it’s still magic.

It’s a win if we want to make our index method as simple as possible.

And that goes for all of Rails, really. The very idea of “convention over configuration” is a wholesale embrace of magic. In Ember:

  userService: service(),

How is it that if you write `service()` without naming a service, the function defaults to the name of the property it is bound to? That makes absolutely zero sense in JavaScript, and the more you know about JavaScript, the more magical this becomes.

There is, in fact, no way to find the code that does this unless somebody whispers in your ear that Ember does compile-time chicanery, and you aren’t really programming in JavaScript, you’re programming in Ember, a language that has a lot in common with JavaScript, and that compiles to JavaScript.

You can decide for yourself if this is a win. Lots of people like it.

Exactly. I use the term a lot with fully dynamic code without extremely strong documentation every step of the way. With 10 layers of dynamic invocation (typical in Ruby on Rails) it is almost impossible to know without a debugger what code path your code will take.

In practice, at work, I haven't found people using magic to describe code that they don't understand, but rather code that uses obscure or somewhat hidden features to try and do things in a smart way.

For instance if a method relied on it being modified at run time by it's caller to ensure sanity of it's input I would call it magical.

I hope you appreciate that the word “magic” is relative to the person using it. For you, it’s self-modifying code. But to someone else, it’s mapping. And to a third person, it’s monkey-patching. And to a fourth person, it’s lazy evaluation of infinite streams, and to a fifth person, it’s ...

...and the point is that magic is a word that's often used alongside readability to deride source code that isn't done the way you would have done it.

“Intuitive equals familiar”—Jef Raskin

In your specific case, the argument is much narrower: Scala's for comprehension is, in itself, a very magical construct that you have to read from the documentation, while map is a function you can actually find in the code and follow: I'd argue that scala's 'for' is its most magical feature, even more than implicits. So anyone that finds for easier than map doesn't really understand what for really does, just an approximation from another language.

That said, there is such thing as not using language features that your team isn't familiar with: There is cognitive cost in, say, understanding higher kinded types, adding cats as a project dependency, or, god forbids, shapeless. There is such thing as excessive complexity, or using features that make anyone that didn't write the code suffer to understand it.

So yes, there are definitely good reasons to say that something is magic, and wishing for a lower cognitive load. Now, the map function isn't it though. I'd just avoid making it dotless, because it's not really any clearer, especially in a case where it's a higher order function, like in your example.

There is no magic other than the unfamiliar. The problem is not magic or the unfamiliar, it is complexity.

A technique can be understood, and still a pain in the ass to use; then it is bad. This is of course subjective, by comparison with other ways of solving the same problem.

Or, a technique can be useful, but too difficult to learn to justify using it on something simple; then it is inappropriate, and should be replaced by simpler, presumably less concise, code. This is of course subjective, dependent on the code's audience (readers, reviewers, maintainers). This is the root of the article's map vs for-loop example: different levels of familiarity with the technique of mapping functions over something.

Working towards a rule to distinguish between magic and powerful would be interesting and I for one would like to see some of HN take some swings here.

On first pass I'm having issues imagining a specific function that would be part of a standard library that is sufficiently magic that it warrants avoiding. Learning is just a part of programming. Something that initially seems like magic can quickly becomes one of your favorite hammers and make you more productive (just found reduce-kv the other day, it's awesome). If it's in the standard library then at least being able to use it seems wise. Note I'm not saying one should or could reasonably be demanded to know everything about the standard library, but rather that being asked to know something in it going forward from a point is likely reasonable.

Magic tends to live in side-effect heavy libraries and frameworks. Rails is full of magic that can make you very productive for specific tasks, but learning all the magic takes a lot of time and the incantations are not necessarily generalizable so when you switch over to Django or Node you need to go learn new magic, or sometimes the magic your looking for doesn't exist anymore and you need to learn what was actually happening (this happened to me switching from Rails to Django, I forget what in particular but it was somewhat jarring realizing how much I had been taking for granted). Learning something like map will translate even if the literal syntax doesn't, knowing and relying on x-many separate opt-args for defining an ActiveRecord relationship to save explicitly typing a dozen lines might be magic not worth it especially if the whole team has to know.

Edit: Just wanted to add that although I took a shot at a rough distinction I want to clarify that I am not suggesting that I have the correct answer, I only hope to elicit the thoughts of better thinkers and posters.

A few years ago, I was working on a testing automation tool. The tests were written in Cucumber, and run on IronRuby, which hooked into a C# application and would press buttons and otherwise move around the application testing that everything did what it was supposed to. My job was to write the layer that sat between the Cucumber tests and the C# application and called the proper events when a Cucumber step was called.

The initial approach was to start the C# and IronRuby processes and have the IronRuby process watch the C# process for events, but this quickly broke down; the C# process ran in an STAThread, and in order to allow a second thread to interact with that it would have to be an MTAThread, and with 500K lines of existing code, there was no telling what changing that would do. So instead we inserted a callback in main. As soon as the CLR had loaded, if a command line parameter had been given, we started the IronRuby interpreter from the C# application. After hooking into some events we would drop out of the IronRuby interpreter and the C# application would start as normal, only going into IronRuby when an event handler was called.

There was just one problem. The IronRuby interpreter was starting in unmanaged code, and the CLR was seeing it as a noop. So when it got to the if statement that checked the command line parameter, it optimized the whole if statement out of existence.

My pair and I spent hours reading docs, trying to bend the optimizer to allow the if statement without hurting application performance. Finally, my pair inserted the following line into the if statement, after the call to the IronRuby interpreter:

    Console.PrintLn("magic!");

It worked. The optimizer recognized the print statement as C# code and left the if statement alone.

That's always what I think of when I think of magic in code.

Many programming languages have features about which I would say, "You should either use them never, or all the time, but nowhere in between."

map is one of those features. Closures are another.

The reason is simple. The cost of learning the feature gets amortized over all of the times you use it. It really isn't worth learning the feature just to see it used a single time. But if you've paid the price to learn it, then there is no reason not to use it wherever it makes sense.

Unfortunately there are some very large languages with many reasonable working sets of features, but sharp corners when you combine the wrong ones. C++ is notorious for this, but it is hardly the only one. In that case it is important to agree on the subset of features that you will use. And not introduce others willy nilly.

I would therefore appreciate the right to use map. But if my co-workers do not know it, I would think hard about how hard to press for adding it to what is needed to understand my code.

That makes no sense. Maps and closures are tools like any other. Just as with many other tools, they are powerful, and should be used WHEN APPROPRIATE.

With maps, for instance, yes generally declarative programming is preferable to imperative programming (http://stackoverflow.com/questions/1784664/what-is-the-diffe...)

However, I've seen monster Java8 stream and C# LINQ statements that would benefit from additional intermediary variables, helper functions, and even cases where loops make things more clear.

btuilly probably meant map as in mapreduce not map as in hashmap.

Btilly rather clearly meant it in the sense of the higher order function (from which mapreduce gets its name), a fact deanCommie clearly grasped.

>Just as with many other tools, they are powerful, and should be used WHEN APPROPRIATE.

You can't work in a vacuum on large projects. To be an effective engineer you have to understand the limitations of your team members. Forgoing map in lieu of a more verbose loop is completely appropriate if you're working with a bunch of junior people that need to maintain and fix bugs in the same codebase.

I didn't downvote you, but I'm guessing the reason for your downvotes is that in this day and age, even junior programmers should be expected to understand map. It's such a simple concept that anyone who writes code for a living should be reasonably expected to understand it. If you're on a team with a bunch of programmers and nobody understands map, GTFO. It will not end well.

In fact, map is simpler than a loop.

So maybe I'll start to use python metaclasses all the time ;)

But good point poiting out to think hard about it when having co-workers working with your code, readability counts too !

So I agree with pretty much everything you said, but it surprised me that people would consider "map" and "closure" a) at all difficult to learn and b) an all-or-nothing business. Could you elaborate? I've found them to be complementary to traditional iteration, object, and macro oriented patterns—they have their strength and weaknesses, and the idea of seeing them either nowhere or everywhere is weird.

You underestimate the bias caused by our initial language. If you're an imperative programmer who has seen some OO, you can learn what map and closures are, but still have no idea what design patterns fit with them. And will be taken aback the first time you encounter them.

The first time I encountered a program that used them in an interesting way, I spent 20 minutes of utter confusion trying to figure out where anything HAPPENED. When I finally found the magic line, I went to the person who wrote it and said, "Why didn't you at least leave a comment!??" He looked at me, slightly puzzled, and said, "What comment would have helped?"

Today these are much more widely understood than they used to be. Mostly because JavaScript forced people to learn it. Though at one point I did have a role in popularizing the concepts in the Perl community with http://www.perlmonks.org/?node_id=34786.

Maps have a very clear and discrete semantic, no need for a flag or a comment, when someone reads the code if they are not familiar with map(x, things) then they will look it up and quickly become familiar. On the other hand closures say "the rules about variables that you might have thought were the rules are not the rules, these are not the rules you are looking for, move on, move on". People who haven't seen them before will be genuinely puzzled.

Yes, a good programmer will find out what is going on, they will be a better programmer there after. However alot of code is picked up by contractors or off shore, or even just by an intern and what happens then is that they hide their incompetence and introduce lots of bugs, or just fake the tests or lie about the tests.

I can't understand why you would use Scala and not use a simple higher order function such as map. It communicates the intentions clearly and is easy to understand.

I can't fathom how hard life with Scala has to be if map() is something you have issues with. Someone is clearly way in over their head. At least have him go through a tutorial or two, jesus...

My personal cutoff for magic is if it's determined at compile time, I'm happy.

If someone has proven to the compiler that a piece of code has a certain nonobvious behaviour, I'm usually intrigued.

If my interpreter starts parsing Roman numerals as constants because a gem depends on a gem which depends on gem `roman` (I'm looking at you Ruby), I'm not amused.

Scala's slick package is a fantastic example of advanced language features being used for good.

There is nothing magical about computing (unless you use Source Mage [1]). But, I have to agree with the author: people tend to use the word magic if they cannot immediately grasp the meaning of a piece of code. This code might use a language construct we are not familiar with, or implement a complex O(log N) algorithm, or it might use a magic number [2]. In most cases, however, it is worth to spend time to understand such code.

[1]: https://en.wikipedia.org/wiki/Source_Mage

[2]: https://en.wikipedia.org/wiki/Fast_inverse_square_root#Overv...

I wanted to get back into using Sorcerer, but the authors website has gone down (looks like for good this time).. sad

sorcerer.silverice.org

You could try the Wayback Machine: http://wayback.archive.org/web/*/http://sorcerer.silverice.o...

> Don't lazily conflate the unfamiliar with magic. It's short-sighted and annoying.

Why? Because magic exists, so save the word for when you see actual magic; don't waste the word on a metaphor?

Is it me or does anyone else think that a single map function application is a really bad example? I've seen lots of errors introduced by for loops that have been malformed (although modern syntax like for(x:allthese) {stuff} helps ) but also map(stuff:x) means do all these, while for means do them one at a time in sequence

WRT pragmatism, why

    val transformedCollection = collection map transformer

instead of

    val transformedCollection = collection.map(transformer)

?

The scala style guide suggests that the dotless notation be used for pure (side-effect-less) operations.

If there is a language where the style guide is rarely followed, it's Scala.

The dotless notation gets complicated easily the moment you try to extend it past one function call, as you start counting to figure out the order of operations. In fact, even using the shortcut of just using a function name instead of {x=> f(x)} or f(_) is seen as bad form in plenty of places, as knowing if a function has zero parameters, one m ten, or if it's varargs is very important information.

I'd limit dotless to very few cases, like symbolic operators, and very organic DSLs, and even then, it doesn't really work all the time: Just look at ScalaTest: Even Bill's wizardry doesn't let us say that "x should be 2": We have to write "x should be(2)", or "x shouldBe 2".

If it wasn't for symbolic operators, which are useful in some cases, I'd go as far as to not miss the existence of the feature at all.

Agreed the style guide is a guideline at best.

I've moved to the dotless notation because I like that it provides a way to indicate pure code. Postfix operators are poorly supported though.

  (xs map { x => x.key -> x }).toMap

If learning Lua has taught me one thing it is that I hate function calls without parentheses.

Same with Ruby. So hard to understand what is going on - and unlike Scala there is not even any and typechecking to help.

Magic is OK provided that it always delivers what it claims to offer (for any reasonable use case).

Most of the underlying details of the TCP, HTTP and SSL protocol stack are magic to me but it works pretty damn well - I'm really glad that I don't have to understand it fully in order to use it.

I sometime use the term magic to highlight code that is out of the current context/scope.

That ICA, FFT or MLP module is magic...

The module maybe complex but it is not magic. However it is something that I probably don't _need_ to understand to solve the problem I am currently working on.

Now I think about it, normally only very reliable code is called magic - the stuff that always goes wrong has some other choice descriptors.

I foolishly picked C++ for an assignment I did at university once and I summed up the experience as "non-deterministic programming" to a friend. To me, it seemed as if I was ending up with different results when running the same code. Then I suddenly got everything working, and that was "magic". In retrospect, it was a much too hard language for me at the time.

Until I got a grasp of the generated methods in Rails I considered them magical. You try to grep for them and they're nowhere to be found!

What if you are not there to explain your "magic" (which is just another word for clever here)? I get useful abstractions, but if your code has reached a point where you have to explicitly explain what your code does to fellow programmers, then you should probably revisit it and make it better. If your code is clever but not readable, go back and make it readable.

Sounds like you guys just need a in house style guide..it does get weird when I can tell a person wrote this half of a program and then tell someone else wrote the other half due to spacing or bracket returning

That's why when I code something that I know will befuddle readers, I guard it with a warning about dark magic and an ascii wizard.

readability is always more important. The "magic" is just a sign of bad code or code that is too heavily abstracted. why have everybody learn this so called magic when you can be more explicit? this is all religious programming views but i personally am pretty tired of things being way too abstracted. it makes getting shit done take too long.

OP here.

This is what I tried to address. "Too abstracted" is far too subjective. Is "map" on a collection too abstract?

Dont get me wrong, code can be too abstract. But dont mistake too abstract for the unfamiliar.

That's fine as long as you don't try to get away with this for convoluted or spaghetti code.

Reply on news.ycombinator.com