Callbacks are Pretty Okay
andrewkelley.me
andrewkelley.me
In the examples, each line of code with a named function appears more readable, but this is only because its meaning has become less dense—and as there is also now more code to read, the two factors arguably cancel each other out. Meanwhile the new code has more non-locality: the eye of the reader must jump between the various function invocations and declarations. That's a lot of jumping, and it will get much worse as complexity grows.
Introducing a name has costs as well as benefits. For example, it adds indirection: the name now stands between the reader and the thing that is named. To overcome this, a name should say something about the problem the code is solving. If all it says is something about the code itself, that's not good—it adds weight and obscures the problem (because you're now taking more code to solve it). It feels like you're simplifying code by factoring it in this way, but you may only be spreading it out more, making pieces which are logically bound together further apart from each other.
A sign that this is happening is when you can't think of a good name for the code you are factoring, or when the name refers only to the code itself. "wtfFriendAction" is a comical example, but I think "gotFriends" is more instructive: it tells you nothing more than "I am the callback for getFriends", and thus adds no information to the original code, only verbosity.
And yes it can be hard to name functions, and I think the OP has not created very good example, but a good name that accurately describes what the function does means I don't have to read it if I don't want to.
A better example would have been some node.js code that has 5 levels of nesting, each with a some significant code within.
You've just got to name them.
The trouble with small examples is that their problems don't seem like a big deal because their entire codebase isn't a big deal. You have to apply them many times over to get any sense of their compounding effect, which is what really matters. But by definition, that's out of scope of a discussion whose examples are small. The only way out of this paradox is to pay close attention to apparently small effects.
Functions are good. Short methods are good. Reusability is good. Properly naming your functions is good. Separation of concerns is good.
Complaining about a couple of extra carriage returns and a few extra characters to properly name the method is insanity.
If that function is required multiple places, you make a proper function out of it, and pass that function instead of a lambda.
Callbacks allow for abstractions like map, filter, reduce... Used properly, they can actually increase reusability instead of prohibiting it.
wtfFriendAction is additionally a very, very poor name. It should be called "addFriendToFriendList" or something. Another thing is that the author himself states that the function doesn't actually makes sence, hence the ugly name.
That's the point — using callbacks you end up with functions which do not make sense...
Edit: spelling
Hammering these dogmas harder doesn't make them any truer, it just creates a feeling that you know what you're doing in a "If the English language was good enough for Jesus Christ then it's good enough for the state of Texas" kind of way. We have almost no empirically verified knowledge about software development. One of the few things we do know (sort of—the studies are old and shaky) is that code size is a predictor of error rates. Think that's relevant?
As Clojure, F#, Haskell and friends show, it's better to have 100 functions working on one datastructure, then having ten functions for ten datastructures.
Relevant link: http://www.slideshare.net/gvwilson/bits-of-evidence-2338367
What makes those languages more friendly towards smaller functions has to do with how easy it is to create a new function in those languages. Creating a function in C/C++ for instance, is a pain becouse of the boilerplate involved (header files...).
In the OP's example, he's imposing named functions on himself when he doesn't actually need to. For some reason he also declares functions within the same scope that he calls it, which just makes it cluttered in my oppinion. He would probably be alot more happy with promises.
Code Complete still has many lessons to teach, but it's certainly not gospel in this day and age (unfortunately my copy is buried in some boxes from a move and I can't find it at the mo to re-read that chapter!).
I personally have seen massive changes to the way we code, people just seem to understand how to split out code into more logical chunks than they used to. Even as recently as 8 years ago, when I start professionally, I remember people used to constantly ask how to pass variables by reference, global variables, all sorts of nastiness that meant that methods weren't really being used properly. These days I honestly haven't seen a by ref pass or a global variable for years (apart from in an old, truly terrible VB.Net program I've just been given to maintain & update, ugh).
Programming really has changed massively. And there certainly are fairly famous proponents of short functions out there, Kent Beck, Robert Martin. It's advice that's changed over the last 20 years, but I don't know whether there's studies to back it up, all I can give you is that the general consensus seems to have changed.
Back to the point, I certainly don't hate long methods, what I hate is methods that do many different things. That's just bad code in my book, especially if they start bleeding lines into each other and get jumbled up as tends to happen as your code ages.
I admit it's very easy to go too far down the rabbit hole, with a massive tree of confusingly nested methods and it's a balancing game.
But when you see a long function you can usually tell straight away whether it's justified or not. Is it doing a complicated thing that justifies a hundred or two hundred lines or is it just doing loads of little simple things.
And often if it's doing simple things, invariably other functions in the same code file will also be doing those simple things with tiny variations and it can be refactored to reduce the TLOC and improve maintainability when you need to update that functionality. And that's usually a sign of a programmer how hasn't learnt to do DRY yet and is relatively inexperienced (in the .Net world anyway, I'm not having a dig at all, I appreciate js still massively changes every year).
A function that does one thing and one thing only is:
- generally short
- testable
- name it intelligently and then others don't even have to read the code to know what it does
- easier to spot bugs
- functionality doesn't get mixed up
- reusable code becomes a lot more obvious
Some of these hold especially true with javascript given its lack of block scope.
Long functions are generally a strong code smell, in my experience anyway. They generally contain bad code, regardless of whether they'd have been broken up or not. Functions that start declaring random anonymous objects, which I see a lot in javascript, are especially dubious. It's an artefact of javascript's design that because it's so difficult to declare reusable objects and namespaces and manage code between files a lot of javascript programmers just end up writing these huge blobs of code.
As to where I see this happening in javascript, as an example I was reading Meteor.js's code the other day and was actually surprised at just how bad the code is. Every function seems to do 5 different things, it declares object right in the middle of functions rather than extracting it out, loads of frankly crazy coding practices.
EDIT: In fact, now that I think about it, I remember once using that bit of CC to argue that long functions were better discussing it with a friend. Funny how you can do a complete 180 over time.
This is true only if the function decomposition is intelligently done. The problem is that with callbacks, the function decomposition is forced on you based on what needs to be done asynchronously (i.e., mostly I/O). As a result, it breaks apart the flow of control into little tiny pieces that may be only loosely coupled with the most logical way to decompose functionality.
So your statement may be true in general, but it may not be applicable when it comes to how to best handle callback functions.
As many as needed.
> If I add a dozen, is that better?
If your programs path needs to do 12 things, then yes. Each function should do one thing. If in documenting your function, you use the word and, that's a safe bet that you are actually writing two functions as one.
> Short methods are good—have any evidence for that? I don't think you do, because such evidence as there is points in the opposite direction (look it up in Code Complete)
Unless there is another reference I missed, Code Completes reference referred to no more than a screen's height, which at that time was ~ 25 lines. Assuming not every line is a line of code performing some action, I'd still consider that a small function.
> It's proper to name functions properly? I agree! What's "properly"?
That's hard to answer, which is why so much has been written on it. Couple that with community idiosyncrasies (like Obj-C naming vs Java), you have a lot of "proper" ways to define functions. Within the realm of the community standards, I subscribe to the thought that the method names should tend to be verb oriented. Talk about actions, or perform an action. After all, functions do something.
> We have almost no empirically verified knowledge about software development.
Well, there is quite a lot. Much of it's older (in the history of software).
> One of the few things we do know (sort of—the studies are old and shaky) is that code size is a predictor of error rates. Think that's relevant?
Be careful about this. It's not as open and shut as you'd like to think. Indeed, it's about as verifiable and confirmed as cyclomatic complexity is. Without going into too much detail and sticking strictly the context here: "code size" is an awful metric.
By that notion:
int a = b + c;
int hours = timesPerformed + hoursPerPeformance;
All of those are valid deductions. After all, while it might seem obvious to remove comments, they could cause confusion, right?
And this ignores even the basic question: is a line merely a line on the screen? Or is it a languages statement? A single operation? Would a complex for clause with multiple assignments and statements count as one line?
All this is to say that code size is not the holy grail of error rates.
My own though is that taking CC and LoC and putting them together is best. A function with low complexity and with fewer lines of code is better than a longer function. Many smaller, lower complexity functions are also better than a single highly-complex function that is longer than any single function, but smaller than all the others when combined.
In practice I find I'm a terrible judge at where the function boundaries should be. Several times I've tried to reorganize a project where functions had gotten gradually too big, ended up making them too small and numerous and having a pain debugging, then finally settling on some intermediate position. All the overly-certain talk of "as many functions as you need" misses these nuances.
(I wrote recently about global vs local readability: http://akkartik.name/blog/readable-bad)
So, you think it that code should be measure by the size of the code?
a = b + c; is better than the other?
Care to explain why?
I meant that everytime somebody refers to the value of controlling code size (like gruseom did in this thread), we have to deal with the inevitable strawman of:
int a = b + c;
int hours = timesPerformed + hoursPerPeformance;
(PG has also said this many many times with reference to arc.)
1. A study by Basili and Perricone found that routine size was inversely correlated with errors; as the size of routines increased (up to 200 LOC), the number of errors per LOC decreased (1984).
2. Another study found that routine size was not correlated with errors, even though structural complexity and amount of data were correlated with errors (Shen et al. 1985)
3. A 1986 study found that small routines (32 LOC or fewer) were not correlated with lower cost or fault rate (Card, Church, and Agresti 1986; Card and Glass 1990). The evidence suggested that larger routines (65 LOC or more) were cheaper to develop per LOC.
4. An empirical study of 450 routines found that small routines (those with fewer than 143 source statements, including comments) had 23% more errors per LOC than larger routines (Selby and Basili 1991).
5. A study of upper-level computer-science students found that students' comprehension of a program that was super-modularized into routines about 10 lines long was no better than their comprehension of a program that had no routines at all (Conte, Dunsmore, and Shen 1986). When the program was broken into routines of moderate length (about 25 lines), however, students scored 65% better on a test of comprehension.
6. A recent [sic!] study found that code needed to be changed least when routines averaged 100 to 150 LOC (Lind and Vairavan 1989).
7. In a study of the code for IBM's OS/360 operating system and other systems, the most error-prone routines were those that were larger than 500 LOC. Beyond 500 lines, the error rate tended to be proportional to the size of the routine (Jones 1986a).
8. An empirical study of a 148 KLOC program found that routines with fewer than 143 source statements were 2.4 times less expensive to fix than larger routines (Selby and Basili 1991).
The 25 lines is just one of the studies, related to students of CS in particular. In general, it seems like the data indicates that somewhere between 100 and 200 is optimal for minimum errors per line, lowest cost to modify, less need to modify, and overall undesirability. So the "short, 5-10 line functions" seems to be not only entirely unsupported by empirical data, but actually harmful in general.
Study 2 also supports complexity with errors, not size (small or large).
I hadn't realized how old these studies were, however. For some reason, I thought they were from the 90's. I'd be wary about relying on, what is, frankly outdated assumptions.
You'd be wary, and so you'd replace these studies with what exactly? Which of their assumptions are outdated?
Let's not fall into the trap of thinking that programmers before us didn't understand as much. As far as I can tell, the issues were much the same. A greater problem with older studies is that their sample sizes were mostly small and their experiments never replicated.
I'm going to be frank, in the 80s, programmers sucked compared to today's programmers. If you took an 80s programmer and time shifted him to today, he'd look like he had no experience and really struggle until he'd spent a lot of time learning new stuff.
You've grown up with software engineering in the last 10 years. Have you honestly not seen the sea change in that time?
Be it programming languages that have shifted dramatically from the original OO in Java/C# 1.0, javascript inspiring massive changes in those languages with anonymous types, closures, etc., with unit testing, TDD. There's so much that's changed, so, so much I can't even think of half of it.
I said it below, who today passes ByRef? Or uses a global variable? 10 years ago that was common.
Just 10 years ago! The field is moving so amazingly rapidly!
And one of the real things I've seen, something you can't describe unless you've got experience, which I've no doubt you have, is that methods now actually do what they say they do. They didn't used to. They used to do other things, and touch global vars, and get passed in variables to muck around with.
And on top of that, people now seem to actually understand OO. Not just blindly recanting what polymorphic means, they can actually split up and create objects sensibly. They really couldn't do it very well 10 years ago. No-one really understood it, it sounded like a good idea but only a tiny percentage actually understood what it really meant.
And yet you think studies done back then, using techniques that would get people fired today, are worth referencing?
LOOK at the dates in the parent post. 1986? No Java, no Javascript, no Python, no Ruby, no C#. But COBOL had just released an update!
You'd better watch out for those GOTOs!
I honestly am perplexed that you can even start to think those studies are still relevant today.
You mention a long list of rules and practices, but I'm not sure they've had as unadulterated a benefit as you think. Replacing GOTOs with objects has more often than not resulted in a more structured form of spaghetti (https://en.wikipedia.org/wiki/Yo-yo_problem, for example) that's no better. There were already people programming with closures and anonymous types, there's just more of them now. That's probably better, but not night-and-day. Lots of people still write spaghetti with closures and callbacks.
Arguing about whether the past is better or the present is better is a waste of time. We're just trading generalities at this point about a topic that permits no easy generalization. One way I've been trying to avoid this trap is with the mantra (generality :), "languages don't suck, codebases suck." It forces me to remember that it's possible to write good code in a bad language, and (often) to write bad code in a good language. Intangible abilities, domain properties, accidental design decisions and cultural factors have a huge effect that it's easy to ignore and underestimate by clinging to reassuring rules about the parts that seem more tangible. But you won't grow until you try to stare into the abyss.
(Edit: By 'you' in that last sentence I didn't mean you specifically, mattmanser.)
No, I wasn't claiming that at all. It wasn't the passage of time, but what occurred during that passage of time that mattered.
mattmanser goes into more detail.
Methodologies. Languages. Practices. Tools.
You'd have to assume that nothing has changed in the last 20-30+ years in the world of programming to combat errors and defects, which is decidedly not the case.
This isn't at all to say that programmers before us didn't understand as much. Just they had different limitations.
I'll admit, I've been doing far more reading on the subject because of this thread. I still stand by the concept that a function should do one thing, and only one thing. The size of the function isn't important, but the nature of a function doing one thing generally leads to smaller functions.
I also think duplication should be removed. This also tends to remove excessive code.
On that note, and interesting discussion can be found here:
I did say "specifically" :)
Unless you are going to sit here and suggest that ARC or GC don't help decrease memory errors, or that languages like Python or Java haven't helped move things along. Heck, even C has been steadily improved over the years. Compilers are getting smarter. Tools like valgrind. Agile methodologies, and formalized code testing. Static analysis and even improve code reviews. Heck, even simple things like IDEs and editors.
So much has changed, so much as evolved. Does that mean everything is wrong? No. But relying on studies that can't be replicated and don't account for common programming practices and environments today is dangerous.
Your answer is that "so much has changed", you're "not going to sit here and list everything", and my question is "really just being lazy"? That's a little hard to take seriously.
a function should do one thing, and only one thing.
I'm sure we all agree that functions should be well-defined. However, this is curiously separate from the question of how long they should be, because there are often different possible decompositions of functions into legitimate "one things" at distinct granularities.
For example, I have a rather long function in my code that builds a graph. That's clearly "one thing": you pass it what it needs to build a graph, and it returns the graph. But I could also extract smaller functions from it that also do "one thing"—to add a vertex, say, or establish an edge. In both decompositions—buildGraph by itself vs. buildGraph, addVertex and establishEdge as an ensemble—all the functions do "one thing". It's just that in the first example there is one coarse-grained "one thing", while in the second there are three finer-grained "one thing"s.
So cohesion (a function should do one thing only) doesn't tell us much about whether to prefer longer or shorter functions. All other things being equal, the coarse decomposition is arguably better than the finer-grained one, because it's simpler in two ways: it has fewer parts (1 vs. 3 in the above example), and the total code is smaller (it saves 2 function declarations).
Actually, it's fairly well established. DRY: Don't repeat yourself. Yes, if you need to assign 50 values, and each is unique, than breaking them up might be difficult. However, these aren't the rules. These are the exceptions. Heck, even your graph example could be an exception.
However, in my experience, most code does contain hits on how things can be broken up into smaller, more manageable pieces.
> the coarse decomposition is arguably better than the finer-grained one
=) The finer-grained one is arguably better than the coarse decomposition because it removes repeated lines of code and makes it easier to test the smaller methods. That the total number of lines of code is increase is unimportant, because you rarely look at all the total code in a module. Rather, you are focused on individual components. If the individual component is smaller, it makes it easier to understand.
I appreciate your point of view. And I agree, it's not easy to make things smaller without losing something. However, that doesn't mean we shouldn't try.
Please do go into detail. What is some of this "quite a lot" of knowledge that sheds light on the questions at hand? It would be great to have new references.
"code size" is an awful metric
This has come up on HN before and the evidence does not support what you're saying. The gold standard in recent work [1] found that more complicated metrics have no value beyond simple program length: "After controlling for size none of the metrics we studied were associated with fault-proneness anymore."
it's about as verifiable and confirmed as cyclomatic complexity is
Do you mean they're both awful? Actually, if what you say is literally true, it follows by Occam that cyclomatic measurements are worthless, since they are far more complicated and no more valuable than the most trivial metric there is.
The studies may be weak and often old, but at least there is a body of evidence around this, and the body of evidence points reasonably consistently to the conclusion that code size is (a) the best measurement of complexity that we have, and (b) the best predictor of error rates. (The only other one I know of that has repeatedly been verified is code inspections, and that's a process not a metric.) Contradictory evidence is welcome.
It's true that there's a debate around how best to measure code size. But that's a separate question. That also has received some pretty high-quality discussion on HN in the past--see the link below. It changed my thinking, for one; I highly recommend the comments by anon_d in that thread.
[1] http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.20.2...,
via http://blog.vivekhaldar.com/post/10669678292/size-is-the-bes...,
discussed at https://news.ycombinator.com/item?id=3037293.
As for names, the problem is that he is forced to make a new callback whenever he does an IO operation and then needs to make up a name for it on the spot. Ideally you should never be forced to split code into functions unless you actually want to do so and when you can choose where to create functions its much easier to create good abstractions and good names.
The cool thing about promises is that your methods are pretty portable. For example:
function getUserFriends(userName, next) {
db.users.findOne({name:userName}, foundOne);
function foundOne(err, user) {
if (err != null) return next(err);
db.friends.find({userId:user.id}, foundFriends);
}
function foundFriends(err, friends) {
if (err != null) return next(err);
return next(null, friends);
}
}
var Q = require('q');
function getUserFriends(userName, next) {
var deferred = Q.defer();
db.users.findOne({name:userName}, deferred.makeNodeResolver());
return deferred.promise;
}
function foundOne(user) {
var deferred = Q.defer();
db.friends.find({userId:user.id}, deferred.makeNodeResolver());
return deferred.promise;
}
function foundFriends(friends) { ... }
getUserFriends.then(foundOne).then(foundFriends);
var Q = require('q');
function getUserFriends(userName) {
return Q.nfcall(db.users.findOne, {name:userName});
}
function foundOne(user) {
return Q.nfcall(db.friends.find, {userId:user.id});
}
function foundFriends(friends) { ... }
getUserFriends.then(foundOne).then(foundFriends);The await solution (or any equivalent) has the benefit to present the call as a synchronous call which is trivial to read and understand. It also has the benefit to support all existing and proven error handling methods. try catch, etc.
This is the best solution to asynchronous calls that I have seen so far.
I'm currently using c++ asio, and it's a nightmare.
bla().
then(something).
then(someOther).
catch(ohNoes);
Second, if you try to get better code structure by breaking up your program into named functions ('something', 'someOther' above), you loose locality, i.e. your error handler does not share a scope with its causing code. You can fix that by passing the entire promise to a processing function, but this again obscures things.
Compared to a program that just does:
x := bla();
var y;
try {
y = something(x);
} catch(...) { ... }
return someOther(y);
I'd be hugely in favour of go's approach with very cheap goroutines, so you don't actually have to use CPS just because your runtime isn't clever enough to use more than one thread (ahem node). OTOH go's lack of exceptions kills the whole argument by bringing back the error handling of the 80s :-(
Replacing anonymous functions with named functions works fine unless the anonymous functions are closures. If they are, you're going to have to add parameters to them to pass state in, and you're in real trouble if you want to have them modify closed state, because in JavaScript you've got no pass-by-reference. But closures that modify closed variables are hard to reason about in the usual nested callback syntax model.
What the async 'syntactic sugar' approaches do is replace nested closure scopes with simple control-structure scoping that's easy to reason about - even if the 'callbacks' are modifying state which then gets made available to subsequent callbacks.
The fact is, in JavaScript, lifted functions can close over local variables but the fact that they do is not obvious - in fact it's so counterintuitive that you're generally best to avoid doing so.
In the first example he gives, two functions that, in the original, are nested, in the refactoring are placed in the same scope, rather than nested within one another. His amended solution prevents the second function from closing over the scope of the first, which is possible in the original, or an async form. If you wanted to share state from the first function to the second, you'd have to realize you could bring the second function into the scope of the first, and then move it, and then you're back where you started with nested callbacks. But amending this code is made harder than it should be by the decision to factor the two closures out to the widest possible scope, rather than the narrowest.
And the fact that ctrl-F closure yields no results just confirms the feeling that the examples gave me that the author is playing around with closures without knowing what he's actually doing.
I don't see how that can lead to good code. Better than a million nested callbacks? Sure. Readable? Nope. How do you even name all those callbacks?
Callbacks are in no way "this generation's goto", they do not in any way inhibit the ability to analyse or prove correct a program, and all alternatives to callbacks amount to incredibly fancy syntax sugar for hiding what is really happening behind the scenes anyway.
A callback is a function call, or put another way, a specification of an exact set of arguments grouped in a formal specification that is sent to a deterministic little machine (the function) to execute immediately. On completion the machine will provably produce the result it promised to return in its owner's manual (i.e. the declared return code).
None of this is "goto" in any way. In goto land, without knowing the exact set of inputs a program receives and whole-system simulation, it is utterly impossible to make assumptions about even the most minimal pieces of state in the program at any given moment.
Contrast to a function call. A function call is a fixed guarantee about the state of the system at the point a machine begins executing. Furthermore a function call has a vastly restricted set of valid behaviours: at some point it must terminate, and prior to termination, update the system state (stack) to include its return value. And upon termination, result in the calling machine to resume operation.
All this syntax sugar shit is whack. Great, you typed 24 characters instead of 3 lines to declare a lambda. Hyper productive, go you. How does this progress toward the point where I can say "computer, make me a coffee!"?
If you're genuinely interested in what this generation's Goto might look like, take 30 minutes to watch http://vimeo.com/71278954 .. our UIs are utterly trapped in pre-1970s thinking, our communication networks are so utterly idiotic that we STILL have to write custom code to link disparate chunks of data and logic together, we're STILL writing goddamned function calls in a freaking text editor (something that was solved LONG ago). All the things like this.
I can't ask my computer to make me a cup of coffee, and it responds. I can't describe in restricted English a simple query problem and have the millions of idle machines around the world coordinate meaningfully to answer my question (and our best chances of this.. freebase.com.. got bought up by Google and left to rot as some vestigial appendage of their antispam dept no doubt).
Computing is in a sad state today, but not because of fucking callbacks. It's in a sad state today because we're still thinking about problems on this level at all.
Node.js wasn't an innovation. It was shit that was solved 60 years ago. I wish more people would understand this. Nothing improves if we all start typing 'await' instead of defining callbacks.
Innovation in our industry has been glacial since at least the early 90s.
Edit: and as a final note, I wholeheartedly welcome you to nitpick the hell out my rant, wax verbose on the definitions of provability, the concept of a stack, the use of English versus American language, why Vim is awesome, and in the process once more demonstrate the amoeba-like mentality 90% of our industry is trapped in.
Go and build a programming language your Mom can use. Preferably by talking to her computer. Please, just don't waste it on any more of this insanity.
Question: what exactly would you like your computer--that beige box under your desk--to do, if you ask (or rather, command) it do do that? This sounds more like an embodied intelligence/robotics problem than a problem with constraint-solving goal-directed AI, per se.
Unless you want it to use Mechanical Turk to fire off a task to have a human make you a coffee. I could see that working, but it's probably not what you'd expect to happen.
People who think this is doable with current technology usually don't have much clue in how to build a programming language.
"Computer, create a rule for my car."
"OK. Which car? Do you mean Red Toyota Purchased 1997?"
"Yes."
"OK."
"Computer, when car is at location House, then boil kettle."
"Do you mean Kettle in Kitchen of 48 My Road?"
"Yes."
"OK. Is that all?"
"No. Computer, this rule is only valid on weekdays."
"OK. So when car Red Toyota Purchased 1997 is at location 48 My Road on Monday, Tuesday, Wednesday, Thursday, Friday, boil Kettle in Kitchen of 48 My Road?"
"Yes, but only if my iPhone was not inside the car."
"Do you mean Jemima's Apple iPhone 4Gs?"
"Yes."
"OK."
...
We will get there, but maybe in 5 or 10 years.
I don't see how that view implies ignorance of current technological limitations.
>Computing is in a sad state today, but not because of fucking callbacks. It's in a sad state today because we're still thinking about problems on this level at all.
He seems to imply that it's sad today is today and not 20 years from now. So what?
People have dreamed about replacing "programming" with "dialogue systems" since at least the 60s; they are like flying cars, always 10 or 20 years away. We might be closer now, but it's not like we were not dreaming about this since before I was born.
In the meantime, we have code to write, maybe it's worth doing something about this callback thingy problem just in case the singularity is delayed a bit.
I was responding to your insinuation that OP must not be knowledgeable about language implementation to hold the views he/she does. In this context, technological limitations are irrelevant because that was not the point; the point was an opinion about the relative effort/attention these problems receive.
I'm admittedly not really invested in this area myself so I don't really care, but it's disingenuous to try and discredit OP's views like that. At least this response is more of a direct counter-opinion.
Instead I see these new languages/frameworks which have quite steep learning curves and replace the original implementation with the same amount of code or even more.
As long as 99% of 'real coders' keep saying that (more) code is the way to go, we're not on the right track imho. I have no clue what exactly would happen if you throw Google's core AI team, IBM Watson's core AI team and a few forward thinking programming language researchers (Kay, Edwards, Katayama, ...) in a room for a few years, but I assume we would have something working.
Even if nothing working would come out, we need the research to be done. With the current state we are stuck in this loop of rewriting things into other things, maybe marginally better to the author of the framework/lib/lang and a handful of fans resulting into millions of lines of not very reusable code. Only 'reusable' if you add more glue code than the original code in the first place adding only to the problem.
Trust me, the PhBs would love to get rid of coders, we are hard to hire and retain. This was at the very heart of the "5th gen computing movement" in the 80s and it's massive failure in large part led to the following second "AI winter."
> I have no clue what exactly would happen if you throw Google's core AI team, IBM Watson's core AI team and a few forward thinking programming language researchers (Kay, Edwards, Katayama, ...) in a room for a few years, but I assume we would have something working.
What do you think these teams work on? Toys? My colleague in MSRA is on the bleeding edge of using DNNs in speech recognition, we discuss this all the time (if you want to put me in the forward thinking PL bucket) over lunch...almost daily in fact. There are many more steps between here and there, as with most research.
So you are unhappy with the current state of PL research, I am too, but going back and trying the same big leap that the Japanese tried to do in the 80s is not the answer. There are many cool PL fields we can develop before we get to fully intelligent dialogue systems and singularity. But if you think otherwise, go straight to the "deep learning" field and ignore the PL stuff, if your hunch is right, we won't be relevant anyways. But bring a jacket just in case it gets cold again.
> What do you think these teams work on? Toys? My colleague in MSRA is on the bleeding edge of using DNNs in speech recognition, we discuss this all the time (if you want to put me in the forward thinking PL bucket) over lunch...almost daily in fact. There are many more steps between here and there, as with most research.
No definitely not toys :) But I am/was not aware of them doing software development (language) research. Happy to know people are discussing it during lunch; wish I had lunch companions like that. Also I should've added MS; great PL research there (rise4fun.com).
I wasn't suggesting a big leap; I'm suggesting considerably more research should be put into it. Software, it's code, it's development and bugs are huge issues of our time and I would think it important to put quite a bit more effort in it.
That said; any papers/authors of more cutting edge work in this field?
The last person to take a serious shot at this problem was Hugo Liu at MIT. Alexander Repinning has been looking at conversational programming as a way to improve visual programming experiences; this doesn't include natural language conversation, but the mechanisms are similar.
With some instant feedback a bit like [1] this at least feels feasible.
[1] http://blog.stephenwolfram.com/2010/11/programming-with-natu...
It's not about dreaming. It's about action and attitude. Continuing down the current path of iterating on the existing SW/HW paradigm is necessary, but in that 20 years, it's not going to lead to strong AI. Our narrow minded focus on Von Neumann architecture permeates academia. When I was in college I had a strong background in biology. Even though my CS professor literally wrote a book on AI, he seemed to have a disdain for any biologically inspired techniques.
Recently, I've seen a spark of hope with projects like Stanford's Brains in Silicon and IBM's TrueNorth. If I was back in school, this is where I'd want to be.
http://www.technologyreview.com/news/517876/ibm-scientists-s...
http://en.wikipedia.org/wiki/Fifth_generation_computer
1982 was 30 years ago, BTW.
As 5th generation shows small pockets of researchers haven't forgotten evolution has given each of us a model to follow for making intelligent machines. I hope they continue down this road because faster calculators aren't going to get us there in my lifetime. You feel differently?
As the article mentions "CPU performance quickly pushed through the "obvious" barriers that experts perceived in the 1980s, and the value of parallel computing quickly dropped", where as 30 years later single threaded CPU performance has gone from doubling every 2 years to 5-10% per year since ~2007. Combine that with the needs of big data, and the time is right to reconsider some of these "failures".
Parallel programming is the biggest challenge facing programmers this decade, which is why we get posts like this on callback hell. Isn't it possible that part of the problem lies in decisions made 50 years ago?
I'm not saying we need to start from scratch, but with these new problems we're facing, maybe it's time reconsider some of our assumptions.
"It is the mark of an educated mind to be able to entertain an idea without accepting it." -Aristotle
http://www.extremetech.com/extreme/164050-discovery-may-make...
The schemes that have been successful since then, like MapReduce or GPU computing, have been very pragmatic. It wasn't until very recently that a connection was made between deep learning (Hinton's DNNs) and parallel computing.
The hierarchical model in Hinton's DNNs is a promising approach. My biggest concern is that all of the examples I've seen are built with perceptrons, whose simplicity makes them easy to model but share almost nothing in common with their biological counterparts.
Intelligent machine learning is 5 to 10 years off, just as it has been since the early 1980s.
Of course, now the idea of that sort of blue-sky research is almost universally dead and the most advanced machine learning will likely be developed at Google as a side effect of much effort put into placing more relevant ads in front of us, which is kind of hilarious.
Microsoft even has a pretty slick thing called on{X} that does some amazing tricks with rules reacting to network events and geolocation tools.
Although I confess that commercializing a system that lets you asynchronously start potentially unmonitored exothermic reactions in your home? Probably hard. :D
Eliza ran in 16KB of RAM. The logic flow is easy, all you need is a sufficiently large dictionary.
Being able to communicate effectively, in a bidirectional manner, with a computer is a skill. It's a skill people will probably always have to learn, one way or another. It's more likely that more people will learn these skills than that we'll devise a way for computers to be the smart ones in the conversation any time soon.
It is not hard.
The real barrier to something like that is in finding the right way to infer context for speech recognition and natural language parsing. You could easily bolt a system like that onto IFTTT and Twilio and get good results right out of the gate.
Free idea for you, there. Go make a startup.
It's obvious to me that they do: humans are not that good at keeping track of these fragmented chains of flow control. They're just more balls in the air for the juggler.
In building systems, we are using many layers of abstractions all the time (or "syntax sugar" as you say).
All the other 99% of us need to do is take juggling classes for the next several years and reorient all of our work around juggling. Just quit doing things the way you've done (successfully) for the past several years. Problem solved.
It wasn't solved, it was replaced by a different, and worse, set of problems. Other people having minds organized differently from yours isn't evidence of a usability problem.
The problem with programming isn't the syntax. The problem is teaching a Martian to smoke a cigarette: Giving instructions to something which shares no cultural background with you and has no common sense.
(Of course, the classic 'teach a Martian to smoke a cigarette' problem is rigged, because the students are never told what instructions the Martian can follow; therefore, anything they say will be misinterpreted in various creative ways depending on how funny the teacher is. On the other hand, playing the game fairly by giving the students a list of things the Martian knows how to do would reduce the thought experiment to a tedious engineering problem.)
> Go and build a programming language your Mom can use.
This sexism is worse than anything else in your barely-thought-through rant of a post. The blatant, unexamined sexism in this statement keeps fully half of the potential programmers from picking up a damn keyboard and trying.
I'd throw in 'ageism' in there as well, being someone who is now 'of a certain age'.
Crazy.
Probably not. I'm a left-of-center guy, but I'm so sick of this politically correct bullshit coming out of left field. The complete inability to understand that not everyone is thinking of the broad, social contexts of oppression everytime they utter a statement.
You KNOW WHAT HE MEANT when he said that statement, but in the typical, annoying trait unique to yuppie white assholes, you seek to distance yourself from a heritage of being an oppressor by constantly pointing out racism/sexism/ageism. It's a game, and it doesn't matter whether what you are pointing at is REAL, AND AFFECTS SOMEONE, it just matters that you score your points to prove to the professional victim class that you're not one of the bad guys, even though you look like one.
Please, feel free to protest the JIF peanut butter slogan: "Choosy moms choose JIF". But no, there isn't an organized professional victims organization around single dads, so nobody will be protesting that, because why would you? There are no points to score.
Newsflash: Your parents and grandparents were probably racist bigots like every other cracker in this country. Your game does nothing to help. A woman who is being denied a promotion at work because of her gender will get zero help from your bullshit game. She will instead be hurt by it, because of the "crying wolf" that idiots like you do for your game that causes eye-rolling in 95% of the population.
It seems to be a common tactic to accuse someone of white-guilt to quiet them down.
We're just reinventing the same thing because some sh*ty stuff got momentum. Take websockets, how is now a webserver better then IRCD hacked with an implmentation of fcgi?
PS: vim is awesome :)
1) EVERYTHING is 'syntax sugar'.
2) There's nothing intrinsically wrong with gotos, they just aren't very well suited for human brains. Computers can execute goto statements very efficiently.
Callback-hell smells very much like gotos to me. It's very easy to do the wrong thing and easy to create very hard to read, hard to understand, and hard to maintain code.
The OPs counter argument argument is against treating some syntactic sugar as special and novel when it is in fact not. Even voice commands are syntactic sugar, but they're way better for the lay user.
Although I dunno what he's talking about with "I can't talk to my computer." My phone is learning a lot of tricks, really fast.
> Callback-hell smells very much like gotos to me. It's very easy to do the wrong thing and easy to create very hard to read, hard to understand, and hard to maintain code.
This is mostly because people try to treat higher order functional code as in fact just a fancy syntax. Writing higher-order functional code (code that creates, consumes, and modifies functions) requires a set of alternative disciplines that most people in our industry not only don't learn, but actively despise and in many cases mock.
Even otherwise brilliant, smarter-than-me people I look up to do this and then declare the entire functional concept bankrupt. It drives me nuts. Once upon a time, when people saw OO code then turned their nose and said, "You shouldn't need to define hierarchies and ontologies just to reason about your code! How stupid is that!" But then proven models came out and the industry half-assed adopting it (remember Taligent? No? Look it up).
So many developers these days are whining and grousing about how static type systems inhibit their freedom and higher order functions are whacky propeller-head notions that only nerds take seriously. And yet they wonder why the bulk of the industry moves at a glacial pace. I'm happy that Erlang's nearly-30-year-old proposition and some of the implications Needham's duality are finally reaching mainstream computing.
Well it's a hell of a lot harder to write a good verb than it is to write a good noun. Similarly, writing good adverbs is nearly impossible (how many books on Lisp macros do you know of?). If I had to teach my Mom to code, I wouldn't teach her how to zip, fold, map & reduce lists-of-lists on the fly, I'd teach her the FullName noun.
Callbacks are just gotos that return. They can also pass along non-global context, like error information. It's a subtle distinction, and to be up in arms about it is indeed strange.
The industry moves slowly because they can afford to. A few million dollars can feed a hundred developers. The codebases get so large, the teams so big, that lowest-common-denominator kind of code will always prevail. Remember what I was going to teach my mom? Not lisp macros, no. Simple nouns, simple mechanisms.
> Well it's a hell of a lot harder to write a good verb than it is to write a good noun.
See, that's your indoctrination talking. Really both are about equally hard. The actual definition of zip is pretty simple; assuming you have trained yourself to think about it the right way. This is no different from OO. The idea that imperative programming is "natural" is sort of a myth.
> (how many books on Lisp macros do you know of?)
Quite a few, actually! But I'm not sure why this matters;. Lips macros have 0 to do with not only this conversation, but this entire family of abstractions. Macros bear no resemblance to what we're talking about.
> If I had to teach my Mom to code, I wouldn't teach her how to zip, fold, map & reduce lists-of-lists on the fly, I'd teach her the FullName noun.
Why? People think verb-first all the time, describe things in verb-first ways, and act in verb first ways. They do it all the time, an it's not unnatural.
> Callbacks are just gotos that return.
Not really.
> They can also pass along non-global context, like error information.
If they are implemented with continuations, they do a lot more. But see also coroutines.
> The industry moves slowly because they can afford to.
I submit that the resurgence of the small software shop and the incredible successes that small software startups have been seeing is a counter-argument to this. As backwards as the average Node.js shop is, they're still light-years ahead of the befuddled, ossified monstrosities that they compete with.
> A few million dollars can feed a hundred developers.
You should be ashamed of this remark.
> The codebases get so large, the teams so big, that lowest-common-denominator kind of code will always prevail.
Bridges are not constructed this way.
> Remember what I was going to teach my mom? Not lisp macros, no. Simple nouns, simple mechanisms.
Stop patronizing people. You're pretty smug for someone who doesn't know lisp. I thought being smug was my job as a lisp hacker!
Yes yes yes. I definitely agree. It's "sort of a myth," but it also sort of true. Zip is indeed quite simple, but it wouldn't make any sense at all unless you knew what a list was. Case in point, zipping two lazy (infinite) lists like they're eager won't work at all; each version of a list would have its own zip. The verb will be more or less derived from the noun. A verbless noun makes sense, but a nounless verb? I think there is some dependency.
Ask some programmers if they learned function calls before they learned variable assignment. I'm obviously betting they didn't, but I'd be curious if I were wrong.
I really don't know what's "natural". I'd like to know, but I don't. I do play guitar. (poorly.) Playing a good chord is a lot harder (for a beginner) than playing a good note; I've seen plenty struggle, including myself. Now though, both are about equally hard. I have no preference. The actual structure of a -7 chord is pretty simple once you start to think in the right way. For some reason, though, beginners seem to like playing major chords and single notes. Similar story: when I was a child, I learned how to write the letters before I learned how to write the words. When I was a slightly older child, I learned the chess pieces before I learned the chess openings. It all goes hand in hand, but something's got to come first. I figure it's probably got something to do with how the brain acquires new patterns, but I know even less about neurology than I do programming.
I intended to relate adverbs to lisp macros, but I don't have a rock solid thesis on the matter. Macros can arbitrarily change the nature of functions just like adverbs change the meaning of nouns. In either case, overuse leads to crappy writing. I'd argue they're more awkward to write, but not because of any indoctrination. Just a personal thought.
I'd even bet cash that even a room of grad students could brainstorm/generate new nouns much faster than they can generate adverbs. This might not prove anything.
> You should be ashamed of this remark. > Bridges are not constructed this way. > Stop patronizing people.
I got confused by all this. I didn't intend to patronize anyone. Sorry.
Linguistically inaccurate. But it also seems irrelvant. "Zipping things" is a great example of a placeholder noun, anything that can be zippable, right? People do this all the time. "Driving" implies that you have a thing to drive, but the act of driving is clear and distinct in people's heads despite the fact that it can represent a lot of different actions.
> Ask some programmers if they learned function calls before they learned variable assignment. I'm obviously betting they didn't, but I'd be curious if I were wrong.
The answer to this question is irrelevant, but also hard to understand. Variables and functions are deeply intertwined ideas because most function calls take variables.
SICP taught functions first, and it was widely acclaimed.
> Macros can arbitrarily change the nature of functions just like adverbs change the meaning of nouns.
I do not see an interpretation of macros that is concordant with this metaphor. Macros let you define new parts of speech entirely, hand-tooled to let you perfectly express what you want the way you find most natural.
> I'd even bet cash that even a room of grad students could brainstorm/generate new nouns much faster than they can generate adverbs. This might not prove anything.
I do not think this is relevant. But if you'd like to see an example of how complicated this is, look at any note from one partner to another. Mine go like this: "Dave, Please pick these items on your way home:" and then a list. Which is a function (in IO, so monadic since it has side effects)> But that is a verb THEN a list of nouns.
That's because the problem is analyzing the goto statements, not executing them. And computers suck at it, less than humans, but still suck.
But the industry has no need for that stuff.
It occurs to me that if you're using a whole bunch of anonymous functions with nested lexical closures, then the shared state between them starts looking a lot like global variables.
And globals are evil, but relative to the other nightmares that Gotos could produce, they're pretty mild.
Computing is in a sad state today, but not because of
fucking callbacks. It's in a sad state today because
we're still thinking about problems on this level at all.
The stuff we're talking about here has irked some pretty decent brains enough for them to go out and make ... another textual programming language.
Folks like Musk are an inspiration indeed for boldly thinking at the high level they do and seeing it through. However, that's exactly what's easy for press to write up on. Stuff like the moon speech is what's easy to communicate. It is also simple to attribute some great deed to one of these great "visions". But behind each such vision lies the incremental work of millions - each of it, I say, worth every penny - unsung heroes, all of them.
On a related point, not all the time where we've heard a call to greater action do we see the ability in those folks to imagine how that might come about.
edit: typos.
Like the city you live in right now?
> Folks like Musk are an inspiration indeed for boldly thinking at the high level they do and seeing it through.
"Folks like Musk" are smart enough to know that the progress of humanity, the great and meaningful leaps of technology to strive for, are accomplished by people solving those hard problems under a common banner.
A consequence you've evidently yet to learn, or are afraid to realize.
That's an interesting aspect of their work you point out. It made me think about describing my "banner" and add it to my HN profile.
It would be cool if we can all identify such "banners" under which we work in our HN profiles. Note that the banner doesn't have to be a technological breakthrough. In my case, for example, I wish for cultural impact.
These days these "banners" are LLCs or C-corps. :)
It is still, a nice exercise to do it though ... and I might turn mine into an LLC anyway :)
They are very much analogous to GOTO -- or even COMEFROM.
They have a similar effect to the control flow, and a similar adverse effect on the understanding of the program.
And Dijkstra's core argument against GOTO applies 100%:
"""My second remark is that our intellectual powers are rather geared to master static relations and that our powers to visualize processes evolving in time are relatively poorly developed. For that reason we should do (as wise programmers aware of our limitations) our utmost to shorten the conceptual gap between the static program and the dynamic process, to make the correspondence between the program (spread out in text space) and the process (spread out in time) as trivial as possible."""
>all alternatives to callbacks amount to incredibly fancy syntax sugar for hiding what is really happening behind the scenes anyway.
If, for, while, foreach --heck even map, reduce et co, etc are also syntax sugar for GOTO. Your point? Or adding the weasel word "fancy" somehow makes this particular syntax sugar bad?
Not to mention that there's nothing "incredibly fancy" about await, async, promises et co.
And if I wanted to know "what's really happening behind the scenes" I wouldn't programmer in Javascript in the first place.
I've found that nesting to indicate asynchronous control flow can be quite a good device for keeping the program text closely related to its dynamic execution. (It's true that such code is hard to read in JavaScript, but I don't think that's because of nesting per se.) It allows you to lay out synchronous logic along one dimension (vertically) and asynchronous along another (horizontally). I hypothesize that there's a quite good notation waiting to be brought out there; unfortunately, such experimentation tends to be done only by language designers in the domain of language design, when it really ought to be done in the context of working systems—and that's too hard to be worth the trouble unless you're programming in something like a Lisp.
That's like saying that "by using FOR instead of GOTO we worsen the conceptual gap between the program (in text) and the process (in time)".
Only we don't worsen it -- we just abstract it to a level in which we can reason about it better.
Higher level is not worse -- except if you consider "more distance from what is actually happening at the CPU" as worse. Which is not: historically the higher the distance, the more programmers got done.
We don't need to keep track of the actual low level process in time -- which with callbacks we're forced to. We just need to know how the steps of the process relate. Which await and co offer in a far better form than callbacks.
"Time" here doesn't mean CPU time, it means logical time--the temporal (as distinct from textual) order of operations in the program. To put it another way, the "time" that matters for program clarity is not when stuff happens at lower levels (e.g. when does the OS evict my thread) but what happens when in the source code itself. This is not so far from your phrase, "how the steps in the process relate", so I don't see the disagreement.
I certainly don't agree, though, that the OP's design recommendations lead to higher-level or easier-to-reason-about code. Do you really think they do?
That's a concise way to put it and I'll certainly remember and reuse it! Did you come up with it or did you come across it somewhere?
Unless you imply "goto" can branch conditionally (e.g. je/jne/jg etc. from ASM) but that's not "goto". "Goto" means unconditional jump.
Do you want callbacks? why instead of writing var a = 1 + 2; don't like to write:
var a; sum(1,2, function(result) { a = result; });
Hooray! We've solved the halting problem. Without formal analysis, it's not possible to show (in general) that a function will terminate.
More seriously, I've seen function behavior that's just as bad as other types of flow control (including gotos).
You don't exercise your Big-O thinking here, you exercise your Big-Theta.
1. Avoid nontrivial anonymous functions.
2. Function declarations after the code that actually does things.
I wonder if his advice is good for all functional-style code in non-functional languages.
Are you fucking kidding me?
Personally I agree that not putting too much logic in anonymous functions makes it more readable. I write my own code that way, but some might disagree.
I don't care about whether functions are above or below where they are used, as long as it is consistent - but event then it is a minor thing. And since it isn't enforced by the language (e.g. like in Clojure where you have to def the var/function above the reference) it is going to vary across code bases anyway.
I don't really think either solves callbacks, nor that callbacks is a problem to be solved. Most comes down to pros and cons seen through the filter of your personal preferences.
Sure you can use a language that compiles to javascript. Does the pros outweigh the cons for you? Great, use it/too bad move right along.
Not everything is right or wrong, good or bad - in fact most things aren't.
If one isn't careful, and because JS doesn't have the visual delineator of blocks, instead all these rules you have to memorize, this could easily could lead to bugs, especially in large code-bases.
I think it's good to rely on closures to keep from polluting your scopes and to not use uneccesary function names, which is going to lead to one giant imperative statement (the one with the sequence of commands for the computer to perform) rather than a lot of little functions that you can pass around. Secrets of the JS ninja goes really in depth about this.
Global scope pollution is completely irrelevant here - declarations are available in the surrounding scope, not global. If the surrounding scope is global (which it never is in node.js), then that's your problem.
Also, where did I write anything about global scope? I said it was good to rely on closures to keep from polluting your scopes, and that there was no need for with unnecessary function names. You're literally belittling me out on things I didn't even write. Am I upvote-target practice here or something?
test()
def test():
print "Hello."I like this async syntax but I'm more fond of native JS generator trampoline. I think the JSOperation with dependencies could even use generators in its implementation.
All these solutions coming about after the C# async article are very enlightening to this problem of the synchronic vs diachronic vs asynchronic.
The JSOperations lib is in alpha but can be found here https://github.com/iandrewfuchs/JSOperations/blob/master/REA...
With the state of JS compilers/minifiers and gzip, I really feel that that particular argument can't even be made in most JS situations.
My code becomes cleaner and more maintainable when I am forced to name each chunk of 5-10 lines by putting them into separate functions/methods with defined inputs and outputs. This compartmentalization is naturally encouraged by java, C#, ruby, and python, but Javascript actually seems to discourage it.
The ready availability of anonymous callbacks in JS tends to allow for "function creep", where you blink twice and suddenly you have 50 tangled lines of code without obvious control flow or separation of concerns; sticking to named functions seems like it could definitely help.
Take a look at the following users repos to see exactly how clean javascript code can be.
https://github.com/learnboost https://github.com/component https://github.com/visionmedia
For some people, callbacks take more cognition than other solutions, but for someone who has been using callbacks through and through, this is a natural progression and makes sense. It is more straightforward and you don't have to understand further abstractions like in a compile to JS language or with promises.
https://gist.github.com/radiosilence/2fd82aaf3b1721448ae1
Except your function names make more sense.
Callbacks aren't the problem. People using them are the problem. Keep your logic as flat as possible and callback-hell will dissolves before your eyes.
Programming dogma is so annoying.
How about this then? I haven't tested. Perhaps functions need to be in a different order, but we only have one level of nesting:
Instead of investigating and finding out for sure, I communicated my incomplete knowledge without context. You're right - I should have investigated and then communicated the correct knowledge. Any LiveScript bashing was unintentional.