Things that separate the men from the boys and/or major concepts in software | Hacker News Reader

Things that separate the men from the boys and/or major concepts in software

19 pointshenning40 comments

Here are a handful of things I've had a really tough time with in programming which seems be something you either get or don't, and there's a big difference between those who master it and those who don't. These are in no particular order.

* Recursion and thinking in higher-order functions

* Continuations

* Monads

* mod_rewrite

* Pointers and the process of building a low-level mental model of machine execution in assembler/C

Also there was a huge weedout in my abstract algebra class in college, separating those who could do rigorous proofs from those who couldn't.

"Turtles all the way down" in Smalltalk was a significant moment for me as well.

What other major things like this have you come across?

View on HN

Locality of concerns. I've come to realize that the fundamental issue in programming is that we all look at the universe through finite brain-sized portholes. Higher order languages, structuring systems in abstraction hierarchies, and software architecture research are all attempts to organize arbitrarily large systems so that they can be understood in brain-sized chunks. One might say that well-designed large systems have a sort of fractal nature; it is possible to zoom in and out, and gain useful insight at multiple different levels.

I have a, perhaps strange, opinion that these topics are not that difficult, just difficult to explain and usually explained poorly.

Once you've gained knowledge, it takes a certain type of empathy to explain it effectively. I think this skill occurs less commonly in left dominant math/engineering oriented brains. This is why some programming books read like poorly translated stereo instructions and developer positions requiring customer-service skills tend to pay very well.

This doesn't negate the intelligence or persistence needed to pick these programming skills up in any way--they remain difficult for these reasons.

Actually I still find continuations hard to grasp. I still have to stop and think before writing code involving continuations, the same way I have to stop and think before driving from A to B in Cambridge.

It doesn't feel like continuations match the way the mind works. Ditto for nested backquotes, though I've never been able to think of a better way to do what they do.

I was mainly thinking of difficulty understanding as opposed to difficulty doing.

Our minds aren't impressive short term memory-wise. That might be part of the nested backquotes problem. Memory prodigies usually don't have greater short term memories--they're just better at "chunking" which is really a compression strategy.

Cambridge roads are Escheresque. I've been to Manhattan a few times, but my mental map of Manhattan is better than of Boston, where I live. You gave a talk about compression and beauty, if I remember correctly. Maybe it's all related--beautiful things are more memorable.

I agree with ericb that many apparently difficult concepts are made harder to understand by an abundance of poor explanations. But in the same way, some of them are made harder to understand by poor design. Continuations might be an inconvenient factorization of their underlying concepts.

Monads seem more fundamental to me, in that they capture a broader class of ideas, and seem less machine-oriented. But the overwhelming number of awful monad tutorial blog posts suggests that they aren't any more convenient to think about than continuations.

I think maybe we're lacking a good framework for thinking about concurrency in general. Threads seem simple at first, but make it easy for even very skilled hackers to get tangled up in race conditions and deadlocks. Coroutines are perhaps easier to grasp than continuations, and easier to reason about than threads, but seem less fundamental.

What about multi-threaded programming and atomic operations on shared data? Seems like something many people flub up or aren't even aware of.

How come mod_rewrite is on there? Isn't that basically just regular expressions?

As for a more general suggestion: databases. Seriously. It is embarrassing how many aren't, but just about every programmer these days has to be very comfortable with understanding database querying and optimization, because that's one of those few lines of code which can make or break an entire application.

I totally agree. Regex gets easier and easier as you do more. It's just hard to do in the beginning (personally I started with trial and error).

But definitely, robust database design is among the hardest things I've had to do. I mean, it's one thing to just set up a couple of tables and write queries to get your desired data. It's a whole different ball game to ensure that it'll be scalable, and designing with consideration that you'll eventually have to split your database into shards as it grows.

In a way, it's a subcategory of what I think is the hardest thing, which is designing for scalability. Just because something works with a handful of users doesn't mean it'll work when you have millions of users and hundreds of simultaneous connections, which most of us are striving for.

To me what separates the men from the boys in software development is the ability to get the job done effectively. In my experience, while the boys are playing with programming methods for the sake of it, the men are actually implementing functionality using the techniques that fit the job.

I must admit spending a fair bit of time as a boy though :)

I really agree with this, because I've done so many things just for the heck of it, and for learning new cool things, etc while other big boys/men out there are implementing simple useful things which make them money.

Well, you first have to be a boy to then grow into a man then (makes sense). Because to know what techniques fit the job, you have to know the available techniques, and to know them you have to learn them, and to learn them you either have to know or trust that they are or might be useful.

You can't really know how useful they are before you know how they work, but you won't be bothered to learn how they work if you don't know how useful they are. It's a bootstrapping problem the solution of which is to accept that there's a percentage of new things you learn that will turn out not to be useful to you. So, there's a lot of value in using unfamiliar techniques just for the sake of it.

I agree. Aquiring the right tools is important. As long as you remember that the tools are there to get the job done not the other way round.

Preferably you can do the majority of your 'technique aquisition' on someone else's time so when you are coding in your own startup (or mine) you can focus on the problem at hand. I know I've implemented things before in ways that I now realize were pointlessly clever.

"There are only two hard things in Computer Science: cache invalidation and naming things" -Phil Karlton

In all seriousness, I agree with your list with the exception of mod_rewrite. I'd also add mastery of algorithms and complexity theory to the list.

The notion of pointers is interesting - to me, as a relative novice to programming, pointers seem as the intuitive way of doing things - and not at all hard. What this implies is that different people have very different mental models of the world and how it works. I know people that I would certainly consider smarter than myself who have trouble understanding the concept of pointers - so it's not just about intelligence, though a certain baseline is probably needed.

On the other hand I have serious trouble understanding faith and the thought of a god - this is one of the things that just doesn't seem to fit into my mental model of the world. So understanding certain abstract concepts seems to have some kind of correlation to how you view the world, and what you find to be easy and what you find to be hard.

I'm not quite sure I brought the point across in a coherent manner - but one can hope...

This insightful News.YC thread that I stumbled upon a while back seems to explain C pointers in a different manner better than I have seen elsewhere:

http://news.ycombinator.com/item?id=87348

(Other than the stuff about casting between pointers and ints though.. )

"That was the lightening bolt moment. The problem with pointers in C is that the [star][varname] syntax means two different things, and they are actually contradictory from a certain perspective. In one context it means "declare a variable that is a pointer" and in the other it means "deference". Whenever my languages-brain bit looked at this it couldn't deal with it, recognised it as a contradictory positions and basically shut me down, but not in a way that let me work out what was going on. Have a look at it yourself from that perspective - hopefully you'll see what I always did. Now whenever I look at C code I instead just think "everything is an int and pointers are all about casting ints, and a star when you declare something is your way of telling the compiler that the type of this thing is int-pointer instead of just int". Thus I now have no difficulties at all."

Very well written, thanks for the link.

I never really understood pointers fully until I learned a bit of assembly as part of a computer architecture course. Once you realize that the computer stores everything as numbers, its like: ooooh. Addresses are just numbers written in Hex to scare away people who don't do number-base switching. In face, computers just play around with numbers mostly in different bases to ease playing around with them.

One of the problems I had with them in C was the weird syntaxes. `load` & `store` instructions are much easier to understand.

Where faith is concerned, remember that it is your faith in the US Government (assuming you are in the US) that transforms an elaborate print on paper into money. As it isn't backed by a precious commodity, it is inherently worthless. Collective faith makes the system work.

To me, science and philosophy, the 2 things normally used to refute the idea of God actually prove it better than anything else. Science and philosophy are searches for universal laws and principles. What makes these principles universal, i.e., applicable in all instances? If there were no intelligent design, these laws and principles would perhaps be better described as leaky abstractions, as opposed to concepts which always apply. Think of things like pi, the golden ratio etc, which are so oft-repeated in nature, science and engineering. Seems far-fetched that all this 'just happened.' Supreme intelligence was at work.

BTW, although I am not a Christian, I am the son of a Minister and a Christian woman, so I am biased.

Greetings from Jamaica!

Benchmarking - optimizing performance (at the right time) is one area when junior people never even think of the things that senior people routinely do

Elegance - my definition is Elegance = Functionality / Lines of Code. Sometimes maximizing elegance takes everything you've got. (And dramatically reduces long term maintenance cost.)

Variable naming - (not a joke, although it seems like one). This shows that you really understand what you're doing and makes life easier for those who have to maintain your work. Real programmers never have to protect their jobs.

Structuring of data - both internally and on disk. Streamlining data structures often reduces the need for complex processing. Eliminating the need to be elegant is often more elegant than being elegant.

Macros. Especially macro-defining macros.

1. Maturity. The ability to not whine about every little problem one runs into, but Google it down quickly and fix it right away. This is about persistence. I find it difficult to work with people I have to motivate at every step of the way through a project.

2. Holding a program in one's head. I'm not sure really whether to attribute this to PG or Peter Norvig, or somebody else. The best way to understand this is really to experience it in practice. Everyone finds it difficult to keep a mental image of a project once it starts to grow, but it seems like some programmers have an especially hard time with this.

Oh, and I'd add cryptography because it's difficult enough that you probably shouldn't write your own unless you've been formally trained and have put some time in as a practitioner.

Implementing cryptography isn't that hard, the algorithms are fairly simple. You can write RSA for example in a couple of hours if you already have BigInt implemented. The hard part is the math behind the algorithms that makes them work and be secure, but thats what we have Schneier for...

Although I wouldn't recommend implementing them yourself, unless you have a very good reason for it, as Linus once said "Given sufficient eyeballs all bugs are shallow", in Cryptography that is doubly important.

Of course implementing someone else's algorithm is fairly easy. Coming up with the next RSA, though, is very difficult.

Actually, I would say that the math behind the algorithms is not the hard part (someone else already did the hard work, as you mention); the hard part about crypto is learning all of the subtle ways that you can screw something up. This is the reason you should never implement your own crypto code -- by definition it will suck for a long, long time.

Why are monads on the list?

FWIW I've never read a clear and simple definition of a monad (that more than one person agreed upon). The word "monad" appears to be very nearly replaceable by the word "opinion" with all ensuing ramifications. Perhaps someone can provide a clear definition of "monad" here!-))

But seriously, I can't help but think that monads occupy a position similiar to Tait's quaternions: useful to some but not popular enough to achieve a critical level of adoption.

all that it takes to get a concept is time and effort and nothing more. do enough examples and exercises related to a concept and you get a pretty good model in your head of what the concept is actually about.

When you say mod_rewrite, you refer to mod_rewrite or to regular expressions?

The ability to quickly learn new things like monads and mod_rewrite.

Parsing and translation.

really grok source control, merging / it's not a windows share with revisions on top of it.

Please explain source control, because I still don't understand it. I don't understand how a source control system could do merging without knowing about the format of the files. It's currently based on a string merge, but this doesn't seem principled and it's not correct in many cases, or is it? On the other hand it often works in practice.

Category theory in general, because it encompasses areas of computer science, mathematics, and physics.

Why should someone who is interested in programming languages learn category theory and what is the best (preferably online) text?

A Gentle Introduction to Category Theory looks pretty good: http://wwwhome.cs.utwente.nl/~fokkinga/mmf92b.html

After having read a few pages category theory somehow seems similar to OOP. As with OOP you package data with its operations (morphisms).

> Quoting Asperti and Longo [1]: “The basic premise of category theory is that every kind of mathematically structured object comes equipped with a notion of [ . . . ] transformation, called ‘morphism’, that preserves the structure of the object.” By means of the categorical language one cannot express properties of the internal structure of objects. The internal structure of objects is accessible only externally through the morphisms between the objects.

What is described here sounds a lot like information hiding/encapsulation. Is category theory similar to OOP in this way?

Category theory is important to type theory and semantics (Pierce, see below, also wrote several books on types in programming languages) and offers a different representation of computational flow (monads, arrows).

Benjamin Pierce's "Basic Category Theory for Computer Scientists" (not online) is terse and complete [1].

Wikipedia is good for definitions and there's also a Wikibook on category theory: http://en.wikibooks.org/wiki/Category_theory

There are some brief, informative, and fun lectures on YouTube by the Catsters: http://www.youtube.com/profile_play_list?user=TheCatsters

The natural transformations playlist is where to start, just be ready to use the aforementioned references.

[1] Natural transformations are the original point of category theory. A text that doesn't cover them is incomplete.

Thank you and thanks neilc. What is the right order to watch the lectures by the Catsters? Top-to bottom as listed on that page?

Are there any data structures based on category theory, like sets & graphs which are inspired by mathematical theories?

Category theory shares a lot with set theory, but they are different - they make different assumptions and each is better suited at describing certain types of systems. Graph theory is also its own topic in math, though sets can be used to talk about them

As for the Catsters' videos, I'm unsure about the best order. The recommendation to start with the natural transformations playlist came from someone in #haskell (Cale, maybe?). I'd suggest diving in to whatever sounds interesting and backing up to review other videos that contain any new terms.

Reply on news.ycombinator.com