I have no idea what I'm doing
codon.com
codon.com
If you look closely enough, nobody really knows exactly what they're doing.
When you realize this, you're no longer blinded by the thought that you or other people know everything about something.
Then you also realize that all the stuff that is created by people isn't necessarily how it should be, it's just the outcome of the circumstances that the people were in.
With this realization you allow yourself to think more creatively (outside the box) about things.
However, the author and me don't have any idea what we're doing so I wouldn't take his or my story too seriously.
Really liked this piece and wanted to add something to it that I hope resonates with HN. I have myself run across all of this, particularly the complexity part, and have heard anecdotal evidence from others that I know, that when one finishes a function or a library of moderate complexity, and you step back from it for a while, and you come back to it, it looks really impressive. It's now no longer something that can be grokked at a glance. It looks complicated, with all kinds of persnickety details. You remember the "why" of certain unintuitive parts - evidence of problems you ran into and had to solve. The code is the product of many hours of iteration - and the code expanded, contracted, and settled into the shape it's currently in - a shape which, if presented to your earlier self, you would say "I created that? Gosh, it looks like something a professional programmer might have made." (or it might look like a horrendous hacky mess, of course.)
I have done projects that seemed small, and then by the end they were enormous, but I knew them intimately.
But when I start a project with the idea of it being that big in my head from the start, I'll get stressed and psych myself out, sometimes from doing it at all.
There's something to be said for ignoring the forest for planting the tree, to turn a proverb on it's head.
For example. I was recently trying to figure out how to make an autocomplete list faster, as the database was too slow. I already had a functional test taking in search parameters and returning results. But there would be no way to write a test for the actual implementation of the indexed/faster search classes until I figured out a real approach. In the beginning, I had no idea if this would require a separate daemon process to do indexing, or if it could index realtime, or caching, or memoization, or where it might need to keep an index once created. These are just a couple of the unknowns. Once I figured out a workable concept, I wrote tests for those classes and functions. Then I refactored multiple times, altering the tests accordingly.
You probably even "called your shot" and make guesses about what would be successful before you even approached the problem.
Then you wrote a test and saw whether your guess was right or not. If not, then you misunderstood something and you dug deeper.
Software, in some ways, in simulated science.
I can understand the author's use of the Beaver analogy but it may not be the right analogy.
Beavers build these dams to keep the water level at a certain height. This not only offers protection from predators but also makes it easy to access food.
http://en.wikipedia.org/wiki/Beaver#Dams
They may have poor eyesight but they have a keen sense of hearing.
May be he meant that the Beaver does not know why its teeth grow continuously but it uses it to cut down trees and make dams.
Over millions of years, beavers that behaved like this prospered. The beaver doesn't know why or even think about it. No more than you think about your particular cell enzyme reactions.
We reformed this continent to suit ourselves. Most attempts to protect some bit of land and trees in the name of preservation is ... shortsighted at the least.
We humans just happen to be the ones on top right now doing the unbalanced expansion thing.
That's rich, coming from a human.
Also, from Wikipedia: "the beaver have transformed Alhambra Creek from a trickle into multiple dams and beaver ponds, which in turn, led to the return of steelhead and North American river otter in 2008, and mink in 2009.[43][44]"
"When young are born, they spend their first month in the lodge and their mother is the primary caretaker while their father maintains the territory. In the time after they leave the lodge for the first time, yearlings will help their parents build food caches in the fall and repair dams and lodges." (Source: Wikipedia)
For instance, most spiders build webs, and it would seem they do it by no other mechanism than pure genetic instinct. However, there don't seem to be any external 'cues' as to when the web is finished (in contrast to the water falling cue we assume beavers use). The spider builds the web until it's done building the web. Did the spider intend to build a web, or was it compelled to?
This is an interesting comment I think, because if you take it far enough it ends up having implications for free will.
Consider the exercise described in this excerpt from a Sam Harris lecture on free will. https://www.youtube.com/watch?v=1-Xp7mvOcVM (only 14m long and quite fascinating). I'll briefly describe the gist of it for non-video folks.
Exercise: Think of any city in the world. That's it, just think of the name of a city. No limits or restrictions, any city from any country in the world, just pick one.
Having chosen one, consider WHY you chose the city you did. It might seem like your choice was utterly uncompelled by external factors; you specifically made a reasoned choice, or you engaged your whims, or you just went with the first one that popped into your mind, or whatever. In any case, the city you ultimately chose was completely up to your own free will. Right?
Well, we can quickly begin to narrow down the field of possible choices. For one, it's clear that nobody who did the above exercise thought of a city that they'd never heard of. So for each person, the city they chose necessarily came from the set of cities they'd actually heard of. And even then, some filtering or selection "algorithm" is clearly happening behind the scenes, because there exist cities you've heard of, but which didn't happen to occur to you at all just now.
So some cities would seem to have a higher or lower chance of coming to mind than others, based on factors unique to the individual. For example, a city someone just mentioned 5 minutes ago and you're now thinking of, a city that's been featuring in the news lately, or the cities you've personally lived in, or you made a memorable trip to once, or which are culturally fascinating to you, etc etc.
And then consider the actual process that occurred in your head when you started to choose your city. In general, thoughts and ideas largely just kinda "occur" to you, by some process that's not under your conscious control. So in starting to think of a city, the names of some cities just started popping into your head. But what was controlling which cities were popping into your mind, and which weren't? Obviously you can't have consciously controlled this brainstorming process, because this would require you to have thought of the city before you thought of it.
So you wanted to think of a city, and your brain just started conjuring them into your conscious mind somehow. Since you didn't consciously supervise which cities were popping to mind and which weren't, it's as if your brain is doing something like accessing your memories, experiences, prejudices, preferences, etc; and bringing cities into your mind by some automatic process that's not consciously available to you.
But, what determines the list of cities that someone has heard of in their lifetime, and which have therefore made it into the brain to be available for this exercise? Well, many things that were never under your control to begin with. An obvious example: many people end up living much of their lives in the same city they were born. Which means at least one city is in your brain merely because your parents happened to give birth/raise you there, and the thought of moving away never occurred to you (or it did, but you stayed anyway). You don't choose where you were born.
And as you say, everything works like this. So perhaps you thought of Rome because you had Italian food for lunch yesterday, therefore biasing yourself towards thinking of it. But then why did you make that particular lunch choice? Perhaps a coworker made the lunch decision that day, meaning the reason Rome just occurred to you was because of a decision your coworker made. Where's the free will in that? Or perhaps you did decide lunch on your own that day. How did you go about this? Obviously by wanting to think of lunch options, and your brain started popping ideas into your head somehow....
GP can reason about what he's doing. He knows that he's commenting on HN. He can come up with a plausible reason why (recreation, perhaps). In a more interesting scenario, say building a dam, GP would cut down logs with the express intention of building a dam to make a pond.
The beaver doesn't know what it's doing, literally. It can't tell you why, it doesn't have a greater plan of building a dam, etc.
You're really talking about motivation, and I'll agree that the motivation is roughly the same for GP and the beaver--some mix of chemicals in their respective brains influencing behavior.
That's because beavers don't speak English.
> it doesn't have a greater plan of building a dam, etc.
Just because you've never seen beavers with little tiny HP calculators poring over pages of drawings and equations does not mean that beavers are mindlessly purposelessly building the dams. You think they just throw stick and through some coincidence, a beaver lodge appears? That's crazy. Beavers build those dams with the intent that it will protect them from predators; they intend to dam up creeks in order to submerge the eventual entrance to their lodge. It is deliberate.
Source?
Beavers are best known for their dam-building. They maintain their pond-habitat by reacting quickly to the sound of running water, and damming it up with tree branches and mud. Early ecologists believed that this dam-building was an amazing feat of architectural planning, indicative of the beaver's high intellect. This theory was tested when a recording of running water was played in a field near a beaver pond. Despite the fact that it was on dry land, the beaver covered the tape player with branches and mud.
-- Richard P.B. (1983). "Mechanisms and adaptation in the constructive behaviour of the beaver (C. fiber L.)". Acta Zoologica Fennica 174: 105–108.
>as soon as they had comfortably bestowed Sandy and me on our horse, I lit my pipe. When the first blast of smoke shot out through the bars of my helmet, all those people broke for the woods, and Sandy went over backwards and struck the ground with a dull thud. They thought I was one of those fire-belching dragons they had heard so much about from knights and other professional liars. I had infinite trouble to persuade those people to venture back within explaining distance. Then I told them that this was only a bit of enchantment which would work harm to none but my enemies. And I promised, with my hand on my heart, that if all who felt no enmity toward me would come forward and pass before me they should see that only those who remained behind would be struck dead. The procession moved with a good deal of promptness. There were no casualties to report, for nobody had curiosity enough to remain behind to see what would happen. --Mark Twain, A CONNECTICUT YANKEE IN KING ARTHUR'S COURT
My friend, I think you are implying intention where there is none. It's true that the effect of beaver dam-building is advantageous, but there was never any deliberation about it on the beavers part. It's a curious case of effort without decision. One can easily think of a situation where a beaver might actually do better to not build a dam - but a beaver will try to build one anyway. It's just what he does.
Do humans eat to keep from starving? Do we have sex to have babies? Do we love our babies so that they don't die? I don't think so. We do these things because it's built into us to do them, just like the beaver.
Either way, fascinating stuff.
Now, the fun part, is that dogs, through similar genetic iterations, have evolved to love catching said ball and bringing it back to us.
Life is fun.
The real question is if the beaver realizes this. I can think of situations where humans have done things that I believe it would have been better for them not to do based on my own information which may be different from theirs.
How do you know that?
>One can easily think of a situation where a beaver might actually do better to not build a dam - but a beaver will try to build one anyway. It's just what he does.
A beaver is not an NPC in a video game. It has actual intelligence. Maybe not much, but more than can be modeled in such a trivial way. And humans for our part are quite often found behaving irrationally and against our better interests.
>Do humans eat to keep from starving?
We eat because we're hungry, we farm to keep from starving. Why did we launch a man to visit the moon? Is the fact that the beavers haven't sent their own delegation to the moon evidence of their inferiority? Maybe they just prefer non-interventionist foreign policy.
It is an important and popular fact that things are not always what they seem. For instance, on the planet Earth, man had always assumed that he was more intelligent than dolphins because he had achieved so much-the wheel, New York, wars and so on whilst all the dolphins had ever done was muck about in the water having a good time. But conversely, the dolphins had always believed that they were far more intelligent than man for precisely the same reasons. (23.1)
Shoshin was a great lesson for me. When I was training in martial arts I was told that shoshin was akin to fashioning one's self after an empty vessel. If the vessel is full then it cannot receive any more water. I took it to mean that one must periodically empty their minds of preconceived notions and ideas and be receptive to new ones when seeking new knowledge.
It has also taught me that I have a hard time appreciating what I am capable of and what I do know. I may fight feelings of inadequacy but I must remind myself that my bar is much higher today than it was when I started out. It is good to not feel like you know what you are doing all of the time. It's an opportunity to learn and fill yourself with new knowledge. As long as I know what I don't know I feel that I can find it out and fill the gaps.
I think what makes it difficult is the amount of competition there is these days and how high we have raised the bar. Capital and growth demand experience and knowledge but we need to take time for development too. You may want to hire the best but maybe you need to take some time to help people develop into the best they can be? But we should all take time to be thankful for what we have done and are able to do. It's easy to forget how painful it all once was.
A talk worth watching, in my opinion, if you've ever been concerned about feeling like you don't know what you'd doing.
How poorly does O'Reilly pay their authors? A book the size of most of their tomes, and at the prices they charge, should be a 5-figure advance in any other part of the industry.
And also bear in mind that an advance is just that: it comes off the royalties, so you may not end up making any money for quite a while.
If you want to make significant money as a tech book author, I'm guessing you have to do what Brennan Dunn does, and self-publish your book and market it yourself. I'm guessing that after you've published an O'Reilly book, you could make a pretty nice sum if you do this on your next book.
No complaints though! O'Reilly is a brilliant publisher and I never had any expectation of making money from it. It was just a thrill to be able to get the bloody thing off my chest at last.
e.g. I don't know what I'm doing, but then neither does anyone else but I am stating it so therefore I'm actually better than them and I have impostor syndrome so that actually makes me better at what I do than you, and everyone who says they know what they're doing suffers from the Dunning–Kruger effect, etc.
This has become a bit of a meme -- the whole "I'm better because I claim to be worse" bit.
I get where you are coming from, but I think the best point in this somewhat rambly essay is that it is fine to come out and talk about how to deal with these odd mental effects, especially when they are keeping us from getting work done.
Usually when people begin kicking up a dust, the solution is to stop kicking– in this case I think that means to avoid making claims, and focus instead on doing great work!
When they are sincere, sure. But increasingly they are becoming like the thin girl forever talking about how she feels fat (or, I suppose, the not thin girl declaring that anyone thinner is anorexic). The Dunning–Kruger thing is referenced so frequently on here and other technology sites that it is like the attempt at a great equalizer: People aren't better than me, and the only way they can think they are is the Dunning–Kruger effect.
It is some sort of race to the bottom / lowest common denominator equalization thing. I'm not saying that the feeling of inadequacy at times isn't real, but it is becoming a badge and measure of superiority.
These missives never simply say "sometimes I feel like an impostor". They invariably mention the whole Dunning-Kruger effect, and invariably claim that this self-doubt is because one has achieved such heights of knowledge that they've gained full awareness of all the things they don't know. It always turns this so-called vulnerability into some sort of heightened awareness.
I'm not "attacking" anyone in any case. I'm pointing out that this is a bit of a trend right now to show ones enlightenment by claiming one's vulnerability.
Recently I found this note on my desk at home. It says “I HAVE NO IDEA
WHAT I’M DOING”. I don’t remember writing it, but it was in my house
on my desk in my handwriting, so I obviously did.
So what did it mean? Was it a cry for help? A product idea? A topic
for a blog post? Or could it be an idea for a conference talk?
I’ll begin by showing you some compelling evidence that I have no idea
what I’m doing.
If you only take one thing away from this talk, make it this: beavers are
idiots. They have no idea why they’re building these huge structures; they just
blindly do it.
"One of the primary reasons beavers build dams is to surround their lodge with water for protection from predators." [...] "On land, the beaver's short legs and wide body made them slow and vulnerable to their enemies. However, unlike most of their historic predators, beavers are excellent swimmers. As a result beavers evolved to have a strong preference to remain in or very close to the safety of the water. The need for safety is the primary reason beavers build dams to create ponds." -- http://www.beaversolutions.com/about_beaver_biology.asp
Sounds like they have a pretty good idea why they build dams.
But enough about me. As you’ve seen, I frequently find myself in situations
where I don’t know what I’m doing, and I usually hit a brick wall and
feel disappointed. But it’s not just me! You have no idea what you’re doing
either.
Dunning-Kruger does not explain or justify a total lack of forethought.
All of these animals look remarkably like something in their environment,
but none of them has any meaningful understanding of why they look that way.
They don’t know what they’re imitating. The stick insect doesn’t know anything
about eucalyptus; that’s just what it looks like.
An interviewer asked Richard Feynman to explain how magnetism works. He explained that it was a force that interacted with certain things. The interviewer asked, but why? Richard's answer was a long way of saying "just because, you moron."
You don't need to understand every layer of the onion. Every layer has another layer, and once you understand every single of them, and grok quantum physics and multiple dimensions and relativity and whatever else affects that onion's properties, that won't change that you need to cook that onion until it's soft before you add in meat to get the flavor out of it. If you know how to cook it, you know what you're doing.
It's not that the author has no idea what he's doing. He's just an idiot.
The author of this talk is not abnormal to do things without remembering them. I write things down all the time only to come across them a year later with no recollection.
It must be nice to be so self-assured of one's intellectual superiority though. Most of us have no idea what we're doing, after all.
This is completely different than what the author (and other commenters) are suggesting, which is that one should have a completely in-depth understanding of all aspects of a given idea as far as human culture is capable of understanding.
If you are walking, you know what you are doing. If you are reading, you know what you are doing. If you're commenting on a web forum, you know what you are doing. You are not confused. You know what typing is. You know what the words you want to say are. You know what you intend to say. This is knowing. This is doing. That's it. That is it.
Most of us, except for maybe the mentally handicapped or the catatonic, know what we are doing. There is no big secret or mystery or hidden understanding. It is not correct to suggest that the majority of people do not comprehend the actions they take. Most people comprehend their actions, even if subconsciously in many cases.
If you watch the video, Feynman gets pissed off. He's not pissed off because he can't answer it, because he could just state "I would have to explain all of physics to you for you to be satisfied." He's pissed because this person doesn't even know what they're asking for.
This author doesn't know what he's talking about when he says he doesn't know what we're doing. But that has nothing to do at all with the rest of us.
The best comparison I can make is to say that most of us don't know why we do anything. There is seemingly no purpose to the universe, or why any of it does what it does. Yet there is completely obvious purpose to everything.
I drink because i'm thirsty. Why am I thirsty? Because my body needs to be hydrated. Why hydrated? Because it's made mostly of water. Why water? Continue that logic train until you run out of ways to explain why the universe exists. Nobody needs to know why the universe exists. But we do know when we're thirsty, and why we drink, which is what we're actually doing, so we do know what we're doing, and why.
We do know why we do things, and what we do. Anyone telling you different is selling something.
The author has projected knowledge onto individual beavers, which is...asinine, to use your word.
The beavers don't look around and think "man, my legs are too short for this location. I need a lodge, dam, and pond!" They just do it. They don't "have a pretty good idea why."
The species as a whole has a motivation (survival), but it's a motivation selected by evolution, not by thought on any level.