Comment by Radim | Hacker News Reader

Comment by Radim | Hacker News Reader

"Switching from experimenting a lot and thinking a little to experimenting a little and thinking a lot was a key turnaround in productivity. When debugging with long iteration times, you really need to pour time into the hypothesis-forming step - thinking about what all the possibilities are, how likely they seem on their own, and how likely they seem in light of everything you’ve seen so far."

Not to glorify by-gone days, but isn't there a certain charm to having to wait your turn to plug punch cards into a mainframe?

Having a "single shot", with long iteration cycles, indeed does something strange to the way you approach programming and bugs and program design.

Anecdote: my long iteration cycles were caused by my mom restricting access to my C64. I filled page after page (paper!) with code, anxious for any chance to try it. I like to think I learned something during these "mental dry runs"…

On topic: I wrote a rant about the "Mummy Effect" of trying to reproduce ML papers (which we do regularly): https://rare-technologies.com/mummy-effect-bridging-gap-betw...

In my first programming class (1981) - FORTRAN! - we keypunched cards, handed them to a clerk to run, and waited hours for the results. We learned to be careful and to think things through. But nothing like Knuth...

"When I was at Stanford with the AI project [in the late 1960s] one of the things we used to do every Thanksgiving is have a computer programming contest with people on research projects in the Bay area. The prize I think was a turkey.

[John] McCarthy used to make up the problems. The one year that Knuth entered this, he won both the fastest time getting the program running and he also won the fastest execution of the algorithm. He did it on the worst system with remote batch called the Wilbur system. And he basically beat the shit out of everyone.

And they asked him, "How could you possibly do this?" And he answered, "When I learned to program, you were lucky if you got five minutes with the machine a day. If you wanted to get the program going, it just had to be written right. So people just learned to program like it was carving stone. You sort of have to sidle up to it. That's how I learned to program." [0]

[0] http://www.softpanorama.org/People/Knuth/index.shtml

As they say, a few days in the laboratory might save a few hours in the library!

Alternately, a few days of doing might save a few hours of thinking. :-)

The flavour I'd always heard was a day of coding saves an hour of planning

Hamming of the hamming code fame was saying exactly this in one of his lectures. His computer was slow enough that one iteration in finding optimal trajectory for a missile took several minutes, enough to allow him to get insight into what the 'curious' results meant.

(In that particular story, it meant that for a ballistic missile the optimal trajectory was to shoot it straight up and steer it on the way down instead of steering it on an elliptic path from the get go)

Where can we find more about this story?

Look up "you and your research" on youtube. He mentioned that story in a couple of lectures, one time in the one about simulation:

https://youtu.be/O5Ml5kPouG8?list=PL2FF649D0C4407B30&t=688

I recommend the whole course though.

The REPL is often misused and is now a curse.

The idea behind the REPL is for fast iteration in development, but the key thing missing is thinking. Most people write code at the REPL to see what will happen. They don't formulate a hypothesis before hitting the enter key. It's more of a guess driven development methodology, keep trying till it works.

If you want to improve your program experience, slow down on the run/compile cycle. Think it through.

I've been facing similar issues recently and have been slowly drifting towards similar ideas mentioned in the article. Though, coming from a more traditional software dev background, I'm still in the process of internalizing how the popular start-up mantra-'build fast and fail fast' doesn't really work in ML.

Also, logging every experiment is another discipline I'm teaching myself. And I fell in love with org-mode in the process.

When I was doing my PhD, I once resorted to turning on all logging, printing about 60 pages of the resulting output, taping them to the floor of the lab, and crawling around on all fours with a printout of the source code tracing the execution to try and prove to myself that a result I was seeing wasn't due to just a bug in the source code. In an odd way, it's maybe my fondest memory from my dissertation work.

Computers have incredible potential in helping us think about and solve problems. Most programming environments have difficulty in showing up that potential however (they are think to program rather than program to think). Especially in machine learning, it is interesting to wonder what kind of tool on a computer would help us solve problems if it isn’t a programming environment or notebook/playground.

I'd say this is generally good advice in science & engineering. My EE and ME coworkers are forced to do this because the each cycle is so slow and expensive.

Reply on news.ycombinator.com