1. Youthful trial and error is really powerful. I had access to a 286 during my formative years, along with the manual for DOS 4.0. I sat down with that manual and typed in every command to see what it did, until I got to one (I think it was RECOVER?) that overwrote all 30 Mb of the hard drive with garbage files. Then I had to learn how to install an OS. After that, I stopped entering random commands, but fortunately I'd already discovered BASICA and EDLIN.
2. Being able to hold a program in your brain and write it down in a notebook is a huge advantage. I've done all of my best work with pen and paper.
I basically followed the same path you did. Once you have to wipe and reinstall you have gotten to a comfortable level with your systems and tend not to fear making mistakes.
I've got this feeling that we have neural pathways built for typing and, subconsciously, we favor phrases, words, and sentences that are easy to type. I know I even have a tic where if I pause to think I'll insert extraneous commas.
It's kind of like how musicians tend to improvise around licks and scales that they have practiced a lot. There's something to thinking about the idea of a song on paper that is more creative than unintentionally falling back on chords/forms that are easy to play. (Adam Neely discusses this very well here: https://www.youtube.com/watch?v=6c_LeIXrzAk)
I have horrible handwriting and am a VERY slow writer (and my hand gets cramped after writing a few words). I know whenever I have to write something by hand, I tend to choose small words and short sentences. On the other hand, I am a very fast typist and can type faster than I can think in most cases, so I tend to be much more verbose.
Yes, the medium we use matters.
I can definitely type faster than I think, though. So my hands always end up in this kind fo wait() loop while my brain processes. Sometimes, I look back at my previous sentences and realized that my hands kept going for a while before by brain even bothered to stop them. I guess I don't have that problem as much with pens.
Shortly after I took out the 20Mb HDD on my dad's work pc with RECOVER he bought me my own little home computer to wreck instead :)
When Stack Overflow and Google do not help, you start doing it yourself. You use tools like a debugger. If the tools do not help, you can look for other tools. If you cannot find them, you can develop your own. You can look at your code, the code of your operating system, and every library you use. You can trace and observe every instruction a CPU performs.
In my experience there are only two ways to really learn about programming: teaching and debugging.
I'll add curiosity to stubbornness too. I very much have to understand how a system is actually working before I'm willing to call it quits and call something successfully debugged. I've definitely worked with folks in the past who don't have that drive; they're more concerned about making it work than understanding why it does/doesn't work. My revelation last night was "deep in this (internal) library, a URL is being manipulated using a badly conceived regex, and that's why this URL is getting mangled".
The curiosity to understand it on a deep level pretty much defined my academic career too. I'd already been programming (in some capacity) for about a decade by time I turned 18 and finished high school. Computer Science was a pretty obvious career path. But I ended up doing an EE/CS dual major, entirely because I wanted to understand the pieces below software as well. A few years after undergrad, I realized that a lot of people struggled to debug complex "cloud" systems, and went back to do an M.Sc. to study the problem more carefully.
(And, per cup-of-tea's comment, I spent a fair bit of time during my M.Sc. teaching as well, including taking on a Spring & Summer session 200-level CS course as a sessional lecturer)
Where do I sign? When I get really stuck on a bug it's sometimes taken weeks to solve in a never ending game of software dominoes. In very rare cases it ended up being a bug that you could only solve with a soldering iron and some creative rewiring of interrupts.
Bugs can be crazily obscured. To the point where you wish you hadn't started but the sunk cost of time invested is such that there is no way back either. This is one of the reasons I was so happy to switch to Unix (and later Linux), the fact that in most cases the program crashes with a core file and access to post mortem debugging (and in many cases a nice stacktrace of how you got there) made short work of most of the bugs. Of course that means you now only have the hardest categories to deal with.
"All causes shall give way. I am in code
Stepped in so far that, should I wade no more,
Returning were as tedious as go o'er."
My substitute is to externalize my brain: Write things down. Journaling. Blogging.
Debuggers are still useful but I don't actually recall the last time when I hit a bug that required interactive debugging.
Microcontrollers: As a side rant, I'm perpetually grouchy with the fact that Arduinos don't have debugger support. I was using AVRs well before the Arduino was a thing (still have my STK500 kicking around...) and I feel like it's a pretty big lost opportunity for hobbyists to not be able to do a deep dive into what is actually going on for when they want to "level up" their MCU skills. Printfs over serial are great, but they often dork with the timing enough to change the character of bugs; JTAG debugging does too, but you can often insert breakpoints with surgical precision to see what's happening without breaking the timing of what you're trying to debug.
One thing that's been fascinating me this past decade or so is that clearly programming and lots of other 'nerdy' stuff is quite mainstream now, and as a result the 'old-fashioned' concept of a hacker (or the 'hacker ethos' or whatever) seems to have become less of a specific subculture.
But at the same time, I've been trying to do some 'code school' style teaching and I find that there's still a very distinct difference between the coders who mostly just want a job and see coding as a viable path (and even enjoy it, much of the time), and coders who have that same kind of 'hacker' mentality.
It's hard to describe, because it's not just about skill or 'good code' or 'productive results'. I guess it's some degree of unusual passion or excitement about the whole thing. And definitely a tendency to just 'hack it' and coming up with sometimes horrifyingly creative solutions to the problem.
For example, one guy I was teaching was incredibly messy in his code style, undisciplined, and clearly copy/pasting stuff from Stack Overflow on occasion. But when we talked about computers, programming, what kind of cool stuff he'd like to build, he was clearly different from the others. Where the others would sometimes diligently go through the Rails Tutorial chapter by chapter, with him sometimes the only thing that got him to actually learn was to give him something that 1) tickled his particular passions (games), and 2) was problematic enough that he'd be cursing at his computer until well after the 'official' study time.
And while his solution rather predictably was often messy, different from what he'd been taught, and somewhat unconventional, he usually figured it out and there was almost always some 'cleverness' to it.
In case you haven't seen them, the SysInternals tools can serve as an interesting middle ground before popping the DLLs into an assembler. There's one (whose name escapes me) that is basically strace-for-Windows, and I've used it to successfully solve some pretty gnarly Win32 problems before.
I mentioned elsewhere in this thread that I did an EE/CS dual degree after having programmed for a long time beforehand. I really don't understand why, but some of the EE-only software I've worked with has been absolutely atrocious.
One of my favourite moments in undergrad was in my Digital Communications class. I don't remember exactly what we were doing, but it involved Matlab and some kind of signal decoding. I handed in my assignment and got called in to have a one-on-one with the professor; he was certain that I had cheated and just hard-coded the answers to my assignment instead of actually writing the decoder. Why? Because my code ran instantaneously instead of sitting and grinding for 30+ seconds. What was my secret? I'd seen a couple of triply-nested for loops in the code and decided they could all be accomplished with a matrix multiply instead of doing all of that computation by hand.
In the same class, a friend (also an EE/CS dual) and I came up with the dynamic programming solution to the Viterbi decoding problem as well, and again, got taken to task for having a solution that ran too fast. Sigh...
Be warned that FANUC is very litigious about that sort of thing. I can't go into specifics, but they've sued companies out of existence for less.
This is what the previous generation pondered:
I sometimes wonder if growing up before pre-built computers and lots of books about computers was actually a boon for my problem solving skills. I spent so many hours with nothing to go on but some schematics and my own persistent trial-and-error. If I'd done the same stuff these days I don't think I'd have learned half as much (although would I have done more, different things? Probably...)
We all work within a context of abstractions, and in any given generation, poking and prodding at those abstractions and trying to understand what happens below it is immensely valuable and instructive. For one generation, below it might mean pen and pencil, and for the other it might mean looking into what jQuery actually does.
Let's not forget that 'pen and pencil' was once upon a time an abstraction inaccessible to many too!
I guess what I'm saying is that the determined problem-solving mentality that were required before Google/SO are still important, but its possible to learn them even with Google/SO.
It ties into one of the best things about programming / software engineering / software architecture, which is that it's a self-scaling system. Everyone eventually finds the level at which it becomes 'too hard', so it's always a challenge.
We're going to need you to time-box that fix. If you don't have a solution within a day, assign the Jira to Bob so he can take a look at it.
In my current gig, I guess I'm Bob. Whenever someone can't figure out the solution to a problem, it just gets punted over to me to figure out wtf someone did a couple of years ago that makes no sense but coincidentally worked for exactly the problem they had. /rant from last night.