How do these kinds of advancements in math happen? Is it a momentary spark of insight after thinking deeply about the problem for 20 years? Or is it more like brute forcing your way to a solution by trying everything?
Yup, I think stubbornness/perseverance is the most useful transferrable skill I got from doing research math. It's a double-edged sword though as I often just can't give up working on something when I really should in my tech job.
You're really standing on the shoulders of giants when you rely on the classification of finite simple groups.
Giants whose work has (dirty little secret) never truly been verified. The proof totals about 10,000 pages. At the end of the effort to prove it there were lots of very long papers, with a shrinking pool of experts reviewing them. There have been efforts to reprove it with a more easily verified proof, but they've gone nowhere.
Hopefully, the growing ease of formalization will lead to a verification some day. But even optimistically that is still a few years out.
My understanding was that the so called "second generation proof" of the classification of finite simple groups led by Gorenstein, Lyons, Solomon has been progressing slowly but steadily, and only the quasithin case had a significant (but now fixed) hole. Are there other significant gaps that aren't as well known?
Still they have a couple more books of proof left, and I have to wonder how carefully it will be reviewed. This will still be a massive improvement, but I'd be a lot happier if the entire proof could be formalized.
Plus there is still a possibility that there proves to be another significant hole.
If any theorem needs to be formalized, this is the one. No other theorem is this big, this hard to prove, and this important to get right.
It's happened several times that I struggled with a bug for hours, then suddenly came up with a key insight during the commute home (or while taking a shower or whatnot), while not actively thinking about the problem. I can't explain this any other way than some kind of subconscious "brainstorming" taking place.
As a side note, this doesn't mean the hours of conciously struggling with the problem were a waste. I bet that this period of focus on the problem is what allows for the later insights to happen. Whether it's data gathering that alows the insights to happen, or even just giving importance to the problem by focusing on it. Most likely it's both.
(n.b. historically speaking in the literature, "subconscious" was the word used to describe Jung's "woo-woo" ideas about a collective subconscious shared across populations, and unconscious was the word used for ideas about a single brain a la Freud which is what we are talking about here)
For me, the subconscious is the wellspring of sudden insights.
I believe this is apocryphal. Watson likely said this because he stole Rosalind Franklin's research.
This is probably something I need to research more, because the questions scientists are dealing with are too deep to ignore the one big question, and I wonder who struggled with it.
Mind you he was helped by a nice coincidence. Franklin knew all 230 space symmetry groups, and so had to sort through them. Watson only really knew one - his PhD thesis was on a protein with the same group as DNA.