>I’ve assigned this problem [binary search] in courses at Bell Labs and IBM. Professional programmers had a couple of hours to convert the above description into a program in the language of their choice; a high-level pseudocode was fine. At the end of the specified time, almost all the programmers reported that they had correct code for the task. We would then take thirty minutes to examine their code, which the programmers did with test cases. In several classes and with over a hundred programmers, the results varied little: ninety percent of the programmers found bugs in their programs (and I wasn’t always convinced of the correctness of the code in which no bugs were found).
>I was amazed: given ample time, only about ten percent of professional programmers were able to get this small program right. But they aren’t the only ones to find this task difficult: in the history in Section 6.2.1 of his Sorting and Searching, Knuth points out that while the first binary search was published in 1946, the first published binary search without bugs did not appear until 1962.
The invariants are "tricky", not necessarily hard but also not trivial to where you can convert your intuitive understanding back into code "with your eyes closed". Especially since most implementations you write will only be "subtly flawed" rather than outright broken. Randomizing an array is also one of the algorithms in this class, conceptually easy but most implementations will be "almost right", not actually generating all permutations.
The rest is somehow orthogonal to the LLM and is just pure practice. It is very easy to procrastinate with an LLM without actual practice.
It allowed me to actually see the problem space and something like the "beauty of classical algorithms". It shifted my "unknown unknowns" into "known unknowns". I had failed so many times to achieve exactly that without an LLM in the past.
The best procrastination device ever built because it validates you the entire time. Great post, even beyond the algorithms example.