Beyond the Hype of AI: Machine Intelligence and the Pancake Problem

> And then there are people who – for whatever reason – try to sell the idea that malicious AI will take over the world.

Often this is a covert marketing/bragging: "Investors, it is with I sad heart that I inform you our product may simply be too goddamn amazing, and the massive gains you will make from partnering with us may come at the cost of negatively affecting other people you will never meet. However, if we don't do it, someone else will, therefore we must reduce the risk by creating new barriers to unsafe potential competition, for the sake of your investment and/or the children."

> The authors speak specifically about formal reasoning in mathematics, but I think it’s pretty safe to say that there is actually no reasoning go­ing on whatsoever in LLMs and therefore, there’s also no intel­li­gence.

I'm reminded of "Benny's Rules" [0] a study on the design of math curricula, and an eponymous human child whose process was actually about applying textual rules rather than doing the intended math. However, his rules still worked well enough that it gave the illusion of progress.

> Benny believed that the fraction 5/10=1.5 and 400/400=8.00 because he believed the rule was to add the numerator and denominator and then divide by the number represented by the highest place value. Benny was consistent and confident with this rule and it led him to believe things like 4/11=11/4=1.5.

[0] https://blog.mathed.net/2011/07/rysk-erlwangers-bennys-conce...