nonagono

Many of his arguments make “logical” sense, but one way to evaluate them is: would they have applied equally well 5 years ago? and would that have predicted LLMs will never write (average) poetry, or solve math, or answer common-sense questions about the physical world reasonably well? Probably. But turns out scale is all we needed. So yeah, maybe this is the exact point where scale stops working and we need to drastically change architectures. But maybe we just need to keep scaling.

Well clearly not, since at least 100 pages of content here are specifically about differentiable programming and not prerequisites :)

More seriously, it's about doing the impossible. Formally, some functions are nondifferentiable, period. But it would be cool if we could actually "more or less" differentiate them. For that we'll necessarily need a bag of tricks which is now coalescing into "techniques" and "principles".

Cf. numerical analysis. It takes a page or two to set up your definitions and show that many functions are badly conditioned, period. And yet we still want to compute them, so we've been building the bag of tricks for almost a century now.

It’s an introduction to a relatively niche new subfield. If I (an expert in the field but not the subfield) want to learn about differentiable programming, my only option before this monograph was to read through tens of random papers which use different presentation styles, terminology etc. Now I can read through the second half of this, around 100 pages, and jump back to the first half if there’s a prerequisite I don’t know.

That’s how most subfields are born. Assorted papers -> monograph -> textbook. The first arrow is defining the subfield as a discrete topic, which is immensely valuable. Only after you have that you can start optimizing for presentation to nonexperts.

My hypothesis is that Japanese people just appreciate high-quality products more. In the sense that even poor people, which of course exist in Japan, are willing to save up money and occasionally splurge on upscale items, vs poor people in USA who prefer to "average out" their spending and just buy the best that they can afford daily.

Some evidence:

1) Japan has a higher percentage of iPhone users (65.88%) compared to USA (56.74%), while being poorer than USA on most metrics (GDP, median income, PPP-adjusted income etc.)

2) I don't have quantitative data, but just from experience Japan has so many more high-fashion stores, expensive pastry bakeries, expensive candy makers etc, when compared to USA. And if the stores are there, the customers must exist too.

Because people in high executive positions usually have no intrinsic aversion to lying, and in this case, from a cold rationality perspective, lying is obviously more effective than admitting guilt.

(Expanding: people who know the details hate him anyway, and it’s better for people who don’t know the details to hear confusing accusations from both sides than an admission of guilt. Admissions of guilt are very rarely unilaterally effective.)

It's just not worth it. Writing programs in a distributed fashion is hard, setting up compute servers on employee machines is awkward, and few companies actually need to run large compute jobs. So while it's in theory possible, the non-compute costs of doing this would eat up any compute savings.

According to zooko, one of the authors, in new-ish cpus blake3 beats sha256 even with hardware acceleration: https://twitter.com/zooko/status/1419403567320821760