Hacker News Reader

> One of the biggest pons asinorums that people get hung up on is the idea that "it just imitates the data, therefore, it can never be better than the average datapoint (or at least, best datapoint); how could it possibly be better?"

Hmm - the phrasing that perhaps holds more water is that LLMs just imitate the data, which means that novel ideas / code tends to be smashed against the force of averaging when fed into an LLM. E.g. NotebookLM summaries/podcasts are good infotainment but they tend to flatten unconventional paragraphs into platitudes or common wisdom. Obviously this is very subjective and hard to benchmark.

> Obviously this is very subjective and hard to benchmark.

I agree, but it also feels very obvious once you've been exposed to it enough times. The internet is filled of written or spoken AI slop that can generally be spotted with ease by trained eyes and ears.

The problem making a bear-proof trash can is that there's significant overlap between the smartest bears and the dumbest tourists.

> and even a linear model with a few parameters

Using a simple average of past performance to predict future performance is also a technique that is often disturbingly effective vs. standard practice. I suppose technically that is a linear model, but really deserves its own class.

Up to a point where the prediction runs afoul of the time horizon and changing unmodelled circumstances.

They do not have sufficient explicit risk or variance management. Makes them highly fragile. There are more robust versions of the estimators... Still have a problem.

Remember 2008? That market ran on these easy models.

Yes, exponential smoothing in forecasting is another classic example of the robustness of simple models. You can throw all your fancy ARIMAs and Box-Cox transforms at a time-series, and much of the time, it is hard to distinguish from a simple moving average.

Specifically, the Makridakis M forecasting competitions (https://en.wikipedia.org/wiki/Makridakis_Competitions) have shown for a long time that beating the baselines is shockingly difficult.

In fact, classic machine learning only really started to convincingly win with the second-to-last, M5: https://www.sciencedirect.com/science/article/pii/S016920702... ; and neural methods only just sort of began working with the latest one, M6: https://www.sciencedirect.com/science/article/pii/S016920702... . (Possibly with M7 we'll see scaled-up meta-learning Transformers finally start beating the Bayesian or decision-tree forecasters. But I don't know if or when a M7 might be held.)

> pons asinorums

This is a new one for me, so, in the spirit of the article, I will "act in the world to acquire more information as needed".

> An obstacle which will defeat a beginner or foolish person. [from 17th c.]

> From New Latin pons asinorum, from Latin pōns (“bridge”) + genitive plural of asinus (“donkey”). Literally, “bridge of donkeys”.

https://en.wiktionary.org/wiki/pons_asinorum