gwern

Stross just made that up, as pure post hoc ergo propter hoc. There are no sources, and he got it from Hannu: https://gwern.net/review/quantum-thief#fn2 Stross has chosen to never revisit the topic to try to substantiate his suggestion.

This quote winds up being rather exemplary: for example, that one parenthetical description manages to make at least 3 errors: 1. Fyodorov was born in 1823, so he obviously could not have invented anything in the '18th century' (ie. 1700s); 2. Cosmism included many things, not just the 'Great Common Task', and the Great Common Task itself went far beyond reviving ancestors, including many overall more important things like colonizing the entire universe or conquering death; 3. and further, the revival part was not about computer simulation at all (that's Hannu's _Quantum Thief_ fictional version of the idea that he came up with for his Sobornosts!) but reviving them physically, in the body, possibly using cloning - and was no more about "inside of a simulation" than Jesus reviving the dead was.

You're right that Hannu made great use of Cosmism as world-building in the Quantum Thief trilogy which I highly recommend (see my review above) - but that could only work because the ideas of Cosmism are so novel & exotic, and not part of Western transhumanism. If they really were as foundational as Stross claims, the 'taproot' of Western ideas, they would make about as exciting fictional worldbuilding as suggesting that you have some sort of 'laws' for AIs, starting with 'An AI may not injure a human being or, through inaction, allow a human being to come to harm."...

So I am suspicious something else is going on to produce the overall uncanny effect of wrongness: possibly they used Photoshop to paste on a bunch of unrelated images to some low-quality stock-art backgrounds?

Specifically, the Makridakis M forecasting competitions (https://en.wikipedia.org/wiki/Makridakis_Competitions) have shown for a long time that beating the baselines is shockingly difficult.

In fact, classic machine learning only really started to convincingly win with the second-to-last, M5: https://www.sciencedirect.com/science/article/pii/S016920702... ; and neural methods only just sort of began working with the latest one, M6: https://www.sciencedirect.com/science/article/pii/S016920702... . (Possibly with M7 we'll see scaled-up meta-learning Transformers finally start beating the Bayesian or decision-tree forecasters. But I don't know if or when a M7 might be held.)

By the way, note that this applies to LLMs too. One of the biggest pons asinorums that people get hung up on is the idea that "it just imitates the data, therefore, it can never be better than the average datapoint (or at least, best datapoint); how could it possibly be better?"

Well, we know from a long history that this is not that hard: humans make random errors all the time, and even a linear model with a few parameters or a little flowchart can outperform them. So it shouldn't be surprising or a mystery if some much more complicated AI system could too.

I was just linked, by an intelligent, educated, skeptical person who regularly uses AI generative tools, to an interesting report on the darknet markets (which I used to be an expert on): https://tornews.org/legendary-darknet-vendor-crimsonlotus-ta... I was not familiar with this site, but I figured it was probably one of many successors to DeepDotWeb, and I've been out of the scene for a long time and wouldn't know about it; but I eventually realized, after several minutes of reading and then checking, that this was an AI slop website. This is because 'Obsidian Bay' doesn't appear to be real, the photo is obviously AI-generated, and the closing commentary by 'Ella Vargas, darknet researcher at the Cyber Crimes Institute' appears to be a fake person at a non-existent organization. Is anything in the article real? Probably not.

Ahhhhh, but you say you don't care about it being real, you say you only care about the prompt - in which case you're fine: some things in it are old events relabeled. (eg. there really was a large darknet market which was famous and had eluded LE and was taken down in a global operation in part due to packet timing attacks! You did learn something true from that accurate prompt/input yielding that slop output! It was just called 'Silk Road 2' and that happened over a decade ago.) So, I guess you have no issue with tornews.org. Maybe you should subscribe. All these articles sound quite exciting and there's a lot of them...

Personally, I think it's a bad site. I could have fallen for it - I simply got lucky that this one was so easy to note the style, factcheck, and debunk. For example, I could have instead read https://tornews.org/massive-dark-web-drug-operation-busted-i... : if you read this, this sounds very plausible and exciting, and it doesn't immediately come off as AI slop because it is so detailed and doesn't sound too much like 4o.

This, it turns out, is because it's based on a real bust: https://www.suffolkcountyda.org/suffolk-county-district-atto... https://www.newsday.com/long-island/fentanyl-bust-suffolk-k7... Heavily rewritten to obscure the sources being plagiarized. How much of it is real? Well, the assertions I spotchecked seemed to be real (ie. copied without too much distortion from the police press conference)... but I have no idea about the rest of it, and this article lends credibility to the other articles.

(Why does this week-old site exist at all and even has a Twitter account? https://x.com/tornews_org It's almost certainly either for the affiliate marketing revenue, in the best-case scenario, or is a phishing scam, in the worst-case scenario. Since it's so young and so reliant on fake content, likely the latter. I didn't check any of the onion links they so helpfully provide, but even if they are all legit right now, that simply means they are not yet scamming. To pay for the AI and domain name and labor here, you only need to phish one DNM user who will deposit a few hundred dollars of cryptocurrency. And this site has already started to pollute Google with its 'facts'.)

If this is really your response, I agree with your original comment about you being burned-out.

Because I routinely catch people, often very intelligent and educated people, confidently posting LLM materials that they have not factchecked and which are wrong (such as, say, about history, where they might make an argument about human evolution based on cave-dwelling where 4o was merely off by a few hundred thousand years), and I still find confabulations in my own use of even frontier models like o1-pro or Gemini-2.5-pro (leaving aside o3's infamous level of crimes, which I think is probably unrepresentative of reasoning models and idiosyncratic to it).

And in this case, the prompt looks like it was very low quality. The author had plenty of chances to put in details from his own real experience as an actual Iranian - any intelligent, observant college undergrad routinely buying coffee ought to have plenty to say - and instead, it's super-vague waffle that... well... an LLM could have written about just about any country, swapping out a few clauses. ("Why drinking coffee in America has become so complicated")

> You're clearly allergic to basic LLM-style

This is not a 'basic' LLM style. It is a very specific, recent chatbot style. (Note that visarga was able to instantly tell it was the recent 4o, because that style is so distinct compared to the previous 4o - never mind Claude-3, Gemini, Llama, Grok etc.)

Further, there should be no single 'LLM-style'; it makes me sad how much LLM writing capability has been collapsed and degraded by RLHF tuning. Even my char-RNN outputs from 2015, never mind GPT-3-base in 2020, showed more occasional sparks of flair than a 2023 ChatGPT did.

> or at least the masquerading of LLM text as human, so I'm curious what you'd consider worse: 1. LLM-generated text reflecting an accurate prompt/input, or 2. genuine human BS wanting to be taken seriously? (e.g. The Areas of My Expertise by John Hodgman if it wasn't in jest)

#1 is worse (if unedited/factchecked/improved etc and just dumped out raw), because there will be much more of it and the intermingling of fact and fiction makes it harder to factcheck, harder to screen out of future training corpuses, and overall more insidious. Human BS serves as costly proof-of-work and because it is costly, once you recognize you are reading BS from someone like Elon Musk or Sam Altman, you can switch modes and ignore the factual content and ask, 'why is he writing this? what purpose does this BS serve? who is the audience here and how are they using it?' and get something quite useful out of it. I have learned a lot from statements by humans where little or none of it was factually true. Whereas a LLM output may mean nothing more than 'some unattended code spent $0.0001 to spam social media with outputs from a canned prompt', if even that.

...It is in a format that resembles a published article because it is going to be a published article? "This is a preprint of a chapter that will appear in the book Designing an Intelligence, published by MIT Press." on the first page.

> As for the graph, it’s too generic

A history of RL from DQN to AlphaProof/LLM computer use in Gemini is not 'generic', and could not be.

> it doesn’t provide any real value

It provides value to people who were not around then and not familiar with how RL attention peaks and crests, and a similar chart about TD-Gammon and Deep Blue, say, would likewise be useful for the many people who did not actually live through those eras, and helps contextualize material from back then. (I did, and maybe you did, and so it's not useful to us, but there exist other, younger people in the world, who are not us{{citation needed}}.) And the fact that these cycles exist is something worth reflecting on - Karpathy and others have reflected on how there were expectations of DRL leading to AGI in the 2015-2020 period, which wound up being swamped by self-supervised learning and DRL relegated to a backwater (and contributed very directly to many major events like how OA and DM became like they are now - and why Sutton is at Keen rather than DM with Silver), but now suddenly becoming super-relevant again.

No, I didn't learn anything from the article (except how susceptible HN has become to even 4o-level LLM outputs, of course). I learned something from your comment, and you learned that from a Google search. Do you see the difference?

> So the actual content of the article is perfectly plausible.

So in other words... it added nothing to even a superficial familiarity with the topic.

> Seems like the same tired take on third wave coffee, without much specific to Iran.

> This article is a bit whiny about the new, and doesn't talk enough about the qualities of the old.

> Funny how even a repressive theocracy can seem so familiar. I guess that's globalization for you.

> One thing the author fails to take note of here is that Iranians have historically been extremely precious about their bougie little drinks.

> This is a poor take on what's actually a rich cultural shift towards variety seeking.

:thinking_face:

Look at the author's previous blog posts: low effort, not even correctly spelled or written, like https://adelbordbari.github.io/album/2025-1-25-the-horror-an... (as expected from an Iranian ESL) - and then this one is suddenly boom: perfectly spelled, em-dashes all over, and where did all these <br> come from? Who writes a Github Pages Markdown Jekyll post with a bunch of <br>s in it...? An LLM, obviously.

https://adelbordbari.github.io/about/

> I’m studying artificial intelligence at university...I’m busy with my storybook that’s supposed to be published by June 2024 as well.

As opposed to all the other Apple decisions?

By the way, which law requires Apple to provide geofencing to a health insurance app such it can't even be installed outside a particular country, and ties its hands completely on the matter?

This sounds like a verbose way of saying "this is really the fault of Apple".

I would bet Roser is wrong here and he can't provide a reference for his claims here. I'd predict that they do decline exponentially, it is just that (1) we have already produced a huge number of them so it's hard to observe the next doubling of cumulative production, and (2) the learning rate exponent is a lot worse than in the more obvious examples like solar panels or deep learning.

Does this really show much that https://arxiv.org/abs/2301.03728#facebook (uncited) and other earlier work did not?

However, the stochastic parrot arguments (and Gary Marcus in many of his writings) made specific, unambiguous empirical predictions about how LLMs would never be able to do many things, such as 'add numbers'. For example, in the original Bender & Koller 2020 paper "Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data" (which lays out the core ideas which are the justification for the 2021 paper that actually introduced the rhetoric of 'stochastic parrot') made many clear, falsifiable statements about what LLMs would never be able to do; here's one of them: https://aclanthology.org/2020.acl-main.463.pdf#page=14

> To get a sense of how existing LMs might do at such a task, we let GPT-2 complete the simple arithmetic problem 'Three plus five equals'. The five responses below, created in the same way as above, show that this problem is beyond the current capability of GPT-2, and, we would argue, any pure LM.

One could say many things about this claim, but not that it is a 'strictly philosophical one'.

I'm still not quite sure how to think of this. Maybe as being like unrolling a diffusion model, the equivalent of BPTT for RNNs?

Forty-six percent of the global population has never hired a human programmer either because a good human programmer costs more than $5 a day{{citation needed}}.

Like that gorgeous orange (letterpress, I assume) manicule you see in the middle of OP - that line-work will never work when scaled down and put into a web page as a normal little manicule. The details get lost in a blur, the hand becomes a solid blob of black, it looks goofy and out of place (note: for the mouse pointer, this is a feature, not a bug), or it ceases to look like a hand at all to someone who isn't primed for it by an article.

I can tell you that because I tried one that looked a lot like that. I went through literally hundreds of manicules from Wikimedia Commons, Font Awesome, Google Images, and articles like this, trying dozens of them on my actual website, and it took that long to find a single one I was at all happy with in context. I was shocked. No other icon I use on my site was that hard to find an adequate version of. (We really wanted a manicule for our use-case, so we persevered.)