Comment by ben_w | Hacker News Reader

Comment by ben_w | Hacker News Reader

Parent Full thread

> "[To Lex Fridman:] We don't want our next model to be shockingly good" (Really ?)

Yes, really.

They're strongly influenced by Yudkowsky constantly telling everyone who will listen that we only get one chance to make a friendly ASI, and that we don't have the faintest idea what friendly even means yet.

While you may disagree with, perhaps even mock, Yudkowsky — FWIW, I am significantly more optimistic than Yudkowsky on several different axies of AI safety, so while his P(doom) is close to 1, mine is around 0.05 — this is consistent with their initial attempt to not release the GPT-2 weights at all, to experiment with RLHF in the first place, to red-team GPT-4 before release, asking for models at or above GPT-4 level to be restricted by law, and their "superalignment" project: https://openai.com/blog/introducing-superalignment

If OpenAI produces a model which "shocks" people with how good it is, that drives the exact race dynamics which they (and many of the people they respect) have repeatedly said would be bad.

influence from yudkowsky is surprising, considering if you've ever touched hpmor, you'd realize the dude is a moron.

re p(doom): the latest slatestarcodex[0] has a great little blurb about the difficulties of applying bayes to hard problems because there's too many different underlying priors which perturb the final number, so you end up fudging it until it matches your intuition.

[0] https://www.astralcodexten.com/p/practically-a-book-review-r...

I find it curious how many people severely downrate the intelligence of others: to even write a coherent text the length of HPMOR — regardless of how you feel about the plot points or if you regard it as derivative because of, well, being self-insert Marty Stu/Harry Potter fanfic[0] — requires one to be significantly higher than "moron", or even "normal", level.

Edit: I can't quite put this into a coherent form, but this vibes with Gell-Mann amnesia, with the way LLMs are dismissed, and with how G W Bush was seen (in the UK at least).

Ironically, a similar (though not identical) point about Bayes was made by one of the characters in HPMOR…

[0] Also the whole bit of HPMOR in Azkaban grated on me for some reason, and I also think Yudkowsky re-designed Dementors due to wildly failing to understand how depression works; however I'm now getting wildly side-tracked…

Oh, and in case you were wondering, my long-term habit of footnotes is somewhat inspired by a different fantasy author, one who put the "anthro" into "anthropomorphic personification of death". I miss Pratchett.

If you think that volume is a replacement for quality, you really need to read more. If you think volume is correlated with intelligence, you really need to read and write more.

In case you still don't believe me, you are welcome to hop onto your favorite fan fiction site, such as AO3 [0], and search for stories over [large k] words.

[0] https://archiveofourown.org/works/search?work_search%5Bquery...

That volume coherently. Obviously mere volume can be done by a literal monkey on a literal typewriter.

Also, I didn't say "correlated with intelligence", what I said was more of a cut-off threshold — asserting that one cannot be an actual moron given writing coherently on that scale is more of a boolean than a correlation.

I do need to write more (my own attempt at a novel has been stalled at "why can't I ever be satisfied with how I lead into the dramatic showdown!" for some years now, as none of my attempts have pleased me once written); but as for reading? Well, if you think my taste must result from insufficient breadth and depth, I must wonder what you think of Arthur C. Clarke, Francis Bacon, Larry Niven, Alexandre Dumas, Adrian Tchaikovsky, Neal Stephenson, Robert Heinlein, Alastair Reynolds, Isaac Asimov, Carl Jung, … I'm going to stop there rather than enumerate my whole library, but I can critique each in a different way without resorting to calling them playground insults, even the ones I dislike.

But I will say it was interesting to contrast Chris Hadfield's "An Astronaut's Guide to Life on Earth" with Richard Wiseman's "Shoot for the Moon" — or H. G. Wells with Jules Verne.

It's bad writing. It's objectively terrible writing in fact. Purely on a volume standpoint, the average novel is 100K words. The Brothers K is HALF the length.

If we ignore that it's a rewriting of JKR's 7 novel series, which gives it a certain amount of coherency, Yudkowsky violates almost every writing guideline in a bad way. In fact, I could probably write an infinitely long coherent essay describing the ways HPMOR violates a reader's mind. It would be easy given almost 700K source material.

But to point at some gaping holes, the plot has no pacing, and the entire story is a badly written self insert where the mc goes around and "fixes" JKR's plot-holes by writing themself into a corner.

The solution to this, is, of course, to write another 20K words of expecto patronum, dispel the plothole with more rationalist bullshit.

Your average fanfic is probably better written.

> It's bad writing. It's objectively terrible writing in fact. Purely on a volume standpoint, the average novel is 100K words. The Brothers K is HALF the length.

I infer you favour Blaise Pascal: "I'm sorry I wrote you such a long letter; I didn't have time to write a short one."

I always thought that was Mark Twain who said that, so I asked ChatGPT who initially told me to not self harm myself and that I’m not alone... But, yep, I have been misinformed my whole life. It was Pascual. Apparently the chat bot thinks I’m having an existential crisis over this but thank you for educating me.

yud's goal was to spread concepts and popularity about his rationality cult, and he arguably did that very successfully with HPMOR. i don't think he's an idiot if he successfully reached his goals, and is also currently basically providing himself a full time job (and employing many others too) doing exactly what he wants to do with no clear profit motive to the people who fund him

his writing may be a little cringe at times and not anywhere near the prestige of "real writers" but it's perfectly entertaining for his intended audience

It's interesting that you're slinging around moron accusations at Yudkowsky seemingly unaware that slatestarcodex thinks very highly of his intelligence and Scott's blog is downwind of Yudkowsky.

Reply on news.ycombinator.com