montebicyclelo

Yep agree, good details, and this doesn't contradict my point above, about people going "overboard" with the comparison.

I do think when making these comparisons, it is worth emphasising that neural nets are really different. E.g. I used to see people equating LLMs to n-gram models, etc. which is overly simplistic, (especially in the early days when the models weren't as good).

This is fun, but historically people have gone a bit overboard with saying that models like this, or n-gram language models, are anywhere close to large neural network models. There is certainly a connection though.

More fragmented sleep, with people talking in the night, etc. I guess that's different from the low, unnatural, rumbles of cars and aeroplanes the majority of people have to put up with.

Really cool work. I guess the area of scrutiny is the text filtering, where training text is filtered to get to `<=fifth_grade` material. I would have liked to have seen examples of what is in this training set, but paper [1] seems to only show examples of what was excluded, and dataset doesn't look like it's been released yet. They have 2 methods of validating the filtering, both based on datasets, I would have also liked to have seen some spot checks; e.g. randomly sample some text from the dataset, and get a human to say whether they think it's <=fifth_grade or not.

(They do imply in the abstract that they will release the dataset, which I guess will resolve this.)

[1] https://arxiv.org/abs/2608.13545

Apparently, the French typically start with "no" because it leaves them room to say "yes" later, which is harder to go back on.

> “Answering ‘non’ gives you the option to say ‘oui’ [yes] later; [it’s] the opposite when you say ‘oui’, you can no longer say ‘non’!

https://www.bbc.co.uk/travel/article/20190804-why-the-french...

Cool doc, thanks for sharing. Amazingly, a quick search brought up one of the articles they were shown setting (from the few words mentioned):

https://www.nytimes.com/1978/07/01/archives/150-hurt-as-truc...

worth noting you don't usually read _that_ many books in your lifetime. e.g. 20 a books a year for 40 years is just 800 books

side note: i've noticed people writing with no caps, unusual grammar, splling mistakes, etc. which does differentiate their output from typical llm output - although ofc llms can probably imitate that

Did the ycombinator podcast which included giving advice to startup founders just a few days ago:

https://www.ycombinator.com/library/Vy-jeff-dean-the-1-rule-...

The article is heavily leaning on the paper "The Illusion of Thinking" [1].

It could be boiled down to: in 2025 this paper showed that "thought traces" in the models of the time could sometimes be inaccurate or misleading. Today they still might be, although OpenAI says actually they are accurate for their modern models, (based on internal research, rather than published research).

[1] https://arxiv.org/abs/2506.06941

I was wondering whether this might have an explanation for why I don't really like major music, preferring minor / modal / non-functional harmony... it does:

> Throughout this derivation of different chords, you will note that a gradual progression or degradation from high-theme/low-complexity (Major Triad, Harmonic Series chords) to low-theme/high-complexity (Minor and Ambiguous chords). This progression is suggested by our measure of interestingness from Section 2.4 "Interestingness: Just Enough Complexity". That is, as we progress, more and more of the theme of the Harmonic Series is lost and more and more complexity is introduced. Notice that this progression seems to mirror that of musical sophistication as well: musically untrained listeners like Major chords while more musically trained listeners are more tolerant to loss of theme and more interested in complexity. (I met a signal processing engineer who had played piano for something like 18 years and who simply did not like Major chords at all.) Other fields seem to progress similarly: white wine is preferred by new wine drinkers, whereas more "complex" red wines are an acquired taste.

Fun, and enjoyed the track it uses:

https://music.youtube.com/watch?v=-tJ23mRbmps

Hmm, there's a difference between unnecessary arguments about every tiny detail, and productive arguments.

I've seen many healthy technical disagreements; often leading to new insights coming to light, assumptions being made explicit, everyone leaving with a better understanding, sometimes resulting in one party conceding, sometimes resulting in a compromise. Guess it requires a certain level of maturity / people arguing in good faith.

> Then cite them as blog posts

My point is it's still useful to have a somewhat authoritative place to cite (high quality) blog post level content. arXiv has formatting requirements and doesn't go down like random personal sites.

> a LaTeX PDF can launder epistemic status

True to a certain extent, although something people are aware of and they can judge the content themselves (hopefully).

Well, some blog posts are worth citing.

Isn't the directionality important. I.e. it is currently possible to run useful / great models locally, but on high end machines; and in a few years we will likely be able to run even better models on standard machines.

"why not alternative", would be better framed as, "here's a fun variation" — because both approaches are just playing around with technology, for fun / curiosity / exploration. Storing in the pixels is a fun approach, resulting in something Rube Goldberg-esque.

Ever considered second hand slightly older gens? Even M1 is still great for many use cases. E.g. often corps are selling them on Ebay, in pretty good condition.

Reminds me of this [1] HN post from 9 months ago, where the author trained a neural network to do world emulation from video recordings of their local park — you can walk around in their interactive demo [2].

I don't have access to the DeepMind demo, but from the video it looks like it takes the idea up a notch.

(I don't know the exact lineage of these ideas, but a general observation is that it's a shame that it's the norm for blog posts / indie demos to not get cited.)

[1] https://news.ycombinator.com/item?id=43798757

[2] https://madebyoll.in/posts/world_emulation_via_dnn/demo/

Originally wrote this in 2022 to improve my understanding of deep learning internals. Recently refreshed it, removing CuPy to keep it lighter and educational. (Plus modernized the stack, e.g. with uv and improved CI.)

For an example of an AI generated song that's gone viral in the last few days, getting millions of views on Spotify / Youtube, see this post from earlier today:

"Tell HN: Viral Hit Made by AI, 10M listens on Spotify last few days" [1]

[1] https://news.ycombinator.com/item?id=46600681

Yep, it is convincing. And it seems the video is performed by a human, although I did think it could be AI. They say in the description:

> I couldn't resist putting a face to this gem.

After saying it is AI:

> How people was touched by this version of « Papoutai » by stromae made by AI, I was also touched as y’all, as an independent Artist I wanted to put my emotions and my soul on this masterpiece

https://www.youtube.com/watch?v=bQ8GbwQV5zE

As a hobbyist, shaders is up there as one of the most fun types of programming.. Low-level / relatively simple language, often tied to a satisfying visual result. Once it clicks, it's a cool paradigm to be working in, e.g. "I am coding from the perspective of a single pixel".

    bulk_send(
        generate_expiry_email(user) 
        for user in db.getUsers() 
        if is_expired(user, date.now())
    )

(...Just another flavour of syntax to look at)

Incorrect Pytorch gradients with Apple MPS backend...

Yep this kind of thing can happen. I found and reported incorrect gradients for Apple's Metal-backed tensorflow conv2d in 2021 [1].

(Pretty sure I've seen incorrect gradients with another Pytorch backend, but that was a few years ago and I don't seem to have raised an issue to refer to... )

One might think this class of errors would be caught by a test suite. Autodiff can be tested quite comprehensively against numerical differentiation [2]. (Although this example is from a much simpler lib than Pytorch, so I could be missing something.)

[1] https://github.com/apple/tensorflow_macos/issues/230

[2] https://github.com/sradc/SmallPebble/blob/2cd915c4ba72bf2d92...

Agreed, and its larger context window is fantastic. My workflow:

- Convert the whole codebase into a string

- Paste it into Gemini

- Ask a question

People seem to be very taken with "agentic" approaches were the model selects a few files to look at, but I've found it very effective and convenient just to give the model the whole codebase, and then have a conversation with it, get it to output code, modify a file, etc.

At your own risk — one place is ebay sellers with a large number of positive reviews, (and not much negative), who are selling lots of the same type of MacBook pros. My assumption is they've got a bunch of corporate laptops to sell off.

On the contrary; now might be a good time to get an M1 Max laptop. A second hand one, ex-corporate, in good condition, with 64Gb RAM, is pretty good value, compared to new laptops at the same price. It's still a fantastic CPU.

> nanochat is also inspired by modded-nanoGPT

Nice synergy here, the lineage is: Karpathy's nano-GPT -> Keller Jordan's modded-nanoGPT (a speedrun of training nanoGPT) -> NanoChat

modded-nanoGPT [1] is a great project, well worth checking out, it's all about massively speeding up the training of a small GPT model.

Notably it uses the author's Muon optimizer [2], rather than AdamW, (for the linear layers).

[1] https://github.com/KellerJordan/modded-nanogpt

[2] https://kellerjordan.github.io/posts/muon/

I did the same. I was sad to lose the comments, but the ads were awful and I don't particularly want someone elses ads / tracking on my hobby site. I switched to gisqus [1], which is powered by GitHub discussions, which seems to be working ok. (The site is hosted on GH pages so seems reasonable to also use GH discussions for the comments.)

[1] https://giscus.app/

> One of the simplest CRDT strategies is Last-Write-Wins (LWW):

> Each update gets a timestamp (physical or logical).

> When two devices write to the same field, the update with the latest timestamp wins.

Please also have a robust synced undo feature, so it's easy to undo the thing you don't want that gets written. Apps that sync often seem to be stingy about how much "undos" they store/sync (if any).