Hacker News Reader

albertzeyeron John Carmack talk at Upper Bound 2025

I think he argues that they would not be able to play Atari games this way (I don't know; maybe I also misunderstood).

But also, he argues a lot about sample efficiency. He wants to develop algorithms/methods/models which can learn much faster / with much fewer data.

albertzeyeron John Carmack talk at Upper Bound 2025

His goal was not just to solve Atari games. That was already done.

His goal is to develop generic methods. So you could work with more complex games or the physical world for that, as that is what you want in the end. However, his insight is, you can even modify the Atari setting to test this, e.g. to work in realtime, and the added complexity by more complex games doesn't really give you any new additional insights at this point.

albertzeyeron Gemini Diffusion

> Google's first LLM to use diffusion in place of transformers.

But this is a wrong statement? Google never made this statement? You can have a Transformer diffusion models. Actually Transformers are very standard for all of the discrete diffusion language models, so I would expect Gemini Diffusion also uses Transformers.

Edit Ah sorry, I missed, this was already addressed, also linked in the post: https://news.ycombinator.com/item?id=44057939 Maybe my remaining post is still useful to some.

The difference is, it's an encoder-only Transformer, and not a decoder-only Transformer. I.e. it gets fed in a full sequence (but noisy/corrupted), and it predicts the full correct sequence. And then you can iterate on that. All frames in the sequence can be calculated in parallel, and if you need only a few iterations, this is faster than the sequential decoding in decoder-only models (although speculative decoding also gets you some speedup for similar reasons). Those discrete diffusion models / encoder-only Transformers are usually trained with BERT-like masking, but that's actually an active field of research. It's really a pity that they don't provide any details here (on training and modeling).

I wonder how this relates to Gemini. Does it use the same modeling? Was the model checkpoint even imported from Gemini, and then further finetuned for discrete diffusion? Or knowledge distillation? Or is it just branding?

albertzeyeron Project Verona: Fearless Concurrency for Python

4GL and 5GL are already taken. So this is the 6GL.

https://en.wikipedia.org/wiki/Programming_language_generatio...

But speaking more seriously, how to get this deterministic?

albertzeyeron I don't like NumPy

It seems the main complaint is about confusing shapes / dimensions. Xarray has already be mentioned, but this is a broader concept, called named dimensions, sometimes also named tensors, named axes, labeled tensors, etc, which has often been proposed before, and many implementations exists.

https://nlp.seas.harvard.edu/NamedTensor

https://namedtensor.github.io/

https://docs.pytorch.org/docs/stable/named_tensor.html

https://github.com/JuliaArrays/AxisArrays.jl

https://github.com/ofnote/tsalib

https://github.com/google-deepmind/tensor_annotations

https://github.com/google-deepmind/penzai

https://github.com/tensorflow/mesh/

https://github.com/facebookresearch/torchdim

https://xarray.dev/

https://returnn.readthedocs.io/en/latest/getting_started/dat... (that's my own development; unfortunately the doc is very much outdated on that)

albertzeyeron Why do LLMs have emergent properties?

Just by the mathematical definition of interpolation (https://en.wikipedia.org/wiki/Interpolation), any function which approximates the given regression points (training data), and is defined for values in between (unseed data), will interpolate it. Maybe you think about linear interpolation specifically, but there are many types of interpolation, and any mathematical function, any neural network is just another form of interpolation.

Interpolation is also related to extrapolation. In higher dimensional spaces, the distinction is not so clear. In terms of machine learning, you would call this generalization.

The question is more, is it a good type of interpolation/extrapolation/generalization. You measure that on a test set.

And mathematically speaking, your brain also is just doing another type of interpolation/extrapolation.

albertzeyeron Jargonic Sets New SOTA for Japanese ASR

Are there any details on what they changed to improve over other existing models?

albertzeyeron Mercury: Commercial-scale diffusion language model

It links to this file: https://drive.google.com/file/d/1j1ofmm8iBaVreGC5TSF1oLsrOqB...

But all but the first page seems to be missing in this PDF? There is just an abstract and (partial) outline.

albertzeyeron Welcome to the Era of Experience [pdf]

That link is dead now?

But I guess you mean this? https://www.sciencedirect.com/science/article/pii/S000437022...

"Reward is enough", David Silver, Satinder Singh, Doina Precup, Richard S. Sutton, 2021.

albertzeyeron Multi-Token Attention

Why do you say FlexAttention is too buggy? I have heard about a lot of successful usages of it, and never heard about any such problems.

Also note, depending on your model dimensions and sequence lengths, often the attention computation plays only a minor role (maybe 10% overall or so), and the MLP computation dominates.

albertzeyeron James Webb Space Telescope reveals that most galaxies rotate clockwise

For the matter-antimatter ratio, you would not expect 50:50, or would you? Because 50:50 would be a highly unstable system? In any case, you would expect that unstable states would be highly unlikely, and it would converge into a stable state.

I'm not sure about the other examples. But maybe it's a similar reason that it is not a 50:50 ratio?

albertzeyeron Ask HN: Any insider takes on Yann LeCun's push against current architectures?

Jürgen Schmidhuber has a paper with Lucas Beyer? I'm not aware of it. Which do you mean?

albertzeyeron Stack Traces Are Underrated

Stack traces are very valuable. Sometimes it can even help to attach them to some object creation, when you later wonder why/how/where this object was created. E.g. in TensorFlow, every single Tensor had a traceback attached to it, so when there was any error later on, it would show you where it was created. This is maybe less needed now with eager mode, but you might have other similar situations.

One problem with stack traces is maybe that they can be too verbose. E.g. if you print them for any warning you print to log (or stdout). Sometimes they will be extremely helpful for debugging some problem, but in many cases, you maybe don't need them (you know why you get the warning and/or you don't care about it).

You could also add more information to the stack trace such as local variables. That can be even more helpful for debugging then, but again adds more verbosity.

For example, we often use this to add information about relevant local variables: https://github.com/albertz/py_better_exchook

One solution to the problem with verbosity is when you have foldable text output. Then the stack trace is folded away (not shown in all details) and you can unfold it to see the details. See the DomTerm demo here: https://github.com/albertz/py_better_exchook#domterm

Some more on text folding:

https://github.com/PerBothner/DomTerm/issues/54

https://gitlab.com/gnachman/iterm2/-/issues/4950

https://github.com/xtermjs/xterm.js/issues/1875

https://gitlab.freedesktop.org/terminal-wg/specifications/-/...

https://github.com/vercel/hyper/issues/1093

albertzeyeron Performance of the Python 3.14 tail-call interpreter

To clarify: The situation is still not completely understood? It's not just only the computed gotos, but there is some other regression in Clang19? Basically, the difference between clang19.nocg and clang19 is not really clear?

Btw, what about some clang18.tc comparison, i.e. Clang18 with the new tail-call interpreter? I wonder how that compares to clang19.tc.

albertzeyeron Fish 4

> However, there should be no direct impact on users.

I find this quite impressive, that they rewrote the whole Fish core, but everything keeps working exactly in the same way (except very few minor things which change in only minor ways, which they list).

albertzeyeron Hyperspace

> This got me wondering why the filesystem itself doesn't run a similar kind of deduplication process in the background.

I think that ZFS actually does this. https://www.truenas.com/docs/references/zfsdeduplication/

albertzeyeron Hyperspace

I wrote a similar (but simpler) script which would replace a file by a hardlink if it has the same content.

My main motivation was for the packages of Python virtual envs, where I often have similar packages installed, and even if versions are different, many files would still match. Some of the packages are quite huge, e.g. Numpy, PyTorch, TensorFlow, etc. I got quite some disk space savings from this.

https://github.com/albertz/system-tools/blob/master/bin/merg...

albertzeyeron We don't need startups, we need Digital-Mittelstand

Yes I read it. I don't think these suggestions really help much, or are really new. E.g. simplify bureaucracy and regulations, that is what most parties already claim to do since many many years. I think many people really want that, but it seems to be difficult to really do this. VAT exemptions is probably also in the program of a couple of parties. English language is already pretty standard in many IT companies in Germany.

albertzeyeron We don't need startups, we need Digital-Mittelstand

There are a lot of such companies already in Germany. Basically most small/medium sized IT companies in Germany fit that classification. So I don't really see how this is now a new approach. And I'm also not sure what is proposed here to be done now. Invest more into IT infrastructure? That's a goal that many parties claim to do since many years. Many people fully agree with this. However, currently, the main topic in Germany is again immigration. Second topic is economy, but the opinions on how to improve economy widely varies. Many of the more traditional parties want to invest more into car companies.

albertzeyeron DeepSeek Open Source FlashMLA – MLA Decoding Kernel for Hopper GPUs

I also just read that paper. But I wonder, even though MLA is strictly more powerful, do you really gain by that in experiments? This paper doesn't really do too much experimental comparisons. GQA on the other side should still be faster (no need to an extra linear transformation).

albertzeyeron Show HN: Txeo – A Modern C++ Wrapper for TensorFlow

I have worked both with the TensorFlow C++ API and the TensorFlow Python API. While the TF Python API is basically only a wrapper around the TF C++ API, it adds a lot of things on top, e.g. many higher-level functions you would want to use to define neural networks, etc. If you know PyTorch, think about torch.nn. Most crucially, calculating the gradients, i.e. doing backprop/autograd, was also purely implemented in Python. Even to define the gradient per each operation was done in Python. The C++ core did not know anything about this. (I'm not exactly sure how much this changed with eager mode and gradient tapes though...)

So, that makes implementing training with only the C++ API quite a big task. You first need to define all the gradients, and then implement backprop / autograd.

albertzeyeron Grok3 Launch [video]

Yes that's certainly true. I was a bit hesitant to post a link from Gary Marcus. But I was mostly posting it for the Elon tweet. I assume the tweet is not fake. So you can ignore about Garys opinion here and just take Elons tweet as it is.

albertzeyeron Grok3 Launch [video]

https://garymarcus.substack.com/p/elon-musks-terrifying-visi...

I'm not sure if this was a very bad joke by Elon, or if Grok 3 is really biased like that.

albertzeyeron The History of S.u.S.E

I think this was my first (or among my first) Linux distributions, around 2000 or so. I did not had Internet access back then. I had some coding experience (mostly Basic; I was just learning C++). I made a dual installation with Windows. I don't really remember the details anymore, but I struggled with really using it, and after I played around with it (maybe it lead to some system freeze and then I did a cold restart), at some point it would not boot anymore. But also, I was not really learning anything about how it works, and did not really understand too much. I think I also tried Red Hat later but had similar experience.

Then some time later, I got Internet access, and I read about Gentoo, and the tutorial to install it from scratch was really well written and easy to follow, and that helped me really a lot in understanding how Linux really works, what components are involved in the whole system. I continued using that for many years. Whenever there was some problem, I was able to understand it and fix it.

Nowadays, I just want to use sth which is so widespread that it is very well supported, and I can expect that any problem can be found on the Internet with some solution, and sth where I just need to spend only a minimal amount of effort to keep it running and up-to-date, so I chose Ubuntu (already couple of years ago; not sure if my choice today would be the same, but for now I stick to it). So this is sth which basically just works. But I wonder, for a newcomer who really wants to understand how Linux works, I think I would rather recommend Gentoo or Arch Linux or so. I think I also prefer the rolling release concept (Arch, Gentoo) over the point release development model (Ubuntu, Debian, Suse, Slackware, etc).

albertzeyeron ArXiv LaTeX Cleaner: Clean the LaTeX code of your paper to submit to ArXiv

It's really a pity that they do this now. Some of their older papers had actually quite some valuable information, comments, discussions, thoughts, even commented out sections, figures, tables in it. It gave a much better view on how the paper was written over time, or how even the work processed over time. Sometimes you also see some alternative titles being discussed, which can be quite funny.

E.g. from https://arxiv.org/abs/1804.09849:

%\title{Sequence-to-Sequence Tricks and Hybrids\\for Improved Neural Machine Translation} % \title{Mixing and Matching Sequence-to-Sequence Modeling Techniques\\for Improved Neural Machine Translation} % \title{Analyzing and Optimizing Sequence-to-Sequence Modeling Techniques\\for Improved Neural Machine Translation} % \title{Frankenmodels for Improved Neural Machine Translation} % \title{Optimized Architectures and Training Strategies\\for Improved Neural Machine Translation} % \title{Hybrid Vigor: Combining Traits from Different Architectures Improves Neural Machine Translation}

\title{The Best of Both Worlds: \\Combining Recent Advances in Neural Machine Translation\\ ~}

Also a lot of things in the Attention is all you need paper: https://arxiv.org/abs/1706.03762v1

albertzeyeron The AMD Radeon Instinct MI300A's Giant Memory Subsystem

You might be true for the market.

However, that target audience, those hobby enthusiasts, hobby developers, also university labs with low budget, those are the people who will develop the future open source frameworks, and ultimately/implicitly those are the people who can have a quite big impact on the future development of brand recognition and the open source ecosystem around the hardware. Those people can shape the future trends.

So, only looking at the market, how much units you would sell here, that totally ignores the impact this might have indirectly in the future.

albertzeyeron Who would buy a Raspberry Pi for $120?

Can you give some example? I mostly also care about energy efficiency. How much energy would the alternatives consume?

albertzeyeron A minimax chess engine in regular expressions

The development writeup for the Doom is interesting, with many details. https://nicholas.carlini.com/writing/2019/javascript-doom-cl...

There was one month time to complete the competition. But it seems you were allowed to reuse any existing other code.

Looks like this was quite fun to work on.

(I feel a bit sad that I would never be able to get one month of free time to work on this now, due to family and job...)

albertzeyeron 1584 LAN party photos from 1996 – 2010 from Australia's island state archived

Why do you think these activities are obsolete? Meeting the other people face-to-face will always be nice, or not? And also playing together in the same room is much nicer than when you are separated.

I'm quite sure you will still be able to find such events, if you want to try. I know a couple of friends who still do that.

Speaking of something which has a similar feeling: There are the hacker events, such as the CCC, which is even right now: https://events.ccc.de/congress/2024/infos/index.html

(Unfortunately, I wasn't able to attend any of these since a while due to having small kids... But I definitely will join them sooner or later again. Maybe then with my kids.)

albertzeyeron Python Meta Circular Interpreter (2019)

Somewhat similar: My PyCPython project (https://github.com/albertz/PyCPython), where I wrote a C interpreter in Python, and then tried to interpret CPython. I never really completed this project though.