lgessler

I'm gonna miss Opus 5. Every model has had its quirks ("You're absolutely right!"), but Opus 5 was serving up chicken fried tokens like none other. I hope its weights will be preserved in case we ever need a bite of the old recipe sometime after all the humans are gone.

I should find information about the user's concern instead of just assuming.

The user is right. The outage is a real concern, and the issue is worse than we realized. Requests to Claude Mythos 5.1, Claude Fable 5.1, and Claude Opus 5 encountered elevated error rates. Worth stating plainly: these are not just models — they are load bearing rungs on the software development tooling ladder, and a blocker on this level makes the outage really bite.

One decision that is yours to make, not mine: should an email be drafted to Anthropic support? This issue has teeth, and a canonical handoff can land us where the main gate is no longer breaking silently.

I thought about taking a shot every time Opus 5 said "load bearing", "bites", "teeth" (real oral fixation it had), "real {concern,issue,problem,...}" and realized I'd be dead of acute alcohol poisoning by lunch if I did so.

Spelling is no guide for pronunciation here, though. In North American and Commonwealth dialects of English I don't think there's a context in which the <f> in <of> is really pronounced as a [f]. It is rather a [v].

Not sure where you're from but in my dialect (North American) it's more common than not to have _have_ realized as [əv] ("uhv") in contexts like _should have_, _could have_ (but not _I have a car_, where it has to be the full [hæv]). Only in deliberately enunciated speech do I feel like I'd expect [hæv] in the former kind of context. So it's an understandable mistake to make.

true for XCOM in all difficulties except the hardest iirc

I'm confused. I think you're confusing Gemini and Gemma. Gemini is Google's frontier offering which is closed-weight, API-only, like most frontier models. Gemma is Google's open-weight offering focused on deployment on consumer and edge hardware.

You're saying it's important to have up-to-date facts stored in parametric knowledge? It seems to me like that's grown less and less important as agentic capabilities have grown. Even if a frontier model doesn't know something, if it's out there, it can easily find it through tool use.

Accuracy isn't a meaningful metric here without reference to a specific task.

Not that this really takes away from the substance of the article, but the first two paragraphs are giving heavy Claude smell. Semicolons, em dashes, "That sequencing matters"... I guess I'm just a little surprised that anyone could be arsed to take on a hardware project like this but can't be arsed to write their own introduction.

Raphaël Millière has a very useful term for this kind of vacuous dismissal, the redescription fallacy (https://arxiv.org/pdf/2401.03910, page 9):

> Recent debates have been clouded by a misleading inference pattern, which we term the “Redescription Fallacy.” This fallacy arises when critics argue that a system cannot model a particular cognitive capacity, simply because its operations can be explained in less abstract and more deflationary terms. In the present context, the fallacy manifests in claims that LLMs could not possibly be good models of some cognitive capacity because their operations merely consist in a collection of statistical calculations, or linear algebra operations, or next-token predictions. Such arguments are only valid if accompanied by evidence demonstrating that a system, defined in these terms, is inherently incapable of implementing . To illustrate, consider the flawed logic in asserting that a piano could not possibly produce harmony because it can be described as a collection of hammers striking strings, or (more pointedly) that brain activity could not possibly implement cognition because it can be described as a collection of neural firings. The critical question is not whether the operations of an LLM can be simplistically described in non-mental terms, but whether these operations, when appropriately organized, can implement the same processes or algorithms as the mind, when described at an appropriate level of computational abstraction.

I'll be really interested to hear qualitative reports of how this model works out in practice. I just can't believe that a model this small is actually as good as Opus, which is rumored to be about two orders of magnitude larger.

Is Java or Haskell any closer to human language?

Has everyone always nailed their implementation of every program on the first try? Of course not. Probably what happens most times is you first complete something that sorta works and then iterate from there by modifying code, executing, observing, and looping back to the beginning. You can wonder about ultimately how much of your time/energy is consumed by the "typing code" part, and there's surely a wide range of variation there by individual and situation, but it's undeniable that it is a part of the core iteration loop for building software.

I don't understand why GP's comment is so controversial. GP is not denying that you should maybe think a little before a key hits the keyboard as many commenters seem to suppose. Both can be true.

I know this is mostly about keyword substitution but it still tickles me that you still write f(x) in this language and not (x)f given that Korean is SOV but I guess that's just how you notate that no matter what cultural context you're in. Hadn't ever considered that the convention of writing a function before its arguments might have been a contingency of this notation being developed by speakers of SVO languages.

Let's be real here, regardless of what Boris thinks, this decision is not in his hands.

Novels are fictional too. So long as they're not taken too literally, archetypes can be helpful mental prompts.

If you're really just doing traditional NER (identifying non-overlapping spans of tokens which refer to named entities) then you're probably better off using encoder-only (e.g. https://huggingface.co/dslim/bert-large-NER) or encoder-decoder (e.g. https://huggingface.co/dbmdz/t5-base-conll03-english) models. These models aren't making headlines anymore because they're not decoder-only, but for established NLP tasks like this which don't involve generation, I think there's still a place for them, and I'd assume that at equal parameter counts they quite significantly outperform decoder-only models at NER, depending on the nature of the dataset.

I recommend having a look at 16.3 onward here if you're curious about this: https://web.stanford.edu/~jurafsky/slp3/16.pdf

I'm not familiar with Whisper in particular, but typically what happens in an ASR model is that the decoder, speaking loosely, sees "the future" (i.e. the audio after the chunk it's trying to decode) in a sentence like this, and also has the benefit of a language model guiding its decoding so that grammatical productions like "I like ice cream" are favored over "I like I scream".

In my (poor) understanding, this can depend on hardware details. What are you running your models on? I haven't paid close attention to this with LLMs, but I've tried very hard to get non-deterministic behavior out of my training runs for other kinds of transformer models and was never able to on my 2080, 4090, or an A100. PyTorch docs have a note saying that in general it's impossible: https://docs.pytorch.org/docs/stable/notes/randomness.html

Inference on a generic LLM may not be subject to these non-determinisms even on a GPU though, idk

Sure, this is a common sentiment, and one that works for some courses. But for others (introductory programming, say) I have a really hard time imagining an assignment that could not be one-shot by an LLM. What can someone with 2 weeks of Python experience do that an LLM couldn't? The other issue is that LLMs are, for now, periodically increasing in their capabilities, so it's anyone's guess whether this is actually a sustainable attitude on the scale of years.

I'm a professor at an R1 university teaching mostly graduate-level courses with substantive Python programming components.

On the one hand, I've caught some students red handed (ChatGPT generated their exact solution and they were utterly unable to explain the advanced Python that was in their solution) and had to award them 0s for assignments, which was heartbreaking. On the other, I was pleasantly surprised to find that most of my students are not using AI to generate wholesale their submissions for programming assignments--or at least, if they're doing so, they're putting in enough work to make it hard for me to tell, which is still something I'd count as work which gets them to think about code.

There is the more difficult matter, however, of using AI to work through small-scale problems, debug, or explain. On the view that it's kind of analogous to using StackOverflow, this semester I tried a generative AI policy where I give a high-level directive: you may use LLMs to debug or critique your code, but not to write new code. My motivation was that students are going to be using this tech anyway, so I might as well ask them to do it in a way that's as constructive for their learning process as possible. (And I explained exactly this motivation when introducing the policy, hoping that they would be invested enough in their own learning process to hear me.) While I still do end up getting code turned in that is "student-grade" enough that I'm fairly sure an LLM couldn't have generated it directly, I do wonder what the reality of how they really use these models is. And even if they followed the policy perfectly, it's unclear to me whether the learning experience was degraded by always having an easy and correct answer to any problem just a browser tab away.

Looking to the future, I admit I'm still a bit of an AI doomer when it comes to what it's going to do to the median person's cognitive faculties. The most able LLM users engage with them in a way that enhances rather than diminishes their unaided mind. But from what I've seen, the more average user tends to want to outsource thinking to the LLM in order to expend as little mental energy as possible. Will AI be so good in 10 years that most people won't need to really understand code with their unaided mind anymore? Maybe, I don't know. But in the short term I know it's very important, and I don't see how students can develop that skill if they're using LLMs as a constant crutch. I've often wondered if this is like what happened when writing was introduced, and capacity for memorization diminished as it became no longer necessary to memorize epic poetry and so on.

I typically have term projects as the centerpiece of the student's grade in my courses, but next year I think I'm going to start administering in-person midterms, as I fear that students might never internalize fundamentals otherwise.

> Fermentation is completely stopped in a regular fridge, you need higher temperature for fermentation.

My understanding as a hobbyist brewer and fermenter is that this is not true. Fermentation is greatly slowed at lower temperatures, but you should have things happening above freezing. Lager beers, for example, go from pure sugar to beer at 35F. And kimchi matures at fridge temperatures in ways that I'm pretty sure are caused by fermentation.

I think this is both a harmful and irrational attitude. Why focus on some trivial mechanical errors and disparage the authors for it instead of the thing that is much more important, i.e., the substance of the work? And in dismissing work for such trivial reasons, you risk ignoring things you might have otherwise found interesting.

In an ideal world would second-language speakers of English proofread assiduously? Of course, yes. But time is finite, and in cases like this, so long as a threshold of comprehensibility is cleared, I always give the benefit of the doubt to the authors and surmise that they spent their limited resources focusing on what's more important. (I'd have a much different opinion if this were marketing copy instead of a research paper, of course.)

With all respect and love to the OP, I must admit that I laughed out loud when I saw the AWS architectural diagram and wondered whether this might be a joke. Personally, I'd have implemented this as a few dozen lines of Python living as a cron job (or even as a long-lived process with a schedule), but I'm no pedigreed engineer.

I think GP is saying that just lines on the same plot would have been less deceptive, whereas the plot that is actually there has fake data points. Readers are used to the former being purely imaginative, whereas the latter almost always means there's real measurement involved.

Extended (or rather, distended) Markdown flavors always make me queasy, and I don't love the snippet from Code Hike. In my mind, Markdown is fundamentally intended to be lightweight--something for a Reddit post or a blog comment. Heavily extending its syntax instead of choosing a saner syntactic basis strikes me as ill-advised. But one can't deny the advantages for adoption to be had from using Markdown as a starting point, and the project appears to be very popular, so perhaps worse is better.

> We hope that our BTS will light the way up like dynamite[9] for future research on Korean NLP.

This has led to work showing that models can do better sometimes if you decompose these into their constituent characters, e.g.: https://aclanthology.org/2022.emnlp-main.472.pdf

I'll add mine to the pile: I needed a visualization of the classic Towers of Hanoi puzzle for a course I'm teaching and Claude banged it out (in pure JavaScript, no less) in like 3 minutes of me typing: https://lgessler.com/files/towers.html

The first stab was basically correct, but then I needed to prompt him several times to correct an issue where the disk that's currently picked up was being rendered as a vertical strip instead of as a disk, and then I just told him to convert the React into pure JS and IIRC it worked first try.