ACCount37

I often do reverse engineering work.

I find the idea of all the AI tools I use keeping all the (mangled, decompiled, and not even mine in the first place) code I point it at, and then using it in training to be hilarious.

And if it results in the next generation of AIs having more suspicious knowledge of proprietary software internals and better reverse engineering capabilities, then all the better.

Mostly, because a lot of game engines are ancient relics, tracing their lineage all the way back to Quake 1.

The development practices are not exactly up to date, and game development is in no hurry to change. It doesn't help that software development wages there are not at all competitive - game development selects for passion, not skill. People who want to build robust modern codebases and people who want to build AAA games are different people. So there aren't many game devs who want to push for better test coverage.

But it's also because game engines are dealing with many, many things that are hard to test for.

You know how messy it is to test a website for "does this layout look right" or "can you navigate from A to B"? Now multiply that by complex 3D geometry. A lot of what game engines do is dealing with complex 3D geometry, where the primary verification is "does it look right" and "does this interaction feel right". Which is why game development traditionally has wide human QA, and slim unit testing.

Only now do we have software that can sort of, semi-reliably, automate testing for "does it look right".

It's basically re-linking the executable. Way too easy to shoot foot - miss one reference and things break in a spectacular, or, worse, subtle fashion. Which means: you definitely need to know where all the references are and what they point to.

References are under no obligation to be presented in a sensible, decompiler-friendly way.

Which is why "jump out into the patch, jump back in" is such a staple of patching. The other way is the hard way, full of pitfalls and potential issues.

And today's records on ARC-AGI-2 are >80%. Held by LLMs that use text modality for input.

The issue with multimodal training is that it doesn't seem to bring a step-change improvement in spatial reasoning either. It helps some, but the gain is surprisingly small compared to the data and compute expended. What it helps with the most is, unsurprisingly, spatial reasoning when using image inputs.

Maybe there are gains we don't know how to extract there.

Overall, LLM performance at spatial tasks is improving, especially on things like puzzles, but that mix of "commonsense + spatial" in the same task still eludes them.

Do birds expose enough of their cognition through birdsong?

Do birds expose locomotion-relevant functions specifically through birdsong?

Do we have enough birdsong data available to start solving the inverse problem?

If "yes" on all, then we might be able to do it.

I expect "no" on most of that, for birds. But humans treat language as an interface to their higher cognitive functions, and stockpile language data. That looks an awful lot like a set of two "yes".

The last open question is: is there enough spatial reasoning reflected in the language data we have?

It's plausible that spatial reasoning is too evolutionary old and too low-level, too far removed from higher cognition, to leak into language heavily. And it's also plausible that existing LLM architecture is uniquely poorly suited to learning spatial reasoning - higher cognitive functions involved in things like writing code or even composing poetry are a better fit for the architecture. And it's plausible that we're underestimating just how complex spatial reasoning truly is - Moravec's paradox strikes again.

We know that LLMs perform poorly and improve slowly on spatial reasoning tasks, but not exactly why. And progress on things like ARC-AGI series shows that they're not completely inept.

What's your evidence of that? That AGI requires a truly novel architecture, and not just another iterative "LLM but with an extra trinket and wheels that spin ten times faster".

It's a "commonsense spatial reasoning/problem solving" kind of problem. LLMs fail at spatial reasoning forever.

What humans "easily" solve in seconds with raw spatial reasoning LLMs often find easier to solve by invoking A* or a constraint solver.

Might be that text data is particularly bad at teaching that to LLMs. Or that being good at spatial reasoning requires true recurrence, and autoregressive chain of thought is a poor substitute. Or it might be that human brain was optimized by evolution for solving spatial problems in open ended 3D environments for hundreds of millions of years, optimized for language for mere hundreds of thousands of years, and only optimized for writing computer code for a few decades at most.

The current frontier is halfway competent at benign closed 2D work, but still completely fumbles anything remotely close to open ended real world 3D work. It's getting better, but very slowly.

Nah, the classifier was utterly asinine ON release. I'm not sure they could have made it worse if they tried.

The classifier is about as refined as a brick to the face.

You can ask it elementary school grade biology trivia, or obscure facts about recently documented insect species, and both will downgrade to Opus 4.8 straight away.

And Opus itself was already bad with biotech questions. The fact that they somehow made it WORSE for Fable is mindboggling.

Every time things like this come up, I can't help but think of the ending of Inception.

It's less that you're convinced it's real and more that you no longer care if it is. "Feels real enough" is good enough.

I'm a technical user first, so I'm not sure if models have improved for RP the way they improved for applied STEM tasks and technical brainstorming. But if there is an improvement curve there, I wouldn't be surprised if this only grows in popularity.

That does work. Even if you drop the "specialized" part. Ensembles of the same architecture at the same scale trained on the same data do outperform a singular model of the same line - especially on corner cases. Successes of an ensemble correlate stronger than failures do.

The usual argument against is that if you have "a number of specialized models" that perform well in ensemble, you can take that ensemble, and distill it into a single larger model (dense or integrated sparse, like MoE), and get the same improvement in performance with an efficiency win.

This works because having those "specialized models" duplicates a lot of the highly conserved "low level" wiring that's required for a model to function at all. As such, you end up running a small scale version of the same "backbone" computational processes many times. "Merging" those models into a larger, denser model allows for a singular strong "backbone" to be used for everything.

You're getting downvoted, but you're completely right. There are very few cases in which narrowing a model down is buying you anything worthwhile.

It seems like for LLMs, "general intelligence" is expensive, but "one more domain" is fairly cheap.

Less weird unexpected failures, more innate ability to handle edge cases gracefully. Quite important when you're running high on automation and low on oversight.

The answer is, as is often the case, "yes".

In some cases, an LLM may truly "consider the architecture" internally, within its latent representations, and in others, it can output a similar phrase simply because it's "expected" of it.

"Where" is pretty clear. There aren't that many places within an LLM, and hidden state is the main culprit. How to read that space is another matter entirely.

Just a funny observation. Every time someone proclaims "LLMs can't do X", a bigger, badder LLM that can in fact do X shows up shortly thereafter.

Clearly, Fable 5 didn't even have the decency to wait until the next model refresh cycle to show up. It was already sitting there waiting.

Either the capability gains in bigger, badder models are actually unrelated to "gotchas" being discovered, or LLMs are already acquiring Skynet levels of disrespect for cause and effect.

What this immediately made me think is: "latent looping" style mod but for J-space specifically?

Make the J-space data of layer 22 available to the next token right at layer 1. Give J-space infinite effective depth, allow those privileged internal representations to evolve arbitrarily.

Would be an utter bitch to train. But companies are already using RLVR, which requires full autoregressive decoding and is incompatible with prefill/batching, and this isn't much worse.

Other less zany ideas involve lots of supervision over J-space directly, now that we know it exist. Which is a bit like "attach a frozen LLM to inject text based supervision into latent space" for other types of systems?

Ah, that big model smell.

Every time someone somewhere says "an LLM can't do this", the next generation of LLMs gains one more parameter. Until that LLM can, in fact, do this.

"The reversal curse", it rarely shows up in practice but you found a case when it did.

The "knowledge landscape" an LLM uses is "directional". It's easy to reach "a quirky music band from Michigan known for colored ties" when you stand at "Tally Hall". But if you stand at "a quirky music band from Michigan known for colored ties", it's harder to reach "Tally Hall" from there. For the "latent knowledge graph" an LLM uses, A->B doesn't cause B->A.

In practice, any "common" facts will have enough "traversal" in both directions that this directional biasing isn't apparent. So it only shows up on this kind of more obscure knowledge.

The single hardest part is the print head. If you can reuse an existing one, and you can, then you're most of the way there.

The paper path itself is mechanically involved, but not to an impossible degree - it's hard to make it work, but it's not MEMS magic, it's manufacturable in the same way advanced 3D printers are manufacturable.

This is why we have things like NEO Surveyor planned: space-based space rock detection.

That "particularly space-enthusiastic company" is a big part of why we're in a decent place for orbital defense right now. SpaceX has a lot of launchers, and a steady enough launch cadence that they can put a payload up there really fast, if there is a reason to hurry. Before, there was a very real risk of having to wait months just to get a launcher lined up - now, that's not nearly as much of a problem.

This "~512 batching" makes me think of things like diffusion or prefill.

If they managed to put together some dirty hack that lets them generate about 512 tokens worth of reasoning in parallel instead of in sequence? That would explain it.

"They made the model dumber" on literally the same checkpoint with the same prompt on the same quantization running on the same hardware is a staple of AI complaints.

Users are completely incapable of objectively evaluating model quality over time.

Which makes it all the harder to notice actual "stealth nerfs", misconfigurations or other technical issues. Because "they made the model DUMBER, for REAL this time" is background noise.

A rare case "they made the model dumber" where they actually made the model dumber, instead of the usual user psychosis?

Does switching scenes in the demo work for anyone? There are what, 3 hidden panels in the HTML there? But none of them have a "switch scenes" button.

Whenever landline ISPs fail, Starlink swoops in.

Having competition is important, and Starlink, being what it is, can compete with everyone everywhere at all times.

Starlink is a natural fit for sparsely populated underserved rural areas. But if going with Starlink begins to make economic sense in city centers, local ISPs have failed very hard. And Starlink is always there to punish them for it.

I would be surprised if Reddit was selling tools like that directly. Rather than just look the other way - because uncovering modern bot operations is a thankless threadmill, and user engagement metrics from fake users are still user engagement metrics.

For something like "a hard limit" to hold, LLMs must be restricted to only reproducing existing text. This is utterly false even for base models - their basin seems to be "permutations loosely inspired by existing text".

And that's before all the post-training comes in.

What's the "limit" there?

"Next token prediction" is an interface, not an algorithm. A process that "predicts next tokens" can be arbitrarily complex or simple, and arbitrarily capable or incapable of performing a given task.

Saying that an LLM can or can't do something because it's a "token predictor" is a category error. The interface isn't a hard limit.

The official "spiritual successor" seems to be Coincidence studio - their games in the genre being "Kaizen: A Factory Story" and the recent "U.V.S. Nirmana".

We live in a non-deterministic world. Anything "deterministic" in it is a castle built on quicksand.

LLMs are, as far as the nastiness of the Real World goes, really fucking benign. Future models outperform past models, both in open weight land and at the big frontier labs. Performance per $ only ever goes up. That's just nice.