strangescript

It can't reason -> It can't make new discoveries -> It can only tie together bespoke missed data -> It can make some basic discoveries -> ??????

more important things to yell about now like global id and age verification and doing everything in their power to hamstring AI development

shh, you are going to upset people that think drinking is evil and cutting it has no draw backs

The T1000s are going to be chasing Gary down and he is going to look over his shoulder and explain to them they aren't really AGI

I am not convinced hallucinations is a solvable problem in a single self contained model. In one hand we approach these things with the idea we are building a real intelligence modeled after our assumptions of our own brain. In the other we want them to not have any of the failings of the things they are being modeled after.

Humans mis-remember and make up things all the time, completely unintentionally. It could be a fundamental flaw in large neural networks. Impressive data compression and ability to generalize, but impossible to make "perfect".

If AI becomes cheap and fast enough, its likely a simple council of models will be enough to alleviate 99% of the problem here.

"If a machine is consuming and transforming incalculable amounts of training data produced by humans, discussed by humans, and explained by humans. Why would the output not look identical to human reasoning? If I were to photocopy this article, nobody would argue that my photocopier wrote it and therefore can think. But add enough convolutedness to the process, and it looks a lot like maybe it did and can."

But its not copying it. That is the entire point. Its using the training data to adjust floating point numbers. If you train on a single data piece over and over again, then yes it can replicate it, just like you can memorize lines of a school play, but its still not copied/compressed in the traditional, deterministic sense.

You can't argue "we don't know how they work, or our own brains work with any certainty" and then over-trivialize what they do on the next argument.

People suffer brain damage and come out the other side with radically different personalities. What happened to "qualia", or "sense of self", where is their "soul". Its just a mechanistic emergent property of their biological neural network.

Who is to say our brains aren't just very high parameterized biological floating point machines? That is the true Occam's Razor here, as uncomfortable as that might make people.

We haven't reached a peak on scaling/performance, so even if an old model can be commoditized, a new one will be created to take advantage of the newly freed infra. Until we hit a ceiling on scaling, tokens are going to remain expensive relative to what people are trying to do with them because the underlying compute is expensive.

The problem with the Bitter Lesson is that it doesn't clearly define what is a computational "hack" and what is a genuine architecturally breakthrough. We would be no where without transformers for example.

Is the name meant to be a jab at you know who or am I reading too much into it?

“It’s only going to get better” "For LLMs to keep improving, we need one of three things: more data, more power or a breakthrough." "Breakthroughs are rare."

This is very short sighted. The amount of research and papers coming out on AI is staggering. Maybe raw power gaining breakthroughs are rare, but you literally only need one to have exponential impact.

If you look back at the history of LLMs, its a never ending timeline of "well, they will never be able to do this" that have been proven wrong, going all the way back to the early 20-teens.

Seems like a very strange bet to make at this point in time.

If coding agents progress like the predictions, there is no world where many people are not out of jobs. Not just coders, but code related jobs. There is so much infra around humans not writing good code, or not being able to plan well.

If agents get so good that they overcome these obstacles then most mid tier companies dev staff is going to be a couple of people making sure the agents are online and running.

Vibe coding is just the canary in the coal mine.

I was literally typing band-aid when I scrolled down.

Many systems are going to have to come up with better solutions to the problems AI will pose for legacy processes.

The diagrams in this article are amazing if you are somewhere in between a novice and expert. Seeing all of the new models laid out next to each other is fantastic.

We are the frog in the warm water...

who said anything about creating something more intelligent than us, these articles have the air of "why are we wasting our time on this stuff", people like gary marcus link them, meanwhile they get better week over week

"You know that really hard test thing that most humans on the planet can't do, or even understand, yeah, LLMs kind of suck at it too"

Meanwhile Noam "well aschtually..."

I love how people are still betting against AI, its hilarious. Please write more 2000-esk "The internet is a fad" articles

Does valve even leak the game titles you purchase to card processors? Don't they have some plausible deniability here?

I feel like this is more about Linux remaining reliable and the other platforms getting worse, rather than Linux getting substantially better.

I no longer have a windows machine. If I can't get a game to run on Linux then I don't bother with it. I played Clair Obscur start, to finish, in Mint on a 3090 with zero issues. I just forced it to load in steam since linux isn't officially supported.

I mean, thats the point, they aren't the same. Concorde was one dimensional, AI is not.

I think the difference between all previous technologies is scope. If you make a super sonic jet that gets people from place A to place B faster for more money, but the target consumer is like "yeah, I don't care that much about that at that price point", then your tech sort is of dead. You are also fully innovated on that product, like maybe you can make it more fuel efficient, sure, but your scope is narrow.

AI is the opposite. There are numerous things it can do and numerous ways to improve it (currently). There is lower upfront investment than say a supersonic jet and many more ways it can pivot if something doesn't work out.

The great thing but terrifying thing about innovation is we rarely select how it plays out. People create great ideas and concepts, but they rarely play out exactly how the initial researchers/inventors expected. Did Apple know we would spend all day doom scrolling when it created the iPhone? Did it want that? Would they have viewed that as desirable? Doubtful. But what was the alternative? Not make a smart phone and then wait until someone else does create one who has even less concern for people's well being. Or better yet, how could they have even predicted the outcome in 2007?

Humanity has never been able to put the innovation genie back in the bottle. At best we have delayed it, but even those situations require there be a finite resource that can be easily regulated and controlled. AI is not one of those things.

better yet, wait 10, let me know how it goes

This entire concept hinges on AI not getting better. If you believe AI is going continue to get better at the current ~5-10% a month range, then hand waiving over developer productivity today is about the same thing as writing an article about the internet being a fad in 1999.

A byte tokenized model is naturally 100% multi-lingual in all languages in its data set. There just isn't a lot of reason for teams to spend the extra training time to build that sort of model.

If your model is getting confused by python, its a bad model. Python is routinely the best language for all major models.

Reading the title and knowing exactly what this is about kind of makes me feel old to be honest.

If you work with model architecture and read papers, how could not know there are a flood of new ideas? Only few yield interesting results though.

I kind of wonder if libraries like pytorch have hurt experimental development. So many basic concepts no one thinks about anymore because they just use the out of the box solutions. And maybe those solutions are great and those parts are "solved", but I am not sure. How many models are using someone else's tokenizer, or someone else's strapped on vision model just to check a box in the model card?

Who said I wanted my outcomes to be deterministic. Why is it that the only way we accept programming is for completely deterministic outcomes, when the reality is that is an implementation detail.

I am a real user and I am on a general purpose e-commerce site and my ask is "I want a TV that is not that expensive", then by definition the user request is barely deterministic. User requests are normally like this for any application. High level and vague at best. Then developers spend all their time on edge cases, user QA, in the weeds junk that the User does not care about at all. People dont want to click filters and fill out forms for your app. They want it to be easy.

I am sure my brain activity is lower when I use a calculator instead of doing math by hand as well.

-- Nothing like a 100% accurate statement getting downvotes because people hate AI

google was way to obsessed with immediate success during this time, if they had that same mentality now they would have given up on LLMs and now they are in first in most categories