For those who've been sniffing this since early 2010, it's so blindly obvious they've already dropped llms on the floor and moved onto deeper alternative research.
For the rest of us, we're still catching coke bottles from the sky and building places of worship around them
There should be papers on fundamental limitations of LLMs then. Any pointers? "A single forward LLM pass has TC0 circuit complexity" isn't exactly it. Modern LLMs use CoT. Anything that uses Gödel's incompleteness theorems proves too much (We don't know whether the brain is capable of hypercomputations. And, most likely, it isn't capable of that).
Ever since "AI" was named at Dartmouth, there have been very smart people thinking that their idea will be the thing which makes it work this time. Usually, those ideas work really well in-the-small (ELIZA, SHRDLU, Automated Mathematician, etc.), but don't scale to useful problem sizes.
So, unless you've built a full-scale implementation of your ideas, I wouldn't put too much faith in them if I were you.
If you have something that gives a sticky +5% at 250M scale, you might have an actual winner. Almost all new ML ideas fall well short of that.
But, I shouldn't have said anything.
I was hired on to help run these experiments with a machine they were trying to perfect. They wanted to do all this stuff with running two machines in parallel, dumping the refuse from one into another machine, all kinds of silly stuff.
I looked at it and in a few minutes I sketched in my notebook a solution to their problem that would achieve the maximum theoretical yield without all these wacky experiments.
But then what would be my job? I was hired to run these experiments not solve their whole problem in 5 minutes. So I didn't tell them. I was getting paid so i didn't care if they were wasting time. I'm a smiley little trans woman anyway, they wouldn't have listened. I did work out the math though on my own.
A few months later, a team of phds at a university we were working with hit on a rudimentary form of the same idea, so at that point I revealed to the boss, oh, look I've worked out the math for that already. And I wrote up the code, and we hit the maximum theoretical yield and it ran 100x faster computationally, and i got a pat on the back. But the reason they listened was because a man with a degree said it.
Why would I bother giving them ideas if they won't listen anyway? If i had given them the answer months earlier, I would not have gotten any kind of bonus even in the unlikely event that I had been taken seriously.
So now of course youre all telling me im insane, and thats fine because honestly I have no incentive to give you answers ahead of time. Go suck eggs. Figure shit out on your own.
You guaranteed that you wouldn't get recognition by keeping the idea to yourself. Your victimization was by your own design, and it stayed where it started: in your head. And ironically you were the one being passive agressive and showing bias towards your coworkers.
Nothing is punished so swiftly and severely as outshining your boss or making him look stupid.
Embrace cringe