> And AI is a computer. So the naive expectation would be that it would be capable of doing as well as a computer.
Why would you judge an AI against the expectations of a naive person who doesn't understand capabilities AIs are likely to have? If an alien came down to earth and concluded humans weren't intelligent because the first person it met couldn't simulate quantum systems in their head, would that be fair?
That piece reports that the note in question was not taken down by a central authority in Twitter, and was instead voted down. You gotta read the correction at the bottom.
This is totally wrong. Fine-tuning techniques were a prerequisite for turning academically impressive but boring and useless models into useful, compelling user products like ChatGPT. Fine-tuning complements fundamental model improvements -- it doesn't substitute for them.
This isn't true. In fact, it's the reverse of true. If you think a bit more carefully about your argument, you'll realize that you've asserted that the single most revolutionary advance of modern deep networks (i.e. network architectures whose performance scales neatly with their parameter counts & training epochs) automatically portends "the plateau of forward progress."
It may be rational and it may be irrational. All I'm saying is I want to see an argument. My impression of the AI space generally is that there are many, many obvious ideas which are simply waiting to be picked up off the ground and tested, and that it's not clear to anyone how much better (what are we even quantifying this with -- log loss?) the base LLM capabilities have to be before they're suitable for making tools that automate large quantities of work. Even if you assume that 240B is the limit for how many parameters number of engineers who have the ability to fine-tune GPT-4 or whatever Google is about to come out with is vanishingly small compared to the number of engineers who are participating in the open-source ML community, and the number of engineers in the open-source ML community is vanishingly small compared to the number of engineers in the long-tail of app developers who will ultimately adopt LLMs to their use-cases. Even assuming that GPT-4 is the best an LLM can possibly be, (which, again, I've seen no argument for), the widening of LLM availability and the building of practical tooling is a strong reason to believe that the utility of LLMs to concrete products will dramatically increase in the next 4-5 years.
If the most intelligent machines ever observed in the universe do not count as "intelligent," then we have a semantic, and not a substantive difference of opinion.
To me, if artificial intelligence means anything, it means automating mundane tasks -- most of which humans can accomplish already. If someone tells me they're an AI pessimist because they think automating tasks like "converse intelligently about nearly any subject at length" and "drive my car anywhere while I sleep in the back" isn't an impressive AI achievement, then I think our disagreement pertains to our ambitions for the field, and lies outside the realm of the technical.
Yes, but how do they know where we are in that growth phase? If you confidently tell me that "GPT-4 is the plateau," I want to know how you know that, specifically. "Well, because all deep learning technologies eventually slow down" is not a good argument. You need to show me the diminishing returns, or give me a good theoretical argument for why we're reaching them.
Humans will never be intelligent. They're optimized for producing offspring, not reasoning. Humans may appear to be intelligent from a distance, but talk to one for any length of time and you'll find they make basic errors of reasoning that no truly thinking being would fall for. /s
This is exactly the point. The pessimists get the pedantic thrill of pointing out that, 60 years ago, some proponents were overconfident. But they neglect to notice the larger picture, which is one of extraordinary progress. They'll be sitting in the passenger seat of a self-driving car, arranging their travel itinerary with a chatbot fluent in English and 60 other languages, and smugly commenting on HackerNews about how "AI pessimists got it right in the 60s."
1. AI has not been "around the corner since the 1950s," and 2. if you think there hasn't been forward progress in AI since the 1950s, you have no valid opinions on this subject.
This entire piece is based on one massive, unsupported assertion, which is that LLM progress will cease. Or, as the author puts it, "we are at the tail end of the first wave of large language model-based AI... [it] ends somewhere in the next year or two with the kinds of limits people are running up against." I want to know only one thing, which is what gives him the confidence necessary to say that. If that one statement is untrue, the thesis of the piece completely fails, and I do not know how any one person alive today can be certain that that statement is true. Has Paul Kedrovsky or Eric Norlin spent $150M training a 2T parameter model that no one's heard about? Do they have access to classified OpenAI docs which state that GPT5 exists already, and it's no better than GPT4? Without this sort of information, "LLMs are not going to get smart enough for widespread practical use" is an unsubstantiated bet, and nothing more.
This sounds a lot like a disagreement over product decisions that the development team has made, and not at all what people normally think when you say "a new paper has proved GPT-4 is getting worse over time."
The paper does things like ask GPT-4 to write code and then check if that code compiles. Since March, they've fine-tuned GPT-4 to add back-ticks around code, which improves human-readable formatting but stops the code compiling. This is interpreted as "degraded performance" in the paper even though it's improved performance from a human perspective.
By your definition, is a blind person capable of reasoning about visual data? Is a deaf person capable of reasoning about auditory data? Can a physicist understand the molecules, atoms, & subatomic particles which he or she can only interact with via a fundamentally textual theory? I would submit that there's no fundamental reason why an LLM needs access to more than text to derive human-level world models.
I'm not saying that the current LLMs have derived human-level world models (they haven't). It's just that, to me, the theory that textual data is categorically not enough to do so is necessarily empirical. To back up the assertion, you'd need to construct metrics which present text-only LLMs fail to succeed with, and then you need to show how multi-modal LLMs did succeed with those same metrics. So far, I don't think adding multi-modality to LLMs actually has improved their general-purpose reasoning ability, which I consider evidence against this theory. But then I read people online just asserting it as though it's an obvious truth derivable from philosophical first-principles. It's odd to me.
I used it extensively to debug a very thorny MySQL version upgrade. In my experience, it knows a lot about MySQL, and can reason about weird behavior very very effectively. A typical use is asking it something like, "Prior to my MySQL upgrade, my DB integration tests all passed, but now, they're non-deterministically failing with the following error. What could be causing this?" It then proposes hypotheses, which I either accept or push back on with new information. Surprisingly, this process led to it actually debugging a great number of my problems.
You need to define "true understanding" in some concrete, measurable way, or accept that you're just using it to avoid thinking clearly about AI capabilities.
"No true understanding" is the new "no true Scotsman." If you're still repeating this line in mid-2023, you're not meaningfully engaging with the technological developments that are occurring in front of your eyes.
Yes, there is a hype cycle that afflicts every new piece of tech. The existence of hype isn't evidence either way. What makes smart people think that this time is different is a straightforward comparison of the capabilities of LLMs and the skills needed to perform various human jobs, including (ominously) high-paying, scarce jobs.
People's freakouts are completely reasonable. The pace of development in this field is unreal and shows few signs of slowing down. The primary blocker to adoption in many fields is just software integration, which will come within a few years at most. At that point we'll see what the limits of LLM capabilities are, and which professions are no longer economically viable.
People repeating this take in the face of so much overwhelming evidence to the contrary look so ridiculous that at this point, you just have to laugh at them. Yeah, sure, it's not reasoning. That hour-long exchange I just had with it, where it helped me trouble-shoot my totally bespoke DB migration problems step by step, coming up with potential hypotheses, ways of confirming or disconfirming theories, interacting in real time as I gave it new constraints until we collaboratively arrived at a solution -- that was definitely not "reasoning." Why isn't it reasoning? No explanation is ever given. I'm waiting for the moment when someone tells me that it's not "true reasoning" unless it's produced in the Reasoning region of France.
> the atomization of intra-family bonds that seems so immoral to you is actually caused by the migration, driven by underlying economic opportunities and not some NIMBY anti-growth mindset
The point is that we have to make a trade-off between free internal migration and NIMBY. If you mix those two ideas together, you get misery. In deciding which is worth prioritizing, perhaps you should consider that free internal migration is a literal bedrock principle of the American state, dating back prior to the Constitution, enshrined legally in the Constitution, on which huge quantities of American legal and economic life have been built for the last 250 years, whereas NIMBY is an ad-hoc political coalition of wealthy homeowners and crotchety neighborhood activists who'd rather no one new get to live in their city.
Yep. And the core team has that "happy cult" vibe where they think that if they're (passive-aggressively) polite enough, they don't have to worry about anyone's opinion outside their insular little group. I tried to use it for a little project about a year ago, ran into some problems, and, searching around, found my problems were 1. not unique to me, and 2. were definitely not going to be solved by the language designers, despite a lot of people having the same suggestions. Their attitude is "either you design your app how we'd prefer you to or you don't use Elm at all." This doesn't work in any other technical project, but they think it's going to work for them for some reason.
The policy they've chosen to advocate directly harms me. Not in a, "oh, no, there's now a construction project nearby" sort of way. In a, "I literally cannot afford to live in this city anymore, so I have to leave" sort of way. I will deride the hell out of the extremely privileged status quo interests who put their precious aesthetic opinions above the basic needs of their children and grandchildren. If you loudly advocate your selfish and in doing so you hurt the hell out of large groups of people, they will dislike you for it.
"A huge comeback" is a big overstatement. Overall, it's still a burnt-out husk of its former glory. But it is true that the city proper has developed something which it lacked circa 2000, which is non-auto jobs (Quicken Loans + some other tech startups), a downtown art scene, young people, music, some hip food, and so on. In other words, it's germinated the embers of a much smaller, bougier, more livable town than existed at any time since the mid-late 80s.
I think everything I wrote is accurate. Bandera was a vicious anti-Semite. Bandera did collaborate with the Nazis. Bandera is venerated in Ukraine (since 2014 and especially since 2022). Maidan was made possible by armed support from neo-nazi political parties. Based on my reading, I do not think that Ukraine is a particularly anti-Semitic place, and I think that they like Bandera mainly because they see him as an anti-Russian fighter. But it's easy for me to understand why Germans are much more uncomfortable with the symbolism that I am.
They do. They also understand that there collaborators were Ukrainian nationalists (who were extremely anti-semitic and enthusiastically participated in pogroms) who are increasingly venerated in Ukraine. As an American, I'm able to compartmentalize this, and rationalize that Ukrainians are venerating people like Bandera for their anti-Russian partisanship, not for their anti-Jewish politics. But this is less of an option for Germans, who for very understandable reasons have a greater sensitivity to anti-Semitism than most nations.