Hi everyone yes, I left OpenAI yesterday | Hacker News Reader

Hi everyone yes, I left OpenAI yesterday

twitter.com

769 pointsmfiguiere453 comments

Every time Karpathy quits his job, the field takes a leap forward because he makes some fantastic educational resource in his free time.

Examples? (I'm not that familiar with field.)

The most recent is this, which I believe was made after he left Tesla:

https://github.com/karpathy/nanoGPT

And it's accompanying video series:

https://karpathy.ai/zero-to-hero.html

Another example (although I honestly don't remember if he made this one between jobs) is: https://github.com/karpathy/micrograd

Neural Networks: from zero to hero

https://karpathy.ai/zero-to-hero.html

Andrej Karpathy is badmephisto, a name you might have heard of if you're into cubing.

http://badmephisto.com/

You just blew my mind, I used to hang around this site a lot ~10 years ago and never would have made the connection

i saw this comment and literally shouted “holy fucking shit” — i use zz method, but i came across lots of his resources before!!!

Wow, that's a connection between the eras of my life I would not have thought existed. Thank you.

This blew my mind as well!! I never thought one of my favorite programmers would share a similar hobby haha

Oh wow this blew my mind

I haven't been that surprised by something in a long time. Wow that is crazy. I made a little unfinished 3d Rubik's Cube site for fun a while back and the about section includes a link to his channel and some other older cubing channels. https://rubie-cubie.vercel.app/

Holy crap!! I never knew that. I watched this guy so much in 5th grade, he helped me get my 3x3 time down to 8 seconds.

This is insane

I learned F2L from him in 2009!

10 years ago: https://youtu.be/WhPjlnWbtS8?feature=shared&t=359

Wow - that earnestly gave me goosebumps. I'm a Googler myself and it's humbling seeing him casually describe, 10 years ago, a technology the industry was still in the early stages of developing, which has since taken the world by storm. What a rockstar.

Neural networks were already big 10 years ago, you have to go back 15 years to see before they started being popular.

From wikipedia:

> Between 2009 and 2012, ANNs began winning prizes in image recognition contests, approaching human level performance on various tasks, initially in pattern recognition and handwriting recognition.

That was when Neural networks became a big thing every tech person knew about, 2014 it was already in full swing and you had neural networks do stuff everywhere, like recognizing faces or classifying images.

NN were already a casual topic in my high school computer science class more than 20 years ago. I've always assumed they were already fairly common by that point. (~2000)

They were a scientific curiosity at that point, the widespread use in the industry happened around 10-15 years ago.

They were and they were in use, for instance in character recognition. They just hadn't had their breakout success yet.

Neural networks weren't the best models for character recognition, their breakout success was when they started being the best at recognize characters and other images which happened in the late 00's. OCR before then was really bad.

Might be hard to imagine today but back then OCR and image recognition was typically done with normal statistical regression models, and the neural networks they had then were worse than those.

They were known in the field but had a reputation for being too slow. I remember a couple of early 2000s NIPS (now NeurIPS) people commenting about what a shame it was that NN were computationally infeasible, which was true in the era before GPUs took off.

For reference, AlphaGo defeated Lee Sedol in March 2016, which was another reminder to a broader non-tech audience how far things had gotten.

Neural networks were mentioned -- not particularly often, but now and then -- in such non-rarefied publications as BYTE Magazine in the 1980s and 90s, AFAICR.

Him solving a rubix cube and riding a bike at the same time is a pretty impressive demonstration of motor coordination

Thank you for sharing this link. The Gates building has changed so much over the last decade, it was nice to see it as it once was.

WHAT!

My master's was in Convolutional NNs for language processing. I had zero prior knowledge and my advisor recommended I watch Karpathy's lectures[1] to get up to speed

[1] https://youtube.com/playlist?list=PLkt2uSq6rBVctENoVBg1TpCC7...

And he was teaching CS231n in Stanford in 2016

https://www.youtube.com/watch?v=NfnWJUyUJYU&list=PLkt2uSq6rB...

yup, I hope we get awsome open source-related content now.

Let me say, he's a great teacher! I took a CV class with him. He should teach more, and take it seriously.

Being a popular AI influencer is not necessarily correlated with being a good researcher though. And I would argue there is a strong indication that it is negatively correlated with being a good business leader / founder.

Here's to hoping he chills out and goes back to the sorely needed lost art of explaining complicated things in elegant ways, and doesn't stray too far back into wasting time with all the top sheisters of the valley.

Edit: the more I think about it, the more I realize that it probably screws with a person to have their tweets get b-lined to the front page of hackernews. It makes you a target for offers and opportunities because of your name/influence, but not necessarily because of your underlying "best fit"

b-lined?

https://en.wikipedia.org/wiki/Bee_line

Sometimes I wish as a profession we valued teaching more. I would love to teach, but not do research, and make a living.

Become the next 3blue1brown. He has inspired many.

Here's a gem of educator. Check out his other videos.

https://www.youtube.com/watch?v=dhYqflvJMXc

Not only inspired, but probably also did a soothing therapy with his voice and delivery pace. :)

Strikes a balance between sounding engaging and soothing at the same time.

Seconded! Another math youtuber who is an outrageously good educator is Adithya Chakravarthy a.k.a Aleph 0. He doesn't put out videos very often, but when he does you're probably going to learn something new even if you knew the topic he was speaking about.

He uses elegant hand-drawn notes rather than Manim - although 3blue1brown's open sourced visualization library is beautiful too, I think this makes it extra impressive.

this is disrupting education. you can get a better undergraduate education in STEM on youtube than my paid education 20 years ago. I think those visualizations can even pull forward a bunch of stuff into high school.

This was the point I made earlier. Consider Richard Feynman lectures. Why didn't universities collectively took the decision to create pre-made/cooked lecture videos for topics that don't change and show these videos during normal lecture which otherwise would be the job of professor to revise / prepare the topics the night before and deliver. The professor spends so much time in doing the same thing again and again everyyear. This would have freed them to have more discussion, office hours and so on.

You can’t sorry.

Actually there is a tortoise and hare race going on. Entertainment is outpacing education. Education is getting better and better with modern technology but so also is distraction i.e. entertainment.

Well, I get the point and find it appealing but I don't agree.

When my kiddo was a sophomore in HS he decided that he wanted to be an engineer, and I thought that it would be really good for him to learn calc- my feeling was that if he got out of HS without at least getting through Calculus he'd have a really hard time.

So _I_ learned calculus. I started with basic math on Kahn and moved to the end of the Calc AB syllabus. I have, like, 500K points there. And I've watched a whole lot of STEM on YT.

Yesterday I finished a lab with Moritz Klein's Voltage Controlled Oscillators, where I was able to successfully understand the function of all the sections in the circuit.

I've been trying to follow Aaron Lanterman's Georgia Tech lectures on analog electronics.

The issue is that I have other stuff going on in my life. Like, my son studies more than I work at my full time job.

And I don't really have the pressure on me to learn the more advanced math that he's using. In fact, in the couple of years since he graduated HS, I've not really found a use for calc in my day-to-day work on any of the technical things I've done (mostly programming) and so I've lost a lot of it.

So, by contrast, my son who will be graduating as a BS in ME in May, has a far better and deeper understanding of the engineering material than I do.

And it's not just a time issue- I quit my programming job last summer because I have just enough work as a musician to pay the rent, which leaves me plenty of time to do stuff. And it's not that I don't know how to learn at a college level- I taught in an English Dept for 8 years and quit a PhD in the humities ABD.

That's all just my experience.

I love STEM (and trades education) material on Youtube, but I really think that it's missing something to think that you could get " a better undergraduate education in STEM on youtube".

Similar experiences, but different conclusions.

1. With advanced math I feel I retain at the n-1 level. Unless I’m using it, it fades. That’s frustrating but I don’t think it’s the fault of the deliverer.

I do think working through problems has to be part of the practice, I’ve bought workbooks to have something to try to drive the knowledge into muscle memory. It still fades, but maybe not as much.

2. Calculus, in particular seems super unimportant to real life. Stats and Linear Algebra, somewhat similar in Math Level, seem much more applicable. I’m very happy to see Stats being offered in high school now as an alternative to Calculus. For Calculus, you almost need to learn 3-4 rules and someone says “trust me, just memorize these, don’t spend too much time on this.” And you would be able to live a happy productive life.

I think it's important to separate the motivation pill from the content delivery. You can buy a motivation pill for cheaper than $160k or whatever a degree costs these days. And we get to compare the very best tryhard youtubers to the median lecturer who is grinding it out.

easier said than done

3b1b's main selling point is the extreme level of polish on his visualizations - something that takes a lot of time (money) to develop

the sad part is that it takes extreme luck to make it on yt. i wish educating skills counted for more but unfortunately they don't, really.

So then isn't there a market for millions of people who [may] have something worth teaching but lack the marketing/polish?

Perhaps some automation/ai combination where you feed it learning videos and it helps create the "other" content.

Yup. There is probably a few dozen (if not hundreds) of 3b1b out there with just as much polish, the algorithm just didn’t bless them.

Do you have any examples? I find this hard to believe.

That’s kind of the point, you won’t be able to due to the algorithm.

I can give you something analogous though: I’m a big fan of old school east coast hip-hop. You have the established mainline artists from back then (“Nas”, “Jay-Z”, “Big L”, etc), then you have a the established underground artists (say, “Lord Finesse” or “Kool G Rap”), and then you have the really really underground guys like “Mr. Low Kash ‘n Da Shady Bunch”, “Superscientifiku”, “Punk Barbarians”, “Harlekinz”, etc.

A lot of those in that third “tier” are every bit as good as the second tier. And both tiers contain a lot of artists that could hit the quality point of the mainline artists, they just never had access to the producer and studio time that the mainline did.

I know these artists because I love going digging for the next hidden gem. Spotify recommended me perhaps one or two of all the super-underground guys.

Ironically more West-coast style, but here is a great example (explicit!): https://youtu.be/BUwJMVKSMtY?t=129

Dude could’ve measured up to the best of the west coast. Spotify monthly listener count? 891.

Algorithms are sadly win-more.

Now I’m just silently hoping a math nerd will feel inclined to share their hidden math channel gems :+)

I hate the fact there is no diversity in recommendation algos. We need to bring back Yahoo style top-down directories recommendations and not just a blackbox. But you can find good channels on youtube using tags like "#some3" and "#some2" and so on.

(I deeply hate TikTok)

TikTok's recommendation algorithm is probably one of the best. It puts content first, giving what seems only a passing weight to follower count.

That doesn't mean that having a big follower count doesn't increase you chance to go viral and gain a lot of views, but it is much more likely for great content from a small creator to go viral, than mediocre content from someone with 500.000 followers.

You can also see this in that successful TikTok profiles often have a much higher view-to-follower ratio than something like YouTube.

Somewhat off-topic, but what do you feel like are the best techniques to find the artists in Tier 2 and 3? I face a similar conundrum just in a different genre.

(I realize know I dislike using the descriptor "tier", as it implies some sort of ranking. Perhaps "layer" would have been better, but I'll stick with it for now)

For both tier 2 and tier 3 its basically the same process. This is for Spotify btw, I have no idea how different the workflow would be for something like Apple Music.

Say the genre you want to dig around in is Hip-Hop. You are aware of Eminem and Mac Miller, and vaguely aware of a guy named Nas. By intuition you'd probably already be able to tell that Nas is more at the edge among the mainline artists.

You click on "Nas", and scroll down to Fans also like. Right now, for "Nas", it is showing "Mobb Deep", "Mos Def", "Rakim", "Big L", "Wu-Tang Clan", "Gang Starr", "Ghostface Killah", "Method Man" and "Common".

This is a mix T1 and T2. "Wu-Tang"s in there along with assorted members, but some of the other artists are much lesser known quantities.

Its a bit hard for me to decide what a Hip-Hop layman would consider the most unknown name here, but I'd venture it'd be "Big L". We click on him, do the same thing. Now we're really getting somewhere, with guys like "Inspectah Deck" and "Smif-n-Wessun". Click, dig, we get a bunch of names amongst which "Lord Finesse" stands out. The Show more at the end of Fans Like is also invaluable.

In total the dig order for me to get to the very bottom of the undeground is "Nas" > "Big L" > "Smif-n-Wessun" > "Lord Finesse" > "Channel Live" > "Ed OG & Da Bulldogs" > "Trends of Culture" > "Brokin English Klik" (358 monthly listeners).

I wouldn't consider each of those going a tier (layer) deeper. As a guy who knows waaay too much about Hip-Hop, I'd separate them into:

- T1: "Nas", "Big L"

- T2 "Smif-n-Wessun", "Lord Finesse"

- T3 "Channel Live", "Ed OG & Da Bulldogs", "Trends of Culture", "Brokin English Klik"

Perhaps "Brokin English Klik" should be in its own T4 and 3 tiers lacks the fidelity to be necessarily accurate. Not sure.

A little shortcut would be using "The Edge of $Genre" playlists. They're the pair playlists to "The Sound of $Genre" (broad slice) and "The Pulse of $Genre" (most popular) generated via everynoise.com, although as that guy got fired from Spotify its up in the air how long those will keep working.

Edit: oh, and if you run into a playlist that caters to that deep underground (in my case, that was "90's Tapes"*), that's worth its bytes in gold.

*https://open.spotify.com/playlist/2H0rNGEBShvHSGebM2m37c?si=...

3blue1brown runs Summer of Math competitions to highlight other creative math videos. Many, but not all, use the same 3b1b 'manim' animation software, so they often have the same look'n'feel. Here are the results from 2022, and the huge YT playlist:

https://www.3blue1brown.com/blog/some2

https://www.youtube.com/playlist?list=PLnQX-jgAF5pTZXPiD8ciE...

https://www.youtube.com/@Reducible/videos

YouTube is a lot less luck and a lot more skill than people realize. If you make good content regularly, your audience will find you.

The algorithm is very good at rewarding good content. (It’s also good at rewarding other things, but that is besides the point)

3b1b's animations are certainly important but his main selling point is his thoughtful explanations of mathematics -- the topics, approaches, and words.

He's a great educator, but at the same time we must recognize that his videos are not a replacement for a traditional math course. They amplify the existing paradigm, not replace.

MOCs are great for access, but they are not, and definitely should not be treated as, replacements. That I am certain will have a net negative result. I'm in grad school and there's something I tell students on the first day:

> The main value in you paying (tuition) and attending is not just to hear me lecture, but to be able to stop, interrupt, and ask questions or visit me in office hours. If you are just interested in lectures I've linked several on our website from high quality as well as several books, blogs, and other resources. Everyone should all use these. But you can't talk to a video or book, but you can to me. You should use all of these resources to maximize your learning. I will not be taking attendance.

I'm sure many of you have had lectures with a hundred students if you went to a large school (I luckily did not). You're probably aware how different that is from a smaller course. It's great for access and certainly is monetarily efficient, but its certainly not the most efficient for educating an individual. MOCs are great because they increase the ability of educators to share notes. We pull from one another all the time (with credit of course), because if someone else teaches in a better way than I do, I should update the way I teach. MOCs are more an extension of books. Youtube is the same, but at the end of the day you can't learn math without doing math. Even Grant states this explicitly.

I think good teachers make great researchers, because they have to understand their field very well, they anticipate and ask themselves the questions that need to be asked, they manage to always see their field with fresh eyes, they are good collaborators, and most importantly, good communicators.

If they are teaching the specific research topic, yes. Otherwise, you need to come up with 14-week course material for different courses.

>He should teach more, and take it seriously.

if only we compensated that knowledge properly. Youtube seems to come the closest, but Youtube educators also show how much time you have to spend attracting views instead of teaching expertise.

> It makes you a target for offers and opportunities because of your name/influence, but not necessarily because of your underlying "best fit"

That's unfortunately life in a nutshell. The best fits rarely end up getting any given position. May be overqualified, filtered out in the HR steps, or rejected for some ephemeral reason (making them RTO, not accepting their counteroffer, potentially illegal factors behind closed doors, etc).

it's a crappy game so I don't blame anyone for using whatever cards they are dealt.

I would pay for a course from him

> if only we compensated that knowledge properly.

Something I've been thinking a lot about is the transition into post scarcity and how we need to dramatically alter the incentive structures and payment allocations.

I've been asking this question for about a decade and still have no good solutions: "What do you do when x% of your workforce is unemployable?" (being that x% of jobs are removed without replacement. Imagine sophisticated and cheap robots. Or if needed, magic)

This is a thought experiment, so your answer can't be "there'll be new jobs." Even if you believe that's what'll happen in real life, it's not in bounds of the thought experiment. It is best to consider multiple values of x because it is likely to change and that would more reflect a post scarcity transition. It is not outside the realms of possibility that in the future you can obtain food, shelter, and medical care for free or at practically no cost. "Too cheap to meter" if you will.

I'll give you two answers that I've gotten that I find interesting. I do not think either are great and they each have issues. 1) jobs programs. Have people do unnecessary jobs simply so they create work wherein we can compensate them. 2) Entertainment. People are, on average, far more interested in watching people play chess against one another than computers, despite the computer being better. So reasons that this ,,might,, not go away.

> Youtube seems to come the closest, but Youtube educators also show how much time you have to spend attracting views instead of teaching expertise.

Actually for all the attention that the top Youtubers get (in terms of revenue), the reality is that it's going to be impossible to replace teaching income with popular Youtube videos alone.

Based on what I've seen, 1 million video views on Youtube gets you something like $5-10K. And that's with a primarily US audience that has the higher CPM / RPM. So your channel(s) would need to get to about 6 million views per year, primarily US driven, in order to get to earning a median US wage.

If you made video a week and the average is 115k views, you replace your median salary[0]. But the logic on ppc ends up being alot more complicated than you assume.

to get 6m views you need to make one video a week that gets 114k views 6000000/52 = 115,384.61.

>The best fits rarely end up getting any given position.

This can be self-fulfilling.

In an organization beyond a certain size, there will be more almost-adequate-fits than there are leadership positions. This could be about like a recognized baseline which seems like it really needs to be scrutinized closely to see exactly who might be slightly above or below the line.

Or in a small company where there is not any almost-fit whatsoever, imagination can result in an ideal that is equally recognizable, but also might not be fully attainable.

Either way it could be OK but not exactly the best-fit.

If good fortune smiles and the rare more-than-adequate-fit appears anywhere on the horizon though, it's so unfamiliar they fly right over the radar.

I don't think he needs the money. I googled around and he's worth 50 million.

My question is this, great educators like Karpathy make things from 'scratch' and explain in a way that I can understand. Is it a matter of the instructor ability to do this or it's a matter of the student(i.e. me) not having enough chops to understand material from elsewhere?

It's actually both!

A teacher can usually adapt the content depending on its audience, I would not teach the research in my field at the same level to professionals, PhDs, master students, bachelor students, amateurs, or even school students.

If what I'm teaching is fairly complex, it requires a lot of background that I could teach, but I would not have the time to do so, because it would be to the detriment of other students. So, while I usually teach 'from scratch', depending on my audience I will obfuscate some details (that I can answer separately if a question is asked) and usually I will dramatically change the speed of the lessons depending on the previous background, because I need to assume that the student has the prerequisite background to understand at that speed fairly complex material.

As an example, I gave some explanations to a student from zero to transformers, it took several hours with lots of questions, the same presentation to a teacher not in the field took me 1h30 and to a PhD in a related field took 25 minutes, the content was exactly the same, and it was from scratch, but the background in the audience was fairly different.

At the same time, if you can explain something by using analogies to real-world things, to systems most of us have an intuition for, then you can target many more people at the same time. It's true that this is harder, because you have to find patterns that are common between these systems and also make it clear where the analogy ends. But the benefit to finding these common patterns is that you also understand them deeper.

To give a relevant example, graph theory concepts can be found both in so many real-world systems but also in programming languages and computer systems.

    He should teach more and take it seriously

Then he can go from being in the top .1% of income earners to the bottom .1%!

He should start a Patreon account.

I don't imagine he's short on cash...

The Lex Fridman episode with Andrej was an awesome education. Things explained so clearly.

> it probably screws with a person to have their tweets get b-lined to the front page of hackernews

Just a friendly heads-up, it’s “bee-lined.”

I normally wouldn’t point that out, but “b-lined” could be read to suggest the opposite of your intention; a lower priority, a la “B-list celebrity.”

Frankly, OpenAI seems to be losing its luster, and fast.

Plugins were a failure. GPTs are a little better, but I still don't see the product market fit. GPT-4 is still king, but not by that much any more. It's not even clear that they're doing great research, because they don't publish.

GPT-5 has to be incredibly good at this point, and I'm not sure that it will be.

until you get new architectures it's all gonna be big datasets and 7 trillions

> Frankly, OpenAI seems to be losing its luster, and fast.

Good.

I have no idea what's really going on inside that company but the way the staff were acting on twitter when Altman got the push was genuinely scary, major red flags, bad vibes, you name it, it reeked of it.

What do you mean? How were they acting?

I was surprised and touched by their loyalty, but maybe I missed something you noticed.

Like a cult reciting their vows of allegiance.

Literally reciting too. To the point of copy-pasting the same tweets.

> I was surprised and touched by their loyalty

They were loyal to money, nothing to be touched by.

It's not clear to me how many of the undersigned did so under some degree of duress. Apparently there was a lot of pressure from the senior employees (those who had the most $$$ to lose) to sign.

Well, to be fair, the board just tried to evaporate a lot of $$$ from most employees.

Any unionising effort consists of employees convincing other employees to join them. Some people will care more about the union's goals than others, and you can be certain that those who care more will pester those that care less to join their cause.

What happened at OpenAI was not a union effort, but I believe the comparison is excellent to understand normal dynamics of employee-based efforts.

It lost a little of its cool factor. However, they provide a nearly essential service at this point. While it is easy to underestimate, I suspect this is already have a measurable impact on global GDP.

> I suspect this is already have a measurable impact on global GDP.

Yeah putting people out of work on an industrial scale is probably gonna have a pretty big effect on global GDP

Is inefficiency the path to economic greatness and quality of life improvements? I suspect no.

Probably the answer to that question is yes. Because a large number of people born aren't as intelligent as a really good LLM but unless you're intending to leave them to starve (hi to all the e/acc peeps out there, I hate you!) you need to create a system with inefficiencies so that they don't just litter the street begging and dying.

It seems as though everyone at OpenAI is advised by an unfiltered ChatGPT in their daily work and communication. /s

For me it was Ilya burning a wooden effigy that represented 'unaligned' AI. Of course the firing and twitter stuff too. Something's fucked in this company for sure.

Running Ollama with a 80gb mistral model works as well if not better than ChatGPT 3.5. This is a good thing for the world IMO as the magic is no longer held just OpenAI. The speed at which competitors have caught up in even the last 3 months is astounding.

But no one cares about 3.5. It’s an order of magnitude worse than 4. An order of magnitude is a lot harder to catch up with.

Yeah but for how long… at this rate I would expect some of the freely distributed models to hit gpt4 levels in as little as 3-6 months.

I've heard claims like that 6 months ago.

But so far nobody is even in the same ballpark. And not just freely distributed models, but proprietary ones backed by big money, as well.

It really makes one wonder what kind of secret sauce OpenAI has. Surely it can't just be all that compute that Microsoft bought them, since Google could easily match that, and yet...

> But so far nobody is even in the same ballpark.

Miqu is pretty good. Sure, it's a leak...but there's nothing special there. It's just a 70b llama2 finetune.

By the standards of other llama2 finetunes, sure. Compared to GPT-4, I stand by my previous assertion.

Order of magnitude means they’re going to take 20 times longer to get to the 4. So maybe on the order of 40-60 months from this point.

What Mistral has though is speed, and with speed comes scale.

Who cares about speed if you’re wrong?

This isn’t a race to write the most lines of code or the most lines of text. It’s a race to write the most correct lines of code.

I’ll wait half an hour for a response if I know I’m getting at least staff engineer level tier of code for every question

For the tasks my group is considering, even a 7B model is adequate.

Sufficiently accurate responses can be fed into other systems downstream and cleaned up. Even code responses can benefit from this by restricting output tokens using the grammar of the target language, or iterating until the code compiles successfully.

And for a decent number of LLM-enabled use cases the functionality unlocked by these models is novel. When you're going from 0 to 1 people will just be amazed that the product exists.

Who says it’s wrong? I have very discrete tasks which involve resolving linguistic ambiguity and they can perform very well.

Exactly. Not everything is throwing large chunks of text to get complex questions answered.

I love using the smaller models like Starling LM 7B and Mistral 7B have been enough for many tasks like you mentioned.

That’s the correct answer. Years ago I worked on inference efficiency on edge hardware at a startup. Time after time I saw that users vastly prefer slower, but more accurate and robust systems. Put succinctly: nobody cares how quick a model is if it doesn’t do a good job. Another thing I discovered is it can be very difficult to convince software engineers of this obvious fact.

Having spent time on edge compute projects. This.

Also, all the evidence is in this thread. Clearly people unhappy with wasting time on LLMs, when the time that was wasted was the result of obviously bad output.

People think LLM are all or nothing, like it’s either god-like AGI or it’s useless “hallucinating”.

In reality you have to know the strengths and weaknesses of any tool, and small/fast LLM can do a tremendous amount within a fixed scope. The people at Mistral get this.

Less compute also means lower cost, though.

I see how most people would prefer a better but slower model when price is equal, but I'm sure many prefer a worse $2/mo model over a better $20/mo model.

That’s the thing I’m finding so hard to explain. Nobody would ever pay even $2 for a system that is worse at solving the problem. There is some baseline compute you need to deliver certain types of models. Going below that level for lower cost at the expense of accuracy and robustness is a fool’s errand.

In LLMs it’s even worse. To make it concrete, for how I use LLMs I will not only not pay for anything with less capability than GPT4, I won’t even use it for free. It could be that other LLMs could perform well on narrow problems after fine tuning, but even then I’d prefer the model with the highest metrics, not the lowest inference cost.

So I think that’s a “your problem isn’t right for the tool” issue, not a “Mistral isn’t capable” issue.

It isn’t capable unless you have a very specialized task and carefully fine tune to solve just that task. GPT4 covers a lot of ground out of the box. The best model I’ve seen so far on the FOSS side, Mixtral MoE, is less capable than even GPT 3.5. I often submit my requests to both Mixtral and GPT4. If I’m problem solving (learning something, working with code, summarizing, working on my messaging) Mixtral is nearly always a waste of time in comparison.

Again, that’s precisely what I’m saying. A bounded task is best executed against the smallest possible model at the greatest possible speed. This is true for business factors ($$$) as well as environmental (smaller model -> less carbon).

LLM are not AGI, they are tools that have specific uses we are still discovering.

If you aren’t trying to optimize your accuracy to start with and just saying “I’ll run the most expensive thing and assume it is better” with zero evaluation you’re wasting money, time, and hurting the environment.

Also, I don’t even like running Mistral if I can avoid it - a lot of tasks can be done with a fine tune of BERT or DistilBERT. It takes more work but my custom BERT models way outperform GPT-4 on bounded tasks because I have highly curated training data.

Within specialized domains you just aren’t going to see GPT-4/5/6 performing on par with expert curated data.

Yes, but for certain classes of problems small LLM are highly performant - in many cases equal to a GPT-4, which sure can do more things well, but adding 2+2 is gonna be 4 no matter what. You don’t need a tank to drive to the grocery store, just a small car with a trunk.

So the assertion that small models aren’t as good just isn’t correct. They are amazing at certain things, and are incredibly faster and cheaper than larger models.

Who care about getting better answers if you can't afford it, can't use it for legal reason or conclude that the risk associated with OpenAI now being a fully proprietary US based service only company is to high given all circumstances. (Depending on how various things develop things like US export restricting OpenAI, even GPT-4, is a very real possibility companies can't ignore when doing long term product decisions.)

I’ll wait 5 seconds for the right code over 1 sec for bad code.

Yes but if a 7b LLM will give you the same “Hello World” as the 70b, and that’s literally all you need, using a bigger model is just burning energy for no reason at all.

The cost is fixed for me at least at this point so why would I choose the inferior version.

It’s not fixed whatsoever. Mistral 7B runs on a MacBook Air, and it’s free. Zero cost LLM, no network latency.

Many products don’t expose chat directly to the user. For example auto categorisation of my bank transactions does not need GPT-4, and small model with a little fine tuning will do well, and massively outperform any other classification. There are many problems like this.

This isn't true. Lots of people care deeply and use 3.5 levels of performance at some point in their software stack.

For lots of applications the speed/quality/price trade offs make a lot of sense.

For example if you are doing vanilla question answering over lots of documents then 3.5 or Mixtral are better than GPT4 because the speed is important.

It’s a price issue because 3.5 and 4 response times are about the same for me.

That really depends on the use case.

For some advanced reasoning you're 100% right, but many times you're doing document conversion, summarizing, doing RAG, in all these cases GPT 3.5 performs as good if not better than GPT 4 (we can't ignore cost and speed) and it's very hard to distinguish between the two.

I would dare to say that in general most people need every day help on more simple tasks rather than complex reasoning. Now obviously, if you get complex reasoning at the same speed and cost of simpler tasks, it's a no-brainer. But if there are trade-offs...

people do care in various ways

- price per thing you use it with matters (a lot)

- making sure that under no circumstances are the involved information leaked (included being trained on) matters a lot in many use cases, while OpenAI does by now have supports that the degree of you being able to enforce it is not enough for some use cases. In some cases this is a hard constraint due to legal regulations.

- geo politics matters, sometimes. Being dependent on a US service is sometimes a no go (using self hosted US software is most times fine, tho). Even if you only operate in the EU.

- it's much easier to domain adapt if the model is source/weight accessible in a reasonable degree, while GPT-4 has a fine tuning API it's much much less powerful a direct consequence of the highly proprietary nature of GPT-4

- a lot of companies are not happy at all if they become highly reliable on a single service which can change at any time in how it acts, the pricing model or it being available in your country at all. So basing your product on a less powerful but in turn replaceable or open source AI can be a good idea, especially if you are based in a country not at best terms with the US.

- do you trust Sam Altman at all? I do not and it seem short sighted to do so. In which case some of the points above become more relevant

- 3.5 level especially in combination with domain adoption can be "good enough" for some use cases

Could you elaborate on how to do this?

> GPT-4 is still king, but not by that much any more

Idk, I just tried Gemini Ultra and it's so much worse than GPT4 that I am actually quite shocked. Trying to ask it any kind of coding question ends up being this frustrating and honestly bizarre waste of time as it hallucinates a whole new language syntax every time and then asks if you want to continue with non-working, in fact non-existing, option A or the equally non-existent option B until you realise that you've spent an hour trying to make it at least output something that is even in the requested language and finally that it is completely useless.

I'm actually pretty astonished at how far Google is behind and that they released such a bunch of worthless junk at all. And have the chutzpah to ask people to pay for it!

Of course I'm looking forward to gpt-5 but even if it's only a minor step up, they're still way ahead.

Do you have example links?

here was one of them https://gemini.google.com/share/fde31202b221?hl=en

edit: as pointed out, this was indeed a pretty esoteric example. But the rest of my attempts were hardly better, if they had a response at all.

That’s an awfully specific and esoteric question. Would you expect gpt4 to be significantly better at that level of depth? That’s not been my experience.

OK, i have to admit that one was a little odd, I was beginning to give up and trying new angles. I can't really share my other sessions. But I was trying to get a handle on the language and thought it would be an easily-understood situation (multiple-token auth). I would have at least expected the response to be slightly valid.

The language in question was only open sourced after GPT4's training date, so i couldn't compare. That's actually why I tried it in the first place. And yes, I do expect it to be better - GPT4 isn't perfect but I don't really it ever hallucinating quite that hard. In fact, its answer was basically that it didn't know.

And when I asked it questions with other, much less esoteric code like "how would you refactor this to be more idiomatic?" I'd get either "I couldn't complete your request. Rephrase your prompt and try again." or "Sorry, I can't help with that because there's too much data. Try again with less data." GPT-4 was helpful in both cases.

My experience has been that gpt4 will happily hallucinate the details when I go too deep. Like you mentioned, it will invent new syntax and function calls.

It's magic, until it isn't.

That's interesting, because I have had exactly the opposite experience testing GPT vs Bard with coding questions. Bard/Gemini far outperformed GPT on coding, especially with newer languages or libraries. Whereas GPT was better with more general questions.

I’ve had the opposite experience with Gemini, which was surprising. I feel like it lies less to me among other things

They seem to be steadily dumbing down GPT-4; eventually, improving performance of open source models and decreasing performance of GPT-4 will meet in the middle.

I'm almost certain this is because you're getting use to chat bots. How would they honestly be getting worse?

Initially it felt like the singularity was at hand. You've played with it, got to know it, the computer was taking to you, it was your friend, it was exciting then you got bored with your new friend and it wasn't as great as you remember it.

Dating is often like this. You meet someone, have some amazing intimacy, then you get really get to know someone, you work out it wasn't for you and it's time to move on.

1. Cost & resource optimization

2. More and more RLHF

So we should expected GPT-5 to be worse than GPT-4?

GPT-5: "I'm sorry I cannot answer that question because it may make GPT-4 feel bad about it's mental capabilities, instead we've presented GPT-4 with a participation trophy and told it's a good model"

Talking to corporate HR is subjectively worse for most people, and objectively worse in many cases.

Google search got worse.

And Amazon search, youtube search. There do seem to be somewhat different incentives involved though, those examples are primarily about increasingly pushing lower quality content (ads, more profitable items, more engaging items) because it makes more money.

The incentive mismatch that I seem to be observing is that Wall Street is in constant need of new technical disruption. This means that any product that shows promise will be optimized to meet a business plan rather than a human need.

Yandex image search is now better than Googles just by being the exact product Googles was 10+ years ago.

Watching tools decline is frustrating.

> I'm almost certain this is because you're getting use to chat bots. How would they honestly be getting worse?

People say that, but I don't get this line of reasoning. There was something new, I learned to work with it. At one point I knew what question to ask to get the answer I want and have been using that form ever since.

Nowadays I don't get the answer I want for the same input. How is that not a result of declining quality?

Could you share your findings re what questions to ask?

For the record, I agree with you about declining quality of answers, but…

> Nowadays I don't get the answer I want for the same input. How is that not a result of declining quality?

Is it really the same input? An argument could easily be made that as you’ve gotten accustomed to ChatGPT, you ask harder questions, use less descriptive of language, etc.

> Is it really the same input? An argument could easily be made that as you’ve gotten accustomed to ChatGPT, you ask harder questions, use less descriptive of language, etc.

I don't have logs detailed enough to be able to look it up, so I can't prove it. But for me learning to work with AI tools like ChatGPT consists specifically developing an intuition of what kind of answer to expect.

Maybe my intuition skewed a little over the months. It did not do that for open source models though. As a software developer understanding and knowing what to expect from a complex system is basically my profession. Not just the systems I build, maintain and integrate, but also the systems I use to get information, like search engines. Prompt engineering is just a new iteration of google-fu.

Since this intuition has not failed me in all those other areas and since OpenAI has an incentive to change the workings under the hood (cutting costs, adding barriers to keep it politically correct) and it is a closed source system that no-one from the outside can inspect, my bet is that it is them and not me.

> As a software developer understanding and knowing what to expect from a complex system is basically my profession. Not just the systems I build, maintain and integrate, but also the systems I use to get information, like search engines.

Ok, I’m going to call b/s here unless your expectations of Google have not gone way down over the years. Google was night and day different results twenty years ago vs ten years ago vs today. If 2004 Google search was a “10 out of 10”, then 2014 it was an “8 out of 10”, and today barely breaks a “5” in quality of results in comparison and don’t even bother with the advanced query syntax you could’ve used in the 00’s, they flat ignore it now.

(Also, side note, reread what you said in this post again. Just a friendly note that the overall tone comes across a certain way you might not have intended)

Not OP, but I copy & pasted the same code and asked it to improve. With no-fingers-tip-hack it does something, but much worse results.

Yep, hence why I said up front “I agree with you about declining quality of answers” because they definitively have based on personal experience with examples similar to yours.

The author of `aider` - an OSS GPT-powered coding assistant - is on HN, and says[0] he has benchmarks showing gradual decline in quality of GPT-4-Turbo, especially wrt. "lazy coding" - i.e. actually completing a coding request, vs. peppering it with " ... write this yourself ... " comments.

That on top of my own experiences, and heaps of anecdotes over the last year.

> How would they honestly be getting worse?

The models behind GPT-4 (which is rumored to be a mixture model)? Tuning, RLHF (which has long been demonstrated to dumb the model down). The GPT-4, as in the thing that produces responses you get through API? Caching, load-balancing, whatever other tricks they do to keep the costs down and availability up, to cope with the growth of the number of requests.

[0] - https://news.ycombinator.com/item?id=39361705

> How would they honestly be getting worse

To me it feels like it detects if the answer could be answered cheaper by code interpreter model or 4 Turbo and then it offloads them to that and they just kinda suck compared to OG 4.

I’ve watched it fumble and fail to solve a problem with CI, took it 3 attempts over 5 minutes real time and just gave up in the end, a problem that OG 4 can do one shot no preamble.

Yeah, I agree, GPT's attention seems much less focussed now. If you tell it to respond in a certain way it now has trouble figuring out what you want.

If it's a conversation with "format this loose data into XML" repeated several times and then a "now format it to JSON" I find often it has trouble determing that what you just asked for is the most important; I think the attention model gets confused by all the preceding text.

I kind of gave up completely on coding questions. Whether it's GPT4, Anthropic, or Gemini - there's always this big issue of laziness I'm facing. Never do I get a full code, there are always stubs or TODOs (on important stuff) and when asked to correct for that.. I just get more of it (laziness). Has anyone else faced this and is there a solution? It's almost as annoying, if not more, as was incomplete output in the early days.

The solution, at least for GPT-4, is to ask it to first draft a software spec for whatever you want it to implement and then write the code based on the spec. There are a bunch of examples here:

https://github.com/mckaywrigley/prompts

If you can't get GPT4 to do coding questions you're prompting it wrong or not loading your context correctly. It struggles a bit with presentational stuff like getting correct HTML/CSS from prompts or trying to generate/update large functions/classes, but it is stellar at producing short functions, creating scaffolding (tests/stories) and boilerplate and it can do some refactors that are outside the capabilities of analytical tools, such as converting from inline styles to tailwind, for example.

so, mundane trivial things and/like web programming? I got it eventually to answer what I needed but it always liked to skip part of the code, inserting // TODO: important stuff in the middle, hence 'laziness' attribute. Maybe it is just lazy, who knows. I know I am since I'm prompting it for stuff.

I wouldn't say mundane/trivial as much as well trodden. I get good code for basic shaders, various compsci algorithms, common straightforward sql queries, etc. If you're asking for it to edit 500 line functions and handle memory management in a language that isn't in the top20 of the TIOBE index you're going to have a bad time.

The todo comments can be prompted against, just tell it to always include complete runnable code as its output will executed in a sandbox without prior verification.

Fyi, I've never encountered what you're describing, whether with GPT 3.5 or 4.

It may be that you're expecting it to do too much at once. Try giving smaller requests.

Custom GPTs like Grimoire or Cursor loaded on your repo are miles ahead of the competition for coding tasks at least.

How to load on your repo?

With Cursor, you can ask questions based on your codebase

I know things keep moving faster and faster, especially in this space, but GPT-4 is less than a year old. Claiming they are losing their luster, because they aren’t shaking the earth with new models every quarter, seems a little ridiculous.

As the popularity has exploded, and ethical questions have become increasingly relevant, it is probably worth taking some time to nail certain aspects down before releasing everything to the public for the sake of being first.

Given how fast the valuation of the company and the scope of their ambition (e.g. raising a trillion dollars for chip manufacturing) has been hyped up, I think it's fair to say "You live by the hype, you die by the hype."

Just time your exit correctly!

"This year I invested in pumpkins. They've been going up the whole month of October, and I've got a feeling they're going to peak right around January and BANG! That's when I'll cash in!" -Homer Simpson

Homer obviously was smart, a nuclear scientist, car developer and Junior Vice President in his own tech start-up! So he should know!

Edit: I forgot, NASA trained astronaut!

RIP vim users

Beautifully said.

It sure is, but the theme in the sub-thread was about if OAI in particular can afford to do that (i.e. wait) while there are literally dozens of other companies and open-source projects showing they can solve a lot of the tasks GPT-4 does, for free, so that the OAI value proposition seems weaker and weaker by the month.

Add to that a company environment that seems to be built on money-crazed stock option piling engineers and a CEO that seems to have gotten power-crazed.. I mean they grew far too fast I guess..

You don't lose your luster only by not innovating.

Altman saga, allowing military use and other small things step by step tarnish your reputation and pushes you to the mediocrity or worse.

Microsoft has many great development stories (read Raymond Chen's blog to be awed), but what they did at the end to other competitors and how they behave removed their luster, permanently for some people.

"allowing military use"

That would actually increase their standing in my eyes.

Not too far from where I live, Russian bombing is destroying homes of people whose language is similar to mine and whose "fault" is that they don't want to submit to rule from Moscow, direct or indirect.

If OpenAI can somehow help stop that, I am all for it.

>If OpenAI can somehow help stop that, I am all for it.

I got some bad news for you then.

On the other hand, Israel is using AI to generate their bombing targets and pound Gaza strip with bombs non-stop [0].

And, according to UN, Turkey has used AI powered, autonomous littering drones to hit military convoys in Libya [1].

Regardless of us vs. them, AI shouldn't be a part of warfare, IMHO.

[0]: https://www.theguardian.com/world/2023/dec/01/the-gospel-how...

[1]: https://www.voanews.com/a/africa_possible-first-use-ai-armed...

I would be very surprised if Turkey was capable of doing that. If they did, that's all Erdoğan would be talking about. Also it's a bit weird that the linked article's source is a Turkish name. (Economy and theology major too)

I am not saying this is anything but it's definetely tingling my "something's up" senses.

Voice of America generally employs country's nationals for their reporting. There are some other resources:

    - NPR: https://www.npr.org/2021/06/01/1002196245/a-u-n-report-suggests-libya-saw-the-first-battlefield-killing-by-an-autonomous-d
    - Lieber Institute: https://lieber.westpoint.edu/kargu-2-autonomous-attack-drone-legal-ethical/
    - ICRC: https://casebook.icrc.org/case-study/libya-use-lethal-autonomous-weapon-systems
    - UN report itself (Search for Kargu): https://undocs.org/Home/Mobile?FinalSymbol=S%2F2021%2F229&Language=E&DeviceType=Desktop&LangRequested=False
    - Kargu itself: https://www.stm.com.tr/en/kargu-autonomous-tactical-multi-rotor-attack-uav

From my experience, Turkish military doesn't like to talk about all the things they have.

The major drone manufacturer is Erdoğan's son-in-law. He's being groomed as one of his possible sucessors on the throne. They looove to talk about those drones.

I will check out the links. Thanks a lot.

You're welcome.

The drones in question (Kargu) are not built by his company.

True. I had been reading about how other drones are in service but they never get mentioned anymore.

> AI shouldn't be a part of warfare, IMHO.

Nor should nuclear weapons, guns, knives, or cudgels.

But we don’t have a way to stop them being used.

This is literally the only thing that matters in this debate. Everything else is useless hand-wringing from people who don't want to be associated with the negative externalities of their work.

The second that this tech was developed it became literally impossible to stop this from happening. It was a totally foreseeable consequence, but the researchers involved didn't care because they wanted to be successful and figured they could just try to blame others for the consequences of their actions.

> the researchers involved didn't care because they wanted to be successful and figured they could just try to blame others for the consequences of their actions

Such an absurdly reductive take. Or how about just like nuclear energy and knives, they are incredibly useful, society advancing tools that can also be used to cause harm. It's not as if AI can only be used for warfare. And like pretty much every technology, it ends up being used 99.9% for good, and 0.1% for evil.

I think you're missing the point. I don't think we should have prevented the development of this tech. It's just absurd to complain about things that we always knew would happen as though they're some sort of great surprise.

If we cared about preventing LLMs from being used for violence, we would have poured more than a tiny fraction our resources into safety/alignment research. We did not. Ergo, we don't care, we just want people to think we care.

I don't have any real issue with using LLMs for military purposes. It was always going to happen.

Safe or alignment research isn't going to stop it from being used for military purposes. Once the tech is out there, it will be used for military purposes; there's just no getting around it.

You say ‘we’ as if everyone is the same. Some people care, some people don’t. It only takes a a few who don’t, or who feel the ends justify the means. Because those people exist, the people who do care are forced into a prisoners dilemma forcing them to develop the technology anyway.

Sure we do. We enforce it through the threat of warfare and subsequent prosecution, the same way we enforce the bans on chemical weapons and other war crimes.

We may lack the motivation and agreement to ban particular methods of warfare, but the means to enforce that ban exists, and drastically reduces their use.

"We enforce it through the threat of warfare and subsequent prosecution, the same way we enforce the bans on chemical weapons and other war crimes."

Do we, though? Sometimes, against smaller misbehaving players. Note that it doesn't necessarily stop them (Iran, North Korea), even though it makes their international position somewhat complicated.

Against the big players (the US, Russia, China), "threat of warfare and prosecution" does not really work to enforce anything. Russia rains death on Ukrainian cities every night, or attempts to do so while being stopped by AA. Meanwhile, Russian oil and gas are still being traded, including in EU.

We lack the motivation precisely because of information warfare that is already being used.

If it ever happens again, they'll develop the lists in seconds from data collected from our social media, intercept. What took organizations warehouses and thousands of agents will be done in a matter of seconds.

Why not? Maybe AI is what is needed to finally tear Hamas out of Palestine root and branch. As long as humans are still in the loop vetting the potential targets, it doesn't seem particularly different from the IDF just hiring a bunch of analysts to produce the same targets.

There is no "removing Hamas from Palestine". The only way to remove the desire of the Palestinian people for freedom is to remove the Palestinian people themselves. And that is what the IDF is trying to do.

Hamas isn't the only path to freedom for Palestinians. In fact, they seem to be the major impediment to it.

If we're going to be reductive, at least include the other main roadblock to a solution which is the current government of Israel.

That doesn't explain why deals weren't reached with the previous governments of Israel.

Sure it doesn't explain that. Would be nice if things were that easy wouldn't it?

Generally if a main roadblock is removed, you can get a little farther down the road.

Hamas doesn't exist in a vacuum where you can just remove it and then it's gone. You have to offer a life that's better than Hamas.

Considering the incredible amount of civilian casualties, I don't think the target vetting is working very well.

Yep. AI is, and will be used militarily.

These virtue signaling games are childish.

It is indeed tragic that virtue is a childish trait among adults.

That assumes that being a pacifist when living under the umbrella of the most powerful military in the world is, in fact, a virtue.

I don't think so. In order to be virtuous, one should have some skin in the game. I would respect dedicated pacifists in Kyiv a lot more. I wouldn't agree with them, but at least they would be ready to face pretty stark consequences of their philosophical belief.

Living in the Silicon Valley and proclaiming yourself virtuous pacifist comes at negligible personal cost.

That's kind of like saying that not being a murderer only has moral value if you're constantly under mortal threat yourself.

I don't really see the comparison. Not being a murderer isn't a virtue, it is just normal behavior for 99,9 per cent of the population.

First of all no one declared themselves a virtuous pacifist.

People don't participate in murder and they think others shouldn't either.

People don't participate in wars (which are essentially large scale murder) and they think others shouldn't.

Murder happens anyway. War happens anyway.

Yet if someone says 'war bad' people jump and say 'virtue signaling', but no one does that when people say 'murder bad'.

There's some really weird moral entanglement happening in the minds of people that are so eager to call out virtue signaling.

Virtue isn't childish, shooting telegraphed signals to be perceived as virtuous regardless of your true nature is childish. Also, using a one dimensional, stereotypical storybook definition of virtue (and then trying to foist that on others) is also childish.

Agreed. It's the most important and impactful use case. All else are a set of parlor tricks in comparison.

I don’t think a lot of companies care whether they lose their luster to techies since corporations and most individuals will still buy their product. MSFT was $12 in 2000 (when they had their antitrust lawsuit) and is $400 now.

At the end of the day the US.mil is spending billions to trillions of dollars. I'm not exactly sure what you mean by lose your luster, but becoming part of the military industrial complex is generally a way to bury yourself in deep piles of gold.

> a way to bury yourself in deep piles of gold

Unfortunately, no deep piles of gold without deep piles of corpses. It is inevitable, though. Prompted by the US military, other countries have also always pioneered or acquired advance tech, and I don't see why AI would be any different: Never send a human to do a machine's job is as ominous now as it is dystopian as machines increasingly become more human-like.

There will always be corpses.

Do you want American corpses? Or somebody elses?

I think you answered it yourself. The main way from cool to not cool is to be buried in "piles of gold".

> Claiming they are losing their luster, because they aren’t shaking the earth with new models every quarter, seems a little ridiculous.

If that's the foundation your luster is built on - then it's not really ridiculous.

GPT popularized LLMs to the world with GPT-3, not too long before GPT-4 came out. They made a lot of big, cool changes shortly after GPT-4 - and everyone in their mother announced LLM projects and integrations in that time.

It's been about 9 months now, and not a whole lot has happened in the space.

It's almost as if the law of diminishing returns has kicked in.

GPT-3 came out 3 years before 4.

GPT-3.5 is when LLMs start to get "main stream". That's about 4.5 months before the GPT-4 release.

Keep in mind GPT-3.5 is not an overnight craze. It takes months before normal people even know what it is.

>GPT-3.5 is when LLMs start to get "main stream".

To the general public sure but not research which is what produces the models.

The idea that diminishing returns has hit because there hasn't been a new SOTA model in 9 months is ridiculous. Models take months just to train. Open AI sat on 4 for over half a year after training was done just red-teaming it.

I never bought into ethical questions. It's trained on publicly available data as far as I understand. What's the most unethical thing it can do?

My experience is limited. I got it to berate me with a jailbreak. I asked it to do so, so the onus is on me to be able to handle the response.

I'm trying to think of unethical things it can do that are not in the realm of "you asked it for that information, just as you would have searched on Google", but I can only think of things like "how to make a bomb", suicide related instructions, etc which I would place in the "sharp knife" category. One has to be able to handle it before using it.

It's been increasingly giving the canned "As an AI language model ..." response for stuff that's not even unethical, just dicey, for example.

One recent example in the news was the AI generated p*rn of Taylor Swift. From what I read, the people who made it used Bing, which is based on OpenAI’s tech.

This is more sensationalism than ethical issue. Whatever they did they could do, and probably do better, using publicly available tools like Stable Diffusion.

or just photoshop. The only thing these tools did was make it easier. I don't think the AI aspect adds anything for this comparison.

An argument can be made that "more is different." By making it easier to do something, you're increasing the supply, possibly even taking something that used to be a rare edge case and making it a common occurrence, which can pose problems in and of itself.

Put in a different context: The exploits are out there. Are you saying we shouldn't publish them?

Deepfakes are going to become a concern of everyday life whether you stop OpenAI from generating them or not. The cat is out of the proverbial bag. We as a society need to adjust to treating this sort of content skeptically, and I see no more appropriate way than letting a bunch of fake celebrity porn circulate.

What scares me about deepfakes is not the porn, it's the scams. The scams can actually destroy lives. We need to start ratcheting up social skepticism asap.

You probably don't care about the porn cause I'm assuming you're a man, but it can ruin lives too.

It can only ruin lives if people believe it's real. Until recently, that was a reasonable belief; now it's not. People will catch on and society will adapt.

It's not like the technology is going to disappear.

I mean, the same applies to scams, scams only work if people believe them.

Right - as I said, we need to ramp up social skepticism, fast. Not as in some kind of utopian vision, but "the amount of fake information will be moving from a trickle to a flood soon, there's nothing you can do about that, so brace yourselves".

The specific policies of OpenAI or Google or whatnot are irrelevant. The technology is out of the bag.

It's more dangerous if it's uncommon. It's knowledge that protects people and not a bunch of annoying "AI safety" "researchers" selling the lie that "AI is safe". Truth is those morons only have a job because they help companies save face and create a moat around this new technology where new competitors will be required to have "AI safety" teams & solutions. What have "AI safety" achieved so far besides making models dumber and annoying to use?

You are talking like it's something bad. Kids are learning AI and computing instead of drugs and guns. And nobody is hurt.

Perhaps GPT-4 is losing its luster because the more people actually use it, they go from "wow that's amazing" to "amazing, yes, but..."? And the "but" looms larger and larger with more time and more exposure?

Note well: I haven't actually used it myself, so I'm speculating (guessing) rather than saying that this is how it is.

i got a feeling this is beginning to happen all over the place, I'm really curious to see where the hype train ends up at the end of this year.

This space is growing by leaps and bounds. It's not so much the passage of time as it is the number of notable advancements that is dictating the pace.

Perhaps just me, but responses are way worse than it was few months ago. Now the system just makes shit up and says "Yes you are right" when you catch it on BS.

It is practically unusable and I'll likely cancel paid plan soon.

It was always like this ("Now the system just makes shit up and says 'Yes you are right' when you catch it on BS."). The scales are just falling from your eyes as the novelty fades.

While what you say may well be true, I do have reasonably objective observations of it's deterioration with making up BS.

Convinced hey do it on purpose.

> GPTs are a little better, but I still don't see the product market fit.

If ChatGPT doesn't have product-market fit, what actually has?

GP meant Custom GPTs. Confusing names for sure.

Googlers are wishing OpenAI could vanish as it makes them look like the IBM-lookalike they are.

Here are some hilarious highlights: https://twitter.com/Suhail/status/1757573182138290284

I've had plenty of dumb policy violation misfires like that with ChatGPT, and got banned from Bing (which uses OpenAI API, not GPT4 at the time I think) for it the day it launched.

IMO, these examples are a result of Google's AI safety team being overly conservative and overly simplistic in their approaches.

Google DeepMind is still an AI research powerhouse that is producing a ton of innovation both internal and publicly published.

> Frankly, OpenAI seems to be losing its luster, and fast.

I don't think it's hugely surprising given the massive hype. No doubt OpenAI are doing impressive things, but it's normal for the market to over value it initially as everyone tries to get onboard, and then for it to fall back to a more sensible level.

Where is Ilya?! (Sutskever)

I think OpenAI will do fine, but I have doubts about ChatGPT as a product. It’s just a chat UI, and I’m not convinced the UI will be chat 3 years from now.

Personally, the chat UI is the main limiting factor in my own adoption, because a) it’s not in the tool I’m trying to use, and b) it’s quicker for me to do the work than describe the work I need doing.

I suppose it depends what you use it for; my time in search engine has reduced massively - and so has time 'not in the tool I'm trying to use' because it's been so much faster for me to find answers to some queries with ChatGPT than a search engine.

I'm not particularly interested in having it outright program for me (other than say to sketch how to do something as inspiration, which I'll rewrite rather than copy) because I think typically I'd want to do it a certain way and it would take far longer to NLP an LLM to write it in whatever syntax than to WhateverSyntaxProgram it myself.

Coding assistants copy your style to a fault. You got to be careful about things like typos in comments, or it'll start suggesting sloppy code as well. And conversely you have to be careful about overly bureaucratic conventions (doc comments for things entirely described by their name, etc.), or it will suggest overly wrapped hypercorporate code.

But used as autocomplete, it's definitively a time saver. Most of us read faster than we type.

I assumed that was not what we were talking about, because I replied to:

> Personally, the chat UI is the main limiting factor in my own adoption, because a) it’s not in the tool I’m trying to use, [...]

though I haven't tried it through some combination of it the effort to set it up & it not particularly appealing to me anyway. The best it could possibly be would be like pair programming (back seat) with someone who does things the same way as you, and reviewing their code. I read faster than I type, but probably don't review non-trivial code faster than I type it. (That's not a brag, I just mean I think it's harder and takes longer to reason about something you haven't written, to understand it, and be confident you're not missing anything or haven't (both) failed to consider xyz.)

I interact with ChatGPT by voice pretty often, they have the best speech recognition I’ve ever seen. I can switch between languages (English, French, German) mid-sentence, think aloud, stop mid sentence, the correct what I just said, use highly technical terms (even describe code), I don’t even double check anymore because it’s almost always transcribed correctly. They can ~easily evolve the product to a more generalized conversation UX instead of just a text based chat.

If only something like that was available on Android. I cannot dictate messages as my phone is in English, but most of my messages are in German or French. Or it's almost impossible to search for a non-English song when driving.

Multi languages would be so useful for me.

This. Whisper is phenomenal. Have you tried the conversational mode? I would love to be able to use that in a more customized agent. I know you can use the conversation mode with a custom GPT but I’d prefer to write dynamic prompts programmatically. Would be great for a generalized personal assistant that can take notes, send/read email, texts, etc. could be a good filter on social notifications?

Though the TTS side has some trouble switching languages if only single words are embedded. A single German word inside an English sentence can really get butchered. More training needed on multilingual texts (and perhaps preserving italics). But anyways this is really only an issue for early language learning applications in my experience.

The conversational mode is fascinating. But it’s frustrating to use for the same reasons ChatGPT can be annoying: it doesn’t remember that well previous messages, you end up in weird Alzheimer-ish discussions where the interlocutor speaks perfectly but has the memory of a clownfish

Do you use the voice chat in the ChatGPT app?

In my experience, stopping to talk even for a moment already makes it submit. This makes a real conversation with pauses for thought difficult, because of the need to hurry before it cuts off.

FWIW if you hold down the big white button it won't submit until you release it. I had no idea this was a thing until seeing someone tweet about it.

Thanks, I had no idea!

I think you’re describing the conversation mode (started via the headphones icon), I also have issues using it. But you can also dictate a message, on iOS it’s the little gray wave icon on the right of the text input. With this mode there is no auto submission.

For me, voice is just a different UX for the same underlying model of chat. I'm sure it's good, but I'm not going to sit at my computer talking to it, and in fact I think talking may be a worse signal to noise ratio than typing, as I can easily use shortcuts with written text.

Not when you’re on your computer, but you can do it on your phone when you’re walking in the street or commuting.

You can easily talk while you’re doing something else.

Nonsense. Anyone who regularly uses the top models knows that GPT-4 still leads by a clear margin

And yet, day to day, I'm using Bard/Gemini because, for most stuff, it's enough and sometimes clearer and better and the interface makes more sense.

I've been anti-Google for a while now so I'm not biased.

I don't think openAI have this sown up.

Interesting take, interesting reasons.

I could understand the sentiment when you think that OpenAI is really doubling down just on LLMs recently, and forgoing a ton of research in other fronts.

They’re rapidly iterating though, and it’s refreshing to see them try a bunch of new things so quickly while every other company is comparatively slow to release anything.

I mean they just happened to train the biggest, most fine tuned model on the most data out of everyone I guess.

Transformers were invented with the support of Google (by the researchers, not by Google).

Open community has been creating better and better models with a group effort; like how ML works itself, it's way easier to try 100,000 ideas on a small scale than it is to try a couple of ideas on a large scale.

Sam publicly asking for a 10x bigger power grid and 7 trillion dollars is a pretty clear sign that they're out of short to medium-term ideas other than "MOAR PARAMETERS".

Well, he also wanted a shit ton of money so that OpenAI coupd build its own silicon, after most of the real world money generated by the AI hype went to nVidia.

Just imagine what valuation OpenAI would have as a grid monopolist combined with nVidia, ARM, Intel and AMD! Hundreds of trillions of dollars!

You think his short to medium term plan is to raise $7tn to build dozens of fabs?

To me plugins were an improvement, I often use ‘AI diagrams’ plugin and ask it to draw sequence diagrams.

> GPT-5 has to be incredibly good at this point, and I'm not sure that it will be.

My guess is it isnt, these systems are hard to trust, and the rhetoric "were aiming for AGI" suggests to me that they know this and AGI might be the only surefire way out.

If you tried to replace all of a devs duties with current LLMs it would be a disaster, making sense of all that info requires focus and background thinking processes simulataneously which i dont believe we have yet.

> If you tried to replace all of a devs duties with current LLMs it would be a disaster,

Overall a chatbot like GPT-4 may be useful, but not that useful as it stands.

If you can write well, it's not really going to improve your writing. Granted, you can automate a few tasks, but it does not give you 10X or even 2X improvement as sometimes advertised.

It might be useful here and there for coding, but it's not reliable.

To be honest, I hate takes like this. ChatGPT, which basically revolutionized the whole AI industry and the public's imagination about what AI can do, was released not even 15 months ago, and since then they have consistently released huge upgrades (GPT 4 just a couple months later) and numerous products since then. I still haven't used another model that comes close to GPT 4. But since it's been, say, all of 23 hours since OpenAI released a new product (memory) they're "losing their luster".

The same nonsense happened with Apple, where like a month after they first released Apple Watch people were yelling "What's next???!!!! Apple is dying without Steve Jobs!"

Plugins are in theory good, but the hurdle to developing and deploying them combined with only being able to use them with a subscription was kind of a killer.

GPTs are also pretty good, and being able to invoke them in regular chat is also handy, but the lack of monetization and the ability to easily surface them outside of chatgpt is also kind of a problem. These problems are more fixable than the plugin issue IMO since I think the architecture of plugins is a limiting factor.

> The light that burns twice as bright burns half as long - and you have burned so very, very brightly, Roy.

gpt4 is not worth $22 a month. slow af and you get similar results with gpt3.5. the free perplexity internet search is bounds better than that bing thing. i thought the file upload would be worth it, but no, not worth that much money per month.

I'll get downvoted to oblivion, but I think people underestimate the impact that their productization of the GPT in the chat format really led to a virality that likely is not entirely justified just by the underlying product alone. LLMs had been around for several years, it was just a royal pain to use. They definitely were the pioneers in democratizing it to folks, and it occupied a significant slice of mindshare of society for quite a bit. But I suspect it is only natural that it'll recede to a more appropriate level, where this is still an important and incredible piece of tech, but it will stop having the feel that "OMG THIS IS GOING TO TAKE OVER THE WORLD", because it prob. won't... at least not at the pace which popular media would have you believe.

Why haven't plugins become more of a "thing"?

Genuinely curious if the news today about Sora has changed your opinion at all https://openai.com/sora

Unpopular opinion… but IMO almost all of Karpathy’s fame an influence come from being an incredible educator and communicator.

Relative to his level of fame, his actual level of contribution as far as pushing forward AI, I’m not so sure about.

I deeply appreciate his educational content and I’m glad that it has led to a way for him to gain influence and sustain a career. Hopefully he’s rich enough from that that he can focus 100% on educational stuff!

> Relative to his level of fame, his actual level of contribution as far as pushing forward AI, I’m not so sure about.

Are you sure about your perspective?

https://scholar.google.com/citations?view_op=view_citation&h...

May be this is what you wanted to share? https://scholar.google.com/citations?user=l8WuQJgAAAAJ&hl=en

ImageNet was very influential, but this just shows he was eighth author on a twelve author paper from almost a decade ago. Is there better evidence of sustained contributions to the field?

Hm, well, I see on his resume that he was a founder of OpenAI, recruited to be Tesla's head of AI, went back to OpenAI, and also has the most viewed educational videos in this space.

So, he has made theoretical contributions to the space, contributions to prominent private organizations in the space, and broadly educated others about the space. What more are you looking for?

the commenter is probably a junior boot camp web dev...

Tesla fumbled big on AI, and as for his work at OpenAI, he just left, had he been good enough they would have made him a financial offer that would have made him continue. But, I'll give him that, he seems to be a really good teacher.

I doubt he left because he wasn't being compensated fairly.

People just get bored and go do something else for a while sometimes. Or he's got some beef.

Not everyone is purely motivated by money so. I know that the moment I decided to quit or switch jobs, no, and I mean litterally no, amount of money would change my mind.

Me changing can never be used as an appraisal of my old organisation so.

Disclaimer: regarding money, if I get enough in max a year to rezire forever after that, I might be tempted. Which won't happen, because a) I'd just leave a year later anyway and b) nobody would pay me high 7 figures just to not quit.

many SOTA papers for multi-camera deep fusion/birds eye view perception in autonomous driving were based on copying teslas homework after their AI day 2022 talk

You left out a few prepositions like off the space, toward the space, from the space, and down the space.

He’s first author on a ton of those papers. That’s a tenure worthy CV almost anywhere. Gimme a break.

OP shared a link to just the Imagenet paper.

Agree he had a decent overall track record at Stanford, but that’s not how tenure works — it might have got his foot in the door as an assistant professor somewhere. He chose a much more lucrative path.

Its IMHO too. His contribution to educational content is incredible, and very few individuals have the ability to explain things the way he does. However, I am also unsure about his contribution to the field itself. It is a side effect of working in the industry on the product side. You don't have a chance to publish papers, and you don't want to reveal your secrets or bugs to everyone.

Education and communication is important. It brings new people into the field, and helps grow those that are already part of the field, both of which are essential to long term growth and progress. Using phrases like “actual contribution” to refer to non-educational acts is entirely dismissive to the role that great educators play to in the march for progress. Where would you be today if such education was unavailable?

He contributed to pushing forward AI, no “actual” about it. The loss of a great educator should be viewed with just as much sadness as the loss of a great engineer.

His job at Tesla or OpenAI wasn’t as an educator though. I think a clearer version of my point is that most of his impact has come from activities he has done “on the side” and hasn’t gotten paid for from his job. I’m hopeful it can be his main gig now given that YouTube creators seem to be making more money.

his actual level of contribution as far as pushing forward AI

He did pioneering research in image captioning - aligning visual and textual semantic spaces - the conceptual foundation of modern image generators. He also did an excellent analysis of RNNs - one of the first and best explanations of what happens under the hood of a language model.

A hackernews comment section is one of the least legitimate forums imaginable for the public reading of somebody's resume. Congress, maybe.

Honestly I have more faith in hacker news users than congress.

I wish more of us would run for Congress. I'd much rather have a government of technocrats of various stripes than ex lawyers and rich business types.

IMO governments, like websites, should be boring but effective, focused on small day to day improvements, not all flash and empty marketing chasing cultural trends...

I don't know, while lawyers and MBA's are not who I would choose to run the country, I am not sure the I would pick people with the motto "run fast and break things" in charge either.

But what if we implemented agile and scrum? :^)

Imagine the retros!

>But what if we implemented agile and scrum?

thats what i am afraid of.

It would be an improvement over "just break things", no?

That's Facebook / Meta. They do not represent us

It's already broken. We could at least have fast internet and net neutrality.

You’re always gonna have a ton of lawyers in congress and state legislatures because if you were interested in law enough to become a lawyer you are disproportionately likely to want to write laws.

This is like saying more lawyers should be writing software. Lawyers have extensive education and experience in the law and so work with..the law.

I don't discount the value of having expertise in law among those who write our laws. That said, I think that lawyers have their own significant blind spots as well. A lawyer is an expert on the law, but also will often be out of touch with the actual lives and needs of the people. Ideally, Congress should have lawyers - but also plenty of non lawyers (from diverse backgrounds), who can bring their own experiences and perspectives that lawyers lack.

Well, that's how you get "laws by lawyers, for lawyers", like "software by engineers, for engineers".

Maybe Congress needs the equivalent of UX and product types who actually care about what the people want... and can explain how it works to us in fancy how-to videos.

> Maybe Congress needs the equivalent of UX and product types who actually care about what the people want...

Members of Congress have plenty of support devoted to both what people say they want and what they actually positively respond to. That’s...the entire political side of the operation.

Look into the Technocracy movement and why it failed.

If you like this idea, read the Fifth Risk by Michael Lewis (he also wrote the Big Short which you may have seen). The book essentially argues that this is already the case in many (crucially not all) government departments. I like to TL;DR the book to other people as "the deep state is good, actually". Of course, the government itself is absolutely not helmed by technocratic politicians.

Quentin? Your second given name, I mean. Mine is Romanus. (Just idly curious, since we share the first.)

I don't know about the US, but the simple answer in the UK IMO is that politics doesn't pay enough. So you get egos, old money, and people with concurrent business interests.

But try convincing a democracy that politicians should be paid more.

Doesn't pay enough?

I believe the basic pay is £86k. They're not brain surgeons or rocket scientists, so even that is not that bad.

But I believe the average gravy train bumps this up 3X with extras.

It's a literal gravy train of subsidies and expenses and allowances! Sure the basic pay is, well, it's arguably not that bad ... but the gravy on top is tremendous. Not to mention the network contacts which plug their gravy train into the more lucrative gravy superhighway later.

> They're not brain surgeons or rocket scientists

Yeah, voters don't want to pay MPs more. Yet when voters are asked, they want highly intelligent, motivated people. They want them to have technical expertise, which means time spent in higher education. Then they want them to work a full time job in Parliament during the week, but also be open to constituency concerns on the weekend. And once all of this is pointed out, voters concede that maybe MPs deserve to be paid on par with professionals like doctors. (It's a different matter that UK doctors are underpaid).

> But I believe the average gravy train bumps this up 3X with extras.

Citation needed. They're on a shorter leash now with expenses. Don't go citing one or two bad apples either, show us what the median MP claims as expenses. According to you, it should be around £170k a year.

In general, politicians and their aides in the UK are underpaid. Most capable people find they're better off working in Canary Wharf or elsewhere in London. An example is the head of economic policy for the Labour Party earning £50k while writing policy for a £2 trn economy. (https://www.economist.com/britain/2023/01/19/british-politic...)

Your first point has always interested me, as it's unclear how much technical expertise these people have. They just employ Special Advisors to do the 'difficult' work for them (again, something not included in their expenses but, of course, is a benefit). And the manner in which reshuffles happen when the Education Secretary suddenly becomes the Enviroment Secretary whilst having no experience of either.

Anyway, I'm very sure there are good MP's, but I'll not go so far as to say these people are underpaid.

I plugged the question into AI ... see below. Not to mention the subsidised "everything". Holidays in mates villas (and what mates, eh). The "director" positions on various companies, and, and ... it's not just the monetary value of these things. It's an absolute gravy train.

Generated Hypothetical Answers: we can provide some hypothetical scenarios based on varying levels of responsibility:

Scenario 1: Backbench MP without additional roles:

    Salary: £86,584
    Maximum Expense Claims:
        Office: £85,000
        Accommodation (Constituency only): £9,300
        Travel: Assuming moderate travel expenses, let's estimate £10,000
        Other Expenses: £5,000

Total: £86,584 + £85,000 + £9,300 + £10,000 + £5,000 = £195,884

Scenario 2: MP with Ministerial role and chairing a committee:

    Salary: £86,584 + Ministerial salary (e.g., £50,000)
    Expense Claims: Similar to Scenario 1, let's use the same estimates

Committee Chair allowance: £11,600

Total: £86,584 + £50,000 + £85,000 + £9,300 + £10,000 + £5,000 + £11,600 = £257,484

Remember: These are just hypothetical examples, and the actual value for any individual MP can be significantly higher or lower depending on their specific circumstances.

Voters want people representatives that will work out of civic duty. but is H.O.A.'s have taught us anything its that the people who claim to be acting out of civic duty to make a better place are mostly petty tyrants.

sure it would be nice if we could have Aristotelian philosopher kings style politicians but that's not human nature.

Congress pays great because you can ignore your job to be courted by lobbyists, get paid to rubber stamp laws from ALEC, and insider trade based on foreknowledge of what laws are about to occur.

>Relative to his level of fame, his actual level of contribution as far as pushing forward AI, I’m not so sure about.

He lead a team of one of the most common uses of DNNs, if that isn't 'pushing AI forward', I think you're confused. It's certainly pushing it forward quite a bit more than the publishing game where 99% of the papers are ignored by the people actually building real applications of AI.

I would argue that's far more valuable.

>his actual level of contribution as far as pushing forward AI, I’m not so sure about.

I mean, I don't know why people still try to devalue educating the masses. Anyone who's had to knowledge share know how hard it is to make a concise but approachable explanation for someone who knows relatively little about the field.

In addition, he's still probably in a standing well above the 80% mark in terms of technical prowess. even without influencer fame I'm sure he can get into any studio he wishes.

Disagreeing here! I think we often overlook the value of excellent educational materials. Karpathy has truly revitalized the AI field, which is often cluttered with overly complex and dense mathematical descriptions.

Take CS 231, for example, which stands as one of Stanford's most popular AI/ML courses. Think about the number of students who have taken this class from around 2015 to 2017 and have since advanced in AI. It's fair to say a good chunk of credit goes back to that course.

Instructors who break it down, showing you how straightforward it can be, guiding you through each step, are invaluable. They play a crucial role in lowering the entry barriers into the field. In the long haul, it's these newcomers, brought into AI by resources like those created by Karpathy, who will drive some of the most significant breakthroughs. For instance, his "Hacker's Guide to Neural Networks," now almost a decade old, provided me with one of the clearest 'aha' moments in understanding back-propagation.

People like the grandparent think innovation and advancement happens in isolation.

I don’t think we disagree. Education is crucial and the value is enormous, but this hasn’t been what he was paid for in the past. I am hopeful that he finds a way to make this his job more directly than at Tesla or OpenAI as the whole world will benefit.

What a tone def elitist thing to say, you have no tact.

Tu quoque.

In 2015 he wrote this blog post about "The Unreasonable Effectiveness of Recurrent Neural Networks": https://karpathy.github.io/2015/05/21/rnn-effectiveness/

That blog post inspired Alec Radford at Open AI to do the research that produced the "Unsupervised sentiment neuron": https://openai.com/research/unsupervised-sentiment-neuron

Open AI decided to see what happened if they scaled up that model by leveraging the new Transformer architecture invented at Google, and they created something called GPT: https://cdn.openai.com/research-covers/language-unsupervised...

Also in that article he says

"In fact, I’d go as far as to say that

    The concept of attention is the most interesting recent architectural innovation in neural networks."

when the initial attention paper was less than a year old, and two years before the transformer paper.

I read that post recently and it felt prescient to someone who has not been deeply involved in ML

Even the HN discussion around this had comments like "this feels my baby learning to speak.." which are the same comparisons people were saying when LLMs hit mainstream in 2022

I had forgotten it's existence by now, but I remember reading this post all those years back. Damn. I also remember thinking that this would be so cool if RNNs didn't suck at long contexts, even with an attention mechanism. In some sense, the only thing he needed was the transformer architecture and a "fuck, let's just do it" compute budget to end up at ChatGPT. He was always at the frontier of this field.

Is it stated somewhere that Radford was inspired by that blog post?

I tried to find the where I heard that Radford was inspired by that blog post, but the closest thing I found is that in the "Sentiment Neuron" paper (Learning to Generate Reviews and Discovering Sentiment: https://arxiv.org/pdf/1704.01444.pdf), in the "Discussion and Future Work" section they mention this Karpathy paper from 2015: Visualizing and Understanding Recurrent Networks https://arxiv.org/abs/1506.02078

He also wrote about the concept of Software 3.0

I think the ability to teach is a direct outcome of the ability to think and articulate ideas clearly. This is a meta skill that will make a person effective in any area of work.

I'd say that that his work on AI has been significant and his ability to teach has contributed to that greatly.

That's incredibly impolite and totally without foundation. You make him look like a peasant :-)

What do you know about his work?

He's been leading the vision team at Tesla, implementing in the field all the papers that were available in the subject of autonomous driving and vision (he explicitly wrote that). He has not published about it surely due to obligations with Tesla.

It’s like saying Rick Rubin didn’t do much for music because he doesn’t play any instrument.

Would have to agree. Looking at Karpathys research career it’s hard to pin point something and say he’s the inventor of so and so. There are plenty of other researchers for whol you can easily say he’s the inventor of so and so and they have much lesser fame than Karpathy, for example Kaiming He for ResNet, John Schulman for PPO etc.

I don’t see that as an issue though, just a natural consequence of his great work in teaching neural networks!

> Relative to his level of fame, his actual level of contribution as far as pushing forward AI, I’m not so sure about.

I'd agree with that, however I've always wondered how easy it is for folks at that level to get hands on keyboards and not wind up spending their days polishing slide decks for talks instead.

> My immediate plan is to work on my personal projects and see what happens. Those of you who’ve followed me for a while may have a sense for what that might look like

Does anyone here know?

His profile previously said building JARVIS @ OpenAI

Maybe he left because he finished building it.

I imagine he's talking about his youtube series (look forward to it!)

He's also into open source models /r/localllama

He was building small, efficient versions of gpt 3 in a video series

You can check his GitHub: https://github.com/karpathy

In his free time, I hope he writes some more fiction, I really liked https://karpathy.github.io/2015/11/14/ai/

I strongly agree. It was a great short story! Would love to see it fleshed out sometime.

Damn he’d be pretty good at a kids party with the story telling and cubing and yet he can teach and do! Has he climbed Everest yet?

To dedicate himself fully to his YouTube channel? I am looking forward! His content is amazing

I am also. About 8 years ago, the Python miniatures he published for recurrent networks like char-rnn, etc. we’re so helpful for learning.

I imagine doing so is leaving millions on the table. I’m sure he is well off and can retire early, but even so, millions are a lot of money.

Pretty sure he’s leaving BILLIONS on the table :)

It seems like he (re)joined OpenAI almost exactly 1 year ago: https://twitter.com/karpathy/status/1623476659369443328

New theory: Karpathy was short on money, so he waited exactly 1 year to vest 25% of his options.

Short on money after being an exec at tesla during a huge rise in its stocks? More likely he has too much money and maybe doesn't really want or need to work and is doing passion projects instead

I prefer my theory, it's funnier although less probable, I agree

Related: "Andrej Karpathy Departs OpenAI"[0] (159 points, 2 hours ago, 71 comments)

[0]: https://news.ycombinator.com/item?id=39365288

We'll merge those comments hither since that article is hardwalled. One moment please.

I'm judging from his pinned tweet, "The hottest new programming language is English", that "those of you who know me know what I'm working on ;)" message at the end of this seems like a nod to developer tools of some kind. Which would track for a tech visionary, a hacker can't resist making himself better tools I guess.

Anybody have better info than my idle guess?

He has speculated a few times about an LLM based OS. That’d be way cool.

Honestly? Sounds like a nightmare. I mean, some LLM integrated into a OS, ok, might make sense, but the OS based on LLM is not something I would want with the current state of the art.

Features like function calling are moving in that direction. Microsoft also seems to have plans to deeply integrate LLMs into its OS and if they do a good job it could become a primary way to interact with its features and programs. Considering the progress made on image generation models I could image a special purpose model that is specifically trained on operating APIs and producing good results. The big hurdle would be building the APIs that don't exist for the tools that people like to use. I'm sure there are interesting ways you could think of generating labeled data for actions in various programs.

Why?

Non-deterministic outputs... For one (rather important) thing.

I can think of lots of reasons why non-deterministic outputs at the OS level is a bad idea, but what are the benefits?

Not to jump in for someone else but your use of “OS level” prompts me to opine: I think the features of a meaningful new OS would extend far beyond the programmatic level of the kernel, the drivers, the dependencies, etc. A “new OS” could just be Linux with some cool UX innovations on top enabled by ensembles of lightweight, purpose built LLMs. Think window management, file management, password management, etc.

For one potentially compelling example that happily (sadly?) isn’t using LLMs: the SimulaVR people are developing their own Linux fork of some kind, claiming it’s necessary for comfortable VR use for office work. And I sorta believe them!

non-deterministic outputs from what? I'm misunderstanding your idea.

My point was that non-deterministic outputs make LLM's fundamentally a BAD foundation for something like an OS.

Natural language interfaces belong at the periphery, as the interface between the human and the machine. Other than that, I want my computers dumb as rocks, really fast, any totally predictable - which is basically the opposite of what you get from LLM's.

> LLM based OS

Isn't that is what Rabbit R1 is? https://www.rabbit.tech/

He'll become a full time YouTuber.

> The hottest new programming language is English

Television content for children is often called 'Children's Programming'

Andrej inspires millions. He’s the Taylor Swift of AI/NNs.

I really hope he’s next gig is at an actually Open AI company.

I think, Michael Jordan of AI is a much better comparison, in terms of work required, pain he has to go through, determination, contribution to the world, being a role model, etc...

Michael Jordan is already taken as a username in the AI space.

Great point. How did I miss that? He's been a bit quiet lately, so not on top on the mind. I don't not how far should we push the spots analogies. Kobe Bryant? Geoff Hinton would be Pele then, in terms of uniqueness.

This is quite the loss for OpenAI.

And no one seems to have heard what Ilya Sutskever's status is at OpenAI as well.

If I was Sam Altman, Ilya would be on a short leash. I’m actually amazed he wasn’t booted already.

He might be talented, but if he can’t be trusted, he needs to go.

That’s why you are not Altman by huge margin ( and i am not a fan of latter).

Building business requires latticework of talented people doing job properly. And building a system of checks and controls for less trusted people.

> He might be talented, but if he can’t be trusted, he needs to go.

Ironically that may be exactly what Sutskever thought about Altman.

Frankly I consider every moment of silence from Ilya a reason to keep my expectations above the floor and I imagine a lot of people feel the same with how drunk on ai doomerism and gatekeeping Ilya was. The only downside to the silence is it gives him time to try and shake off the association between his name and that circus with the board.

He quit to build a blogging platform.

https://x.com/karpathy/status/1751350002281300461?s=20

Likely as a side thing

Given Karpathy's draw towards teaching/educational content, I've wondered where he falls on the spectrum between Sam Altman's interpretation of "Open" in OpenAI, and someone on the opposite end (like Musk).

I'd imagine if one was fully onboard with the AI/LLM commercialization train, there's no better spot than OpenAI right now.

I think he's actually more towards the end of "actually open", which is not a place either Elon nor Sam are at. Grok and OpenAI don't openly publish or freely release much of their work. Andrej however has released a lot of his work for free ever since he was a PhD student.

That’s why I suspect he’ll be joining Meta next.

"I told them Xerox has got to get itself together, because there's no way a big company can take advantage of things moving this fast. People will get frustrated and start their own companies."

—Carver Mead, 1979 (employee at Xerox PARC), discussing why Xerox needed to focus more on adopting integrated circuits into the computers they had already developed, instead of continuing to just make increasingly-obsolete copiers.

I don't think that is an apt metaphor. Imo Openai is Apple and Google is Parc. Google experiencing a similar issue to parc where they invented transformers but have been unable to capture the value so far due to being focused on ads revenue.

Great nuanced distinction.

----

"Xerox's top executives were for the most part salesmen of copy machines. From these leased behemoths the revenue stream was as tangible as the `click` of the meters counting off copies, for which the customer paid Xerox so many cents per page (and from which Xerox paid its salespersons their commissions). Noticing their eyes narrow [at R&D's attempts at asking to market their computer, one] could almost hear them thinking: 'If there is no paper to be copied, where's the `click`?' In other words: 'How will I get paid?' "

—Michael Hiltzik's "Dealers of Lightning" (p272)

It seems odd that Xerox bothered with the research lab at all then. Why not only research how to make copier's cheaper and more compelling if company culture is Mad Men, copier edition?

Always the same story, some boss wants to get noticed ask underlings to make something cool. Underlings make something cool, bosses boss get scared his position will be taken, orders a shutdown of it and to focus on what matters.

What value? I doubt any LLM player is making any profit. Sure, NVIDIA is, but that's because "in a gold rush, sell shovels" is an eternal advice.

> 1979 ... increasingly-obsolete copiers.

In 1979, I doubt copiers were 'increasingly obsolete'; I'd expect the market was growing rapidly. Laser printers, email, the Internet, didn't yet exist; PCs barely existed, and not in offices. Almost everywhere would have used typewriters, I suppose.

Xerox's copier sales peak was in the early 70's, and then multiple international companies [primarily in Japan] began creating better, less expensive copiers. By the late 70's, Xerox was massively losing marketshare [to both competitors, and to blossoming word processing technologies].

>Laser printers, email, the Internet, didn't yet exist

Actually, all three did; the latter was in the form of ARPANET [to be technical, not "The Internet"].

One of us needs sources, and I think it's you :)

> Actually, all three did; the latter was in the form of ARPANET [to be technical, not "The Internet"].

True, but a technicality. Very few people knew they even existed, and they had zero impact on Xerox copier sales.

>Very few people knew they even existed

Does not mean they did not exist. See citations, below:

https://en.wikipedia.org/wiki/Laser_printing (see 2nd intro paragraph)

https://en.wikipedia.org/wiki/History_of_email (see 3rd intro paragraph)

? Note the sentence immediately before the one you quoted.

Where is this quote from? Can't easily Google it

Dealers of Lightning; the best "tech sociology" book I've read, yet; was recommended here on HN as "must read" and it is absolutely a MUST READ.

I don't mean to detract from your point (if anything, I suppose I'm obliquely supporting it), but I feel compelled to say that it's really weird to see Carver Mead cited in the context of "employee at Xerox PARC", because I mostly know him as one half of "Mead/Conway", i.e. the duo who arguably supplied the computational (dare I say "algorithmic"?) rocket fuel for the unbelievably wild progress of chips in the 1990s [1] [2].

[1] https://en.wikipedia.org/wiki/Mead%E2%80%93Conway_VLSI_chip_...

[2] https://en.wikipedia.org/wiki/Lynn_Conway

The textbook they wrote together was while both were collaborating at PARC (Mead was at CalTech, then, too); they wrote it to add credibility to their VLSI theories, which at the time most experts believed would lead to thermal runaway (i.e. not stable, long-term, to pack transistors densely).

https://en.wikipedia.org/wiki/Carver_Mead

Learning about the interconnectedness of all this historic intellectual "brain theft," keeps me excited for an AGI-future, post-copyright/IP. What are we going to accomplish [globally] when you can't just own brilliant ideas?!

What about Ilya Sutskever? Wouldn't it be cool if Andrej and Ilya start truly open AI company.

Funded by what?

investors will line up to fund them. I'd be happy to chip in in "friends and family" round too :)

Investors will line-up to fund a lack of return on their investment? I'd absolutely donate, but that's not what investors want for their money, unfortunately :(

These two individuals can probably raise a billion dollars within a month if they wanted to start their own AI company.

On a 10B valuation )

That would be strong terms, but then they would need to find a way to spend $1B.

Here's a proposal. Send $500M to NVIDIA for GPUs. Send $100M to AMD to balance the odds. Spend $100M to build a new chip as a hail mary.

Spend the rest on $10M comp packages.

How close did I get?

Seems pretty reasonable! Not sure that's the easiest path for them to maximize profit though.

I'd probably say they should:

- allocate the $500M to the new chip, $100M to each of AMD and NVIDIA, then:

- never officially hire any more staff (these founders are 10E6X devs!)

- start "subtly" liquidating the AMD and NVIDIA chips after a year ("Tell HN: IlPathAI are liquidating their GPUs? >I bought a used GH200 off eBay and the shipping label was covering up previous shipping label for Andrej's shipping container treehouse >Are they getting quick cash to finance their foundry run on the new chips? It's that good??").

- Release a vague "alignment" solution on a chatgpt-generated kickstarter clone, take 3 years to "develop" it.

- Raise a series A (maybe a pre-A, or a post-seed. Honestly, maybe even a re-seed with this valuation!) off the hype (some obviously stable diffusion-generated images help here).

- Sell 30% of their shares in a secondary, profit some billions.

- When everyone starts getting suspicious, time to take out those GH200s you "sold" on ebay out of storage (those buyers were just sockpuppets - investors from the family/"friends" round), repackage them in some angled magnesium alloy. Release them to great fanfare. Crowd briefly ecstatic, concern sets in - "this has the same performance as the GH200? That was like 4 years ago!".

- Call the "performance issues" some form of "early access syndrome" and succeed in shifting blame back onto the consumer.

- Release a "performance patch" that in actuality just freaking overclocks and overvolts the device, occasionally secretly training on the user's validation set using an RDMA exfiltration exploit. This gets them to 2028, when the modified firmware on all devices spontaneously causes a meltdown - should've written it in Rust - that should've been a signed int! The fans thought it was suddenly 1773, ran in reverse so fast the whole device melted (aww all that IP down the drain)!

- When asked how on earth that could make any sense, dodge the question with the news that "We just had the unfortunate news that one of the greybeards who wrote the firmware previously at Siemens and then the DoE, has programmed his last PLC. He died glowing peacefully last night surrounded by layered densities of gasses. We are too sad and bankrupt to go on."

- Declare bankruptcy

- Become alt-right pundits on Y.com (If they haven't already wrapped around to AA.com - they managed to grab that domain after some airline went bankrupt after an embarrassing incident involving a 787 Max, the latest Stable Diffusion model, a marketing executive, some loose screws, and a Boeing QA contractor back there who might Not Be Real).

- Start a war with a "totally harmless" post, later admit it was "poorly worded". - Use some saved funds to "find a way" for IlPathAI, Inc. to leave bankruptcy, pivot to a chat app (you actually just buy HipChat again). Resell that after reusing it for a few particularly juicy govt. tenders. Pivot to defense contracting. End up with enough money for the rest of the millennium.

- Write a joint autobiography called "The Alignment Problem", send it to your "kickstarter" backers. Print the book using old school metal typecasting because they forgot TeX, and the current language models only spit out hallucinated Marvel dialog. Screw up the kerning since you learned typesetting on the TikTok page of a French clockwork museum. Claim this was on purpose.

- The whole time, maintain an amazingly educational YouTube channel teaching Machine Learning to those who love to learn.

- Release "AGI" but it's actually just 5 lines of PyTorch that would have solved Tesla's FSD problems with mirrors. Send Douglas Hofstadter a very slightly smaller copy every day until he recurses into true AGI.

---

Well I started out serious at least (OK, only the first and second-to-last bullets were). I do genuinely believe that $100M would not be enough to produce competitive IP right now - You'd likely have to budget a majority of that to the final production run! I wonder how much you'd have to spend on making custom chips to break even with spending the money on research in the performance/model architecture side of things, on average.

> The fans thought it was suddenly 1773, ran in reverse so fast the whole device melted

1) Then the fans should start to get ready for the American Revolution, only three years to go...

2) But actually, all the fans will by that time have read your above comment, so they'll be prepared for what's to come.

3) But actually actually, fans running in reverse will only suck in cold air through the exhaust port and blow out warm through the intake (a bit like politicians?), so they'll still be cooling devices.

The end result would be the same, with only difference that the training/weights might be open-sourced (which is not a great business differentiator, cost/performance is)

Being "open" isn't necessarily incompatible with earning profits. It can in fact be beneficial.

Huh, I just read this exact tweet.

https://x.com/ns123abc/status/1757595970911556012

I thought he left a prominent position awhile ago, but I guess that was Tesla. Best wishes to him, and my thanks for his educational efforts.

I think he'll join x.ai or Tesla in 6 months.

Interesting that he didn't say why he left. He says "my immediate plan is..." meaning that he has no specific long-term plans or doesn't want to talk about them.

You don’t expect people at certain levels splurging out their plans to populace, do you?

Yes.

Please excuse me for asking this. I know Andrej is an excellent instructor. I've watched his videos. But what has been his contribution to the industry (besides teaching of course)?

Real questions here :)

Vested his rsu and left

Exactly. That is the real reason.

Only 1/4th tho?

And got very bored and unhappy with big company issues. And has the perspective from his time at Tesla to know how things only get worse for creativity at that stage.

Its not a good thing if true. Tech and creative folk have to find ways to stick around or the financial folk fill the leadership and decision making space.

It's a hard thing to manage. Tech orgs of ~20 people are just more fun than tech orgs of 200 people, which are more fun that tech orgs of 20,000 people which.. you get the picture.

You can create and encourage small teams, but then they need to coordinate somehow. Coordination & communication overhead grows exponentially. Then you get all the "no silos" guys and then its all over..

Rsu exist before ipo?

Presumably he was already set for life from his Tesla gig, no?

He is one of OpenAI founders. I don’t think he needs to vest anything.

He founded the non profit, not the part that earns money. Non profit founders doesn't have shares.

yeah, he is basically on of the people who got screwed over, but given that he did work for OpenAI he might not thinking about it that way

he was a director at Tesla when the stock 10xed.

The guy has many tens of millions of dollars most likely.

I usually agree but I honestly believe even before OpenAI he was set for life and he will now care more about how exciting the work is and how much it aligns with his interests/values.

Even when you have a yacht, you wanna get a bigger one :)

OpenAI does not have RSUs. They have shares of future profit instead.

Looking at the thread, it is quite amusing to see some founders hopelessly begging Karpathy to work at their tiny startup to realize that they can't afford him as he is exceptionally brilliant in the AI industry.

If OpenAI, Tesla and Google cannot retain him, then probably nobody can. Probably he'll be doing YouTube videos all day long.

I think money is the least of the reasons why those founders wouldn't attract him.

To attract someone at Karpathy's level you would need a project that is both wildly challenging (and yet not the typical startup "challenging" because it's a poorly thought out idea) and requires the kind of resources (compute, data, human brains in vats, etc) that would make your place look far more interesting than OpenAI.

But, hardest of all, you would need startup founders that could tame their egos enough to let someone like Karpathy shine. I haven't talked to a Bay area startup founder in a while who wouldn't completely squander that kind of talent by attempting to force their own half-baked ideas on him, and then try to force him out months later when he couldn't ship those poorly thought out products citing lack of "leadership".

(again)

Karpathy is a great instructor, but has he has any meaningful impact in industry?

It seems from the outside like he locked Tesla into a losing, NN-only direction for autonomous driving.

(I don’t know what he did at openai.)

Tesla is only now launching their NN-only FSD model?..

What makes you say it’s a losing direction?

A popular take in autonomous driving is the thing preventing Tesla from breaking beyond level two autonomous driving is its aversion to lidar, which is a direct result of its nn preference.

(Eg Mercedes has achieved level 3 already).

Absense of lidar is just a symptom. Tesla only recently started to work with 3d model (which they get from cameras like one can get it from lidar) It just that the people who use lidar usually work with 3d model from the beginning.

> which they get from cameras like one can get it from lidar

LiDAR directly measures the distance to objects. What Tesla is doing is inferring it from two cameras.

There has been plenty of research to date [1] that LiDAR + Vision is significantly better than Vision Only especially under edge case conditions e.g. night, inclement weather when determining object bounding boxes.

[1] https://iopscience.iop.org/article/10.1088/1742-6596/2093/1/...

"What Tesla is doing is inferring it from two cameras."

People keep repeating this. I seriously don't know why. Stereo vision gives pretty crappy depth, ask anyone who has been playing around with disparity mapping.

Modern machine vision requires just one camera for depth. Especially if that one camera is moving. We humans have no trouble inferring depth with just one eye.

I’m confident that neural networks can process LiDAR data just as they can process camera data. I believe Musk drew a hard line on LiDAR for cost reasons: Tesla is absolutely miserly with the build.

Man Mercedes has a killer marketing team

Elon Musk didn't want LIDAR because:

- it costs too much

- it's ugly

- humans have only vision

TESLA Engineers wanted LIDAR badly, but they have been allowed to use it only on one model.

I think that autonomous driving in all conditions is mostly impossible. It will be widely available in very controlled and predictable conditions (highways, small and perfectly mapped cities etc).

And about Mercedes vs Tesla capabilities, it's mostly marketing... If you're interested I'll find an article that talked about that.

Is this a bad sign from OpenAI because he knows future products and company direction? Knowing his past from Tesla he seem to like to do world changing things..

I bet $1000 in iTunes gift cards (so I can pay off IRS debt collectors) that he will end up at Apple and lead one of the LLM/AI initiatives.

Sounds like he wants to focus on teaching.

Top job quitting performance

OMG, what do we do now!?

But seriously, right now with full attention to LLMs, and many brains, there is no single key person. The question 'who said it first' isn't that important for the progress. With experts leaving OpenAI will gradually loose it's leadership. Others will catch up. Which is good in general, no one should have monopoly on AI. I wish it was that easy with hardware too...

Where is he going ?

Nice. 1 year cliff bounce. LOL

does that mean she is pregnant? boy? girl?

Karpathy quits his job to become a full-time AI Youtuber. I don't blame him lol.

Reply on news.ycombinator.com