edouard-harris

This seems like a remarkable chain of reasoning. Out of curiosity, what would convince you that there is a path from LLMs (say) to AGI? What would you expect progress along that path to look like?

Surely that can't be right? The US + Canada together have a population of under 380M. Even your low-end ARPU number of $1200 per year would imply North American revenue above $450B, far higher than Google's entire 2023 annual revenue of $300B [0].

(Even correcting for market share isn't enough, as Google Search has over 90% penetration in the US [1].)

[0] https://www.statista.com/statistics/266206/googles-annual-gl...

[1] https://www.similarweb.com/engines/united-states/

There's a hidden advantage to writing like this. It comes up when, e.g., you're communicating on a controversial topic in a forum like Twitter with a history of forming mobs against folks who communicate on controversial topics.

Suppose an angry reader is looking for a reason to form a mob against you. If you write in simple sentences, that makes it easy for the angry reader to process your statements and go after them. But if your sentences are more complicated, the angry reader needs to decode them logically before they can justify their anger. Angry people tend to be poor at logic. So when they run into this kind of writing, they often get bored before they have a chance to get outraged. You get your message across, and the angry reader moves on to the next tweet in their feed. Everybody wins.

This doesn't work 100% of the time. But if you do it right, it cuts down on a lot of negative virality. I'm not saying this is or isn't Patrick's intentional strategy. I have no idea. It's just one among many consequences of this communication style.

> The entire purpose of starting a company is to get acquired.

Certainly not in terms of expectation value. Most of the value of a startup at any given funding round is driven by the possibility of a public listing. This is true even though most successful startup outcomes are acquisitions.

> Of the literally hundreds of companies that YC has invested in, only 5 have gone public.

The correct number is 18, not 5. [1]

[1] https://www.ycombinator.com/topcompanies/valuation (select the "Public" tab.)

This is the correct take. As the cost to do a bad thing decreases, the amount of political will society needs to exert to do that bad thing decreases as well.

In fact, if that cost gets low enough, eventually society needs to start exerting political will just to avoid doing the bad thing. And this does look to be where we're headed with at least some of the knock-on effects of AI. (Though many of the knock-on effects of AI will be wildly positive.)

I've heard the licenses Meta releases its AI models under described as "open access", to differentiate them from the stricter "open source" definition.

If Steve Jobs had sold paint.

While the trend they're highlighting is absolutely real, it looks like the math in their first example isn't quite right:

> For example, Carrefour said a bottle of sugar-free peach-flavored Lipton Ice Tea, produced by PepsiCo, shrank to 1.25 liters (0.33 gallons) from 1.5 liters (0.3 gallons), resulting in a 40% effective increase in the price per liter.

Assuming the price per unit stayed the same (which is the implication), this is a 20% effective increase, not 40%.

Agree with you on the cultural events, but the NYC subway is definitely dirty compared to the systems of the two other cities I've lived in (Toronto and London).

I was mostly in LIC and Manhattan though. Maybe it's not so dirty in the other boroughs?

In practice, the real number is probably closer to 36% than to 65% (and likely even lower).

Here's the problem: the article doesn't account for the capacity factors of the various new installed sources they mention [1]. A capacity factor is the actual electrical energy produced by a generating unit, divided by the maximum possible electrical energy that generating unit could have produced if it were operating continuously at full power. And under real-world conditions, most renewable energy sources produce at far below their theoretical capacity maximum.

For example: wind turbines have a capacity factor around 36%, and solar installations are under 25% [2]. If we apply these factors to correct the numbers in the article, the picture becomes much bleaker: the headline 1475 MW of new wind capacity drops to just 531 MW, while the headline 2530 MW of installed solar drops to 632 MW. Now let's generously assume that the 100 MW of hydro and 29 MW of biomass from the article both operate at a 100% capacity factor. Under those assumptions, total new installed renewable capacity drops from 4134 MW to 1292 MW — more than threefold. [3]

And of course, the real-world shortfall is even worse than this. Electricity markets need to clear continuously, and the capacity factor for solar when the sun isn't shining is not 25%, but 0%. That means if the hours of peak demand coincide with hours during which solar is offline (which they do during much of the year) there is no amount of solar you can install that will keep the grid online. To sum up: there is no way to escape the need for reliable baseload power. [4]

[1] At least, the article doesn't mention correcting for capacity factors. And the FERC source document it cites gives me a blank page when I click on it in Firefox, meaning there is no way to be sure whether anyone applied this correction. I'm assuming they didn't, because citing high "headline" capacity numbers like this is unfortunately very common in discussions about renewable energy sources.

[2] Capacity factor numbers are for installations in the United States during 2022, from the Energy Information Administration: https://www.eia.gov/electricity/monthly/epm_table_grapher.ph...

[3] From the article: 2530 MW of solar, 1475 MW of wind, 100 MW of new hydro, 29 MW of new biomass, 2259 MW of natural gas.

[4] Even battery installations don't get you there. Between 2022 and 2026, the US is expected to add only 24 minutes worth of battery storage to its grid. This includes residential, non-residential, and grid-scale installations. Sources: https://pv-magazine-usa.com/2022/09/14/u-s-installed-a-recor... https://www.eia.gov/energyexplained/electricity/electricity-...

Hey — you're quite correct and I appreciate this feedback. Thank you.

I just tested GPT-4 on the Winograd schemas below, which I made up and (as far as I know) are novel.

It scored 3/4, incorrectly answering "the pouch" instead of "the baby kangaroo" for Question 2 of Schema 1. And now that I look at the question, one could easily imagine the kangaroo being afraid to jump into a pouch that's too large! In hindsight, if a human complained to me that this question was ambiguous, I'd have a hard time disputing them.

Granted these are just two examples, but they aren't cherry-picked. Moral of the story: before one makes a strong claim such as "there isn't any evidence a model can do X", it's often worth making a minimal effort to investigate one's own claim.

========================================

Winograd Schema 1:

Sentence 1: The baby kangaroo was afraid to jump into the pouch because it was too small. Question 1: What was too small? (Correct answer: the pouch)

Sentence 2: The baby kangaroo was afraid to jump into the pouch because it was too large. Question 2: What was too large? (Correct answer: the baby kangaroo)

Winograd Schema 2:

Sentence 1: The gardener couldn't grow the flowers in the greenhouse because it was too hot. Question 1: What was too hot? (Correct answer: the greenhouse)

Sentence 2: The gardener couldn't grow the flowers in the greenhouse because they were too delicate. Question 2: What were too delicate? (Correct answer: the flowers)

https://www.aitracker.org/ is good for a general audience, but doesn't go into as much details as some of the more research-oriented roundups.

> I've more often heard of the opposite problem.

Yeah, my guess is this is the more common case for a randomly selected startup. But I suspect Casey doesn't advise a randomly selected sample of startups.

I mean, even if that is exactly what "x-risk research" turns out to be, surely even that's preferable to a catastrophic alternative, no? And by extension, isn't it also preferable to, say, a mere 10% chance of a catastrophic alternative?

> Meanwhile while their first engineering hire, the person who probably wrote half the damn code for years, gets < 1% of the equity and none of the glory.

For whatever it's worth, market comp for Engineer #1 equity is significantly above 1%. Depends on a lot of things (number of founders, how many founders are already technical, whether hire is pre or post PMF, prior experience, etc.) but by and large <1% is not going to be competitive at this level.

As one more data point: I've angel-invested in a couple of Nigerian startups, and I can confirm this tracks with the black-market rates my portcos are seeing over there. The divergence (between official and actual forex rates for NGN/USD) first took off in September '22 and has stuck around since.

> Remember Alamada wasn’t a normal customer. It was acting as a market maker/counterparty of last resort to many other FTX customers. In that case there could be situations where ‘normal customer liquidation rules’ shouldn’t apply

This might be a reasonable argument in a vacuum, except that:

> [SBF] told investors and prospective investors that FTX had top-notch, sophisticated automated risk measures in place to protect customer assets, that those assets were safe and secure, and that Alameda was just another platform customer with no special privileges. [emphasis added]

And also:

> Bankman-Fried also told investors, and directed other FTX and Alameda employees to tell investors, that Alameda received no preferential treatment from FTX. For example, Bankman-Fried told the Wall Street Journal in or around July 2022: “There are no parties that have privileged access.” Likewise, in a Bloomberg article published in or about September 2022, Bankman-Fried claimed that “Alameda is a wholly separate entity” than FTX. In the same article, Ellison is quoted as stating about Alameda: “We’re at arm’s length and don’t get any different treatment from other market makers.” Bankman-Fried made similar statements directly to investors. [emphasis added]

(The above are direct quotes from the SEC complaint [1] against SBF.)

Singh might argue that he wasn't aware of the private statements to investors. But the WSJ and Bloomberg stories show the "arm's length" claim was something they consistently messaged to the public at the highest levels. To argue that Alameda's role as market maker of last resort justified a privileged status would be inconsistent with all their prior public claims to the contrary. "We lied to the public repeatedly about our risk management" isn't a defense; it's a confession.

[1] https://www.sec.gov/litigation/complaints/2022/comp-pr2022-2...

While it's true he's saying that founders can create a lot of value without employees, he's certainly not suggesting employees don't also create value. Obviously they do, or founders wouldn't hire anyone.

> if you have employees, you have to share your profits with your employees

Needless to say, this already happens via equity and options. There's a reasonable debate to be had over what constitutes "fair" profit-sharing. But the market clears where it clears, and individual tech employees definitely have meaningful leverage even under current macro conditions.

It's worth noting that at least one startup (Adept AI) is literally building exactly this as a full end-to-end system.

See: https://www.adept.ai/act

> For what it's worth, Searle is a naturalist (so he thinks brains are "just" meaty machines) and he would also disagree with you.

I definitely disagree that Searle's conclusion in the Chinese room experiment applies to the kinds of AI systems we're building today.

I agree that nobody to my knowledge has figured out how to measure sentience, so I don't think that fits under the "measurable" heading. And this does suggest to me — because this was your hand-picked example — that your position as a substance dualist doesn't affect your concrete expectations about what AI systems will actually be able to do in the future. Is that right?

> I'm a substance dualist so I'm admittedly a bit biased

Thanks, I appreciate you pointing this out. Given that's the case, it might not actually be possible to reconcile our two views and we can just agree to disagree. (I was a physicist for ten years, so it's challenging for me to inhabit a position that contends that thought and matter are non-overlapping magisteria.)

I guess one question I might ask you is: how do you think substance dualism will affect your expectations about AI going forward? For example, is there any specific, measurable thing that you expect a future AI system absolutely will not be able to do? Or do any such "forbidden" things have to fall into categories like "thought", "feeling", etc. — i.e., the magisterium of the "mental", as opposed to the "physical" — under substance dualism?

Can you give a concrete example of an output ChatGPT could give — or an experiment one could run on it — that would persuade you that it's intelligent? (For whatever definition of "intelligent" you prefer.)

If you can't think of such a test, then it seems strange indeed to confidently claim that ChatGPT has failed to satisfy an untestable condition.

But if you can think of such a test, and any human fails it — does that mean the human who fails it isn't intelligent?

Yeah, in any given instance, "nothing happens" is true more likely than not. On the other hand, this is equally true in a game of Russian roulette. In both cases, "nothing ever happens" is not necessarily the right lesson to draw from the observation.

The nature of tail risk is that it seem completely impossible until, one fine day, it manifests explosively. Various other corners of crypto have recently been exposed to this lesson, often at some expense. It doesn't seem unreasonable to expect more damage to the ecosystem before the lesson is well and truly learned.

That seems a bit unfair. Every successful founder is irrationally optimistic about their own business — that's partly how they became successful in the first place. It doesn't seem at all unlikely that Zuck, Lütke, the Collisons, and many, many others all made the same wrong directional bet and ended up over their skis for perfectly sincere reasons.

In case one has trouble recalling, way back in the dark ages of 2.5 years ago, when these investments were first being made, neither the duration nor the outcome of the pandemic were at all clear to anyone.

Indeed. And in this case, it was from one day to the next.

Yep. This is the same strategy Instagram used a few years ago in copying Snapchat's Stories format, and likely has the same goal. The idea is not that Instagram can take back TikTok's audience — those users are well and truly lost — but rather that it can deny existing Insta users any reason to even try TikTok in the first place. (Because they can get the same form factor without downloading a new app.)

From what I've heard, the Stories copying strategy did successfully contain (but did not roll back) the Snapchat threat to Insta. So there's some reason to believe they can hold the line against TikTok with this approach too, even absent the possibility of a government ban.

I expect this sort of convergence will continue in the future. Whatever one might think of fast-following as a strategy, scale plus rapid copying of features does seem to work.

> It's basically been disproven at this point that simply lowering cost does not necessarily lead to better outcomes for the country.

After untangling the double-negative, this is equivalent to saying: "It's basically been proven at this point that simply lowering cost necessarily leads to better outcomes for the country."

Is this in fact what you meant? The rest of your comment suggests you may believe the opposite.

This might make sense as a response solely to the title of the article, but I have to admit I find it puzzling as a reaction to its content. Notwithstanding the title, the article mentions a model called Minerva that scored fully 50% on the MATH dataset of high-school/undergrad mathematical problems. For comparison, a human computer science PhD student scored 40%. [1]

For context, Minerva came out this July. When it was tested on a national math exam, it scored higher than that year's class of graduating high school seniors. [2] A mere eight months (!) earlier, OpenAI had announced [3] they'd trained a language model that solved math word problems almost as well as an average middle-schooler. So even if you believe — rightly or wrongly — that current capabilities aren't very impressive, it's worth remembering that your understanding of current capabilities might not be entirely accurate, even if it's only a few months out of date.

Incidentally, it may be worth looking at some examples of these models' outputs before deciding what they can or can't do. Here's Minerva solving some math problems, for example:

https://dl.airtable.com/.attachmentThumbnails/32f10fc59039c7...

https://dl.airtable.com/.attachmentThumbnails/f6f86e0edd14d1...

https://dl.airtable.com/.attachmentThumbnails/d9bc743dca1015...

https://dl.airtable.com/.attachmentThumbnails/51562ef70c2b14...

https://dl.airtable.com/.attachmentThumbnails/c081906cc4d23b...

I'll admit I find it challenging to interpret these results as "passable garbled nonesense [sic]", though perhaps I'm not being demanding enough. At any rate, when these models go from beating 10-year olds at math to beating 18-year olds at math in the span of 8 months, one does start to wonder how much of the hype is really due to over-interpretation — and what the next 8 months have in store.

========================

[1] https://arxiv.org/pdf/1709.04326.pdf (For comparison, a 3-time Mathematics Olympiad gold-medalist scored 90%.)

[2] This was Poland's National Math Exam. Minvera scored 65%; the 2021 national average was 57%. See https://www.aitracker.org/ and Section 3 of the Minerva paper at https://arxiv.org/pdf/2206.14858.pdf

[3] https://openai.com/blog/grade-school-math/

> Fun fact: the newly anointed YC president Garry Tan automatically blocks anyone on Twitter who follows anyone who advocates for housing reform or even likes any tweets from such [1].

While the article you cited talks about Garry blocking people on Twitter, it doesn't mention anything about housing reform.