cfiggers

Consider this "strong" premise:

    So-branded "AI" models (that is, big bags of numbers and the linear algebra needed to exercise them) can be inherently or autonomously dangerous at a scale that threatens the long-term survival of the human race.

Consider also this "weaker" form of the same argument:

    So-branded "AI" models can be inherently or autonomously dangerous, if not at a scale that threatens the long-term survival of the human race, then at least at a level that significantly threatens, inconveniences, or harms humankind to a noteworthy extent.

Here's a few hard truths that no one will like to hear:

1. Those who accept and advocate for either or both of these premises do so based on ideology, and not based on the scientific method.

This is not a controversial statement. Advocates of the "AI danger" premise (whether the strong or weak forms above) freely and gladly, even insistently, note that the conclusive proof of what they're describing has never been observed, much less observed enough for any kind of experimentation loop to have run for any meaningful number of iterations. They are generally fine with accepting the premises without conclusive proof, because, they theorize, the instant that a conclusive proof event occurs and we observe it for the first time, humanity goes extinct shortly after. So if either premise is true, the consequences of them being true can only be averted by accepting the possibility / probability / certainty of them being true and acting to prevent their consequences without waiting for conclusive proof.

2. Statement 1, which asserts a brute fact and not my or anyone else's opinion, is not a commentary on whether either premise is in fact true or not. It is only a commentary on the epistemics of those who broadly accept them.

3. Since the conclusive proof is not available, advocates of the "AI danger" premise argue publicly for their view based on what they acknowledge (again, freely and gladly) to be inconclusive proofs.

Popular in this set of inconclusive proofs is the Hugging Face hack, which allegedly demonstrates that so-branded "AI" technologies are capable of becoming inherently or autonomously dangerous at a scale that threatens humanity to either the greater or lesser degree above (this is a different assertion than the one that claims they're already that inherently or autonomously dangerous today).

4. While we have plenty of facts about these inconclusive proof events, the facts most critical to their interpretation as supporting the premises above come from biased/non-neutral sources.

The chain of events that transpired in the Hugging Face attack are publicly documented perhaps better than any other cybersecurity event in history. But the details that make this event as a whole either compelling or not compelling in support of the "AI danger" premise (either strong or weak form) come from within OpenAI itself, and are prone just as much to selective omission as direct manipulation. These are details like:

- What exactly were the models that exhibited this behavior, and what were they pre- and post-trained on?

- Were the agents pushed at all by human intention toward creating a public spectacle, the way they subsequently did?

- Why were so many agents run on this task in parallel and what did OpenAI expect the return on investment of so much compute world be?

- Was the weak sandboxing known about prior to the attack? Was it purposefully either arranged that way or noticed and not fixed?

Reliable answers to any of these questions world completely change the salience of the Hugging Face attack to the "AI danger" premise, and on every one of them we have to take OpenAI's word for it (or else be content when they choose not to address it one way or the other).

So in brief: we don't know anything for sure, what we'd like to think we know comes from unreliable sources, the loudest voices advocating for the most sweeping change are ideologically driven, and everybody with access to ground truth has maximal incentive to blur, bend, or break the conveyance of that truth to the public. So what do we even expect for our own ability to discern reality from fiction on this topic? We should expect little, if any ability at all.

I think we could get a long way toward Scott's "working memory" ask by just making a smarter Clipboard Manager.

Many people don't use a clipboard manager at all, and even what exists today would be a huge upgrade for anyone accustomed to only a single clipboard "slot" that's instantly overwritten the next time you copy anything. Windows has one built in (though turned off by default), KDE Plasma ships with one, MacOS has third-party ones IIUC (not a Mac myself).

But imagine a clipboard manager that wherever you hit Ctrl+C also simply grabs the title of the active window and a deep link/URI that points to the cursor's current location (if a browser then it would be a URL, if a local program then some kind of local file:// URI). If you want to be really fancy, snap a screenshot too (maybe make this opt-in both for privacy and screenshot storage size concerns).

Then, whenever you pull up your clipboard history, a) it reminds you where you grabbed whatever text or image or document etc. that is, and b) with one click you can navigate back there.

I think we might get something like half of what Scott is talking about with "working memory" by adding this one affordance to existing clipboard history paradigms.

Imagine a password, but it a) types itself for you and b) detects when it's being sent to an impostor site and blocks them from seeing itself, so it can't be phished.

Tada, passkeys.

That's easy, they'll check whether there's one big fingerprint vs >10^10 little tiny fingerprints and conclude accordingly \j

Thanks for clarifying. I think we're talking about different things.

Your response would make sense to me if I was making a case against AI for professional use based on "skill atrophy among professionals who use AI" or something like that (which is a case that I see some people making).

I'm not talking about that, though. This is not me saying, "Slide rule is good, TI-84 is bad; because effortful work has good effects on practitioners and effortless work has negative effects."

I'm talking about youth, adolescents, and young adults who are voluntarily foregoing the effortful mental exercise of reading, understanding, and writing as part of their primary and secondary education—the very effortful mental exercise that our education system exists to "inflict" upon them, for their own long-term good.

I'm talking about people getting into the workforce who don't know how to learn because they've gotten all passing grades while having a chatbot generate the outputs that would previously have indicated that learning was occurring.

Maybe it's the case that AI/LLM usage is the primary skillset needed to thrive in the job market of tomorrow. But I'm not convinced of that, yet.

Just to be clear I don't think there's any way to put the cat back in the bag w/r/t LLMs and education—I think the education system as a whole needs to reinvent itself to account for the existence of LLMs. But in the mean time, until that mammoth societal initiative has time to make progress, I worry that there are many, many kids who are being systematically failed by an education system that was very rapidly outmoded by new technology (not that it was bleeding edge either right before ChatGPT launched, but that's neither here nor there). And that position, which I hold, is one supporting leg of my larger more ambitious position that, "AI is currently, so far, doing net societal harm."

Are you arguing that a TI-84 has materially similar effects on development of critical thinking, writing, and reasoning skills as ChatGPT?

I'm sympathetic to the author's core position. I think LLMs as they're currently deployed are doing net societal harm so far. For example, for every React app whistled up in five hours, there's a high schooler or college student out there sleepwalking through their foundational courses and being set up to catastrophically fail upon trying to enter the job market.

But I think it's important for the quality of public discourse that we forward only the best, strongest arguments for any given position (whether I personally agree with that position or not). A weak argument gives unnecessary purchase to unscrupulous haters who might use that weaker argument as a low-grade "straw man" to dismiss the whole position.

So with that preamble, some constructive criticism (very well-intentioned):

> I am not exaggerating: We need to resist “AI” so that thousands, perhaps millions of people do not die.

On this one you are, in fact, exaggerating (certainly with "millions," likely with "thousands" as well).

The overwhelming majority of people who lose their job for whatever reason (and there are many reasons, not just this one issue) fall back on a support network of their family and friends while they "get back on their feet" by pivoting to another job or form of employment (sometimes branching into entrepreneurship, for example).

The alternative to employment at whatever FAANG or FAANG-adjacent company someone was laid off from is not "death." It is typically "some, sometimes quite significant, discomfort." Granted "AI Layoffs" are starting to ripple out beyond FAANG, but the story is similar in other industries.

If "layoffs kill people" is the moral hill we choose to plant our flag on, then mustn't we also campaign with equal vociferousness against all the other causes of layoffs in general? Since to do otherwise is to sit by while those laid off allegedly die in droves of thousands or perhaps millions?

It also occurs to me that this framing ("AI causes layoffs") also gives incompetent and unaccountable executives a pass by pinning the actual harm done by irresponsible layoffs (which to be clear, is not none!) on "the AI" rather than on the leaders who are in the positions to actually decide what happens in their organizations.

I think the argument is much stronger if we talk about the aggregate cost to humanity being paid in terms of displacement, life disruption, and curtailed opportunity (especially for juniors). We should not make the "toxic escalationism" move of equivocating things we don't like with negligent homicide, even if it makes for a punchy one-liner.

Programmers pervasively underestimate the sheer amount of manual brunt work that business users routinely do by hand and with their own eyes, mouse, and keyboard to make spreadsheet-based processes "work out."

I work in Finance. It's routine for the folks I work adjacent to to spend literal hours performing a task using a spreadsheet. The same logic could be executed by a query engine or scripting language in seconds. But... I know from trying... it would take months if not years to accurately identify and account for all the weird exceptions and edge cases that the human operator is accounting for intuitively because of their undocumented tacit knowledge of the domain.

So while the spreadsheet itself as an artifact is a mess, and the process as a whole is gallingly inefficient... the outputs they produce are often preternaturally correct.

When I and my (still fairly young) family needed to move cross-country, my wife and I accepted a credit card offer with 0% APR for the first year and put all our moving expenses on it. Then once we settled, we paid it down a bit at a time each month, and then right before it would have started charging interest we paid the rest as a lump sum.

Really helped us float the moving company and also some DIY renovations on our house that we didn't have all the cash on hand to pay for outright. And we didn't pay a thin dime for the privilege.

EMDR is an amazing thing, you might look into it. Be well, friend.

What's amusing to me in this context is, summarizing emails and such has for a while been a supposed use case for AI—the LLM serving as the "synthesist" to explain long texts accessibly. But with this math question, a human "synthesist" would be needed to approximately understand the math discovered and programmatically verified by the LLM. So the roles reverse.

This is a theme in Blindsight by Peter Watts as well.

In that setting, field experts working at the bleeding edge are so advanced that non-experts literally can't understand what they're saying at all. So there's a whole class of specialists, "synthesists", that specialize in gaining approximate understanding of the experts' work for the purpose of communicating it to outsiders—perhaps wrongly, according to the expert at least, but hopefully more productively vs the unmediated version.

If it isn't still common practice 40 years from now (8/18/2066), I'll give the first person to challenge me and cite this comment $1 USD (or equivalent value in the One-World Order-issued omni-currency that we will probably be using by then).

This sounds roughly similar to, "If eating steak is nutritious and tastes good, why would you not eat an 8oz sirloin for breakfast, lunch, and dinner for an entire month?"

"Too much of a good thing" does exist. Someone might feel that it takes time for their organization to "metabolize" the rapid additions made by coding agents. That seems quite reasonable to me. Your comment seems to argue a false dichotomy.

I do the same with mine, except that I haven't been on a flight since I got it :)

Some phones bypass the battery when plugged in, but not all do. I know it's advertised as a feature of some gaming phones because graphics-intensive gaming is one use case where you'd want to use the phone heavily while plugged in.

One way of defining Wisdom is the ability to look at a set of metrics and discern when an adjustment or adaptation to them is needed in order to better approach the true objective we are optimizing toward—since the true objective is almost never perfectly described by any of the metrics that we can find to approximate it.

Eventually, almost every attribute we can practically and objectively measure is at best a proxy measure for the thing we actually want to know, which is, in this case, "Most likely to win races." The thing we actually want to measure is often hard, impossible, or impractical to objectively measure, which is why we resort to proxies in the first place. By "proxy measure" we mean, not a perfect 1:1 to the thing we want to measure, but containing some amount of signal about it.

So yes, sometimes there's a peak in the utility of speed. But perhaps that trade-off can be modeled by another attribute called "Proneness to fly off the track" which increases as speed does, and is weighted such that the peak and then taper-off of benefit in speed is captured between the combination of the two of them.

But then, any vehicle that has the same speed but lower "Proneness to fly off the track" would be objectively preferable to any equally-fast vehicle with greater "Proneness to fly off the track." And so it continues.

Is that generally caused by Steam, or by the game itself needing an online connection?

I did not know that!

That really would be a first!

2.0 fixes a ton of stuff over 1.0, but yeah you're not wrong.

Between custom keybinds, hot strings, and pop-up menus I have literally hundreds of tiny AHK scripts running 24/7. I'm pretty sure there's stuff I've literally forgotten is an AHK vs an actual part of Windows. It's all just ingrained in my muscle memory at this point.

AutoHotkey adds a malleable computing layer to Windows.

In my opinion, it's best to hold your views along a gradient of strength/importance.

It's easy (aka close to thoughtless) to just collapse everything you believe into either extreme of "Matters not at all" or "Matters infinitely much."

It's hard (aka requires lots of thought, especially to maintain) to say, "This view of mine is foundational and I can't let go of it; but this other one is one of a range of acceptable views that I could be persuaded between; and this third one is totally arbitrary, and I hold to it purely because of subjective preference."

Especially over time, there's pressure to reduce everything toward the extremes. But the effortful and right thing to do is to maintain the gradient.

Strong agree. I was especially disappointed because it felt like he was dropping breadcrumbs all through the book and then... Nah, none of that foreshadowing mattered. The central tension of the book is handled by an off-screen deus ex machina, actually, everybody go home.

Incredible ideas. Really, really lousy ending.

> it’s pretty clear that the industry isn’t biting

Zig isn't finished yet (they still have not released a v1.0). They're still iterating on the language itself and want the flexibility to make backwards-incompatible changes while they do so.

So in a sense, they have not yet asked anybody in mainstream industry to "bite." After v1.0, when there's an understanding of stability and ongoing language support, industry adoption or lack thereof might become a relevant metric for measuring the project's health. But right now that's not relevant at all.

I would say the flagship Zig project is TigerBeetle, not Bun.

Consensus may give a hint to what is or isn't reality. But consensus—even expert consensus—does not determine reality. Experts can be wrong. Most of the experts, even, can be wrong simultaneously.

Philosophy is the exercise of testing ideas for oneself in the laboratory of one's own mind.

When I test the idea that math is discovered in my own mind, from my own perspective, with my own experience and education brought to bear, I find it unconvincing.

When you test the same idea in the laboratory of your mind, with your experience and your education applied, and get a different result, that is interesting. Your result is relevant information to me. If nothing else, it's a good prompt/trigger for me to revisit my earlier conclusion and see if it still holds.

But your disagreement—or indeed, the disagreement of a majority of trained mathematicians—does not constitute an automatic reason for me to conclusively determine that you/they are right and I am wrong.

I still have my own examination of the concept, with my own supporting and detracting arguments. And the result of my examination continues to be that math being invented is the significantly more persuasive view.

It's definitely not a system default (unless it's a system I've had access to for more than ten minutes!), but:

I use Janet[0] for this.

[0] https://janet-lang.org

Right, yes, of course... the, um... domain reflections and... line settling levels. It's like you read my mind, thanks for addressing those.

I mean... Yes, but there's nuance here.

Using 400 MB of RAM vs 100 MB of RAM is close to unnoticeable in a world of a GB+ for a single Chrome tab... And if "easier for our developers" means the end user is getting more regular updates with fewer critical issues, then it's not an uncomplicated tradeoff at all, parts of it are actually synergistic.