AIPedant

The problem is that Trump, Musk, Lindell, etc are all extremely arrogant and constantly disregard sound legal advice. Their lawyers aren't merely associated with a controversial client; their professional reputation is put at risk because they might lose easily winnable cases due to a client's dumb tweet. You have to be a crappy lawyer (or an unethical enforcer like Alex Spiro and Roy Cohn) to even want to work with them.

It doesn't matter if court was adjourned, she was still at work and performing her official duties. In particular she asked the ICE agents if they had a judicial warrant and was told no, it was administrative. A federal judicial warrant clearly outranks a local judge and it would be fine to arrest her if she defied it. It is not clear that an order from the executive branch does the same. That doesn't mean local judges are immune from federal prosecution (e.g. corruption charges if someone takes money to rule favorably), but there is a fairly high bar, I don't think her behavior even comes close to probable cause for obstruction of justice or shielding an undocumented immigrant.

The reason every other administration besides Trump refused to go into local courthouses to deport people wasn't about "whether people should be safe from immigration enforcement," it was about separation of powers.

The part that makes it not so clear cut is that this is really a constitutional issue rather than a criminal one. It is not credible that the judge was trying to ensure the man could keep living in the US undocumented. She was defending her court. From a judge's POV, arresting a plantiff/defendant in the middle of a trial is a violation of their right to a trial and impedes local prosecutors' abilities to seek justice.

Ultimately this seems like Trump asserting that the federal executive branch has unfettered veto authority over local judicial branches. That doesn't sit well with me.

The Cursor founders (technically the company is called Anysphere, Inc) are all young MIT grads. What they needed is a 40-year-old with a degree from Fitchburg State who could say "Woah, don't do that! It's not worth the long-term risk!"

I was outraged to learn that the test is basically a combination of multiple-choice and true-false. What a misleading pile of crap this study is!

a) Doing well on multiple-choice tests does not really imply anything about doing well in a real lab setting, especially for an AI.

b) Using multiple-choice tests to compare LLMs to humans has an obvious flaw: LLMs are probably superhuman at guessing multiple-choice answers based on superficial statistics! We all learned as children that there are ways to game multiple-choice questions even if you have no idea what the answer is.

c) It is just unacceptably lazy for AI researchers to do this. They aren't teachers on a tight deadline. There is no justification whatsoever for AI researchers to use Scantrons.

I am truly disgusted with the poor scientific rigor in AI research. So depressing.

This is a frustrating comment. I said "I was surprised to learn" because I had the same impression you did, but then I learned something new. It seems like you're just rejecting my conclusion out of hand without bothering to learn anything.

Eric Berger's book in particular suggests that, before Falcon 1 was successful, Musk was much more humble and collaborative with the other early SpaceX hires, and typically deferred to their expertise. He was always reckless and megalomaniacal. But after Falcon 1 he became much worse.

I hated Elon Musk long before it was cool: I was a fan of Tesla in the early days, and when I read Musk's "super-secret master plan" for Tesla I thought "yeesh, the board chairman is an idiot, where did they find this bozo?" (I knew a bit about SpaceX but somehow didn't make the connection.)

That said, I was surprised to learn much later that, by all accounts, Elon Musk was a competent and resourceful leader in SpaceX's early days. Maybe these stories are just his personality cult in action, but I found it plausible. It appears he once knew his place as an engineering manager, without LARPing as a Chief Engineer (he didn't appoint himself to CTO until quite a bit later). I worked for a really good manager who didn't know how to code, but he knew a lot about software and was very good about pulling back on coding things vs pushing forward on software design. It seemed like Musk was similar at SpaceX.

Which is all to say that celebrity is a helluva drug. I don't think Musk was ever an especially "high-IQ individual," and his first marriage suggests he's always been a misogynistic loser. But being anointed "a real life Tony Stark!" seems to have destroyed his brain. Ketamine probably doesn't help.

I nominally agree with this point - AGI is theoretically possible according to the Church-Turing thesis, we can “just” solve the Schrödinger for every atom in the human body.

The more salient point is that when a model reads “dog” it associates a bunch of text and images vaguely related to dogs. But when a human reads “dog” they associate their experiences with dogs, or other animals if they haven’t ever met a dog. In particular, cats who have met dogs also have some concept of “dog,” without using language at all. Humans share this intuitive form of understanding, and use it with text/speech/images to extend our understanding to things we haven’t encountered personally. But multimodal LLMs have no access to this form of intelligence, shared by all mammals, and in general they have no common sense. They can fake some common sense with huge amounts of text, but it is not reliable: the space of feline-level common sense deductions is not technically infinite, but it is incomprehensibly vast compared to the corpus of all human text and photographs.

> One of the biggest pons asinorums that people get hung up on is the idea that "it just imitates the data, therefore, it can never be better than the average datapoint (or at least, best datapoint); how could it possibly be better?"

Hmm - the phrasing that perhaps holds more water is that LLMs just imitate the data, which means that novel ideas / code tends to be smashed against the force of averaging when fed into an LLM. E.g. NotebookLM summaries/podcasts are good infotainment but they tend to flatten unconventional paragraphs into platitudes or common wisdom. Obviously this is very subjective and hard to benchmark.

In a squishy philosophical sense, I think LLMs are doomed to hallucination/confabulation because we’re making systems that use language and hoping they figure out the basics of communication. That certainly is not how it works in humans:

> Did I not, then, as I grew out of infancy, come next to boyhood, or rather did it not come to me and succeed my infancy? My infancy did not go away (for where would it go?). It was simply no longer present; and I was no longer an infant who could not speak, but now a chattering boy. I remember this, and I have since observed how I learned to speak. My elders did not teach me words by rote, as they taught me my letters afterward. But I myself, when I was unable to communicate all I wished to say to whomever I wished by means of whimperings and grunts and various gestures of my limbs (which I used to reinforce my demands), I myself repeated the sounds already stored in my memory by the mind which thou, O my God, hadst given me. When they called some thing by name and pointed it out while they spoke, I saw it and realized that the thing they wished to indicate was called by the name they then uttered. And what they meant was made plain by the gestures of their bodies, by a kind of natural language, common to all nations, which expresses itself through changes of countenance, glances of the eye, gestures and intonations which indicate a disposition and attitude--either to seek or to possess, to reject or to avoid. So it was that by frequently hearing words, in different phrases, I gradually identified the objects which the words stood for and, having formed my mouth to repeat these signs, I was thereby able to express my will. Thus I exchanged with those about me the verbal signs by which we express our wishes and advanced deeper into the stormy fellowship of human life, depending all the while upon the authority of my parents and the behest of my elders.

(From the Confessions of St. Augustine)

In an abstract philosophical sense, who knows? But any real organism in the real universe has a finite limit on how long they can be quiescent, so it is fine to say that they're alive until they can no longer be revived, even if that takes hundreds of thousands of years. Death implies irreversible damage.

The "over 100,000 deaths" seems like a slight exaggeration, this paper bounds the estimate at 40,000-80,000: https://www.nature.com/articles/s41467-022-33627-9 Wikipedia's "Epidemiology of snakebites" had similar estimates.

That's still a lot of people! But I was skeptical of 100,000, especially without a source.

Agreed - to be clear I was saying it confabulated analyzing the visual details of the photo to back up its actual reasoning of reading the EXIF. I am not sure that “low‑slung pre‑Alpine ridge line, and the latitudinal light angle that matches mid‑February at ~47 ° N” is actually evident in the photo (the second point seems especially questionable), but that’s not what it used to determine the answer. Instead it determined the answer and autocompleted an explanation of its reasoning that fit the answer.

That’s why I mentioned the case where it made up things that weren’t in the photo - “drives on the left” is a valuable GeoGuesser clue, so if GPT looks at the EXIF and determines the photo is in London, then it is highly probable that a GeoGuesser player would mention this while playing the game given the answer is London, so GPT is probable to make that “observation” itself, even if it’s spurious for the specific photo.

I just noticed that its explanation has a funny slip-up: I assume there is nothing in the actual photo that indicates the picture was taken in mid-February, but the model used the date from the EXIF in its explanation. Oops :)

One vague possibility: hallucinations are mitigated by looking at uncertainty in token prediction, the idea being that it’s a proxy for factual uncertainty, and the system is more likely to confabulate a court case or whatever if it only has a fuzzy idea of what comes next. But this won’t work for reasoning models which solve math problems: the next token for “x=“ is highly uncertain until you’ve done the computation!

I’ve also seen a few of those where it gets the answer right but uses reasoning based on confabulated details that weren’t actually in the photo (e.g. saying that a clue is that traffic drives on the left, but there is no traffic in the photo). It seems to me that it just generated a bunch of hopefully relevant tokens as a way to autocomplete the “This photo was taken in Bern” tokens.

I think the more innocuous explanation for both of these is what Anthropic discussed last week or so about LLMs not properly explaining themselves: reasoning models create text that looks like reasoning, which helps solve problems, but isn’t always a faithful description of how the model actually got to the answer.

You would think that Cursor's leadership would be aware of other cases where LLM customer support went awry - e.g. that Canadian airline whose chatbot promised a bereavement discount, ending with a judge ordering them to honor the chatbot's BS.

I suspect Cursor told themselves that they are super-smart AI experts who would never make an amateur mistake like the airline, they will use prompt engineering + RAG. With this, it will be unpossible that the LLM could make a mistake.

My eternally grumpy point is that if we are on the cusp of human-level AGI, then surely we can make frog-level and fish-level AI right now. But ANN-powered robots (or AI agents in physics-based video games) seem extremely slow and stupid compared to a frog or a fish. And instead of asking difficult scientific questions about what intelligence really means, AI researchers keep shoving human facts and heuristics in LLMs to give an illusion of human-level intelligence. This is precisely the same mistake yesteryear's AI researchers made when developing Lisp expert systems: ignore any tricky science, focus on trivia, tricks, and (most importantly) vibes.[1]

I am not even convinced transformer ANNs are as smart as spiders, and I don't mean the cleverer jumping spiders. Here is my thought experiment: lets say you trained a transformer-powered robo-spider on one gazillion examples of spiderwebs in nature - between rocks, bushes, etc - and verified that in such natural environments you had "superspider" performance (whatever that means). Now test the robo-spider on a pantry, attic, garage, etc. Will the robo-spider be able to reliably spin a functional indoor web as well as a real spider? I doubt it.

I could be wrong of course! Maybe transformers can figure out the underlying geometric/physical principles in a reliable way. But zooming out a bit: despite the success of Be My Eyes / etc, I don't think any of us will live long enough to see an AI replacement for a seeing-eye dog. ("We are very sorry about your mother, there was an edge case where the AI didn't realize that green trucks were dangerous.")

[1] More people should seriously consider that LLMs are similar to Lisp expert systems with an easier user interface, but trading reliability for breadth and ease of development. I use Scheme all the time, clearly Lisp expert systems are useful, as are LLMs. But it is also clear that Lisp will never be a model for human/etc intelligence. See also Drew McDermott's classic paper, "Artificial Intelligence Meets Natural Stupidity" https://dl.acm.org/doi/10.1145/1045339.1045340

I think a very direct answer to this is pointing to the hot water Cursor found itself in after an AI customer support agent made stuff up.

- Do you want an LLM salesbot to close a deal your company isn't actually able to fulfill?

- Do you think your company will use AI more intelligently and reliably than the people who made a popular LLM coding system?

The dataset is not “jazz,” it’s Ultimate Guitar’s tablature of jazz songs, so the data is very low quality.

Margaret Sanger's creepy and scientifically repudiated motivations are not relevant to the discussion. It seems like you are playing a dishonest word game: "eugenics" is a set of ideologies and policies which specifically excludes individual choice so that society can impose its views of genetic fitness. But you are using a slippery and unspecified definition that seems to change shape depending on what point you're trying to make.

As to birth control and abortion, you are flat out 100% wrong, they have nothing whatsoever to do with eugenics. These exist because people don't want to disrupt their lives with the difficulty of raising a child. In 99.9% of cases the genetic fitness of that child is completely immaterial. Birth control was also not significant in Margaret Sanger's eugenics writing: she advocated sterilization, and her support of birth control distinguished her from other eugenicists since she thought "fit" mothers still had a right to limit the number of "fit" offspring.

Which leads to another point: you are misstating "positive eugenics." Positive eugenics is encourage "fit" people to have more kids, not to encourage everyone to "reproduce in the best way possible." Again it seems like you are being plain dishonest - which is especially frustrating because I don't think you're arguing for eugenics, you're just trying to win an argument on the internet, and that argument rests on a dumb word game which ignores reality.

Genetic testing actually is a bit more relevant, but there are sharp distinctions between "medically justifiable interventions" and "attempting to optimize your children," and even optimizing your children is not the same thing as eugenics. It is irresponsible and childish to use the same term without being careful.

  The goal of enhancing individuals and the human species by engineering the genes related to some characteristics and traits is not to be confused with the barbarous projects of eugenics that planned the simple elimination of human beings considered as 'imperfect' on an ideological basis. However, it impinges upon the principle of respect for human dignity in several ways. It weakens the idea that the differences among human beings, regardless of the measure of their endowment, are exactly what the recognition of their equality presupposes and therefore protects. It introduces the risk of new forms of discrimination and stigmatization for those who cannot afford such enhancement or simply do not want to resort to it. The arguments that have been produced in favour of the so-called liberal eugenics do not trump the indication to apply the limit of medical reasons also in this case.

(https://unesdoc.unesco.org/ark:/48223/pf0000233258_eng)

I don’t think “organic computer” is a good way to think of a brain. The fact that human brains + some scratch paper can implement any Turing-complete model of computation is very interesting. But that’s not true of chimpanzee brains or orca brains or crow brains, all of which belong to intelligent, thinking animals.

Going the other way, it’s not clear that a Turing-complete model of computation can tractably implement (say) crow cognition. Turing machines can solve arbitrary systems of Schrödinger equations, so theoretically we could simulate every atom in a crow’s body and get an AI crow[1]. But that’s obviously intractable for any known physical computer, and would remain intractable even if we moved to proteins rather than atoms. So are there higher-order “primitives” of crow cognition that can be implemented on a Turing machine? Or is the problem akin to integration, where neat “symbolic” solutions are impossible for most brains, and only “numeric” approaches work?

[1] This is also true for humans: even if you take a loopy quantum consciousness approach, AGI is theoretically possible. Sometimes you see people arguing against AGI on the grounds of Gödel-incompleteness, but this is a mystical nonsense understanding of what Gödel actually proved.

I would guess it's the opposite of "bad best-practices", namely that there are no common best practices. Everyone intuitively understands how black-box redaction works, but the ways inadequate redaction fails is a bit technical and not intuitively obvious, so it's a task that's ripe for ignorant overconfidence.

  Now imagine if the odds of simple life becoming intelligent life that we could detect and could become spacefaring is 1 in 1 million.

Why not assume the odds are 1 in 200 million, so that there's only one such planet in the whole galaxy and there's no Great Filter to worry about? It seems just as valid, except "200 million" is slightly less aesthetically pleasing.

We already know that human-type technological civilization is extremely unlikely, since the planet was full of somewhat intelligent bipedal dinosaurs for about 100m years, yet none of them seemed to engage in mining, large-scale construction, severely disruptive extinctions, nuclear power... (humans will leave a thick geological layer of concrete and pollution, along with plentiful unique minerals coming from slag, plastic, etc.)

I really have no patience for this sort of p(doom) nonsense. If you make up numbers you can come to any quantitative conclusion you want. Mildly tweaking totally unfalsifiable odds makes the Fermi "paradox" go away entirely.

I think the point is that they still seem too rushed (perhaps reckless is a better term), since these are causing serious problems for planes in the area and generating very nasty pollution - what if debris hits an airliner? Nobody's saying they have to get Starship 100% right the next time, they are human. But I am very concerned that SpaceX simply has bad internal testing and design, maybe inadequate computer simulation or telemetry, who knows. But the concern is they'll improperly judge Starship as good-to-go once it launches successfully, having somewhat blindly applied a bunch of duct tape without properly understanding the cause of the failures.

A key limitation is that the early universe had virtually no heavy elements whatsoever, and it is far too fantastical to suppose there might have been life in the form of hydrogen-positron crystals or whatever. It took billions of years of supernovas for protoplanetary disks to include metals, complex organic molecules, etc, and perhaps even longer to make planets with enough chemical entropy to support life.

Orcas could very well be smarter than us, but it is extremely unlikely they'll ever evolve hands to build tools - it would be a radical and crippling change in their skeleton with absolutely no evolutionary pressure to do so. Primates (and koalas) didn't evolve hands with opposable thumbs to build tools, we did so because our ancestors got pushed into the trees. And orcas really need their flippers to be good in the water.

Technically that is the only natural source, it can be made industrially with hydrogen sulfide and methanol, both of which are very abundant in the interstellar medium and on gas giants (Neptune has large clouds of hydrogen sulfide). The reaction is catalyzed by aluminum oxide, which also occurs naturally. On a world with an aluminum-rich crust, I could see dimethyl sulfide being produced in volcanoes. It was striking that this planet had thousands times higher concentration of DMS than Earth; maybe it has vastly more algae, maybe it's purely geological.

  I think it is AGI, seriously.  Try asking it lots of questions, and then ask yourself: just how much smarter was I expecting AGI to be?

That's his whole argument!!!! This is so frustrating coming from a public intellectual. "You don't need rigorous reasoning to answer these questions, baybeee, just go with your vibes." Complete and total disregard for scientific thinking, in favor of confirmation bias and ideology.

No: an LLM that doesn't confabulate will certainly get things wrong in some of the same ways that honest humans do - being misinformed, confusing similar things, "brain" damage from bad programming or hardware errors. But LLM confabulations like the one we're discussing only occur in humans when they're being sociopathically dishonest. A lawyer who makes up a court case is not a "human being wrong," it's a human lying, intentionally trying to deceive. When an LLM does it, it's because it is not capable of understanding that court cases are real events that actually happened.

Cursor's AI agent simply autocompleted a bunch of words that looked like a standard TOU agreement, presumably based on the thousands of such agreements in its training data. It is not actually capable of recognizing that it made a mistake, though I'm sure if you pointed it out directly it would say "you're right, I made a mistake." If a human did this, making up TOU explanations without bothering to check the actual agreement, the explanation would be that they were unbelievably cynical and lazy.

It is very depressing that ChatGPT has been out for nearly three years and we're still having this discussion.

Question 8 on the form, “check if any part of this transaction is to facilitate a private party transfer,” is what he lied about. I phrased it loosely.