js8

Today, the difficult shit is how to get the resources from the people who use them to turn difficult shit into an easy shit. Basically, a war. I am not sure I want people to move on to that.

I agree with the 2nd sentence onwards, and I think I expressed it in the other comments I made here. I am not really sure what was your point about "execution", though. Humans might have inner interpreter loops as well.

I wish Terry Tao would prepend the guest names on his blogposts, this is not the first time it's confusing.

I disagree we understand LLMs, your view is quite reductionist. It's like saying we understand human brain because we understand physics and evolution, or that we understand some program because it's just a binary code in a known instruction set.

I think Dennett's theory of intentionality (see https://en.wikipedia.org/wiki/Intentional_stance) applies here. We do understand LLMs from physical stance (ML algorithm and inference), but we don't fully understand them from design stance (it's internal workings have been evolved so it's hard to tell the functional units) and from the intentional stance they are a complete mystery.

And I talk about obstacles to this understanding elsewhere in this thread.

Not really sure if it's sarcasm, but let me make a side remark.

It's really stunning how much more effective the "Standard ML" notation (embraced by Haskell, Lean etc.) is compared to writing proofs in classical logic.

This "UX problem" is, I think, the reason why is mathematical community embracing automated provers maybe 50 years later than they could have. Automated people wanted the better language, but the mathematicians largely resisted.

So seeing this, it would be preposterous for me to think that any language, natural or not, has the last say in this. We're gonna be stuck with learning new languages and formalisms for a long time.

I disagree, to keep it short, what you're describing is not understanding, it's superstition. And I think it's a wrong direction of engineering, relying on some sort of irreproducible expert intuition, one that has been successfully replaced by enlightenment and scientific method.

There are 3 major obstacles in understanding LLMs:

1. They use inscrutable internal language of embeddings

2. They communicate in natural language which is itself ambiguous

3. The weights and training inputs are being hidden as a "trade secret"

"with the added advantage of having been trained on a HUGE number of codebases"

This doesn't really mean much unless we understand what is the quality and relevance of these sources for the problem at hand. Without this understanding it's just a superstition.

I agree LLMs are useful tools, but I am dismayed by a lot of cargo-culting around them, which happens because we don't understand them. I think we will have much better tools when we understand what is the expectation and what is the algorithm they run.

Somebody else said that the magician analogy was poor. I like magic tricks, but it took many years of cultural change (influenced by people like Houdini, Randi, Penn & Teller) to stop illusionists (and mentalists) make claims they have supernatural abilities, or people believing it on their own (a magician pretending to be able to catch a bullet was shot by an audience member who didn't understand the distinction).

It is detrimental, I think, to treat LLMs as if they have magical abilities ("superintelligence") rather than understanding they just run some clever algorithm. The fear for (programming) jobs comes from that framing; nobody fears of their job because of compilers, since compilers are understood.

(And it actually runs against kind of "socialist" framing of the problem, which I agree with, that is why should people be worried about the jobs in the first place, when society is getting richer as a result of better tools?)

"You could say the same thing about human programmers, but I've never heard anyone say they think that programmers "execute" Jira tickets."

Yes I could. We have "executives", for starters. And first "computers" were actual humans.

"That was eons ago."

Yes, technically I should call them LRMs (large reasoning models) not LLMs. But that doesn't seem relevant here, to my point they encode some logic (which we want to be close to classical logic, i.e. behavior of words like "true", "and", "not" and so on matches).

I would agree with EDW, and I have argued here in a similar way.

I am not against use of NL in negotiation or poetry. If you find ambiguity useful there, be my guest. But engineering specifications, mathematics, as well as other sciences or even philosophy would IMHO benefit from more rigor.

I also strongly disagree with the notion that logical or programming languages cannot express ambiguity. (It actually took me many years to understand.) I used to think you need something like fuzzy logic or probability, but that's unsatisfactory in some ways. Eventually, I settled for a really simple understanding of the problem.

Take lambda calculus for instance. I define the term to be ambiguous iff it has a normal form. So it is ambiguous if it expects additional argument, which resolves (part of or all) the ambiguity. Terms with no normal form are completely unambiguous, their "output" is completely given.

In classical logic, this corresponds to formulas that are conditioned on additional assumption. Again, the extra assumption can resolve the ambiguity.

So it is kind of my conviction (although we could show that by translating an LLM as a program into LC) that all the words in natural language can be formalized as sufficiently complicated lambda terms, that all have normal forms and react to each other in a way that resolves some ambiguity without ever resolving all of it.

Well.. I think this is a big open problem in philosophy.

On one hand, you have things like Lean (calculus of inductive constructions), these are relatively simple formal logics (just in more practical notation) that let you define any conceivable type, which is akin to specification.

On the other hand, there is a rich set of modal and fuzzy logics that can help with aspects of reasoning in natural language. I think these can be defined in the former, but nobody has really made a good agreement as to how.

So the main difficulty is for any such language to gain traction, people who speak it.

Instead, we trained LLMs and they came up with something (evolved to reason). I think the future philosophical research will need to answer what exactly do LLMs bring to the table in terms of formalization of natural language.

I disagree with each sentence for a different reason.

LLMs interpret (so, "execute" in a way) natural language in the sense they have internal logic that assigns to the sequence of tokens in context a next token. If we delineate the input and output into a series of logical statements, we can think of it as a program that builds a logical statement from a list of input statements. So it encodes derivation in some logical system.

However, the internal logical system is informal in the sense that the above rules are not guaranteed to be sound on the fragment of classical logic encoded in the natural language. It is a close approximation, though, so it often works.

To add, half of my problem with natural language would be resolved by agreeing on exact definitions, which is kinda what LLMs do internally. However, they don't surface this formalization very well(even with open weights it's difficult), which makes it pretty unusable.

I am not sure what your counterargument is. But in mathematics and computation, people have tried for at least 150 years to move away from natural language, and figure out stable foundations that can be externalized. I think there is a good reason for that - you save time correcting errors due to different interpretation.

I answered this to myself - stop worrying about LLMs. It's pretty simple: due to Curry-Howard isomorphism, programming languages are just notations for some type of formal logic.

Now ask yourself a question, what language do you want to maintain the programs in? Do you think natural language is going to be easier and more maintainable than formal logic?

The answer is no. So you need programmers, people who can read the formal description and adapt it to new requirements.

LLMs are amazing technology, but the truth is - natural language just kinda sucks. Therefore, you don't really need them (see also https://en.wikipedia.org/wiki/AI_effect ).

I think people love LLMs for the same reasons they love magicians. But just like the magician employs a hidden trick, LLM just runs some algorithm you don't see or understand.

So worrying about LLMs taking programming job is kinda like worrying that a magician will take a warehouse worker job, because they can levitate stuff. Meanwhile, we already have automated programmer - it's called a compiler.

It's basically a convention. The alternative is to raise interrupts of course, but that might be application specific, or use other invalid instructions than the designated one, but they might work differently on other processor types.

Philosophy yes, but Dennett's design vs intentional stance is a more appropriate tool here.

Design stance is, LLM is a next word predictor.

Intentional stance is, LLM can reason, according to what rules?

Graeber actually points out that centrally-planned economies had formally higher work hours than was the reality (people slacked off more), and that it was akin to workers collectively making a choice working less for the overall lower production.

But the obvious problem is that it's still a societal taboo, so it's not UBI because it's not universal. It's probably lot more efficient to actually openly embrace shorter work hours, so that people can stop pretending and choose to do something more useful.

Also, existence of BS jobs is often driven by high status individuals, so it is comparatively less free than UBI.

I originally read the headline as "Silicon Valley Is Transforming Military-Industrial Complex".

Is Microsoft still trying to embrace, extend, extinguish Netscape after all those years? :-)

No, the term itself is not stigmatizing. People stigmatize legitimate psychiatric diagnoses, and you're perpetuating this idea in your comment, by blaming the term for the moral failure. These labels don't exists to stigmatize people, but to study the problem and help people.

I was bound to agree with you before I RTFA. The classification is useful for a psychiatrist, i.e. someone who wants to help the person.

"Bad coworker" can have many causes. In prolific AI psychosis, I think the assumption (differential diagnosis) is, they became a "bad coworker" due to use of AI, so they were a "good coworker" before.

> Maybe you can upgrade digital systems until they are 'fully fortified' in theory, but humans have an upper limit what they can understand / perceive / critically analyse (not to even speak about engineered high pressure situations) that can't be upgraded.

There is an answer to that - simplify and abstract. Lots of human software is unnecessarily complex, often caused by backwards compatibility and general human creativity.

Take sendmail vs postfix as an example of this process.

Not to mention, the idea that Germany, 3rd or 4th largest economy on Earth, cannot defend itself on its own and needs American troops "to defend itself" is just ridiculous.

The Americans are there as a front of US empire (IIRC Rammstein is being used for Iran war).

It's not clear if it is foolish.

The situation will probably be like with airline pilots. The plane is mostly an automated system, but we simply don't let automation to work through exceptional situations, without human review. Therefore, lots of pilot training (and their presence) is still required.

But it's also hard to guess what exactly skill should the pilots have. It's clear that lots of things from manual aircraft do not translate well to automated.

The "next-token predictor" is taking a design stance (see Dennett). But to take the intentional stance is more interesting in case of large language/reasoning models.

No, it's not a problem. The suppliers cannot keep increasing the prices to match UBI, because they will face increased competition from people who now have other options (more freedom) thanks to UBI.

For example, let's say I buy vegetables. With UBI, I might spend some time to grow my own. Or buy from a neighbor who expands his production.

It's true that price of labor might rise with UBI, but that can only rise to the share of labor as input. It will also force to replace labor-intensive products with capital-intensive (automation), which is a good thing.

The US GDP per capita is 94k a year. The average salary is 69k a year; this is a proxy for the price of labor. So you can easily have 20k UBI for everyone in the US, the country is rich enough.

No, you cannot say the same about computers, because the technology is almost never the problem. The problem is cultural.

The deregulation since the 70s created oligarchs and encouraged lack of moral scruples. See also https://news.ycombinator.com/item?id=49416055

It's a coincidence that it occurred with the rise of the Internet (computers existed before) and AI. These coincidences happened in the past with other technologies.

So the OP has a valid position to boycott AI, just like in 1870s they should have boycotted trains or in 1910s the oil industry.

Of course you can! It's called boycott.

The workers who strike in a factory don't do that because they think the thing being produced is being socially useless. They do it because they demand social change.

You should ask the billionaires about "dignity of work", they are the welfare queens. And also about "independence" on the people who work for them. What you're saying is just classist prejudice.

I don't understand what you're saying. I agree that food insecurity could be fixed; it's a redistribution problem. UBI is a way to fix it.

But I don't see how we are forcing people to be low class with UBI (as opposed to having them unemployed or doing precarious work that doesn't pay even the basic needs).