intended

I am not sure how that link was made.

Good old ML, which is significantly simpler than LLMs, was capable of ensuring people would not be hired simply because of their names.

The fact that it is misaligned is also not being contended, if anything that contention is made easier to support.

When models are anthropomorphized intuitions of how humans behave end up driving discussion and ideas off track while being too attractive to avoid. This isn't helped when the terminology from the labs and other sources is "intelligence" "intent" and so on.

LLMs simulated more than simple autocomplete.

The autocomplete analogy is rebutting a different point: namely the fidelity of the simulation to reality.

This specific argument is valid. As sophisticated a simulation an LLM is, it is not “thinking” in the same sense we assume other people are thinking.

I am not making an argument about free will, or the uniqueness of human thought, just that the correspondence to how humans reach conclusions and how the simulation produces outputs do not match on a 1:1 basis; as a result attributing traits builds incorrect intuitions.

The greatest predictor of AI productivity I have seen has been operator maturity.

AI usage tends to atrophy skills, unless used in a very mature and self aware manner.

All of which depend entirely on the operator having developed a mature theory of mind, and experience with living and working within their own head.

The model seems defensible, but for some reason it isn’t passing the sniff test. Maybe its that the extreme scenario seems wildly optimistic, or that the observed modest productivity gains are framed as either the trend or “just the start of something amazing !”.

The issues are that

1) The median AI use case isn’t increasing productivity but expanding capability and reducing the need to communicate across teams. Ex: if someone needs a dashboard or a ppt designed, now they don’t have to ask another team. AI is substituting for intern and early career workers in someone else’s firm.

This is also borne out in the hiring data.

2) The cases that show maximal productivity gains are when AI is leveraged by experts.

This is a problem, because the evidence for automation gains is poor. AI automation seems to be going to the same place ML projects went to die.

If augmentation is the primary lever that lifts up productivity, and it is dependent on AI being matched to experts, then the limiting factor is expertise.

This makes the fact that AI is both, taking out early career roles AND one shooting education, a divine irony.

Where exactly are those future experts supposed to come from? The ones with taste and the ability to deliver us this AI driven GDP growth?

AI is also setting up a conflict between liability, insurance and productivity, and thats a whole point in itself. However it will be expressed as a drag on productivity.

This imbroglio looks like a point in the journey for a firm that is realizing it is not going to make money selling shovels during a gold rush, and that it has more to gain from just… being vertically integrated across an industry.

Open models have definitely hampered the ability to sell tokens at a premium, so mass market adoption is impossible.

But then take someone like Jane Street, for example. They self report making $30bn leveraging LLMs. It’s a defensible assumption that they are making profit on it.

Perhaps it’s more profitable for OpenAI/Anthropic to build their own funds. They have the capital, compute, they can afford to recruit teams and buy any IP/Data required.

This isn’t a fully fleshed out argument, but it is the first time it feels like the winds are changing.

There are multiple aircraft manufacturers for smaller aircraft, and there are several showing up for electric aircraft today.

For jets, the capex required to build them is massive, and its never going to have an upstart just up and build a jet.

The regulations in place are the reason why its safe to fly, and airlines have the safety records they do.

This is the degree of safety end consumers want to be reassured that its safe to fly. The checks and balances in places create the market that you are hoping to give over to cheaper upstarts.

The market doesn't exist without that safety regulation.

See how people feel uneasy when flying Boeing after the decay of their engineering and quality culture became public knowledge.

> Meanwhile the fields that got to be regulated last brought us the most wealth, advances and innovation: high tech & software.

Financial markets are highly regulated. They are one of the most structured markets out there.

The outcomes in all those markets, whether they are good or bad, is a direct result of how the markets are structured.

The 2008 financial crisis, for example, was a direct result of regulators being defanged and underfunded for multiple years.

If you look at technology: innovation amazing in the era of unlimited horizons, the era of "move fast and break things."

However, the cost of externalities from inventions like social media, were not captured in that market structure. This means that firms can absolutely pollute or addict people in the information economy, capturing the profit but socializing the costs.

The ability of dominant players to consume/acquihire upstarts and keep their market position, is a drag on innovation and consumer outcomes today.

The situation that was faced in the early days of a new technology is different than what is faced in its maturity, when the advantages of the nimble upstart give way to the massive overlord.

At each stage, you have to craft the incentives and rules to ensure you achieve the outcomes you (as a nation) actually priortize.

The choices you are presented with, are always a function of market structure, aka the interplay of incentives, regulation, and competition.

You have a market because your taxes pay for many things, ranging from policing, to courts, to power and standards underlying how various systems work.

The choices presented to you are always a trade off vs other options. Having regulation that dictates how plug points and voltage works for example, reduces the choices you have, but the baseline standard means many other things can be built because standardization enables modularity.

The position that there exists some platonic ideal "choice" isn't even useful as an illustrative argument in this context.

Markets that are effective at ensuring better outcomes for the median or average consumer are things that have to be crafted. Figuring out the correct extent of intervention, (whether you champion less regulation or more) is the issue.

For arguments sake, let’s assume two people watch an event and found religions based on their perception of it.

This is purely a matter of belief, based on what they saw. No one is lying in this situation, they just perceived different things.

This seems like a fantastic extrapolation from your premise, but the degree of change you are arguing for will reach exactly this point, multiple times throughout history.

Fundamentally, you are arguing for a different physical reality than the one you inhabit.

This is always going to be the case, because regulation is typically a form of friction.

It is the shape of regulation that matters, and ideology on either side of the divide is religion.

Good regulation and regulators would punish anti competitive behavior, but this will also act as a drag on firms, while preventing firms from reaching certain sizes and potential economies of scale.

The point is that regulation deserves to be understood and crafted well.

Those economies of scale are meaningless to consumers if it means a single firm squeezes everyone in the supply chain and captures the benefit for itself.

This is also a specific weakness in American conversations. The divide between free markets, competition, and regulation are ideological.

If you want good markets you need good regulation and incentive structuring.

Markets, left to their own devices, do not end up automatically being competitive in favor of consumers.

But above all, if AI wasn’t hyped as a threat to humanity, to jobs, to security, regulation could have been cautious.

To add insult to injury, given how voters are tired of technology, expecting a different move from governments is a losing bet.

That said, I think there is a lot of appeal to go hunting for firms, since it really feels like corporations and their owners are engaging in predatory behavior as opposed to customer centric behavior.

I get the appeal, but lying is a sub-category of deception, and deception itself is a child of error.

Meaning deception is inherently something that the physics of reality allows.

In the most simplistic sense, the camouflage of moths that look like snakes, or a chameleon’s ability to change colour, is deception.

In that sense, deception is the ability to fool the sensors of a specific category of targets. It follows that detection is easier if you manage to identify a category of signals that the deceiver has not accounted for (and the detector can access).

Deception of this nature is critical for things like revolutions to occur. Without the ability to hide and blend in, the most dominant faction will always hold sway.

The rule of the dominant faction, even in a pure truth world, is an issue because errors and randomness exist.

You can have people witness an event and based on the physical position they occupied, perceive different things occurring.

Error and time pressure is sufficient to ensure that individuals and groups make suboptimal decisions, that lead to rule and domination based on erroneous information.

As long as error exists, deception will exist and so lying will exist.

How would you estimate actual gains and losses to education for LLMs? What would good look like?

You are free to disagree, and you are free to have an opinion. I would prefer your position to be the one that dominates, however the data on the ground doesn’t support the position.

If I can make a guess, the issue is that you are looking at the potential for individual learning improvement but I am referring to what is currently happening at scale.

If you remember school, most students didn’t want to be there. Homework exercises are something you would rather avoid.

America and the west, as class mobility became increasingly dependent on education, repeated the patterns in China and India. People went to college not because they wanted to study the subject, but because it got you a decent income when you got out.

In all these situations, you have people who aren’t there for the journey, but there for the outcome. (For that matter, even people who want to study, will take the easy way out.)

Having seen people use AI to study, it can be useful, especially if the underlying text books and notes are bad. However there are already many YouTube videos made by experts covering those topics, so the uplift here isn’t magical.

> I really disagree with you that the technology itself is inimical to learning.

This is a matter of belief, and in a vacuum it’s something most people would agree with. The issue is what the outcomes actually are.

This isn’t a metaphor/analogy that works. There is a parallel in the introduction of a tool that has more “firepower” but that isn’t the crux of the issue when it comes to intelligence.

You can give fable to someone who has never written code before and someone who has experience writing SW. Your outcomes are very different.

The crux of the matter is that you need to know what “good” looks like to use automation or augmentation tools well. This is why it’s a greater boost to people with experience.

https://literacytrust.org.uk/research-services/research-repo...

We also know from colleges and schools that students coming in are having difficulty doing the reading needed for classes, even in literature classes.

AI is making this worse, since it’s unleashed an epidemic of cheating and the tool itself is inimical to learning. (Learning is friction. The tool to remove friction also removes learning)

The issue isn’t that we have stupid people then and now. The issue is that trends of literacy and development now are changing direction.

As I said clearly, I haven’t made any point on the computational substrate. The point is on the computation being run.

I dont need to bother about free will for my argument.

Please take a look at what I am saying as it has little overlap with your objection.

The simulation solution is backed by their own applied example, and is also the logical intervention based on how the problem is set up.

Perhaps Bainbridge is doing more heavy lifting for me, by filling in blanks and gaps that others are finding.

the less friction the less growth.

So even if they copied from SO they would have better knowledge than an Ai user.

This isn’t speculation either, theres an MIT study which this is based on. https://www.media.mit.edu/publications/your-brain-on-chatgpt...

https://www.sciencedirect.com/science/article/abs/pii/000510...

The article is based on evidence, and states it clearly in the body.

Firms aren’t just selling products, they are selling reliability and taking on liability.

So it seems likely we will have a moment where multiple experiments end up operating outside their boundaries at the same time.

It is what computation is being run.

Humans have intent, let’s take this as an assertion.

Models run simulations that act similar to intent. However they are not the same as intent and the simulation is not a 1:1 correspondence.

There is no “intent” here, there is pseudo intent. If you are only concerned with outcomes and not the actual nuts and bolts of how those outcomes are achieved, this distinction will be meaningless to you.

If you are actually thinking about what is going on, and what can be done to prevent such outcomes, then assuming there is any such thing as “ethics” results in misaligned assumptions at best, and wasted effort looking in the wrong directions at worst.

If the agents acted based on “ethics” then the solution would be to check the ethics they believe in and change those.

However there is no belief system at play here, simply a simulation which was instantiated in a certain way. Which brings us to the annoying voodoo part of LLM training. Everything goes back to how the initial training data is shaped.