ADeerAppeared

There's a fundamental problem with all these "summary" tasks, and it's obvious from the disclaimer that's on all these AI products: "AI can be wrong, you must verify this".

A summary for which you must always read the un-summarized text is useless as a summary, this should be obvious to literally everyone, yet AI developers stick their heads in the sand about it because RAG lets them pretend AI is more useful than it actually is.

RAG is useless, just fucking let it go and have AI stay in it's lane.

Yes, though this scenario isn't all that interesting to discuss:

* In the case of big IP holders (e.g. media companies, news organizations) this is just obtaining consent. The only fun quirck is that OpenAI's purchasing of this drastically lowers the "AI training is fair use" claim by proving the existance of a market.

* In the case of platforms like Reddit, it just kicks the problem one layer down. The platform does obtain "consent" through it's ToS. (Beware that this consent is legally weak, and won't protect your ass from anything outside copyright) But users will still see it as "stealing" and may flee the platform.

There's still a notable shift away from the clearnet.

I disagree, consent is important here, and the "move fast and break things" mentality is what's causing problems.

#1 It's not all that certain AI training is fair use. There could be significant damages if it is found to not be fair use.

#2 "We're going to steal your shit even if you won't give us consent to do that" is a bad idea. We're already seeing the practical results: People stop/reduce posting their works to the clearnet. What's your scraper bot going to do? Create an account on every 'private' forum and immediately get sued? Pay a subscription fee to every single Patreon page?

And the big one, #3: It destroys public support for your technology, which is essential if you want to survive the oncoming government regulation.

Look at the public response to various tech companies stopping their AI rollout in europe over EU regulations. Varying from "Yes, this is exactly what we asked for" to "Good fucking riddance".

And it will get only worse yet. Europe's privacy authorities are already issuing quiet statements that the scraping of social media posts, such as is done for AI data collection, is not legal under the GDPR. (The law's pretty clear on this, it doesn't matter that it was "posted publicly", you're not allowed to use personal data like that) There's already whispers of going after the LLMs themselves, as they contain and continue to process personal data as well.

AI needs public support, and the lack of consent is slowly bleeding it dry.

> beyond the ad-hominem

Note that the "ad-hominem" pertains to only those who care about extinction risk, not AI developers in general.

> which have nothing to do with scaling

Except they are. There's no fundamental change in architecture, it's all transformers. OpenAI's "multi-modal" capabilities are not a redesign of their AI, it's systems bolted together.

Bolting a text-to-image generator & image-to-text describer to an LLM doesn't approach "General Intelligence", it's a system of three specific capabilities.

You could consider that new system "more general" than a lone LLM, but it only approaches General Intelligence in the same way that the series "1+1+1+1+..." approaches infinity. Is each individual step "closer to infinity"? Sure. Is it a useful way to achieve infinity? No, because you'll be stuck adding new steps forever the same as a conventional computer software.

> And it's very blatantly not the case that the entire field is 'only really trying to make them bigger'.

There are some, not many but I'll grant you some, researchers working on actually generalizing these systems.

Their funding is a rounding error to AI spending, their work isn't going to hit production for years. It's not an extinction risk.

You're linking conservative think-tanks, not the original source from the smithsonian.

If you'd look at the original infographic, it's about which norms and values are considered "default" in the US, inherited from (white) European values.

With "Hard work" in specific:

> Protestwant Work Ethic

> * Hard work is the key to success

> * Work before play

> * "If you didn't meet your goals, you didn't work hard enough"

And I'm sorry but the notion that these are human-invented norms and values, not "laws of the universe", is pretty fucking milquetoast.

If you genuinely believe all failures at your job are the result of you not hustle-grinding hard enough, that not even a single element contributing to those failures has been out of your control, I have a bridge to sell you.

The point is that it's a separate discussion, and that losing yourself in that discussion rather than adressing the material harm of the discrimination, is a problem.

And I get it, the discussion is deeply alluring. There is a deep amount of nuance involved. Your point about punctuality is begging for a response about how "punctuality is racist" looks dumb on the surface but belies a deeper point about polychronic cultures that has a lot of legitimate merit to it.

I'll refrain from giving you that full ramble lest I betray my original point, if you or a lurker wants it toss a reply and we can have a go. But note that your link uses a baity headline to grab your attention for a more nuanced point.

> What I am more interested in is the constant pressure on "safety risks" without anything that feels tangible to me so far.

It's marketing. "Our AI is so powerful it's going to destroy the world" is just a way to market "Our AI is so powerful, give us money".

The only people who sincerely care about "extinction risk" are the weird cultists. In the real world there's essentially zero chance of LLMs/Current-GenAI scaling up into AGI, nevermind AGI that'd be an extinction risk. (Yes, that phrasing is cheeky. AGI is a different kind of AI, not just specialized models made bigger. But we're only really trying to make them bigger, and not build general intelligence from the ground up)

> I believe there is indeed risk using models that could be biased but I don't believe that is a new problem.

It's the same old problem with most software, and it's similarly ignored.

These models are biased, and attempts to control that bias are a shitshow. (As Google conveniently showed everyone)

The problem is twofold:

1. This bias has severe real world impact. https://www.theverge.com/21298762/face-depixelizer-ai-machin... This is an article from 2020. We still haven't fundamentally addressed problems like this. These tools are still used by law enforcement and businesses making life-impacting decisions.

2. There's a widespread sentiment that "computers can't be racist". Whenever the bias of these systems hits the news or otherwise gets attention, they're often colloquially described as "racist" (or "sexist", etc), which triggers a swift counter from many techbros going "Um akshually it's not racist, it's merely skin tone reflectivity/it's merely the data set/etc"^[1]

The argument effectively being, "It's not racism because it's not intended, it's merely sparkling discrimination". Yet, this is used as a thought terminating cliché. Nothing is done about the discrimination. Everyone just goes home. "It's not racist, we're not bad people, job done." Leaving the harm of the discrimination unsolved.

This is not without cause. The only way to really "un-bias" these systems effectively would be to extensively curate the dataset and admit that the systems are of very limited capability and should not be used in (non-research) production environments.

Both of these are "impossible". Curating the dataset for current generative-AI would take years and years. And admitting AI shouldn't be used for anything where the bias may have a material impact on the outcome kills the hype bubble. AI firms and developers don't want to address the problem, because the problem is really hard and annoying.

But despite these costs, we should still do it. Because it is the morally correct thing to do. And because regulators are going to tear every company involved a new one if they don't.

---

[1]: A footnote to pre-empt something: I don't care what side of this argument you're on. Whether you believe "racism" must include an element of intent or can be done by machines and systems without intent, there is discrimination with material harm on real people. Whether you call that discrimination "racism" or "sparkling machine discrimination" does not matter. The harm matters, and must be stopped.

> What do you make of the placebo effect?

The placebo effect is measurable. If there is no measurable improvement, there's no placebo effect either.

Bear in mind that what most claims in favour of the polygraph measure is not truth but potentially-false confession. Extracting false confessions is relatively easy, it's also completely f-ing useless to wider society and massively harmful to the victim.

I am referring to their service rather than their LLM in specific.

Perplexity is in the business of using an LLM to paraphrase existing content, then serving that up as their own "work" in a way that directly harms the original content they took.

It's not even a question of "Is AI training copyright infringement", they're just doing copyright infringement with AI. And it's horribly common already.

People aren't going to get it, because you don't get them.

People have the right to learn non-copyrightable elements from your code.

The claim is that AI learns copyrightable elements.

> Someone is likely to design an LLM that is specifically trained to do exactly that.

Perplexity AI.

The simple version is that code is copyrightable as an expression. And the underlaying algorithm is patentable.

The legal term you're looking for here is the "Abstraction-Filtration-Comparison" test; What remains if you subtract all the non-copyrightable elements from a given piece of code.

Legally? No.

In spirit? ... Probably?

Unlike most LLMs, Github copilot can trivially solve their copyright problem by just using only code they have the right to reproduce.

They have a giant corpus of code tagged with license, SELECT BY license MIT/Equivalent and you're done, problem solved because those licenses explicitly grant permission for this kind of reuse.

(It's still not very cash money to take open source work for commercial gain without paying the original authors, and there's a humorous question if MIT-copilot would need to come with a multi-gigabyte attribution file, but everyone widely agrees it's legal and permitted.)

The only reason you'd hack a filter on top rather than doing the above is if you'd want to hide the copyright problem. It's an objectively worse solution.

> If you analogy was you were a human who memorized every variation of a problem (and every other known problem)

This is mere assumption. AI is supposed to work like that, but that's a goal, and not the result of current implementations. Research shows that they do memorize solutions as well, and quite regularly so. (This is an unavoidable flaw in current LLMs; They must be capable of memorizing input verbatim in order to learn specific facts.)

> and there was a tiny perctange of a chance where you reproduced that exact varation of one you memorized

This is copyright infringement. Actionable copyright infringement. The big music publishers go after this kind of accidental partial reproduction.

> but then added an after the fact filter so you don't directly reproduce it...

"Legally distinct" is a gimmick that only works where the copyright is on specific identifiable parts of a work.

Changing a variable name does not make a code snippet "legally distinct", it's still copyright infringement.

Where it gets ethnically dubious is that:

1. The copilot team rushed to slap a copyright filter on top to keep these verbatim examples from showing up, and now claims they never happen.

2. LLMs are prone to paraphrasing. Just because you filter out verbatim copies doesn't mean there isn't still copyright infringement/plagiarism/whatever you want to call it. The copyright filter is only a legal protection, not a practical protection against the issue of copyright infringement.

Everyone who knows how these systems work understand this. The copilot FAQ to this day claims that you should run copyright scanning tools on your codebase because your developers might "copy code from an online source or library".

Github has it's own research from 2021 showing that these tools do indeed copy their training data occasionally: https://github.blog/2021-06-30-github-copilot-research-recit...

They clearly know the problem is real. Their own research agreed, their FAQs and legal documents are carefully phrased to avoid admitting it. But rather than owning up to the problem, it's "Ner ner ner ner ner, you can't prove it to a boomer judge".

> This is the same for humans, if we were asked to compute multiplication without thinking about it for longer than a few milliseconds.

Not to be a jerk but "LLMs are just like humans when humans don't think" is perhaps not the take you intended to have.

> I have never in my life applied strict logic to any problem lol.

My condolences.

No, but seriously. If you've done any kind of math beyond basic arithmetic, you have in fact applied strict logical rules.

I explicitly say that LLMs could do it in my response. As a show of good faith you should try reading the entire comment.

Yes, I'm using simple examples to demonstrate a particular difference, because using "real" examples makes getting the point across a lot harder.

You're also just wrong. I did in fact test, and both GPT 3.5 Turbo and 4o failed. Not only with the rule change, but with the mere task of providing possible moves. I only included the admission that they may succeed as a matter of due diligence, in that I cannot conclusively rule out they can't get the right answer because of the randomization and API-specific pre-prompting involved.

> "For chess board r1bk3r/p2pBpNp/n4n2/1p1NP2P/6P1/3P4/P1P1K3/q5b1 (FEN notation), what are the available moves for pawn B5"

The difference is what I attempt to describe at the end there.

Humans apply fixed strict rules about how to break up problems, like multiplication.

LLMs simply guess. That's a powerful trick to get some more capability for simple problems, but it just doesn't scale to more complex ones.

(Which in turn is a problem because most tasks in the real world are more complex than they seem, and simple problems are easily automated through conventional means)

I should clarify, "of the problem given" refers to the problem given in a prompt.

As you note, transformer (and indeed, most ML) models do create a "world model". They're useful for 'specific' intelligence tasks.

The problem for general tasks lies in their inability to create specific models. To stick with the board game example: The model can't handle differently shaped boards, or changes to the rules.

I could ask a human and a chess-trained AI system to, for a given chess board state and piece, what places that piece can move to. Both have their model of chess.

But if I then ask, "With the rule change that the pawn can always move two spaces", the AI cannot update their model. Where for the human this would be trivial. The human can substitue in new logic rules, the AI cannot.

And that is very core of what's required for generalized logic and "thinking" in the way most tasks require it. What's so troublesome about current generative AI is that it's trained to be extremely general (within the domain of text generation), so their internal models aren't all that good.

Ask an LLM the chess problem above and you might even get a good answer out, but it doesn't generalize to all such chess problems, especially not more complex ones.

> Doing math scales to infinity only given an error rate of zero

This is true, I had omitted it for simplicity; It is still the same approach applied to scaled problems. Humans don't execute it perfectly, but computers do.

With humans, and any other fallible but "true" math system, the rate of errors is roughly linear to the size of the problem. (Linear to the # of steps, that is)

With LLMs and likewise systems, this is different. There is an "exponential" dropoff in accuracy after some point. The problem-solving approach simply does not scale.

> you will see that its approach will not be dissimilar to how a human would reason about the question internally.

"Not dissimilar", but nevertheless a mere approximation. It doesn't apply strict logic to the problem, but guesses what steps should be followed.

This looks like reason, but is not reason.

> But correctly completing "234 * 452 =" is a lot simpler to do by doing math than by having memorized all possible multiplications.

There's a fatal flaw in this theory: We can trivially test this and see that LLMs aren't "doing math".

"Doing math" is an approach that scales to infinity. The same technique to solve a multiplication of 3 digit numbers applies to solving a multiplication of 500 digit numbers.

Ask GPT 3.5 to multiply "234 * 452 =" and it'll correctly guess 105768. Ask "234878 * 452 =" and it gives an incorrect '105797256'

Ask GPT 4o, and you'll get correct answers for that problem. Yet even with the added external tools for such questions, it has the same failure mode and breaks down on larger questions.

These models are architecturally limited to only language modelling, and their capabilities of anything else are restricted by this. They do not "do math". They have a language-model approximation of math.

This can be observed in how these models perform better "step by step"; Odds are you'll see GPT 4o do this if you try to replicate the above. (If it doesn't, it fails just as miserably as GPT 3.5)

What's happening there is simple, the token context is used as a memory space. Breaking the problem down into parts that can be guessed or approximated through language modelling.

Beware of hyping this as "AI can think and has memory!" though. This behaviour is a curious novelty, but not very generalizeable. There is still no "math" or thought involved in breaking up the problem, merely the same guessing. This works reasonably only for cases where extensive training data is available on how to do this. (Such as math.)

It's not a bathtub curve. Your low-level and "high"-level tasks are the same thing: Probabilistic text generation.

It's not reasoning about your code, nor about the explanation it gives you.

> AI has no concept of "these four parts have to be closely connected, building a whole".

AI can't think. It doesn't create an internal model of the problem given, it just guesses. It fails at all these "middle" tasks because they require abstract reasoning to be correct.

Yes. That is a problem.

Tech execs are strongly pressured to keep the AI hype going at all costs, which just makes the damage worse when it inevitably collapses.

> I just don’t understand why product market fit isn’t established and revenue found before launching — these failures seem to elementary to avoid.

You can ask that question of the entire tech industry.

The answer is pretty straightforward: Because being a normal financially solvent business is not good enough. A simple technical improvement is not good enough.

Pichai needs another trillion dollars added to Google's market cap, or it's not good enough.

(And VC has the same problem, all of tech is broken.)

No. AI firms are a good example of precisely why copyright was put in place.

Right now, AI is taking people's original works and rehashing them in a way that directly competes with the original work. Some AI firms (e.g. perplexity) just have their LLMs paraphrase the work lightly.

This is a problem because it drives original work out of business. Even setting aside matters of originality, artistic value, and AI being vapid slop:

Gen-AI is and will remain a derivative work that is reliant on original human-made work

ChatGPT is not going to do investigative journalism. If we let AI push all journalists into bankruptcy, the news just becomes an endless sewer of PR statements recycled into AI slop.

If you don't want Google's woke "diverse nazis" to push real news into bankruptcy, copyright is needed to stop them.

> why would we insert this not-even-stupid noise into the information space?

Because US tech would "collapse" otherwise.

Big Tech's stock price is entirely propped up by AI hype. Metaverse didn't go anywhere so they need a new bubble.

US tech startups are even worse. The VC culture is completely incapable of building useful companies, AI is the last hype. Just a rat-race to grab the last funding and get bought out before the end of ZIRP kills the startup scene entirely.

There's no real demand for any of this shit. Even the web spam renaissance is mostly hustle-bros losing money. The writing is already on the wall with investment slowing down, and AI product after AI product bombing.

You can say that, it doesn't matter. The statistics show that these tools reduce views.

And really, "I'm going to replace my entire news intake with the AI slop even if it's entirely hallucinated lies or propaganda" is perhaps not something you ought to say out loud.

> I think there is a real content dilemma here at work

It's not really a dilemma.

This is exactly what copyright serves to protect authors from. Perplexity copied the content, and in doing so directly competes with the original work, destroying it's market value and driving the original author out of business. Literally what copyright was invented to prevent.

It's the exact same situation as journalists going after Google & social media embeds of articles, which these sites propagandized as "prohibiting hyperlinking", but the issue has been the embedded (summary of the) content. Which people don't click through, and this is the entire point of those features for platforms like Facebook; Keeping users on facebook and not leaving.

This is why quite a few jurisdictions agreed with the journalists and moved to institute restrictions on such embedding.

By all practical considerations, perplexity is doing the exact same thing and trying to deflect with "we used an AI to paraphrase".

> The incentives of Google and website owners were more or less aligned.

The key difference here is that linking is and always has been fine. Google's Book search feature is fair use because the purpose is to send you to the book you searched for, not substitute the book.

Google's current AI summary feature is effectively the same as Perplexity. People don't click through to the original site, the original site doesn't get ad impressions or other revenue, and is driven out of business.

> What will happen if:

What will happen is what already is happening: Journalists are driven out of business, replaced by AI slop.

And then what? AI needs humans creating original content, especially for things like journalism and fact-finding. It'd be an eternal AI winter, all LLMs doomed to be stuck in 2025.

It's in every AI developer's best interest to halt the likes of Perplexity immediately before they irreparably damage the field of AI.

> How would an LLM training on your writing reduce your reward?

Because you're not getting the ad impressions anymore. The harsh reality is that people do not click on to sources, so when sites like Perplexity copy your content, you lose the revenue on that content.

This, in turn, drives all real journalism out of business. And then everyone's screwed, including these AI reposting sites.

Next time you "um akshually", do your homework first.

> These are pretty strong guarantees, and really make it difficult for Apple to bypass.

These guarantees rely entirely on trust in the hardware but it's not your hardware.