615 karma · joined February 23, 2024
The problem here is that Google claims immunity on being a platform. "It's not use doing the trademark infringement, it's General-ScamCo"
And as it stands right now, US law is quite favourable to that defense. Even when in reality, it's obvious that advertising platforms are Publishers. That they not only make editorial decisions, but that making those decisions is the entire fucking point of them.
It does not tell you how it got that key. A compromised server would send you the key all the same.
You still have to trust in the security infrastructure. Trust that Apple is running the hardware it says it is, Trust that apple is running the software it says it is.
Security audits help build that trust, but it is not and never will be proof. A three-letter-agency of choice can still walk in and demand they change things without telling anyone. (And while that particular risk is irrelevant to most users, various countries are still opposed to the US having that power over such critical user data.)
The idea that these are people who are "broken" and need to be "fixed" is something a lot of Deaf people are very opposed to.
What are you on about? This is precisely the kind of situation where a best-effort can be done.
Though let us swap your example of "sex lines" for something legitimately harmful. Scamming callcenters.
Should they stop all scams? No, they can't. Inspecting phone call traffic to monitor for scams would be extremely invasive. Should they disconnect the firm they and everyone else knows are scammers? Yes.
Thus the point. When these companies don't make a best-effort, the government gets involved.
The only added complexity here is that the chat apps themselves can't do anything. But the big tech giants could stop making the problem infinitely worse, big platform holders who already monitor publicly-posted content could work to ensure their reports are actioned upon.
I'm not talking about "all of privacy" being lost in one go, I had assumed this would be obvious to all readers.
I'm talking about the specific matter at hand; Message scanning, with the point of how this failure mode poses a threat for further and further infringements of privacy down the line.
> If privacy dies, it ends creativity, civil society and funding for politicians.
Cute poetry, but the aftermath of 9/11 clearly shows otherwise.
Consider for a moment how even these "anti-terrorism" measures are seeing shockingly little pushback. Despite al-Qaeda being an ocean and continent away.
You're a fool if you think any politician would risk their career to revoke message scanning once implemented. The attack-ads write themselves, "[POLITICIAN] wants to let pedophiles sextort your children".
These proposals keep returning because there are legitimate concerns that are being left unadressed by industry self-regulation.
When nobody wants to have a conversation about good measures, politicians are going to try again, even the new ones.
It is sincerely horrifying what the reckless deployment of generative AI is doing in this space. And it's all dismissed as "Well you can't stop math anyway so we're not going to do anything" and (implicitly) "The ends justify the means, sacrificing a few children is worth it to get AGI". These are terrible optics.
If tech does not hold itself accountable, privacy is going to die sooner or later. The politicians need only win once.
He's speaking the truth.
Google has, for a long while now, let scammers just buy advertisements to get their fake scam page to the top of the results. And not just major banks, various open source software have been subject to this exact attack.
It's imperative for security that you install adblockers on all their devices.
I disagree.
The moment you accept the need to always check the source, there's no fucking point to this anymore.
If you have to check the source ... just provide the links like Google Search does. There's no point in a summary whose full original text you always have to read.
> From what I've seen, it is true that these errors stemmed from uncritically accepting satire/unreliable sources rather than hallucinations.
And so, this is why I don't think this holds up; If we consider this a failure mode of the user, the entire tool is pointless.
It's plainly obvious that the intended benefit of AI Overview is not having to read the pages of the search result. And with that intent, this is not a failure of the user.
The dirty secret of a lot of AI is that they rush out a hotfix for any such issues that pop up, and then just insist the issue was fake. "You can't replicate the issue, our AI is flawless, DO NOT LOOK AT THE PATCH ROLLOUT BEHIND THE CURTAIN".
They're not.
Take "This means that AI Overviews generally don't “hallucinate” or make things up in the ways that other LLM products might."
This is just a lie, in two ways:
1. It's still an LLM, feeding search results into the prompts and asking for a summary reduces the probability of hallucination compared to directly asking the question in the prompt, it's still a probability non-negligibly above zero. This kind of pretending you can "fix" the hallucination problem by feeding data into the prompt is extremely wrong and dangerous.
2. It's missing the entire point. "Um akshually the LLM didn't hallucinate, we simply gave it Reddit user Fucksmith's post which it confidently restated as truth" (or indeed, the much funnier, "we gave it The Onion, which it confidently restated as truth") is functionally the same as a hallucination to the end user. The mechanism here does not matter.
These are both fundamental critical issues to AI overviews. "Rare" hallucinations are unacceptable on a tool used billions of times a day. And the LLM paraphrasing lies/satire/ignorance/etc into truth is a fundamental flaw of using LLMs to paraphrase like this.
You can't just patch this issue, the entire thing is fundamentally flawed.
It is that.
These AI firms have been very vocal about their "ends justify the means" attitude. (Which is doubly damning considering how little these firms care about any purported AI safety for their AGI dreams. Months of "AI is going to kill us all" while they couldn't even be arsed to stop using CSAM-contaminated training data, nevermind using quality training data to ensure alignment)
And "Foreign state actors generated some propaganda texts" is very small as AI-problems in the here-and-now go.
To the user
But these tools are marketed as if you do only need to run them once to get a good result; The companies behind them would really want you to stop hammering the button that deletes their money.
As an aside:
> For evaluating prompts and running in production; your hallucination rate is inversely proportional to the number of times you sample.
This isn't really true, and requires you to fuzz the prompt itself for best effect. Making the "spam the LLM with requests" problem much worse.
The reason is slightly different here.
What's so desirable here is an AI system with such general intelligence that it is capable of such mathematics by itself as a consequence. Not because the mathematics is so useful, but because the required reasoning capabilities are at such a level that, we could speak of an artificial intelligence that is meaningfully "general" about any problem.
It's a decent approximation of "able to solve any problem" that we can still reasonably test.
> Let's be a little more concrete: do you think FormalGeo [1] is a big deal?
It looks to be an interesting approach in modelling mathematics, and their use of machine learning is an interesting novelty that may pave the way to more useful general mathematics systems, but I can't find much about how these systems might interop with current/'generative' AI systems.
And that last bit is one of the big roadblocks for current AI. They're very weak at reasoning, but we can't directly interop to (e.g.) LLMs, so we can't compensate for that weakness.
The problem is that it's not particularly useful: As the problem complexity increases, the user will need to be increasingly specific in the prompt, rapidly approaching being fully exact. There's simply no point to it if your prompt has to (basically) spell out the entire program.
And at that point, the user might as well use the backing system directly, and we should just write a convenient input DSL for that.
Quite the opposite, it's the holy grail of all AI.
Consider various work that isn't (and can't) be done by computers/robots/etc right now.
The intelligence constraint is universally, a required amount of problem solving. Even the "low skill" labour requires it.
And to perform such problem solving, you need advanced logic and reasoning capabilities, which is the same thing as novel mathematics, just applied to a different end.
This nitpicking is a red herring.
The issue that separates "AGI" from current AI systems is the lack of generality. (Humour me.)
In particular, the lack of reasoning capability. And what the pessimists argue here is that there is no road to get there for current systems. Transformers are approximation machines, and are generalized for that specific task. But that's also where it stops, they can't do things that aren't such pattern-approximation.
Optimizing a transformer for arithmetic isn't a step towards AGI, because it is not generalizing. You'd need to do this for every conceivable task and subtask. This is the exact reason why imperative-programmed AI architectures were discarded.
Put bluntly, this approach will never get you a transformer that won't shit itself when asked to do novel reasoning tasks, such as novel mathematics. (Which I will remind the reader, anything but the basic programming work counts as)
And critically, the fundamental architecture of these transformer systems doesn't allow the combination of them into other AI systems to acquire generalized capabilities. There's no way to make an LLM hook into a computer-algebra-system, you can only feed 'finished' output of one system into another.
On one hand, their support for outsourcing programmes; "Training Indians on how to use AI", suggests they realize AI tooling without human cleanup is a crapshoot.
On the other hand, they keep digging. This kind of gaslighting is an old and proven trick for genuinely rare problems, but it doesn't work if your issues are fairly common, as they'll get replicated before you can get a fix out.
Similarly, they're gambling with immense legal risks and sacrificing core products for it. They're betting the farm on AI, it may kill the company.
You don't even need an LLM for that. Google will almost certainly have tested.
The test result is just politically-unacceptable within the company: It doesn't work, it's a architectural issue inherent to the technology, we can't fix it.
Instead, they just rush to patch any specific, individual errors that show up, and claim that these errors are "rare exceptions" or "never happened".
What's going on here is that Google (and most other AI firms) are just trying to gaslight the world about how error-prone AI is, because they're in too deep and can't accept the reality themselves.
The problem being the aforementioned conflict of interest. The moment you're a competitor to Google Search (or any of Google's other products), they can kill your API access and ruin you.
It's very hard, getting harder as websites grow more hostile to crawlers. (With more and more putting up login walls to keep poorly written AI crawlers out.)
But the real problem is that tech is culturally unused to dealing with monopolists "on even ground". The wider tech community operates on the implied worldview that they can take on anyone by just asking daddy A16Z for a duovigintillion dollars. For 25 nearly-continuous years, you could just outspend to get into even the most hostile markets.
That economic climate is just ... gone now. No more free money. You can't just buy your way to the top anymore, the business has to financially work.
And most importantly: You can't outspend the incumbents anymore, but they still can outspend you.
That's the dumbest part of it all; Kagi's not an entirely new search engine. It's Google's index but without the ad-tech juicing, and some clever use of a few other search indices to specialize results.
(I.e. Google could unshit itself at any moment, it just doesn't want to.)
That's the big problem here. All the new search engines rely on the infrastructure of the old ones. It would take very little collusion between Google and Microsoft to crush something like Kagi. And the moment any new competitor gets big, they'll do it.
What's needed is an anti-trust breakup. There's simply too big a conflict of interest between Google, the search engine product, and Google, the ad-tech giant earning money off the websites.
Google is using their search engine monopoly to control the wider web, and fill it with spammy SEO garbage to juice their advertising revenues.
But to say something useful, let me try to elaborate my general criticism here:
> Prior systems either dumped massive amounts of cognitive load in the investigators face or took man years of effort to create a specific workflow, and in an adversarial dynamic space like fraud you need a much more dynamic approach to different types of new attacks.
This begs a question: Why didn't a computer system to summarize this data already exist? Or rather, what stopped the prior systems from doing this work? (And I'll consider conventional machine learning; classifiers and the like, as traditional computer systems here)
And there's generally two options here:
1. Conventional computer systems absolutely could do this work, but they just haven't been built. (Say, because nobody signed off on the R&D but would sign off on AI hype R&D)
2. The LLM system is doing a task the conventional computer system cannot do.
Number one's problem is simple: It's just inefficient and wasteful. Number two is a red flag: There's very little overlap between the things a conventional computer system cannot do, and the things you can trust an LLM to do reliably.
As you describe this system, selecting which data is relevant for fraud investigation is a very traditional classification task. Using normal machine learning for that is basically industry standard.
So what's the LLM actually doing? Subtract the hard logic of normal software, and the classification of machine learning, and the answer is generally: A complex nuanced reasoning task.
But that's precisely what LLMs are not to be trusted for, because they are incapable of that kind of reasoning.
> Listen. When John Henry battled the steam drill he did win, but it killed him.
You're missing the point I was making with that remark. It's not about firing people or not.
It's that these systems are dangerous to evaluate from a high level. It's very easy to miss externalities that'll tip the entire endeavour into a net-negative. You need the investigation of what exactly the AI systems are doing, on a specific detailed level.
E.g.:
> This is useful if say you have business people or whatever writing effectiveness or whatever testing where they can provide a specification of policy and a well prompted LLM can generate pretty exhaustive cucumber tests (which can be pretty redundant and formulaic when asserting positive and negative cases exhaustively) which can then be revised by hand as needed.
"A specification that has been prompted into sufficient detail" is just a program. You're describing the most inefficient declarative programming stack on the planet.
Granted, the programming stack to actually declare business rules this way isn't very good, but using AI here is just an error-prone transpiler.
It's very easy to "looks good to me" these tests and claim the project a success, yet miss subtle errors in the generated tests. I remain skeptical about how well these tests will hold up in the longer term.
> The bigger AI companies like OpenAI and Anthropic are not losing money in their monetized APIs.
[CITATION NEEDED]
Inference is more expensive than claimed, used extensively as a 'slot machine' with users trained to just keep re-generating until they get something useful, and only keeps getting more expensive as model quality has to go up.
And in practice, training the model is far less one-off than claimed. Current tools are not sufficient.
> Every place I’ve worked is absolutely reducing costs and doing new / more business as a result of their LLM use.
Unless you are working in SEO, Marketing, or spam, I don't believe you.
LLMs aren't reliable enough to replace actual human labour. While it's true many companies are fooling themselves into believing they're reducing costs, in practice other staff is picking up the slack. This is unsustainable unless your company has massively overhired.
Things like "AI generated software tests" are a farce. The consequences aren't immediate, but will show up long term.
Copilot+ runs locally, but has the quality of 2 years ago, it's basically useless for anything but shitposts and spam.
The cloud AI tools all burn hideous amounts of money, all ran at a loss.
AGI is a red herring, and will not happen. The architecture of generative-AI systems simply doesn't permit the required logic and reasoning capability.
Even the "Actually, Indians" concept of outsourcing the tertiary sector to the developing world by way of having low-skill workers clean up AI generated trash is unviable. (It both doesn't work, and is politically doomed.)
What's going on here is that tech companies are tearing up everything to pump their stock prices after the covid-tech-boom and ZIRP ended. Burn down their core products to keep the bubble going just a bit longer.
* For the data sources, we know there's a lot of CSAM in there, and that OpenAI knowingly shipped this data to 3rd party companies to tag it. Time ran an article about their Kenyan parters, who quit on them because "Look at this image. If the image is CSAM, push the button" is a horrifying job.
* There's copyright. AI relies extensively on hiding the true scale of the copyright problem by filtering infringing content out of the AI responses. But the engineers of those systems know the dirty details. (It's dubious that such filtering would bring them back into compliance with fair use; LLMs are paraphrasing machines and copyright filters do not cope well with paraphrasing)
* Discrimination is a shitshow. LLMs discriminate as their dataset isn't reflective of reality, but of what and how we choose to record reality. And as Google's "Diverse Nazis" image generation shows, messing with prompts doesn't work to fix this. It'll be discrimination lawsuits either way. All major AI firms know of this problem, and internally investigate it to avoid gaffes like Google's.
In practice, the boring answer to the question is "all of the above and then some". A big problem for OpenAI is that these are massive problems, for which their engineers could be dragged into a courtroom to testify about.
While it is technically possible for you to read some GPL'd code, memorize it, and then reproduce it later by accident. That's not how programmers work. What humans remember is not the copyrightable code, but the patentable algorithm. (And there are few algorithms that are simple enough to memorize on a cursory reading, novel enough to be patentable, but not so novel you'll remember it's source)
AI does not work in algorithms. It's a language model. It deals purely in the copyrightable code. (Both figuratively; LLMs are structurally incapable of the high level abstract reasoning required, and literally by way of the training data)
That makes the AI completely pointless.
The GDPR mandates that consent is given affirmatively, with this kind of "oh we put it in the EULA nobody reads" being explicitly called out as non-compliant.
You're missing the point.
Yes. It's financially suboptimal to do this. But it's destroying society.
And while this requires policy change to fix, policy change is still "SUBOPTIMAL".