Hacker News Reader

Comment by nkrisc | Hacker News Reader

The tool worked correctly and as intended, but due to a bug it did not work correctly nor as intended.

To be fair, that quote in the original article could have more context. By "The tool" they meant "AI-assisted support tool"[1]; perhaps they meant that the issue was not an AI hallucination inherent of the tool, but a fixable bug.

[1]: https://www.documentcloud.org/documents/28202858-meta-ai-ag-...

In that case, the statement is so meaningless as to be useless. Why should we care how Meta splits up their microservices? The tool still failed. They just want to redefine the "tool" as something else, anything else, to avoid having to admit something negative about their precious AI.

> The LLM correctly generated tokens according to user input, however due to a bug in a separate code path, the system did not properly verify the email address

> Nginx correctly handled the user requests according to the HTTP standard, however due to a bug in a separate code path, the system did not properly verify the email address

I mean, I think many of us are curious and enjoy hearing more details about how and where bugs like this occur. What's wrong with that?

I'd love to read a proper technical post-mortem, but this obviously isn't it. It's a carefully-worded statement from a lawyer meant to minimize liability and reputational damage to the company.

There is nothing wrong with that, and nobody is saying there is. In fact, it is exactly what is being requested here!

It seems to me like they're saying the agent made the tool call they expected, but the harness didn't reject it like they expected it to.

But it sounds like it's not even a harness issue if they have a process where they send a reset email to an address that isn't associated with the account.

This isn't (just) a validation issue, and shouldn't be at the harness level.

Sounds like they are saying the agent did not malfunction, and this vuln could have been triggered by a human support agent too.

It probably could have been, but how likely is that compared to with the AI agent? I'd assume (and I'm ready to look like an idiot if I'm wrong) that the humans are trained to send the verification code to the email address on file, rather than any address the client asks them to. I'd certainly assume most of them are more afraid of the consequences than the AI is.

For sure. Social engineering attacks on human support staff are common and well known, but the skill floor is non-trivial; you need to actually be able to convince a human of your ruse.

Having a support agent likely made it easier to enumerate the vuln, and certainly made it easier to scale out exploitation once it was discovered.

I think they’re blaming a tool function so as not to admit the overall agent process was shit.

But it’s irrelevant, outside of PR. We know at least THREE bad components to this process and they were constituent parts.

Kind of interesting that LLMs are basically being sold as having “human-like” reasoning capabilities, but in this case when “obamawhitehouse” asked to have it’s password reset sent to bob12345667@gmail.com the LLM didn’t question it and just triggered the process that happened to have a bug.

Humans support agents certainly fall prey to social engineering all the time, but I can’t think of a case where it was done on this scale so easily.

I get the joke, but it's a relevant nuance that the new code, the chatbot, did not have 'the bug'. I still think that the mistake and head that should roll should be the one that published the chatbot.

But it's important to acknowledge that there was a 'bug' in an underlying tool and not in the chatbot, and still PIP/fire those responsible for publishing the chatbot and exposed an otherwise internal tool to the public, and not those that introduced the 'bug' to an internal tool.

Why should the chatbot team necessarily take the blame? For all we know, they could have got approval from the tool team to make it public, and passed additional security review for making it public.

Also, why fire anyone after a single mistake?

I did mention PIP/fire, but to be fair, this looks like the worst security issue in the history of Meta, a company known for an almost impeccable cybersecurity clean sheet.

So yeah, firing somebody or a group of people is on the table. Especially when like 10% of the company was fired last week for unrelated reasons. If you are gonna do it, fire the people who slash the value of your company by billions of dollars.

How not to do blameless postmortem lmao. Non of the the engineers involved in this incident had anything to do with the company-wide layoff. I'm deeply sorry if the layoff affected you. But blame firing/piping more engineers for an incident should NOT be on the table. The negative sentiment towards meta engineers on this post is just wild.

>But blame firing/piping more engineers for an incident should NOT be on the table.

There has to be a level of fuck up where a resignation is appropriate, maybe this doesn't meet your bar, but surely you recognize that there exists a limit of incompetence that proves that one is not up to the demands for the job.

I used to be on your camp, blameless postmortems, the truth is more important than assigning blame and in all likelihood it's a systemic problem. But with time I realized two things, 1 there's actually incompetent people, 2 if you wrongly get blamed and you don't blame someone else, then it's your head that rolls, hate the game not the player, you have to assign blame to someone else if you are accused.

Reply on news.ycombinator.com