Mudge's redacted Twitter whistleblower disclosure [pdf]
s3.documentcloud.org
s3.documentcloud.org
This leads me to believe that this is a document for public consumption adopting the aesthetics of a letter of concern sent to regulators, and that this document is being submitted to the court of public opinion. I don't doubt that Twitter executives are borderline fraudulent and may have crossed the line into outright fraud - I'd be unsurprised to learn that about any group of executives at any large company. But this document has more the feeling of propaganda than a serious appeal to regulators.
https://www.gwern.net/docs/darknet-markets/2013-ulbricht-com...
The way things are presented in sections with numbered subsections, the way some things in the beginning appear to be redacted, are clearly meant to convey the authority of an official document submitted to a legal body.
I'm sure this doesn't violate any sort of law and that you are permitted to write a letter like this and submit it to regulators. What I'm suggesting is that the stylistic choices they make are indicative of the audience they are writing for, and that it is different from their stated intentions. And that suggests to me that this document has an agenda and is willing to accomplish it through deceptive means (those being, lying about the purpose and audience of the document to enhance it's perceived authority by giving the general public the impression they are the fly on the wall of an official proceeding rather than reading a document for their own consumption that may have no bearing on the outcome of this dispute), which is my personal standard for calling something propaganda.
This is Elon’s attempt to burn US taxpayer money helping launder him out of having to pay ~$10b to walk away from Twitter. It sounds like Mudge has his own litigation regarding severance. So it was a no-brained to join Musk’s mercenaries.
This issue is not worth taxpayer time. You want less regulation? Then stop crying wolf to Congress, and more importantly, stop just doing dumb stuff like buying a company you can’t actually run.
This is HN and it only serves as a court of opinions, and Elon isn't popular right now, but let's keep ourselves intellectually honest.
Twitter has had several security issues, including Saudi spies[1], a large-scale hack of several celebrities[2], and even recent ones[3]. Twitter is one of the largest tech companies and a very important town square.
Mudge was hired to do a job by Dorsey, and the newly appointed execs made it very clear that they don't give a shit about security, when the lack of safety measures has been demonstrated to be dangerous to the public.
[1] https://techcrunch.com/2022/08/09/twitter-spy-convicted-saud... [2] https://www.thewrap.com/teen-mastermind-behind-elon-musk-joe... [3] https://www.hackread.com/twitter-data-breach-accounts-sold-h...
> Twitter is one of the largest tech companies and a very important town square.
94th on the chart [1] - as for a town square I respectfully disagree, its a website - 12th according to rankings. Just nudges infront of the other famous town square, pornhub and a little further behind, linkedin. Half the 'traffic' of reddit, and 1/3 of apple somehow, not to mention the giant in the room, facebook. Surely if there is a town square that is it ... i doubt many would even agree with that.
> the newly appointed execs made it very clear that they don't give a shit about security
Allegedly
> when the lack of safety measures has been demonstrated to be dangerous to the public.
I am yet to see said demonstration. Are we worried that XYZ blue checkmark twitter users public tweets are not encrypted at rest on their servers? Or that the entire corpus of the library of twitter might be lost in an catastrophic event?
[1] https://companiesmarketcap.com/tech/largest-tech-companies-b...
[2] https://www.semrush.com/blog/most-visited-websites/ (oh how i miss alexa)
I'll stay tuned to see if it happens, like everyone else.
The use of the Mudge is certainly the weakest of the indicators I've presented; what do you think of the strongest of them, the unusual use of emphasis? What do you think of the three of them taken together? If you were the author of this document and you just found out that the public would never see it, only officials at the SEC et al - would you feel like this document was likely to be effective for that audience? Is there truly nothing about it you'd change in that circumstance?
I do think there is a good reason for bodies which are meant to dispassionately find facts and enforce regulations to prefer plain language and straightforward argumentation. But even if that weren't the case, it really wouldn't reflect on my argument. Additionally, everything the document alleges could be true, and the allegations worthy of discussion - I haven't argued that they aren't true or that we shouldn't be discussing them, I have no way of knowing if they are true and am ambivalent about whether we should give them oxygen - and that would not reflect on my argument.
I go into slightly more detail on this point here: https://news.ycombinator.com/item?id=32603317
No, I don't believe it would read the same without emphasis. Especially since the emphasis is often on superlatives rather than the substance of certain statements. I'm not sure what to say other than, "emphasis is a tool of graphic design that does, in fact, function." It guides the eye and enables the reader to skim, and if you took away the emphasis, this document would be less effective, at least if the goals of the document are what I have argued.
Your questions seem reasonable except the fact they're about him!
Do you understand his (very populist) history?!
It's pretty apparent that one of the intentions of this document is to get Musk out of hot water - totally separate from "protecting the internet" and not a particularly populist goal.
I'd speculate this represents a collaboration between Musk and Mudge's legal teams, probably in exchange for Musk funding litigation from Mudge regarding wrongful termination. That's all speculation, it's just what makes the most sense to me.
I'd write like there's no way this doesn't leak to the public eventually. It's Twitter we're talking about. Juicy gossip is the point of the platform so of course it's going to get spread. I don't know that it undercuts its message though.
Seems like he wrote it with multiple audiences in mind and I don't see how that detracts from his claims or message.
I would claim they did not write for more than one audience. I'd refer you to my other comments in this thread for that argument.
- Lack of development and testing environments; engineers build, deploy, and test code directly on the production environment
- >50% of employees having access to the live production environment and sensitive user data (getting _worse_ over time)
- Lack of logging of what people did with their production environment access
- 30% of employees' systems had disabled software updates
- Twitter "has never held proper licenses to the data sets and/or software" they used for some ML models
- "The majority of the systems in the data centers were running out of date software no longer supported by vendors"
- Misleading the board (e.g. trumpeting "we have endpoint monitoring software on 92% of employee systems!" but neglecting to mention that endpoint monitoring software reported 30% of employees' systems had disabled software updates)
- Misleading the FTC (e.g. implying that data was deleted when users closed their accounts, when in fact it was not)
The other sections are much more interesting.
He's basically saying "the execs didn't prioritize what I thought was important! It's illegal!"
There being an entire section titled "Lying about Bots to Elon Musk" makes this entire document seem flimsy at best.
It's entire possible to make whistleblower compaints about security issues at Twitter without trying to testify to an impending lawsuit.
I thought the whistle blowing was about state actors being granted 100% access to user data. Instead it dedicates the first 20 pages whinging about a non issue. I lost any motivation to read the docs any further.
He also complains about data center ops quality at Twitter. While Mudge pretended this is extremely unusual, I have never worked at any fortune 500 that would meet his criteria of fault tolerance, and I have worked at several. With original architects disappearing and employee churn this is impossible. And why does he think this needs to be whistle blown? He sounds really immature.
Is there any thing of interest beyond the elon musk drama?
--- edit: one thing I regret is not changing the cnn.com URL to the original WaPo reports a few days ago, since they were obviously much better - so I'll add them here:
https://www.washingtonpost.com/technology/interactive/2022/t...
https://www.washingtonpost.com/technology/2022/08/23/peiter-...
---
Twitter CEO Parag Agrawal on whistleblower story - https://news.ycombinator.com/item?id=32565019 - Aug 2022 (82 comments)
Twitter’s former security chief says company lied about bots and safety - https://news.ycombinator.com/item?id=32564630 - Aug 2022 (2 comments)
Ex-Twitter exec blows the whistle, alleging reckless cybersecurity policies - https://news.ycombinator.com/item?id=32562815 - Aug 2022 (597 comments)
Not so recent, but related:
Twitter shakes up its security team - https://news.ycombinator.com/item?id=30026171 - Jan 2022 (110 comments)
Twitter names famed hacker 'Mudge' as head of security - https://news.ycombinator.com/item?id=25115754 - Nov 2020 (172 comments)
https://news.ycombinator.com/item?id=30026171 ("Twitter shakes up its security team")
Further, all dang is doing here is referencing other HN submissions that relate to the same topic. Are you claiming that he's picking and chosing related links based on some form of bias?
Almost makes it look like the same PR people work for both Twitter and FB. Or, these PR tactics are google-able and everyone rips the same template now.
The best way to avoid this problem is to have a secondary datacenter that is hot, and to never let both fail. Of course, this is what twitter is doing. Designing a DR plan where you can lose multiple primary/secondary datacenters is hard enough that basically no one does it.
This is something that really doesn't belong in the whistleblower complaints because it wasn't under his responsibilities, twitter wasn't lying to anyone about having properly implemented DR plans, and twitter's DR plan isn't out of the ordinary across the industry.
What sounds immature, unprofessional, and out of depth is Agrawal's mass e-mail tarnishing Zatko's reputation, a tactic that your post seems to repeat from a different angle.
I'd rather have the theory of general relativity written in crayon, than the most beautiful calligraphic illustration of nonsense. Wouldn't you?
I should have used the word "seem" rather than "sound" - "sound" does make it seem like I'm criticizing tone.
With all due respect, no you aren't. There isn't any criticism of the actual content in your post. It's all critiquing the tone - what style he used when writing, who you think he thinks his audience is, how informal his phrasing is. None of that has ANYTHING to do with content, it's all about form.
My comment is based on the following:
Much of the first 15-20 pages are complaints about the company optimizing and incentivizing what he believes to be the wrong metric. There is nothing illegal about it. Twitter are clear about it, shareholders know what they are trying to do. He's naive if he thinks it's a legal issue. It's probably not even a moral or business issue - if raw bot count becomes a user experience problem, it will cause mDAU to drop.
He appears unprofessional in his constant wanting to go to the board - he was a security lead, it's not his place to go to the board. He reported to a guy who reported to a guy who reported to the board.
If the smoking gun is the bot section, then there is nothing here at all.
For example, Mudge claiming he was ordered not to present a report to the board. He might not be happy about that, but that's perfectly fine, it's not down to him to chose what the CEO decides should be presented to the board, you can have perfectly reasonable disagreements about what's appropriate and at the end of the day the call is down to the CEO. The fact that there are other cases where board members intervene to tell Mudge this reinforces that.
Or the claim that the CEO instructed Mudge to send the board documents they both knew were misleading. This is an explosive claim, but it seems highly unlikely that Mudge can prove Agrawal knew the documents to be false, and misleading is impossible to know, because he can't know the context they were presented in. I think it's just highly unlikely that Twitter's CEO is so incompetent that he's just moustache twirling and lying to everyone, it seems highly likely this will come down to Agrawal having a different opinion or interpretation of the facts.
What really undermines the claim is when we get to this section:
>Agrawal’s tweet was a lie. In fact, Agrawal knows very well that Twitter executives are not incentivized to accurately “detect” or report total spam bots on the platform.
Mudge is massively over-reaching here. At best, an argument can be made that at some point there are some perverse incentives where allowing spam bots could inflate numbers to make the company look successful. But even if that argument were convincing, which it isn't, Agrawal clearly doesn't believe it. It's trivial to make Agrawal's argument here.
That's why this looks like big claims, but unsupported claims. Because where it's clear that people can reasonably believe what they say but disagree with each other, Mudge claims one side must be lying. What could easily presented as openness, honesty and transparency about the challenges the company faces("Mudge asked the Head of Site... what the underlying spam bot numbers were. Their response was "We don't really know.”) - Mudge basslessly claims this is essentially proves they were acting in bad faith.
This all looks designed to be explosive on first sight, but not actually correct in the detail.
Isn't it? Does the CEO get to carte blanche decide to hide security holes from the board?
So it's up to the CEO to make the judgement on when something rises to the importance of needing to notify the board or needs input from the board. The board is there for strategy, not for operations. So as long as there's nothing relevant to the high level strategy, there's no real reason to tell them about individual operational issues.
An obvious example here, is that Mudge cites that Agrawal tells the board that they have end point management, but Mudge knows the end point management shows lots of devices aren't secure. The end points being insecure isn't a problem to bring to the board, as long as you have a strategy to rectify it- which presumably they do, since that's why they've instituted end point management in the first place. And remember, the board knows that they've got security issues and they're addressing them, because the board approved them hiring Mudge as head of security in the first place. So the question is why the Head of Security thinks he needs to go and advise the board of all of the operational issues he's tackling. They know there are issues, that's why he's employed.
It is clear to me that Argrawal is trying to obfuscate and confuse. Every time he is asked about the total number of bots on Twitter, Argrawal subtlety changes the subject to mDAU. He knows this is not the same as total users, but he chooses to change the subject and pretend he's answering the question. This is very shady in my book, and his attempt at subterfuge and lack of transparency is very troubling. I've seen other CEOs do similar things when they don't want to talk about something potentially damaging. The fact that Argrawal artlessly tried to smear Mudge earlier in the week does not make me trust that he has good and forthright intentions. Argawal seems very shady, and I wonder how you can trust him?
He isn't obliged to engage on every question in the exact way the questioner asks.
And this isn't just some arbitrary thing that Agrawal has made up either, there's one very specific reason why you would care about mDAUs not bots. mDAUs are the number of people you can advertise to, and Twitter's revenue is advertising. It's like saying Twitter has an army of 10,000 cats in it's basement. It might be true, but it's not relevant to the value of the business.
There's some places where they show that twitter's public statements, after a security incident, around their security posture were misleading. That's potentially illegal, but even that could be a grey area.
There's some places where they show twitter didn't comply with creating and enforcing an SLDC, but if you've ever worked on an SLDC, you know it's a joke that's there for compliance checkbox reasons. Also, this is certainly a grey area because most likely to actually comply with this, you only need to have a document generated, training for how it should be used, and very basic processes in your git workflows. I'd be surprised if they hadn't done enough to legally comply with this.
The statements around "employees with production access" are vague. What _kind_ of production access do they have? If you're an engineer, you have production access, because your code can be deployed there. If you're in ops/support, you have production access, because you need to access user data. Hard to properly judge the seriousness of this without knowing more details.
For me, the lack of audit access seems to be the biggest legal issue, especially if they were lying to regulators about their progress.
It's not that the accusations aren't bad. They paint a picture of an org that has serious security issues, seriously management and organizational health/culture issues, flailing executives, and an incompetent board. But, it really seems like most of it simply isn't illegal.
This is simply a complaint making statements without any real form of evidence. As you say in your final statement, this is just painting a picture of an organization. It's definitely not fair to take everything stated here at face value with all of the context which is clearly missing.
It being a statement in a complaint document does not in any way mean that it is true (or has been "shown" to be).
If you've had an attack, you disclose, and what you discuss in terms of the protections you had in place, and the actions you're going to take afterwards weren't completely accurate, that's more of a grey area.
Sounds very similar to how the Saudi Arabian government spied on Twitter users using a plant but in this case knowingly supported by Twitter executives.
At a minimum it's worth spending brain cycles considering the impact on "national security".
Arguing over the blame and legal implications could be interesting in the context of dispassionate technical legal analysis, but most of that discourse seems little more than a couch for people to signal their biases for the parties involved.
Did Twitter really think that continuous denial, lies and more PR deception was going to be that easy to get away with?
Not this time.
If Elon hadn't just happened to tender an offer, want to back out, and put them under a court spotlight ... yeah, they probably did.
And, to be fair, unless someone goes to jail, then ... yeah, they'll just get away with it as the cost of doing business.