Hacker News Reader

Comment by dpweb | Hacker News Reader

No disrespect to those challenged with protecting such a huge target, but why do admin tools even have these capabilities? I could see needing to disable a user account or change some attributes, but why would an admin ever need to tweet from it? There shouldn't be tools with God privileges even for admins. Not surprising human error was involved in a breach this huge. So, how many people had access to this tool? Is there a killswitch for the tool itself available to very few, really very few, persons? edit: I dont know if the tool can tweet but surprised 2FA can be stripped without a human being confirming (ie.. the acct owner's social media person), especially for famous people.

View on HN

The admin tool was used to change the email on an account, then the attacker reset the password and got full access to the account. Apparently having 2FA enabled did not stop this attack (admin tool probably had the power to strip 2FA from accounts).

So while the tool did not directly have the ability to tweet, it effectively did.

I feel like the power to reset emails and remove 2fa should be only held by a very small subset of customer support, with proper training.

That seems like it was the case, but the attackers got access to lower privileged accounts and used them to find who had that access so they could target them.

The key being "proper training". Those few god-level admins should be drilled enough to defeat a phone-phishing campaign. In fact, they should probably have custom procedures to look after their own credentials.

I work in customer service supervising entry level employees. The amount of power they have at any given time is astounding and it's by sheer ignorance or benevolence that more isn't embezzled en masse or this information isn't used for personal gain. My entire team of newly trained staff have access to your bank account information, where and when your payment was posted by IP, and we can strip 2fa or mobile numbers at whim. This coupled with inexperienced agents often leaves multiple accounts compromised. Having a select few engineers who don't work weekends always helps. We don't train the agents to tell them that they could potentially ruin customers' weeks by pushing the wrong button and it happens way too often to be standard, but as long as investors are happy and banks are good to reverse charges with no penalties here we are. Tech companies are good to throw caution to the wind.

I've coded/supported a number of admin tools for a number of large companies (banks, telcos, etc) so I'll take a stab at "why" First, god mode can be implemented cheap and fast. God mode also makes day to day support easy as admins/support staff can do just about everything fast and easy - so they typically love it and will often fight to keep it. More than once I have had managers turn down proposals to tighten up security because of the cost and false beliefs that because the tool is behind a firewall, better security is not needed. Taking a schedule and/or budget hit to implement tighter security is not going to get them a promotion or bonus. Sad but true. In the vast majority of cases I've seen, paying for security improvements becomes incentivized for managers after their company gets burned by a breach.

Also, admin tools are often "afterthoughts", there is usually a motley collection of them, and often considered as an expense/cost to be minimized and not a revenue generating asset that gets more budget and attention.

My uninformed guess is that there isn't a "tweet as this user" button (because obviously there's no legitimate use case for that), but there is a "change this user's email address" button (because you might need to do that in order to help someone who's locked out of their account), and if you can do that you can take over someone's account. Obviously something like this would be detected quickly, which makes it less scary in some ways than a "tweet as this user" button, but of course this particular attack did not seek to evade detection once it was launched.

Of course, some of the targeted users presumably had 2FA enabled. How to do account recovery with 2FA in a consumer context is a complicated problem and I'm not aware of any good answers, but there's certainly an argument that the protections in place there weren't adequate and I wouldn't be surprised to see them changing soon.

I would also hope that rank-and-file support staff can't change users' email addresses, and the attackers had to spear-phish one of a smallish number of people whom more complicated account-recovery cases are escalated to. But who knows if that's how it works.

> How to do account recovery with 2FA in a consumer context is a complicated problem and I'm not aware of any good answers

I've always wondered why there isn't more use of time delays for this sort of thing.

If there was a notification e-mail and a 7-day wait, that would offer a fair chance for the real account holder to cancel the change. Not 100% - the user might be on holiday - but it would catch a lot, and hence decrease attackers' motivation. And while a 7-day wait is inconvenient, for services like Twitter and Steam losing access for a week isn't the end of the world.

"Tweet As User" is the basis for almost all social media management tools and Twitter still doesn't have very fine-grained permissions grants. https://developer.twitter.com/en/docs/basics/apps/guides/app...

We had a running gag in our social media startup of tweeting "poop" from people who left their phones/computers unlocked ... someone did it to an employee that was logged in as a customer (corporate brand) context and that was the end of that 'joke'.

Did they tweet directly from the admin tools? My impression was that they used the admin tools to reset the password and then take over the account, ultimately tweeting like any normal user would do.

Do we know for sure that the admin tools can do all this? My understanding was that the tools enabled password resets, which allowed the attackers to tweet from the accounts themselves.

> For 45 of those accounts, the attackers were able to initiate a password reset, login to the account, and send Tweets.

It is my understanding that they used the tools to update the email of the account, then reset the password to log into and make a new password such they could log in and tweet. Do you have any source that says that they could use the support tools to tweet directly from?

Internal admin tools grow over time. They don't spring fully-formed into the modern company with all the correct access controls and auditing they ought to require at that point. They carry a lot of cruft from things that were needed early on and aren't needed anymore.

Furthermore, they're a classic cost center, not a lot of love or budget goes into reducing their tech debt, or bulwarking them up against a sophisticated adversary. Red teaming yourself full time is expensive and not profitable. What's the worst that happens from a breach like that? Well, Equifax is still going strong!

I recall being party to an amusing conversation at a major network services provider at a team meeting for people with access to such tools, to the effect of:

- Alright, we're modifying <internal tool A> to lock down access to accounts related to <major political figure>. You will no longer be able to use <internal tool A> on <accounts>, only select supervisors will have that access.

%%% ah, okay, that makes sense

# uh, hey, regarding <internal tool B>, which allows us to look up <thing that would provide equivalent access to internal tool A>? does that still work the same?

- Uh, yeah, it does.

# okay?

%%% ... silence ...

- Alright, next item!

To the best of my knowledge, that was never addressed. <internal tool A> has audit logs. <internal tool B> doesn't.

Admin tools having the capability to change email and 2FA settings is a necessity, but Twitter clearly needs to greatly increase the security.

I don't know what all Twitter uses, but I know that many companies have various methods of authentication depending on how much damage can be done:

- Logging on using a username/password and 2FA is enough for some activities.

- More sensitive operations have to be done on hardware that has a certificate installed and backed by something like Windows Hello.

- Even more sensitive operations require a JIT account and a certificate stored on a separate hardware key such as a yubikey.

- Very sensitive work gets done on a secure device that is very locked down and can detect changes to the hardware that may suggest tampering.

- Some stuff simply isn't allowed to be done remotely, even with the above restrictions.

Obviously not every company needs such a complex setup, but for someone as high profile as Twitter, you'd expect more thought to be put into this.

You have a need for a tool allows you to see the UI “as the user does” so you can respond to support requests and maybe someone thinks it’s easier to just copy the cookie.

This isn’t the right way to do it, but given they work at Twitter I could imagine this isn’t the first big mistake they’ve made.

This sounds like speculation. Is there evidence that Twitter has a tool that allows employees to see the UI as the user does?

it's not copying the cookie, but there are absolutely third party UIs via a user's API key, it's not a huge leap of faith to assume Twitter has similar internally.

A tool that allows an employee to access a user's API key? That sounds like a bad idea, especially if that tool is accessible to support personnel.

The question was interpreted as “why do admin tools have these features” on account of that was in the first sentence, and not, as you may have imagined, a request for a twitter employee to explain all of their tools, or a request for a twitter employee to justify creating these tools.

I doubt the admin tweeted from the tool. The admin changed the email on the account, then did a password reset, then logged in as the account then tweeted.

They probably had permissions to change/reset the access credentials which can be used to gain access as a the user.

Reply on news.ycombinator.com