Incident report: Employee and customer account compromise
twilio.com
twilio.com
I do [edit: NOT] use my personal cell number for anything for security reasons, so even after their insistence that it was safe to use I refused and therefore I was unable to get past the first step of the signup process and went with another provider. After reading this I am feeling validated that I didn't cave.
Good for you. Your refusal and my refusal protect one another, refusing collectively is stronger than refusing alone.
My SMS spam is almost non-existent, compared to others.
But one thing, beyond my desire to not give out my number... it is pointless regardless.
When I worked in office, I had mobile access. At home, a bit rural, my access is nil via 4g/5g. I just have no access.
My mobile forwards after 6 rings to my voip, so that works well. But for SMS auth? Hello! I cannot do that!
I have been an ebay customer for 21, yes 21 years. I can no longer log in, as they now insist I enter a mobile number to continue.
Gee thanks ebay.
(No, SMS won't work via voip, they check numbers, even ported numbers)
21 years. Years of thousands, even >10k spent reliably per year.
Gone as a customer.
Calling paypal support, results in people literally unable to understand ... anything. They repeat a mantra off their screen without deviation. Many cuatomer support people I spoke to, were barely paying attention.
I really don't get it. SMS is barely secure to begin with.
They are willing to throw away accounts, just for pennies on tracking.
21 years.
It does say that short number support is not guaranteed, though.
Somehow, even though you say they ban them, I was able to submit it and get a verification message. Funny how that works, when someone just parrots the same thing without actually trying it out.
Proof: https://imgur.com/a/75yKkFl
It's annoying as hell because I would very much like to sell a thing or two and don't have many platform choices.
> The process to close the account may take up to 30 days from this notice. eBay will send a message to the email address registered on file, confirming that the account has been closed and, unless on hold, restricted, or suspended, that data associated with the account has been deleted.
I am curious if it is because they care slightly less if it's a CA VOIP number, if it came from a decent pool of numbers, or something else. Or if they will lock you out later and force you to contact the "risk assessment" team and use this as a datapoint.
Interestingly it's an old landline number that got ported over to voip.ms, so you'd expect that if it were trusted due to reputation, it'd also only be accepted as a landline. Yet the validation experience has no problem with it.
* Numbers are generally designated when they're created by regulatory agencies as "landline", "mobile", "VOIP", etc.
* You can provision yourself a mobile number in a provider like Twilio or a competitor if you don't like them, and it will have all the capabilities you want
* Thanks to technological advancements, you can do things like overlay mobile capabilities on non-mobile numbers, but only if you're running all of the traffic through a company that has that tech
* "VOIP" numbers are a vanishingly small corner case for these companies and are a pain in the ass to support. They are a tiny portion of the numbering space and lots of smaller Telco companies just won't complete calls or texts to/from VOIP numbers. Companies like Twilio rely on those smaller companies for last-mile completion or origination of calls.
TL;DR Provision yourself a "mobile" number using an API-enabled SaaS telco platform, which you can use exactly how you want, no actual mobile phone needed, with all the capabilities you want. The "VOIP" number will only continue to cause you more and more issues over time.
Realistically, was I being targeted? No. But it's a sad state of affairs that I have to even think about such things. I ended up looking up the company online and finding their contact email address to let them know about the mistake, so they could contact the person directly (which they did, and thanked me).
[1] "If you operate a pseudonymous Twitter account, we understand the risks an incident like this can introduce and deeply regret that this happened. To keep your identity as veiled as possible, we recommend not adding a publicly known phone number or email address to your Twitter account."
https://privacy.twitter.com/en/blog/2022/an-issue-affecting-...
This is my 2FA mule:
https://kozubik.com/items/2famule/
There are others like it, but this one is mine.
Whenever I hear things like this, I like to remind people that they should be most concerned about outcomes, not about what satisfies their indignance. I agree that this situation is ridiculous, but being mad about it isn't going to change the state of the world. If you're genuinely worried about giving out your primary phone number for 2FA or account verification purposes, then this is a solution for you. Being pissed at companies for leaking data is not a solution.
To my defense, I didn't argue that being pissed at some company would solve anything :)
See my comments elsewhere in thread, but a "VOIP" number is a ridiculously tiny corner case in the world of telco, hence lack of support.
In fact, twilio even started offering special numbers that are flagged and "vouched for" that should be treated as non-VOIP ... but they aren't.
If you really need it to work, it needs to be a number from a physical SIM card.
I guess it's nice to have the backup, though, in case the local internet connection goes down. The 500MB/mo plan bumps it to just $6/mo.
The /r/twitch subreddit is full of people who think you should absolutely give twitch your phone number for verification (fun fact: Twitch doesn’t even allow you to turn on 2FA until you give them your phone number). Even when a few days later, twitch got their data leaked, they’ll reaffirm that you are an idiot for caring.
And those are users, not even companies.
I don’t have a work number and this is an absurd reason to get one, so we just canceled the account.
> We're aware that some users would much rather prefer not to use SMS for 2FA usage […] this is a known issue
> Please be noted that as of right now, you can only access [TOTP] after submitting a valid phone number. If possible, we'd recommend you just provide a personal phone number as a workaround
I’d already borrowed a coworker’s phone once for the initial account verification, but requiring it multiple times crossed a line.
Hm, i wonder what the point is to warn them beforehand?
The number of people who regularly fell for it was worrisome. Falling for it meant auto-enrollment in a mandatory security awareness training. Failing to take the training would result in deactivation of the individual's network credentials.
I don't know if these campaigns are actually effective at changing people's behavior, but they certainly revealed how effective spearphishing is.
I've considered creating a browser plugin which marks the address bar red/green when you're on URLs the company (dis)trusts. Seems like this would be something that should exist already.
I’m also curious how much customer data an attacker should have been able to get into from outside. Is there no 2fa-auth vpn needed to get in? Or is there just lots of customer data hanging out in email/whatever?
A "zero trust" approach where nothing is trusted and you pass through an SSO with MFA like Okta is better in that Okta do a better job in security than "random network team running an unpatched firewall from vendor X" do on average. And each service has an auth layer and doesn't implicitly trust some IPs are good.
In any case, a phishing that captures login+password+mfa means game over in both scenarios.
This is one of the worst pieces of advice to be repeated ad deafenun.
I'm sure there are some VPNs that are implemented a poorly as you describe, but I'm not sure that's the common case like you seem to think.
Which you get from TLS. Strong authentication and authorization per-web-application using short-lived tokens through OAuth/OIDC is about a billion times more robust than any VPN network security.
90% of what VPNs are useful for is connecting you to a non-routable network.
TLS is another _layer_ of security, but it still reveals who you are talking to via SNI and DNS queries, and worse: how often you are talking to them.
It's another story that will drum up support for another integrated authentication system. I say this as someone who was recently targeted in an attack.
It's peek-a-boo, I get that, but damn is it frustrating.
But they'll get what they want. They'll get their secure system.
The only 2FA that protects against this is a hardware token, like a Yubikey, if used in U2F/FIDO2 mode. Twilio does use Yubikeys for some roles and access types, but not all, and presumably there was enough sensitive data that was only gated by TOTP 2FA.
I think in this day and age, if I were running a company infosec department, I would mandate U2F/FIDO2 for access to every system, and try to structure things so employees don't need to access company systems on their mobile device. (Yes, I know it's possible to do hardware 2FA on mobile, but I feel like it'd create a large IT support burden since it's not always so straightforward.) And if there are company services that aren't suitable for hardware 2FA, they need to be rare exceptions that are granted, and need to be firewalled off from the rest of the company.
I wonder how those factors impact the likelihood that an employee might be susceptible to phishing.
Even in the Techcrunch article, they have not specified anything. Are these customers other businesses or regular users using Twilio apps like Authy?
If the 'hackers' got too deep into the systems they might create a big mess. Authy 2fa tokens are backed up on Twilio servers (opt-in) unlike Aegis/andOTP. If you lose access to your offline 2fa tokens (phone stolen/lost), Authy can re-send these token from their servers. Users need to wait 24-48 hours for the whole recovery process to be over.
https://support.authy.com/hc/en-us/articles/115012672088-Res... https://authy.com/phones/reset/?proceed=true
But for this method, your phone number linked to Authy Account, email Id linked to Authy Account are needed. The Process is Started by old-school SMS based OTP sent to the linked number. You then have to Cancel it via a email sent to you, if you think some is doing is maliciously without your consent.
aka... There were a bunch of ways employees could access the customer data unaudited, so we don't have evidence of that. Rather than say "we don't believe you were impacted in this attack", we're using weasel words to set your mind at rest, when we really have no idea.
Shouldn’t having multiple domains for different use cases be considered a security risk going forward? Is it already considered a bad practice?
(Maybe they already have this, the article doesn’t say)
Very unfortunate incident, and rather scary if this is targeting multiple companies.
(Twilio also owns Authy for 2FA)
Twilio even offers that as a service (as do most bulk-SMS providers): https://support.twilio.com/hc/en-us/articles/223181348-Alpha...
In the US, alpha senders aren't allowed. Shortcode (5 or 6 digit numeric senders) spoofing is highly frowned upon to the point where you can't send from the same shortcode with multiple aggregators.
Other countries vary between anything goes, aggregator takes anything in the sender field, some high profile names are checked, or all sender names need to be run through the telecoms regulator and validated by the aggregator and cell carrier.
Of course, there's (usually?) no customer id transparency between the carrier and the aggregator, so if there's a misconfiguration at the aggregator, one customer may be able to send with a registered sender of another customer.
So, really, Twilio employees should be trained that the company will never send SMSes to employees for these sorts of purposes. But it only takes a few people to get fooled when their guard is down.
That's the difficult thing about defending against security-related attacks: the defender needs to be perfect 100% of the time, but the attacker only needs to get lucky once.
(Disclosure: I'm a former employee who has been getting these phishing texts, the most recent of which was yesterday... not that I have access to any systems anymore. I guess they are going off leaked employee lists that are at least 5 months out of date.)
0: https://twitter.com/SwiftOnSecurity/status/15535388679192616...
It seems like hardware-based 2FA with something like FIDO/U2F would have easily stopped this attack and generally make stealing credentials really difficult (it's still technically possible if you're able to spoof the target domain, but that's much harder than registering a fake domain).
Kind of weird that they only mention "security training" as a remedy, which is obviously never going to be 100% effective, instead of a more effective technical solution.
The real story is that Twilio, likely due to one of their largest products being Authy, still refuses to move away from SMS 2FA. They need to finally join the rest of the industry and move on to webauthn, internally and externally for their customers.
Exactly, that is absolutely not sophisticated - it's often as simple as "scraping LinkedIn".
The entire attack process includes testing the credentials as a necessary part of getting the 2FA code.
Agreed. A former colleague pointed out that company comms of this sort would never have exclamation marks in them. They don't look as amateurish as some of the poorly-spelled/poorly-punctuated/poor-grammar phishing attempts I've seen, but they don't look particularly legitimate to me either.
But remember that, in a company of over 8,000 people, many of them very non-technical (tech companies are staffed by people of all levels of technical proficiency), all it takes is one or two or three people to fall for it. And maybe they were tired, or had a beer or two in them, or something like that.
If this is in fact the case, I wonder if Twilio becomes even more liability for such customer damages. Note: additionally, Twilio uses it's own Authy for MFA.
Twilio has really stepped up KYC efforts over the past couple years. I don't think an anonymous attacker could send this volume of messages through the platform without getting noticed. And if they were using Twilio to send these messages, it would have been trivial to block them.
For everyone: If you have more than a handful of employees, one of them is guaranteed to be phished. If that can result in a meaningful compromise, work on your security until it doesn't. Then work on securing against attackers tricking that employee into downloading and running malware, because guess what, that's also going to happen.
I suspect most companies would never know that someone was sharing an employees login and downloading data...
Well if you want slightly more confidence, use my tool to catch thieves...[1] Turns out most thieves are rather tempted by some cryptocurrency that appears discarded and up for grabs.
Given the "sophistication" of some of these attacks I would be surprised if the people behind it would even recognize a crypto wallet file name, let alone know how to import it and actually steal the coins.
I remember seeing another service that provides usernames, emails or links to seed into your DB and will alert when the link is requested or spam starts coming into that. I think that will be more effective.
The usual caveats of data from surveys applies...
But the biggest problem is still the human factor aka "you can't patch stupidity".
Wow, what amazing technology could this be! Maybe it's one of those shitty websites or apps that collect your entire phone book and leave it on some Mongo NoSQL public database? Maybe its Twitter that pinkie-promised to not store phone numbers, then stored and lost them?
Is this another case of caller-ID spoofing? That needs to be illegal/impossible.
If you don't want to use that, for some reason, or if you want additional protection, other solutions include using dedicated machines (not personal cell phones and certainly not whatever devices the attacker was using) ideally with hardware-locked keys (TPMs etc.) to access the corporate network or at least to access sensitive systems like customer data, or giving people separate privileged accounts that they don't use for day-to-day access, or establishing a two-person rule for logging into sensitive systems (so two people with the same access need to get successfully phished at the same time), or setting up some real-time auditing of changes (e.g., a Slack channel gets notified when people manually log into prod).
Would it be stupid to force mTLS for employees?
Its pretty frustrating that there's a well known technical solution that prevents this type of attack from being feasible and companies like twilio simply have not prioritized it yet. Replacing SMS and Push based 2fa with webauthn is the best bang for your buck security upgrade available to companies right now.
EDIT a bit later:
> We have reemphasized our security training to ensure employees are on high alert for social engineering attacks, and have issued security advisories on the specific tactics being utilized by malicious actors since they first started to appear several weeks ago. We have also instituted additional mandatory awareness training on social engineering attacks in recent weeks. Separately, we are examining additional technical precautions as the investigation progresses.
Security training is ineffective against phishing attacks like this. Please stop wasting your money and your employees time; implement the technical solution that nullifies the attack.
Traffic that gets identified as marketing traffic as opposed to traffic that will involve a human responding (like a chat bot to schedule with your doctor's office) actually costs Twilio much more to send because the Telco carriers charge more because they have to put more into mitigating the risk of getting blocked and sued for spamming.
So the "good customers" are lower volume but higher margin, and generally more likely to grow at a sustainable pace and not disappear off the map, as the have a lot of code around using your APIs, not just "Send a kajillion messages". Those "marketing" customers will jump off your platform in a blink of an eye if someone offers them cheaper prices and less oversight on their spammy practices.
I wouldn’t say that. There’s a significant effect of training on the number of people that click. The big problem is that the attackers have to only get it right the one time.
I understand your frustration but you are missing the point.
2FA with a valid (non-VOIP) SIM card is not for your security. They tell you it is, and they dress up the process as if it is, but they are lying to you.
Twilio, et. al, have a brutal, unrelenting scam/spam problem and they have no solution for it. They have nothing. If they had a solution they would have deployed it long ago.
Instead, they do what they can: throw sand in the gears and slow down the bad actors with ridiculous, obviously absurd (put in any phone number we have never seen before to prove that it's you) mechanisms and hope the real customer base doesn't balk.
That's an interesting point I hadn't heard that before. Don't banks essentially do some form of this with KYC checks?
Would expect being smart about bringing in some components of that process would reduce spam while not introducing too much user friction.
You also have to realize that Twilio is not a Telco, they ride on top of big Telco company infrastructure and ultimately has to answer to big Telco, and big Telco is very risk averse and generally has to answer to regulators. So Twilio doesn't want spammers either because they get dinged, blocked, or marked as spam by the Telcos.
Ultimately there's probably more that the underlying Telco infrastructure could do more to support anti-spam measures but these companies are dinosaurs that move very slowly. It's easy to deride them from the position of enlightened SaaS company employees, but they're the ones in the capital-intensive business of maintaining worldwide networks of copper wires in the ground, physical switches, systems designed in the 1980s or the 1950s and complying with incredibly complex and outdated regulatory structures. It's the "legacy system" of a developer's nightmare to the 100th power.
Having a good flow for self-serve customers to sign up and get going is the cheapest way to get new business, and developers will not even look twice at you if it's hard to get going in a self-service manner on you platform.
However, it's exactly that low friction that gets spammers going easily.
So what do you do?
You try to come up with heuristics and even ML models to identify bad actors early. It's an ongoing constant battle, a game of whack-a-mole where the moles have basically zero cost overhead, no legal entity associated with them, and can just keep popping up over and over and over forever.
So you're staffing a fraud detection team of engineers, you have lawyers and finance people on this, you have customer service for customers who get caught as a false positive - it's all an extremely expensive pain in the ass.
I wrote you an email a year ago about how the Authy app is dumb and SMS "2fA" is NOT a second factor! To re-iterate: SMS is not authenticated, it can be spoofed (using your platform), there are no delivery guarantees, no read receipts, it's not encrypted... and most importantly: It defers security to Phone Carriers, who have the security posture of an air seal made of swiss cheese.
Once again: Dump the Authy App. Promote actual 2fA technology like U2F, now WebAuthn, hardware keys, OR _actual_ TOTP which cannot be reset using an sms. My 67 year old Mother, grandmother of 12, knows how to use hardware keys and TOTP. I'm embarrassed for you guys that you still can't figure it out.
Sincerely,
-The I told you so department
When you say tokens can be reset using SMS, which tokens are you referring to? Is this reset something specific to Twilio?
I can't imagine a scenario where SMS would enable the resetting of other 3rd party TOTP secrets, but I may not be understanding your comment.
#2: The authy app lets you recover your account using SMS. So anyone that wants to pull off a simjack attack on you can login to your Authy as you and obtain the keys to the kingdom.
The entire point of TOTP is the "Secret" is held locally in an oracle. If you break that constraint, you've broken the security of the protocol.
[1] https://authy.com/blog/how-the-authy-two-factor-backups-work...
I'm aware of the simjack risk, but that would require:
1. That I'm using cloud backups
2. That the attacker has also obtained my backup key
None of this seems fair to summarize as:
> Your security tokens can be reset using SMS.
I'm not claiming Authy is perfect, but it seems to use a reasonable approach for people who don't fall under the "high value target" category.
For U2F, there is no possibility for a user mistakenly approving one site's challenge on another site, if the challenge request is coming from (and the response would be sent to) https://badsite.com, then any challenge that's not for https://badsite.com would be automatically rejected by the browser even before asking the user anything. (This is the type that is usually implemented through a USB key.)
Is what you're responding to, and such an attack cannot work with them. The parent comment already clearly understands the flaws of Authy, you don't need to talk through it.
I'll try to explain the key difference between totp and webauthn style flows, as it relates to security here.
Conceptually, you can think of it as the hardware token (the yubikey or whatever) gets the site domain name the user is on from a trusted source (the browser), and then sends back a secret that is specific to that hardware device and domain. If they're on the real site, the token sends the right secret, but the attacker can't intercept it since it's sent directly between the local browser and usb device. If they're on a fake site, the secret will only work for that fake domain, not the real one, so the attacker can't forward it and have it work.
Many large tech companies use hardware tokens of this sort now, and for a company of twilio's size it's quite reasonable to expect that they provide such a token to employees and mandate using it when accessing customer data.
Either way, not only there is poor security at Twilio, but I would expect that they will be fined in the multi-millions of dollars for this breach.
By whom? Most breaches do not involve fines of any sort.