Alameda lost tens of millions because of a fat fingering mistake
adityabaradwaj.com
adityabaradwaj.com
Once, when I was much younger, I had a side gig dealing poker at an underground club in NYC. One morning at the end of 10 hours dealing, I accidentally exposed a burn card which turned out to be something one of the players was representing (bluffing), with about $10k on the table. It was an honest mistake - literally a fat finger mistake. I was tired and my dexterity slipped. The player jumped up, upset the chips and started shouting that I was cheating and in league with his opponent.
The manager came over to calm things down. He was the young son of a mob boss, whose portfolio encompassed a variety of entertainment franchises. In order to calm the player, he pulled me off the table, told me he believed it was an honest mistake, and then threatened my life if it ever happen again. I quit then and there. Only after I had quit and refused to come back do I think he actually knew for certain that I wasn't pulling a scam.
[edit] I'm really enjoying that people are enjoying this post, but I want to clarify what I was getting at: The original article assumes that SBF et. al. concluded by some internal mechanism that this was an "honest mistake", and that this is sufficient. That assumption belies the way criminal organizations actually work, which is to say, they wouldn't assume it was an honest mistake, but they'd happily say it was one and let off a patsy if they had orchestrated it. I kind of hope I can tilt the conversation toward this aspect of the OP's revelation?
"Build what makes you happy..." is a darn good philosophy, btw.
Let's just say your money isn't going to anything good if you play in one of those places.
The Manhattan one was poker only. Steel door with a camera, get buzzed into a second door where you talk to somebody and they only buzz you up if they recognize you.
The poker room had two nice tables and a little bar where you could buy drinks. TVs playing sports. Seemed to be Asian-run.
[edit] There was also one night I showed up to work only a few hours after the cops had raided the poker room, and they were just getting back up and running.
Thanks! I wish I'd say it's my daily driver, but I still work for money.
Seriously though, I think making clients happy still makes me happy, if I like what they're doing and I get to impress them a bit. I guess it would be more fun to be able to live only on passion projects, but I know I'm lucky even to be able to keep the hope alive and keep creating.
There are dealers who are true shuffle mechanics and either know for sure or have a very good idea of who has what. I'm not that skilled.
I re-watched the movie "Casino" recently, and there's a scene where the Joe Pesci character cans a floor man for not stopping a slot machine run after the first or second time someone hits a jackpot in a night. Paraphrasing the movie: "Either he was in on or he's a fucking moron, either way we can't have him here". To bring it back to the original topic, I think in the gambling world, the default assumption is that there's a scam and the employee is guilty until proven innocent. I wonder why that wouldn't apply to a shady firm like Alameda, too, unless management was "in on it". Come to think of it, I guess that's the point of my story, which I didn't realize when I wrote it.
Robert de Niro. Joe Pesci plays a thug.
To answer your real question - my guess is that this was a place that had a history of moving fast and breaking things. But probably more importantly - there was no way for that person to benefit from the mistake. The BTC price recovered automatically through the quick arbitrage market. The only way they could have benefited would have been if they were the counter party in the sale, which is highly unlikely.
Now, why FTX was a place that operated like this is another question altogether.
There is nothing that says the BTC “mistake” wasn’t a fraud. But for it to have worked (and you’d only have one shot), the setup would have needed to be clear to outside observers.
The other argument for it being a mistake is that if you really wanted to have exploited Alameda/FTX (not sure which, or if there was a distinction), the person could have tried to take more. The amount is small enough to be a “mistake”, but not large enough to make the higher ups question it. Maybe the gains would have been enough for a corrupt party, but something makes me doubt it.
Meanwhile, I just find it hard to believe that the accepted narrative - backed up by this story - is that SBF lost $10M in this little fuckup and pardoned the employee who made the error. Seriously? This article? How did he know the employee was innocent, how was that proven to be a mistake, and what happened to the employee? Seems even more plausible given this paper-thin explanation that the flash crash was deliberately triggered to move some money off book. The fat fingered employee, if they exist, gets the blame and is nobly pardoned.
I certainly would hope the FBI is putting together a list of all the unrelated counterparties who profited during that event, just to rule out related ones.
I've lost $50M for my employer (a large, reputable, very by-the-book tech company). I've also gained $100M for my employer, on a project that was canceled because it didn't make enough money. (I half-jokingly asked "Well would you spin it out, I'd love to have $100M?", but it was too tied to corporate infrastructure.) My current project is on track to lose ~$10M/year for my employer, but it's considered strategically important, and so we've run it up the chain and every indication is that it's going to launch anyway.
It just doesn't matter when you make billions. It's like how you stop clipping coupons when you start making a six-figure salary, because the time and attention needed to sweat $0.50 isn't worth it when your paycheck is $10K/month.
SBF may have been "worth billions," but he sure didn't have enough escape cash on hand when he needed it.
Your division, project managers, VPs may throw around 9-figures worth of investment capital on loss leaders, party on the Riviera and give everyone Christmas bonuses, but all that means squat to someone like SBF once he's under indictment.
A cold hard $10 million in pocket, off the books, earned illegally, that can't be tracked back is worth a lot more than controlling a billion dollar budget or even getting a $100 million paycheck.
The very fact that it's considered a rounding error would work to the benefit of whoever had set up the scam. The real scam is setting up a corporation where you can pass off a rounding error large enough to let you escape federal custody, and that's where he fell short.
I'm pointing out why it would have been a worthwhile scam for him from my own experience of launching the first serious Bitcoin casino and coming to the rational, purely cost-based determination that there was no way to make enough money from opening it to Americans to let me avoid the likely consequences of that if the feds decided to treat Bitcoin as currency and go for me, which I now know they would have. I determined I could probably pull $10M in a year, but it wouldn't be nearly enough. I'd need to have at least $50M to have a chance. But this is the thought process you can probably project upon SBF running a flash crash, not the "too small to worry about" idea. $10M of misplaced money is right in the ballpark for someone a little stupid who's considering running for the islands.
It's a bit rarer to have limit orders at say 1/4 the current price but not extremely low, but they still happen. Enough to have plausible deniability at least. Check the order books; my recollection is that this was still O(hundreds) of people.
An escalation (though obviously not to violence) makes perfect sense for a second error of that type, however. It's the kind of mistake that seems innocent enough but if it's deployed in a pattern it can obviously assist the bluffee.
Fool me once, fool me twice, etc. It's not uncommon for dealers to be in on scams, obviously.
The dealer sees the tragedy coming down the pike, and as a last ditch effort he spoils the whole game to prevent an even larger loss from occurring after the river card.
For example, on this particular hand, maybe the cards were preloaded into the deal to actually harm Sam. This was done deliberately because we need to keep some volatility across hands to make the cheating more subtle. So, on this hand, Sam is supposed to bet a little bit early, then fold. He loses some of his money, but the whole game seems on the up-and-up.
Except, oh no!, Sam appears to think this is one of the planned winning rounds, so he just keeps betting! Sam has made an error, and may lose big. I (the dealer) am going to suck it up and pretend to make a mistake. I'll get yelled at in public, but the bosses will thank me later for rescuing Sam.
Let's say you have a seriously strong hand, and there's only one way you could see that you could be beat. The opponent may have that hand now, or they may be hanging around in an attempt to upgrade to that hand while bluffing in the meantime.
In some cases, the above scenario is obvious. Everyone at the table knows well enough what each player is attempting to project. Every player knows which cards to look for to be revealed. It can be devastating to the bluffing player for that card to be accidentally exposed as a burner card.
Given there was a burner card exposed, then there was at least one more card to be added to the community cards. At this point, the bluffing player would be in a tough spot. The player with the stronger hand would likely be raising / re-raising to call the bluff and / or get more chips into the pot.
Continuing the game after the exposed card would have been pointless. You also feel stupid getting exposed on a bluff, when you otherwise might not have had to show your cards. You're probably going to be very pissed.
Edit: The dealer would have had no way of knowing what the burner card was, but the dealer could have known what card the bluffing player was looking for. However, I don't see how it would make sense to expose that card intentionally as a cheating move. Maybe the dealer could have used that as a signal to expose the bluff. My guess is the bluffing player was bluffing in an attempt to make their hand (fake it until you make it) and the other player sensed that. Maybe the other player wasn't convinced and would have won the hand regardless. But at the point of seeing the exposed card, they both knew it was over.
NOTE: We would have to know more about how this played out to do much more speculation than the above.
Good question. It's not like the errant trader couldn't have had a friend of a friend put in a bunch of $10k limit buys the day before, knowing the price would recover nearly instantly...
One, I had put down $500 in twenties, and he was giving me chips, except he was going to give me $1,250 in chips, like he thought they were fifties. Pit Boss looked at him, looked at the chips, "What are you doing??", let him fix the error.
A couple of hands in, dealer gets an Ace, so he checks the hole card. Satisfied, he continues, and we all bet and play out the hand, until he goes to turn over the hold card... and has Blackjack with a King.
Pit Boss again, situation explained, voids the hand and returns everyone's stake to them.
What blew my mind was that after two mistakes like this in a matter of minutes, the dealer wasn't pulled off the table.
They often weight the rules back towards the players in other strange ways, like, being able to double down at any time (even on the 3rd or 4th card), resplit aces, as well as surrender.
A lot of Eastern European casinos don't even deal the "down card" for the dealer until the players have finished.
That’s not just Eastern European, it’s European in general. In fact, that ruleset is well-known as ENHC (European no-hole-card) among blackjack players.
Does anyone know what the dealer training looks like in practice? I assume people will make silly mistakes for a while. Are they expected to practice until perfection before they start working?
Would love to have a properly trained Vegas dealer weigh in here.
I can easily believe that development was lax without proper monitoring, controls and testing. Typically for any kind of trading system there'll be multiple layers to help catch these kinds of errors or deliberate fraud. This could include some or all of:
1. A separate system that monitors and clears orders looking for pricing mistakes;
2. A risk management system that looks at your total position and can block orders if they would make your position too long or short;
3. Another system might monitor actual assets vs custodial assets to see if you ever "leak" assets.
To trade any US securities you'd have to go through SEC compliance to make sure you have sufficient systems and monitoring in place, particularly if your system is facing unsophisticated investors. Crypto has largely avoided this kind of scrutiny and rigor.
Part of all this is separate systems help prevent both accidents and fraud. A pricing check in a trading system could be broken by a code change or bypassed by a malicious change. Subverting an external system you may not have access to gets more difficult.
In interactions of bluff, bluff, and double bluff suspicion never dies.
Quitting indignantly on principal is either a sure sign of innocence or a true con artist cutting and putting distance from a 'scam', staying can be just as ambiguous.
I'm not doubting your innocence here, just doubting that there was any action you could take that would convince a suspicious mind that you were clean.
I dare say had you stayed then eventually something else would have happened around you down the track (that's just life). Parting ways and moving on saved you from that at least.
I'm sure the doubt remained. What you wrote kind of reinforces my point that it's very hard to be sure that an expensive mistake was an honest mistake.
Personally, though, I didn't quit to try to double-bluff. I'm a self-preservationist. I'm in the business of keeping myself alive, and I draw the line at working for someone who threatens me.
I'm aware that quitting could've seemed shady, but at that point it was the least of my worries. And I doubt it's the normal course of behavior, either, if someone were caught skimming. Someone who was guilty of malfeasance, who was offered to keep the job, would have almost certainly chosen to stay and lay low while finding some new way to scam them, because that person needs either the money or the frisson. I enjoyed 'em both but I didn't need either enough to risk my life on another accident.
Someone who lacks any trust will always find reasons to assume the worst, as you say. There's a bit of a corollary, though, in my experience, which is: People who choose to continue working around those people also mostly can't be trusted. And I feel like the trust-less parties are well aware of that.
There could be nothing more at play here than rational self-interest. You quit to preserve your life; this is rationally the best move for your risk-adjusted interests. The house gets to say (possibly very visibly - did you quit on the spot?) that the dealer no longer works here, which mollifies the customer, because the potentially-corrupt dealer is no longer in a position where they can influence outcomes. So it's the best outcome for the house. And then the hand has already been played, so the customer now has a sunk cost on their bluff, but you're gone, so at least they know that if you are corrupt, you won't be dealing any more hands to them. So it's the best outcome for the customer.
You don't need to trust people who aren't in a position to hurt you anyway. That's why walking away is sometimes so powerful: you just change a bad situation by removing yourself from the equation, and start over somewhere better.
Who said anything about quitting indignantly on principle? The sequence of events goes like this:
1. Dealer deals a round of cards, does not intend to reveal a card, but does anyway.
2. Dealer is informed that if that happens again, he'll be executed.
No amount of indignation is necessary to get the dealer to quit. All that's necessary is that the dealer believe the assurance he's given in step 2. The mistake is guaranteed to happen again -- if the dealer was able to avoid doing it, that's what would have happened in step 1.
the patsy in that scenario would have been the guy screaming the game was rigged
or you, if they had popped you in front of the gambler to prove they took his allegations seriously
the patsy is not "in on" what's happening. They might let you off if they thought it was intentional but that somebody more important had put you up to it
wiktionary
patsy: (informal, derogatory) A person who is taken advantage of, especially by being cheated or blamed for something.
Someone at Celsius manually approved a bogus transaction on a hacked website...
https://www.coindesk.com/markets/2021/12/03/crypto-lender-ce...
Badger later reimbursed Celsius with inflation on their token that would be released slowly over several years. "restitution".
Someone at Celsius thought that they could sell that restitution token and ended up forfeiting it... which was another $22m loss...
https://www.thismorningonchain.com/articles/defi/celsius-mad...
Then they tried to get the DAO to vote to give them back their lost tokens, without even admitting who it was that lost the tokens... and everyone voted against it.
Someone exploited a weakness in CloudFlare and was able to replace the Badger website. When someone clicked on the site to execute an approval transaction, it went to the attacker first, which gave the attacker full control over their wallet.
It did take a manual step on the part of Celsius though... which should have been checked more closely. The UX around that checking is really terrible though and when someone is trying to do something quickly, they aren't always going to check. This is a big failure of wallets these days.
Balancer.fi just had a similar attack happen to them where the .fi registry allowed a nameserver change.
https://twitter.com/Balancer/status/1704552288201883809
It is also clear that the frontends really need to be hosted in a way that they can't be modified.
Years ago the advice was IPFS and IPNS.
I agree. This is not the Web3 dream everyone was promising us when frontends and nft media assets themselves are mutably stored on some server relying on serveral entities in the DNS chain to maintain security, behave, and stay available.
We kind of have the 'secure lock' with https doing part of the work, but it is kind of irrelevant if DNS is pointing to some hackers site.
This isn't just crypto... it is your bank too.
Further we used to have HPKP to further protect the security chain but it ended up being dangerous for various reasons. Monitoring certificate transparency logs for any re-issuences of your domain's certificates is the current detection method as http is pretty heavily penalized in todays browsers.
Moving millions of dollars customer funds around via a browser wallet is insanely bad, they should have had well tested scripts that interact with the smart contracts directly.
And for me personally, I will say every time the economy booms for a few years I start to ask: gosh is this ever going to stop?
For example with real estate, I haven't bought any because the prices always seem insane to me. But I'd be far wealthier (on paper at least) if I'd bitten the bullet and bought into the real estate market a few years ago.
Like, was Steve Jobs (turtleneck) covering something up? Zuck (t shirts)? Newsom with his Bruce Wayne hair?
Back then "nerd" seemed to have a much more negative connotation and seemed to describe people who were closer to "gross" than "quirky", like the trope of them constantly having a runny nose or sneezing everywhere due to allergies. Steve on the other hand came off as very clean and insistent on dressing nicely, even if in his own style.
I feel like Zuck was/is similarly seen as a sham for his entire robotic persona. Perhaps not as a scam, but he certainly never seemed to be all that liked by people.
I don't know anything about Newsom to comment on that one. But one other example where I was telling people it was probably a scam beforehand was Nikola. I don't recall what about his behavior did it for me, but something was just off about him in the way SBF was that just screamed deception.
Just leverage long 50x forever until they're all billionaires or FTX is bankrupt.
The hype cycle marketing was targeted by geography and social class.
Sadly, this sort of thing happens in traditional trading firms as well.
https://en.wikipedia.org/wiki/Oil_futures_drunk-trading_inci... https://en.wikipedia.org/wiki/Knight_Capital_Group#2012_stoc... https://en.wikipedia.org/wiki/Nick_Leeson#Barings_Bank https://en.wikipedia.org/wiki/Rogue_trader https://www.foxnews.com/story/typing-error-causes-225m-loss-...
- It lasted 12 seconds.
- Total BTC traded was 290.54
- Only 2% of that was at price of 10K or below.
- Most volume happened in $20-30K range, over 50% discount to mkt.
Here is a tweet from someone (rightly) speculating that it had been Alameda: https://twitter.com/TheoryBitcoin/status/1452345411759398923
With multiple layers of mutualised failsafes.
That being said, the Lightning Network was deployed on the Bitcoin blockchain in 2018. It allows for thousands of transactions per second, thanks to a very elegant solution[1]. Unlike traditional blockchains, the Lightning Network is not blockchain-based, freeing it from the limitations we are all familiar with. However, it inherits all the security properties of its underlying blockchain. While the Lightning Network has its fair share of shortcomings, its scalability remains an open question. Nonetheless, it has been working great for over 5 years.
Moving fast is great, but you need structure to support it. If you front load your development with guard rails that ensure you're always on track, then you "aim small, miss small" so to speak. If you do none of that and just hope that you can respond to problems as they arise then you've really not aimed at all and can potentially hit a target, but likely also hit a house.
Hang on a second. It caused a dip in the price of Bitcoin, that is not a "market crash". Bitcoin is a toy and not a real market, its price is set almost entirely by bullshit wash trades and other forms of fraud.
Similarly, printing a billion tokens and trading one for a dollar doesn't give you a billion dollar market cap, even though "all the stocks do it!" Those stocks also back productive systems, have financial statements, forecasts, and a million other things that make them a billion dollar (or larger) market capped company.
Things are worth whatever people are willing to pay. Bitcoin may be the original sin of crypto bros, but it still appears to be worth money, because it's impossible to make one for free, or obtain one for free. And it has utility. You can buy drugs or avoid the banking system using it.
Silicon Valley Bank had buildings. It was literally worthless at the end. A company is just as fictitious as a bitcoin, only existing because people all agree that "company" is a thing.
(EDIT: I changed my example to SVB from Enron)
> a company is just as fictitious as bitcoin
Nonsense.
Bitcoin's really pretty different; the value is ~entirely speculation driven. It does not, and cannot, pay dividends, or buy back its own 'shares', or anything like that. No-one would ever consider buying all the bitcoins (in the same way they might take the $20bn company private); the sole value of bitcoin is in other people wanting to buy bitcoin.
I think this may have potentially been true when everything was going up volume-wise, but as competition improves there's less margin for error.
But that's because the holes were actually directed added utility from that perspective.
Regulators were… not pleased.
(Quick English summary of incident: https://www.foxnews.com/story/typing-error-causes-225m-loss-... )
That is good for unsophisticated customers, but it creates a disincentive for market makers to provide liquidity when there is a fat finger event. That in turn leads to larger price dislocations on such events.
(The reason is that the market maker faces adverse selection. Providing liquidity is a bet from the market maker that the price dislocation will revert. Price adjustment reduces the profit the market maker makes from mean reversion. But if the price dislocation was caused by a real news, the market maker will eat the full losses.)
Seen them on the stock market too. Back in my day I wrote a few trading bots, they never went anywhere positive after about a week of profit.
Seems to me now that the smartest robo-trader algorithm is...
10 wait for human mistake...
20 goto 10I was trading crypto back in 2016-2017. Making a few thousand here and there. Mostly just from time in the market during the run-up of ethereum. The whole thing drove me crazy because I thought it was so stupid, like gambling. So in 2017 I withdrew my several thousand dollars and put in a trade on coinbase pro for 5 etherum at $100. (I had left $500 to do this, the most I was willing to risk at the time) On 2018-11-25 this trade was executed in a flash crash.
I had totally forgotten about this trade and when I was going to buy a house in 2021 I was looking for any change in the seat cushions thinking I may have left a small amount in these accounts. Well imagine my shock when the amount was just short of $18,000.
When I was young I "moved fast and broke things", then had brushes with the law that fortunately didn't result in jail. Silicon Valley prides itself on second and third order thinking, but do people ever consider what might come after "moving fast and breaking things"
(and members that create orders to do that a lot either get their membership removed or pay fines)
Wouldn't this sell just be gobbled up quickly by buyers? Why would it move the BTC price so dramatically to the downside? Surely this couldn't have been that much volume at near all-time highs of BTC in 2021?
This seems to imply that risk checks are somehow either unnecessary or impractical for manual trades, which is completely untrue. This is 100% a case of 'we chose not to implement that'.
Saved!
I guess we have different definitions of "invisible"
Fat finger errors would often result in unintentional gains. We incorporated them into our models as if they were losses of the same magnitude.
Perhaps the title could be updated to reflect that this article is not referring to the city?
The Athlete's Glitch
Our client team had an amazing NFL season. But with the start of a new NFL season, but our NFL team was buzzing for a different reason – the unveiling of our off-season updates to the beloved iPad app used by one of the major NFL teams.
Our app, which was primarily used for training and past game analysis, had undergone a major overhaul, and we were proud of the sleek new UI/UX designs. But just as the athletes started to get into the grind of their training, our office started receiving some unexpected feedback.
"Hey! The app is taking me to the wrong sections!" one message read.
"I keep hitting the wrong button. Something's off," said another.
Confusion took over our team. We had spent months meticulously planning, designing, and testing these updates. Automated tests had been executed to perfection, internal testing hadn’t shown a single glitch. What was happening?
Then, a video came in. It was from a coach, showing one of the top athletes trying to use our app. We watched intently as his fingers moved over the screen. The problem became immediately apparent: his fingers, sculpted by years of athletic training and naturally larger than the average, were simply too big for our redesigned interface. Every time he tried to tap a specific function, his finger would unintentionally touch the adjacent ones.
Turns out, in our endeavor to create a sleeker, modern UI, we had inadvertently shrunk the size of the clickable buttons and packed them tightly in a grid. This might have looked aesthetically pleasing and worked perfectly for our testers, but for the athletes with their robust fingers, it was a recipe for frustration.
We convened an emergency meeting. Our lead designer, Marcus, broke the silence, "Our primary users are these athletes. We should've considered their physical attributes in our designs. It's my oversight."
Our project manager, Clara, nodded in agreement. "We need to fix this and roll out an update ASAP. We can't have the team struggling with this during their crucial training period."
The next few days were a blur of coding, designing, and testing. With feedback from some of the players, we reintroduced larger buttons and ensured enough space between them, all while maintaining the sleek look of our new design. It was a lesson learned the hard way, but it reiterated the importance of understanding our users' needs and physical attributes.
The next feedback we got was from the star quarterback, "Perfect! Back in the game with this. Thanks, TechTouch."
The NFL season kicked off with roaring crowds, and our app, now more user-friendly than ever, was right there with the athletes, assisting them every step of the way.