OptiFi Program Incident Report
medium.com
medium.com
I’m quite sure the hacker who pressed the button feels plenty bad enough already, and I hope the people around them were as kind as the people around me were.
In the comment benreesman only said that enough data was deleted to cause 50-100M$ in damages. They might very well have managed to save half the data, but the lost half was already quite expensive?
Edit: thanks for replies! :)
See "How big is a billion?" from Numberphile:
You can even read about it in this paper which publicly described the system contemporary to that event: http://www.quinonero.net/Publications/predicting-clicks-face...
If you read it and are still steamed up, well, then you hold that opinion honestly, but I suspect most would be like, “cool story bro”.
All of these things benefit from advances in binary classifiers, and have therefore been massively subsidized by click prediction R&D. Maybe those benefits outweighs the costs, maybe they don’t.
We have GPUs because shoot-em-up games were “the big thing” in the 90s. Net win? Personal preference. But entertainment if one form or another often pushes advances in computing. The BOPR stuff from the linked paper originated on XBox Live matchmaking.
It’s perfectly fair to be like: I don’t like the ad-funded internet. There are tons of plausible reasons why someone wouldn’t like the ad-funded Internet, but most reasonable objections to it are of the form: “my internet experience is worse than it would be without ad funding”, past that you rapidly wander into “other people’s preferences are inferior to mine”.
Well, it would probably bankrupt most companies, but it wouldn't bankrupt most people's employer.
The average employee works at a larger than average company. Similar for the average customer or average investor.
That's just because big companies have more employees / customers / investors to contribute to that kind of average.
I find myself lately viewing crypto as mostly a bad idea, or a bad implementation of an idea with potential, or even outright as a Ponzi scheme.
These smart contacts are really a bad idea. We have contracts in life with lawyers, these tend to work out pretty well because we can use our human intelligence to arbitrate them. The idea of code as law is really dumb. I say that as a programmer who works on code all day. Computers are dumb like your calculator. They do exactly what you tell them to. Which sounds great until you realize how hard it is to describe what you want in a way the computer can understand. That's why I make a good salary and people think my job is difficult. The reality is computers often do what you tell them, but it's not what you wanted. Rather you have the instructions wrong or you made a mistake. That's why software is riddled with bugs. Avoiding that is very, very, very difficult. Nobody gets it right. These crypto finance bros will definately not get it right. And when they get it wrong, there's no human, no court to appeal to. How's that an improvement?
Being able to very easily put a price tag on sloppy programming is intriguing to me.
Certain fields have always had a high cost of programming error, including a cost in human life. It's just that cryptocurrency combines this with a first-to-market rush that's somehow still going on, encouraging a rush to error.
But with crypto, the effect is much more direct. The programmer is handling money much more directly and if something goes wrong, they are much more directly affected and not being insulated from the effects.
It seems like the crypto community would do anything to avoid the legal/regulatory system and it's established processes of operation requirements, insurance and liabilities.
Doing that they regularly fail at performing the most basics of basic financial duties like not getting hacked and not throwing the keys to the kingdom.
I've yet to hear of a bank loosing funds and getting away with telling "tough luck" to their customers, but we've witnessed many crypto "banks" doing just that.
That also involved failing to have a viable rollback plan. It affected so many people so badly that questions were asked in Parliament; TSB were down for a week.
Now with cryptocurrency we've disintermediated the bank to produce a much better solution: your contract can be down forever and there's no Parliament to ask questions in.
This aspect of crypto is a 1960s paranoid computer fear come to life, and somehow it’s often presented as an improvement over the existing system of human checks and balances.
Where do you think your income and bank balances are tracked and stored? On pieces of paper?
Your bank analogy is silly and nowhere near analogous.
https://www.bloomberg.com/news/articles/2022-08-15/citi-sues...
You can say that's an edge case today but I and OP are saying, the future will look more like crypto looks today. Not a bright future.
Given it was an accidental early repayment of a loan, this isn't quite the slam dunk you think it is. If they had paid a company they didn't owe money to, they could get the money back through the courts.
Crypto is meant to evade those courts.
> “To believe that Citibank, one of the most sophisticated financial institutions in the world, had made a mistake that had never happened before, to the tune of nearly $1 billion, would have been borderline irrational,” he wrote.
https://www.nytimes.com/2021/02/16/business/citibank-revlon-...
That's as close to "code is law" and other nonsense of the crypto libertarians as you can get.
> Recipients of cash wired in error are typically required to return it.
I don’t agree. I trust that loopholes like that will be slowly rectified with legislation if not present today.
In fact, it’s likely that crypto will (problematically?) be heading the same way. There was a recent case of a crypto buff who found a bug in some project and made off with a few $Million and I think the courts said he could be arrested and expected to return it, just as if he made off with cash. Importantly, they basically said “blockchain isn’t the source of truth to the courts” which was the guys defense. IMO a bright future for people, but not for a crypto venture.
You have cash in your house and someone breaks in to steal everything. The insurance will _maybe_ cover your loss given that you secured it with basic security. _Maybe_ the police will investigate and arrest the burglar.
But even in this situation, you were better off putting your savings in any bank account where any fraudulent transaction can be reverted with a button.
Now, you have your crypto wallet. It gets emptied by some random bot. Well, you are as fucked than with your cash, except that nobody will cover your loss and nobody will investigate your case since the burglar is probably from another country.
You can insure the crypto you have. It’s probs expensive to the point of being not worth it.
People investigate crypto hacks though. And if the perpetrators are in a jurisdiction that you have some legal availability to you can totally use legal means. Basically any western nation will allow such a suit.
https://www.coindesk.com/policy/2021/12/22/teenage-suspect-i...
You can strike out "financial" and that's already the realty we're living in. We got lucky that Y2K was not an issue today. We wouldn't probably be able to fix enough code and nowadays much more is under direct computer control than back then.
Nobody "wants" that but it happens gradually. With crypto, it didn't happen organically but crypto blasted onto the scene from the side of complete digitization. Looking at that, it's easy to say that nobody would want that but while you're looking this way, traditional finance is creeping towards complete digitization as well, just behind your back.
> This aspect of crypto is a 1960s paranoid computer fear come to life, and somehow it’s often presented as an improvement over the existing system of human checks and balances.
I'm not saying that crypto is an improvement as it is now. Nor that it will ever be (it might but I have my doubts). But what you're missing is that crypto is complete wild west, like traditional finance was maybe in 1900. The whole history of finance is a sequence of fuckups and laws and regulations that were imposed to prevent similar fuckups to occur and we still got the 2007 financial crisis, after 150 years of improvements.
I can try to stay as far away from crypto as possible but as I said in my previous post, there are aspects that are interesting even if most of it is completely nuts.
A mea culpa of “The one thing we purport to be good at we actually have literally no understanding of and when shit doesn’t work we just run it a few times with different arguments.” My god.
You're welcome.
I find the prompt for dangerous ops useful. GitHub will ask you to repeat the name of the repo before you destructive actions, Terraform will ask you to say yes to the prompt. These are all good things.
The better alternative will be instead of asking for confirmation to ask to explain the intent by picking one of the options or writing it down. Ideally, an alternative must be suggested. This will avoid automatic reaction, because there's no one clear path to the goal.
It's almost never the case that blaming the user is actually going to help nor that adding more eyeballs will prevent people from making mistakes. If it's routine, we'll apply it to the wrong entity. If it's not routine, we'll not understand all the implications of our actions.
Also, to stretch the analogy I used further: one way to avoid lighting one's own house on fire by mistake is to not make the things you light on fire look like houses. In the case above, and in many other cases (like the big Atlassian outage earlier this year) the problem wasn't so much that the user was deliberately deleting important stuff, it's that they couldn't tell the difference between the class of unimportant things they thought they were deleting from and the class of important things they would stop and think long and hard about before deleting.
And then there's a regulatory problem: investors trust their money to businesses which have not earned that trust, because of whatever magical thinking that exists on this market. At least the company seems to be able to return the money, but will it be sanctioned for this failure? There should be a regulatory incentive to do better next time.
It seems like this was in the "deploy to production" stage, and there has to be some mechanism for doing that.
(It also seems unnecessarily complicated; I'd appreciate a plaintext explainer of what actually went wrong)
Any changes to production must be tested before release. They did not do it.
So I would just confirm but not realise what I was confirming.
People would tape one button on, then wonder why it would chop a finger sometimes
Yes it is annoying. Yes, there may be better choices. But it is there for a reason
Funny that people still ask me "hey, why do you trust those evil banks? you should do all in crypto instead!" and then laugh at me when I tell them I trust banks more than some random dev on the internet...
That's when most crypto companies right now are very small. Can you imagine the chaos if crypto were to actually become big and dev count were to grow to 1k+ people spread across multiple offices?
There's definite efficiency gains with crypto (a dex like Uniswap can do massive volume with very few developers for instance), but there needs to be a way to limit the cost of human errors.
That's reality.
Same thing would have happened if they had put all that cash on a boat, and accidentally sunk it.
The normal banking industry is operating chainsaws very cautiously, with a lot of safety equipment and training. The cryptocurrency industry may be operating the same chainsaws, but they're trying to juggle them naked, on a floor slick with the blood of their peers.
Are you saying that it is impossible to get valuables out of a boat that sunk?
"Treasure trove of gold and jewels recovered from a 366-year-old shipwreck in the Bahamas"
If you'd prefer you can throw your gold bullion into a volcano, or perhaps mix it in with radioactive fuel.
But there were a few instances in WW2 where large amounts of value had to be transferred by boat, and in at least one case sunk by enemy action. HMS Edinburgh: https://www.warhistoryonline.com/instant-articles/hms-edinbu...
This is hardly exclusive to crypto.
> Can you imagine the chaos if crypto were to actually become big and dev count were to grow to 1k+ people spread across multiple offices?
Yeah I can already imagine in other places, just ask Santander Bank. [0]
[0] https://www.theguardian.com/business/2021/dec/30/santander-b...
Per the article, they were very likely to recover most of that money (and were legally entitled to recover all of it, but were unwilling to do so for image reasons). So exactly the opposite of what happens with DeFi.
see: Knight Capital
Lessoned we learned harshly EVERY DEPLOYMENT NEEDS A RIGOROUS PROCESS AND SINGLE POINT FAILURE CAN BE AVOIDED. PLEASE DON’T RUSH LIKE WHAT WE DID, ESPECIALLY FOR DEFI PROJECTS.
Best wishes to the company and individuals involved for recovery of assets and success going forward.
However there were other assets on the chain that are locked using the program which are now inaccessible because its inoperable.
Consider it like an operation that closes and deletes your user-account but returns you an archive of your home directory when it's complete. However all files that weren't in your home directory that were encrypted with your local user key now can't be read.
It's likely infeasible to predicate program deletion on the program no longer being referenced in other programs/contracts.
Disclaimer: I'm not a Solana expert (or a crypto advocate for that matter)
If you want something where some developer can change the rules anytime, then just pay the money direct to the developer, and hope he pays you back one day.
Contrary to the HN zeitgeist, I have a lot of excitment for decentralization medium- to long term. But it seems almost everything that's happening in the DeFi space today is about as decentralized as a traditional client server app with mysql.
As for this issue, just look no further than the DAO hack and see if that has taught crypto anything about so-called decentralization; clearly it hasn't.
> - We will adopt a peer-surveillance approach which requires at least 3 peers to engage in the deployment process. They have the responsibility to remind the main deployer of any potential risk, and make sure each step complies with the deployment guides and norms.
> - In case anything abnormal happens during the deployment process, such as bad network status or insufficient deployment fee, we should calm down and have a discussion with peers to make sure each operation is safe. Meanwhile, we should mark down every command line and returned message for further reference.
I wonder if that’s enough stack of Swiss Cheese to prevent such an accident from happening again. Hopefully they expand on the “we should calm down” to not be limited to saying “calm down please”. Calming down is quite difficult when you’re in the throes of something, and proper procedures tend to be put aside for the sake of pressure relief.
I wish them swift recovery from their predicament.
The first company I worked for sold access to data on a per minute basis and it was like $1.50/minute or something (dialup access in the 90s to an interactive ClarionDB session).
So any outage would be quantified by some curious programmer. I remember even small outages being quantified in hot washes as 10 minutes x 2000 users x $1.5 = $30k outage.
I remember one of my bosses was nicknamed “Commander Crash” as he once crashed the whole system for hours based on a single live config file edit.
No one was fired but it was actually liberating knowing how much mistakes cost.
Towards later in the article they say "Here it turned out that we didn’t really understand the impact and risk of this closing program command line. ‘solana program close’ is actually for closing the program permanently and sending the SOL tokens in the buffer account used by the program back to the recipient wallet."
What?!
solana program close
The program's designers should likely have chosen the verb "shutdown", instead of "close".
solana program lose-all-money
If it’s a simple change, why not do it? Though, I can see your point if the change is monumental but barely reduces rate of mistakes.
Otherwise why the edited title that explicitly points this out?
As in, you have all this crypto with made up value, and we are going to “cash out” due to a mistake, and now you have whatever we stated the value is in cash?
Doesn't even look like the Solana cli supports an HSM for this usecase.
Whoops, I lost everything! Computers are useless.
In the crypto world all changes are permanent and immutable.
There is no recourse.
That’s the difference.
I mean look at online banking - in the UK it's only in the last few _years_ that we've been required to do 2fa when buying things online. The earliest online banks didn't even have 2fa to log in and move money around, and at some point HTTPS didn't even exist so everything was sent in plaintext. SMS is still mostly plaintext. Things evolve.
Crypto is all about disintermediation. But that doesn't work when people's money / savings are on the line.
So we can easily tell exactly where this leads: Once enough pain has been sustained through errors like this, DeFi code bugs, fraudulent transfers, etc... a whole industry of HUMANS will pop up that will (a) become an intermediary in crypto/DeFi transactions to ensure that fraud, errors, etc are prevented; (b) introduce mechanisms to revert transactions (controlled by said HUMANS); (c) perform the massive paperwork on compliance (KYC, register big transaction, check against blacklists, etc)
Is there really a different outcome?
Crypto fans claim finance is an overbloated industry and they are here to disrupt it. In reality all they are doing is recreating exactly the same system but with one more layer of indirection (instead of transferring dollars you transfer a stablecoin that points to a dollar).
Will there be exciting use cases for crypto at the edges? maybe. Will it 'disrupt' the broader financial system, in the sense that it will make it better/faster/cheaper? I don't see it.
When they do that they'll realize they no longer get any benefit from the underlying asset being a cryptocurrency (since it is no longer trustless as the humans are able to reverse transactions) and will switch back to a trust-based system with a good old database as the ledger.
Defi is just the market hearing that over and over again and saying, "OK. I heard you like ponzi schemes..." They all already have humans throughout--- if nothing else, in the form of the people walking off with the windfalls.
The fact that the tech keeps blowing up is mostly just an artifact that the tech is just there as obfuscation for the fraud that underlies and motivates the enterprise. The technology is not well thought out because it doesn't need to be, it's not managed by people with high technical expertise because people with such expertise see through the schemes and can find better things to do with their time than to help rip people off. Given the economic or centrally trusted points of failure-- a competent and ethical engineer would usually tell you adding smart contract gunk to these schemes is an unacceptable source of risk without meaningful benefit (except perhaps as pretext to hide from law enforcement) and so the systems the world gets are the ones built by people who were less than competent and ethical.
Apologies to the rare few things under that banner that aren't fraud -- but they're part of a lemon market where they can't be distinguished from their more fraudulent compatriots, so that kind of guilt by association is inevitable.
But what does that matter? The point of decentralization is not to abolish humans, services, platforms, user-experience designs. The blockchain acts as a neutral base layer, and competing services can operate on top of the same shared data. OpenSea has limited control over the NFTs being listed on it, which is why you can buy and sell the same assets on other platforms.
What was lost was not money, but electronic conkers of which a small number people paid real money for.
If they welch on it that’s another matter, but the stated time is tomorrow so…so far so good?