3rd largest Bitcoin exchange lost its wallet.dat | Hacker News Reader

3rd largest Bitcoin exchange lost its wallet.dat

bitomat.pl

117 pointsmef98 comments

Everyone keeping their bitcoins in exchanges/(think banks, or broker accounts) instead of their own wallets is completely subverting the point of the idea of bitcoin.

BTC has some great unique properties as a p2p cryptocurrency - which goes out the window when everything sits in a few exchanges rather than a bunch of wallets.

Find a way to easily trade bitcoin without centralizing wallets and you got the thousand bitcoin idea.

Next step is to turn the thousand bitcoin idea into an actual reality and get rich.

I thought one can directly send bitcoin to another address without the need for a middleman?

Yes, but dollars and euro and yens almost always require a middleman, unless you're doing cash exchange.

Ah, I did not realise that this was about a "transformation" between bitcoin and normal money. But in that case wouldn't it make sense to keep the money on the exchange just as long as needed?

The bitcoin exchanges typically require 6 confirmations of a transaction in the block chain before crediting your account with bitcoins. This takes about 1 hour.

If you are speculating on bitcoins and you hear about some news that might cause the price to drop, then you don't want to wait 1 hour before you can sell them.

Can you offer more money for the verification, to get it done faster? I think the default is 0.01 BTC, so if you offer 0.05 BTC will you get priority?

At the moment even no-fee transactions usually make it onto the block chain as quickly as possible. The ~1 hour (6 block) confirmation time is due to the recipient being cautious that the sender won't try to double-spend the money by rewriting history with lots of computation power. As more time elapses from when the transaction occurred, this becomes harder, and not as a function of your transaction fees.

The transaction fee only helps get it into the first block sooner. It does nothing for the X confirmations after the first.

Of course one can. But what he's talking about are large-scale exchanges, where one makes transactions with funds deposited directly on the exchange.

You can always do that if you trust the person enough. The idea with exchanges is essentially to act as an escrow service, but one where the two parties don't have to directly know who each other is. I don't think you can decentralize trust, realistically. Someone ultimately has to be responsible for a bad transaction or handling of BTC.

Ah, but if the people running the _exchanges_ are too incompetent to safeguard their wallets, as in this case, how bad will the average user be?

It's a free market, someone else will step in and create a more secure exchange.

Only if it's worth it to them to do so. Could be altruism, could be money, but the mere lack of a good product doesn't guarantee someone to step in.

What unique properties of a cryptocurrency are negated by a functional exchange?

One of the problems with ever realizing a "true free market" in bitcoin trading is the time spent waiting for block confirmations. There are barriers to entry in that you can not just transfer your coins to an exchange and sell instantly. Because of this a lot of people who trade bitcoins keep a certain amount in their exchange accounts.

I guess the bitcoin economy is ruthless on its entrepreneurs and users.

Failure to backup, improper trust in individuals, improper understanding of the technology, poor security practice, and bad decisions will wipe out early adopters' wealth continuously and relentlessly.

I doubt we will see much of the early adopters retaining their bitcoin wealth from these early days. Those who did are either incredibly lucky, or incredibly competent.

I am an early adopter of bitcoin and I hope to survive or avoid the many dangers and poor decisions that ensnared many of peers on the road to bitcoin richness.

I think that's a bit over the top. It really isn't that hard to keep your coins in your own wallet and back it up.

What this demonstrates to me is that people are too trusting. You really shouldn't trust any of these services until they become more transparent in who is backing them, what their infrastructure looks like, their disaster plans, etc.

It isn't, but since the primary thing to do with Bitcoin is to speculate it, keeping it on your PC is problematic.

If this kind of thing keeps happening, isn't Bitcoin inherently deflationary? With gold, for example, only a small proportion is ever permanently "lost"; even shipwrecks can be recovered in the future. And of course more gold is always being dug up. But if bitcoins from a fixed set are slowly being lost in various ways, irreparably shrinking the money supply, it seems like it'd have trouble being a viable long-term currency.

The 21M BTC limit already made it deflationary, so lost BTC just increases the rate slightly. The early adopters tend not to complain.

In practice, when used to back a currency, gold is usually also inherently deflationary; mining of new gold generally cannot keep up with real economic growth.

Yes, BTC is inherently deflationary, but one of the fascinating things about it is that since it can be infinitely subdivided, this may not be as much a problem as it for meatspace currencies.

Currently BTC is divisible down to 8 decimal places, enabling a maximum possible 2.1 quadrillion atomic units. But that is only an artifact of the data structure used in the current implementation, and could potentially be modified to allow even more granularity.

So, as the value of BTC rises against goods, services, and other currencies, the market can reprice in smaller and smaller increments of BTC, and the incentive to save/hoard that is typically associated with deflationary monetary systems may be weaker with BTC.

> the incentive to save/hoard that is typically associated with deflationary monetary systems may be weaker with BTC.

I don't see how that is possible. The coins may undergo a process similar to a stock split but any holders of the coins end up with equal purchasing power. The incentive to hoard does not go away if the currency is deflationary.

Was this a relatively young BitCoin exchange? One would hope that the wallet.dat was backed up somewhere else (another EC2 instance, EBS, offline).

This doesn't bode well for establishing credibility in BitCoin exchanges and operators. Although nobody necessarily needs to use the exchanges, for better or worse, people use them to establish market values to more easily value their transactions.

It's obviously easy to criticize after the fact, but I am genuinely curious how backing up critical data wasn't considered. Unless, this was a very young / recently launched BitCoin exchange as a weekend hobby project that took off?

They've been around for a few months and were one of the higher volume exchanges. They were doing ~800 BTC in trades/day for the month prior to this, so ~$11k/day with their commission of .006*2 (I think it was .006, and on both sides) they were pulling in ~$130/day.

Everything (transactions, withdrawals, deposits) was free of charge.

Only way of profiting legally from the exchange was interest on the capital deposited by users on exchange owners bank account.

Or "losing" the wallet.dat after a few months.

You're operating in an unregulated market. Your legal protections for when someone outright steals your money are much weaker. You might be able to sue them... if you could get to Poland. Not only is your market unregulated, you don't share a jurisdiction with the "bank" unless you live in Poland.

So it's not the only way of profiting legally as much as it's the only way of profiting ethically.

I don't think this particular case was malicious, but I think people are vastly underestimating their risk exposure to a malicious exchange. Thankfully, the bad guys seem as incompetent as the people running bitomat.

If it's a scam, lying about losing the file is useless, since all the BTC transactions are publicly logged. You can just input their address(es) on the Bitcoin Block Explorer and see if any of them are being used.

Of course, that doesn't mean you'll have any legal recourse against them.

The addresses being used don't necessarily indicate the exchange owner was lying. An enterprising AWS engineer might have been able to recover the wallet somehow. At a value of $200,000+ USD it'd be worth looking for.

Here's that plan laid out: Sell BTC for EUR; deposit EUR in a checking account/savings account; receive a bank's interest on the EUR. As customers demand the BTC in the amounts enumerated in BTC that you still owe them (in great numbers after some announcement like the above, or otherwise), buy BTC with EUR and sell it to the customers. Pray continuously that you don't lose your shirt with these conversions, particularly given that unexpected spikes in BTC demand will come precisely at those times that you'll tend to lose out on a conversion.

You might, extremely conservatively, sell BTC to people in exchange for a promise of a greater amount of future BTC (i.e., you might lend BTC out). Step 1 in that scheme: find anyone who will borrow BTC who isn't either a lunatic (and therefore unlikely to be able to repay the loan) or a thief. After subtracting lunatics and thieves, you're left with people who will take some BTC, sell it for EUR, attempt to make money, and then at a future date buy BTC for with any EUR they've made. I.e., with people who will need to pray almost as hard and as long and with as much chicken blood and magic circles as an exchange that took your suggestion to get interest from a bank. Your borrowers won't have to fear a sudden demand for the money, but they'll still have to fear the BTC:EUR exchange rate.

Meanwhile, approximately 100% of BTC holders are speculating in the currency.

Sometimes you don't appreciate regulation until it's gone.

An exchange failed on its incompetency and now they are trying to liquidate and giving back their customers' money. The market works even if the consequence is quite sour and painful.

This will means that exchanges will tend toward competency rather than incompetency.

Regulations is something that you will have to design first or else you get regulatory capture or "TOO BIG TO FAIL" or some kind of unjust monopoly. Adding a regulator means that we will have to shift our focus in making sure that the regulators are doing its job rather than ruthlessly killing off companies that can't keep customers' money safe.

Except with a lost wallet.dat there will be a lot of BTC that cannot be given back. Customers are likely going to take a haircut because an incompetent exchange lost their money. In the real world the people running the exchange would be going to jail.

I don't understand the point you are trying to make.

Incompetent exchange goes out of business. Said exchange try to compensate their customers. Customers learn better to scrutinize other exchanges for their backup policies and how they run their business.

Of course, this is the free market. Failure to be competent, either as an investor, user, or exchange, result in monetary loss.

To others, this is horrible. To me, this is the way I like it.

I wish for the accurate consequence of my decisions, not a jacked up distorted picture of reality. This prevent crisis from becoming a minor blip in the bitcoin economy into something that fundamental destroy the bitcoin economy.

Adding regulators who have a hold on the whole Bitcoin economy with no alternative for said regulators, with no incentives to for regulators to regulate the market right is a recipe for disaster. You want regulation and regulators? Show me the incentives for them to make things right. Show me why they will not fall prey to regulatory capture. Give me precise details of how this is all suppose to work and what the rules are.

That there's not a regulated option... To use BitCoins you have to go into the underbelly where you're quite likely to be stolen from and have no recourse. While it's great you enjoy the ability to be mercilessly preyed upon, it's not great that there is no safe option.

This one was just (apparently) incompetence. Wait until the real crooks get going.

The safe option is to use an established government backed currency, I want bitcoin to succeed, but I can't honestly tell people about it without describing it as an incredibly risky investment (right now).

Bitcoin has unique properties in terms of exchanging wealth. If you don't actually need those properties though you probably shouldn't be using bitcoin for anything other than playing with it.

This is an inefficiency of the market. You must have some level (perhaps not rigidly set) of inefficiency at which you would conclude that some type of government regulation would make the market more efficient.

Otherwise you'd accept any level of inefficiency as the market working, at which point it's just an ideological label.

Although it's not clear that regulation of bitcoin is even possible, we can still consider losses due to its absence.

What makes you think the government can run an economy?

Our economy, despite recent "deregulation" is the most regulated, especially when you count criminal and civil law as well, in the world. There are countless laws against selling known-bad products (lemon cars, toxic mortgages, etc) and we could have almost everyone involved in the recent collapse behind bars if we tried.

We pay through the nose for the regulation, especially in the collateral damage from false positives and economic friction, and it's totally worthless at catching the biggest criminals (in terms of dollars stolen) in world history.

If I want regulation I can use a regulated exchange. Or rather, if I felt like using beta-software I'd be doing it and would be thankful to be allowed even if I had just lost my test fund. Forced regulation would remove the choice.

And ultimately regulation implies that rules stop the bad guys and we know that's true. Thanks, but I'll wait until an open-source and audit-able exchange comes along, and then I'll listen to the experts I respect.

You might have better luck in online debates if you argue against the actual points made, instead of whatever that was.

Where do you live that incompetent people who run banks go to jail?

Apparently a different branch of the multiverse that forked from ours way back near the beginning of time.

There is a movement among the users to exchange lost bitcoins into bitomat shares so the owner compensate for his errors with parts of his own company.

I disagree. Regulation has given people a false sense of security, leading to them trusting these unknown and untested entities.

Instead of regulation what they need is more transparency. Tell your customers who you are, how you are building your service, what you will do in the case of a disaster.

In this particular case you have someone who is playing with the technology without having even a basic understanding of how to build the required infrastructure.

How do you ensure transparency without regulation?

Adam Smith's invisible hand will take care of it.

What's to prevent a firm from lying about who they are and what they are doing? Without regulations (and accompanying penalties for breaking the rules), it's the wild West. It sounds good on a libertarian check list, but in practice the people who can least afford it get absolutely destroyed.

Bankruptcy.

Plus, backup is just a good policy. Practice it lest you end up homeless or have to find another job.

It's hard to go bankrupt when you steal for a living.

Believe it or not, it's even harder to go bankrupt without regulation... Bankruptcy refers to the legal status of an entity!

This thread has just about reached the point of absurdity.

In theory, you'd have trusted independent third-parties that would verify and vouch for your company if you asked them to. Rating Agencies are an example of those.

We know how well they work in real life, though.

That's a good question. I don't think regulation really improves things though, it just adds another layer of obscurity and involuntary regulation (is there any other kind?) goes against my libertarian sensibilities.

As for the people who 'can least afford it', well they need to learn how to be paranoid, the sooner the better.

Reputation is, to me, the best way to ensure trust. Build a public reputation through transparency and actions. Of course this takes time, but in the end is better than than an external entity imposing regulations.

On the bright side, if the wallet has truly been lost and not stolen, everybody else's BTC got just a bit more valuable.

In other news, funneling thousands of trades through one wallet.dat... how on earth do you successfully back that up when it has new data every minute? If exchanges are going to become a "thing", I'd think you'd want something like multiple redundant networked databases storing the BTC.

The individual trades in exchanges don't hit the wallet. Only deposits and withdrawal's do. The trades themselves are handled by the exchanges database.

You can pre-generate addresses in a wallet and then do a backup. This backup is valid and can be used to recover funds as long as that pre-generated pool doesn't run out.

There's also a 'backupwallet' RPC command that the bitcoin daemon provides to safely copy/backup the wallet. So lots of options available.

Not necessarily,

The hit to confidence in bitcoin might decrease the value of the remaining bitcoins more than the loss of these bitcoins increased it. I suppose we'll have to look at the exchanges...

Looks like the admin stored the wallet.dat and backups on an ec2 instance, and then shut it down while doing a RAM upgrade without realizing he would lose everything on the disk.

Translation of post:

DECLARATION I hereby inform all users of the service www.bitomat.pl of system failure that occurred on 26 July 2011 and its consequences. At the outset I would like to apologize for such a long delay in publication of this statement. I explained that it was dictated only by the good service investigation conducted to determine the causes of failure and the people responsible for it. Unfortunately, to date, despite intensive efforts, could not determine these issues. However, I believe that the longer pause to disclose this communication at this stage would be unreasonable. I am also aware that service users are the appropriate explanation. On 26 July 2011 at about 23:00, I noticed that absorbs all Bitcoin server machine resources, and probably not used for making. So there was need to increase the amount of RAM in the server. As a result of that procedure - suddenly the whole virtual machine has been erased, all data stored on the server has been lost!, Including records concerning bitcoinowego portfolio and its backups (backups). I have taken action have established that the disappearance of the data was the result of the introduction of virtual server settings, which he never would have introduced. Amazon Web Services Company, which is located servers, website says that the machine that has been cleared has been set up in such a way as to be irretrievably destroyed automatically with the data on disks attached to it at the time of her arrest by the shutdown. We are constantly trying to determine who made changes to these settings and whether it will be able to recover lost data. Unfortunately partnership with Amazon Web Services, which was placed servers service is difficult. Once I realized that I deleted the machine have redeemed the biggest package of technical support, I talked to the manager, asked about the security of disk space, I explained, so far unsuccessfully. Still exerts pressure on the Amazon Web Services to accelerate their activities but without concrete results. At the moment I am unable to clearly determine the causes of crashes, I suppose that it is the result of actions of third parties, which are causing the server tried to cancel to hide their illegal activities, or intentionally wanted to website disappeared. If my suppositions are confirmed, the fact will let police and prosecutors. At the same time if possible take action through which it would be possible to recreate lost data. But what I need to interact with the server's owner, and that as I mentioned above is difficult. At this point I wish to inform and assure you that your cash deposited into your bank account service and not converted into BTC and unpaid cash from the sale of BTC remain safe and intact. Any further findings will you keep. At the same time I am counting on your help in solving the problem. I realize that the situation is very difficult, and you fear for the fate of their BTC. We are constantly working on a solution to the crisis, and I'm open to your suggestions. Currently going to: cancel all active orders so far, to restore service to operate to allow the performance of any operation (in particular the payment of PLN). Please your suggestions and ideas. I wish to inform you that I had several conversations with potential investors from home and abroad. Www.bitomat.pl service is on sale for EUR 17,000 BTC. If interested, please contact us at bartek@szabat.com. Best regards Bartek Shabbat Service Administrator www.bitomat.pl

> Www.bitomat.pl service is on sale for EUR 17,000 BTC

What?

That was my reaction as well. Not just the fact that it's ambiguous whether they're trying to sell for 17k EUR or 17k BTC, but the fact that they're even trying to sell it at all, and then the buyer is presumably on the hook for the BTC that were lost as well!

I currently had ~13 BTC in my Bitomat account- when I login they correctly show my balance, so I'm slightly confused as apparently their database wasn't lost, but their wallet data was? How does that happen? And why was the wallet data not backed up on a non-EC2 instance?

All of this is making me lose some faith in the exchanges- between the MtGox BS and this, is TradeHill the only exchange that can be trusted? Can they even be trusted? It seems they are by far the most competent from a technical perspective, but at this point who wants to trust any of these exchanges with any significant amount of money?

Exactly, they screwed up big time by not knowing one of the fundamental things about EC2, now they're trying to sell the thing, and they can't decide on a currency on top of that?

However, it looks like the original page says BTC, so it looks like just a translation error.

Given the price they're selling for I assume they'd use that money to refund the funds from the lost wallet.

That'd be a very dangerous assumption for anyone considering the purchase.

> EUR 17,000 BTC

EUR here is just some strange artifact of google translate

> Serwis www.bitomat.pl zostaje wystawiony na sprzedaż za kwotę 17000 BTC.

translates to:

> Www.bitomat.pl service is on sale for EUR 17,000 BTC.

Maybe they don't say EUR much in polish and they used a "for money" kind of phrase and gtranslate was being "helpful."

Yea- why would BTC translate to BTC, but "kwote" translates to EUR?

Google Translate is heuristics-based, and can occasionally produce strange results. For instance, when translating Irish-language text, it will sometimes translate Baile Ath Cliath (Irish for Dublin, the capital of Ireland) to London, the capital of the UK.

This is likely because its corpus for Irish is made up largely of Irish government documents (all Irish govt documents are translated into Irish, but Irish isn't used by many people and there's not all that much else written in it), and these documents are rather similar to British civil service documents.

"kwote" means "amount"

you could translate that "za kwote" snippet as: ... for sale for the amount of 17k BTC.

I don't understand how this is news to people. Bitcoin was designed to be a secure, distributed, anonymous country. What do you expect to happen when you stop relying on Bitcoin to store your money and start relying on some random Polish webapp?

Like many 'random Polish web apps' this one was among the top in its class. Poland has world-class software talent, please don't downplay it.

Didn't mean to disparage Polish software development; just was assuming the average user of the site wasn't Polish, leaving them even less legal security— which I see, glancing at the site now, was wrong. So apologies to any offended.

With that said, I think you could pick a better site to be the shining beacon of Poland's world class talent than the site that just lost a bunch of people's money because the only place they were keeping it was on a single EC2 instance store. If this is the top, I'm not sure I wanna see the rest of the class.

> just was assuming the average user of the site wasn't Polish, leaving them even less legal security

People should get rid of that "poland is europe's mexico" notion. It's the EU - you have the same legal security as in the UK, France or Germany.

These particular developers, though, were probably not world-class. Data this important simply should not be lost due to trouble with an EC2 instance; any data on an EC2 instance, even on EBS (though somewhat less so for EBS) should be considered liable to vanish at any moment.

A user's bitcoin "wallet" is really their private key. It seems they didn't lose their database, so they still have a record of how many bitcoins each user should have, but they don't have access to those coins.

1. How do you "increase the amount of RAM" on an EC2 instance other than spinning up a new larger instance?

2. Why didn't they use EBS and take frequent snapshots to S3?

It sounds like they were running an instance store EC2, then fired up a larger instance and stopped the first one? This has to be a textbook case of how not to use EC2.

You can't. Instance types are fixed. He was likely using an EBS-root configured AMI, where the mount point for EBS is /. These are EBS-backed, but since the EBS block is automatically created when you spool up the instance; it is also automatically deleted when you terminate the instance, unless you specify otherwise.

This.

I have to repeatedly tell people that EBS-backed is not persistent. Amazon did everyone a big disservice by calling something -backed when it was actually transient.

It is persistent until you delete it. What more could you ask for?

persistent until you terminate the instance. Unlike a EBS volume that you explicitly attach, which persists beyond the termination of the instance you attach it to.

This is not actually true. If you Stop an EBS-root EC2 instance, the EBS Volume can be safely detached and moved along to other instances, or you can restart the instance and your data will be ok. Additionally, There's no excuse for not snapshotting your EBS volumes of any sort, and these snapshots persist beyond the lifetime of the volume. I find it just a little sad that relatively few (less than all) EBS customers don't snapshot their volumes.

You do not have to move your volumes: while the instance is in a "stopped" state you can change it's instance type and then restart it. EBS-root let's you change hardware as fast as you can reboot with just three commands.

You are, of course, correct. Notice in my post that you replied to I was careful to use the word terminate, not stop, when referring to what I assume he did to his instance.

edit: After re-reading my post it probably wasn't clear to people who don't use EC2; so your reply is probably for the best. :)

And why would anyone want to immediately terminate an instance when you can just stop it instead?

Cynical translation of that: I didn't understand the setup and made a change which deleted everything. I haven't found anyone else to blame yet but I'm desperately trying.

There was need to increase the amount of RAM in the server. As a result of that procedure - suddenly all data stored on the server has been lost! [..] the disappearance of the data was the result of the introduction of virtual server settings, which he never would have introduced.

For Amazon persistent storage, you don't just change a setting, you need to pay extra for EBS storage and design your service to use it, so saying the following:

We are constantly trying to determine who made changes to these settings and whether it will be able to recover lost data.

Suggests a continuing lack of understanding, and that makes it seem more likely that the service was built on quicksand and is now sunk. Even if we assume that they did have EBS volumes available and someone had redesigned their service to not use them, that they have no access audits, no regular setup audits, no long term backups, no revision control history, no change procedure and no email records (and nobody who remembers anything about it) to give them a clue what happened, that's pretty damning in itself for a financial service.

Once I realized that I deleted the machine [..] At the moment I am unable to clearly determine the causes of crashes

Maybe an artefact in the translation, but you can't accept that you caused it while also being unable to determine the cause!

I suppose that it is the result of actions of third parties, which are causing the server tried to cancel to hide their illegal activities

If this is a serious admission that their Amazon account was hacked, the assurance that unpaid cash from the sale of bit coins is safe seems a bad idea, and hiding it at the bottom of another announcement is really poor form. If it's not a serious admission then it reeks of pointing the blame anywhere except 'here' and is also poor form.

Is anyone else reminded of Leafyhost about now?

Am I the only one reminded of the Eve Investment Bank?

http://news.softpedia.com/news/Eve-Online-Economy-Suffers-70...

Does anyone know if they disclosed the address at which they stored their bit coins?

There's a reason real banks use mainframes, not cloud services...

This is not why they use mainframes instead of the cloud. This is why they employ competent sysadmins.

Other thread about same subject here: http://news.ycombinator.com/item?id=2828091

The time seems right for the first bitcoin insurance company.

Only problem. Who would insure a data file of unknown value?

That's not really a problem. You'd insure individual bitcoin addresses, and the amount of bitcoins assigned a bitcoin address is public.

The real problem is figuring out whether an address has been actually stolen, or whether the user is indulging in a spot of insurance fraud.

And nothing of value was lost.

I can sell my BitCoins immediately for about $250 (what I originally put in). How can you sit and say it has "no value"?

Theoretically you can sell them, but what if nobody's buying?

Theoretically you can work to make money, but what if nobody's hiring?

Seriously, was that your best argument?

Supply and demand. Right now there is a demand for BTC which means people can supply them for a cost.

Some people believe that price and value are the same thing. Others, not so much. It's not inconceivable that Ted here holds the latter opinion.

I don't really understand the difference I guess. Does Ted like to do something else with his dollar bills that I don't want to or can't imagine?

> At this point I wish to inform and assure you that your cash deposited into your bank account service and not converted into BTC and unpaid cash from the sale of BTC remain safe and intact

The funds are likely held in aggregate in a single bank account under a single name. Without having the data showing which accounts hold which amounts, I wonder how those funds could be returned to the rightful owners.

It sounds like they lost the bitcoins, not the database.

Ha ha! A fool and his money are soon parted. Anyone stupid enough to waste real currency on bitcoins deserves what he gets. Bitcoins are the modern equivalent of speculating on "limited edition" Beanie Babies. The truth hurts so I expect all the bitcoin fanboys will downvote me now. Go ahead do your worst. :-)

Reply on news.ycombinator.com