It was not a DDoS: MtGox got 20k new accounts per day | Hacker News Reader

It was not a DDoS: MtGox got 20k new accounts per day

bitcointalk.org

124 pointsspeeder102 comments

If someone did a little bit of analysis on MtGox they might have concluded that setting up new accounts was the most effective way to do a DoS attack. Not that I'm saying that's what happened, just that it seems naive to say "we were down due to new account creation, and therefore we should rule out the possibility of a DoS attack!"

The MtGox employees talk about 10 Gbps of traffic hitting their front-end servers. They are desperately working with Black Lotus and Prolexic to help. That is a textbook network DDoS. "Creation of the accounts" is not the source of the slowness and unresponsiveness of the site.

https://bitcointalk.org/index.php?topic=166578.msg1737375#ms...

They are already using prolexic (check DNS to verify) and it sure looks like DDoS to me.

But saying that's what it was is a great way to keep the bubble going.

There is no way that 20K database inserts in a day should affect them in this way. They likely experienced a DDoS and either didn't want to say it or didn't recognize it and actually believed it was because of their 20K new users. Either way, given the issues they have had and the ~$1.2 billion market cap loss just from this one incident, it begs the question: Should this company be at the center of something as important as Bitcoin?

These types of issues have been faced and solved by myriad other companies, many of them with fewer resources. It seems that every major player in the Bitcoin space has trouble conducting themselves as actual professional businesses and are just fine with causing losses ranging from thousands to billions of dollars in this case. If just a few decent companies came together to provide the services that Mt. Gox and other current "leaders" are failing to provide, Bitcoin could quickly become a much more powerful force than it already is. The limiting factor for Bitcoin is no longer demand; it is the lack of reliable infrastructure for buying and selling them.

"The limiting factor for Bitcoin is no longer demand; it is the lack of reliable infrastructure for buying and selling them."

There is a big demand, I don't understand why VCs or YCombinator are not pushing a few startups in this direction.

see https://coinbase.com/

I know about coinbase and use it, but it's very limited. U.S. only, is not a real exchange (buying bitcoins is often not possible), etc. And quite frankly, support is lacking too.

Buying bitcoins is still extremely complex for most of the world. It needs to be simplified further.

Coinbase is probably in it for the long run, and they are probably doing things to enable them to outlast the competition.

However.

On Monday I was planning to place an order for bitcoin on Coinbase and I was informed that the order would go through on Friday at the market rate at that time. That is somewhat worthless and extremely dangerous in a volatile market. For now, I consider their service unusable.

It's been that way for a while. I'm hoping they change the way that works, as right now the only way to guarantee you'll get a buy order in is to use a tool like coinbase-trader(https://github.com/martindale/coinbase-trader), literally polling their API until it goes through.

Coinbase with its ACH stuff isn't really that workable. It takes too long. The opportunity is there, but Coinbase isn't meeting it. Bitinstant has turned into a Ponzi scheme and is well on its way to failing. If Coinbase or someone else started accepting Moneygram and cash bank deposits, they could quickly capture a huge pot of new business. On the selling side, no one is faster or more reliable than FastCash4Bitcoins.com. If someone could perform like they do on the bitcoin buying side, they would win a ton of business.

IMO, the limiting factor has been the amount of people willing to sell bitcoins. So the price ended up going up and up and up.

And now that we've reached a point where the value of it is so unrealistic (determined by because everyone is so drastically delusional about it's value and also the causes of the given events -- as your post and other comments demonstrate), those same people are now willing to sell / cash-out, and others will follow their lead.

> There is no way that 20K database inserts in a day should affect them in this way

I doubt its the 20k new accounts. It's probably the 20k new accounts' (per day) trading activity.

1) They should put a captcha on their sign up form. Bitcoin has gotten significant press in the pass week, but not enough to bump registrations by 10x (60k in March alone to 20k per day)

2) "The number of trades executed triple in the last 24hrs." But how much has the volume changed? Is there a way of somehow filtering out what would be "panic" induced trading (perhaps by removing # trades that happen in a large swing) to see if trade frequency has changed?

Volume really jumped up - eyeballing it, conservatively, about $15mm in the last 24 hours of trading volume:

http://bitcoin.clarkmoody.com/widget/chart/

> Bitcoin has gotten significant press in the pass week, but not enough to bump registrations by 10x

I don't know, well publicised offers of free money might actually result in quite high uptake.

http://i.imgur.com/cGjIS9x.png

There's definitely a great deal of automated trading happening. I suspect it's less DDoS than ad-hoc HFT bots, though.

I think these are done in bursts specifically to cause lag. Nobody would bother to buy 0.1 BTC in 100 separate orders at the same price. Often the orders are cancelled anyway.

Actually, I believe this is bitcoin washing. Its money laundering of sorts.

Won't you get destroyed in fees doing this?

Which is more, paying x% on 1BTC of x% a hundred times on 0.01BTC?

I'm not familiar with the cost structure, but if there were a per-transaction cost then it would be:

which is more, paying x% + y once on 1 BTC or paying x% + y a hundred times on 0.01 BTC?

Clearly the latter is more, assuming of course that `y` is positive (and it would be a very strange market that inverted the structure)

On Bitfloor, you actually get up to 0.1% rebate on limit orders. From the docs:

"Any order that is on the order book will receive a rebate when filled. Think of this as a negative fee (you actually get extra funds). This is done to provide an incentive for some traders to place their orders onto the order book for others to better understand market conditions."

Same thing happens on the equity markets, with the difference being pocketed by the exchanges (they'll pay .29 cents per share to liquidity providers and charge .30 to liquidity takers)

Mt. Gox I believe has a flat percentage fee on all trades so someone can make as many micro trades as they want and still pay the same fee as one large trade.

It's pretty easy to solve by charging users who make over a certain limit of micro trades. I wonder why they haven't done this?

This actually seems like a large flaw in their fee structure. Every traditional online brokerage/trading service that I have looked at in the past has used a flat fee. This incentives individuals to reduce their number of transactions in order to limit the overall fee's paid.

Dear quandryfoe

My name is Danjuma Sule, one of the sons of major Gen Gumel Danjuma Sule, The late Nigeria's former minister of mines and power in the regime of the late former Nigeria's military Head of state, Gen Sanni Abacha. When my father died he left me an inheritance of 9830422.33333421 bitcoins. Unfortunately bitcoin exchanges are not yet set up in Nigerian currency and I am in need of a young techno wizard with a bank account denominated in US dollars to assist me in gaining access to my inheritance.

It is on this basis I am seeking for assistance. Your percentage is negotiable. Please note; your age and profession doesn't really matter in this transaction. Waiting for your immediate response and bank account specifics.

Regards, Danjuma Sule

Why is your name in green?

"Noob"

Edit: I mean green equals new account.

It's an 'HN Verified' account; this person has passed all identity/credit checks and may -- nay, should -- be issued convertible debt at will.

How can I get one of these "HN Verified" accounts? Are you the person in charge of verification I should be sending my credit card information to?

From the same place you get 4chan gold accounts.

Next to the pool on the roof.

If you're creating a new account because you KNOW its going to be downvoted, isn't that an indication that you just shouldn't make the comment?

Maybe he was just committing to the joke.

Similarly downvoted him.

HN is a serious place for serious topics. We should be talking about bitcoin.

I can't tell if you're being sarcastic or not...

It's actually a very wonderful comment.

The ironic thing about this comment is that, unlike with a USD bank account, I would be happy to give someone my public bitcoin address.

I imagine sharing your IBAN or local equivalent does not imply any significantly larger danger than sharing your bitcoin address, am i missing something?

The scary thing is that there are no checks on withdrawals from your account. Anyone with your account number (which is available on all checks) could in theory withdraw money from your account.

That would depend on the bank and country. In the UK, for example, you can set up a direct debit and withdraw funds directly, using just the bank details. But you would need to have a UK bank account to do so, and they would presumably not just allow anyone to start running direct debits.

It is not as easy as it sounds. Those kind of withdraws (direct debits) needs special accounts to start with, and can be cancelled talking to the bank (during a "grace period", typically 2 weeks). The process to determine if that withdraw was legitimate is done with the money on the original account. If you have more than normal rate of "rejections", your special account will be cancelled, and could be difficult to run direct debits again...

The specifics can change from country to country, of course. I know that in the US direct debits are rare (while in Europe are quite common), so maybe the process changes.

Anyway, your bank account number is still sensitive info, as there are scams that can challenge the whole process (generating lots of direct debits, or trying to duplicate legitimate ones to go "undercover")

I'm impressed by how many commenters know the cause of someone else's infrastructure problems with such certainty.

Are you new to technology? This is the norm. Everybody is the smartest person in the room, regardless of how little they know about the details of the implementation.

Déjà vu.

I find this hard to believe... mtgox, bitcointalk, bitcoincharts were all just experiencing "too much traffic" today?

bitcointalk has a massive donations budget, and we all know mtgox has a lot of money and a huge interest in preventing something like this from happening.

I think it was a DDoS combined with a few opportune sells to create a panic on purpose to be quite honest.

Tinfoil hat replies welcome.

Theymos (main bitcointalk admin) is seeing very high traffic to the forums: 250 requests per second. He thinks this is legit traffic, not a DDoS: https://bitcointalk.org/index.php?topic=172672.msg1799471#ms...

Then what is the reason for the mass panic/selling?

Mtgox the exchange (not Bitcointalk the forum) is under DDoS which caused the panic selling.

The crushing inevitability of Bitcoin's failure?

It briefly crossed the 200€ mark yesterday. Maybe a lot of people wanted to cash out at that point. As soon as the price was dropping, more people decided it was a good time to sell.

While it was dropping, there were two ways you could expect it to go. Either it was dropping and was going to stay low, in which case you should sell, or it would rise back up in which case you should sell and then buy when it got cheap.

250 requests per second is high?

Maybe they should start to use an SSD and some decent software.

250 HTTP request per second of static files is peanuts (heck, my website can sustain 10 thousand). But 250 HTTP requests affecting dynamic content on bitcointalk.org is a bigger engineering pb: the average page is 100kB of dynamically rendered HTML supporting per-user theme, and most HTTP requests modify content (forum post) or are expensive requests (post searches).

Theymos has been improving the forum software, and is offering bounties to people to implement certain things.

I moved all my bitcoins over to BitStamp (https://www.bitstamp.net/) earlier today. MtGox has always felt a little unreliable at multiple levels. They sound earnest in their post mortems, but it's hard to tell if they are sufficiently competent with web technology to be doing what they're doing.

you did very well. mtgox doesn't deserve to the first and probably not even the second largest exchange.

I moved to Bitcoin-24 (https://bitcoin-24.com/ - mainly EUR), they seem pretty popular and have no fees for trading.

Bitstamp is no better. If anything, the (comparatively) low trade volume makes it very dangerous. Yesterday, I put in an instant (market) order at about 5pm. It was fulfilled in about 2 hours, for $50 dollars less than the price was when I put it in. I would expect for market orders to be either fulfilled or canceled immediately, not linger pending for 2 hours.

Market orders are relisted every ~10mins at the market price until fulfilled. If you want to sell at no less than some $X, that's what limit orders are for. You used a market order when what you wanted was a limit order; that's not Bitstamp's fault (unless the fault is in making it clear what the difference is, but IIRC they do explain in the FAQ).

Bitstamp was down for a large portion of the crash yesterday as well.

Magic The Gathering Online eXchange (MtGOX) refitted to trade a virtual currency which then had to scale massively in a short period of time.

Yeah, I'd say there are serious architectural problems. This is not a software "design" problem though, it's the inevitable real-world result of software "evolution". It happens.

I read they're working on a new platform built from scratch. I bet it's going to kick ass as they've got some real world experience now about what works, what does not and what needs to be hardened.

Bitcoin needs a high performance, extremely secure, open source trading engine, with a standardized API that traders can use to arbitrage the exchanges.

The tricky part is transferring other currencies in and out, and between the exchanges. "Ripple" also sounds promising for this purpose.

There's nothing like that already? When you say "need" do you know it doesn't exist already? I'm genuinely curious and hoping you know the answer.

There was one called Intersago but it looks defunct: http://bitcoin.stackexchange.com/questions/1447/is-there-an-...

I came across this project on bitcointalk, which isn't really an exchange but looks interesting: https://github.com/FellowTraveler/Open-Transactions

Okay, not a DDoS. Let's go with that for a sec. Absurd load? Even when the data feeds are working, they show huge bursts of transactions going through. (A link downthread shows a screen full of tiny transactions being executed)

MtGox is synonymous with 'bitcoin' at this point. Everyone who knows what a bitcoin is knows what that site is.

Are all those 20k accounts legitimate?

To be clear - latency on transactions was at around 600 seconds in the middle of the day - that is, an order to "buy" at the market, or "sell" at the market, took 600+ seconds to execute.

For about 20 minutes, they buy/sell buttons were non-responsive. You could click on them - but nothing happened. I have a suspicion (without evidence other than observation) that at one point today the buy/sell functions were disabled.

More like 3600 seconds for over an hour.

The buy/sell buttons always worked, as far as I know, but when the lag is an hour I can see how it would seem broken.

I experienced two failure scenarios today - they were very different.

Failure Scenario #1 - Buy/Sell Buttons were not working. You would click them - but absolutely nothing was happening. On #MtGoxLive there were a number of us all experiencing exactly the same thing. A few believed it might have been a javascript issue.

Failure Scenario #2 - Once the buttons were activated/fixed - your order was placed in queue instantly, but remained "pending" - on #mtgoxlive, ;;goxlag showed lag time of about 600 seconds - My order went from pending to "executed" exactly according to ;;goxlag.

I consider the first a bug, but the second seems to be the system failing in the way it was designed to.

The NPR Planet Money podcast just released a podcast Tuesday on Bitcoins: http://www.npr.org/blogs/money/2013/04/09/176688096/episode-...

It was the first time I had heard about Bitcoins, and so did many I think!

Considering how volatile bitcoins are, any downtime DDoS or not, affect users a great deal. Unlike NASDAQ that can rollback/shutdown the market, MtGox has to maintain 24/7/365. I dare say their market's uptime is more critical than the regular stock market. I wonder how their stack is setup. Anyone from MtGox?

Well when everyone only trades through one website this is what happens.

I trade at another exchange that's inside the SEPA zone, has a 6x lower fee, and doesn't require ridiculous amounts of papers to verify yourself (which is illegal to request without a permit from the Dutch CBP, but I can hardly object because then I simply can't transfer fiat currency out of gox). The trading lag was also much lower there.

But are the other markets as rabid as MtGox? I think that is why alot of people even put up with the higher fees.

Also when MtGox comes back online in ~3 hours I believe we will be in for some more bot action manipulation, especially with the temporary fee waive now.

> Unlike NASDAQ that can rollback/shutdown the market, MtGox has to maintain 24/7/365. I dare say their market's uptime is more critical than the regular stock market.

You are trivializing the real markets. What makes you think this is something that NASDAQ can just "do"? They had a system problem causing delays on Facebook's IPO, and they're still dealing with the shit-storm from that now. Everyone who had poor executions (or missed executions) as a result of the delay is trying to get compensation from the exchange for their technical problem. I haven't checked the exact number recently, but I want to say $21MM is the pool that the exchange has to puke to the "plaintiffs".

Because bitcoin exchanges are not regulated in the same fashion as stock exchanges, I doubt that MtGox will be on the hook for anyone's poor executions.

> I wonder how their stack is setup.

As someone else pointed out, their stack is setup wrong if account management has _any_ impact on their matching engine. Hopefully, it was just the web servers which host both accounts-management and trading UIs. Regardless, there was clearly some resource-sharing which should not have happened.

the fact that they can have issues doesn't mean they cannot suspend a stock as part of normal operations if they rise or fall too fast. At least, most stock markets seem to be doing it without issues most of the time.

>but I want to say $21MM is the pool that the exchange has to puke to the "plaintiffs".

This is actually back in the news. And the value that the banks are looking to recover is closer to $500MM.

http://www.swissinfo.ch/eng/business/UBS_unimpressed_with_Fa...

Well, MtGox proved me wrong earlier today. :) Matter of fact, still down now.

Also I still think it would be interesting to see their setup, wrong or not.

For comparison, Facebook acquired their first 100M users at an average rate of 60k accounts per day.

Facebook users generally don't intend to transact though.

Especially not in a way where the exact order of each transaction is critical, so there is no opportunity for any parallelism.

So while this is an astute observation, a properly run exchange would go through a lot of effort to separate the matching engine (the inherently single-threaded part of the exchange) from the portions of the system that accept orders and spew data. The registration system and web site also don't belong anywhere they could possibly contend with the matching engine. Real exchanges process orders of magnitude more volume on a single symbol than mtgox does, yet have latencies measured in microseconds rather than tens of minutes.

The matching engine should be separated completely from the rest of system as this is the one component where you want to align with cpu cache and all other kind of crazy low level optimizations. I do agree that mtgox system is slow but we should keep in mind the resource difference both in time and money the large stock exchanges have had time to build their systems. Most of the stock exchanges also don't build their own system but rely on buying one from a independent organization or another stock exchange.

A small nitpick: Volume is one metric but it does not matter in reality the important part is the amount of transactions on a specific symbol.

A small nitpick: Volume is one metric but it does not matter in reality the important part is the amount of transactions on a specific symbol.

That's quite right, thanks :)

Real venues incentivize participants to post orders in round lots (generally 100 shares) and some of them penalize participants who add liquidity with extraordinarily low fill rates. These rules help reduce load on the matching engine, and afaik mtgox has no analogous rules.

All the more impressive.

I never thought it was a DDoS, can't understand why people think this. Lots of exchanges have been down, one even seemed hacked (turned only a related service was hacked), and meanwhile the price only went up.

The price drop is due to panic selling and lag. It was almost inevitable, also constant in the news. Now it happened and we can move on. The price is back to about 75% of what it was already, and I don't think it'll take more than a few months to break the previous price record. Or seeing how fast it's recovering, perhaps not more than a few days.

There was a DDoS, as there has been an on going one for quite a while now and you can find so on the mtgox website news section.

However it was not a DDoS in the classical sense that brought mtgox down to its knees but nonetheless a DDoS has occured.

mtgox supports a weird feature whereby you're allowed to enter any buy or sell order even if you don't have funds. This means that whenever the prices matches your order the system needs to check whether you actually can with your balance fill that position.

Via the api, it is possible to put tiny transaction even smaller amounts than what you're allowed via the webgui.

It doesn't take much review of the ask and bid to find out that there were a huge a mount of micro tiny transactions all over the place. Those are in an ever higher volume when system begins lagging furiously and people panic.

These are perhaps HFT, or maybe not, but those are the source of the lag and therefore the source of the panic.

This is DDoSing and it's also mtgox fault for having a a toy exchange and us bitcoiners for using it.

I long ago moved to bitstamp. You can find your local exchange i'm sure.

It looks like whoever was behind recent events saw this as message claiming it wasn't a DDoS, chuckled and responded by... making a stronger DDoS. MtGOX is down.

The press release did say that they may take the exchange down for a few hours. I'm not sure if that meant unannounced downtime like this though.

It's amazing how what is currently the largest Bitcoin exchange service can be so unprofessional. You would think they could have anticipated this.

Did anyone notice there was a SEVEN MILLION (USD $) transaction yesterday?

http://blockchain.info/tx/5d9ef693d41cb3bb4c6d98e70ea8b2cc91...

Actually, $7M based on $100 value. If it was @ $200, well $14 Million USD.

Maybe some bitcoins being shuffled around in the exchange?

And now, guess what MtGox advises everybody do? Go drink some Champaign! To celebrate their massive success! I'm not kidding-- look at their latest bulletin on why they're suspending trading for the rest of today.

You've got to be KIDDING ME. They don't seem to give two shits about their users or their money.

I've been waiting for over a week to get my account verified so I can start using my account (I'm not in the US), but they only seem to verify a few hundred a day and there are still 5000 people ahead of me.. quite frustrating.

I'm in Australia, it was 4 working days. IIRC The price went up from $60 to about $170 during that time, very frustrating.

At least I'm verified now - but I have to say their lag time and apparent saling issues still concern me a lot.

If you want to run a succesful pyramid scheme, step 1 is find desparate poor people, reddit is a great starting point. Ring up their greed impulse and the rest writes itself.

The hundreds of thousands of dollars people funnel at a time to 419 schemes indicate that there are plenty of desperate, greed-clouded middle-classed persons and higher-classed suckers. I suppose Pyramids/MLMs also work well among the lower classes, but they're hardly limited to them, diversify!

Someone, please, compete with Mt.Gox more. :( Also, don't store your wallets on Mt.Gox. Oh what a target they must be.

In general I agree with the advice to not store large quantities of btc on Mt. Gox, but it's not as dangerous as it used to be a year ago and may be less dangerous than storing them locally in an encrypted wallet... Especially if one uses Windows. If one is using two-factor authentication with Mt. Gox, even if I somehow got infected with a keylogger it wouldn't really matter for Mt. Gox whereas my local wallet would be in trouble. And while we'll see how Mt. Gox responds to user stupidity that leads to their btc being stolen, I'm confident that if it was squarely Mt. Gox's fault (e.g. someone executed a successful social engineering attack on them to get around yubikey), they would make things right.

Real Life Mario Brothers?

Reply on news.ycombinator.com