Coinbase down for maintenance
coinbase.com
coinbase.com
I am not excusing their behavior as they need to be able to handle it, especially when their server status affects the bank accounts of real people. But I feel that it is understandable with out condoning it. The simplest explanation is likely correct in this case.
I don't think many of us could have predicted the volume and interest in BTC a year ago and neither did Coinbase's purchasing and infrastructure teams.
I think I would have predicted that they'd need to run at a very large scale, and be able to handle massive spikes. Whether or not I could have convinced them to make the investment is another story.
It's a gamble no matter what, but moving too quickly or moving too slowly results in the company going under.
In that case, I think straddling the line of "overwhelmed when there is a big spike" is a good spot to be in all things consitered.
Currently they are signing people up at a dramatic rate despite their server issues. If they start laying people off because they spun up to fast they could start a panic that would kill them.
They have to weigh all of this in when deciding on an infrastructure spend.
I would also like to think from a purely personal level that they don't want to hire and fire people like that as these are real people with real families involved.
Coinbase has been my recommendation to anyone new to the scene who wants to buy/use bitcoin.
And obviously once finished, move all funds to paper wallet.
Scaling is a solved problem. You have AWS, Google Cloud, and MS Azure to choose from with financial institution/HIPAA-level offerings.
By most engineers in the field it is a difficult problem with ongoing work.
Also, your system gets more and more complex as you scale. More monitoring systems needed, more attack vectors to watch for, deployment systems, backup systems... Imagine the security level coinbase needs to have. It's somewhat unprecedented since a hack could make off with a lot of valuable assets in an instant. (even just from their hot-wallet)
You saying scaling is solved makes me question what you think their infrastructure looks like. It's not just an AWS loadbalancer with some shards behind it. It's A LOT more complex than that.
Are they not using cloud, which should enable easy scaling? Or is their architecture not designed for good scaling?
Cloud solves everything guys!!! Yes buy more Cloud now that's 10% down in the exchanges lol...
Most distributed systems don't really scale ass you add nodes beyond a factor of 10 to 100. If they did, you wouldn't need to hire super expensive people to handle those systems.
It's absolutely bonkers amounts of growth!
Edit: more concrete numbers:
In January of 2016 they had around 2.6 million users.
On November 29th they announced they had over 13 million users.
I don't know how much computing power is actually needed when you're just buying BTC. When I used Coinbase recently to take a small BTC position, I only made maybe 4 transactions in total. Is this really a lot to handle?
I'd imagine you can't use scaling tricks like eventual consistency or simple sharding here. Traders are going to be pissed if their most up to date information from the exchange is a second behind what they can see.
It's already at the point where they have a line in their FAQ for the GDAX API that tells people where to colocate from for the lowest latency.
And all of that data needs to be stored damn-near indefinitely, and securely. And integrated with traditional financial systems.
I'd love to see some information on what it actually takes, as i'm just guessing here, but seeing as there aren't any bitcoin exchanges that can seem to keep the servers running smoothly during spikes, i'm guessing it's not a trivial problem to solve.
I'm not saying it's impossible, but the amount of growth they have seen is substantial and unprecedented. Nobody could have predicted it, and the rate of increase just keeps going up.
They are clearly new in this space, as all cryptocurrency companies are. I don't think that means that they should just give up and go home because they couldn't keep the servers up during spikes of insane activity...
And a lot more operating capital, too.
They have to scale at the rate their revenue or financial backers will be able to sustain. Based on the number of new users signing up to the service the down time is not yet a significant deterrent. If use drops because of downtime they will likely need to reevaluate a large capital spend to solve the issue.
Given the recent increase in news about cryptocurrency I've expected "technical difficulties" with Coinbase. Heck, most mining pools are sweating bullets right now.
Hypergrowth = hyperpains.
hypergains = hyperpains
At least with the Cabal theories you know someone knows what's going on. You might even have someone to blame/jail at the end of the tunnel!
Therefore he thinks the people exchanges are either part of the "flash-crashes" (and buy up assets themselves, because they know exactly where most of the stop-loss orders are placed), or they get kickbacks from the whales manipulating the market, by giving them access to the database with all the limit and stop-loss orders.
Stop-losses are very important in investing but I'd recommend always executing them manually at market rate. Set a price alert to notify you. Of course if the exchanges are down you're sol, but if BTC is on a bullet train to $10 your exchange stop-loss probably won't fill at your price anyway.
No, no, no, no ...
You do not ever want to insert a market order[1] ... the market can be manipulated and high frequency traders can buy/sell your public market order at an artificially low/high price.
The answer to getting "stopped out" is not to insert a market order - it is to create a fill-or-kill at a specific price or other more specific order type ...
[1] https://www.marketwatch.com/story/this-is-why-you-never-use-...
Getting "stopped out" is fine, it's part of investing. You need to stop your losses. I'm just saying don't put the order on the books for the exchange to pluck like a ripe berry.
I'll go out on a limb and say that much of the recent spike can probably be attributed to the 100k++ accounts open on Thanksgiving weekend getting funded today and everyone losing their shit mashing the Market Buy button which get instantly executed. The mechanics of vast market buy orders with no liquidity is well known.
Fractional reserve banking is a prisoner's dilemma, we all marginally benefit from the status quo of credit/debit/interest, but in a crisis, the first people to pull out win, and everyone else loses.
It stores your shares for you, but it’s not loaning them out. The only way you get money out of Coinbase is if someone buys your shares.
1 BTC at Coinbase is only worth whatever Fiat currency Coinbase is willing to buy it from you for. If that's $1 for all your Bitcoins, tough luck.
They're under no obligation to guarantee a sale price for BTC.
Then how do they pay you when you come to withdraw your cash?
Bank runs are how fraudulent exchanges are caught: Bitcoin is self-auditing, and either you have it or you don't, there aren't IOUs or paper receipts.
Fractional reserve banking is a prisoner's dilemma, we all marginally benefit from the status quo of credit/debit/interest, but in a crisis, the first people to pull out win, and everyone else loses.
If there's a run on the bank, the little people are gonna get screwed, no doubt about it, no matter what store of value we're talking about (seashells, gold, paper currency, cryptocurrency, gold-pressed latinum, etc).
https://engineering.coinbase.com/november-29th-december-1st-...
e: actually, tbf, it was a buy and a sell:
https://blockchain.info/address/366k4bqibNrbUyby4xtkDT1demdc...
But it does demonstrate the market can just about handle large transactions.
https://www.reddit.com/r/Bitcoin/comments/7i7mix/great_gdaxc...
Can we change the link to https://twitter.com/coinbase/status/938802155925913600
Not me. Detailed status pages are awesome. Visibility into the service is awesome.
However, financial institutions are not currently obligated to maintain their clients' ability to transact those assets, let alone within a specific time-frame. Many people have sued their brokerages for not letting them sell stock in a timely fashion to take advantage of price changes; essentially all of them have lost or settled for pennies on the dollar.
Is this going to end in a re-discovery of exactly why we have FDIC? Or the SEC? How do people even know how much bitcoin activity is real and not essentially faked?
Popular Google Searches by year, by my estimation.
2016: "what is bitcoin"
2017: "what is ethereum"
2018: "what is a wash trade"
2019: "can you eat squirrel"
Define fake
Essentially, if you interact with banks and have too many chargebacks or complaints, you'll be dropped. Or your fees will get massive. And that's if you find a payment processor who's happy with you taking money for cryptocurrencies in the first place.
And when the whole thing blows up, I disappear. My payment gateway will drop me, but what do I care?
Oh, you copied and pasted it from yesterday...
(I actually originally tweeted it, like two weeks ago, but no one reads my twitter, which is to their credit.)
[0] https://www.cryptocompare.com/mining/calculator/btc?HashingP....
The only effective circuit breaker is some form of synchronized method, but will the different exchanges cooperate? They really should put in circuit breakers this week across all exchanges so that when really bad stuff the panic doesn't destroy tons of people. It will be bad, but at least if the trades can complete and things unwind in a moderately slow fashion, it won't just 100,000 attempted transactions per second with random successes.
In the situation with ETH, a bunch of automated margin calls happened because of one large order on the books, dumping the price down to 10 cents! GDAX/Coinbase repaid a lot of people who lost money then, even though they didnt have to. They may now have some dumb heuristics that slow down trades / whatever if a bunch of large orders are going through, or price delta over a short time period goes too high. I remember looking at the charts today on GDAX and the 5 minute candle went from 20k open to 16k close before trading halted! I also heard from others that there were a lot of ~19k sells of 100 coins each on the books and executing.
So yeah, just speculating that this may bave been the case of someones boss at GDAX/Coinbase telling engineering "please don't make us do that again", after they ate the cost last time! :-)
"Decentralized! Deregulated! But coordinate to protect us from ourselves, don't let us lose money!"
That said, while I don't think the exchanges need to (or realistically could) coordinate on circuit breakers, I do think it's entirely reasonable for there to explore this type of mechanism on a per-exchange basis in the case of GDAX and other exchanges where margin is allowed, to prevent the whole "cascading margin call" thing that happened with ETH to happen again. That could take a lot of forms, including limiting the size of market orders to not eat 30% of the book, or not allowing 1 market order to trigger margin calls if (insert criteria here)
(That said, they could also just say "that's how margin and trading goes on low-volume, highly volatile assets" and technically be correct, but much like their decision to reimburse some traders, this is more of a business decision and wanting to keep your customers. Not sure what the "right" answer is.)
Assuming they're adding thousands and thousands of customers quickly, I'm not surprised that their systems and support are lagging. Comes with the (Wild West) territory.
With a cryptocurrency there's nobody to call, you have to convince 50% + 1 of the mining pool to undo any major breach - which is almost certainly never going to happen.
That being said, anyone care to elaborate how the price of bitcoin is determined? I mean it's easy when there is a large central bank monitoring this figure. However, how exactly does it work for bitcoin? What makes it worth the $15K value when there is no "goods" to back it up. Are there anyone here buying bitcoin as an investment?
So to give you a little more info: 1) Yes, there are big players (aka "whales"), and the rest of the "small fish", all playing in the same pond. The interesting thing about cryptos is that the are more "money-like" than "stock-like" when it comes to trading. Meaning just like money, where I can buy a fraction of a $, I can also buy a fraction of a bitcoin. So there are a lot of smaller players who will buy, say .01 BTC for $150 right now.
But while there is a general consensus that bitcoin is $15k by virtue of people paying that much for it, the "whales" (people with a lot of money or coins), can cause issues because they can put a lot of volume on or take a lot of volume off the order book at once. This happened to Ether this year, dumping the price for a few minutes from >$300 down to about 10 cents for a few minutes! https://blog.gdax.com/eth-usd-trading-update-5d8142b5bdc1
In fact, there is some speculation that a lot of the rise right now is from Wall St. / big firms buying up a lot of coin, as futures trading on Bitcoin is coming in the coming weeks! (But I don't know enough to comment on that theory)
As far as preventing exchanges themselves from colluding... it would be difficult but not impossible, because again the price on the exchanges comes from the traders, not the exchanges themselves. (Theoretically it is possible that an exchange could do a bunch of "wash trading" / make up a bunch of fake trade volume to make demand look higher than it was, but given that there are a number of exchanges in a number of different countries, coordination there might be difficult)
We don't really know.
>Or is there a general consensus that the price is $15K. And wouldn't there have to be people that we willing to pay this price? What prevents the exchanges from colluding and raising the price artificially?
Well the exchanges could be running buy-bots but I think the more likely collusion is going to be big-players selling to themselves, if a big player is filling up both sides of the order book they can make it look like there is a lot of activity and drive the price up that way. Now is this that or is it organic? I'm not sure we have a good way to tell.
A lot of the market movement is caused by people setting fixed prices on their transactions. Eg: Sell 5 BTC at X Or buy 5 BTC at Y. If X or Y is never reached the transaction never takes place.
Generally the exchanges just connect a buyer to a seller. You are not buying the exchanges coins but rather the coins of another individual/trader. The exchanges are just middle men facilitating the sale.
The exchanges make money on a per transaction basis on fees. If the market grinds to a halt because there are no transactions the exchanges make no money.
Now there are many possible reasons for the recent spike, some malignant such as potential pump n dump scheme or artificial inflation due to the upcoming futures exchanges and the desire to profit from shorting and some benign such as all the media attention and fear of missing out or just genuine belief in the potential of BTC as a currency or asset.
No idea which reason is accurate, could be all of them or none.
- "Market Maker" - By placing an entry in the order book, which says something like: "I have up to 100 Bitcoin I'm willing to sell at $15,000.01 a piece", or "I'm willing to buy up to 100 bitcoin at 14,999.99 a piece".
- "Market Taker" - You can do this by looking at the book and saying: "I will take that deal". You can also do a "market order", saying: "I want to buy 100 BTC at market price", in which case, the exchange will match you with as many sellers as it takes to fulfill your order of 100 BTC, working from the lowest offered price on up.
The price is determined by either whatever the last trade executed at, or a weighted average of the buy side and sell side of the order book. If you hear someone talking about the spread, that is the difference between the highest buy price offered on the book, and the lowest sell price offered on the book. (2 cents in my above fictitious example).
The exchanges make their money by running the order book, matching makers and takers, and charging a small fee for the transaction. (On GDAX for example, market makers get charged no fee, market takers get charged some fraction of a %). The exchanges make money on the volume of trades going through them.
It can get a bit more complicated, but I hope that gives you an idea!
From the exhanges POV this is useful because more orders = more money for then.
From the market traders POV (the taker) this is useful because a lower spread and more volume means there is less chance of buying something they can't later sell.
For the market maker, they make their money on volatility (aka the movement if the price). If they think the price is going up, they will offer to buy at a price higher than the current highest buy offer. This reduces the spread and increases the odds of someone taking their offer. They will then later sell those coins, and make a small profit on the movement.
If they think the price is going down, they will place a sell order cheaper than the lowest sell, and then buy back later, making small profit whole keeping the same amount of coin / stock / whatever.
Basically a "market-making" trader is less in it for long-term, and more interested in making money on the movement.
Aside: You may have also heard of "short selling". This is when a market maker is betting the price is going to go down, but doesn't have anything to sell. In a short sale, they will borrow the stock/coin from someone else (such as the exhange or their broker), and sell it, promising to buy it back later to repay the loan of said stock / coin. Easy way to bet on perfornance of an asset you don't owb, but dangerous since if you are wrong and price goes up, the person who loaned it to you will want you to repay it back sooner rather than later, costing you money!
there are many orders that are above or below the current market price. those wont get cleared until the trades that are closer to market clear, and they may not get cleared at all if the market moves the other direction
But coinbase goes down for maintenance all the time
* ID verification by either webcam or id pic upload fails, and eventually one reaches the daily quota, locking the feature for 24 hours (seems fair though)
* removing and re-adding bank accounts fails, the site says the account is already linked and it shows some ghost duplicates (i.e. the same bank account multiple times with different statuses)
* bank account verification fails, the site says the small amounts deposited don't match
* some buying operations fails due to API timeouts when collecting money from linked bank accounts
What are the other exchanges that you can trade BTC and are they more reliable than Coinbase?
If one exchange implemented circuit breakers, people would simply move their trading elsewhere.
It seems Coinbase struggling to cope up:
$0000 - $1000: 1789 days
$1000- $2000: 1271 days
$2000- $3000: 23 days
$3000- $4000: 62 days
$4000- $5000: 61 days
$5000- $6000: 8 days
$6000- $7000: 13 days
$7000- $8000: 14 days
$8000- $9000: 9 days
$9000-$10000: 2 days
$10000-$11000: 1 day
$11000-$12000: 6 days
$12000-$13000: 17 hours
$13000-$14000: 4 hours
$14000-$15000: 10 hours
$15000-$16000: 5 hours
$16000-$17000: 2 hours
$17000-$18000: 10 minutes
$18000-$19000: 3 minutes
EDIT: You currently need $70m to bump the market above 20k on GDAX. That's a lot for retail investors but not much if institutional investors get involved.
We're talking probably low 2-digit millions to make big moves I think.
Almost none.
If there are just a few transactions at a price, well, we assume that is the worth of all bitcoins in existence.
Which is a crazy assumption, but hey.
Almost always you see minor movement on the order book except maybe sometimes at the actual mid point it may update a bit. The chart is almost always balanced until the past few weeks it's be HEAVILY weighed towards the sell side.
71K+ BTC had been already traded by the time it hit high 19k.
All you need is back and worth trades in th market. It can even be market manipulation where some buy and sell between themselves.
Bitcoin has massive market liquidity problem. Once people want to sell, the price drops like a stone. This is why it would be convent to have "technical problems" or do DDoS attack to prevent sells and allow markets to calm.
The insane rise in the valuation of Bitcoin. How many times we have seen a currency or an asset that represents this kind of momentum? Clearly no one has seen this coming (that soon).
Also, the move to $19,000 seems to have been some kind of liquidity issue on GDAX, the price did not get that high on other exchanges.
$0000 - $1000: 1789 days
$1000- $2000: 1271 days
$2000- $4000: 85 days
$4000- $8000: 96 days
$8000- $19000: 20 days
Edit: Even on a logarithmic scale, bitcoin is experiencing exponential growth.
Here is the same data with (approximated) percentage moves:
$0000 - $1000: 1789 days
+100% ($2000): 1271 days
+50% ($3000): 23 days
+33% ($4000): 62 days
+25% ($5000): 61 days
+20% ($6000): 8 days
+16% ($7000): 13 days
+14% ($8000): 14 days
+12.5% ($9000): 9 days
+11% ($10000): 2 days
+10% ($11000): 1 day
+9% ($12000): 6 days
+8% ($13000): 17 hours
+7.5% ($14000): 4 hours
+7% ($15000): 10 hours
+6.5% ($16000): 5 hours
+6.25% ($17000): 2 hours
+5.8% ($18000): 10 minutes
+5.5% ($19000): 3 minutes
Anyway, it's more the volume that is of importance here than the price when it happens.
Actually, those in the Bitcoin economic space have been saying this was going to happen for ~3 years
Hyperbitcoinization - Daniel Krawisz - March 29, 2014 http://nakamotoinstitute.org/mempool/hyperbitcoinization/
2 9s, or 99% uptime? Or is it worse than that?
During major price swings? About 10% uptime at best. They're almost always down when the price jumps or drops.
> Star burns through a succession of nuclear fusion fuels:
Hydrogen burning: 10 Myr
Helium burning: 1 Myr
Carbon burning: 1000 years
Neon burning: ~10 years
Oxygen burning: ~1 year
Silicon burning: ~1 day
I hope Bitcoin's end is not as violent. :/
[1] http://www.astronomy.ohio-state.edu/~pogge/Ast162/Unit3/supe...
These past couple of days, whenever a coin soars a ridiculous amount, on any given exchange, that exchange seems to mysteriously go down. For example: Binance and Bitfinex were down for "maintenance" (or just plain down), while IOTA was experiencing a surge in price (from ~1.10 to 6$). Similarly this happens with Bitcoin today (it seems Quadriga had some issues and Coinbase; not sure about others).
Something is up, and it's weird.
My conspiracy theory: big money institutions are testing pump and dumps for when short contracts come into play. On the one hand, the cryptocurrency world wants to remain as unregulated as possible; on the other, a lot of individuals are going to get reamed by big whales.
Add to the fact that people are also DDOS websites (doesn't take a "big money institution" to do that) and it seems the explanation is clear...
So yeah, I find these weird circuit breaker patterns suspicious. The worst part is "partial" degradation, when you can't execute trades, but someone else can. It's very easy to hide all sorts of irregularities behind these partial outages.
I'm sure they would pay you a lot of money to solve their scaling issues, if you were so inclined.
I'm much more inclined to believe the downtime is caused by scaling issues rather than some big conspiracy,
EDIT: Just to give some insight, one of the problems we run into is that usually only one issue is exposed during an outage. For example, lets say we have 10 functions (or services or whatever) in a row that all call each other. All the odd numbered ones have issues where they fail at scale. During the first downtime, function one will fail and start to throw errors, but this means the full traffic load isn't reaching the rest of the functions. By the time the problem is solved, traffic has usually died down, so function 1 is fixed, but 3,5,7,9 still have issues that haven't been exposed.
Additionally, at a quickly growing company like Coinbase, the code isn't going to stay the same for very long, so even once those functions are fixed, new issues are introduced.
It's easy to say they should be load testing at 10x their expected load to prevent these type of issues, but it's really hard to replicate production traffic. Users do weird and unexpected shit that can cause problems.
It's funny how quickly people jump to overly-complex or borderline conspiratorial explanations when the obvious is staring us in the face...
Coinbase might be a large well funded organization but it's still run by developers with giant backlogs of tickets and conflicting priorities to build new stuff vs maintain old stuff.
This times 1,000.
As the bitcoin market grows GDAX, Bitfinex, etc are probably going to have to borrow some ideas/talent from NASDAQ & other big electronic exchanges to stabilize things.
Shouldn't an exchange be able to handle volume? Seems like a basic requirement.
Regardless, I still believe that with the introduction of shorts, were going to see bigger swings.
They're not all shitty me-too Rails apps built by junior engineers who figured "how hard can it be?". But most of the ones I've seen are.
But what levels of volume? 100 quadrillion trades per nanosecond? Sure they could scale their infrastructure to those levels, but why? You overprovision, get estimate your worst-case scenario and work towards that. But in the end, if there is a huge rush, like a 10000% increase or more, even the largest exchanges would break down.
Exchanges have incentive to handle as much volume as possible to make as much via fees as possible, but no site can handle infinite volume. The coinbase app has been climbing the iTunes charts into the top 20 the last couple days. Just before the site went down the price was running up by hundreds of dollars per minute, attracting tons of trader traffic trying to profit from it. Even if they have basic scaling measures in place for the basic infrastructure, they most likely experienced unprecedented exponentially increasing volume that was inevitably bound to break something
I've seen exchanges running 10 or 100 times that amount with no problem, ran by only a couple of graduates out of school.
Either way, doesn't matter. Exchanges have no performance problem, it's a big queue, you just queue requests.
I'm sure as they grow they'll harden their systems and/or do a total rewrite. In fact I wouldn't be surprised if they're doing it already since the big boys are starting to show interest with futures.
Exchanges are just websites like any other. They're build to handle a certain amount of traffic based on the average amount plus a little extra to handle spikes.
If they only get (And I'm pulling numbers out of my ass here just to make a point) an average of 10 trades per second with occasional spikes to 100 trades per second, how much infrastructure should they build? What capacity should they build to? 1,000 trades per second? 10,000?
There's a cost analysis involved. I think it's a bit unrealistic for them to have enough infrastructure to handle spikes an order of magnitude higher than their typical peak loads.
Big-money institutions won't do this. Random individuals taking advantage of the fact that lots of value rides on relatively little volume? Much more likely. Combined with the fact that the exchanges, intentionally or unintentionally, lock liquidity when volatility spikes and it's 1907 all over again.
[1] https://www.investopedia.com/terms/w/washtrading.asp
Disclaimer: This is not securities advice. Do not buy or sell anything based on this Internet comment.
So the listing of Bitcoin futures on the Chicago exchange might not be a good thing for the Bitcoin price at all.
Attention from price surges increases visitor traffic, and exchanges are understaffed or unprepared for the 10-100x increase in traffic volume. There's no need for complicated stories, evil corporations, or any of that nonsense. Stick with straightforward cause and effect.
Seriously though, it's worth considering that this kind of thing could be possible even if we don't know whether or not it has anything to do with these specific outages.
These services are like banks, but the traffic they handle is more like a news site, due to the extreme volatility of cryptocurrencies. Super-security combined with extremely large traffic fluctuations.
Whoever is on PagerDuty at Coinbase these days - I feel for ya.