Avoid Vendor Lock-In Risk with Multi-Cloud Deployments
cockroachlabs.com
cockroachlabs.com
For example, if they use two-phase commit, what is the granularity of locks ( table, row, etc )during an open transaction? Table locks during two-phase commit with, say, 50ms coast-to-coast latency would be an immediate dealkiller.
Or, if it's eventually consistent as most asynchronously distributed DBs are, they can't claim AC.
I really wish Cockroach had better documentation - not so much how to use it, but how it works so as to let engineers reason about performance characteristics.
There's also a bunch of technical tech-talks linked from our company's website: https://www.cockroachlabs.com/community/tech-talks/
https://www.cockroachlabs.com/docs/stable/frequently-asked-q...
Last I heard CockroachDB is like "Spanner without atomic clocks". Not sure if that helps but at least the WAN latency is not an afterthought.
https://www.cockroachlabs.com/docs/stable/frequently-asked-q...
https://www.cockroachlabs.com/docs/stable/frequently-asked-q...
As explained in the blog (as linked by another response) and the documentation (https://www.cockroachlabs.com/docs/stable/strong-consistency... and https://www.cockroachlabs.com/docs/stable/distributed-transa...) CockroachDB does not use locks, and write intents in the concurrency control scheme are expressed by K/V pair i.e. per table cell.
Meanwhile, I acknowledge that CockroachDB does not yet convey well its performance model so that engineers can get an intuition about it. There is documentation, but it's either too detailed (like the design doc) or too high-level (most of the docs site). This needs a new sort of documentation, which we are working on.
> Employs Raft, a popular successor to Paxos.
I don't believe that is correct and that is not a good meme to promote. Successor implies the 'deprecation' of Paxos, which is certainly not the case. For the architectural sweet spot where both protocols can be used, it is fair to say Raft provides the more accessible approach, and that for those who insist on rolling their own consensus mechanism.
Paxos and Raft serve distinct architectural concerns in the shared context of providing a distributed state with formally defined consistency guarantees. And to the point, it is entirely reasonable to see a distributed system that utilizes both to address system component specific concerns.
CockroachDB, ironically (in context of your OP), in fact could benefit from a vendor lock-in were the candidate cloud provider to provide atomic clocks and GPS cards in their offerings.
Literally all GPS does is provide a way to determine the time very accurately in relation to a bunch of synchronized atomic clocks in orbit. (As a side benefit, knowing the time like this will also give you your position.)
Windows for concurrent operations are bounded by clock uncertainty. If you say you did something at 2:00 and I say I did something at 3:00, you probably did your thing before me. But if it's 1:57 vs 1:59, there's strong possibility my clock is fast or yours is slow. If I know both our clocks are within 5 seconds of the correct time, then I can say your action happened before mine. So accurate clocks allow you to provide ordering to more events.
Additionally, If you do identify operations as concurrent, you can explicitly deal with them. Dealing with them is doable, but expensive. If your clocks are in sync with a small margin of error, the rate at which you will have to do expensive things to resolve concurrent operation is low.
Google's Spanner database uses atomic clocks to have a max clock skew of 7ms. That eliminates most concurrent conflicts, and dealing with conflicts is fast because we can recognize conflict quickly. Basically, it waits 7ms on all writes to see if any other nodes have a conflicting write.
CockroachDB can operate in "spanner mode", but without atomic clocks it uses a max clock skew of 250ms. This is a much bigger window and identifying conflicts on write takes a lot longer.
CockroachDB recommends that you don't run in "spanner mode" because of the performance hit. Instead, the default mode does lazy conflict detection, where instead of waiting after all writes, it will sometimes wait to read data until the clock skew window has passed.
Here's a great read on the topic:
https://www.cockroachlabs.com/blog/living-without-atomic-clo...
The "eventual consistency" part is simply acknowledging that we can't beat C when our databases are in North America, South America, Europe, Africa, Australia, and Asia. One-way comms takes time. 2 way comms, even more so.
The tradeoff is, that changes can come in, and 'eventually' will propagate to the rest of the DB nodes. It does sound not ideal initially, however this type of operation would allow a localDB running as AP to sync when back online.
The examples I'm deriving come from recent DBs, like GUN http://gun.js.org/#step1
a) Using "lock in" features like Cloud Formation, Lamba, Cloudwatch, etc, to do so, such that you use some of the "value" of the cloud.
==or==
b) Rolling your own solutions for these peripheral functions (deployment, backup, monitoring, etc) and running them on the bare VPS instances. In which case...why are you on this expensive cloud? You can get a VPS with lower bandwidth costs elsewhere.
Basically being sort of "half in" on the cloud, where you use it as a glorified VPS makes no sense to me. Either be all the way in, or all the way out.
K8s on AWS, at least right now, means you are in the business of being a plumber. For example, if I remember right, the solid k8s ingress controller means ELB, which has issues like the need for cache warming. ALB is better, but has only bleeding edge support as an ingress controller in K8s. Similar for getting it installed on AWS in the first place.
Rackspace or Digital Ocean or similar is a cheaper better place if you want to roll your own k8s solution.
Google Cloud Platform is definitely cheaper than AWS, and their hosting of Kubernetes seals the deal for me. The only problem? Legacy. If I want to link two virtual machines across cloud environments, it introduces somewhat unacceptable latency, and throughput will be less reliable. So connecting to things like extremely busy Redis or AMQP clusters is not really as safe and downtime is more likely.
Still, I don't feel like Kubernetes is terrible on AWS. It works fairly well. I'm not using Ingress resources right now, just a bunch of services with their own ELBs.
Why not just rent a cheaper unfurnished home if you insist on bringing your own...
The furniture analogy is a bit flawed. I can get comfortable in a new, furnished house, but furnishings don't come with vendor lock-in. Cloud services naturally do, at least the highly proprietary ones. I'm not saying put the furniture in the basement, I'm just saying throw a slipcover over it so we don't touch it directly. :)
DigitalOcean meanwhile is a lot more limited. The new stuff they've added is nice with firewalls and load balancers, but the tooling with AWS is more complete and I can utilize a fair bit of that tooling from within Kubernetes, including things like Amazon's certificate provisioning.
Basically to be clear, I'm saying maybe now embracing GCP and Azure seems like a solid plan, but years down the road you might want to have some more mobility. If you're already using things you can bring with you to the next provider, you're going to be ahead of the curve.
https://kubernetes.io/docs/concepts/cluster-administration/f...
https://medium.com/google-cloud/experimenting-with-cross-clo...
It's like developing your app to work with MySQL, PostgreSQL, sqlite, Oracle, SQL Server....
Unless that is a business requirement, don't do it.
It reduces you to the lowest common denominator and the chances that you will actually switch from one DBMS to another is low and probably would require lots of work regardless.
In almost all cases, YAGNI.
And used Postgres-specific features in the schema.
sigh
Multi-cloud is a huge decision that need lots of planning and application level support. I'd go so far as to say it is an order of magnitude easier to design an application that can be moved between clouds easily than it is to design a multi-cloud application. This still defeats vendor lock-in.
I think CockroachDB is interesting tech, but this blog post reads like a pretty empty marketing piece.
I've been through datacenter power outages, refusal to renew, and difficulty in getting circuits in a timely manner in multiple Tier 4 datacenters.
Self-hosting is only as good as your planning, strategy, and execution, which is much the same as the cloud world. You're probably safer running a single EC2 instance than running a single dedicated server in some random colo, but certainly running your own geographically distributed infrastructure is likely better than relying on a single AWS availability zone (for example).
However, it's a great idea to open up peoples' eyes to the idea of how being on a major cloudy provider can make you collateral damage of an attack that never would have targeted you if you were in DIY mode.
I don't think this should apply to anyone running some hosts for web applications though.
It's not so much that most of these would be permanent - but it does add risk of downtime from the "not ops domain" to the ops domain.
And multiple providers isn't a universal fix - but it might mitigate some risk.
What I would be curious, is, how much faster the db would be if it was in c++/rust instead of java/go. I know bigtable claims 3x faster than hbase. Scylla claims ~10x (though also different architecture) faster than Cassandra. Trinity ~2x compared to Lucene. etc etc.
Also: incoming comments about the name of the db.
Agreed, but until there are some published performance metrics, I think it's an assumption more than a truth. Premature optimization...
Like even TiDB (mysql protocol, golang) uses TiKB (kv in rust,rocksdb) to make it faster.
For some reason(s) nobody is using LMDB on distributed dbs (only know actordb). While for single-nodes there are many cases that have switched and been better.
Sadly had to give up on CockroachDB. They've got good engineers on their team and have some nice funding as well so I'm hopeful that they'll make big improvements in this department eventually. But it'll take quite a lot of work to bring it up to speed - excuse the pun :)
Oh and I don't think Go is any limiting factor here. It's more of an architectural/engineering issue.
We're also talking about wide-area storage. Any latency from Go over C++ should be negligible: the Go GC is sub-ms latency now and will just get better [1] while for inter-DC network latency you can expect 50-2/300 ms latencies. The GC just doesn't matter here. Go also gives you control over memory layout, and has an ever-improving compiler [2]. C++ would only really be necessary for sub-ms tail latency systems which are really rare.
I think the gap between BigTable and Hbase is likely due to Google spending more time optimizing BigTable. They do use it for basically everything not built on Spanner.
[1] See recent activity on https://github.com/golang/go/issues/10958 [2] Go has a ways to match GCC, but you can work around the language for now to get close to C. If you want to improve things here, maybe contribute to gccgo or gollvm so Go users can get all of LLVM & GCC's optimizations.