Building ClickHouse Cloud from scratch in a year
clickhouse.com
clickhouse.com
Haven’t used ovh but do and vultr offering were severely limited when i tried it
For a high performance service like ClickHouse the nodes may need to be optimized and that's not often done in the fully managed solutions.
For EKS they often take forever to get to the current version of k8s and that might be a problem.
Whilst performance of the master nodes have increased recently they might not be up to scratch for what's required if you're doing a lot of operations.
All in all managed does not mean fit for all requirements. It certainly is great for at least 80% of cases but not all.
Well in this case they are providing systems management services as in the ClickHouse database as a service.
Disclaimer: I work for Altinity.
Given sytse is the CEO and founder of GitLab, I am sure he’s not interested in getting vendor locked. Community solutions like kops help keep things open and accessible to everyone.
> REST APIs are independent of technology used, which helps avoid any dependency between Control Plane and Data Plane. We were able to change the language from Python to Golang in the Data Plane without any changes or impact to the Control Plane. They offer a lot of flexibility, decoupling various server components which can evolve independently. > They can scale efficiently due to the stateless nature of the requests - the server completes every client request independently of previous requests.
Im not sure what REST apis are being compared against? Something like RPCs? If so, then using an interchange format like protobufs allows for automatic client generation in several languages. Additionally, schema changes are much less painful.
But, as of yet, it has not come up.
Maybe things have improved, and maybe I was using the "wrong" languages, but when we tried this at a previous company the generated code was deemed so awful that we abandoned that idea almost on day 1. To this day I tend to believe the promise of "write once, generate anywhere!" to be mostly BS.
Not even picking on protobuf specifically here - last time I tried a REST generative approach (swagger?) the results were similarly unusable. The whole concept is a fantasy, unless things have staggeringly improved.
Out of all the options for generating typesafe interfaces this seems the most flexible and generates usable code.
It's so much more expensive in the long run, though. Everything talks in HTTP. Everyone can read and understand HTTP. Proprietary binary protocols add a layer of complexity to literally everything else you do, and require a translation layer to talk to the front end.
I wish there was a little more info on the team size and which parts took the most engineering effort to build.
With that said, it sounds like they went with a simpler design than what is described in the AWS article they linked in their blog post. It just sounds like they're able to spin up multiple data plane cells per region, which is what every DBaaS provider needs to do.
(to GP: Powerpoint and Keynote can go a long way.)
You can then say this role can read/write to “s3://bucket/${account-tag-name}/“ and have it enforced.
Because they use a single service account per client, which has no limits as far as I know, they can add a IAM assume role policy to say “the client-ID session tag must match the Kubernetes service account name” or something like that.
What’s annoying is that you can’t go any more granular than the service account name - currently there is no way to create a pod IAM role with specific pre-baked tags, so you can’t say like “this IAM session can only be assumed when tag X equals the k8s pod label value Y”.
Being able to do this would be insanely valuable for specific and fairly niche IAM problems. As I understand it, it’s a limitation with Kubernetes.
The pod IAM roles stuff leverages Kubernetes stuff, and the token that’s mounted into the container is a YAML representation of a Kubernetes token object. There are no fields or other way to add this information into the object.
You would need to encode it into the JWT itself, which isn’t possible or something.
I’m half remembering this, and I can’t find the issue on Guthub because everything has been shuffled around since.
At least, that's what I would have done, given the timeframe.
It requires some considerations. For example, the linear read/write performance on typical c5d/m5d/r5d instances is just around 4 GB/sec, which is faster than 25 Gbit network, but slower than 40 Gbit network. You get 25 Gbit network on larger normal instances in AWS, otherwise, it requires "n" - network-optimized instances. S3 read performance is higher - 50 Gbit/sec achieved from a single server, higher values will require either more CPU or uncompressible data (the data is usually compressed "too much" for a 100 Gbit network to become a bottleneck). But S3 is a tricky beast, and it won't give a good performance unless you read in parallel and with carefully selected ranges from multiple files. The higher latencies of S3 are expected to be around 50 ms unless you are being throttled (and you will be throttled), which is worse than both local SSDs and every type of EBS. S3 is cheap and saves cost on cross-AZ traffic... unless you make too many requests. So, a lot of potential troubles and a lot of things to improve.
For example, at this moment, I'm doing an experiment: creating 10, 100, and 500 Clickhouse servers, and reading the data from s3, either from a MergeTree table or from a set of files.
Ten instances saturate 100 Gbit/sec bandwidth, and there is no subsequent improvement.
JFYI, 100 Gbit/sec is less than one PCI-e with a few M.2 SSDs can give.
There are request limits but these are per partition. There is also an undocumented hard cap on list requests per second which is much lower than get object.
We did use multiple VPCs, that might make a difference
It also works on a laptop, or even without installation.
I think all the choices they made makes so much sense.
This essentially means they must be running multiple dataplane clusters however they might have just a single control plane.
Thoughts?
Whilst hoping a single customer doesn’t exceed the limits of a cell =)
We picked these ten technologies to translate YAML configs to cloud bills and put ClickHouse inside.
From the blog post: "the fastest baseline here is ClickHouse server running on an AWS m5d.24xlarge instance that uses 48 threads for query execution. As you can see, an equivalent cloud service with 48 threads performs very well in comparison for a variety of simple and complex queries represented in the benchmark" so there can be a small difference depending on the sizing but it's something to consider on a case per case basis and often other dimensions need to be taken into account (operational cost, bottomless storage, linear scalability, reliability etc.).
Basically, we're looking for the person who has at least some expertise with operating CH in production and paying big money. In four month offering $6K-$7K/month we've got ziltch.
Running a database on Kubernetes backed by an object based storage? Not even close.
Don't get me wrong, Clickhouse folks are great and their DB is fantastic, but your scale may need some adjustments.