It's a demonstration. Kafka is a very popular input for Clickhouse. So are data lakes. The point is neither need Kafka at all.
88 karma · joined April 14, 2011
We're heavily inspired by Turbopuffer. I'd say we are comparable to them when they launched in terms of perf and scale. But they've obviously invested heavily since then, so we're not going to match them on raw perf at scale right now. Our goal is to be a pretty competitive OSS offering over the long term though.
The next biggest lift for us to get much closer is quantization. If we squeeze more signal into fewer bits, we will improve performance end to end.
1. the reason it's slow as you select more series over longer periods of time is that the series has to be pulled for each time bucket in the range, and then the samples have to be pulled for each bucket. By compacting older buckets and merging samples together, historical queries should be pretty comparable to 'more recent' cold queries. 2. We don't pre-cache all the metadata today. If we did that, then we could parallelize sample loads much more efficiently, lowering latency. 3. There is a lot of room to do better batching and tune the parallelism of cold reads.
We've only been at this for a couple of months. THe techniques to improve latency on object storage are well known, we just have to implement them.
Another benefit is this: all the data is on S3, so spinning up more optimized readers to transform older data to do more detailed analysis is also an option with this architecture.
It's a meaningful change in calculation of running yourself vs paying someone to do it for you IMO.
> Is this Magical Pixie Dust I can sprinkle on my application?
> No, not quite. Exactly-once processing is an end-to-end guarantee and the application has to be designed to not violate the property as well. If you are using the consumer API, this means ensuring that you commit changes to your application state concordant with your offsets as described here.
I think that is a pretty clear statement that end-to-end exactly once semantics doesn't come for free. It states that there needs to be additional application logic to achieve this and also specifies what must be done.
I write as one of the implementors of exactly once guarantees in Kafka.
3. Yes, it has been tested empirically. Quoting from the article:
> We wrote over 15,000 LOC of tests, including distributed > tests running with real failures under load and ran them > every night for several weeks looking for problems. This > uncovered all manner of issues, ranging from basic > coding mistakes to esoteric NTP synchronization issues > in our test harness. A subset of these were distributed > chaos tests, where we bring up a full Kafka cluster with > multiple transactional clients, produce message > transactionally, read these messages concurrently, and > hard kill clients and servers during the process to > ensure that data is neither lost nor duplicated.
> Also, is there a reason not to use large pages directly for the mmap'd sets if you know you're going to have them hot at all times? (I assume they read the entire file on start?)
We could use large pages directly. But, as I mentioned in the article, the performance gains would be negligible compared to the gains that come from having things in memory in the first place. These are not very large memory systems and the page table / TLB miss overhead doesn't seem to be biting us. We are just following the mantra 'pre-mature optimization is the root of all evil' :)
$1400/month gets a one-bed apartment in Mountain View. I used to pay $700/mo for a private room and bathroom in a 2 bed apartment in Mountain View. Those prices seem over hyped.
This lays bare the root of procrastination. It is also provides forceful, direct ways for dealing with it. I don't know a single person who has read it and not taken something positive away.
And it's a really quick read. I went through it in an evening.
But it did take 3 rejections spanning a month before I got through.
The reason for the delay?
My service is invitation only. I had a 'login' and a 'request invite' screen in the app. The 'request invite' screen accepted the email address of the user requesting the invite.
Apple said 'We do not permit invitation only apps on the app store. Please remove the request invite button'. I tried arguing with them, citing popular apps (like pinterest) which behave identically.
But to no avail. I had to remove the button before they approved the process.
Apple were really thorough. They actually logged in using the credentials I provided and played with the whole end to end service.
They even called me to explain their original rejection.
So I was both impressed by their thoroughness, and miffed by their lack of standards.
On the whole, I don't like this approval process. It seems whimsical and is not good for the small-guys: sometimes they land hard on us, or at other times let total crap in, which devalues the trust people have on non-popular apps in general.
I have an MVP which I have been testing with early users. But scaling up marketing is the really hard bit. It is something which will take time, because its not really in your control. You have to keep showing up in the right places in tasteful ways.
And then people _may_ convert.
But I think it all can be done with a side income. Right now, if you have web-development + mobile skills, getting freelance jobs to pay the bills is not a bad option.
All of this translates directly into developing a taste for elegant (programming) interfaces and well structured architectures.
In the past, I put together a curated list of pdfs for anyone who wants to start studying and practicing Dijkstra's mathematics. Here is what I recommend:
EWD1300 - The notational conventions of the mathematics : http://www.cs.utexas.edu/users/EWD/ewd13xx/EWD1300.PDF
After that, I have this collection of EWDs and other documents that offer the best, graded, introductions to the idioms and concepts of Dijkstra's mathematics:
What sets these books apart is that they try to tackle the problem of _how_ to design elegant algorithms. In the process they introduce crucial concepts like invariants which, once you understand them, become indispensable tools in reasoning and designing algorithms.
More generally, their stress on elegant proofs develops a taste for mathematical elegance which would serve all programmers well as they design new systems.
These books are much too under-appreciated, in my opinion.
He explained the basic concepts of online marketing and the basics of marketing strategy in a supremely succinct and inspiring way. It was golden! It was entertaining! Highly recommended!