182 karma · joined August 12, 2013
The code tend to be loaded with primitives that express ownership semantics or error handling. Every time something changes (for instance, you want not just read but also modify values referenced by the iterator) you have to change code in many places (you will have to invoke 'as_mut' explicitly even if you're accessing your iterator through mutable ref). This could be attributed (partially) to the lack of function overload. People believe that overload is often abused so it shouldn't be present in the "modern" language. But in languages like C++ overload also helps with const correctness and move semantics. In C++ I don't have to invoke 'as_mut' to modify value referenced by the non-const iterator because dereferencing operator has const and non-const overloads.
Async Rust is on another level of complexity compared to anything I used. The lifetimes are often necessary and everything is warpped into mutliple layers, everything is Arc<Mutex<Box<*>>>.
That's easy. EBS and similar solutions comes with the price. They're very expensive. Especially, when you need a lot of IOPs. You may be saving on cross-AZ traffic but you will pay ridiculous amount of money on storage. If you have replication you can use attached storage which is way cheaper.
Also, this metadata database looks like a bottleneck. All writes and reads should go through it so it could be a point of failure. It's probably distributed and in this case it has its own complex failure modes and it has to be operated somehow.
Also, putting things from different partitions into one object is also something I'm not very keen about. You're introducing a lot of read amplification and S3 bills for egress. So if the object/file has data from 10 partitions and I only need 1, I'm paying for 10x more egress than I need to. The doc mentions fanout reads from multiple agents to satisfy a fetch request. I guess this is the price to pay for this. This is also affects the metadata database. If every object stores data from one partition the metadata can be easily partitioned. But if the object could have data from many partitions it's probably difficult to partition. One reason why Kafka/Redpanda/Pulsar scale very well is that the data and metadata can be easily partitioned and these systems do not have to handle as much metadata as I think WarpStream have to.
With page cache it's OK, because the FTL layer of the drive will work with 32MiB blocks but in case of Redpanda the drive will struggle because FTL mappings are complex and GC has more work. If Kafka would be doing fsync's the behaviour would be the same.
Overall, this looks like a smearing campaign against Redpanda. The guy who wrote this article works for Confluent and he published it on his own domain to look more neutral. The benchmarks are not fair because one of the systems is doing fsyncs and the other does not. Most differences could be explained by this fact alone.
About that 1M writes thing. You have two options. 1) Organize data by metric name first, or 2) by timestamp. In case of 2) the updates will be linear but reads will have huge amplification. In case of 1) updates will be random, but reads will be fast.
Single table design will be prone to high read/write amplification due to data alignment. Usually, you need to read many series at once so your query will turn into full table scan. Or it will read a lot of unneeded data which happened to be located near the data you need. Writes will be slow since your key starts with metrics name. Imagine that you have 1M series and each series gets new data point every second. In your scema it will result in 1M random writes.
Cardinality of the table will go through the roof, BTW. Every data point will add the key. Good luck dealing with this.
The main misconception about TSDB's is that it's just a data with timestamp. TSDB's has multi-dimentional data model, time is only one of the dimensions.
All these platforms have some problems. E.g. Nginx is PITA if your processes need to communicate with each other. Erlang/OTP has this nasty mailbox problem (O(N^2) behavior of the selective receive). The cooperative multitasking is a piece of Victorian-era technology that is so painfully bad for various reasons, but people still tend to believe that is solves something.
On the other hand, modern operating systems are awesome. The thread schedules are awesome. Thread schedulers can dynamically adjust thread priorities depending on the load and solve some nasty synchronization problems (like priority inversion or starvation) for you. It's not 1995 and OS schedulers can switch threads in O(1) and most server apps can use a thread per connection approach without any problems. The only thing you need to get right is a synchronization, but there are a lot of tools that can help (like Thread Sanitizer). It's much harder to get the synchronization right with cooperative multitasking (good luck finding that priority inversion on implicit lock caused by dumb channel use pattern) than with normal threads.
Leaf nodes contain data from one series (this data should be read together) and SSTable with time-series data contains many series and there is no guarantee that all these series will be used by the query.
>> The Prometheus solution also sequentially places compressed chunks for the same series.
I'm not really that familiar with Prometheus internals, especially with indexing part. As I understand it doesn't align writes so there is a lot of write amplification on the lower level that translates to cell degradation and non-optimal performance, but I can be wrong here.
Random reads and writes are significantly slower if you write everything from one thread. To speed everything up you should write in parallel (for example using Linux AIO + O_DIRECT, or libuv + O_DIRECT). OS level buffering and many OS threads will deliver good random write throughput as well.
There are other effects to consider, e.g. read-write interference.
This makes sense now. I've found out that the compression algorithm performance numbers affect the overall performance in a big way. On modern SSD the entire workload is CPU bound.
2. Because there is a lot of data-structures. I'm using tree per series. The database can simply store hundreds of thousends of series. Creating WAL per series is not feasible.
3. It maintains a list of roots.
4. One I/O operation per node. You will fetch a leaf node for every ~1000 data points and a superblock for every 32 leaf nodes. It's not as bad as it sounds because you will read data for one series only. To span over 4 levels the series should contain tens of millions of points.
5. Yes. You will need a beefy machine for this with a lot of RAM.
6. Random reads are fast on modern SSDs. It's optimized for SSD (I simply don't have a computer with HDD).
7. It stores only composable aggregations - min, max, count, sum, min/max timestamps.
8. All series names is stored in memory. During the query time this memory is scanned using regexp to find relevant series names and they ids. This is a kind of a temporary solution. It works good enough for the datasets with small cardinality (around 100K series).
- "there is no longer a single file per series but instead a handful of files holds chunks for many of them"
- "We partition our horizontal dimension, i.e. the time space, into non-overlapping blocks. Each block acts as a fully independent database containing all time series data for its time window."
I don't believe this will work out well because it will introduce read amplification during query time (compared to file per series approach that they're using now). And I'm really curious how they managed to get 20M writes per second on laptop. The article states that they're using compression algorithm from Gorilla paper and Gorilla paper authors claims that they managed to get 1.5M on a single machine.
> On their face, the benefits of using exceptions outweigh the costs, especially in new projects. However, for existing code, the introduction of exceptions has implications on all dependent code. If exceptions can be propagated beyond a new project, it also becomes problematic to integrate the new project into existing exception-free code. Because most existing C++ code at Google is not prepared to deal with exceptions, it is comparatively difficult to adopt new code that generates exceptions.
And most importantly, I don't think that they're disabling exceptions using compiler flag. I'm sure that they just using error codes for error handling instead of exceptions.
> but many C++ programmers disable exceptions for performance
I have never seen a project like this for 10+something years. The only piece of code with disabled exceptions that I've seen was created for Atmel controllers. And in modern C++ exceptions is zero-cost. Your paying for exceptions only when they occur so, nobody is disabling exceptions now.
This is not the only line that bugs me but I don't have enough time for it right now.
I find exceptions really useful but also, I think that with exceptions people tend to loose very useful panic/error dichotomy. I saw projects without a single "panic" in the code. All those projects gravitated towards dumb error handling mechanisms aka "log all errors and continue".