bluestreak

This ran on Java 11. This isn't so much of an issue for us. We are trying to avoid allocations, even as trivial as those. There are other examples that allocated where they should not, for example this lambda will allocate.

  () -> System.out.println(1)

I lost hope in escape analysis quite frankly.

Rnd is something I have written because Java's Random is slow and clunky.

https://github.com/questdb/questdb/blob/master/core/src/main...

here is one: https://github.com/questdb/questdb. Disclaimer, I work on this project. Main reason we use Java is speed of development (which increased with amount of base libraries written) and ease of testing.

We have written a database in zero GC java and one thing I have not seen any evidence of "escape analysis".

   @State(Scope.Thread)
   @BenchmarkMode(Mode.AverageTime)
   @OutputTimeUnit(TimeUnit.NANOSECONDS)
   public class EscBenchmark {

       Rnd rnd = new Rnd();

       public static void main(String[] args) throws RunnerException {
           Options opt = new OptionsBuilder()
                   .include(EscBenchmark.class.getSimpleName())
                   .warmupIterations(5)
                   .measurementIterations(5)
                   .forks(1)
                   .addProfiler(GCProfiler.class)
                   .build();

           new Runner(opt).run();
       }

       @Benchmark
       public int testEscapeAnalysis() {
           int[] tuple = {0, 2}; // esc analysis? where are you?
           return tuple[rnd.nextPositiveInt() % 2];
       }
   }

And the output of GC profiler:

  Benchmark                                                     Mode  Cnt     Score     Error   Units
  EscBenchmark.testEscapeAnalysis                               avgt    5     8.234 ±   0.029   ns/op
  EscBenchmark.testEscapeAnalysis:·gc.alloc.rate                avgt    5  2647.216 ±   9.275  MB/sec
  EscBenchmark.testEscapeAnalysis:·gc.alloc.rate.norm           avgt    5    24.000 ±   0.001    B/op
  EscBenchmark.testEscapeAnalysis:·gc.churn.G1_Eden_Space       avgt    5  2643.140 ± 177.137  MB/sec
  EscBenchmark.testEscapeAnalysis:·gc.churn.G1_Eden_Space.norm  avgt    5    23.963 ±   1.613    B/op
  EscBenchmark.testEscapeAnalysis:·gc.count                     avgt    5   157.000            counts
  EscBenchmark.testEscapeAnalysis:·gc.time                      avgt    5   103.000                ms

QuestDB | Technical Content Writer | Full Time | London,UK or Remote

At QuestDB we help developers handle explosive amounts of data while getting them started in just a few minutes with the simplest and most accessible time series database.

We are looking for an experienced Technical Content Writer to join our fast growing team. You will be writing technical documentation for developers along with explainers and tutorials. You should have demonstrable experience working closely with engineers and product managers to understand and document features along with programming experience and knowledge of database technologies. The role requires a good command of English, and great communication and self-motivation to succeed in a remote environment. You will report to the CTO and work closely with the engineering and product teams.

Find out more about the position @ https://questdb.io/careers/technical-content-writer/

Hi HN,

Author here. QuestDB is a fast SQL open source database for time series. About a month ago we launched on HackerNews [1].

Today, I am excited to announce QuestDB 5.0.3 [2]. This new release includes our changes in memory mapping strategy, giving us better performance, as well as some major changes to the Postgres wire protocol support.

In this blog post, we explain our journey to improve QuestDB's performance, especially regarding memory management. Relying as much as possible on the kernel and avoiding extra layers turned out to be very successful.

Thanks

Vlad

[1] https://news.ycombinator.com/item?id=23975807

[2] https://questdb.io/getstarted/

Hi HN,

Author here. QuestDB is a fast SQL open source database for time series. About a month ago we launched on HackerNews [1].

Today, I am excited to announce QuestDB 5.0.3 [2]. This new release includes our changes in memory mapping strategy, giving us better performance, as well as some major changes to the Postgres wire protocol support.

In this blog post, we explain our journey to improve QuestDB's performance, especially regarding memory management. Relying as much as possible on the kernel and avoiding extra layers turned out to be very successful.

Thanks

Vlad

[1] https://news.ycombinator.com/item?id=23975807

[2] https://questdb.io/getstarted/

Thanks for the explanation Nikita. Any branching, hashing etc will increase execution time. The execution example is on laptop, which has 2 memory channels. 0.13s is absolute max that this laptop is able to muster as far as memory throughput goes. Usually 4 threads is enough to saturate memory.

I have written this piece of code: https://github.com/questdb/questdb/blob/master/core/src/main...

This sums 64bit values and using AVX2 it will sum 1Bn in 0.26s. Incrementing conditionally will not be as fast and will throw vectorization out of the window too.

QuestDB is quite intensive on cores. For example it is not a good idea to put two threads on hyperthreded cores that share the same physical core. Also it isn't good idea to put threads on cores that belong to different physical CPUs on multi-socket server. With bare metal box you will have visibility of all of these conditions. On virtualized boxed - this will depend on your luck really. That said - if no one is hammering the same core questdb is hammering - performance will be very similar to metal box.

I wonder what does this query compile to? In terms of C-code equivalent?

   SELECT cab_type, count(*)
   FROM trips
   GROUP BY cab_type;

From execution time it seems to me that this is a straight sum() of 32-bit integers. "cab_type" has two distinct values and if stored 32bit value for "green" is 0 and "yellow" is 1, straight sum of these integers will produce the desired outcome and explain performance. That said the same performance will not extend to key that has three or more distinct values.

Thanks! This is something we built in house. In fact, ‘mpsq’ did all of that. All credit should go to him!

if you can send all three in the same message, Influx Line Protocol for example, we will store them as 3 columns in one table. Does this help?

Our best performance currently is in C++ code and LLVM is something we are considering using to compute expressions, such as predicates and select clause items. This is most likely to be way faster than what we can currently do in Java. What I would like to know if LLVM can optimize all the way to AVX512?

We also need to experiment with hugepages. The beauty is that if read and write are separated - there is no issue with writes. They can still use 4k pages!

OLTP is not a good fit, if your workflow consists from INSERT/UPDATE/DELETE statements

:)

Eventually both. We are starting with baby steps, e.g. get data from A to B quickly and reliably. Replication/HA will be first of course. Then we want to scale queries across multiple hosts. Since all nodes have the same data - they may as well all participate. Sharding will be last. We are thinking of taking a route of virtualizing tables. Each shard can be its own table and SQL optimiser can use them as partitions of single virtual table. We already take single table and partition it for execution. Sharding seems almost like a natural fit.

The strategy is to multicast data to several nodes simultaneously. Data packets are sequence to allow receiver identify data loss. When loss is detected receiver finds breathing space to send a NACK. The packet and the nack would identify missing data chunk with O(1) complexity and sender then re-sends. Overall this method is lossless and avoids overhead of contacting nodes individually and sending same data over the network multiple times. This is useful in scenarios where several nodes participate in query execution and getting them up to date quickly is important.

Over the network streaming is not yet available. Someone has mentioned Kafka support, how useful would that be to stream processed (aggregated) values and/or actual table changes?

Thank you for the kind words!

This is incredibly useful, thank you! It would be awesome if we could chat more about your use cases at some point. Drop us a line on hello at questdb.io or join our slack. Whichever is easier for you.

We store "dimensions" as table columns with no artificial limits on column count. If you able to send all dimensions in the same message, they will be stored on one row of data. If dimensions are sent as separate messages, current implementation will store them on different rows. This will make columns sparse. We can change that if need be and "update" the same row as dimensions arrive as long as they have the same timestamp value.

There is an option to store set of dimensions separately as asof/splice join separate tables.

Thank you for the kind words and constructive feedback. We are here to build on feedback like this. Grafana plugin is coming soon.

We are working on building a solid PostgreSQL support insofar as allowing ODBC driver to execute this type of query from Excel. This is work in progress with not that much left on it.

There will be two different flavors of replication:

- TCP-based replication for WAN - UDP-based replication for LAN and high traffic environments

We are currently building foundation elements of this replication, such as column-first and parallel writes. These will go into and always be part of QuestDB. TCP-replication will go on top of this foundation and also part of QuestDB. UDP-based replication will be a part of a different product we are building that will be named Pulsar.

It has been asked here: https://github.com/questdb/questdb/issues/261. Definitely on our road map. It would be good if you could share your story why you need arrow?

Thank you! We have quite active slack and we try to listen to and help our community there. Feel free to join!

thank you!

- replication is in the works, this is going to be both TCP and UDP based, column-first, very fast.

- yes, benchmarks are indeed are done on second pass over the mmaped pages. First pass would trigger IO, which is OS-driven and dependant on disk speed. We've seen well over 1.5Gb/s on disks that support this speed. Columns are mapped into memory separately and they are lazy accessed. So the memory footprint depends on what data your SQLs actually lift. We go quite far to minimize false disk reads by working with rowids as much and possible. For example 'order by' will need memory for 8 x row_count bytes in most cases.

- durability is something we want user to have control over. Under the hood we have these commit modes:

https://github.com/questdb/questdb/blob/master/core/src/main...

NOSYNC = means OS flushes memory whenever. That said, we use sliding 16MB memory window when writing. Flushes will trigger by unmapping pages. ASYNC = we call msync(async) SYNC = we call msync(sync)

thank you for sharing! The core of memory management is abstracted away. All of the query execution logic is unaware of the source of memory pointer. That said we are still learning and really appreciate your feedback. There are some places where we could not beat aggregation of julia, but the delta wasn't very big. This could have been down to mapped memory. We will definitely try things with direct memory too!

Java was the starting point. Back in the day Rust wasn't a thing and C++ projects were quite expensive to maintain. What Java does for us is IDE support, instant compilation time and super easy test coverage. For things that does require ultimate performance we do use C/C++ though. These libraries are packaged with Java and transparent to end user.

"normal" database does not preserve the order of data as it comes in. To get the data out you have to constantly rely on indexes or "order by" constantly to get chronological order back. Time series database should maintain the order of data and not rely on indexes to have data make sense again.

thanks! What differentiates us from other time series databases is the performance. Both for ingestion and queries. For example we can ingest application performance metrics via Influx Line Protocol and query them via SQL and both should faster than incumbents

thank you, life is an interesting experience :) I used to work with Fabio, Vortexa CEO and had to turn down an offer of being first employee there to focus on QuestDB. They are an absolute awesome bunch of guys and deserve every bit of their success!

What makes QuestDB different from other tools is the performance we aim to offer. We are completely open on how we achieve this performance and we serve community first and foremost.