Re-examining our approach to memory mapping
questdb.io
questdb.io
How do I test with realistic data but never risk my customer's data in the process? Either way I'm going to make a copy of real user data. In one case it's much harder to have shared references to live data than in the other.
There are lots of failure modes for rehearsals that can take out the cluster. If you're doing it live, how do you know you don't have a request loop or just introduced an n+1 fanout in a previously well-behaved service? One that can take out said service and parts of its dependency graph? What if I write a query that crashes processes?
On the other hand, it's 'common sense' for staging to use similar hardware to production so that discrepancies don't catch you by surprise. But common sense isn't common, and people can rationalize the hell out of not spending money on precautions, and so they never are equivalent if they are partitioned. Which also makes thing like perf analysis very low fidelity.
More likely that nobody is right, everything is a tradeoff (sucks), and which pain points can your organization collectively stomach?
Author here. QuestDB is a fast SQL open source database for time series. About a month ago we launched on HackerNews [1].
Today, I am excited to announce QuestDB 5.0.3 [2]. This new release includes our changes in memory mapping strategy, giving us better performance, as well as some major changes to the Postgres wire protocol support.
In this blog post, we explain our journey to improve QuestDB's performance, especially regarding memory management. Relying as much as possible on the kernel and avoiding extra layers turned out to be very successful.
Thanks
Vlad