gsubes

why these title changes?!? Does one really have to write separate blog posts with the same title so they don't get changed?

Our tests showed than even with larger messages (100k price ticks per request) pipes were still a magnitude slower.

I am wondering about this myself, even SynchronousQueue (where the spin wait was taken from) is slower than Mapped Memory. Though ASpinWait spins a magnitude longer before sleeping, maybe that is the difference. Or the fact that Memory Mapping goes Off-Heap and instantiates no objects during transfer.

The use case is financial strategy backtests where normally you iterate through the whole dataset and only keep a lookback window in memory to access older data (one backtest iteration per thread per cpu core). Only occasionally having to jump to a completely new data window on demand.

The only downside with the jumps is that they will have to load worst case 9999 data points to reach the desired 10000 index of the file chunk. But with most strategies this is no problem and can be fixed by increasing the in-memory lookback window on demand. Having smaller files for smaller chunks would have decreased read performance because of having to switch files too often. So in that regard the read amplification is not a real problem.

And yes, currently the inmemory AHistoricalCache and file based ATimeSeriesDB is only for date keys, but could be made generic if desired (simple pull request).

Anyway a full db server definitely has more things to account for than this custom solution for this specific problem. While the point here is to show that a custom made solution can be a lot faster than general purpose timeseries databases.

Or do you have different opinions here?

Well, true the title could have been better chosen, forgot there was a timeseriesDB in .NET

Though the original title still was true about the approach in the link being more than 30 times faster than a pure LevelDB solution (which is also often used for this sort of storage).

Database servers like influxdb or druid provide flexibility at the cost of performance. If you want 3 million inserts per second and 14 million reads per second you have to roll your own solution (like the one in the link). Though if you know a database that performs like that, I would appreciate enlightenment. :)

Any query language, network access, non-binary storage format will slow the data system down, so we had to create our own embedded timeseries system to get this speed.