Hacker News Reader

Comment by lopatin | Hacker News Reader

Hey this looks really well put together, nicely done! Maintaining a connection to all these exchanges, and managing that data, is no easy task.

A few random questions:

1. Can you speak more to the "synchronized clock" part? You augment data feeds timestamps with your own?

2. What kind of database did you choose for this? How much data are you managing?

Also, (promise you'll give me a discount if tell you this?) this seems underpriced. I haven't seen anyone sell truly high resolution depth data going months back yet. For BitMEX nonetheless. But maybe i'm out of the loop. Anyways, congrats on the launch!

View on HN

Thanks for the feedback, really appreciate it.

1. Yes every message has also local timestamp (100 ns precision).

2. Currently it's around 4-5TB of compressed data (so 25-35TB uncompressed - need to check to be sure).

Indeed pricing is supposed to be very affordable as it's targeted at independent algo traders so without spending huge amounts of $$$ they can have good data to backtest on professional level (arguably if you can call crypto trading professional as some argue). Happy to provide you with the discount, please get in touch with me via email if interested.

Edit: some obsolete info redacted

Your timestamping approach is interesting.

It is well known that exchanges like bitmex suffer from special latency issues. People pay very well to study that and provide mitigation.

Many approaches can be done. But running on a single computer, let alone a single VM, is very dangerous.

I worked for a company that offered multiple timestamps + hardware timestamping for accuracy.

A VM will cause too much latency. Even very well configured, your jitter will be above 100ns by several orders of magnitude.

So be very careful about considering your extra timestamp as authoritative.

You are spot on the other challenge: database. The person who focused on that achieved impressive results using a special hardware + software mix.

Thanks for the details. Flat files in the cloud, and heavy caching on the client sounds like the most effective way to go about it to me too. I work at a company that collects data like this for the traditional derivatives markets and fwiw about half of our data is in S3 too. The other half, Dynamo.

I appreciate you making this accessible to indie algo traders, it's def what we need. Will keep an eye on this.

Reply on news.ycombinator.com