Hacker News Reader

Comment by dodders | Hacker News Reader

Assuming 40 pairs, 2 ticks per second and 1kb to store a tick (including timestamps, indexes etc), I reckon that's about 70TB for 40 years worth of data, not counting weekends.

I've assumed only outright ticks, hence the 40 pairs - in reality most banks store the crosses as well, which can be up to 1,600 pairs (40 x 40), which will get you into the PB range for the 40 years.

View on HN

One kilobyte per tick seems quite generous. All of the EURUSD tick data for 2013 from histdata.com (the source mentioned in the article) is only 515MB (~20GB for 40 years, ~824GB for 40 pairs).

I would say a "tick" comprises the timestamp (stored as a long int) and ten levels of the order book (bid price, ask price, bid size, ask size) each stored as a double-precision float or a long int, so that's

  (1 + 4 * 10) * 8 = 328 bytes

per tick, so 1KB isn't far off. Obviously not every level changes on every tick, so there are opportunities for compression, that can be significant.

Note that the "tick data" from histdata.com gives you prices sampled every 1 second (so not every tick) for the top level of the order book, and doesn't give you any size information at all.

Not the surprise: you need less than 3bytes on average to store a tick...

Reply on news.ycombinator.com