Ask HN: Advice for storing medium-sized, variable time data
Each time I'm recording a single integer for about 500 different things. But it's not always for the same things. Essentially I'm recording a varying subset of the most significant values of a bigger pool of data.
I've never had any contact with databases before and currently I'm recording that data into json files (one for each day). I've also thought about csv. But both are obviously file-basede storage solutions and I'm thinking the amount of data might get too big for that.
So are databases a better solution? And if so: which? As far as I understand, most common dbs are relational which are essentially tables. The problem is that if every new tick is a new row, the number of columns will be large (in the 1000s) and increasing over time. I take it that's problematic.
Ideas, suggestions?