Ask HN: What stack technologies to use to build analytics platform?
I can't used an API 3rd party service based solution (like Keen.io), because it would be far to expensive at this volume.
The things I need to track and show are quite basic: pageviews, referral breakdown, device breakdown. It needs to be charted on a line chart so I can show the increase or decreases in various traffic sources over time (for example it might show direct, mobile apps, social, etc along with the number of views at each time interval).
I've been really considering how to build this. Our main app is in Rails (not going to use that other than the interface). Most of our low level stuff is written in Go.
At a very basic and noob level.... should I be storing logs rather than MySQL rows? I've been trying to read up on analytics development and a lot of people use log parsing. If I opt to use MySQL is that a poor choice? Data points will be several hundred million per month.
I suppose the only good thing is is that the dashboard is fairly simple. Showing views over time, broken out by source is really the main thing. It isn't like I want to show something like Google Analytics where you can break it out 40 different ways.
I'd love any frameworks, insight, tips, schema suggestions, tech suggestions... really anything, to help me on this task. Thanks!