Is anyone else considering the same? I know this is standard for companies like Amazon and Facebook, but what about everyone else?
If you are using a self hosted setup, what is it? Custom built, open source, etc?
I don't use them, but they are building enterprise-grade self-hosting.
Disclaimer: I am working on a project in the marketing analytics space. I don't use them and this isn't an endorsement, just a pointer to research more!
Seems like they have some decent companies using them too.
I don't know if we want to go that direction though. Plus I wasn't even sure who is actually self hosting these days. Kind of reminds me of the Silicon Valley show with the "box" in the data center dilemma.
The bigger the data the more self-hosting will make sense for you, but the less for your customers.
This works for snowplow because they sell services, which only really works for bigger companies who have the resources to self-host and to pay the services fees.
I've been looking at Snowplow and it seems really cool.
About every company who don't outsource everything + every company that adheres to strict privacy rules + every company that is forbidden to share user data because of regulations + every company that is beyond what free GA has to offer.
That's like a shit tons of companies, a lot of which are established and have money. Just sayin'
The only aspect I didn't like was it was PHP. Not really my strong language - if I wanted to customize it.
> However, it didn't scale that well for us.
What kind of scaling problems did you encounter ?
Towards the end, IIRC, the calculation of unique visitors within a custom range was also slow.
I don't know how many visitors and page views you have, but that's a red flag for most established sites.
As a side effect same system detects malware and cyber attacks on other websites pretty well as well.
Do you pull any info regarding the IP addresses, or is it only the raw logs that you're going through?
Mostly playing with raw logs and then even RAW-er logs using Splunk Stream (thing that switches network interface in promiscuous mode and gives me all data for all protocols and any context I ever want).
For example I can analyze anomalies in web hits and anomalies in web session to discover new, previously unknown traffic sources and patterns.
It helped to discover 2 new classes of cyberattacks I didn't know were targeting my server.
In fact, most of the systems we had were all homegrown and while initially it was a great idea - the maintenance part of it took a lot of time away from optimizing on generating revenue.
We spent a lot of time trying to figure out the structure of our self-hosted web analytics platform, how it tracks data, how it stores data, etc. The majority of our company were engineers but we still wasted a lot of time fiddling with the self-hosted analytics.
There will likely be a subset of customers who would be interested in self-hosted but I'm willing to bet that they are more likely to be companies more engineering-oriented.
Yeah I was thinking the same thing - that companies who are self hosting are more likely to be engineering-oriented. Otherwise like you said it's just too much of a pain to handle it yourself.
May I ask, why did you self-host vs use an external service? To save money, for greater security and privacy?
Ultimately though, the business demanded to switch to something more robust and we went with Omniture.
I do believe that self-hosted is an option if it's easy to maintain and it also delivers on business features like reporting and data exploration.
Thanks for the insight.
With all the regulations and policies on data protection, using something not self-hosted is just not going to happen. (I believe that if there was a possibility that their users were affected by a 3rd party breach, the fine is around 1000x the project budget).
If this means a little less information about the audience, that is perfectly acceptable.
It tracks a few million records a month so not high scale and runs on a single $5 digital ocean instance.
I did attempt to make it Sass for hosting resellers as an upsell but never made any progress. It's just running for myself these days.
Also, can I ask how big your company is? A few million records is decent volume.
No company actually. Just side projects. The largest being searchcode.com
Feel free to email me if you want further details on either. Details in my profile.
I've actually been looking at Snowplow and that seems pretty badass so far. Might work for what I'm looking at.