Surprised nobody mentioned Matomo Analytics yet:
It's FLOSS, comes both self-hosted and as a paid service and it's pretty good. Perhaps good enough for your use case?
It's FLOSS, comes both self-hosted and as a paid service and it's pretty good. Perhaps good enough for your use case?
1. Do you grab the country from the IP address before anonymising it? (I know VPN's are a thing, but we would like to know where our customers are coming from)
2. Why encrypt with a daily key instead of just hashing the IP address and storing the hash? Do you ever decrypt the IP address?
3. How have you found CloudWatch? I'm logging all requests to our own database, because it's so simple to do. What's the bonus for Cloudwatch?
Thank you for your questions.
Disclaimer: I also work on WTM from time to time, so I hope can answer these questions to some degree :)
1. WhoTracksMe is served via CloudFront, it only get's the IP that CF reports. To get the country, there are two ways that can be used. IP => Country mapping via a Database or additionally, because CF has multiple edge locations, if needed the you could see which edge location served the request, that will help you narrow down the location at a country level. Ofcourse, in both cases if the user is using VPN, you would only records the VPN location.
2. The problem we want to avoid is same IP with same hashing algorithm will produce same value. Which can then be used to co-relate user activity across days. Second, depending on what hashing scheme one is using, rainbow tables can be used to get back the original value. Therefore, to avoid, we used the approach of daily key. Now that I write, we could also use some hash + daily_salt. This should give the same guarantees. - Needs to be checked.
3. In this setup we are not using Cloudwatch but Cloudfront, which is the CDN provider from AWS.
2. Chances of a hash collision on IP address is pretty small (i.e. statistically insignificant). But adding a daily salt would decrease the odds, for sure.
3. Sorry, my bad, I meant Cloudfront, but it's Sunday night here and there has been wine ;) Same question: how has Cloud{{thing}} worked out for you? Is it worth the extra complexity?
The problem, as I understand GP, is not so much the fear that two different IPs might collide. The problem is that seeing the same hash, you know that it refers to the same IP, and as such you can correlate users/sites over time (a privacy invasion).
Now, everyone takes tracking for granted as if the option of not being tracked is some sort of unachievable goal.
That's the issue. It isn't tracking A as opposed to tracking B. It's tracking period.
I'm not selling adverts to them. I'm not capturing personal information. all I'm doing is trying to understand who my customer is and why they're not buying my product.
Why is this unethical?
And why is it OK to analyse server logs, but not OK to use javascript, to do this?
The data in your server logs is data that people sent to you on purpose (by visiting a page on your site, or filling out a form on your site). If you collect data about people by sending them code to execute and send data back to you, they're not meaning to send you that data; you're doing it behind their back.
I'm glad you try to use data you collect responsibly, but if people aren't meaning to send you data, you're not entitled to it.
But I'd agree that letting the users use a public resource on a condition that they should allow arbitrary scripts to be executed on their side can be considered unethical.
If you walk into a shop, you have an expectation to be observed (both to check you're not shoplifting, and also to see how customers behave). They don't tell you that you'll be observed, it's just one of those things you expect when you walk into someone's shop.
We effectively have a notice on the front door that says we will be observing their behaviour while they're in our shop. They have every opportunity to walk away. I really don't see how this is unethical.
The problem is that this is a poor analogy to computer networking. I send you an HTTP request; you send me an HTTP response. I'm not visiting your shop; we're sending each other messages.
My expectation is that you won't follow me around, observing how long I look at certain parts of your message, gauging my reaction to different parts of your message. That this is the norm on the web is an unfortunate reality, but it doesn't make it okay.
I totally understand that that's what's happening behind the scenes, but that's like saying "I vibrated the air in specific harmonics" rather than "I said yes".
Could you please disclose the name of your company so I can be sure to avoid it like the plague?
If I were visiting your shop, it's totally reasonable for you to check to make sure I'm not stealing anything. But I can't steal anything of yours by sending you messages! (Setting aside the possibility of me hacking your server, but that's not what you're sending me javascript to check on.) Hopefully this expresses more clearly why I don't think the shop analogy holds water.
If "sending messages" is too primitive of an abstraction, we can imagine exchanging letters in the mail, or buying a newspaper, or something like that.
(BTW, I'm sorry for the unconstructive sister comment here - I definitely don't condone it.)
Maybe its time for shops to have EULA's covering the entire door before I open it.
That's not true: people have no idea what is sent and they're not choosing what's sent (with the exception of some forms.)
It's OK to collect display width, or estimate the time on a page even though people didn't send it to you on purpose.
all I'm doing is trying to understand who my customer is and why they're not buying my product.
Then ask them. The same way companies did for hundreds of years before javascript tricks made people lazy.
These things can be achieved by simply talking to your customers. Why is that so hard to do?
Also, what makes you think that your interest in knowing your customer's behavior is more important than your customers' right to privacy?
Startup 101 "Mom Test" stuff. If you ask people what they think, you get their intentions. If you measure their behaviour, you get actual data. Talking to customers is excellent, but measuring what they actually do verifies that they're not lying to you because they're nice people.
>Also, what makes you think that your interest in knowing your customer's behavior is more important than your customers' right to privacy?
I don't understand this. If I ran a physical shop and watched my customers browse to understand their behaviour, no-one would think this was unethical, intrusive or violating their privacy. Why is it all of those things online?
Or is your objection just to the Javascript? If I "violate their privacy" by analysing server logs it's OK, but adding javascript to measure behaviour is "bad, hmmkay"?
> Or is your objection just to the Javascript? If I "violate their privacy" by analysing server logs it's OK, but adding javascript to measure behaviour is "bad, hmmkay"?
If you misuse any data you collect from anywhere, you're being a jerk. We're talking about collecting data in the first place. If you collect data by having a webpage visitor execute code on their own computer in the background and send data back to you, they clearly aren't meaning to send you that data, so you shouldn't do it.
> Startup 101 "Mom Test" stuff. If you ask people what they think, you get their intentions. If you measure their behaviour, you get actual data. Talking to customers is excellent, but measuring what they actually do verifies that they're not lying to you because they're nice people.
Startup 101 is anathema to treating people well. That some data about people could help you run your business better doesn't mean you're entitled to that data.
I respect that you have this point of view, though. In that I'm glad there are people out there fighting this fight. It needs to be fought, and we need to be questioning these boundaries.
Thank you for forcing me to question mine.
I'm purely talking about tracking your behaviour on our site. We want to know if you spent 20 mins reading our "about" page and then looked at our pricing page, and then vanished.
In terms of the rest... country is probably interesting, language preference is interesting (when do we need to translate from English because we have enough customers not speaking English), platform is probably interesting (purely so we can prioritise development to match our customer base). Where you got referred from is probably interesting, I guess.
But all of that we'll be getting from the http request and ip address rather than any malign tracking cookie nonsense.
I don't want you to know this sort of information. It's too invasive.
To folks saying to “Just use server logs”, it’s simply too hard to connect the dots unless you really know what you’re doing. Digging through my nginx logs to find that the majority of my users are dropping off at the second step of sending an invoice would be nearly impossible. Oh I could connect those logs to my api’s database, now I need a place to aggregate that data together. Maybe I could send all of that data together to elasticsearch and query with kibana. Now I need to learn how To set those things up.
Or I could use Matomo for some analytics real quick
GUI is nice, currently trying it out but not with php code but with embeded js like other tools use.
No it's not, just stop overarchitecting your stuff.
> Digging through my nginx logs to find that the majority of my users are dropping off at the second step of sending an invoice would be nearly impossible. Oh I could connect those logs to my api’s database, now I need a place to aggregate that data together. Maybe I could send all of that data together to elasticsearch and query with kibana. Now I need to learn how To set those things up.
You can log requests/responses however you want on your server, it doesn't have to be an nginx log. Just put whatever request data that matters to you into your database immediately after you send a response.
> Or I could use Matomo for some analytics real quick
"Not being a jerk to my website visitors is too hard."
You can still certainly be a jerk with data you collect in your server logs. (For example, I'd be pretty angry if you sold your log data to google or some other advertiser.) But your server is your computer, and the data in server logs is the data making up the HTTP requests that were explicitly sent to your server. That's your data, not mine.