Hacker News Reader

Comment by seiji | Hacker News Reader

We built a page logger. Any time a page was loaded, we'd keep track of it.

They reinvented the access log? By hand? Are people really that detached from how servers work?

I assumed this just meant they built something on top of access logs that told them what percentage of page loads also loaded JavaScript. I don't know about your webserver, but mine doesn't tell me this kind of thing out of the box.

Anyway, it's pretty irrelevant whether they added logging in their app (one line of code?) or enabled their webserver's built in logging.

Access logs that write continually from different processes or threads (like PHP) can cause System IO to reach critical proportions. In any non-trivial enterprise application it's completely reasonable to have a stat-logic-layer for recording events... The access log can still do it's own sorta thing - this is more true if you have more than 1 server.

Surprised no ones mentioned the noscript plugin.

Since the user is on facebook they'd have facebook unblocked, but noscript still blocking everything else.

How would you discern a noscript user from bot?

You wouldn't, but if the vast majority of your ad click visitors have javascript disabled, and only a tiny percentage of your other visitors do, what conclusion would you draw?

You don't, but it's very unlikely that there were enough noscript users to match the stats. From the post:

"...in all of our years of experience, only about 1-2% of people coming to us have JavaScript disabled, not 80% like these clicks coming from Facebook."

It's a different marketing block.

For example, target something for the Reddit.com users and you'll see 90%+ block ads.

With facebook granular targetting you could very specifically end up with a target segment with a large proportion of noscript type plugin users. Without more details on their campaign I would not be so quick to place blame purely on bots.

But this is about users who click on ads on Facebook. The sort of users who have javascript disabled are going to self-select out of that group pretty strongly.

And even among the most technically literate groups, I'd be amazed if you saw 80% use of noscript. So many sites require javascript that it's easier to browse with it enabled.

> How would you discern a noscript user from bot?

There are a few tricks to do that, described on the old ha.ckers.org weblog, or perhaps on the related sla.ckers.org forum. No idea if they still work, though ..

You could always see if you could trigger NoScript's built-in XSS or frame hijacking protection :) Bots will not notice, and users get a scary warning message and believe your ad is trying to hack them. Not good for your conversion rates, but a pretty sure way to tell them apart.

Unless you're targeting noscript users there's no way they account for a significant portion of those 80% of clicks.

What you say is valid, however no-script users are a small minority of internet users. I doubt their numbers would tip the results in either direction.

This could be a businessperson's interpretation of what happened.

Or they might have indeed reinvented it. Kids these days and their Heroku! Back in my day we'd process each HTTP request by hand. And we'd do it uphill both ways in the snow.

using awk.

Hey now, awk is simply lovely. It's ridiculous how popular it is to slight it these days.

Indeed.

Most of the parsing required for the logfile analysis I describe is written in awk, and goes through roughly 1 million lines of logfile in about 75s on a commodity 2Ghz Intel Xeon.

Pre-caching of host lookups is done via xargs and a suitably high '-j' value. Running on a pre-sorted and de-duped list of IPs, this takes another few minutes. Lookups are read into hash tables for faster processing in the main awk parser (avoids system calls, DNS latency, and especially negative result caching failures).

It's fast and simple. Not a 100% solution, but quick to add bits to over time as new needs arise.

How does 'using awk' count as a slight?

It was the proper add-on to the previous comment kids today don't learn awk. Becuase they have Perl, Ruby, and Python which are supersets of the unix command line tool set. Awk was the king in the years before Perl.

I never really learned to use cut, so I sitll use awk for one-liners the deal with columns in delimited-data text-files.

Sorry, I read it as a slight because it came right after "uphill both ways in the snow".

Which usually precedes "Get off my lawn!"

Anyway, if I want to take gratuitous pokes at a language, I usually choose XML; sometimes java.

Which, of course, I've just been informed is the "preferred" integration language for the project I'm presently working on. Sigh.

You had awk? Ha! Luxuries! Why when I was a young programmer we had to write the code in the snow with our pee, and a compiler was just a word for the pilot of the hovering dirigible that read the instructions and passed them to the ALU, which was another fellow with an abacus. They would wrap the re- sults around a rock, and drop it on my house when the program would exit. We had to walk uphill...

Using butterflies: http://xkcd.com/378/

With VB6 you can check IP Addresses.

Please guys. This is not Reddit.

Damn straight. Cheap jokes are not allowed.

And that is how to do a c-c-c-c-c-combo breaker on Hacker News.

This is besides the point.

Is it though? It seems they spent a lot of time throwing more and more javascript at the problem until they realized they couldn't figure out what was happening. If they had real logging from the beginning, it would have unconfused them much sooner.

They were taking orthogonal measurements to see if the pattern would appear over and over. Also it sounds like each successive measurement technique was at a lower level than the previous technique. Sounds pretty thorough. I'd love to see some actual data though.

How would you use the access log to determine if client-side javascript was being executed?

For the page in question, count total hits in the server access log and subtract total hits reported by the Javascript analytics software.

Check if it's even being fetched for a start. Assuming it's not inline.

Browser-cache my friend. You'll need to poll the server on page load (or even better in this case, after you load the ad)

Use a javascript function to poll the server for something.

thanks for coining the word "unconfused"

Access logs can't record if javascript is enabled. There are other things I would be interested in if I were doing this too, I'd check the browser's TZ, check if flash was enabled, etc. and do it across sessions.

Depends how far they went profiling their visitors, it clearly says in the article they were tracking if js was enabled, so yes, you'd need to roll something or another by hand. GA relies solely on js for example.

easy: < script > (new Image()).src = document.location.href + "Hey, i have javascript < / script >

And then compare in the log.

genius

The way I understand this is that they want to log who accesses their pages on FaceBook's servers.

I am not familiar with how things work on FaceBook, but I would think that FaceBook users do not have access to the logs that FaceBook's servers write.

Reply on news.ycombinator.com