Analytics on the Cheap | Hacker News Reader

Analytics on the Cheap

0x74696d.com

78 pointsksri27 comments

Be sure to check out https://github.com/snowplow/snowplow before rolling out your own custom analytics solution..

Yeah, using the Cloudfront CDN logs is a very similar approach. If we'd known about Snowplow and it was production-ready when this service were written, it would certainly be worth looking at.

That being said, this service does double-duty for us: when an analytics ping comes in from the video player we're also using that same data to write "bookmarks" for resuming play. This data has different liveness requirements than the analytics and can't be done as a nightly batch. So we'd end up having to write our own Collector (in the Snowplow architecture) anyway to perform that fan-out of incoming events.

Hey 0x74696d - Snowplow co-founder here. Very cool post! The Snowplow Kinesis architecture gives you "fan-out" for free - you could write your bookmarking service as a Kinesis KCL app which reads from the Kinesis enriched event stream written by our Kinesis Enrich (https://github.com/snowplow/snowplow/tree/master/3-enrich/sc...). In this case you'd be using our Scala Stream Collector (Spray with a Kinesis back-end), not the CloudFront CDN Collector.

Nice. If I was starting this project today (it's pretty mature at this point) that'd definitely be something I'd look into.

That being said, when Kinesis previewed it had only Java bindings for KCL. Not sure if that's still the case, but that'd be a limiting factor for our shop unfortunately.

You're right 0x74696d - originally the KCL was Java only. The Java KCL now includes something called the MultiLangDaemon, which means you can write apps in other languages. There is an official Python KCL (https://github.com/awslabs/amazon-kinesis-client-python) but no others I know of yet. There's also AWS Lambda for processing Kinesis streams with JavaScript, and of course you can use Storm or Spark Streaming, although those are JVMish too.

Very interesting. You say "Within an hour or so each GET shows up in the logs for the S3 bucket wherever we’re sending the logs for our S3 example.bucket." Is this frequency configurable? I also see at the link below that the logs are best effort and they may drop messages. How is this for you in practice?

http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerLogs.ht...

The frequency isn't configurable. We execute the ingest daily for a 24 hour window that's IIRC 4 hours old. This works fine for us in practice.

That's really cool. You're saving money in 2 places:

1) not paying for SQS, Kinesis, or some other kind of queue to store the events.

2) not paying for a backend queue processor job to read the events from the queue and write them to S3/Redshift.

You still need a backend processor to do the Redshift copy commands, but this is much lighter weight and you'd never need to scale this process.

You'd also be paying some additional money for the S3 logging but probably pretty minimal.

Very cool!

Kinesis is not hosted Kafka in the way that ElasticCache is hosted Redis. It was developed internally within Amazon.

Oh? Edited to say "The long story short of this is that it’s like a hosted Kafka." Thanks for the correction!

Source?

Should have added a disclaimer. I work for Amazon.

Author here. Happy to answer any questions.

The font color is unreadable on ubuntu/chrome.

Weird. It's #000 on #FFF, unless my meager CSS skills have failed me. Maybe it's just too thin or something? I'll see what I can do but probably not today. /points to the markdown link in sibling comment.

The 'Lato' font is unreadably thin for me on Linux. If I turn it off (via inspect element) and allow it to use the next one (Helvetica) then it looks fine.

Chrome renders thin fonts stupidly thin on Linux (remember that Chrome uses it's reinvents the whole wheel when it comes to rendering!).

Are you using chrome by any chance?

Yeah, should've said, Chrome.

Looks clean on retina mac. Maybe a font rendering issue?

Chrome renders thin fonts stupidly thin on Linux (remember that Chrome uses it's reinvents the whole wheel when it comes to rendering!).

Are you using chrome by any chance?

I can't read it due to firewall :-(

You couldn't stick it in a gist or something could you?

I host on Github pages, so the original markdown is available at https://github.com/tgross/tgross.github.io/blob/master/_post... Does that help?

Also, what kind of corporate firewall lets you have access to Github but not tech blogs? We're hiring. Just sayin'

Actually the company firewall here blocked it because it automatically recognized the hexy domain name as 'suspicious'. Which if you think about it is kind of smart...

Worth knowing too - people sometimes hide analytically servers off on cryptically named domains, and maybe they get eaten by firewalls more than you'd think.

Same here, Trend Micro blocks your domain. It even goes as far as to call it a "Verified fraudulent page or threat source."

Meh. It's my personal blog. I don't care enough about that. =)

Thanks!

I guess they knew you'd say that - your company is blocked too!

> Also, what kind of corporate firewall lets you have access to Github but not tech blogs?

IBM's (although it doesn't block all tech blogs, and who's to know why they've decided to block yours.)

Reply on news.ycombinator.com