pnachbaur

Out of curiosity, how do you weigh the straightforward cost of a service like Algolia vs the full TCO of bringing it in-house?

I tend to think of Time Series data as being several orders of magnitude larger than 23 million data points per week (38 per second) but now I can't seem to find a good definition of Time Series data. Anyone have thoughts on the rough threshold between event data and time series data? I think of arrays of hundreds/thousands of individual sensors that take 10 measurements a second as "different" than user generated data that is time-ordered.

Most boarding schools get compared to Hogwarts, and when you add a focus on 'technological wizardry' ... ¯\_(ツ)_/¯

We absolutely do not sell any data, and only use it internally to debug issues and optimize the system :)

You could use median instead of filtering out outliers explicitly so your averages don't get skewed.

If you go that way, be sure to read Jepsen: Elastic Search [1]

[1] http://aphyr.com/posts/317-call-me-maybe-elasticsearch

Reminded me of Storehaus [1] but taken further. curious to see what K/V drivers they've written so far.

https://github.com/twitter/storehaus

There is a video under "June 18 | Seattle" on the linked page

If you're interested in push notifications based on analytics data, check out Pushpop [1] (doesn't explicitly support iOS yet)

https://github.com/keenlabs/pushpop

I actually hadn't been using Cover, but I'm a big fan of Aviate. I greatly prefer it over the stock/Samsung experience.

JSON originated from Javascript alone

We emphasize "event data" which goes beyond blanket monitoring. This is "big data" because what you're tracking could still happen at high volume and velocity. We leave the third V, variety, up to you - ultimately you will know your business best and can create and extend the data model you need. Making a backend code change to add a single event collection could immediately lead to millions of very rich data points!

My colleague wrote an excellent blog post on about the event data approach and I think you might enjoy the read! https://keen.io/blog/53958349217/analytics-for-hackers-how-t...

The tutorial that helped me figure out how to use dynamic JSON data was the one on Path Transitions: http://bost.ocks.org/mike/path/

Is this entirely an open source project, or are there plans to do something like Keen.io?

Sounds like solid advice to companies - what about advice for young engineers?

I can certainly relate to the challenges of working on a big data platform intended to immediately satisfy varied customers...

But mainly I want to say I'm super impressed with the Google Analytics 'storification'. I can imagine the difficulties in bringing that level of quality to myriad data sources, but I'm excited to see you succeed!

I'm still surprised there is no SVM implementation for Mahout (http://stackoverflow.com/questions/10482646/recently-svm-imp...).

The article explicitly says "Characters with the same name can exist on different servers, so the mind-boggling 3.8 million unique names found in the dataset was not expected."

The comments on the source point out the same mechanism that aims to keep the US from killing him could also been seen as incentivizing the US protecting him from people who want everything released right away.

Framework for email marketing campaigns, but with a bunch of features.

http://www.exacttarget.com/products

(My company recently switched from Blue Hornet and ExactTarget gives us much more control)

It sounds like that would fall into the "Updates" category

Well, except that we don't pay as much attention to every human being.

Feels like a filler post to hop on a trend. Also, I just can't agree that MapReduce was a bad idea in the first place. Especially when the author goes on to laud the ecosystem built on top of MR/HDFS.

This is well done! An internal project I'm working on will soon need to provide documentation like this.

I think you could do something really interesting simply by connecting feed usage data and content data, which (I believe) was zarify's idea of 'Pandora for newsfeeds'.

There are companies along this line[1], but I'd prefer a solution with just rss usage/content data, and not the social streams as well.

[1] Prismatic and Percolate come to mind

It's a SDK for mobile apps (Android/iOS). There doesn't seem to be a page with every product on it, but the product link at the top leads you to explanations like: http://try.crashlytics.com/reports/

I've dreamed of building something like this for the analytics platform I'm building; my jaw dropped watching the demo.

Really valuable article. As a junior employee I find I'm treading a fine line trying to practice these values, especially when the 'culture' isn't always trickling down. I'd enjoy reading more about this stuff from that perspective, which I suppose falls into the category of 'managing your manager'?

I want to be an intern! Sadly, I'm no longer a student.

Boston, MA | Warner Brothers - Turbine

We're looking for more Data Platform Engineers

We're building a truly massive platform to support all Warner Brother's game studios: we want to measure everything from millions of concurrent users across multiple platforms. Yes, you get to play with data from Mortal Kombat!

We're using:

Python/Java/C/R

HAProxy/Nginx/Redis/Storm/Hadoop/Vertica/Cognos/Tableau

Everything in AWS (Puppet/Fabric)

If you're passionate about video games and data or have questions, contact pnachbaur@turbine.com

Full medical/vision/dental, 401k matching, etc