kiyoto

This rings so true. Last April, I had a chance to chat with an engineer at an (explosively growing) early stage B2C startup. He gave a talk about "data science", so I walked up to him afterwards and asked him if he considered himself to be a data scientist and did anything that's remotely like statistical inference.

He said, "nah, most of what I do is collecting a lot of data and just segmenting them in a bunch of ways. Pivot tables go really far as far as product insights are concerned."

Then, he added, "that whole data science thing in my talk...that was to grab attention."

I am NOT saying statistics and machine learning is useless. They are tremendously useful. But chances are good that it pays a lot more early on to collect a lot of data and ask basic questions from different perspectives.

Sorry that my point wasn't clear: it's not so much that I experience issues with in_tail. in_tail is actually solid at detecting log rotation, etc. My point was this is not a trivial problem, and a lot of the code inside in_tail.rb is for detecting and handling log rotation gracefully.

I see what you are getting at, but my experience as one of the "mailing list" people for Fluentd (an open source log collector) has been a bit different, especially when a lot of data is written to a file rapidly: For example, when you try to tail a log file with log rotation reliably, you run into all sorts of edge cases. Such issues shouldn't exist if files and streams were truly interchangeable in the context of logging.

I used to think the same. But after meeting with some of its engineers and designer (yes, one), my perspective has changed quite a bit. Yes, many things they do range from insane to socially unacceptable, but they've got the brains.

Also, while they are making a lot of mistakes, they are getting a few key things very right as a startup: 1. building a strong culture and brand 2. dominating a niche before expanding out to other verticals (Geoffrey Moore's bowling ally theory).

Enough people use it to have the company valued above a billion dollars, have their first MongoDB World (http://world.mongodb.com) and have a HN darling like Stripe as a power user.

Sure, MongoDB has its shortcomings, but so does <Someone's favorite database>. It's much more productive to understand the pros and cons of using it and use it (or not user it) appropriately.

>A common pattern for people using Samza is actually to accumulate a large window of data and then rank using a complex brute force algorithm that may take 5 mins or so to produce results.

I think the argument eventually comes down to what people mean by "batch" and "stream". Some people might describe the aforementioned Samza use case as (micro-)batch as opposed to stream processing.

All in all, I found your post insightful. If a system with fewer moving parts can handle the same data processing requirements, that will be appealing to the majority of users.

As someone who has one foot in marketing and one foot in development (I am a developer evangelist), here is what's happening.

1. Google's infrastructure is evolving atop MapReduce (FlumeJava/MillWheel).

2. Google's PR decided to call it "not using MapReduce anymore" because in marketing, "beyond <current fad>" sounds really cool.

3. The rest of the world's PR/press/marketing fall for Google's clever PR.

Either way, it is great to see Google making its core technology accessible as part of their PaaS =)

> I've seen some videos of real Vim pros at work

Yeah, I know one of these people too. I know one for Emacs, Eclipse, Excel and of course SublimeText.

Honestly, I don't see why some folks obsess over text editors/IDEs. Familiarity and mastery matter way more than features. Sure, a truly barebone editor like Notepad.exe or nano might be insufficient, but any editor with a decently-sized following has enough features to make its users comparably productive - as long as you learn how to use them (This is almost a self-fulfilling claim: if an editor has a sizable following, that means it got something right for its users).

I mean, how else do we explain that Rob Pike uses Acme, Joshua Bloch uses Emacs to write Java, and Paul Graham uses vi to write code and prose?

>We were able to reach the speed we needed by writing our own Go client to ship off the logs to our cluster

That's a fair argument: when written intelligently, Go has great performance.

The other reason is performance v. flexibility. Any generic log collector (Logstash, Flunetd, etc.) needs to support multiple data sources and outputs and is probably less optimized for the specific situation at hand.

Just curious: what's the buffering strategy for your custom agent written in Go? Does it support file-/memory-based buffering?

Fluentd maintainer here. A lot of folks indeed use Fluentd for application log collection. Perhaps because we embrace JSON as the core data structure, MongoDB is a common data destination (Second most popular plugin) http://www.fluentd.org/plugins/all

It might be a case of the Dunning-Kruger effect. Either way, I am willing to concede that I misused the term (perhaps I am the one being an idiot here). I am still surprised that the blog post got published on Uber's blog though.

Thanks for a great summary.

>For logs, you can either use a shared directory with the host or use a remote log collection service like logstash or papertrail.

I would not put Logstash (software) and Papertrail (SaaS) in the same bucket though. With Logstash (or any other data collectors like Flume, Fluentd, etc.) you need a backend system to go with it.

Sorry to nitpick, but logging is something I work on/think about all day, so I thought to point it out.

I agree. Not knowing anything does not make anyone an idiot. Not realizing that you do not know something, however, kind of does.

The author probably has no background in experimental design, so I would spare him of statistical scrutiny. This includes the author's furtive attempt to sneak in causation into a simple regression.

Or, if I were to give the author of the benefit of the doubt, Uber realized that this potentially inaccurate statistical claim ("Uber is inducing a drop in DUI") can help them win people's goodwill in areas where their operations have been thwarted/stunted by bureaucracy.

If so, they are really shrewd (for most laypeople don't understand anything about statistical inference), but I am not sure if I would trust a company that resorts to such a disingenuous campaign.

I actually want Uber to be a success. I love their service and envision it to be a real-time bidding platform for all sorts of (offline) services. But at the same time, I would be concerned if a company of such great promise is already acting intellectually dishonestly.

Hence, I really, really hope the author is just an idiot.

I love the contrast between the 1st and 2nd link on HN right now (2014/06/06 10:49AM)

http://gyazo.com/b5fb5ded81f2aab53cd4185c65158afa

That's arguable. I never had a chance to estimate this in earnest, but I have the hypothesis that the majority of HFT transactions are with one another. Sure, they collectively provide liquidity to non-HFT investors (because the order flow has to come from somewhere), but I also had the hypothesis that if the clearing houses (CME/NYSE/etc.) could improve their matching engines and/or consolidate disparate exchanges, that might solve the liquidity problem without HFTs.

I want to shed some light on this topic. As a former HFT quant, I find the general "Main Street" sentiment toward the profession to be misguided.

1. HFT does NOT take anything from most investors most of the time. Most investors trade on a much longer investment horizon than any HFT firm. If you are a normal investor, you can lump up most of HFTs into the same bucket as the exchange itself. Most of them engage in some combinations of (i) pure arbitrage (ii) very short-termed statistical arbitrage (iii) market-making based on (i) and (ii). None of this is really that relevant to most investors. The only time HFTs can screw over a lot of investors is when their software goes awry. But this is a risk that any computerized system has, high frequency or not. If there is a major bug in Chicago Mercantile Exchange's matching engines, that would be a total disaster.

2. HFT firms do have huge execution risks: if you are making a two-sided market, there is a chance you can get "swept", meaning when the value of the underlying moves faster than you can react, you get filled on one side of the order without a realistic chance of hedging for a profit. HFT firms do a ton of research into estimating their execution risks. (edit: This point is often glossed over, making HFTs look like this evil superpower group of nerds exploiting other investors. That's not the case)

3. Finally, most HFT traders don't go into the profession solely for the money, just like most people do not apply for YC solely for their passion. Most of my former coworkers weren't that greedy and led pretty modest lives despite making hundreds of thousands of dollars. For them, high frequency trading had a locally optimal balance of tackling intellectually challenging problems while getting paid handsomely.

I won't say HFT is the most valuable thing that its practitioners can be doing. I think the (trading) world would operate just fine without all these micro-second level transactions. But they are NOT the next subprime mortgage, and I just hope folks stop commenting on stuff they have no clue about.

Sorry for sounding like an ass, but I thought I would provide my slightly different perspective, precisely because Japanese was not particularly remarkable pronunciation-wise (but still interesting semiotics-wise)

I contribute to one of the Nikkei publications.

Thanks for teaching me how to pronounce Japanese words! I am not all that familiar with the language. My limited experience includes being born in this little clubbing district called Roppongi, growing up in this little city called Tokyo until I was fourteen, and having read somewhere between 3,000 and 5,000 books in it. Even with such a limited knowledge of Japanese, I ostensibly have a monthly column in a nationally circulated magazine.

I didn't say this was a great example to showcase the nuance and intricacy of Japanese. All I said was how semiotics was interesting in general, my observation being an (admittedly boring) example of it.

In Japanese, the sign itself is called "attomaaku" ("at mark") but it's pronounced as "atto" when dictated. So someone's email would be johnsmith-at-gmail-dot-com, and if you ask a Japanese person to pronounce the symbol, they would say "atto". However, if you show them the symbol and ask them _what it is_, they probably would say "attomaaku".

Semiotics is fascinating.

A great post!

>When building a binary classifier, many practitioners immediately jump to logistic regression because it’s simple. But, many also forget that logistic regression is a linear model and the non-linear interaction among predictors need to be encoded manually. Returning to fraud detection, high order interaction features like "billing address = shipping address and transaction amount < $50" are required for good model performance. So one should prefer non-linear models like SVM with kernel or tree based classifiers that bake in higher-order interaction features.

I am not seeing the argument here. Can we just not encode binary conditions with dummy variables?

Wow. If my HN top page analysis is correct, this story went from 5th place to 24th place in a minute. Probably got penalized for ring-voting?

Thanks a lot!

Yea, I felt like a total n00b when I came to the web startup world a few years ago. This sounds ridiculous, but one and only database I had used until that point is kdb+ (kx.com). I had no idea about the performance/tradeoffs of any other databases.

I agree with you that properly-structured SQL databases can scale horizontally/vertically. That said, I've noticed that the set of people who know SQL performance well and the set of data analysts/statistically inclined folks do not overlap much (myself included), and frankly, data analysts should be able to focus on analysis, not SQL optimizations.

In a way, this is the problem Impala (and other MPP databases) solves at many companies: it's not that their data analysis cannot be handled with MySQL/Oracle, but it's cheaper and quicker to throw all the data in HDFS and query via Impala (sans some cost associated with setting up/maintaining Impala).

I write a lot (in Japanese) on the web and have had some meaningful traffic. I think the most I got on a single day is like 150k visitors. And I've tried everything from raw HTML to SSG to Wordpress to a homegrown CMS.

The truth of the matter is, for most web writers, any CMS is plenty performant. You probably won't even need a cache, and if you end up needing one, it's usually pretty straightforward to add it. And it's pretty self-evident to me that updating one document and saving it is easier than re-generating the updated parts of your website with some script before git-this-git-that.

One good argument against CMS is security: many of them have a history of security woes. This is partly why I ended up writing my own very minimal CMS engine. All it does is read some markdown files and render them. For me, that achieves both security (or at least if not, it's easier for me to go through my 100 lines of Ruby code than the Wordpress codebase) and low friction.

>A side benefit is that when someone says "What's the probability that the new purple dragon logo outperforms the old one?", you can give them an answer without backpedaling and explaining null hypotheses, p-values, significance levels, and all that jazz.

This is surprisingly valuable. As a former quant/math person, I am shocked time and again how most self-professed data driven people have no idea about inferential statistics. I've learned over time that most people just want to see data and descriptive statistics, preferably in visual representations, and interpret them somewhat creatively and pretty much non-rigorously.

This is really cool. I can tell you guys actually know statistics.

w/r/t testing multiple hypotheses, I've long wondered why false discovery rate never gets mentioned in the A/B-testing context. Any thoughts?

Thanks for sharing this. I think the truth lies somewhere between annual prepay and month-to-month. Most SaaS companies end up offering a mix of the two =)

As someone who works at a SaaS company, the annual prepay has one downside: it narrows the middle portion of the funnel (qualifying leads) compared to month-to-month.

Of course, the biggest selling point of SaaS over licensed software is service: not having to manage your software (and often hardware) is a huge pain reliever for the buyer.

The other selling point is lack of financial commitment (at least with month-to-month): you do not have to buy into the service for the next whole year. Instead, you can try and see how it fits into your business. If you see no fit or can't get value, you can cut the loss and move on. You cannot do this with the annual prepay model.

Time and again, I've been surprised how customers who were unwilling to commit to a year ended up prepaying for a year when the time for renewal came. If you are forcing the annual prepay model to your customers, you are kicking this type of "cautious at first but eventually valuable" customers off your sales funnel.

A popular way to combat this is free trials (anywhere between two week and ninety days). This is a decent solution, but I've seen first hand that whether one is paying for a SaaS strongly influences usage/penetration into the customer organization.

edit: minor edits for clarification

SQL is scary.

I've talked to hundreds of customer prospects for Treasure Data, and by far the biggest surprise I encountered is how few people know their way around SQL. I am not talking about just "business" people but also programmers.

Contrast this dearth of SQL-proficient population with the massive, massive Excel user base: I jokingly tell my friends that Excel is by far the most popular programming language =p

I agree. We hired our frontend engineer through HN. And it was surely not a dedicated link like the ones YC companies get to post but one of those monthly job boards =)