ktamura

Hadoop didn't fail us. We failed Hadoop.

The decline of Hadoop as a software category is Software Product Marketing 101: it did not identify pervasive killer use cases critical to running . Yes, it's true that Hadoop was a revolutionary way to store and process massive datasets on commodity hardware, but what's the use case for that? If you are Visa/AMEX (fraud detection), Facebook/Google (various ML-based data products) and a few other types of companies with obvious applications of massive data processing, yes, Hadoop has been great.

But here's the thing: beyond a few such corner cases, it never found a use case that enterprise data warehouse couldn't handle.

Then came Redshift, then BigQuery, and now Snowflake (as a BigQuery on AWS, really). While there are some key technical differences between Redshift and BigQuery/Snowflake, they are all _much_ cheaper than the previous generation of data warehouses (Vertica, Netezza, Greenplum, etc.) The lower price meant greater access, and developers who previously couldn't imagine using data warehouses could finally spin one up with a credit card swipe.

Hadoop, too, took a lot of collateral damage because many developers realized that they didn't need much of Hadoop beyond SQL-on-Hadoop.

Redshift was a beautiful feat of product strategy and marketing: They just took what used to cost a lot and offered it for much less in an environment where developers already had a lot of data (AWS). This was much simpler to execute than what Hadoop had to do: introduce new technology, identify use cases, and finally compete with incumbent solutions.

We failed Hadoop (as you can see from www.cloudera.com, even Cloudera, the Hadoop company, hardly mentions Hadoop on its top page). Not the other way around.

Thanks! Airflow was/has been a great source of inspiration =)

If simplicity and non-Python-centricity matter, I encourage folks to look into Digdag [1][2].

It's Ansible for Workflow Management.

While both Luigi and Airflow (somewhat rightfully) assume the user to know/have affinity for Python, Digdag focuses on ease of use and helping enterprises move data around many systems.

If we learned one thing from today's S3 outage, it's not enough to use multiple cloud infrastructure providers: you should probably have your data in multiple cloud providers as well.

[1] https://www.digdag.io

[2] https://github.com/treasure-data/digdag

So glad to see this. As someone who learned both Excel and SQL on the first job (it's called being an analyst at a trading firm), I always felt that SQL was a "more powerful but clunkier cousin" of Excel and wished there was a tutorial for SQL aimed at Excel ninjas.

Well, fast forward half a decade and nobody wrote it, so I did so awhile back. Of all the things I wrote for work, it continues to be the most popular:

https://blog.treasuredata.com/blog/2014/12/05/learn-sql-by-c...

>Cortana is actually pretty cool.

A fellow Windows user (with a decent experience on Linux as well). Is there a way to use Cortana from PowerShell/cmdline?

>The business itself may not be a great business due to the amount of cost it takes to run it -- but it's necessary for the running of other ventures.

The latter half of the statement applies to Stripe as well. The big difference between GitHub and Stripe is that Stripe's business model is more directly tied to the growth of entrepreneurship economy: the more money goes through Stripe-powered businesses, the more money Stripe makes. In GitHub's case, the correlation is less direct and less predictable.

When Amazon Athena actually matures, wouldn't it solve at least the interactive query needs, probably at a much lower/elastic price point than Qubole?

The real question to ask is, will Amazon contribute back to open source? Presto itself is plenty proven and scalable: after all, it was created at Facebook.

It's not Presto per se, but running any data processing workload against unoptimized data formats is the issue.

Then again, both BigQuey and Snowflake require that you move data into their storage engine (Redshift too), and that's an additional step that's proportional to the size and complexity of your data. At the same time, it's stupid to store your logs as OLAP optimized formats and completely lose legibility. In sum, Athena trades off performance for convenience.

No matter what database vendors say, you can't defy the principles of computer science.

I did. As a matter of fact, I was this close to going to Rutgers to study combinatorics and graph theory for a Ph.D.

Once I realized that I was not going to be that good at math (I bailed on a Ph.D program a few months before I was supposed to start), I went into quant finance to make money. I was okay at it, but not great. From there, I somehow ended up in the software world (mostly, I'd say, because I learned to program as a quant trader, certainly not through my coursework).

I ended up in marketing by chance: I am the 1st employee at the current startup and naturally evolved into that role because I am technical but not an engineer and like to write/say stuff.

Marketing, especially product marketing, is one of the most under-rated and under-recognized role in the startup world, but in many B2B businesses, it's just as important (or so I'd like to believe) as engineering and sales.

I'm still amused/saddened by how little theoretical work you need to do at Stanford to get a statistics BS (they call it Mathematical & Computational Science). I get it: you don't need a rigorous foundation of math to be useful as a workaday "data scientist". That said, university is probably the last time you get to indulge/immerse yourself in deep, rigorous, mathematical thinking. Not requiring students to take theoretical courses is such a pity.

In hindsight, I am glad that I studied math at Stanford (also I ended up doing something completely different: marketing). It pushed me to think deeply and patiently about a problem at length and taught me how to mix intuition with analytic rigor.

I believe it was Paul Graham who said that math was one of the better subjects to pursue in school because it's one of the most difficult. I kind of agree with pg on this one: I was nowhere near the top of my math classes, but whenever I took theoretical CS or stats classes, I found them shockingly easy and did well with much less effort.

I still stand by my earlier comment: https://news.ycombinator.com/item?id=12915345

TL;DR; It's probably a good thing for Silicon Valley, considering the alternative outcome (all other things equal): Trump still winning the election without a single Silicon Valley influencer.

I wasn't talking about Palantir specifically but the valley in general =)

This is a stroke of genius by Peter Thiel. In hindsight, he was hedged. If Clinton won, his Trump endorsement would have been chalked up to his contrarian, eccentric bent. If Trump won -and he did- Thiel's bet would have a huge return.

His self-interest aside, this is a huge deal for Palantir, the company that he helped found and has many federal governments as their client. Also, in a sense, Thiel ensured that Silicon Valley didn't completely lose touch with the White House and will have some leverage in influencing them. For starters, immigration is a key issue for Silicon Valley for which Thiel can be a voice of reason to influence the new administration.

I don't agree with many things Thiel says or does. That said, this was a brilliant execution.

Yeah, this is a great move. That said, again, here's the philosophical difference: with BQ, you need to "Contact your sales representative if you are interested in flat-rate pricing". There's nothing wrong with this, but it reflects their philosophy that compute should be something pay-per-use, not pay-to-procure.

Again, I expect this "talk to sales" part to change in the future as BQ matures, but I imagine this kind of CIO friendly thinking to be new (if not de-prioritized) at Google.

Before this thread turns into a vim-v-emacs-esque flame war.

I really think it's a good thing that AWS and GCP are punching each other in the cloud data warehousing market. It means that the market is maturing, and we are all benefiting from their arms race against each other.

I am of the opinion that Redshift and BigQuery are philosophically different enough that performance differences, while important, shouldn't be the deciding factor. I've written about this in a blog post awhile back, and it might be relevant for folks weighing their options

https://blog.treasuredata.com/blog/2016/06/09/redshift-bigqu...

Disclosures:

1. I don't work at either.

2. My employer, however, partners with both.

What are you using for container/pod logging?

This is just great.

I have a couple of mutual friends with Jess Lee from my time at Stanford, and no one from back in the day thought she was exactly the type to become a venture capitalist.

The rest of her story is probably more well-publicized: choosing a PM job at Google as opposed to an engineering job at Intuit, turning from a No. 1 Polyvore fan to its CEO who led its acquisition by Yahoo! And now, she's the first (honestly, I am surprised that she's the first) US investing partner at Sequoia.

There are many things I am not particularly fond of about Silicon Valley, but her well-deserved ascent in the valley embodies so much of the positive qualities about Silicon Valley.

Unsurprising at all. All businesses/sectors are at the mercy of policymakers, and it's natural to invest some of the newfound wealth into influencing public policies. The size of lobbying budget is more or less a proxy for the size and the growth of the sector.

The irony, of course, is Silicon Valley's holier-than-thou self-righteousness in relation to Wall Street and/or lobbying.

Their dataset reeks of startup hustling.

I just downloaded the training set [1] and plotted some of its descriptive statistics [2]. It looks that all features are uniform distributions and the response variable is Bernoulli coin-flipping. In layman's terms, you can't really come up with a good predictive model with this training set.

I give them the benefit of the doubt that they wanted to have something in place to push the website live, but I cannot imagine any serious data scientist not noticing this.

[1] http://datasets.numer.ai/57feb95/numerai_training_data.csv

[2] https://cl.ly/051G3Y2Z2O0W

Over the years, I've tried many digital to-do lists, including one I wrote myself. Today, I just use a notebook.

The trick that I made work is to have two to-do lists: daily and quarterly.

For daily, it's simple. I have a variation of the Bullet Journal [1], but interleaved with meeting notes, ideas, etc.

But many ideas are bigger than what you can accomplish/benchmark on a daily basis. For those, I have two pages per quarter that I fill with "quarterly goals". I try to keep the list under 20-30 items and keep them high-level. Every week, I look at the same two pages and make sure that I am making progress toward them. If not, I go back to my daily notes and see if I worked on the right stuff the previous week. This is a variation on the idea that I stole from Aaron Levie [2].

For me, the tactile/tangible nature of paper note-taking, as well as its flexibility in formatting, has been incredibly useful.

[1] http://bulletjournal.com/

[2] https://www.fastcompany.com/3021586/most-productive-people-b...

"The Power Broker" by Robert Caro.

At first glance, it's a 1,000-page, detailed biography of a (in)famously effective city planner Robert Moses. At its core, it's a lucid examination of the anatomy of power. It has completely changed how I think about power, where it comes from (whether it's at work or in the greater market/world) and how it is retained and lost.

It's also a great antithesis to junk-food journalism and online reading that we've become addicted to. Caro's research is thorough and his writing is inspiring.

>That's not what happened at all. He built the system entirely himself before getting in contact with us.

Good to know. While what I said was meant to be hypothetical, I am happy to stand corrected.

>creating a neural network that can do real work is within the abilities a single individual.

This really depends on said "individual." My biggest issue was the tacit conflation between two themes, one valid and another reeking of "feel good" marketing:

1. That a reasonably technical person can use neutral network to do useful things, no small part thanks to something like TensorFlow (valid)

2. That such individuals are prevalent in agriculture (???)

I am all for hero-making: it's at the heart of marketing and customer advocacy. However, as someone who has been doing technical marketing for awhile, I just find this story exceptional in both senses of the word (as others commented, perhaps I turn out to be wrong!)

While it's a cool concept and agriculture is ripe for technology disruption, the enterprise software product marketer in me finds it to be a calculated marketing stunt, if not an disingenuous one.

TensorFlow is, in no way shape or form, ready to be used by anyone but very technical people with a solid knowledge of both software engineering and machine learning. In fact, it's probably unfair to single out TensorFlow because every deep learning toolkit I have seen is still in the same "extremely early adopter" phase. Yet, this article makes it seem like a line of business user (farmer) with some background in technology (former embedded systems engineer) can use deep learning to transform his/her business -when, in reality, much expert help is needed to make it work. In other words, it would have been a lot more plausible/genuine if the article read like:

1. A progressive farmer got in touch with Google.

2. Google dispatched their solutions architect to work with them.

3. Hey, a cool, early prototype is working!

I am not trying to be a hater. Deep learning has huge potential, and Google, among others, is doing a lot to make it accessible. That said, this type of over-promising ahead of market reality is what gives cutting edge technology a bad name.

>A Docker distro of Kubernetes would do very well in enterprise on-prem or private cloud environments. They already have a great developer experience. Companies will pay for support on both.

Some early data here: https://twitter.com/dberkholz/status/720686568671940608

I cannot recommend using Slack via a browser enough. It achieves several goals:

1. It sandboxes resource usage, preventing Slack from being a permanent resource hog. Is your Slack tab running hot on Chrome? Just kill it for now. There are many browser extensions for managing resource usage per tab.

2. Closing the Slack browser tab means no disruptive notifications. Sure, you can snooze Slack @-mentions, etc., but that's too much of "working hard to make the tool work."

3. Better workflow (seriously) Perhaps this is just me (or my function as a marketing person at a startup), but I work almost entirely inside the browser (GMail, Google Docs, various sites for research, SaaS apps). Using Slack as a standalone desktop app means I have to focus away from the browser. Using Slack as a browser tab means I can treat it just one of several web apps I use regularly.

4. Bonus point: No need to update your Slack client =)

Neither am I =) It's a matter of doing fact-checking/analysis.

Some of the juxtapositions/language are bizarre.

>2. Of the YC companies that use hosting providers, 55% use AWS, 13% use Cloudflare and 6% use Rackspace.

Comparing Cloudflare and AWS? Where's GCP or Azure?

>3. Of the YC companies that use a CDN, 68% use Amazon CloudFront, 7% use Cloudflare and 5% use Fastly.

Okay, so there's a separate section for CDN...

>8. Only 22% of YC sites use a third party CMS. Of those, 83% use WordPress.

"3rd party" CMS? As opposed to...a custom made web app?

>11. In terms of installs, the most common Analytics providers are Google Analytics (451), Google Universal Analytics (314), Mixpanel (183) and Optimizely (129).

I thought Google Universal Analytics was an updated version of Google Analytics.

That said, this kind of market research is incredibly valuable.

I've always found Palantir's sales&marketing strategy to be unique and fascinating among Silicon Valley startups.

Palantir is a high-touch, B2B enterprise software startup, and as such, they are no different than other startups when it comes to sales&marketing. They need to find leads (= potential customers), close them as customers and retain them.

Yet, Palantir's leadership and employees have repeatedly emphasized that they don't do sales or marketing. The OP confirms that they meant that they do sales and marketing differently than other companies.

Retaining lobbyists and policy advisers is one example. Typically, enterprise software companies retain industry analyst firms like Gartner, Forrester, etc. and "influence" their opinions via briefing/consulting. These industry analyst firms also work with technology buyers and make recommendations to them based on their knowledge of software/service providers and the needs of the buyer.

To be sure, no analyst firm is 100% pay to play because if they favor select vendors too much, they lose credibility, access to information and hence influence. That said, it's naive to think that they are 100% free from monied vendor incentives.

It did not occur to me until reading this post that the same model applies to lobbying, and that's where Palantir invested their influencer marketing budget.

Palantir often emphasizes their technological innovation and unique culture (again, a great marketing strategy to attract great technical talent and put them in sales engineering roles). However, I've come to believe that Palantir's greatest innovation is in their business strategy.