kiyoto

I thought this was the project that aaronsw started awhile back? Is it coming back to life?

For all his shortcomings, LKY has shown the world how well a tiny city-state can do economically with geopolitical leverage and strategic thinking. Especially when it comes to diplomacy vis-a-vis the West, most Asian political leaders are confused (and confusing) at best and belittled (and bewildering) at worst, but no leader of the West fucked with LKY in his prime.

https://www.youtube.com/watch?v=VexrmTacOAA

Once in awhile, I watch that footage of Meet the Press to remind myself what a sharp political leader he was. RIP the benevolent dictator.

Can you share the specifics of your experience (the cluster size/use case/etc.)

I am partly asking this because you clearly feel strongly about this topic (your account was created an hour ago, most likely to comment on this).

I agree with this 100%. SQL works incredibly well for purely declarive, relatively simple operations. However, the moment you start doing "Do this, then that, then that", it starts to look (literally the appearance) daunting.

I myself come from the R/Python background, and my hypothesis is this difficulty around composability (especially for dplyr/pandas crowd) is what deters data scientists from writing more SQL.

Here is the github repo: https://github.com/airbnb/airpal

Treasure Data, Mountain View, CA.

We are looking for a Growth Engineer to help grow our B2B SaaS business (15M Series B in Jan 2015)

You are:

- resourcefully relentless and relentlessly resourceful.

- a big believer in using programming to automate tedious tasks.

- technical but most interested in having direct business impact.

- not afraid of digging into the data to prove/disprove ideas.

We are rebuilding our growth strategy, and you will be a core member of this effort.

If interested, please email me at kiyoto@treasure-data.com with an example of an intelligent growth tactic you have implemented.

That's exactly what I am talking about. Your anti-example is a case of library-level abstraction. Now, if you are saying that JavaScript doesn't lend itself to writing extensible machine learning routines whereas Lua does, it's a different story.

"I wanted to write something that is concise, easy to understand, easy to extend, and reasonably efficient. There is a subtle trade-off between conciseness and extensibility: If we use a very high level language that already has a "neural_network_train" function, we can write very concise code but we lose the ability to change the training algorithm. If we use a very low level language that only provides primitive arithmetic operations, all the algorithm details are exposed and modifiable but the code is bulky and difficult to understand. For a happy medium, the code should reflect the level at which I think of the problem"

The tradeoff is not between concision and extensibility, but high- and low-level computations.

Even if the language natively implements a "neural_network_train" function, as long as the language also offers low-level primitives to implement all the necessary parts of the neural_network function, the language is no less extensible than the OP's suggested alternative. For example, almost 100% of R users use "lm" to run linear regressions, but R has all the necessary pieces to implement the linear regression calculation (either by inverting matrices or running iterative gradient descent algorithms)

The OP conflates the library-level abstraction and language-level abstraction. I am with him in that there is a trade off between concision and extensibility w/r/t language-level abstraction. The library-level abstraction is pragmatically important (i.e., you would not use OCaml to run websites) but theoretically uninteresting (Ocaml can certainly express all the needed computation for a web server).

Thanks for the good feedback.

>I agree that the problem of schema rot is an issue

I am glad to hear that, and as someone who once landed a patch on PHP (proof! https://bugs.php.net/bug.php?id=61124&edit=3), I feel terrible that it read like an advert. I will consider writing a more technically sound followup.

Curious: do you feel this "schema rot" pain today? If so, can you elaborate on it?

It really isn't (Disclaimer: I work there and wrote the entry)

In fact, you can replace Treasure Data with anything that is:

1. Schemaless storage

2. Allows you to output results easily and reliably to MPP databases.

What I noticed is that a lot of people tout various MPP engines, cloud or on-premises, as the only storage layer with any queryable interface that they need, and we think that's a bad approach.

Some folks are starting to realize this. For example, here is an article by Martin Fowler: http://martinfowler.com/bliki/DataLake.html that talks about the concept of "data lake". The problem is, most data lakes are utterly inaccessible to the data scientist/analyst camps.

EDIT: some grammatical clarifications

You clearly work in the web industry =)

perl does one thing hundred times better than (server-side) JavaScript: when you need to get data out of a server that you have no sudo privilege because your boss asked you to get stats on your trades ASAP.

This happens shockingly frequently in finance, and perl (and actually, awk) was the only scripting language that I could safely assume was installed.

Some folks in the Fluentd community uses Graylog2+Fluentd as an alternative to EFK (Elasticsearch Fluentd Kibana) See http://www.fluentd.org/guides/recipes/graylog2

>Looks good for the basics, but as some other folks mentioned here, it would benefit from some articles about design, and even more importantly, rationale.

I agree about rationale, but probably in a different way. For a great many people, especially in the "data analyst camp" (i.e., people coming from finance/consulting background with mind-blowing Excel skills), SQL is not something they want to learn in the vacuum but more like necessary evil. For them, it's hugely valuable to learn how and why SQL can be useful.

http://blog.treasuredata.com/2014/12/05/learn-sql-by-calcula... is my humble attempt to address that issue, and it seems to have resonated well with the data analyst type folks.

Just offering a non-programmer's perspective.

Interesting. This can be used with Fluentd as well with the JSONFormatter (https://www.fluentd.org)

Treasure Data (https://jobs.lever.co/treasure-data), where I've been for the last three years, employs several people to work exclusively on Fluentd, an open source project (https://www.fluentd.org)

We do this for a strategic reason, which I outlined here: https://www.fluentd.org/blog/unified-logging-layer

>More interestingly, things like revenue/visitor have a known probability distribution.

This actually depends. Based on my experience at a very early stage startup, it was definitely not the case for some datasets (I even tried fiddling with various well-known distribution's parameters).

If I recall correctly, I believe that the bootstrap had some asymptotic guarantee on the rate of convergence (although my memory is hazy on this)?

EDIT: never mind, it is asymptotic, hence not finite-sample necessarily.

I agree. Bootstrapping is somehow not covered in most "first course in statistics" yet it is very valuable in practice for data people, especially at startups: Because you don't have that much data, and most of your data do not naturally follow anything like the normal distribution, it could be misleading to use the normal theory to estimate variance. The bootstrap helps you combat this.

For the logging layer, have you considered/looked at Fluentd? Disclaimer: I am a Fluentd maintainer and would love to hear feedback from folks.

Some people like using Grafana with InfluxDB backend and Fluentd for log collection.

A great article. A couple of open source projects for CEP/event processing.

1. Esper (http://www.espertech.com) has been around for awhile. This is the CEP engine a lot of people are familiar with.

2. Norikra (http://norikra.github.io/) is a schema-less event processing engine, often used with Fluentd (https://www.fluentd.org) as the data collector (Fluentd can do stream processing as well).

3. Apache Storm (https://storm.apache.org) has been popular in the Hadoop community, often with Kafka as the event source.

Not quite. I used to be a 'data scientist' in finance. Their jargon for this position is "quantitative analyst".

The two definitely have intersections. Here is what's in DataScience \cap Statistics

1. A solid understanding of basic inferential statistics.

2. A healthy dose of skepticism about drawing conclusions from data.

3. The ability to model the problem and communicate how data fits into the model.

Here are the skills that most data scientists I know do NOT have but statisticians do.

1. A solid mathematical foundation of statistics: I am guessing because this stuff is only taught in graduate-level statistics classes, which most data scientists have never taken.

2. The research frontier of statistics, or the "what's new" of the statistics academia.

3. Coming up with new statistical methodologies (or discovering anything new and contributing to the foundation of statistics). This is not entirely true, but most data scientists I know are users of existing techniques/ideas.

Here are the skills that most statisticians I know do NOT have but data scientists do.

1. "Vocational" programming skills: most statisticians know R/Matlab, but many of them lack the experience of acquiring and cleaning data before doing analysis. I don't think this is a matter of ability but experience and lack of exposure.

2. Communicating statistical conclusions to non-statisticians: some of my statistician friends have a knack for this, but it is a skill that's hugely important for data scientists. People who want to be data-driven and ask for statistical conclusions far outnumber people who can actually understand statistical subtleties, and almost by survivors' bias, all data scientists I know are excellent at getting their conclusions across the organization even if others do not fully understand the full ramificaiton.

(Disclaimer: I am a Fluentd maintainer)

Here are some comparisons out in the wild (Note: neither of them is a Logstash or Fluentd maintainer afaik)

- http://jasonwilder.com/blog/2013/11/19/fluentd-vs-logstash/

- https://blog.deimos.fr/2014/05/13/logstash-vs-fluentd/

Also, if you have any question or doubt about Fluentd, please feel free to email me at kiyoto@treasure-data.com

It is not "better" but different.

1. Easier to extend that syslog-ng if you have a modest knowledge of Ruby

2. Easy to configure file- and memory- based buffering and failover.

3. Advanced filtering out of the box.

4. Rich plugin ecosystem with 300+ plugins.

At least that's what I've heard from the users who switched from syslog-ng to Fluentd. I am happy to learn more about what makes syslog-ng great since I've never used it seriously myself =)

Glad to hear that! I wrote https://www.digitalocean.com/community/tutorials/elasticsear... a while back.

Some people find EFK (Elasticsearch Fluentd Kibana) to be another compelling alternative to Splunk (Disclaimer: I am one of the maintainers of Fluentd)

http://docs.fluentd.org/articles/free-alternative-to-splunk-...

"The bottom line is that we realize we have to continually raise the bar, to get you home from the bar. We are aware of this challenge, from both a technological as well as people perspective, and are doing what we can to fulfill our mission of bringing reliable rides to all who want them."

Yes, because it's really critical to get home from the bar in a state of total inebriation =)

Also, I didn't know giving reliable rides was Uber's mission. I thought it was "evolving the way the world moves." This may sound like nitpicking but if you pause for a moment, the difference, as far as a company mission goes, between "giving reliable rides" and "evolving the way the world moves" is pretty big.

First of all, John Foreman is great. Read his book "Data Smart" and http://analyticsmadeskeezy.com/blog/

(disclaimer: I am in no way tied to John Foreman. Also, I work at a company that provides a data processing/collaboration SaaS...for big data! http://www.treasuredata.com)

A quote from the OP:

>If your business is currently too chaotic to support a complex model, don't build one. Focus on providing solid, simple analysis until an opportunity arises that is revenue-important enough and stable enough to merit the type of investment a full-fledged data science modeling effort requires.

This is consistent with what we see in our customers. The use cases we see most with processing big data boils down to generating reports.

Generating reports may sound really prosaic, but as I learned from our customers, most organizations are very, very far from providing access to their data in a cogent, accessible manner. Just to generate reports/summaries/basic descriptive statistics, incredibly complex enterprise architectures have been proposed, built by a cadre of enterprise architects and deployed with obscenely high maintenance subscription fees billed by various vendors. That's the reality at many companies.

As bad and confusing the buzzword "big data" is, one good byproduct is that it has forced slow-moving enterprises to rethink their data collection/storage/management/reporting systems.

Finally, I am starting to see folks do meaningful predictive modelling on top of large-ish data (in the order of terabytes). Some of them are our customers at Treasure Data, some aren't, but they are definitely not "build[ing] a clustering algorithm that leverages storm and the Twitter API" but actually doing the hard work of thinking through how (or if) the data they collect is meaningful and useful.

And that's a good thing.

As a math major who went to grad school in CS theory, I find the article's title vexing. It projects a faux dichotomy of computer scientists v. mathematicians when, in reality, it's mathematicians v. mathematicians.

Many areas of computer science, at least the areas in which Leslie Lamport is best known, is a strongly mathematical discipline inspired by models of computing. Dr. Lamport probably considers himself as a "computer scientist", but I bet it is really far from what many laypeople (even the HN crowd) imagine to be computer scientists.

Another issue is, mathematicians have as much to do with mathematical pedagogy in secondary education as computer scientists have to do with science experiments in high school. Again, the title is really misleading in this regard.

Anyway, I am glad that a great mind like Leslie Lamport and David Mumford (definitely a mathematician) are thinking hard about how to make math education better.

It depends on where you look. If anything, anime/manga (or anything that is categorized/miscategorized under the Otaku category) are gaining wider adoption in the West by the day, and for music, "Baby Metal" is gaining traction: babymetal.net

>If you can talk intelligently about the whole grab bag of stuff these teams use, that'll get you in the door. Understanding RDBMSes, data warehousing concepts, and ETL is a big plus for people doing infrastructure work.

This is sadly true, for now. I don't think folks here disagree with the "true" part. Let me explain why it is "sadly" and "for now".

The biggest issue with big data is most of it sits unused. In many organizations, HDFS ends up being an alternative to NetApp storage servers, storing terabytes of data with the hopes of them being useful one day.

In fact, if you already get to that stage of using HDFS as a storage server, you must have a decent ETL team that can put data into HDFS with a menacing combination of ad hoc scripts and a workflow that looks like a cobweb produced by a deranged spider. For now, knowing the ins and outs of various semi-functional open source components and the tenacity, patience and skill to deal with the gnarliest of ETL tasks get you a high-paying data engineering job.

But, in the long term, there will be a big change.

1. Tools are getting better: many data practitioners are realizing there are huge gaps between different data infrastructure components, and they are trying to fill these gaps. There is a lot of attention given to query execution engines (Presto, Impala, Spark, etc.) but I find data collection/workflow management tools are just as critical (if not higher leverage) right now. Tools like Fluentd (log collector) [1], Luigi (workflow engine) are OSS software in this direction.

2. Data-related cloud services are becoming really, really good: huge kudos to services like AWS, GCP, Heroku (through Addons). They are quickly building a great ecosystem of data processing/analysis/database components that frankly work better than most self-administered OSS counterparts. (Disclaimer: my perception might be colored here since I work for a data processing/collaboration SaaS myself [3])

So, back to the question. I think aspiring data engineers have two distinct career paths:

1. Becoming an expert in a particular data engineering component: this would be building a query execution engine, designing a distributed stream processing system, etc. (It would be awesome if you decide to release as open source)

2. Becoming an expert on quickly and effectively deploying cloud services to get the job done: this is the skill most desired among data engineers at startups.

What not to become is one of these OSS DIY bigots: not good enough to build truly differentiating technology, but adamant about building and running their own <up and coming OSS technology>. These folks will be wiped out in the next decade or so.

[1] https://www.fluentd.org [2] http://luigi.readthedocs.org [3] http://www.treasuredata.com