Comment by shadowmint | Hacker News Reader

Comment by shadowmint | Hacker News Reader

Parent Full thread

I doubt it.

The rise and fall of data scientists (if they ever rise) will be swift.

Thing is, it's easy to collect a huge pile of data and then retrospectively look at the data set and see patterns in it (very often, patterns that don't actually exist but some over fitted model kind of suggests exist).

It's much much harder to get anything tangibly useful from that data.

Sometimes there is no tangibly useful data. It's all just random.*

'What have you got to show the last 2 weeks of effort?' ... 'Well, we definitely proved that there's absolutely no correlation between our manager's new initiative and online sales; that small spike we saw was within seasonal variation'. How long do you think you'll have a job?

What people need is better tools to auto-classify data into data sets and perform real time analysis of their data (like Splunk, which is amazing), so everyone can look at summary information in real time, at any time, and you can respond dynamically to changes.

* - I know, I know, no data of that sort is totally random. ...but it's often so noisy that no statistically valid information is recoverable.

I work in risk analytics and think this comment is dead on. Tools such as Splunk can wrap metadata facilitating analysis around arbitrary input data (e.g. admin logs, vulnerability scan results, etc). I expect that such tools will reduce the need for some of the work currently done by data scientists.

With that said, there is still a need for human driven analysis of risk data in order to compose models that prioritize and collate incoming signals in ways to assist in risk management decisions. These models are highly domain specific (e.g. a model for evaluating IT infrastructure assets' risk across a global banking business) and are nontrivial to design.

I suspect that data scientists will be useful in bridging the gap between automated risk analytics collection tools and corporate risk management by assisting in coming up with risk models based on the volumes of incoming data.

Edit: or maybe "quants", not "data scientists" will fill this role. Regardless of the naming convention, it's an interesting and complex space to watch.

Nah, data science is AI-complete, like natural language understanding. Think of current state of data science as just the next iteration / rebranding of statistics, powered by newest developments in CS, machine learning and software industry. There is no shortage of statistical data exploration and modeling applications, and we are pretty far from full software automation in this space.

AI-complete, awesome terminology you coined there.

what does it mean?

Solving it would mean solving AI.

If you are at a large company with a data-driven marketing department (hint: the good ones are), the statement

"Well, we definitely proved that there's absolutely no correlation between our manager's new initiative and online sales; that small spike we saw was within seasonal variation"

will be much appreciated and valued. The data scientist doesn't need to report to the head of Marketing, and won't feel any pressure to make someone else's manager look good.

I generally agree that people need better tools. However it is also very difficult to give people the right sort of visualization and analysis tools. If you pick a very, very specific niche, then doable. But people have tried for years and years for more generalized tools and they still mostly suck in general. This doesn't mean it is hopeless just that it is much harder than many people realize.

Good comment, and much of it also applies to vague optimism that some people put into Machine Learning methods.

Reply on news.ycombinator.com