Comment by dredmorbius | Hacker News Reader

Comment by dredmorbius | Hacker News Reader

Parent Full thread

Don't think that "data science" is a solid career trajectory for the next 20-40 years. It's a fad, driven by "big data" and the presumption that a vast an invasive dossier on individuals translates to value through marketing.

The (sorry) data on that is thin.

And it would hardly be the first time that a hugely promising career field has gone bust.

Over my lifetime, I've seen the technical "hot jobs" go from:

⚫ Highway and civil engineering (1960s - early 1970s)

⚫ Nuclear engineering (1970s to March 28, 1979).

⚫ Defense and aerospace engineering (1980s through 1992).

⚫ Doctor / medicine (through the mid 1980s and the rise of HMOs / "managed healthcare")

⚫ Petroleum engineering (1960s - 1980, 1999 - present, big empty hole 1980 - 1999)

⚫ Genetic engineering (mostly false starts: late 1970s / early 1980s, again early 1990s, again post dot-com bust -- but it's never really launched)

⚫ "Data Scientist" (2009 - present)

⚫ "Webmaster" (1997 - 2001)

⚫ "Mobile Apps Developer" (2007 - ~ nowish, starting to fade)

Just because you're getting your time in the sun now doesn't mean the glow won't fade.

Sure, the salaries may come down as more competent people enter the field (something that has been happening a lot more slowly than expected IMO), but calling data science a "fad" or saying it isn't a solid career trajectory is ridiculous. It's just a new name for something that has existed for decades. My first job was in a group that has been creating useful products using machine learning since the early 1990s. If that group was formed today it would be called the "data science" team. There are plenty of other examples, like "quants" at finance companies and actuaries at insurance companies. These sorts of jobs have existed for a long time and have paid well the whole time.

I'd also point out that most of the jobs on your list are specific roles in specific industries - data science as a concept will be more resilient because it's an evolving set of skills (as is programming/software engineering) that span basically the whole economy. The day people no longer have a reason to build systems using computers is the day these jobs will go away.

> The day people no longer have a reason to build systems using computers is the day these jobs will go away.

Sooo, not too long after some clever person figures out how to automate 80% of the grunt work of building an analytics pipeline and their colleagues dumb down the 20% left so that the people inhabiting the C-suite can work it on their iPads...?

I know at least three startups working that space, right now. And there are probably another 300 on the way.

This is like saying that all the programming jobs are going to go away because of some visual programming language just over the horizon. The fact is we are generating more and more data every day, and there are infinite ways to formulate and look at it. The tools that C-suite can use will never be powerful enough to do what a data scientist does because of the variety of data that exists. You can only dumb it down to the lowest common denominator, and the LCD will not be very useful unless the scope is extremely narrow.

>and there are infinite ways to formulate and look at it.

Of course there aren't. The best way is to know exactly the causal process generating the data, thus enabling you to predict your variable-of-interest perfectly from the limited data.

This isn't currently possible, of course, but there's one best way to interpret data, not an infinity.

Of course there are, because there are infinite forms of data. And to say that there is always one best way to interpret data, well, I don't even know where to begin with such a naive declaration.

Philosophically, there is only one best way to interpret data - if you know the underlying causes then that provides the only explanatory power you need. A similar argument could be made to our understanding of multiple physical phenoma prior to Newton deriving His equations - your statement is equivalent to saying that there are multiple valid interpretations of how the planets revolve around the sun and apples fall from trees.

The data we care about are not the direct result of physical laws, they are the result of chaotic systems both natural (eg. weather) and human (eg. stock markets).

For your tortured analogy to bear any fruit you have to assume that we're just waiting to find a dead simple set of equations that explains all human endeavor, and after that finite memory and computation power won't be an issue because, hey, cause and effect.

Sometimes it's an arms race. The minute you automate away some analytics step, the smart analysts start building on top of that, raising the bar for the competition.

Can you actually automate statistical analysis like that? I don't mean automating the actual math, I mean writing software to process raw data, decide what the meaningful statistical computations are, and then present the information output appropriately.

As far as I'm aware doing probability and statistics correctly is all about framing the problem correctly. You can, for example, accidentally assume that two events are independent when they're really dependent. Is it even possible to detect that kind of mis-framing automatically?

Yes .. it is possible. I may be getting the name wrong but I saw a demo from Ben Shneiderman's research group a few years ago where they did every possible statistical analysis on pairs of variables to look for high correlation. To their credit, they weren't selling the idea as automatic stats analysis. Their domain was health data and they were merely stating that their tool could give scientists an insight as to what factors to look at. Still ... it made me wonder where this could go.

We mock MBAs because they think there is a skill called "business administration" which can be applied to any industry. The truth is "data science" is something that you do not are. Domain knowledge trumps all.

And throughout that time actuaries (the original data scientists) have had one of the best-paying and highest job satisfaction levels. Statisticians and data analysts have been in demand for a long time and likely will continue to be.

Yes. Though keep in mind that actuaries are protected by big regulatory moats. An actuary is not someone who practices actuarial science, but someone who has a piece of paper saying that he can.

It really is hilarious that you actually have to have a certification to practice statistics. Think about how absurd that is. People write code that controls airplanes, medical devices, emi emissions...which are of course certified/regulated on a product basis. But to just apply stats...you need a certification. Something is wrong here.

I practice statistics all the time without a certificate. What are you talking about?

http://www.beanactuary.org/exams/

OK, so you just worded what you said above poorly. You implied that any practice of stats required certification. My comment covered both statisticians, and actuaries, but just used actuaries as an example.

Why does it matter that actuaries are certified? It's a professional role which requires dealing with some pretty heavy concepts. Employers presumably want to be sure that candidates meet some level of capabilities.

I guess I should add that the web world and modern data scientists kind of changed this but it was mostly because there were a bunch of smart PhDs who left academia and other companies to go work at rapidly growing startups, and their abilities helped grow the companies. Growing startups don't really care if their "Actuaries" don't have credentials other than beign able to monetize site data.

Nooo. No.

An actuary is someone who does actuarial work. Most people join the profession right out of college. They have no piece of paper other than their degree. They are still actuaries.

A credentialed actuary is someone who has gone through the entire educational system, which are a series of exams that are typically taken while working as an actuary. Credentials and the practice rights that come with them permit an actuary to do exactly one new thing. Only a credentialed actuary can opine on the adequacy of an insurance company's loss reserves (to make sure that the company has set aside enough money for all of the claims it will eventually have to pay).

Not many actuaries issue formal statements of actuarial opinion. All of the other work that we do can be legally done by anyone. There is nothing stopping an insurance company from hiring a large number of statisticians or whatever to do the day-to-day work, only bringing in a credentialed actuary to do the reserve opinion once a year. Companies generally don't do this because actuaries are more than just statisticians; they're business professionals with very deep domain knowledge. Credentialed actuaries command a premium in the marketplace not because of any "regulatory moats", but because the specialized education itself is very valuable. If that ever stops being the case then nothing is stopping the market from correcting itself.

Yes, the truth is more nuanced than my flippant comment portrayed it. I considered becoming an actuary once, but decided against.

True. I've worked with and for several over the years. It's rather an underappreciated career path.

That said, the business model for actuarial work typically hasn't been mining every last shred of individual and personal information (caveats, below) for retail advertising. Instead it's been measuring, tracking, and modeling risks within insurance, and many of the business lines don't concern individuals. You've got business, shipping, and industrial insurance, though yes, also life, healthcare, auto, home, and fire.

In particular, insurance is a fairly regulated market with exemptions from anti-trust collusion for the specific purpose of facilitating underwriting boards -- these are multi-company associations which pool risk data. It tends to make for a slightly less cut-throat competitive environment than IT. Also that personal data isn't bandied about quite so casually.

Though as I said, that's changing. I've been informed by industry insiders that a common practice now in automobile insurance is to purchase "smog reports" from states. These include a tremendous amount of data (most of it specific to vehicle performance), of which the salient element is the miles driven since the previous check. Turns out that your auto mileage is a significant risk predictor.

Though the fact that data gathered for one stated purpose (clean air) is being used for another (rate adjustment and fraud detection) is troubling.

Anyone with the skills to be a webmaster in 1997 likely still has a promising career trajectory if they want it. However, I don't know of any webmasters from that period who pegged their career to the term "webmaster."

"Webmaster" in 1997 covered a wide range of sins, and part of the change has been that what used to be one job is now a raft of them -- design, front-end coding, markup, back-end, ops, database, and more. Smaller shops may combine several of these, but it's not unusual to see each as its own specialty.

That's part of what I was getting at.

But the other was that "Webmaster" was a really hot ticket for a few years. It's faded markedly.

I can't see my parent comment at the moment, but I saw a recent "ask HN" post where the question concerned where all the Ruby-on-Rails jobs had gone. What had been a really hot ticked for the past few years is starting to fade. A lot of kids are starting to learn their first talent-shift lessons.

This is a fantastic post. My career has only spanned 15-20 years so far, but what I've learned is that every "hot" career ebbs and flows. Law used to the be the place to go, now nobody wants to go, but it'll get to the point again where it's a "hot" career.

Lets hope not.

Yeah, I currently have some data sciency skills. Still in core software engineering. I think I will stick to what I am currently doing.

Reply on news.ycombinator.com