Hacker News Reader

Comment by cageface | Hacker News Reader

ML definitely seems to be more on people's radar these days but as far as I can tell the job market is pretty small.

My experience has been the opposite:

http://metaoptimize.com/qa/questions/2542/job-prospects-in-m...

I guess it depends what you're comparing against. Against general purpose programming jobs, yes, the market is small. But, of all CS PhDs, people who do ML are the most sought-after.

That was actually my question. :)

I'm sure CS PhD's are in good shape but breaking into ML from another programming discipline seems pretty hard.

This is true - my boss has been pressuring me to do the legwork required to get at least a masters in stats or CS so I can carry a higher title in our institutional research department.

Just for those outside of academia - state schools really drink their own Kool-aid. Associate/assistant directors require a masters degree in anything while directors require PhD credentials.

> I'm sure CS PhD's are in good shape but breaking into ML from another programming discipline seems pretty hard.

That's because ML isn't a "programming discipline". It's pretty much pure statistics, optimization algorithms, and linear algebra these days, and those algorithms are HARD to code, HARD to scale.

Those algorithms are actually surprisingly simple to code most of the time, if you're using a high-level language and aren't worried about scaling. Scaling them is hard, but for that you can already find good, scalable implementations out there of the basic building blocks.

In my experience coding machine learning algorithms is actually easier than the hairier sorts of programming (multithreaded, distributed, very low-level, etc) if you do the math first. Most errors are come from doing the math worngly (or not doing it at all) and most slowness are due to missing very obvious optimizations (which a good programmer will pick up on sometimes even if it's not explicit in the papers that describe the technique; some papers unfortunately assume you will pick up on the obvious optimizations yourself).

That's been my experience so far as well. The algorithms are surprisingly straightforward. Understanding the underlying theory well enough to get good results is the hard part.

I think most of this is due to the fact that most ML techniques need lots of specific knowledge to apply correctly. Many introductions focus too much on specific algorithms, for example, while ensemble methods are probably best in real-world data. The pactices of building training and test sets, of regularizing, of doing proper feature engineering, and of making structured models (for example) are more easily learned in a mentor-student relationship than by reading blogs (reading papers might take you a long way, but you won't build an intuition as to why things work and why they fail without lots of experimentation or advice from someone who already has this intuition) and/or AI books, and ML books are usually far too technical (or too superficial in the technical side, as is programming collective intelligence).

Most of the ML books I've worked with so far seem a bit overly formal. Steven Marsland's book seems to strike the best balance between theory and implementation I've seen, even if the Python code is a little clumsy.

You think so? Well okay, it's not a topic for everybody, but with increasing data loads, it's about time people think more about machine learning technologies to use to get to know more about their data. I do both Operations Research and Machine Learning and see many possibilities to improve a company's knowledge about the data they have - even if they don't know it (yet).

Especially as with unsupervised ML you can definitely find patterns in your data, groups of similar data and trends, you can do forecasts, imply relationships, and much more.

It's a hard topic, and you definitely need a lot of time to get into the really bloody details, but it is definitely worth it. And it makes a lot of fun if you like maths and statistics.

I agree it's a fascinating field. I've been doing nothing for the past few months but intensive study of the fundamentals. I've had to review a lot of math but I've really been enjoying it. It also seems reasonable to me to expect that demand for those skills will grow.

My impression of the current job market, however, is that it's tough for somebody without formal academic qualifications to crack.

There I agree, without the academic qualifications, people will hardly listen to you talking about formal Hilbert spaces, Jaccard index, etc.

Depends on how you define the market. I think ML is still too much a pie in the sky research topic for a job, but it's in huge "demand" as a startup founder

i.e. If you can actually do ML, you're much more valuable as a startup founder than looking for a 9-5 job, unless it happens to be at say, Google.

My guess is it'll be another decade before ML and statistics become seriously sought after in "normal" corporate programming.

We are in the process of writing up and hiring three positions from Associate director to database admin here at KU for Institutional Research (the focus is on machine learning and prediction). I'd say it's a fair indicator in a 'business' that's faced 2%-8% annual budget cuts in the past three years that people even here are paying attention.

From my IR background - it's about bloody time.

Reply on news.ycombinator.com