Where does this trend go? Will there an over-supply of data scientists soon?
It’s getting worrisome.
There is an undersupply of technically skilled people who have the knowledge and experience to sort this mess out and bring it into a production system (I'm talking about data engineers and machine learning engineers).
I think this will correct itself in the sense that those who only did some online courses will not be able to find a data science job so they will either retrain themselves to become a data engineer or they will pick a different field (maybe the field they came from initially. I suppose having domain knowledge AND some data science skills could be very useful).
Both data science and trading have low entry barriers - you can easily find what appears to be opportunities or a working strategy / model. It's much harder to find something that will stand the test of time. In both disciplines you can easily get the feeling you are doing something right, while in reality you are going in a very wrong direction - finding spurious correlations, overfitting or worse.
I think most of those data scientists don't really know their craft.
[1]: https://www.knowledgescientist.org/
[2]: https://dmccreary.medium.com/from-data-science-to-knowledge-...
There already is an oversupply at least for beginner-level candidates. The trend will be that more people who switch to data science will have to eventually give up because they can't find a job. And that companies will increase their requirements for academic credentials even further (a PhD is already a minimum requirement at some places).
The field itself will be fine, there are plenty of interesting and difficult problems to work on.
It’s easy to hire someone to do “machine learning” off in the corner, it’s harder to adopt data literacy through your org and get people to thoughtfully use data to make decisions. Including knowing the value and limits of the data you’re looking at. And this last item is truly needed to really leverage the data whether it’s analysis or more complex machine learning.
Yes, the overwhelming majority thinks that. I can't blame them really. Most of the content out there about this is by data virgins who've never touched actual data, mostly Tweeting/blogging/Mediuming/YouTubing/audience building. This is due to the fact that relatively few are actually doing this in real life. ML projects can be quite expensive and you're either in a large organization's ML team or, like us, a company that large organizations hire either for lack of an internal team, or to support their team. That is, if you want to work on actual problems, with real data, and real stakes, real infrastructure questions, etc.
Recently, though, there has been an uptick in the amount of MLOps content. That, too, has been innundated by people writing for "that site on Medium".
That is the way it is for now and the field is immature. There will be a lot of churn. Most people will give up. Some will keep up and learn and actually get to do that and evolve. So, one may be dismissive and it would be understandable. But we could also see it from a stand-point of a conversion funnel, and all that is driving people through that funnel. The content may irk me, but if it's driving someone to the field and that person sticks and learns... Sysadmin, webmaster, webdev, IT... all went through something similar.
We have been building our own platform[0] to leverage what we have learned and at least remove some of the toil from our plate and ease off a bit of the hiring pressure resulting of that shortage. Our PhD people needed to tap on someone's shoulder to set up or fix their environment, or deploy their model. We're removing taps on shoulders, reducing the cardinality of the profiles we need, and the skills required to operate. These issues can kill a small company, as you either had to hire people who could do it all, or people who were specialized. And you're left with people jumping around putting out fires on several projects at a time. It can drive many people crazy.
- [0]: https://iko.ai