Hacker News Reader

Comment by gas9S9zw3P9c | Hacker News Reader

Definitely. I came from a software engineering background into AI 6+ years ago, and I was lucky to be there at the right time. From purely a career perspective, leaving aside that this may be your life's passion, I believe that right now is the worst time to get started in AI. These is so much noise and competition but very few actual jobs or value created. It's just a research PR machine. Just as with Data Science, companies that think they need AI often just need better data collection, pipeline and infrastructure engineering. Good backend/infrastructure/data engineers are so much harder to find these days, and these skills IMO provide much more value than doing some kind of modeling.

View on HN

> Good backend/infrastructure/data engineers are so much harder to find these days, and these skills IMO provide much more value than doing some kind of modeling.

After all these years of FOMO on AI, this is music to my ears.

Right, I’ve been reading about machine learning for about 5 years, have read hundreds of articles about different techniques, and have often tried to explore ways it could be used.

However, I’ve never found a practical use in software engineering.

Every time I think I discover something that could use machine learning, I usually don’t have any data to work with or don’t have a clear definition of what the inputs and outputs would be.

In the end, I find a way to develop a solution that doesn’t need AI and often makes me realize that AI would not have been able to provide the required reliability.

I even participated in a self-driving RC car competition (running on raspi 3). I ended up scrapping the ML solution because it could only run at 10 hertz which was not even close to what was needed.

I ended up writing a custom algorithm in C that ran at 200fps and could do “advanced” moves like backing up and course correcting.

I’m not saying that ML doesn’t have its place. I’m saying that right now it’s all about images and nlp and has huge costs.

In any practical usage, the ML parts that I would need are already available as a service that I can use almost free (like azure cognitive services).

> In the end, I find a way to develop a solution that doesn’t need AI and often makes me realize that AI would not have been able to provide the required reliability.

This was brought home to me back in the 1990s at a place where we were trying to popularise expert system technology (a positively prehistoric form of AI). It was one of my first jobs in industry after leaving academia. The goal was to create a system that could predict the speed at which a unit of military vehicles could move so that a contact report expert system could decide whether two groups of vehicles could have feasibly moved between two points in a given time. The system could use this to help decide whether two reports referred to two different enemy units or a single unit that had moved from point A to point B.

After quite a lot of time cranking out and debugging rules to describe the movement behaviours, I realised that some simple convoy arithmetic would do just as good a job - units are often constrained by the speed of the slowest vehicle, and respect inter-vehicle distances. For most purposes this simple arithmetic was just as good (and orders of magnitude faster) than the complex rule engine.

I think there is a lot of more practical deep learning that are coming out, but it often times working with the people who know the field intimately.

For example, in my PhD thesis project, I am making a series of deep learning models to make a cardiac MRI autopilot. I've build a series of deep learning networks to localize the cardiac landmarks that define the cardiac imaging planes. And our group has even made it into a clinical prototype that works within our imaging workflow.

I think the field is shifting in that ML technologies are increasingly requiring domain level knowledge in order to make a practical endpoint.

I've been involved in a lot of Data Science projects, and the most successful ones are the ones where the Data Scientist either did or was part of the infrastructure and Data Engineering.

Most of the failed projects basically were "Hire a Data Scientist to make a magic model" and nothing else was supported. Basically, "Here's some data, do some magic." I read somewhere that 90% of data is low/zero value, and I'd agree with that.

There's absolutely a place for people doing pure statistical or ML/Deep Learning modeling. But the rest of the organization has to support them so they can have that narrow focus. A lot of places want to take a shortcut and not do the data work.

5 years ago the FOMO was from Hadoop. Hadoop Here, Hadoop there...people just forgot you have to actually collect a lot of high quality data to make Hadoop useful. It's AI now.

Reply on news.ycombinator.com