Comment by JHonaker | Hacker News Reader

Comment by JHonaker | Hacker News Reader

Parent Full thread

I mean, don’t just take my word for it. Here’s Yann LeCun [1].

I will concede that I forgot about DeepFakes. How’d I do that? But still, cool now we can live in a dystopian future.

The point that I was trying to make still stands. Assuming that more data and more compute will solve all of our problems is a ridiculous position. The field might be moving incredibly fast, but most of the work I see is not great. Everyone is hyped. People claiming that we can replace all of statistics and machine learning with pure deep learning models forget about how hard it is to actually get these things working in the first place. Do you have 100 TPUs to run full blast for a week? I mean consider the series of Nature papers a few years ago. Google says, “I can predict earthquakes using NNs!” Somebody comes along and says, “I thought about the problem for a little bit and beat you with logistic regression.”

Self-driving cars have been just around the corner for the last 10 years. Just like with the rest of stats and ML, it’s not the place you have a lot of data that’ll kill you. It’s the edge cases.

[1] https://m.facebook.com/yann.lecun/posts/10157253205637143?no...

Be careful, the narrative that you need tons of hardware to do anything is pushed by companies selling licenses to cloud APIs and cloud training platforms. In reality people routinely publish new results at AI conferences while only having access to a few GPUs. Not everything is about hardware. There are thousands and thousands of AI papers. Not all of them are about scaling up some model like GPT-3.

Yes, there is a lot of hype, a lot of misuse of ML by people who don't understand it. Yes, in some cases it makes no sense to use a deep neural net if logistic regression (which is a single-layer NN) will work. This is generally the case when your features are already very processed, curated and meaningful as in your earthquake example. But you will never solve e.g. image or speech understanding with logistic regression.

Perception on high-dimensional, complex, messy real-world signals is where deep nets excel. And they have unparalleled performance there, they blow everything out of the water. It's hard to overemphasize this. The last decade has turned many applications from does-not-work into works. A zero-one transition. Not 10% better, but it was hopeless and now it is "quite nice". Still not divine perfection, but it's far from just being hype.

It can be hard for outsiders to appreciate this, but before 2010 or so, everything in computer vision was very fragile for real-world applicability. The algorithms only worked in one specific setting, like one particular assembly line, with all parameters hand tuned in the pipeline. You needed a lot of experience and intuition to configure your feature extraction pipeline and the regression method and it was still very fragile. Much of research was only demonstrated on toy examples, synthetic data, or only on a few images. The shift to huge datasets is also something from the last 10-15 years. Before that people cherry-picked example, where their hand-crafted algorithm with hand-tuned parameters finally produced something resembling an okay result. I'm only exaggerating a little. These things really didn't work. Many of the concepts were already in place, but it still didn't work.

Could you list some conferences and papers one could use to catch up on ML and AI from zero?

Papers are meant for when you're already familiar with the basics. It also depends on your knowledge of prerequisites and goals.

But it's best to take courses and read textbooks. There are good free courses from reputable universities out there. I'd recommend starting with some math and CompSci foundations. The internet is full of recommendations because everyone and their mom is asking the same question. How to learn AI from zero. Most of those people then don't stick with it of course because it takes time and effort.

The other way is to just read "easy" tutorials and "instant" courses without any foundation, learn just the buzzwords and then be totally confused about what is actually going on.

Reply on news.ycombinator.com