Practical Deep Learning for Coders 2019 | Hacker News Reader

Practical Deep Learning for Coders 2019

fast.ai

556 pointsjph0075 comments

FYI, Google just completed adding all the pre-requisites for this course to their Colab service, which provides a GPU-powered Jupyter Notebook for free. There are no usage restrictions, other than that each session must be no longer than 12 hours (after 12 hours, you can just create a new session). So this means you can complete the whole course, and do your own projects, even if you don't have a credit card.

Here's the setup guide: https://course.fast.ai/start_colab.html

That's great! I almost burn up my laptop running the NLP classes and wanted something more scalable that paperspace(is still a great way to run fastai).

Thanks Jeremy!

I've gone through each iteration of the fast.ai courses and must say that it is worth it every time. The latest 2019 courses are again excellent. Even though I've covered the concepts before, I always learn a ton from every lesson.

Although it is recommended to dedicate ~10 hours per lesson, it is very possible to go through each video, then circle back to focus on one of the concepts/problems presented.

I recommend giving the first lesson a try. It is very rewarding feeling like you can actually do or build something at the end of each lesson.

If you haven't done so already, would you recommend going through the old lessons? Or is the current version good enough?

The courses move with the fastai and Pytorch libraries so its best to do the latest course so you don’t run into features that have been deprecated or renamed or miss out on new features.

I don't have the time to immediately work on this course, but I might later this year.

What I'd like to know, as someone who has completed (in the past) the following MOOCs:

1. ML Class (plus about half of AI Class) - 2011 2. Udacity's CS373 Course - 2012 3. Udacity's Self-Driving Car Engineer Nanodegree - 2016/17

...and gained a lot of knowledge about ML and DL thru them - would this fast.ai course be worthwhile to pursue as an addition to my prior education on the subjects? Would I find it just as challenging (from a cursory glance it looks to be a "yes", but I wanted another opinion)?

I'm open to suggestions from everyone here, not just jph00/fast.ai ...

Also - and this I would also like opinions on - would my time be better spent pursuing other areas I know I am not proficient in - mainly learning about probability/stats, as well as basics in calculus...

For instance, in the prior courses, there was always mention of - for say backprop - stuff about the "chain rule" and other "rules" for calculus, and I feel I need to understand that more. That is, I am wanting to understand things in ML and DL at a lower level; how the "black boxes" really work. So would I be better off pursuing that, or just continuing with courses like this one?

Or maybe both at the same time (if that is even possible)?

I've also considered that I might need to take those subjects (probability/stats/calc) as actual courses via community college or similar (or online MOOCs).

Any thoughts or suggestions are welcome!

Whilst it's absolutely not required knowledge for most deep learning, if you are interested in more fully understanding the chain rule, my friend Terence Parr (the ANTLR guy) and I wrote a fairly thorough tutorial for matrix calculus https://explained.ai/matrix-calculus/index.html .

Ive watched the first two episodes of fast ai and one thing i can say is that the whole approach of this course is the opposite of what the courses you listed are structured like. Fast Ai jumps right into "how to use the script/tools" while slowly teaching the concepts behind why they work in comparison to the typical ML/AI course that involves complex learning calculus and statistics before doing anything practical.

SDC alumni here as well with a couple of ML courses under my belt as well. Did any of these courses help you advance or pivot your career? I'm still struggling and looking for advice how to move forward into AI/ML (I have a Master in Mech as well). Any advice will be highly appreciated!

This is the best course. Unless you really some basic. Highly recommend.

This. I can tell anybody thinking about taking this course, do it! The way Jeremy Howard explains things is so down to earth and clear, you'll be surprised at how easy all this ML/DL nonsense is :)

But seriously, there's a lot to learn here, and if you are looking to make your entry into the hype field of the decade I would strongly suggest starting here first

(Alternatively, for the more math inclined, perhaps take Andrew Ng Coursera course first: https://www.coursera.org/learn/machine-learning)

I second the Andrew Ng Coursera course; I took it in 2011 when it was called "ML Class" (pre-Coursera) - yep, I was one of the beta testers!

Just wanted to say, your first course made me, someone with out a degree, extremely interested in the field, and empowered me to pursue a career in the industry. You making ML "uncool" really made me believe I could do it, when previously I had been quite self-conscious about my lack of academics.

I did realize however I was still going to be quite a hard sell for companies without the credentials. But that has lead me towards obtaining a B.S. in Data Management/Analytics from WGU.

Thank you for your incredible content and I look forward to completing this course. Your enthusiasm and teaching style make the class a real joy.

Could someone recommend in which order should the fast.ai courses be taken for someone new to the AI field completely? Should "Introduction to Machine Learning for Coders" be the first? or I could dive into "Practical Deep Learning" right now, followed up later perhaps by "Cutting Edge Deep Learning for Coders" ? What is the difference between the two deep learning courses also?

If you want to get to building ML models right away, take Practical Deep Learning for Coders. Cutting Edge Deep Learning is a part 2 to that class, with more advanced topics.

The Intro to Machine Learning, from my understanding, is more of a normal class that will take a deeper dive into some of the concepts and theories that aren't necessary for practical implementation, but will be helpful for understanding the whole process.

I personally recommend starting with this newest course.

The deep learning courses are specifically focused on deep learning whereas machine learning seems to have a broader scope. I think it just depends on what you are more interested in.

Can we have one in Julia please?

https://julialang.org/blog/2019/01/fluxdiffeq

+1 for the idea

I use Python and TensorFlow at work but for at home side projects I am very much enjoying Julia and Flux. I haven’t tried FluxDiffEq yet.

Jeremy, please :-)

I'm so excited to dive into this.

One question: is there a reason you opted for PyTorch over Keras? I had the impression that Keras was the go-to for "easy as ABC" neural networking.

EDIT: bonus question! I'm really fascinated with gameplay agents like AlphaGo and more recently AlphaStar. If I finish this course, will I have what I need to start work on a toy version of some gameplaying agent? If not, could you recommend where I could go next to start exploring that area?

I discussed this a bit here: https://www.fast.ai/2017/09/08/introducing-pytorch-for-fasta...

Also some info in the fastai release post: https://www.fast.ai/2018/10/02/fastai-ai/

Once you've done fast.ai, you can look at OpenAI's crash course in reinforcement learning: https://blog.openai.com/spinning-up-in-deep-rl/

If you're interested in a Go version of the AlphaGo algorithm, I wrote one: http://github.com/gorgonia/agogo :)

You can start with this book if you want to go straight to making a gameplay agent https://www.manning.com/books/deep-learning-and-the-game-of-...

The one advantage of pytorch is that you can make your graphs dynamic.

Tensorflow also created Eager - their dynamic environment

In my view, this is the best, most accessible course on the planet for learning the how to of deep learning as quickly and efficiently as possible.

Highly recommended!

Well, very similar to one I just released (even in name)[1]. Specifically, I targeted taking one problem through several network iterations. I think it can be overwhelming to go through all those different networks. Once you get the basics new networks don’t aren’t really too difficult. I haven’t taken their class before (and wouldn’t want to spend money). I think that’s a huge barrier, there’s not much of a reason to require setup on sagemaker.

[1] https://austingwalters.com/neural-networks-to-production-fro...

It's free and there are no ads, and you can run it for free on colab or GCP.

I have no idea about where your sagemaker comment comes from.

Thanks for posting this, I have started working up some cloudformation scripts with the goal of getting spot instances into play for GPU nodes on AWS @ https://github.com/frankamp/fast-ai-aws-advanced not quite there yet, but using layering to get separation of concerns means a user can swap out for a bigger gpu temporarily without losing anything. Cheers.

Does your definition of "coders" include those of us with at most middle school mathematics knowledge? I've tried some machine learning courses before, but couldn't make sense of the vast majority of information.

Yes it'll be a bit more work for you, but it should be OK. I provide links to more math background where required in the course.

Can the exercises in the fast.ai deep learning course be done in another language other than python? And how much time in weeks should it take to complete the course? If my laptop doesn't have a GPU, can I still do all exercises?

The whole course builds upon the fastai library written in python so there is no point in not using python. But I feel like you can gain a pretty good understanding of deep learning in general by watching the videos.

The live course was held over a time of 7 weeks. It should take around 70h to finish it in total but I really depends on how deep you want to dive in and how easy it is for you to understand.

> If my laptop doesn't have a GPU, can I still do all exercises?

To answer this, while technically you could do the exercises, it is possible that it will take longer (perhaps much longer) to train models and whatnot without a GPU. Where you'll really have a hard time will be anything for deep learning and large neural networks. What might take a few seconds or minutes to train on a GPU might take hours with just a CPU with a few cores.

But not to worry - in the course it explains that you can use a variety of "cloud based" GPU systems that are fairly cheap to access (provided you have a credit card). So if you can't get access to a GPU, there are other ways that won't put you in the poorhouse (just make absolutely sure to completely shutdown and/or delete your VM instances you spin up).

Also - while you could use another language, it is best to stick with Python for now, because it has become almost the defacto language for ML and Deep Learning, at least as far as for education and such. Mainly because it has such great libraries for scientific computing available, plus it is easily approachable. It also has a great interface to Tensorflow (and ultimately CUDA and NVidia GPUs - also, get used to the fact that if you do anything DL related, you will be using NVidia almost exclusively; while AMD and others have their own DL hardware, it isn't nearly as well supported - maybe it will be better supported in the future).

That said - Tensorflow does have decent support in a number of languages, especially C++ - so if you are more familiar with that language, you'll be set to do some amazing things once you complete the course.

But Python, again, is really where it's at. Tensorflow is a great library that abstracts away a lot of the pain in dealing with CUDA. But even it has pain - and there are other libraries layered over it that make things even easier to work with (ie - Keras). Seriously; in the training I've had with DL (mainly thru Udacity), we had to learn how to build a simplified DL "library" of methods for building simple neural networks.

Once you learn about Tensorflow, you will have an epiphany of "OMG so much easier!". Then - when you learn about Keras, it's yet another epiphany - it makes Tensorflow look "difficult". Of course, it's best to learn in this manner if you can, so that you understand how the lower levels work - instead of everything being a black-box.

Of course, at some level there will be "black-boxes"; for instance, you may or may not want to learn how Tensorflow interfaces with CUDA. Or how CUDA interfaces with the GPU. Or certain other methods in Tensorflow or some other library that perhaps wraps certain methods and functionality. Honestly, things are much nicer today than they were even 5 years ago. You can decide just what level of depth you want to explore down to.

Why not simply run on a cloud solution (like Google Colab), which can connect to GPU (and even TPU!) instances and is incredibly easy to work with out of the box?

Jeremy mentioned elsewhere in the thread that you can do this :)

I took the in-person version of this course in Fall 2018, and it's definitely approachable for someone who doesn't yet know any linear algebra or remembers calculus all that well. You just start doing things with deep learning right away. After these quick wins, you'll hopefully get some more intuitive breakthroughs of what's actually happening. Then and only then do they suggest digging into the math more deeply (I'm still doing this last part).

As someone just wrapping up the FastAI Intro to Machine Learning course and moving on to DL, I'm really glad this came out now.

Thanks for all the knowledge, Jeremy!

Does anyone else find it potentially problematic we have all these "get coding fast - for coders!" type resources available, when most programmers don't have the mathematical foundations to truly understand ML/AI/DL?

It's funny, programmers scoff at non-technical folk using drag-and-drop tools to replicate certain programming functionality but I see the same thing happening here.

The course is amazing. I was frustrated in late 2018 when they promoted their code to "v1" but kept iterating with breaking changes on a literal daily basis. I have stopped using their library because of this+insufficient time. However, their work is so amazing that I will pick it up again when I have more time. Yes, it's so good that I will deal with daily breaking changes to use it.

The changes were just during the development phase. Should be quite settled now.

Wonderful. I should add that you and Sebastian were always extremely responsive and helpful troubleshooting issues with that popped up due to the breaking changes.

Any recommendations for getting up to date with Pytorch? I could pick some cheap books from packt or maybe good blogs ?

It's worth noting Jeremy and team are making these invaluable resources available for free, even though a lot of folks would be extremely happy to pay for them.

It's amazing to me that learning materials of such exceptional quality and deep practical application are available without a paywall, especially when everyone else seems to be cashing in on the AI hype with a myriad of paid nanodegrees and such.

Thank you, Jeremy and team, for everything you're doing!

I finished the 2018 edition of the class two weeks ago but I am happy to go through the latest version now. The class was so thought provoking. I came up with a lot of good ideas for my thesis project while watching the videos.

Hey Jeremy, just FYI this release seems to have broken the links to the Introduction to Machine Learning for Coders course. For example, I can no longer access it from the fast.ai front page (I get a 404 error.)

Many thanks for letting me know - I've fixed that link now. FYI the ML course is now at: http://course18.fast.ai/ml

Thank you for the link. It's currently broken here : https://www.fast.ai/2017/09/08/introducing-pytorch-for-fasta... at this line:

"As we developed our second course, Cutting-Edge Deep Learning for Coders, we started to hit the limits of the libraries "

Hey, this course looks really interesting! Although I'm hesitating to take it because I'm not sure what the difference is between Machine Learning and Deep Learning. Can anyone explain?

Machine Learning is a catch-all term for optimizing statistical models with data.

Simplest example that you are very likely already familiar with is that of a 'best fit line' to some xy scatter plot. This starts by making an assumption (model choice) that the relationship between `x` and `y` is linear, e.g. `y=mx + b`, then you can use data (xy points) to figure out the most likely values for `m` and `b`. You can then make predictions for new `x_new` values by plugging them into your known line to get `y_new`.

Machine learning often manifests in a two step process: first feature extraction, and then fitting features to a desired output. Deep learning combines these as an end-to-end process to eliminate 'human in the loop' problems that occur from feature extraction.

Example: you want to predict who should win a chess game in a given board state

Feature extraction (what information you think matters): what pieces does white have, what pieces does black have, is white in check, is black in check, how many valid squares can white king move to, how many valid squares can black king move, etc...

* Fitting: make an assumption about the relationship between features and outcome (model choice), fit model using data (features, outcome)

The Deepblue 2 model that played Kasparov used around 8000 features (not sure if this is the feature vector size or # of features). As you can imagine, feature extraction is highly dependent on expert knowledge of the problem and will often fail to cover unknown situations/cases.

Deep learning models aim is to avoid limitations of expert knowledge by using raw data (e.g. occupancy of each square on a chess board) and extract features implicitly rather than relying on explicit human formulas. It has also opened up new possibilities for areas where expert knowledge has made little progress in the past (e.g. there is not much an expert can say about what pixel features are might indicate a dog/cat is contained in an image).

Though this end-to-end nature also makes it harder to spot biases and errors. There is some work that first trains with deep learning and then tries to infer a simpler model that is understandable for humans.

I took half of ML before. The pace is a bit slower and topics are slightly different. ML goes into using random forests and focuses more on working on structured data. I think if you have programming experience and are more interested in Deep Learning than the Deep Learning course is great.

DL is a branch of ML.

This is amazing! :) I took portions of it before but motivates me to go back and work on it again. Is it just me or the overview notes from Lesson 6 & 7 are the same?

Also wanted to add that as someone who hasn't professionally coded or have a degree in CS, the learning curve was a bit steep (which is why I took a break from taking the course).

Are there any resources or good ways to get up to speed on getting sufficient coding experience/knowledge so that I can really digest the content?

There's a lot of Python learning resources here: https://forums.fast.ai/t/recommended-python-learning-resourc...

My main recommendation is to try to do as many things as you can with code, instead of manually or with domain-specific tools. Even if it's slower at first, solving your own problems with code is the best way to become a productive coder.

So we should watch these newer ones instead of the older ones?

For sure. It's a giant step over last year's course.

Has the classic ML course been deprecated? I noticed that following link no longer works: https://course.fast.ai/lessonsml1/lesson1.html

EDIT: looks like they've been moved to the following link instead: http://course18.fast.ai/ml.html

This is awesome, many thanks.

Jeremy, I know you're a fellow fan of J -- is it possible to do modern ML with J? It seems like numpy is just a worse version of the array paradigm from J/APL, but I'm guessing all the specialized ML packages are the real value of python here? Any idea of anyone has done similar work in J?

What are the main differences with last year's course ? Also can we expect http://course18.fast.ai/part2.html to be updated in the near future ?

Is there a way to create lifestyle config for Sagemaker manually instead of using fast.ai created IAM ARN and config?

The lessons are fairly brief compared to Coursera\Udacity

About 14 hours of lessons per part for fast.ai. A lot of Udacity courses I've seen are shorter than that. I certainly wouldn't call it "brief"!

The thing that separates fastai course from other courses is the use of fastai library (built on top of pytorch). fastai library makes it dead simple to train neural network faster and with more accuracy.

Write a neural network is easy. But tweaking hyper parameters takes lot of time and knowledge. fastai library implements cyclical learning rates which varies the learning rate during training along with some other defaults for other hyperparameters.

https://github.com/fastai/fastai

Great course, looking forward to checking it out. I skimmed thru the overview and dismayed to see nothing on object detection still :/

Object detection is in part 2. You can use the 2018 part 2 for now. We had hoped to move it to part 1 this year, but couldn't figure out a way to make it easy enough yet.

fast.ai courses are EXCELLENT. I've read/watched many interviews with both creators and have been even more impressed by their drive to bring deep learning to everyone, not just the AI specialists.

Congrats, Jeremy and crew!

Many thanks. I'm really happy with how it turned out. I hope you all like it! :)

I'm really interested in learning the math basics for deep learning. Is there an online guide that you can point us to if we want to learn the basics of calculus, linear algebra, probability theory? In other words is there a "fast.ai" version of "math for deep learning" out there?

Yes for linear algebra, though it's probably best suited to people with at least some exposure to matrix/vector math (for example you've used SVD but have no idea how or why it works): https://www.fast.ai/2017/07/17/num-lin-alg/

For probability, machine learning is more about statistics (the two are related, but courses explicitly about probability will cover different things), so I would lean towards that. An Introduction to Statistical Learning in R (ISLR) is a frequently recommended book. You can ignore the R and do the exercises in Python.

If you actually want to learn about probability, you can look at MIT's course: https://ocw.mit.edu/resources/res-6-012-introduction-to-prob...

EDIT: If you've never been exposed to calculus, many people swear by Khan Academy's videos.

This is great. Thanks you!

I would also recommend the MIT calculus courses on edX. It has been a great refresher for me since the last I looked at calc was 8 years ago.

I'll check it out, thanks!

Reply on news.ycombinator.com