Hacker News Reader

Comment by iraphael | Hacker News Reader

It should be noted that transfer learning is an umbrella term for many ideas that revolve around transferring what one model has learnt into another model. The method described here is a type of transfer learning called fine tuning.

View on HN

Correct. There are multiple different ways to transfer knowledge in between tasks. We are talking here about transfer learning with deep neural networks where it is proven to work with several advantages over training a model end to end on your own. There are multiple decisions you need to make even with transfer learning like which layer to use for transfer, how much fine-tuning should be done, based on how much data you have which we are trying to automate.

Yes transfer learning is a fairly umbrella term encompassing a lot of different approaches. We tried to give an example of the one most commonly used in NNs almost exclusively with regards to feature extraction. Do you have some resource that lists a variety of transfer learning approaches? Happy to work with you in creating a aggregated list.

Well for starters, fine-tuning can be done in a variety of different ways. You can pretrain your model with a larger, different dataset, or you can train an autoencoder that learns some useful representation of that larger dataset and use the encoder as a base for fine tuning.

Another approach I've seen that was really cool is Model Distillation [0], which is basically the training of a smaller NN with the inputs and outputs of a larger NN (where the output is slightly modified to increase gradients and make training faster).

[0] https://arxiv.org/pdf/1503.02531v1.pdf

Reply on news.ycombinator.com