Hacker News Reader

You do something similar in Udacity's SDC nanodegree but you use their simulator rather than a new car. Interestingly through trial and error on my project, ELU activation functions was the only thing that prevent vanishing gradient problems. You use the same activation function and I'm curious why you selected that one. I always wonder why it was the only function that worked for me.

View on HN

problem is most likely that your initialization are bad (see e.g. https://openreview.net/pdf?id=H1gsz30cKX for explanation). make sure to use variance scaling, taking activation into account (relu cuts variance by half). probably you need to multiply all initial kernel weights by 2. make sure that initial prediction of model before training is same order of magnitude as typical target, not saturated to zero or other extreme value. batchnorm and skip connections can also ease problems of bad initialization, so worth trying.

Because you can still find the best gradient with ELU. Say you use sigmoid and you have big weights then your gradient can get stuck

Thanks for the answer! I also used Relu with no luck either. I figured Relu would work better than ELU but it didn't work for this application. Would you happen to know why ELU is superior to ReLU?

RELU is really good for mostly positive weights. ELU is good for both

Reply on news.ycombinator.com