Hacker News Reader

Comment by dirtyaura | Hacker News Reader

Really great work on visualizing neurons!

Is anyone working with LSTMs in a production setting? Any tips on what are the biggest challenges?

Jeremy Howard said in fast.ai course that in the applied setting, simpler GRUs work much better and has replaced LSTMs. Comments about this?

View on HN

Yes the bulk of our business is time series. This includes everything from hardware break downs to fraud detection. I think Jeremy has some good points but in general, but I wouldn't assume that everything is binary. (By this, I mean look at these kinds of terse statements with a bit of nuance)

Usually as long as you have a high amount of regularization and use truncated backprop through time in training you can learn some fairly useful classification and forecasting problems.

Beyond that standard neural net tuning applies. Eg: normalize your data, pay attention to your weight initialization, understand what loss function you're using,..

So if LSTMs are purposely forgetting, do you need less training data than a CNN?

LSTMs don't "forget" more than "remember the things that matter". They don't necessarily need less data. They do have a limit on the "length" of time steps they can handle though. Eg: You can't do thousands in to the future (maybe a few hundred or so)

The long part of "LSTM" means remember good long ranging dependencies.

I hope that wasn't quite what I said :)

I said, IIRC, that they're very similar in terms of results and GRUs are a little simpler. I've seen some papers show better results for GRU, and visa versa.

Oh also I did say that CNNs often beat RNNs, even for time series, language, and audio. So I generally try them first. You can always stick an RNN on top of the CNN and get the best of both worlds.

I just wanted to take the chance to say thank you for the course. As someone who is interested in the topic but isn't a practicioner, just watching it all work in practice has been fascinating and highly enlightening.

Our chatbot uses LSTM(GRU) for prediction, we see about the same performance as LSTM, but training is more efficient.

A recent paper exploring various variatons of neural machine translation architectures found that LSTMs consistently outperformed GRUs in that task.

https://arxiv.org/abs/1703.03906

> simpler GRUs

What does GRU stand for?

Gated recurrent unit, it has fewer gates than LSTMs usually.

Reply on news.ycombinator.com