Not related to the original post, but I recommend giving CuDNNLSTMs a try (if using Keras like the OP), the speed increase is absurd. (2-3x).