Hi, glad to see you here. We were using spark for logistic regression training but switch to MPI now, because of the 2GB problem. I think LR was spark's killer feature, would you make it better? Thanks.
Can you talk about these limitations and your experience?
https://issues.apache.org/jira/browse/SPARK-1476
https://issues.apache.org/jira/browse/SPARK-6235
You'll hit this bug when your model size is larger than 2GB.
BTW, Recomputation of RDDs may result in duplicated accumulator updates. So please do not use accumulator in your trainer for gradient summation. They know that, but they said they will not fix that.