Hacker News Reader

Reynold from Databricks and the Apache Spark project here.

Would you mind shooting me an email rxin at databricks.com so I can understand more the issues you run into?

Hi, glad to see you here. We were using spark for logistic regression training but switch to MPI now, because of the 2GB problem. I think LR was spark's killer feature, would you make it better? Thanks.

Oh wow... I didn't even know these limitations (and in fact ,they are not very Google-able).

Can you talk about these limitations and your experience?

https://issues.apache.org/jira/browse/SPARK-139

https://issues.apache.org/jira/browse/SPARK-1476

https://issues.apache.org/jira/browse/SPARK-6235

You'll hit this bug when your model size is larger than 2GB.

BTW, Recomputation of RDDs may result in duplicated accumulator updates. So please do not use accumulator in your trainer for gradient summation. They know that, but they said they will not fix that.

https://issues.apache.org/jira/browse/SPARK-732

https://issues.apache.org/jira/browse/SPARK-5490

Reply on news.ycombinator.com