Apache Singa, a Distributed Deep Learning Platform | Hacker News Reader

Apache Singa, a Distributed Deep Learning Platform

singa.incubator.apache.org

92 pointsbased218 comments

There are so many Apache projects these days (mostly seem to be in the incubator) - where do they all come from? I had once thought that Apache was a dumping ground for the technology of failed startups, but a lot of these projects seem to find new life in the incubator - so they must be doing something right.

The list of big name Apache projects is nuts, look at their project lists and you can easily pull out dozens of big projects and even incubator has a few popular projects.

It's hard to find whats worthwhile and what isn't.

Lots of these (including this one) come out of universities, or university-corporate research partnerships. Personally I'm very glad that universities nowadays seem to be more okay with publishing tangible results (i.e. source code) as opposed to "just" the research papers. I don't think it's always been that way. I can't begin to imagine the amount of work that is rotting on someone's old usb stick, because they weren't allowed to publish their PhD theses's source code.

where do they all come from?

From what I've seen, there are two major sources, and one slightly less prevalent one:

1. Big companies. If you go through the incubator, you'll find quite a few projects where there was a large code grant, and many initial contributors, from one large company or another. Facebook, IBM, Yahoo, Oracle, etc.

2. Academia. A lot of projects are existing OSS projects being developed as part of academic research, and now the team wants to formalize the project structure and get the project to a wide audience.

The slightly less prevalent one:

3. Just ordinary OSS projects developed $WHEREVER, not necessarily affiliated with any company or university, and the team just decide that Apache would make a good home for their project.

Anyway, I definitely wouldn't call the ASF "a dumping ground for the technology of failed startups". If anything the incubator is designed to prevent that sort of thing. If projects don't successfully complete incubation, they don't become full fledged TLPs. Now, there are people who quibble over how long some projects linger in the incubator, sometimes in a sort of half-dead/half-alive zombie state, but by and large the process works pretty well.

No GPU support => not interesting. GPUs are an order of magnitude faster. If you're doing deep learning on CPUs you're wasting your time and/or money.

if your data exceeds memory then a GPU is worthless. distributed makes it scalable

It's not trivial to take any existing algorithm for training and scale it not only cross-GPU but also cross machine. Training tends to be very bandwith intensive. In fact that is the entire reason Nvidia is so heavily focused on this with their upcoming GPUs (Pascal). I am not saying it's not possible, just that it's a lot harder than "distributed makes it scalable".

This is wrong. Training data can be streamed through GPU memory during training. It's your parameters that can't exceed GPU memory. You can get GPUs with 12 GB of memory, and they also support float16 so they can be twice as memory efficient as CPUs. If your model has more parameters than that, then you'll be waiting months or years for a single model to train using CPUs, even distributed.

Furthermore, almost any technique you use to distribute and scale training will work just as well regardless of whether the computations are happening on CPUs or GPUs.

At least back in 2012, Google research seemed to suggest that distributed CPU training of large models could sometimes be preferable to fitting within the limits of GPUs:

http://research.google.com/archive/large_deep_networks_nips2...

This is also not quite right. Models whose parameters are too big to fit on one GPU can be trained by splitting them across multiple GPUs, as was done here, for example: http://papers.nips.cc/paper/5346-sequence-to-sequence-learni...

According to the paper, the parameters fit on one GPU (or at least that one GPU was able to train the model). It was just too slow, so they trained on 8 GPUs in parallel. But those GPUs were still on the same machine (one node, multiple GPUs).

SINGA provides an abstraction for defining all known models,a neural net structure that is easy for model and data partitioning, and a parallelism model that supports both synchronous, asynchronous and hybrid training frameworks. Processing at each node could the be CPU, GPU, or CPU-GPU based.

It is in their schedule for december http://singa.incubator.apache.org/develop/schedule.html

It's only single node, multiple GPUs, though, which already exists in other frameworks.

With the parallelism model and abstraction it has to support model/data or hybrid partitioning, and synchronous/asynchronous or hybrid training, it should be easy to extend to GPU cluster. However, training is required only periodically, and if it can be done on existing clusters as efficiently, why not?

And Cuban hackers everywhere chuckle at the unfortunate slang cuss word chosen as the name of this project... http://www.urbandictionary.com/define.php?term=Singa

Singh is actually the Indian Sanskrit root word for Lion and is common in south Indian language of Kannada, Tamil and Malayalam which has Simha, Singham and Simham as the word for lion respectively. The Sri Lanka's ethnic majority are called Sinhalese which again is derived from the Indo-Aryan Sinhala language bearing similar influence, meaning the "lion people" or "people with lion blood", while a sword-wielding lion is the central figure on the modern national flag of Sri Lanka.The island nation of Singapore (Singapura) derives its name from the Malay words singa (lion) and pura (city), which in turn is from the Tamil-Sanskrit சிங்க singa सिंह siṃha and पुर புர pura.

Reply on news.ycombinator.com