Comment by doctorpangloss | Hacker News Reader

Comment by doctorpangloss | Hacker News Reader

Parent Full thread

> Deep learning enthusiasts probably will stick to NVIDIA cards

We've been building the wrong hardware for ML for a while now.

A TPU doesn't delivery supremacy for problems over GPU hardware. It's a Google senior engineering retention and PR project.

Exceedingly few problems resemble image recognition. You wouldn't be able to tell from the research, because due to the tools it's sort of the only affordable thing to do.

Playing Go barely looks like it. Or more accurately, Go doesn't look like most other games. Molecular physics and synthesis barely looks like it. Neural programs/neural Turing machines that matter (converting complex sim code like game rules and physics into neural networks via learning) may never achieve the accuracy or performance/watt necessary to really compete with just making a bunch of CPUs or straight up dedicated hardware.

Word embeddings are a disaster. People keep trying to do innovative stuff with it, and it's 2018 and we've just just recently got entity names from Wikipedia. I think Peter Norvig, having basically only machine translation to point to, is really going to eat crow when non-perceptual AI tasks sort of have nothing to do with stuff Google has developed. But what do I know.

I'm sure someone's going to trot out some obscure TPU or GPU-driven thing. It's been years! Awesome innovations in hardware really do lead to obvious, immediate crazy cool stuff. I'm really talking about the Kinect as an example here, in that it was really hyped and not much came of it. The TPU, and ML-specified hardware targeted to today's problems, is a slower-motion Kinect of our time.

Mining cryptocurrency, the most recent innovation, is the opposite of cool.

I'm confident these tools exist to capitalize on the subsidy from the gaming industry. AMD is wise to not chase ML features.

> talking about the Kinect as an example here, in that it was really hyped and not much came of it

The underlying technology was acquired by Apple and is used on iPhones for FaceID, https://www.theverge.com/circuitbreaker/2017/9/17/16315510/i...

> PrimeSense was never solely responsible for the technology in Microsoft’s Kinect — as evidenced by the huge improvements Microsoft made to Kinect 2.0 on its own — and it’s also obvious that Apple is doing plenty of new software and processing work on top of this hardware. But the basic idea of the Kinect is unchanged. And now it’s in a tiny notch on the front of a $999 iPhone.

Matrix multiplication is the bread-and-butter of ML (both training and inference), for fairly obvious and solid reasons. Luckily for us, it's also very important in all sorts of domains, including graphics, rendering and gaming. That's all there is to it. There's no real "subsidy from the gaming industry", only a cross-subsidy situation in which common problems that are shared across domains can take advantage of the increased scale that comes from sharing the GPU-compute platform.

I'm not sure what you mean by "ML features", but AMD/ATI's dismissive attitude towards the ML and GPU-compute communities is looking quite silly by now.

I'm not sure what you mean by "ML features", but AMD/ATI's dismissive attitude towards the ML and GPU-compute communities is looking quite silly by now.

I disagree, ML computing has tons of competition and its margins will be quickly diminish. I think they are right to focus on gaming.

> Exceedingly few problems resemble image recognition.

It makes it extremely difficult to take any of what you just said seriously knowing that you made this statement.

I don't know dude, if there weren't such lucrative jobs for academics at places like Google and pharmaceutical research companies, the discourse wouldn't be so narrowly focused on doing blah blah neural networks on huge libraries of images and molecular hashes.

People with academic backgrounds would be less afraid to point out how much vaporware machine learning has produced. They'd take stock in that their tenure, or six year PhDs, give them an opportunity to not chase trends so aggressively. They'd shill online less, for some giant faceless corporation like Google.

I'm not going to shit on a bunch of great stuff people have made. But the scary thing is that Noam Chomsky is probably right, that AI science hasn't advanced much at all. That we're still in an AI winter where it matters, the science, despite all the money and attention.

> the discourse wouldn't be so narrowly focused on doing blah blah neural networks on huge libraries of images and molecular hashes.

Do you mind clarifying what exactly your issues are with these specific applications of machine learning? Is it that you think there isn't really any meaningful value that can be gained in working within these areas? Or is it just that you want people to spend more time researching a different topic that you think would be more interesting/compelling?

Care to expand? Prima facie this is a very reasonable stance. Further, image data has a lot of structure not present in other data and it's far from obvious what to do with ML techniques in other contexts.

This is coming from a physicist starting to investigate ML tools in my own work.

Some of the greatest advancements in research and application of machine learning have been within image recognition in the past few years. The inflection point and subsequent boom of deep learning circa 2012 is basically marked by the ImageNet competition of that year.

I think we are at a point where the cool hardware will actually be low power inference (deployed on basically the class of models we have now).

What is the right hardware for ML?

That will certainly depend on the kind of ML you want to run. What is the right hardware for computing?

They absolutely need tensor/matrix multiply cores. Incremental fp32 improvements aren't impressive.

Seems like what you need for inference is usually very different from what you need for training.

Probably Larabee. Too early and too expensive for its time.

The Japanese PEZY machine looks incredibly interresting. Too bad it is only used in that one super-computer.

Its overall design sits somewhere between a GPU and CPU. Like a GPU, it supports 8-way hyperthreading (GPUs typically support 10. CPUs usually only support 2... maybe 4 for Larabee). Like a GPU, its programmed through an OpenCL-like language (I know you don't like that, but its how it works).

However, the overall layout is very CPU-like as well: a "City" is composed of "Villages", which is composed of multiple "Processing Elements". Each PE is scalar like a normal CPU, although 8-way SMT. So its not like a GPU's SIMD or SIMT architecture in this regards.

--------

See information here:

https://en.wikichip.org/wiki/pezy/pezy-scx/pezy-sc

It really is an interesting design that toes the line between CPU and GPU.

Is PEZY still a going concern after the arrests?

So that's why Xeon Phi has been so successful!

/s

Real talk, though. Why do you think larabee is the "right" hardware?

I'd assume because of the extra serial processing power of a CPU.

x86 isn't the right choice though. As the chips get smaller, all the x86 cruft starts to become a larger portion of each core.

RISCV with 40-ish instructions plus a SIMD extension would probably be ideal. They've already proven that those designs use less space per core than ARM.

The truth is that for many types/sizes of models AVS2 or AVS512 in Xeons is as fast and GPUs.

I am interested in seeing benchmarks regarding this. I believe this is true for models with truly huge embeddings, but otherwise if your models are even a little compute dense then GPUs are faster.

My experience is that you need the additional caveat of a streaming fairly homogeneous and highly parallelizable approach or the memory transfer, branching and communication overheads will eat away nearly all the GPU gains.

I've also noticed that a lot of the time, people are comparing GPU to naive implementations (and sometimes, implementations written in dynamic languages) instead of to highly tuned BLAS or MKL implementations. For a large swathe of problem types, using the fastest math library will reduce the CPU/GPU gap to less than an order of magnitude or less.

No, they're not. Straight front the horse's mouth:

https://software.intel.com/en-us/mkl/features/benchmarks

In no case will you see it get close to 10TFLOPS. GPUs easily do this, and can approach 100TFLOPS with tensor cores.

I don't dispute this. If you can remove memory transfer overhead and meet the requirements I mentioned, then GPUs will be much better. But there are many problems that do not fit that (SIMD per warp) regime. In short, GPUs are no panacea and have their own trade-offs and bottleneck sensitivities, just like everything else.

I agree that they are no panacea, but we also have an example of the Intel Xeon phi, which did have high speed hbm memory. That too did not compete with GPUs. I think the majority of it has to do with the GPU chip itself being extremely simple, with no Branch prediction, no branch difference, very simple caching behavior, etc. They are able to allocate a much larger portion of the die to just throw fma units at it.

> comparing GPU to naive implementations (and sometimes, implementations written in dynamic languages) instead of to highly tuned BLAS or MKL implementations

In your experience, are these comparisons using naive GPU implementations as well? If so, that still seems like a valuable comparison.

Well, I'm speaking from personal experience, for many (but not all) text-classification IO-intensive models with large embedding tables Xeon on TF compiled with AVX2 or AVX514 and FMA is as fast as GPU. Obviously these models need to be on the lower spectrum of computational complexity.

This is not true. I've benchmarked it, and you can see yourself by looking at the MKL benchmarks versus cuBLAS. cuBLAS is always significantly faster.

Reply on news.ycombinator.com