The battle between GPUs and CPUs in the datacenter is real, but I wouldn't count out regular CPUs yet. Even with GPU-ready frameworks like Tensorflow, Theano, and Pytorch, it still requires a fair amount of domain expertise to get good performance out of GPUs. These frameworks can automatically offload compute kernels to the GPU, but actually getting good performance still requires you to understand the memory model, what data to pin in GPU memory, how to use the limited GPU memory effectively, etc.
The new instruction sets that Intel have been adding (e.g. all of the recent vectorized FMA instructions) go a long way to making Intel CPUs competitive for a lot of this ASIC/GPU stuff out of the box for things like neural networks. Nvidia has been off to a really great start in this area, and they have carved out a very respectable lead, but I think it's premature to say that Intel (or AMD) are having an existential crisis.
You have to wonder why they started with inference and not the other way round. Perhaps it's the case that you train a model once, then use it for predictions many, many, many times, thus using more aggregate compute power. (I used to work at Google, but the above is based on public information and speculation alone.)
The paper does confirm that CPUs are closer to GPUs for smaller or latency-sensitive workloads than most people assumed. At least on 2015-vintage hardware, that is. And they're a lot less trouble to set up and provision.
At least from what I know of ASICs (which is admittedly pretty high-level), this direction is the natural way I'd expect it to go, because the inference problem is much closer in scale to the kind of problem ASICs are already widely used for and good at. Fast inference can be done with a chip of a size similar to what you see in existing ASICs, like specialized DSP chips. Speeding up training is a lot more speculative. You could try scaling up to massively powerful individual chips that can each replace an NVidia GPU (but with lower cost / lower power budget), or improve distributed training so you can effectively use a much larger number of much weaker customized chips. And there are more significant memory and memory-bandwidth issues with training, so it's less of the pure-compute type of problem that ASICs are more often clear wins for. All of which is probably possible with enough effort, but less straightforward, especially to end up with something that beats GPUs on relevant metrics.
I think that was the GP's point? The training is done on GPUs, the execution is done on ASICs, and neither step involves Intel. That's what alarms them. Dedicated hardware might help this, but that seems iffy to me. ML and neural networks are still a fast-evolving field, and instructions that might help today could be useless next year. ASICs can be thrown out and replaced, but instructions that Intel adds to its ISA will be there forever.
True, but for training, in near future, it is still going to be GPU, probably Nvidia GPU only, not too much Intel could do.
> requires a fair amount of domain expertise to get good performance out of GPUs
Rightfully so, but invisible to end user/programmer, if you ever use those frameworks and understand their abstraction level. This falls into the realm of few experienced engineers in Google/FAIR/Nvidia and other respected institutions, who understand their business well.
I think Intel is indeed having an existential crisis: they are losing the future, the future of computation is shifting, away from their comfort zone, and they yet figured out how to stop or catch up with the new trend. If CPU becomes a stock controller component, just like a WiFi-chip, how is Intel going to justify its hefty marketing valuation? It is about time for Intel to do some hard soul searching.
One style of training that would be a good fit for the current TPU architecture is some of the recent work on reinforcement learning without gradients:
As to whether the TPU can perform efficient training or whether it chokes like current FPGAs due to insufficient memory bandwidth is a different question. That would be my guess but I'd love to be proven wrong.
I only know of Nervana's ASIC that is promising to be used for training and Google hinted at it for future work. Does anyone know of other processor, ASIC or FPGA's that are built for training?
GPUs can transpose matrices in shared memory and register space in several different ways that are all efficient. I'm guessing that you are describing the transposed matrices required for calculating deltas and weight gradients here? If so, then I see, it's more than just the fact it has one-tenth the bandwidth of a consumer GPU it's just a less flexible architecture in the first place. One would hope they fix that for the second generation TPU which I assume they're working on already. That said I've yet to encounter a network that requires double-precision math to converge. Have you?
Half Precision can be tricky, but the same trick Nervana uses (dynamic fixed point) can be emulated in software and it has been used for graphics for decades. I specifically made use of it in an OpenGL engine 20 years ago for high resolution texture sampling.
As for Nervana IP, IMO they need to ship something soon or NVIDIA'S going to ship a GPU on par with it.
Will the same happen to other non-x86 groups? Who knows. But we also see that even Intel's previous "accelerator" divisions have moved focus to being "CPU-driven" as well (Xeon Phi), so we have that data point.
I also remember some of the articles from a decade ago that were sure Intel's "might" would help it succeed in the mobile market. I'm not entirely sure if it's as bad for Intel in the machine learning market, because right now there's pretty much only one competitor rather than many.
It's usually much easier to be second in a market than the 5th, as some customers would inevitably get some raw deals from the incumbent, and they'll want an alternative. But if the machine learning market starts becoming entrenched around GPUs, that second player may end-up being AMD, which I think will have decent alternatives to Nvidia's training and inference GPUs in a year or two (mainly for 2019 with 7nm Navi, though).
reminds Sun in 1998 and the onward years when SPARC was like North Korea border patrol in preventing any x86 penetration into Sun even though it was clear that x86 was the future.
The Sun 386i "Roadrunner" was launched in 1988.
A release of Solaris for PC compatibles began shipping in 1993.
In 2002, Sun introduced its first general purpose x86 system, the LX50.
In 2003, Sun announced a strategic alliance with AMD to produce x86/x64 servers based on AMD's Opteron processor; this was followed shortly by Sun's acquisition of Kealia, a startup founded by original Sun founder Andy Bechtolsheim, which had been focusing on high-performance AMD-based servers. Hardware began shipping in 2004.
You don't even have to look that far back. As recently as two years ago, I remember comments here proclaiming with certainty that Intel would crush ARM because they had finally broken through the power wall.
I think a lot of folks, Intel included, have a strong belief in the inevitability of x86 dominance. But ultimately most of the inherent value of x86 is compatibility with Windows software (and DOS before it). Take that away and a lot of the advantages disappear. (Not to downplay the impressive technical and process achievements Intel has made, of course...)
Where/when did you try buying? I didn't think it was difficult to get the 1st-gen Phi cards while they were selling them -- especially as it seemed like nobody wanted them. Admittedly I never bought any myself, my employer picked up some 31S1P from a local server OEM for testing when they were massively discounted.
Note that the Knights Landing, the current generation is released socketed only for now, so you'd need to buy a server or the ninja dev platforms Intel sells through Colfax.
Still, I would have very much liked to play around with the 'phi'.
> Admittedly I never bought any myself, my employer picked up some 31S1P from a local server OEM for testing when they were massively discounted.
That's the kind of deal I was looking at too. Maybe they were massively discounted because they were already EOL'd?
Well, if you're still interested, KNL is a significantly better architecture, so you could just try now.
> That's the kind of deal I was looking at too. Maybe they were massively discounted because they were already EOL'd?
Refreshed my memory and I was wrong; there was some discussion to jump on the $125-$195/card deal Colfax had late 2014 [1] (likely dumping stock due to EOL), but we figured that they are not even worth that much. In comparison, the one card we have was actually bought just 10-11 months before for ~$1500.
Intel acquired Digital's StrongARM unit as part of a settlement for a patent infringement suit.
What advantage does using an ARM core for this have over simply using an x86 core?
So I stand by my point:
> What advantage does using an ARM core for this have over simply using an x86 core?
But if you used 68000, Z80, 6502 chips as a co-processor to an 8088 CPU or Intel X86 you could assign tasks to them to free up the main CPU and even doing parallel processing.
Sure Intel can make low power CPUs but if you have an X86 and ARM chip working together, like an Apple Macintosh that wants to go to ARM eventually but needs X86 code to run as well, if they make Macs with an Intel X86 and ARM chips they can run iOS apps on the ARM and MacOS apps on the Intel to run more apps than a Mac normally has.
Edit:
I like to also add if Apple put am ARM chip on their Macs with an Intel chip and required the ARM for newer MacOS versions they can lock out the Hackintosh users from pirating their OS.
Parallel programming is already possible using a CPU with multiple cores, which I think Intel would love to add more of if developers found usages for them that end users love. No necessity for a CPU using a different instruction set, which just makes programming this more complicated.
EDIT: And if there are active threads that require much less CPU power on some core(s), for modern CPUs the respective core will already enter energy saving states aggressively (giving lots of the advantages that adding low power cores has).
You got the Raspberry PI Zero series that only costs $5, a system with an ARM CPU on one card. What could Intel chip do that and still cost $5?
The more cores added to an Intel chip, the more electricity it uses and the higher price it becomes.
Sure it would be complex, the Commodore Amiga was complex but people figured out how to program the co-processors, etc.
Right now for Bitcoin mining, they use GPUs or ASIC chips to solve SHA hashes.
It is much cheaper to produce as few versions of a chip as possible and deactivate cores (e.g. by fuses) for the cheaper versions. Also energy saving functions in processors have become so much better for the last, say, ten years that one has not to care anymore for energy usage of silicon that is not used.
> Sure it would be complex, the Commodore Amiga was complex but people figured out how to program the co-processors, etc.
At that time no less complex design was feasible. With today's technology it is. Why should one prefer to program a convoluted beast over programming a much more clean design?
We have so many people warning us about AI, and most of them are famous people who don't even know how it works because of how complex it has become. People often fear what they don't understand.
It is also why some people program in Haskell even if it is more complex than Python or Ruby on Rails.
I am talking about having an ARM CPU as a co-processor to an AMD or Intel system. You could even modify the X86 chips to have an ARM core or two because the design of the ARM chip is simpler than the Intel, which is why the Raspberry PI Zero is only $5 and is on a small circuit board powered by a USB cable.
The need for ARM code comes from mobile devices and in the Post-PC, Post-Microsoft, and Post-Windows era we need to look to alternatives to the way the PC used to work which was simple but out of date.
When JFK talked about going to the Moon, he didn't say it because it would be simple, he said we should do it because it is hard or complex. When he said that he had no idea how the rocket would get to the Moon, etc.
I am talking about a PC/Mobile hybrid using X86 and ARM technology that can run code made for both types of chips. One that both chips can work together as a team by assigning tasks to them.
Imagine if you will, Apple makes a new Hybrid Mac, it has an Intel CPU and ARM Co-Processor. It runs a new version of MacOS that has a virtual environment to run an ARM OS like iOS inside of it and has a touch screen. A developer can use XCode to create ARM iOS apps and then test them on the ARM chip which is the same in the latest iPhone/iPad, etc.
Do you think MacOSX was simple or complex to design? It was based on Unix instead of writing from scratch. It is designed to only work on Apple branded PCs aka Macintoshes. If it were simple to design it would have met the same fate as Copeland and other failures like the Apple Lisa or Apple III. Instead, it took Unix which has a higher learning curve and put the Mac GUI on top of it to make it simpler.
Also, Linux do you think to hack the Linux kernel is simple or complex? Do you read all of the cussing Linus does when someone submits unstable code to it that they never tested? Do you know how hard that is, and to the user they know nothing about it and think it a simple thing?
Do you not know that electronics use math and formulas to work? Do you know that Algebra, Trigonometry, Geometry, Calculus I and II, and Differential Equations as well as Linear Algebra are also not simple but complex that students struggle to try and learn? What about Statistics as well as Finacial math? Not simple but complex. There is a lot of complexity done in computers and programs or apps or whatever you want to call it. To the user it appears simple to those of us who went to college and learned data processing aka information systems and computer science we know how complex a computer is. The Amiga was so good because not that it was simple or played video games, but it could emulate the Macintosh with AMax and eve if it has a slower 68000 than a real Mac the co-processors allowed it to run faster than a Mac with a faster chip in it. End users didn't know how complex emulating a Monochrome PC DOS machine, or a Macintosh was, and only saw how easy it was to do so.
We are getting into the Mobile device age and the Internet of Things age and soon Qunatum computers and the Qubits that will change the way computers work and be more complicated than the X86 PCs or ARM based mobile devices are. The future belongs to the complex design that works better than the simple design.
This is the same mistake that most tech companies make, the people who can debug and write programs who are experts that write complex code, design complex computer systems, and make it so there are fewer bugs and fewer design flaws and save the company millions in expenses for better quality control that needs less help desk hand holding support?
Do you ever notice when these people are fired or downsized that quality control suffers? That the OS and apps get bloated and slow, and suffer more bugs and flaws in it, even hackers stealing data like passwords and email addresses because someone didn't sanitize code because management wanted it more simple than complex. That they hired 20something out of college or dropout people to replace the 30+ or 40+ year old experts for less money.
Sure you could say programmers are a dime a dozen, you get 500+ resumes a day for a programmer's job? You go with the person willing to work for less money to cut costs, but the customer suffers.
Did you know why Apple has been more successful than Microsoft? Steve Jobs learned Six Sigma quality control to make Apple hardware and software a better quality than Microsoft's. Microsoft got rid of the experts that made Windows stable or made a good mobile phone and Vista and Windows Powered devices were awful. Microsoft tried to deny it, but buying a Macintosh or iPhone was better than anything Microsoft put out because of quality control. Without Steve Jobs and his experts at Apple, Apple is starting to suffer as Microsoft had when quality was bad.
Google and Amazon had this problem but found a way to fix it by hiring better experts and quality control people.
So tell me did Evolution and Natural Selection makes a simple human brain or a complex one? It made a complex one that we have problems trying to figure it out how it works, while we try to copy it to an AI program as a series of nodes in a grid or matrix. If the human brain was so simple, we couldn't even figure out how to make computers. Yet because it is complex it is a superior design. So if complex is better, why settle for simple?
Fifteen or so years ago I read someone mentioning that the ratio of processor state transitions per bit of program state was growing almost exponentially.
Anecdote: Starting in the early eighties I started using CAD programs. And they were painfully slow. Yet around 1995 suddenly they weren't 'slow'. That was huge. Since then improvements in processor speed and memory hasn't provided nearly as much functional improvement.
What I think is we're in a world were current processor designs are over kill of much of the programming tasks people need done. What people want is not more speed but better efficiency. That points towards using excess silicon to implement task specific coprocessors, not for more unneeded GPU performance gains.
This leaves Intel in a bad spot. Especially since everyone and their dog is loath to design in Intel parts if they can help it.
Intel has Xeon Phi, but what else are they doing? There is so much opportunity in new computing architectures but it seems like Intel has been taken by surprise. I hope this giant wakes up.
It is essentially what modern cores have with a their split caches, except those things are called "caches" because the architecture then has a lot of machinery for pretending to be a von Neumann Machine with a huge memory. This pretending is done because that's the kind of machine that software expects.
But the first unices ran on machines with memories only as big as a modern L1-cache. Moreover, networking and parallelism are everyday issues for software now. It will be interesting to see if software evolves to explicitly recognise that it is really running on a network of small machines + a hierarchy of big external storage devices, of which DRAM is just one.
The difference is of course that we should not see the whole cluster of cores as a Von Neumann machine, the memory bandwidth is just too low to drive multiple cores with some external memory, memory must be as close as possible to the cores themselves. Quote from wtallis lower in this sub-thread :
> You can't pretend that memory is all equidistant from the processor cores unless you want everything running equally slow.
If you are a large multi-national where your product pipeline is shallow due to competitors creation better products than your R&D.
I think the Xeon Phi is an example of something which fell flat.
And intel doesn't have these people. They don't want the embarrassment and are too entrenched in their own processes. The technology as of this year can compete or stomp intel into the ground. We can program ASICs to handle any task, just about as fast as it can handle the electron flow. The only real setback is the fact the instructions are not hard-coded in the actual physical logic. That's literally the only slow-down. Were it not for that translation layer right there, ASICs would come out on top, no problem. Specialization is the future. General purpose was good enough for 30 years, it's time to move on.
Is that like La Brea Tarpit? Just say ASIC.
An ASIC is an Application Specific Integrated Circuit. In particular, a GPU is not an ASIC. Saying Custom Application Specific Integrated Circuit Processor is doubly redundant.
https://en.wikipedia.org/wiki/Application-specific_integrate...
Where you're right is that Nirvana was a chip company founded for this ASIC and I think the term "custom" is misnomer. You could argue that Google isn't a chip company so theirs are custom.