Pixel Visual Core: Google’s first custom-designed co-processor
blog.google
blog.google
This together with 10bit video profile for h265 (think phone hardware is capable now but haven't seen any phones with 10bit video yet) would make mobile video a whole lot better, right now it frankly sucks for anything with high contrast or dynamic range.
I would love to see the apples to apples comparison of the Pixel tensorflow computation running on a vanilla snapdragon GPU vs this custom core.
Also will be interesting to know why halide and not opencv.
It seems that this is more an attempt at locking the tensors to target custom metal, so that it can't be copied and made to work on other phones.
For examples, the Google Camera NX version is a port of the HDR+ feature on compatible phones.. Even the cheap Xiaomi. They actually give results comparable to Nexus.
However, by locking the model to a very custom image processor (and custom IR) , this can't be ported. This looks to me as a version of walled garden more than a performance win.
Also your argument that it’s not about perf makes no sense to me. Sure other phones have HDR, I bet its way slower and more energy intensive than the Pixel 2. Except maybe the iPhone, which may also have custom silicon. If they had custom HDR tech in software they wanted to keep they could just keep the source closed, or patent it.
I find it telling that they compared cpu vs tpu... Not GPU vs tpu.
My question is why do you care about opencv when benchmarking neural network performance on various hardware?
People don’t use both for the same problem, nobody is using “tensorflow code running on the halide ir”.
Both approaches use a lot of basic operations applied to matrices, so the custom ASIC supports either, but separately. HDR sounds like a better fit for a classical approach, so their HDR filter is probably written in Halide, with no machine learning or TensorFlow anywhere.
Ask HN: What happened to tensorflow lite? | https://news.ycombinator.com/item?id=15494368 (Oct 2017, no comments)
>zitterbewegung: They announced it at Google I/O in May See : https://techcrunch.com/2017/05/17/googles-tensorflow-lite-br... I can't find another announcement is this vaporware?
Firstly yes Apple is well ahead in CPU core design and has been since 2013 when they brought out the first 64-bit ARM chip. And yes I know 64 bit by itself doesn't account for the performance increase, but the 64-bit ARM core architecture is much more streamlined and efficient than 32-bit which had accumulated all sorts of cruft over the years. The A7 wasn't just 64-bit, it was step change up in performance creating a gap that the other ARM vendors have never caught up with.
That's not the whole story though, core design alone isn't the only reason Apple chips are faster, another is they have dramatically more cache memory on chip that other designs. The A10 had a total of 7MB of L2 and L3 cache, more than double the maximum configuration for the 835. Apple SOCs are massive, which makes them expensive.
Finally, they can build in customised capabilities in hardware to support specific high level features, such as the secure enclave and the neural engine in A11 Bionic to support image post-processing effects and face recognition. This is clearly what Google is trying to do, but in an add-on chip because they're not (yet?) up to designing their own SOC.
So it's not just the the Apple SOCs are faster, the reasons why they are faster are on several different axes. Some SOC vendors apparently say their profit margin lies in shaving off just one or two square millimetres of die size. Market forces like that would never produce a chip like the A11. So really this approach by Google of designing their own custom co-processor, and maybe eventually their own SOCs, could be the only way to close this gap.
Multiple designers could run circles around Qualcomm, but the marginal cost of doing so combined with the number of CPUs you'd have to mint to do so makes it a losing proposition. Apple's in the unique position of being able to leverage Intel to bend Qualcomm over a barrel, and Qualcomm fires back with a gazillion lawsuits and New York Times articles that cry "woe is me, Apple is eating my lunch!"
On a different topic, benchmark navalgazing is probably not going to help anyone. Every company that's been designing CPUs from the mid 80s on has been writing benchmark defeating code, and I'm 100% certain PA SEMI/Apple is no different; they certainly have the silicon space to waste writing Application-specific-accelerators for acing benchmarking tasks, especially on a platform with as much opacity as Apple holds over the iPhone.
If Qualcomm is that big of a drag on Android (or not-Apple, but it amounts to the same thing) smartphones, maybe the solution is for Google to buy them outright and rearrange their practices.
Sure, it'd be expensive, but not anything Google couldn't afford.
>Qualcomm doesn't really seem capable of keeping up with what Apple is doing. Snapdragon 835 came out 6 months after the A10, and scored about 2/3 as well at single core, and less than 10% on Geekbench 4. A11 scored twice what the 835 did on single core, and over 50% more on multi core, and the 835 is going to be the best chip available in the U.S. for Android until sometime next year.
Qualcomm and Apple are on different release cycles. The SD 845 will be in the same ballpark in multicore score and about 25% slower in single core than the A11.
As for Qualcomm and Apple being on different release cycles...that's exactly the problem! They're 6 months behind AND still have worse performance, plus due to their deal with Samsung, it's going to take even longer than that for any other manufacturers to ship devices with 845.
As another comment said, it's not really going to be profitable for any company to try and compete with Apple on this. Which is why if it is going to happen, it's going to have to be done by someone like Google who has a vested interest in it beyond profitability from the chips themselves. If ARM reference designs can't keep up with what Apple is doing, and nobody else is willing to invest to go beyond them, Android will fall behind in the high end of the market.
Is it like a integer GPU? The massive amount of ALUs suggests something like that.
But later, I see "Notably, because Pixel Visual Core is programmable, we’re already preparing the next set of applications.". So, is it FPGA-like?
Edit: wrong name
That unit is massive, and wouldn't fit in a small die like this.
It seems likely it's optimized for the same kind of operations though, even if it doesn't have a matrix multiply unit.
I would guess the main intended use for this silicon is running neural networks, even though the initial use case is for photos.
Nearly all AI things on phones (voice recognition, google assistant's local features, keyboard predictive language model, offline translation, etc.) are severely compute limited, and could perform much better with this silicon.
Some features are obvious candidates to put on-device, like realtime recognition of the contents of a photo, realtime wavenet voice synthesis, yet compute limitations preclude it.
To my eyes, the shadow boost is a little too much. I'd ideally like a slider to customise exactly how much brightness you're adding. Personally, I'd probably turn most of these down a touch.
Google clearly has fantastic technology driving this image processing, but I think they need a little more work on their taste.
I also don't know if Google actually consults with real photography experts when creating these machine learning techniques. Apple made a big deal out of doing that at the last event to create "beautiful photos". It's not enough to just "mathematically improve" something. It should look very pleasing to the human eye, too.
Perhaps if Google did that, too, it would have avoid issues such as this one, or the somewhat unrealistic look of its portrait mode blur in photos. It's hard to explain, but to me the Pixel 2 portrait shots look like the person doesn't "belong" in the environment. Like they have a linear-blur wallpaper behind them and they photograph themselves in front of it. Or almost like they are photoshopped into the image. I wouldn't say it's that bad. And a majority of people probably won't notice it immediately, but I still think it needs to be improved (perhaps, I don't know, by actually using a dual-camera setup like Apple and Samsung?).
https://www.gsmarena.com/google_pixel_2_xl-review-1670p4.php...
There are many non-obvious issues with this as well. You're most likely thinking of, for example, an non-centered subject can create an interesting composition and algorithmically centering the subject would not create better looking photos.
However, in film's history, we've seen older technology briefly preferred for no other reason than momentum. An example off the top of my head is higher frame rate video does not have the "movie feel" that 24 fps film does. That doesn't mean 24 fps is ideal, its simply familiar.
But I think the (depressing) truth of it is that I notice it because I've had fast lenses and spent a lot of time taking photos, so I know what natural looking bokeh looks like. And bokeh is also just an artifact of a lens and aperture combo, so who is to say that extreme blur with a sharp line as they are doing isn't just going to be the new standard?
I suspect the next generation will come to prefer this look (if it continues) and find the "natural" effect with real glass as weird.
I guess most people won't load the image into a paint package to fix it, so their aim is just to automatically do a job nearly as good as a trivial change a human would make. Still, I was hoping for more.
I think they just chose bad examples. The results from this Google Research blog post are much much more impressive:
https://research.googleblog.com/2017/04/experimental-nightti...
I wish my smartphone could do _that_ automatically.
Exposure fusion from multiple images has much higher usable dynamic range and lower noise.
> Using Pixel Visual Core, HDR+ can run 5x faster and at less than one-tenth the energy than running on the application processor (AP).
But if you need the computer to do all the color adjustment for you, so you can post your snapshot on facebook or print it at the local drug store with no manual intervention, this might be a good enough result.
This thread seems to be filled with amateur professionals thinking that their iPhone or Pixel will magically do something optically it can't (yet?). It's good enough, and in some cases a spectacular ROI (both Apple/Google/Samsung/LG/etc) for the price. The photography giants wouldn't sell or develop new products if they felt a lot of pressure in this space. You shouldn't have an expectation that your $1000 phone competes with a few thousand in camera, glass and filters in the right hands. It won't.
Edit: words
This is key. I own some decent camera gear and I'm amazed at what is coming out of all of the top end phones. Of course I can critique and find faults, but the pictures that non-photographers can take with their phones in a matter of seconds are down right amazing.
I actually wonder if the gear heads feel their domain threatened? I'm waiting for the day when I don't have to lug around my camera and multiple pounds of lenses in order to get all the shots I want :)
But that simply highlights that simply owning good gear doesn't mean you can shoot good pictures. You still need to learn lighting, composition, posing your subjects, etc etc, which was always the case!
>My point was that phones can and are making great pictures without requiring a bag full of gear, and that it's highlighting that gear is secondary to the skills you listed.
I'd phrase it differently though, and I'm sure you'd agree. I'd say that gear isn't secondary, but gear is a tool that should be used when appropriate. You still need carry an external flash/light/etc if you want to light your subject in poor light. You still need to carry a long lens if you want to take a picture of an eagle perched in a tree. You still need a tripod if you want long exposures, etc, etc.
> You shouldn't have an expectation that your $1000 phone competes with a few thousand in camera, glass and filters in the right hands. It won't.
IMHO, it doesn't even beat $500 worth of camera equipment.
Them looking bland is a function of them being made in challenging circumstances.
It will be a looooong time until we see that on a pro camera with interchangeable lenses, if ever.
If you can, you might be able to postprocess it... But as far as I know, no DSLR cameras even have gyros, nor sufficiently high burst rates, so that kind of photography is out of the question.
It comes out to over a gigabyte of data per image... and you can't compress it before processing...
I guess the long and short of it is these camera companies aren't software people--except isn't Sony a big camera company? You'd think they'd be able to bridge the gap.
So long as it involves multiple pictures and the inputs (exposure params)are computed are taken done when the photo is captured, then it does things you can't do with image processing in any single image.
You can always extend numerically the DR of any set of values but you will stumble upon some noise eventually.. This HDR+ module is supposed to have less noise in the shadows OR have better sun effects / beautiful highlights (candles for example) / good looking clouds. A good test would be shooting in strong direct sunlight and look at the shadows.
I couldn't see whether that was done though, it could be done with most of the same effect by simply exposing e.g. +2EV and -2EV and simply deciding afterwards how to blend those images into a HDR given the image content (faces, ...)
This is almost exactly how you'd make super-resolution pictures (adding a nearest-point scaling up step at some point! Yet their super-resolution app (Photoscan) has some sort of artificial upper resolution limit. If they do plan to expose this capability to 3rd parties, I hope someone makes a proper scan/super-resolution app! Also tried Office Lens, also left disappointed :(
Everything you need is available in the camera2 RAW API.
Using the users hand movements help cancel out various types of per-physical-pixel noise too.
Or is Samsung actually right not to go down that path and the economics of Android handsets just don't support the significant extra spend and commensurate price bumps that sort of investment would require? The Pixels look like great devices, but without dramatically upping their distribution and marketing game is this really a vanity project for Google? I hope not, I'm crossing my fingers that with the talent they acquired from HTC, Google is really intent on making a splash in the market with the Pixel and not just in the tech press.
Google has ML experts, compiler experts, etc. who probably contributed a lot to this. I guess it is a perk of keeping domain experts in the company who are there to advise when time comes.
I am routinely blown away by how well it performs at all lighting. Everyone i've shown pictures from low light scenarios is always impressed.
HDR+ is a big reason I bought the Pixel 2, I know its going to keep getting better.
It's going to seem like magic to the user.
I've held out for so long, it's finally time to replace my HTC One m7.
On the plus side, the co-processor seems programmable, which means quality can improve without the need to upgrade the hardware.
The basic idea of using special hardware makes sense. Apple has done the same thing, and as you get more and more data off the sensor this seems like the only way to do analysis/adjustment in real time without wasting a ton of power.
Googles ad revenue is so important that they should do absolutely anything and everything to protect it, spending billions on R&D to keep android at the forefront is essential, and at the moment they are only really competitive on the software side (A11 smokes latest snapdragon chips, as does the A10....), offering a custom chip would give android the vertical integration that Apple leverage to be so dominant.
They say it uses Tensorflow, but is it a limited subset of Tensorflow designed only for image processing, like running a pretrained model on a TPU?
Is that hardware only accessible by the camera? If so, isn't that wasteful?
"HDR+ will be the first application to run on Pixel Visual Core. Notably, because Pixel Visual Core is programmable, we’re already preparing the next set of applications. The great thing is that as we port more machine learning and imaging applications to use Pixel Visual Core, Pixel 2 will continuously improve. So keep an eye out!"