Comment by quietbritishjim | Hacker News Reader

Comment by quietbritishjim | Hacker News Reader

Parent Full thread

quietbritishjim

> You really want to use ... C/C++ for inference, not Python ...

> You need ... C/C++, not Python.

I think this is a red herring. Usually for deep learning you just use Python to plug together the libraries that actually do the processing, and those are written in terms of C/C++. You can see that in the article where the numpy array returned from OpenCV's video capture API is passed directly to tensorflow. Python never touches the individual pixels of the image directly, and once that's inside tensorflow it's irrelevant that a Python object briefly represented it.

> with a Jetson Xavier

Well that's obviously the real difference. It's not even just the same general type of computer but a bit faster - the Jetson has a decent NVidia GPU on board whereas the Raspberry Pi is doing the processing on its extremely limp CPU. Indeed that's the whole point of the Jetson; it's basically an NVidia graphics card with extra components strapped to it to turn it into a full computer.

> You really want to use DeepStream ... not TensorFlow

I'm not familiar with DeepStream, so I'm not so sure about this, but again this is unlikely to make a great deal of difference. It's certainly not the main factor at play here: that's definitely the Jetson's GPU, which of course TensorFlow can certainly use (via CUDA and CUDNN, as does DeepStream). It's true that using TensorRT can provide a speed boost on a Jetson, but even that's possible with TensorFlow, although admittedly you have to remember to call it specifically but it's just three or four lines of (Python!) code. There are already so many ways it's unavoidable to tie yourself into NVidia's ecosystem, it seems like a bad idea to tie yourself in further in a totally avoidable way like this.

[Edit: I just realised that the image is being streamed to a remote computer that's doing the inference. The general point remains though. The totally different architecture (including having to transfer data over the network) and hardware are the actual reason for the performance difference, while C/C++ vs Python and DeepStream vs TensorFlow are tiny details.]

If something were to be more "neutral" what would you hope to see exactly? Something performant is typically going to be framework/hardware specific.

Sorry, I'm not sure what you mean by "neutral". Are you talking about my suggestion to avoid DeepStream? If so:

The frameworks that work on multiple types of hardware, like TensorFlow and (probably most popular now) PyTorch, have separate backends for their different targets. Each of these backends have huge amounts of platform-specific code, and in the case of the Nvidia backend, that code is written in terms of CUDA just as DeepStream is. That's how they achieve good performance even though the top-level API is hardware generic. The overwhelming majority of deep learning code, both the actual learning and the inference, is written in terms of these frameworks rather than NVidia's proprietary framework. Admittedly I haven't played with NVidia's library, but I highly doubt there's a serious performance difference - it's even possible that the open-source libraries are faster due to the greater community (/Google) effort to optimise them.

It does look like DeepStream does a lot more of the processing pipeline than just the inference. In that case it's going to be a lot more tricky to get the whole pipeline on the GPU using those TensorFlow or PyTorch. At the end of the day, if only DeepStream does what you need, I'm not saying you necessarily shouldn't use it - just that you should ideally attempt to avoid it if reasonably possible.

I think the difference with the jetson xavier is the tensor cores. The xavier is different from the pi (and even the jetson nano), like 100x different.

The Raspberry Pi doesn't have any "tensor cores" at all. According to Wikipedia, it actually does have a "Broadcom VideoCore IV" GPU, but I don't think this processor is ever used for deep learning. So if you did inference on the Pi then it would have to be on the CPU; inference is slower even on a meaty desktop CPU than on a GPU, never mind the low-powered CPU on the Pi.

That is all academic, as the whole point of the article is actually that the processing isn't done on the Pi but on the remote server. In that case the difference (if there even is one, I don't see a frame rate mentioned in the article) is indeed down to the difference in power of the respective GPUs, as you're alluding to, or to do with the fact that the article is having to stream the image frames over the network (it doesn't even seem to compress them) whereas the parent comment's idea just processes them locally.

You know what, you're right and I'm wrong.

I went back and looked carefullier, and I must have read "on the edge", then "testing it locally" then "integrating tensorflow" and thought they moved it. But it doesn't actually do it on the edge at all. I think I need to learn to read.

As I said in another comment here, I and lots of other commenters misread it that way too. I definitely find it funny they took "on the edge" to mean "anywhere on my local network", rather than just on the actual device capturing the data.

Reply on news.ycombinator.com