Intel Extension for TensorFlow | Hacker News Reader

Intel Extension for TensorFlow

github.com

136 pointshochmartinez34 comments

Intel did a great thing for people interested in ML and numeric research by making their MKL library and compiler free and cross platform compatible. Even today on my AMD zen3 Ryzen machine intel's MKL linked numpy and pytorch are in some operations 10* (yes that is really ten times) faster in comparison with the next best alternative (openBlas etc). I was shocked to discover how much of a difference MKL makes for cpu workloads. This is mostly because it makes use of AVX2 cpu extensions which make certain matrix operations a lot faster.

They crippled the performance on non-Intel CPUs on purpose until recently.

Intel's anti competitive behaviour follows them throughout their history.

Yes, this is true, but even back then (during the time they crippled it) it was possible to pretend to run on Intel hardware. I haven't done this, but I read it was possible.

I just felt your first sentence, without a qualifier, gave too much credit looking at the context.

They crippled their own consumer CPUs by retroactively disabling AVX512

That was my previous experience, but have you tried linking to AMD AOCL recently? I would not expect the performance gap between Intel MKL and AMD AOCL to still be as large as you describe on a Zen 3.

Would you know if AOCL is supported by numpy? And are there any benchmarks out there? (Especially interested in Zen4)

Since BLAS (and LAPACK) have an essentially well-defined ABI, anything dynamically ELF-linked doesn't need specific support. You can just LD_PRELOAD, say, OpenBLAS or blis+libflame (as in AOCL) to use them. You can also make a trivial libblas.so.3 shim to put on LD_LIBRARY_PATH. (Debian has a heavyweight approach to linear algebra alternatives, with separate builds of the whole library.) See, e.g., https://fx.srht.site/posts/2019-10-20-library-aliases.html

Compiling eigen uses avx512 just fine. I don't think it is quite the discriminator it once was. However, I amp happy they made it open and it would be great of oneAPI became a real alternative to cuda for more tasks. I tried getting it to work with flux.jl a while back and ran into some difficulty.

That's simply implausible for any proper measurement of the large-dimension level-3 operations that normally matter (and probably others I haven't measured). OpenBLAS certainly uses AVX2 assembler on Zen if it's built and run correctly, but you can get >60% of MKL's AVX2 performance with a plain C micro-kernel. The only plausible reason for an order of magnitude difference on large GEMM would be multi-threaded versus serial.

I keep saying there's far too much mythology around MKL, disproved experimentally, and if you're using Zen I don't know why you wouldn't use AMD's version of BLIS. It doesn't even make sense to talk about MKL on Zen without the version, since the story keeps changing.

OpenBLAS and BLIS actually are free software and cross-platform. MKL is still proprietary, and certainly doesn't run on the POWER platform I support. Also note that it only got the small matrix performance relevant to tensorflow after libxsmm showed the way.

For context, in other Python ML packages like scikit-learn running the Intel extension on a compatable CPU can result in a massive performance increase.

Although since TensorFlow models should be trained on a GPU unlike sklearn, that's less useful, and there are better tools for CPU inference. (e.g. SavedModels or ONNX)

yes, but Intel now has mainstream GPUs that are reasonably (~ RTX 3060) performant, which is what I assume this is for

Are you talking about Intel Arc? I'm yet to see any ML relevant benchmarks on Intel Arc. If you are aware of any please let me know.

Same here. At 16GB of VRAM and only $349, it could fill a really nice slot for DL.

You won't for some time as ARC is only supported on Linux kernel 6+. And honestly, their hardware is pretty bad for ML given you can pick up a used 3090 for $600. I just don't think there will be an ML push for ARC with current gen.

> And honestly, their hardware is pretty bad for ML given you can pick up a used 3090 for $600.

How do you know it's bad? Do you have benchmarks?

Just checked, used 3090s are going for $700-900 on eBay.

hardwareswap.reddit.com is the best way to buy used computer hardware.

It won’t be able to make use of CUDA, which is more than enough reason.

I think a recent Linux version requirement shouldn't hamper benchmarking if you're a somewhat technical user.

You can get the latest Linux pacakges for eg Ubuntu here: https://wiki.ubuntu.com/Kernel/MainlineBuilds - probably other distributions have similar easy ways since it's needed so often in bug reporting for users to tell if a hw support bug is in a distro specific kernel change or not.

It presumably has to support Aurora urgently: https://www.alcf.anl.gov/aurora

I don’t know much about Intel’s toolchain: people say good stuff about TBB and things like that.

Historically doing inference on Intel gear was mostly about whether or not to target AVX2 or AVX512 when building Eigen or whatever. A few years ago the net win was AVX2 because the de-clock and re-clock just killed you.

What’s the game these days? Long term I doubt inference will be done on x86, but I think a lot of people still do it.

For the curious, Apple has an analogous Tensorflow Metal plugin to allow for Apple Silicon (and AMD GPU) acceleration using the same plugin architecture.

https://developer.apple.com/metal/tensorflow-plugin/

Which is basically unusably buggy.

For example, tf.sort only sorts up to 16 values and overwrites the rest with -0. Apparently not fixed for over one year: https://developer.apple.com/forums/thread/689299

Also, tf.random always returns the same random numbers: https://developer.apple.com/forums/thread/69705

Although I guess these bugs are not the fault of Tensorflow's plugin architecture but rather Apple's implementation.

I believe the second link should be https://developer.apple.com/forums/thread/697057

It is sorta the fault of tensorflow... Tensorflow should have had a test suite for plugins that verifies that all operations at least lead to correct results no matter which backend is in use.

> Also, tf.random always returns the same random numbers:

reminds of this particular xkcd [1], How someone fucks up that bad is beyond me.

1. https://xkcd.com/221/

I love the hardware Nvidia makes, but for the good of our field and for progress, we need our ML stacks to work with all other vendors. CUDA is the wrong platform to build our future on.

https://github.com/intel/intel-extension-for-pytorch PyTorch version. Looks like it's been supported a lot longer than TensorFlow, the repo dates back over a year.

Sorry for my ignorance, why does TensorFlow in this repo readme have an asterisk? (It writes as TensorFlow*).

I too am wondering that. Doesn’t seem to be any corresponding asterisk explanation.

I assume they meant that say "* Tensorflow is a trademark owned by a third party and has no affiliation with Intel"

Does this work with jax jit?

Disappointed by the lack of benchmark comparison in the README

Do Nvidia and AMD have something like this?

Reply on news.ycombinator.com