37ef_ced3

Linus Torvalds:

  And I claim that that is the real problem with AVX-512 (and pretty much any vectorization). I personally cannot find a single benchmark that does anything I would ever do - not even remotely close. So if you aren't into some chess engine, if you aren't into parsing (but not using) JSON, if you aren't into software raytracing (as opposed to raytracing in games, which is clearly starting to take off thanks to GPU support), what else is there?

Answer? Neural net inference, e.g., https://NN-512.com

If you need a little bit of inference (say, 20 ReNet50s per second per CPU core) as part of a larger system, there's nothing cheaper. If you're doing a small amount of inference, perhaps limited by other parts of the system, you can't keep a GPU fed and the GPU is a huge waste of money.

AVX-512, with its masked operations and dual-input permutations, is an expressive and powerful SIMD instruction set. It's a pleasure to write code for, but we need good hardware support (which is literally years overdue).

There are completely stand-alone systems like:

https://NN-512.com

The stand-alone code generator (a statically linked executable written in Go with no dependencies outside the Go standard library) generates stand-alone POSIX C code for the neural net, requiring only gcc to compile.

Also see Fabrice Bellard's LibNC:

https://bellard.org/libnc/

C API. Small library, no external dependencies, available for Linux and Windows.

And https://NN-512.com

You're in for a big surprise. Try using the language.

Spend some time using Go, and you will be impressed by its performance.

You'll wonder, "Were all those haters on Hacker News misinformed?"

Convolutions are fused into convolutions, elementwise operations are fused into convolutions, everything is inlined except where function calls are needed for pthread work units (and those work units are all custom/arbitrary).

So...

...you were unable to decipher this Hacker News comment thread...

...unable find some C code and build it with GCC and make an executable for yourself...

...but you think you can reverse engineer the executable?

Here is the C code for a DenseNet-121 generated by NN-512:

https://nn-512.com/browse/DenseNet121

Even if you had the C code available to you, you would have a hard time producing the input graph.

Good luck reverse engineering it after GCC has compiled it!

NN-512 has an incredibly flexible code generator. It can easily be tweaked to produce completely different code for the same convolution, so everyone can apply their own twist to defeat the reverse engineers ("the intellectual property thieves").

This will not be able to reverse engineer fully-customized, fully-fused neural networks generated by NN-512:

https://NN-512.com

NN-512 generates custom code for all the operations, custom units of work for the threads, custom code around tensor edges, everything is fused and unrolled and customized. If they can deduce the network graph specification from the AVX-512 code, I will be astonished.

If you can do it, show me. But I know you can't.

Anyone who cares about model privacy will use their own variant of a tool like NN-512. It's security through obscurity, but that's the best you can hope for if you are distributing an executable.

Are you seriously suggesting that the language should not have notation for allocating zeroed primitive types and receiving the address of the allocation?

  var (
      p1 = new(int)
      p2 = new(*int)
      p3 = new(**int)
      p4 = new(complex64)
      p5 = new(complex128)
  )

I feel like you must be joking. Should we say

  var (
      c complex128
      p = &c
  )

every time?

You can think of Go as the modern representative the Plan 9 and Oberon schools of thought -- their answer to (and replacement for) C and C++.

The fact that it was developed at Google is incidental. That's where Robert Griesemer (Oberon) and Rob Pike (Plan 9) and Ken Thompson (Unix, Plan 9) and Russ Cox (Plan 9) and Ian Lance Taylor, etc., happened to be employed.

It's used like this

  p := new(int)

since you can't say

  p := &int{}

For small-scale transformer CPU inference you can use, e.g., Fabrice Bellard's https://bellard.org/libnc/

Similarly, for small-scale convolutional CPU inference, where you only need to do maybe 20 ResNet-50 (batch size 1) per second per CPU (cloud CPUs cost $0.015 per hour) you can use inference engines designed for this purpose, e.g., https://NN-512.com

You can expect about 2x the performance of TensorFlow or PyTorch.

Go. The obvious right choice.

But Meta is in the advertising business and Google is a competitor.

As an engineer who uses Go and appreciates it as a replacement for most purposes where you might use C or C++, the complaints just seem bizarre.

You've got people who argue against explicit error checking and people who don't understand the important role of nil pointers and people who don't see the massive net win of garbage collection in concurrent programs, and so on.

Endlessly.

It feels like a waste of time defending the language when the people criticizing it seem to hold such a vastly different point of view. It's like trying to convince people that Natural Born Killers is a good movie or that Primus makes good music. There's an unbridgeable chasm between you and the people you're trying to convince.

From a customer's perspective, it isn't clear that a hardware/software taxi driver is in any way better than a human taxi driver, particularly with (human) driver assist preventing collisions and all the other warnings provided by a modern car.

In both cases someone else is driving for the customer. With a human driver (plus driver assist braking and collision warnings) you have the most flexible, sophisticated intelligence on Earth driving. With a robotaxi you have something inferior. But maybe it's a lot cheaper, right?

The robotaxi can only compete on price because that's its only advantage. If you own a car that drives itself, that's a different story. Everybody can see the value proposition.

But is the robotaxi actually cheaper at all? We would have to look at the cost of the hardware (how often do lidars fail and how much do they cost to replace?) and the cost of the software development and the cost of the fallback human remote operators (fleet monitoring and teleoperation) and the years of huge R&D investment (billions of dollars) to evaluate whether a robotaxi fleet is indeed cheaper. So how much cheaper is it, exactly? 5%? 10%? 15%?

As a customer, would you pay a little more to have the most flexible, sophisticated intelligence on Earth (human brain + driver assist) or would you want to save a few dollars and risk having some dumb piece of software strand you in the middle of the road somewhere?

We all use Google Maps or Apple Maps when driving and most of us have seen these systems do boneheaded things. Just imagine the dumb things a robotaxi could do. It's hard for a normal person to be excited about this. I don't know a single person who is excited by robotaxis.

Yes, but did he buy her a horse?

From a customer's perspective, it isn't clear that a software taxi driver is in any way better than a human taxi driver, particularly with (human) driver assist preventing collisions and all the other warnings provided by a modern car.

In both cases someone else is driving for the customer. With a human driver (plus driver assist braking and collision warnings) you have the most flexible, sophisticated intelligence on Earth driving. With a robotaxi you have something inferior. But maybe it's a lot cheaper, right?

The robotaxi can only compete on price because that's its only advantage.

If you own a car that drives itself, that's a different story. Everybody can see the value proposition.

We would have to look at the cost of the hardware and maintenance and fallback remote operators and the R&D investment to evaluate whether a robotaxi fleet is indeed cheaper. How much cheaper is it, exactly? 5%? 10%? 15%?

Would you pay a little more to have the most flexible, sophisticated intelligence on Earth (human brain + driver assist) or would you want to save a few dollars and risk having some dumb piece of software strand you in the middle of the road somewhere?

We all use Google Maps or Apple Maps when driving and most of us have seen these systems do boneheaded things. Just imagine the dumb things a robotaxi could do.

It's hard for a normal person to be excited about this. I don't know a single person who is excited by robotaxis.

Colin Percival's Tarsnap: https://www.tarsnap.com/

Colin is "Security Officer Emeritus, FreeBSD".

Animats, I was a fan of your Downside site: http://downside.com/

Downside's goal is to remind people that we're experiencing a financial bubble, and that financial bubbles always burst. (We originally wrote that in April, 2000. Any questions?)

And here we are, living through another huge bubble.

In other words, you believe the cost of the human is subtracted but there is no additional hardware/maintenance/remote-operation/R&D-investment cost to add?

Anyway, I think we can all agree that the only benefit of a robotaxi is that it's cheaper. And we don't know how much cheaper. So it's not surprising that most people aren't excited about them.

I don't know a single person who cares about robotaxis. In my experience, if you talk to people about robotaxis they just don't see it as big step forward. "What's the point?"

Well, we'd have to look at the cost of the hardware and maintenance and fallback remote operators and the R&D investment to evaluate whether a robotaxi fleet is indeed cheaper. And how much cheaper is it, exactly? 5%? 10%? 15%?

Would you pay a little more to have the most flexible, sophisticated intelligence on Earth (human brain + driver assist) or would you want to save a few dollars and risk having some dumb piece of software strand you in the middle of the road somewhere?

We all use Google Maps or Apple Maps when driving and most of us have seen these systems do boneheaded things. Just imagine the dumb things a robotaxi could do.

So robotaxis are perhaps mainly attractive to the very low income parts of population, people who might buy an inferior product to save a dollar or two. People who buy store-brand ketchup instead of Heinz ketchup even though it doesn't taste as good.

It's hard for a normal person to be excited about this.

From a customer's perspective, it isn't clear that a software taxi driver is in any way better than a human taxi driver, particularly with human driver assist preventing collisions and all the other warnings provided by a modern car.

In both cases someone else is driving for the customer. With a human driver (plus driver assist breaking and collision warnings) you have the most flexible, sophisticated intelligence on Earth driving. With a robotaxi you have something inferior. But maybe it's a lot cheaper, right? The robotaxi can only compete on price because that's apparently the only advantage.

The idea of a robotaxi seems popular among people who don't like being around other people... a population perhaps over-represented here on Hacker News. Personally I don't see the point and I know I'm not alone in wondering.

If you own a car (e.g., a Tesla) that drives itself, that's a different story. Everybody can see the value proposition.

This code is really nice. How will you profit?

Let's be real.

Not long from now all creative and productive work will be done by machines.

Humans will be consumers. Why learn a skill when it can all be automated?

This will eliminate what little meaning remains in our modern lives.

Then what? I don't know, who cares?

TYLENOL® Extra Strength Caplets

Acetaminophen

500 mg (in each caplet)

2 caplets every 6 hours

1,000 milligrams is not "pushing the daily limit". That is a normal adult dose and by a wide margin within what is considered safe for even chronic daily use.

Like many people, when I have a headache or a sore muscle I take the dose recommended on the Tylenol bottle: 1,000 milligrams. It is not even remotely dangerous.

Furthermore, if the original commenter is using acetaminophen to reduce rare psychological pain the margin of safety is even larger because she's using it only rarely.

Commenter was explicit about dosage and pretty clear about frequency. A single dose of 1,000 milligrams in a 24 hour period is indeed small and universally considered to be safe for people with a normal liver.

Antihistamines like Benadryl are anticholinergics. They are harmful to the brain and everyone should avoid them. In contrast acetaminophen is universally considered to be safe for people with a normal liver.

AVX-512 is an elegant, powerful, flexible set of masked vector instructions that is useful for many purposes. For example, low-cost neural net inference (https://NN-512.com). To suggest that Intel and AMD should instead make "existing REPB prefixes a lot faster" is missing the big picture. The masked compression instructions (one of which is used in Lemire's article) are endlessly useful, not just for stripping spaces out of a string!