Hacker News Reader

> The competition in this space is great but I can't help but wonder what would happen if instead all these companies pooled their resources and went after the goal collectively.

It would probably slow down. 9 women can't have a baby in 1 month. Besides that, the disagreements about approach, politics, or eventual competitive interests would probably bring things to a halt for a long time.

I don't think the solutions to this problem are resource-constrained. Many companies would happily find more resources in order to be first to market with this technology.

Also agreed, when hedge funds don't silo their quants, instead of seeing 50 different strategies from 50 quants, they get 50 variants of the same strategy, source:

https://www.amazon.com/gp/product/1119482089/ref=ppx_yo_dt_b...

Exactly. Look at Human Brain Project: $1B and 10 years later what exactly have they achieved? Just like you said - disagreements about approach, politics, or eventual competitive interests did bring things to a halt for a long time

The human brain project was DOA from day #1. Unrealistic goals, no clear reason why more money would lead to better results and no concrete deliverables that anybody needed.

Same can be said about level 5 autonomy!

To me, the fatal flaw of HBP (or USA's competing HBI) is that they are akin to "cargo cult science", the idea that we can replicate the superficial structures to a significant enough degree that they system they impart will suddenly somehow become activated.

But just like the Melanesians with their coconut-shell headsets, there won't be anyone listening on the other end...

If we replicate a car with a sufficient accuracy it will start. I don’t see why this wouldn’t apply to any other piece of machinery, including a brain.

HBP should have tried a simpler task first, e.g. replicate a fruit fly’s brain.

Having replicated the fruit fly's brain, what are you going to do?

Replicate the rest of the fruit fly so that you can install the brain in it? And then replicate the world so that your software-simulated fruit fly has a natural context in which to operate?

Or just stimulate the brain with random inputs not associated with any real-world stimulus?

I’m pretty sure simulating fly’s neural inputs would be a far easier task than simulating its brain. After verifying it works correctly we would proceed to simulating a more complex brain, say a frog. And so on.

To simulate the fly's neural input, you need to simulate the fly's entire environment, including fluid dynamics for the air around the fly, the physics of every object the fly interacts with, hormonal responses to changes in the flies blood concentration of various substances (O2, glucose, ...).

This already strains our technical capabilities (at least for the amount of money we are willing to spend on it).

For any animal whose behavior is the product of a lot of learning, and who deals with other such animals in daily life, you have to solve all the problems you had to solve for the fly and also deal with a much larger connectome and also deal with the fact that, until you can passably simulate mouse, you will never be able to simulate the way a mouse learns to behave in the presence of another mouse.

No need to simulate any of the environment. You only need to record the neural inputs to a fly’s brain. Sure that’s also challenging, but nowhere as challenging as simulating the entire fly’s brain. My point is if you manage to accomplish the latter, the former would be a breeze.

> You only need to record the neural inputs to a fly’s brain.

I don't think that's true. As soon as your simulated fly's behavior diverges from the actual fly's, all of the recorded input after that point is invalid/useless because it will not match the simulated fly's position/orientation/whatever.

Also, how many dollar of investment and years into the future do you think we are from being able to record all of a fly's neural input while it is moving freely?

To start with you only need to study input-response pairs (sensory input causing motor commands). You enter the neural inputs from a real fly into the simulation, and compare responses of the real fly and the simulated one. Once you understand what's going on, proceed to do sequence of inputs, and compare the sequence of responses. The goal is not to produce the identical sequence of actions by tuning the simulation, it's understanding how the actions are being computed from the inputs.

Having a detailed simulation like this would greatly accelerate Numenta's style research, where instead of piecing together information from published papers, you would get it straight from the experiments you control.

>> if instead all these companies pooled their resources and went after the goal collectively.

That would be a bad idea because like in evolutionary processes you need this diversity of ideas to locate better local optima even if it will take longer.

Standards shouldn't emerge too soon. I think for self driving tech, at the current stage, competition is good because there are lots of unsolved questions. Competition will ensure the best tech is ultimately available to consumers.

Of course, it's not a binary choice. Things like data should probably be pooled but the use of data in tech should compete.

Ditto validation tech frameworks. If those are not standardized then people will not be able to make an informed choice about which solution is the safest other than to wait a decade and do a bodycount.

I think that there is still a need for some brilliant insights and breakthroughs, it isn't just a matter of getting the work done.

So actually, I think it's one of these situations where having a lot of independent efforts might be worth it.

> There is so much duplication going on

In the self-driving world, the duplication is necessary - different companies are taking different directions, and nobody really knows which will work out.

In the ML hardware world, the duplication is mostly unnecessary. People are developing their own inference hardware ASIC's because they're relatively simple (compared to designing a CPU from scratch, designing a TPU is pretty simple because there are so few operations, and no complex out-of-order execution), and you can't buy one off the shelf yet.

As soon as ML hardware becomes available to buy off the shelf without a massive price premium, everyone will switch to that.

I do research in ML hw field: there are currently a couple hundred designs to run a convolutional NN inference. A couple of dozen have been built. They have pretty different underlying technologies (CMOS, floating gate, ReRAM/memristors, etc), different ideas (systolic arrays, analog crossbars, cache organization, lookup tables, data reuse, TDM, using spikes, etc), wildly different power (from microwatts to hundreds of Watts), size, speed, precision, flexibility, cost, ease of use/integration, etc. This is just convnet inference. Lots more is needed to do training in hw, again with multiple choices on how to do it.

So which one design you suggest we all use for all our ML needs?

The duplication I see in the commercial world in ML inference hardware is in designs similar to the TPU... So a big ~128x28 accumulating mat-mul array, with enough memory throughput to get one operand in and out fast, and enough cache to store the other operand (weights) and switch between which weights are used so the mat-mul array can very efficiently do larger matrix sizes.

Also lookup tables for a bunch of activation functions.

That basic design can efficiently implement nearly any neural net architecture as long as the layer sizes are at least 128x128 and fixed point is okay.

The other exotic designs you suggested are more academic research things, and not yet deployed at scale in anyone's datacenters.

We could see some of this play out in the China EV market in the coming years. State sponsored subsidies around infrastructure standardization. Combined with foreign investment and competition spurring innovation.

What I've seen personally is what can be loosely termed "emergent consensus". Historical competitors (and often it gets whittled down to two giants, such as Boeing and Airbus) will work in secret on research. But after years of experimentation arrive at very similar outcomes. An optimal answer that could only be arrived at through constant trial and error, product evolution and iteration.

Regarding Karpathy's PyTorch presentation I don't thing anything that wasn't already public was revealed. The FSD board with custom NPUs is a Work of Art. I like that there are dual redundant streams. And the scale of the dataset is already well know: 4096 HD-images per step!

If I had to speculate, the "Dojo Cluster" may be envisioned as an effort to share data and compute with industry partners as a cloud SaaS product and ancillary revenue stream. But that is pure speculation ;)

Inside Tesla’s Neural Processor In The FSD Chip

https://fuse.wikichip.org/news/2707/inside-teslas-neural-pro...

Fortunately, some companies do share a significant amount of what their cars have learned so far. Uber publishes a ton of papers about their self-driving research [0][1]. Waymo released an open autonomous driving dataset, and publishes papers as well [2][3].

Of course, papers and data aren't code. But I think a lot more is being shared than people realize.

[0] https://eng.uber.com/author/raquel-urtasun/ [1] https://eng.uber.com/research/?_sft_category=research-self-d... [2] https://waymo.com/open [3] https://arxiv.org/pdf/1812.03079.pdf

I don't know about sharing tech, but there should definitely be a shared evaluation benchmark, and some kind of oversight agency should be involved. The idea would be: if you want to be permitted to operate an AV on public roads, you need to demonstrate that your vehicle's vision system can detect pedestrians and obstacles with near-perfect accuracy on a large shared image database, most of which is NOT distributed to researchers.

Competition is good. It keeps you honest. The Manhattan project experienced competition, of the do-or-die type. It only was not internal, it was from the Nazis, Japan, and later the soviets.

Right now, the competition in the self-driving area is metaphorically as close to do-or-die as you can have in peace time. GM as a regular car manufacturer is toast, Cruze is pretty much their only hope. Uber is bleeding, if they pull self-driving off, they are kings. The German manufacturers are watching in disbelief as Tesla is starting to eat their pie. Conversely, Tesla knows that in what they are doing (electric cars) they don't enjoy any fundamental moat. If the Germans get their act together, they'll be able to make equally performant electric cars, but true luxury. The only one that's not really about survival is Waymo.

I wonder if the French civil nuclear electricity program that led to this level of low carbon emissions, or the TGV (high speed rail system), could be good examples of what you're asking for.

Maybe the companies actually building the nuclear plants and trains and rails were actually in competition ?

There was a time in the medival ages where alchemists were kidnapped by kings and held in chambers so they would only generate knowledge for them. This obviously lead to a similar duplication to the one you describe, right up to calculus where Newton kept the thing hidden in a drawer and then Leibnitz had the same idea.

Once that kind of secrecy was gone our whole technical progress was accelerated, because people could build on the discoveries of other people.

Right now we are going back to the alchemist model in some ways (the highest profile people work for the big companies and don’t share their discoveries). This makes progress slower.

> the highest profile people work for the big companies and don’t share their discoveries

I have to strongly disagree with this for the specific case of AI/ML. The big company labs are publishing open access papers non-stop, often with code and sometimes even datasets. They're more open than some areas of academia, in fact.

Really? I heard the opposite from someone in the field, who told me that they do publish but it is never the really relevant stuff. I can’t really judge that myself to be honest

I work in the field. New and relevant stuff is published frequently by Google, Facebook and Microsoft (as well as smaller companies like NVidia etc). Apple very occasionally publishes too.

Leon Gatys, the guy who published A Neural Algorithm for Artistic Style that revolutionized the entire industry while he was at a university is now employed at Apple. They have published at least one thing that was semi-interesting on cognitive functioning in older adults and how it can be expressed through smartphone usage: http://delivery.acm.org/10.1145/3310000/3300398/a168-gordon....

Interesting and certainly compelling, but not earth-shaking like style transfer has been.

Ideally everyone would collaborate on inputs and compete on outputs. All the data gathering, tagging, mapping etc could be put into a shared domain, and then after that the companies decide what to do with it and how to commercialise it.

Easier said than done, but I think it would strike the right balance between reducing duplicate work, and incentivising progress.

You can apply this line of reasoning on many markets, like the pharma or food industry which also have safety concerns. It strikes me as the kind initiatives EU attempts nowadays when realizing we are running behind on some tech and want to leverage the one possible advantage we have as a great centralizing power. Not too different from communist states, actually. I agree with the sentiment that redundancy seems wasteful, but it seems to me a necessary evil as a driving force in development, as with the right to private property in general.

I'm reminded of the Manhattan project and I don't think they would have succeeded in their goal if they had tried to run 10 of those at once. There just aren't that many really great scientists in a space this narrow.

They built complete factories for three different technologies, and finally delivered two totally different bomb designs.

Which is a sign they were pretty worried about choosing to put all their smart guys on one path, and picking the wrong one. And the number of smart guys they had was certainly a real constraint. Even under their very stressed circumstances, exploration and competition were important.

This is correct - read the fantastic “making of the atomic bomb” - the planners kept competition alive for as long as possible because no one really knew which path was going to work. In contrast the nazi project, among other reasons, failed because it focused nearly exclusively on heavy water and didn’t have internally competing projects.

It's kind of a different class of problem. With self driving you can have someone like Hotz or the Cruise guys knock up a system with limited resources and have it work quite well. You just can't do that with developing nuclear weapons.

Also self driving is not a problem where you can put a bunch of geniuses in a room and have them calculate the correct design. There are too many unknowns. It needs experimentation and trial and error.

Fair point. And the moon mission. Autonomous driving is such a consumerist issue though, seems quite well suited for a free market dynamic. I'd rather see a great joint effort on fusion energy or something.

Talk about twisting the argument. The Manhattan project was in competition with the Soviets and the Nazis, it was less a result of some unified front of all the smartest scientists in the world, and more of a result of "if we don't succeed first, someone much worse probably will".

Your logic of "if there were 10 of them simultaneously, it wouldn't have worked out" is flawed and self serving in that if you ignore all competition and just divide a single competitor into any number of smaller entities, of course at some point they'll be too small to be viable.

Competition is good

Choice is good, alternative implementations are good. But I think competition is bad. It is wasteful and antisocial.

Competition is good and directly leads to the progress we’ve seen in the western world.

"Progress" is a rhetorical device used to paint whatever actually happened in a positive light. But coincident with the positive elements of the rise of western society, is a whole lot of tragedy. Wars with unprecedented brutality and scale, genocide, factory farming, mass extinctions and the systematic destruction of the planet, selfishness as a defining cultural trait.

Replacing competition with cooperation may have accelerated the positive aspects of progress (e.g. science), and maybe slowed or prevented many of the negative ones.

Here's a treatise on the topic, if you're interested in this counterpoint: https://www.alfiekohn.org/contest/

I think the diversity you see in cameras and lidar placement and existence is worth it enough to have different paths forward. Tesla seems insistent that it can be done sans lidar. It's definitely worth it to see which approach works best.

>Imagine every car manufacturer having a completely different take on what a car should be like from a safety perspective. We have standards bodies for a reason

Roads are also governed by public bodies. Road signs are standardized and public.

I think the government should take a much larger role in defining self driving cars. For example, rather than using computer vision to recognize signs, signs could be active standardized beacons; instead of having to recognize lanes, they could be repainted with rfid chips that are trivial for cars to recognize and follow.

Avoiding driving into people is also something that was somewhat regulated by crosswalks with pedestrian lights. Would it be absurd for the crosswalk to know roughly how many people are at it, then broadcast this to the car, rather than having the car have to recognize them?

There are many things the government could do with transportation infrastructure that would benefit everyone, many of which are literally impossible for companies to do separately. Can you imagine if we had to wait until IBM (or Siemens or Google or Apple) got into the business of launching satellites before we got GPS? There is a good chance that to this day cell phones wouldn't know their location or give anyone any mapping applications.

To me, self driving cars are similar. Many parts of transportation are a public good.

> Would it be absurd for the crosswalk to know roughly how many people are at it, then broadcast this to the car

Yes, completely absurd. For one, many places are too sparse, poor, or unstandardized for this to be remotely economical. Two, people often don't cross at crosswalks (jaywalking). Level 5 AV is a thing where 99% coverage isn't good enough. You need a lot of nines.

That's why many think lidar systems are a crutch. If lidar can't work in snow or heavy rain (at least 1-2% of days in the north), then you need fallback which must still be >99% as effective to avoid incidents. But then why not just use the fallback?

Generating data? Sure. But for actual use in the data pipeline? Makes you rely on an ultimately untenable solution.

In sparsely populated areas smart roads could inform the car that there hasn't been any movement whatsoever in the whole area for hours. (Along the entire road and adjacent to the roads). How the car responds (driving somewhat faster, perhaps) could mean a large measure of safety as compared to when there are a large group of people about to cross a rural road at night that the car may or may not see visually.

Anyway it's just one example. RFID or similar embedded beacons in the road paint would make roads much easier for cars to follow. They are expensive for any one company to do but cheap for the government to do if it is done everywhere at once and lasts 5-10 years. Road signs that broadcast what they are (instead of needing to be "seen" and interpreted visually instead of through radio by the car), are similar.

Finally, a government standard could coordinate cars into a caravan, avoiding pileups for example, and giving the participating cars several advantages you can find by Googling "car caravan". (Though this might be achieved by industry.)