Comment by rcarr | Hacker News Reader

Comment by rcarr | Hacker News Reader

This is getting pretty close to how I think we get to the general purpose humanoid robot. This is how I see it playing out:

- You have your Boston Dynamics style humanoid robot at the job site, lets say it's a bricklayer for the purposes of this example.

- You have a human somewhere offsite in an open room with an omnidirectional treadmill floor, and cameras and depth sensors positioned all around the room. They're wearing a Hollywood style motion capture suit and have a VR headset on so they can see what the humanoid robot sees through their cameras.

- The human then acts as they would on site, walking up to the pile of bricks, picking them up, placing them etc. The robot moves in real time on the job site, mimicking whatever action the human performs. I don't know if you'll need props to do this properly or if the muscle memory from years on the job will be enough for the humans to get the motions right.

- You log all the data. You then have someone watch through the video stream, labelling each action that is being performed.

- You run it all through a machine learning algorithm, until you get to the point where you can just send the architectural plan to the robot and essentially say "Build this wall for me".

The first 3/4 points were all almost exactly from the 2008 movie Sleep Dealer to the point where I thought you were referencing that

Never heard of it before, any good?

It's worth watching because it is so plausible.

Commenting for myself later :D.

there's a favorites feature that might interest you.

And yet, as far as I know we don't even have mature implementations of this even for machines with much courser movement whose movement are controlled by a loose physical mapping to the human operator's movements. Things like excavators with dual joystick controls.

This is precisely how Comma and Tesla do self driving.

comma does raw input to output behaviour cloning like you propose. interestingly, tesla does not. via there last public ml update, theyre self driving architecture still involves a path generator and evaluator combo, and a lot of seperately designed sensor fusion steps. parking is done via ppo reinforcement learning i think.

In short term perhaps but not long-term.

I believe they will send out a team to digitize the job site.

The team will then create a digital twin.

The architect will map everything into this twin.

The computer system will simulate the build steps.

Robots will be brought onto the job site and get a finetuned model (if necessary) and will build it automatically.

No, long term we somehow get something magical like LLMs that swoop in and magically pull off these tasks.

None of that. We get LLM on the jobsite and it tells worker how it’s done.

Often nothing as cheap as humans in terms of energy consumption. I doubt LLMs will beat that energy consumption.

Humans are .. complicated. You cannot just compensate a human’s caloric expense (although many companies would love that).

Anywhere a human gets involved shit gets expensive. Interestingly robots are pretty human-intensive, from design to programming and maintenance. Only at scale does the “human factor” go down and things become affordable.

Also, humans are squishy, you need safety requirements to keep them intact (seeing as human repair is incredibly costly and takes ages to complete, often with no guarantee that repair completes and performance levels go back to nominal), humans need sleep and can't work around the clock, and they have a mind of their own, which can make them difficult to deal with.

Indeed. Most of them are also bad planners and lose track of priorities very, very easily. Besides that, there is non-stop internal strife and quarreling as they are deeply social creatures that need to be in constant competition with one and other in order to be happy. You need another class of humans to "manage" these issues.

Once those two layers interact, you get a whole new dimension of problems you get to deal with. The creation of yet another layer of "management" is inevitable. It's basically management all the way up.

But man, once you get a bunch them aligned and motivated the sky is the limit.

I once read a novel or short story with this concept. The robots were piloted (on-site) for training and then let loose on their own. I forget the plot besides that though or who may have wrote it.

Yes, it's called reinforcement learning.

Reply on news.ycombinator.com