Software and Hardware for General Robots

In my robotics experience I've come to the conclusion that some problems just need to get back to the physics, and trying to put labels on things like "grasp" is fraught with problems that constrain things in unhelpful ways.

It's more useful to think in terms of what forces are needed to be applied to the environment to accomplish a task. Then you can build a solver to develop innovative physics based solutions that are unconstrained by the semantics.

A thoughtful article. It is becoming increasingly obvious that the integration of field/home robotics into our daily lives is going to require us to adapt to the technology, more so than it adapts to us. For example, a (new) house constructed with a robotic-assistant in mind might alleviate some of the issues mentioned in the text; a gantry robotic arm, for example, with full access to the entire house. Washers/dryers designed to be integrated with robotic manipulators; things of this nature.

The example, of opening a package of dates, is actually pretty complex task even for humans. A child or even an adult with weaker fingers won't be able to do so.

There are plenty of kitchen tasks which are way easier for robot, for exampling making a bowl of cereal. Which brings me to the second point. You say:

> The only viable hardware for a robot meant to do any task in human spaces is an adult-sized humanoid, with two-arms, two-legs, and five fingers on each hand.

but I think this is only true if robot is constrained to using tools designed for humans. In the strawberry example, the knife is designed for multiple fingers, and cutting board has no elevated border to work well with the human knifes.

I think a properly designed "kitchen arm" (with maybe compliant/under-actuated fingers, a few sharp blades, a vacuum grip, etc..) as well as robot utensils (like uneven cutting boards that the food does not roll off from) would allow robots to do a large fraction of the kitchen tasks.

And maybe your bag of dates would be cut open instead of opened nicely -- but you'd still get to eat them.

I believe robotics hardware is near and damn it there, and has been for a while now. We're already seeing semi-reliable robots such as 'smart' hoovers enter homes.

The biggest problem by far is the software - in particular the AI. The biggest companies in the world have thrown billions of dollars at AI, Universities have had some of the most brilliant minds among their ranks, and we essentially got some (impressive) slightly better search algorithms.

There are absolute fundamental questions (+) that need to tackled in order to have the kind of generalized AI such environments require. Most AI (that I'm aware of) currently lacks the ability to do anything other than optimize itself for strictly specified scenarios/environments.

Personally I quite like the information theoretic approaches to self-motivated agents, there are some nice mechanisms out there such as empowerment [1]. It's not the full picture, but it's a step in the right direction. I don't think this is something we can throw larger neural networks and computation resources at and hope it solves itself.

(+) This is the subject of a paper I am currently writing.

[1] https://arxiv.org/abs/1310.1863

I'm not a roboticist, but can the ability to do general purpose manipulation be built up from a universe of known simpler manipulation tasks using something a bit like transfer learning? Is this used? Are there methods that don't need this?

Also, what would a good interface between a Software 1.0 program and a Software 2.0 program look like in robot software? I mean, what would the boundary between (3) and (4), and (4) and (5) look like in this imaginary stack?:

  (5) Autonomous controller (software 2.0)
  (4) A high level interface for giving instructions to (3), and finding out what (5) is doing
  (3) Motor manipulation controller (software 2.0)
  (2) A daemon for converting NN outputs into safe hardware control outputs (software 1.0)
  (1) OS kernel (software 1.0)

I’d love to work on the AI side of this. Are there any good arms I can buy for not too expensive and start trying out ideas?

Could I really have a pick_up_object() start up? There’s actually demand for that?

Robotics founder here. IMHO the problem with notions about general robotics is that it doesn't exist: robots are very specific if they are any good, at least if you measure success as cost vs. effectiveness. This is unlikely to change greatly due to physics demanding specific and greatly disparate automation solutions for many tasks.

Great article. What do you think of this paper? https://arxiv.org/abs/1909.10893

This type of research makes me optimistic about possibilities for generalization in the future.

Manual mode with onbody interface solves a lot of transitional problems.

See https://www.xprize.org/prizes/avatar

I actually registered a company named "General Biomimetics" in Delaware and have so much ambition and so many (unfortunately mostly vague) ideas about this. Specifically I have been thinking about washing dishes and other tasks in the kitchen. So its been something I have been spending at least two days a week on (I have a job).

But due to the depth of the problem and not having resources or much knowledge, it was maybe a little silly to create the company. But I like the idea of reserving that name, just in case I ever get anywhere.

From the hardware side, I feel like some of the robotics issues can be resolved by "just" copying people more closely. For example, it seems like the way real arms and muscles work should provide more leverage and force than the typical servo setup. And having five fingers provides the potential that manipulations could be copied from people.

There is also a very promising new type of artificial muscles called HASEL.

Of course, in order to efficiently build these human-like limbs, "all" we need is a way to 3D print with several materials at once, including a new type of conductive ink that can handle high voltages for the HASEL muscles.

But the starting point to me is a robot that can actually understand what it's looking at. In that it sees with depth, and understands the composition of objects and their orientation, etc.

Capsule networks seem interesting but also maybe are a bit computationally expensive and unproven? Also he seems to be focused on just the transformation matrix, but it seems like there are more aspects of the state that could be relevant and maybe are unique to different object types. But I am slowly trying to understand capsules anyway.

I have seen a few ideas about more general neural network-based systems that suggest it is necessarily to have multiple neural networks, or networks of networks, or neural modules, etc.

To me it seems like the ideal thing would be to have some standard shapes for networks or modules and also be able to reuse and adapt them for different tasks.

So my vague ideas now are something like: standard-shaped modules, trained on core modeling tasks such as finding 3d surfaces in 2d images. But at the same time somehow segmenting into different objects. And the potential high-level objects should be able to feed into the potential low-level understanding and visa-versa.

My intuition is that ideally there is a sort of 3d wireframe overlayed on the 2d image, identifying each object and sub-object with its exact dimensions, shape and orientation. Kind of like I've seen in one or two science fiction movies. So if I can somehow generate all of that, I know I have properly decoded the image.

Today I was looking at a GAN tutorial. But I have never made a CNN before, so decided that must be first.

Usually I think about this stuff for awhile and then just decide I don't really know what to do and then go back to Coursera. I finish Ng's first class and am looking at the hyperparameters one. I feel like I need to make some actual neural networks on my own though, because mainly Ng is teaching me how to convert from math notation to vectorized Python as much as anything.

If nothing else, this is really motivating me to learn about existing AI techniques. Which I feel like, to be a good programmer, I actually should be able to use things like Tensorflow etc. for narrow AI tasks.