Skeletal Tracking Beta | Hacker News Reader

Skeletal Tracking Beta

developer.leapmotion.com

147 pointsMatetricks40 comments

I hacked the Leap controller at the SF Music Hack day. My team and I built an app called leapJ: multiplayer DJ'ing with your body. (https://www.hackerleague.org/hackathons/music-hack-day-san-f...)

Leap is a great little device. My main complaint is that the API is too low-level. We would just fire events roughly every 20ms, in a markovian (stateless) way. For this reason, the instrument control was somewhat janky, because we didn't have the time to de-jitter hand paths / infer a tracking spline.

My wishlist for leap is:

* Higher level APIs, that apply the sort of sophisticated filtering and tracking that many people are liable to roll themselves.

* Multiple leap devices per computer. Needing a separate computer per leap is a big barrier to working with several leaps at once.

* ARM support. I'd love to have a RaspberryPI for each leap. Then they would be truly mobile. For right now, I think I'm going to have to get a mini ITX motherboard,

You need a smoothing library. I welcome our low level access, you can always pile what you need on top, the converse is not possible.

What would be nice is some intelligence pushed down into the Leap, like the ability to run Lua or JVM code directly so that realtime multidevice is possible.

I think he meant augmenting your API with higher-level stuff, but not removing any of the low level access. :-)

This is amazing! I bought the LeapMotion a while ago and forgot it soon enough because the hand, finger and gesture recognition was very basic and didn't feel natural (at least to me and my friends who tried it). Now they push the boundaries of that small, old piece of hardware with a software update and show what it's really capable of. I can only imagine how impressive this must feel when you use it with an oculus rift and interact with a virtual reality environment. The sticking-blocks-together example is just the beginning of what's possible.

EDIT: Okay, I tried some of the examples and am excited! The recognition part is excellent, I can throw things up in the air and catch them again, all very natural. What's missing though are good physics. Objects glitch around my hands, wiggle mid air, get stretched even when they should be solid. But I guess that's Unity's problem, not LeapMotion's.

Yeah it's basically a holodeck. Exciting stuff. It's funny how on the shows the holo-deck was always something that took a whole lot of electricity so they could only have one or two on the ship, and you'd have to book time on it. It looks like in real life holodecks are much less power hungry than faster than light starship engines, and that more people desire their own holo-deck than want to go to space. Makes sense. Entertainment has historically been much more popular than exploration.

A holodeck is just another means of exploration. Spatial exploration (be it space, underwater, etc) is not really that interesting. If we went to Mars right now, we'd be really, really, really bored. There's nothing there except rocks.

Using the Oculus Rift (and associated VR technologies like the Leap), I can experience all sorts of amazing things that I've never experienced before. Like fighting aliens for the survival of humanity. Or jumping through floating sky scrapers. And I can create new experiences for others.

Exploration has never been more popular. It's just moved from the spatial to the conceptual.

Honest question? Why is the first example use case always killing? I'd probably be a lot more into video games if there was anything beyond Mario maturity games that wasn't a murder fantasy.

I don't want to fight aliens. I'd love to interact with real aliens though. That's an exploration that you could never have virtually.

I wish it wasn't, but that's just what's already there. Two reasons why:

1. Killing is a very easy interaction to implement. It's much simpler to make an AI that tries to kill you than one that tries to engage you in meaningful conversation.

2. The intensity of violence compensates for the disassociation that resulted from the interface. In other words, you were just watching the game on the monitor. You didn't real feel "there" like you do with VR. What I find, is when I'm playing violent games with VR, it's less enjoyable and more stressful. Fighting becomes an unpleasant necessity.

I think as the technology progresses, we'll see more games focus much less on violence. Like that Voyager bridge demo for the Oculus Rift. If someone made the whole ship with the same level of detail, and had no people or enemies or violence, I could still spend many hours just walking around and exploring, and I would probably enjoy it more than shooting things.

Portal, paper mario, braid, flight sims, sim city, tropico, civilization(? you do kill people in that), kerbal space program, etc.

I'm not by any means going to deny the flood of games that are based on violence. Most of the mentioned games were off the top of my head, but when I went looking for more, I was definitely a little surprised by just how tough it was to find well known contenders.

I can see how a casual brush with games might not bring a lot of the good ones to the front, though. I like games that really change how I think about a problem (portal, braid, paper mario), let me experience things I can't pragmatically experience right now (flight sims), or complex strategic games with some satire (civ/tropico).

I'm interested in games because I think they can be a really valid art form and a powerful way to convey concepts and ideas.

I'm incredibly excited for how far the oculus rift and new motion tracking techs will let me take that.

I remember the first time I played Assassins Creed. I mean, the game is about murder. It's in the title. My brother got frustrated at me after an hour of watching and yelled "STOP CLIMBING SHIT AND EXPLORING. KILL SOMEONE." I think he left after I spent another fifteen minutes exploring the ancient middle east.

I think more about, say, sending a probe places, having it scan the area with minute detail, and then broadcasting it back so we can walk through it virtually. We've talked about theoretical wormholes existing at an atomic level. If you could transmit data back and forth through a wormhole to an interstellar probe with insanely high resolution laser scanning, then recreate that world virtually, how far off are we from actually visiting the place in question? Really does make you wonder how important our senses are in the grand scheme of reality.

If we can transmit information through space instantaneously, our physical location becomes almost meaningless.

Who cares where your brain is if you can send and receive data somewhere 2000 lightyears in the same time it would take to send and receive data from somewhere 40cm away?

Sadly, even the theory behind such a concept is still in its infancy - and of course it may eventually be proven to be absolutely impossible with the same great certainty with which we've defined the mass of an electron.

I can also call/Facetime a friend that lives in a different part of town/country/world but that doesn't replace a physical presence.

If you disagree, try to convince my mother why I shouldn't have to make at least once-every-couple-month visits.

>small, old piece of hardware with a software update and show what it's really capable o

It was always about the software. Hardware is just 2 parallax USB 3.0 cameras in one small box. There is ZERO hardware processing.

I wrote a case about the Leap for my MIT Entrepreneurship class and interviewed a bunch of the founding team.

These guys are great -- but they still havent found their killer app. Thats partly because the tech is ahead of its time -- but the vision is clearly excellent.

David Holz the founder is the brains behind this, has been working on a better UI since he was literally a kid in Florida. See: http://www.businessweek.com/articles/2012-05-24/david-holzs-...

Their CEO is the founder of Quattro Wireless founder, an ex-Apple exec Andy Miller who is a really smart guy. SO they do really have a good team, and lots of VC backing.

Perhaps this new skeletal tracking stuff will help this really catch on. Best of luck to LEAP!

It seems like the type of company that needs to be acquired for its technology, by someone with not just the vision for the technology, but also the funds, and perhaps the platform for it, to really take advantage of it.

It looks like it is time to literally dust off my leap motion.

I'm wondering where I'll find mine...

I just updated my LeapMotion library for Processing [0] with support for the new LeapMotion SDK. Mine provides a thin wrapper over the SDK; there are several other libraries for Processing with higher level APIs (e.g. additional gestures).

[0] https://github.com/heuermh/leap-motion-processing

The new version is really a step up. Props to the dev team for being responsive as well. I sent them an issue during the beta, and they had pushed a fix a few days later. There's still glitches here and there, but those usually result from one hand being on top of the other or something similar.

I'm excited to see where they go with this. I'm not sold on using the leap for my everyday computing tasks, but I think there are some great applications for it.

People who have a Leap: does this work as well, in terms of tracking the hands, as it appears to in the demo?

If so, I may have to buy a Leap...

Yup. It works just like that. Now, keep in mind that it's not going to work perfectly ALL the time. If you get into complex occlusion cases it may not be able to reconstruct the skeleton perfectly.

I've never had mine work anywhere near as well as the demo implies, even in well lit environments and placed in a pre-tested spot.

edit: That said, maybe the new API is ridiculously better, but I doubt it.

the new software really is orders of magnitude better.

Answering my own question...

OK, the darn thing's only £70, so I'm going to buy one and test it out. Blog report coming.

("Only" £70? Yep. Currently, the best hand-tracking solution I'm aware of, from Animazoo, costs £13,000. £70 is a pretty good deal even if it has some limitations.)

It's not as good as the video implies... still quite easy to get it to glitch out. Notice in the demo video that they flip their hands pretty quickly. if you rotate them slowly things get a bit funky. I guess it's pretty hard to compensate for occluded fingers.

It does work just like the video although it is not perfect. They said on Reddit that they are going to keep pushing software updates to improve the tracking.

Big props to Leap Motion on the skeletal tracking. It was very easy to use. Took me around an hours to dust off my 3d trig and build this: www.youtube.com/watch?v=FX7pQlB6-IY&feature=youtu.be

So easy to use...

How is this technology so much more accurate than Microsoft's Kinect? That's awesome! I love seeing the new guy develop a better product than a old giant.

Very, very different problem domains and technology.

Consider: how much variation is there between bodies and hands? How much larger is the work envelope of the Kinect vs the Leap? How much more information do you get from the Kinect vs the Leap (depth buffer, positional audio, etc.)?

Wow. Could this be used as a form of biofeedback when trying to learn (or improve) complicated movements in sports?

It's hands-only, so there's a limited range of movements it can track.

However, if you apply the same question to a Kinect 2 or, say, another full-body mocap suit, then the answer is definitely.

One of these days when I have free time (no time soon, then) I shall write a routine for my mocap suits to track and give realtime feedback on various martial arts techniques.

> to track and give realtime feedback on various martial arts techniques

Hehe, that was exactly the application I was thinking of :)

Wondering if this would be useful for 3D level editing/navigating - or simply for our model browser... hmm...

This looks like it would be fantastic for teaching sign language...

I wonder if there's a PHP library?

You could use JS to make AJAX calls to PHP, since it has a JS library.

I think you could get PHP to connect using Websockets.

don't your deltoids get tired?

If I am sitting, my hands get tired very soon. However, if I stand up it works very well for me. I guess you get used to it anyway.

Meh.

Reply on news.ycombinator.com