Microsoft demos 'holoportation' 3D presence tech with HoloLens | Hacker News Reader

Microsoft demos 'holoportation' 3D presence tech with HoloLens

research.microsoft.com

423 pointsImpossible109 comments

I was waiting for this, knew it was only a matter of time before Microsoft came out with something like this since it's right in their field of office products.

Hopefully this is the catalyst needed to create better work/life balance and lower the barrier for remote work and make it just as productive as being present in the office.

For remote work, what are the advantages of this over a good video conferencing/screen sharing type of setup.

It's really hard to say exactly what it is, just as I can't tell you exactly why being physically next to somebody is better than a video conference, but it is. This would bring us one step closer to being physically present without actually being so just as video brought us one step closer than we were with just phone calls.

Field of view and peripheral vision?

Better visibility of non-verbal interaction cues. Also, not sharing screen real-estate between the person you're talking to and the work you're collaborating on.

But only if they manage to drastically reduce/eliminate the latency that's present with video conferencing.

It gets us one step closer by allowing virtual eye contact, but if the latency is still there, it will still be subpar to in-person interactions.

The other thing that wasn't so evident by this video is that everyone you were taking to would have to be wearing Hololenses, so it will still be a bit awkward.

Not trying to be a downer, just that as a remote worker, I think about all of these things that make remote interactions hard.

> The other thing that wasn't so evident by this video is that everyone you were taking to would have to be wearing Hololenses, so it will still be a bit awkward.

I remember seeing the guy who invented the selfie stick present it to Canada's Dragon's Den: https://www.youtube.com/watch?v=Bw7YbcoCkbs. My reaction was: this is ridiculous and stupid. Nobody will ever use that let alone buy it.

Let's just say I was slightly wrong.

Not only that, but people go crazy over VR. This will take off, I'm sure.

https://quikpod.com/quik-pod-selfie-stick-cbc-dragons-den-20...

You're sure it will take off. But at the same time you confess that you are very bad at predicting these kind of things. So, not sure what you're saying.

> So, not sure what you're saying.

That naysayers are usually wrong.

Though most things fail... So bettering against something is generally a good bet.

I said that sinxoveretothex_atThatTime was wrong.

The reason I dismissed the product then was because of the aesthetics. The same reason that my parent commenter pointed to to suggest holoportation wouldn't work.

I gave the example of VR. I can also give the example of "portable" phones and "portable" computers (have you seen how fugly those things were back then?). Or bluetooth earpieces.

That being said, I'm not predicting the end of in-person meetings or anything remotely like that. I wrote too candidly when I wrote "this will take off". Let's just say that we'll see more and more of that type of technology in the future.

I expect this to reach Bluetooth earpieces level of ubiquitousness: not quite without haters and not quite everyone-has-one, but rather everyone-knows-someone-who-uses-one kind of thing.

That dragon's den episode is baffling. The guy's reason for not licensing it to other companies is baffling. His valuation is baffling. I could practically feel them trying to not look too eager to take his 30% while calmly forcing out the words.

Here is something that never took off. And never will despite the popularity of this video. https://www.youtube.com/watch?v=YrtANPtnhyg

The difference? Selfie stick does exactly what it says it does and isn't more awkward, lowres, latency prone or difficult to use than a normal person would expect.

Unfortunately it's blocked in the UK "This video contains content from BBTV_SonyPictures, who has blocked it in your country on copyright grounds."

this is an alternative video https://www.youtube.com/watch?v=Nzu_5xd-AlM

I know it may sound trivial, but I work from home almost every day using short pants and no shoes. What I want to say is that home is still home, my kids will appear and yield, my tv might be turned on, and the dog would eventually bark. By no means I would want to transform my home in an office, I work from home for lot of things that wont be allowed in an office ever. And thats fine, I guess.

I work for MSFT but don't work on HoloLens.

Not all remote work is created equal. There are certain kinds of work that require richer kinds of interaction. E.g. whiteboarding, prioritizing with Post-It notes, brainstorming. Anyone who tries to "dial in" to one of these meetings is perpetually behind those in the room given they can't see what's on the wall/whiteboard. I could imagine technology like this making that kind of remote work much more productive.

We already have the technology needed to be much better at remote meetings, but people don't use it.

What technology do you have in mind? I've suffered through far too many awful Google Hangout meetings - if anything can improve remote meetings then I'd love to hear about it.

Nice "demo"...

Microsoft lately seems to be really good at being a "proof of concept developer." Which is cool and all that, but then people wait and forget why it was so cool.

As a comparison, Apple holds back their tech till it is polished and ready for market, so people are excited and start literally lining up to get whatever it is... And when they get it for the most part they aren’t disappointed, because it is pretty much exactly what was presented on stage at the show and they can show it off while its still a new shiny idea.

This is a demo from Microsoft Research, which does research into new areas, not a product group.

This is a point worth repeating. Does Apple even publish its research? I've benefitted from many papers published by Microsoft and Google's research departments, either directly through learning something I can apply myself, or indirectly through processes that end up in open source technology.

Does Apple even publish research? Some Googling failed to turn up anything. In fact, I found an HBR paper that said Apple specifically does not publish its research.

IMO, this makes Apple much less praiseworthy than Microsoft/Google/Facebook.

Steve Jobs abandoned Apple Advanced Technology Group. Apple isn't interested int technology researches.

off topic, but anyone know if there wifi mesh software went any further than this ? http://research.microsoft.com/en-us/downloads/994abd5f-53d1-...

That sounds insulting to the scientific research itself. Comparing the huge amount of research Microsoft Research division does to whatever Apple does to produce their consumer devices is really ridiculous. You can hold back on a 1-inch increase in screen-size until it's "market-ready" but you can't apply that to the real research areas.

Others like Apple actually deliver their products, and they work exactly like advertised. Microsoft is well known for announcing and promising products/vapourware years in advance. Like the HoloLens Minecraft demonstration at E3 2015. What a betray to the public eye it was. HoloLens has for technical reasons a very narrow field of view, the 3D content is only visible in a small area of the view, and cannot deliver the high hopes that were faked in the E3 demo. In the end it's just another augmented reality glass in prototype stage that at the moment fails to deliver and general public would be very unhappy with it. To prevent negative reviews of their demo unit Microsoft of course selected the audience carefully, so you will find just a few objective articles that mention the current problems like the incredible small field of view. But don't fall for the PR hype, they have a lot of money to spend, without trying it out yourself.

What people also always forget, because they fail to mention it, is the high amount preparation to set such a demonstration up. You need a 3D laser scan of the room and all objects, everything has to be static and shouldn't be moved. Than you need several high precision tracking devices connected to a central computer. The augmented reality glass ("HoloLens") needs to be connected via cable as well, otherwise you get those ugly video lags (seen in the clip). And you will see even the girl often just half of the body if you are standing next to her, because the glass shows only in a small area in the center the 3D augmented graphics as overlay (semi-transparent), and around that area you see your normal view through the glass without the 3D. So it's very annoying that you have to turn your head the whole time to directly point it towards where you want to focus, otherwise you see nothing of relevancy in 3D at all. It's bigger problem than with virtual reality glasses that show a dark background around the visible area (if at all). So this technology isn't new, but to get it ready beside very rough prototypes that are of little use for general public, it will take another 3-5 years, that's for sure.

Others like Apple actually deliver their products, and they work exactly like advertised

Check the other comments: it's not really false what you say, more like giving a twist to reality or comparing apples to oranges. This is about the Microsoft research department, not about their product delivery line. And the latter is what you meant when saying Apple - which indeed does deliver properly working products. Just like, say, Microsoft delivers keyboars or mice or Visual Studio or Surface.

Its a demo. Quite a good one.

Microsoft lately seems to be really good at being a "proof of concept developer."

AFAIK not really just lately: the research department has been going strong for tens of years, coming up with stuff like this which is on the edge of what currently available tech can do. It's maybe just lately it gets more press/HN coverage than 20 or so years ago.

Microsoft Research was started in 1989, to replicate the Bell Labs model.

(I work there.)

I never hear or see anybody mention this, but I feel that it is important to note that the actual surface area of the HoloLens is currently way to small to give anything resembling the presented experiences.

They always show the hololens used from a third perspective. Filmed with a camera, not the actual Hololens capture.

The hololens that I got to use (admittedly, almost a year ago now) was just nowhere near the expierence of any of the demo's. It's just a tiny tiny augmented display, compared to the massiveness of the headset.

Let's say it's more toward Google Glass than towards Oculus rift.

I really believe in this (type of) product, I have clients in certain fields where augmentation is just a obvious next step, but it is not going to be what they currently display.

And they know it, because there isn't a single image on the internet where you see what the Hololens user is actually seeing. Though I'm sure I'll be proven wrong at that in the comments below.

[Edit] Yes, I see there is a small render of the Hololens in the bottomleft corner, but you really need to see it compared to your full field-of-vision to get how small it is.

As far as i know only dev-kits are out and the Hololens is not consumer ready. The screen size is something they need to fix before then, i totally agree, but that problem is of a "beta hardware" type of problem and something time can easily solve - I believe.

They are actually pretty close to the FoV limit that the display technology they are using can provide. Magic Leap is going to have the same problem unless they have found a new technology since their original hints. Also long term use will probably burn your retinas out, they are likely running those lasers at the barely safe limits for brief exposure.

Don't know who told you that but that's not true at all. I've used test hardware that was the same display and had a full FOV (just needed to be connected to external power and GPU)

No lasers in use, just fancy optics

These folks have a ted talk video on their front page where they show the view from inside their headset. Seems reasonably expansive.

https://www.metavision.com/

They've even got a much wider field of view (they claim around 90°) which should be useful for most things that don't attempt to be fully immersive.

Likely wont be coming to market though. It's also tethered.

http://augmentl.io/ar-manufacturer-meta-being-accused-of-ste...

https://www.youtube.com/watch?v=MMALYjGcKFc

The human fovia is only ~7 degrees. Besides motion, we don't perceive much outside that range. The important technical developments here will be to paint the right details in the right part of the retina.

Pretty cool, but my first thought is: how authentic will communication be when "holoporting" to each other if you both have those big hololens mounted on your head? It will be hard to pick up on each other's facial expressions and impossible to look into each other's eyes.

Hololens will eventually be a contact lens. Progress is progress.

Might as well say that this will eventually be transporter or holodeck. The best we can assume at this point is that they'll reduce this down to something like a pair of large sunglasses, but even that's a bit optimistic without some revolutionary changes in optics.

We'll just replace our eyeballs.

We'll just replace our heads.

Sure, eventually, but I can't imagine that on any reasonable timeframe.

In a generation or two they'll do facial tracking. I imagine you'll get a full 3D model made of you, and it'll pick up the facial movements and transpose them onto your avatar

This is possible with current consumer-grade webcams. You'll video conference like you do today, except you'll see life-like rendered avatars of each participant. Participants with VR headsets will be able to see everyone else sitting at a table with them. If you don't have a headset, you'll see the same video conference UI you see today, just with avatars instead of video.

Next step - replace the avatar with an AI employee bot trained on your in-work responses, and we can all go off to the beach.

And never come back.

Yea, perhaps. Like in the recent "face2face: realtime face capture and re-enactment video"[1]. and your reference to "your avatar" reminds me of the scene from the matrix where Morpheus describes "residual self-image"[2]

[1] https://www.youtube.com/watch?v=ohmajJTcpNk

[2] https://youtu.be/AGZiLMGdCE0?t=26

See also Ghost in the Shell:

https://www.youtube.com/watch?v=poKi7YyuamI

Here's an example of Faceshift, which is completely markerless (Apple acquired them last year): https://www.youtube.com/watch?v=RLfAmNDNgHk

And here's a good example of the relatively high fidelity (marker-based) tracking that's happening in real-time: https://www.youtube.com/watch?v=OdQpJgLs51w

There is also FaceRig which you can purchase on steam: http://store.steampowered.com/app/274920/

I think facial expression is already possible, probably not nuanced enough, but that's just dependent on the resolution of the cameras and holograms.

Getting eye-level detail will be a pain because of the mask, but the real fun is going to happen when this can all be done with just a pair of glasses (and hopefully contact lenses in the far future).

Is it just me or HoloLens sounds a bit like a Hooli product?

All jokes aside, given that the rendering quality improves, this could have interesting applications for scientists researching facial and body expressions in certain environments. It would enable them to replay scenes and emotions which current 2D/3D video fails to capture, especially because it allows you to change your viewpoint.

I doubt this will be a game changer for teleconferencing, mainly due to the oversized device you have to wear, but then again all these gadgets tend to look a bit creepy in their initial phases (e.g. the Google Glass prototype).

NASA already uses Hololens to get the scientists to feel like they're walking on Mars. Rovers basically capture the footage from Mars in all perspectives, these pictures get transferred back to Earth, and scientists are able to "walk" on Mars with as low delay as possible (of course, it takes a while to transfer the images down to Earth). It's a remarkable thing and I really do feel like Hololens (or some other Augmented Reality project) will have a huge impact on the scientific community.

I did an interview with a software developer from NASA about this project: https://opensource.com/life/16/1/scale14x-interview-parker-a...

Why would they use AR and not VR? AR actually seems useless for that use case.

AR would allow you to interact with your co-workers much more naturally than VR

Here's the actual page on the research - http://research.microsoft.com/en-us/projects/holoportation/

Compare this to previous work at Microsoft, just last year: http://techxplore.com/news/2015-07-microsoft-hololens-video-...

How do they define which aspects of the captured image to extract, model, and transmit? If they were just modeling the humans, I could seeing using motion detection, or infrared perhaps, but they seem to have also extracted and modeled the little girl's toys.

So i'm curious what method they use to extract just specific bits of the image captured in the holodeck. Perhaps this is answered in one of their papers.

If anyone knows and has a quick overview, or link to a relevant paper that'd be great.

The setups are identical, so it seems obvious that they capture everything that differs (i.e., not part of the floor and cubes).

The easiest thing is probably to "calibrate" the scene with nobody in it, since the cameras appear to be fixed.

I realize that I'll probably never get something like this, but dang, that was cool.There seemed to be a distinct quality difference in the realtime created models, and the prerecorded models. The realtime ones had issues, where as the prerecorded ones (like the girl with the dog) were smooth and nice. I definitely think that if this goes mainstream, higher quality models will be generated in most cases and only use real time stuff if there isn't one available.

Honestly, given how rapidly technology is advancing, I think that it's not unreasonable to expect a consumer-friendly version of this within 10-20 years. It's happening!

Five years.

Downvote me if you want but HOLY FUCK! I got goosebumps when he started replaying the interaction in reverse! This is really reaaaally cool!

It was really cool! the replay with him and his daughter reminded me of a scene in the Minority Report, but much cooler!

This is pretty exciting because of its potential for telementoring. I've been working on a research project using tablets and augmented reality for surgical telementoring in austere environments ( https://engineering.purdue.edu/starproj/ ), but a setup like what MS has here would offer so many benefits. Imagine a trainee surgeon being able to feel as if a mentor surgeon was actually present and able to gesture and give instruction during the course of a live surgery, for example.

I see they learned from their previous campaigns, where they focused on the ability to "work from anywhere" and were criticized[1] for creating a strange, dystopian vision of all work and no leisure. Now, they are focusing much more on the personal aspects in their marketing, even though their primary market is fairly likely to be business.

[1] https://signalvnoise.com/posts/3683-microsofts-dystopian-pit...

If you look at the tv that's in the room, it looks like it's actually showing the real live 3d reconstruction with what I'd guess is a 500-1000ms latency.

So the main video that is in sync (when the guy has it's own holograph in superposition) was certainly edited afterwards..

He seemed extremely careful not to let his daughter walk through him. He always made room for her. I wonder if there have been studies done to assess how creepy it might be to have a family member walk right through you...

I think it was more of a gut reaction of "here's something moving towards me, let me get out of its way", rather than being extremely careful. I don't think we're at all ready for having things passing through us.

I believe he was demonstrating spatial awareness, or perhaps it's merely because the compositer doesn't do well with intersection.

I also found it interesting (though possibly scripted) how he stuck his hand out at 2:55 as if to try to stop her from falling.

Remarkably impressive, but I have to admin to being a bit creeped out by the save and relive stuff at the end. A bit too close to the dream machine in "Until the End of the World" for my taste.

Give them another 4 years time to perfect their technology, that is 2020, another 5 years again to iterate and adoption. I can see in 10 years time this will have massive impact to office work. Less Business trip to travel.

I dont think it will replace business travel. But there will be a lot less of it. And it is also dependent on the countries broadband quality. ( May be in the future the Job requirement will write you need 100Mbps broadband + low latency results. )

So, an idea:

How much work would it take to segregate just the augmented reality displays as a pair of glasses that a user can wear while receiving the display data wirelessly? I say this because once you can do that, you can have (for instance) an executive conference room setup with a number of identical conference rooms scattered around the world, each with a server performing the hololens rendering logic and rendering across _n_ glasses per room. No bulky hololens computers-on-head and an eerily realistic recreation of one conference room with all participants in-room.

I don't know the exact patents Microsoft filed on the display technology they have for the visors, but I suspect it's not yet easy enough to compress them into anything close to Google Glass yet (you'd need larger glass in any case), but one can hope.

Once we get there anyone will be free to re-enact Kingsman at will

Install this at sports stadiums and not only make my own viewing more "embedded" in the action, but allow replay to get any angle (granted lower res than they are used to for really tight calls). I imagine that many sports organizations would be lining up outside MS research's door with cash in hand. Though the tech probably needs to advance quite a bit more to capture that many 3D objects in real time.

Just this week I wrote a blog about how to build your own holographic studio by using multiple Kinects and RoomAlive Toolkit. Nowhere near the sophisticated capture that Microsoft Research demoed of course. http://smeenk.com/build-your-own-holographic-studio/

This is not cool just for work. I can definitely see myself using it to feel closer to my family, a few thousands miles away.

Also, this is research. Too many of these comments are bashing it, mostly because it comes from MS I guess. We can still hate their products, but the research is really useful.

Hey, this looks pretty impressive. It also makes me want to imagine some of the video game uses of the technology, like say, how cool a Five Nights at Freddy's game would be based on this tech...

To some degree, it also reminds me a bit of a certain villain from Teen Titans...

Pretty cool demo! Maybe they got inspired by what this guy did a while back with 3 kinects: https://www.youtube.com/watch?v=Ghgbycqb92c?

Thats very impressive, looking forward to seeing this advance in the next few years.

It's expensive, which makes me want to think of who would buy these first. There might be a market in medicine (think Vilayanur Ramachandran style phantom limb therapy) and psychiatry.

Shrinking it all and putting it on the coffee table was a nice trick.

So amazing. Can't wait until all you need is a few tiny webcams and something like magic leap allows it to be in your glasses so you can actually see the other person unobstructed.

The concept is great but the equipment must be very expensive.

It seems multi-national company may make money by providing a private room to enjoy the 'holoportation service'.

This team must have the best remote standups.

Very impressed! I can't even begin the possibilities that this tech could bring to the table.

Microsoft Office 3D here we come, walking past life size graphs and real life spreadsheets

I can't even, the video "demo" has been so obviously post-produced.

very very cool. The guy in the video is an excellent presenter too.

The case against supersonic plane travel..

These things will be the end of all of us.

Microsoft just made PORN more convenient!

It's very cool!

Url changed from http://www.gamasutra.com/view/news/268997/Microsoft_demos_ho....

1st thought: next Skype

2nd thought: 4 cameras and some space required

3rd thought: command-w

The first and only promising technology from Microsoft I've ever seen :)

Microsoft seems to be fiddling around in research while Sony runs away with the Virtual Reality cup.

As a point of clarification, this is Augmented Reality, which is actually quite different than Virtual Reality in practice and scale.

So it's an apples to bananas comparison.

The biggest difference at this stage seems to be people handing over money for VR while AR, well......

In a very real way they are "the same" in that Microsoft and Sony have each chosen to bet on AR or VR, not both.

The are the same in that they are both "immersive headset technology interfaces".

Looks very much like a betamax versus VHS battle and AR at this stage is a distant betamax.

Yeah... no. The VR industry is in a bubble right now. This is actually something that was talked about at GDC. A massive portion of the VR industry is betting on video games, and there's several good economical reasons why we think the high-end VR products (PSVR, Oculus, etc) will vastly undersell.

Sony is the one who's likely to suffer most, in fact, as the tie to their console limits their market even further.

The Vive has one camera on the headset, so it can do a kind of AR too..

No, it doesn't and this has already been discussed ad-nausea around the VR world.

I'm sorry but could you provide some explanation (hint or link)? I'm probably wrong but your correction is a bit short IMHO.

I don't understand why they need to pretend to support features they haven't developed. It makes something that is obviously advanced feel cheaper and fake in some way. The daughter is obviously a recording, but they call it "live", say she can't hear him, then he gives her audio queues. Who is editing this and what are they thinking?

He says "you can only hear me but you can't see me" (since she isn't wearing a hololens). I think it is actually happening live between him and her.

Yup and you hear an echo of his voice (presumably because she's in an adjoining room). That wasn't pre-recorded.

She also glances a few times at something (a screen?) at the top right corner.

Reply on news.ycombinator.com