Jump – Experiences like you're actually there | Hacker News Reader

Jump – Experiences like you're actually there

google.com

445 pointsjulianpye191 comments

I love all the experimentation in this area but I'm afraid systems like this are going to fall into a sort of uncanny valley of VR. Your brain is incredibly good at picking out the discrepancies between the real world and a high-resolution 120fps image with surround sound - just as it is incredibly good at recognizing objects and motions masquerading as human.

It's like going from 2D/24FPS to 3D/48FPS. You'd think it would be a huge increase in perceived realism but it's quite the opposite. What you want is magic window, which is what you get when a film or even a game really pulls you in and your perception sort of synchronizes with something completely out of step with your physiology and totally unbelievable for many other reasons. Instead, you get a feeling of unreality or surreality, and the final gap between the media and your perception is unbridgeable.

I think that's part of why VR/3D/HFR stuff has always been so unconvincing, and why it's no coincidence that it's finally only catching on when we pair it with totally unrealistic games and demos. You need the fantasy. Because it's so clearly not reality, and anything that attempts to replicate reality too closely will be rejected.

Anyway, cool tech though, and I'm sure I'll enjoy using it some day. But for now I think the foreseeable future of VR is in virtual environments, not in the duplication or capture of real ones.

I think the reason VR is better with games and demos is because it's hard to properly record in the real world. That's what Google is trying to solve here and I expect the results are far ahead of current solutions. It's both using a lot of hardware and a lot of software, I'm very excited to check out the results!

I think that's a slightly pessimistic view. This experimentation is what will eventually enable the bridging of the uncanny valley. And I think that day will be here sooner than we think.

You're talking about Presense, which is the point at which your brain flips and is convinced what its seeing is real.

Michael Abrash has a wonderful deck he wrote while at Valve about what it will take for VR to create Presense in the viewer. It's worth a read if you care about how we'll navigate past the uncanny valley of VR you describe: http://media.steampowered.com/apps/abrashblog/Abrash%20Dev%2...

Heh, I figured there was probably a word for it out there somewhere invented by the people actually doing the stuff. I'll take a look at this, thanks.

Since people might not be familiar with the term yet, I feel it's useful to point out that it's spelled "presence" with a 'c', as in, you feel as if you were actually present in a virtual reality.

Haha I was reading the parent and I was like "Pre-sense?" "What would that have to do with feeling immersed in VR?" This cleared it up, thanks.

Heh, and for a moment there I thought Pre-Sense was some Google Tech - similar to Adsense - for monetizing 3D VR :)

There's a ton of really interesting stuff on his old Valve blog:

http://blogs.valvesoftware.com/abrash/

I freaked out while a piece of metal came to my face when i was watching Mad Max Fury Road in 3D. Is my brain fucked up?

I actually loved 3D/48FPS - I felt like I was watching a play in a large diorama. I've regularly wondered if different theaters tuned it differently, or if people's brains just have differences, since so many other people reported 48FPS as looking fake.

If you already felt like it looked like a play, then that's why people didn't like it. Sets look like they would on a stage, not what the stage represents. If the story takes place in a castle in 18th century England, you want the viewer to be thinking "I'm in 18th century England right now" and not "I'm in a theater watching a film about 18th century England". Creating a film is a lot about suspension of disbelief, so if your brain is saying "that looks like a set" instead of "that looks like a castle", then you've lost that battle.

Or then it is like the case with youngsters and mp3s, where the studies have found that they actually prefer the "sound" of compressed music - with compression artifacts and all. If a hifi enthusiast (~tunesmith) would say that uncompressed music sounds better, you wouldn't shoot his argument down by saying that improved fidelity and dynamics just allow you to hear the music was recorded and postprocessed in a studio.

What I'm trying to say with this mp3 analogue is that it is very much a matter of habit and preference. Technically lower quality video does not lead to better immersion if it is not something you have learnt to expect.

What killed 48FPS for me was how obvious it made the transitions between heavily CG/post-corrected scenes and fairly unedited ones. The overall color tones and movements weren't consistent. To me the Hobbit 2 was visually half film, half video-game.

I could see a movie with no CG (or careful use of it) being very enjoyable in 48FPS.

Pay careful attention to her reaction:

https://i.imgur.com/avVTybvh.jpg

The brain is actually fairly easily tricked.

http://blogs.discovermagazine.com/notrocketscience/2011/05/2...

You can get immersed easily in any video game (I felt vertigo the first time I jumped from a building in Minecraft), but you can always tell what is real.

Sure, the brain is trickable. We had people jumping out of the way of the train and gunshot in "The Great Train Robbery" as well, of course. Notably the guillotine simulator that is so effective is a virtual environment, which is what I was suggesting will be the rule for a while in VR.

You can get the same reaction from playing any other game with an element of surprise on just a normal display.

I differentiate between jump scares/immersion and presence.

Imagine playing a hack and slash game. A user is not reacting every time their character is hit. They may have an initial reaction to a surprise attack that catches them off guard, but no more. There is some level of immersion but there is no level of presence. They never feel like they might be in danger and their body does not react as if it is in any danger.

Presence would be having a feeling of discomfort anytime your avatar was injured. The brain having tricked itself into thinking it is the avatar. When the avatar gets hit by a weapon, the user's body sweats and tenses up, believing it was just hit by an actual weapon.

Air-pressured tactile gloves can help develop a sense of presence. For example, pressure when grabbing a rock may trick your brain into believing the rock is real.

I wanted to focus more on the studies than the .gif. Having a mannequin stabbed and the participant's body reacting is a sense of presence and not simply a jump scare. The brain couldn't tell its own body from that of a mannequin.

How does something like this compare? I would like to try it:

https://www.youtube.com/watch?v=cML814JD09g&feature=youtu.be

(The bit with the spider webs especially registered with me as something that might be a trigger for making your brain go wonky.)

If they have material so when you swing your sword - you actually hit something. Or when you shoot your gun - it provides some kick and feedback as if you were firing a real weapon then I can see this working really, really well.

The audio cues are there - the visual cues are there. Add in some haptic feedback and maybe even smell (ie. make a room full of dead bodies and blood smell rancid) and you've hit every sense but taste. I can easily imagine it would cause a state of presence if done right.

Obviously their graphics will have to be "good enough" (although sadly it won't reach their CGI'd video editing level anytime soon :( )

That's the stock, first time seeing VR reaction for alot of people.

What really matters though is persistence of engagement. Outside of those of us who've drank the koolaid or have a vested interest, how often the general public will come back to VR and how long they will stay inside it will be the critical factor for success.

That rate is still well below the threshold needed.

See also this guy's reaction (around 1:20):

https://youtu.be/Odax7F3tWhM?t=1m20s

It's not about whether the brain can be tricked or jump scares will work, it's about whether you'll go "Oh wow" or "My head hurts this looks stupid"

Like an 8-bit guillotine vs this one.

The original commenter actually made the exact distinction between VR (what you linked) and capturing real life video

> Pay careful attention to her reaction: > https://i.imgur.com/avVTybvh.jpg

Note: This jpg is actually a gif (?)

That's the power of Virtual Reality

You sound like someone who hasn't tried Oculus Rift. In that case I'm happy to inform you that you're wrong.

I've tried the Rift, and I completely agree with him.

agree that oculus rift doesn't feel real? I own a rift too, and I don't understand how someone can easily say it doesn't feel real if they actually tried it. Sure it brings nausea sometimes, and sure a lot of the graphics are crappy, but can you so certainly say that it doesn't feel real? I haven't seen anyone who's tried oculus rift that say "huh how is this different from a video game, it's obvious it's fake"

I've tried an Oculus Rift (DK1 only though), so here you go: how is it different from a video game, it's obvious it's fake.

Genuinely though, I found it a far less engaging, far less immersive experience than a normal 2D screen. Maybe the later models are better, I don't know, but I definitely don't see what the fuss was about.

It doesn't feel real, because I know its fake. My brain isn't fooled into believing that I exist in this different world. I fulling know I'm looking at a video game, in fact I found the raster lines reminiscent of classic arcade games.

I had fun, but what the article in this thread is talking about is a 'holodeck' feeling. The uncanny valley is where the space between where your brain registers the image as imaginary and is ok with it (3d cartoons) and where it rejects the image as if its offended at being lied to (polar express).

The rift is very much in the former, my brain knew it was a game.

I've tried it up until Crescent Bay, and it's fabulous. But I still feel it is very much for carefully fashioned, virtual experiences, and will have trouble replicating things via video, however many gopros you put on the task.

"These mobile phones will never reach a mass market! The batteries are far too big!"

You might be interested in this talk that Michael Abrash gave about 2 months ago at Facebook F8 [1]. It starts at around 00:42:20.

It is a fascinating talk where he provides a number of interesting demonstrations regarding how the brain interprets information, including depth perception, color interpretation, movement, auditory cues [2] etc. Highly recommended.

[1] https://www.youtube.com/watch?v=UDu-cnXI8E8&t=2540

[2] Absolutely fascinating. What you hear depends on what you are looking at, not just on what you are hearing. This demonstration starts at approximately 01:02:00 - https://www.youtube.com/watch?v=UDu-cnXI8E8&t=3720

I found that talk to be juvenile. If I want to look at optical illusions I'll google search them myself.

>so clearly not reality, and anything that attempts to replicate reality too closely will be rejected

You should see some Oculus demos in person before making claims that VR is easily rejected by your body. There are some roller coaster demos that can make you nauseous within a minute.

To be fair a reasonable amount of people find Oculus rift makes them nauseous in any application.

But 99% of times it's motion sickness. You only get it if the camera moves, which is not the case here.

Totally disagree.

A few months ago I was walking around the Unreal 4 Paris Apartment demo [0] with a Occulus DK2. The resolution isn't quite there, but it definitely felt real. I think that VR can definitely provoke feelings of reality, at least if the people working on it keep going forward in resolution like they have been.

[0]: https://youtu.be/Y6PQ19BEE24

I think it's worth correcting your use of "Uncanny valley". My understanding (see links below) is that "Uncanny Valley" refers to virtual beings trying to trick you into thinking they're real but coming off as creepy instead -- e.g. the Tom Hanks voiced train conductor in "The Polar Express". Creepy. See here: http://en.wikipedia.org/wiki/Uncanny_valley and the new yorker article here: http://www.newyorker.com/magazine/2014/04/28/pixel-perfect-2

It does not refer, as you suggest, to the general case of VR not seeming real. There is a special thing that happens when we see fake beings trying to pawn themselves off as real. It think it's worth preserving the distinction.

It is readily apparent that the commenter is using an analogy ("a sort of uncanny valley of VR"). Let's not get sidetracked.

Well put and fascinating points.

I;m not entirely sure if this is adding to or disagreeing with the points you make…

A few weeks ago I commented on some thread about robot maids and such. My point was and is that when we picture a far off idea that we suspect is coming (AI, VR, human-less hotel-resorts) we need mental placeholders where people go now. Technology tends to replace people in the way reddit replaces librarians but we can’t picture reddit in advance until the world wide web or something similar exists. So, flying cars, robot mades and sexy female AIs. They’re placeholders for future automation or changes that make the need go away rather then meeting the need.

Back to Jump. Our starting point is a made up idea of what VR entertainment looks like. Game of Thrones in VR. Jerry Springer in VR. Xuxa in VR.

You make a valid point that this won’t work. I totally agree. But, I’m fairly optimistic about the ingenuity of artists. Lots of things run into these kinds of problems. A conversation between two actors pretending to be doctors is not uncanny, but there are arts to making it palatable. They will figure out POV issues. Should you be a character or an invisible eye like in flat films? Maybe they could use filters to make it look like ‘A Scanner Darkly’ to get over the issue you describe.

This is what art does. They’ll figure it out.

> A few weeks ago I commented on some thread about robot maids and such. My point was and is that when we picture a far off idea that we suspect is coming (AI, VR, human-less hotel-resorts) we need mental placeholders where people go now. Technology tends to replace people in the way reddit replaces librarians but we can’t picture reddit in advance until the world wide web or something similar exists. So, flying cars, robot mades and sexy female AIs. They’re placeholders for future automation or changes that make the need go away rather then meeting the need.

That's an incredibly interesting point. I've never seen 'future' technology like that before, but it's certainly an excellent analogy. Looking back to how people saw the future in the 20th century, you can see that many things did not come to exist in the way they were imagined, but do exist now in some other fashion.

It's incredibly difficult to predict what will be the next reddit, or the next anything. It remains to be seen if/when VR will take off, and how it will differ as we see it right now. Certainly what's interesting is that we're watching our VR dreams being turned into tangible products that can bring real VR experiences.

VR has already taken off. It's not everyday mainstream yet, but it is well on its way.

I do like that way of looking at future technology, but I think we're close enough to VR "happening" that we don't need that sort of thought experiment.

I'm unconvinced about the existence of an uncanny valley in frame rate.

I think it has more to do with what we're accustomed to.

High FPS on large panning scenes is amazing, and the conversion back to low fPS is just as noticeable and jarring.

A friend of mine staunchly opposed high FPS until the 60fps NSFW reddit changed his mind.(well, for HD porn anyway).

The problem with 48fps on the big screen was that it didn't feel "filmic". It actually WAS more realistic, but in that specific instance this actually hurt the message of the content. When we are recreating live events, concerts etc, we aren't going for filmic, we are going for the feeling of being there. I have tried the Paul McCartney example on GearVR which uses a similar technique as Jump - it's not perfect but it is a lot better than watching the concert on a screen.

TL;DR - We are approaching what we might want in a captured live experience, where early 48fps3d movie experiences may have passed what we want in a filmed one.

My experience of The Hobbit was that the 24fps version felt like being in Middle Earth, and the 48fps felt like being on the film set - which strictly speaking was the more realistic experience, but far less satisfying.

I always find sports footage strange to watch on TV. I have a feeling its because of a higher frame rate, and less blurring of movement. Its kind of crisper but less absorbing.

People have been watching movies at 24fps for so long that they've been trained to put their brains into movie mode and suspend their disbelief when they see something at that rate.

There were similar outcries at the advents of colour movies and "talkies". Once people have watched a few HFR movies, I'm sure they'll get used to it and learn how to suspend their disbelief again.

Maybe, maybe not - it may also depend on the movie being made. I thought the Hobbit was kinda tacky compared to LotR, it seemed to have a much smaller budget, especially in the CGI and extras/makeup department.

But, I'll give it the benefit of the doubt for now - 3D was kinda crappy too at first, but some years later the movie makers have improved the technique to make it far less annoying. It's become more than just a gimmick now.

To me it felt like a video game, and in fact the action supported this. I was left wondering why someone would go to the trouble of making a computer game that you cannot play.

I'm still holding onto hope that we'll surpass the uncanny valley and get to a point where we can create things that are totally believable. There's going to be a weird time before we get there but I don't think that should be viewed as meaning it's impossible.

I wonder how much of that uncanny value would simply be due to the unfamiliarity of the technology. For example, I'm sure we all know about the stories of people freaking out when they saw http://en.wikipedia.org/wiki/L'Arriv%C3%A9e_d'un_train_en_ga... though there's some doubt as to whether or not they actually ran to the back of the room. Either way, my point is that I'm sure all new technologies at first seem either unnaturally boring or unnaturally exciting. Eventually, people get used to them and that effect wears off. At this point, I think the only real obstacles standing in the way of widespread VR adoption are the possibility of motion sickness and the resolution of the displays (screen door effect). Otherwise, it seems like the tech has matured and would be very useful to a lot of people.

Something that's maybe not clear from that page, but was mentioned in the Google IO keynote: this is not just stitching of multiple video streams.

Google does some heavy-duty machine learning / computer vision on their servers to extract 3D information from these video streams (they mentioned having depth data). Then they presumably re-generate seamless stereoscopic 360-degree video from this 3D model. That's how they can get stereo from mono data.

It's something like those hyperlapse videos from Microsoft, they also first do number crunching on lot of video data to generate 3D model and then use this to fill in missing data.

Yes. I wrote the code for the http://www.3d-4u.com 3D camera. You definitely can not use simple still image stitching algorithms. Per pixel depth estimation (via dense stereo matching) and subsequent blending must be used. I would say more, but I had to sign a long NDA before I worked on this.

This looks like 180, not 360.

Yep -- same seam issues though.

i was expecting the cube to get solved once the page finished loading.. I'm dissapointer

> get stereo from mono data

Why do you call it mono data? Given how close are each lense from another, I'm guessing, that anything far enough (0.5m?) from the cameras is beign recorded by at least two cameras at all times.

Because simply stitching it will create one image/video stream from one perspective. If you have 2 eyes, you have stereo vision so can perceive depth. This technique allows for the synthesis of stereo video from a rig that doesn't directly capture stereo video. Pretty cool stuff

The point the GP was making was that you can treat each pair of cameras as a pair of 'eyes' for the stereoscopic analysis. And you get every combination of adjacent cameras as a pair so a ring of 8 cameras represents 8 distinct pairs (call them camera 'a' and camera 'a' + 1 clockwise)

By processing those streams you should be able to derive high quality depth information from the combined video streams, and given the overlap in views re-compute a 'view' from any direction.

A lot of early work was done using the streetview cameras as well to do street facing feature extraction using similar techniques but as those cameras were essentially single shot (discrete pictures over time rather than video) you end up with the "slewing" effect of moving from one spot to the next.

Presumably Google will go out and do street view with this sort of setup and get a continuous terrain option so that you can really walk around as if you were there (caveat occlusion effects)

I don't know specs of the HERO4, but the GoPro HERO3 has a 170 degree view angle - so that 16 camera rig has a _significant_ overlap. I suspect you could generate both "adjacent camera" and "every second camera" stereoscopic pairs without too much difficulty, and you can probably even extract useable data out of the "every third camera" pairs and their ~30 degree view overlap. My guess is everything seen by that rig is viewed by 3 distinct pairs (a&a+1, a&a+3, and a&a+3), and that the entire scene from a 16 camera 170fov rig could be considered to have 16 a&a+1 pairs, 16 a&a+2 pairs and 16 a&a+3 pairs giving you three stereo views with different baselines for every point in the scene. I'm guessing there's a _lot_ of spatial information in there.

Actually, I just realised all those HERO4's are mounted in portrait orientation, so they won't be getting 170 degrees of horizontal view, and you probably aren't going to get useful overlap between the a&a+3 camera pairs. (but it does mean there's only two 20 degree wide conical polar "blindspots")

I had thought we would never get 360 + stereo. I wonder how much touch up work this will require and what kinds of scenes will provide problems with incorrect depth heuristics.

There are a couple of consumer ready 360 camera options available to purchase today ($300-$400), albeit not 3d steroscopic. I have personally played with the ricoh theta and the kodak pixpro and think they are good products. Here are some links if you want to check them out, I don't work for any of these companies but am a VR enthusiast...

Good for stills and video https://theta360.com/en/about/theta/

Action camera similar to GoPro http://kodakpixpro.com/Americas/cameras/activeCam/sp360.php

http://www.vsnmobil.com/products/v360

I took an sp360 on track. Review: pass. I can dig out the video if you care.

Agreed, the SP360 is just awful. Shame.

Looking at the top comments - I guess I'm completely crazy. I 100% thought this was a late April's fools. They legitimately think people are going to buy a 3D printed (~$12 hat), that holds about $5k worth of gopros? So... this is like another Nexus Q/really smart guy's horrible idea that's just going to disappear with no explanation?

Maybe they would given some basic incentive. Free internet, some discount, gift card?

I wouldn't underestimate what some people will do for a little spiff.

Maybe they won't do it again, but they might do it once, or on occasion. If one could manage churn reasonably? Who knows?

That said, not on your life. Not doing it.

And why not? It's not about the hardware - it's about the videos they can create with a simple tool (and a few thousand to buy the cameras), which puts this technology within reach of the semi-professionals and higher-end hobbyists, instead of specialized equipment in Google Streetview cars. Like how Cardboard gives simple VR to anyone with a smartphone. It's not about whether it's a killer app, it's about planting the idea into people's minds - and eventually, about creating content. Because that's Google's trade - content. Cool 3D videos = more views than a regular gopro video = more advertising income.

If my assumption is correct that Google has a monopoly on search and user-published video, then your argument seems illogical to me. They're already getting almost 100% of the available pie, so where would the growth come from?

Even if there is room to grow - how much extra money do you think will actually come from this venture and why?

No, this product seems like bullshit which is designed to keep nerds talking about the Google brand. It's a flashy, go-nowhere project that nobody is going to use. I'll bet you $100 that people generally do not remember it a year from now.

Looks great, though as other commenters have pointed out, 16 GoPros makes for an expensive rig.

I've been watching this space for a while and pretty much all of the consumer varieties (usually Kickstarter-backed) have pretty terrible video quality, so it looks like I'll be sitting this one out for a while yet.

They don't claim this to be a consumer product.

As others have said, the cost of this is peanuts compared to professional cameras.

I'd bet one of the first killer-apps for VR will be sports: Imagine several of these rigs set up around (and over) a football field, so that spectators can choose to "be" at the right point on the field to observe the game, and have complete freedom to look around the field.

I think you'd be surprised how nice it is to have a team of producers and camera operators doing the work of finding the interesting views for you. People attend live sports for the atmosphere and shared experience, not the view.

Though a court side camera at a basketball game would make an interesting vantage point... People definitely do sit court side for the view.

That's true - but for my preferred sport (motorcycle racing) sometimes it'd be _really_ nice to not have to "settle" for the producers choices of what to see (that last MotoGP race had, from memory, 3 critical overtaking manoeuvres happen while the producer had decided to cut away to showing the team/girlfriend in the pit garage). We don't have the bandwidth (yet), but I dream of the day when there's a cluster of camera arrays (similar to that 16 GoPro rig) at every on track vantage point (the current camera locations would be fine to start with), with a few hundred video streams coming out of the race track and the option to use whatever smart software you can buy/subscribe-to/write, or even maybe get a curated feed from someone other than the on-track producer. The World Superbike Championship telecast is already using some pretty good image processing software to paint real-time telemtery overlays onto a bike in the video feed - it's still making mistakes, but it's pretty impressively good considering it's happening honest-to-god-live. (I can't find any public videos to link to there, sorry - Dorna are obviously really good about sending takedown notices to Youtube, and all the good video on their own site is behind their paywall...)

How is Google going to be involved though? They don't have the licenses for sports casting.

I'll take your bet - how much? - I doubt TV networks are going to do anything with it and if they do, they won't be licensing it from Google.

16 gopros isn't much more than a mid-range Canon DSLR, which TONS of amateurs manage to pony up the cash for.

I don't think price will be a big inhibitor here, honestly.

Price is a huge inhibitor for a mid-range Canon DSLR, so it'll probably be a huge inhibitor for this system too.

Yeah but compare it to a Red or a Hasselblad and it's utter peanuts.

Oh certainly, I just wanted to make the point that photography is a notoriously expensive hobby to begin with.

Uhhmm... how much do you think gopro 4's cost?

About $2000

ahahha what? more like $8,000

It might be more cost-effective to use the Xiaomi Yi

http://www.engadget.com/2015/04/29/xiaomi-yi-camera-versus-g...

It's an open spec. Use Rasberry Pi and camera modules. Buy 'em used. You can probably build the rig for <$1500

The interesting aspect here is the iPhone availability for Cardboard and the use of Youtube as the platform for 3D VR.

Youtube already had stereoscopic support, but there were not many clients available, content was limited and difficult to create with 3rd party solutions that all cost money. It never took off and couldn't ride the wave of the studio and CE company driven 3D momentum, which quieted down after a while anyway.

This time Google ticks all the right boxes. Client devices on iOs and Android. The videoplayer everyone uses. A $5 cardboard enabler. A capture platform with GoPro and free assembly software.

I'm looking forward to trying out Cardboard on iOS. Anyone have a recommendation for a good viewer manufacturer? (I'm a bit surprised at how much some of the kits are, given they're made of, er, cardboard).

I am a bit partial as I am one of the founders of this company but here is a shameless plug for our vr viewers... http://www.dodocase.com/collections/virtual-reality

Thanks for the shameless plug :) I like that you called out the material quality/selection in your viewer. I'm actually kind of fascinated at the level of guidance Google gives to help partners make these. Anyway, pre-ordered and look forward to it! (iPhone 6)

GoPro is a third party solution that costs money (a lot). What makes it different from before?

The playback devices were basically your PC with anaglyph glasses (terrible experience) or a PC with Nvidia 3D setup. This is the main difference.

Editing software were expensive Sony setups ($5K), plugins for Adobe AE or very feature-poor setups such as from Magix (around $200). Editing was very tricky, since there were few standards here, just many custom ways to do it. This is a somewhat difference, since it can be assumed that the path from capture to quickly experienced content may be very short and allow for a camera to youtube experience.

Creation devices were custom 3D rigs as expensive as the GoPro setup or consumer cameras like the Fuji W3 camera series. In this space, you are right, there is not so much difference there.

16 GoPros? Sounds expensive. Certainly not something I would buy, since it would gather dust on the shelves along with my other gadgets after a few initial uses.

That being said, this may be the right time to reinvest in 360 degree stereoscopic video since the viewing technology is finally catching up. Point Grey has been making similar spherical cameras since the 90's [1]

Pretty cool idea overall. Other questions I had: Does it have a single data interface? How do you charge this behemoth.

[1] http://www.ptgrey.com/360-degree-spherical-camera-systems

xiaomi's sports camera is about $70, gopro's cost is about the same I guess

> gopro's cost is about the same I guess

The GoPro Hero4 costs $400.

talking about cost here, not price. gopro is way over-priced.

But the GoPro Hero is only $130 and is physically the same size.

The first GoPro is garbage. I have one, and I have the 3+ one. The first one is noisy, has horrible compression, absolutely horrible at low light, doesn't shoot 4K, doesn't do 60fps at 1080p.

> The first GoPro is garbage [...] doesn't shoot 4K, doesn't do 60fps at 1080p.

Truly we live in the future. If you told me even 15 years ago these would be legitimate complaints for a sub-$200 camera I would have called you crazy.

15 years is an eternity in technology. Check out the processors from 2000, they are all garbage by every reasonable metric. They are only good as heaters now.

And the lower resolution and features probably don't matter when you stitching together 16 streams.

The 16-GoPro rig isn't for a home user, it's for YouTube channel owners and other video publishers who are making money from their videos and want some more immersive content.

Well, I guess my point is that this is a bit overkill. A professional content producer may want higher quality than consumer gopro cameras.

Whereas a home user (who may also want to create such immersive content) is left out in the lurch.

Well I'm sure you'll be able to one day have a rig of 16 RED Epic cameras and crunch through all the 6K videos, but plenty of neat scenes can be recorded with the current version. 16 GoPros aren't cheap (though this could be quite a nice package for a rental shop), but all-in will still be quite a bit cheaper than just one RED Epic body (without lens!).

i don't think you have any idea what you're talking about.

lots of special-interest (cars, outdoors, extreme sports, etc.) youtube channels are filmed primarily on gopro rigs (suction-mounted, body-mounted, drone-mounted, etc.) and they make millions of dollars of revenue.

"professional" means a lot of different things. the content production market is massive and very profitable.

My brother makes professional videos on his phone, as well as a RED camera. It entirely depends on what he's trying to do, and what the people paying for it are looking for.

with a 90 degree FOV (such as oculus) you are looking at the feed from 4 go pros at any moment. the hero 3+ does 4K, so you're reducing four 4K feeds into a single 1080p/2 (oculus DK2) feed.

When you're going from 33 megapixels to 1 megapixel, i'm not sure how much optics is going to matter...

Gopro's are what $400? That'd be $6400, which yeah may be out of the price range of your average consumer but my brother who does film work will routinely spend more than $2000 on a lens. If there is a market for these kind of videos, then it's a drop in the bucket.

If you want a cheaper 360 degree camera, consider looking at the RICOH Theta, which retails for $300 on Amazon:

https://theta360.com/en/

http://www.amazon.com/Ricoh-Theta-Degree-Spherical-Panorama/...

Has anyone see the opposite of this done - where you have a 360 degree array of cameras pointed at a target? This would allow the viewer to pan around the point of interest during an action sequence.

I'd like to see an array of cameras using quadcopters flying around a target - say a whale jumping out of the water or a football player. This would allow the viewer to see all side of the action. Great for sports playbacks.

I don't think a quadcopter can hold cameras with enought distance apart from each other in order for a shot to be any useful, I mean think about it, whale jumping or a football players bodies are much bigger then a quadcopter radius, so the 360 views will most probably be useless

Well, if you have 16 quadcopters - each with one camera... :-)

or just one quadcopter, with one camera, that moves around the object

The comment I replied to mentioned a jumping whale, not exactly a still scene...

That was the Matrix bullet time shot.

https://www.youtube.com/watch?v=ibwTDmxsXuI

Yes, just like this but with an array of quadcopters that auto-track a target.

Wasn't this invented by Disney in the 50s? http://en.wikipedia.org/wiki/Circle-Vision_360%C2%B0

I was just thinking about how the idea has been around a long time. Even back in the early 2000s I was working on a project that used a camera on a special tripod that would let you take pictures in 360 degrees and stitch them together with some software in post (quicktime I think?).

But I think the neat thing is using readily-available GoPro cameras and stitching the video together quickly. That takes a lot of processing power to do on the fly.

QuickTimeVR Toolkit. It was painfully slow on my Quadra and the stitching usually had to manually corrected by nudging the images around (didn't help that this was pre-digital cameras and errors in slide registration and scanning compounded) but I thought it was close to magic - even if it was running in MPW ;-)

I'm curious how they are planning to record audio. Binaural sound can give a sense of presence even without video. The common approach to recording it is by placing microphones in your own or a fake heads ears and that won't work in VR. It'd be interesting to see an array of microphone elements perhaps in a cleverly shaped mold to go along with this.

Each of those 16 GoPros have microphones, so I think you could get great results by processing all of those audio signals.

The best idea-0 approach would be to record Ambisonics with a tetrahedral array, and ship B format or a derivative to the client.

In theory that could be processed just-in-time to produce an appropriate stereo image based on head tracking.

http://en.wikipedia.org/wiki/Ambisonics

In practice the computational overhead would likely be on par with graphics processing and you might want an APU :)

I wonder how the vertical FOV is - will you be able to look up and look straight down? Seems like this is optimized for horizontal 360.

I wonder if this is what gets VR to take off. Seems like Oculus will be way too expensive for practical use. Once people start getting content out there all you'll need is a $20 cardboard kit to take advantage of that.

You could have several of these say at a music festival hanging around and watch them from your house at your leisure without having to deal with heat and crowds. And worse you go see an artist on one stage and realize they're a dud and you have to now travel to another stage. With this, boom, you can switch in an instant.

VR has been "around the corner" for 30 years. At some point Moore's law will give enough resolution and speed that images will seem real. I thought Oculus had reached that point, but I havent seen a demo yet.

Wait a little longer and off the shelf solutions approach that point too.

The big risk to the 3D / VR market is generally the possibility that there is a percentage of the population that just isn't "wired" for it, i.e. they can't actually experience immersive 3D without physical revulsion from their sensory experiences not lining up.

When you start designing a product for only N% of possible consumers, you're in trouble if N isn't large enough.

This is actually my biggest fear.

I am counting the days till the commercial release of the Rift and haven't tried any of the units yet. I got motion sickness from Mirror's Edge in < 5min. and have difficulty playing other games in tight indoor spaces or games I can't crank the FOV up on.

If we finally get awesome, immersive technology, and I have a genetic trait that makes me unable to use it without getting ill, I will be truly depressed. This level of VR immersion will allow a whole new range of sensory experiences and I want to be a part of that from the beginning.

I get eyestrain/headache everytime I see a 3D movie in the cinema, and I believe that I'm not the only one experiencing this.

But this issue affecting N% of possible costumers certainly hasn't stopped 3D movies from succeeding at the box office. (I guess N would be similar for VR.)

This rig isn't the answer. Content creators are already using better rigs to experiment with creating immersive experiences.

Using Cardboard is disarmingly cool the first time you see it, but it's incredibly clunky and not immersive enough. And over time they literally fall apart.

The Samsung Gear VR is a much more likely candidate to drive mass adoption. It's higher quality than both the cardboard and the Oculus. If you already have a Samsung phone it's a small investment to get a great VR experience.

I think LightField camera technology is ultimatly going to win the VR camera wars, it provides much better depth information. If you havn't seen the otoy tech yet it is worth checking out...

http://home.otoy.com/otoy-demonstrates-first-ever-light-fiel...

Lightfields don't lose because of how they look, it's the lack of compression and difficulty in capturing them for real scenes that have held them back.

The apartment scene on the linked page was about 40MB compressed, IIRC. True that was a static scene, and admittedly once you start doing 30 FPS (perhaps no need for 90+ FPS that the HMD's require, because head movement can be rendered independantly of animation), you're up to 70GB/minute. Lots, for sure, but not out of the question.

It's not out of the question, but it's back to distribution by physical media and the only unequivocal technical advantage is in the rendering of reflections and refractions.

This is really exciting. I came up with the same design last year and have been really wanting to make a rig like this. My gym had treadmills with screens, and you could select a virtual hike through places like Oregon and New Zealand. So I really want to capture some of these hikes with this rig, and replay them while walking or running on an Omni [1]. I'm really excited about the potential for VR to turn exercise into an enjoyable experience, instead of a chore.

And one day I would love to travel around the world with a team and film a documentary like Samsara. [2]

[1] https://www.kickstarter.com/projects/1944625487/omni-move-na... [2] http://www.imdb.com/title/tt0770802/

The most realistic virtual reality I have ever participated in was the Duke immersive Virtual Environment (DiVE). There is a program that simulates a kitchen. There are cereal boxes, silverware, etc that you can pick up and even throw across the room. When I opened the refrigerator door, I instinctively moved my body out of the way. The crazy thing is that it's not even the best tech out there. The resolution was crappy and the physics were a little bit off. But if you stopped actively paying attention to the details, even for just a moment, it was enough to make some part of your brain think it's real. VR doesn't have to much better (if at all) in order to be really immersive.

If you're at Duke or somewhere nearby I highly recommend checking it out. They have visiting hours fairly frequently. http://virtualreality.duke.edu/

How did it compare to Oculus Rift? I've never experienced either but I'm curious....

I've tried the Oculus DK1 but not the DK2. They are similar in terms of resolution and tracking accuracy. There are two main differences. The DiVE is a cube-shaped room with six projected walls. So you can fit multiple people into the simulation at once (only one person gets head tracking and the proper perspective). The DiVE also had a "wand" which you could use to interact with your environment. I would bet there are similar peripherals for use with an Oculus Rift, but I haven't tried them.

Next up: Street View 3D?

I used to think VR was mostly for gaming, but with this, and expeditions that they demoed, I'm starting to get really excited about VR apps for the masses.

That'll be awesome! Walking on any street of the world, just as if you were there? Why not!

You'll need some fancy ML/CV algos to take photo-only data from this to turn it into 3D environments, but Google could already create Street View in 3D with their self driving car data their collecting using their cloud point laser scanners.

https://gigaom.com/2014/12/18/google-maps-for-android-has-a-...

It appears that someone has already done something like this [0]. In 2013 during 30C3 in Hamburg, someone bought in a bicycle which you used to bike through cities virtually using an Oculus rift [1]. Unfortunately I couldn't find any extra information about that project, but I do seem to remember that they had to cache everything locally because Google blocked them after 10000 API calls a day.

[0] https://github.com/troffmo5/OculusStreetView [1] https://instagram.com/p/ibU1IXFhCz/

Did anyone else immediately think of this?

http://digital--underground.tripod.com/id9.htm

This is a really fun hack that really did make me smile and laugh... but I do wonder what the sensor/processing/display latency is like on hardware/software that wasn't originally designed to do this.

I once played with an early Oculus, it basically amounted to something that wasn't really succeeding at tricking my senses other than giving me a headache and making me want to vomit at the same time.

I'm sure similar things were said about early cinema though.

Then there's this: https://www.panono.com/#/en/home

This rig does stereoscopic shots from two different perspectives at once, which the Panono does not. That's just a flat still shot camera(s).

Looking at the picture immediately makes me think of it as an oversized crown or tiara you're supposed to wear on your head. Sort of a "Glass 3.0".

The "Camera Crown" is an entirely impractical idea of course (that is, not the actual idea), but that does lead me to wonder how it's meant to be moved through spaces, if at all.

I expect these will be probably hard mounted to moving surfaces during sporting events.

The event that comes to mind is obviously motor racing, specifically Nascar. I think being able to experience crashes in 3D stereoscopic would be a big selling point for people.

wooow, multi-billion-dollar-megacorporation striped together 16 expensive cameras with a piece of plastic :OO

so what, actually. They find ways to continue maximizing profit from their commercial platform. Which was at its most-cool when it was community-mostly.

I had a book of chindogu from at least 10 years ago. They had an invention that looked shockingly like the jump camera. Here is an image which shows a scan of a few of the images including the relivent one: http://www.tofugu.com/wp-content/uploads/2012/02/chindogu-un...

I find it kind of funny that Google is now using a system that was invented so long ago as a joke. Why not use any of the other system to capture 360 3d images? Why did Google decide 16 cameras was the right way to go?

The only reason I can think is you might get higher res images out of 16 cameras than with only a few and a hemispherical reflector or a motor, but is it really worth the extra cost and points of failure?

It's not for capturing 3D images.

It's for capturing 3D movies, which can then be stitched together and viewed immersively. There's similar technology used in "Circle 360" theaters in Disneyworld (EPCOT) and other places, but the cost has generally been in the content-production side of things.

Cost is still in the content-production side, but by basing it on commercial off-the-shelf components, they're bringing it closer to the masses.

360Fly has a single 360 camera (shipping this year). Much more consumer friendly but similar concept. https://360fly.com/

Has anyone though through doing this with binocular vision in all directions? just thinking about the geometry, I can't thing of a way to do it without a truly egregious amount of sensors.

I wonder if you couldn't just display the same video to each eye, offset by 10 degrees or so.

So, if you're at an event or something (like sxsw) do you wear this as a hat? or have a large pole... or how does filming with this actually work?

Mounted on a tripod like any other camera system. Most people will view the content, few will record (16 cameras is a lot of money/hassle/data).

I guess you havnt used a GearVR yet then.

GearVR is a VR viewer and this is a VR capture system, two different things. Hardware wise it competes with some indie brackets that similarly fit an array of GoPro cameras:

http://www.video-stitch.com/camera-360/

All mount to a tripod. GoPro is even coming out with their own:

http://gopro.com/news/nick-woodman-talks-virtual-reality-and...

The clever part here is the software.

Far out, this is a nice experiment. I wonder, though, how viewers will be able to focus on something that close without additional lenses.

If they can do it in real-time, this can replace the LIDAR used in self-driving cars, which can reduce the cost and make them affordable.

LIDAR will almost always be a better option when you don't have particulate matter that obstructs the beam because you can measure the distances directly which vastly reduces latency. Round trip time to a remote server means you have already lost and also introduced some very nasty failure modes. This is the reason why valve uses a LIDAR like system for Lighthouse rather than try to do image processing like on the kinect.

Imagine with high enough bandwidth where you could outsource the cost of a taxi to remote operators like drone pilots.

Almost impossible to read that font on my machine.

edit: I'm on Linux/Firefox. I assume it has to do with what font sets I have installed.

Yeah, not sure why it has font weight of 100.

If you're going to make that kind of complaint, you really should at least mention what you're using to view the page.

I agree. Edited my comment.

People who have retina screens tend to forget that most of the world still doesn't have retina screens.

I suppose you're talking about the "Roboto" font in #737373. It's also hard to read with Chrome on Windows 7 and a standard 1920x1200 monitor. The same thing happened when Google Fi launched. Either they don't test enough on certain platforms or it's just that corporate design takes precedence over good readability on certain platforms.

I guess that it is a "Firefox issue". I have the same problem with Firefox under Windows 7 (2560x1440, newly calibrated screen). The font renders a little better in Chrome and in IE 11 it looks good.

How does it do stereoscopy?

Probably with some simple optical flow view morphing. There's enough data from all the cameras to enable creating a stereo image.

There is definitely overlap in the FOV of the cameras that are adjacent to each other (GoPros in 16:9 generally have about a 120 degree field of view, though specifics depend upon which GoPro models is being used), so I assume they timesync the individual videos and use that overlap information to determine relative depths of things.

In order to do Stereoscopy properly, you need the axes of the camera barrel to intersect having a set of cameras all radially pointed outward doesn't achieve this.

The end user never sees the raw video in this case so that's not a requirement -- they're generating lots of synthetic stereoscopic views for the user by post-processing the video and accounting for depth.

I'll be interested to see what data they're sending to the video player to provide both stereoscopy and 360-panoramic video at the same time. The methods I can think of offhand require real-time pixel-shuffling to assemble the proper view during playback but maybe they figured out something else.

I'm guessing it's planes all the way down.

so, based on the first image, I need to press 16 buttons to start capturing, and take 16 sd cards out before I can view them?

GoPro cameras can communicate over wifi, so probably not.

my hero 3 drains its battery super fast when on wifi

> "GoPro's Jump-ready 360 camera array uses HERO4 camera modules and allows all 16 cameras to act as one. It makes camera syncing easy, and includes features like shared settings and frame-level synchronization"

No more details than that, but GoPros have an accessory port in the back that makes it possible to control and I'd assume they're using that. You can even supply power through this port so in theory you could add expansion battery packs to this rig and keep the cameras charged.

GoPro's can be controlled by a separate remote control (http://shop.gopro.com/accessories/smart-remote/ARMTE-002.htm...) which can signal up to 50 cameras at the same time. You can also access SD cards via USB without needing to remove them (as well as supply power). So google could do all this with no hardware mods to the cameras. From my test, GoPro's remote typically syncs cameras in under 5ms but sometimes it can be longer. Having a time offset between cameras can be dealt with in software in many cases.

Is it April 1st already?

Maybe they could write white on white and then it would finally be completely unreadable, nice design as usual Google

this might just "jump start" VR (pornos)

Ok sorry couldn't let this word pun slip :-p

Did anyone see this coming? Quite a successful secret by Google.

yes, vihart has been working on it for a while:

http://elevr.com/elevrant-camera-circles-for-stereo-spherica...

I don't understand why this comment is so heavily downvoted. I also was surprised how any information about google working on something like this has never appeared in my view.

This was the 2nd comment here at the beginning. Maybe there's tendency to downvote sibling comments? Also in the first version I had a grammar mistake. I wrote "Did anyone saw this coming?"

> Our 3D alignment approach creates a beautiful, seamless panorama, so you you won't see borders where cameras are spliced together.

Ironic how there's a seam in this sentence.

Reply on news.ycombinator.com