Scalable Large Scene Neural View Synthesis | Hacker News Reader

Scalable Large Scene Neural View Synthesis

waymo.com

64 pointsJyaif12 comments

This is using Neural Radiance Fields (aka NeRF) that first appeared about two years ago: https://www.matthewtancik.com/nerf

Originally, it took more than 1 GPU-week to train a single static scene. After multiple rounds of algorithmic optimizations, it's now practical to train & render whole city blocks, as seen in the Waymo demo.

The best part of it, is the answer to the classic question: "how many polygons?". Because the answer is zero.

These guys train representations for some scenes in seconds:

https://nvlabs.github.io/instant-ngp/

Hey, I used to go to that AMC theater!

NeRF is one of the big "holy sh*t" moments for me in recent years. I've always thought using polygons as our only way to represent non-trivial 3d shapes felt awfully inelegant, like using raster images instead of vector graphics.

Definitely recommend checking out the HN discussion of the original NeRF paper[0]

[0] https://news.ycombinator.com/item?id=22637721

> If you give it a bunch of photos of a scene from different angles, this machine learning method lets you see angles that did not exist in the original set.

Yeah wasn't sure what I was seeing.

If NERF interest you, don't miss NVidia last month paper, Instant Neural Graphics Primitive : https://news.ycombinator.com/item?id=29959700 which helps bridge the gap between traditional voxel maps, and neural network representation.

Imagine matching the output of the block nerf + GPS data combined with a live camera view to get ultra-high positioning precision... Or to use changes over time in 2 views to detect anomalies or maintenance heuristics... So many potentially interesting use cases

I wish a day comes when the compute at inference is more optimized than the raw camera sensor processing

Is this too much to ask for? :/

Actually, determining camera position is done quite differently and more efficiently: https://en.wikipedia.org/wiki/Simultaneous_localization_and_...

(I spent about a year writing an optimiser for camera localisation from multiple views)

What is the size of the model in question?

UPD: found in arxiv: 0.25M parameters (e.g. ~1MB) per block; demo video used 16 blocks, so 16MB total for Mission Bay.

"An intern"

So this is a speculative side project that isn't integrated into the main flow of work going on at Waymo. That suggests to me that there isn't anything as technical and forward thinking as this going on in the core teams. If they did they would show it off because clearly anything flashy they are quite eager to show off.

They appear to be building trains with wheels, pretty uninspiring.

The intern in question is Matthew Tancik, who is one of the primary driving forces behind Neural Radiance Fields. It is one of those cases, where the employer (Waymo) is lucky to have such an intern.

I can't comment on the second part of the message, because I don't know what I am allowed to say (I work for Google, but not Waymo).

As a PhD student alongside Matt, it’s not any intern. Ph.D. internships are much more scoped to specific lines of work and are a way to potentially fund promising researchers for the long term.

In fact, Berkeley AI Research has a built-in way for companies to fund BAIR research and to bring on BAIR Ph.D. students as interns: https://bcommons.berkeley.edu/home

Reply on news.ycombinator.com