HDR-NeRF: High Dynamic Range Neural Radiance Fields | Hacker News Reader

HDR-NeRF: High Dynamic Range Neural Radiance Fields

xhuangcv.github.io

151 pointsPaulHoule36 comments

Every few months I see new mind blowing NeRF results, but I wonder if there is a usable project that I could try it myself. Be it open source project, or commercial product.

Also I'm not quite sure what time scales are we talking about, with turning bunch of images to nerf model, and then using that model to generate novel views.. hours on powerful gpu? minutes on cpu?

Check out https://github.com/NVlabs/instant-ngp.

Trains a nerf in a couple of seconds.

I went on a trip to Italy last year and took a video while walking around Michelangelo’s David. Even with that relatively poor image quality this let me turn it into a pretty high quality nerf.

Nerf studio

https://docs.nerf.studio/en/latest/

Does this actually add much more than just taking the original images with an 'HDR' mode on a camera?

It lets you synthesize views that you didn’t take with the camera.

I am into making stereograms and this sort of technology would help, for instance maybe the separation of my two cameras is more or less than I wanted so I could make up the views I really wanted.

NeRFs aren't limited to small camera orbits around one perspective of a scene either.

https://jonbarron.info/zipnerf/

They didn't turn HDR mode on, that's the novel aspect of the research. They took SDR photos with different positions and exposure levels, and then merged them into the NeRF.

Before modern sensors with huge dynamic range for a single photo, the "old school" way of getting HDR was to put your camera on a tripod, and take 5+ shots with different exposure levels. This can then be merged in software, but obviously this only works on static scenes and a static camera position.

The HDR-NeRF model lets you move the camera around, you don't need a tripod.

PS: Something I find hilarious is that the paper talks about HDR and shows "HDR-SDR" comparisons... but both pictures are SDR JPG images. It's pathetic how the PC world is pathologically unable to uplift imaging standards while the TV and Camera industries with far smaller R&D budgets are casually sailing past us into the future.

But what advantage does not turning HDR mode on give vs doing so is the question. I guess you could be trying to generate a NeRF with a really old smart phone?

Use today could be in turning older non-HDR images into HDR NeRFs, but also as a foundation that somebody else will build something completely new on top of. Who knows what that may be.

Right now the initial hot industrial application for NeRF's are recovering radiance fields from ordinary video footage. You can have someone just walk around with an ordinary camera or phone, just making sure they have decent coverage. Then back in the studio after a bunch of crunching you get a 3d virtual set you can position a camera in arbitrarily. The quality of this is considerably better than previous approaches that were based on recovering 3d meshes.

The work in the original post can now extend that process to capturing HDR data, which should enable better quality when compositing with other footage or rendered elements. And you don't need some special HDR video camera, just an ordinary one can sample the same thing.

AFAIK other NeRF methods don't let you use HDR data, just LDR images, so you would have to pre-process your HDR photos (tone-mapping) and you wouldn't be able to control exposure afterwards as previous necessary information for doing so would not be present when inferring novel views.

HDR-NeRF allows you to control both exposure and position when synthetizing new images. Your intuition is on the point though, with enough data/compute you could build a method that uses the photos with full dynamic range information from a HDR camera instead of "stitching" LDR photos of varying exposures like this method does.

Perhaps it's possible to recover overblown images from video for forensic purposes. I'd be interested in seeing some results.

Your PS reminds me of the old ads for those Matrox 30bpp graphics cards, which ‘showed how good these cards looked’ compared with standard graphics cards. Which we watched, obviously, on a computer with a standard graphics card.

Also the TV salesman telling me that a 100Hz panel would look jerky, and showing me a video on a 200Hz screen that allegedly illustrated the difference.

I thought this was essentially how "computational photography" worked, like in the Google Pixel photo app: The camera takes several pictures with different exposure, maybe even from different lenses, and an AI model merges them.

It is. Source: I built the ProHDR app that iOS copied

I see these videos on YT, "walking through Tokyo 4K" stuff like that. It would be cool to have something that can watch the video and generate a street level map with depth of what it saw. You can then go through with a camera head level and navigate it. Can be done but yeah.

The technology would also be killer in real estate markets. Imagine being able to take a video and give potential tenants the ability to see the apartment/house/etc in detail.

Probably easiest solution is a 360 video camera since YT allows you to view them... but not sure how good of quality/distortions.

You can already go on zillow and see 3d walk through by jumping from one spherical panorama to another. They even make 3d models out of it.

I cynically think it wouldn't be a killer, because real estate agents prefer to have photos carefully made with fish-eye lens that makes rooms look bigger.

Check out zip nerf which seems to do exactly that https://jonbarron.info/zipnerf/

looks amazing, my naive take is ML is a brute force way to do something, not guaranteed to be correct but close enough

on the other hand, I'd be curious how much worse a procedural way to do frame by frame analysis would be

Should have the label 2022

I feel smarter just reading the title

How do they decide where to use the little letters in these acronyms? Seems like it should either be HDR-NRF or something like HiDR-NeRF. Come on folks. Naming consistency, please.

NeRF was already standardized in the literature by people who are not these authors [0] and HDR is also already standardized by people who are not these authors [1]. Go with the flow unless you have a very good reason not to.

HDR is probably HDR and not some other clever name because there was no obvious clever name with an easy pronunciation. Whereas NRF is just so close to NeRF -- an actual word people are familiar with -- it's begging to be pronounced that way.

[0] https://arxiv.org/abs/2003.08934

[1] https://en.wikipedia.org/wiki/High_dynamic_range

Doesn't the vowel for e in "neural" match the e in in Nerf?

Yes, the vowel matches, but you could imagine the original authors had chosen NRF, pronounced "en ar eff". But the fact that the e from Neural was there, and the word nerf was there, just a perfect alignment.

TIL some people pronounce "neural" as /nɝəl/ (ner-al) instead of /njʊɹəl/ (new-ral).

aka its NeRF or nothing

> HDR is probably HDR and not some other clever name because there was no obvious clever name with an easy pronunciation.

Wouldn't matter, people would complain anyway. HDRs gonna H8.

I will complain because "HDR" is often usually used to refer to the process of /removing/ HDRness from data, ie, tone-mapping, so you can display it on an SDR display. Or so you can make one of those /r/shittyHDR images that looks like it's been deep fried.

This one looks like it's real though.

Tone mapping was all the rage 10-20 years ago. There was even a name for it (the X look) but I forgot which unfortunate photographer's name was associated with it.

> Tone mapping

For anyone else wondering:

> Tone mapping is a technique used in image processing and computer graphics to map one set of colors to another to approximate the appearance of high-dynamic-range images in a medium that has a more limited dynamic range. Print-outs, CRT or LCD monitors, and projectors all have a limited dynamic range that is inadequate to reproduce the full range of light intensities present in natural scenes. Tone mapping addresses the problem of strong contrast reduction from the scene radiance to the displayable range while preserving the image details and color appearance important to appreciate the original scene content.

> […]

> The goals of tone mapping can be differently stated depending on the particular application. In some cases producing just aesthetically pleasing images is the main goal, while other applications might emphasize reproducing as many image details as possible, or maximizing the image contrast. The goal in realistic rendering applications might be to obtain a perceptual match between a real scene and a displayed image even though the display device is not able to reproduce the full range of luminance values.

https://en.wikipedia.org/wiki/Tone_mapping

More importantly they missed the opportunity to call it NeRF-HDR, pronounced Nerf Herder.

I personally appreciate abbreviations that easily roll of the tongue and seem like words on their own. I believe it's important to be able to talk fluently about these things, to better relate to and understand them.

Reply on news.ycombinator.com