Hacker News Reader

What I've learned over the years with digital cameras -- you can't trust the preview image. The histogram tells much more of the story (if not the whole truth). My photographs have become much better (in a technical sense) after I learned to expose almost to the point of blowing highlights, making sure there is little blank space at the right (where the highlights are displayed) of the histogram as possible (hence the term Expose To The Right). As someone downthread stated, this improves the signal-to-noise ratio of the captured data. There's an unwritten assumption that the camera's built-in auto-exposure capability isn't sophisticated enough to expose the scene properly, and the photographer has to take control. This is often the case.

And speaking of SNR, there are a few tricks astro-photographers have been using for years that occasionally find their way into mainstream photography. One is image stacking, where you combine multiple photographs of the same scene (useful for landscapes if a little blur isn't a problem for you) into a single image. This pushes the shot noise (the random distribution of noise) down even further. I've gotten clean images out of cheap cameras, and even from cheap cell phone cameras (given enough images).

The other technique is dark frame subtraction, where you take an photo with the lens covered at the same ISO/shutter/aperture as the rest of the set. Digital sensors have readout noise and heat noise which tends to be very predicable. Subtracting this noise gives you a cleaner shot as well.

On a side note, I do wish when talking about ETTR and even digital photography in general that the term "overexpose" wasn't used when explaining the technique of overriding the cameras default metering with exposure compensation. Overexposure would be blowing the highlights, or at least the highlights of details that the photographer considers important (e.g. blowing the highlights of street lights is usually okay). The "over" portion makes it sound like the photographer is doing too much of something, when in reality ETTR is a technique of giving enough exposure, but not too much.

> One is image stacking

Since you seem to have experimented with this, I'd like to ask: don't you need a certain "minimum" time to capture the faintest of stars, because their signal otherwise wouldn't rise above the noise floor? I've been curious about this with stacking in atrophotography for a while.

> The other technique is dark frame subtraction, where you take an photo with the lens covered at the same ISO/shutter/aperture as the rest of the set. Digital sensors have readout noise and heat noise which tends to be very predicable. Subtracting this noise gives you a cleaner shot as well.

For long exposures this heat noise becomes more problematic: long exposures heat up the sensor quite a bit. Have you tried building your own customised electric coolbox to keep temperatures down? It can be a fun project!

I actually used this in a different context: during long Skype sessions at home with my people abroad, my phone heats up a lot and the video would start stutter. Putting it on a cooling block actually fixes that. This had more to do with the system not having to scale back CPU performance to let the phone cool down, but a side-effect according to people on the other end of the line, was that the image quality improved too

> don't you need a certain "minimum" time to capture the faintest of stars, because their signal otherwise wouldn't rise above the noise floor?

I'm not an expert or even very knowledgeable on image stacking, but I'll share my experience.

I've taken a stacked picture of the Orion constellation. 15 exposures at 5 seconds each, and combined them to get this photo:

https://www.flickr.com/photos/abliskovsky/31756623694/

In none of the individual exposures is the Orion Nebula visible, but in this photo it's pretty clear. So in the end I'm fairly sure that your statement is true, but the signal required is very, very faint.

Do you know of any good resources for "getting into" image stacking for a complete newbie? 2 years ago I tried stacking images I took of Jupiter in my 16" dob (taken with a Canon DSLR), but it was terrible and I'm not confident I used the app I found (I think it was AstroStack..) for image stacking correctly.

I don't have any particular advice. I searched the web for astrophotography stacking and used some software I don't remember the name of to do it. Sorry I can't be more helpful!

RegiStax is popular for solar and planetary imaging. I've used it several times on H-alpha solar photography. It has a fairly well defined workflow and there are lots of video tutorials on how to use it.

Awesome, I will check it out. Thanks! Also looks like it runs under Wine on Linux, which is good news for me!

Nice result!

Ok, just for fun: I'll try to make a quick back-of-an-envelope calculation and see if we can make an educated guess...

Let's use the old Sunny 16 rule, and its equivalent for moonlight, which also gives us an estimate for difference in light between day and night[0]:

> Daylight (sunny day): Correct exposure for this case is given by the Sunny 16 Rule: 1 / ISO [seconds] @ f/16. So at ISO 100, a typical exposure is 1/125 @ f/16 (that's 1/3 stop less exposure than the rule calls for, but it's the closest standard shutter speed.)

> Full moon: to get an equivalent exposure at night with a full moon, the rule is: 1 / ISO [days] @ f/4. That's right, DAYS. For ISO 100, that means 1/100 of a day @ f/4.

> That works out to something like 14.7 minutes; I just round it up to 15 minutes. So 15 minutes @ f/4, or FOUR HOURS @ f/16.

> That's 21 stops difference from sunlight to moonlight.

So on a sunny day you need f/16, 100 ISO, shutter speed 1/100.

flickr says you shot at f/5.6, unknown ISO (I'll go with quasi-conservative 6400 ISO), 5 seconds.

Aperture increased by 1.5 stops

ISO increase by 6 stops

500x longer shutter time => log2(500) -> 2^9 = 512 => bit less than 9 stops.

So that's between 16-17 stops of extra light sensitivity

That's five stops short of the 21 stops mentioned above. However, we're not aiming for a properly exposed single picture, we're aiming for minimum exposure to capture a signal!

Even the first digital SLRs had a dynamic range of five to six stops, so theoretically should be able to capture something in the shadow regions. Nowadays ten stops is about the norm I believe, with some of the better cameras going up to twelve. So that should clearly be above the minimum signal required

Also, the moonlight-rule above is very conservative: it's about exposing a scene in moonlight, not about capturing stars. Stars are light sources themselves, and probably brighter than objects reflecting moon light, even in a full moon. Also, the rule-of-thumb was based on film, which actually suffered from something called "reciprocity failure"[1] so it might exaggerate the required exposure.

So yes, this should work out fine, and as your picture shows it clearly does :)

[0] https://www.flickr.com/groups/11947580@N00/discuss/721576207...

[1] https://en.wikipedia.org/wiki/Reciprocity_(photography)#Reci...

I don't actually know the answer to your question. My experience with image stacking is daytime landscape photography. I've found that stacking can greatly improve the shadow detail. The ETTR technique gives nice blue skies, and restrains other highlights, but naturally, with shorter exposure times, the darker portions of the scene get less exposure than they would if you allowed the skies to blow out.

I'm only guessing, but there must be a minimum time for the exposure. Stacking won't change that. My assumption (and I'm not a statistician) is that the noise floor moves lower with stacking. Taking multiple images and stacking them is basically taking multiple samples. The random (Poisson shot noise) is averaged out across multiple images while the signal (the star) is a constant across the images.

As you can see from the linked article, there's an amazing amount of detail that can be recovered from the shadows if you know how to get to it. In a dark sky environment, you'd be limited to the camera's sensitivity (the analog gain applied using the ISO setting), but in the city environment, almost certainly the sky light is the limiting factor.

I have not experimented with custom cooling. Aside from some moon photos and photos of the recent partial (in my area) solar eclipse, I'll leave astrophotography to others.

> The histogram tells much more of the story (if not the whole truth).

I feel that this is worth expanding on.

Because a raw file isn't actually image data and instead just straight readouts from the sensor, it doesn't make sense to show the histogram of raw data; the data has to be manipulated into a jpeg first. The histogram in the camera will show the lightness values for what the in-camera image processing computer does to convert the raw file into a jpeg, but the in-camera image processor is usually not as advanced as a desktop image processor. It's probably running older image processing algorithms and has to worry about battery life among other things.

So generally what this means is that if your camera is telling you that parts of your image are over or under exposed and therefore clipped, it might still be possible to recover those parts of the image in a more powerful image processor. It's also worthwhile, if you've got really old raw files, to go back and reprocess them with modern software. Raw file processing has come quite a long way in the past decade!

That is the right technique in general when overexposing would clip highlights. Here's that's not really a risk, so the other direction works better.

What article doesn't explain is the physics behind why ETTR captures more light data: our perception of light scales logarithmically, while a CCD captures it linearly. Each stop is a doubling (or halving) of the amount of light captured. CCDs work by increased voltage based on captured photons.

Imagine back to when digital photography just started, when we were stuck with 10-bit RAW images (before processing them to 8 bit JPGs). That's 1024 different values per channel, not a whole lot of room to begin with. Because we're converting a linear signal to a logarithmic one, that means that highest stop in the image takes up 512 of those values.

EDIT: As rightfully pointed out in a reply, 8-bit JPGs are not in linear colour space any more - so this logic does not apply there (however, JPG compression is more harsh on shadow details, since it's lossy, and it tries to move that loss of information to the parts of the image where you're less likely to notice, like the shadows, but we're digressing)

By that logic, the low light data should have a lot more banding than the highly exposed data. Try it with level tools on a RAW file and see for yourself.

So you can imagine that when the dynamic range of (part of) the scene that you want to capture is less than the dynamic range of your sensor, you want to be on the right edge of the histogram (all else being equal - if this causes motion blur due to long exposure it's pretty useless)

Caveat: this is potentially outdated knowledge, perhaps the analog/digital converter in modern cameras actually converts the voltage of a CCD to a (pseudo)logarithmic scale for the digital output, which might partially circumvent this issue (although the sensor is still linear). And nowadays many of the better cameras have 14-bit A/D converters - the usefulness of which also largely depends on the quality of the sensor - so there's a lot more detail in the shadows than before.

You should write an article on the subject. Seems there are a lot of interested people in the process and you've got some domain specific knowledge to lay it out nicely.

I agree. I’d love to read an article about this!

Thanks for the compliment, but I doubt I'll do better than the Wikipedia summary[0] or the original article suggesting the method from 2003, which is still really good[1]. The 2011 followup is also worth a read[2].

If you want a good website in general for explaining photography techniques plus the physics behind it, I highly recommend Cambridge in Colour[3].

By the way, per the example of the second linked essay: if we try to capture 10 stops in 12 bit channels, the darkest stop only has 32 discrete available in the RAW file.

However, this might be somewhat less of an issue these days because most cameras have a really high resolution, and most images are resized for screen displays. So we can average out some banding (and image noise, for that matter).

Example: I have a 20 megapixel camera. A 1080p screens is just 2.1 megapixels. So even for a full-screen export, that's roughly ten "source pixels" per "output pixel". Which would mean that the output pixel, being an, could have 300 different values. The distribution won't be like that of course, but it still should help.

[0] https://en.wikipedia.org/wiki/Exposing_to_the_right

[1] https://web.archive.org/web/20150209012804/http://www.lumino...

[2] https://web.archive.org/web/20150213002736/http://www.lumino...

[3] http://www.cambridgeincolour.com

While the color depth of a JPG is 8-bits, it is not a linear 8 bits. The sRGB color space is gamma corrected (non-linear) and is roughly equivalent to taking the square root of a linear pixel value.

The same thing holds true for a processed RAW image as well.

Thanks for the reminder about JPGs, edited in!

When you say processed RAW, you mean after importing to image editing software, right? Which implies you wouldn't see much of a difference in posterisation in the high and low exposure areas

Right, after importing to image editing software.

Still, for a very long time the RAW file itself used to be linear in storing the signal. Perhaps not any more though.

It is still linear as far as I know. And the linearity is quite good, within a few percent. There is a community of astrophotographers interested in making photometric measurements with DSLRs (American Association of Variable Star Observers). I think the harder part of the problem is getting a standard candle for calibration of your DSLR DN values to actual SI quantities.

https://www.aavso.org/dslr-observing-manual

Huh, that sounds like a very interesting (but different) problem to tackle, given the vast differences between models.

I'm going to be pedantic and air a pet-peeve of mine. "Raw" isn't an acronym. It's just shorthand for raw image data. It's not even the file extension of most (any?) camera raws. Why does everyone insist on writing it in capital letters?

Never really thought about that. I think I subconsciously interpret RAW as only meaning "file of unprocessed image data from a camera", whereas raw has more possible meanings. I know this shouldn't matter, because it's never confusing in context what one would mean.

I guess there's also the visual "symmetry" with JPG or JPEG.

Probably because it's cased that way in camera menus (correctness notwithstanding).

Another possible reason not outlined the other replies you received to this comment - If I were to say "copy the raw file" someone less experienced may think "okay, take the fresh jpeg file and copy it", interpreting "raw" in a more conventional, non-technical sense. Capitalizing it emphasizes that it refers to the file format, even if not the actual file extension. Sure, I could say "copy the NEF file" or "copy the CRW/CR2 file", but that means I'd have to know whether the user was using Nikon or Canon. "RAW" lets the speaker eliminate ambiguity in non-technical contexts without needing to be so specific as naming the actual extension.

I'd rather see a comparison between two 30-second exposures at 1600 and 6400 than what was presented here. That would give a much better idea of how much detail you're losing from sensor gain.

If the scene has a lot of variation in brightness, then I'll always opt to save the highlights over the shadows. Shadow detail recovers remarkably well, while clipped highlights are ghastly.

ETTR is a real technique to maximize SNR. Underexposing and then boosting shadows is just a good way to guarantee that your image has no contrast, and comes out looking flat and grey all over.