HTP (highlight tone priority) and DUAL_ISO? OK? No-no?

Started by l_d_allan, September 21, 2013, 08:56:47 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

luxluminis

Audionut, thanks for the Physics Class. I appreciate that. But still... perhaps i did not make clear that i stopped believing my Canon Metering System 3 years ago. I first started to work with UNI WB and in the progress devleoped myself even better usable Picture Styles which were as accurate in determining an 98% Version of the true RAW Data but also allowing to judge colors on the camera screen. I basically use them even in documentary photography becasue they show much better real results than UNIWB.

When exposing RAW Files i usually always target 1.5 to 2 Stops (depends on the scne and if above grey or below grey dominates) above the Canon Evaluative Metering without blowing highlights when i use HTP.  The sensor of the 1 DX is capable to record highlights up to 3.5 Stops above Spot metered brightest Highlights. With 3.3 stops above the brightes highlights you are always on the safe side. But that is not very practical when it comes to fast changing situations. You can do this for still Photography, take your time and spot meter the brightest highlights, add 3.3 stops and you get a perfectly filled RAW Histogram as exposed to the right as it can get... I have developed my own technique to use it practically. You dont even have to check your histogram, if you are able to identify the brightest highlights. But this is easy. When not sure just use the spot meter and check which spot shows the shortest exposure time.

Practically you can set your camera to AV Mode, set the exposure compensation to + 3.3 and then check in your scene by pointing the spot meter on some highlights which give you the shortest shutter speeds. Then you push the Asterix Button to save this exposure and you reframe your final scene. I dont know if this works with other camers, but it works with the 1 DX as it can handle up to +/- 5 stops exposure compensation.

I was stunned when i saw that i could increase even the images of a SONY RX1 siginficantly with this technique. At Base ISO.

I defintely dont judge RAW files by the Exposure Meter. I know that even without HTP turned on, all the RAW Files exposed normally are extremely UNDEREXPOSED. If you open such files in RPP64, they look truly awful, a dull mess....  And the standard and hidden boost they get in Lightroom, etc. is responsible for a lot of loss of detail.

I was shocked when i saw this the first time, especially with a SONY sensor. I could not believe that the difference is that huge, even at BASE ISO. But it is.

So, if i understand you correct, you say, that i can achieve the same amount-of-photons-caught-result without HTP turned on and a shorter exposure (or lower SIO) at the same aperture. I am asking, because when i use the exact same exposure, i blow the highlights massively without HTP (and even have less shadow detail, at least as far as the clipping in RPP64 shows), while i have perfect highlights with HTP turned on and a tiny bit more details in the shadow. RPP64 at lest shows less clipping in the shadows. If you want,i can upload some screenshots out of RPP on my dropbox if this works with this forum. I could also send you the RAW files, if you want to check them with Raw Digger, so we then know which version catches more light, if there is one...

If this is what you are saying, shooting without HTP could theoretically mean, that i could use a lower ISO. But i am pretty sure, you cant determine the difference regarding Noise whatsoever. All i can provide right now is a comparison of a equally exposed, one without HTP and one with HTP at the same exposure settings. I must make additionally a comparison between with HTP activated and without (but then with different, shorter shutter speeds but again exposed to the right).

If what you say is correct, it would make no sense to use HTP in RAW photography, but there probably would still be good use for HTP in JPG photography. Anyway, thanks for your time.


luxluminis

All RAW filesare not treated for Sharpness or Noise Reduction in Lightroom or any other app. Just the standard predefined LR values. I only adjusted the Exposure to compensate for the ETTR "overexposure" which in fact is not a true overexposure, when you open the files in a real Raw Converter like RPP64 they are just perfectly exposed RAW Files. In Lightroom the ETTRed files look massively overexposed due to the standard push these converters perform to match the push the camera manufacturers in-camera and in their converters, so you have to compensate down 2 stops. Normale exposure will be stated as normal. All the other files are ETTR. And of course the JPGs are also normal.


ISO 12.800 with HTP activated opened in RPP64:

https://www.dropbox.com/s/nzlf67heypnfj4j/HTP12800MAXETTR-opened%20in%20RPP64-noadjustments-atall.png?dl=0

ISO 12.800 without HTP opened in RPP64:

https://www.dropbox.com/s/wvbly47fxsmu2dt/ISO12800-ETTR%20same%20values%20as%20with%20HTP%20version%20but%20without%20HTP%20active%20opened%20in%20RPP64.png?dl=0

HTP12.800 (on the left side) vs 12.800 without HTP, Shadows:

https://www.dropbox.com/s/5rob81af548mj7y/HTP12800vsNOHTP12800-shadows.png?dl=0

HTP12.800 (on the left side) vs 12.800 without HTP, Midtones:

https://www.dropbox.com/s/1psc4soo1izqzge/HTP12800vsNOHTP12800-midtones.png?dl=0

HTP12.800 (on the left side) vs 12.800 without HTP, Highlights:

https://www.dropbox.com/s/2xzguts5r23zve3/HTP12800vsNOHTP12800-highlights.png?dl=0

HTP12.800 (on the left side) vs HTP 1.250, Shadows:

https://www.dropbox.com/s/brvvy7taghd07bs/HTP12800MAXETTRvsHTP1250normallyexposed-shadows.png?dl=0

HTP12.800 (on the left side) vs HTP 1.250 , Midtones:

https://www.dropbox.com/s/qaeykjtmbeb7xt8/HTP12800MAXETTRvsHTP1250normallyexposed-midtones.png?dl=0

HTP12.800 (on the left side) vs HTP 1.250 , Highlights:

https://www.dropbox.com/s/ie4mi24vjz7be12/HTP12800MAXETTRvsHTP1250normallyexposed-highlights.png?dl=0

And to show the mess, the normal exposure looks in a true RAW Converter, hefre the normally exposed ISO 12800 pened in RPP64. Lightroom would show you a normally exposed file, but the reality is this one:

https://www.dropbox.com/s/4lqzk1nz9q2hd03/Normal%20Exposure%20ISO%2012800%20opened%20in%20RPP64.png?dl=0

And finally, the proof why you should always shoot RAW...

JPGs ISO 12.800, left is without In Camera Noise Reduction, right is one with LOW and one with NR set to High.

https://www.dropbox.com/s/e6en2z7qgdyon1x/JPG-ISO12800-normally-exposed-left-is-without-incam-NR-right-is-withincam-NR-Low.png?dl=0

https://www.dropbox.com/s/e6en2z7qgdyon1x/JPG-ISO12800-normally-exposed-left-is-without-incam-NR-right-is-withincam-NR-Low.png?dl=0

What i will add later, will be a comparison like i described it in my previous post. ETTR with HTP and ETTR without HTP, but at a different Exposure.

Please tell me, if you want the RAW Files, Audionut. I dont have Raw Digger.


Audionut

Quote from: luxluminis on November 02, 2015, 10:47:39 AMWhat i will add later, will be a comparison like i described it in my previous post. ETTR with HTP and ETTR without HTP, but at a different Exposure.

Pointless.  As soon as you adjust exposure you gain an advantage, or disadvantage, through the difference in noise in the light itself. 


In your comparisons, you forgot to compare those images to ISO 6400 with HTP OFF at the same exposure settings.  I'll give you a hint why this is important..............You're ISO 12800 with HTP ON images aren't really ISO 12800.


Look at these two images.
ISO 200 - HTP OFF


ISO 200 - HTP ON


The green, red and blue mountains in these images are the data.  To the right means higher signal, either through increased gain in the camera electronics, or increased scene luminance.  And obviously, further to the left means lower signal, either through less gain in the camera electronics, or decreased scene luminance.

With a scene that has a fixed amount of light (a fixed scene luminance), at the same exposure settings, please explain to me how the ISO 200 shot with HTP ON has a lower signal?

luxluminis

With a different Exposure i meant an exposure which does not oversaturate the highlights and does not clip them. HTP and NON HTP both at same ISO and aperture but with a different shutter speed, so both would be identically exposed to the right and show the same histogram (at least with my RAW preview on the camera display. It would be interesting to then see the exact values in RAW Digger.

I will also try the same with ISO 6400 for the NON HTP Version.

luxluminis

And of course just for showing of the others who perhaps do not know how dramatically MAX ETTR alters the RAW files here a comparison of 2 RAW files at ISO 12.800. Again, just exposure compensation in Lightroom for the ETTRed file. No sharpening, No Noise Reduction.

The left one is a "correct" exposure as the 1 DX exposes it without exposure compensation. The right and clean one is ETTRed by 2 stops to the right, dialed back down 2 stops in LR.

https://www.dropbox.com/s/96ssebp7k3mjf6e/Bildschirmfoto%202015-11-02%20um%2013.34.39.png?dl=0







luxluminis

Thanx for the link to this smart ETTR tool. Unfortuantely i canĀ“t use it without ML. I dont think that there will by an ML version for 1DX anytime soon respectively ever... There are not enough users... :)

To sum the discussion up:

You say, that it is pointless to use HTP in general when shooting RAW, because there wont be a visible difference when the files are corrected afterwards to a pleasant or correct exposure in any RAW Converter. HTP users think they pack more light on the sensor due to the ability of more positive exposure compensation without clipping, but in the end basically end up with the same histogram you can achieve one ISO step lower with less positive exposure compensation.

That seems to make sense. Altough my quick tests with 12800HTPT and 6400 without HTP indicate that with HTP 12.800 i can add 1/3 more light before the histogram starts clipping. i can add without HTP at ISO 6400 2/3 of a stop exposure compensation and with HTP and 12800 it is 2 stops. If i ignore the 1 stop due to the ISO shift of HTP, i still have 1/3 stop more headroom when exposing to the right with HTP. So that is somehow an irregularity in the sensible theory.

Audionut

Quote from: luxluminis on November 02, 2015, 01:25:18 PM
HTP and NON HTP both at same ISO and aperture but with a different shutter speed, so both would be identically exposed to the right

No........They would not.

I'm not interested in continuing this discussion until you understand the difference between exposure (the amount of light hitting the sensor) and ISO (post exposure gain), and how these functions affect different sources of noise in the images.

Some stuff for you to read.

http://www.clarkvision.com/articles/iso/
https://en.wikipedia.org/wiki/Shot_noise
http://theory.uchicago.edu/~ejm/pix/20d/tests/noise/index.html

There is no irregularity in the sensible theory, only in your understanding of the fundamental aspects of the issues at hand.

Quote from: luxluminis on November 02, 2015, 03:52:54 PM
Altough my quick tests with 12800HTPT and 6400 without HTP indicate that with HTP 12.800 i can add 1/3 more light before the histogram starts clipping.

The JPG based histogram on your fancy 1DX?
http://www.magiclantern.fm/forum/index.php?topic=12096.0#post_JPG

Go away and understand the basics first.

luxluminis

I assume you are not familiar with different concepts of altering the JPG preview histogram through Picture Style Settings and or White Balance to get a pretty accurate outlook on the real raw data. Otherwise your comment makes no sense. This is what i use on my "fancy 1DX". Ever heard of UNIWB? Just to name one possibility. I dont need to install 3rd party hacks to know how good my RAW exposure is or how far i can go with my ETTR. I nail it every time. And with nailing i mean after controling it in RPP64 or leveling it for further procedure in Lightroom. I dont read channel data. I process photos. I admit i am more the practical guy who earns his living with photography, not with nerdism. I need concepts that work consistently and without extra effort. If using HTP was meaningless, i can live with that. It did not cost me any image quality or more effort. But anyway, thanx for your time and your lessons about physics.

How about your basics regarding photography? Any work to show?

Perhaps you wont need it, as you have tools that think and do the math for you while ETTRing, but others might find interest in the practical hack i described in the other comment about how to nail always the MAX ETTR:

"Practically you can set your camera to AV Mode, set the exposure compensation to + 3.3 and then check in your scene by pointing the spot meter on some highlights which give you the shortest shutter speeds. Then you push the Asterix Button to save this exposure and you reframe your final scene." This delivers a MAX ETTRed exposure. Easy as that.

Guess what: You dont even have to "understand the basics" of light or physics to nail a perfect RAW exposure. And you dont need no 3rd party tools. And of course you dont need any of this for a great photo.... :)

Have fun!

Audionut

When it comes to composition skills, framing, background etc, I suck.

Want to know the difference between you and I?

I haven't attacked your ability to photograph.  You came here, dug up a two year old thread, attacked my understanding of the processes involved with HTP, with a clear lack of understanding in the basics needed to comprehend the subject.  When it became abundantly clear that you were completely ignoring the time I was investing in helping you to fully understand the processes involved, and gave you one last push towards understanding, you went and played the princess card.

Thumbs up mate.



dfort

QuoteI have a 1Dx, I'm awesome.

Fell off my chair laughing at that one.

Hey I operate a Flame, I'm a badass editor. (Though some may say that the movies I worked on suck.)

Getting back to the subject, it seems to me that what we'd really like is a camera that would never overexpose a photograph.

http://magiclantern.fm/forum/index.php?topic=15797.0
http://petapixel.com/2015/08/18/mit-created-a-camera-that-will-never-overexpose-a-photograph/#more-176916

I didn't go to MIT but I did go to Art Center College so this makes perfect sense to me:


DeafEyeJedi

Indeed, it does and Thanks again for yet another friendly reminder!

[emoji106]

btw, I enjoy reading your comprehensive responses @Audionut!
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

Audionut

Quote from: dfort on November 02, 2015, 08:37:41 PMFell off my chair laughing at that one.

I was having a staring contest with the personal text and blinked first.  ;D  No offense intended DEJ, I blink first all the time.  :)

Quote from: dfort on November 02, 2015, 08:37:41 PM
Hey I operate a Flame, I'm a badass editor. (Though some may say that the movies I worked on suck.)

I'm pretty sure you took some footage from a different movie, with the same characters, and edited it into the end of From Dusk Till Dawn.   :P

Quote from: dfort on November 02, 2015, 08:37:41 PM
Getting back to the subject, it seems to me that what we'd really like is a camera that would never overexpose a photograph.

That would help with one aspect of the noise problem.  Since by definition, a fully overexposed image = 100% noise.  ;)
The trick is being able to capture the amount of light that we want, without violating our preferred motion and DOF preferences.

Think of the sunny 16 rule.  If you want to have a narrower DOF then f/16, you must adjust the shutter to compensate, which adjusts the capture of the motion.  To defeat this adjustment of shutter, you can use an ND filter, but the ND filter reduces the amount of light captured, which means that the light being captured contains more noise.



In this image (representing the noise profile of a 6D @ ISO 100), the red line represents the noise level of the camera, the faint yellow line represents the maximum quality obtainable if all of the noise was only shot noise (Poisson noise), with the faint grey line representing the maximum quality obtainable if the noise was only electronic noise.

Left to right is darker to brighter, with top to bottom representing quality, higher meaning more quality.

The very right hand side side of the red line represents the saturation point of the sensor (at lowest ISO), or the saturation point of the downstream electronics when gain is applied (ISO is increased).  A 1EV difference represents a doubling or a halving.

So we can see that at the saturation point of the sensor in this ISO 100 profile, any noise that the camera produces is below the noise level contained in the light itself.  The light that we are capturing has more noise then the camera electronics are making.  If we look closely enough, it's not until we are four stops below this saturation point, that the noise the camera produces starts to become noticeable.  For reference, 18% luminance (midtone) is around 3.5 stops from saturation.

As we move further away from this saturation point, we can see that we are no longer shot noise limited, we start to become read noise limited.  In other words, the noise that the camera electronics is making is greater then the noise in the light.

Here is a noise profile from the same camera @ ISO 6400.


If you can understand my nerdy stuff above, two things should be immediately obvious.  The saturation point is no longer delivering eight point something EV of quality, only five point something EV of quality instead.  And that the image is shot noise limited to around six stops below the saturation point, instead of only four stops @ ISO 100.

Two things are happening here.  The first is that the amount of light we can capture @ ISO 6400 is less than the amount of light we can capture @ ISO 100, and hence, the very right hand side of the graph in the ISO 6400 graph is representing a smaller amount of light captured.  We can see this being represented by the maximum quality of the image being less than the ISO 100 graph.

The second is that as we increase ISO in current Canon cameras, the efficiency of the downstream electronics becomes greater (they produce less noise), which means that even though we capture less overall light, the light that we do capture is being represented more faithfully.  That is, the camera electronics don't affect the image quality until we are at some point further away from the saturation point.

edit:  We can also see the difference in quality for the same amount of light hitting the sensor between ISO 100 and ISO 6400.  The difference between ISO 100 and ISO 6400 being six stops.  For simplicity lets start at the 13EV line on the vertical scale.  Measuring back six stops brings the light level to 7EV on the vertical scale.  The 7EV on the vertical scale in the ISO 100 graph is the equal amount of light hitting the sensor at 13EV on the vertical scale in the ISO 6400 graph.  In other words, there's around 0.6EV more quality @ ISO 6400 then ISO 100 for that amount of light hitting the sensor.

This is how we can ETTR with ISO on Canon cameras, because ISO controls an aspect of the overall noise.  In Sony/Nikon cameras for instance, ISO does not adjust the amount of noise that the camera produces (mostly anyway), so using ISO on these cameras from an SNR standpoint is useless.  The only reason to use ISO on these cameras is so that you can actually see the rendered image on the back of the camera, and I know of quite a few people who are asking the powers to be to address this issue.

edit:  So as per my edit above, we can see that as we travel further then 6EV from the from the sensor saturation at ISO 100, read noise (camera electronic noise) becomes a significantly larger proportion of the total noise in the image.  Whereas, from this same amount of light hitting the sensor @ ISO 6400 (13EV on the vertical scale) the noise remains entirely shot noise dominated for a further 6EV (six stops).

So if you're still following along, really there are two aspects that need addressing.

Being able to capture more light.  More light is always more better from an SNR standpoint.  We shouldn't have to worry about deviating away from our required motion and DOF requirements to respect the saturation point of the sensor.  The only concern should be making sure we don't nuke our models out of existence with light.  When we can capture more light without having to reduce the shutter time, close the aperture or use an ND filter, we can extend the DR by pushing more signal further from the shot noise limit in the shadows.

Reducing electronic noise.  This means that whatever light we do capture, the amount of noise contained in the image is always shot noise dominated.

I didn't intend for this post to be so long, but there you go.