Magic Lantern Forum

Developing Magic Lantern => Modules Development => Topic started by: l_d_allan on April 20, 2013, 02:11:30 AM

Title: Auto ETTR based on RAW histogram (ettr.mo)
Post by: l_d_allan on April 20, 2013, 02:11:30 AM
https://bitbucket.org/hudson/magic-lantern/src/tip/modules/ettr/README.rst?

Quote from: Original Post
Quote from: a1ex on April 17, 2013, 11:43:04 PM
Alright, now you have true RAW histogram and zebras. 5D2 only.

Very cool. Great work.

Would it be possible to have "Automatic ETTR" capability? It would perhaps select the maximum exposure that only blew out one of the RGBG channels. That would let ACR PV2012 do its magic at a near optimal level.

The photographer would obviously be responsible for not including a specular highlight, such as the sun or chrome, etc.

For my "use case" on panoramas, I'd pick a bright frame that excluded specular highlights, determine the suggested, automatic ETTR exposure from ML, and then set the camera on Manual with that exposure.
Title: Re: Auto ETTR based on RAW histogram?
Post by: Audionut on April 20, 2013, 03:11:55 AM
I don't see how this would be much faster then doing it in live view.  And what would ML adjust to increase exposure, shutter, aperture or ISO?  You might be happy with ISO, others with shutter etc.
Title: Re: Auto ETTR based on RAW histogram?
Post by: Marsu42 on April 21, 2013, 01:59:20 PM
Quote from: Audionut on April 20, 2013, 03:11:55 AM
And what would ML adjust to increase exposure, shutter, aperture or ISO?  You might be happy with ISO, others with shutter etc.

Imho longer shutter and maybe lower iso for handheld would be good since it doesn't change the picture/dof, the apex computation of these things is already there, that doesn't seem to be a problem to me.

But I thing auto-ettr is an absolute stellar idea since it is a bit of a fix for the major shortcoming of Canon's current crop sensor: low shadow resolution and noise. I often do bracketing not because of a high dr scene, but I want to ettr in raw but don't quite know how much is safe (after highlight recovery in acr), so this feature would save time and shutter cycles.
Title: Re: Auto ETTR based on RAW histogram?
Post by: garry23 on April 21, 2013, 04:27:40 PM
I have written this to hopefully help the broader ML community better understand some of the 'feature requests', in this case ETTR.

Obviously it is written from an 'IMO stance' as ETTR and other strategies to maximize dynamic range, or captured image data fidelity, are not universally agreed upon.

First, to maximize the captured data's fidelity for post processing I believe we need to try and accomplish several things with our exposures (other than ensure they are in focus etc): minimize noise, maximize S/N and capture the most tonal information on each sensor element (RGBG). However, trying to accomplish all these at once, for a real world scene, is near impossible.

For example, to minimize noise we should only shoot with a camera cooled to its lowest operating temperature, eg to minimize dark current noise. The longer we shoot and if we shoot on hot days this noise contribution will increase, just like entropy.

To maximize S/N we should seek to capture the maximum number of photons, and no more, ie achieve a Full Well situation. However, although we may be able to do this for a real scene, it will only be achieved in the few sensor elements in the brightest part of the scene, it a very small % of the overall statistics of the captured image. For instance the subject/focus of the scene may be in the mid tones or lower, ie not a specular highlight that is creating the Full Well situation.

I believe we all now know that DSLR cameras do not capture and process light like our eyes or film. The process is linear and thus this is why there is apparent merit in ETTR and bracketing strategies. That is trying to get the maximum tonal graduation into the capture, without 'blowing out' important data.

So far so good.

I think bracketing is not 'contentious' as we usually are on a tripod and at the base/lowest ISO, ie where we can guarantee that some of the sensor elements capturing the scene information we deem important will be at their Full Well level, albeit only a few %, unless we 'over bracket'.

I think the issue comes when we introduce the ISO factor, ie when doing a handheld bracketing set or seeking a handheld ETTR single exposure. In both cases we may need to increase the ISO to achieve a good shutter speed, eg the slowest bracket greater than 1/FL, say OR greater than 1/50, say.

I for one take a lot of handheld 3-brackets on my 5DMkIII and have confidence that increasing the ISO will not create too many issues in post processing. However, from my reading on sensors etc, I will not increase the ISO above about 1600-3200, as this will transition me from the region where the camera noise sources dominate to where the sensor limitation dominate, ie I'm just not capturing enough photons at high ISO. This transition will vary for each camera, but the bottom line is, that if we follow an ETTR strategy, there is an upper limit (ISO) we should all be aware of.

In conclusion, I believe ML is on the right track by giving the user choices to maximize DR and S/N wrt the scene, ie extended bracketing (although with my 5DMkIII this is less importance compared to my 50D) and a RAW LV histogram (a transformational feature).

Finally, IMHO, using all the ML features without understanding the camera-system's limitations could bring disappointment.
Title: Re: Auto ETTR based on RAW histogram?
Post by: l_d_allan on April 25, 2013, 03:48:51 PM
Quote from: Audionut on April 20, 2013, 03:11:55 AM
I don't see how this would be much faster then doing it in live view.

To me, the issue is that that histogram from 0-255 ends up being fairly low resolution when you really want to see what is going on from the equivalent of 245 to 255 illumination per channel.

A real nice-to-have capability oriented to ETTR would be a RAW-based histogram from 245-255 for just the 10x region where you were zoomed into. That would enable avoiding the specular highlights that are ok to be blown out.

QuoteAnd what would ML adjust to increase exposure, shutter, aperture or ISO?  You might be happy with ISO, others with shutter etc.

I would think I'd invariably be interested in shutter speed. At some point, the exposure time gets too long in dark scenes, and then you can use reciprocity to get an equivalent exposure.  For example, ML might calculate ISO 100, f8, and end up with 4 minutes. I might be willing to use ISO 400 and f5.6 to use 30 seconds (if I did the math correctly).
Title: Re: Auto ETTR based on RAW histogram?
Post by: garry23 on April 26, 2013, 12:32:36 AM
May I simply endorse the above and ask if it would be possible to factor in a user variable that allows the user to select the entire histogram or a fraction of it, from the 100% highlight end. It could be in stops or fractions of the full histogram.
Title: Re: Auto ETTR based on RAW histogram?
Post by: a1ex on April 26, 2013, 07:29:53 AM
You can just count the stops till overexposure on the histogram, or read the spotmeter, and then count the clicks on the shutter speed dial. So if your test picture is underexposed a bit, you can tell at a glance how many stops you have to adjust.
Title: Re: Auto ETTR based on RAW histogram?
Post by: garry23 on April 26, 2013, 03:29:55 PM
Alex

I suggested the above as I thought you had worked outa way to display a RAW histogram in LV, rather than 'just' in review mode.

In other words if a RAW histogram can be displayed in LV I would like to suggest a user option to display, say, the right hand highlight stop as a zoomed in histogram would be very useful

Title: Re: Auto ETTR based on RAW histogram?
Post by: a1ex on April 29, 2013, 02:00:41 PM
Here's a semi-automatic implementation: on the histogram, ML now shows by how many stops you can push the exposure to the right without clipping (or, with clipping the green channel only, if you prefer).

So, take a test picture, adjust the exposure by that number, and get the optimal SNR.

https://bitbucket.org/hudson/magic-lantern/commits/37bec2dc9ea3
Title: Re: Auto ETTR based on RAW histogram?
Post by: 1% on April 29, 2013, 04:31:24 PM
BTW: 5d2 method working for 600D but the alignment/pitch is off. I'm only getting red channel and zebra image is distorted. The state transition is the same.... I wish I knew how to verify that I have the correct reg because its not working on M and the transition is the same... maybe just a sync issue.
Title: Re: Auto ETTR based on RAW histogram?
Post by: a1ex on May 21, 2013, 10:12:16 PM
Got it working.

This will probably convince me to shoot in auto ETTR, because it exposes exactly the way I want.

Going to try it this weekend at a small local event.
Title: Re: Auto ETTR based on RAW histogram?
Post by: RenatoPhoto on May 21, 2013, 10:52:59 PM
Quote from: a1ex on May 21, 2013, 10:12:16 PM
Got it working.

This will probably convince me to shoot in auto ETTR, because it exposes exactly the way I want.

Going to try it this weekend at a small local event.

I just tried it.   Seems that it needs two or three pictures before it get its right. 
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: a1ex on May 21, 2013, 10:54:51 PM
If your picture is overexposed, it has no way to know by how much. So, start with an underexposed one and will get it right in one step.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: RenatoPhoto on May 21, 2013, 11:01:58 PM
Yea, that is what I though it was happening.  At first I could not get it to work but then I noticed small changed in exposure from one pic to the next!  I will take some pictures and see how good I can recover shadows and not over expose.

Also noticed that on the last build 5D3 I am getting bad frames on all RAW modes when I press zoom.  Every time, every 40 frames or so I get a few bad ones. 
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: RenatoPhoto on May 21, 2013, 11:06:28 PM
Quote from: a1ex on May 21, 2013, 10:54:51 PM
If your picture is overexposed, it has no way to know by how much. So, start with an underexposed one and will get it right in one step.
In big changes of light it take three steps.  When I go from dark ground to something with sky it takes three shots.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: a1ex on May 21, 2013, 11:07:55 PM
Tip: in the Histogram menu, set the EV indicator to ETTR hint. It will show you by how much it will change the exposure.

And even if you turn off Auto ETTR, you will still get the hint, so you can do the adjustment manually if you want.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: RenatoPhoto on May 21, 2013, 11:17:33 PM
Thats great.  Maybe the cam should take consecutive pics until the hint is near zero and then stops.  Although it does not always go to zero but say maybe +- 0.5 is good enough. 

I tried just pressing the button burst, but no adjustment was made.  Can it calculate fast enough to just burst three or four pics?
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: RenatoPhoto on May 21, 2013, 11:25:46 PM
Its great.  I tested the highlights and could recover them pretty well.  There was no apparent overblown areas in the highlights.

This is good for Canon, double the shutter counter=more spare parts.  :D

Congratulations!!!!
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: a1ex on May 21, 2013, 11:27:51 PM
It doesn't work in burst because you can't change settings in the middle of shooting. So... single pics only.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: 1% on May 21, 2013, 11:52:13 PM
Its really good. Much better than P mode. Has to take a pic first I guess unless you open LV?
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: a1ex on May 21, 2013, 11:57:39 PM
I didn't enable it for LV yet, but should work. Maybe hook it on histogram update?
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: 1% on May 22, 2013, 12:00:10 AM
Would go together.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: clint on May 22, 2013, 08:03:46 AM
IS this working on 60D yet?..... I know you guys have been busy giving the camera industry a bit of a shock..... anyway, I tried the RAW histogram with no success, perhaps I missed something obvious.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: a1ex on May 22, 2013, 08:25:12 AM
Nope, only 5D2, 5D3 and 6D for now. One has to port the photo mode raw support. See my porting guide here: http://www.magiclantern.fm/forum/index.php?topic=5240.msg32128#msg32128

Just got it working in LiveView. Now THAT is really usable (it's effectively reading my mind). This is how Canon should have implemented the metering in the first place.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: vertigopix on May 22, 2013, 08:32:01 AM
Hope that somebody will port it...
I haven't the skill to do this  :-[ but It's looking very interesting !
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: platu on May 23, 2013, 08:31:19 AM
I have been shooting using the Auto ETTR feature today and just have to say that it's amazing.  So much more advanced than Canon's own metering.  This deserves a lot more attention but I understand about the excitement of Raw video.  Great job Alex!
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: Audionut on May 23, 2013, 08:35:28 AM
It is very cool.  A little customization built in also so that you can tweak it to your liking.

I know quite a few people now who have been harping on about wanting something like this, rather then Canon's 12% grey.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: a1ex on May 23, 2013, 09:35:58 AM
Another tip: combine this with Post Deflicker from shoot menu. This one will save exposure compensation info in sidecar files, so when you develop the raw shots, all your pictures will appear equally exposed.

Also try it instead of bulb ramping: enable intervalometer, auto ETTR and post deflicker. In theory, this should give 100% flicker-free timelapse on the entire exposure range (from 1/8000 to 30 seconds). I'll try it this weekend hopefully.

Both methods use robust statistics (that percentile thingie from the old bulb ramping), so I expect very good results.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: ozcancelik on May 23, 2013, 09:58:59 AM
So we have accurate deflickering! I hope it works well. Are we gonna use your deflickering application? Or it doesn't need any deflicker touch in post?
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: X-RAY on May 23, 2013, 10:22:38 AM
This is really the next big feature...but as platu already noted, just doesn't get the same excitment as RAW. I think its because its not as comprehensible.

I've got a question on the RAW histogram. So as I understand, before that the histogram was created from a jpg-like presentation...so you didn't get an idea of the headroom which you are able to recover in the shadows / highlights. Is it right, that now in the RAW histogram it shows the whole range of information? That would be so much better. ^^


Some ideas on that feature:

Auto ETTR needs just a little bit of time, is it already optimized for speed? Maybe it would be helpful when there is a indicator that the Auto ETTR "has just set the values". Like a beep or something.

Any chance to get it working in other modes than M? Like C-Modes? It would be great having a mode reserved just for Auto ETTR.

It only alters the shutter speed and iso values. Perhaps it would be reasonable to set in the options which value should be calculated.
At the moment: Aperture uncalculated + iso / shutter calculated

Wouldn't it also be great to have the Auto ETTR on the *-Button (exposure) in M-Mode? Then you really don't need to toggle the feature when you want to use it or not.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: a1ex on May 23, 2013, 10:33:07 AM
For deflickering: my app is no longer needed, all you need to do is (for ufraw workflow):

rename 's/UFR$/ufraw/' *
ufraw-batch *.ufraw --out-type=jpeg


For XMP, I have no idea how to use it, I guess it's similar to the DNG video workflow.

Raw histogram is computed from 14-bit raw data trimmed to 12-bit (count the intervals, each bar is 1 stop).

Auto ETTR can be assigned to some shortcut, but I'm not sure which is the best way to do it. The * button is hard to detect, we can't tell the difference between it and half-shutter.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: Audionut on May 23, 2013, 10:33:49 AM
Quote from: X-RAY on May 23, 2013, 10:22:38 AM
I've got a question on the RAW histogram. So as I understand, before that the histogram was created from a jpg-like presentation...so you didn't get an idea of the headroom which you are able to recover in the shadows / highlights. Is it right, that now in the RAW histogram it shows the whole range of information? That would be so much better. ^^

Yes, it's based on RAW data, so it's accurate.  It's not let down by all the problems of trying to read a histogram based of processed JPG data.

Quote from: X-RAY on May 23, 2013, 10:22:38 AMAuto ETTR needs just a little bit of time, is it already optimized for speed? Maybe it would be helpful when there is a indicator that the Auto ETTR "has just set the values". Like a beep or something.

A beep would be nice.  It is fairly easy to tell when your settings have been changed though.

Quote from: a1ex on May 23, 2013, 10:33:07 AM
Auto ETTR can be assigned to some shortcut, but I'm not sure which is the best way to do it. The * button is hard to detect, we can't tell the difference between it and half-shutter.

M-Fn, DOF or AF-ON would be my choices for 5D3.

edit:  Actually, what about that silly rate button or the other useless one above it?
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: a1ex on May 23, 2013, 10:39:58 AM
From those buttons, RATE is the only one easy to intercept. For other cameras... SET and INFO/DISP can also be used (just like with expo presets).

DOF is also possible, but only with Canon lenses.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: Audionut on May 23, 2013, 10:44:54 AM
RATE would be the best pick then imo.  Afaik, the only decent use people have been getting out of it, is marking the first shot in a series.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: Audionut on May 23, 2013, 11:06:15 AM
Quote from: a1ex on May 23, 2013, 09:35:58 AM
Another tip: combine this with Post Deflicker from shoot menu. This one will save exposure compensation info in sidecar files, so when you develop the raw shots, all your pictures will appear equally exposed.

Oh wow, this works good  :)  I like the post calibration info on the screen also.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: X-RAY on May 23, 2013, 12:03:49 PM
But rate is only on 5D3. On 5D2 I think Set would be ok. But also the DOF-Button would also be perfect for me. I never use that.
INFO/DISP isn't good to reach.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: eyeland on May 23, 2013, 02:07:04 PM
Again, what a sweet implementation, I am sure that many more stills shooters will try out ML in the light of this!
Great that the non-video people get some love from ML as well, we wouldn't want them to feel left out :)
For me, being almost equally interested in video and stills these days, this is all so amazing...
+1 to assigning the rate button
Can't wait to try this out on a star timelapse
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: Simdub on May 23, 2013, 03:10:12 PM
Wow, very nice implementation  :). Thanks again ML Team! After some tries, I can assure you it works very well! Just have done an example here. I took 2 pictures of the same scene, one underexposed (aprx. 1 maybe 1.5 stop underexposed) and the other one I let ML's autoETTR make his job. Then I used Ligthroom 4 to adjust them (no noise reduction or sharpness increase). They have the same configuration (except exposition of course). So here are the results:

                shutter speed : 1/160 - iso : 200 - f/2.2                                   shutter speed : 1/60 - iso : 400 - f/2.2
(http://nsa34.casimages.com/img/2013/05/23/130523031412762430.jpg)   (http://nsa33.casimages.com/img/2013/05/23/130523031413491027.jpg)

100% crop:

(http://nsa33.casimages.com/img/2013/05/23/130523031412412118.jpg)   (http://nsa34.casimages.com/img/2013/05/23/130523031413170707.jpg)

Even if I knew benefits of ETTR I wasn't a big fan of it (Ok, I wasn't brave enough). But now, this features will obviously change my mind. All the shots I took were very concluant. Even if we have to took 2 pictures for one scene, I think this could be a game changer in exposure modes. So thank you again, I find this feature really helpfull ;)!

PS: second shot isn't as sharp as the first one, it's because I set ETTR's slowest shutter at 1/60 sec. You can change this and set it at 1/125 and so on for scenes where there are some mouvements!
PS2: Sorry for my bad english, I just wanted to share those results :).
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: 1% on May 23, 2013, 04:34:25 PM
BTW... for raw histo and all that:

(http://i39.tinypic.com/6egntf.jpg)
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: platu on May 24, 2013, 12:48:01 AM
Using Auto ETTR with silent pictures enabled, the maximum number of photos it will let me setup in the intervalometer is just a small fraction of the space I have available on my CF card.  I'm guessing that it is calculating the space available based on CR2 file sizes and hasn't been updated to account for the smaller file size of silent pics.  Am I understanding this correctly?
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: noisyboy on May 24, 2013, 01:01:12 AM
Sweet feature! Can't wait to try this. I would dare say that ML DSLR's (in particular the mkIII) is fast becoming one of the most all rounded and feature packed camera on the market.

Mad props!  8)
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: RenatoPhoto on May 24, 2013, 01:50:29 AM
Just tried a time lapse with ETTR enabled but did not work because I set Image Review = OFF (In Canon Menu) to save power.  The ETTR program did not adjust exposure and I did not know it until I reviewed the images in my pc. 

For now make sure Image Review = 2 seconds.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: platu on May 24, 2013, 02:53:03 AM
Also, post deflicker does not work when silent pictures is enabled.  For me, one of the most exciting prospects of ETTR is being able to use the combination of Auto ETTR,  post deflicker, and silent pics (to save my shutter) for timelaspes.  That is potentially the ultimate combo.  I realize that Auto ETTR was just released so it may already be part of Alex's plans for all I know.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: tkyisepic on May 24, 2013, 05:13:03 AM
Does this feature work with long exposure photography with ND filters?
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: JoshuOne on May 24, 2013, 10:12:33 AM
Greetings, I was hoping that someone could tell me how to implement the auto ETTR feature on the 500d, I am using the latest nightly and could not find the feature in the menus. Thanks so much to all the devs who work on this awesome software.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: RenatoPhoto on May 25, 2013, 11:11:20 PM
Quote from: JoshuOne on May 24, 2013, 10:12:33 AM
Greetings, I was hoping that someone could tell me how to implement the auto ETTR feature on the 500d, I am using the latest nightly and could not find the feature in the menus. Thanks so much to all the devs who work on this awesome software.

Probably not enabled yet.  Follow the specific development thread to find the latest on this camera.  For more information about AutoETTR go here:

ETTR (Exposure to the Right): - -History & Beginners Guide --
http://www.magiclantern.fm/forum/index.php?topic=5693.0
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: 1% on May 28, 2013, 07:45:05 PM
I think this is what broke voice tags... would also be nice to have it in movie mode with shutter doubleclick.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: a1ex on May 28, 2013, 08:56:52 PM
Voice tags are working here.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: X-RAY on May 28, 2013, 09:04:46 PM
@ alex
I tested the 5D2 build 1fcfef6c9425 from a.d.
But the shortcut buttons don't work for me. Only the "Always On" mode seems to work. The others move the mirror but don't calculate values. Ist it already tested and working with other 5D2s?
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: a1ex on May 28, 2013, 09:13:09 PM
On my 5D2 it's working. ExpSim enabled and LiveView is set to photo mode, right?
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: X-RAY on May 28, 2013, 09:25:16 PM
Ah sorry ... I think ExpSim was it. You're great ^^
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: 1% on May 28, 2013, 09:58:13 PM
For some reason when I start voice tags on 600D and 6D it just flashes the light once and doesn't start recording... set is only blinking the light. Even if I comment out the check for QR mode it doesn't work.... but it does start during normal photo mode... I see the events when I turned on event spy. Looking at it some more.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: 1% on May 31, 2013, 01:20:34 AM
*bump


Auto ETTR doesn't touch aperture does it? Outside I get overexposed frames if I don't set the aperture. I know many want fixed DOF, etc but maybe as an option.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: Audionut on May 31, 2013, 04:34:55 AM
Aperture could be set like a safety shift in Canon menu I guess.
At last resort, stop down aperture.

On another note, I set this up on my brother in laws 550D.  After being amazed, he looked at me and said, "But you said not to use auto modes".  lol.

Did anyone else know that when you load the raw_rec module, you get RAW histograms in live view?  Am I just slow!

On his 550D, Auto ETTR only worked in live view.  I couldn't get it to work at all in photo mode.  And yet on my 5D3, it doesn't work in liveview.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: a1ex on May 31, 2013, 06:45:04 AM
The math is already complex only with balancing shutter and ISO (with all those rounding limitations from Canon), good luck adding the aperture ;)

Also, the ETTR is also good for timelapse, where you want constant aperture. It's not intended to be a full auto mode. You can combine it with expo lock btw.

In LiveView, it works if picture quality is raw. If you shoot jpeg... of course it shows the jpeg histogram.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: lederschnautz on June 08, 2013, 07:44:20 PM
Hi there,

in the Beginners Guide to ETTR Timelapse, RenatoPhoto wrote
QuoteNOTE:  If you set your intervalometer at 30 seconds and your slowest shutter speed at 30 sec, you will have problems.  Set the intervalometer at least = Slowest Shutter speed plus the review image setting, in Canon Menu, plus 3 more seconds.
Would it be possible, to change the interval during the course of a shooting? So i start during daytime with short intervals, giving a lot of frames to work with and only if the shutter gets to slow, the intervals get longer.
If this question is already answered somewhere, please don't crucify me, i couldn't find anything about it.

Thanks to all the Devs for putting their time in this awsome Firmware, and to all the people contributing great information in these forums.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: a1ex on June 08, 2013, 07:45:09 PM
Set ETTR to 30s and intervalometer to 10s, should do the trick. Didnt't try it though.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: ismael on June 21, 2013, 05:10:13 PM
About ETTR.

In a dark scene, what would be better in terms of reducing noise?

- expose to the right using high ISO's? (assuming I'm already maxed out in terms of iris and shutter)?
- expose properly and have the advantage of using a lower ISO and therefore less noise?

This is not exactly about the feature itself from ML but more of a general wondering about the tecnique.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: jerkjackson on August 20, 2013, 10:07:49 AM
hi there,

i'm new to ML and the forums here - but so far am quite impressed! tried the auto ettr on a 5dII mainly for timelapse.

i'm currently planning a future project which will involve a lot of timelapse and will probably use ML for exposing to the right. the only thing that i'm not 100% happy with is, that it auto-ettr only seems possible in full RAW mode. am i missing something here? is it possible to use auto ettr with SRAW1 and SRAW2 modes as well?

thanks,
joerg
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: a1ex on August 25, 2013, 09:11:48 PM
Last week I had some time to play with ETTR and I think I've found a way to avoid the severe underexposure in scenes with high dynamic range (e.g. like this (http://www.magiclantern.fm/forum/index.php?topic=7835)).

https://bitbucket.org/hudson/magic-lantern/commits/fcdc23eb5d7d

How it works:
- If the midtones get too noisy, the ETTR algorithm will stop underexposing.
- Same thing if the shadows get too noisy.

How to quantify "too noisy"?

For midtones it's easy: the median "brightness" gives you the signal level, so we can use the SNR. So, when I say the midtone SNR is 5 EV, this means half of the image has a SNR less than 5 EV, and the other half has a SNR higher than 5 EV. Pretty easy and statistically robust.

For shadows, I've chosen the 5% percentile. So, in this context, the shadows having a SNR of 3 EV means 5% of the image pixels have a SNR lower than 3 EV, and the other pixels will be brighter than that.

Keep in mind that 5% is bigger than you may think, since it refers to image area, not linear size. The linear percentage is roughly 22% (1/4.5).

So, you can choose 2 noise limits, one for midtones and one for shadows, and the image will not get noisier than that (but you may lose some highlights). This setting has the highest priority, so it will sacrifice highlights even if you disallow clipping from the other settings.

I'm not yet sure what are some good defaults for this new trick, so it's turned off by default.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: Audionut on August 26, 2013, 06:46:32 AM
This is handy.

Shouldn't the midtones be 13-18% luminance?  3EV below important highlights.

In it's current configuration, 6EV looks to be a good default.

Default ETTR 0% clipping                                                                                                                   5EV midtones
(https://s15.postimg.cc/tn6mslvhn/Capture.jpg)(https://s15.postimg.cc/hlb8yhou3/Capture-2.jpg)




6EV midtones                                                                                                                                       7EV midtones
(https://s15.postimg.cc/vezlnimkb/Capture-3.jpg)(https://s15.postimg.cc/s8523w9u3/Capture-4.jpg)




8EV midtones
(https://s15.postimg.cc/rvdnxoh9n/Capture-5.jpg)


8EV midtones can be ruled out due to highlight loss, although to be fair, the skin tones have been captured correctly.  7EV has also blown highlights beyond recovery.  But lets weigh up midtone detail.

5EV midtones                                                                                                                               
(https://s15.postimg.cc/q3kp2uasb/Crops.jpg)

6EV midtones
(https://s15.postimg.cc/p1aikb7ej/Crops-2.jpg)

7EV midtones
(https://s15.postimg.cc/mjyrd10cr/Crops-3.jpg)

8EV midtones
(https://s15.postimg.cc/taf8mh2y3/Crops-4.jpg)

8EV is obviously clean because it was captured correctly.
7EV is nice and clean also because it was captured at only 1EV below.
6EV still retains nice clean detail in the midtones (skin).  Deep shadows are getting noisy.
5EV has noise above my own personal threshold.  Deep shadows are very noisy, but more importantly, noise in the midtones has become to high.

Conclusion.  You're like Santa Clause, but better.  You bring toys to play with every other week  ;D

On a more serious note!
6EV is where it's at.  6EV lifts the midtones to a suitable SNR while retaining very good highlight detail.

Here is a quick blend of the 6EV shot.
(https://s15.postimg.cc/n9hjpdlgr/Blended.jpg)


On scenes with greater dynamic range, my conclusion holds true imo.  5EV midtone setting will blow highlight detail without lifting that midtone detail to a sufficient SNR.  Might as well blow the highlight detail a stop further and get those midtone to a sufficient SNR.

Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: a1ex on August 26, 2013, 07:01:07 AM
QuoteShouldn't the midtones be 13-18% luminance?  3EV below important highlights.

I prefer to specify this in terms of noise (because this is the reason to sacrifice the highlights). So, at ISO 100, with 11 stops of DR, 3 EV below clipping level would be 8 EV of SNR if my math is not broken.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: Audionut on August 26, 2013, 09:36:38 AM
Yes that does make sense.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: RenatoPhoto on August 30, 2013, 05:05:15 PM
Quote from: a1ex on August 25, 2013, 09:11:48 PM
How it works:
- If the midtones get too noisy, the ETTR algorithm will stop underexposing.
- Same thing if the shadows get too noisy.

I am impressed with the results! 
I love AETTR and this function makes it easier to use because I do not need to tweak the highlight ignore so much.

It appears that by using these two settings the module adjusts to one or the other noise levels.  I assume that whichever (midtones or shadows) gives the higher correction level, then that one is implemented.  Maybe this is an weighted additive correction such as 1/2midtones + 1/2shadows.   
I was wondering if you could confirm the logic implementation because this part is not clear.

The final result is that the highlight ignore is adjusted based on each individual scene which is exactly what I have been looking for, i.e. some logic to automate the highlight ignore feature.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: a1ex on August 30, 2013, 05:10:11 PM
The logic is right in the quotes (so it will use the highest correction level).
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: Marsu42 on August 30, 2013, 06:15:17 PM
Alex, could you add an "analyze only" option that beeps noticeably when the raw histogram has the amount of blown highlights in the ettr options? This way, the module would be very useful even if not going for ettr, but shooting regularly in other modes like I usually do.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: a1ex on August 30, 2013, 06:17:52 PM
I have the histogram always enabled for that (the ETTR hint from there uses the parameters from auto ETTR).
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: Marsu42 on August 30, 2013, 06:25:44 PM
Quote from: a1ex on August 30, 2013, 06:17:52 PM
I have the histogram always enabled for that (the ETTR hint from there uses the parameters from auto ETTR).

The histogram from live view?! I seem to be doing the complete opposite vs. you, as I never shoot in lv, it wouldn't work for me, even if I miss a lot of the great ml lv features.

Most of the time I shoot animals with fill flash(es) in changing light in a semi-automatic (Av with ml autoiso, ml autoexpo) mode - and having an audible hint when the dynamic range rises too much for that iso setting would be invaluable ... because the exposure changes so often auto ettr isn't really helpful here. And even in full manual with flash an "analyze only" ettr option could signal too many blown specular highlights.

Btw as you know this request is really a tiny addition (beep instead of ettr), I'm sure I'm not the only one that'd find it useful.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: RenatoPhoto on August 30, 2013, 06:38:07 PM
Is it possible to use Canon Exposure value or BV value (recently found for Autoexposure module) + 1 EV for a quick first approximation for AETTR?
The mem location was found here: http://www.magiclantern.fm/forum/index.php?topic=7208.msg67025#msg67025

I noticed that AETTR sometime takes a few shots to converge.  But If I take the Canon suggested Exposure value and add +1EV an use these settings as begging point for AETTR then the solution converges rapidly!
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: a1ex on August 30, 2013, 06:55:03 PM
The histogram is there in photo mode too.

Canon exposure value is not much better than random from what I've noticed. But you can use it, just flip to some auto mode.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: RenatoPhoto on August 30, 2013, 07:13:38 PM
Quote from: a1ex on August 30, 2013, 06:55:03 PM
Canon exposure value is not much better than random from what I've noticed. But you can use it, just flip to some auto mode.

I had tried this in the past hoping that Canon values would speed up convergence but sometimes it never converged due to some conflict so I stop doing this.  I will try it again.

Thanks
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: Marsu42 on August 30, 2013, 07:48:39 PM
Quote from: a1ex on August 30, 2013, 06:55:03 PM
The histogram is there in photo mode too.

Ok, I again have learned something - but what I really want is to get the camera tell me about blown highlights *without* looking at the display, no matter if live view or not.

That's because when shooting animals my eye is glued to the viewfinder, and I often haven't got time to check for the correct exposure and end up with clipping, as you pointed out Canon's metering cannot really be relied upon so a ml warning would be appreciated...
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: a1ex on August 30, 2013, 07:50:19 PM
The hardest part is to think where I should put that in the menu.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: RenatoPhoto on August 30, 2013, 08:22:24 PM
Quote from: a1ex on August 30, 2013, 06:55:03 PM
But you can use it, just flip to some auto mode.

Ok, I see that it works better now but unfortunately the Midtones and Shadows do not work since (Auto Iso).  If the AutoEttr is ON can it automatically disable the AutoIso to gain better control?
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: a1ex on August 30, 2013, 08:27:39 PM
I prefer to let the user choose whatever settings he wants, and just print a warning.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: RenatoPhoto on August 30, 2013, 08:55:02 PM
Quote from: a1ex on August 30, 2013, 08:27:39 PM
I prefer to let the user choose whatever settings he wants, and just print a warning.
In C1 (shutter priority) if I put ISO in some manual value, then AutoETTR module does nothing and reaches a limit.  If I flip to Auto ISO then AETTR works but Midtones and Shadows system does not work.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: Marsu42 on August 30, 2013, 09:05:37 PM
Quote from: a1ex on August 30, 2013, 07:50:19 PM
The hardest part is to think where I should put that in the menu.

Um, just add a Tigger Mode "Always warn" with the help line explaining that it beeps when the highlights are blown according to the ettr settings? The decisive part for me here is that I'd not need to look away from the vf.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: Audionut on August 30, 2013, 11:57:01 PM
Quote from: RenatoPhoto on August 30, 2013, 08:55:02 PM
In C1 (shutter priority)

Try Tv mode rather then a custom mode.

Quote from: Marsu42 on August 30, 2013, 09:05:37 PM
Um, just add a Tigger Mode "Always warn" with the help line explaining that it beeps when the highlights are blown according to the ettr settings? The decisive part for me here is that I'd not need to look away from the vf.

Good idea.  Being able to set a warning for the midtones/shadows would be very handy too.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: RenatoPhoto on August 31, 2013, 12:06:55 AM
Quote from: Audionut on August 30, 2013, 11:57:01 PM
Try Tv mode rather then a custom mode.

Negative Expo limits reach and no adjustment made.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: Audionut on August 31, 2013, 12:43:26 AM
In the meantime, stick to M mode and use the exposure meter.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: Marsu42 on August 31, 2013, 12:11:11 PM
Quote from: Audionut on August 30, 2013, 11:57:01 PM
Good idea.  Being able to set a warning for the midtones/shadows would be very handy too.

Absolutely, I simply didn't want to push my luck by requesting too much, someone might tell me to do it myself and submit a pull request... but if we have a low warning beep for clipped shadows & a high for blown highlights (+ probably another beep for histogram midtone weight) there has to be an added option to toggle these individually. For me, the "blind" raw highlight warning is the most important one though.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: RenatoPhoto on August 31, 2013, 02:11:03 PM
I have had this idea in my mind about AETTR so here it is:

There has been many times where the metering of the entire scene is not necessary.  Is there a way to reduce the metering (ETTR )analyzing area to 3/4, 1/2, 1/4 of the center image?  This is specially important for wildlife video and photography where the optimum distance to the subject cannot be easily obtained.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: ceew on September 01, 2013, 01:08:04 AM
Sorry if this has been requested before..

I have been choosing aperture priority with m settings and in low light situations the shutter speed is about 1/100 per second and the iOS ramps up..

Is it possible to have the iOS setting used displayed above the histogram.. I have the histogram in the lower right position so having it maybe just above or below the histogram would be awesome.. Sometimes the iso is just too high .. Etc.. 12800 where I would probably choose to compromise my dof rather than choose such a high iso or even shutter speed, ie or using a tripod etc..

And also hard choose the current best  iso  choices 160,320,640,1250 etc.. Rather than 200,400,600,800.. Etc..

But the iso display above or near the histogram would be first on my wish list..

Canon 5d m3

Thank you..
Magic lantern is just awesome..
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: STELYA on September 23, 2013, 09:45:32 AM
Just started playing with this one and I may be missing something ...
How to use it properly ? I mean I am in AV mode ... ETTR trigger is on 2xHalfShutter or AlwaysON , highlight goodbye is from 2-10% , EV from 1-4 .... RAW histogram/histogram hint enabled...clip GREEN.

I point my 6D at a high contrast scene and take one Canon measured photo then I enable ETTR by pressing 2xHalfShutter and it goes into LV ...beeps and thats it...Then I take another pic and it is almost the same as original... Am I using this feature wrong ? When to press shutter to take ETTRed photo? I remember it was working in early stage but not sure as I played with it just little.

Id like to use it instead of Canon's metering all the time, it is something I was reading about before ML and it's a great feature.
using latest alpha

Tnx
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: Marsu42 on September 23, 2013, 09:52:11 AM
Quote from: STELYA on September 23, 2013, 09:45:32 AM
Just started playing with this one and I may be missing something ...

To try auto-ettr, use it in live view because it can re-meter w/o flipping the mirror - then you'll instantly see the result with different sources.

Quote from: STELYA on September 23, 2013, 09:45:32 AM
Id like to use it instead of Canon's metering all the time, it is something I was reading about before ML and it's a great feature.

I'd also like to do this, but the lag of a-ettr is about 3 seconds - if the scene changed since then, your exposure will be broken.

This can result in almost comical situations, recently I was a-ettr'ing a horse going between trees from shadow to sun wit about a 5sec interval and ml managed to mis-expose every shot :-p ... its such a shame Canon only meters for jpeg, but in many situations it cannot be helped.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: a1ex on September 23, 2013, 09:59:06 AM
I had some attempts to integrate it with Canon metering, but it ended up screwing the exposure completely.

Canon metering seems to be strongly center-weighted, while ETTR is not. So, moving the camera a little changed Canon metered value enough to clip the highlights or cause ETTR to require another test picture.

The only case when it helped was with the camera on a tripod, pointing it to a static scene, and adjusting the aperture manually back and forth. In this case, Canon meter was good enough to get the correct exposure instantly or with at most one intermediate test pic. But in most other cases that I've tried, it didn't help.

Maybe something like "adjust settings if Canon meter shows a difference of 2 stops or more" would work better.

If any of you wants to try, I can share a patch.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: Marsu42 on September 23, 2013, 10:12:16 AM
Quote from: a1ex on September 23, 2013, 09:59:06 AM
Maybe something like "adjust settings if Canon meter shows a difference of 2 stops or more" would work better. If any of you wants to try, I can share a patch.

Absolutely, I'm always willing to try stuff if there is no danger of bricking my camera(s)... and this sounds like it would be able to address my main concern with a-ettr, at least a bit.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: STELYA on September 23, 2013, 11:02:47 AM
I could try as well no worries :)

But to do it I just need to be clear what I have to do in order to get it to work...

I activate it (double press)
It goes into scan mode...beeps and returns to normal...
Whaddaya need to do next? Take the shot (when to take it) ?
Do I need to hold down second shutter down all the way through scanning and then do a full press?
And where can I see that Ive done it right?
Trying this in Av mode btw.


Cheers
Srry for being a noob here ;)
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: Audionut on September 23, 2013, 11:53:31 AM
a1ex, wouldn't it be better to use AETTR to work out the needed EC?  There then comes a reliance on the photographer when using AETTR in semi-auto modes to ensure the scene does not change dramatically.  Just as with normal EC determinations.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: a1ex on September 23, 2013, 11:56:38 AM
Yes, my implementation was based on EC.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: Audionut on September 23, 2013, 12:47:45 PM
From a photography standpoint, autosnap is probably the best option to use.  If your exposure settings were already correct before autosnap, you get your photo.  Else, you still get photos while AETTR is doing its calculations.  ie:  You don't miss your moments.

It would be handy to have autosnap linked to set/half shutter.  In this way, when needed you can press set/halfS have have autosnap perform its actions, but most importantly, the setting is then reset back to a state where it needs set/halfS to trigger the operation again.  This would save having to enter the menu to disable autosnap, or, running around in autosnap mode killing the shutter.

All other options, link to dualISO, SNR limits etc would remain fixed at the last calculation.

Where lighting conditions change significantly, AETTR would be a quick exposure option where images are still taken while the calculation is being performed.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: v8rrc on October 01, 2013, 12:30:26 PM

Is it possible to change the "Link to Dual ISO" option so that it'll use Dual ISO just for ettr pictures and not for the ones taken with out ettr without having to go back to the menu to turn Dual ISO on and off every time?
It's just that I find that if I'm taking a lot of snapshots Dual ISO is just a lot more work and if I've got a proper shot I just use the half press double click and its a nuisance to have to go into the menu every time to turn Dual ISO on and then off again.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: Marsu42 on October 01, 2013, 12:44:30 PM
Quote from: Audionut on September 23, 2013, 12:47:45 PM
From a photography standpoint, autosnap is probably the best option to use.  If your exposure settings were already correct before autosnap, you get your photo.  Else, you still get photos while AETTR is doing its calculations.  ie:  You don't miss your moments.

If the moments lasts longer than 5 seconds, which doesn't qualify as "moment" to me :-p ... anyway: I find auto-snap awkward because you never know if ettr will take another picture, so you have to hold the camera with your eye to the vf and wait what happens, always seems a bit comical to me. With "always on" you at least have control over taking another shot or not and also don't use much time when pressing the shutter manually.

Quote from: v8rrc on October 01, 2013, 12:30:26 PM
Is it possible to change the "Link to Dual ISO" option so that it'll use Dual ISO just for ettr pictures and not for the ones taken with out ettr without having to go back to the menu to turn Dual ISO on and off every time?

+1 ... imho ettr should "silently" use dual_iso w/o permanently enabling/disabling it or modifying the dual_iso settings as this will screw up the dual_iso values in normal shooting mode. So basically ettr should save dual_iso values before taking the shot, then restore them.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: tobi_ml on October 24, 2013, 09:28:41 PM
I tested the AETTR features today and just can say wow ;D! But still I have one question:
AETTR changes the ISO and the Shuttertime, but it does NOT change the aperture - Am I right so far ?
I tried it to combine it with the TV Mode to get also a aperture ramping, but this gave me a flickering (One time the completly opened aperture like bevor in Manual Mode, then the correct aperture from the TV Mode and so on...) with an intervall of about 1 second  :o. I made the timelapse in silent mode (Raw DNG, NO dual iso) on a 5D3.
So which is the best method to controll all 3 - the aperture, the shuttertime and the ISO - only with AETTR in M Mode (or are their better solutions to do this)?
Thanks for your answers :)
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: RenatoPhoto on October 25, 2013, 03:32:19 AM
If you are talking about timelapse then
http://www.magiclantern.fm/forum/index.php?topic=8431.0
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: tobi_ml on November 01, 2013, 04:29:53 AM
Oh yes, I forget to mention that, any other hints for "silent" day to night timelapses?
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: gideonplus on January 21, 2014, 01:28:26 PM
is it possible to make the ETTR in Video RAW\MLV changing only the ISO first
and NOT changing the shutter speed which is imporant for not having stoboscopic
effect when the shutter is more then twice faster then frame rate in video mode ?
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: dallasill0 on January 28, 2014, 02:31:45 PM
I have 6d 1.1.3 (October) Tragic Lantern, but in ETTR settings I dont find Shadow SNR and Midtones SNR... Anybody knows why???
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: Audionut on January 28, 2014, 05:20:56 PM
October was a long time ago.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: l_d_allan on January 29, 2014, 03:21:51 AM
Quote from: dallasill0 on January 28, 2014, 02:31:45 PM
I have 6d 1.1.3 (October) Tragic Lantern, but in ETTR settings I dont find Shadow SNR and Midtones SNR... Anybody knows why???

The latest TL for the 6d has time-stamp of Jan 18,  2014.

I just installed the Jan 18 version. Seems fine. However, there doesn't seem to be any way to determine just what version is active.

Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: 1% on January 29, 2014, 03:51:21 AM
The date on the zip is what you have. ETTR should be 1:1 with ML.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: Audionut on March 06, 2014, 02:29:32 AM
I moved the development discussion into it's own thread.

http://www.magiclantern.fm/forum/index.php?topic=10808.0

Cheers.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: PaulJBis on March 17, 2014, 01:29:24 PM
Hello. I've been trying AutoETTR for a sunset timelapse this weekend, and I have some questions about it.

First of all, until now I was shooting using the old bulb ramping feature. When I shoot sunset timelapses, my opinion is that it doesn't matter if the latter shots (the ones at night) have a lower exposure value than the earlier ones; after all, it's supposed to be at night, and if you try to overexpose them so that they match the daylight shots, you'll risk blowing out the lights in the shot. Thus, until now I was using bulb ramping to ramp down the EV value.

I understand that this is not possible with AutoETTR, since its philosphy is to expose always to the right in order to decrease noise, and if you want to ramp down EV, you can always do it later in post. Which is fine... except that sometimes, when it's really dark at night, AutoETTR simply can't increase the exposure anymore. The only way it can do so is by increasing the ISO to ridiculous levels (3200, 6400), which kind of defeats the purpose of avoiding noise. (It happened to me this weekend).

Thus, I'm wondering if there's any way to "cap" the ISO levels, or the EV values that AutoETTR aims for. One way I'm thinking of doing it is by activating the advance intervalometer and adding keyframes in it for ISO, and then using the "slowest shutter" option. Is there any other way? Or, in more general terms: is there a way to ramp up/down the EV target of AutoETTR + adv_int.mo for situations like these?


Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: a1ex on March 17, 2014, 01:38:32 PM
Quote from: PaulJBis on March 17, 2014, 01:29:24 PM
The only way it can do so is by increasing the ISO to ridiculous levels (3200, 6400), which kind of defeats the purpose of avoiding noise.

I have yet to see a higher ISO noisier than a lower one (in electrons). If you want to reduce the ISO (all other variables being equal), it's not to get less noise, but to capture more highlights.

FYI:
http://www.clarkvision.com/articles/evaluation-canon-5diii/index.html
http://www.clarkvision.com/articles/evaluation-canon-7d/index.html
and so on

on-topic: change auto ISO limit in Canon menu.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: PaulJBis on March 17, 2014, 02:08:40 PM
Quote from: a1ex on March 17, 2014, 01:38:32 PM
on-topic: change auto ISO limit in Canon menu.


Quick question: but I thought that AutoETTR didn't work if you set the ISO in auto mode?

(I'm reading the links right now. Will post later with questions if I have doubts :-) ).
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: garry23 on March 17, 2014, 02:10:05 PM
Alex

At the risk of complicating this post I would like to try an idea on you, that relates to a feature request I posted the other day.

The idea is to bring an ML 1 deg spot together with AETTR.

At the moment my understanding is that the AETTR algorithm scans the entire scene and works out the shift in exposure to ensure the user's criteria is met.

I use manual (ie non ML) ETTR a lot, but with a spot meter. My work flow is simple, scan the scene with my 1 deg spot meter and select the 'hottest' part of the image that I wish to see shifted to the right, ie the highlights that I want to see in zone 7 or 8, rather than the camera's mid tone zone 5.

Through calibration I know on my 5DMkII the DR is about 7.2Ev and the highlight edge is at 3.2Ev. So I can safely shift by up to 3Ev.

I use an external meter as this is lens independent.  BTW on my 5DMkIII I would need to use about a 350mm lens to achieve a 1 deg spot - see here: http://www.spotmetering.com/1deg.htm

If ML could emulate a 1 deg spot, ie by noting the lens FL and the sensor size and returning an exposure restricted to the ML 1 deg spot, this would mean ML is a real alternative to, say, the Sekonic meter, which I use. If the lens doesn't return its FL, then ML would use a default FL that the user could change, ie to account  for a manual lens.

Once there is an ML 1 deg ML spot meter built in, the user would than be able scan the scene in LV, select the part of the scene to AETTR and let ML do the rest, ie shift the exposure or at least indicate what that shift would be.

The two killer features here are a 1 deg spot (or a selection of angles?) coupled to the user selecting the area of the scene to shift from.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: a1ex on March 17, 2014, 02:19:24 PM
Well, the spotmeter algorithm is quite a lot simpler than ETTR.

I could add a window size option for ETTR, which is the simplest way to implement a spotmeter without reinventing the wheel. However, there's a important difference:

- In the current configuration (full-screen), the SNR limits and highlight ignore percentage are very important to get a good balance between noise and highlight detail, given the limited DR in Canon cameras.

- With a small spotmeter, these settings will have no effect and will become confusing (a spotmeter can simply do a plain average and that's it).

So I'm not sure if - for a UI perspective - spotmeter really makes sense as a sub-mode in ETTR, or as a completely separate tool.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: garry23 on March 17, 2014, 03:03:53 PM
Alex

I understand and would humbly suggest that it would be well worth doing.

I still believe, but I'm not a coder, that by first getting a 1deg spot in ML, the ability to survey the scene in LV mode, as now but a 1deg, would be great.

If ML could then return the exposure at that 1deg point in the scene and maybe give an ETTR hint, that would be good.

'All' I'm suggesting is to create an inbuilt 1deg spot meter that is independent of FL, ie lens/sensor coupled.

If this can be linked to an ETTR shift great, but honestly that is a bonus, as if I know the 1deg spot exposure reading, I can manually shift to zone 7 or 8.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: PaulJBis on March 17, 2014, 09:14:16 PM
Quote from: a1ex on March 17, 2014, 01:38:32 PM
I have yet to see a higher ISO noisier than a lower one (in electrons).

Okay, I just did a few tests. You were right!!   :o I guess I was still thinking about ISO in terms of film stocks...

However, my larger point stands: I just finished another sunset timelapse using AutoETTR, and at the end (when it was dark), it blew my highlights to the point that now I can't recover them in post. Thus, I'd like to know if there's a way to ramp EV so that, at the end of the timelapse, the software aims for a different value than at the beginning.

Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: a1ex on March 17, 2014, 09:28:54 PM
You probably had SNR limits set too high.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: PaulJBis on March 17, 2014, 09:52:13 PM
I had them on their default values, 6 and 2. So if I lower them, I'll get a picture less "to the right" at the cost of more potential noise, right? I'll experiment with other values then.
Title: Re: [DONE] Auto ETTR based on RAW histogram?
Post by: PaulJBis on March 18, 2014, 07:58:06 PM
Okay, I've been rereading this thread, paying special attention to the posts where this new feature (SNR limits) was introduced. However, there's one thing I don't understand:

Quote from: a1ex on August 25, 2013, 09:11:48 PM
So, when I say the midtone SNR is 5 EV, this means half of the image has a SNR less than 5 EV, and the other half has a SNR higher than 5 EV. Pretty easy and statistically robust.


Why are you talking about SNR in terms of EV units instead of, say, dB?

(Please note: I'm no professional developer nor physicist, just a humble amateur photographer. Pointers to basic notions are welcome. After googling I found this explanation:

http://www.imatest.com/docs/noise/

But I don't know if it's relevant to the subject).

Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: a1ex on March 18, 2014, 08:24:04 PM
Yes, I prefer to use EV units, since that's how you dial the exposure.

So, with these SNR limits, you ask "don't underexpose more than that", and ETTR will do that religiously (even if that means it will burn up to 95% of your image - worst case).

Note that ETTR approximates SNR to signal / optical black noise (that is, it assumes a constant sensor noise model, as in the column A from your link). On 5D3 ISO 100, this is true until around 4 EV above the noise floor. So, ETTR assumes the SNR follows the black line from this graph (that is, it assumes 1 EV of slower shutter == 1 EV of less noise, which is only true in shadows):

(http://a1ex.magiclantern.fm/bleeding-edge/profile-5D3-iso100.png)

(so you can ask for 8 EV in midtones, but in practice you'll only get 6)

I could change it to be more exact, but I need to know some more data about each ISO. Right now, the black line is completely determined by dynamic range figures, which I copied from DxO. If I also consider the max SNR in highlights, I could get a better approximation.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: a1ex on May 08, 2014, 08:44:28 PM
Some slides I've prepared a while ago: comparing Canon's exposure meter with ETTR. Hope it helps understanding the difference between the two, and the motivation behind ETTR.

(http://a1ex.magiclantern.fm/bleeding-edge/ettr/canon1.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/ettr/canon2.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/ettr/ettr1.jpg) (http://a1ex.magiclantern.fm/bleeding-edge/ettr/ettr2.jpg)

So, ETTR analyzes everything from your scene, but doesn't care where they are placed (the exposure decision is based only on the histogram), while Canon metering is much too center-weighted (or focus-point-weighted) for my taste.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: Audionut on May 09, 2014, 05:15:44 AM
When iso_research is completed, I would like to add get_iso_research_dynamic_range.
Can you use the SNR curves from raw_diag to help with accuracy?

The biggest problem with Canon metering, IMO, is that it is dumb.  It doesn't care about highlights, shadows, SNR, DR.  Simply, find the average, and place it at -4 EV.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: barepixels on May 20, 2014, 09:42:20 AM
Someone asked on page 3 how come AETTR can't work with SRAW1 or SRAW2.  Can someone explain why the option is not available.  For doing time-lapse I find SRAW1 is half the file size, Thus I can shoot twice as long without changing memory card.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: dmilligan on May 20, 2014, 12:57:12 PM
Quote from: barepixels on May 20, 2014, 09:42:20 AM
Can someone explain why the option is not available.
The camera behaves differently when in these modes and so they would need to be reverse engineered (like everything else in ML).
Title: ETTR and back button focus
Post by: garry23 on August 31, 2014, 01:10:38 PM
Because I usually use SET to call ETTR I have only just noticed a conflict with back button focus.

When I move ETTR to double press, it works as it should off the exposure button, but it also triggers off a double press of the back button.

Is there any way to decouple ETTR being triggered off the back button when ETTR is used by a double press?
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: tetsusaiga on September 01, 2014, 02:26:46 PM
PaulJBis:  You said in your post above that you "just finished another sunset timelapse using AutoETTR, and at the end (when it was dark), it blew my highlights to the point that now I can't recover them in post. Thus, I'd like to know if there's a way to ramp EV so that, at the end of the timelapse, the software aims for a different value than at the beginning."

When you say "ramp EV exposure," are you talking about the "Exposure target" parameter in AutoETTR? How have your sunset/sunrise timelapses come along so far with AutoETTR? I'm trying to learn all that I can so any help or tips would be appreciated.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: dmilligan on September 01, 2014, 03:35:12 PM
PaulJBis problem was likely too high of "SNR Limits", the SNR limits will override the clipping limit (things will get overexposed) to maintain the requested SNR. Solution is to set the SNR limits lower.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: garry23 on September 01, 2014, 03:48:35 PM
Re my shutter half press question, does any one know how to just get this on the shutter button, ie and not the back button.

I would have thought no one would want the double press working on the back button.

Or am I wishing for unobtanable things?
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: PaulJBis on September 02, 2014, 12:18:36 AM
Quote from: tetsusaiga on September 01, 2014, 02:26:46 PM
PaulJBis:  You said in your post above that you "just finished another sunset timelapse using AutoETTR, and at the end (when it was dark), it blew my highlights to the point that now I can't recover them in post. Thus, I'd like to know if there's a way to ramp EV so that, at the end of the timelapse, the software aims for a different value than at the beginning."

When you say "ramp EV exposure," are you talking about the "Exposure target" parameter in AutoETTR? How have your sunset/sunrise timelapses come along so far with AutoETTR? I'm trying to learn all that I can so any help or tips would be appreciated.

In the context of that post, I didn't mean the "Exposure target" parameter that you can set in AutoETTR. I meant the internal EV target that AutoETTR sets when it performs its calculations.

BTW; I've been out of the loop, shooting timelapses (among many other things). Will try to catch up tonight and see if there's been anything interesting.

Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: jackmoro on October 19, 2014, 10:15:45 AM
Hello.

Is it possible (miracle  :o) to make ETTR calculate exposore only for working area in RAW recording? Like if we use 2:20 aspect, some area will be not be used in real footage, but it might (often) has some bright parts.

Alex.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: moodlover on March 28, 2015, 04:44:33 AM
When exposing a scene, should we always go for ettr? Without ML, in photo mode, I did a test where I'd expose a shot twice, once at the metered exposure for 0 and once at +2 stops over. In lightroom when I applied -2ev to the over one to match the exposures, the over one had extremely clean shadows (no noise) compared to the normally metered one.

Can/does this apply when recording raw video? As in, a) will ettr always ensure we capture a good exposure and b) will going over ettr allow us to reduce exposure in post and give a better image?

https://luminous-landscape.com/the-optimum-digital-exposure/
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: dubzeebass on March 28, 2015, 05:51:20 AM

Quote from: moodlover on March 28, 2015, 04:44:33 AM
Can/does this apply when recording raw video?

a) will ettr always ensure we capture a good exposure and
b) will going over ettr allow us to reduce exposure in post and give a better image?

Yes it applies as long as you don't blow out any parts of the image

a) it may overexpose and require pulling it back. Overexposure != good exposure but you've got the gist of it.
b) see first sentence of this reply :)

Optimally for more efficient workflow you'd use a light meter to match the exposures.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: a1ex on June 12, 2015, 06:00:29 AM
Last night I wanted to film some cactus flowers which only last for one night, and 30 seconds weren't enough in my previous attempt. So, instead of adding extra lights, I modified ETTR to work with very long exposures.

https://bitbucket.org/hudson/magic-lantern/commits/63884c935209

That means:
- you can now set a slowest shutter speed of up to one hour
- if it requires a shutter speed longer than 30 seconds, it will switch to bulb mode, and will configure Bulb Timer with the new exposure time
- once it no longer requires a long exposure, it will switch back to M

The change is not in the nightly yet - I need to do some more tests, since I've modified the ETTR code in quite a few places.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: josepvm on June 13, 2015, 12:32:13 AM
Very useful addition !!!  Many thanks, A1ex.

I've build the ettr-bulb branch for 500D and tested it indoors, in low light, and it works very well.

At first attempt I tried to use the feature in liveview mode with no success, ETTR rises ISO and not exposure.
Then I have tried in photo mode, setting "Autosnap" to trigger ETTR, with bulb timer enabled, and this way it works fine.


I will test it also outdoors, during next days, with a strong ND filter, to capture seashore landscapes like this one (Dual-ISO):

https://dl.dropboxusercontent.com/u/44995840/20131230-ee16002-jv200.jpg (https://dl.dropboxusercontent.com/u/44995840/20131230-ee16002-jv200.jpg)

Adjusting exposure in this picture with such a strong ND filter, and sunset's continuously changing light was a pain. I needed to meter the scene with the filter removed, and then calculate the new exposure using the filter coefficient ... Now Auto ETTR will make it much easier, no need to remove and put again the filter every time !


Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: garry23 on June 13, 2015, 02:26:57 AM
I'm being stupid.

Just tried to compile this but can't work out what to use instead of what I normally use:

hg clone -r unified https://bitbucket.org/hudson/magic-lantern

The branch says use this:

hg pull && hg update ettr-bulb

But that doesn't work.

Can some expert correct me please.

I'm compiling in the cloud.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: josepvm on June 13, 2015, 08:04:29 AM
Use a different directory to start from and launch:

hg clone -r ettr-bulb https://bitbucket.org/hudson/magic-lantern
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: garry23 on June 13, 2015, 02:47:05 PM
@josepvm

Many thanks, that worked. Like many on the ML forum, once pointed in the right direction I'm OK :-)

Have loaded on to my 5D3 and confirmed the tweaked module is loaded, as I now see the extended time in the slowest shutter.

But I can't see to get it 'to work', ie it just switches to ever higher ISOs.

Have tried it in LV and non-LV modes, using SET and AutoSnap and M and B modes.

Any guess as to what I'm doing wrong?
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: garry23 on June 13, 2015, 03:09:09 PM
OK worked a few things out.

'normal' ETTR doesn't require canon review to be on: LE ETTR apparently does.

The LE-ETTR is now going into capture mode, but doesn't stop, ie I have to intervene and the image is totally overexposed.

BTW I'm using a 24-105 Canon F/4L with an ND.

I've also tried exposure sim on and off.

At the moment, I still can't seem to capture an LE image.

I will keep experimenting.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: josepvm on June 13, 2015, 03:41:14 PM
You need to set "bulb timer" active, in ML's menu ( "shoot" section ), set M mode in photo view, adjust desired aperture and ISO, and take the first picture with a half shutter press. If ETTR is set to "autosnap" it will continuously take photos until the correct exposure time is found.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: garry23 on June 13, 2015, 04:07:37 PM
@josepvm

Thanks for those tips.

Still having problems as LE ETTR seems to seek out a high ISO solution.

Am I missing a trick :-)
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: josepvm on June 13, 2015, 04:12:48 PM
In live view mode ETTR seeks for high ISO, yes. Do not use live view. Set your desired ISO and simply take the first picture with a half shutter press, and let it run, automatically taking as many pictures as needed, until exposure time settles in the correct value.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: garry23 on June 13, 2015, 04:15:24 PM
I'm not in LV mode.

LE ETTR still goes for high ISO.

Strange
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: a1ex on June 13, 2015, 04:22:26 PM
Can you take a video of the issue?
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: garry23 on June 13, 2015, 04:32:59 PM
Alex

Sorry cant do that (at least not to day).

Settings are:
- Auto ETTR = Press SET, Slowest 8m, Highlight Ignore .1%, everything else off
- Expo Override OFF
- Expo Sim OFF
- Bulb Timer 8m (but grey out as not in LV)

Question: should I do a half shutter press in M or B mode, to trigger LE ETTR?
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: a1ex on June 13, 2015, 04:40:51 PM
Change the trigger mode to either "Always ON" or "Auto Snap".
Then, do a full shutter press if you are in M, or a long half-shutter press if you are in B.

If you meter with SET, this uses the LiveView image. It's not possible to figure out a 5-minute exposure from a 1/30 test image, so you must meter from an actual long exposure.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: garry23 on June 13, 2015, 04:46:56 PM
Alex

OK but what seems to occur in B is that it appears to meter, then it goes into capture mode, ie I get about 5 or 6 beeps and the canon clocks starts counting.

But I stop it after a min or so and everything is totally over exposed.

PS half press doesn't trigger in M mode

PPS I should have said, the exposure should be between 1-2min
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: josepvm on June 13, 2015, 04:57:56 PM
Set a shorter time in "bulb timer" (40 seconds) , ETTR in "Auto snap" mode, take the first picture, and let it run, the first picture probably will not be correctly exposed, but ETTR will automatically correct exposure time and it will take also automatically a second picture, and a third one if needed. 
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: garry23 on June 13, 2015, 05:09:53 PM
Sorry still not functioning as you.

I'm in B, ML Bulb is set to 40s, Autosnap on, I do a half long press and capture starts. It takes a picture (40s) then does a second at 1/25.

Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: josepvm on June 13, 2015, 05:13:57 PM
Is the second picture at 1/25 well exposed? If yes, recheck ISO and aperture settings.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: garry23 on June 13, 2015, 05:17:30 PM
No, totally under.

If I switch to canon metering 0Ev is about 3s.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: a1ex on June 13, 2015, 05:23:50 PM
Try starting with bulb timer from 1 second.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: josepvm on June 13, 2015, 05:27:58 PM
Check if the aperture is effectively closing to the desired value when taking the pictures, there have been problems with some lenses in FRSP, I know that was using live view, but who knows... try pressing DOF button and slghtly unlocking the lens from mount ...
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: garry23 on June 13, 2015, 05:34:24 PM
Alex's hint at 1s seems to have stabilized things.

I'm using a manual aperture now, ie Canon and ML see F/00.

I seem to be getting solutions now, but ML/Canon gets confused regarding image taking and, out of frustration, I switch on and off to reset things.

Also, can't get LE-ETTR to trigger in M mode.

Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: a1ex on June 13, 2015, 05:37:00 PM
Quote from: garry23 on June 13, 2015, 05:34:24 PM
Also, can't get LE-ETTR to trigger in M mode.

Quote from: garry23 on June 13, 2015, 04:46:56 PM
PS half press doesn't trigger in M mode

Press the shutter button a bit harder.

Quote from: a1ex on June 13, 2015, 04:40:51 PM
do a full shutter press if you are in M
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: garry23 on June 13, 2015, 05:43:03 PM
 :)

OK I'll practice a bit more  ;)

Here are some thoughts so far.

Bulb timer doesn't get reset after an LE-ETTR capture. For instance if I originally have the ML Bulb at 1s and LE-ETTR finds the solution at 120s, ML bulb is at 120s when I next use LE-ETTR, unless I manually reset to 1 s. This threw me a few times.

For ML enhanced LE work we have this emergent capability and David M's ND Bulb Module (which is not yet in the nightlies).

The LE-ETTR is great for when you don't really 'care' too much about shutter time.

If you want a 'spot on' shutter time, then the ND Bulb Module is the way to go.

Maybe there is a way to combine to two approaches into an new LE module, ie keep ETTR 'pure' for shutter speeds less than Bulb.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: a1ex on June 13, 2015, 05:48:12 PM
Quote from: garry23 on June 13, 2015, 05:43:03 PM
Bulb timer doesn't get reset after an LE-ETTR capture.

I wanted this to work for timelapse, so it's not meant to be reset - it will fine-tune the solution as the lighting is changing.

To "reset" it, you could start from M mode. After the long exposure image is taken, you will be in B mode, but your mode dial will be still on M. So, flip the dial back and forth to reset the mode (and get back to M), then you can start over.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: garry23 on June 13, 2015, 05:49:43 PM
Alex

Have been doing the flipping thing :-)

BTW it still wont trigger, via a long half press, from M, it does from B.

Cheers

Garry
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: garry23 on June 13, 2015, 05:55:21 PM
Alex

How about a user choice, ie reset bulb or not :-)
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: garry23 on June 13, 2015, 07:28:37 PM
Alex/josepvm

Have experimented more and restricted my workflow to starting in M with AutoSnap, which requires a full press to initiate.

I have ML Bulb Timer enabled, but don't worry about the time (as ETTR will control).

I set Canon ISO to 100 and the exposure at a few sec, eg 4s, (which will be the autosnap test exposure).

I then carry out a full shutter press in M mode.

AutoSnap takes a 4s test exposure (and captures this image to the card).

It then automatically takes the 'real' exposure.

One thing I have noted is that I need to set % highlights higher than in normal ETTR mode, eg 1% or higher. If I don't the ETTR solution doesn't look pushed to the right enough.

BTW my test image has no speculars, it's flat.

So what could be 'better'?

At the moment the user has little feedback on the LE shutter time (I assume the beeps at the start are indicative, eg 3 beeps = 3 mins). Is there a way to flash up the LE time on the screen prior to taking the LE?

Also, as i said previously, this and David M's ND Bulb are complementary. Hence it would be good to have both running.

Bottom line: great feature if you are not too worried about what the actually time is, ie in dynamic Timelapses or where you are taking a still at 'simply' a long exposure. DM's ND Bulb Module is there (hopefully soon) for the occasions when you want a specific time, ie I can dial in my variable ND or change things to ensure a time.

Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: josepvm on June 13, 2015, 09:06:46 PM
Tested your procedure, It works fine for me, also.

And yes, after a test shot with only few seconds of exposure, ETTR is able to correctly expose the second shot ( aprox. ~ 1 minute exposure ) So no need to waste time in a very long test exposure, that's great !

I do not see any need to change the highlight ignore setting, I get the exposure that I like  ... but I always set lower ETTR's SNR limits (4 EV for midtones, 1 EV for shadows) than default ones (6 EV for midtones, 2 EV for shadows).  The reason is that I capture mostly landscapes, and I don't want that sky highlights get clipped at all. If the foreground gets too dark, I use Dual-ISO or bracketing.

If you feel that in long exposures you are getting underexposed images, you can increase the highlight ignore value, as you have done, or alternatively increase the SNR limits. I think the second method is more reliable to expose more the shadows and midtones, regardless of the presence or absence of specular highlights.


Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: garry23 on June 13, 2015, 09:36:58 PM
I set both SNRs to zero, to guarantee no clipping.

If you set SNRs to non-zero you risk clipping.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: garry23 on June 14, 2015, 12:13:11 AM
Alex/josepvm

I thought you may be interested in this post: http://photography.grayheron.net/2015/06/further-le-experiments.html
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: DeafEyeJedi on June 14, 2015, 01:29:09 AM
Definitely a great read and thanks for sharing the link @garry23!
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: josepvm on June 14, 2015, 08:26:23 AM
A nice article, thanks !
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: massimo on September 20, 2015, 09:24:12 PM
Hi,
I tried to use the aettr on my 5d2 but the ISO keep limited to 1600 max, is it by 5d2's auto iso limit?
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: dmilligan on September 21, 2015, 02:11:37 AM
ISOs above 1600 on 5d2 are just digital manipulations and not actually increased analog amplifier gains. Therefore there is no improvement in SNR, and using them is pointless (except for JPEG and h264, but aettr is for raw only). Just increase exposure in post.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: garry23 on September 21, 2015, 07:56:27 AM
If you haven't read this, you may find this of value: http://www.clarkvision.com/articles/iso/
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: rsmith02 on April 11, 2016, 02:02:43 PM
[nevermind, redownloaded the package and found this module. It would be nice to have the non-included ones in one place to download, though.]
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: Dionisgr on July 13, 2016, 03:44:06 PM
Good time of day. If you use auto ETTR all the shots turn out overexposed! In Sunny weather, to remove not possible! Is this normal?
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: Walter Schulz on July 13, 2016, 03:46:19 PM
Sample and settings used would be fine.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: Dionisgr on July 13, 2016, 04:00:43 PM
I apologize for my poor knowledge! Could you point me to where I can find step by step instructions to use Auto ETTR
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: Walter Schulz on July 13, 2016, 04:27:49 PM
Start with https://builds.magiclantern.fm/#/features -> Modules -> ettr (link)
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: Dionisgr on July 13, 2016, 04:39:20 PM
A huge thank you! ;)
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: garry23 on July 13, 2016, 06:31:40 PM
Also put all SNRs to off or zero.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: mr.smith on June 12, 2017, 12:28:09 AM
Many thank you for ML team.

Can you be able to chose MAX ISO(like slowest shutter) when using ETTR?

I am using ETTR and it is very very useful.
But when I am in the dark, the ISO will be 12800(it is 5D3's max ISO) soon.
If I shoot with 12800 ISO, it always appears dead/hot pixels.
SO, I want to chose max ISO(such as 4000) when using ETTR.

If it is possible, please create it on next build.
(Both Nightly Build and Experimental Build, please.)


*Canon 5D3
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: a1ex on June 12, 2017, 01:35:38 AM
Asked and answered (http://www.magiclantern.fm/forum/index.php?topic=5200.msg107005#msg107005).

I'd be also interested to see a proof that ISO 4000 (or other close value) gives less hot pixels compared to ISO 12800 (at identical shutter speeds, of course).
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: mr.smith on June 27, 2017, 04:08:52 AM
Hi,

I uploaded the pics.(5D3 1.2.3 2017March30build, 1920x1080)
Lower ISO also has dead/hot pixel.
But it does not stand out and less than higher ISO.
Therefore, I want to be able to chose MAX ISO(like slowest shutter) when using ETTR.

The ISO will be 12800(it is 5D3's max ISO) soon, although I am in not so dark.
I always think "It does not need 12800 now." and change to 3200 or 6400.
Higher ISO is noiser, so I don't want to use ISO 12800 anymore.

If it is possible, please create it on next build.
(Both Nightly Build and Experimental Build, please.)

ISO3200

(https://thumb.ibb.co/hAqgA5/3200.jpg) (https://ibb.co/hAqgA5)


ISO6400

(https://thumb.ibb.co/bHCHiQ/6400.jpg) (https://ibb.co/bHCHiQ)


ISO12800

(https://thumb.ibb.co/c0QsHk/12800.jpg) (https://ibb.co/c0QsHk)


Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: dmilligan on June 27, 2017, 04:20:29 AM
Quote from: mr.smith on June 27, 2017, 04:08:52 AM
Higher ISO is noiser
Sounds like you didn't read the link a1ex posted in the previous response.

Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: a1ex on June 27, 2017, 08:05:59 AM
To compare the pictures, you need to render them at equal brightness (using exposure adjustments in post).

Here's a similar test done with 6D (http://www.magiclantern.fm/forum/index.php?topic=10028.msg97321#msg97321).
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: Psih on July 30, 2017, 10:34:17 PM
ETTR use 1/2 stop for change exposure. Is it true? If yes, why not 1/3 stop?
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: a1ex on July 31, 2017, 02:41:55 PM
To keep the math simple.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: Psih on July 31, 2017, 11:05:08 PM
I have some problem, when I using ETTR with 1/2 stop for shooting timelapse and postprocess it with LRTimelapse. LRTimelapse can work only with step 1/3 stop. Can I change some settings ETTR for change step from 1/2 to 1/3?
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: a1ex on August 01, 2017, 12:40:54 AM
Quote from: Psih on July 31, 2017, 11:05:08 PM
I have some problem, when I using ETTR with 1/2 stop for shooting timelapse and postprocess it with LRTimelapse. LRTimelapse can work only with step 1/3 stop. Can I change some settings ETTR for change step from 1/2 to 1/3?

You'll have to change the code, and it's not trivial. You'll need to disallow 1/2-stop exposures and account for the rounding used in 1/3-stop mode (actually 3/8 and 5/8).

To my knowledge, LRTimelapse is a commercial program, so you might have better luck with feature requests there.

That said, we have our own deflickering tools that do not use the EXIF info at all. Some examples:
- Deflicker module (http://www.magiclantern.fm/forum/index.php?topic=10496) (original version, not very practical, as deflickering is best done in post)
- Bridge script (http://www.magiclantern.fm/forum/index.php?topic=8850) (similar algorithm, but requires proprietary software)
- mlvfs --deflicker (http://www.magiclantern.fm/forum/index.php?topic=13152) and mlv_dump --deflicker (http://www.magiclantern.fm/forum/index.php?topic=18975) (for MLV timelapse, e.g. silent pics or raw video)

The same deflickering algorithm is available in Darktable.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: Psih on July 06, 2018, 12:32:32 PM
The smaller the steps, the better the results - therefore I need this option and I am ready to pay for this option. Can somebody do it?
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: Audionut on July 06, 2018, 01:58:21 PM
Quote from: Psih on July 06, 2018, 12:32:32 PM
The smaller the steps, the better the results

Proof.  Use images, we already understand the science.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: Psih on July 06, 2018, 02:12:06 PM
I'm getting better results, when I stop shoot with ETTR and start use qDSLRdashboard. The transition between day and night is more smooth. I can't provide any proof, only my experience on shooting timelapse. I'm regularly shoot timelapses and I want this option and I can pay. If it is impossible, I will search another way (timelapse+ for example).
I'm sorry for double post.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: Audionut on July 06, 2018, 02:17:40 PM
Quote from: Psih on July 06, 2018, 02:12:06 PM
The transition between day and night is more smooth.

Post processing issue.  The following may help.

Quote from: a1ex on August 01, 2017, 12:40:54 AM
That said, we have our own deflickering tools that do not use the EXIF info at all. Some examples:
- Deflicker module (http://www.magiclantern.fm/forum/index.php?topic=10496) (original version, not very practical, as deflickering is best done in post)
- Bridge script (http://www.magiclantern.fm/forum/index.php?topic=8850) (similar algorithm, but requires proprietary software)
- mlvfs --deflicker (http://www.magiclantern.fm/forum/index.php?topic=13152) and mlv_dump --deflicker (http://www.magiclantern.fm/forum/index.php?topic=18975) (for MLV timelapse, e.g. silent pics or raw video)

The same deflickering algorithm is available in Darktable.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: Psih on July 06, 2018, 03:05:36 PM
Post processing the same in both case (ETTR and qDSLRdashboard) and I don't think that problem is in one. I think, if this feature is so difficult, I can find another way. Timelapse+ cost 400$. If nobody can't solve this problem, this is sad, but I will buy Timelapse+.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: Audionut on July 06, 2018, 03:49:51 PM
If you're not willing to spend time solving the post processing issue, part with your $400.  That choice is entirely yours.

Exposure matching is exposure matching.  Doesn't matter how different the images are, whether those images are captured with 1/3 stop or 1/2 stop exposure settings (exposure is changing during day > night even with exact same camera exposure settings), there is an exposure difference, and this exposure difference needs to be calculated.

If the post processing software is struggling with the difference between camera exposure settings (1/3 vs 1/2 stop), then the issue lies within the post processing software.

Having narrower exposure differences is always better (from a pure quality standpoint).  Mostly for pixel peeping, but credit where it's due.  However, and to answer your other question.

Quote from: a1ex on August 01, 2017, 12:40:54 AM
You'll have to change the code, and it's not trivial. You'll need to disallow 1/2-stop exposures and account for the rounding used in 1/3-stop mode (actually 3/8 and 5/8).

I would advise leaving the comfortable box, trying other post processing software, before parting with $400, but it's your $'s and your decision.  Personally I've had good results with the bridge script, but I was only playing, nothing serious, and I've not used the mlvfs or mlv_dump options.

Quote from: a1ex on August 04, 2013, 07:38:08 PM
For me, the end goal is not to provide a finished product that everybody can consume. I'd rather see it as an open software platform where others can program their own enhancements and share them with the community

This is a development based community, where we contribute with skills and knowledge, not monetary offers.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: a1ex on July 06, 2018, 06:45:53 PM
Just FYI - an older example with the deflicker algorithm used in the tools mentioned earlier:



Don't know whether the transition is smooth enough for your needs or not, but I'm pretty sure that using 1/3 stops instead of 1/2 won't make any noticeable difference. You may notice exposure jumps much larger than 1/2 EV in the input video (the one displayed in smaller size).
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: c_joerg on July 09, 2018, 01:56:15 PM
Quote from: Audionut on July 06, 2018, 03:49:51 PM
Exposure matching is exposure matching.  Doesn't matter how different the images are, whether those images are captured with 1/3 stop or 1/2 stop exposure settings

Just for understanding.
This video is made with 1/3 stops end I can't correct the pumping sun in post because the core of the sun is blown out.

If I were to create these recordings with ETTR then you could probably correct it but then I would lose a lot of information especially in the dark areas.

I really like the CHDK part where I can change exposure in 1/96 stops...
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: Audionut on August 03, 2018, 11:38:20 AM
Quote from: c_joerg on July 09, 2018, 01:56:15 PM
If I were to create these recordings with ETTR then you could probably correct it but then I would lose a lot of information especially in the dark areas.

There's built in protection for shadows. (https://www.magiclantern.fm/forum/index.php?topic=5693.msg40369#msg40369).

My bold.
Quote from: a1ex on August 25, 2013, 09:11:48 PM
Last week I had some time to play with ETTR and I think I've found a way to avoid the severe underexposure in scenes with high dynamic range (e.g. like this (http://www.magiclantern.fm/forum/index.php?topic=7835)).

https://bitbucket.org/hudson/magic-lantern/commits/fcdc23eb5d7d

How it works:
- If the midtones get too noisy, the ETTR algorithm will stop underexposing.
- Same thing if the shadows get too noisy.

How to quantify "too noisy"?

For midtones it's easy: the median "brightness" gives you the signal level, so we can use the SNR. So, when I say the midtone SNR is 5 EV, this means half of the image has a SNR less than 5 EV, and the other half has a SNR higher than 5 EV. Pretty easy and statistically robust.

For shadows, I've chosen the 5% percentile. So, in this context, the shadows having a SNR of 3 EV means 5% of the image pixels have a SNR lower than 3 EV, and the other pixels will be brighter than that.

Keep in mind that 5% is bigger than you may think, since it refers to image area, not linear size. The linear percentage is roughly 22% (1/4.5).


So, you can choose 2 noise limits, one for midtones and one for shadows, and the image will not get noisier than that (but you may lose some highlights). This setting has the highest priority, so it will sacrifice highlights even if you disallow clipping from the other settings.

I'm not yet sure what are some good defaults for this new trick, so it's turned off by default.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: mothaibaphoto on November 10, 2018, 03:16:26 PM
ETTR in recent builds is not usable in low light.
I used to shoot a lot of timelapses before 4K video era begins.
I even shared my hard earned settings for getting amazing sunrises:
https://www.magiclantern.fm/forum/index.php?topic=18730.msg178621#msg178621
Recently i tried to shoot timelapse with recent build and failed.
Now i try to sort out whats wrong.  What i found: as long, as i reduce light, the image on LCD gets more pixelated,
histigram shows overexposure, ETTR shortens exposure(!!!), but screen gets only brighter and brighter and, finally complete white.
It saves completely black DNG, of course.
I compared the ettr source of recent build with the old one.
I found suspicious new code:
if (overexposed_percentage > 0 && (auto_ettr_midtone_snr_limit || auto_ettr_shadow_snr_limit) && !dual_iso)
I enabled DualISO, to test, whether this is the source of problem - and it disappeared.
Instead of white screen I gets correctly exposed images with several second exposures.
I decided to comment this out and try without DualISO - problem persist.
DualISO fixes the problem somehow, but that code is not the source of it.
So, i stuck on it, have no more ideas, my humble skills don't allow me to find why it used to work but no more :(
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: a1ex on November 10, 2018, 03:56:20 PM
Can you upload some sample images used by ETTR to meter from?

In other words, if you have used ETTR outside LiveView, the CR2 files should be enough, but if you have metered in LiveView, you will need to provide DNG frames from LiveView.

Quote from: mothaibaphoto on November 10, 2018, 03:16:26 PM
What i found: as long, as i reduce light, the image on LCD gets more pixelated,

If you have attempted to meter long exposures from 1/30" LiveView frames... I'm not sure what I can do.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: mothaibaphoto on November 11, 2018, 12:08:24 AM
Quote from: a1ex on November 10, 2018, 03:56:20 PM
If you have attempted to meter long exposures from 1/30" LiveView frames... I'm not sure what I can do.
Yes, it's seems that is the point: ETTR (not me) tries to calculate exposure from that mess.
Finally, it deals with completly white LCD preview, but saves completely black DNG.
And only if DualISO is activated, it calculates exposure from actually taken images.
How can I avoid this? As I mentioned, I shoot timelapse:
Photo mode Live view, silent Full-Res, compressed DNG (tried MLV, the same story).
It used to work all the time this way.
I think it's unnecessary to upload the perfectly black DNG.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: a1ex on November 11, 2018, 12:50:19 AM
Quote from: mothaibaphoto on November 11, 2018, 12:08:24 AM
I think it's unnecessary to upload the perfectly black DNG.

If ETTR metered from there, the image might be relevant.

I've tried to reproduce the issue on crop_rec_4k build and I think I've figured it out. During the timelapse, camera stays outside LiveView, in powersaving mode, with mirror up; the screen turns on for a few seconds for reviewing each image. ETTR is definitely metering from captured images.

When pointing the camera from some relatively dark scene (after it settled to 0.5" at ISO 100) to some very dark scene (that would have required 16" at some higher ISO), the first image was nearly black (as expected) and ETTR switched the exposure to 1/8000 (without any intermediate steps). That only happens when it believes the image is extremely overexposed.

Checking the white level of the black DNGs revealed values below 2500, which is clearly incorrect. This appears to be introduced by a change discussed around here (https://www.magiclantern.fm/forum/index.php?topic=20579.msg190437#msg190437): in photo mode, ML raw backend tries to guess the white level starting from the value reported by Canon (as that appeared to be a better approximation than some hardcoded value, i.e. what earlier builds did). The issue is that in FRSP mode, unlike regular photo mode, Canon code does not report any white level; it reports 0, so ML ended up scanning for the brightest pixels in the actual image. If these brightest pixels appear as a peak on the right side of the raw histogram (quite likely to happen with a dark frame), ML assumes the image is overexposed and picks a white level right below that "pitch black".

Please try some less bleeding-edge build (either unified or lua_fix) and report back.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: mothaibaphoto on November 11, 2018, 06:40:09 AM
Thank you, a1ex, you are very helpful, as always!!!
I updated that code in raw.c: when silent enabled, your old code is running:

    if (!lv)
    {
if (is_module_enabled("silent"))
{
/* at ISO 160, 320 etc, the white level is decreased by -1/3 EV */
/* in LiveView, it doesn't change */
int iso = 0;
if (!iso) iso = lens_info.raw_iso;
if (!iso) iso = lens_info.raw_iso_auto;
static int last_iso = 0;
if (!iso) iso = last_iso;
last_iso = iso;
if (!iso) return 0;
int iso_rounded = COERCE((iso + 3) / 8 * 8, 72, 200);
float iso_digital = (iso - iso_rounded) / 8.0f;

if (iso_digital <= 0)
{
raw_info.white_level -= raw_info.black_level;
raw_info.white_level *= powf(2, iso_digital);
raw_info.white_level += raw_info.black_level;
}

raw_info.white_level = autodetect_white_level(raw_info.white_level);
raw_info.dynamic_range = compute_dynamic_range(black_mean, black_stdev_x100, raw_info.white_level);
}
else
{
/* start at Canon's white level, and autodetect from there
* Canon's guess may be up to 0.38 EV below the true value - or maybe more?
* http://www.magiclantern.fm/forum/index.php?topic=20579.msg190437#msg190437
*/
int canon_white = shamem_read(0xC0F12054) >> 16;
raw_info.white_level = autodetect_white_level(canon_white);
raw_info.dynamic_range = compute_dynamic_range(black_mean, black_stdev_x100, raw_info.white_level);
printf("White level: %d -> %d\n", canon_white, raw_info.white_level);
}
    }


That fixes the problem, but... now it occasionally ens up stuck with "Raw error".
Need long(5+ sec) exposures, high ISO(400+) and DualISO disabled.
There are 6 occurences of that message in source, so I made all 6 different and found,
that it is the first one in the auto_ettr_step() procedure in ettr module.
It looks like i leave DualISO enabled even if overcast :)
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: a1ex on November 11, 2018, 07:00:59 AM
Quote from: mothaibaphoto on November 11, 2018, 06:40:09 AM
now it occasionally ens up stuck with "Raw error".

I highly doubt leaving dual iso enabled is going to fix this. The code snippet you have found simply selects some different heuristic for fixing overexposed highlights (a quirk that I need to fix somehow, but not related to your errors).

The "raw error" coming from raw.c also has different causes. Look for this line:

    else if (QR_MODE) // image review after taking pics


and you'll be able to see these conditions. Wherever the code is returning 0, that's some error condition. If you turn these dbg_printf's into plain printf's, you'll see these messages on the console.

Most likely, the failure reason is some black level check gone wrong, possibly caused by hot pixels. Look for this line:

            /* for debugging: if black check fails, save the bad frame as DNG */


and enable that block, i.e. turn it into if(1). You'll get these bad frames saved as DNGs on the card.

These DNGs will allow me to re-run the entire thing (raw overlays + ETTR code) from QEMU, on exactly the same image that caused trouble.

Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: mothaibaphoto on November 11, 2018, 07:38:46 AM
That debugging kinda works... once, and crashes the camera with ERR70 on top LCD :)
Nethertheless, here is bad dng and 3 generated crash logs:
https://www.dropbox.com/sh/8h4ssn4l8v1mjjo/AABFZ9T1zwDKMFhSJWakZxAfa?dl=0
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: a1ex on November 11, 2018, 07:51:54 AM
Indeed, a hot pixel in the optical black area. Need to use robust statistics for checking the noise levels, i.e. MAD instead of stdev.

That ERR70 must be caused by ML allocating a huge chunk of memory while Canon code is still processing the image. Adding some delay might fix the error. Removing that malloc and saving the DNG directly might work, too. Both changes will interfere with LiveView (where I had to use that debugging code).
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: mothaibaphoto on November 11, 2018, 09:02:20 AM
Ok, thank you!!!
I had that "Raw error" occasionally from time to time before.
It's not so serious problem due it's rare.
At least, now I know that it could be avoided by DualISO( I'm sure, tried several times).
That incorrect canon's white level with silent was real show stopper, and gladly it fixed.
Title: Re: Auto ETTR based on RAW histogram (ettr.mo)
Post by: yoback on November 14, 2019, 12:27:57 AM
Hey, can someone tell me where can I download it for canon 6d? I was serchin for a 2 hours..