Messages

Saturate offset is not adjusting the black level per se, it's only adjusting the offset*.
B/W offset isn't any form of gain, it's digitally shifting the analog signal into the recorded bit depth.
...
*The offset is so far down into the noise floor that it does not effect black in the recorded image.  The offset can be adjusted higher then the default value, but there doesn't appear to be any gain to DR.

It's a bit of an aside, but I object to your characterizing SaturateOffset as only an "offset", and the implication that changing SaturateOffset is not useful for improving DR.  However, perhaps I am reading too much into your choice of words.  SaturateOffset changes the digitized value of the black level without changing the digitized value of the white level, which has DR consequences.

In my mental picture of the electronics, there are two relevant DACs:

• The first DAC produces the reference voltage for the black-level clamp, and this clamping occurs before the ADTG/DFE amplification stage.  The corresponding register is SaturateOffset (0xc0f0819c).
• The second DAC produces an analog offset that is added to the signal after both amplification stages, before it is fed into the ADC.  The corresponding register is B/W offset (0xC0F08034).

Thus, changing B/W offset just shifts the digitized values because it just shifts the signal up or down in analog space.  There is no change in the dynamic range because the black level and white level move together and the read noise stays the same.

On the other hand, SaturateOffset is more than just an offset because it comes before at least one stage of amplification.  At the output of the amplifier, it becomes directly related to the separation between the black level and white level (which is the saturation at the rail of the amplifier).  Increasing SaturateOffset decreases this separation, meaning that saturation occurs at a lower analog signal, so your DR must decrease.  Conversely, decreasing SaturateOffset must increase your DR.  (The changes in DR may be negligible if the register change is small).

For example, at ISO 100 on the 50D, you can achieve most of the DR improvement by just changing these two registers, without touching any amplifier gains.  (While changing B/W offset itself doesn't change DR, increasing it provides more room to decrease SaturateOffset.)

Presumably yes, but as is the way with such things, only "when it's ready".

My guess is that this will probably be it.  While I can't rule out the discovery of additional levers to manipulate, it seems rather unlikely.

any news about expanding DR for 50D?

The latest data shows small improvements to dynamic range: ~0.25 EV at ISO 100, and about half that at higher ISOs.

Here's the table of all the data I've collected for the 50D.  The highlights:

• Most of the x160 ISOs are digital pulls from the next-higher ISO, so have the same dynamic range as that full ISO.
• Normal ISOs greater than 1600 are analog pushes using only DFE gains, and there is no reduction in read noise as measured in electrons (the read noise in ADUs scales directly with ISO).  Thus, using ISOs greater than 1600 throws away highlights without any improvement in the shadows.
• The two expanded ISOs are digital pushes that multiply the ISO 3200 data by 2 and 4.  The discreteness can be seen in the CR2 files.

Link to Google Docs

Okay, I think I understand what's causing the confusion.  The January 7th fix made sure that the LV overlay and the ML menu show the same shutter speed: the one that LV is actively using to simulate the exposure.  That is, if you have a certain shutter speed dialed in, but you stop down the aperture or increase the ISO from the lowest, LV uses a different shutter speed than the one you have dialed in to simulate the effect.  For example, with a maximum aperture of f/4:

• Exposure settings are ISO 100, f/4, and 1/500s => displayed shutter speed is 1/550s
• Exposure settings are ISO 400, f/4, and 1/500s => displayed shutter speed is 1/140s

That's because LV is using a shutter speed a factor of 4 slower than the dialed-in shutter speed to simulate the higher ISO.

Prior to the January 7th fix, the LV overlay would show the dialed-in shutter speed rather than the shutter speed being used by LV.  However, since RAW video is simply recording the LV feed, the dialed-in shutter speed is the incorrect one to display.  You can easily demonstrate this to yourself by trying to dial in a shutter speed longer than the frame cadence, such as 1 second.

Does this make sense?  If you still think that the displayed shutter speed is wrong, you can try to prove it by taking a video of a moving source and looking at the blur.  Note that you can't necessarily trust the metadata in the RAW video because I'm not sure it's being saved entirely correctly.

[magic.cfg]

I don't have ML installed at the moment. On TL the Headers are: Expo; Overlay; Movie; Shoot; Focus; Display; Prefs; Modules; Debug; Modified.

I don't see any help which shows the TL version.

Ah, yeah, I forgot that TL has the "Help" tab disabled.  You can look up the build information in the generated text file magic.cfg in the ML/settings folder on your card.

I assumed there would be a way to write/print all the current camera setting to a log file for transfer to a computer. If it's not available at the moment would it be easy to introduce? It might save some time when people are looking for help on the forums. Maybe it's worth adding? I was going backwards and forwards between the two cards and I thought I'd like a paper print out and I could do a "spot the difference" and find the setting which was causing the problem.

The settings can be found in the *.cfg text files in the ML/settings, which is very similar to what you are asking for.

Edit: revised method to find build info

Problem somewhat understood now. With the latest build, the shutter speed values display wrong. Thinking my shutter speed was 1/48, it was actually something like 1/390. Had to switch to an earlier build to find correctly displayed shutter speeds. Hopefully this gets fixed at some point.

How much earlier of a build did you go back to to get the correct shutter speeds?  I fixed a bug with the shutter-speed display on January 7th, but it's entirely possible that there are still issues.  Can you provide more details about which exposure mode you're in and how you are changing your settings?

I'm sure there will be an easy way to get each card to print out all the settings so I can compare them but it wasn't obvious to me.

You can go to the "Modified" tab in the ML menu (all the way to the right).  All the settings that are different from their defaults should be listed here.

What's the best way to check which version is on the camera? It seems like a Noob question.

You can go the "Help" tab in the ML menu, and there should be lines at the bottom that will give you build information.

Nice. On 50D, from your older results, I believe you can recover all the highlights only via Saturate Offset (and you might even benefit from *increasing* the DFE gain, though it's probably not noticeable).

However, I remember reading 50D has its ISO 100 somehow pulled from 200 (not sure exactly how), so things may be different at ISO 200 and above.

The low white level of 13432 is the case at only ISO 100.  At higher ISOs, the white level is 15760, so just like with the 5D2, decreasing the DFE gain could be useful.  (This does suggest that ISO 100 is handled differently in some way...)

Can you tell me some default values for these DFE registers and send me a RAM dump via PM? (Debug -> Dump ROM and RAM -> RAM4.bin). You may also look them up yourself in the hexdump and only show me the relevant section (e.g. if register 1d02 is 0x456 and you search for 0x1d020456, you will find something that looks like an array of values for these registers; it should be fairly obvious where this array starts and stops).

As an example, the default values for ISO 100 are:
1d02 = 0x419
1d04 = 0x41a
1d06 = 0x41e
1d08 = 0x41c

I'll send you the array by PM.

@ayshih, can you write down a table with register values at all the ISOs? (photo mode, of course)

Okay, I've already have some partial tables, but I might as well put together a comprehensive table tomorrow.

5D2 mistery solved!!!

Keyword: SendDataToDfe (it's yet another catgory of registers). Updated ADTG GUI.

For the 50D:

Code Select Expand

        SEND_DATA_TO_DFE_FUNC = 0xffa71598;


I confirmed that each of the four 1d0x registers changes the gain of a fourth of the columns, but I haven't yet done anything more (e.g., generating the analogue of your 5D2 graph).

Are the whites recorded a solid white (16383)? I'd like to see the CR2 files.

The green channels are solid white at 16383; the red channels are at 16065, and the blue channels are at 16001. ...  What needs more investigation is the "rise" before each channel reaches its max with the tweaked settings.

I take that back.  It looks like the overexposed regions are not solid white, and the pixel-to-pixel differences in saturation is what is producing that "rise" in the histogram.  Here are plots of where the pixels have values shy of the overall white level (in one of the green channels), for the default settings (13200..13400) and the tweaked settings (16100..16350).  I'd expect to see just the border of the overexposed region.


There does not appear to be any noticeable banding in the per-column black level when looking at the top black bar.

Are the whites recorded a solid white (16383)? I'd like to see the CR2 files.

The green channels are solid white at 16383; the red channels are at 16065, and the blue channels are at 16001.  Here are the CR2 files for default settings and tweaked settings.

I haven't looked too closely at the CR2 files, but here's a quick comparison of their histograms (with the channels lumped together).  What needs more investigation is the "rise" before each channel reaches its max with the tweaked settings.

(I probably need to include Canon's digital gain in my formula; can you check the default value of c0f08030 at iso 100?)

At ISO 100, the default value of digital gain (0xc0f08030) is 0x1000, so I did not change it.

Yes, old-generation cameras do not use 888x registers from ADTG (there may be some other registers).

For 5D2, I've got a tiny improvement (0.15 stops) by setting SaturateOffset (0xc0f0819c) from 0x66f to 0x66f + 32 - 1024 - 624 and B/W offset (0xC0F08034) from 0x1991 to 0x1991 + 624.

On my 50D, the white level at ISO 100 is only 13432, so I can adjust those same registers to expand the range from 1023..13432 to 31..16383, which gives an apparent DR increase of 0.4 EV.



I increased B/W Offset (0xc0f08034) from 0x1079 to 0x1c00 to max out the white level, and decreased Saturate Offset (0xc0f0819c) from 0xf87 to 0x20.  (I just now realized that these are essentially the same settings you cited for the 5D2.)  I find it curious that not only is this setting for B/W Offset "nice" as a multiple of 256, it also shifts the meaning of the Saturate Offset to correspond to the actual empirical black level (0x20 ~= 31).

[adtg_gui.mo]

For adtg_gui.mo, I've tested adding the DIGIC hooks for the 50D by grabbing the addresses from stubs.S, and they appear to work fine.  I can see registers such as SaturateOffset (0xc0f0819c) , and I can still see ADTG registers (which is mentioned in a comment as an issue for the 5D Mark II).

Code Select Expand

    else if (streq(camera_model_short, "50D")) // http://www.magiclantern.fm/forum/index.php?topic=6751.msg63322#msg63322
    {
        ADTG_WRITE_FUNC = 0xFFA11FDC;
        CMOS_WRITE_FUNC = 0xFFA12190;
        ENGIO_WRITE_FUNC = 0xFF97D904;  // from stubs
        ENG_DRV_OUT_FUNC = 0xff97d794;
    }


Have you tried making small adjustments to 0x8880?

I'm still getting the hang of looking at ADTG registers, but I don't believe the 50D uses that particular register, so I don't know how to change the black level.  (As an aside, I also haven't figured out how to change the ADTG gains on the 50D.  Inspired by Greg's testing with the 500D, I tried overriding the ADTG3[0] and ADTG3[3] registers, but they don't appear to affect anything.)

Thinking more about the effect I've described, it looks like the black-level clamp just has a characteristic time for clamping to the reference voltage (presumably generated by the register value and a DAC), and actually oscillates a bit about the desired level as it gets there.  The amplitude of that oscillation then corresponds to how much the clamp needs to shift the signal in voltage.  If you consider the signal on a given line in an AC-coupled sense, a massive overexposure have a high average charge per pixel, which means that pixels in the optical-black region will be very far from the reference voltage, and the resulting amplitude of the oscillation to be large.  The 5D Mark III presumably has a different, more effective black-level clamp.

If that's the right description, my guess is that there'd be no benefit from small changes to the black level.  Changing the black level by a significant amount could help to stabilize the optical-black region for images that are significantly overexposed, but would compromise normally exposed images, so that doesn't really make sense as a change.

Edit: I got a little mixed up about the direction of the needed change.

@ayshih: for these cameras I would point the camera to a light bulb in the middle of the image, and expose until the edge pixels become dark. If you have like 1% of the image overexposed, it's enough for DR calculation.

Yup, agreed.

[raw_diag.mo]

As discussed on a recent pull request, a massively overexposed frame affects the left optical-black region for the 50D (but does not in the 5D Mark III).  This effect complicates determination of the black level and the read noise, and consequently can throw off the measurement of dynamic range.

The first plot shows the histogram for raw_diag.mo with only minor overexposure.  The measured read noise is comparable to that of a dark frame with the same exposure settings.  The second plot shows the histogram when the image is massively overexposed.  You can see that the read noise appears to be larger, which appears to reduce the dynamic range by ~0.2 EV.


Diving into the images themselves, here are the histograms for pairs of columns, starting from the left edge of the sensor (the bottom-most line) and going to just before the image itself (the top-most line).  The solid lines (as opposed to the dashed lines) are the columns in the inner optical-black region used by raw_diag.mo to determine the black level and the read noise.  The first plot, corresponding to the minor overexposure, shows that all of the columns in the inner optical-black region have about the same mean and standard deviation.  The second plot, corresponding to the massive overexposure, shows that all of the columns have about the same standard deviation, but there is significant drift in the mean, and averaging over this drift is what is producing the appearance of a larger read noise and a slight decrease in the black level.


Thus, the read noise isn't actually changing, which is good, but the apparent drift of the black level is not good.  I don't know whether this points to some form of sensor bleed or a deficiency of the black-level clamp.  Either way, on cameras with sensors like the 50D's, take care not to overexpose images by too much.

[raw_diag.mo]

Regarding raw_diag.mo, this is a very minor quibble, but you may want to tweak the definition of the optical-black region for the left black bar.  You start at y = 20 instead of y = raw_info.active_area.y1 + 20 as is done in raw.c.  The 50D, at least, has a structured pattern for the first ~thirty lines of the top black bar, so starting the optical-black region that early picks up that pattern.  It's so few pixels that it probably doesn't really affect any histograms, but you might as well play it safe.

First, that thread is (now) in the forum for Tragic Lantern, which is a more experimental and possibly more unsafe fork of Magic Lantern.  The Magic Lantern nightly builds for the 50D are pretty stable and full-featured (including RAW recording), so I encourage you to stick with them rather than immediately going to more experimental builds.

Second, the first post in that thread hasn't been updated to reflect the progress since June 2013, so you're effectively using an outdated version of Tragic Lantern, which seems like an especially bad idea.  If you wish to continue using Tragic Lantern, you can find the most recent Tragic Lantern builds linked near the end of that thread.

2) Then take the modified files from http://www.magiclantern.fm/forum/index.php?topic=5586.0 and put them in their respective locations (".DS_Store" and "autoexec.bin" in the root directory, then place the "MODULES" folder and the other ".DS_Store" file from there too  into the "ML" folder) (unlike other guides, the MODILES folder don't exist), now for simplicity's sake, in the "ML" folder, make a new folder called "scripts" for no reason (it will make booting faster because the camera looks for it)

It sounds like you are installing a significantly older modification of Magic Lantern.  Using nightly build for the 50D will smooth out some of the parts of your guide (e.g., issues with the "scripts" folder), and more importantly, the nightly build will likely perform better than the version you are running.  You should update to the nightly build unless you have a specific reason not to.  After you have completed your step 1, make sure to delete *everything* before you copy over the files from the nightly build (which contains the whole directory structure, including the "modules" and the "scripts" folders).

If that last 0.1-0.2 EV of shadow detail is more important then 1EV of highlight detail, you may benefit from ISO 6400 vs ISO 3200. ...

Thanks, Audionut, for that detailed explanation.  I was getting a bit confused by the discussion of noise and the claim of "fake" ISOs, and your explanation (as well as one of your links) helped greatly to straighten me out.  In essence, going to higher ISOs reduces the read noise as measured in equivalent electrons, but once the read noise is much smaller than the photon shot noise, the SNR will improve only slightly at best.  Thus, ISOs >~ 6400 can appear to be "fake" even if they are real analog gain, simply because you're already dominated throughout the image by photon shot noise.

The ML module implements the method faithfully, so you're asking whether the method is dependable.  For my own part, I've been very satisfied with DotTune.  When I first upgraded to the 50D, I painstakingly made lots of test shots and manual adjustments to figure out good AFMA values for my lenses.  Then I found out about DotTune, and I was pleased (but frankly also a little irritated!) with how easy it was, and produced AFMA values very close (<~1 tick difference) to what I had already determined.

Now, there are definitely times when DotTune misfires and gives a bad result, but I chalk that up to user error, such as not setting the perfect focus initially or accidentally bumping the camera when switching to manual focus.  Running through the procedure a few times can help confirm whether your value is trustworthy.

I suspect that those that see consistently incorrect AFMA values from DotTune are making some mistake in the setup/procedure and not realizing it.

hello, I am using ML with 5D2. Sometime between Jan 11 and Jan 18 the temperature display was messed.
Instead of the temperature in C and F a 3 digit value is displayed like: 144 raw

There's been a recent push to confirm that the temperatures were being accurately converted from the raw value for each camera, and cameras for which calibration data were not taken yet were reverted to display just the raw values for temperature.  See this thread for more information.  Temperature measurements for the 5D Mark II were just provided yesterday, so I imagine you'll have human-readable temperatures again in the nightly builds in the near future.

@ayshih: Thanks for your reply.
The 60D hast no AF configuration, that is one missing feature to the 50D, propably to make a bigger difference to the 7D at this time.

I use the 70-200 IS II.

I tried it again, the values in the original Canon menu are the same as in the DotTune module menu, vice versa - so this works.
I tried with -20 to 20 manual and with the option "all lenses" instead of "this lens" - this seems to be the problem. "all lens"es works fine. There might be a problem, that its a zoom lens? In the Canon menu I got two values W and T (Wide and Tele it seems), but the one value in the DotTune module seems to stores to both settings (propably ok), but I again get these problems, that I even get at -100 and +100 a focus indication.
Anyway, no matter... I can use the "all lens" option and remember the values at 70mm and 200mm (and other lenses) and can store these manually with the original Canon menu. It is a big hit.... but "unfortunately" my 70-200 after the test want to set a "0" value - and this seems ok - because this lens IS very sharp - i never had problems with it. ;-)

Ah, yes, I forgot that AFMA is missing from the 60D.  It sounds like your specific lens has not been registered in your camera in the way that initializes everything properly for the DotTune module.  See these posts from the DotTune thread:

http://www.magiclantern.fm/forum/index.php?topic=4648.msg29451#msg29451
http://www.magiclantern.fm/forum/index.php?topic=4648.msg29772#msg29772

How about, using "this lens", try first setting arbitrary AFMA values to W and T, different from each other, and then trying the DotTune module again.

Regardless, this is not an issue specific to the nightly builds, so further discussion should probably move to the DotTune thread.

Walter.  You haven't really been of any help.  All your post come off as smart mouthed negative posts.  If you can't be more positive then don't respond to my queries.
I got some good positive feed back from a few others that helped.  That is what the forums should be about.

I disagree with your reaction; I have the same questions as Walter.

So knowing this, you can't shoot video on the CF cards because the CF card with the ML firmware is taking up the only available slot on the 5DMKII.
I was doing all this to get a clean image recorded to an Atomos Ninja 2.  I assume then that the camera will send the video to the recorder vs. the CF slot.

I don't understand why you seem disconcerted that you are required to have the CF card with the ML code on it in the camera.  Just because the ML code is on the CF card doesn't stop you from recording video (either H.264 or RAW) to that same CF card, as many people on this forum can attest to.  And, regardless, if you're recording video through the HDMI connection using the Atomos Ninja 2, and not using the ML's capabilities to record video, it would appear to sidestep any issue you may have with the CF-card slot.  Walter was confused, and so am I.

Are you saying that I need to keep a designated CF card with the downloaded ML firmware handy to set the camera up anytime I want to use the ML menus to set the camera up for shooting video?  So, when I take the CF card out of my camera and install a clean CF card, I will not have the ML menus.
That seems to be what is happening.  I can't access the ML menus without the card being in place.
The moment I take the CF card out with the ML firmware, the camera only shows the Canon 2.1.2 latest firmware.

Yes, this is the normal behavior.