Dual ISO - massive dynamic range improvement (dual_iso.mo)

Started by a1ex, July 16, 2013, 06:33:50 PM

Previous topic - Next topic

0 Members and 4 Guests are viewing this topic.

a1ex

Vanilla processing:

mlv_dump M26-1444-30-frames.MLV --dng --no-fixcp --no-stripes
cr2hdr *.dng --same-levels
dcraw -W -b 4 *.DNG
ffplay -loop 0 M26-1444-30-frames_%06d.ppm


There is some flicker - let's see if it's natural flicker or bug in cr2hdr. MlRawViewer (1 month old is too old, really?) doesn't show any obvious flicker.

The debug options in cr2hdr won't help much, as they are all after matching the exposure between the two sub-images (that is, after scaling the raw data). Let's go back to octave:

for f = dir('*.dng')'
    f.name
    im = read_raw(f.name);
    g1 = im(1:4:end, 2:4:end);
    imwrite(uint8(g1 / 64), [f.name(1:end-4) ".jpg"]);
end


Oh no, the "crawling" effect!


phase = 0;
for f = dir('*.dng')'
    f.name
    im = read_raw(f.name);
    g1 = im(1+phase:4:end, 2-mod(phase,2):4:end);
    phase = mod(phase+1, 4);
    imwrite(uint8(g1 / 64), [f.name(1:end-4) ".jpg"]);
end
system("ffplay -loop 0 M26-1444-30-frames_%06d.jpg")


No flicker obvious to me (just lots of aliasing, as we are previewing at 480x270, without any filtering), so it must be a bug in cr2hdr. Let's try the experimental changes.


cd modules/dual_iso
hg up e72eb09ffa30 -C
make cr2hdr; sudo cp cr2hdr /path/to/executables


Running the first script gives similar result (still no luck).

TODO: fix the math in cr2hdr.c, match_exposures. This is going to be tricky.

Using the data from each sub-image (dark or bright) gives no flicker. The issue appears after matching the brightness between them. The issue is this variation in fitting results:

Linear fit      : y = 0.063813*x + 108.308081
Linear fit      : y = 0.053660*x + 142.179592
Linear fit      : y = 0.062154*x + 105.105966
Linear fit      : y = 0.063109*x + 102.554616
Linear fit      : y = 0.061640*x + 115.000017
Linear fit      : y = 0.063022*x + 114.077421
Linear fit      : y = 0.064436*x + 101.677465
Linear fit      : y = 0.063109*x + 102.554616
Linear fit      : y = 0.063109*x + 108.302178
Linear fit      : y = 0.063285*x + 113.235900
Linear fit      : y = 0.060960*x + 110.928908
Linear fit      : y = 0.061044*x + 109.146648
Linear fit      : y = 0.061725*x + 114.726728
Linear fit      : y = 0.061129*x + 118.265047
Linear fit      : y = 0.062327*x + 104.553750
Linear fit      : y = 0.063022*x + 102.833685
Linear fit      : y = 0.062587*x + 111.972896
Linear fit      : y = 0.057352*x + 132.357319
Linear fit      : y = 0.057992*x + 122.541289
Linear fit      : y = 0.057912*x + 117.377329
Linear fit      : y = 0.053069*x + 140.393025
Linear fit      : y = 0.053290*x + 145.259863
Linear fit      : y = 0.058883*x + 119.573745
Linear fit      : y = 0.061554*x + 105.396036
Linear fit      : y = 0.061982*x + 111.904583
Linear fit      : y = 0.063548*x + 108.517969
Linear fit      : y = 0.064525*x + 101.649343
Linear fit      : y = 0.064346*x + 94.735446
Linear fit      : y = 0.062674*x + 111.319366
Linear fit      : y = 0.064974*x + 106.084113
Linear fit      : y = 0.063109*x + 102.175959


If you have some other footage that shows flicker (in particular, extreme cases), I'd like to see it as well (small cuts like this are OK).

smasry

QuoteMlRawViewer (1 month old is too old, really?) doesn't show any obvious flicker.

I stand corrected; I was sure development of MlRawViewer was discontinued, but am happy to be wrong in this case.

QuoteIf you have some other footage that shows flicker (in particular, extreme cases), I'd like to see it as well (small cuts like this are OK).

I have plenty; I was worried that something's wrong with my 5D3 or my workflow since I seem to be getting flickering in most if not all dual-iso footage, while no-one else seems to be reporting anything. Perhaps it's because I push it beyond your recommendations of 400/1600 or 200/1600 and go all the way to 100/1600, leaving fewer midtones?

Unfortunately, I've deleted a lot of test clips, taken while I was learning, and experimenting with, this feature. Here are more clips from the same shoot, though:

https://www.rojoynegroclub.com/images/tmp/M26-1449.MLV
https://www.rojoynegroclub.com/images/tmp/M26-1450-100-frames.MLV
https://www.rojoynegroclub.com/images/tmp/M26-1451-166-frames.MLV
https://www.rojoynegroclub.com/images/tmp/M26-1454-48-frames.MLV
https://www.rojoynegroclub.com/images/tmp/M26-1455-66-frames.MLV
https://www.rojoynegroclub.com/images/tmp/M26-1458-52-frames.MLV
https://www.rojoynegroclub.com/images/tmp/M26-1502-312-frames.MLV
https://www.rojoynegroclub.com/images/tmp/M26-1503-144-frames.MLV

The last two are more extreme, as they are looking towards the sun. I think the setting are still the same (100/1600), but can't remember for sure.


a1ex

Update: still aliased like hell, but flicker-free! Enjoy the power of robust statistics!

http://builds.magiclantern.fm/jenkins/view/Utilities/job/cr2hdr-20bit/

I expect this to solve many problems caused by imprecise exposure matching, not just flicker (the most common one being horizontal stripes). You may want to run it with --no-stripe-fix (this workaround might not be needed any more - to be tested). There are a few other (small) changes I didn't manage to test yet, but they appear harmless at first sight.

Rendering script used for testing:

# assumes only 1 MLV is present in the working directory (no more, no less)
mlv_dump *.MLV --dng --no-fixcp --no-stripes
cr2hdr *.dng --same-levels --no-stripe-fix
dcraw -W -b 4 *.DNG
ffplay -loop 0 $(basename *.MLV .MLV)_%06d.ppm


TODO: make good use of this to solve the aliasing issue: http://www.magiclantern.fm/forum/index.php?topic=20999

Caveat: I did not run my "usual" test suite, as it would require swapping the hard drive, so please consider the latest build highly experimental (and try the previous one in case of problems). As usual, feel free to report issues - in particular, troublesome test images.

Danne

Great work. Here´s a binary for mac users:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/cr2hdr

Tested against one test file from smasry.

This is an interesting added option:
Troubleshooting options:
--force         : force processing even if the image is not Dual ISO


QuoteTODO: make good use of this to solve the aliasing issue: http://www.magiclantern.fm/forum/index.php?topic=20999
This would be great for dualiso.

smasry

Thanks, @a1ex, can't wait to test this! And thanks, @danne for already building the mac cr2hdr binary!

Yes, aliasing is bad; I should have invested in a wider lens and shot this in crop mode. It wouldn't have been too terrible then.

As for the aliasing reduction using your script, hugin, enfuse and all the other options listed in that thread, it looks fantastic, just one problem: we lose the RAW image, leaving less data to work on, in the colour-correcting stage. I hope I'm wrong on this issue, would a 16-bit TIFF or EXR may give us the same leeway as a 16-bit DNG?

Danne

Well, would be nice to see some comparison tests on this theme. I updated the hugin/enfuse script just now by the way.

a1ex

Quote from: smasry on December 04, 2017, 09:24:08 AM
Yes, aliasing is bad; I should have invested in a wider lens and shot this in crop mode. It wouldn't have been too terrible then.

It would have been just as bad - exercise for the reader.

Hint: it was already discussed.

If spatial aliasing is an issue, but you have no problems with temporal aliasing, I suggest looking into the plain old HDR video. The optical flow tricks are MUCH easier to use on that kind of footage.


OlRivrRat

                     @Dane

           "Here´s a binary for mac users:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/cr2hdr"

           So what do we do with This to be sure Switch Uses It?
ORR~DeanB  ~~  80D-ML  &  SL1+ML  &  5D2+ML  &  5DC+ML  &  70D+ML(AliveAgain)


a1ex

Regarding aliasing in videos:

Here's a very simple octave script that only works on static scenes (with nearly no motion) with crawling artifacts. It's based on an older idea from stevefal, and seems to work quite well on some of the clips posted lately (as long as the motion from one frame to another is next to none).

You'll now see with your own eyes what exactly a "slight temporal artifact" is.

This script also serves as a minimal proof of concept code for Dual ISO blending (useful for understanding how the algorithm works, without having to dig through thousands of lines of code). In other words, this script is very useful as documentation. It operates on unprocessed 14-bit .dng frames (as extracted with mlv_dump), but requires some metadata from cr2hdr logs.


# file name prefix
# M26-1444-30-frames_000000.dng -> prefix = "M26-1444-30-frames_"
prefix = "M26-1444-30-frames_"


# "crawling" phase - possible values: 0, 1, 2, 3
# from ISO pattern on the first rendered frame (cr2hdr log), or trial and error:
# dBBd = 0
# ddBB = 1
# BddB = 2
# BBdd = 3
p = 3

# hardcoded - copy from cr2hdr log
black       = 2047       # black level (14-bit)
white       = 15000      # white level (14-bit), can be approximate, must be slightly lower than actual clipping point
gain        = 15.3       # from "Linear fit" messages - gain part (actually ISO ratio)
offset      = 24         # from "Linear fit" messages - offset part, invert sign
black_adj   = 3          # fine-tune until blacks look right (fixme: why is it needed?)

# from start+1 to end-1 (frame numbers to process)
# we need one frame before and one after
for i = 1:29
    # fixme: read each image only once
    a = read_raw(sprintf("%s%06d.dng", prefix, i-1));
    b = read_raw(sprintf("%s%06d.dng", prefix, i));
    c = read_raw(sprintf("%s%06d.dng", prefix, i+1));

    # print the pattern for current frame (cross-check with cr2hdr log)
    # for example:       cr2hdr *.dng | grep pattern
    # cross-check with:  octave crawl.m | grep pattern
    iso_pattern = circshift("dBBd", [0 p])

    # rebuild hi-iso data for frame b, taking bright fields from frames a and c
    # fixme: this only works when there's little or no motion between the frames
    # this recovers the full resolution from perfectly static (!) scenes
    # but introduces massive aliasing whenever something moves
    hi = b;
    hi(mod(p+0,4)+1:4:end,:) = a(mod(p+0,4)+1:4:end,:);
    hi(mod(p+3,4)+1:4:end,:) = c(mod(p+3,4)+1:4:end,:);

    # rebuild lo-iso data for frame b, taking dark fields from frames a and c
    lo = b;
    lo(mod(p+1,4)+1:4:end,:) = c(mod(p+1,4)+1:4:end,:);
    lo(mod(p+2,4)+1:4:end,:) = a(mod(p+2,4)+1:4:end,:);

    # cancel black delta (see "Find black delta..." in cr2hdr.c)
    # fixme: incorrect offset scaling in cr2hdr logs
    black_delta = 2 * offset * 16 / (gain + 1)
    hi -= black_delta / 2;
    lo += black_delta / 2;

    # remove black level, match image brightness
    # (darken hi to match lo)
    hi -= black;
    lo -= black;
    hi /= gain;

    # approximate dual iso blending (smooth transition from lo-iso to hi-iso)
    # fixme: compute noise profiles and use Kalman optimal averaging formula
    T = @(x) min(max((x - 3000) / 8000, 0), 1);
    F = @(x) ((sin((x-0.5)*pi)+1)/2).^4;
    f = F(T(hi .* gain));
    x = lo .* f + hi .* (1-f);
    plot(1:15000, F(T(1:15000)));

    # restore the black level
    x += black;

    # fine-tune black level
    x += black_adj;

    # check image levels
    percentiles = [10 50 90 99 99.9 99.99 99.999];
    levels = round(prctile(x(:), percentiles))';
    printf("%g\t", percentiles); printf("\n");
    printf("%d\t", levels); printf("\n");

    # output as PGM -> DNG
    # todo: clean way to save a DNG from octave
    # pgm2dng: http://a1ex.magiclantern.fm/bleeding-edge/pgm2dng.c
    # note: pgm2dng hardcodes a black level of 2048 and a white level of 16382
    # multiply 14-bit image and black/white levels by 4 to get 16-bit output
    out = sprintf("frame%d.pgm", i);
    imwrite(uint16(x * 4), out);
    system(sprintf("pgm2dng %s", out));
    system(sprintf("exiftool %s -BlackLevel=%d -WhiteLevel=%d -overwrite_original", [out(1:end-4) ".DNG"], black * 4, white * 4));

    # update the phase for next frame
    # todo: is the "crawling" always going in this direction?
    p = mod(p+1, 4);
end


Caveat: it requires tuning for every single clip (mostly copying stuff from cr2hdr log). It is currently tuned for M26-1444-30-frames.MLV (linked earlier). You'll also need pgm2dng (a very hackish tool I use for experiments, to get DNGs out of octave).

This also gives fairly good results on the clip from 70MM13 (p=0, gain=7.86, offset=10.61).

Possible improvements:
- find out where there is motion (how? aliasing may easily confuse any algorithm I can think of) and use regular interpolation in these areas.
- apply motion compensation (optical flow) and interpolate only the missing bits (difficult)

Have fun!

70MM13

Here's a couple of frames extracted from test shots of iso 53 vs 100...

I did nothing to avoid pink highlights, simply reducing analog gain to look for noise improvement, and got a good result!

Both recordings are using identical conditions and camera settings, one with iso regs enabled, the other disabled.  Exposure was adjusted in post to match.

These images have NO noise reduction of any kind. I raised the raw white level in rawtherapee to reduce the pink highlights but only a quick adjustment to make it less distracting.

I'll be using this!




R

Has anyone experienced problems with last version (2017-12-04 00:01) of cr2hdr processing 5DII files?

I notice two things in Lightroom that I think they happen since I updated cr2hdr.

First, it seems the processed Dual ISO DNG it has lost orientation info as now LR doesn't turn my images automatically.

Second, LR is unable to find a lens profile of the DNG processed with cr2hdr but it can find it automatically in the original Dual ISO file.

Anyone has noticed this also?

Danne

Upload an example CR2 and version of cr2hdr? Windows, mac?

R

- unprocessed dualiso: https://www.amazon.es/clouddrive/share/2hOTnDMQlaR59dlhFOp5faiVnOftB4sWivsdVIiHL5K

- processed dualiso: https://www.amazon.es/clouddrive/share/N5smrclgCuyIQOPP5bO80qnh0TMaryMC825RSzsGakM

Things I have noticed since build 2017-12-04 00:01.

1) Metadata loss. Unprocessed dng has the right metadata.
- Exposure: 1/125 at f8.0
- Focal Length: 30mm
- Model: Canon EOS 5D Mark
- Lens: EF 17-40 f4 L USM

Processed dual iso has lost metadata.
- Exposure: 1 sec at f1.0
- Focal Length: ?
- Model: Canikon
- Lens: ?

So Lr warns that it's unable to locate a matching lens profile automatically.

2) Even after manually assign lens profile something seems wrong. Before the last build 90% I needed -2 points of distortion correction for the 17-40mm. Now I need about +10. ?¿?¿?¿ Weird.

3) Orientation info seems lost? Before last build Lr automatically rotate the images to right orientation, now I have to do it manually.

Lr 7.1 on Windows 8.1.

Danne

I used the latest version for mac and it transcodes just fine over here, camera name and lens name is represented. Are you using cr2hdr directly or some other app?

R

Quote from: Danne on January 27, 2018, 05:50:45 PM
I used the latest version for mac and it transcodes just fine over here, camera name and lens name is represented. Are you using cr2hdr directly or some other app?

I'm using Dual ISO Converter plugin for Lightroom. Could it have something to do the fact that I've downloaded last build of cr2hdr and overwrite the cr2hdr found in the Lr plugin?


Danne


R

Quote from: Walter Schulz on January 27, 2018, 07:24:02 PM
@R: Sharing an unprocessed Dual-ISO file might be helpful ... content of plug-in bin dir, too. Exiftool, dcraw, cr2hdr

Like the one here? --> https://www.magiclantern.fm/forum/index.php?topic=7139.msg196445#msg196445

( https://www.amazon.es/clouddrive/share/2hOTnDMQlaR59dlhFOp5faiVnOftB4sWivsdVIiHL5K )

The ..\Adobe Lightroom Classic CC\cr2hdr.lrplugin\bin looks like this:




Walter Schulz

Quote from: R on January 27, 2018, 07:31:01 PM
Like the one here? --> https://www.magiclantern.fm/forum/index.php?topic=7139.msg196445#msg196445

Must be alcohol induced blindness ...
Problem with snapshot: No reliable version info. Hard to tell what we are trying to reproduce without your files.

R

About https://www.magiclantern.fm/forum/index.php?topic=7139.msg196445#msg196445, it turned out to be incompatibilities between Lr plugin and exiftool .

cr2hdr.3.0-170316-02-MLV-DEV Plugin uses exiftool 10.43 and cr2hdr-win 2017-12-04 00:01 comes with exiftool 10.02. If you want to update the plugin bin folder with new releases of cr2hdr-win, just replace cr2hdr.exe and dcraw.exe but not exiftool.exe.

theBilalFakhouri

Quote from: theBilalFakhouri on April 27, 2017, 09:45:37 AM
I Have problem with Canon 700D
when i turn on dual iso module automatically ISOless LV err(8 ) appear on screen

and the module do not work anymore
WHY THIS HAPPENING ?
any Ideas? a1ex

dual_iso ISOless ( 8 ) & (15) err finally solved for Canon 700D that have this problem!
The problem was in CMOS (0) Registers at ISO 100, It was different in my camera.
So I changed it to match the code in dual_iso.c
        is_700d = 1;   
        FRAME_CMOS_ISO_START = 0x4045368e;
        FRAME_CMOS_ISO_COUNT =          6;
        FRAME_CMOS_ISO_SIZE  =         34;

        PHOTO_CMOS_ISO_START = 0x40452444;
        PHOTO_CMOS_ISO_COUNT =          6;
        PHOTO_CMOS_ISO_SIZE  =         16;

It's Working!





Now the time to try it:
magiclantern-crop_rec_4k.2018Apr11.700D115_Iso_Less_8_15_resolved (Updated with 10/3/2018 changes)

Walter Schulz

Issue related to ISOless message on 650D? Any advice how to corner this?

theBilalFakhouri

Quote from: Walter Schulz on February 11, 2018, 02:10:52 PM
Issue related to ISOless message on 650D? Any advice how to corner this?
Yes, I think it's the same way I did you can do, it's about CMOS 0 Registers at ISO 100, Some cameras have different registers than others for
the same model, (I don't know yet if it will be more than two Registers for same model, waiting 700d users to feedback)

So you have to see your registers on your camera by adtg_gui.mo (Assuming adtg_gui.mo is working on canon 650d)

1-Set ISO at 100
2-Set camera on movie mode 1080/24
3-After loading the modules, turn on adtg_gui in Debug submenu
4-Turn on dual_iso
5-Go to PLAY mode then go back to liveview
6-Check your CMOS 0 Registers data and write it down (In the white box)
VRAM0" border="0

7-Now go to photo mode
8-Take a picture and Check your CMOS 0 Registers data and write it down again.

--> Now in dual_iso.c
This value for movie mode

        FRAME_CMOS_ISO_START = 0x********;

and this for photo mode:

        PHOTO_CMOS_ISO_START = 0x********;

Leave the (0x) and change the others to the values you wrote, Compile and test it in the camera!

dfort

Quote from: theBilalFakhouri on February 11, 2018, 02:58:08 PM
1-Set ISO at 100
2-Set camera on movie mode 1080/24
3-After loading the modules, turn on adtg_gui in Debug submenu
4-Turn on dual_iso
5-Go to PLAY mode then go back to liveview
6-Check your CMOS 0 Registers data and write it down (In the white box)

I got 0x4045368e on my 700D, just like you did:


Quote from: theBilalFakhouri on February 11, 2018, 02:58:08 PM
7-Now go to photo mode
8-Take a picture and Check your CMOS 0 Registers data and write it down again.

I got 0x40452044:


So following your instructions it looks like my camera is fine with the current code:

modules/dual_iso.c
    else if (is_camera("700D", "1.1.5"))
    {
        is_700d = 1;   

        FRAME_CMOS_ISO_START = 0x4045328E;
        FRAME_CMOS_ISO_COUNT =          6;
        FRAME_CMOS_ISO_SIZE  =         34;

        PHOTO_CMOS_ISO_START = 0x40452044;
        PHOTO_CMOS_ISO_COUNT =          6;
        PHOTO_CMOS_ISO_SIZE  =         16;

        CMOS_ISO_BITS = 3;
        CMOS_FLAG_BITS = 2;
        CMOS_EXPECTED_FLAG = 3;
    }


However, your 700D is different?

Quote from: theBilalFakhouri on February 11, 2018, 12:53:02 PM
        PHOTO_CMOS_ISO_START = 0x40452444;

It's Working!

It doesn't make sense that these registers are different on the same model camera. This is what happens if I use your build on my camera: