Author Topic: Dual ISO - massive dynamic range improvement (dual_iso.mo)  (Read 1358076 times)

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10558
  • 5D Mark Free
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #2925 on: December 02, 2017, 12:22:36 AM »
Vanilla processing:
Code: [Select]
mlv_dump M26-1444-30-frames.MLV --dng --no-fixcp --no-stripes
cr2hdr *.dng --same-levels
dcraw -W -b 4 *.DNG
ffplay -loop 0 M26-1444-30-frames_%06d.ppm

There is some flicker - let's see if it's natural flicker or bug in cr2hdr. MlRawViewer (1 month old is too old, really?) doesn't show any obvious flicker.

The debug options in cr2hdr won't help much, as they are all after matching the exposure between the two sub-images (that is, after scaling the raw data). Let's go back to octave:
Code: [Select]
for f = dir('*.dng')'
    f.name
    im = read_raw(f.name);
    g1 = im(1:4:end, 2:4:end);
    imwrite(uint8(g1 / 64), [f.name(1:end-4) ".jpg"]);
end

Oh no, the "crawling" effect!

Code: [Select]
phase = 0;
for f = dir('*.dng')'
    f.name
    im = read_raw(f.name);
    g1 = im(1+phase:4:end, 2-mod(phase,2):4:end);
    phase = mod(phase+1, 4);
    imwrite(uint8(g1 / 64), [f.name(1:end-4) ".jpg"]);
end
system("ffplay -loop 0 M26-1444-30-frames_%06d.jpg")

No flicker obvious to me (just lots of aliasing, as we are previewing at 480x270, without any filtering), so it must be a bug in cr2hdr. Let's try the experimental changes.

Code: [Select]
cd modules/dual_iso
hg up e72eb09ffa30 -C
make cr2hdr; sudo cp cr2hdr /path/to/executables

Running the first script gives similar result (still no luck).

TODO: fix the math in cr2hdr.c, match_exposures. This is going to be tricky.

Using the data from each sub-image (dark or bright) gives no flicker. The issue appears after matching the brightness between them. The issue is this variation in fitting results:
Code: [Select]
Linear fit      : y = 0.063813*x + 108.308081
Linear fit      : y = 0.053660*x + 142.179592
Linear fit      : y = 0.062154*x + 105.105966
Linear fit      : y = 0.063109*x + 102.554616
Linear fit      : y = 0.061640*x + 115.000017
Linear fit      : y = 0.063022*x + 114.077421
Linear fit      : y = 0.064436*x + 101.677465
Linear fit      : y = 0.063109*x + 102.554616
Linear fit      : y = 0.063109*x + 108.302178
Linear fit      : y = 0.063285*x + 113.235900
Linear fit      : y = 0.060960*x + 110.928908
Linear fit      : y = 0.061044*x + 109.146648
Linear fit      : y = 0.061725*x + 114.726728
Linear fit      : y = 0.061129*x + 118.265047
Linear fit      : y = 0.062327*x + 104.553750
Linear fit      : y = 0.063022*x + 102.833685
Linear fit      : y = 0.062587*x + 111.972896
Linear fit      : y = 0.057352*x + 132.357319
Linear fit      : y = 0.057992*x + 122.541289
Linear fit      : y = 0.057912*x + 117.377329
Linear fit      : y = 0.053069*x + 140.393025
Linear fit      : y = 0.053290*x + 145.259863
Linear fit      : y = 0.058883*x + 119.573745
Linear fit      : y = 0.061554*x + 105.396036
Linear fit      : y = 0.061982*x + 111.904583
Linear fit      : y = 0.063548*x + 108.517969
Linear fit      : y = 0.064525*x + 101.649343
Linear fit      : y = 0.064346*x + 94.735446
Linear fit      : y = 0.062674*x + 111.319366
Linear fit      : y = 0.064974*x + 106.084113
Linear fit      : y = 0.063109*x + 102.175959

If you have some other footage that shows flicker (in particular, extreme cases), I'd like to see it as well (small cuts like this are OK).

smasry

  • New to the forum
  • *
  • Posts: 28
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #2926 on: December 02, 2017, 01:03:17 PM »
Quote
MlRawViewer (1 month old is too old, really?) doesn't show any obvious flicker.

I stand corrected; I was sure development of MlRawViewer was discontinued, but am happy to be wrong in this case.

Quote
If you have some other footage that shows flicker (in particular, extreme cases), I'd like to see it as well (small cuts like this are OK).

I have plenty; I was worried that something's wrong with my 5D3 or my workflow since I seem to be getting flickering in most if not all dual-iso footage, while no-one else seems to be reporting anything. Perhaps it's because I push it beyond your recommendations of 400/1600 or 200/1600 and go all the way to 100/1600, leaving fewer midtones?

Unfortunately, I've deleted a lot of test clips, taken while I was learning, and experimenting with, this feature. Here are more clips from the same shoot, though:

https://www.rojoynegroclub.com/images/tmp/M26-1449.MLV
https://www.rojoynegroclub.com/images/tmp/M26-1450-100-frames.MLV
https://www.rojoynegroclub.com/images/tmp/M26-1451-166-frames.MLV
https://www.rojoynegroclub.com/images/tmp/M26-1454-48-frames.MLV
https://www.rojoynegroclub.com/images/tmp/M26-1455-66-frames.MLV
https://www.rojoynegroclub.com/images/tmp/M26-1458-52-frames.MLV
https://www.rojoynegroclub.com/images/tmp/M26-1502-312-frames.MLV
https://www.rojoynegroclub.com/images/tmp/M26-1503-144-frames.MLV

The last two are more extreme, as they are looking towards the sun. I think the setting are still the same (100/1600), but can't remember for sure.


a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10558
  • 5D Mark Free
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #2927 on: December 04, 2017, 01:04:34 AM »
Update: still aliased like hell, but flicker-free! Enjoy the power of robust statistics!

http://builds.magiclantern.fm/jenkins/view/Utilities/job/cr2hdr-20bit/

I expect this to solve many problems caused by imprecise exposure matching, not just flicker (the most common one being horizontal stripes). You may want to run it with --no-stripe-fix (this workaround might not be needed any more - to be tested). There are a few other (small) changes I didn't manage to test yet, but they appear harmless at first sight.

Rendering script used for testing:
Code: [Select]
# assumes only 1 MLV is present in the working directory (no more, no less)
mlv_dump *.MLV --dng --no-fixcp --no-stripes
cr2hdr *.dng --same-levels --no-stripe-fix
dcraw -W -b 4 *.DNG
ffplay -loop 0 $(basename *.MLV .MLV)_%06d.ppm

TODO: make good use of this to solve the aliasing issue: http://www.magiclantern.fm/forum/index.php?topic=20999

Caveat: I did not run my "usual" test suite, as it would require swapping the hard drive, so please consider the latest build highly experimental (and try the previous one in case of problems). As usual, feel free to report issues - in particular, troublesome test images.

Danne

  • Hero Member
  • *****
  • Posts: 3537
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #2928 on: December 04, 2017, 01:42:40 AM »
Great work. Here´s a binary for mac users:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/cr2hdr

Tested against one test file from smasry.

This is an interesting added option:
Code: [Select]
Troubleshooting options:
--force         : force processing even if the image is not Dual ISO

Quote
TODO: make good use of this to solve the aliasing issue: http://www.magiclantern.fm/forum/index.php?topic=20999
This would be great for dualiso.

smasry

  • New to the forum
  • *
  • Posts: 28
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #2929 on: December 04, 2017, 09:24:08 AM »
Thanks, @a1ex, can't wait to test this! And thanks, @danne for already building the mac cr2hdr binary!

Yes, aliasing is bad; I should have invested in a wider lens and shot this in crop mode. It wouldn't have been too terrible then.

As for the aliasing reduction using your script, hugin, enfuse and all the other options listed in that thread, it looks fantastic, just one problem: we lose the RAW image, leaving less data to work on, in the colour-correcting stage. I hope I'm wrong on this issue, would a 16-bit TIFF or EXR may give us the same leeway as a 16-bit DNG?

Danne

  • Hero Member
  • *****
  • Posts: 3537
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #2930 on: December 04, 2017, 11:42:11 AM »
Well, would be nice to see some comparison tests on this theme. I updated the hugin/enfuse script just now by the way.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10558
  • 5D Mark Free
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #2931 on: December 04, 2017, 01:46:09 PM »
Yes, aliasing is bad; I should have invested in a wider lens and shot this in crop mode. It wouldn't have been too terrible then.

It would have been just as bad - exercise for the reader.

Hint: it was already discussed.

If spatial aliasing is an issue, but you have no problems with temporal aliasing, I suggest looking into the plain old HDR video. The optical flow tricks are MUCH easier to use on that kind of footage.

70MM13

  • New to the forum
  • *
  • Posts: 14
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #2932 on: December 06, 2017, 04:22:52 AM »
Moved to own thread...

OlRivrRat

  • Senior
  • ****
  • Posts: 261
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #2933 on: December 06, 2017, 06:05:31 AM »
                     @Dane

           "Here´s a binary for mac users:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/cr2hdr"

           So what do we do with This to be sure Switch Uses It?
ORR~DeanB  ~~  80D-ML  &  SL1+ML  &  5D2+ML  &  5DC+ML  &  70D+ML.111B(DEAD)


a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10558
  • 5D Mark Free
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #2935 on: December 06, 2017, 03:51:32 PM »
Regarding aliasing in videos:

Here's a very simple octave script that only works on static scenes (with nearly no motion) with crawling artifacts. It's based on an older idea from stevefal, and seems to work quite well on some of the clips posted lately (as long as the motion from one frame to another is next to none).

You'll now see with your own eyes what exactly a "slight temporal artifact" is.

This script also serves as a minimal proof of concept code for Dual ISO blending (useful for understanding how the algorithm works, without having to dig through thousands of lines of code). In other words, this script is very useful as documentation. It operates on unprocessed 14-bit .dng frames (as extracted with mlv_dump), but requires some metadata from cr2hdr logs.

Code: [Select]
# file name prefix
# M26-1444-30-frames_000000.dng -> prefix = "M26-1444-30-frames_"
prefix = "M26-1444-30-frames_"


# "crawling" phase - possible values: 0, 1, 2, 3
# from ISO pattern on the first rendered frame (cr2hdr log), or trial and error:
# dBBd = 0
# ddBB = 1
# BddB = 2
# BBdd = 3
p = 3

# hardcoded - copy from cr2hdr log
black       = 2047       # black level (14-bit)
white       = 15000      # white level (14-bit), can be approximate, must be slightly lower than actual clipping point
gain        = 15.3       # from "Linear fit" messages - gain part (actually ISO ratio)
offset      = 24         # from "Linear fit" messages - offset part, invert sign
black_adj   = 3          # fine-tune until blacks look right (fixme: why is it needed?)

# from start+1 to end-1 (frame numbers to process)
# we need one frame before and one after
for i = 1:29
    # fixme: read each image only once
    a = read_raw(sprintf("%s%06d.dng", prefix, i-1));
    b = read_raw(sprintf("%s%06d.dng", prefix, i));
    c = read_raw(sprintf("%s%06d.dng", prefix, i+1));

    # print the pattern for current frame (cross-check with cr2hdr log)
    # for example:       cr2hdr *.dng | grep pattern
    # cross-check with:  octave crawl.m | grep pattern
    iso_pattern = circshift("dBBd", [0 p])

    # rebuild hi-iso data for frame b, taking bright fields from frames a and c
    # fixme: this only works when there's little or no motion between the frames
    # this recovers the full resolution from perfectly static (!) scenes
    # but introduces massive aliasing whenever something moves
    hi = b;
    hi(mod(p+0,4)+1:4:end,:) = a(mod(p+0,4)+1:4:end,:);
    hi(mod(p+3,4)+1:4:end,:) = c(mod(p+3,4)+1:4:end,:);

    # rebuild lo-iso data for frame b, taking dark fields from frames a and c
    lo = b;
    lo(mod(p+1,4)+1:4:end,:) = c(mod(p+1,4)+1:4:end,:);
    lo(mod(p+2,4)+1:4:end,:) = a(mod(p+2,4)+1:4:end,:);

    # cancel black delta (see "Find black delta..." in cr2hdr.c)
    # fixme: incorrect offset scaling in cr2hdr logs
    black_delta = 2 * offset * 16 / (gain + 1)
    hi -= black_delta / 2;
    lo += black_delta / 2;

    # remove black level, match image brightness
    # (darken hi to match lo)
    hi -= black;
    lo -= black;
    hi /= gain;

    # approximate dual iso blending (smooth transition from lo-iso to hi-iso)
    # fixme: compute noise profiles and use Kalman optimal averaging formula
    T = @(x) min(max((x - 3000) / 8000, 0), 1);
    F = @(x) ((sin((x-0.5)*pi)+1)/2).^4;
    f = F(T(hi .* gain));
    x = lo .* f + hi .* (1-f);
    plot(1:15000, F(T(1:15000)));

    # restore the black level
    x += black;

    # fine-tune black level
    x += black_adj;

    # check image levels
    percentiles = [10 50 90 99 99.9 99.99 99.999];
    levels = round(prctile(x(:), percentiles))';
    printf("%g\t", percentiles); printf("\n");
    printf("%d\t", levels); printf("\n");

    # output as PGM -> DNG
    # todo: clean way to save a DNG from octave
    # pgm2dng: http://a1ex.magiclantern.fm/bleeding-edge/pgm2dng.c
    # note: pgm2dng hardcodes a black level of 2048 and a white level of 16382
    # multiply 14-bit image and black/white levels by 4 to get 16-bit output
    out = sprintf("frame%d.pgm", i);
    imwrite(uint16(x * 4), out);
    system(sprintf("pgm2dng %s", out));
    system(sprintf("exiftool %s -BlackLevel=%d -WhiteLevel=%d -overwrite_original", [out(1:end-4) ".DNG"], black * 4, white * 4));

    # update the phase for next frame
    # todo: is the "crawling" always going in this direction?
    p = mod(p+1, 4);
end

Caveat: it requires tuning for every single clip (mostly copying stuff from cr2hdr log). It is currently tuned for M26-1444-30-frames.MLV (linked earlier). You'll also need pgm2dng (a very hackish tool I use for experiments, to get DNGs out of octave).

This also gives fairly good results on the clip from 70MM13 (p=0, gain=7.86, offset=10.61).

Possible improvements:
- find out where there is motion (how? aliasing may easily confuse any algorithm I can think of) and use regular interpolation in these areas.
- apply motion compensation (optical flow) and interpolate only the missing bits (difficult)

Have fun!

70MM13

  • New to the forum
  • *
  • Posts: 14
Re: Dual ISO - massive dynamic range improvement (dual_iso.mo)
« Reply #2936 on: December 09, 2017, 05:18:08 AM »
Here's a couple of frames extracted from test shots of iso 53 vs 100...

I did nothing to avoid pink highlights, simply reducing analog gain to look for noise improvement, and got a good result!

Both recordings are using identical conditions and camera settings, one with iso regs enabled, the other disabled.  Exposure was adjusted in post to match.

These images have NO noise reduction of any kind. I raised the raw white level in rawtherapee to reduce the pink highlights but only a quick adjustment to make it less distracting.

I'll be using this!