[IMPOSSIBLE] dual ISO H.264

[globalphotobank]

BRITOM
that clip you have there don't look like good news. See the child's body twist and her head go back. If someone did that to my child, my instant reaction would be to boot f$#! the B%@!# right in the face.

[apefos]

A1ex showed a color chart in other topic and talked about his interest in better color preview in camera LCD when recording dual iso raw.

This makes me to go back here in this topic in post number 24 when he talked about hue differences between the two isos. At that moment I did not pay attention to the issue, because my mind was focused in the first challenges, to perceive lines retention and enabling h264 recording.

But after finishing the algorithm and seeing the color chart from A1ex and his informations about color differences, I perceived my job was incomplete. I need to consider the information from A1ex which says there is a hue difference between the two isos.

So I did some tests with the 600D today at night here to perceive how is the color behavior. I recorded some dual iso h264 videos showing different colors and in computer I confirmed there is a hue difference, but not just a hue difference, there is also a white balance difference between the two isos in the image.

This will make necessary to do a precise test in the same way I did the grey wall test. But this time with colors. I will need to find the hue difference and white balance difference between isos and calculate formulas for each line in the wave considering the amount of merging between them, because merging changes the hue and white balance difference... I will also need to perceive if color formulas must be applyed before of after luminance formulas... just another challenge...

Ginger HDR replied my email. They asked me If I did a try to reconstruct one still image using the algorithm. This made me to realize that I need to show the algorithm working before someone compile it into a computer software. (oh! obvious thing!)

So, after the color tests and color formulas implementation I will try to reconstruct the lines in a still image manually using the algorithm.

[britom]

@globalphotobank @everybody, i really sorry, i pasted the wrong gif link... my apologies to everyone

[a1ex]

I'm interested in fixing the dual iso preview for photos, not for video. The problem is slightly different: you have to match a small and color-distorted image (720x480 YUV) with some reference image. You no longer have to match colors for each scanline, but only at macroscopic level, because the heavy downsampling already took care of the scanlines.

So, I've tried to find Canon's response curves and how they change when using dual ISO, and for this purpose I've generated the following test images:

gradient-neutral-100.CR2
gradient-neutral-100-1600.CR2

(extract the embedded JPEG previews from them and compare them with the raw data)

The biggest issue is now obvious: besides the color cast, as soon as one of the ISOs gets clipped, the color info is lost. I think it's now clear that, unless you find a way to disable Canon's desaturation feature in bright highlights (maybe with a picture style?), the entire dual ISO from H.264 is just a waste of time.

If anyone would like to prove me wrong, please use technical arguments, not spirits.

[apefos]

Ok A1ex, I understood exactly what you want, just give me some time.

[mk11174]

@apefos  Here is the version of Dual_Iso you wanted for TL 2.0 NewMem, already tested it on my 600D and all is fine.
https://bitbucket.org/mk11174/magic-lantern/downloads/ML_600D_TL_NewMem_Dual_ISO_H264_Nov_26_2013.zip

[apefos]

@MK11174

Many Thanks! I downloaded and instaled in card. It is working OK! I got no waterfalling behavior in first tests and module loads fine. Now the dual iso indicator turns green, great!

[apefos]

@A1ex

I did a look in the CR2 files you uploaded in Digital Photo Professional and in Picture Style Editor, mainly in the 100-1600 and compared with the extracted jpg.

To be honest I do not believe a Picture Style will help you in this preview. Let me explain why: The nature and concept of dual iso is the iso 100 capture good highlights and iso 1600 capture good shadows, so most times the iso 1600 will be clipped in things like clouds, open windows and so on. When the image is clipped there is no information to recover because everything will be above 255, so no way to recover anything. I did lots of try to lower the highlights and what was clipped in the 1600 iso lines turned into a flat grey image. And the low iso lines turned into something like false color preview. Professional photografers will not like this.

My idea to solve this preview issue is you enable this functions in some camera buttons:

when dual iso module is enabled:
Press left arrow = disable (suspend) dual iso module and set camera to same low iso of dual iso module
Press right arrow = disable (suspend) dual iso module and set camera to same high iso of dual iso module
Press set = enable dual iso module again (kill suspend)

This way people can see the shadows recovered by the high iso, see the highlights preserved by low iso and see the mixed image if they want. And see things fast, just pressing one button. Only this will give to the professional photografers the perfect preview to see exactly what they are capturing in shadows and in highlights.

If you can, you also can enable camera buttons to change the low and high iso without the need to go into the ML menu, so the photografer see and change low and high iso separately and change them fast:

when seeing low iso press up or down arrows to change it
when seeing high iso press up or down arrows to change it

[bnvm]

I don't think A1ex was talking about using picture styles to fix the preview, he was referring to the white clipping of the highlights that the camera does.

Imaging a bright red light, if you take a picture of the red light you might expect that when the values are clipped in the image that you would end up with a bright red pixel but that is not what happens the camera automatically converts the overly bright red into white destroying any of the color information what might have been there. This is not the case for raw since it is just raw sensor data. What this means is that unless you can find a way of stopping the camera from converting clipped highlights into white there won't be anyway to correctly merge the 2 iso's together when there is clipping.  He was suggesting that maybe a picture style could do this.

Have you ever tried to merge an hdr when there are objects in the scene that are moving. I had done several in the past at night with traffic and I can tell you that these incorrect clipped highlights will cause some nasty color artifacts when merged turning bright purple or pink, etc...

[apefos]

Oh guys, you are so hungry!!!

I will show something I am working on since I got my 600D in november 2012. Don't ask anything now! 

Is this what you want?

[apefos]

 

[apefos]

 

[a1ex]

I'm marking the topic as IMPOSSIBLE, until you will be able to prove me wrong.

[arturochu]

I'm marking the topic as IMPOSSIBLE, until you will be able to prove me wrong.

badass

[djsubject]

what's the difference between HDR and dual ISO?

[apefos]

ok, no problem, I understand. People wants to see things working.

I will stop posting for a while and concentrate my efforts to finish the demonstration video editing for the aps project I showed in the photos above.

After this I will finish the color tests to improve the algorithm and I will reconstruct a still image extracted from h264 video using the algorithm to prove it works.

It will take time.

I will stop posting for a while until I can show something working.

Thanks.

[ItsMeLenny]

what's the difference between HDR and dual ISO?

HDR is "high dynamic range".

HDR usually is constructed from taking multiple pictures at different exposures and combining them all together to get my dynamic range, as in you can see whats in the dark regions as well what is in the super bright regions.

Dual ISO is another way to construct HDR images from one picture, by scanning each line of pixels at different ISOs to create an under and overexposed image interlaced together.

Both of them then need further editing in post.



And to apefos:
You talk a lot of gibberish on different posts which is also sometimes completely irrelevant.
And then make multiple posts in a row. Would you mind toning it down just a little please.

[g3gg0]

ok, no problem, I understand. People wants to see things working.

i think it will work. definitely.
BUT...

when you raise details in single H.264 blocks in vertical direction by interlacing hi/lo exposure, then you lose some bits of your available bandwidth
for describing that interlace pattern. this means you lose horizontal detail and vertical detail across the lines.
because the interlaced lines are visible strongly, the small differences caused by other gradients is very low.

my assumption:
due to the lossy compression, you will lose a decent amount of vertical resolution and also a bit of the horizontal resolution.
you have this detail loss in addition to the resolution loss in under/overexposed areas btw.

can you do me a favor?
can you record the common resolution check patterns in one video with dual iso enabled, disabled, ... a few times?
i would recommend about 5 seconds per state and about 5 ON/OFF combinations.
this with 3 video files please: one video with a low CBR, one with high CBR and one video with VBR (normal quality)

we then can look at the video frames and data rates.

[a1ex]

i think it will work. definitely.

I've proved it can't work, even if you start from uncompressed YUV (see post #103).

If you still don't understand: let's say you have a function y = f(x), where x ranges from say -128 to +128, and let's say you have f(x) = x/100 (or something close to that). You can look at y and you should recover x. There's one little issue: you are working on integer math

[g3gg0]

I've proved it can't work, even if you start from uncompressed YUV (see post #103).

If you still don't understand: let's say you have a function y = f(x), where x ranges from say -128 to +128, and let's say you have f(x) = x/100 (or something close to that). You can look at y and you should recover x. There's one little issue: you are working on integer math

ah sorry, i didnt want to say it will work with the current state of implementation.
should have added "somehow", that would explain what this fragment of sentence should have expressed.

if you squeeze the integer range with a custom pic style / lut, you will lose color resolution but can cover the exposure ranges.
this will for sure raise some other problems too, no doubt!
and even if there is a workaround that is capable of cover the re-quantization (14 to yuv) properly, the H.264 compression is still lossy.

what i wanted to make clear:
H.264 is based on removing data your eye (brain) doesnt really need to see whats happening.
but in the case dual_iso the obvious thing isnt the interlaced bright/dark line thingy, but the relatively small deltas on top of that.
and H.264 algorithms will now try hard in reducing that "unimportant" data.

there could be one lucky circumstance: the vertical pattern equals the maximum vertical frequency that can be expressed with DCT parameters.
this would allow the encoder to use only fewer bits for describing that pattern. but i am quite sure the encoder can not detect that 1/2 * f_max pattern.
it will most likely detect a 1.9999998 * f_max + 0.00000001 * (f_max-1) - 0.00000001 + (f_max - 2) etc.
so we lose a lot more bits on describing that pattern.
-> reduced details in the whole image

[apefos]

@a1ex if you think it will not work no problem, but other people are free to do tests if they want, so stop bothering us and go work in your raw tasks. you are the main developer but you made this thing open source so please respect it's nature. we have the module enabled, we do not need you and your "impossible words"anymore, let us try our success or our fail alone, you are fired of dual iso h264 development, SHUT YOUR MOUTH AND GO AWAY, NOW!

@g3gg0 here is what you asked me:

a jpg sequence with the fist wave starting at line 01 (balck and white because i did not implemented wb and hue correction to the algorithm yet)

https://www.dropbox.com/s/8z0qghs027qtxdm/jpg192seq.zip

here are the resolution chart tests in diffetent bitrates:

https://www.dropbox.com/s/mgyqi26r3j39els/MVI_1869.MOV

https://www.dropbox.com/s/y59aqo92qtbbpre/MVI_1870.MOV

https://www.dropbox.com/s/g7ddce61rgxeeka/MVI_1871.MOV

https://www.dropbox.com/s/3sbnmmjgkdc8jqm/MVI_1872.MOV

https://www.dropbox.com/s/klwl50juhqg22i0/MVI_1874.MOV

https://www.dropbox.com/s/fqwecc61bn9vov0/MVI_1875.MOV

https://www.dropbox.com/s/mgyqi26r3j39els/MVI_1869.MOV

[Andy600]

Speechless!

[apefos]

Soon I will release a picture style.

I will ask for a small donation.

I will use some amount of the donations to support the developers who are willing to work in the dual iso h264 project until we see if it will work or not. I believe it will.

http://www.magiclantern.fm/forum/index.php?topic=9408.0

[g3gg0]

@a1ex if you think it will not work no problem, but other people are free to do tests if they want, so stop bothering us and go work in your raw tasks. you are the main developer but you made this thing open source so please respect it's nature. we have the module enabled, we do not need you and your "impossible words"anymore, let us try our success or our fail alone, you are fired of dual iso h264 development, SHUT YOUR MOUTH AND GO AWAY, NOW!

wtf??

[g3gg0]

i guess that is time to say that we need this particular kind of spirit even less.