[IMPOSSIBLE] dual ISO H.264

[jagnje]

Is it possible to have dual ISO for H.264 video?

[1%]

well it can be recorded as dual ISO but I don't think it adds any info and there is no workflow.

[a1ex]

I'm afraid the compression will make it impossible to recover the image. Didn't try, but I'm not eager to.

[jagnje]

don`t know but I imagined something like this

but also in 1080p and not limited to 50fps.
I sure know it would come in handy when filming skaters going through shade areas and into the sun.

[painya]

All you would need is a dng2hdr right? Just export the h.264 frames, and throw it in there.

[jagnje]

im not sure how that would help recover the underexposed or overexposed areas.

[apefos]

I am also interested in this feature, to record dual iso in h264.

I did a try to enable dual iso without enablind the raw video module, but the green dot keeps a little dark and the dual iso just get enabled when I enable the raw video, so to test it, the developers need to make the dual iso independent from raw video module.

I am using the T3i / 600D and no money to get a 5D3. In 600D I am limited to record continous 24p raw at 960x544. The dual iso uses alternate lines for different exposure, so I belive the dual iso just will be ok using 1920x1080 resolution, so no way to get good quality dual iso raw from 600D at 24fps 16:9.

Also, even I could get a 5D3, considering the raw files large size, it would be impossible for me to shoot a short film or documentary in 1920x1080 raw...

So due to these reasons I would like to experiment the dual iso in H264 MOV. I think if I increase the CBR bitrate it would help the encoder to get better image using dual iso. Today I am using 1.3x because this makes the files playable in camera.

I think the workflow for H264 dual iso would be simple... convert the H264 video into a JPG image sequence and do the same workflow, drag and drop over the converter. Maybe it would need another converter? (JPG2HDR)? Or even better to drag and drop the MOV file direct over the converter: MOV2HDR

The HDR video is great for situations without so much movement and it works good in 720p 50p or 60p to get the overexposed frames without optical flow... but the dual iso can do this in images with movement in 1920 x 1080 because it does not need optical flow...

So can you enable the dual iso to work with H264? I would like to test it also.

Thanks

[chris_overseas]

For dual-ISO to work as intended it needs to be dealt with BEFORE any further image processing takes place. I can't see h264 dual-ISO ever working due to the nature of the h264 compression and the image processing that is performed in the encoding phase. h264 will "smear" the high ISO and low ISO lines together somewhat, making it impossible to accurately reconstruct a sensible final image. That's assuming there's even a 1-1 mapping from the sensor to a horizontal line. Any resizing/binning will trash things completely.

[apefos]

I understand the compression problem, it could destroy the separation from the lines making different iso useless, but even knowing this I would like to try... maybe it can give us an average result... why not to try?, it is not so difficult to enable dual iso for H264...

I also got another idea, maybe impossible, to implement a similar function from dng2hdr inside camera, so the camera would do the interpolation and iso merge before compression... this is the "dream"...

I believe someday the manufacturers will implement this kind of thing using a sensor similar to the 70D, there is also a new patent from canon showing the dual pixel af with different size pixels, great for different exposures...

[dmilligan]

I understand the compression problem, it could destroy the separation from the lines making different iso useless, but even knowing this I would like to try... maybe it can give us an average result... why not to try?, it is not so difficult to enable dual iso for H264...

There's no reason to waste time trying, it wont work, it just wont, please accept that.

I also got another idea, maybe impossible, to implement a similar function from dng2hdr inside camera, so the camera would do the interpolation and iso merge before compression... this is the "dream"...

Custom, realtime image processing at 24fps is impossible. I don't even think a super high end computer could handle dual iso conversion + toning at 24fps, much less the measly ARM cpu inside these cameras. Plus, you'd be 'baking in' the curve/toning, since h264 is only 8 bit.

[apefos]

So maybe someday the manufacturers or ml will give us two H264 or H265 streams from adjacent pixels in the sensor recorded to the SD card simultaneously, so we can do the hdr merging in the post production.

Or these two streams could be in just one recorded file in top/bottom or side by side, just like 3d...

Also, if the next generation Canon dslr come with digic6 and 1080p 50p 60p, the hdr video with alternate iso in subsequent frames will be better to use, recording h264 or mp4 in 1080p 50p or 60p to get the 25p or 30p overexposed frames (shadows) without optical flow, this would be enough for good quality low movement images, 16 fstops dr, 11 from the custom picture style + 5 from the hdr video...

in narrative or documentary 1080p raw video is difficult for me to use due to the files are so large and no sound in camera...

[a1ex]

I understand the compression problem, it could destroy the separation from the lines making different iso useless, but even knowing this I would like to try...

Take the embedded JPEG preview from a dual ISO CR2 and get something useful out of it. Post the results, I'm curious how far you can get without having access to raw data.

[3pointedit]

Hmmm, aliasing isn't really affected by the compression and iirc the blurring comes from the resizing stages.

This might just be a way to get 30fps HDR from 1080 h264.

A1ex the raw embedded JPEG won't have temporal compression. I'll try shooting an interlace video at different sizes to see what is retained.

[apefos]

If interlaced video (60i and 50i) can be recorded by camcorders and consumer cameras in 17Mbps it is because the lines separation can be retained... if we use 1.3x CBR maybe it will be enough to keep the lines separation in a enough result, if this can shoot in 1080p and interpolated and merged in post and then downscaled to 720p or even to 640x360p for smartphone view it will be great,

Many thanks for testing the idea... If it can give an average result will help lots of people... no need to be 100% perfect...

If you need a beta tester just allow me to download a version of dual iso module enabled to work in h264... I can test in 600D/t3i

thanks and good luck in the tests!!!

[1%]

I think all you'd have to do is go to line ~322

change:
int raw_mv = mv && lv && raw_lv_is_enabled();

to

int raw_mv = mv && lv //&& raw_lv_is_enabled();

Quick look, I didn't see any other checks/blocks. EOSM can be tricked from photo mode -> movie mode without raw because of the constant LV or it could not too long ago.

Now is this worth it? That I don't know.

[3pointedit]

For cameras with poor raw performance like my 550d  this would mean cinemascope hdr files. Eliminating temporal errors from 720 50/60p but potential vertical aliasing errors. Not sure how registration of images would fare after merging or scaling?

Would be neat to test, as I just cant get useful raw from sd card rebel.

[apefos]

Hi 1%,

It seems to be so easy to modify the dual iso module to allow h264 recording... problem is I do not know how to edit these files. Could you please do this change and allow us to download the dual iso module for h264 from your bitbucket? I have the 600D/T3i running your NewMem build so I can test it.

Many thanks!

[3pointedit]

I can simulate these files as combed footage, I guess the real question will be the effect of interpolation at the edges of high contrast.

[apefos]

i do not know if this is in the dual iso module or in the autoexec.bin... i would like to do this change to enable dual iso recording in h264 for the 600d/t3i, how can I open the file, change and save? see:

go to line ~322

change:
int raw_mv = mv && lv && raw_lv_is_enabled();

to:
int raw_mv = mv && lv //&& raw_lv_is_enabled();

[Africashot]

I don't think dual ISO in h264 could possibly work, first the cpu would not handle dual read out from the sensor and compressing to h264, secondly I believe the compression process would ruin any chance of recovery and last I do not see a way for developing the dual ISO frames out of the .mov file...

[a1ex]

@apefos: please do your homework first. After you will get reasonably good results, post them and I'll enable it for video.

FYI: recovering the image from YUV (be it compressed or not) is one or two orders of magnitude more difficult than from raw data (linear and not debayered). I wanted you to see this with your own eyes, but you didn't seem to get the message.

[apefos]

@a1ex: I think now I understood the message, now I am doing the homework! Thanks.

I did a try now with the 600D with dual iso module enabled and raw rec 1728x992 enabled and I can shoot a JPG large fine in dual iso and the photo retains the lines... great! I just changed the Pic Quality to Large Fine only to allow camera to save the jpg to the card.

now it is night here, so tomorrow morning I will shoot some jpg photos to do a better perceive in the lines retention and upload to show the results.

Tomorrow I can also try to separate the two fields in the jpg photo using a software in computer, interpolate the lines and merge with enfuse to see what happens...

[apefos]

I have good news. It works!

I took a picture in JPG Small Fine with dual iso and raw rec modules enabled, Pic Quality was set to Small Fine.

I deinterlaced and interpolated the same jpg file two times, one for iso 100 and one for iso 800 and then I merged it with enfuse gui in default settings. To separate the isos you just need to choose even or odd field in the deinterlace moment.

the workflow for video will be simple, export an image sequence from the mov h264 file, deinterlace and interpolate each image two times, one for low iso and one for high iso, enfuse each pair with enfuse gui. (all can be done in batch processing)

Probably a simple filter can be created for Virtual Dub to do this task.

see the original dual iso jpg, the two isos deinterlaced and interpolated jpg and the enfused result jpg: (can I try it in h264 video?) thanks!

[3pointedit]

Ooh nice result! I wonder if it's possible to stick this into the h264 encoder? a1ex is there enough processing power in a Rebel like the 550D?

[a1ex]

I took a picture in JPG Small Fine with dual iso and raw rec modules enabled, Pic Quality was set to Small Fine.

Interesting, so Canon's downsampling algorithm did half of the job here. The thing is, with video you have 2 lines at iso 100 and 2 at iso 1600. Can you try to blend the image at full resolution?

I'm interested in this for in-camera preview (not final output), but I'm having big trouble with color casts caused by demosaicing. Just compare the two sub-images at low ISO and the two sub-images at high ISO (they have very different hues). I'm not sure how to solve it.