[IMPOSSIBLE] dual ISO H.264

[apefos]

Let's go on...

I took a look in the Large Fine JPG in the computer and yes, each iso uses TWO lines. So this is impossible to separate using deinterlace... So I did a 50% resize in the image (downscale) to discard half of the lines and the image changed to ONE line per iso as was expected and the deinterlace+interpolation works...

about video: I recorded a RAW video in the 600D using 1728x992 @ 8 fps with dual iso enabled. The DNG sequence in the timeline shows TWO lines per iso, so it will need a downscale to 50% size to make deinterlace works. I can render the DNG sequence into a H264 video file, and then export a jpg sequence from it  to test if the compression will retain the lines separation, but before it I have a question:

why using two lines per iso instead of just one in the dual iso video? this would make sense for interlaced footage, but for progressive footage one line per iso would be enough... maybe the reason is to respect the bayer pattern? because bayer pattern is GRGRGR and next line is BGBGBGBG. is this the reason or the reason is just to get a better separation? Maybe in the dual iso h264 video we can use one line per iso because the h264 video is debayered in camera just like the jpg images are... one line per iso would introduce less aliasing in the final result and no need a 50% downscale before doing the deinterlace... share your thoughts...

about color cast in sub-images: it is normal issue due to when you change brightness the perceive of saturation changes. The saturation level is the same in the two sub-images, but less brightness makes the eye to feel there is more saturation and more brightness in the other image makes the eye to perceive there is less saturation. But saturation level is the same and things become ok in the merged image. The look of the low brightness image tends to me more red and the look of the high brightness image tends to be more green, but in the merged image everything becomes ok. No need to worry about this for the h264 dual iso video, after merging the final result will be ok. If a merged preview could be enabled in camera probably it would be ok also...

if a merged preview could be enabled in camera maybe the camera could record it into a h264 video!!! but there will be enough processing power to do it in 1920x1080 @24fps ? And using two lines per iso a 50% downscale would be a must before deinterlace+interpolate+merge, but using one line per iso would not need the 50% downscale... or another algorithm for deinterlace would be needed, probably the implemented deinterlace method in the CR2HDR.exe is already doing the deinterlace for two lines per iso or is it doing a 50% downscale?...

More about preview in camera: when looking at the dual iso in live view, I perceived that the high iso prevails over the low iso image, maybe due to more light on it, and I cannot judge the exposure for the low iso. I need to disable the dual iso module to see each iso and find the better iso combination before enablind the dual iso for recording. this way I can get great exposue for the low and high iso lines, it is impossible to set the exposure with dual iso enabled. Also, the white balance in raw recording is wrong at the moment of recording, colors need to be tweaked in ACR when importing the DNG sequence to timeline. Maybe the color cast in dual iso preview is just because the raw recording cannot show correct colors in live view, I always see some magenta cast when previewing dual iso in the lcd. I think this is not a big problem, I can deal with it.

[apefos]

password: diuhjc

Not a valid vimeo URL[/vimeo]

[jagnje]

yes! making late night brainstorming possible!

[dmilligan]

@apefos, have you read and understood this (esp. section 5):

http://acoutts.com/a1ex/dual_iso.pdf

[apefos]

Yes, I did a read in the dual iso pdf right now

It says exactly what I was guessing in my previos post, dual iso uses two lines to get red and blue information for each iso considering the bayer patern desing: line1= RGRGRG, line2=GBGBGB

The pdf paper does not say one line per iso is impossible, it just say: "good luck interpolating them" and I would like to try my luck!

For h264 recording the camera will do the debayer process before encoding to H264, so we have a chance for the one line per iso to work. The interlaced television video uses one line per field, so one line separation is possible. also in the tests I did in the video above proves that H264 and JPG can retain information in one line per iso.

As I said in the video, we can work in a second dual iso module designed for h264 and leave the current dual iso module just for raw, this way we will not disturb the raw recording in dual iso, we will develop a second experimental and independent module for h264 (dual_iso_raw.mo & dual_iso_h264.mo)

if one line per iso works it will be just great, if it does not work all we need to do is to go back to two lines per iso. why not to try? let's do a test in a module specific for h264 with one line separation and see what happens...

"I want to believe."

[dmilligan]

Yes, I did a read in the dual iso pdf right now

It says exactly what I was guessing in my previos post, dual iso uses two lines to get red and blue information for each iso considering the bayer patern desing: line1= RGRGRG, line2=GBGBGB

The pdf paper does not say one line per iso is impossible, it just say: "good luck interpolating them" and I would like to try my luck!

I don't think you fully understand how deybayering (demosaicaing) works or you would understand why having only one line is impossible for this (esp. when you have no control over the demosaicaing algorithm - it's being done by the camera).  Think about it: your upper iso is going to only contain green and red information and the lower iso is only going to contain green and blue information (or vice versa). How are you going to make anything useful out of that? You'd have either no red or no blue information in your shadows and the same would be true of the highligts.

Plus, you do realize that the way you are currently doing this, you are sacrificing 75% vertical resolution and gaining very little in terms of dynamic range (from your sample video, I saw pretty much no improvement in dynamic range, simply a 75% reduction in resolution). The way cr2hdr works, you retain full resolution in the mid-tones and only sacrifice 50% resolution in the highlights and shadows. You need to understand how that works, and implement it on 8-bit, nonlinear data (probably almost impossible).

also in the tests I did in the video above proves that H264 and JPG can retain information in one line per iso.

No, it only proves that the particular H264 compression you used in post can retain this information, not that the compression (and more importantly, other image processing such as the slight upscale to 1080p) applied by the camera would retain it.

[apefos]

different opinons and suppositions are normal... mainly when things are not proven yet in the real world... I am the kind of person that does not give up until I see things proven in the real world tests, no matter what people says. we are not loosing anything doing this... and we are loosing more time talking than testing...

the interlaced footage 60i and 50i from dslr and camcorders uses one line per filed and it works. why the one iso per line would not work? it is the same principle. let's see how the camera will demosaic and debayer the sensor with one line per iso, we cannot say things before seeing it...

using deinterlace+interpolation in one line per iso video will turn 1920x1080 into 1920x540 for each iso and it will interpolate pretty well. if we use two lines we would need a special software for deintercale images considering two lines to avoid the 50% downscale. But the best will be to see what happens recording one line per iso before to worry about how to deinterlace two lines per iso images...

@a1ex , @1% , I did the home work, and i did pretty well, so can you enable the h264 recording in the dual iso module and also enable two versions, one version with two lines per iso and other version with one line per iso? this way I can test it and we will se the real world results. you can create these two modules separated from the module for raw (dual_iso_h264_oneline.mo and dual_iso_h264_twolines.mo) it deserves a try, only a real world test will give us the real results, all talking is just suppositions, we need to see what happens in the camera and in the computer.

if you preffer I can do the tests in private videos and not share the module, send me a pm, or I can do it public if there is no problem.

thanks

[dmilligan]

different opinons and suppositions are normal... mainly when things are not proven yet in the real world...

the burden of proof is on you to prove it is feasible, not on us to prove to you that it isn't

the interlaced footage 60i and 50i from dslr and camcorders uses one line per filed and it works. why the one iso per line would not work?

you clearly still don't understand a bayer array or anything I said: http://en.wikipedia.org/wiki/Color_filter_array

it is the same principle.

no it's not, dual iso is not the same as frame interlacing, they are extremely different things

let's see how the camera will demosaic and debayer the sensor with one line per iso, we cannot say things before seeing it...

yes we can if we know enough about how the sensor works:

If you make every other line a different ISO, either all the blue sensels or all the red sensels will be at the same ISO and there will be none at your other ISO

Example:
ISO100 RGRGRGRG
ISO800 GBGBGBGB
ISO100 RGRGRGRG
ISO800 GBGBGBGB
ISO100 RGRGRGRG
ISO800 GBGBGBGB

a1ex spells this out plainly in the pdf, in a section you quoted, but clearly didn't understand fully. "good luck interpolating that" is sort of sarcasm, he really means its impossible to interpolate that. Plus, you don't have any control of the interpolation, the camera is going to be doing it according to it's own hardwired demosaicing algorithm that is not expecting different ISOs, it will be completely wrong.

@a1ex , @1% , I did the home work, and i did pretty well, so can you enable the h264 recording in the dual iso module and also enable two versions, one version with two lines per iso and other version with one line per iso? this way I can test it and we will se the real world results. you can create these two modules separated from the module for raw (dual_iso_h264_oneline.mo and dual_iso_h264_twolines.mo) it deserves a try, only a real world test will give us the real results, all talking is just suppositions, we need to see what happens in the camera and in the computer.

1% has given you the exact code change you need to do to make it work (except he left out a semicolon), why don't you just do it yourself? (see here to get started: http://www.magiclantern.fm/forum/index.php?topic=991.0)

I'm not saying this is totally impossible, but IMO it will be very hard and not worth the effort. You might manage to get back to the 14-bit dynamic range of the original RAW, but you are going to pay a penalty in resolution, moire, and aliasing that will not make it worth it.

[apefos]

Are you playing the devil's advocate? or you just do not have the inovative spirit? The inovative spirit does not listen to the impossible, it prefers to fail doing it, because doing it allow a chance for success or to learn another way to reach the success. you need to watch the JOB movie to try to implement some inovative spirit on you.

I still believe one line per iso is possible because the camera algorithm uses the surrounding pixels to reconstruct the hole image. in raw the one line per iso could be a different history, but for h264 we have a chance: maybe the camera debayer algotihm can reconstruct the rgb data for all the pixels and all the image even with one line per iso, let's give a try to the camera debayer algorithm using one line per iso. if it fails we just need to go back for two lines per iso, it is so simple to try...

I just do not understand why people keep saying "impossible, impossible" without just giving it a try... just trusting in theory. somethimes teory can be surpassed, I believe the camera debayer algorithm can surpass this teory. If you want do to something you just need to close your eyes and ears to people who say impossible, it is better to fail trying to do... nobody will die doing this... and the cameras will not explode...

Unfortunately I do not know how to compile and do not have time to learn. I did what I could, now it is up to the developers to implement it and allow us to test. I think it deserves a try in one line per iso and in two lines per iso.

I will stop posting for a while... I have work to do here and I think it is also time for stop talking and allow  the developers to work on it.

[arrinkiiii]

...you are going to pay a penalty in resolution, moire, and aliasing that will not make it worth it.

Exactly ! 

[Stedda]

... now it is up to the developers to implement it and allow us to test. I think it deserves a try in one line per iso and in two lines per iso.

... I have work to do here and I think it is also time for stop talking and allow  the developers to work on it.

 

Where do these people come from?

[apefos]

It seems I found a bug in dual iso. Sometimes in 600D T3i in 1728x992 and in 960x544 the recorded RAW video in Dual Iso shows moving lines. the lines scroll down, I called it waterfalling behavior. See the video: (password = diwfb)

Not a valid vimeo URL[/vimeo]

[dmilligan]

Not a bug, cr2dng can handle it (which lines are which are automatically determined by the algorithm)

[apefos]

Interesting, so it is safe for RAW recording! But for H264 recording maybe it could be a problem because it changes the lines position for deinterlace...

It seems I found another bug, I did some try to drag the DNG sequence over the CR2HDR but I got this screen and nothing happens... (using 600D T3i) see:

[dmilligan]

you need to install dcraw

[apefos]

thanks!

the dcraw and the exif are in the same folder of cr2hrd now. it works, but there is a line which says:

black subtract didn't work

and the reconstructed images are underexposed, see the text: (I found this: "After postprocessing, you will get a DNG that looks like a dark ISO 100 shot, but you can bring the exposure back up and be delighted by how little noise is present in those recovered shadows.") so it seems to be working, but there are lots of magenta or cian hot pixels...

[apefos]

I did more tests and in 1280x720 raw recording in dual iso the waterfalling problem did not happen yet. Sometimes it happens in 1728x992 and in 960x544, It seems to be an intermittent problem, it comes and goes...

For Raw dual iso, as said in previous post, the merging algorithm can deal with this issue, but if h264 dual iso will be implemented this bug needs to be fixed for computer deinterlace and interpolation, or a special extracting program needs to be created.

For RAW recording I understood the need of two lines per iso, the two lines combo will have the red, green and blue information to allow the image reconstruction for each iso.

But for h264 dual iso, probably the camera debayer algorithm can reconstruct the image using just one line per iso, because this is the nature of the camera debayer algorithm, it uses the surrounding pixels to reconstruct each pixel from the bayer pattern. In the interlaced footage from camcorders with 1920x1080 pixels sensors the even and ood fields differs a lot from each other in moving images and the camera can reconstruct it and no problem. also there are lots of light changes in images and the camera debayer algorithm always do a great job. just remember the canon hv20 camcorder. it is a 1920x1080 sensor and records 24p and 60i pretty good, no moire and no aliasing.

These are the reasons I strongly encourage the developers to try one line per iso in dual iso h264 recording. maybe it can surprise us with great results. and the workflow is easy to do in batch processing in computer. the video I did proves the lines separation can be retained.

if we can get two images with 1920x540 for each iso, the interpolation will be pretty good, and the hdr video will shine. there is also the instant hd and blow up plugins whic can do this interpolation without aliasing. if one line per iso in dual iso h264 works it will be revolutionary... no need lots of cards, the inexpensive Rebels will do it at 1920x1080, sound in camera... very small or no aliasing...

I do not know how to compile and it is hard for me to learn... I hope the devs give it a try...

sorry for lots of talk about this feature, but it is very promissing and I am really interested on it

thanks!

[painya]

What if you could record to an atomos ninja at a different ISO while simultaneously recording h.264? 

[apefos]

I think the external recorder can be good idea to allow less compression in recorded video and a cleaner result, but for the low budget people the main goal is to record in h264 in the memory card. After doing the test in the video, I believe the h264 recording will be fine to retain the lines. But first, the main challenge is to test if the camera debayer algorithm can reconstruct the image when using one line per iso, and send a good even/odd fields with the different isos in a reconstructed image  to h264 recording or to the external recorder. I believe yes, but it needs real world tests.

[dmilligan]

But for h264 dual iso, probably the camera debayer algorithm can reconstruct the image using just one line per iso, because this is the nature of the camera debayer algorithm, it uses the surrounding pixels to reconstruct each pixel from the bayer pattern. In the interlaced footage from camcorders with 1920x1080 pixels sensors the even and ood fields differs a lot from each other in moving images and the camera can reconstruct it and no problem. also there are lots of light changes in images and the camera debayer algorithm always do a great job. just remember the canon hv20 camcorder. it is a 1920x1080 sensor and records 24p and 60i pretty good, no moire and no aliasing.

If you make every other line a different ISO, either all the blue sensels or all the red sensels will be at the same ISO and there will be none at your other ISO

Example:
ISO100 RGRGRGRG
ISO800 GBGBGBGB
ISO100 RGRGRGRG
ISO800 GBGBGBGB
ISO100 RGRGRGRG
ISO800 GBGBGBGB

a1ex spells this out plainly in the pdf, in a section you quoted, but clearly didn't understand fully. "good luck interpolating that" is sort of sarcasm, he really means its impossible to interpolate that. Plus, you don't have any control of the interpolation, the camera is going to be doing it according to it's own hardwired demosaicing algorithm that is not expecting different ISOs, it will be completely wrong.

[apefos]

[NedB]

@apefos: "Doubting" Thomas, right? Clever. @dmilligan and others: maybe it's best to stop responding? You seem to be just encouraging him  Cheers!

[apefos]

ok, guys, I give up! I learned something in my life: "I agree that we disagree" (about the need of testing it). I was just hoping and wondering that h264 could be different from raw and could handle one line per iso with the internal camera debayer algorithm, maybe someday I learn to compile...

[bnvm]

You don't need to learn to compile for this, what you need to do is learn about the process of debayering and how it works. Once you understand that and understand that the dev's don't have any control over the camera's internal debayerer you will see why single line iso's won't work.

[apefos]

Ok, let's forget about one line per iso...

Can I just ask for enable the dual iso recording in h264? (just enable and keep the two lines per iso)

a1ex told me he would enable it after my homework, and I did the homework in photo and video...

@apefos: please do your homework first. After you will get reasonably good results, post them and I'll enable it for video.

thanks!