Video - dual iso for 5D2

[reddeercity]

Further examination of the Cdng from MLVFS with the dual iso option enable and CR2HDR with Exiftool

This one was process though MLVFS With dual iso enabled/default setting

Code: [Select]

ExifTool Version Number         : 10.09
File Name                       : M13-2146_000000.DNG
Directory                       : C:/Users/FX8350/Desktop/New folder
File Size                       : 3.3 MB
File Modification Date/Time     : 2016:03:13 22:46:43-06:00
File Access Date/Time           : 2016:03:16 13:57:24-06:00
File Creation Date/Time         : 2016:03:16 13:57:24-06:00
File Permissions                : rw-rw-rw-
File Type                       : DNG
File Type Extension             : dng
MIME Type                       : image/x-adobe-dng
Exif Byte Order                 : Little-endian (Intel, II)
Subfile Type                    : Full-resolution Image
Image Width                     : 1856
Image Height                    : 928
Bits Per Sample                 : 16
Compression                     : Uncompressed
Photometric Interpretation      : Color Filter Array
Fill Order                      : Normal
Make                            : Canon
Camera Model Name               : Canon EOS 5D Mark II
Strip Offsets                   : 65536
Orientation                     : Horizontal (normal)
Samples Per Pixel               : 1
Rows Per Strip                  : 928
Strip Byte Counts               : 3444736
Planar Configuration            : Chunky
Software                        : MLVFS
Modify Date                     : 2016:02:13 21:46:43
CFA Repeat Pattern Dim          : 2 2
CFA Pattern 2                   : 0 1 1 2
Exposure Time                   : 1/53
F Number                        : 1.4
ISO                             : 100
Sensitivity Type                : ISO Speed
Exif Version                    : 0230
Subject Distance                : 180 m
Focal Length                    : 50.0 mm
Lens Model                      : 50mm
DNG Version                     : 1.4.0.0
Unique Camera Model             : Canon EOS 5D Mark II
Black Level                     : 7176
White Level                     : 60000
Default Crop Origin             : 0 0
Default Crop Size               : 1856 928
Color Matrix 1                  : 0.5309 -0.0229 -0.0336 -0.6241 1.3265 0.3337 -
0.0817 0.1215 0.6664
Color Matrix 2                  : 0.4716 0.0603 -0.083 -0.7798 1.5474 0.248 -0.1
496 0.1937 0.6651
As Shot Neutral                 : 0.4580008629 1 0.5775139181
Baseline Exposure               : 0
Calibration Illuminant 1        : Standard Light A
Calibration Illuminant 2        : D65
Active Area                     : 0 0 928 1856
Forward Matrix 1                : 0.8924 -0.1041 0.176 0.4351 0.6621 -0.0972 0.0
505 -0.1562 0.9308
Forward Matrix 2                : 0.8924 -0.1041 0.176 0.4351 0.6621 -0.0972 0.0
505 -0.1562 0.9308
Time Codes                      : 00:00:00.00
Frame Rate                      : 23.976
Baseline Exposure Offset        : 0
Aperture                        : 1.4
CFA Pattern                     : [Red,Green][Green,Blue]
Image Size                      : 1856x928
Megapixels                      : 1.7
Shutter Speed                   : 1/53
Focal Length                    : 50.0 mm
Light Value                     : 6.7
-- press RETURN --


This was process thought BarracudaGUI for cr2hdr dual iso

Code: [Select]

ExifTool Version Number         : 10.09
File Name                       : M13-2146_000000 (2).DNG
Directory                       : C:/Users/FX8350/Desktop/New folder
File Size                       : 3.6 MB
File Modification Date/Time     : 2016:03:15 15:48:31-06:00
File Access Date/Time           : 2016:03:16 14:00:28-06:00
File Creation Date/Time         : 2016:03:16 14:00:28-06:00
File Permissions                : rw-rw-rw-
File Type                       : DNG
File Type Extension             : dng
MIME Type                       : image/x-adobe-dng
Exif Byte Order                 : Little-endian (Intel, II)
Image Description               :
Make                            : Canon
Camera Model Name               : Canon EOS 5D Mark II
Orientation                     : Horizontal (normal)
Software                        : MLVFS
Modify Date                     : 2016:02:13 21:46:43
Artist                          :
Subfile Type                    : Full-resolution Image
Image Width                     : 1856
Image Height                    : 928
Bits Per Sample                 : 16
Compression                     : Uncompressed
Photometric Interpretation      : Color Filter Array
Strip Offsets                   : 4330
Samples Per Pixel               : 1
Rows Per Strip                  : 928
Strip Byte Counts               : 3444736
X Resolution                    : 180
Y Resolution                    : 180
Planar Configuration            : Chunky
Resolution Unit                 : inches
CFA Repeat Pattern Dim          : 2 2
CFA Pattern 2                   : 0 1 1 2
Black Level                     : 8192
White Level                     : 40000
Default Crop Origin             : 0 0
Default Crop Size               : 1856 928
Active Area                     : 0 0 928 1856
Opcode List 1                   : (Binary data 28 bytes, use -b option to extrac
t)
XMP Toolkit                     : Image::ExifTool 10.02
Subject                         : Dual-ISO
Copyright                       :
Exposure Time                   : 1/53
F Number                        : 1.4
Exposure Program                : Not Defined
ISO                             : 100
Sensitivity Type                : ISO Speed
Exif Version                    : 0230
Date/Time Original              :
Shutter Speed Value             : 1
Aperture Value                  : 1.0
Exposure Compensation           : 0
Max Aperture Value              : 1.0
Subject Distance                : 180 m
Metering Mode                   : Unknown
Flash                           : No Flash
Focal Length                    : 50.0 mm
Sub Sec Time                    :
Sub Sec Time Original           :
Focal Length In 35mm Format     : 0 mm
Lens Model                      : 50mm
TIFF-EP Standard ID             : 1 0 0 0
Serial Number                   :
DNG Version                     : 1.3.0.0
DNG Backward Version            : 1.3.0.0
Unique Camera Model             : Canon EOS 5D Mark II
Color Matrix 1                  : 0.4716 0.0603 -0.083 -0.7798 1.5474 0.248 -0.1
496 0.1937 0.6651
Analog Balance                  : 1 1 1
As Shot Neutral                 : 0.3895824105 1 0.4863276282
Baseline Exposure               : undef
Baseline Noise                  : 1
Baseline Sharpness              : 1.333333333
Linear Response Limit           : 1
Calibration Illuminant 1        : D65
Frame Rate                      : 23.976
Aperture                        : 1.4
CFA Pattern                     : [Red,Green][Green,Blue]
Image Size                      : 1856x928
Megapixels                      : 1.7
Shutter Speed                   : 1/53
Focal Length                    : 50.0 mm
Light Value                     : 6.7
-- press RETURN --

There a few thing that are different , White level , Black Level are off

mlvfs-- Black Level   : 7176
                 White Level   : 60000
                 Strip Offsets : 65536
                  Forward Matrix 1 : 0.8924 -0.1041 0.176 0.4351 0.6621 -0.0972 0.0505 -0.1562 0.9308
                  Forward Matrix 2  : 0.8924 -0.1041 0.176 0.4351 0.6621 -0.0972 0.0505 -0.1562 0.9308

cr2hdr---Black Level  : 8192
                 White Level  : 40000
                  Strip Offsets  : 4330
                   Color Matrix 1 : 0.4716 0.0603 -0.083 -0.7798 1.5474 0.248 -0.1496 0.1937 0.6651
No time code , only one forward matrix & it not right.

So I don't think its the dual iso module problem , it some think to do with cr2hdr as I extracted the cr2hdr process one with mlvfs on windows7 with Pismo quick mount

[DeafEyeJedi]

Excellent work @reddeercity!

[jpegmasterjesse]

I went looking for proof that this gets us the kind of dynamic range we're looking for. With MlRawViewer the MLV file looks like what you'd expect - interlaced, but data-rich.  I exported as DNG's from MlRawViewer from there and used Cr2HDR to convert.  Perhaps that's a bad workflow, because the resulting dngs were missing any information about shadows or blacks.

I imported the original, interlaced DNG's into lightroom. Added some grain and noise reduction.  The interlacing isn't so noticeable.
https://gfycat.com/WastefulGraciousAndeancat

The point of that being that YES, this is a step in the right direction.  Even before the conversion, I definitely felt as if I was working with the ISO's I specified in Dual-ISO.

THANKS!  If I can provide some more objective information for anyone, please let me know.

[Danne]

Feel free to test out A1ex experimental cr2hdr build for the 5D mark 2 in MLP. (thanks dfort for compiling)

From discussion here
http://www.magiclantern.fm/forum/index.php?topic=7139.msg164375#msg164375

Here is a version of MLP with the experiment cr2hdr build for 5D mark 2 (and other cams to it seems). It will work right out of the box for developing the dng files. Havn,t had time to test it out and don,t own a 5d mark 2.

MLP_5D2
https://www.dropbox.com/s/vawstdji3yif8sk/MLP_5D2.zip?dl=0

Works like this
1 - Download and then double click MLP(.workflow) to install
2 - Right click and select MLP from the contextual menu on a folder containing dual iso MLV or RAW or MLV-lite footage. Do this twice cause first time MLP creates a folder structure then quits. The second time will start processing. Same white level(avoid flicker) will be applied to all files and it works multithreaded.

[DeafEyeJedi]

Great progress everyone on board so far and this is why I love you guys. Humans are amazing. Especially in this Community. Thanks for the quick update on MLP aiming for 5D2 users (Hello there we didn't forget about you!) and honestly none of this would have happened this promptly if it weren't for your suggestion to post my test files from my friend's 5D2. Bold move, @Danne!

[emoji106]

[reddeercity]

Just tested update CR2HDR process the dual iso without any problem , there is a little difference in the mid tone area
there seems to more mid detail with MLVFS  but more shadow detail with the update CR2HDR.

One think I notice is with cr2hdr process file in resolve I had to use less exposure then mlvfs process file
by about half , e.g. cr2hdr was about 2.5 where mlvfs was 4.6 , interesting .

Link to my dropbox with Dual ISO Cdng if any one would like check them out

Non process dual ISO --> https://www.dropbox.com/s/qnzst1571q766pu/Non_process_M13-2146.MLV.frame_000000.dng?dl=0
New CR2HDR dual ISO       https://www.dropbox.com/s/3w24ifc4opurmgi/New_CR2HDR_M13-2146.MLV.frame_000000.DNG?dl=0
Old CR2HDR dual ISO        https://www.dropbox.com/s/3al9oe8oerahtzv/Old_CR2HDR_M13-2146_000000.DNG?dl=0
MLVFS dual ISO (PC ver.)   https://www.dropbox.com/s/c7punwsgezdmopt/M13-2146_000000.dng?dl=0

Non process Dual ISO 100/800 1856x928  from 5D2

Non_process_M13-2146.MLV.frame_000000 by RedDeerCityTV, on Flickr

New CR2HDR process though BarracudaGUI


New_CR2HDR_M13-2146.MLV.frame_000000 by RedDeerCityTV, on Flickr

MLVFS process dual iso on windows7

M13-2146_000000 by RedDeerCityTV, on Flickr

Old CR2HDR , wrong white & black levels


Old_CR2HDR_M13-2146_000000 by RedDeerCityTV, on Flickr

[tecgen]

Regarding the 50d, someone should try to manually compare (with ADTG-gui) the actual values CMOS[0] for the iso photo mode with those of the video mode(for the same iso).
iso 100, 200, 400, 800, 1600.

As far as I understand the implementation for the 5D2 and 50D is exactly the same, just with different address values.

Could someone tell us in more detail, how we can find out the values for the 50D? When we know the value for other cameras next to the 5D2, that also use only one register, we could change the implementation to a more generic "one register" solution, right?

[waza57]

For 50D :

As far as I understand the implementation for the 5D2 and 50D is exactly the same, just with different address values.

I don't know at all but if my little method can be applied for 50D:

Without dual iso:
We do learn what is the CMOS[0] address when LiveView is ON. (start at iso 100 for example)
is it the same as photo mode (probably not)?
What is his value and is it the same than photo mode (for same iso 100) ?

and make the same thing for iso 200.... it would be enough to have an idea about 50D.

To do this :
We must have a 50d (I have not).
Enable adtg_gui module after compiled it for your nightly builds installed( except if a precompiled one exist, I don't know that).
Then we could investigate the addresses of CMOS[0] and its values.

[DeafEyeJedi]

MLP_5D2
https://www.dropbox.com/s/vawstdji3yif8sk/MLP_5D2.zip?dl=0

Works like this
1 - Download and then double click MLP(.workflow) to install
2 - Right click and select MLP from the contextual menu on a folder containing dual iso MLV or RAW or MLV-lite footage. Do this twice cause first time MLP creates a folder structure then quits. The second time will start processing. Same white level(avoid flicker) will be applied to all files and it works multithreaded.

Question though ... if I were to test 5D2 files after installing this version ... Do I need to reinstall back to the original MLP from your thread? Wasn't sure if this was designed to be used ONLY with 5D2 atm? If so, would be nice if we could add 'MLP_5D2' under the service list unless you think you will eventually make this into the classic MLP?

Thanks, D!

[Danne]

You can run it as usual use the command to transfer the other stuff. Or grab the cr2hdr(I renamed it cr2hdr20bit) from the 5D2 version and put it in regular MLP if you know where to do it.

[DeafEyeJedi]

Got it -- Will do just that. Thanks!

However, we may have a problem with the new cr2hdr binary (I ran this twice just to be sure) and both converted DNG's and ProRes4444 show a bunch of random dropped/pinkish frames (or something else strange is going on) but when I then took the original MLV's (same 5D2 test files that I posted from earlier) and rendered through MLVFS (which still uses the older Binary -- not the one that @dfort uploaded) via DR12 which spitted out fine like normal as it seems.

[dfort]

Humm--looks like people are posting the same comments on this topic and the main Dual ISO topic. Makes things a bit confusing.

In any case, that cr2hdr I posted is for testing so it looks like you got to be the crash test dummy 

I don't have any 5D2 Dual ISO MLV files but the posted tests do look very promising. Maybe the 7D and 50D are next?

[Danne]

Isn,t David using his own created dual iso algoritm in MLVFS?
Sean, can you try the same clip with the regular MLP and check for dropped frames?

[DeafEyeJedi]

Humm--looks like people are posting the same comments on this topic and the main Dual ISO topic. Makes things a bit confusing.

In any case, that cr2hdr I posted is for testing so it looks like you got to be the crash test dummy 

I don't have any 5D2 Dual ISO MLV files but the posted tests do look very promising. Maybe the 7D and 50D are next?

It's still online from my earlier post if you would like to play with. I'm also trying to get ahold of my co worker to shoot some more Dual-ISO footage from his 5D2 just to have options for us to troubleshoot this bad boy down.

Isn,t David using his own created dual iso algoritm in MLVFS?
Sean, can you try the same clip with the regular MLP and check for dropped frames?

Ah, well the one thing I'm surprised about is the fact that @dmilligan hasn't updated his cr2hdr binary yet (which I thought was needed in order for 5D2 Dual ISO to work?) however I then ran another test running with just MLVFS and default settings for cr2hdr20-bit and applied Cinelog-C in AE so here's what it looks like...

Also here's the quick comparisons of in between MLP and MLVFS...

@Danne -- I am currently rendering the same test files from 5D2 with the latest MLP from original post of your thread just to see what happens with the random dropped/pinkish frames on both the DNG's and ProRes4444 ... Stand by!

*EDIT*

Just finished spitting out DNG's and ProRes4444 which showed these same 'Green Cast' issues from few days ago BUT no dropping/pinkish frames.

It seems with old cr2hdr which shows green cast and new cr2hdr which shows random pinkish frames but no green cast ... Weird?

How exactly can we narrow down the culprit on the latest cr2hdr?

Also let me know if I need to report this in the orignal Dual-ISO thread for @a1ex to see or not just yet? [/size][/size][/b]

[reddeercity]

Seems to be ok on the PC side with the new updated cr2hdr
No green cast

[reddeercity]

Just a quick report , I did have a Liveview freeze up with dual iso module loaded,
basically I was doing very quick pan without recording raw video and needed a battery pull
re-start cam every back to normal.
so watch out for any quick motion with cam as this can freeze the 5D2 Liveview with dual ISO.

[Danne]

The problem seems to be with the experimental cr2hdr build @deafeyejedi. Maybe good ide to continue this discussion in the other dual iso thread?

If you report green cast still going on you have to clarify what you are testing. If you run the official non experimental build of MLP of course there will be a green cast. If you run the experimental 5D2 build it should not.
The test were after was the dropframe issue. If it really is an issue. Hard to say. Maybe let reddeercity try out the file with the barracuda workflow.

[tecgen]

Just a quick report , I did have a Liveview freeze up with dual iso module loaded,
basically I was doing very quick pan without recording raw video and needed a battery pull
re-start cam every back to normal.
so watch out for any quick motion with cam as this can freeze the 5D2 Liveview with dual ISO.

Do you still use the dual iso module on your 5d2? Did your Liveview freeze just once or did it happen more often?

@all
Did anyone except Nikfreak already tried to implement dual iso video for the 50d?

[reddeercity]

No I don't use it much and no I haven't had a lockup but I don't pan fast so I don't have problems
Works great but a nightmare in processing , unless you have a very fast Computer & Raid0 or storage SSD .

[tecgen]

> No I don't use it much and no I haven't had a lockup but I don't pan fast so I don't have problems

Good to hear that it the approach works somehow reddeercity. Maybe your freeze of the Liveview on the was related to something else. I'll test it on the 5d2 by myself later.
First I want to check out if we can get dual iso video on the 50D. I'm sure the implementation for the 50D is very similar to the 5d2. We know already the address of CMOS[0] in movie mode on the 50D. I used the implementation that works for the 5d2 and tried to get it working for the 50D.
https://bitbucket.org/tecgen/magic-lantern/branch/5D2-50D-dualiso-video#diff

I don't know why, but dualiso does not yet work when Liveview is turned on. I checked the CMOS[0] address for ISO 100 in movie mode. It's correct. I played around a bit with atdg_gui. By changing the value to 8, 18 or 28 I was able to record a raw video of something that looks similar to dualiso. I need to check the DNGs if these are just horizontal stripes or not.

> Works great but a nightmare in processing , unless you have a very fast Computer & Raid0 or storage SSD .
Sure it needs more time and steps in postprocessing. But it's the same on every dualiso video supported camera model. No matter if your using a 5d3 or 550d. It just takes some time I think it would be cool to have this dualiso video feature on all ML cameras. I don't want to waste the effort waza57 put into the implementation for the 5d2. At least this implementation should be in the nightly build when its stable enough.

[Deadcode]

Please help me!

I tried the dual ISO module above. It worked but once the camera freezed and i couldn't do anything. So i removed the battery. I know it's possibile to cause permanent damage on the camera or CF card, maybe that happened this time.

After restoring the camera to the original firmware, i still get unusable raw files and unusable .mov-s.
Check it out: https://drive.google.com/drive/folders/0Bwoum3ZKc_jNR3lLbmV3S1RodUk?usp=sharing

The strange thing is: ML Raw video is not affected! only still picture raw and compressed video. (jpg's are not perfect, but OK)
What can i do? If i have to send the camera to repair what module should be changed?

Thank you for your help!


edit: 5D markII, Lexar 64GB 1066x

[a1ex]

If i have to send the camera to repair what module should be changed?

The card.

[dfort]

how you use DUAL ISO in video recording ?

It's an old module that was being tested on the 5D2.  I don't have any documentation on who created it.

Here's how to find out more about that old module. In the modules directory there is a shell script called module_hginfo_dump.sh. Run it on the module--I ran it from the modules directory:

Code: [Select]

./module_hginfo_dump.sh /[PATH_TO]/dual_iso.mo
Name        : Dual ISO
Author      : a1ex
License     : GPL
Summary     : Dynamic range improvement by using two ISOs in one picture
Forum       : http://www.magiclantern.fm/forum/index.php?topic=7139.0
Description : Increases dynamic range by sampling the sensor at two different
              ISOs, switching ISO for every other line pair. This trick cleans up
              shadow noise, resulting in a dynamic range improvement of [...]
Help page 1 : Quick start
             
               *  Start at ISO 100 in Canon menu [...]
Help page 2 : Tips and tricks
             
               *  Do not use dual ISO for regular scenes that don't require a very [...]
Last update : ab1e90c on 2015-11-24 09:50:19 UTC by a1ex:
              cr2hdr: moved safeguard from median_int_wirth to kth_smallest_i...
Build date  : 2016-03-13 16:24:44 UTC
Build user  : cmoi@kiranfou7

The "Last update" is the changeset so knowing that you can search for it on Bitbucket and if you have a development environment setup you can roll back to that version like this:

Code: [Select]

hg update ab1e90c
Now you can look at the source code for that old module.

So how different is it from the current unified version?

Code: [Select]

hg diff -r ab1e90c:unified modules/dual_iso/dual_iso.h
Not much though there is one significant change:

Code: [Select]

diff -r ab1e90c88b95 -r 36acb0f9c3e2 modules/dual_iso/dual_iso.c
--- a/modules/dual_iso/dual_iso.c       Tue Nov 24 11:50:19 2015 +0200
+++ b/modules/dual_iso/dual_iso.c       Fri Feb 02 22:07:41 2018 +0100
@@ -556,8 +556,12 @@
 
     int mvi = is_movie_mode();
 
-    int raw = mvi ? FRAME_CMOS_ISO_START && raw_lv_is_enabled() : ((pic_quality & 0xFE00FF) == (PICQ_RAW & 0xFE00FF));
+    /* default checks will not work here - we need full-sized raw in photo mode */
+    int raw = mvi ? raw_lv_is_enabled() : ((pic_quality & 0xFE00FF) == (PICQ_RAW & 0xFE00FF));
 
+    if (mvi && !FRAME_CMOS_ISO_START)
+        MENU_SET_WARNING(MENU_WARN_NOT_WORKING, "Dual ISO does not work in movie mode on your camera.");
+   
     if (!raw)
         menu_set_warning_raw(entry, info);
 }
@@ -832,7 +836,7 @@
         CMOS_FLAG_BITS = 2;
         CMOS_EXPECTED_FLAG = 0;
     }
-    else if (is_camera("700D", "1.1.4"))
+    else if (is_camera("700D", "1.1.5"))
     {
         is_700d = 1;   

Why was that block put in there? Let's run "blame" on dual_iso.c -- this can be done on Bitbucket. Looks like blocking dual_iso on the 5D2 was put there for a reason, though the reason isn't clear in the commit message.

Time to search the forum. Looks like someone got quite far with it here: Video - dual iso for 5D2 (recursive link) However, no pull request was submitted.

[reddeercity]

https://bitbucket.org/waza57/magic-lantern/commits/7be4cd7f975ef25e98143d10da26712e5ec8ca77?at=dual_iso_video_5d2
Yea it works just fine 

[dfort]

Great! Thanks reddeercity

@waza57 -- What are you up to these days? I see you're still active on the forum and Bitbucket.