3K/UHD 5D2 Raw development and Other Digic IV Cams

[reddeercity]

Trying to reproduce a1ex's full width experiment on 5d2 , It not totally right but close
I had a problem with cmos[2]  for some reason I couldn't get 0x00e 

For full width LiveView:
CMOS[2] = 0x00E
C0F06008: 0x27705DB (only the lowest half appears to matter on this camera)
C0F06084: 0x30037 (copied lowest half from photo mode)
C0F06088: 0x46A0B87 (copied lowest half from photo mode)
=> 5632x1074 in mlv_rec, 12.5 FPS.

everything else work out , not sure why I can't , the best I could get is 0xe , So left it at default "0x10e" (will try later again on "cmos[2]" )
Did a Image dump , it did save a successful DNG & Liveview 422 image

I was able to save a mlv in 10bit 12.5fps 3520x1074 about 100 frames (max size in normal 3x crop_mode)
Here a sample frame M02-0002_00040.png


The DNG dump was 5632x1074 , 5.239 aspect ratio . RAW-004.DNG tif file RAW-004.tif

Active Area    :52 160 1127 5792

Code Select Expand

File Name                       : RAW-004 (2).DNG
File Size                       : 11 MB
File Modification Date/Time     : 2018:10:02 00:01:54-06:00
File Access Date/Time           : 2018:10:02 00:51:47-06:00
File Creation Date/Time         : 2018:10:02 00:51:47-06:00
Camera Model Name               : Canon EOS 5D Mark II
Orientation                     : Horizontal (normal)
Software                        : Magic Lantern
Subfile Type                    : Full-resolution Image
Image Width                     : 5792
Image Height                    : 1127
Bits Per Sample                 : 14
Compression                     : Uncompressed
Photometric Interpretation      : Color Filter Array
Strip Offsets                   : 33792
Samples Per Pixel               : 1
Rows Per Strip                  : 1127
Strip Byte Counts               : 11423272
X Resolution                    : 180
Y Resolution                    : 180
Planar Configuration            : Chunky
Resolution Unit                 : inches
CFA Repeat Pattern Dim          : 2 2
CFA Pattern 2                   : 0 1 1 2
Black Level                     : 1277
White Level                     : 16200
Default Crop Origin             : 0 0
Default Crop Size               : 5632 1075
Active Area                     : 52 160 1127 5792
Unique Camera Model             : Canon EOS 5D Mark II
Color Matrix 1                  : 0.4716 0.0603 -0.083 -0.7798 1.5474 0.248 -0.1
496 0.1937 0.6651
Analog Balance                  : 1 1 1
As Shot Neutral                 : 0.473635 1 0.624
Calibration Illuminant 1        : D65
Frame Rate                      : 25
CFA Pattern                     : [Red,Green][Green,Blue]
Image Size                      : 5792x1127
Megapixels                      : 6.5


The liveview dump


I what to reproduce all of a1ex crop_rec experiment on the 5d2 , specially the 50p 1856x704
I would like to put these resolution's in the crop_rec module at some point , even though the write speed on the 5d2 is slow right now it would be nice to have this part done.

[reddeercity]

This is interesting , after recording that short mlv file @ 3520x1074 @12.5fps I was getting very high right speeds and I
don't change anything else

Code Select Expand

# Config file for module mlv_rec (MLV_REC.MO)
mlv.video.enabled = 1
mlv.res.x = 9
mlv.bpp = 0
mlv.aspect_ratio = 16
mlv.write_speed = 8527
mlv.preview = 2
mlv.display_rec_info = 2
So 85.27MB/s  , not sure why maybe 12.5 fps ? can't see it
this is a mystery why , but I like it !

[theBilalFakhouri]

Not really a mystery it's same in all other cameras

Yes, it's normal.

If capture speed is greater than writing speed (i.e. recording is not continuous): memory bus is the main limit. If you capture at a higher data rate (e.g. higher resolution and/or FPS), there won't be enough bandwidth left for file I/O at full speed. I'm not sure how DMA priorities are working, or whether there's any way to adjust them. This is why the hacks that freeze LiveView are able to achive slightly better speed - they turn off some "useless" image streams, freeing some memory bandwidth.

Otherwise... the buffer will be mostly empty and the speed will be limited by image capture speed. In this case, mlv_lite will try to "benchmark" only when the card is actively writing, and may print the "idle" time as well. However, this method is just an approximation: there will be some overhead caused by small buffer sizes (as the raw recording task starts writing to card as soon as it has completed frames in the buffer, rather than waiting for a huge contiguous chunk to be completed). No big deal, as in this case, recording will be continuous anyway. It may affect the estimated time for the next recording.

[reddeercity]

Yes finally 50p on the 5D2 



Got a1ex's experiment to run on my 5d2 , it was a little tricky , had a few freezes with c0f0713c & c0f07150
and take me a while to figure out that I need to start with 30p NTSC in the Canon Menu 



but after that I needed to adjust the registers in a certain order or else it would lockup/freeze
So it go like this :

Code Select Expand

CMOS[1]
c0f06088
c0f06008
ADTG12[100c]
c0f0713c
c0f07150
c0f06014
Got not too bad of a image , needs work yet to clean up totally , did a image dump , got DNG, 422 .
I was able to record a squeezed 1856x704 10bit @50p for about 1000 frames , after unsqueezed I get 1856x1172
then crop off the bottom 266 lines to get 1856x906 .Very smooth frame rate , lots of detail , can't wait to try  some slow-mo shots 



Here the RAW-1880x705.DNG from the image dump ,  squeezed (1880x704)


Un-squeezed to 1880x906


here a short 56Mb I exported from MLVProducer Un-Squeezed with 3x5 option M03-0028_1856x906_50p_h264.mov





exported frame from mlvproducer un-squeezed & cropped


Now to get Full width working properly , that's my next challenge .

[ch_d]

OMG! ML team - You are the best!

[reddeercity]

This is a continuation of post #278 -- Seeing how far I push the frame rate , got to 71.737fps before it froze/Locked up , had to pull battery.
I was looking for 75fps , I thinks it there just need to play around more , even thinking I can push it to 100 fps   

[Igor_Braun]

Really? 70FPS on 5d mk II? Guys i'm following you here on this topic but never write. So this my first time. You are awesome.

[CITY-U1001]

reddeercity , show test video in slow mo

[reddeercity]

Will it looks like I made a mistake on the 50p experiment from a1ex & I'm happy I did !
After going thought it again , I made a error by not refreshing Liveview after setting CMOS[1]
So after the 3x5 unsqueezed  I get 1856x1172    not 1856x906 .
M05-2017_1856x1172_50p_h264.mov
Got a really clean image now but black level is off a little as per a1ex post .

Here the squeezed RAW-007_1880x705_50p.DNG from image dump


RAW-007_1880x705_50p_Unsqueezed_1880_1172_50p.png


Exported frame with some CC from MLVProducer


Liveview 422 image dump


Screen shot of Liveview


In short , This is As exciting As UHD (2880x1080) on the 5D2 or maybe better in some cases .
To sum it up we are within reach of stable 50p @ 1856x1127 (records @ 1856x705) that's better then the standard
1856x1044 @ 10 & 14bit 24fps , maybe it's time to look in to 3x1 & 1x3 binding to reduce aliasing & moire pattern .

[IDA_ML]

A nice step forward, Reddeercity!  Keep going!

For those of you who still question if it is worth the effort to use Magic Lantern in these old Digic 4 cameras, please watch this video:

https://www.youtube.com/watch?v=WTr2L3uCYts

Imagine, we also have 4K crop recording working ...

[goldenchild9to5]

@reddeercity That's unbelievable.. the 5D Mark 2 capable of doing slowmotion video WOW.  Great progress Team 

[reddeercity]

Dialed down the frame rate to 48.015 fps have a really clean image now ,


black levels are very close "1654-16200" , seems to clean up at "1730" in mlvproducer


8 second test clip [email protected] h264.mov


Frame from mlvproducer with good black levels @ 1730 with some CC ,
Looks almost perfect M06-2218_000011856x1172_48p.png

So I think 48fps is a better fit for the 5D2 (black levels are better mainly), one issue off the bat I notice I was locked to 1/333 on the shutter plus had a frozen Liveview.
Thou this could be because of ADTG_GUI taken up resource's slowing down liveview I guess .
The 2 frames different is not noticeable 50->48 and I have a better black level . More work needed of course , getting closer     

Edit: here's the Cdng from mlvfs M06-2218_000000_1856x704_48p.dng

Found something interesting about global shutter on CMOS sensors https://www.qimaging.com/ccdorscmos/triggering.php
Just wondering if I'm close to a global shutter with 48fps with 1/333 of a sec ?
I never thought it could be possible on a CMOS , just know that CCD could be global shutter , Thing you learn while researching 



from my file I'm at 3ms @ 48fps so theoretically 70-75fps is maybe a possibility or at least 60fps
Unless I'm miss understand this .

[Levas]

Interesting article about global shutter on CMOS.
But if I understand the stuff in the article right they're using external light source for global shutter on CMOS.
They're using a LED or Laser which can be switched on and of fast enough for controlling exposure.
CMOS is exposed, but about only half of the shutter time is used for exposure, the external light source kicks in every frame and exposing for a shorter time then shutter time is set.
So no global shutter for normal cinema situations unfortunately, more for like fabrication/quality control and stuff.

[reddeercity]

Good news I got near 100fps Bad the image is all noise distortion  but didn't crash !



I still looking for stable 60fps , we do have 48p & 50p with black levels issue that dose work .

Changed Timers

Code Select Expand

c0f06008 0x23b023b -> 0x23b011d   **this alone gave me 59.94** (looks familiar , never try just that alone)
c0f06014 0x577 ->   0x348


Liveview is mess , I did record a few MLV's

Code Select Expand


Image Width                     : 1856
Image Height                    : 704
Camera Model Name               : Canon EOS 5D Mark II
Strip Offsets                   : 65536
Rows Per Strip                  : 704
Strip Byte Counts               : 2613248
Planar Configuration            : Chunky
Software                        : MLVFS
Exposure Time                   : 1/250
F Number                        : 4.0
ISO                             : 800
Black Level                     : 1792
White Level                     : 16200
Default Crop Origin             : 0 0
Default Crop Size               : 1856 704
Calibration Illuminant 1        : D65
Active Area                     : 0 0 704 1856
Frame Rate                      : 99.781

Showing the right frame rate & black level (1792)
Ok , back to finding stable 60fps .

[ilia3101]

Incredible progress!

Also what are the next steps to getting 3k centered?

[Igor_Braun]

If you guys need help with one more 5d mark II for your tests, let me know

[reddeercity]

Incredible progress!

Thanks , a1ex figured it out originally now I'm hooked  on the hi-frame rate 

Also what are the next steps to getting 3k centered?

I think the c0f0713c & c0f07150 is not set right , from my understanding (which is very limited)
I believe this controls the window size in Liveview (I maybe wrong) but from a1ex experiment with full width
and height liveview those reg's where needed .
This should be addressed in future updates to the crop_rec.module
I what to add same different presets anyways .
Also we need to implement a1ex's new "CONFIG_EDMAC_RAW_SLURP/CONFIG_EDMAC_RAW_PATCH" into the 5d2 4k branch
that how we have 10-12bit full frame liveview builds on the experiments page.

I'm starting to come to the end of my experiments with Hi-Frames rate , I'm starting to see really clean images now (black level not as bad now)
I increased the vertical to 720p from 704 this give me after unsqueezed 1856x1200 , but it has better black levels -- there still are not right
10bit yields the best image so far, black level are 98 (should be around 112) image cleans up at black level 75
and 14bit is 1559 and cleans up at 1200 black level , before at 704 black level where all over the place and couldn't clean up .
changed

Code Select Expand

c0f06088 0x4f40432 ->0x2e40432 =>720p
Before it was 0x2d40432 => 704
FYI -- the default is 0x4f40432 =>1250x1856 , so take the first set of numbers "4f4" that's the vertical resolution
"432" the horizontal
hence 0x2e40432 =>720x1856 ,  we read this as 1856x720
I encourage other with d4 cam not just 5d2 to try out a1ex experiments , there are very simple & will help you understand
how the camera works or at least with different resolutions & frame rates
Edit: this was in 50fps , didn't try 48 yet

[reddeercity]

Found a module on the crop_rec 4k 5d2 branch that "cf_acc.mo , added it to the module list to complied
Did that ok now what is it ? is it for over clocking the CF card interface ?
I haven't loaded it yet just trying to fine some back ground info , looking at the code
seem to be able to bypass canon code and access the cylinder/blocks directly or something like that .
Can anybody enlighten me please 

[a1ex]

https://bitbucket.org/hudson/magic-lantern/branch/hdparm

Might be useful for overclocking, too.

[reddeercity]

Ok , loaded it up and it reads the CF Card info
Printed this on the screen


Some useful info at least , this is a Lexar 1066x 32GB
I see the card runs as a PCMCIA & the UDMA in Advance i/o --0x0003  , UDMA6 ? More then likely from the dm-spy logs

[reddeercity]

https://bitbucket.org/hudson/magic-lantern/branch/hdparm

Might be useful for overclocking, too.

Thanks a1ex , I'll have a look at this .

[reddeercity]

Might be useful for overclocking, too.

It going to take little time to get my head around "hdparm" branch for overclocking , a lot of stuff there .

Also got my new replacement CF card today (Lexar 1066x 64GB) the old one couldn't format just came up as "RAW"
So I did a fresh benchmark speed test in a few different modes to see where it at before I start to
adjust reg's to force it to UMDA 7 (I'm doing this on a new card because it I smoke it , it has warranty  )
1 min bench test , Movie Liveview , first run


Photo Liveview , better speed


Great Speed !! Play mode GD on


There is quite a bit of difference between those 3
So why is there so much overhead in Movie Liveview ?
Or is it just Liveview taken up that extra resources
Normally I record raw video at about 72-77MB/s on my other Lexar 1066x CF cards .
Even if we could get the write speed in play mode (87.4MB/s) for movie liveview , That's enough to record 2880x1080 continuously .

Need to do more investigation .

Edit: this was in 1080 30p fps , just checking raw video write speed --
1:1 averages 74MB/s & 3x cropmode 2144x1074 12bit  averages 75MB/s and pecks at 79MB/s

[reddeercity]

Working on increasing height in 3x Crop_mode  , I'm up to 2152x1331

Code Select Expand

Default Crop Size               : 2152 1331
Active Area                     : 52 160 1383 2312
Image Size                      : 2312x1383

RAW-002_2152x1331.DNG
The last 300 line at the bottom is because I haven't implemented a1ex's redirect buffer for larger image in the edmac capture
(that's coming very soon , along with a updated crop_rec with added presets up to full width )

Also got 3200x1200 working , did have good success  with 3168x1152


M13-2030_000000_3168x1152.dng

Did get 3168x1330 , still needs more work to get 24fps (at 21 fps right now)

[reddeercity]

Try to force CF card to UDMA7 on Lexar 1066x from this post here and here
The Reg is only accessible with "DIGIC POKE" , there no 0xC062850C in the ADTG_GUI

Code Select Expand

0xC062850C --->1000102
As waza57 said -- corrupt the file system , the file are hidden .
I'm in the process of recovering the files ( it's not a fast process that's for sure) didn't hurt the card so far
the card is still in FAT32 , and can read -- file explorer says the card is 64GB with 7 GB in space used up with data(files)
So before I format again and install magic lantern I what to recover the setting files .
before it lost the file system I took a photo with my iPhone , and there's a increase for sure with some interesting info


70.7MB/s write in the other test a few post up it was 57-63MB/s
but the read is crazy 2260.4MB/s 
So maybe that reg affects the "read only" ?
more investigation
Edit: Strange thing when it lost the file system it changed the name of the card to
"EOS_S_GIGIL" not sure why something with canon I guess

[waza57]

@ reddeercity

Great job with fps !

"EOS_S_GIGIL"

Same thing for me...