3K/UHD 5D2 Raw development and Other Digic IV Cams

[reddeercity]

@ reddeercity
Great job with fps !

Thanks , a1ex did all the heavy  lifting I'm trying get to fine tune it.
There still a black level problem (green cast) , I can't fix yet -- it happens when ADTG_12[100c] 0x2 -> 0x4  is applied
(It's the one that centers the image in liveview) . If you change to 0x3 or 0x5 the image is has blue cast
So on my quest to find reg's that would help I did find the reg for black level in liveview  , but sadly it only for liveview & not for raw image in the buffer

Code: [Select]

C0F08034 0x1c00 (default)
C0F08034 0x1c00 ->1b00 (green cast)
C0F08034 0x1c00 ->1d00 (red cast)
C0F08034 0x1c00

C0F08034 0x1b00

C0F08034 0x1d00


This what happen after ADTG_12[100c] 0x2->0x4 is applied 
but before I refresh liveview which will center the image



I think I may have kind of figured out the "EOS_S_GIGIL"
after recovering some of the data with diskinternals.com/partition-recovery
the CF card came up as "EOS_DEVELOPFAT32"  , I made a disk image of the card and loaded it up in "HxD" hex editor and the beginning
of the image I have this

Code: [Select]

ëX.CanonEOS..@ ......ø..?...?...±9t.Ÿ;..........................€.)
....EOS_DEVELOPFAT32   ..BOOTDISK.FA.........................................
ëX.CanonEOS..@ ......ø..?...?...±9t.Ÿ;..........................€.)....EOS_DIGITALFAT32   ......I'm thinking by setting

Code: [Select]

0xC062850C --->1000102 We may have accessed a canon service mode or canon development mode for maybe internal testing .
Al this only happen when I tried to record raw video , I first ran the bench.mo and it seem ok at the point.



@a1ex I'm trying to compile "hdparm" branch , getting a fatal error .
I first  tried to complied the 5d2 platform on my new VM " Up to date Ubuntu"
didn't work  , it was looking for a older tool chain "gcc-arm-nne-eabi-4-8-2013q4"
Tried to install but I don't know how to set it in the path for compiling , lucky I still have the Old Clone of
Ubuntu with all the old  tool chain pre-installed . So I complied it and all worked except  the cf_acc module fail to build
So I tried  to compile the module separately and got the following 

Code: [Select]

ml@ml-pc:~/CF_Card-Over-Clocking_magic-lantern/modules/cf_acc$ make
Updated HGVERSION
[ README   ]   module_strings.h
[ CC       ]   cf_acc.o
[ CC       ]   hdparm/identify.o
hdparm//identify.c:7:25: fatal error: linux/types.h: No such file or directory
 #include <linux/types.h>
                         ^
compilation terminated.
make: *** [hdparm/identify.o] Error 1
ml@ml-pc:~/CF_Card-Over-Clocking_magic-lantern/modules/cf_acc$

Seems to look for this "linux/types.h" and from what I can tell there is nothing related to "Linux" anywhere
Is there something missing ? do I need to add something or should I just comment it out ?
As a test , I tried to compile on the 5D3 113 platform -- still the same error .

I notice this is on the old core , still has raw_rec.mo .
 

[reddeercity]

I knew it !
There a Service menu I found in the rom
I don't know how to access it but maybe that "EOS_DEVELOPFAT32" did something to the card .
Something along the lines like magic lantern does when it boots

Code: [Select]

"********** FACTORY ADJUSTMENT MENU VER 0.01 **********":
"0. Exit from Factory Adjustment":
"1. Leak Check":
"2. SDRAM Check":
"3. ROM Check":
"4. SDIO Implementation Check":
"5. Video Adjustment":
"6. A Adjustment":
"7. USB Check":
"8. Adjustment Data Display and Change":
"9. Check Flag Display and Initialization":
"A. ALL Check":
"D. Debug Command":
"R. Data Receive":
"Y. Input Unique":
"Z. Input_Device_Unique":
"invalid input":
"CheckPro> ":
"--- LeakCheck ---":
"--- Finish ---":
"K218 System & Display Check & Adjustment program has started.":


[Igor_Braun]

I shot my cat with your build. She liked 2880:1080 so much. Everything was ok but froze grayscale preview.
https://drive.google.com/file/d/1YiBTSvS1JYzhThcNcmEZbBaBjnLibsjX/view?usp=sharing
Reddeercity, you told that you finally objective here is 4096:1550.
Look, 2.64:1 - it's a very special cinematic aspect ratio.
Do you plan to bring some options to choose between 2.64:1, 2.35:1, 2:1 or maybe 16:9?
I think 2.35:1 is the most optimal option here. It could be 3642:1550 (2.35:1) for example.

[reddeercity]

............. Everything was ok but froze grayscale preview.

right now that one of the limitation , press half shutter before recording will give a full color preview in liveview

Do you plan to bring some options to choose between 2.64:1, 2.35:1, 2:1 or maybe 16:9?

Yes , but the choices will be limited to CF card write speed . 3k to maybe 3.5k (currently at 3168x1330 @ 21 fps)
no sense trying to record 4k @ 24p for 2 seconds .
I can implement full width @ 12.5 fps (5632x1074) and full height @ 4 fps (5632x3752)
Which I plan on doing later on (in a few week or so)
I'm really focused  right now on CF card overclocking (increasing write speed beyond 80MB's)

[IDA_ML]

I can implement full width @ 12.5 fps (5632x1074) and full height @ 4 fps (5632x3752)
I'm really focused  right now on CF card overclocking (increasing write speed beyond 80MB's)

Reddeercity,

If you could implement 5632x3752@4 fps that would be fantastic!  This means very smooth timelapse videos at 1 to 4 fps which really look very beautiful, especially in combination with Dual ISO. 

What about the 7D?  It writes at 90 MB/s as it is.  No overclocking would be necessary to get reasonably long recording times at the high croped resolutions.  2560x1440 at 10bits lossless and 24 fps would be a dream come through for this fantastic old camera.

[reddeercity]

Calling all 5D2 users -- 
magiclantern-crop_rec-4k-5D2-eXperimental.2018Oct22.5D2212.zip

Updated File
magiclantern-crop_rec-3k-1080p-5D2-eXperimental.2019Feb01.5D2212.zip

I implemented  a1ex's redirect buffer  that's used in the official 10-12bit "raw_slurp" in the  https://builds.magiclantern.fm/experiments.html for 5D2.
Work very well with 10bit 1:1 FHD 1856x1044 @ 30fps & 24fps with out issue with mlv_rec  (did a short 30 second test at both frame rates).

More important , it works great with the "Crop_Rec" module , no problem recording 2880x1080@24fps
This was the first step to bring 5D2 in to the main "crop_rec 4k branch"

There's a little interesting side benefit from using the new redirect buffer from a1ex ,
I can get 3008x1138 with the 2880x1080 crop_rec preset , just need to set aspect ratio to 1:1
go to 10x zoom then back to 5x zoom (3x crop_mode) basically refresh liveview
that resolution is a little too unstable , but it proves that the buffer is working correctly
Now I can push forward with some different preset .

So now you have stable 10-12bit in 1:1 Full HD with 3k crop_rec , more usable now .
To come full hd 48fps or maybe 50 & Full resolution 5.6k @4 fps  to start with.
Enjoy

@IDA_ML a developer with 7d really needs to implement the code ,
I don't have one so it would be nearly  impossible for me .
Sorry .

[Danne]

Do you have your code uploaded? Would like to check it out.

[reddeercity]

Do you have your code uploaded? Would like to check it out.

Here's the source code Crop_Rec-4k-5d2-magic-lantern_redercit_buffer_a1ex-code-10-22-2018.zip
The changes are in

Code: [Select]

platform/5D2.212/internals.h
platform/5D2.212/stubs.S
src/raw.c
Edit: I edited the first post in thread with the latest build ,
I'll update it with information & new build when necessary
This will help make it easy to find the new builds for 3k,4k crop_rec

[Danne]

Thanks. By the way. Could you tell me the ISO ADTG registers for the 5D mark 2?
ADTG2[8882]
ADTG2[8884]
ADTG2[8886]
ADTG2[8888]

[masc]

@reddeercity: thanks for your build! I played a bit around with it. 10 and 12 bit seems to work very stable for me. I got no issues so far, @25fps (PAL) and 30fps (NTSC), FHD and 2144x1074 working nicely. The only thing I did not get to work is 3k. What I tried:
- activate mlv_rec, mlv_snd, crop_rec
- restart camera
- start liveview
- goto 5x zoom
- select resolution 2880
- select aspect ratio 1:1
- leave ml menu
- goto 10x zoom
- goto no zoom
- goto 5x zoom
If I now record, I get 2144x1074. Or if I go into ml menu I see the same resolution.
What am I doing wrong? I tried with Canon menu PAL (25fps) and NTSC (30fps).

Edit: forget what I wrote... so stupid... never worked with crop_rec... I found it now!
Edit2: oh yeah... after getting the right settings to work, the quality of the resulting clips are brilliant at 2880x1080!!! Great work!

[reddeercity]

Thanks, as always I'm only a part or a bigger picture 
Most of the credit for the crop_rec goes to waza57 for porting it to 5d2 .

[reddeercity]

Now that I have the redirect buffer working I can test some full width in 3x crop_mode  from a1ex's experiments .
I modify the resolutions in mlv_rec , added 3008, 3840, 4096, 5632 (H)
I successfully reproduced the full width 5632x1074 @ 12.5fps , can record continuously in 10bit

Code: [Select]

For full width LiveView:
CMOS[2] = 0x00E
C0F06008: 0x27705DB (only the lowest half appears to matter on this camera)
C0F06084: 0x30037 (copied lowest half from photo mode)
C0F06088: 0x46A0B87 (copied lowest half from photo mode)
=> 5632x1074 in mlv_rec, 12.5 FPS.


5632x1074-12.5fps-M23-2045.mov 40Mb , Short 12 second H264 test

selected 4096 from menu to record 4k
4096x1074-12.5fps-M23-2047.mov 30Mb , 8 second H264 test

@Danne there's no

Code: [Select]

ADTG2[8882]
ADTG2[8884]
ADTG2[8886]
ADTG2[8888]on the 5D2 , those reg's are for Digic 5 cams only

[reddeercity]

Next to get full height  as per a1ex experiment , in 3x crop_mode -- 2144x1330 @ 9.98fps (hopefully I can increase frame rate)

Code: [Select]

C0F06014: 0xEDC
C0F06088: 0x0x56A04BA
CMOS[1] => F6A
C0F0713C from 0x476 -> 0x576
C0F07150 from 0x49C -> 0x59C
9.98fps

2144x1330-9.9fps-M23-2259.mov 40Mb H264 clip
2144x1330-M23-2259_00001.png PNG frame export from mlvproducer

[reddeercity]

Continuing from yesterday with full height exercise ,

2144x1586

2144x1586-9.9fps-M24-2225.mov 30Mb h264 clip
2144x1586-M24-2225_00001.png Frame exported from mlvproducer

Code: [Select]

C0F06014: 0xEDC
C0F06088: 0x0x66A04BA
C0F0713C from 0x476 -> 0x676
C0F07150 from 0x49C -> 0x69C

2144x1842

2144x1842-9.9fps-M24-2227.mov 30Mb h264 clip
2144x1842-M24-2227_00001.png Frame exported from mlvproducer

Code: [Select]

C0F06014: 0xEDC
C0F06088: 0x0x76A04BA
C0F0713C from 0x476 -> 0x776
C0F07150 from 0x49C -> 0x79C


[benoit]

@reddeercity ->  Calling all 5D2 users -- 
Congratulations, the first step in crop_rec is here !
I tested a 10bits 2752x1080 file and it is ok but liveview is frozen during recording.
I tested a dual_iso_crop_rec mixed video but i got an "oldapi" error message on loading dualiso module. I know I'm asking a little too much for a first step ...
But it's was so existing that I tested !
Thanks

[masc]

I tested a 10bits 2752x1080 file and it is ok but liveview is frozen during recording.

Japp same here. But somewhere I read "liveview is for wimps", so I thought this is state of the art...

[benoit]

Perhaps I'm a wimps ...    I want to see what i'm filming  ! I'm not only use my cam with a tripod  !

[dariSSight]

I must be doing something very wrong, so I'll take the hit. Please breakdown what our setting should be, I only see static lines when I try to use 3.5K 1.1 Crop setting.

[reddeercity]

Updated the First Post with very basic screen shot tutorial
Short Tutrial for Setting up Crop_Rec Module (3.5k 1:1 Preset) on the 5D Mark ii
Hopefully this should clear up any confusion

[dariSSight]

I don't know what I'm doing wong but I can't wait to find out, I've follow all tutorial steps. I attached photos to represent setting, also showing that in 1856x1044 1:1 and 10x zoom everything looks OK, but x5 3.5K 2880x1080 is all static lines. PLEASE HELP ME SOLVE THIS!

 

[reddeercity]

You don't follow the tutorial , I don't know how to make it any simpler .
This is bleeding edge stuff , Go back to the tutorial and match your screen shots to the
ones in the first post , Do there match exactly in the Movie Tab  ?
Hint: something to do with fps

Code: [Select]

1:For best results start with 30 fps ntsc in the canon Liveview menu
2:Load crop_rec  & mlv_rec , mlv_snd can be load but reduces the amount of frames recorded .
3:Next setup Liveview preview , This is the most important part without this set correctly it will not work !
 -Liveveiw needs to be set to ML Gray Scale-
4:Next go in to the crop_rec menu select 3.5k preset .
 -Note ! this only work when you enter 5x zoom/3x crop_mode as per the hi-lighted texts says-
Did you do this exactly ?
The frame shots tell a different story .
Only load & enable the bear minimum to run crop_rec

[dariSSight]

Wow, @Reddeercity you’re affirmative I use both FPS setting, Thanks for the smack on the back of my head. So is it necessary to use Exact FPS on 10bit 12bit? When using 5x is it recommended to use 24mm to get a 50mm framing? Thanks Again to Reddeercity, Alex, Waza57 for the phenomenal job of making my trusty 5D Mark II at least a FHD RAW recorder, with full frame sensor RAW our images will meets up to wide range of end Presentations Format.

[reddeercity]

Working on presets , Got a nice clean 3168x1330 but sadly only 12fps at the moment (looking to get 23.976 fps)
I need to understand the A & B Timers better , this was just luck -- started at 7fps and worked my way up to 12 fps

3168x1330 @ 12fps


3168x1330_12fps-M29-2249.mov h264 10 second 30Mb clip
3168x1330-12fps-frame-M29-2249_00001.png
3168x1330-RAW-017.DNG from image dump

Code: [Select]

Default Crop Size               : 3176 1331
Active Area                     : 52 160 1383 3336
These are the overridden reg's ** Note** the times (6008 & 6014) are for 7fps I didn't save the timers for 12 fps



I did find that "CMOS[1] need to set early -- if not , thing don't work right

and you get this 2.38 A.R. ( that what I'm looking for)

[reddeercity]

working on a 4k preset (4096)

Code: [Select]

c0f06088 - 46a088a =>4096x1074
c0f06008 - 2770477 => 16.389 fps
Clean dng from image_dump plus clean dng's in .mlv's

4104x1075-RAW-019.DNG from image_dump


After cropping off the  black space it's only 3718x1074 , needs more reg's added

3718x1075-RAW-019.png

[reddeercity]

Ok got 4096x1074 @ 18fps working -- It's almost useable as is 

Needed to add CMOS[2] 0x10E ->0xE (same as full width 5632)


A & B Timers setting for 18 fps


For same reasons mlv_rec in crop_mode cut off the 48 pixels from 4096 to 4048 , not sure why .


4048x1074-18fps-M02-2240_00001.png

If you look at the dng from the image _dump it's 4104x1074 and there no black space on the right side like the PNG .
4104x1075-18fps-RAW-020.DNG
4104x1074_image_dump-RAW-020.png used dcraw to export a ppm them saved it to png with Irfan View

4096x1074-M02-2240_000000.dng Cdng exported from MLVFS (quick mount)
there a 48pixel black area on the right , thinking the problem is mlv_rec  , I see in the code there's a 160 pixel crop , so may be change that to 100 maybe

Code: [Select]

Subfile Type                    : Full-resolution Image
Image Width                     : 4264
Image Height                    : 1127
Bits Per Sample                 : 14
       ..........
Default Crop Size               : 4104 1075
Active Area                     : 52 160 1127 4264


Code: [Select]

Black Level                     : 1001
White Level                     : 16200This is also interesting , can the black level really be 1001 ?
Image looks great , so I guess this is ok , but strange -- I recorded  at 10bit .

Recorded a short .mlv about 10 seconds exported it to h264 from mlvproduder 4096x1074-18fps-M02-2240.mov

So close now to 24 just need 6 more frames per second .

Now it time to work on Lossless compression , since I'm close to having the presets done.
So far I have ,  full liveview , 2880x1080, 3168x1330, 5632x1074 ,4096x1074
I'm thinking of added some of the near full height in 2k mode e.g. 2144x1842 or 2144x1586.
I would like to add 48 FPS in FHD (1856x720) but it's too unstable right now , need a lot of work yet.