3K/UHD 5D2 Raw development and Other Digic IV Cams

[ItsMeLenny]

96 isn't possible on any camera https://docs.google.com/spreadsheets/d/16cgnRivbUv7nA9PUlCLmLdir3gXdIN3pqzCNAAybepc/edit#gid=5
in digic IV. Unless anybody's tried overclocking the SD card speed.
Probably wonldn't need to if RAW compressed (or lossless whatever stupid name people were calling it) could work.

[ilia3101]

96 isn't possible on any camera https://docs.google.com/spreadsheets/d/16cgnRivbUv7nA9PUlCLmLdir3gXdIN3pqzCNAAybepc/edit#gid=5
in digic IV. Unless anybody's tried overclocking the SD card speed.
Probably wonldn't need to if RAW compressed (or lossless whatever stupid name people were calling it) could work.

This isn't MB/s card speed, it's sensor speed- Megapixels per second. I was just coming up with some theoretically possible resolutions/framerates that I want to try and get on the 5D2 in crop mode. They wouldn't record continuously of course, about 6 seconds in 10 bit (but it is 60fps, so a 14 second long shot once slowed down to 24)

There's also a chance that none of this is possible at all in crop mode.

[reddeercity]

I'm looking at keeping 4k (4096x590=>4096x1770 23.976fps) or as I call it
Vertical compressed , don't like the name 1x3 or 3x1 or what ever it's called (less confusing)
I did a short 4 min talking head clip (me talking about 4k on the 5d2  ) Yea that continuous @69MB/s write speed @ 10bit uncompressed
I found it to be not as bad as I thought , yes there a very small amount of aliasing
but with some heavy grading and anti-aliasing filters is more then useable
thou you must select your shots wisely to avoid this little issue .(fine patterns etc. .. you know the drill)
Here a frame of the close to final grade , still needs a little tweaking .


5D Mark ii Real 4K.tiff

Workflow:
MLV App 1.4 wins , to exact dngs to full height (590->1770)
Imported the dng's in to Adobe CS6 A.E. ACR set project to 32bit float
(I tried mlv app first , but I can't get the image to look right the skin tone where really bad & had had too much noise in the shadow ,
sadly I when back to A.E. so slow 6Hrs to generate 6300 4k tiff's  compared to 90 mins on mlv app maybe it better on the MAC never tried).
Imported the tiffs in to Apple compressor , made a real ProRes4444XQ (Apple ProRes is much better , 16bit+alpha)
The real ProRes4444XQ can be real pushed hard for grading , take a lot to break the image .
Imported in to FCPX edit & final grade , added my LUT there , "ArrilInterneg_KodakVision_sRGB.cube .

So do the other 5D2 users think I should keep 4K preset in compressed vertical ?
The sample above should be noted that I was a little out of focus (hard to do by yourself  )

FYI: 4mins of 4096x590@ 23.976fps = 17.7GB (mlv_rec file) that got blown up to 350GB of tiffs = 40GB ProRes4444XQ
So at 4096x1770 , 1 dng = 9MB , 1 Tiff = 58MB x6300 frames
Cam settings 400 ISO 1/50th 180-degree shutter , Canon Ef24-70 f2.8L @ 24mm f4.0

[masc]

So do the other 5D2 users think I should keep 4K preset in compressed vertical ?

Any of your posted presets would be nice to try out for me. But unfortunately I don't get one single preset to work over here...

[ilia3101]

So do the other 5D2 users think I should keep 4K preset in compressed vertical ?

First impression when I opened it: wow this looks sharp, second impression: Oh there's a lot of alising (even on the out of focus monitor).

I still think the 1:1 presets look better, though you could keep this one for a wider field of view. Also a softer lens might reduce aliasing.

[benoit]

As presets need to be frequently updated, I like the Ilia idea of creating a struct to store different presets.
If it's possible to do it, it's more suitable, it's open, it's simple (no need to compile every time).
Indeed, we can exchange an xml file (or whatever file format)  that store 5D2 presets between users.
IMHO, I think 3K/UHD raw developpement and registers discover will be more easy and quickly spread over users.

David

[ilia3101]

As presets need to be frequently updated, I like the Ilia idea of creating a struct to store different presets.
If it's possible to do it, it's more suitable, it's open, it's simple (no need to compile every time).
Indeed, we can exchange an xml file (or whatever file format)  that store 5D2 presets between users.
IMHO, I think 3K/UHD raw developpement and registers discover will be more easy and quickly spread over users.

David

Well with my idea (C structs in the source code) you'd still need to compile it each time, but it would still be massively easier than what it is now (loads of random bits spread all over the code where you need to add stuff).

And your idea of having it in a file is amazing if it could come true, but it seems unlikely to me that anyone able to do that will find time (and for my idea). It also requires a format simpler than xml, maybe csv, or even a binary format that would need to be edited with a hex editor.

I tried recreating reddeercity presets with adtg_gui (crop rec disabled), no success yet, can't find some of the registers 🙁
@reddeercity Also are the adtg_gui screeenshots showing all the registers you changed?

[reddeercity]

@Ilia3101 , Your right too much aliasing , check out the sample below  -- better ?
Yes when I post Reg's screen shots it all the reg's  needed for the what ever preset I was working on.
No sure why you can't reproduce , 5D2 is really touchy , if you adjust reg's out order the cam will lock up or will not respond .
You have to enable (in the advanced tab) ENGIO registers , then refresh , view or play a h264 or cr2 file to update the reg's
Rule of thumb : First thing Slow down Timer"A" first then Timer"B" , I usually do all my work around 10fps then after I set raw rez.
and head timers I increase timers "B" to the correct frame rate (23.976) (but slowly) .
Yes "C" code can be hard to understand sometimes , but there is a reason for everything 
If you read the crop_rec code you will see that it  needs calls from adtg to apply the modified regs.


Ok I'm very close to getting 3xcrop (5xzoom) to work with Lineskipping 
I had to start in 1:1(3x3 FHD) set the timers , resolution & cmos[2] reg
Then switch to 3x crop_mode (5x zoom) set to adtg12[100c] to line skipping (0x5->0x6)
Adjust frame blanking adtg1[106] 0x301->0x1 lastly cmos[5]
Seem to be cleaner , haven't notice any aliasing yet but I'm still adjusting
here a sample to pixel peep


3x_crop_mode_Line-Skipping_4096x1776_M19-2238_frame_47.png

[banertop]

yep!

definitely looks much better!!

[ilia3101]

does look better

[ilia3101]

Yes when I post Reg's screen shots it all the reg's  needed for the what ever preset I was working on.

good

No sure why you can't reproduce , 5D2 is really touchy , if you adjust reg's out order the cam will lock up or will not respond .
You have to enable (in the advanced tab) ENGIO registers , then refresh , view or play a h264 or cr2 file to update the reg's
Rule of thumb : First thing Slow down Timer"A" first then Timer"B" , I usually do all my work around 10fps then after I set raw rez.
and head timers I increase timers "B" to the correct frame rate (23.976) (but slowly) .

Thank you for this, really useful! I had enabled ENGIO, but had no idea about the playing a file trick. Also had no idea about the order of setting them. I think this will help a lot next time I try.

Yes "C" code can be hard to understand sometimes , but there is a reason for everything 
If you read the crop_rec code you will see that it  needs calls from adtg to apply the modified regs.

I still think it could be modified to make those calls based on some data in a simple to understand struct. I would like to do this if I ever get an understanding of how crop rec presets work right now.

[Danne]

Well one way to proceed is to create actual presets in crop_rec.c. From there one can alter registers from within a sub menu structure. Then include all relevant registers and connect them to the presets so it´s possible to modify anything but from a starting point where all registrys are already set.

[reddeercity]

Ok I got crop_rec to work with my new preset 2.4k (2400x1330 @ 23.976)
But I do have a problem , waza57 added a rom hack to increase height in 5x Zoom
before a1ex implemented "Raw_Slurp" Redirect Buffer for extended resolution in 5D2/D4 Cam .
As a result my preset is  2368x1266 @24.028fps 

Code: [Select]

Active Area     : 0 0 1266 2368
Frame Rate      : 24.028
So I'm currently going thought the crop_rec code line by line to figure out how to disable it/remove it
as it's no longer needed as a result the 2 hacks are fighting each other causing conflict

@waza57 is there a quick way to disable the rom hack you added ?

So once that rom hack is disable , I can implement all my preset including
4K line skipping in 5x zoom , got that to work now @ 23.976fps (4096x590=>4096x1770)



We are almost thought the forest now and I can see a clearing with rolling grasslands to play in 

[reddeercity]

Boy this ROM hack from waza57 for height is really hard to disable , more I think I'm close the more error's I get  & can't compile 
On the Bright side  My preset code structure works , after I found where to add the code.

2400x1330 23.976fps +unfrozen liveview

Code: [Select]

line_592-598 
                  if (is_5D2)
                {
                cmos_new[1] = 0xB0D;    /* pink highlights without this */
                cmos_new[2] = 0x10E;    /* read every column, centered crop */     //waza57 cmos 6
                break;
                }
                break;   
    

Code: [Select]

****Line_685-693****
/* raw buffer centered in zoom mode */
            case CROP_PRESET_CENTER_Z:
                cmos_new[1] = (!is_5D2)
                  ? 0xB0D          /* vertical (first|last) */
                  : -1;
                cmos_new[2] = (!is_5D2)
                ? 0x10E            /* horizontal offset (mask 0xFF0) */
                : -1;
                break;


Code: [Select]

****Line_1508-1544***
static inline uint32_t reg_override_zoom_fps(uint32_t reg, uint32_t old_val)
{
    /* attempt to reconfigure the x5 zoom at the FPS selected in Canon menu */
    if (is_5D2)
    {
     int timerA = 680 ;         
     int timerB = 1472 ;     
     int a = reg_override_fps_nocheck(reg, timerA, timerB, old_val);
        if (a) return a;
           switch (reg)
            {
                /* raw resolution (end line/column) */
                /* X: (3072+140)/8 + 0x17, adjusted for 3072 in raw_rec */
               
                 case 0xC0F06084:
                    return (old_val & 0x0000000)  ;
               
                case 0xC0F06088:
                    //return (old_val & 0xFFFF0000) + 0x630 ;
                    return (old_val & 0x00000000) + 0x56A053A;   //0x4b00648 or origin value chang nothing

                case 0xC0F0713C:
                    return (old_val & 0x0000000)  + 0x566;

                case 0xC0F07150:
                    return (old_val & 0x0000000) + 0x58C;
               
                case 0xC0F08184:
                    return (old_val & 0x0000000) + 0x56A;

                case 0xC0F08188:
                    return (old_val & 0x0000000) + 0xA07;

               case 0xC0F08518:
                    return (old_val & 0x0000000) + 0x56A053A;

               }

Plus another Good thing I discovered tonight , after testing my Reg's with ADTG_gui to make sure I have all of them right
I thought I would test HDMI to my Zacuto EVF and see if it works & Yes! I can preview in real time over the HDMI
while recording raw @ 2400x1330 23.976fps 
Even pass it thought to my Ninja HDMI recorder , thou the image is gray scale when recorded but Half Shutter press give
a color preview . It's still center crop but at least you can have finer focus level special if you have diopter as I do for the EVF
Still haven't figured out how to scale it to LCD(720x480) or HDMI(1920x1080) yet .

I'll start fresh tomorrow on the crop_rec.c , hopefully I'll has a better luck .


Edit: I see there maybe a quick way to disable the ROM hack

edmac-memcpy.c , fio-ml.c
          I need to disable call to function "clean_d_cache()" otherwise i can't patch ROM addresses
 

[70MM13]

I really hope you get a 5D4...  Your persistence and determination are badly needed for that unfortunate unappreciated BEAST!

[reddeercity]

Ok that didn't work , there more that's needed to override this going to have to take a drive deep .
It may be more then just Crop_Rec.

[reddeercity]

Found the ROM_Hack from waza57 , now I open up a can of worms 
just freezes in x5 zoom with crop_rec enable , before at least I had a image , thou it was all messed up.
going to have to back track again , this maybe over my head , I thought by disabling the hack I could get this to work.
not too sure now .

[aprofiti]

Boy this ROM hack from waza57 for height is really hard to disable , more I think I'm close the more error's I get  & can't compile 
On the Bright side  My preset code structure works , after I found where to add the code.

I was wondering if isn't better to make Digic 4 camera build on top of official crop_rec_4k branch? This is something should be done before or after.

At the moment I can remember @dfort for lossless compression here and a new one here , which is a fork of crop_rec_4k if I'm not wrong; then there is waza57 repository, which I'm not of what is capable of doing and I imagine that @reddeercity is working on top of that because is the one which compile, is it right?

Can someone make me an overview of the difference on those repository and what is needed to make d4 compile on top of crop_rec_4k?

[Igor_Braun]

Hi, Reddeercity, this is about zoom jumping out (the problem of my camera I described)
https://drive.google.com/file/d/1aa8kzCQueoRzHM1PoiVRr0jjbkSrG8Za/view?usp=sharing

[reddeercity]

Success !! disabled the "Rom Hack" and I can use the "raw_slurp" redirect buffer for extended rez.

[Danne]

Adding above gives this when compiling:

Code: [Select]

           if (!patch_active)   waza57 changes to effective update for 5d2
                                       ^
crop_rec.c:1660:71: error: invalid suffix "d2" on integer constant
           if (!patch_active)   waza57 changes to effective update for 5d2
                                                                       ^
crop_rec.c: In function 'crop_rec_init':
crop_rec.c:2313:9: error: 'RAWROMHEIGHT_WRITE' undeclared (first use in this function)
         RAWROMHEIGHT_WRITE = 0xFFCAA1F4;
         ^
crop_rec.c:2314:9: error: 'MEM_RAWROMHEIGHT' undeclared (first use in this function)
         MEM_RAWROMHEIGHT = 0x00000467;
         ^
crop_rec.c:2316:9: error: 'RAWROMWIDTH_WRITE' undeclared (first use in this function)
         RAWROMWIDTH_WRITE = 0xFFCAA1EC;   // waza57 patch ROM for more height
         ^
crop_rec.c:2317:9: error: 'MEM_RAWROMWIDTH' undeclared (first use in this function)
         MEM_RAWROMWIDTH = 0x00000FCE;
Did you also disable this?

Code: [Select]

       // RAWROMHEIGHT_WRITE = 0xFFCAA1F4;
       // MEM_RAWROMHEIGHT = 0x00000467;
       
       // RAWROMWIDTH_WRITE = 0xFFCAA1EC;   // waza57 patch ROM for more height
       // MEM_RAWROMWIDTH = 0x00000FCE;
Please upload a working crop_rec.c version that you know compiles.

[reddeercity]

Ok Finally got Crop_Rec working correctly on 5D2 with "raw_slurp" redirect buffer !
I started fresh with a simple  3k @ 23.98fps preset (3008x1074 @ 23.98fps)


magiclantern-crop_rec-3k-5D2-eXperimental.2019Jan28.5D2212.zip
Updated file
magiclantern-crop_rec-3k-5D2-eXperimental.2019Jan31.5D2212.zip
Updated File
magiclantern-crop_rec-3k-1080p-5D2-eXperimental.2019Feb01.5D2212.zip

Unfrozen Liveview while recording raw video 

I was having problem with c0f0713c & c0f07150 , as I have this all wrong in the code
and figured out that if you reduce 7150 more then 713c
you actually reduce over head for more frame FPS , that's how I squeezed out 23.98 @ 3008x1074 ,
if you leave those reg's alone the best you can do is 18fps at the same resolution .

The other big problem I have (haven't solved yet) is I need different "CMOS[1]" vertical offsets
setting then the one in "crop_rec" seems it just call's the one vertical offset in default 5x Zoom mode (0xE6A)
where in fact I need to have at least 3-4 different vertical offset (one per preset)
Unless there's something I missed , I just worked around the default CMOS[1] offset
and adjust from there . That why I haven't increase the vertical from default.

So this is the first test build for everyone to try , You only need to load

Code: [Select]

5D2_212.sym
crop_rec.mo
mlv_rec.mo
Same as before , canon menu 30fps -- enable 3.5k preset then enter in to
5x zoom , should take a few seconds before it comes up , half shutter will give a full color preview.
when you start recording , there should be a short 1-2 second delay then the preview will switch to real
time (canon preview) in b/w , as noted in my other posts it a center crop , but it is in real time 

Do Not ! enable "frame override" as this will lockup/freeze cam .
FYI - there's no mlv_lite.mo right now ,I need to add lossless to it before it can compile .
Test it as normal , let me know what breaks if anything . I'll add more preset after I figure out how
to code CMOS[1] offset's correctly .

@danne not sure where you when wrong , I checked the code I posted and it looks right
but I check it againist the source . when I have every thing done I'll post my code , it far too messy right now.

[ilia3101]

Incredible work @reddeercity!!! Can't wait to try it out.

3008x1074x24 is only 77Mpix/s, so there should still be plenty of  room for height increases if the overhead can be reduced more.

[dariSSight]

Thank for Anything that you give us RedDeerCity but What's the best 1:1 Resolution? I do notice if I'm going to use x5 then to get the equlavate of a 50mm I need to use 24-35mm right?

[masc]

Thank you @reddeercity! I tried it out:
so far it is working over here. Mostly it is working with superb picture quality. In one test clip the first 2 pictures were corrupted and after some seconds maybe 10 frames came in as "corrupted bundle". Other test clips were 100% fantastic.
This might not be of interest: I can't quit the crop mode by using zoom button. I have to use menu button. But I think, to make this safe was not your goal (no real problem so far)
Keep up the great work!