3K/UHD 5D2 Raw development and Other Digic IV Cams

[a1ex]

@a1ex , the math for "B"timer works 1640(base) +75 = 1715 , but for "A" timer 610 seem to be unchanged even though I have (-2018)

That happens when it hits the lower limit for timer A, as declared in fps-engio.c. That is, the value sent to hardware won't be below the declared limit, no matter what you dial in the UI.

To be able to dial a lower value, edit that limit (FPS_TIMER_A_MIN).

If timer A is too small, the side effects begin to be visible on the right side (usually noise instead of image data). If you keep reducing it, the image will break. The ideal value is the smallest value that still gives clean image (with correct image data on the right border).

I think at 10x zoom is really  1:1 pixel

So is the x5 zoom. The only difference between these modes is how the preview is rendered. In both x5 and x10 modes, the captured raw image is identical. So, I see no reason why anything would work better in x10, rather than in x5.

For the UI, I prefer to leave the x10 mode untouched, for focusing.

[reddeercity]

Will a little good & bad news , first the good -- I figured out what reg's & timers I need for extended resolutions as seen in the previous post's with the Liveview_dumps I've posted .
Now the bad -- the problem I'm having is I'm lock in to the frame buffer (LiveviewBuffer) of either 1880x1248 or 2144x1076 (3x crop_mode) I can't extent the buffer ,
so I'm getting a doubling of the same frame buffer as seen in 4096x1248 here & here . I still can't extend the 3x crop window to 3k like @waza57 did (If you are out there waza57 please can you comment on your experiment ) But I have been able to extent 1:1 (FHD) to 2144x1076 from 1880x1248 . Thou it uses the frame resolutions of 2.1k it's still has the 1880x1248 frame buffer , it just overlaps part the same frame buffer to fill the frame resolution , I guess that has something to do with the edmac .

Liveview_dump 422 , frame res. is 2144 (V) in 1:1 FHD and is filled with the 1856 (V) Liveview buffer , see the black bar then part of the next same frame


I guess I I'm stuck now at this point , been going around in circles and unless I can figure out the liveview buffer (This may be beyond my ability  )
a1ex is there any hint for this ? I'm lost on the Liveview buffer , or I'm I missing a step . I would think I should be able to save single dng even in extended res.

For the UI, I prefer to leave the x10 mode untouched, for focusing.

Ok , just looking to see if there is any usable information that can be used

[reddeercity]

Switch back to "ISO_Research_Branch" getting much better results , thou the buffer is still a problem.
Finding "CMOS[0]" can change the Liveview to full width of the sensor from a center crop.
Here are the setting I found to be the most stable and cleanest image while there's still lots of room for improvement .

What I did to get the results that I'm posting -- Load "iso research branch" build (I added 4096 (V) & 3840 (V) to mlv_lite.c  for extended resolution .
Loaded

Code Select Expand

adtg_gui.mo
crop_rec.mo
edmac.mo **(only for info)**
mlv_lite.mo

enabled "crop_rec.mo" , set to 1:1
enabled  "adtg_gui.mo" , toggled on "digic reg's" then enabled the 3K 5d3 preset(I changed the reg's to work with 5d2 ,6808-->6088 , 6804 -->6084)
that it , and trial & error got me to this point .

Code Select Expand

ADTG[0x1061] 0x249 -->3ca      **(1:1 frame Blanking)**
CMOS[0] 0x20b -->0xe1b
CMOS[1] 0xc00 -->0x10
CMOS[2] 0x40e -->0x10c

Here's the Liveview image dump for those setting -- frame resolution set to 4096x1250



And the HD image dump for the above Liveview dump




when CMOS[0] is disabled (no over riding reg's) the image turn back to center crop .

Code Select Expand

ADTG[0x1061] 0x249 -->3ca      **(1:1 frame Blanking)**
CMOS[1] 0xc00 -->0x10
CMOS[2] 0x40e -->0x10c

Liveview image dump -- frame resolution set to 4096x1250



HD image dump -- frame resolution as set to 4096x1250



So any one with a d4 (digic4) cam 7d , 50d etc. ... can follow along -- the process should be very similar

[reddeercity]

I'm starting to smell success !
Made go great process today with "crop_rec_4k_5D2" branch from "wasa57" in 3x crop_mode
I was able to get the vertical to 2304 (V) , max is 2144(V) thou I couldn't save a dng at that rez ,
but I was about to save a 2252x1075 dng after ml crop it , so really it's 2416x1127 the un-crop size or the "Raw Size"

Code Select Expand

Active Area  : 52 160 1127 2416

Here the Original dng from the Image_dump 2256x1075-RAW-000.DNG



here is also 2256x1075-RAW-000.png from the same image dump

Code Select Expand

ExifTool Version Number         : 10.45
File Name                       : RAW-000.DNG
Directory                       : C:/dcraw
File Size                       : 4.6 MB
File Modification Date/Time     : 2018:08:24 22:18:22-06:00
File Access Date/Time           : 2018:08:11 23:06:24-06:00
File Creation Date/Time         : 2018:08:11 23:06:24-06:00
File Permissions                : rw-rw-rw-
File Type                       : DNG
File Type Extension             : dng
MIME Type                       : image/x-adobe-dng
Exif Byte Order                 : Little-endian (Intel, II)
Image Description               :
Make                            : Canon
Camera Model Name               : Canon EOS 5D Mark II
Orientation                     : Horizontal (normal)
Software                        : Magic Lantern
Modify Date                     :
Artist                          :
Subfile Type                    : Full-resolution Image
Image Width                     : 2416
Image Height                    : 1127
Bits Per Sample                 : 14
Compression                     : Uncompressed
Photometric Interpretation      : Color Filter Array
Strip Offsets                   : 33792
Samples Per Pixel               : 1
Rows Per Strip                  : 1127
Strip Byte Counts               : 4764956
X Resolution                    : 180
Y Resolution                    : 180
Planar Configuration            : Chunky
Resolution Unit                 : inches
CFA Repeat Pattern Dim          : 2 2
CFA Pattern 2                   : 0 1 1 2
Black Level                     : 1278
White Level                     : 16200
Default Crop Origin             : 0 0
Default Crop Size               : 2256 1075
Active Area                     : 52 160 1127 2416
Opcode List 1                   : (Binary data 28 bytes, use -b option to extrac
t)
Copyright                       :
Exposure Time                   : 0
F Number                        : 0
Exposure Program                : Not Defined
ISO                             : 0
Exif Version                    : 0221
Date/Time Original              :
Exposure Compensation           : 0
Max Aperture Value              : 1.0
Metering Mode                   : Unknown
Flash                           : No Flash
Focal Length                    : 0.0 mm
Sub Sec Time                    :
Sub Sec Time Original           :
Focal Length In 35mm Format     : 0 mm
TIFF-EP Standard ID             : 1 0 0 0
Serial Number                   :
Lens Model                      :
DNG Version                     : 1.3.0.0
DNG Backward Version            : 1.3.0.0
Unique Camera Model             : Canon EOS 5D Mark II
Color Matrix 1                  : 0.4716 0.0603 -0.083 -0.7798 1.5474 0.248 -0.1
496 0.1937 0.6651
Analog Balance                  : 1 1 1
As Shot Neutral                 : 0.473635 1 0.624
Baseline Exposure               : undef
Baseline Noise                  : 1
Baseline Sharpness              : 1.333333333
Linear Response Limit           : 1
Calibration Illuminant 1        : D65
Frame Rate                      : 25
CFA Pattern                     : [Red,Green][Green,Blue]
Image Size                      : 2416x1127
Megapixels                      : 2.7
Shutter Speed                   : 0


Just adjusting with digic poke in the debug menu ,

Code Select Expand

c0f06084 - 0x30002
c0f06088 - 0x469ff77

[honza123]

Great !

[reddeercity]

Now that a1ex fixed adtg_gui.c in iso-resaerch branch I can proceed to get the full raw buffer for full raw width
Here's (magiclantern-iso-research-4k-D4.2018Aug29.5D2212.zip) the Up dated build I'm using , did some quick adjustments to 6084 & 6088 . Got a error printed on the screen

Code Select Expand

Default raw buffer too small (38MB , need 2148x16377 59MB - reallocating
[BKT] giving up
black 2/5 stdev too large (47077/100 ref 15535/100



Will at lest I know I'm working off a 38MB buffer -- do I need a 59MB buffer ?
@a1ex , what would  the buffer size the 5D3 uses for 4k extended res ? does it need to be 59MB

Here are the 6084 & 6088 settings , I first enable the digic reg's , exited to liveview press the play button reviewed a cr2 photo
re-entered adtg_gui.mo enabled the 3k preset for crop rec -- adjusted reg's 6084 & 6088



Shot a short 25 second 720p video on my iPhone , showing the error being printed on the screen with some other info
https://drive.google.com/file/d/129ONClwQUoZrQeJ9tg0moK7YnIeRdzKm/view?usp=sharing

the setting for 6088 was just a random number to see if the changes do really take effect , and they do

[a1ex]

59MB is way more than what you can allocate contiguously on these cameras; that number comes from the value dialed to C0F06088:

0x3FF90433 -> split in two -> 0x3FF9 and 0x433.
0x1 -> split in two -> 0x0 and 0x1 (better don't touch this one; leave it at 0x10036).
Difference: 0x3FF8 and 0x3FD.
The latter is X resolution divided by 2 (number of columns read out at once). That's how you've got 2148.
The former is Y resolution, but higher bits might be unused (hardware does not interpreted them, i.e. it works in the same way whether these bits are 1 or 0). However, ML interprets them.
So, 0x3FF8 = 16376. ML requests memory for 2148x16377 (looks like in your source, it adds 1). The hardware delivers 0xFF8 (4088) or 0x7F8 (2040); likely the former.

In your previous example, 0x469 - 3 = 1126; not sure how to interpret 0xff77 - 2 to get close to 2256...

In any case, I suggest adding small quantities to either the low halfword (for X) or to high one (for Y).

As you increase X, you may have to increase FPS timer A by the same amount (1 step = 2 pixels on 5D2), and possibly other registers that use similar values. For example, on 700D, timer A must be written to 0xC0F06008, 0xC0F0600C, 0xC0F06010, 0xC0F06824, 0xC0F06828, 0xC0F0682C and 0xC0F06830; the last 4 must be in sync and if you don't increment them, you will get just black bars.

As you increase Y, you may have to increase FPS timer B and PowerSaveTiming registers.

Other registers can be figured out by comparing x5 (1:1 crop) with plain 1080p (that's pretty much all you've got on 5D2). On 700D, comparing the regular 1080p with the 1:1 crop mode from Canon firmware was very helpful.

[reddeercity]

Thanks @a1ex that's exactly what I was looking for , makes sense now .
I was blinding adjusting , now I have more organized approached thanks.
 

[reddeercity]

From 6088 "default" 4f40432 in 1:1 FHD
4f4 =1268 (V or X)  432 = 1074 x 2 = 2148 (H or Y)
I tried 5f40432 to see what happens
5f4 = 1524 (V)  432 = 1074 x 2 =2148 (H)
increased vertically by 256 pixels if I did the math right .

[reddeercity]

Looking up now , 
I'm taken baby steps , able to increase the height now with
c0f06088 , A & B timers , target height is 2160 -- I'm at 1760

Code Select Expand

cmos[2]  40E --> 10E

Code Select Expand

c0f06088
4f40432 ->6f40432

Code Select Expand

Low Jello , 180d
A timers  --> 792
B timer --> 1750 (around there)





disregard the 3200 , I haven't adjust the lower half yet (432) i did quickly set the lower half to 632 to the first half of 6f4 and got it to lock to 2880x1760 & 3072x1760
but it lock up , needs more reg's to make it stable , I'm slowly finding the reg that correspond to the 5d3/D5 camera.
Can't save a dng yet , thou liveview dumps look promising .

[a1ex]

Made a couple experiments on 5D2, too. Initial notes:

- black level depends on raw type (either 1024 or 1792; need to sit down to find it)
- pink preview still stubborn
- CONFIG_EDMAC_RAW_SLURP kinda works, but gives corrupted frames
- timing for raw_lv_vsync appears to be correct
- calling raw_lv_vsync from the same place where Canon code would call it (StartImagePass x1/x5 CrawAddr / KindOfCraw), did not help
- patching the SetEDmac call from the above routine appears to work:

Code Select Expand


static void raw_lv_setedmac_patch(uint32_t* regs, uint32_t* stack, uint32_t pc)
{
    /* R0: EDMAC channel */
    /* R1: output buffer */
    /* R2: EDMAC info (geometry) */
    /* R3: flags */

    int width, height;
    int ok = raw_lv_get_resolution(&width, &height);
    if (ok)
    {
        if (redirected_raw_buffer)
        {
            /* optionally update output buffer */
            regs[1] = redirected_raw_buffer;
        }

        /* update EDMAC image size */
        int pitch = width * raw_info.bits_per_pixel / 8;
        static struct edmac_info dst_edmac_info;
        dst_edmac_info.xb = pitch;
        dst_edmac_info.yb = height - 1;
        regs[2] = (uint32_t) &dst_edmac_info;

        /* we can override this here */
        EngDrvOut(RAW_TYPE_REGISTER, lv_raw_type);
    }
}

static void raw_init()
{
    ...

    /* SetEDmac from StartImagePass_x1/x5 CrawAddr / KindOfCraw */
    /* FIXME: why it fails when the hook is placed on the BL instruction?! */
    patch_hook_function(0xFFA08B20, MEM(0xFFA08B20), raw_lv_setedmac_patch, "RAW LV x5");
    patch_hook_function(0xFFA08008, MEM(0xFFA08008), raw_lv_setedmac_patch, "RAW LV x1");
}


Still need to figure out why our code called from the same place doesn't work well, but the patched one does.

Now, adtg_gui stuff. FPS timer A has a small quirk - one unit means 4 pixels on this camera, but one unit of C0F06088 means 2 pixels. So, the easiest experiment is:
- enable ADTG registers and (in the advanced tab) ENGIO registers
- go to x5 zoom, enable mlv_rec's grayscale preview
- override FPS timer A (C0F06008) from 0x2770277 to 0x2770377 (increase X by 0x100)
- this change lets you increase raw resolution register (C0F06088) by 0x200, i.e. from 0x46A04BA to 0x46A06BA
- this gives 3168 pixels horizontally in mlv_rec: (0x6BA - 0x36)*2 = 3336, minus 160px black bar = 3176, then 3168 is multiple of 32.

Next is adjusting CMOS registers for centering the image horizontally - reply #10. Will edit later.



Part 2. Let's try to get full width LiveView.

Target values: the ones from photo mode.

Code Select Expand


              x1     x5 centered        photo    description
C0F06084:   0x10036     0x30036        0x10037   start row/column
C0F06088: 0x4F40432   0x46A04BA      0xEDD0B87   stop row/column (column: 1 unit = 2 pixels)
C0F06008: 0x23B023B   0x2770277      0x5DB05DB   FPS timer A (1 unit = 4 pixels horizontally)
C0F06014:     0x6D5       0x4FF          0xEDC   FPS timer B (1 unit = 1 pixel vertically)
CMOS[1] :     0xC00       0xE6A          0xC00   vertical start/stop
CMOS[2] :     0x40E       0x10E            0x8   horizontal pos & binning mode
CMOS[3] :       0x5         0x5            0x7   ???
CMOS[4] :     0x244       0x244          0x244   ???
CMOS[5] :       0x1        0x1           0xC05   ???


Code Select Expand


CMOS[2]:
xx-- -------- binning mode (00: read every column, 01: bin every 3 columns, 10 and 11: bin even more columns?!)
--xx xxxx---- horizontal position (0: start from left, 0x2a: thin stripe of valid data, 0x2b: no more valid data)
---- ----xxxx ??? (0x8 in photo mode, 0xE in LiveView, double highlights with 0x8 in LV)


For full width LiveView:
CMOS[2] = 0x00E
C0F06008: 0x27705DB (only the lowest half appears to matter on this camera)
C0F06084: 0x30037 (copied lowest half from photo mode)
C0F06088: 0x46A0B87 (copied lowest half from photo mode)
=> 5632x1074 in mlv_rec, 12.5 FPS.



Now, to increase height.
C0F06014: 0xEDC (copied from photo mode) => 4.2 FPS, clean image, so far so good.
C0F06088: 0x56A0B87 (increase Y by 0x100). Image breaks. Undo the change and look for other registers that might be related to timer B (close to 0x4FF)
First candidates: HEAD timers. Attempting to change them results in camera lockup. Back to the drawing board.

Some success starting from vanilla x5 and adjusting just the height:
- change C0F07150 from 0x49C to 0x59C and pray it won't lock up
- change C0F0713C from 0x476 to 0x576 and pray it won't lock up
- change C0F06088 from 0x46A04BA to 0x56A04BA => mlv_rec shows 2144x1330, but image is not good on the bottom half
- try to change CMOS[1] => E69 looks better, but image stutters; F6A looks even better, also with stutter, still some image data missing
- look for powersave timing registers: ADTG1[1047] from 0x550 to 0x650 appears to do the trick. maybe not needed? go to 1x and back to x5 to refresh.
=> 2144x1330 in mlv_rec with clean preview, still with some stutter with ML grayscale preview, but looks fine with Canon preview.

Let's try some more:
- C0F07150: 0x69C
- C0F0713C: 0x676
- C0F06088 to 0x66A04BA
=> 2144x1586 apparently working just fine.

- C0F07150: 0x79C
- C0F0713C: 0x776
- C0F06088 to 0x76A04BA
=> 2144x1842 apparently working just fine.

- CMOS[1]: 0xC00
- C0F07150: 0x79C
- C0F0713C: 0x776
- C0F06088 to 0x76A04BA
=> 2144x2354 apparently working just fine.

Battery empty, will continue tomorrow.

[reddeercity]

Holy Cow !!!  ,  it's not my birthday & I know it's not 'Xmas , just look outside to make sure it wasn't snowing 
5D2 & D4 users  a1ex has given use a amazing gift ! the proof that 4k is a real possibility on 5d2 or at least 3k 

This is incredible , thanks @a1ex -- so I guess I should try and reproduce some of your experiments (by the way I like your experiments way better then mind)
I knew about these one 's but anyway lock up cam

Code Select Expand

C0F07150:
C0F0713C
"CONFIG_EDMAC_RAW_SLURP" sound very encouraging

Anyhow here we go , let see if I can save a good dng in extended rez.

[reddeercity]

easiest experiment

Ok I can reproduce , got even an dng to export thought the image dump -- thou it's all distorted

Code Select Expand

Default Crop Size               : 3176 1075
Active Area                     : 52 160 1127 3336

dng from image dump converted to png so I can post it


Liveview .422 image


Here the original 3176x1075 RAW-001.DNG & LV-007.422 image .

Code Select Expand

ExifTool Version Number         : 10.45
File Name                       : RAW-001.DNG
Directory                       : C:/dcraw
File Size                       : 6.3 MB
File Modification Date/Time     : 2018:08:31 21:37:14-06:00
File Access Date/Time           : 2018:08:31 21:52:05-06:00
File Creation Date/Time         : 2018:08:31 21:52:05-06:00
File Permissions                : rw-rw-rw-
File Type                       : DNG
File Type Extension             : dng
MIME Type                       : image/x-adobe-dng
Exif Byte Order                 : Little-endian (Intel, II)
Make                            : Canon
Camera Model Name               : Canon EOS 5D Mark II
Orientation                     : Horizontal (normal)
Software                        : Magic Lantern
Subfile Type                    : Full-resolution Image
Image Width                     : 3336
Image Height                    : 1127
Bits Per Sample                 : 14
Compression                     : Uncompressed
Photometric Interpretation      : Color Filter Array
Strip Offsets                   : 33792
Samples Per Pixel               : 1
Rows Per Strip                  : 1127
Strip Byte Counts               : 6579426
X Resolution                    : 180
Y Resolution                    : 180
Planar Configuration            : Chunky
Resolution Unit                 : inches
CFA Repeat Pattern Dim          : 2 2
CFA Pattern 2                   : 0 1 1 2
Black Level                     : 8833
Default Crop Size               : 3176 1075
Active Area                     : 52 160 1127 3336

White Level                     : 16200
Default Crop Origin             : 0 0
DNG Version                     : 1.3.0.0
DNG Backward Version            : 1.3.0.0
Unique Camera Model             : Canon EOS 5D Mark II
Color Matrix 1                  : 0.4716 0.0603 -0.083 -0.7798 1.5474 0.248 -0.1
496 0.1937 0.6651
Analog Balance                  : 1 1 1
As Shot Neutral                 : 0.473635 1 0.624
Baseline Exposure               : undef
Baseline Noise                  : 1
Baseline Sharpness              : 1.333333333
Linear Response Limit           : 1
Calibration Illuminant 1        : D65
Frame Rate                      : 25
CFA Pattern                     : [Red,Green][Green,Blue]
Image Size                      : 3336x1127
Megapixels                      : 3.8


On to the next experiment .

[reddeercity]

Got to full width "5632x1075" for a short time then lock up , but I was able to save a image dump only good thing I got was a Liveview .422
scrambled up image , I didn't get a chance to change 6008 before it locked up .



Here the LV-008.422 from the rez of 5632x1074

[a1ex]

Hm, the DNGs should be clean, not sure what's missing. Did you start overriding from x5 zoom?

Anyway, back to full height.

CMOS[1]: 0xC00
C0F06014: 0xEDC (increased by 0x9DD)
C0F07150: need to increase by 0x9DD from 0x49C to 0xE79.
C0F0713C: need to increase by 0x9DD from 0x476 to 0xE5E.
C0F06088: 0x46a04ba + 0x9DD0000 = 0xE4704BA (a little lower than full height)
=> 2144x3600 in mlv_rec; the DNG from VRAM dumps looks clean.

Code Select Expand


Default Crop Size               : 2152 3600
Active Area                     : 52 160 3652 2312


Still need to reach 0xEDD04BA for full height (150 pixels left). Dialing this value doesn't work.

Let's increase everything by 150 units (0x9DD + 150 = 0xA73).

CMOS[1]: 0xC00
C0F06014: 0xF72
C0F07150: 0xF0F
C0F0713C: 0xEE9
C0F06088: 0xEDD04BA
=> 2144x3750 in mlv_rec, 9.6 FPS.

Let's add the full width:

CMOS[2] = 0x00E
C0F06008: 0x27705DB
C0F06084: 0x10037 (same as photo mode)
C0F06088: 0xEDD0B87 (same as photo mode)
=> 5632x3752 in mlv_rec, 4 FPS, DNG looks clean to me.

Code Select Expand


Default Crop Size               : 5632 3752
Active Area                     : 52 160 3804 5792




May require some fine-tuning to match a CR2, maybe the frame rate can be slightly increased, but this completes the proof of concept: arbitrary resolutions, up to full-res LiveView, are possible on 5D2 and likely on all other models already running ML.

This experiment was actually a continuation of the 700D full-res LV PoC.

Will commit the changes and update the experimental builds to make sure the above can be reproduced by any of you.

[waza57]

I have the same problem with C0F07150 and C0F0713C I can't override them correctly in my implementation of a 
crop_rec module for 5D2. Maybe that's why I'm limited in height

A few month ago i posted a dng with a promising size from my experiences in "dm-spy-experiments" branch.
It took a lot of time to implement this method in a module of crop_rec_4k branch.
I've seen that Reddeercity spend a lot of time time here, it's encouraged me to continue
 
for the moment only "CROP_PRESET_3X" and "CROP_PRESET_CENTER_Z".For those who want to test the code is there:

https://bitbucket.org/waza57/magic-lantern/src/crop_rec_4k_5D2/
Download link:
magiclantern-Nightly.2018Sep02.5D2212.zip
 
I hope it will be useful. It works for me but at your own risk (Digic poke is enable don't use it if you do not know what you are doing)

SETTINGS:
   
      Canon menu : NTSC 30
      ML menu    : set resolution to maximum ,  preview grayscal
      Modules to load: crop-rec, mlv play, mlv rec, silent
   
NOTES for developers:

I don't use CONFIG_EDMAC_RAW_SLURP.

I started from this:
https://www.magiclantern.fm/forum/index.php?topic=10111.msg123909#msg123909

https://www.magiclantern.fm/forum/index.php?topic=10111.msg125581#msg125581

And especially this where i decide to try increment height with patching ROM addresses.
https://www.magiclantern.fm/forum/index.php?topic=19336.msg183200#msg183200

Thanks to GREG!

For crop_rec.mo works i need to make change in functions of:

raw.c
          "raw_lv_get_resolution" ->  ...uint32_t top_left  = shamem_read(0xC0F06800)... to retrieve the size like this
          "autodetect_black_level"->  disabled,  otherwise  i got "raw detect error"
          "#define BLACK_LEVEL 1792"  otherwise bad black level
         
edmac-memcpy.c , fio-ml.c
          I need to disable call to function "clean_d_cache()" otherwise i can't patch ROM addresses

lv-img-engio.c
          I need to add this otherwise impossible to compile : "if defined(FEATURE_EXPO_ISO_DIGIC) && defined(CONFIG_EDMAC_RAW_SLURP)"
         
Thanks to a1ex for this project, without him, I would never have discovered the joys of programming in C.

TODO list:

- Increase height frame.
- mlv_lite module compilation.
- Compressed raw activation. ( new idea here too :Danne Idea )
- Shutter speed question ( is this bad?)
- compatibility with Dual_iso module question.
- ISO adjustment activation (https://www.magiclantern.fm/forum/index.php?topic=19336.msg205719#msg205719)
- Faulty HDMI connection
- Bad preview solution

* And all i will notice in your feedback

[a1ex]

Right, it took a lot of trial and error to figure out the meaning of all these registers. Back then, when I started to write the initial 4K hack, I didn't understand how this works well enough to make a generic implementation; that's why crop_rec ended up preset-based, with each resolution being manually tuned.

Now looking into 500D. This one is interesting because its 1080p mode is limited to 20 FPS. It does have 720p30 though. Let's compare them:

Code Select Expand


          1080p20  720p30  x5 zoom    photo
C0F06008  50F050F 5630563  58F058F  AF90AF9
C0F06014      4D1     304      3FF      C83
C0F06084    20040   20040    2004A    1004C
C0F06088  43D06C0 29506C0  360085A  C84132C
C0F0713C      440     294      370   unused
C0F07150      46E     2A1      388   unused


First mystery is at timer A: both 1080p20 and 720p30 use the same horizontal resolution, yet there are different values of timer A. Dialing down the value from 720p (0x563) to 0x50F does not work; image breaks below 0x543. Why?!

1080p: 1664x1083
720p: 1664x659

It's definitely one of the ADTG registers. Locked all of them to their values from 1080p, switched to 720p et voilà! 32 FPS with timer A = 50F!

Narrowed down to ADTG[100C] - the line skipping factor. Boo...



Full width (from x5):

FPS timer A: 1 unit = 2 pixels.
C0F06088: 1 unit = 1 pixel.
2 * 0x58F - 0x85A = 710
2 * 0xAF9 - 0x132c = 708
=>
C0F06008: 58F0AF9
C0F06084: 2004C
C0F06088: 360132C

Appears to work, but need to implement the display filters first in order to check the results. Will revisit later.

[reddeercity]

Ok starting over , trying the first experiment again to get 3k (H) in 3x crop
Still can't get a clean dng out , even try a fresh iso-reserach branch clone no go , there must be something I'm missing!
I can't get it to work at all with mlv_rec , kind of works with mlv_lite  -- so a1ex are you using mlv_lite?
even with mlv_lite I'm still at max rez of 2144 even if I try the full width , I get the res in the menu 5623 but still say max 2144.

It work better with my very hacky "raw_slurp" with the default raw at 4096X4096 (the one I post come from that )

Do I need to add that patch code you posted ? to made it work so I can export clean dng's ?

I followed your procedure

Code Select Expand

- enable ADTG registers and (in the advanced tab) ENGIO registers
- go to x5 zoom, enable mlv_rec's grayscale preview
- override FPS timer A (C0F06008) from 0x2770277 to 0x2770377 (increase X by 0x100)
- this change lets you increase raw resolution register (C0F06088) by 0x200, i.e. from 0x46A04BA to 0x46A06BA
- this gives 3168 pixels horizontally in mlv_rec: (0x6BA - 0x36)*2 = 3336, minus 160px black bar = 3176, then 3168 is multiple of 32.

When I got it to show the right size 3168x1074 it was with mlv_lite & "raw_slrup" with default raw buffer 4096x4096 which I thing is 32MB

I also had Frame Ride disabled , and I did see with the full width 5632 It was at 12.5 fps but image was garbage .

Should I try the 4k crop record branch with ADTG_GUI.mo ?
This confusing me why I can't it to work , the image is always garbage/noise/distorted .

@waza57 , good to see you back here !

[waza57]

....some unnecessary deleted modes at the moment in the crop_rec module :

new link : magiclantern-Nightly.2018Sep02.5D2212.zip

@ a1ex
Do you see a ADTG[100C] with adtg_ gui ? i'don't see it with 5D2.

@reddeercity
i have tried to simply copy the files from the adtg_gui folder of iso-research into the crop_rec_4k branch. it seems to work.

[reddeercity]

Great job @waza57 ! Tried you crop_record 4k build , and I can confirm 3xcrop 3.5k preset works @ 2880x1080 23.976 10bit out of the box !

Liveview image dump is good LV-011.422


Image dump dng also good RAW-003.DNG

Code Select Expand

Default Crop Size               : 2884 1080
Active Area                     : 54 264 1134 3148
Black Level                     : 1792
White Level                     : 16200


Code Select Expand

File Size                       : 6.4 MB
File Modification Date/Time     : 2018:09:01 23:34:10-06:00
File Access Date/Time           : 2018:09:01 23:58:53-06:00
File Creation Date/Time         : 2018:09:01 23:58:53-06:00
File Permissions                : rw-rw-rw-
File Type                       : DNG
File Type Extension             : dng
MIME Type                       : image/x-adobe-dng
Exif Byte Order                 : Little-endian (Intel, II)
Make                            : Canon
Camera Model Name               : Canon EOS 5D Mark II
Orientation                     : Horizontal (normal)
Software                        : Magic Lantern
Subfile Type                    : Full-resolution Image
Image Width                     : 3192
Image Height                    : 1200
Bits Per Sample                 : 14
Compression                     : Uncompressed
Photometric Interpretation      : Color Filter Array
Strip Offsets                   : 33792
Samples Per Pixel               : 1
Rows Per Strip                  : 1200
Strip Byte Counts               : 6703200
X Resolution                    : 180
Y Resolution                    : 180
Planar Configuration            : Chunky
Resolution Unit                 : inches
CFA Repeat Pattern Dim          : 2 2
CFA Pattern 2                   : 0 1 1 2
Black Level                     : 1792
White Level                     : 16200
Default Crop Origin             : 0 0
Default Crop Size               : 2884 1080
Active Area                     : 54 264 1134 3148
DNG Version                     : 1.3.0.0
DNG Backward Version            : 1.3.0.0
Unique Camera Model             : Canon EOS 5D Mark II
Color Matrix 1                  : 0.4716 0.0603 -0.083 -0.7798 1.5474 0.248 -0.1
496 0.1937 0.6651
Analog Balance                  : 1 1 1
As Shot Neutral                 : 0.473635 1 0.624
Baseline Exposure               : undef
Baseline Noise                  : 1
Baseline Sharpness              : 1.333333333
Linear Response Limit           : 1
Calibration Illuminant 1        : D65
Frame Rate                      : 25
CFA Pattern                     : [Red,Green][Green,Blue]
Image Size                      : 3192x1200
Megapixels                      : 3.8

Recorded a short 12 seconds raw video @10bit 2880x1080 23.976 here the ProRes_2880x1080_24p version of it , around 600 MB
The original raw video M01-2333.MLV , it's about a 1GB


Here a frame from that


Awesome job waza57 , works great so far .
I must of really messed up the raw.c somewhere in my builds , I have to check out your code and see where I when wrong .
Thanks again 

If anyone missed it 5D2 10 bit 3K raw video is a reality !
Another Pipe Dream comes true 

[Ottoga]

If anyone missed it 5D2 10 bit 3K raw video is a reality !
Another Pipe Dream comes true 

So cool Reddeercity, persistence and perseverance really paid off.

[scotophorus]

i've been reading your updates on this thread for the last year and a half and now i have to spam you some thanks for this great achiement: holy sh*t! that's amazing! thank you very much @alex @reddercity @waza57 @dfort and everyone involved

[a1ex]

@ a1ex
Do you see a ADTG[100C] with adtg_ gui ? i'don't see it with 5D2.

Yes, I see it after getting out of LiveView and back. Works as expected.

Looking for a 720p mode on 5D2? Now we need to figure out how to reduce the resolution in order to increase the frame rate

On 700D I've ran into a restriction that appears to be related to the live preview: modified resolution could not have an area (WxH) smaller than in the original video mode. The behavior is likely there on earlier models as well. What I think it's going on: the image processing modules that create the preview image will not get enough input data, and they will just keep waiting, freezing the preview and eventually crashing the camera. If the modified resolution is larger, the modules get enough input data, possibly with incorrect dimensions, they discard the extra data and the camera ends up displaying something (a garbled image); and it doesn't lock up.

edit: reducing resolution seems to work, to some extent, LiveView is frozen, camera didn't crash after ~ 1 minute (didn't try more), there are some black level issues, mlv_rec shows 1856x704 at 50.004 FPS.

[kwstas]

it's alive! congrats guys!

I can confirm
20 sec mlv, 2880x1080 @ 10 bit, though a -116px black at the right
13 sec mlv, 2540x1080 @ 12 bit

my card does 77mb at average

--
edit: cropped mode a bit unstable yet, seems not to be working now

[waza57]

@ a1ex

OK, thanks. I have clone  crop-rec4k branch and work on it with a copy of the adtg_gui folder of iso-research branch.
The things compile and seems to work like this and permitted a minimal  crop_work module to work, but:

adtg[100c] is not present even if i refresh livevew or go to play mode and back or go to x5 an go to play mode and...and.. ...nothing ...nothing.

C0F0713C  and C0F07150  i pray for  it won't lock up but god was not in my build.....

it's why , i would like to know, if it's possible:
 
1 - what is exactly the reference of your commit.
2 - Also, if you are currently using your "raw_lv_setedmac_patch" solution?
3 - if yes , is this solution works only with CONFIG_EDMAC_RAW_SLURP
4- And because I'm a little lost , what this solution does concretely.

@reddeercity

happy birthday anyway!

@kwstas

Do you use the new download link ? what is the problem ? have you apply my recommanded settings ?