3K/UHD 5D2 Raw development and Other Digic IV Cams

[reddeercity]

@Ilia3101 no not yet , but I'll give it try soon.
I'm actually trying to find out why HDMI is broken in the nightly builds (unified) no one has reported it
and as far as I can tell it's been broken since the "New Core" (I use the Old Core for production work I must have
HDMI for my production workflow) , so I'm undecided to which way to go . I can implement 10-12bit
and my experiments in UHD plus waza57 code in to the "Old Core" I'm starting to see too many restriction with the new core as
it must be compatible with Digic 5 Camera (5D3) and I'm not to sure about the Digic's 4 (5D2) life expectedly or support .
I know it sound discouraging , but I have to wonder if all this time & effort is really worth it after all .
I can deal with a limited 1:1 resolution but not HDMI , that the deal breaker .
So I need to recess this thread in the direction it goes .
Sorry to be off topic a bit .

[reddeercity]

reddeercity, Thanks for that but it seems timer B don't increase height like with other models.
I tried a lot of things but nothing works.
Perhaps ADTG [8xxx] but unfortunately I don't succeed to see it with adtg-gui module in 5D2.
An example :

effective width: 2840
effective height: 1082

Being trying to reproduce this from @waza57 but no go , I clone crop_rec_4k_5d2 branch and update to comment f73a13b -- I can load crop_crop.mo but it's disable , the mlv_lite and adtg_gui.mo didn't make .
So can you tell me how you got this please ? was this on mlv_lite or mlv_rec ? I see in the "Digic Poke" where you code is , can you share the setting in there ?
In the mean time I'll play with more , maybe by chance i can find the right setting to reproduce this 
Edit: your working in this right ?

Edit:2 it seem I'm still missing some code in raw.c 9a0b318 plus some comment's I missed , I thought the f73a13b comment was the consolidation of the code from waza57 , I'll continue tomorrow

[reddeercity]

I'm getting some error compiling now in raw.c , I'm not sure why

Code Select Expand

../../src/raw.c: In function 'raw_lv_get_resolution':
../../src/raw.c:682:5: error: invalid storage class for function 'raw_update_params_work'
int raw_update_params_work()


Code Select Expand

int raw_update_params_work()
anything to do with "invalid storage class for function 'raw_update" etc. .....
Can some one please help explain this error .
I'm working off waza57 "crop_rec_5d2"
https://bitbucket.org/waza57/magic-lantern/commits/9a0b318953a02cc12adc09e44253c5772abeb939#Lsrc/raw.cF678T678
Edit: Here my source code I'm working off I uploaded to my bitbucket downloads
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/2018-4-26-crop_rec_5d2.zip

[reddeercity]

I put aside "waza57 crop_rec_5d2" code on hold and when back my code and my some improvements (It's gets better) 
in raw.c I added

Code Select Expand

#ifdef CONFIG_5D2
#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0xF000)
#end
In adtg_gui.c I changed 0x6800 -->0x6084/8

Code Select Expand


static int res3k_reg(int reg)
{
    if (regs[reg].dst == 0xC0F0)
    {
        switch (regs[reg].reg)
        {
            case 0x6084/8:                /* C0F06084/8 - raw resolution */
                return 0x52801AB;       /* from 0x528011B -> 3072px in raw_rec */
        }
    }

and added in mlv_lite.c the extended resolutions

Code Select Expand

3840, 4096





 


I was able to extend the horizontal to1076 from 1044 , not much but a start 
I did dump the image buffer this what I'm getting , (please remember I did not do any timer adjustment yet)
The HD Dump

The LV Dump

The second HD Dump after trying to move the image down to center

Playing around with liveview preview

Screen shot -- I didn't do any timer adjustment yet


Here the build I compiled (the latest updated source + ISO Research)

Code Select Expand

# Magic Lantern Nightly.2018May03.5D2212 (f683753bc4e3+ (iso-research))
# Built on 2018-05-03 03:54:46 UTC by ml@ml-pc
# Configuration saved on 2018/05/02 22:40:24
beta.warn = 2
menu.first = -4
movie.log = 1
rec.notify = 0
enable-liveview = 2
fps.preset = 1
fps.override.idx = 32
fps.override = 1
battery.drain.rate.rev = 59
hist.log = 0
spotmeter.draw = 0
clear.preview = 4
zebra.draw = 0
disp.mode.x = 149

So What I did was load the adtg_gui.mo ,  mlv_lite.mo  &  Crop_Rec.mo with crop_rec I put the cam in to 1:1 3x mode so every pixel & line (by pass line skipping)
then loaded the 3k 5d3 preset but I changed the regs as noted above .
It changed CMOS[1]0xc00 --> 0x2cb & CMOS[2] 0x40e --> 0x10e
I'm able to switch between all res's up to 4096 (I added 3840 & 4096) yes I can set it to 3520 also , before I was lock to just 3584 and thou the cmos[2]
I was able to change the  horizontal by 32 lines , (not much but a start)

I really just simplified the code after learning what I read in the 10-12bit thread 
I think this is the right path , though the registers just like the 5d3 only make sense being there's so closely related -- brother from a different mother 
more to come , next is to clean the raw image up and start recording a image only if for a few second until the buffer is full then I need to finish the lossless compressed raw
code in silent .c
The last thing is the content of MENUS.CFG

Code Select Expand

0 3F77F0FE 3D41FF14 Overlay\Zebras
0 3F783078 3D556563 Overlay\Spotmeter
0 409A2D5E 3E935DBF Movie\FPS override
0 3FF3E2C6 3CA1335E Movie\Crop mode
0 409BA922 3F3448D6 Movie\RAW video
0 3F711935 3B573CCE Movie\Movie Tweaks
0 3FFA4E97 3E70270A Display\Clear overlays
0 407B0F0F 3F208C48 Display\Anamorphic
0 3FFB325A 3EF96A01 Debug\Memory patches
0 3FF8BDA4 3EF33695 Debug\Show console
0 409C1067 3F4B1C91 Debug\ADTG Registers
0 3F7308B9 3BE6B0C1 Debug\Lens info
0 40BC1075 40007E74 Debug\Screenshot - 10s
0 3FFA92E5 3EDB7170 Debug\Dump image buffers
0 3F7346F1 3BFDC26F Debug\Don't click me!
0 3F7E7825 3F10815C Debug\Free Memory
0 3F7156EE 3B6CC2E3 Movie Tweaks\Movie Logging
0 3F7194B7 3B8237FD Movie Tweaks\REC/STBY notify
0 3F71D290 3B8F3D97 Movie Tweaks\Force LiveView
0 3F72CA91 3BD1B7F5 FPS override\Desired FPS
0 3F728C79 3BBEA73B FPS override\Optimize for
0 3F787002 3D6ABC54 Histogram\Scaling
0 3F75B891 3CA48BF9 Advanced\DIGIC Registers
0 3F78EF46 3D8E03DC Advanced\Log Registers Now
0 3FF9750E 3EAE7395 Advanced\1:1 3K crop mode (5D3)
0 4079DAD8 3FA5BBBF RAW video\Resolution
0 3FF5D867 3D364505 RAW video\Aspect ratio
0 3FFF9DD2 3FDE2613 RAW video\Preview
0 3F747EF9 3C4C5756 RAW video\Advanced...
0 3F7EB94A 3F1EF4B2 Free Memory\Allocated RAM


[reddeercity]

some progress cleaning up Lv @ 4096x1044

Image dump of Liveview in "framing preview"

adjustments CMOS[1]  with ADTG, 0x106E, Shutter blanking for LiveView 1x

Screen capture Liveview


Even thou it in 4096x1044 It seem it retaining the 1856 buffer & just mirroring  it
maybe my size is not right but with SRM buffer my understanding there should enough 32mb per frame , right ?

Code Select Expand

RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0xF000)
I thing this what keeping me from saving a DNG on dumps , it does export a DNG but it's empty
I'll continue tomorrow .

[reddeercity]

Still haven't got the raw image right yet but I can increase the vertical to 1250 , so that's 4096x1250 .
It's was a matter of just changing Aspect Ratio to 1:1 or 3:2 .

Image dump


Edit: Gave a go at 3x crop mode to see if I could get something there got a interesting result

Just played around with c0f0 0x6084 & c0f0 0x6088 reg's and when to the change the res in the raw video menu .
I don't think there any there yet but interesting

Edit:2 I thing I have to rework the buffer or I need to fully implement raw_slurp  with SRM memory
I can easily put the sensor in to 1:1 without the 3k preset in adtg_gui.mo

Code Select Expand

cmos[1] 0xc00 --> 0xbc0  (height)
cmos[2] 0x40e --> 0x10e (width)
With that I choose up to 4096x1250 , I got that by just switching in to 3x crop_mode and reading what the reg's at cmos 1&2  are.
plus I think I found the reg's for c0f0 0x6084 & 0x6088 in 3x crop_mode

Code Select Expand

c0f0 0x6084 --> 0x10036 (1:1 FHD)
c0f0 0x6084 --> 0x30036 (3x crop_mode)
c0f0 0x6088 --> 0x4f40432 (1:1 FHD)
c0f0 0x6088 --> 0x46a04ba (3x crop_mode)

[reddeercity]

Having some trouble get the custom buffer to compile for the higher res redirect using srm memory  as per a1ex  talks said here  I found the comment  crop_rec_4k#Lsrc/raw.cT149  and I finished implmenting the rest of raw_slurp for srm memory as per what a1ex has done for the 5d3 for 4k . So my error is this
src/raw.c:578:39: error: 'DEFAULT_RAW_BUFFER'...........
see below -- from the terminal window

Code Select Expand

../../src/raw.c:161:0: warning: "RAW_LV_BUFFER_ALLOC_SIZE" redefined [enabled by default]
#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0x1000)
^
../../src/raw.c:147:0: note: this is the location of the previous definition
#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0xF000)
^
../../src/raw.c: In function 'raw_lv_realloc_buffer':
../../src/raw.c:578:39: error: 'DEFAULT_RAW_BUFFER' undeclared (first use in this function)
         if (raw_lv_buffer != (void *) DEFAULT_RAW_BUFFER)
                                       ^
../../src/raw.c:578:39: note: each undeclared identifier is reported only once for each function it appears in
make: *** [raw.o] Error 1

if you have trouble following what posted from the raw.c I uploaded the whole file to my bitbucket downloads folder
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/adtg_gui_4k_raw_slurp_redierct_buffer-raw.c
here  part of the code, I hope someone can help me understand the problem
I added more code then needed  so everyone can get the idea of with happening

 

Code Select Expand

71- /*********************** Camera-specific constants ****************************/
72-
73- #ifdef CONFIG_EDMAC_RAW_SLURP
74- /* undefine so we don't use it by mistake */
75- #undef RAW_LV_EDMAC
76-
77- /* hardcode Canon's raw buffer directly */
78- /* you can find it from lv_raw_dump, arg1 passed to dump_file:
79-  *
  80- * raw_buffer = get_raw_buffer()
  81- * sprintf_maybe(filename, '%08lx.mm1', raw_buffer)
  82-  * ...
  83-  * dump_file(filename, raw_buffer, 7*something...)
  84- */
..................
146- #ifdef CONFIG_5D2
147- #define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0xF000)
148- #define DEFAULT_RAW_BUFFER_SIZE (9*1024*1024)     (I added this as I thought it be big enough , not too sure thou -- is this my problem  not sure how to find it out )
149- #endif
150-
151- #ifndef DEFAULT_RAW_BUFFER_SIZE
152- /* todo: figure out how much Canon code allocates for their LV RAW buffer - how? */
153- #warning FIXME: using dummy DEFAULT_RAW_BUFFER_SIZE
154- #define DEFAULT_RAW_BUFFER_SIZE (9*1024*1024)
155- #endif
156-
157- /* for higher resolutions we'll allocate a new buffer, as needed */
158- /* all cameras using CONFIG_EDMAC_RAW_SLURP should be able to handle this */
159- /* SRM_BUFFER_SIZE matches the full-res image size, as 14-bit uncompressed (actually a bit larger, but not much) */
160- #define CONFIG_ALLOCATE_RAW_LV_BUFFER
161- #define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0x1000)
..........................

549- #ifdef CONFIG_EDMAC_RAW_SLURP
550-
551- /* requires raw_sem */
552- static void raw_lv_free_buffer()
553- {
554-   printf("Freeing LV raw buffer %x.\n", raw_lv_buffer);
555-   if(raw_allocated_lv_buffer) {
556-        free(raw_allocated_lv_buffer);
557-        raw_allocated_lv_buffer = 0;
558-   }
559-   raw_lv_buffer = 0;
560-   raw_lv_buffer_size = 0;
561- }
562-
563- /* requires raw_sem */
564- static void raw_lv_realloc_buffer()
565- {
566-   int width, height;
567-  int ok = raw_lv_get_resolution(&width, &height);
568-  if (!ok)
569-  {
570-      ASSERT(0);
  571-     return;
572-   }
573-
574-    int required_size = width * height * 14/8;
575-    if (DEFAULT_RAW_BUFFER_SIZE >= required_size)
576-   {
577-      /* no need for a larger buffer */
578-      if (raw_lv_buffer != (void *) DEFAULT_RAW_BUFFER)     ----- (this where the error is  not sure to resolve it )
579-      {
580-          printf("Default raw buffer OK for %dx%d (%s)", width, height, format_memory_size(required_size));
581-
582-          if (raw_lv_buffer && raw_lv_buffer == raw_allocated_lv_buffer)
583-          {
584-             printf(" - back to default.\n");
  585-            raw_lv_free_buffer();
  586-         }
  587-         else if (raw_lv_buffer)
  588-         {
  589-               printf(": %x -> %x\n", raw_lv_buffer, DEFAULT_RAW_BUFFER);
   590-        }
  591-         else
  592-         {
  593-              printf(".\n");
  594-           }
   595-       }
   596-
   597-      raw_lv_buffer = (void *) DEFAULT_RAW_BUFFER;
   598-      raw_lv_buffer_size = DEFAULT_RAW_BUFFER_SIZE;
   599-      return;
   600-   }
   601-
   602-    if (raw_lv_buffer_size >= required_size)
   603-    {
   604-          /* no need for a larger buffer */
    605-       return;
    606-    }
    607-
    608-   printf("Default raw buffer too small (%s", format_memory_size(raw_lv_buffer_size));
    609-   printf(", need %dx%d %s) - reallocating.\n", width, height, format_memory_size(required_size));
    610-
    611-   if (raw_lv_buffer && raw_lv_buffer != (void *) DEFAULT_RAW_BUFFER)
    612-   {
    613-       ASSERT(0);
    614-        return;
     615-   }
    616-
    617- #ifdef CONFIG_ALLOCATE_RAW_LV_BUFFER
    618-         raw_allocated_lv_buffer = fio_malloc(RAW_LV_BUFFER_ALLOC_SIZE);
    619-         raw_lv_buffer = raw_allocated_lv_buffer;
    620-         raw_lv_buffer_size = RAW_LV_BUFFER_ALLOC_SIZE;
    621-         return;
622- #endif /* CONFIG_ALLOCATE_RAW_LV_BUFFER */
623-
624-          /* you should enable CONFIG_ALLOCATE_RAW_LV_BUFFER
625-           * or find some other way to reserve memory for the RAW LV buffer */
626-          ASSERT(0);
627- }
628-
629- #endif  /* CONFIG_EDMAC_RAW_SLURP */
630- #endif /* CONFIG_RAW_LIVEVIEW */
631-


[reddeercity]

It look like I have problems with the redirect buffer
crop_rec_4k#Lsrc/raw.c-614

Code Select Expand

if (raw_lv_buffer != (void *) DEFAULT_RAW_BUFFER)
I added

Code Select Expand

#ifdef CONFIG_5D2
#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0x1000)
#define DEFAULT_RAW_BUFFER_SIZE (9*1024*1024).
#endif
Is there a way to find the "DEFAULT_RAW_BUFFER_SIZE" ?
I just copy the default values from
crop_rec_4k#Lsrc/raw.cT146

Code Select Expand

#define DEFAULT_RAW_BUFFER_SIZE (9*1024*1024)
#endif
but if I'm using "SRM_Buffer" do I really need the "Default_Raw_Buffer_Size"
this is confusing me , I did find the "working buffer size" in one of my logs but seem too small

Code Select Expand

1726C> GuiMainTas:ffa4522c:18:03: (PUB) SetImageWorkMemory 1:0x40d00000, 2:0x41700000
172A7> GuiMainTas:ffa45274:18:03: (PUB) SetImageWorkMaxPixel W:2200,H:1872,0x5e4340
as it for 2200x1872 pixels.

[reddeercity]

Will every think works as far as getting in to 1:1 pixel mode in FHD and ml can select any resolution from 1856 to 4096 up to 1:1 Aspect Ratio 
and it will be able (theoretical specking) to capture up full width 5634 . Very clever software , Magic Lantern !
I starting to see the genius of the code -- the whole infrastructure is very flexible , once I followed a1ex's comments , it's all very clear now   ,
the 5d2 works so very close to 5d3 but with I small difference really , e.g. , c0f0 0x6800 0x6804 (d5/5d3) c0f0 0x6088 0x6084 (d4/5d2)
Indeed the 5d3's has the same cmos [2] reg's (0x10E) ,  is the same as the 5d2 cmos[2] reg's (0x10E) for 1:1 pixel FHD and the vertical offset cmos[1]  is a little different
but close enough to get started on cleaning up the offset .
The height limitations is equal to the height of the image buffer , in my case it's still using either the full HD buffer (1880x1250) in 1:1 or 3x crop 2144x1076 and that's the new kettle of fish Has to be solved .
I reproduce every thing in 3xcrop_mode and can get up to 4096x1076 , but the height is lock to the 3x buffer , that why I can get 1250 vertical in 1:1 because the image buffer size ,
the default full raw buffer size need to be found so the redirect buffer code infrastructure can be used with SRM Memory like all the rest of the cam or almost all .
If that can't work then a custom redirect image buffer for d4 cam will need to be written , maybe similar to what I was working on  before raw_slurp

Code Select Expand

#ifdef CONFIG_5D2
#define RAW_LV_BUFFER_ALLOC_SIZE (2040*1267)
#endif
that's equal to the full image buffer full image buffer on 5d2 .

I have a request from any d4 cam users (50d ,550d etc. ..) but I prefer 50d or 7d , I'm interested in if we can put others d4 cams in to 1:1 FHD 4k mode
you need ADTG_GUI.mo from the ISO-Research branch , the best way is to compile with the latest source +the iso-research branch .
some small mod's to adtg_gui.c  adtg_gui.c-1127 change 0x6804 to 0x6084/8 & C0F06804 to C0F06084/8
and in mlv_lite.c add 3840 & 4096 to the list of resolutions that sure do it. If you can't compile of has a problem let me know here and I can provide a build.
What I'm looking for is the cmos[1] , cmos[2] & c0f0 0x6084 & c0f0 0x6088 you'll need to enable digic reg's and put the cam in to 3x crop_mode and record the setting at those reg's
c0f0 0x6084 should be something like 0x30036 in 3x crop mode & 0x10036 in 1:1 mode at least that what the 5d2 says & cmos[2] should be 0x10e in 3x crop 1:1 is 0x40e same thing
that the 5d2 info so if other cam report the same then we can port UHD or at least a true crop record of the sensor in 1:1 maybe 2.5k so who knows.

[reddeercity]

Will I got the redirect buffer to compile in raw.c -- stupid mistake in the "#ifdef CONFIG_5D2" forgot to close the #ifdef with "#endif"
any how as I said before I implemented "raw_slurp" for "srm memory" for the redirect buffer . link to the raw.c file below if any one wants to follow along closer
redirect_buffer_4k_res_iso-research_branch_raw.c
added

Code Select Expand

#ifdef CONFIG_5D2
/* for higher resolutions we'll allocate a new buffer, as needed */
#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0x1000)
#define DEFAULT_RAW_BUFFER (4096*4096)
#endif
not sure if it working yet , just start to experimenting  , I don't think the "default_raw_buffer" is correct I just pick the biggest one there was I saw
in the EDMAC 4096*4096 . I did try 3574*1268 but when I enable raw vide(mlv_lite) all there was "0" or 0x0 for size so I changed to 4096*4096 and raw vide work again at
least before I enabled the adtg_gui.mo . Needed to be reworked more , I wanted at least at this point to be able too compile and it does so fine tuning  buffer sizes
and or finding the correct size is next .

the redirect buffer

Code Select Expand

/* requires raw_sem */
static void raw_lv_realloc_buffer()
{
    int width, height;
    int ok = raw_lv_get_resolution(&width, &height);
    if (!ok)
    {
        ASSERT(0);
        return;
    }

    int required_size = width * height * 14/8;
    if (DEFAULT_RAW_BUFFER_SIZE >= required_size)
    {
        /* no need for a larger buffer */
        if (raw_lv_buffer != (void *) DEFAULT_RAW_BUFFER)
        {
            printf("Default raw buffer OK for %dx%d (%s)", width, height, format_memory_size(required_size));

            if (raw_lv_buffer && raw_lv_buffer == raw_allocated_lv_buffer)
            {
                printf(" - back to default.\n");
                raw_lv_free_buffer();
            }
            else if (raw_lv_buffer)
            {
                printf(": %x -> %x\n", raw_lv_buffer, DEFAULT_RAW_BUFFER);
            }
            else
            {
                printf(".\n");
            }
        }

        raw_lv_buffer = (void *) DEFAULT_RAW_BUFFER;
        raw_lv_buffer_size = DEFAULT_RAW_BUFFER_SIZE;
        return;
    }

    if (raw_lv_buffer_size >= required_size)
    {
        /* no need for a larger buffer */
        return;
    }

    printf("Default raw buffer too small (%s", format_memory_size(raw_lv_buffer_size));
    printf(", need %dx%d %s) - reallocating.\n", width, height, format_memory_size(required_size));

    if (raw_lv_buffer && raw_lv_buffer != (void *) DEFAULT_RAW_BUFFER)
    {
        ASSERT(0);
        return;
    }

#ifdef CONFIG_ALLOCATE_RAW_LV_BUFFER
    raw_allocated_lv_buffer = fio_malloc(RAW_LV_BUFFER_ALLOC_SIZE);
    raw_lv_buffer = raw_allocated_lv_buffer;
    raw_lv_buffer_size = RAW_LV_BUFFER_ALLOC_SIZE;
    return;
#endif /* CONFIG_ALLOCATE_RAW_LV_BUFFER */

    /* you should enable CONFIG_ALLOCATE_RAW_LV_BUFFER
     * or find some other way to reserve memory for the RAW LV buffer */
    ASSERT(0);
}

In the edmac-memcpy.c -- added for raw_slurp (I just took the code from the here

Code Select Expand

/** this method bypasses Canon's lv_save_raw and slurps the raw data directly from connection #0 */
#ifdef CONFIG_EDMAC_RAW_SLURP
   ......
#elif defined(EVF_STATE)
uint32_t raw_write_chan = 0x12; /* 60D and newer, including all DIGIC V */
#elif defined(CONFIG_5D2)
uint32_t raw_write_chan = 0x02; /* testing 0x01 and 0x02 */
#endif

Code Select Expand

platform/5D2.212/consts.h
#define SRM_BUFFER_SIZE 0x25D0000   /* print it from srm_malloc_cbr */

Code Select Expand

platform/5D2.212/internals.h
/** this method bypasses Canon's lv_save_raw and slurps the raw data directly from connection #0 */
#define CONFIG_EDMAC_RAW_SLURP
#define CONFIG_ALLOCATE_RAW_LV_BUFFER

adtg_gui.c I changed c0f0 0x6804 to 6084/8

Code Select Expand

static int res3k_reg(int reg)
{
    if (regs[reg].dst == 0xC0F0)
    {
        switch (regs[reg].reg)
        {
            case 0x6084/8:                /* C0F06084/8 - raw resolution */
                return 0x52801AB;       /* from 0x528011B -> 3072px in raw_rec */
        }
    }

    return 0;
}

[reddeercity]

another piece of the puzzle of the redirect buffer
I just added

Code Select Expand

/* for higher resolutions we'll allocate a new buffer, as needed */
/* all cameras using CONFIG_EDMAC_RAW_SLURP should be able to handle this */
/* SRM_BUFFER_SIZE matches the full-res image size, as 14-bit uncompressed (actually a bit larger, but not much) */
#define CONFIG_ALLOCATE_RAW_LV_BUFFER
#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0x1000)

I updated the raw.c and it's on my bitbucket downloads I renamed it to adtg_gui_redirect_4k_buffer_raw.c for those who are following my progress .

I also found out how to find the "default_raw_buffer" -- compile the allocate-raw-lv-buffer branch with these comments from a1ex e799e89

Raw backend -- attempt to autodetect Canon's raw buffer size
Usage:
- compile with CONFIG_MARK_UNUSED_MEMORY_AT_STARTUP (config-defines.h)
- open the console (Debug menu) and enable something that uses LiveView RAW features (raw video, raw histogram etc)
- test by starting the camera in all video modes (photo, 1080p, 720p, crop, x5 etc)
- take a screenshot or write down the console messages

This something all Digic 4 cam user can do easily and would help with other features beside 4k/UHD
It you do please just post result on this thread .
looking for :
5D2 -- (I should be able to get this one , but it won't hurt to post other results)
50D
7D
550d
600d (? maybe )
did I miss any ? for digic4

[reddeercity]

some progress in 3x crop_mode , did a image dump LV-005.422 LV-004.422 , starting to get the image to move to the left and push out the ghost image .
you can see the black border shrinking , tried to export the 422 image but the "YUV 422 Convertor" didn't recognizes the resolution but it could be view in the app.
most notable things I'm adjusting is the "A" &"B" Timers & the 3x shutter blanking & cmos[4]
screen shots of app with 422 dumps



yea some how I enabled dual iso with cmos [4]
And one last thing the A & B timers make it more stable , seems timer "B" with higher numbers (greater then 1910<) keeps the image in extended resolutions
so it's not only totally depended on just cmos's reg's , timers have a great deal to do with it also .

[reddeercity]

I got QEMU up & running finally and i'm looking for some other d4 cam rom's to investigate .
mainly looking for 50D plus T3i (600D) T2i(550d) , T1i (500D)
I have 7D 203 (firmware)   rom . Just PM a link to the roms  , don't post  links here
Thanks.

[histor]

Just a strange bug in the broken3x-12bit build - I've seen it twice with different cards. It destroys the \ML\modules\ folder (or does something wrong to the filesystem). Finally it fails with "Can't find 5D2_212.sym"

[Walter Schulz]

5D2 handling takes special care. Never! remove card without waiting some seconds for write operations to complete. Yes, cam will access card *after* opening card compartment lid!
Are you sure you took that into consideration?

[histor]

The card was in all the time. As well as battery. It's something build specific, I guess.

[reddeercity]

Just a strange bug in the broken3x-12bit build - I've seen it twice with different cards. It destroys the \ML\modules\ folder (or does something wrong to the filesystem). Finally it fails with "Can't find 5D2_212.sym"

Thanks for the feed back on the bleeding edge build (not for the light hearted) 
What @Walter said still hold true , even if you open the cf card door to fast while the red light still blinking (writing file for ml) you can get wipe out files or corruption
Yes ,  I've seen this happen before on just the plain nightly builds , but i find it's usually a user error . Pressing  buttons/switch's to fast or too many times , etc. ....
But to really know I would need more info , like all the cam setting , any error messages , etc. ...... , If you clear the setting file in ML on CF card does it still happen ?
If you put the card in your reader , are all the ML files still there ? after ml can't find the .sys file ?
What CF cards are you using ? (needs to be @ least 1000x)

To be clear , this build has limited feature support . That means only basic functions , e.g. all manual adjustment not Auto feature.(frame over ride is OK to use )
Cam must set to manual M mode and Only use pure analog ISO's (100,200,400,800) not digital ISO's (160,320,500,600,etc. ...)
You must turn off(disable) mlv_rec and or mlv_lite before you adjust ISO's or it will crash/lockup .

One more thought , after you get the "Can't find 5D2_212.sym" turn cam off and pull the battery out and try a reboot from cold.
Does it work now ? I've sometimes seen ML miss loading files like "crop marks" or module file , I just pull battery out and then reboot
all is fine usually after that .

[histor]

I've packed logs and settings. http://disk.karelia.pro/crfYi80/broken3x-12bit-bug.zip
In short - files from the build stay unaltered but all modules vanished. That's what cart reader sees. While previous crash I saw some alien characters instead of the filenames. Windows can do nothing with them but format.  The second time I've just copied the Modules\ back – and everything works with the same settings.
What camera sees when  the error occurs:  the message about 5D2_212.sym and no Global draw overlays. You can shoot, can't call ML menus. It's interesting, what was before that, but I can't roll my mind back.
There was no intensive card writing, the cards are OK (SanDisk Extreme Compact  SDCFXSB-128G-G46 and smaller SDCFXSB-064G-G46). No video recording, only lazy taking pictures in Live view. ISO1000 is a bad setting, but it doesn't stop taking stills or working live view. Another bad setting is leaving mlv_lite, mlv_rec and mlv_snd enabled together. That's a cruel way to get 10bit raw with sound. And that's not a sound of explosion : ) Is there any legal way to do it?
In fact 10-12bit is a fascinating feature for 5d2 and I'm watching  every reddeercity's  post with great attention. I'm not afraid of multiple crashes now, I have to accommodate with the limitations - to avoid them in the field.
Maybe that's not very useful information, it's hard to trace it from actual user error to the final result. My only question is: why ML rewrites modules on shutdown? and what for?
To end with something good – I'm actually charmed with ML for a decade.

[a1ex]

Can you find a way to reproduce the issue? Can you do the same without entering LiveView? Can you narrow it down to some particular module?

Can you get the same issue with regular builds?

ML does not rewrite modules on shutdown; it just saves its settings. However, if there's a bug that causes some sort of memory corruption, its side effects can be just about anything. I'm unable to find other explanation for this behavior.

[IDA_ML]

In fact 10-12bit is a fascinating feature for 5d2 and I'm watching  every reddeercity's  post with great attention.

So am I.  I have the 7D and the footage that it provides at 2520x1200 resolution and 10 and 12 bits is absolutely gorgeous.  It has this very exciting film look in it that, in my opinion, only the 7D has.  Don't know about 5DMkII.  I do hope that Reddeercity, with the help of other developers,  succeeds with his most valuable work in porting 2,5, 3 and 4k-crop recording to the 7D.  I keep my thumbs pressed and watch this development with great anxiety!

[reddeercity]

Being spending time working new angles to get UHD cleanly , I thought I would try and reproduce waza57 experiment in 3x crop mode which he got 2880(V)
-- to date I still can't reproduce his work (I need his variables for reg's 6084 & 6088) . In any case I pushed on ,
I being experimenting with the dm-spy build (has a menu option to select reg's and change them with half shutter button)
-- so find the reg you what to change e.g. (c0f06084 , etc. ....)


Play around with 6084 & 6088 , I noticed that in 3x crop if I change the reg 6084 from 30036(default) to 30037 or 30038 I can chose extended rez's e'g' "3520"
but not 30036 thou the image is distorted at reg 30038
3520x1076

I still can't save a dng/mlv at this point though I can dump the Vram and get images from Liveview.
Things took a turn for the better when decided to check out the reg's in 10x zoom . I noticed that most the reg's are
the same as 3x crop as far as 6084 & 6088 are concerned but
10x reg's from dm-spy

Code Select Expand

[REG] @@@@@@@@@@@@ Start ADTG[1]
[REG] ADTG:[0x105f0301]
[REG] ADTG:[0x10610301]
[REG] ############ Start CMOS
[REG] CMOS:[0x20b]
[REG] CMOS:[0x1e6a]
[REG] CMOS:[0x210e]
[REG] CMOS:[0x3005]
[REG] CMOS:[0x4242]
[REG] CMOS:[0x5c01]
[REG] @@@@@@@@@@@@ Start ADTG[2]
[REG] ADTG:[0x14e00a]
[REG] ADTG:[0x150609]
[REG] @@@@@@@@@@@@ Start ADTG[3]
[REG] ADTG:[0x9011c]
[REG] ADTG:[0xb811c]

3x crop (5x zoom)

Code Select Expand

Start ADTG[1]
[REG] ADTG:[0x105f0301]
[REG] CMOS:[0x20b]
[REG] CMOS:[0x1e6a]
[REG] CMOS:[0x210e]
[REG] CMOS:[0x3005]
[REG] CMOS:[0x4242]
[REG] CMOS:[0x5c01]
Start ADTG[2]
[REG] ADTG:[0x14e00a]
[REG] ADTG:[0x150609]
Start ADTG[3]
[REG] ADTG:[0x93d1a]
[REG] ADTG:[0xbbd1a]

the differences is at "ADTG[3]"
0x9011c & 0xb811c (10x)
0x93d1a & 0xbbd1a (3x)
here the 10x3.5k_dm_spy_dm-0006.log & 10x3.5k_digic11.LOG , why I'm I calling this 3.5k LOG's ?
Will I can chose up to 3520 (V) in MLV_lite  right out of the box without any modification's to any reg's and I get a clean Image from vram for Liveview (still no dng , "raw error")
10x zoom mode Liveview dump , I selected 3520 in mlv_lite

Original LV-000.422 at 3520x1076



Original LV-001.422



So I know that 10x zoom is disabled in raw.c if it's enabled can it record the 10x zoom Liveview ?
I think at 10x zoom is really  1:1 pixel , if not how could I select 3520 ? I notice in the dm-log that there is
2 different zoom mode for raw zoom mode "1" (3xcrop) & zoom mode "2" (10x zoom)
part of the log the deal with 10x image process

Code Select Expand

63A82> LiveViewMg:ffa09310:99:02: StartImagePass_x10
63AFC> LiveViewMg:00096224:00:00: *** StartEDmac(0x12, 0x2), from ffa093e8
63B2E> LiveViewMg:00096790:00:00:     addr 1b07800, ptr 1b07800, size
63B9F> LiveViewMg:00096224:00:00: *** ConnectWriteEDmac(0x5, 0x0), from ffa09418
63BFB> LiveViewMg:00096224:00:00: *** RegisterEDmacCompleteCBR(0x5, 0xffa08fa0 "WriteEDmacCrawCompleteCBR_x10", 0x0), from ffa09428
63C56> LiveViewMg:00096224:00:00: *** StartEDmac(0x5, 0x1), from ffa0944c
63C81> LiveViewMg:00096790:00:00:     addr 43e28a4, ptr 43e28a4, size
63CBE> LiveViewMg:ffa0946c:99:02: StartImagePass_x10 CrawAddr : 443e28a4 / KindOfCraw : 0
63D65> LiveViewMg:00096224:00:00: *** StartEDmac(0x8, 0x1), from ffa096b4
63D97> LiveViewMg:00096790:00:00:     addr 44afb4, ptr 44afb4, size
63DD9> LiveViewMg:ffa09df4:99:02: StartIntermediatePassMagnify Addr : 0x50000080
63E3E> LiveViewMg:00096224:00:00: *** StartEDmac(0x0, 0x0), from ffa09e18
63E66> LiveViewMg:00096790:00:00:     addr 10000080, ptr 10000080, size
63ED1> LiveViewMg:00096224:00:00: *** ConnectWriteEDmac(0x11, 0x19), from ffa0b5f4
63F04> LiveViewMg:ffa0b63c:99:02: StartQuarkYuvPass 0x5C578400
63F5C> LiveViewMg:00096224:00:00: *** StartEDmac(0x11, 0x2), from ffa0b660
63F82> LiveViewMg:00096790:00:00:     addr 1c578400, ptr 1c578400, size
6422E> LightMeasu:00096224:00:00: *** SetHPTimerAfter(0x61a8, 0xff86727c, 0xff86727c, 0x0), from ff867310
65BF4> **INT-6Ah*:00096840:00:00: *** TryPostEvent(LiveViewMgr, 0x2, 0x0, 0x0), from ff8d8828
65CBA> **INT-6Dh*:ff9a407c:00:02: [ENG] WriteDMACInterrupt(5)(0x2)(0)
65D01> **INT-6Dh*:00000558:99:02: WriteEDmacCrawCompleteCBR_x10
65D30> LiveViewMg:00096098:00:00: *** LVState: (2) --2--> (3)          ff8e17a8 (x=6bac4c z=0 t=0)
65D73> LiveViewMg:ff8e17c4:98:02: lvVDInterrupt
65DE1> LiveViewMg:00096224:00:00: *** register_interrupt("HEAD3", 0xd9, 0xff986e2c, 0x0), from ff986ee0
65E59> LiveViewMg:00096224:00:00: *** register_interrupt("HEAD4", 0xe0, 0xff986f4c, 0x0), from ff986f98
65EA9> LiveViewMg:ff8da96c:98:02: GetWbIntegFrameForWb 0
65EDD> LiveViewMg:ff8e8fb4:08:01: lvcaeGetWbIntegFrame(sync)
65F02> LiveViewMg:ff8e8fcc:08:01: WBIntegFrame(WB:0x152a40,FST:0x1529dc)
65F79> LiveViewMg:ffa0adfc:99:02: SetWbFstPasParameter
65FAA> LiveViewMg:ffa0b00c:99:02: SetWbIntegParameter
66012> LiveViewMg:ffa06df8:98:02: ReleaseEngineResource Res:4, Free:fffffffc Count:0
66065> LiveViewMg:ff8e1b48:98:02: DispGain:1024, ISO:89
66091> LiveViewMg:ff8e1b70:98:02: WB 1982 1024 1024 2016(4037)
66110> LiveViewMg:ff8e1718:98:02: setHorizontalIrcutData(po0=100)
66198> LiveViewMg:ff863840:82:02: GetVramSize (PUB)
661E8> LiveViewMg:ff867ea8:82:03: SetPBForLV ZoomMode=2, LVType::Info=0, LVType::DispType=0
6621D> LiveViewMg:ff867ed4:82:02: SetPBForLV (3066)
662D1> LiveViewMg:ff9a77d0:00:01: [CLKSAVER] ��ClockSave Out��
6631E> LiveViewMg:ff8638c4:82:01: ImgDDev SelectParameter DispType=0
66349> LiveViewMg:ff863840:82:02: GetVramSize (PUB)
6637C> LiveViewMg:ff8637e4:82:01: ImgDDev Reg c0f140e0 01b07800
663AE> LiveViewMg:ff8637e4:82:01: ImgDDev Reg c0f140e4 01b07da0
663E1> LiveViewMg:ff86449c:82:02: AsyncEnableImageVBufferForPlayBackAndWait (PUB)
66409> LiveViewMg:ff863fb4:82:01:  RequestNotifyBlank (PRI)

This what I find very interesting

Code Select Expand

661E8> LiveViewMg:ff867ea8:82:03: SetPBForLV ZoomMode=2, LVType::Info=0, LVType::DispType=0
6621D> LiveViewMg:ff867ed4:82:02: SetPBForLV (3066)

ZoomMode=2 so can I assume this is still 14bit raw but out a different channel .
I wonder if SetPBForLV (3066) is the Zoom window for capture ?
as a little side project I'll try to enable 10x zoom recording and see if this work out of the box 
That would be nice to just have to push the zoom button twice (10xzoom) to get UHD , this could change everything I've being work on for the last year.
Is this a "secret" special bug that canon dev. forgot to close ?  in the same lines as the 50d video that wasn't supposes to be active , here's hoping 

[reddeercity]

Same more info on the above post -- I forgot to add the A & B timers info

Code Select Expand

Desired FPS -------23.976(from30)
Optimize for --------- Low Light
Shutter range --------1/24-1/109
FPS timer A -----------610(FT-28)
FPS timer B -----------1630(FT-10)
Main Clock ------------24.00MHz
Actual FPS ------------24.137

I just check a plain ml build and In 10x zoom you can't select anything over 2k so it must be the FPS timer .
It was a kind of a guess factor , I remembered something that a1ex said about A timers need to be low like (540) can't go below 610
and b timer need to be around 1600-1700 I think , so I play around to get it close and it appears to work in 10x zoom


Also I think 3x crop_mode is a resample of FHD and there's some pixel binding going on , I took 3 screen shots , first FHD then 3x cropmode (5xzoom) and last 10x zoom .
1:1 FHD (line skipping)

3x crop_mode (5x zoom)

10x zoom

I still believe 10x is truly 1:1 of the 5616x3744 with some cropping in movie mode and I think the back lcd sceen is mapped to the sensor in 1:1 so just a 720x480 of the 5.6k image
it I use the digital dolly I can scroll to the top of the image  same as  full res image but with a little cropping maybe 16x9 ? not sure.
I do not get the same results if I use the dolly in 3x crop , I'll post a example of this later on this week .

[reddeercity]

One more quick little post , while in 10x zoom I did a image dump at 2880 & 3520 (V)
and it seem that the height is being increased in 2880 not sure why ,
maybe the A & B timers are close in tune with 2880 then 3520 (V) .

2880(V)3520(V)

[reddeercity]

More research in to 10x Zoom to understand if this mode is really 1:1of the sensor or just a resample :
Adjusting the A & B timers are giving my some very interesting results ,
I started with 29 from 30 fps in "low jello" mode , A-timers = 610 (FT-2018) was not changed , I started B-timer at 2333 (FT+ 977)
and increase to 2575 (FT+1219) . In mlv_lite (raw video) set resolution to 3520 (V)

A-timer 610 (FT-2018) B-timer 2333 (FT+977)


A-timer610(FT-2018)-B-timer2400(FT+1044)


A-timer610(FT-2018) B-timer2450(FT+1094) 


A-timer610(FT-2018) B-timer2500(FT+1144)


A-timer610(FT-2018) B-timer2525(FT+1169) 


A-timer610(FT-2018) Btimer2550(FT+1194)


A-timer610(FT-2018) B-timer2575(FT+1219)


A-timer610(FT-2018) B-timer2581(FT+1225)


Now do I have the full res 5.6K image ? I need to apply this with the adtg_gui module so I can adjust the cmos reg's
I still don't know what size the buffer is in 10x Zoom .

This is what the default Liveview Image dump in 10x zoom looks like , I use a plain nightly ML build

[reddeercity]

Continue on to research 10x zoom with ADTG_GUI "ISO_Resarch_Branch"
finding the same results with Frame timers A & B as with dm-spy .
Started with 23.976 from 30 fps
default  A & B timers for 10x zoom

Code Select Expand

A-timer 610(-22)
B-timer 1640 (x1.28)

Code Select Expand

A-timer 610(FT-2018)
B-timer 1715 (FT+75)
Here the LV-005.422 for those timer , getting full res Liveveiw just like the above post .

Loaded the Edmac.mo and it seems that 10x zoom using the same buffer as 3x crop
@a1ex , the math for "B"timer works 1640(base) +75 = 1715 , but for "A" timer 610 seem to be unchanged even though I have (-2018) so with the math that's "-1408" is this being
applied ? do I have to modify them though the reg's (head timers) ? to get 540