3K/UHD 5D2 Raw development and Other Digic IV Cams

[ilia3101]

@ Ilia3101 here you go Source Code with modification magic-lantern-adtg-gui-4k-test_source_code.zip

Thanks a lot, I'll see if I can come up with any useful changes.

Can't save a dng yet , tried a image dump I can save the liveview422 see  post #90  here is the original LV dump 422 image  LV-002.422 in Black & White ML perview  I guess I should correct my self it did save a DNG file but it was "0" bytes and there was a HD Liveview dump image HD-002.422 which was normal looking .
I think to clean it up it could be as easy as fps A & B timing  or even the  Cmos  register , once there a clean LV image with ML B/W preview I think it may save a dng dump , there a raw record error when I try to save .raw file (Yes I'm still using raw 1st version not mlv lite , it's very basic just record raw once I have something stable I'll move to MLV Lite)

I have an idea why the preview is all noise: The 3K preset adjusts the preview parameters as well as raw parameters, so image framing is correct on 5D3 at the 3K res(is this true?), so maybe it ruins the preview on 5D2.
I'll try and find where the presets are in the code and comment out the changes in it that are irrelevant, maybe the image will be clear then.

@ Ilia3101 I will also PM you with the test 3.5k build I was using to do my test  that I posted in #90 , I don't what to post it publicly so it get out in the wild.

Thx

[reddeercity]

A small correction I did not use the 3x crop buffer in raw.c

Code: [Select]

#define RAW_LV_BUFFER_ALLOC_SIZE (2040*1267) it's the 1:1 buffer , just look at post#90 in the Edmac screen shot on EDM#5 , address "43e28a4" 3570x1267 (3570/14*8 = 2040) . You could try the 3x crop buffer which is 4046 x 1127 so (4046/14*8 = 2312)

I have an idea why the preview is all noise: The 3K preset adjusts the preview parameters as well as raw parameters, so image framing is correct on 5D3 at the 3K res(is this true?), so maybe it ruins the preview on 5D2.
I'll try and find where the presets are in the code and comment out the changes in it that are irrelevant, maybe the image will be clear then.

Look at this for help adtg_gui.c-1127 and also this adtg_gui.c-1343

[reddeercity]

Found some helpful information for "CONFIG_EDMAC_RAW_SLURP"  treatment on 5D2 , as many times I read over this code I never see this 
I think this is a better way then "RAW_LV_BUFFER_ALLOC_SIZE" but not sure if this works on digic 4 cam's  so far it works with digic5 ,
I'll give it try , this is the link to the code starts at line 407 allocate-raw-lv-buffer#Lsrc/raw.cT407

Code: [Select]

408  * How to find buffer dimensions for CONFIG_EDMAC_RAW_SLURP:
409  * - Go to LV and use lv_save_raw
410  * - Check the RAW_LV_EDMAC debug info
411  *  - Suppose it reports W: 0xA3A H: 0x3C7 (taken from 1100D in LV mode)
412  * - EDMAC W is the number of bytes per line
413  *   - The W resolutions is computed as: W * 8 / 14 (raw buffer is 14 bits per pixel)
414  *   - Thus 0xA3A  8 / 14 -> 1496 pixels
415  * - EDMAC H is the number of "jumps"
416  *   - The H resolutions is H + 1 -> 0x3C8 -> 968 pixels
Edit: This will have to be add to "platform / 5D2.212 /  internals.h"

Code: [Select]

/** this method bypasses Canon's lv_save_raw and slurps the raw data directly from connection #0 */
#define CONFIG_EDMAC_RAW_SLURP not to sure about "connection #0" thou -- this maybe a wild goose chase

[ilia3101]

Haven't really managed to get anywhere
I tried changing the presets, separating the 3k one out in to 2 parts, but didn't really help.

I suspect I might have the same exact 'soft brick' you complained about before, as I had tried out your changes a while ago and seen results, now I'm not seeing anything.

Every time I try and do anything with magic lantern I always slow down eventually and give up for a bit, then I try again and the cycle repeats

Interesting about the SLURP thing though.

[a1ex]

not to sure about "connection #0" thou -- this maybe a wild goose chase

Connections are probably hardcoded (or configured in a way we don't currently understand) to various image processing modules. So far, all the D4 and D5 models get the LiveView RAW data from connection 0. Connection 6 and 7 are pass-through (whatever you transfer there using a Read EDMAC channel will be copied on the other side, to a Write EDMAC channel configured for the same connection).

See this diagram (from http://www.magiclantern.fm/forum/index.php?topic=18315.msg188630#msg188630 )



A read EDMAC channel will read the data from RAM and send it to some image processing module.

A write EDMAC channel will get the data from some image processing module and will write it into RAM.

The data can be read via some input module (such as DSUNPACK, ADUNPACK, UNPACK24, or others - possibly unnamed), where you can configure the input bit depth. In this case, the input stream can be 10-bit, 12-bit, 14-bit or 16-bit, configured using DSUNPACK_MODE / ADUNPACK_MODE / UNPACK24_MODE / 0xC0F371FC / etc. In this case, the image processing module that actually does the work (e.g. JPCORE) probably receives normalized data.

A similar process happens on the output side, where a PACK module is used (PACK16, PACK32). Remember the PACK32_ISEL and PACK32_MODE:
- PACK32_ISEL probably means "wire the input of PACK32 module to whatever other image processing module that outputs Bayer data, in various places in the pipeline";
- PACK32_MODE configures the output bit depth of whatever image data arrives to the PACK32 module (10/12/14/16).

Currently, the uncompressed bit depth selection is done in raw_lv_request_bpp (raw.c, crop_rec_4k branch).

Experiments on the above can be made on existing code that's known to work (raw_twk for digic 4/5, EekoAddRawPath for digic 5), or on FA_MaxSelectTestImage / FA_SubtractTestImage (low-hanging fruit for understanding the image processing modules).

One interesting note from the crop_rec_4k thread, where I've implemented the 10/12-bit lossless compression by darkening the input raw data (so the input and output would still be 14-bit, but there will be fewer levels actually used by the image - as many as a 12-bit or a 10-bit image). For lossless compression, the entropy of a 10/12-bit stream is similar (maybe identical?) to the entropy of a 14-bit stream with each value shifted by 4 or 2 bytes (integer division by 16 or by 4).

How does that work?

raw.c:raw_lv_request_digital_gain:
- lv_raw_gain is written to SHAD_GAIN_REGISTER
- RAW_TYPE_REGISTER is set to 0x12 (DEFCORRE)
- this image happens to be scaled by digital ISO gain and is not affected by bad pixels.

When the digital gain is not set, RAW_TYPE_REGISTER is set to CCD (probably the first stage where the raw data gets in the digital domain).

For a better understanding, set RAW_TYPE_REGISTER to DEFCORRE (0x4 on digic 4) without overriding SHAD_GAIN, and notice what happens at ISO 320 vs 400. Repeat for RAW_TYPE_REGISTER set to CCD. Then start overriding SHAD_GAIN with any values you want, even something like this:

Code: [Select]

if (get_halfshutter_pressed())
{
    EngDrvOut(SHAD_GAIN_REGISTER, rand());
}


Now look at the problems that appeared from this change (the 10/12-bit lossless implementation):

- First of all, the "slurp" EDMAC channel (the one that writes the raw data into memory) had to be configured for exact resolution; the autodetection from raw_lv_get_resolution (0xC0F0680x/0xC0F0608x) is not exact - it's often off by one on the vertical direction, although the exact reason is unknown. Being off by one on 5D3 resulted in the raw data being correct only on every other frame (although I don't really understand why that happens). The issue was only with RAW_TYPE_REGISTER set to DEFCORRE, but it all worked fine when it was set to CCD.

- Next, take a look at this bug: in a video mode with increased vertical resolution, the darkening works only on the top side, on an area equal to default Canon resolution in that video mode. That means, we have to reconfigure some more registers - probably in the image processing pipeline, all the way from CCD (sic) to DEFCORRE. Which ones? I don't know - couldn't find them in adtg_gui. I hope to find them by emulating the LiveView in QEMU, but that's going to be a really long journey.

That's why, for now, the 10/12-bit lossless compression only works in video modes with unmodified resolution (plain 1080p, plain 720p and 5x zoom).

[reddeercity]

@Ilia3101 -- baby steps  just pick one thing to focus on and learn as much as you can , look closer at what the code is doing in the 3k 5d3 preset , there a lot of clues there even thou it for the 5d3 the 5d2 share some similarities in this respect
it may not look like there much progress right now but we are laying out the frame work for more then 3.5k video this will also lead to FHD 10/12bit and even in time compressed raw .

@a1ex thanks for the lesson , things are getting clearer .

[Wayne H]

@Alex that is an excellent and highly detailed explanation on how you Magic Lantern magicians are squeezing 3.5K and 4K out of the 5d3,

I was wondering a while back why the 12bit and 11-8bit lossless options were only available on the 3.5K mode, and not in the other UHD and 4K modes, but after doing some magic lantern reading, i quickly understood that the 3.5K mode works differently to all the other high resolution modes,

Makes me appreciate just how much work you guys do in your own free time, LIVEVIEW is incredibly complex. (it took Canon 6 months just to get the 5d3 to spit out clean and mirrored HDMI via a firmware update) AMAZING WORK GUYS.

[reddeercity]

Ok I'm going to try and get "CONFIG_EDMAC_RAW_SLURP" for 5D2 working ,  from this allocate-raw-lv-buffer#Lsrc/raw.cT407 tell how to go about it , but I still don't understand .

Code: [Select]

 
408  * How to find buffer dimensions for CONFIG_EDMAC_RAW_SLURP:
409  *   - Go to LV and use lv_save_raw
410  *   - Check the RAW_LV_EDMAC debug info
411  *   - Suppose it reports W: 0xA3A H: 0x3C7 (taken from 1100D in LV mode)
412  *   - EDMAC W is the number of bytes per line
413  *   - The W resolutions is computed as: W * 8 / 14 (raw buffer is 14 bits per pixel)
414  *   - Thus 0xA3A  8 / 14 -> 1496 pixels
415  *   - EDMAC H is the number of "jumps"
416  *   - The H resolutions is H + 1 -> 0x3C8 -> 968 pixels

from line 411 -- W: 0xA3A H: 0x3C7  how do I figure out what these compute to , I must be missing something here . Also how do I implement "lv_save_raw" from line "409" there must be some code I missed . I remember some where it was posted but not sure where , @dfort didn't you go though this with the eosm ? I can't seem to find it , if not I have to do more searching .

[dfort]

@dfort didn't you go though this with the eosm ? I can't seem to find it , if not I have to do more searching .

I can't find it either. This is the closest that I could come up with: http://www.magiclantern.fm/forum/index.php?topic=5601.msg179673#msg179673

Maybe write something in dubug.c calling lv_save_raw and print it on the screen or console? There's a hint in lv_rec.c:

Code: [Select]

    /* this causes the function to hang!? */
    if(data.options.rawMode)
    {
        //~ bmp_printf( FONT(FONT_MED, COLOR_WHITE, COLOR_BLACK), 30, 20 * yPos++, "Make sure you ran call('lv_save_raw')");
        call("lv_save_raw", 1);
        msleep(200);
        raw_update_params();
    }

The rest of it makes sense to me:

0xA3A (hex) = 2618 (decimal)
2618 *  8 / 14 -> 1496 pixels

[reddeercity]

I can't find it either. This is the closest that I could come up with: http://www.magiclantern.fm/forum/index.php?topic=5601.msg179673#msg179673

Maybe write something in dubug.c calling lv_save_raw and print it on the screen or console? There's a hint in lv_rec.c:

Code: [Select]

    /* this causes the function to hang!? */
    if(data.options.rawMode)
    {
        //~ bmp_printf( FONT(FONT_MED, COLOR_WHITE, COLOR_BLACK), 30, 20 * yPos++, "Make sure you ran call('lv_save_raw')");
        call("lv_save_raw", 1);
        msleep(200);
        raw_update_params();
    }


@dfort thanks , that will help a lot

The rest of it makes sense to me:

0xA3A (hex) = 2618 (decimal)
2618 *  8 / 14 -> 1496 pixels

I never thought hex , I light bulb just came on as soon as I that.

[dfort]

It might help if you read how dmilligan found the addresses for the EOSM starting around here:

https://www.magiclantern.fm/forum/index.php?topic=5601.msg175969#msg175969

Yeah, it is confusing and doesn't translate across platforms but it might get you a step closer.

There's also several methods of finding some of this stuff. It appears that in the code snippet you posted it is showing, "How to find buffer dimensions for CONFIG_EDMAC_RAW_SLURP" using the 1100D as an example. The size is 1496 x 968. Isn't that the size of the full raw buffer?

Another way of getting the full raw buffer is to set the camera's movie settings to the mode you want (say 1920x1080/24 in the Canon menu) and while the camera is in movie mode shoot a simple silent still DNG. Take that DNG and run it through dcraw using the -4 and -E options. The output file (in .pgm format) will be the size of the full raw buffer. That's what I had to do when working out the focus pixel maps for cameras that show focus pixels in raw video.

To get the entire raw buffer from a DNG, including OB area: dcraw -4 -E
To get the active area only: dcraw -4 -D

That's the second time today that I quoted this tip. OB means out of bounds. There's actually an area that gets cropped out when viewing the DNG of the full raw buffer and using that -E option will show you the entire raw buffer.

[reddeercity]

@dfort thanks , I'll give that a go

[reddeercity]

So some more tests -- with these latest attempt I  see some progress in  Liveview  @  3584x1068



The first image with the pink noise bar across the top of liveview was with a half shutter press and the second was with Zebras & Focus Peck enable -- see image below



So this leading me to believe the there is either a vertical resolution out of sync or fps timers .
I guess I should say what my configuration was  I'm still using my custom build with modified Liveview buffer as noted in other post here on this thread .
Camera setting are real basic , Preview I use "ML B/W preview for liveview , frame override enable to 23.976 . I only loaded adtg_gui.mo , crop_rec.mo , raw_rec.mo (I use .raw format for testing)

I notice something very interesting , on first boot I enable adtg_gui.mo & the rest -- mess with  cmos regs etc. ... when I got the image all messed up I did a reboot to clear cmos registers and that's when it happen  it automatically boot to my custom buffer , now I was back to 3584x1068 -- the strange thing adtg_gui.mo was not enable just crop_rec 1:1 , there was no memory patches . That what the screen shots are from , the non enabled cmos registers , just crop record . The code from crop_record comes from the 5D3 and I have not change it at all , I should mention before I rebooted and adtg_gui.mo enabled I had use the 5d3 3K preset so unless I rebooted to quickly and those cmos registers so how stayed in memory I can't seeing it but you never know. So as far as I know all this is done with the custom liveview buffer I'm using

Code: [Select]

#ifdef CONFIG_5D2
#define RAW_LV_BUFFER_ALLOC_SIZE (2040*1267)
#endif
Plus the other changes I made to the code here
Setting:
crop_rec.cfg

Code: [Select]

# Config file for module crop_rec (CROP_REC.MO)crop.preset = 1magic.cfg

Code: [Select]

# Magic Lantern Nightly.2017Apr14.5D2212 (bd8e89d9dccf+ (iso-research))
# Built on 2017-04-14 03:42:23 UTC by ml@ml-pc
# Configuration saved on 2017/10/29 19:17:40
beta.warn = 29
menu.first = -3
movie.log = 1
rec.notify = 0
enable-liveview = 2
fps.override.idx = 11
battery.drain.rate.rev = 69
hist.log = 0
spotmeter.draw = 0
zebra.draw = 0
disp.mode.x = 149
raw_rec.cfg

Code: [Select]

# Config file for module raw_rec (RAW_REC.MO)
raw.video.enabled = 1
raw.res.x = 20
raw.dolly = 1
raw.preview = 2
raw.warm.up = 2
I also tested this with HDMI output connected my atomos ninja hdmi hard drive recorder , uploaded to my bitbucket downloads link below
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/00150_H264.mp4
there's some shearing etc. ... but did not crash the camera , one thing I notice thou was the ml scopes where blank no feed back for the image .
at the end of the video I try to record raw video but got a "Raw error" this was expected .
But in the next file after I cycled the hdmi connection I then had the ML scopes came back see it in the link below
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/00153_H264.mp4

So I can also take a CR2 photo in this configuration and it all right no problem below is a short video that I take with my iPhone with me pressing the shutter fully to get a cr2 photo link below also
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/IMG_2273.MOV
https://bitbucket.org/reddeercity/magic-lantern_10-12bit/downloads/_MG_8668.CR2

I notice as I panned the camera left to right the preview that was ml b/w the distortion would move or pulse and of course with half shutter I get color distortion , but it's not a static frozen image

[reddeercity]

I notice there one line of pixel you can see here the LV-004.422 image
Liveview dump

[reddeercity]

Going over ARM Configuration Bus Access to Peripherals which I been led to believe to be in the 5d2 and all digic 4



Look at the red hi-lighted box , that looks like the resizer registers I have been adjusting in my earlier tests I posted with adtg_gui.mo 
the IPIPE Registers are interesting DM365 IPIPE register set

....These include for example register offset ranges 0x44-0x1CC and 0x1F0-0x228.
According to the code these address ranges contain registers for OTF Defect Pixel Correction, Lens Shading Correction, Noise Filtering and CFA interpolation. ....

will that seem to be dslr stuff    , need to dig deeper .

[reddeercity]

Could this be the Sensor Interface (ISIF) – RAW Data Processing Flow for canon digic 4 or all eos ?



http://www.ti.com/lit/ug/sprufg8c/sprufg8c.pdf

I think this my be a very importance pdf with registers that could map out to what is needed for development


Edit: interesting , Image Pipe Interface with dark frame subtraction ?

[ItsMeLenny]

I have nothing valuable to add I just wanted you to know I get so excited when there's a new reply in this thread.

[DeafEyeJedi]

I have nothing valuable to add I just wanted you to know I get so excited when there's a new reply in this thread.

+1 

[70MM13]

I have nothing valuable to add I just wanted you to know I get so excited when there's a new reply in this thread.

Agreed!

[domo94]

I am coming from a 550d thread.

To help with further development with 550d, and hopefully to get this software ported with Canon 7D to get the most consistent and solid 10bit 1080p recording I need!

How can I help?

[dfort]

There are lots of parts to this.

There's the 10bit/12bit part that apparently requires either CONFIG_EDMAC_RAW_SLURP and/or CONFIG_RAW_LIVEVIEW but that can't be right because if you notice that there is an experimental build for the 1100D that supports 10bit/12bit using a branch where RAW_SLURP isn't defined on that camera. However, if RAW_SLURP is required that has been cracked on the 1100D in the allocate-raw-lv-buffer branch.

Then there's lossless compression where it seems that we found all the stubs for the 7D but haven't figured out how to get it working properly yet.

Finally, there's the crop_rec module. Several cameras are working with the 3x3 mode which allows shooting nice quality 60fps raw video without that terrible line skipping that shows lots of aliasing artifacts in the 1280x720p mode. However, to get to the juicy stuff that includes 4k raw video will require a lot more work that so far is only supported on the 5D3.

Attacking this all at once is overwhelming, believe me I tried. I'd suggest taking one of these topics and start reading from the beginning like a novel to see how these puzzles were eventually solved.

[reddeercity]

Actually it's really ISO_Research Branch with crop_record that's it . Here  is my source code I'm working from 
If you read my thread from the start it's all explained.
10/12bit & lossless compression is not part of this right now so has no bearing on Digic IV UHD/3.5K raw video development ,
Thou I will implement 10bit (as this work on 5D2) after I can get a unscrambled stable image  (very close now)

Digic IV cams work with RAW_LV_BUFFER_ALLOC_SIZE  as long as you find the right buffer e.g. 5D2

Code: [Select]

#define RAW_LV_BUFFER_ALLOC_SIZE (2040*1267)I would suspect 550d would be the same but for 7D it may not because of the dual cpu there are more difficulty

I am coming from a 550d thread , To help with further development with 550d

Thanks , look here Greg was helping with 500d with some promising result but stop.
So search Greg's posts on my thread & on the forum he has some very good info to share e.g. 500D, trying to increase the resolution of the video raw (crop mode).
In fact from his initial experiment's a1ex started the whole 4K/UHD for 5D3  crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView he credited in the first post .

If you can compile the source code (you need a working environment I recommended a prebuild VM of Ubuntu  Quick Guide and Solutions for VirtualBox that the one I use)
can be ran on Windows or MAC , It's all setup nothing to be configured --install VirtualBox for your OS and start working.
The other useful thing is to decompile your ROM Dump files look here for help in disassembling.

...... ported with Canon 7D to get the most consistent and solid 10bit 1080p recording I need!

This is off topic but here you go 
Will there is 10bit 1080p already with the 7D with the experimental 10/12bit build , but only works in 3x cropmode ,
can be downloaded from my bitbucket account 10bit_12bit_raw_2016Dec01.7D203.zip
1:1 liveview produces corrupted image every other frame usually the top 1/3 , that's why it's not in the main build area and only works in cropmode with audio by the way 
 

[dfort]

Yes, I've been following this topic and I am able to compile Magic Lantern and disassemble ROMs. In fact I compiled that 7D build you pointed to on your last post exactly one year ago. I took it down because 10bit/12bit wasn't supported on the 7D (or 5D2, 50D, 500D, 550D).

According to the experiments download page "Only models with CONFIG_EDMAC_RAW_SLURP are compiled." However, I thought it was worth noting that there is a 10bit/12bit build for the 1100D available for download. It is being compiled on the raw_video_10bit_12bit branch and if you look through the code you'll see that CONFIG_EDMAC_RAW_SLURP isn't defined for the 1100D in that branch.

Digic IV cams work with RAW_LV_BUFFER_ALLOC_SIZE  as long as you find the right buffer e.g. 5D2

Code: [Select]

#define RAW_LV_BUFFER_ALLOC_SIZE (2040*1267)

You defined RAW_LV_BUFFER_ALLOC_SIZE in raw.c but you didn't define CONFIG_EDMAC_RAW_SLURP in internals.h. You are working on the iso-research branch but for this experiment you will probably have better luck using the allocate-raw-lv-buffer branch. Note that neither iso-research or allocate-raw-lv-buffer has the 10bit/12bit code so you'll need to merge those changes with the raw_video_10bit_12bit branch. [EDIT - Yeah, you're doing all that.] For everything you're trying to do your goal should be to eventually get it working in the crop_rec_4k branch. Good luck!

Note: It would really help if you would update your Bitbucket repository instead of posting a link to your local repository.

[dfort]

Let me backpedal a bit. This thing is rather complicated and hard to grasp all at once.

Instead of blabbing away let me offer a bit of code to find something that might help. A while back I was searching for the SRM_BUFFER_SIZE on some cameras that I didn't have access to so I made a custom build to print it out on the LCD screen. This is something that you will probably need as a1ex noted in this post.

src/exmem.c  (based on the unified branch)

Code: [Select]

static void srm_malloc_cbr(void** dst_ptr, void* raw_buffer, uint32_t raw_buffer_size)
{
    if (!srm_buffer_size)
    {
        /* we can't tell how much to allocate; the allocator tells us */
        srm_buffer_size = raw_buffer_size;
    }
    else
    {
        /* it should tell us the same thing every time */
        ASSERT(srm_buffer_size == raw_buffer_size);
    }
   
    /* return the newly allocated buffer in the output variable */
    *dst_ptr = raw_buffer;
   
    /* announce it's done */
    give_semaphore(srm_alloc_sem);
   
    bmp_printf(FONT_LARGE, 50, 300, "SRM_BUFFER_SIZE 0x%x", srm_buffer_size);
}

The only change is that last line in order to print the SRM_BUFFER_SIZE on the screen when shooting a simple silent DNG, like this on the 7D:



So this can be added to the consts.h file (malloc section):

Code: [Select]

#define SRM_BUFFER_SIZE 0x1F80000   /* print it from srm_malloc_cbr */
@tecgen found it for the 550D - 0x1F68000 - I should add that he found it on his own.

He said he was going to try to find it on the 50D and 5D2 but I haven't seen any updates from on this. So how about we look for these? I put up builds for these cameras on my downloads page so we can document the SRM_BUFFER_SIZE. If anyone wants to look for it on a camera that isn't listed I'd be glad to post a build or it can be compiled from the unified-find-SRM_BUFFER_SIZE branch in my repository.

I also did a screenshot of edmac on the 7D and see something very different from the 5D2. Maybe the channels we want are running on the master processor so they aren't visible on the slave? I don't know.

[Andy600]

@dfort - Just ran your build on the 50D and the SRM buffer size shows as 0x1b14000

useful bit of code