3K/UHD 5D2 Raw development and Other Digic IV Cams

[ilia3101]

I can finally 'help' with getting 3k on the 5D2! thx dfort

Code: [Select]

0x25d0000
37MB buffer?

[reddeercity]

From adtg_gui.c-1121 This is the 3K 5d3 preset I use on my 5D2

Code: [Select]

static int res3k_reg(int reg)
{
    if (regs[reg].dst == 0xC0F0)
    {
        switch (regs[reg].reg)
        {
            case 0x6804:                /* C0F06804 - raw resolution */
                return 0x52801AB;       /* from 0x528011B -> 3072px in raw_rec */
with buffer

Code: [Select]

#ifdef CONFIG_5D2
#define RAW_LV_BUFFER_ALLOC_SIZE (2040*1267)
#endif
that end up to be 3584x1068 .
A hint from nikfreak from here I see where I my have a problem , just dawn on me
that the registers are a little different 
•C0F06804 (Digic5)
•C0F06084 / C0F06088 (Digic4)

Dam Dyslexia 

As seen in this piece of code adtg_gui.c-126 full res liveview uses C0F06084

Code: [Select]

   * 5D2 photo:
     * 0xC0F06088.hi - 0xC0F06084.hi is the vertical resolution (3804)
     * 0xC0F06084.hi shifts the image in one-line increments, so it must be the first scanned line
     * (0xC0F06088.lo - 0xC0F06084.lo) * 2 is the horizontal resolution (5792)

So it look like I need to make a few changes to my source code to take that in account .
Stay tuned

[DeafEyeJedi]

This is all smoking hot... keep it up guys!

[reddeercity]

Something extra to add , I thing I know way I'm lock to 3584x1068 , I see that cmos[1] is set to 0x802 and if
I convert the hex to decimal so 0x802 = 2050 *14 /8 = 3587.5 -- my custom buffer 2040x1267 , so I my change the buffer to 2050 instead of 2040 and see what happens

[IDA_ML]

A very hot topic, indeed!  I keep checking for new posts at least twice a day.  Am a right by thinking that there might be some hope for the good old 7D too?  Excellent work, guys!  I keep my thumbs pressed!

[dfort]

Checking it twice a day? Really?

Of course it would be much easier if we all upgraded our older cameras for something not so old but not too new either.

Anyway--I did some playing around with the 7D over the weekend. One of the things I wanted to see was all the changes that @reddeercity actually did so I found which changeset he started with and managed to put it into a branch in my repository. This should provide a clearer picture of what is going on. I also made a dummy pull request of the stuff I worked on which was trying to get CONFIG_EDMAC_RAW_SLURP working. I used the allocate-raw-lv-buffer branch because it was pointed out somewhere to look at how the 1100D was done done on this branch but it didn't compile! Turns out that it still had a problem in the code that was resolved a while back. BTW, the 1100D is an EVF_STATE camera while the 50D, 5D2, 7D and other cameras we're looking at are LVState cameras. None of the LVState cameras have CONFIG_EDMAC_RAW_SLURP working--yet. Anyway, here's my work in progress:

https://bitbucket.org/daniel_fort/magic-lantern/pull-requests/15/searching-for-config_edmac_raw_slurp-on/diff

There are lots of pieces to this puzzle and it would be great to get at least some of the answers we found cataloged in some way. David (a.k.a. reddeercity) how about modifying your first post and adding the SRM_BUFFER_SIZE for the various cameras that we have this resolved? At some point down the line we'll need this and having them in one place instead of scattered throughout various posts would be much easier to work with.

Another thing worth sharing are these screenshots I did on the 7D.

First the default:


Next, what happens after recording a bit of raw video again at the default raw video settings:


Of significance is what happens with channels 6 and 10. Note this in the code:

edmac-memcpy.c

Code: [Select]

#elif defined(CONFIG_7D)
uint32_t edmac_read_chan = 0x0A;  /*Read 0x19 0x0D 0x0B 0x0A(82MB/S)*/
uint32_t edmac_write_chan = 0x06; /* Write 0x5 0x6 0x4 (LV) */

Channel 6 is the write channel and 10 (0x0A hex) is the read channel.

Now let's put the camera in zoom mode, max out the image size and record a clip--albeit a very short clip.





Pretty cool, eh? (Speaking Canadian.)

Not sure what channel 16 is for but it is interesting that it is the largest and the address didn't change.

[reddeercity]

  Yea , 2 more things -- wear a toque get a beer and I will make you a honorary Canadian 

[reddeercity]

Ok now down to business -- That large buffer is what I would focus on from channel #16 , That's how I choose the 5d2 buffer on channel #5 in the Edmac.
If you navigate  to channel #16 on the 7D I would expect to see a different address then the default in raw_rec.c

Code: [Select]

#if defined(CONFIG_500D) || defined(CONFIG_550D) || defined(CONFIG_7D)
#define RAW_LV_EDMAC 0xC0F26008
#endifThat's why I changed

Code: [Select]

define RAW_LV_EDMAC 0xC0F04508 to

Code: [Select]

define RAW_LV_EDMAC 0xC0F04500Something interesting -- if you take Edmac#16 (on 7D) buffer "4410x1225"
4410/14*8 = 2520
So this could be your custom buffer 2520 x 1127 1225-- this is how I came up with my buffer.

....... at least some of the answers we found cataloged in some way. David (a.k.a. reddeercity) how about modifying your first post and adding the SRM_BUFFER_SIZE for the various cameras that we have this resolved?

Yes I will update the first post with screen shots and buffers information on all digic iv that are being actively be worked on ,
this means any one following this thread with a digic iv cam that don't see your camera on the first post will need to be involved.
At this point the camera's with CF Cards have priority 5D2 , 7D , 50D -- then the 550d & 500d but not limited in that order 

[reddeercity]

Also I use the crop_rec.mo  to get the 3.5k . I enable the 1:1 preset which I sure give me every line etc. ...

Code: [Select]

  switch (crop_preset)
        {
            /* 1:1 (3x) */
            case CROP_PRESET_3X:
                /* start/stop scanning line, very large increments */
                /* note: these are two values, 6 bit each, trial and error */
                cmos_new[1] = (is_720p())
                    ? PACK12(14,10)     /* 720p,  almost centered */
                    : PACK12(11,11);    /* 1080p, almost centered */
               
                cmos_new[2] = 0x10E;    /* read every column, centered crop */
                cmos_new[6] = 0x170;    /* pink highlights without this */
                break;


[IDA_ML]

Checking it twice a day? Really?

Of course it would be much easier if we all upgraded our older cameras for something not so old but not too new either.

O, absolutely!  Of course, you are right, Dfort, it would be much better if we upgraded our cameras to something newer but do we really have that choice?  The only alternative and better model to the 7D that supports Magic Lantern is the 5D3 but how many people on this forum can afford this body and the lenses for it?  I also own the 100D which is a fantastic little ML camera, perfectly suited for travel, especially after the latest ML developments, but it deffinitely is not appropriate for serious work, considering the short battery life, the limited vertical resolution, lack of VAF filter and especially the very annoying "hick-ups" (random jumps throughout the video) in the 5x-magnification mode which makes the camera unuseable in that most important for serious practical video shooting mode.

It may be interesting for you to know that this is not only my opinion.  I have discussed the issue with several  friends and colleagues of mine who use their 7Ds, 5D2s and even 50Ds on a daily bases and they all share these same thoughts.  That is why, the excitement about the latest developments concerning these old Digic IV cameras among the video shooting community is tremendous!  I am sure, Mr. Reddeercity who obviosly loves his 5DMk2 so much, understands what I mean :-))).

And finally, a moral issue that I would like to share here.  You say, upgrading to newer models. But why, if the old models continue working so well, are built like tanks, are extremely reliable and in many practical situations outperform the newer ones?  Why would I want to sell my trusted 7D that I payed so much money for back in 2011 if it still works and looks like brand new and in fact, I have not used up even 50% of its shutter life?  How much does it cost to recycle a 7D and how much energy would it take?  Wouldn't we all make a friendly gesture to our Mother Nature if we use our cameras more consciously and do not upgrade them so often?  In fact, I think that Magic Lantern fulfills a mission in that respect - by giving new life to older cameras it helps protecting our environment.  Another reason why I like it so much and why I am so gratefull to all our developers - the real missionaries!

[dfort]

@IDA_ML - I'm not arguing with you, just hinting that it would be easier if we all had 5D3's. My first ML capable camera was an EOSM. I got it because thought it could do almost everything the other ML capable cameras could do. I does, depending on your definition of almost.

Back on topic--

Something interesting -- if you take Edmac#16 (on 7D) buffer "4410x1225"
4410/14*8 = 2520
So this could be your custom buffer 2520 x 1127 -- this is how I came up with my buffer.

How did you get 1127 from 1225?

What I'm looking for is how to translate this from the 1100D to the 7D and other cameras that use LVState.

Code: [Select]

#ifdef CONFIG_1100D
#define RAW_LV_BUFFER_ALLOC_SIZE (3906*968)
#endif

Note that originally it was coded like this:

Code: [Select]

#define RAW_LV_BUFFER_ALLOC_SIZE (3906*766)
"Blame" shows that G. Condello (a.k.a. @nanomad) wrote that line but I searched all over and couldn't find out how he got the "3906" part. The "968" part is spelled out in this old comment you already posted:

Code: [Select]

408  * How to find buffer dimensions for CONFIG_EDMAC_RAW_SLURP:
409  *   - Go to LV and use lv_save_raw
410  *   - Check the RAW_LV_EDMAC debug info
411  *   - Suppose it reports W: 0xA3A H: 0x3C7 (taken from 1100D in LV mode)
412  *   - EDMAC W is the number of bytes per line
413  *   - The W resolutions is computed as: W * 8 / 14 (raw buffer is 14 bits per pixel)
414  *   - Thus 0xA3A  8 / 14 -> 1496 pixels
415  *   - EDMAC H is the number of "jumps"
416  *   - The H resolutions is H + 1 -> 0x3C8 -> 968 pixels
So is W width and H height of one of the buffers? Maybe but "W: 0xA3A H: 0x3C7" would translate to 2618x967. Note that the 1100D specifications show a maximum video resolution of "720p HD video at 25 or 30 fps" so maybe, just maybe, that's a reason why it didn't even need CONFIG_EDMAC_RAW_SLURP for it to work in the raw_video_10bit_12bit branch.

I also went back into the history of the source code and couldn't figure out when these "W:" and "H:" values were reported in hex but maybe (and this is just an uneducated guess) the values for the 7D should be taken from the maximum size of channel 16:

Code: [Select]

#ifdef CONFIG_7D
#define RAW_LV_BUFFER_ALLOC_SIZE (4410*1226)
#endif

Note the +1 on the "H:" value as noted in that old comment we are referencing.

This seems logical because the 7D can record higher resolution than the 1100D. Then again so many things in ML aren't what they seem so I might be totally off base on these assumptions.

By the way, your calculation of the width of the "custom buffer" matches up with the zoom mode raw buffer. I shot silent stills in various modes and this is what exiftool is showing on the DNG files:

Code: [Select]

mv640
Default Crop Size               : 1736 694
Active Area                     : 26 256 720 1992

mv720
Default Crop Size               : 1736 694
Active Area                     : 26 256 720 1992

mv1080
Default Crop Size               : 1736 1156
Active Area                     : 26 256 1182 1992

zoom
Default Crop Size               : 2520 1200
Active Area                     : 26 0 1226 2520

FRSP
Default Crop Size               : 5202 3466
Active Area                     : 50 158 3516 5360

Note the height of the zoom full raw buffer? Back to my question, "How did you get 1127 from 1225?" Would you agree that it should be 1226? That would mean that your "custom buffer" is really just the zoom mode buffer.

Exercise for other Digic IV cameras: Shoot silent DNG's at various video modes. Just change the Canon movie settings and shoot the silent stills with the camera in movie mode. Don't forget to do one for zoom (a.k.a. 5x crop mode). If anyone wants to try this but hasn't mastered exiftool, post a link to your DNG's and we'll check them out for you.

[a1ex]

RAW_LV_BUFFER_ALLOC_SIZE answered here. You need to allocate space for how much raw image data *you* plan to save from connection #0 (you are reconfiguring the LiveView resolution, after all). For experiments, you could always allocate one SRM buffer, which is the largest contiguous chunk you can get from Canon's memory allocators - a 14-bit full-res uncompressed^*) image won't be larger than that. This is done on 5D3 for full-res LV (crop_rec_4k).

^*) "14-bit uncompressed" is important; it can get larger if you request 16-bit data, or after lossless compression (yes, a compressed image *can* be larger than the input image). Currently there's no known way to store such image in a contiguous chunk of memory. For lossless, it usually works, because - in practice - most of the time, the compressed image ends up smaller. Still, you also have to handle the case where compressed is larger than uncompressed (otherwise you'll overwrite other stuff in memory, sooner than you may think).

1100D *has* CONFIG_EDMAC_RAW_SLURP (check the build log from the Experiments page). Don't remember anyone reporting back for this model, so I don't know whether the 10/12-bit build works or not (or whether to commit the allocate-raw-lv-buffer merge, for that matter).

[dfort]

Thanks for clearing things up though at the same time a few things got muddied up.

1100D *has* CONFIG_EDMAC_RAW_SLURP (check the build log from the Experiments page). Don't remember anyone reporting back for this model, so I don't know whether the 10/12-bit build works or not (or whether to commit the allocate-raw-lv-buffer merge, for that matter).

So this whole thing about using the 1100D as an example might just turn out to be fool's gold? I could have sworn that raw_video_10bit_12bit 1100D/internals.h *doesn't* have CONFIG_EDMAC_RAW_SLURP defined while allocate-raw-lv-buffer 1100D/internals.h *does*. Looked at it again and I still don't see it. I also looked all over the build log and that didn't didn't turn up any nuggets.

However, this turned on a light:

RAW_LV_BUFFER_ALLOC_SIZE is how much we need for a full-res image (max W * max H * 14/8).

Looking at the exiftool output for the active area of a full-res silent image, 5360 * 3516 * 14/8 = 32980080 or 0x1F73C70 hex. Hum, sounds familiar:

Code: [Select]

#define SRM_BUFFER_SIZE 0x1F80000   /* print it from srm_malloc_cbr */
So does that mean something like this should work?

raw.c

Code: [Select]

#ifdef LV_STATE
#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE)
#endif

Haven't tried it and my syntax might be out of whack but that's the idea.

[a1ex]

Another tip: each autoexec.bin has human-readable metadata - open it with nano and check the header. That tells you exactly where it was built from (and whether there were any uncommitted changes).

SRM_BUFFER_SIZE would work if you allocate with srm_malloc_suite. With the malloc wrappers, there will be a small overhead from the memory backend, so... just reduce the size by 0x100 bytes or so. Didn't try, but this should be generic for any model that's going to implement full-res LiveView.

[reddeercity]

How did you get 1127 from 1225?

Sorry , mistake I meant 2520x1225 . But it seems I may be wrong after reading a1ex posts need to investigate more

[dfort]

Don't forget about this:

Code: [Select]

416  *   - The H resolutions is H + 1
That means it should be 2520x1226 and that works but we can go much further than that.

SRM_BUFFER_SIZE... just reduce the size by 0x100 bytes or so.

Did some experimenting and this is working:

Code: [Select]

#ifdef CONFIG_7D
#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0x1000)
#endif

The generic "#ifdef LV_STATE" didn't work and that's the least amount of shaving off of SRM_BUFFER_SIZE that worked.

Let me qualify how well this allocat-lv-raw-buffer experiment is currently going. Up until now I was just checking DNG frames extracted from RAW and MLV files and they were looking good so I thought I'd render a video file and it was doing the old shake rattle and roll. In other words the DNG images looked clean but it appeared to be changing the capture area randomly. I backed out all the changes and guess what, it was still shaking so whatever I'm doing didn't cause this and didn't make it any worse.

So is CONFIG_EDMAC_RAW_SLURP working? Maybe, I don't know. Looks like the thing to do is to switch over to the raw_video_10bit_12bit branch and see what happens.

[reddeercity]

Cool , I'll play with this on 5d2 and see what happens

[reddeercity]

Try out the code on the 5D2 , no problem every thing work normal in 1:1 & crop mode -- recorded 2 short .raw file 

Code: [Select]

#ifdef CONFIG_5D2
#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0x1000)
#endif
At least no ill effects , complied it on the iso_research branch I'm using , but did not load adtg_gui yet to see if it changes anything.

Edit: going though the DM-Spy Log from the 5D2

Code: [Select]

E05D3> GuiMainTas:ffa4522c:18:03: (PUB) SetImageWorkMemory 1:0x40d00000, 2:0x41700000
E060A> GuiMainTas:ffa45274:18:03: (PUB) SetImageWorkMaxPixel W:2200,H:1872,0x5e4340
E0640> GuiMainTas:ffa44db8:18:03: (PUB) SetEffectiveSizeOfJpeg(w:7488 h:4992) 2995
E066D> GuiMainTas:ffa44dd0:18:01: SetEffectiveSizeOfJpeg w:7488 h:4992
So can I assume that 0x5e4340 is 2200x1872 ?

I'm giving "RAW_SLURP" another try -- that why I was thing 0x5e4340 may work for that like this ?

Code: [Select]

#ifdef CONFIG_5D2
#define DEFAULT_RAW_BUFFER MEM(MEM(0x5e4340))
#endif
I add

Code: [Select]

#define CONFIG_EDMAC_RAW_SLURPin to platform/5D2.212/internals.h
did I miss anything ?

I'll update later tonight.

[dfort]

Houston, we have a problem. This is using a nightly build:

[EDIT] Nothing to see here -- discovered it was a post production problem. Camera is fine.

--snip--

Pressing on, I uploaded allocate-raw-lv-buffer branch builds to my downloads page. Included are builds for the 7D, 5D2, 50D, 550D and even the 1100D. Yeah, I know, not very interesting. It just seems to be an old branch that doesn't even have mlv_lite but it works quite differently on these cameras.

@reddeercity - I have no idea what you are doing but interested in how your experiment turns out. This is what I did with the allocate-raw-lv-buffer branch:

https://bitbucket.org/daniel_fort/magic-lantern/pull-requests/15/searching-for-config_edmac_raw_slurp-on/diff

[EDIT] Got it working! 10bit/12bit on the 7D. I've got to do some more cleanup on the code because it is messing up the Digic V cameras but thought I'd put out the good news. Since there are so many requests from 7D users I uploaded an early build if anyone wants to be a volunteer guinea pig.

[ilia3101]

You got it working!!??? Maybe getting it for the 5D2 is similar. How did you do it??

I see your commit has CONFIG_EDMAC_RAW_SLURP defined for 5D2, could it be working yet?

Surely it can't be that far if you found it for 7d already, the difficult camera

[Danne]

dfort with the magic fingers. 7D working with 10/12 bits, tested just now:
10bit sample
https://bitbucket.org/Dannephoto/magic-lantern/downloads/M06-1005_samples_10bit_7D.zip
12bit sample
https://bitbucket.org/Dannephoto/magic-lantern/downloads/M06-1006_samples_12bit_7D.zip
14bit sample
https://bitbucket.org/Dannephoto/magic-lantern/downloads/M06-1007_samples_14bit_7D.zip



Any chance to get a build for testing on 550D?

[Andy600]

@dfort and for the 50D if you can.

[IDA_ML]

Dfort, you fully deserve your "Hero Member" status!  I can't wait to test.  Thank you so much!

[reddeercity]

Ok can we please stay on topic here , any 10-12bit 7d should be discuss
either on the 7d or 10-12bit thread no here . This is for UHD/3.5k development.

[Danne]

Code: [Select]

Ok can we please stay on topic here , any 10-12bit 7d should be discuss
either on the 7d or 10-12bit thread no here . This is for UHD/3.5k development.I would rather welcome the guy(dfort) who potentially could hold the key to the raw_slurp code snippet for the 5D mark II. But hey, that´s just me.
Keep up the good work everyone. 10bit on 7D is golden.