Author Topic: Canon 40D (Read 471923 times)

Walter Schulz · « **Reply #500 on:** March 22, 2020, 12:40:54 AM »

Quote from: reddeercity on March 22, 2020, 12:30:47 AM

On the CF card write speed , your saying you only get around 20MB/s ?
Is this on a type 1 or type 2 CF card ?

"Type" = Physical dimension (thickness) of a CF-card. Type II is thicker (5 mm) and was used for rotational media (as in harddisks, see MicroDrive).
Here is some info about CF modes

Quote

Explanations
The CF interface is derived from the ATA Interface. The CF 2.0 specification allowed PIO-4 mode that has a theoretical limit of about 15.9MB/s. CF 3.0 introduced two additional PIO modes that are not part of the original ATA specification: PIO-5 and PIO-6, where PIO-6 has a theoretical limit of about 23.8MB/s. Most CF cards with speed ratings from 80x to 166x use PIO-6 to achieve this speed. However, there is currently no FireWire reader with support for PIO-5 or PIO-6. This means that PIO-6 cards run in the slower PIO-4 mode, and therefore run slower than with current USB 2.0 readers. Support for MDMA modes is most likely not that popular, but you can't really tell the difference between PIO-4 and MDMA-2 when a card reader is used, because PIO only occupies the card reader controller, not the host CPU. CF 3.0 also introduced two additional MDMA modes, MDMA-3 and MDMA-4 using the same timing as PIO-5 and PIO-6.
CF 1.0: Max. PIO-2 (240ns), 7.95MB/s.
CF 2.0: Max. PIO-4 (120ns), 15.89MB/s.
CF 3.0: The new CF PIO modes PIO-5 and PIO-6 and the ATA UDMA modes UDMA-0 to UDMA-4 were introduced, offering a theoretical limit of about 63MB/s for UDMA-4.
CF 4.0: The remaining ATA UDMA modes UDMA-5 and UDMA-6 were added, allowing up to about 127MB/s.
CF 5.0: 48-bit commands were added. This command set allows a transfer block size of up to 32MB compared to 128kB with the previous command set. This will allow actual transfer rates to come closer to the theoretical limit.
CF 6.0: The new CF UDMA mode UDMA-7 (24ns/166M) was added, allowing up to about 158MB/s transfer rate.

In case you have plans on using CF cards as regular ATA device: For all timings of 100ns or faster (UDMA-5, UDMA-6 and UDMA-7), the CompactFlash specification only allows one device per bus with a maximum distance of 0.15m (6in) from the card connector to the host controller, at 3.3V supply voltage! In addition to that, 80-wire cables must be used for all UDMA modes.

Current UDMA CF Cards and UDMA CF readers support UDMA-5 or UDMA-6, with real-world transfer rates of more than 80MB/s. This is already beyond the theoretical transfer rate of UDMA-4. Current USB 2.0 UDMA CF readers are maxed out at about 32MB/s, less than 40% of the actual card performance.

Source: Late and lamented http://www.hjreggel.net/cardspeed/index.html (Current = Years and years ago)

heder · « **Reply #501 on:** March 22, 2020, 10:13:51 AM »

Quote from: Danne on March 21, 2020, 10:18:27 AM

Nice work! How close are you to mjpeg stream? How would that work in practice? mv1080p compressed into mjpeg? Recorded as single images? Curious.

Danne, Atleast the older cameras like 40d,450d,1000d were built with jpeg streaming capabilities, were each images is downsampled, mjpeg compressed and finally tranfered to the pc. We can utilize this in magic lantern and save each mjpeg into a file, which will result in a motion-jpeg stream. A tool like ffmpeg supports this non-standard format. The interessing features is that each frame is a full image.

Quote from: reddeercity on March 22, 2020, 12:30:47 AM

Great Job @heder , I've been watching your progress with great interest
One of my long term goals in to have Mjpeg compression -> .avi (4.2.2 8bit)
I know the stream is there but was unable to fully understand it to be implemented in d4 cams
this something that All ML cams can use specially the older d4 e.g. 500d etc. ...

On the CF card write speed , your saying you only get around 20MB/s ?
Is this on a type 1 or type 2 CF card ? spec. say it can use either one .
There is a branch that has to do with CF bus clocking or overclocking ,
I've been looking in to this for the 5d2 & 50d .
Some useful link from previous conversation's about CF card speed
https://www.magiclantern.fm/forum/index.php?topic=12862.msg206010#msg206010
https://www.magiclantern.fm/forum/index.php?topic=12862.msg206011#msg206011
https://www.magiclantern.fm/forum/index.php?topic=19336.msg206067#msg206067etc. .....

eddeercity, as far as I can tell the 40d only support pio-6 mode but seems to be underclocked (14 mb/s), I wasted alot of time digging into the firmware code looking for UDMA, without success, I used 4 different new card from sandisk, kingstons, transcend (ranging from 866x down to 166x). I did alot of blackbox testing with the timing and ended with approx 20 mb/s which the pio-6 specification support. These are approx 3mm cards so I guess CF-1. The firmware supports CompactFlash Specification Revision 3.0 (found "CFA3.0" strings in firmware) http://rumkin.com/reference/aquapad/media/cfspc3_0.pdf

I documented some of my finding here https://magiclantern.fandom.com/wiki/Register_Map/40D
---

Theoretically: the sensor was a readout rate of 96Mpixel/sec, thats 4004004 pixels per second at 23.976 or ~9 fps at full sensor size. Both should be doable with the CF limited bandwidth (jpeg compression), but there is along list of issue to be solved. I need to understand the yuv-mjpeg livemode "chain" inorder to beable to change the width and height of the images and many many other issues. The yuv-mjpeg chains seem to be is abit more complex that "raw video", this one really needs alot of cache hijacking.

Danne · « **Reply #502 on:** March 22, 2020, 12:58:36 PM »

Quote from: heder on March 22, 2020, 10:13:51 AM

Danne, Atleast the older cameras like 40d,450d,1000d were built with jpeg streaming capabilities, were each images is downsampled, mjpeg compressed and finally tranfered to the pc. We can utilize this in magic lantern and save each mjpeg into a file, which will result in a motion-jpeg stream. A tool like ffmpeg supports this non-standard format. The interessing features is that each frame is a full image.

Sounds like a game changer to me

.

Ant123 · « **Reply #503 on:** March 22, 2020, 07:07:40 PM »

Quote from: heder on March 22, 2020, 10:13:51 AM

We can utilize this in magic lantern and save each mjpeg into a file, which will result in a motion-jpeg stream.

https://www.magiclantern.fm/forum/index.php?topic=8119.msg212429#msg212429

heder · « **Reply #504 on:** March 22, 2020, 07:48:43 PM »

Quote from: Ant123 on March 22, 2020, 07:07:40 PM

https://www.magiclantern.fm/forum/index.php?topic=8119.msg212429#msg212429

Yes exatly.

Quote from: a1ex on February 23, 2019, 09:34:06 AM

The JPEG buffer is is populated *and* freed in lvcdevResourceGet, so... that's pretty much the only place for a hook, if you want to reuse Canon's approach. You could place a hook (patchmgr: patch_hook_function) directly into 0xFFAD5870, but that won't solve much (the hook code will only execute when needed, so it will be a bit cleaner). You should also speed up the copying process with either dma_memcpy (HPCopy) or edmac_memcpy; that should get rid of dropped frames.

On newer models (DIGIC 5, IIRC also 60D and newer DIGIC 4), this function no longer runs automagically while in LiveView, so this approach is only valid for older models.

There is another drawback of this JPEG stream: it's not using the full LiveView resolution. On 450D, CR2 resolution is 4312x2876 (with dcraw), so I'd expect something close to 1436x958 in LiveView. On 5D2, CR2 resolution is 5634x3753, LiveView resolution is 1880x1248, LV-JPEG resolution is 1024x680. I'm pretty sure the hardware can deliver MJPEG at full LiveView resolution; "just" need to figure out how to how to request a full-size YUV buffer (most Canons do this while recording H.264) and how to call the JPEG encoder on an arbitrarily-sized YUV buffer (likely very similar to the lossless encoder).

BTW - disabling this JPEG stream on old models could be useful for reducing power consumption. Nothing measured; just theory.

This is the real challenge

heder · « **Reply #505 on:** March 24, 2020, 08:46:11 PM »

When running with call("lv_save_raw",1) the raw EDMAC channel has the following sizes

LiveView x1 = raw bayer format = 1336 x 872
LiveView x5 = raw bayer format = 1952 x 814
LiveView x10 = raw bayer format = 1952 x 814

Ant123 · « **Reply #506 on:** March 25, 2020, 04:49:50 PM »

Did you manage to get the image using "lv_save_raw" ?

heder · « **Reply #507 on:** March 26, 2020, 08:00:09 PM »

No, i didnt borther yet, i just analysed the RAW edmac register values.

Im stuck in gui stuff, menus in liveview isnt working, which is needed for raw video.

heder · « **Reply #508 on:** March 31, 2020, 11:02:36 AM »

* Menus in livemode is now working

* mlv_lite framework is up and running

- modules vsync is called from my own 40d vsync task (hmm, should use the commonm vsync func, but ok for now)
- module version structs "VERS" in output header turns into alot of garbage => disabled for now.
- small hacks set to 0
- SRM_job set to 0
- Maximum memory set to 40MB
- edmac_copy_rectangle_cbr_start fails for now ...

The camera save correctly and mlv_dump accepts the and decodes the output file correctly

(if I skips dmac_copy_rectangle_cbr_start and just put in dummy data using memset), so I need to get dmac_copy_rectangle_cbr_start verified.

heder · « **Reply #509 on:** April 02, 2020, 03:19:47 PM »

Yay .. finally a nice looking raw recorded frame from mlv_lite ! .. (without edmac support).

1 fps at 1312x872 (x1 mode) using memcpy(), approx each 20th frame is ok. Had to use memcpy to copy line of line. I need to hack the raw edmac system more before I can use it. My current hijacked vsync function is in the wrong spot, and the 40d contnuesly writes the raw edmac destionation register before performing the raw tranfers, and that pointer changes. mlv_lite overwrites this pointer, but without proper timing the camera just overwrites mlv_lites address again, stupid 40d, surrender.

kitor · « **Reply #510 on:** April 03, 2020, 10:46:33 PM »

MLV on 40D

Even if that's single frame, congrats. I'll check it just for fun, after whole covid crap ends (dad has pristine 40D with only ~10k frames).
Wish I had enough knowledge to move R forward, but with time that I can spent, I'll maybe finally upgrade my cam to latest FW and update the stubs...

heder · « **Reply #511 on:** April 04, 2020, 07:50:57 AM »

Updated open beta to 1.0.2

https://www.magiclantern.fm/forum/index.php?topic=1452.msg224594#msg224594

Fixes =
Menu's are now working in LiveMode (use joystick instead of scrollwheel)

I'll be "out of office" for the next 10 days, corona is driving the kids nuts, easter vaccation @ home. mlv_lite will take some more time, its abit more complex than I thought.

JackV88 · « **Reply #512 on:** April 12, 2020, 06:24:47 PM »

Hi, I am a new member and I have a problem. How do I download from this link?
https://bitbucket.org/jmheder/magic-lantern/branch/vxworks

Another thing, does focus peaking work for vintage lenses?
sorry, I hope to be helped. Happy Easter.

heder · « **Reply #513 on:** April 12, 2020, 09:48:47 PM »

Hi JackV88, welcome.

I assume you wish to download the source code for the 40d. You need mercurial command "hg" and download my master branch https://bitbucket.org/jmheder/magic-lantern and then switch to vxworks branch. If you were only looking for binaries, then look in the https://bitbucket.org/jmheder/magic-lantern/downloads folder.
The unified master branch for all cameras are here https://bitbucket.org/hudson/magic-lantern

I cant answer you regarding focus peaking, i have not used any time in that, im spending time om video atm.

heder · « **Reply #514 on:** April 14, 2020, 02:25:42 PM »

I realized why I can't get every raw frame. The raw save EDMAC functionallity system does'nt run on each frame

, but only sometimes, and even more stange ... the interval between each raw frame is'nt even consistent, really odd. So now I need to create my own sync/hook function, that tranfers each frame.

Wlad81 · « **Reply #515 on:** April 16, 2020, 08:47:42 PM »

So, this camera wouldn't be able to output the full speed raw video stream (*.mlv)?

heder · « **Reply #516 on:** April 17, 2020, 06:12:10 PM »

Quote from: Wlad81 on April 16, 2020, 08:47:42 PM

So, this camera wouldn't be able to output the full speed raw video stream (*.mlv)?

I don't think this is a raw issue aka. speed bottleneck, but the digic 3 cameras are old and abit different from the digic 4 cameras.
The sensor readout rate is really high = 96 Mpixel (like 5D2) and the internal EDMAC is above 300 MB/s, so there plenty of room for
raw tranfers. The only bottleneck I see is CF card 20 MB/s (and this is with modified timing), but ofc this one had nothing to do with
raw trafering. Perhaps the developers did'nt need raw tranfers while debugging, and did'nt implement raw tranfers on each frame ?,
or due to battery usage or some other reason.

I have tracked down the specific function and code that executes the raw tranfers. This function also tranfers the yuv interpolated image
for displaying on the screen. In fact there a function for each mode x1,x5,x10. In all 3 tranfer functions - only when some conditions are
correct will the raw edmac tranfer be allowed. I'm currently working on using cache tricks to force raw tranfers on each frame, when we will
know for sure.

heder · « **Reply #517 on:** April 21, 2020, 02:42:49 PM »

Raw video using single buffer is now running, all frames are good

Theta Sigma · « **Reply #518 on:** April 21, 2020, 02:52:34 PM »

Thanks so much for the continued work! It almost seems as if the finish line is becoming visible on the horizon now.

Once you collect your trophy, might I interest you in running the 80D marathon afterward?

heder · « **Reply #519 on:** April 21, 2020, 03:08:56 PM »

Quote from: Theta Sigma on April 21, 2020, 02:52:34 PM

Thanks so much for the continued work! It almost seems as if the finish line is becoming visible on the horizon now.

Once you collect your trophy, might I interest you in running the 80D marathon afterward?

It was quite a nice experience to see each frame was good, finally, like getting a birthday present

I think I'll be done
with 40D in 2021 (oh dear), still so much to do. But the future ? there are so many cameras that are interessing, I can't
say were I want to go after 40d, we will see, but im only interessed in cameras that is missing ML. For now it's the 40d
(and properly helping 1300D users abit).

Danne · « **Reply #520 on:** April 21, 2020, 03:14:59 PM »

Nice progress @heder. Not many around here nowadays that can accomplish these things. How many frames per second are you getting?

critix · « **Reply #521 on:** April 21, 2020, 03:17:47 PM »

Quote from: heder on April 21, 2020, 03:08:56 PM

(and properly helping 1300D users abit).

Thank you for taking the time to help us move on.

heder · « **Reply #522 on:** April 21, 2020, 03:31:52 PM »

Quote from: Danne on April 21, 2020, 03:14:59 PM

Nice progress @heder. Not many around here nowadays that can accomplish these things. How many frames per second are you getting?

I have not yet begun to do fps testing. I only have the basic system running : single buffer, no edmac and only in x1 mode, but completing these
is only a question of time, no additional hacks needed. The CF Card is however only 20 mb/s, so this one sets the limit. Raw edmac is however only
a stepping stone, mjpeg video in x5/x10 mode is primary target, but that one is even harder than raw video.

Ant123 · « **Reply #523 on:** April 21, 2020, 04:51:18 PM »

heder
Maybe you will commit changes to your repository?

heder · « **Reply #524 on:** April 21, 2020, 05:58:52 PM »

Quote from: Ant123 on April 21, 2020, 04:51:18 PM

heder
Maybe you will commit changes to your repository?

Ok, they're updated on bitbucket, but it aint pretty at all, it's currently one big spagetti hack. I will work on making in more pretty, anyways

* The raw tranfering is started with call("lv_save_raw",1).
* mlv_lite set the image buffer via a fake call EngDrvOut(0xFFFFFFFF,mlv_lite_buffer);
* The raw tranfering is stopped with call("lv_save_raw",0) and a fake call EngDrvOut(0xFFFFFFF0,0);
* mlv_lite is also currently a big mess

On each frame Canon calls the same function which perform multiple dma tranfer (including YUV), sometimes is also issues a raw tranfer, but always writes a value 0xc0f08030.

1. If canon writes the edmac 4 destination, then I know that a raw frame will be tranfered, and I set an internal raw flag, and the only thing I will do it change the edmac 4 destination the mlv_lite buffer

2. A write to 0xc0f08030 will always occur AFTER a possible edmac 4 destination write, so here I check that the internal raw flag is set, if not set, then canon did not start the tranfer, and I will start the raw tranfering myself with the correct mlv_lite buffer pointer.

Now I *just* need to redo all the spagetti mess

News:

Author Topic: Canon 40D (Read 471923 times)