Canon 40D

[domasa]

Now is visible. Incredibly 

[heder]

Still working, current status ... 

[heder]

I'm stuck at the moment  . My issue in inside LiveView with the HD buffers.

When the camera is started in liveView the HD buffers are correct, and my histogram is correct and it shows the histogram of LiveView.
When I capture a image in LiveView, the HD buffer seems to be locked to the image taken and thus the histogram is locked to the image taken.
So when I return to Live View, the HD buffer is locked to the captured image and does no longer show the histogram of LiveView. When I take
another image the HD buffer gets updated to this new image, and the histogram is changed. The buffer returned to me is not HD buffer with
LiveView content but the image buffer of the last captured image.

I'm going to make screenshot work for 40D (yuv mode) so I can dump correct images and then I guess I need to run the 40D FW in QEMU
with breakpoints in the firmware. There is a function "lvGetResources" that prints out the correct LiveView YUV buffer address in the log, but
ghidra fails to decode which function(s) call lvGetResources. I would like to get that call tree.

[srsa]

I would like to get that call tree.

In case qemu can't get that far, you could also try using cache hacks (src/cache_hacks.h) on the camera.

[heder]

YES thanks, good idea, I just need to intercept the call using cache hacks and read the LR register, then I can  find the return address of the caller

[heder]

I'm having trouble, I hoped I could get a version before xmas, but it looks badly. It's a simple requirement I thought was allready done, but its not : The position of the HD vram buffers are not static.

I havent found the caller of lvGetResources yet, QEMU/gdb won't break, and cache hacks won't do the trick in the real camera either, I'v had troubles in the past with cache hacks on 40D, like I can't use cache hacks on the canon debugmsg function, this does however work in QEMU. My last try is going to use cache hack in QEMU (next week) - This should work.


Scenarios tested:

1. Live Mode

-> Start camera
-> goto LiveMode
-> save all HD buffer 1,2,3.
Result : All three images is Live HD images 

2. Live Mode + image capture
-> Start camera
-> goto LiveMode
-> capture image (take photo)
-> re-enter LiveMode
-> save all HD buffer 1,2,3.
Result : All three images is the captured photo - NOT live mode images 

3. Photo Mode
-> Start camera
-> Capture image (take photo)
-> Goto Live Mode
-> save all HD buffer 1,2,3.
Result : All three images is noise 

.. back to basics ..

[srsa]

I'v had troubles in the past with cache hacks on 40D

I've had similar experience on PowerShots.
I found that the firmware's (and CHDK's) cache manipulation functions counter-act cache hacks. The locked part of cache remained locked, but the content vanished from time to time.
The real solution (IMHO) would be a rewrite of all offending cache manipulation functions, so that they don't touch the locked part. A less reliable way would be re-adding the hacks at the end of each such function.
I chose an even less reliable solution: I re-added the hacks every 10ms, using a task that cycles once every 10ms. It was good enough for doing research.

As far as I know, some ML ports and modules rely on cache hacks - I have no idea how they get away without experiencing similar issues.

One more tip: If I see it correctly, lvGetResource only has data references (no b or bl targets it directly). You could get away just by patching those references using the data cache.

[heder]

Srsa .. You are right about cache hacks. The QEMU "Digic" processor works better in some cases, but on LvGetResources I ran out of luck. Ghidra shows no direct reference to lvGetResource, but alot of function is'nt called directly but indirectly just like lvGetResource, but in most cases I am able to find a pointer to the function, being passed as a argument to another function, but again with lvGetResource I ran out of luck. Ghidra is really good, but in all cases is does'nt understand functions that does'nt start with a stmdb, like if there a mov before the stmdb ghidra get confused, and having to manually make these function is waste of time, and I have a feeling that the pointer to the function or direct call is hidden somewhere.

I wanted was to find the memory (struct) that points to the LV buffers, as this is the very last task before making a beta build.

For now in working in a different direction, I'm have dump'ed the 256 MB ram and have disected this into images as green(rgb),grey,luma(YUV422),chroma(YUV422) in 8 different sizes (512,720,864,972,1024,1944,3888), which resulted im over 6000 images I need to review, which was converted into a movie using ffmpeg, which I need to watch . If the LiveView buffers are there, I should beable to find them.

if this failed I might try your to 10 ms re-adding hack cache method.

[srsa]

Ghidra shows no direct reference to lvGetResource, but alot of function is'nt called directly but indirectly just like lvGetResource, but in most cases I am able to find a pointer to the function, being passed as a argument to another function, but again with lvGetResource I ran out of luck.

Assuming lvGetResource is the function that references the "[LV] lvGetResource 0x%lX (0x%lX)" string, there are 4 data references pointing to it (= the function's address appears 4 times). These references are a few kBytes "above" the function. It looks like Ghidra doesn't automatically notice references like that. I found them by placing the cursor to the function's first instruction and choosing "Search" -> "For direct references" in the CodeBrowser's menu. I then had to turn those bytes into pointers (P button). The references all appear to be part of a huge struct, and that struct is referenced from a function. I'm not familiar with this style of code as PowerShots implement state machines differently.

[heder]

Assuming lvGetResource is the function that references the "[LV] lvGetResource 0x%lX (0x%lX)" string, there are 4 data references pointing to it (= the function's address appears 4 times). These references are a few kBytes "above" the function. It looks like Ghidra doesn't automatically notice references like that. I found them by placing the cursor to the function's first instruction and choosing "Search" -> "For direct references" in the CodeBrowser's menu. I then had to turn those bytes into pointers (P button). The references all appear to be part of a huge struct, and that struct is referenced from a function. I'm not familiar with this style of code as PowerShots implement state machines differently.

Oh yes, I see it clearly now. Huge struct, that will take some time to dig into that. I never learned the "Search" -> "For direct references" until now, thank you for pointing that one out.

[heder]

I have located LiveView, but they're not at a fixed address.

I dumped the ram and converted into different size and formats 512,720,864,972,1024,1080,1944,3888 versus grey,green (rgb),luma(Y422),chroma(Y422),chroma(YUV411) and finally converted them into a movie with ffmpeg, that way I analyzed around 10000 images. Quite effective. This method took some time but it's really nice. I found multiple images in different sizes all over the place, and I found what I was looking for the LiveView buffer. The images I found (smaller size) was:

1. HD buffers
Numbers    : 32
Size          : 1024 x 680
Address    : Dynamic (changes when a image is taken)
Encoding   : YUV422

2. LiveView buffers
Numbers    : 2
Size       : 720 x 214
Address    : Dynamic (changes when a image is taken)
Encoding   : YUV411

3. Miniature (NOT LV image)
Numbers    : 1
Size       : 180 x 320
Address    : Dynamic (changes when a image is taken)
Encoding   : YUV422

Next step is to locate the memory that references these images - or finding the pointers passed to lvGetResources .. seems to be same issue  .

[heder]

Finally I made some progress again . I had to ditch the previous attempt and re-read the disassembled code in order to hunt the correct HD images, this was a pretty good idea. A function called within lvGetResources actually stored the main input parameter (first argument to lvGetResources) at 0x1c804. This pointer is static allocated and points to structure in memory (possibly allocated), at offset 0x94 in this struct is a pointer to one of the current YUV HD images. The image pointer is always correct and is updated after each shot. Now I can shoot images in livemode with correct histogram and vectorscope.

[heder]

I'm working on the bootloader, using the recovery branch.

My main issue is that the old bootloader does not work everytime as expected - it too much a hack, so I need to remake it the right way.

It seems that Call("") on 40D supports : 

"EnableFirmware"
"DisableFirmware"
"EnableBootDisk"
"DisableBootDisk"
"EnableMainFirm"
"DisableMainFirm"

Any one tryed these out on other cameras ? or have clues to which one to use inorder to make autoexec.bin being loaded ?

[heder]

Started working again, it's rainly constantly here in denmark, so no time for astrophotography.

Digged into FPS override mode and finally got it working (low light mode only). It turns out there is no shadow memory area on Canon
40D, so a lot of peripheral registers can't be read. The solution was to create virtual (fake) registers for fps register A and B which is
used in fps override. 

Can't do above 30 fps, only below.

Live Mode buffer images : Upper image is standard live mode in "M" mode (30 fps) with the settings that gave me the brightess image. Lower image is 1 fps with
exposure lock (ISO 3200 "H") and approx 1 second imtegration.  Hot pixels are present

[Dezmond2]

Where download current version for install and testing ?

[heder]

Hi Dezmond2

I'm working on the bootloader, that the last thing needed.

I have a basic working bootloader for 40D (abit similar to 450D and 5D classic bootloaders), but it's not perfect, and I wish to have a bootloader that cannot be activated on the wrong cameras or wrong 40D versions, as this might brick the camera. I have no idea when it done, but my hope is within the next 3-4 weeks.

I will post the complete packages here in this thread.

[heder]

Notes on jpeg recording

The jpeg in live mode 1024x680 (no zoom) has maximum size 330.8k. The header is hardcoded (and so are the jpeg-edmac channels), but the actual file can vary in length, and is typically smaller. I can't not find the true lenght of the jpeg unless I search the entire 330.8k block for the jpeg end container, easy solution is just to save the entire 330.8k block.  Inorder to have a efficient recording system, these images must be copied using edmac into larger buffers, which can be saved. Larger buffers increase the write speed to the CF card, small buffers lower the bandwidth. The FIO file system used dma.  The big issue there is not the size - the jpeg is small, but it the fact that there are no "free" edmac avilable on the DIGIC III, (free = not used by firmware). All edmac are used (no wonder that the DIGIC4 have many more of these). The firmware even seems to be using some edmac for multiple operations. So stealing a edmac is gonna be difficult- but I guess not impossible. The jpeg-edmac channels used for encoding and tranfering the jpeg image is used in approx 5ms per frame, so in the remaining time they seems to be unused. At 24 fps I have around 36 ms for tranfering the image to another location, before the jpeg-edmac is used again, this should be possible.

Minimum CF bandwidth needed at 24 fps = 24 * 330.8k  ~ 8 mb/s. 

These bandwidth requirements are also below the capabilities of the CF system, the bandwidth limit is around 14 mb/s and overclocked around 20 mb/s, so this should also be possible, and higher fps should also be possible. I could also "just" try to hijack a simple function which is called just after jpeg tranfering completes, then save the frame directly, but this requires that the CF system is capable of 8 mb/s in small blocks and that the LV system can cope with the additional code overhead. Also, even more interessing, the Live Mode jpeg is 1024x680, but the raw image is larger 1334x872, and in zoom x5 mode, the raw size is close to HD = 1872x1066 pixels.

[heder]

Hurray ! it's beta release day for 40D

This is the first public release series of  ML for 40D. Please take notice that this is a strictly experimental version, there are no guarantees - if it breaks, you get to keep both pieces. If you find bugs, please let me know, in this thread. If you try this build out, please keep in mind that I have strictly focused on getting main features supported, while spending little time on minor issues. Features/Modules that crashed the camera was removed, but nothing more. "If it aint broke dont fix it". There are options and menus that does not work, but nor crashes the camera. Most modules were removed. The complete package is larger than normal ML packages, as I don't have a efficient bootloader encryptor, but just an old inefficient version.

Latest version: "1.0.6" (18 Feb 2021)
Features: Basic ML support with exposure control, intervalometer and live mode functionality
Important: Trashcan button to enter Magic Lantern. The Q button on 40D is "picture" button.
Important: If the camera crashes, please remove the battery.
Canon FW: 1.1.1
Src: https://github.com/jmheder/ml/raw/master/vxworks_1.0.6.zip
Compiler used: gcc-arm-none-eabi-4_7-2014q2-20140408-linux.tar.bz2
Download (1.0.6): https://github.com/jmheder/ml/raw/master/magiclantern-Nightly.2021Feb18.40D111.zip
Older version:
Download (1.0.5): https://github.com/jmheder/ml/raw/master/magiclantern-Nightly.2020Aug23.40D111.zip
Download (1.0.4): https://github.com/jmheder/ml/raw/master/magiclantern-Nightly.2020Jul29.40D111.zip
Download (1.0.0): https://github.com/jmheder/ml/raw/master/magiclantern-Nightly.2020Jan24.40D111.zip

The main features

Bootloader and installation
See the video in the next reply.

Exposure control
Basic exposure time, F ratio, ISO, color temperature, Av,Tv,ISO lock is available, but I haven't tested them all, so I expect additional issues here. If you wish to use exposure simulation, enable this in the canon menu - and enable it in the ML menu aswell.

Live Mode
Not all overlays is support yet. AF Focusing in Live Mode will produce a temporary corrupted green screen, it not serious, it's just annoying and will go away automatically. Remember to enable live mode in (canon menu), also enable focusing in Live Mode (canon menu)

Raw video
Basic mlv_lite module is supported

Intervalometer.
Intervalometer with or without bulb is supported, but there are of issue (see below) Practice and you will learn. Easy solution: turn off review time and keep one seconds delay between each shot.

Astrophotography
For those into astrophotography, the camera with this ML build is a great combination, the intervalometer with bulb is working, and Live Mode plus low fps and a large exposure time will enable you focus on stars, a great tool for fine adjustment of the focus. Astro module added (star focusing only for now)


List of known minor issues

A. --
B. --
C. Exposure simulation. Don't change ExpSim in exposure control, ML always think it's enabled (remember to enable it via canon menu)   
   also you can use "info" button in Live Mode to verify that canon's own function to verify exposure simulation is within tolerances.
D. LiveMode. When AF focusing in Live Mode a corrupted green screen will occur, it's not important, just annoying.
E. --
F. Intervalometer, this ML version think image review time is fixed to 3(?) sec (you might see incorrect warnings in lower screen)
G. --
H. Intervalometer, in bulb mode keep exposure time below "take pic every sec" + review time, if you set review time
   in canon menu to zero, you should be able to do "crazy pic" if needed.
I. Exposure. Exposure times above 15 ?(20?) seconds is not correctly recognized by ML in exposure menu.
J. --
K. Live Mode. To have a nice Live Mode (with bars) the internally view aspect ratio is changed, a few times you
   might see the bar positions incorrectly. To remove triple bars either exit and enter Live Mode or press the AF focus
   button.     
M. --
N. Canon Menu. Really odd colors in canon menu can be caused by a previous crash (after crash : console is on). Enable console and 
   disable it again to resolve this issue.
O. Color Temperature. ML recognized some color temperature as names like "Sunny", if the temperature was set from Canon menu / dials.
   When changing the color temperature within ML you only get to see the temperature as a number
P. Menu. Some options does not work, but nor crashes the camera.
Q. --

[heder]

Additional info - Source code repository details & future:

Branch: https://bitbucket.org/jmheder/magic-lantern/branch/vxworks

I have merged unified into vxworks and updated the code the 40D platform code, thus this vxworks repository includes more or the lastest unified code (end 2019), which is great news!. The downside is that the 40D is a digic III processor and quite different from digic 4/5..., and this can be seen in the code. No shadow memory, low resolution display, few edmac, fewer PROP's, address space is different and so on.I had to create a simulation layer to keep the code somewhat clean. So I don't think 40D will be included into mainstream "unified", it will take to much time. I will however keep my focus on supporting the  40D with bug fixes and possibly more features, like modules and even try to get mjpeg stream recording up and running.

[heder]

Installation on 40D

The build include is a simple "bootflag flipper" bootloader, like the 5Dc,400D's. Update the camera with ML-SETUP.FIR, this will take some seconds. After the screen goes black, a few seconds will pass, then bootloader will run. When the bootloader is running the follow led pattern can be seen:

very fast blue led blinking   = bootloader is running (this will allways occur at first)
solid blue led                     = camera bootflag was enabled, please remove the battery
very slow blue led blinking  = camera bootflag was disabled, please remove the battery
solid red led                       = Incorrect firmware version (use Canon 40D FW = 1.1.1), please remove the battery

You will also need EOSCard to make the CF card bootable, see the video

Quick Video

Small mistake around 1:50,  "Bootloader is running, bootloader is on". I should have said  "Bootloader is running, camera bootflag is on"

[ilex]

Congrats!!! Up with the great work!!Thanks

[ilex]

Hi Heder
I've tried to install your ML in the Eos 40D FW 1.1.1.
When doing so the blue LED does what you say, it flashes fast about 20 times, then solid and then slow.
I remove the battery and switch the camera back on and it remains without changing the FW and does not charge the ML.
How can I run it?
What am I doing wrong?

Excuse my poor English.

Greetings

[heder]

Hi ilex.

It would be a good idea to read the installation guide. https://wiki.magiclantern.fm/install

But ... a quick introduction. The bootloader (the one you installed) will enable the camera to boot magic lantern firmware from the card ... but ONLY if the CF card is prepared correctly.  You forgot to make the bootflag in the CF-card partition table set true and set the volumen label to be "EOS_DEVELOP". You can use EOS_Tool.exe to do that.

1. Upgrade the Canon 40D with the FIR file and make sure that the BLUE led is solid at the end (solid blue = camera bootflag enabled).
2. Put your CF into your CF card reader.
3. Download https://pel.hu/down/EOScard.exe and run it. Press the EOS_DEVELOP and bootdisk button and then save
4. Make sure you have autoexec.bin (magic lantern) on the CF card.

[ilex]

Hi heder,
Thanks for your quick answer.
I used ML on an EOS 50D until two years ago and I currently have it installed on an EOS M and neither had this problem with the boot up.
I will try to install it during the day according to your advice and the ML installation page.
Thanks for your great work to port ML to this old EOS 40D, which will help to give it a new life.
Greetings

[heder]

Hi
My 40d bootloader does not charge the CF card, like other bootloaders. You Will have to use Eos-Tool. Maybe later in 2020 i Will change the bootloader to work like all others.