Magic Lantern Forum

Developing Magic Lantern => General Development Discussion => Topic started by: a1ex on January 25, 2016, 09:29:53 AM

Title: Portable ROM dumper
Post by: a1ex on January 25, 2016, 09:29:53 AM
Lately I've got a few softbricked cameras to diagnose, and struggled a bit with the ROM dumper from bootloader: it wasn't quite reliable. A while ago, g3gg0 reimplemented it with low-level routines (which worked on his camera, but not on mine). Today I looked again at the old approach, and it looks like the file I/O routines from bootloader had to be called from RAM, not from ROM.

So, I've updated the code and need some testing. I've emulated this in QEMU, but the results may be different on real hardware.

What you have to do:

- download autoexec.bin (http://a1ex.magiclantern.fm/debug/portable-rom-dumper/autoexec.bin)
- place it on a card without any important data on it (it might corrupt the filesystem if anything goes wrong)
- the display looks roughly like this: http://www.magiclantern.fm/forum/index.php?topic=14732
- after it's finished, look on the card, and you will find 4 files: ROM[01].BIN and ROM[01].MD5.
- you don't have to upload them, just check the MD5 checksum:
  - Windows: you may use http://www.winmd5.com/ (http://WinMD5)
  - Mac, Linux: md5sum -c *.MD5
- repeat the test on the same card (let it overwrite the files), then on a card with different size (and maybe different filesystem).

Some cameras have only ROM1 connected, so dumping ROM0 will give just random noise. In this case, the ROM0 checksum may not match, but that's OK.

The ROM dumper should recognize all ML-enabled cameras, except for 5D2, 50D and 500D. These old models do not appear to have file writing routines in the bootloader (or, at least I could not find them). The QEMU simulation works even on exotic models like 1200D or EOS M2.

So, you don't have to upload any files or screenshots. Simply verify the MD5 checksums on your PC (if in doubt, paste the md5sum output).

That's it, thanks for testing.
Title: Re: Portable ROM dumper
Post by: nikfreak on January 25, 2016, 06:46:35 PM
Both 100D.100B and 70D.111A get stuck at "Dumping ROM0...". No LED activity neither any sign of placing / writing files to sdcard on both cams.
Title: Re: Portable ROM dumper
Post by: Walter Schulz on January 25, 2016, 07:04:01 PM
650D
Code: [Select]
Magic Lantern Rescue
----------------------------
- Model ID: 0x301 650D
- Boot flags: FIR=0 BOOT=-1 Ram=-1 UPD=-1
- ROMBASEADDR: oxFF0C0000
- Dumping ROM0...

... and doing nothing else. No LED activity, no file.

7D: Pretty much the same. Different strings for Model ID and ROMBASEADDR of course.
Title: Re: Portable ROM dumper
Post by: mk11174 on January 26, 2016, 09:17:59 AM
700D works and gives Files, But ROM1.MD5 checksum changes each time, ROM0.MD5 stays the same.

I used winmd5free, I ran 1 test, got the checksum from ROM0.MD5 and copy and pasted it in the Original text box, then ran test again and loaded new ROM0.MD5 and checksum was same as first.

Did same steps for ROM1.MD5, but checksum was differ.

Did I even do it correctly or were you wanting something else?
Title: Re: Portable ROM dumper
Post by: a1ex on January 26, 2016, 01:00:04 PM
On 700D, ROM0 does not appear to be connected, so you are interested only in ROM1.

So, in the Browse dialog, you select ROM1.BIN, and check if the number in the "Current file MD5 checksum value" box is the same as the one from ROM1.MD5. However, I already know, from the user with the bricked 700D (http://www.magiclantern.fm/forum/index.php?topic=16487), that it doesn't work well on this camera :(

Any 600D user can try it?
Title: Re: Portable ROM dumper
Post by: mk11174 on January 26, 2016, 04:30:09 PM
Yeah, no same  :(

Only can try with 1 card though, don't have any extra cards at this time.  :-[
Title: Re: Portable ROM dumper
Post by: mk11174 on January 26, 2016, 08:47:15 PM
600D

Model ID: 0x286 600D
Boot flags: FIR=0 BOOT=-1 RAM=-1 UPD=-1
ROMBASEADDR: 0XFF010000
Dumping ROM0...

nothing on card no led
Title: Re: Portable ROM dumper
Post by: a1ex on March 04, 2016, 07:02:09 PM
Made some updates. Can you try again on 600D, 650D and 700D?

I don't expect it to work on 7D, but who knows.

edit: it's much slower this time (5-10 minutes per file), and there's no LED activity
Title: Re: Portable ROM dumper
Post by: Walter Schulz on March 05, 2016, 11:24:27 AM
7d hangs after line "- Dumping ROM 0. . .".

650D does simply fly through "-Dumping ROM 0. . .". Much too fast, about 20 seconds between ROM and MD5 line.
Hangs after ROM 1 message. MD5 file seems to be corrupted. Windows wants to check the card, other files seems to be out of order, too.

Boys, take warning about card corruption seriously!
Title: Re: Portable ROM dumper
Post by: DeafEyeJedi on July 26, 2016, 12:03:49 PM
7d hangs after line "- Dumping ROM 0. . .".

I can also confirm this as well for the 7D.
Title: Re: Portable ROM dumper
Post by: a1ex on July 29, 2016, 12:06:45 PM
I managed to get correct ROM dumps on 5D3 on an 8GB card that was previously failing (filesystem corruption). It's still not perfect, the first file saved ends up corrupted, but seems to be a small step forward.

The issue: Canon's bootloader routines for file I/O copy the data to some cacheable (!) memory; when that buffer reaches 0x4000 bytes, it's written to card using DMA. However, patching Canon code to use an uncacheable memory pointer didn't help.

Before, I was disabling data and instruction caches, and I could get correct dumps on an old 256MB (yes, MB) card.

Today I've tried to disable the write buffer (SCTLR bit 3), but that didn't seem to help. What helped was disabling the cache and write buffer in the memory regions that are set up at camera startup (see http://magiclantern.wikia.com/wiki/Memory_map ):
Code: [Select]
static inline void disable_all_caches()
{
    asm(
        "mrc p15, 0, r0, c1, c0, 0\n"   /* read SCTLR */
        "bic r0, #0x4\n"                /* disable data and unified caches */
        "bic r0, #0x8\n"                /* disable write buffer */
        "bic r0, #0x1000\n"             /* disable instruction cache */
        "mcr p15, 0, r0, c1, c0, 0\n"   /* write back SCTLR */
        "mov r0, #0\n"
        "mcr p15, 0, r0, c2, c0, 0\n"   /* disable data cache on all memory regions */
        "mcr p15, 0, r0, c2, c0, 1\n"   /* disable instruction cache on all memory regions */
        "mcr p15, 0, r0, c3, c0, 0\n"   /* disable write buffer on all memory regions */
        : : : "r0"
    );
}

Still, the first file is always saved with incorrect checksum on the 8GB card. To work around this, I dump the two ROMs twice => 4 files. First has incorrect checksum, next 3 are fine, and the behavior is very consistent.

All 4 files end up correct on the 256MB card.

I've updated the autoexec.bin from the first post, and I'd like you to try it on all ML-enabled cameras, except 7D.
Title: Re: Portable ROM dumper
Post by: Walter Schulz on July 29, 2016, 08:12:59 PM
(http://picload.org/image/rrocplrl/img_20160729_195800.png)

Used a 1 GB card formatted with cam.
EOScard with "classic" ML.
Replaced autoexec.bin.
Inserted into cam and - after a while - done.

Anything else to test on 650D? Other card sizes, file systems?
Title: Re: Portable ROM dumper
Post by: a1ex on July 29, 2016, 09:07:54 PM
Yes, on the card, from a Linux or Mac shell, run this command: md5sum -c *.MD5

For Windows, cygwin might have the equivalent command, otherwise you may have to use a GUI.
Title: Re: Portable ROM dumper
Post by: Walter Schulz on July 29, 2016, 09:34:08 PM
Or commandline (Powershell 4.0)

Code: [Select]
PS H:\> get-filehash ROM*.BIN -Algorithm MD5

Algorithm       Hash
---------       ----
MD5             D5929B7A5AD99F511FC83D7D0B48B85F
MD5             A1C1204A28D2453F11374C8690CB0C35
MD5             D5929B7A5AD99F511FC83D7D0B48B85F
MD5             A1C1204A28D2453F11374C8690CB0C35

EDIT: BTW, is it intended to use lowercase for first bin and uppercase for the rest?

rom0a.bin
ROM1A.BIN
ROM0B.BIN
ROM1B.BIN
Title: Re: Portable ROM dumper
Post by: a1ex on July 29, 2016, 10:02:39 PM
Looks like it finally worked, right?

The case difference is not intended. Maybe it has to do with the corruption I'm experiencing for the first file?
Title: Re: Portable ROM dumper
Post by: nikfreak on December 06, 2016, 10:49:05 PM
canon released a bunch of firmware updates recently. Any plans on updating the portable dumper so it works for all cameras?
Title: Re: Portable ROM dumper
Post by: a1ex on December 06, 2016, 10:55:13 PM
Code: [Select]
[...]

Testing portable ROM dumper...
   5D: skipping
  5D2: skipping
  5D3: ROM0.BIN: OK ROM1.BIN: OK
  5D4: ROM1.BIN: OK
   6D: ROM0.BIN: OK ROM1.BIN: OK
   7D: ROMs not saved
 7D2M: ROM1.BIN: OK
  40D: skipping
  50D: skipping
  60D: ROM0.BIN: OK ROM1.BIN: OK
  70D: ROM0.BIN: OK ROM1.BIN: OK
  80D: ROM1.BIN: OK
 400D: skipping
 450D: skipping
 500D: skipping
 550D: ROM0.BIN: OK ROM1.BIN: OK
 600D: ROM0.BIN: OK ROM1.BIN: OK
 650D: ROM0.BIN: OK ROM1.BIN: OK
 700D: ROM0.BIN: OK ROM1.BIN: OK
 750D: ROM1.BIN: OK
 760D: ROM1.BIN: OK
 100D: ROM0.BIN: OK ROM1.BIN: OK
1000D: skipping
1100D: ROM0.BIN: OK ROM1.BIN: OK
1200D: ROM0.BIN: OK ROM1.BIN: OK
 EOSM: ROM0.BIN: OK ROM1.BIN: OK

[...]
Title: Re: Portable ROM dumper
Post by: nikfreak on December 08, 2016, 12:03:05 PM
Can you post a dumper for 7D1 please?
Title: Re: Portable ROM dumper
Post by: DeafEyeJedi on December 08, 2016, 06:53:23 PM
Can you post a dumper for 7D1 please?

+1 as this should open some doors for newer firmwares that were released by Canon recently.
Title: Re: Portable ROM dumper
Post by: nikfreak on January 02, 2017, 02:47:53 PM
Can you post a dumper for 7D1 please?

or maybe update the portable binary to save the 7D ROM?
Title: Re: Portable ROM dumper
Post by: a1ex on January 02, 2017, 03:28:50 PM
I don't have one ready; the portable codebase probably requires some sort of IPC communication.

Understanding it could be useful for other things as well (QEMU, 7D2, running code on master without FIR).

On 7D, I don't even know how to blink the LED from bootloader context (source) (http://www.magiclantern.fm/forum/index.php?topic=17610), but can be found out from the dm-spy branch (binary search to find out where the LED starts to blink with the usual register commands, to see what Canon code does).

If you have enough patience, g3gg0 committed some QR code experiments on the portable codebase. Might be useful.
Title: Re: Portable ROM dumper
Post by: dfort on February 02, 2017, 12:46:46 AM
Used the portable dumper to get a 700D.115 ROM1.BIN. Would this be the first step to porting a firmware update?

Notes: An exfat formatted card didn't work and neither did an old 8GB card but a fairly new 32GB SanDisk Extreme PRO worked--the second time I tried it. The first time it only saved the ROM0B.MD5 file. Interesting that the card was renamed to EOS_DEVELOP.
Title: Re: Portable ROM dumper
Post by: dfort on October 20, 2017, 10:23:02 PM
Got my hands on a 7D and the first thing I tried was to get a dump of the latest 2.0.6 firmware just to see if some of the stubs that I'm not finding porting 2.0.3 to the crop_rec_4k branch. Yeah, that's a long shot. Of course I hit the wall with the Portable ROM dumper. I also tried the DUMP1300.FIR and dumperM2.fir, another long shot that didn't work. Is there any way to dump the 7D ROM? There must be but maybe the original porting notes are long forgotten? I couldn't dig up anything in previous forum posts.
Title: Re: Portable ROM dumper
Post by: dfort on October 21, 2017, 07:16:38 AM
Just an interesting observation. On the 7D the portable ROM dumper works fine in QEMU but not on the camera!

(https://farm5.staticflickr.com/4447/37565757710_d5786ecbc4.jpg) (https://flic.kr/p/ZeyjZL)

Of course to run QEMU you need a ROM dump. That's a Catch 22.
Title: Re: Portable ROM dumper
Post by: nikfreak on October 21, 2017, 10:59:02 AM
check also my requests above.
You'll be amazed that adtg_gui makes problems, too.
For the moment I'd rather focus on getting dual_iso as well as lossless compression to work (instead of adding support for 2.0.6)
Title: Re: Portable ROM dumper
Post by: dfort on October 21, 2017, 07:29:30 PM
I'm just trying to get familiar with the 7D and going through every stub and constant is one way to do that. Updating the firmware at this point isn't a priority, getting it ported to crop_rec_4k is, though it won't be as easy as the 6D and 650D. Of course if you're working on it the odds are good.

Back on topic, I did a little more digging into the recovery branch and found a way to compile a portable ROM dumper.

Code: [Select]
# recovery (portable display test, ROM dumper, CPU info...)
hg update recovery -C
cd ../../platform/portable.000
make install ML_MODULES_DYNAMIC= CONFIG_BOOT_DUMPER=y

With the latest commit on the recovery branch [1cfee5fb3c40 (https://bitbucket.org/hudson/magic-lantern/commits/1cfee5fb3c40917f546d6e581434833ddcfcf196)] all I get is noise on the screen, this happens in both QEMU and on the camera.

(https://farm5.staticflickr.com/4494/23982597168_39a5363475.jpg) (https://flic.kr/p/Cxg5AC)

This first appeared in commit 987a55497b04 (https://bitbucket.org/hudson/magic-lantern/commits/987a55497b04f9bd6f722e8845a0196677f913d9), disp_dump: added experimental display based dumper.

So stepping back one commit I can make a portable ROM dumper that works in QEMU but not on the camera. It is a little different than the pre-compiled dumper posted by a1ex on the first post of this topic.

(https://farm5.staticflickr.com/4488/37803264672_8c8e720540.jpg) (https://flic.kr/p/ZAxBxo)

Regressing all the way back to the big bang of the recovery branch I still couldn't get a ROM dumper that works in camera on the 7D.

Another issue I've been having is not being able to get to the Canon menu, much less the ML menu, in QEMU but that's a different topic.
Title: Re: Portable ROM dumper
Post by: a1ex on October 21, 2017, 09:51:58 PM
Well, 7D is a different beast. ML was ported to this camera a long time ago - check the old wiki (http://magiclantern.wikia.com/wiki/7D_support) for notes.

The recent dumper experiments are for 5DS (and they are also be useful for 7D2 (http://www.magiclantern.fm/forum/index.php?topic=13746.msg190105#msg190105)). Both are dual digic models, with the same quirk: the portable ROM dumper works in QEMU, but not on the camera. You need a second camera to decode the blinks.

Emulation would require starting two instances of QEMU and modeling the communication between them somehow. I remember reading about a QEMU fork with multiprocessor support (maybe the Xilinx version, or something based on that). Low priority from my side, for now.
Title: Re: Portable ROM dumper
Post by: dfort on October 21, 2017, 10:50:39 PM
I found old Google Group discussions like this one (https://groups.google.com/forum/#!topic/ml-devel/Dh2jGGyh280) and this (https://groups.google.com/forum/#!topic/ml-devel/ljZ4Ko8lu30) that pre-dates the old wiki.

Then there's this CHDK universal dumper (http://chdk.wikia.com/wiki/Canon_Basic/Scripts/Dumper) that looks interesting. It runs on "CanonBasic (http://chdk.wikia.com/wiki/Canon_Basic)" scripting language and is listed as working on the 7D as well as most other EOS cameras. I tried following the tutorials but no luck.

Oh well, though I discovered something new but what's old is new to me.
Title: Re: Portable ROM dumper
Post by: a1ex on October 21, 2017, 11:00:03 PM
EOS firmware doesn't have Canon Basic, though DIGIC 5 models apparently have a similar scripting language (http://www.magiclantern.fm/forum/index.php?topic=2864.msg190986#msg190986). It's not present on DIGIC 6, unlikely to be in D7, so for me there's very little reason to look into it.

Canon Basic dumper is only for PowerShot firmware (that includes EOS M3 and newer models).
Title: Re: Portable ROM dumper
Post by: g3gg0 on October 22, 2017, 01:33:36 AM
there is an advanced display based dumper available, allowing reasonable transfer speeds.

here a vid: https://www.youtube.com/watch?time_continue=117&v=9mNhph9BfNA
the current implementation is quite a bit faster and doesnt take so long during dot calibration iirc.

how it works:
"master" camera booted in dumper, displays some patterns.
"slave" camera, in video mode, scans the screen and waits for the dots to appear.
after they went through setup, ROM content is shown on screen and slave camera writes a plain ROM image to its card structured dump file, that must be "assembled" on PC later.

how to use:
- get the recovery branch (https://bitbucket.org/hudson/magic-lantern/branch/recovery) and build the dumper (make sure disp_dump() is being called) for your camera to dump (lets call it master)
- save the "rom_dump" module sources
- switch to unified and compile the "rom_dump" module there
- put it into your fully supported camera (lets call it slave)
- master: boot rescue system
- slave: aim your lens towards the master camera
- slave: fine tune the focus to get a good separation between dots (check corners too!)

hint: if the "all dot" display time on master is too short for proper lens setup, reboot it and do one more phase.
generally its a good idea to simply let the master run its dumping process and checking on slave that all positions (4 corners and center)
give a proper white dot and close to black lines inbetween

- as soon the "all dot" display has fully faded on master and you have a black screen, start the module on slave (Debug -> Dump memory)
- slave: waits for the first 16 "setup dots" to appear one after another
- slave: waits for corner dots to calculate dimensions
- slave: waits for all dots

then it should work.
later when you are done with dumping, use the unpack_block (gcc -o unpack_block -c unpack_block.c) utility to re-assemble a proper ROM file.

hints:
- i worked with a zoom lens which worked fine
- disable IS (!!) as it will slightly move all the time
- use a power supply, it will take some time ;)  (dont remember, maybe 4 hours?)
- when you ran out of battery, modify the dump area in the dumper and start where it ended before
- when you have multiple partial rom dumps (DUMP.BIN) due to some troubles, just concatenate all files using e.g. cat command and pass it to unpack_block
- unpack_block will only pick blocks that weren't damaged and have a valid CRC
- if a fly passes by, then you might miss a block or two ;)
- if you constantly have trouble due to lens quality or a scratch in display, head to disp_dump.c (https://bitbucket.org/hudson/magic-lantern/commits/987a55497b04f9bd6f722e8845a0196677f913d9?at=recovery#Lsrc/disp_dump.cT16) and play with the values (e.g. increase BIT_WIDTH/BIT_SPACING and decrease BIT_COUNT_* so all will fit on screen)
- when modifying parameters in the dumper, the module will autodetect these changes (transmitted via setup dots)
Title: Re: Portable ROM dumper
Post by: dfort on October 22, 2017, 02:50:22 AM
Wow, that is so cool. I've got to try it.

So the 7D "master" is set up and working properly because it is displaying the patterns. I thought something was wrong, then again I didn't wait 4 hours for it to finish running "The Matrix" like code.

I take it that the "slave" can be another ML enabled camera and doesn't have to be the same model, all that's need is for it to run the "rom_dump" module. That rom_dump module is in the recovery branch but you're saying it needs to be compiled in the unified branch--Hum, ok. Is the EOSM a fully supported camera? I've got power supplies for both the 7D and EOSM.

Need to do this when I've got plenty of time and when there are no flies in the room.  ;D
Title: Re: Portable ROM dumper
Post by: g3gg0 on October 22, 2017, 10:51:32 AM
So the 7D "master" is set up and working properly because it is displaying the patterns. I thought something was wrong, then again I didn't wait 4 hours for it to finish running "The Matrix" like code.
wondering what you want with the 7D - do you mean 7D2?

I take it that the "slave" can be another ML enabled camera and doesn't have to be the same model, all that's need is for it to run the "rom_dump" module. That rom_dump module is in the recovery branch but you're saying it needs to be compiled in the unified branch--Hum, ok. Is the EOSM a fully supported camera? I've got power supplies for both the 7D and EOSM.

exactly, any camera where the raw backend works, should be fine.
i used the raw backend because of the resolution you get there. HD bufs could also work, but i felt like raw is the more supported choice ;)

just try it and report :)

when starting the module, it needs a possibly pitch black image. at least everything should have the same dark color.
it then does a full-screen averaging to get the base (off) color. then it waits for some dots to appear. 16 dots to be exact.
the significant jump in brightness is considered as "on" level.

after all 16 dots were located, they are used to communicate the dimensions of the matrix.
and then the matrix size is determined. etc.
Title: Re: Portable ROM dumper
Post by: 12georgiadis on October 22, 2017, 01:32:54 PM
wondering what you want with the 7D - do you mean 7D2?
G3gg0, I think it's about the 7DmkI which can become beast with a great crop_rec_4k port. It has the potential to handle all the features seen with the 5dmkIII.

Title: Re: Portable ROM dumper
Post by: g3gg0 on October 22, 2017, 06:05:58 PM
hmh why then run the rom dumper - you can simply run ML ?
Title: Re: Portable ROM dumper
Post by: dfort on October 22, 2017, 06:36:34 PM
I'd like to take a look at the 2.0.6 firmware for the 7D. Maybe there's something in there that is missing from 2.0.3? Doing a firmware update on this camera seems to be at the bottom of everyone's list so I'm not really pushing that though I do tend to be a bottom feeder when it comes to things to do in ML.

Besides, if running that master/slave setup is the way to get a dump out of some of the new cameras it wouldn't hurt getting familiar with the process.
Title: Re: Portable ROM dumper
Post by: g3gg0 on October 22, 2017, 06:54:33 PM
okay then it is experimenting with stuff, got it :)
Title: Re: Portable ROM dumper
Post by: dfort on October 23, 2017, 06:28:22 AM
Having issues getting this to work. First of all, here's my setup:

(https://farm5.staticflickr.com/4458/24019142318_ba236c85d5.jpg) (https://flic.kr/p/CAuocq)

The 7D is the "master" and the EOSM is the "slave." The image looks nice and sharp--using a 55mm f/2.8 Nikon Micro Nikkor. Never mind the low battery warning, that always happens with this power adapter using the wimpy 120 volts AC here in the U.S.

(https://farm5.staticflickr.com/4501/37823285006_ca73562db4.jpg) (https://flic.kr/p/ZCjdTy)

I tried several times but the display always shows "C0" and there is no card write activity. Screenshot didn't work with rom_dump running so I had to do a phone photo.

(https://farm5.staticflickr.com/4448/37872115941_31db62fffb.jpg) (https://flic.kr/p/ZGCuCe)

Looking at the code it should be giving feedback on the screen like "waiting for SEQAAAA", "waiting for idle", "waiting for XRES", etc. If it is having problems it should be printing out something like "Data calib dot 2 detection failed" but I'm getting nothing like that, just "C0".

- if you constantly have trouble due to lens quality or a scratch in display, head to disp_dump.c (https://bitbucket.org/hudson/magic-lantern/commits/987a55497b04f9bd6f722e8845a0196677f913d9?at=recovery#Lsrc/disp_dump.cT16) and play with the values (e.g. increase BIT_WIDTH/BIT_SPACING and decrease BIT_COUNT_* so all will fit on screen)
- when modifying parameters in the dumper, the module will autodetect these changes (transmitted via setup dots)

I doubt theres any problem with the lens and there's no scratch in the display. Everything looks nice and sharp and all the dots seem to be showing up on the screen. I guess it wouldn't hurt playing around with the values but thought I'd report my first failed attempt.

Any tips like what ISO setting and shutter angle worked best? Maybe it needs FPS override? The EOSM always defaults to mv720 at 29.97 so maybe that's the problem?
Title: Re: Portable ROM dumper
Post by: Kharak on October 23, 2017, 09:41:14 AM
Sorry, but is that the Matrix ? :)
Title: Re: Portable ROM dumper
Post by: dfort on October 23, 2017, 01:07:01 PM
The Matrix is a 1999 science fiction action film. Google "Matrix code" and you'll see a resemblance between the graphics used in the movie and the patterns on the LiveView screen of the "master" camera when using the display based ROM dumper.

Back on topic -- switched the "slave" camera to a 5D3.113 and it started working right away. One camera is on AC, the other on battery. I'm starting by doing a dump of the 7D.203 to see if it matches the regular ML dump before venturing into the still unknown 2.0.5 and 2.0.6 firmware. I've got a rather busy week so it might take me a while before I have anything to report.

[EDIT] Other than to report that a fully charged battery won't make it all the way through the process so it will either have to be done in sections following the instructions posted by @g3gg0 or both cameras need to be on external power supplies. The 4MB file processed with unpack_block and disassembled with disassemble.pl (http://www.magiclantern.fm/forum/index.php?topic=12177.0) matches perfectly with the ML produced ROM dump.
Title: Re: Portable ROM dumper
Post by: g3gg0 on October 23, 2017, 04:47:22 PM
okay thanks for your feedback!

so on the EOS M the module fails to find the C0-dot in the raw area. maybe i am missing some code to init raw buffer access.
never checked how to do that on EOS M.

good to hear that it finally worked :)
Title: Re: Portable ROM dumper
Post by: nikfreak on October 23, 2017, 05:38:42 PM
dfort, send me the 2.0.6 dump as private link by PM plz once you have it. Thx in advance
Title: Re: Portable ROM dumper
Post by: dfort on October 23, 2017, 06:25:19 PM
Sure will @nikfreak it will probably take me a few days until I've got another large block of time to run this. Since it takes so long and the test with the 7D.203 went so well I'm going straight for the 7D.206.

@g3gg0 - It would be nice if it worked on the EOSM, it is working perfectly on the 5D3.113.

One thing I noticed is that if I compile for the EOSM in the recovery branch that camera also calls disp_dump() and starts displaying the dots on the screen instead of dumping the ROM. No problem if I back up to a changeset before you added the experimental display based dumper.
Title: Re: Portable ROM dumper
Post by: g3gg0 on October 23, 2017, 07:32:12 PM
btw, the reason why i made this weird dumper...

i wanted to dump the 5Ds and i didnt want to use a GPIO to toggle a LED that is sampled with a LDR, then fed into a sound card, then saved as audio, then reconstruct the data.
with a data rate that is smaller than snail mail ;)

the QR code experiment tend to have failed because binary transfers are a pain in the ass and the decoders were even more pain in the ass.
(thats why i will close this branch soon)

and if we have such a nice display and soooooo many light sensitive sensor elements in a camera (the image sensor...), why then not use them along with a module that decodes the data?


possible ToDos:
 - not only OOK (on off keying), but also use color and brightness (0%, 50%, 100% for each RGB) gives 9 possible states and thus ~3 bits of information per pixel
 - compress the binary data, which would imply we are no longer block aligned
Title: Re: Portable ROM dumper
Post by: sombree on October 25, 2017, 05:01:36 PM
Should it be this fast?
https://drive.google.com/open?id=0B1pPwPvDPAHRQWR2RXAxYWlfZjQ
Title: Re: Portable ROM dumper
Post by: g3gg0 on October 25, 2017, 08:46:27 PM
woohooo, thats quite fast :D
no, it should be slower by a factor of 10 or so...

which camera model? 80D?
Title: Re: Portable ROM dumper
Post by: sombree on October 25, 2017, 09:11:49 PM
Yes, it's 80D with autoexec.bin built from recovery branch with some modifications (https://hastebin.com/ugavohazir.diff) :)
Title: Re: Portable ROM dumper
Post by: dfort on October 26, 2017, 12:03:04 AM
Wish the dump I'm attempting would go that fast. Here's my new setup:

(https://farm5.staticflickr.com/4512/37902523892_b7dce0ec5e.jpg) (https://flic.kr/p/ZKjkRf)

After the master (7D) displays all dots twice and goes to a black screen I start the slave (5D3) and it goes through the setup. I tried getting a shot of some of that exciting action:

(https://farm5.staticflickr.com/4505/37879972566_e663aeedab.jpg) (https://flic.kr/p/ZHjL8h)

Then it starts--Yay!

(https://farm5.staticflickr.com/4472/37879974016_32bbcf15f5.jpg) (https://flic.kr/p/ZHjLyh)

Though at some point is usually starts failing:

(https://farm5.staticflickr.com/4493/37224095844_81ae12abf6.jpg) (https://flic.kr/p/YHndPW)

This goes on for hours and eventually the 5D3 seems to be telling me to get a life because it shuts down--always when I'm not watching.

Tried to see what I could get out of a failed dump:

Code: [Select]
./unpack_block DUMP.BIN 1.BIN
First header received, block size is 738 byte
Base address is 0xFC000000
Missing block of 0x1CD4 bytes at 0xFC0002E2-0xFC001FB5, assuming 0xFF
Missing block of 0x1CD4 bytes at 0xFC002298-0xFC003F6B, assuming 0xFF
Missing block of 0xBB62 bytes at 0xFC00424E-0xFC00FDAF, assuming 0xFF
0xFC152E92: Data CRC error, trying to correct
  Bit #296 error, fixed
0xFC1545A2: Data CRC error, trying to correct
  Bit #296 error, fixed
...
0xFC17524E: Data CRC error, trying to correct
  [ERROR] correction failed
  => Unpack failed for 0x2E2 bytes at 0xFC17524E-0xFC17552F, assuming 0xA5
Dumped 0xFC000000-0xFC175530

Instead of figuring out where to restart, how much to dump, etc. I thought why not dump smaller chunks so I looked through the code and it looks like this is where the change should be made:

reboot.c
Code: [Select]
    print_line(COLOR_CYAN, 3, "  Magic Lantern Rescue\n");
    print_line(COLOR_CYAN, 3, " ----------------------------\n");
   
    print_model();
    prop_diag();
    print_bootflags();
    find_gaonisoy();
   
    disp_dump(0xFC000000, 0x02000000);

    #if defined(CONFIG_BOOT_DUMPER)
        /* pick one method for dumping the ROM */
        #if defined(CONFIG_FULLFAT)
            dump_rom_with_fullfat();
        #else
            dump_rom_with_canon_routines();
        #endif
    #endif

Specifically, disp_dump(0xF8000000, 0x01000000) should be what we need for a ROM1.BIN from the 7D, right? Hum--if the model is recognized maybe the dump addresses could be adjusted? So far I believe that the only currently supported camera that the portable dumper cannot be used on is the 7D. Also, it would be nice if the display dumper would be called only if the portable dumper fails.

Those options look intriguing so I looked up where they are being defined and found this in the recovery branch Makefile.user.default:

Code: [Select]
# Work in progress - load ML as position-independent code (PIC)
CONFIG_PIC          = n

# recovery: use the fullfat library for low-level card access
CONFIG_FULLFAT      = n

# recovery: print CHDK cpuinfo
CONFIG_CPUINFO      = n

# recovery: dump ROM from bootloader (either FULLFAT or Canon routines)
CONFIG_BOOT_DUMPER  = n

# recovery: enable boot flag from bootloader
CONFIG_BOOT_BOOTFLAG = n

Should I add any of this to my Makefile.user?
Title: Re: Portable ROM dumper
Post by: g3gg0 on October 26, 2017, 08:58:20 PM
well my personal favorite is:
leave it as a swiss army knife - you can never make it fit for every purpose.
people who are into dumping cameras, will know how and what to change.
(dont know if alex agrees on this)

but it is absolutely a good idea to make it "detecting" if the portable dumping failed and then show the dot-dumper.
Title: Re: Portable ROM dumper
Post by: a1ex on October 26, 2017, 09:02:18 PM
There is a catch - usually, when the portable dumper fails, it freezes. To implement a fallback mechanism, one would probably have to set up a hardware timer and an interrupt - not worth the hassle IMO.

I'd just add a config option for the new dumper (already done locally, just not yet tested).
Title: Re: Portable ROM dumper
Post by: dfort on October 27, 2017, 01:01:04 AM
I was thinking of all sorts of options but a simple one that builds either a portable dumper or a display dumper should be good enough.

Success!

Got a good looking 7D.206 ROM1.BIN using the display dumper. No luck getting a ROM0.BIN but that shouldn't be a show stopper.

...disp_dump(0xF8000000, 0x01000000) should be what we need for a ROM1.BIN from the 7D, right?

I was right about that but it kept failing at different points. Sure, unpack_block should fix it but after 24 hours it was still working on the dump so I tried doing it in smaller chunks.

Code: [Select]
disp_dump(0xF8000000, 0x00400000);
disp_dump(0xF8400000, 0x00400000);
disp_dump(0xF8800000, 0x00400000);
disp_dump(0xF8C00000, 0x00400000);

This gave me four DUMP.BIN files that could be concatenated together:

Code: [Select]
cat 1-0xF8000000/DUMP.BIN 2-0xF8400000/DUMP.BIN 3-0xF8800000/DUMP.BIN 4-0xF8C00000/DUMP.BIN > ROM_concatenated.BIN

Then run through unpack_block:

Code: [Select]
./unpack_block ROM_concatenated.BIN ROM1.BIN
First header received, block size is 738 byte
Base address is 0xF8000000
Missing block of 0x1CD4 bytes at 0xF80002E2-0xF8001FB5, assuming 0xFF
Missing block of 0x1CD4 bytes at 0xF8002298-0xF8003F6B, assuming 0xFF
Missing block of 0xBB62 bytes at 0xF800424E-0xF800FDAF, assuming 0xFF
Missing block of 0xFA bytes at 0xF83FFF06-0xF83FFFFF, assuming 0xFF
Missing block of 0x3323D4 bytes at 0xF85DDA56-0xF890FE29, assuming 0xFF
Missing block of 0xBAD7A bytes at 0xF8945098-0xF89FFE11, assuming 0xFF
Missing block of 0xFA bytes at 0xF8BFFF06-0xF8BFFFFF, assuming 0xFF
Missing block of 0x8FC7C bytes at 0xF8C301B6-0xF8CBFE31, assuming 0xFF
Missing block of 0xD7214 bytes at 0xF8EE8D8C-0xF8FBFF9F, assuming 0xFF
Dumped 0xF8000000-0xF8FFF660

Yay! A good looking ROM1.BIN file that is the right size.

(https://farm5.staticflickr.com/4458/37952674741_35424aa64f.jpg) (https://flic.kr/p/ZPKnX4)

I tried that same trick to get a ROM0.BIN but everything I tried (F0000000 - F7000000) ended up like this:

(https://farm5.staticflickr.com/4448/37952673741_7d26353cc8.jpg) (https://flic.kr/p/ZPKnDP)

Go figure.

In any case I'm happy with what I've got out of this exercise.
Title: Re: Portable ROM dumper
Post by: g3gg0 on October 27, 2017, 05:58:36 PM
wondering why it failed in the first place.
maybe some counter overflowing? hmmm