Portable ROM dumper

[dfort]

I'm just trying to get familiar with the 7D and going through every stub and constant is one way to do that. Updating the firmware at this point isn't a priority, getting it ported to crop_rec_4k is, though it won't be as easy as the 6D and 650D. Of course if you're working on it the odds are good.

Back on topic, I did a little more digging into the recovery branch and found a way to compile a portable ROM dumper.

Code: [Select]

# recovery (portable display test, ROM dumper, CPU info...)
hg update recovery -C
cd ../../platform/portable.000
make install ML_MODULES_DYNAMIC= CONFIG_BOOT_DUMPER=y
With the latest commit on the recovery branch [1cfee5fb3c40] all I get is noise on the screen, this happens in both QEMU and on the camera.



This first appeared in commit 987a55497b04, disp_dump: added experimental display based dumper.

So stepping back one commit I can make a portable ROM dumper that works in QEMU but not on the camera. It is a little different than the pre-compiled dumper posted by a1ex on the first post of this topic.



Regressing all the way back to the big bang of the recovery branch I still couldn't get a ROM dumper that works in camera on the 7D.

Another issue I've been having is not being able to get to the Canon menu, much less the ML menu, in QEMU but that's a different topic.

[a1ex]

Well, 7D is a different beast. ML was ported to this camera a long time ago - check the old wiki for notes.

The recent dumper experiments are for 5DS (and they are also be useful for 7D2). Both are dual digic models, with the same quirk: the portable ROM dumper works in QEMU, but not on the camera. You need a second camera to decode the blinks.

Emulation would require starting two instances of QEMU and modeling the communication between them somehow. I remember reading about a QEMU fork with multiprocessor support (maybe the Xilinx version, or something based on that). Low priority from my side, for now.

[dfort]

I found old Google Group discussions like this one and this that pre-dates the old wiki.

Then there's this CHDK universal dumper that looks interesting. It runs on "CanonBasic" scripting language and is listed as working on the 7D as well as most other EOS cameras. I tried following the tutorials but no luck.

Oh well, though I discovered something new but what's old is new to me.

[a1ex]

EOS firmware doesn't have Canon Basic, though DIGIC 5 models apparently have a similar scripting language. It's not present on DIGIC 6, unlikely to be in D7, so for me there's very little reason to look into it.

Canon Basic dumper is only for PowerShot firmware (that includes EOS M3 and newer models).

[g3gg0]

there is an advanced display based dumper available, allowing reasonable transfer speeds.

here a vid: https://www.youtube.com/watch?time_continue=117&v=9mNhph9BfNA
the current implementation is quite a bit faster and doesnt take so long during dot calibration iirc.

how it works:
"master" camera booted in dumper, displays some patterns.
"slave" camera, in video mode, scans the screen and waits for the dots to appear.
after they went through setup, ROM content is shown on screen and slave camera writes a plain ROM image to its card structured dump file, that must be "assembled" on PC later.

how to use:
- get the recovery branch and build the dumper (make sure disp_dump() is being called) for your camera to dump (lets call it master)
- save the "rom_dump" module sources
- switch to unified and compile the "rom_dump" module there
- put it into your fully supported camera (lets call it slave)
- master: boot rescue system
- slave: aim your lens towards the master camera
- slave: fine tune the focus to get a good separation between dots (check corners too!)

hint: if the "all dot" display time on master is too short for proper lens setup, reboot it and do one more phase.
generally its a good idea to simply let the master run its dumping process and checking on slave that all positions (4 corners and center)
give a proper white dot and close to black lines inbetween

- as soon the "all dot" display has fully faded on master and you have a black screen, start the module on slave (Debug -> Dump memory)
- slave: waits for the first 16 "setup dots" to appear one after another
- slave: waits for corner dots to calculate dimensions
- slave: waits for all dots

then it should work.
later when you are done with dumping, use the unpack_block (gcc -o unpack_block -c unpack_block.c) utility to re-assemble a proper ROM file.

hints:
- i worked with a zoom lens which worked fine
- disable IS (!!) as it will slightly move all the time
- use a power supply, it will take some time   (dont remember, maybe 4 hours?)
- when you ran out of battery, modify the dump area in the dumper and start where it ended before
- when you have multiple partial rom dumps (DUMP.BIN) due to some troubles, just concatenate all files using e.g. cat command and pass it to unpack_block
- unpack_block will only pick blocks that weren't damaged and have a valid CRC
- if a fly passes by, then you might miss a block or two
- if you constantly have trouble due to lens quality or a scratch in display, head to disp_dump.c and play with the values (e.g. increase BIT_WIDTH/BIT_SPACING and decrease BIT_COUNT_* so all will fit on screen)
- when modifying parameters in the dumper, the module will autodetect these changes (transmitted via setup dots)

[dfort]

Wow, that is so cool. I've got to try it.

So the 7D "master" is set up and working properly because it is displaying the patterns. I thought something was wrong, then again I didn't wait 4 hours for it to finish running "The Matrix" like code.

I take it that the "slave" can be another ML enabled camera and doesn't have to be the same model, all that's need is for it to run the "rom_dump" module. That rom_dump module is in the recovery branch but you're saying it needs to be compiled in the unified branch--Hum, ok. Is the EOSM a fully supported camera? I've got power supplies for both the 7D and EOSM.

Need to do this when I've got plenty of time and when there are no flies in the room. 

[g3gg0]

So the 7D "master" is set up and working properly because it is displaying the patterns. I thought something was wrong, then again I didn't wait 4 hours for it to finish running "The Matrix" like code.

wondering what you want with the 7D - do you mean 7D2?

I take it that the "slave" can be another ML enabled camera and doesn't have to be the same model, all that's need is for it to run the "rom_dump" module. That rom_dump module is in the recovery branch but you're saying it needs to be compiled in the unified branch--Hum, ok. Is the EOSM a fully supported camera? I've got power supplies for both the 7D and EOSM.

exactly, any camera where the raw backend works, should be fine.
i used the raw backend because of the resolution you get there. HD bufs could also work, but i felt like raw is the more supported choice

just try it and report

when starting the module, it needs a possibly pitch black image. at least everything should have the same dark color.
it then does a full-screen averaging to get the base (off) color. then it waits for some dots to appear. 16 dots to be exact.
the significant jump in brightness is considered as "on" level.

after all 16 dots were located, they are used to communicate the dimensions of the matrix.
and then the matrix size is determined. etc.

[12georgiadis]

wondering what you want with the 7D - do you mean 7D2?
G3gg0, I think it's about the 7DmkI which can become beast with a great crop_rec_4k port. It has the potential to handle all the features seen with the 5dmkIII.

[g3gg0]

hmh why then run the rom dumper - you can simply run ML ?

[dfort]

I'd like to take a look at the 2.0.6 firmware for the 7D. Maybe there's something in there that is missing from 2.0.3? Doing a firmware update on this camera seems to be at the bottom of everyone's list so I'm not really pushing that though I do tend to be a bottom feeder when it comes to things to do in ML.

Besides, if running that master/slave setup is the way to get a dump out of some of the new cameras it wouldn't hurt getting familiar with the process.

[g3gg0]

okay then it is experimenting with stuff, got it

[dfort]

Having issues getting this to work. First of all, here's my setup:



The 7D is the "master" and the EOSM is the "slave." The image looks nice and sharp--using a 55mm f/2.8 Nikon Micro Nikkor. Never mind the low battery warning, that always happens with this power adapter using the wimpy 120 volts AC here in the U.S.



I tried several times but the display always shows "C0" and there is no card write activity. Screenshot didn't work with rom_dump running so I had to do a phone photo.



Looking at the code it should be giving feedback on the screen like "waiting for SEQAAAA", "waiting for idle", "waiting for XRES", etc. If it is having problems it should be printing out something like "Data calib dot 2 detection failed" but I'm getting nothing like that, just "C0".

- if you constantly have trouble due to lens quality or a scratch in display, head to disp_dump.c and play with the values (e.g. increase BIT_WIDTH/BIT_SPACING and decrease BIT_COUNT_* so all will fit on screen)
- when modifying parameters in the dumper, the module will autodetect these changes (transmitted via setup dots)

I doubt theres any problem with the lens and there's no scratch in the display. Everything looks nice and sharp and all the dots seem to be showing up on the screen. I guess it wouldn't hurt playing around with the values but thought I'd report my first failed attempt.

Any tips like what ISO setting and shutter angle worked best? Maybe it needs FPS override? The EOSM always defaults to mv720 at 29.97 so maybe that's the problem?

[Kharak]

Sorry, but is that the Matrix ?

[dfort]

The Matrix is a 1999 science fiction action film. Google "Matrix code" and you'll see a resemblance between the graphics used in the movie and the patterns on the LiveView screen of the "master" camera when using the display based ROM dumper.

Back on topic -- switched the "slave" camera to a 5D3.113 and it started working right away. One camera is on AC, the other on battery. I'm starting by doing a dump of the 7D.203 to see if it matches the regular ML dump before venturing into the still unknown 2.0.5 and 2.0.6 firmware. I've got a rather busy week so it might take me a while before I have anything to report.

[EDIT] Other than to report that a fully charged battery won't make it all the way through the process so it will either have to be done in sections following the instructions posted by @g3gg0 or both cameras need to be on external power supplies. The 4MB file processed with unpack_block and disassembled with disassemble.pl matches perfectly with the ML produced ROM dump.

[g3gg0]

okay thanks for your feedback!

so on the EOS M the module fails to find the C0-dot in the raw area. maybe i am missing some code to init raw buffer access.
never checked how to do that on EOS M.

good to hear that it finally worked

[nikfreak]

dfort, send me the 2.0.6 dump as private link by PM plz once you have it. Thx in advance

[dfort]

Sure will @nikfreak it will probably take me a few days until I've got another large block of time to run this. Since it takes so long and the test with the 7D.203 went so well I'm going straight for the 7D.206.

@g3gg0 - It would be nice if it worked on the EOSM, it is working perfectly on the 5D3.113.

One thing I noticed is that if I compile for the EOSM in the recovery branch that camera also calls disp_dump() and starts displaying the dots on the screen instead of dumping the ROM. No problem if I back up to a changeset before you added the experimental display based dumper.

[g3gg0]

btw, the reason why i made this weird dumper...

i wanted to dump the 5Ds and i didnt want to use a GPIO to toggle a LED that is sampled with a LDR, then fed into a sound card, then saved as audio, then reconstruct the data.
with a data rate that is smaller than snail mail

the QR code experiment tend to have failed because binary transfers are a pain in the ass and the decoders were even more pain in the ass.
(thats why i will close this branch soon)

and if we have such a nice display and soooooo many light sensitive sensor elements in a camera (the image sensor...), why then not use them along with a module that decodes the data?


possible ToDos:
 - not only OOK (on off keying), but also use color and brightness (0%, 50%, 100% for each RGB) gives 9 possible states and thus ~3 bits of information per pixel
 - compress the binary data, which would imply we are no longer block aligned

[sombree]

Should it be this fast?
https://drive.google.com/open?id=0B1pPwPvDPAHRQWR2RXAxYWlfZjQ

[g3gg0]

woohooo, thats quite fast
no, it should be slower by a factor of 10 or so...

which camera model? 80D?

[sombree]

Yes, it's 80D with autoexec.bin built from recovery branch with some modifications

[dfort]

Wish the dump I'm attempting would go that fast. Here's my new setup:



After the master (7D) displays all dots twice and goes to a black screen I start the slave (5D3) and it goes through the setup. I tried getting a shot of some of that exciting action:



Then it starts--Yay!



Though at some point is usually starts failing:



This goes on for hours and eventually the 5D3 seems to be telling me to get a life because it shuts down--always when I'm not watching.

Tried to see what I could get out of a failed dump:

Code: [Select]

./unpack_block DUMP.BIN 1.BIN
First header received, block size is 738 byte
Base address is 0xFC000000
Missing block of 0x1CD4 bytes at 0xFC0002E2-0xFC001FB5, assuming 0xFF
Missing block of 0x1CD4 bytes at 0xFC002298-0xFC003F6B, assuming 0xFF
Missing block of 0xBB62 bytes at 0xFC00424E-0xFC00FDAF, assuming 0xFF
0xFC152E92: Data CRC error, trying to correct
  Bit #296 error, fixed
0xFC1545A2: Data CRC error, trying to correct
  Bit #296 error, fixed
...
0xFC17524E: Data CRC error, trying to correct
  [ERROR] correction failed
  => Unpack failed for 0x2E2 bytes at 0xFC17524E-0xFC17552F, assuming 0xA5
Dumped 0xFC000000-0xFC175530

Instead of figuring out where to restart, how much to dump, etc. I thought why not dump smaller chunks so I looked through the code and it looks like this is where the change should be made:

reboot.c

Code: [Select]

    print_line(COLOR_CYAN, 3, "  Magic Lantern Rescue\n");
    print_line(COLOR_CYAN, 3, " ----------------------------\n");
   
    print_model();
    prop_diag();
    print_bootflags();
    find_gaonisoy();
   
    disp_dump(0xFC000000, 0x02000000);

    #if defined(CONFIG_BOOT_DUMPER)
        /* pick one method for dumping the ROM */
        #if defined(CONFIG_FULLFAT)
            dump_rom_with_fullfat();
        #else
            dump_rom_with_canon_routines();
        #endif
    #endif

Specifically, disp_dump(0xF8000000, 0x01000000) should be what we need for a ROM1.BIN from the 7D, right? Hum--if the model is recognized maybe the dump addresses could be adjusted? So far I believe that the only currently supported camera that the portable dumper cannot be used on is the 7D. Also, it would be nice if the display dumper would be called only if the portable dumper fails.

Those options look intriguing so I looked up where they are being defined and found this in the recovery branch Makefile.user.default:

Code: [Select]

# Work in progress - load ML as position-independent code (PIC)
CONFIG_PIC          = n

# recovery: use the fullfat library for low-level card access
CONFIG_FULLFAT      = n

# recovery: print CHDK cpuinfo
CONFIG_CPUINFO      = n

# recovery: dump ROM from bootloader (either FULLFAT or Canon routines)
CONFIG_BOOT_DUMPER  = n

# recovery: enable boot flag from bootloader
CONFIG_BOOT_BOOTFLAG = n

Should I add any of this to my Makefile.user?

[g3gg0]

well my personal favorite is:
leave it as a swiss army knife - you can never make it fit for every purpose.
people who are into dumping cameras, will know how and what to change.
(dont know if alex agrees on this)

but it is absolutely a good idea to make it "detecting" if the portable dumping failed and then show the dot-dumper.

[a1ex]

There is a catch - usually, when the portable dumper fails, it freezes. To implement a fallback mechanism, one would probably have to set up a hardware timer and an interrupt - not worth the hassle IMO.

I'd just add a config option for the new dumper (already done locally, just not yet tested).

[dfort]

I was thinking of all sorts of options but a simple one that builds either a portable dumper or a display dumper should be good enough.

Success!

Got a good looking 7D.206 ROM1.BIN using the display dumper. No luck getting a ROM0.BIN but that shouldn't be a show stopper.

...disp_dump(0xF8000000, 0x01000000) should be what we need for a ROM1.BIN from the 7D, right?

I was right about that but it kept failing at different points. Sure, unpack_block should fix it but after 24 hours it was still working on the dump so I tried doing it in smaller chunks.

Code: [Select]

disp_dump(0xF8000000, 0x00400000);
disp_dump(0xF8400000, 0x00400000);
disp_dump(0xF8800000, 0x00400000);
disp_dump(0xF8C00000, 0x00400000);
This gave me four DUMP.BIN files that could be concatenated together:

Code: [Select]

cat 1-0xF8000000/DUMP.BIN 2-0xF8400000/DUMP.BIN 3-0xF8800000/DUMP.BIN 4-0xF8C00000/DUMP.BIN > ROM_concatenated.BIN

Then run through unpack_block:

Code: [Select]

./unpack_block ROM_concatenated.BIN ROM1.BIN
First header received, block size is 738 byte
Base address is 0xF8000000
Missing block of 0x1CD4 bytes at 0xF80002E2-0xF8001FB5, assuming 0xFF
Missing block of 0x1CD4 bytes at 0xF8002298-0xF8003F6B, assuming 0xFF
Missing block of 0xBB62 bytes at 0xF800424E-0xF800FDAF, assuming 0xFF
Missing block of 0xFA bytes at 0xF83FFF06-0xF83FFFFF, assuming 0xFF
Missing block of 0x3323D4 bytes at 0xF85DDA56-0xF890FE29, assuming 0xFF
Missing block of 0xBAD7A bytes at 0xF8945098-0xF89FFE11, assuming 0xFF
Missing block of 0xFA bytes at 0xF8BFFF06-0xF8BFFFFF, assuming 0xFF
Missing block of 0x8FC7C bytes at 0xF8C301B6-0xF8CBFE31, assuming 0xFF
Missing block of 0xD7214 bytes at 0xF8EE8D8C-0xF8FBFF9F, assuming 0xFF
Dumped 0xF8000000-0xF8FFF660

Yay! A good looking ROM1.BIN file that is the right size.



I tried that same trick to get a ROM0.BIN but everything I tried (F0000000 - F7000000) ended up like this:



Go figure.

In any case I'm happy with what I've got out of this exercise.