RE contributions

[Indy]

hi,

Just to let you know, I put on the bitbucket most of my python scripts (the public ones)
https://bitbucket.org/hudson/magic-lantern/src/fa4b9a00d0ca859ea86a4a0c9b0b144ef2e9b02b/contrib/indy/readme.TXT?at=unified

"it is working at least for me" ;-)

Indy

[1%]

The fir scripts let you get a bin from the canon updates? I need to look in 5d3 fir.... also that property dumper dumps props with values? That would be a godsend.

[Indy]

updates need to be decrypted first.
the script (dec_fir.py) is not public because it contains keys and crypto algorithms from Canon.

Indy

[g3gg0]

http://upload.g3gg0.de/pub_files/fdc99f4a738e331e63009bdc7ac12f71/PropertyEditor.exe

it lets you dump properties into some xml format.
and maybe somewhen i will add importing, if it is needed

windows:  run "PropertyEditor.exe [ROM-Filename]"
linux: run (with mono)  "mono PropertyEditor.exe [ROM-Filename]"

[1%]

Awesome have to use all of these on the 6D bins

[g3gg0]

did just check with 60D and 600D.
7D seems to save properties somewhere else. (well, on master it seems)

[1%]

I get:

Code Select Expand



[I] Autodetected property offset: 00080000
[Block: 00, Length: 00001054, Flag: 00000000] [Base: 00080000]
  [E] Invalid block size
[Block: 01, Length: 00000000, Flag: 00000000] [Base: 00081000]
  [E] Invalid block size
[Block: 02, Length: 00001054, Flag: 00000000] [Base: 00082000]
  [E] Invalid block size
[Block: 03, Length: 00000000, Flag: 00000000] [Base: 00083000]
  [E] Invalid block size
[Block: 04, Length: 00001054, Flag: 00000000] [Base: 00084000]
  [E] Invalid block size
[Block: 05, Length: 00000000, Flag: 00000000] [Base: 00085000]
  [E] Invalid block size
[Block: 06, Length: 00001054, Flag: 00000000] [Base: 00086000]
  [E] Invalid block size
[Block: 07, Length: 00000000, Flag: 00000000] [Base: 00087000]
  [E] Invalid block size
[Block: 08, Length: 00001054, Flag: 00000000] [Base: 00088000]
  [E] Invalid block size
[Block: 09, Length: 00000000, Flag: 00000000] [Base: 00089000]
  [E] Invalid block size
[Block: 0A, Length: 00001054, Flag: 00000000] [Base: 0008A000]
  [E] Invalid block size
[Block: 0B, Length: 00000000, Flag: 00000000] [Base: 0008B000]
  [E] Invalid block size
[Block: 0C, Length: 00001054, Flag: 00000000] [Base: 0008C000]
  [E] Invalid block size
[Block: 0D, Length: 00000000, Flag: 00000000] [Base: 0008D000]
  [E] Invalid block size
[Block: 0E, Length: 00001054, Flag: 00000000] [Base: 0008E000]
  [E] Invalid block size
[Block: 0F, Length: 00000000, Flag: 00000000] [Base: 0008F000]
  [E] Invalid block size
[Block: 10, Length: 00001054, Flag: 00000000] [Base: 00090000]
  [E] Invalid block size
[Block: 11, Length: 00000000, Flag: 00000000] [Base: 00091000]
  [E] Invalid block size
[Block: 12, Length: 00001054, Flag: 00000000] [Base: 00092000]
  [E] Invalid block size
[Block: 13, Length: 00000000, Flag: 00000000] [Base: 00093000]
  [E] Invalid block size
[Block: 14, Length: 00001054, Flag: 00000000] [Base: 00094000]
  [E] Invalid block size
[Block: 15, Length: 00000000, Flag: 00000000] [Base: 00095000]
  [E] Invalid block size
[Block: 16, Length: 00001054, Flag: 00000000] [Base: 00096000]
  [E] Invalid block size
[Block: 17, Length: 00000000, Flag: 00000000] [Base: 00097000]
  [E] Invalid block size
[Block: 18, Length: 00001054, Flag: 00000000] [Base: 00098000]
  [E] Invalid block size
[Block: 19, Length: 00000000, Flag: 00000000] [Base: 00099000]
  [E] Invalid block size
[Block: 1A, Length: 00001054, Flag: 00000000] [Base: 0009A000]
  [E] Invalid block size
[Block: 1B, Length: 00000000, Flag: 00000000] [Base: 0009B000]
  [E] Invalid block size
[Block: 1C, Length: 00001054, Flag: 00000000] [Base: 0009C000]
  [E] Invalid block size
[Block: 1D, Length: 00000000, Flag: 00000000] [Base: 0009D000]
  [E] Invalid block size
[Block: 1E, Length: 00001054, Flag: 00000000] [Base: 0009E000]
  [E] Invalid block size
[Block: 1F, Length: 00000000, Flag: 00000000] [Base: 0009F000]
  [E] Invalid block size


[g3gg0]

can you look for the firmware version string like e.g. "2.0.3" in the full rom dump?
then look for a find where you see a *lot* FF FF FF bytes in front of the location.

see e.g.: http://upload.g3gg0.de/pub_files/2d39eb8caed13175612063dcc89241f4/properties.PNG

[g3gg0]

i see, 6D has changed property format.
updating...

[g3gg0]

solved. 6D uses blocks larger than 0x1000 bytes.
now i am autodetecting the block size by the first block's size.
this will fail if the first block is empty, then you will have to specify it manually. (either 0x1000 or 0x2000)

http://upload.g3gg0.de/pub_files/cbd65f7af070379954c4ec3da697c196/PropertyEditor.exe

[1%]

Thanks, I think it dumped everything now.

Output is slightly different though.

Ie 0x80010006

would be

    [Property: 02010006, Length: 0000000C] [Base: 000A6178]
?


Length is different... or are props numbered differently in rom?

[g3gg0]

well, there are a few more property sections (just realized that not all is in the autodetected one)

for example 600D:

TUNE: 0xBE0000
FIX:  0xB60000
RING: 0xAE0000
RASEN: 0xB00000
LENS:  0xEC0000

all with different block sized etc. just figuring out how to autodetect that ....

Code Select Expand


v3 = PROPAD_CreateFROMPropertyHandle((unsigned int *)&v14, 1, (int)&unk_F8B60000, 0x10000u, 8u, 0x80000u, 0x1000000);
  if ( v3 & 1 )
    DryosDebugMsg(139, 6, "PROPAD_CreateFROMPropertyHandle : FIX (%#x)", v3);
  v14 = 0x2000000;
  v4 = PROPAD_CreateFROMPropertyHandle((unsigned int *)&v14, 1, (int)&unk_F8AE0000, 0x10000u, 2u, 0x1000u, 0x1000000);
  if ( v4 & 1 )
    DryosDebugMsg(139, 6, "PROPAD_CreateFROMPropertyHandle : RING (%#x)", v4);
  v14 = 0x4000000;
  v15 = 83886080;
  v16 = 234881024;
  v5 = PROPAD_CreateFROMPropertyHandle((unsigned int *)&v14, 3, (int)&unk_F8B00000, 0x10000u, 6u, 0x20000u, 0x1000000);
  if ( v5 & 1 )
    DryosDebugMsg(139, 6, "PROPAD_CreateFROMPropertyHandle : RASEN (%#x)", v5);
  v14 = 184549376;
  v6 = PROPAD_CreateFROMPropertyHandle((unsigned int *)&v14, 1, (int)&unk_F8EC0000, 0x10000u, 6u, 0x20000u, 0x1000000);
  if ( v6 & 1 )
    DryosDebugMsg(139, 6, "PROPAD_CreateFROMPropertyHandle : LENS (%#x)", v6);


[1%]

6D has those sections too with separate dumps like on 600D.. I only tried getting the main props but instead got a 400mb 00.bin Looks a bit like ram dump. The other functions for tune/fix/ring/rasen look to be the same.

[g3gg0]

update

http://magiclantern.fm/modules/modules/PropertyEditor.exe/PropertyEditor.exe

use it with arguments:
1: ROM-filename
2: FROM base address
3: blocksize
4: number of blocks

How to get the parameters:
look for a call to CreateFROMPropertyHandle and check the arguments
the number of blocks (argument 4:) is "sectorsize * sectors / blocksize"

int PROPAD_CreateFROMPropertyHandle(unsigned int *prop_types, int prop_count, unsigned int base_address, unsigned int sectorsize, unsigned int sectors, unsigned int blocksize, int a7)

if you are not sure how to calculate, you can just pass 0 as block count or completely omit it.
then the dumper will only dump until the first block marked "valid" is found.
if you supply that count, it will dump all blocks, even old and invalid ones.

for 7D:
PropertyEditor.exe ROM1.bin 0x910000 0x040000 0x1
PropertyEditor.exe ROM1.bin 0xA00000 0x2C0000 0x1

Type  Offset  Properties
FIX:  0x910000 (0x00000000)
TUNE: 0xA00000 (0x01000000)

for 600D:
PropertyEditor.exe ROM1.bin 0xAE0000 0x001000 0x20
PropertyEditor.exe ROM1.bin 0xB00000 0x020000 0x3
PropertyEditor.exe ROM1.bin 0xB60000 0x080000 0x1
PropertyEditor.exe ROM1.bin 0xBE0000 0x2C0000 0x1
PropertyEditor.exe ROM1.bin 0xEC0000 0x020000 0x3

Type  Offset  Properties
RING: 0xAE0000 (0x02000000)
RASEN 0xB00000 (0x04000000 0x05000000 0x0E000000)
FIX:  0xB60000 (0x00000000)
TUNE: 0xBE0000 (0x01000000)
LENS  0xEC0000 (0x0B000000)


for 60D:
PropertyEditor.exe ROM1.bin 0x9D0000 0x001000 0x20
PropertyEditor.exe ROM1.bin 0x9F0000 0x020000 0x3
PropertyEditor.exe ROM1.bin 0xA50000 0x080000 0x1
PropertyEditor.exe ROM1.bin 0xAD0000 0x370000 0x1
PropertyEditor.exe ROM1.bin 0x00C000 0x000800 0x4

Type  Offset  Properties
RING: 0x9D0000 (0x02000000)
RASEN 0x9F0000 (0x04000000 0x05000000 0x0E000000)
FIX:  0xA50000 (0x00000000)
TUNE: 0xAD0000 (0x01000000)
LENS: 0xEC0000 (0x0B000000)
CUST: 0x00C000 (0x03000000)


for 6D:
PropertyEditor.exe ROM1.bin 0x080000 0x002000 0x20
PropertyEditor.exe ROM0.bin 0x060000 0x020000 0x3
PropertyEditor.exe ROM0.bin 0x0C0000 0x100000 0x1
PropertyEditor.exe ROM0.bin 0x1E0000 0x020000 0x3
PropertyEditor.exe ROM0.bin 0x020000 0x001000 0x20


Type  ROM  Offset  Properties
RING: ROM1 0x080000 (0x02000000)
RASEN ROM0 0x060000  (0x04000000 0x05000000 0x0E000000)
FIX:  ROM0 0x0C0000  (0x00000000)
LENS: ROM0 0x1E0000  (0x0B000000)
CUST: ROM0 0x020000 (0x03000000)

[1%]

Rom0 is the ram segment or...

I just get same dump from 0xff000000 for both rom0.bin and rom1.bin

The roms dissected from the firmware updates start at 0xff00 or at program area (5d3  0xf80c0000)?

[g3gg0]

ROM0: 0xF0000000 - 0xF7FFFFFF (when camera has a ROM0, then its 8 or 16M, so its 0xF0000000-0xF0FFFFFF)
ROM1: 0xF8000000 - 0xFFFFFFFF (most cameras have 16M, so its 0xF8000000-0xF8FFFFFF)

[Indy]

I'm happy to see it is useful !
again and as usual, excellent work G3gg0!

any idea about lens00.bin content ?
it seems it contains vignetting and chromatic aberration tables for correction...
I can provide 60D and 550D data if needed.

would it be useful to create a custom update with modified properties / bitmap / strings ?
yes it is risky.

Indy

[g3gg0]

upload it to http://upload.g3gg0.de/ - i will look what is inside.

i played with this thought a long time.
did the same for nokia phones - providing tools for extracting, modifying and repacking language packs.
that was, because nokia sold some phones only in some countries and did not put translation for e.g. europe into it.
so there was a "market" for such translation tools.

not sure if the userbase we are focusing on needs such translations.

[g3gg0]

LENS content seems to be blocks of 0xA90 byte size.
blocks end with a sequence of: 22 22 22 22

these blocks are split into two parts, separated by 44 44 44 44. first is 0x360 bytes, second part is 0x730.
second part consists mostly of words with 0x18 byte size, some with 0x20 byte

thats what i can see from the hex file

[g3gg0]

LENS format: (property 0x0B000000, can be modified with EOSUtiliy's abberation profile manager)

0x00-0x30 header
0x00: 0x0F3C
0x02: number of LENS_ENTRIES
0x04: 0x0030 (this header size)
0x06: 0x0080 (number of reserved LENS_ENTRIES)

0x30-0x830 LENS_ENTRIES:
0x10 bytes per entry
format:
0x00: 0x00000034 lens_id, see PROP_LENS second word. (0x0032001D = nifty fifty, 0x00000034 = 18-55 IS II)
0x04: 0x000000FF
0x08: 0x00000000
0x0C: start offset in LENS_DATA

0x830-... LENS_DATA: (offset depends on header information)
[header with 0x20 byte size]
0x000: 0x0000
0x002: 0x0001
0x004: 0x0012 (min focal length)
0x006: 0x0037 (max focal length)
0x008: 0x00FA (min focus distance in mm. 450, 250, 280, 340, 380, ....)
0x00A: 0x0000
..
0x01E: 0x0000

[section with 0x340 byte size]
0x000: 0x0012 (focal length 1)
0x002: 0x0017 (focal length 2)
0x004: 0x0021 (focal length 3)
0x006: 0x0037 (focal length 4)
0x008: 0x0FA0 (unknown 4000, 2222, ...)
0x00A: 0x0A6B (unknown 2667, 1481, ..., its above value / 1.5)
0x00C: 0x0535 (unknown 1333, 741, ..., its above value / 2)
0x00E: 0x0000
..
0x33C: 0x44444444

[section with 0xF0 byte size]
0x000: unknown
...
0x010: 0x0012 (focal length 1)
0x012: 0x0017 (focal length 2)
0x014: 0x0021 (focal length 3)
0x016: 0x0037 (focal length 4)
...
0x0EC: 0x33333333

[section with 0x640 byte size]
0x000: 0x0012 (focal length 1)
0x002: 0x0017 (focal length 2)
0x004: 0x0021 (focal length 3)
0x006: 0x0037 (focal length 4)
...
0x63C: 0x22222222

[1%]

would it be useful to create a custom update with modified properties / bitmap / strings ?

That would be cool. Especially to add back properties missing for certain things, etc. Also fix annoying things with cannon FW.

Ditch the canon fw/interface completely if at all possible.... but that would be a mountain of work.

[Indy]

I was on the path with my parse_lens*.py scripts. I'll study your findings tomorrow.
thank you, you definitely went further !

[coutts]

Thank you for this Indy, great 

[g3gg0]

here some parsed lens data how i understand it:

http://upload.g3gg0.de/pub_files/a13e3d003e3620bac8641b1e7278e253/lens_bin.lensxml

using this tool:

http://upload.g3gg0.de/pub_files/421dd6883b808b44c8dd85f4fe54dd63/LensDumper.exe

[g3gg0]

update, added indy's structuring of lens data content