Menu

Show posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Show posts Menu

Messages - nkls

#1
Feature Requests / Re: draw enhancements
May 27, 2017, 11:03:45 AM
It's an optimized implementation of the Midpoint circle algorithm. The wikipedia article describes the math behind it and gives some hints on how it can be changed to draw incomplete arcs. Sounds like a good C programming task! :)
#2
Nice work! Finally some running cameras to toy around with.  :D

I've managed to merge the latest changes into qemu v2.9.0-rc1. It's probably not a perfect merge, the camera gui hangs up more often than on v2.5.0 for me.

The patches to qemu code merged quite well, but someone with more knowledge of them should probably review whether they are ok/redundant in 2.9.0.

A major difference is that I've replaced the interrupt thread with a QEMUTimer. This solved som iothread lock bug, but I guess it should work better than the thread since it's now synced to the guest clock and not the host system.
#3
I got one of these the other day and have been playing around with it a bit.

Some interesting stuff:

There seems to be a telnet server in the firmware which can be enabled by adding:
TELNET=1
to the CONFIG file. If I connect I get a welcome message
Connected to 192.168.1.116.
Escape character is '^]'.
Welcome to FlashAir
ESC R4161 built 16:20:27, Oct  6 2014
>

but I'm not able to enter any commands. Maybe there is some login required?

There is a builtin fdump utility that can be used to dump memory content at startup.

COMMAND=fdump 0x00C00000 -f fdump.dat -l 0x00100000

Sometimes this hangs the card, but I've managed to dump segments as large as 0x00200000 bytes. There is also a -m flag (mount filesystem) I haven't tried yet which might make things more stable.

The processor seems to be some Toshiba MeP processor. The bytecode can be interpreted by binutils configured with --target=mep. (link for the lazy)

binutils-mep/bin/mep-objdump -D -b binary -mmep rom.bin
00000000 <.data>:
       0:       08 d8 01 00     jmp 0x100
       4:       18 df 08 00     jmp 0x8e2
       8:       00 00           nop
       a:       00 00           nop
       c:       00 00           nop
...
      fe:       00 00           nop
     100:       00 70           di
     102:       28 59           mov $9,40
     104:       59 79           stc $9,$cfg
     106:       00 59           mov $9,0
     108:       58 79           stc $9,$rpe
     10a:       03 eb a0 41     lw $11,(0x41a000)
     10e:       b5 cc 00 10     and3 $12,$11,0x1000
     112:       b5 cb 20 00     and3 $11,$11,0x20
     116:       2a 6b           srl $11,0x5
     118:       5a 6c           srl $12,0xb
     11a:       c0 1b           or $11,$12
...
     8e2:       48 df 08 00     jmp 0x8e8
     8e6:       02 70           ret
     8e8:       f0 cf ac ff     add3 $sp,$sp,-84
     8ec:       fa 00           sw $0,($sp)
     8ee:       06 41           sw $1,0x4($sp)
     8f0:       0a 42           sw $2,0x8($sp)
     8f2:       0e 43           sw $3,0xc($sp)
     8f4:       12 44           sw $4,0x10($sp)
     8f6:       16 45           sw $5,0x14($sp)
     8f8:       1a 46           sw $6,0x18($sp)
     8fa:       1e 47           sw $7,0x1c($sp)
     8fc:       22 48           sw $8,0x20($sp)
     8fe:       26 49           sw $9,0x24($sp)
     900:       2a 4a           sw $10,0x28($sp)
     902:       2e 4b           sw $11,0x2c($sp)
     904:       32 4c           sw $12,0x30($sp)
     906:       36 4d           sw $tp,0x34($sp)
     908:       3a 4e           sw $gp,0x38($sp)
     90a:       3e 4f           sw $sp,0x3c($sp)
     90c:       1a 71           ldc $1,$lp
     90e:       42 41           sw $1,0x40($sp)

Boot entry at 0x0 and interrupt at 0x4 perhaps?

There is also this iSDIO interface available. It would be interesting to expose it to ML and see what may come out of it.
#4
Seems like the script can't find rst2html, are you sure that it's installed?
#5
Interesting, the size difference seems to be the result of an alignment issues with some versions of ld. By adding the -N flag (Do not page align data, do not make text readonly) when linking autoexec the file size is decreased by almost 10% on my host os (arch linux), and by 5% on an ubuntu VM. As far as I can tell the only difference is that a bunch of zero padding is removed from autoexec.bin.


autoexec: $(ML_SRC_REBOOT_OBJ)
    $(call build,LD,$(LD) \
        -o $@ \
        -e _start \
        -N \
        -Ttext $(AUTOEXEC_BASE) \
        $^ \
        --cref \
        -Map=autoexec.map \
    ); \
    $(READELF) -l magiclantern | $(GREP) -C 2 MemSiz
#6
@count
Welcome back! All credits to nikfreak for the 100D port, he has done an awesome job of putting ML on our cameras.

My repo at https://bitbucket.org/niklastisk/magic-lantern is just a fork of his with dm-spy-experiments for the 100A firmware, and some other experiments.

Regarding QEMU, my latest 100D development is found at https://bitbucket.org/niklastisk/qemu/branch/eos-develop-new which currently is only a few commits (100D stuff) from the mainline repo. QEMU is somewhat complicated to use with 100D since it has an extra flash chip which must be dumped. For that there is a module, but the addresses are only ported/tested for 100D.100A. The GUI won't boot up yet, but I somehow managed to get the status screen showing once. Hopefully that's a sign of being close to getting the GUI running, but there's still stuff to do.

@nikfreak
May I ask what configuration you are using when compiling your builds? What gcc version and what Makefile.user flags are you using? My builds are always much larger than yours, and I've been trying to pinpoint the cause of the size difference.
#7
I've fixed the serial flash reads, turns out the QUAD mode discards the last 16 bytes in each 0x800-byte block. Now the property tables are read correctly.

https://bitbucket.org/niklastisk/qemu/branch/eos-develop-new

The spell sets for 100D are included as well, and I even got the GUI showing (once)! There still seems to be problems with the touch panel, but I don't know if they are critical or not.
#8
General Development / Re: Compiling ML with GCC 6
July 29, 2016, 06:50:22 PM
I can't see any problem with either of the changes really.

Using vsnprintf from dietlibc/musl seems like a good idea, especially since Canon's implementation is non-standard and obviously differ between DIGIC 4 and 5. And as mentioned, it's needed for printing 64-bit integers.

Would the stdin/stdout pipe into DryosDebugMsg then? And are you thinking about wrapping the FIO_* functions into FILE's such that we can use fopen and fwrite as well?
#9
General Development / Re: Compiling ML with GCC 6
July 29, 2016, 12:44:57 PM
I looked into my 100D rom again, and behold:


100D:FF6F47F0  BL      unk_double_0x61584 ; [R0,R1] = f(R0,R1,R2,R3)
100D:FF6F47F4  STRD    R0, [SP]
100D:FF6F47F8  ADR     R2, "***** GuiUtilOlcData GUI_GetOlcTvForCode8def UniqueNum(%lf)"
100D:FF6F47FC  MOV     R1, #6
100D:FF6F4800  MOV     R0, #131
100D:FF6F4804  BL      DryosDebugMsg


Seems like the double's are 64-bit after all, and the 100D's prinft support floats.

I can't find any explicit use of double in the src dir, but there are cases where doubles should be used implicitly:
src/fps-engio.c:    float frame_duration_orig = 1000.0 / default_fps;
I'm not sure if the -ffast-math flag will demote them here or not.

Anyways, floats are preferable unless high-precision is needed, e.g. more than roughly 7 decimal digits.
#10
General Development / Compiling ML with GCC 6
July 29, 2016, 11:12:21 AM
GCC 6 is released, and the linux distro I'm using (arch linux) has decided to upgrade my compiler. Ignoring the fact that I can downgrade/recompile/use a VM and the risk of using untested GCC versions, there is one major issue with the newest version:

The support for the -fshort-double flag is removed, i.e. demoting all doubles to floats.

As far as I understand, the ABI used by Canon actually has 32-bit doubles instead of 64-bit, so it's not just a flag we are using to make ML faster. I've compiled and tested ML with GCC 6 without the -fshort-double flag and all the basic stuff works, but I suspect that printf("%f") and such are broken, and that the binaries should be larger since there might be extra double precision code in there.

Are there other issues that should be expected? Any way to work around this?

My thought are:
Hope that GCC includes a -m32bit-doubles flag for the ARM architecture in the future.
Create a custom arm-eos ABI and recompile GCC.

It's not a big issue right now since we can use GCC 4/5, but in a few years we might be missing out on any new gcc goodies.
#11
I've managed to get the dm-spy-extra branch to run for 100D (fw 100A) with full logging, sources are at:
https://bitbucket.org/niklastisk/magic-lantern/branch/100D-mpu-spy

Still no success at emulating the 100D though, just including the correct mpu messages did not work out.


On a side note, me and a1ex discovered some interesting things about the memory management of the 100D's 256M RAM when using the CONFIG_MARK_UNUSED_MEMORY_AT_STARTUP configuration:

      p1=&memtest; p3=UNCACHEABLE(p1);
      init:00b1c75c:00:00: POINTER   p1         p2         p3
      init:00b1c77c:00:00: ADDRESS   0x00B7F204 0x10B7F204 0x40B7F204
      init:00b1c7a8:00:00: (init)    0x00000123 0x00000123 0x00000123
      init:00b1c7e0:00:00: memtest++ 0x00000124 0x00000123 0x00000123
      init:00b1c818:00:00: (*p1)++   0x00000125 0x00000123 0x00000123
      init:00b1c850:00:00: (*p2)++   0x00000125 0x00000123 0x00000123
      init:00b1c888:00:00: (*p3)++   0x00000125 0x00000124 0x00000124

(source code: http://pastebin.com/vHWcH9ij)

My interpretation is that (0x10000000 | ptr) creates a pointer that wraps around the 256M ram, but bypasses the cache and is read-only such that write attempts fail silently. This obviously won't work for 512M cameras. :)
#12
I had another look at the flash memory yesterday, and I figured out a few things. Your code appears to load the serial flash correctly for the 100D without changes, and it might be that it's just the spells causing the property errors. (Still no canon gui.)

The chip in my camera is a Winbond W25Q128FV (datasheet), and not a Macronix as I thought before. The Manufacturer Code and device ID is {0xEF,0x40,0x18} which is what I got from SPI instruction 9Fh. This doesn't matter much since it works anyways, but it's good to have the actual data sheet to base the flash emulation of.

There are two high-level interfaces (with the same debug names) for the serial flash. One uses SPI-only transfer with "serial layout", while the other is "QUAD" read which reads data through DMA, or SPI if the read block is small enough. The DMA read has this shifted data layout, maybe due to physical wiring limitations or obfuscation or whatnot. The code you posted is used by the property manager, and is from the QUAD interface, which has to un-shift the SPI data to ensure proper data layout.

This explains why the version string in the first block is readable -- it is read and written by the SPI-only interface which don't expect DMA layout. I've used the address of the SPI-only interface in my dumper, so the data in the dump should be the same as in the actual flash.

I've also tried to change the block size, and there is no difference between the data received from a read-all-at-once dump file and a 1024-byte fio_malloc'ed dumper.
#13
Wow, I'm impressed! Nice job! :) I won't be able to try it out in the upcoming two weeks, but I'll give it a go after that.

You are right about the sf data being manipulated in some way, I wondered what strange format would produce data like:

0000fff0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00010000: 0fff f000 0dcf 70f0 0000 0000 1d0f 70f0  ......p.......p.
00010010: 0000 0000 0040 3000 0000 0000 10c0 0000  .....@0.........
00010020: 0fff ffff f010 0000 10c0 0000 0fea fdcb  ................
00010030: a080 0000 1180 0000 0000 0000 0000 0000  ................
00010040: 0000 0000 0000 0000 0020 0000 1180 0000  ......... ......

but now that you say it it looks very half-byte shifted at 0x10000.

Interestingly, the first block contains the version string "4.2.1" which is not half-byte shifted.

00000000: 4603 0080 342e 322e 3100 0000 0000 0000  F...4.2.1.......
00000010: 0000 0000 ffff ffff ffff ffff ffff ffff  ................
00000020: ffff ffff ffff ffff ffff ffff ffff ffff  ................


Is it just me or is the write address not incremented in your fix?

So wait, is the data still unshifted when you use uncacheable memory? What happens with the data block exactly -- is the first half-byte always set to zero, and the last half-byte discarded? Any indication of the higher-level calls requesting more data than asked for, or is the last byte of a read always assumed to be bogus? (note to self)

You could also try to force it to only use non-DMA/only-DMA with gdb and see if that works better, I recall getting different results when doing that.

Quote from: a1ex on June 12, 2016, 12:47:44 PM
Guess we'll have to dump the RAM address where the serial flash contents is loaded (or maybe attempt to read the entire serial flash with a single call in the dumper).
It'd be tricky to allocate 16MB(?) for the full flash wouldn't it? It would also be interesting to see how e.g. one 2kb read compares to two 1kb reads, maybe we just need to overlap the block reads by a few bytes and unshift them to make it work.
#14
Nice to see things pushed to the main repo and that you've sorted out the open ends I left in there! It seems to work and looks correct, so I think it's all good.

I remember trying to get the debug messages showing through pure gdb for a while but then gave up and hacked it together in C instead. It might be possible to some extent but I bet it'll be ugly. Maybe count %-signs and branch to different cases?

The gdb-qemu link isn't ideal either, the current:
set *0xCF999001 = *0xCF999001
triggers the IO-handle when it reads from *0xCF999001 but not when it writes to it and I can't figure out why. Not that it matters much, but it would be nice to be able to send values to qemu through a single address.
#15
The chip install code is run quite early in the init task, this is how it looks in IDA:


As you can see there is a flag (derived from FROM) which determines what chip to install. On a second thought, those routines must be for setting up writing to main flash.

The function call before "SerialFlash Initialize" is definitely for SROM though, I guess there would be some similar call in 70D but not in 5D3? Maybe the 5D3 has more space in main flash memory and thus no need for it?
In 100D the output is

[SF] InstallSerialFlash 6 0xc022c0d4 0xf0000000 0x1000000 1

so it's similar.

I think I've got most of the protocol working, take a look in this file in 100D-testing. It gets messy once you try to emulate DMA since it is shared with the SD memory transfer.


Quick answer: (too much of a hack to push to bitbucket)

unsigned int eos_handle_tio ( unsigned int parm, EOSState *s, unsigned int address, unsigned char type, unsigned int value )
{
    unsigned int ret = 1;
    const char * msg = 0;
    int msg_arg1 = 0;

    switch(address & 0xFF)
    {
        case 0x00:
            if(type & MODE_WRITE)
            {
                if((value == 0x08 || value == 0x0A || value == 0x0D || (value >= 0x20 && value <= 0x7F)))
                {
                    printf("\x1B[31m%c\x1B[0m", value);
                    return 0;
                }
            }
            else
            {
                return 0;
            }
            break;

        case 0x04:
            msg = "Read byte: 0x%02X";
            msg_arg1 = s->tio_rxbyte & 0xFF;
            ret = s->tio_rxbyte & 0xFF;
            while (ret == 0 || ret == EOF) {
                ret = getchar();
            }
            return ret;
            break;
       
        case 0x08:
            /* quiet */
            return 0;

        case 0x14:
            if(type & MODE_WRITE)
            {
                if(value & 1)
                {
                    msg = "Reset RX indicator";
                    s->tio_rxbyte |= 0x100;
                    return 0; /* nkls: quiet */
                }
                else
                {
                    /* quiet */
                    return 0;
                }
            }
            else
            {
                if((s->tio_rxbyte & 0x100) == 0)
                {
                    msg = "Signalling RX indicator";
                    ret = 3;
                }
                else
                {
                    /* quiet */
                    return 3; /* nkls: signal something on the line */
                    //return 2;
                }
            }
            break;
    }

    io_log("TIO", s, address, type, value, ret, msg, msg_arg1, 0);
    return ret;
}

is there a hidden menu option in other firmwares as well?
#16
Thanks guys, and thanks for the info! I was trying to to get 100D working and I guess I learned a bit about the qemu internals on the way. I wouldn't call myself an expert on the system but I know my way around bigger C/linux projects like this. :)

I never realized the dm-spy-experiments dumped the mpu calls, I guess I have to make one of 100D when I get the time!

Yeah, I'm using IDA for the decompiling and code analysis.


About that serial flash though, I'm under the impression that it's only available in newer cameras (100D and 70D?), and that it is necessary to emulate them to get to the main dialog in qemu. But correct me if I'm wrong.

In 100D there are references to two flash chips PC28F128M29EWHF and K5C262GATMAF50 which are both 16MB iirc. There is a function "IsAddressSerialFlash" which is used by the property manager (or some flash access abstraction) to determine if the data is stored in flash rom (same as firmware) or serial flash rom (controlled over serial interface). I think I had it setup with DMA, interrupts and whatnot but I never made it to the main dialog. :( I'll give it another shot sometime but atm I'm too busy with other projects.

Here are some extracts from the factory menu, note the S and V options in FACTADJ and the SIO6 menu for accessing the serial rom.

FACTADJ

********** FACTORY ADJUSTMENT MENU VER 0.01 **********
0. Exit from Factory Adjustment
1. Leak Check
2. SDRAM Check
3. ROM Check
4. HDMI Check
5. Video Adjustment
6. ICU Version Check
7. Audio Check
8. Adjustment Data Display and Change
9. Check Flag Display and Initialization
A. ALL Check
B. ALLP Check
F. FROM ID Check
P. Power Domain Check
S. Serial Flash Check
V. Serial Flash Version Check
Z. Input_Device_Unique

S
Serial Flash Check.
346
Select OK or NG

V
SROM Version Check.
4.2.1
Select OK or NG


FROMUTILITY

************ FROMUTILITY MENU Ver 0.09 ************
[Type:346 Body:DC Revision:0.00]
0.Exit from FROM Menu
1.Erase Sector(0x20000)
2.Erase Chip
4.Write from card
5.Write from DRAM
6.Firm   flag 0xF8000000 0x00000000 ON
7.Boot   flag 0xF8000004 0x00000000 OFF
8.UpDate flag 0xF800000C 0xFFFFFFFF OFF
9.Create Boot Disk
A.Exec Program from card
B.ALL Block Unprotect
C.Connect card
G.Memory Dump
I.Write Data
J.Direct Jump
P.Power Domain
S.SROM Menu
U.Firm update
Z.RCbind.bin update

S

**** SROM(SIO6) Menu ****
0.Exit from SROM Menu
1.Erase Chip   0x01000000
2.Erase Block  0x0000F000
3.Erase Sector 0x00001000
4.Write Data
5.Write from Card
6.SROM Dump(SIO Read)
7.SROM Dump(QUAD Read)
8.Get Info
9.Get Version

8
8.Get Info 0x80000346

9
9.Get Version 4.2.1
#17
I've pushed my changes to qemu to my bitbucket now, it's located at:

https://bitbucket.org/niklastisk/qemu (branches: eos-develop and 100D-testing)

It was a while since I touched the code but I hope parts of it can be useful to someone at least.

Things I recall I've done:

  • updated qemu to 2.5.50, I haven't tried to merge with master in a while but it may work with the latest version
  • moved all eos related code to /hw/eos and moved sections of eos.c to their own files
  • DebugMsg printing, see eos_workspace for how to use it
  • tried to get 100D serial flash to work without much success, but the code is there
  • probably some other stuff, look at the commits...
#18
Nice work, nikfreak!

It's my first time using ML but I love it so far. :) No bugs to report other than whats been mentioned already.

I've made a module to dump the serial flash if anyone is interested, it only works for 100D.100A as of now, the addresses for the other versions must be ported and added to the source. It's probably only useful for developers, but I think it might be needed to run canon fw in qemu and I guess a dump could be used to restore the properties if they would get messed up at some point.
#19
Hello,

I'm trying to emulate 100D with qemu 2.3.0 and I'm running into some issues. I've got it to run quite far but there seems to be multiple issues and I'm not sure which one to try to fix next...

I've attached the log file for the longest I've got it running, and the major changes from what's in the repo is:

- Added addresses for 100D
- No boot flag, just canon stuff
- Asserts aren't fatal, they just print to serial and return to caller (GDB hack)
- DebugMsg prints to log (with color output, GDB/QEMU magic)
- Sequencer events are delayed if being propagated too early, to avoid errors (again some GDB magic)
- Added some SPI EEPROM code
- Grepped out the USB/HDMI/MIC clobber


So some questions:

Is there an SPI EEPROM in 60D?
The 100D firmware tries to load some properties from there and isn't happy with getting zeros. Not sure whether real data is necessary for the emulation, but it seems to be the reason why some asserts are raised.

There are some "unknown spells", should I try to figure out what they do? For the 60D, are the spell responses dumped from the camera or reversed from firmware?

It also complains about I2C and RTC at some points, but I don't think those are fatal errors.

Somehow it crashes to a $pc=4 state which I 'm trying to track down atm, but any thoughts on what to do to make it run further would be very appreciated. :)

This is the qemu log:
https://www.dropbox.com/s/y92sf7babtf3t2w/100D.log?dl=0
#20
I got to the point where I felt I couldn't get further without running ML on the camera, but then I gave up when I didn't have time to figure out how the bootdisk fir compilation works. I have experience from hacking/digging around in proprietary arm code from the old dumbphone days, so I should be able to do some good to this project if I just get the time.
#21
So it seems like the two versions of firmware 1.00 can be differentiated by their "ICU Firmware Version", one is 3.4.3 and one is 3.4.7.

I've pushed the stubs.S I've been working on to this repo, it also describes how you can determine what ICU version a dump has.
#22
Yeah, Canon made it hard for us on this one. Since there is no way to switch from one 1.00 to the other I guess we have to maintain two versions until Canon releases a firmware update.

Anyhow, if the code runs in QEMU would it run on the camera as well? Since there are stubs missing and constants are wrong it would barley be functional but would the ML menu show up if I compiled it and loaded it to the camera?
#23
There are actually two different firmwares with version number 1.00 around, it's been mentioned here and in fw-signature.h in the repo.

The MD5 of my roms are:

46226ec402c825a4796212e84e53846d  D100-ROM0.bin
489543fee0032a6e657005971c567903  D100-ROM1.bin

and I suspect that at least your checksum of ROM1.bin is different than mine.

Regarding the ROM_OFFSET, in firmware_entry I have:

ROM:FF0C0098                 LDR     R0, =sub_FFA9970C
ROM:FF0C009C                 LDR     R1, =assert     ; 0x1900
ROM:FF0C00A0                 LDR     R3, =unk_87690
ROM:FF0C00A4
ROM:FF0C00A4 loc_FF0C00A4                            ; CODE XREF: firmware_entry+B0
ROM:FF0C00A4                 CMP     R1, R3          ; Copy initial RAM
ROM:FF0C00A4                                         ; 0x1900-0x87690
ROM:FF0C00A8                 LDRCC   R2, [R0],#4
ROM:FF0C00AC                 STRCC   R2, [R1],#4
ROM:FF0C00B0                 BCC     loc_FF0C00A4
ROM:FF0C00B4                 LDR     R1, =0xC52E8
ROM:FF0C00B8                 MOV     R2, #0

so I have to use
#define RAM_OFFSET (0xFFA9970C - 0x1900)
to get the qemu specific things running.

Also, my signature in fw-signature.h evaluates to 0x2EBD343C even without QEMU, so to get signature checking to work I have to redefine SIG_100D_100.

Since the ML GUI is running in QEMU with these changes and all debug messages shows up in the console, I'm quite sure that they are correct.
#24
I've managed to get the GUI running in QEMU, and my camera has a different revision of 1.00 than the one you have been working on nikfreak.

The two revisions (or are there more?) of the 100D 1.00 seems to have everything same in RAM, i.e. relocated functions and static stuff are at the same addresses but the RAM_OFFSET is different. Functions in ROM are shuffled around though. Any thought on how we are to distinguish between the different 1.00's?