Author Topic: How to run Magic Lantern into QEMU?!...  (Read 64229 times)

kennetrunner

  • New to the forum
  • *
  • Posts: 15
Re: How to run Magic Lantern into QEMU?!...
« Reply #150 on: May 11, 2017, 05:07:41 PM »
@a1ex   I have qemu 2.9, and tried to install the patch contrib/qemu/qemu-2.5.0.patch   - which fails :-(
Do you have an updated patch file ?

.. ken
1300D.110

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 9831
  • 5D Mark Free
Re: How to run Magic Lantern into QEMU?!...
« Reply #151 on: May 11, 2017, 09:58:20 PM »
Yes, on the qemu-2.9.0 branch.

However, some parts of the test suite fail in 2.9.0 (but are OK on 2.5.0), and 2.5.0 has a few more features on the EOS side (in particular, the memory tracing and related tools). Syncing them shouldn't be very difficult.

2.9.0 can be interesting for EOS M5, as it might emulate multi-core CPUs a little better (note: the self-test fails for this model) and for porting the code to other QEMU variants (e.g. Panda 2.0 for its binary instrumentation plugins, or Xilinx QEMU*) if you want to look into UHS), but for the moment I've switched back to 2.5.0 (simply because it works and I'm used to it).

*) Xilinx QEMU is currently based on 2.6.0, so the patch for it is probably something in-between.

kennetrunner

  • New to the forum
  • *
  • Posts: 15
Re: How to run Magic Lantern into QEMU?!...
« Reply #152 on: May 12, 2017, 02:48:58 PM »
So, after a ton of hurdles I shelved my own attempts at getting qemu 2.9 running...
Instead I downloaded the vbox image from http://www.magiclantern.fm/forum/index.php?topic=7579.msg134989#msg134989 and compiled qemu 2.5 and it **seemed** to complete successfully...

However, now when I run
Code: [Select]
./run_canon_fw.sh 550D I get
Code: [Select]
qemu-system-arm: unsupported machine type
When I list the supported machine types, the only Canon one to show up is canon-a1100

Seems like I'm missing a step, somewhere ?... any pointers ?

1300D.110

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 9831
  • 5D Mark Free
Re: How to run Magic Lantern into QEMU?!...
« Reply #153 on: May 12, 2017, 03:50:03 PM »
The vanilla QEMU does not include our modifications.

See the install log from the build server for reference.

dfort

  • Hero Member
  • *****
  • Posts: 1590
Re: How to run Magic Lantern into QEMU?!...
« Reply #154 on: May 17, 2017, 03:08:44 PM »
I just started into trying to get the qemu branch working on macOS Sierra 10.12.4.

The ML repository needs to be in the home (user) directory or it won't work. The script in magic-lantern/contrib/qemu/install.sh seems to run fine up until this point:

Code: [Select]
Next steps:
===========

1) Compile QEMU

   cd /Users/rosiefort/qemu/qemu-2.5.0
   ../configure_eos.sh
grep: /proc/cpuinfo: No such file or directory
   make -j

Running ../configure_eos.sh (without the cpuinfo) or "./configure --target-list=arm-softmmu --disable-docs --enable-sdl" results in:

Code: [Select]
ERROR: Cocoa and SDL UIs cannot both be enabled at once
QEMU can also be installed via Homebrew but of course it doesn't include the Magic Lantern modifications.

Maybe it is a matter of figuring out how to configure the "cpuinfo" for the Mac?
Code: [Select]
sysctl -n machdep.cpu.brand_string
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz

Though it is probably much more than just that. Maybe it is the clang compiler? Any hints?
EOSM.202 EOSM.203 EOSM2.103 700D.115 5D3.*

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 9831
  • 5D Mark Free
Re: How to run Magic Lantern into QEMU?!...
« Reply #155 on: May 17, 2017, 03:11:35 PM »
Comment out the affected line; plain make is fine.

Or, try something like this:

Code: [Select]
echo "   make -j`grep -c processor /proc/cpuinfo || sysctl -n hw.ncpu || echo 1`"

Compiling without SDL should be fine (it's used that way on the build server, where it runs without GUI).

dfort

  • Hero Member
  • *****
  • Posts: 1590
Re: How to run Magic Lantern into QEMU?!...
« Reply #156 on: May 17, 2017, 03:52:07 PM »
Code: [Select]
echo "   make -j`grep -c processor /proc/cpuinfo || sysctl -n hw.ncpu || echo 1`"
grep: /proc/cpuinfo: No such file or directory
   make -j8

Ah ha, so the processor on this machine is:
Code: [Select]
sysctl -n hw.ncpu
8

So I tried:
Code: [Select]
make -j8
Please call configure before running make!
make: *** No rule to make target `trace/generated-events.h', needed by `Makefile'.  Stop.
make: *** Waiting for unfinished jobs....
make: *** [config-host.mak] Error 1


Changing configure_eos.sh so it uses gcc-5 instead of clang gives the warning noted in the script but it still comes up with:
Code: [Select]
C++ compiler c++ does not work with C compiler gcc-5 --std=gnu99
Disabling C++ specific optional code

ERROR: Cocoa and SDL UIs cannot both be enabled at once

It looks like the issue is setting this up so the configure script will select only the SDL UI. Need to dig into it a bit more.
EOSM.202 EOSM.203 EOSM2.103 700D.115 5D3.*

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 9831
  • 5D Mark Free
Re: How to run Magic Lantern into QEMU?!...
« Reply #157 on: May 17, 2017, 04:00:25 PM »
To specify the C++ compiler, try this:

Code: [Select]
CC="clang" CXX="clang++" \
    ./configure --target-list=arm-softmmu --disable-docs --disable-sdl \
    --extra-cflags="-Wno-error=deprecated-declarations" $*

It almost compiles with clang, except for a tiny function which you can safely comment out, and some warnings. Will look into them.

For SDL, try --disable-sdl.

dfort

  • Hero Member
  • *****
  • Posts: 1590
Re: How to run Magic Lantern into QEMU?!...
« Reply #158 on: May 17, 2017, 04:48:09 PM »
Almost there. Are you seeing these same errors?
Code: [Select]
/Users/rosiefort/qemu/qemu-2.5.0/hw/arm/../eos/dbi/logging.c:196:9: error:
      function definition is not allowed here
        {
        ^
/Users/rosiefort/qemu/qemu-2.5.0/hw/arm/../eos/dbi/logging.c:200:16: error: use
      of undeclared identifier 'close_idc'; did you mean 'closedir'?
        atexit(close_idc);
               ^~~~~~~~~
               closedir
/usr/include/dirent.h:102:5: note: 'closedir' declared here
int closedir(DIR *) __DARWIN_ALIAS(closedir);
    ^
2 errors generated.
make[1]: *** [hw/arm/../eos/dbi/logging.o] Error 1
make[1]: *** Waiting for unfinished jobs....
/Users/rosiefort/qemu/qemu-2.5.0/hw/arm/../eos/dbi/memcheck.c:91:20: warning:
      unused function 'set_uninitialized' [-Wunused-function]
static inline void set_uninitialized(uint32_t addr)
                   ^
/Users/rosiefort/qemu/qemu-2.5.0/hw/arm/../eos/dbi/memcheck.c:99:20: warning:
      unused function 'set_freed' [-Wunused-function]
static inline void set_freed(uint32_t addr)
                   ^
/Users/rosiefort/qemu/qemu-2.5.0/hw/arm/../eos/dbi/memcheck.c:107:20: warning:
      unused function 'clr_freed' [-Wunused-function]
static inline void clr_freed(uint32_t addr)
                   ^
3 warnings generated.
1 warning generated.
make: *** [subdir-arm-softmmu] Error 2

Hint on where to find the tiny function that can be commented out?
EOSM.202 EOSM.203 EOSM2.103 700D.115 5D3.*

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 9831
  • 5D Mark Free
Re: How to run Magic Lantern into QEMU?!...
« Reply #159 on: May 17, 2017, 05:01:06 PM »
Yes, exactly there.

dfort

  • Hero Member
  • *****
  • Posts: 1590
Re: How to run Magic Lantern into QEMU?!...
« Reply #160 on: May 17, 2017, 05:34:25 PM »
Tried something different. Since cr2hdr doesn't compile on clang we need to install a different compiler on the Mac.
Code: [Select]
CC="gcc-5" \
    ./configure --target-list=arm-softmmu --disable-docs --disable-sdl \
    --extra-cflags="-Wno-error=deprecated-declarations" $*

Then ran "make -j8" like before and it compiled. There were some other errors but it finished compiling.

Got to run off to work now but hope to get back to this soon.
EOSM.202 EOSM.203 EOSM2.103 700D.115 5D3.*

dfort

  • Hero Member
  • *****
  • Posts: 1590
Re: How to run Magic Lantern into QEMU?!...
« Reply #161 on: May 18, 2017, 12:51:49 AM »
These are the errors I got running the gcc-5 compiler from the Homebrew distribution on the Mac:
Code: [Select]
In file included from /usr/include/Availability.h:190:0,
                 from /usr/include/stdlib.h:61,
                 from /Users/rosiefort/qemu/qemu-2.5.0/include/qemu/osdep.h:35,
                 from /Users/rosiefort/qemu/qemu-2.5.0/include/qemu-common.h:15,
                 from block/raw-posix.c:24:
/System/Library/Frameworks/CoreFoundation.framework/Headers/CFDateFormatter.h:53:34: error: expected ',' or '}' before '__attribute__'
     kCFISO8601DateFormatWithYear API_AVAILABLE(macosx(10.12), ios(10.0), watchos(3.0), tvos(10.0)) = (1UL << 0),
                                  ^
/System/Library/Frameworks/CoreFoundation.framework/Headers/CFDateFormatter.h:80:126: error: 'introduced' undeclared here (not in a function)
 CFDateFormatterRef CFDateFormatterCreateISO8601Formatter(CFAllocatorRef allocator, CFISO8601DateFormatOptions formatOptions) API_AVAILABLE(macosx(10.12), ios(10.0), watchos(3.0), tvos(10.0));
                                                                                                                              ^
  CC    crypto/cipher.o
/System/Library/Frameworks/CoreFoundation.framework/Headers/CFURL.h:777:39: error: 'deprecated' undeclared here (not in a function)
 const CFStringRef kCFURLLabelColorKey API_DEPRECATED("Use NSURLLabelColorKey", macosx(10.6, 10.12), ios(4.0, 10.0), watchos(2.0, 3.0), tvos(9.0, 10.0));
                                       ^
/System/Library/Frameworks/CoreFoundation.framework/Headers/CFURL.h:777:39: error: 'message' undeclared here (not in a function)
 const CFStringRef kCFURLLabelColorKey API_DEPRECATED("Use NSURLLabelColorKey", macosx(10.6, 10.12), ios(4.0, 10.0), watchos(2.0, 3.0), tvos(9.0, 10.0));

It did finish compiling so did it work? Followed instructions with the rom dumps and mounted the sd card image and this is what happened:
Code: [Select]
./run_canon_fw.sh EOSM
./run_canon_fw.sh: line 10: losetup: command not found
usage: grep [-abcDEFGHhIiJLlmnOoqRSsUVvwxZ] [-A num] [-B num] [-C[num]]
[-e pattern] [-f file] [--binary-files=value] [--color=when]
[--context[=num]] [--directories=action] [--label] [--line-buffered]
[--null] [pattern] [file ...]
./run_canon_fw.sh: line 10: losetup: command not found
usage: grep [-abcDEFGHhIiJLlmnOoqRSsUVvwxZ] [-A num] [-B num] [-C[num]]
[-e pattern] [-f file] [--binary-files=value] [--color=when]
[--context[=num]] [--directories=action] [--label] [--line-buffered]
[--null] [pattern] [file ...]
CHK version_gen.h
  LINK  qemu-ga
  LINK  ivshmem-server
  CC    block/raw-posix.o
In file included from /usr/include/Availability.h:190:0,
                 from /usr/include/stdlib.h:61,
                 from /Users/rosiefort/qemu/qemu-2.5.0/include/qemu/osdep.h:35,
                 from /Users/rosiefort/qemu/qemu-2.5.0/include/qemu-common.h:15,
                 from block/raw-posix.c:24:
/System/Library/Frameworks/CoreFoundation.framework/Headers/CFDateFormatter.h:53:34: error: expected ',' or '}' before '__attribute__'
     kCFISO8601DateFormatWithYear API_AVAILABLE(macosx(10.12), ios(10.0), watchos(3.0), tvos(10.0)) = (1UL << 0),
                                  ^
/System/Library/Frameworks/CoreFoundation.framework/Headers/CFDateFormatter.h:80:126: error: 'introduced' undeclared here (not in a function)
 CFDateFormatterRef CFDateFormatterCreateISO8601Formatter(CFAllocatorRef allocator, CFISO8601DateFormatOptions formatOptions) API_AVAILABLE(macosx(10.12), ios(10.0), watchos(3.0), tvos(10.0));
                                                                                                                              ^
/System/Library/Frameworks/CoreFoundation.framework/Headers/CFURL.h:777:39: error: 'deprecated' undeclared here (not in a function)
 const CFStringRef kCFURLLabelColorKey API_DEPRECATED("Use NSURLLabelColorKey", macosx(10.6, 10.12), ios(4.0, 10.0), watchos(2.0, 3.0), tvos(9.0, 10.0));
                                       ^
/System/Library/Frameworks/CoreFoundation.framework/Headers/CFURL.h:777:39: error: 'message' undeclared here (not in a function)
 const CFStringRef kCFURLLabelColorKey API_DEPRECATED("Use NSURLLabelColorKey", macosx(10.6, 10.12), ios(4.0, 10.0), watchos(2.0, 3.0), tvos(9.0, 10.0));
                                       ^
block/raw-posix.c: In function 'hdev_open':
block/raw-posix.c:2129:23: warning: variable 'kernResult' set but not used [-Wunused-but-set-variable]
         kern_return_t kernResult;
                       ^
make: *** [block/raw-posix.o] Error 1

Looks like the same error messages.

Note that I'm running macosx 10.12.4 and there doesn't seem to be a readily available losetup. Is this necessary?
EOSM.202 EOSM.203 EOSM2.103 700D.115 5D3.*

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 9831
  • 5D Mark Free
Re: How to run Magic Lantern into QEMU?!...
« Reply #162 on: May 18, 2017, 12:57:19 AM »
Searching this error message gives http://stackoverflow.com/questions/41143981/macos-sierra-corefoundation-error-while-compiling-wxwidgets-for-simspark

To me, it looks like qemu didn't finish compiling.

dfort

  • Hero Member
  • *****
  • Posts: 1590
Re: How to run Magic Lantern into QEMU?!...
« Reply #163 on: May 22, 2017, 12:11:20 AM »
QEMU on Mac -- Yay!


This might be a real hack, here's what I did. First of all the CoreFoundation.framework issues affect the Homebrew gcc-5 compiler so I went back to the macos clang.

~/qemu/configure_eos.sh
Code: [Select]
CC="clang" CXX="clang++" \
    ./configure --target-list=arm-softmmu --disable-docs --disable-sdl \
    --extra-cflags="-Wno-error=deprecated-declarations" $*

The compilation errors we had with clang were because of this section of code, so I commented it out.

~/qemu/qemu-2.5.0/hw/eos/dbi/logging.c line#194
Code: [Select]
        /* QEMU is usually closed with CTRL-C, so call this when finished */
//        void close_idc(void)
//        {
//            fprintf(idc, "}\n");
//            fclose(idc);
//        }
//        atexit(close_idc);

Loaded up the EOSM ROM0.BIN, ROM1.BIN and had to create an SFDATA.BIN using the sf_dump module and placed them in ~/qemu/EOSM. Mounted the sd.img and ran:

Code: [Select]
./run_canon_fw.sh EOSM
Now just because it launches doesn't mean that it is working properly but I wanted to share my progress.
EOSM.202 EOSM.203 EOSM2.103 700D.115 5D3.*

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 9831
  • 5D Mark Free
Re: How to run Magic Lantern into QEMU?!...
« Reply #164 on: May 22, 2017, 11:45:27 AM »
Nice. This is the cocoa interface, right? Do you have menus that allow you to switch between serial console, VGA and so on?

BTW, after this commit it should install cleanly on clang and/or Mac.


dfort

  • Hero Member
  • *****
  • Posts: 1590
Re: How to run Magic Lantern into QEMU?!...
« Reply #165 on: May 23, 2017, 02:51:37 PM »
Speaking for all of us Mac users -- Thanks!

Here are a few things Mac users who want to get this working should watch out for.

install.sh prints some instructions at the end of the process. On osx it prints out the grep error instead of keeping silent.
Code: [Select]
1) Compile QEMU

   cd /Users/rosiefort/qemu/qemu-2.5.0
   ../configure_eos.sh
grep: /proc/cpuinfo: No such file or directory
   make -j8

It is caused by this line:

install.sh
Code: [Select]
echo "   make -j`grep -c processor /proc/cpuinfo || sysctl -n hw.ncpu || echo 1`"

Switching the positions around eliminated the grep error on osx and created a sysctl command not found on Linux so I tried this to suppress the error messages and it worked:
Code: [Select]
echo "   make -j`grep -c processor /proc/cpuinfo 2> /dev/null || sysctl -n hw.ncpu 2> /dev/null || echo 1`"


This instruction doesn't work for osx:
Code: [Select]
3) Mount the included SD (or CF) image (you may use mount.sh)mount.sh calls "kpartx" which isn't available for osx. Mounting sd.img can be done by simply double clicking the sd.img icon but I'm not sure if that worked properly because when running qemu it shows this:
Code: [Select]
SD: CMD12 in a wrong state
[SDIO] Error
SD: CMD12 in a wrong state
[SDIO] Error
SD LOAD OK.
Open file for read : AUTOEXEC.BIN
SD: CMD12 in a wrong state
[SDIO] Error
SD: CMD12 in a wrong state
[SDIO] Error

I looked for an alternate for kpartx and maybe hdiutil will work? I haven't figured out if there is some special way it needs to be invoked to mount the img files as loopback devices but either of these seem to work:
Code: [Select]
hdiutil mount sd.img
or
hdiutil attach sd.img


Back to install.sh, at the end it is supposed to list some camera models but on osx it displays this:

Code: [Select]
   Note: Canon GUI emulation (menu navigation, no LiveView) only works on
   usage: grep [-abcDEFGHhIiJLlmnOoqRSsUVvwxZ] [-A num] [-B num] [-C[num]]
[-e pattern] [-f file] [--binary-files=value] [--color=when]
[--context[=num]] [--directories=action] [--label] [--line-buffered]
[--null] [pattern] [file ...]

That problem is partially because the install script calls tests/run_tests.sh. Here is the output of running that script on osx:
https://pastebin.com/0T5GLRh5

Not sure if any of this would prevent qemu from running properly on osx. I'm just starting on this. Running the minimal autoexc.bin works on every ROM dump I've got but I haven't been able to go much beyond that.
EOSM.202 EOSM.203 EOSM2.103 700D.115 5D3.*

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 9831
  • 5D Mark Free
Re: How to run Magic Lantern into QEMU?!...
« Reply #166 on: May 24, 2017, 10:05:48 AM »
Code: [Select]
SD: CMD12 in a wrong state
[SDIO] Error

That's fine, I get those too. I'm not sure if this indicates an emulation bug / incomplete model, or it's just how the (simplified) SD driver used in bootloader is supposed to behave (note: CMD12 is STOP_TRANSMISSION).

In the main firmware, you'll get a similar error about CMD1; this one is OK, as it appears to be the way Canon code probes for MMC cards. Regular SD cards are probably not supposed to reply to CMD1 outside the SPI transfer mode (at least that's my understanding), so the SD emulation backend prints some messages. The full conversation can be watched with -d sdcf (or -d sdcf,io for more details) and cross-checked with the SD specification.

If the reader is familiar with SD protocol, I'd welcome any insights (in particular, for the UHS-I initialization sequence).


Note that I'm running macosx 10.12.4 and there doesn't seem to be a readily available losetup. Is this necessary?

To avoid corrupting the data on the SD image, it's best to prevent starting the emulation if the image is mounted by the user (as there will be two processes wanting to write on the same card image, without knowing about each other). This is done on Linux by checking /proc/mounts (losetup is used with "loopback" devices - that is, when mounting an image as a filesystem). Don't know how this works on Mac, and don't know whether my method is portable across other Linux distributions either.



The other issues appear to be (more or less) just annoyances (as they don't print what the user expects to see) and probably easy to fix.