Compiling Magic Lantern on Windows 10 (using its Linux subsystem)

[g3gg0]

if someone wants to use an XServer on windows, i used to run vcXsrv and it works quite well.

[irvcobb]

As one with some Linux experience, no fear of the command line, and little compiling experience, my observations may be of value?

Starting with a clean Linux install on Win 10, I can get all the way to here with no problem:

Code Select Expand

qemu$ ./run_canon-fw.sh 600D
make: Entering directory '/home/irv/qemu/qemu-2.5.0'
Please call configure before running make!
Makefile:47: recipe for target 'config-host.mak' failed
make: *** [config-host.mak] Error 1
make: Leaving directory '/home/irv/qemu/qemu-2.5.0'


Then a cd qemu-2.5.0 and ./configure seem to sort things.

Did I miss something?

Compiling now.

[g3gg0]

did you run install.sh under magic-lantern/contrib/qemu ?

[irvcobb]

Spoke too soon. Failed with this:

Code Select Expand

hw/arm/. ./eos/dbi/logging.o: In function 'eos_logging_ init':
/home/irv/qemu/qemu-2.5.0/hw/arm/. ./eos/dbi/logging.c:1801: undefined reference to 'memory_set_access_logging_cb'
target-arm/op_helper.o: In function 'helper_log_ldr':
/home/irv/qemu/qemu-2.5.0/tar get-arm/op_helper.c:634: undefined reference to 'log-ldr-cb'
target-arm/op_helper.o: In function 'helper_log_str':
/home/irv/qemu/qemu-2.5.O/tar et-arm/op_helper.c:639: undefined reference to 'log_srt-cb'
collect2: error: ld returned 1 exit status
Makefile:193: recipe for target 'qemu-system-aarch64' failed
make[1] *** [qemu-system-aarch64] Error 1
Makefile:184: recipe for target 'subdir-aarch64-softmu' failed
make: *** [subdir-aarch64-softnunu] Error 2
make: Leaving directory ' /home/irv/qemu/qemu—2.5.O'


[irvcobb]

did you run install.sh under magic-lantern/contrib/qemu ?

Yes.

[g3gg0]

i tried also:
https://pastebin.com/YaCiuRFk
what does it look like when you run install and the commands it tells you to run?

[dfort]

this is the procedure described there?, right?

Right.

Seems to be a mercurial issue when not trusting files. Don´t think another location gonna fix it.
https://stackoverflow.com/questions/8535866/not-trusting-file-hg-hgrc-from-untrusted-user-root-group-dev

Actually, moving the repository from /mnt/c/Users/dfort to /home/dfort did get rid of those message. Though that hides it from regular Windows programs like SourceTree.

Got QEMU working!

• Start VcXsrv
• Open a PowerShell or cmd window
• Type bash
• Finally, start QEMU like this (all on one line):

Code Select Expand

DISPLAY=:0 ./run_canon_fw.sh 5D3,firmware="boot=1"

[dfort]

QEMU is working--great, but how are you guys mounting the card image so you can install ML?

I was going to suggest adding kpartx  to the package list because it is required for the mount.sh script, but it isn't working.

Code Select Expand

sudo kpartx -av sd.img
Kernel not configured for semaphores (System V IPC). Not using udev synchronisation code.
/dev/mapper/control: open failed: No such device
Failure to communicate with kernel device-mapper driver.
Check that device-mapper is available in the kernel.
Incompatible libdevmapper (unknown version) and kernel driver (unknown version).
device mapper prerequisites not met

[irvcobb]

i tried also:
https://pastebin.com/YaCiuRFk
what does it look like when you run install and the commands it tells you to run?

I got it sorted. PEBCAK.   

[a1ex]

- find out how did g3gg0 get that nice GUI for QEMU (mine has no menus)

Figured it out - installing libgtk2.0-dev *before* building QEMU does the trick. Updated the screenshot.

The DryOS shell:

QEMU is working--great, but how are you guys mounting the card image so you can install ML?

One scriptable way is using mtools - if the image uses FAT16/FAT32. I'm currently using it in the test suite (tests/run_tests.sh) and in this script: run_ml_all_cams.sh (which is not meant for interactive use, but for running some - usually short - test on more than one camera model, then saving a log and/or some screenshots). Without any mounting tools, we can use that script to copy ML on the card image:

Code Select Expand


# this is non-interactive (no display, but we'll ask it to save a screenshot, just for kicks)
# tip: use INCREMENTAL=1 to avoid running "make clean", and AUTOEXEC_ONLY=1 to avoid building/copying modules
ML_PLATFORMS="60D.111/" ML_CHANGESET="unified" TIMEOUT=10 SCREENSHOT=1 ./run_ml_all_cams.sh

# now we have ML on the card, so we can run it interactively
./run_canon_fw.sh 60D,firmware="boot=1"




Note: the "boot=0" or "boot=1" (which looks that way because I couldn't figure out how to add custom command-line options in QEMU, so I've hijacked an existing one) is used to change the boot flag in the ROM. On the very first ML install, the ROM dumps have the boot flag turned off. Subsequent ROM dumps (e.g. after formatting the card) will have the boot flag enabled. For cameras that do not run ML yet, the boot flag will be disabled. Therefore, for this example I'm assuming nothing about the boot flag state and always setting it manually.

Todo: a "make install_qemu" would be handy.

[bpv5P]

Great, now people can run open source code in a closed source system...
Isn't it simple to do a debian live usb? Faster compilation times, no bugs.
Microsoft is trying hard to gain market again, since the inception of android.

[g3gg0]

Great, now people can run open source code in a closed source system...
Isn't it simple to do a debian live usb? Faster compilation times, no bugs.
Microsoft is trying hard to gain market again, since the inception of android.

please dont shift this into a "closed source vs open source" flamewar kind of discussion.

1. for various reasons i am working with windows. my life is not "for ML only" and so i cannot simply switch to another OS.
2. booting a VM with linux, which i did for years now, is just extra overhead - rebooting into another OS is not an option.
3. i don't care whether my kernel is called NTKERNEL or Linux. i focus practical solutions, not idealism.
4. Open Source implies openness, which allows exactly this and that's what makes Open Source cool
5. this pipe dream of having a full blown GNU/* environment without quirks and compromieses no matter which OS i use, is finally real - why not accept it?
6. its the counterpart of WINE - its not noticeable slower.

and finally:
-> if its possible and doesn't cost too much effort, it will be done

[bpv5P]

please dont shift this into a "closed source vs open source" flamewar kind of discussion.

I'm not, but seems you are. I'm using meritocracy here, not idealism. Booting on the real system instead of 'emulation' has many benefits, beyond usability.

3. i don't care whether my kernel is called NTKERNEL or Linux. i focus practical solutions, not idealism.

You would be surprised how practical a unix system is, compared to Windows. Especially for development.

4. Open Source implies openness, which allows exactly this and that's what makes Open Source cool

I'm not stopping people from doing it and I'm not against it. Actually companies do this all the time with open source code.
What I'm saying is, again, that booting on real OS has many benefits.

5. this pipe dream of having a full blown GNU/* environment without quirks and compromieses no matter which OS i use, is finally real - why not accept it?

It's not the same. Can you tell what's going on in your system, without the code? You don't.
And, if you don't, how can you fix bugs or be sure it's secure? You g3gg0, for example, should be concerned about the security point, since you're designing the crypto module for ML. See all the accusations against Windows 10 about privacy. People that brings cryptography to general public are being monitored and this is not "tinfoil hat" bullshit, it's real.
If the crypto module ever go mainline, you should be concerned about these points, since it would potentially influence many people in need for this feature (like journalists in war zones).
But, you are the devs, not me. I'm just a random guy.

[irvcobb]

Successfully complied qemu and discovered "next steps" 1-3 which I was too dull to see yesterday.

In step 3, mount.sh throws an error because it needs kpartx. So install kpartx. Then this:

Code Select Expand

Kernel not configured for semaphores (System V IPC). Not using udev synchronisation code.
/dev/mapper/control: open failed: Permission denied
Failure to communicate with kernel device-mapper driver.
Incompatible libdevmapper (unknown version) and kernel driver (unknown version).
device mapper prerequisites not met

What am I missing?

[dfort]

What am I missing?

I asked the same question (about kpartx) and a1ex came up with a good suggestion.

[bouncyball]

5. this pipe dream of having a full blown GNU/* environment without quirks and compromieses no matter which OS i use, is finally real - why not accept it?
6. its the counterpart of WINE - its not noticeable slower.

Yeah all is right except maybe the fact that WSL (windows subsystem for Linux) can't run 32bit ELFs, only 64 bit ones.

[a1ex]

Tried gdb (required to run some models in QEMU, including 5D3). The one from Ubuntu repository doesn't work (it stops the emulation when it shouldn't). Tried https://launchpad.net/gcc-arm-embedded (gcc 5.4, Linux installation tarball), but that didn't work either, as it has 32-bit binaries.

As I'm currently on a slow connection, I'll let somebody else figure it out (best guess: compile the 64-bit version from sources, or use the Windows install package). The command you need to get running is:

Code Select Expand


./run_canon_fw.sh 5D3/firmware="boot=0" -s -S & arm-none-eabi-gdb -x 5D3/patches.gdb


or similar for any other camera model that has a patches.gdb file.

[irvcobb]

I asked the same question (about kpartx) and a1ex came up with a good suggestion.

Thanks for the help, I'm trying it now.
Obviously I'm having real trouble with reading comprehension this week. :-(

[dfort]

best guess: compile the 64-bit version from sources

This worked for my system.

Code Select Expand

sudo apt-get build-dep gdb-arm-none-eabi
sudo apt-get source --compile gdb-arm-none-eabi
sudo dpkg -i gdb-arm-none-eabi_7.11.1-0ubuntu1~16.5+9_amd64.deb

Note the name of the deb file because

Code Select Expand

sudo dpkg -i gdb-arm-none-eabi.deb
doesn't work--and after it is all over there's a mess to clean up but it works!

Slight change on the command:

Code Select Expand

DISPLAY=:0 ./run_canon_fw.sh 5D3,firmware="boot=0" -s -S & arm-none-eabi-gdb -x 5D3/patches.gdb

I keep getting:

Code Select Expand

Program received signal SIGTRAP, Trace/breakpoint trap.
0xff13659c in ?? ()
on the 5D3 but it does the same thing on the Mac.

However, it works great on the 700D!

Ok, my question. How do you copy to sd.img without mounting it?

This worked for me:

Code Select Expand

. ./mtools_setup.sh
mcopy -o -i $MSD ../magic-lantern/minimal/5D3.113/autoexec.bin ::




Yeah, I know, where's the Canon GUI? It works on the 700D:

[g3gg0]

yep, its 64 bits only, no 32 bit ABI adaption layer.
so probably good to upvote the uservoice regarding this issue. i already voted for it a year ago, but its not essential.

[a1ex]

I keep getting:

Program received signal SIGTRAP, Trace/breakpoint trap.
0xff13659c in ?? ()

on the 5D3 but it does the same thing on the Mac.

I don't have this issue on Ubuntu with gdb 7.10.1.20160616-cvs (gcc-arm-none-eabi-5_4-2016q2). The older gdb 7.6.0.20131129-cvs (gcc-arm-none-eabi-4_8-2013q4) works too (boots the GUI on 5D3), but some features from the GDB scripts (IIRC semaphores and message queues) require gdb newer than 7.7.

[dfort]

Interesting--I've got:

Mac (installed in: ~/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-gdb)

Code Select Expand

(gdb) show version
GNU gdb (GNU Tools for ARM Embedded Processors) 7.6.0.20131129-cvs


Ubuntu on Windows (installed in: /usr/bin/arm-none-eabi-gdb)

Code Select Expand

(gdb) show version
GNU gdb (7.11.1-0ubuntu1~16.5+9) 7.11.1


Neither boots the GUI on the 5D3 but they both do on the 700D.

[a1ex]

Found a 64-bit gdb in the PPA from https://launchpad.net/gcc-arm-embedded and updated the install script from qemu to use that one. Unfortunately, it doesn't boot the 5D3 GUI either.

Then noticed the gcc-arm-embedded builds were moved here: https://developer.arm.com/open-source/gnu-toolchain/gnu-rm

After double-checking on Ubuntu, all the 64-bit gdb versions I've tested (gdb-arm-none-eabi 7.10, gcc-arm-embedded 7.12 git) fail to boot the 5D3 GUI, while the 32-bit one (gdb-arm-none-eabi:i386 7.10) works just fine.

TODO: other than checking the 32-bit Windows binaries, not sure what other options we have...

[dfort]

Wonder if that's the same issue with the EOSM2. I can't get to the ML menus in QEMU and if I close the Canon menu I can't get them back. Does it work in Ubuntu?

Compiling ML works fine on Windows 10 using Linux subsystem or Cygwin on other versions of Windows so we have plenty of options but QEMU may be a good reason to setup an Ubuntu VM.

[a1ex]

The issues on M2 are new DryOS task hooks (for ML menu) and lack of LiveView emulation (for Canon menu only working once).