Compiling Magic Lantern on Windows 10 (using its Linux subsystem)

[g3gg0]

It is possible to compile magic lantern and qemu on windows, without any third-party-programs like cygwin, MSYS, VirtualBox etc by solely using windows' native linux compatibility layer.

Magic Lantern


For those who didn't know, microsoft added wrappers to allow linux code to execute properly.
You have just to enable it, as described on microsoft's website.
This gives you "bash" the famous native linux shell executable directly within windows.

OS Preparation

After you installed ubuntu, you should install a few standards tools.

Depending on the Windows 10 installation you have, you might be able to simply execute "bash" via Win+R or a menu entry called Bash or Ubuntu etc.
Then, in bash run: 

Code: [Select]

sudo apt-get update
sudo apt-get install make gcc gcc-arm-none-eabi mercurial gcc-mingw-w64 python3-docutils zip

There were also cases when you had to install python2 - your mileage may vary.

Code: [Select]

sudo apt-get install python2


Download

directly clone magic lantern from the mercurial repository using this command:

Code: [Select]

hg clone -u unified https://bitbucket.org/hudson/magic-lantern
it will download the latest version, the unified branch.


Configuration

first determine the exact arm-gcc compiler version you have either by executing

Code: [Select]

ls /usr/lib/gcc/arm-none-eabi/
or by entering

Code: [Select]

arm-none-eabi-gcc- [TAB] [TAB]


then use your favorite text editor in either linux or windows and create a file named Makefile.user with only this content:

Code: [Select]

GCC_VERSION=-4.8.2
ARM_PATH=/usr


open a windows shell at the folder where your makefiles are and run 'bash'.
and you should be able to compile Magic Lantern on windows with *native* compile speed



here an "all-in-one" script by a1ex, a bit modified:

Code: [Select]

# prepare system
sudo apt-get update
sudo apt-get install make gcc gcc-arm-none-eabi mercurial gcc-mingw-w64 python3-docutils zip

# download and prepare ML
hg clone -u unified https://bitbucket.org/hudson/magic-lantern
cd magic-lantern
echo "GCC_VERSION=-`ls /usr/lib/gcc/arm-none-eabi/`" > Makefile.user
echo "ARM_PATH=/usr" >> Makefile.user

# preparation complete, now build ML
cd platform/5D3.123
make zip

# desktop utilities
cd ../../modules/mlv_rec
make mlv_dump.exe
cd ../../modules/dual_iso
make cr2hdr.exe

# ports in progress (100D, 70D)
hg update 100D_merge_fw101 -C # use TAB to find the exact name
hg merge unified # or lua_fix or whatever (optional)
cd ../../platform/100D.101
make zip

# 4K with sound
hg update crop_rec_4k_mlv_snd -C
cd ../../platform/5D3.123
make clean; make zip

# quick build (autoexec.bin only, without modules)
cd ../../platform/5D3.123
make zip ML_MODULES_DYNAMIC=

# recovery (portable display test, ROM dumper, CPU info...)
hg update recovery -C
cd ../../platform/portable.000
make zip ML_MODULES_DYNAMIC=




QEMU (or: how to run Canon OS within qemu within the linux environment within windows 10 on a x64 CPU)

If you were successful with compiling magic lantern, then why not start compiling qemu?



install missing packages (review those please)

Code: [Select]

sudo apt-get update
sudo apt-get install zlib1g-dev libglib2.0 autoconf libtool libsdl-console flex bison libgtk2.0-dev mtools
sudo apt-get install libsdl-console-dev

the last one - libsdl-console-dev - caused some trouble. i could not download some (unnecessary) drm graphics drivers.
i used aptitude to inspect the status and don't ask me what i did, but aptitude asked me if i want to examine its recommendations and i accepted them.
suddenly libdrm was held back and all other packages got installed.

you probably have to switch to the qemu branch

Code: [Select]

hg update qemu

then it is time to compile qemu using the script in contrib/qemu/install.sh.
make sure your magic lantern path is named "magic-lantern" else the script will abort.

hint by a1ex, doesn't happen on my system:
for some reason, the output from install.sh is truncated
opening a new terminal appears to fix it (?!)
if it still doesn't work: ./install.sh |& tee install.log
then open install.log in a text editor to read it

when it's done, do what it says:
    a) cd `pwd`/some_path_here"
    b) ../configure_eos.sh"
    c) make -j4   (or the numer of cores your CPU has)

if you now run the run_canon_fw.sh you get an error telling you:

Code: [Select]

qemu-system-arm: -chardev socket,server,nowait,path=qemu.monitor,id=monsock: Failed to bind socket to qemu.monitor: Operation not permitted

my assumption is, that either unix domain socket implementation in WSL is buggy or at least incompatible to qemu.
so the script run_canon_fw.sh needs some patches before it runs - remove those lines:

Code: [Select]

    -chardev socket,server,nowait,path=qemu.monitor,id=monsock \
    -mon chardev=monsock,mode=readline \


enjoy!

[Danne]

Did you make it work? I´ve been doing some testing cause wanted to port "Switch" to windows this way. Just finished porting Switch to Linux.
Its a bit tricky and betaish on windows. Ok, here´s my tryout to compile mlv_dump.

I had to install make(additional)
sudo apt-get install make

The result was this:(sorry the shitty formatting. I´m on windows 10(virtual box)
root@DESKTOP-NOI3633:~/magic-lantern/modules/mlv_rec# make mlv_dump                                                     make: cc: Command not found                                                                                             make: stat: ../../Makefile.user: Input/output error                                                                     [ HOST_CC  ]   mlv_dump.host.o                                                                                          /bin/sh: 1: -I../../src: not found                                                                                      Makefile:58: recipe for target 'mlv_dump.host.o' failed                                                                 make: *** [mlv_dump.host.o] Error 127                                                                                   root@DESKTOP-NOI3633:~/magic-lantern/modules/mlv_rec#

One big question mark is where are the files even downloaded(located)? After some research you can find the stuff here:

Open up cmd and enter:

Code: [Select]

cd %localappdata%\Lxss\

Code: [Select]

start . (enter)
Go into root folder and find magic-lantern and change your Makefile.user.default to what g3gg0 describes above.
Rename Makefile.user.default to Makefile.user
Now go back to ubuntu bash terminal and compile. You´ll find the goodies doing step 1 again. Well no goodies yet since compiling didn´t work. Yet...


Here is what I get when trying to compile platform(5D mark III) . (upload ing pic since formatting doesn´t cut it.

[g3gg0]

okay thanks for your feedback. tried also installing package gcc ?

my makefile.user has only one additional difference that should have no impact:

Code: [Select]

HOST_CC?=$(shell which gcc)
HOST_LD?=$(shell which ld)
HOST_AR?=$(shell which ar)

can you run "make V=1" and check if there is something obvious?
e.g. wrong compiler being called etc

to run the compiler i used to open up cmd.exe, change to the ML path and execute "bash".
as i am on windows insider, my installation might be a bit different though

[Danne]

I really been messing with mac the last 10 years but I will do my best.

make V=1 What´s this, never heard of?

Not sure how you mean here:

to run the compiler i used to open up cmd.exe, change to the ML path and execute "bash".

Tried following, going from cmd.exe:

Code: [Select]

C:\Users\Danne\AppData\Local\lxss\root\magic-lantern\platform\5D3.113>bash -c make V=1
make: *** No targets specified and no makefile found.  Stop.
make V=1 in bash/ubuntu


Trying to install gcc gets me this:(wrong version?)

Code: [Select]

gcc is already the newest version (4:5.3.1-1ubuntu1).
Have some things todo but I´ll check in later today.

[g3gg0]

hi danne,

i cannot compare with my system, as i am at work right now.
but it calls "cc" only which is incorrect. it should call :
/usr/bin/arm-none-eabi-gcc
which get constructed from $(ARM_PATH) etc

can you restore the original Makefile.user.default so it is existent next to your Makefile.user?

also the  "| -v " is a bit weird. should call some executable there?

[Danne]

Keep getting errors. Am I calling the correct gcc? Here is my setup:

[g3gg0]

my guess is that your Makefile.user is causing the wrong cc to get called (?)
you have copied it properly over from Makefile.user.default?

[onesolo]

then modify/add those lines in your Makefile.user:

Code: [Select]

GCC_VERSION=-4.8.2
ARM_PATH=/usr
ARM_LIBGCC_PATH=/usr/lib/gcc/arm-$(ARM_ABI)/$(subst -,,$(GCC_VERSION))
CC=$(CROSS_COMPILE)gcc

open a windows shell at the folder where your makefiles are and run 'bash'.
(as i am on windows insider, my installation might be a bit different though)

and you should be able to compile Magic Lantern on windows with *native* compile speed

So, I did fine until I got the part to the makefile.user ....
Can anyone help me?
Where is that makefile.user file?! and after that, what should I do? I don't understand what are the steps that one has to do in order to compile.
Do I have to download the source code of ML ?! If yes, from where and to where?

[Danne]

Ok, it works. And fast! Let me break it down so not others do my(stupid) mistakes:

Install Ubuntu on windows. See first post
Install necessary compiler tools. See first post

1 - open up cmd.exe print bash and hit enter
2 - download magic lantern.
hg clone -u unified https://bitbucket.org/hudson/magic-lantern
3 - apply the changes in makefile suggested in g3gg0 first post
4 - print cd magic-lantern(and then go to whatever you want to compile)
5 - when compiling is done print cmd.exe hit enter then print start .(dot) hit enter again(will open up the current folder with the goodies)

Retrospect:
See to it that file extensions are visible in windows. I missed the .default part in makefile.user.default and when erasing it really didn´t erase the extension at all.
Another thing. My lost in folder space shenanigans in the second post are maybe good for getting acquinted with Ubuntu folder tree on windows otherwise I´d stay off those commands.
Phew, good exercise. Now I have good idea what to do when porting "Switch" over to windows.
Thanks g3gg0.

[g3gg0]

update: added qemu (holy shit is that cool....)
and thanks for your feedback, can you also review this one?

[Danne]

I think I got it working. Here is my loose notes on my qemu install.

libs installs fine

Code: [Select]

sudo apt-get install zlib1g-dev libglib2.0 autoconf libtool libsdl-console flex bison
Terminal ending with following last lines:

Code: [Select]

173 new root certificates were added to your trust store.
Import process completed.
Done 
done.
root@DESKTOP-NOI3633:/mnt/c/Users/Danne#
Now install following:

Code: [Select]

sudo apt-get install libsdl-console-devSeems to get stuck at:

Code: [Select]

Processing triggers for man-db (2.7.5-1) ...  When I try to copy the line the process seems to abort. Anyway. I might come back here if next steps are unsuccessful.

Next:
open cmd.exe

Code: [Select]

bash

Code: [Select]

cd magic-lantern/contrib/qemu

Code: [Select]

./install.sh
This pops up:

Code: [Select]

This will setup QEMU for emulating Magic Lantern. 
Thou shalt not be afraid of compiling stuff on Linux ;)
Continue? [y/n]
Hit y script runs and ends with:

Code: [Select]

Connecting to download.qemu.org (download.qemu.org)|172.99.69.163|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 25464996 (24M) [application/x-bzip2]

     0K ........ ........ ........         100% 1.79M=14s

2017-08-08 00:57:22 (1.79 MB/s) - ‘qemu-2.5.0.tar.bz2’ saved [25464996/25464996]
And script comes to this conclusion:

Code: [Select]

./install.sh: line 38: cd: .git: No such file or directory
Initialized empty Git repository in /mnt/c/Users/Danne/qemu/qemu-2.5.0/.git/
Hold on! Script wasn´t even done yet...Script continues and:

Code: [Select]

Enjoy!
This will setup QEMU for emulating Magic Lantern.
Thou shalt not be afraid of compiling stuff on Linux ;)
Continue? [y/n]
Crap! Mistakingly hit another button than y got this:
root@DESKTOP-NOI3633:/mnt/c/Users/Danne/magic-lantern/contrib/qemu#
Trying to hit y anyway
root@DESKTOP-NOI3633:/mnt/c/Users/Danne/magic-lantern/contrib/qemu# y
No luck:

Code: [Select]

y: command not found  Ok, maybe qemu already is installed?
Ran the installer again and:

Code: [Select]

Next steps:
===========

1) Compile QEMU

    cd /mnt/c/Users/Danne/qemu/qemu-2.5.0
    ../configure_eos.sh
    make -j1

Setting up QEMU on Linux...
Using gcc --std=gnu99 / g++ with -Wno-error=deprecated-declarations
Options>
Disabling libtool due to broken chain support

Script continues ans install, do make etc...

Now I remove these lines from the run_canon_fw.sh script

Code: [Select]

   -chardev socket,server,nowait,path=qemu.monitor,id=monsock \
    -mon chardev=monsock,mode=readline \
Let´s run the script

Code: [Select]

cd /mnt/c/Users/Danne/qemu

Code: [Select]

./run_canon_fw.sh
Some hick up on libutils. I hit a button then the compiling bible continues. I think for 20 minutes?

Finally this:

Code: [Select]

DebugMsg= (overriden) 
qemu-system-arm: -M: requires an argument
root@DESKTOP-NOI3633:/mnt/c/Users/Danne/qemu# j
That´s it?



*by the way should the 1) Compile QEMU instruction be changed from:

Code: [Select]

cd /mnt/c/Users/Danne/qemu/qemu-2.5.0 to:

Code: [Select]

cd /mnt/c/Users/Danne/qemu/qemuThe run_canon_fw.sh script is located in qemu folder, not qemu-2.5.0

[g3gg0]

i tried again yesterday - installation works as i described - without any glitch.
that .git message is normal

[Danne]

Ok, great. Thanks for qemu install

[dfort]

Nice!

Just tried it out and thought I'd document my mistakes so others won't have to suffer through them.

I couldn't install from the "store" even after creating a Microsoft account and signing up for the "Windows Insider" program:



The trick was to use the alternate installation method, turning on the Developer settings:



Then just type "bash" in a cmd window:



Note that the browser in the background is on the installation instructions in case you need to find that URL.

The packages didn't install on the first attempt but after doing a recommended "sudo apt-get update" it went smoothly. Well, almost. The instructions are to create a Makefile.user file that has:

Code: [Select]

GCC_VERSION=-4.8.2
I kept getting this error:

Code: [Select]

make: *** No rule to make target '/usr/lib/gcc/arm-none-eabi/4.8.2/libgcc.a', needed by 'gcc-libgcc.a'.  Stop.
Turns out I had a different version of gcc so I had to make this change:

Code: [Select]

GCC_VERSION=-4.9.3
I also made the mistake of cloning the magic-lantern repository in /mnt/c/User/dfort instead of the "real" home directory, /home/dfort. No sweat, just copied it to my home directory and that got rid of some weird messages about "untrusted" files.

One package you might consider adding is "zip" in order to "make zip" or even a cr2hdr-win.zip. Interesting that I couldn't build a Linux mlv_dump but running "make mlv_dump.exe" worked perfectly to cross compile a Windows binary.

Looking forward to trying QEMU next.

[g3gg0]

can you call "lsb_release -a" and post the result?
maybe when installing from store you get a newer release.
this would explain the newer compiler version.

the directory where the project is in, is not important.
you can leave it anywhere, then you can edit it with windows tools easily.

[a1ex]

I've set up this Win10 VM, which is a fresh installation with bash, visual studio and a few others preinstalled. The Linux subsystem is based on Ubuntu Xenial.

Package list is mostly OK; had to run "sudo apt-get update" before, and also had to install "zip".

To find out what GCC version you have, type:

Code: [Select]

arm-none-eabi-gcc- [TAB] [TAB]

Result: in my case it was 4.9.3.

For Makefile.user I prefer rewriting only the lines I need changed (rather than copying the entire Makefile.user.default). Therefore, I've created it from scratch (just these 2 lines):

Code: [Select]

GCC_VERSION=-4.9.3
ARM_PATH=/usr

In a nutshell:

Code: [Select]

# prepare system
sudo apt-get update
sudo apt-get install make gcc gcc-arm-none-eabi mercurial gcc-mingw-w64 python3-docutils zip
hg clone -u unified https://bitbucket.org/hudson/magic-lantern
cd magic-lantern
echo "GCC_VERSION=-4.9.3" > Makefile.user
echo "ARM_PATH=/usr" >> Makefile.user

# preparation complete, now build ML
cd platform/5D3.123
make zip

# desktop utilities
cd ../../modules/mlv_rec
make mlv_dump.exe
cd ../../modules/dual_iso
make cr2hdr.exe

Some more:

Code: [Select]

# ports in progress (100D, 70D)
hg update 100D_merge_fw101 -C # use TAB to find the exact name
hg merge unified # or lua_fix or whatever (optional)
cd ../../platform/100D.101
make zip

# 4K with sound
hg update crop_rec_4k_mlv_snd -C
cd ../../platform/5D3.123
make clean; make zip

# quick build (autoexec.bin only, without modules)
cd ../../platform/5D3.123
make zip ML_MODULES_DYNAMIC=

# recovery (portable display test, ROM dumper, CPU info...)
hg update recovery -C
cd ../../platform/portable.000
make zip ML_MODULES_DYNAMIC=

For QEMU, I had to install some additional packages and an X server (see e.g. Running Linux desktop apps on the Windows Subsystem for Linux):

Code: [Select]

# not sure if this step is needed
sudo nano /etc/apt/sources.list
# add deb-src entries (copy the deb ones and replace deb with deb-src)
sudo apt-get update
sudo apt-get build-dep qemu

# from g3gg0
sudo apt-get install zlib1g-dev libglib2.0 autoconf libtool libsdl-console flex bison
sudo apt-get install libsdl-console-dev

# needed for GUI
sudo apt-get install libgtk2.0-dev

# needed to access the virtual SD/CF images
sudo apt-get install mtools

# install QEMU
hg update qemu -C
cd contrib/qemu
./install.sh
# for some reason, the output from install.sh is truncated
# opening a new terminal appears to fix it (?!)
# if it still doesn't work: ./install.sh |& tee install.log
# then open install.log in a text editor to read it

# anyway, now follow the instructions from install.sh
cd /path/to/qemu/qemu-2.5.0
../configure_eos.sh
make -j4
cd ..

# copy the ROMs from your camera (e.g. sdcard/ML/LOGS/ROM*.BIN -> qemu/5D3/ROM*.BIN)

# now run the emulation
export DISPLAY=:0
./run_canon_fw.sh 60D,firmware="boot=0"



TODO:
- find out how did g3gg0 get that nice GUI for QEMU (mine has no menus) [DONE]
- install a GUI for Mercurial [g3gg0 suggests TortoiseHg]
- check build environment for common errors and suggest how to fix / what to install if things go wrong (also on other usual operating systems)
- mount the SD image used in QEMU and install ML there [halfway done]
- sample debugging session with GDB+QEMU, maybe also IDA
- run the QEMU test suite (a large part of it won't work, as it's hardcoded for my ROMs)
- how to run a Lua script
- bonus: show how to build CHDK
- ... ?

[g3gg0]

The Linux subsystem is based on Ubuntu Xenial.

okay, thats the difference. (could not test a fresh Win10 as i only have 2MBit/s )
mine is based on trusty and apt-get upgrade doesn't do anything.
then i will update my post later with your information.

[bouncyball]

@g3gg0

mine is based on trusty and apt-get upgrade doesn't do anything.

Do:

Code: [Select]

sudo do-release-upgrade
and you're gonna get upgraded system to Xenial

bb

[dfort]

can you call "lsb_release -a" and post the result?

Code: [Select]

dfort@DELL-LAPTOP:~/qemu$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.2 LTS
Release:        16.04
Codename:       xenial

the directory where the project is in, is not important.
you can leave it anywhere, then you can edit it with windows tools easily.

When magic-lantern is in the Windows home directory I'm getting these messages:

Code: [Select]

not trusting file /mnt/c/Users/dfort/magic-lantern/.hg/hgrc from untrusted user root, group root
It doesn't seem to cause any problems. Of course you're right if you want to use Windows tools put it somewhere Windows can access it. I searched for the location of /home/dfort using Windows Explorer and couldn't find it.

Just started trying to get QEMU working. Right off the bat I was getting this:

Code: [Select]

Could not initialize SDL(No available video device) - exiting
After reading a1ex's post I thought I'd try it without "build-dep qemu" just to see if that is really necessary. libgtk2.0-dev installed fine but then I got:

Code: [Select]

gtk initialization failed
And "build-dep qemu" needs the source files:

Code: [Select]

dfort@DELL-LAPTOP:~/qemu$ sudo apt-get build-dep qemu
[sudo] password for dfort:
Reading package lists... Done
E: You must put some 'source' URIs in your sources.list
The article a1ex pointed out recommends using Cygwin X or vcXsrv but it looks like a1ex found a more elegant solution. BTW--nice to see nano being used. It is much easier to learn than vi.



Of course to build from source requires a compiler and related tools so now I've got a full-blown Linux development environment in Windows.

Still:

Code: [Select]

export DISPLAY=:0
./run_canon_fw.sh 5D3,firmware="boot=1"
...
gtk initialization failed


[Danne]

export DISPLAY=:0
./run_canon_fw.sh 5D3,firmware="boot=1"
...
gtk initialization failed

Try installing XMING and start it.
https://sourceforge.net/projects/xming/
Also install x11-apps

Code: [Select]

sudo apt-get install x11-appsWrite:
export DISPLAY=:0 enter in terminal
Then run your commands.

I searched for the location of /home/dfort

Open up cmd.exe write start .(dot) enter?

I just ported mlv_dump processing through Switch(Linux version in Windows  ) Some more things to to do before releasing...



 

[dfort]

Open up cmd.exe write start .(dot) enter?

Nope--not there.

Tried your suggestions installing Xming but still no QEMU x11 window. Also tried VcXsrv--no go. Got a lot of stuff installed now that probably isn't necessary.

Code: [Select]

sudo apt-get install xorg openbox x11-apps x11-xserver-utils libsdl-console-dev kitchen-sink
Ok--maybe not the kitchen sink yet.

[Danne]

Kitchen sink .

Seems to be a mercurial issue when not trusting files. Don´t think another location gonna fix it.
https://stackoverflow.com/questions/8535866/not-trusting-file-hg-hgrc-from-untrusted-user-root-group-dev

[g3gg0]

Code: [Select]

sudo do-release-upgrade


doh. right. i meant 'dist-upgrade', not 'upgrade' but this doesnt help either.
thanks!

[g3gg0]

- find out how did g3gg0 get that nice GUI for QEMU (mine has no menus)

oh eh. i used VNC only, no GUI at all...
this might explain the localhost:5900

- install a GUI for Mercurial

i can absolutely recommend TortoiseHg and its workbench

[g3gg0]

I couldn't install from the "store" even after creating a Microsoft account and signing up for the "Windows Insider" program:
The trick was to use the alternate installation method, turning on the Developer settings:

this is the procedure described there?, right?