Magic Lantern Forum

Developing Magic Lantern => General Development => Topic started by: g3gg0 on August 06, 2017, 10:32:09 PM

Title: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: g3gg0 on August 06, 2017, 10:32:09 PM

It is possible to compile magic lantern and qemu on windows, without any third-party-programs like cygwin, MSYS, VirtualBox etc by solely using windows' native linux compatibility layer.

Magic Lantern
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fthumb.ibb.co%2FijyvUv%2Fimage.png&hash=96a4300cfa8c6241bc83de5c5e169f39) (https://ibb.co/ijyvUv)

For those who didn't know, microsoft added wrappers to allow linux code to execute properly.
You have just to enable it, as described on microsoft's website. (https://msdn.microsoft.com/en-us/commandline/wsl/install_guide)
This gives you "bash" the famous native linux shell executable directly within windows.

OS Preparation

After you installed ubuntu, you should install a few standards tools.

Depending on the Windows 10 installation you have, you might be able to simply execute "bash" via Win+R or a menu entry called Bash or Ubuntu etc.
Then, in bash run: 

Code: [Select]

sudo apt-get update
sudo apt-get install make gcc gcc-arm-none-eabi mercurial gcc-mingw-w64 python3-docutils zip



Download

directly clone magic lantern from the mercurial repository using this command:

Code: [Select]

hg clone -u unified https://bitbucket.org/hudson/magic-lantern
it will download the latest version, the unified branch.


Configuration

first determine the exact arm-gcc compiler version you have either by executing

Code: [Select]

ls /usr/lib/gcc/arm-none-eabi/
or by entering

Code: [Select]

arm-none-eabi-gcc- [TAB] [TAB]


then use your favorite text editor in either linux or windows and create a file named Makefile.user with only this content:

Code: [Select]

GCC_VERSION=-4.8.2
ARM_PATH=/usr


open a windows shell at the folder where your makefiles are and run 'bash'.
and you should be able to compile Magic Lantern on windows with *native* compile speed :)



here an "all-in-one" script by a1ex, a bit modified:

Code: [Select]

# prepare system
sudo apt-get update
sudo apt-get install make gcc gcc-arm-none-eabi mercurial gcc-mingw-w64 python3-docutils zip

# download and prepare ML
hg clone -u unified https://bitbucket.org/hudson/magic-lantern
cd magic-lantern
echo "GCC_VERSION=-`ls /usr/lib/gcc/arm-none-eabi/`" > Makefile.user
echo "ARM_PATH=/usr" >> Makefile.user

# preparation complete, now build ML
cd platform/5D3.123
make zip

# desktop utilities
cd ../../modules/mlv_rec
make mlv_dump.exe
cd ../../modules/dual_iso
make cr2hdr.exe

# ports in progress (100D, 70D)
hg update 100D_merge_fw101 -C # use TAB to find the exact name
hg merge unified # or lua_fix or whatever (optional)
cd ../../platform/100D.101
make zip

# 4K with sound
hg update crop_rec_4k_mlv_snd -C
cd ../../platform/5D3.123
make clean; make zip

# quick build (autoexec.bin only, without modules)
cd ../../platform/5D3.123
make zip ML_MODULES_DYNAMIC=

# recovery (portable display test, ROM dumper, CPU info...)
hg update recovery -C
cd ../../platform/portable.000
make zip ML_MODULES_DYNAMIC=




QEMU (or: how to run Canon OS within qemu within the linux environment within windows 10 on a x64 CPU)

If you were successful with compiling magic lantern, then why not start compiling qemu?

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fthumb.ibb.co%2FeRFB4v%2Fimage.png&hash=93422176682b739c8bbd659a178cfe4c) (https://ibb.co/eRFB4v)

install missing packages (review those please)

Code: [Select]

sudo apt-get update
sudo apt-get install zlib1g-dev libglib2.0 autoconf libtool libsdl-console flex bison libgtk2.0-dev mtools
sudo apt-get install libsdl-console-dev

the last one - libsdl-console-dev - caused some trouble. i could not download some (unnecessary) drm graphics drivers.
i used aptitude to inspect the status and don't ask me what i did, but aptitude asked me if i want to examine its recommendations and i accepted them.
suddenly libdrm was held back and all other packages got installed.

you probably have to switch to the qemu branch

Code: [Select]

hg update qemu

then it is time to compile qemu using the script in contrib/qemu/install.sh.
make sure your magic lantern path is named "magic-lantern" else the script will abort.

hint by a1ex, doesn't happen on my system:
for some reason, the output from install.sh is truncated
opening a new terminal appears to fix it (?!)
if it still doesn't work: ./install.sh |& tee install.log
then open install.log in a text editor to read it

when it's done, do what it says:
    a) cd `pwd`/some_path_here"
    b) ../configure_eos.sh"
    c) make -j4   (or the numer of cores your CPU has)

if you now run the run_canon_fw.sh you get an error telling you:

Code: [Select]

qemu-system-arm: -chardev socket,server,nowait,path=qemu.monitor,id=monsock: Failed to bind socket to qemu.monitor: Operation not permitted

my assumption is, that either unix domain socket implementation in WSL is buggy (https://github.com/Microsoft/BashOnWindows/issues/134) or at least incompatible to qemu.
so the script run_canon_fw.sh needs some patches before it runs - remove those lines:

Code: [Select]

    -chardev socket,server,nowait,path=qemu.monitor,id=monsock \
    -mon chardev=monsock,mode=readline \


enjoy!

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: Danne on August 06, 2017, 11:50:15 PM

Did you make it work? I´ve been doing some testing cause wanted to port "Switch" to windows this way. Just finished porting Switch to Linux.
Its a bit tricky and betaish on windows. Ok, here´s my tryout to compile mlv_dump.

I had to install make(additional)
sudo apt-get install make

The result was this:(sorry the shitty formatting. I´m on windows 10(virtual box)
root@DESKTOP-NOI3633:~/magic-lantern/modules/mlv_rec# make mlv_dump                                                     make: cc: Command not found                                                                                             make: stat: ../../Makefile.user: Input/output error                                                                     [ HOST_CC  ]   mlv_dump.host.o                                                                                          /bin/sh: 1: -I../../src: not found                                                                                      Makefile:58: recipe for target 'mlv_dump.host.o' failed                                                                 make: *** [mlv_dump.host.o] Error 127                                                                                   root@DESKTOP-NOI3633:~/magic-lantern/modules/mlv_rec#

One big question mark is where are the files even downloaded(located)? After some research you can find the stuff here:

Open up cmd and enter:

Code: [Select]

cd %localappdata%\Lxss\

Code: [Select]

start . (enter)
Go into root folder and find magic-lantern and change your Makefile.user.default to what g3gg0 describes above.
Rename Makefile.user.default to Makefile.user
Now go back to ubuntu bash terminal and compile. You´ll find the goodies doing step 1 again. Well no goodies yet since compiling didn´t work. Yet...


Here is what I get when trying to compile platform(5D mark III) . (upload ing pic since formatting doesn´t cut it.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs1.postimg.org%2F61gomoiqn%2FScreen_Shot_2017-08-07_at_00.15.44.png&hash=6c291d26e12c5f8bbdeb42bca77529c9)

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: g3gg0 on August 07, 2017, 07:45:22 AM

okay thanks for your feedback. tried also installing package gcc ?

my makefile.user has only one additional difference that should have no impact:

Code: [Select]

HOST_CC?=$(shell which gcc)
HOST_LD?=$(shell which ld)
HOST_AR?=$(shell which ar)

can you run "make V=1" and check if there is something obvious?
e.g. wrong compiler being called etc

to run the compiler i used to open up cmd.exe, change to the ML path and execute "bash".
as i am on windows insider, my installation might be a bit different though

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: Danne on August 07, 2017, 08:45:42 AM

I really been messing with mac the last 10 years but I will do my best.

make V=1 What´s this, never heard of?

Not sure how you mean here:

Quote

to run the compiler i used to open up cmd.exe, change to the ML path and execute "bash".

Tried following, going from cmd.exe:

Code: [Select]

C:\Users\Danne\AppData\Local\lxss\root\magic-lantern\platform\5D3.113>bash -c make V=1
make: *** No targets specified and no makefile found.  Stop.
make V=1 in bash/ubuntu
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs1.postimg.org%2Fb30ntegof%2FScreen_Shot_2017-08-07_at_08.48.08.png&hash=4876dd685cdb86451d289ea7b3478b57)

Trying to install gcc gets me this:(wrong version?)

Code: [Select]

gcc is already the newest version (4:5.3.1-1ubuntu1).
Have some things todo but I´ll check in later today.

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: g3gg0 on August 07, 2017, 09:03:53 AM

hi danne,

i cannot compare with my system, as i am at work right now.
but it calls "cc" only which is incorrect. it should call :
/usr/bin/arm-none-eabi-gcc
which get constructed from $(ARM_PATH) etc

can you restore the original Makefile.user.default so it is existent next to your Makefile.user?

also the  "| -v " is a bit weird. should call some executable there?

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: Danne on August 07, 2017, 02:42:41 PM

Keep getting errors. Am I calling the correct gcc? Here is my setup:
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fs1.postimg.org%2Flv7exdupb%2FScreen_Shot_2017-08-07_at_14.47.39.png&hash=6d4eb71b80bfa1d45940c189358aaa6b)

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: g3gg0 on August 07, 2017, 03:37:52 PM

my guess is that your Makefile.user is causing the wrong cc to get called (?)
you have copied it properly over from Makefile.user.default?

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: onesolo on August 07, 2017, 03:48:10 PM

Quote from: g3gg0 on August 06, 2017, 10:32:09 PM

then modify/add those lines in your Makefile.user:

Code: [Select]

GCC_VERSION=-4.8.2
ARM_PATH=/usr
ARM_LIBGCC_PATH=/usr/lib/gcc/arm-$(ARM_ABI)/$(subst -,,$(GCC_VERSION))
CC=$(CROSS_COMPILE)gcc

open a windows shell at the folder where your makefiles are and run 'bash'.
(as i am on windows insider, my installation might be a bit different though)

and you should be able to compile Magic Lantern on windows with *native* compile speed :)

So, I did fine until I got the part to the makefile.user ....
Can anyone help me?
Where is that makefile.user file?! and after that, what should I do? I don't understand what are the steps that one has to do in order to compile.
Do I have to download the source code of ML ?! If yes, from where and to where?

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: Danne on August 07, 2017, 04:30:39 PM

Ok, it works. And fast! Let me break it down so not others do my(stupid) mistakes:

Install Ubuntu on windows. See first post
Install necessary compiler tools. See first post

1 - open up cmd.exe print bash and hit enter
2 - download magic lantern.
hg clone -u unified https://bitbucket.org/hudson/magic-lantern
3 - apply the changes in makefile suggested in g3gg0 first post
4 - print cd magic-lantern(and then go to whatever you want to compile)
5 - when compiling is done print cmd.exe hit enter then print start .(dot) hit enter again(will open up the current folder with the goodies)

Retrospect:
See to it that file extensions are visible in windows. I missed the .default part in makefile.user.default and when erasing it really didn´t erase the extension at all.
Another thing. My lost in folder space shenanigans in the second post are maybe good for getting acquinted with Ubuntu folder tree on windows otherwise I´d stay off those commands.
Phew, good exercise. Now I have good idea what to do when porting "Switch" over to windows.
Thanks g3gg0.

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: g3gg0 on August 08, 2017, 01:30:13 AM

update: added qemu (holy shit is that cool....)
and thanks for your feedback, can you also review this one?

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: Danne on August 08, 2017, 07:20:24 AM

I think I got it working. Here is my loose notes on my qemu install.

libs installs fine

Code: [Select]

sudo apt-get install zlib1g-dev libglib2.0 autoconf libtool libsdl-console flex bison
Terminal ending with following last lines:

Code: [Select]

173 new root certificates were added to your trust store.
Import process completed.
Done 
done.
root@DESKTOP-NOI3633:/mnt/c/Users/Danne#
Now install following:

Code: [Select]

sudo apt-get install libsdl-console-devSeems to get stuck at:

Code: [Select]

Processing triggers for man-db (2.7.5-1) ...  When I try to copy the line the process seems to abort. Anyway. I might come back here if next steps are unsuccessful.

Next:
open cmd.exe

Code: [Select]

bash

Code: [Select]

cd magic-lantern/contrib/qemu

Code: [Select]

./install.sh
This pops up:

Code: [Select]

This will setup QEMU for emulating Magic Lantern. 
Thou shalt not be afraid of compiling stuff on Linux ;)
Continue? [y/n]
Hit y script runs and ends with:

Code: [Select]

Connecting to download.qemu.org (download.qemu.org)|172.99.69.163|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 25464996 (24M) [application/x-bzip2]

     0K ........ ........ ........         100% 1.79M=14s

2017-08-08 00:57:22 (1.79 MB/s) - ‘qemu-2.5.0.tar.bz2’ saved [25464996/25464996]
And script comes to this conclusion:

Code: [Select]

./install.sh: line 38: cd: .git: No such file or directory
Initialized empty Git repository in /mnt/c/Users/Danne/qemu/qemu-2.5.0/.git/
Hold on! Script wasn´t even done yet...Script continues and:

Code: [Select]

Enjoy!
This will setup QEMU for emulating Magic Lantern.
Thou shalt not be afraid of compiling stuff on Linux ;)
Continue? [y/n]
Crap! Mistakingly hit another button than y got this:
root@DESKTOP-NOI3633:/mnt/c/Users/Danne/magic-lantern/contrib/qemu#
Trying to hit y anyway
root@DESKTOP-NOI3633:/mnt/c/Users/Danne/magic-lantern/contrib/qemu# y
No luck:

Code: [Select]

y: command not found  Ok, maybe qemu already is installed?
Ran the installer again and:

Code: [Select]

Next steps:
===========

1) Compile QEMU

    cd /mnt/c/Users/Danne/qemu/qemu-2.5.0
    ../configure_eos.sh
    make -j1

Setting up QEMU on Linux...
Using gcc --std=gnu99 / g++ with -Wno-error=deprecated-declarations
Options>
Disabling libtool due to broken chain support

Script continues ans install, do make etc...

Now I remove these lines from the run_canon_fw.sh script

Code: [Select]

   -chardev socket,server,nowait,path=qemu.monitor,id=monsock \
    -mon chardev=monsock,mode=readline \
Let´s run the script

Code: [Select]

cd /mnt/c/Users/Danne/qemu

Code: [Select]

./run_canon_fw.sh
Some hick up on libutils. I hit a button then the compiling bible continues. I think for 20 minutes?

Finally this:

Code: [Select]

DebugMsg= (overriden) 
qemu-system-arm: -M: requires an argument
root@DESKTOP-NOI3633:/mnt/c/Users/Danne/qemu# j
That´s it?



*by the way should the 1) Compile QEMU instruction be changed from:

Code: [Select]

cd /mnt/c/Users/Danne/qemu/qemu-2.5.0 to:

Code: [Select]

cd /mnt/c/Users/Danne/qemu/qemuThe run_canon_fw.sh script is located in qemu folder, not qemu-2.5.0

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: g3gg0 on August 08, 2017, 08:46:08 AM

i tried again yesterday - installation works as i described - without any glitch.
that .git message is normal

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: Danne on August 08, 2017, 11:19:03 AM

Ok, great. Thanks for qemu install :)

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: dfort on August 09, 2017, 07:52:52 AM

Nice!

Just tried it out and thought I'd document my mistakes so others won't have to suffer through them.

I couldn't install from the "store" even after creating a Microsoft account and signing up for the "Windows Insider" program:

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm5.staticflickr.com%2F4366%2F36059705260_c3c3bcebe5.jpg&hash=1439f0118e30c259921d101592b992b3)

The trick was to use the alternate installation method, turning on the Developer settings:

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm5.staticflickr.com%2F4409%2F36059745160_ee1bac04f2.jpg&hash=8edbaae707683881bb1fb70e42e467a2) (https://flic.kr/p/WWtBW9)

Then just type "bash" in a cmd window:

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm5.staticflickr.com%2F4405%2F35649025143_a70dbbd74e_c.jpg&hash=ed4942b7a88e1114b82626e2e2ce0c97) (https://flic.kr/p/Wjbzbx)

Note that the browser in the background is on the installation instructions in case you need to find that URL.

The packages didn't install on the first attempt but after doing a recommended "sudo apt-get update" it went smoothly. Well, almost. The instructions are to create a Makefile.user file that has:

Code: [Select]

GCC_VERSION=-4.8.2
I kept getting this error:

Code: [Select]

make: *** No rule to make target '/usr/lib/gcc/arm-none-eabi/4.8.2/libgcc.a', needed by 'gcc-libgcc.a'.  Stop.
Turns out I had a different version of gcc so I had to make this change:

Code: [Select]

GCC_VERSION=-4.9.3
I also made the mistake of cloning the magic-lantern repository in /mnt/c/User/dfort instead of the "real" home directory, /home/dfort. No sweat, just copied it to my home directory and that got rid of some weird messages about "untrusted" files.

One package you might consider adding is "zip" in order to "make zip" or even a cr2hdr-win.zip. Interesting that I couldn't build a Linux mlv_dump but running "make mlv_dump.exe" worked perfectly to cross compile a Windows binary.

Looking forward to trying QEMU next.

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: g3gg0 on August 09, 2017, 09:52:34 AM

can you call "lsb_release -a" and post the result?
maybe when installing from store you get a newer release.
this would explain the newer compiler version.

the directory where the project is in, is not important.
you can leave it anywhere, then you can edit it with windows tools easily.

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: a1ex on August 09, 2017, 10:11:13 AM

I've set up this Win10 VM (https://developer.microsoft.com/en-us/windows/downloads/virtual-machines), which is a fresh installation with bash, visual studio and a few others preinstalled. The Linux subsystem is based on Ubuntu Xenial.

Package list is mostly OK; had to run "sudo apt-get update" before, and also had to install "zip".

To find out what GCC version you have, type:

Code: [Select]

arm-none-eabi-gcc- [TAB] [TAB]

Result: in my case it was 4.9.3.

For Makefile.user I prefer rewriting only the lines I need changed (rather than copying the entire Makefile.user.default). Therefore, I've created it from scratch (just these 2 lines):

Code: [Select]

GCC_VERSION=-4.9.3
ARM_PATH=/usr

In a nutshell:

Code: [Select]

# prepare system
sudo apt-get update
sudo apt-get install make gcc gcc-arm-none-eabi mercurial gcc-mingw-w64 python3-docutils zip
hg clone -u unified https://bitbucket.org/hudson/magic-lantern
cd magic-lantern
echo "GCC_VERSION=-4.9.3" > Makefile.user
echo "ARM_PATH=/usr" >> Makefile.user

# preparation complete, now build ML
cd platform/5D3.123
make zip

# desktop utilities
cd ../../modules/mlv_rec
make mlv_dump.exe
cd ../../modules/dual_iso
make cr2hdr.exe

Some more:

Code: [Select]

# ports in progress (100D, 70D)
hg update 100D_merge_fw101 -C # use TAB to find the exact name
hg merge unified # or lua_fix or whatever (optional)
cd ../../platform/100D.101
make zip

# 4K with sound
hg update crop_rec_4k_mlv_snd -C
cd ../../platform/5D3.123
make clean; make zip

# quick build (autoexec.bin only, without modules)
cd ../../platform/5D3.123
make zip ML_MODULES_DYNAMIC=

# recovery (portable display test, ROM dumper, CPU info...)
hg update recovery -C
cd ../../platform/portable.000
make zip ML_MODULES_DYNAMIC=

For QEMU, I had to install some additional packages and an X server (see e.g. Running Linux desktop apps on the Windows Subsystem for Linux (https://ctrl.blog/entry/how-to-x-on-wsl)):

Code: [Select]

# not sure if this step is needed
sudo nano /etc/apt/sources.list
# add deb-src entries (copy the deb ones and replace deb with deb-src)
sudo apt-get update
sudo apt-get build-dep qemu

# from g3gg0
sudo apt-get install zlib1g-dev libglib2.0 autoconf libtool libsdl-console flex bison
sudo apt-get install libsdl-console-dev

# needed for GUI
sudo apt-get install libgtk2.0-dev

# needed to access the virtual SD/CF images
sudo apt-get install mtools

# install QEMU
hg update qemu -C
cd contrib/qemu
./install.sh
# for some reason, the output from install.sh is truncated
# opening a new terminal appears to fix it (?!)
# if it still doesn't work: ./install.sh |& tee install.log
# then open install.log in a text editor to read it

# anyway, now follow the instructions from install.sh
cd /path/to/qemu/qemu-2.5.0
../configure_eos.sh
make -j4
cd ..

# copy the ROMs from your camera (e.g. sdcard/ML/LOGS/ROM*.BIN -> qemu/5D3/ROM*.BIN)

# now run the emulation
export DISPLAY=:0
./run_canon_fw.sh 60D,firmware="boot=0"

(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fwin10%2Fqemu-60D.png&hash=938de7f682be7f31c2754772c0010713)

TODO:
- find out how did g3gg0 get that nice GUI for QEMU (mine has no menus) [DONE]
- install a GUI for Mercurial [g3gg0 suggests TortoiseHg]
- check build environment for common errors and suggest how to fix / what to install if things go wrong (also on other usual operating systems)
- mount the SD image used in QEMU and install ML there [halfway done]
- sample debugging session with GDB+QEMU, maybe also IDA
- run the QEMU test suite (a large part of it won't work, as it's hardcoded for my ROMs)
- how to run a Lua script
- bonus: show how to build CHDK
- ... ?

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: g3gg0 on August 09, 2017, 01:56:58 PM

Quote from: a1ex on August 09, 2017, 10:11:13 AM

The Linux subsystem is based on Ubuntu Xenial.

okay, thats the difference. (could not test a fresh Win10 as i only have 2MBit/s ;) )
mine is based on trusty and apt-get upgrade doesn't do anything.
then i will update my post later with your information.

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: bouncyball on August 09, 2017, 06:52:40 PM

@g3gg0

Quote from: g3gg0 on August 09, 2017, 01:56:58 PM

mine is based on trusty and apt-get upgrade doesn't do anything.

Do:

Code: [Select]

sudo do-release-upgrade
and you're gonna get upgraded system to Xenial :)

bb

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: dfort on August 09, 2017, 07:02:52 PM

Quote from: g3gg0 on August 09, 2017, 09:52:34 AM

can you call "lsb_release -a" and post the result?

Code: [Select]

dfort@DELL-LAPTOP:~/qemu$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.2 LTS
Release:        16.04
Codename:       xenial

Quote from: g3gg0 on August 09, 2017, 09:52:34 AM

the directory where the project is in, is not important.
you can leave it anywhere, then you can edit it with windows tools easily.

When magic-lantern is in the Windows home directory I'm getting these messages:

Code: [Select]

not trusting file /mnt/c/Users/dfort/magic-lantern/.hg/hgrc from untrusted user root, group root
It doesn't seem to cause any problems. Of course you're right if you want to use Windows tools put it somewhere Windows can access it. I searched for the location of /home/dfort using Windows Explorer and couldn't find it.

Just started trying to get QEMU working. Right off the bat I was getting this:

Code: [Select]

Could not initialize SDL(No available video device) - exiting
After reading a1ex's post I thought I'd try it without "build-dep qemu" just to see if that is really necessary. libgtk2.0-dev installed fine but then I got:

Code: [Select]

gtk initialization failed
And "build-dep qemu" needs the source files:

Code: [Select]

dfort@DELL-LAPTOP:~/qemu$ sudo apt-get build-dep qemu
[sudo] password for dfort:
Reading package lists... Done
E: You must put some 'source' URIs in your sources.list
The article a1ex pointed out (https://ctrl.blog/entry/how-to-x-on-wsl) recommends using Cygwin X (http://x.cygwin.com/) or vcXsrv (https://sourceforge.net/projects/vcxsrv/) but it looks like a1ex found a more elegant solution. BTW--nice to see nano being used. It is much easier to learn than vi.

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm5.staticflickr.com%2F4355%2F36419123856_b46a40724f_z.jpg&hash=de23bf85c033d0b938ebc55a3f2e294b) (https://flic.kr/p/XuewGy)

Of course to build from source requires a compiler and related tools so now I've got a full-blown Linux development environment in Windows.

Still:

Code: [Select]

export DISPLAY=:0
./run_canon_fw.sh 5D3,firmware="boot=1"
...
gtk initialization failed


Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: Danne on August 09, 2017, 07:14:45 PM

Quote

export DISPLAY=:0
./run_canon_fw.sh 5D3,firmware="boot=1"
...
gtk initialization failed

Try installing XMING and start it.
https://sourceforge.net/projects/xming/
Also install x11-apps

Code: [Select]

sudo apt-get install x11-appsWrite:
export DISPLAY=:0 enter in terminal
Then run your commands.

Quote

I searched for the location of /home/dfort

Open up cmd.exe write start .(dot) enter?

I just ported mlv_dump processing through Switch(Linux version in Windows  :P) Some more things to to do before releasing...



 

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: dfort on August 09, 2017, 08:10:39 PM

Quote from: Danne on August 09, 2017, 07:14:45 PM

Open up cmd.exe write start .(dot) enter?

Nope--not there.

Tried your suggestions installing Xming but still no QEMU x11 window. Also tried VcXsrv--no go. Got a lot of stuff installed now that probably isn't necessary.

Code: [Select]

sudo apt-get install xorg openbox x11-apps x11-xserver-utils libsdl-console-dev kitchen-sink
Ok--maybe not the kitchen sink yet.

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: Danne on August 09, 2017, 08:33:56 PM

Kitchen sink ;).

Seems to be a mercurial issue when not trusting files. Don´t think another location gonna fix it.
https://stackoverflow.com/questions/8535866/not-trusting-file-hg-hgrc-from-untrusted-user-root-group-dev

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: g3gg0 on August 09, 2017, 10:50:34 PM

Quote from: bouncyball on August 09, 2017, 06:52:40 PM

Code: [Select]

sudo do-release-upgrade


doh. right. i meant 'dist-upgrade', not 'upgrade' but this doesnt help either.
thanks!

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: g3gg0 on August 09, 2017, 11:13:04 PM

Quote from: a1ex on August 09, 2017, 10:11:13 AM

- find out how did g3gg0 get that nice GUI for QEMU (mine has no menus)

oh eh. i used VNC only, no GUI at all...
this might explain the localhost:5900 ;)

Quote from: a1ex on August 09, 2017, 10:11:13 AM

- install a GUI for Mercurial

i can absolutely recommend TortoiseHg (https://tortoisehg.bitbucket.io/) and its workbench (https://tortoisehg.bitbucket.io/img/vt_history.png)

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: g3gg0 on August 09, 2017, 11:22:18 PM

Quote from: dfort on August 09, 2017, 07:52:52 AM

I couldn't install from the "store" even after creating a Microsoft account and signing up for the "Windows Insider" program:
The trick was to use the alternate installation method, turning on the Developer settings:

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm5.staticflickr.com%2F4409%2F36059745160_ee1bac04f2.jpg&hash=8edbaae707683881bb1fb70e42e467a2) (https://flic.kr/p/WWtBW9)

this is the procedure described there? (https://msdn.microsoft.com/en-us/commandline/wsl/install_guide), right?

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: g3gg0 on August 09, 2017, 11:24:36 PM

if someone wants to use an XServer on windows, i used to run vcXsrv (https://sourceforge.net/projects/vcxsrv/) and it works quite well.

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: irvcobb on August 10, 2017, 12:28:39 AM

As one with some Linux experience, no fear of the command line, and little compiling experience, my observations may be of value?

Starting with a clean Linux install on Win 10, I can get all the way to here with no problem:

Code: [Select]

qemu$ ./run_canon-fw.sh 600D
make: Entering directory '/home/irv/qemu/qemu-2.5.0'
Please call configure before running make!
Makefile:47: recipe for target 'config-host.mak' failed
make: *** [config-host.mak] Error 1
make: Leaving directory '/home/irv/qemu/qemu-2.5.0'

Then a cd qemu-2.5.0 and ./configure seem to sort things.

Did I miss something?

Compiling now.

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: g3gg0 on August 10, 2017, 12:37:43 AM

did you run install.sh under magic-lantern/contrib/qemu ?

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: irvcobb on August 10, 2017, 12:41:16 AM

Spoke too soon. Failed with this:

Code: [Select]

hw/arm/. ./eos/dbi/logging.o: In function 'eos_logging_ init':
/home/irv/qemu/qemu-2.5.0/hw/arm/. ./eos/dbi/logging.c:1801: undefined reference to 'memory_set_access_logging_cb'
target-arm/op_helper.o: In function 'helper_log_ldr':
/home/irv/qemu/qemu-2.5.0/tar get-arm/op_helper.c:634: undefined reference to 'log-ldr-cb'
target-arm/op_helper.o: In function 'helper_log_str':
/home/irv/qemu/qemu-2.5.O/tar et-arm/op_helper.c:639: undefined reference to 'log_srt-cb'
collect2: error: ld returned 1 exit status
Makefile:193: recipe for target 'qemu-system-aarch64' failed
make[1] *** [qemu-system-aarch64] Error 1
Makefile:184: recipe for target 'subdir-aarch64-softmu' failed
make: *** [subdir-aarch64-softnunu] Error 2
make: Leaving directory ' /home/irv/qemu/qemu—2.5.O'


Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: irvcobb on August 10, 2017, 12:44:12 AM

Quote from: g3gg0 on August 10, 2017, 12:37:43 AM

did you run install.sh under magic-lantern/contrib/qemu ?

Yes.

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: g3gg0 on August 10, 2017, 01:00:32 AM

i tried also:
https://pastebin.com/YaCiuRFk
what does it look like when you run install and the commands it tells you to run?

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: dfort on August 10, 2017, 02:14:51 AM

Quote from: g3gg0 on August 09, 2017, 11:22:18 PM

this is the procedure described there? (https://msdn.microsoft.com/en-us/commandline/wsl/install_guide), right?

Right.

Quote from: Danne on August 09, 2017, 08:33:56 PM

Seems to be a mercurial issue when not trusting files. Don´t think another location gonna fix it.
https://stackoverflow.com/questions/8535866/not-trusting-file-hg-hgrc-from-untrusted-user-root-group-dev

Actually, moving the repository from /mnt/c/Users/dfort to /home/dfort did get rid of those message. Though that hides it from regular Windows programs like SourceTree.

Got QEMU working!

• Start VcXsrv
• Open a PowerShell or cmd window
• Type bash
• Finally, start QEMU like this (all on one line):

Code: [Select]

DISPLAY=:0 ./run_canon_fw.sh 5D3,firmware="boot=1"
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm5.staticflickr.com%2F4417%2F36334084311_f8f92668d2_z.jpg&hash=123af26ea23a057657795a9602464845) (https://flic.kr/p/XmHFpv)

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: dfort on August 10, 2017, 03:29:46 AM

QEMU is working--great, but how are you guys mounting the card image so you can install ML?

I was going to suggest adding kpartx  to the package list because it is required for the mount.sh script, but it isn't working.

Code: [Select]

sudo kpartx -av sd.img
Kernel not configured for semaphores (System V IPC). Not using udev synchronisation code.
/dev/mapper/control: open failed: No such device
Failure to communicate with kernel device-mapper driver.
Check that device-mapper is available in the kernel.
Incompatible libdevmapper (unknown version) and kernel driver (unknown version).
device mapper prerequisites not met

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: irvcobb on August 10, 2017, 04:12:54 AM

Quote from: g3gg0 on August 10, 2017, 01:00:32 AM

i tried also:
https://pastebin.com/YaCiuRFk
what does it look like when you run install and the commands it tells you to run?

I got it sorted. PEBCAK.   :)

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: a1ex on August 10, 2017, 09:12:17 AM

Quote from: a1ex on August 09, 2017, 10:11:13 AM

- find out how did g3gg0 get that nice GUI for QEMU (mine has no menus)

Figured it out - installing libgtk2.0-dev *before* building QEMU does the trick. Updated the screenshot.

The DryOS shell:
(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fwin10%2Fqemu-60D-drysh.png&hash=c2c283a7edac8f2ecb0dcc08a5136c80) (https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fwin10%2Fqemu-60D-drysh2.png&hash=d45d5c28df5a02f91ec99fdb336b0c7f)

Quote from: dfort on August 10, 2017, 03:29:46 AM

QEMU is working--great, but how are you guys mounting the card image so you can install ML?

One scriptable way is using mtools - if the image uses FAT16/FAT32. I'm currently using it in the test suite (tests/run_tests.sh) and in this script: run_ml_all_cams.sh (which is not meant for interactive use, but for running some - usually short - test on more than one camera model, then saving a log and/or some screenshots). Without any mounting tools, we can use that script to copy ML on the card image:

Code: [Select]

# this is non-interactive (no display, but we'll ask it to save a screenshot, just for kicks)
# tip: use INCREMENTAL=1 to avoid running "make clean", and AUTOEXEC_ONLY=1 to avoid building/copying modules
ML_PLATFORMS="60D.111/" ML_CHANGESET="unified" TIMEOUT=10 SCREENSHOT=1 ./run_ml_all_cams.sh

# now we have ML on the card, so we can run it interactively
./run_canon_fw.sh 60D,firmware="boot=1"

(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fwin10%2Fqemu-60D-ml.png&hash=50eeb2ce679d8c153397d20db9d7a798)

Note: the "boot=0" or "boot=1" (which looks that way because I couldn't figure out how to add custom command-line options in QEMU, so I've hijacked an existing one) is used to change the boot flag in the ROM. On the very first ML install, the ROM dumps have the boot flag turned off. Subsequent ROM dumps (e.g. after formatting the card) will have the boot flag enabled. For cameras that do not run ML yet, the boot flag will be disabled. Therefore, for this example I'm assuming nothing about the boot flag state and always setting it manually.

Todo: a "make install_qemu" would be handy.

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: bpv5P on August 10, 2017, 10:36:14 AM

Great, now people can run open source code in a closed source system...
Isn't it simple to do a debian live usb? Faster compilation times, no bugs.
Microsoft is trying hard to gain market again, since the inception of android.

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: g3gg0 on August 10, 2017, 11:16:14 AM

Quote from: bpv5P on August 10, 2017, 10:36:14 AM

Great, now people can run open source code in a closed source system...
Isn't it simple to do a debian live usb? Faster compilation times, no bugs.
Microsoft is trying hard to gain market again, since the inception of android.

please dont shift this into a "closed source vs open source" flamewar kind of discussion.

1. for various reasons i am working with windows. my life is not "for ML only" and so i cannot simply switch to another OS.
2. booting a VM with linux, which i did for years now, is just extra overhead - rebooting into another OS is not an option.
3. i don't care whether my kernel is called NTKERNEL or Linux. i focus practical solutions, not idealism.
4. Open Source implies openness, which allows exactly this and that's what makes Open Source cool
5. this pipe dream of having a full blown GNU/* environment without quirks and compromieses no matter which OS i use, is finally real - why not accept it?
6. its the counterpart of WINE - its not noticeable slower.

and finally:
-> if its possible and doesn't cost too much effort, it will be done

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: bpv5P on August 10, 2017, 12:48:00 PM

Quote from: g3gg0 on August 10, 2017, 11:16:14 AM

please dont shift this into a "closed source vs open source" flamewar kind of discussion.

I'm not, but seems you are. I'm using meritocracy here, not idealism. Booting on the real system instead of 'emulation' has many benefits, beyond usability.

Quote

3. i don't care whether my kernel is called NTKERNEL or Linux. i focus practical solutions, not idealism.

You would be surprised how practical a unix system is, compared to Windows. Especially for development.

Quote

4. Open Source implies openness, which allows exactly this and that's what makes Open Source cool

I'm not stopping people from doing it and I'm not against it. Actually companies do this all the time with open source code.
What I'm saying is, again, that booting on real OS has many benefits.

Quote

5. this pipe dream of having a full blown GNU/* environment without quirks and compromieses no matter which OS i use, is finally real - why not accept it?

It's not the same. Can you tell what's going on in your system, without the code? You don't.
And, if you don't, how can you fix bugs or be sure it's secure? You g3gg0, for example, should be concerned about the security point, since you're designing the crypto module for ML. See all the accusations against Windows 10 about privacy. People that brings cryptography to general public are being monitored and this is not "tinfoil hat" bullshit, it's real.
If the crypto module ever go mainline, you should be concerned about these points, since it would potentially influence many people in need for this feature (like journalists in war zones).
But, you are the devs, not me. I'm just a random guy.

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: irvcobb on August 10, 2017, 08:31:00 PM

Successfully complied qemu and discovered "next steps" 1-3 which I was too dull to see yesterday.

In step 3, mount.sh throws an error because it needs kpartx. So install kpartx. Then this:

Code: [Select]

Kernel not configured for semaphores (System V IPC). Not using udev synchronisation code.
/dev/mapper/control: open failed: Permission denied
Failure to communicate with kernel device-mapper driver.
Incompatible libdevmapper (unknown version) and kernel driver (unknown version).
device mapper prerequisites not met
What am I missing?

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: dfort on August 11, 2017, 05:46:16 AM

Quote from: irvcobb on August 10, 2017, 08:31:00 PM

What am I missing?

I asked the same question (about kpartx) and a1ex came up with a good suggestion (http://www.magiclantern.fm/forum/index.php?topic=20214.msg188309#msg188309).

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: bouncyball on August 11, 2017, 08:52:51 AM

Quote from: g3gg0 on August 10, 2017, 11:16:14 AM

5. this pipe dream of having a full blown GNU/* environment without quirks and compromieses no matter which OS i use, is finally real - why not accept it?
6. its the counterpart of WINE - its not noticeable slower.

Yeah all is right except maybe the fact that WSL (windows subsystem for Linux) can't run 32bit ELFs, only 64 bit ones.

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: a1ex on August 11, 2017, 09:28:44 AM

Tried gdb (required to run some models in QEMU, including 5D3). The one from Ubuntu repository doesn't work (it stops the emulation when it shouldn't). Tried https://launchpad.net/gcc-arm-embedded (gcc 5.4, Linux installation tarball), but that didn't work either, as it has 32-bit binaries.

As I'm currently on a slow connection, I'll let somebody else figure it out (best guess: compile the 64-bit version from sources, or use the Windows install package). The command you need to get running is:

Code: [Select]

./run_canon_fw.sh 5D3/firmware="boot=0" -s -S & arm-none-eabi-gdb -x 5D3/patches.gdb

or similar for any other camera model that has a patches.gdb file.

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: irvcobb on August 11, 2017, 03:15:12 PM

Quote from: dfort on August 11, 2017, 05:46:16 AM

I asked the same question (about kpartx) and a1ex came up with a good suggestion (http://www.magiclantern.fm/forum/index.php?topic=20214.msg188309#msg188309).

Thanks for the help, I'm trying it now.
Obviously I'm having real trouble with reading comprehension this week. :-(

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: dfort on August 12, 2017, 08:32:58 AM

Quote from: a1ex on August 11, 2017, 09:28:44 AM

best guess: compile the 64-bit version from sources

This worked for my system.

Code: [Select]

sudo apt-get build-dep gdb-arm-none-eabi
sudo apt-get source --compile gdb-arm-none-eabi
sudo dpkg -i gdb-arm-none-eabi_7.11.1-0ubuntu1~16.5+9_amd64.deb
Note the name of the deb file because

Code: [Select]

sudo dpkg -i gdb-arm-none-eabi.debdoesn't work--and after it is all over there's a mess to clean up but it works!

Slight change on the command:

Code: [Select]

DISPLAY=:0 ./run_canon_fw.sh 5D3,firmware="boot=0" -s -S & arm-none-eabi-gdb -x 5D3/patches.gdb
I keep getting:

Code: [Select]

Program received signal SIGTRAP, Trace/breakpoint trap.
0xff13659c in ?? ()on the 5D3 but it does the same thing on the Mac.

However, it works great on the 700D!

Ok, my question. How do you copy to sd.img without mounting it?

This worked for me:

Code: [Select]

. ./mtools_setup.sh
mcopy -o -i $MSD ../magic-lantern/minimal/5D3.113/autoexec.bin ::

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm5.staticflickr.com%2F4426%2F36377984861_9aa0412a4f_z.jpg&hash=1c1c6fe681e3f22e5661206bf711ee75) (https://flic.kr/p/XqAFvx)

Yeah, I know, where's the Canon GUI? It works on the 700D:

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Ffarm5.staticflickr.com%2F4416%2F36515516065_aed9e8290e_n.jpg&hash=d0d938457beab3d932e315a7e5637c88) (https://flic.kr/p/XCKyKc)

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: g3gg0 on August 12, 2017, 10:10:46 AM

yep, its 64 bits only, no 32 bit ABI adaption layer.
so probably good to upvote the uservoice (https://wpdev.uservoice.com/forums/266908-command-prompt-console-bash-on-ubuntu-on-windo/suggestions/13377507-please-add-32-bit-elf-support-to-the-kernel?page=1&per_page=20) regarding this issue. i already voted for it a year ago, but its not essential.

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: a1ex on August 12, 2017, 11:16:41 AM

Quote

I keep getting:

Program received signal SIGTRAP, Trace/breakpoint trap.
0xff13659c in ?? ()

on the 5D3 but it does the same thing on the Mac.

I don't have this issue on Ubuntu with gdb 7.10.1.20160616-cvs (gcc-arm-none-eabi-5_4-2016q2). The older gdb 7.6.0.20131129-cvs (gcc-arm-none-eabi-4_8-2013q4) works too (boots the GUI on 5D3), but some features from the GDB scripts (IIRC semaphores and message queues) require gdb newer than 7.7.

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: dfort on August 12, 2017, 05:36:27 PM

Interesting--I've got:

Mac (installed in: ~/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-gdb)

Code: [Select]

(gdb) show version
GNU gdb (GNU Tools for ARM Embedded Processors) 7.6.0.20131129-cvs

Ubuntu on Windows (installed in: /usr/bin/arm-none-eabi-gdb)

Code: [Select]

(gdb) show version
GNU gdb (7.11.1-0ubuntu1~16.5+9) 7.11.1

Neither boots the GUI on the 5D3 but they both do on the 700D.

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: a1ex on September 27, 2017, 12:05:53 AM

Found a 64-bit gdb in the PPA from https://launchpad.net/gcc-arm-embedded and updated the install script from qemu to use that one. Unfortunately, it doesn't boot the 5D3 GUI either.

Then noticed the gcc-arm-embedded builds were moved here: https://developer.arm.com/open-source/gnu-toolchain/gnu-rm

After double-checking on Ubuntu, all the 64-bit gdb versions I've tested (gdb-arm-none-eabi 7.10, gcc-arm-embedded 7.12 git) fail to boot the 5D3 GUI, while the 32-bit one (gdb-arm-none-eabi:i386 7.10) works just fine.

TODO: other than checking the 32-bit Windows binaries, not sure what other options we have...

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: dfort on September 28, 2017, 02:15:17 AM

Wonder if that's the same issue with the EOSM2. I can't get to the ML menus in QEMU and if I close the Canon menu I can't get them back. Does it work in Ubuntu?

Compiling ML works fine on Windows 10 using Linux subsystem or Cygwin on other versions of Windows so we have plenty of options but QEMU may be a good reason to setup an Ubuntu VM.

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: a1ex on September 28, 2017, 01:26:20 PM

The issues on M2 are new DryOS task hooks (http://www.magiclantern.fm/forum/index.php?topic=15895.msg189856#msg189856) (for ML menu) and lack of LiveView emulation (http://www.magiclantern.fm/forum/index.php?topic=15895.msg187913#msg187913) (for Canon menu only working once).

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: jml2017 on December 24, 2017, 11:38:12 AM

Hello ML/Linux Gurus,

I am trying to install QEMU and the procedure described in the first post works for me until the ./install.sh step.

At the very beginning of the script execution it exits with the following messages


(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fthumb.ibb.co%2FeMr8aR%2Fimage.png&hash=b6867707deb4abe4daedcb6f14530b10) (https://ibb.co/eMr8aR)


I added the command to the profile and checked that the path is now correct with "echo $PATH" command:
The path variable contains
/home/myusername/gcc-arm-none-eabi-5_4-2016q3/bin

I also checked that this folder exists and is populated with files.
However, the script still exits at the same point... I restarted the bash with no success.

It may be obvious for practitionners, but my experience of linux is quite limited.  :(

I am running Windows 10 with Ubuntu installed as per first post.

Thanks for your help.

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: a1ex on December 24, 2017, 11:43:16 AM

Incidentally I was currently testing QEMU installation on WSL and noticed the same issue - try "hg pull; hg update qemu -C" to get the fix.

No guarantees the next steps are actually working - will try them later after a short break.

edit: seems to work; still want to re-test on a fresh installation.

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: jml2017 on January 03, 2018, 10:26:09 PM

Ok, with this command, the script goes further. But now it complains that I dot have the right GDB

(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fthumb.ibb.co%2FdrDjCG%2Fimage.png&hash=edd8880f6f5225bbe102731883890606) (http://ibb.co/drDjCG)


I guess the best choice is option 5. However I am not sure how to install this from https://launchpad.net/gcc-arm-embedded. Which command line should I use?

Thanks for the help.
 

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: a1ex on January 03, 2018, 11:02:20 PM

For trying out current stuff, I hope option 4 should work fine (that's also because most of the GDB patches that were causing trouble are no longer needed (http://www.magiclantern.fm/forum/index.php?topic=2864.msg190823#msg190823)).

For 5, just unzip a Linux 64-bit build in your home directory and add it to your PATH.

I've ran some parts of the test suite with gcc-arm-none-eabi-6-2017-q2-update, gcc-arm-none-eabi-6-2017-q2-update and gdb 8.0.50.20171008-git (all 64-bit) and worked fine. However, if I get the 5D3/patches.gdb back from changeset 93e57ce57802 (https://bitbucket.org/hudson/magic-lantern/commits/93e57ce57802) and intentionally break the RTC emulation - so the patch will execute something - all these 3 64-bit versions will fail. The 32-bit gdb from gcc-arm-none-eabi-5_4-2016q3 (and any other 32-bit versions I've tried) will work just fine.

Recently I've managed to crash the 32-bit gdb on 5D classic code, and the 64-bit gdb 8.0.50.20171008-git worked to some extent (so, the 32-bit GDB isn't bug-free either).

My conclusion so far: for serious GDB use on DIGIC 4 and 5 firmwares, you are more likely to get in trouble with a 64-bit GDB (but you should still be able to run the examples from the QEMU guide (https://bitbucket.org/hudson/magic-lantern/src/qemu/contrib/qemu/README.rst?at=qemu&fileviewer=file-view-default)).

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: jml2017 on January 04, 2018, 09:29:32 PM

I selected 4, and now it complains again about the PATH even though the PATH is correct...


(https://www.magiclantern.fm/forum/proxy.php?request=http%3A%2F%2Fthumb.ibb.co%2FbNNzEw%2Fimage.png&hash=5bc2c1592c94d54da4249128e8d4dc43) (http://ibb.co/bNNzEw)

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: a1ex on January 04, 2018, 09:43:44 PM

Hm, that's a bug; it should have installed arm-none-eabi-gcc/gdb from the package manager, so they should be already in PATH and recognized as valid...

Do you get any output if you type arm-none-eabi-gdb at the bash prompt?

(will look into it, but need to reinstall the VM first)

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: jml2017 on January 05, 2018, 04:50:37 PM

That is what I get:

-bash: /home/jml2017/gcc-arm-none-eabi-5_4-2016q3/bin/arm-none-eabi-gdb: cannot execute binary file: Exec format error

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: a1ex on January 05, 2018, 04:55:36 PM

Hm, it (or maybe you) somehow downloaded the 32-bit toolchain (which WSL can't run), and you have added it to PATH. If you ran the command suggested by the script, close the bash window and open a new one, as that change is temporary.

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: a1ex on January 07, 2018, 04:05:32 PM

Made some WSL changes to QEMU install script, but for some reason I'm unable to bring the Win10 VM back to life (and I'm not currently on a fast internet connection either). Would appreciate some help with testing latest changes (both from scratch and on top on some existing installation).

Please include an installation log, if possible, or some screenshots from the process (for example, the prompts you may get during installation). Thanks.

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: jml2017 on January 12, 2018, 11:12:30 PM

Hello,
Reinstalled Ubuntu from scratch and ran the script install.sh for QEMU with no issues until I get...

---------------------------------------------------------------------------------------------------------------
Initialized empty Git repository in /home/username/qemu/qemu-2.5.0/.git/

*** Please tell me who you are.

Run

  git config --global user.email "you@example.com"
  git config --global user.name "Your Name"

to set your account's default identity.
Omit --global to set the identity only in this repository.

fatal: empty ident name (for <qemu@magiclantern.fm>) not allowed

-----------------------------------------------------------------------------------------------------

What should I do?

Thanks.

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: a1ex on January 12, 2018, 11:24:47 PM

Okay, that must be a newer git than what I've tested; will check.

If this patch works for you, I'll commit it:

Code: [Select]

--- a/contrib/qemu/install.sh
+++ b/contrib/qemu/install.sh
@@ -389,6 +389,7 @@
   git init
   # git requires a valid email; if not setup, add one for this directory only
   git config user.email || git config user.email qemu@magiclantern.fm
+  git config user.name || git config user.name qemu-eos
   git add . && git commit -q -m "$QEMU_NAME vanilla"
 fi
 cd ..


Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: jml2017 on January 13, 2018, 10:39:28 AM

Okay QEMU compiles with no extra errors after adding your extra line to the script.

I am now at the step of running ./run canon_fw.sh 6D. However it errors because it cannot open /6D/SFDATA.BIN. If I understand well, I should be able to retrieve this file by running some module sf_dump on my 6D. However, I do not see this module on the ML camera menus. Where is it?

Thanks

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: a1ex on January 13, 2018, 12:05:09 PM

It's covered under Installation (https://bitbucket.org/hudson/magic-lantern/src/qemu/contrib/qemu/README.rst#rst-header-installation) (no binary available yet, but I should provide one).

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: jml2017 on January 13, 2018, 07:14:42 PM

I followed steps from https://www.magiclantern.fm/forum/index.php?topic=19232.0 (https://www.magiclantern.fm/forum/index.php?topic=19232.0)

I compiled the module with
 ~/magic-lantern/modules/sf_dump$ make

Then I had to mount the SD card  so that Ubuntu subsystem of windows 10 sees it:

sudo mkdir /mnt/SD
sudo mount -t drvfs F: /mnt/SD (F: is the SD drive under windows 10)

Then  ran :
make install CF_CARD=/mnt/SD

Finally, when starting my 6D with the SD card, I could load the sf_dump module and run it to get the SFDATA.BIN.

So I ran again ./run_canon_fw.sh 6D and got the following output:

Code: [Select]

/sbin/losetup
/sbin/losetup
make: Entering directory '/home/username/qemu/qemu-2.5.0'
CHK version_gen.h
LEX convert-dtsv0-lexer.lex.c
make[1]: flex: Command not found
BISON dtc-parser.tab.c
make[1]: bison: Command not found
LEX dtc-lexer.lex.c
make[1]: flex: Command not found
make: Leaving directory '/home/username/qemu/qemu-2.5.0'
./run_canon_fw.sh 6D &

DebugMsg=0x67c8 (from GDB script)
Lockdown read 0
Lockdown read 0
Lockdown read 1
Lockdown read 1
Lockdown read 2
Lockdown read 2
Lockdown read 3
Lockdown read 3
Lockdown read 4
Lockdown read 4
00000000 - 00000FFF: eos.tcm_code
40000000 - 40000FFF: eos.tcm_data
00001000 - 1FFFFFFF: eos.ram
40001000 - 5FFFFFFF: eos.ram_uncached
F0000000 - F0FFFFFF: eos.rom0
F1000000 - F1FFFFFF: eos.rom0_mirror
F2000000 - F2FFFFFF: eos.rom0_mirror
F3000000 - F3FFFFFF: eos.rom0_mirror
F4000000 - F4FFFFFF: eos.rom0_mirror
F5000000 - F5FFFFFF: eos.rom0_mirror
F6000000 - F6FFFFFF: eos.rom0_mirror
F7000000 - F7FFFFFF: eos.rom0_mirror
F8000000 - F8FFFFFF: eos.rom1
F9000000 - F9FFFFFF: eos.rom1_mirror
FA000000 - FAFFFFFF: eos.rom1_mirror
FB000000 - FBFFFFFF: eos.rom1_mirror
FC000000 - FCFFFFFF: eos.rom1_mirror
FD000000 - FDFFFFFF: eos.rom1_mirror
FE000000 - FEFFFFFF: eos.rom1_mirror
FF000000 - FFFFFFFF: eos.rom1_mirror
C0000000 - DFFFFFFF: eos.iomem
[EOS] loading './6D/ROM0.BIN' to 0xF0000000-0xF0FFFFFF
[EOS] loading './6D/ROM1.BIN' to 0xF8000000-0xF8FFFFFF
[EOS] loading './6D/SFDATA.BIN' as serial flash, size=0x800000
[MPU] warning: non-empty spell #2 (Complete WaitID = 0x80000001 Mode group) has duplicate(s): #6
[MPU] warning: non-empty spell #52 (PROP_VIDEO_MODE) has duplicate(s): #53

[MPU] Available keys:
- Arrow keys   : Navigation
- PgUp, PgDn   : Sub dial (rear scrollwheel)
- [ and ]      : Main dial (top scrollwheel)
- SPACE        : SET
- DELETE       : guess (press only)
- M            : MENU (press only)
- P            : PLAY (press only)
- I            : INFO/DISP (press only)
- Q            : guess (press only)
- L            : LiveView (press only)
- Shift        : Half-shutter
- 0/9          : Mode dial (press only)
- B            : Open battery door
- C            : Open card door
- F10          : Power down switch
- F1           : show this help

gtk initialization failed
[MPU] WARNING: forced shutdown.

For clean shutdown, please use 'Machine -> Power Down'
(or 'system_powerdown' in QEMU monitor.)
What did I do wrong?

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: a1ex on January 13, 2018, 09:05:21 PM

You need to install an X server (see earlier in this thread) and export the DISPLAY variable. This step is specific to WSL.

Make sure you can run other GUI programs (such as gitk).

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: jml2017 on January 13, 2018, 11:23:30 PM

Ok I installed VcXsrv.

Now I get this error:

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fthumb.ibb.co%2Ff83Aj6%2Fimage.png&hash=b0d4ed2dd8584711a9f3d9197182bdda) (https://ibb.co/f83Aj6)

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: a1ex on January 13, 2018, 11:31:58 PM

Press M and start navigating the menu. For some reason, the 6D starts with main display off - was your camera configured that way before dumping the ROM? Or press I(nfo) to turn it on.

The red messages are printed by Canon firmware to the serial console (https://bitbucket.org/hudson/magic-lantern/src/qemu/contrib/qemu/HACKING.rst#rst-header-serial-console).

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: jml2017 on January 14, 2018, 12:50:45 AM

Ok it works now  :)


(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fthumb.ibb.co%2FmxQO46%2Fimage.png&hash=c996c0446b4454fb5bdb8c490b178785) (https://ibb.co/mxQO46)

I'd like to generate the zebras and histogram I get with ML on the real camera.

I tried this command with some image
env QEMU_EOS_VRAM_PH_QR_RAW='/path/to/IMG_1234.CR2' ./run_canon_fw.sh 6D,firmware="boot=1", but I do not get anything.

What is the procedure to emulate the capture from an image and the image review that we get right after?

Thanks again.

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: clanlee on April 09, 2018, 09:50:37 AM

I need some assistance.  I followed the instructions and got to the point where I can call up the QEMU window but I can't do anything from there.  It looks like Magic Lantern is not loaded so I have nothing but the QEMU window.  I tried pressing M for menu and all other combinations of buttons but nothing happens.

How do I load Magic Lantern within the Ubuntu environment?  Do I have to copy the contents of the nightly build somewhere?  Thank you.

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: dfort on April 09, 2018, 01:27:58 PM

Quote from: clanlee on April 09, 2018, 09:50:37 AM

Do I have to copy the contents of the nightly build somewhere?

Yes, it needs to be in the sd.img or cf.img depending on which (virtual) camera you are using. I had some problems on the EOSM and it turned out there was a problem with the SFDATA.BIN file I was using.

Did you follow these instructions?

https://bitbucket.org/hudson/magic-lantern/src/qemu/contrib/qemu/README.rst

If you have a problem with the user guide or suggestions, a1ex is looking for feedback -- post it on the  How to run Magic Lantern into QEMU?!...  (https://www.magiclantern.fm/forum/index.php?topic=2864.0) topic.

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: clanlee on April 09, 2018, 08:49:53 PM

Quote from: dfort on April 09, 2018, 01:27:58 PM

Yes, it needs to be in the sd.img or cf.img depending on which (virtual) camera you are using. I had some problems on the EOSM and it turned out there was a problem with the SFDATA.BIN file I was using.

Did you follow these instructions?

https://bitbucket.org/hudson/magic-lantern/src/qemu/contrib/qemu/README.rst

If you have a problem with the user guide or suggestions, a1ex is looking for feedback -- post it on the  How to run Magic Lantern into QEMU?!...  (https://www.magiclantern.fm/forum/index.php?topic=2864.0) topic.

Yes, I was following those instructions but as a linux newbie, I ran into a road block when trying to mount the sd.img file.  I'm trying to setup the environment for the 5D3.123.  I wasn't sure where to post the question since this is specific to the Windows 10 running Ubuntu so I thought this might be a good place.

This is where I ran into the problem.  According to the instructions:

Quote

mount the card image (sd.img or cf.img) as /whatever/EOS_DIGITAL, then run make install from your platform directory:

# from the magic-lantern directory
cd platform/60D.111
make clean; make
# make sure your virtual card is mounted (this step is operating system specific)
make install
# make sure your virtual card is no longer mounted

I ran make clean; make command in the platform/5D3.123 folder.

When I attempted to mount the sd.img file, it's failing due to my lack of knowledge in linux commands.  I tried the following command:

Quote

sudo mount -t ~/qemu-eos/sd.img ~/magic-lantern/platform/5D3.123/EOS_DIGITAL -o loop

But received a message that it can't find the path in /etc/fstab.  When I listed the contents of fstab, it's nothing like the suggestions I found when searching for answer.  It's probably due to the fact that it's a virtual environment in Windows. Any help would be appreciated.

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: a1ex on April 21, 2018, 07:59:51 PM

Reproduced; looks like mounting the images in WSL is different; however, it's not really needed. You should be able to use "make install_qemu".

Sorry for the delay when it comes to WSL issues; the Windows VM is painfully slow :(

It took about 1 hour to run the install script, then 15 minutes just to execute the above command (compared to 1 min + 25 seconds on the local machine). It wasn't a fresh install, just hg pull, hg update, ./install.sh and make install_qemu (so all the dependencies were already there). Hoping a physical Win10 installation is faster than that...

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: ewinemiller on July 17, 2020, 11:58:55 PM

Hey folks,

I'm giving this a whirl and running into python problems when it tries to compile the modules. The error I get is

/bin/sh: 1: python: not found

I found a reference in one thread where adding

PYTHON=python2.7

to the Makefile.user helped with this problem. For me not so much. Also tried python2. I know it's using the value because if I change it to python3, it yells at me about a syntax error in the script.

  File "../readme2modulestrings.py", line 40
    print "static char __module_string_%s_name [] MODULE_STRINGS_SECTION = %s;" % (a, c_repr(name))
          ^
SyntaxError: invalid syntax

If I type python2 or python2.7 in bash, I do get the 2.7 python shell so it does seem to be installed correctly.

Any suggestions? I did pick Ubuntu 20.04LTS for my Linux version and I'm trying to compile Danne's branch (though I had the same problem with unified).

Thanks.

Title: Re: Compiling Magic Lantern on Windows 10 (using its Linux subsystem)
Post by: ewinemiller on July 18, 2020, 01:38:08 PM

Switched to Unbuntu 18.04LTS and it works fine.

Quote from: ewinemiller on July 17, 2020, 11:58:55 PM

I did pick Ubuntu 20.04LTS for my Linux version