Compiling Magic Lantern with Cygwin/MinGW-64

[garry23]

@dfort

Just to say I really appreciate your efforts here.

As soon as I am able to give it some time, and know things are 'fixed', I will take the compile plunge 

Cheers

Garry

[dfort]

@garry23

Things are fixed! a1ex merged the changes into the unified branch so you should merge unified from the main repository into your working branches.

I just ran through the tutorial and everything worked as expected. From bare metal to a fully functional ML build took less than an hour on a very slow 10 year old laptop.

[dfort]

I ran through the tutorial on a much faster 64-bit Windows 7 desktop system and went from zero to finished ML build in 15 minutes. Also learned a few things along the way like it is better to setup a 32-bit Cygwin installation even though the computer has a 64-bit processor. Why? Because the command line tools like mlv_dump won't build properly with the 64-bit gcc used in Cygwin. That's only important if you want to build Cygwin native applications, you really should be using MinGW-64 to build Windows native applications.

Updated the tutorial so Windows 64-bit users should have a better experience.

[dfort]

Tried compiling the command line tools. mlv_dump, raw2dng, built fine but there's a problem with cr2hdr:

Code: [Select]

$ make cr2hdr.exe
Updated HGVERSION
[ README   ]   module_strings.h
Help page 1: Quick start
Help page 2: Tips and tricks
CROSS=1 make cr2hdr.exe
make[1]: Entering directory '/home/dfort/magic-lantern/modules/dual_iso'
[ i686-w64-mingw32-gcc ]   cr2hdr.exe
cr2hdr.c:544:20: error: conflicting types for ‘module_get_string’
 static const char* module_get_string(const char* name)
                    ^
In file included from cr2hdr.c:56:0:
../../src/module.h:268:13: note: previous declaration of ‘module_get_string’ was here
 const char* module_get_string(int mod_number, const char* name);
             ^
make[1]: *** [Makefile.cr2hdr:18: cr2hdr.exe] Error 1
make[1]: Leaving directory '/home/dfort/magic-lantern/modules/dual_iso'
make: *** [Makefile.cr2hdr:21: cr2hdr.exe] Error 2


[Danne]

Seems a problem on mac too. If you erase the conflicting string

Code: [Select]

const char* module_get_string(int mod_number, const char* name); from module.h cr2hdr will compile. Not the solution but some baby step direction 

Here´s some more info.
https://stackoverflow.com/questions/5691650/conflicting-types-error-when-compiling-c-program-using-gcc


You could rename module_get_string(two places) in cr2hdr.c to something like  my_module_get_string and get rid of the conflict as well.

[dfort]

Interesting -- this must be affecting Linux also. Can anyone verify?

[Danne]

I noticed another error in crop_rec_4k branch when compiling cr2hdr. Complexity of bleeding edge?

Code: [Select]

[ gcc-5    ]   cr2hdr
cr2hdr.c:544:20: error: conflicting types for 'module_get_string'
 static const char* module_get_string(const char* name)
                    ^
In file included from cr2hdr.c:56:0:
../../src/module.h:274:13: note: previous declaration of 'module_get_string' was here
 const char* module_get_string(int mod_number, const char* name);
             ^
cr2hdr.c: In function 'module_get_string':
cr2hdr.c:546:34: error: '__module_strings_MODULE_NAME' undeclared (first use in this function)
     module_strpair_t *strings = &__module_strings_MODULE_NAME[0];

[dfort]

Created bug report: cr2hdr fails to build

[Danne]

https://bitbucket.org/hudson/magic-lantern/commits/37426e9f8f9f5c6e1bcd69b7b2f7f85eafa6f456?at=crop_rec_4k
 

[dfort]

Thanks -- works now. Should get that change into the unified branch.

[Danne]

Not the same changes exactly but that minor conflicting string nedds to be fixed in unified. Glad it works on my mac now. Thanks g3gg0.

[dfort]

Thanks g3gg0!

Fixed.

[Danne]

Had some nice progress wth building a cygwin environment inside a windows 10 os set up in virtual box on my mac. The first 4 commands in first post was all it took to get compiling working.
Then I pulled down some branches I work with on my mac and then got the errors recognized here.
http://www.magiclantern.fm/forum/index.php?topic=17099.msg165820#msg165820

Code: [Select]

Updated HGVERSION
[ README   ]   module_strings.h
Traceback (most recent call last):
  File "../readme2modulestrings.py", line 113, in <module>
    txt = run('cat README.rst | grep -v -E "^:([^:])+:.+$" | ' + rst2htmlCommand + ' | python ../html2text.py -b 700 | sed "s/\r$//"')
TypeError: cannot concatenate 'str' and 'NoneType' objects
../Makefile.modules:39: recipe for target 'module_strings.h' failed
make: *** [module_strings.h] Error 1
Solution usually is to symlink python like following but this failed
for f in /usr/bin/*.py; do ln -s $f "${f/.py/}"; done

Reading further down the forum post link above user Escho install python3. This is also what worked with cygwin. Adding the packages for python3 solved the issues compiling mlv_dump and also cr2hdr on windows 10.

All in all a great tutorial dfort.

[Danne]

Been digging around cygwin so gonna share my latest findings.

FIrst script should change python2-docutils to python3-docutils.

Code: [Select]

%UserProfile%\Downloads\setup-x86.exe -s http://cygwin.mirrors.pair.com -q -P mingw64-i686-gcc-core,gcc-core,make,mercurial,python3-docutils,perl,wget,unzip,zip,diffutils
This won´t be needed anymore

Code: [Select]

for f in /usr/bin/*.py; do ln -s $f "${f/.py/}"; done

I was a bit sloppy building mlv_dump without the extension .exe which made mlv_dump dependent on cygwin1.dll. After a few reinstallations I actually noticed dfort wrote about this at the bottom of the first post, hmm.

Alway do e.g make mlv_dump.exe or make cr2hdr.exe, not make mlv_dump

[dfort]

If you stay within the Cygwin environment you'll never know.

Best to always compile using make mlv_dump.exe because that binary will run both inside and outside of Cygwin.

If anyone wants a real challenge try getting QEMU running in Cygwin.

[dfort]

Whoa--this is really weird.

Traced down the problem to what seems to be an issue with Cygwin's docutils. This affects both python2 and 3.

This is with 0.13.1-1 of python2-docutils

Code: [Select]

$ ls -la /usr/bin/*.py
-rwxr-xr-x 1 Dan None  6488 Mar 14 21:30 /usr/bin/msgfmt.py
-rwxr-xr-x 1 Dan None 22097 Mar 14 21:30 /usr/bin/pygettext.py
-rwxr-xr-x 1 Dan None   594 Mar 13 23:12 /usr/bin/rst2html.py
-rwxr-xr-x 1 Dan None  1140 Mar 13 23:12 /usr/bin/rst2html5.py
-rwxr-xr-x 1 Dan None   791 Mar 13 23:12 /usr/bin/rst2latex.py
-rwxr-xr-x 1 Dan None   600 Mar 13 23:12 /usr/bin/rst2man.py
-rwxr-xr-x 1 Dan None   764 Mar 13 23:12 /usr/bin/rst2odt.py
-rwxr-xr-x 1 Dan None  1698 Mar 13 23:12 /usr/bin/rst2odt_prepstyles.py
-rwxr-xr-x 1 Dan None   601 Mar 13 23:12 /usr/bin/rst2pseudoxml.py
-rwxr-xr-x 1 Dan None   637 Mar 13 23:12 /usr/bin/rst2s5.py
-rwxr-xr-x 1 Dan None   871 Mar 13 23:12 /usr/bin/rst2xetex.py
-rwxr-xr-x 1 Dan None   602 Mar 13 23:12 /usr/bin/rst2xml.py
-rwxr-xr-x 1 Dan None   670 Mar 13 23:12 /usr/bin/rstpep2html.py
-rwxr-xr-x 1 Dan None 18545 Mar 14 21:29 /usr/bin/smtpd.py

And this is with the current 0.13.1-2

Code: [Select]

Dan@Laptop-PC /usr/bin
$ ls -la /usr/bin/*.py
-rwxr-xr-x 1 Dan None  6488 Mar 14 21:30 /usr/bin/msgfmt.py
-rwxr-xr-x 1 Dan None 22097 Mar 14 21:30 /usr/bin/pygettext.py
-rwxr-xr-x 1 Dan None 18545 Mar 14 21:29 /usr/bin/smtpd.py

Where did rst2html go? No wonder my symlink trick stopped working. You can still go back one version on the Cygwin installer but the default is to upgrade you to the latest.

[Danne]

Leaves us with both cygwin and crosscompiler mingw-64 broken. At least cygwin let's you compile mlv_dump

[dfort]

...crosscompiler mingw-64 broken. At least cygwin let's you compile mlv_dump

mingw64-i686-gcc isn't broken, at least not in Cygwin. There's a problem on the Mac with the Homebrew package of mingw64-i686-gcc that I'm using in the tutorial.

[dfort]

The fix for now is to launch the Cygwin installer, find python2-binutils, revert to 0.13.1-1 then run my one-liner:

Code: [Select]

for f in /usr/bin/*.py; do ln -s $f "${f/.py/}"; done
That should get you back up and running.

I had some other issues trying to work this out because I had the Python for Windows installed and Cygwin would try to run the rst2html script from that package and it kept failing.

Sent a message to the Cygwin mailing list to see if there is a reason those docutils scripts vanished.

[dfort]

It looks like python2-binutils won't be fixed anytime soon. Turns out that python2-binutils and python3-binutils use the same scripts and for some reason they can't be in both packages at the same time.

python2-binutils 0.13.1-1 works
python3-binutils 0.13.1-1 broken

python2-binutils 0.13.1-2 broken
python3-binutils 0.13.1-2 works

Long story short -- I switched the installation so it uses python3-binutils. I didn't do this before because I didn't want to move to python3 but as it turns out Cygwin defaults to using python2 with the way we're installing the ML development environment. Go figure.

Code: [Select]

$ python --version
Python 2.7.13

So the installation on the first post is working again. If you start with a clean slate just delete C:\cygwin and run the tutorial again.

[Danne]

Another tip coming from Bouncyball is to strip mlv_dump.exe from unneeded stuff.
make mlv_dump.exe && strip mlv_dump.exe

Worked with MinGW-64 but only after I replaced mac strip with the one from MinGW-64 into the xcode version here:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/strip

If not wanting to replace strip following should work when running make on mlv_dump:
make mlv_dump.exe && /usr/local/Cellar/mingw-w64/5.0.2_1/toolchain-i686/i686-w64-mingw32/bin/strip mlv_dump.exe

[dfort]

Those paths are for Mac. Haven't tried it on Cygwin yet but it should work.


Sent from my iPad using Tapatalk

[Danne]

Oops. Thought MinGW-64 was for mac only...

[dfort]

MinGW is Minimalist GNU for Windows and it is available for Mac, Linux and Windows development environments though the binaries will most likely only run on Windows.

I tried that trick on Cygwin and it works. Either version of strip seems to do the trick:

Code: [Select]

strip mlv_dump.exe

Code: [Select]

i686-w64-mingw32-strip mlv_dump.exe
Before: 631,091 mlv_dump.exe
After:  154,112 mlv_dump.exe

[Audionut]

Thanks dfort.  Nice and easy.

Now just to work my head around SourceTree.