Compiling Magic Lantern on a Macintosh

[Levas]

What do you see when running this?

Code: [Select]

python -c "import sys; print('\n'.join(sys.path))"

I see this:

Code: [Select]

/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload
/Users/magic_lantern/Library/Python/2.7/lib/python/site-packages
/usr/local/lib/python2.7/site-packages
/usr/local/Cellar/numpy/1.14.5/libexec/nose/lib/python2.7/site-packages
/Library/Python/2.7/site-packages
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC

Haven't try to reinstall python 2, is it better or needed for compiling ?

[Danne]

It seems to be there, must have been paths.
Well, I would play around with butterflow and then reinstall python2. Probably more futureproof.

[Levas]

Couldn't get butterflow to work (gonna give it a try on a new user account, so when things get messy again, I delete the user account  )

So I reinstalled python 2, needed to do some forced relinking, changed the default.user.file back to python2 and ML compiling is working again with python2, thanks 

[dfort]

Yeah, I also had issues when installing dependencies for various applications. Good to hear you got python2 working again.

[dfort]

For those of us who like to live on the bleeding edge--Apple released a major OS update called Mojave (10.14). After running the update I had to re-install the xcode command line developer tools. There's also a Homebrew update for Mojave. Seems I did everything right but I'm getting this error when compiling ML:

Code: [Select]

[ XOR_CHK  ]   ../../build_tools/xor_chk
In file included from /usr/local/Cellar/gcc@5/5.5.0_2/lib/gcc/5/gcc/x86_64-apple-darwin17.3.0/5.5.0/include/stdint.h:9:0,
                 from ../../build_tools/xor_chk.c:2:
/usr/local/Cellar/gcc@5/5.5.0_2/lib/gcc/5/gcc/x86_64-apple-darwin17.3.0/5.5.0/include-fixed/stdint.h:27:32: fatal error: sys/_types/_int8_t.h: No such file or directory
compilation terminated.
make: *** [../../build_tools/xor_chk] Error 1
I also tried wiping out brew and starting over with Danne's Compiler.app but I'm still getting this error.

There's also problems building the ML command line tools:

Code: [Select]

make mlv_dump
Using ~/gcc-arm-none-eabi-5_4-2016q3/bin/arm-none-eabi-gcc (preferred).
[ HOST_CC  ]   mlv_dump.host.o
In file included from mlv_dump.c:22:0:
/usr/local/Cellar/gcc@5/5.5.0_2/lib/gcc/5/gcc/x86_64-apple-darwin17.3.0/5.5.0/include-fixed/math.h:45:23: fatal error: sys/cdefs.h: No such file or directory
compilation terminated.
make: *** [mlv_dump.host.o] Error 1
The MinGW compiler works fine so there seems to be an issue with Mojave and the Homebrew gcc compiler.

[EDIT] Workaround is to remove the Makefile.user section that points to the Homebrew gcc-5 compiler. This works for compiling ML but not for the command line tools, mlv_dump and cr2hdr.

[Danne]

Could you try uninstall command line tools and reinstall them?

[dfort]

Could you try uninstall command line tools and reinstall them?

Been there, done that. Seems to be a problem with the Homebrew version of gcc-5 not being compatible with Mojave.

[Danne]

Just upgraded command line tools and now facing the same issues as you describe.

[Danne]

Ok, fix seems this:
https://stackoverflow.com/questions/52509602/cant-compile-c-program-on-a-mac-after-upgrade-to-mojave

Install macOS_SDK_headers_for_macOS_10.14.pkg by finding it here:
/Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg

Conclusion: command line tools will not install the same way anymore. Will fix the script later.
Compiles platform build
Compiles both mlv_dump and cr2hdr

[dfort]

Ok--this command fixed it:

Code: [Select]

open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg

[Danne]

Should rework the script to do proper reinstalls/updates.

[Danne]

Compiler.app updated with fix for Mojave:
https://bitbucket.org/Dannephoto/compiler/commits/5ccdf722a114040229c8680d2d97ce660ca831bd

[scrax]

Code: [Select]

[ CP       ]   newlib-libm.a
[ CP       ]   gcc-libgcc.a
[ CC       ]   disp_direct.o
[ CC       ]   font_direct.o
[ CC       ]   footer.o
[ XOR_CHK  ]   ../../build_tools/xor_chk
/bin/sh: cc-5: command not found
make[1]: *** [../../build_tools/xor_chk] Error 127
make[1]: *** Waiting for unfinished jobs....
make: *** [600D.102] Error 2

that is my problem compiling ML following this guide, on Osx ElCapitain 10.11.16

had to change the Makefile in build_tools/ to have correct gcc-5 instead of cc-5 (like in the error)

from this:

Code: [Select]

$(call build,XOR_CHK,$(HOST_CC) $< -o $@)to this:

Code: [Select]

$(call build,XOR_CHK,gcc-5 $< -o $@)

[DeafEyeJedi]

Great catch @scrax and thanks for the much needed reminder... Gonna give this script of @Danne's another go while on Mojave 10.14.3!   

[dfort]

Did you make a Makefile.user file? The advantage of doing this is that it isn't tracked so you can customize your build environment without making any changes to the source code. You can put in several options in there and uncomment them as needed. Here's what's in mine:

Makefile.user.default

Code: [Select]

#
# Host compiler settings
#
HOST_CC=gcc-5
HOST_LD=gcc-5
HOST_AR=$(shell which ar)

# CONFIG_QEMU = y
# LOG_INTERRUPTS = y
# CONSOLE_DEBUG = y
# CONFIG_DEBUGMSG = y
# CONFIG_DEBUG_INTERCEPT_STARTUP = y
# CONFIG_DEBUG_INTERCEPT = y
# CONFIG_GDB      = y
# CONFIG_GDBSTUB  = y
# CONFIG_MMIO_TRACE=y

# Recovery branch options:
# CONFIG_BOOT_FULLFAT=y
# CONFIG_BOOT_DUMPER=y
# CONFIG_BOOT_SROM_DUMPER=y

# VERSION = $(call eval_once,VERSION,raw_video_10bit_12bit_LVState.$(shell LC_TIME=EN date +'%Y%b%d').$(MODEL)$(FW_VERSION))

You can do the same with the modules. Here, I was testing a bunch of modules in one of the bleeding edge branches to see if they will compile.

Makefile.modules.user

Code: [Select]

ML_MODULES += \
 mlv_rec \
 pic_view \
 ettr \
 dot_tune \
 autoexpo \
 arkanoid \
 deflick \
 img_name \
 bench \
 selftest \
 adv_int \
 edmac \
 ime_base \
 ime_null \
 ime_rot \
 ime_std \
 img_name \
 io_crypt \
 iso_regs \
 mem_spy \
 mpu_dump \
 plot \
 raw_diag \
 sf_dump \
 trace \
 bulb_nd \
 bolt_rec \
 adtg_gui \
 raw_twk \
 edmac \
 filepref \
 

[scrax]

Did you make a Makefile.user file? The advantage of doing this is that it isn't tracked so you can customize your build environment without making any changes to the source code. You can put in several options in there and uncomment them as needed. Here's what's in mine:

Makefile.user.default

Code: [Select]

#
# Host compiler settings
#
HOST_CC=gcc-5
HOST_LD=gcc-5
HOST_AR=$(shell which ar)

# CONFIG_QEMU = y
# LOG_INTERRUPTS = y
# CONSOLE_DEBUG = y
# CONFIG_DEBUGMSG = y
# CONFIG_DEBUG_INTERCEPT_STARTUP = y
# CONFIG_DEBUG_INTERCEPT = y
# CONFIG_GDB      = y
# CONFIG_GDBSTUB  = y
# CONFIG_MMIO_TRACE=y

# Recovery branch options:
# CONFIG_BOOT_FULLFAT=y
# CONFIG_BOOT_DUMPER=y
# CONFIG_BOOT_SROM_DUMPER=y

# VERSION = $(call eval_once,VERSION,raw_video_10bit_12bit_LVState.$(shell LC_TIME=EN date +'%Y%b%d').$(MODEL)$(FW_VERSION))

You can do the same with the modules. Here, I was testing a bunch of modules in one of the bleeding edge branches to see if they will compile.

Makefile.modules.user

Code: [Select]

ML_MODULES += \
 mlv_rec \
 pic_view \
 ettr \
 dot_tune \
 autoexpo \
 arkanoid \
 deflick \
 img_name \
 bench \
 selftest \
 adv_int \
 edmac \
 ime_base \
 ime_null \
 ime_rot \
 ime_std \
 img_name \
 io_crypt \
 iso_regs \
 mem_spy \
 mpu_dump \
 plot \
 raw_diag \
 sf_dump \
 trace \
 bulb_nd \
 bolt_rec \
 adtg_gui \
 raw_twk \
 edmac \
 filepref \
 

I have a Makefile.user made by editing the default one and on the host compiler settings part it's same as yours

HOST_CC=gcc-5
HOST_LD=gcc-5

but when compiling $(HOST_CC) became "cc-5" instead of "gcc-5" so had to put manually in the build_tools/Makefile

[dfort]

but when compiling $(HOST_CC) became "cc-5" instead of "gcc-5" so had to put manually in the build_tools/Makefile

Hum--doesn't make sense. Have you tried it without Makefile.user.default? As far as I know it is really only needed when compiling cr2hdr. Everything else should work out of the box on the Mac. Maybe the Homebrew compiler is configured in a way that clashes with the Apple compiler?

Code: [Select]

which gcc
/usr/bin/gcc
which cc
/usr/bin/cc
which cc-5 <-- I don't have this on my system
which clang
/usr/bin/clang
which gcc-5
/usr/local/bin/gcc-5 <-- Hombrew compiler


[scrax]

Hum--doesn't make sense. Have you tried it without Makefile.user.default? As far as I know it is really only needed when compiling cr2hdr. Everything else should work out of the box on the Mac. Maybe the Homebrew compiler is configured in a way that clashes with the Apple compiler?

Code: [Select]

which gcc
/usr/bin/gcc
which cc
/usr/bin/cc
which cc-5 <-- I don't have this on my system
which clang
/usr/bin/clang
which gcc-5
/usr/local/bin/gcc-5 <-- Hombrew compiler


Yes is same here no cc-5 an other compiler are in same places as yours.

Code: [Select]

which clang gcc gcc-5 cc cc-5
/usr/bin/clang
/usr/bin/gcc
/usr/local/bin/gcc-5
/usr/bin/cc

I've not tried without Makefile.user but tried to change gcc-5 with cc in build_tools/Makefile and it worked too so maybe I can make an alias for cc-5 to cc, what you think? (don't work)

[DeafEyeJedi]

@Danne's magic still appears to be working fine on Mojave with the exception of few Modules being failed to build. Will need to investigate this a little deeper.

Now I gotta figure out how to get @Danne's presets to include... probably need to find his branch. There's 181 of them for me to dig through via Compiler. 

[dfort]

@scrax - Just noticed you're a developer so you probably know more about this than I do but here's what is going on at my end when I tried to replicate the error.

For sure there is no space between the 'g' and 'cc-5' in your Makefile.user file? Try compiling without the Makefile.user (and of course without your changes to build_tools/Makefile) you'll get this warning but it should work:

modules/mlv_rec

Code: [Select]

make mlv_dump
...
ld: warning: The i386 architecture is deprecated for macOS (remove from the Xcode build setting: ARCHS)

If that works for you try setting gcc-5 on the command line:

Code: [Select]

make mlv_dump HOST_CC=gcc-5
@DeafEyeJedi - The branch you are looking for is crop_rec_4k_mlv_snd_isogain_1x3_presets in Danne's repository.

[DeafEyeJedi]

@DeafEyeJedi - The branch you are looking for is crop_rec_4k_mlv_snd_isogain_1x3_presets in Danne's repository.

Thanks for pointing me into the right path. Seems I've succeeded with getting @Danne's branch set to default via Compiler. Looks sexy!   

[scrax]

@scrax - Just noticed you're a developer so you probably know more about this than I do but here's what is going on at my end when I tried to replicate the error.

I'm more a tester than a developer, also I forgot all what I learned about ML 

I've removed Makefile.user and xor_chk got compiled right without the need to edit it's Makefile and that's what I was looking for so at this point seems better for me to not use the homebrew gcc at all.

Thank's dfort 

[Danne]

@Deafyeyjedi
Here are some steps if you want to use my fork and the branch used for bleeding edge eosm:

For other users. Compiler.app can be found here:
https://www.magiclantern.fm/forum/index.php?topic=21882.msg199370#msg199370

Once you double clicked Compiler.app and installed all dependencies do this:

1 - Download my repository. In terminal do:

Code: [Select]

hg clone https://Dannephoto@bitbucket.org/Dannephoto/magic-lantern magic-lantern_dannephotoThis will download my repo and rename it to magic-lantern_dannephoto. It should land in your user folder

2 - Drag the repository named magic-lantern_dannephoto onto Compiler.app. Once the menu opens choose:
(b)  branches

3 - You´ll see at the top that your branch right now is:
current branch: default.
What you want is to be in:
current branch: crop_rec_4k_mlv_snd_isogain_1x3_presets

The top of source tree should look like this right now:
1   crop_rec_4k_mlv_snd_isogain_1x3_presets
2   digic6-dumper
3   recovery
4   crop_rec_4k_mlv_snd_isogain_1x3_presets_EOSM2
5   crop_rec_4k_5D2_rd_stuff
...

4 - So to get there simply print 1 at the bottom since the branch you want is the first one(hg source tree always keeps latest first) like this:
Please enter your selection number below and press enter:
1

If it looks like this at the top all has worked:
current branch: crop_rec_4k_mlv_snd_isogain_1x3_presets

5 - Now go back to:
(m)  main

6 - To compile select platform first in main menu:
(p)  platform

7 - select 17 for eosm:
17   EOSM.202

8 - Choose c for compiling and just wait until it´s done. When ready Compiler.app will open up the platform folder that you compiled from:
(c)  compile

The whole point of this tool is that you can enter ml folders by navigating the menus rather than opening up manually.

[DeafEyeJedi]

@Danne
Thanks so much for your thorough explanation. I was able to get it configured properly per your request. Now it's even more sexier! 

Just spat out a build for 100D... Perhaps let's see how bloody this bad boy can be, ain't that right @OlRivrRat? 

[Danne]

hehe, nice desktop image too