Compiler.app(https://s18.postimg.cc/sin6ogmux/Screen_Shot_2018-04-17_at_22_scaled.png)
Compiler.app is an automation tool aiming to be an easy one click solution setting up a magic lantern development environment on a mac. Main purpose is to get fast access into som basic functions such as changing branches, compile on a platform, module etc. Hopefully this also will encourage newcomers to get straight into compiling and testing. The tool can be used even if you already have a development environment installed.
Sourcehttps://bitbucket.org/Dannephoto/ml_compiler-git/src/master/
Downloadhttps://bitbucket.org/Dannephoto/ml_compiler-git/downloads/ml-compiler.dmg
Howto:
- Download Compiler.app then Double click Compiler.app and follow instructions(link above)
(Bypass gatekeeper by holding down ctrl when clicking the application the first time) - To change or add a repository just drag your magic-lantern repository onto Compiler.app
Most of the stuff is a continuation of what is brought into this thread (https://www.magiclantern.fm/forum/index.php?topic=16012.0/) by dfort. Bigger part of dfort quick installation script is included into Compiler.app.
(license gpl)
Example usage:Here are some steps if you want to use my fork and the branch used for bleeding edge eosm:
Once you double clicked compiler.app and installed all dependencies do this:
1 - Download my repository. In terminal do:
hg clone https://[email protected]/Dannephoto/magic-lantern magic-lantern_dannephotoThis will download my repo and rename it to magic-lantern_dannephoto. It should land in your user folder
2 - Drag the repository named
magic-lantern_dannephoto onto Compiler.app. Once the menu opens choose:
(b) branches3 - You´ll see at the top that your branch right now is:
current branch: default.
What you want is to be in:
current branch: crop_rec_4k_mlv_snd_isogain_1x3_presets
The top of source tree should look like this right now:
1 crop_rec_4k_mlv_snd_isogain_1x3_presets
2 digic6-dumper
3 recovery
4 crop_rec_4k_mlv_snd_isogain_1x3_presets_EOSM2
5 crop_rec_4k_5D2_rd_stuff
4 - So to get there simply print
1 at the bottom since the branch you want is the first one(hg source tree always keeps latest first) like this:
Please enter your selection number below and press enter:1
If it a the top looks like this all has worked:
current branch: crop_rec_4k_mlv_snd_isogain_1x3_presets
5 - Now go back to:
(m) main6 - To compile select platform first in main menu:
(p) platform7 - select 17 for eosm:
17 EOSM.2028 - Choose c for compiling and just wait until it´s done. When ready Compiler.app will open up the platform folder that you compiled from:
(c) compile
@Danne: very cool! Got the toolchain and the repos. Is it also possible to compile with compiler.app? I only found make clean...
Thanks masc :)
Yes, compiling possible. You navigate through menus, first enter (b) branches and select a branch, head back to (m) main then get into (p) platform and select your camera and this will happen:
(https://s18.postimg.cc/u738ibbnt/Screen_Shot_2018-04-05_at_14.25.34.png)
Just print make zip and when done the folder will open up serving you the goodies :)
Yeaha. That works. Nice.
On installing there comes the question "Press RETURN to continue or any other key to abort" ... there is no difference between RETURN and other keys.
But hey, I can compile! :D
Is it possible to install to another directory? All is installed to /Users/<name>/
Great!
Yes, I´m aware of the abort failure. Will fix it. It´s in the script from dfort. Thanks for pointing that out.
You can drag any repo onto Compiler.app and that will become the repo you work from after this. It´s only installing default magic-lantern repo on vanilla installs(If this is what you´re asking regarding installing to another directory). This means you can move repos anywhere after downloading them and when dragged to Compiler.app work from there.
*update
Well, it´s actually not continuing after not hitting enter. The script correctly skips brew installation but immidietaly goes to the next command in the script which is brew related so one will think it´s installing anyway. I put in a safety check if brew already installed so it skips the ruby command all together:
if ! [ -f "/usr/local/bin/brew" ]
then
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
fi
brew install gcc@5 python wget mercurial
@Danne - updated the quick install script from the Compiling Magic Lantern on a Macintosh (https://www.magiclantern.fm/forum/index.php?topic=16012.0) tutorial with your changes. Since the script lives on the forum and is not under version control please check it to make sure we stay in sync.
A better option is if you have the script in your repository I can just link to it from the tutorial so it will always be in sync. I already put in a link to this topic in the tutorial.
Nice one dfort :)
The quick installation script would be put into a repository you mean?
Or it could be detached into a subscript in Compiler.app and the referred/linked to but then it wouldn´t be a copy paste thingy since I did some minor changes to fit the compiler.
Maybe best to create a repo here? I could update it as I add changes in Compiler.app.
Only loss will be that it will be linked and not a direct code snippet in your forum post...
@Danne: I don't mean the magic-lantern folder. I mean the gcc-arm-none-eabi-4_8-2013q4 & gcc-arm-none-eabi-5_4-2016q3 folder. I would like to have them somewhere else...
@masc: as long as you have arm-none-eabi-gcc in your PATH, ML makefiles should pick it up without any tweaking. This (https://bitbucket.org/hudson/magic-lantern/src/unified/Makefile.setup.toolchain) looks up the compiler, so you can see what it tries.
General instructions: https://www.magiclantern.fm/forum/index.php?topic=991
@Danne: yeah, looks like we're already out of sync. I was thinking of something along the lines of this stackoverflow tip (https://stackoverflow.com/questions/192292/bash-how-best-to-include-other-scripts). My quick install script is meant to be copied and pasted into the terminal so the part that writes the file to disk before executing then deleting it when finished needs to be changed.
I should be able to do something like this only with the quick install script:
cd ~
wget https://bitbucket.org/Dannephoto/compiler/raw/default/Compiler.app/Contents/main.command
bash main.command && rm main.command
Try it.
BTW--looks like we're both using a non-portable bash shebang and should probably change it to "#!/usr/bin/env bash" like we did in the QEMU scripts (https://bitbucket.org/hudson/magic-lantern/pull-requests/834/qemu-build-tweaks-1/diff).
I removed gcc-arm-none-eabi-4_8-2013q4 and gcc-arm-none-eabi-5_4-2016q3 and then added:
brew install caskroom/cask/gcc-arm-embedded
This seems to run ok, tried compiling for 5D mark III, mlv_dump and cr2hdr. Picks up:
Using /usr/local/bin/arm-none-eabi-gcc (from PATH).
If this seems a valid change I could replace the (preferred) gcc-arm versions with this?
@dfort. I´l look into your suggestions.
Seems to work nice @dfort. Nice suggestion. I added a few changes to have this work with (c) option too. I would suggest for this command:
[ ! -f "`which brew`" ] && /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
[ ! -f "`which wget`" ] && brew install wget
cd ~
sleep 1 && clear && echo "Press enter to proceed" &
if [ -f main.command ]; then rm main.command; fi
wget https://bitbucket.org/Dannephoto/compiler/raw/default/Compiler.app/Contents/main.command
bash main.command && rm main.command
So no more excuses for Mac users not setting up a Magic Lantern development environment!
Thanks for keeping the install script separate so that it can be run as a stand alone script. Now it is under revision control just like a "real" app. Linked to this project from the Compiling Magic Lantern on a Macintosh (https://www.magiclantern.fm/forum/index.php?topic=16012.0) tutorial.
Nice collaboration dfort and good idea to keep the script as a separate tool inside Compiler.app.
Quote from: Danne on April 05, 2018, 05:29:13 PM
I removed gcc-arm-none-eabi-4_8-2013q4 and gcc-arm-none-eabi-5_4-2016q3 and then added:
brew install caskroom/cask/gcc-arm-embedded
This seems to run ok, tried compiling for 5D mark III, mlv_dump and cr2hdr. Picks up:
Using /usr/local/bin/arm-none-eabi-gcc (from PATH).
I tried that, but for me it does not work. That means compiler.app does not find the arm compiler and wants to install it. Did I miss something?
Should work by just bypassing and neglect to download and install the missing gcc folders or erase following from main.command inside Contents folder in Compiler.app to get rid of the nagging message:
https://bitbucket.org/Dannephoto/compiler/src/8a16fef6c98a38b1dec78bcab4099e826e920659/Compiler.app/Contents/main.command?at=default&fileviewer=file-view-default
lines 64-71:
if ! test -d ~/gcc-arm-none-eabi-4_8-2013q4;
then
gcc4=$(tput setaf 1)$(echo gcc4_8-2013q4 missing)$(tput sgr0)
fi
if ! test -d ~/gcc-arm-none-eabi-5_4-2016q3;
then
gcc5=$(tput setaf 1)$(echo gcc5_4-2016q3 missing)$(tput sgr0)
fi
Thx Danne! That works! I commented these lines out. 8)
Great masc. And good tip from A1ex.
Added some refinements to the (b) branches menu item:
(https://s7.postimg.cc/e1812f5wr/Screen_Shot_2018-05-16_at_15.12.20.png_500px.jpg)
keyword search...
(https://s7.postimg.cc/audhist6j/Screen_Shot_2018-05-16_at_15.12.32.png_500px.jpg)
After keyword search
(https://s7.postimg.cc/4goefjw0b/Screen_Shot_2018-05-16_at_15.12.38.png_500px.jpg)
Error when installing Compiler.app:
'main.command' can not be opened because Sandbox is not allowed to open documents in Terminal.
P.s. 'Show contents' in Compiler.app and running 'main.command' in Terminal did the trick!
Interesting. Did you open it by ctrl click and open the first time?
By the way. What OS are you using?
Yep. Tried Ctrl-click the first time. Failed.
The latest macOS 10.13.4 on an iMac (Retina 5K, 27-inch, 2017).
P.S. As far as I 'know' it is working alright now. Still learning! ;-)
Cool. I'll see if I can reproduce this and then upload a fixed version. Compiling working as expected?
Tried to compile 17 80D.101
I get the next error after each command:
********************************************************
WARNING: module mlv_rec failed to build, deleting
********************************************************
Unfortunately the terminal is cleared after compiling.
So it is impossible to see the other errors.
I will remove the magic-lantern and qemu-eos folders and start from scratch.
80D, that camera hasn't got magic lantern working yet I think?
What branch were you compiling in?
You can scroll up in terminal after compiling to see what else didin't work even after window is cleared.
You're right. 80D isn't working yet. All modules failed to build.
Did compile in 80D-troubleshooting branch.
What is best way to test the Compile.app install?
I also got this warning:
*** WARNING: 64-bit GDB is known to have issues.
Are there ROM-dumps for the 80D and 6D Mark II?
I use crop_rec mlv lite with sound support however either of the cams you mention has ml running but are in very early stages.
Search on the forum for questions about rom dumps. Not sure where to find these but any help porting ml will be appreciated.
@kotik
Pushed a new version but don´t know if it will work better. Only thing changed is a routine for when the app itself could contain spaces, e.g Compiler 2.app which would make the comman script to break. It runs over here on sierra without sand box errors so who knows...
Thnx. Worked even without Ctrl-click the first time.
Maybe because the folders 'magic-lantern' etc. where already present?
Where can I find crop_rec mlv lite with sound support?
It is almost impossible to search all those branches.
Or maybe you made the sandbox exception permanent for the main.command script already? Thanks for testing!
@kotik
You can narrow down branches by selecting (s) and specify a keyword. Try printing crop_rec and enter to fetch all branches with that word inside. It should be there. Something like crop_rec_4k_mlv_lite_snd I think it's called.
@Danne
The search example in the Compiler.app suggests 'croprec' not 'crop_rec'.
Yes, I know. Will change that when I can. Anyway. Just write crop and you'll be fine.
It is pretty interesting. On which OS you are working now?
When I compile the crop_rec_4k_mlv_snd_isogain_1x3_presets branch via the instructions in the first post and enable sd_uhs I get the following message on boot up
sd_uhs.mo is not supported for your camera
I know I've chosen the right camera under platform because the .zip its named crop_rec_4k_mlv_snd_isogain_1x3_presets_2019Apr07.6D116.zip
A previous crop_rec build with sd_uhs.so compiled by someone else was working but I'd love to use your latest build.
Well, you need to enable it for your specific cam:
Check for this place in sd_uhs code:
if (is_camera("6D", "1.1.6"))
{
sd_setup_mode = 0xFF325A20;
sd_setup_mode_in = 0xFF325AA8;
sd_setup_mode_reg = 1; /* switch variable is in R1 (likely all D5 other than 5D3) */
sd_set_function = 0xFF78F308;
/* sd_uhs_patch(); */ }
Change this line to:
sd_uhs_patch(); }
And if you want to loose the complaint on screen add the 6D into the mix:
if ((!is_camera("EOSM", "2.0.2")) && (!is_camera("100D", "1.0.1")) && (!is_camera("700D", "1.1.5") && (!is_camera("6D", "1.1.6")))
{
NotifyBox(2000, "sd_uhs.mo is not supported for your camera");
}
Untested territory. I don´t own this cam myself.
Quote from: Danne on April 08, 2019, 07:13:08 AM
Well, you need to enable it for your specific cam:
Check for this place in sd_uhs code:
if (is_camera("6D", "1.1.6"))
{
sd_setup_mode = 0xFF325A20;
sd_setup_mode_in = 0xFF325AA8;
sd_setup_mode_reg = 1; /* switch variable is in R1 (likely all D5 other than 5D3) */
sd_set_function = 0xFF78F308;
/* sd_uhs_patch(); */ }
Change this line to:
sd_uhs_patch(); }
And if you want to loose the complaint on screen add the 6D into the mix:
if ((!is_camera("EOSM", "2.0.2")) && (!is_camera("100D", "1.0.1")) && (!is_camera("700D", "1.1.5") && (!is_camera("6D", "1.1.6")))
{
NotifyBox(2000, "sd_uhs.mo is not supported for your camera");
}
Untested territory. I don´t own this cam myself.
Just found it as you were replying. Enabled and working flawlessly so far! 80+ second recording at max 1824x1026 @ 30 fps 14-bit lossless. Will report back if there are problems.
Appreciate your efforts greatly!
Now if only we can get decent 60fps video without aliasing (at 16:9 without losing FoV)! lol
Good stuff.
This all installed fine and setup dependencies (AFAIK), but no matter what repo/branch combination I select, I just get this error on compile for all modules.
module.h:344:28: fatal error: module_strings.h: No such file or directory
Any ideas?
This is caused by ML assuming that "python" resolves to python 2 (breaks the module string generation so the header file is missing). You almost certainly have "python" resolve to python 3 (Catalina? I don't have a Mac to test).
The quick hack way to fix is to make python point to python 2. The slightly nicer way is to edit ML source so that it explicitly calls python2. The best way would be to remove the dependencies on python2 entirely but I haven't done that yet.
I think the changes for option 2 are split over these three commits:
https://bitbucket.org/stephen-e/ml_200d/commits/?search=python
DO NOT apply those commits blindly to Danne's repo, that's from one of my repos and they're messy commits with changes for multiple things. Copying blindly will break your compile. I'm in the middle of tidying this up so should have a nice, single commit this week.
Thanks for responding. Lack of time here.
Porting ml to git should be next. I hope I find time for this soon. Looking at your port seems not that straight forward.
Removing hg dependency is easy, I have a single commit for that:
https://github.com/reticulatedpines/magiclantern_simplified/commit/3163d41cee35fb54b4308e1f32318ff87de53645
This will mean you can't inspect autoexec.bin to tell which commit it's built from. I have a TODO to make it stamp with hg and git if they're both available, and fallback to no stamp if not.
Likely applies cleanly to yours, feel free to cherry-pick it. I am currently merging together unified, qemu and lua_fix branches as those seem to be the important branches.
Oh, if you meant "taking ML mercurcial repo and putting it into Git source control", then that's here:
https://github.com/reticulatedpines/magiclantern
Has all branches. It's a little old but probably no changes happened since then (haven't checked). You can clone from there to your own Git repo.
Quote from: names_are_hard on May 26, 2020, 06:33:59 PM
This is caused by ML assuming that "python" resolves to python 2 (breaks the module string generation so the header file is missing). You almost certainly have "python" resolve to python 3 (Catalina? I don't have a Mac to test).
The quick hack way to fix is to make python point to python 2. The slightly nicer way is to edit ML source so that it explicitly calls python2. The best way would be to remove the dependencies on python2 entirely but I haven't done that yet.
I think the changes for option 2 are split over these three commits:
https://bitbucket.org/stephen-e/ml_200d/commits/?search=python
DO NOT apply those commits blindly to Danne's repo, that's from one of my repos and they're messy commits with changes for multiple things. Copying blindly will break your compile. I'm in the middle of tidying this up so should have a nice, single commit this week.
Thanks for the reply, and thanks you Danne for the Mac script!
I'm only on 10.13.6 on this machine, but I do have Python 2.7, 3.7 and 3.8 installed for other stuff.
Python --version gives me Python 2.7.16 though.
Is there a way to check what version of Python this is referencing?
I'm pretty sure that error is python mixup, as it's exactly the same on mine and I know what causes it there. It does look like your "python" is 2, which is what ML expects, but you're using Danne's stuff around ML to compile, right? I have no idea what that does.
If you scroll further back in the errors, do you see something like this?
Building module adv_int...
[ README ] module_strings.h
$//"README.rst | grep -v -E "^:([^:])+:.+$" | rst2html | python ../html2text.py -b 700 | sed "s/
Traceback (most recent call last):
File "../html2text.py", line 782, in <module>
wrapwrite(html2text(data, baseurl))
File "../html2text.py", line 715, in html2text
return optwrap(html2text_file(html, None, baseurl))
File "../html2text.py", line 150, in optwrap
for line in wrap(str(para), BODY_WIDTH):
File "../html2text.py", line 44, in <lambda>
wrap = lambda text, width: word_wrap(text, width, extent_func)
File "/home/user/code/magiclantern/magiclantern_simplified/modules/align_string_proportional.py", line 21, in word_wrap
lookup = dict((c, extent_func(c)[0]) for c in set(text))
File "/home/useer/code/magiclantern/magiclantern_simplified/modules/align_string_proportional.py", line 21, in <genexpr>
lookup = dict((c, extent_func(c)[0]) for c in set(text))
File "/home/user/code/magiclantern/magiclantern_simplified/modules/rbf_read.py", line 64, in extent_func
w += font.wTable[ord(c) - font.charFirst]
AttributeError: 'Bunch' object has no attribute 'wTable'
This commit may fix it for you, confirmed to work to make 50D build and run in Qemu for me, on a mixed python2 / python3 Linux system:
https://github.com/reticulatedpines/magiclantern_simplified/commit/0801e337c10b243d6a5c19c0b63efa23142823de
If you don't want to mess around with that, you'll probably have to wait for Danne to test / fix it, I don't have a Mac system to test with. If you apply it to your source and it still doesn't build, I can try and help with any errors.
I think docutils are either missing or mixed up. One fast way to fix this mght be this in terminal:
pip install docutils
pip2 install docutils
pip3 install docutilsMy script in the tool should have installed these already but maybe pip isn´t working or something else causing it not to install. What happens when you run the pip commands from terminal?
Some background info:
https://www.magiclantern.fm/forum/index.php?topic=16012.msg155422;topicseen#msg155422
Also check your
Makefile.user.default in ml root folder:
Quote from: dfort on October 14, 2015, 02:36:14 AM
We're almost ready to build Magic Lantern but first we've got to make one small change first. Go into the magic-lantern directory, find the file named Makefile.user.default and you'll see this block of code:
#
# Host compiler settings
#
HOST_CC=$(shell which gcc)
HOST_LD=$(shell which ld)
HOST_AR=$(shell which ar)
Now make a new file called Makefile.user and put in just these lines:
#
# Host compiler settings
#
HOST_CC=gcc-5
HOST_LD=gcc-5
HOST_AR=$(shell which ar)
Warning: DON'T EDIT Makefile.user.default -- Put the changes you want to make to Makefile.user.default in your Makefile.user file. Think of this as a way to customize the Magic Lantern build environment without modifying the code.
Ok pip3 returns
Requirement already satisfied: docutils in /usr/local/lib/python3.7/site-packages (0.16)
Looking again I can also see a lot of /bin/sh: python2: command not found
python3.7 and python3.8 have presumably been installed by some other brew apps, so maybe even though macOS still targets python2.7, Homebrew defaults to 3.7?
But I'm a bit out of my league here....
Please try the makefile fix.
If you're getting "python2: command not found" then I guess Mac doesn't have that command (so, "python" is python3 and there's no python2 at all). You have Python 2 installed, but no way to run version 2 specifically. Make a symlink or alias so that "python2" runs the binary for python 2. Macs are setup in a way that makes it annoyingly difficult to run python programs, I've seen this before.
You can uninstall python 3 and it should jump into python 2.
I guess so, but:
- presumably they have Python 3 installed for a reason, installing Xcode can install Python 3 so uninstalling may be impractical
- Python 2 is EOL and out of support so some path that moves away from it is preferred
- Apple have stated future versions of OSX will not have Python 2
Might still be a good workaround for now.
Ok I'm in business!
So I think yes that's crux of it. It seems, at least on 10.13.6 anyway, macOS has python2.7 installed under python, but thats it.
Brew has installed other python versions on my system at some stage, but 'which python2' gives me nothing so its going nowhere.
So it was as easy as installing python2 via brew, although I had to do it from a commit as python@2 didn't want to work.
For anyone following, this commit did the trick
brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/86a44a0a552c673a05f11018459c9f5faae3becc/Formula/
[email protected]Thanks for the help!
Cool! Apple really doesn't seem to like following standards!