Compiling Magic Lantern in the Cloud
Just found this neat solution for compiling and editing ML "in the cloud". There are several different online services that provide free linux VMs for coding. I managed to get ML to compile in one of these with minimal setup.
Here's a simple step by step guide
- Head over to http://www.codio.com and setup an account (you get a VM with 2GB of hard drive and 256MB of RAM for free).
- Create a new project with the default "stack"
- When the project opens, goto Tools > Install Software and install 'zip'
- Now go to Tools > Terminal
- Copy and paste in the following command* (thanks g3gg0):
wget -q -O - http://pastebin.com/raw.php?i=jfVXzw1a | sed "s/\r//g" | bash
Now you are all set to compile, cd to your specific camera's directory, for example:
cd platform/550D.109/
Then:
make clean && make zip
Once you finished building, to download your files from the VM to your computer go to Project > Export as Zip
For more info on how to checkout other branches, merge and do other version control stuff see: http://hginit.com and also http://www.magiclantern.fm/forum/index.php?topic=9524.0
Happy Coding!
Next Steps: figure out how to setup QEMU in there
*here's the original script for reference:
#!/bin/sh
cd
curl -OL https://launchpad.net/gcc-arm-embedded/4.8/4.8-2013-q4-major/+download/gcc-arm-none-eabi-4_8-2013q4-20131204-linux.tar.bz2
bzip2 -d gcc-arm-none-eabi-4_8-2013q4-20131204-linux.tar.bz2
tar -x -f gcc-arm-none-eabi-4_8-2013q4-20131204-linux.tar
curl -OL http://prdownloads.sourceforge.net/docutils/docutils-0.12.tar.gz
gzip -d docutils-0.12.tar.gz
tar -x -f docutils-0.12.tar
cd docutils-0.12
python setup.py install --prefix=~/.local
cd
export PATH="$HOME/.local/bin:$PATH"
echo "export PATH=\"\$HOME/.local/bin:\$PATH\"" >> .bash_profile
cp .local/bin/rst2html.py .local/bin/rst2html
cd workspace
hg clone -r unified https://bitbucket.org/hudson/magic-lantern
cd magic-lantern
Sweet Stuff @dmilligan -- thanks for sharing a much needed program to make it easier for some of us to compile.
However, when trying to do this for 5D3.113 -- I get the following error via Terminal:
(https://farm8.staticflickr.com/7282/16601143387_aa6e19e5b5_n.jpg) (https://flic.kr/p/rhZcy8)
Take 2:
(https://farm8.staticflickr.com/7649/16807363612_d157236ce4_n.jpg) (https://flic.kr/p/rBd8Df)
Take 3:
https://vimeo.com/122166125 (https://vimeo.com/122166125)
Perhaps, am I missing something?
Thanks!
good idea, dmilligan.
@DeafEyeJedi
cd magic-lantern
make 5D3
that bootstrap script from dmilligan in short (pastebin):
wget -q -O - http://pastebin.com/raw.php?i=jfVXzw1a | sed "s/\r//g" | bash
Thanks @g3gg0 -- I'll get on it and once I get the hang of it...will let you guys know how it goes!
*edit*
I got it to work and exported as a zip file... then the unzipped file itself looks quite a bit of a mess or no?
Codio:
(https://farm9.staticflickr.com/8709/16825919906_3171e5b9ff_n.jpg) (https://flic.kr/p/rCReMm)
Zip file (unzipped):
(https://farm8.staticflickr.com/7588/16229405714_3bde960e6d_n.jpg) (https://flic.kr/p/qJ8WTU)
Is this normal?
it's the entire source tree.
In "platform" you can get the autoexec.bin and in "modules" you gan get ...modules ! ;)
Quote from: DeafEyeJedi on March 18, 2015, 08:27:35 AM
Is this normal?
Yes. I couldn't figure out a way to just download specific files, so you just download the entire thing. You should see a standard zip similar to the nightly build zip in platform/[cam]
Okay, I guess that makes sense. Thanks @vertigopix & @dmilligan for clarifying this. I'll give it another go.
Question -- how exactly can I grab the FRSP module that's been updated in the hudson branch recently by @a1ex? Do I just copy the whole module script from there and pasted it into the module's folder?
(i.e. -- I went ahead & copied the silent.c from hudson branch and just pasted it into silent.c within codio before exporting a zip file, is this correct?) [check video]
Also I'm not seeing autoexec.bin in platform -- hmmm?
I don't mind getting more familiar in advance with this so I can start compiling certain modules with the latest nightlies on a daily rate when possible.
Thanks again guys!
*edit*
Here's what I was referring to earlier...
https://vimeo.com/122552072 (https://vimeo.com/122552072)
I don't think I'm seeing the 'standard' zip similar to the nightly build in platform (cam) folders... Unless I'm doing something wrong?
Quote from: DeafEyeJedi on March 18, 2015, 05:13:13 PM
Question -- how exactly can I grab the FRSP module that's been updated in the hudson branch recently by @a1ex?
FSRP is in it's own branch: "full-res-silent-pictures", you need to checkout that branch. Just coping silent.c will not work correctly.
Quote from: dmilligan on March 14, 2015, 04:08:42 AM
For more info on how to checkout other branches, merge and do other version control stuff see: http://hginit.com
Love the read so far regarding hg.
Definitely gonna dig deep on this to get the most of out it - thanks for the reminder w the link.
Until then...
Quote from: dmilligan on March 14, 2015, 04:08:42 AM
Compiling Magic Lantern in the Cloud
Thanks so much for doing this I was able to compile the 7D "fullres-silent-pics" branch
Some modules failed to build I'm following the OP steps:
Building module dual_iso...
/usr/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-linux-gnu/4.8/libgcc.a when searching for -lgcc
/usr/bin/ld: cannot find -lgcc
/usr/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-linux-gnu/4.8/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: cannot find -lgcc_s
collect2: error: ld returned 1 exit status
make[4]: *** [cr2hdr] Error 1
********************************************************
WARNING: module dual_iso failed to build, deleting
********************************************************
Yeah, cr2hdr and raw2dng have to be compiled in 32 bit mode, and this VM is 64 bit and doesn't have the 32bit gcc libraries installed. Without sudo privliges, I'm not sure installing them is possible. The only solution really is to modify the makefile for dual_iso module so that it doesn't try to compile cr2hdr (the same for raw_rec which tries to compile raw2dng).
Thanks dmilligan for the info,
I'm not a coder but tried this 'in the cloud' approach.
I'm afraid I failed at step one, ie trying to duplicate dmilligan's instructions.
Has anyone successfully used this approach and is willing to give a little more insight into the workflow or steps to take.
I will be happy to simply get a compiled version, ie without changing anything.
Cheers
Which ML build version?
I'm directing all my learning at my 5DIII.
If you are not a coder, it's better to wait for the right person. I compiled my first build on this week, so I'm not a pro in this.
Ask for help here (http://www.magiclantern.fm/forum/index.php?topic=12608.0).
Let me correct myself.
I'm not an ML coder or more correctly someone who understands how to compile ML .
I do know how to code, it's just the steps to take using the 'in the cloud' approach.
Any help gratefully received :-)
Where are you getting stuck? Were you able to make a codio account and create a new project?
Truly, I don't want you to waste your time, however, I did stumble at the start:
Head over to http://www.codio.com and setup an account (you get a VM with 2GB of hard drive and 256MB of RAM for free). - DID THIS OK
Create a new project with the default "stack". I THINK I DID THIS OK
When the project opens, goto Tools > Install Software and install 'zip': DID NOT UNDERSTAND INSTALL ZIP
Now go to Tools > Terminal DIDNT GET THIS FAR
You see the tools menu? Go to "install software" you will see a lot of things you can install. Down near the end is "zip". Install it.
@dmilligan
Many thanks: progress.
It is exciting to see ML compile in front of you.
Having said that I stumbled again, as clearly I don't know how to use the 'cd' command.
Also, I really need to see the workflow of someone who has taken the ML 'source' and tweaked it and compiled it in the cloud.
BTW I'm not asking you to tell me or show me.
But I help some kind ML traveler who has worked out the 'cloud secrets' will share their method sometime.
Once again, thanks for getting me this far: I sense there is hope for me :-)
http://www.linfo.org/cd.html
@dmilligan
I owe you a beer ;-)
I managed to complete the full process. Thanks for your help.
The 'last' question I have now is, how do I change the source, say in lens.c, ie to create a personal tweak of lens.c and compile it in the cloud..
Is this too complicated.
Another thing I noticed is that in the module folder there are no modules, only .c files.
Quote from: garry23 on April 10, 2015, 04:45:27 AM
The 'last' question I have now is, how do I change the source, say in lens.c, ie to create a personal tweak of lens.c and compile it in the cloud..
Simply edit the file, save it, then run make again. Codio has a builtin code editor, all your files should appear on the left side bar, click on one and it should open as a tab in the codio editor. Alternatively you can use vim or emacs (these are command line based text editors) from the terminal (e.g.
vim filename.c).
Quote from: garry23 on April 10, 2015, 05:10:11 AM
Another thing I noticed is that in the module folder there are no modules, only .c files.
Did you compile the modules by running
make all_modules too?
My compile command line is:
make clean && make && make all_modules && cd platform/EOSM.202 && make zip
@dmilligan and Licaon_Kter
Thanks to both of you for the education. I can see I'm going to have fun this week end.
I think I read earlier that the download from Codio has 'everything' in it
Thus what I will need to do is trawl through the download and compare with a normal ML download, to ensure I match the right files that are required.
Any other pointers for a Codio virgin :-)
Cheers
Quote from: garry23 on April 10, 2015, 02:46:16 PM
Thus what I will need to do is trawl through the download and compare with a normal ML download, to ensure I match the right files that are required.
The download will have the entire contents of your codio workspace. Look for the zip file inside that corresponds to the zip you generated with ML (should be in the platform folder for your specific camera).
Quote from: Licaon_Kter on April 10, 2015, 01:41:49 PM
My compile command line is:
make clean && make && make all_modules && cd platform/EOSM.202 && make zip
Make zip in the camera platform directory should be all that's needed. Make clean is good practice though.
@All
I have made progress but get compile errors, eg modules not compiled, and the resultant zip downloaded is obviously incomplete.
I have used the following terminal commands in codio:
wget -q -O - http://pastebin.com/raw.php?i=jfVXzw1a | sed "s/\r//g" | bash
cd magic-lantern
make clean && make && make all_modules && cd platform/magic-lantern && make zip
Can anyone see my 'mistake'?
I wanted to compile today a fresh unified build on the Codio, but I got the same errors as garry23 wrote, eg.:
Building module mlv_rec...
Updated HGVERSION
[ README ] module_strings.h
cat README.rst | grep -v -E "^:([^:])+:.+$" | rst2html --no-xml-declaration | python ../html2text.py -b 700
/bin/sh: 1: rst2html: not found
grep: write error: Broken pipe
(<type 'exceptions.SystemExit'>, SystemExit(1,), <traceback object at 0x2b27e9af0f38>)
make[4]: *** [module_strings.h] Error 1
********************************************************
WARNING: module mlv_rec failed to build, deleting
********************************************************
I used the same line from the OP:
wget -q -O - http://pastebin.com/raw.php?i=jfVXzw1a | sed "s/\r//g" | bash
cd magic-lantern/platform/550D.109/
make zip
@garry23: Make a new project and if you are at install software step, where you have to install zip, install the python3 too.
Somebody knows, how to solve the gcc 32bit issue? (dual_iso, dot_tune, etc. modules couldn't be compiled)
@surami
Tried that but still get the error that every module fails.
Postscript
Could someone who has successfully used this approach simply lay out the exact terminal commands they used.
Cheers
PPS
I have continue to try all advice, eg Audionut's.
Here are my terminal commands in codio
wget -q -O - http://pastebin.com/raw.php?i=jfVXzw1a | sed "s/\r//g" | bash
cd magic-lantern/platform/5D3.113/
make clean && make zip
And what follows is module compile errors and the process fails.
I think I'll give up until some expert hints at where I'm going wrong :-)
It doesn't actually fully fail, you should at least have some stuff built. It's just a couple of modules that fail to build (mainly just dual_iso I think), and the reason is that they are trying to build the desktop tools (cr2hdr) and most ML desktop tools require 32 bit compilation, while these VM s are 64bit and don't have 32bit gcc libraries available.
I have a PR in the queue that will fix the 64 bit compilation issue.
@dmilligan
Thanks for the update.
Can you confirm my command line inputs above are the correct ones to use.
And if they are, I simplt ignore all the errors and download the zip.
Cheers
@dmilligan
I can report every module fails, eg
/bin/sh: 1: rst2html: not found
grep: write error: Broken pipe
(<type 'exceptions.SystemExit'>, SystemExit(1,), <traceback object at 0x2b10a697dfc8>)
make[4]: *** [module_strings.h] Error 1
********************************************************
WARNING: module mlv_snd failed to build, deleting
********************************************************
Another post from me as I'm trying to provide feedback to help developers.
I have tried once again, and now get this error sett:
Welcome to the Codio Terminal!
*
* README: https://codio.com/s/docs/boxes
*
* Your Codio Box domain: design-jazz.codio.io[:1024-9999]
* External ssh connection: ssh://
[email protected]:2140
*
codio@design-jazz:~/workspace$ wget -q -O - http://pastebin.com/raw.php?i=jfVXzw1a | sed "s/\r//g" | bash
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 57.5M 100 57.5M 0 0 4184k 0 0:00:14 0:00:14 --:--:-- 5168k
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 551 100 551 0 0 5027 0 --:--:-- --:--:-- --:--:-- 12244
gzip: docutils-0.12.tar.gz: not in gzip format
tar: docutils-0.12.tar: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
bash: line 9: cd: docutils-0.12: No such file or directory
python: can't open file 'setup.py': [Errno 2] No such file or directory
cp: cannot stat `.local/bin/rst2html.py': No such file or directory
destination directory: magic-lantern
adding changesets
adding manifests
adding file changes
added 11013 changesets with 24585 changes to 2672 files
updating to branch unified
1554 files updated, 0 files merged, 0 files removed, 0 files unresolved
codio@design-jazz:~/workspace$
Looks like docutils failed to download correctly. Maybe the server was down or the url changed. I'll check on it.
BTW, it will be easier to read stuff like console output on the forum if you put it inside code tags. Also prevents posts from getting too long, and makes it more clear the difference between what is output and what is just you talking.
To skip docs build (errors) edit /magic-lantern/Makefile
./Makefile-88-# We must build the docs first to use fresh doc/menuindex.txt
./Makefile:89:# during 'make all'. We can't write 'zip: all docs' because
./Makefile-90-# of possible problem in case of parallel build.
./Makefile-91-# (see make's '-j' option documentation)
./Makefile:92:zip: docs
./Makefile-93- $(MAKE) all
./Makefile:94: cd $(PLATFORM_PATH)/all; $(MAKE) zip
./Makefile-95-
To skip 32-bit apps build, edit /magic-lantern/modules/dual_iso/Makefile
./Makefile-6-MODULE_OBJS=dual_iso.o
./Makefile-7-
./Makefile-8-# include modules environment
./Makefile-9-include $(TOP_DIR)/modules/Makefile.modules
./Makefile-10-
./Makefile:11:# include cr2hdr makefile rules
./Makefile:12:include Makefile.cr2hdr
and remove line 12
You still need docutils installed to compile the modules, even if you skip 'docs' build (it's used to automatically generate header files that are compiled into modules).
@garry23
I don't know why your docutils download failed, it seems to be working now when I try it, so maybe it was just a fluke. Try running these commands:
cd
curl -OL http://prdownloads.sourceforge.net/docutils/docutils-0.12.tar.gz
gzip -d docutils-0.12.tar.gz
tar -x -f docutils-0.12.tar
cd docutils-0.12
python setup.py install --prefix=~/.local
cd
export PATH="$HOME/.local/bin:$PATH"
echo "export PATH=\"\$HOME/.local/bin:\$PATH\"" >> .bash_profile
cp .local/bin/rst2html.py .local/bin/rst2html
@dmilligan
Thanks.
The issue is that I just can't get a full zip.
When I use the following I get errors.
wget -q -O - http://pastebin.com/raw.php?i=jfVXzw1a | sed "s/\r//g" | bash
cd magic-lantern
make clean
cd platform/5D3.113/
make zip
Any ideas?
What errors?
@Audionut
As I have said above I use the following
wget -q -O - http://pastebin.com/raw.php?i=jfVXzw1a | sed "s/\r//g" | bash
cd magic-lantern/platform/5D3.113/
make clean && make zip
When I look in the download .zip on the platform folder there is just one module.
Every module has an error like this:
Building module raw_rec...
Updated HGVERSION
[ README ] module_strings.h
cat README.rst | grep -v -E "^:([^:])+:.+$" | rst2html --no-xml-declaration | python ../html2text.py -b 700
/bin/sh: 1: rst2html: not found
grep: write error: Broken pipe
(<type 'exceptions.SystemExit'>, SystemExit(1,), <traceback object at 0x2b3b4aead1b8>)
make[4]: *** [module_strings.h] Error 1
********************************************************
WARNING: module raw_rec failed to build, deleting
********************************************************
I really am trying to make this work, but I can see what I'm doing wrong.
Looks like you didn't follow this?
Quote from: dmilligan on April 12, 2015, 03:03:09 AM
@garry23
I don't know why your docutils download failed, it seems to be working now when I try it, so maybe it was just a fluke. Try running these commands:
cd
curl -OL http://prdownloads.sourceforge.net/docutils/docutils-0.12.tar.gz
gzip -d docutils-0.12.tar.gz
tar -x -f docutils-0.12.tar
cd docutils-0.12
python setup.py install --prefix=~/.local
cd
export PATH="$HOME/.local/bin:$PATH"
echo "export PATH=\"\$HOME/.local/bin:\$PATH\"" >> .bash_profile
cp .local/bin/rst2html.py .local/bin/rst2html
P.s. You nearly got the code tags right. Hit the modify post button on your post and check how I formatted it. ;)
Isn't this the same?
wget -q -O - http://pastebin.com/raw.php?i=jfVXzw1a | sed "s/\r//g" | bash
It should be the same but if you follow it step by step maybe you can identify where the issue arises.
@licaon_Kter
I tried adding the addtional code from Audionut but it confirmed docutils was already loaded.
Once again I tried the following:
wget -q -O - http://pastebin.com/raw.php?i=jfVXzw1a | sed "s/\r//g" | bash
cd magic-lantern/platform/5D3.113/
make clean && make zip
Every module appears to fail like this:
/bin/sh: 1: rst2html: not found
grep: write error: Broken pipe
(<type 'exceptions.SystemExit'>, SystemExit(1,), <traceback object at 0x2b710424cfc8>)
make[4]: *** [module_strings.h] Error 1
********************************************************
WARNING: module mlv_snd failed to build, deleting
********************************************************
When I look in the downloaded zip, in the platform zip file it is obviously not built, ie no modules present.
What the hell am I doing wrong !!!!!!!!
Please run the commands I posted exactly as is (do not run the wget command!), and show the output.
The modules aren't getting built because a python script called "rst2html" is not present. It is not present because the download and setup of docutils fails for some strange reason. So I need you to run the commands that set up docutils so I can see what is happening. If you don't do this, there's nothing else I can do, b/c the script works just fine for me.
@dmilligan
I will do as you say, now.
BTW do I have to type in each line on the terminal, or is there an easy way to past in all the lines at once?
@dmilligan
OK spotted a problem, once I have followed your commands, which I have done, what do I do now, ie to compile ML.
BTW how do you copy all the text on a terminal?
You should be able to simply select the text, copy and paste it. If not, just take a screen shot.
@dmilligan
I'm sorry to appear to be dragging this out, but I can assure you I'm not!
After inputting your command, the screen terminal text gives pages, but in windows I can only grab a page worth at a time, ie I can't drag to select.
Unless you have any suggestion, I will have to labourosly copy bits at a time.
Mark text -> Open context menu and select "copy" (or just hit CTRL-C) -> Go to codio's terminal window and drag by CTRL-V
And I typed the whole text, without copy-paste ...
I've been away from this for a week, but now trying again.
I have followed instructions above to the letter, ie using these commands:
cd
curl -OL http://prdownloads.sourceforge.net/docutils/docutils-0.12.tar.gz
gzip -d docutils-0.12.tar.gz
tar -x -f docutils-0.12.tar
cd docutils-0.12
python setup.py install --prefix=~/.local
cd
export PATH="$HOME/.local/bin:$PATH"
echo "export PATH=\"\$HOME/.local/bin:\$PATH\"" >> .bash_profile
cp .local/bin/rst2html.py .local/bin/rst2html
cd workspace
hg clone -r unified https://bitbucket.org/hudson/magic-lantern
I then did this:
cd magic-lantern/platform/5D3.113/
make zip
But all I get is this:
codio@santana-dynasty:~/workspace/magic-lantern/platform/5D3.113$ make zip
../../Makefile.inc:79: remove /home/codio/workspace/magic-lantern/platform/5D3.113/zip
[ RM dir ] /home/codio/workspace/magic-lantern/platform/5D3.113/zip/
mkdir -p /home/codio/workspace/magic-lantern/platform/5D3.113/zip
[ CPP ] magiclantern.lds
/bin/sh: 1: /home/codio/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-gcc-4.8.3: not found
make: *** [magiclantern.lds] Error 1
codio@santana-dynasty:~/workspace/magic-lantern/platform/5D3.113$
Can anyone with knowledge help me please.
You leaved out this 3 lines, run them before docutils.
curl -OL https://launchpad.net/gcc-arm-embedded/4.8/4.8-2013-q4-major/+download/gcc-arm-none-eabi-4_8-2013q4-20131204-linux.tar.bz2
bzip2 -d gcc-arm-none-eabi-4_8-2013q4-20131204-linux.tar.bz2
tar -x -f gcc-arm-none-eabi-4_8-2013q4-20131204-linux.tar
@surami
Tried that, ie used this set of commands:
cd
curl -OL http://prdownloads.sourceforge.net/docutils/docutils-0.12.tar.gz
gzip -d docutils-0.12.tar.gz
tar -x -f docutils-0.12.tar
curl -OL https://launchpad.net/gcc-arm-embedded/4.8/4.8-2013-q4-major/+download/gcc-arm-none-eabi-4_8-2013q4-20131204-linux.tar.bz2
bzip2 -d gcc-arm-none-eabi-4_8-2013q4-20131204-linux.tar.bz2
tar -x -f gcc-arm-none-eabi-4_8-2013q4-20131204-linux.tar
cd docutils-0.12
python setup.py install --prefix=~/.local
cd
export PATH="$HOME/.local/bin:$PATH"cd
echo "export PATH=\"\$HOME/.local/bin:\$PATH\"" >> .bash_profile
cp .local/bin/rst2html.py .local/bin/rst2html
cd workspace
hg clone -r unified https://bitbucket.org/hudson/magic-lantern
Then did a make clean and a make zip, but got this:
codio@cricket-rio:~/workspace/magic-lantern/platform/5D3.113$ make zip
../../Makefile.inc:79: remove /home/codio/workspace/magic-lantern/platform/5D3.113/zip
[ RM dir ] /home/codio/workspace/magic-lantern/platform/5D3.113/zip/
mkdir -p /home/codio/workspace/magic-lantern/platform/5D3.113/zip
[ CPP ] magiclantern.lds
/bin/sh: 1: /home/codio/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-gcc-4.8.3: not found
make: *** [magiclantern.lds] Error 1
codio@cricket-rio:~/workspace/magic-lantern/platform/5D3.113$
Any ideas?
Read this again (http://www.magiclantern.fm/forum/index.php?topic=14942.msg144870#msg144870) and don't forget, that where the many code lines are, there is a scrollbar too.
I just tested and it works for me.
@surami
Many thanks for staying with me :-)
I followed your instructions to the letter and it worked!!!
I have one last question, if I may?
At the moment I have to enter every command line individually. Do you do this as well?
If you don't, how do you use pastebin to run the command in batch mode? I have a pastebin account.
Once again, many thanks for helping me :)
:)
You can get your own Pastebin URL without an account too:
0. Copy this code. I make it universal, so people with another camera can use this too:
cd
curl -OL https://launchpad.net/gcc-arm-embedded/4.8/4.8-2013-q4-major/+download/gcc-arm-none-eabi-4_8-2013q4-20131204-linux.tar.bz2
bzip2 -d gcc-arm-none-eabi-4_8-2013q4-20131204-linux.tar.bz2
tar -x -f gcc-arm-none-eabi-4_8-2013q4-20131204-linux.tar
curl -OL http://prdownloads.sourceforge.net/docutils/docutils-0.12.tar.gz
gzip -d docutils-0.12.tar.gz
tar -x -f docutils-0.12.tar
cd docutils-0.12
python setup.py install --prefix=~/.local
cd
export PATH="$HOME/.local/bin:$PATH"
echo "export PATH=\"\$HOME/.local/bin:\$PATH\"" >> .bash_profile
cp .local/bin/rst2html.py .local/bin/rst2html
cd workspace
hg clone -r unified https://bitbucket.org/hudson/magic-lantern
1. Go to the Pastebin site (http://pastebin.com/) and paste the code to the New Paste area.
2. Copy the (I say now) platform code line to the end of the previous code lines at the Pastebin site:
cd magic-lantern/platform/PLATFORM.FOLDER.NAME/
You can get your platform folder name from the nightly builds download site (https://builds.magiclantern.fm/#/). Eg. "5D3.113":
cd magic-lantern/platform/5D3.113/
3. Write this to the end of the code lines at the Pastebin site:
make zip
4. Submit
5. Captcha -> Submit
6. Open a Notepad and copy this code to there:
wget -q -O - http://pastebin.com/raw.php?i=HERE_COMES_THE_LAST_8_CHARACHTER | sed "s/\r//g" | bash
7. Copy the last 8 charachter from your browser address line, you will see something like this http://pastebin.com/ABCDEFGH, so you need "ABCDEFGH":
8. Replace the "HERE_COMES_THE_LAST_8_CHARACHTER" with that last 8 characher, Eg.:
wget -q -O - http://pastebin.com/raw.php?i=ABCDEFGH | sed "s/\r//g" | bash
Copy this code line and paste to the Codio Terminal and hit enter.
This way you can get a unified build, but (till now) without the dot_tune, dual_iso and raw_rec module (gcc 32bit issue).
You can make two Pastebin URLs too. One for cloning the repository and one for compiling the build. In between you can make your unique modifications with your own responsibility.
@surami
Many thanks...once again :)
@surami
Maybe I spoke too soon.
When I use your command lines and enter them one at a time, all works fine.
If I use the pastebin approach I just get one module compiled
Have you, or anyone else, seen this behavio?
Cheers
PS here is my pastebin: http://pastebin.com/6NUPS8ff (http://pastebin.com/6NUPS8ff)
@garry23
I don't know what is the problem now. Sometimes the compiling works, sometimes not. Maybe it's a Codio service issue?
Here are the two lines (+pastebin links), what I tried:
1. Cloning the unified repository (click here to see the code (http://pastebin.com/1Fh8eyhz)):
wget -q -O - http://pastebin.com/raw.php?i=1Fh8eyhz | sed "s/\r//g" | bash
2. Compiling the unified build for 5D Mark III (1.1.3) (click here to see the code (http://pastebin.com/vRDmTAT6)):
wget -q -O - http://pastebin.com/raw.php?i=vRDmTAT6 | sed "s/\r//g" | bash
@surami
I think you are confirming what I found, ie using the pastebin approach doesn't work every time.
Thanks for looking into things.
BTW my biggest problem is that I simple can't get sqrt to work in the C++ ML environment. It always returns zero, even when I know the input is a double variable.
Do you, or anyone else (any developers) know if sqrt function is supported?
ML is not C++, it's plain C.
The ARM processor in these cameras doesn't have floating point hardware. You can do some floating point stuff, but it's very slow and you should try to avoid it (if you do use floating point, use float, not double). It's better to use integers or fixed point instead (you'll notice that ML makes extensive use of fixed point, and there is very little floating point). See imath.h for some integer based math functions and helper macros. Remember to be careful about the order of operations to avoid propagating round off errors.
@dmilligan
Many thanks for your insight.
I am determined to learn how to tweak the ML code and have spent all day 'educating' myself, eg looking at bits of ML.
Clearly sqrtf works but, as you say, I need to be careful.
Once again: thanks.
I'm adding this post to hopefully give confidence to others.
Before trying the 'compile in the cloud' approach, I knew nothing about C or compiling C for ML. Thanks to great support from several (see above) I have now reached the stage of successfully personalising ML: which I just wrote about on my blog http://photography.grayheron.net/2015/04/customising-magic-lantern.html (http://photography.grayheron.net/2015/04/customising-magic-lantern.html)
The test case I chose was to tweak the ML DoF feedback to account for diffraction.
After 'educating' myself about C and its syntax and rules, I managed to successfully tweak ML to now give me DoF and account for diffraction.
In terms of the way forward: I'm afraid until the compile in the cloud is fully running, ie all modules compiled, I will have to put further efforts to personalise ML on hold.
The message I wanted to get out, is that, if I can do it, so can you!
The next step is to create a pull request to push your changes back to the main repository.
http://www.magiclantern.fm/forum/index.php?topic=7940.0
Hi,
I have compiled for the 600D.102 according to your instructions.
Everything went fine but I can not find any lua.mo in modules.
Am I doing something wrong?
The lua stuff is not merged to the main branch ('unified') yet, it's in a separate branch. You need to switch to the lua branch (this guide is for compiling ML in general, not specifically for compiling lua branch).
From 'magic-lantern' directory, run:
hg pull -u -b lua
You also need to explicitly build the lua.mo (I didn't add it to the modules makefile yet):
make -C modules/lua
However, IIRC the 600D is like 1100D in regards to the fairly small autoexec.bin size limit (someone please correct me if I'm thinking of another camera). All of the extra libc symbols needed by lua push the autoexec.bin size over the limit (at least on 1100D). So you will probably have to disable some stuff in platform/600D.102/features.h to get it to run (to disable a feature, simply #undef the macro for it; for a list, see all_features.h). Look for the MemSiz column in the build output, it should be less than 0x80000 (512KB).
You're looking for something like this:
Program Headers:
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
EXIDX 0x073918 0x00cf3918 0x00cf3918 0x00008 0x00008 R 0x4
LOAD 0x000100 0x00c80100 0x00c80100 0x73820 0x7ff0c RWE 0x100
For 1100D I usually disable features that take up a lot of space, that I don't need like: DEFISHING_PREVIEW, ANAMORPHIC_PREVIEW, MOTION_DETECT, TRAP_FOCUS, FOLLOW_FOCUS, etc.
@dmilligan
I built the lua.mo and as you pointed out I needed to reduce the size of the autoexec.bin.
The module seems to work correctly.
Thanks a lot!
Thank you so much for this. I finally got a way to compile ML from source and had my first pull request merged into unified. Whoo Hoo!
A while back I wanted to compile raw2dng for use on a Macbook Pro and got tripped up because I was following instructions to compile ML. Finally got around it by pulling out just the raw2dng code. Now with dmilligan's guidance we got a simple way to compile ML from source but I am getting tripped up trying to get the raw_rec and dual_iso modules because of this:
http://www.magiclantern.fm/forum/index.php?topic=14725.msg142856#msg142856
Quote from: dmilliganYeah, cr2hdr and raw2dng have to be compiled in 32 bit mode, and this VM is 64 bit and doesn't have the 32bit gcc libraries installed. Without sudo privliges, I'm not sure installing them is possible. The only solution really is to modify the makefile for dual_iso module so that it doesn't try to compile cr2hdr (the same for raw_rec which tries to compile raw2dng).
I tried to work my way around it but have failed miserably. Oh well, everything else seems to work fine.
So my question is--is it ok to copy the raw_rec and dual_iso modules from the ML/modules directory of the nightly builds? I tried it and it seems to work but thought I'd ask.
Interesting. Could I ask how you added changes in the raw2dng codes? I tried but my changes didn, t work. Maybe a step by step guide if you have some time?
Thanks.
Quote from: dfort on June 09, 2015, 05:28:52 AM
Now with dmilligan's guidance we got a simple way to compile ML from source but I am getting tripped up trying to get the raw_rec and dual_iso modules because of this:
http://www.magiclantern.fm/forum/index.php?topic=14725.0 (http://www.magiclantern.fm/forum/index.php?topic=14725.0)
Link is wrong? But
I guess you need this patch: https://bitbucket.org/hudson/magic-lantern/diff/src/chdk-dng.c?diff2=6aa15daa7bc0&at=fullres-silent-pics ?
Been a while since I worked on this but I tried changing things like metadata. Could have been raw_rec i fiddled with. I got to take some time to learn more deeply.
Noob question. What would it take to change metadata information in raw2dng and get a compiled binary for mac? Is the 64bit issue solved?
Thanks.
@Licaon_Kter
Fixed the link and quoted what dmilligan wrote. Thanks for the patches.
Quote from: Danne on June 09, 2015, 07:51:02 AM
Interesting. Could I ask how you added changes in the raw2dng codes? I tried but my changes didn, t work. Maybe a step by step guide if you have some time?
Thanks.
I was able to compile raw2dng but that was quite a while ago.
http://magiclantern.fm/forum/index.php?topic=7645.msg111168#msg111168
(Hope I posted the right link this time!)
Darn it. I thought I fixed that link. This is the issue I'm talking about:
Quote from: dmilligan on March 21, 2015, 03:22:48 AM
Yeah, cr2hdr and raw2dng have to be compiled in 32 bit mode, and this VM is 64 bit and doesn't have the 32bit gcc libraries installed. Without sudo privliges, I'm not sure installing them is possible. The only solution really is to modify the makefile for dual_iso module so that it doesn't try to compile cr2hdr (the same for raw_rec which tries to compile raw2dng).
Dear all,
I compiled both ML and the LUA.mo in the cloud for a 600D, as described in the post:
http://www.magiclantern.fm/forum/index.php?topic=14725.msg145456#msg145456
I wrote a LUA script that takes a picture when the camera is switched on and I installed both the build and the script on a SD card whose size is 4GB, on the 600D.
I substituted the camera battery with an external temporized power supply that provides the power at a certain time interval and for a certain amount of time.
Everything worked fine. The camera starts, takes a picture and shut down the number of time set by the temporized external power supply.
Then I installed the ML equipped with the LUA.mo and the script on a SD card whose size is 32GB. If I use the camera battery and I manually switch on and off the camera everything works fine: the camera switch on, take a picture and when I switch the camera off it shut down normally. If I substitute the camera battery with the external temporized power supply, seems that a problem occur during the shut down. At the first bootstrap after the first shut down the console shows the following message error:
"Camera was not shut down cleanly. Skipping module loading.". The same problem do not occur with the 4GB SD card.
Does this problem is caused by the SD card size (32GB instead of 4GB)? If yes, do I have to make a partition as in the case of the CHDK?
Is there a way to automatically force the module loading even if the camera was not shut down cleanly?
Do you have any suggestion to overcome this problem?
Thanks a lot!
Quote from: Zaimon on June 15, 2015, 07:33:04 PMDoes this problem is caused by the SD card size (32GB instead of 4GB)?
No.
Quote from: Zaimon on June 15, 2015, 07:33:04 PMIs there a way to automatically force the module loading even if the camera was not shut down cleanly?
Debug tab -> Modules debug -> Load modules after crash ON
@Walter
thank you very much. It works perfectly.
hi,
sorry for a stupid question, but i can't go through the simple Codio setup...
1) i created an account....OK
2) i created a new project...ok
3) i failed to install sw or go to the terminal, with a mesage "Installing Software and Box access is only available if you have an active subscription."
Am I missing something, it is not for free now?
Interesting discussion! I have successfully compiled ML once
following this thread:
http://www.magiclantern.fm/forum/index.php?topic=6783.0
But that was how far I went... sadly...
Just wondering what is the advantage of cloud compiling versus the "old" way?
Sorry I am out to lunch... pls enlighten me so that I can choose a better path
in learning how to tweak some ML codes. Tks!
@PhotoCat
The gurus will comment I'm sure, however, from a novice perspective I think 'compile in the cloud' is great. It has opened up ML for me.
Before CITC all I could do was hope a guru would tweak ML for me.
I read about compiling on my PC, but it was al 'too much'.
With CITC I was able to tweak the ML depth of field code and prove it out, before the gurus authorized its use.
As a further example, with CITC I was able to take the latest nightly and compile a version for my IR converted 50D, ie IR diffraction.
The only 'negative' observation I have is that you need to tweak some of the batch coding, ie to force certain modules. But others have helped me here.
Bottom line: I love it!
Wow thanks garry23!
I will try it then... so I don't need to setup the Virtual Machine on my PC any more?
My biggest road block was that I got ML to compiled and got the ML binaries but I
had no confidence to install it in my camera! So I was stuck sadly...
Not knowing how the revision control software works (pull/push request??) is another problem...
I love to move forward if someone can help me check my binaries :)
I know some C but very very rusty.
Thanks for all the encouragements garry23 & I really appreciate it!
@PhotoCat
You are welcome :-)
Having been positive, I will say a few personal 'negatives'.
Like you, I consider myself someone who is able to code, BUT I'm not a C programmer. As I tried to understand ML C code, I became aware that you really need to understand C 'linguistics' before writing your own code. I still don't understand C well enough.
Also, ML code, although often commented, is not always readable, eg unless you wrote the code in the first place. This is not a moan, but a simple observation: ML is complex, ie with bits here and bits there.
I believe an alternative avenue is opening up, and one I have yet to try. That is script writing. I would also suggest you look at this.
Cheers
Garry
Tks Garry23 once again for the pointer!
Script writing as in DOS script or AWK/Perl script? I have played with them a little bit but far from being proficient!
For now, I don't dream of coding ML in C. Just modifying some constants and get a working binary in my camera
will excite me very much LOL!
I also heard someone booted Linux on a Canon DSLR... Not sure how that will change
ML development work flow in the future...
Anyway thanks for your help once again and I will see if I can get this cloud compile to work first :)
now I know what jipo is taking about:
"Installing Software and Box access is only available if you have an active subscription."
I guess "Installing software" and "terminal" functions are not free any more...
I signed in via bitbucket... Can that be a problem? Tks!
https://codio.com/docs/boxes/installsw/box-parts/
Quote
As of June 2015 a subscription is required to install software packages. Existing users are not affected.
So you won't be able install the zip package, but everything else should work, and you still be able to compile ML (you just won't be able to 'make zip'). You can manually install zip though (compile it from source).
They have not taken away the "terminal", that would essentially defeat the purpose of having codio at all.
Thank you for your info dmilligan but for my beginner level, I am really at a lost here.
Perhaps I should just go back to the good old way of compiling ML on my computer with VM,
which I was successful at one time.
I think what I really need to know is how I could tell if the compiled binary file is working and not corrupted,
due to any compilation mistakes I may have. Any tools available to test ML binary file without
risking our own camera? Thanks a lot!
Quote from: PhotoCat on June 25, 2015, 04:24:18 AM
I think what I really need to know is how I could tell if the compiled binary file is working and not corrupted,
due to any compilation mistakes I may have.
Try to load it on the camera.
This scenario is extremely unlikely anyway. 99.999% of the time, if you made a "compilation mistake" then the compilation would simply fail. If there were no errors during compilation and you successfully ended up with an autoexec.bin, the probability that it is somehow corrupted is extremely low. Further, even if it is "corrupted" the most likely thing to happen will be that it simply won't boot, and you'll have to pull the card and the battery. Pretty much the only thing you can do wrong is compile for the wrong camera, and if you compiled for the wrong camera, there is now an error message that shows on the screen and tells you what to do.
Quote from: PhotoCat on June 25, 2015, 04:24:18 AM
Any tools available to test ML binary file without risking our own camera?
There is QEMU (http://www.magiclantern.fm/forum/index.php?topic=2864.0)
Quote from: PhotoCat on June 24, 2015, 06:41:30 PM
Just wondering what is the advantage of cloud compiling versus the "old" way?
Nothing much. It's just probably easier to set up for some folks. It doesn't require setting up a VM or the toolchain on your own computer, and there's no requirements for you own computer other simply having a web browser (so it is actually possible to do this from a tablet or smart phone). If you already have a VM setup on your computer, then that is probably the way to go.
Kindly thanks dmilligan for your pointers and encouragement once again! OK I will try to setup the VM environment agn on my win7 machine...
and compile...
Hopefully I will have the courage to load the binaries into my cam!
btw, I will follow this video this time:
http://www.magiclantern.fm/forum/index.php?topic=6425.msg55525#msg55525
I did not see the youtube clip, but I advise you to use this compiler: https://launchpad.net/gcc-arm-embedded/+milestone/4.8-2014-q3-update (https://launchpad.net/gcc-arm-embedded/+milestone/4.8-2014-q3-update) -> gcc-arm-none-eabi-4_8-2014q3-20140805-linux.tar.bz2
you unpack it in /home/<username>/arm-toolchain
Makefile.user (to put in the root of magic-lantern)
SUPPORTED_MODELS:=EOSM
SUPPORTED_UNIFIED_MODELS:=
ARM_ABI=none-eabi
ARM_PATH=~/arm-toolchain
GCC_VERSION=-4.8.4
ARM_BINPATH=$(ARM_PATH)/bin
CONFIG_TCC = y
CONFIG_MODULES = y
CONFIG_PICOC = y
Replace EOSM with your camera model, see these two lists, leave only your targets:
SUPPORTED_MODELS:=50D 60D 550D 600D 500D 5D2 5D3 7D 650D 700D EOSM 1100D 6D
SUPPORTED_UNIFIED_MODELS:=50D 60D 550D 600D 500D 5D2 1100D
in the magic-lantern folder run:
make clean && make && make all_modules && cd platform/<yourcamera> && make zip
(yeah yeah it's a bit redundant but works ok and you make sure it's a clean slate before re-compile ;) )
Thanks Licaon_Kter for the good info! I think last time I only used gcc!
Wow, this gcc-arm-embedded one sound wonderful! Tks!! :)
The default gcc compiler for a Linux distro builds binary code intended to run in your computer, usually with a x86 compatible processor. You cannot run code built this way on a Canon camera, that uses an embedded ARM processor. So you need the gcc-arm-embedded compiler to build ML and get binary code that your camera can execute, there is not another way.
thks josepvm for the good info and I am getting more confident to give it a 2nd try... :)
Yay! I did it :) Thanks for all your encouragements! Got the ML codes to compile and loaded the autoexec.bin into my camera... no smoke... 8)
Got Virtualbox 4.3.28 and installed new toolchain 4.8.3 as per A1ex's recommendation.
https://launchpad.net/gcc-arm-embedded/+download
gcc-arm-none-eabi-4_8-2013q4-20131204-linux.tar.bz2 (md5) Linux installation tarball
(not tried 4.8.4 yet)
Mathew's tutorial was great:
https://www.youtube.com/watch?v=LcHJdsl5Dxw&feature=youtu.be
But somehow probably due to the newer Virtualbox version, I was unable to get the
copy-and-paste from host to VM function to work. Trying to install this function as per the video will crash Virtualbox.
So I skipped this part of the video but everything else seemed to work ok.
BTW, sorry to hijack this thread! Now what?? Should I open a new thread or post in an existing thread?
How do I get help from now on as I am ready to do some very very minor mods to the codes?
Also I need to know the procedure to submit my new codes (if any) to the ML team... absolutely no clues...
Any pointers would be appreciated! Thanks!
Hello everyone -
I've decided to bite the bullet once again and get myself into this whole world of CITC... In effort to keep the development alive in the ML community. I ain't no coder but am pretty computer savvy and definitely have the patience for troubleshooting down bugs and whatnot.
Here's my 2nd take and since it has quite been a while since I last played with this... so I got to this point where it says:
(https://c1.staticflickr.com/1/284/19097048438_3c82b81bd4_n.jpg) (https://flic.kr/p/v6xnaQ)
Here's the clip of the entire process:
https://vimeo.com/132172185
Perhaps it has to do with 'abort: destination 'magic-lantern' is not empty' according to Terminal?
Any suggestions or thoughts?
Quote from: PhotoCat on June 30, 2015, 04:53:19 AMNow what?? Should I open a new thread or post in an existing thread?[/size][/font][/size]How do I get help from now on as I am ready to do some very very minor mods to the codes?Also I need to know the procedure to submit my new codes (if any) to the ML team... absolutely no clues...
Start a thread here: http://www.magiclantern.fm/forum/index.php?board=25.0 I guess
@DeafEyeJedi
:
montage is missing, says so right there, looking for it like this might help: https://packages.debian.org/search?searchon=contents&keywords=montage (https://packages.debian.org/search?searchon=contents&keywords=montage)
so install imagemagick?
I'm finding that Codio is running a little spotty. Sometimes it works, other times it dumps. However, I was able to get familiar with what is needed to work with and compile ML and I was finally able to get it working on my MacBook--no virtual box necessary.
In any case, compiling in the cloud is a good starting point for users that don't have a complete development platform to work with. Too bad that Codio started locking out new users from some of the feature needed to compile ML.
"Terminal and Box access is only available if you have an active subscription."
It doesn't look like this works anymore. Or am I missing something?
There are many free online IDE's with a linux vm, I frequently use https://c9.io or https://nitrous.io
Has anyone recently and successfully used Codio for compiling in the Cloud?
I have followed the instructions on page 1, which worked for me about a year ago, but the Codio UI has changed since then and I'm getting nowhere.
If someone is currently using Codio, I would be grateful if they could share their workflow.
Cheers
Garry
It's probably worth scrapping this idea and moving on. Just use a VM like I've mentioned in other threads. If we can get a Linux VM setup easily, maybe we can replace this topic with instructions for doing that.
Try this:
Install VirtualBox: https://virtualbox.org/wiki/Downloads
Install a prebuilt Ubuntu VM: http://virtualboxes.org/images/ubuntu/
Run the bootstrap script from the OP in a command prompt
wget -q -O - http://pastebin.com/raw.php?i=jfVXzw1a | sed "s/\r//g" | bash
I'm not a coder but tried this 'in the cloud' approach.
I failed at step one,
Has anyone successfully used this approach and is willing to give a little more insight into the workflow or steps to take.
I found some errors about module_strings.h while trying to compile the 500D firmware.
eg:
Building module mlv_lite...
Updated HGVERSION
[ README ] module_strings.h
Could not find rst2html/rst2html5/rst2html.py/rst2html5.py.
Please install python-docutils (pip install docutils).
[ CC ] mlv_lite.o
In file included from mlv_lite.c:48:0:
../../src/module.h:344:28: fatal error: module_strings.h: No such file or directory
#include "module_strings.h"
^
compilation terminated.
make[5]: *** [mlv_lite.o] Error 1
********************************************************
WARNING: module mlv_lite failed to build, deleting
********************************************************
Even though I could successfully compile the firmware, there were no modules. Below is my compilation log.
https://raw.githubusercontent.com/wanghurui/hello_world/master/screen.log (https://raw.githubusercontent.com/wanghurui/hello_world/master/screen.log)