Compiling Magic Lantern in the Cloud

[garry23]

@licaon_Kter

I tried adding the addtional code from Audionut but it confirmed docutils was already loaded.

Once again I tried the following:

Code: [Select]

wget -q -O - http://pastebin.com/raw.php?i=jfVXzw1a | sed "s/\r//g" | bash

cd magic-lantern/platform/5D3.113/

make clean && make zip
Every module appears to fail like this:

Code: [Select]

/bin/sh: 1: rst2html: not found                                                                                                                   
grep: write error: Broken pipe                                                                                                                   
                                                                                                                                                 
(<type 'exceptions.SystemExit'>, SystemExit(1,), <traceback object at 0x2b710424cfc8>)                                                           
make[4]: *** [module_strings.h] Error 1                                                                                                           
                                                                                                                                                 
********************************************************                                                                                         
WARNING: module mlv_snd failed to build, deleting                                                                                                 
********************************************************   
When I look in the downloaded zip, in the platform zip file it is obviously not built, ie no modules present.

What the hell am I doing wrong !!!!!!!!

[dmilligan]

Please run the commands I posted exactly as is (do not run the wget command!), and show the output.

The modules aren't getting built because a python script called "rst2html" is not present. It is not present because the download and setup of docutils fails for some strange reason. So I need you to run the commands that set up docutils so I can see what is happening. If you don't do this, there's nothing else I can do, b/c the script works just fine for me.

[garry23]

@dmilligan

I will do as you say, now.

BTW do I have to type in each line on the terminal, or is there an easy way to past in all the lines at once?

[garry23]

@dmilligan

OK spotted a problem, once I have followed your commands, which I have done, what do I do now, ie to compile ML.

BTW how do you copy all the text on a terminal?

[dmilligan]

You should be able to simply select the text, copy and paste it. If not, just take a screen shot.

[garry23]

@dmilligan

I'm sorry to appear to be dragging this out, but I can assure you I'm not!

After inputting your command, the screen terminal text gives pages, but in windows I can only grab a page worth at a time, ie I can't drag to select.

Unless you have any suggestion, I will have to labourosly copy bits at a time.

[Walter Schulz]

Mark text -> Open context menu and select "copy" (or just hit CTRL-C) -> Go to codio's terminal window and drag by CTRL-V

And I typed the whole text, without copy-paste ...

[garry23]

I've been away from this for a week, but now trying again.

I have followed instructions above to the letter, ie using these commands:

Code: [Select]

cd
curl -OL http://prdownloads.sourceforge.net/docutils/docutils-0.12.tar.gz
gzip -d docutils-0.12.tar.gz
tar -x -f docutils-0.12.tar
cd docutils-0.12
python setup.py install --prefix=~/.local
cd
export PATH="$HOME/.local/bin:$PATH"
echo "export PATH=\"\$HOME/.local/bin:\$PATH\"" >> .bash_profile
cp .local/bin/rst2html.py .local/bin/rst2html
cd workspace
hg clone -r unified https://bitbucket.org/hudson/magic-lantern
I then did this:

Code: [Select]

cd magic-lantern/platform/5D3.113/
make zip
But all I get is this:

Code: [Select]

codio@santana-dynasty:~/workspace/magic-lantern/platform/5D3.113$ make zip                                                           
../../Makefile.inc:79: remove /home/codio/workspace/magic-lantern/platform/5D3.113/zip                                               
[ RM dir ]  /home/codio/workspace/magic-lantern/platform/5D3.113/zip/                                                                 
mkdir -p /home/codio/workspace/magic-lantern/platform/5D3.113/zip                                                                     
[ CPP      ]   magiclantern.lds                                                                                                       
/bin/sh: 1: /home/codio/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-gcc-4.8.3: not found                                           
make: *** [magiclantern.lds] Error 1                                                                                                 
codio@santana-dynasty:~/workspace/magic-lantern/platform/5D3.113$   
Can anyone with knowledge help me please.

[surami]

You leaved out this 3 lines, run them before docutils.

Code: [Select]

curl -OL https://launchpad.net/gcc-arm-embedded/4.8/4.8-2013-q4-major/+download/gcc-arm-none-eabi-4_8-2013q4-20131204-linux.tar.bz2
bzip2 -d gcc-arm-none-eabi-4_8-2013q4-20131204-linux.tar.bz2
tar -x -f gcc-arm-none-eabi-4_8-2013q4-20131204-linux.tar

[garry23]

@surami

Tried that, ie used this set of commands:

Code: [Select]

cd
curl -OL http://prdownloads.sourceforge.net/docutils/docutils-0.12.tar.gz
gzip -d docutils-0.12.tar.gz
tar -x -f docutils-0.12.tar
curl -OL https://launchpad.net/gcc-arm-embedded/4.8/4.8-2013-q4-major/+download/gcc-arm-none-eabi-4_8-2013q4-20131204-linux.tar.bz2
bzip2 -d gcc-arm-none-eabi-4_8-2013q4-20131204-linux.tar.bz2
tar -x -f gcc-arm-none-eabi-4_8-2013q4-20131204-linux.tar
cd docutils-0.12
python setup.py install --prefix=~/.local
cd
export PATH="$HOME/.local/bin:$PATH"cd
echo "export PATH=\"\$HOME/.local/bin:\$PATH\"" >> .bash_profile
cp .local/bin/rst2html.py .local/bin/rst2html
cd workspace
hg clone -r unified https://bitbucket.org/hudson/magic-lantern
Then did a make clean and a make zip, but got this:

Code: [Select]

codio@cricket-rio:~/workspace/magic-lantern/platform/5D3.113$ make zip                                                                                                                                                                     
../../Makefile.inc:79: remove /home/codio/workspace/magic-lantern/platform/5D3.113/zip                                                                                                                                                     
[ RM dir ]  /home/codio/workspace/magic-lantern/platform/5D3.113/zip/                                                                                                                                                                     
mkdir -p /home/codio/workspace/magic-lantern/platform/5D3.113/zip                                                                                                                                                                         
[ CPP      ]   magiclantern.lds                                                                                                                                                                                                           
/bin/sh: 1: /home/codio/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-gcc-4.8.3: not found                                                                                                                                               
make: *** [magiclantern.lds] Error 1                                                                                                                                                                                                       
codio@cricket-rio:~/workspace/magic-lantern/platform/5D3.113$             
Any ideas?

[surami]

Read this again and don't forget, that where the many code lines are, there is a scrollbar too.

I just tested and it works for me.

[garry23]

@surami

Many thanks for staying with me :-)

I followed your instructions to the letter and it worked!!!

I have one last question, if I may?

At the moment I have to enter every command line individually. Do you do this as well?

If you don't, how do you use pastebin to run the command in batch mode? I have a pastebin account.

Once again, many thanks for helping me 

[surami]

You can get your own Pastebin URL without an account too:

0. Copy this code. I make it universal, so people with another camera can use this too:

Code: [Select]

cd
curl -OL https://launchpad.net/gcc-arm-embedded/4.8/4.8-2013-q4-major/+download/gcc-arm-none-eabi-4_8-2013q4-20131204-linux.tar.bz2
bzip2 -d gcc-arm-none-eabi-4_8-2013q4-20131204-linux.tar.bz2
tar -x -f gcc-arm-none-eabi-4_8-2013q4-20131204-linux.tar
curl -OL http://prdownloads.sourceforge.net/docutils/docutils-0.12.tar.gz
gzip -d docutils-0.12.tar.gz
tar -x -f docutils-0.12.tar
cd docutils-0.12
python setup.py install --prefix=~/.local
cd
export PATH="$HOME/.local/bin:$PATH"
echo "export PATH=\"\$HOME/.local/bin:\$PATH\"" >> .bash_profile
cp .local/bin/rst2html.py .local/bin/rst2html
cd workspace
hg clone -r unified https://bitbucket.org/hudson/magic-lantern1. Go to the Pastebin site and paste the code to the New Paste area.
2. Copy the (I say now) platform code line to the end of the previous code lines at the Pastebin site:

Code: [Select]

cd magic-lantern/platform/PLATFORM.FOLDER.NAME/You can get your platform folder name from the nightly builds download site. Eg. "5D3.113":

Code: [Select]

cd magic-lantern/platform/5D3.113/3. Write this to the end of the code lines at the Pastebin site:

Code: [Select]

make zip4. Submit
5. Captcha -> Submit
6. Open a Notepad and copy this code to there:

Code: [Select]

wget -q -O - http://pastebin.com/raw.php?i=HERE_COMES_THE_LAST_8_CHARACHTER | sed "s/\r//g" | bash7. Copy the last 8 charachter from your browser address line, you will see something like this

Code: [Select]

http://pastebin.com/ABCDEFGH, so you need "ABCDEFGH":
8. Replace the "HERE_COMES_THE_LAST_8_CHARACHTER" with that last 8 characher, Eg.:

Code: [Select]

wget -q -O - http://pastebin.com/raw.php?i=ABCDEFGH | sed "s/\r//g" | bash
Copy this code line and paste to the Codio Terminal and hit enter.

This way you can get a unified build, but (till now) without the dot_tune, dual_iso and raw_rec module (gcc 32bit issue).

You can make two Pastebin URLs too. One for cloning the repository and one for compiling the build. In between you can make your unique modifications with your own responsibility.

[garry23]

@surami

Many thanks...once again 

[garry23]

@surami

Maybe I spoke too soon.

When I use your command lines and enter them one at a time, all works fine.

If I use the pastebin approach I just get one module compiled

Have you, or anyone else, seen this behavio?

Cheers

PS here is my pastebin: http://pastebin.com/6NUPS8ff

[surami]

@garry23

I don't know what is the problem now. Sometimes the compiling works, sometimes not. Maybe it's a Codio service issue?

Here are the two lines (+pastebin links), what I tried:
1. Cloning the unified repository (click here to see the code):

Code: [Select]

wget -q -O - http://pastebin.com/raw.php?i=1Fh8eyhz | sed "s/\r//g" | bash2. Compiling the unified build for 5D Mark III (1.1.3) (click here to see the code):

Code: [Select]

wget -q -O - http://pastebin.com/raw.php?i=vRDmTAT6 | sed "s/\r//g" | bash

[garry23]

@surami

I think you are confirming what I found, ie using the pastebin approach doesn't work every time.

Thanks for looking into things.

BTW my biggest problem is that I simple can't get sqrt to work in the C++ ML environment. It always returns zero, even when I know the input is a double variable.

Do you, or anyone else (any developers) know if sqrt function is supported?

[dmilligan]

ML is not C++, it's plain C.

The ARM processor in these cameras doesn't have floating point hardware. You can do some floating point stuff, but it's very slow and you should try to avoid it (if you do use floating point, use float, not double). It's better to use integers or fixed point instead (you'll notice that ML makes extensive use of fixed point, and there is very little floating point). See imath.h for some integer based math functions and helper macros. Remember to be careful about the order of operations to avoid propagating round off errors.

[garry23]

@dmilligan

Many thanks for your insight.

I am determined to learn how to tweak the ML code and have spent all day 'educating' myself, eg looking at bits of ML.

Clearly sqrtf works but, as you say, I need to be careful.

Once again: thanks.

[garry23]

I'm adding this post to hopefully give confidence to others.

Before trying the 'compile in the cloud' approach, I knew nothing about C or compiling C for ML. Thanks to great support from several (see above) I have now reached the stage of successfully personalising ML: which I just wrote about on my blog http://photography.grayheron.net/2015/04/customising-magic-lantern.html

The test case I chose was to tweak the ML DoF feedback to account for diffraction.

After 'educating' myself about C and its syntax and rules, I managed to successfully tweak ML to now give me DoF and account for diffraction.

In terms of the way forward: I'm afraid until the compile in the cloud is fully running, ie all modules compiled, I will have to put further efforts to personalise ML on hold.

The message I wanted to get out, is that, if I can do it, so can you!

[Audionut]

The next step is to create a pull request to push your changes back to the main repository.

http://www.magiclantern.fm/forum/index.php?topic=7940.0

[Zaimon]

Hi,

I have compiled for the 600D.102 according to your instructions.
Everything went fine but I can not find any lua.mo in modules.

Am I doing something wrong?

[dmilligan]

The lua stuff is not merged to the main branch ('unified') yet, it's in a separate branch. You need to switch to the lua branch (this guide is for compiling ML in general, not specifically for compiling lua branch).

From 'magic-lantern' directory, run:

Code: [Select]

hg pull -u -b lua

You also need to explicitly build the lua.mo (I didn't add it to the modules makefile yet):

Code: [Select]

make -C modules/lua

However, IIRC the 600D is like 1100D in regards to the fairly small autoexec.bin size limit (someone please correct me if I'm thinking of another camera). All of the extra libc symbols needed by lua push the autoexec.bin size over the limit (at least on 1100D). So you will probably have to disable some stuff in platform/600D.102/features.h to get it to run (to disable a feature, simply #undef the macro for it; for a list, see all_features.h). Look for the MemSiz column in the build output, it should be less than 0x80000 (512KB).

You're looking for something like this:
Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  EXIDX          0x073918 0x00cf3918 0x00cf3918 0x00008 0x00008 R   0x4
  LOAD           0x000100 0x00c80100 0x00c80100 0x73820 0x7ff0c RWE 0x100


For 1100D I usually disable features that take up a lot of space, that I don't need like: DEFISHING_PREVIEW, ANAMORPHIC_PREVIEW, MOTION_DETECT, TRAP_FOCUS, FOLLOW_FOCUS, etc.

[Zaimon]

@dmilligan

I built the lua.mo and as you pointed out I needed to reduce the size of the autoexec.bin.
The module seems to work correctly.

Thanks a lot!

[dfort]

Thank you so much for this. I finally got a way to compile ML from source and had my first pull request merged into unified. Whoo Hoo!

A while back I wanted to compile raw2dng for use on a Macbook Pro and got tripped up because I was following instructions to compile ML. Finally got around it by pulling out just the raw2dng code. Now with dmilligan's guidance we got a simple way to compile ML from source but I am getting tripped up trying to get the raw_rec and dual_iso modules because of this:

http://www.magiclantern.fm/forum/index.php?topic=14725.msg142856#msg142856

Yeah, cr2hdr and raw2dng have to be compiled in 32 bit mode, and this VM is 64 bit and doesn't have the 32bit gcc libraries installed. Without sudo privliges, I'm not sure installing them is possible. The only solution really is to modify the makefile for dual_iso module so that it doesn't try to compile cr2hdr (the same for raw_rec which tries to compile raw2dng).

I tried to work my way around it but have failed miserably. Oh well, everything else seems to work fine.

So my question is--is it ok to copy the raw_rec and dual_iso modules from the ML/modules directory of the nightly builds? I tried it and it seems to work but thought I'd ask.