Getting started with development

[Alex_Palomo]

Where do you find the virtual machine? I've tried to downlaod the VM of the front page but it is not available anymore.
Thank you!

[Licaon_Kter]

Guess it's easier to make one from scratch, say using Lubuntu and these: https://developer.arm.com/open-source/gnu-toolchain/gnu-rm/downloads

BTW, do you want to build something new or to test those in the works?  Did you see the new page with experiments builds: https://builds.magiclantern.fm/experiments.html ?

[Audionut]

What about this one:  http://www.magiclantern.fm/forum/index.php?topic=7579.msg134989#msg134989

[reddeercity]

Yea that one works it's the one I use to compile on VM in Win7 started a thread here with a short video tutorial .

[Alex_Palomo]

Thank you guys! You've been such a great help!!! Now that I can compile ML, it's time to dev.
Your video has been very useful to me reddeercity.
Licaon_Kter, my idea is to build something that I've been investigating. Sure that I will let you know!
Have a good day!

[henrus]

http://mercurial.selenic.com/wiki/Tutorial

Replying to an old post, but in case anyone is reading them, this link is no longer working. A good Mercurial Tutorial can be found here: http://hginit.com/

[a1ex]

Preparing to update the build system, with the goal of making the environment setup as painless as possible, on all major operating systems. I need some help with testing this PR.

The stuff I've been tweaking lately includes:
- Mac fixes (seems like it's impossible not to break this one, even with apparently minor changes)
- parallel build fixes (make -j8)
- fixing minor quirks

New features from the PR:
- top-level build rules (zip, 5D3_zip, 60D_install_qemu etc)
- compiler autodetection (use any arm-none-eabi-gcc from your executable path, or any gcc-arm-none-eabi from your home directory, without editing the Makefiles)
- Linaro arm-gcc support (experimental, see PR description for details)
- nicer error messages, toolchain suggestions if none is installed

There was good feedback on the PR from veterans around here, but some things have changed and I'm still finding OS-specific issues, so I'd like to encourage more users (newcomers too!) to give it a try. Just compile from the "makefile-updates" branch instead of unified, and let me know how it goes:

Code: [Select]

hg clone -u makefile-updates https://bitbucket.org/hudson/magic-lantern
cd magic-lantern
make
# follow error messages, install dependencies and so on

In particular, I'm looking for cases where the error message is unclear when some build tool is not found and has to be installed. Or, for the build process not working on some widely used operating system (even if it didn't work before). Or, to suggest better ways of doing things (I'm far from being a Makefile expert - mostly copy/pasting from Stack Overflow).

You should be able to build ML under any recent Linux distro, Mac and Windows (both Cygwin and WSL). Would be nice to know if it works on BSD as well

Just FYI: there were some attempts to automate the environment preparation for various operating systems, but they are beyond the scope of this PR. One such attempt is in QEMU's install.sh (Mac demo), others are in various sticky topics on this forum. The former is also waiting to be merged into mainline, so feel free to give it a try as well.

[dfort]

The stuff I've been tweaking lately includes:
- Mac fixes (seems like it's impossible not to break this one, even with apparently minor changes)
- parallel build fixes (make -j8)
- fixing minor quirks

It is working great on this Mac.

Glad to see you addressed one of my pet peeves - when compiling in the camera directory, running "make clean" wouldn't clean out build_tools/xor_chk so I kept having to run "make clean" from the top directory which takes a lot longer.

Using "make -j8" speeds things up on this system:

make zip	__	44 sec.
make -j8 zip	__	31 sec.

On the Compiling Magic Lantern on a Macintosh Quick installation script I'm doing this:

Code: [Select]

make -j`grep -c processor /proc/cpuinfo 2> /dev/null || sysctl -n hw.ncpu 2> /dev/null || echo 1`
Wouldn't it work in a Makefile by applying what you've got in src/Makefile.src to automatically detect the number of processors and make parallel builds as the default? (You probably already thought of that and decided against it, right?)

[a1ex]

Overriding -j directly in the Makefile apparently worked in some older versions, but stopped working in newer ones. Found this workaround, but seemed too complicated - ended up just printing a suggestion.

I compile often without modules, or with a single module enabled (for example, make 5D3.113_install_qemu ML_MODULES=silent). That's where you'll see the biggest speed-up. Modules are compiled sequentially (there are some unsolved dependencies) - that's what's slowing down the build process. After make clean, and after having all the files cached (i.e. not the first run), my times are:
- For the above command: 10s with -j1, 4.2s with -j4 and 3.6s with -j8.
- Without compiling any modules (make 5D3.113_install_qemu ML_MODULES=): 9.2, 3.3 and 2.7
- With the default set of modules (make 5D3.113_install_qemu): 33, 22, 21.

Enabling parallel build for modules (comment out .NOTPARALLEL in modules/Makefile) drops the last number to 9.5 seconds. Unfortunately this relies on a bug in the build system to work out the dependencies between modules (e.g. mlv_snd depends on mlv_rec, but mlv_snd compiles quicker, being simpler). For some reason, the modules are compiled twice (couldn't figure it out yet), and that happens to fix the missed dependencies for mlv_snd.

In any case, parallel build is not able to make things 4x as fast, as you would expect for a quad-core CPU, but it's getting close.

There's also CONFIG_CCACHE=y, which makes the build process *really* fast (0.3 seconds for make 5D3.113), although I didn't experiment much with it, and I'm unable to tell whether this caching is safe or not. It was broken before this PR.

For modules, the slowest step is getting its last changeset from Mercurial (each module is tagged with the last changeset that affected it directly).

[dfort]

Figured you already thought of it. I thought running though the modules twice was a feature, not a bug. Guess it is necessary for parallel processing because of module dependencies.

Is this a good place for a feature request?

When running "make install" please also copy the ML-SETUP.FIR to the card.

That's another pet peeve of mine. Just now I installed ML on a card that I didn't realize didn't have the boot flag enabled on the card. Fortunately I found out before heading out because I wasn't taking my computer and card reader with me.

[a1ex]

Good catch - solved that too (along with some other minor quirks).

[dfort]

Another feature request -- make firmware update. Automatically updates the camera and ML to the latest Canon firmware. 

[a1ex]

CHDK has it

[dfort]

You know I was just kidding but someone did it--well almost. Some things still need to be found.

Code: [Select]

makefile.inc: MEMISOSTART will need updating if the size of canon DATA + BSS has changed. Hopefully that will not be the case across different version of firmware on the same camera
Sounds like what happened on the 5D3.135 update.

In any case, back on topic -- nice improvements on the build system.

[Danne]

Ran the latest environment fixes:

Code: [Select]

hg clone -u makefile-updates https://bitbucket.org/hudson/magic-lantern
cd magic-lantern
make
# follow error messages, install dependencies and so onSeemed to work:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/Make.txt

But when trying to compile cr2hdr on my mac I get the same issues as before if not fixing the makefile.user file:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/dual_iso.txt

[a1ex]

Alright, so the build system itself is fine (hopefully not only on my system). You could try additional stuff, like "make install", "make zip", check if it builds the same thing as the "unified" branch (a bit hard to check, as we don't have fully reproducible builds - at least the date/time of each build will be different).

The warnings or errors you've got are not from the Makefiles.

[Danne]

The issue is related to "Host compiler settings" on mac. Created a Makefile.user with following changes and cr2hdr compiles just fine:

source:
http://www.magiclantern.fm/forum/index.php?topic=16012.0

Code: [Select]

#
# Host compiler settings
#
HOST_CC=$(shell which gcc)
HOST_LD=$(shell which ld)
HOST_AR=$(shell which ar)

Code: [Select]

#
# Host compiler settings
#
HOST_CC=gcc-5
HOST_LD=gcc-5
HOST_AR=$(shell which ar)
cr2hdr is created and in terminal:

Code: [Select]

dans-MBP:dual_iso dan$ make cr2hdr
Using ~/gcc-arm-none-eabi-5_4-2016q3/bin/arm-none-eabi-gcc (preferred).
[ gcc-5    ]   cr2hdr
dans-MBP:dual_iso dan$
Will do some more testing make zip etc and post findings in this post.


Make zip

Code: [Select]

cd magic-lantern
time make -j8 zip

real 2m36.460s
user 3m38.229s
sys 0m40.887shttps://bitbucket.org/Dannephoto/magic-lantern/downloads/Make_J8.txt
vs

Code: [Select]

cd magic-lantern
time make zip
real 3m59.816s
user 3m13.599s
sys 0m40.220shttps://bitbucket.org/Dannephoto/magic-lantern/downloads/Make_zip.txt

[a1ex]

Errors from "make zip" fixed in a289367.

Also checked whether "make -j8", both with old and new makefiles, compiles the same thing - the trick was to rename the branch so the metadata header has the same length as with unified, then compare the binaries with vbindiff. From my side, it's ready for including in mainline.

edit: merged!

[dfort]

Finally got a chance to try it at work - HP Z840 running Windows 7 and Cygwin:

Code: [Select]

Tip: to compile faster, try one of these:
    make -j32
    export MAKEFLAGS='-j32'

Yeah, it is a monster.

From the top directory:

Code: [Select]

make EOSM      - 25 sec.
make -j32 EOSM -  4 sec.

Just running my phone as a stopwatch so it isn't very accurate.

So how about a full build?

Code: [Select]

make EOSM_zip      - 2 min. 24 sec.
make -j32 EOSM_zip - 1 min. 35 sec.

Not as dramatic but a big improvement. Cygwin isn't fast but it is super easy to setup and fairly non intrusive. I've run it on everything from an old 32-bit Vista laptop to this monster.

[Danne]

Cool, speedy!
Has cygwin time installed? If so run i.e:

Code: [Select]

time make -j32 EOSMTo get specific processing times

[dfort]

Nice trick!

Code: [Select]

time make EOSM
...
real    0m4.319s
user    0m0.733s
sys     0m2.750s

time make -j32 EOSM
...
real    0m3.697s
user    0m2.068s
sys     0m9.598s

time make EOSM_zip
...
real    2m23.027s
user    0m32.539s
sys     1m5.425s

time make -j32 EOSM_zip
...
real    1m34.908s
user    0m33.797s
sys     1m13.671s

So what does this mean? Looks like I was only measuring the "real" time.

Oh--wait. Seems there's some memory caching going on. Even though I was running "make clean" between builds these times are very different. Look at what happens if I run this in a new terminal:

Code: [Select]

time make -j32 EOSM_zip
...
Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  LOAD           0x0000e0 0x0009e1e0 0x0009e1e0 0x656c8 0x78a34 RWE 0x100

[ CP       ]   EOSM_202.sym
[ CC       ]   version.o


[ CP       ]   install
../../platform/EOSM.202/version.c:5:12: error: redefinition of 'build_id'
 const char build_id[] = "97b73e7a781e (unified)";
            ^
../../platform/EOSM.202/version.c:2:12: note: previous definition of 'build_id' was here
 const char build_id[] = "97b73e7a781e (unified)";
            ^
../../platform/EOSM.202/version.c:6:12: error: redefinition of 'build_date'
 const char build_date[] ="2017-12-07 22:56:23 UTC";
            ^
../../platform/EOSM.202/version.c:3:12: note: previous definition of 'build_date' was here
 const char build_date[] ="2017-12-07 22:56:23 UTC";
            ^
../../platform/EOSM.202/version.c:7:12: error: redefinition of 'build_user'
 const char build_user[] = "dfort@QANTAS";
            ^
../../platform/EOSM.202/version.c:4:12: note: previous definition of 'build_user' was here
 const char build_user[] = "dfort@QANTAS";
            ^
make[2]: *** [../../Makefile.filerules:23: version.o] Error 1
make[2]: *** Waiting for unfinished jobs....
cp ML-SETUP.FIR /home/dfort/magic-lantern/platform/EOSM.202/zip/
[ VERSION  ]   ../../platform/EOSM.202/version.bin
[ OBJCOPY  ]   magiclantern.bin
[ STAT     ]   magiclantern.bin
magiclantern.bin: 415432 bytes
make[2]: Leaving directory '/home/dfort/magic-lantern/installer/EOSM.202'
make[1]: *** [../../platform/Makefile.platform.extras:52: installer_check] Error 2
make[1]: *** Waiting for unfinished jobs....
make[1]: Leaving directory '/home/dfort/magic-lantern/platform/EOSM.202'
make: *** [Makefile:24: EOSM.202_zip] Error 2

real    0m4.113s
user    0m1.507s
sys     0m6.026s


[dfort]

I discovered a slight issue with the new Makefile system. Doing a "make clean" in either the platform or the top directory will not clean out the installer directory. Not a big deal but it shows up in the SourceTree app which beginners like me are using:

[a1ex]

Solved both, and a few others I've found meanwhile:

https://bitbucket.org/hudson/magic-lantern/pull-requests/898/makefile-updates-part-2

[digiboy]

Hi,
I use ubuntu 18.04.
I try compile and receive error:
~/MagicLantern/magic-lantern/platform/550D.109$ make install
Using /usr/bin/arm-none-eabi-gcc (from PATH).
[ MKDIR    ]   ML directory structure...
mkdir: cannot create directory ‘/ML’: Permission denied
../../Makefile.setup:149: recipe for target 'prepare_install_dir' failed
make: * [prepare_install_dir] Error 1

Do you know why script want create folder in /ML ?
Where I can change this default folder "/ML" to something else?

[a1ex]

It could not figure out where your card is mounted. True, this can be handled a little nicer.

Try this:

Code: [Select]

make install CF_CARD=/media/EOS_DIGITAL     # path where your card is mounted, or any other directory

Or, "make zip" followed by manual installation.