Magic Lantern Forum

Developing Magic Lantern => General Development Discussion => Topic started by: Francis on August 13, 2013, 02:19:50 AM

Title: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: Francis on August 13, 2013, 02:19:50 AM
It seems like lots of people are using wine to run the command line tools provided with Magic Lantern. Here is how to compile and create symbolic links to the binary to make the workflow a little bit simpler ;).

I'm assuming you are fairly comfortable using a command line interface and navigating the filesystem with commands like cd. If you aren't then this is not the resource for you, you have plenty of other reading to do before you will be ready for this. Edit: for emphasis.

You're going to need a few tools for this: MacPorts, hg, dcraw, and a pre-compiled arm toolchain.

First step is download and install MacPorts (http://www.macports.org/), a command line-based package system similar to APT for linux. After installation is complete you are going to want to install a few packages.

In terminal type 'sudo port install mercurial' to download install hg', which we will use to download the source. You'll be prompted for your administrator acct. password. Follow any warnings or suggestions on updating the package list.

Next we're going to download dcraw, which is needed by cr2hdr and raw2dng for conversion purposes. Again in terminal type 'sudo port install dcraw'.

We're done with MacPorts for now and can use hg to download the source tree. Navigate in Terminal to whereever you wish to download the ML source files and type 'hg clone -r unified https://bitbucket.org/hudson/magic-lantern'. This will create a folder ./magic-lantern/ which contains all the source files.

Now that we have the source we need to get a pre-compiled toolchain (https://launchpad.net/gcc-arm-embedded) to compile the code. Save and extract.

Now we have to tell the ML build files where they can fine the compiler. open ./magic-lantern/makefile.user.default in a text editor and and change the following line to indicate where you extracted the toolchain.

Change
Code: [Select]
ARM_PATH=~/gcc-arm-none-eabi-4_7-2012q4
to
Code: [Select]
ARM_PATH=~/path/to/compiler
For me its
Code: [Select]
ARM_PATH=~/Downloads/Magic_Lantern/gcc
Be sure to not end your path with '/'

Save this new file as makefile.user (dropping the .default extension). Now we are ready to build.

First, you must compile the autoexec.bin and associated .sym file for your camera.

Now, in terminal, go to your source ./magic-lantern/modules/raw_rec

Type make. You will get some warnings about not having compiled the source for any model but these instructions are purely for getting the utilities to compile. You'll end up with our first utility, raw2dng, amongst other things like the module files.

Now we are going to make a link to raw2dng so that it can be run as a command in terminal no matter what directory you are in.

Type 'sudo ln -s raw2dng /usr/bin/raw2dng' This creates a symbolic link to the original file.

Now we are going to do the same for cr2hdr. Enter ./magic-lantern/modules/dual_iso

Type 'make' then 'sudo ln -s cr2hdr /usr/bin/cr2hdr'.

Voila now these 2 utilities are compiled and ready to be run from any directory through the terminal. For example with dual ISO images, enter the folder with the images and type 'cr2hdr IMG_0000.cr2'

To be totally honest there are probably some bumps you are going to run into and files you may have to edit to get everything compiling. I'll do my best to help but I have very limited programming experience and mostly have gotten through this with the help of Coutts 'developing on Mac' wiki article and a lot of trial and error.
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: Danne on August 13, 2013, 07:34:16 AM
Supercool. I, ll get right on it. Thank you very much
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: REDcineX on August 13, 2013, 07:30:21 PM
I am getting an error because it cannot find the file.

"/bin/sh: /Users/Admin/Downloads/magic-lantern/gcc/bin/arm-none-eabi-gcc-4.7.3: No such file or directory"
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: REDcineX on August 13, 2013, 07:31:26 PM
I looked up the file in the directory and found that on mac it is a .alias instead of an executable that surround it.




EDIT: And it doesn't link to an original file..
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: REDcineX on August 13, 2013, 07:35:53 PM
After downloading a few versions of the toolchain, I found an .exe of the 4.7.3 file. Now Terminal tells me that it cannot find the -ld file. I found the file and it is, as previously stated, a .alias file as well. It points no where.


"Macbook-Pro-i7-26gHz-2012:dual_iso Admin$ make
[ MODULE   ]   dual_iso.mo
/bin/sh: /Users/Admin/gcc-arm-none-eabi-4_7-2012q4/bin/arm-none-eabi-ld: No such file or directory
make: *** [dual_iso.mo] Error 127"

Proof that file is in directory:
(http://i1230.photobucket.com/albums/ee486/azactro/ScreenShot2013-08-13at102954AM.png) (http://s1230.photobucket.com/user/azactro/media/ScreenShot2013-08-13at102954AM.png.html)

Makefile.user:
(http://i1230.photobucket.com/albums/ee486/azactro/ScreenShot2013-08-13at103708AM.png) (http://s1230.photobucket.com/user/azactro/media/ScreenShot2013-08-13at103708AM.png.html)
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: Francis on August 13, 2013, 08:16:10 PM
You downloaded the windows version of the toolchain. That is why they are all .exes
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: REDcineX on August 13, 2013, 08:18:10 PM
I have tried the windows, linux, and mac version. All with the same results. It just happened to be the windows one I took a screenshot of.


(http://i1230.photobucket.com/albums/ee486/azactro/ScreenShot2013-08-13at110937AM.png) (http://s1230.photobucket.com/user/azactro/media/ScreenShot2013-08-13at110937AM.png.html)
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: Francis on August 13, 2013, 08:33:07 PM
That screenshot doesn't show the full path but I'm assuming you removed the Windows files and put these in the same place. First in your makefile.user you have gcc version =4.7.4 instead of 4.7.3 but that probably isn't your only problem. Did you download the toolchain from the page I linked to? If so why use an older version? The current download links on that main page are 2013q2. That probably isn't a problem either.

I'm not sure what to tell you but if your first impulse when it didn't work was to try the Windows version instead, you're probably not ready for this. Just download this binary (http://http:/avlmotofoto.com/cr2hdr) and then make the symbolic link again. You probably should delete the one you've possibly already made.

Edit: Actually the 4.7.4 line probably is most of your problem. Change that to 4.7.3 and download the current toolchain. In my version they are all unix executables, none of them are links.
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: REDcineX on August 14, 2013, 12:37:43 AM
My first urge wasn't to use the windows version. I have parallels installed so I was going to see if I could use anything on windows. I rendered it pointless as others have made the mac version work.

I have tried both the new version of the toolchain and old versions because depending on which one I download I get more .alias (link) files. That is exactly how it looks after extracting the mac .tar.bz2

EDIT: I downloaded the current toolchain and changed it back to 4.7.3. My problem is still "/bin/sh: /Users/Admin/Downloads/gcc-arm-none-eabi-4_7-2013q2/bin/arm-none-eabi-ld: No such file or directory"

Because it appears as a link/alias and goes nowhere. I don't understand how it is doing that..
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: REDcineX on August 14, 2013, 12:47:32 AM
Fixed the problem of the .alias files. It was my unarchiver.

Now it still won't work, but at least that problem above is out of the way.

Macbook-Pro-i7-26gHz-2012:dual_iso Admin$ make
[ MODULE   ]   dual_iso.mo
[ STRIP    ]   dual_iso.mo
[ EXPORTS  ]   dual_iso.sym
[ DEPENDS  ]   dual_iso.dep
Not checked (compile ML for these cameras first):
    100D, 1100D, 40D, 500D, 50D, 550D, 5D2, 5D3, 5DC, 600D, 60D, 650D, 6D, 700D, 7D, EOSM
make: *** [dual_iso.dep] Error 1
Macbook-Pro-i7-26gHz-2012:dual_iso Admin$ sudo ln -s cr2hdr /usr/bin/cr2hdr
Password:
ln: /usr/bin/cr2hdr: File exists


Is my problem that the file is already there so it won't overwrite it? Where can I actually find that file and erase it?
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: Francis on August 14, 2013, 01:08:16 AM
Delete it with the rm command. You don't have to use sudo (which switches to the admin/root account to perform a command) if you are already logged in as the admin.

To delete type
rm /usr/bin/cr2hdr

Then link again. If it says you don't have permission then add the sudo to the beginning of it.

Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: zappa1976 on August 14, 2013, 01:42:14 AM
Francis,

sorry if I'am asking again your help but i have some errors.

I link a screenshoot with errors I have.

(http://imageshack.us/a/img850/9168/hz4a.png)

I attached also 2 other screen shoot to show my settings.

(http://imageshack.us/a/img541/9208/yjpn.png)

(http://imageshack.us/a/img32/2601/b0wh.png)

Please can you check if are they ok?

Many thanks in advance!
Saverio
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: REDcineX on August 14, 2013, 02:13:18 AM
Hehe. One step at a time. Successfully deleted the previous file. Unfortunately, it still won't recognize "cr2hdr" as a command.

Macbook-Pro-i7-26gHz-2012:~ Admin$ rm /usr/bin/cr2hdr
rm: /usr/bin/cr2hdr: Permission denied
Macbook-Pro-i7-26gHz-2012:~ Admin$ sudo rm /usr/bin/cr2hdr
Password:
Macbook-Pro-i7-26gHz-2012:~ Admin$ cd magic-lantern/modules/dual_iso
Macbook-Pro-i7-26gHz-2012:dual_iso Admin$ make
[ DEPENDS  ]   dual_iso.dep
Not checked (compile ML for these cameras first):
    100D, 1100D, 40D, 500D, 50D, 550D, 5D2, 5D3, 5DC, 600D, 60D, 650D, 6D, 700D, 7D, EOSM
make: *** [dual_iso.dep] Error 1
Macbook-Pro-i7-26gHz-2012:dual_iso Admin$ sudo ln -s cr2hdr /usr/bin/cr2hdr
Macbook-Pro-i7-26gHz-2012:dual_iso Admin$ sudo ln -s cr2hdr /usr/bin/cr2hdr
ln: /usr/bin/cr2hdr: File exists
Macbook-Pro-i7-26gHz-2012:dual_iso Admin$ cd ..
Macbook-Pro-i7-26gHz-2012:modules Admin$ cd ..
Macbook-Pro-i7-26gHz-2012:magic-lantern Admin$ cd ..
Macbook-Pro-i7-26gHz-2012:~ Admin$ cd desktop
Macbook-Pro-i7-26gHz-2012:desktop Admin$ cd untitledfolder
Macbook-Pro-i7-26gHz-2012:untitledfolder Admin$ cr2hdr _MG_9518.CR2
-bash: cr2hdr: command not found
Macbook-Pro-i7-26gHz-2012:untitledfolder Admin$
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: REDcineX on August 14, 2013, 02:19:47 AM
Francis,

sorry if I'am asking again your help but i have some errors.

I link a screenshoot with errors I have.

(http://imageshack.us/a/img850/9168/hz4a.png)

I attached also 2 other screen shoot to show my settings.

(http://imageshack.us/a/img541/9208/yjpn.png)

(http://imageshack.us/a/img32/2601/b0wh.png)

Please can you check if are they ok?

Many thanks in advance!
Saverio


Can you post a screenshot of what is in "bin"?
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: REDcineX on August 14, 2013, 02:44:00 AM
I believe you need to remove the "/Users/Saverio" in the ARM path on your makefile.user.
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: zappa1976 on August 14, 2013, 02:54:33 AM
I believe you need to remove the "/Users/Saverio" in the ARM path on your makefile.user.
I tried but I have the same error.. :-[
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: Francis on August 14, 2013, 03:41:04 AM
I tried but I have the same error.. :-[

You don't need to change anything in your makefile.user. You've got it built. Now the only problem is making the link.

Try typing out the whole path when you are making the link.

sudo rm /usr/bin/cr2hdr <--to get rid of the old link
sudo ln -s /Users/Saverio/magic-lantern/modules/dual_iso/cr2hdr /usr/bin/cr2hdr

It's working for me with those exact same steps.
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: zappa1976 on August 14, 2013, 03:47:00 AM
I believe you need to remove the "/Users/Saverio" in the ARM path on your makefile.user.

Something now it is working (I download the second toolchain "gcc-arm-none-eabi-4_7-2013q1"  with 4.7.3 instead of "gcc-arm-none-eabi-4_7-2013q2" with 4.7.4...is it right??)

I attached a screenshoot with the Error 1 I still have..

(http://imageshack.us/a/img829/3756/uerg.png)

I don't know how to solve the error..
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: REDcineX on August 14, 2013, 03:51:54 AM
@Francis Do you know why I am still getting a "command not found" error?

Look at reply #12 for the terminal
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: Francis on August 14, 2013, 05:50:16 AM
Here is what I did, and have redone multiple times:

First start with this
Code: [Select]
Francis-Danforths-MacBook-Pro:~ danforthf$ sudo rm /usr/bin/cr2hdr
Password:
Francis-Danforths-MacBook-Pro:~ danforthf$ cd magic-lantern/modules/dual_iso/
Francis-Danforths-MacBook-Pro:dual_iso danforthf$ make clean
[ RM ]  dual_iso.o dual_iso.mo *.o *.d
[ RM ]  cr2hdr
Francis-Danforths-MacBook-Pro:dual_iso danforthf$

This basically resets you to scratch.


Then -->
(http://avlmotofoto.com/cr2hdr_example.png)

I don't know what else to tell you guys. The hard part is done. Both of you have compiled the source. Maybe google 'making symbolic links in OSX' or '/usr/bin Mac' or something like that.
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: Danne on August 14, 2013, 07:47:17 AM
Here,s another one :)
I did all the steps an cleaned up also. When I hit make this is what I get?
(http://s24.postimg.org/gi68plxnp/Ska_rmavbild_2013_08_14_kl_08_04_49.png)
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: REDcineX on August 14, 2013, 08:22:34 AM
I get the same thing as Danne below. And I would imagine the problem is the in the toolchain as I mentioned before, the file says 4.7.4 (not 4.7.3). So we are calling a file that is not there. Of course, it can't find it. If I change it to 4.7.4 in the makefile.user, I get past that error, but I still do not get cr or cr2hdr as a command at the end.

Macbook-Pro-i7-26gHz-2012:~ Admin$ sudo rm /usr/bin/cr2hdr
Password:
Macbook-Pro-i7-26gHz-2012:~ Admin$ cd magic-lantern/modules/dual_iso/
Macbook-Pro-i7-26gHz-2012:dual_iso Admin$ make clean
[ RM ]  dual_iso.o dual_iso.mo *.o *.d
[ RM ]  cr2hdr
Macbook-Pro-i7-26gHz-2012:dual_iso Admin$ make
[ GCC      ]   cr2hdr
[ GCC      ]   cr2hdr
[ GCC      ]   cr2hdr
~/Downloads/gcc-arm-none-eabi-4_7-2013q2/bin/arm-none-eabi-gcc-4.7.3  -Os -mthumb-interwork -march=armv5te -D__ARM__ -Wp,-MMD,./dual_iso.d -Wp,-MT,dual_iso.o -nostdlib -fomit-frame-pointer -fno-strict-aliasing -I../../include  -Wall -W -Wno-unused-parameter -Wno-unused-function -Wno-implicit-function-declaration -Wno-missing-field-initializers -Wno-format -Wdouble-promotion -ffast-math -fno-trapping-math -fsingle-precision-constant -fshort-double -std=gnu99 -Winline -I../../platform/all -I../../platform/all/include -I../../src -I../../picoc   -DMODULE -DMODULE_NAME=dual_iso    -c -o dual_iso.o dual_iso.c
/bin/sh: /Users/Admin/Downloads/gcc-arm-none-eabi-4_7-2013q2/bin/arm-none-eabi-gcc-4.7.3: No such file or directory
make: *** [dual_iso.o] Error 127


Macbook-Pro-i7-26gHz-2012:dual_iso Admin$ make
~/Downloads/gcc-arm-none-eabi-4_7-2013q2/bin/arm-none-eabi-gcc-4.7.4  -Os -mthumb-interwork -march=armv5te -D__ARM__ -Wp,-MMD,./dual_iso.d -Wp,-MT,dual_iso.o -nostdlib -fomit-frame-pointer -fno-strict-aliasing -I../../include  -Wall -W -Wno-unused-parameter -Wno-unused-function -Wno-implicit-function-declaration -Wno-missing-field-initializers -Wno-format -Wdouble-promotion -ffast-math -fno-trapping-math -fsingle-precision-constant -fshort-double -std=gnu99 -Winline -I../../platform/all -I../../platform/all/include -I../../src -I../../picoc   -DMODULE -DMODULE_NAME=dual_iso    -c -o dual_iso.o dual_iso.c
[ MODULE   ]   dual_iso.mo
[ STRIP    ]   dual_iso.mo
[ EXPORTS  ]   dual_iso.sym
[ DEPENDS  ]   dual_iso.dep
Not checked (compile ML for these cameras first):
    100D, 1100D, 40D, 500D, 50D, 550D, 5D2, 5D3, 5DC, 600D, 60D, 650D, 6D, 700D, 7D, EOSM
make: *** [dual_iso.dep] Error 1
Macbook-Pro-i7-26gHz-2012:dual_iso Admin$ ls
Makefile   cr2hdr      cr2hdr.o   dual_iso.d   dual_iso.o   optmed.h
chdk-dng.o   cr2hdr.c   dual_iso.c   dual_iso.mo   dual_iso.sym   qsort.h
Macbook-Pro-i7-26gHz-2012:dual_iso Admin$ sudo ln -s /Users/Admin/magic-lantern/module/dual_iso/cr2hdr /usr/bin/cr2hdr
Password:
Macbook-Pro-i7-26gHz-2012:dual_iso Admin$ cd ~/Desktop
Macbook-Pro-i7-26gHz-2012:Desktop Admin$ cd untitledfolder
Macbook-Pro-i7-26gHz-2012:untitledfolder Admin$ ls
M12-1656.RAW   _MG_9518.CR2   _MG_9521.CR2   _MG_9523.CR2   _MG_9524.CR2   _MG_9526.CR2   _MG_9527.CR2
Macbook-Pro-i7-26gHz-2012:untitledfolder Admin$ cr
-bash: cr: command not found
Macbook-Pro-i7-26gHz-2012:untitledfolder Admin$ cr2hdr _MG_9524.CR2
-bash: cr2hdr: command not found
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: Danne on August 14, 2013, 08:33:28 AM
Nice to see I,m not alone ;)
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: Francis on August 14, 2013, 06:18:44 PM
You can check to see if the link is made by doing the following

Code: [Select]
Francis-Danforths-MacBook-Pro:dual_iso danforthf$ cd /usr/bin
Francis-Danforths-MacBook-Pro:bin danforthf$ ls -al
total 291648
drwxr-xr-x  1209 root   wheel       41106 Aug 14 12:05 .
drwxr-xr-x@   14 root   wheel         476 Feb  8  2012 ..
..............
-rwxr-xr-x     1 root   wheel      377872 Oct 24  2010 cpp-4.0
-rwxr-xr-x     1 root   wheel      166128 Oct 24  2010 cpp-4.2
-rwxr-xr-x     1 root   wheel        1540 Aug  1  2009 cpuwalk.d
lrwxr-xr-x     1 root   wheel          54 Aug 14 12:05 cr2hdr -> /Users/danforthf/magic-lantern/modules/dual_iso/cr2hdr
-rwxr-xr-x     1 root   wheel         807 May 19  2009 crc32
-r-xr-xr-x     1 root   wheel         816 May 19  2009 crc325.10.0
-r-xr-xr-x     1 root   wheel         815 May 19  2009 crc325.8.9
................
Francis-Danforths-MacBook-Pro:bin danforthf$

Obviosly you have to scroll a good ways up to see the line.

Next check the file /etc/paths by opening in a text editor to see if '/usr/bin' is in there. If it isn't add it as the top line. Be sure to save as plain text.
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: Danne on August 14, 2013, 08:49:09 PM
O, ll check into this when I have a little time.
Big thanks Francis for helping out.
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: REDcineX on August 14, 2013, 11:21:12 PM
Success!

Last login: Wed Aug 14 00:31:07 on ttys000
Macbook-Pro-i7-26gHz-2012:~ Admin$ cd desktop/untitledfolder
Macbook-Pro-i7-26gHz-2012:untitledfolder Admin$ ls
M12-1656.RAW   _MG_9521.CR2   _MG_9524.CR2   _MG_9527.CR2
_MG_9518.CR2   _MG_9523.CR2   _MG_9526.CR2
Macbook-Pro-i7-26gHz-2012:untitledfolder Admin$ cr2hdr _MG_9523.CR2

Input file     : _MG_9523.CR2
Full size      : 5792 x 3804
Active area    : 5634 x 3753
White level    : 15834
Black borders  : 158 left, 51 top
Black level    : 972
ISO pattern    : BddB GBRG
Estimating ISO difference...
ISO difference : 4.21 EV (1847)
Black delta    : -6.81
Interpolation  : mean23-chroma5x5-contrast
Matching brightness...
Looking for hot pixels...
Hot pixels     : 1438
Full-res reconstruction...
Alias filtering...
Dark chroma filtering...
Building contrast map...
Dilating contrast map...
Smoothing contrast map...
ISO overlap    : 4.8 EV (approx)
Black adjust   : -3
Output file    : _MG_9523.DNG
    1 image files updated


My problem was this:

lrwxr-xr-x     1 root   wheel        49 Aug 13 23:07 cr2hdr -> /Users/Admin/magic-lantern/module/dual_iso/cr2hdr

When I ran the ls -al I saw the folder said "module" not "modules"
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: Danne on August 15, 2013, 12:44:11 AM
Congrats! I, ll check this out more thoroughly later.
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: Francis on August 15, 2013, 05:52:49 AM
I hope it was worth it.  :)
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: jerrykil on October 09, 2013, 07:03:36 AM
I hope it was worth it.  :)
very worth it, thanks
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: dfort on March 30, 2014, 07:50:15 PM
When trying to post a question here I got:

"Warning: this topic has not been posted in for at least 120 days.
Unless you're sure you want to reply, please consider starting a new topic."

Not sure if starting a new topic is appropriate so I'll just ask my question.

I followed the instructions for compiling on Mac and it tripped up looking for python.

makefile.user has: PYTHON=python2

For Mac shouldn't it be: PYTHON=python

I made that change and it got a little further but it didn't finish:

[ CC       ]   raw_rec.o
[ MODULE   ]   raw_rec.mo
[ STRIP    ]   raw_rec.mo
[ EXPORTS  ]   raw_rec.sym
00001960 raw_movie_filename
00001964 raw_video_enabled
[ DEPENDS  ]   raw_rec.dep
Not checked (compile ML for these cameras first):
    1100D, 500D, 50D, 550D, 5D2, 5D3, 600D, 60D, 650D, 6D, 700D, 7D, EOSM
make: *** [raw_rec.dep] Error 1
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: dmilligan on March 30, 2014, 08:57:37 PM
It's failing running a python script that checks the dependencies for the modules. I had it set to python2 so that it would use version 2 of python which is apparently the version that works correctly. What happens if you put the PYTHON=python2 back in?
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: dfort on March 31, 2014, 01:53:58 AM
Thanks for the tip dmilligan. This is what happens:

[ DEPENDS  ]   raw_rec.dep
/bin/sh: python2: command not found
make: *** [raw_rec.dep] Error 127

This is a new MacBook Pro running 10.9.2. I think I followed the instructions to install MacPorts and Xcode properly. According to whereis, python is in /usr/bin/python but returns null for python2.

Edit: Looks like this MacBook has python 2.6 and 2.7, the default being 2.7.5. I tried changing python2 to python2.6 and python2.7 and didn't get the command not found error but it always winds up with:

make: *** [raw_rec.dep] Error 1
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: dmilligan on March 31, 2014, 01:24:38 PM
IIRC python2 was on my system by default, though I upgraded to 10.9 from earlier versions, Apple may well be including different versions of things like this by default now, IDK. maybe you can try downloading some different 'official' versions of python and try them (use fink or brew or whatever) rather than the Apple flavored version. I'm not at my mac right now so I can't check what the actual version of python is working and where it came from, but I'll reply again when I am.
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: dfort on March 31, 2014, 06:35:08 PM
No rush. I've got a busy week and can't get back to this until the weekend. So this error is caused by python?

Quote from: dfort
[ DEPENDS  ]   raw_rec.dep
Not checked (compile ML for these cameras first):
    1100D, 500D, 50D, 550D, 5D2, 5D3, 600D, 60D, 650D, 6D, 700D, 7D, EOSM
make: *** [raw_rec.dep] Error 1
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: dmilligan on March 31, 2014, 07:00:35 PM
it's either that or you didn't compile ML itself first before trying to compile whatever module this is, I'm assuming you did that right?
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: dfort on April 01, 2014, 02:46:46 AM
I was just following the instructions at the top of this topic:

Quote
In terminal, go to your source ./magic-lantern/modules/raw_rec

Type make. You will get some warnings about not having compiled the source for any model but these instructions are purely for getting the utilities to compile. You'll end up with our first utility, raw2dng, amongst other things like the module files.

Maybe everything went fine? Guess I'll have to wait until Friday night when I get home to find out.
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: Audionut on April 01, 2014, 03:26:14 AM
I updated the OP. 

You must build the autoexec.bin for your camera first, before compiling modules.
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: dfort on April 07, 2014, 05:31:46 AM
Thanks--inching a little closer. I noticed the change in the original post:

Quote
First, you must compile the autoexec.bin and associated .sym file for your camera.

I wasn't sure how to do this so I tried compile autoexec.bin for the EOSM this way.

1st try:
Code: [Select]
$ make EOSM
make -C  /Users/rosiefort/magic-lantern/platform/EOSM.202
[ VERSION  ]   ../../platform/EOSM.202/version.bin
make[1]: truncate: No such file or directory
make[1]: *** [../../platform/EOSM.202/version.bin] Error 1
make: *** [EOSM] Error 2

Ah ha--looks like truncate is missing. This should probably be added to the OP:
Code: [Select]
sudo port install truncate
This is what happened next:
Code: [Select]
$ make EOSM
make -C  /Users/rosiefort/magic-lantern/platform/EOSM.202
[ VERSION  ]   ../../platform/EOSM.202/version.bin
truncate: illegal option -- s
Internal error!
make[1]: *** [../../platform/EOSM.202/version.bin] Error 1
make: *** [EOSM] Error 2

Now I'm stuck again. I tried to build other platforms listed in the Makefile with the same results. I searched the wiki and forums for a fix but ran out of time for this weekend. I'm taking off tomorrow for another week of work away from my computer. No rush, but if anyone could give me a hint at what to try next I'd really appreciate it.

BTW--do I really have to compile ML before compiling raw2dng? My goal is simply to get raw2dng_cs2x2 running on a Mac to deal with the pink dots in EOS-M raw footage. Thanks!
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: Audionut on April 07, 2014, 05:39:28 AM
http://www.magiclantern.fm/forum/index.php?topic=7936.msg78404#msg78404
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: dfort on April 07, 2014, 06:02:47 AM
Thanks Audionut for pointing that out but I think I figured it out. The heading on this post is a bit misleading:
Quote
Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]

Correct me if I'm wrong but the way to compile raw2dng is to go to the modules/lv_rec directory and do this:
Code: [Select]
$ make raw2dng
[ GCC      ]   raw2dng
[ GCC      ]   raw2dng
raw2dng.c:70:103: warning: format specifies type 'int' but the argument has type
      'unsigned long' [-Wformat]
  ...= %d, should be 192", sizeof(lv_rec_file_footer_t));
       ~~                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
       %lu
raw2dng.c:39:77: note: expanded from macro 'FAIL'
#define FAIL(fmt,...) { fprintf(stderr, "Error: "); fprintf(stderr, fmt, ## __VA_ARGS__)...
                                                                            ^
1 warning generated.
[ GCC      ]   raw2dng

This produced a raw2dng executable. Now I need to figure out how to switch on the options to make this behave like raw2dng_cs2x2.
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: kichetof on April 07, 2014, 09:33:11 PM
There're some problems with the latest update of Xcode 5.1

When I compile cr2hdr I encounter this error:

Code: [Select]
make cr2hdr
REBUILDING
[ README   ]   module_strings.h
[ gcc      ]   cr2hdr
clang: error: unknown argument: '-mno-ms-bitfields' [-Wunused-command-line-argument-hard-error-in-future]
clang: note: this will be a hard error (cannot be downgraded to a warning) in the future
make: *** [cr2hdr] Error 1

If I disable -mno-ms-bitfields on Makefile.cr2hdr, it works fine!

Quote
CR2HDR_CFLAGS=-m32 -mno-ms-bitfields -O2 -Wall -I$(SRC_DIR) -D_FILE_OFFSET_BITS=64 -fno-strict-aliasing

I tried to bypass this error to set CFLAGS with CFLAGS=-Wunused-command-line-argument-hard-error-in-future and CFLAGS=Wno-error=-Wunused-command-line-argument-hard-error-in-future but nothing to do. Only remove the flag works for the moment

Some infos here (http://stackoverflow.com/a/22355874), here (http://forums.getpebble.com/discussion/comment/98335/#Comment_98335) and here (https://twistedmatrix.com/trac/ticket/6974)
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: dfort on April 11, 2014, 04:39:06 PM
Success--I got raw2dng compiled and working on my Mac. To get rid of the pink and/or green focus dots it was simply a matter of uncommenting this line.

Code: [Select]
Before:
//~ #define CHROMA_SMOOTH
After:
#define CHROMA_SMOOTH

Developers--excuse me if I'm making this very elementary but I'd like to make this clear to non-programers who have cameras with pink dot issues and are on Macs.

The is my output of "make raw2dng"
Code: [Select]
$ make raw2dng
[ GCC      ]   raw2dng
[ GCC      ]   raw2dng
raw2dng.c:70:103: warning: format specifies type 'int' but the argument has type
      'unsigned long' [-Wformat]
  ...= %d, should be 192", sizeof(lv_rec_file_footer_t));
       ~~                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
       %lu
raw2dng.c:39:77: note: expanded from macro 'FAIL'
  ...{ fprintf(stderr, "Error: "); fprintf(stderr, fmt, ## __VA_ARGS__); fpri...
                                                           ^
raw2dng.c:622:13: warning: unused function 'chroma_smooth_3x3'
      [-Wunused-function]
static void chroma_smooth_3x3(unsigned short * inp, unsigned short * out...
            ^
raw2dng.c:957:13: warning: unused function 'chroma_smooth_5x5'
      [-Wunused-function]
static void chroma_smooth_5x5(unsigned short * inp, unsigned short * out...
            ^
3 warnings generated.
[ GCC      ]   raw2dng

Note that this creates a program that must be run from the command line, drag and drop doesn't work, so you'll need to read up on how to use the command line. Basically, using the Terminal app, navigate into the directory you want to save the .dng frames then--

Code: [Select]
[PATH_TO_raw2dng] [PATH_TO_*.RAW_FILES]
Of course you can put the compiled raw2dng program in a location that can be found in your "PATH" but setting it up depends on your environment and which shell you're using.

For any one else that wants to compile raw2dng on a Mac--please note that the instructions in the OP (original post) won't get you there and developers pointing to other posts are only going to lead you further down the rabbit hole. All that you need is in the magic-lantern/modules/lv-rec directory of the source code. You will need to get the gcc compiler running on your Mac but that can be as simple as installing Apple's Xcode from the app store. It is a free download. You'll also need to install mercurial and dcraw, MacPorts make that pretty easy. What you don't need to do is set up a complete ARM cross-compiling setup if you just want to get raw2dng working.

It would be great if the pre-compiled GUI raw to dng Mac apps could have an option to turn on CHROMA_SMOOTH for users who aren't able to compile ML.
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: Hastyr on April 21, 2014, 07:06:47 PM
I'm working to compile cr2hdr myself - even though I found a MacOS binary online.

It looks like I worked past the following error, though I still cannot build cr2hdr so this workaround may be not be correct.

truncate: illegal option -- s
Internal error!

the problem appears to be in: magic-lanter/src/Makefile.src
two lines have the following:
Code: [Select]
@truncate -s 284 $@and:
Code: [Select]
truncate -s 0 $@ ; \
the macports truncate command doesn't support the -s switch.  So I changed the lines to read:
 
Code: [Select]
@truncate $@ 284and
Code: [Select]
truncate $@ 0 ; \respectively.

now building the magic lantern build dies in 7D_master.203
Code: [Select]
_log truncated to save space_
make -C  /Users/foo/tmptmp/magic-lantern/platform/7D_MASTER.203
[ VERSION  ]   ../../platform/7D_MASTER.203/version.bin
[ CPP      ]   magiclantern.lds
[ AS       ]   entry.o
[ CC       ]   master_boot.o
../../platform/7D_MASTER.203/master_boot.c: In function 'master_callback':
../../platform/7D_MASTER.203/master_boot.c:161:13: error: implicit declaration of function 'ptpbuf_add' [-Werror=implicit-function-declaration]
             ptpbuf_add(0x4005, callstack, strlen(callstack));
             ^
../../platform/7D_MASTER.203/master_boot.c:177:27: warning: initialization makes pointer from integer without a cast [enabled by default]
         uint8_t *buffer = bkpt->ctx[0];
                           ^
cc1: some warnings being treated as errors
make[1]: *** [master_boot.o] Error 1
make: *** [7D_MASTER] Error 2

which I'll work on as I get time...
some cameras do get built.  If I try a make in dual_iso i get the above error

'-mno-ms-bitfields'

disabling it causes a new set of errors unfortunately. 
Code: [Select]
make
[ gcc      ]   cr2hdr
cr2hdr.c:344:34: error: use of undeclared identifier '__module_strings_MODULE_NAME'
    module_strpair_t *strings = &__module_strings_MODULE_NAME[0];
                                 ^
cr2hdr.c:1208:51: warning: if statement has empty body [-Wempty-body]
        if(system("octave --persist iso-curve.m"));
                                                  ^
cr2hdr.c:1208:51: note: put the semicolon on a separate line to silence this warning
cr2hdr.c:1587:55: warning: if statement has empty body [-Wempty-body]
        if(system("octave --persist fullres-curve.m"));
                                                      ^
cr2hdr.c:1587:55: note: put the semicolon on a separate line to silence this warning
cr2hdr.c:1919:56: warning: if statement has empty body [-Wempty-body]
            if(system("dcraw -d -r 1 1 1 1 edges.dng"));
                                                       ^
cr2hdr.c:1919:56: note: put the semicolon on a separate line to silence this warning
cr2hdr.c:1933:59: warning: if statement has empty body [-Wempty-body]
            if(system("dcraw -d -r 1 1 1 1 edge-map.dng"));
                                                          ^
cr2hdr.c:1933:59: note: put the semicolon on a separate line to silence this warning
cr2hdr.c:2264:51: warning: if statement has empty body [-Wempty-body]
        if(system("octave --persist mix-curve.m"));
                                                  ^
cr2hdr.c:2264:51: note: put the semicolon on a separate line to silence this warning
5 warnings and 1 error generated.
exiftool-bridge.c:58:9: warning: variable 'mode' is used uninitialized whenever 'if' condition
      is false [-Wsometimes-uninitialized]
    if (exif_file)
        ^~~~~~~~~
exiftool-bridge.c:70:9: note: uninitialized use occurs here
    if (mode != 0)
        ^~~~
exiftool-bridge.c:58:5: note: remove the 'if' if its condition is always true
    if (exif_file)
    ^~~~~~~~~~~~~~
exiftool-bridge.c:51:13: note: initialize the variable 'mode' to silence this warning
    int mode;
            ^
             = 0
1 warning generated.
make: *** [cr2hdr] Error 1
and no cr2hdr binary.

Next stop is a search of the forums again.  Any timesaving feedback would be appreciated.

Thanks,
Gus
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: Hastyr on April 21, 2014, 11:48:30 PM

and no cr2hdr binary.

Next stop is a search of the forums again.  Any timesaving feedback would be appreciated.

A bit more success!

I found that the following change prevents compiling cr2hdr on MacOS in my configuration..
Code: [Select]
https://bitbucket.org/hudson/magic-lantern/commits/5b78fb409ac81d2bec6b9fecdd352af26d3b27a5
I commented it completely out (the change is self consistent) and despite a slew of errors I am able to develop a binary.  When I use it to post process my photos it work but not as expected. The colors are "off".

Still this is still a work in progress, but getting closer

Thanks,
Gus

Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: kichetof on April 26, 2014, 12:28:49 PM
On my side I have this problem since Xcode update to compile cr2hdr.

I need to remove -mno-ms-bitfields to be able to compile :(
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: philmoz on June 14, 2014, 01:06:08 AM
On my side I have this problem since Xcode update to compile cr2hdr.

I need to remove -mno-ms-bitfields to be able to compile :(

Or you can install GCC 4.8 or 4.9 (using homebrew or macports).
http://www.ficksworkshop.com/blog/14-coding/65-installing-gcc-on-mac

This will also allow you to build the new cr2hdr-20bit version of cr2hdr - XCode does not support nested functions.

Phil.
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: jamespero on July 03, 2014, 06:00:41 PM
Hey everybody,

I'm new to ML and somewhat new to C programming. I'm trying to get up to speed with ML development so that I can try and get development for the SL1/100D moving a little faster. I'm currently dealing with an error while trying to compile:

Code: [Select]
make -C  /Users/James/documents/repositories/magic-lantern/platform/100D.100
[ VERSION  ]   ../../platform/100D.100/version.bin
[ VERSION  ]   ../../platform/100D.100/version.c
[ CC       ]   version.o
[ CC       ]   boot-hack.o
../../src/boot-hack.c: In function 'my_task_dispatch_hook':
../../src/boot-hack.c:302:5: error: implicit declaration of function 'qprintf' [-Werror=implicit-function-declaration]
     qprintf("dispatch context->pc=0x%08X entry=0x%08X\n", context->pc, task->entry);
     ^
cc1: some warnings being treated as errors
make[1]: *** [boot-hack.o] Error 1
make: *** [100D] Error 2

Based on my experience with programming in php and javascript and a little with arduino, I'm guessing qprintf is not exactly a required element in getting ML to compile but I'm not familiar with qprintf specifically unlike printf and sprintf. Can someone tell me if this is something I need to worry about or how to get this error resolved?

[EDIT]

I'm thinking there's an include that missing from boot-hack.c like stdio.h or the like, but again I'm new to ML and mostly new to C so I'm not entirely sure.
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: dmilligan on July 04, 2014, 04:11:20 AM
it's for QEMU, what branch are you trying to compile? did you do a make clean?
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: jamespero on July 08, 2014, 03:14:04 AM
it's for QEMU, what branch are you trying to compile? did you do a make clean?

I'm trying to compile count's 100D wrk-qemu branch - https://bitbucket.org/berlincount/magic-lantern/commits/all

I've got qemu installed via homebrew, but I think I'm still missing something. I've run a make clean and that seems to complete without error.

I guess at this point I need a little guidance with qemu to make sure I have it installed and configured... :/
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: dfort on June 27, 2015, 10:13:11 PM
I've been having a PM exchange with Danne over compiling raw2dng and mlv_dump command line tools on the Mac and thought I'd share our success and failures with others on the forum. This seemed to be the appropriate forum topic to post this.

Apparently raw2dng, mlv_dump and cr2hdr can be compiled without having to install a cross platform ARM environment in order to build ML.

Starting with a virgin Mac, here's how to compile raw2dng:

1. Install Xcode. This is fairly easy because it is an Apple app.

2. Install the command line tools. This is a part of Xcode but apparently it isn't installed automatically. MacPorts has the instructions on how to get the command line tools working: https://www.macports.org/

3. Install MacPorts. Again, the MacPorts instructions are fairly easy to follow.

4. Install mercurial and dcraw via MacPorts. This is really easy:

Code: [Select]
sudo port install mercurial
sudo port install dcraw

5. Now using the terminal go into a directory where you would like to put the Magic Lantern source code and run this:

Code: [Select]
hg clone -r unified https://bitbucket.org/hudson/magic-lantern
Note that there are lots of other ways to do this that will get you the same results. This works too:

Code: [Select]
hg clone https://bitbucket.org/hudson/magic-lantern
cd magic-lantern/
hg update unified

6. cd into the directory where raw2dng.c and the Makefile for it is located:

Code: [Select]
cd [PATH TO YOUR WORKING DIRECTORY]/magic-lantern/modules/lv_rec
7. Now simply type this in the terminal.

Code: [Select]
make raw2dng
You will get some warning messages but don't worry, in that directory you will find the compiled raw2dng command line tool. It is easy to spot because it doesn't have a file extension.

By the way, what I wanted was a version of raw2dng with chroma smoothing turned on in order to take care of the pink focus pixels on EOS-M raw video. This was very simple to do because it is spelled out very clearly in the raw2dng.c code starting on line 33:

Code: [Select]
/* useful to clean pink dots, may also help with color aliasing, but it's best turned off if you don't have these problems */
//~ #define CHROMA_SMOOTH

I just removed the "//~" and it worked!

Next up, mlv_dump:

Trying to figure this one out on the forum topic was confusing because people were trying all sorts of things but the issue seems to be that the code was written on Linux systems and Apple Xcode sets up the developing environment differently. Long story short:

Code: [Select]
sudo port install gcc47 +universal; sudo port select --set gcc mp-gcc47; hash gcc
It takes a long time to run this, about a couple of hours on my system, so patience! Once it finishes you should be able to compile mlv_dump as well as raw2dng without having to make any source code tweaks.

Finally, cr2hdr:

Running "make cr2hdr" in the magic-lantern/modules/dual_iso directory gave me this error:

Code: [Select]
/bin/sh: rst2html: command not found
It looks like the Mac doesn't have rst2html installed. From what I found out that is a part of docutils but there is no docutils in MacPorts. I tried installing py-docutils. That didn't get me rst2html but it did install /opt/local/bin/rst2html-2.7.py so I made a soft link to it like this:

Code: [Select]
sudo ln -s /opt/local/bin/rst2html-2.7.py /opt/local/bin/rst2html
That did it--whew!

Hope this helps other ML users who want these command line tools running on their Macs. During my many attempts at compiling these tools I ended up installing some unnecessary components--hope I didn't leave out any necessary ones! Maybe one of them is truncate? If so, it can also be installed through MacPorts.
Title: Re: Compiling and setting up ML command line tools for Macs [cr2hdr and raw2dng]
Post by: Danne on June 27, 2015, 11:18:50 PM
Very helpful stuff. Wouldn,t be anywhere near this stuff  if not for these instructions.
You got inte cr2hdr as well. Awesome. There is one command line regarding linear conversion when choosing adobe dng conversion I want to check out. Finally possible.
Good work.