MinGw for Windows command line tools

[dfort]

I set up a MinGw environment on a Windows system just to compile the command line tools so there shouldn't be a need to set up Virtual Box. There was no problem building mlv_dump but I'm having problems with cr2hdr and raw2dng. Any clues what's going on here? One issue might be that this is a 32-bit system and maybe these tools need a 64-bit compiler?

raw2dng:

Code Select Expand

[ HOST_CC  ]   raw2dng
[ HOST_CC  ]   raw2dng
raw2dng.c: In function 'main':
raw2dng.c:107:5: warning: unknown conversion type character 'z' in format [-Wformat=]
     if (sizeof(lv_rec_file_footer_t) != 192) FAIL("sizeof(lv_rec_file_footer_t) = %zu, should be 192", sizeof(lv_rec_file_footer_t));
     ^
raw2dng.c:107:5: warning: too many arguments for format [-Wformat-extra-args]
raw2dng.c:109:5: warning: implicit declaration of function 'fseeko' [-Wimplicit-function-declaration]
     fseeko(fi, -192, SEEK_END);
     ^
[ HOST_CC  ]   raw2dng
raw2dng.o:raw2dng.c:(.text.startup+0x62): undefined reference to `fseeko'
collect2.exe: error: ld returned 1 exit status
make: *** [raw2dng] Error 1


cr2hdr:

Code Select Expand

Updated HGVERSION
[ README   ]   module_strings.h
LC_TIME=EN hg log . -r 'reverse(ancestors(.))' -l 1 --template '{date|hgdate}
{node|short}
{author|user}
{desc|strip|firstline}'

'LC_TIME' is not recognized as an internal or external command,

operable program or batch file.


(<type 'exceptions.SystemExit'>, SystemExit(1,), <traceback object at 0x024A2710>)
make: *** [module_strings.h] Error 1


[dmilligan]

You might have better luck cross-compiling from linux.

[Licaon_Kter]

What shell do you use? Bash or something from Cygwin, right? Hope you're not using Windows provided cmd.

You've also installed mingw64-x86_64-headers right?

[dfort]

Let me take this one OS at a time so it doesn't get confusing.

You might have better luck cross-compiling from linux.

Installing a virtual Linux machine on Windows then running MinGw in that seemed a little convoluted to me that's why I tried MinGw in Windows. All I wanted to build were a few of the command line tools so I figured that would be easier than cross compiling. It was pretty easy to do in Cygwin but the issue with Cygwin is that you need the .dll files to travel with the .exe files.

So I took dmilligan's advice and at least for raw2dng it was almost too easy. Running Ubuntu I installed the MinGw base package (gcc-mingw-w64-base) and changed the Makefile like this:

Code Select Expand

# MINGW=~/mingw-w32/bin/i684-w64-mingw32-gcc
MINGW=/usr/bin/i686-linux-gnu-gcc

And it worked. Figuring that maybe ML looks in that location for all the command line tools I set up the directory structure I made a symbolic link:

Code Select Expand

ln -s /usr/bin/i686-linux-gnu-gcc ~/mingw-w32bin/i686-w64-mingw32-gcc

So now it works without having to modify the Makefile. Oh joy. Next I tried cr2hdr. Woo hoo, compiled right out of the box. Finally, mlv_dump which is the only one that I was able to build on Windows. This started out a bit trickier because it wasn't looking for the compiler in the same path as the other tools. Instead it was looking in the path for i686-w64-min32-gcc which is not the way it is named in the Ubuntu installation. So another symbolic link:

Code Select Expand

ln -s /usr/bin/i686-linux-gnu-gcc /usr/bin/i686-w64-mingw32-gcc

[EDIT: better yet change the compiler name in Makefile.user.default]

However, no joy this time.

Code Select Expand

dan@dan-laptop:~/Desktop/magic-lantern/modules/mlv_rec$ make mlv_dump.exe
[ MINGW_GCC ]   mlv_dump.w32.o
[ MINGW_GCC ]   ../lv_rec/raw2dng.w32.o
[ MINGW_GCC ]   lzma/Threads.w32.o
lzma/Threads.c:5:21: fatal error: process.h: No such file or directory
#include <process.h>
                     ^
compilation terminated.
make: *** [lzma/Threads.w32.o] Error 1

Looking in Threads.c there is:

Code Select Expand

/* Threads.c -- multithreading library
2009-09-20 : Igor Pavlov : Public domain */

#ifndef _WIN32_WCE
#include <process.h>
#endif

#include "Threads.h"

I'm stumped. Is there a header file missing?

[dfort]

What shell do you use? Bash or something from Cygwin, right? Hope you're not using Windows provided cmd.

You've also installed mingw64-x86_64-headers right?

The headers were installed automagically by the MinGw installer.

I was using the Windows Command Prompt. Which is your preferred bash shell for Windows? I didn't know the Cygwin shell was an option, it works in a jail/sandbox/chroot/whatever in the Cygwin environment.

The big news is that I finally got a MinGw cross compiler working--on a Mac! There are several scripts available that will download and build GNU tools but this is the one that got me the same compiler version that is hard coded in the ML source:

https://ffmpeg.zeranoe.com/blog/?p=414#more-414

It is way overkill considering the only files I needed so far were i686-w64-mingw32-gcc and i686-w64-mingw32-ar. I used the build options from the example:

Code Select Expand

bash ./mingw-w64-build-3.6.7 --build-type=multi --default-configure --disable-shared --enable-gendef

[Licaon_Kter]

I was using the Windows Command Prompt. Which is your preferred bash shell for Windows? I didn't know the Cygwin shell was an option, it works in a jail/sandbox/chroot/whatever in the Cygwin environment.

It works meaning it finds apps, but it does not handle unix tricks, like the LC error, since you use Cygwin it's better to use bash/zsh/whatever unix shell.

As far as Cygwin goes I recommend https://babun.github.io ( zsh shell, pact install <cygwin package> and so on )

[dfort]

As far as Cygwin goes I recommend https://babun.github.io ( zsh shell, pact install <cygwin package> and so on )

I tried out babun and it worked great but it has the same issues as Cygwin. Actually, I think it is pretty much the same as Cygwin because the cr2hdr.exe it created needs cygwin1.dll and cyggcc_s-1.dll in order to work. I think that the MinGw cross compile method uses static libraries but I'm not sure. Maybe it links to the standard Microsoft libraries? All I know is that this is quite a puzzle to solve.

[dfort]

By the way, the right way to use MinGw on Windows is through their MSYS which is a Unix like environment complete with a bash shell, etc. I actually was able to compile ML on a Windows system with it.

However, I'm still having problems getting the command line tools to compile. Going back through my notes it looks like I'm back to where I started with cr2hdr:

Code Select Expand

Updated HGVERSION
[ README   ]   module_strings.h
LC_TIME=EN hg log . -r 'reverse(ancestors(.))' -l 1 --template '{date|hgdate}
{node|short}
{author|user}
{desc|strip|firstline}'

'LC_TIME' is not recognized as an internal or external command,

operable program or batch file.


(<type 'exceptions.SystemExit'>, SystemExit(1,), <traceback object at 0x024A2710>)
make: *** [module_strings.h] Error 1

[Licaon_Kter]

Better comment out the mercurial check in readme2modulestrings.py maybe.

[dfort]

I found the line in readme2modulestrings.py and commented it out but then I got a slew of warnings and error messages.

[EDIT -- Uh, isn't the backslash "\" an escape character in Windows? Maybe that's the problem? Or maybe it is not interpreting the "\n" properly and seeing them as newline characters?]

Code Select Expand

Updated HGVERSION
[ README   ]   module_strings.h
[ gcc.exe  ]   cr2hdr
In file included from cr2hdr.c:60:0:
module_strings.h:13:5: warning: missing terminating " character [enabled by default]
     "Increases dynamic range by sampling the sensor at two different
\n"
     ^
module_strings.h:13:5: error: missing terminating " character
module_strings.h:14:1: error: stray '\' in program
     "ISOs, switching ISO for every other line pair. This trick cleans up
\n"
^
module_strings.h:14:3: warning: missing terminating " character [enabled by default]
     "ISOs, switching ISO for every other line pair. This trick cleans up
\n"
   ^
module_strings.h:14:1: error: missing terminating " character
     "ISOs, switching ISO for every other line pair. This trick cleans up
\n"
^
module_strings.h:15:5: warning: missing terminating " character [enabled by default]
     "shadow noise, resulting in a dynamic range improvement of
\n"
     ^
module_strings.h:15:5: error: missing terminating " character
module_strings.h:16:1: error: stray '\' in program
     "around 3 stops, at the cost of reduced vertical resolution,
\n"
^
module_strings.h:14:2: error: 'n' undeclared here (not in a function)
     "ISOs, switching ISO for every other line pair. This trick cleans up
\n"
  ^
module_strings.h:16:2: error: expected ',' or ';' before 'n'
     "around 3 stops, at the cost of reduced vertical resolution,
\n"
  ^

-----------cut out a big section because it is way too long to post-------------------

module_strings.h:85:5: error: missing terminating " character
module_strings.h:86:1: error: stray '\' in program
     MODULE_STRING(__module_string_i_name, __module_string_i_value)
^
module_strings.h:86:3: warning: missing terminating " character [enabled by default]
     MODULE_STRING(__module_string_i_name, __module_string_i_value)
   ^
module_strings.h:86:1: error: missing terminating " character
     MODULE_STRING(__module_string_i_name, __module_string_i_value)
^
module_strings.h:87:5: warning: missing terminating " character [enabled by default]
     MODULE_STRING(__module_string_j_name, __module_string_j_value)
     ^
module_strings.h:87:5: error: missing terminating " character
module_strings.h:88:1: error: stray '\' in program
     MODULE_STRING(__module_string_k_name, __module_string_k_value)
^
module_strings.h:88:3: warning: missing terminating " character [enabled by default]
     MODULE_STRING(__module_string_k_name, __module_string_k_value)
   ^
module_strings.h:88:1: error: missing terminating " character
     MODULE_STRING(__module_string_k_name, __module_string_k_value)
^
module_strings.h:89:5: warning: missing terminating " character [enabled by default]
MODULE_STRINGS_END()
     ^
module_strings.h:89:5: error: missing terminating " character
module_strings.h:90:1: error: stray '\' in program
module_strings.h:90:3: warning: missing terminating " character [enabled by default]
module_strings.h:90:1: error: missing terminating " character
module_strings.h:91:5: warning: missing terminating " character [enabled by default]
module_strings.h:91:5: error: missing terminating " character
module_strings.h:92:1: error: stray '\' in program
module_strings.h:92:3: warning: missing terminating " character [enabled by default]
module_strings.h:92:1: error: missing terminating " character
module_strings.h:93:5: warning: missing terminating " character [enabled by default]
module_strings.h:93:5: error: missing terminating " character
module_strings.h:94:1: error: stray '\' in program
module_strings.h:94:3: warning: missing terminating " character [enabled by default]
module_strings.h:94:1: error: missing terminating " character
module_strings.h:95:5: warning: missing terminating " character [enabled by default]
module_strings.h:95:5: error: missing terminating " character
module_strings.h:96:1: error: stray '\' in program
module_strings.h:96:3: warning: missing terminating " character [enabled by default]
module_strings.h:96:1: error: missing terminating " character
module_strings.h:97:5: warning: missing terminating " character [enabled by default]
module_strings.h:97:5: error: missing terminating " character
module_strings.h:98:1: error: stray '\' in program
module_strings.h:98:3: warning: missing terminating " character [enabled by default]
module_strings.h:98:1: error: missing terminating " character
module_strings.h:99:5: warning: missing terminating " character [enabled by default]
module_strings.h:99:5: error: missing terminating " character
module_strings.h:100:1: error: stray '\' in program
module_strings.h:100:3: warning: missing terminating " character [enabled by default]
module_strings.h:100:1: error: missing terminating " character
module_strings.h:112:5: warning: missing terminating " character [enabled by default]
module_strings.h:112:5: error: missing terminating " character
module_strings.h:113:1: error: stray '\' in program
module_strings.h:113:3: warning: missing terminating " character [enabled by default]
module_strings.h:113:1: error: missing terminating " character
module_strings.h:115:1: error: invalid initializer
module_strings.h:114:5: error: expected ',' or ';' before string constant
make: *** [cr2hdr] Error 1


The thing is that if I run the command in the line that I commented out in bash:

Code Select Expand

LC_TIME=EN hg log . -r 'reverse(ancestors(.))' -l 1 --template '{date|hgdate}\n{node|short}\n{author|user}\n{desc|strip|firstline}'

The output seems fine:

Code Select Expand

1439942737 14400
1f35f64464e0
david
Merge ml_dng and cr2hdr-20bit

Hard coding this into readme2modulestrings.py

Code Select Expand

#last_change_info = run("LC_TIME=EN hg log . -r 'reverse(ancestors(.))' -l 1 --template '{date|hgdate}\n{node|short}\n{author|user}\n{desc|strip|firstline}'")
last_change_info = """1439942737 14400
1f35f64464e0
david
Merge ml_dng and cr2hdr-20bit"""


results in the same error message as above.

Something else that I tried was to replace Mercurial. I was using the Windows install from the Mercurial site so I tried one that is installed in the MinGW environment and got the same results.

If anyone is interested in installing Mercurial in MinGW it is almost as simple as:

Code Select Expand

pip install Mercurial

Almost because it needs the Microsoft Visual C++ Compiler for Python 2.7. Fortunately it is a free download.

[dfort]

Sorry, can't just leave this alone. I played around with trying to compile cr2dng with a MinGW/MSYS setup on Windows and I did it, but with a crazy workaround.

I was chasing my tail at first because the things that I tried seemed to work, then not work. The reason is because on the first "make cr2hdr" this error comes up:

Code Select Expand

Updated HGVERSION
[ README   ]   module_strings.h
LC_TIME=EN hg log . -r 'reverse(ancestors(.))' -l 1 --template '{date|hgdate}
{node|short}
{author|user}
{desc|strip|firstline}'

'LC_TIME' is not recognized as an internal or external command,

operable program or batch file.


(<type 'exceptions.SystemExit'>, SystemExit(1,), <traceback object at 0x024A2710>)
make: *** [module_strings.h] Error 1

Ok--seen that before but it does create an "hgstamp" file that looks good and a "module_strings.h" file that is formatted improperly. Here's a snippet of it:

Code Select Expand

static char __module_string_f_value[] MODULE_STRINGS_SECTION =
    "Increases dynamic range by sampling the sensor at two different
\n"
    "ISOs, switching ISO for every other line pair. This trick cleans up
\n"


Running "make cr2hdr" again without doing a "make clean" will result in the long error message as it tries to parse the "module_strings.h" file.

My workaround was to run "make cr2hdr" so it will create "hgstamp" then replace the improperly formatted "module_strings.h" file with a properly formatted version from another system, then edit "Makefile.cr2hdr" removing "$(MODULE_STRINGS)" and finally running "make cr2hdr" again. This resulted in a working cr2hdr.exe binary.

I'm curious into what is going on here. Is it the MinGW/MSYS environment to blame or is there a portability issue in the code? It compiles fine in Cygwin, Linux, Mac and I've got it cross compiling Mac to Windows using the MinGW gcc compiler so there's plenty of options. However, it would be great to get this working properly in the MinGW/MSYS environment because many people are on Windows and this doesn't require setting up virtual machines or creating binaries that require cygwin .dll files.

[dfort]

The other command line tool that isn't building in the MinGW/MSYS Windows environment is raw2dng. Searching for others with this problem the solution turned out to be pretty simple. Adding this to raw2dng.c in the definitions area:

Code Select Expand

#define fseeko fseek // needed for MinGW fseeko64 also works

Like the comment says, fseeko64 also works. Maybe substituting one of these for this line?

Code Select Expand

fseeko(fi, -192, SEEK_END);

What would be the issue if either fseek or fseeko64 is used instead of fseeko? Just wondering.

[EDIT: Just found this in mlv_dump.c so maybe someone else ran into this issue?]

Code Select Expand

#if defined(__WIN32)
    return fseeko64(stream, offset, whence);
#else
    return fseeko(stream, offset, whence);
#endif

[EDIT2: Tried it out and it works so I submitted a pull request--lv_rec: use fseeko/fseeko64 depending on platform as in mlv_dump.c.]

[dfort]

I seem to be having quite a conversation with myself on this topic. Guess that's what spending too much time trying to figure out ML source code does to you.

So--if anyone besides myself is interested, I did manage to work out the issues with building the command line tools. In fact an unexpected side benefit was being able to compile ML on a Windows system without requiring a virtual machine running Linux. Don't rejoice just yet, the autoexec.bin files I've created are not working yet but that's an issue for a different topic.

Here is the pull request for modifying readme2modulestrings.py so that it will work on Windows with MinGW.

[Edit: This pull request resolves the autoexec.bin issue.]

[Danne]

Keep it up Dan. Valuable, interesting work.