Compiling ML on Windows with msys/MinGW - The Sequel

[dfort]

Many have tried and failed -- and so have I but this is soooo close.

Cut to the chase -- There is an issue with xor_chk in a Windows msys/MinGW build environment that prevents it from creating a working autoexec.bin. This isn't anything new, it was noted back in the original xor_chk pull request.

There is an open bug report on the issue that points back to this discussion.

Before anyone says to give it up and use Cygwin, Linux, Mac or even Linux in a virtual box on Windows please let me give you the back story. I was compiling some Mac binaries of the command line tools, cr2hdr, mlv_dump and raw2dng from dmilligan's ml_dng fork to help testers who wanted to try it out but couldn't compile the code on their own. Then a few Windows users asked if someone could supply them some binaries too. Looking at this as a challenge, I dusted off my old 32-bit Windows/Linux dual-boot laptop and gave it a go. Cygwin was super easy, but the .exe files compiled in that environment required cygwin1.dll and that wasn't very practical for distribution. Cross compiling in Linux worked but I had a 32-bit machine and testers wanted 64-bit binaries. I finally settled on cross compiling on my MacBookPro which worked like a charm.

In the process I spent more time that I should have trying to get the msys/MinGW setup working and I was eventually able to compile the command line tools with just a few Windows specific changes to a couple of files. Seeing that there might be some value in this I posted these pull requests:

lv_rec: use fseeko/fseeko64 depending on platform as in mlv_dump.c.

Modified readme2modulestrings.py so that it will work in a MinGW Windows environment.

At this point I did something crazy--I tried compiling ML for my camera and it worked! Well, sort of. The autoexec.bin didn't work. It looked fine, I mean it was the right size. There were a few differences when comparing it to a working autoexec.bin byte by byte and everything else (modules, etc.) worked. Eventually I tracked down the problem to xor_chk.exe. At first the only quirk seemed to be that "make clean" wouldn't remove it but there was something else very wrong here. Copying over the xor_chk.exe along with cygwin1.dll from the Cygwin build environment I had on the same computer and re-running the platform build created a working autoexec.bin.

Ok so the problem is narrowed down to just this one file, apparently the only one that the host system needs to build in order to complete a working autoexec.bin. Everything else is handled by the ARM cross compiler.

So why is it important to work on this? Well it does give users and developers more options. Ok, developers will still prefer a POSIX compliant environment but setting up msys/MinGW is almost as easy as Cygwin and much easier than running a Linux virtual machine on Windows. The original project at www.mingw.org may be outdated but there is the msys2 project and an easy installer for it here that can be used with the MinGW-w64 - for 32 and 64 bit Windows project that leapfrogs what is available in Cygwin. In addition, the compiled command line tools will run on a stock Windows machine without any extra .dll files--which is where this whole thing started.

[dfort]

Since taking the compiled xor_chk.exe from Cygwin worked I tried another test cross compiling Mac to Windows. It didn't work. So is there something in xor_chk.c that works fine in POSIX compliant environments but not on Windows? It is a very small source file so I'll post it here.

Code: [Select]

#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>


int main (int argc, char *argv[])
{
    if(argc != 2)
    {
        printf("Invalid parameter count (%d)\n", argc);
        return -1;
    }
    FILE *f = fopen(argv[1], "r+");
   
    if(!f)
    {
        printf("Failed to open file\n");
        return -1;
    }
   
    uint32_t checksum = 0;
    uint32_t data = 0;
    uint32_t read = 0;
   
    /* make sure we read from the beginning */
    fseek(f, 0, SEEK_SET);
   
    while(!feof(f))
    {
        if(fread(&data, 4, 1, f) != 1)
        {
            break;
        }
       
        checksum ^= data;
        read++;
    }

    /* modify checksum */
    data ^= checksum;
   
    fseek(f, -4, SEEK_END);
   
    if(fwrite(&data, 4, 1, f) != 1)
    {
        printf("Failed to write\n");
        return -1;
    }
   
    fclose(f);
   
    return 0;
}

[dmilligan]

So you should be able to test it easily. All it does is takes a filename as a command line parameter, computes a checksum of the file and then writes it back to the last 4 bytes of the file.

It should be simple to create a sample file run it through and then check the output with a hex editor.

If I had to guess, I'd say it's something with the fseek call.

[dfort]

Thanks!

Now I have a better understanding of what  xor_chk does. I'm seeing more differences than just the last 4 bytes between the working and non-working autoexec.bin files so that must be a clue that something is going wrong.

fseek was my first guess too.

[dfort]

Got it working! It only took one tiny change:

Originial:

Code: [Select]

FILE *f = fopen(argv[1], "r+");
Change:

Code: [Select]

FILE *f = fopen(argv[1], "rb+");
According to the fopen man page:

The mode string can also include the letter 'b' either as a last character or as a character between the characters in any of the two-character strings described above.  This is strictly for compatibility with C89 and has no effect; the 'b' is ignored on all POSIX conforming systems, including Linux.  (Other systems may treat text files and binary files differently, and adding the 'b' may be a good idea if you do I/O to a binary file and expect that your program may be ported to non-UNIX environments.)

I'll test it on Cygwin, OS-X and Linux before making a pull request just to be sure.

The next issue is that "make clean" doesn't remove xor_chk.exe in a Windows MSYS/MinGW environment. I've already tried some successful hacks for it but still looking for a more elegant solution.

[DeafEyeJedi]

Excellent work @dfort! [emoji106]

[dfort]

Thanks!

Here we go, pull request #658 build_tools for Windows.

Tested on OS-X, Linux, Cygwin and of course MSYS/MinGW.

The next thing to do would probably be a tutorial on how to setup a MSYS/MinGW Windows development environment, the easy way. I did it the hard way--many times over.

[Marsu42]

Many have tried and failed -- and so have I but this is soooo close.

Yeah, getting mingw to work reminds me of being a donkey chasing after a carrot hanging from a stick in front of his nose :-)

Just one comment: There's a big distinction between msys/cygwin-"mingw" and "native" mingw without any emulation layer. If you're downloading "mingw" from somewhere, you're probably also running bash which automatically means you're using msys - which in turn is an outdated cygwin fork. If you do so, imho it doesn't make a lot of sense to use a windows-native make and so on, but you can just stick with using the unix toolchain with the emulation layer. In this case, only dedicated programs which don't benefit from the emulation layer will be "native" windows - for example like the arm gcc compiler from launchpad.

As already commented in one of your pull requests, feel free to get a "native" mingw build system running, but there's little (newsgroup) support for it, and you'll certainly end in "line-ending"-hell and "path separator / \"-hell sharing scripts in both native windows and cygwin/msys-emulation. For the average user, getting cygwin compilation working (it nearly does except for some modules) will be the quickest way to windows-based success as cygwin has a nice installer, pulling/updating all dependencies and preventing problems from sharing win tools (mercurial, latex, ...) with the unix-ish ml build system. In any case the autoexec.bin and .exe tools will be the same as the compiler is native win32.

[dfort]

Although I was able to get ML and the command line tools working in the original MSYS/MinGW system for new users I'd recommend MSYS2 and MinGW-w64. I understand your arguments against MinGW and favoring Cygwin except one--the compiled command line (.exe) programs that are compiled in Cygwin are not native win32 (or win64) they must be distributed with the cygwin1.dll. That could be a problem. In fact that's the reason I even started going down this path.

The code changes needed to get ML to compile on a MSYS/MinGW are about the same as what is needed for OS-X. Let's give users a choice. (Ugh--now I'm sounding like a politician.)

I didn't expect to start a debate over the merits of Cygwin vs. MSYS/MinGW. In fact my development system is a Mac. It is my newest computer, with it I can compile ML and command line tools for both OS-X and Windows and it is the computer that I use with Lightroom, Photoshop, Premiere, Resolve and all the other programs that support my photography/filmmaking addiction. Uh oh--hope this doesn't start a Mac vs. Windows debate.

I'm working on a tutorial on building ML with MSYS2/MinGW-w64 for beginners. Stay tuned.

[dfort]

Setting up a development environment quickly and easily using MSYS2/MinGW-w64

Start out by heading over to the MSYS2 one click installer page. Disclaimer - this is going to take more than one click before you can compile ML.

I've only got an old 32-bit Windows system so if you try this on a 64-bit system please leave some feedback.

I recommend the default install location on the main (C:\msys32) drive. In fact just use all the defaults that the easy installer recommends and don't try to overthink it. After the install wizard finishes it will open up a terminal window. Don't feel intimidated if you aren't familiar with using the terminal. Eventually you should learn some basic commands but for now you and simply copy and paste what you need. The first thing to do as noted in the easy installer webpage is to update the system packages.

Code: [Select]

pacman --needed -Sy bash pacman pacman-mirrors msys2-runtime
Go ahead and highlight what is in that code box and copy to the clipboard. To paste in the terminal you will quickly learn that good ol' Ctrl-V doesn't work. Right click in the terminal window and select Paste from the pop up menu. You can learn the keyboard shortcuts if you like, I never did.

This will run the command to update the packages. Of course your answer should be y or Y, Windows is case insensitive so it doesn't matter. I ran the installation many time to make sure that this tutorial really does work and sometimes if the network connection wasn't solid it wouldn't download all the packages. In that case simply run the command again. Fortunately the terminal keeps a history so use the up arrow key to bring up the last command you entered in the terminal.

After this process is finished close the terminal. Don't try to get fancy and type "exit" in the terminal because it won't work and you'll have to force quit the "bash" process. Simply click the "X" on the upper right hand corner to close window.

Bring up your start menu and you will see a folder named "MSYS 32bit" or I guess "MSYS 64bit" if you installed the 64-bit version. In that folder are three "shells" -- I can hear the laughs from anyone who has seen "Demolition Man." In any case, I'll be using the "MinGW-w64 Win32 Shell." Do not use the MSYS2 Shell, it won't work with the way this setup is configured. Note that it is possible to configure MSYS2 Shell to compile ML but the command line tools will need the msys-1.0.dll to run so let's not go there.

Then copy/past this next command and run it:

Code: [Select]

pacman -Su
Guess what? You now have MSYS2 setup and it only took a few left and right clicks. Now it is time to get the packages needed to compile Magic Lantern. There aren't that many in my list because some packages will install several other packages that it depends on. You will need to make a decision if you want any of the command line tools like cr2hdr or mlv_dump to compile as 32-bit or 64-bit native. If you have a 32-bit computer like me you have no choice but on a 64-bit system you can actually install both versions and compile for either 32-bit or 64-bit targets. For now keep it simple and just select one.

32-bit compiler

Code: [Select]

pacman -S make diffutils wget zip unzip tar perl mingw-w64-i686-gcc mingw-w64-i686-python2-docutils
64-bit compiler

Code: [Select]

pacman -S make diffutils wget zip unzip tar perl mingw-w64-x86_64-gcc mingw-w64-x86_64-python2-docutils
Hang on, we're almost there. Canon cameras are obviously not Windows computers so we will need one more compiler to build something that will run on the ARM Architecture used in the camera. To figure out which ARM compiler to get refer to the ML code, specifically: Makefile.user.default -- ARM_PATH=~/gcc-arm-none-eabi-4_8-2013q4 notice that little squiggly mark after the equal sign? That means that the path starts in your home directory and surprise, you're already there. Let's make it easy, just copy/paste the following in your terminal:

Code: [Select]

wget https://launchpad.net/gcc-arm-embedded/4.8/4.8-2013-q4-major/+download/gcc-arm-none-eabi-4_8-2013q4-20131204-win32.zip && mkdir gcc-arm-none-eabi-4_8-2013q4 && unzip -n gcc-arm-none-eabi-4_8-2013q4-20131204-win32.zip -d gcc-arm-none-eabi-4_8-2013q4 && rm gcc-arm-none-eabi-4_8-2013q4-20131204-win32.zip
Now you have a system that can compile ML. The only thing that is missing is the ML source code itself. The code is stored on a Mercurial repository and since the Mercurial software is a free download from the developers it makes sense to get it directly from the Mercurial website.

Just make sure you leave the option checked to add Mercurial to the search path. It would also be a good idea to restart your system at this point.

Note that you can also install Mercurial via pacman but I had issues with that way of installing it. One of the advantages of msys is that it doesn't lock you in to only what is available in that world. For more information I recommend reading dmilligan's excellent Mercurial Tips but if you're only interested in grabbing the code and rolling your own--read on.

Ok--let's get the code and start compiling. If you grab the unified branch as instructed in a1ex's Getting started with development tutorial:

Code: [Select]

hg clone https://bitbucket.org/hudson/magic-lantern
cd magic-lantern
hg update unified
At this point you can jump over to any other tutorial on how to compile ML and the command line tools. Here's the way I do it for the EOSM:

Code: [Select]

cd magic-lantern/platform/EOSM.202/

Code: [Select]

make zip
This will create a zip file just like the nightly builds.

If you are interested in the command line tools you should be able to go into the appropriate directory in magic-lantern/modules/ and compile the tools. There are still some quirks getting all the command line tools to compile but I found that to be the case for all of the development setups that I have tested out. The binary files that this MSYS/MinGW setup creates should be portable to other Windows computers. What I mean by that is that it isn't dependent on cygwin1.dll or msys-1.0.dll files that may not be installed on the target computer.

So you want to start hacking away on the code? There's lots of choices when it comes to text editors, I would recommend using Notepad++. It is designed to work with source code and you can check if the source has Linux, Mac or Windows line endings--don't mix them up! Here is a screenshot of the interface, notice that posix.c has Windows (CR+LF) line endings.

Also consider using the excellent SourceTree app to keep track of your changes and easily switch between branches.

One more tip. If you are going to do lots of coding with this system I'd recommend making shortcuts of your msys home directory and shell batch file in easy to find places.

Finally, consider this (and all ML code for that matter) as a work in progress. Have fun experimenting. My next project is checking out how to compile native Windows versions of the command line tools using Cygwin as recommended by Marsu42.

Happy Hacking!

[dfort]

Setting up a development environment with the old school MinGW

Are you more familiar with the original MinGW project? No problem, it can also be configured to build ML and the command line tools. The command line tools will be linked to the Windows libraries and thus be portable to other computers. The original MinGW project has been a little slow to update so it doesn't have 64-bit support yet though there are ways around this. You can check out some advanced MinGW tricks from this post but let's start with a simple installation to get started.

Go to mingw.org and Download the mingw-get-setup.exe installer. You will want to pick the mingw-developer-tool and msys-base, note that picking the developer tools will automatically select msys-base. The installer is a bit old school, you need to click and select Mark for installation then got to the Installation menu and Apply Changes. Keep it simple and just accept the default installation options.

Navigate your way to the installed directory: C:\MinGW\msys\1.0 and launch msys.bat This will launch a terminal and also create your home directory.

Doing copy/paste in the old school terminal is a little trickier - click on the C:\ icon on the upper left corner to bring up the pop up menu. Paste is in the Edit submenu. If you want to select text you need to choose Mark from the menu then press Enter. Try it out:

Code: [Select]

mount C:/MinGW /mingw
What did that command do? It was needed to set up the development tools. It took me a long time to figure that one out because it doesn't seem to be documented very well.

There are a few other packages needed for ML. Instead of searching through the installer it is easier to do it on the terminal:

Code: [Select]

mingw-get install msys-zip msys-unzip msys-wget
As long as you're getting the hang of copy/paste in the terminal, run the following to get the ARM cross compiler:

Code: [Select]

wget --no-check-certificate https://launchpad.net/gcc-arm-embedded/4.8/4.8-2013-q4-major/+download/gcc-arm-none-eabi-4_8-2013q4-20131204-win32.zip && mkdir gcc-arm-none-eabi-4_8-2013q4 && unzip -n gcc-arm-none-eabi-4_8-2013q4-20131204-win32.zip -d gcc-arm-none-eabi-4_8-2013q4 && rm gcc-arm-none-eabi-4_8-2013q4-20131204-win32.zip
If you look closely you'll notice there's a slight difference from the MSYS2 version because this version isn't set up to check ssl certificates.

Python isn't included as a standard package so we'll install the official version from python.org download latest python 2 for Windows (2.7.10) and run the installer. One thing to watch out for is to make sure and select the option to add python.exe to your path. Either "Will be installed on local hard drive" or "Entire feature will be installed on local hard drive" will work because ML requires only a minimal amount of Python features.

There are a few things that need to be done to get python set up for ML. Go back to the MSYS terminal and use the following command to install the python docutils:

Code: [Select]

pip install docutils
Ignore the message to update pip, you won't need it again. Next you need to make a few links so python2 and rst2html can be found:

Code: [Select]

ln -s /c/Python27/python.exe /c/Python27/python2.exe

Code: [Select]

ln -s /c/Python27/Scripts/rst2html.py /c/Python27/Scripts/rst2html
Finally, make sure that your system is set up to recognize the .PY file extension. On the version of Windows that I'm using I need to right click on the Computer icon, select Properties then select Advanced system settings, go to the Advanced tab, Environmental Variables... button then scroll down on the bottom window to select PATHEXT and Edit--whew!

The last program to install is Mercurial. That is an easy one and needs no tweaking to get it to work.

I won't repeat everything that was in the previous tutorial so just grab the code and start building:

Code: [Select]

hg clone https://bitbucket.org/hudson/magic-lantern
cd magic-lantern
hg update unified
What's the advantage of the old school version of MinGW over the newer projects? It seems that there is less of a problem compiling the command line tools, though they will always be 32-bit unless you can figure out how to install the 64-bit tool chain.

Have fun!

[jkruser]

dfort, great job on the write up, I was able to get msys2 up and compiling with relatively little fuss on a 64 bit Windows 8 box.  I did find that Mercurial's suggestion window to restart was more of a requirement as hg was not recognized until after a full restart of Windows.

Just to validate, my build of the nightly branch at the time of the post was 1,551,114 bytes, what are you seeing?

[dfort]

Thanks for the feedback. Good to know that it is working on 64-bit systems.

My nightly is 1,551,116 bytes zipped and 2,167,768 bytes unzipped. This is for an EOSM build so if you are on a 70D it might be slightly different.

[jkruser]

dfort, my build was of the universal (marked EOS M) branch you linked to in the guide just as a test.  The 2 byte difference was from the zip algorithm in 64 bit.  Once unzipped, I have the same 2,167,786 byte size.  A proper MD5 hash probably isn't needed to confirm.

[dfort]

Great--so everything is working? Did you get this working with the 70D sources? Hopefully nikfreak can help you out with that.

[jkruser]

Dfort, do you have a list of changes you made to get unified to build?  I pulled a copy of the central repository and just like you mentioned, it fails to build.  Once I get that up and running, I'll get tinkering with the 70D branch.

[dfort]

Yes, there are only a few changes that are covered by these two pull requests:

https://bitbucket.org/hudson/magic-lantern/pull-requests/656/modified-readme2modulestringspy-so-that-it/diff
https://bitbucket.org/hudson/magic-lantern/pull-requests/658/build_tools-for-windows/diff

You can just replace these files if they don't clash with nikfreak's 70D code:

modules/last_change_info.sh

Code: [Select]

#!/bin/sh

# Called from readme2modulestrings.py
# also useful by itself to look at a formatted report
# of the extracted version info

LC_TIME=EN \
hg log . -r 'reverse(ancestors(.))' -l 1 --template \
'{date|hgdate}\n{node|short}\n{author|user}\n{desc|strip|firstline}'
modules/readme2modulestrings.py

Code: [Select]

# extract module strings from README.rst

import sys, re
import subprocess
from datetime import datetime

from align_string_proportional import word_wrap
from rbf_read import extent_func, rbf_init_font
rbf_init_font("../../data/fonts/argnor23.rbf")

def run(cmd):
    try:
        p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        out = p.communicate()
        if p.returncode or out[1]:
            print >> sys.stderr, cmd
            print >> sys.stderr, out[0]
            print >> sys.stderr, out[1]
            exit(1)
        return out[0]
    except:
        print >> sys.stderr, sys.exc_info()
        exit(1)

# see http://gcc.gnu.org/bugzilla/show_bug.cgi?id=37506 for how to place some strings in a custom section
# (you must declare all strings as variables, not only the pointers)

def c_repr(name):
    if "\n" in name:
        s = "\n"
        for l in name.split("\n"):
            s += "    %s\n" % ('"%s\\n"' % l.replace('"', r'\"'))
        return s
    else:
        return '"%s"' % name.replace('"', r'\"')

strings = []
def add_string(name, value):
    a = chr(ord('a') + len(strings))
    print "static char __module_string_%s_name [] MODULE_STRINGS_SECTION = %s;" % (a, c_repr(name))
    print "static char __module_string_%s_value[] MODULE_STRINGS_SECTION = %s;" % (a, c_repr(value))

    strings.append((name, value))

def declare_string_section():
    print
    print('MODULE_STRINGS_START()')
    for i, s in enumerate(strings):
        a = chr(ord('a') + i)
        print "    MODULE_STRING(__module_string_%s_name, __module_string_%s_value)" % (a, a)
    print('MODULE_STRINGS_END()')

inp = open("README.rst").read().replace("\r\n", "\n")
lines = inp.strip("\n").split("\n")
title = lines[0]

used_lines = []
for l in lines[2:]:
    if l.startswith("..") or l.strip().startswith(":"):
        continue
   
    used_lines.append(l)

inp = "\n".join(used_lines)
inp = inp.split("\n\n")

add_string("Name", title)

# extract user metadata from RST meta tags
tags = {}
for l in lines[2:]:
    l = l.strip()
    m = re.match("^:([^:]+):(.+)$", l)
    if m:
        name = m.groups()[0].strip()
        value = m.groups()[1].strip()
        if value.startswith("<") and value.endswith(">"):
            continue
        add_string(name, value)
        tags[name] = value

if "Author" not in tags and "Authors" not in tags:
    print >> sys.stderr, "Warning: 'Author/Authors' tag is missing. You should tell the world who wrote your module ;)"

if "License" not in tags:
    print >> sys.stderr, "Warning: 'License' tag is missing. Under what conditions we can use your module? Can we publish modified versions?"

if "Summary" not in tags:
    print >> sys.stderr, "Warning: 'Summary' tag is missing. It should be displayed as help in the Modules tab."

# extract readme body:
# intro -> "Description" tag;
# each section will become "Help page 1", "Help page 2" and so on

# render the RST as html -> txt without the metadata tags
# sed command at end is because Windows inserts CR characters all over the place. Removing them should be benign on other platforms.
txt = run('cat README.rst | grep -v -E "^:([^:])+:.+$" | rst2html --no-xml-declaration | python ../html2text.py -b 700 | sed "s/\r$//"')

desc = ""
last_str = "Description"
help_page_num = 0
lines_per_page = 0
for p in txt.strip("\n").split("\n")[2:]:
    if p.startswith("# "): # new section
        help_page_num += 1
        add_string(last_str, desc)
        desc = ""
        last_str = "Help page %d" % help_page_num
        lines_per_page = 0
        p = p[2:].strip()
    desc += "%s\n" % p
    lines_per_page += 1
    if lines_per_page > 18:
        print >> sys.stderr, "Too many lines per page\n"
        exit(1)

add_string(last_str, desc)

# extract version info
# (prints the latest changeset that affected this module)
# Window can't handle this so run it from a shell script
if sys.platform == 'win32':
last_change_info = run("sh ../last_change_info.sh")
else:
last_change_info = run("LC_TIME=EN hg log . -r 'reverse(ancestors(.))' -l 1 --template '{date|hgdate}\n{node|short}\n{author|user}\n{desc|strip|firstline}'")

if len(last_change_info):
    last_change_date, last_changeset, author, commit_msg = last_change_info.split("\n")
    split = last_change_date.split(" ")
    seconds = float(split[0])
    last_change_date = datetime.utcfromtimestamp(seconds).strftime("%Y-%m-%d %H:%M:%S UTC")
   
    # trim changeset to 7 chars, like Bitbucket does
    last_changeset = last_changeset[:7]
   
    # trim commit msg to 700px
    size = extent_func(commit_msg)[0]
    if size > 700:
        new_size = 0
        new_msg = ""
        for c in commit_msg:
            new_size += extent_func(c)[0]
            if new_size > 700:
                break
            new_msg += c
        commit_msg = new_msg + "..."
       
    add_string("Last update", "%s on %s by %s:\n%s" % (last_changeset, last_change_date, author, commit_msg))

build_date = datetime.utcnow().strftime("%Y-%m-%d %H:%M:%S UTC")

# echo called from python in Windows behaves differently, better to avoid it.
if sys.platform == 'win32':
build_user = run('whoami').replace("\n", "") + "@" + run('hostname').replace("\n", "")
else:
build_user = run("echo `whoami`@`hostname`")

add_string("Build date", build_date)
add_string("Build user", build_user)

declare_string_section()

build_tools/Makefile

Code: [Select]

# Makefile for the various tools used in the build process

ifndef TOP_DIR
# if this is used as a standalone makefile, we need to include some additional stuff
TOP_DIR=..
include $(TOP_DIR)/Makefile.setup
endif

$(XOR_CHK): $(XOR_CHK).c
$(call build,XOR_CHK,$(HOST_CC) $< -o $@)

clean::
$(call rm_files, $(XOR_CHK) $(XOR_CHK).exe)

build_tools/xor_chk.c

Code: [Select]

#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>


int main (int argc, char *argv[])
{
    if(argc != 2)
    {
        printf("Invalid parameter count (%d)\n", argc);
        return -1;
    }
    FILE *f = fopen(argv[1], "rb+");
   
    if(!f)
    {
        printf("Failed to open file\n");
        return -1;
    }
   
    uint32_t checksum = 0;
    uint32_t data = 0;
    uint32_t read = 0;
   
    /* make sure we read from the beginning */
    fseek(f, 0, SEEK_SET);
   
    while(!feof(f))
    {
        if(fread(&data, 4, 1, f) != 1)
        {
            break;
        }
       
        checksum ^= data;
        read++;
    }

    /* modify checksum */
    data ^= checksum;
   
    fseek(f, -4, SEEK_END);
   
    if(fwrite(&data, 4, 1, f) != 1)
    {
        printf("Failed to write\n");
        return -1;
    }
   
    fclose(f);
   
    return 0;
}
Well, there is another pull request to build raw2dng but you probably don't need to deal with that if you're only interested in building ML.

One more time saver, to get the 70D source you can try this:

Code: [Select]

hg clone -r 70D-merge https://nikfreak@bitbucket.org/nikfreak/magic-lantern
By the way, I just put up a tutorial on how to set up a Cygwin development environment, so just in case this setup isn't cutting it for you.

http://magiclantern.fm/forum/index.php?topic=15894.msg154435;topicseen#msg154435

[An147]

Hi @dfort! I followed tutorial for starting development with MSYS2, looks like there are some new problems arise since 2015.

I'm trying to build ML for Canon 5D2, unified branch.

First, there is no mingw-w64-x86_64-python2-docutils package now, so I run:

Code: [Select]

pacman -S mingw-w64-x86_64-python-docutilsIt fixed Error 2 from make...Ok.

Next error was "/bin/sh: line 1: hg: command not found".
Ok, I setup Mercurial right in MSYS2

Code: [Select]

pacman -S mercurial
Next

Code: [Select]

[ LD       ]   autoexec
which: no gcc in (/usr/local/bin:/usr/bin:/bin:/opt/bin:/c/Windows/System32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0/:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl)
[ XOR_CHK  ]   ../../build_tools/xor_chk
../../build_tools/xor_chk.c: line 7: syntax error near unexpected token `('
../../build_tools/xor_chk.c: line 7: `int main (int argc, char *argv[])'
make: *** [../../build_tools/Makefile:11: ../../build_tools/xor_chk] Error 2
It said there is no gcc, and indeed there was no gcc.exe in C:\msys64\usr\bin
Solved by running

Code: [Select]

pacman -S gcc
Now I'm stuck with this error:

Code: [Select]

[ SCRIPTS  ]   install_extra_data
"/bin/sh: line 1: /d/Job/MagicLantern/Repo/magic-lantern/platform/5D2.212/zip/ML/scripts/${f\#\#../../scripts/}: bad substitution
make: *** [../../Makefile.inc:41: install_extra_data] Error 127"
I'm running GNU Make 4.3 Built for x86_64-pc-msys and gcc (GCC) 10.2.0
There is an issue in repository that shows a solution: https://foss.heptapod.net/magic-lantern/magic-lantern/-/issues/2944
But the Makefile.top has "scripts/${f\#\#../../scripts/}" at line 48 now, not 80. And I didn't understand

The fix is to replace '#' with '#'

. Is it a mispelling?
Can you help me with this, please? Maybe there is a more up-to-date solution to compile ML on Windows?

[names_are_hard]

Most of these problems you describe are generally "mingw environment provides a more modern set of tools than ML expects".  There are two approaches to fix this.  Use older tools (an older version of mingw?  Or use WSL and install an older Linux, Ubuntu 16 or 18 perhaps).  Or update ML to work with more recent tools.

I have a fork of ML that I have updated to work with more modern tools (e.g., the problem with # in make was fixed there in Nov 2020).  You could try that to see if it works:
https://github.com/reticulatedpines/magiclantern_simplified

[An147]

Thanks for your reply!

Ok, I can try your fork, or maybe running older Linux in virtual machine. In the end I'm interested in 10/12/14bit RAW video branch with crop_mod for 5D2, but even basic version is a good start.

But then I'm just interested:
So, other people continue developing ML using Mercurial, and you continue to develop some other branches, so they are independent, right?
It just seems strange to me, but I'm appreciate your repo with Git! Good luck!

[names_are_hard]

WSL will be easier to get working, though perhaps less convenient for you to work with.  If you use my repo in mingw and it builds, you know the problem is likely official ML not building with modern tools.

I've never had commit access to the official repo, so I couldn't use it.  I also found trying to use mercurial I lost work multiple times, because I didn't understand it.  Feels like it must have dangerous defaults.  Since mercurcial is almost dead, I couldn't be bothered learning it, preferring to port the repo to git (and at the time, bitbucket was removing all mercurial repos, so, why not?).  The official repo also has several important long lived branches, and I was finding it annoying needing to swap between them to work with different things, so I merged these in my repo (lua_fix, unified, qemu and digic6-dumper).

At some point I'd like to get my work merged into official, but that won't be for a while - the purpose of my repo is primarily working on modern Digic cams that are unsupported, Digic 6, 7 and 8.  We're getting there but there is still a long way to go.