Author Topic: Compiling Magic Lantern with Cygwin/MinGW-64  (Read 16699 times)

dmilligan

  • Developer
  • Hero Member
  • *****
  • Posts: 3165
  • 60Da / 1100D / EOSM
Re: Compiling Magic Lantern with Cygwin/MinGW-64
« Reply #25 on: March 22, 2016, 03:47:07 AM »
I'm pretty sure that's a spambot

vstrglv

  • Member
  • ***
  • Posts: 134
Re: Compiling Magic Lantern with Cygwin/MinGW-64
« Reply #26 on: November 22, 2016, 07:54:32 AM »
I am trying to compile with
 "hg clone -r unified https://bitbucket.org/hudson/magic-lantern
cd magic-lantern/platform/5D3.113/
make clean && make zip",
but there is an error for raw_rec.mo:
Code: [Select]
[ GCC      ]   raw2dng
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: cannot find -lm
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0//libgcc_s.dll.a when searching for -lgcc_s
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0/libgcc_s.dll.a when searching for -lgcc_s
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0//libgcc_s.dll.a when searching for -lgcc_s
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0/libgcc_s.dll.a when searching for -lgcc_s
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: cannot find -lgcc_s
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0//libgcc.a when searching for -lgcc
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0//libgcc.a when searching for -lgcc
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0/libgcc.a when searching for -lgcc
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0/libgcc.a when searching for -lgcc
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0//libgcc.a when searching for -lgcc
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0/libgcc.a when searching for -lgcc
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: cannot find -lgcc
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: cannot find -lcygwin
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/w32api/libadvapi32.a when searching for -ladvapi32
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/w32api/libadvapi32.a when searching for -ladvapi32
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/w32api/libadvapi32.a when searching for -ladvapi32
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: cannot find -ladvapi32
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/w32api/libshell32.a when searching for -lshell32
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/w32api/libshell32.a when searching for -lshell32
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/w32api/libshell32.a when searching for -lshell32
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: cannot find -lshell32
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/w32api/libuser32.a when searching for -luser32
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/w32api/libuser32.a when searching for -luser32
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/w32api/libuser32.a when searching for -luser32
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: cannot find -luser32
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/w32api/libkernel32.a when searching for -lkernel32
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/w32api/libkernel32.a when searching for -lkernel32
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/w32api/libkernel32.a when searching for -lkernel32
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: cannot find -lkernel32
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0//libgcc_s.dll.a when searching for -lgcc_s
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0/libgcc_s.dll.a when searching for -lgcc_s
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0//libgcc_s.dll.a when searching for -lgcc_s
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0/libgcc_s.dll.a when searching for -lgcc_s
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: cannot find -lgcc_s
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0//libgcc.a when searching for -lgcc
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0//libgcc.a when searching for -lgcc
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0/libgcc.a when searching for -lgcc
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0/libgcc.a when searching for -lgcc
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0//libgcc.a when searching for -lgcc
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0/libgcc.a when searching for -lgcc
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: cannot find -lgcc
collect2: error: ld returned 1 exit status
Makefile:16: recipe for target 'raw2dng' failed
make[4]: *** [raw2dng] Error 1

********************************************************
WARNING: module raw_rec failed to build, deleting
********************************************************
What is wrong?
Canon 5D3, SanDisk Extreme PRO,160MB/s, 256GB

dfort

  • Hero Member
  • *****
  • Posts: 1596
Re: Compiling Magic Lantern with Cygwin/MinGW-64
« Reply #27 on: November 22, 2016, 09:17:20 AM »
That's odd, it is building raw2dng.exe. As I recall that doesn't build properly with a 64-bit compiler.

I can't reproduce the error because my PC is a very slow and old 32-bit laptop. Here are some things you can try.

Go into the modules/lv_rec directory and open up the file named Makefile with a text editor. Find this line:

Code: [Select]
all:: raw2dng
and comment it out like this:

Code: [Select]
# all:: raw2dng
That doesn't solve the issue with being able to build raw2dng with a 64-bit compiler. For that you can look for this block of code:

Code: [Select]
raw2dng.exe: FORCE
$(call build,MINGW,$(MINGW_GCC) -c $(SRC_DIR)/chdk-dng.c -m32 -mno-ms-bitfields -O2 -Wall -I$(SRC_DIR))
$(call build,MINGW,$(MINGW_GCC) -c raw2dng.c -m32 -mno-ms-bitfields -O2 -Wall -I$(SRC_DIR) -D_FILE_OFFSET_BITS=64)
$(call build,MINGW,$(MINGW_GCC) raw2dng.o chdk-dng.o -o raw2dng.exe -lm -m32)

and change that second "call build" line to include the -std=c99 switch:

Code: [Select]
$(call build,MINGW,$(MINGW_GCC) -c raw2dng.c -m32 -mno-ms-bitfields -O2 -Wall -I$(SRC_DIR) -D_FILE_OFFSET_BITS=64 -std=c99)
If neither of these works you can delete your 64-bit Cygwin installation and install a 32-bit version in its place.

Please post which "fix" worked for you.
EOSM.202 EOSM.203 EOSM2.103 700D.115 5D3.*

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 9840
  • 5D Mark Free
Re: Compiling Magic Lantern with Cygwin/MinGW-64
« Reply #28 on: November 22, 2016, 09:35:03 AM »
Searching for (parts of) the error message is a good idea. Just be careful with dashes, e.g. search for:
Code: [Select]
x86_64-pc-cygwin/bin/ld: "cannot find -lgcc"

Without quotes, -lgcc in the search string means "return pages that do not contain lgcc".

http://stackoverflow.com/questions/30119573/compile-32bit-code-from-cygwin64

vstrglv

  • Member
  • ***
  • Posts: 134
Re: Compiling Magic Lantern with Cygwin/MinGW-64
« Reply #29 on: November 22, 2016, 05:33:18 PM »
Thank you for reply dfort and alex. dfort, i have tried your fixes, but it does not work. No raw2dng.exe, no raw_rec.mo. May be i have to install cygwin32.
Now commented # all:: raw2dng in modules/raw_rec and raw_rec.mo appeared!
Canon 5D3, SanDisk Extreme PRO,160MB/s, 256GB

dfort

  • Hero Member
  • *****
  • Posts: 1596
Re: Compiling Magic Lantern with Cygwin/MinGW-64
« Reply #30 on: November 22, 2016, 08:11:57 PM »
I don't have a 64-bit PC to test out these issues. On the tutorial I included this:

Quote
...What if you have a 64-bit system?
Note that most module exe helpers are targeted for x86 code, i.e. even if you run cygwin x64 (which you should on a x64 Windows) you need to use the i686 mingw toolchain.
Please leave feedback if you get mingw64-x86-64-gcc-core working and on which command line tools--it is going to need some tweaks to the ML source code.

That might be a little confusing but basically it means that if you are going to compile the command line tools, like raw2dng.exe, it has been tested to work with the 32-bit compiler (i686) but not with the 64-bit compiler (x64).

I wasn't aware that building a platform automatically builds raw2dng but now that I see that does. It was the same with cr2hdr until just recently with this commit. The other popular command line tool, mlv_dump, isn't built when a platform is compiled so I thought that was preferred method. Build rules for raw2dng are in both raw_rec and lv_rec so perhaps we should look into that.

Of course this doesn't resolve another issue which is that raw2dng.exe can't be built with a 64-bit compiler. I'm not sure if this is true for all 64-bit compilers but it is a problem when I cross compile a Windows binary on a Mac:

Code: [Select]
[ MINGW    ]   raw2dng.exe
raw2dng.c: In function 'find_and_fix_cold_pixels':
raw2dng.c:1213:9: error: 'for' loop initial declarations are only allowed in C99 or C11 mode
         for (int y = 0; y < h; y++)
         ^
raw2dng.c:1213:9: note: use option -std=c99, -std=gnu99, -std=c11 or -std=gnu11 to compile your code
raw2dng.c:1215:13: error: 'for' loop initial declarations are only allowed in C99 or C11 mode
             for (int x = 0; x < w; x++)
             ^
raw2dng.c:1233:5: error: 'for' loop initial declarations are only allowed in C99 or C11 mode
     for (int p = 0; p < cold_pixels; p++)
     ^
raw2dng.c:1243:9: error: 'for' loop initial declarations are only allowed in C99 or C11 mode
         for (int i = -4; i <= 4; i++)
         ^
raw2dng.c:1245:13: error: 'for' loop initial declarations are only allowed in C99 or C11 mode
             for (int j = -4; j <= 4; j++)
             ^
make: *** [raw2dng.exe] Error 1

Simply adding the "-std=c99" switch worked for me but it should be tested on other 64-bit systems.

Thank you for reply dfort and alex. dfort, i have tried your fixes, but it does not work. No raw2dng.exe, no raw_rec.mo. May be i have to install cygwin32.
Now commented # all:: raw2dng in modules/raw_rec and raw_rec.mo appeared!

I'm a bit confused with that comment. So commenting out the "all" rule worked for you? This should build the raw_rec module but not raw2dng.exe.

What was it that didn't work, adding the "-std=c99" switch?

@alex - Which compiler are you using on Linux and are you cross compiling Windows binaries? Would adding the "-std=c99" switch to the raw2dng make rules have any negative consequences? It seems to work fine with the Windows binaries that I have compiled on my Mac.
EOSM.202 EOSM.203 EOSM2.103 700D.115 5D3.*

vstrglv

  • Member
  • ***
  • Posts: 134
Re: Compiling Magic Lantern with Cygwin/MinGW-64
« Reply #31 on: November 22, 2016, 08:28:55 PM »
So commenting out the "all" rule worked for you? This should build the raw_rec module but not raw2dng.exe.

What was it that didn't work, adding the "-std=c99" switch?


Yes, if i comment "all", raw_rec module  is built but not raw2dng.exe
If add "-std=c99" switch, raw_rec module  is not built.
But it is possible to compile raw2dng.exe by "make raw2dng.exe" from modules/raw_rec folder. The same for cr2hdr.exe from modules/dual_iso
Canon 5D3, SanDisk Extreme PRO,160MB/s, 256GB

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 9840
  • 5D Mark Free
Re: Compiling Magic Lantern with Cygwin/MinGW-64
« Reply #32 on: November 22, 2016, 08:29:37 PM »
@alex - Which compiler are you using on Linux and are you cross compiling Windows binaries?

The one from Makefile. The nightly build server compiles cr2hdr.exe and mlv_dump.exe, but raw2dng binaries weren't updated lately (maybe I should set up a job for it as well).

Quote
Would adding the "-std=c99" switch to the raw2dng make rules have any negative consequences?

Nope.

BTW, gcc 5.x uses -std=gnu11 by default, whereas gcc 4.x uses -std=gnu89.

dfort

  • Hero Member
  • *****
  • Posts: 1596
Re: Compiling Magic Lantern with Cygwin/MinGW-64
« Reply #33 on: November 22, 2016, 09:27:45 PM »
Ok--looks like @vstrglv is able to compile ML so I'll test some Makefile changes that should keep these issues from coming up again.
EOSM.202 EOSM.203 EOSM2.103 700D.115 5D3.*

vstrglv

  • Member
  • ***
  • Posts: 134
Re: Compiling Magic Lantern with Cygwin/MinGW-64
« Reply #34 on: December 06, 2016, 04:20:38 PM »
Quote
Ok--looks like @vstrglv is able to compile ML so I'll test some Makefile changes that should keep these issues from coming up again.
Thank you. Now this error has gone.
Canon 5D3, SanDisk Extreme PRO,160MB/s, 256GB

dfort

  • Hero Member
  • *****
  • Posts: 1596
Re: Compiling Magic Lantern with Cygwin/MinGW-64
« Reply #35 on: May 27, 2017, 02:06:11 AM »
A while back a user came up with a problem: Unable to build ML with Cygwin 32. I'm now helping another user having problems with this tutorial so I decided to delete my Cygwin installation and rebuild it by following the tutorial. The installation seemed to go fine but I got the same compile errors that the other user reported:

Code: [Select]
make[1]: Leaving directory '/home/Dan/magic-lantern/tcc'
[ CC       ]   module.o
[ AR       ]   strrchr.o
[ AR       ]   dietlibc.a
[ AR       ]   lib_a-setjmp.o
[ AR       ]   newlib-libc.a
[ CP       ]   newlib-libm.a
[ CP       ]   gcc-libgcc.a
[ LD       ]   magiclantern
module.o: In function `module_load_symbols.constprop.2':
module.c:(.text+0x428): undefined reference to `tcc_add_symbol'
module.o: In function `_module_load_all':
module.c:(.text+0x498): undefined reference to `tcc_new'
module.c:(.text+0x4a4): undefined reference to `tcc_set_options'
module.c:(.text+0x4c8): undefined reference to `tcc_delete'
module.c:(.text+0x504): undefined reference to `tcc_delete'
module.c:(.text+0x720): undefined reference to `tcc_delete'
module.c:(.text+0x75c): undefined reference to `tcc_add_file'
module.c:(.text+0x7bc): undefined reference to `tcc_relocate'
module.c:(.text+0x7e8): undefined reference to `tcc_relocate'
module.c:(.text+0x850): undefined reference to `tcc_delete'
module.c:(.text+0x8bc): undefined reference to `tcc_get_symbol'
module.c:(.text+0x8e4): undefined reference to `tcc_get_symbol'
module.c:(.text+0x90c): undefined reference to `tcc_get_symbol'
module.c:(.text+0x934): undefined reference to `tcc_get_symbol'
module.c:(.text+0x95c): undefined reference to `tcc_get_symbol'
module.o:module.c:(.text+0xc4c): more undefined references to `tcc_get_symbol' follow
module.o: In function `_module_load_all':
module.c:(.text+0xcac): undefined reference to `tcc_delete'
module.o: In function `module_load_task':
module.c:(.text+0xf7c): undefined reference to `tcc_load_offline_section'
module.o: In function `module_load':
module.c:(.text+0x1040): undefined reference to `tcc_new'
module.c:(.text+0x104c): undefined reference to `tcc_set_options'
module.c:(.text+0x1070): undefined reference to `tcc_delete'
module.c:(.text+0x1084): undefined reference to `tcc_add_file'
module.c:(.text+0x1098): undefined reference to `tcc_relocate'
module.c:(.text+0x10a8): undefined reference to `tcc_delete'
module.o: In function `module_get_symbol':
module.c:(.text+0x10c8): undefined reference to `tcc_get_symbol'
module.o: In function `module_exec':
module.c:(.text+0x10f0): undefined reference to `tcc_get_symbol'
module.o: In function `module_unload':
module.c:(.text+0x126c): undefined reference to `tcc_delete'
make: *** [../../src/Makefile.src:195: magiclantern] Error 1

I'm at a loss to figure out what is going on here and how to fix it. Just before deleting my previous Cygwin installation I downloaded the latest ML and compiled just fine. Of course I no longer have the previous installation--ugh!

[EDIT]

For what it is worth turning off the module configuration allows compilation to complete but of course no modules are built.

Makefile.user.default
Code: [Select]
CONFIG_MODULES      = n
EOSM.202 EOSM.203 EOSM2.103 700D.115 5D3.*

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 9840
  • 5D Mark Free
Re: Compiling Magic Lantern with Cygwin/MinGW-64
« Reply #36 on: May 27, 2017, 09:46:33 AM »
Full log?

Best guess: I'm using a trick to strip most TCC symbols - everything but the API we use - in tcc/Makefile near localsyms. This file should contain all TCC's symbols except those listed in the error message. Some of the tools used to create this file might not be available on Windows.

dfort

  • Hero Member
  • *****
  • Posts: 1596
Re: Compiling Magic Lantern with Cygwin/MinGW-64
« Reply #37 on: May 27, 2017, 07:33:48 PM »
Full log

I'm guessing that between September 2015 when I wrote this tutorial and March 2017 when this issue was reported something changed in Cygwin. Maybe one of the tools used in tcc/Makefile? Long shot, grep was updated to 3.0.

[EDIT] It looks like the "symbols"command is missing in Cygwin while OSX has it. I never heard of that command.
EOSM.202 EOSM.203 EOSM2.103 700D.115 5D3.*

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 9840
  • 5D Mark Free
Re: Compiling Magic Lantern with Cygwin/MinGW-64
« Reply #38 on: May 27, 2017, 08:05:40 PM »
Try:
Code: [Select]
cd tcc   # from magic-lantern root dir
make clean
make

dfort

  • Hero Member
  • *****
  • Posts: 1596
Re: Compiling Magic Lantern with Cygwin/MinGW-64
« Reply #39 on: May 27, 2017, 08:12:33 PM »
Results:

Code: [Select]
[email protected] ~/magic-lantern/tcc
$ make
~/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-gcc-4.8.3.exe -o libtcc.o -c libtcc.c -DTCC_TARGET_ARM -DWITHOUT_LIBTCC -I.  -Os -fno-strict-aliasing -Wno-pointer-sign -Wno-sign-compare -Wno-unused-result
~/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-gcc-4.8.3.exe -o tccpp.o -c tccpp.c -DTCC_TARGET_ARM -DWITHOUT_LIBTCC -I.  -Os -fno-strict-aliasing -Wno-pointer-sign -Wno-sign-compare -Wno-unused-result
~/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-gcc-4.8.3.exe -o tccgen.o -c tccgen.c -DTCC_TARGET_ARM -DWITHOUT_LIBTCC -I.  -Os -fno-strict-aliasing -Wno-pointer-sign -Wno-sign-compare -Wno-unused-result
~/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-gcc-4.8.3.exe -o tccelf.o -c tccelf.c -DTCC_TARGET_ARM -DWITHOUT_LIBTCC -I.  -Os -fno-strict-aliasing -Wno-pointer-sign -Wno-sign-compare -Wno-unused-result
~/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-gcc-4.8.3.exe -o tccasm.o -c tccasm.c -DTCC_TARGET_ARM -DWITHOUT_LIBTCC -I.  -Os -fno-strict-aliasing -Wno-pointer-sign -Wno-sign-compare -Wno-unused-result
~/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-gcc-4.8.3.exe -o tccrun.o -c tccrun.c -DTCC_TARGET_ARM -DWITHOUT_LIBTCC -I.  -Os -fno-strict-aliasing -Wno-pointer-sign -Wno-sign-compare -Wno-unused-result
tccrun.c: In function 'tcc_relocate_ex':
tccrun.c:190:18: warning: #warning FIXME: why does it overflow without this extra RAM when loading the big adtg_gui? [-Wcpp]
                 #warning FIXME: why does it overflow without this extra RAM when loading the big adtg_gui?
                  ^
~/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-gcc-4.8.3.exe -o arm-gen.o -c arm-gen.c -DTCC_TARGET_ARM -DWITHOUT_LIBTCC -I.  -Os -fno-strict-aliasing -Wno-pointer-sign -Wno-sign-compare -Wno-unused-result
~/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-ld -r -o libtcctmp.o libtcc.o tccpp.o tccgen.o tccelf.o tccasm.o tccrun.o arm-gen.o
#~ ~/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-objcopy libtcctmp.a libtcc.a --localize-symbols localsyms
~/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-objcopy libtcctmp.o libtccx.o --localize-symbols localsyms
EOSM.202 EOSM.203 EOSM2.103 700D.115 5D3.*

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 9840
  • 5D Mark Free
Re: Compiling Magic Lantern with Cygwin/MinGW-64
« Reply #40 on: May 27, 2017, 08:22:47 PM »
Hm, no error here...

Code: [Select]
cat localsyms
~/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-readelf libtcctmp.o -Ws
~/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-readelf libtccx.o -Ws

dfort

  • Hero Member
  • *****
  • Posts: 1596
Re: Compiling Magic Lantern with Cygwin/MinGW-64
« Reply #41 on: May 28, 2017, 06:14:08 PM »
Here are my results of those commands.
EOSM.202 EOSM.203 EOSM2.103 700D.115 5D3.*

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 9840
  • 5D Mark Free
Re: Compiling Magic Lantern with Cygwin/MinGW-64
« Reply #42 on: May 28, 2017, 06:31:14 PM »
Okay, tcc_new and the other missing symbols should *not* be in localsyms, but for some reason, they are. You could try running the localsyms rule from Makefile manually (maybe step by step) to see where it fails. Outside Makefile, it would look somewhat like this:
Code: [Select]
readelf libtcctmp.o -Ws | awk "{print \$8}" | sort | uniq | grep -v tcc_new

The output should contain all other symbols, except tcc_new.

dfort

  • Hero Member
  • *****
  • Posts: 1596
Re: Compiling Magic Lantern with Cygwin/MinGW-64
« Reply #43 on: Yesterday at 01:25:33 AM »
Ran that command:

Code: [Select]
readelf libtcctmp.o -Ws | awk "{print \$8}" | sort | uniq | grep -v tcc_new
through each of these listed in the Makefile:

Code: [Select]
# I'm sure there's a cleanest solution, but... I just don't know how
localsyms: libtcctmp.o
@$(READELF) $< -Ws | $(AWK) "{print \$$8}" | sort | uniq \
| grep -v "^tcc_new$$" \
| grep -v "^tcc_delete$$" \
| grep -v "^tcc_add_file$$" \
| grep -v "^tcc_relocate$$" \
| grep -v "^tcc_get_symbol$$" \
| grep -v "^tcc_add_symbol$$" \
| grep -v "^tcc_set_options$$" \
| grep -v "^tcc_load_offline_section$$" \
> $@

In every case the one I ran did not appear in the list. Just to be sure I also tried the ARM version of readelf with the same results:

Code: [Select]
~/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-readelf libtcctmp.o -Ws | awk "{print \$8}" | sort | uniq | grep -v tcc_new
I can see those tcc_* appearing in the Cygwin tcc/localsyms but are not in the Mac tcc/localsyms. So I take it that might be the problem? By the way running "make clean" will not remove the localsyms file though that probably isn't an issue.
EOSM.202 EOSM.203 EOSM2.103 700D.115 5D3.*

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 9840
  • 5D Mark Free
Re: Compiling Magic Lantern with Cygwin/MinGW-64
« Reply #44 on: Yesterday at 01:59:47 AM »
The results of readelf ... grep commands were the same on both Mac and Cygwin?

As a workaround, you can manually delete these symbols from "localsyms", then run "make" again. The localsyms file will not be re-created (unless you also edit some TCC sources), but libtccx.o (which is the one used by ML) will be re-created using your modified localsyms.

dfort

  • Hero Member
  • *****
  • Posts: 1596
Re: Compiling Magic Lantern with Cygwin/MinGW-64
« Reply #45 on: Yesterday at 02:33:44 AM »
The results of readelf ... grep commands were the same on both Mac and Cygwin?

I didn't run the readelf ... grep commands on the Mac. All I did was compare the two localsyms files.

Found the problem and a solution!

The problem is with grep in the current Cygwin distribution. I haven't seen any reports on it but this must have been an issue for a while. Switching to another version will clear up the issue. At one point I was suspecting grep but struck it out because it seemed highly unlikely:

I'm guessing that between September 2015 when I wrote this tutorial and March 2017 when this issue was reported something changed in Cygwin. Maybe one of the tools used in tcc/Makefile? Long shot, grep was updated to 3.0.

Here's what users following this tutorial should do until grep is fixed in Cygwin. Re-run the installer, choose the "Up To Date" view option, search for "grep" then click on the circle with the arrows where it says "Keep" until you find the previous version of grep.



Cygwin setup will replace grep and it should work. You might have to delete tcc/localsyms file. Note that after you successfully build ML at least once you can go back to that buggy version of grep and you'll be able to continue building ML because the localsyms file doesn't get overwritten with each build or even by running "make clean" this may or may not be a good thing. It drove me nuts while trying to get this Cygwin tutorial working again.
EOSM.202 EOSM.203 EOSM2.103 700D.115 5D3.*

dfort

  • Hero Member
  • *****
  • Posts: 1596
Re: Compiling Magic Lantern with Cygwin/MinGW-64
« Reply #46 on: Yesterday at 09:25:01 PM »
Got some answers from the Cygwin mailing list. To sum it up, there's a change coming in grep version 3 that will also affect other platforms that affects the Windows platform. The fix for ML is to do this:
Code: [Select]
-    @$(READELF) $< -Ws | $(AWK) "{print \$$8}" | sort | uniq \
+    @$(READELF) $< -Ws | tr -d '\r' | $(AWK) "{print \$$8}" | sort | uniq \

If you're interested in the nitty gritty details:

https://sourceware.org/ml/cygwin-announce/2017-02/msg00035.html

And the responses from the mailing list:
Code: [Select]
cygwin Digest 29 May 2017 17:14:45 -0000 Issue 10265
...
---------- Forwarded message ----------
From: Daniel Fort <[email protected]>
To: [email protected]
Cc:
Bcc:
Date: Sun, 28 May 2017 23:34:36 -0700
Subject: grep-3.0-2 issues within Makefile
grep-3.0-2 binary will not function as expected when the -v option is
used in a Makefile.

Resolution - downgrade to grep-3.0-1.

When using Cygwin to build Magic Lantern users stated reporting a
build errors on new Cygwin installs around November 2016. The
resolution was to downgrade grep to the previous version.

The discussion and instructions on how to prepare Cygwin to compile
Magic Lantern are in this forum topic:

http://www.magiclantern.fm/forum/index.php?topic=15894.msg154435#msg154435

Using a Cygwin install that includes the default grep-3.0-1 will
result in errors when running the follow Makefile code:

localsyms: libtcctmp.o
    @$(READELF) $< -Ws | $(AWK) "{print \$$8}" | sort | uniq \
        | grep -v "^tcc_new$$" \
        | grep -v "^tcc_delete$$" \
        | grep -v "^tcc_add_file$$" \
        | grep -v "^tcc_relocate$$" \
        | grep -v "^tcc_get_symbol$$" \
        | grep -v "^tcc_add_symbol$$" \
        | grep -v "^tcc_set_options$$" \
        | grep -v "^tcc_load_offline_section$$" \
        > $@


---------- Forwarded message ----------
From: Marco Atzeri <[email protected]>
To: [email protected]
Cc:
Bcc:
Date: Mon, 29 May 2017 10:52:11 +0200
Subject: Re: grep-3.0-2 issues within Makefile
On 29/05/2017 08:34, Daniel Fort wrote:
grep-3.0-2 binary will not function as expected when the -v option is
used in a Makefile.

Please note the last grep announcement
https://sourceware.org/ml/cygwin-announce/2017-02/msg00035.html

and the changes between text and binary mounts.

...
--

and what is the error ?

Regards
Marco


---------- Forwarded message ----------
From: Eric Blake <[email protected]>
To: [email protected]
Cc:
Bcc:
Date: Mon, 29 May 2017 06:39:53 -0500
Subject: Re: grep-3.0-2 issues within Makefile
On 05/29/2017 03:52 AM, Marco Atzeri wrote:
> On 29/05/2017 08:34, Daniel Fort wrote:
>> grep-3.0-2 binary will not function as expected when the -v option is
>> used in a Makefile.
>
> Please note the last grep announcement
> https://sourceware.org/ml/cygwin-announce/2017-02/msg00035.html
>
> and the changes between text and binary mounts.
>

>> Using a Cygwin install that includes the default grep-3.0-1 will
>> result in errors when running the follow Makefile code:
>>
>> localsyms: libtcctmp.o
>>     @$(READELF) $< -Ws | $(AWK) "{print \$$8}" | sort | uniq \

Most likely, $(READELF) is producing \r\n-terminated output. The
solution, then, is to rewrite the line to:

$(READELF) $< -Ws | tr -d '\r' | $(AWK) ...

>
> and what is the error ?

Most likely, grep is not filtering as expected, because now that it is
treating your data as binary rather than text, your explicit $ anchor is
only matching \n instead of \r\n.

--
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org



---------- Forwarded message ----------
From: Eric Blake <[email protected]>
To: [email protected]
Cc:
Bcc:
Date: Mon, 29 May 2017 06:44:07 -0500
Subject: Re: grep-3.0-2 issues within Makefile
On 05/29/2017 06:39 AM, Eric Blake wrote:

>>> localsyms: libtcctmp.o
>>>     @$(READELF) $< -Ws | $(AWK) "{print \$$8}" | sort | uniq \
>
> Most likely, $(READELF) is producing \r\n-terminated output.

That said, WHAT is $(READELF) actually expanding to? If it is the cygwin
binutils version, it should NOT be outputting \r\n in the first place.
Generally, you don't get \r\n output unless you are mixing non-cygwin
programs into the pipeline.

--
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org



---------- Forwarded message ----------
From: Brian Inglis <[email protected]>
To: [email protected]
Cc:
Bcc:
Date: Mon, 29 May 2017 09:11:02 -0600
Subject: Re: grep-3.0-2 issues within Makefile
On 2017-05-29 05:44, Eric Blake wrote:
> On 05/29/2017 06:39 AM, Eric Blake wrote:
>>>> localsyms: libtcctmp.o
>>>>     @$(READELF) $< -Ws | $(AWK) "{print \$$8}" | sort | uniq \
>> Most likely, $(READELF) is producing \r\n-terminated output.
> That said, WHAT is $(READELF) actually expanding to? If it is the cygwin
> binutils version, it should NOT be outputting \r\n in the first place.
> Generally, you don't get \r\n output unless you are mixing non-cygwin
> programs into the pipeline.

Cygwin .o files are not ELF format and not recognized as such by readelf

$ readelf src/astro/sofa/20160503_a/c/build/zr.o -Ws
readelf: Error: Not an ELF file - it has the wrong magic bytes at the start

or file

$ file src/astro/sofa/20160503_a/c/build/zr.o
src/astro/sofa/20160503_a/c/build/zr.o: data

running Cygwin readelf on cross builds which produce ELF .o don't
generate "\r":

$ file util/*.o
util/ntp-keygen.o:      ELF 32-bit LSB relocatable, ARM, EABI5 version 1
(SYSV), not stripped, with debug_info
util/ntp-keygen-opts.o: ELF 32-bit LSB relocatable, ARM, EABI5 version 1
(SYSV), not stripped, with debug_info
util/ntptime.o:         ELF 32-bit LSB relocatable, ARM, EABI5 version 1
(SYSV), not stripped, with debug_info
util/tickadj.o:         ELF 32-bit LSB relocatable, ARM, EABI5 version 1
(SYSV), not stripped, with debug_info
util/version.o:         ELF 32-bit LSB relocatable, ARM, EABI5 version 1
(SYSV), not stripped, with debug_info

$ readelf util/*.o -Ws | wc -lwcL
    460    3479   26678      84
$ readelf util/*.o -Ws | grep $'\r' | wc -lwcL
      0       0       0       0

and it's not the Cygwin mingw binutils

$ /usr/bin/x86_64-w64-mingw32-readelf util/*.o -Ws | grep $'\r' | wc -lwcL
      0       0       0       0

and just to prove this detects "\r"

$ echo $'\r' | grep $'\r' | wc -lwcL
      1       0       2       0

so culprit must be native Mingw binutils readelf.

--
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada


I need to make sure this is working on previous versions of grep and other platforms but it looks like it is a rather benign change.

[EDIT] This issue should only affect the Windows platform and is caused by the Windows line endings created by the pre-compiled ARM toolchain. Mixing non-Cygwin packages in a Cygwin environment is not recommended but the only other option is to build the ARM toolchain from scratch--unless the Linux binary works in the Cygwin environment? Never mind, that didn't work.
EOSM.202 EOSM.203 EOSM2.103 700D.115 5D3.*