Compiling Magic Lantern with Cygwin/MinGW-64

Started by dfort, September 22, 2015, 02:04:07 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

dmilligan


vstrglv

I am trying to compile with
"hg clone -r unified https://bitbucket.org/hudson/magic-lantern
cd magic-lantern/platform/5D3.113/
make clean && make zip",
but there is an error for raw_rec.mo:
[ GCC      ]   raw2dng
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: cannot find -lm
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0//libgcc_s.dll.a when searching for -lgcc_s
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0/libgcc_s.dll.a when searching for -lgcc_s
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0//libgcc_s.dll.a when searching for -lgcc_s
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0/libgcc_s.dll.a when searching for -lgcc_s
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: cannot find -lgcc_s
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0//libgcc.a when searching for -lgcc
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0//libgcc.a when searching for -lgcc
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0/libgcc.a when searching for -lgcc
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0/libgcc.a when searching for -lgcc
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0//libgcc.a when searching for -lgcc
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0/libgcc.a when searching for -lgcc
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: cannot find -lgcc
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: cannot find -lcygwin
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/w32api/libadvapi32.a when searching for -ladvapi32
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/w32api/libadvapi32.a when searching for -ladvapi32
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/w32api/libadvapi32.a when searching for -ladvapi32
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: cannot find -ladvapi32
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/w32api/libshell32.a when searching for -lshell32
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/w32api/libshell32.a when searching for -lshell32
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/w32api/libshell32.a when searching for -lshell32
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: cannot find -lshell32
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/w32api/libuser32.a when searching for -luser32
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/w32api/libuser32.a when searching for -luser32
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/w32api/libuser32.a when searching for -luser32
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: cannot find -luser32
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/w32api/libkernel32.a when searching for -lkernel32
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/w32api/libkernel32.a when searching for -lkernel32
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/w32api/libkernel32.a when searching for -lkernel32
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: cannot find -lkernel32
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0//libgcc_s.dll.a when searching for -lgcc_s
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0/libgcc_s.dll.a when searching for -lgcc_s
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0//libgcc_s.dll.a when searching for -lgcc_s
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0/libgcc_s.dll.a when searching for -lgcc_s
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: cannot find -lgcc_s
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0//libgcc.a when searching for -lgcc
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0//libgcc.a when searching for -lgcc
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0/libgcc.a when searching for -lgcc
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0/libgcc.a when searching for -lgcc
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0//libgcc.a when searching for -lgcc
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-cygwin/5.3.0/libgcc.a when searching for -lgcc
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/ld: cannot find -lgcc
collect2: error: ld returned 1 exit status
Makefile:16: recipe for target 'raw2dng' failed
make[4]: *** [raw2dng] Error 1

********************************************************
WARNING: module raw_rec failed to build, deleting
********************************************************

What is wrong?
Canon 5D3,1.1.3; Canon EOS M,202,  CF-SanDisk Extreme PRO,160MB/s, 256GB, SD-SanDisk Extreme Pro, 170MB/s, 128GB.

dfort

That's odd, it is building raw2dng.exe. As I recall that doesn't build properly with a 64-bit compiler.

I can't reproduce the error because my PC is a very slow and old 32-bit laptop. Here are some things you can try.

Go into the modules/lv_rec directory and open up the file named Makefile with a text editor. Find this line:

all:: raw2dng

and comment it out like this:

# all:: raw2dng

That doesn't solve the issue with being able to build raw2dng with a 64-bit compiler. For that you can look for this block of code:

raw2dng.exe: FORCE
$(call build,MINGW,$(MINGW_GCC) -c $(SRC_DIR)/chdk-dng.c -m32 -mno-ms-bitfields -O2 -Wall -I$(SRC_DIR))
$(call build,MINGW,$(MINGW_GCC) -c raw2dng.c -m32 -mno-ms-bitfields -O2 -Wall -I$(SRC_DIR) -D_FILE_OFFSET_BITS=64)
$(call build,MINGW,$(MINGW_GCC) raw2dng.o chdk-dng.o -o raw2dng.exe -lm -m32)


and change that second "call build" line to include the -std=c99 switch:

$(call build,MINGW,$(MINGW_GCC) -c raw2dng.c -m32 -mno-ms-bitfields -O2 -Wall -I$(SRC_DIR) -D_FILE_OFFSET_BITS=64 -std=c99)

If neither of these works you can delete your 64-bit Cygwin installation and install a 32-bit version in its place.

Please post which "fix" worked for you.

a1ex

Searching for (parts of) the error message is a good idea. Just be careful with dashes, e.g. search for:

x86_64-pc-cygwin/bin/ld: "cannot find -lgcc"


Without quotes, -lgcc in the search string means "return pages that do not contain lgcc".

http://stackoverflow.com/questions/30119573/compile-32bit-code-from-cygwin64

vstrglv

Thank you for reply dfort and alex. dfort, i have tried your fixes, but it does not work. No raw2dng.exe, no raw_rec.mo. May be i have to install cygwin32.
Now commented # all:: raw2dng in modules/raw_rec and raw_rec.mo appeared!
Canon 5D3,1.1.3; Canon EOS M,202,  CF-SanDisk Extreme PRO,160MB/s, 256GB, SD-SanDisk Extreme Pro, 170MB/s, 128GB.

dfort

I don't have a 64-bit PC to test out these issues. On the tutorial I included this:

Quote
...What if you have a 64-bit system?
Quote from: Marsu42 on September 22, 2015, 04:10:34 PM
Note that most module exe helpers are targeted for x86 code, i.e. even if you run cygwin x64 (which you should on a x64 Windows) you need to use the i686 mingw toolchain.
Please leave feedback if you get mingw64-x86-64-gcc-core working and on which command line tools--it is going to need some tweaks to the ML source code.

That might be a little confusing but basically it means that if you are going to compile the command line tools, like raw2dng.exe, it has been tested to work with the 32-bit compiler (i686) but not with the 64-bit compiler (x64).

I wasn't aware that building a platform automatically builds raw2dng but now that I see that does. It was the same with cr2hdr until just recently with this commit. The other popular command line tool, mlv_dump, isn't built when a platform is compiled so I thought that was preferred method. Build rules for raw2dng are in both raw_rec and lv_rec so perhaps we should look into that.

Of course this doesn't resolve another issue which is that raw2dng.exe can't be built with a 64-bit compiler. I'm not sure if this is true for all 64-bit compilers but it is a problem when I cross compile a Windows binary on a Mac:

[ MINGW    ]   raw2dng.exe
raw2dng.c: In function 'find_and_fix_cold_pixels':
raw2dng.c:1213:9: error: 'for' loop initial declarations are only allowed in C99 or C11 mode
         for (int y = 0; y < h; y++)
         ^
raw2dng.c:1213:9: note: use option -std=c99, -std=gnu99, -std=c11 or -std=gnu11 to compile your code
raw2dng.c:1215:13: error: 'for' loop initial declarations are only allowed in C99 or C11 mode
             for (int x = 0; x < w; x++)
             ^
raw2dng.c:1233:5: error: 'for' loop initial declarations are only allowed in C99 or C11 mode
     for (int p = 0; p < cold_pixels; p++)
     ^
raw2dng.c:1243:9: error: 'for' loop initial declarations are only allowed in C99 or C11 mode
         for (int i = -4; i <= 4; i++)
         ^
raw2dng.c:1245:13: error: 'for' loop initial declarations are only allowed in C99 or C11 mode
             for (int j = -4; j <= 4; j++)
             ^
make: *** [raw2dng.exe] Error 1


Simply adding the "-std=c99" switch worked for me but it should be tested on other 64-bit systems.

Quote from: vstrglv on November 22, 2016, 05:33:18 PM
Thank you for reply dfort and alex. dfort, i have tried your fixes, but it does not work. No raw2dng.exe, no raw_rec.mo. May be i have to install cygwin32.
Now commented # all:: raw2dng in modules/raw_rec and raw_rec.mo appeared!

I'm a bit confused with that comment. So commenting out the "all" rule worked for you? This should build the raw_rec module but not raw2dng.exe.

What was it that didn't work, adding the "-std=c99" switch?

@alex - Which compiler are you using on Linux and are you cross compiling Windows binaries? Would adding the "-std=c99" switch to the raw2dng make rules have any negative consequences? It seems to work fine with the Windows binaries that I have compiled on my Mac.

vstrglv

Quote from: dfort on November 22, 2016, 08:11:57 PM
So commenting out the "all" rule worked for you? This should build the raw_rec module but not raw2dng.exe.

What was it that didn't work, adding the "-std=c99" switch?


Yes, if i comment "all", raw_rec module  is built but not raw2dng.exe
If add "-std=c99" switch, raw_rec module  is not built.
But it is possible to compile raw2dng.exe by "make raw2dng.exe" from modules/raw_rec folder. The same for cr2hdr.exe from modules/dual_iso
Canon 5D3,1.1.3; Canon EOS M,202,  CF-SanDisk Extreme PRO,160MB/s, 256GB, SD-SanDisk Extreme Pro, 170MB/s, 128GB.

a1ex

Quote from: dfort on November 22, 2016, 08:11:57 PM
@alex - Which compiler are you using on Linux and are you cross compiling Windows binaries?

The one from Makefile. The nightly build server compiles cr2hdr.exe and mlv_dump.exe, but raw2dng binaries weren't updated lately (maybe I should set up a job for it as well).

QuoteWould adding the "-std=c99" switch to the raw2dng make rules have any negative consequences?

Nope.

BTW, gcc 5.x uses -std=gnu11 by default, whereas gcc 4.x uses -std=gnu89.

dfort

Ok--looks like @vstrglv is able to compile ML so I'll test some Makefile changes that should keep these issues from coming up again.

vstrglv

Quote
Quote from: dfort on November 22, 2016, 09:27:45 PM
Ok--looks like @vstrglv is able to compile ML so I'll test some Makefile changes that should keep these issues from coming up again.
Thank you. Now this error has gone.
Canon 5D3,1.1.3; Canon EOS M,202,  CF-SanDisk Extreme PRO,160MB/s, 256GB, SD-SanDisk Extreme Pro, 170MB/s, 128GB.

dfort

A while back a user came up with a problem: Unable to build ML with Cygwin 32. I'm now helping another user having problems with this tutorial so I decided to delete my Cygwin installation and rebuild it by following the tutorial. The installation seemed to go fine but I got the same compile errors that the other user reported:

make[1]: Leaving directory '/home/Dan/magic-lantern/tcc'
[ CC       ]   module.o
[ AR       ]   strrchr.o
[ AR       ]   dietlibc.a
[ AR       ]   lib_a-setjmp.o
[ AR       ]   newlib-libc.a
[ CP       ]   newlib-libm.a
[ CP       ]   gcc-libgcc.a
[ LD       ]   magiclantern
module.o: In function `module_load_symbols.constprop.2':
module.c:(.text+0x428): undefined reference to `tcc_add_symbol'
module.o: In function `_module_load_all':
module.c:(.text+0x498): undefined reference to `tcc_new'
module.c:(.text+0x4a4): undefined reference to `tcc_set_options'
module.c:(.text+0x4c8): undefined reference to `tcc_delete'
module.c:(.text+0x504): undefined reference to `tcc_delete'
module.c:(.text+0x720): undefined reference to `tcc_delete'
module.c:(.text+0x75c): undefined reference to `tcc_add_file'
module.c:(.text+0x7bc): undefined reference to `tcc_relocate'
module.c:(.text+0x7e8): undefined reference to `tcc_relocate'
module.c:(.text+0x850): undefined reference to `tcc_delete'
module.c:(.text+0x8bc): undefined reference to `tcc_get_symbol'
module.c:(.text+0x8e4): undefined reference to `tcc_get_symbol'
module.c:(.text+0x90c): undefined reference to `tcc_get_symbol'
module.c:(.text+0x934): undefined reference to `tcc_get_symbol'
module.c:(.text+0x95c): undefined reference to `tcc_get_symbol'
module.o:module.c:(.text+0xc4c): more undefined references to `tcc_get_symbol' follow
module.o: In function `_module_load_all':
module.c:(.text+0xcac): undefined reference to `tcc_delete'
module.o: In function `module_load_task':
module.c:(.text+0xf7c): undefined reference to `tcc_load_offline_section'
module.o: In function `module_load':
module.c:(.text+0x1040): undefined reference to `tcc_new'
module.c:(.text+0x104c): undefined reference to `tcc_set_options'
module.c:(.text+0x1070): undefined reference to `tcc_delete'
module.c:(.text+0x1084): undefined reference to `tcc_add_file'
module.c:(.text+0x1098): undefined reference to `tcc_relocate'
module.c:(.text+0x10a8): undefined reference to `tcc_delete'
module.o: In function `module_get_symbol':
module.c:(.text+0x10c8): undefined reference to `tcc_get_symbol'
module.o: In function `module_exec':
module.c:(.text+0x10f0): undefined reference to `tcc_get_symbol'
module.o: In function `module_unload':
module.c:(.text+0x126c): undefined reference to `tcc_delete'
make: *** [../../src/Makefile.src:195: magiclantern] Error 1


I'm at a loss to figure out what is going on here and how to fix it. Just before deleting my previous Cygwin installation I downloaded the latest ML and compiled just fine. Of course I no longer have the previous installation--ugh!

[EDIT]

For what it is worth turning off the module configuration allows compilation to complete but of course no modules are built.

Makefile.user.default
CONFIG_MODULES      = n

a1ex

Full log?

Best guess: I'm using a trick to strip most TCC symbols - everything but the API we use - in tcc/Makefile near localsyms. This file should contain all TCC's symbols except those listed in the error message. Some of the tools used to create this file might not be available on Windows.

dfort

Full log

I'm guessing that between September 2015 when I wrote this tutorial and March 2017 when this issue was reported something changed in Cygwin. Maybe one of the tools used in tcc/Makefile? Long shot, grep was updated to 3.0.

[EDIT] It looks like the "symbols"command is missing in Cygwin while OSX has it. I never heard of that command.

a1ex

Try:

cd tcc   # from magic-lantern root dir
make clean
make

dfort

Results:

Dan@Laptop-PC ~/magic-lantern/tcc
$ make
~/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-gcc-4.8.3.exe -o libtcc.o -c libtcc.c -DTCC_TARGET_ARM -DWITHOUT_LIBTCC -I.  -Os -fno-strict-aliasing -Wno-pointer-sign -Wno-sign-compare -Wno-unused-result
~/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-gcc-4.8.3.exe -o tccpp.o -c tccpp.c -DTCC_TARGET_ARM -DWITHOUT_LIBTCC -I.  -Os -fno-strict-aliasing -Wno-pointer-sign -Wno-sign-compare -Wno-unused-result
~/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-gcc-4.8.3.exe -o tccgen.o -c tccgen.c -DTCC_TARGET_ARM -DWITHOUT_LIBTCC -I.  -Os -fno-strict-aliasing -Wno-pointer-sign -Wno-sign-compare -Wno-unused-result
~/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-gcc-4.8.3.exe -o tccelf.o -c tccelf.c -DTCC_TARGET_ARM -DWITHOUT_LIBTCC -I.  -Os -fno-strict-aliasing -Wno-pointer-sign -Wno-sign-compare -Wno-unused-result
~/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-gcc-4.8.3.exe -o tccasm.o -c tccasm.c -DTCC_TARGET_ARM -DWITHOUT_LIBTCC -I.  -Os -fno-strict-aliasing -Wno-pointer-sign -Wno-sign-compare -Wno-unused-result
~/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-gcc-4.8.3.exe -o tccrun.o -c tccrun.c -DTCC_TARGET_ARM -DWITHOUT_LIBTCC -I.  -Os -fno-strict-aliasing -Wno-pointer-sign -Wno-sign-compare -Wno-unused-result
tccrun.c: In function 'tcc_relocate_ex':
tccrun.c:190:18: warning: #warning FIXME: why does it overflow without this extra RAM when loading the big adtg_gui? [-Wcpp]
                 #warning FIXME: why does it overflow without this extra RAM when loading the big adtg_gui?
                  ^
~/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-gcc-4.8.3.exe -o arm-gen.o -c arm-gen.c -DTCC_TARGET_ARM -DWITHOUT_LIBTCC -I.  -Os -fno-strict-aliasing -Wno-pointer-sign -Wno-sign-compare -Wno-unused-result
~/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-ld -r -o libtcctmp.o libtcc.o tccpp.o tccgen.o tccelf.o tccasm.o tccrun.o arm-gen.o
#~ ~/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-objcopy libtcctmp.a libtcc.a --localize-symbols localsyms
~/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-objcopy libtcctmp.o libtccx.o --localize-symbols localsyms

a1ex

Hm, no error here...


cat localsyms
~/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-readelf libtcctmp.o -Ws
~/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-readelf libtccx.o -Ws


a1ex

Okay, tcc_new and the other missing symbols should *not* be in localsyms, but for some reason, they are. You could try running the localsyms rule from Makefile manually (maybe step by step) to see where it fails. Outside Makefile, it would look somewhat like this:

readelf libtcctmp.o -Ws | awk "{print \$8}" | sort | uniq | grep -v tcc_new


The output should contain all other symbols, except tcc_new.

dfort

Ran that command:

readelf libtcctmp.o -Ws | awk "{print \$8}" | sort | uniq | grep -v tcc_new

through each of these listed in the Makefile:

# I'm sure there's a cleanest solution, but... I just don't know how
localsyms: libtcctmp.o
@$(READELF) $< -Ws | $(AWK) "{print \$$8}" | sort | uniq \
| grep -v "^tcc_new$$" \
| grep -v "^tcc_delete$$" \
| grep -v "^tcc_add_file$$" \
| grep -v "^tcc_relocate$$" \
| grep -v "^tcc_get_symbol$$" \
| grep -v "^tcc_add_symbol$$" \
| grep -v "^tcc_set_options$$" \
| grep -v "^tcc_load_offline_section$$" \
> $@


In every case the one I ran did not appear in the list. Just to be sure I also tried the ARM version of readelf with the same results:

~/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-readelf libtcctmp.o -Ws | awk "{print \$8}" | sort | uniq | grep -v tcc_new

I can see those tcc_* appearing in the Cygwin tcc/localsyms but are not in the Mac tcc/localsyms. So I take it that might be the problem? By the way running "make clean" will not remove the localsyms file though that probably isn't an issue.

a1ex

The results of readelf ... grep commands were the same on both Mac and Cygwin?

As a workaround, you can manually delete these symbols from "localsyms", then run "make" again. The localsyms file will not be re-created (unless you also edit some TCC sources), but libtccx.o (which is the one used by ML) will be re-created using your modified localsyms.

dfort

Quote from: a1ex on May 29, 2017, 01:59:47 AM
The results of readelf ... grep commands were the same on both Mac and Cygwin?

I didn't run the readelf ... grep commands on the Mac. All I did was compare the two localsyms files.

Found the problem and a solution!

The problem is with grep in the current Cygwin distribution. I haven't seen any reports on it but this must have been an issue for a while. Switching to another version will clear up the issue. At one point I was suspecting grep but struck it out because it seemed highly unlikely:

Quote from: dfort on May 27, 2017, 07:33:48 PM
I'm guessing that between September 2015 when I wrote this tutorial and March 2017 when this issue was reported something changed in Cygwin. Maybe one of the tools used in tcc/Makefile? Long shot, grep was updated to 3.0.

Here's what users following this tutorial should do until grep is fixed in Cygwin. Re-run the installer, choose the "Up To Date" view option, search for "grep" then click on the circle with the arrows where it says "Keep" until you find the previous version of grep.



Cygwin setup will replace grep and it should work. You might have to delete tcc/localsyms file. Note that after you successfully build ML at least once you can go back to that buggy version of grep and you'll be able to continue building ML because the localsyms file doesn't get overwritten with each build or even by running "make clean" this may or may not be a good thing. It drove me nuts while trying to get this Cygwin tutorial working again.

dfort

Got some answers from the Cygwin mailing list. To sum it up, there's a change coming in grep version 3 that will also affect other platforms that affects the Windows platform. The fix for ML is to do this:
-    @$(READELF) $< -Ws | $(AWK) "{print \$$8}" | sort | uniq \
+    @$(READELF) $< -Ws | tr -d '\r' | $(AWK) "{print \$$8}" | sort | uniq \


If you're interested in the nitty gritty details:

https://sourceware.org/ml/cygwin-announce/2017-02/msg00035.html

And the responses from the mailing list:
cygwin Digest 29 May 2017 17:14:45 -0000 Issue 10265
...
---------- Forwarded message ----------
From: Daniel Fort <[email protected]>
To: [email protected]
Cc:
Bcc:
Date: Sun, 28 May 2017 23:34:36 -0700
Subject: grep-3.0-2 issues within Makefile
grep-3.0-2 binary will not function as expected when the -v option is
used in a Makefile.

Resolution - downgrade to grep-3.0-1.

When using Cygwin to build Magic Lantern users stated reporting a
build errors on new Cygwin installs around November 2016. The
resolution was to downgrade grep to the previous version.

The discussion and instructions on how to prepare Cygwin to compile
Magic Lantern are in this forum topic:

http://www.magiclantern.fm/forum/index.php?topic=15894.msg154435#msg154435

Using a Cygwin install that includes the default grep-3.0-1 will
result in errors when running the follow Makefile code:

localsyms: libtcctmp.o
    @$(READELF) $< -Ws | $(AWK) "{print \$$8}" | sort | uniq \
        | grep -v "^tcc_new$$" \
        | grep -v "^tcc_delete$$" \
        | grep -v "^tcc_add_file$$" \
        | grep -v "^tcc_relocate$$" \
        | grep -v "^tcc_get_symbol$$" \
        | grep -v "^tcc_add_symbol$$" \
        | grep -v "^tcc_set_options$$" \
        | grep -v "^tcc_load_offline_section$$" \
        > $@


---------- Forwarded message ----------
From: Marco Atzeri <[email protected]>
To: [email protected]
Cc:
Bcc:
Date: Mon, 29 May 2017 10:52:11 +0200
Subject: Re: grep-3.0-2 issues within Makefile
On 29/05/2017 08:34, Daniel Fort wrote:
grep-3.0-2 binary will not function as expected when the -v option is
used in a Makefile.

Please note the last grep announcement
https://sourceware.org/ml/cygwin-announce/2017-02/msg00035.html

and the changes between text and binary mounts.

...
--

and what is the error ?

Regards
Marco


---------- Forwarded message ----------
From: Eric Blake <[email protected]>
To: [email protected]
Cc:
Bcc:
Date: Mon, 29 May 2017 06:39:53 -0500
Subject: Re: grep-3.0-2 issues within Makefile
On 05/29/2017 03:52 AM, Marco Atzeri wrote:
> On 29/05/2017 08:34, Daniel Fort wrote:
>> grep-3.0-2 binary will not function as expected when the -v option is
>> used in a Makefile.
>
> Please note the last grep announcement
> https://sourceware.org/ml/cygwin-announce/2017-02/msg00035.html
>
> and the changes between text and binary mounts.
>

>> Using a Cygwin install that includes the default grep-3.0-1 will
>> result in errors when running the follow Makefile code:
>>
>> localsyms: libtcctmp.o
>>     @$(READELF) $< -Ws | $(AWK) "{print \$$8}" | sort | uniq \

Most likely, $(READELF) is producing \r\n-terminated output. The
solution, then, is to rewrite the line to:

$(READELF) $< -Ws | tr -d '\r' | $(AWK) ...

>
> and what is the error ?

Most likely, grep is not filtering as expected, because now that it is
treating your data as binary rather than text, your explicit $ anchor is
only matching \n instead of \r\n.

--
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org



---------- Forwarded message ----------
From: Eric Blake <[email protected]>
To: [email protected]
Cc:
Bcc:
Date: Mon, 29 May 2017 06:44:07 -0500
Subject: Re: grep-3.0-2 issues within Makefile
On 05/29/2017 06:39 AM, Eric Blake wrote:

>>> localsyms: libtcctmp.o
>>>     @$(READELF) $< -Ws | $(AWK) "{print \$$8}" | sort | uniq \
>
> Most likely, $(READELF) is producing \r\n-terminated output.

That said, WHAT is $(READELF) actually expanding to? If it is the cygwin
binutils version, it should NOT be outputting \r\n in the first place.
Generally, you don't get \r\n output unless you are mixing non-cygwin
programs into the pipeline.

--
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org



---------- Forwarded message ----------
From: Brian Inglis <[email protected]>
To: [email protected]
Cc:
Bcc:
Date: Mon, 29 May 2017 09:11:02 -0600
Subject: Re: grep-3.0-2 issues within Makefile
On 2017-05-29 05:44, Eric Blake wrote:
> On 05/29/2017 06:39 AM, Eric Blake wrote:
>>>> localsyms: libtcctmp.o
>>>>     @$(READELF) $< -Ws | $(AWK) "{print \$$8}" | sort | uniq \
>> Most likely, $(READELF) is producing \r\n-terminated output.
> That said, WHAT is $(READELF) actually expanding to? If it is the cygwin
> binutils version, it should NOT be outputting \r\n in the first place.
> Generally, you don't get \r\n output unless you are mixing non-cygwin
> programs into the pipeline.

Cygwin .o files are not ELF format and not recognized as such by readelf

$ readelf src/astro/sofa/20160503_a/c/build/zr.o -Ws
readelf: Error: Not an ELF file - it has the wrong magic bytes at the start

or file

$ file src/astro/sofa/20160503_a/c/build/zr.o
src/astro/sofa/20160503_a/c/build/zr.o: data

running Cygwin readelf on cross builds which produce ELF .o don't
generate "\r":

$ file util/*.o
util/ntp-keygen.o:      ELF 32-bit LSB relocatable, ARM, EABI5 version 1
(SYSV), not stripped, with debug_info
util/ntp-keygen-opts.o: ELF 32-bit LSB relocatable, ARM, EABI5 version 1
(SYSV), not stripped, with debug_info
util/ntptime.o:         ELF 32-bit LSB relocatable, ARM, EABI5 version 1
(SYSV), not stripped, with debug_info
util/tickadj.o:         ELF 32-bit LSB relocatable, ARM, EABI5 version 1
(SYSV), not stripped, with debug_info
util/version.o:         ELF 32-bit LSB relocatable, ARM, EABI5 version 1
(SYSV), not stripped, with debug_info

$ readelf util/*.o -Ws | wc -lwcL
    460    3479   26678      84
$ readelf util/*.o -Ws | grep $'\r' | wc -lwcL
      0       0       0       0

and it's not the Cygwin mingw binutils

$ /usr/bin/x86_64-w64-mingw32-readelf util/*.o -Ws | grep $'\r' | wc -lwcL
      0       0       0       0

and just to prove this detects "\r"

$ echo $'\r' | grep $'\r' | wc -lwcL
      1       0       2       0

so culprit must be native Mingw binutils readelf.

--
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada



I need to make sure this is working on previous versions of grep and other platforms but it looks like it is a rather benign change.

[EDIT] This issue should only affect the Windows platform and is caused by the Windows line endings created by the pre-compiled ARM toolchain. Mixing non-Cygwin packages in a Cygwin environment is not recommended but the only other option is to build the ARM toolchain from scratch--unless the Linux binary works in the Cygwin environment? Never mind, that didn't work.

dfort

So it looks like some of these issues are Déjà vu.

Removing localsyms was something we did for the lua module:
https://bitbucket.org/hudson/magic-lantern/pull-requests/692/remove-localsyms-in-modules-lua/diff

And we needed to make some tweaks for Magic Lantern to compile on Windows:
https://bitbucket.org/hudson/magic-lantern/pull-requests/660/cr2hdr-build-rules/diff
https://bitbucket.org/hudson/magic-lantern/pull-requests/655/lv_rec-use-fseeko-fseeko64-depending-on/diff
https://bitbucket.org/hudson/magic-lantern/pull-requests/658/build_tools-for-windows/diff

As far as the best way to "fix" the issue of mixing a Windows native ARM toolchain in a Cygwin environment--I've been trying to compile the ARM toolchain in Cygwin but haven't quite got it yet. The easier method would be to just deal with the Windows line endings in the Makefile in a way that doesn't affect other platforms which is something that we already have figured out.

Finally, suggestions for how to improve one line of code keeps coming. This one is from Brian Inglis on the Cygwin mailing list.
I've found moving functionality into awk runs faster than creating one
additional pipe and another process, so to avoid creating multiple
additional pipes and processes, you could change your awk script to do
all the filtering (with appropriate line breaks allowed and quoting
required by your make environment inserted), saving 8 pipes and greps
(some of which could also be saved by a single egrep (grep -E) using "|"
or basic grep using "\|") e.g. (all one-liners)

egrep -v
'/^tcc_(new|delete|add_file|relocate|get_symbol|add_symbol|set_options|load_offline_section)$/'

OR

grep -Ev
'/^tcc_(new|delete|add_file|relocate|get_symbol|add_symbol|set_options|load_offline_section)$/'

OR

grep -v
'/^tcc_\(new\|delete\|add_file\|relocate\|get_symbol\|add_symbol\|set_options\|load_offline_section\)$/'

OR awk

$8 ~ /\r$/ { sub( /\r$/, "", $8)}; $8 !~
/^$|^tcc_(new|delete|add_file|relocate|get_symbol|add_symbol|set_options|load_offline_section)$/
{print $8}

with a bit more work, it can generate a unique list, using associative
array indices, saving 1 pipe and uniq, and a lot of input to and work
for sort (some of which could also be saved using "sort -u" instead of
uniq):

$8 ~ /\r$/ { sub( /\r$/, "", $8)}; $8 !~
/^$|^tcc_(new|delete|add_file|relocate|get_symbol|add_symbol|set_options|load_offline_section)$/)
{++a[$8]}; END {for (i in a) print i}

and if you are using gawk, also sort the indices, giving a sorted array
with the original indices as values, having sequential numeric indices,
saving 1 pipe and uniq (which could also be saved using "sort -u"):

$8 ~ /\r$/ { sub( /\r$/, "", $8)}; $8 !~
/^$|^tcc_(new|delete|add_file|relocate|get_symbol|add_symbol|set_options|load_offline_section)$/)
{++a[$8]}; END {n = asorti(a); for (i = 1; i <= n; ++i) print a[i]}

HTH

a1ex

Very nice. To me, the second version looks the most readable. The awk versions are a lot more complex than ARM assembler :P

And regarding execution time, on a 9-year old laptop:

time make localsyms
real 0m0.040s
user 0m0.004s
sys 0m0.008s


so I don't see a valid reason for optimizing it (it's only called once in the build process).

dfort

Made a pull request that will fix the Cygwin grep 3.0 issue with precompiled Windows ARM toolchain. Also added some other tweaks to the tcc Makefile. Tested on Cygwin and Mac development environments.