Trouble getting ML running on SDXC card with exFAT under GNU+Linux [SOLVED]

Started by alMalsamo, January 12, 2013, 08:27:01 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

alMalsamo

Hallo, first time ML user here, and I am very excited to get going!

I got ML 2.3 working great on a 32GB SDHC card using the firmware update method on my 60D, so I have a small taste of how things should be running!  I normally only use 128GB Lexar SDXC cards in this body, however.  I want to set up every SDXC card I have to make ML run as soon as I load the card, but I can't get my first one working!


I am running Debian GNU/Linux at the moment, and I am trying to use the make_bootable.sh script found in the Google Groups page linked in the instructions PDF.  It is annoying that something seemingly as simple as turning on a bootable flag in the SDXC card's exFAT filesystem is tripping me up, but here is the output I have from.  I had to use the third version, make_bootable3.sh as the other two were not working. Also, I compiled 'exfat_sum.c' and set both the resulting exfat_sum executable and make_bootable3.sh script file +x permissions.


# ./make_bootable3.sh /dev/mmcblk0p1

MagicLantern card pacher Only

Applying EXFAT parameters on /dev/mmcblk0p1 device:
writing EOS_DEVELOP at offset 130
writing BOOTDISK at offset 122
recompute checksum. old=0x0, new=0xf801fffff801ffff


After this finishes, I can no longer mount the card on the computer, nor the camera, and I must reformat again in the camera for it to be functional again.   :'(


Getting ML working on my 128GB cards is a must, so anyone with some tips on how to properly set exFAT filesystems to be bootable under GNU+Linux (or even Mac OS X CLI), please impart your wisdom!


Thanks!

nanomad

EOS 1100D | EOS 650 (No, I didn't forget the D) | Ye Olde Canon EF Lenses ('87): 50 f/1.8 - 28 f/2.8 - 70-210 f/4 | EF-S 18-55 f/3.5-5.6 | Metz 36 AF-5

alMalsamo

Quote from: nanomad on January 12, 2013, 03:44:15 PM
give EOSCard a try and see if it works for you

I don't have a Windoze box around, and it would be a damn shame if I can't set this to be bootable without using a Windoze-only GUI app.  Almost any other filesystem I can make bootable from the CLI easily, but Microsoft just HAD to use their power and money to get the SD Card Association to "standardise" on yet another one of their heavily patented, proprietary filesystems for SDXC as a trojan horse....   /end mini-rant


I guess I will try to run it with Wine.  Or perhaps I should try the firmware update method with the SDXC card?  Is there a reason why the instructions only mention the bootable method for SDXC cards?  Does the firmware update method not work for SDXC cards?

nanomad

EOS 1100D | EOS 650 (No, I didn't forget the D) | Ye Olde Canon EF Lenses ('87): 50 f/1.8 - 28 f/2.8 - 70-210 f/4 | EF-S 18-55 f/3.5-5.6 | Metz 36 AF-5

alMalsamo

I've solved the obstacles to getting exFAT bootable under GNU+Linux!  I never tried running EOSCard with Wine, as I wanted a native solution to get this working.  I will explain what I found out for those who are curious, and provide attachments to the fixed files for those who want something quickly working.

Problem #1: make_bootable scripts have incorrect line breaks

The files on the Google Groups page are problematic and ideally should not be linked to in the official ML instruction PDF and wiki as they are.  The make_bootable scripts are not executable on my systems (I tried on multiple distributions of GNU+Linux) because the endings of the lines are CRLF which stands for a Carriage Return character followed by a Line Feed character.  This is the default newline notification for text files under DOS/Windows, but in Unix-based OSes (including GNU+Linux and Mac OS X), newlines are notified by only a Line Feed (LF) character, with no CR in sight.  Bash was unable to execute the files as they exist on the Google Groups page because it needs all CRLF to be converted to LF in order to sanely determine where the line breaks are.  It's tricky because in a normal text editor, you do not see the difference, but if you use for example the 'file' command on the bash scripts, you will see output similar to this:

make_bootable.sh: Bourne-Again shell script, ASCII text executable, with CRLF line terminators

This last part of the output of 'file' tells us why this script is not executable by default.  To convert CRLF line breaks to LF ones, there is a utility called dos2unix (most likely available as a package of the same name in your distribution of choice) which does the job quickly and easily.  After running this program on the make_bootable script, file now reports it as such:

make_bootable.sh: Bourne-Again shell script, ASCII text executable

With no mention of CRLF in sight.  After doing this change, the script posted by 'arm.indy' on the Google Groups page became executable on my systems.  Unfortunately, while the scripts seem to execute and exit with success, they caused my SDXC cards to not only be unbootable, but completely unmountable and unreadable on both my camera and PC.  Debugging the mounting errors on my PC, I found the fault to lie with the exfat_sum program, which brings me to...


Problem #2: exfat_sum.c was written by a 32-bit OS user, for 32-bit OS users (won't work on 64-bit!)

After getting the make_bootable script working, I found out that the exfat_sum step was calculating the wrong checksum on my systems, which are all running 64-bit operating systems.  The problem lies in the exfat_sum.c source code file, where there are 4 instances of "unsigned long".  long numbers are 32 bits on a 32-bit OS, but they are 64 bits on a 64-bit OS, meaning for those of us on 64-bit OSes will have different (incorrect!) values calculated by the compiled exfat_sum program.  To fix this, I changed every instance of "unsigned long" to "unsigned int" in the source, because int values are always 32 bits no matter what platform you are running.  Changing these lines, recompiling, and running my fixed make_bootable.sh script finally produced a mountable, bootable exFAT filesystem. 

I copied the ML directory and autoexec.bin file to the exFAT filesystem, and Magic Lantern was instantly working on my 128GB cards!  Note that I had previously initialised my camera with a 32GB SDHC card with FAT32.  I do not believe you can skip this step and still have a working SDXC/exFAT card with ML on it, even if you successfully make the exFAT filesystem bootable using the above methods (or other known working methods such as EOSCard utility on Windows).


If you are uninterested in duplicating the steps I gave, I have attached the fixed versions of the 2nd make_bootable.sh script and exfat_sum.c files in this post.  I hope this information or files are useful to other *nix users down the line!

a1ex

The fixed exfat_sum.c is here: https://bitbucket.org/hudson/magic-lantern/src/tip/contrib/make-bootable

I don't understand what changes you did to make_bootable.sh, other than removing the autodetect feature (which works for me). The newlines in the hg version are also correct here.

scrax

Quote from: alMalsamo on January 14, 2013, 05:13:14 PM
... and Magic Lantern was instantly working on my 128GB cards! 

Great, so you used the fixed exfat_sum.c on a card bigger than 16GB without issue?
Cause so far I have reports of only 16GB Exfat card confirmed working.
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus | EOS 5D MLbeta5- EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX, PS, LR, RawTherapee, LightZone -no video experience-

alMalsamo

Quote from: a1ex on January 16, 2013, 10:06:58 AMI don't understand what changes you did to make_bootable.sh, other than removing the autodetect feature (which works for me).

I don't understand why myself, but as I said the files linked on the Google Groups page had CRLF for their line breaks when I downloaded them, when they should be just a LF character.  I didn't otherwise remove any features, so I'm not sure what you are referring to.


Quote from: scrax on January 16, 2013, 10:24:17 PM
Great, so you used the fixed exfat_sum.c on a card bigger than 16GB without issue?
Cause so far I have reports of only 16GB Exfat card confirmed working.

Yes, it's working with no issues on my 128GB SDXC Lexar card with exFAT!

scrax

I know why, you and Alex are not talking about the same file version ;)
The problem with line ending is not present in the make_bootable.sh you can found in the source repo, and in that version there is the code for checking and finding the card automatically so your patched file seems like a reduced version of the one in the code.

Probably the link you followed in the guide has to be updated too.
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus | EOS 5D MLbeta5- EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX, PS, LR, RawTherapee, LightZone -no video experience-