Folders created by macOS Big Sur on exFAT cards are unreadable in Canon EOS cams

[a1ex]

There were some recent reports from Mac users who couldn't install ML on exFAT cards, when using the latest version of macOS. The first report I've received was from a 60D user who just upgraded from Catalina to Big Sur. Some other reports followed shortly, so I've decided to take a closer look. Tests were done on macOS Big Sur 11.0.1.

Affected models:
- 550D, 60D, 600D, 700D, 100D, EOS M, 5D3 (both SD and CF), 200D, M50 (tested in QEMU).
- 500D does not support exFAT, so this problem does not apply here.
- Likely all other DIGIC 4..8 models that support exFAT (SD or CF, doesn't matter).
- DIGIC X models: not tested, but expecting them to behave just like DIGIC 8.
- PowerShot cameras: not tested.
- non-Canon hardware: not tested, but anything is possible

Steps to reproduce on real hardware:
1) Take a 64GB SD card (or larger)
2) Format it in the camera (550D or newer) => it will be formatted as exFAT.
3) Unzip ML files on the card from macOS Big Sur (which will create autoexec.bin, ML-SETUP.FIR and the ML directory)
4a) Attempt to install ML by running Firmware Update with ML-SETUP.FIR:

Result: "ML directory not found! Please copy all ML files."

The second screenshot contains some ad-hoc diagnostic output, showing that a FIO_FindFirst/FindNext from the camera is able to find the ML directory, but it's not able to read its contents.

4b) Make the card bootable manually (e.g. with make_bootable.sh) and attempt to run ML
Result: ML will not be able to read the contents of the ML subdirectory. It won't be able to read its fonts, modules, scripts and so on.


Steps to reproduce in QEMU:
1) Create an empty 64GB image

Code: [Select]

rm -f sd64.img
truncate -s 64G sd64.img
2) Format it in the virtual camera.
Edit run_canon_fw.sh to use the newly created image, launch the emulation for your favorite DIGIC 4/5 camera (it must boot the GUI in the emulator and it must support exFAT) and format the image. Turn off the virtual camera.
3) Mount the card image into macOS Big Sur.
e.g. start here and add these definitions into OpenCore-Boot.sh:

Code: [Select]

  -device ide-hd,bus=sata.3,drive=SDCard
  -drive id=SDCard,if=none,format=raw,file=/path/to/qemu-eos/sd64.img
4) Using macOS Big Sur, download ML for your favorite DIGIC 4/5 camera (that's all we've got for now) and unzip it on the virtual card.
5) Eject the virtual card and run the emulation.

Shortcut: here's a 64GB SD image already prepared for 60D, which you can use to reproduce this bug. Tip: decompress (unxz) it on a BTRFS filesystem - that way, it will only take a few megabytes on the disk

Once confirmed on a fully supported camera, you can also test on newer models, such as 200D or M50, with a minimal test program, found on the digic6-dumper branch in minimal/qemu-fio, using the modified source: minimal.c

Compile with:

Code: [Select]

hg clone https://foss.heptapod.net/magic-lantern/magic-lantern/
cd magic-lantern/
hg up digic6-dumper -C
cd minimal/qemu-fio
wget https://a1ex.magiclantern.fm/bleeding-edge/BigMac/minimal.c -O minimal.c
# note: make install_qemu won't work on exFAT card images
# mount the 64GB SD image as EOS_DIGITAL, so "make install" will autodetect it
make MODEL=200D CONFIG_QEMU=y install
# run the emulation from the qemu-eos directory

Analysis

The minimal test code linked earlier will output something like this:

Code: [Select]

Trying SD card...
    filename     size     mode     timestamp
--> DCIM         00020000 00000010 30/09/2017 12:15
--> MISC         00020000 00000010 30/09/2017 12:15
--> .fseventsd   00020000 0000003a 13/12/2020 22:33
--> .Trashes     00020000 0000003a 13/12/2020 22:33
--> autoexec.bin 00002900 00000020 13/12/2020 18:28
--> ._autoexec.b 00001000 00000022 13/12/2020 22:35
--> ML           00020000 00000038 03/07/2018 16:20
--> ML-SETUP.FIR 00008d5c 00000020 13/12/2020 17:14
--> ._ML-SETUP.F 00001000 00000022 13/12/2020 22:35
--> ._ML         00001000 00000022 13/12/2020 22:35
Trying DCIM dir...
    filename     size     mode     timestamp
--> 100CANON     00020000 00000010 30/09/2017 12:15
--> EOSMISC      00020000 00000010 30/09/2017 12:15
Trying ML dir...
FIO_FindFirstEx error 1, test failed.

Notice the attribute of the ML directory (0x38), created by macOS Big Sur, and compare it with the attribute of directories created by the camera (0x10). Thanks Lorenzo33324 on the Discord channel, for spotting the difference!

According to the official exFAT specification from Microsoft, the FileAttributes field may use the following bits:


Valid bitmasks for exFAT are:

Code: [Select]

- 0 -> 0x01: ReadOnly
- 1 -> 0x02: Hidden
- 2 -> 0x04: System
- 3 -> 0x08: Reserved1
- 4 -> 0x10: Directory
- 5 -> 0x20: Archive

In our case, the ML directory created by macOS Big Sur has the attributes set to 0x38, meaning: Archive, Directory, Reserved1. This is a problem - the directory created by Big Sur does not have valid attributes.

The exFAT driver from DryOS does not know how to interpret the Reserved1 bit, so it does not recognize ML (created by Big Sur) as a valid directory.

To confirm, I've mounted the SD card image under a Win10 virtual machine and ran the following command on the root directory of the card:

Code: [Select]

attrib -a ML

Result: the ML directory became readable for the above test code, which was ran on the virtual camera.

You can get the same outcome by manually patching the attribute of the ML directory. In the attached 64GB card image (sd64.img), patch the byte at offset 0x20402A4 from 0x38 to 0x30, recompute the checksum (e.g. with fsck.exfat) and the ML directory becomes readable for the camera.

Caveat: the above workarounds will not "magically" fix your filesystem in order to use Magic Lantern. For that, you'd also have to modify the attributes of all subdirectories of the ML directory.

Another test you may want to run: delete the DCIM directory from the exFAT card and re-create it from Big Sur.
Expected outcome: camera won't be able to save the images.
On 60D, I've got an error at startup: "Card cannot be accessed". This will force you to format the card, losing any data you might have there. The filesystem was fully readable under Linux (FUSE exfat 1.3.0) and Windows 10.

ML bug? Apple bug? Canon bug?

As the issue can be reproduced with plain Canon firmware, it's clearly not a bug in ML.

Microsoft says this about the "reserved" bits:


What happens is that:
1) Apple used a bit declared as "reserved" in the exFAT specification. Whether it was intentional or by mistake, I have no idea.
2) Canon interpreted that bit as in "this is not a directory".

Therefore... both Apple and Canon seem to have misused the exFAT specification. Thanks g3gg0

Conclusion

The current version of macOS Big Sur - at least 11.0.1 - creates directories with invalid attributes on exFAT. While the filesystem drivers from other operating systems, like Windows or Linux, will tolerate these invalid attributes, the exFAT driver from Canon cameras will not.

In other words, the directories created from Big Sur 11.0.1 will not be recognized as valid directories in Canon EOS cameras.

This will affect users who will try to install Magic Lantern on exFAT cards, regardless of the camera generation, from DIGIC 4 until at least DIGIC 8, likely also DIGIC X (but not tested).

Regular users are unlikely to notice this bug, as triggering it requires the user to *create* a directory from macOS Big Sur, and to use it somehow in the camera.

This is not a ML bug. I can attempt to find a workaround as time permits, but... no guarantees.

[Walter Schulz]

Big Sur 11.2 has the same issue.
Tested and reproduced with 650D.

[dionhouston]

Hi all,

Thank you for this extremely helpful post.  I tried a variety of things to remove extended file attributes with no success, but then I discovered a pretty easy workaround.

If you format the SD card to FAT32, and follow the regular install instructions, it will install.  You can then immediately format the card, and ML will then change the format to ExFAT and life once again is normal.

Hope this helps someone else that may come here.  Definitely not an ML bug -- Canon should be able to ignore it.  But don't think I'm going to get many more firmware updates for this 70D to fix it

Dion

[Walter Schulz]

Thanks! We kind of noted your workaround, just not here. Would you mind, a1ex? ;-)
https://mobile.twitter.com/autoexec_bin/status/1338228845103689729
https://www.reddit.com/r/MagicLantern/comments/jzlnlf/ml_directory_not_found/gfpyeis/?utm_source=reddit&utm_medium=web2x&context=3

[Walter Schulz]

Anyone around able to test if this issue still exists with macOS 13.x Monterey?

How to test without re-install:
Create a folder (name with 8 characters or less) on your ML enabled card with ExFAT file system.
Insert card to camera
Enable file_man.mo in Modules tab.
Restart camera
Access Debug tab -> File Manager
Try to locate the folder you created.

Report results, please!

[Walter Schulz]

Canon EOS 7D which can read exFAT cards.

My EOS 7D cannot use ExFAT and it cannot use SD cards. Please check again.

[DeafEyeJedi]

Thanks for pointing me to this thread @Walter Schulz. Following this just in case I run into this again.

I had just recently upgraded to Monterey after being on Catalina for several years. Skipped Big Sur.

This workaround is doable. Thank you @a1ex and everyone else who brought this up!