ExFAT on compatible cams: Forced ExFAT format with cards 8-32 GByte?

[Walter Schulz]

Hello!

Cams with ExFAT support will treat cards with 32 GByte or less like this:

- If cam is formatting: You get FAT32.
- If card is formatted with a cardreader and ExFAT file system: Cam will read/write without problem.

So: If you want to use ExFAT's perks with cards 8 - 32 GByte and format the card inside cam you have to -
1) insert card into cardreader,
2) format it with ExFAT,
3) copy ML installation files to card,
4) insert card into cam and run firmware update
5) restart cam
And if you were clever you copied ML directory contents in step 1.5 or you have to run through some additional steps to make your MLV/RAW record work your way.

Q:
Is it possible to bypass Canon's limitation and enable the cam to format with ExFAT for cards with 8-32 GByte?
If not: Maybe a warning displayed in the format menu if a 32 GByte card (or less) with ExFAT is detected?


I can life without such options, of course. Quite a low priority.

[a1ex]

I believe - but I'm not yet sure - that the decision for what filesystem to use might be around "DryEFatFormat(%d)".

Example from 5D3 (variable names are from Canon strings if in CamelCase, or my own guess if in lower_case):

Code: [Select]

  ...
  if ( CardType == 0x1000 && Blks > 0x10000000 )    // if CF > 128GB?
    CardType = 0x2000; // treat it as a SD card?
  ...
  if (CardType == 0x1000) // if CF ?
     fs_type = 0; // FAT?
  else if (CardType == 0x2000 && Blks >= 0x4000000) // if SD > 32GB ?
     fs_type = 4;  // ExFAT?
  else
     fs_type = 0;

Looks a bit too simplistic, but it seems to be kinda consistent with some user reports:

Both of my 5D3s with firmware 1.2.3 format 64GB cards as follows: CF 64GB: FAT32. SD:exFAT.

I formatted both CF cards, the 64GB in my 123 and the 128GB in my 113, and both were "reduced" to fat32
[...]
Of course the major weakness with my argument is, that I haven't tried cards larger than 128GB.

Another snippet that chooses partition type for SD:

Code: [Select]

if blocks < 0x7FA8           // 15.95M
  fs_id = 1;                 // FAT12 < 16MB
elseif blocks < 0x10000      // 32M
  fs_id = 4;                 // FAT16 < 32M
elseif blocks < 0x400200     // 2048.25M
  fs_id = 6;                 // FAT16 > 32M
elseif blocks < 0xFB0400     // 8032M
  fs_id = 11;                // FAT32
elseif blocks < 0x4000000    // 32768M
  fs_id = 12;                // FAT32 LBA
else
  fs_id = 7;                 // ExFAT

For CF, the code is a little different, but there is a table that shows fs_type for various block sizes:

Code: [Select]

> 0x800      => 0
> 0x4000     => 1
> 0x10000    => 2
> 0x4000000  => 3
> 0xFFFFFFFE => 3

Anyone has a 256GB CF card and a 5D3, to check what filesystem it chooses? I believe CF > 128GB is handled with the SD routines in Canon code (that would give ExFAT) but I'm not 100% sure.

[DeafEyeJedi]

256MB CF card, did I read that right?

If so, I'll see if my Grandfather has any since he has been using CF cards on his Nikons DSLR's since the 90's... If not then I can probably scoop one up online for a decent price.

[a1ex]

256GB, typo

( the card I use for many ML experiments is actually a 256MB one )

I'm actually interested in formatting a 64GB SD as FAT32 from the camera. Reason: mounting ExFAT on my Linux system is not exactly straightforward (have to type the mount commands manually, because the GUI doesn't recognize it, and then I have to fiddle with permissions to be able to read/write from it as a regular user, without sudo).

[Walter Schulz]

650D handles SD-cards as described above. 32GB and below: FAT32. Bigger: ExFAT

[a1ex]

I think I've just formatted a 16 GB SD card as EXFAT from 5D3

Anyone else interested in this feature?

[Walter Schulz]

650D? Please? Pretty, pretty please?

[axelcine]

@a1ex - I'd love it

[DeafEyeJedi]

Would love this feature as well. Could this be duplicated for the CF slot on the 5D3 as well?

[kontrakatze]

If it would be possible to format the cf internally to exFAT, the problem with mlv_play and more than 10 chunks/files would be obsolete, or am I wrong? Right now I'm hesitating to change the format to exFAT, because if it gets hectic I usually format the cf internally instead of deleting all files manually. But this would change the format to FAT32 again and the option of mlv_rec  to create files bigger than 4GB would lead to trouble.

[Walter Schulz]

No, you are right. Workaround while waiting for a nightly build: Change of workflow would help.
- The card is in your cardreader. Delete DCIM folder after transferring files.

[a1ex]

Well, that bug will be still present, but it will be a bit easier to work around it.

Update: the patch worked for a 16GB CF as well (5D3).

Update2: also formatted a 64GB SD as FAT32, so I can finally use it without console black magic

[a1ex]

Uploaded the source: https://bitbucket.org/hudson/magic-lantern/src/card_fmt/modules/card_fmt/

I'm not going to provide a binary without some serious field testing, because this is a bit like playing Russian Roulette with your data. The only testing I did was copying ML files on the card and checking if it still loads. I don't even fully understand the code I've patched.

I did some sanity checks, using an experimental library - patch manager. So, if you want to add support for another camera, chances to patch the wrong thing are very small.

I encourage you to review the patch before running it (Debug -> Memory Patches).

Compatibility:
- 5D3 1.1.3 (tested with 16GB SD/CF formatted as exFAT, and 64GB SD formatted as FAT32)
- 5D3 1.2.3 (not tested)
- 650D 1.0.4 (not tested)

Compiling:
- the module won't work with the nightly - you need to compile the full ML from the card_fmt branch.
- for 5D3.123, you need to merge the 5D3-123 branch (hg update card_fmt && hg merge 5D3-123)

Hacking:
- to print Canon messages during format, compiling dm-spy-experiments is helpful (but those messages are not exactly verbose).
- by adding custom hooks dm-spy-extra.c, you can increase the verbosity a bit (example available for 5D3.113)

Code: [Select]

   FileMgr:ff482d98:21:03: TotalSectors:0x7438000
   FileMgr:ff482dac:21:03: EraseSectors:0x100000
   FileMgr:000b71f8:00:00: *** sd_choose_filesystem(0x7438000, 0x14fd48), from ff729fc0
   FileMgr:000b71f8:00:00: *** sd_choose_filesystem_fs_type(0x0), from ff729fc0
   FileMgr:000b71f8:00:00: *** sd_choose_filesystem_fs_type_2(0x0, 0x3), from ff729fc0
   FileMgr:000b71f8:00:00: *** sd_choose_filesystem_ret(0x1), from ff729e70
   FileMgr:000b71f8:00:00: *** sd_format_params(0x0), from ff729e70
   FileMgr:000b71f8:00:00: *** DryEFatGetSystemAreaLength_ret(0xa040), from ff729e70
   FileMgr:ff482e1c:21:03: Erase:0x00000000 - 0x000fffff
...
   FileMgr:ff5bb814:21:05: DryEFatFormat(1)
   FileMgr:000b71f8:00:00: *** DryEFatFormat(0x6420c, 0x7438000, 0x0, 0x1, 0x0, 0x0, 0x14fd88), from ff5bb8f8
   FileMgr:000b71f8:00:00: *** sd_choose_filesystem(0x7438000, 0x14fd18), from ff729fc0
   FileMgr:000b71f8:00:00: *** sd_choose_filesystem_fs_type(0x0), from ff729fc0
   FileMgr:000b71f8:00:00: *** sd_choose_filesystem_fs_type_2(0x0, 0x3), from ff729fc0
   FileMgr:000b71f8:00:00: *** sd_choose_filesystem_ret(0x1), from ff729e70
...
   FileMgr:ff2bbb68:24:03: FIO_Format End

Test reports welcome.

[Walter Schulz]

Dang! Old noob issue: Can compile ML card_fmt branch without modules but "make card_fmt" fails with Error 1.

[a1ex]

Code: [Select]

$ hg clone https://bitbucket.org/hudson/magic-lantern/
$ cd magic-lantern/
magic-lantern$ hg update card_fmt
magic-lantern$ cd platform/5D3.113/
magic-lantern/platform/5D3.113$ make
magic-lantern/platform/5D3.113$ cd ../../modules/card_fmt/
magic-lantern/modules/card_fmt$ make
Will load on:
    5D3
Not checked (compile ML for these cameras first):
    1100D, 500D, 50D, 550D, 5D2, 600D, 60D, 650D, 6D, 700D, 7D, EOSM


[mothaibaphoto]

Iso-research merged with unified has reduced recording times even without adtg_gui loaded. Is that due patch manager? Please pay attention to possible performance downgrade.

[Walter Schulz]

Thanks, a1ex! First 32 GByte ExFAT format out-of-the-cam.

This one is showing in Debug tab:

[kontrakatze]

Wow! Impressive! To bad, I have no idea how I could test this. My last programming experiences are lots of years in the past....
Is it a module? If yes, can I just download it somewhere, put it to modules on the latest nightly so I can test it on my cards (several Lexar 128gb, 5d Mark III, 1.1.3)?

[Walter Schulz]

Not that difficult if you run OS X. I followed dfort's instructions and I really have no idea how to setup a development machine. If you are running Windows: I followed budafilms' and reddeercity's posts about Hackintosh (= OS X on standard PC hardware) and have 2 machines now. One dual boot OS X/W7, one dedicated. You might get same results (slower, though) running a virtual Mac in Oracle VM on Windows.
And it is not "only a module" but a dedicated build and very experimental. If a1ex says he does not fully understand the parts he patched ... well ... it's not for production and not even for "just testing". Take care!
It's not intended to be used with other modules. Correct, a1ex?

[a1ex]

Iso-research merged with unified has reduced recording times even without adtg_gui loaded. Is that due patch manager?

On the code side, the only thing I can think of is the "small hacks/extra hacks" (now done via patch manager) failed to apply, though I've checked it and seems fine. Are there any errors in Debug -> Memory patches? Were you on 1.1.3 or 1.2.3?

There are no other things (such as extra tasks or things running in background) that could affect recording performance.

Maybe this "modified" filesystem results in slower card access. Make sure you account for this variable in your experiment.

Otherwise, please provide numbers.

It's not intended to be used with other modules. Correct, a1ex?

Shouldn't interfere.

[Walter Schulz]

Shouldn't interfere.

I know you have reservations about Lua.mo ...

Code: [Select]

Scanning modules ...
Load modules ...
   [i] load: CARD_FMT.MO
   [i] load: lua.mo
Linking ...
tcc: error: undefined symbol 'console_puts'
  [E] failed to link modules

[a1ex]

Seems to work fine here, both in unified and card_fmt branch...

[Walter Schulz]

User error, again. Mixed up nightly build and card_fmt branch files. Works now.

[axelcine]

Wow - what an effort you excellent people do towards managing the card formatting possibilities. Amazing.

I tried my little card tricks with my 700D: An old 32GB 300x fat32 card was reformatted in the cam: Fat32. Then I formatted a brand new 32 GB Lexar 1000X, which was exFat'ted just a few days ago - one of the cards I use with my 5DIII.123 btw. Result: Fat32.

OK. Forced exFat on the old 32GB card via my PC, reinstalled the next-to-latest ML (Feb 24 - hadn't seen the Feb 28!!!), fir'red the good old 700D: ML runs happily on an exfatted old 300x SD-card - ridiculous of course, cause the card won't do any good with serious moviemaking in raw.

When I returned my fast Lexar card to my 5DIII.123 after reformatting to exFat and reinstalling ML - the cam wouldn't eat the .fir update. No way, Sir. Don't like the taste.

OK, the reformatting circus probably destroyed the hidden "signature file". Reformatted card in 5D: Fat32 - no surprise, but the cam got to set her fingerprint. Back in PC, reformat to exFat, reinstall ML, and this time the 5D ate it all up with a satisfied burp in the end. Now the signature was alright.

Got to think about it. I didn't use low level formatting, either in cams or on PC, so obviously some kind of file thing has of course survived the formatting, which isn't the same as a wipe.

[mothaibaphoto]

On the code side, the only thing I can think of is the "small hacks/extra hacks" (now done via patch manager) failed to apply, though I've checked it and seems fine.

As far as I can understand "small hacks/extra hacks" keep working (i.e, AWB and AE disabled or not).
DualISO is the only failed thing i've noticed.

Are there any errors in Debug -> Memory patches? Were you on 1.1.3 or 1.2.3?
Maybe this "modified" filesystem results in slower card access. Make sure you account for this variable in your experiment.
Otherwise, please provide numbers.

No errors, i'm on 1.1.3, I cloned iso-research 11 feb and didn't update since then, so no "modified" filesystem.
I see alot of changes, I should update then repeat my testing.
Edit:
I still didn't update but made speed test - "unified" vs "patch manager patched unified" as it was on 11 feb.
Everything the same - settings, warmed cards, 30 fps non-crop mode, adtg_gui not loaded in patched version:
17 vs 26 sec average.