ExFAT on compatible cams: Forced ExFAT format with cards 8-32 GByte?

Started by Walter Schulz, December 02, 2014, 05:39:55 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Walter Schulz

Hello!

Cams with ExFAT support will treat cards with 32 GByte or less like this:

- If cam is formatting: You get FAT32.
- If card is formatted with a cardreader and ExFAT file system: Cam will read/write without problem.

So: If you want to use ExFAT's perks with cards 8 - 32 GByte and format the card inside cam you have to -
1) insert card into cardreader,
2) format it with ExFAT,
3) copy ML installation files to card,
4) insert card into cam and run firmware update
5) restart cam
And if you were clever you copied ML directory contents in step 1.5 or you have to run through some additional steps to make your MLV/RAW record work your way.

Q:
Is it possible to bypass Canon's limitation and enable the cam to format with ExFAT for cards with 8-32 GByte?
If not: Maybe a warning displayed in the format menu if a 32 GByte card (or less) with ExFAT is detected?


I can life without such options, of course. Quite a low priority.

a1ex

I believe - but I'm not yet sure - that the decision for what filesystem to use might be around "DryEFatFormat(%d)".

Example from 5D3 (variable names are from Canon strings if in CamelCase, or my own guess if in lower_case):

  ...
  if ( CardType == 0x1000 && Blks > 0x10000000 )    // if CF > 128GB?
    CardType = 0x2000; // treat it as a SD card?
  ...
  if (CardType == 0x1000) // if CF ?
     fs_type = 0; // FAT?
  else if (CardType == 0x2000 && Blks >= 0x4000000) // if SD > 32GB ?
     fs_type = 4;  // ExFAT?
  else
     fs_type = 0;


Looks a bit too simplistic, but it seems to be kinda consistent with some user reports:

Quote from: tron on February 24, 2016, 11:13:16 PM
Both of my 5D3s with firmware 1.2.3 format 64GB cards as follows: CF 64GB: FAT32. SD:exFAT.

Quote from: axelcine on February 24, 2016, 07:40:59 PM
I formatted both CF cards, the 64GB in my 123 and the 128GB in my 113, and both were "reduced" to fat32
[...]
Of course the major weakness with my argument is, that I haven't tried cards larger than 128GB.

Another snippet that chooses partition type for SD:

if blocks < 0x7FA8           // 15.95M
  fs_id = 1;                 // FAT12 < 16MB
elseif blocks < 0x10000      // 32M
  fs_id = 4;                 // FAT16 < 32M
elseif blocks < 0x400200     // 2048.25M
  fs_id = 6;                 // FAT16 > 32M
elseif blocks < 0xFB0400     // 8032M
  fs_id = 11;                // FAT32
elseif blocks < 0x4000000    // 32768M
  fs_id = 12;                // FAT32 LBA
else
  fs_id = 7;                 // ExFAT


For CF, the code is a little different, but there is a table that shows fs_type for various block sizes:

> 0x800      => 0
> 0x4000     => 1
> 0x10000    => 2
> 0x4000000  => 3
> 0xFFFFFFFE => 3


Anyone has a 256GB CF card and a 5D3, to check what filesystem it chooses? I believe CF > 128GB is handled with the SD routines in Canon code (that would give ExFAT) but I'm not 100% sure.

DeafEyeJedi

256MB CF card, did I read that right?

If so, I'll see if my Grandfather has any since he has been using CF cards on his Nikons DSLR's since the 90's... If not then I can probably scoop one up online for a decent price.
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

a1ex

256GB, typo :P

( the card I use for many ML experiments is actually a 256MB one :P )

I'm actually interested in formatting a 64GB SD as FAT32 from the camera. Reason: mounting ExFAT on my Linux system is not exactly straightforward (have to type the mount commands manually, because the GUI doesn't recognize it, and then I have to fiddle with permissions to be able to read/write from it as a regular user, without sudo).

Walter Schulz

650D handles SD-cards as described above. 32GB and below: FAT32. Bigger: ExFAT

a1ex

I think I've just formatted a 16 GB SD card as EXFAT from 5D3 :D

Anyone else interested in this feature?

Walter Schulz


axelcine

EOS RP, 5dIII.113/Batt.grip, 5dIII.123, 700d/Batt.Grip/VF4 viewfinder + a truckload of new and older Canon L, Sigma and Tamron glass

DeafEyeJedi

Would love this feature as well. Could this be duplicated for the CF slot on the 5D3 as well?
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

kontrakatze

If it would be possible to format the cf internally to exFAT, the problem with mlv_play and more than 10 chunks/files would be obsolete, or am I wrong? Right now I'm hesitating to change the format to exFAT, because if it gets hectic I usually format the cf internally instead of deleting all files manually. But this would change the format to FAT32 again and the option of mlv_rec  to create files bigger than 4GB would lead to trouble.

Walter Schulz

No, you are right. Workaround while waiting for a nightly build: Change of workflow would help.
- The card is in your cardreader. Delete DCIM folder after transferring files.

a1ex

Well, that bug will be still present, but it will be a bit easier to work around it.

Update: the patch worked for a 16GB CF as well (5D3).

Update2: also formatted a 64GB SD as FAT32, so I can finally use it without console black magic :)

a1ex

Uploaded the source: https://bitbucket.org/hudson/magic-lantern/src/card_fmt/modules/card_fmt/

I'm not going to provide a binary without some serious field testing, because this is a bit like playing Russian Roulette with your data. The only testing I did was copying ML files on the card and checking if it still loads. I don't even fully understand the code I've patched.

I did some sanity checks, using an experimental library - patch manager. So, if you want to add support for another camera, chances to patch the wrong thing are very small.

I encourage you to review the patch before running it (Debug -> Memory Patches).

Compatibility:
- 5D3 1.1.3 (tested with 16GB SD/CF formatted as exFAT, and 64GB SD formatted as FAT32)
- 5D3 1.2.3 (not tested)
- 650D 1.0.4 (not tested)

Compiling:
- the module won't work with the nightly - you need to compile the full ML from the card_fmt branch.
- for 5D3.123, you need to merge the 5D3-123 branch (hg update card_fmt && hg merge 5D3-123)

Hacking:
- to print Canon messages during format, compiling dm-spy-experiments is helpful (but those messages are not exactly verbose).
- by adding custom hooks dm-spy-extra.c, you can increase the verbosity a bit (example available for 5D3.113)


   FileMgr:ff482d98:21:03: TotalSectors:0x7438000
   FileMgr:ff482dac:21:03: EraseSectors:0x100000
   FileMgr:000b71f8:00:00: *** sd_choose_filesystem(0x7438000, 0x14fd48), from ff729fc0
   FileMgr:000b71f8:00:00: *** sd_choose_filesystem_fs_type(0x0), from ff729fc0
   FileMgr:000b71f8:00:00: *** sd_choose_filesystem_fs_type_2(0x0, 0x3), from ff729fc0
   FileMgr:000b71f8:00:00: *** sd_choose_filesystem_ret(0x1), from ff729e70
   FileMgr:000b71f8:00:00: *** sd_format_params(0x0), from ff729e70
   FileMgr:000b71f8:00:00: *** DryEFatGetSystemAreaLength_ret(0xa040), from ff729e70
   FileMgr:ff482e1c:21:03: Erase:0x00000000 - 0x000fffff
...
   FileMgr:ff5bb814:21:05: DryEFatFormat(1)
   FileMgr:000b71f8:00:00: *** DryEFatFormat(0x6420c, 0x7438000, 0x0, 0x1, 0x0, 0x0, 0x14fd88), from ff5bb8f8
   FileMgr:000b71f8:00:00: *** sd_choose_filesystem(0x7438000, 0x14fd18), from ff729fc0
   FileMgr:000b71f8:00:00: *** sd_choose_filesystem_fs_type(0x0), from ff729fc0
   FileMgr:000b71f8:00:00: *** sd_choose_filesystem_fs_type_2(0x0, 0x3), from ff729fc0
   FileMgr:000b71f8:00:00: *** sd_choose_filesystem_ret(0x1), from ff729e70
...
   FileMgr:ff2bbb68:24:03: FIO_Format End


Test reports welcome.

Walter Schulz

Dang! Old noob issue: Can compile ML card_fmt branch without modules but "make card_fmt" fails with Error 1.

a1ex


$ hg clone https://bitbucket.org/hudson/magic-lantern/
$ cd magic-lantern/
magic-lantern$ hg update card_fmt
magic-lantern$ cd platform/5D3.113/
magic-lantern/platform/5D3.113$ make
magic-lantern/platform/5D3.113$ cd ../../modules/card_fmt/
magic-lantern/modules/card_fmt$ make
Will load on:
    5D3
Not checked (compile ML for these cameras first):
    1100D, 500D, 50D, 550D, 5D2, 600D, 60D, 650D, 6D, 700D, 7D, EOSM

mothaibaphoto

Iso-research merged with unified has reduced recording times even without adtg_gui loaded. Is that due patch manager? Please pay attention to possible performance downgrade.

Walter Schulz

Thanks, a1ex! First 32 GByte ExFAT format out-of-the-cam.

This one is showing in Debug tab:


kontrakatze

Wow! Impressive! To bad, I have no idea how I could test this. My last programming experiences are lots of years in the past....
Is it a module? If yes, can I just download it somewhere, put it to modules on the latest nightly so I can test it on my cards (several Lexar 128gb, 5d Mark III, 1.1.3)?

Walter Schulz

Not that difficult if you run OS X. I followed dfort's instructions and I really have no idea how to setup a development machine. If you are running Windows: I followed budafilms' and reddeercity's posts about Hackintosh (= OS X on standard PC hardware) and have 2 machines now. One dual boot OS X/W7, one dedicated. You might get same results (slower, though) running a virtual Mac in Oracle VM on Windows.
And it is not "only a module" but a dedicated build and very experimental. If a1ex says he does not fully understand the parts he patched ... well ... it's not for production and not even for "just testing". Take care!
It's not intended to be used with other modules. Correct, a1ex?

a1ex

Quote from: mothaibaphoto on February 28, 2016, 05:51:37 PM
Iso-research merged with unified has reduced recording times even without adtg_gui loaded. Is that due patch manager?

On the code side, the only thing I can think of is the "small hacks/extra hacks" (now done via patch manager) failed to apply, though I've checked it and seems fine. Are there any errors in Debug -> Memory patches? Were you on 1.1.3 or 1.2.3?

There are no other things (such as extra tasks or things running in background) that could affect recording performance.

Maybe this "modified" filesystem results in slower card access. Make sure you account for this variable in your experiment.

Otherwise, please provide numbers.

Quote from: Walter Schulz on February 28, 2016, 07:03:14 PM
It's not intended to be used with other modules. Correct, a1ex?

Shouldn't interfere.

Walter Schulz

Quote from: a1ex on February 28, 2016, 07:18:49 PM
Shouldn't interfere.

I know you have reservations about Lua.mo ...
Scanning modules ...
Load modules ...
   [i] load: CARD_FMT.MO
   [i] load: lua.mo
Linking ...
tcc: error: undefined symbol 'console_puts'
  [E] failed to link modules

a1ex

Seems to work fine here, both in unified and card_fmt branch...

Walter Schulz

User error, again. Mixed up nightly build and card_fmt branch files. Works now.

axelcine

Wow - what an effort you excellent people do towards managing the card formatting possibilities. Amazing.

I tried my little card tricks with my 700D: An old 32GB 300x fat32 card was reformatted in the cam: Fat32. Then I formatted a brand new 32 GB Lexar 1000X, which was exFat'ted just a few days ago - one of the cards I use with my 5DIII.123 btw. Result: Fat32.

OK. Forced exFat on the old 32GB card via my PC, reinstalled the next-to-latest ML (Feb 24 - hadn't seen the Feb 28!!!), fir'red the good old 700D: ML runs happily on an exfatted old 300x SD-card - ridiculous of course, cause the card won't do any good with serious moviemaking in raw.

When I returned my fast Lexar card to my 5DIII.123 after reformatting to exFat and reinstalling ML - the cam wouldn't eat the .fir update. No way, Sir. Don't like the taste.

OK, the reformatting circus probably destroyed the hidden "signature file". Reformatted card in 5D: Fat32 - no surprise, but the cam got to set her fingerprint. Back in PC, reformat to exFat, reinstall ML, and this time the 5D ate it all up with a satisfied burp in the end. Now the signature was alright.

Got to think about it. I didn't use low level formatting, either in cams or on PC, so obviously some kind of file thing has of course survived the formatting, which isn't the same as a wipe.
EOS RP, 5dIII.113/Batt.grip, 5dIII.123, 700d/Batt.Grip/VF4 viewfinder + a truckload of new and older Canon L, Sigma and Tamron glass

mothaibaphoto

Quote from: a1ex on February 28, 2016, 07:18:49 PM
On the code side, the only thing I can think of is the "small hacks/extra hacks" (now done via patch manager) failed to apply, though I've checked it and seems fine.
As far as I can understand "small hacks/extra hacks" keep working (i.e, AWB and AE disabled or not).
DualISO is the only failed thing i've noticed.
Quote from: a1ex on February 28, 2016, 07:18:49 PM
Are there any errors in Debug -> Memory patches? Were you on 1.1.3 or 1.2.3?
Maybe this "modified" filesystem results in slower card access. Make sure you account for this variable in your experiment.
Otherwise, please provide numbers.
No errors, i'm on 1.1.3, I cloned iso-research 11 feb and didn't update since then, so no "modified" filesystem.
I see alot of changes, I should update then repeat my testing.
Edit:
I still didn't update but made speed test - "unified" vs "patch manager patched unified" as it was on 11 feb.
Everything the same - settings, warmed cards, 30 fps non-crop mode, adtg_gui not loaded in patched version:
17 vs 26 sec average.