UHS-I / SD cards investigation

[Danne]

This was never necessary, but the quick tests performed during patching are not sufficient to make sure the preset is reliable.

Besides this, you have to make sure there will be absolutely no other card access during the initial tests. The code is not thread safe and any kind of card access during the inital tests, from either Canon firmware or ML, will lock up the camera and may result in a corrupted filesystem. Should this happen (camera locking up), you must format the card (even if it appears to work fine at first sight, it's almost certainly not OK and some of your files are probably already lost at that point).

I'm not saying this just for kicks; I've actually experienced corrupted filesystems many times while testing this module. If in doubt, please review the code before running it, and make sure you are not using it for anything important. Not joking. There is a reason why this is not available as a regular download.

P.S. more stuff coming soon (in particular, better compatibility with certain Sandisk cards). Still experimenting. No success with thread safety yet.

P.P.S. Not all 95MB/s cards are compatible with the hardcoded 160MHz preset. I've just got two of these.

Thanks for shedding some light on this topic, especially on this

make sure there will be absolutely no other card access during the initial tests.

Looking forward to what you have up your sleeve regarding sandisk cards .

P.S. more stuff coming soon (in particular, better compatibility with certain Sandisk cards). Still experimenting. No success with thread safety yet.

Anyone with lua knowledge that could shed light on this?

@Danne...I initially had the same issue as @OlRivRat where I had to change the name of the lua file to get it to appear in the scripts menu, but that was easy to fix.

[Levas]

Besides this, you have to make sure there will be absolutely no other card access during the initial tests. The code is not thread safe and any kind of card access during the inital tests, from either Canon firmware or ML, will lock up the camera and may result in a corrupted filesystem. Should this happen (camera locking up), you must format the card (even if it appears to work fine at first sight, it's almost certainly not OK and some of your files are probably already lost at that point).

That one is new for me, until now, nothing bad happened here.
Does Canon firmware or ML do much random accessing the card ?

[a1ex]

Generally speaking, card access is quite predictable - look at the LED blinks. For example, if you switch to photo mode during the overclocking tests, Canon firmware may want to read from the card in order to display the last photo. Another example: ML settings are saved every time after you change some setting in ML menu. Or, if you start recording before the tests are completed.

Finally, if you use the Lua script and there are other scripts that happen to auto-load (or read their config files or whatever) during the overclocking tests, you may have a little surprise.

Being a race condition, the outcome is not deterministic; it doesn't mean that you will get a lockup every single time a card access is made during these tests. However, the low-level tests are not thread-safe (we have to tell Canon code not to use the card during these tests, and I don't know how to do that). If the other card access happens in the middle of the low-level write test, for example, the camera will lock up and the sectors written during that test (which may or may not overlap existing files or directories on the card) will contain garbage.

[alpicat]

@a1ex  Thanks for your work on this and the warning about card access, that's very useful information. So maybe it's safer not to use the Lua script (to autorun the overclock test) as I guess it adds even more to the risk.

[Danne]

Added three new builds(eosm, 100D, 700D) top of downloads section:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/

I noticed my embarrasing lua script wasn´t even doing what it should do so rewrote the lot of it. Maybe a little less embarrasing today  :

Code: [Select]

-- enable SD overclocking

    console.hide()
    if menu.get("Debug", "SD overclock", "MAY CAUSE DATA LOSS") == "MAY CAUSE DATA LOSS"
    then
    if menu.get("Movie", "RAW video", "") == "OFF"
    then
    display.notify_box("enabling of SD overclocking")
    menu.set("Debug", "SD overclock", "ON")
    msleep(1000)
    display.notify_box("please wait...")
    msleep(2000)
    display.notify_box("patching...")
    msleep(2000)
    display.notify_box("done!")
    else
    menu.set("Movie", "RAW video", "OFF")
    display.notify_box("enabling of SD overclocking")
    menu.set("Debug", "SD overclock", "ON")
    msleep(1000)
    display.notify_box("please wait...")
    msleep(2000)
    display.notify_box("patching...")
    msleep(3000)
    menu.set("Movie", "RAW video", "ON")
    display.notify_box("done!")
    end
    else
    display.notify_box("Please enable sd_uhs.mo")
    msleep(2000)
    display.notify_box("Please enable sd_uhs.mo")
    msleep(2000)
    return
    end
The script will now look if RAW video is set to off or not and will temporarily turn the setting off while patching, then reeanble RAW video again when done.
I tried meet a condition while RAW video was set to ON but lua seems rather picky here. For instance:

Code: [Select]

if menu.get("Movie", "RAW video", "") == "ON" will not work since it also needs what else is written after ON:

Code: [Select]

if menu.get("Movie", "RAW video", "") == "ON, 1736x584" will work. Is there some other way to check for when a menu item is on or not?   

[a1ex]

Lots of examples on this one in api_test.lua.

[Danne]

Thank you. Examples needed.

[Danne]

Posted new builds for these cams:
magiclantern-Nightly.2018May26.6D116sd_uhs.zip
magiclantern-Nightly.2018May26.70D112sd_uhs.zip
magiclantern-Nightly.2018May26.100D101sd_uhs.zip
magiclantern-Nightly.2018May26.650D104sd_uhs.zip
magiclantern-Nightly.2018May26.700D115sd_uhs.zip
magiclantern-Nightly.2018May26.EOSM202sd_uhs.zip


Find it here(includes enabling SD overclocking patch lua script):
https://bitbucket.org/Dannephoto/magic-lantern/downloads/

As always. Any risks included are your responsiblity, not mine. And the more feedback the merrier...

[Tony Weller]

magiclantern-Nightly.2018May26.EOSM202sd_uhs.zip

Working fine 

[theBilalFakhouri]

It's also working flawlessly on 700D. Thanks @Danne!

[dfort]

I also tested Danne's build on the 700D. Super easy to set it up.

• Turn on lua and sd_uhs modules
• Set SDoverclock.lua script to Autorun

It adds to the startup time but other than that it just runs in the background without having to think about it.

Been a while since we've seen card benchmarks so here is a set using the SanDisk Extreme Pro 32GB UHS-I (95/90MB/s read/write speed, video speed C10, U3, V30)

Photo Mode (normal)


Photo Mode (with sd_uhs module)


Movie Mode (normal)


Movie Mode (with sd_uhs module)


Of course the real test is how it performs in the wild so I'm loading up Danne's build on all my cards and taking it out on a three week trip. I usually just shoot stills when on vacation but I'll try shooting some raw video this time. Hope to come home with something to share.

[EDIT] Well maybe I won't set this up on all the cards. A 128GB SanDisk card showed pretty much the same speed improvements as the 32GB card of the same make/model but a couple of Komputerbay 64GB cards are showing something very different.

komputerbay 64GB Photo Mode (normal)


komputerbay 64GB Photo Mode (with sd_uhs module)


The write speed does improve but it is almost half the speed of the SanDisk card. Sorry, another example of you get what you pay for.

[andy kh]

i think the previous build works better for me on 70D. latest build crash twice. i shoot for an hour(not continues) yesterday with the 25May build without crashing. i wil do more test and report

[Danne]

@andy_kh
Would be nice to know a little more about your crashes. What settings used etc. Reproduceable or not? Any logs?
It´s more or less the same build except for changing this:
https://bitbucket.org/Dannephoto/magic-lantern/commits/19ca4bea793d0fcdb3756ddd21b41b326f4d9c99
By the way. Can you send me your 25May build to look at?
Cannot see anything that would affect the 70D build but who knows.

@dfort
Have a nice trip. Right now I added some more stuff to my own lua script for faster action. Since I mostly play around with the 5x zoom in 2520x1080p setting I put this at the bottom of the script. On my eosm it´s a bit of a hassle getting the 5x zoom, so easy letting lua do it instead:

Code: [Select]

while camera.mode ~= MODE.MOVIE do
   display.notify_box("enable MOVIE mode")
   msleep(1000)
end

    menu.set("Movie", "RAW video", "ON")   
    camera.shutter.value = 1/50
    camera.iso.value=400
-- ready to enter x5 crop mode
    lv.zoom = 5
-- fixme: doesn't seem to work if you have fine-tuned it
    menu.set("RAW video", "Resolution", 2560)
    menu.set("Overlay", "Global Draw", "OFF")
    menu.set("Display", "Clear overlays", "OFF")
    menu.set("RAW video", "Data format", "12-bit lossless")
Noticed I set resolution to 2560? That´s because my cams don´t have the 2520 in the scroll down menu.

I then manually set FPS override to around 24. Would really like to get that menu item to work with lua too but it seems very unstable. For instance:

Code: [Select]

menu.set("FPS override", "Desired FPS", "24.002")Will set 16.003 fps on my cam. Maybe it´s something in cam not updating correctly when going through lua. Needs some more understanding.
https://www.magiclantern.fm/forum/index.php?topic=14828.msg199769#msg199769

[50mm1200s]

So, I got a Lexar Pro (1066x) 32GB, and it's able to record only 4 seconds on MLV at 50D's maximum res (without crop, I think it's 1556x10xx or something). I don't know why, but I was not able to record more than these 4 seconds, while my Komputerbay 64GB is able to record up to 2min in the same resolution.

I'm posting this here because I could try anything on this card, since I have no much use for it. If @a1ex or @dford want to fry this one in some crazy experiment, feel free to send me a module for 50D and I'll do it.

[OlRivrRat]

                    @Danne

           My SL1 has also had a few "Soft Bricks" with 26May Build. Can't say for sure what was in process when it happens >

It's random ~ I'll try to pay closer attention.

[Levas]

Maybe not really the right topic, BUT I think it fits best here, for the user group.

Danne compiled some nice builds with the simplified SD_UHS overclock module.
And my guess is that the users of this build would be happy if they could select 10bit lossless for all iso's.

Few months ago I altered the MLV_lite module a little so I could select 8/9/10 or 11 bit on every iso.
I was just curious as hell how 8 bit raw looks like on iso 100 
And since then I know, shadows can look horrible posterised.
So disclaimer, Alex is right in how he implemented the lossless bit options, that is for maximum quality.
But I think 10 bit lossless is usable for video on all iso's, it's worse then 14 bit ofcourse, but 10 bit raw video isn't a bad option in video world.
I even think the 8 bit lossless can be used for modern art like projects or video clips, it has some posterised style which some could like 

So here's the source of this MLV_Lite
https://drive.google.com/file/d/1VcIMktzPpt_c5m1uIDtuU9pf8WlQ1Zow/view?usp=sharing
And here's a compiled version:
https://drive.google.com/file/d/1CJ7hXialyM7dYra6CUYSwyh8Dkl5bkm7/view?usp=sharing
And here is how the menu looks like.


Not sure if it works on all cams, I guess so, maybe Danne could check it out and help you out.

[Danne]

Nice Levas. Will check it out. You should work from bitbucket branch stuff for easier overview. Even the sd_uhs speed up patching is coming from your code to begin with . What branch did you add your changes in mlv_lite.c by the way? The one with audio working hopefully?

If you want to check 8bit output you can actually do that directly with mlv_dump(on steroids) since buncyball got that feature working a while ago with -b option. So just pass a 14bit file and select -b 8 and you´ll see how shitty it looks .

Edit: checking hg diff it seems you worked from the audio branch...

[Danne]

Bunch of new builds uploaded, thanks to Levas. Check two posts above. Not something I missed but could be good to have in the future:
magiclantern-Nightly.2018May28.6D116sd_uhs_allbits.zip
magiclantern-Nightly.2018May28.70D112sd_uhs_allbits.zip
magiclantern-Nightly.2018May28.100D101sd_uhs_allbits.zip
magiclantern-Nightly.2018May28.650D104sd_uhs_allbits.zip
magiclantern-Nightly.2018May28.700D115sd_uhs_allbits.zip
magiclantern-Nightly.2018May28.EOSM202sd_uhs_allbits.zip

All builds in this downloads section:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/

Commit:
https://bitbucket.org/Dannephoto/magic-lantern/commits/9b53eaef80680d3f131d1309ab62184267f654b1

new branch:
https://bitbucket.org/Dannephoto/magic-lantern/branch/crop_rec_4k_mlv_lite_snd_sd_uhs_HDR_ext_all_bits

I encourage testers to compile for themselves. For mach users I simplified compiling with this tool:
Compiler.app
https://www.magiclantern.fm/forum/index.php?topic=21882.msg199370#msg199370

[andy kh]

Lossless isnt working for the 70D yet. But stil i wil test the new build

[Levas]

@Danne, wow you're quick 
If I'm not mistaken I altered the mlv_lite in March, from the crop_rec_4K build, so indeed with audio.
It should be almost the same as the crop_rec_4K builds on the experiments page here on the ML site (which are from March 10th 2018).

If you want to check 8bit output you can actually do that directly with mlv_dump(on steroids) since buncyball got that feature working a while ago with -b option. So just pass a 14bit file and select -b 8 and you´ll see how shitty it looks .

Damn, there was an easier way of checking how 8 bit raw looks in iso 100 

I must check on the options in MLV_dump more often, I recently find out that you could also export in lossless dng's instead of the full size dng's, gonna be saving a lot of diskspace here 

[Danne]

Yes, lossless option works great. If I'm not mistaken much of the code is derived from mlrawviewer compression code A Baldwin created. Bouncyball refined some code implementation from "footage app" guy and then polishing, polishing...

[theBilalFakhouri]

Ran a brute force (random) search for the above registers, and...

SDR104 @ 160MHz

Code: [Select]

static uint32_t uhs_regs[]     = { 0xC0400600, 0xC0400604,/*C0400608, C040060C*/0xC0400610, 0xC0400614, 0xC0400618, 0xC0400624, 0xC0400628, 0xC040061C, 0xC0400620 };   /* register addresses */
static uint32_t sdr50_700D[]   = {        0x3,        0x3,                             0x4, 0x1D000301,        0x0,      0x201,      0x201,      0x100,        0x4 };   /* SDR50 values from 700D (96MHz) */
static uint32_t sdr_80MHz[]    = {        0x3,        0x3,                             0x5, 0x1D000401,        0x0,      0x201,      0x201,      0x100,        0x5 };   /* underclocked values: 80MHz = 96*(4+1)/(5+1) */
static uint32_t sdr_120MHz[]   = {        0x3,        0x3,                             0x3, 0x1D000201,        0x0,      0x201,      0x201,      0x100,        0x3 };   /* overclocked values: 120MHz = 96*(4+1)/(3+1) */
static uint32_t sdr_132MHz[]   = {        0x2,        0x2,                             0x2, 0x1D000201,        0x0,      0x100,      0x100,      0x100,        0x2 };   /* overclocked values: 132MHz?! (found by brute-forcing) */
static uint32_t sdr_160MHz[]   = {        0x2,        0x3,                             0x1, 0x1D000001,        0x0,      0x100,      0x100,      0x100,        0x1 };   /* overclocked values: 160MHz = 96*(4+1)/(2?+1) (found by brute-forcing) */


Maybe a dumb question

@a1ex Can you explain what the math of this calculations ?
e.g. How in 120 MHz mode did you calculated the values from the registers ?  /* overclocked values: 120MHz = 96*(4+1)/(3+1) */
Which register have the value 96 ? and why did you multiply it with (4+1) ? and what is the registers for (4+1) ? etc..

Can you explain what the math ? How and what's happening ?

[a1ex]

See reply #41. At first, things appear to make sense, but once you start tweaking some more registers, things are no longer obvious and frequency was guessed from maximum read speed.

The presets I've tried last week (take with a grain of salt; not yet double-checked):

Code: [Select]

static uint32_t sdr50_700D[]   = {        0x3,        0x3,                             0x4, 0x1D000301,        0x0,      0x201,      0x201,      0x100,        0x4,               0    };   /* SDR50 values from 700D (96MHz) */
static uint32_t sdr_96MHz_b[]  = {        0x3,        0x3,                             0x4, 0x1D000301,        0x0,      0x201,      0x201,        0x0,        0x4,               0    };

static uint32_t sdr_80MHz_a[]  = {        0x3,        0x3,                             0x5, 0x1D000401,        0x0,      0x201,      0x201,        0x0,        0x5,               0    };   /* underclocked values: 80MHz = 96*(4+1)/(5+1) */

static uint32_t sdr_120MHz_a[] = {        0x3,        0x3,                             0x3, 0x1D000201,        0x0,      0x201,      0x201,      0x100,        0x3,               0    };   /* overclocked values: 120MHz = 96*(4+1)/(3+1) */
static uint32_t sdr_120MHz_b[] = {        0x3,        0x3,                             0x3, 0x1D000201,        0x0,      0x201,      0x201,        0x0,        0x3,               0    };

static uint32_t sdr_132MHz_a[] = {        0x2,        0x2,                             0x2, 0x1D000201,        0x0,      0x100,      0x100,      0x100,        0x2,               0    };   /* overclocked values: 132MHz?! */
static uint32_t sdr_132MHz_b[] = {        0x5,        0x2,                             0x2, 0x1D000001,        0x0,      0x201,      0x301,      0x100,        0x2,               0    };   /* overclocked values: this one works on SanDisk Extreme Pro 32GB 95MB/s V30 (5D3) */
static uint32_t sdr_132MHz_c[] = {        0x2,        0x2,                             0x2, 0x1D000200,        0x1,      0x101,      0x100,    0x10100,    0x10003,               0    };

static uint32_t sdr_160MHz_a[] = {        0x2,        0x3,                             0x1, 0x1D000001,        0x0,      0x100,      0x100,      0x100,        0x1,               0    };   /* overclocked values: 160MHz = 96*(4+1)/(2?+1) (found by brute-forcing) */
static uint32_t sdr_160MHz_b[] = {        0x3,        0x3,                             0x2, 0x1D000200,        0x1,      0x301,      0x100,    0x10100,    0x10003,               0    };
static uint32_t sdr_160MHz_c[] = {        0x3,        0x3,                             0x2, 0x1D000200,        0x1,      0x301,      0x100,    0x10000,    0x10003,               0    };
static uint32_t sdr_160MHz_d[] = {        0x3,        0x3,                             0x2, 0x1D000200,        0x1,      0x301,      0x100,        0x0,    0x10003,               0    };


[theBilalFakhouri]

@a1ex

I'd like to play with values without changing it in the code and recompiling it every time.

Can I adjust these registers by adtg_gui? if not, Can anyone knows the coding better than me make these registers adjustable via sd_uhs with a submenu ? then we can run an overclocking with the settings we did 

[dinobike]

Had anyone tried the hack with this one??