UHS-I / SD cards investigation

[Danne]

This version of mlv_lite.mo should work with 14 bit uncompressed in my build:
https://bitbucket.org/Dannephoto/switch_mini-git/downloads/mlv_lite.mo

[theBilalFakhouri]

Tested a little more: recording as long as possible with 5K frtp setting from above:
1st file stopped after 11.8GB, 4272frames, 2:58min. No error visible in file.
2nd file stopped ("card full", while 36GB are still left) after 11.3GB. I can't open the file, because of error "Invalid blocksize '0', corrupted file".

Okay, that's is good enough to know, thanks for testing!, bad news: don't use this patch

The patch seems to prevent SD card from switching to 48 MHz, but won't solve the real issue, the "instability", at least at this point we might know where to look to be able fix the instability issue


Here is the source code I used if someone needs it:
https://drive.google.com/file/d/1uvDeQeKemtfD_a9m4HOPU30DJYnI7vv2/view?usp=sharing

These lines are the patch for EOS M:

Code Select Expand

if (is_camera("EOSM", "2.0.2"))
{
patch_instruction(0xff642a98, 0x13a01001, 0x13a01000, "0xff642a98");
patch_instruction(0xff6427cc, 0x13a01001, 0x13a01000, "0xff6427cc");
}

[ainu]

bad news: don't use this patch

I tested the module (https://www.magiclantern.fm/forum/index.php?topic=12862.msg236743#msg236743) on two sd cards:
1. Kingston SDA3 64GB, on which previously only the 160 MHz preset worked steadily
2. Sandisk Extreme Pro 256GB (SDSDXXY-256G-GN4IN) all presets work well.

Results:
1. 192 and 240 MHz presets work stably for recording. The benchmark shows a recording speed of ~ 80 Mb\sec (240MHz) in play mode and ~ 71 Mb\sec in video mode, mlv files are recorded and played normally (start recording without run benckmark. If first run benckmark and then try record raw video then "card full" error).
At the same time, the read speed in the benchmark is ~ 850mb\sec (192) and ~ 1100 mb\sec (240). Everything was tested with no photos and videos on the card.
If you take a photo, then when you start the benchmark, a pause appears and the photo is not displayed correctly as the background of the benchmark. However, the write speed is the same 71\80 Mb\sec.
At the same time, the read speed AND write speed are displayed correctly on the 160 MHz preset.
2. Everything works with this card in the same way as before.

In light of this, I assume that some sd cards need other presets and value locks.
I do not believe that with the normal recording of files, an error in their reading (abnormal speed in the benchmark, inability to read a photo from the card) is the fault of the card itself.

[ainu]

I tried to pick up a preset, better than 240 MHz.
I found it:)
https://drive.google.com/file/d/13WHC6LE9u5OFMuLL1dSbbv4zZIICWo1z/view
register 0xC0400614 = 0x1d000201

Look at 620, I set it to 0x0 and its color changed to gray.
Without this change, it was impossible to obtain a stable reading speed (after the first write cycle in the benchmark, the speed of the first read cycle immediately fell).

Results on sd sandisk 256gb extreme pro (170mb\s) with this preset:
Play mode Write 97.4 Read 140 mb\s
https://drive.google.com/file/d/1IU_VMWSVvxCWgftdihUygrNn266wxWw7/view?usp=sharing
Video mode Write 82,2 Read 105,5 mb\s
https://drive.google.com/file/d/1kDcZVdm-Y5UW2qtTqRVdBiQmMnTUpYmg/view?usp=sharing

At the same time, I can record raw videos and files open normally.
Record indicator https://drive.google.com/file/d/13vY8fDGzSZOxx5Z2Mm9c_xiSdnl1ivyI/view?usp=sharing
Raw video settings https://drive.google.com/file/d/18t2nmtp4DPVM8c0COOiHOCXnaf0StwMt/view?usp=sharing
MLV lite screenshot https://drive.google.com/file/d/1cl0Ti0vUkjsC9-yoP4f0unDKOdh0gpMs/view?usp=sharing

Prior to this, 240 mhz speeds were in play mode W 95, R 105.5 mb\s and in video mode W 76 R ~ 85 mb\s.

With kingston sda3 64GB, this preset does not work, because in the current version of the module, where you can change settings, there is no blocking of values.
And as I found out earlier, for this sd card, blocking values ​ ​ is necessary (see the previous post).

Please check this preset on other cameras and sd cards where the 240 mfz preset works normally. You may see improved write and read speeds. If so, share the results on which camera and with which sd card it turned out.


@theBilalFakhouri, can you say when you set the value of the register to 0x0, does it really correspond to this value, or does the register simply remain unchanged and the patch does not apply to it? This is important for understanding what is really happening.

In the process of selecting register values ​ ​ for the preset, I noticed a number of laws that I will try to describe further. These patterns are only assumptions, but you can confirm or refute them, as well as share your experience in selecting values ​ ​ for presets.
The register 0xC0400600 can take values from 0 (?) to F, among which 3 groups are allocated (from faster write speed to lower):
- 0x8, 9, a, b, c, d, e, f
- 0x3
- 0x1, 2, 4, 5, 6, 7
Within each group, if you change the register, the write rate remains the same.

The register 0xC0400604 can take values from 0 (?) to F, among which 4 groups are allocated (from faster write speed to lower):
- 0x3, 7, b, f
- 0x2, 6, a, e
- 0x1, 5, 9, d
- 0x0, 4, 8, c
Within each group, if you change the register, the write rate remains the same.

The 0xC0400610 register can take values from 0 (?) to F, with the following distribution according to the recording speeds in video mode (values for sandisk extreme pro 256gb):
0x1 - 20.7mb\s is a possibly incorrect value that returns the write rate to "48mhz."
0x2 - 82 mb\s
0x3 - 77,1 mb\s
0x4 - 67 mb\s
0x5 - 57,8 mb\s
0x6 - 51,3 mb\s
... - recording speed drops as value increases
0xf - 26,2 mb\s

The 0xC0400614 register can take a lot of values, but the very speed of writing and reading is not directly affected. It affects in general whether the preset will work correctly, or the read/write speed will drop by 20.7-21 mb\s aka "48 mhz mode."
The value of this register may be associated with the value of the 0xC0400610 register.
I will explain by example:
At value of register 0xC0400610 equal 0x2, any value of register 0xC0400614 excellent from 0x1d000201 does prest not working (speed falls up to 20.7 mb\s).
A similar picture in the case of a register value 0xC0400610 0x1 - for it you need 0x1d000001 in the register 0xC0400614.
At the same time, if the register value 0xC0400610 equal to 0x3, 4, etc., the values ​ ​ of the register 0xC0400614, at which the preset will work (the speed does not fall to 20.7 mb\s), there can be a set (without any pattern of distribution).

The register 0xC0400614 can take values from 0 (?) to F. In this case, if the value of the register 0xC0400614 does not match the value of the register 0xC0400610 (with the exception of 0x0), then the write\read speed drops to 20.7 mb\s, making the preset inoperative.

Register 0xC0400618 (I tried all values from 0 to F in different combinations with other registers), registers 0xC0400624, 0xC0400628, 0xC040061C probably have no effect on the write speed\read and the operation of the preset.
For example, I made the following settings (for registers 0xC0400618, 0xC0400624, 0xC0400628, 0xC040061C) and got the same recording speed in video mode 82 mb\s, as if I left the default values.
https://drive.google.com/file/d/1HjVij4y-vICXrqGdpfRnANIQP7qLdEnp/view?usp=sharing

I ask you to check whether these patterns are observed on other cameras and sd cards and publish the results here.

[Danne]

Vey nice findings @ainu .
Tested your numbers on the eos m but couldn´t make it work. Should test further on 5d3 as well. Hopefully @theBilalFakhouri will jump into your findings soon.

[theBilalFakhouri]

Interesting numbers , unfortunately these presets didn't work in 700D:

https://drive.google.com/file/d/13WHC6LE9u5OFMuLL1dSbbv4zZIICWo1z/view
....
https://drive.google.com/file/d/1HjVij4y-vICXrqGdpfRnANIQP7qLdEnp/view?usp=sharing

(it switches to 48 MHz)

I need you to double check your values (just in case if you changed it by accident while making your tests), and perform one more test, your preset in the first link is:

Code Select Expand


static uint32_t uhs_regs[]       = { 0xC0400600, 0xC0400604,/*C0400608, C040060C*/0xC0400610, 0xC0400614, 0xC0400618, 0xC0400624, 0xC0400628, 0xC040061C, 0xC0400620 };
static uint32_t sdr_240MHzPlus[] = {        0x8,        0x3,                             0x2, 0x1D000201,        0x0,      0x201,      0x201,      0x100,        0x0 };

I added it into sd_uhs.mo with name of "240MHz +", here is the module compiled:
https://drive.google.com/file/d/1KQHxJfKnSHP5jy83dIutPW3ZIXeQUEJS/view?usp=sharing

Please select "240MHz +" preset, restart the camera, and run benchmarks in Play mode and share the result (use your Sandisk card).

Here is the source code (just in case):
https://drive.google.com/file/d/1mKRqyzHrGS0xKcRM26tHb4HCIN12ZyKu/view?usp=sharing


@Levas

How "240MHz +" preset performs on your 6D, could you make a test?


@ainu

Good job by experimenting with these, I will take a closer look into the patterns you have founded (currently no much time), keep it up!

[ainu]

Please select "240MHz +" preset, restart the camera, and run benchmarks in Play mode and share the result (use your Sandisk card).

I tested it many times. Results is the same.
Photo mode W 97.5 R 140.5
https://drive.google.com/file/d/14D-tuIO91JulJ5EuGAwRzpRom4KLfSgL/view?usp=sharing
Video mode W 82.5 R 105,4
https://drive.google.com/file/d/1YzaUZtFihtPa_RJUesfsG3ONuFGOhqvK/view?usp=sharing

@theBilalFakhouri, can you add 240+ preset to module with value locks? I want to test it on Kingston SDA3. I think that this sd card can achive comparable read\write speed.

[Danne]

Tested numbers on 5d3 but getting the 5000mb/s write and read speed so what's nicely delimiting 70D speed isn't working with 5d3 seemingly. Still like to see more tests though.

[Walter Schulz]

I'm sure you didn't but have you accidently tested using a full card?

[Danne]

Sandisk 64 gb 170Mb/s card was tested. Plenty space left.
@theBilalFakhouri could you send me a build like the one ainu is using? Also source code for this build since patching routines also seems significant for the tests.

EDIT: Tested with ainus sd_uhs.mo build but still not working better so I guess 70D is the fastest cam with sd slot at the moment.

[theBilalFakhouri]

Ok, cool, thanks for the retesting!

It seems we have Sandisk card reader in our cameras

240MHz Plus preset founded by @ainu on his 70D:



(I reposted this above picture from @ainu from this reply)

And yes, it's:
- 97.5 MB/s write speed
- 140.5 MB/s read speed!

-Preset seems stable on 70D from the benchmarks

@theBilalFakhouri, can you add 240+ preset to module with value locks? I want to test it on Kingston SDA3. I think that this sd card can achive comparable read\write speed.

Some SD cards (like your Kingston card) doesn't accept these high speeds, it mostly won't work, but I can add it (maybe tomorrow) I will PM you . .

Tested numbers on 5d3 but getting the 5000mb/s write and read speed so what's nicely delimiting 70D speed isn't working with 5d3 seemingly. Still like to see more tests though.

5000MB/s or these very high numbers in benchmarks means card can't be accessed, or can't be accessed normally which means the overclock didn't work (mostly it's a Bench.mo bug, can't deal in these type of cases) so there no actual write process happening on the card.

@theBilalFakhouri could you send me a build like the one ainu is using? Also source code for this build since patching routines also seems significant for the tests.

ainu is probably using ArcziPL's custom build, source code same as what we have (crop_rec_4k_mlv_lite_snd), only the difference a working lossless compression for 70D and some fixes for FPS code for 70D, nothing related to patches routine, at some point I asked ArcziPL for his source code, and I got it, will try to find it and will PM you

Raw video settings https://drive.google.com/file/d/18t2nmtp4DPVM8c0COOiHOCXnaf0StwMt/view?usp=sharing

@ainu
You know that you can use 14-bit lossless compression instead of 14-bit uncompressed and this would cut data rate to nearly to half, you don't have to use 14-bit uncompressed (unless you are making write speed tests)

[ainu]

Yes, I use using ArcziPL's custom build. At 14 bit lossless I can record about 2240x... with 240mhz+ preset.

at some point I asked ArcziPL for his source code, and I got it, will try to find it and will PM you

This will be very good! Please PM to me too. I want to make a working movie log when shooting Raw video (current work only on non raw video or raw + h.264 proxy).

May be I'm wrong about registers 0xC0400624, 0xC0400628
I test value 0x1 in 0xC0400620 with 0x2 in 0xC0400610 - all speed drops after first write test (0xC0400624 and 0xC0400628 = 0x201), but if I change both registers 0xC0400624, 0xC0400628 to  0x301, then I achive in Play mode W 96,6 R105,5 mb\s, Video mode W 20,7 R 20,7 mb\s.

Code Select Expand

static uint32_t uhs_regs[]       = { 0xC0400600, 0xC0400604,/*C0400608, C040060C*/0xC0400610, 0xC0400614, 0xC0400618, 0xC0400624, 0xC0400628, 0xC040061C, 0xC0400620 };
static uint32_t sdr_test1[] = {        0x8,        0x3,                             0x2, 0x1D000201,        0x0,      0x301,      0x301,      0x100,        0x1 };

If I set to 0x301 only 0xC0400628, then I achive in Play mode W 96,6 R 140,4 mb\s, Video Mode W 82 R 20.7 mb\s.

Code Select Expand

static uint32_t uhs_regs[]       = { 0xC0400600, 0xC0400604,/*C0400608, C040060C*/0xC0400610, 0xC0400614, 0xC0400618, 0xC0400624, 0xC0400628, 0xC040061C, 0xC0400620 };
static uint32_t sdr_test2[] = {        0x8,        0x3,                             0x2, 0x1D000201,        0x0,      0x201,      0x301,      0x100,        0x1 };

[ainu]

Oh, man's, you don't belive me...
https://drive.google.com/file/d/1JmaM36AY8KLrB3WaIx4fEqvDIVW9LfDq/view?usp=drivesdk
Play mode W 96.2 R 192.1 mb/s and photo files write and read correctly.
Video mode 20.7 
at settings
https://drive.google.com/file/d/1KXFelYhaihiDB28fPYnrOzXyjlqsFjks/view?usp=drivesdk

Edit:Update settings image! Previous picture was wrong.

[Levas]

Tested the build Bilal posted on the 6d.

Quickly benchmarked the 240+ preset in both play and video mode...but both my sandisk cards (95mb/s and 170mb/s) drop to ~20mb/s read/write speed 

Will look into the source and test a little further but I think these 240+ settings won't work for 6d 

[theBilalFakhouri]

This preset (240MHz +) appears to work (partially) in 700D:

Code Select Expand

{ 0xC0400600, 0xC0400604,/*C0400608, C040060C*/0xC0400610, 0xC0400614, 0xC0400618, 0xC0400624, 0xC0400628, 0xC040061C, 0xC0400620 };
{        0x8,        0x3,                             0x2, 0x1D000201,        0x0,      0x100,      0x100,      0x100,        0x2 };

(I applied this patch to prevent the switch to 48 MHz, without it switches to 48 MHz)

-Benchmarks:
Write speed: 91.6 MB/s
Read speed: +800 MB/s (there is a problem when reading to card with this preset, card can't be accessed, it a bit strange because write speed benchmarks are working fine)
Write speed: 91.8 MB/s
Read speed: +4000 MB/s (same thing)
(.ppm file didn't save correctly)

-Benchmarks with regular 240 MHz preset:
Max write speed: ~90.3 MB/s

So there is ~1 MB/s improvement, I think this is the maximum write speed could be achieved with my Sandisk 170 MB/s SD card.

I think above preset may work with 70D too normally as this preset (which was founded by ainu), but not sure, a test is required, could you try it @ainu?

Currently 240 MHz + presets aren't stable for other DIGIC 5 cameras, except for 70D:

TO DO:
Figure out why

It might be from:
-ainu's SD card version, his card might just accept these overclocked values out of the box
-SD overlock registers
-other SD controller parameters (e.g card current (same as power limit?), driver strength... other things?) ــــــــــــــــــــــــــــــــــــــــ> (I am trying to understand these, and how to control it)


If a1ex was around, I would ask him for these two useful things which might help us a lot to understand what is happening:

Xilinx QEMU includes UHS emulation (along with some other nice stuff) and is based on QEMU 2.6.x (at the time of writing). Currently we have patches for QEMU 2.5.0 and 2.9.0, and for the Xilinx version, it will be something in-between. If there is interest, I can commit the patches.

And source code of:

sd_uhs_d.mo (rename to sd_uhs.mo, otherwise it doesn't work for some reason)
Supported models for this diagnostic version: 5D3 1.1.3 and 700D 1.1.5.

700D: it won't perform any overclocking (will only run some tests to print card capabilities returned by CMD6).
5D3 1.1.3: it will only install the basic version of the hack (with 700D parameters) and will run some tests to print card capabilities.

-Currently getting 240MHz + preset to work stably is low priority for me (so currently I won't make further tests with it), I will just stick with regular 240 MHz preset or at least 192 MHz and my interest to make it work on 5D3 (or get 240MHz working on EOS M).


ainu has mentioned that the regular 192/240 MHz presets are working fine with his Sandisk Extreme PRO UHS-I U3 170 MB/s (SDSDXXY-256G-GN4IN):

2. Sandisk Extreme Pro 256GB (SDSDXXY-256G-GN4IN) all presets work well.

[names_are_hard]

From strings in the mo file, it might be this commit: https://foss.heptapod.net/magic-lantern/magic-lantern/-/commit/1d9bdf74695ad9439d8062480366b84e533dfc02

Possibly it is based on that with unpushed changes.

[ainu]

This preset (240MHz +) appears to work (partially) in 700D:

Code Select Expand

{ 0xC0400600, 0xC0400604,/*C0400608, C040060C*/0xC0400610, 0xC0400614, 0xC0400618, 0xC0400624, 0xC0400628, 0xC040061C, 0xC0400620 };
{        0x8,        0x3,                             0x2, 0x1D000201,        0x0,      0x100,      0x100,      0x100,        0x2 };

a test is required, could you try it @ainu?

I try, but no luck - R/W speed 20.7 mb.

Within a few days, I want to get the result of the sandisk 256gb card recording speed test (in the cardrider and usb 3.0 slot) to understand the maximum above which not to step.

Currently 240 MHz + presets aren't stable for other DIGIC 5 cameras, except for 70D:
TO DO:
Figure out why
...
-other SD controller parameters (e.g card current (same as power limit?), driver strength... other things?)

Is it possible to determine these parameters by measuring the supply voltage, current, etc. on the contacts of the card?
I can assemble an extender from the microSD->SD adapter and SD card slot so that I can access the card contacts while using it in the camera.

From strings in the mo file, it might be this commit

Thank you. I will see it later. May be it help me to underdstand what is happening.

[srsa]

And source code of:

sd_uhs_d.mo (rename to sd_uhs.mo, otherwise it doesn't work for some reason)

To get the source code of a binary module, follow this guide.

[names_are_hard]

Huh, interesting.  Seems weird compared to listing the commit hash so you can lookup the whole thing.

Due to the forced repo change, the hash in this module can't be used to find the base commit.  So you get the diff but it's not that useful (unless there's some way to map Heptapod hashes to bitbucket ones?  Last I remember there wasn't).

My local bitbucket hg copy doesn't have the hash so I suspect this is some local branch of Alex's that never got pushed.  But I suck at hg so I could easily be doing something wrong.  Does hg sync everything?  I vaguely recall it doesn't so maybe there was a public branch but I never had it locally?

[theBilalFakhouri]

I try, but no luck - R/W speed 20.7 mb.

Okay, thanks for testing!
(We should really understand how these registers works and what it affect)

From strings in the mo file, it might be this commit: https://foss.heptapod.net/magic-lantern/magic-lantern/-/commit/1d9bdf74695ad9439d8062480366b84e533dfc02

Possibly it is based on that with unpushed changes.

Nah, unfortunately it's not that commit, yeah it's unpushed changes

To get the source code of a binary module, follow this guide.

Thanks for mentioning this! that guide did the trick (I can see the changes, but didn't test it yet), awesome hack to hack the hack

[srsa]

Due to the forced repo change, the hash in this module can't be used to find the base commit.

I have a working copy of the old repo, the output of hg log does list the following changeset:

Code Select Expand

changeset:   17218:b2db3dc9dd4f
branch:      sd_uhs
user:        alex@thinkpad
date:        Wed Apr 04 22:32:31 2018 +0200
summary:     sd_uhs: minor fixes and fine-tuningsThe hash also appears as one of the last commits of the sd_uhs branch on heptapod. Searching for that hash or the commit msg (on the heptapod web interface) gets me no results though.

[theBilalFakhouri]

And source code of:

sd_uhs_d.mo (rename to sd_uhs.mo, otherwise it doesn't work for some reason)

To get the source code of a binary module, follow this guide.

Source code has been restored: sd_uhs_d.c, compiled the module and made a test, output appears to be the same as the sd_uhs_d.mo provided by a1ex

[names_are_hard]

Yay, problem sorted!  Turns out it was the commit I already suggested (but couldn't prove because heptapod didn't preserve commit hashes).

Not very relevant, but my mistake was guessing that "hg log b2db3dc9dd4f" would tell me about that commit (because searching for how to do this didn't get me any useful results...).  Instead it silently does nothing.  "hg log|grep b2db3dc9dd4f" does work.  Hg is weird and I'm glad I don't have to use it.

[Teamsleepkid]

I bought that sd card that was pictured. I hope It helps. I get a lot of drop outs using 5k frtp. Be careful doing some of the stuff in this thread I somehow bricked my camera. Then I freaked out and was going to beg people to help me fix it. But then I found a thread on fixing bricked cameras and fixed mine myself somehow lol. Anyone getting 5k frtp to run reliably? Im on the eos m. Anyway thanks for your work. Oh also it did seem to work really well when I loaded the module. Before a bricked my cam. Thanks for your work:)

[theBilalFakhouri]

@Teamsleepkid
http://www.catb.org/~esr/faqs/smart-questions.html#beprecise

I bought that sd card that was pictured. I hope It helps. I get a lot of drop outs using 5k frtp.

Could you share a picture for the card?
Have you tried running card benchmarks, result?

Be careful doing some of the stuff in this thread I somehow bricked my camera. Then I freaked out and was going to beg people to help me fix it. But then I found a thread on fixing bricked cameras and fixed mine myself somehow lol.

How did you brick your camera, what happened? how did you fix your *bricked* camera?

Anyone getting 5k frtp to run reliably? Im on the eos m.

It should work fine, if the problem is only early recording stop then you should run benchmarks in camera for your card, and see if write/read speeds are stable or not, you may want to format the card in camera (low level format) at least for once to get stable write speed, is there another problem with 5k frtp?

You may consider using lower bit-depths for 5K FRTP, 14-bit lossless might not be continuous. it's off-topic here, report 5k frtp problems in the dedicated thread for Danne's builds for EOS M.

Anyway thanks for your work. Oh also it did seem to work really well when I loaded the module. Before a bricked my cam. Thanks for your work:)

Which module? could give me a link for it.