Messages

the 5k worked at ~1:23 seconds before the buffer filled up @ 14 bits.
Will have to try how long it does constant recordings, looks like @Gabriielangel mentioned 3 minutes. Did the camera auto stop or did you stop it?

On that particular test, the lvface+aewb hack was enabled,  and I stopped it myself at 3 minutes  (And I used the Sandisk Extreme Pro 170-128GB, with the regular sandisk extreme 256GB, I got about 1 minute ).
If you want to record longer, use 12bits, which will give you around 20 minutes continuous if your scene is not too bright. Like @theBilalFakhouri said, record time are scene and ISO dependent

Slightly off-topic and shape of things to come. Stable recording? Nope, just benchmark!
Running with 280 MHz now:

Is this build available to test for EOS M? I just received my 140MB/s Rated Write card. Might as well test it while I'm doing the resolution tests.

A new card tested today on EOS M:
Sandisk Extreme Pro 200MB/s-256GB
PNM: SR256
OID: 2022/06
Rated R/W: 200/140 MB/s

Tested with Danne's Latest Build Sept29-2022


Tested with Latest Module on Bilal's post.


Records on par with the earlier Sandisk Extreme Pro Cards.

Thank you again to everyone involved!

[Data Rate Upper Limits]

Data Rate Upper Limits Warning: Long Post Ahead

Like I said in the previous post, I would like to create a test which can be easily and quickly replicated by anyone, in order to have a coherent comparison between the performances of various cards at different resolutions.

Because filming the LCD Screen leads to moiré artifacts which biases the results, I went outside to film various scenes at 2.5K to get a baseline. I will then create a pattern which can be printed on plain paper and filmed with the camera, with enough details to generate files of comparable size to real life recordings.

This test will also check where the "Data Rate Overload" occurs in real life (See my previous post)

The motivation for going through this is as follows:

1- I just purchased a high-capacity 200MB/s card to test the upper limits, which cost me half the price of the Camera.
2- The Students I help would rather spend that money on beer...
3- It took me 4 months of reading and testing before being able to use that EOS M to its full potential. This negates any low-cost benefit afforded by the cam. But if there was a set of guidelines written in a few posts, newcomers would be able to use the cam to its full potential in a fraction of the time (And with cheaper cards, if necessary).
4-It's always nice to have a list of several alternative cards. For example, the latest Sandisk Extreme Pro did not work well until the latest Overclock module.

Short Answer:
1- A test pattern which can generate 70-80MB/s files at the target exposure and 90MB/s+ at ettr would simulate real-life conditions.
2- The Bench.mo would be a lot closer to real-life use if it also ran a sustained-write test (1-2 min?)

3- With 240MHz Overclock, there is no need to heavily underexpose the image to get decent recording times at 12bits, and 14bits with little restrictions (2.5K tested here, but theoretically applies to 5k frtp and 1080p).

I avoid recording at 10 bits, as it gives a reddish tint to the noise in the image, and more noise overall. So 10bits should be carefully exposed to minimize the noise, and 12/14 bits should be preferred if there are lots of dark regions in the image.

Look at scene 5 below to see a real-life example of exposure triggering a frame corruption message.

Details:
Exposure being a key factor in achieving near-continuous recordings (Or long enough clips), let's establish what I consider a bright-enough, well exposed image. In the examples, I also included graded versions, so that it shows that the tests are not unreasonably brightly exposed.

I had the scopes in my MLV App modified to make it easier to evaluate the image. I left those in every screenshot, so that it shows that no clipping is taking place. (Click or download images to enlarge)



ScopesLabel.cpp


A few examples where I exposed with histogram at 1.0 and the main subjects Grey and Green with hints of yellow, when using the False Colors:

Example 1 Before and After Grading:




Example 2 Before and After Grading:




Example 3 Before and After Grading:




These 3 examples yield between 62 and 64MB/s @2.5K 14bits, and the other clips I have top at 79MB/s which means that it is possible to get a guaranteed 30 seconds per clip (Max 1:45min with lvface, or 3:00+ with lvface+aewb hack) with a Sandisk Extreme 256, without having to heavily underexpose the image. With these parameters, the images are quite clean, with some light noise in the shadows, like any other camera, even the expensive ones... (See the previous video I posted: https://bit.ly/3eKo5e9)

Exposing a little hotter brings the noise lower, at the expense of shorter recording times. Which leads to the next test.

Examples recorded with Histogram showing 0.2

You can look at the Data Rates Here. Each group of 10-12-14 or 10-12-14-14 bits are the exact same image, recorded at different bit Depths. The extra 14 bit recording is to check consistency  or a lower exposure.


Scene 1, very bright Highlights, 80.39 MB/s @2.5k 14bits


Scene 1 Graded


Scene 2, Well balanced scene with some bright elements (The shiny objects are usually what chokes the recording). 74.27 MB/s @2.5k 14bits


Scene 2 Graded


Scene 3, Well Balanced scene with less shiny spots (Allows a brighter overall image) 76.5 MB/s
@2.5k 14bits


Scene 3 with histogram at 1.0, lowers the data rate by 6.5 MB/s, and you get richer colors too


Scene 3 Graded


Scene 4, Closeup with a lot of details and a very shiny object. This combination made the recording stop early at 14bits, with 91.23 MB/s


Scene 4 Graded


Scene 5, Triggers frame corruption.

This scene, with a lot of details and reflective mosaic triggered a frame corruption error at 14 bits.


Graded


Recorded with exposure right before it triggered a frame corruption, 91.59 MB/s at 2.5k, 14bits


Lowering Histogram to 1.0 and 2.0 yields a data rate of 88.1 and 76.16 MB/s respectively




When the image is that bright, bumping up the ISO has a negligible effect on data rates:



Also note that moving the camera makes motion blur, and a blurry image decreases the data rate.
I won't post those clips, as this post is long enough already!

I did not let the camera record long in this test, to speed things up. For actual recording times, Have a look at the SD card tests I posted, which were in line with the Data Rates obtained here.

Maybe everyone could chime in as to what constitutes "a long enough clip recording time" to get the job done at higher resolutions.

Build Used: crop_rec_4k_mlv_snd_raw_only_2022Sep15.EOSM202
Cards: Sandisk Extreme Pro 128GB 170MB/s, Sandisk Extreme 256GB
Lens Ef-M 32mm F1.4

If I set the camera right on the focus point where it breaks on demand, and then switch presets:

2.5k 14bits and 12bits : Breaks
2.5k 10bits : Record times are similar to a very exposed image, it just stops recording after a while, without any corruption.

2.8k 14bits  : Breaks
2.8k 10bits and 12bits : Record times are similar to a very exposed image, it just stops recording after a while, without any corruption.

5k frtp 10bits and 12 bits : Seems to be unaffected by this issue.
5k frtp 14 bits : Breaks

1080p 10bits and 12bits : Seems to be unaffected by this issue.
1080p 14 bits : Breaks

If we look at my previous tests in an informal manner, and add 27.9MB/s (The additional rate for a "noisy" picture) to the data rate figures:

https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FZGHKYLw%2FSandisk-Extreme-SDSQXA1-256-G-GN6-MA-Report2.png&hash=e09c4547a1295a140c4a23cf1a2f91e9

(I didn't publish it, but 5k frtp is 71.7 and 60.2 MB/s for 14 and 12 bits respectively.)

We get one more hint that maybe too much data is being sent to the codec or that the way the codec encodes the extra information yields more data than the card can handle?

[Details:]

Follow up on the Frame Corruption discussion started here: https://www.magiclantern.fm/forum/index.php?topic=25841.msg240550#msg240550

The problem appears to be that the extra details introduced by focusing on the LCD screen at ISO 400 require more bandwidth than the card can handle.

Details:

I first tried modifying reg_713c and reg_7150 by +-10 each and compared the recorded file sizes to those recorded with the standard preset. Differences are marginal.
So those regs are probably timing related (High values distort the preview). I will test more in the coming days.

I then recorded files slightly soft-focused, and then focused a hair soft right before it breaks. There is a  27.9 MB/s difference between soft and almost full-focus.

So I recorded more files, starting a tiny bit soft focused and getting closer to full focus until it gave the corrupted frames error.
The closest file I got to full focus averages 94.7 MB/s, which is already close or above the benchmarked speed of the card (Sandisk Extreme Pro 170 128GB, 92.8-95.7MB/s)

So the problem appears to be that the extra details introduced by focusing on the LCD screen at ISO 400 require more bandwidth than the card can handle.

For the tests, I did a MLV fast-pass export of frames 25 to 96 (72 frames or 3 seconds) for each file I recorded, to facilitate comparison (Divide the file size by 3 to get MB/s):





This screenshot is for the folder where I MLV fast-passed 3 seconds for each file, for the focus sharper until it breaks test:

This is within my capabilities, no problem.
Maybe someone in the know could chime in, if they have the explanation handy(Actual purpose of each reg, so I can take potential problems into consideration)?

BTW, for those reading this out of context, this problem can be mitigated by:

1-Using a lower ISO (Less noise to encode)
2-Underexposition (Which results in a larger number of darker patches, easier to encode)
3-Avoiding tiny little details across the frame. For example, if I need to film a medium shot with blowing tree leaves without a subject in the foreground (Blurs the background a bit), I use the soft lens.

When I run into this problem, it is often because I move the camera, and the reflexions I tamed with the polarizer reappear suddently because of the change of angle.

I will move any findings, if any, in your main thread, to keep this one less cluttered.

@Danne, if you read closely, I said that I left the overlays on for ease, because exposure is an important parameter (I specified, because we had an exchange about this).
I ran the test with and without the Histogram and spot meter, and it did it too. I am able to replicate the test results on demand.

I think that because it is easier to make it crap out 10 times out of 10 when ISO is set to 400, I suspect that the extra noise (Extra details) is one of the reasons.
But I could be wrong.

Edit: Do you know what reg_713c and reg_7150 Stand for? The values are -2000 to +2000. That's a lot of combinations to test

I did a quick record this morning. I will be able to tell you about the data rates of the different sections in the mlv file this evening.

In the video I start with a soft focus of my test image. I stop recording myself;
Second trial I focus a little more, It stops recording itself;
Third trial I get sharp focus. You see the errors (As Danne said, while recording) I let it go a little then stop recording.

Video Demonstration: https://bit.ly/3Uxkkcs

Test conditions to replicate easily:
2.5k 1:1 Preset
Lens at f3.5
Iso 400 (Makes it easier to replicate)
Lens at 66 cm from the screen
Exposure as seen on screen (It will still happen without the Overlays, I left those on for ease)
Record the still first frame of the test file (The rest is to test something else): https://bit.ly/3LwH4W9

This is not limited to Lab settings, I ran into this problem during the summer when shooting certain foliage and animal closeups (Fur) under harsh sun, but could not figure it out.

If that's the only way, I will take that route then.
It is a different kind of endeavour, as I will need a second tripod, etc.

1- Your're right. It works now regardless of the build. The overclock isn't on by default.

2-I am trying to come up with a repeatable way of testing the record capabilities of a card at various resolutions, as the speed shown in the benchmarks doesn't mean that the camera will record at a certain resolution.
I found out that with certain types of images (Notably if I focus on the LCD monitor too closely) The amount of detail seems to be too much for the Lossless compressor / Data Rate (It just won't record).

I get a bunch of frame corruption and compression messages, but those are displayed too fast for me to read. It would be easier if those messages were saved on the SD card.

Working Fine on my EOS M, but I had to use it with your Jul 31 Build to see the CID (Otherwise it showed a bunch of zeroes). Maybe it needed another restart?

1- Is is ok to post the show CID Info on the forum?
2- Is there a way for me to get the error messages displayed on screen to a file? It used to work a while back, but I don't know how I got it to do it back then.

- Added show CID info option

Very clever addition to help us identify differences... Thanks!

[New version out:]

Shameless bump 

New version out:

Works well for me. Able to record 3 minutes consistently at 10bits. Thanks!

Some Benchmarks for the Sandisk Extreme Pro 170 128GB (The one which didn't work at 240MHz at first. Does anyone have a better way to call it?)



And the one for the Sandisk Extreme 256GB for comparison:




I hope the layout isn't too confusing. I will fill the missing spots as things progress. If something useful is missing, please let me know.

What I found out is that in order for this kind of bench to be significant, each measurement must be taken at least 3 times (Which I didn't do for the Record until Failure column).

For 2.8k 10 bit 192MHz, i did, and I got 3 completely different values: 3:50, 6:53 and 16:28 minutes with lvface hack.

It is also important to record a consistently exposed frame. Try recording 2.8k 14 bit with the lens cap on / off to see what I mean

There seems to be some kind of thermal throttling going on, because some times, even though I have Green Light / Immortal; the recording stops. At which point even recording 2.5k 10 bits won't go past 4 seconds (And like I said previously, the temperature displayed is at or above 50 Degrees C). You then have to turn off the camera, wait a little, and start again.

The Sandisk Extreme Pro SD 170 runs slightly cooler at 240MHz than the Sandisk Exterme Micro SD

Finally, Remember that when you use lvface+aewb, you cannot change the shutter angle (Shutter Fine-Tuning) and you will need to fix the white balance for every clip. So if you are strict about color accuracy or if you have a lot of clips to process, it will make your life easier if you film a white card at the beginning of each take.

The cuts in most movies are less than 20 seconds long, so... This new 240MHz mode rocks!

Can someone explain to me please what are ALL differences advantages/disadvantages between ''stock'' 1080p raw mode and so called ''5k'' mode?

If you are technically inclined, you can start by reading this: https://wiki.magiclantern.fm/pixel_binning

Otherwise, The main differences are:

Field of view:
1080p will give you a wider field of view than 5K frtp, because a larger portion of the sensor is sampled.

Resolution:
5k frtp will give you a more detailed image with less aliasing and less moiré.
Technical reason being that although a smaller portion of the sensor is sampled, none of the pixels are discarded (and more pixels from the sensor contribute to the final image). Each line is kept and each group of 3 columns is averaged.
for 1080p, only 1 line out of 3 contributes to the final image, and each group of 3 columns is averaged.

Disk Space:
5k frtp will obviously consume a lot more disk space than 1080p

High Frame rates:
1080p gives you the ability to record at 40fps Edit: 50fps with the latest 240Mhz Build to get slow motion effects, 5k frtp does not.

Using an external monitor:
It is a lot easier to use an external monitor in 1080p mode.

If you want to see what can be done using the 1080p mode, have a look at this guy's channel, he knows what he is doing: https://www.youtube.com/channel/UCJ28_AuqkDcipyqHoRTV76Q  AND this channel for 5k frtp: https://www.youtube.com/c/DavidKhoirul/videos

For good tutorials, you can check out Zeek's channel: https://www.youtube.com/c/ZeekEOSZ

And if you REALLY want to get comfortable with this camera, the only way is to start from the first post of the current thread and read everything up to this point, as there is no documentation per se

Great!

I have results I will post later today. There seems to be some temperature throttling going on.
The temperature polling is not always realtime, but I saw that when it stops recording (On 15 min+ trials, I kept recording while I were watching movies yesterday; the recording stops and then you see that the temp is 50-52 Degrees C)

I tested Today's build with 5kflv (Downloaded 2 hours ago),

It doesn't work for me at 23.976 fps (lvface+aewb), but it works at 19.999 fps (Even with only lvface, And at that res, the sharpness is quite nice).

I tried with both Sandisk Extreme Pro cards I have, with SDR50 and SDR104 patches.

Trials here: (I stopped by myself close to 20 seconds if it kept going. 10 am+ is the newest card, 11 am+ is the older card)

Thanks for sharing tests.
What if you disable histogram? Do you enable both hacks as well from bilal? lvface + aewb?

Leaving the histogram on doesn't extend the recording times. I reviewed the clips, and I did not get any frame corruptions though.
We're beyond the 20 minute mark at 2.5k, so it's hard to know. I will try at 3K on the next batch.

It works well at displaying overs (and avoids the pink frames) at 2.5k and 5kfrtp (And 1080p I assume?), but not accurate at all at other resolutions.
You could add an entry in your special menu as False Colors + Histogram. I put the False Colors on the SET button, and it is very quick to operate on/off.

I posted the figures for lvface+aewb in the Sandisk Ultra post last week.

Also, I previously reported that the 3k 24p preset was finnicky regarding the shutter, but it happens to all presets as soon as lvface+aewb is enabled (Sometimes 1/49, sometimes 1/50, without enabling shutter fine-tuning)

I were wondering if a static target gave the lossless compression too much slack when testing.
It turns out that the difference between a moving or a static image is 1MB/s @ 2.5k 14 bit.

BUT

The difference between a frame exposed at 92% exposure (0.2 EV before clipping) and a frame exposed at 80% (about 1.0 EV before clipping)
is about 3.3 MB/s

Looking at Walter's Evo Plus figures (I will have a look at my Sandisk numbers tomorrow, there are a lot...), I think the exposure is worth taking into account when testing.

This Samsung EVO Plus 256 GB MB-MC256K you benchmarked sits at a very comfortable price point here in Canada. If those cards prove to be as reliable as the Sandisk  Extreme Pro,
it will give the extra space required to record 3K 1:1 to everyone.

Benchmark for Sandisk Ultra 128GB SDSQUA4-128GB (Red and Grey) Micro SD

This card is a No-Go.

@240 Mhz


@192 MHz


@160 Mhz


OC Off



This is a U1 rated card. So I guess I aimed a little Too Low!

[Method:]

Some tests to check for corrupted frames.

Sandisk Extreme 256 Micro SD (Red and Gold) SDSQXA1-256G-GN6MA

crop_rec_4k_mlv_snd_raw_only_2022Sep15.EOSM202.zip
Didn't touch anything, except for the 2.39:1 Aspect Ratio.

Results: (Click to enlarge)


Camera Temperature peaked at 54 Degrees C.
I did not notice anything obvious as far as image quality goes, which could have happened because since the camera gets hot to the touch, the sensor could run hotter too.
The card stopped recording when I was close to full capacity (About 10GB free space left). At first I thought it was because of the heat, but only erasing all the files on card restored the speed.

There is a drop of speed when you get in the vincinity of 54 Degrees C. Happened after about 30 minutes recording the 30 seconds clips, sooner when recording 3 minutes clips (Those are still copying to disk as I am writing this).
Taking the card out and blowing air on it fixed it.
I only got corrupted clips when I got close to the full capacity of the card. Everything else is corruption free.

Method:

I shot the following chart at 3k, 2.8k, 2.5k; at every bitrate. (If someone has a more appropriate chart, please let me know)
Since the moiré happening because I am recording an LCD screen has some movement into it, I guess it should put enough strain on the compressor to be realistic.


Shot at 4 Exposure levels, to check how dark we have to go...





Files:







Benchmarks

@240Mhz


@192Mhz


@160Mhz


Overclock Off


I can also upload the file size and MLV app screenshots for the 3 minutes test, if someone needs to see those.

Also, my Sandisk Extreme Pro 170 128GB which didn't work @240MHz now works flawlessly.
Will add max recording times to the mix when I get a chance.

Thank you to all those involved for pushing it a step further!

Something is worth taking into account:

You need to restart the camera, load a new preset and try to record a short clip at least twice after copying the new files to the SD Card.
If you don't, you may get the impression that it isn't working, but it maybe will...

Does anyone knows why?

In my case, at first I got a few error messages, saying "frame corruption" on screen, but after restarting, even my Sandisk Extreme 256GB (Red and Gold) works @240MHz!!!!!


The Sandisk Extreme Pro 170 MHz 128GB which did not work a few days ago @240 also works now!
Will report frame corruptions later.


Here are the error messages I got before the 3rd restart, if it can be of any help:

ML ASSERT:
0
at mlv_lite.c:2892 (compress_task), task compress_task
lv:1 mode:3

compress_task stack: 1de948 [1de9d8-1dd9d8]
0x0009EC30 @ bddb24:1de978
0x0009E558 @ 9ec8c:1de948

Magic Lantern version : Nightly.2022Sep15.EOSM202
Mercurial changeset   : e4663790da85 (crop_rec_4k_mlv_snd_raw_only) tip
Built on 2022-09-15 10:08:56 UTC by da.
Free Memory  : 219K + 1891K

ML ASSERT:
0
at mlv_lite.c:2873 (compress_task), task compress_task
lv:1 mode:3

compress_task stack: 1ded50 [1dede0-1ddde0]
0x0009EC30 @ bd6d10:1ded80
0x0009E558 @ 9ec8c:1ded50

Magic Lantern version : Nightly.2022Sep15.EOSM202
Mercurial changeset   : e4663790da85 (crop_rec_4k_mlv_snd_raw_only) tip
Built on 2022-09-15 10:08:56 UTC by da
Free Memory  : 219K + 1890K

ML ASSERT:
0
at mlv_lite.c:2873 (compress_task), task compress_task
lv:1 mode:3

compress_task stack: 1ded50 [1dede0-1ddde0]
0x0009EC30 @ bd69b0:1ded80
0x0009E558 @ 9ec8c:1ded50

Magic Lantern version : Nightly.2022Sep15.EOSM202
Mercurial changeset   : e4663790da85 (crop_rec_4k_mlv_snd_raw_only) tip
Built on 2022-09-15 10:08:56 UTC by da
Free Memory  : 219K + 1895K

ML ASSERT:
0
at mlv_lite.c:2873 (compress_task), task compress_task
lv:1 mode:3

compress_task stack: 1ded50 [1dede0-1ddde0]
0x0009EC30 @ bd6920:1ded80
0x0009E558 @ 9ec8c:1ded50

Magic Lantern version : Nightly.2022Sep15.EOSM202
Mercurial changeset   : e4663790da85 (crop_rec_4k_mlv_snd_raw_only) tip
Built on 2022-09-15 10:08:56 UTC by da
Free Memory  : 220K + 1911K

ahaaaaa today in the morning I have downloaded the build and it said uploaded 17min ago or so. I assumed Bilals 240MHz was part of it already.

In short:
Downloaed 2022Sep15EOSM202.zip and only after replacing with the original from Bilal ( https://www.magiclantern.fm/forum/index.php?topic=26634.msg240128#msg240128 ) it worked again but I have the feeling that now the option is missing for sdcard frequency

When using Bilal's codes, you have to go in the DEBUG tab, at the bottom. You will see the Overclock frequency there.