LiveView hacks (write speed improvement)

[IDA_ML]

In my experience, the 700D provides less noise and a cleaner image in the darks compared to the EOS-M.  I don't know why.  It could be a different sensor or some kind of noise reduction at RAW level but overall the image of the 700D looks cleaner in the shadows.

[Skinny]

less noise and a cleaner image in the darks compared to the EOS-M

It could be better power filtration, different dc/dc converters and all analog stuff around the sensor and a/d conversion. EOS-M is so compact they could sacrifice some performance in order to fit everything in that tiny form-factor..

For example, in audio world two different audio interfaces with identical chips inside can have different characteristics. And it is mainly because of a difference in power, shielding and all that stuff. Basically, analog stuff around the digital chip.

[Danne]

Dxo rates eos m sensor higher than 700d
https://www.apotelyt.com/compare-camera/canon-700d-vs-canon-m

I expect more or less no visible difference given test setup is the same. Anybody having both cameras could post two cr2 files for us to compare. Needless to say. Same test condition needed.

[vastunghia]

After some Easter family shooting, I can conclude that these LV hacks boosted my 5D3 (1.1.3) write speed from 115-120 MB/s to 130-135 MB/s in real-life situations.

Switched to 14-bit 3.5K (that should be called 3.7K actually IMO) definitively. I just cannot do exact ETTR in very bright conditions and complex scenes -- in these cases I have to stop down 1/3 to 1 EV in order to achieve continuous recording, sometimes also killing Global Draw. Happy to pay that price though.

I'm definitely getting 140 MB/S on my 5d3 (113)

What cards are you using? I definitely cannot reach above 135 MB/s.

Sergio

[mlrocks]

After some Easter family shooting, I can conclude that these LV hacks boosted my 5D3 (1.1.3) write speed from 115-120 MB/s to 130-135 MB/s in real-life situations.

Switched to 14-bit 3.5K (that should be called 3.7K actually IMO) definitively. I just cannot do exact ETTR in very bright conditions and complex scenes -- in these cases I have to stop down 1/3 to 1 EV in order to achieve continuous recording, sometimes also killing Global Draw. Happy to pay that price though.

What cards are you using? I definitely cannot reach above 135 MB/s.

Sergio

Did you use sd card overclocking on 5D3? How long was your shooting session? Did you experience overheating and related corrupted frames? Thanks.

[vastunghia]

Did you use sd card overclocking on 5D3? How long was your shooting session? Did you experience overheating and related corrupted frames? Thanks.

Yes, overclocking enabled, as well as card spanning of course.

I took 96 clips in a 48h time frame approximatively, all 3.5K 1:1 centered @14 bit (some of which using Dual ISO as well), with the following stats:

• 32 automagically stopped , all with write speed > 135 MB/s (150 on average)
• the remaining 64 achieved continuous recording (short clips, 30 secs on average, apart from a longer one lasting more than 3 minutes), with write speed up to 135 MB/s (130 on average)

I would say that in broadly half of cases I got a slightly corrupted first frame (typically resulting in a pink area in the top 1/4 of the image), but apart from that only one clip had a (completely) corrupted frame in the middle of it (i.e. MLV App skipped it when converting to DNG). So this seems pretty reliable to me.

Also, no particular overheating observed (but again, my shooting sessions were very quick).

HTH

Sergio

[mlrocks]

Yes, overclocking enabled, as well as card spanning of course.

I took 96 clips in a 48h time frame approximatively, all 3.5K 1:1 centered @14 bit (some of which using Dual ISO as well), with the following stats:

• 32 automagically stopped , all with write speed > 135 MB/s (150 on average)
• the remaining 64 achieved continuous recording (short clips, 30 secs on average, apart from a longer one lasting more than 3 minutes), with write speed up to 135 MB/s (130 on average)

I would say that in broadly half of cases I got a slightly corrupted first frame (typically resulting in a pink area in the top 1/4 of the image), but apart from that only one clip had a (completely) corrupted frame in the middle of it (i.e. MLV App skipped it when converting to DNG). So this seems pretty reliable to me.

Also, no particular overheating observed (but again, my shooting sessions were very quick).

HTH

Sergio

Thank you for the detailed answer.
Is dual ISO on 5D3 reliable?
Also to IDAML:
Is dual ISO on 700D reliable?
No pink flash? Moire severe? etc.

If dual ISO works on 650D/700D and 5D3, basically these cameras are comparable to Arri Amira, which is the best news camera due to its high dr.

[theBilalFakhouri]

On 650D/700D first you need to select a stable RAW video mode, all presets in crop_rec "Crop mode" are stable (doesn't produce pink frames) with/without Dual ISO, also the normal Canon modes like (1080p, 720p, movie crop mode, x5 mode).

But if you are using arbitrary resolutions with crop_new (Crop mode V2), some settings would have pink/corrupted frames either with or without Dual ISO. crop_new isn't well fine-tuned yet, some settings will work without any problem like my favourite 1736x2214 1x3 @ 23.976 FPS (10-bit lossless), never had a problem with this mode on my 700D, also made a lot of tests with Dual ISO, this RAW video mode is stable.

Reducing FPS or RAW resolution a little in crop_new "Crop mode V2" should help to eliminate on pink/corrupted frames issues with your custom settings (don't push it to the limits).

But unfortunately currently I am not using Dual ISO much on 700D because it has flickering issue on most RAW modes, I tested MLVApp, latest version of cr2hdr (on linux), both produce the same problem.
Some modes does work without the flickering issue like 4.5K 1x3 in "Crop mode".

[mlrocks]

On 650D/700D first you need to select a stable RAW video mode, all presets in crop_rec "Crop mode" are stable (doesn't produce pink frames) with/without Dual ISO, also the normal Canon modes like (1080p, 720p, movie crop mode, x5 mode).

But if you are using arbitrary resolutions with crop_new (Crop mode V2), some settings would have pink/corrupted frames either with or without Dual ISO. crop_new isn't well fine-tuned yet, some settings will work without any problem like my favourite 1736x2214 1x3 @ 23.976 FPS (10-bit lossless), never had a problem with this mode on my 700D, also made a lot of tests with Dual ISO, this RAW video mode is stable.

Reducing FPS or RAW resolution a little in crop_new "Crop mode V2" should help to eliminate on pink/corrupted frames issues with your custom settings (don't push it to the limits).

But unfortunately currently I am not using Dual ISO much on 700D because it has flickering issue on most RAW modes, I tested MLVApp, latest version of cr2hdr (on linux), both produce the same problem.
Some modes does work without the flickering issue like 4.5K 1x3 in "Crop mode".

Thanks for your detailed answer, Bil. Is the UHD 1X3 immune to the flickering issue? I will try 4.5K 1x3 with dual ISO later.

Update:
I just tested 4.5K 1x3 with dual ISO, there is no flickering.

[theBilalFakhouri]

Thanks for your detailed answer, Bil. Is the UHD 1X3 immune to the flickering issue? I will try 4.5K 1x3 with dual ISO later.

Unfortunately, UHD 1x3 does have flickering issue, all other 1x3 presets have this problem except for 4.5K 1x3 which is weird . .

I just tested 4.5K 1x3 with dual ISO, there is no flickering.

Yeah, I am wondering how this preset don't introduce flickering, it worth digging into it. Will take a further look in future.

[theBilalFakhouri]

@vastunghia

Nice stats, thanks for sharing the results.

[theBilalFakhouri]

..
How i can capture 5D2 log to provide You with correct addresses for lvface and awbd functions that are needed to make it work on 5D2 ?

I got a log from Skinny (thanks), it seems 5D2 doesn't have these functions . Initially I thought it would be there because 60D and 600D both have these functions, but 5D2 and 550D don't.
So not all DIGIC 4 cameras have these.

I took a quick look into 5D2 log, it might be there some similar functions, no idea yet if it would work. Will take a further look later.

[theBilalFakhouri]

...
2.8K – 10 bits – 23.981fps
- Before: around 11s
- After: continous (manually stopped after more than 10 minutes)

3K (3008x1280) – 10 bits – 23.3fps (reg_6014 = -480 / reg_6008 = 0)
- between 10 and 15s

Great job Bilal, the 2.8K can now be used and the 3K seems not that far

Thanks for sharing, it's definitely an enhancement .

[theBilalFakhouri]

A Kingston Canvas 128gb (not the fastest, but cheap) - > benchmark CF only: 115W, 146R

But I tested my Komputerbay 128gb now too. I reached unbelievable 156mb/s - > ~7848mb / 1203 frames x 23.976 = ~156 (after 40s stopped automatically) - > Benchmark CF only 119W, 152R
Benchmark CF + SD: 155W

Your Komputerbay CF card matches my Sandisk CF card benchmarks in Photo mode, same write speed ~119 MB/s, I thought your card might be faster.
~156 MB/s in LiveView isn't an accurate number, buffer size has an effect too on the calculations . . I will make tests later and will try to examine write speed based on your settings

ut there is no change in Benchmark with and without your hacks.

Yeah, because the hacks only works in LiveView . .

[vastunghia]

Is dual ISO on 5D3 reliable?

Dual ISO gave me no problem on the 5D3 so far. And in particular scenes, if used wisely, it yields impressive results.

Nice stats, thanks for sharing the results.

Thank you again Bilal, you are doing a terrific work!

If dual ISO works on 650D/700D and 5D3, basically these cameras are comparable to Arri Amira, which is the best news camera due to its high dr.

Yes, I love my 5D3 with ML, it is capable of delivering fantastic image quality at a stunning dynamic range.

In my view these are the two missing features that would make the 5D3 a real Raw workhorse:

• HDMI output with correct framing (or at least not so cropped) -- especially in 1:1 modes (don't even know if this is technically feasible): without this, checking focus (which is critical at high resolutions of course) is a pain in the neck
• SD overclock @higher speeds: on the 70D I can set it to 160 / 192 / 240 MHz, so porting this to the 5D3 (once again, if technically feasible) would potentially unlock continuous recording in more situations

Yeah, this is my personal list for Santa... 

Sergio

[mlrocks]

Unfortunately, UHD 1x3 does have flickering issue, all other 1x3 presets have this problem except for 4.5K 1x3 which is weird . .

Yeah, I am wondering how this preset don't introduce flickering, it worth digging into it. Will take a further look in future.

Another thing to consider:

The MLV App processing time for a 4.5k 1x3 dual ISO footage is about 4 times longer than a 4.5k 1x3 without dual ISO, about 10 to 12 times longer than a 1920x1280 1x1 without dual ISO. For fast turnaround time projects, 1x1 2k actually has such an advantage.

[gabriielangel]

Is there a script or a way to do frame stacking (For Noise Reduction) from within MLV app?

[masc]

Another thing to consider:

The MLV App processing time for a 4.5k 1x3 dual ISO footage is about 4 times longer than a 4.5k 1x3 without dual ISO, about 10 to 12 times longer than a 1920x1280 1x1 without dual ISO. For fast turnaround time projects, 1x1 2k actually has such an advantage.

DualISO processing is very complex and the actual algorithm is single threaded only. Yes - this is slow, but a lot of math.
But I don't think non-dualiso 4.5K is 2.5x..3x slower than 2K 1x1: the amount of pixels is more or less the same. The only difference is a single 3x stretching - but this is way faster than you write. As soon as you rescale 2K to e.g. 4K and 4.5K to 4K both will need more or less the same processing time.

Is there a script or a way to do frame stacking (For Noise Reduction) from within MLV app?

You can average n frames to one single frame. But this feature is more for creating dark frames. You won't get a video with that.

[mlrocks]

DualISO processing is very complex and the actual algorithm is single threaded only. Yes - this is slow, but a lot of math.
But I don't think non-dualiso 4.5K is 2.5x..3x slower than 2K 1x1: the amount of pixels is more or less the same. The only difference is a single 3x stretching - but this is way faster than you write. As soon as you rescale 2K to e.g. 4K and 4.5K to 4K both will need more or less the same processing time.

No offense, MASC. MLV App is a great software.
I am trying to configure my 650D as a practical news camera, balancing all of the major factors. I think the 12 bit 4.5k 1x3 dual ISO is probably the closest to the ARRI Amira, which is amazing, considering the price tag difference between the two. For short deadline projects, 1920x1280 14 bit 24 p is a good choice. For cinematic feelings, 5.2k 1x3 10 bit is the best, due to its super 35mm sensor size.

[mlrocks]

No offense, MASC. MLV App is a great software.
I am trying to configure my 650D as a practical news camera, balancing all of the major factors. I think the 12 bit 4.5k 1x3 dual ISO is probably the closest to the ARRI Amira, which is amazing, considering the price tag difference between the two. For short deadline projects, 1920x1280 14 bit 24 p is a good choice. For cinematic feelings, 5.2k 1x3 10 bit is the best, due to its super 35mm sensor size.

Interestingly, compared with 10 bit footage, 14 bit 2k 1x1 AR 3-2 24 fps, when shooting with auto/manual ettr, and elevating exposure one stop in MLV App, the output is very nice, similar to alexa/amira's high key footage, the noise in the dark area is not apparent. So, 14 bit is actually better than 10 bit, when using ettr, may gain one to 2 stops dr. For 10 bit shooting, better avoid ettr, because much of the information pushed to the dark side is lost and not saved in raw files.
In this term, 14 bit 4.5k 1x3 with ettr and dual iso may have a dr of 16 stops, 12 stops from the 650d raw file, 1-2 stops from ettr, 2-3 stops from dual iso. This dr is on par with the legendary alexa and amira's dr. 4.5k 1x3 raw also should be on par with the alexa/amira's 2.8k prores. The sensor size in 4.5k 1x3 mode is about 230 mm2, almost the same as 4-3's 224 mm2, although a little bit smaller than alexa/amira's super 35mm, this is good enough for news shooting, which requires large in focus zones. Imagine a lowly 650D/700D is Alexa/Amira or Red One in disguise, this is really something to talk about.

https://www.provideocoalition.com/alexa-dynamic-range-its-all-in-how-you-use-it/
https://www.newsshooter.com/2020/05/27/the-arri-alexa-is-10-years-old/

[koopg]

Great build
5D3  v1.1.3
3.5k 14bit lossless + dual ISO
aspect ratio 2:20
Dual card  with Sandisk x1066
SD over locked
First frame almost always half pink (personally don't care)
I can get continues rec as long there not too much over exposure.
Its a game chager.

Thank you all devs!


Sent from my SM-P905 using Tapatalk

[mlrocks]

Great build
5D3  v1.1.3
3.5k 14bit lossless + dual ISO
aspect ratio 2:20
Dual card  with Sandisk x1066
SD over locked
First frame almost always half pink (personally don't care)
I can get continues rec as long there not too much over exposure.
Its a game chager.

Thank you all devs!


Sent from my SM-P905 using Tapatalk

For dual ISO on 5d3, did you see flickering on the bright area?

[Wushuliu]

I've been out of the ML world for a few years but this seems like a tempting development to jump back. What camera would people recommend for highest continuous resolution 16:9 to 2:20 and hdmi monitor out?

[mlrocks]

I've been out of the ML world for a few years but this seems like a tempting development to jump back. What camera would people recommend for highest continuous resolution 16:9 to 2:20 and hdmi monitor out?

Only 650d/700d can do correct high resolution hdmi out. EOS M may do that also. Thanks to Bil's work. Hope he can figure out the 5d3's realtime preview and hdmi out.
650D/700D UHD 1X3 preset has AR 16-9, 4K 1x3 preset has AR 2. I tested that both HDMI out works fine with my Zacuto 3 inch EVF.
5D3's 5.7k 1x3 preset has incorrect framing unstretched hdmi out. I can use Zacuto EVF's anamorphic ratio of 2 to do certainly level of unstretching that is good enough for rough framing, but the hdmi out framing is not correct.

[mlrocks]

Interestingly, compared with 10 bit footage, 14 bit 2k 1x1 AR 3-2 24 fps, when shooting with auto/manual ettr, and elevating exposure one stop in MLV App, the output is very nice, similar to alexa/amira's high key footage, the noise in the dark area is not apparent. So, 14 bit is actually better than 10 bit, when using ettr, may gain one to 2 stops dr. For 10 bit shooting, better avoid ettr, because much of the information pushed to the dark side is lost and not saved in raw files.
In this term, 14 bit 4.5k 1x3 with ettr and dual iso may have a dr of 16 stops, 12 stops from the 650d raw file, 1-2 stops from ettr, 2-3 stops from dual iso. This dr is on par with the legendary alexa and amira's dr. 4.5k 1x3 raw also should be on par with the alexa/amira's 2.8k prores. The sensor size in 4.5k 1x3 mode is about 230 mm2, almost the same as 4-3's 224 mm2, although a little bit smaller than alexa/amira's super 35mm, this is good enough for news shooting, which requires large in focus zones. Imagine a lowly 650D/700D is Alexa/Amira or Red One in disguise, this is really something to talk about.

https://www.provideocoalition.com/alexa-dynamic-range-its-all-in-how-you-use-it/
https://www.newsshooter.com/2020/05/27/the-arri-alexa-is-10-years-old/

After so much consideration, I finally settle down on 14-bit lossless UHD 1X3 with auto/manual ETTR without dual ISO on my 650D for raw news shooting, due to its AR 16-9, relative good low light performance, good image quality, acceptable pp computing time, correct real time color high resolution preview and magic zoom 2, etc. 14-bit lossless UHD 1X3 for a complex scene has a peak writing speed of 85 MB/s, I will go to user interface 2 (almost equivalent to global draw off but with certain essential information) and underexpose the scene to maintain the continuous recording, or go 12-bit lossless. Most of the time, though, the writing speed is below 78 MB/s, allowing to use the global draw in user interface 1.