LiveView hacks (write speed improvement)

Started by theBilalFakhouri, April 07, 2022, 06:20:22 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Levas

Quote from: Danne on April 08, 2022, 03:19:37 PM
"one more hack" caused freeze. Will investigate further.

The same with 6d, but it's not just a freeze. Trashcan button to enter ML menu is responsive.
In ML menu you can see that it's starting raw video, but it never starts  :P
Switching off the camera gives clean shutdown, so no ordinary freeze.

Danne

Yeah, something strange there Levas :).
By the way bilal. More hacks says exposure and white balanse will be locked. Testee behaviour but seems exposure is still changeable while recording on my eos m. Changing, iso, shutter and aperture, while filming, also changes exposure on the screen?

theBilalFakhouri

Currently avoid using Cartridge Cancel a.k.a "one more hack" on EOS M and 6D. as I wrote in the first post I have no idea what it does.
Be aware this is the only hack which may produce RAW data corruption in some cases (during my tests only first frame was affected, other were fine. but still there is a possible RAW data corruption).

I found also Cartridge Clear function (we are not using it in our hacks), calling it would suspend all EDMACs channel activity, probably Cartridge may mean EDMACs channels.

Of course take these hacks and info with grain of salt. :) This thread is a nice proof of concept that proves we can gain extra write speed.

@names_are_hard suggested that we may be able to lower other tasks priority in LiveView, without killing the tasks (without suspending aewb).
There are more stuff to dig into it, if someone want a faster progress feel free to contribute.

Danne

Yes, very cool stuff.
Tested some more and using more hacks freezes when hd 1080p is selected. Probably since it´s in crop mode. Regular x3 zoom and 1x3 binning modes(anamorphic all works. So don´t use these hacks with HD 1080p mode for now.

theBilalFakhouri

Quote from: Bender@arsch on April 07, 2022, 10:15:43 PM
3.5K Preset in 3584x1730, at 23.976fps at 14bit lossless. With your Hacks, SD Overclocking and card spanning. Filming static scene.
I double checked the filezise on my computer. mb / frames x 23.976.

Without your hacks I reached up to 139mb in my older tests

Cool, what CF card are you using? mind running CF card benchmarks in Photo mode?

Quote from: Bender@arsch on April 07, 2022, 10:15:43 PM
Sometimes the first frame is corrupt but the rest is good.

Yeah, I had the same thing. This happens with "One more hack" enabled, other hacks didn't produce first frame corruption. Double checking with "One more hack" disabled is a good idea.

Quote from: Bender@arsch on April 07, 2022, 10:15:43 PM
But I think this writespeed is really bleeding edge: If I use raw zebras histogram and waveform (with Kill global draw), the record stops automatically after a while (also with green indicator), camera shutdown and say Battery low. With dummy Battery it stops, but without shutdown. And without waveform it works fine (with Battery) .

Sometime I have speed drops and suddenly stops on 700D. more tests are needed for me to make sure and figure out what's happening. I will make more tests on 5D3 in future and will see if I can reproduce your issue too.

theBilalFakhouri

Quote from: lightwriter on April 08, 2022, 10:32:13 AM
This is beyond awesome! Definitely Black Magic Lantern ;)
I fully trust in your builds, and I've never had any problem. I assume that's also the case with this one :)

Thanks, you may want to make some real tests too with the new stuff before giving a full trust :).

theBilalFakhouri

Quote from: Danne on April 09, 2022, 03:03:01 PM
Tested some more and using more hacks freezes when hd 1080p is selected. Probably since it´s in crop mode. Regular x3 zoom and 1x3 binning modes(anamorphic all works. So don´t use these hacks with HD 1080p mode for now.

Just tested the new hacks in movie crop mode on 700D, it seems lvfaceEnd hack is causing freezing here too, only in MCM. but "One more hack" appears to work.

Danne

Tested all three hacks separately but nothing worked when in mcm mode on eos m. Other presets seems fine though :).

theBilalFakhouri

Quote from: Danne on April 09, 2022, 03:46:45 PM
Tested all three hacks separately but nothing worked when in mcm mode on eos m. Other presets seems fine though :).

Nice, could you post some results from EOSM, write speed with and without the hacks (recording time in a static scene before and after).

Danne

Testing against 2.8k in 12bit, 24fps I get around 600 frames with the hack lvface + aewb and without it records about a 100 frames. With 10bit I get continuous while without the hack around 260 frames. I think we are recording around 65MB/s here.
Correction. One more hack will still freeze all modes.

Bender@arsch

Quote from: theBilalFakhouri on April 09, 2022, 03:06:18 PM
Cool, what CF card are you using? mind running CF card benchmarks in Photo mode?

A Kingston Canvas 128gb (not the fastest, but cheap) - > benchmark CF only: 115W, 146R

But I tested my Komputerbay 128gb now too. I reached unbelievable 156mb/s - > ~7848mb / 1203 frames x 23.976 = ~156 (after 40s stopped automatically) - > Benchmark CF only 119W, 152R
Benchmark CF + SD: 155W

But there is no change in Benchmark with and without your hacks.

QuoteI will make more tests on 5D3 in future and will see if I can reproduce your issue too.

I think the shut down problem was the Battery itself, maybe not full recharged or something.

2blackbar

Holy crap ! Just tried it on eos M , i set 2.8k 10 bit, 24fps , sd overclock 192-240, got 61-62MB rec speed and i was able to record one minute, it stopped at 01:05...
Danne heres HDMI output when i record 2.8k in this new version, half of the screen is pink stripes
So that means now 3k is possible to implement on eos M ?
I set 3k and reg 6014 to 480 and got 23.3 fps , i can get 30secs this way but 23.980 wont get me even 5secs with 6014 set to 525.My cardwrite speed on PC is 80MB
Sadly all other non crop modes are broken, and hang the camera when pressing record, so i cant record in fullsensor width modes.
Ok turning off lvface hack fixes camera hanging on non crop modes but its the one hack that helps the most to gain extra speed.
How i can capture 5D2 log to provide You with correct addresses for lvface and awbd functions that are needed to make it work on 5D2 ?


Ramick

Just tried on my eos M,

Test conditions:
- 192mhz mode
- Lvface +aewb hacks
- "One more hack" OFF => as Danne mentioned direct freeze
- Same over exposed scene for all tests
- Sandisk Extreme Pro 170mb/s

2.8K – 10 bits – 23.981fps
- Before: around 11s
- After: continous (manually stopped after more than 10 minutes)

3K (3008x1280) – 10 bits – 23.3fps (reg_6014 = -480 / reg_6008 = 0)
- between 10 and 15s

Great job Bilal, the 2.8K can now be used and the 3K seems not that far :)

gabriielangel

Quote from: Danne on April 09, 2022, 08:16:24 AM
Anybody testing this hack with eos m and hdmi out?

How do I do eos-m screen captures? The hdmi output is acting strange:

If I use a phone with usb cam installed, It outputs 2K 1:1 correctly, but in other modes it outputs only the interface overlay, but the video preview is garbled.

If I connect the eos m to the hdmi input of my Asus Computer Monitor, the 2k 1:1 mode doesn't work, 5K ana and 1080p work but freeze, and 2.5K and 2.8K modes output half a squeezed video preview with the second half pink.

So, I went back to the June 2021 build to make sure: only 2k 1:1 works when using the phone as monitor, and using the Asus Monitor, only 5k Ana and 1080P outputs correctly. I still get half a squeezed screen in 2.5k and 2.8k

So it seems like it depends on what is on the hdmi receiving end, because Zeek said it works for him in all modes when using the june 2021 build.

Any ideas or other tests  I could do to help out?

Danne

First post is updated with a fresh eosm build from original code. This version skips "One more hack"  and also skips running in hd 1080p mode.

ZEEK

Quote from: 2blackbar on April 09, 2022, 05:32:36 PM
Danne heres HDMI output when i record 2.8k in this new version, half of the screen is pink stripes
To fix that PINK HDMI screen: in ML tab 3, go to 'customized buttons', Disable x3 Crop Toggle and Focus Aid
EOS M

lightspeed

I don't have one more hack on the eosm

is it not on the eosm?

gabriielangel

EOS M Test Results using HDMI Monitor:

Build: crop_rec_4k_mlv_snd_raw_only_2022Apr10.EOSM202.zip, link in first post
SD Card: Sandisk Extreme Pro 170 MB/s, 128GB
192MHz Hack

In order for the hdmi output to work reliably, the presets need to be changed using the "Tap Display". When connecting an hdmi monitor, the screen tap doesn't work (The eos screen is disabled). So we could go to 3rd ML Menu, customized buttons, and put "INFO SELECTABLE" to "Off" and  change "Dropdown List" to "INFO", but then lose whatever function was assigned to the Info button. We could also disconnect the hdmi cable to select presets, but I don't think this would be recommended as the Mini hdmi connector isn't "Industrial Strength) :)

Also, in order for all the different presets to work reliably, without Pink / Green half screen, squeezed display, freezed display; in 3rd ML Menu Disable x3 Crop Toggle and Disable Focus Aid. Those 2 functions won't be needed if using a field monitor with pinch-to-zoom or zoom functions.

If using a phone with USB Camera or another capture app installed as a monitor, only the 2k 1:1 preset works (Tested with LG V30, Samsung Galaxy S8, Hagbis UHC06 capture dongle)

Edit: When using a capture dongle with a phone and usb camera, the dongle needs to be connected to the phone BEFORE powering it on. That way, 2K, 2.5K and 2.8K presets work (But the screen shows only part of the actual recorded image). When changing presets, resolution, etc, the EOS-m needs to be restarted. My hdmi monitor (Asus) doesn't work with 2.5K (At any aspect ratio), but some people got their monitors working when using a 16:9 ratio). Maybe we could post working configurations in a dedicated thread to make it easier for everyone... My USB camera Video Format settings are: MJPEG, 1920x1080 60p

The Benchmark module doesn't write the results as a .ppm file (image file) on the card anymore, but I get 61.9 / 63.9 MB/s Write speed, which are the exact same figures I had when running the June 26 2021 Build. The recording times have been expanded significantly though:

5K frtp:
14bit 23.976 fps, recorded over 3 minutes once, and several 1 minute clips in a row, no interruptions (Orange light).
12bit 23.976 fps, records continuously (>3 minutes).

2.5k:
14bit 23.976 fps, toggles between green and orange light, Always records past 1 minute, but on one trial, it stopped after 2:30 minutes.
12bit 23.976 fps, records continuously  (>3 minutes)

2.8K:
12 bit 23.976 fps, records between 1-2 minutes.
10 bit 23.976 fps, records continuously, orange light.

For the tests, I shot tree branches towards an overcast sky, with the histogram showing between E1.0 and E0.4

Danne

Nice test report!
So, just to be sure. All tests reported here includes the use of a hdmi monitor? Except the usb camera test that is.

gabriielangel

Quote from: Danne on April 11, 2022, 03:27:03 AM
Nice test report!
So, just to be sure. All tests reported here includes the use of a hdmi monitor? Except the usb camera test that is.

Correct. I used the HDMI input of an ASUS PB278Q. If you could point me to some info on how the hdmi out works, I could try to find out why the usb dongles and the lower end field monitors are so finnicky.

Danne

Unfortunately not using hdmi port personally so nice that this could be tested like this. What about frame corruption? Did it seem like files were ok or did you spot any pink frames here and there?

gabriielangel

Those long clips take forever to convert! I will let you know in a few days, as I need to look at everything very closely to spot focus dots, glitches, etc. So that whoever reading the comment can use it accordingly.

Danne

I see. I think some general use cases will reveal any issues pretty early but I could be wrong. But great if you can check some on your side since you seem to do a good job here.

gabriielangel

Some results.
I recorded two batches of about 30 clips each. I reviewed Both batches today, so here are my findings:

When adding the hack "Lvface+aewb", using HDMI monitor or not, if I set the shutter fine tune so that I get between 1/46 and 1/48 (a little above 1/46 gives the appropriate blur we're used to, 1/48 seems faster than expected), the value of the shutter jumps to 1/30 as soon as I hit record. It shows in the recorded file, as you see the exposure ramping up within the first 10 frames or so (using other fine tune values will shift the exposure relative to the value).

Putting  Shutter fine tune to "off" will keep a steady 1/50.

Using only "Lvface" instead of "Lvface+aewb" fixes this Shutter issue, but 2.8k recording will redline at 12 bit and record about 20 seconds.




So using only the "Lvface" hack and hdmi monitor, the figures are as follows:

2.8K:
12 bit 23.976 fps, records about 20 seconds.
10 bit 23.976 fps, records continuously, between green and orange light.
No pink or dropped frames, focus pixels can be seen clearly  while recording, but are taken care of with mlvapp.

Following Danne's hint about resolution:
14bit 23.976 fps, if I lower the resolution to 2656x1112, I get at least 30 seconds, oscillating between red and orange light.
14bit 23.976 fps, If I lower the resolution to 2608x1092, I get at least 1 minute (Stopped myself on each trial), always in orange light, no red.

2.5K:
Speed Results are the same, but lots of pink frames regardless of the bit depth. even restarting the camera does not fix it. So, not usable with an HDMI monitor for now.
Edit: See Reply#42, it works with a phone and capture dongle in my case (And no pink frames), when connecting the dongle before turning on the phone, and also when powering off the camera after each settings / preset change)

5K ana frtp: The results are similar to my previous post (Reply #42), no pink or dropped frames and no focus pixels.

2K: Continuous recording at every bit depth. No corrupt frames.

Out of 60 clips, I got 3 corrupted clip, which would not open (2.8k).


When using the hdmi monitor, when switching presets, the connection to the monitor is often lost. You have to shutdown the camera and it runs fine on restart.

Like I mentioned in the previous post, using the hdmi monitor disables the tap screen dropdown list. So I have to assign the dropdown list to the info button.
But, when pressing info, you have to turn the wheel quickly, because it chooses a preset by itself as soon as you stop spinning the wheel.

Also, I noticed that every preset uses a different center (Those scan different parts of the sensor) and if I use the 2.5K Centered preset as a reference, the presets are often a little to the right, with the 5K preset scanning towards the lower portion of the sensor (Which at least give you a chance to get the eyes in better focus if filming a person when using an older lens).

It would be good to disclose the exact offset figures, as this would allow people to choose the right lens accordingly. Unless you use an ef-m 32mm f1.4 or, to a slightly lesser extent, ef-m 22mm f2; you don't get edge to edge sharpness. So, if you use a Helios or a c-mount lens, you will get uneven bokeh and vignetting, as those vintage lenses get softer and blurrier very quickly as soon as you deviate from the center. (Not too apparent if you do nature shots, but a lot more if you frame a subject off center.)
Zooming also requires reframing, when symmetry is needed. A non technical person will easily overlook those details and will have to learn the hard way, after wondering for hours if they were drunk the day of the shoot  :)

Danne

Interesting. So 2.8k shows better results than 2.5k when it comes to pink frames.
So there's a shutter issue when +aewb i selected? And only with hdmi I suppose?