LiveView Investigation

[Grognard]

@Bilal.

I wonder if there way in 3.5k mode (5DIII) when recording to force framing in B&W and not switching between Black and White and color.

Maybe it could improve speed? It's the case in anamorphic, only black and White and higher framerate. In practice it's a very low resolution in black and white but almost in real time.

Thank you.

[theBilalFakhouri]

This can be done easily, in mlv_lite.c iirc it's in:

Code Select Expand

unsigned int raw_rec_update_preview(unsigned int ctx)

You can force black and white preview all time by changing:

Code Select Expand

        (need_for_speed && !get_halfshutter_pressed())
            ? RAW_PREVIEW_GRAY_ULTRA_FAST
            : RAW_PREVIEW_COLOR_HALFRES

to

Code Select Expand

        (need_for_speed && !get_halfshutter_pressed())
            ? RAW_PREVIEW_GRAY_ULTRA_FAST
            : RAW_PREVIEW_GRAY_ULTRA_FAST

Also forcing B&W preview option can be implemented in a cleaner way under RAW video submenu.

-Will this improve speed?
-Not much really, only in idle (while not recording).

-Is there a way to increase ML Framing preview speed?
-Yes, but the cost is higher chance of corrupted frames while using Framing preview.

This part of code:

Code Select Expand

    /* be gentle with the CPU, save it for recording (especially if the buffer is almost full) */
    msleep(
        (need_for_speed)
            ? ((queued_frames > valid_slot_count / 2) ? 1000 : 500)
            : 50
    );

Controls when it's okay to refresh Framing preview depending on RAW recording state, this part of code add delays while recording for Framing preview to slow down refresh rate.
If you removed this part of code you will have semi real-time preview in B&W during RAW recording, of course this will make CPU usage 100% all time and there is high chance you will have corrupted frames.

But we can probably fine-tune the delay (reduce it), play with 1000 and 500 values (decrease them).
1000 and 500 are in milliseconds.

[Grognard]

Thank you for your reply!

In pratical when recording, we can switch between crop realtime preview and framing greyscale. Framing is usefull to check the correct frame before switching back to crop realtime preview which is stable.
If we can finetune the delay and achieve an almost realtime b&w preview for at least 5 seconds (not really need more) it will be enough for following moving subjects without a proper realtime preview.

[theBilalFakhouri]

Last year (24/7/2022) I shared a list for my plans on DIGIC 4/5 models on Discord (did I share it somewhare on the forum too?):

My current plans on D4/5 (without guarantees):
-Improve SD overclocking, e.g. making 240 MHz SD overclocking stable on 100D/EOS M/5D3.
-Improve memory management among crop_rec, silent picture and RAW video (on 700D/100D).
-Understanding preview, at least how to control it for our purposes; for real-time correct preview in crop modes, clean HDMI (without black borders).
-Lossless on DIGIC 4 if possible.

My highest priority is 240 MHz SD overclocking on 100D, currently I am putting all of my ML time into SD overclocking.
Good news: I got 240 MHz from "it doesn't work at all" state (on 100D) to "it's working partially", I found a new piece in the puzzle.

And hey, speaking today . . we achieved the first goal, stable 240 MHz overclocking across all DIGIC 5 models

-Improve SD overclocking, e.g. making 240 MHz SD overclocking stable on 100D/EOS M/5D3. Done!

Beside, many W90 SD cards now work with the *new* sd_uhs at 240 MHz (have we created a list for working SD cards yet? ), until now nobody reported burned SD card which is good
I wouldn't achieve this without the help of community of course, having a 100D made things easier to me by working on hardware directly, and definitely pushed the excitement and the feel of the challenge.

Also, the work was based on other dev work who is a1ex (literally, his previous amazing work still pushes/excites me to work more on this project), we are missing you.



Okay, so what would be my next goal after SD overclocking on DIGIC 5 from above list?
We all know, time to work again on it . . .




Sneak-peek on what's I am working on/doing in background :

-This month I started to work again on preview, my understanding is improved
-I found a new missing pieces which affect preivew (the pieces which don't show in adtg_gui)
-Now I can fully patch mv1080 in mv720 for the first time with working preview and valid RAW data (no cut):

Doing the same experiment but by patching mv1080 mode into mv720, will have working YUV HD path and RAW data but both streams will have only 1736x696 from the top of frame working, the bottom part of frame will be frozen, something is restricting the amount of data, figuring this out (by comparing 720p vs 1080p configuration, and this time it's not the displayed registers in adtg_gui) will allow us to process more than 1736x1160 in LiveView in 1080p mode.

That means, the new findings can control the limit of processed RAW data in LiveView, unfortunately and untill now my tests to increase processed RAW data in mv1080 *weren't successful

-We can shift preview on screen (LCD/HDMI) which mean we can center it, no need to move ML top bar to bottom
-Also get rid of bottom vram artifacts using Canon configuration, no need to draw black box (current 1x3 presets look like native video modes with these )
-I got the first proof of concept for clean HDMI on 700D, still not perfect though, I ended up with that "I should work on other part of preview first" then clean HDMI.
-If you followed the previous work on preview, current amount of RAW data which we can process in x5 mode is around ~2.7 MP:

The reason why all of 1x3 real-time presets are limited to ~2.7 MP because of preview, going above ~2.7 MP resolution will make Preview frozen or you will lose part of the horizontal /vertical preview (there is a limit here, more research is required).

Huh, but why it's ~2.7 MP? It's 2.7 MP in x5 mode . . two days ago I thought about it and figured out that 2520x1080 (default RAW resolution in x5 mode) is ~2.7 MP

-*I went back and looked into the new findings which made patching mv1080 into mv720 successful, and found out that they were related to two EDMAC channels, and more likely it's a buffer size issue, I believe that explain why I couldn't go over 1736x1160 in 1080p and over 2.7 MP in x5, but I got over 1736x696 in mv720, because mainly -I think- I redirected one of EDMAC channel buffer address for mv720 to mv1080 buffer address which should be theoretically larger.

Right now I am still working on EDMAC findings. take everything with grain of salt and be patience please

-This is applicable to 700D and should be also applicable to similar models
-5D3 have few similarities, but I am still missing a starting point, didn't look deeper yet

-Things will be shared when it's ready, it's might not be for free, too early to talk about these . . today I just wanted to tell that I have some good news about preview thing
-As always, there is no guarantee of success (it could be there Hardware limits or I might never figure it out completely), there is no time-line regarding when this will be out or ready, I hope it won't take years

I am documenting everything I found, hopfully no infos would be lost from my side.

[andy kh]

great finding bilal

[Danne]

Very cool. I guess we could get a good mv1080p mode for eos m here? What about increasing fps here? Will it work better than with regular 1080p?
Is this patch also valid for h264? Might get mv1080p 50 or 60 fps?

[theBilalFakhouri]

Thanks andy kh!


Oh yes! it fu*%ing EDMAC thingie , I could process more than 1736x1160 for the first time in mv1080, by redirecting EDMAC mv1080 buffer address to x5 buffer address which made me think it would rasies the limit to ~2.7 MP instead of ~2 MP, but . . I could porcess ~3.7 MP now in LiveView in mv1080 mode:

Proof of Concept, exceeding the limits:

[gifv]https://i.imgur.com/0LlKXKy.mp4[/gifv]


Un-touched Vram dumps from camera:
https://drive.google.com/drive/folders/1bbRPhuVe4WJFDAyG-oiP8iqi3xaPCvtp?usp=share_link

-and the funny part is that EDMAC channel (which causes limit issue) is used for clearning focus pixels in preview could we just disable it completely?
We might save some memory bandwidth and reduce overhead.

-Unfortunately, I got corrupted frames in 1736x2154 @ 23.976 FPS with real-time preview, this mode isn't usable for RAW recording yet, the cause is the preview, it could be it's from x5 EDMAC buffer which I re-used or a hardware limit,
more tests are needed. Reducing FPS to ~21 FPS helped in this preset (recording was fine).

-I didn't test increasing preview for horizontal resolution, last time we had black bar limit, still no idea where this comes from.
-Also, the preview is now being calculated automagically in crop_rec, there are no hardcoded values for each preset.

I need now to work out how to direct it correctly to our own buffer, that might help . .

[iaburn]

Everything you say is too technical for me to understand the real implications, but sounds like magic, you are a wizard! 

[theBilalFakhouri]

Very cool. I guess we could get a good mv1080p mode for eos m here?

Thanks!
Yes, I think we can patch mv720 to mv1080 with working preview, without using MCM on EOS M (that already worked on 700D).

What about increasing fps here? Will it work better than with regular 1080p?

1080p mode has 1736x1160 RAW resolution, if we could reduce vertical resolution to 1736x976 16:9 or 1736x738 2.35:1 and got preview working (not frozen) we can definitely gain more FPS.

Is this patch also valid for h264? Might get mv1080p 50 or 60 fps?

The sensor isn't fast enough to scan 1736x976 at 60 FPS, on 700D 1736x868 @ 50 FPS is possible for example, but I have no idea if H.264 encoder is fast enough to encode that.
Anyway, I will not touch H.264 things, H.264 is already horrible on our cams, MJPEG make more sense if you ask me (too early to even talk about it)

[ML700D]

1080p mode has 1736x1160 RAW resolution, if we could reduce vertical resolution to 1736x976 16:9 or 1736x738 2.35:1 and got preview working (not frozen) we can definitely gain more FPS.

When recording I accidentally use 5x zoom in 1736x976 mode 25fps (canon setting) and the fps changed to 29.954 maybe it gives you something to investigate..
I like using this "accidental mode" when use wide lens 🤣
but sometimes I got freeze LV too..

[Danne]

Thanks for the files bilal. I converted it to a MLV with raw2mlv but probably was MLV already  and opened it in Mlv App. Unstretched and hell yeah. Wizardry in the making. Here´s the MLV file for playing:
https://bitbucket.org/Dannephoto/raw2mlv/downloads/RAW-002.MLV

[ML700D]

max resolution is 2520 x 1080 from 1736 x 976 from what I have tried..

[theBilalFakhouri]

When recording I accidentally use 5x zoom in 1736x976 mode 25fps (canon setting) and the fps changed to 29.954 maybe it gives you something to investigate..

I like using this "accidental mode" when use wide lens 🤣

off-topic here, please reports any *issues* related to my builds in my dedicated thread Bilal's crop_rec_4k experiments for 650D / 700D (T4i / T5i)

That mode is called x5 mode which can be entered by pressing magnification button in LiveView, and 30 FPS is normal in x5 mode, x5 mode is always 30 FPS by default (that's how Canon made it) and it discard FPS settings in Canon menu.

max resolution is 2520 x 1080 from 1736 x 976 from what I have tried..
..

Same thing, in default x5 mode (without crop_rec) the maximum resolution is 2520x1080 @ 29.954 FPS by Canon default, that's normal.

but sometimes I got freeze LV too..

Steps to reproduce?

[theBilalFakhouri]

Everything you say is too technical for me to understand the real implications, but sounds like magic, you are a wizard! 

Thanks! spending a lot of time and doing a lot of experiments helps with making sense of things
But for sure, I never understand every little thing, but I understand things enough.

[ML700D]

Steps to reproduce?

I'm not aware of that..
but I will report then in your thread Bilal's crop_rec_4k experiments for 650D / 700D (T4i / T5i) if it happens again.

[theBilalFakhouri]

Processing 1736x2214 @ 22 FPS in LiveView works and recording is fine, increasing FPS higher than that will start to introduce problems like choppy preview and corrupted frames, setting FPS to 23.976 freezes both RAW and preview streams.
I think we hit memory bandwidth limit here (could be processing a.k.a Hardware limit?), not a buffer size this time.

I am trying to change EDMAC#24 channel (the one which correct focus pixels, called "Safari") to *other channel, but this requires patching 10 ROM addresses (I am using patch_instruction) which works, but when enabling crop_rec, not all of the addresses will patched (cache collision), I need to find an other way to change the channel.

*Because some EDMAC channels **performs worse than others, I am just geussing that EDMAC#24 channel has limited bandwidth, assuming the problem is coming from it . .
  If this doesn't work (if changing the channel doesn't perform better), we need to find a way to skip focus pixel correction step, and pass LiveView streams without it, hoping this will fix the issue,
  of course, I am still assuming that  the problem is coming from "Safari" process. I don't how to disable Safari and and pass streams . . yet.

**In the past, I tried some EDMAC write channels for RAW recording, some of them introduced corrupted frames, while others were perfect. And of course in our builds we are using the perfect ones.

[ShittyWebsite]

Perhaps good news for 5D3? 😍

[theBilalFakhouri]

Perhaps good news for 5D3? 😍

There is no news for 5D3 . . at least, not yet.

But if 700D was powerful enough to process 1736x2214 @ 22 FPS in LiveView + record RAW video + doing lossless compression in the same time , 5D3 should be even more powerful, it has faster CPU and RAM.

My tests on 5D3 were by comparing x5 and x10 modes, I was always trying to figure out how cropping is happening, no luck so far.
But in the coming weeks I will start to compare mv720 vs mv1080 configurations on 5D3, might find something there.

Will spend all of my ML time on Preview this year.

[dpjpandone]

-Also, the preview is now being calculated automagically in crop_rec, there are no hardcoded values for each preset.

Woah! That is wonderful!

[dpjpandone]

**In the past, I tried some EDMAC write channels for RAW recording, some of them introduced corrupted frames, while others were perfect. And of course in our builds we are using the perfect ones.

I had this problem in 2014-2015 working on 7D. Tearing in the recorded image (like a split screen) when external monitor was taxing CPU. Changing to different EDMAC channel fixed it. I wish i could tell you why.....

[theBilalFakhouri]

I splitted the post regarding "Framing" preview, it's being off-topic here. Here is it:
Fine-tuning Framing preview to speed it up

Let's continue there.

[theBilalFakhouri]

Processing 1736x2214 @ 23.976 FPS in LiveView works
yeah yeah ez ez

[theBilalFakhouri]

It seems the hardware is powerful enough, thanks Canon!

[iaburn]

Processing 1736x2214 @ 23.976 FPS in LiveView works
yeah yeah ez ez

Does it means that real time preview on the EOS M at 1736x2178 might be possible??   

[theBilalFakhouri]

Does it means that real time preview on the EOS M at 1736x2178 might be possible??   

Yes, that's exactly what does that mean

EOS M/EOS M2/700D/650D/100D are benafited from this research and achievements, because their internals are very similar, almost identical.
Previous LiveView work (current 1x3 presets with real-time, a.k.a frtp presets on EOSM) was started on 700D then ported and fine-tuned by Danne to EOS M (it's almost copy-paste stuff here).