Bilal's crop_rec_4k experiments for 100D / SL1

Started by theBilalFakhouri, June 06, 2022, 12:50:10 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

theBilalFakhouri

Welcome to:


-What is this?
It's a port from my custom builds for 650D/700D to 100D, please read 650D/700D thread for more details then come back here and read the differences.

-Port status:
This port is a work in progress, expect some missing things compared to 650D/700D builds, please report bugs if you found some.


-What's missing:
-crop_new module (Crop mode V2), not ported yet (didn't make tries), will work on it soon (should work after making some tweaks).

-crop_rec (Crop mode):
25 FPS isn't working in UHD 1x3 preset (hardware limit), only 23.976 FPS works.

"One more hack" doesn't work on 100D, it freezes LiveView same as EOS M.

-What's working:
All other things not mentioned above are working fine :) (well, HDMI output not tested yet).

-Quirks (not related to my build, but I have noticed it):

When selecting 3K and 1440p presets you need to turn off then on RAW video multiple times to let RAW video update RAW video resolution parameters.

Recording stops very early on 100D e.g. when recording high resolutions like 2560x1440p @ 23.976 FPS at 10 bit lossless with 192 MHz overclock compared to 700D (on 100D it's 1-2 seconds while on 700D it's up to 11 seconds, same settings, same scene, same everything).

Edit: Fixed the second issue, also provided more details for the first issue, read this.

-Download:
crop_rec_4k.2022Jun19.100D101.zip

-Focus Pixel Maps:
I have added them too.

-Other notes:
-Will update this OP and add more details, also make it nicer with more photos and showcasing short videos as porting progressing.

theBilalFakhouri


Danne

Probably possibble to reuse existing pixel maps. Trial and error in mlv app should be helpful.

theBilalFakhouri

Quote from: Danne on June 06, 2022, 11:52:11 AM
Probably possibble to reuse existing pixel maps. Trial and error in mlv app should be helpful.

I have tried, changed focus maps name to match my 3K and 1440p presets in terms of width and height (with black areas, I got width and height from mlv_dump), but MLVApp crashes for some reason
Edit: it seems I have downloaded wrong files, existing pixel maps appear to work! I have just added them in MLVApp repo.

On 700D I have captured maximum resolution in each Binning mode, e.g. 5208x3478 1:1, 1736x3478 in 1x3, then I sent them to dfort and he created the focus pixel maps for them, this way the same pixel map will work in any resolution, you just need to rename the same map to match mlv_dump height and width.

theBilalFakhouri

Added focus pixel maps for 100D in MLVApp repository, should be downloaded automatically when you open a MLV clip.
You can also Download them manually (add them in MLVApp folder root).

No need to use Chroma Smoothing anymore.

GianlucaM83

Thanks for bring back magic lantern to canon 100D with fresh feature  :)
I am so happy that I still have this great camera  8)

DeafEyeJedi

THIS is the SHIT!  8)

I AM game.

Gonna test this one out along with the LiveView Hack builds that you did.

Thank you guys!
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

theBilalFakhouri

Thanks guys @GianlucaM83 @DeafEyeJedi!
Looking forward for your reports/results :D

Thanks to donors who gave me the opportunity to work on 100D.

theBilalFakhouri

I figured out what's causing these:

Quote from: theBilalFakhouri on June 06, 2022, 12:50:10 AM
-Quirks (not related to my build, but I have noticed it):

When selecting 3K and 1440p presets you need to turn off then on RAW video multiple times to let RAW video update RAW video resolution parameters.

Recording stops very early on 100D e.g. when recording high resolutions like 2560x1440p @ 23.976 FPS at 10 bit lossless with 192 MHz overclock compared to 700D (on 100D it's 1-2 seconds while on 700D it's up to 11 seconds, same settings, same scene, same everything). WTF? aren't both cams have same buffer size? (will look if there is a solution in other branches).


It's from DEFAULT_RAW_BUFFER_SIZE (in raw.c) which is too small, on 100D it's 5.2 MB vs 650D/700D/EOSM it's 24 MB!
I tried to increase it (I changed 100D buffer size value to 700D buffer size value) and above problems are no longer there, but of course this won't work, the camera will freeze (I made this test just to make sure if above issues are coming from DEFAULT_RAW_BUFFER_SIZE).

I doubled checked if 5.2 MB default buffer size on 100D might be a typo or something, and it seems correct :(
Here was the commit for it, details how we got it, I run a1ex test and it was reporting 5.2 MB in all video modes.

So for now:
-Can we change DEFAULT_RAW_BUFFER to another one (which has bigger size)? or it must be Canon RAW buffer?
-Can we increase DEFAULT_RAW_BUFFER_SIZE a little more (until above issues are no longer there), would this cause problem? how can we increase it in a safe way? from where 5.2 MB limit is coming from?

I need some help here from other Devs.

theBilalFakhouri

Quote from: a1ex on March 07, 2018, 11:36:03 PM
Unless somebody starts porting the 4K and full-res LiveView on these cameras, it won't make any difference. ...

100D/EOSM2 x5: 2592 * 1108 * 14/8 = 5025888 (smaller than 5.2MB).

...

So now we need at least 3144 * 1337 * 14/8 = 7,356,174 = ~ 7.4 MB buffer size.

theBilalFakhouri

-Some info:
Total shoot memory size is ~91 MB
Total  SRM  memory size is ~31 MB
Total:                             ~122 MB 

We are using both of these for RAW video recording (saving frames to memory before writing it to SD card), beside there is 24 MB from LV RAW buffer on 700D (5.2 MB on 100D) which is being used for RAW buffer.

I remembered that we actually allocate buffer size as needed in very high resolution (when RAW frame size is higher than LV RAW buffer size a.k.a DEFAULT_RAW_BUFFER)
like Full-Res LiveView on 700D or 1440p and 3K on 100D (we are not restricted to 24 MB on 700D or 5.2 MB on 100D, see):

On 700D when I use Full-Res LiveView preset (5208x3478), new RAW buffer gets allocated from shoot memory, and allocation size "RAW_LV_BUFFER_ALLOC_SIZE" is defined as "SRM_BUFFER_SIZE - 0x1000"
You can see how much is SRM_BUFFER_SIZE in consts.h for each model, on 700D/100D it's ~31 MB, so in this case LV RAW buffer will take ~31 MB from shoot memory and the remaining ~59 MB will get allocated by mlv_lite (for RAW video recording), and SRM memory won't be used for both LV RAW buffer and RAW video recording which is weird! why SRM memory won't be allocated for RAW video recoding?

which mean it's waste of ~31 MB of memory, probably the code in raw.c need some modifications (for SRM memory issue)?

Okay, back to 100D, when selecting 1440p (needs ~6.7 MB RAW buffer size) or 3K (needs ~7.4 MB RAW buffer size) presets, the new RAW buffer will get alloctaed from shoot memory, and you might already know that allocation size is defined as "SRM_BUFFER_SIZE - 0x1000" which is ~31MB which is higher than what we actually need!

So we are wasting ~24 MB of shoot memory, remaining would be 59 MB, and for some reason SRM memory won't be allocated by mlv_lite (already said that), so we are wasting total of ~55 MB of memory for RAW video recording, and that's why RAW video recording stops very early on 100D when recording at 3K or 1440p.

So after figuring out above things, I changed "RAW_LV_BUFFER_ALLOC_SIZE" to match RAW frame size (a tiny bit higher), from "SRM_BUFFER_SIZE - 0x1000" to "raw_info.width*raw_info.height*14/8 + 0x1000", by doing that we have left of ~83 MB from shoot memory, and also SRM memory will get allocated by mlv_lite (RAW video) :)

Which mean --> longer recording times at higher resolutions on 100D :D

-And regarding why you will need to turn off and on RAW video multiple times to update RAW video parameters:
That's because RAW video will allocate all shoot memory when it's on, turning off RAW video will free RAW video buffer and set shoot memory free then this thing will let crop_rec (and raw.c) allocate new RAW buffer from shoot memory (for high resolutions), then by turning on RAW video again will let it allocate remaining shoot memory.

More like a race condition, for sure this is fixable, need some code modifications for raw.c and mlv_lite.c and probably crop_rec.c.

-By figuring above things will open the door for some enhancements for Full-Res LiveView recording.

-Test build?
Sure: crop_rec_4k.2022Jun19.100D101.zip

Note: You will need to turn off and on RAW video multiple times after selecting 1440p or 3K presets, to let RAW video update its parameters!!
Please report for any issues
.

theBilalFakhouri


theBilalFakhouri

Quote from: theBilalFakhouri on June 15, 2022, 12:03:36 PM
-Some info:
Total shoot memory size is ~91 MB
Total  SRM  memory size is ~31 MB
Total:                             ~122 MB 

Above results were when "Image quality" was set to JPG from Canon settings, read Image quality and its effect on memory (more RAM 4 free) for more details.

So by setting "Image quality" to RAW then restarting the camera, you get:

Total shoot memory size is     ~95 MB
Total  SRM  memory size is     ~62 MB
Total:                                 ~157 MB 

(do your calculations for the rest of reply #10 :P)

--> Which has positive impact on RAW recording times.

theBilalFakhouri

-Results on 100D (because of extra RAM):

-2560x1440p @ 23.976 FPS 10-bit lossless, 192 MHz SD overclock, sound recording on, lvface + aewb hacks are on, GD off (~70 MB/s required write speed reported by RAW video):

Before: ~17 seconds.
After:  ~50 seconds.

-Other scene (same settings):

Before: First try: ~30 seconds,  Second try: ~33 seconds.
After:   First try: 1 Minuets and 57 seconds,  Second try: 1 Minuets and 27 seconds.

Skinny

@theBilalFakhouri congrats! very nice results! :) almost two minutes is impressive for 2560x1440 resolution.

theBilalFakhouri

Thanks @Skinny, I am happy that I could enhance recording times on 100D with high resolutions (it was only 1-2 seconds maximum at 2560x1440p with same above settings on 100D :P).

Could you do this test on 5D2? also 550D because . . why not :).


theBilalFakhouri

UHD 1280x2160 1x3 16:9 preset is alive on 100D :) , also 25 FPS now works in 4.3K 1x3 preset:

New build: crop_rec_4k.2022Jun19.100D101.zip

-UHD 1x3 probably won't have 25 FPS, this time it's a hardware limit (slow sensor speed).
-Also a new focus pixel map should be created for UHD 1x3 preset, some focus pixels in the bottom of frame aren't covered with exiting focus pixel map (editing current pixel map might be easy? sorry never played with them).
But there is a tutorial somewhere in the forum for creating focus pixel map, currently I will upload the exiting map to MLVApp repo, also it will be included here.




It was CMOS 7 issues, not FPS Timer B:
-UHD 1x3: changed it from 0xB86 to 0xB66.
-4.3K 1x3: changed it from 0xB87 to 0xB67. (0xB87 works fine at 23.976 FPS, but 0xB67 was required to remove artifacts when using 25 FPS, I don't have such problem on 700D?)

ArcziPL

M50.110 [main cam] | G7X III [pocket cam] | 70D.112 [gathers dust] | M.202 [gathers dust] | waiting for M5II

SebastianC

Thank you, Bilal! Great job!

Eos 100d 4k crop video(4.3k mode)
Up to 5-9 seconds every record.

https://youtu.be/qFi05EyY87o

Camera: Canon 100d
Lens: EF 35mm f2 is
crop_rec_4k.2022Jun19.100D101
SD card: Sandisk Ex pro 128GB 170mb.
4K Crop mode: 4080 × 1836

Music: The Easy Winners - E's Jammy Jams

I hope long to record because no mode has long record times, all modes have up to 15 seconds per time.

theBilalFakhouri

@SebastianC

No problem, have fun :)

Quote from: SebastianC on June 24, 2022, 03:06:10 AM
I hope long to record because no mode has long record times, all modes have up to 15 seconds per time.

Actually all 1x3 presets are continuous but with these settings:

-Enable 192 MHz SD overclock
-Turn on the new hacks which are "lvface + aewb" from RAW video submenu

You might get continuous recording with these at 14-bit lossless (mostly depends on scene), but if you want 100% continuous recording set "bit-depth" to 12-bit from "Crop mode" submenu (you must keep 14-bit lossless in RAW video submenu).

3K and 1440p presets won't be continuous, you can lower its resolution or aspect ratio from RAW video submenu, also use lower bit depths in these presets.

SebastianC

Quote from: theBilalFakhouri on June 24, 2022, 07:36:09 PM
@SebastianC

No problem, have fun :)

Actually all 1x3 presets are continuous but with these settings:

-Enable 192 MHz SD overclock
-Turn on the new hacks which are "lvface + aewb" from RAW video submenu

You might get continuous recording with these at 14-bit lossless (mostly depends on scene), but if you want 100% continuous recording set "bit-depth" to 12-bit from "Crop mode" submenu (you must keep 14-bit lossless in RAW video submenu).

3K and 1440p presets won't be continuous, you can lower its resolution or aspect ratio from RAW video submenu, also use lower bit depths in these presets.

I will try later today,Great! excited!

SebastianC

Quote from: SebastianC on June 24, 2022, 08:07:40 PM
I will try later today,Great! excited!
I tested today:

4.3k :1440x1836,12bits,23.973fps,record up to 4m 45s.

1440p :2560x1440,10bits,record up to 1m 24s.

1736-976: record up to 2m 23s

Thank you for your help, Bilal! You are a genius!

GianlucaM83

hi.
i performed some tests with the latest build "crop_rec_4k.2022Jun19.100D101".

This is the SD card i use:
SanDisk Extreme Pro, 32Gb, 95 MB/S UHS-1, Class 10, U3, V30

These are the settings i used:
SD overclock 192 MHz
Small hacks on + more hack "lvface + aewb".
Preview: realtime
Global draw: On, all modes + focus Peak on + histogram: RAW RGB, Log

these are the results:

-UHD 1X3 1280x2160 - No audio
12 bit Lossless, 23.973 fps
Frames 64, Duration 2 Sec

-4K 1x3: 1x3 Binning 1360x1976 - No audio
12 bit Lossless, 23.973 fps
Frames 65, Duration 2 Sec

-4K 1x3: 1x3 Binning 1360x1976 - No audio
10 bit Lossless, 25 fps
Frames 71, Duration 2 Sec

I suspect that my sd card is not compatible with the sd overclock module at all  :(

if you need I can upload the example mlv files and provide more details


sorry for my bad English

Walter Schulz

Load bench.mo in modules tab. Change overclocking frequency to 160 MHz. Restart camera.
Access Debug tab -> Benchmarks -> Card benchmarks -> Quick R/W benchmark (1 min) and start it.
If card defaults to 19-21 MByte/s -> Lower frequency, restart and retest.