Magic Lantern Forum

Experimental builds (WIP) => crop_rec and derived builds => Topic started by: theBilalFakhouri on June 06, 2022, 12:50:10 AM

Title: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on June 06, 2022, 12:50:10 AM
Welcome to:
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2F47fVSm4%2FBilal-Build-for-100-D.png&hash=e68bd5e0c6650317cb05ee2705fa6afd)


-What is this?
It's a port from my custom builds for 650D/700D (https://www.magiclantern.fm/forum/index.php?topic=25784.msg231049#msg231049) to 100D, please read 650D/700D thread for more details then come back here and read the differences.

-Port status:
This port is a work in progress, expect some missing things compared to 650D/700D builds, please report bugs if you found some.


-What's missing:
-crop_new module (Crop mode V2), not ported yet (didn't make tries), will work on it soon (should work after making some tweaks).

-crop_rec (Crop mode):
25 FPS isn't working in UHD 1x3 preset (hardware limit), only 23.976 FPS works.

"One more hack" doesn't work on 100D, it freezes LiveView same as EOS M.

-What's working:
All other things not mentioned above are working fine :) (well, HDMI output not tested yet).

-Quirks (not related to my build, but I have noticed it):

When selecting 3K and 1440p presets you need to turn off then on RAW video multiple times to let RAW video update RAW video resolution parameters.

Recording stops very early on 100D e.g. when recording high resolutions like 2560x1440p @ 23.976 FPS at 10 bit lossless with 192 MHz overclock compared to 700D (on 100D it's 1-2 seconds while on 700D it's up to 11 seconds, same settings, same scene, same everything).

Edit: Fixed the second issue, also provided more details for the first issue, read this (https://www.magiclantern.fm/forum/index.php?topic=26511.msg239224#msg239224).

-Download:
crop_rec_4k.2022Jun19.100D101.zip (https://bitbucket.org/bilal_fakhouri/magic-lantern/downloads/crop_rec_4k.2022Jun19.100D101.zip)

-Focus Pixel Maps:
I have added them too (https://www.magiclantern.fm/forum/index.php?topic=26511.msg239140#msg239140).

-Other notes:
-Will update this OP and add more details, also make it nicer with more photos and showcasing short videos as porting progressing.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on June 06, 2022, 10:21:56 AM
Updated the build, same link:

-Download:
crop_rec_4k.2022Jun06.100D101.zip (https://bitbucket.org/bilal_fakhouri/magic-lantern/downloads/crop_rec_4k.2022Jun06.100D101.zip)

-Fixed white bar in the right in 1920x1280 1:1 preset.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: Danne on June 06, 2022, 11:52:11 AM
Probably possibble to reuse existing pixel maps. Trial and error in mlv app should be helpful.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on June 06, 2022, 12:58:29 PM
Probably possibble to reuse existing pixel maps. Trial and error in mlv app should be helpful.

I have tried, changed focus maps name to match my 3K and 1440p presets in terms of width and height (with black areas, I got width and height from mlv_dump), but MLVApp crashes for some reason
Edit: it seems I have downloaded wrong files, existing pixel maps appear to work! I have just added them in MLVApp repo.

On 700D I have captured maximum resolution in each Binning mode, e.g. 5208x3478 1:1, 1736x3478 in 1x3, then I sent them to dfort and he created the focus pixel maps for them, this way the same pixel map will work in any resolution, you just need to rename the same map to match mlv_dump height and width.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on June 06, 2022, 02:30:06 PM
Added focus pixel maps for 100D in MLVApp repository, should be downloaded automatically when you open a MLV clip.
You can also Download (https://drive.google.com/file/d/1LJaOYqqZT3kREext4EQPVcER-kinkXaD/view?usp=sharing) them manually (add them in MLVApp folder root).

No need to use Chroma Smoothing anymore.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: GianlucaM83 on June 06, 2022, 06:05:00 PM
Thanks for bring back magic lantern to canon 100D with fresh feature  :)
I am so happy that I still have this great camera  8)
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: DeafEyeJedi on June 06, 2022, 07:15:06 PM
THIS is the SHIT!  8)

I AM game.

Gonna test this one out along with the LiveView Hack builds that you did.

Thank you guys!
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on June 13, 2022, 12:52:53 AM
Thanks guys @GianlucaM83 @DeafEyeJedi!
Looking forward for your reports/results :D

Thanks to donors  (https://www.magiclantern.fm/forum/index.php?topic=26250.msg239078#msg239078) who gave me the opportunity to work on 100D.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on June 13, 2022, 01:10:42 AM
I figured out what's causing these:

-Quirks (not related to my build, but I have noticed it):

When selecting 3K and 1440p presets you need to turn off then on RAW video multiple times to let RAW video update RAW video resolution parameters.

Recording stops very early on 100D e.g. when recording high resolutions like 2560x1440p @ 23.976 FPS at 10 bit lossless with 192 MHz overclock compared to 700D (on 100D it's 1-2 seconds while on 700D it's up to 11 seconds, same settings, same scene, same everything). WTF? aren't both cams have same buffer size? (will look if there is a solution in other branches).


It's from DEFAULT_RAW_BUFFER_SIZE (in raw.c) which is too small, on 100D it's 5.2 MB vs 650D/700D/EOSM it's 24 MB!
I tried to increase it (I changed 100D buffer size value to 700D buffer size value) and above problems are no longer there, but of course this won't work, the camera will freeze (I made this test just to make sure if above issues are coming from DEFAULT_RAW_BUFFER_SIZE).

I doubled checked if 5.2 MB default buffer size on 100D might be a typo or something, and it seems correct :(
Here was the commit  (https://foss.heptapod.net/magic-lantern/magic-lantern/-/commit/9936e350480223bf1da5a9e198698d97a3049421)for it, details how we got it (https://www.magiclantern.fm/forum/index.php?topic=5601.msg196632#msg196632), I run a1ex test and it was reporting 5.2 MB in all video modes.

So for now:
-Can we change DEFAULT_RAW_BUFFER to another one (which has bigger size)? or it must be Canon RAW buffer?
-Can we increase DEFAULT_RAW_BUFFER_SIZE a little more (until above issues are no longer there), would this cause problem? how can we increase it in a safe way? from where 5.2 MB limit is coming from?

I need some help here from other Devs.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on June 13, 2022, 02:40:26 AM
Unless somebody starts porting the 4K and full-res LiveView on these cameras, it won't make any difference. ...

100D/EOSM2 x5: 2592 * 1108 * 14/8 = 5025888 (smaller than 5.2MB).

...

So now we need at least 3144 * 1337 * 14/8 = 7,356,174 = ~ 7.4 MB buffer size.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on June 15, 2022, 12:03:36 PM
-Some info:
Total shoot memory size is ~91 MB
Total  SRM  memory size is ~31 MB
Total:                             ~122 MB 

We are using both of these for RAW video recording (saving frames to memory before writing it to SD card), beside there is 24 MB from LV RAW buffer on 700D (5.2 MB on 100D) which is being used for RAW buffer.

I remembered that we actually allocate buffer size as needed in very high resolution (when RAW frame size is higher than LV RAW buffer size a.k.a DEFAULT_RAW_BUFFER)
like Full-Res LiveView on 700D or 1440p and 3K on 100D (we are not restricted to 24 MB on 700D or 5.2 MB on 100D, see (https://foss.heptapod.net/magic-lantern/magic-lantern/-/commit/7ad3754768b26a106d057550ca359089ba837fcf)):

On 700D when I use Full-Res LiveView preset (5208x3478), new RAW buffer gets allocated from shoot memory, and allocation size "RAW_LV_BUFFER_ALLOC_SIZE" is defined as "SRM_BUFFER_SIZE - 0x1000"
You can see how much is SRM_BUFFER_SIZE in consts.h for each model, on 700D/100D it's ~31 MB, so in this case LV RAW buffer will take ~31 MB from shoot memory and the remaining ~59 MB will get allocated by mlv_lite (for RAW video recording), and SRM memory won't be used for both LV RAW buffer and RAW video recording which is weird! why SRM memory won't be allocated for RAW video recoding?

which mean it's waste of ~31 MB of memory, probably the code in raw.c need some modifications (for SRM memory issue)?

Okay, back to 100D, when selecting 1440p (needs ~6.7 MB RAW buffer size) or 3K (needs ~7.4 MB RAW buffer size) presets, the new RAW buffer will get alloctaed from shoot memory, and you might already know that allocation size is defined as "SRM_BUFFER_SIZE - 0x1000" which is ~31MB which is higher than what we actually need!

So we are wasting ~24 MB of shoot memory, remaining would be 59 MB, and for some reason SRM memory won't be allocated by mlv_lite (already said that), so we are wasting total of ~55 MB of memory for RAW video recording, and that's why RAW video recording stops very early on 100D when recording at 3K or 1440p.

So after figuring out above things, I changed "RAW_LV_BUFFER_ALLOC_SIZE" to match RAW frame size (a tiny bit higher), from "SRM_BUFFER_SIZE - 0x1000" to "raw_info.width*raw_info.height*14/8 + 0x1000", by doing that we have left of ~83 MB from shoot memory, and also SRM memory will get allocated by mlv_lite (RAW video) :)

Which mean --> longer recording times at higher resolutions on 100D :D

-And regarding why you will need to turn off and on RAW video multiple times to update RAW video parameters:
That's because RAW video will allocate all shoot memory when it's on, turning off RAW video will free RAW video buffer and set shoot memory free then this thing will let crop_rec (and raw.c) allocate new RAW buffer from shoot memory (for high resolutions), then by turning on RAW video again will let it allocate remaining shoot memory.

More like a race condition, for sure this is fixable, need some code modifications for raw.c and mlv_lite.c and probably crop_rec.c.

-By figuring above things will open the door for some enhancements for Full-Res LiveView recording.

-Test build?
Sure: crop_rec_4k.2022Jun19.100D101.zip (https://bitbucket.org/bilal_fakhouri/magic-lantern/downloads/crop_rec_4k.2022Jun19.100D101.zip)

Note: You will need to turn off and on RAW video multiple times after selecting 1440p or 3K presets, to let RAW video update its parameters!!
Please report for any issues
.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on June 15, 2022, 01:17:34 PM
-Uploaded the source code for Devs:
magic-lantern-bilal-100d (15-6-2022).7z (https://drive.google.com/file/d/1evYxe7KXE5tEKR1UBLGHzlfRXvfXy3HT/view?usp=sharing)
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on June 15, 2022, 06:42:53 PM
-Some info:
Total shoot memory size is ~91 MB
Total  SRM  memory size is ~31 MB
Total:                             ~122 MB 

Above results were when "Image quality" was set to JPG from Canon settings, read Image quality and its effect on memory (more RAM 4 free) (https://www.magiclantern.fm/forum/index.php?topic=26521.msg239231#msg239231) for more details.

So by setting "Image quality" to RAW then restarting the camera, you get:

Total shoot memory size is     ~95 MB
Total  SRM  memory size is     ~62 MB
Total:                                 ~157 MB 

(do your calculations for the rest of reply #10 (https://www.magiclantern.fm/forum/index.php?topic=26511.msg239224#msg239224) :P)

--> Which has positive impact on RAW recording times.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on June 16, 2022, 03:35:48 PM
-Results on 100D (because of extra RAM (https://www.magiclantern.fm/forum/index.php?topic=26511.msg239232#msg239232)):

-2560x1440p @ 23.976 FPS 10-bit lossless, 192 MHz SD overclock, sound recording on, lvface + aewb hacks are on, GD off (~70 MB/s required write speed reported by RAW video):

Before: ~17 seconds.
After:  ~50 seconds.

-Other scene (same settings):

Before: First try: ~30 seconds,  Second try: ~33 seconds.
After:   First try: 1 Minuets and 57 seconds,  Second try: 1 Minuets and 27 seconds.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: Skinny on June 16, 2022, 04:20:54 PM
@theBilalFakhouri congrats! very nice results! :) almost two minutes is impressive for 2560x1440 resolution.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on June 16, 2022, 10:44:41 PM
Thanks @Skinny, I am happy that I could enhance recording times on 100D with high resolutions (it was only 1-2 seconds maximum at 2560x1440p with same above settings on 100D :P).

Could you do this test (https://www.magiclantern.fm/forum/index.php?topic=26521.msg239231#msg239231) on 5D2? also 550D because . . why not :).
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: Skinny on June 18, 2022, 06:38:55 PM
@theBilalFakhouri sure, done: https://www.magiclantern.fm/forum/index.php?topic=26521.msg239289#msg239289
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on June 19, 2022, 03:43:04 AM
UHD 1280x2160 1x3 16:9 preset is alive on 100D :) , also 25 FPS now works in 4.3K 1x3 preset:

New build: crop_rec_4k.2022Jun19.100D101.zip (https://bitbucket.org/bilal_fakhouri/magic-lantern/downloads/crop_rec_4k.2022Jun19.100D101.zip)

-UHD 1x3 probably won't have 25 FPS, this time it's a hardware limit (slow sensor speed).
-Also a new focus pixel map should be created for UHD 1x3 preset, some focus pixels in the bottom of frame aren't covered with exiting focus pixel map (editing current pixel map might be easy? sorry never played with them).
 But there is a tutorial somewhere in the forum for creating focus pixel map, currently I will upload the exiting map to MLVApp repo, also it will be included here (https://www.magiclantern.fm/forum/index.php?topic=26511.msg239140#msg239140).



It was CMOS 7 issues, not FPS Timer B:
-UHD 1x3: changed it from 0xB86 to 0xB66.
-4.3K 1x3: changed it from 0xB87 to 0xB67. (0xB87 works fine at 23.976 FPS, but 0xB67 was required to remove artifacts when using 25 FPS, I don't have such problem on 700D?)
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: ArcziPL on June 19, 2022, 11:35:04 AM
Well done! You're doing a fantastic work!
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: SebastianC on June 24, 2022, 03:06:10 AM
Thank you, Bilal! Great job!

Eos 100d 4k crop video(4.3k mode)
Up to 5-9 seconds every record.


Camera: Canon 100d
Lens: EF 35mm f2 is
crop_rec_4k.2022Jun19.100D101
SD card: Sandisk Ex pro 128GB 170mb.
4K Crop mode: 4080 × 1836

Music: The Easy Winners - E's Jammy Jams

I hope long to record because no mode has long record times, all modes have up to 15 seconds per time.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on June 24, 2022, 07:36:09 PM
@SebastianC

No problem, have fun :)

I hope long to record because no mode has long record times, all modes have up to 15 seconds per time.

Actually all 1x3 presets are continuous but with these settings:

-Enable 192 MHz SD overclock
-Turn on the new hacks which are "lvface + aewb" from RAW video submenu

You might get continuous recording with these at 14-bit lossless (mostly depends on scene), but if you want 100% continuous recording set "bit-depth" to 12-bit from "Crop mode" submenu (you must keep 14-bit lossless in RAW video submenu).

3K and 1440p presets won't be continuous, you can lower its resolution or aspect ratio from RAW video submenu, also use lower bit depths in these presets.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: SebastianC on June 24, 2022, 08:07:40 PM
@SebastianC

No problem, have fun :)

Actually all 1x3 presets are continuous but with these settings:

-Enable 192 MHz SD overclock
-Turn on the new hacks which are "lvface + aewb" from RAW video submenu

You might get continuous recording with these at 14-bit lossless (mostly depends on scene), but if you want 100% continuous recording set "bit-depth" to 12-bit from "Crop mode" submenu (you must keep 14-bit lossless in RAW video submenu).

3K and 1440p presets won't be continuous, you can lower its resolution or aspect ratio from RAW video submenu, also use lower bit depths in these presets.

I will try later today,Great! excited!
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: SebastianC on June 25, 2022, 03:49:50 AM
I will try later today,Great! excited!
I tested today:

4.3k :1440x1836,12bits,23.973fps,record up to 4m 45s.

1440p :2560x1440,10bits,record up to 1m 24s.

1736-976: record up to 2m 23s

Thank you for your help, Bilal! You are a genius!
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: GianlucaM83 on June 26, 2022, 10:56:24 AM
hi.
i performed some tests with the latest build "crop_rec_4k.2022Jun19.100D101".

This is the SD card i use:
SanDisk Extreme Pro, 32Gb, 95 MB/S UHS-1, Class 10, U3, V30

These are the settings i used:
SD overclock 192 MHz
Small hacks on + more hack "lvface + aewb".
Preview: realtime
Global draw: On, all modes + focus Peak on + histogram: RAW RGB, Log

these are the results:

-UHD 1X3 1280x2160 - No audio
12 bit Lossless, 23.973 fps
Frames 64, Duration 2 Sec

-4K 1x3: 1x3 Binning 1360x1976 - No audio
12 bit Lossless, 23.973 fps
Frames 65, Duration 2 Sec

-4K 1x3: 1x3 Binning 1360x1976 - No audio
10 bit Lossless, 25 fps
Frames 71, Duration 2 Sec

I suspect that my sd card is not compatible with the sd overclock module at all  :(

if you need I can upload the example mlv files and provide more details


sorry for my bad English
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: Walter Schulz on June 26, 2022, 11:09:42 AM
Load bench.mo in modules tab. Change overclocking frequency to 160 MHz. Restart camera.
Access Debug tab -> Benchmarks -> Card benchmarks -> Quick R/W benchmark (1 min) and start it.
If card defaults to 19-21 MByte/s -> Lower frequency, restart and retest.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: GianlucaM83 on June 26, 2022, 12:35:41 PM
Load bench.mo in modules tab. Change overclocking frequency to 160 MHz. Restart camera.
Access Debug tab -> Benchmarks -> Card benchmarks -> Quick R/W benchmark (1 min) and start it.
If card defaults to 19-21 MByte/s -> Lower frequency, restart and retest.
Thanks for the help
In the tests I get a write and read speed of about 21 mb/s in all three modes "160 MHz - 192 MHz - 240 MHz"
If I do the test in video mode I get these errors: Alloc error: buffer=16777216

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2Fx8d4nQQR%2FMVI-7866-MOV-snapshot-02-04-877.jpg&hash=5de52c7029f72a0bdbff5a857923ed79)
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on June 26, 2022, 01:04:38 PM
@GianlucaM83

160MHz overclocking is working fine here using my Sandisk Extreme PRO 32GB UHS-I U3 95MB/s.
You must restart your camera after changing SD overclocking frequency, if you didn't do that the setting won't be applied.

Let's start fresh:
-Unload all modules, load only bench.mo, restart the camera.
-Switch to Photo mode (no LiveView)
-Run cards benchmarks, share the result.
-Load sd_uhs.mo, restart the camera.
-Select 160 MHz, restart the camera.
-Run cards benchmarks, share the result.

If I do the test in video mode I get these errors: Alloc error: buffer=16777216

Turn off RAW video before running benchmarks in movie mode.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: GianlucaM83 on June 26, 2022, 01:54:09 PM
@GianlucaM83

160MHz overclocking is working fine here using my Sandisk Extreme PRO 32GB UHS-I U3 95MB/s.
You must restart your camera after changing SD overclocking frequency, if you didn't do that the setting won't be applied.

Let's start fresh:
-Unload all modules, load only bench.mo, restart the camera.
-Switch to Photo mode (no LiveView)
-Run cards benchmarks, share the result.
-Load sd_uhs.mo, restart the camera.
-Select 160 MHz, restart the camera.
-Run cards benchmarks, share the result.

Turn off RAW video before running benchmarks in movie mode.

ok. I followed your instructions. 
these are the results:

only bench.mo loaded:
Write speed 41.8 mb/s
Read speed 43.6 mb/s

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FNybGTcCD%2FIMG-Fresh-sd-off.jpg&hash=c54bc7be2fa790eb6bc9efe107ca154a) (https://postimg.cc/NybGTcCD)

sd_uhs.mo Loaded (restarted camera), 160 MHz selected (restarted camera)
Write speed 21.1 mb/s
Read speed 21.8 mb/s

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2Fv1GHcBhD%2FIMG-SD160mhz-on.jpg&hash=28fac1fe019064d3c7f2d0967c23a914) (https://postimg.cc/v1GHcBhD)

I cannot understand the reason for this result

further info I can provide
the sd is formatted in FAT32 and I did a deep format in camera before installing magiclantern

Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on June 26, 2022, 02:14:35 PM
Okay, thanks for re-testing.

In my custom 100D build I am using my 160 MHz preset which worked fine using my SD cards, let's test a1ex's 160 MHz preset with this build:
crop_rec_4k.2022Jun19.100D101_160MHz.zip (https://bitbucket.org/bilal_fakhouri/magic-lantern/downloads/crop_rec_4k.2022Jun19.100D101_160MHz.zip)

Replace all ML files with the provided build (or make a clean install, format SD card in PC then install the provided build), test 160 MHz preset and see if it will drop to 21 MB/s or not, report back please.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: GianlucaM83 on June 26, 2022, 02:52:50 PM
Thanks for your patience  :)
I did a clean install
unfortunately i get the same speed even with this version.
i think the problem is just my SD, maybe it is damaged.
i will buy a new one and come back with new information.
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FLqmCwK9r%2FIMG-7886.jpg&hash=f496ff5d28b5f9dd24ae43b5e3279dea) (https://postimg.cc/LqmCwK9r)
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: Walter Schulz on June 26, 2022, 02:57:07 PM
Maybe giving this version a try first:
https://builds.magiclantern.fm/jenkins/job/crop_rec_4k/80/artifact/platform/100D.101/magiclantern-crop_rec_4k.2018Jul22.100D101.zip
There are several sd_uhs.mo variants to try.
Just copy it to ML\modules directory.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on June 26, 2022, 03:20:53 PM
@GianlucaM83

One last test from my side, test 160 MHz again with this build:
crop_rec_4k.2022Jun19.100D101_160MHz_SD_Configuration.zip (https://bitbucket.org/bilal_fakhouri/magic-lantern/downloads/crop_rec_4k.2022Jun19.100D101_160MHz_SD_Configuration.zip)

i think the problem is just my SD, maybe it is damaged.

Maybe your 95 MB/s variant is different from mine . . or requires different overclock settings.

i will buy a new one and come back with new information.

Make sure it's Sandisk Extreme PRO 170 MB/s UHS-I U3 version which works with 192 MHz overclock.


Maybe giving this version a try first:
https://builds.magiclantern.fm/jenkins/job/crop_rec_4k/80/artifact/platform/100D.101/magiclantern-crop_rec_4k.2018Jul22.100D101.zip

Let's test that build with official sd_uhs from a1ex:
https://drive.google.com/file/d/1xWYubqqr0DceaqMhXqiIovyF_7ep5vfc/view?usp=sharing

After running a1ex's sd_uhs version, a log would be saved in ML/LOGS folder in SD card (SD_UHS.LOG), please upload it somewhere.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: GianlucaM83 on June 26, 2022, 03:35:53 PM
Ok done. this is the .LOG file
https://drive.google.com/file/d/1sP4CJKMZlBXwItMUhhkYXB1fLYEjs23E/view?usp=sharing (https://drive.google.com/file/d/1sP4CJKMZlBXwItMUhhkYXB1fLYEjs23E/view?usp=sharing)
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: GianlucaM83 on June 26, 2022, 03:46:20 PM
I used the sd_uhs from a1ex.  because i hadn't seen the first link with your build, sorry.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on June 26, 2022, 03:56:58 PM
Cool, so 160 MHz does actually work with your SD card. You need to check if it stable (doesn't drop to 21 MB/s), you can do this by recording RAW video or run Benchmarks multiple times in movie mode and photo mode.


Last thing I need you to test is:

One last test from my side, test 160 MHz again with this build:
crop_rec_4k.2022Jun19.100D101_160MHz_SD_Configuration.zip (https://bitbucket.org/bilal_fakhouri/magic-lantern/downloads/crop_rec_4k.2022Jun19.100D101_160MHz_SD_Configuration.zip)

If that worked, then I would know what causing the issue (the drop to 21 MB/s when using 160 MHz in my custom build).
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: GianlucaM83 on June 26, 2022, 04:15:29 PM
I installed the latest test build you provided. Something has changed. it looks better now

Write speed 67.3 MB/s
Read speed 71.9 MB/s

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FLJDG3BLk%2FIMG-7889.jpg&hash=76618c8b3cf6531b758f8154401877c2) (https://postimg.cc/LJDG3BLk)
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on June 26, 2022, 04:18:40 PM
Nice, please make some RAW recording tests and see if it stable (doesn't drop to 21 MB/s).
Will post more details later.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: GianlucaM83 on June 26, 2022, 04:56:09 PM
ok I'm back  :P
I recorded two raw videos. duration one minute per video, because the battery was getting low.
It's really stable, I think it could keep shooting continuously.

These are the settings

-UHD 1X3 1280x2160
12 bit Lossless, 23.973 fps
Frames 1504, Duration 1 Min, But I think it's continuous

-4K 1x3: 1x3 Binning 1360x1976
12 bit Lossless, 23.973 fps
Frames 1493, Duration 1 Min, But I think it's continuous

Bilal you are great  8)
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: GianlucaM83 on June 29, 2022, 09:01:44 AM
I had the opportunity to try the build more thoroughly.
The speed of the SD seems constant. I can record for several minutes without problems.
One problem I noticed is that, in every RAW video I have recorded, on the top of the frame only, there is some kind of intermittent noise and strange vertical stripes.
This is most visible in outdoor footage, instead in the recordings made at home the problem is very little noticeable.
This happens in both mlvapp and davinci resolve using the uncompressed dng files extracted with mlvapp.
The problem is also visible by opening the mlvs files with the FastcinemaDNG program.
I upload some MLVS test files.
Note that. The mlvs files I have uploaded contain only a few frames extracted from the original mlvs files.
I exported them from mlvapp using the MLV flatpass method. this is because the original files are too big to upload directly to google drive.
if you need other files or other tests I can do it
Thanks for the help

https://drive.google.com/drive/folders/1-COgSnAwgprsIif1iA1tzZ2iQcoonMWL?usp=sharing (https://drive.google.com/drive/folders/1-COgSnAwgprsIif1iA1tzZ2iQcoonMWL?usp=sharing)
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on June 29, 2022, 10:21:02 AM
@GianlucaM83

Can you cut MLV files even further? file sizes are still a little huge to me.
2 Seconds per clip should be enough.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: GianlucaM83 on June 29, 2022, 10:42:43 AM
Yes of course, sorry
Link to the same folder but with smaller files:
https://drive.google.com/drive/folders/1-COgSnAwgprsIif1iA1tzZ2iQcoonMWL?usp=sharing (https://drive.google.com/drive/folders/1-COgSnAwgprsIif1iA1tzZ2iQcoonMWL?usp=sharing)
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on June 29, 2022, 11:07:58 AM
Thanks for reporting, it's CMOS 7 (https://www.magiclantern.fm/forum/index.php?topic=26511.msg239303#msg239303) issue, a little tweak should fix the problem, I will release a new build soon.

Do you know if this problem happens in other presets? or when using 25 FPS with some presets?
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: GianlucaM83 on June 29, 2022, 11:21:01 AM
No problem.
unfortunately i only got to try these two presets at 23.973 fps. Maybe later or tomorrow i will be able to try the other presets at 25 fps too and I will report the results  :)
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on June 29, 2022, 11:33:06 AM
Thanks, it would be cool if you test every preset so we can fix all issues if there are ones.
I did only test presets indoor, didn't notice problems. as you said, it seems to happen (the artifacts) at certain lighting conditions.

Here is a new build, hopefully will fix artifacts for 4K/UHD 1x3 presets:
crop_rec_4k.2022Jun29.100D101.zip (https://bitbucket.org/bilal_fakhouri/magic-lantern/downloads/crop_rec_4k.2022Jun29.100D101.zip)

Please re-test mentioned presets, no need to hurry, take your time.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: GianlucaM83 on June 30, 2022, 10:58:13 AM
I did some more tests with new build. I shot some raw video under the same lights conditions as last time.
Used only UHD 1X3 and 4k 1x3 presets in both 25fps and 23,973 fps.
Unfortunately, the results are the same as last time. On the top of the frame I have that strange effect. videos shot indoors have no problem.
I noticed a few videos with corrupted frames at the beginning.
I share the link to the folder where I uploaded all the mlvs files. Inside it you will find subfolders with the name of the used presets.
Take a look when you can
If you need more tests just ask  :)
https://drive.google.com/drive/folders/145oB6eJZbft6W7xLb768njG7Il9LAoUR?usp=sharing (https://drive.google.com/drive/folders/145oB6eJZbft6W7xLb768njG7Il9LAoUR?usp=sharing)
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on July 01, 2022, 03:27:43 AM
Weird, I am unable to reproduce these artifacts on my 100D, tried with a lot of lighting conditions, there are no artifacts.

Could you share your *camera and ML settings? does these artifacts appear in LiveView too?
Are you using FPS override? or any other settings?

Is focus box centered on screen? e.g. turn off crop mode, press zoom button, make sure x1 printed on screen (by Canon overlays), press SET button to center the focus box, select UHD or 4K preset, artifacts are still there?

...videos shot indoors have no problem.

It actually does have the same problem, look closely at all indoor clips (above silver Coca Cola), in some areas the artifacts are there but they are not very visible.

I noticed a few videos with corrupted frames at the beginning.

How to reproduce the issue? :-\ .. I never had corrupted frames . .
*Your camera/ML settings please?

*For ML settings, please zip SETTINGS folder inside ML folder in SD card and upload it somewhere, but first make sure your settings do produce the artifacts.

I think I have another solution for the artifacts, used by Danne on EOS M for the same presets which is changing vertical image offset a little to the top, this way both UHD/4K 1x3 presets won't have accurate centering on sensor (I am trying to avoid this option) but at least it will clear the artifacts . . let's try this option as last resource, will share images show the current offset and the new offset if we decided to do it this way.



@100D users:
Do you have these artifacts when using 4K/UHD 1x3 presets?

It looks like this (vertical lines in the top of frame):

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FhYJZ8BL%2FArtifacts.png&hash=c27cb20d58ee9fd5c2ce649869243143)
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: GianlucaM83 on July 01, 2022, 09:20:07 AM
I am not using any particular setting.
Almost all settings are the default ones except these:
I only enabled the "focus peack" and " raw histogram" in the overlay section. In the RAW video section I activated the two options "small hack" and "more hacks lvface+aewb". Fps ovveride is not active.
In the canon menu I set the video format to PAL and the recording format to 1920x1080 25 fps.
I have tried disabling crop mode, putting the focus rectangle in the center in 1x mode and re-enabled crop mode but the problem remains.
I noticed that these artifacts are also visible in the preview on the live view, before I start recording and while I am recording.
These are the modules I have enabled:
crop_rec - dual_iso - mlv_lite - mlv_play - mlv_snd - sd_uhs.

This is the configuration file:
https://drive.google.com/file/d/1H-n_Q9_rn-ioQV_99a5FQO4MHXrV3TyP/view?usp=sharing (https://drive.google.com/file/d/1H-n_Q9_rn-ioQV_99a5FQO4MHXrV3TyP/view?usp=sharing)

Thanks
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: Danne on July 01, 2022, 12:21:27 PM
Disable histogram.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: GianlucaM83 on July 01, 2022, 07:06:32 PM
@Danne
I did it but nothing changes. I also disabled everything in the overlay section. I also tried a fresh installation but the artifacts remain.
I leave a short clip with the operation I usually do to record raw video. to see if I missed something.


One more thing, I tried the 1920x1280p 1:1 preset it seems there are no artifacts with it.
https://drive.google.com/file/d/1YUVXB5CODWrKc8VtWEDrEE326ojRB6hn/view?usp=sharing (https://drive.google.com/file/d/1YUVXB5CODWrKc8VtWEDrEE326ojRB6hn/view?usp=sharing)


I personally think it would be helpful if some other users would also provide their feedback on this.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on July 01, 2022, 11:17:31 PM
Thanks for the video, I will provide some more tests to run later, but for now please try:

-Set White Balance to specific value instead of Auto.
-Switch to Photo mode, go to ML settings, Expo tab and turn on ExpSim.
-Restart camera.

Now try UHD or 4K preset, artifacts are still there?
If yes, I am going to provide another set of tests.

-Does 4.5K and 4.3K 1x3 presets work without these artifacts? try 23.976 and 25 FPS.
-If you want to upload MLV clips, please cut them to 5 frames instead of two seconds, that would be enough too . .

I started to suspect that it could be your camera sensor is slightly different from mine (slight hardware differences?), I could be wrong of course . . BTW, where was your 100D manufactured? (you can read this on sticker in the bottom of camera)
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: GianlucaM83 on July 02, 2022, 08:17:57 AM
Okay I did this:
- I switched to photo mode.
- I turned on expsim mode in the expo tab.
-I set the white balance to a personal value (5200k).
-restarted the camera
- I recorded some raw video in all 1x3 modes, both 24.973 fps and 25 fps
Unfortunately, artifacts are visible in all modes  :(
I put the link to mlvs file, this time 5 frames per clip.
https://drive.google.com/drive/folders/1IQ1bpWKOI9T0aRATwlj_1tfRJxkmpViS?usp=sharing (https://drive.google.com/drive/folders/1IQ1bpWKOI9T0aRATwlj_1tfRJxkmpViS?usp=sharing)

Regarding the possibility that our cameras have a slightly different sensor, perhaps that could be true, but I don't know much about it.
On the bottom of my camera is a little label that says: Made in TAIWAN, and the serial number.
I don't know if this might be relevant, maybe it doesn't make any difference but, I think it is a camera for the European market. i think this because some time ago, I sent the camera to a service center for auto focus calibration of the kit lens.
The repair was under warranty and the camera was sent by Canon to a lab in Germany.

Edit: I did some more tests at home with low artificial light, the artifacts are almost not visible  :o
I think this is going to drive me crazy :P
https://drive.google.com/drive/folders/1_acY0asl8sB-pPAq7ju74X2mASjPJP71?usp=sharing (https://drive.google.com/drive/folders/1_acY0asl8sB-pPAq7ju74X2mASjPJP71?usp=sharing)
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: Danne on July 02, 2022, 12:56:32 PM
Could you test this version?
https://bitbucket.org/Dannephoto/magic-lantern_dannephoto_git/downloads/crop_rec_4k_mlv_snd_isogain_1x3_presets_100D_2022Jul02.100D101.zip

Doesn´t include bilals later stuff but you could try if sensor behaves good in general.

Source:
https://bitbucket.org/Dannephoto/magic-lantern_dannephoto_git/src/master/

Go to this branch:
crop_rec_4k_mlv_snd_isogain_1x3_presets_100D
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: GianlucaM83 on July 02, 2022, 04:19:11 PM
Hi Danne
I installed your build.
I did some testing, at home for now.
But I need help to understand which presets I should use and how I should set them. Because the camera crashed a few times during recording and in the clips I made there are many corrupted frames.
I think it's my fault, I got used to bilial's build which had slightly different settings.
link with the various crash logs:
https://drive.google.com/drive/folders/1flNTGxPTwwTybtRWXZYDTQ5kQqRcHn1U?usp=sharing (https://drive.google.com/drive/folders/1flNTGxPTwwTybtRWXZYDTQ5kQqRcHn1U?usp=sharing)

However, I was able to record some clips.
With your build, in these lighting conditions there seems to be no artifacts on the top of the frame, but I think it is better to do some more tests in other lighting conditions as well, and set more appropriate settings to avoid the corrupted frames and crashes
Some MLVs test files:
https://drive.google.com/drive/folders/12Y-iMQ1-m7xHrJPqAFiZaQN4dLbHsJG9?usp=sharing (https://drive.google.com/drive/folders/12Y-iMQ1-m7xHrJPqAFiZaQN4dLbHsJG9?usp=sharing)

Thanks guys
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: Danne on July 02, 2022, 04:37:46 PM
Did you test regular 14bit vs 12 or 10bit in Bilals build? Looks a little like gain registry issue but mostly guessing here.
Would be nice to have access to bilals code online for faster checking.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on July 02, 2022, 04:59:54 PM
@GianlucaM83

Thanks for testing, I will provide more things to test (later today or tomorrow).
Hardware differences was a guess, could be wrong, my 100D made in Taiwan too. I don't think it's something related to auto focus calibration.

@Danne
My 100D doesn't have any artifacts and my tests was done with the same build and settings used by GianlucaM83.
I don't think it's Analog gain problem, well, @GianlucaM83 could redo your tests with in 14-bit lossless only:

-Set 14-bit lossless from RAW video submenu
-Turn off bit-depth option from "Crop mode"
-Restart camera
-Test UHD and 4K 1x3 presets

-Artifacts are still there?

Also, could you test 3K and 1440p 1:1 presets?
So far the issue seems affect only 1x3 Binning modes and only GianlucaM83's 100D.

More things are coming from my side soon, currently a little busy.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: Danne on July 02, 2022, 05:07:07 PM
I'll grab my 100 soon and join the testing squad  8)
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: GianlucaM83 on July 03, 2022, 10:30:14 AM
-I tested UHD 1X3 and 4k 1x3 presets in 14 bit lossless. artifacts still appear  :(
-I tested the 3k and 1440p 1:1 presets. There are no visible artifacts. At least I can't see them. maybe the 1:1 mode don't give problems on my camera.  :)

Obviously to get a good recording time I had to reduce the bits to 10, use 23.976 fps and lower the resolution.
As usual I put some sample mlv files made with both presets at different resolutions and bit depht. take a look when you can.
Thanks

1440p 1:1 preset:
https://drive.google.com/drive/folders/1kgkTOUrFuPfsMWtJ3_cgCHkFS2CAspjh?usp=sharing (https://drive.google.com/drive/folders/1kgkTOUrFuPfsMWtJ3_cgCHkFS2CAspjh?usp=sharing)

3k 1:1 preset:
https://drive.google.com/drive/folders/1_pfTlm_dCFb3NkO-dw6yzbkbsJ3USpaE?usp=sharing (https://drive.google.com/drive/folders/1_pfTlm_dCFb3NkO-dw6yzbkbsJ3USpaE?usp=sharing)

Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: Danne on July 03, 2022, 10:51:39 AM
Tested all 1x3 presets. Hats off to you bilal and all 100D owners. THose presets you created are perfectly working both 24 and 25fps. Seems very stable too. Not like my build ;).
I keep nagging about code but you should keep a fresh codebase along with your builds. Equally imprtant as posting builds.
Keep up the good work!
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: Danne on July 03, 2022, 11:16:58 AM
By the way. I think I know how to fix GianlucaM83 issue. Had a similar issue with 5d3. I need the crop_rec.c code to check.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: GianlucaM83 on July 03, 2022, 11:41:58 AM
Guys your work is amazing  8)
I am very happy that the build works well on your cameras and for all the users who have this great camera  :)

Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on July 03, 2022, 03:56:52 PM
Thanks @Danne :)

I keep nagging about code but you should keep a fresh codebase along with your builds. Equally imprtant as posting builds.

Yeah, you are right, currently I am preparing for a Github solution (will re-write 650D/700D code then apply the new changes maybe in separate branch for 100D), that would fix this problem.

By the way. I think I know how to fix GianlucaM83 issue. Had a similar issue with 5d3. I need the crop_rec.c code to check.

Here is the new fresh code:
magic-lantern-bilal-100d (29-6-2022).7z (https://drive.google.com/file/d/1dLCSrN5YXb0jBukYW3tE9jwkOxuMhSY1/view?usp=sharing)

I am aware of the fix could be changing the offset on sensor a little to the top, but I am trying to avoid this (https://www.magiclantern.fm/forum/index.php?topic=26511.msg239495#msg239495) currently.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: Danne on July 03, 2022, 04:13:24 PM
Not offset, other regs.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on July 03, 2022, 04:17:43 PM
Okay, waiting your Magic :)

@GianlucaM83

I am looking into 1440p/3K right now, thanks for the tests again. Also I am preparing new tests as backup plan.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: Danne on July 03, 2022, 05:41:06 PM
Please try this build @GianlucaM83
https://bitbucket.org/Dannephoto/magic-lantern_dannephoto_git/downloads/crop_rec_4k_2022Jul03.100D101.zip

I changed registry to this in crop_rec code:
Code: [Select]
        /* PowerSaveTiming registers */
        /* after readout is finished, we can turn off the sensor until the next frame */
        /* we could also set these to 0; it will work, but the sensor will run a bit hotter */
        /* to be tested to find out exactly how much */
        adtg_new[4]  = (struct adtg_new) {6, 0x8172, nrzi_encode(readout_end + 1) }; /* PowerSaveTiming ON (6D/700D) */
        adtg_new[5]  = (struct adtg_new) {6, 0x8178, nrzi_encode(readout_end + 1) }; /* PowerSaveTiming ON (5D3/6D/700D) */
        adtg_new[6]  = (struct adtg_new) {6, 0x8196, nrzi_encode(readout_end + 1) }; /* PowerSaveTiming ON (5D3) */
        adtg_new[7]  = (struct adtg_new) {6, 0x8173, nrzi_encode(fps_timer_b - 5) }; /* PowerSaveTiming OFF (6D/700D) */
        adtg_new[8]  = (struct adtg_new) {6, 0x8179, nrzi_encode(fps_timer_b - 5) }; /* PowerSaveTiming OFF (5D3/6D/700D) */
        adtg_new[9]  = (struct adtg_new) {6, 0x8197, nrzi_encode(fps_timer_b - 5) }; /* PowerSaveTiming OFF (5D3) */
        adtg_new[10] = (struct adtg_new) {6, 0x82B6, nrzi_encode(readout_end - 1) }; /* PowerSaveTiming ON? (700D); 2 units below the "ON" timing from above */
       
        /* ReadOutTiming registers */
        /* these shouldn't be 0, as they affect the image */
        adtg_new[11] = (struct adtg_new) {6, 0x82F8, nrzi_encode(readout_end + 1) }; /* ReadOutTiming */
        adtg_new[12] = (struct adtg_new) {6, 0x82F9, nrzi_encode(fps_timer_b - 1) }; /* ReadOutTiming end? */


Sidenotes. I see you work from  crop_rec_4k branch. I would prefer working from crop_rec_4k_mlv_snd as it has g3gg0 mlv_snd fixes. Anyway. Please test and report back. Very short of time over here unfortunately.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on July 03, 2022, 05:49:41 PM
Sidenotes. I see you work from  crop_rec_4k branch. I would prefer working from crop_rec_4k_mlv_snd as it has g3gg0 mlv_snd fixes.

I have added them to crop_rec_4k branch a long time ago (look at previous commits).


I will check your fix and play with it on my 100D.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: GianlucaM83 on July 03, 2022, 08:28:48 PM
Thanks Danne
I downloaded the build and repeated the tests.
I noticed that there are some new modules in this build.
I only loaded the ones I had already loaded in the previous builds.
Unfortunately the artifacts are still there in the 1x3 modes.
One thing I noticed but maybe it is not important. In all the builds I have tried.
When I activate the crop mode in the video mode. On the liveview of the camera I see the same artifacts that are in the recorded video.
 While recording a raw video I see at the top of the live view the same colored, flashing bands that I see later in the video I recorded.
Currently the preview is set to realtime. if I try to change it to another option it resets itself when I exit the magic lantern menu.
i don't know if i explained myself correctly. my English sucks.

Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: Danne on July 03, 2022, 08:55:05 PM
Hm, I think it's related to powertime registry but hard to know for sure. Is it always happening or only sometimes?
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: SebastianC on July 03, 2022, 08:57:49 PM
Thank you Danne! I will try new !
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: GianlucaM83 on July 03, 2022, 09:04:58 PM
Danne.
It happens every time and only when i enter the 1x3 crop mode in video mode.
if i activate the 1440p 1:1 preset there are no visible artifacts on the liveview or even in the video i recorded. maybe it is just a coincidence
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on July 03, 2022, 09:29:51 PM
I noticed that there are some new modules in this build.
I only loaded the ones I had already loaded in the previous builds.

The new modules are there in repo, it either not supported yet or it's for development stuff, don't use them.
I always delete them before uploading a new build for 100D, it's okay for now because we are providing testing build, and these builds won't be used after we solve the problem.

One thing I noticed but maybe it is not important. In all the builds I have tried.
When I activate the crop mode in the video mode. On the liveview of the camera I see the same artifacts that are in the recorded video.
While recording a raw video I see at the top of the live view the same colored, flashing bands that I see later in the video I recorded.

Seems normal, these type of artifacts should be shown in both LiveView and recorded RAW data. (why you have these artifacts on your 100D isn't normal)

Currently the preview is set to realtime. if I try to change it to another option it resets itself when I exit the magic lantern menu.

Yeah, for presets which have correct real-time preview there is no need for Framing preview, I made it like that.
Probably I should make an option for the user too, to let him select which preview he want to use.


Thank you Danne! I will try new !

Do you have the same artifacts  (https://www.magiclantern.fm/forum/index.php?topic=26511.msg239495#msg239495)which GianlucaM83 has?
If no, there is nothing new to test here.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: GianlucaM83 on July 03, 2022, 09:43:05 PM
just another little test I don't know if it can make sense.
maybe it's the summer heat playing tricks on me  :o

-UHD 1X3 preset, 12 bit, 23,976 fps.

400 ISO, artifacts are visible
https://drive.google.com/file/d/1P8o22RTubL9hY1vmsD2z7yQ7BGjsephS/view?usp=sharing (https://drive.google.com/file/d/1P8o22RTubL9hY1vmsD2z7yQ7BGjsephS/view?usp=sharing)

800 iso, artifacts are not visible
https://drive.google.com/file/d/16Q3pZ2neZvQb5dPC3Xa9_S-RVlQL9dCu/view?usp=sharing (https://drive.google.com/file/d/16Q3pZ2neZvQb5dPC3Xa9_S-RVlQL9dCu/view?usp=sharing)
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on July 03, 2022, 09:58:43 PM
@GianlucaM83 let's try these:

First, backup your camera ROM files, you will find them in ML/LOGS folder, copy them to a safe place (also let me have them, zip them then upload it somewhere and send me the link via personal message), please Don't share the link publicly, it's copyrighted material.

Tests:

1- Let me take a look into your 100D registers:

-Download this adtg_gui.mo (https://drive.google.com/file/d/1QfBQkORaKQ-cT9-Y4QOm-mpsHhecFOmm/view?usp=sharing), put it in ML/modules folder in SD card
-Unload all modules, and load only adtg_gui.mo, restart camera
-Set video mode to 1080p24 to gain extra speed

-Follow the following video, you can see in the video I am entering Canon menu and toggling between x1/x5/x10 modes, you should do that too to let "ADTG registers" bring the registers into it:

-After you finish, you will find "adtg.log" file in ML/LOGS folder in SD card, upload it somewhere and share it.

2- Try this build:
crop_rec_4k.2022Jul03.100D101_delay.zip (https://bitbucket.org/bilal_fakhouri/magic-lantern/downloads/crop_rec_4k.2022Jul03.100D101_delay.zip)

-Just install the build and try UHD or 4K 1x3 presets and feedback please.
-In this build I made the delay longer before refreshing LiveView from 100 MS to 500 MS (MS = Milliseconds), I need to make sure if it's not a race condition.

3- Update your camera firmware to this one:
https://pel.hu/down/v101-sl1-100d-x7-win.zip

-Format your SD card in PC (remove ML)
-Put CCF16101.FIR in SD card
-Make sure battery is fully charged
-Run firmware update process

Edit:
Note: I don't know if this would affect or reset autofocus calibration data (https://www.magiclantern.fm/forum/index.php?topic=26511.msg239517#msg239517) for your kit lens!!

-Then install ML build and see if UHD or 4K 1x3 presets still have the artifacts or not.


4- I will prepare a build for manual CMOS 7 fine-tuning, will post it with more details later.

Please make these tests in order.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on July 03, 2022, 11:58:37 PM
Artifacts were reproduced on my 100D :) :

Thanks to ADTG registers LOG provided by @GianlucaM83, I noticed 5 ADTG registers which have different values on GianlucaM83's 100D, which are:

ADTG2[c00d]:    0x5249
ADTG2[c00e]:    0x1
ADTG2[c00f]:     0x1
ADTG2[c010]:    0x2
ADTG2[c011]:    0x2

-On my 100D it's:

ADTG2[c00d]:    0x5000
ADTG2[c00e]:    0x53
ADTG2[c00f]:     0x52
ADTG2[c010]:     0x52
ADTG2[c011]:     0x52

I copied GianlucaM83's 100D values to my 100D, then the artifacts appeared in 1x3 presets.
These registers are gain registers.

-Let's try my 100D values on GianlucaM83's 100D using this build:
crop_rec_4k.2022Jul04.100D101_ADTG.zip (https://bitbucket.org/bilal_fakhouri/magic-lantern/downloads/crop_rec_4k.2022Jul04.100D101_ADTG.zip)

@GianlucaM83
Please try provided build and 1x3 presets then feedback please.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: GianlucaM83 on July 04, 2022, 12:30:26 AM
With this latest build the artifacts seem to be gone.
I will try tomorrow in other lighting conditions. But I think this is the right way.
Thanks Bilal and also to the others who helped me  :)
You guys are great  8)
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: GianlucaM83 on July 04, 2022, 08:32:38 AM
Today I tested the new build:"crop_rec_4k.2022Jul04.100D101_ADTG".
In the same lighting conditions where I had artifacts before.

-Tested all 1x3 presets. 12 bit, 23.976 fps, 25 fps.
The artifacts are no longer visible  :)
I put the link with the test MLV files:
https://drive.google.com/drive/folders/19E3IRpgkBfjoIsFjB7MfDkf6fDS4xKcO?usp=sharing (https://drive.google.com/drive/folders/19E3IRpgkBfjoIsFjB7MfDkf6fDS4xKcO?usp=sharing)

Bilal you did it  8)
Thanks!
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: DeafEyeJedi on July 08, 2022, 12:44:56 AM
Wonderful progress guys!
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on September 10, 2022, 03:53:18 AM
-What's not working:

SD Overclocking:
During my tests 192 MHz SD overclock will only work with Sandisk Extreme PRO UHS-I U3 170 MB/s version, and it seems stable.

240 MHz doesn't work even when using Sandisk Extreme PRO UHS-I U3 170 MB/s.
Both 192 MHz and 240 MHz don't work when using Sandisk Extreme PRO UHS-I U3 95MB/s.

New sd_uhs update:
https://www.magiclantern.fm/forum/index.php?topic=26634.msg240128#msg240128

State:
240 MHz does work to some point (https://www.magiclantern.fm/forum/index.php?topic=26634.msg240127#msg240127) using Sandisk Extreme PRO UHS-I U3 170 MB/s.
192 MHz now works on my Sandisk Extreme PRO UHS-I U3 95MB/s.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: GianlucaM83 on September 10, 2022, 09:44:08 AM
First of all, thank you Bilal and everyone who contributed to make this new module free for everyone  :)
I did a bench with my new sd card: Sandisk Extreme Pro 170 MB/s U3.
Seems that it can handle the 192 MHz preset correctly, instead with the 240 MHz preset it drops to 20MB/s.
Before this update, the same SD only worked with the 168 MHz preset. So this seems like a huge improvement to me.
Later I will also try it with my old  Sandisk Extreme PRO UHS-I U3 95MB/s
Thanks again!

Sandisk Extreme Pro 170 MB/s U3: New sd_hs module.192MHz preset:
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2F7L0gJfB7%2Fbench192mhz.jpg&hash=af9541c345954119d87f97999dbb7f00)

Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on September 10, 2022, 09:57:42 AM
I did a bench with my new sd card: Sandisk Extreme Pro 170 MB/s U3.
Seems that it can handle the 192 MHz preset correctly, instead with the 240 MHz preset it drops to 20MB/s.
Before this update, the same SD only worked with the 168 MHz preset. So this seems like a huge improvement to me.

Cool, what is your card capacity for Sandisk Extreme Pro 170 MB/s U3?
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: GianlucaM83 on September 10, 2022, 10:12:46 AM
I bought the 64 GB version
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: SebastianC on September 10, 2022, 04:21:28 PM
My 100D 3K can record more than 20-30seconds now with new 240 MHz.

Before I just only can record less than 5-10 seconds.

Good job Bilal!
Thank you for your work!
 :)
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on September 15, 2022, 10:51:41 AM
Another better 240 MHz for 100D:
https://www.magiclantern.fm/forum/index.php?topic=26634.msg240328#msg240328

-It works better at high temps using my Sandisk 170 MB/s and 100D, it didn't drop to 21 MB/s after 5 minuets of recording :D
-SDR104 patch which required to enable high OC for some cards now works with the newest 240 MHz preset, and it's stable
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: GianlucaM83 on September 16, 2022, 05:27:23 AM
I did a bench with the newest_sd_uhs_SDR104_patchmodule.
The 240 MHz preset now works on my SD as well  8)
Thanks!

This is the result:
240 MHz Preset
Sandisk Extreme Pro 170 MB/s U3 64GB

Write speed: 83.7 MB/s
Read speed:  85.7 MB/s
Write speed: 83.5 MB/s
Read speed:  85.7 MB/s

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2Fd1sJX8sJ%2Fbench-240-mhz-newestmodule-16-09-2022.jpg&hash=ad8492dc37b39c315a84a0405b454512)
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: Walter Schulz on September 16, 2022, 06:27:07 AM
@Bilal: 10 MByte/s less than rest of the bunch? Did you get simiiar results?
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on September 16, 2022, 08:59:59 AM
@GianlucaM83 Awesome, thanks for the test!

@Walter
Yeah it seems a little slow at 240 MHz (but still higher than 192 MHz (https://www.magiclantern.fm/forum/index.php?topic=26511.msg240231#msg240231)), for me I can get up to ~90 MB/s using my 170 MB/s SD card on 100D (same on 700D, my SD card limit is ~90 MB/s).

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.ibb.co%2FVSgLgKT%2Fbench0.png&hash=b065a15540d5e8d7405022f582709dc5) (https://ibb.co/7WnSn9K)


@GianlucaM83
Your SD card should perform around ~90 MB/s (or a little higher) with 240 MHz. I suggest to run benchmarks with fast card reader on PC.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: GianlucaM83 on September 16, 2022, 10:05:54 AM
I did a beanchmark of my SD card on my PC as you suggested. I used My Kingston FCR-HS4 card reader which is usb 3.0, connected directly to a usb 3.0 port on the motherboard.
I used the CrystalDiskMark program with the default settings and the aja system software. Unfortunately, I do not know any other softwares.
These are the results:
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2Fs2f9wgBb%2FIMG-20220916-084432.jpg&hash=4546e493fa8896b6f5e347f76e9c12dd) (https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FwvyNGy94%2Fbench-PC.jpg&hash=01d45ca9f93589b09e9315e0fc5bd138)
what do you think about these results, do you think they are in the normal range for my SD?
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: Walter Schulz on September 16, 2022, 10:28:36 AM
Yes, they are.
Your Kingston FCR-HS4 (as mine) does not know anything about overclocking and runs pure UHS-I protocols as specified by SD Card Association. Means: UHS-104 is the limit.
You can compare your numbers with https://www.cameramemoryspeed.com/reviews/sd-cards/sandisk-extreme-pro-170mbs-uhs-i-u3-v30-64gb-sdxc-memory-card/
In this test all cardreaders but one are running non-proprietary transfer modes. The only one using overclocking stands out.

Maybe Bilal can get behind it.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: GianlucaM83 on September 16, 2022, 10:35:04 AM
Yes, they are.
Your Kingston FCR-HS4 (as mine) does not know anything about overclocking and runs pure UHS-I protocols as specified by SD Card Association. Means: UHS-104 is the limit.
You can compare your numbers with https://www.cameramemoryspeed.com/reviews/sd-cards/sandisk-extreme-pro-170mbs-uhs-i-u3-v30-64gb-sdxc-memory-card/
In this test all cardreaders but one are running non-proprietary transfer modes. The only one using overclocking stands out.
Thanks Walter Schulz for the explanation  :)
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on September 16, 2022, 12:09:30 PM
Thanks for the benchmarks.

I don't always trust CrystalDiskMark benchmarks (at least for fake SD cards), it wasn't really accurate (https://www.magiclantern.fm/forum/index.php?topic=24315.msg218540#msg218540) when I bought fake (https://www.magiclantern.fm/forum/index.php?topic=24315.msg218496#msg218496) Sandisk Extreme PRO 170 MB/s SD cards.
The other software (AJA) benchmark seems closer in term of write speed benchmarks compared to camera benchmark. Of course ignore read benchmarks (because we are using 192 MHz for reads on 100D when using 240 MHz).

Could you also test with Black Magic Disk Speed Test? Here is a download link (https://drive.google.com/file/d/1OsKTt-GJdGb0ngPBURmZdXlP2nncTtug/view?usp=sharing) from my Google Drive (I don't know where is the official download link)
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: GianlucaM83 on September 16, 2022, 02:07:04 PM
I did a test with Disk Speed Test.
Write speed is 85.5 MB/s
(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fi.postimg.cc%2FPJMGDMPB%2FDisk-Speed-Test.png&hash=3cdcd005d4cbe1b6e405baafeb26dc8c)
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: Skinny on September 16, 2022, 08:35:02 PM
@theBilalFakhouri congrats with new achievements, cool stuff guys! Do 300mhz, why stop at 280 :D
@GianlucaM83 nice virus ti :) I can see a lot of musicians here on ML forum for some reason :o
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: Walter Schulz on September 16, 2022, 08:41:34 PM
Do 300mhz, why stop at 280 :D

Because next stop is not 300 but 384 and that is a tough call.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on September 17, 2022, 01:59:04 AM
@GianlucaM83
Thanks for the benchmarks, your SD card is doing on average (88 + 82 + 85.5) / 3 = ~85 MB/s which is close to 83.7 MB/s from camera.
If we exclude CrystalDiskMark, (82 + 85.5) / 2 = 83.7 MB/s. Well, I don't think this is an accurate way to do measurements, also because SD card frequency is unknown using PC card reader.

Last suggested test:
Perform low level format in camera, do benchmarks in Play mode at 240 MHz.

But anyway, in all cases 240 MHz is working fine here, the slower write speed in 240 MHz could be from your SD card or your 100D.
The only accurate way to verify is to test another Sandisk 170 MB/s SD card on your 100D and test your current Sandisk 170 MB/s SD card on other DIGIC 5 model at 240 MHz.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on September 17, 2022, 02:05:52 AM
@Skinny

Thank you!

Yeah as Walter said, other frequencies above 240 MHz are 280 MHz and 384 MHz. Will double check later if we can select something among 280-384 MHz range.
We are so lucky that our Canon cameras (DIGIC 5 models) from ~10 years ago support unofficial UHS-I speeds! Impressive work Canon ;D

Well, I must get 280 MHz stable (if possible) then think about higher than 280 MHz frequencies possibilities.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: theBilalFakhouri on September 17, 2022, 02:27:31 AM
Next plans for 100D port:

At current stage I think I got most things from my 650D / 700D port to 100D, main things were:

-Crop mode (crop_rec) presets
-New hacks for RAW video
-And recently 240 MHz SD overclocking
-Memory allocation fix for high resolutions
-And other things

I need to check HDMI state too for some crop mode presets.

Remaining thing is Crop mode V2 (crop_new) module, more likely I will not port it. on 700D it's still confuse users, also the module isn't 100% stable (causes corrupted frames at some settings).
My alternative idea for this is to make more stable presets for Crop mode (crop_rec). Will discuss later what new presets to include in crop_rec, it will be something like more Time-Lapse modes and high frame rate modes.

Well, I might give crop_new a try on 100D.

100D users: feel free to give suggestions, your opinions and to discuss. please don't ask for new features, only if it related to my 650D / 700D port.
Title: Re: Bilal's crop_rec_4k experiments for 100D / SL1
Post by: thatguyjay on November 17, 2022, 12:08:48 AM
Hey guys, I'm a noob poking into this subject matter. I had an SL1 in my closet and just heard about all this RAW shooting capability added. Thank you for all your work on this. I can't seem to get my settings to work optimally and I thought I could use some advice.

I installed the latest build I could find: crop_rec_4k.2022Jul04.100D101_ADTG

I'm using a 128GB SD card like the one in the pic above: Sandisk Extreme Pro

I was only able to find one setting that worked to record more than 5 seconds before buffering/crashing the camera or showing pink stripes in LiveView: 1280x720 10-bit.

I also tried turning on crop mode but that seemed to be an additional X factor.

I guess there are some modes that work and some that don't and it has to do with the speed of the SD card? Any advice on recommended settings that will get me to at least UHD would be very helpful.

I'm trying to get to at least 1920x1080 to use this as a B camera with some interviews IDK if I'll have an answer by then, but I'm shooting Monday. I would be happy to support development on the SL1 if it would help get us over the finish line for this camera's capabilities.