Bilal's crop_rec_4k experiments for 100D / SL1

[GianlucaM83]

Okay I did this:
- I switched to photo mode.
- I turned on expsim mode in the expo tab.
-I set the white balance to a personal value (5200k).
-restarted the camera
- I recorded some raw video in all 1x3 modes, both 24.973 fps and 25 fps
Unfortunately, artifacts are visible in all modes 
I put the link to mlvs file, this time 5 frames per clip.
https://drive.google.com/drive/folders/1IQ1bpWKOI9T0aRATwlj_1tfRJxkmpViS?usp=sharing

Regarding the possibility that our cameras have a slightly different sensor, perhaps that could be true, but I don't know much about it.
On the bottom of my camera is a little label that says: Made in TAIWAN, and the serial number.
I don't know if this might be relevant, maybe it doesn't make any difference but, I think it is a camera for the European market. i think this because some time ago, I sent the camera to a service center for auto focus calibration of the kit lens.
The repair was under warranty and the camera was sent by Canon to a lab in Germany.

Edit: I did some more tests at home with low artificial light, the artifacts are almost not visible 
I think this is going to drive me crazy
https://drive.google.com/drive/folders/1_acY0asl8sB-pPAq7ju74X2mASjPJP71?usp=sharing

[Danne]

Could you test this version?
https://bitbucket.org/Dannephoto/magic-lantern_dannephoto_git/downloads/crop_rec_4k_mlv_snd_isogain_1x3_presets_100D_2022Jul02.100D101.zip

Doesn´t include bilals later stuff but you could try if sensor behaves good in general.

Source:
https://bitbucket.org/Dannephoto/magic-lantern_dannephoto_git/src/master/

Go to this branch:
crop_rec_4k_mlv_snd_isogain_1x3_presets_100D

[GianlucaM83]

Hi Danne
I installed your build.
I did some testing, at home for now.
But I need help to understand which presets I should use and how I should set them. Because the camera crashed a few times during recording and in the clips I made there are many corrupted frames.
I think it's my fault, I got used to bilial's build which had slightly different settings.
link with the various crash logs:
https://drive.google.com/drive/folders/1flNTGxPTwwTybtRWXZYDTQ5kQqRcHn1U?usp=sharing

However, I was able to record some clips.
With your build, in these lighting conditions there seems to be no artifacts on the top of the frame, but I think it is better to do some more tests in other lighting conditions as well, and set more appropriate settings to avoid the corrupted frames and crashes
Some MLVs test files:
https://drive.google.com/drive/folders/12Y-iMQ1-m7xHrJPqAFiZaQN4dLbHsJG9?usp=sharing

Thanks guys

[Danne]

Did you test regular 14bit vs 12 or 10bit in Bilals build? Looks a little like gain registry issue but mostly guessing here.
Would be nice to have access to bilals code online for faster checking.

[theBilalFakhouri]

@GianlucaM83

Thanks for testing, I will provide more things to test (later today or tomorrow).
Hardware differences was a guess, could be wrong, my 100D made in Taiwan too. I don't think it's something related to auto focus calibration.

@Danne
My 100D doesn't have any artifacts and my tests was done with the same build and settings used by GianlucaM83.
I don't think it's Analog gain problem, well, @GianlucaM83 could redo your tests with in 14-bit lossless only:

-Set 14-bit lossless from RAW video submenu
-Turn off bit-depth option from "Crop mode"
-Restart camera
-Test UHD and 4K 1x3 presets

-Artifacts are still there?

Also, could you test 3K and 1440p 1:1 presets?
So far the issue seems affect only 1x3 Binning modes and only GianlucaM83's 100D.

More things are coming from my side soon, currently a little busy.

[Danne]

I'll grab my 100 soon and join the testing squad 

[GianlucaM83]

-I tested UHD 1X3 and 4k 1x3 presets in 14 bit lossless. artifacts still appear 
-I tested the 3k and 1440p 1:1 presets. There are no visible artifacts. At least I can't see them. maybe the 1:1 mode don't give problems on my camera. 

Obviously to get a good recording time I had to reduce the bits to 10, use 23.976 fps and lower the resolution.
As usual I put some sample mlv files made with both presets at different resolutions and bit depht. take a look when you can.
Thanks

1440p 1:1 preset:
https://drive.google.com/drive/folders/1kgkTOUrFuPfsMWtJ3_cgCHkFS2CAspjh?usp=sharing

3k 1:1 preset:
https://drive.google.com/drive/folders/1_pfTlm_dCFb3NkO-dw6yzbkbsJ3USpaE?usp=sharing

[Danne]

Tested all 1x3 presets. Hats off to you bilal and all 100D owners. THose presets you created are perfectly working both 24 and 25fps. Seems very stable too. Not like my build .
I keep nagging about code but you should keep a fresh codebase along with your builds. Equally imprtant as posting builds.
Keep up the good work!

[Danne]

By the way. I think I know how to fix GianlucaM83 issue. Had a similar issue with 5d3. I need the crop_rec.c code to check.

[GianlucaM83]

Guys your work is amazing 
I am very happy that the build works well on your cameras and for all the users who have this great camera 

[theBilalFakhouri]

Thanks @Danne

I keep nagging about code but you should keep a fresh codebase along with your builds. Equally imprtant as posting builds.

Yeah, you are right, currently I am preparing for a Github solution (will re-write 650D/700D code then apply the new changes maybe in separate branch for 100D), that would fix this problem.

By the way. I think I know how to fix GianlucaM83 issue. Had a similar issue with 5d3. I need the crop_rec.c code to check.

Here is the new fresh code:
magic-lantern-bilal-100d (29-6-2022).7z

I am aware of the fix could be changing the offset on sensor a little to the top, but I am trying to avoid this currently.

[Danne]

Not offset, other regs.

[theBilalFakhouri]

Okay, waiting your Magic

@GianlucaM83

I am looking into 1440p/3K right now, thanks for the tests again. Also I am preparing new tests as backup plan.

[Danne]

Please try this build @GianlucaM83
https://bitbucket.org/Dannephoto/magic-lantern_dannephoto_git/downloads/crop_rec_4k_2022Jul03.100D101.zip

I changed registry to this in crop_rec code:

Code: [Select]

        /* PowerSaveTiming registers */
        /* after readout is finished, we can turn off the sensor until the next frame */
        /* we could also set these to 0; it will work, but the sensor will run a bit hotter */
        /* to be tested to find out exactly how much */
        adtg_new[4]  = (struct adtg_new) {6, 0x8172, nrzi_encode(readout_end + 1) }; /* PowerSaveTiming ON (6D/700D) */
        adtg_new[5]  = (struct adtg_new) {6, 0x8178, nrzi_encode(readout_end + 1) }; /* PowerSaveTiming ON (5D3/6D/700D) */
        adtg_new[6]  = (struct adtg_new) {6, 0x8196, nrzi_encode(readout_end + 1) }; /* PowerSaveTiming ON (5D3) */
        adtg_new[7]  = (struct adtg_new) {6, 0x8173, nrzi_encode(fps_timer_b - 5) }; /* PowerSaveTiming OFF (6D/700D) */
        adtg_new[8]  = (struct adtg_new) {6, 0x8179, nrzi_encode(fps_timer_b - 5) }; /* PowerSaveTiming OFF (5D3/6D/700D) */
        adtg_new[9]  = (struct adtg_new) {6, 0x8197, nrzi_encode(fps_timer_b - 5) }; /* PowerSaveTiming OFF (5D3) */
        adtg_new[10] = (struct adtg_new) {6, 0x82B6, nrzi_encode(readout_end - 1) }; /* PowerSaveTiming ON? (700D); 2 units below the "ON" timing from above */
       
        /* ReadOutTiming registers */
        /* these shouldn't be 0, as they affect the image */
        adtg_new[11] = (struct adtg_new) {6, 0x82F8, nrzi_encode(readout_end + 1) }; /* ReadOutTiming */
        adtg_new[12] = (struct adtg_new) {6, 0x82F9, nrzi_encode(fps_timer_b - 1) }; /* ReadOutTiming end? */

Sidenotes. I see you work from  crop_rec_4k branch. I would prefer working from crop_rec_4k_mlv_snd as it has g3gg0 mlv_snd fixes. Anyway. Please test and report back. Very short of time over here unfortunately.

[theBilalFakhouri]

Sidenotes. I see you work from  crop_rec_4k branch. I would prefer working from crop_rec_4k_mlv_snd as it has g3gg0 mlv_snd fixes.

I have added them to crop_rec_4k branch a long time ago (look at previous commits).

I will check your fix and play with it on my 100D.

[GianlucaM83]

Thanks Danne
I downloaded the build and repeated the tests.
I noticed that there are some new modules in this build.
I only loaded the ones I had already loaded in the previous builds.
Unfortunately the artifacts are still there in the 1x3 modes.
One thing I noticed but maybe it is not important. In all the builds I have tried.
When I activate the crop mode in the video mode. On the liveview of the camera I see the same artifacts that are in the recorded video.
 While recording a raw video I see at the top of the live view the same colored, flashing bands that I see later in the video I recorded.
Currently the preview is set to realtime. if I try to change it to another option it resets itself when I exit the magic lantern menu.
i don't know if i explained myself correctly. my English sucks.

[Danne]

Hm, I think it's related to powertime registry but hard to know for sure. Is it always happening or only sometimes?

[SebastianC]

Thank you Danne! I will try new !

[GianlucaM83]

Danne.
It happens every time and only when i enter the 1x3 crop mode in video mode.
if i activate the 1440p 1:1 preset there are no visible artifacts on the liveview or even in the video i recorded. maybe it is just a coincidence

[theBilalFakhouri]

I noticed that there are some new modules in this build.
I only loaded the ones I had already loaded in the previous builds.

The new modules are there in repo, it either not supported yet or it's for development stuff, don't use them.
I always delete them before uploading a new build for 100D, it's okay for now because we are providing testing build, and these builds won't be used after we solve the problem.

One thing I noticed but maybe it is not important. In all the builds I have tried.
When I activate the crop mode in the video mode. On the liveview of the camera I see the same artifacts that are in the recorded video.
While recording a raw video I see at the top of the live view the same colored, flashing bands that I see later in the video I recorded.

Seems normal, these type of artifacts should be shown in both LiveView and recorded RAW data. (why you have these artifacts on your 100D isn't normal)

Currently the preview is set to realtime. if I try to change it to another option it resets itself when I exit the magic lantern menu.

Yeah, for presets which have correct real-time preview there is no need for Framing preview, I made it like that.
Probably I should make an option for the user too, to let him select which preview he want to use.

Thank you Danne! I will try new !

Do you have the same artifacts which GianlucaM83 has?
If no, there is nothing new to test here.

[GianlucaM83]

just another little test I don't know if it can make sense.
maybe it's the summer heat playing tricks on me 

-UHD 1X3 preset, 12 bit, 23,976 fps.

400 ISO, artifacts are visible
https://drive.google.com/file/d/1P8o22RTubL9hY1vmsD2z7yQ7BGjsephS/view?usp=sharing

800 iso, artifacts are not visible
https://drive.google.com/file/d/16Q3pZ2neZvQb5dPC3Xa9_S-RVlQL9dCu/view?usp=sharing

[theBilalFakhouri]

@GianlucaM83 let's try these:

First, backup your camera ROM files, you will find them in ML/LOGS folder, copy them to a safe place (also let me have them, zip them then upload it somewhere and send me the link via personal message), please Don't share the link publicly, it's copyrighted material.

Tests:

1- Let me take a look into your 100D registers:

-Download this adtg_gui.mo, put it in ML/modules folder in SD card
-Unload all modules, and load only adtg_gui.mo, restart camera
-Set video mode to 1080p24 to gain extra speed

-Follow the following video, you can see in the video I am entering Canon menu and toggling between x1/x5/x10 modes, you should do that too to let "ADTG registers" bring the registers into it:

-After you finish, you will find "adtg.log" file in ML/LOGS folder in SD card, upload it somewhere and share it.

2- Try this build:
crop_rec_4k.2022Jul03.100D101_delay.zip

-Just install the build and try UHD or 4K 1x3 presets and feedback please.
-In this build I made the delay longer before refreshing LiveView from 100 MS to 500 MS (MS = Milliseconds), I need to make sure if it's not a race condition.

3- Update your camera firmware to this one:
https://pel.hu/down/v101-sl1-100d-x7-win.zip

-Format your SD card in PC (remove ML)
-Put CCF16101.FIR in SD card
-Make sure battery is fully charged
-Run firmware update process

Edit:
Note: I don't know if this would affect or reset autofocus calibration data for your kit lens!!

-Then install ML build and see if UHD or 4K 1x3 presets still have the artifacts or not.


4- I will prepare a build for manual CMOS 7 fine-tuning, will post it with more details later.

Please make these tests in order.

[theBilalFakhouri]

Artifacts were reproduced on my 100D :

Thanks to ADTG registers LOG provided by @GianlucaM83, I noticed 5 ADTG registers which have different values on GianlucaM83's 100D, which are:

ADTG2[c00d]:    0x5249
ADTG2[c00e]:    0x1
ADTG2[c00f]:     0x1
ADTG2[c010]:    0x2
ADTG2[c011]:    0x2

-On my 100D it's:

ADTG2[c00d]:    0x5000
ADTG2[c00e]:    0x53
ADTG2[c00f]:     0x52
ADTG2[c010]:     0x52
ADTG2[c011]:     0x52

I copied GianlucaM83's 100D values to my 100D, then the artifacts appeared in 1x3 presets.
These registers are gain registers.

-Let's try my 100D values on GianlucaM83's 100D using this build:
crop_rec_4k.2022Jul04.100D101_ADTG.zip

@GianlucaM83
Please try provided build and 1x3 presets then feedback please.

[GianlucaM83]

With this latest build the artifacts seem to be gone.
I will try tomorrow in other lighting conditions. But I think this is the right way.
Thanks Bilal and also to the others who helped me 
You guys are great 

[GianlucaM83]

Today I tested the new build:"crop_rec_4k.2022Jul04.100D101_ADTG".
In the same lighting conditions where I had artifacts before.

-Tested all 1x3 presets. 12 bit, 23.976 fps, 25 fps.
The artifacts are no longer visible 
I put the link with the test MLV files:
https://drive.google.com/drive/folders/19E3IRpgkBfjoIsFjB7MfDkf6fDS4xKcO?usp=sharing

Bilal you did it 
Thanks!