Canon 100D / SL1

Started by nikfreak, October 19, 2015, 10:41:29 PM

Previous topic - Next topic

0 Members and 2 Guests are viewing this topic.

bouncyball

I guess Switch uses focus pixel fix by default. If focus pixel map is applied it produces phantom focus pixel artifacts all over the image. I guess it's because of this image has so many vertical curtain lines with contrasty transition.

IDA_ML

Hello Dfort,

As you requested, I have now shot another Movie Crop Mode (MvCM) file of yesterday's high-contrast scene in the 8 ... 11 bit losslessly compressed mode but this time without Dual ISO.  To make sure, you have exactly the same camera settings and do not compare apples with oranges, I have shot also the same scene in Dual ISO 100/800.  For your comparison, I am providing also another pair of clips shot at 14-bit lossless with and without Dual ISO, respectively to check and see if the bitrate of the LL compressed videos has an influence on the focus pixels and the Dual ISO behavior.  All 4 clips were recorded at the following camera settings:

         F9,1; 1/50s and basic ISO 100, (100D, EF 24/2,8 IS, Oct. 23-rd, 2017 build)

You can download the archive (634 MB) with the 4 clips in the next 7 days from here:

https://we.tl/5h1L5IHogQ

I am sorry, I was unable to shorten the clips.  As I explained above, shortening with MLV_dump produced corrupt frames.

Happy testing!

bouncyball

I hate 'wetransfer.com'. When speed surpasses 8mb/sec download always stops and, the ugliest thing is, it can not be resumed. Tried 20times already to no avail...

IDA_ML

I am sorry to hear that.  I don't have this problem since speed rarely exceeds 2-3 MB/s here.  Could you suggest another site for uploading files that does not have this speed issue? 

Danne

Quote from: bouncyball on January 17, 2018, 09:39:00 AM
I guess Switch uses focus pixel fix by default. If focus pixel map is applied it produces phantom focus pixel artifacts all over the image. I guess it's because of this image has so many vertical curtain lines with contrasty transition.

What´s the correct workflow here when it comes to dualiso MLV files?

bouncyball

Quote from: Danne on January 17, 2018, 10:44:48 AM
What´s the correct workflow here when it comes to dualiso MLV files?
I have no idea :P

But in this particular case fixing of not existent focus pixels is an enemy.

Danne

I see, we are discussing lossless 8-12 bit files not 14bit? So simply disabling fpm files for lower bits might do the trick.

bouncyball

Quote from: IDA_ML on January 17, 2018, 10:36:10 AM
I am sorry to hear that.  I don't have this problem since speed rarely exceeds 2-3 MB/s here.  Could you suggest another site for uploading files that does not have this speed issue?
IDK, google drive always worked for me.

IDA_ML

Bouncyball,

I have just uploaded the package with the Movie Crop Mode MLVs onto another storage server called "tranzit".  Here is the download link, active for 120 hours:

http://tranzit.dir.bg/load.php?id=WktUhR9ZqKqj6rXE3016921

It will prompt you to type a distorted 4 symbol code into a box and then you have to click on the blue line to start the download.  Please let me know if you succeeded.




bouncyball


dfort

Quick report.

I can't get the 8...12bit Dual ISO isn't working with MLVFS or mlv_dump/cr2hdr. The normal shots worked fine but are quite underexposed.



Looks like maybe the 2-pass map isn't needed on the 100D mv1080crop 8...12bit? Hard to tell with this test because there are so few focus pixels showing up.



Here's the 14bit lossless Dual ISO after playing around with it. Recovered lots of shadow detail but it is a bit noisy. No focus pixels, though.



Well, almost. There were a few along the edges that persisted.



By the way, earlier I was getting lots of focus pixels. When testing out MLVFS I often pull out the fpm files to see the focus pixels. Guess what? I forgot to put them back.  ::)

All fine now!

IDA_ML

Thank you for your feedback, Dfort.  Well, it does not look all that bad at all.  I am very happy to see that there is no aliasing on the Dual ISO 14-bit shot which may make the Movie Crop Mode quite usable with Dual ISO.  Yes, the shot is quite noisy but keeping in mind the extreme contrast this scene was shot at and how much you had to push the shadows to recover detail, I think, noise is to be expected.   

As far as underexposure with the 8...11 bit LL files is concerned, their exposure settings were the same as with the 14-bit ones.  They look underexposed on the screen due to the lower bit rate but in fact, they are normally exposed.  If you try to bring them to normal exposure in ACR you will see that their noise levels are about the same as with the 14-bit LL.  When I shoot videos at 8...11 bit LL, I trust much more the histogram than the dark image on the screen.

dfort

Sorry, I meant that all of the shots were underexposed. You're far from clipping the highlights.



This is what the Dual ISO histogram looks like before processing:



I'm not an expert at exposing Dual ISO but this looks underexposed to me.

Focus pixel fixing seems to be working, though maybe the 100D doesn't need the 1-pixel offset pass that the 700D needed. What would really help is shooting that test pattern @theBilalFakhouri used. It is still available on his Google Drive from the link in his post.

The main issue here is that cr2hdr isn't processing your 8...12bit Dual ISO shot.

IDA_ML

Dfort,

Do you mean the file PIXEL SCANNER V4.mp4?  Should I play it on my computer screen and record it from there?  In which mode should I do that - normal uncropped, Movie Crop, 5x-Crop, Dual ISO or something else?

dfort

@theBilalFakhouri detailed how he did it but even if you just put the camera on a tripod and shoot it off your computer screen it should work.

Every video mode at every bit rate and compression setting would be best but right now what I'm most interested in are mv1080crop and 5x zoom -- the 1:1 sampling video modes using 8...11bit, 12bit and 14bit lossless compression.

Don't bother testing Dual ISO with the test pattern. We know there is a problem but it shouldn't affect the focus pixels.

IDA_ML

All right, I'll see what I can do.

canneloni

I shot the test pattern in different modes on the 100D. (8...11 Bit lossless, 12 Bit lossless each in mv1080, mv1080moviecrop, mv1080_5xZoom). The settings used were f4, shutter 1/50th, ISO 200 and manual WB at 6100 Kelvin (WB adjust with PushWB from shortkey menu in liveview) exposure was measured as about 1 stop underexposed.

If this is usefull to anyone you can download it here:

https://drive.google.com/open?id=11RJPTv3pQpRpDSfLCild9BL8XGh1D8xX

Edit: I used the build IDA linked in this post. I was not able to record continuously with the above settings and even got some errors when i tried to record 14 bit lossless. Had to restart the camera after i tried. 
100D.100B ; Canon 18-55 STM ; Canon 50 1,8 II ; Canon 75-300 4,0 - 5,6 III ; Sigma 17-50 2,8

dfort

Great test @canneloni !

I need to take a closer look but right away look at the psychedelic display of focus pixels on mv1080:



Compared to mv1080crop -- without fixing the focus pixels:



Thanks for this.

dfort

Ok, this is what is going on with the 12bit zoom mode and most likely also on the mv1080crop mode. [EDIT] This also applied to mv1080crop.

It doesn't matter if the current focus pixel map is used or the --no-fixfp in mlv_dump or if I remove all of the map files from MLVFS, the results are the same. Focus pixels pop up near contrasty borders.



These aren't quite as obvious as they are on the mv1080 test but they are there. They are usually green or magenta and fall into a pattern. (Click on the image for a closer look.)



It turns out that on these tests the pattern was shifted 8 pixels to the right. On the 100D the pattern repeats every 12 pixels so you can also say that it was shifted 4 pixels to the left. Is it something that happens all of the time or can it be in the "normal" position on one shot and shifted on another shot using the same settings? Since we've seen this happen before on other lossless compressed footage let's make a second pass shifting the map 8 pixels to the right:



I'll check the mv1080crop shots and run some tests on the files from @canneloni and @IDA_ML submitted for testing and if everything checks out I'll update the focus pixel map files.

By the way, I'm using both mlv_dump (the latest version on steroids) and MLVFS on these tests along with dcraw and ACR so if you notice a color shift, it is probably just me using different apps.

IDA_ML

Dfort,

You have just acomplished another achievement!  I tried with MLVFS your new focus pixel maps that you updated a few hours ago.  I used the old Dual ISO 100/800 file that I shot with the 100D in the 5x-crop mode at 2520x1080 resolution a few days ago.  And guess what - THERE IS NOT A SINGLE FOCUS PIXEL LEFT !!!  The file is amazingly clean!  We now have just the Dual ISO artifacts in the overexposed highlight areas.  See for yourself:



 

If CR2HDR could be fixed as A1ex suggested then this 5x-crop mode will be perfectly suited for high-qualty Dual ISO shots.  If also the hiccup issue could be resolved in this mode, then this would be a dream come true for the small but amazing 100D.

Thank you so much, Dan!

dfort

Whoa--the Dual ISO lines on the highlights are egregious!

It would also be nice to get rid of the focus pixels that are on the edges of the frame on your shot:



IDA_ML

Now that Dfort has made such a remarkable progress with fixing focus pixels on the 100D and other cameras, it is probably the right time to raise again the issue with the "hiccup" bug that makes one of the most creative modes in the 100D unuseable - the 5x-crop mode in which high-quality video recording is possible at resolutions up to 2520x1080.  The problem is described in detail in #4 here:

http://www.magiclantern.fm/forum/index.php?topic=16040.msg195484;topicseen#msg195484

I have also prepared a test video clearly illustrating the hiccup jumps with moving objects in the scene:



Sofar, it is not clear what causes these jumps.  If someone could provide an explanation and suggest a fix, I am sure, this will be greatly appreciated by those who use ML for video shooting with the 100D.



dfort

What were the exact settings you used?

I know that it could get confusing because there are several "crop" modes but you're using zoom mode which is set when the Canon menu is at 1920x1080 and you press the magnifying glass button until you get into 5x magnification. However, to record the largest sized image you need to use one of the reduced bit settings and lossless compression. Is this happening at 14bit lossless or only 8...12bit lossless? Maybe this has something to do with it?

Quote from: a1ex on December 30, 2017, 09:58:10 AM
...8...12-bit lossless modes (they use raw type 0x12 DEFCORRE, other modes use 0x10 CCD aka the first step in the pipeline).

Maybe there is a problem using 0x12 DEFCORRE in zoom mode? Maybe experimenting with a different PREFERRED_RAW_TYPE?

src/raw.c
#define RAW_DEBUG_TYPE   /* this lets you select the raw type (for PREFERRED_RAW_TYPE) from menu */


There's a "warning FIXME" in the skip values so maybe this needs to be changed?

[EDIT] I was looking at the unified branch, this was fixed in the crop_rec_4k branch but I'll leave what I posted here because maybe this is causing some issues? Though I doubt it.

raw.c
        #if defined(CONFIG_650D) || defined(CONFIG_EOSM) || defined(CONFIG_100D)
        #warning FIXME: are these values correct for 720p and crop modes?
        skip_top    = 28;
        skip_left   = 74;
        skip_right  = 0;
        skip_bottom = 6;
        #endif


From the files I've been looking at for the cameras with the focus pixel issue it seems to me that it should be:

       skip_top    = 28;
        skip_left   = 72;
        skip_right  = 0;
        skip_bottom = 0;


Note that the skip values are different on the 700D. These cameras are very similar so maybe this should also apply to the 100D/650D/EOSOM?

        #ifdef CONFIG_700D
        skip_top    = 28;
        skip_left   = 72;
        skip_right  = 0;
        skip_bottom = zoom ? 0 : mv1080crop ? 0 : 4;
        #endif


Could it be that the skip_bottom in zoom mode that might be causing the "hiccup" issue?

canneloni

@Dfort
If you can provide a build I'd be happy to test it for you. If you need anything else, just let me know!
100D.100B ; Canon 18-55 STM ; Canon 50 1,8 II ; Canon 75-300 4,0 - 5,6 III ; Sigma 17-50 2,8

IDA_ML

Quote from: dfort on January 20, 2018, 07:05:49 PM
What were the exact settings you used?

I know that it could get confusing because there are several "crop" modes but you're using zoom mode which is set when the Canon menu is at 1920x1080 and you press the magnifying glass button until you get into 5x magnification. However, to record the largest sized image you need to use one of the reduced bit settings and lossless compression. Is this happening at 14bit lossless or only 8...12bit lossless? Maybe this has something to do with it?

Dfort,

I use the 5x-magnification mode exactly as you describe it above.  I am not talking about the Movie Crop Mode where everything works fine.  I just select the RAW video mode and press the magnifying glass button until I get into 5x magnification.  Then I focus, meter and start recording.  The hiccup behavior occurs ONLY while recording, at random locations in the clip, at ANY resolution, ANY framerate and ANY bitrate (uncompressed or losslessly compressed) and ANY Preview selection in that 5x-crop mode.  It is not observed during preview of the scene on the camera screen before recording is started but it is clearly seen upon in-camera playback  of the recorded clip.  Finally, hiccup jumps are observed even in very old builds - it has always been there, also in the RAW video (MLV) mode at 5x-magnification. 

Unfortunately, I cannot answer the other questions and comments of yours.  I just don't understand them.  This is the reason why I asked if hiccups are observed with other similar cameras - 650D, EOS-M, 700D.  It is very easy to check and see that - just use the above settings and shoot a clip with moving objects that is at least 5 s long, (reduce resolution if necessary).  If your cameras generate hiccups, you will see them by playing back the clip. 

I can confirm that hiccups are not observed with the 7D.