Canon 100D / SL1

[IDA_ML]

@IDA_ML: I assume you enable fps override in 5x to extend the recording duration? if true please report back if the hickups also appear with fps override disabled

Yes, I do enable fps override at 5x-magnification.  I performed tests at different fps settings from 24 to 2 fps and the hick-ups with moving objects are always there, even at very low resolutions.  I believe, I also tested with fps override disabled and they were still there but I will double check again when I get back home from work.

[IDA_ML]

Nikfreak,

I can now confirm that the hick-ups occur also at fps override disabled.  I shot several clips at different resolutions and there were hick-ups in all of them. 

[IDA_ML]

Is nobody able to provide an answer to my question #936 on the previous page 38?  I am sure, this issue concerns many users who want to use image stabilization while taking silent full-resolution pictures.

[Lensbeginner]

Hi,
I'm evaluating a(nother) camera purchase based on the need of ~2h unattended recording each time.
I'm already an advanced photo-amateur.
I've read conflicting informations regarding continuous recording times... ranging from a few seconds (that's with audio, if I understood well) to a not better specified "continuous".

My question: is it possible to attain ~2h uninterrupted (without gaps due to spanning, glitches, whatnot) 29.97fps 720p 8/10bit RAW recording, no audio is fine by me, and a bitrate around 30Mbps (3.75MBps or 13.5GB/h) on the Canon 100D?

Thanks in advance.

[Walter Schulz]

No. Uncompressed RAW data rate without compression will be around 27 MByte/s. Compression ratios of 27:3.75 will not be seen in RL.

[Lensbeginner]

No. Uncompressed RAW data rate without compression will be around 27 MByte/s. Compression ratios of 27:3.75 will not be seen in RL.

Ok, 27MB not Mb. Thanks.
So no chance at all to record ~2h even with Magic Lantern mod, even h264, mgpeg2, AVCHD, you name it?

[Walter Schulz]

No problems recording 2h RAW at all (presumed overheating isn't an issue).

It's just your specs making it impossible to match.

[Lensbeginner]

No problems recording 2h RAW at all (presumed overheating isn't an issue).

It's just your specs making it impossible to match.

Thanks.
In the meantime I found a Magic Lantern fps calculator (Excel), and worked out that I really need h264 to obtain reasonable file sizes on one hand and resolution on the other.

[Walter Schulz]

No seamless recording in H.264 mode. Some frames will get lost every 29:59.

[IDA_ML]

Hello all 100D users,


After reading Dfort's recent developments on removing focus pixels:

https://www.magiclantern.fm/forum/index.php?topic=16054.525

and Bouncyball's implementation of these developments in the latest MLV_DUMP.exe:

https://www.magiclantern.fm/forum/index.php?topic=18975.150

I decided to check them out with the 100D.
 

Here is what I did:
---------------------

1) I downloaded the latest MLV_DUMP.exe from here:

https://bitbucket.org/bouncyball/magic-lantern/downloads/

2) I downloaded the latest focus pixel maps (FPM) as prepared by Dfort from here:

https://bitbucket.org/daniel_fort/ml-focus-pixels/src/02873af5448795686684b921ade8ca830e9f6cfb/focus_pixel_map_files/?

Then I simply replaced the old FPM files in my MLVFS_x86 directory with the new ones.  In this way, I was able to test MLVFS.

3) I shot several clips with the 100D and the October 23-rd, 2017 build, downloaded from here:

https://bitbucket.org/ehoutsma/magic-lantern/downloads/

The reason I use this build is that it works with synchronous sound in the 8...11 bit losslessly compressed mode at the normal 1736x976 and the 1808x1008 (Movie Crop Mode) resolutions at which continuous recording is possible. 

I converted the MLV clips shot in this way, into cDNG sequences using both: the new MLV_DUMP.exe and the MLVFS and inspected randomly selected frame grabs at 100% magnification using Adobe Camera RAW (ACR) and Photoshop.

Results:
--------

   - Normal RAW video recording mode at 1736x976 resolution:   No Focus pixels
   - Movie Crop Mode (RAW video) at 1808x1008  resolution:      No Focus pixels

4) I also tested how Dual ISO works with the Normal RAW video recording mode at 1736x976 resolution just by activating Dual ISO 100/800 and repeating the test.  After converting the Dual ISO MLV file into a cDNG sequence, I applied CR2HDR to it to convert the frames into normal DNG files with the icreased dynamic range.  Also in this case, NO Focus Pixels were observed, however, as expected, aliasing was so strong that it makes this type of Dual ISO video useless.

5) I tried to test Dual ISO also with the Movie Crop Mode (RAW video) at 1808x1008  resolution.  Unfortunately, after extracting the cDNG frames from the Dual ISO MLV file, the CR2HDR that I downloaded from here:

https://builds.magiclantern.fm/utilities.html 

failed to perform the CR2HDR operation.  It starts convering but then it crashes and the Dual ISO DNG-frame remains unconverted.  CR2HDR worked fine with the sequences from #4 though.  It also worked with Dual ISO files shot at the maximum 2520x1080 resolution in the 5x-magnification mode but I will provide results from that test later.

[IDA_ML]

Dual ISO FPM test on the 100D at the maximum 2520x1080 resolution with the 8 ... 11-bit losslessly compressed mode
==========================================================================

There are several reasons why this 5x-magnification high-resolution mode would be very interesting to use with Dual ISO.  Here are a few:

1) At low ISO (100 and 200 ISO), it provides very high video quality (a lot of fine detail, beautiful colors, smooth tone transitions,  excellent dynamic range, etc.) which makes it ideal for shooting landscape videography, smooth timelapses, etc. 

Even at high ISO-values, if the scene is properly exposed, the results are quite satisfactory.  I often film at ISO 800 and 1600 with the 100D and am quite satisfied with the image quality.  Here is a frame grab from a clip from this morning that I shot at ISO 800:





On this photo, I deliberately pushed the shadows a little bit in ACR to create an impression of the noise in the dark areas.  As you see, the noise is monochromatic, (see the TV set), meaning that it can easily be reduced in post to pleasing levels without degrading fine detail too much.

2) When shooting events in rooms and halls that are not well lit, (e. g. people dancing under lights hanging from the ceiling), the background that occupies a large area of the frame, is often too dark.  When you try to push the shadows a little bit to let some more  detail come up, the noise becomes too intrusive as shown in the above photo.  In this case, if the clip is shot at Dual ISO, the noise in the dark areas of the clip may be greatly enhanced.  The same applies also to high-contrast landscape scenes, city light videography at night, etc.

For these reasons I decided to check out Dual ISO in that mode too. For the test I shot the same scene at exactly the same camera settings but at Dual ISO 800/3200.  Then I applied MLV_Dump and CR2HDR to get the final result and applied a few corrections to it in ACR to match the photo from #1.  Here is the result:






On this photo, despite the fact that the high-ISO setting is 3200, the noise on the TV set is almost identical to the one from the previous photo shot at ISO 800.  This means that the Dual ISO technique has 2 stops potential for improving shadow noise in videos shot in this mode.  No focus pixels are visible (excellent work, Dfort !!!)  Unfortunately, there are other artifacts visible in the photo but they have nothing to do with the focus pixels.  We have now overexposure in the highlights and, therefore, magenta colorization on those areas (see the wall).  Also Dual ISO horizontal lines are clearly visible on the overexposed areas.  Those lines were visible on the frame grab right after the CR2HDR processing, so they do not have anything to do with focus pixels either.

My impression is that the above artifacts are caused by the quite extreme Dual ISO settings of 800/3200 that I had to use for this clip due to the dim light.  This probably caused reduction of the dynamic range and overexposure of the highlights.  To verify this, I shot another clip of a much better lit scene where I used Dual ISO 100/800.  The CR2HDR procedure reported 11,5 stops dynamic range on that frame.  Here is the result:






Here we see some small focus pixel residues in the highlights but basically, the result matches the one obtained by Dfort here:

https://www.magiclantern.fm/forum/index.php?topic=16054.msg195983;topicseen#msg195983

As he demonstrated, these can be fully removed by adding some chroma smoothing.  There are other artifacts that could be more of a concern.  They appear along the edge of the cupboard, on the frame of the painting and on the plant in the background.  I am not sure what their origin might be but they could also be lens related.

CONCLUSIONS:
==========

Dfort and Boncyball have done a hell of a job in removing focus pixels in the losslessly compressed modes of the 100D.  The resulting video clips are very clean and free of focus pixels up to the maximum resolution of 2520x1080.  Even Dual ISO works well at this resolution and provides fairly clean results.  Care should be taken to not use too extreme Dual ISO settings and to properly expose the scene.  Overexposure in the highlights should be avoided.  Further tests are necessary to see if the Dual ISO technique will prove usefull in real-life filming conditions but if yes, this is another reason why the quite annoying "hick-up" issue that appears with moving objects in this mode should be fixed.  Nikfreak, you are our last hope in that respect ... !

[dfort]

I see lots of problems on one of your shots. Left over focus pixels and a pattern over light flat areas (not to mention too many refrigerator magnets):



Could you post trimmed down MLV's of your 100D tests so we can take a closer look?

Code Select Expand

-- MLV output --
  -f frames           frames to save. e.g. '12' saves frames 0 to 12, '12-40' saves frames 12 to 40. forces --no-audio switch


Example--save the first 4 frames to output.mlv.

Code Select Expand

mlv_dump -f 4 [input.mlv] -o [output.mlv]


[IDA_ML]

Dfort,

Here you go - 3 shortened MLVs (5 frames each) from the originals:

https://we.tl/vUnFAIAPFX

Could not remove the refrigerator magnets, sorry :-))).  Link will be active in the next 7 days.

[dfort]

Looks pretty good over here. How are you processing?

This is with MLVFS and Adobe Camera Raw.





Do your blinds have that pattern? I don't think that's a Dual ISO pattern. [EDIT] I think therefore I'm wrong.

This is what it looks before the cr2hdr process:



I don't see any focus pixels anywhere on this shot.

[EDIT] I do see some issues on the other two shots. Need to take a closer look at what is going on.

[a1ex]

Do your blinds have that pattern? I don't think that's a Dual ISO pattern.

It is a Dual ISO pattern.

Issue: the sample video was recorded at 11-bit lossless (black 2047, white 3467), but cr2hdr expects full-range input (white level anywhere between ~10000 and 16383). Scaling the input data before cr2hdr processing (e.g. "b = (a - 2048) * 8 + 2048" in octave) gives correct results.

This has to be fixed in either mlv_dump or cr2hdr.

[IDA_ML]

Dfort, A1ex is right.  It is a Dual ISO pattern. 

My workflow:
==============

1) Applying MLV_Dump to the original Dual ISO MLV file from the camera to extract the DNG sequence.  No switches are used, just the plain MLV_dump;

2) Applying CR2HDR from the Utilities page to the DNG sequence to obtain the final DNGs with increased dynamic range.  The Dual ISO pattern is clearly seen on the overexposed areas after this operation.

3) Applying basic corections to the final DNGs in ACR (Blacks, Whites, Shadows/Highlights, White balance, Lens corrections).  Export to JPEG.

============================

A1ex,

The CR2HDR package from the Utilities page does not work with Dual ISO MLVs shot in the Movie Crop Mode on the 100D, (in step #2 above).   It starts converting the DNG frame but in the middle of it, it crashes and the frame remains unconverted.  Would you like me to upload a 5-frame sample of such a MLV file for you to have a look?

I expect that Dual ISO will work well with Dual ISO files shot in the Movie Crop Mode  since there is no aliasing in that mode.

[a1ex]

I believe it's the same issue; you can upload, but I won't be able to look into it these days (maybe just for confirming the issue).

[IDA_ML]

Here is a Dual ISO (100/800) MLV file shot with the Movie Crop Mode at 1808x1008 resolution.  Unfortunately, I was unable to reduce its length to 5 fames as I did with the other files.  All 5 frames were corrupt.  That is why, I am uploading the entire MLV file as shot with the 100D (350 MB).  Sorry about that.  Here is the download link (active for 7 days):

https://we.tl/y3dAM8dhP6

For those with a limited bandwidth, here is the 10-th frame of the cDNG sequence (3 MB), extracted with MLV_Dump in the normal way, which works just fine:

https://we.tl/EkkDxmc4K5

If CR2HDR is applied to this single frame the crash is observed.

[dfort]

Besides the cr2hdr issue -- I see dots!

@IDA_ML - Could you do a short test using the same settings but without Dual ISO?

[a1ex]

I only get the dots with the downloaded DNG (which was created with mlv_dump on steroids), but I'm unable to reproduce them with vanilla mlv_dump (with default settings, it finds 0 cold pixels and does not activate vertical stripe correction).

Scaling the input data to full range in octave (before running cr2hdr) fixes the cr2hdr issue. Without full-range scaling, no crash happens; it's just unable to match the two exposures and gives up.

[bouncyball]

I also can confirm that there are no focus pixels. Full 20bit dualiso processing can not handle it but preview mode gives this image:





It has some colored aliasing (because of the low quality dualiso processing) and lots of vertical lines of the curtain

[bouncyball]

It looks like 8-10bit lossless. White level is 3467.

Edit: in mlv_dump os it should be processed with '--no-fixfp' option to turn off focus pixel fix and do not touch healthy pixels.

[IDA_ML]

@IDA_ML - Could you do a short test using the same settings but without Dual ISO?

Yes, Dfort, I could but not now.  It's dark outside :-(((.  I will try to shoot another clip without Dual ISO tomorrow before I go to work.

By the way, I was not quite able to get your clean result from your post #963 when I tried to reproduce it with MLVFS.  The focus dots left from MLV_dump are gone but I still get strange artifacts in the highlights, behind the fridge.  See for yourself:





Did you apply chroma smoothing or some other trick to get this clean result?

@Bouncyball

I am glad you got this Dual ISO Movie Crop Mode frame properly converted.  Do you think, we can get rid of the aliasing artifacts with the existing cr2hdr tools or higher quality processing will be necessary?

[bouncyball]

IDK, I did not try to scale the data to full range as a1ex did. If I got it right, as he mentioned, for this particular case in 20bit high quality mode (cr2hdr), two exposures can not be matched and this is the issue.

Preview mode can be used in MLV App or MLVFS, then you can try to get rid of aliasing with some workflow, suggested by a1ex or Danne on other threads.

Edit: beside aliasing this low quality image has other artifacts too (but no focus pixels).

[dfort]

Glad to hear about the absence of focus pixels though I tried different apps keep seeing them. Maybe I need to take a vacation. The video I posed was done on Switch. It usually does a good job on Dual ISO.

Looks like your latest has focus pixels and the Dual ISO pattern.



The focus pixel map file for the 100D in mv1080crop mode is one of the most complicated with 206,400 mapped pixels covered in two passes. Here is what the pattern looks like:



On the 650D/700D/EOSM I had to create four pass map files to get it working with 8...12bit lossless mv1080crop.