Author Topic: Fixing Focus Dots from Canon 5D Mark III & How To Avoid This Issue In The Future (Read 8721 times)

PaulHarwood856 · « **on:** October 12, 2019, 11:19:03 PM »

Hello Magic Lantern Community,

I've been editing a choir concert lately, and have noticed several Focus Dots from 1080p 12 bit lossless raw video from the Canon 5D Mark III. I used ISO 1600 and a Sandisk Extreme Pro CF Card. I messaged Danne and dfort, and they have helped me a lot. I created a fpm file with coordinates of where the focus dots are. I used Fix Bad Pixels On, Aggressive, and Interpolation 3. Also, I have Fix Focus Dots On. However, I keep seeing more focus dots in my footage and have had to move on and edit the project with how the footage currently is. My questions for the community are: How can I avoid this in the future? & What is the best way to remove Focus Dots? Here is a link to the files: https://drive.google.com/open?id=1gi9tnSDQenCl32aRLhaXNFLlRqpyWdWa

Any input on this would be greatly appreciated. Thank you.

- Paul

Danne · « **Reply #1 on:** October 12, 2019, 11:37:15 PM »

Sorry I am a bit short on time lately.
In Switch I kept a workflow which will allow for creating a dot map through the use of imagej. Very long since I used it. You reach some instructions by selecting (m) mlv_dump and in that menu at the bottom there's an alternative to open up a file and create this list. Follow instructions on installing imagej etc.
The best I can do atm.

Checked you file. how can iso 1600 give so many focus pixels? Really fishy.

a1ex · « **Reply #2 on:** October 13, 2019, 12:05:17 AM »

Quote from: PaulHarwood856 on October 12, 2019, 11:19:03 PM

How can I avoid this in the future?

Easy - by using the builds from the download page. There are no focus dots on the 5D Mark III, that I know of.

However, there are some low-level settings (lv_af_raw) that cause some kind of subtle focus dots. If these were changed in the build you were using, then... sorry, it wasn't me who introduced this problem

The pattern looks quite interesting though - I've never tried to map it. These dots are very likely introduced by Canon software, but what exactly they are used for, I have no idea.

Quote

What is the best way to remove Focus Dots?

I don't know what is the *best* way (that's an optimization problem), but the dots should not be there in the first place - as explained earlier. Of course, since they are in your clip, you can use the FPM method to fix them - it will work, but there might be better ways. Anyway - I've noticed one of dots at (1406,915), but I could not find it in the FPM file. IIRC, the coords are relative to the top-left pixel on the sensor, not in the recorded image (since one map needs to cover all possible resolutions from a particular video mode), and since I don't remember the exact details... let's reverse engineer the offset. By comparing the DNG with and without that pixel in the FPM file, I've noticed a difference at (1254,783). So, to interpolate our bad pixel, you'd have to write "1558 1047" in the FPM. I could not find it in your file, but I could confirm that, by adding it, that particular dot disappears.

If you already mapped your bad pixels - in the FPM file - you know what you have to do: shift them into the right place. I don't know which of the dots from your list is the one I've identified - worst case, you have only 295 possibilities to try

Good luck!

Danne · « **Reply #3 on:** October 13, 2019, 12:24:30 AM »

Hm, yes, what build is used(hopefully not mine

).
I tried picking coordinates with imagej building a pixel list and when exporting the dng files through Switch (m) option with an allbadpixels.map file it works but as there´s a lot of them and maybe at random places it´s problematic.

EDIT:
Created an allbadpixels.map file:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/allbadpixels.map

Put it next to your MLV file and transcode it to dng files with Switch. Tried finding most of them but I probably missed a few.

PaulHarwood856 · « **Reply #4 on:** October 13, 2019, 12:34:42 PM »

Hey Danne,

No need to apologize. I really appreciate your help. I'm on Windows and have used batch_mlv_in-out before. I'll try your fpm file next to the MLV file and see how it goes. Should I be worried about a faulty sensor? I found it strange at ISO 1600 to have so many focus dots. It does seem fishy.

- Paul

PaulHarwood856 · « **Reply #5 on:** October 13, 2019, 12:40:10 PM »

Hey Alex,

I thought it was a bit odd to face this issue with the 5D Mark III. I messaged Danne and dfort and they gave me advice on how to fix this issue. I was told they are focus dots. In MLV App, with the fpm file installed, I chose Fix Focus Dots. This is the experimental build, the most recent one. Are you saying I won't face this issue with a nightly build? I'll make sure to check the (lv_af_raw) settings.

Quote

If you already mapped your bad pixels - in the FPM file - you know what you have to do: shift them into the right place.

In regards to shifting the dots in the right place, is there a certain order you're referring to?

Thank you for your input Alex. I do appreciate it.

- Paul

Danne · « **Reply #6 on:** October 13, 2019, 12:47:49 PM »

The map file provided could work in batch_mlv. Put it next to your mlv file. If not working let me know.
It's not focus pixels.
Most important. Tell exact ml build you are using.

a1ex · « **Reply #7 on:** October 13, 2019, 01:48:07 PM »

Quote from: PaulHarwood856 on October 13, 2019, 12:40:10 PM

I'll make sure to check the (lv_af_raw) settings.

These are not visible on the user interface (at least not in my builds); it's a string in Canon firmware, or a keyword for developers to search for more info, if you prefer.

It's a setting I did not touch in any of my builds, because I know the consequences

Danne · « **Reply #8 on:** October 13, 2019, 08:54:37 PM »

Quote

Hey Danne,

A while back I was having a few artifacts in my 7D ml raw footage with 60 fps. The solution was to type 10, which opted out of removing cold pixels (Batch_mlv_in and out). Unfortunately I have similar artifacts in my 5D Mark III 1080p 12 bit lossless ml raw video.

Edit: never mind. I misinterpreted the message.
Still need to know what build was used here...

PaulHarwood856 · « **Reply #9 on:** October 14, 2019, 10:40:31 PM »

Hey Danne,

I just sent you a message. I am using the 4K Crop Record Experimental Build for 5D Mark III 1.2.3 July 22nd, 2018. I messaged you the settings I use as well. Here is a link to the settings and build I'm using:

https://drive.google.com/open?id=1MtVsdi6GTOsyAUkSYOAjkF7ip8vXPGRc

I noticed this issue again from filming yesterday. The artifacts were in my footage, and it was a long take. Just like the choir concert... Might have to do with recording too long or overheating.

- Paul

Danne · « **Reply #10 on:** October 14, 2019, 11:33:35 PM »

It's so strange. I can't test atm. Could you test if the issue is gone with another build? Maybe one of these?
https://www.magiclantern.fm/forum/index.php?topic=23041.msg216899#msg216899

PaulHarwood856 · « **Reply #11 on:** October 15, 2019, 01:23:53 PM »

Hey Danne,

I haven't tried a your new build yet, but I noticed something. The artifacts I'm having with the Canon 5D Mark III 1.2.3 Experimental Build July 22nd, 2018 are the same exact issue with the Canon 7D when shooting slow motion ML Raw 14 bit. This happened with 2 different Canon 7D ML Nightly Builds. However, disabling cold pixel fix in Batch MLV fixed the issue. Another resolution was passing through original raw data without processing. I believe disabling vertical stripes fix would solve the issue as well. So, I'm starting to think there is a way of solving this issue without focus pixel maps. If the issue is resolved with the Canon 7D ML Raw 14 bit, then Canon 5D Mark III 12 bit lossless should have a solution. I've tried disabling cold pixel fix, disabling vertical stripes fix, both fixes combined, and passing through original raw data. None of these solutions which work with the Canon 7D work with the Canon 5D Mark III. Here is a link to the post I made on this forum about the Canon 7D:

https://www.magiclantern.fm/forum/index.php?topic=22771.msg205825#msg205825

Here is a link to a long take a couple days ago with the same artifacts issue with the Canon 5D Mark III:

https://drive.google.com/open?id=1LU2huAeMjHANKasbL2ONqmrXR0fgqzaw

- Paul

Danne · « **Reply #12 on:** October 15, 2019, 01:46:34 PM »

Well, please test. I just tried disabling both cold pixels and vertical stripe correction but no difference. Maybe the 12bit stream is one containing those pesky dots. A1ex states his builds don´t show these spots but maybe you found a way on higher iso and in 12bit. Testing 14bit or another build should tell.

PaulHarwood856 · « **Reply #13 on:** October 15, 2019, 02:12:11 PM »

Hey Danne,

Ok I understand. I will test a new build soon as well as 14 bit on the current build. I'm hoping there is a way to fix this issue without a focus pixel map. If the same artifacts occurred in Canon 7D ML Raw, and there is a fix for that, then there should be a solution. Did you by any chance check the ML settings I have in the Google Drive link? Maybe it's a setting in the current build I'm using? I'll make sure to do the tests though. It will be great to avoid this issue in the future.

Oh and one other idea. Do you know if reflashing the 1.2.3 firmware would make any difference?

- Paul

Danne · « **Reply #14 on:** October 15, 2019, 02:37:15 PM »

I am not on firmware 1.2.3. And too much work atm but if I were you I would do an hour of testing out your own ideas and what others suggested. Good luck.

a1ex · « **Reply #15 on:** October 15, 2019, 05:39:22 PM »

Yes, please test 12-bit vs 14-bit. Firmware version shouldn't matter.

I've double-checked lv_af_raw in the code, so I'm definitely not using it, but there may be other similar things in Canon firmware that are unknown to me. Unable to test on the camera, as I'm on 1.3.6 for now. (I know about the WIP from Danne & co., but it's not booting yet.)

DeafEyeJedi · « **Reply #16 on:** October 15, 2019, 08:45:31 PM »

Thanks @a1ex for pointing this thread which got my attention here. Now I've reached out to @PaulHarwood856 earlier this morning via texts and he's currently out shooting test comparisons between 12-bit & 14-bit.

Hoping you're using @Danne's latest (2019Oct13) build. I'm looking forward to seeing the results as I'm very curious about this issue. This is the first I've heard of to have a FPM created by @dfort for a 5D3 body.

PaulHarwood856 · « **Reply #17 on:** October 16, 2019, 02:51:45 AM »

Hello everyone,

I tested Danne's recent build from October 13th, found here:

https://www.magiclantern.fm/forum/index.php?topic=23041.msg216899#msg216899

I haven't encountered the artifacts at ISO 1600 with long takes. I used MLV App and tested 12 bit lossless and 14 bit lossless. Tested 3.5K and 1920x818 with frames per second at 23.976 and exact.

It was really hot out, and in order to test ISO 1600, I had to use a variable nd. I tried to reproduce the issue by shooting long takes out in the heat. The only issue I saw was vertical banding. Other than that there was some of lens flaring with the Canon 16-35 f2.8 Mark I.

I'll have to test more, but so far I haven't encountered the issues I was having before.

Also, before installing Danne's build, I uninstalled Magic Lantern (was using the July 22nd, 2018 Crop Rec 4K Experimental Build) by formatting both the SD and CF card. I used Sandisk Extreme Pros. I then used a card reader to copy the files from Danne's build to both memory cards. Then I updated the firmware, rebooted, and experienced a few issues with loading modules. There were errors. Fortunately I was able to get the modules working by loading mlv lite. I will provide a link when I get a chance of all the settings I used. I decided to restore ML settings to default, then changed settings from there. There were no error files on my SD card or CF card.

Thanks for all of your help. I'll continue testing, but so far, so good

- Paul

Danne · « **Reply #18 on:** October 16, 2019, 06:15:39 AM »

Maybe a bummer here changing the builds. Testing 12bit vs 14bit on your first build would be interesting now. Are you able to reproduce tha issue again on that build?
Module errors will appear whrn certain modules aren't loaded together so that's normal.

DeafEyeJedi · « **Reply #19 on:** October 16, 2019, 06:23:10 AM »

Quote from: Danne on October 16, 2019, 06:15:39 AM

Maybe a bummer here changing the builds. Testing 12bit vs 14bit on your first build would be interesting now. Are you able to reproduce tha issue again on that build?
Module errors will appear whrn certain modules aren't loaded together so that's normal.

Absolutely great call here @Danne — I may have the 2018July22nd build laying around in my download folders on either Mac’s.

Pretty sure Paul would still have a copy on his end.

PaulHarwood856 · « **Reply #20 on:** October 16, 2019, 12:12:49 PM »

Hey Danne,

I still have the July 22nd, 2018 1.2.3 Crop Rec 4K Build on my other CF Cards. This build is on the Experimental Builds Page found here:

https://builds.magiclantern.fm/experiments.html

I'll see if I can reproduce the issue when I have some spare time. The artifacts occur with 1920x818 12 bit lossless ML Raw video with frames per second override on at exact fps and ISO 1600.

- Paul

PaulHarwood856 · « **Reply #21 on:** October 30, 2019, 11:57:48 AM »

Hello everyone,

I'm currently using Danne's October 15th Build: crop_rec_4k_mlv_snd_isogain_1x3_presets_2019Oct15.5D3113.zip found on this page: https://www.magiclantern.fm/forum/index.php?topic=23041.msg216899#msg216899

I filmed at a park in Delaware on Monday 10-28-2019. I decided to record in ISO 1600 with an nd filter on (SLR Magic Variable ND MK2). Last weekend I went to a park in Pennsylvania and shot at ISO 100 with Danne's build, and got no artifacts/dead pixels/focus dots (whatever these artifacts are). No artifacts at all, a few clips had vertical bands.

When I shot at ISO 1600 in Delaware, I now have artifacts in my videos, and even though I had 3.5K module enabled with 23.976fps override at 2:35.1, only 1920x818 was recorded when I used 5x crop. When I shot at ISO 100 I didn't have this issue. At ISO 1600, I did still get vertical bands on a few clips (both ISO 100 and 1600 I have had vertical bands on a few clips).

So, I've tested the artifacts issue on 3 different days/shoots. I've found ISO 1600 1920 x 818 (no x5 crop) is where the issue occurs. What can I do to figure out this issue? Is it possible to share an MLV that has this issue and for someone to take a look? If I take pictures of all my settings, is there something I need to look for or maybe someone can take a look? I'm worried my camera might have an issue, but am not sure how to tell. Any advice would be greatly appreciated. Thanks.

- Paul

PaulHarwood856 · « **Reply #22 on:** October 30, 2019, 12:19:38 PM »

Hello everyone,

Forgot to mention I only recorded in 12 bit lossless, and forgot to test 14 bit. I'm going to take pictures of all of the settings and provide an MLV, DNG Sequence, and XQ file. I'm using MLV App. The only adjustments I'll use are Cinelog-C and lens vignette correction. I won't touch any settings in MLV App. The DNG sequence will be lossless. I'll post here when I have the files ready.

- Paul

Danne · « **Reply #23 on:** October 30, 2019, 01:11:27 PM »

Quote from: PaulHarwood856 on October 30, 2019, 12:19:38 PM

Hello everyone,

Forgot to mention I only recorded in 12 bit lossless, and forgot to test 14 bit. I'm going to take pictures of all of the settings and provide an MLV, DNG Sequence, and XQ file. I'm using MLV App. The only adjustments I'll use are Cinelog-C and lens vignette correction. I won't touch any settings in MLV App. The DNG sequence will be lossless. I'll post here when I have the files ready.

- Paul

12bit compared with 14bit. Same settings otherwise. Most important step.

PaulHarwood856 · « **Reply #24 on:** October 30, 2019, 01:36:16 PM »

Hey Danne,

Thank you for the quick response. Ok I understand. I'm going to record a really long take in 12 bit lossless and 14 bit. I'll try 14 bit lossless and the other bit depths as well.

- Paul

News:

Author Topic: Fixing Focus Dots from Canon 5D Mark III & How To Avoid This Issue In The Future (Read 8721 times)