I've not tried dark frame subtraction before so I just gave it a go with my 6D at 1824x1026 14bit lossless and MVL App 1.13 but when subtracting a dark frame using a short (8-9s) uncompressed video I recorded with the lens cap on I get a sea of bright fully saturated pixels in dark areas with sharp details.
Actually I still get the bright saturated noise without dark frame subtraction just not as bad and even with raw processing completely disabled.
Also with most of the debayer algorithms the bright pixels are black but with AMaZE or AHD its really overpowering!
It only seems to happen at ISO 6400 and above.
Bright saturated noise, sounds like hot pixels ?
MLV app has option to fix bad pixels.
BUT, what helped me a lot to get rid of most of those hot pixels is to do a dead/hot pixel remap in camera.
you can google it, canon pixel remapping.
How it's done.
-Remove lens and put on body cap.
-Turn on camera and to get best results, enable liveview and set it to iso 6400 and leave it that way for a minute or more so the sensor becomes noisy.
-Now do a sensor cleaning in Canon menu (almost in the latest tab of canon menu, 'sensor cleaning' and choose 'clean now'.
After this, you should have much less hot pixels.
Darkframe subtraction is mostly helpful to remove vertical line noise in raw video.
But there is even a better way for 6d.
For low light, you should use LV raw type 0x12 (only works for normal raw recording, not working for high resolution crop module modes)
Not sure if you have a build where you can alter 'LV raw type', it should be available on top in the DEBUG tab of ML menu.
Debug tab -> LV raw type option
Default LV raw type = 0x10
For low light, set it to 0x12, highlight LV raw type option, click set button, use cursor keys to set cursor below the 0 from 10 value, and use scroll wheel on top of DSLR to set it to value 2, so you get 12.
Now use set button to exit and you should have LV raw type value of 0x12
Now shoot a scene in low light, in this mode, even iso 25600 looks great.
Every time the camera is turned off, the LV raw type is set to default 0x10 value.
If you don't have the LV raw type option in DEBUG tab, try the build in this post:
https://www.magiclantern.fm/forum/index.php?topic=25782.msg228784#msg228784 (https://www.magiclantern.fm/forum/index.php?topic=25782.msg228784#msg228784)
Is this raw type option only exist for 6D? 25600 iso sounds crazy :o :D
No, different LV raw types exist on all cameras.
It's a debug options, so disabled in most ML builds.
In some 6d builds, I have the option enabled and available in Debug menu.
Lv raw types are data streams of the live view raw data. There are a lot of raw type data streams, some are more altered then the other.
There are raw types which contain purple pixels (dead/hot pixels), while on other the dead/hot pixels are fixed and not present.
So it seems that canon does some fixing on the raw data stream.
The difference between 0x10 and 0x12 on the 6d:
0x10 -> can be used with high resolution crop rec options and 0x10 has highest dynamic range.
In high iso and low light, vertical line noise becomes visible in image.
0x12 -> only works for normal available resolutions, doesn't work with the altered crop rec module resolutions.
Dynamic range is slightly lower ( 0.5 stop ). No vertical line noise in high iso low light situations.
The above difference means that canon is doing some dark frame correction in camera on raw type 0x12. Explains the lower dynamic range and missing vertical line noise.
Not sure if the above applies to other camera (lv raw type 0x10 vs 0x12).
Thanks. I did some search on the forum and found other mentions, but it seems that nobody implemented this on other cameras at least yet. And I wonder why, because it could be very nice option to use in low light. Maybe on other cameras it's just different and actually working low noise raw type it is exclusive to 6D..
It's not difficult to test yourself, all you need is a build for your camera, where the option is enabled.
You can ask in the forum of your camera if someone can make a build where it is enabled.
The option is in the 'raw.c' file, located in the 'src' folder in the source files.
For example in the source on magic lantern page:
https://foss.heptapod.net/magic-lantern/magic-lantern/-/blob/branch/unified/src/raw.c (https://foss.heptapod.net/magic-lantern/magic-lantern/-/blob/branch/unified/src/raw.c)
In this example, it's on code line 37
#undef RAW_DEBUG_TYPE /* this lets you select the raw type (for PREFERRED_RAW_TYPE) from menu */
All someone has to do is change the 'undef' to 'define' and you will have an option in debug menu to alter the raw type value which will be used for raw video recording.
So the line has to be changed to:
#define RAW_DEBUG_TYPE /* this lets you select the raw type (for PREFERRED_RAW_TYPE) from menu */
There are a lot of raw types value to choose from, some might freeze your camera, but in that case, pull out battery and try again : ;D
There aren't that many clean raw type streams, most contain the purple pixels.
I think for Digic V an Digic VI the values are the same, so I expect default is 0x10 on other cams and first thing to try would be value 0x12.
Then do some comparison in low light high iso (6400 or higher) and see if there is any difference.
Edit: you can't see changes to raw type in canon liveview, but it will do show up in ML preview, the slow low resolution preview option available in raw video menu.
ok, sounds very interesting :) I'll ask in 5D2 thread since this is the camera I'm using.. thank you for very interesting suggestion. I really like night scenes, but never go beyond iso 800 in raw, maybe 1600 sometimes but it is already too noisy.
@CommanderLake: no idea how to help with this few information. Maybe you should upload your darkframe and a frame of the clip which brings up the problem. So we could analyse and help.
Quote from: Levas on March 14, 2022, 08:54:07 AM
It's not difficult to test yourself, all you need is a build for your camera, where the option is enabled.
Has anyone compiled this for the 5d3 (113)?
@IDA_ML
Noticed that there is not much reaction to make a build with raw type enabled.
So compiled one for 5d3 - 113.
Not sure if it works, it's on your own risk, can't test it on a 5d3.
Also made a 6d build from the same source and tested it on my 6d, works and with raw_type selection in debug tab.
(sourcecode is "crop_rec_4k_mlv_snd_isogain_1x3_presets"
Couldn't figure out how to compile without crop_rec module and sd_uhs module.
So I unpacked the zip file, uploaded the complete folder to google drive and removed the sd_uhs module and crop_rec module manually.
(crop_rec and sd_uhs modules in this build are made for 6d, not sure if they work on 5d3, so removed them)
Since this build is based on the crop_rec_4k source, you can of course use crop_rec modules from other builds with this build, just copy the crop_rec.mo file from another build in the ML/module/ directory
As you know, if it breaks you get to keep both pieces :P
https://drive.google.com/drive/folders/10a3g3M_b5n8DufDAaZMagcWsAXN8c_14?usp=sharing (https://drive.google.com/drive/folders/10a3g3M_b5n8DufDAaZMagcWsAXN8c_14?usp=sharing)
For quick test to see If it makes any difference for vertical line noise.
Enable the following modules first time you start the camera with this build:
-mlv_lite
-mlv_play
-mlv_snd
Restart camera
Go to debug tab in ML menu and check LV raw type is 0x10
Now set iso to 25600 and shoot some ML raw video in low light.
After that, set LV raw type to value 0x12 and shoot another raw video in low light.
Curious, see any difference on 5d3 ?
@Skinny
Tried to compile a same build for 5d2, but didn't succeed, error messages and no build.
Probably because the source is the "crop_rec_4k_mlv_snd" branch, where the 5d2 is not fully supported ???
I've just learn to compile for 700D from bilal's source code here (https://www.magiclantern.fm/forum/index.php?topic=25784.0)
I have tested it and works.
download: 700D-DEFINE-RAW-DEBUG-TYPE.zip (https://drive.google.com/file/d/1lneaJNzmjPa5r0DirJcIhTQry6DE04ax/view?usp=sharing)
Quote from: IDA_ML on March 15, 2022, 04:46:10 PM
Has anyone compiled this for the 5d3 (113)?
I use Danne's source code (https://bitbucket.org/Dannephoto/magic-lantern_dannephoto_git/src/master/)
here I tried to compile it for you Dannephoto-5d3.113-define_raw_debug_type.zip (https://drive.google.com/file/d/1dphj2v8e_m9TqFnGelDQ5T--ZIow5xgV/view?usp=sharing)
copy *.mo files from your ML/modules (except 5d3_113.sym) and ML/script from your 5d3.113 build to the same folder in this build
I hope it helps.
Quote from: Levas on March 14, 2022, 08:54:07 AM
It's not difficult to test yourself, all you need is a build for your camera, where the option is enabled.
You can ask in the forum of your camera if someone can make a build where it is enabled.
The option is in the 'raw.c' file, located in the 'src' folder in the source files.
For example in the source on magic lantern page:
https://foss.heptapod.net/magic-lantern/magic-lantern/-/blob/branch/unified/src/raw.c (https://foss.heptapod.net/magic-lantern/magic-lantern/-/blob/branch/unified/src/raw.c)
In this example, it's on code line 37
#undef RAW_DEBUG_TYPE /* this lets you select the raw type (for PREFERRED_RAW_TYPE) from menu */
All someone has to do is change the 'undef' to 'define' and you will have an option in debug menu to alter the raw type value which will be used for raw video recording.
So the line has to be changed to:
#define RAW_DEBUG_TYPE /* this lets you select the raw type (for PREFERRED_RAW_TYPE) from menu */
There are a lot of raw types value to choose from, some might freeze your camera, but in that case, pull out battery and try again : ;D
There aren't that many clean raw type streams, most contain the purple pixels.
I think for Digic V an Digic VI the values are the same, so I expect default is 0x10 on other cams and first thing to try would be value 0x12.
Then do some comparison in low light high iso (6400 or higher) and see if there is any difference.
Edit: you can't see changes to raw type in canon liveview, but it will do show up in ML preview, the slow low resolution preview option available in raw video menu.
Thanks for the idea..
Curious, can you see any difference in vertical line noise between raw type 0x10 and 0x12 on the 700d ?
I didn't see any vertical line but horizontal line
I will try again later and see if there is any difference.
Here are some test 0x10 and 0x12 at ISO 12800
0x10 using binning mode 1x3 looks greenish with striped like horizontal line
(https://i.ibb.co/j36CBpf/Whats-App-Image-2022-03-17-at-10-00-52-PM.jpg)
0x10 crop_rec OFF ISO 12800
(https://i.ibb.co/5YYRXVK/Whats-App-Image-2022-03-17-at-10-01-48-PM.jpg)
change to 0x12
(https://i.ibb.co/tx6qxBF/Whats-App-Image-2022-03-17-at-10-03-45-PM.jpg)
0x12 ISO 12800
(https://i.ibb.co/bNmF19B/Whats-App-Image-2022-03-17-at-10-04-21-PM.jpg)
I didn't see any different so far.. except using binning 1x3 with iso 12800 got very dark image
Quote from: Levas on March 16, 2022, 08:00:43 PM
@Skinny
Tried to compile a same build for 5d2, but didn't succeed, error messages and no build.
Probably because the source is the "crop_rec_4k_mlv_snd" branch, where the 5d2 is not fully supported ???
@Levas thank you anyway :)
@ML700D but.. is this a standart canon preview? what about actual recorded raw footage?
Quote from: Levas on March 14, 2022, 08:54:07 AM
Edit: you can't see changes to raw type in canon liveview, but it will do show up in ML preview, the slow low resolution preview option available in raw video menu.
Quote from: Skinny on March 17, 2022, 05:20:08 PM
@Levas thank you anyway :)
@ML700D but.. is this a standart canon preview? what about actual recorded raw footage?
no, that is ML live view, the footages are both same when open in mlvapp and mlv_play afaik.
Quote from: Levas on March 14, 2022, 08:54:07 AM
Edit: you can't see changes to raw type in canon liveview, but it will do show up in ML preview, the slow low resolution preview option available in raw video menu.
I will try ss from mlv_play later.. sorry.
Quote from: ML700D on March 17, 2022, 04:32:17 PM
Here are some test 0x10 and 0x12 at ISO 12800
0x10 using binning mode 1x3 looks greenish with striped like horizontal line
...
Not related to RAW type, these artifact happen because we are tweaking preview registers in 1x3 modes (also in other presets with correct real-time preview). These artifact
only affects LiveView but not the recorded RAW data, in lower bit-depths it becomes even more visible.
BTW, on 700D the maximum analog ISO is 3200. Both 6400 and 12800 ISOs are digital.
I see, thanks bilal
btw have you ever try this 0x12 things?
I have found this thread and wanted also to contribute and/or ask what went wrong. For the first I would just share two screenshots and you can decide if you really need the footage, too.
https://drive.google.com/drive/folders/1fFFIf96Lh9A9hcOaG3PYxu7u0dXWTJbM?usp=sharing
Especially on the back of the black pullover its in the after screenshot so much worse. But I can definetly see a change and am not sure if its to the better or the worse :D
In this case the after is worse.
The after has the vertical lines which have that Magic lantern high iso raw fingerprint :P
Dark frame subtraction should solve the vertical line issues.
So if you haven't mistaken before and after...then it looks like your dark frame is not made with the same settings as the shot you extracted it from.
All settings must be equal, resolution settings in ML, iso setting and shutter time setting. Aperture shouldn't make a difference, I always close it as much as I can for dark frame creation.
It looks like your dark frame are recorded in different resolution or other iso setting.
The MLVapp states if the resolutions are different and doesnt allow you to use wrong darkframe video. Iso is definetly equal because I have looked it up in the info tab and I never change shutterspeed but it should be the same. And yeah I hope that the aperture doesnt make a difference but I can try to make one with the most closed down just for testing. I will try to upload the source files and link them two that you can try it out for yourself and compare what went wrong :)
So the dark frame was recorded in different conditions, after some time? Sensor temperature can make a lot of difference in the noise and patterns.. dark frame should be shot as close as possible to the original conditions. And not only things like themperature, but even battery charge will probably affect some noise patterns. But I don't believe the aperture will make any difference...
Oh okay, yeah the dark frame was definetly recorded at a totally different day at totally different temperature. With the battery I am not sure because I let it run all the time from a dummy battery adapter and power directly from a powerbank.
To be honest I hoped I would record once the dark frames for various settings and have them "forever" in the folders for future darkframe substractions and to not record every single time a darkframe video for every single shot I take
Should work better even if the darkfram was recorded afterwards. Subtraction works best with 14bit and maybe ok with 12bit files. 10bit files not good imo.
@cannibalferox you can try "pre-heating" your camera, let it run for a few minutes with high iso recording video.. then record mlv for dark frames. maybe it will be close enough..