Author Topic: mlv_dump on steroids (Read 69821 times)

Danne · « **Reply #100 on:** July 05, 2017, 10:09:05 AM »

That's a nice idea but it would prolong every clip with around two seconds. Also note that recording a darkframe for every clip isn't needed so then maybe it would be useful as a menu option.

Another idea I have is to build a lua script which would record a short darkframe sample of each iso on any specific cam and then the user could use these files in a dark frame storage. Let's say one loads the script and then it ends when all files are recorded instead of manually having to do this.

ilia3101 · « **Reply #101 on:** July 05, 2017, 10:18:03 AM »

Yeah, it would be nice as a menu option. A script is a good idea, but I think noise pattern might be different at different shutter speeds, probably a good approximation though. Does anyone else think it's worth adding this as an option?

bouncyball · « **Reply #102 on:** July 05, 2017, 10:44:10 AM »

I don't think that electronic rolling shutter can be turned off and sensor still be scanned and data read out still be functional. Otherwise we'd be able to use whole sensor as optical black area. Am I wrong?

If mechanical shutter could be closed in live view that would be another story.

g3gg0 · « **Reply #103 on:** July 05, 2017, 01:47:31 PM »

alex and i already discussed that stuff while ago.
main problem is, we maybe could make a 0-exposue readout. this is a BIAS frame iirc.
a DARK frame involves the correct exposure time and ISO (or at least correct exposure and a fitting BIAS frame for the ISO etc)

we could somewhen also move the mirror/shutter down and then expose for a DARK frame,
but currently we do not know how to achieve this.
this is most likely controlled in the MPU which we didnt reverse engineer yet.

bouncyball · « **Reply #104 on:** July 25, 2017, 07:27:18 PM »

Hi guys!

Some more updates on the 1st post, download link is the same.

regards
bb

Danne · « **Reply #105 on:** July 25, 2017, 10:46:25 PM »

The power of "c". Great work B.

Running time command comparing creation time building the first croprec focus pixel file for eosm:
fmputil
real 0m0.021s
user 0m0.012s
sys 0m0.005s

fpm.sh
real 0m4.717s
user 0m10.848s
sys 0m2.304s

bouncyball · « **Reply #106 on:** July 26, 2017, 12:21:04 PM »

Nice, it's about 225 times faster

Kharak · « **Reply #107 on:** August 16, 2017, 09:39:56 PM »

@bouncyball @Danne @a1ex @ g3gg0 @anyone else interested.

Bug Report:

I talked to A1ex about this one, as I feared the Lossless compression in 5D3 had messed up badly on two shots I made, it looks like data corruption or failed compression. To my great relief and anyone else using Lossless MLV, it is not the camera. A1ex thinks it is some kind of filtering or some bug of some sort in the 'mlv_dump on Steroids' causing this.

Quote from: a1ex on August 15, 2017, 07:13:36 PM

It's not underflow (that was my first guess, but then I've compared the DNGs in octave and it's more likely to be some sort of filtering operation). If the mlv_dump on steroids has some option to output unprocessed raw data, you could try that.

JPEG screenshot:
https://mega.nz/#!hAo32YwD!iSr0T0XTJdKZ3FJupV6tApUhk7rcixlr5xde8PW7PJI

DNG:
https://mega.nz/#!IIBSDbBT!r4cwb58zCnbuZwcxcPoXY9rixJ2jr3uok5iFQktgtzQ

MLV:
https://mega.nz/#!lZ4A0SyD!Fl1SxrgbM0XtP7RL8vPwK43q9ePhBkSR2iK1bowCsiw some 500 MB.

The "corruption" is stuck in specific spots, its actually in many different places if you look closely its just much more visible in the "high probable aliasing" of the building. Perhaps Vertical Stripe fix messing up?

Both shots where this messed up are of this same building, but it might just aswell be in other shots I have, I just really noticed it on this one.

Converted with Danne's Batch Script combined with "his" mlv_dump on steroids.

5D3 - 123
June 19th build

Let me know if you want any more info.

Danne · « **Reply #108 on:** August 16, 2017, 09:56:42 PM »

I tried processing the file over here on windows and I couldn´t reproduce the artefacts in dng 245. Are you using any additional settings in the menu when developing? I just ran it without any adding.

Kharak · « **Reply #109 on:** August 16, 2017, 10:01:04 PM »

No, no additional processing.

And it is reproducable, again and again.

With your Batch Script Menu, I tried outputting Non-Processed Raw (3). But the program crashes.

Danne · « **Reply #110 on:** August 16, 2017, 10:16:46 PM »

Just tried -p option with and without spaces, in the same folder to another drive etc, without darkframes folder and with. It all worked outputting raw dng files with the setting -p. Strange.

You could try outside the script by:

Code: [Select]

mlv_dump -p --dng INPUT.MLV

You don´t have any darkframes in your folder? Could you try disbling vertical stripes fix. Also disabling cold pixel fix.

Kharak · « **Reply #111 on:** August 17, 2017, 12:02:23 AM »

´´It is not the Darkframes causing it, it is the combination of Cold Pixel Fix with Darkframes causing this corruption. First tried disabling Vertical Stripe fix with DF's and then with --no-fixcp and now it works with Darkframes.

Thank you Danne and A1ex!

Danne · « **Reply #112 on:** August 17, 2017, 01:04:29 AM »

Nice catch Kharak.

a1ex · « **Reply #113 on:** August 17, 2017, 01:09:31 AM »

Can you upload a dark frame for this one, so I can add it to the test suite?

Kharak · « **Reply #114 on:** August 17, 2017, 09:08:38 AM »

Yes, will update this post when done.

Edit:

DF MLV straight out of camera:
https://mega.nz/#!dAg0jQjZ!bs2kX3TwL5UD-kXL7_fCgRQI_nC7-eFwLYD0l6gYKFY ISO 100 216 MB

Averaged DF via Mlv dump on Steroids:
https://mega.nz/#!1AgkwAwR!J0-1xq7emM9-2UFIL4Kk0NMtKMxcrK0Gemow6vmR7F8 2.6 MB

bouncyball · « **Reply #115 on:** August 17, 2017, 10:17:20 AM »

@kharak

Switch uses both mlv_dump versions, for simplicity me and Danne are calling them: ml-dng and steroid version

.
Can you be more specific about version you used? Steroid supports two interpolation methods for bad pixel removal - from mlvfs (more like cr2hdr) and from raw2dng which is used by standard mlv_dump.

Try both in conjunction with dark frame:

Code: [Select]

mlv_dump -bpi 0 
mlv_dump -bpi 1

I'm really curious if there will be difference.

thanx
bb

Danne · « **Reply #116 on:** August 17, 2017, 10:43:10 AM »

Actually, he is using batch_mlv which uses your steroid version only named mlv_dump

I should rename it. Anyway. Interpolation method tests are important. I havn't inserted this to batch_mlv but as soon as Kharak uploads his darkframe shot we could check this.

bouncyball · « **Reply #117 on:** August 17, 2017, 10:46:25 AM »

Ah well, never mind. I tried it myself with your posted darkframe sample. Either bad pixel removal method causes artifacts, different ones thought. Can't say which one is better

both are ugly in it's own way.

I guess if dark/flat frame is used there is no need to fix badpixels or stripes after that.

bb

bouncyball · « **Reply #118 on:** August 17, 2017, 10:54:00 AM »

@Danne: ah I forgot that Kharak uses windows

Danne · « **Reply #119 on:** August 17, 2017, 01:34:49 PM »

I just ran the provided darkframe and matching MLV file from Kharak through my version of mlv_dump in Switch and it was not affected by any corruption in file nr 245. My ml-dng version should be closer to unified mlv_dump. I did not disable cold pixel fix.
https://bitbucket.org/Dannephoto/magic-lantern/branch/ml-dng-unified_11b
I could reproduce the corruption with mlv_dump_on_steroids in Switch.

Not affected version of mlv_dump:
terminal output:(beginning)

Code: [Select]

MLV Dumper
-----------------

Mode of operation:
   - Input MLV file: 'M31-1527.MLV'
   - Decompressing before writing DNG
   - Enforcing 14bpp for DNG output
   - Convert to DNG frames
   - Subtract reference frame '../avg_14bit_EOS5DMarkIII_res_1920x800_iso_100_fps_59.976000.MLV'
   - Output into 'M31-1527_1_2017-07-31_0001_C0000_'
File M31-1527.MLV opened
File M31-1527.M00 not existing.
Loading subtract (dark) frame '../avg_14bit_EOS5DMarkIII_res_1920x800_iso_100_fps_59.976000.MLV'

terminal output(ending):

Code: [Select]

Processing...

Vertical stripes correction:
  1.000  1.000  1.004  1.003  0.998  1.001  1.000  1.003

Frame0 : cold pixels found: 0                             

Reached end of chunk 1/1 after 367 blocks
Processed 348 video frames at 59.98 FPS (5.80 s)
Done

bouncyball · « **Reply #120 on:** August 17, 2017, 05:26:17 PM »

Hmm... It says: "Frame0 : cold pixels found: 0" this equals to not touching data and not fixing anything hence no artifacts.

Now I understand better what's happening. After dark frame subtraction, MLVFS/cr2hdr bad pixels scaning code tries to find all flavors of bad pixels eg: cold, hot, dead...

raw2dng code is for cold pixel revealing only. In steroid for revealing bad pixels I've used only advanced scaning code from MLVFS but for interpolation both methods are available trough the switches.

bb

Danne · « **Reply #121 on:** August 17, 2017, 07:30:39 PM »

For reliability sake, all pixel fix methods should be default set to off. When pixel peepers see dead, cold, hot things in their dng files they should reprocess with correct pixel method set.
Otherwise we are gonna keep on narrowing down these little never ending mysteries for a long time

Kharak · « **Reply #122 on:** August 17, 2017, 09:56:31 PM »

I think I have only ever seen the Cold Pixel Counter say 1, once. Are Cold Pixels usually an issue?

How does a Cold Pixel look like? Is it a dead pixel e.g. Black?

bouncyball · « **Reply #123 on:** August 18, 2017, 11:11:30 AM »

I guess it's more like dead pixel and is nearly black. Someone correct me please

Danne · « **Reply #124 on:** August 18, 2017, 12:36:21 PM »

Think they can come in different colors.
http://www.magiclantern.fm/forum/index.php?topic=17795.msg171191#msg171191

News:

Author Topic: mlv_dump on steroids (Read 69821 times)