mlv_dump on steroids

Started by bouncyball, February 16, 2017, 07:10:10 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Danne

That's a nice idea but it would prolong every clip with around two seconds. Also note that recording a darkframe for every clip isn't needed so then maybe it would be useful as a menu option.

Another idea I have is to build a lua script which would record a short darkframe sample of each iso on any specific cam and then the user could use these files in a dark frame storage. Let's say one loads the script and then it ends when all files are recorded instead of manually having to do this.

ilia3101

Yeah, it would be nice as a menu option. A script is a good idea, but I think noise pattern might be different at different shutter speeds, probably a good approximation though. Does anyone else think it's worth adding this as an option?

bouncyball

I don't think that electronic rolling shutter can be turned off and sensor still be scanned and data read out still be functional. Otherwise we'd be able to use whole sensor as optical black area. Am I wrong?

If mechanical shutter could be closed in live view that would be another story.

g3gg0

alex and i already discussed that stuff while ago.
main problem is, we maybe could make a 0-exposue readout. this is a BIAS frame iirc.
a DARK frame involves the correct exposure time and ISO (or at least correct exposure and a fitting BIAS frame for the ISO etc)

we could somewhen also move the mirror/shutter down and then expose for a DARK frame,
but currently we do not know how to achieve this.
this is most likely controlled in the MPU which we didnt reverse engineer yet.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

bouncyball

Hi guys!

Some more updates on the 1st post, download link is the same.

regards
bb

Danne

The power of "c". Great work B.

Running time command comparing creation time building the first croprec focus pixel file for eosm:
fmputil
real   0m0.021s
user   0m0.012s
sys   0m0.005s

fpm.sh
real   0m4.717s
user   0m10.848s
sys   0m2.304s

bouncyball

Nice, it's about 225 times faster :)

Kharak

@bouncyball @Danne @a1ex @ g3gg0 @anyone else interested.

Bug Report:

I talked to A1ex about this one, as I feared the Lossless compression in 5D3 had messed up badly on two shots I made, it looks like data corruption or failed compression. To my great relief and anyone else using Lossless MLV, it is not the camera. A1ex thinks it is some kind of filtering or some bug of some sort in the 'mlv_dump on Steroids' causing this.

Quote from: a1ex on August 15, 2017, 07:13:36 PM
It's not underflow (that was my first guess, but then I've compared the DNGs in octave and it's more likely to be some sort of filtering operation). If the mlv_dump on steroids has some option to output unprocessed raw data, you could try that.

JPEG screenshot:
https://mega.nz/#!hAo32YwD!iSr0T0XTJdKZ3FJupV6tApUhk7rcixlr5xde8PW7PJI


DNG:
https://mega.nz/#!IIBSDbBT!r4cwb58zCnbuZwcxcPoXY9rixJ2jr3uok5iFQktgtzQ

MLV:
https://mega.nz/#!lZ4A0SyD!Fl1SxrgbM0XtP7RL8vPwK43q9ePhBkSR2iK1bowCsiw some 500 MB.

The "corruption" is stuck in specific spots, its actually in many different places if you look closely its just much more visible in the "high probable aliasing" of the building. Perhaps Vertical Stripe fix messing up?

Both shots where this messed up are of this same building, but it might just aswell be in other shots I have, I just really noticed it on this one.

Converted with Danne's Batch Script combined with "his" mlv_dump on steroids.

5D3 - 123
June 19th build


Let me know if you want any more info.
once you go raw you never go back

Danne

I tried processing the file over here on windows and I couldn´t reproduce the artefacts in dng 245. Are you using any additional settings in the menu when developing? I just ran it without any adding.

Kharak

No, no additional processing.

And it is reproducable, again and again.

With your Batch Script Menu, I tried outputting Non-Processed Raw (3). But the program crashes.
once you go raw you never go back

Danne

Just tried -p option with and without spaces, in the same folder to another drive etc, without darkframes folder and with. It all worked outputting raw dng files with the setting -p. Strange.

You could try outside the script by:
mlv_dump -p --dng INPUT.MLV


You don´t have any darkframes in your folder? Could you try disbling vertical stripes fix. Also disabling cold pixel fix.

Kharak


´´It is not the Darkframes causing it, it is the combination of Cold Pixel Fix with Darkframes causing this corruption. First tried disabling Vertical Stripe fix with DF's and then with --no-fixcp and now it works with Darkframes.

Thank you Danne and A1ex!
once you go raw you never go back

Danne


a1ex

Can you upload a dark frame for this one, so I can add it to the test suite?

Kharak

Yes, will update this post when done.

Edit:

DF MLV straight out of camera:
https://mega.nz/#!dAg0jQjZ!bs2kX3TwL5UD-kXL7_fCgRQI_nC7-eFwLYD0l6gYKFY ISO 100 216 MB


Averaged DF via Mlv dump on Steroids:
https://mega.nz/#!1AgkwAwR!J0-1xq7emM9-2UFIL4Kk0NMtKMxcrK0Gemow6vmR7F8 2.6 MB
once you go raw you never go back

bouncyball

@kharak

Switch uses both mlv_dump versions, for simplicity me and Danne are calling them: ml-dng and steroid version :).
Can you be more specific about version you used? Steroid supports two interpolation methods for bad pixel removal - from mlvfs (more like cr2hdr) and from raw2dng which is used by standard mlv_dump.

Try both in conjunction with dark frame:

mlv_dump -bpi 0
mlv_dump -bpi 1

I'm really curious if there will be difference.

thanx
bb

Danne

Actually, he is using batch_mlv which uses your steroid version only named mlv_dump  :P
I should rename it. Anyway. Interpolation method tests are important. I havn't inserted this to batch_mlv but as soon as Kharak uploads his darkframe shot we could check this.

bouncyball

Ah well, never mind. I tried it myself with your posted darkframe sample. Either bad pixel removal method causes artifacts, different ones thought. Can't say which one is better :P both are ugly in it's own way.

I guess if dark/flat frame is used there is no need to fix badpixels or stripes after that.

bb

bouncyball

@Danne: ah I forgot that Kharak uses windows :)

Danne

I just ran the provided darkframe and matching MLV file from Kharak through my version of mlv_dump in Switch and it was not affected by any corruption in file nr 245. My ml-dng version should be closer to unified mlv_dump. I did not disable cold pixel fix.
https://bitbucket.org/Dannephoto/magic-lantern/branch/ml-dng-unified_11b
I could reproduce the corruption with mlv_dump_on_steroids in Switch.

Not affected version of mlv_dump:
terminal output:(beginning)
MLV Dumper
-----------------

Mode of operation:
   - Input MLV file: 'M31-1527.MLV'
   - Decompressing before writing DNG
   - Enforcing 14bpp for DNG output
   - Convert to DNG frames
   - Subtract reference frame '../avg_14bit_EOS5DMarkIII_res_1920x800_iso_100_fps_59.976000.MLV'
   - Output into 'M31-1527_1_2017-07-31_0001_C0000_'
File M31-1527.MLV opened
File M31-1527.M00 not existing.
Loading subtract (dark) frame '../avg_14bit_EOS5DMarkIII_res_1920x800_iso_100_fps_59.976000.MLV'



terminal output(ending):
Processing...

Vertical stripes correction:
  1.000  1.000  1.004  1.003  0.998  1.001  1.000  1.003

Frame0 : cold pixels found: 0                             

Reached end of chunk 1/1 after 367 blocks
Processed 348 video frames at 59.98 FPS (5.80 s)
Done

bouncyball

Hmm... It says: "Frame0 : cold pixels found: 0"  this equals to not touching data and not fixing anything hence no artifacts.

Now I understand better what's happening. After dark frame subtraction, MLVFS/cr2hdr bad pixels scaning code tries to find all flavors of bad pixels eg: cold, hot, dead...

raw2dng code is for cold pixel revealing only. In steroid for revealing bad pixels I've used only advanced scaning code from MLVFS but for interpolation both methods are available trough the switches.

bb

Danne

For reliability sake, all pixel fix methods should be default set to off. When pixel peepers see dead, cold, hot things in their dng files they should reprocess with correct pixel method set.
Otherwise we are gonna keep on narrowing down these little never ending mysteries for a long time :)

Kharak

I think I have only ever seen the Cold Pixel Counter say 1, once. Are Cold Pixels usually an issue?

How does a Cold Pixel look like? Is it a dead pixel e.g. Black?
once you go raw you never go back

bouncyball

I guess it's more like dead pixel and is nearly black. Someone correct me please :)

Danne