Author Topic: mlv_dump on steroids  (Read 15516 times)

Danne

  • Hero Member
  • *****
  • Posts: 3480
Re: mlv_dump on steroids
« Reply #100 on: July 05, 2017, 10:09:05 AM »
That's a nice idea but it would prolong every clip with around two seconds. Also note that recording a darkframe for every clip isn't needed so then maybe it would be useful as a menu option.

Another idea I have is to build a lua script which would record a short darkframe sample of each iso on any specific cam and then the user could use these files in a dark frame storage. Let's say one loads the script and then it ends when all files are recorded instead of manually having to do this.

Ilia3101

  • Member
  • ***
  • Posts: 236
Re: mlv_dump on steroids
« Reply #101 on: July 05, 2017, 10:18:03 AM »
Yeah, it would be nice as a menu option. A script is a good idea, but I think noise pattern might be different at different shutter speeds, probably a good approximation though. Does anyone else think it's worth adding this as an option?
5D2

bouncyball

  • Senior
  • ****
  • Posts: 362
Re: mlv_dump on steroids
« Reply #102 on: July 05, 2017, 10:44:10 AM »
I don't think that electronic rolling shutter can be turned off and sensor still be scanned and data read out still be functional. Otherwise we'd be able to use whole sensor as optical black area. Am I wrong?

If mechanical shutter could be closed in live view that would be another story.

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3052
Re: mlv_dump on steroids
« Reply #103 on: July 05, 2017, 01:47:31 PM »
alex and i already discussed that stuff while ago.
main problem is, we maybe could make a 0-exposue readout. this is a BIAS frame iirc.
a DARK frame involves the correct exposure time and ISO (or at least correct exposure and a fitting BIAS frame for the ISO etc)

we could somewhen also move the mirror/shutter down and then expose for a DARK frame,
but currently we do not know how to achieve this.
this is most likely controlled in the MPU which we didnt reverse engineer yet.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

bouncyball

  • Senior
  • ****
  • Posts: 362
Re: mlv_dump on steroids
« Reply #104 on: July 25, 2017, 07:27:18 PM »
Hi guys!

Some more updates on the 1st post, download link is the same.

regards
bb

Danne

  • Hero Member
  • *****
  • Posts: 3480
Re: mlv_dump on steroids
« Reply #105 on: July 25, 2017, 10:46:25 PM »
The power of "c". Great work B.

Running time command comparing creation time building the first croprec focus pixel file for eosm:
fmputil
real   0m0.021s
user   0m0.012s
sys   0m0.005s

fpm.sh
real   0m4.717s
user   0m10.848s
sys   0m2.304s

bouncyball

  • Senior
  • ****
  • Posts: 362
Re: mlv_dump on steroids
« Reply #106 on: July 26, 2017, 12:21:04 PM »
Nice, it's about 225 times faster :)

Kharak

  • Hero Member
  • *****
  • Posts: 708
Re: mlv_dump on steroids
« Reply #107 on: August 16, 2017, 09:39:56 PM »
@bouncyball @Danne @a1ex @ g3gg0 @anyone else interested.

Bug Report:

I talked to A1ex about this one, as I feared the Lossless compression in 5D3 had messed up badly on two shots I made, it looks like data corruption or failed compression. To my great relief and anyone else using Lossless MLV, it is not the camera. A1ex thinks it is some kind of filtering or some bug of some sort in the 'mlv_dump on Steroids' causing this.

It's not underflow (that was my first guess, but then I've compared the DNGs in octave and it's more likely to be some sort of filtering operation). If the mlv_dump on steroids has some option to output unprocessed raw data, you could try that.

JPEG screenshot:
https://mega.nz/#!hAo32YwD!iSr0T0XTJdKZ3FJupV6tApUhk7rcixlr5xde8PW7PJI


DNG:
https://mega.nz/#!IIBSDbBT!r4cwb58zCnbuZwcxcPoXY9rixJ2jr3uok5iFQktgtzQ

MLV:
https://mega.nz/#!lZ4A0SyD!Fl1SxrgbM0XtP7RL8vPwK43q9ePhBkSR2iK1bowCsiw some 500 MB.

The "corruption" is stuck in specific spots, its actually in many different places if you look closely its just much more visible in the "high probable aliasing" of the building. Perhaps Vertical Stripe fix messing up?

Both shots where this messed up are of this same building, but it might just aswell be in other shots I have, I just really noticed it on this one.

Converted with Danne's Batch Script combined with "his" mlv_dump on steroids.

5D3 - 123
June 19th build


Let me know if you want any more info.
once you go raw you never go back

Danne

  • Hero Member
  • *****
  • Posts: 3480
Re: mlv_dump on steroids
« Reply #108 on: August 16, 2017, 09:56:42 PM »
I tried processing the file over here on windows and I couldn´t reproduce the artefacts in dng 245. Are you using any additional settings in the menu when developing? I just ran it without any adding.

Kharak

  • Hero Member
  • *****
  • Posts: 708
Re: mlv_dump on steroids
« Reply #109 on: August 16, 2017, 10:01:04 PM »
No, no additional processing.

And it is reproducable, again and again.

With your Batch Script Menu, I tried outputting Non-Processed Raw (3). But the program crashes.
once you go raw you never go back

Danne

  • Hero Member
  • *****
  • Posts: 3480
Re: mlv_dump on steroids
« Reply #110 on: August 16, 2017, 10:16:46 PM »
Just tried -p option with and without spaces, in the same folder to another drive etc, without darkframes folder and with. It all worked outputting raw dng files with the setting -p. Strange.

You could try outside the script by:
Code: [Select]
mlv_dump -p --dng INPUT.MLV

You don´t have any darkframes in your folder? Could you try disbling vertical stripes fix. Also disabling cold pixel fix.

Kharak

  • Hero Member
  • *****
  • Posts: 708
Re: mlv_dump on steroids
« Reply #111 on: August 17, 2017, 12:02:23 AM »
I tried without Darkframe Subtraction and the corruption is gone. So the Darkframe Subtraction is causing this weird corrupt splotches being stuck across the image.

So what do I take from this?

Are my Darkframes not good enough for this shot? I was thinking as I made these DF's from 100 -128000 that the sensor temerature, probably was not matching the normal sensor temperatures it normally operates under. It was about 28-30 C when I was recording the DF's, definitely 30 when I was done with ISO 12800. Especially after a1ex had mentioned, that ideally one should make DF's before and/or after every shot. The Dark Current most likely behaves a lot more differently when the Sensor temperature is in the 40 to 50 C. The outside temperature was not hot, about 19-20, the Sun had a punch to it, but I really don't think I was in the 50's. I rarely get that high temperatures on the 5D3, only after shooting for longer periods in high ISO's. The 5D2 ends up there all the time and Hot pixels start emerging.

Looking at the previous shots, I estimate I recorded about 10 shots, not really consecutively, before this one. We were driving in a car through Reykjavik shooting whatever caught my eye, of which I had some breaks between, not running non-stop. I probably turned of the camera aswell between some shots as I always try to conserve battery. If this information is of any help.


Edit: It is not the Darkframes causing it, it is the combination of Cold Pixel Fix with Darkframes causing this corruption. First tried disabling Vertical Stripe fix with DF's and then with --no-fixcp and now it works with Darkframes.

Thank you Danne and A1ex!
once you go raw you never go back

Danne

  • Hero Member
  • *****
  • Posts: 3480
Re: mlv_dump on steroids
« Reply #112 on: August 17, 2017, 01:04:29 AM »
Nice catch Kharak.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10456
  • 5D Mark Free
Re: mlv_dump on steroids
« Reply #113 on: August 17, 2017, 01:09:31 AM »
Can you upload a dark frame for this one, so I can add it to the test suite?

Kharak

  • Hero Member
  • *****
  • Posts: 708
Re: mlv_dump on steroids
« Reply #114 on: August 17, 2017, 09:08:38 AM »
Yes, will update this post when done.

Edit:

DF MLV straight out of camera:
https://mega.nz/#!dAg0jQjZ!bs2kX3TwL5UD-kXL7_fCgRQI_nC7-eFwLYD0l6gYKFY ISO 100 216 MB


Averaged DF via Mlv dump on Steroids:
https://mega.nz/#!1AgkwAwR!J0-1xq7emM9-2UFIL4Kk0NMtKMxcrK0Gemow6vmR7F8 2.6 MB
once you go raw you never go back

bouncyball

  • Senior
  • ****
  • Posts: 362
Re: mlv_dump on steroids
« Reply #115 on: August 17, 2017, 10:17:20 AM »
@kharak

Switch uses both mlv_dump versions, for simplicity me and Danne are calling them: ml-dng and steroid version :).
Can you be more specific about version you used? Steroid supports two interpolation methods for bad pixel removal - from mlvfs (more like cr2hdr) and from raw2dng which is used by standard mlv_dump.

Try both in conjunction with dark frame:
Code: [Select]
mlv_dump -bpi 0
mlv_dump -bpi 1
I'm really curious if there will be difference.

thanx
bb

Danne

  • Hero Member
  • *****
  • Posts: 3480
Re: mlv_dump on steroids
« Reply #116 on: August 17, 2017, 10:43:10 AM »
Actually, he is using batch_mlv which uses your steroid version only named mlv_dump  :P
I should rename it. Anyway. Interpolation method tests are important. I havn't inserted this to batch_mlv but as soon as Kharak uploads his darkframe shot we could check this.

bouncyball

  • Senior
  • ****
  • Posts: 362
Re: mlv_dump on steroids
« Reply #117 on: August 17, 2017, 10:46:25 AM »
Ah well, never mind. I tried it myself with your posted darkframe sample. Either bad pixel removal method causes artifacts, different ones thought. Can't say which one is better :P both are ugly in it's own way.

I guess if dark/flat frame is used there is no need to fix badpixels or stripes after that.

bb

bouncyball

  • Senior
  • ****
  • Posts: 362
Re: mlv_dump on steroids
« Reply #118 on: August 17, 2017, 10:54:00 AM »
@Danne: ah I forgot that Kharak uses windows :)

Danne

  • Hero Member
  • *****
  • Posts: 3480
Re: mlv_dump on steroids
« Reply #119 on: August 17, 2017, 01:34:49 PM »
I just ran the provided darkframe and matching MLV file from Kharak through my version of mlv_dump in Switch and it was not affected by any corruption in file nr 245. My ml-dng version should be closer to unified mlv_dump. I did not disable cold pixel fix.
https://bitbucket.org/Dannephoto/magic-lantern/branch/ml-dng-unified_11b
I could reproduce the corruption with mlv_dump_on_steroids in Switch.

Not affected version of mlv_dump:
terminal output:(beginning)
Code: [Select]
MLV Dumper
-----------------

Mode of operation:
   - Input MLV file: 'M31-1527.MLV'
   - Decompressing before writing DNG
   - Enforcing 14bpp for DNG output
   - Convert to DNG frames
   - Subtract reference frame '../avg_14bit_EOS5DMarkIII_res_1920x800_iso_100_fps_59.976000.MLV'
   - Output into 'M31-1527_1_2017-07-31_0001_C0000_'
File M31-1527.MLV opened
File M31-1527.M00 not existing.
Loading subtract (dark) frame '../avg_14bit_EOS5DMarkIII_res_1920x800_iso_100_fps_59.976000.MLV'


terminal output(ending):
Code: [Select]
Processing...

Vertical stripes correction:
  1.000  1.000  1.004  1.003  0.998  1.001  1.000  1.003

Frame0 : cold pixels found: 0                             

Reached end of chunk 1/1 after 367 blocks
Processed 348 video frames at 59.98 FPS (5.80 s)
Done

bouncyball

  • Senior
  • ****
  • Posts: 362
Re: mlv_dump on steroids
« Reply #120 on: August 17, 2017, 05:26:17 PM »
Hmm... It says: "Frame0 : cold pixels found: 0"  this equals to not touching data and not fixing anything hence no artifacts.

Now I understand better what's happening. After dark frame subtraction, MLVFS/cr2hdr bad pixels scaning code tries to find all flavors of bad pixels eg: cold, hot, dead...

raw2dng code is for cold pixel revealing only. In steroid for revealing bad pixels I've used only advanced scaning code from MLVFS but for interpolation both methods are available trough the switches.

bb

Danne

  • Hero Member
  • *****
  • Posts: 3480
Re: mlv_dump on steroids
« Reply #121 on: August 17, 2017, 07:30:39 PM »
For reliability sake, all pixel fix methods should be default set to off. When pixel peepers see dead, cold, hot things in their dng files they should reprocess with correct pixel method set.
Otherwise we are gonna keep on narrowing down these little never ending mysteries for a long time :)

Kharak

  • Hero Member
  • *****
  • Posts: 708
Re: mlv_dump on steroids
« Reply #122 on: August 17, 2017, 09:56:31 PM »
I think I have only ever seen the Cold Pixel Counter say 1, once. Are Cold Pixels usually an issue?

How does a Cold Pixel look like? Is it a dead pixel e.g. Black?
once you go raw you never go back

bouncyball

  • Senior
  • ****
  • Posts: 362
Re: mlv_dump on steroids
« Reply #123 on: August 18, 2017, 11:11:30 AM »
I guess it's more like dead pixel and is nearly black. Someone correct me please :)

Danne

  • Hero Member
  • *****
  • Posts: 3480
Re: mlv_dump on steroids
« Reply #124 on: August 18, 2017, 12:36:21 PM »