Magic Lantern (RAW) Video format v2.0 (mlv_rec.mo)

Started by g3gg0, July 15, 2013, 10:58:23 PM

Previous topic - Next topic

0 Members and 3 Guests are viewing this topic.

Danne

Hi g3gg0!
It happens on any MLV file I test with. Confirmed the same behaviour from dfort and deafeyejedi and your included binary in first post. Here is a -v output.

Last login: Sun May 22 20:43:53 on ttys005
Daniels-MBP:~ Daniel$ /Users/Daniel/Desktop/Myfiles/mlv_dump -f 1 --cs5x5 --dng -v -o hej /Users/Daniel/Desktop/Myfiles/M04-0025.MLV

MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: '/Users/Daniel/Desktop/Myfiles/M04-0025.MLV'
   - Verbose messages
   - Convert to DNG frames
   - Output into 'hej'
File /Users/Daniel/Desktop/Myfiles/M04-0025.MLV opened
File /Users/Daniel/Desktop/Myfiles/M04-0025.M00 not existing.
Processing...
File Header (MLVI)
    Size        : 0x00000034
    Ver         : v2.0
    GUID        : 11410529140746060919
    FPS         : 23.976000
    File        : 0 / 1
    Frames Video: 450
    Frames Audio: 95
Block: RAWI
  Offset: 0x00000034
    Size: 180
    Time: 205.985000 ms
    Res:  1920x1080
    raw_info:
      api_version      0x00000001
      height           1318
      width            2080
      pitch            3640
      frame_size       0x00493450
      bits_per_pixel   14
      black_level      2047
      white_level      15000
      active_area.y1   28
      active_area.x1   146
      active_area.y2   1318
      active_area.x2   2078
      exposure_bias    0, 0
      cfa_pattern      0x02010100
      calibration_ill  1
Block: INFO
  Offset: 0x000000e8
    Size: 16
    Time: 210.371000 ms
Block: RTCI
  Offset: 0x000000f8
    Size: 44
    Time: 0.001000 ms
     Date:        04.12.2015
     Time:        00:23:59 (GMT+0)
     Zone:        ''
     Day of week: 5
     Day of year: 337
     Daylight s.: 0
Block: EXPO
  Offset: 0x00000124
    Size: 40
    Time: 0.002000 ms
     ISO Mode:   0
     ISO:        3200
     ISO Analog: 112
     ISO DGain:  0/1024 EV
     Shutter:    29994 microseconds (1/33.34)
Block: LENS
  Offset: 0x0000014c
    Size: 96
    Time: 0.003000 ms
     Name:        '20mm F1.4 DG HSM | Art 015'
     Serial:      ''
     Focal Len:   20 mm
     Focus Dist:  51 mm
     Aperture:    f/0.00
     IS Mode:     0
     AF Mode:     0
     Lens ID:     0x000000FA
     Flags:       0x00000000
Block: IDNT
  Offset: 0x000001ac
    Size: 84
    Time: 0.004000 ms
     Camera Name:   'Canon EOS 5D Mark III'
     Camera Serial: 'C587A29EA'
     Camera Model:  0x80000285
Block: WBAL
  Offset: 0x00000200
    Size: 44
    Time: 0.005000 ms
     Mode:   9
     Kelvin:   2500
     Gain R:   608
     Gain G:   1024
     Gain B:   400
     Shift GM:   0
     Shift BA:   0
Block: STYL
  Offset: 0x0000022c
    Size: 52
    Time: 0.006000 ms
     picStyle:   129
     contrast:   0
     sharpness:  0
     saturation: 0
     colortone:  0
Block: WAVI
  Offset: 0x00000260
    Size: 32
    Time: 507.541000 ms
    wav_info:
      format           1
      channels         2
      samplingRate     48000
      bytesPerSecond   192000
      blockAlign       4
      bitsPerSample    16
Block: RTCI
  Offset: 0x00000280
    Size: 48
    Time: 508.823000 ms
     Date:        04.12.2015
     Time:        00:23:59 (GMT+0)
     Zone:        ''
     Day of week: 5
     Day of year: 337
     Daylight s.: 0
Block: EXPO
  Offset: 0x000002b0
    Size: 48
    Time: 510.421000 ms
     ISO Mode:   0
     ISO:        3200
     ISO Analog: 112
     ISO DGain:  0/1024 EV
     Shutter:    29994 microseconds (1/33.34)
Block: LENS
  Offset: 0x000002e0
    Size: 96
    Time: 510.457000 ms
     Name:        '20mm F1.4 DG HSM | Art 015'
     Serial:      ''
     Focal Len:   20 mm
     Focus Dist:  51 mm
     Aperture:    f/0.00
     IS Mode:     0
     AF Mode:     0
     Lens ID:     0x000000FA
     Flags:       0x00000000
Block: WBAL
  Offset: 0x00000340
    Size: 48
    Time: 510.473000 ms
     Mode:   9
     Kelvin:   2500
     Gain R:   608
     Gain G:   1024
     Gain B:   400
     Shift GM:   0
     Shift BA:   0
Block: VIDF
  Offset: 0x00000370
    Size: 3632624
    Time: 529.645000 ms
   Frame: #0000
    Crop: 152x136
     Pan: 146x133
   Space: 3792

Vertical stripes correction:
  1.000  1.000  0.998  1.008  0.996  0.996  0.995  0.995
Segmentation fault: 11
Daniels-MBP:~ Daniel$


ewinemiller

Anybody see corruption like this? I've been getting a couple of frames every recording like this. I'm using the December 5D3.123 build.

Thanks.


beauchampy

Use August 7th 2014 for 1.2.3
Use anything post 11th April 2016 for 1.1.3

Debug logging causes the pink corrupt frames (it's turned off in those builds).

Lars Steenhoff

QuoteUse August 7th 2014 for 1.2.3
Use anything post 11th April 2016 for 1.1.3

Debug logging causes the pink corrupt frames (it's turned off in those builds).



Very useful info, and I think it should be on the front page somewhere!

Many user don't know this and will be put off after seeing pink frames using the latest 1.2.3 nightly.

or can we remove the offending builds and put them in an archive somewhere?

This would help many users who encounter this problem


Jockerl

can someone please compile a new build for 1.2.3 with logging disabled :)

Edit: or is it possible to use the newest mlv-rec.mo out of the 1.1.3 nightly, which does not have the logging enabled with the latest 1.2.3 nightly?

Lars Steenhoff

you could use the mlv module from August 7th 2014 for 1.2.3 in the December build.

but yea a new build based on more recent branch would be welcome


beauchampy

Quote from: Lars Steenhoff on July 22, 2016, 02:18:17 PM
you could use the mlv module from August 7th 2014 for 1.2.3 in the December build.

but yea a new build based on more recent branch would be welcome



the issue with that module is i think it doesn't allow for a single raw clip longer than 40gb / ~8.5 mins

omananda

Greetingz ...

None of the players and converters work for me on a mac (Yosemite 10.10.5)

Any suggestions? I would love to test the quality of the .mlv files and see how they can be edited?

Kind Regards, OM

Walter Schulz


a1ex

Small update to mlv_dump:

- fix cold pixel processing when saving only a subset of the frames
- cold pixel fix is now enabled by default
- fine-tuned vertical stripe fix + option to disable it
- option to scan every single frame for cold pixels (slow, but I remember some reports with moving cold pixels, for example with very bright specular highlights)
- experimental flat field (gain) correction, details here
- minor fix to dark frame subtraction mode (proper white level and keep sub-black values)


--no-fixcp          do not fix cold pixels
--fixcp2            fix non-static (moving) cold pixels (slow)
--no-stripes        do not fix vertical stripes in highlights

-t mlv_file         use the reference frame in given file as flat field (gain correction)


These updates were tested here.

Windows build: https://builds.magiclantern.fm/jenkins/view/Other%20tasks/job/mlv_dump/

Will ask g3gg0 to update the first post and maybe set up a Mac build.

mothaibaphoto

Get some weird artifacts processing with chr2hdr DualISO dng files extracted from FRSP MLV files with this last version. Previous version(with --fixcp2 and --fix-stripes) works pretty well. On slow connection now maybe later can upload examples.

Danne

Tested developing a dualiso file. Some bug with wb(dng corruption). Dfort found this as well yesterday.

Latest mlv_dump(wrong wb)
unprocessed dng
https://drive.google.com/file/d/0B4tCJMlOYfirekEtV05PRHpzMHM/view?usp=sharing

Comparison dng(no surprises with wb) used an older build of mlv_dump
unprocessed dng
https://drive.google.com/file/d/0B4tCJMlOYfirRk5POWM4UXJJX1U/view?usp=sharing

Original MLV(dualiso) shortened(7mb)
https://drive.google.com/file/d/0B4tCJMlOYfirdTNKRldId1oxRXM/view?usp=sharing

mlv_dump(latest) compiled for mac.
https://drive.google.com/file/d/0B4tCJMlOYfirVVR1VFNOSExPLUk/view?usp=sharing


*update.
Stripes correction causing corruption in dng files.


dfort

Not sure why this is being discussed in the MLV Lite topic but for what it is worth here are my findings.

bouncyball has a version of mlv_dump that fixes the wb issues that Danne pointed out. His version is using dmilligan's dng module.

https://bitbucket.org/bouncyball/ml_dng-branch/branch/ml_dng

I thought it would be a good project to add the latest changes to this version of mlv_dump and submit a pull request to the unified branch. This way we can start migrating to the modular dng codebase a little bit at a time instead of all at once. raw2dng is a dependency of mlv_dump but it seems to be broken (wb issues) in the ml_dng branch.

https://bitbucket.org/hudson/magic-lantern/pull-requests/603/dng-module/diff

Now for the strange part. I tried to update raw2dng with bouncyball's latest changes hoping that would fix the wb issues like it did with mlv_dump but that triggered the wb issue in dual iso in mlv_dump that Danne is pointing out. Reverting back to the original (broken) raw2dng and mlv_dump is fine. I tried reverting bouncyball's changes in the unified branch but it had no effect on the mlv_dump wb issue.

So now I've got a working mlv_dump and a broken raw2dng in my mlv_dng branch and a broken mlv_dump and working raw2dng in the unified branch but there's no way to mix and match the two codebases.

Is there a way to break the dependency between mlv_dump and raw2dng? That is beyond my coding skills.

[EDIT] Here's a link to the mlv_dump I'm working on if anyone want to test it: https://bitbucket.org/daniel_fort/magic-lantern/downloads/ml_dng-test_2016-10-12.zip
And a link to my ml_dng work in progress: https://bitbucket.org/daniel_fort/magic-lantern/branch/ml_dng-mlv_dump

dmilligan

I can't really pin down the white balance "issue" you are referring to. (I see nothing recent about wb in the MLV lite thread).

I'm also a bit confused about your statements about a broken raw2dng, since RAW format does not record any white balance information, there's no possible way raw2dng could do anything other than simply hard code some arbitrary white balance.

dfort

Quote from: dmilligan on October 13, 2016, 12:17:22 AM
...I'm also a bit confused about your statements about a broken raw2dng...

Sorry about the confusion, this is not specific to MLV Lite and my comments should probably be moved to another thread.

There seems to be a long standing issue with mlv_dump as noted in your comment on this post:

Quote from: dmilligan on October 01, 2015, 01:00:13 PM
Once the DNG module PR I have open is merged, then mlv_dump will also handle WB correctly.

So it turns out that a few of us are using mlv_dump from your ml_dng branch and it indeed handles white balance properly. I take it you've been busy on other projects so I've been working on trying to move this forward so that everyone will benefit.

However, as you know mlv_dump references raw2dng when compiling and even though mlv_dump is fine, there's a problem with raw2dng in your ml_dng branch.

No point continuing the discussion on this thread. I'll post some more details in the Update mlv_dump topic though come to think of it that also somewhat off topic--I'm having one of those days.  ???

[EDIT] Just realized that I mistakenly thought this was the MLV Lite topic. Anyway, I was still off topic bringing up raw2dng.

mothaibaphoto

To continue my previous post:
I'm not about WB. DNG processed with two last builds(Aug 28, 2016 11:09 PM and Sep 5, 2016 10:19 PM) have the same WB for me. But the very last build(Sep 5, 2016 10:19 PM) produce files incompatible with cr2hdr - i get corrupted image with artifacts. If I run that last build with --no-stripes option - everything is ok(except stripes :) ). Therefore, its something wrong with very last tunes of stripe fixes.
Example file
https://www.dropbox.com/s/7fitsoqbrn34v1f/corrupt.mlv?dl=0

dfort

@mothaibaphoto - I don't see those issues on your FRSP file. Running mlv_dump -v corrupt.mlv shows lots of extra metadata that doesn't seem necessary but other than that I don't see anything wrong with it. This was processed on cr2hdr with the default options:



[EDIT] Note that --fix-stripes is no longer an option in mlv_dump.

Danne

I can confirm the finding from mothaibaphoto. --no-stripes fixes the issue with the dual iso files posted before.

DeafEyeJedi

I can confirm this as well. Looks good to me and this was processed with @Danne's cr2hdr app.



*edit*

I just notice this and wasn't sure if this was to be expected re: metadata after its been rendered for Dual-ISO process.



Would this be problematic for Timelapse type of work where certain scripts requires proper tags for exiftool and such?
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

Danne

Deafeyejedi. You tend to forget. MLP is mlv_dump, cr2hdr nothing less. Scripts in a wrapper :)

DeafEyeJedi

Ah, that makes sense and Thanks @Danne -- which was why I mentioned that I used your cr2hdr app (not MLP) even tho they both use the same binaries, right?

Another one of the moments I guess...  ::)
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

dfort

Where are the artifacts? I might not be looking close enough?

mothaibaphoto

Artifacts

@DeafEyeJedi: Whats wrong with that meta and why its matters?
@dfort: Extracting with -f option brings mlv with that frames and metadata for the rest.
It's kinda feature of mlv_dump :)

dfort

Ok--able to reproduce it here. The problem is on the current unified branch. Sorry, I've been trying some things out and probably used the wrong version when I ran that first test. If you're on a Mac try this one or if you can compile you can pull it from my bitbucket repository. It uses the ml_dng module from dmilligan and has all the latest from unified in it. You can find more information on this one on the update mlv_dump topic. This version doesn't show those artifacts though there's something strange going on with the white letters on the road if you look really close. In any case I didn't try any extra option on mlv_dump or cr2hdr so see if I could get rid of them.



By the way, the -f option in mlv_dump is just for extracting frames. You didn't specify which frame to pull but both frames in your MLV file exhibit the same issues.

-f frames           frames to save. e.g. '12' saves the first 12 frames, '12-40' saves frames 12 to 40.

mothaibaphoto

@dfort:  Update mlv_dump topic is great idea and i have various workarounds for issue i encounter,
but all this has nothing to do with message i want to bring to public:
Last version of vertical stripes correction code incompatible with DualISO.

Quote from: dfort on October 14, 2016, 08:59:26 AM
something strange going on with the white letters on the road if you look really close.
Cromatic Aberrations/Purple Fringing
Quote from: dfort on October 14, 2016, 08:59:26 AM
You didn't specify which frame to pull but both frames in your MLV file exhibit the same issues.
I mean I extracted 2 frames from 300 frames sequence, this why it contains too much metadata.