MLV Lite

[dmilligan]

MLV Lite
The sweet taste of MLV with none of the extra calories™

Recently I've been thinking about ways to do away with the old RAW file format (it has a lot of problems). I had originally thought of creating a sort of hybrid between RAW and MLV formats, with an MLV header at the beginning and then just a dump of raw data after that like the RAW format, but a1ex helped me realize that's it not hard at all to modify raw_rec to generate completely valid MLV files ("MLV Lite" naming credits also go to a1ex). So, that's what I've done, and it seems to be working. Now it needs just needs some testers.

Download: raw_rec.mo (now available in nightly builds)
PR: https://bitbucket.org/hudson/magic-lantern/pull-requests/685/proposal-completely-replace-the-old-raw

There should be virtually no difference between the way this raw_rec operates and the one in the current nightlies with the sole exception that you will get MLV files out of it. You should see almost identical performance and behavior.

There are some caveats to what you get compared to the full mlv_rec:
1. Metadata is only for what the settings were when recording was started (e.g. you get expo metadata, but if you change exposure, you don't get that new information).
2. No audio
3. No card spanning (recording to both CF and SD card)

This needs to be thoroughly tested so we can convince the main devs to merge it. Here are something things you should test:
1. High FPS
2. File splitting at 4GB limit
3. What happens when card runs out of space?
4. Heavy CPU usage options (GD overlays, crop mode preview, etc.)
5. Metadata is correct?
6. Try various MLV converters to make sure they can all handle the files
7. Write performance comparisons (this raw_rec vs. previous raw_rec vs. mlv_rec)

Make sure to fully describe every situation you tested and the results (whether or positive or negative) and tell which camera you're using, and it'd be good to have tests for both 5D3.113 and 123.

[dfort]

Just a quickie test shows that focus pixel fixing with MLVFS is working and it looks like the camera is able to record more frames at the higher resolutions before hitting the "Frame skipped. Stopping." Of course there should be more testing. Just wondering, is dynamic panning, a.k.a. "digital dolly" supported? It looks like the pan information is saved for every frame.

[reddeercity]

Sound Interesting  , This is a good idea  if nothing changes as in speed then I'm 100% for New Fast MLV Lite !
 I'll test on my 5d2 and compare to the old Raw and report back tomorrow .
 

[dmilligan]

@dfort
It's saved for every frame b/c the spec requires it, but currently it's not updated after recording starts. I suppose it wouldn't be a problem to update it though.

[Danne]

Brilliant. Will test as soon as I can.

[kgv5]

I made some simple tests and really didnt notice any difference (5d3, 1.1.3) in speed vs MLV rec (I tried for example crop mode with 2,8K res and fps override 1080p 38fps and recording duration in seconds was practically the same). But i guess it needs more tests with different settings.

[Danne]

Gotta love your your work dmilligan. Did some morning tryouts. Gotta get to work. Here are my findings.

5D mark III, ML version 1.1.3

MLV Lite         1792x606, 50fps  2178 frames, global draw off   
MLV                1792x606, 50fps  850 frames, global draw off, sound off

MLV Lite         1920x1080 25fps, with or without global draw (focus peaking) Continuously!
                       Viewing the first 1200 frames I couldn,t find any corrupted frames due to global draw on(focus peaking). This is great stabilty for PAL users.
MLV                1920x1080 25fps around 400 to 600 fps depending on global draw on or off, sound off

Problems with invalid header size. In preview window (mlv play on camera) files previewing works for around 1180 frames but I recorded for whole 5 minutes 1920x1080. Probably the first chunk of MLV works then comes the invalid header size issue.

Post processing
The first MLV chunk convert in post not the others filming longer sequences. Used MLP(mlv_dump), worked splendidly(the first MLV chunk).
Same goes for MLVFS, only first chunk of files pops up, M00 etc doesn,t.

Couldn,t get the files to play in MlRawViewer, too bad Baldand seems to have left the ML development.

Didn,t test against format RAW(no time) but I reckon the numbers are identical or close comparing with MLV Lite

Great work.

[reddeercity]

Did a Quick test with the latest Nightly.2016Feb13.5D2212, 5D2
1856x1004 , 1856x1044  @23.976 and 3x crop mode 2048x930 exacted with mlv.dll with pismo mount on WinPro 7
I did not have time today to compare with raw format.

I did have so problems loading the new mlv lite module , there some log files . Will post them later.
Had problems with the overlays not being disabled Like full MLV when you hit record.
So that Slowed down the frames. The overlay Kill performance on the 5d2
If I kill overlays , it seems to kept the speed but not sure as there no debug to see the data recorded.
Well that not totally true , after messing around a bit I would able to enable the data graph with buffer data rate
but I had to disable the Overlays globally so no meters etc...
Would be nice to have it behave like FULL-MLV.

1856x1004 recorded continuously @ 23.976 74.5MB/s Same as the old raw( did not try any HDMI device at this time )
1856x1044 stop at 800 frame (old raw 1880x1058 @ 23.976 is continuous. Very old build I use from Oct24/13)
I did get a few pink/corrupted frames with 1856x1044 no problems with 1856x1004 @ 23.976p .

3x Crop 2048x930 @ 23.976 is continuous , same as the Old Raw nightly build I use (Oct24/13)
***Note*** I use that build because it faster then all of Raw builds to-date.
When I converted the 3xCrop mode mlv lite files all of the 3xCrop files are Pink , just like the black level is wrong.
I remember when crop mode video was being implemented there was the same issue then
back in fall of 2013 if I'm not mistaken .

I need to do more testing  , from what I have  seem so far I like. 
I will post log files and the pink/corrupted dng's in a day or so.
Tested on a CF Lexar 64GB 1066x card

 
 

[dmilligan]

The first MLV chunk convert in post not the others filming longer sequences. Used MLP(mlv_dump), worked splendidly(the first MLV chunk).
Same goes for MLVFS, only first chunk of files pops up, M00 etc doesn,t.

Yeah, I was worried chunk splitting would have issues. Didn't have time to test it (takes too long to record a sample that hits the 4GB barrier). Can you send the first ~20MB of the split file or post a screenshot of a hexdump of the start of the file?

Would be nice to have it behave like FULL-MLV.

Well that's sort of missing the point. If you want something that behaves like the full MLV, then use the full MLV. This should behave exactly like raw_rec except give you MLV files. I don't have any plans to modify the feature set of raw_rec, that's outside the scope of this effort. I just want to switch out the format and make sure everything works and achieves the same level of performance as the old raw_rec. If we can achieve that goal and verify it with good test reports, then we can be done with the old raw format once and for all.

[Danne]

Code: [Select]

mlv_dump -o output.M00 -f 20 input.M00Or renamed to MLV extension gives a segmentation fault 11 but here,s a screen dump. Tell me if you need anything else.
Hex_dump
https://drive.google.com/file/d/0B4tCJMlOYfirbHE5d2RrcUFhSWc/view?usp=sharing

Or did you need the first chunk MLV? Anyway here,s the first file as well, shortened.
https://drive.google.com/file/d/0B4tCJMlOYfiraEMxaXNtekcwRms/view?usp=sharing

[hjfilmspeed]

This is an incredible idea!

[Ottoga]

3. What happens when card runs out of space?

Just a thought, that could also be applied to mlv_rec.mo.

When enabling the module, why not:
1.     check the available space on the card
2.     calculate how many minutes of recording time is available based on the resolution and crop mode set at the time of pressing the
        start recording button. (Max recording)
3.     reduce the calculated available time by say 1 minute (to maintain some free space overhead) - (Allowed Recording)
4.     display this on the screen with a continue y/n? option.
5.     Automatically stop recording when the allowed recording time has been reached.

I think that this would allow the MLV file/last segment to be closed correctly thereby eliminating the number of "Card full" issues raised with MLV recording. Step 4 would give the operator a chance to clear space on the card or swap it out for a larger capacity one if the allowed capacity was insufficient for their needs and/or rethink their filming strategy to work around the space constraint.

[menoc]

Simply Put: "MLV_REC without the overhead . . ."

[dmilligan]

@Ottoga,

Again, adding new features like this are beyond the scope of this effort.

I just want to know what happens when the card runs out of space. Are you still able to convert the file?

[dmilligan]

The first MLV chunk convert in post not the others filming longer sequences. Used MLP(mlv_dump), worked splendidly(the first MLV chunk).
Same goes for MLVFS, only first chunk of files pops up, M00 etc doesn,t.

I think I fixed the issue. Binary updated, please retry.
https://bitbucket.org/dmilligan/magic-lantern/downloads/raw_rec.mo

[reddeercity]

Ok I hear you on the scope of this ,
More information  , Spanning works only up to 1 file  2 or more will crash any app I try.
Tested on MLVFS (Mac) (I don't have the latest updated version)
Raw2Cdng ver.1.65 (Win) , note the 3xCrop file looked find in the preview window (not Pink)
MLVProducer (ver.2224-Intel)(Win) also crashed on 2 or more spanning files , but there again the 3X Crop file looked ok (not Pink)
Here the crash log from MLVProducer  mlvproducer log file

Seems the mlv.dll with Pismo was the only app producing pink(wrong black level I think) frame from 3xCrop mode.
Maybe the mlv.dll needs up dating , not sure.
 
when the Cf card was full , I didn't know and try to record just came back with a error message "file corrupted or something to that effect".
Made 3 files , that crashed all PC apps but MLVFS did not crash ,  open up the web page but display nothing , just files names.
Here the link to those 3 file ,(very Small)  MLV_Lite_Card_Full.zip
After that rebooted the cam with that same card , camera displayed a "Full Card" message.

When I first tried to load that module  , I had problems here is the Log files it generated ASSERT00-18.zip . It keep generating files until I reboot the Cam. then it was ok

Lastly here is the pink/corrupted dng's frames in a zip file only 30Mb  dng's frames.zip

Edit: Notice you posted a updated version or the raw_rec.mo will try it out tomorrow & report back.

[Danne]

Tried the latest raw_rec here
http://www.magiclantern.fm/forum/index.php?topic=16650.msg162413#msg162413

Will post dumps later

Longer sequences works with MLVFS. Tested 5 minute clip, about 7000 frames, 28gb file
Longer sequences won,t work with mlv_dump. Only the first chunk will create dng files.
Mlv_dump will convert to dng files if it has only one spanned file recorded, M00, then it processes both the files but only the first if the recording has more than one extra chunk, then it will only process the first chunk called MLV.
More testing later.
Work...


Here are some files.
I shortened the first MLV and the M00 is an entry hex_dump.
MLV_lite. Originally three chunks, only the first will process in mlv_dump
https://drive.google.com/file/d/0B4tCJMlOYfirbml5WXdFQVFXcEE/view?usp=sharing

Regular mlv file, three chunks. Processing works with mlv_dump (comparing maybe)
https://drive.google.com/file/d/0B4tCJMlOYfirMU5LN21BQXV1SU0/view?usp=sharing

If you need something else let me know.

[DeafEyeJedi]

Loving this Project and it's progress so far ... sorry I have been out and about with work and life as we all know, right?!

Anyway here's what I was able to accomplish with reading from MLVFS with the new raw_rec.mo files that I downloaded and placed into the Binaries folder under ML.

I then test shot a bunch of slo-mo's in varieties FPS.



Noticed the two empty lines (MLV #2148 & #2256) which were spanning files with .M00, .M01, etc in their respective destination -- not sure what could be the culprit here?

However, I have tried running some of the new raw_re.mo MLV's that show up 'Black Screen of Death' upon viewing footage in MLRawViewer ... Not all but some and will need to investigate further more to confirm this more in depth.



Until I tried DR12 while running MLVFS and the first file recorded shows up fine (screenshot below) as oppose to MLRV (screenshot above). Maybe this has to do with mlv_dump? Or the header?



I am currently running some of those files with @Danne's recent updated version of MLP and will report back once I find the results. I am also in the middle of updating @AWPStar's MLVProucer on my Mac under Wine and will report that back as well.

Gotta get some sleep and off to work early in AM. Thanks again David for your excellent innovations that just seems to never end! 

[dmilligan]

Seems the mlv.dll with Pismo was the only app producing pink(wrong black level I think) frame from 3xCrop mode.
Maybe the mlv.dll needs up dating , not sure.

Pismo is obsolete: http://www.magiclantern.fm/forum/index.php?topic=13152.msg161902#msg161902

When I first tried to load that module  , I had problems here is the Log files it generated ASSERT00-18.zip . It keep generating files until I reboot the Cam. then it was ok

That error is actually coming from mlv_rec. You probably had both loaded, which is causing some sort of conflict.

@Danne, @DeafEyeJedi
Recording multiple chunks is working just fine here, so I'm not sure what's going on, and everything in the sections of the files you sent me looks correct. The only thing that's "wrong" is the fileCount field (it's always 0, b/c we don't go back and update it after we've created new chunks), but it's allowed to be wrong:

- fileCount field in the header may get set to the number of total chunks in this recording

And files like this process just fine for me in mlv_dump and MLVFS.

What's the output of mlv_dump -v for the files with multiple chunks?
If you rename each .MXX chunk to .MLV will they process by themselves?
(make sure to delete any .IDX files before trying anything new)

@DeafEyeJedi,
Anything in mlvfs log file? ~/.mlvfs.log

[Danne]

Here is terminal output from a working set of three spanned files (regular MLV)

Code: [Select]

MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: '/Users/Daniel/Desktop/mlv/regular mlv/M17-0810.MLV'
   - Convert to DNG frames
   - Output into 'hej'
File /Users/Daniel/Desktop/mlv/regular mlv/M17-0810.MLV opened
File /Users/Daniel/Desktop/mlv/regular mlv/M17-0810.M00 opened
File /Users/Daniel/Desktop/mlv/regular mlv/M17-0810.M01 opened
File /Users/Daniel/Desktop/mlv/regular mlv/M17-0810.M02 not existing.
Processing...

Vertical stripes correction:
  1.000  1.000  0.995  1.021  0.997  0.997  0.995  1.004
Reached end of chunk 1/3 after 2764 blocks
Reached end of chunk 2/3 after 2687 blocks
Reached end of chunk 3/3 after 1 blocks
Processed 2683 video frames
Done


Here,s the output from three spanned files recorded with mlv_lite

Code: [Select]

MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: '/Users/Daniel/Desktop/mlv/mlv_lite not working/M17-0807.MLV'
   - Convert to DNG frames
   - Output into 'hej'
File /Users/Daniel/Desktop/mlv/mlv_lite not working/M17-0807.MLV opened
File /Users/Daniel/Desktop/mlv/mlv_lite not working/M17-0807.M00 opened
File /Users/Daniel/Desktop/mlv/mlv_lite not working/M17-0807.M01 opened
File /Users/Daniel/Desktop/mlv/mlv_lite not working/M17-0807.M02 not existing.
Processing...

Vertical stripes correction:
  1.000  1.000  0.995  1.016  0.999  0.997  0.998  1.003
[ERROR] VIDF: File ends in the middle of a block
Processed 1182 video frames
Done

If you rename each .MXX chunk to .MLV will they process by themselves?

Yes, working.

Verbose output
First lines

Code: [Select]

MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: '/Users/Daniel/Desktop/mlv/mlv_lite not working/M17-0807.MLV'
   - Verbose messages
   - Convert to DNG frames
   - Output into 'hej'
File /Users/Daniel/Desktop/mlv/mlv_lite not working/M17-0807.MLV opened
File /Users/Daniel/Desktop/mlv/mlv_lite not working/M17-0807.M00 opened
File /Users/Daniel/Desktop/mlv/mlv_lite not working/M17-0807.M01 opened
File /Users/Daniel/Desktop/mlv/mlv_lite not working/M17-0807.M02 not existing.
Processing...
File Header (MLVI)
    Size        : 0x00000034
    Ver         : v2.0
    GUID        : 7343463891422747826
    FPS         : 25.000000
    File        : 0 / 0
    Frames Video: 0
    Frames Audio: 0
Block: RAWI
  Offset: 0x00000034
    Size: 180
    Time: 0.768000 ms
    Res:  1920x1080
    raw_info:
      api_version      0x00000001

Verbose output
Last lines

Code: [Select]

Block: VIDF
  Offset: 0xfff7221c
    Size: 3633152
    Time: 48853.163000 ms
   Frame: #1182
    Crop: 152x132
     Pan: 146x133
   Space: 4064
[ERROR] VIDF: File ends in the middle of a block
Processed 1182 video frames
Done

Code: [Select]

[ERROR] VIDF: File ends in the middle of a blockSeems to be what is happening here.

[Ottoga]

MLVLite Tests
--------------------
Camera:   7D
ML Build:   magiclantern-Nightly.2016Feb13.7D203
raw_rec.mo  (2nd lite build)

Cards used: Sandisk Extreme III 8GB 30MB/s
                  KB 128gb 1000x

PC:    ASUS Intel i5 quad core laptop


Tested with:
    -    Default raw resolution with continuous recording to generate spanned file mlv, m00, m01... etc. The longest went to m14.
    -    Some alternate resolutions
    -    5x and 10x crops
    -    fps override
    -    ettr "always on" for all tests
    -    global draw off for all tests
    -    kept recording and creating files unill both cards ran out of space

General observations

1)    When in 10x crop mode recording failed to start when pressing the record start button.
       There are 14 MLV files (M18.1841-1855) created with a 1kb size and in the same time window there is a file called backup.raw 512kb that was created. I believe
       that these files are all related the 10x recording failure.
       Whilst pressing the start record button, I ultimately managed to observe the following message that flickered onto the screen after each button press:
                  "Raw detected error".

      Link to backup.raw file: http://1drv.ms/1R9MEeo

2)    When the card ran out of space the following occured:
      :    The message "Movie recording stoped automatically, Data corruption @slot nn, frame nnnnn (where nn  is a numeric number)
           This message appeared a few times before ultimately staying on screen. The frame number changed for each occurance of
           the message, can't remember whether the slot numumber did.
      :    The CF access LED remained lit.
      :    I pressed the stop recording button, waited a bit but nothing changed so I powered the camera off (the LED stayed lit).
    At this stage,
      :    for the 1st (8gb) card, I removed and reinserted both the battery and the CF card (thnking that the camera had frozen).
           -  Powered the camera backup and receive a message that the camera had not been shut down correctly. Otherwise the camera
              booted normally, including ML.
      :    for the second (128gb) card I just powered the camera backup again.
              I received a "CF Card is Full" message, however the camera booted normally, including ML.
      :    I was able to copy all files off the CF cards to my local harddrive.

    From this, my inital observation would be that for the 7D at least, running out of space whilst recording will not brick the camera
    or corrupt the CF card.

3)  With the exception of the 1kb size MLV files all of the others mounted without error in MLVFS.

     localhost view:   http://1drv.ms/1Lv3NLp

     CMD Lin view:   http://1drv.ms/1Lv3TlY

4)  Excluding the 1kb size MLV files all of the others successfully loaded into and played in MLVProducer (version: mlvp.alpha.build2224b.INTEL)

5)  Via the Dokan v1rc mounts I can successfully import the DNG's into my Photo and video editing applications. A quick visual scan of the individual frames didn't spot
     any bad or pink frames.


Let me know if there are any other specific tests that you would like me to do.

[Danne]

@Ottoga.
Will your files convert properly using mlv_dump?

[Ottoga]

@Danne

I haven't tried this. I'll give it a go in the morning and report back.

[Ottoga]

@Danne  @Dmilligan

5)  Via the Dokan v1rc mounts I can successfully import the DNG's into my Photo and video editing applications. A quick visual scan of the individual frames didn't spot
     any bad or pink frames.

Correction to the above.  Upon closer inspection I found that for the recordings that were in progress at the time that the CF card ran out of space: the last frame of each was corrupt.


I processed all of the multi-chunk mlvs and a couple of single chunk ones through mlv_dump. The tests resulted in a mixed bag of outcomes.

At launch, mlv_dump listed all of the available file chunks to be processed but listed the next logical one in the chain as "Not Existing" Ref: to the linked screen shots for examples. I suspect that this is standard mlv_dump behaviour.

A number of files failed to process all of the available chunks with a common error message:  ERROR VIDF: File ends in the middle of a block

The following links are the mlv-dump screenshots for the files related to the recordings that were in progress at the time the CF cards ran out of space. Both of these files suffered the above error however, it was on the last (and corrupt) frame that it occurred.

M18-1704.mlv and last file on the 8gb card:      http://1drv.ms/1oO2276   1 Chunk, 1399 of 1400 frames processed
M18-1856.mlv and last file on the 128gb card:  http://1drv.ms/1Qpej7U    9 chunks, 13508 of 13509 frames processed

The following links are for files where mlv_dump failed to process all of the available frames.

M18-1658.mlv:  http://1drv.ms/1oO2rXh     2 chunks, only processed the first 3276 of 4050 frames before erroring out
M18-1805.mlv:  http://1drv.ms/1oO2Rgh     3 chunks, only processed the first 1458 of 44214 frames before erroring out

The following links are for files processed through mlv_dump that processed error free.

M18-1655.mlv: http://1drv.ms/1Ktv59V         1 chunk,    1031 frames processed
M18-1808.mlv: http://1drv.ms/1oO37fd       16 chunks, 23150 frames processed
M18-1826.mlv: http://1drv.ms/1XATnR8        1 chunk,      601 frames processed (fps 1 frame per second)

I was running multiple instances of mlv_dump. To verify the results I re-ran a number of the mlvs with a single instance of mlv_dump running and got the same results.


My thoughts are that mlv_dump is not able to handle the mlv_lite files that had skipped frames or where the mlv_lite file closes abruptly due to things like non default resolutions and high resolution crop mode.

Cheers... Otto

[dmilligan]

The issue here is the 4GB file size limit on Fat32. The file IO routines in Canon firmware respond to this limit in different ways for different cameras (they vary slightly in other ways too). Some cameras will fail immediately if there's not enough room before the 4GB for the current requested write, while others will fill up to the end of 4GB exactly, then fail. We could just always stop before getting to the limit, but we don't necessarily know in advance if there is a 4GB limit or not (ExFAT does not have a 4GB limit). The only way to know if there is a 4GB limit is to actually hit it (well, I suppose there might be other ways to find out).

The old raw format simply allowed frames to span across files, so one chunk might end in the middle of a frame and then the remaining part of the frame would start the next file. MLV format does not allow this.

If mlv lite hits the 4GB limit, it will start the next chunk with the current frame that failed to write, so there's actually nothing wrong with the next chunk file (that's why if you rename it to MLV it will process correctly). The problem is what happens at the end of the previous chunk. If the camera doesn't write any data when there's not enough room left under the 4GB limit, then there's no problem, but if the camera writes some of that data, then the file will end in the middle of a frame. So mlv lite needs to go back and clean it up.

Here's a visual example:
                                                     4GB Limit->|
[MLVI][VIDF1-01010101][VIDF2-01010101][VIDF3-01010101][VIDF4-01010101]

Sometimes this happens:
[MLVI][VIDF1-01010101][VIDF2-01010101][VIDF3-01010101]
Yay! everything is fine

but sometimes this happens:
[MLVI][VIDF1-01010101][VIDF2-01010101][VIDF3-01010101][VIDF4-0101

What it tries to do if this happens is replace the last VIDF with a NULL so it will be ignored:
[MLVI][VIDF1-01010101][VIDF2-01010101][VIDF3-01010101][NULL-0101]

It appears that the cleanup does not work reliably on all cameras. If it doesn't happen right, then mlv_dump will give you an error that the file ends in the middle of a block. There's actually no reason mlv_dump couldn't just keep processing if this situation happens, and simply emit a warning, rather than an error. The frame that got chopped off will be in the next chunk in it's entirety.

The same sort of thing happens when the card runs out of space, but in this situation we don't even bother to try and clean it up, there's nothing going to come after this frame anyway. If the last frame is corrupted, well, that's to be expected. As long as all the rest of the file processes correctly, I'm not worried about it.