Squeezing the last bit of performance out of MLV Lite (for testers)

[a1ex]

Topic split from MLV Lite thread.

I'm trying to squeeze the last bit of performance out of MLV Lite, to make it as fast as (or maybe faster than) the original raw_rec. You can help by testing different builds and checking which one is faster.



Original message:

I made a small experiment with mlv_lite.

In dmilligan's implementation, frames are 4K larger than with the original raw_rec (because of the MLV headers stored for each frame, and also because of alignment issues). In my version, frame sizes are identical to the original raw_rec.

What I'd like to know: is there any noticeable speed difference between the two versions? How does it compare to the old raw_rec?

My build: raw_rec.mo
to be compared with the one from first post, and to the old raw_rec.

Caveat: my version does not handle 4GB limit well; I'll fix it only if the speed difference is noticeable.

[Frank7D]

a1ex, I tried your raw_rec.mo in my 7D and got an error when turning on camera:

Code: [Select]

Scanning modules...
Load modules...
  [I] load: RAW_REC.MO
Linking..
tcc: error: undefined symbol 'camera_serial'
  [E] failed to link modules

Build version: Nightly.2015Mar14.7D203

[Licaon_Kter]

Build version: Nightly.2015Mar14.7D203

2015? Retry with Apr 01 2016 build I guess.

[Frank7D]

Yes, that works. Thanks!

[Frank7D]

I tried the new a1ex raw_rec and the one from the front page. I didn't see a big difference, but I won't report results in detail just yet because I ran into an issue that might need to be addressed first:

The raw_rec from the front page produced mlv files that opened in mlvfs but wouldn't through mlrawviewer.

The raw_rec from a1ex produced mlv files that would not open in mlvfs (old Pismo pc version) or through mlrawviewer. Since that's how I look at mlvs, I couldn't examine them.

I'm using 1.1.3 mlrawviewer. When I can I'll get 1.3.3 to see if it works better.

[reddeercity]

@A1ex , thanks for the new test mlv lite build . Ran the raw_rec on 5D2 on magiclantern-Nightly.2016Apr11.5D2212.
Not problem with the module ,  normal operation so far . Recorded 28GB in different lengths some with spanning files .
This was just a quick test , well need to do a more in depth testing , but at this point there is a significant increase in write speed .
Test resolution was 1856x1004 @ 23.976p to CF Lexar 64GB 1066x card .

This was my test with mlv lite from dmilligan build Re: MLV Lite Reply #44
1856x1004 @ 23.976 with GD (global draw)  enabled  - 1134 Frames
1856x1004 @ 23.976 with GD disabled - continuous until full (74.5MB/s write speed)
With A1ex build ,
1856x1004 @ 23.976 with GD enabled , (default configuration) 1990 frames (800 more frames)
1856x1004 @ 23.976 with GD disabled , recorded continuous but had no idea what it was , all I know it span more then 2 files
But was flying blind so I stop recording after a few minutes. 
It would but very nice if the over lays could be disabled like Full MLV when recording raw with just the file data rate & buffer info.

Edit: I wasn't sure before  but to confirm ,  5 Spanning files (5700 frames) from 5D2 & all look very good no corruption extracted with  chmee's  raw2cdng ver, 1.7.4
This PC  program never fails to read mlv/raw 

[a1ex]

Indeed, the corruption when spanning files is minor (does not result in data loss) - it's very similar to this one.

Since the method is promising, I've committed the source.

[a1ex]

Here's another experiment:

raw_rec.mo

I'm trying to see if the EDMAC accepts alignments lower than 4096 bytes, and - at least on 5D3 - it accepted 64 just fine. That means, we can align everything at 512 bytes for optimal write speed, and we can use dmilligan's method of placing VIDF headers before the frame without increasing frame size.

How does this one perform in terms of speed?

This build should do proper cleanup for spanning files (dmilligan's method).

[Ottoga]

Tried it on my 7D.
Global draw off, max resolution and again at 5x. Continuous recording ok.  With global draw on and 5x recordings recording stopped after approx 30 seconds.

Output was 3 x multi chunk  files and 1 x single chunk.

All were able to be opened and processed in MLVP.

[reddeercity]

Here's another experiment:raw_rec.mo
How does this one perform in terms of speed?

About the same or little faster , did change the preview option to see if that increased write speed with out frame drop.
1856x1004 @ 23.976 ,  Canon Preview , SRM job memory , Small hacks. Average Write Speed was 70.5MB/s
1908 frames -- expected frames 2240 (card cold)
2370 frames -- expected frames 2501
1665 frames -- expected frames 1901 (After a reboot to check card space)
1566 frames -- expected frames 1650
2330 frames -- expected frames 2515
1899 frames -- expected frames 2090 (Auto Preview)
1972 frames -- expected frames 2170 (Auto Preview)
2440 frames -- expected frames 2610 (ML Grey/Hacked Preview)
  823 frames -- expected frames   900 (ML Grey/Hacked Preview with half shutter pressed for color preview)

3x Crop mode
1920x1038 @ 23.976 , Canon Preview : 1468 frames -- expected frames 1566
1920x1038 @ 23.976 , Hacked Preview : 3166 frames -- expected frames 3900
2048x930   @ 23.976 , Auto Preview    : Continuous (I let go for about 5minutes )
2048x1024 @ 23.976 , ML Grey Preview : 863 frames expected 960 frames.

This build should do proper cleanup for spanning files (dmilligan's method).

Seems to be ok , now I can extract with MLVFS & Prismo , 1:1 images are ok but the 3x Crop mode are
pink , wrong black level according to exiftool : Black Level : 1023

Code: [Select]

ExifTool Version Number         : 10.09
File Name                       : M11-1634_000000.dng
Directory                       : D:/New folder (5)
File Size                       : 3.7 MB
File Modification Date/Time     : 2016:04:11 16:33:54-06:00
File Access Date/Time           : 2016:04:11 16:47:38-06:00
File Creation Date/Time         : 2016:04:11 16:47:38-06:00
File Permissions                : rw-rw-rw-
File Type                       : DNG
File Type Extension             : dng
MIME Type                       : image/x-adobe-dng
Exif Byte Order                 : Little-endian (Intel, II)
Subfile Type                    : Full-resolution Image
Image Width                     : 2048
Image Height                    : 930
Bits Per Sample                 : 16
Compression                     : Uncompressed
Photometric Interpretation      : Color Filter Array
Fill Order                      : Normal
Make                            : Canon
Camera Model Name               : Canon EOS 5D Mark II
Strip Offsets                   : 65536
Orientation                     : Horizontal (normal)
Samples Per Pixel               : 1
Rows Per Strip                  : 930
Strip Byte Counts               : 3809792
Planar Configuration            : Chunky
Software                        : MLVFS
Modify Date                     : 2016:03:11 16:33:54
CFA Repeat Pattern Dim          : 2 2
CFA Pattern 2                   : 0 1 1 2
Exposure Time                   : 1/50
F Number                        : 2.2
ISO                             : 800
Sensitivity Type                : ISO Speed
Exif Version                    : 0230
Subject Distance                : 63 m
Focal Length                    : 50.0 mm
Lens Model                      : 50mm
DNG Version                     : 1.4.0.0
Unique Camera Model             : Canon EOS 5D Mark II
Linearization Table             : (Binary data 87187 bytes, use -b option to extract)
Black Level                     : 1023
White Level                     : 15000
Default Crop Origin             : 0 0
Default Crop Size               : 2048 930
Color Matrix 1                  : 0.4716 0.0603 -0.083 -0.7798 1.5474 0.248 -0.1
496 0.1937 0.6651
As Shot Neutral                 : 0.4580008629 1 0.5775139181
Baseline Exposure               : 0
Calibration Illuminant 1        : D65
Active Area                     : 0 0 930 2048
Frame Rate                      : 23.976
Baseline Exposure Offset        : 0
Aperture                        : 2.2
CFA Pattern                     : [Red,Green][Green,Blue]
Image Size                      : 2048x930
Megapixels                      : 1.9
Shutter Speed                   : 1/50
Focal Length                    : 50.0 mm
Light Value                     : 4.9In 1:1 mode the black level looks normal
Black Level   : 1792
Sample frame from each in my dropbox
3x crop wrong black level 2048x930 And 1:1 1856x1004 Cdng correct black level   I remember back a few years when this was a issue before MLV , looks the same .

All tests were with Overlays enabled .

[a1ex]

For the black level issue, can you try the module on top of latest nightly?

[Steve_FR]

Eos M comparing 4/12 nightly and a1ex's newest raw_rec.mo (MLV Lite 1.1):

Average of 160 frames (5 sample test) at 1728x736 5x @ 23.97 fps (4/12 raw-rec.mo mlv lite 1.1)

Average of 228 frames (5 sample tests) at 1728x736 5x @ 23.97 (raw 1.0 nightly)

Raw rec 1.0 still offering slightly higher performance on Eos M. ML's displayed megabytes/sec was at around 36mb/s for MLV Lite 1.1, and around 40.5mb/s for Raw 1.0. I don't know if that's important all, but thought I'd mention it.

[DeafEyeJedi]

Here's another experiment:

raw_rec.mo

This link isn't working (just gives me a text file) when the link from this post works but that wouldn't be the latest one to experiment, correct?

[Walter Schulz]

@DeafEyeJedi: Open link's context menu and select "Save as ..."

[DeafEyeJedi]

That did it and Thanks @Walter Schulz!

[reddeercity]

For the black level issue, can you try the module on top of latest nightly?

Ok , so I'm not to sure what you are asking so I did some test with older nightly builds (maybe that's what you want ?)
magiclantern-Nightly.2016Jan14.5D2212 , magiclantern-Nightly.2016Feb13.5D2212 , magiclantern-Nightly.2016Apr01.5D2212 , magiclantern-Nightly.2016Apr11.5D2212 .
They all product pink frames with black level of 1024 , for Jan, Feb & Apr 01 I used dmilligan's mlv lite V.1.0 . & use your ver 1.1 on the apr11 build.
I also did a quick test with the old raw with the builds I tested and all produce normal black level of 1792 in 3xCrop mode @ 2048x930 23.976 .
But one thing catch me off guard when I though I would try the mlv lite module from dmilligan's mlv lite V.1.0 in 1:1 mode @ 1856x1004 23.976
with the magiclantern-Nightly.2016Jan14.5D2212  just to see how long I could record to get a comparison to the latest nightly (Apr/11/2016) with your build a1ex , and boy did that surprise me
I got 2409 frames out of expected frames of 2600 
that the most yet ! just by a little but my question is what happen between Jan14 build and Apr11 build to slow thing down of is there something in play here I don't understand.

Edit: I just remembered that I report the 3x crop pink frame problem in the raw2cdng thread as I was using chmee's app , never thought it was a module issue
http://www.magiclantern.fm/forum/index.php?topic=5618.msg162891#msg162891

[a1ex]

Latest = opposite of older. Download 2016Apr12 (latest when I asked you to run the test), replace the module, and try it.

Look for "black" in the changelog for that build.

If you notice any speed difference between 2016Jan14 and 2016Feb13, it's either pure luck, or the inherent noise in your testing method. The only change was this.

[Danne]

Recorded 10 files of each setting on an eos 7D. Printed out highest performance achieved. Used latest nightly build (magiclantern-Nightly.2016Apr13.7D203) 598036c

RAW
60fps  1728x584 (focus peaking on) highest achieved. 338 frames
mlv_lite(dmilligan)
60fps  1728x584 (focus peaking on) highest achieved. 336 frames
mlv_lite(a1ex)
60fps  1728x584 (focus peaking on) highest achieved. 338 frames


RAW
25fps  1728x972 (focus peaking on) highest achieved.  822 frames
mlv_lite(dmilligan)
25fps  1728x972 (focus peaking on) highest achieved.  896 frames
mlv_lite(a1ex)
25fps  1728x972 (focus peaking on) highest achieved.  899 frames


Retested RAW 25fps  1728x972 one extra time to see what it would achieve with card at its warmest.
Last run(best out of ten)
RAW
25fps  1728x972 (focus peaking on) highest achieved.  872 frames

[Frank7D]

I tried testing an earlier module but had an issue reading the MLVs (see April 11 post).
Before I try testing the latest, I have a question:
Is there a recommended protocol for testing (minimum number of clips, etc.) that will allow us to see meaningful differences in the results as opposed to random variation?

[a1ex]

The differences are amplified when the write speed needed is just a little above what can be achieved. You could aim for about 2000 frames until it stops.

To get consistent results, try to start from the same initial conditions (e.g. card formatted in camera, then restarted, maybe warmed up, whatever other settings you prefer). Just do the same in all the tests, so the numbers can be comparable.

To evaluate the variations, you could record 10 clips, write down number of frames from each one, and compute:
- median
- MAD
- min
- max

Why median/mad and not mean/stdev? It's hard to make a lot of experiments, and the distribution is not exactly Gaussian - in particular, the outliers can be quite far from the average. Mean and stdev are easily influenced by outliers, while median and MAD are much more robust.

Min and max are mostly for evaluating how much luck you had

(I should probably split the topic and come up with a Lua script that does all the testing automatically)

[reddeercity]

this is the build I tested with a1ex v1.1 mlv lite.
Magic Lantern Nightly.2016Apr13.5D2212
Camera   : 5D2
Firmware : 212
Changeset: 598036c017f7 (unified) tip
Built on : 2016-04-12 22:11:00 by jenkins@magiclantern.fm-e

@a1ex The 3xCrop mode video frames are pink like the other nighty & mlv lite ,  black level:1024 , 1:1 Ok

1856x1004 @ 23.976p Preview , Canon Liveview , GD enabled (default mode)
I did two set of 10 recording (that all the time I have today) , stopping to write down the captured frames
between recording then continuing to record as fast as I could so the camera didn't cooled down.

Capture frames   Expected frames    Temp in degrees C

1775              1995
1818              1938
2305              2427                49
1834              1943                51
1918              2042                55
2620              2892                56
2411              2616                60
2012              2415                63
1566              1754                60
1574              1674                64

71-72 MB/s write speed
_______________________________________________________

1915              2498                38
1187              1242                45
1822              1958                47
2389              2498                51
2083              2247                54
1747              1918                60
1961              2058                59
1938              2075                60
2178              2351                60
1862              1973                60

71-72 MB/s write speed
__________________________________________________________

3x Crop Mode 1920x1038 23.976

1014              1055                44
1417              1968                47
1217              1300                50
1408              1515                56
1285              1333                56
1382              1586                56
 904               1000                60
1069              1326                60
1113              1265                60
1312              1419                60


74-75.5 MB/s write speed

[Danne]

What I'd like to know: is there any noticeable speed difference between the two versions? How does it compare to the old raw_rec?

@reddeercity
Did you compare your results? I don,t understand how to determine which build is the faster one. Am I missing something?

[a1ex]

@a1ex The 3xCrop mode video frames are pink like the other nighty & mlv lite ,  black level:1024 , 1:1 Ok

Ah, this one is the workaround for showing correct preview. Then, you can safely ignore the black level issue; the fix for this particular case requires a change in the raw recorder as well, because the raw recorder is the one who changes it. In other words, it will be OK in the nightly. For this thread, I'm only interested in recording speed.

The raw_rec from nightly should be already OK.

Black level issue can be safely ignored only with the builds posted earlier in this thread, only on 5D2/50D, and only in 5x zoom (all conditions true). If you encounter it in any other conditions, that's a bug that should be reported.

@reddeercity
Did you compare your results? I don,t understand how to determine which build is the faster one. Am I missing something?

+1.

[a1ex]

Please find a script that analyses a directory with MLV or RAW files and computes some robust statistics - including the ones I've mentioned earlier.

The script should be very fast, so you don't need to actually download the files from the card; just run it and copy the output.

Code: [Select]

#!/bin/bash
#
# Extract number of frames from all RAW/MLV files from
# current directory and compute some statistics on them.
#
# Dependencies: mlv_dump, xxd, octave.

all=""

# MLV files are processed with mlv_dump
for f in `ls *.MLV`; do
    frames=$(mlv_dump $f | grep -Po "(?<=Processed ).*(?=video frames)")
    all="$all $frames"
    echo "$f: $frames"
done

# RAW files are processed with this trick from dfort:
# http://www.magiclantern.fm/forum/index.php?topic=16054.msg163309#msg163309
# This supports up to 4 extra chunks; you can add as many as you need.
for f in `ls *.RAW`; do
    r00=`basename $f .RAW`.R00
    r01=`basename $f .RAW`.R01
    r02=`basename $f .RAW`.R02
    r03=`basename $f .RAW`.R03
    if [ -e $r00 ]; then f=$r00; fi
    if [ -e $r01 ]; then f=$r01; fi
    if [ -e $r02 ]; then f=$r02; fi
    if [ -e $r03 ]; then f=$r03; fi
    frames=$(printf "%d" $(printf "%.s 0x%s %.s" `xxd -s -0xb4 -l 4 -p -e $f`))
    all="$all $frames"
    printf "$f:%5d frames\n" $frames
done

# print the five number summary
octave --eval "quartiles = prctile([$all])"

# also print median, MAD, IQM, IQR
# MAD and IQR scaled to match stdev on Gaussian distribution
octave --eval " \
    x = [$all]; \
    med = median(x); \
    mad = median(abs(x - median(x))); \
    iqm = mean(x(x>=prctile(x,25) & x<=prctile(x,75))); \
    iqr = diff(prctile(x, [25 75])); \
    printf('Median/MAD: %d +/- %d\n', round(med), round(mad * 1.4826)); \
    printf('IQM/IQR   : %d +/- %d\n', round(iqm), round(iqr / 1.349));"


Tested on Linux, should run on Mac as well.

To run it, copy it somewhere (I saved it to ~/raw_stats.sh) and give it executable permissions (chmod +x ~/raw_stats.sh).

Sample test run (this is raw_rec from lua_fix branch, should be identical to unified, 5D3, 1920x1280, 25fps, Komputerbay 32GB 1000x):

Code: [Select]

/media/EOS_DIGITAL/DCIM/100EOS5D$ ~/raw_frames.sh
ls: cannot access *.MLV: No such file or directory
M14-1636.RAW:  370 frames
M14-1637.R00: 1377 frames
M14-1638.RAW:  834 frames
M14-1639.R00: 1455 frames
M14-1640.RAW:  446 frames
M14-1641.RAW:  927 frames
M14-1642.RAW:  481 frames
M14-1643.RAW:  737 frames
M14-1644.RAW:  460 frames
M14-1645.RAW:  401 frames
quartiles =

    370    446    609    927   1455

Median/MAD: 609 +/- 321
IQM/IQR   : 648 +/- 357

To compare two test runs, you can simply compare the last two lines, which contain the summary for the entire experiment (these two methods that estimate the same thing - central tendency and variability). Basically, if the difference is too small compared to the variability, we can't really tell than one build is faster than the other.

Where's the threshold? I don't know, I'm not a statistician. If you know, just chime in.


Checking the repatability:

A second test run (10 clips) gave these values:

Code: [Select]

quartiles = 316.00    521.00    620.50   1137.00   1235.00
Median/MAD: 621 +/- 196
IQM/IQR   : 713 +/- 457

A third test run:

Code: [Select]

quartiles = 357    462    620   1040   1522
Median/MAD: 620 +/- 294
IQM/IQR   : 667 +/- 428


[goldenchild9to5]

Great Job & thank you @a1ex and Team Magic Lantern for all the new updates..