Magic Lantern Forum

Developing Magic Lantern => General Development => Topic started by: a1ex on April 10, 2016, 11:36:35 AM

Title: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: a1ex on April 10, 2016, 11:36:35 AM
Topic split from MLV Lite (http://www.magiclantern.fm/forum/index.php?topic=16650.msg164520#msg164520) thread.

I'm trying to squeeze the last bit of performance out of MLV Lite, to make it as fast as (or maybe faster than) the original raw_rec. You can help by testing different builds and checking which one is faster.





Original message:

I made a small experiment with mlv_lite.

In dmilligan's implementation, frames are 4K larger than with the original raw_rec (because of the MLV headers stored for each frame, and also because of alignment issues). In my version, frame sizes are identical to the original raw_rec.

What I'd like to know: is there any noticeable speed difference between the two versions? How does it compare to the old raw_rec?

My build: raw_rec.mo (http://a1ex.magiclantern.fm/bleeding-edge/mlv_lite/raw_rec.mo)
to be compared with the one from first post, and to the old raw_rec.

Caveat: my version does not handle 4GB limit well; I'll fix it only if the speed difference is noticeable.
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Frank7D on April 10, 2016, 09:55:41 PM
a1ex, I tried your raw_rec.mo in my 7D and got an error when turning on camera:

Scanning modules...
Load modules...
  [I] load: RAW_REC.MO
Linking..
tcc: error: undefined symbol 'camera_serial'
  [E] failed to link modules


Build version: Nightly.2015Mar14.7D203
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Licaon_Kter on April 10, 2016, 10:10:52 PM
Quote from: Frank7D on April 10, 2016, 09:55:41 PM
Build version: Nightly.2015Mar14.7D203
2015? Retry with Apr 01 2016 build I guess.
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Frank7D on April 10, 2016, 10:20:37 PM
Yes, that works. Thanks!
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Frank7D on April 11, 2016, 02:39:20 AM
I tried the new a1ex raw_rec and the one from the front page. I didn't see a big difference, but I won't report results in detail just yet because I ran into an issue that might need to be addressed first:

The raw_rec from the front page produced mlv files that opened in mlvfs but wouldn't through mlrawviewer.

The raw_rec from a1ex produced mlv files that would not open in mlvfs (old Pismo pc version) or through mlrawviewer. Since that's how I look at mlvs, I couldn't examine them.

I'm using 1.1.3 mlrawviewer. When I can I'll get 1.3.3 to see if it works better.
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: reddeercity on April 11, 2016, 06:12:15 AM
@A1ex , thanks for the new test mlv lite build . Ran the raw_rec on 5D2 on magiclantern-Nightly.2016Apr11.5D2212.
Not problem with the module ,  normal operation so far . Recorded 28GB in different lengths some with spanning files .
This was just a quick test , well need to do a more in depth testing , but at this point there is a significant increase in write speed .
Test resolution was 1856x1004 @ 23.976p to CF Lexar 64GB 1066x card .

This was my test with mlv lite from dmilligan build Re: MLV Lite Reply #44  (http://www.magiclantern.fm/forum/index.php?topic=16650.msg162823#msg162823)
1856x1004 @ 23.976 with GD (global draw)  enabled  - 1134 Frames
1856x1004 @ 23.976 with GD disabled - continuous until full (74.5MB/s write speed)
With A1ex build ,
1856x1004 @ 23.976 with GD enabled , (default configuration) 1990 frames (800 more frames)
1856x1004 @ 23.976 with GD disabled , recorded continuous but had no idea what it was , all I know it span more then 2 files
But was flying blind so I stop recording after a few minutes. 
It would but very nice if the over lays could be disabled like Full MLV when recording raw with just the file data rate & buffer info.

Edit: I wasn't sure before  but to confirm ,  5 Spanning files (5700 frames) from 5D2 & all look very good no corruption extracted with  chmee's  raw2cdng ver, 1.7.4
This PC  program never fails to read mlv/raw  ;)

Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: a1ex on April 11, 2016, 08:56:54 AM
Indeed, the corruption when spanning files is minor (does not result in data loss) - it's very similar to this one (http://www.magiclantern.fm/forum/index.php?topic=16650.msg162583#msg162583).

Since the method is promising, I've committed the source (https://bitbucket.org/hudson/magic-lantern/commits/6b986781ebbd129ac7ea2452f527d44553804f23).
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: a1ex on April 11, 2016, 10:02:27 AM
Here's another experiment:

raw_rec.mo (http://a1ex.magiclantern.fm/bleeding-edge/mlv_lite/edmac64/raw_rec.mo)

I'm trying to see if the EDMAC accepts alignments lower than 4096 bytes, and - at least on 5D3 - it accepted 64 just fine. That means, we can align everything at 512 bytes for optimal write speed, and we can use dmilligan's method of placing VIDF headers before the frame without increasing frame size.

How does this one perform in terms of speed?

This build should do proper cleanup for spanning files (dmilligan's method).
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Ottoga on April 11, 2016, 12:10:03 PM
Tried it on my 7D.
Global draw off, max resolution and again at 5x. Continuous recording ok.  With global draw on and 5x recordings recording stopped after approx 30 seconds.

Output was 3 x multi chunk  files and 1 x single chunk.

All were able to be opened and processed in MLVP.
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: reddeercity on April 12, 2016, 01:52:21 AM
Quote from: a1ex on April 11, 2016, 10:02:27 AM
Here's another experiment:raw_rec.mo (http://a1ex.magiclantern.fm/bleeding-edge/mlv_lite/edmac64/raw_rec.mo)
How does this one perform in terms of speed?
About the same or little faster , did change the preview option to see if that increased write speed with out frame drop.
1856x1004 @ 23.976 ,  Canon Preview , SRM job memory , Small hacks. Average Write Speed was 70.5MB/s
1908 frames -- expected frames 2240 (card cold)
2370 frames -- expected frames 2501
1665 frames -- expected frames 1901 (After a reboot to check card space)
1566 frames -- expected frames 1650
2330 frames -- expected frames 2515
1899 frames -- expected frames 2090 (Auto Preview)
1972 frames -- expected frames 2170 (Auto Preview)
2440 frames -- expected frames 2610 (ML Grey/Hacked Preview)
  823 frames -- expected frames   900 (ML Grey/Hacked Preview with half shutter pressed for color preview)

3x Crop mode
1920x1038 @ 23.976 , Canon Preview : 1468 frames -- expected frames 1566
1920x1038 @ 23.976 , Hacked Preview : 3166 frames -- expected frames 3900
2048x930   @ 23.976 , Auto Preview    : Continuous (I let go for about 5minutes )
2048x1024 @ 23.976 , ML Grey Preview : 863 frames expected 960 frames.

Quote from: a1ex on April 11, 2016, 10:02:27 AM
This build should do proper cleanup for spanning files (dmilligan's method).
Seems to be ok , now I can extract with MLVFS & Prismo , 1:1 images are ok but the 3x Crop mode are
pink , wrong black level according to exiftool : Black Level : 1023

ExifTool Version Number         : 10.09
File Name                       : M11-1634_000000.dng
Directory                       : D:/New folder (5)
File Size                       : 3.7 MB
File Modification Date/Time     : 2016:04:11 16:33:54-06:00
File Access Date/Time           : 2016:04:11 16:47:38-06:00
File Creation Date/Time         : 2016:04:11 16:47:38-06:00
File Permissions                : rw-rw-rw-
File Type                       : DNG
File Type Extension             : dng
MIME Type                       : image/x-adobe-dng
Exif Byte Order                 : Little-endian (Intel, II)
Subfile Type                    : Full-resolution Image
Image Width                     : 2048
Image Height                    : 930
Bits Per Sample                 : 16
Compression                     : Uncompressed
Photometric Interpretation      : Color Filter Array
Fill Order                      : Normal
Make                            : Canon
Camera Model Name               : Canon EOS 5D Mark II
Strip Offsets                   : 65536
Orientation                     : Horizontal (normal)
Samples Per Pixel               : 1
Rows Per Strip                  : 930
Strip Byte Counts               : 3809792
Planar Configuration            : Chunky
Software                        : MLVFS
Modify Date                     : 2016:03:11 16:33:54
CFA Repeat Pattern Dim          : 2 2
CFA Pattern 2                   : 0 1 1 2
Exposure Time                   : 1/50
F Number                        : 2.2
ISO                             : 800
Sensitivity Type                : ISO Speed
Exif Version                    : 0230
Subject Distance                : 63 m
Focal Length                    : 50.0 mm
Lens Model                      : 50mm
DNG Version                     : 1.4.0.0
Unique Camera Model             : Canon EOS 5D Mark II
Linearization Table             : (Binary data 87187 bytes, use -b option to extract)
Black Level                     : 1023
White Level                     : 15000
Default Crop Origin             : 0 0
Default Crop Size               : 2048 930
Color Matrix 1                  : 0.4716 0.0603 -0.083 -0.7798 1.5474 0.248 -0.1
496 0.1937 0.6651
As Shot Neutral                 : 0.4580008629 1 0.5775139181
Baseline Exposure               : 0
Calibration Illuminant 1        : D65
Active Area                     : 0 0 930 2048
Frame Rate                      : 23.976
Baseline Exposure Offset        : 0
Aperture                        : 2.2
CFA Pattern                     : [Red,Green][Green,Blue]
Image Size                      : 2048x930
Megapixels                      : 1.9
Shutter Speed                   : 1/50
Focal Length                    : 50.0 mm
Light Value                     : 4.9

In 1:1 mode the black level looks normal
Black Level   : 1792
Sample frame from each in my dropbox
3x crop wrong black level 2048x930 (https://www.dropbox.com/s/kz4dqxrjj0k9ybg/M11-1634_000000.dng?dl=0) And 1:1 1856x1004 Cdng correct black level  (https://www.dropbox.com/s/fw5zhyzylqwujnr/M11-1651_000084.dng?dl=0)  I remember back a few years when this was a issue before MLV , looks the same .

All tests were with Overlays enabled .
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: a1ex on April 12, 2016, 11:10:23 AM
For the black level issue, can you try the module on top of latest nightly?
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Steve_FR on April 12, 2016, 10:34:34 PM
Eos M comparing 4/12 nightly and a1ex's newest raw_rec.mo (MLV Lite 1.1):

Average of 160 frames (5 sample test) at 1728x736 5x @ 23.97 fps (4/12 raw-rec.mo mlv lite 1.1)

Average of 228 frames (5 sample tests) at 1728x736 5x @ 23.97 (raw 1.0 nightly)

Raw rec 1.0 still offering slightly higher performance on Eos M. ML's displayed megabytes/sec was at around 36mb/s for MLV Lite 1.1, and around 40.5mb/s for Raw 1.0. I don't know if that's important all, but thought I'd mention it.
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: DeafEyeJedi on April 13, 2016, 05:36:22 AM
Quote from: a1ex on April 11, 2016, 10:02:27 AM
Here's another experiment:

raw_rec.mo (http://a1ex.magiclantern.fm/bleeding-edge/mlv_lite/edmac64/raw_rec.mo)

This link isn't working (just gives me a text file) when the link from this post (http://www.magiclantern.fm/forum/index.php?topic=16650.msg165538#msg165538post) works but that wouldn't be the latest one to experiment, correct?
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Walter Schulz on April 13, 2016, 05:55:43 AM
@DeafEyeJedi: Open link's context menu and select "Save as ..."
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: DeafEyeJedi on April 13, 2016, 06:05:16 AM
That did it and Thanks @Walter Schulz!
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: reddeercity on April 13, 2016, 08:03:25 AM
Quote from: a1ex on April 12, 2016, 11:10:23 AM
For the black level issue, can you try the module on top of latest nightly?
Ok , so I'm not to sure what you are asking so I did some test with older nightly builds (maybe that's what you want ?)
magiclantern-Nightly.2016Jan14.5D2212 , magiclantern-Nightly.2016Feb13.5D2212 , magiclantern-Nightly.2016Apr01.5D2212 , magiclantern-Nightly.2016Apr11.5D2212 .
They all product pink frames with black level of 1024 , for Jan, Feb & Apr 01 I used dmilligan's mlv lite V.1.0 . & use your ver 1.1 on the apr11 build.
I also did a quick test with the old raw with the builds I tested and all produce normal black level of 1792 in 3xCrop mode @ 2048x930 23.976 .
But one thing catch me off guard when I though I would try the mlv lite module from dmilligan's mlv lite V.1.0 in 1:1 mode @ 1856x1004 23.976
with the magiclantern-Nightly.2016Jan14.5D2212  just to see how long I could record to get a comparison to the latest nightly (Apr/11/2016) with your build a1ex , and boy did that surprise me
I got 2409 frames out of expected frames of 2600  ???
that the most yet ! just by a little but my question is what happen between Jan14 build and Apr11 build to slow thing down of is there something in play here I don't understand.

Edit: I just remembered that I report the 3x crop pink frame problem in the raw2cdng thread as I was using chmee's app , never thought it was a module issue
http://www.magiclantern.fm/forum/index.php?topic=5618.msg162891#msg162891
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: a1ex on April 13, 2016, 08:23:53 AM
Latest = opposite of older. Download 2016Apr12 (latest when I asked you to run the test), replace the module, and try it.

Look for "black" in the changelog for that build.

If you notice any speed difference between 2016Jan14 and 2016Feb13, it's either pure luck, or the inherent noise in your testing method. The only change was this (https://bitbucket.org/hudson/magic-lantern/changeset/2a1f972cdbd87e77059539622fd4308618fa9ae6).
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Danne on April 13, 2016, 11:20:10 AM
Recorded 10 files of each setting on an eos 7D. Printed out highest performance achieved. Used latest nightly build (magiclantern-Nightly.2016Apr13.7D203) 598036c

RAW
60fps  1728x584 (focus peaking on) highest achieved. 338 frames
mlv_lite(dmilligan)
60fps  1728x584 (focus peaking on) highest achieved. 336 frames
mlv_lite(a1ex)
60fps  1728x584 (focus peaking on) highest achieved. 338 frames


RAW
25fps  1728x972 (focus peaking on) highest achieved.  822 frames
mlv_lite(dmilligan)
25fps  1728x972 (focus peaking on) highest achieved.  896 frames
mlv_lite(a1ex)
25fps  1728x972 (focus peaking on) highest achieved.  899 frames


Retested RAW 25fps  1728x972 one extra time to see what it would achieve with card at its warmest.
Last run(best out of ten)
RAW
25fps  1728x972 (focus peaking on) highest achieved.  872 frames
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Frank7D on April 13, 2016, 03:55:41 PM
I tried testing an earlier module but had an issue reading the MLVs (see April 11 post).
Before I try testing the latest, I have a question:
Is there a recommended protocol for testing (minimum number of clips, etc.) that will allow us to see meaningful differences in the results as opposed to random variation?
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: a1ex on April 13, 2016, 04:53:25 PM
The differences are amplified when the write speed needed is just a little above what can be achieved. You could aim for about 2000 frames until it stops.

To get consistent results, try to start from the same initial conditions (e.g. card formatted in camera, then restarted, maybe warmed up, whatever other settings you prefer). Just do the same in all the tests, so the numbers can be comparable.

To evaluate the variations, you could record 10 clips, write down number of frames from each one, and compute:
- median
- MAD
- min
- max

Why median/mad and not mean/stdev? It's hard to make a lot of experiments, and the distribution is not exactly Gaussian - in particular, the outliers can be quite far from the average. Mean and stdev are easily influenced by outliers, while median and MAD are much more robust.

Min and max are mostly for evaluating how much luck you had :P

(I should probably split the topic and come up with a Lua script that does all the testing automatically)
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: reddeercity on April 14, 2016, 03:49:26 AM
this is the build I tested with a1ex v1.1 mlv lite.
Magic Lantern Nightly.2016Apr13.5D2212
Camera   : 5D2
Firmware : 212
Changeset: 598036c017f7 (unified) tip
Built on : 2016-04-12 22:11:00 by [email protected]

@a1ex The 3xCrop mode video frames are pink like the other nighty & mlv lite ,  black level:1024 , 1:1 Ok

1856x1004 @ 23.976p Preview , Canon Liveview , GD enabled (default mode)
I did two set of 10 recording (that all the time I have today) , stopping to write down the captured frames
between recording then continuing to record as fast as I could so the camera didn't cooled down.

Capture frames   Expected frames    Temp in degrees C

1775              1995
1818              1938
2305              2427                49
1834              1943                51
1918              2042                55
2620              2892                56
2411              2616                60
2012              2415                63
1566              1754                60
1574              1674                64

71-72 MB/s write speed
_______________________________________________________

1915              2498                38
1187              1242                45
1822              1958                47
2389              2498                51
2083              2247                54
1747              1918                60
1961              2058                59
1938              2075                60
2178              2351                60
1862              1973                60

71-72 MB/s write speed
__________________________________________________________

3x Crop Mode 1920x1038 23.976

1014              1055                44
1417              1968                47
1217              1300                50
1408              1515                56
1285              1333                56
1382              1586                56
904               1000                60
1069              1326                60
1113              1265                60
1312              1419                60


74-75.5 MB/s write speed
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Danne on April 14, 2016, 10:42:11 AM
QuoteWhat I'd like to know: is there any noticeable speed difference between the two versions? How does it compare to the old raw_rec?

@reddeercity
Did you compare your results? I don,t understand how to determine which build is the faster one. Am I missing something?
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: a1ex on April 14, 2016, 12:33:48 PM
Quote from: reddeercity on April 14, 2016, 03:49:26 AM
@a1ex The 3xCrop mode video frames are pink like the other nighty & mlv lite ,  black level:1024 , 1:1 Ok

Ah, this one is the workaround for showing correct preview. Then, you can safely ignore the black level issue; the fix for this particular case requires a change in the raw recorder as well, because the raw recorder is the one who changes it. In other words, it will be OK in the nightly. For this thread, I'm only interested in recording speed.

The raw_rec from nightly should be already OK.

Black level issue can be safely ignored only with the builds posted earlier in this thread, only on 5D2/50D, and only in 5x zoom (all conditions true). If you encounter it in any other conditions, that's a bug that should be reported.

Quote from: Danne on April 14, 2016, 10:42:11 AM
@reddeercity
Did you compare your results? I don,t understand how to determine which build is the faster one. Am I missing something?

+1.
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: a1ex on April 14, 2016, 05:15:49 PM
Please find a script that analyses a directory with MLV or RAW files and computes some robust statistics - including the ones I've mentioned earlier.

The script should be very fast, so you don't need to actually download the files from the card; just run it and copy the output.


#!/bin/bash
#
# Extract number of frames from all RAW/MLV files from
# current directory and compute some statistics on them.
#
# Dependencies: mlv_dump, xxd, octave.

all=""

# MLV files are processed with mlv_dump
for f in `ls *.MLV`; do
    frames=$(mlv_dump $f | grep -Po "(?<=Processed ).*(?=video frames)")
    all="$all $frames"
    echo "$f: $frames"
done

# RAW files are processed with this trick from dfort:
# http://www.magiclantern.fm/forum/index.php?topic=16054.msg163309#msg163309
# This supports up to 4 extra chunks; you can add as many as you need.
for f in `ls *.RAW`; do
    r00=`basename $f .RAW`.R00
    r01=`basename $f .RAW`.R01
    r02=`basename $f .RAW`.R02
    r03=`basename $f .RAW`.R03
    if [ -e $r00 ]; then f=$r00; fi
    if [ -e $r01 ]; then f=$r01; fi
    if [ -e $r02 ]; then f=$r02; fi
    if [ -e $r03 ]; then f=$r03; fi
    frames=$(printf "%d" $(printf "%.s 0x%s %.s" `xxd -s -0xb4 -l 4 -p -e $f`))
    all="$all $frames"
    printf "$f:%5d frames\n" $frames
done

# print the five number summary
octave --eval "quartiles = prctile([$all])"

# also print median, MAD, IQM, IQR
# MAD and IQR scaled to match stdev on Gaussian distribution
octave --eval " \
    x = [$all]; \
    med = median(x); \
    mad = median(abs(x - median(x))); \
    iqm = mean(x(x>=prctile(x,25) & x<=prctile(x,75))); \
    iqr = diff(prctile(x, [25 75])); \
    printf('Median/MAD: %d +/- %d\n', round(med), round(mad * 1.4826)); \
    printf('IQM/IQR   : %d +/- %d\n', round(iqm), round(iqr / 1.349));"



Tested on Linux, should run on Mac as well.

To run it, copy it somewhere (I saved it to ~/raw_stats.sh) and give it executable permissions (chmod +x ~/raw_stats.sh).

Sample test run (this is raw_rec from lua_fix branch, should be identical to unified, 5D3, 1920x1280, 25fps, Komputerbay 32GB 1000x):

/media/EOS_DIGITAL/DCIM/100EOS5D$ ~/raw_frames.sh
ls: cannot access *.MLV: No such file or directory
M14-1636.RAW:  370 frames
M14-1637.R00: 1377 frames
M14-1638.RAW:  834 frames
M14-1639.R00: 1455 frames
M14-1640.RAW:  446 frames
M14-1641.RAW:  927 frames
M14-1642.RAW:  481 frames
M14-1643.RAW:  737 frames
M14-1644.RAW:  460 frames
M14-1645.RAW:  401 frames
quartiles =

    370    446    609    927   1455

Median/MAD: 609 +/- 321
IQM/IQR   : 648 +/- 357


To compare two test runs, you can simply compare the last two lines, which contain the summary for the entire experiment (these two methods that estimate the same thing - central tendency and variability). Basically, if the difference is too small compared to the variability, we can't really tell than one build is faster than the other.

Where's the threshold? I don't know, I'm not a statistician. If you know, just chime in.




Checking the repatability:

A second test run (10 clips) gave these values:

quartiles = 316.00    521.00    620.50   1137.00   1235.00
Median/MAD: 621 +/- 196
IQM/IQR   : 713 +/- 457


A third test run:

quartiles = 357    462    620   1040   1522
Median/MAD: 620 +/- 294
IQM/IQR   : 667 +/- 428
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: goldenchild9to5 on April 14, 2016, 06:24:34 PM
Great Job & thank you @a1ex and Team Magic Lantern for all the new updates..
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Frank7D on April 14, 2016, 09:36:58 PM
I just made a spreadsheet (copied it off a helpful website) that compares two data sets and tells us if their means are statistically significantly different (with, say, 95% confidence).

It requires every value measured to be entered, so for instance the number of frames in each clip.

I'll try it on any test results someone wants (I can make it available too, if it proves useful).
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Danne on April 14, 2016, 10:30:13 PM
Great stuff. I couldn,t get the script to work on my mac A1ex. I didn,t work on it very hard and I,m squeezing in these tests between other things. For what it,s worth I started building a small script to do some indicating calculations, median, out of ten files. From my pragmatic testing I,d say there is maybe a slight edge to A1ex build. A lot of the clips went to 34 seconds whereas dmilligan build stopped at 33 seconds. Here are the results and the script.

First run(A1ex mlv_lite)
Canon EOS 5D Mark III

Frames(each file)
733
1012
906
981
991
975
993
692
966
973

Max
1012

Min
733

Median
922.2


Other metadata
Fps
29.97

Resolution
1920x1080


Second run(dmilligan mlv_lite
ERR:1 md:0x 0 ml:0

Frames(each file)
973
965
957
961
984
970
972
963
976
984

Max
984

Min
957

Median
970.5


Other metadata
Fps
29.97

Resolution
1920x1080



Third run(A1ex mlv_lite)
Canon EOS 5D Mark III

Frames(each file)
1029
1020
1015
1029
1014
1017
1007
1035
997
999

Max
1035

Min
997

Median
1016.2


Other metadata
Fps
29.97

Resolution
1920x1080



Fourth run(dmilligan mlv_lite)
ERR:1 md:0x 0 ml:0

Frames(each file)
787
961
961
966
966
1006
1003
1010
1003
1003

Max
1010

Min
787

Median
966.6


Other metadata
Fps
29.97

Resolution
1920x1080



The script. Put in a .command file and double click next to 10 mlv files. Gotta fix the min value. Not working properly.

#!/bin/bash
workingDir=`dirname "$0"`
cd "${workingDir}"

#Camera name
#Frames
#Max/Min
#Median
#Other metadata
#Fps
#Resolution

lock=$(ls *.MLV | awk 'FNR == 10 {print; exit}')

if ! [ -n "$lock" ]
then
clear
echo "
No cheating. 10 MLV files required"
sleep 2
exit 0
fi

#Camera name
mlv_dump -m -v $(ls *.MLV | awk 'FNR == 1 {print}') | grep 'Camera Name:' | awk '{ print $3,$4,$5,$6,$7,$8; exit }' | cut -d "'" -f2  > mlv_lite.txt


#Frames
echo "
Frames(each file)" >> mlv_lite.txt
mlv_dump -m -v $(ls *.MLV | awk 'FNR == 1 {print}') | grep 'Processed' | awk '{ print $2; exit }' >> mlv_lite.txt
mlv_dump -m -v $(ls *.MLV | awk 'FNR == 2 {print}') | grep 'Processed' | awk '{ print $2; exit }' >> mlv_lite.txt
mlv_dump -m -v $(ls *.MLV | awk 'FNR == 3 {print}') | grep 'Processed' | awk '{ print $2; exit }' >> mlv_lite.txt
mlv_dump -m -v $(ls *.MLV | awk 'FNR == 4 {print}') | grep 'Processed' | awk '{ print $2; exit }' >> mlv_lite.txt
mlv_dump -m -v $(ls *.MLV | awk 'FNR == 5 {print}') | grep 'Processed' | awk '{ print $2; exit }' >> mlv_lite.txt
mlv_dump -m -v $(ls *.MLV | awk 'FNR == 6 {print}') | grep 'Processed' | awk '{ print $2; exit }' >> mlv_lite.txt
mlv_dump -m -v $(ls *.MLV | awk 'FNR == 7 {print}') | grep 'Processed' | awk '{ print $2; exit }' >> mlv_lite.txt
mlv_dump -m -v $(ls *.MLV | awk 'FNR == 8 {print}') | grep 'Processed' | awk '{ print $2; exit }' >> mlv_lite.txt
mlv_dump -m -v $(ls *.MLV | awk 'FNR == 9 {print}') | grep 'Processed' | awk '{ print $2; exit }' >> mlv_lite.txt
mlv_dump -m -v $(ls *.MLV | awk 'FNR == 10 {print}') | grep 'Processed' | awk '{ print $2; exit }' >> mlv_lite.txt

#Max/Min
echo "
Max" >> mlv_lite.txt
grep -Eo '[0-9]+' mlv_lite.txt | sort -rn | head -n 1 >> mlv_lite.txt

echo "
Min" >> mlv_lite.txt
grep -Eo '[0-9]+' mlv_lite.txt | sort | head -n 1 >> mlv_lite.txt

echo "
Median" >> mlv_lite.txt

echo $(awk 'NR == 04 {print; exit}' mlv_lite.txt)/10+$(awk 'NR == 05 {print; exit}' mlv_lite.txt)/10+$(awk 'NR == 06 {print; exit}' mlv_lite.txt)/10+$(awk 'NR == 07 {print; exit}' mlv_lite.txt)/10+$(awk 'NR == 08 {print; exit}' mlv_lite.txt)/10+$(awk 'NR == 09 {print; exit}' mlv_lite.txt)/10+$(awk 'NR == 10 {print; exit}' mlv_lite.txt)/10+$(awk 'NR == 11 {print; exit}' mlv_lite.txt)/10+$(awk 'NR == 12 {print; exit}' mlv_lite.txt)/10+$(awk 'NR == 13 {print; exit}' mlv_lite.txt)/10 | bc -l | awk ' sub("\\.*0+$","") ' >> mlv_lite.txt


#Other metadata
#Fps
echo "

Other metadata
Fps" >> mlv_lite.txt
mlv_dump -m -v $(ls *.MLV | awk 'FNR == 1 {print}') | awk '/FPS/ { print $3; exit }' | awk ' sub("\\.*0+$","") ' >> mlv_lite.txt
#Resolution
echo "
Resolution" >> mlv_lite.txt
mlv_dump -m -v $(ls *.MLV | awk 'FNR == 1 {print}') | awk '/Res/ { print $2; exit }' >> mlv_lite.txt


open mlv_lite.txt

osascript -e 'tell application "Terminal" to close first window' & exit
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: a1ex on April 14, 2016, 10:39:11 PM
Quote from: Frank7D on April 14, 2016, 09:36:58 PM
I just made a spreadsheet (copied it off a helpful website) that compares two data sets and tells us if their means are statistically significantly different (with, say, 95% confidence).

It requires every value measured to be entered, so for instance the number of frames in each clip.

Sounds great.

I'm working on a Lua script that would do the following test (all happens on the camera):

- have a few different versions of raw_rec in a directory, from which it will select a random module
- format the card (simulated keypresses in Canon menu)
- restart the camera (which will also reload the new module)
- record 10 clips
- write down number of frames from each clip to a log file
- repeat the test overnight

So, if all goes well, I'll give you a big data set to analyze. One entry (for 10 clips) would look somewhat like this:

Raw_rec version : a32a5c30
Recorded frames : 282,562,1299,966,768,691,796,528,842,588
Quartile summary: 729.5 frames (562...842)


In a nutshell, the script would try different versions (all uploaded on card in advance), record 10 test clips, write down the number of frames, format the card, repeat.
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Frank7D on April 14, 2016, 11:13:37 PM
I just put Danne's results through the spreadsheet. I combined the two sets of a1ex numbers into one and did the same with the dmilligan numbers.

The means for the two (consolidated) sets are amazingly close: a1ex: 969.2 dmilligan: 968.55

Critical T Value (@ 95% confidence): 1.69
Our T -Stat: 0.03

When the T -Stat is less than the Critical T Value (as in this case), that means the difference is not significant (as we could guess from looking at the means).

a1ex, I'm ready for that data whenever it's done.
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: a1ex on April 15, 2016, 01:14:14 AM
Here's the script I'm currently running: rawbench.lua (https://bitbucket.org/hudson/magic-lantern/src/raw_benchmark/scripts/extra/rawbench.lua).

It's not yet compatible with the nightly - only the latest lua_fix branch can handle it at the time of writing. It was a pretty strong test for the scripting API - I was thinking it's almost ready to merge, but this little script revealed a lot of bugs on the file I/O side.

Be careful with this script (it's destructive, it formats your card without asking).
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: reddeercity on April 15, 2016, 03:26:34 AM
Quote from: Danne on April 14, 2016, 10:42:11 AM
@reddeercity
Did you compare your results? I don,t understand how to determine which build is the faster one. Am I missing something?
I did the test with dmilligan mlv lite build in the MLV Lite thread (http://www.magiclantern.fm/forum/index.php?topic=16650.msg162823#msg162823)  so I didn't post any thing.
Plus I didn't have time the other day.

QuoteOk back on topic , I did a more comprehensive test with MLV Lite module on my 5D2 with the updated raw_rec.mo
I use the latest Nightly Build plus I compared mlv lite to full mlv ( use 2 different nighty for full MLV , feb15/2014 &Feb13/2016) and used the Old Raw format for Oct24/2013.
All tests done on CF Lexar 32GB 1066x
MLV Lite: magiclantern-Nightly.2016Feb13.5D2212  dmilligan mlv lite2nd build
1856x1004 @ 23.976 with GD (global draw)  enabled  - 1134 Frames
1856x1004 @ 23.976 with GD disabled - continuous until full (74.5MB/s write speed)
1856x1004 @ 23.976 with GD disabled + HDMI enabled (Ninja HDMI Hard Drive recorder connected) - 3215 Frames
1856x1044 @ 23.976 With GD enabled - 1089 Frames
1856x1044 @ 23.976 With GD disabled - 1662 Frames

The results of a1ex mlv lite test (http://www.magiclantern.fm/forum/index.php?topic=17091.msg165725#msg165725)

3x Crop mode:
1920x1038 @ 23.976 with GD enabled - 1046 Frames (76.9 MB/s Write Speed)
1920x1038 @ 23.976 With GD disabled - 2354 Frames
1920x1076 @ 23.976 With GD enabled - 724 Frames
2048x930 @ 23.976   With GD disabled -  continuous until full (74.8 MB/s Write speed)
2048x1024 @23.976  With GD disabled - 1016 Frames

All said and done a1ex mlv lite build produce 700 to 800 more frames in 1:1 and about 300 frames in 3x crop.

Now I can only speak for the 5D2 , so as you say you are looking for speed . Well  If there was a switch to toggle Overlays off & on
Like full mlv , just the date rate file info stuff then that would just about equal the speed that the Original Raw did back in Oct/24/2013
Quote1856x1004 @23.976 + HDMI enabled (Ninja HDMI Hard Drive recorder connected) (74.6MB/s Write speed ) - continuous until full
1856x1044 @ 23.976  (77.5MB/s Write speed) -  continuous until full
1872x1012 @ 23.976 (76 MB/s write Speed)   -  continuous until full
1920x1038 @ 23.976 (79.7 MB/s write speed) - continuous until full
This is what I compare All Nightly Builds too , it's the Gold standard for 5D2.
So I look for the largest Continuous resolution the Camera can do that why I still use 1872x936 mlv full.
My 2 cents

Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: a1ex on April 15, 2016, 08:03:50 AM
Looks like I forgot to initialize the random seed, so the script picked the same build every time.

@Frank7D: how significant is this difference?


Recorded frames : 379,1260,1307,1306,1061,892,572,1022,568,375
Quartile summary: 957 frames (568...1260)



Recorded frames : 329,900,1010,471,568,518,840,1257,407,471
Quartile summary: 543 frames (471...900)


(you guessed it - best and worst case runs with the same build, test done by the script)
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Frank7D on April 15, 2016, 04:08:30 PM
Best case mean: 874
Worst case mean: 677

Critical T: 1.75
T -Stat: 1.29

Since The Critical T is larger, the difference in the means is not significant.

Intuitively, if you look at how much the number of frames varies within each set, it seems like there's a large random factor. You probably would need a larger number of data points to get a meaningful comparison. That doesn't mean each run needs to be larger though. You could just combine several runs of ten.
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Danne on April 15, 2016, 04:47:32 PM
I wonder if It would help if I ran let,s say 40 recordings in a row and gave you the numbers from both mlv_lite versions? Now if A1ex is suddenly serving us another raw_rec.mo to test it would be a little painful  :P
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Frank7D on April 15, 2016, 05:46:24 PM
I think as long as the camera model and settings used are the same, we can combine results from multiple testers (so no one person has to do so much). After all, we would like to know how the builds perform in general, so multiple cameras would be just fine, it seems to me.
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: a1ex on April 15, 2016, 06:22:04 PM
Got the first dataset. The script crashed at some point, so it's not as complete as I would like.


Raw_rec version : ML/MODULES/RAW_REC/14E2991.MO
Recorded frames : 286,759,717,1185,1273,1264,795,865,613,634
Raw_rec version : ML/MODULES/RAW_REC/54E90DE.MO
Recorded frames : 419,1261,1314,1319,961,574,453,1262,613,562
Raw_rec version : ML/MODULES/RAW_REC/60247BB.MO
Recorded frames : 374,717,922,997,624,518,614,581,1116,812
Raw_rec version : ML/MODULES/RAW_REC/CA5315A.MO
Recorded frames : 558,1319,876,968,1329,1311,1262,511,574,-1,1141549314
Raw_rec version : ML/MODULES/RAW_REC/DE4E1D1.MO
Recorded frames : 349,781,700,304,1260,1260,1240,1264,674,635
Raw_rec version : ML/MODULES/RAW_REC/14E2991.MO
Recorded frames : 449,640,1321,1333,1330,1313,1306,848,-1,1075910152
Raw_rec version : ML/MODULES/RAW_REC/C350071.MO
Recorded frames : 249,736,1276,1261,1270,1260,493,807,418,824
Raw_rec version : ML/MODULES/RAW_REC/14E2991.MO
Recorded frames : 916,1300,1318,1041,849,776,1265,767,538,-1,-190905046
Raw_rec version : ML/MODULES/RAW_REC/C350071.MO
Recorded frames : 778,1188,826,831,403,741,1263,1124,549,755
Raw_rec version : ML/MODULES/RAW_REC/CA5315A.MO
Recorded frames : 892,1114,416,696,1319,1310,1302,898,634,-1,-2103303980
Raw_rec version : ML/MODULES/RAW_REC/14E2991.MO
Recorded frames : 372,1037,700,1010,1260,1260,880,681,816,716
Raw_rec version : ML/MODULES/RAW_REC/CA5315A.MO
Recorded frames : 394,1293,1321,1331,1080,610,615,1248,315,-1,-2078138256
Raw_rec version : ML/MODULES/RAW_REC/60247BB.MO
Recorded frames : 361,995,812,920,640,493,532,623,756,987
Raw_rec version : ML/MODULES/RAW_REC/14E2991.MO
Recorded frames : 1280,1318,1299,1317,1313,1260,437,531,-1,603720293
Raw_rec version : ML/MODULES/RAW_REC/C350071.MO
Recorded frames : 905,963,589,1081,656,1164,1271,1236,666,270
Raw_rec version : ML/MODULES/RAW_REC/60247BB.MO
Recorded frames : 441,790,892,1094,1078,1077,971,576,548,664
Raw_rec version : ML/MODULES/RAW_REC/C350071.MO
Recorded frames : 1248,1316,1307,1317,951,1044,1149,553
Raw_rec version : ML/MODULES/RAW_REC/60247BB.MO
Recorded frames : 315,946,1089,812,669,662,704,621,782,567
Raw_rec version : ML/MODULES/RAW_REC/14E2991.MO
Recorded frames : 902,910,1031,954,1308,1319,1306,678,300,-1,1825290946
Raw_rec version : ML/MODULES/RAW_REC/DE4E1D1.MO
Recorded frames : 601,788,1133,1272,1268,1266,904,830,585,-1,-1994319992
Raw_rec version : ML/MODULES/RAW_REC/60247BB.MO
Recorded frames : 382,1003,857,855,775,812,574,475,683,684
Raw_rec version : ML/MODULES/RAW_REC/14E2991.MO
Recorded frames : 1123,1330,1311,1315,1311,1168,582,692
Raw_rec version : ML/MODULES/RAW_REC/C350071.MO
Recorded frames : 407,1153,1219,469,744,1162,1266,1260,665,459
Raw_rec version : ML/MODULES/RAW_REC/CA5315A.MO
Recorded frames : 820,708,586,928,1322,1302,1259,799,1036,-1,1652791484
Raw_rec version : ML/MODULES/RAW_REC/14E2991.MO
Recorded frames : 472,814,912,1266,1260,1263,915,707,686,-1,-926858671
Raw_rec version : ML/MODULES/RAW_REC/54E90DE.MO
Recorded frames : 671,1266,1260,1263,463,827,940,1067,553,-1,1150832025
Raw_rec version : ML/MODULES/RAW_REC/14E2991.MO
Recorded frames : 999,1267,1125,728,575,1225,1122,706,818,-1,855478615
Raw_rec version : ML/MODULES/RAW_REC/14E2991.MO
Recorded frames : 916,1197,744,517,942,1158,1261,697,719,-1,-271791549
Raw_rec version : ML/MODULES/RAW_REC/60247BB.MO
Recorded frames : 640,826,603,625,737,900,1010,900,777,739


The obvious outliers at the end seem to indicate a bug when card gets full.
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Frank7D on April 15, 2016, 06:56:38 PM
Thanks a1ex. One question:
Will I be seeing those same module names in the future as well? (just wondering if it's worth my time to build in something in the spreadsheet to extract and sort the info according to those specific names).
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: a1ex on April 15, 2016, 07:05:45 PM
No, I'll change them, but the name format will stay the same. I think it's worth writing a script that would parse a log file in this format, group data by build version, sort them by performance and compute how significant the differences are.
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Frank7D on April 15, 2016, 11:12:49 PM
I modified my spreadsheet to automate more things but it's still more manual than it needs to be. I'll try to refine it. In any case, I should be faster next time.

a1ex, I put your data sets in (excluding the outliers you mentioned) and of the six modules the only meaningful difference in the means was with 60247BB.MO, which all the other five modules beat.

So, tied for first are:
14E2991.MO (mean=963, set size=89)
CA5315A.MO (mean=952, set size=36)
C350071.MO (mean=907, set size=48)
54E90DE.MO (mean=897, set size=19)
DE4E1D1.MO (mean=901, set size=19)

and in sole possession of last place is:
60247BB.MO (mean=742, set size=60)
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: a1ex on April 16, 2016, 02:37:46 PM
Thanks - the slower one had a mistake (entire frame was aligned at 64 bytes instead of 512; I wanted to align only the EDMAC). The other ones had various fine-tunings that don't seem to make a difference.

Here's another set (I've ordered the lines manually and dropped the last two numbers from each line, since that's where card may get full).


Raw_rec version : ML/MODULES/RAW_REC/36B111D.MO
832,1318,1299,1282,1151,1209,1177
966,1262,1211,1244,1231,1276,1266
1039,1285,857,1283,1291,1285,1280
941,1257,959,1283,1288,1283,1292
1073,1153,1163,1283,1297,1284,1132
707,1289,1260,1282,1066,1215,919

Raw_rec version : ML/MODULES/RAW_REC/E203CF9.MO
778,1260,976,1168,1262,1260,1274
1135,1069,1263,1284,1294,1260,1066
932,1159,1136,1040,1260,1265,1249
971,1035,990,1268,1260,1261,1147
1128,1161,1223,1038,1000,1260,1116

Raw_rec version : ML/MODULES/RAW_REC/14E2991.MO
1027,1116,1037,1318,1308,1282,1260
669,1259,1251,1166,1007,1169,1265
837,1172,1184,1148,818,1268,1256
890,1154,875,1268,1258,1263,949
916,1170,1171,1266,1154,1174,1137
890,1260,1187,1267,1171,1103,956
1050,1170,1206,1145,795,1174,1260
1038,957,999,958,1260,1260,1108
767,1161,1065,1031,1264,1262,1038


I've found a way to compute the T value in octave, but I'm not yet sure how to interpret it. The help says:

-- Function File: [PVAL, T, DF] = t_test_2 (X, Y, ALT)
     For two samples x and y from normal distributions with unknown
     means and unknown equal variances, perform a two-sample t-test of
     the null hypothesis of equal means.

     Under the null, the test statistic T follows a Student distribution
     with DF degrees of freedom.

     With the optional argument string ALT, the alternative of interest
     can be selected.  If ALT is "!=" or "<>", the null is tested
     against the two-sided alternative 'mean (X) != mean (Y)'.  If ALT
     is ">", the one-sided alternative 'mean (X) > mean (Y)' is used.
     Similarly for "<", the one-sided alternative 'mean (X) < mean (Y)'
     is used.  The default is the two-sided case.

     The p-value of the test is returned in PVAL.


So I tried:

>> a = [ <paste first dataset> ](:);
>> b = [ <paste second dataset> ](:);
>> c = [ <paste third dataset> ](:);

>> [median(a), median(b), median(c)]
ans =
   1258.5   1161.0   1166.0

>> [p,t] = t_test_2(a, b, ">") % is a > b ?
p =  0.19725
t =  0.85642

>> [p,t] = t_test_2(b, c, ">") % is b > c ?
p =  0.13103
t =  1.1282

>> [p,t] = t_test_2(a, c, ">") % is a > c ?
p =  0.022017
t =  2.0388

>> [p,t] = t_test_2(a, b, "<") % is a < b ?
p =  0.80275
t =  0.85642

>> [p,t] = t_test_2(b, c, "<") % is b < c ?
p =  0.86897
t =  1.1282

[p,t] = t_test_2(a, c, "<") % is a < c ?
p =  0.97798
t =  2.0388


So, to me it looks like a (36B111D) is a little faster than the other two (E203CF9 and 14E2991). I actually expected it, as it fixed a bug introduced with the SRM memory backend, where we've got memory buffers larger than 32MB, which are slower to write (limit is 32MB-512B). The latter is the maximum size that can be written with one DMA transfer, so larger buffers would require two transfers; the second transfer is usually small (slower to write).

Between the other two, I'm not sure what conclusion to draw. Median is a tad lower on b, but the difference is tiny. The T number is small, so probably the difference is not significant, but the P values (0.13 vs 0.87) seems to indicate that b might be a little bit faster. Should I run the test between those two for a longer time, or should I consider them equal?

(P.S. To see what those codes mean, look them up on the repository (https://bitbucket.org/hudson/magic-lantern/commits/all).
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Frank7D on April 16, 2016, 05:32:12 PM
A: 36B111D.MO
Count = 42
Mean frames = 1178

B: E203CF9.MO
Count = 35
Mean frames = 1150

C: 14E2991.MO
Count = 63
Mean frames = 1115

A vs. B
T -Stat: 0.856418986
Critical T: 1.665996224

A vs. C
T -Stat: 2.03877008
Critical T: 1.66319668

B vs. C
T -Stat: 1.128192618
Critical T: 1.667572281

So A beat C (36B111D.MO beat 14E2991.MO)

I notice that your "t" value equals my "T -Stat" value.

"Should I run the test between those two for a longer time, or should I consider them equal?"

Maybe. My spreadsheet couldn't find a winner between A and B or B and C, so maybe B needs more points.
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: DeafEyeJedi on April 16, 2016, 06:47:59 PM
Holy Moly! It's been awhile since I last jumped into this puddle of mud. Excellent progress you all have made so far to those that contributed and to me this is just a wonderful piece of Art.

Love the collaborations you all have created into this thread and now that I'm finally back at home from being on road away from my lab. It seems 5D3, 7D have been throughly tested.

Does this mean it would be best for me to do this speed test with a different camera model other than those two I've spotted so far?

I also noticed there are new Nightlies that just came out last night (54e90de) ... Should we be testing this on the latest, correct?

I'd prefer to use @a1ex's build to squeeze the last bit of performance out of this. Is it wrong for me to feel this way?  :P
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: a1ex on April 16, 2016, 07:58:45 PM
You can try these two pull requests, and let us know if you see any difference in practice:

- (RAW) https://bitbucket.org/hudson/magic-lantern/pull-requests/710/raw_fixes-part-2/diff
- (MLV Lite) https://bitbucket.org/hudson/magic-lantern/pull-requests/685/proposal-completely-replace-the-old-raw

Builds (these two should be identical speed-wise):
- raw: raw_rec.mo (http://a1ex.magiclantern.fm/bleeding-edge/mlv_lite/32m-raw/raw_rec.mo) (e78c7b8)
- mlv: raw_rec.mo (http://a1ex.magiclantern.fm/bleeding-edge/mlv_lite/32m-mlv/raw_rec.mo) (30bb06a, not tested)

The change that gave the highest score in the above benchmark is included in both.

For core, any recent version that loads the module is fine. Latest nightly won't make any difference, compared to the previous one (see change log). The nightly with the black level fix should make a difference if you actually had problems with the black level. Speed-wise, I don't expect any differences coming from the core.

So, the questions for these two builds are:
- are the speeds identical in both? (as you could see, this question is not very easy to answer)
- how do they compare (speed-wise) with the raw_rec from the nightly?

If the answers are favorable, that means we have achieved dmilligan's goal with MLV Lite: save valid MLV instead of RAW, with performance identical to the old raw_rec. Plus a tiny speed boost.

There's still room for increasing the speed, with MLV only (can't be done with raw). We can exploit a MLV feature that allows the frames to be in any order in the file. No idea how much the speed gain could be, though.

I'm tempted to merge the current version of mlv_lite to nightly if it actually proves to be identical to the original raw, as it seems pretty solid and the code changes are minimally invasive. After that, we are free to explore further optimizations.
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Frank7D on April 16, 2016, 08:32:28 PM
By the way, just for clarity, when the versions "tie" that doesn't prove they're definitely the same, just that we haven't been able to show they're different. It could be that a larger data set would reveal a difference.
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Ottoga on April 17, 2016, 03:30:50 PM
@A1ex
Do you still want more test data from a 7d?  If so give me a link to your test raw_rec modules and I'll run your test script against them.
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: a1ex on April 17, 2016, 05:46:34 PM
Yes, with builds from reply #42.

I'm waiting for test results for those builds before continuing, because I want to merge things into nightly.
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Ottoga on April 18, 2016, 03:23:46 AM
@a1lex
Is there a compiled version of the rawbench.lua script available. I don't have the facilities to compile.
Also, in your post you say that this script is not compatible with the latest nightly, is this still the case? if so is there a test build of a compatible nightly available?
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: a1ex on April 18, 2016, 07:28:40 AM
Lua scripts are interpreted directly on the camera (no need to compile, just place it in the ML/SCRIPTS directory).

I'll test lua_fix again today, and if it passes on 5D3, it will be merged in the next nightly.

edit: sorry, I've already found two regressions in core code, and a major issue on the I/O side. I've fixed them, but the changes were not trivial, and I don't want to do another rushed merge.
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Ottoga on April 19, 2016, 05:27:09 AM
@A1ex

I'm making some progress, the script is trying to run. however, I see the following error message on the screen when I turn the camera on and enter the ML menu system.



Stack Traceback:
[c]: in function "require"
ML/SRIPTS/RAWBENCH.LUA:21: in main chunk



Looking at the script, I assume that I'm missing addition scripts called "logger" and "stats"


Camera conditions are:
April 19, 2106 nightly
Camera AV mode
rawbench.lua installed within the ML\scripts directory
Modules loaded:  raw-rec.mo, lua.mo
ML - Global draw off, defaults for all other settings
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: a1ex on April 19, 2016, 02:25:41 PM
I didn't merge lua_fix yet. Will test it again tonight.

edit: sorry, I'm very tired. I managed to update the script to analyze MLV files as well (code committed), fixed a few more minor things in Lua, but I don't have enough energy to run the tests for merging into nightly. I could really use some help here (http://www.magiclantern.fm/forum/index.php?topic=14828.msg164972#msg164972)...
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: a1ex on April 20, 2016, 07:41:43 AM
Result from last night (1920x1080, 30 fps, forgot global draw on at default settings):


Raw_rec version : ML/MODULES/RAW_REC/32M-RAW.MO
728,1163,1073,1172,1059,983,682,639
458,1091,1132,1145,1135,1074,845,740
328,1044,1116,1045,1136,1137,944,1017
1011,1134,1048,1153,819,1162,1139,747
899,1187,1162,1122,1172,1179,931,652
827,1136,1127,1000,1094,1093,1124,972
660,1064,1050,1062,1054,1044,931,891

Raw_rec version : ML/MODULES/RAW_REC/UNIFIED.MO
853,1067,1056,1045,1073,1063,1090,778
445,1060,1085,1066,1072,874,907,976
1043,1069,1037,1066,1047,1092,926,825
876,1087,852,963,1050,960,1083,1073
1036,1076,1037,1004,878,1084,874,986
1044,1088,1072,972,1052,939,842,1049
954,978,1047,967,1020,1019,1019,945
527,987,909,991,997,832
699,915,1007,982,909,883,910,949

Raw_rec version : ML/MODULES/RAW_REC/32M-MLV.MO
689,872,864,774,776,794,646,793
786,810,873,730,771,797,781,883


Three problems:
- the MLV version appears slower (or maybe my script is not handling the MLVs properly; investigating)
- the random number generator doesn't seem very good (it picked MLV only two times)
- the script stopped recording for some reason after about 3 hours (I suspect overheating)
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Frank7D on April 20, 2016, 08:32:24 AM
A: 32M-RAW.MO
Count = 56
Mean frames = 992

B: UNIFIED.MO
Count = 70
Mean frames = 972

C: 32M-MLV.MO
Count = 16
Mean frames = 790

A vs. B
T -Stat: 0.70
Critical T: 1.66

A vs. C
T -Stat: 4.11
Critical T: 1.70

B vs. C
T -Stat: 5.80
Critical T: 1.71

So A and B both beat C (32M-RAW.MO and UNIFIED.MO beat 32M-MLV.MO)
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: a1ex on April 20, 2016, 08:47:37 AM
Partial results after some small tweaks to the script:

rawbench.lua (https://bitbucket.org/hudson/magic-lantern/src/raw_benchmark/scripts/extra/rawbench.lua) (history) (https://bitbucket.org/hudson/magic-lantern/history-node/raw_benchmark/scripts/extra/rawbench.lua?at=raw_benchmark)

5D3, 1920x1080, 30 fps, global draw off, builds from reply #42 vs current unified:


For next experiment
===================

Raw_rec version : ML/MODULES/RAW_REC/32M-RAW.MO

===============================================================================
ML/SCRIPTS/RAWBENCH.LUA - 2016-4-20 08:26:24
===============================================================================

[08:26:30] Recording test clip 1 of 8...
[08:27:10] Recording stopped.
[08:27:30] Recording test clip 2 of 8...
[08:28:21] Recording stopped.
[08:28:41] Recording test clip 3 of 8...
[08:29:32] Recording stopped.
[08:29:52] Recording test clip 4 of 8...
[08:30:43] Recording stopped.
[08:31:03] Recording test clip 5 of 8...
[08:31:53] Recording stopped.
[08:32:13] Recording test clip 6 of 8...
[08:32:59] Recording stopped.
[08:33:19] Recording test clip 7 of 8...
[08:34:00] Recording stopped.
[08:34:20] Recording test clip 8 of 8...
[08:34:36] Recording stopped.

===============================================================================
ML/SCRIPTS/RAWBENCH.LUA - 2016-4-20 08:36:28
===============================================================================

A:/DCIM/100EOS5D/M20-0826.RAW: 1065
A:/DCIM/100EOS5D/M20-0827.R00: 1360
A:/DCIM/100EOS5D/M20-0828.R00: 1273
A:/DCIM/100EOS5D/M20-0829.R00: 1260
A:/DCIM/100EOS5D/M20-0831.R00: 1352
A:/DCIM/100EOS5D/M20-0832.R00: 1216
A:/DCIM/100EOS5D/M20-0833.RAW: 1091
A:/DCIM/100EOS5D/M20-0834.RAW: 310
Recorded frames : 1065,1360,1273,1260,1352,1216,1091,310
Quartile summary: 1238 frames (1078...1312.5)

For next experiment
===================

Raw_rec version : ML/MODULES/RAW_REC/UNIFIED.MO

===============================================================================
ML/SCRIPTS/RAWBENCH.LUA - 2016-4-20 08:37:57
===============================================================================

[08:38:02] Recording test clip 1 of 8...
[08:38:32] Recording stopped.
[08:38:52] Recording test clip 2 of 8...
[08:39:43] Recording stopped.
[08:40:03] Recording test clip 3 of 8...
[08:40:54] Recording stopped.
[08:41:14] Recording test clip 4 of 8...
[08:41:50] Recording stopped.
[08:42:10] Recording test clip 5 of 8...
[08:42:56] Recording stopped.
[08:43:16] Recording test clip 6 of 8...
[08:44:01] Recording stopped.
[08:44:21] Recording test clip 7 of 8...
[08:45:02] Recording stopped.
[08:45:22] Recording test clip 8 of 8...
[08:45:53] Recording stopped.

===============================================================================
ML/SCRIPTS/RAWBENCH.LUA - 2016-4-20 08:47:45
===============================================================================

A:/DCIM/100EOS5D/M20-0838.RAW: 657
A:/DCIM/100EOS5D/M20-0839.R00: 1268
A:/DCIM/100EOS5D/M20-0840.R00: 1261
A:/DCIM/100EOS5D/M20-0841.RAW: 817
A:/DCIM/100EOS5D/M20-0842.R00: 1186
A:/DCIM/100EOS5D/M20-0843.RAW: 1160
A:/DCIM/100EOS5D/M20-0844.RAW: 989
A:/DCIM/100EOS5D/M20-0845.RAW: 745
Recorded frames : 657,1268,1261,817,1186,1160,989,745
Quartile summary: 1074.5 frames (781...1223.5)

For next experiment
===================

Raw_rec version : ML/MODULES/RAW_REC/32M-MLV.MO

===============================================================================
ML/SCRIPTS/RAWBENCH.LUA - 2016-4-20 08:49:14
===============================================================================

[08:49:19] Recording test clip 1 of 8...
[08:49:55] Recording stopped.
[08:50:15] Recording test clip 2 of 8...
[08:50:56] Recording stopped.
[08:51:16] Recording test clip 3 of 8...
[08:51:52] Recording stopped.
[08:52:12] Recording test clip 4 of 8...
[08:52:47] Recording stopped.
[08:53:07] Recording test clip 5 of 8...
[08:53:43] Recording stopped.
[08:54:03] Recording test clip 6 of 8...
[08:54:44] Recording stopped.
[08:55:04] Recording test clip 7 of 8...
[08:55:45] Recording stopped.
[08:56:05] Recording test clip 8 of 8...
[08:56:35] Recording stopped.

===============================================================================
ML/SCRIPTS/RAWBENCH.LUA - 2016-4-20 08:58:27
===============================================================================

A:/DCIM/100EOS5D/M20-0849.MLV: 916
A:/DCIM/100EOS5D/M20-0850.MLV: 1046
A:/DCIM/100EOS5D/M20-0851.MLV: 812
A:/DCIM/100EOS5D/M20-0852.MLV: 802
A:/DCIM/100EOS5D/M20-0853.MLV: 878
A:/DCIM/100EOS5D/M20-0854.MLV: 1028
A:/DCIM/100EOS5D/M20-0855.MLV: 1055
A:/DCIM/100EOS5D/M20-0856.MLV: 784
Recorded frames : 916,1046,812,802,878,1028,1055,784
Quartile summary: 897 frames (807...1037)

For next experiment
===================

Raw_rec version : ML/MODULES/RAW_REC/UNIFIED.MO

===============================================================================
ML/SCRIPTS/RAWBENCH.LUA - 2016-4-20 09:00:11
===============================================================================

[09:00:16] Recording test clip 1 of 8...
[09:00:57] Recording stopped.
[09:01:17] Recording test clip 2 of 8...
[09:01:57] Recording stopped.
[09:02:17] Recording test clip 3 of 8...
[09:02:58] Recording stopped.
[09:03:18] Recording test clip 4 of 8...
[09:04:09] Recording stopped.
[09:04:29] Recording test clip 5 of 8...
[09:05:20] Recording stopped.
[09:05:40] Recording test clip 6 of 8...
[09:06:30] Recording stopped.
[09:06:50] Recording test clip 7 of 8...
[09:07:26] Recording stopped.
[09:07:46] Recording test clip 8 of 8...
[09:08:17] Recording stopped.

===============================================================================
ML/SCRIPTS/RAWBENCH.LUA - 2016-4-20 09:10:09
===============================================================================

A:/DCIM/100EOS5D/M20-0900.RAW: 1087
A:/DCIM/100EOS5D/M20-0901.RAW: 1030
A:/DCIM/100EOS5D/M20-0902.RAW: 1069
A:/DCIM/100EOS5D/M20-0903.R00: 1317
A:/DCIM/100EOS5D/M20-0904.R00: 1260
A:/DCIM/100EOS5D/M20-0905.R00: 1313
A:/DCIM/100EOS5D/M20-0906.RAW: 905
A:/DCIM/100EOS5D/M20-0907.RAW: 682
Recorded frames : 1087,1030,1069,1317,1260,1313,905,682
Quartile summary: 1078 frames (967.5...1286.5)

For next experiment
===================

Raw_rec version : ML/MODULES/RAW_REC/32M-RAW.MO

===============================================================================
ML/SCRIPTS/RAWBENCH.LUA - 2016-4-20 09:11:38
===============================================================================

[09:11:43] Recording test clip 1 of 8...
[09:12:04] Recording stopped.
[09:12:24] Recording test clip 2 of 8...
[09:13:09] Recording stopped.
[09:13:29] Recording test clip 3 of 8...
[09:14:20] Recording stopped.
[09:14:40] Recording test clip 4 of 8...
[09:15:31] Recording stopped.
[09:15:51] Recording test clip 5 of 8...
[09:16:27] Recording stopped.
[09:16:47] Recording test clip 6 of 8...
[09:17:33] Recording stopped.
[09:17:53] Recording test clip 7 of 8...
[09:18:38] Recording stopped.
[09:18:58] Recording test clip 8 of 8...
[09:19:39] Recording stopped.

===============================================================================
ML/SCRIPTS/RAWBENCH.LUA - 2016-4-20 09:21:31
===============================================================================

A:/DCIM/100EOS5D/M20-0911.RAW: 476
A:/DCIM/100EOS5D/M20-0912.R00: 1188
A:/DCIM/100EOS5D/M20-0913.R00: 1340
A:/DCIM/100EOS5D/M20-0914.R00: 1355
A:/DCIM/100EOS5D/M20-0915.RAW: 805
A:/DCIM/100EOS5D/M20-0916.R00: 1205
A:/DCIM/100EOS5D/M20-0917.RAW: 1095
A:/DCIM/100EOS5D/M20-0919.RAW: 1071
Recorded frames : 476,1188,1340,1355,805,1205,1095,1071
Quartile summary: 1141.5 frames (938...1272.5)

For next experiment
===================

Raw_rec version : ML/MODULES/RAW_REC/32M-MLV.MO

===============================================================================
ML/SCRIPTS/RAWBENCH.LUA - 2016-4-20 09:23:00
===============================================================================

[09:23:05] Recording test clip 1 of 8...
[09:23:26] Recording stopped.
[09:23:46] Recording test clip 2 of 8...
[09:24:27] Recording stopped.
[09:24:47] Recording test clip 3 of 8...
[09:25:23] Recording stopped.
[09:25:43] Recording test clip 4 of 8...
[09:26:09] Recording stopped.
[09:26:29] Recording test clip 5 of 8...
[09:27:09] Recording stopped.
[09:27:29] Recording test clip 6 of 8...
[09:28:00] Recording stopped.
[09:28:20] Recording test clip 7 of 8...
[09:29:01] Recording stopped.
[09:29:21] Recording test clip 8 of 8...
[09:30:02] Recording stopped.

===============================================================================
ML/SCRIPTS/RAWBENCH.LUA - 2016-4-20 09:31:54
===============================================================================

A:/DCIM/100EOS5D/M20-0923.MLV: 481
A:/DCIM/100EOS5D/M20-0924.MLV: 1036
A:/DCIM/100EOS5D/M20-0925.MLV: 936
A:/DCIM/100EOS5D/M20-0926.MLV: 541
A:/DCIM/100EOS5D/M20-0927.MLV: 979
A:/DCIM/100EOS5D/M20-0928.MLV: 685
A:/DCIM/100EOS5D/M20-0929.MLV: 954
A:/DCIM/100EOS5D/M20-0930.MLV: 952
Recorded frames : 481,1036,936,541,979,685,954,952
Quartile summary: 944 frames (613...966.5)

For next experiment
===================

Raw_rec version : ML/MODULES/RAW_REC/32M-MLV.MO

===============================================================================
ML/SCRIPTS/RAWBENCH.LUA - 2016-4-20 09:33:36
===============================================================================

[09:33:41] Recording test clip 1 of 8...
[09:34:17] Recording stopped.
[09:34:37] Recording test clip 2 of 8...
[09:35:18] Recording stopped.
[09:35:38] Recording test clip 3 of 8...
[09:36:13] Recording stopped.
[09:36:34] Recording test clip 4 of 8...
[09:37:09] Recording stopped.
[09:37:29] Recording test clip 5 of 8...
[09:38:10] Recording stopped.
[09:38:30] Recording test clip 6 of 8...
[09:39:11] Recording stopped.
[09:39:31] Recording test clip 7 of 8...
[09:40:12] Recording stopped.
[09:40:32] Recording test clip 8 of 8...
[09:41:07] Recording stopped.

===============================================================================
ML/SCRIPTS/RAWBENCH.LUA - 2016-4-20 09:43:00
===============================================================================

A:/DCIM/100EOS5D/M20-0933.MLV: 908
A:/DCIM/100EOS5D/M20-0934.MLV: 1034
A:/DCIM/100EOS5D/M20-0935.MLV: 798
A:/DCIM/100EOS5D/M20-0936.MLV: 938
A:/DCIM/100EOS5D/M20-0937.MLV: 954
A:/DCIM/100EOS5D/M20-0938.MLV: 1034
A:/DCIM/100EOS5D/M20-0939.MLV: 952
A:/DCIM/100EOS5D/M20-0940.MLV: 898
Recorded frames : 908,1034,798,938,954,1034,952,898
Quartile summary: 945 frames (903...994)

For next experiment
===================

Raw_rec version : ML/MODULES/RAW_REC/32M-MLV.MO


So, unless there's a problem with the script analyzing the MLVs (which I'll double-check shortly), it looks like something is really slowing it down. Figure out what...
Title: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: DeafEyeJedi on April 20, 2016, 09:20:58 AM
Still trying to get the script to run on 5D3 but here's what I was able to accomplish from the past few nights of practice runs.

5D3, 1920x1080p, 30p global draw ON with builds from reply #42 vs the current nightly (also did 60p @ 1728x584p just for kicks).

Unified (nightly):
30p = 31s
60p = 10s

e78c7b8:
30p = 35s
60p = 11s

30bb06a:
30p = 27-28s (probably due to overheat? will test some more)
60p = 8-10s

I also noticed you added the rawbench.lua file on your recent post ... downloaded and will place it into script folder within ML and hopefully it'll run it overnight without overheating it? Ha.

*edit*

I just started to run the script on 5D3 or at least thought so but I keep getting this error saying "unexpected symbol near '<\5>' ... any idea what I could be doing wrong?

Thanks, @a1ex!
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: a1ex on April 20, 2016, 09:31:17 AM
To run the script, compile lua_fix and run the tests on top of that branch.

Under ML/MODULES/RAW_REC you can place binaries from other branches, so the script will pick them for testing.
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: a1ex on April 20, 2016, 01:02:11 PM
Another test: I've aligned the MLV header at 512 bytes.

5D3, 1920x1080, 30 fps, global draw on. 32M-MLV.MO is the one from reply #42 (30bb06a), and HDR512.MO is the one with header aligned at 512 bytes (b658401).

Filtered log, with grep -E "(Recorded|version)":

Raw_rec version : ML/MODULES/RAW_REC/32M-MLV.MO
Recorded frames : 632,475,772,726,659,702,642,754
Raw_rec version : ML/MODULES/RAW_REC/HDR512.MO
Recorded frames : 793,1019,976,944,1114,1017,714,577
Raw_rec version : ML/MODULES/RAW_REC/32M-MLV.MO
Recorded frames : 688,724,559,786,863,786,785,867
Raw_rec version : ML/MODULES/RAW_REC/HDR512.MO
Recorded frames : 1032,1053,927,1139,939,1055,754,659
Raw_rec version : ML/MODULES/RAW_REC/HDR512.MO
Recorded frames : 689,890,921,802,1101,1049,918,1133
Raw_rec version : ML/MODULES/RAW_REC/32M-MLV.MO
Recorded frames : 422,721,762,778,879,869,706,868
Raw_rec version : ML/MODULES/RAW_REC/32M-MLV.MO
Recorded frames : 797,780,847,679,577,847,768,799
Raw_rec version : ML/MODULES/RAW_REC/32M-MLV.MO
Recorded frames : 629,793,770,754,751,845,783,835


Full log:

For next experiment
===================

Raw_rec version : ML/MODULES/RAW_REC/32M-MLV.MO

===============================================================================
ML/SCRIPTS/RAWBENCH.LUA - 2016-4-20 12:15:20
===============================================================================

[12:15:25] Recording test clip 1 of 8...
[12:15:51] Recording stopped.
[12:16:11] Recording test clip 2 of 8...
[12:16:32] Recording stopped.
[12:16:52] Recording test clip 3 of 8...
[12:17:23] Recording stopped.
[12:17:43] Recording test clip 4 of 8...
[12:18:14] Recording stopped.
[12:18:34] Recording test clip 5 of 8...
[12:19:05] Recording stopped.
[12:19:25] Recording test clip 6 of 8...
[12:19:56] Recording stopped.
[12:20:16] Recording test clip 7 of 8...
[12:20:42] Recording stopped.
[12:21:02] Recording test clip 8 of 8...
[12:21:32] Recording stopped.

===============================================================================
ML/SCRIPTS/RAWBENCH.LUA - 2016-4-20 12:23:29
===============================================================================

A:/DCIM/100EOS5D/M20-1215.MLV:  632 frames, 2.136047 GB
A:/DCIM/100EOS5D/M20-1216.MLV:  475 frames, 1.605415 GB
A:/DCIM/100EOS5D/M20-1217.MLV:  772 frames, 2.609222 GB
A:/DCIM/100EOS5D/M20-1218.MLV:  726 frames, 2.453751 GB
A:/DCIM/100EOS5D/M20-1219.MLV:  659 frames, 2.227303 GB
A:/DCIM/100EOS5D/M20-1220.MLV:  702 frames, 2.372635 GB
A:/DCIM/100EOS5D/M20-1221.MLV:  642 frames, 2.169846 GB
A:/DCIM/100EOS5D/M20-1222.MLV:  754 frames, 2.548386 GB
Recorded frames : 632,475,772,726,659,702,642,754
Quartile summary: 680.5 frames (637...740)

For next experiment
===================

Raw_rec version : ML/MODULES/RAW_REC/HDR512.MO

===============================================================================
ML/SCRIPTS/RAWBENCH.LUA - 2016-4-20 12:25:14
===============================================================================

[12:25:19] Recording test clip 1 of 8...
[12:25:50] Recording stopped.
[12:26:10] Recording test clip 2 of 8...
[12:26:51] Recording stopped.
[12:27:11] Recording test clip 3 of 8...
[12:27:52] Recording stopped.
[12:28:12] Recording test clip 4 of 8...
[12:28:48] Recording stopped.
[12:29:08] Recording test clip 5 of 8...
[12:29:53] Recording stopped.
[12:30:14] Recording test clip 6 of 8...
[12:30:54] Recording stopped.
[12:31:14] Recording test clip 7 of 8...
[12:31:45] Recording stopped.
[12:32:05] Recording test clip 8 of 8...
[12:32:31] Recording stopped.

===============================================================================
ML/SCRIPTS/RAWBENCH.LUA - 2016-4-20 12:34:28
===============================================================================

A:/DCIM/100EOS5D/M20-1225.MLV:  793 frames, 2.680199 GB
A:/DCIM/100EOS5D/M20-1226.MLV: 1019 frames, 3.444039 GB
A:/DCIM/100EOS5D/M20-1227.MLV:  976 frames, 3.298707 GB
A:/DCIM/100EOS5D/M20-1228.MLV:  944 frames, 3.190552 GB
A:/DCIM/100EOS5D/M20-1229.MLV: 1114 frames, 3.765122 GB
A:/DCIM/100EOS5D/M20-1230.MLV: 1017 frames, 3.437279 GB
A:/DCIM/100EOS5D/M20-1231.MLV:  714 frames, 2.413193 GB
A:/DCIM/100EOS5D/M20-1232.MLV:  577 frames, 1.950158 GB
Recorded frames : 793,1019,976,944,1114,1017,714,577
Quartile summary: 960 frames (753.5...1018)

For next experiment
===================

Raw_rec version : ML/MODULES/RAW_REC/32M-MLV.MO

===============================================================================
ML/SCRIPTS/RAWBENCH.LUA - 2016-4-20 12:36:12
===============================================================================

[12:36:17] Recording test clip 1 of 8...
[12:36:48] Recording stopped.
[12:37:08] Recording test clip 2 of 8...
[12:37:39] Recording stopped.
[12:37:59] Recording test clip 3 of 8...
[12:38:25] Recording stopped.
[12:38:45] Recording test clip 4 of 8...
[12:39:16] Recording stopped.
[12:39:36] Recording test clip 5 of 8...
[12:40:11] Recording stopped.
[12:40:32] Recording test clip 6 of 8...
[12:41:02] Recording stopped.
[12:41:23] Recording test clip 7 of 8...
[12:41:54] Recording stopped.
[12:42:14] Recording test clip 8 of 8...
[12:42:49] Recording stopped.

===============================================================================
ML/SCRIPTS/RAWBENCH.LUA - 2016-4-20 12:44:46
===============================================================================

A:/DCIM/100EOS5D/M20-1236.MLV:  688 frames, 2.325317 GB
A:/DCIM/100EOS5D/M20-1237.MLV:  724 frames, 2.446991 GB
A:/DCIM/100EOS5D/M20-1238.MLV:  559 frames, 1.88932 GB
A:/DCIM/100EOS5D/M20-1239.MLV:  786 frames, 2.65654 GB
A:/DCIM/100EOS5D/M20-1240.MLV:  863 frames, 2.916786 GB
A:/DCIM/100EOS5D/M20-1241.MLV:  786 frames, 2.65654 GB
A:/DCIM/100EOS5D/M20-1242.MLV:  785 frames, 2.65316 GB
A:/DCIM/100EOS5D/M20-1243.MLV:  867 frames, 2.930305 GB
Recorded frames : 688,724,559,786,863,786,785,867
Quartile summary: 785.5 frames (706...824.5)

For next experiment
===================

Raw_rec version : ML/MODULES/RAW_REC/HDR512.MO

===============================================================================
ML/SCRIPTS/RAWBENCH.LUA - 2016-4-20 12:46:32
===============================================================================

[12:46:38] Recording test clip 1 of 8...
[12:47:19] Recording stopped.
[12:47:39] Recording test clip 2 of 8...
[12:48:19] Recording stopped.
[12:48:40] Recording test clip 3 of 8...
[12:49:15] Recording stopped.
[12:49:35] Recording test clip 4 of 8...
[12:50:21] Recording stopped.
[12:50:41] Recording test clip 5 of 8...
[12:51:17] Recording stopped.
[12:51:37] Recording test clip 6 of 8...
[12:52:18] Recording stopped.
[12:52:38] Recording test clip 7 of 8...
[12:53:09] Recording stopped.
[12:53:29] Recording test clip 8 of 8...
[12:54:00] Recording stopped.

===============================================================================
ML/SCRIPTS/RAWBENCH.LUA - 2016-4-20 12:55:56
===============================================================================

A:/DCIM/100EOS5D/M20-1246.MLV: 1032 frames, 3.487977 GB
A:/DCIM/100EOS5D/M20-1247.MLV: 1053 frames, 3.558953 GB
A:/DCIM/100EOS5D/M20-1248.MLV:  927 frames, 3.133095 GB
A:/DCIM/100EOS5D/M20-1249.MLV: 1139 frames, 3.849617 GB
A:/DCIM/100EOS5D/M20-1250.MLV:  939 frames, 3.173653 GB
A:/DCIM/100EOS5D/M20-1251.MLV: 1055 frames, 3.565712 GB
A:/DCIM/100EOS5D/M20-1252.MLV:  754 frames, 2.548386 GB
A:/DCIM/100EOS5D/M20-1253.MLV:  659 frames, 2.227303 GB
Recorded frames : 1032,1053,927,1139,939,1055,754,659
Quartile summary: 985.5 frames (840.5...1054)

For next experiment
===================

Raw_rec version : ML/MODULES/RAW_REC/HDR512.MO

===============================================================================
ML/SCRIPTS/RAWBENCH.LUA - 2016-4-20 12:57:41
===============================================================================

[12:57:46] Recording test clip 1 of 8...
[12:58:17] Recording stopped.
[12:58:37] Recording test clip 2 of 8...
[12:59:13] Recording stopped.
[12:59:33] Recording test clip 3 of 8...
[13:00:09] Recording stopped.
[13:00:29] Recording test clip 4 of 8...
[13:01:05] Recording stopped.
[13:01:25] Recording test clip 5 of 8...
[13:02:10] Recording stopped.
[13:02:30] Recording test clip 6 of 8...
[13:03:11] Recording stopped.
[13:03:31] Recording test clip 7 of 8...
[13:04:07] Recording stopped.
[13:04:27] Recording test clip 8 of 8...
[13:05:13] Recording stopped.

===============================================================================
ML/SCRIPTS/RAWBENCH.LUA - 2016-4-20 13:07:10
===============================================================================

A:/DCIM/100EOS5D/M20-1257.MLV:  689 frames, 2.328698 GB
A:/DCIM/100EOS5D/M20-1258.MLV:  890 frames, 3.008042 GB
A:/DCIM/100EOS5D/M20-1259.MLV:  921 frames, 3.112816 GB
A:/DCIM/100EOS5D/M20-1300.MLV:  802 frames, 2.710618 GB
A:/DCIM/100EOS5D/M20-1301.MLV: 1101 frames, 3.721184 GB
A:/DCIM/100EOS5D/M20-1302.MLV: 1049 frames, 3.545433 GB
A:/DCIM/100EOS5D/M20-1303.MLV:  918 frames, 3.102677 GB
A:/DCIM/100EOS5D/M20-1304.MLV: 1133 frames, 3.829339 GB
Recorded frames : 689,890,921,802,1101,1049,918,1133
Quartile summary: 919.5 frames (846...1075)

For next experiment
===================

Raw_rec version : ML/MODULES/RAW_REC/32M-MLV.MO

===============================================================================
ML/SCRIPTS/RAWBENCH.LUA - 2016-4-20 13:08:54
===============================================================================

[13:08:59] Recording test clip 1 of 8...
[13:09:20] Recording stopped.
[13:09:40] Recording test clip 2 of 8...
[13:10:11] Recording stopped.
[13:10:31] Recording test clip 3 of 8...
[13:11:02] Recording stopped.
[13:11:22] Recording test clip 4 of 8...
[13:11:53] Recording stopped.
[13:12:13] Recording test clip 5 of 8...
[13:12:49] Recording stopped.
[13:13:09] Recording test clip 6 of 8...
[13:13:45] Recording stopped.
[13:14:05] Recording test clip 7 of 8...
[13:14:36] Recording stopped.
[13:14:56] Recording test clip 8 of 8...
[13:15:32] Recording stopped.

===============================================================================
ML/SCRIPTS/RAWBENCH.LUA - 2016-4-20 13:17:28
===============================================================================

A:/DCIM/100EOS5D/M20-1309.MLV:  422 frames, 1.426285 GB
A:/DCIM/100EOS5D/M20-1310.MLV:  721 frames, 2.436851 GB
A:/DCIM/100EOS5D/M20-1311.MLV:  762 frames, 2.575424 GB
A:/DCIM/100EOS5D/M20-1312.MLV:  778 frames, 2.629501 GB
A:/DCIM/100EOS5D/M20-1313.MLV:  879 frames, 2.970863 GB
A:/DCIM/100EOS5D/M20-1314.MLV:  869 frames, 2.937065 GB
A:/DCIM/100EOS5D/M20-1315.MLV:  706 frames, 2.386154 GB
A:/DCIM/100EOS5D/M20-1316.MLV:  868 frames, 2.933685 GB
Recorded frames : 422,721,762,778,879,869,706,868
Quartile summary: 770 frames (713.5...868.5)

For next experiment
===================

Raw_rec version : ML/MODULES/RAW_REC/32M-MLV.MO

===============================================================================
ML/SCRIPTS/RAWBENCH.LUA - 2016-4-20 13:19:15
===============================================================================

[13:19:20] Recording test clip 1 of 8...
[13:19:51] Recording stopped.
[13:20:11] Recording test clip 2 of 8...
[13:20:42] Recording stopped.
[13:21:02] Recording test clip 3 of 8...
[13:21:38] Recording stopped.
[13:21:58] Recording test clip 4 of 8...
[13:22:29] Recording stopped.
[13:22:49] Recording test clip 5 of 8...
[13:23:15] Recording stopped.
[13:23:35] Recording test clip 6 of 8...
[13:24:11] Recording stopped.
[13:24:31] Recording test clip 7 of 8...
[13:25:02] Recording stopped.
[13:25:22] Recording test clip 8 of 8...
[13:25:53] Recording stopped.

===============================================================================
ML/SCRIPTS/RAWBENCH.LUA - 2016-4-20 13:27:49
===============================================================================

A:/DCIM/100EOS5D/M20-1319.MLV:  797 frames, 2.693718 GB
A:/DCIM/100EOS5D/M20-1320.MLV:  780 frames, 2.636261 GB
A:/DCIM/100EOS5D/M20-1321.MLV:  847 frames, 2.862709 GB
A:/DCIM/100EOS5D/M20-1322.MLV:  679 frames, 2.294899 GB
A:/DCIM/100EOS5D/M20-1323.MLV:  577 frames, 1.950157 GB
A:/DCIM/100EOS5D/M20-1324.MLV:  847 frames, 2.862709 GB
A:/DCIM/100EOS5D/M20-1325.MLV:  768 frames, 2.595703 GB
A:/DCIM/100EOS5D/M20-1326.MLV:  799 frames, 2.700478 GB
Recorded frames : 797,780,847,679,577,847,768,799
Quartile summary: 788.5 frames (723.5...823)

For next experiment
===================

Raw_rec version : ML/MODULES/RAW_REC/32M-MLV.MO

===============================================================================
ML/SCRIPTS/RAWBENCH.LUA - 2016-4-20 13:29:36
===============================================================================

[13:29:41] Recording test clip 1 of 8...
[13:30:07] Recording stopped.
[13:30:27] Recording test clip 2 of 8...
[13:30:58] Recording stopped.
[13:31:18] Recording test clip 3 of 8...
[13:31:49] Recording stopped.
[13:32:09] Recording test clip 4 of 8...
[13:32:40] Recording stopped.
[13:33:00] Recording test clip 5 of 8...
[13:33:31] Recording stopped.
[13:33:51] Recording test clip 6 of 8...
[13:34:27] Recording stopped.
[13:34:47] Recording test clip 7 of 8...
[13:35:18] Recording stopped.
[13:35:38] Recording test clip 8 of 8...
[13:36:14] Recording stopped.

===============================================================================
ML/SCRIPTS/RAWBENCH.LUA - 2016-4-20 13:38:10
===============================================================================

A:/DCIM/100EOS5D/M20-1329.MLV:  629 frames, 2.125908 GB
A:/DCIM/100EOS5D/M20-1330.MLV:  793 frames, 2.680199 GB
A:/DCIM/100EOS5D/M20-1331.MLV:  770 frames, 2.602463 GB
A:/DCIM/100EOS5D/M20-1332.MLV:  754 frames, 2.548386 GB
A:/DCIM/100EOS5D/M20-1333.MLV:  751 frames, 2.538246 GB
A:/DCIM/100EOS5D/M20-1334.MLV:  845 frames, 2.855949 GB
A:/DCIM/100EOS5D/M20-1335.MLV:  783 frames, 2.6464 GB
A:/DCIM/100EOS5D/M20-1336.MLV:  835 frames, 2.822151 GB
Recorded frames : 629,793,770,754,751,845,783,835
Quartile summary: 776.5 frames (752.5...814)
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Frank7D on April 20, 2016, 08:17:07 PM
For Reply #55:

A: 32M-MLV.MO
Count = 40
Mean frames = 742

B: HDR512.MO
Count = 24
Mean frames = 926

A vs. B
T -Stat: 5.54
Critical T: 1.68

B beat A (HDR512.MO beat 32M-MLV.MO)
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: a1ex on April 21, 2016, 12:22:24 AM
I've merged lua_fix, so you should be able to run the automatic tests with the next nightly.

How to use:

* highly recommended: a power adapter, so you can leave the camera overnight running the test
* set up your raw recording preferences
* review rawbench.lua (https://bitbucket.org/hudson/magic-lantern/src/raw_benchmark/scripts/extra/rawbench.lua) (don't run code from strangers without understanding what it does)
* save rawbench.lua (https://bitbucket.org/hudson/magic-lantern/src/raw_benchmark/scripts/extra/rawbench.lua) into ML/SCRIPTS/ on your card
* save stats.lua (https://bitbucket.org/hudson/magic-lantern/src/raw_benchmark/scripts/lib/stats.lua) into ML/SCRIPTS/LIB/
* create ML/MODULES/RAW_REC and put there any raw_rec.mo files you want to test (any recent versions would work), either old raw or mlv_lite, renaming them as needed (8.3 filenames)
* mlv_rec.mo binaries might work as well (not tested)
* start the camera in movie mode and follow the instructions on the script console
* when you think it's enough, stop the camera and look at ML/LOGS/RAWBENCH.LOG




To test old versions that can't run on this code base (such as reddeercity's "gold standard" raw_rec, which doesn't run on latest core), you can use this trick:
* place the old ML build on the card
* record a few clips manually
* place the latest nightly on the card, including this script
* start the camera outside LiveView; the script will analyze the clips (count the number of frames), but it will not record anything
* check ML/LOGS/RAWBENCH.LOG
* format the card, keep ML (it will also keep the log)

As you can see, the analysis happens on camera; you don't have to run anything on the PC.




But if you don't mind processing the files on the PC, here's an alternative workflow for manual testing:
* place the old ML build on the card
* record a few clips manually
* place the card in your card reader
* run these scripts (http://www.magiclantern.fm/forum/index.php?topic=17091.msg165745#msg165745) on your DCIM directory to analyze the videos (Linux/Mac; on Windows it might work with Cygwin or msys, not tested)
* copy the output from the console




You'll probably want to test the latest MLV Lite (raw_rec.mo (http://a1ex.magiclantern.fm/bleeding-edge/mlv_lite/hdr512/raw_rec.mo)) against the "classic" raw_rec from reply #42, and against the one from unified.

So, the questions that should be answered by testing are basically the same:

- are the speeds identical in latest MLV Lite vs latest "classic" raw_rec?
- how do they compare (speed-wise) with the raw_rec from the nightly?
- any bugs in mlv_lite that are not present in the "classic" raw_rec? (I've found one (http://www.magiclantern.fm/forum/index.php?topic=16650.msg165990#msg165990) with this script, and hopefully fixed it, but... who knows)
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: a1ex on April 21, 2016, 07:28:58 AM
New data set:

Raw_rec version : ML/MODULES/RAW_REC/32M-MLV.MO
Recorded frames : 671,1045,995,725,771,704,772,564
Raw_rec version : ML/MODULES/RAW_REC/HDR512G.MO
Recorded frames : 710,1253,1455,1326,1430,898,691,536
Raw_rec version : ML/MODULES/RAW_REC/32M-MLV.MO
Recorded frames : 451,1054,959,1045,1038,755,666,798
Raw_rec version : ML/MODULES/RAW_REC/UNIFIED.MO
Recorded frames : 832,1315,1316,1308,1178,1260,708,528
Raw_rec version : ML/MODULES/RAW_REC/32M-RAW.MO
Recorded frames : 1036,1049,983,1129,1114,1175,845,1066
Raw_rec version : ML/MODULES/RAW_REC/32M-RAW.MO
Recorded frames : 806,997,1206,1368,1346,866,1066,731
Raw_rec version : ML/MODULES/RAW_REC/HDR512G.MO
Recorded frames : 924,1259,1389,1080,1094,961,874,899
Raw_rec version : ML/MODULES/RAW_REC/32M-MLV.MO
Recorded frames : 560,1048,1036,719,478,742,1067,880
Raw_rec version : ML/MODULES/RAW_REC/32M-RAW.MO
Recorded frames : 1042,1353,1193,1342,1353,1077,862,565
Raw_rec version : ML/MODULES/RAW_REC/HDR512G.MO
Recorded frames : 818,1096,637,1200,1403,1460,1384,542
Raw_rec version : ML/MODULES/RAW_REC/32M-RAW.MO
Recorded frames : 458,694,862,1282,1374,1224,735,624
Raw_rec version : ML/MODULES/RAW_REC/32M-MLV.MO
Recorded frames : 660,1046,1021,1045,1050,888,771,952
Raw_rec version : ML/MODULES/RAW_REC/32M-RAW.MO
Recorded frames : 586,1361,1346,1360,976,1293,1218,603
Raw_rec version : ML/MODULES/RAW_REC/HDR512G.MO
Recorded frames : 630,1257,849,884,956,1260,1255,850
Raw_rec version : ML/MODULES/RAW_REC/32M-MLV.MO
Recorded frames : 747,577,753,1043,1064,954,1040,929
Raw_rec version : ML/MODULES/RAW_REC/32M-MLV.MO
Recorded frames : 805,695,952,1040,1034,1024,1037,962
Raw_rec version : ML/MODULES/RAW_REC/32M-RAW.MO
Recorded frames : 1039,1286,1357,1359,1350,1274,770,455
Raw_rec version : ML/MODULES/RAW_REC/32M-RAW.MO
Recorded frames : 356,1060,1211,1173,1284,967,968,915
Raw_rec version : ML/MODULES/RAW_REC/HDR512G.MO
Recorded frames : 940,1258,1077,1055,958,1427,1260,640
Raw_rec version : ML/MODULES/RAW_REC/UNIFIED.MO
Recorded frames : 710,964,1004,928,1310,1151,799,1029
Raw_rec version : ML/MODULES/RAW_REC/32M-RAW.MO
Recorded frames : 1126,679,1337,1351,1330,1343,734,881
Raw_rec version : ML/MODULES/RAW_REC/32M-MLV.MO
Recorded frames : 305,1027,952,1031,798,594,869,637
Raw_rec version : ML/MODULES/RAW_REC/32M-RAW.MO
Recorded frames : 1318,1335,1343,1272,1278,722,871,449
Raw_rec version : ML/MODULES/RAW_REC/32M-MLV.MO
Recorded frames : 740,763,756,660,534,774,1042,772
Raw_rec version : ML/MODULES/RAW_REC/32M-MLV.MO
Recorded frames : 746,921,1037,1036,1031,1030,929,734
Raw_rec version : ML/MODULES/RAW_REC/32M-RAW.MO
Recorded frames : 817,1280,1337,1364,1257,1342,728,536
Raw_rec version : ML/MODULES/RAW_REC/UNIFIED.MO
Recorded frames : 754,1297,1318,1046,833,1057,841,842
Raw_rec version : ML/MODULES/RAW_REC/UNIFIED.MO
Recorded frames : 1013,1310,951,944,1329,1165,763,1070
Raw_rec version : ML/MODULES/RAW_REC/UNIFIED.MO
Recorded frames : 412,1009,608,817,1192,1134,844,824
Raw_rec version : ML/MODULES/RAW_REC/UNIFIED.MO
Recorded frames : 1056,1121,1298,1311,1315,852,1063,431
Raw_rec version : ML/MODULES/RAW_REC/UNIFIED.MO
Recorded frames : 892,1260,1311,1038,699,918,1031,890
Raw_rec version : ML/MODULES/RAW_REC/HDR512G.MO
Recorded frames : 819,1256,1424,607,1211,1394,1240,633
Raw_rec version : ML/MODULES/RAW_REC/32M-RAW.MO
Recorded frames : 683,1336,986,726,1110,1335,1328,743
Raw_rec version : ML/MODULES/RAW_REC/32M-RAW.MO
Recorded frames : 1186,1086,965,1355,1332,1327,1078,569
Raw_rec version : ML/MODULES/RAW_REC/32M-MLV.MO
Recorded frames : 300,956,726,866,867,947,563,800
Raw_rec version : ML/MODULES/RAW_REC/HDR512G.MO
Recorded frames : 1031,1269,1395,1399,1383,848,505,506
Raw_rec version : ML/MODULES/RAW_REC/32M-RAW.MO
Recorded frames : 1026,1356,1255,1358,848,1193,642,736
Raw_rec version : ML/MODULES/RAW_REC/32M-MLV.MO
Recorded frames : 323,1032,1030,1028,665,815,635,860
Raw_rec version : ML/MODULES/RAW_REC/UNIFIED.MO
Recorded frames : 1244,1307,1041,1295,1260,1264,821,378
Raw_rec version : ML/MODULES/RAW_REC/UNIFIED.MO
Recorded frames : 439,1064,1009,1063,1038,1260,1232,608
Raw_rec version : ML/MODULES/RAW_REC/UNIFIED.MO
Recorded frames : 754,562,1250,1296,1311,1296,1132,827
Raw_rec version : ML/MODULES/RAW_REC/32M-MLV.MO
Recorded frames : 333,778,1048,1029,1037,881,752,1036
Raw_rec version : ML/MODULES/RAW_REC/32M-RAW.MO
Recorded frames : 1036,1346,1348,1273,1285,1278,875,408
Raw_rec version : ML/MODULES/RAW_REC/32M-MLV.MO
Recorded frames : 340,868,546,604,763,688,758,947


Good news: the script did no longer crash, and there were no errors in the log :)

I also wrote a quick analysis script: analyze-dataset.py (http://a1ex.magiclantern.fm/bleeding-edge/mlv_lite/analyze-dataset.py)

Results:

Quartile stats: median (Q1 ... Q3)
Raw_rec version : ML/MODULES/RAW_REC/32M-MLV.MO:  863 frames (722 ... 1030)
Raw_rec version : ML/MODULES/RAW_REC/UNIFIED.MO: 1040 frames (833 ... 1260)
Raw_rec version : ML/MODULES/RAW_REC/HDR512G.MO: 1079 frames (849 ... 1265)
Raw_rec version : ML/MODULES/RAW_REC/32M-RAW.MO: 1120 frames (855 ... 1331)

P =

           1  1.7114e-07  8.1831e-08  5.2004e-11
  1.7114e-07           1     0.42529     0.21991
  8.1831e-08     0.42529           1     0.81414
  5.2004e-11     0.21991     0.81414           1

T =

           0     -5.4388     -5.6262     -6.9187
      5.4388           0    -0.79972     -1.2308
      5.6262     0.79972           0    -0.23547
      6.9187      1.2308     0.23547           0


The versions are:
hdr512g: 2ab79de
32m-mlv/raw: binaries from reply #42 (30bb06a/e78c7b8)
unified: 4eacd1f
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Ottoga on April 21, 2016, 09:11:01 AM
@A1ex

Having mixed success with the rawbech.lua script. Feedback can be found here: https://www.dropbox.com/s/q5s1whs1vsqwph7/Rawbench_Testing.7z?dl=0
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: a1ex on April 21, 2016, 09:46:10 AM
Quotestdev too large

This is from the raw backend trying to compute the black level (it may also help diagnosing pink frames); you can ignore it for this test.

QuoteLooking at the log files generated, I can't tell which raw module was executing

This happens right after formatting the card, before restarting. I don't see it in the log file though, but I pushed a possible fix.

QuoteI also didn't see the rawbench.lua script under any menu

The menu code is commented out; the script now runs unattended.

QuoteHowever, assuming that it is working correctly, for this script to provide you with any meaningful feedback I think that it needs to limit the size of the size of the clips that it records. To say "2 minutes" before stopping and moving to the next iteration.

The script assumes you have picked recording settings that do not result in very long clips. But, indeed, I can see why a timeout can be useful, so I've hardcoded a 1-minute timeout.

QuoteI think that the rawbench or stats scripts are not correctly handling instances when there is just a single raw clip in the DCIM folder on the card (even if it is a multi chunk one).

True, but I don't see it as a problem, since you need many test clips to get a relevant result. I just delete those lines from the logs.

Thanks for the detailed feedback.
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Ottoga on April 21, 2016, 11:14:13 AM
@A1ex

QuoteThe script assumes you have picked recording settings that do not result in very long clips. But, indeed, I can see why a timeout can be useful, so I've hardcoded a 1-minute timeout.

At the minute mark the error in the linked screenshot was displayed and the camera continued recording.

Screenshot:  https://www.dropbox.com/s/3d119euqnr2dz0i/20160421_184655.jpg?dl=0

Sorry, its a bit blurry

Quote
Which raw_rec module is running?
Quote

Looking at the log files generated, I can't tell which raw module was executing



This happens right after formatting the card, before restarting. I don't see it in the log file though, but I pushed a possible fix.

Nothing in the log file.

[Edit] The log Contents



===============================================================================
ML/SCRIPTS/RAWBENCH.LUA - 2016-4-21 19:37:58
===============================================================================

[19:38:03] Recording test clip 1 of 8...

===============================================================================
ML/SCRIPTS/RAWBENCH.LUA - 2016-4-21 19:42:31
===============================================================================

A:/DCIM/100EOS7D/M21-1938.M02: 4654 frames, 12.745171 GB
ML/SCRIPTS/LIB/stats.lua:29: attempt to perform arithmetic on a nil value (field '?')
stack traceback:
   ML/SCRIPTS/LIB/stats.lua:29: in function 'globals.getMedian'
   ML/SCRIPTS/LIB/stats.lua:69: in function 'stats.quartiles'
   ML/SCRIPTS/RAWBENCH.LUA:216: in function 'globals.check_files'
   [C]: in function 'globals.xpcall'
   ML/SCRIPTS/RAWBENCH.LUA:322: in function 'globals.run_test'

===============================================================================
ML/SCRIPTS/RAWBENCH.LUA - 2016-4-21 19:43:59
===============================================================================

[19:44:04] Recording test clip 1 of 8...

Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: a1ex on April 21, 2016, 11:50:39 AM
Stopping at 1 minute should be OK now.

The other... I'm not sure where to look. Do you see the raw_rec version messages on the screen (including "For next experiment"), right after formatting the card, but before restart?
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Ottoga on April 21, 2016, 12:02:53 PM
Can't look at this again for a couple of hours.  Will test it again tonight though before hitting the sack.
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Ottoga on April 21, 2016, 02:57:54 PM
@A1ex

QuoteStopping at 1 minute should be OK now.

The other... I'm not sure where to look. Do you see the raw_rec version messages on the screen (including "For next experiment"), right after formatting the card, but before restart?

The 1 minute restart is working fine now thanks. I've also tried it with settings that won't record continuously and it is looping through that nicely as well.

Also, I am seeing the raw_rec version messages on screen. They don't stay for long, but they are there.

Thanks for your time and effort to get this script working for. I'll put a recharged battery in tomorrow and let it run until it dies. It should give a few cycles of good info for you.

Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: a1ex on April 21, 2016, 05:50:33 PM
Another test worth trying is api_test.lua (it also tests file I/O, among other things). Might give some hints.
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Frank7D on April 21, 2016, 06:07:58 PM
For reply #58:

A: 32M-MLV.MO
Count = 104
Mean frames = 829

B: HDR512G.MO
Count = 56
Mean frames = 1050

C: UNIFIED.MO
Count = 80
Mean frames = 1012

D: 32M-RAW.MO
Count = 112
Mean frames = 1061

A vs. B
T -Stat: 5.63
Critical T: 1.66

A vs. C
T -Stat: 5.44
Critical T: 1.66

A vs. D
T -Stat: 6.92
Critical T: 1.65

B vs. C
T -Stat: 0.80
Critical T: 1.66

B vs. D
T -Stat: 0.24
Critical T: 1.66

C vs. D
T -Stat: 1.23
Critical T: 1.65

B, C and D are tied (means are not significantly different).
B, C and D beat A (32M-MLV.MO)


EDITED to correct some numbers (the spreadsheet is a work in progress) but it didn't change the bottom line.
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Ottoga on April 23, 2016, 11:59:59 AM
@A1ex

Here are the test results after running rawbench.lua. I ran two series of tests, one with global draw on and the other with it off. I transposed the log file details into a spreadsheet in a format that is bit more readable for me. I'm not sure if all the required log entries have been written into the log or not. As such, the original log files are supplied as well.

I also ran the api_test.lua through a couple of cycles as you suggested and have included the log file.

Test results can be found here:  https://www.dropbox.com/s/ohr3qlav4ozh810/Rawbench_Testing_7D.7z?dl=0

Let me know if you want me to try anything else.
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Frank7D on April 23, 2016, 06:32:05 PM
For Reply #67

GDraw ON

A: raw_rec.mo
Count = 15
Mean frames = 1061

B: raw_lite.mo
Count = 10
Mean frames = 1118

C: 30bb06a.mo
Count = 20
Mean frames = 939

D: e78c7b8.mo
Count = 5
Mean frames = 926

E: Don't know
Count = 5
Mean frames = 920

A vs. B
T -Stat: 0.88
Critical T: 1.73

A vs. C
T -Stat: 1.55
Critical T: 1.70

A vs. D
T -Stat: 1.10
Critical T: 1.89

A vs. E
T -Stat: 1.21
Critical T: 1.89

B vs. C
T -Stat: 2.27
Critical T: 1.75

B vs. D
T -Stat: 1.90
Critical T: 1.83

B vs. E
T -Stat: 2.24
Critical T: 1.83

C vs. D
T -Stat: 0.10
Critical T: 1.94

C vs. E
T -Stat: 0.15
Critical T: 1.94

D vs. E
T -Stat: 0.03
Critical T: 1.89

B beat C, D and E

GDraw OFF

A: raw_rec.mo
Count = 15
Mean frames = 1826

B: raw_lite.mo
Count = 15
Mean frames = 1720

C: 30bb06a.mo
Count = 15
Mean frames = 1848

D: e78c7b8.mo
Count = 10
Mean frames = 1576

A vs. B
T -Stat: 0.68
Critical T: 1.70

A vs. C
T -Stat: 0.16
Critical T: 1.70

A vs. D
T -Stat: 1.14
Critical T: 1.72

B vs. C
T -Stat: 0.97
Critical T: 1.70

B vs. D
T -Stat: 0.68
Critical T: 1.72

C vs. D
T -Stat: 1.41
Critical T: 1.72

No winners


Bottom line:

Global Draw ON:
B beat C, D and E

Global Draw OFF:
No winners

If we're going to do much of this, would it be possible to get the data in a predictable format?
Perhaps something like a1ex provided in Reply #58, i.e.:

Raw_rec version : ML/MODULES/RAW_REC/32M-MLV.MO
Recorded frames : 671,1045,995,725,771,704,772,564
Raw_rec version : ML/MODULES/RAW_REC/HDR512G.MO
Recorded frames : 710,1253,1455,1326,1430,898,691,536
Raw_rec version : ML/MODULES/RAW_REC/32M-MLV.MO
Recorded frames : 451,1054,959,1045,1038,755,666,798
Raw_rec version : ML/MODULES/RAW_REC/UNIFIED.MO
Recorded frames : 832,1315,1316,1308,1178,1260,708,528
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Ottoga on April 23, 2016, 07:38:40 PM
@Frank7D

Happy to provide the feedback in what ever the preferred format is.

I don't know what the following are/represent or how they are calculated. Do you have a sample spreadsheet that I could follow?

T -Stat: 0.97
Critical T: 1.70

The raw_rec version info has been manually captured while the script was running and added after the event as the logging is not capturing this information. I can add the path if it's important even though the script looks for them in a specific location.



Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Frank7D on April 23, 2016, 08:28:05 PM
Ottoga,

Here are the formulas I am using. Note that they do not show all the math since I am using some Excel functions ("DEVSQ" etc.).
Sum of the Squared Deviations
x=DEVSQ(Data Set A)+DEVSQ(Data Set B)

Pooled Sample Variance
y=x/([Count of A]-1+[Count of B]-1)

Standard Error of Difference in Means
z=SQRT(y*(1/[Count of A]+1/[Count of B]))

Our T -Stat
=([Mean of A]-[Mean of B])/z

U (used in DF calculation)
U=[Mean of B]^2/[Mean of A]^2

Degrees of Freedom
V=(1/[Count of A]+U/[Count of B])^2/(1/([Count of A]^2*([Count of A]-1))+U^2/([Count of B]^2*([Count of B]-1)))

Critical T Value @ 0.95 (95% confidence)
=TINV(0.05*2,V)


If the T -Stat is larger than the Critical T Value, then the difference in the means of the two data sets is significant.

If someone wants the spreadsheet, I can upload it.

As far as the data format, it doesn't matter much to me except that it would be nice if it were consistent.

EDIT: I should probably give the site where I got these formulas:
http://policeanalyst.com/performing-a-statistical-t-test-in-excel/ (http://policeanalyst.com/performing-a-statistical-t-test-in-excel/)
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Ottoga on April 24, 2016, 04:45:02 PM
@Frank7D

Interesting article. Also, thanks for your analysis. I've updated my spreadsheet as well for future use.
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: a1ex on April 26, 2016, 07:55:02 PM
I think I know why the raw_rec version didn't appear in the log file. When formatting the card, I should have closed the log file and reopened it afterwards...

Maybe the reason it worked fine on 5D3 was... a Canon bug?!
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Ottoga on April 27, 2016, 12:03:44 PM
@a1lex

I'll try it again tonight and let you know if it is now logging correctly.

btw: When the script says finish, have a little sleep.... My camera appears to go into preview mode of a clip (don't know which one).  After the sleep period the script resumes and continues as per normal. As a result though, the camera never really gets the intended break and chance to cool a little.

Here is a link to a video captured with my phone:  https://www.dropbox.com/s/880kd2t74rddk21/20160427_193655.mp4?dl=0
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: a1ex on April 27, 2016, 12:10:08 PM
Got it - I ran the script without mlv_play. So, when it went to preview mode on my camera, there was no activity.
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Ottoga on April 27, 2016, 12:31:54 PM
@A1ex

Thanks, I just need to remember to unload the mlv_play module as well before running the script again.

The script is still running so no feedback on the logging yet.
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Ottoga on April 28, 2016, 01:47:39 AM
@A1lex - logging is working correctly now. Thanks
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: a1ex on November 09, 2016, 11:47:59 PM
Benchmarked this change (https://bitbucket.org/hudson/magic-lantern/commits/47e2c4efc13b90025c0d16e390f4461ffce618b6?at=unified) against the previous one:


Raw_rec version : ML/MODULES/RAW_REC/2AB.MO
345,712,1001,921,1116,1202,1437,612
374,1130,1334,1156,1165,620,873,838
425,1157,665,888,1256,1334,954,799
958,1255,1457,1336,1331,1012,902,244
834,1256,1322,1309,1179,508,862,616
532,1256,1328,1237,650,858,1238,640

Raw_rec version : ML/MODULES/RAW_REC/47E.MO
678,789,1260,1487,1502,1126,769,516
377,1181,1252,903,942,1137,790,671
379,1259,625,556,982,1389,1260,430
980,657,1011,1263,1260,1260,879,603
772,719,994,1273,1273,1168,1036,832
893,1356,1495,1143,697,1283,1185,647
454,1260,1148,845,666,843,1258,513
663,1224,1113,486,642,1337,1183,923
980,1239,644,796,1260,1251,1264,786

Quartile stats: median (Q1 ... Q3)
Raw_rec version : ML/MODULES/RAW_REC/2AB.MO:  980 frames (689 ... 1256)
Raw_rec version : ML/MODULES/RAW_REC/47E.MO:  981 frames (688 ... 1259)

P =

           1     0.98743
     0.98743           1

T =

           0   -0.015784
    0.015784           0


Need to remember how to interpret these numbers, but the speed appears to be the same.
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Frank7D on November 17, 2016, 07:21:21 PM
A: 2AB.MO
Count = 48
Mean frames = 967

B: 47E.MO
Count = 72
Mean frames = 968

A vs. B
T -Stat: 0.015784
Critical T: 1.66

T -Stat is less than Critical T, so A and B are tied.
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Frank7D on November 17, 2016, 07:42:59 PM
Posting to be notified.
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: a1ex on December 02, 2016, 11:14:21 AM
Partial results for 3 recent changes from mlv_lite (https://bitbucket.org/hudson/magic-lantern/commits/branch/new_raw_format).

5D3, KB 32GB 1000x, 1920x1080 30fps, global draw on (raw histogram, spotmeter, level indicator, fast zebras).


Raw_rec version : ML/MODULES/RAW_REC/451.MO
627,825,721,808,957
705,826,687,735,910
598,821,812,756,678
467,703,819,764,866
555,920,676,647,727
519,473,717,706,897
688,729,741,717,876

Raw_rec version : ML/MODULES/RAW_REC/AAC.MO
798,873,947,895,995
793,986,753,806,784
751,799,763,735,448
851,971,957,897,653
496,831,837,769,965
636,945,879,962,704
671,839,894,887,885
666,707,778,951,984

Raw_rec version : ML/MODULES/RAW_REC/2DC.MO
694,997,812,890,819
627,701,920,795,982
694,549,805,712,683
784,857,987,876,499
601,915,907,457,703

Quartile stats: median (Q1 ... Q3)
Raw_rec version : ML/MODULES/RAW_REC/451.MO:  727 frames (680 ... 821)
Raw_rec version : ML/MODULES/RAW_REC/2DC.MO:  795 frames (691 ... 894)
Raw_rec version : ML/MODULES/RAW_REC/AAC.MO:  834 frames (752 ... 921)

P =

           1     0.29402   0.0045415
     0.29402           1     0.17861
   0.0045415     0.17861           1

T =

           0     -1.0589     -2.9284
      1.0589           0     -1.3602
      2.9284      1.3602           0


Same experiment after running a bit longer (edit):

Raw_rec version : ML/MODULES/RAW_REC/451.MO
627,825,721,808,957
705,826,687,735,910
598,821,812,756,678
467,703,819,764,866
555,920,676,647,727
519,473,717,706,897
688,729,741,717,876
682,891,731,615,827
697,702,799,627,701
522,694,722,866,884
709,1020,930,906,911
698,859,883,913,865
717,895,849,803,754
708,867,813,753,898

Raw_rec version : ML/MODULES/RAW_REC/AAC.MO
798,873,947,895,995
793,986,753,806,784
751,799,763,735,448
851,971,957,897,653
496,831,837,769,965
636,945,879,962,704
671,839,894,887,885
666,707,778,951,984
773,832,913,963,816
705,919,893,894,933
805,809,809,1017,885
788,918,825,949,813
694,647,957,980,980
684,768,737,867,877
739,864,726,761,836

Raw_rec version : ML/MODULES/RAW_REC/2DC.MO
694,997,812,890,819
627,701,920,795,982
694,549,805,712,683
784,857,987,876,499
601,915,907,457,703
814,904,927,798,961
791,985,981,909,976
567,801,822,820,966
842,869,966,876,817
934,860,717,869,763
860,888,966,963,877
764,784,782,876,903
842,972,904,956,856
683,723,980,964,895
693,634,801,891,838
851,975,975,967,876
695,806,886,889,859
694,877,826,847,729

Quartile stats: median (Q1 ... Q3)
Raw_rec version : ML/MODULES/RAW_REC/451.MO:  747 frames (698 ... 866)
Raw_rec version : ML/MODULES/RAW_REC/AAC.MO:  832 frames (762 ... 917)
Raw_rec version : ML/MODULES/RAW_REC/2DC.MO:  857 frames (782 ... 909)

P =

           1  0.00087472  0.00033621
  0.00087472           1     0.85151
  0.00033621     0.85151           1

T =

           0     -3.3996     -3.6659
      3.3996           0    -0.18749
      3.6659     0.18749           0


Side note: I expect 451 to have the same performance as 47e (from previous experiment); however, comparing them requires swapping the config files too (or manually crafted config files, with config autosave disabled). That's because of the fine resolution control added recently. Will give it a try later.
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: DeafEyeJedi on December 02, 2016, 07:08:53 PM
Absolutely wonderful progress @a1ex and Thanks for the fix with this!
Title: Re: Squeezing the last bit of performance out of MLV Lite (for testers)
Post by: Frank7D on December 03, 2016, 07:24:21 AM
For Reply #80:

A: 451.MO
Count = 70
Mean frames = 763

B: AAC.MO
Count = 75
Mean frames = 828

C: 2DC.MO
Count = 90
Mean frames = 832

A vs. B
T -Stat: 3.39962
Critical T: 1.66

A vs. C
T -Stat: 3.66587
Critical T: 1.65

B vs. C
T -Stat: 0.18749
Critical T: 1.65

B and C both beat A
B and C are tied