Messages

Quick test, did a recording using mlv_rec with mlv_snd on the crop_rec_4k branch, same as two days ago.

I can record fine, and I end up with the following files:

Code Select Expand

M19-0910.M00
M19-0910.M01
M19-0910.M02
M19-0910.M03
M19-0910.M04
M19-0910.M05
M19-0910.M06
M19-0910.MLV

But when I stop the recording I get the famous "Audio failed to stop: 4" error, the RAW video option in the movie menu says "Stopping..." and ML is "stuck". Can't shutdown the camera, can't start another recording.


I was using the following settings:


Edit:
I think I found the source of the problem. I was recording in 8kHz (since I only need some audio to sync, and don't want to waste to much bandwidth) and when I stop the recording the camera returns "Audio failed to stop: 4". Returning the setting to 48kHz eliminates the problem.

Another issue is, the files (MVI_xxxx.DAT and IMG_xxxx.MLV) are empty when the camera crashes. This is using the MLV_lite module with H264 proxy enabled.

Mlv_lite with proxy is the combination I'm testing atm.

1280x720@10b ran for 8:22 before the camera crashed.

Mlv_rec + mlv_snd isn't usable because the audio won't stop.

Sorry, edited my post a bit.

I'm running: crop_rec_4k.2017Oct16.700D115

Lowering the bitrate (to 0.1x) for the video seems to help a bit. Recording for almost 3 minutes now.

Edit: I stopped the recording after 7 minutes.

Restarting a recording is hit or miss. Sometimes the live view is dark, sometimes it's the normal live view.

I've been testing h264 proxy and mlv_snd recording on my 700D and I'm either missing something or running into some errors.

I want to have some sort of audio track to be able to sync 3 recordings (2 camera's and a screen capture)

Recording H264 proxy is hit or miss, especially using lossless recording, but even 12bps or 10bps isn't stable. I get flickering, pink artifacts, pink bars. After some recordings the live view goes black, and restarting gives bars on the screen.
Recording 1280x720@10bbp makes the camera freeze after 35-50 seconds. I'm running crop_rec_4k.2017Oct16.700D115

Using mlv_rec with mlv_snd records but at the end of the record it errors with: "audio failed to stop 4" error. This is after the first recording, and the camera "softlocks"with a busy notification.

I'll do some more research, but was curious if there were others with the same issues.

MLVFS works, and saves a lot of space, but the development of a 2 minute clip takes 45 minutes.

dcraw / ffmpeg runs at 4-4.5 fps at the moment, on a slow virtual machine running Ubuntu Xenial, using extracted dng's. Storage is a samba mounted network storage, so that's not in it's advantage either.

My MBPr, using external USB3.0 SSD storage runs at approximately 3-3.5 fps using MLVFS.

More testing results:
Profile 2?
Debayer 0

Code Select Expand


find DNG/ -iname '*.dng'  -print0 | sort -z | xargs -d "\n" -0 ~/Switch/bin/dcraw +M -c -6 -W -q 0 -w | ffmpeg -f image2pipe -vcodec ppm -r 25 -i pipe:0 -vcodec prores -pix_fmt yuv422p10le -n -r 25 M23-1452.mov
frame=  100 fps=4.4 q=-0.0 Lsize=   51746kB time=00:00:04.00 bitrate=105975.6kbits/s   


Debayer 3

Code Select Expand


find DNG/ -iname '*.dng'  -print0 | sort -z | xargs -d "\n" -0 ~/Switch/bin/dcraw +M -c -6 -W -q 3 -w | ffmpeg -f image2pipe -vcodec ppm -r 25 -i pipe:0 -vcodec prores -pix_fmt yuv422p10le -n -r 25 -profile:v 2 M23-1452_profile2_q3.mov
frame=  100 fps=2.4 q=-0.0 Lsize=   48464kB time=00:00:04.00 bitrate=99255.2kbits/s   


Quality 3 (Hq)
Debayer 0

Code Select Expand

find DNG/ -iname '*.dng'  -print0 | sort -z | xargs -d "\n" -0 ~/Switch/bin/dcraw +M -c -6 -W -q 0 -w | ffmpeg -f image2pipe -vcodec ppm -r 25 -i pipe:0 -vcodec prores -pix_fmt yuv422p10le -n -r 25 -profile:v 3 M23-1452_profile3.mov
frame=  100 fps=4.4 q=-0.0 Lsize=   77486kB time=00:00:04.00 bitrate=158690.4kbits/s   

Debayer 3

Code Select Expand

find DNG/ -iname '*.dng'  -print0 | sort -z | xargs -d "\n" -0 ~/Switch/bin/dcraw +M -c -6 -W -q 3 -w | ffmpeg -f image2pipe -vcodec ppm -r 25 -i pipe:0 -vcodec prores -pix_fmt yuv422p10le -n -r 25 -profile:v 3 M23-1452_profile3_q3.mov
frame=  100 fps=2.4 q=-0.0 Lsize=   70703kB time=00:00:04.00 bitrate=144800.6kbits/s   

Prores_ks
Quality 2
Debayer 3

Code Select Expand


find DNG/ -iname '*.dng'  -print0 | sort -z | xargs -d "\n" -0 ~/Switch/bin/dcraw +M -c -6 -W -q 3 -w | ffmpeg -f image2pipe -vcodec ppm -r 25 -i pipe:0 -vcodec prores_ks -pix_fmt yuv422p10le -n -r 25 -profile:v 2 M23-1452_proresks_profile2_q3.mov
frame=  100 fps=2.1 q=-0.0 Lsize=   35701kB time=00:00:04.00 bitrate=73114.9kbits/s

Code Select Expand


find DNG/ -iname '*.dng'  -print0 | sort -z | xargs -d "\n" -0 ~/Switch/bin/dcraw +M -c -6 -W -q 3 -w | ffmpeg -f image2pipe -vcodec ppm -r 25 -i pipe:0 -vcodec prores_ks -pix_fmt yuv422p10le -n -r 25 -profile:v 3 M23-1452_proresks_profile3_q3.mov
frame=  100 fps=2.1 q=-0.0 Lsize=   53619kB time=00:00:04.00 bitrate=109811.1kbits/s

Prores_ks 4444

Code Select Expand


find DNG/ -iname '*.dng'  -print0 | sort -z | xargs -d "\n" -0 ~/Switch/bin/dcraw +M -c -6 -W -q 3 -w | ffmpeg -f image2pipe -vcodec ppm -r 25 -i pipe:0 -vcodec prores_ks -pix_fmt yuv444p10 -n -r 25 M23-1452_proresks_4444.mov
frame=  100 fps=1.8 q=-0.0 Lsize=   79918kB time=00:00:04.00 bitrate=163672.1kbits/s   


On the MBPr
100 frames read from MLVFS vs 100 frames read from storage:
Storage:

Code Select Expand


frame=  100 fps= 11 q=-0.0 Lsize=   50995kB time=00:00:03.96 bitrate=105491.5kbits/s speed=0.446x

MLVFS:

Code Select Expand


frame=  100 fps=6.6 q=-0.0 Lsize=   50995kB time=00:00:03.96 bitrate=105491.5kbits/s speed=0.261x   

Isn´t the "set thread manually" to 1 doing exactly what you want? Processing one file at the time?

I think I messed up a setting skipping the conversion to ProRes, which gave the impression it was doing the MLV_dump first for every file, and after that doing the conversion to ProRes.

As soon as you leave the ProRes menu this switch will be reset. Reason for this is that there is a good chance you´ll forget it next time you start switch and then the damage is done.

-1

What is the use for the DNG files after you have converted them to ProRes. You can quite quickly extract them from the MLV and they just take up space.

Can you also add something to clean up the temp dng files, so larger queue's don't need as much space as needed now.

Another improvement would be to finish one file at the time. First extract the dng's from the first file and convert it to ProRes, remove the dng's and then move on to the next file.

Quality check isn't my cup of tea. But the speed improvement is huge! 4:1 instead of 20:1.

Can you also change the output directory to ProRes422 instead of ProRes4444? So you can keep different versions apart?

1522 frames 1:01

Code Select Expand

Total running time:   
0 Days, 00 Hrs, 04 Min, 11 Sec
X to FFmpeg ProRes:   
XxXx
mlv_dump:   
0 Days, 00 Hrs, 01 Min, 22 Sec
raw2dng:   
XxXx
dcraw_FFmpeg ProRes:   
0 Days, 00 Hrs, 02 Min, 44 Sec
cr2hdr(CR2 files):   
XxXx

That's a nice improvement! This way it's a lot easier to use the footage.

Can you also remove the DNG files after the run is complete?

5758 frames of 1360x764:

Using codec yuv422p10le

Code Select Expand

Total running time:   
0 Days, 00 Hrs, 55 Min, 40 Sec
X to FFmpeg ProRes:   
XxXx
mlv_dump:   
0 Days, 00 Hrs, 08 Min, 24 Sec
raw2dng:   
XxXx
dcraw_FFmpeg ProRes:   
0 Days, 00 Hrs, 47 Min, 11 Sec
cr2hdr(CR2 files):   
XxXx


Still quite a lot, 20-30 seconds of work for every second of footage.

This is with the old version (with a small change)

Great! Thanks for looking for solutions! I even like the 422 codec a lot! 444 is way overkill for a lot of use cases. Mine is one of them.

If you want I'll give it a go on bigger files.

I have total running time in the notifier window which should pop up when done.

This summary would be nice to have in the LOG.txt file. You could add the runtime of a task after that task is done.

I am using external drives, because the internal drive is way to small on my MBPr. 256GB isn't that much once you start doing video, but still I think it's a good thing to be able to alter the amount of processes based on preferences, other tasks, etc.

MLV is 43GB.
DNG's are 88GB
Prores is 35GB

The Prores image is ok, but waiting 11 hours for it to build is unpractical and it's more convenient to use the DNG files instead of the Prores file.

Thank you!

I was running the process on a Macbook Pro 13" 2015 i5. I know it's not the fastest, but it's workable.

The MLV is 45.76GB large and contains 31345 frames and it takes 50 minutes to extract the dng files from the MLV.  The process of creating a prores movie takes ages Maybe you can add the running times to the log.txt file?

The SSH part, is simple running the command from a console, terminal, like pressing ctrl+alt+f1 on a linux box. If it's possible to start the run command for a task from the command prompt would be very helpful. I am aware of the fact that multithreaded might be an issue.

It would also be helpful to change the number of parallel tasks, because working with MLV's that are 20 minutes long takes up a lot of disk space. 4x100 GB of temp data is a lot!

Looks good Danne!

Does it also run from the command line, over SSH? That would be awesome because I can remotely start the conversion process (using SSH) on pc's that are more suitable for doing those tasks.

What is a normal render time for near full HD to Prores 4444? I had a 20 minute clip that took 11 hours to render from DNG to Prores. Seems a bit long and impractical.

The null pointer changes are incorrect. Both tests are reversed. Do other cameras then the 5D3 incorporate this test since they can't be affected by the same bug.

For the 700D the solution was to remove the * 2 and / 2 in these statements.

Code Select Expand

TTL_Args.xRes = width * 2;
    TTL_Args.yRes = height / 2;

Oops. I forgot to change the test for 0XE5E5E5E5. it should continue if it is 0XE5E5E5E5 and not if it''s something else.

The doubling only has to be removed for the FRSP, the mlv do need half the height, double the width. I'm not really sure about the simple silent pictures.

I'll test the other option for the set flags later and match it to the 600D and check if that works.

The real test today was a bit of a dissapointment. Recording at 1736x976 wasn't continyous... It stopped after 33000 frames (because the card was full)

Did a "short" testrun, recording 16:9 using 11 bits lossless compression.

Code Select Expand


    FPS         : 25.000000
...
    Frames Video: 23504
...
    Res:  1736x976
...
     ISO:        800
...
     Time: 942568.613000 ms
...
   Frame: #23503


Only thing is the battery went out of juice Tomorrow I'll be recording an interview and will be using this camera as my second (backup) camera, using plugin power. Let's see if we can fill up the SD card without interruption.

1) I altered the scan_A5A5 routine so it won't trip over A5A5A5A5 at these addresses:
0xF8C950E4 - 0xF8C950F0
0xF8C95BD0 - 0xF8C95BDC
0xF8C9F1B0 - 0xF8C9F1B4
If they do appear at another location it will show the warning again.

I haven't had a good look at the buffers yet.

3) I've combined the resources for the 700D * and the 5D3 *. It is working on my 700D and should work on the 5D3. Hope this is a solution you can live with.

I wasn't able to get a compressed Full Resoution Silent Picture DNG but it did work for Simple Silent:

Found the issue with the FRSP. Tricking the compressor in half the height, double the width makes it fail. Removing the width * 2 and height / 2 does grab the lossless dng. I don't know if they are valid, but Rawtherapee and Resolve 14 accept them.

P.S. Here are the entry points for lossless decompression on the 700D. I do get a lot of malloc errors, but replacing malloc with fio_malloc @mlv_play.c:2220 fixed those.

Code Select Expand

    if (is_camera("700D", "1.1.4"))
    {
        Setup_DecodeLosslessRawPath = (void*)0xFF4294DC;
        Start_DecodeLosslessPath = (void*)0xFF4295A4;
        Cleanup_DecodeLosslessPath = (void*)0xFF429708;
    }

I also tried to get this running on the 600D, but I can't get the dm-spy-experiments branch running on the 600D. The led blinks, once per second for a few times, then it blinks fast and it starts over. Same for the startup-log builds. (even the older ones). Am I missing something?

Finding the stubs was easy, except for the setflags bit of course.

Code Select Expand

    if (is_camera("600D", "1.0.2"))
    {
        /* ProcessTwoInTwoOutJpegPath, 600D 1.0.2 */
        TTL_SetArgs     = (void *) 0xFF2155E0;      /* fills TTJ_Args struct; PictureSize(Mem1ToRaw) */
        TTL_Prepare     = (void *) 0xFF2FE354;      /* called right after ProcessTwoInTwoOutJpegPath(R) Start(%d); */
                                                    /* calls [TTJ] GetPathResources and sets up the encoder for RAW */
        TTL_RegisterCBR = (void *) 0xFF2FD424;      /* RegisterTwoInTwoOutJpegPathCompleteCBR */
        TTL_SetFlags    = (void *) set_flags_600D;  /* this function is also inline on 600D??? */
        TTL_Start       = (void *) 0xFF2FE3C4;      /* called next; starts the EDmac transfers */
        TTL_Stop        = (void *) 0xFF2FE3FC;      /* called right after sssStopMem1ToRawPath */
        TTL_Finish      = (void *) 0xFF2FE434;      /* called next; calls UnlockEngineResources and returns output size from JpCoreCompleteCBR */
    }

dfort: https://bitbucket.org/ehoutsma/magic-lantern/src/?at=crop_rec_4k_700d

Just took 2 pictures, one in DNG and one in Lossless DNG (both with the lenscap on), Lossless DNG was 1.4MB, the normal DNG 2.3MB.

I haven't disabled the null pointer warning

I tool the crop_rec_4k branche, took out the 5D3 specific boothacks and that runs like a charm.

I need some help with the raw buffers (for the mlv_lite) but I'll share my code with you tomorrow.

Warning, I did encounter a null pointer error during recording when the card couldn't keep up and the scan_a5a5 routines warns about a null pointer bug all the time...

You can find the ProcessTwoInTwoOutJpegPath for the 700D here. http://www.magiclantern.fm/forum/index.php?topic=18443.msg188163#msg188163

Since I'm unsure about the values above, I won't share a full build.