MLV App 1.14 - All in one MLV Video Post Processing App [Windows, Mac and Linux]

[70MM13]

How about inserting a black frame for every detected corrupted frame?
General question. Is a corrupted frame in one MLV file aborting the whole batch process in MLV app? If not I could live without the export of corrupted files although cool if it would work with black frames.

On a side note. I tried multi rendering on dual iso files since those are really slow to process. Flickering occured so had to run processing with only one copy of Mlv App to get comsistent results. Strange. They shouldn´t interfere with each other?

This needs to be addressed sooner than later!  I'm just getting into dual iso and that's where we NEED to improve rendering speed.  I suppose we can work around it with VMs, but that's getting a little extreme, no?

Maybe it's time to set up a good old fashioned render farm.  If mlvapp won't take advantage of multi cores, a farm could be made with a bunch of "obsolete" machines, but then some kind of job control will definitely be needed!

I just had a thought:  can mlvapp run in a sandbox?  that might solve Danne's discovery.  also, Danne, were you using different drives for the different instances?  That might be a stretch, but again, I'm just thinking out loud...  I'd test it but i'm in the middle of a job so I can't check until tomorrow.  I will definitely try then...

[masc]

Hoaa... what's going on here?!

I don't think that one MLVApp copy interacts with any other copy. If this happens, it could be a problem of the operating system or a defect HDD. So I don't believe in that. If so, please tell me how to reproduce. But those app copies work 100% independant (if you work on different clips at the same time).

In the last year we worked a lot to get nearly all calculations in MLVApp multithreaded. Maybe it is not optimal here and there how these jobs work together (that means I am sure it isn't), but this is hard to make better. Some single calculations like dualiso are single threaded, because we had still no luck.
Deciding to do a farm means deleting all we did in the past year - making MLVApp singlethreaded again. Otherwise a farm makes no sense. If that is faster?! Maybe. Maybe not. I don't know. And I can't try out. On a dualcore there can be just one export at a time (one thread processing in MLVApp + one thread for ffmpeg), on a quadcore 2, etc....

If MLVApp detects a corrupted frame on export, it asks to abort the export of this clip ot of this batch. It does not crash somehow or does something strange. I would again suggest to try another ML build for your camera. I can't remember having seen a corrupted clip from my cameras since I use them, in a way the export would stop. The export only stops, if the header tells "there is data", but there isn't. Most of other corruptions are bypassed.
Until now we had to build up corrupted clips our self by switch some bytes in good clips. It is always better to fight against the reason and not against the effects.

Edit: for ffmpeg export black frames should already be generated. For cDNG the described messagebox is coming with the buttons "Skip frame, Abort current export, Abort batch export". For MLV export the described messagebox is coming with the buttons "Abort current export, Abort batch export". Until now we don't "generate" frames.

[Danne]

Retested dual iso, seems to be working with multiple copies of Mlv App. Not sure what was going on before. Was a while since I tested this. My mac book pro almost lifted from the table(fans blowing ).
Anyway. I always found multiprocessing faster when exporting through ffmpeg but of course architecture could be a mess to build for this.
Still not clear to me how corruption in mlv files are handled(aborting the whole batch or not) but I think it´s not unthinkable to foresee at least one or two files that will stop the batch party if one has filmed bigger amounts of mlv files. Are we certain Mlv App finds each corrupted file prior to export or could this happen also randomly while exporting?

Edit:
One shit ass idea is to select a group of MLV files in the first Mlv App and right click on this group. The user could here get the question if one wants to open up thos e files in another copy of Mlv App, hehe. Then actual working Mlv App copy itself and move the selected files. Oh man, it´s friday...

[masc]

Edit:
One shit ass idea is to select a group of MLV files in the first Mlv App and right click on this group. The user could here get the question if one wants to open up thos e files in another copy of Mlv App, hehe. Then actual working Mlv App copy itself and move the selected files. Oh man, it´s friday...

Hahaha... yes. In german this has the name "verschlimm-bessern" (make it worse-better)
* Just a little joke, I know you all want to help. *

[Danne]

hehe, anyway. Anamorphic modes on eosm recorded in dual iso as always comes out very, very nice. Bouncyball did some really good job here with getting the dual iso files working with all bits. One should start using the cam more often 

[bouncyball]

Hello everyone!

Is the problems happens due to mlv_dump code it should be possible to insert the "relaxed" mode skipping files right bouncyball?

The mlvapp import has nothing in common with mlv_dump code at all

Let me make clear some "corrupted frame" aspects here.
1. mlvapp importer reads bytes from MLV but on that stage does not analyze any frame data itself for corruption.
2. errors come from the fact that _MLV_ itself is corrupted, e.g. specs are violated, header block missing or block size is not matched to the value written in the header etc etc ...
3. error is passed to higher (QT) part of the application, where the proggie (on error) spits out the message box with error and where some actions could be performed (skip, cancel etc).
So it is all about MLV corruption, not the particular frame, hence substitution with black frames is not applicable.

BTW mlvapp has built in brute force method (I've implemented long time ago) if the block header could not be found on common/expected address offset, proggie searches the block name until the end of file. I have one corrupted MLV like this and mlvapp loads it fine, slower though.

Now what can we do: introduce somewhere checking option allowing, on error, stop processing of current MLV (leave all files produced intact) and proceed to the next one without user intervnention. This will finish whole batch despite corrupted MLVs.

The problem aborting at corrupted frames was discussed many times here already. Mostly a corrupted file can't be handled for roundtrips, audio does not fit to video, linking does not work, etc. ... these are only some of the reasons why bouncyball and I decided to aborted export for corrupted clips. If just the last frame is currupted, this might be a a special case - but even than it leads very often to linking issues when working with Resolve or FCPX.

I guess only problem here will be with linking as masc described above. But if one let the option (I described above) checked, he/she already agreed that this could happen.

Regards
bb

[ilia3101]

I don't think that one MLVApp copy interacts with any other copy. If this happens, it could be a problem of the operating system or a defect HDD. So I don't believe in that. If so, please tell me how to reproduce. But those app copies work 100% independant (if you work on different clips at the same time).

+1

Should even be independent even if working on the same clip

[Danne]

Now what can we do: introduce somewhere checking option allowing, on error, stop processing of current MLV (leave all files produced intact) and proceed to the next one without user intervnention. This will finish whole batch despite corrupted MLVs.
I guess only problem here will be with linking as masc described above. But if one let the option (I described above) checked, he/she already agreed that this could happen.

Now we´re talkin´

[70MM13]

it's awesome to see all you heavyweights in discussion...

[bouncyball]

Now we´re talkin´

Yes, and one more thing, the real single frame corruption can be detected only when decoding wrong lossless raw data by LJ decoder (it spits error that data can not be decompressed). If raw is uncompressed we can not detect corruption until we look at the image on the screen

[mothaibaphoto]

Hi, guys, thank you for your great app!!!
Just downloaded 1.5 (win64) and tried black frame subtraction.
1. 1.5 version crashes constantly after processing 502-504 frames on any file (longer than 504 frames).
If process without black frame it not crashes.
2. 1.4 processes that files well, but... just only one and than i need to terminate the app with task manager.
If i try to select another file after processing one, or add several files to batch - it warns
"Could not open file (path to black frame MLV)" then it warns
"Could not open file (path to MLV to process)".
The only way to close this window is to terminate the app.
If process without black frame, it is possible to select any number of files to process
or add to batch and process too.

[bouncyball]

+1
Should even be independent even if working on the same clip

Haha flickering would happen only if we will implement processing of EVEN and ODD frames in mlvapp. Then one instance of mlvapp should process the odd frames with different settings than second instance even frames

[70MM13]

OK, I am may asking for too much, but since everyone's all here... 

The shot I am currently grading couldn't be done in mlvapp (yet?) because the bright red LED is shining right at the camera, and when grading, it just gets too strong and blows out.

In resolve, I was able to avoid the problem by using a "power zone" (selection mask) to isolate it, first by colour, and also by a circle selection tool to make sure nothing else is touched.  I did this on the first node, and then graded with the second node, without changing the led...

Would this be too much to ask?

see image, selecting the colour of the led for the mask



 

[bouncyball]

I guess @masc should answer that MASK question

[70MM13]

LOL that's an awesome answer!

[bouncyball]

1. 1.5 version crashes constantly after processing 502-504 frames on any file (longer than 504 frames).
If process without black frame it not crashes.

Very odd problem indeed. I have to reproduce this somehow...

Edit: which OS?

[mothaibaphoto]

Edit: which OS?

...
Just downloaded 1.5 (win64) and tried black frame subtraction.
...

[bouncyball]

Ah, OK, thanx .

[masc]

...One should start using the cam more often 

Hehe... oh yes... 95% is implementing this app, 5% is using this app. But hey... in winter it's dark if I could use my cam...

...
The shot I am currently grading couldn't be done in mlvapp (yet?) because the bright red LED is shining right at the camera, and when grading, it just gets too strong and blows out.
...
see image, selecting the colour of the led for the mask



 

Hm... indeed an interesting task. When I see your picture I first think about the Hue vs. Luma diagram and I would try to darken red tones. Have you tried that? Only because only the LED is red.
Masking is in principle not too complicated - I got that working with the linear gradient function. But this more or less doubled the code needed for those parameters. And the code became somewhat hard to understand - I think Ilia hates me for that
The next question is: in what way we select which areas in the picture? I really like the way it is realized in Resolve. But implementing a 2nd Resolve? I think just using Resolve is easier. It is a great software, if such features are needed!

[70MM13]

It may be possible to do with your idea once nodes are an option...  it would probably require some very precise work in a few stages (nodes) to keep it controlled while increasing the overall gain.

I agree that it may be that resolve is required to do such things, but I thought it would be worth asking anyways.

I really mean it when I say that I now prefer mlvapp (except for these difficult situations)

[masc]

@bouncyball: also tried using darkframe and also got a crash on first try. Somewhere about 200 frames were okay, than OSX told me this:

Code Select Expand

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libsystem_kernel.dylib        0x00007fff91479866 __pthread_kill + 10
1   libsystem_pthread.dylib        0x00007fff8e24735c pthread_kill + 92
2   libsystem_c.dylib              0x00007fff8e9c4b26 abort + 125
3   libsystem_malloc.dylib        0x00007fff9981207f free + 411
4   magiclantern.MLV App          0x0000000108ec0e29 load_all_chunks + 345
5   magiclantern.MLV App          0x0000000108ebe22a openMlvClip + 106
6   magiclantern.MLV App          0x0000000108efef8a df_load_ext + 218
7   magiclantern.MLV App          0x0000000108ec3af5 applyLLRawProcObject + 53
8   magiclantern.MLV App          0x0000000108ebd8df getMlvRawFrameFloat + 127
9   magiclantern.MLV App          0x0000000108ebd430 get_mlv_raw_frame_debayered + 48
10  magiclantern.MLV App          0x0000000108ebde15 getMlvRawFrameDebayered + 197
11  magiclantern.MLV App          0x0000000108ebdee0 getMlvProcessedFrame16 + 80
12  magiclantern.MLV App          0x0000000108e839af MainWindow::startExportPipe(QString) + 22943
13  magiclantern.MLV App          0x0000000108e9dc56 MainWindow::exportHandler() + 2198
Looks like there is a problem with darkframe loading. Should use debugger next. Clip is really large... (4.5K).

[bouncyball]

@bouncyball: also tried using darkframe and also got a crash on first try.

Yeah... something is wrong and it's wrong from the beginning. I have to try to nail this with valgrind.

[IDA_ML]

Hello guys,

I played a little with version 1.5 and I love it!  What a valuable piece of software!  Congratulations to everyone involved.

Recently, I managed to borrow a 5D3 and shot a few test files to check the latest MLVApp.  Here is a file of a high-contrast scene shot in the 1x3 mode at Dual ISO 100/800. 

https://we.tl/t-z6hpWfLjDk

The noise suppression in the darkest areas is generally very good but these magenta fringing artifacts along the straight lines of the laundry stand and the painting on the right are killing me.   Could you guys please have a look?  Thanks.

[masc]

Something in load_all_chunks seems to be wrong using "free". But what? Code looks not wrong. Using debugger the app exports in slow-motion and does not crash. LOL. As usual.

[masc]

Hello guys,

I played a little with version 1.5 and I love it!  What a valuable piece of software!  Congratulations to everyone involved.

Recently, I managed to borrow a 5D3 and shot a few test files to check the latest MLVApp.  Here is a file of a high-contrast scene shot in the 1x3 mode at Dual ISO 100/800. 

https://we.tl/t-z6hpWfLjDk

The noise suppression in the darkest areas is generally very good but these magenta fringing artifacts along the straight lines of the laundry stand and the painting on the right are killing me.   Could you guys please have a look?  Thanks.

Looks like moiree. Using IGV it becomes a little better. But near the painting on the right I don't see something?!