MLV App 1.14 - All in one MLV Video Post Processing App [Windows, Mac and Linux]

Started by ilia3101, July 08, 2017, 10:19:19 PM

Previous topic - Next topic

0 Members and 5 Guests are viewing this topic.

bouncyball

Quote from: masc on August 15, 2017, 08:00:19 PM
Ah okay. Did I handle that sparately anywhere?
Nope, but I always have to do it when using plain C and I hate this so much, hahaha :P

masc

LOL, okay. Got it now.  8)

I did something against all the debug outputs now. It is on github.

Another question: can you open files bigger than 2GB on Linux? On OSX it is no problem, on Windows some while(..) will never end... It think the comparisons between uint64_t and long are the problem (see compiler warnings), but I did not get it fixed until now... maybe I'll wait for Ilia to let him fix that. :)
5D3.113 | EOSM.202

12georgiadis

Quote from: masc on August 15, 2017, 06:39:23 PM
Superb! Thank you so much! I changed that and commited it into the repos.
I don't get that. What did you do? \" is just a " in QString. It is for getting spaces in the paths and filenames to work, for example.

But very cool, that you got it working on Linux! Is it possible to make a kind of package, or has it to be compiled always for using it on Linux? I'm not deep enough in Linux...
Hej, cool, fine that it works for you too now. Thanks. MlRawViewer is still better in performance, because it uses GPU - we don't do for now.

Can cdng hold processed picture data? I thought it is also a kind of RAW, that is why I did not think about exporting cdng yet (don't know if Ilia3103 did).

What do you mean with "fixing issues settings"?

We just started developping this little app, we'll see where our way goes.  ;D


Ok, didn't know that it was cpu based.
Cdng may Be a good export option because we often Color correct after the editing but mlv are still not covered by NLEs. ProRes is ok with cinelog and xq option (not ffmpeg 10bits only export). With that option, we'd have the choice to correct raw Before and after the editing.
Fixing issues settings : bad pixels, v. Stripes... etc (all options available in mlvfs/mlv dump/ mlv producer)



Envoyé de mon iPhone en utilisant Tapatalk

ilia3101

I'm still going to be gone for a week ish(no code), but now I've got more of an internet connection.

@masc Sessions looks perfect, does it save it in a format yet? I'd like to be able to read that on the cocoa GUI in the future too (that needs lots of catching up).
See if you can fix any of the bugs easily, if you can, why wait for me :D
And yes, CDNG is a raw format (holds only Bayer data and almost identical to DNG), but it's ok to do some processing like removing vertical stripes or fixed pattern noise. I have thought about it, I like the idea of making a separate DNGing 'library', but I really have no idea how the dng format works.
Also audio_mlv.c might get done soon ;)

@bouncyball oh, I thought I had already put is_active to use... I'm probably going mad then. Maybe that's why I always had annoying problems trying to do certain things before.

Lots of good progress going on here.

And proper log output curves are still on the to-do list  :-\

bouncyball

Quote from: masc on August 15, 2017, 08:35:22 PM
Another question: can you open files bigger than 2GB on Linux? On OSX it is no problem, on Windows some while(..) will never end... It think the comparisons between uint64_t and long are the problem (see compiler warnings), but I did not get it fixed until now... maybe I'll wait for Ilia to let him fix that. :)
Yes I noticed those warnings too.

As I see windows build is 32 bit thas is why "-D_FILE_OFFSET_BITS=64" switch needed to pass to the compiler to seek more than 2gb files.Try:

win32: QMAKE_CFLAGS_RELEASE += -msse2 -std=c99 -D_FILE_OFFSET_BITS=64

or compile x64 binary with mingw.

I don't experience this issue because Linux binary is 64bit.

regards
bb

bouncyball

Quote from: Ilia3101 on August 15, 2017, 09:39:15 PM
And yes, CDNG is a raw format (holds only Bayer data and almost identical to DNG), but it's ok to do some processing like removing vertical stripes or fixed pattern noise. I have thought about it, I like the idea of making a separate DNGing 'library', but I really have no idea how the dng format works.
Also audio_mlv.c might get done soon ;)
Here all latest stuff for ML raw data handling/processing ever written/modified by ML developers or me :) It's all structured in my version of mlv_dump on steroids:
1. all latest cDNG handling stuff - cDNG Library and more.
2. latest Raw Processing Stuff modified and restructured. Includes stripes correction, bad/focus pixel fix, pattern noise fix and chroma smooth.

Quote from: Ilia3101 on August 15, 2017, 09:39:15 PM
@bouncyball oh, I thought I had already put is_active to use... I'm probably going mad then. Maybe that's why I always had annoying problems trying to do certain things before.
Maybe :D

regards
bb

masc

Quote from: Ilia3101 on August 15, 2017, 09:39:15 PM
@masc Sessions looks perfect, does it save it in a format yet? I'd like to be able to read that on the cocoa GUI in the future too (that needs lots of catching up).

Thx. Yes it saves in a XML, I called it MASXML (MLV App Session XML). It holds all files and parameter settings of the session. Atm I have only one file opened with your lib, so caching makes no sense. All cached frames get deleted when changing the clip. So I disabled it for now.

Quote from: Ilia3101 on August 15, 2017, 09:39:15 PM
See if you can fix any of the bugs easily, if you can, why wait for me :D

I just wait because I did not got it resolved yet ;) ... the 2 whiles work with a small change, but I run into other problems then.

Quote from: Ilia3101 on August 15, 2017, 09:39:15 PM
And yes, CDNG is a raw format (holds only Bayer data and almost identical to DNG)

So if we only want MLV -> cDNG, why not just using mlv_dump? It does exaclty that. This means just not needed complexity in this app, because it is a processing app and exporting to cDNG means exporting unprocessed frames... maybe we could just add a menu item which "remote controlles" mlv_dump. Or did I missunderstood?

Quote from: bouncyball on August 16, 2017, 09:02:09 AM
As I see windows build is 32 bit thas is why "-D_FILE_OFFSET_BITS=64" switch needed to pass to the compiler to seek more than 2gb files.Try:

win32: QMAKE_CFLAGS_RELEASE += -msse2 -std=c99 -D_FILE_OFFSET_BITS=64

or compile x64 binary with mingw.

Thanks again! Unfortunately that does not do the job :( I have still the problem with the never-ending while loops. When looking for Qt on Windows 64bit on the Qt download page, there is only a version with VC compiler (or we have to build it ourself). But when trying before I did not get it to work with MSVC. :( Maybe I should spend more time on that again.

The processing stuff of mlv_dump looks really interesting! Bad pixel fix, pattern noise and so on... would be great to have it in MLV App!
5D3.113 | EOSM.202

Danne

QuoteSo if we only want MLV -> cDNG, why not just using mlv_dump?
+1

bouncyball

Quote from: masc on August 16, 2017, 12:13:18 PM
maybe we could just add a menu item which "remote controlles" mlv_dump. Or did I missunderstood?
Yup, would be right decision, the whole point of this app is to skip creation of intermediate files. This also concerns DNGs. When you implement ffmpeg vial lib, MLV App is gonna be truly end to end processing tool.

Quote from: masc on August 16, 2017, 12:13:18 PM
The processing stuff of mlv_dump looks really interesting! Bad pixel fix, pattern noise and so on... would be great to have it in MLV App!
If you'll have any questions regarding processing I'm ready to answer with pleasure :)

bb

12georgiadis

What is still missing in osx MLV apps in general is a GUI that covers the preview function and the corrections before transcoding/rewrapping. If we can avoid MLV => CDNG and do all corrections/check in MLV, it would be perfect. I'm currently shooting a documentary and I'm using this workflow :

0) shoot lossless MLV + H264 proxys with sound
1) offload cards and checksum with rsync
2) visual check with Mlrawviewer/MLV app
3) open MLVFS to generate virtual CDNGs => fix settings (v. stripes etc.) (sometimes also using Swtich from Danne and MLV_Dump)
4) Resolve 14 => cinelog or Alexa Lut => sync sound from H264 proxys (find average offset) => export 444 12 bits XQ
5) Fcpx => import 444 XQ => generate proxys + lut r709
6) editing on any NLE
7) online edit, back on Resolve 14, color correct
8) Export APR 444 (DSM)

ideal workflow would be :

0) shoot lossless
1) offload, visual check, apply LUT, color correct, fix settings, sync sound, export APR 444 12 XQ + APR Proxys in the same time (or CNDG + DPX/TIFF if necessary) ALL this steps in one app and one batch ;-)


bouncyball

@Ilia3101, @masc

I forgot one thing to ask

In amaze_demosaic.c there are 3 inline funcs:

__inline float xmul2f(float d)
__inline float xdiv2f(float d)
__inline float xdivf( float d, int n)

gcc (under linux) or mingw-gcc (under windows) always complains and outputs error that they are not defined.

changing "__inline" to "static INLINE" solves the issue.
How do you compile w/o applying these changes?

masc

Quote from: bouncyball on August 16, 2017, 03:14:19 PM
If you'll have any questions regarding processing I'm ready to answer with pleasure :)

:) I think Ilia or me, we will do that! Cool!

Quote from: bouncyball on August 16, 2017, 07:23:12 PM
gcc (under linux) or mingw-gcc (under windows) always complains and outputs error that they are not defined.
changing "__inline" to "static INLINE" solves the issue.
How do you compile w/o applying these changes?

Okaay, I had always one big problem under windows, maybe this could solve it: I don't get everything compiled with
win32: QMAKE_CFLAGS_RELEASE += -msse2 -std=c99
so I did a compiler run with this before
win32: QMAKE_CFLAGS_RELEASE += -msse2
some module(s?) compile, others with error, I switch the flag again and the other modules compile and link in the end. Not the best to do so... I know... but I found no other solution so far. Maybe your hint solves this issue. I'll try tomorrow! Do you think that's it?

Quote from: 12georgiadis on August 16, 2017, 04:19:10 PM
1) offload, visual check, apply LUT, color correct, fix settings, sync sound, export APR 444 12 XQ + APR Proxys in the same time (or CNDG + DPX/TIFF if necessary) ALL this steps in one app and one batch ;-)

I think this is not so far away from what I think MLV App could do in the future ;)
5D3.113 | EOSM.202

12georgiadis

Quote from: masc on August 16, 2017, 10:06:00 PM
:) I think Ilia or me, we will do that! Cool!

Okaay, I had always one big problem under windows, maybe this could solve it: I don't get everything compiled with
win32: QMAKE_CFLAGS_RELEASE += -msse2 -std=c99
so I did a compiler run with this before
win32: QMAKE_CFLAGS_RELEASE += -msse2
some module(s?) compile, others with error, I switch the flag again and the other modules compile and link in the end. Not the best to do so... I know... but I found no other solution so far. Maybe your hint solves this issue. I'll try tomorrow! Do you think that's it?

I think this is not so far away from what I think MLV App could do in the future ;)

Great ! I'll Be here to test and give feedback ;-)


Envoyé de mon iPhone en utilisant Tapatalk

masc

Quote from: bouncyball on August 16, 2017, 07:23:12 PM
@Ilia3101, @masc

I forgot one thing to ask

In amaze_demosaic.c there are 3 inline funcs:

__inline float xmul2f(float d)
__inline float xdiv2f(float d)
__inline float xdivf( float d, int n)

gcc (under linux) or mingw-gcc (under windows) always complains and outputs error that they are not defined.

changing "__inline" to "static INLINE" solves the issue.
How do you compile w/o applying these changes?

I changed that and tried again on windows. That does not change anything for me. Other source files need the -std=c99 flag, if I activate that the compiler brings 172 errors for these few lines in amaze_demosaic.c:

#define min(a,b) \
   ({ typeof ((a)+(b)) _a = (a); \
      typeof ((a)+(b)) _b = (b); \
     _a < _b ? _a : _b; })

#define MAX(a,b) \
   ({ typeof ((a)+(b)) _a = (a); \
       typeof ((a)+(b)) _b = (b); \
     _a > _b ? _a : _b; })

#define SQR(a) \
   ({ typeof (a) _a = (a); \
     _a * _a; })

Do you have an idea why? The output is to long for a message in this forum :(
Mostly it is something like "expected ';' before '_a'" or "....before '_b'".
5D3.113 | EOSM.202

ilia3101

@masc..

1. Yes, it is a good idea to just use mlv_dump for dng export, to avoid unnecessary complexity.

2. The AMaZE code is a disgusting mess to be honest, I don't even know what language you could call it, but if I remember correctly it compiled with some c++ standards flags.

3. Also could you get me a nice screenshot of the Qt GUI that you like so I can put it in the first post?

4. About caching: I think it should work like this... When there is many clips open, the one being worked on should be allowed full cache (whatever we did when only one clip was supported), and others would be allowed 0 cache.. Of course this would need for me to rewrite the caching to actually work so it can be adjusted in size while running.

@bouncyball...

Could you possibly by any chance provide a Linux build for @masc to put on the release page?

bouncyball

@masc

Which QT and mingw versions do you use?
I use QT 5.9.1 with mingw 5.3.0 and able to compile on one pass :)
Btw under windows as well. However can not fix the MLV size >2GB issue yet.

12georgiadis

Ok, I think I found out where I had issues. When I try to open it my files from a complex folder structure (like I often use for editing) it crashes. But when I moved them to a simple one, like HDD root, it opens perfectly.

bouncyball

@masc

try:
win32: QMAKE_CFLAGS_DEBUG += -msse2 -D_FILE_OFFSET_BITS=64
In conjunction with replacing all "fseek" with "fseeko". Should resolve >2gb isuue.

I also compiled MLV App with mingw64-x64 (using MSYS2 enviroment, very cool stuff), got 64bit native binary for windows but 2gb issue persisted. Then I changed all fseek calls with fseeko and 2.8GB MLV loaded and processed :) I'm too lazy to compile libpng and zlib again for win32 MLV App testing. Try yourself what I said abobe, I think it should help even for 32bit binary.

Meanwhile trying to compile all static MLV App... not succeded yet...

bb

masc

Quote from: Ilia3101 on August 17, 2017, 11:24:15 AM
@masc..

1. Yes, it is a good idea to just use mlv_dump for dng export, to avoid unnecessary complexity.

2. The AMaZE code is a disgusting mess to be honest, I don't even know what language you could call it, but if I remember correctly it compiled with some c++ standards flags.

3. Also could you get me a nice screenshot of the Qt GUI that you like so I can put it in the first post?

4. About caching: I think it should work like this... When there is many clips open, the one being worked on should be allowed full cache (whatever we did when only one clip was supported), and others would be allowed 0 cache.. Of course this would need for me to rewrite the caching to actually work so it can be adjusted in size while running.

@bouncyball...

Could you possibly by any chance provide a Linux build for @masc to put on the release page?

3.: http://image.ibb.co/h7tvma/Bildschirmfoto_2017_08_12_um_12_21_47.png this was the last one...
4.: yes, this should easily be possible. I did it also like this (you provide this functionality), but I had some problems - some crashes and some pinkish frames. So I disabled it... with some debugging this should work!
5.: I was talking with bouncyball about that... some posts above. But to build a linux package that needs some extra work ;)

Quote from: bouncyball on August 17, 2017, 12:21:14 PM
@masc
Which QT and mingw versions do you use?
I use QT 5.9.1 with mingw 5.3.0 and able to compile on one pass :)
Btw under windows as well. However can not fix the MLV size >2GB issue yet.

Good question. I think it is a Qt 5.6.0 and MinGW 4.9.x or something like that on Windows. Maybe I should just update MinGW. But with this 2GB issue, the best option will be to try it again with MSVC I think.

Quote from: 12georgiadis on August 17, 2017, 07:57:18 PM
When I try to open it my files from a complex folder structure (like I often use for editing) it crashes. But when I moved them to a simple one, like HDD root, it opens perfectly.

Hm, sounds strange. At least my part can handle each path I think. Maybe I look for that deeper in the sources...
Quote from: bouncyball on August 17, 2017, 08:14:41 PM
try:
win32: QMAKE_CFLAGS_DEBUG += -msse2 -D_FILE_OFFSET_BITS=64
In conjunction with replacing all "fseek" with "fseeko". Should resolve >2gb isuue.
...
Try yourself what I said abobe, I think it should help even for 32bit binary.
Wow, sounds good... I'll try that tomorrow when I'm back on my windows PC! Thanks!
5D3.113 | EOSM.202

bouncyball

HERE is a proof of concept ;) Win x64 MLV App with > 2GB support. Just exported mov for 4.2gb MLV.

bouncyball

Quote from: Ilia3101 on August 17, 2017, 11:24:15 AM
Could you possibly by any chance provide a Linux build for @masc to put on the release page?
Can not compile all static binary yet. There is no chance to put libs side by side with binary in Linux. It's not gonna work.

It's how it looks like on my desktop :)



masc

Quote from: bouncyball on August 17, 2017, 08:14:41 PM
I'm too lazy to compile libpng and zlib again for win32 MLV App testing. Try yourself what I said abobe, I think it should help even for 32bit binary.

The win32 versions of libpng and zlib are in the repos. You mustn't compile again...

Quote from: bouncyball on August 17, 2017, 08:52:14 PM
HERE is a proof of concept ;) Win x64 MLV App with > 2GB support. Just exported mov for 4.2gb MLV.

Yeaaayyy... will try it tomorrow!  8) And: so cool to see it on Linux! Hahahaa
5D3.113 | EOSM.202

Danne


mario1000

Quote from: bouncyball on August 17, 2017, 08:52:14 PM
HERE is a proof of concept ;) Win x64 MLV App with > 2GB support. Just exported mov for 4.2gb MLV.

Dear bouncyball,

I would like to try this version, but after start an error message appears: "This application fails to start because it could not find or load the Qt platform plugin "windows" in "". Reinstallating the application may fix this problem."
I use Windows 8.1 . Any suggestions?

Thanks in advance!

masc

@bouncyball:
Same for me. Should be a problem with the libraries. You can try opening on your system after renaming your Qt installation folder...

I tried what you wrote for >2GB file support with Win32... but the problem still persists :(
5D3.113 | EOSM.202