Author Topic: raw2mlv processing  (Read 2547 times)

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 5935
raw2mlv processing
« on: November 10, 2019, 03:45:44 PM »
Ilia3101 is building a raw2mlv conversion tool. Very cool. Now we are actually able to wrap CR2 files and dng files back to our beloved MLV format. Well, more raw formats are possible but for now I focused on CR2 and dng files.

This automator app has only one purpose. To utilize Ilia3101 raw2mlv conversion tool, code here:
https://github.com/ilia3101/LibMLV

Forum post:
https://www.magiclantern.fm/forum/index.php?topic=24386.0
https://github.com/ilia3101/LibMLV


raw2mlv.app(Mac only!)
https://bitbucket.org/Dannephoto/raw2mlv/src/master/
Instructions and source code

Download:
https://bitbucket.org/Dannephoto/raw2mlv/downloads/raw2mlv.dmg

regarding gatekeeper
To supress gatekeeper hold ctrl button down(macOS Sierrra, Mojave, Catalina) while opening the application the first time. You can also change permissions from within privacy/security settings.



Good for? You tell me but I use it as follows:
1 - Transcode dualiso CR2 files into MLV files accessible in Mlv App
2 - Wrap timelapse CR2 file sequences into one chained MLV file accessible in Mlv App
3 - Wrap dng files back into MLV format

TODO:
Adding more options
Adding more raw format options
Add folder support. (Drag n´ drop only atm)


Ilia3101

  • Contributor
  • Hero Member
  • *****
  • Posts: 689
Re: raw2mlv processing
« Reply #1 on: November 10, 2019, 07:23:41 PM »
Yay!
5D2, MLV App

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3325
  • 5D3 | M1 | 7D | 70D | SL1 | M2 | 50D
Re: raw2mlv processing
« Reply #2 on: November 11, 2019, 03:27:11 AM »
Oh yeah baby!!! I friggin' love the second option (02) for getting all CR2's into a single MLV file.

This is nuts especially for time-lapse stuff!  8)
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

IDA_ML

  • Hero Member
  • *****
  • Posts: 654
Re: raw2mlv processing
« Reply #3 on: November 11, 2019, 10:23:51 AM »
If an option could be added to MLVApp to open single CR2 files, including DualISO ones, I think, I would say Goodbye to ACR for ever.  I like the results from MLVApp much better and for me such an option would be revolutionary! 

Genious work, Ilia3101 and Danne!

Kharak

  • Hero Member
  • *****
  • Posts: 903
Re: raw2mlv processing
« Reply #4 on: November 11, 2019, 04:00:49 PM »
Waiting anxiously for a Windows version :)
once you go raw you never go back

cmh

  • Contributor
  • Freshman
  • *****
  • Posts: 61
Re: raw2mlv processing
« Reply #5 on: November 11, 2019, 04:36:01 PM »
raw2mlv.exe (Windows x64)

Usage
There's two ways to create an mlv from your raw files:
- drop the files on raw2mlv.exe and you'll find an output.mlv in the folder where your files are located (this method might be limited to a maximum of 433 files).
- copy the files in a directory containing raw2mlv, open a powershell windows, navigate to you folder (or open one from the current folder using shift + right clic) and use the command line.

 
Example:
.\raw2mlv.exe "E:\DCIM\100CANON\IMG_168*.CR2" -o "C:\Users\SomeDude\Desktop\test.mlv"
This command will open all CR2 files starting by IMG_168 from the folder E:\DCIM\100CANON\ and create an mlv file on SomeDude's desktop.
Open this mlv in MLVApp and each frames will be made of the CR2s to be manipulated and saved.

Here's a quick demo.

Download
https://github.com/cmhamiche/raw2mlv/raw/master/raw2mlv.zip

Note
This is not a port of Danne's app but a placeholder for ilia3101's Windows build.

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 5935
Re: raw2mlv processing
« Reply #6 on: November 11, 2019, 04:50:13 PM »
Cool. Feel free to maintain a windows version in above post.

Ilia3101

  • Contributor
  • Hero Member
  • *****
  • Posts: 689
Re: raw2mlv processing
« Reply #7 on: November 11, 2019, 06:58:16 PM »
@cmh Thank yuou wow!!! I knew it would compile on Windows.

Really wishing I had Windows (rare occasion) right now, I wanna test this.

I hope to create a makefile that willl work everywhere.
5D2, MLV App

Walter Schulz

  • Contributor
  • Hero Member
  • *****
  • Posts: 6876
Re: raw2mlv processing
« Reply #8 on: November 11, 2019, 07:14:05 PM »
You can. Install an Windows 10 evaluation version inside a virtual machine.
https://www.microsoft.com/en-us/evalcenter/evaluate-windows-10-enterprise
Photogs and videographers: Assist in proof reading upcoming in-camera help!. Your input is wanted and needed!

Ilia3101

  • Contributor
  • Hero Member
  • *****
  • Posts: 689
Re: raw2mlv processing
« Reply #9 on: November 11, 2019, 07:26:55 PM »
I will use that thankjs.
5D2, MLV App

cmh

  • Contributor
  • Freshman
  • *****
  • Posts: 61
Re: raw2mlv processing
« Reply #10 on: November 11, 2019, 07:29:22 PM »
If I'm being honest I'm not a dev and I barely understand what I'm doing. I'm just familiar with some unix systems.
I've yet to understand how to link dynamic library to get rid of some more dlls.
Adapting compile.sh to add Windows should be a trivial task if you check the build instructions I posted.

Ilia3101

  • Contributor
  • Hero Member
  • *****
  • Posts: 689
Re: raw2mlv processing
« Reply #11 on: November 11, 2019, 07:34:48 PM »
The instructions are very nice and useful. Thank you.

I agree it would be nice to remove some of the DLLs,  I will see what I can do
5D2, MLV App

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3325
  • 5D3 | M1 | 7D | 70D | SL1 | M2 | 50D
Re: raw2mlv processing
« Reply #12 on: November 11, 2019, 09:13:47 PM »
If an option could be added to MLVApp to open single CR2 files, including DualISO ones, I think, I would say Goodbye to ACR for ever.

It's bound to happen. Bet Adobe is about to shit a brick, if not already and yet once this option arise it would indeed be revolutionary!  8)
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 5935
Re: raw2mlv processing
« Reply #13 on: November 11, 2019, 10:35:19 PM »
Really cool progress on your windows post @cmh. Thanks for that.

I updated with possibility to alter bitdepth, already included in raw2mlv. First post updated:

2blackbar

  • Member
  • ***
  • Posts: 234
Re: raw2mlv processing
« Reply #14 on: November 12, 2019, 12:57:33 AM »
LOVE WINDOWS VERSION ! Thank You guys a lot, i can now edit cr2 files in single mlv

garry23

  • Contributor
  • Hero Member
  • *****
  • Posts: 1815
Re: raw2mlv processing
« Reply #15 on: November 12, 2019, 07:09:59 AM »
To help those of us who are not up to speed on mlv and 'video stuff', I wonder if some kind person could capture the workflow to achieve the following use case to simulate an  LE image.

Capture a sequence of .cr2s, say, 20.

Bring these together into an mlv file using the new tool.

Process the mlv in some app to merge the individual files together.

A1ex suggested the above in a previous post, but truly I have no idea about the video tools that have been created for mlv.

Cheers

Garry

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 5935
Re: raw2mlv processing
« Reply #16 on: November 12, 2019, 07:17:50 AM »
Usage:
There's two ways to create an mlv from your raw files
- open a powershell from raw2mlv's folder (shift + right clic) and use the command line (raw2mlv.exe --help to get the arguments).
- drop your files on raw2mlv.exe and you'll find an output.mlv in the folder where your files are located.

Example:
.\raw2mlv.exe "E:\DCIM\100CANON\IMG_168*.CR2" -o "C:\Users\SomeDude\Desktop\test.mlv"
This command will open all CR2 files starting by IMG_168 from the folder E:\DCIM\100CANON\ and create an mlv file on SomeDude's desktop.
Open this mlv in MLVApp and each frames will be made of the CR2s to be manipulated and saved.

Above doesn't work for you @garry23?
Sidenote: funny thing I asked for explaining vids of your lua script once but not really happening right ;)?

cmh

  • Contributor
  • Freshman
  • *****
  • Posts: 61
Re: raw2mlv processing
« Reply #17 on: November 12, 2019, 10:28:17 AM »
Here's a quick demo on Windows.


Don't mind the quality (uploading it took longer than creating it) but it should do the job.
Keep in mind that the raw2mlv I made available is just Ilia3101's I compiled for Windows and not a port of Danne's fancier OSX tool.

Edit: even tho I haven't tried yet, there's ways to change the framerate either in raw2mlv or MLVApp.
That said, other than stop motion animations, I'd love to see a timelapse smoothed with Resolve's optical flow.

Kharak

  • Hero Member
  • *****
  • Posts: 903
Re: raw2mlv processing
« Reply #18 on: November 12, 2019, 01:12:53 PM »
When I have more than 430 dng's in a folder, the executable fails.
windows gives me this message: "The filename or extension is too long".

I have a bunch of folders with DNG's that I want converted to MLV, one folder I have has 430 dng's and it works, but another with 495 wont convert and I get that message. Folders with less than 430 convert fine and ones with more dont.
once you go raw you never go back

cmh

  • Contributor
  • Freshman
  • *****
  • Posts: 61
Re: raw2mlv processing
« Reply #19 on: November 12, 2019, 01:26:55 PM »
I can't help much on that.
Have you tried moving those files to a shorter path like C:\DNG.
If if doesn't help (and probably won't because I suspect that a long ass concatenated string is created) you could also try to remove the 260 character path limit by editing the registry.
I haven't try myself but if you feel adventurous and made backups, here's the procedure:
- run regedit.exe as administrator
- locate [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem]
- change data value [LongPathsEnabled] (DWORD) to "1"
- close regedit and restart Windows

Could you report if you make it work somehow?

Ilia3101

  • Contributor
  • Hero Member
  • *****
  • Posts: 689
Re: raw2mlv processing
« Reply #20 on: November 12, 2019, 01:56:57 PM »
I seem to remember that windows needs to use special versions of the file reading/writing functions for files over 4GB, do you think file size could be the problem?
5D2, MLV App

Lars Steenhoff

  • Senior
  • ****
  • Posts: 454
Re: raw2mlv processing
« Reply #21 on: November 12, 2019, 02:56:21 PM »
What about the lens information, can this also be transferred form the raw to the mlv?

cmh

  • Contributor
  • Freshman
  • *****
  • Posts: 61
Re: raw2mlv processing
« Reply #22 on: November 12, 2019, 03:04:51 PM »
I don't have the necessary knowledge to answer that question Ilia3101, I've never written any code and I've no idea how fwrite is implemented in Visual C++.
I'm talking out of my ass but while I see some fwrite in raw2mlv.c, I can't find any buffer meaning that while it's cached, I guess it's relying on the operating system flushing (or disk controller whatever). I think it's good practice to define a buffer and split things up.

That said you might need some more info from Kharak, maybe he's working on a fat32 card or something.
Meanwhile I'll create 200 DNG to test this out...

Edit: so I've created 210 CR2s, made a 6,5gb mlv that contains the 210 frames and it's working fine. I'll try with DNGs...
Working fine with DNG using a long path. I'll try 500 files...

Ilia3101

  • Contributor
  • Hero Member
  • *****
  • Posts: 689
Re: raw2mlv processing
« Reply #23 on: November 12, 2019, 03:30:42 PM »
@Lars steenhoff I will add lens info soon, just haven't done it yet. As well as quite a lot of things.

Don't worry cmh, we did this in MLV App before and I can make the change later, it's not big. Although as it seems to work for you maybe it is completely fine.

Maybe Karak is using FAT32.

Can you test if this runs on your PC: https://drive.google.com/open?id=1SzEU5xDILbE6xMxWMTBUQ3b-TWkWDjA4

I have tried to compile with microsoft msvc and it works for me. Only one DLL needed for this: libraw :P
5D2, MLV App

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 5935
Re: raw2mlv processing
« Reply #24 on: November 12, 2019, 03:33:38 PM »
I seem to remember that windows needs to use special versions of the file reading/writing functions for files over 4GB, do you think file size could be the problem?
On mac I think it could be around 5000-6000 files before terminal chokes. Easy workaround using find xarg command but let´s see if mac gets some broken workflow first before developing further.