Magic Lantern Forum

Using Magic Lantern => Raw Video => Topic started by: bronson on May 21, 2013, 09:56:00 AM

Title: Raw2dng development ideas
Post by: bronson on May 21, 2013, 09:56:00 AM
Hi, things seems to be moving quickly... :D
I'm not a coder, but i'm getting the idea that raw2dng could be also a management tool.

With a gui, it could take an entire raw folder and create dng's subfolders with raw's names.
Perhaps, also creating a folder with a low res mp4 proxyfile that it could be useful before developping dngs.

The first step with the gui could be easy... but could not be the last...

Thanx to all for your amazing work..
Title: Re: It would be great with raw2dng...
Post by: zachnfine on May 21, 2013, 10:53:23 AM
As long as we're offering up feature suggestions:

Frame numbers based on time-of-day timecode, generated using the file creation date as a starting point if no metadata is available.

Each dng sequence written to its own folder.

Addition or alteration of the dng files such that they'll work in davinci resolve. That software works with dng sequences ok, but not ours.

I plan to have a look at the source code and see if I can't contribute, but am not the best coder and might as well list those ideas here first in case they spark the interest of a developer who's already up to speed on the project.
Title: Re: Raw2dng development ideas
Post by: bronson on May 21, 2013, 11:37:59 AM
I agree with zachnfine.

Keeping a movie info in txt with all good exif values could be also useful.. (resolution, focal, shutter speed, iso, date, lens info...)

Any other improvements are welcome.
Title: Re: Raw2dng development ideas
Post by: powerpc on May 21, 2013, 12:37:17 PM
I have been thinking about this for a couple days because I took more than 200 raw files last Sunday on ITTF.  I made a small code to deploy all the raw files to dng and put them in separate folders. The names begin with the name of each raw file name. Just put the raw files, raw2dng.exe and my small program in a same folder and click on it, it will do the work.
The prog is on my ftp server. please feel free to try


 ftp://a:[email protected]/Volume_2/raw2dngb.rar (ftp://a:[email protected]/Volume_2/raw2dngb.rar)


;)
Title: Re: Raw2dng development ideas
Post by: X-RAY on May 21, 2013, 12:49:52 PM
@ powerpc
Really nice! Helps a lot. That was exactly what I had to do each time. Put the raw-file in a folder and drop the raw to raw2dng to get the dngs in the folder. Now its so much simpler. Also nice to have the info.txt. But I think it would also be helpful to have the resolution and fps in the folder-name? like "M0000001_1920_720_29667" for 1920x720 and 29.667fps
Title: Re: Raw2dng development ideas
Post by: powerpc on May 21, 2013, 12:58:21 PM
@X-RAY

Ha, you are always on line
I am happy it helps and thank you for the idea. I will work on it tonight.
Title: Re: Raw2dng development ideas
Post by: bronson on May 21, 2013, 02:11:27 PM
GREAT powerpc !!!

Thank you for the share..
Title: Re: Raw2dng development ideas
Post by: LucaBrasi on May 21, 2013, 02:35:07 PM
Great Stuff Powerpc!!! thanks! worked a treat!
Title: Re: Raw2dng development ideas
Post by: X-RAY on May 21, 2013, 02:37:35 PM
;-)
just addicted to the latest developments. but also simply in another timezone (UTC +1).
Title: Re: Raw2dng development ideas
Post by: powerpc on May 22, 2013, 02:27:34 AM
 
Hi, I have made the change on the folder names and updated, the same link and filename

ftp://a:[email protected]/Volume_2/raw2dngb.rar (ftp://a:[email protected]/Volume_2/raw2dngb.rar)


  :P

x, I am in Paris, the same zone
Title: Re: Raw2dng development ideas
Post by: 1% on May 22, 2013, 03:06:32 AM
Does it work with 4gb changes? Also only makes folders if I use it as an "open with" in windows.
Title: Re: Raw2dng development ideas
Post by: hirethestache on May 22, 2013, 05:15:40 AM
Is the idea on the horizons of developing a proprietary native premiere/aftereffects plugin that would be able to playback the recorded files without any encoding?
Title: Re: Raw2dng development ideas
Post by: tin2tin on May 22, 2013, 07:20:11 AM
Would it be possible to add an option to add a destination folder? Like -o "c://my-path/"?
Title: Re: Raw2dng development ideas
Post by: mjneubrander on May 22, 2013, 06:24:12 PM
I would like an option to export say the (first, middle and last frame) or (every ##th frame) so I can make a preview before extracting all the frames. Im working on a little something with UFRaw and batch processing. Before starting UFRaw from the management tool I would like to build a preview so I can decide on which clips to use. I don't want to process every frame in a clip just to throw away B-reel. I have a SSD so processing a ~3 min clip takes up the whole thing :( Space is precious.
Title: Re: Raw2dng development ideas
Post by: bronson on May 22, 2013, 06:52:07 PM
Powerpc did a great job with his update, but it would be very cool if someone could write a gui for raw2dng.
It could always work also with the lastest build.
It could batch rename, increment, create folders, and my dream, make an mp4 preview as a proxy.

8)
Title: Re: Raw2dng development ideas
Post by: Stedda on May 22, 2013, 07:16:17 PM
Quote from: bronson on May 22, 2013, 06:52:07 PM

It could batch rename, increment, create folders, and my dream, make an mp4 preview as a proxy.

8)

RAWanizer
Title: Re: Raw2dng development ideas
Post by: IliasG on May 22, 2013, 07:34:00 PM
I find useful to

- run all DNGs through exiftool to change exif tags like model name, WB as shot,  Black and White Levels, dates, .. go on.

- run all DNGs through Adobe DNG Converter and resave them as losslessly compressed. Size goes from 3.6MB to 2.1-2.2MB on average when it includes a medium jpeg preview (1024X576) and a bit less with the "minimal" preview.
This "medium" preview can be a substitute of the requested mp4 preview when extracted as jpeg sequence ..

If one choose to compress to Lossy DNG, size goes to less than 1MB but the result is a 8bit-log dng file .. someone has to check if it is robust enough for heavy gradation ..

If the above could be automated in this "rawtoDNGb" utility it would be the nice ..

A related GUI utility is Olli's TurboDNGimporter .. http://www.visualbakery.com/Tools/DNGImporter.aspx
Title: Re: Raw2dng development ideas
Post by: bronson on May 22, 2013, 08:50:39 PM
@Powerpc
Sorry, I just used your tool and  i didn't see that it was raw2dng(b)... I thought that you had rewritten the main raw2dng code.
Sorry for this, it works like a charm..

Merci à toi...
PS même time zone que toi, mais plus dans le sud... :)

@Stedda
thanx for this info, but what's RAWanizer, I googled it, but it seems that i did not find any informations about this..
Title: Re: Raw2dng development ideas
Post by: Peter on May 22, 2013, 09:37:06 PM
I did a simple batch script which works per drag&drop and puts all DNGs of the dragged RAW file in a subfolder where the RAW file is located. This doesn't delete or move any files.

* Create a folder somewhere, put raw2dng.exe and mkdng.cmd (this script) there
* Create a shortcut of mkdng.cmd and put it on the Desktop
* Drag the RAW file to the shortcut

@echo off
IF /I "%~x1"==".RAW" goto OK
echo No Raw File!
goto END
:OK
IF NOT EXIST %~d1%~p1%~n1 MD %~d1%~p1%~n1
%~d1
cd %~p1%~n1
%~d0%~p0raw2dng.exe %1
:END
pause


For those who don't need more at the moment ;-)
Title: Re: Raw2dng development ideas
Post by: powerpc on May 23, 2013, 01:18:14 AM
Updated, now it allows inputting destination path, and choosing to put separate folders or not.
Title: Re: Raw2dng development ideas
Post by: Naturalbornsamy on May 24, 2013, 12:28:50 AM
Hi everyone,
do you know if I can find a similar script than Power PC one for mac platform?

Thx in advance; )
Title: Re: Raw2dng development ideas
Post by: tin2tin on May 24, 2013, 12:45:12 AM
Quote from: powerpc on May 23, 2013, 01:18:14 AM
Updated, now it allows inputting destination path, and choosing to put separate folders or not.
Thank you.  :D Is it still the same link?
Title: Re: Raw2dng development ideas
Post by: scrax on May 24, 2013, 01:12:14 AM
Quote from: Naturalbornsamy on May 24, 2013, 12:28:50 AM
Hi everyone,
do you know if I can find a similar script than Power PC one for mac platform?

Thx in advance; )

Here raw2dng.app (http://www.magiclantern.fm/forum/index.php?topic=5508.0)
Title: Re: Raw2dng development ideas
Post by: powerpc on May 24, 2013, 01:17:39 AM
to tin2tin,

Yes, is the same link :P
Title: Re: Raw2dng development ideas
Post by: RenatoPhoto on May 24, 2013, 01:56:34 AM
Quote from: powerpc on May 22, 2013, 02:27:34 AM
 
Hi, I have made the change on the folder names and updated, the same link and filename

ftp://a:[email protected]/Volume_2/raw2dngb.rar (ftp://a:[email protected]/Volume_2/raw2dngb.rar)

Thanks it works but I have the following problem:

Created directory g:/1
put all files (xxx.raw, raw2dng.exe, raw2dngb.exe)  in directory g:/1
clicked on raw2dngb.exe

Created new directory xxx and put DNG files in it.

But.. it deleted raw2dng.exe file...
Title: Re: Raw2dng development ideas
Post by: tin2tin on May 24, 2013, 07:36:10 AM
Quote from: powerpc on May 24, 2013, 01:17:39 AM
to tin2tin,

Yes, is the same link :P
Comodo throws a malware alert. This sometimes happens if ex. upx is used. Is it possible to pack it less compressed or something?
Title: Re: Raw2dng development ideas
Post by: Grunf on May 24, 2013, 11:04:55 AM
My suggestion:

-. Possibility of using dark-current substraction (take one raw pic with lens cover on, save and let raw2dng substract it from raw files. This will average out FPN and eventual pixel errors (as this is not being done by camera any more when using raw).

Title: Re: Raw2dng development ideas
Post by: Naturalbornsamy on May 24, 2013, 12:01:35 PM
Quote from: scrax on May 24, 2013, 01:12:14 AM
Here raw2dng.app (http://www.magiclantern.fm/forum/index.php?topic=5508.0)
Thanks, but I mean the raw2dngb script. (it's for a batch extract of several .raw files right?)
Title: Re: Raw2dng development ideas
Post by: scrax on May 24, 2013, 12:47:32 PM
Quote from: Naturalbornsamy on May 24, 2013, 12:01:35 PM
Thanks, but I mean the raw2dngb script. (it's for a batch extract of several .raw files right?)
For what I understood raw2dngb.exe is doing the same for win that is doing raw2dng.app, only difference is that  raw2dng.app has a GUI and you don't see command line terminal. But if you want you can find the script inside the package.

Also raw2dng.app will not let you choose a destination directory, it will use the same dir of the raw file/folder to save the dng.
Title: Re: Raw2dng development ideas
Post by: fatpig on May 24, 2013, 03:29:06 PM
Hello guys, I did something very similar. Give it a try. :)

RAW2DNG BATCHelor 1.1 Beta

(http://www.fatpigtures.com/software/images/batchelor-screenshot-1.1.gif)

http://www.fatpigtures.com/software (http://www.fatpigtures.com/software)

It is a GUI with renaming, and thumbnail generation option.
It puts all your DNGs in separate folders with a folder.jpg and
also renames the DNGs and each folder.
You can also drag n drop your folder into the field on top.

I would be happy if you like it.
Any Questions will be answered here.

Making an MP4 preview will not be very problematic. If there is real need for it, I can try to implement that aswell.

I created a new thread for it and all future versions: ]http://www.magiclantern.fm/forum/index.php?topic=5645.0] (http://www.magiclantern.fm/forum/index.php?topic=5645.0)
Title: Re: Raw2dng development ideas
Post by: RenatoPhoto on May 24, 2013, 04:10:21 PM
@ fatpig

I tested it and works well..  Since I do not care for thumbnails I did not install imagemagick and still works.  In future dev may want to enable a selection box for thumbnails.  A proxy may be nice for quick review of video.

Thanks for your support of ML
Title: Re: Raw2dng development ideas
Post by: fatpig on May 24, 2013, 04:52:58 PM
happy to hear that.
thanks for your suggestion,
I think I will do that in the next release.

I created a new thread for it and all future versions: http://www.magiclantern.fm/forum/index.php?topic=5645.0 (http://www.magiclantern.fm/forum/index.php?topic=5645.0)
Title: Re: Raw2dng development ideas
Post by: powerpc on May 24, 2013, 05:01:26 PM
to RenatoPhoto, Could you please try g:\1 "not g:/1"

I believe it has some bugs, I will check it out
Title: Re: Raw2dng development ideas
Post by: bronson on May 24, 2013, 09:27:56 PM
 :) :) :) :) :)
fatpig, you make may day !!!

The h264 will be useful, we could edit with proxies and replace them by dng's sequence.

Title: Re: Raw2dng development ideas
Post by: marten on May 24, 2013, 09:34:44 PM
Have a look at this and see if it's useful :)

http://www.magiclantern.fm/forum/index.php?topic=5557.0 (http://www.magiclantern.fm/forum/index.php?topic=5557.0)
Title: Re: Raw2dng development ideas
Post by: fatpig on May 25, 2013, 12:58:23 AM
@marten,
seems your RAWanizer pretty much does everything the BATCHelor does, and already has the proxy available.
I will test it out- sounds sweet!

Are there any differences in the workflow?
what do you use to extract the tiffs from the dngs?
Title: Re: Raw2dng development ideas
Post by: tin2tin on June 01, 2013, 08:28:56 AM
Quote from: powerpc on May 21, 2013, 12:37:17 PM
The prog is on my ftp server. please feel free to try
 ftp://a:[email protected]/Volume_2/raw2dngb.rar (ftp://a:[email protected]/Volume_2/raw2dngb.rar)
I'm still getting a Comodo Malware alert. Would it be possible to add the destination switch code to the official raw2dng code? So the official raw2dng will be fully fledged?  :D
Title: Re: Raw2dng development ideas
Post by: guentergunter on June 17, 2013, 08:45:41 AM
Quote from: Naturalbornsamy on May 24, 2013, 12:28:50 AM
Hi everyone,
do you know if I can find a similar script than Power PC one for mac platform?

Thx in advance; )

Also, there is RAWMagic (http://www.magiclantern.fm/forum/index.php?topic=6218.0). It has a nice GUI and converts spanned files on the fly to CinemaDNG with a choosable output folder 8)