The CinemaDNG Discussion (raw2cdng)

[duey101]

Here are my tests with v.1.49...

Convert RAW files from SD card:
16 bit linear, party mode(ON) Gamma 1.0 = Result - no issues.
12 bit logarithmic, party mode(ON) Gamma 1.0 = Result - Strong green cast.
12 bit linear, party mode(ON) Gamma 1.0 = Result - no issues.
10 bit logarithmic, party mode(ON) Gamma 1.0 = Result - no issues.

16 bit linear, party mode(OFF) Gamma 1.0 = Result - no issues.  Faster conversion speeds!
12 bit logarithmic, party mode(OFF) Gamma 1.0 = Result - Strong green cast.  Faster conversion speeds!
12 bit linear, party mode(OFF) Gamma 1.0 = Result - no issues.  Very fast conversion!
10 bit logarithmic, party mode(OFF) Gamma 1.0 = Result - Slight pink cast and lots of added noise.

Question:  Does v1.49 work with .mlv files?  I tried dropping a couple on the GUI but nothing happened.  I have mlv_dump.exe in the raw2cdng folder as well.  The conversion speeds are very fast on my 1280x720 15 second RAW/MLV clips.  This is definitely a time saver.

One other thing, I noticed a few cold/dead pixels in my raw/mlv files, wondering if that is from the conversion or my raw settings?

[chmee]

@duey101
[MLV] it should. mlv_dump.exe (with audiosupport) is inside the zip. If you drag&drop mlv, you should generally see a question by windows, if it can open mlv_dump.exe. Please enable logging and look into the logfile. maybe it turns out, you have to copy the mlv-files first onto the hdd instead of converting straight from the card.

[coldpixels] i cant decide, from where the pixels are coming. Do you have tested 16bit/without partymode? (these files are openable in speedgrade, rawdigger or photoshop/acr) i do not recalculate the raw-data in this setting..

[tonybeccar]

@chmee, holly molly I didnt see your new version yesterday.. will try it now! Abd oh, too bad then, but good advice, will try the 32bit filters then..

I wonder what news about mlv_dump we'll be seeing... And also I hope you make the command line a priority!!!

Thanks SO much chmee!!! NOW we have a workflow!

[tonybeccar]

?? ProRes on Windows in Premiere Pro CC 
Where are you seeing this. I have been working in there and don't see this.
The only way to encode prores on windows it though FFmeg in a standalone program
like "Cinemartin" or 5DtoRGB (I use that for all my h264)

Yeah, it IS true! I'm running 7.2.1 (4).. the file formats are all different now.. I have a lot of image sequences for example.. and inside Quicktime MOV I have this:

https://www.dropbox.com/s/eicb6qmz1aya7pl/prores.jpg

But whenever I hit export I get a compiler error... maybe I screwed things up myself.. but it IS there!

[duey101]

Man... all I can say is thanks!  This converts my .RAW files with lightning speed.  I'm still not sure on the dead/cold pixels issue.  I noticed they are on all my files whether shot in MLV or RAW.  I'll try some other settings tomorrow.  I'm shooting in MLV/RAW on a 60D with...
1280x720
16:9
fps override on 24fps
Global Draw off
Frame skipping off
Preview - Hacked No Prev
Extra hacks - On

The pixels appear in both MLV/RAW files.  You can see here...



The other issue is with MLV files.  Here is what my raw2cdng window and raw2cdng folder contents looks like...



But if I try to drag an MLV file onto the window nothing happens... although I can see a DOS window flash briefly as soon as I drop the file in, but nothing happens then, the file does not drop onto the GUI and Win 7 x64 doesn't ask any questions.  Not sure why I can't get the MLV files to work.  Any ideas?

The RAW files work perfectly as far as I can tell, pull right up in ACR for batch DR correction, and edit smoothly in PPro CC 7.2.1 (4).

[chmee]

@duey101 i bet, the mlv_dump does show a "file not found" error, maybe of a longer path-tree or unusual symbols in pathnames.. Please try to rename the rawcdng-folder. All points to underlines.

[deadpixels] please try another converter (fi raw2dng) and examine the pictures with rawdigger, lightroom or something else. just to clear, if its sensordata or software-creation-failure.

regards chmee

[duey101]

I moved the folder to my root D:/ rive and now it works.  Renamed the folder to raw2cdng only and that seems to have worked, takes a lot longer to convert MLV>RAW though!  But the RAW to CDNG is really fast!   

[chmee]

raw->dng will be faster next versions (with mlv, please forgive me, i take the advantages of the reference-implementation of g3gg0)

[duey101]

Here is some interesting stuff...

Convert with raw2dng, clean no weird pixels



Then convert with raw2CDNG... 12/16 bit Partymode On and Off same results, funky pixels.



And then I loaded the same file in PPro direct with Drastic's plugin installed again (after uninstall) and I get this... funky pixel hell!



Hmm....

[chmee]

the screenshots are all from premiere? resolve? photoshop? (fi acr uses algorithms to hide these pixels, my dng's and the drastic-importer have no "repair-sensels"algorithms..)

[duey101]

The first two are from Lightroom 5, I converted the raw's in raw2dng/raw2cdng and opened both files in LR5 and took screen caps.  The last is from a single frame export from the PPro timeline.

[arturochu]

my premiere doesn't even have quicktime mov option! i'm running latest version, i would love to export to prores in windows, i think you have something installed in your pc that activated those codecs.

[arturochu]

Yeah, it IS true! I'm running 7.2.1 (4).. the file formats are all different now.. I have a lot of image sequences for example.. and inside Quicktime MOV I have this:

https://www.dropbox.com/s/eicb6qmz1aya7pl/prores.jpg

But whenever I hit export I get a compiler error... maybe I screwed things up myself.. but it IS there!

okj i got it, you can export from premiere cc in windows IF you install mediareactor, then a lot more codecs will appear, including prores, i've already donde this but actually i can export to prores with no problems! 

[tonybeccar]

okj i got it, you can export from premiere cc in windows IF you install mediareactor, then a lot more codecs will appear, including prores, i've already donde this but actually i can export to prores with no problems! 

Lol, yes installed it and uninstalled it.. another victim of the mediareactor greatness..

[duey101]

Your converter is looking VERY NICE!!!  After more testing today I have found that to convert MLV files with raw2cdng they need to be in the same directory as the raw2cdng program and that needs to be on a root drive for example, D:/raw2CDNG works for me.  At least this seems to be the case for me.  While processing the MLV files this window pops up...



But the files finish converting (slowly) and then appear in the raw2cdng GUI, after that they convert really fast to CDNG.  The mlv_dump process is the only bottle neck in this method of converting MLV files.  Here are some more test images from today.

raw2dng, no issues.  (4:1 magnification)



raw2CDNG, very slight problem on a few pixels, or maybe blown highlights or something?  Almost undetectable.   (4:1 magnification)



MLV files converted through raw2cdng...

File 1



File 2



Both MLV conversions are crispy clean, no pixel problems.  I completely uninstalled DMR and Preview, not sure if that had anything to do with it.  But while it was installed I did some more conversions and got this clean raw2cdng file when opened in Light Room 5...



But then when I opened the exact same .dng file in PPro I go these pixels showing up...



Still testing...

[chmee]

@duey101 thx a lot for your testing parcour.

(/me) important toDo: native mlv with splitfile-support

* Are these corrupted Sensels static?
* did you tried rawdigger with both files? (raw2dng vs. raw2cdng)
* are you using a proper modeltag?
* if you want, send me the test-dngs to raw2cdng@phreekz.de

regards chmee

[duey101]

* Are these corrupted Sensels static? 
Static as in do they move around when the video is played on the timeline?  No they do not move but remain where they are.  What does Sensel actually mean?  (sorry, new to these terms)

* did you tried rawdigger with both files? (raw2dng vs. raw2cdng)
I never used that program but downloaded just now, have no idea what it does.  LOL!!!  When I open both files in Rawdigger the raw2CDNG file looks okay, the raw2DNG file looks pinkish.

* are you using a proper modeltag? 
I'm not sure, this is how it looks when I convert...



I got that model name off of some list that I think was EXIF data for specific cameras, where do I get the correct name for a 60D?

In some of the RAW files I get these pink corrupted frames (2 or 3 per clip).

[duey101]

After converting several formats, H.264>>ProRes (with Bigasoft Converter) Cineform>> (GoPro Studio) and a few others I have tried, these CDNG files are the best and edit in real time seamlessly even with effects dropped on them and in full res.  The native H.264 files are horrible to try and edit with and even ProRes files bog down in 1/2 and 1/4 res.  The CDNG workflow is the way to go.  It is really smooth in PPro editing wise so far.  Glitch free scrubbing and playback at full res, on my machine anyway.  You don't have to import all the clips in as a sequence or anything like that, just convert and drag into the project bin from the Media Browser in PPro.

You are on the right path, and close with this.  The best thing is how FAST it converts the RAW files!  Fast!  MLV is another story because of that bottleneck, so yes, native MLV may be the way to go.

Oh... on some clips I get weird color banding in PPro like in this image (3X magnification) screen shot from PPro single frame export.



But doesn't appear if the same CDNG is opened in Photoshop.  Only looks like that in Premiere.

[chmee]

thank you for the motivational words

[static sensels] The Name Sensel comes from the fact, its the "pixelated" part of a sensor, but not a pixel, because 2/3 of its destinated value is calculated. Wiki says, its a blend of "sensor" and "cell". Part of an array of sensors.

[Rawdigger] has one big advantage, its good to analyze raw data - so if there are real dead pixels, they should be present in both files, as rawdigger do not reconstructs and refines the picturedata.. it seems, its the case.



[banding][moiré] do you record in zoomed mode? these "problems on hard contrasty edges are logical because of the bayer-structure - in premiere more than in ACR/Lightroom (..better debayer-algorithms). BUT i've been asked messages before if its depending on my tool. Yes, it seems, my recalculation makes it more obvious - and No, its already there..

[corruptedFrames] Oh.. maybe there is an "coincidence" with the amount of splitted parts? 2 damaged Frames with 3 SplitFiles?

my 5 cents. regards chmee

[duey101]

Okay that's good to know and thanks for the education.  When that dead pixel shows up in the rawdigger does that mean my sensor actually has a dead pixel in it?!!!    Or is is that during the recording in raw a dead pixel happens?  I hope my sensor isn't being damaged.  Still new to all this technical stuff.

Those shots with the color banding were shot in 5X crop mode so I get how that would effect the image.  Overall, when I shoot raw on my 60D I can get 400-420 frames in MLV mode and about 7-10 seconds in RAW mode @1280x720 which scales up quite well to 1920x1080.

As far as the pink corrupt frames not sure what the cause of that is.

[chmee]

Do a simple CR2-raw-Picture and load it into rawdigger. Look, if the deadpixels are present. And Yes, Dead or Hot-Pixels/Sensels are typical - but in your case its a lot If its really that much, i would contact your distribution.. i won't believe, the local dealer will handle that, but another higher instance of canon.. as a gesture of goodwill maybe..

ahh, btw. these corrupted Frames are not caused by conversion, but seems to be a "problem" with too high recording-datarate.. MLV is a little bit slower than raw_rec.
http://www.magiclantern.fm/forum/index.php?topic=10257.0

[duey101]

I loaded a raw CR2 file and didn't see any deal pixels so what does that mean?  Why do they show up in the RAW/MLV converted CDNG's?  I don't get it. 

[chmee]

just one more test.
Do they show up, if you're using the reference converter? fi. raw2dng, mlv2dng, mlv_dump?
Do they disappear, if you're using lower resolutions? (datarates)

I'm not soo into it, to decide now, where the failure's coming from.. sry.

[duey101]

I just ran Rawdigger and used a picture of a white wall (off white) and then blew it up in Rawdigger.  Looks like this...



Other than these wht/grey colors there are no black/hot/weird looking pixels, they all look like this so I think my camera is fine, no dead pixels or other colored pixels at all.

Let me test the other options and get back to you.

[duey101]

Okay, after another battery of tests here is the result.

raw2dng - no issues
raw2CDNG convert raw file, no issues, convert mlv file no issues.
mlv2dng - no issues.

All the files are clean including the 10X crop.  Now...

If I take the raw2CDNG file and open it in PPro, drag it to the project window, open it in the source monitor, blow it up to 400%... THEN I see the dead/black/sensel issues.  Look here...



But outside of PPro these pixels from the same DNG/CDNG files are not present.  I sent you the converted files in email.  I don't know with the mlv2dng or raw2dng files cause they don't open in PPro (because they are not Cdng I assume).  EDIT: And if I output the sequence from PPro the black/dead/sensels show up in the output video, but oddly enough not in the files outside of PPro.