The CinemaDNG Discussion (raw2cdng)

[feureau]

Oh, Sweet! Thanks, chmee! :3

[Jbowdach]

THis looks like a fantastic diagnostic tool to help resolve issues w problematic MLV files. THanks so much! Looking forward to the next version of raw2cdng.
Please let us know how we can use it to either fix files, and\or use it to provide you with useful diagnostic information

[evanbuzzell]

In beta 6, I still cannot scroll through the files without the app crashing

[ChadMuffin]

Just a thought for a request. I have seen in the MLV to DNG Batch Converter v1.9 there is an option to use an ACR profile when exporting the DNGs. The idea behind it is to apply an ACR preset that is a flat profile or one that you would create for what ever you have in mind. Then, you could either bring those into AE to export to ProRes or what ever codec you want OR bring those right into Premiere or Resolve with that preset applied. That would save a ton of clicks, especially with larger projects with bringing them with a flat preset applied already. However, I do not know if Premiere or Resolve will read that applied preset since cDNG is a different process than just the normal DNG (to my knowledge).

[sergiocamara93]

However, I do not know if Premiere or Resolve will read that applied preset since cDNG is a different process than just the normal DNG (to my knowledge).

No, they won't. PP uses Speedgrade framework which doesn't read ACR and the same goes for Resolve (where you can apply a flat profile - BMPCC - and then a LUT of your choice to all your footage. Plus it encodes faster).

Forgot to mention that MLV to DNG Batch Converter v.1.9 generates DNG, not CDNG. So PP won't read them at all, for Resolve it's about the same between the two formats right now (reads both equally).

[ChadMuffin]

@sergiocamara93

Thanks for the info! Never mind about my request then. heh.

[kaco]

Hi, from my 2TB shooting I have a trouble opening one of my file. The loader bar is still loading, both raw2cdng and mlvdiag.

What i did -- downloaded all the mlv's, deleted those 1 bytes and batch renamed all the mlv's (because of Resolve round-trip method).

The debug info:

Code: [Select]

[drop] ------
[drop] -- file K:\xxx\002_M0336.MLV will be analyzed now.
[drop] is MLV
[setFileinfoData] set/changed Filedata
[drop] FileinfoData set
[createMLVBlocklist] started
[createMLVBlocklist] splitFiles counted (1)
[createMLVBlocklist] indexing Blocks in K:\xxx\002_M0336.MLV

My theory is that the end of this file was splitted into those 1byte chunks and sice I deleted them, raw2cdng don't know where to find them? The progress bar is still active, but the disc activity is zero.

EDIT: But I deleted many 1 byte files and from 399 MLV's only with this I am getting error. The file exctracts with MLV Converter 1.9.2 fine.

[chmee]

@kaco
can you open it with mlvdiag? Oh Sorry, see now, you tried. ok. will change the reader module in mlvdiag/raw2cdng to allow open/read "corrupted"/strange blocks.

[kaco]

@chmee: now tested with mlv convertor 1.9.2 ... the files open, but won't extract... since it's a 80gb of data I think there's something important in it.. oh, god...
Do you think there is a chance to get the data from the file?

tested it also with mlv2dng:

Code: [Select]

mlv2dng.exe 002_M0336.MLV
Exception! Header not recognized: ''

Exception! Header not recognized: ''
mlv_dump:

Code: [Select]

mlv_dump.exe 002_M0336.MLV

 MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: '002_M0336.MLV'
   - Verify file structure
File 002_M0336.MLV opened
File 002_M0336.M00 not existing.
Processing...
[ERROR] Missing file header
Processed 0 video frames
Done


[chmee]

Do you have a Hex-File-Editor like HXD? Open the file with it, copy the first 1MB (or 5 or 10) to a new file and send it to me. raw2cdng@phreekz.de (i will code such a button into mlvdiag).. Ahh, please write down the original and ml-firmware-version and body, maybe its a hint for @a1ex on a bug. maybe..

[chmee]

as you assumed in your pm, this file is empty, full with zeros.

regards chmee

[pompeiisneaks]

Chmee, I've got visual studio 2012 and have been playing with the source a bit, added some exception code due to warnings, etc.  I'd be happy to help out on the code base, I'm trying to get my C# skills up, I've got a class on it this semester, so any tinkering I can do will help me.  Also, I can help try and add some more try/catch blocks for some of the I/O type errors we've been seeing, but wanted to make sure you're needing help with the program.

~Phil

[chmee]

@Phil
of course, lets start a workflow with bitbucket.

[Jbowdach]

hey chmee,

here are a few logs from attempting to decode MLV files from my latest vacation clips. These are all shot on a 7D. Tried numerous times, and it seems some of these files just dont want to decode, they just crash the application. Only several files out of almost a TB of video tho, so not horrible. Just wish I knew what was causing these crashes. Please let me know if I can provide any further info about any specific file.

Smaller log1 error:https://www.dropbox.com/s/3szc3mruyhop361/raw2cdng.2.debug_crash630_alaska.log
Log 2 error log: https://www.dropbox.com/s/pschwkg4derpi1e/raw2cdng.2.debug_crash2_630alaska.log
Log 3 Errors: https://www.dropbox.com/s/ts3z31ejyunnn8q/raw2cdng.2.debug_crash3_630alaska.log
BIG log file: https://www.dropbox.com/s/houu1koi7ff3yw1/raw2cdng.2.debug.log

[arturochu]

chmee what happened to the development of this program? not a single update in two months? bummer.

[ChadMuffin]

+1

[chmee]

guys. sorry, really.. it just has been full o' work, and it is still. next days i'll publish a beta7 with some fixes but *SORRY* still no chroma and vertical banding. on the end of this month i'll start my holiday-time with more time for this project again.

to be honest, i didnt found the trick with chroma and vertical banding - i just spent far too little time to understand the sourcecode - and i think about changing the whole code to byte-arrays (instead of uint16) to be on a better track for that. cant say, if it changes the transcoding time. on the todolist is a c#-dll-library for mlv-things as well, cause i use nearly the same code on raw2cdng and mlvdiag. makes sense but positive news: a german guy mailed, he wants to help on raw2cdng, maybe we get all the transcoding-code in c++, hoping it speeds up the thing.
 
regards chmee

[ChadMuffin]

Glad to see more things coming and expanding the team! Excited for more developments.

[arturochu]

+1

[chmee]

short info on some work: as someone stated i didnt used the WBAL-data from MLV. the values are kind of tricky. why? The values inside WBAL are only usable under some circumstances:

these are (by now) the interesting data-fields in WBAL

wb_mode - from 0 to 6 and 9
kelvin - only accurate if wb_mode = 9
wbgain_r ,_g and _b - only accurate if wb_mode = 6

If you use wb-Presets (tungsten, shade, cloudy etc), i only see the preset, but neither kelvin nor wb_gain is right (there will be old values from the moment you used them last time). If you use KELVIN (manual setting), only kelvin will be right, the wbgain-values are not refreshed. as well, wb_mode MANUAL leads to accurate wbgain-values, but kelvin wont be refreshed.

(by the way)

wbs_gm - is the known green-magenta-shift, i assume, its a simple value shift green versus red/blue
wbs_ab - assume its a YPbPr-Shift (Lab)

Why do i write this? it seems, it makes sense to write down all necessary fields/values and save them as lists into the main ml-modules or into the converter. If you look into a cr2-file from your body, you ll find the presets. more: to get the most informations we should save the values from all possible Kelvin-Steps.

(1) i did pictures with all kelvin-steps (5DIII 2.500-10.000).
(2) saved them into a folder
(3) then i used the great exiftool to extract the wbgain-fields.

save this code into a file named wb_manual.fmt into this folder:

Code: [Select]

$ColorTemperature,${WB_RGGBLevelsAsShot;tr/ /,/}
and run this via command-line inside the folder you saved the cr2-files.

Code: [Select]

exiftool -p wb_manual.fmt *.CR2 > list.csv
voila, you ve got a commaseparated list - kelvin to rggb-values. here you see it inside excel:


then do this format-file (name it wb_presets.fmt) on only one cr2-file to save the presets:

Code: [Select]

Daylight,$ColorTempDaylight,${WB_RGGBLevelsDaylight;tr/ /,/}
Shade,$ColorTempShade,${WB_RGGBLevelsShade;tr/ /,/}
Cloudy,$ColorTempCloudy,${WB_RGGBLevelsCloudy;tr/ /,/}
Tungsten,$ColorTempTungsten,${WB_RGGBLevelsTungsten;tr/ /,/}
Fluorescent,$ColorTempFluorescent,${WB_RGGBLevelsFluorescent;tr/ /,/}
Flash,$ColorTempFlash,${WB_RGGBLevelsFlash;tr/ /,/}call it:

Code: [Select]

exiftool -p wb_presets.fmt [CR2-FILENAME] > wb_presets.csvafter that we've got the rggb-values for the wb_mode's 0-5. for 5DIII fw 1.1.3 these are the values:

Code: [Select]

Daylight,5200,2052,1024,1024,1678
Shade,7000,2362,1024,1024,1431
Cloudy,6000,2208,1024,1024,1544
Tungsten,3200,1479,1024,1024,2497
Fluorescent,3714,1796,1024,1024,2399
Flash,6320,2284,1024,1024,1520
so, now i've got all values to give the cinemadngs (nearly) accurate rggb-values to simplify the workflow inside the editing tools.

and now? whats the problem? it seems (look inside a cr2) these rggb-values from 5dIII are not accurate for all bodies. i will build this things into the next raw2cdng-version, but maybe its necessary to do these rggb-lists for every modell - and maybe these values change with the firmware. if someone's got some time for that, try to build these lists as well..

@g3gg0 , @baldand and @a1ex was this work somewhere done already?

@jbowdach i didnt forgot you, right now i just have the time to solve actual things. if you've got a "smaller" non-functional file, please send it to me.

regards chmee

[kaco]

Hey there,

unless I did not make any mistake, the values are different on my 5D mark III's:

1. Older 5D mark III, bought in 2012, SN 013021001***, FW 1.1.3

Code: [Select]

2500,1182,1024,1024,31392600,1226,1024,1024,30222700,1274,1024,1024,29212800,1326,1024,1024,28192900,1369,1024,1024,27523000,1413,1024,1024,26893100,1450,1024,1024,25763200,1489,1024,1024,24673300,1526,1024,1024,23993400,1565,1024,1024,23303500,1606,1024,1024,22603600,1638,1024,1024,22083700,1672,1024,1024,21533800,1708,1024,1024,20973900,1739,1024,1024,20564000,1768,1024,1024,20164100,1802,1024,1024,19784200,1833,1024,1024,19384300,1859,1024,1024,19074400,1886,1024,1024,18724500,1913,1024,1024,18434600,1942,1024,1024,18114700,1967,1024,1024,17804800,1990,1024,1024,17534900,2009,1024,1024,17275000,2028,1024,1024,16995100,2048,1024,1024,16755200,2068,1024,1024,16495300,2089,1024,1024,16315400,2110,1024,1024,16135500,2131,1024,1024,15965600,2153,1024,1024,15775700,2171,1024,1024,15635800,2189,1024,1024,15475900,2208,1024,1024,15336000,2226,1024,1024,15176100,2241,1024,1024,15076200,2255,1024,1024,14946300,2270,1024,1024,14836400,2284,1024,1024,14716500,2300,1024,1024,14586600,2315,1024,1024,14486700,2330,1024,1024,14366800,2346,1024,1024,14276900,2362,1024,1024,14157000,2378,1024,1024,14047100,2394,1024,1024,13967200,2405,1024,1024,13897300,2416,1024,1024,13807400,2427,1024,1024,13727500,2439,1024,1024,13647600,2450,1024,1024,13577700,2461,1024,1024,13487800,2473,1024,1024,13417900,2485,1024,1024,13328000,2497,1024,1024,13268100,2509,1024,1024,13178200,2521,1024,1024,13118300,2533,1024,1024,13038400,2545,1024,1024,12988500,2551,1024,1024,12938600,2558,1024,1024,12878700,2570,1024,1024,12828800,2576,1024,1024,12768900,2583,1024,1024,12719000,2595,1024,1024,12669100,2602,1024,1024,12609200,2608,1024,1024,12569300,2615,1024,1024,12509400,2628,1024,1024,12459500,2635,1024,1024,12419600,2641,1024,1024,12359700,2655,1024,1024,12319800,2661,1024,1024,12259900,2668,1024,1024,122110000,2675,1024,1024,1215

Code: [Select]

Daylight,5200,2068,1024,1024,1649
Shade,7000,2378,1024,1024,1404
Cloudy,6000,2226,1024,1024,1517
Tungsten,3200,1489,1024,1024,2467
Fluorescent,3714,1817,1024,1024,2372
Flash,6320,2305,1024,1024,1492
1. Newer 5D mark III, bought in 2014, SN 268021002***, FW 1.1.3

Code: [Select]

2500,1159,1024,1024,30132600,1201,1024,1024,29052700,1247,1024,1024,28042800,1295,1024,1024,27092900,1334,1024,1024,26483000,1376,1024,1024,25833100,1413,1024,1024,24793200,1450,1024,1024,23783300,1485,1024,1024,23153400,1522,1024,1024,22503500,1558,1024,1024,21853600,1589,1024,1024,21363700,1621,1024,1024,20853800,1654,1024,1024,20323900,1683,1024,1024,19934000,1713,1024,1024,19564100,1745,1024,1024,19204200,1774,1024,1024,18834300,1799,1024,1024,18534400,1824,1024,1024,18244500,1846,1024,1024,17924600,1872,1024,1024,17654700,1896,1024,1024,17364800,1917,1024,1024,17114900,1935,1024,1024,16865000,1956,1024,1024,16595100,1975,1024,1024,16365200,1993,1024,1024,16115300,2013,1024,1024,15945400,2032,1024,1024,15775500,2052,1024,1024,15605600,2072,1024,1024,15425700,2093,1024,1024,15295800,2110,1024,1024,15135900,2127,1024,1024,15006000,2144,1024,1024,14856100,2158,1024,1024,14756200,2171,1024,1024,14646300,2185,1024,1024,14526400,2198,1024,1024,14426500,2212,1024,1024,14316600,2226,1024,1024,14216700,2241,1024,1024,14096800,2255,1024,1024,14006900,2270,1024,1024,13897000,2284,1024,1024,13787100,2300,1024,1024,13717200,2310,1024,1024,13647300,2320,1024,1024,13557400,2330,1024,1024,13487500,2341,1024,1024,13417600,2351,1024,1024,13327700,2367,1024,1024,13267800,2378,1024,1024,13177900,2389,1024,1024,13118000,2399,1024,1024,13048100,2411,1024,1024,12968200,2422,1024,1024,12908300,2433,1024,1024,12828400,2444,1024,1024,12778500,2450,1024,1024,12738600,2461,1024,1024,12668700,2467,1024,1024,12628800,2473,1024,1024,12578900,2485,1024,1024,12519000,2491,1024,1024,12479100,2497,1024,1024,12429200,2509,1024,1024,12379300,2515,1024,1024,12329400,2521,1024,1024,12289500,2533,1024,1024,12229600,2539,1024,1024,12189700,2545,1024,1024,12149800,2558,1024,1024,12089900,2564,1024,1024,120410000,2570,1024,1024,1198

Code: [Select]

Daylight,5200,1993,1024,1024,1611
Shade,7000,2284,1024,1024,1378
Cloudy,6000,2144,1024,1024,1485
Tungsten,3200,1450,1024,1024,2378
Fluorescent,3714,1751,1024,1024,2280
Flash,6320,2217,1024,1024,1460
For instance also the shutter sounds a bit different on these, so maybe there's some little design difference?

I also doublechecked that no WB shift was set on both and that both had same settings, same standard profile, disabled all "enchance" functions, etc.

[chmee]

@kaco
thanks a lot. i assumed, that every sensor has got its own indivdual adjustment - maybe once on build or maybe once on switch on or firmware-update.. means, every 5DIII has a slightly other characteristic curve. but for the job as a converter-app with static rggb-values it should be enough - OR - i/we apply this data as external files - i dont think its worth it.

now could be interesting to see, how the curves differ on other models.

thanks and regards chmee

[kaco]

For me it's totally OK to use some "average" values. This values will still be used only when creating proxies and then finetuned in the final color grade.

[Jbowdach]

Im having sync issues when I autosync in Resolve. Theres a consistent gap at the beginning of the audio. Anyone else having this issue or other audio sync issues?

[chmee]

@jbowdach
i just did a small test with ~25 files, recorded (in terms of timecode) in nearly same time. after autosync it went good, no gaps, no real problems. i assume, my timecode-calculations for the audiotrack going wrong if they're broken framerates. did you checked the timecode inside the files?



just found that: maybe thats kind of similar?

regards chmee

btw: i implemented the WBAL-usage and resolve, premiere and photoshop behaves quite well. did some more exception-trapping and debug-writing. beta7 is near.