The CinemaDNG Discussion (raw2cdng)

[chmee]

Hey there.. I'm currently trying to realise a CinemaDNG-Converter. First i thought it would just be an existing xmp-packet inside the dng-frame, but it wasn't. After reading the CinemaDNG-Definition i saw, there are about 35 mandatory tags.

DNG Specification 1.4.0.0 - June 2012 - CinemaDNG Specification 1.1.0.0 - September 2011

The CinemaDNG-File is just a TIFF/PE File (as DNG is also) with these CDNG-Tags. I wrote an App for Win7, building a CDNG-Header with the raw-pictures from the Camera-raw-file. They are 14Bit, isnt it? I figured out, that Resolve and Speedgrade cant handle 14Bit Raw-Frames! Just 8, 10 or 16Bit, maybe 12Bit. So, now i'm sitting on converting these 14bit to 16Bit-Chunks. These CinemaDNG-Sequences are recognised by Resolve and Speedgrade, but they are chaotic because of wrong handling/converting.

If asking, why the simple DNG-Sequences are not usable, it might be that
(+) missing CDNG-Tags
(+) IFD0-Entry is the jpg-Preview in 128x96px 8Bit - thats the one you see in Resolve.
(+) Raw-Data are packed into a subIFD - as in the DNG-Definition described.
(+) they are 14bit.
 
Questions:
(*1) Raw-Data - Subpixel-Chunks, MSB or LSB first? LittleEndian, so LSB first, right?
(*2) is it worth that? Maybe another Fileformat would be more usable? DPX? EXR? TIF?

regards chmee


regarding raw2cdng

i decided to stay on the same name and same tool with integrated mlv-convert (since 1.4.5). Resolve, Speedgrade and all ACR-using-Apps are coping well with the cdng's. With 1.5.0 (coming next) there s a new GUI, and on NAB(07.04.2014) Adobe releases a bigger update promising a better workflow with cdng. 1.5.0 is online. since 1.6.5 audiosync seems stable, colormatrices are refreshed. you find more info on my webblog.

please look in my blog-entry to find the newest version!

bitbucket-sourcecode c# (with 1.5.0beta2 i've left it into the free world)
https://bitbucket.org/chmee/raw2cdng

actual release : raw2cdng 1.6.5 for windows
http://www.magiclantern.fm/forum/index.php?topic=5618.msg136306#msg136306

regards chmee

[a1ex]

1) https://bitbucket.org/hudson/magic-lantern/src/tip/src/raw.h . Don't ask me about endianness stuff, I've just scrambled these bits until it started to look good

ARM is little-endian.

[squig]

lmao.

thx for coming chmee, save us from Adobe hell.

[CaptainHook]

I figured out, that Resolve and Speedgrade cant handle 14Bit Raw-Frames! Just 8, 10 or 16Bit, maybe 12Bit. So, now i'm sitting on converting these 14bit to 16Bit-Chunks.

BMCC DNG's are 12bit log that unpack as 16bit linear in resolve.

[scrax]

Good topic, since a lot of people seems to miss that compatibility.

My suggestion is (if possible), instead of making a converter for win, will not be better to make ML generate a raw with all what needed and make raw2dng able to convert directly to cinemaDNG? That way we will have a multiplatform solution, also 12 and 10 bit raw are under development so probably the converter should be able to handle different bitrates.

[chmee]

of course. Right now the quest on 10/12Bit is the right way to get cinemaDNG(@10Bit) straight ooCam. finally, there will be a converter computerside - and the integration/rewriting of the cdng-header is the far smallest problem. I'm just sitting to solve the 16bit converting thang, maybe after that we will have a cdng-converter till we implement the idea into raw2dng. its the experience "we" are doing now.

ah, if someone misses the thread 10/12bit -> http://www.magiclantern.fm/forum/index.php?topic=5601.100

regards chmee

[sicetime]

I work in post, and have assimilate's scratch and resolve, I am ready and willing to test any post workflows that come out of this. Thanks for the great work, I think this one of the most exciting ml threads I have read!

[EOSHD]

Here's how the ML DNG metadata differs to Cinema DNG (Blackmagic Cinema Camera)

http://www.eoshd.com/uploads/dng-metadata.png
(Blackmagic Cinema Camera EXIF on the left, Canon raw DNG on the right)

As you can see stills standard DNG EXIF is very different to Cinema DNG.

DaVinci Resolve seems to read the reduced-resolution metadata for 5D Mark III footage.

There seems to be other errors like JPEG compression for the full resolution image and uncompressed for the reduced resolution image?

[sicetime]

Any update on this?

[EOSHD]

My research shows the OP was correct, the problem is -

(+) IFD0-Entry is the jpg-Preview in 128x96px 8Bit - thats the one you see in Resolve.
(+) Raw-Data are packed into a subIFD - as in the DNG-Definition described.

Stills DNG has support for multiple images in one DNG at different resolutions (subfiles) and Cinema DNG only has one 'primary-image'. The raw data in the stills DNG is in a subfile, and seemingly the primary file is a thumbnail. Resolve reads the thumbnail data in IFD0-Entry but cannot access the raw data packed into a subIFD since it lacks support for this file structure.

Could we change Raw2DNG so subIFD becomes ifd0? I.e. pack the raw data into IFD0 instead of SUBIFD? Is that a trivial metadata change or more complex? There is a DNG SDK and Cinema DNG docs, someone just needs to understand the file format and write the correct structure into Raw2DNG's code.

I asked a Blackmagic guy via John Brawley about Resolve supporting stills DNG, and Pete confirmed it doesn't yet support DNG only Cinema DNG. I am sure they are working on it but probably quicker to change Raw2DNG.

This thread at EXIFTool was useful in pinpointing the problem - http://u88.n24.queensu.ca/exiftool/forum/index.php?topic=4927.0

[chmee]

just to give you some food Here's the link to the app. (win)

*refresh* to 0.91
---
*lol* sorry, compiling while Championsleague-Finals is a bad idea.
0.92 - http://www.dslr-kleinanzeigen.de/Download/raw2cdng.0.92.zip

*!* NOT FUNCTIONAL*!*
(+) you need installed Microsoft .net 4.0 Runtime
(+) converting limited to 5 frames
(+) just for dev/analysing

problems so far:
(*) not enough time to solve, because workin' hard. pokin' around.. tuesday i'm back free on this thing.
(*) 14bit is functional with working image
(*) 16bit shows converting problems - have no time to finish successfully. (but is workable in resolve)
(*) both are not recognised by photoshop/acr, but viewable with fe irfanview. help appreciated (maybe colormatrix)
(*) for investigating on the converting problem try rawdigger, switch between the Sensel-channels.
(*) it looks like an array not NULLED in loop. rewriting the convert-function back to "simple" bitshifting and masking.

@EOSHD
if the IFDs would be changed, it wouldnt be no more a DNG. My Application throws out a CDNG with 14Bit - but they're not readable in Resolve Speedgrade etc pp.. The 16Bit-Files are readable, but not usable

For the devs here. Thats the converting loop: Do you see any Problems in it?

Code: [Select]

// c# code
// source - byte[] - rawstream 14bit
// chunks - int - stripByteCounts
// output -> Dest - byte[]

        public static byte[] to16(byte[] source, int chunks)
        {
            byte[] Dest = new Byte[chunks/14*16];
            UInt32 tt = 0;
           
            // because of bytestream i take 7 Bytes for (odd)RGRG (even)GBGB
            for (var t = 0; t < chunks; t += 7)
            {
                Dest[tt++] = (byte)(source[t] >> 2);
                Dest[tt++] = (byte)((source[t] << 6) | (source[t + 1] >> 2));
 
                Dest[tt++] = (byte)(((source[t + 1] & 0x3) << 4) | (source[t + 2] >> 4));
                Dest[tt++] = (byte)((source[t + 2] << 4) | (source[t + 3] >> 4));
 
                Dest[tt++] = (byte)(((source[t + 3] & 0xF) << 2) | (source[t + 4] >> 6));
                Dest[tt++] = (byte)((source[t + 4] << 2) | (source[t + 5] >> 6));
 
                Dest[tt++] = (byte)(source[t + 5] & 0x3F);
                Dest[tt++] = source[t + 6];
            }
            return Dest;
        }
regards chmee

[swinxx]

i just started playing around with resolve 9 lite - mac version - and have to say that it is hands down the best color grading program i know.
and the best thing is: it is free. the limitation is 1920x1080, but thats ok for me. you can get really great results
the tracking function is so amazing and easy to use.

so i´m really waiting hard for some conversion tools that can convert the raw file from the camera into a raw format that can be used in resolve.

i wanna thank all the developers and enthusiast for all their commitment.

[g3gg0]

if you had posted the whole code, i would have tried to fix it

[chmee]

https://github.com/chmee/misc/tree/master/%5Bcsharp%5Draw2cdng

Sorry, i'm there..

[notdabod]

Does anyone know of a Mac version for a cinema DNG converter?

Thanks

[squig]

Does anyone know of a Mac version for a cinema DNG converter?

Thanks

There is no cinemaDNG converter yet.

[squig]

I tried saving a DNG to a DNG in ACR and opened it in resolve, it was just thumbnail size but it was in colour.

[IliasG]

Since the basic problem for Resolve is that it cannot read the subIFD info in exif and only reads IFD0, I tried to change the exif IFD0 info by hand and it does work but some crucial tags cannot be changed - deleted - moved .. blocked by exiftool's author for security reasons.

If it worked to the end the next step would be a scipt to automate the whole work and could be a interesting feature of the dng2raw + .. GUIs.

But as it is now I think a request to Phill Harvey to deblock changing those tags and even provide an option for exiftool to make easy these changes is the best I am thinking of. Anyone interested (I am not) can post a request ..

http://u88.n24.queensu.ca/exiftool/forum/index.php/board,4.0.html?PHPSESSID=61s2u55elgkbd4g06blqf67nr5

The other option is, the ML team to change the dng2raw code so that it writes the exif in the IFD0 subgroup looks more difficult to me because an expert in exif data is needed.

[IliasG]

I tried saving a DNG to a DNG in ACR and opened it in resolve, it was just thumbnail size but it was in colour.

What you saw is the "preview" jpeg which is embedded in every DNG. It could be full size (DNG converter gives 3 size options .. full - medium - small) but even then you would have in Resolve jpegs not raws. It's because Resolve reads only the IFD0 exif info which in PhotoDNG describes the jpeg preview and the raw data are described - taged in the subIFD group.

[chmee]

@IliasG
To be honest.. Did you built a DNG-File that opens in Resolve with 14Bit Data?
What TIFF-Tags do you want to change, you're not able to?

Edit: I tried to use smaller sizes (720x400px) for generating small sample files. Camera Raw behaves really strange with these data.. https://github.com/chmee/misc/tree/master/%5Braw%5Dsamples (look into the dng-metadata. the colormatrix shows an inf"inite" entry)

regards chmee

[IliasG]

Hello chmee,

No I didn't build any DNG file, I stopped after I could not change the needed tags.

Blocked tiff tags. 0x0117 (StripByteCounts) - 0x111 (StripOffsets). As you understand these are the most significant tags ..

I cannot open any of your samples (ACR 5.7, Lightroom 4.4-demo, Dcraw,) nor read - change any exif tag with exiftool !!!... "Error: Not a valid TIFF (looks more like a XMP) - 1280_540.dng"

I think, You have to ask for help from Phil Harvey, he is THE EXPERT in exifs and a very good guy.

 

[chmee]

StripByteCounts and StripOffsets.. thought that. It makes no sense to change them in EXIF-Tool. If you want to change them, take a hex-editor. Just not to be misunderstood - by now i dont generate the header, i just use another one and rewrite the important tags. it makes no sense rewriting the whole tiff-header-creation thing, thats already in the magiclantern package - but i m no c(++)-guy.

(*) The samples in the github are dng's converted with raw2dng.exe from 5D3-raws - no changes by me! strange, isnt it? these are raw samples on where we can start talking about - one database and changes in it.
(*) by now i am just interested in the converting from 14 to 16 bit. i m a bit puzzled, that just appending zwo bits to the core data didnt worked, but hey, (michael jackson) "you are not alone" - we're on the way to find a solution..
(*) by the way, irfanview tries his best to show strange data - and in my case it does..

regards chmee

[IliasG]

But if they (the samples) straight from raw2dng why there is no exif ??. Ohh now I see it .. corrupted download ..

Redowloading ..

Believe it or not I had not read this thread from the start

You improved alot !!.

I am looking at 704X400
When changing the bitdepth to 16 logic say you have to change StripByteCounts 
704*400*14/8 = 492800 while it is 504000 in exif !!.
704*400*16/8 = 563200

Don't know if StripOffsets need a change to 38400... as is the usual for raw2dng built DNGs

[chmee]

704*400*14/8 = 492800 while it is 504000 in exif !!.

because its 720x400@14Bit -> StripByteCounts 504000. StripOffsets is a calculated value, where the raw-data starts. Look with a hexeditor into it.. we're talkin 'bout files coming straight from raw2dng.exe(!)

regarcs chmee

[IliasG]

because its 720x400@14Bit -> StripByteCounts 504000. StripOffsets is a calculated value, where the raw-data starts. Look with a hexeditor into it.. we're talkin 'bout files coming straight from raw2dng.exe(!)

regarcs chmee

And where did 704 came from !!. This proves that if you are searching Exif tags instead of going to sleep you get blind

I have no experience nor the free time to start digging with hex editors ..

You mean that you removed the thumbnail and the subIFD header so the raw data start at 1536 ??.