The CinemaDNG Discussion (raw2cdng)

Started by chmee, May 23, 2013, 10:46:55 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

chmee

no. cinema dng and "simple" dng differs much - in terms of mandatory tags. i use a cdng-template, i merged myself from looking into another cdng-frames. just to clarify some things:

(*) dng cant be converted into cinema dng with only two or three another tags. but maybe these videoapps understand dng and dont need any cdng-tags?!
(*) main problem is, 14bit bayer is not useable for cinemadng, because no videosoftware can read 14bit frames as sequence. (my app is able to output 14bit cdng - but there is no software able handling it)
(*) dng batch converting is a "half baked" solution, because it ends in 8bit-sequences with loosing all benefits
(*) converting into tiff-rgb 16bit needs demosaicing and color matrix handling -> not my intention (ffmpeg?)
(*) my app (in actual state) does some weird things while converting to 16bit-bayer - and thats the point of "not-usability"

(+) the bayer structure is ok, because cinema dng can handle it, also the apps, that are handling cdng
(+) it must be converted into 10 or 16bit bayer. 10bit would be worse than 16bit, but definitively better than 8bit baked.

regards chmee
[size=2]phreekz * blog * twitter[/size]

hjfilmspeed

I feel like RAW to Cinema DNG GUI would be amazing!!!! Opening right up in the free version of resolve would be alot of fun!

chmee

Just to give you some hope :) The app is working, when i'm writing synthetic data (10, 12 and 16Bit Output) - but i'm stuck understanding the "simple" 14Bit-RAW-Data. Taking the short RAW-Example put on my server (https://github.com/chmee/misc/tree/master/%5Braw%5Dsamples (zipped raw samples 1280x540))

NEED YOUR HELP FOR THE LAST THING!

Example from 1280_540.raw - first 7 Bytes describe the first 4 Sensels ( RGRG)


hex - 40 29 42  c7 4c  9c 2a
dec - 64 41 66 199 76 156 42

binary:
| 01000000 | 00101001 | 01000010 | 11000111 | 01001100 | 10011100 | 00101010 |

as far as i understand restructuring to 14 bit goes:

| 01000000001010 | 01010000101100 | 01110100110010 | 01110000101010

dec - 4106 | 5164 | 7474 | 7210

BUT this data dont correspondend with the image i see in 14bit. its a uniform background - greyish on Blacklevel 2046, reddish with blacklevel 0. both pixels Pixel1(RG) and Pixel2(RG) have to be (more or less) same value, what is going on?


regards chmee
[size=2]phreekz * blog * twitter[/size]

chmee

ok, here's a v0.99 for beta testing. thanks to reyalp for the hint.
http://www.phreekz.de/wordpress/2013/06/magiclantern-raw2cdng-cinema-dng/

all responses, hints are very welcome. regards chmee
[size=2]phreekz * blog * twitter[/size]

olik

Hello chmee,

this is very exciting. However, after converting a raw file that I created with the 5dmk2 without changing the bitrate, I could not open the dngs with resolve 9. It did not "see" the dngs. So I opened one dng with photoshop where I got presented a very contrasty image, barely recognizable without blue channel (will post dng tomorrow).

arturochu

same problem as last post, i thought it would make a single cdng file but made a lot of dngs that i cannot open in resolve
Chu

chmee

thanks for response. actually i just tested with 5DIII-Files. So it would be fine you're leaving somewhere a short & small raw-file from your bodies. Ah btw. No, it is a filesequence with cinema-dng pictures (their ending is dng as well) - for using in resolve its mandatory changing to 16bit! olik, post a screenshot. (seems i have to delete the 14bit-converting :)

regards chmee
[size=2]phreekz * blog * twitter[/size]

gogauze

I did a quick test with my 5D3 and converting into 16bit, works like a charm. I'll be even more happy when one of the Mac folk ports it to OSX. I am sincerely impressed with your skill.

squig

What's the increase in file size converting to 16bit?

gogauze

Fairly negligible really. I think it increased by maybe 10% on the quick test that I ran.

IliasG

@ Chmee,

Congratulations, and many thanks.

Regarding file size.

A good solution would be to store the data in the exported cine-DNG at 12bit log (after applying a LUT on the 14bit linear raw data) and then inject a inverse LUT (12bit log to 16bit linear) in the corresponding exif tag. It's the same as BlackMagick do and Nikon's "visually lossless" raws also. This would give a 12/16 file size.

What happens if we pass this 16bit cine-DNG through Adobe DNG converter. If it does not destroy the cine-DNG format is would give a respectable size decrease as by default it applies lossless compression which for such files can compress at about half the size.

olik

Tested again for the 5dmk2, this time I changed the bitrate from 12 to 16bit. This solved my issue, I can now import to resolve. I have a strong green cast though, but it is perfectly correctable. Wonderful job! I love it :-)

Here the C-dng in resolve


vs h264 (same camera settings)


and the first DNG of the sequence
https://picasaweb.google.com/lh/photo/tZ7PtaVRzHpdgX1PKcC_0aHmTw6ITH9KGra-l8uiJak?feat=directlink

a1ex

Quote from: olik on June 03, 2013, 01:33:35 PM
I have a strong green cast though

The black level is too high; it's probably hardcoded at 2048, which is usually right for 5D3, but too big for 5D2. The one from exif should be a good starting point.

scrax

I can load the app in Osx with mono and load in a file, but how to start it?


EDIT, maybe it did all, I got a lot of dng but with some stranges names due to the win \ for folders I suppose

Also mono gives this error when loading it:
mono[75121] <Error>: kCGErrorInvalidConnection: CGSGetWindowTags: Invalid connection
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus | EOS 5D MLbeta5- EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX, PS, LR, RawTherapee, LightZone -no video experience-

scrax

So far on osx when converting a raw file I got this file structure at the end:

rawfilefolder/M0000001.RAW
rawfilefolder\ML_16/   <--this is not a subfolder of rawfilefolder it's another empty folder
rawfilefolder\ML_16\ML_16_000000.dng
rawfilefolder\ML_16\ML_16_000001.dng
rawfilefolder\ML_16\ML_16_000002.dng
...  and so on with files having the win folder structure in the name.
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus | EOS 5D MLbeta5- EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX, PS, LR, RawTherapee, LightZone -no video experience-

olik

While we already talk about using this software on different operating systems, I tried using it on Ubuntu Linux with mono, without luck;

mono '/raw2cdng.0.99.exe'
Missing method EnableVisualStyles in assembly raw2cdng.0.99.exe, type System.Windows.Forms.Application

Unhandled Exception: System.IO.FileNotFoundException: Could not load file or assembly 'System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' or one of its dependencies.
File name: 'System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'
[ERROR] FATAL UNHANDLED EXCEPTION: System.IO.FileNotFoundException: Could not load file or assembly 'System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' or one of its dependencies.

File name: 'System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'


has anyone an idea?


scrax

I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus | EOS 5D MLbeta5- EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX, PS, LR, RawTherapee, LightZone -no video experience-

chmee

thanks for the testers.

summarizing for the next versions
* >2GB filehandling
* path-strings uniform and no windows-dumb backslash
* pink fringing
* vertical lines (could be solved with 0xc619 or 0xc61b )
* 10 and 12 bit files (no logarithmic with LUT for now - think, there is a tag for it - linearizationTable 0xc618)
* batchmode?!
* handling other bodies with their metadata (colormatrix, whitelevel and so on)

regards chmee
[size=2]phreekz * blog * twitter[/size]

a1ex

Quote from: chmee on June 03, 2013, 10:02:33 PM
* vertical lines (could be solved with 0xc619 or 0xc61b )

This is black level for each column, but for the banding issue we need EV correction for each column (mod 8 ). Do you know any tag for that?

chmee

ahh. ok. there is no metadata-array for ev correction - as i see that now.

Some Question there have been:

14 to 16Bit -> ~14.2% increase (16/14 = 8/7)
14 to 12Bit -> ~14,2% decrease(12/14 = 6/7)

@IliasG
No, logarithmic transforming is not my intention. for 12Bit i will cut LSB 2 Bit - we'll gonna see, how many data in the shadows is going to be lost - i quite think, its negligible :) Dont forget - the Magiclantern Team, all the coders around it gave you a weapon you never had the money for that - and this "techspec-more is better"- blahblah doesnt make a good film - your movie is not depending on logarithmic 16bit, but on "substance"!

@olik
what version-number of .net-framework is running in mono? it needs 4.0
[size=2]phreekz * blog * twitter[/size]

silwerfedlt

This is truly amazing and I´ve tested your tool shortly tonight with my 5D2 and I had to lower the blacklevel alot I think I had it on 1024 (just trying different settings to figure out how it works..). If it was on 2000 I had a very dark picture with alot of green. What would be the right setting for 5D2? Whys does it doffer from 5D3?

I´m new to resolve so I have to learn alot to be able to grade thorougly in the program but the basics are the same as the comp programs I use daily. (The Foundry Nuke and Eyeon Fusion).

Its so nice to be able to skip the Adobe programs - I don´t trust them at all for grading RAW sequences...


Cheers

jtkleine

Just wanted to chime in, that I managed to open a file originating from 550D in Resolve. I did run raw2cdng under Parallels in OS X Lion, in 16bit mode of course. The file looked basically OK, but it was just something I shot in a minute. I couldn't do any actual testing, as the clip had horrific noise and no real black or white to begin with. Just wanted to see if it works at all. Sorry, I have not time to do anything more helpful right now.

BTW, I tried also in Speedgrade, just for fun. That didn't work at all (I didn't expect it to). The image was mostly green. I don't think Speedgrade even supports CDNG, maybe only Alexa and R3D.

Chmee you are doing as a big favor. No much point in grading raw video footage anywhere else than in Resolve -- at the moment at least. I tested earlier some clips in AE and graded them wit Color Finesse. That workflow didn't make much sense. I mean, to first bake the raw data into static rgb and then start grading. Very frustrating. So thanks, great stuff.

olik

Quote@olik
what version-number of .net-framework is running in mono? it needs 4.0

I have mono-runtime 2.10.8.1 but it seams that 4.0 is not even out, 3.0.10 was just released as a beta (http://www.go-mono.com/mono-downloads/download.html) am I missing sth? :)

chmee

@jtkleine - please leave a screenshot, what body you are using? maybe its a 12bit body? is some 12bit-Body able to raw-output? if you want to, just upload a short/small rawfile to analyse, whats happening.

http://dslr-kleinanzeigen.de/Tools/Body_Database/index.php?filter=prodeqcanon|rawbiteq12

@olik dont know much about mono - i just use linux-derivates for xbmc :)

generally, please wait till thursday, for now i'm working all day long - i cant do any changes.
[size=2]phreekz * blog * twitter[/size]

jtkleine

OK, I shoot a small clip and upload the raw file a little later. Where should I upload it? Maybe I just put in on a server and link it here. A screen shot too. But I don't have time today to fiddle with the black level settings in raw2cdng, so I just use the default.