The CinemaDNG Discussion (raw2cdng)

Started by chmee, May 23, 2013, 10:46:55 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

DaveAbbott

Amazing! Thanks Irri!

Now all I need to do is to solve my black point issue and I can delete 800GB of .TIFs and give Lightroom a breather.

Irri

Don't thank me; there are people doing fantastic work here, and I'm not one of them :)

Glad you got it working.

Oedipax

I can't get raw2cdng running under Mono on my Mac, either. I'm running the latest version of Mono (just installed from Macports). No luck in CrossOver, either.

chmee

@IliasG
after some thoughts and discussions i think, it makes sense to lower the bitamount without losing any informations - so i would give the logarithmic idea a try. The final challenge would be to get the rawdata in a 10bit@log space and compressing it - because we can :)

(small patchversion is on the way - just got a job today, so i think the "bigger" patches will be on weekend)
[size=2]phreekz * blog * twitter[/size]

squig

Quote from: Oedipax on June 06, 2013, 04:09:28 AM
I can't get raw2cdng running under Mono on my Mac, either. I'm running the latest version of Mono (just installed from Macports). No luck in CrossOver, either.

We won't have to wait much longer http://www.magiclantern.fm/forum/index.php?topic=6218.msg46920;topicseen#msg46920

scrax

Quote from: Oedipax on June 06, 2013, 04:09:28 AM
I can't get raw2cdng running under Mono on my Mac, either. I'm running the latest version of Mono (just installed from Macports). No luck in CrossOver, either.
Don't install with macport, use the precompiled binary from mono site, that worked for me. I've posted the link in raw2dng.app topic
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus | EOS 5D MLbeta5- EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX, PS, LR, RawTherapee, LightZone -no video experience-

IliasG

Quote from: chmee on June 06, 2013, 10:46:15 AM
@IliasG
after some thoughts and discussions i think, it makes sense to lower the bitamount without losing any informations - so i would give the logarithmic idea a try. The final challenge would be to get the rawdata in a 10bit@log space and compressing it - because we can :)

(small patchversion is on the way - just got a job today, so i think the "bigger" patches will be on weekend)

I was away from my PC last week (and I'll be away again) so I could not give any try on this but today I'll be at home ...

Just a summary of my thoughts for the log curve.

We can clip most of the unused "below black point" levels. Now it is around 2048/14bit, we can keep 64 or 32 of them (or less) if we want a footroom just in case something goes wrong with Black Point. and say 64 or 32 over the Black Point.

It's desirable to keep linearity at the darks.
So for 14to12bit the lowest (say 32 below black + 32 ) could be mapped 1:1 giving a 4X slope at the linear part and then start the log part.

For 14to10bit it is a bit more complicated as a 16X slope would needed for 1:1 mapping but this way we will have heavy compression at highlights. I am thinking of an intermediate solution, slope less than 8X ( 8X gives levels density equivalent to 13bits) and maybe use smaller linear part and less (or zero)"below black" levels.

Anyway we can try for a start with rec.709 curve (linear with slope 4.5 and then gamma=2.22)
http://www.poynton.com/notes/colour_and_gamma/GammaFAQ.html#gamma_correction

fatpig

@chmee:

As soon as you implement Commandline, I will be happy to include it in BATCHelor. :)
(it's sticky'd if you don't know it.)
Is it planned?

Still need a 5DII .RAW File? I have one.

danielcreed

don't know if this will help but it states here that black level is 1023 and white level is 15600

http://lclevy.free.fr/cr2/

as far as the 5d Mark 2 goes


a1ex

The values change depending on video mode; 1023 is just a rough starting point. On the same link, it says that black point is found by averaging the black borders from the image (which ML does).

IliasG

Quote from: IliasG on June 06, 2013, 03:09:11 PM
...

Anyway we can try for a start with rec.709 curve (linear with slope 4.5 and then gamma=2.22)
http://www.poynton.com/notes/colour_and_gamma/GammaFAQ.html#gamma_correction

Quick and dirty calculations in Excel spreadsheet for rec.709 curves.

Input is a typical Canon raw, 14bit linear. Clipped at -31 below the Black Level, mapped to 10bit gamma rec.709.

The inverse mapping (linearization table) which should be injected in exif tag is from 10bit gamma to 16bit linear with Black Level at 141/16bit.

http://rapidshare.com/files/3859944133/curves-14bit_to_10bit_gamma-rec709.xlsx

I changed slightly the offset to fight a discontinuity at the crosspoint.

Gives a good balance between the linear part and highlight compression. The levels's density at the darks is a bit better than 12bit linear while keeping highlight compression robust enough to withstand heavy grading (300 levels for the last stop).

hjfilmspeed

I know it would be an extra step but are there any dng to cinema dng converters?

zach915m

I was able to successfully convert a couple DNG files to 16 bit DNG and now it only does 14 bit files and then when trying to upconvert to 16 bit I get this error message:

"index was outside the bounds of the array"

Any idea how to fix this? 

Thanks for any help.

Zach

budafilms

RAWMAGIC VS RAW2DNG

Same shot: Build June 6 1920x720 25p no skipped frame. 5D MARK III. Scandisk 60 mb/s

- RAWMAGIC (beta4) open in DaVinci Resolve setting withe balance tungsten - exported as pro res HQ
- RAWTODNG (0.11) open in ACR setting tungten exported from compressor as pro res HQ

Nothing more applied, any corrections, all filters desactivated.

Imported both to FINAL CUT X and exported as h264 1920x1080

Facts:
- the more saturation and black levels in RESOLVE (appears to be sharpness).
- Magenta dots in RESOLVE (See the hair)
- blurry in ACR - maybe less black level

I really donĀ“t know if is RESOLVE whocreates de magenta o RAWMAGIC

The video to download:
https://www.dropbox.com/s/vjrfigp0xi4t2pi/prueba%20RAW%20MAGIC%20vs%20RAW2DNG.mov

chmee

just a short sign of life of me. just got the flu (in summer, hate it).
brain apparatus is on 30%-can and -10%-want-strength :)

(*) batchmode is coded
(*) most code is rewritten
(*) no more 14bit-output, because not needed
(*) blacklevel and whitelevel read from raw-data
(*) colormatrix read from raw-data

(+) sitting on 12bit linear mode - after finishing that, exe will be published

by now
(-) no adjustments to new stripesize-changes (extremly urgent on list)
(-) no 10bit logarithmic (urgent on todolist) (thx iliasg, appreciate your work) (think, we have to hide the 14bit in 16bit-words - we know, resolve cant handle 14bit)
(-) banding issue for 5diii-sources (urgent) (thx to thomas worth)
(-) pink fringes (investigating) (its a canon-kind problem - adobe is handling that manually so to say)
(-) commandline-mode (in one of the next versions)
(-) compression (will come after 10bit logarithmic)

@zach - two days ago, a1ex changed the bytestripelength - did you used some "abnormal" picture-sizes? have to update my 5diii with the new code to investigate..

regards chmee
[size=2]phreekz * blog * twitter[/size]

zach915m

Hey Chmee,

I believe it must have something to do with what Alex did to the bytestripelength as I tried this further today on two other systems and they both came up with the same error.  My only change in workflow was using the most recent version of Magic Lantern on my 5D Mark III instead of the late May version I was using before.  All footage was shot at 1920 by 1080.

Sorry to hear about you getting the flu that is bad news!  Feel better!

tin2tin

Quote from: chmee on June 10, 2013, 01:01:01 PM

(*) batchmode is coded
(*) most code is rewritten
(*) no more 14bit-output, because not needed
(*) blacklevel and whitelevel read from raw-data
(*) colormatrix read from raw-data

(+) sitting on 12bit linear mode - after finishing that, exe will be published
Great to hear you're making progress. Thank you for investing your time in this much needed tool. Now you have succeeded with the 14 to 12 bit conversion, I can't help my curiosity on the question of this process is too heavy to implement in camera or not(as discussed in the 12/10 bit conversion thread), but I guess it is premature to ask those questions.  ;)

chmee

http://www.phreekz.de/wordpress/2013/06/magiclantern-raw2cdng-cinema-dng/



version 1.0 is out - you know. its beta - and dont be too much optimistic :) the 12bit mode is built because of the easy way and a hope, that cutting the lsb 2 bits are not that visible :) the result is still far better than the "old" canon output.

@tin2tin:
the conversion from 14 to 12 bit is not that big work - but it needs some memory and some bitshifting and masking. my assumings are, if between sensor and dma-controller is no "processing-unit" able to handle that. hmm. the first approach was, to drive all the sensordata thru the digic - and this was the bottleneck.

regards chmee
[size=2]phreekz * blog * twitter[/size]

a1ex

Quotelower whitelevel by 15%

On what cameras is this needed? If raw2dng has pink highlights too, it's a bug in ML and you should report it.

chmee

so far i understand the pink highlights problem - its no problem of data handling but of managing the "beyond whitelevel sensels". the checkbutton is a fast built in tester if its changing something. if you search after "canon raw pink highlights" you (we) see, its a problem since years - manually managed by the raw-converters - and the dcraw-community is the only one trying to solve that by explaining the problem.

example links:

may 2012 - http://sourceforge.net/p/ufraw/discussion/434060/thread/b43582e3/
march 2009 - http://www.luminous-landscape.com/forum/index.php?topic=32942.0
feb 2010 - http://www.dpreview.com/forums/thread/2748822
oct 2011 - http://code.google.com/p/rawtherapee/issues/detail?id=1037
dec 2009 - http://www.flickr.com/groups/photomatix/discuss/72157623070526860/

regards chmee
[size=2]phreekz * blog * twitter[/size]

a1ex

In photo mode (CR2): white level depends on digital ISO, e.g. at 160, 320 and so on it's 1/3-stop lower than at full-stop ISO. Oddly enough, at ISO 250 it's not higher than at ISO 200.

In movie LiveView: digital ISO does not affect raw data, so the white level is always the same. The level from exif should be correct.

In photo LiveView, ML alters the white level to adjust the histogram so it matches the CR2 one. That's one of the reasons why you should not record raw video in photo mode.

chmee

Quote from: a1ex on June 11, 2013, 12:57:15 PM
.In photo LiveView, ML alters the white level to adjust the histogram so it matches the CR2 one. That's one of the reasons why you should not record raw video in photo mode.
So. meaning, in m and videomode it shouldnt happen?
[size=2]phreekz * blog * twitter[/size]

a1ex

M is photo mode, movie mode is movie mode. Don't mix them.

tin2tin

A quick test with a file from a 60D in Movie mode, a couple of ML builds back: http://we.tl/4Mv47vaiRS

There seems to be a bug in 12 bit + 15% unchecked which makes the screen black with green monocrome highlights.

chmee

the moviemode reacts on the mode-dial.. so, the only 100% functional way is to set on moviemode AND mode-dial on m.
now properly said?

@tin2tin investigating - made a dumb mistake (last minutes) lowering the blacklevel by 4 not by 2 should be the right way. on the way. thx for the raw.
[size=2]phreekz * blog * twitter[/size]