Author Topic: The CinemaDNG Discussion (raw2cdng)  (Read 921346 times)

chmee

  • Contributor
  • Hero Member
  • *****
  • Posts: 1031
  • raw2cdng
Re: The CinemaDNG Discussion (raw2cdng)
« Reply #800 on: May 09, 2014, 01:14:46 PM »
@artyg
please send me a short MLV example to raw2cdng@phreekz.de. actually i'm a bit confused because of the actual changes regarding the blacklevel. i didn't saw those problems with earlier ml-builds..

https://bitbucket.org/hudson/magic-lantern/pull-request/484/black-level-fix/diff

please @all
the actual ml-builds are quite new (especially regarding the blacklevel-things), some of them (1.2.3 for 5DIII) are bleeding edge. dont change your system if you want to work productive. if you need them, use an older build, something between ~feb09(mlv-audio) and ~apr27.. i am still testing my code with a 5DIII 1.1.3 build from ~apr20, because its workin' and stable. you are not bound to use the newest builds!

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: The CinemaDNG Discussion (raw2cdng)
« Reply #801 on: May 09, 2014, 05:49:24 PM »
The black level bug was introduced in February 11. I am currently waiting for code review for the fix before merging it.

If April builds are any better than May ones (especially regarding black level), please show exactly where the difference is (bug report).

reddeercity

  • Contributor
  • Hero Member
  • *****
  • Posts: 2293
Re: The CinemaDNG Discussion (raw2cdng)
« Reply #802 on: May 09, 2014, 09:52:11 PM »
dose this problem refer to 5D3 Only ? as there has never been a problem with 5d2
but I'm still on Feb16th NB, no problems.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: The CinemaDNG Discussion (raw2cdng)
« Reply #803 on: May 10, 2014, 07:28:07 AM »
There are more causes for this bug.

The one I've talked about earlier only affects 5D3 and 60D (it appears when changing video modes and recording right away, and I could reproduce it and fix it).

It can also appear if a pink (corrupted) frame appears right when ML is computing the black level (though this one is quite hard to reproduce).

It can also appear because of some Canon code overwriting the raw buffer - remember that raw mode is just a debugging flag written by Canon, but they didn't design it for everyday usage. For example, in 500D, if you take a picture in LiveView and hold the shutter button for a little longer, the raw buffer will be overwritten by LiveView data and the raw zebras will be wrong. This has nothing to do with raw video, but it's a test case that can be reproduced (very useful for testing the fix).

It may also appear because of race conditions (I've tried to avoid it and make the raw backend thread-safe, but I'm not an expert in multithreaded programming, so I may have missed something).

The workaround from MLV (which is enabled by default - not by me) is incorrect for 5D2 and many other cameras. The pull request removes the workaround and fixes the raw backend properly IMO. It checks the optical black area, so it should also detect bad black levels caused by a pink frame (though not with 100% chances).

On the pull request page, I've listed the relevant tests needed to make sure the fix is portable, and got zero feedback on these for about one week. I simply didn't have the time to do these early tests myself, and without code review and early feedback, I'm not going to include the fix in the nightly builds.

Jbowdach

  • Freshman
  • **
  • Posts: 74
  • Editor\Colorist for Cinetic Studios
Re: The CinemaDNG Discussion (raw2cdng)
« Reply #804 on: May 10, 2014, 09:47:45 PM »
Aside from the black level issue, I cant get the application to process MLVs from my 7D without it crashing. I've tried several sets of MLV files recorded at different times, on different cards, with different version of ML\TL. Is there anyway I can provide a log of source of diagnostic info for this? I have no other alternative mlv2cdng workflow, as this has worked so well

Many thanks!

chmee

  • Contributor
  • Hero Member
  • *****
  • Posts: 1031
  • raw2cdng
Re: The CinemaDNG Discussion (raw2cdng)
« Reply #805 on: May 10, 2014, 10:18:33 PM »
@jbowdach please be patient for 1 or 2 days, i will a provide a beta4 with more debug-infos (and some bugfixes)

Jbowdach

  • Freshman
  • **
  • Posts: 74
  • Editor\Colorist for Cinetic Studios
Re: The CinemaDNG Discussion (raw2cdng)
« Reply #806 on: May 11, 2014, 01:16:22 AM »
@jbowdach please be patient for 1 or 2 days, i will a provide a beta4 with more debug-infos (and some bugfixes)
@chmee many thanks, you are awesome! Please let me know how I can help provide any debug info

chmee

  • Contributor
  • Hero Member
  • *****
  • Posts: 1031
  • raw2cdng
Re: The CinemaDNG Discussion (raw2cdng)
« Reply #807 on: May 11, 2014, 10:52:53 PM »
first: @a1ex it was not my intention to leave bad words on testing and generally on your work - understanding bugs in my code with a glitch on ml-side makes it very difficult to find the problem. so please @all it was no call-up to not test new builds.. your tests are of course necessary.

beta4 is online
* raw-bug (i hope) fixed
* more templates for filename generating
* proxy jpegs
* more debug infos in log (debug-button on top)
http://www.phreekz.de/wordpress/2014/04/magiclantern-raw2cdng-1-5-0/

regards chmee

Jbowdach

  • Freshman
  • **
  • Posts: 74
  • Editor\Colorist for Cinetic Studios
Re: The CinemaDNG Discussion (raw2cdng)
« Reply #808 on: May 11, 2014, 11:21:42 PM »
first: @a1ex it was not my intention to leave bad words on testing and generally on your work - understanding bugs in my code with a glitch on ml-side makes it very difficult to find the problem. so please @all it was no call-up to not test new builds.. your tests are of course necessary.

beta4 is online
* raw-bug (i hope) fixed
* more templates for filename generating
* proxy jpegs
* more debug infos in log (debug-button on top)
http://www.phreekz.de/wordpress/2014/04/magiclantern-raw2cdng-1-5-0/

regards chmee

Thank very much, good sir! I am already testing it now to see if it fixes any of the crashing issues I had w beta3 and I have debug enabled so I can provide you debug info. Your work is VERY much appreciated as it's easily the best method for MLV -> DNG + WAV at the moment.

EDIT: Please find my debug log on pastebin below and issue files on bitbucket. It crashed as I was converted MLV files to 16bit CNDNGs (trying to avoid extra options so not using 12bit downconversion or any other options yet) -
Debug Log - http://bit.ly/1jw3160

reddeercity

  • Contributor
  • Hero Member
  • *****
  • Posts: 2293
Re: The CinemaDNG Discussion (raw2cdng)
« Reply #809 on: May 11, 2014, 11:24:05 PM »
Thank chmee, Just downloaded the new version (1.5.0 beta 4) and with give it a work out  :)
One question thou, Is there a way of getting the normal 16bit Cdng as in Version 1.4.9 ?
I can't edit the Cdngs or use them in any Apple produces, as there are all pink cast (for Adobe)
but I like the new version with the file name generator etc.. . I can use the older version it fine but just wondering .  :D

Jbowdach

  • Freshman
  • **
  • Posts: 74
  • Editor\Colorist for Cinetic Studios
Re: The CinemaDNG Discussion (raw2cdng)
« Reply #810 on: May 11, 2014, 11:44:29 PM »
Thank chmee, Just downloaded the new version (1.5.0 beta 4) and with give it a work out  :)
One question thou, Is there a way of getting the normal 16bit Cdng as in Version 1.4.9 ?
I can't edit the Cdngs or use them in any Apple produces, as there are all pink cast (for Adobe)
but I like the new version with the file name generator etc.. . I can use the older version it fine but just wondering .  :D

off topic real quick:  thanks for your videos online. Helped me get started w the workflow when VERY few mlv videos working w sync sound seem to be online

chmee

  • Contributor
  • Hero Member
  • *****
  • Posts: 1031
  • raw2cdng
Re: The CinemaDNG Discussion (raw2cdng)
« Reply #811 on: May 11, 2014, 11:59:01 PM »
@reddeercity  :o huh. didnt noticed such a glitch. please send me a wrong and a working dng. i'll do my best to get you into the position using the new gui :)

@jbowdach you as well, send me a short ( short! :) ) not functional mlv from your 7d to analyze it on my pc.

--> raw2cdng@phreekz.de

regards chmee

Jbowdach

  • Freshman
  • **
  • Posts: 74
  • Editor\Colorist for Cinetic Studios
Re: The CinemaDNG Discussion (raw2cdng)
« Reply #812 on: May 12, 2014, 02:21:54 AM »
@reddeercity  :o huh. didnt noticed such a glitch. please send me a wrong and a working dng. i'll do my best to get you into the position using the new gui :)

@jbowdach you as well, send me a short ( short! :) ) not functional mlv from your 7d to analyze it on my pc.

--> raw2cdng@phreekz.de

regards chmee

hmmm....short..let me see what i can find that errors out thats not huge.

EDIT: Okay, I have NOTHING thats not massive. GOnna shoot a test tonight thats not too big for you to analyze . Many thanks :)

EDIT 2: Now of course, all my very short clips are working. 5-13 seconds. Possibly something with large files and split M01s, etc?

reddeercity

  • Contributor
  • Hero Member
  • *****
  • Posts: 2293
Re: The CinemaDNG Discussion (raw2cdng)
« Reply #813 on: May 12, 2014, 05:55:28 AM »
Testing Beta4, Had crashes with jpeg proxy enabled & 16bit output or when I previewed then try to convert without proxy enabled crashed also
will test some more.
@chmee here is a screen shot of FCPX Right hand side is from the Beta 4 Raw2cdng & on the left is the 1.4.9 ver. raw2cdng.
http://500px.com/photo/70187941/fcpxca~-by-david-miazga?from=user_library

Jbowdach

  • Freshman
  • **
  • Posts: 74
  • Editor\Colorist for Cinetic Studios
Re: The CinemaDNG Discussion (raw2cdng)
« Reply #814 on: May 12, 2014, 06:06:27 PM »
Tried with 10 separate clips and couldn't get them to crash, OF COURSE....dont want to upload 4GB MLVs, so will try again in or day or so

chmee

  • Contributor
  • Hero Member
  • *****
  • Posts: 1031
  • raw2cdng
Re: The CinemaDNG Discussion (raw2cdng)
« Reply #815 on: May 12, 2014, 06:21:42 PM »
@jbowdach
my longest testfile is a 3strip-sequence. so, i will record a longer one today and do some tests. looking, if there are "out of memory" or some IO-exceptions doin this.

Jbowdach

  • Freshman
  • **
  • Posts: 74
  • Editor\Colorist for Cinetic Studios
Re: The CinemaDNG Discussion (raw2cdng)
« Reply #816 on: May 13, 2014, 06:47:22 AM »
thanks! Ive been trying to shoot full cards, so I 59GBs of MLVs + M001, M002, etcs. Seems be errors on various clips tho, not just REALLY long ones. my specific tests for you were all below 20 seconds tho

EDIT: Posted another log to bitbucket

artyg

  • New to the forum
  • *
  • Posts: 32
Re: The CinemaDNG Discussion (raw2cdng)
« Reply #817 on: May 13, 2014, 07:52:23 PM »
@artyg
please send me a short MLV example to raw2cdng@phreekz.de. actually i'm a bit confused because of the actual changes regarding the blacklevel. i didn't saw those problems with earlier ml-builds..

https://bitbucket.org/hudson/magic-lantern/pull-request/484/black-level-fix/diff

please @all
the actual ml-builds are quite new (especially regarding the blacklevel-things), some of them (1.2.3 for 5DIII) are bleeding edge. dont change your system if you want to work productive. if you need them, use an older build, something between ~feb09(mlv-audio) and ~apr27.. i am still testing my code with a 5DIII 1.1.3 build from ~apr20, because its workin' and stable. you are not bound to use the newest builds!

I downgrade my 5D3 firmware to 1.1.3 and install ML 24Apr. And pink highlites stay... Then I run beta4 in CDNG 12bit (not maximezed) mode and programm work correct! No pink. I test my old files, and they all convert correct! So, I think 1.2.3 firmware work correct too... But some strange things with 1.2.3 happen. Sometimes 5D3 does not turn on!.. I decided to stay in 1.1.3 - this firmware more stable. Thanks, chmee!

So, one bug i find. In 12bit maximized mode pink highlights appear... (post on bitbucket)
Canon 5D3 SanDisk Extreme Pro 64Gb 160MB/s

chmee

  • Contributor
  • Hero Member
  • *****
  • Posts: 1031
  • raw2cdng
Re: The CinemaDNG Discussion (raw2cdng)
« Reply #818 on: May 15, 2014, 08:13:54 AM »
please @all i need more testfiles from other bodies than mine (5DIII 1.1.3).

(*) do a short (max 100MB) mlv or raw record, 10 frames
(*) with high dynamics, it should have overexposed and dark areas as well
(*) some greyish things and some colored (colorchecker, if you have)
(*) some kind of testpattern (search via google)?
(*) write down build date (checksum?) and body with canon-firmwareversion
(*) the files will be collected and released/listed on a site

if you want, you can send them to me (raw2cdng@phreekz.de)

regards chmee

@artyg 12bit pinkhighlights, yes i know. thats the reason i ask for more videofiles from other bodies.

ted ramasola

  • Moderators
  • Hero Member
  • *****
  • Posts: 1250
Re: The CinemaDNG Discussion (raw2cdng)
« Reply #819 on: May 15, 2014, 08:17:24 AM »
Do you need a mkII file?

What setting should the BLACK FIX be ON or OFF ?
5DmkII  / 7D
www.ramasolaproductions.com
Texas

chmee

  • Contributor
  • Hero Member
  • *****
  • Posts: 1031
  • raw2cdng
Re: The CinemaDNG Discussion (raw2cdng)
« Reply #820 on: May 15, 2014, 08:56:23 AM »
yes, 5DII as well. best is to deliver both and name them.

ted ramasola

  • Moderators
  • Hero Member
  • *****
  • Posts: 1250
Re: The CinemaDNG Discussion (raw2cdng)
« Reply #821 on: May 16, 2014, 09:08:23 AM »
Sent you 4 mlvs from 5D2 and 7D to raw2cdng@phreekz.de

2 settings for each cam. With black fix ON and OFF.

Used ML nightly may 14 build for both cams.
5DmkII  / 7D
www.ramasolaproductions.com
Texas

chmee

  • Contributor
  • Hero Member
  • *****
  • Posts: 1031
  • raw2cdng
Re: The CinemaDNG Discussion (raw2cdng)
« Reply #822 on: May 16, 2014, 09:48:58 AM »
i'm already downloading. thanks.

chmee

  • Contributor
  • Hero Member
  • *****
  • Posts: 1031
  • raw2cdng
Re: The CinemaDNG Discussion (raw2cdng)
« Reply #823 on: May 18, 2014, 09:59:09 PM »
raw2cdng 1.5.0 beta5 online

* stabilizing code (memory leak if bigger files fixed, memory leak if preview was used fixed)
* proxy jpgs now with gamma 2.0
* busy-progressbar while reading data from drag/dropped files

http://www.phreekz.de/wordpress/2014/04/magiclantern-raw2cdng-1-5-0/

now there are two important things for me:
* @reddeercity please send me (as written above) these two different dng's - want to fix that.
* 12bit output isnt fine at all - have to fix that.

regards chmee

Jbowdach

  • Freshman
  • **
  • Posts: 74
  • Editor\Colorist for Cinetic Studios
Re: The CinemaDNG Discussion (raw2cdng)
« Reply #824 on: May 19, 2014, 07:22:58 AM »
many thanks, will test and report back