5D MK3: Problem with red channel in RAW files getting GREEN CLIPS (Dec builds)

Started by carlballou, January 08, 2014, 06:30:31 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

carlballou

I have searched google and the forum here, and unable to find any similar problems. This is randomly happening with two different December builds with the 5d mk3 on two different camera bodies. The raw video has a problem with the RED channel that is baked into the raw data images on a few random raw clips.

The workflow is as follows:

1) ML correctly installed to SD card.
2) Recording to Lexar 1000x 128gb and 64gb CF cards, formatted Exfat with OSX disk utility.
3) Offloading with Pomfort's Cliphouse to backup drives.
4) Converting with RAWmagic to Cinema DNG.
5) Grading in DaVinci Resolve to Prores....

I have been shooting with ML on the mk3 for a while now with no problems other than the usual, decided to shoot a short film and now I am having this issue. Day 1 we had this issue on B-camera, 1 random clip per mag. Day 2 it appeared on A-camera, 2 random clips the last mag of the day. Day 3 not present at all. Day 4 and 5 to follow starting tomorrow.

It appears the red channel is crushed and "baked" into the raw data, causing the clip to look green in the shadows and extremely crushed. There are a few reference screen shots attached for one example of this issue. One is showing the flat raw image brought into Resolve, versus the standard profile applied by Cliphouse and the 3rd is the problem clip showing the green channel shift. There is also a comparison of the histogram and scopes for a normal raw clip vs the issue attached. Both clips are shot with the exact same camera settings.

This issue occurs on RAW clips on the card itself, so Cliphouse and the copied files are not the issue. Also, with clips converted with RAWmagic vs cinema2dng still present in addition to the .raw files. I have downloaded 2 RAW utilities called "RawTherapee" and "RawDigger" to further analyze the EXIF data to see if there is a correction that can be made and applied to all images in the raw sequence. Maybe it is possible that ML is not closing the clips correctly after recording??? I will post any updates as they become available. Just wondering if anyone else is experiencing this issue....








NedB

@carlballou: Take a look at http://www.magiclantern.fm/forum/index.php?topic=7122.msg93340#msg93340. The .dng's I was looking at looked very similar (crushed blacks, green tint) to those from your "bad" clip.

Try the fix I suggested, I think it very well may work. Don't know why it happens, but if the fix is repeatable and makes sense, then this is a good workaround. Once I adjusted the "Exif:BlackLevel" tag, the resulting (fixed) .dng's. looked exactly like .dng's from my "good" clips. Hope this helps. Cheers!
550D - Kit Lens | EF 50mm f/1.8 | Zacuto Z-Finder Pro 2.5x | SanDisk ExtremePro 95mb/s | Tascam DR-100MkII

a1ex


NedB

@a1ex: The essence of this problem (if it is the same one I had a week or so ago) is that it seems to be unpredictable. Nevertheless, perhaps there is a clue in the fact that both of us (that is, @carlballou and I) seem to have "good" clips to compare the "bad" one(s) to. This means that we were both shooting several takes of the same scene. Perhaps there is a time between takes which we should respect and failed to (like stopping recording of an .mlv clip and hardly waiting at all before hitting the record button again, or maybe playing back the .mlv we just shot, then stopping the playback (or not?) with a half-shutter, then starting recording again (while the playback "stopping" is maybe still going on in the background))?

In any case, when I compared a typical .dng from a "good" clip with one from a "bad" clip, using exiftool, the only difference I found was in the aforementioned "Exif:BlackLevel" tag. If @carlballou is able to fix his problem clip with the same procedure, that would indicate that the tag is intermittently being written wrongly, OR that the mlv_dump.exe tool sees something different in the "bad" clip which doesn't allow it to save .dng's with the correct black level.

Let's wait and see if @carlballou's .dng's can be fixed with the same (or similar) fix. If he also finds a "wrong" black level of (I think it was) 2590, instead of the correct level (2046 IIRC), then we will know more.

In the meantime thank you for all you do for Magic Lantern and its users. Cheers!
550D - Kit Lens | EF 50mm f/1.8 | Zacuto Z-Finder Pro 2.5x | SanDisk ExtremePro 95mb/s | Tascam DR-100MkII

a1ex

I'm 100% sure the black level will fix it.

ML omputes it on the fly, during standby periods (not while recording), from the left optical black bar (you can see this area in silent pics with "dcraw -4 -E foobar.dng", but it's not recorded in raw videos for speed reasons). Possible cases when it can go wrong:

1) the raw frame gets corrupted like this: http://www.magiclantern.fm/forum/index.php?topic=8190.0

When this happens, the black bar area will contain useful image data => the computed black level will be always higher than the correct one.

2) there are some hot pixels in the black area (unlikely IMO)

3) the black level is updated while changing resolution / switching LV modes, when the raw resolution from ML variables does not match the contents of the buffer (race condition)

Some statistical tests might help detecting this condition in post, and maybe even fixing most of it.

Note: in both raw_rec and mlv_rec, the black level (and all other raw parameters) are recomputed right before recording. So, I would rule out 3, because that would result in completely corrupted videos.

Do you have the memory hack enabled?

NedB

@a1ex. Great news.

1) Didn't notice 1st frame corruption in my "bad" clip, so I can't say whether this case applies.
2) Seems like this would cause the problem to appear much more often than it does. I've only seen it in one clip, ever.
3) If we're ruling out 2 and 3, then the problem is probably 1, right? One "good" result of this is that it only affects metadata, and can be fixed in post.

My "bad" clip was shot on 12/24 and unfortunately I do not remember if the memory hack was enabled. To which exact menu item in which ML menu are you referring?

Statistics: How much does the [Exif:BlackLevel] tag vary? Is it always near to 2048 or so, or does it have great variations? If the correct value is always near 2048 (or perhaps near some calculable combination of other parameters like ISO, etc.), then maybe there could be a test which compares this calculation to the "expected" value and offers a choice, maybe right in mlv_dump, to correct the tag.
550D - Kit Lens | EF 50mm f/1.8 | Zacuto Z-Finder Pro 2.5x | SanDisk ExtremePro 95mb/s | Tascam DR-100MkII

a1ex

That first frame is not recorded.

I believe there is a feedback loop on the sensor that looks at the optical black area and tries to keep the black level at some fixed value. Look in the dual ISO PDF and in the ADTG chip datasheet (referenced there).

This value can be 2048, 1024, 1700something and probably others (it's camera-specific and sometimes video mode-specific). On 5D3 I think you can assume it's always 2048.

Creating a table of usual values for all cameras and hardcoding it may help, but I prefer to use portable code without camera-specific tweaking whenever possible. So, I'd try to identify the actual issue and fix it instead.

Also look at g3gg0's experiment from here: http://www.magiclantern.fm/forum/index.php?topic=9861

You can notice the black level varies inside the frame; I think it's possible to estimate it somehow from the raw data itself. Notice the black level variation on columns and lines is nothing else than the well-known fixed pattern noise (some call it banding).

NedB

I will look at the two documents you referenced.

I agree the issue should be resolved correctly and fixed. OTOH, the "bad" black level in my bad clip was 2590, which wouldn't appear anywhere on this (short) list, presumably.

If the problem really only occurs every hundred (or more) clips, maybe it's not worth fixing in ML or in camera. Especially if simply changing the exif tag in post fixes it completely.

Waiting to see what @carlballou has to say, and what his "bad" black level is, and if others chime in with the same problem(s), which is (are) also easily fixed.

As for estimating the black level from the raw frame itself, certainly the bad .dng's have a certain "look" which sets them apart: crushed blacks and an overall green tint. Seems like it would be easy to check for both of these in post-processing and offer a suggested fix. But maybe the easiest "check" is just our two eyes when we review the .dng's.

Again, ATM it doesn't seem necessary to insure that ML always writes the correct black level, if it is getting it right 99+% of the time. Right?
550D - Kit Lens | EF 50mm f/1.8 | Zacuto Z-Finder Pro 2.5x | SanDisk ExtremePro 95mb/s | Tascam DR-100MkII

carlballou

Hey guys,

I have not been able to reproduce the issue again in the last two days. It seems to me that the issue was completely random on the clips affected.

By the way, we are using a Tv Logic 5.6in for HDMI monitoring w/ SDI loop out. The image frequently freezes on standby but is quickly fixed by pressing the zoom button.

@a1ex - The memory hack is set to OFF, and we are shooting 1920x1038 (1.85:1)

@NedB - are you using a windows machine? I am unable to get "exiftoolGUI" working correctly on OSX, even with the ported version here: http://hvdwolf.github.io/pyExifToolGUI/

When I load the .RAW file, all the tabs and fields are empty. Trying to load a DNG crashes the program..... but the program does work for still images such as a CR2 from the 5D.

Any other tool I found for OSX that claims to be able to edit and view EXIF data doesn't work correctly. I'm running 10.8.5 here.

Does anyone know a reliable way to view/edit the EXIF data on a mac and apply batch corrections to the .RAW video file or DNG folder?

NedB

@carlballou: Yes, I'm on Windows and Linux. If you want to send me one of the .dng's from the "bad" clip, along with another from one of the good clips (another take from the same scene), I'll take a look at the tags and let you know what I find. I don't know offhand of a way to get exiftool to work on a Mac.

Also, I don't think you should expect exiftool, if it were to work on Mac, to be able to open or process a .raw or .mlv file. You have to extract the .dng's first. Not sure why the ported GUI isn't working for you.

If you do send the .dng's, I'll get back to you tomorrow, as it's already 1am here (Germany). Cheers.
550D - Kit Lens | EF 50mm f/1.8 | Zacuto Z-Finder Pro 2.5x | SanDisk ExtremePro 95mb/s | Tascam DR-100MkII

carlballou

http://we.tl/04xfemxpF6

Here is a link if anyone else wants to compare. It is wetransfer and will expire next week but.... 1500 is the "bad" file and 1503 is the normal one.

I also just noticed the "bad" file DNG's are looking normal in OSX Preview or pressing spacebar in Finder, but corrupt and green in Resolve, Cliphouse, After Effects, etc...

carlballou

So far in OSX... I am able to make some extreme corrections with the RawTherapee application to the raw white/black points. I did it by eye to get a "normal" looking raw histogram.

See the screenshots below on another example clip that I have with the same issue as the other.

This may be the solution I end up using, as I only have 5 total shots with this problem. But I would still like to find an answer eventually on exactly how far off the black points are versus the normal clips and why this problem even happened in the first place.




NedB

@carlballou: Naaahh, you don't have to use RT at all. Check your mail, I applied the fix (changing the exif:BlackLevel tag to the correct value) to the "bad" .dng you sent me and it is fixed. This is really only a metadata problem, that is, your data is absolutely ok.

For anyone else with a random clip now and then which has a green tint and crushed blacks: (as I've said before) just use exiftool/exiftoolGUI, and modify the value of the exif:BlackLevel tag. For the new value, use the value of the same tag present in a .dng which looks correct and was shot in more or less the same conditions.

Cheers!
550D - Kit Lens | EF 50mm f/1.8 | Zacuto Z-Finder Pro 2.5x | SanDisk ExtremePro 95mb/s | Tascam DR-100MkII

a1ex

I found a really stupid bug that might be related to this (black level was recomputed continuously). Here's a quick fix, not really tested:


diff -r 436d9c773740 src/raw.c
--- a/src/raw.c Fri Jan 10 10:18:38 2014 +0200
+++ b/src/raw.c Fri Jan 10 16:24:20 2014 +0200
@@ -325,10 +325,13 @@

static int autodetect_black_level(int* black_mean, int* black_stdev);
static int compute_dynamic_range(int black_mean, int black_stdev, int white_level);
-static int autodetect_white_level();
+static int autodetect_white_level(int initial_guess);

int raw_update_params()
{
+    //~ static int k = 0;
+    //~ bmp_printf(FONT_MED, 250, 100, "raw update %d called from %s ", k++, get_task_name_from_id(get_current_task()));
+
     #ifdef RAW_DEBUG
     console_show();
     #endif
@@ -637,21 +640,27 @@
         prev_delta_x = delta_x;
         prev_delta_y = delta_y;
     }
-   
+
+    /* black and white autodetection are time-consuming */
+    /* only refresh once per second or if dirty, but never while recording */
+    static int bw_aux = INT_MIN;
+    int recompute_black_and_white = !recording && (dirty || should_run_polling_action(1000, &bw_aux));
+
     if (dirty)
     {
         raw_set_geometry(width, height, skip_left, skip_right, skip_top, skip_bottom);
         dirty = 0;
     }

+    if (!recompute_black_and_white)
+    {
+        /* keep the old values */
+        return 1;
+    }
+
+    int black_mean, black_stdev;
     raw_info.white_level = WHITE_LEVEL;
-
-    static int black_mean, black_stdev;
-
-    int black_aux = INT_MIN;
-    if (!lv || dirty || should_run_polling_action(1000, &black_aux))
-        raw_info.black_level = autodetect_black_level(&black_mean, &black_stdev);
-       
+    raw_info.black_level = autodetect_black_level(&black_mean, &black_stdev);

     if (!lv)
     {
@@ -1091,6 +1100,9 @@

static int autodetect_black_level(int* black_mean, int* black_stdev)
{
+    //~ static int k = 0;
+    //~ bmp_printf(FONT_MED, 250, 50, "black refresh: %d ", k++);
+   
     /* also handle black level for dual ISO */
     int mean1 = 0;
     int stdev1 = 0;


I expect it to give a little speed boost when running with global draw on, but I didn't run any comparison.

carlballou

This problem has yet to occur again after updating to a January build.

FYI: If anyone else has this problem in the future, I was able to successfully fix the black level on all the affected clips using EXIFTOOLGUI on a Windows laptop to batch process the DNG's to "2047" for my clips.

Just select the files, add the black level to be modified and click the current value. At the bottom, enter the correct value and the line becomes yellow. Then you must press the Enter key to enable the change.

I was also unable to batch process more than 1,000 frames at once. So a couple batches were required for a few of the shots. The program is kind of a pain but it worked.

mhimhi


jgerstel

Got the same issue, green images, it only happens when shooting in cropped mode and changing FPS a lot, like 12 FPS. The blacklevel in my DNG's were 2470 (no this is not a typo!)

The way I solved it is using cmd tool, example:

open terminal

studios-mac-pro:M08-1453 studio$ cd /Volumes/EOS_DIGITAL/DCIM/100EOS5D/M08-1453
studios-mac-pro:M08-1453 studio$ exiftool -Blacklevel=2047 *.dng

Hope this helps

Dotzor

Quote from: jgerstel on June 09, 2014, 12:01:13 PM
Got the same issue, green images, it only happens when shooting in cropped mode and changing FPS a lot, like 12 FPS. The blacklevel in my DNG's were 2470 (no this is not a typo!)

The way I solved it is using cmd tool, example:

open terminal

studios-mac-pro:M08-1453 studio$ cd /Volumes/EOS_DIGITAL/DCIM/100EOS5D/M08-1453
studios-mac-pro:M08-1453 studio$ exiftool -Blacklevel=2047 *.dng

Hope this helps

I tried this, I copied your code and changed to path to match my path but terminal gave me the 'this is an unknown command'...
are these two separate lines right?
And if it works should i do this for every single frame? because that is going to take a while...

https://forums.adobe.com/message/6587847#6587847

Satis

Quote from: Dotzor on July 27, 2014, 08:05:28 PM
I tried this, I copied your code and changed to path to match my path but terminal gave me the 'this is an unknown command'...
are these two separate lines right?
And if it works should i do this for every single frame? because that is going to take a while...

https://forums.adobe.com/message/6587847#6587847
No man, you have to have the exiftool.exe (under mac it could be slightly different) in the same folder, because what you do here is calling this command by
exiftool , the -Blacklevel=2047 is it's parameter, which means it's going to set the black level 2047 to *.dng, as every single file with dng file extensions (that's what * does, selets every file, the .dng part specifies it to dngs), so you don't need to mess around those files one by one individually.

Also, I had this problem myself, thanks for this thread guys, I managed to fix it!
50D IR, 600D, 550D

patpat74

Hi!
It seems that I have quite the same problem, but the 2047 blacklevel trick just give me a pink image... :-(
May somneone help me ? (i just understood how to fix the vertical banding, now i have thoses green/black images...new day new fight!!)

I don't know coding... I0'mn on W7 64bits....  I don't even know how to put an image on this forum ;-)

Thanks ! and thanks to ML.... On the shootings without bugs, it's so great... :-)

The_big_S

Hi all,

so I've encountered the same problem. I've changed the black point vaules for the DNG's in exiftool, but that doesn't seem to have worked. Has anyone got to that point and had to find another fix?



dmilligan


The_big_S

Ok, sample DNG can be downloaded from here: http://we.tl/aGmIpWeITl

Any help really appreciated, thanks so much!

dmilligan

That DNG is very strange. What program did you use to generate it? The WhiteLevel is 46480 which is larger than the largest value you can encode in 14 bits (16384). The image data appears to have been manipulated in some strange manner. There are large areas that are crushed to complete black (in other words 0), which would never happen. The lowest values you'll see should be around the blacklevel (or 2048), with the exception of a few bad pixels here and there.

The_big_S

I used raw2cdng 1.6.5

I also initially used RAWanizer to create cineform files and all the proxies had the same issue.

It only seemed to occur when shooting on cropped sensor. Was using Nightly.2014May21.5D3113

Does this mean the information is not present in the blacks?