New vertical line issue? 1_2_3 build.

Started by giarcpnw, April 24, 2014, 10:12:29 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

giarcpnw

Hey all, not sure if this is a processing issue or a ML issue.
Canon 5d Mark iii. April 16th 1_2_3 build. I was using the x3 crop function to shoot some owls. All the x3 crop stuff looks killer. I turn it off and shot normal and I got these tiny vertical lines in the "bright" section of the image.
It may be in the dark parts but I can't see it. I cranked the contrast and darkness in the frames so it sticks out more but i tried grading this and it ends up looking like compression artifacts in the video once exported to ProRes. Any ideas? It did not appear to be the "banding" issues i can find by searching. Two stills attached. Again, x3 crop looked fine. regular 1920x1080 raw resulted in the lines. I used Raw Magic to process these to DNGs

Look for the lines in the blue sky
https://www.dropbox.com/s/zqkq081quaamwg4/LinesRAWMarkiiiML1_2_3.jpg

No lines in x3 crop version
https://www.dropbox.com/s/jc7q91ryn0q4dka/NolinesX3cropRAWMarkiiiML1_2_3.jpg


Thanks,
Craig

a1ex

Can you upload a DNG processed with raw2dng or mlv_dump?

giarcpnw

I can not unfortunately. I dumped the RAW file after converting to DNGs with RawMagic. Is raw2dng the more preferred app to use?
I can say that RawMagic never gave me an issue until this build of ML and or using the x3 crop option.

I'll see if I can replicate the issue with some new footage and convert using both applications and post the results.

Craig


a1ex

It's the one I can use for troubleshooting.

giarcpnw

Ok,

I replicated it last night. I processed the raw files with the newest RawMagic updated this week) and the 013 version of Raw2dng. (newest i could find for the mac)
I got interesting results.

RawMagic creates lines on the DNGs and raw2dng creates a subtler matrix looking grid. They are hard to see until you grade and then compress and then it looks like weird compression artifacts. But if you crank the sharpness and contrast on the dngs (included in the link) in ACR you can plainly see both on each dng. I was unfortunately able to do this on the x3 crop video also I shot last night.
I looked back at tiff sequences i shot when i had the 1.1.3 version of ML on the camera and i can't see the patterns. That doesn't mean they weren't there, just not as prevalent perhaps. See what you think and if it warrants more investigation i'll be happy to shoot some more tests for you. Just tell me some parameters. I'm curious if this is camera firmware, ML or the processing. Or all three perhaps. Hopefully fixable.

Hopefully i won't have to go back the the 1.1.3 firmware. As it is now this footage would not pass muster viewed much larger than on a computer screen.


This link is a drop box folder with the two same dngs and two tiffs processed from both apps.
https://www.dropbox.com/sh/kyvrl00j8t5detb/Q727jX9-7d#/



a1ex

Can you show the conversion log from raw2dng?

I doubt it has anything to do with Canon firmware version (more likely the algorithm got confused by the scene you were shooting).


giarcpnw

For what it's worth MLVdump 1.0 creates the same matrix style pattern on the DNGs.


a1ex

The log you posted is from ffmpeg. It looks like you used a GUI version of raw2dng (not mine), but I don't know how updated it is, or how to find the log.

You can run the Windows version in the terminal under Wine, and paste the output. It's a troubleshooting step, after all (I want to find out if it's caused by my algorithm or it was present in the raw data, since I couldn't draw a clear conclusion from the current info).

The matrix pattern looks more like the image requires green equilibration when debayering. RawTherapee, Darktable, ufraw and dcraw all have options for correcting it; in ufraw it's called VNG four color interpolation, in RawTherapee it's Demosaicing->Preprocessing->Green equilibration. I'm not sure how exactly it's called in other programs.

More info about this artifact: http://www.pages.drexel.edu/~par24/rawhistogram/40D_Demosaicing/40D_DemosaicingArtifacts.html

giarcpnw

Ok, i'll try on a PC on Monday. No windows machines at the Casa.

So this would be a step to run the raw files through BEFORE exporting them into DNGs? Is this a wide spread problem with Debayering?

I pumped a few out with MLRawViewer and I thought we had a winner but the last clip of 4 introduced the lines again. Could this be an ffmpeg thing?
It just wasn't doing this three weeks ago.

I'll get back to you on monday with the PC raw2dng log.

Thanks,

C




giarcpnw

Ok,

Just cranked out the two raw files through the latest (that i could find) raw2dng.exe on the windows side.

Interesting results. Again the regular RAW mode frame has lost the lines but still exhibits a "matrix" style grid in the sky blue. (1st link) This was the same as what happened when using the raw2dng GUI mac version.  While the x3 cropped mode version has beautiful random pixels that looks like film grain when sharpened. (2nd link)
This happened to some degree (either lines or a grid) in all converters. RawMagic. Mac GUI of raw2dng and also MLV dump and MLRawViewer.

See here. Crank the sharpness in Adobe ACR to see the pattern.
https://www.dropbox.com/s/hzbd1i0ybve8pc4/windowsR2D_grid_000261.dng

https://www.dropbox.com/s/8686cn6l2nyhrq1/windowsR2D_x3_nogrid_000277.dng

The terminal goes away when after running the raw2dnge.exe process so i could not copy any "log" info or... if it's dumps a log file somewhere just need to tell me where to find it. It wasn't in the destination folder with the DNGs


I guess my and others concerns would be: This seems to be a semi repeatable an issue but also random. Seems to happen on some shots but not others. Seems to happen more in normal Raw and MLV modes but not as much in the x3 crop mode. Where is it happening and how do we fix? Is it just my camera? Is it a 1.2.3 thing? I don't recall it happening under 1.1.3 or not that I noticed.
If it's a debayering algorithm issue, is there a way to tell a converter what versions to use. AHD, PPG ,VNG, Bilinear etc.

I don't know. I'm an end user but i'm willing to help you out running more trials if you need them as I really want this to succeed. You've really created a kick ass mod.

Let me know what you need further Alex.

Thanks



a1ex

Copy what raw2dng prints on the console (that one is the log).

giarcpnw

Ah, clearly i'm an apple fanboy. I needed a co-workers help to run this from the console. This is all it gave me back.


Resolution  : 1920 x 1080
Frames      : 360
Frame size  : 3629056 bytes
FPS         : 23.976
Black level : 2047
White level : 15000
Processing frame 1 of 360...
Vertical stripes correction:
  1.000  1.000  0.992  0.993  0.998  0.997  0.992  0.983
Cold pixels : 0
Processing frame 360 of 360...
Done.

To convert to jpg, you can try:
    ufraw-batch --out-type=jpg *.dng

To get a mjpeg video:
    ffmpeg -i %6d.jpg -vcodec mjpeg -qscale 1 video.avi

a1ex

Alright. Can you also try with this version and post the output DNG?

http://acoutts.com/a1ex/raw2dng_totallyraw.exe

giarcpnw


a1ex

Since this is called "totally raw", the vertical lines were there in the raw sensor data. The algorithm from raw2dng tried to correct the issue, and almost succeeded. The maze artifacts are probably a sign that the correction could be improved, but can't tell for sure yet.

Will keep this sample as reference, and when I'll revisit the algorithm I'll try to fix it. Until then, green equilibration is your friend.

giarcpnw

Ok thanks,

Final question till this gets fixed. What app can process a ML raw file for green equilibration? Or is it done in batch to the DNG?

The apps i could find that were mentioned in the link you posted a few days ago are meant for still raw files. And some a few years before RAW video was around.

I may go back to 1.1.3 and see if this is still an issue.

giarcpnw

Maybe i found the answer myself.  Looks like Raw Therapee and apply setting to all DNGs like in Adobe ACR.

I will wrestle with this and post results for others.

http://www.magiclantern.fm/forum/index.php?topic=9836.0

Thanks

giarcpnw

Using RawTherapee you have to apply the "Line Noise" slider under the Demoisaic tab. I had to crank it to about 200-250 to get rid of them.
I also toggle between the "ammaze" or VNG4 demoisaic algorithms to get rid of the matrix pattern if it appears instead of the lines. Only bummer is you have to output to 16bit Tiffs and that is x3 larger than the DNGs.

A1ex,  what could be causing this to only be happening on the full frame shots but not on the x3 crop mode? I find that telling of something. Not sure what though. And what exactly is "Line Noise" since RT has a slider to reduce it. What could cause it not to be there under ML on 1.1.3? I seem to be the only one dealing with this at the moment so I realize it's not a big issue. I bet it's going to pop up more though as people notice it.


a1ex

Upload the exact same scene shot on 1.1.3 with the exact same settings.

giarcpnw

That was my next step. But it's going to take a few days.  ;)
I might try a newer build perhaps of 1.2.3 and see if anything changes first but I shot a lot of raw under 1.1.3 and never saw lines.

DNGs pending......

a1ex

I hope you realize that GUI code tweaks are not going to affect the image quality in any way.

Raw is raw, whether is labeled RAW or MLV. The build date doesn't matter when it comes to image quality (noise, banding, bad pixels and such).

The only exceptions are changing raw type (did that in the very early builds from last summer) and altering ISO registers (only if you have used any tools from the ISO research thread). The differences you see are from the postprocessing tools - some of them attempt to fix these issues, but these tools have nothing to do with the build you use.

giarcpnw

Ok, I get that. So.. playing devils advocate.

Why now and not on my previous raw files (1.1.3_ which I will verify soon)?
And why on the full frame but not on the x3 crop mode?
And if it is post processing.....could something have changed in the 1.2.3 firmware that has an affect on the raw files (meta data or wrapper) that could cause post processing to cause this now... but not in earlier builds?

I swear it just isn't in my earlier raw files from 1.1.3.  I just went back and loaded tiffs from various shots into Pshop and cranked sharpness and the noise patterns are just random and grain like.

I also never shot x3 crop mode until 1.2.3. Could that mode have some affect on how the sensor translates data once it has been used.

I realize there are just too many variables that are unaccounted for. I'm just trying to grab a tangible thread to start narrowing it down, or at least giving you enough data to start narrowing it down.

Let me try the april 24th build of 1.2.3. tonight and see if anything changes. If not I'll go back to 1.1.3 and the last build I used and see what happens.

I'm trying to nail this down so i can shoot a short we have planned. I'm starting to regret having messed with the camera when it was working three weeks ago just fine. Oy!




a1ex

Again. For an apples-to-apples comparison, you need to shoot the exact same thing, with the exact same settings.

giarcpnw

Ok, I have to eat crow. I was wrong....about 1.1.3 and the noise. The lines did show up..... :P

But! I'm not totally wrong about a couple issues. The lines ONLY show up using MLRawViewer to process DNGs. Which is a bummer since it's such a robust app. This might explain why I thought it was only happening after I updated to 1.2.3. I was not using that converter before that. The only way to get rid of these lines is to use RawTherapee and apply the "line noise" function. Maybe other apps but I don't know them.

RawMagic, raw2dng (mac GUI version) and possibly mlvdump (have to test again, lost track of test MLVs) give the mild matrix pattern on DNGs under 1.1.3. Each app a little variation in amount. Easily fixable in Adobe ACR. Apply a mild 8-17 setting on "masking" under the sharpening tab. Will render it gone in most cases. Especially in large swaths of the background (sky, blank walls etc.) Clouds did throw it for a loop. Hence, up to the 17 setting.

I got these same results under the April 24th 1.2.3 nightly build also before I dropped down to the April 29th 1.1.3 nightly. So, I was INCORRECT in assuming it was firmware related. (I will chalk it up to ignorance rather than stupidity since i'm an end user not a programmer) But, I did have lines randomly under older 1.2.3 builds. I have to go look at what version I used. Look at my earlier DNG posts on this thread. One is a Raw Magic and one is a raw2dng I believe. I can not explain why i got the lines on those conversions. I was not able to repeat it.

The only issue that hasn't been explained yet (cough cough) is why this does not happen to videos captured in the x3 crop mode? x3 videos I captured have a very natural noise pattern that is very grain like and barely detectable. This result was had on All ML builds under both 1.1.3 and 1.2.3 Even using MLRawViewer. Can that be explained and can whatever is making it work be applied to Raw data coming from the Full Frame capture? Would greatly expand pushing sharpness and color grading ability raw video for post. I discovered all this after converting to ProRes and then H.264 for Vimeo. These patterns look like wreak havoc once compressed. They look like tears and compression artifacts in end videos.

All DNGS from the 1.1.3 test are below.

https://www.dropbox.com/s/uba3yzpez5m1j1s/M01-1847_Nolines_RawMagic_113.dng

https://www.dropbox.com/s/v7xjv29eip1vl1p/M01-1847_nolines_raw2dngGUI_113.dng

https://www.dropbox.com/s/fpdv0l1qnt3nnca/M01-1847_MLRawView_Lines_113.dng

https://www.dropbox.com/s/b0taxxnv76qj52s/M01-1848_MLRawviewer_x3_nolines.dng