Messages

I may have missed this but once you set the settings for 3k 1:1 crop, what's the diff between the regular record mode and then x5 crop mode? They seem the seem on preview but regular mode will record around 3072x1228 but x5 Crop just stops. Is it cranking the data rate or something?

And does FPS override make any difference in any of thi?

Yes, it occurs in lights mostly. Here's another example. Although, I'd already applied some luminance noise reduction to hide it a bit. This is a portion of a 3072x1308 frame from the April 6th 4k build. Look at the clouds. When in motion it almost looks like a vertical grain.

https://www.dropbox.com/s/6c1hcrw7stm729h/Screen%20Shot%202017-04-11%20at%206.35.25%20AM.png?dl=0

First off, this app is amazing. Thank you. So much better than the MLV_dump.

Now, with the advancement of the 4k raw and 14bit lossless the issue of Vertical Lines has resurfaced. Looks like no one has addressed I yet in the decoder softwares. I tried mlv-dump, cr2hdr and your latest build. How did you address fixing vertical lines in your older builds? And can you do it for the new 4k raw stuff? Or does the Footage app rely on another engine?

Thanks

Craig

Ewald,

As you can see from this string, it was proven not to be a specific ML version issues. I tried 1.1.3 and then 1.2.3 and still got them. Different builds did not make a difference. The only place i did NOT get line was in x3 crop mode. I never really got a logical answer why that was but maybe the sensor data is handled differently in that mode.

I was able to get rid off the bands by using different conversion apps. Raw Magic and MLV Mystic seemed to fix it while MLRawViewer did not. They may have fixed it by now. It's basically if the creator codes in an algorithm to fix the banding in the conversion to DNG process. It's apparently on the raw sensor data which Magic Lantern has no control of. You can also use Adobe Camera Raw and mess with the masking settings to get rid of them across skies and wall. One of my posts describes the process.

Good luck.

I went over this ad nauseum with A1ex two weeks ago. You just have to try different RAW or MLV decoding apps. I got the best results with Raw 2 DNG and Raw Magic. I explain how to use Adobe Camera Raw to mask the patterns. If it get's really bad you have to use Raw Therapee and apply different Demosaicing algorithms and masking.

http://www.magiclantern.fm/forum/index.php?topic=11612.0

Thanks A1ex, for explaining this and putting up with all my questions.   I'll send "Baldand" this thread and see if he can fix the line issues in MLRawViewer.

Craig

Ok, I have to eat crow. I was wrong....about 1.1.3 and the noise. The lines did show up.....

But! I'm not totally wrong about a couple issues. The lines ONLY show up using MLRawViewer to process DNGs. Which is a bummer since it's such a robust app. This might explain why I thought it was only happening after I updated to 1.2.3. I was not using that converter before that. The only way to get rid of these lines is to use RawTherapee and apply the "line noise" function. Maybe other apps but I don't know them.

RawMagic, raw2dng (mac GUI version) and possibly mlvdump (have to test again, lost track of test MLVs) give the mild matrix pattern on DNGs under 1.1.3. Each app a little variation in amount. Easily fixable in Adobe ACR. Apply a mild 8-17 setting on "masking" under the sharpening tab. Will render it gone in most cases. Especially in large swaths of the background (sky, blank walls etc.) Clouds did throw it for a loop. Hence, up to the 17 setting.

I got these same results under the April 24th 1.2.3 nightly build also before I dropped down to the April 29th 1.1.3 nightly. So, I was INCORRECT in assuming it was firmware related. (I will chalk it up to ignorance rather than stupidity since i'm an end user not a programmer) But, I did have lines randomly under older 1.2.3 builds. I have to go look at what version I used. Look at my earlier DNG posts on this thread. One is a Raw Magic and one is a raw2dng I believe. I can not explain why i got the lines on those conversions. I was not able to repeat it.

The only issue that hasn't been explained yet (cough cough) is why this does not happen to videos captured in the x3 crop mode? x3 videos I captured have a very natural noise pattern that is very grain like and barely detectable. This result was had on All ML builds under both 1.1.3 and 1.2.3 Even using MLRawViewer. Can that be explained and can whatever is making it work be applied to Raw data coming from the Full Frame capture? Would greatly expand pushing sharpness and color grading ability raw video for post. I discovered all this after converting to ProRes and then H.264 for Vimeo. These patterns look like wreak havoc once compressed. They look like tears and compression artifacts in end videos.

All DNGS from the 1.1.3 test are below.

https://www.dropbox.com/s/uba3yzpez5m1j1s/M01-1847_Nolines_RawMagic_113.dng

https://www.dropbox.com/s/v7xjv29eip1vl1p/M01-1847_nolines_raw2dngGUI_113.dng

https://www.dropbox.com/s/fpdv0l1qnt3nnca/M01-1847_MLRawView_Lines_113.dng

https://www.dropbox.com/s/b0taxxnv76qj52s/M01-1848_MLRawviewer_x3_nolines.dng

Ok, I get that. So.. playing devils advocate.

Why now and not on my previous raw files (1.1.3_ which I will verify soon)?
And why on the full frame but not on the x3 crop mode?
And if it is post processing.....could something have changed in the 1.2.3 firmware that has an affect on the raw files (meta data or wrapper) that could cause post processing to cause this now... but not in earlier builds?

I swear it just isn't in my earlier raw files from 1.1.3.  I just went back and loaded tiffs from various shots into Pshop and cranked sharpness and the noise patterns are just random and grain like.

I also never shot x3 crop mode until 1.2.3. Could that mode have some affect on how the sensor translates data once it has been used.

I realize there are just too many variables that are unaccounted for. I'm just trying to grab a tangible thread to start narrowing it down, or at least giving you enough data to start narrowing it down.

Let me try the april 24th build of 1.2.3. tonight and see if anything changes. If not I'll go back to 1.1.3 and the last build I used and see what happens.

I'm trying to nail this down so i can shoot a short we have planned. I'm starting to regret having messed with the camera when it was working three weeks ago just fine. Oy!

Si Senor.....

That was my next step. But it's going to take a few days. 
I might try a newer build perhaps of 1.2.3 and see if anything changes first but I shot a lot of raw under 1.1.3 and never saw lines.

DNGs pending......

Using RawTherapee you have to apply the "Line Noise" slider under the Demoisaic tab. I had to crank it to about 200-250 to get rid of them.
I also toggle between the "ammaze" or VNG4 demoisaic algorithms to get rid of the matrix pattern if it appears instead of the lines. Only bummer is you have to output to 16bit Tiffs and that is x3 larger than the DNGs.

A1ex,  what could be causing this to only be happening on the full frame shots but not on the x3 crop mode? I find that telling of something. Not sure what though. And what exactly is "Line Noise" since RT has a slider to reduce it. What could cause it not to be there under ML on 1.1.3? I seem to be the only one dealing with this at the moment so I realize it's not a big issue. I bet it's going to pop up more though as people notice it.

I did it a month ago so I could use an external monitor at HD res. read my issues here:

http://www.magiclantern.fm/forum/index.php?topic=11612.0

I've been getting strange vertical lines at full frame raw capture that weren't there in 1.1.3.  Using the x3 crop mode doesn't have this issue strangely enough. It appears to be some sort of demoisaicing issue. I can repair it with RawTherapee running the "line noise' fix under the raw tab but I will have to do this for pretty much every full frame raw capture shot at this point. Extra steps and it only outputs tiff sequences at 3x the size of the DNGs. Even more of a PITA for the RAW post procedure.  I can't be the only one dealing with this so when it pops up more maybe it will get more press.

I'm considering going back to 1.1.3 as it wasn't there at that firmware.

Craig

It was a version issue. Had to get the newest build for 10.8. Works now.

C

Bringing this back from the dead.

How are you importing multiple DNGs in a sequence? I can get RT to open one dng by right clicking and choosing open with. But when I use the file browser and choose a DNG sequence folder it "counts them" up in the left hand corner and then never displays them in the browser, hence, no way to edit them or batch process them out to tiff sequence. I'm using RT to get rid of line noise and strange demoisaicing patterns on certain takes..

Gracias.

Maybe i found the answer myself.  Looks like Raw Therapee and apply setting to all DNGs like in Adobe ACR.

I will wrestle with this and post results for others.

http://www.magiclantern.fm/forum/index.php?topic=9836.0

Thanks

Ok thanks,

Final question till this gets fixed. What app can process a ML raw file for green equilibration? Or is it done in batch to the DNG?

The apps i could find that were mentioned in the link you posted a few days ago are meant for still raw files. And some a few years before RAW video was around.

I may go back to 1.1.3 and see if this is still an issue.

It appears to have gotten worse.

Full Frame capture
https://www.dropbox.com/s/v3hcympsx6ht5tg/R2D_totalRAW_lines000263.dng

x3 crop. Light lines added to this for the first time.
https://www.dropbox.com/s/lxtgh3m8lkbcbly/R2D_totalRAW_x3_somelines_000133.dng

Ah, clearly i'm an apple fanboy. I needed a co-workers help to run this from the console. This is all it gave me back.


Resolution  : 1920 x 1080
Frames      : 360
Frame size  : 3629056 bytes
FPS         : 23.976
Black level : 2047
White level : 15000
Processing frame 1 of 360...
Vertical stripes correction:
  1.000  1.000  0.992  0.993  0.998  0.997  0.992  0.983
Cold pixels : 0
Processing frame 360 of 360...
Done.

To convert to jpg, you can try:
    ufraw-batch --out-type=jpg *.dng

To get a mjpeg video:
    ffmpeg -i %6d.jpg -vcodec mjpeg -qscale 1 video.avi

Ok,

Just cranked out the two raw files through the latest (that i could find) raw2dng.exe on the windows side.

Interesting results. Again the regular RAW mode frame has lost the lines but still exhibits a "matrix" style grid in the sky blue. (1st link) This was the same as what happened when using the raw2dng GUI mac version.  While the x3 cropped mode version has beautiful random pixels that looks like film grain when sharpened. (2nd link)
This happened to some degree (either lines or a grid) in all converters. RawMagic. Mac GUI of raw2dng and also MLV dump and MLRawViewer.

See here. Crank the sharpness in Adobe ACR to see the pattern.
https://www.dropbox.com/s/hzbd1i0ybve8pc4/windowsR2D_grid_000261.dng

https://www.dropbox.com/s/8686cn6l2nyhrq1/windowsR2D_x3_nogrid_000277.dng

The terminal goes away when after running the raw2dnge.exe process so i could not copy any "log" info or... if it's dumps a log file somewhere just need to tell me where to find it. It wasn't in the destination folder with the DNGs


I guess my and others concerns would be: This seems to be a semi repeatable an issue but also random. Seems to happen on some shots but not others. Seems to happen more in normal Raw and MLV modes but not as much in the x3 crop mode. Where is it happening and how do we fix? Is it just my camera? Is it a 1.2.3 thing? I don't recall it happening under 1.1.3 or not that I noticed.
If it's a debayering algorithm issue, is there a way to tell a converter what versions to use. AHD, PPG ,VNG, Bilinear etc.

I don't know. I'm an end user but i'm willing to help you out running more trials if you need them as I really want this to succeed. You've really created a kick ass mod.

Let me know what you need further Alex.

Thanks

Ok, i'll try on a PC on Monday. No windows machines at the Casa.

So this would be a step to run the raw files through BEFORE exporting them into DNGs? Is this a wide spread problem with Debayering?

I pumped a few out with MLRawViewer and I thought we had a winner but the last clip of 4 introduced the lines again. Could this be an ffmpeg thing?
It just wasn't doing this three weeks ago.

I'll get back to you on monday with the PC raw2dng log.

Thanks,

C

For what it's worth MLVdump 1.0 creates the same matrix style pattern on the DNGs.

Is this what you're after?

https://www.dropbox.com/s/w46yynnbe25hwv4/raw2dngLOG_gridissues.rtf

Thanks

Ok,

I replicated it last night. I processed the raw files with the newest RawMagic updated this week) and the 013 version of Raw2dng. (newest i could find for the mac)
I got interesting results.

RawMagic creates lines on the DNGs and raw2dng creates a subtler matrix looking grid. They are hard to see until you grade and then compress and then it looks like weird compression artifacts. But if you crank the sharpness and contrast on the dngs (included in the link) in ACR you can plainly see both on each dng. I was unfortunately able to do this on the x3 crop video also I shot last night.
I looked back at tiff sequences i shot when i had the 1.1.3 version of ML on the camera and i can't see the patterns. That doesn't mean they weren't there, just not as prevalent perhaps. See what you think and if it warrants more investigation i'll be happy to shoot some more tests for you. Just tell me some parameters. I'm curious if this is camera firmware, ML or the processing. Or all three perhaps. Hopefully fixable.

Hopefully i won't have to go back the the 1.1.3 firmware. As it is now this footage would not pass muster viewed much larger than on a computer screen.


This link is a drop box folder with the two same dngs and two tiffs processed from both apps.
https://www.dropbox.com/sh/kyvrl00j8t5detb/Q727jX9-7d#/

I can not unfortunately. I dumped the RAW file after converting to DNGs with RawMagic. Is raw2dng the more preferred app to use?
I can say that RawMagic never gave me an issue until this build of ML and or using the x3 crop option.

I'll see if I can replicate the issue with some new footage and convert using both applications and post the results.

Craig

Hey all, not sure if this is a processing issue or a ML issue.
Canon 5d Mark iii. April 16th 1_2_3 build. I was using the x3 crop function to shoot some owls. All the x3 crop stuff looks killer. I turn it off and shot normal and I got these tiny vertical lines in the "bright" section of the image.
It may be in the dark parts but I can't see it. I cranked the contrast and darkness in the frames so it sticks out more but i tried grading this and it ends up looking like compression artifacts in the video once exported to ProRes. Any ideas? It did not appear to be the "banding" issues i can find by searching. Two stills attached. Again, x3 crop looked fine. regular 1920x1080 raw resulted in the lines. I used Raw Magic to process these to DNGs

Look for the lines in the blue sky
https://www.dropbox.com/s/zqkq081quaamwg4/LinesRAWMarkiiiML1_2_3.jpg

No lines in x3 crop version
https://www.dropbox.com/s/jc7q91ryn0q4dka/NolinesX3cropRAWMarkiiiML1_2_3.jpg


Thanks,
Craig