Heavy vertical banding Noise removal? ACR vs Neat Video or both etc.

Started by Terry Tibbs, June 18, 2015, 02:40:18 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Terry Tibbs

Ok so the first thing I notice upon viewing my first 5D RAW footage is that it's noisy as hell. I gather that this is due to the incamera noise reduction not being used.

As a photographer, my experience is that the ACR noise reduction is excellent. I don't have a lot of experience with Neat Video, but I'm told it's also very handy.

Does anyone have any advice, or a workflow that suits them?

Wondering if a mixture of both might be good, ie a gentle ACR noise red followed by more specific treatment with neat Video.

Thoughts?

EDIT:

Here's a pic that illustrates what I'm talking about. Some pretty savage vertical kind of banding noise. (I'm still not entirely sure if I used the correct render output in OpencolourIO and AE. In OpenColourIO I used Cinelog-C, and in AE the output render was at DNxHD444.

The noise can be removed I'm sure. Plus the contrast will be a lot different after adding an LUT.

If you scroll across to the right on the image you can see the vertical lines...

F2, iso 1000....

EDIT: Does the way you export from RAW2CDNG make any difference? I exported 16bit maximized. It's just that the footage seems unusually noisy for iso 1000....



Terry Tibbs.


cpc

This type of noise is normal for underexposed footage. There simply isn't enough light fed to the sensor for proper clean darks. Furthermore, the noise is very visible here because your blacks are lifted with the log-ish profile; this also lifts the noise and makes it more apparent.

Neat Video will help a bit, but will soften the image. Better play with ACR's noise reduction first and see where it gets you. But the best you can do is pull black down to hide the noise. You will need to pull it down in correction anyway, because its too high for a proper image.

It is already too late, ofc, but as a general guideline: in low light scenes it is better to raise the ISO in camera in order to bring the image up. With 5dm3 raw iso 1600 is fine, 3200 is useable.

Danne

Yes,  ettr(expose to the right) is key when working with canon sensors. Making 1600 iso fully usable as cpc stated.

Levas


Levas


Levas

Just 'crush' the shadows in post ;)
What also works for me, I made darkframes for the higher isos like 3200 and 6400
I recorded a few seconds MLV file with the same settings and lenscap on.
Used command line option in mlv_dump for averaging all the frames in the mlv and used the resulting frame for dark frame substraction on the normal videoframes.

Terry Tibbs

Yeah, after some ACR noise reduction and the Film Convert LUT it looks better. Was shocked at the level of noise though. Like I said, I was only shooting at iso 1000, let alone 1600 or 3200.

Looks better now: (Doesn't look this noisy when watching actual video footage and is converted to jpeg in the upload):







Danne

Iso 100 will give more noise than iso 1600 if underexposed..

Terry Tibbs

Hey Danne. Thanks! Yeah, when I'm photographing weddings etc I usually expose to the right a little then bring it down a half stop or so and a lot of the noise disappears. What I didn't factor into this equation I think was that C-Log lifts all the shadows. What looked right on the monitor wasn't going to be how it looked in C-Log! Guess there's a way of monitoring for this. Haven't sussed that bit out yet!


Danne

I understand. I think the optimal log would be one that care for ettr. Not saying this is the case here. I don,t really know. I,m working on 3d luts like that a lot. Bring signal back rather than pushing. Have to talk to Andy600 about a cinelog_ettr version ;). Let,s see what he has to say.

Terry Tibbs

Actually yeah he did mention something about ettr to me in an email. Unfortunately I'm no DP and I had a ton of things to worry about such as sucker mounts and lights etc. I did my best, but I didn't manage to use colour charts or ettr. I guess ettr would have been the way to do it as you suggest.

I know it's off topic but on a side note while I seem to have someone's attention you don't happen to know my best export settings in AE? I'm on PC and have tried DNxHD 444. Guessing this is best although I see there is DNxHD 185....

Thanks. It's all I need to know now I think.....


Danne

Go for highest if you have the hard drive space. Not very familiar with dxnhd. THink they can go pretty high320mb or so?  Also some new updated dxnhd format to check out. Probably google is the better bet in this case.

Levas

For right exposure for video, the on display ML raw histogram is a good tool :)
For highest quality you nee the one with all the 4s in it.
185 has something to do with tv standard, maybe gamma 1.85?

EDIT: Oh wait 185 stands for the bitrate...

Terry Tibbs

Awesome chaps. DNxHD 444 it is then. I tried google but couldn't find anything.

Winning! :)

Andy600

Ok, a few comments regarding exposure, noise, Cinelog etc

First and perhaps most importantly, you don't expose for Cinelog as it is a post-applied transform of the linear raw signal (whatever the linear signal is). When shooting ML Raw you should always use raw metering, not the Canon default RGB meters (the settings are in the ML menu).

ETTR is a good method for 'most' shots but I would not recommend it for the image in your original post nor most night, low light or HDR shots in general. The light in the lower left of the frame will significantly saturate the sensor so that an ETTR exposure (using the raw meters) will force the important mids further down towards the noise floor. In a low light shot like this I would expose for the actors with a wide aperture and high ISO (up to 1600 on the 5D Mark III). The light will clip a little but this can be rolled off (soft clipped) using curves.

re: codecs - Cinelog-C (the OCIO part) acts like a compressor and is capable of squeezing the debayered linear signal's RGB values and DR for rendering to a 10bit space without being affected by ICC gamut mapping. If you rendered the sRGB signal (even cinelog v3.0) without the Cinelog-C gamut transform you risk clipping or shifting RGB pixel values. I recommend using the highest chroma subsampling that you have storage capacity for. The absolute minimum requirement for the Cinelog-C signal is 10bit 4:2:2 but 4:4:4 will allow you to do extreme grades to the same level as working directly with the raw files. If you have the latest AVID codec pack your should have DNxHR which is the PC equivalent of ProRes.

re: noise reduction - obviously, a better exposure is the first thing to try but ACR NR will usually suffice but Neatvideo can be added to the mix for stubborn FPN (try increasing the temporal filters if it's FPN)

re: relevant exposure levels between the C300 and Cinelog. C300 Canon log is an algorithm that changes depending on the selected exposure index - this is important for in-camera log recording. With MLRaw we have a linear raw signal to play with and, although it's possible to replicate EI linked transforms (i.e. many variations of the log curve), it is generally considered to be overkill when processing raw images outside of the camera and to use a single, optimum transfer function (usually the one linked to the camera's native ISO) - this is why Alexa Log-C, if/when used in post production, is actually Log-C EI800.

Cinelog works in conjunction with ACR's sRGB ICC profile. It negates the default tone curve and offsets the sRGB transfer function (all added by default in ACR) to linearize the signal, then we add a log formula and an exposure offset relevant to each camera's specified dynamic range - we had to find a good balance between the log curve and SNR while ensuring the DR is not clipped in ACR under default settings while retaining the maximum detail when the exposure slider is used (the Cinelog curve is a fairly complex mix of a basic Cineon log formula and gamma). Don't be scared of using the ACR exposure slider if the image is too bright or too dark - I nearly always pull it down a bit as it produces a thicker negative and it works in a similar way to EI linked in-camera log profiles.

While on the subject of ACR, it is impossible to make accurate camera profiles of Log-C, Slog, BMD Film etc from the actual math formulas because of limitations in the way ACR works (If it was possible I would make the fullset). The log curve always needs to be scaled, offset or manipulated (i.e. by adding a shoulder rolloff portion) in some way to avoid truncating the signal on it's way to After Effects (as can happen with other profiles), hence why the Cinelog V3.0 log curve is unique but once it's passed through ACR and is sitting on the AE timeline it is effectively Cineon log albeit with slightly raised gamma.

re: Canon log - you are correct in your other post about the C500 being the only C-series camera (so far) to record Canon Cinema Gamut. The C300 Canon Log profile gamut (CP Lock) is close to Rec709 primaries (but as I mentioned previously, with some non-linear tweaks). I managed to source a full set of chart exposures shot in Canon log on the C300 today and it's thrown up some interesting things regarding the CPLock gamut and the way the 8bit files work in relation to MLRaw from ACR. Apart from the difference in bit depth, it looks like you will get a much better default  match using a combination of Canon log and Canon Wide DR. I still need to build the OCIO transform (should be ready tomorrow for you to try). A nice side affect is that it seems to produce a much better spread of the C300 signal than Canons own luts and transforms - it's certainly better than the Canon Log to Cineon transfer function they provide - it doesn't clip the signal :)

Anyway, enough waffle from me for tonight. I'll be in touch tomorrow.
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

Terry Tibbs

Hey thanks Andy! Exciting stuff!

Look forward to hearing more!

I'm understanding the essence of a lot of what you're saying but I won't deny a lot of of it is going over my head.

I've ony ever used ACR noise reduction with the most basic settings as I am much more used to using Noise Ninja for photography, but I found this tutorial:

https://youtu.be/R9lPemcMcFE

Long story short, the settings look like this:



He also turns the clarity up a touch.

I've brought it into Premiere and once the same Film Comvert LUT is applied it's not a bad match, although it may be a wee bit too sharp in the end. I might bring the sharpening down a tad on the next attempt which should help with the noise.

Glad I at least got it right about the Cinema Gamut and the C500, and using DNxHD 444 :)

Cheers!


cpc

The problem with ETTR is that even if it will give the cleanest possible image, it is time consuming in post to match shots in a sequence if each shot is exposed using ETTR. I personally never use ETTR.
But what I often do with any camera is I rate the camera slower than the official rating; in other words, equally overexpose all shots. This trades highlights latitude for cleaner image. You get some of the benefits of ETTR but without sacrificing consistency. And image consistency is beneficial on many levels. Rating slower is especially useful for shooting raw or log transfer curves because manufacturers almost always specify ISOs which maximize DR but result in noisy shadows.

The 5dm3 can be beautiful at iso 1600 as long as it is properly exposed. Here is a couple of images from an old short I shot (I don't have a Canon camera anymore); both at 1600, with no noise reduction whatsoever:





The first image is fine as it is, and the second can use a little bit of NR in the shadows. Pretty clean where it counts though. In fact, passing it through the default color NR settings of ACR would probably deliver pretty clean shadows (I used Resolve on this short).

Andy600

Wow that is some serious sharpening!  :o - the resulting image will be filled with ringing artifacts and probably look like a checkerboard up close :D

There is a very good tut on ACR sharpening - I need to find the link but basically, hold down the CTRL key (or maybe it's the ALT key?  ::) - I'm saying this from memory) when making your adjustments. The key is to use the masking control to isolate the edges (push it quite high) then use the Radius slider to limit the sharpening area (keep it at a low value) then just tweak the amount. I rarely touch the Detail slider as it can cause artifacts.

For NR settings in ACR I tend to apply chroma NR first then nudge the luma NR until it starts smoothing the noise a touch but still has some texture - it's personal taste but less is more. NeatVideo has much better control and can be targeted - it's well worth using if you don't mind the extra processing time but it's probably not a good idea to use both at once.

re: Canon - I'll try and simplify things for you when I email you tomorrow - I know I can waffle a bit  ;D
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

Terry Tibbs

lol ok yeah I thought it may have been overkill!

Would be interested to see the link. Will try and get my head around what you're saying RE: CTRL ALT etc but it's a wee bit over my head. But dedication's what you need etc.

With regard to the waffle, that's why I'm here! Go for it! :)

Andy600

@cpc - I'm with you on that. I think ETTR can easily be misunderstood by some users and it's an option that can easily produce bad results if not used correctly. You still have to factor in the dynamic range of the scene and what the camera can actually capture for it to work so it's important to make use of the raw EV offsets (clipping allowances) in scenes with high DR.

I tend to expose for the subject when doing my own shots and deal with any clipped spectrals latter - 90% of the time I don't need/use ETTR, I just make sure the important stuff is in the middle and there is no highlight clipping other than spectrals. If I don't have lighting with me and cant capture what I want in the available light, within the limits of the aperture and ISO of the camera, I don't hit record - there is no point. I know I've missed some good shots with this methodology but if the technology/gear/me isn't up to task for something I think it better to avoid wasting CF card space and a disappointing post review of the footage. You can't polish a t**d  :D
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

Danne

Perfect examples cpc.

Sorry for the misunderstanding. When I say ettr I actually mean this.
I rate the camera slower than the official rating; in other words, equally overexpose all shots -cpc



Danne

Hi @Andy600

QuoteCinelog-C (the OCIO part) acts like a compressor and is capable of squeezing the debayered linear signal's RGB values and DR for rendering to a 10bit space without being affected by ICC gamut mapping.

Would be very interesting if you shared what this means in practice. When compressing RGB values and DR, are you using curve tool and tweaking gamma, contrast or what is used to compress the signals? Do you mean you go from 16-bit to 10-bit space? It is all very abstract right now. You can link if you like.
Thanks

Andy600

@Danne - This is very specific to the workings of ACR and how it interacts with After Effects and ICC profiles. The task is to ensure debayered linear pixel values can be transformed to a log colorspace, rendered (in a 10bit container) and later inverted to produce the same values i.e. there are typically no out of gamut values remaining after the transform that would cause the ICC profile to remap them (depending on the ICC profile's rendering intent) or be truncated by the act of rendering.

Think of it as trying to get a large, square peg into a smaller, round hole - Cinelog reshapes the square peg to match the round hole and because it's an invertible math function we can pretty much get it back to being a larger square peg (reproducing out of gamut values). The formula we use is our IP so forgive me for not sharing that part ;) 

I should add that it also allows super brights (pixels that have values exceeding 1.0 in floating point) to be retained.
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

Danne

@Andy600  :o even more, haha.
Every word has to be googled. Gonna do some digging.