mlv_dump on steroids

Started by bouncyball, February 16, 2017, 07:10:10 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Danne

Try a darkframe. Probably gonna work best. Chroma smooth 2x2 is fine in my world but it might not be needed.
When doing a darkframe film with exact same setting as the footage, lens cap on, shoot 2-3 seconds.
Creating a storage folder with darkframes for batch_mlv shouldn't be too hard. Just need some time.

You need to decompress your footage too before doing the mlv_dump averaging. If it's too many steps just upload a vanilla darkframe and I test it for you.

a1ex

Quote from: Kharak on July 01, 2017, 03:16:29 AM
MLRawViewer manages to get rid of them with its vertical Stripe fix, but introduces artifacts like horizontal stripes and Chunky vertical stripes in black/dark spots and/or high contrast sources instead. Is it in anyway possible to combine the best of both worlds? It is my understanding that MLRawViewer uses OpenGL shaders with combination of different coding languages

Unfortunately, MLRawViewer just locks up my current machine (bringing down the entire operating system), so it's not straightforward for me to play with this code. Otherwise I'd probably maintain this program, to some extent.

Quote
This shot could have been +0.5 EV higher, maybe +1, that's a big maybe.

To see how it would have looked at +1 EV, do this:

exiftool M08-1044-cut_000000.dng -WhiteLevel -BlackLevel
White Level                     : 16200
Black Level                     : 2047

exiftool M08-1044-cut_000000.dng -WhiteLevel=9123


How did I come up with this number? (2047 + 16200) / 2.

Proof that taking the average of black and white is equivalent with increasing the exposure 1 stop (regarding overall look and highlight clipping point, but not noise levels) if left as an exercise for the reader.

Result...

octave:1> a = read_raw('M08-1044-cut_000000.dng');
octave:2> prctile(a(:), [1 5 10 50 90 99 99.9 99.99 100])'
ans =
   2054.0   2066.0   2076.0   2176.0   3285.0   6054.0   8156.0   8828.6   9113.0


That means, not a single pixel would have been clipped in this frame by increasing the exposure by 1 stop (which means, no information would have been lost). You should be able to grade this file identically to the original (please try).

To simulate 2 stops, try white level 5585, and for 1.5 stops, 7050. Math is left as exercise.


octave:3> prctile(a(1:2:end,1:2:end)(:), [1 5 10 50 90 99 99.9 99.99 100])'
   2050.0   2059.0   2067.0   2126.0   2767.0   4212.5   5019.0   5236.2   5364.0


That means, with 2 stops, no pixel on the red channel would have been clipped (therefore, no highlights should be clipped to pure white).

My JPEGs (0, +1, +1.5, +2 from white level, +3, +2, +1.5, +1 from exposure slider; for other settings, change extension to .ufraw):




The highlights in the last two (where some channels are clipped) are desaturated (they do not retain the original colors, but they are not clipped to white either). That's how highlight recovery works in ufraw. To my knowledge, many commercial raw processing programs perform a lot better in this area (curious to see how Danne would render this).

Quote
btw bit off topic, a line above in the link you posted:
> The read noise can be isolated by taking a "black frame" image, an exposure with the lens cap on and the highest
> available shutter speed; there are thus no photons captured, and only the electronic noise from reading the sensor remains.

Is this the actual proper way of doing a Darkframe ? Setting shutter speed to the highest available?

Not sure how exactly you reached this conclusion; this page (next time, make sure the article you are talking about can be identified, e.g. with a link) mentions the highest shutter speed as a way to analyze the read noise component (without e.g. thermal noise, which depends on exposure, or PRNU, which depends on the number of photons captured), not as the best way to capture dark frames.

I'd take the dark frames under the same conditions as the original clip. Probably some settings (like shutter speed) won't matter much, but I didn't experiment with it.

QuoteI have some Vintage lenses that have quite big variations in the Vignetting at the different F-stops, from 1.4 up to 5.6

Flat frame would correct the vignetting in those lenses as well - but you will need one averaged flat MLV at each aperture. Repeat for each ISO and some usual shutter speeds, and you'll probably end up with a 3D matrix of correction frames. That would definitely require some sort of automation.

Danne

Very informative, educative. Will try to find more time do some more testing around it. Just started vacay...

bouncyball

Hey guys! Thanks for lots of very interesting talking here :)

Quote from: a1ex on July 01, 2017, 09:07:47 AM
Otherwise I'd probably maintain this program, to some extent.
I'm sure every community member would be extremely thankful for that :)

g3gg0

Quote from: a1ex on July 01, 2017, 09:07:47 AM
Flat frame would correct the vignetting in those lenses as well - but you will need one averaged flat MLV at each aperture. Repeat for each ISO and some usual shutter speeds, and you'll probably end up with a 3D matrix of correction frames. That would definitely require some sort of automation.

as proposed in early MLV times, we could use FLAT and DARK frame types instead of VIDF.
a "correction.mlv" then would contain (sorted by block time):

MLVI, RAWI, RAWC, LENS, (misc info), EXPO, FLAT, DARK, EXPO, FLAT, DARK, EXPO, FLAT, DARK, EXPO, FLAT, DARK, EXPO, FLAT, DARK, ...

where EXPO and FLAT+DARK are a series of correction frames for the EXPO settings.
(yeah DARK isn't possible as of now so we would omit it, but maybe somewhen we can do that)

possible correction frames: FLAT, DARK, BIAS
(see http://deepskystacker.free.fr/english/faq.htm )
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

Danne

An idea right out of my ass. What about creating one for all isos darkframe with rec trigger? Record a sequence with iso 100 then pause, set iso 200 record a sequence pause, iso 300 etc, then average this all iso MLV file to one hell of a darkframe?

g3gg0

yes, thats what i wrote with the sequence above ;D
EXPO <= ISO, exposure time
LENS <= lens, aperture, focal length


typedef struct {
    uint8_t     blockType[4];
    uint32_t    blockSize;    /* total frame size */
    uint64_t    timestamp;    /* hardware counter timestamp for this frame (relative to recording start) */
    uint32_t    isoMode;    /* 0=manual, 1=auto */
    uint32_t    isoValue;    /* camera delivered ISO value */
    uint32_t    isoAnalog;    /* ISO obtained by hardware amplification (most full-stop ISOs, except extreme values) */
    uint32_t    digitalGain;    /* digital ISO gain (1024 = 1 EV) - it's not baked in the raw data, so you may want to scale it or adjust the white level */
    uint64_t    shutterValue;    /* exposure time in microseconds */
}  mlv_expo_hdr_t;

typedef struct {
    uint8_t     blockType[4];
    uint32_t    blockSize;    /* total frame size */
    uint64_t    timestamp;    /* hardware counter timestamp for this frame (relative to recording start) */
    uint16_t    focalLength;    /* in mm */
    uint16_t    focalDist;    /* in mm (65535 = infinite) */
    uint16_t    aperture;    /* f-number * 100 */
    uint8_t     stabilizerMode;    /* 0=off, 1=on, (is the new L mode relevant) */
    uint8_t     autofocusMode;    /* 0=off, 1=on */
    uint32_t    flags;    /* 1=CA avail, 2=Vign avail, ... */
    uint32_t    lensID;    /* hexadecimal lens ID (delivered by properties?) */
    uint8_t     lensName[32];    /* full lens string */
    uint8_t     lensSerial[32]; /* full lens serial number */
}  mlv_lens_hdr_t;

Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

Danne

Quote(yeah DARK isn't possible as of now so we would omit it, but maybe somewhen we can do that)
So we can uncomment some abort stops in mlv_dump to try this?
With all due respect flat frames needs some more testing and going throughs. As opposed to darkframes it's more tinkering time getting it to work to the better. Removing vignetting often causes grain and other side effects such as unexpected pinks in highlights. Not being able to upload examples isn't ideal here but I see if I find time here. Darkframes works very reliably imo.

a1ex

Quote from: Danne on July 01, 2017, 01:31:57 PM
Removing vignetting often causes grain

Because of exposure pushing in the dark areas? Yes, that's expected.

Quote
unexpected pinks in highlights

White level may need to be adjusted; unfortunately, this will result in clipping useful highlights (that were not clipped in the vignetted area of the frame).

Possible workarounds:
- include some highlight recovery code in mlv_dump (hard, there are an infinite number of methods to do this, as it must guess what's in the clipped channels; that's the job of the raw processing software)
- variable white level support in raw processing programs (does any app support something like a white frame, or other sort of variable white level?)
- apply the vignetting correction after developing the raw image (after the highlight recovery is applied, but while still in some linear space) - depends on the processing app.
- decompose the flat frame into vignette correction (radial averaging, easy) and defects correction (dust spots, vertical stripes, whatever), and apply just the second part.

From all those, the last item appears to be the most practical to me; mlv_dump already does vertical averaging, so it's easy to add a similar option. A very large blurring filter might work as well, if the correction frame is not uniform for some reason.

g3gg0

i would keep mlv_dump as focused as possible.

if some correction is necessary in "mlv-space", i.e. where we act camera specific, where we are tightly coupled to mlv_lite/mlv_rec, i absolutely agree implementing that in mlv_dump.
also some "primitive" corrections like chroma smoothing, bad pixel maps etc is well placed there.
also the DARK, BIAS, FLAT correction is quite camera - or mlv-workflow - specific and its good to place it in mlv_dump.
lens vignetting correction, well, thats something that we get "for free" using FLAT frames.

more complex and generic corrections might be best somewere else. (hightlight recovery for example)
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

bouncyball

Quote from: g3gg0 on July 01, 2017, 02:27:12 PM
i would keep mlv_dump as focused as possible.
I've got a vague feeling that I'm gonna remove a lot of stuff from my PR :P

Quote from: g3gg0 on July 01, 2017, 02:27:12 PM
also the DARK, BIAS, FLAT correction is quite camera - or mlv-workflow - specific and its good to place it in mlv_dump.
DARK, FLAT frame types!
Fantastic, simple, effective proposal for "correction.mlv".

Quote from: g3gg0 on July 01, 2017, 02:27:12 PM
more complex and generic corrections might be best somewere else. (hightlight recovery for example)
Would be so nice to not rely on any raw processor and have several advanced hightlight recovery methods in the arsenal. Could be implemented outside of "mlv_dump.c" just during exporting DNGs. Or maybe introduce plugin framework? ;)

g3gg0

these frames were proposed from the beginning on ;)
well, mlv_dump already has this framework also from the very beginning. lua callouts.

if we integrate highlight recovery - then why not noise reduction, rotation, cropping, lens correction, deflickering, red eye reduction etc.
for me this is definitely raw manipulation and nothing that is a) required to make mlv usable or b) very camera specific.


Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

Danne

This test mainly verifies this
QuoteThat means, not a single pixel would have been clipped in this frame by increasing the exposure by 1 stop (which means, no information would have been lost). You should be able to grade this file identically to the original (please try).
Mainly this suggests pushing f-stop more "to the right" to achieve cleaner shadows prone to fpn and stripes.

Opened up the files in ACR, altered white level acording to a1ex test above http://www.magiclantern.fm/forum/index.php?topic=18975.msg186641#msg186641.
Matched exposure accordingly(+2, +1.5, +1, 0)
Note white level 5585 and white clip warning. Check 4th pic.
From 5th pic and onwards some heavy highlight recovery and acr behaviour.
Final pics good old dcraw -H 2 recovery.

16200


9123


7050


5585 Note that highlight clip warning is shown in ACR, upper right corner





Now some heavy highlight recovery. Slider all the way to the left



The result by eye with highlight recovery applied. Pic one and two without any clipping. One stop push seems ok as stated by a1ex before.
16200


9123(1 stop push)




Next two are showing some funky rebuilding artefacts. Do have in mind we are pulling slider to the maximum.
7050


5585





dcraw output verifies this as well. I used following command
dcraw -T -w -H 2 INPUT.dng


16200


9123


7050


5585

bouncyball

Quote from: g3gg0 on July 01, 2017, 11:36:23 PM
well, mlv_dump already has this framework also from the very beginning. lua callouts.
btw, never used it myself. Should be tested if I broke something after my changes. And no one can do this better than you ;)

Quote from: g3gg0 on July 01, 2017, 11:36:23 PM
if we integrate highlight recovery - then why not noise reduction, rotation, cropping, lens correction, deflickering, red eye reduction etc.
That's right agree, but would be cool, all in one Swiss Army knife, well perhaps red eye is too much :P

g3gg0

Quote from: bouncyball on July 02, 2017, 09:24:19 AM
That's right agree, but would be cool, all in one Swiss Army knife, well perhaps red eye is too much :P


ehm. this was ironic... :)
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

bouncyball


bouncyball

@Danne

Thanks for thorough investigation!

Kharak

So I did a DFA on the example you all are looking at, it did not help with the FPN. I think it actually made it worse. Mentioned this before on 'mlv_dump on steroids' before Bouncyball fixed a bug that only made VS corrections to the first frame; it's as if Darkframe Averaging makes more contrast for the stripes, but this might be bollocks from me. Again, just a hunch and you guys know a lot better than me what the correction/averaging and so on are actually doing.

I took the decompressed file in to MLVProducer where I made a FPN profile to try and remove it. In parts on his cheek and throat, the results were better, but it also introduced vertical FPN stripes in a few other places instead, like in brighter splotches in the background and so on. Comparing MLVProducer's output, the MLV_dump DFA output and just normal Vertical Stripe fix direct from MLV_dump (danne). It looks to me as normal Output from MLV_dump with VS Fix had the best result of the three.
Maybe someone else could have a go with the DFA, in case I made a mistake. I don't remember what ISO I shot this at, after decompression, the MLV says ISO 100 and shutter 1/120, the shutter is accurate for sure. So I assume the ISO is true aswell?

All three outputs vary in Colour and Luminance. Which might have skewed my perception of the Striped results (less contrast, more contrast, green darks, magenta darks etc..) Bottom line is, the FPN is still there. Unfortunately I don't have a lot of time to fiddle around matching these frames and outputs, I got a music video coming up which requires a lot of my attention on planning these days. But I still wanna help out as much as I possibly can, I really wanna get rid of this FPN, I mean I think the FPN in those shadows is surrounded with totally usable shadow details (after NR techniques), its just the Stripes making it unusable (to me). 

MLVProducer made the MLV 2 Frames longer than the other converters, so I don't know what's going on there, I outputted 59.976 from MLVProducer, as it read the file, maybe it doesn't know how to count properly or interpolates somewhere along the line. The software is a bit funky sometimes, but I also never really used it, I was more of a RAW2CDNG guy.

Another question. When a MLV has been decompressed, it shows Metadata like ISO and Shutter speed, but after DNG's have been made, that metadata is gone? Wouldn't it be an idea to include that aswell in the DNG output?
once you go raw you never go back

a1ex

Well, I'm actually looking for some sample footage with dark frame correction, and another one also with flat frame, as I've just started a test suite for mlv_dump. Currently I've used the MLV from this thread (hope you don't mind), and a buggy MLV from another thread.

So, if you have a small dark MLV to match this clip, I can look into it.

Danne

Not sure if these MLV frames and included flatframes could help any. Shot with eosm 14bit. Compressed with lj92(for upload size reasons, bad connection) in post so obviously will need decompressing back.
https://bitbucket.org/Dannephoto/magic-lantern/downloads/eosm_flatframes.zip
332MB

a1ex

Thanks - ideally it should be a real-world example with visible defects that would be corrected by these frames (maybe extract a short clip from one of your dark/flat experiments, where this correction makes a difference).

If possible, it should have some clipped highlights as well.

Currently I'm trying to set up a visual diff, to highlight image changes between different versions. These things are already used in web development - some examples. I've also posted a proof of concept for ML menus.

Danne


Kharak

@a1ex

I am uploading a unprocessed dark mlv, ISO 100 1/120.

I cut the MLV down to 30 Frames, I hope it is enough, the connection makes this the very max size for me 41 some MB. If you need more, you gotta wait 1-2 days and I'll have 3G or possibly real internet ;)


EDIT:
ISO 100 mlv
https://mega.nz/#!8NgVAAaK!9-870FPyEXgTXA9FGaV2za5NxkHRM8WFmJke661sqDo
once you go raw you never go back

a1ex

Thanks - here's my attempt, but unfortunately the dark frame introduced additional artifacts...

Maybe these dark frames really have to be recorded for each clip (ideally at the beginning and at the end - or just at the end, to avoid missing the action).

https://builds.magiclantern.fm/jenkins/view/Utilities/job/mlv_dump_darktest/ (0 = plain, 1 = with dark frame)

ilia3101

Maybe this isn't relevant to this thread, but could there be a darkframe feature added to mlv/lite where it closes the shutter for a short time at the end of recording and records a few frames in "DARK" blocks? Mlv specific apps could take good advantage of this ;)