crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView

[Kharak]

Does april 14th build run the test for the april 4th build?

My connection is so incredibly slow, if i dont have to dowload apr10 just to test for a bug and then download 14th, well that would be swell

[Quentin]

@GutterPump
You remind me of myself when I shoot lens cap 

[GutterPump]

@quentin

Haha it could have been worse, it's really the mess on my desk. But for magic lantern I make an effort.

[hindra]

Hey guys, here's a quick 23 second shot of the 8 bit lossless 3520x1322 at 60 fps with the latest April 14th 1.2.3 build. My hands are really shaky, I'm not a pro and I had my shutter at 1/80th.
ISO 200, F8, 1/80th - SHAKY HANDS - straight out of camera no corrections/lut - 60mbit vbr
Its working

https://vimeo.com/213302351


EDIT: 30 fps

[Teamsleepkid]

thanks for posting the video thats pretty crazy

[a1ex]

Hey guys, here's a quick 23 second shot of the 8 bit lossless ... ISO 200

Here a zip with DNG if some want check how look 8/10/12 bits lossless

https://www.dropbox.com/s/pj7db1izytkj3zj/8_10_12bitslossless_samples.zip?dl=0

Have you noticed the ISO suggestions when selecting a lower bit depth from menu? (at the bottom of screen)

Please refer to this commit for more details, and to this article for some theory regarding bit depth, noise and posterization.

I've considered acceptable a quantization step of twice the noise stdev, which would be just a bit better than "gradient 3-bit tonality" from Fig. 3 in the linked article (log2(256/24) = 3.3), which gives the lowest ISO usable at each bit depth. For the highest ISO, I've considered "wasteful" a quantization step smaller than noise stdev / 2.5.

3520x1322 at 60 fps

Impossible (please check the metadata with mlv_dump -v).

[iheartbmx]

Filled a 32GB card with 3K / 2.35:1 / 23.97fps / 14bit lossless - Completely stable with zero bad frames! (I used cr2hdr v1.2 to extract DNG's on OSX Sierra and it worked perfect)

You guys are amazing!

[Quentin]

@a1ex
I saw ISO suggestions.
8 bit digests noise generated by high ISO ? (I read that raw is divided by 64)
Thanks in advance for the enlightenment
and your hard work

[a1ex]

Yes, at high ISOs, the lowest bits (trimmed for reducing the bit depth) are mostly noise.

At low ISOs, some of them are useful data (especially in shadows).

The last two bits are mostly noise on current sensors (so you can safely use 12-bit at low ISOs).

The posterization from ML previews is real (it matches what I'm getting when opening the DNG in ufraw, for example), so you can actually preview the loss of shadow detail.

For a visual explanation, don't forget to move the mouse over the images from the linked article (see previous post).

[Quentin]

Thats incredible and gives more room to film in low light, broadening the camera capabilities.
Knowing how ETTR works, I can understand what you describe

[hindra]

Have you noticed the ISO suggestions when selecting a lower bit depth from menu? (at the bottom of screen)

Please refer to this commit for more details, and to this article for some theory regarding bit depth, noise and posterization.

Impossible (please check the metadata with mlv_dump -v).

Yes, I did notice the ISO suggestions. I have no idea why I thought I was shooting at 60fps. I re-rendered at 30 fps, I also cant hit 60fps when attempting to re-create. but, 8 bit lossless works! I was trying to use the lowest bitrate to obtain the highest fps and I thought it was 60, I was mistaken. Thank you for those links.

[D_Odell]

Is there any chance MLVFS will start supporting these files without using mlv_dump?

[giarcpnw]

What would be the reason the DNGs from the April 14th build would not open in After Effects? I'm getting can not parse errors. Worked fine with April 10th. Using Footage.app on mac.  Shot 14bit lossless.

[giarcpnw]

What would be the reason the DNGs from the April 14th build would not open in After Effects? I'm getting can not parse errors. Worked fine with April 10th. Using Footage.app on mac.  Shot 14bit lossless.

I think I answered my own question. The first frame of the sequence in all the MLVs I shot tonight were black (I assume corrupted) I removed them from the sequences and then AE and Camera Raw could load the sequence.

Something to note for this build. April 14th.

[a1ex]

Small MLV and steps to reproduce, please.

[GutterPump]

Have you noticed the ISO suggestions when selecting a lower bit depth from menu? (at the bottom of screen)

Of course i noticed these suggestions, now i read your article i'm able to understand better.

I will do differents tests in low light condition with different range of ISO. eg : 12800 ISO to 800.

So, if i understand well, in this way, the noise is our friend if we need less posterization because the 8bits depth ? So it means that if we film a scene very illuminated outside with the 8bits , we have all interest to use a variable filter ND.

[a1ex]

So, if i understand well, in this way, the noise is our friend if we need less posterization because the 8bits depth ?

Correct.

To be effective for reducing posterization, the noise must be applied before the quantization (therefore, analog ISO noise is fine). Adding it in post will not help.

So it means that if we film a scene very illuminated outside with the 8bits , we have all interest to use a variable filter ND.

I disagree here. Lower bit depths were added to reduce the data rate at higher ISOs (where the lowest bits are mostly noise, and because of them, the image doesn't compress well).

I don't think filming with 8 bits is actually your goal. More likely, I assume you actually want to minimize the data rate. With uncompressed data, it's easy (all frames have the same size: W x H x bpp/8). With lossless compression, you can't choose the compression ratio in advance; this is given - roughly - by the entropy of the image.

These "fake bit depth" settings (actually dividing the signal by a power of 2) are one way to reduce the entropy (by removing the least significant bits, because of integer division). That's a lossy operation (going from 14 bits to any lower value). After this step, compression is lossless (that is, the uncompressed data will be identical to what was fed to the compressor).

(That means, 14-bit lossless will give raw data identical to uncompressed 14-bit (100% identical, not one single bit different, contrary to some earlier report), while "12-bit lossless" should be interpreted as "14-bit to 12-bit conversion - lossy by definition - followed by lossless compression", so it should have the same number of useful levels as uncompressed 12-bit. Please note the 12-bit lossless is not identical to 12-bit uncompressed - they differ by a constant value and possibly by some round-off error, and the same is true for lower bit depths.)

Back to entropy - increasing the ISO and underexposing (to keep the image brightness unchanged) will increase noise in the image (at all brightness levels). So, what you may gain by reducing the bit depth, you will lose from the added noise. Here's a quick test (static outdoor scene exposed to the right, plain 1080p, compression rates taken from ML menu):

Bits per pixel      14  12  11  10   9   8
ISO  100 1/100     61% 53% 50% 48% 46% 43%
ISO  200 1/200     62% 54% 51% 49% 47% 44%
ISO  400 1/400     63% 54% 51% 49% 47% 45%
ISO  800 1/800     65% 55% 52% 50% 48% 46%
ISO 1600 1/1600    67% 56% 53% 50% 48% 46%
ISO 3200 1/3200    70% 57% 53% 50% 49% 47%
ISO 6400 1/6250    76% 60% 55% 52% 50% 48%
ISO 12800 1/12500  79% 63% 57% 53% 50% 49%

=> there's nothing to gain by using a lower bit depth at a higher ISO + underexposure.

Of course, if your goal is to record at 8 bits, no matter what, then of course, the best thing to do is to increase ISO at 12800 (or at least 6400) for half-decent results. It's also trivial to change the code to get 6 bits, 4 bits or even one single bit, if that's your goal

My advice: first expose properly (ETTR, zebras, raw histogram, whatever you prefer), and afterwards choose the optimal bit depth. Not the other way.

[Quentin]

Actually that post about compression needed to be in two parts
a) Why do I get these boxy big pixels compression-like in out of focus areas
b) Is 14bit and 14bit lossless visually identical

The question b) is answered already.
The question a), I guess was due to a different reason other than the bit depth

But I understand a1ex words about bit depth. Great explanation

[VinceML]

Hi Guys,

I explain how work and how I setup my 5D for create 3,5K video, here :

https://www.youtube.com/watch?v=kPQMNHhV41M

[Levas]

 
Now with these fake bit depth options, this is getting interesting for the SD card camera's 

@Alex,
The red numbers in your bit depth / compression ratio table, do you mean that the serious loss of info/shadow detail is where the red numbers are ?
And if I understand correct, you're using digital gain, which cut's of the noisy bit, but the image technically stays 14 bit for the compression algorithm, but because of the noisy bit cut off, the compression rate goes even beyond the 50%.

[a1ex]

Correct - the red numbers match my ISO recommendations and are below "gradient 3-bit tonality" from the third figure (labeled Fig. 18) from the noise article. Also added orange = wasteful (there's little or no additional image quality to gain at these settings).

If the theory is correct (which I didn't have the patience to check by pixel peeping), then it's probably best to stay near the transition line to red, on the black side (that is, 11-bit for ISO 100-800, and reduce the bit depth once you need ISO 1600 and beyond).

I believe the recommended settings are visually identical to regular 14-bit (while in 14-bit lossless, the raw data is 100% identical to uncompressed 14-bit). Again, this is not confirmed by pixel-peeping (it's just noise theory). There may be a small color cast in shadows, because the 0.5 LSB offset is not yet applied (but it's easy to fix once @martinhering helps us decode the JPEG stream).

The estimations were done for 3x3 binning (plain 1080p); the 1:1 crop mode may require slightly different values.

At these (probably optimal) settings, the compression rate appears to be consistent (50%), at least on my test scene (it was some foliage exposed to the right, but didn't save it).

(my definition for compression rate is: saved frame size / size of a 14-bit uncompressed frame)

[giarcpnw]

Small MLV and steps to reproduce, please.

Well, i could not duplicate it. I updated Martin's footage app this morning and they process fine. I do however remember it happened right after i tried to capture 12bit and 8 bit lossless and saw this on my screen.
This happens whenever I try to capture any lossless bit rate than other than 14bit lossless. Trying to record like this results in a ton of scrolling errors. (none of which I know what they mean) Have to power off to stop.   Pressing x5 crop WILL ALLOW capture however in this mode. 3072x1308, FPS Override 23.976

[a1ex]

@giarcpnw: already reported, and for the moment I don't know how to solve it. Will probably disable it in crop modes with modified resolution, until I'll understand what happens.

@Quentin: I don't see "boxy big pixels compression-like in out of focus areas" in your post - I only see black text on a blueish background...

[Quentin]

Unfortunately I erased the videos.
But ... how do we attach an image on the post ?
Sorry for my ignorance ?

[hindra]

1080p48 at 12 and 11 bit lossless at 100 iso I get results with a split exposure. Global draw off