7D at Halloween: The Bitrate-Monster (EXPERIMENTAL)

[RenatoPhoto]

This high bitrate is AWESOME!!! \o/

I would like to duplicate your results.  What are the important parameters to obtain such results?

[g3gg0]

okay now found how to enable all-I on 600D using cache hacks.
will try to port that on 7D.

i ordered a maksutov mto 11-ca and want to shoot some high bit rate videos and stack them using e.g. registaxx

update:
all-I works with super high bit rate.
just my analyzer crashes with all videos.
but i see that all frames are I, just cannot look deeper into the details.

but test yourself:
D/L experimental All-I 7D FIR

[1%]

If you're doing astro stuff, maybe messing with deblocking will help. If you had gop 4, all I is just gop1

[P337]

I'm using a SanDisk Extreme 8GB 60mbps card. I just shot some trees.

ISO 160
Shutter 1/250
Aperture f/11
Flush rate: 4
CBR 20.0x
Focus peaking, magic zoom, histogram, waveform and bitrate display were ON
Audio OFF

I recorded 1 minute, 42 seconds of handheld footage. Recording was stopped by me (it would keep recording for the whole 4GB). The file size is 2.38GB.

As g3gg0 rightly said, it's actually VBR. I got a maximum bitrate of 218 while the average was around 170. Although I focused it pin-sharp, the footage is blurry as is all 7D footage. Visually it looks just like ordinary 1x BR to me.

Could anyone please guide me about doing a better comparison?

It might help to know that once you hit an aperture of f/8 the pixels in the 7D start to diffract which will blur the image a bit more and more the further you go passed that threshold.

[RenatoPhoto]

My visual test show little-to-no difference in noise between 1x or 20x.  I applied denoising as mentioned before but still no improvement in final results.

Also tried the ALL-I at 20x and here I noticed a little more noise at 20x.

Tests at iso 12800
f/5.6 and 1.8
Lens 24mm sigma f/1.8
1920x1080 24fps

If anyone gets better results with cbr 20x then I would like to know how.  It is very difficult for me to post images or videos since I have very limited bandwidth.

[feureau]

update:
all-I works with super high bit rate.
just my analyzer crashes with all videos.
but i see that all frames are I, just cannot look deeper into the details.

but test yourself:
D/L experimental All-I 7D FIR

This is great news, g3gg0! I'm trying it out as we speak.
AVInaptic says I'm getting All-I now~!

Out of 139 frames:

Code: [Select]

P-slices                      0 (  0.000 %)
B-slices                      0 (  0.000 %)
I-slices                    139 (100.000 %) ####################
SP-slices                     0 (  0.000 %)
SI-slices                     0 (  0.000 %)
One thing I noticed though, is that increasing the bitrate and using all-I, it seems that I'm seeing far more severe rolling shutter than normal...

Also:

- it seems to be not CBR, but VBR although it says CBR

I think this is VBR too. If I shoot with low ISO and don't have anything moving in the frame, I'm only getting around 35-45 mbps...

Any news on recording audio too, btw? (even at reduced bitrate to save bandwidth for video?) Would be great as guidetracks.


Thanks again for all the hard work, g3gg0!

[feureau]

I would like to duplicate your results.  What are the important parameters to obtain such results?

ML Movie settings:

Code: [Select]

Mode: CBR
CBR Factor 20.0x
Bitrate Info: On
Flush rate: 2
BuffWarnLevel: 50%

The processor percentage thingy at the upper right corner that shows up on the bitrate info box on the upper right stabilizes at 4% on my 7D.

[g3gg0]

My visual test show little-to-no difference in noise between 1x or 20x.  I applied denoising as mentioned before but still no improvement in final results.

interesting. i can see the ISO noise a more with 10x than with 1x when playing with VLC on my computer.
also image quality analysis using a special tool says that the quality is better (factor 2, when bitrate is raised by factor 5 iirc)

sure that your video file shows that high bit rate? try using VLC and look at the statistics.

[akshayverma1]

I did some tests with the All-I port.

These are jpegs at quality setting 10.
1080p at 25FPS
Bitrate: CBR 20x
1/250s, f/5.6, ISO 160
18-135mm lens at 135mm

1) Newspaper closeup: unsharpened


Sharpened:


2) Tree: unsharpened


Sharpened:


Compression artifacts still show up occasionally. All-I is improving the quality but if the bitrate could be constant there would be a major improvement.

Thanks a ton for all the hard work g3gg0!

@P337: Yes! I agree anything lower than f/8 would soften the footage, but a still image taken at the same aperture isn't that soft. I think it's because of the line skipping, as ilguercio pointed out. This time I shot at f/5.6 (which I think is the sharpest for this camera, or even f/6.3) and it's still not sharp.

[1%]

Compare ALL-I vs gop-3 and 4. In 600D tests, all I resolves a little bit less detail... still better than stock tho. With 200Mbps I frames you have a bit more quality so may be better.

Glad someone else noticed the rolling shutter. It does seem to increase but I didn't know if it was better or worse than stock.

[feureau]

I did some tests with the All-I port.

These are jpegs at quality setting 10.
1080p at 25FPS
Bitrate: CBR 20x
1/250s, f/5.6, ISO 160
18-135mm lens at 135mm

1) Newspaper closeup: unsharpened


Sharpened:

That's some really good sharpening work there. What did you use to sharpen these up? I can't believe there's practically no moire on these patterns. Did you use  an anti-aliasing filter such as the VAF7D?

[akshayverma1]

That's some really good sharpening work there. What did you use to sharpen these up? I can't believe there's practically no moire on these patterns. Did you use  an anti-aliasing filter such as the VAF7D?

I sharpened these with After Effects' built in unsharp mask at 200% (wanted to over sharpen a bit), radius: 1px, threshold:0.
Didn't use any anti-aliasing. I guess it's not there because the details are rather large (the pictures aren't cropped). Ink droplets on a newsprint are quite visible.

[P337]

I sharpened these with After Effects' built in unsharp mask at 200% (wanted to over sharpen a bit), radius: 1px, threshold:0.
Didn't use any anti-aliasing. I guess it's not there because the details are rather large (the pictures aren't cropped). Ink droplets on a newsprint are quite visible.

Hey Akshayverma,

Thanks for the tests what bitrates were you averaging and peaking at for those jpegs?  And I assume those were single frames taken from a video right?

@P337: Yes! I agree anything lower than f/8 would soften the footage, but a still image taken at the same aperture isn't that soft. I think it's because of the line skipping, as ilguercio pointed out. This time I shot at f/5.6 (which I think is the sharpest for this camera, or even f/6.3) and it's still not sharp.

Right, line skipping will also effect it as well as the 4:2:0 subsampling and frame compressions.  Even at 1:1 compression rates, about 597 Mbps, it would still only have half the chroma pixels :/ So I doubt it will ever equal to photo image quality

[RenatoPhoto]

This whole thing of bitrates is a mystery to me.  It is great that this additional tool has
been added to 7D and now I am trying to figure out how to make the best of it.  I was really
excited with a previous post where some amazing results were presented with ISO 12800
where the denoising had improved significantly by the use of higher bitrates.  Unfortunately
I ran a test myself and cannot find any such results.  I import the videos to vegas 12
and denoise with Neatvideo and the results are the same shitty noise between 1x or 20x for IBP and
the noise of ALL-I seems worst compared to the standard canon video (no ML) until you get up to
10x CBR. I hope someone can provide more details on their workflow to getting such great results
in the high iso world by using the higher bitrates.

Also I have done some testing at low iso (100) with all the options and again the final results do
not convince me that using higher bitrates produces any better video.  For all comparisons I select an area of
about 320x240 pixels of the whole 1920x1080 so I am looking at at 6x factor where I can see artifacts
and noise very clearly. 

I also ran some test on the 5D3 and found out that the ALL-I video quality (artifacts) is worst and therefore
I will only use IBP video with my 5D3.

I don't have the video analysis tools but by looking at the video properties I found the following bitrates for
all my test subjects:


CBR       IBP        ALL-I
            kbps        kbps

No ML    97573                  updated -->(54588 IBP)
1x          49541      43228
2x          88795        95941
3x          89605        150578
5x          90403        228832
10X        91294        268863
15X        96169        272493
20X        98747        243947

Tested 7D with 24mm sigma f1.8

Note that the standard 7D video has a bitrates of 97573 54588 kbps which is below 2X bitrates and above 1X for both IBP and ALL-I.  In All-I video the bitrates go significantly higher but the artifacts show big time until you get above 2x and thereafter I do not see any improvements as bitrates go up. 

Since the video analysis software presented in this post, is showing significant improvements, I would assume that these improvements are at such small pixel size that my eye cannot distinguish and therefore I do not see the improvements that the analysis would seem to point.  Again I am looking at 6x magnification and cannot match VISUAL IMPROVMENTS TO SOFTWARE ANALYSIS NUMBERS.

Any help to clarify this mystery would be greatly appreciated.

Updated since I made a mistake which was pointed out where the standard video on 7D does not go up to 97Mbs.  I must have mixed up some of the videos.

[P337]

Just to be clear, is a lower number for the "flush rate" speeding up the buffer? Is Flush=1 the fastest or is it Flush=20 and what is Canon's standard flush rate? 

Cause whenever I try to record with the flush rate any lower than 3 I get an err70 and when I set flush to 20 I see my buffer percentage stabilizes around 30% but when Flush is set to 4 it stabilizes around 5%.

Anyway my 1st CF card is benchmarking at 15MB/s (8, 8, 15) I set to CBR 20x and Flush to 3; it buffers out at 100 Mbps Average (125 Mbps peak instant) which is only equal to CBR 2.5x.  Not a very good card for testing high bitrates but I got another one that's benchmarking at 45MB/s that I'll try later.

[g3gg0]

@P337:
it is the number of frames after that buffers should get flushed.
there is no "better" value.
if your buffer runs full (because of high bit rate), but your card could write faster, decrease the value.
lower values so are not "speeding up buffer" but clearing the buffer more often.
canon default is after [fps] frames. as said in main post - every second.

@RenatoPhoto:
what makes you think that the noise is worse?
more noise in video means that there are more details. in this case its high ISO noise (?)
thanks for testing.

[P337]

...I don't have the video analysis tools but by looking at the video properties I found the following bitrates for
all my test subjects:

CBR       IBP        ALL-I
            kbps        kbps

No ML    97573        54588
1x          49541      43228
2x          88795        95941
3x          89605        150578
5x          90403        228832
10X        91294        268863
15X        96169        272493
20X        98747        243947

...

Note that the standard 7D video has a bitrates of 97573 kbps which is close to the 20X bitrate.  In All-I video the bitrates
go significantly higher but the artifacts show big time until you get above 2x and thereafter I do not see any improvements
as bitrates go up.

...

Any help to clarify this mystery would be greatly appreciated.

Something is not right here;  7D with no ML isn't 97 Mbps (h.264 Baseline 5 has it capped at 50 Mbps) and the 7D doesn't have "ALL-I" without Magic Lantern and never had IPB even with ML.

Yes "All-I" on the 5D3 will likely have less detailed image quality than its IPB mode.  We touched upon that in this thread: http://www.magiclantern.fm/forum/index.php?topic=3103.400

597 Mbps for a true 1:1 compression of a 1080p 4:2:0 image at 24fps
35 Mbps for the average 17:1 compression with "IPP" (Which is Canon's stock settings)
86 Mbps for the average 7:1 compression used for "All-I" (Canon gives around 95 Mbps for the 5D3 "All-I" mode)
and
We've seen the 7D maxed at about 250 Mbps which is about a 2.5:1 compression!

...since All-I "does less compression work" and since most compression artifacts and glitches happens due to round off errors during compression it would be expected that All-I would reduce the opportunity for artifacts. 

The problem is that All-I needs quite a bit more bandwidth than IPP and if it doesn't get it then most fine details get lost in the "All-I process". For example, the GH2 with h.264 IPB at 88 Mbps gave a near 1:1 compression look but the 5D3 with h.264 All-I at 95 Mbps didn't.

Being VERY untechnical about it, from what I've "heard":
ALL-I at 100 Mbps would look similar to IPP at 50 Mbps and IPB at 25 Mbps

[P337]

@g3gg0
Ah, I see.  So "Stock" would be like Flush = 24, which would flush the buffer every 24 frames which also happens to be every second when the recording is set to 24fps. 

So setting the Flush to 4 would flush the buffer every 4th frame instead of every second, yeah?

So it's clearing the buffer faster than once per second now right?

Thanks for clearing that up

[g3gg0]

what is the advantage of all-I anyway?

thats the reason why i asked that in the other thread
so all-i is good for what now? is modifying GOP useful at all?
or will a lower GOP value reduce blocking?

maybe just make it configurable and (pro) users can choose what they need.

@P337:
exactly

[P337]

Shorter GOPs are better for recording really fast or really erratic movements (like race cars) because it just records everything without having to try to "predict"your subjects movements (unlike with IPP and IPB) so there is less chance of artifacts caused by prediction errors (like ghosting). 

ALL-I will record the subject's movement more accurately but that needs more bandwidth.  If it doesn't have enough bandwidth, you'll still get the accurate movements but will get far more compression errors (like blockiness).

80% of the time I'd choose IPB (or IPP) but for those times you need "All-I" it's great to have, you need the bandwidth to use it though.

Also "All-I" is easier to edit and color grade cause each frame has its own pixels rather than having to depend on the frame that came before, or people can just transcode to an Intra codec in post

[1%]

Canon defaults for gop are FPS/2... so at 24 its 12 at 30 its 15 and at 60 its 30. These 3 do not need cache hacks and just work.

So setting the Flush to 4 would flush the buffer every 4th frame instead of every second, yeah?

Canon writes the buffer to the card after each GroupOfPictures is finished. So when you shrink gop it writes the buffer out faster. Hence all I writes every frame immediately.

will a lower GOP value reduce blocking?

I think higher bit rate should reduce blocking but you can control the deblocking filter independently, right now canon is doing it. I turn it off and don't get much blocking, go figure.

Also I have done some testing at low iso (100) with all the options and again the final results do
not convince me that using higher bitrates produces any better video.

Then don't be convinced and enjoy your 40mbps max video. Frame size is bigger, QP is lower everything on the encoder end is better. Where do you think that extra file size comes from? My 10MP camera vs my 20MP camera stills don't convince me either... except somehow raws are 20mb vs 10mb and go further in post.

You guys haven't pushed luma/chroma quality up yet, that should add a little bit. According to H264 technical docs it was like 17% increase... the player can't play those back either.

[feureau]

Welp, I did some testing on rolling shutter with and without the bitrate monster.

Settings:
Lens: 18-135 IS. (non-STM version)
ISO: 12800
Shutter: 1/4000
Aperture: f/5.6
Focal length: 135mm

First up: The default firmware:

http://i.imgur.com/nS3aa.jpg

Here's with the bitrate monster at CBR 20.0x and flush at 3, which I believe is the maximum that the ML can pull out of the camera:
http://i.imgur.com/mrbcI.jpg

Here's the two, overlaid on top of each other:
http://i.imgur.com/Zkrqr.jpg

There's a slight increase of rolling shutter in the bitrate monster's but I'd chalk that one up as:

a. Margin of Error due to that I had to do this by hand one setting after another, and not at the same time.
b. Very slight sample variance.

That I'd say there's practically no difference in rolling shutter.

Another thing that I can conclude from the test is that if you look at the grain pattern on the wood part, and the nail details, you can definitely see definition in the All-I version while it's all washed out in the default firmware version. Thus concluding that ALL-I have a definite net positive benefit in image resolution. Keep in mind that this is ISO 12,800.

In summation:

a. Rolling shutter difference is negligible or insignificant
b. ALL-I has a demonstrable benefit in resolution

QED. 

One thing I can't seem to explain is that the highlight seems to be blown out in the default firmware version, while it's more retained in the bitmonster. I'm not sure if something changes in the setting while I load ML (doubt it) or if high bitrate ALL-I somehow lets it retain more highlight detail.

UPDATE: I did some more quick test, and I'm seeing it: The bitmonster seems to be retaining highlight details much better at ISO 12800.

UPDATE2: false alarm on the highlight thing. Apparently, under certain circumstances, after you record some movies with default firmware, then load up ML, and go to movie-live-view mode with ML, it would stop down the aperture by one stop. Not sure what triggers it, it happens rather randomly.

[Digital Corpus]

It might help to know that once you hit an aperture of f/8 the pixels in the 7D start to diffract which will blur the image a bit more and more the further you go passed that threshold.

It's ~f/6.9 for the 7D due to the 18 MP sensor. However, with 1080p video, you're then talking about effectively a ~2.1 MP sensor so the airy disk can technically be larger. On top of that, since the video is created with line skipping, anything over f/7.1 may still in hibit resolution. One more thing I want to look into aside from the pixel-peeping effects of the VAF-7D.

[rebel_rob]

i got an error code at playback with the bit rate at 19.9. ISO was at 4000 with highlight tone priority. i don't know if anyone else has gotten something similar but thought i'd report on it. the highest iso i can use without the error on playback is 2500.

[akshayverma1]

Thanks for the tests what bitrates were you averaging and peaking at for those jpegs?  And I assume those were single frames taken from a video right?

Yes, those are single frames taken from video. Average bitrate for the newsprint was about 150. Peak was 170. For the tree, average was 210, and peak was 260.

I did some high ISO tests with well lit trees last night and got some amazing bitrates - 290 peak and 260 average. It's 35 seconds and 1.04GB.