7D at Halloween: The Bitrate-Monster (EXPERIMENTAL)

[g3gg0]

Okay..

As some guys asked for thorough tests of what the high bitrate reported in this thread really look like,
i decided to post an EXPERIMENTAL release of the 7D bitrate hack.
i repeat. EXPERIMENTAL.

what does that mean?
1) it is only intended for people with decent technical knowledge, because...
2) ... you have to be aware of the possible negative side effects like overheating ....
3) ... or some unrecoverable crash of your camera.
4) and it would only make sense to test if you know what bitrate means
4) although i don't think this will happen - the risk for your camera is higer as with the alpha
5) the usual "be warned" stuff i always tell you still applies

okay. i have to say that, you know.

back to the bitrate hack.


what is it about?
with ML we can drive the bitrate up to factor 3.0x then, depending on your card speed, recording stops.
this is because of the recording buffers that fill too fast.
these buffers are cleared about ONCE per SECOND - if you set 25 fps, they will get written to card after the 25th frame.
if we would set e.g. 10x rate, the buffer would be full after half a seconds or so.
thats the reason for the recording that stops.

what does this hack do?
it flushes the buffers more often. this is configurable.
i pre-set it to 4 frames which works quite well with my 30MiB/s CF card and a rate of < 9.0x.
on 7D this also requires some cache hacks in master firmware.
porting that other models is a lot of simpler. (imho)

what do you want to test?
a) test how far your card can go up.. set bit rate higher and test high detail scenes. report what your highest stable bitrate was and your card type (with benchmark speed).
b) check if the hack is worth its effort. is the video quality good? or would we stick better with 3.0x and this hack is useless?

known issues:
- ERR70 may happen if your card is too slow and/or the flush rate is too low
- important: disable sound recording, else you will get an ERR70 too
- it seems to be not CBR, but VBR although it says CBR


i really would love to see some deep analysis of the videos and your conclusion - is it worth to get implemented in all models?
or should i just drop this code and stick to other things?

here is the DL link for this experimental version.

[ilguercio]

I thought 3.0x was the maximum allowed but looks like it's nothing compared to 9.0x.
Files are going to be HUGE and i unless you're doing a short movie and have a high end computer to process the files most of the people don't want to shoot as such high bitrates. Don't get me wrong, your release has the purpose of being tested by movie guys to see if such stuff is worth. What are your early opinions?

[1%]

I knew CF would stomp SD. I bet 7D can shoot max quality most of the time and never overflow.
When I was just scaling CBR eventually the numbers overflowed and turned negative. But 7d is probably more like 5dII with different MVR config.

[ilguercio]

9.0x CBR is something like 3Gb per minute, quite a lot for any "casual" application. Let's see what our users can do with that.

[Simon]

Looks very interesting ! Could you please port it to work on 550d/T2I ?

[nanomad]

Yes indeed, it would be nice to test how far a stupid camera like the 1100D can go

[Simon]

 

[1%]

Can't go any higher than maximum QP of the encoder. Low end cameras are like 600D. Theoretically I could do BR that high too but SD card is too slow or not enough data unless you crank the ISO.

CBR 9x is just going to make the predictor set highest quality all the time, I tried it, values overflow.

5DII and 7D have a slightly different mvr config + faster card rates.

[RenatoPhoto]

Ran some simple test with Lexar Professional 1000x 32GB
card Benchmark  20.8, 20, 53.8
Recording at night iso 6400

Tested the Defult values  =  ERR70

UPDATED October 29:  After I disabled the Audio then I was able to record much longer videos.  Disregard the results below.

Test flush rate (CBR, 1X, OFF, 4, 70%)
Increase flush rate to 8 = record 4 sec no playback on camera
Continue to increase flush rate @ 10 = record 4 sec and 1 sec playback on camera
Continue to increase flush rate @ 15 = record 6 sec and 1 sec playback on camera
Continue to increase flush rate @ 20 = record 8 sec and 1 sec playback on camera
It would seem that higher flush rate allows longer recording and buffers do no fill up.

Test BuffWarnLevel (CBR, 1X, OFF, 20, 70%)
Using previous best settings of Flush rate= 20
Increase BWL 90% = record 8 sec and 1 sec playback on camera
Increase BWL 100% = record 8 sec and 1 sec playback on camera
Decrease BWL 30% = record 8 sec and 1 sec playback on camera
Conclusion, BWL has no effect on recording

Test CBR begining settings ( CBR, 0.1X, OFF, 20, 90%)
Increase CBR 0.1 = record unlimited time and 1 sec playback on camera
Increase CBR 0.2 = record 15 sec and ERR 70 shut off
Increase CBR 1.4 = record 5 sec and 1 sec playback on camera
Increase CBR 2 = record 3 sec and 3 sec playback on camera
Increase CBR 3 = record 1 sec and no playback on camera - Camera locks up when trying to play
Increase CBR 4 to 9 = record 1 sec and 1 sec playback on camera
Increase CBR 10 = record 1 sec and no playback on camera
Increase CBR 11 to 20 = record 1 sec and 1 sec playback on camera
Conclusion: longest recording at cbr=1, longest playback at cbr=2

Tomorrow I will try to do some recording in the afternoon and to look at video

[g3gg0]

please make sure that audio recording is disabled
else you will face an err too

[akshayverma1]

I'm using a SanDisk Extreme 8GB 60mbps card. I just shot some trees.

ISO 160
Shutter 1/250
Aperture f/11
Flush rate: 4
CBR 20.0x
Focus peaking, magic zoom, histogram, waveform and bitrate display were ON
Audio OFF

I recorded 1 minute, 42 seconds of handheld footage. Recording was stopped by me (it would keep recording for the whole 4GB). The file size is 2.38GB.

As g3gg0 rightly said, it's actually VBR. I got a maximum bitrate of 218 while the average was around 170. Although I focused it pin-sharp, the footage is blurry as is all 7D footage. Visually it looks just like ordinary 1x BR to me.

Could anyone please guide me about doing a better comparison?

[ilguercio]

Don't expect the footage to be pin sharp just because you raised the CBR value.
The problem at the basis on that fuzzyness is how the encoder scans the sensor. It's not downsampling it but skipping lines so you're not going to have amazing quality just because of the bitrate increase.

[akshayverma1]

Thanks for the answer, ilguercio.

I shot some more footage, this time water coming from a tap with lots of fast-moving bubbles. The difference is clear between bitrates lower than 1x, and 1x. There's a lot of compression artifacts at lower bitrates.

I also shot a ceiling fan at 1/500s. Again, there is only minimal difference between 1x and 20x. I have tried applying an LUT (shot in cinestyle), color grading, denoising, and over-sharpening. 20x is only minimally better; it still has some artifacts on the moving blades. Is this because it's VBR?

[g3gg0]

thanks for that investigation.

has anyone an idea where the higher bit rate comes from?
e.g. are all frames effectively (or nearly) I-frames with only few P-blocks?

i have a demo version of the tool AVC Analyzer that is made especially for this kind of analysis,
but i can not analyse the files because the tool crashes with these files

[1%]

Avinaptic will tell you the % of frames that are X QP at least.

Details were small if you look through 600D tests. Visually hard to tell until you grade/convert. Only visible when you zoom in. Rencoding to things like cineform keeps the file sizes so there is more data in there.

[g3gg0]

okay, i shot a comparison that AVC Analyzer could handle.

now some details what the higher bitrate causes.

i shot two videos of a wall with little details, high iso and high f-stop value.
one video with 1.0x and one with 10.0x

first frame (#0): both videos have an I-frame with ~500KiB size. QP of this frame is in both videos 18.

second frame (#1): is P and uses in both videos ~430KiB. It consists of ~89% Intra MBs (MacroBlock) and also both have QP 18.

third frame (#2): the things change.
both videos still have ~89% Intra-MBs. this doesnt change at all in the whole video.
but the QP and thus the size differs slowly. the 1.0x video goes up to QP 22 whereas the 10.0x video goes down to 17.
for the 1.0x this are 262KiB whereas 10.0x uses 490KiB per frame.

the QP, size | QP, size for 1.0x | 10.0x now continues like this:
25, 164299 | 16, 551711
24, 185327 | 15, 611007
24, 186271 | 14, 667463
24, 185715 | 13, 741415
24, 185215 | 12, 797839
24, 185059 | 11, 873799
24, 184827 | 10, 956451
24, 185711 | 10, 952903
24, 186119 | 10, 952615

next I-frame (#12)
24, 237231 | 10, 1000675


so now it is clear that the data is spent for QP parameter which affects quantization.


here some graphical analysis results regarding image quality:
1x
3x
5x
10x

[nanomad]

Looks like the higher nitrate has less denoising but with the same blurriness...I wonder if that's something good or not

[g3gg0]

no, blurriness and denoising both go down to 50%

[nanomad]

Yep, I wonder what  i was reading... any chance you can do a screenshot at 3x or something in the middle?

[g3gg0]

sure, added the results to the post above

[1%]

So no lower QP than 10? In 600D when BR gets high, CBR likes to set qscale of +25. You can shoot ALL-I too and then look at frame sizes.

[g3gg0]

uhm what? qscale 25 is the worst quality.  ah ok 26 is the worst, so 25 is not really better.
and thats the default value?

[feureau]

This high bitrate is AWESOME!!! \o/

I've only been playing with this for a couple-a hours, but it seems my Transcend 133x cards can't record more than CBR x2.5. However, my Sandisk Extreme can record at CBR x20.0 for the full 4GB. As mentioned, it seems it's actually recording on VBR, so I set my 7D to ISO 12,800 to maximize noise, forcing high bitrate with the following settings:

Code Select Expand

# Magic Lantern v2.3.NEXT.2012Oct28.7D203

Start          : 2012/10/30 19:44:03
Lens name      : EF50mm f/1.8 II
ISO            : 12800
Shutter        : 1/48.194s
Aperture       : f/14.0
Focal length   : 50 mm
Focus distance : 0 mm
White Balance  : 1 - Sunny, Magenta 0, Blue 0
Picture Style  : UserDef2 (0,-4,-2,0)
FPS            : 23.976
Bit Rate (CBR) : 20.0x

CSV data:
Time,ISO,Shutter,Aperture,Focal_Len,Focus_Dist
19:44:03,12800,48,14.0,50,0
19:44:09,12800,48,16.0,50,0
19:44:10,12800,48,18.2,50,0



I'm shooting f/14.0 indoor at night. And the result is bleepin' awesome! I've never seen ISO 12,800 denoise so cleanly:

> http://i.imgur.com/gKEyi.jpg


Here's the original frame, for comparison:

> http://i.imgur.com/UwtiV.jpg

Compression noise so low that it's easy for the denoiser to clean sensor noise. I'd say it's pretty much usable, especially after some post processing.

What's weird was that the POS screen and a lot of the areas near the lights looks blown out on the screen when I shot this, but it looks just fine on the video file.

I put this particular file through AVInaptic for analysis. I'm not sure if any of these means anything (especially the errors), but in case anyway want to take a look, here's the analysis result.

My 7D's processor stabilizes at 15% with flushing at every 4 frames at CBR 20. Maybe there's room to increase the CBR (well, VBR?) even higher?

One thing I noticed with high ISO, high bitrate is that, it seems the rolling shutter becomes far more severe compared with regular bitrate, but I haven't done a comparison yet, so this is just a feeling.

I also did a stress test by shooting at 12800 ISO and I kept panning the camera around so the picture would kept changing. It records for 3min. and11sec. before it hits 4GB.

Thank you for all of your hard work, g3gg0! \o/

Let me know if there's anything you want me to try out.



[ About file ]
Name: MVI_6586.MOV
Date: Tue, 30 Oct 2012 20:47:25 +0700
Size: 474,530,285 bytes (452.547345 MiB)

[ Magic ]
File type: ISO Media, Apple QuickTime movie

[ Generic infos ]
Duration: 00:00:19 (19.26925 s)
Container: Apple QuickTime movie (fast start)
Creation time: Wed, 31 Oct 2012 02:44:03 +0700
Modification time: Tue, 30 Oct 2012 20:47:21 +0700
Total tracks: 1
Track nr. 1: video (avc1) []

[ Relevant data ]
Resolution: 1920 x 1080
Width: multiple of 32
Height: multiple of 8
Average DRF: 10.095238
Standard deviation: 0.755656
Std. dev. weighted mean: 0.737784

[ Video track ]
Codec: avc1
Resolution: 1920 x 1080
Frame aspect ratio: 16:9 = 1.777778
Pixel aspect ratio: 1:1 = 1
Display aspect ratio: 16:9 = 1.777778
Framerate: 23.976024 fps
Bitrate: 197005.307005 kbps
Duration (bs): 00:00:19 (19.26925 s)
Qf: 3.962559

[ Video bitstream ]
Bitstream type: MPEG-4 Part 10
SPS id: 0
  Profile: Baseline@L5
  Num ref frames: 1
  Chroma format: YUV 4:2:0
PPS id: 0 (SPS: 0)
  Entropy coding type: CAVLC
  Weighted prediction: No
  Weighted bipred idc: No
  8x8dct: No
Total frames: 462
Drop/delay frames: 0
Corrupt frames: 0

P-slices: 423 ( 91.558 %) ##################
B-slices:   0 (  0.000 %)
I-slices: 39 (  8.442 %) ##
SP-slices:   0 (  0.000 %)
SI-slices:   0 (  0.000 %)

[ DRF analysis ]
average DRF: 10.095238
standard deviation: 0.755656
max DRF: 18

DRF<10:   0 (  0.000 %)
DRF=10: 453 ( 98.052 %) ####################
DRF=11:   1 (  0.216 %)
DRF=12:   1 (  0.216 %)
DRF=13:   1 (  0.216 %)
DRF=14:   1 (  0.216 %)
DRF=15:   1 (  0.216 %)
DRF=16:   1 (  0.216 %)
DRF=17:   1 (  0.216 %)
DRF=18:   2 (  0.433 %)
DRF>18:   0 (  0.000 %)

P-slices average DRF: 10.085106
P-slices std. deviation: 0.689222
P-slices max DRF: 18

I-slices average DRF: 10.205128
I-slices std. deviation: 1.264495
I-slices max DRF: 18

[ Profile compliancy ]
Selected profile: MTK PAL 6000
Resolution: 1920 x 1080 > 720 x 576
Framerate: 23.976024 <> 25
Buffer underflow: 00:00:00 (frame 0)
Buffer underflow: 00:00:00 (frame 1)
Buffer underflow: 00:00:00 (frame 2)
Buffer underflow: 00:00:00 (frame 3)
Buffer underflow: 00:00:00 (frame 4)
Buffer underflow: 00:00:00 (frame 5)
Buffer underflow: 00:00:00 (frame 6)
Buffer underflow: 00:00:00 (frame 7)
Buffer underflow: 00:00:00 (frame
Buffer underflow: 00:00:00 (frame 9)
Buffer underflow: 00:00:00 (frame 10)
Buffer underflow: 00:00:00 (frame 11)
Buffer underflow: 00:00:01 (frame 12)
Buffer underflow: 00:00:01 (frame 13)
Buffer underflow: 00:00:01 (frame 14)
Buffer underflow: 00:00:01 (frame 15)
Buffer underflow: 00:00:01 (frame 16)
Buffer underflow: 00:00:01 (frame 17)
Buffer underflow: 00:00:01 (frame 18)
Buffer underflow: 00:00:01 (frame 19)
Error: Too many violations

This report was created by AVInaptic (18-12-2011) on 30-10-2012 21:17:58

[1%]

Yep, +25 is pretty bad but not the worst quality. QP of 50 or so is the WORST quality but I don't think it normally can set that. Kinda why I went away from CBR, sometimes it got wonky. Here it looks to be pretty decent.  Gettin QP10 with fewer cache hacks.

Like above post, high BR let really bad night footage process into something useful.

Maybe there's room to increase the CBR (well, VBR?) even higher?

Sort of, but not the CBR way. Once you hit 100% QP 10 there's not much higher to go. I think frame quality goes to 87 and highest qscale "cbr" can set is 112. But QP doesn't change over that range, I think frame gets a little bigger but that's it.

[g3gg0]

this would make sense for astrophotography