Bit rate investigation

Started by Audionut, July 19, 2012, 04:54:03 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Andy600

What does the averaging setting do? I get buffer overloads when its on. I'm also noticing that I can get higher bit rates by starting on a less detailed scene like the sky then tilting to the thing I want to shoot. If I hit rec on the detailed scene the buffer usually overloads immediately.

Is there any way we can have selectable config files for fast switching between settings on camera similar to what Miyake has done with selectable binarys?

It would be useful for testing GOP settings and other things. Remembering what I set for each test shot is difficult after ding a few. At the moment I'm trying GOP3 with CBR at about 1.5 which is working out at about 50-60Mbps. Should I increase the P factor to be greater than I?

BTW 1% I'm not seeing any I-frames in Bitrate viewer with my settings. Is that good?
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

1%

Quote
If I hit rec on the detailed scene the buffer usually overloads immediately.

Yes, I noticed this myself... if you jam the buffer out immediately the recording will stop... if you gradually bring it up, it may setttle. Sometimes I point at dark scene right before testing something I'm not sure will work to let the MVR get started.

You're probably in GOP mode in BR viewer. Hit CTRL-F to get into frame mode.


As for encoder profiles its a very good idea once we squeeze everything out of this. I and P and D and Gop are SIZES not ratios. So increase P if you want bigger P frames, I if you want bigger I frames, etc. Gop sizes I'm not sure yet, one is calculated.

Averaging takes the average # for every mode and scales that, usually you end up with bigger numbers. I have to set up debugging to on/off before encoder profiles and maybe we can figure out how the sizes are calculated and what they mean.

One thing I just noticed... CBR (qscale between -6 and -8, BR ~90-107mbps) VBR( qscale -16, BR 100-140mbps and less buffer overruns)... same scene.

Andy600

Thanks 1%. Good to know I'm on the right track with my testing. It's interesting about CBR & VBR on the same scene though my class 10 card isn't letting me hit the upper limits so i need to pick up something better. I can hit about 120Mbps before it stops.

Thanks for the heads-up on BRV, you're right, I was in GOP mode  ???

When you say about I, P D & GOP being sizes is it dimensions or bitrate size?
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

1%

I think size is frame size or something like that.

Just found there are 2 ways to stop the encoder... bitrate too high and complexity too high. I can go along and film not too complex scene at Q-16 and then buffer will instantly fill (from 14%) when I hit a bunch of green trees waving. More control over max requestable bit rate might help with buffer stops, as would figuring out what actually overloads the encoder.

Andy600

Yes I found the same thing.

Have you noticed that at default settings GOP is dynamically set. Moving onto high detail drops GOP to 2 or even 1 sometimes. I filmed something earlier that produced a bunch of I-frames only for a second or 2 then they started to spread out.

Ignore that. Just checked again and it was P-frames colored the same in AVInaptic
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

Leon

Unless I'm misunderstanding you, I think your "2 ways to stop the encoder" are the same thing:

If you encode at a certain quantiser (Q-scale value) then it uses as much bitrate as it needs to maintain a constant "quality" (amount of information loss) within that macroblock (16x16 part of the picture).  If not much is happening then it does not need much bitrate so you don't fill the buffer.  As soon as you present it with a complex scene it suddenly requires much more bitrate to maintain the quality.  Increased bitrate is caused my increased complexity.  Complex patterns, motion and noise (including high ISO) all contribute to increasing complexity.

I also wanted to clarify something about I/P ratio:  In x264, the I/P ratio is how much it increases the *quantiser value* (not bitrate) for I-frames compared to P-frames.

In x264, the quantiser runs from 0 (lossless encoding) to 51 (very poor quality), with 18-20 being sensible for DVD encoding and about 22 being used for 1080p BluRay encoding.  (Of course you would normally encode using a Constant Rate Factor (CRF) rather than a Constant Quantiser Parameter (CQP), which Canon DSLRs almost certainly use.)  Changing the CQP value by 1 usually affects filesize by about ~9%.  Therefore using CQP 17 is about 1.09 * 1.09 * 1.09 times bigger (about 1.3x bigger) than using CQP 20.  Canon is of course using its own scale (no surprise, lol) and the increments of Q-scale may be quite different.

Did you work out what D1 and D2 are?  Maybe they are values for the inter/intra luma quantisation deadzone?  If so, then lowering the threshold would likely improve fine detail considerably but increase bitrate.

See:  http://mewiki.project357.com/wiki/X264_Settings#deadzone-inter.2Fintra

In any case, I would think that all these parameters probably relate directly to parameters that x264 can take.  For example, GOP size is called "keyint" in x264.  Specifically, I wonder what your I and P parameters could be - you can't set the size of I & P frames (only the quantiser ratio).  Setting the GOP tells you when your I-frames are, so all your other frames must be P-frames (since there are no B-frames).

See this page for all the x264 options: 

http://mewiki.project357.com/wiki/X264_Settings

1%

It could be the same stop but seems like the buffer fills instantly and BR before wasn't that high. Like 90 to 110 then suddenly 100% full... so either it requested something crazy like 250-300 or crashed or both.

D values I'm not sure yet... if you lower them the BR+quality drops, raise them and it raises to new heights. Both are also set the same for some reason. Next build is going to have these values on screen (setting) so maybe we'll make sense of it all.

Andy600

I'm thinking we need some kind of repeatable test, something always moving but can be filmed on a locked off tripod. Water from a tap?
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

1%

I did lamp in the rain... but its not raining now :( or should that be :)

Now you can view I/P/D, etc with debug on in 24p/30P

https://bitbucket.org/OtherOnePercent/tragic-lantern/downloads/autoexec.bin.600D.Exfat.Debug *visible while recording*

Andy600

I ran lots of tests but I cant see any difference other than bitrate and GOP increase/decrease. The settings were definitely taking effect but no difference in the end results...so far  :-\
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

Leon

I use TV static to test reliability of different CBR values with different SD cards, with the camera on a tripod.  It's very repeatable and I figure it's about as complex as a scene can get.  Analogue TV is no longer broadcast in my area, so I can switch my TV to analogue tuner and select any station and I just get that snowy erratic static.  Horribly difficult to encode  :)

My set-up is:

• Set TV to detuned analogue static
• Mount camera on tripod ensuring it is perpendicular to centre of TV and at a distance where the static fills the whole frame.  (I always use the same lens, the 50mm f/1.8 for consistency.)
• Set ISO high (I've been using 3200 but 6400 might be better)
• Set aperture to something like f/8 to f/11 to ensure the picture will be crisp with as much detail as possible (but ensuring exposure is correct)
• Perhaps controversially, I set shutter speed higher than usual (1/100) because I think this should capture even more detail (less temporal motion blur) and be harder to encode
• Set PictureStyle (eg Neutral 0,-4,-2,0) and ensure HTP is off.  Probably should turn off ALO too but I haven't bothered.
• I keep audio on because I'll almost always want it on, even just as an emergency backup or for synching audio from my external recorder.  (Mostly I just have my external recorder plugged in and the quality's good enough.)
• I've been testing at 30 fps, on the basis that if it can do that it will be fine with 25 or 24, and 30 fps is sometimes useful to slow down action a bit or reduce the effect of camera shake.
• Lastly, but importantly, I ensure the focus is very accurate, by using live-view Contrast Detection AF rather than "fast" phase detection AF.

From this (with the December 2011 firmware) I established that my "Class 6" Verbatim card could only manage CBR 0.8x reliably.  My UDMA 1 SanDisk manages CBR 1.4x (with audio) and my Class 10 Transcend is variable, managing 1.1x sometimes and 1.3x other times depending when it was formatted.  Of course setting a lower GOP will probably improve these.

1%

Here is a 200mps clip: http://upload.g3gg0.de/pub_files/fc7fd4c62859b2178c6acb0f4af1ad0f/200Mbps_MVI_2425.MOV


When BR is high and you look at videos from the same series which were lower there is a difference, its just hard to quantify and consistently test. Static sounds good for testing stability, but how is it for comparing detail preservation, etc.

Also, gop size affects how motion is displayed.

deleted.account.0

Would this tool help?

MSU Video Quality Measurement Tool (VQMT) is a program for objective video quality assessment. It provides functionality for both full-reference (two videos are examined) and single-reference (one video is analyzed) comparisons.

http://compression.ru/video/quality_measure/video_measurement_tool_en.html
There's even a free version available.

If the camera was locked off on a Tripod & the same scene captured at various bit rates, this may work very well.




Audionut

Quote from: simonrfenton on August 29, 2012, 04:51:52 AM
Would this tool help?

Unfortunately no.  It uses metrics (PSNR, SSIM etc etc) to determine quality.  Good in theory, but you would never be able to record exact scenes.  Any slight deviation would effect the results.

Also, you need a recording to base off.  There is no "perfect" copy to base the experiments off.

Audionut

Quote from: 1% on August 29, 2012, 02:06:50 AM
Here is a 200mps clip:

How about finding a static scene with tons of detail to test also.  Something like what I did in the OP.

3pointedit

Could you use a recording of a water fall, in close up? Very noisy temporaly, and if shot off a screen competely repeatable. You would have to start with a pretty hi res clip first though.
550D on ML-roids

Rush

Easiest way to shoot static - close lens cap + set extreme ISO (around 100000)
Greetings from Russia!

1%

QuoteHow about finding a static scene with tons of detail to test also

Probably need 1 test for detail and one test for motion and 1 test for sustainability.

Quoteasiest way to shoot static - close lens cap + set extreme ISO (around 100000)

600D has no 100k, I think 1250-2500 is probably as high as anyone will go normally. 1250 and below for most cases otherwise you get noise. Also encoder will just be encoding junk... I can record really high MB of black/underexposed but that doesn't translate into real performance in the field. Recording will just stop when scene is correctly exposed and there is movement.

Water and trees are good candidates because of how erratic/detailed they are.

QuoteSetting the GOP tells you when your I-frames are, so all your other frames must be P-frames (since there are no B-frames).

Yep, with the analyzer its either I or P.

QuoteSpecifically, I wonder what your I and P parameters could be - you can't set the size of I & P frames

Lenght of nal unit for I and P? The numbers are pretty big and in that range.

Rush

Quote from: 1% on August 29, 2012, 06:39:36 PM
600D has no 100k, I think 1250-2500 is probably as high as anyone will go normally. 1250 and below for most cases otherwise you get noise. Also encoder will just be encoding junk... I can record really high MB of black/underexposed but that doesn't translate into real performance in the field. Recording will just stop when scene is correctly exposed and there is movement.

There is no difference between any movement and noise. Noise is real stress-test, because amount of noise have great impact on required bitrate.
600D can set ISO up to 632000 (higher value decrease brightness for me). Just open ISO submenu, set analog ISO to 3200 and increase "ML digital ISO".
Greetings from Russia!

1%


nanomad

EOS 1100D | EOS 650 (No, I didn't forget the D) | Ye Olde Canon EF Lenses ('87): 50 f/1.8 - 28 f/2.8 - 70-210 f/4 | EF-S 18-55 f/3.5-5.6 | Metz 36 AF-5

deleted.account.0

1% - Any chance of another unified build? I'd like to do some testing on a 550D.

deleted.account.0

Quote from: Audionut on August 29, 2012, 06:32:18 AM
Unfortunately no.  It uses metrics (PSNR, SSIM etc etc) to determine quality.  Good in theory, but you would never be able to record exact scenes.  Any slight deviation would effect the results.

Also, you need a recording to base off.  There is no "perfect" copy to base the experiments off.

Is there a way to load in a RAW file off SDCARD into RAM, pass this through the encoder at various compression settings, then measure the resulting difference between these clips with the MSU tool? This would give a repeatable starting point.

Rush

Quote from: simonrfenton on August 30, 2012, 02:38:09 PM
Is there a way to load in a RAW file off SDCARD into RAM, pass this through the encoder at various compression settings, then measure the resulting difference between these clips with the MSU tool? This would give a repeatable starting point.
For real, you'll see no difference with still scene - with bitrate >20mbps it will be perfect
Greetings from Russia!

1%

Actually it does seem possible but not sure how to implement it, would it be jpeg or yuv. I'd guess gop would be set to 2 and then you get one I frame and one P frame.