Bit rate investigation

[ilguercio]

1%
Got anything new for us?

[1%]

Not yet, blame work.

[1%]

Ok, made a new unified with everything scalable individually. Watch for MVR resets. The CBR "scale" does nothing since everything is separate. Changing D values makes MVR reset or produces giant bit rates, maybe we'll find out why in testing.

Let me know if you find anything interesting.


http://www.qfpost.com/file/d?g=adnsX5lXl

[Chucho]

Does any body know how [JPCORE] works? It has a lot of H.264 libary functions like JporecoreSliceqpd which is the Slice QP delta Quantization scale it range is -26 to 25. Also JpcorePicpyPicpcD which are the picture py an pc chroma QP offset for the loop-filter it range is -12 to 12. And then you have JpcoreDbfalphaDDbbetaD which is connected to Mvrsetdeblockingfilter the loop-filter alpha/beta table offset are -6 to 6. Making the deblocking filter stronger should improve the muddy shadows. Also MrvSetGopOptSizeFULLHD is set to 12 setting it to 1 should give you all I-frames no?

[1%]

Its worth looking into. Might make more real changes.

; **'mvrSetGopOptSizeFULLHD GOP_OPT_SIZE_FULLHD([0]=%d, [1]=%d, [2]=%d, [3]=%d, [4]=%d)'

Aren't all the arguments of this function just the gop opts?

Does it have another address in the MVR config?

Mvrsetdeblockingfilter the loop-filter alpha/beta table
Where is that?

somewhere here?

*(-4 + sp0) = lr0
*(-8 + sp0) = unk_R5
*(-12 + sp0) = unk_R4
*(-16 + sp0) = arg3
*(-16 + sp0) = BYTE(arg0->off_0x4)

where is sp0?


Maybe:

aAJ_JpcoreSliceqpdD_Qscale_0x00_to_0x20.off_0x1 /*0x6CE1*/ = BYTE((arg0 & 0xF))
aAJ_JpcoreSliceqpdD_Qscale_0x00_to_0x20.off_0x2 /*0x6CE2*/ = BYTE((arg1 & 0xF))

[ilguercio]

Ok, made a new unified with everything scalable individually. Watch for MVR resets. The CBR "scale" does nothing since everything is separate. Changing D values makes MVR reset or produces giant bit rates, maybe we'll find out why in testing.

Let me know if you find anything interesting.


http://www.qfpost.com/file/d?g=adnsX5lXl

Trying it now but it is not easy to spot differences other than pure bandwidth.
Do the guys that have hacked the GH2 know something about those settings already?
They could help us somehow.

[Chucho]

GH2 encoder is software based Canon's is hardware based

[Rush]

Thank you for all that you do here. Hope your investigations will result in better bitrate control.

Just some test of current ML birate control, not with the version posted above (ISO 6400, 1/50, sharpen filter, 400% crop with 2x upscale)
Original 45 mbit

QScale16 165 mbit (in motion details looks even better)


High bitrate really helps to keep details with high ISOs

[1%]

Gop2 only changes when you average. Otherwise it doesn't seem to scale at all.

values are:

(int)5570568
(int)7261685 - when averaging.


I tried to travel to 0x6CE0 and see if de-blocking filter can be changed. I don't know if I'm in the right spot but the value there kept changing while recording.

How to set mvrSetGopOptSizeFULLHD? Would settign the gop opts to specifics instead of scaling produce equivalent results? I want to try all I frame footage.
 

[Rush]

How to set mvrSetGopOptSizeFULLHD? Would settign the gop opts to specifics instead of scaling produce equivalent results? I want to try all I frame footage.

I don't think that all-I will be good idea, because it requires higher bitrate with same image quality.
Increasing GOP length should help with increasing quality maintaining same bitrate

[1%]

If we can set it one way, we can set it the other way too.


Default GOP Options:


// 24P Defaults
// I 62992200 (3c12F48)  p 19594200 (12AFBD8)
//d1 262144 = 40000 d2 262144
//g0 3833856 (3A8000) g1 3309568 (328000) g2 2785284 (2A8004) g3 2260992 (228000) g4 17736704 (10EA400)

Possible D values:

//DSizes: 78643, 288358, 393216, 524288, 629145, 681574

When you change certain options others go to defaults or only to certain values.

http://www.qfpost.com/file/d?g=5N1nnCmy6


600D Fat32 - Adjust Deblocking Filter A/B

[1%]

Having set the deblocking filter correctly, I've gone onto picqpy, piqpc.

So far picqpy=20 is the highest without camera completely freezing. Bit rate increases for sure, original value was 26. Its calculated in jpeg quality from ~51 &0x3f but lower values like 12, 18, don't work.

PicqPC=0 by default. Anything else causes camera to hard freeze like it did with invalid picqpy

Setting through:NSTUB(0xFF1C94A8, mvrSetQscaleYC) allows setting of piqpc but for some reason doesn't change picqpy.

I think I have the array right.... 0 and 1 positions?
it calls:

jpeg_pic_quality(BYTE(*(arg0)), BYTE(arg0->off_0x4))


*ok, it appears either or. Set QPY to 20 get higher BR, set QPc +6 to -6 (58 after math)...

Altering QPC & QPY together causes hard lock. Which one is a better idea?

You can try QPy at 20.

https://bitbucket.org/OtherOnePercent/tragic-lantern/downloads/autoexec.bin.600D.Exfat.Audio.08.PiqQPy.bin

1st noticeable effect... preview is messed up. Videos OK.

Can now set Gop lenght and its verifiable + written. Bad news is when buffer overruns you get error 70. Bit rate seems lower, quality untested yet. Def way less P frames.

Code: [Select]

ASSERT: 0
at Fcreate\FcsMaker.c:2575, task MovieRecorder
lv:1 mode:20


Magic Lantern version : v2.3.NEXT.2012Aug23.600D102
Mercurial changeset   : c0cc0716e6ce+ (unified) tip
Built on 2012-08-24 00:43:15 by user@D610.
Free Memory  : 334K + 1396K

This happens after recording is stopped. I don't know if quickly switching it back at recording end would block the soft error 70 or if there is another way.

[Audionut]

Hi 1%,

Sorry for going awol, life got busy.  Any chance you can make a unified build?

Can now set Gop lenght and its verifiable + written. Bad news is when buffer overruns you get error 70. Bit rate seems lower, quality untested yet. Def way less P frames.

Increasing GOP length should result in more P frames for every I frame.

[1%]

I have to make everything work on other cameras. Deblock filter is ok but picqy, piqc are set by address. Also have to put averaging in a loop so its not just for 24p mode.

I just recorded all I frames and its REALLY easy on the buffer but bitrates are still high 90s/100s with no buffer warnings. Wonder what's up with quality.

-other observations.
1. BR is less controllable... can set to q-16 and 3.0x and camera will go at the speed of the card 40-120mbps with the buffer at 1-4%.
2. File size a little bigger, card constantly written (like buffer skipped).
3. Crop Mode (3x) + FPS override affect bit rate and can crash recording (~140-160mbps)
Graph looks like this:



If we can stop the error 70 and this doesn't adversely impact quality it would be a winner. I've never written such high BR video this consistently.

So what does increasing gop length do? Should be theretically better, right?
Gop size at 20.... buffer runs out much quicker, BR are much lower and look at this spiky thing.



Long ass thread from GH1/2 ppl on this topic:
http://www.personal-view.com/talks/discussion/421/official-low-gop-topic/p1

Compare frames:

http://www.sendspace.com/filegroup/e5qNpEKnFi%2BJnPzvHvBAow

[Rush]

1%, very nice! It looks like I-frames don't use buffer, and P-frames use buffer.
So it seems that we locked with quality - high bitrate ALL-I should look like low bitrate with GOPs...

[Audionut]

So what does increasing gop length do? Should be theretically better, right?

GOP is a Group Of Pictures.  Since the Canon codec only has I & P frames and the default GOP IIRC being 12, a GOP would look like this.

IPPPPPPPPPPP

And this just repeats, IPPPPPPPPPPPIPPPPPPPPPPPIPPPPPPPPPPP

I frames are larger then P frames.  As I mentioned earlier in this thread, they can be up to 4x larger.  I frames need to be larger as they are the reference frames for P frames.  P frames are not a complete picture, they only contain the detail that has changed since the last I frame.

Baring any scene detection that might be in the Canon code, if we increase the GOP size to a more general 10 x framerate (24fpsx10) 240, we can have 1 large I frame followed by 239 P frames.

Where as, with the default Canon GOP size, for the same 240 frames, we would have a total of 20 (large) I frames.

[1%]

I'll try 240 and see what happens. Did you notice quality fall in the sample pics? I don't think its quite as bad as low BR I/P. the frames are about the same size (~5mb) . Bit rate for I only goes up to 140 without buffer filling.

I need a consistant way to test what this is doing to quality, any ideas. I don't have curtains, test chart is hard to tell a diff.

[nanomad]

Quick idea: hack a script in bash that grabs a couple of frames using mencoder and then gives them to PDIFF
http://pdiff.sourceforge.net/

edit: Uhm, that only works if the camera and lighting stays still :/

[1%]

I'll have to try that.

240 is unworkable.. buffer fills at very low BR and recording stops. Qscale says +24? and BR is in the 1/2s (all options at .1x) before I can get it to not stop instantly.

This error 70 at every stop is getting quite annoying, going to have to trace it in that function and see where it produces Assert: 0 and if there is a way to patch whatever it compares

*Gop3 is also easy on the buffer. Q-16 locked it produces high 90s to 110 with ~14% buffer usage. 3 and 6 is what the GH1 people were using.

[Rush]

Good way to see difference is to record with high ISO (3200+) something with fine details.
With bad quality this fine details will be blurry and smudged by noise. With good quality this details will stay fine and noise will be more detailed too. (here is example)

[g3gg0]

i asked the company thomson if we can get a trial of their tool "AVC analyzer"

http://www.thomson-networks.com/products/monitoring-switching/media-file-analyzer

they sent me a dongle with 100 days evaluation for free
that tool is *really* able to look into every single frame and lets you check all coefficients down to the raw stream.
upload the files you want to compare and i will look at them  (or give you access to my PC or a VM via RDP so you can work with it)

want to access?

[g3gg0]

here the output of some movie i recorded and analyzed

http://upload.g3gg0.de/pub_files/c4e2826707bdc0c9a88831de35451f28/H264_Parsing.log

and here screenshots of that tool
http://upload.g3gg0.de/pub_files/f82f98bdf0e202cdfd9340f6b557f7ab/shot1.png
http://upload.g3gg0.de/pub_files/0385b3de70e7d74f6055bf25cbb854a8/shot2.png
http://upload.g3gg0.de/pub_files/a55fd128fc03da9103e3b4cf931ea2ab/shot3.png
http://upload.g3gg0.de/pub_files/5c33631de2a773d9378a1462f3307585/shot4.png
http://upload.g3gg0.de/pub_files/cbdd60d7b681909bd41585f481ffdd99/shot5.png

[Leon]

Sorry to just drop into the conversation – I thought it better to just ask than start a new thread – but I'd like to experiment with using CBR whilst setting a maximum Q-scale value and a minimum Q-scale value.  Is there an easy way to do this?  (My programming ability is limited but not non-existent.)

My reasoning is:

• Perhaps setting a minimum quantiser (e.g. -8) will reduce buffer overruns whilst maintaining a good quality.  With some of my "Class 6" cards I currently have to use CBR 0.8x for reliability, but maybe I could get away with CBR 1.0x limited to -8.

• Perhaps I can establish a maximum quantiser, maybe something like 0, where I never get buffer overruns and could improve quality by using that as another limit.

While I'm asking, does anyone know if -16 to +16 are equivalent to certain CQP quantiser values in x264?

Thanks!

[Audionut]

I'll try 240 and see what happens. Did you notice quality fall in the sample pics? I don't think its quite as bad as low BR I/P. the frames are about the same size (~5mb) . Bit rate for I only goes up to 140 without buffer filling.

Really need a scene shot form the same position (tripod) with the same lighting conditions.
Our eyes tend to bias to the bright side.

I need a consistant way to test what this is doing to quality, any ideas. I don't have curtains, test chart is hard to tell a diff.

Use anything with lots of details/texture..  Also need to test motion.  I'd suggest a static scene and walking through the frame.

[Audionut]

240 is unworkable.. buffer fills at very low BR and recording stops. Qscale says +24? and BR is in the 1/2s (all options at .1x) before I can get it to not stop instantly.

A large GOP size should reduce bitrate, so there must be something else funky going on.

Are you positive that the changes you are making are having the correct effects to GOP?  Record a short sample and load that into bitrate viewer, switch to frame based mode and count the number of frames in between the I frames (large bitrate spikes).
Or send the files to g3gg0 who can check with the stream analyzer.

Once you are sure you are getting the correct changes to GOP, we will need to test if the Canon code has any scene detection.