Bit rate investigation

Started by Audionut, July 19, 2012, 04:54:03 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

1%

It is changing 100%, counted the I frames in BR viewer.

Quotebut I'd like to experiment with using CBR whilst setting a maximum Q-scale value and a minimum Q-scale value

You can already do this on my  builds, all the features work in CBR mode... I actually put qscale on -16 when I did all I frames and that stuff. There should be a unified build up that has this in the repo and works with all cameras. I don't think I looped averaging so its limited to 24p.

I'll have to make some tripod/controlled  videos and upload them somewhere.... like 10s each. Those logs are pretty exact.

Whoever has a 600D, you can make videos too. binaries are up for 1, 3, 240 gop. I do need to see where the err 70 comes from too.. other cameras may not get it just at the end.


Stream cut ~90mbps Gop3 Video

http://www.sendspace.com/file/3db3yd

1%

The error 70 is coming from here:


FF255194: e59f2364 ldr r2, [pc, #868] ; 0xff255500: pointer to 0xa0f
FF255198: e28f1e32 add r1, pc, #800 ; *'Fcreate\\FcsMaker.c'
FF25519C: e28f0e33 add r0, pc, #816 ; 0xff2554d4: pointer to 0x30
FF2551A0: ebf6fc55 bl assert_0
FF2551A4: e5d4c00f ldrb r12, [r4, #15]


It is called here:

str:fcsSetThmMovieDesc_Fcreate\FcsMaker.c+728: assert_0(0xFF2554D4, 'Fcreate\\FcsMaker.c', 2575)

'fcsSetThmMovieDesc : Unknown GOP(%d)'

How to fix this?

*12 and 15 are valid options for most modes.

This bug will also show up when trying to write frame rate into file.

'fcsSetThmMovieDesc : Unknown FrameRate(%d)'


Leon

@1% :  Where can I download your build?  The closest I could find after 15 minutes searching was:  https://bitbucket.org/hudson/magic-lantern/downloads

When you say it will only work at 24fps, is that only if I change the GOP or I/P frames stuff, or will it also not work properly at 25fps or 30fps even if I only change Q-scale values?

Would I be better compiling a more up to date version myself?

Ta.

1%

In my repo, not in ML repo.

https://bitbucket.org/OtherOnePercent/tragic-lantern

When I say they don't work I mean the averaged values will only be set for 24p mode, noting will be set for 30p, etc until you turn off averaging. Regular scaling works, qscale works. Averaging mainly gets you higher numbers.


More on error 70:
Can you change bne to beq like in cracking and send it down the gop 12 branch?

FF255158: e5953040 ldr r3, [r5, #64] ; 0x40 FF255074
FF25515C: e353000c cmp r3, #12
FF255160: 03a0000c moveq r0, #12
FF255164: 0a000002 beq FF255174
FF255168: e353000f cmp r3, #15
FF25516C: 1a000004 bne FF255184

Audionut

Quote from: 1% on August 25, 2012, 05:22:41 AM
Stream cut ~90mbps Gop3 Video

That's looking really good.  GOP size correct and you have a nice low IP ratio.

And a quick look on the laptop the video looks great.  No blocking with the fast motion and details appear to be fine.

nanomad

Quote from: 1% on August 26, 2012, 02:32:59 AM
More on error 70:
Can you change bne to beq like in cracking and send it down the gop 12 branch?


From what I can see it's a static function call (i.e. not using a pointer). Since that code runs in ROM, the only way to do it would be to patch the code cache with the correct address which may or may not work...
EOS 1100D | EOS 650 (No, I didn't forget the D) | Ye Olde Canon EF Lenses ('87): 50 f/1.8 - 28 f/2.8 - 70-210 f/4 | EF-S 18-55 f/3.5-5.6 | Metz 36 AF-5

1%

What about:

http://magiclantern.wikia.com/wiki/Relocation


Can we relocate the whole function and NOP the assert_0 calls?

The function is only called by:


NSTUB(0xFF254B6C, str:PROP_PC_FLAVOR2_PARAM_PROP_PC_FLAVOR3_PARAM_fc)

FF255248: ebffff1e bl str:fcsSetThmMovieDesc_Fcreate\FcsMaker.c


Also, what about attacking NSTUB(0xFF0142FC, assert_0) itself and making it not call NSTUB(0xFF5680D4, assert), just print a log. That might soften a lot of non-serious crashes.

nanomad

I have no clue about side-effects but that should be easier since assert_0 actually calls B assert


ROM:FF014318                 B       assert


So you should place something like that in boot-hack.c (the first defines goes into your platform consts.h and should be edited accordingly)


#define HIJACK_FIXBR_ASSERT 0xFF014318

#define FIXUP_BRANCH_NOLINK( rom_addr, dest_addr ) \
    INSTR( rom_addr ) = B_INSTR( &INSTR( rom_addr ), (dest_addr) )

// This goes near the other "fixes"
FIXUP_BRANCH_NOLINK( HIJACK_FIXBR_ASSERT, nop );

EOS 1100D | EOS 650 (No, I didn't forget the D) | Ye Olde Canon EF Lenses ('87): 50 f/1.8 - 28 f/2.8 - 70-210 f/4 | EF-S 18-55 f/3.5-5.6 | Metz 36 AF-5

nanomad

Yeah, too bad that code segment is never relocated to RAM  :(
EOS 1100D | EOS 650 (No, I didn't forget the D) | Ye Olde Canon EF Lenses ('87): 50 f/1.8 - 28 f/2.8 - 70-210 f/4 | EF-S 18-55 f/3.5-5.6 | Metz 36 AF-5

1%


Andy600

Quote from: 1% on August 27, 2012, 01:20:18 AM
Ok, thanks to g3gg0 and nanomad error 70 is fixed. You're free to set the gop how you like, 1-100 something.

https://bitbucket.org/OtherOnePercent/tragic-lantern/downloads/autoexec.bin.600D.Exfat.Err70-Gone

Hi 1%, I'm trying this out but cant see any change in overall bit rate. I've set GOP to 1 but it looks like it's still recording at the default settings. I'm no expert with this (obviously). What settings should I dial in for an all-I frame test?

Cheers

Andy
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

a1ex

Asserts can be disabled easily from boot-hack.c, my_assert_handler (just don't call old_assert_handler). With this trick I've gained access to menus on a "bricked" 5D2 which was giving ERR70 at every startup (even without ML).

I wouldn't enable it in release builds because of potential side effects.

1%

Quote
Hi 1%, I'm trying this out but cant see any change in overall bit rate.

Did you toggle bit rate on the movie menu to something other than FW default. The setting affects whether ML will set the rates or not. Set CBR from the sub menu and go back to the movie menu, hit set and push the BR up to something with the arrow keys (1.5x, 2.0x, etc). I'd call it a bug but basically I didn't modify that part of the code (that x rate sets nothing).  After that you can go back and lock qscale by switching to VBR or just use encoder menu to change sizes.

QuoteAsserts can be disabled easily from boot-hack.c, my_assert_handler (just don't call old_assert_handler). With this trick I've gained access to menus on a "bricked" 5D2 which was giving ERR70 at every startup

We disabled just that 1 assert, sounds like it was worth the effort vs disabling them all because of what you say about stability. Now we can patch any jump, I have a feeling this will be very handy.

nanomad

Quote from: 1% on August 27, 2012, 05:09:51 PM
Now we can patch ANYTHING, I have a feeling this will be very handy.

Fixed to reflect the awesomness  ;)
EOS 1100D | EOS 650 (No, I didn't forget the D) | Ye Olde Canon EF Lenses ('87): 50 f/1.8 - 28 f/2.8 - 70-210 f/4 | EF-S 18-55 f/3.5-5.6 | Metz 36 AF-5

1%

And in that note, should look at where baseline profile and level is set... at the least could set it to 5.2 according to wikipedia... at the best up it to main or high profile if its in the encoder.

Andy600

Quote from: 1% on August 27, 2012, 05:09:51 PM
Did you toggle bit rate on the movie menu to something other than FW default. The setting affects whether ML will set the rates or not. Set CBR from the sub menu and go back to the movie menu, hit set and push the BR up to something with the arrow keys (1.5x, 2.0x, etc). I'd call it a bug but basically I didn't modify that part of the code (that x rate sets nothing).  After that you can go back and lock qscale by switching to VBR or just use encoder menu to change sizes.

We disabled just that 1 assert, sounds like it was worth the effort vs disabling them all because of what you say about stability. Now we can patch any jump, I have a feeling this will be very handy.

Got it :) I hadn't toggled the bit rate
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

1%

What was wanted with qscale limiting seems possible:

mvrSetLimitQScale LIMIT_QSCALE (L = %d, H = %d)'

I'll have to implement it, but right now I'm hard locking q to -16 when encoding because of greatly reduced buffer usage with reduced gop.

These are all the HW encoder parameters as initialized:


CreateResLockEntry(arg0->off_0x4, *(arg0)) => ret_CreateResLockEntry_FF1C82A4
*0x6244 = ret_CreateResLockEntry_FF1C82A4
CreateResLockEntry(arg0->off_0x8, arg0->off_0xC) => ret_CreateResLockEntry_FF1C82B8
*0x6248 = ret_CreateResLockEntry_FF1C82B8
*0x617C = arg0->off_0x10
*0x6198 = arg0->off_0x14
*0x61B4 = arg0->off_0x18
*0x61D0 = arg0->off_0x1C
*0x61EC = arg0->off_0x20
*0x620C = arg0->off_0x24
*0x6210 = arg0->off_0x28
*0x6168 = arg0->off_0x34
*0x6170 = arg0->off_0x38
*0x616C = arg0->off_0x2C
*0x6174 = arg0->off_0x30
*0x6110 = 1



Decoder sets parameters to decode baseline here... does that mean there are more profiles?



FF2F7C38: STRING: '\tSetConvertParameterForDecodeBaseline ' FF2F7934
FF2F7C60: STRING: '[ThbDec] XA= %d' FF2F7948
FF2F7C74: STRING: '[ThbDec] XB= %d' FF2F7964
FF2F7C88: STRING: '[ThbDec] XN= %d' FF2F7984
FF2F7C9C: STRING: '[ThbDec] YA= %d' FF2F79A4
FF2F7CB0: STRING: '[ThbDec] YB= %d' FF2F79C4
FF2F7CC4: STRING: '[ThbDec] YN= %d' FF2F79E4
FF2F7CD8: STRING: '=================================='



Someone with 5DMkIII fw... is encoder initialized a similar way? I'll keep looking.

Samples:
Here is a Gop4 video (~150MB)
http://www.sendspace.com/file/fipvng

I have a similar for gop3, just have to upload it.
Here: http://www.sendspace.com/file/mgj8ig

1%

Here are encoder parameters for different modes (I hope # is value)



720P 1080P? 1080P Dzoom
pointer to 0x6108
#324 ; 0x144 #316 ; 0x13c #348 ; 0x15c
#328 ; 0x148 #320 ; 0x140 #352 ; 0x160
#16
#120 ; 0x78 #116 ; 0x74 #132 ; 0x84
#20
#148 ; 0x94 #144 ; 0x90 #160 ; 0xa0
#24
#176 ; 0xb0 #172 ; 0xac #188 ; 0xbc
#28
#204 ; 0xcc #200 ; 0xc8 #216 ; 0xd8
#32
#232 ; 0xe8 #228 ; 0xe4 #244 ; 0xf4
#36 ; 0x24 #36 ; 0x24 #36 ; 0x24
#268 ; 0x10c #260 ; 0x104 #292 ; 0x124
#40 ; 0x28 #40 ; 0x28 #40 ; 0x28
#272 ; 0x110 #264 ; 0x108 #296 ; 0x128
#52 ; 0x34 #52 ; 0x34 #52 ; 0x34
#96 ; 0x60 #96 ; 0x60 #96 ; 0x60
#56 ; 0x38 #56 ; 0x38 #56 ; 0x38
#104 ; 0x68 #104 ; 0x68 #104 ; 0x68
#44 ; 0x2c #44 ; 0x2c #44 ; 0x2c
#100 ; 0x64 #100 ; 0x64 #100 ; 0x64
#48 ; 0x30 #48 ; 0x30 #48 ; 0x30
#108 ; 0x6c #108 ; 0x6c #108 ; 0x6c


Extra changes for Dzoom

*0x6274 = arg1
*0x627C = arg2
*0x6284 = arg3

FF1C8628: e5a50018 str r0, [r5, #24]!
FF1C862C: e5a56154 str r6, [r5, #340]! ; 0x154
FF1C8630: e5a57008 str r7, [r5, #8]!
FF1C8634: e5858008 str r8, [r5, #8]



There are also 2 AVC formats which appear:
CanonAVC0006
CanonAVC0005

Anyone know what they are?

Also set in: NSTUB(0xFF04B9E0, AJ_MOVW_SetAvcInfo)

AJ_guess_sortof_a_copy__FROM.R1__TO.R0__LEN.R2(2684 + aAJ_0x1EE0_MOVWptr_struct_0x00_to_0x04_taskName.off_0x4 /*0x1E38*/, arg0, 0xc)

*
When an encode is requested in HD mode, size is hard set

RequestH264Encode(arg0, 1920, 1080, arg1)

What happens if you change this?

More stream parameters in:
NSTUB(0xFF1111F4, AJ_guess_set_movieMode_n_pstream_stuff)

It is only called twice... can this be changed via code page hacking?

AJ_guess_set_movieMode_n_pstream_stuff+476:   RequestH264Encode(arg0, 1920, 1080, arg1)
str:lvcdevStartH264_pInputAddress_pStreamAddress1_+268:   RequestH264Encode(arg1->off_0x4, 1920, 1080, *(arg1))

Andy600

Just been reading LPowell's post about his Flow Motion 2.2 100Mbps for the GH2 which uses GOP3. http://www.personal-view.com/talks/discussion/3337/gh2-flow-motion-v2-100mbps-fast-action-performance-reliability-for-all-class-10-sd-cards/p1

This seems to be the optimal setting for that camera. Maybe a good starting point for testing this build? I'm still learning this so I'm not sure exactly what to set things to. So far I have got it set to GOP3 -16Qscale but bitrate viewer shows it's not a constant bit rate and I cant see any I-Frames.

BTW does anyone know an app similar to Stream Parser that will give us more detailed info? Bitrate Viewer is a bit basic.
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

Audionut

Quote from: Andy600 on August 28, 2012, 03:19:53 AM
BTW does anyone know an app similar to Stream Parser that will give us more detailed info? Bitrate Viewer is a bit basic.

AVInaptic
http://www.videohelp.com/tools/avinaptic

1%

 I think you have to keep qscale locking off if you want to use the CBR function as intended. The parameters in VBR change the same but the encoder is quality minded vs BR minded (qscale is locked, BR will fluctuate based on data and usually evens out). Coincidentally, this is another thing that can be tested in terms of quality.

I frames show up for me with BR viewer.. gop3 = 8 I iframes /sec gop 4 = 6/sec ... for 24p

Quote
Flow Motion v2 boosts the AVCHD encoder's peak bitrate to 100Mbps in all video modes and uses a short 3-frame GOP in all 1080p video modes (and a 6-frame GOP in 720p modes
These improvements enable Flow Motion v2 to capture highly detailed, fast moving images with far more fidelity than the stock firmware's restricted bandwidth can accommodate.

We're already doing this... but what is happening to quality? I can do like 140mbps in gop1, average over 120.


High ISO - Crop Mode - Light w/ Rain
Not sure if CBR/VBR

Gop1
http://upload.g3gg0.de/pub_files/d4c92e3c7850a189c1c192a86ace2f77/Lamp_Gop_1.MOV

Gop3
http://upload.g3gg0.de/pub_files/8e06487dd60083d1e42bd73197ec441c/Lamp_Gop_3.MOV

Gop6

http://upload.g3gg0.de/pub_files/1507715cede6b8b58d476acb7c2bc3ec/Lamp_Gop_6.MOV

Gop12
http://upload.g3gg0.de/pub_files/6186c340ceffd9fd7b4b6b940c14a0f5/Lamp_Gop_12.MOV

Chucho

http://www.jongbel.com/?page_id=19 is trail software that give detail information about the NAL units includes slices I,P,B and there offsets and all the seq_parameter that are set including profile_idc and level_idc.
Correct me if I'm wrong but how I understand it is that the SPS "sequence parameter set" sets the Profile_idc and the Level_idc in the NAL units. Is this only use for decoding and not encoding? [JPCORE] H264_SPS_PPS may lead to some good hints. Also NSTUB(0xFF1C90B0, H264E_OperateNalUnit_fGetSpsPps) may also be a good place to look for some hints.

1%

AJ_H264E_OperateNalUnit_fGetSpsPps is called by RequestH264Encode and sets encode parameters. I didn't know SPS/PPS were profile before.

Where is?
H264_SPS_PPS

Found in:

NSTUB(0xFF1E3310, AJ_JPCORE_SetEncodeH264Parameter)

That analaysis program is good!

Level is 50, profile is 66... I swear I saw that somewhere in the H264 parameters but now I know what to look for.

qp minus is -6 in videos, that is either our QPC/QPy or from deblocking filter (which didn't seem to come off, need to make sure it doesn't get reset per frame.)

Leon

Right.  So if you use "CBR" but "lock" the quantiser at -16, will you not get exactly the same output as using Q-scale with quantiser at -16?  (And still set GOP, etc.) 

I think they should be exactly the same thing, and it would then make sense to call it a modified Q-scale rather than CBR.  (Unless I'm missing something?)  If this works with audio it would be really cool (not that it isn't already really cool ;)).

What would also be cool, in the future, is to have an option where you can set Q-scale and the firmware will automatically vary the I/P ratio to ensure the buffer doesn't overrun.  This would allow more efficient compression, than say using GOP 3 all the time, but still allow use of low quantisers, and not worrying about setting the GOP manually depending on the speed of each of your cards.  (Even if you have to leave a large margin of error, such as trying not to go above 20% full.)

Andy600

Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com