Author Topic: Bit rate investigation  (Read 105513 times)

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D
Re: Bit rate investigation - altering ratio between P and I frame sizes
« Reply #275 on: September 17, 2012, 02:36:57 AM »
Thats pretty easy. You can do it manually now, just set slice to somewhere above 129 before you start recording. After you finish, it leaves it where it ended up. Could have it set back lower when recording finishes or something like that.

JasonATL

  • Freshman
  • **
  • Posts: 54
Re: Bit rate investigation - altering ratio between P and I frame sizes
« Reply #276 on: September 17, 2012, 02:42:34 AM »
Great, thanks!. I wondered what slice did, since it was always different when I finished recording. Must have missed that!

matt2491

  • New to the forum
  • *
  • Posts: 7
Re: Bit rate investigation - altering ratio between P and I frame sizes
« Reply #277 on: September 17, 2012, 04:23:00 AM »
Just donated $25 specifically for the bitrate investigation cause! Yay!

Kabuto1138

  • Freshman
  • **
  • Posts: 87
Re: Bit rate investigation - altering ratio between P and I frame sizes
« Reply #278 on: September 17, 2012, 08:29:32 PM »
Hey Guys, this development is perfect timing for Magic Lantern/CANON users.

With the release of the GH3 and at the price range we are working at is only a matter of time before people start looking elsewhere(the GH3 or Sony cameras) for this high bitrate capabilities out of the box.  People might loose interest in ML very soon if the canon dslr cameras don't have "competitive" bitrates.... probably the most important new feature ML should have.

keep up the great work guys!!

How do we make ALEX be part of this ASAP?

nanomad

  • Administrator
  • Hero Member
  • *****
  • Posts: 2918
  • All your websites are belong to us
Re: Bit rate investigation - altering ratio between P and I frame sizes
« Reply #279 on: September 17, 2012, 08:33:36 PM »
How do we make ALEX be part of this ASAP?

A time machine, probably
The guys here have done a terrific job  ;)
EOS 1100D | EOS 650 (No, I didn't forget the D) | Ye Olde Canon EF Lenses ('87): 50 f/1.8 - 28 f/2.8 - 70-210 f/4 | EF-S 18-55 f/3.5-5.6 | Metz 36 AF-5

Kabuto1138

  • Freshman
  • **
  • Posts: 87
Re: Bit rate investigation - altering ratio between P and I frame sizes
« Reply #280 on: September 17, 2012, 08:39:22 PM »
He's done an AMAZING job.  I don't know why Canon hasn't hired him.  ML is probably responsable for a lot of sales.

Just worried that there is a window here of opportunity.  With the Black Magic Camera, Gh3, A99 etc high bitrates should be a priority?

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D
Re: Bit rate investigation - altering ratio between P and I frame sizes
« Reply #281 on: September 17, 2012, 11:14:18 PM »
There are drawbacks to the other cameras like smaller sensors, lens selection and they don't take stills so well. Also more crop/deeper dof.

But canon is lagging here for sure. The competition is stiff in mid levels.

Sony - I've seen no hacks yet. Over $2k price tag... sony quality and sony reputation. no thanks.
GH3 - If its hacked up and cheap why not. Ripoff lenses but gain IS. Tiny sensor/low mp. Mostly for video.
Black Magic - If software and performance is up to snuff, price is good for a semi-pro and resolution tops most things in that range. If stills are also good and body is comfortable this camera is eating everyone's lunch.

There is simply no market for 5d3 at $4k. At $2k sure. The other cameras ~$1000 are really a good value but where does that leave canon? In the slightly above low end consumer market? Broke film makers? Once semi vendor lock with the lenses wanes they are in for a world of hurt.

The days where it was canon vs $10k+ cameras are soon if not already over. Nikon is pretty good w/stills and lenses are cheaper. Their sensors are still winning but for how long? If anything, copy us faster and release more FW versions with new features. How can you upgrade hardware, sell the camera and do everything possible to not take advantage of it?

Anyways, on topic... instant/average BR is still off so I'm having problems. After dicking with it, its still 33% off and something happens to average BR indicator after 4gb or like 3 minutes. I'm using instant but its not super accurate either (occasionally pops up to like 200-400, not real). I'm having stopping issues at high ISO like 2500. Then again 2500 looks like crap in day time. Need some math help. I'll push what I just did when I test it some more.

JasonATL

  • Freshman
  • **
  • Posts: 54
Re: Bit rate investigation - altering ratio between P and I frame sizes
« Reply #282 on: September 18, 2012, 01:16:06 AM »
I'm using instant but its not super accurate either (occasionally pops up to like 200-400, not real). I'm having stopping issues at high ISO like 2500. Then again 2500 looks like crap in day time. Need some math help. I'll push what I just did when I test it some more.

1% - Just curious: How do you know that 200-400 is not real? The reason that I ask is that VLC was reporting input bitrates exactly that high in some of my early test clips (and I just haven't checked in my later clips). Bitrate viewer does not show them that high, but my version of bitrate viewer only shows average bitrate in one second intervals (is there a way to get it to report bitrate at higher frequency?).

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D
Re: Bit rate investigation - altering ratio between P and I frame sizes
« Reply #283 on: September 18, 2012, 01:27:08 AM »
I did bit rate viewer in gop mode (ctrl-G) and frame mode(ctrl-g) and never saw a bit rate above 2xx) So will have to look at VLC and see if it says anything else. Stream eye also doesn't show anything that high.

To get 400mbps you would need 2x the card write available (50MB). It would have to store the frame in the buffer and write it out in pieces somehow.

miyake

  • Developer
  • Senior
  • *****
  • Posts: 396
Re: Bit rate investigation - altering ratio between P and I frame sizes
« Reply #284 on: September 18, 2012, 09:45:30 AM »
Just idea.

The H264 encoding has a lot of arguments. So it's a little difficult to implement in current ML menus.
So
 -add new config file for encoding parameters like encoding.cfg
 - file contents described current 1%'s values and symbolic name with MS-Excel csv style.
 - ML will loading the file and values into memory.
 - we can choose which profile want to use
    (menu is dynamically added and we can identify symbolic name).

This idea will support dynamic changing profiles without compile environment. And menu is clean. Also the user is able to change detailed parameters, if it understanding. And more, Someone make a "Encoder config generator" on Win/Mac/WEB apps.
How do you think 1%?

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: Bit rate investigation - altering ratio between P and I frame sizes
« Reply #285 on: September 18, 2012, 09:56:03 AM »
The 5D3 firmware can load and parse these INI files:

http://a1ex.magiclantern.fm/bleeding-edge/5D3/H264-ipb.ini
http://a1ex.magiclantern.fm/bleeding-edge/5D3/H264-alli.ini

(the current values in those files contain the default parameters, I think).

I'm thinking to implement profiles in 5D3 too, so this idea can be reused.


nanomad

  • Administrator
  • Hero Member
  • *****
  • Posts: 2918
  • All your websites are belong to us
Re: Bit rate investigation - altering ratio between P and I frame sizes
« Reply #286 on: September 18, 2012, 10:03:42 AM »
Really? Lol
Are those official or undocumented?
EOS 1100D | EOS 650 (No, I didn't forget the D) | Ye Olde Canon EF Lenses ('87): 50 f/1.8 - 28 f/2.8 - 70-210 f/4 | EF-S 18-55 f/3.5-5.6 | Metz 36 AF-5

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: Bit rate investigation - altering ratio between P and I frame sizes
« Reply #287 on: September 18, 2012, 10:04:44 AM »
I've got the field names from strings and the values from memory dumps. And yes, you can change things from there and they take effect (or err70).

nanomad

  • Administrator
  • Hero Member
  • *****
  • Posts: 2918
  • All your websites are belong to us
Re: Bit rate investigation - altering ratio between P and I frame sizes
« Reply #288 on: September 18, 2012, 11:45:28 AM »
Can't we use the mvrTest function to do some controlled tests? It reads a YUV sequence and sets it as an input to the mvr Encoder. The only drawback is that it reads from the A:/ drive but we should be able to patch it easily....
EOS 1100D | EOS 650 (No, I didn't forget the D) | Ye Olde Canon EF Lenses ('87): 50 f/1.8 - 28 f/2.8 - 70-210 f/4 | EF-S 18-55 f/3.5-5.6 | Metz 36 AF-5

Marvin

  • New to the forum
  • *
  • Posts: 31
  • Personality is destiny
Re: Bit rate investigation - altering ratio between P and I frame sizes
« Reply #289 on: September 18, 2012, 12:03:33 PM »
Let me share with you my findings with the H264 parameters in the .ini file, I did tons of experiments with Alex back in May, during the quiet days after the "Hello World" :-)

I'll start with ALL-I.

Code: [Select]
Transform8x8Flag = 2
Profile = 100
Level = 51

It seems that these parameters can't be changed (ERR70 if modified). Profile 100 and Level 51 means "High Profile, Level 5.1" in H264 specifications, this is even more advanced than standard Bluray codec which is High Profile Level 4.1.

Code: [Select]
BitRate = 90000000
This parameter might interest most of the people here :P yes it can be changed to maximum 300000000, which is 300Mbps (specified in H264 standards), compared to original 90Mbps default by Canon. But please note that this value is currently only a cap and does not keep the video file at this bitrate. More on this later.

Code: [Select]
EntropyCodingMode = 0
Another interesting parameter, there are two modes which are "Context-Adaptive Variable Length Coding (CAVLC), Context-Adaptive Binary Arithmetic Coding (CABAC)". default 0 is CAVLC which is lower encoding efficiency but easier for playback. value of 1 enables CABAC which is said to bring 10%-15% quality increase under the same given bitrate.

Code: [Select]
IntraPicInitQP = 20
InterPicInitQP = 20
QpOffsetForB = 0

can't be changed if I remember correctly, seems related to initial Quantitization Parameter, more on this later...

Code: [Select]
MinQpI = 10
MinQpP = 10
MinQpB = 10
MaxQpI = 51
MaxQpP = 51
MaxQpB = 51

an important set of parameters, these values affect the actual video bitrate, for ALL-I, after numerous ERR70s I found out that the three "MinQp" values can be changed up to 7, and "MaxQp" values can be changed up to 20. QP stands for Quantitization Parameter which is the lower the number the better, a value of "0" means lossless encoding :-)

Video shot with these values changed/unchanged did show a noticeable difference in bitrate. Under the same scene and same exposure, I got 70Mbps compared to 40Mbps with Canon default value. Together with the previous "Bitrate" parameter set to 300Mbps I was able to get 151Mbps under ISO12800 compared to 92Mbps default.

Code: [Select]
MinBitrate = 0
MaxBitrate = 0

can't be changed.

Code: [Select]
SarWidth = 0
SarHeight = 0
AspectRatioIdc = 1
VideoFmtAndVspFlag = 81
VideoFullRangeFlag = 1
TimingInfoPresentFlag = 0

not very important parameters, doesn't affect the video much.

Code: [Select]
RateControlEnable = 2
A mysterious parameter and worth looking into, according to some documents, a value of 0 => CBR, 1 => VBR, 2 = Fixed QP. with 0 I get very small video files from 1Mbps to 5Mbps, 1 gives ERR70, 2 is Canon default which must be Canon's secret of manipulating bitrate.

Code: [Select]
ScalingMatrices = 0
pScalingMatrixAddr[0] = 0
pScalingMatrixAddr[1] = 0
pScalingMatrixAddr[2] = 0
pScalingMatrixAddr[3] = 0
pScalingMatrixAddr[4] = 0
pScalingMatrixAddr[5] = 0

Not sure about the exact meanings but don't seem to affect the video much.

For IPB there is only a few differences.

Code: [Select]
Profile = 100
Level = 41
BitRate = 10000000

IPB uses High Profile Level 4.1, the same as Bluray codec. Bitrate can be changed to maximum 62000000.

Code: [Select]
MinQpI = 10
MinQpP = 10
MinQpB = 10
MaxQpI = 35
MaxQpP = 51
MaxQpB = 51

Alex are you sure these are default values? I don't remember LOL, but for IPB, MinQp and MaxQp can be changed to 6 and 30. resulting bitrate can easily hit 62Mbps which is the upper limit set by H264 specifications.

If you look at the metadata of 5D Mark III's .MOV file, it will tell you that the video is encoded by "EOS encoder", which I think it might be talking to the above parameters directly, if a value is not present, for example, if I change "Profile" to 122 which stands for High 4:2:2 encoding, it will give an ERR70...   or if I change the MinQP to something like 2 or 3, video recording will be stuck at "BUSY".

Hope this information is helpful and can save you some time, good luck folks!
Film is truth 24 frames per second.

JasonATL

  • Freshman
  • **
  • Posts: 54
Re: Bit rate investigation - altering ratio between P and I frame sizes
« Reply #290 on: September 18, 2012, 01:28:43 PM »
Miyake - I think that's a great idea! I would be willing to do what it sounds like Marvin has done for the 5D3 in testing sets of parameters for the 600D (and if anyone cares to take this to the 550D, I'll test it, too). What would be really nice is to have a menu that allows me to choose a profile to load and use (sort of like the overlays). At least for this testing phase, it could be more efficient.

Marvin and A1ex -
1. Perhaps I misunderstand, but can I change those parameters on an 5D3 without ML by using the .ini files? How does the 5D3 load them? Do I just put them on the root of the SD or CF card and it automatically loads them?
2. Before I try this, is Err70 an error that needs a battery pull? Or, does it pose a greater risk to the camera?
3. If answers to 1. and 2. are positive, then is there anything I can do along these lines or is all understood by you already?
4. Is the 5D3 the only camera that takes such an .ini file?

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: Bit rate investigation - altering ratio between P and I frame sizes
« Reply #291 on: September 18, 2012, 01:30:07 PM »
JasonATL: this feature is not yet enabled in alpha 1, so you need to wait.

JasonATL

  • Freshman
  • **
  • Posts: 54
Re: Bit rate investigation - altering ratio between P and I frame sizes
« Reply #292 on: September 18, 2012, 02:00:19 PM »
Thanks, A1ex. I appreciate everyone's work on this.

Just trying to offer help in any way possible. Let me know if there is anything that I can test/try that might be helpful.

deleted.account

  • Member
  • ***
  • Posts: 154
Re: Bit rate investigation - altering ratio between P and I frame sizes
« Reply #293 on: September 18, 2012, 02:24:14 PM »
Let me share with you my findings with the H264 parameters in the .ini file, I did tons of experiments with Alex back in May, during the quiet days after the "Hello World" :-)


Code: [Select]
SarWidth = 0
SarHeight = 0
AspectRatioIdc = 1
VideoFmtAndVspFlag = 81
[b]VideoFullRangeFlag = 1[/b]
TimingInfoPresentFlag = 0

not very important parameters, doesn't affect the video much.

Hope this information is helpful and can save you some time, good luck folks!

Personally would like the 'VideoFullRangeFlag' on/off as an option for an ini file. :-)

With the fullrangeflag set on as it is with Canon MOV's it signals to the decompressing codec to squeeze luma levels 16 - 235 in any import into the NLE or any transcode / general encoder output. So it can be normalized into 0-1 ie: 0 - 255 in 8bit RGB terms.

I currently use a patched build of MP4Box to be able to switch this flag off in the h264 stream which involves a remux to mp4.

Although the feed into the camera encoder appears to be raw uncompressed JFIF ie: full luma/chroma levels over 0 - 255, the option to switch the flag off for the 32bit float < 0.0  > 1.0, so overbrights exist may be useful.

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D
Re: Bit rate investigation - altering ratio between P and I frame sizes
« Reply #294 on: September 18, 2012, 04:22:28 PM »
Wish I had .ini loading and QP of 7 :) Way more parameters than Digic IV h.264.  I don't think we have enough parameters to warrant an INI yet.

JasonATL

  • Freshman
  • **
  • Posts: 54
Re: Bit rate investigation - altering ratio between P and I frame sizes
« Reply #295 on: September 19, 2012, 02:11:06 PM »
I finally edited together 20 seconds or so of a waterfall. Shot with Canon FW, GOP=1 (ALL-I), and GOP=3. The links to screen captures are below, too. This difference in quality is huge, in my opinion. I did not test the ML 2.3 CBR 2.0x or something like that on this particular shot. I'm fairly confident from experience that CBR 3.0 would have caused a buffer overflow, but maybe not.


Of course, download the original file to see this without Vimeo's compression.

GOP=3: Avg 149 Mbps, Peak 170 Mbps
GOP=1 (ALL-I): Nearly constant 169 Mbps
Canon FW: 40 Mbps


Still 1: https://picasaweb.google.com/102645356447404039003/September192012#5789851335385308482
Still 2 (300% zoom): https://picasaweb.google.com/102645356447404039003/September192012#5789851289298012066

I think that ALL-I probably looks the best overall here, but GOP=3 is very close. The increased detail you can see in the rocks, leaf, and shadows in the waterfall is visible at 100% and obvious at the 300% zoom, in my opinion.

I also did GOP=2, 4, and 6 (not shown in the video) for this same subject and don't really see a large difference among them.

ilguercio

  • Hero Member
  • *****
  • Posts: 845
Re: Bit rate investigation - altering ratio between P and I frame sizes
« Reply #296 on: September 19, 2012, 02:28:27 PM »
It looks better to me when it comes to compression as the big rock in the foreground is a bit less "blocky".
I suppose it won't make any difference to me, i wouldn't have seen the differences by myself.
I'd prefer having true 1080p but i know it can't happen, or just crop mode on all cameras so we can get rid of line skipping.
Does any of our users have a GHx for comparison?
Canon EOS 6D, 60D, 50D.
Sigma 70-200 EX OS HSM, Sigma 70-200 Apo EX HSM, Samyang 14 2.8, Samyang 35 1.4, Samyang 85 1.4.
Proud supporter of Magic Lantern.

Marvin

  • New to the forum
  • *
  • Posts: 31
  • Personality is destiny
Re: Bit rate investigation - altering ratio between P and I frame sizes
« Reply #297 on: September 19, 2012, 03:28:24 PM »
Nice job Jason :)

How did you manage to achieve 170Mbps? 600D uses H264 Baseline Profile Level 5.0 if I remember correctly, if you look at the specification, it says maximum datarate is set to 135Mbps.

Anyway If 600D can achieve constant 169Mbps and is stable for a few minutes then I'm very optimistic that 5D Mark III might be able to hit constant 300Mbps in ALL-I (max datarate set by H264 spec)
Film is truth 24 frames per second.

JasonATL

  • Freshman
  • **
  • Posts: 54
Re: Bit rate investigation - altering ratio between P and I frame sizes
« Reply #298 on: September 19, 2012, 04:01:24 PM »
Marvin - Thanks. I don't know enough about H264 to answer your question precisely. 1% has been able to change the encoder's parameters. What this could mean is that the encoder is not compliant with the profile specs or that the change in the encoder preferences abandons the profile (is that even possible?). If you are knowledgeable in this area, perhaps you could shed some light on it for us and help us better understand what parameters we can control and how they will help give us the best output here. I was intrigued by your testing with the 5D3 and look forward to being able to get a better bitrate from my 5D3.

ilguercio -I suspect that, if this was your own footage and you were editing and color grading, you would notice the difference. What I have really noticed is that I can sharpen the footage and push the colors much more than before without it "breaking down." What I mean by this is that with sharpening and color grading, the compression artifacts show up quickly. With such a high bitrate resulting in such a good image to start with, the image is more robust to changes. It also looks far less "digital" to me with a much more natural look. This subjective aspect of the picture sometimes isn't easy to pinpoint as to the exact part of the picture to blame. Finally, what I will try to demonstrate in my next video will be a more detailed image. In this case, what looks like "mush" with Canon FW turns into a very nice picture.

I understand your point about resolution. I would like to see more resolution, too. But, the key point here is that the Canon FW is not giving us the best of what (little) resolution we have due to the bitrate and encoder parameters. 1% has at least made huge progress in fixing this. I am excited about the having this on the 5D3 (and other things in development, as well). But, I am quite impressed with what I'm getting out of my 600D now.

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D
Re: Bit rate investigation - altering ratio between P and I frame sizes
« Reply #299 on: September 19, 2012, 04:24:20 PM »
Hehe... we are officially out of spec according to avinaptiic. I think its a good thing.

The spec is for HW playback mostly; so if chinsy HW players can't play your raw its a good thing.

I'd love to change the profile and now I know how to override the "profile" numbers in rom but I haven't a clue what the settings mean. High or at least level 5.1 is probably in there. Maybe then QP can move past 10. I'd like to figure out how to turn off resizing if possible too. We tried a little so far but it didn't work. Might be able to manipulate some of the tables used for x/y. Hopefully this can make for 1:1 720P, etc.

I'm still wanting more. 5d3 has way higher data rate and full frame sensor. But $4k price tag vs ~$600 for 600d.