Magic Lantern Forum

Experimental builds (WIP) => Tragic Lantern => Topic started by: Audionut on July 19, 2012, 04:54:03 AM

Title: Bit rate investigation
Post by: Audionut on July 19, 2012, 04:54:03 AM
Firmware Default
http://dl.dropbox.com/u/34113196/ML/Firmware%20Default.png

Q15
http://dl.dropbox.com/u/34113196/ML/Q15.png

1 has detail in the leaves.  The other has blurry green things with no detail.

Cropped 300% resize to exaggerate.

(http://dl.dropbox.com/u/34113196/ML/Firmware-Defaultcrop.png)
(http://dl.dropbox.com/u/34113196/ML/Q15crop.png)

edit:  Considering it's a poor encoder to start with and uses 40mbps @ default for the quality it produces (lack of fine details), is it really any wonder that doubling the bitrate increases quality?

The Q15 sample above was the max of my card @ 99mbps.  And although it's much better then deafult, IMO, it would take around 200mbps before it produced really good results.
Such a shame that the codec can't be replaced.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on July 19, 2012, 05:32:56 PM
There may be more parameters to the encoder, just have to look. Its clear for me higher bitrate = more data and lower bitrate = less data = lower quality. I mean why else can you put .1 bitrate and record for 1/2 an hour? Higher bitrate not having better quality would mean that the settings are being rejected but they're clearly not.

The problem is the bit rate setting itself. I set to 1.1 and I get bit rates of 45-60... the higher settings just peak the bit rate really high and thats why recording stops. Above 65mbps with card not fresh it stops recording. I've been watching the instant rates and the higher x numbers still seem to give the same base bit rate.

If they only didn't peak it so fast and started at a higher value it would be more stable while keeping quality. As it stands the settings just crank it till you overrun the buffer then try to lower it but never fast enough.  I.e it goes from 59 to 79 or 89 and then the buffer runs out depending on card speed and scene complexity. 
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on July 20, 2012, 08:03:21 PM
Ok, I bought a new card. Patriot EP SDXC 64g. It is tested for at least 15MB/s continuous write. Last night I thought 99mbps is tops on this camera... it is not. I saw 107mbps peak. The reason you're all having trouble is that your cards are not fast enough. Class 10 is 10MB/s continuous write and 100mbps is 12.5MB/s. 

With the new card I can shoot 99mbps all day, complex scene in 2.x factor, 1250 ISO. At 3.0 the BR starts going above the 100s and eating the buffer. I also found out audio is ~40Mbps. I can shoot at 1.4x and get ~61Mbps instant with 1 bar in the buffer. Higher doesn't work. i almost wish I had bought a faster pro card but I'm not paying $180 for a few extra Mb. So while this feature is good when SDXC continuous speeds go up so will bit rate. I don't think there is a limit as 600D supports UHC-1 but maybe not the faster mhz bus speed (anyone try?). That 200mbps? In a year or with expensive cards it might happen.

*Update... I saw instant bit rate of 210.. do you think this is an error?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: ilguercio on July 20, 2012, 08:45:14 PM
I got huge bitrates for a few seconds with my 50D, like 188 mbps, but the that was probably too much for the camera.
Still, if there is no such appreciable gain in quality i see no point in using such bitrates since they're slower to process and tae up a lot of space.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: weldroid on July 20, 2012, 09:03:57 PM
This guy here?
http://www.clickok.se/PartDetail.aspx?q=p:4664058

15 MB/s is what... 120 mb/s, so it should be able to handle more than 100 mb/s.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on July 20, 2012, 09:57:51 PM
15MB/s is about there yes... I think 120mbps stable is a bit hopeful. That card tested 18MB/s on a 1gb file and 15 on a 4gb file. So far 99 is stable for me... above 100 is where it starts to get flaky. Gotta admit 99mbps at high ISO is pretty good. I only paid $US60... much better than US $179


QuoteStill, if there is no such appreciable gain in quality i see no point in using such bitrates since they're slower to process and tae up a lot of space.

According to? The gain comes out in post [but we'll see :)]. Not much "visible" difference between 4:2:2 and 4:2:0 for most applications but then 4:2:2 camera costs $$$$ and 4:2:0 camera is cheap.

Also we don't know whats up with extending 4gb limit or other encoder parameters on 600D. Can 550D do steady 99mbps? Or 60D? I guess 50D didn't do too bad but I think movie mode on that is completely added by ML? People were saying 1.4x is last stable factor on 60D and I think alex was even going to take higher bit rates out; but on 600D 1.4x works with sound and before I could barely do 1.1 or 1.2 at higher complexity + iso.

I think this card was only available since 5/12.... SD was simply not capable of these speeds before without REALLY expensive cards nobody would buy.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Audionut on July 20, 2012, 10:56:02 PM
Quote from: 1% on July 20, 2012, 09:57:51 PMCan 550D do steady 99mbps?


550D - Sandisk Extreme Pro 95MB/s

Firmware default.
(http://i.imgur.com/sNUM8.png)

About the best I can get with in camera audio recording.
(http://i.imgur.com/PrvnR.png)

Max bitrate I can get with off camera audio
(http://i.imgur.com/ImPX4.png)

I don't have a test with ML CBR.  But as has been mentioned already, it is not true CBR.
I'm fairly sure I've seen Alex mention that this has to do with the way the Qscale works.  It's to slow to update iirc.

edit:  As can be seen in the last screenshot, these are all 1080p/30fps.  Could probably squeeze are few more Mbps out of it at 24fps, but I hate motion judder.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: a1ex on July 20, 2012, 11:01:23 PM
True CBR in H.264 may need multiple passes for each frame. As in: compress at some fixed quality factor (QScale). Is frame size too big? Reduce quality and recompress. Too small? Increase quality and recompress. Rinse, repeat.

OK for encoding on PC, but not OK in real time. And the algorithm is hardcoded in Canon firmware - it basically uses frame size info from frame X to decide quality factor on frame X+1. Sure, there are P frames, I frames, GOP and other similar stuff that make changing the algorithm very difficult.

So, unless there are visible benefits of this (which I didn't see yet), it will stay as wontfix.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Audionut on July 20, 2012, 11:09:04 PM
Quote from: a1ex on July 20, 2012, 11:01:23 PMSo, unless there are visible benefits of this (which I didn't see yet), it will stay as wontfix.

Do you not see the visual difference in the screenshots I posted above?

IMO, a true CBR mode wouldn't increase quality per se, but it would allow users to record at the maximum bitrate of their DSLR's/Memory cards without fear of recording being stopped as currently happens with the current modes.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on July 20, 2012, 11:10:45 PM
I don't think we really need a "fix"... I think people need faster cards. Camera write > card write at this point. Were you able to see any other parameters for the encoder? P frames, I frames and GOP I think can alter quality too.

https://docs.google.com/viewer?a=v&q=cache:irMNf5l2YKIJ:www.acti.com/download_file/Product/support/DesignSpec_Note_GOP_20091120.pdf+&hl=en&gl=us&pid=bl&srcid=ADGEESjGSHsSrBtVUnnS1hD4VabEF8doangw_kfGHKdtFzskQPYa18On-7UgzKquwOiCOI939UWnuSnLKkthDpDArZMRhbmcNIRsz0RmPkfMfvfNAo2QOCtcT97oSM8ChjvHYdUHEP3i&sig=AHIEtbQ5C55Uiw30BabakL0sKOumDxhbag

Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: a1ex on July 20, 2012, 11:14:49 PM
GOP length is fps/2 - buf[3] in PROP_VIDEO_MODE. You can't just change it - it will record a few frames and stop. But it will record them with new GOP length, which is a good sign.

Reference frame sizes for P and I are scaled when you change the "X" in bitrate dialog, by a constant factor with current implementation. It is possible to alter their ratio (for example, you can get equal size for I and P frames).
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Audionut on July 20, 2012, 11:16:56 PM
Quote from: 1% on July 20, 2012, 11:10:45 PM
I don't think we really need a "fix"... I think people need faster cards.

If you can find me something faster then a Sandisk Extreme Pro, I'd be all for it.  But at the end of the day, it wouldn't matter as the bitrates are limited by the cameras.

I can get transfers in excess of 80MB/s on a PC.  This is way in excess of the 100Mbps that the camera tops out at.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Audionut on July 20, 2012, 11:21:12 PM
Quote from: a1ex on July 20, 2012, 11:14:49 PM
GOP length is fps/2 - buf[3] in PROP_VIDEO_MODE. You can't just change it - it will record a few frames and stop. But it will record them with new GOP length, which is a good sign.

A larger GOP length would certainly help.  10x fps is considered a fairly standard encoding setting.  Of course, this isn't taking into account any format (blu-ray etc) requirements.

Quote from: a1ex on July 20, 2012, 11:14:49 PMReference frame sizes for P and I are scaled when you change the "X" in bitrate dialog, by a constant factor with current implementation. It is possible to alter their ratio (for example, you can get equal size for I and P frames).

May I ask what the current ratio is? 

If it's not possible to change the GOP length, AFAIK, altering the IP ratio would help.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: a1ex on July 20, 2012, 11:28:18 PM
Need to run custom code to find out - but you can see it with bitrate viewer.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on July 20, 2012, 11:34:43 PM
Hehehe, were there is a will there is a way.

QuoteI can get transfers in excess of 80MB/s on a PC

Read? Write? Remember camera isn't using the new UHC-1 bus speed so real write is probably ~15-20MB/s... they have not made a faster card yet :( Maybe when UHS speed is higher the reggo write will go up like it did. Or we can tweak something to enable this? Far shot...

QuoteNeed to run custom code to find out

The pile of things to try for ML 2.4 is growing but I guess that is a good thing :)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Leon on July 21, 2012, 12:06:28 AM
It'a not just about improving quality.  A big problem is how the max CBR setting that a card can cope with changes unpredictably.  For example, a class 10 Transcend card may manage 1.3x right after formatting, but less than 1.0x after it as been used a bit or some files deleted.  It drives me nuts.  This is quite annoying, and it would be great to have a feature that would automatically reduce the bitrate or CBR setting if required during recording.  I think unexpected stopping / restarting of video is certainly one of the biggest problems with recording video on the 550D.  It happens too often, even at the firmware default.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on July 21, 2012, 01:15:15 AM
What is:



static void load_h264_ini()
{
    gui_stop_menu();
    call("IVAParamMode", CARD_DRIVE "ML/H264.ini");
    NotifyBox(2000, "%s", 0x4da10);
}


And this sets gop and frame sizes ratios in an array?



void opt_set(int num, int den)
{
    int i, j;
   

    for (i = 0; i < MOV_RES_AND_FPS_COMBINATIONS; i++) // 7 combinations of resolution / fps
    {
#ifdef CONFIG_500D
#define fullhd_30fps_opt_size_I fullhd_20fps_opt_size_I
#define fullhd_30fps_gop_opt_0 fullhd_20fps_gop_opt_0
#endif

#ifdef CONFIG_5D2
#define fullhd_30fps_opt_size_I v1920_30fps_opt_size_I
#endif
        for (j = 0; j < MOV_OPT_NUM_PARAMS; j++)
        {
            int* opt0 = (int*) &(mvr_config_copy.fullhd_30fps_opt_size_I) + i * MOV_OPT_STEP + j;
            int* opt = (int*) &(mvr_config.fullhd_30fps_opt_size_I) + i * MOV_OPT_STEP + j;
            if (*opt0 < 10000) { bmp_printf(FONT_LARGE, 0, 50, "opt_set: err %d %d %d ", i, j, *opt0); return; }
            (*opt) = (*opt0) * num / den;
        }
        for (j = 0; j < MOV_GOP_OPT_NUM_PARAMS; j++)
        {
            int* opt0 = (int*) &(mvr_config_copy.fullhd_30fps_gop_opt_0) + i * MOV_GOP_OPT_STEP + j;
            int* opt = (int*) &(mvr_config.fullhd_30fps_gop_opt_0) + i * MOV_GOP_OPT_STEP + j;
            if (*opt0 < 10000) { bmp_printf(FONT_LARGE, 0, 50, "gop_set: err %d %d %d ", i, j, *opt0); return; }
            (*opt) = (*opt0) * num / den;
        }
    }
[/quote]
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: nanomad on July 21, 2012, 01:25:21 AM
Some kind of test probably
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Audionut on July 21, 2012, 02:33:52 AM
Quote from: 1% on July 20, 2012, 11:34:43 PMRead? Write? Remember camera isn't using the new UHC-1 bus speed so real write is probably ~15-20MB/s...

Sorry, that was read speed.  I am unsure how to completely bypass windows caching to determine true write speeds.  But considering the card reaches advertised read speeds, and the advertised write speed is 90MB/s, even if it was only getting 1/2 advertised write speeds, that's still way in excess of what the camera (550D atleast) is capable of writing.

Quote from: a1ex on July 20, 2012, 11:28:18 PM
Need to run custom code to find out - but you can see it with bitrate viewer.

(http://i.imgbox.com/aczySVsH.png)


Ouch, I frames should not need to be anywhere near 2x the size of P frames.

It appears as if it's a variable IP ratio also.  edit:  On second thoughts, it's probably not a variable IP ratio, simply that 235Mbps is the maximum frame size the camera is capable of.

(http://i.imgbox.com/adycEEvI.png)

Even more hurt.  I frames 4x larger then P frames  :(
That makes sense, I'm shooting for low bitrate and Canon is doing it's best to waste bits  ???

A reduction in I frame size would allow an increase in P frame size, for the same GOP size.

edit:  I frames being reference frames for all the P frames in that GOP, they do need to be larger (better quality).  4x larger, no!


Does anyone see the difference in the screenshots I posted on the first page of this thread?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Audionut on July 21, 2012, 02:47:02 AM
Why is GOP size so important.  Look at the last example I posted above.  MVI_8661.  There are approximately 31 I frames in that sequence.  I can tell you that the scene was almost static, a little zoom or 2.  With a massive GOP size, lets say 500 frames (ignoring seeking ability, editing ability, spec requirements etc), a good encoder would have only needed say 3 or 4 I frames.

So instead of having 31 frames @ 48Mbps, we could have gotten away with only 4 frames @ 48Mbps, which would have lowered the overall bitrate of that sequence.  And given I was obviously shooting for a very low bitrate, I could have either had the same quality with a lower final bitrate (filesize), or a higher quality encode at the same bitrate.

Alex.  Regardless of any quality improvements there may or may not be with adjustable GOP and/or IP ratio sizes, there is a small demand for this functionality.  I've already donated once, and I would like to offer a $20 bounty.  After 2.3 goes final of course.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on July 21, 2012, 03:30:28 AM
So 235MBps is the theoretical limit... and camera write would be 30MB/s... no card exists that can do this without UHS mode.. yet. Makes sense what we are getting... about 1/2 of top bit rate at 1/2 write speed.


I see the difference in the leaves, I think they just assume that difference not big enough. Seems encoder quality is so "poor" because canon wastes bandwidth.  How do the ratios look by default... still this bad? What about scenes with motion?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Audionut on July 21, 2012, 08:11:08 AM
Quote from: 1% on July 21, 2012, 03:30:28 AMI see the difference in the leaves, I think they just assume that difference not big enough.

Well it's a fairly significant difference for mine.  The high bitrate encode does a reasonable job of maintaining detail, the default settings just blur.

Quote from: 1% on July 21, 2012, 03:30:28 AMSeems encoder quality is so "poor" because canon wastes bandwidth. 

It's just a poor encoder, period.  And to be honest, at best, the most that GOP and/or IP ratio changes would make (to quality) are similar to what I posted in the first page of this thread.  And if people are having a hard time noticing those differences, I'm fighting a losing battle to get GOP/IP ratio adjustment implemented.

Quote from: 1% on July 21, 2012, 03:30:28 AMHow do the ratios look by default... still this bad? What about scenes with motion?

Default was very similar to the low bitrate encode.  I frames around 4 times larger.  I haven't tested any motion scenes.

I'll find some time in the next few days to run tests on sequences with motion.  I'll also do some bitrate tests involving people.  Maybe if people see the lack of fine details in peoples faces with the default bitrates compared to high bitrate encodes, it might have more of an effect then a leaf.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on July 21, 2012, 08:25:43 AM
It was enough difference for me to go buy a faster card.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: nanomad on July 21, 2012, 10:37:39 AM
Quote from: 1% on July 21, 2012, 08:25:43 AM
It was enough difference for me to go buy a faster card.

Agreed, the difference is definitely there
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: weldroid on July 21, 2012, 11:42:25 AM
Actually, the difference is very easy to visualize: low FPS-es with FPS override let us choose virtually any VBR levels up to (down to) -16 irrespective of SD card capabilities. So just shoot something with lots of details with -8 and -16 and add some sharpening in post...

https://dl.dropbox.com/u/4848144/0.png

vs

https://dl.dropbox.com/u/4848144/-16.png

Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: a1ex on July 21, 2012, 01:48:41 PM
Hm, the first one looks pretty defocused.

Now, depending on scene complexity, with CBR 1x, QScale may be -16, may be -8, may be 0... may be +16 in extreme cases...

So, for timelapse recording it's probably a great idea to set it at -16 without worrying.

Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on July 21, 2012, 04:18:18 PM
Any quick way to lock the ratio to 1.5x I to P? To see what happens... Or set I+P equal?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: a1ex on July 21, 2012, 04:42:01 PM
Yes, in bitrate.c, rather than scaling all those numbers by a constant factor, compute the average first and scale that one.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on July 21, 2012, 05:38:31 PM
MOV_OPT_NUM_PARAMS; = I frame?
MOV_GOP_OPT_NUM_PARAMS = p frame? or gop size?


Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: a1ex on July 21, 2012, 05:54:21 PM
OPT_NUM_PARAMS: those are 2, one is P, the other is I.

The others... I don't know.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on July 21, 2012, 06:20:29 PM
   
int avg=0;

        for (j = 0; j < MOV_OPT_NUM_PARAMS; j++)
        {   int* opt0 = (int*) &(mvr_config_copy.fullhd_30fps_opt_size_I) + i * MOV_OPT_STEP + j;
// int* opt = (int*) &(mvr_config.fullhd_30fps_opt_size_I) + i * MOV_OPT_STEP + j;
            if (*opt0 < 10000) { bmp_printf(FONT_LARGE, 0, 50, "opt_set: err %d %d %d ", i, j, *opt0); return; }

avg += (*opt0) / (j+1) ;
          // (*opt) = avg * num / den;
//          (*opt) = (*opt0) * num / den;

}

int* opt = (int*) &(mvr_config.fullhd_30fps_opt_size_I) + i * MOV_OPT_STEP;
            (*opt) = avg * num / den;




/Bad At C
This will compute the average?

I tried this... not sure if it worked. Overall bit rate is lower but still hits 90s.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on July 21, 2012, 09:22:16 PM
void opt_set(int num, int den)
{
    int i, j;
    int avgI=0;
    int avgP=0;
for ( int qq =0; qq < 9; qq++)
{ int* opt1 = (int*) &(mvr_config_copy.fullhd_30fps_opt_size_I) + qq * MOV_OPT_STEP + 0;
int* opt2 = (int*) &(mvr_config_copy.fullhd_30fps_opt_size_I) + qq * MOV_OPT_STEP + 1;
avgI += (*opt1) / (qq+1);
avgP += (*opt2) / (qq+1);
}




    for (i = 0; i < MOV_RES_AND_FPS_COMBINATIONS; i++) // 7 combinations of resolution / fps
    {
#ifdef CONFIG_500D
#define fullhd_30fps_opt_size_I fullhd_20fps_opt_size_I
#define fullhd_30fps_gop_opt_0 fullhd_20fps_gop_opt_0
#endif

#ifdef CONFIG_5D2
#define fullhd_30fps_opt_size_I v1920_30fps_opt_size_I
#endif

int* opt = (int*) &(mvr_config.fullhd_30fps_opt_size_I) + i * MOV_OPT_STEP + 0;
                  (*opt) = avgI * num / den;
       
opt = (int*) &(mvr_config.fullhd_30fps_opt_size_I) + i * MOV_OPT_STEP + 1;
  (*opt) = avgP * num / den;




+ Delete the first j loop.

Bit rate gets very serious... instant BR catches up to average very quickly. Scaling factors also more noticeable.

Did I do this right?
I can't scale B/R viewer like you guys have it.
(http://i.imgur.com/J1Unc.png)


Still spiked on Lower ISO + No motion

(http://i.imgur.com/YezNn.png)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on July 22, 2012, 03:01:55 AM
.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Audionut on July 22, 2012, 04:13:09 AM
Looks like you're making some good progress there.

I'm probably getting to far ahead here, but are there plans of being able to make this a user adjustable parameter, or will it be a fixed parameter based on testing.

x264 sets it's IP ratio @ 1.4.  So it's probably a good starting point  :)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on July 22, 2012, 05:05:17 AM
Don't know. I'm just testing. I'm not sure which is I/P yet or if its right. I can scale them differently.. I think I almost have to as I can stop recording much easier and I'm seeing much higher bit rates, esp in crop.

*I am going to try to make adjustable independent I/P scaling and maybe a unified bin for testing it.  Then we can see if its doing anything useful.
As is here: https://bitbucket.org/OtherOnePercent/tragic-lantern/changeset/1d40d7f8736d


Have to see if 550D and others have different numbers of movie modes. 600D has 9 so the averaging loop has 9.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on July 22, 2012, 07:08:13 PM
http://www.qfpost.com/file/d?g=VoeCZ1i1x (http://www.qfpost.com/file/d?g=VoeCZ1i1x)

Unified you can test.

500D and 5d2 may be incomplete as their scaling is different.

Also, find if I/P is right.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Audionut on July 23, 2012, 12:58:12 AM
Cheers 1%.

I'll run some tests and report back.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Audionut on July 23, 2012, 04:11:42 AM
What I have noticed so far.

IP ratio should really be 1 setting.   ie:  Setting say 1.5 should mean that I frames are 1.5 times larger then P frames.

As for my tests below, I'm still not sure if I should be posting them as they are very confusing.  ???

All these tests done at CBR 0.5x
(http://i.imgbox.com/adx7WiJZ.png)


Adjusting P setting down, reduces bitrate substantially for the same CBR setting.  Increasing the IP ratio.
Note:  In previous tests, reducing the bitrate in Qscale mode also increased IP ratio.  So it appears that there in no actual adjustment of IP ratio happening here.  Just bitrate changes.
(http://i.imgbox.com/absJTNMA.png)

Adjusting P setting up, slightly increases bitrate.  Same IP ratio.
(http://i.imgbox.com/aboQZEOu.png)

Adjusting I setting either up or down, increases bitrate slightly with very minor changes to IP ratio.

Adjusting P setting to 0.1x and I setting to 3.0x increases bitrate substantially while reducing IP ratio.
(http://i.imgbox.com/adk7JYVb.png)


Unfortunately, this is all scene dependent.  Here is another test at various settings until buffer full recording stop.
Qscale 16
(http://i.imgbox.com/adpd5KVP.png)

P 3.0x - I 0.1x
(http://i.imgbox.com/ackLXmEi.png)

P 0.1x - I 3.0x
(http://i.imgbox.com/aduuunfp.png)

Here, the only changes worth mentioning are that the last test was able to record for longer before buffer full.


Conclusion.  Canon has some funky code going on to adjust IP ratio.  This seems to be based on scene complexity.  Probably to try and help it maintain a constant bitrate.

This patch does not appear to be changing IP ratio.  It appears to me as if it's affecting the bitrate distribution.  For instance, adjusting P ratio down, isn't actually adjusting the ratio of P frames in anyway, simply saying, for this CBR setting, encode at a lower bitrate.  And it's Canons internal code that is making the changes to IP raito.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on July 23, 2012, 06:05:14 AM
It doesn't do anything in qscale mode. That ignores the scale factor and only sets qscale. Try CBR of over 1.

I think this sets p frame and i frame bitrate size. It does use the canon function... that is all we have in bitrate.c. It takes the average of the values and scales it. So for CBR X adjust bit rate factpr of P and I to X. When I encoded at 2.0x and set 1.3 I to 1.0 P, I frame was ~1.4 x the size of the P frame next to it.

I'll try to scale individual numbers vs average next and see if that changes anything. I don't think anyone else has a fast card to test so any test is good.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Audionut on July 23, 2012, 06:22:51 AM
Quote from: 1% on July 23, 2012, 06:05:14 AM
It doesn't do anything in qscale mode. That ignores the scale factor and only sets qscale. Try CBR of over 1.

Sorry.  In the last 3 screenshots.  Qscale -16 was used in the top one.  CBR mode was used in the last 2.  I don't recall what CBR it was, I was just adjusting each one until I was getting the maximum bitrate before buffer full.  I think I got to CBR 1.6x.


Quote from: 1% on July 23, 2012, 06:05:14 AMI think this sets p frame and i frame bitrate size. It does use the canon function... that is all we have in bitrate.c. It takes the average of the values and scales it. So for CBR X adjust bit rate factpr of P and I to X. When I encoded at 2.0x and set 1.3 I to 1.0 P, I frame was ~1.4 x the size of the P frame next to it.

Ok, that makes a little more sense for me.  Except that in all my tests setting 0.1 P and 3.0 I was the biggest difference I could make.  And actually reduced the size of I frames compared to P frames.
edit:  And if I'm understanding you correctly, the exact opposite should have occurred.

I think the biggest problem with the current method is that it's scaling the frames to some internal Canon code.  So the results will always vary (almost unpredictable).

There needs to be a way of just scaling the P frames to the I frames.  So for instance, whatever internal Canon code decides the size (bitrate) of an I frame, lets scale our P frames to that.

P frames always 10% smaller then I frames. Or 20% or 30% etc etc.

Quote from: 1% on July 23, 2012, 06:05:14 AMI'll try to scale individual numbers vs average next and see if that changes anything. I don't think anyone else has a fast card to test so any test is good.

Look forward to testing your changes.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on July 23, 2012, 07:14:24 AM
P frame/ I frame could also be backwards. I can also make it scale in qscale, right now for that it uses a factor of 1 on everything.

After that probably have to look in the firmware. Also... any affect on quality? Seems like not scaling everything the same leads to higher bit rates.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Audionut on July 23, 2012, 07:25:21 AM
Quote from: 1% on July 23, 2012, 07:14:24 AMAlso... any affect on quality? Seems like not scaling everything the same leads to higher bit rates.

I haven't looked at quality yet.  I was getting to much (bitrate & IP ratio) variation in the tests I was doing.  I would like to be able to consistently get P frames a certain percentage smaller then I frames.

Then I can concentrate on getting the same bitrate at various IP ratios to be able to judge quality effects.

I am finding that 0.1 P 3.0 I as it stands, does allow a higher (average) bitrate to be captured.

Quote from: 1% on July 23, 2012, 07:14:24 AMP frame/ I frame could also be backwards.

That makes sense.  I was getting very confused with the options as they currently work.  :)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Audionut on July 23, 2012, 07:56:38 AM
Bug report.

I ratio 1.0 to 1.6 scale up bitrate as would be expected.

I ratio from 1.7 through to 2.9 result is bitrate of 2mbps.  <--Something wrong here.

I ratio 3.0 results in very large bitrate increase.

Assuming this is currently backwards, so this is effecting the P frames.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: a1ex on July 23, 2012, 12:54:34 PM
Small hint: you can try to expose the raw coefficients (2+5 if I remember well) on the user interface, or in config file. This way, the testers can try any values and probably come up with more meaningful results.

After we understand the exact meaning of each coefficient, we can think at some user-friendly controls (like i/p ratio or something like this).
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: ilguercio on July 23, 2012, 02:53:28 PM
I'm sorry if i can't follow you well with the maths but, can you tell me briefly what's going on?
If you need some testing i can do it on my 50D with a pretty good card (8GB 400x) .
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on July 23, 2012, 03:09:53 PM
Try it and see if it works for you. I guess next step is toggling averaging and dumping the coefs to screen. Then once we know what numbers go in there we can load all from the config file. Will be more of a pain to toggle unless you overwrite the config file with PTP or something. I'll switch I/P too.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Chungdha on July 23, 2012, 05:08:20 PM
I ran new 2.3 on 550D with Sandisk extreme PRO 95mb/s and tried out CBR 3x with audio off and it ran quite long till I ran into a too complex scene. But Audio on could only do CBR 1.6x I wonder why the sound recording is taking basically like half the buffer while its should be processing less information than the video part.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on July 24, 2012, 06:02:25 AM
Wonder no more: audio takes up ~40mbps
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on July 24, 2012, 09:52:30 PM
New bin:

* Switch I/P
* Turn on/off I & P averaging
* Use I/P in qscale mode

I tried setting the scale over 3.0x  but that actually produced lower bit rates. I'll have to set up a debug mode so values can be seen on the screen next.

http://www.qfpost.com/file/d?g=SQBAcMzad (http://www.qfpost.com/file/d?g=SQBAcMzad)


With more investigation....

NSTUB(0xFF1C94E0, mvrSetDeblockingFilter) < Good or bad? And how do you set it.

NSTUB(0xFF1C9AB4, AJ_MvrSet_HD_VGA_Gop_size) has no arguments? Sets a bunch of the other parameters.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Audionut on July 25, 2012, 09:35:10 AM
Quote from: 1% on July 24, 2012, 09:52:30 PM
With more investigation....

NSTUB(0xFF1C94E0, mvrSetDeblockingFilter) < Good or bad? And how do you set it.

NSTUB(0xFF1C9AB4, AJ_MvrSet_HD_VGA_Gop_size) has no arguments? Sets a bunch of the other parameters.

Being able to adjust the deblocking filter would be handy.

Cheers 1%, I'll find some time to test the new build tomorrow.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: ilguercio on July 25, 2012, 11:16:48 AM
Can't access ML on your autoexec, trash button doesn't do anything.
I can access it only while in LV mode. Still, the continuous redrawing of the screen doesn't allow me to set things properly.
Send this patch to A1ex and let him compile the whole, maybe?
I can test it right now.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: a1ex on July 25, 2012, 11:19:00 AM
QuotemvrSetDeblockingFilter
I've tried these, but could not notice any effect, even at CBR 0.1. Maybe my eye is not trained for this.

Those mvr functions set values in mvr_config structure. I prefer to set them manually, rather than calling mvr functions (because you have access to all parameters in a portable way).

They have usually one argument - a pointer to an array (or structure) containing all the parameters they use. They are also registered via the call-by-name (eventproc) mechanism.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: ilguercio on July 25, 2012, 11:21:19 AM
Well, i just shut the camera off with this autoexec while still in LV and the camera generated a strange noise or a series of very fast beeps, i can't understand.
What is this? :S
I can record it if you want.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: a1ex on July 25, 2012, 11:27:33 AM
Beeps?! Like autofocus beeps?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: ilguercio on July 25, 2012, 11:31:19 AM
Nope, and by the way beep is always disabled on my 50D (just double checked it) .
This is the noise. It has never done it with any other build, just with this one from 1%.
https://dl.dropbox.com/u/1087972/ML/strange%20noise.WAV
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: a1ex on July 25, 2012, 11:34:10 AM
Very cool - if I know how to get these from normal ML, we could solve the audio sync part :d
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: ilguercio on July 25, 2012, 11:37:43 AM
Lol, it does it only when i turn the camera off while in LV but it is not reproduceable because it doesn't always occur.
Ask 1%, i never experienced such stuff.
What might it be, in your opinion?
I can't understand if it's coming from the little speaker placed in the back cover or it is something related to the screen trying to shut down (and all the electronics related) .
Does the LCD driver have the ability to trigger sounds (ok, that was just a wild guess) .
Anyway, you should probably allow users to fiddle with each available setting related to the encoder.
I can't test with this build, i don't know if it's my 50D that doesn't like it.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: a1ex on July 25, 2012, 11:41:49 AM
If 1% can post a diff of the code, maybe I can figure out what's going on.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: ilguercio on July 25, 2012, 11:43:15 AM
See this, the datasheet of the LCD driver.
https://dl.dropbox.com/u/1087972/ML/hint.jpg
That could be it ;)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: a1ex on July 25, 2012, 11:45:23 AM
So... we triggered the buzzer by mistake? :D
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: ilguercio on July 25, 2012, 11:51:05 AM
Probably, why did it buzz anyway?
It doesn't do it all the time, can you take his code and try to log what happens at shutdown?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: ilguercio on July 25, 2012, 11:53:36 AM
https://dl.dropbox.com/u/1087972/ML/IC-ON-LINE.CN_lc75835w_4254790.pdf
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on July 25, 2012, 05:08:03 PM
If you're having issues with the menu its because I disabled menu piggyback and menu closing to have access to ML while in the movie player. I guess this is the side effect. I can take it off.. I think we've learned all we need from play mode.

2 audio registers may be switched on other cameras from when I was audio testing with them. Can't really think of anything else.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: unity2k on July 25, 2012, 05:18:06 PM
I'd like to add a thought, maybe not the smartest, but anyway. What if upon the start of recording video, a sync sound is recorded, both on camera and on an external audio device, at this point the audio is disabled on camera and the CBR is pushed higher, as the recording approaches the end of what is needed to be filmed, a button push lowers the CBR, audio is re-enabled and a second sync sound recorded? Then in post the sound could be synced back to the footage (think PluralEyes).
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: ilguercio on July 25, 2012, 05:43:01 PM
But why is the something buzzing?
It is not the usual beep.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on July 25, 2012, 05:44:58 PM
Something in audio.c write to those registers on 50D. I only made sure it didn't on 600D.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: ilguercio on July 25, 2012, 05:50:53 PM
But the 50D doesn't have audio so where does it come from?
It could be the LCD driver buzzing but it's just speculating about nothing, really.
Could you generate a beep from the 50D? It has got a tiny speaker for the audio confirmation of the focus and i think that's the only thing it can generate.
What do you think?
Also, what is left about the video compression codec?
Just P and I regulate the amount of data?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on July 25, 2012, 06:04:40 PM
Lots is left about the codec, I looked in the firmware. Can alter the movie recorder to ignore 30 minute and 4gb limit with some effort. that is the only place I see the limit in firmware (so far). Movie recorder is just set to stop on those, nothing in actual file writing stuff.

There are the gop opts to figure out and setting gop length... that deblocking filter.


* look in audio.c and find what is writing to what and then experiment. You'll find your LCD speaker or whatever that makes noise... maybe you have an audio IC thats not used? I don't have 50D. I'll fix the trash button stuff though we have PTP I don't think its needed anymore.

Look through repo/audio.c and see what is writing

https://bitbucket.org/OtherOnePercent/tragic-lantern


*mvrSetD_FULLHD


is this gop length?

The modes:
FULLHD = 1080P?
HD= 720P?
VGA= duh?

5 gop bitrate options for each mode. Good thing is we can now ONLY tweak FULLHD and forget the crazy loops.


50D only:

http://www.qfpost.com/file/d?g=d3NJAuL9W (http://www.qfpost.com/file/d?g=d3NJAuL9W)

Does this fix ML menu on 50D?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Chungdha on July 25, 2012, 09:11:38 PM
Any possibility to lower the sound bitrate as its should not be using that much as sound recorder still use standard sd cards with way less bitrate than that, as max sound be just 192kb/s
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: ilguercio on July 25, 2012, 09:26:14 PM
Well, the guys which have initially analyzed the firmware have found references to AK4646 as audio chip but i can't see it on my camera.
I don't know if it is integrated in something else or what, i even asked lensrentals' Roger Cicala about that and he said he couldn't find it even on 5DII and III so that could be a clue and a hope.
Unfortunately i don't know about C and ARM in general, i can just guess and test but i can't surely write any code.
I'm going to try your autoexec and see if it solved the problem with my 50D.
Thanks.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on July 25, 2012, 09:40:41 PM
Looking at 50D firmware would probably tell all. Or try writing to power registers of 50D for AK style IC and Lapis style IC.


QuoteAny possibility to lower the sound bitrate as its should not be using that much as sound recorder still use standard sd cards with way less bitrate than that, as max sound be just 192kb/s

You can lower sampling rate. The audio is uncompressed. Its something to investigate for sure.

Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: ilguercio on July 25, 2012, 09:46:55 PM
I can't even compile the code, if you know any good manner of getting me started into this i would really appreciate it.
Things started here:
http://chdk.setepontos.com/index.php?topic=6161.75
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on July 26, 2012, 03:55:38 AM
Thats a bit of a process... quickest way is to get current ML source and edit the 50D parts. You can compile on linux and mac (yagarto?) not sure on windows. If you don't know anything about C its going to be a tough road.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on July 26, 2012, 08:45:04 PM
*Update

Set D1 and D2 values. I get abnormal instant bit rates even over 500 but then MvRecorder seems to reset to regular values without stopping. Maybe there are more hidden things in the other registers?


*Update 2

looking through the fw, gop opts also set the timers? like x67e4;   

D1_30fps;   is never set by canon functions, why?

Register    ; 0x2f is set by BOTH gop opts and D setting functions but I don't see it in the MVR.H

I saw the discussion on google ~(2011) and wow, nobody really looked at this besides alex? Nobody wants higher quality videos to compete with GH1?

Any more hints?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Leon on August 01, 2012, 03:19:12 PM
@ 1% :  How on earth can audio be 40 Mbps?  I just demuxed a video shot with the firmware default, and the audio file is 1.5 Mbps, which is what it should be at 48,000 Hz, 16 bit, stereo:  48000 * 16 * 2 = 125 kB/s = 1.5 Mbps.

The question is why does this 1.5Mbps make such a difference to the max video bitrate allowed?  Does anyone know?

@weldroid :  Thanks for those reduced FPS images.  The difference is obvious.  Even if one may be slightly defocused, there's still a lot more compression artifacts in the more compressed frame, and it would be even more noticeable after editing.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: ilguercio on August 01, 2012, 04:47:27 PM
1%
Got anything new for us?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 02, 2012, 02:12:01 PM
Not yet, blame work.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 10, 2012, 11:01:10 PM
Ok, made a new unified with everything scalable individually. Watch for MVR resets. The CBR "scale" does nothing since everything is separate. Changing D values makes MVR reset or produces giant bit rates, maybe we'll find out why in testing.

Let me know if you find anything interesting.


http://www.qfpost.com/file/d?g=adnsX5lXl
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Chucho on August 11, 2012, 02:49:54 AM
Does any body know how [JPCORE] works? It has a lot of H.264 libary functions like JporecoreSliceqpd which is the Slice QP delta Quantization scale it range is -26 to 25. Also JpcorePicpyPicpcD which are the picture py an pc chroma QP offset for the loop-filter it range is -12 to 12. And then you have JpcoreDbfalphaDDbbetaD which is connected to Mvrsetdeblockingfilter the loop-filter alpha/beta table offset are -6 to 6. Making the deblocking filter stronger should improve the muddy shadows. Also MrvSetGopOptSizeFULLHD is set to 12 setting it to 1 should give you all I-frames no?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 11, 2012, 04:22:07 AM
Its worth looking into. Might make more real changes.

; **'mvrSetGopOptSizeFULLHD GOP_OPT_SIZE_FULLHD([0]=%d, [1]=%d, [2]=%d, [3]=%d, [4]=%d)'

Aren't all the arguments of this function just the gop opts?

Does it have another address in the MVR config?

Mvrsetdeblockingfilter the loop-filter alpha/beta table
Where is that?

somewhere here?

*(-4 + sp0) = lr0
*(-8 + sp0) = unk_R5
*(-12 + sp0) = unk_R4
*(-16 + sp0) = arg3
*(-16 + sp0) = BYTE(arg0->off_0x4)

where is sp0?


Maybe:

aAJ_JpcoreSliceqpdD_Qscale_0x00_to_0x20.off_0x1 /*0x6CE1*/ = BYTE((arg0 & 0xF))
aAJ_JpcoreSliceqpdD_Qscale_0x00_to_0x20.off_0x2 /*0x6CE2*/ = BYTE((arg1 & 0xF))
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: ilguercio on August 11, 2012, 05:44:35 AM
Quote from: 1% on August 10, 2012, 11:01:10 PM
Ok, made a new unified with everything scalable individually. Watch for MVR resets. The CBR "scale" does nothing since everything is separate. Changing D values makes MVR reset or produces giant bit rates, maybe we'll find out why in testing.

Let me know if you find anything interesting.


http://www.qfpost.com/file/d?g=adnsX5lXl
Trying it now but it is not easy to spot differences other than pure bandwidth.
Do the guys that have hacked the GH2 know something about those settings already?
They could help us somehow.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Chucho on August 11, 2012, 06:15:26 AM
GH2 encoder is software based Canon's is hardware based
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Rush on August 11, 2012, 11:37:49 AM
Thank you for all that you do here. Hope your investigations will result in better bitrate control.

Just some test of current ML birate control, not with the version posted above (ISO 6400, 1/50, sharpen filter, 400% crop with 2x upscale)
Original 45 mbit
(http://ipicture.ru/uploads/20120811/86E8qOhn.png)
QScale16 165 mbit (in motion details looks even better)
(http://ipicture.ru/uploads/20120811/jQfwYmVf.png)

High bitrate really helps to keep details with high ISOs
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 11, 2012, 04:50:13 PM
Gop2 only changes when you average. Otherwise it doesn't seem to scale at all.

values are:

(int)5570568
(int)7261685 - when averaging.


I tried to travel to 0x6CE0 and see if de-blocking filter can be changed. I don't know if I'm in the right spot but the value there kept changing while recording.

How to set mvrSetGopOptSizeFULLHD? Would settign the gop opts to specifics instead of scaling produce equivalent results? I want to try all I frame footage.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Rush on August 11, 2012, 05:56:44 PM
Quote from: 1% on August 11, 2012, 04:50:13 PM
How to set mvrSetGopOptSizeFULLHD? Would settign the gop opts to specifics instead of scaling produce equivalent results? I want to try all I frame footage.
I don't think that all-I will be good idea, because it requires higher bitrate with same image quality.
Increasing GOP length should help with increasing quality maintaining same bitrate
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 11, 2012, 06:21:18 PM
If we can set it one way, we can set it the other way too.


Default GOP Options:


// 24P Defaults
// I 62992200 (3c12F48)  p 19594200 (12AFBD8)
//d1 262144 = 40000 d2 262144
//g0 3833856 (3A8000) g1 3309568 (328000) g2 2785284 (2A8004) g3 2260992 (228000) g4 17736704 (10EA400)

Possible D values:

//DSizes: 78643, 288358, 393216, 524288, 629145, 681574

When you change certain options others go to defaults or only to certain values.

http://www.qfpost.com/file/d?g=5N1nnCmy6


600D Fat32 - Adjust Deblocking Filter A/B
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 23, 2012, 01:08:52 AM
Having set the deblocking filter correctly, I've gone onto picqpy, piqpc.

So far picqpy=20 is the highest without camera completely freezing. Bit rate increases for sure, original value was 26. Its calculated in jpeg quality from ~51 &0x3f but lower values like 12, 18, don't work.

PicqPC=0 by default. Anything else causes camera to hard freeze like it did with invalid picqpy

Setting through:NSTUB(0xFF1C94A8, mvrSetQscaleYC) allows setting of piqpc but for some reason doesn't change picqpy.

I think I have the array right.... 0 and 1 positions?
it calls:

jpeg_pic_quality(BYTE(*(arg0)), BYTE(arg0->off_0x4))


*ok, it appears either or. Set QPY to 20 get higher BR, set QPc +6 to -6 (58 after math)...

Altering QPC & QPY together causes hard lock. Which one is a better idea?

You can try QPy at 20.

https://bitbucket.org/OtherOnePercent/tragic-lantern/downloads/autoexec.bin.600D.Exfat.Audio.08.PiqQPy.bin

1st noticeable effect... preview is messed up. Videos OK.

Can now set Gop lenght and its verifiable + written. Bad news is when buffer overruns you get error 70. Bit rate seems lower, quality untested yet. Def way less P frames.


ASSERT: 0
at Fcreate\FcsMaker.c:2575, task MovieRecorder
lv:1 mode:20


Magic Lantern version : v2.3.NEXT.2012Aug23.600D102
Mercurial changeset   : c0cc0716e6ce+ (unified) tip
Built on 2012-08-24 00:43:15 by user@D610.
Free Memory  : 334K + 1396K


This happens after recording is stopped. I don't know if quickly switching it back at recording end would block the soft error 70 or if there is another way.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Audionut on August 24, 2012, 03:16:06 AM
Hi 1%,

Sorry for going awol, life got busy.  Any chance you can make a unified build?

Quote from: 1% on August 23, 2012, 01:08:52 AM
Can now set Gop lenght and its verifiable + written. Bad news is when buffer overruns you get error 70. Bit rate seems lower, quality untested yet. Def way less P frames.

Increasing GOP length should result in more P frames for every I frame.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 24, 2012, 04:11:09 AM
I have to make everything work on other cameras. Deblock filter is ok but picqy, piqc are set by address. Also have to put averaging in a loop so its not just for 24p mode.

I just recorded all I frames and its REALLY easy on the buffer but bitrates are still high 90s/100s with no buffer warnings. Wonder what's up with quality.

-other observations.
1. BR is less controllable... can set to q-16 and 3.0x and camera will go at the speed of the card 40-120mbps with the buffer at 1-4%.
2. File size a little bigger, card constantly written (like buffer skipped).
3. Crop Mode (3x) + FPS override affect bit rate and can crash recording (~140-160mbps)
Graph looks like this:

(http://i.imgur.com/A64BL.jpg)

If we can stop the error 70 and this doesn't adversely impact quality it would be a winner. I've never written such high BR video this consistently.

So what does increasing gop length do? Should be theretically better, right?
Gop size at 20.... buffer runs out much quicker, BR are much lower and look at this spiky thing.

(http://i.imgur.com/ahRwA.jpg)

Long ass thread from GH1/2 ppl on this topic:
http://www.personal-view.com/talks/discussion/421/official-low-gop-topic/p1

Compare frames:

http://www.sendspace.com/filegroup/e5qNpEKnFi%2BJnPzvHvBAow (http://www.sendspace.com/filegroup/e5qNpEKnFi%2BJnPzvHvBAow)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Rush on August 24, 2012, 12:14:58 PM
1%, very nice! It looks like I-frames don't use buffer, and P-frames use buffer.
So it seems that we locked with quality - high bitrate ALL-I should look like low bitrate with GOPs...
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Audionut on August 24, 2012, 02:09:34 PM
Quote from: 1% on August 24, 2012, 04:11:09 AM
So what does increasing gop length do? Should be theretically better, right?

GOP is a Group Of Pictures.  Since the Canon codec only has I & P frames and the default GOP IIRC being 12, a GOP would look like this.

IPPPPPPPPPPP

And this just repeats, IPPPPPPPPPPPIPPPPPPPPPPPIPPPPPPPPPPP

I frames are larger then P frames.  As I mentioned earlier in this thread, they can be up to 4x larger.  I frames need to be larger as they are the reference frames for P frames.  P frames are not a complete picture, they only contain the detail that has changed since the last I frame.

Baring any scene detection that might be in the Canon code, if we increase the GOP size to a more general 10 x framerate (24fpsx10) 240, we can have 1 large I frame followed by 239 P frames.

Where as, with the default Canon GOP size, for the same 240 frames, we would have a total of 20 (large) I frames.

Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 24, 2012, 04:50:35 PM
I'll try 240 and see what happens. Did you notice quality fall in the sample pics? I don't think its quite as bad as low BR I/P. the frames are about the same size (~5mb) . Bit rate for I only goes up to 140 without buffer filling.

I need a consistant way to test what this is doing to quality, any ideas. I don't have curtains, test chart is hard to tell a diff.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: nanomad on August 24, 2012, 05:09:24 PM
Quick idea: hack a script in bash that grabs a couple of frames using mencoder and then gives them to PDIFF
http://pdiff.sourceforge.net/

edit: Uhm, that only works if the camera and lighting stays still :/
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 24, 2012, 05:25:21 PM
I'll have to try that.

240 is unworkable.. buffer fills at very low BR and recording stops. Qscale says +24? and BR is in the 1/2s (all options at .1x) before I can get it to not stop instantly.

This error 70 at every stop is getting quite annoying, going to have to trace it in that function and see where it produces Assert: 0 and if there is a way to patch whatever it compares

*Gop3 is also easy on the buffer. Q-16 locked it produces high 90s to 110 with ~14% buffer usage. 3 and 6 is what the GH1 people were using.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Rush on August 24, 2012, 08:12:42 PM
Good way to see difference is to record with high ISO (3200+) something with fine details.
With bad quality this fine details will be blurry and smudged by noise. With good quality this details will stay fine and noise will be more detailed too. (here is example (http://www.magiclantern.fm/forum/index.php?topic=1565.75#msg7150))
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: g3gg0 on August 24, 2012, 08:53:03 PM
i asked the company thomson if we can get a trial of their tool "AVC analyzer"

http://www.thomson-networks.com/products/monitoring-switching/media-file-analyzer

they sent me a dongle with 100 days evaluation for free :)
that tool is *really* able to look into every single frame and lets you check all coefficients down to the raw stream.
upload the files you want to compare and i will look at them  (or give you access to my PC or a VM via RDP so you can work with it)

want to access?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: g3gg0 on August 24, 2012, 09:07:18 PM
here the output of some movie i recorded and analyzed

http://upload.g3gg0.de/pub_files/c4e2826707bdc0c9a88831de35451f28/H264_Parsing.log

and here screenshots of that tool
http://upload.g3gg0.de/pub_files/f82f98bdf0e202cdfd9340f6b557f7ab/shot1.png
http://upload.g3gg0.de/pub_files/0385b3de70e7d74f6055bf25cbb854a8/shot2.png
http://upload.g3gg0.de/pub_files/a55fd128fc03da9103e3b4cf931ea2ab/shot3.png
http://upload.g3gg0.de/pub_files/5c33631de2a773d9378a1462f3307585/shot4.png
http://upload.g3gg0.de/pub_files/cbdd60d7b681909bd41585f481ffdd99/shot5.png
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Leon on August 24, 2012, 11:32:21 PM
Sorry to just drop into the conversation – I thought it better to just ask than start a new thread – but I'd like to experiment with using CBR whilst setting a maximum Q-scale value and a minimum Q-scale value.  Is there an easy way to do this?  (My programming ability is limited but not non-existent.)

My reasoning is:

• Perhaps setting a minimum quantiser (e.g. -8) will reduce buffer overruns whilst maintaining a good quality.  With some of my "Class 6" cards I currently have to use CBR 0.8x for reliability, but maybe I could get away with CBR 1.0x limited to -8.

• Perhaps I can establish a maximum quantiser, maybe something like 0, where I never get buffer overruns and could improve quality by using that as another limit.

While I'm asking, does anyone know if -16 to +16 are equivalent to certain CQP quantiser values in x264?

Thanks!
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Audionut on August 25, 2012, 04:55:24 AM
Quote from: 1% on August 24, 2012, 04:50:35 PM
I'll try 240 and see what happens. Did you notice quality fall in the sample pics? I don't think its quite as bad as low BR I/P. the frames are about the same size (~5mb) . Bit rate for I only goes up to 140 without buffer filling.

Really need a scene shot form the same position (tripod) with the same lighting conditions.
Our eyes tend to bias to the bright side.


Quote from: 1% on August 24, 2012, 04:50:35 PMI need a consistant way to test what this is doing to quality, any ideas. I don't have curtains, test chart is hard to tell a diff.

Use anything with lots of details/texture..  Also need to test motion.  I'd suggest a static scene and walking through the frame.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Audionut on August 25, 2012, 05:03:58 AM
Quote from: 1% on August 24, 2012, 05:25:21 PM
240 is unworkable.. buffer fills at very low BR and recording stops. Qscale says +24? and BR is in the 1/2s (all options at .1x) before I can get it to not stop instantly.

A large GOP size should reduce bitrate, so there must be something else funky going on.

Are you positive that the changes you are making are having the correct effects to GOP?  Record a short sample and load that into bitrate viewer, switch to frame based mode and count the number of frames in between the I frames (large bitrate spikes).
Or send the files to g3gg0 who can check with the stream analyzer.

Once you are sure you are getting the correct changes to GOP, we will need to test if the Canon code has any scene detection. 
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 25, 2012, 05:22:41 AM
It is changing 100%, counted the I frames in BR viewer.

Quotebut I'd like to experiment with using CBR whilst setting a maximum Q-scale value and a minimum Q-scale value

You can already do this on my  builds, all the features work in CBR mode... I actually put qscale on -16 when I did all I frames and that stuff. There should be a unified build up that has this in the repo and works with all cameras. I don't think I looped averaging so its limited to 24p.

I'll have to make some tripod/controlled  videos and upload them somewhere.... like 10s each. Those logs are pretty exact.

Whoever has a 600D, you can make videos too. binaries are up for 1, 3, 240 gop. I do need to see where the err 70 comes from too.. other cameras may not get it just at the end.


Stream cut ~90mbps Gop3 Video

http://www.sendspace.com/file/3db3yd
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 25, 2012, 10:32:36 PM
The error 70 is coming from here:


FF255194: e59f2364 ldr r2, [pc, #868] ; 0xff255500: pointer to 0xa0f
FF255198: e28f1e32 add r1, pc, #800 ; *'Fcreate\\FcsMaker.c'
FF25519C: e28f0e33 add r0, pc, #816 ; 0xff2554d4: pointer to 0x30
FF2551A0: ebf6fc55 bl assert_0
FF2551A4: e5d4c00f ldrb r12, [r4, #15]


It is called here:

str:fcsSetThmMovieDesc_Fcreate\FcsMaker.c+728: assert_0(0xFF2554D4, 'Fcreate\\FcsMaker.c', 2575)

'fcsSetThmMovieDesc : Unknown GOP(%d)'

How to fix this?

*12 and 15 are valid options for most modes.

This bug will also show up when trying to write frame rate into file.

'fcsSetThmMovieDesc : Unknown FrameRate(%d)'

Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Leon on August 26, 2012, 02:01:24 AM
@1% :  Where can I download your build?  The closest I could find after 15 minutes searching was:  https://bitbucket.org/hudson/magic-lantern/downloads

When you say it will only work at 24fps, is that only if I change the GOP or I/P frames stuff, or will it also not work properly at 25fps or 30fps even if I only change Q-scale values?

Would I be better compiling a more up to date version myself?

Ta.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 26, 2012, 02:32:59 AM
In my repo, not in ML repo.

https://bitbucket.org/OtherOnePercent/tragic-lantern

When I say they don't work I mean the averaged values will only be set for 24p mode, noting will be set for 30p, etc until you turn off averaging. Regular scaling works, qscale works. Averaging mainly gets you higher numbers.


More on error 70:
Can you change bne to beq like in cracking and send it down the gop 12 branch?

FF255158: e5953040 ldr r3, [r5, #64] ; 0x40 FF255074
FF25515C: e353000c cmp r3, #12
FF255160: 03a0000c moveq r0, #12
FF255164: 0a000002 beq FF255174
FF255168: e353000f cmp r3, #15
FF25516C: 1a000004 bne FF255184
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Audionut on August 26, 2012, 06:22:49 AM
Quote from: 1% on August 25, 2012, 05:22:41 AM
Stream cut ~90mbps Gop3 Video

That's looking really good.  GOP size correct and you have a nice low IP ratio.

And a quick look on the laptop the video looks great.  No blocking with the fast motion and details appear to be fine.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: nanomad on August 26, 2012, 10:54:14 AM
Quote from: 1% on August 26, 2012, 02:32:59 AM
More on error 70:
Can you change bne to beq like in cracking and send it down the gop 12 branch?


From what I can see it's a static function call (i.e. not using a pointer). Since that code runs in ROM, the only way to do it would be to patch the code cache with the correct address which may or may not work...
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 26, 2012, 05:14:56 PM
What about:

http://magiclantern.wikia.com/wiki/Relocation


Can we relocate the whole function and NOP the assert_0 calls?

The function is only called by:


NSTUB(0xFF254B6C, str:PROP_PC_FLAVOR2_PARAM_PROP_PC_FLAVOR3_PARAM_fc)

FF255248: ebffff1e bl str:fcsSetThmMovieDesc_Fcreate\FcsMaker.c


Also, what about attacking NSTUB(0xFF0142FC, assert_0) itself and making it not call NSTUB(0xFF5680D4, assert), just print a log. That might soften a lot of non-serious crashes.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: nanomad on August 26, 2012, 05:48:52 PM
I have no clue about side-effects but that should be easier since assert_0 actually calls B assert


ROM:FF014318                 B       assert


So you should place something like that in boot-hack.c (the first defines goes into your platform consts.h and should be edited accordingly)


#define HIJACK_FIXBR_ASSERT 0xFF014318

#define FIXUP_BRANCH_NOLINK( rom_addr, dest_addr ) \
    INSTR( rom_addr ) = B_INSTR( &INSTR( rom_addr ), (dest_addr) )

// This goes near the other "fixes"
FIXUP_BRANCH_NOLINK( HIJACK_FIXBR_ASSERT, nop );

Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: nanomad on August 26, 2012, 07:18:43 PM
Yeah, too bad that code segment is never relocated to RAM  :(
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 27, 2012, 01:20:18 AM
Ok, thanks to g3gg0 and nanomad error 70 is fixed. You're free to set the gop how you like, 1-100 something.

https://bitbucket.org/OtherOnePercent/tragic-lantern/downloads/autoexec.bin.600D.Exfat.Err70-Gone
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Andy600 on August 27, 2012, 02:57:55 PM
Quote from: 1% on August 27, 2012, 01:20:18 AM
Ok, thanks to g3gg0 and nanomad error 70 is fixed. You're free to set the gop how you like, 1-100 something.

https://bitbucket.org/OtherOnePercent/tragic-lantern/downloads/autoexec.bin.600D.Exfat.Err70-Gone

Hi 1%, I'm trying this out but cant see any change in overall bit rate. I've set GOP to 1 but it looks like it's still recording at the default settings. I'm no expert with this (obviously). What settings should I dial in for an all-I frame test?

Cheers

Andy
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: a1ex on August 27, 2012, 03:02:43 PM
Asserts can be disabled easily from boot-hack.c, my_assert_handler (just don't call old_assert_handler). With this trick I've gained access to menus on a "bricked" 5D2 which was giving ERR70 at every startup (even without ML).

I wouldn't enable it in release builds because of potential side effects.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 27, 2012, 05:09:51 PM
Quote
Hi 1%, I'm trying this out but cant see any change in overall bit rate.

Did you toggle bit rate on the movie menu to something other than FW default. The setting affects whether ML will set the rates or not. Set CBR from the sub menu and go back to the movie menu, hit set and push the BR up to something with the arrow keys (1.5x, 2.0x, etc). I'd call it a bug but basically I didn't modify that part of the code (that x rate sets nothing).  After that you can go back and lock qscale by switching to VBR or just use encoder menu to change sizes.

QuoteAsserts can be disabled easily from boot-hack.c, my_assert_handler (just don't call old_assert_handler). With this trick I've gained access to menus on a "bricked" 5D2 which was giving ERR70 at every startup

We disabled just that 1 assert, sounds like it was worth the effort vs disabling them all because of what you say about stability. Now we can patch any jump, I have a feeling this will be very handy.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: nanomad on August 27, 2012, 05:18:22 PM
Quote from: 1% on August 27, 2012, 05:09:51 PM
Now we can patch ANYTHING, I have a feeling this will be very handy.

Fixed to reflect the awesomness  ;)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 27, 2012, 05:41:17 PM
And in that note, should look at where baseline profile and level is set... at the least could set it to 5.2 according to wikipedia... at the best up it to main or high profile if its in the encoder.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Andy600 on August 27, 2012, 05:57:10 PM
Quote from: 1% on August 27, 2012, 05:09:51 PM
Did you toggle bit rate on the movie menu to something other than FW default. The setting affects whether ML will set the rates or not. Set CBR from the sub menu and go back to the movie menu, hit set and push the BR up to something with the arrow keys (1.5x, 2.0x, etc). I'd call it a bug but basically I didn't modify that part of the code (that x rate sets nothing).  After that you can go back and lock qscale by switching to VBR or just use encoder menu to change sizes.

We disabled just that 1 assert, sounds like it was worth the effort vs disabling them all because of what you say about stability. Now we can patch any jump, I have a feeling this will be very handy.

Got it :) I hadn't toggled the bit rate
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 27, 2012, 07:16:03 PM
What was wanted with qscale limiting seems possible:

mvrSetLimitQScale LIMIT_QSCALE (L = %d, H = %d)'

I'll have to implement it, but right now I'm hard locking q to -16 when encoding because of greatly reduced buffer usage with reduced gop.

These are all the HW encoder parameters as initialized:


CreateResLockEntry(arg0->off_0x4, *(arg0)) => ret_CreateResLockEntry_FF1C82A4
*0x6244 = ret_CreateResLockEntry_FF1C82A4
CreateResLockEntry(arg0->off_0x8, arg0->off_0xC) => ret_CreateResLockEntry_FF1C82B8
*0x6248 = ret_CreateResLockEntry_FF1C82B8
*0x617C = arg0->off_0x10
*0x6198 = arg0->off_0x14
*0x61B4 = arg0->off_0x18
*0x61D0 = arg0->off_0x1C
*0x61EC = arg0->off_0x20
*0x620C = arg0->off_0x24
*0x6210 = arg0->off_0x28
*0x6168 = arg0->off_0x34
*0x6170 = arg0->off_0x38
*0x616C = arg0->off_0x2C
*0x6174 = arg0->off_0x30
*0x6110 = 1



Decoder sets parameters to decode baseline here... does that mean there are more profiles?



FF2F7C38: STRING: '\tSetConvertParameterForDecodeBaseline ' FF2F7934
FF2F7C60: STRING: '[ThbDec] XA= %d' FF2F7948
FF2F7C74: STRING: '[ThbDec] XB= %d' FF2F7964
FF2F7C88: STRING: '[ThbDec] XN= %d' FF2F7984
FF2F7C9C: STRING: '[ThbDec] YA= %d' FF2F79A4
FF2F7CB0: STRING: '[ThbDec] YB= %d' FF2F79C4
FF2F7CC4: STRING: '[ThbDec] YN= %d' FF2F79E4
FF2F7CD8: STRING: '=================================='



Someone with 5DMkIII fw... is encoder initialized a similar way? I'll keep looking.

Samples:
Here is a Gop4 video (~150MB)
http://www.sendspace.com/file/fipvng

I have a similar for gop3, just have to upload it.
Here: http://www.sendspace.com/file/mgj8ig (http://www.sendspace.com/file/mgj8ig)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 27, 2012, 08:09:07 PM
Here are encoder parameters for different modes (I hope # is value)



720P 1080P? 1080P Dzoom
pointer to 0x6108
#324 ; 0x144 #316 ; 0x13c #348 ; 0x15c
#328 ; 0x148 #320 ; 0x140 #352 ; 0x160
#16
#120 ; 0x78 #116 ; 0x74 #132 ; 0x84
#20
#148 ; 0x94 #144 ; 0x90 #160 ; 0xa0
#24
#176 ; 0xb0 #172 ; 0xac #188 ; 0xbc
#28
#204 ; 0xcc #200 ; 0xc8 #216 ; 0xd8
#32
#232 ; 0xe8 #228 ; 0xe4 #244 ; 0xf4
#36 ; 0x24 #36 ; 0x24 #36 ; 0x24
#268 ; 0x10c #260 ; 0x104 #292 ; 0x124
#40 ; 0x28 #40 ; 0x28 #40 ; 0x28
#272 ; 0x110 #264 ; 0x108 #296 ; 0x128
#52 ; 0x34 #52 ; 0x34 #52 ; 0x34
#96 ; 0x60 #96 ; 0x60 #96 ; 0x60
#56 ; 0x38 #56 ; 0x38 #56 ; 0x38
#104 ; 0x68 #104 ; 0x68 #104 ; 0x68
#44 ; 0x2c #44 ; 0x2c #44 ; 0x2c
#100 ; 0x64 #100 ; 0x64 #100 ; 0x64
#48 ; 0x30 #48 ; 0x30 #48 ; 0x30
#108 ; 0x6c #108 ; 0x6c #108 ; 0x6c


Extra changes for Dzoom

*0x6274 = arg1
*0x627C = arg2
*0x6284 = arg3

FF1C8628: e5a50018 str r0, [r5, #24]!
FF1C862C: e5a56154 str r6, [r5, #340]! ; 0x154
FF1C8630: e5a57008 str r7, [r5, #8]!
FF1C8634: e5858008 str r8, [r5, #8]



There are also 2 AVC formats which appear:
CanonAVC0006
CanonAVC0005

Anyone know what they are?

Also set in: NSTUB(0xFF04B9E0, AJ_MOVW_SetAvcInfo)

AJ_guess_sortof_a_copy__FROM.R1__TO.R0__LEN.R2(2684 + aAJ_0x1EE0_MOVWptr_struct_0x00_to_0x04_taskName.off_0x4 /*0x1E38*/, arg0, 0xc)

*
When an encode is requested in HD mode, size is hard set

RequestH264Encode(arg0, 1920, 1080, arg1)

What happens if you change this?

More stream parameters in:
NSTUB(0xFF1111F4, AJ_guess_set_movieMode_n_pstream_stuff)

It is only called twice... can this be changed via code page hacking?

AJ_guess_set_movieMode_n_pstream_stuff+476:   RequestH264Encode(arg0, 1920, 1080, arg1)
str:lvcdevStartH264_pInputAddress_pStreamAddress1_+268:   RequestH264Encode(arg1->off_0x4, 1920, 1080, *(arg1))
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Andy600 on August 28, 2012, 03:19:53 AM
Just been reading LPowell's post about his Flow Motion 2.2 100Mbps for the GH2 which uses GOP3. http://www.personal-view.com/talks/discussion/3337/gh2-flow-motion-v2-100mbps-fast-action-performance-reliability-for-all-class-10-sd-cards/p1

This seems to be the optimal setting for that camera. Maybe a good starting point for testing this build? I'm still learning this so I'm not sure exactly what to set things to. So far I have got it set to GOP3 -16Qscale but bitrate viewer shows it's not a constant bit rate and I cant see any I-Frames.

BTW does anyone know an app similar to Stream Parser that will give us more detailed info? Bitrate Viewer is a bit basic.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Audionut on August 28, 2012, 05:10:26 AM
Quote from: Andy600 on August 28, 2012, 03:19:53 AM
BTW does anyone know an app similar to Stream Parser that will give us more detailed info? Bitrate Viewer is a bit basic.

AVInaptic
http://www.videohelp.com/tools/avinaptic (http://www.videohelp.com/tools/avinaptic)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 28, 2012, 06:19:32 AM
 I think you have to keep qscale locking off if you want to use the CBR function as intended. The parameters in VBR change the same but the encoder is quality minded vs BR minded (qscale is locked, BR will fluctuate based on data and usually evens out). Coincidentally, this is another thing that can be tested in terms of quality.

I frames show up for me with BR viewer.. gop3 = 8 I iframes /sec gop 4 = 6/sec ... for 24p

Quote
Flow Motion v2 boosts the AVCHD encoder's peak bitrate to 100Mbps in all video modes and uses a short 3-frame GOP in all 1080p video modes (and a 6-frame GOP in 720p modes
These improvements enable Flow Motion v2 to capture highly detailed, fast moving images with far more fidelity than the stock firmware's restricted bandwidth can accommodate.

We're already doing this... but what is happening to quality? I can do like 140mbps in gop1, average over 120.


High ISO - Crop Mode - Light w/ Rain
Not sure if CBR/VBR

Gop1
http://upload.g3gg0.de/pub_files/d4c92e3c7850a189c1c192a86ace2f77/Lamp_Gop_1.MOV

Gop3
http://upload.g3gg0.de/pub_files/8e06487dd60083d1e42bd73197ec441c/Lamp_Gop_3.MOV

Gop6

http://upload.g3gg0.de/pub_files/1507715cede6b8b58d476acb7c2bc3ec/Lamp_Gop_6.MOV

Gop12
http://upload.g3gg0.de/pub_files/6186c340ceffd9fd7b4b6b940c14a0f5/Lamp_Gop_12.MOV
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Chucho on August 28, 2012, 07:02:44 AM
http://www.jongbel.com/?page_id=19 is trail software that give detail information about the NAL units includes slices I,P,B and there offsets and all the seq_parameter that are set including profile_idc and level_idc.
Correct me if I'm wrong but how I understand it is that the SPS "sequence parameter set" sets the Profile_idc and the Level_idc in the NAL units. Is this only use for decoding and not encoding? [JPCORE] H264_SPS_PPS may lead to some good hints. Also NSTUB(0xFF1C90B0, H264E_OperateNalUnit_fGetSpsPps) may also be a good place to look for some hints.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 28, 2012, 07:57:34 AM
AJ_H264E_OperateNalUnit_fGetSpsPps is called by RequestH264Encode and sets encode parameters. I didn't know SPS/PPS were profile before.

Where is?
H264_SPS_PPS

Found in:

NSTUB(0xFF1E3310, AJ_JPCORE_SetEncodeH264Parameter)

That analaysis program is good!

Level is 50, profile is 66... I swear I saw that somewhere in the H264 parameters but now I know what to look for.

qp minus is -6 in videos, that is either our QPC/QPy or from deblocking filter (which didn't seem to come off, need to make sure it doesn't get reset per frame.)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Leon on August 28, 2012, 05:22:18 PM
Right.  So if you use "CBR" but "lock" the quantiser at -16, will you not get exactly the same output as using Q-scale with quantiser at -16?  (And still set GOP, etc.) 

I think they should be exactly the same thing, and it would then make sense to call it a modified Q-scale rather than CBR.  (Unless I'm missing something?)  If this works with audio it would be really cool (not that it isn't already really cool ;)).

What would also be cool, in the future, is to have an option where you can set Q-scale and the firmware will automatically vary the I/P ratio to ensure the buffer doesn't overrun.  This would allow more efficient compression, than say using GOP 3 all the time, but still allow use of low quantisers, and not worrying about setting the GOP manually depending on the speed of each of your cards.  (Even if you have to leave a large margin of error, such as trying not to go above 20% full.)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Andy600 on August 28, 2012, 05:27:09 PM
Quote from: Audionut on August 28, 2012, 05:10:26 AM
AVInaptic
http://www.videohelp.com/tools/avinaptic (http://www.videohelp.com/tools/avinaptic)

Thanks :)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Andy600 on August 28, 2012, 06:14:39 PM
What does the averaging setting do? I get buffer overloads when its on. I'm also noticing that I can get higher bit rates by starting on a less detailed scene like the sky then tilting to the thing I want to shoot. If I hit rec on the detailed scene the buffer usually overloads immediately.

Is there any way we can have selectable config files for fast switching between settings on camera similar to what Miyake has done with selectable binarys?

It would be useful for testing GOP settings and other things. Remembering what I set for each test shot is difficult after ding a few. At the moment I'm trying GOP3 with CBR at about 1.5 which is working out at about 50-60Mbps. Should I increase the P factor to be greater than I?

BTW 1% I'm not seeing any I-frames in Bitrate viewer with my settings. Is that good?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 28, 2012, 07:00:08 PM
Quote
If I hit rec on the detailed scene the buffer usually overloads immediately.

Yes, I noticed this myself... if you jam the buffer out immediately the recording will stop... if you gradually bring it up, it may setttle. Sometimes I point at dark scene right before testing something I'm not sure will work to let the MVR get started.

You're probably in GOP mode in BR viewer. Hit CTRL-F to get into frame mode.


As for encoder profiles its a very good idea once we squeeze everything out of this. I and P and D and Gop are SIZES not ratios. So increase P if you want bigger P frames, I if you want bigger I frames, etc. Gop sizes I'm not sure yet, one is calculated.

Averaging takes the average # for every mode and scales that, usually you end up with bigger numbers. I have to set up debugging to on/off before encoder profiles and maybe we can figure out how the sizes are calculated and what they mean.

One thing I just noticed... CBR (qscale between -6 and -8, BR ~90-107mbps) VBR( qscale -16, BR 100-140mbps and less buffer overruns)... same scene.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Andy600 on August 28, 2012, 08:30:08 PM
Thanks 1%. Good to know I'm on the right track with my testing. It's interesting about CBR & VBR on the same scene though my class 10 card isn't letting me hit the upper limits so i need to pick up something better. I can hit about 120Mbps before it stops.

Thanks for the heads-up on BRV, you're right, I was in GOP mode  ???

When you say about I, P D & GOP being sizes is it dimensions or bitrate size?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 28, 2012, 10:04:57 PM
I think size is frame size or something like that.

Just found there are 2 ways to stop the encoder... bitrate too high and complexity too high. I can go along and film not too complex scene at Q-16 and then buffer will instantly fill (from 14%) when I hit a bunch of green trees waving. More control over max requestable bit rate might help with buffer stops, as would figuring out what actually overloads the encoder.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Andy600 on August 28, 2012, 10:50:47 PM
Yes I found the same thing.

Have you noticed that at default settings GOP is dynamically set. Moving onto high detail drops GOP to 2 or even 1 sometimes. I filmed something earlier that produced a bunch of I-frames only for a second or 2 then they started to spread out.

Ignore that. Just checked again and it was P-frames colored the same in AVInaptic
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Leon on August 28, 2012, 11:02:35 PM
Unless I'm misunderstanding you, I think your "2 ways to stop the encoder" are the same thing:

If you encode at a certain quantiser (Q-scale value) then it uses as much bitrate as it needs to maintain a constant "quality" (amount of information loss) within that macroblock (16x16 part of the picture).  If not much is happening then it does not need much bitrate so you don't fill the buffer.  As soon as you present it with a complex scene it suddenly requires much more bitrate to maintain the quality.  Increased bitrate is caused my increased complexity.  Complex patterns, motion and noise (including high ISO) all contribute to increasing complexity.

I also wanted to clarify something about I/P ratio:  In x264, the I/P ratio is how much it increases the *quantiser value* (not bitrate) for I-frames compared to P-frames.

In x264, the quantiser runs from 0 (lossless encoding) to 51 (very poor quality), with 18-20 being sensible for DVD encoding and about 22 being used for 1080p BluRay encoding.  (Of course you would normally encode using a Constant Rate Factor (CRF) rather than a Constant Quantiser Parameter (CQP), which Canon DSLRs almost certainly use.)  Changing the CQP value by 1 usually affects filesize by about ~9%.  Therefore using CQP 17 is about 1.09 * 1.09 * 1.09 times bigger (about 1.3x bigger) than using CQP 20.  Canon is of course using its own scale (no surprise, lol) and the increments of Q-scale may be quite different.

Did you work out what D1 and D2 are?  Maybe they are values for the inter/intra luma quantisation deadzone?  If so, then lowering the threshold would likely improve fine detail considerably but increase bitrate.

See:  http://mewiki.project357.com/wiki/X264_Settings#deadzone-inter.2Fintra

In any case, I would think that all these parameters probably relate directly to parameters that x264 can take.  For example, GOP size is called "keyint" in x264.  Specifically, I wonder what your I and P parameters could be - you can't set the size of I & P frames (only the quantiser ratio).  Setting the GOP tells you when your I-frames are, so all your other frames must be P-frames (since there are no B-frames).

See this page for all the x264 options: 

http://mewiki.project357.com/wiki/X264_Settings
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 28, 2012, 11:17:55 PM
It could be the same stop but seems like the buffer fills instantly and BR before wasn't that high. Like 90 to 110 then suddenly 100% full... so either it requested something crazy like 250-300 or crashed or both.

D values I'm not sure yet... if you lower them the BR+quality drops, raise them and it raises to new heights. Both are also set the same for some reason. Next build is going to have these values on screen (setting) so maybe we'll make sense of it all.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Andy600 on August 28, 2012, 11:46:41 PM
I'm thinking we need some kind of repeatable test, something always moving but can be filmed on a locked off tripod. Water from a tap?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 28, 2012, 11:59:32 PM
I did lamp in the rain... but its not raining now :( or should that be :)

Now you can view I/P/D, etc with debug on in 24p/30P

https://bitbucket.org/OtherOnePercent/tragic-lantern/downloads/autoexec.bin.600D.Exfat.Debug *visible while recording*
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Andy600 on August 29, 2012, 01:03:57 AM
I ran lots of tests but I cant see any difference other than bitrate and GOP increase/decrease. The settings were definitely taking effect but no difference in the end results...so far  :-\
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Leon on August 29, 2012, 01:10:44 AM
I use TV static to test reliability of different CBR values with different SD cards, with the camera on a tripod.  It's very repeatable and I figure it's about as complex as a scene can get.  Analogue TV is no longer broadcast in my area, so I can switch my TV to analogue tuner and select any station and I just get that snowy erratic static.  Horribly difficult to encode  :)

My set-up is:

• Set TV to detuned analogue static
• Mount camera on tripod ensuring it is perpendicular to centre of TV and at a distance where the static fills the whole frame.  (I always use the same lens, the 50mm f/1.8 for consistency.)
• Set ISO high (I've been using 3200 but 6400 might be better)
• Set aperture to something like f/8 to f/11 to ensure the picture will be crisp with as much detail as possible (but ensuring exposure is correct)
• Perhaps controversially, I set shutter speed higher than usual (1/100) because I think this should capture even more detail (less temporal motion blur) and be harder to encode
• Set PictureStyle (eg Neutral 0,-4,-2,0) and ensure HTP is off.  Probably should turn off ALO too but I haven't bothered.
• I keep audio on because I'll almost always want it on, even just as an emergency backup or for synching audio from my external recorder.  (Mostly I just have my external recorder plugged in and the quality's good enough.)
• I've been testing at 30 fps, on the basis that if it can do that it will be fine with 25 or 24, and 30 fps is sometimes useful to slow down action a bit or reduce the effect of camera shake.
• Lastly, but importantly, I ensure the focus is very accurate, by using live-view Contrast Detection AF rather than "fast" phase detection AF.

From this (with the December 2011 firmware) I established that my "Class 6" Verbatim card could only manage CBR 0.8x reliably.  My UDMA 1 SanDisk manages CBR 1.4x (with audio) and my Class 10 Transcend is variable, managing 1.1x sometimes and 1.3x other times depending when it was formatted.  Of course setting a lower GOP will probably improve these.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 29, 2012, 02:06:50 AM
Here is a 200mps clip: http://upload.g3gg0.de/pub_files/fc7fd4c62859b2178c6acb0f4af1ad0f/200Mbps_MVI_2425.MOV


When BR is high and you look at videos from the same series which were lower there is a difference, its just hard to quantify and consistently test. Static sounds good for testing stability, but how is it for comparing detail preservation, etc.

Also, gop size affects how motion is displayed.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: deleted.account.0 on August 29, 2012, 04:51:52 AM
Would this tool help?

MSU Video Quality Measurement Tool (VQMT) is a program for objective video quality assessment. It provides functionality for both full-reference (two videos are examined) and single-reference (one video is analyzed) comparisons.

http://compression.ru/video/quality_measure/video_measurement_tool_en.html
There's even a free version available.

If the camera was locked off on a Tripod & the same scene captured at various bit rates, this may work very well.



Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Audionut on August 29, 2012, 06:32:18 AM
Quote from: simonrfenton on August 29, 2012, 04:51:52 AM
Would this tool help?

Unfortunately no.  It uses metrics (PSNR, SSIM etc etc) to determine quality.  Good in theory, but you would never be able to record exact scenes.  Any slight deviation would effect the results.

Also, you need a recording to base off.  There is no "perfect" copy to base the experiments off.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Audionut on August 29, 2012, 06:37:48 AM
Quote from: 1% on August 29, 2012, 02:06:50 AM
Here is a 200mps clip:

How about finding a static scene with tons of detail to test also.  Something like what I did in the OP.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 3pointedit on August 29, 2012, 08:10:47 AM
Could you use a recording of a water fall, in close up? Very noisy temporaly, and if shot off a screen competely repeatable. You would have to start with a pretty hi res clip first though.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Rush on August 29, 2012, 02:23:09 PM
Easiest way to shoot static - close lens cap + set extreme ISO (around 100000)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 29, 2012, 06:39:36 PM
QuoteHow about finding a static scene with tons of detail to test also

Probably need 1 test for detail and one test for motion and 1 test for sustainability.

Quoteasiest way to shoot static - close lens cap + set extreme ISO (around 100000)

600D has no 100k, I think 1250-2500 is probably as high as anyone will go normally. 1250 and below for most cases otherwise you get noise. Also encoder will just be encoding junk... I can record really high MB of black/underexposed but that doesn't translate into real performance in the field. Recording will just stop when scene is correctly exposed and there is movement.

Water and trees are good candidates because of how erratic/detailed they are.

QuoteSetting the GOP tells you when your I-frames are, so all your other frames must be P-frames (since there are no B-frames).

Yep, with the analyzer its either I or P.

QuoteSpecifically, I wonder what your I and P parameters could be - you can't set the size of I & P frames

Lenght of nal unit for I and P? The numbers are pretty big and in that range.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Rush on August 29, 2012, 08:43:41 PM
Quote from: 1% on August 29, 2012, 06:39:36 PM
600D has no 100k, I think 1250-2500 is probably as high as anyone will go normally. 1250 and below for most cases otherwise you get noise. Also encoder will just be encoding junk... I can record really high MB of black/underexposed but that doesn't translate into real performance in the field. Recording will just stop when scene is correctly exposed and there is movement.

There is no difference between any movement and noise. Noise is real stress-test, because amount of noise have great impact on required bitrate.
600D can set ISO up to 632000 (higher value decrease brightness for me). Just open ISO submenu, set analog ISO to 3200 and increase "ML digital ISO".
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 29, 2012, 09:34:45 PM
You're right, I tried it just now and it lets you set gain up for an iso of 200k. Haven't tried it in a while. A fully exposed max ISO will help compute highest BR for given settings.

At 1250 in CBR
D value comparisson:
http://upload.g3gg0.de/pub_files/743c0ef6b157e52ef4e753cb244ea9f4/one.tga

.1x
http://upload.g3gg0.de/pub_files/81210227647f962e48af7dbdf06e9276/point1.tga

.5x
http://upload.g3gg0.de/pub_files/a4ba6a53dc63f95f7afe1efac0bbadb0/point5.tga

2x
http://upload.g3gg0.de/pub_files/575df7479153ac68143ca0fdee7d024b/two.tga

Any difference?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: nanomad on August 29, 2012, 10:10:25 PM
Not really
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: deleted.account.0 on August 30, 2012, 02:31:30 PM
1% - Any chance of another unified build? I'd like to do some testing on a 550D.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: deleted.account.0 on August 30, 2012, 02:38:09 PM
Quote from: Audionut on August 29, 2012, 06:32:18 AM
Unfortunately no.  It uses metrics (PSNR, SSIM etc etc) to determine quality.  Good in theory, but you would never be able to record exact scenes.  Any slight deviation would effect the results.

Also, you need a recording to base off.  There is no "perfect" copy to base the experiments off.

Is there a way to load in a RAW file off SDCARD into RAM, pass this through the encoder at various compression settings, then measure the resulting difference between these clips with the MSU tool? This would give a repeatable starting point.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Rush on August 30, 2012, 02:49:12 PM
Quote from: simonrfenton on August 30, 2012, 02:38:09 PM
Is there a way to load in a RAW file off SDCARD into RAM, pass this through the encoder at various compression settings, then measure the resulting difference between these clips with the MSU tool? This would give a repeatable starting point.
For real, you'll see no difference with still scene - with bitrate >20mbps it will be perfect
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 30, 2012, 05:05:15 PM
Actually it does seem possible but not sure how to implement it, would it be jpeg or yuv. I'd guess gop would be set to 2 and then you get one I frame and one P frame.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Chucho on August 30, 2012, 08:37:18 PM
Quote from: simonrfenton on August 30, 2012, 02:38:09 PM
Is there a way to load in a RAW file off SDCARD into RAM, pass this through the encoder at various compression settings, then measure the resulting difference between these clips with the MSU tool? This would give a repeatable starting point.
If you look at the deleted files in your SD card you will notice a MVIxxxx.DAT file for every MVIxxxx.MOV file. The camera first writes this .DAT file before it adds the headers, tags and atoms, it then renames it to .MOV and delete the .DAT file. This code is somewhere at the end of the main firmware look for the .DAT and .MOV strings. This is the only time I've seen that the camera loads a file of the SD card for a movie fuction. I've never been able to recover the .DAT file from the SD card but it's always the same size as the .MOV file so it's after compression. I have been able to produce a MVIxxxx.DAT file by changing the values of a register C0F1xxxx something I can't remember the exact address. I'm  not sure if it's the same .DAT file as the compress .DAT. It was one of the Initialize H264 encode registers.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: deleted.account.0 on August 30, 2012, 11:54:19 PM
My logic is, as we can now do "low frame rate" h.264 movies, (like 1 FPS), then we don't need to work in Real Time.
So yes, we have say 25-30 YUV images on the SDCARD, then have a script to load them into VRAM, one by one.

I would imagine writing this 1 sec test .MOV may take a number of minutes to complete.

The test YUV files needn't be actual real life footage, it could be 1 sec of rendered 3D animation, with lots of detail to test the encoder performance.

Food for thought...
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Chucho on August 31, 2012, 12:25:15 AM
I don't think you can feed YUV footage to the Canon encoder, as I understand it it goes like this,
JPEG path -> H264 encoder (NAL units, ect) MVIxxxx.DAT -> MOV wrapper (headers, tags, atoms) MVIxxxx.MOV.
You have to rewrite a lot of tasks just for a simple test.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 31, 2012, 12:41:14 AM
I dunno... looking at all this, do you think the bottleneck is at the encoder or the jpeg data being fed to it? We can record ridiculously high BR and yet visible quality is only a little different.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: deleted.account.0 on August 31, 2012, 01:43:57 AM
Perhaps the JPEG compressor needs to be adjusted before we'll see improved quality further down the chain...?

How about bypassing the middle man altogether & writing MJPEG .MOV files direct to SD Card? (like the older Canon Powershots used to do).
At low compression ratios, 3:1, the JPEG encoder still gives very good results, supports 4:2:2 colour space and is easier to edit - not to mention bypassing the double compression.

Uncompressed 8-bit YUV 1920x1080@25 FPS is around 103 MB/sec, MJPEG at 7:1 would give about 15MB/sec or 120 mbit/sec.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Leon on August 31, 2012, 01:59:25 AM
4:2:2 video would certainly be nice!

What about getting silent picture video to work properly?  Those .422 files would presumably make a good video?  The main problem when I tried it months ago was that the .422 files often are a mixture of frames (eg top third is one frame, bottom two thirds the next), and there isn't audio.  Perhaps they could somehow be dumped into a .MOV (or any other file), along with audio, even if you had to use a special program on the PC afterwards to convert it into something useable...

I'd also be interested in a 550D-compatible build.  Perhaps with sharpness -1 added in too?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 31, 2012, 02:14:17 AM
Better still... dump buffer that is fed to the JPeg encoder and bypass all of this. Its major work though, more than I know how to do. Although since we can hijack functions now its technically possible.

QuoteUncompressed 8-bit YUV 1920x1080@25 FPS is around 103 MB/sec, MJPEG at 7:1 would give about 15MB/sec or 120 mbit/s

2nd one my card could definitely do... but there is no MJPEG codec, it would have to be made from scratch.

Quote'd also be interested in a 550D-compatible build.

Before that happens need the stubs used for some of the functions in bitrate.c... I only have 600D firmware. I'd also need addresses for where to jump the invalid gop error. 550D also a bit weaker on processing power and no 64bit file size so no files over 4gb. 600D seems like a transition camera to t4i/5dIII.


QuoteJPEG path

Nanomad tried using different paths on T3, can the path be switched so compressor is getting larger files than 1920x1080? I know it can resize because of zoom mode, etc. With less frames in the buffer I think it can handle bigger ones. Canon was really stuck on that long gop encoding.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: ass5001 on August 31, 2012, 03:36:59 AM
Sandisk Extreme Pro 95MB/s
yes it's the best choice for 2x.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: deleted.account.0 on August 31, 2012, 05:22:03 AM
Quote from: 1% on August 31, 2012, 02:14:17 AM
Better still... dump buffer that is fed to the JPeg encoder and bypass all of this. Its major work though, more than I know how to do. Although since we can hijack functions now its technically possible.

2nd one my card could definitely do... but there is no MJPEG codec, it would have to be made from scratch.

How about just a series of Jpeg images written to SDCard? (avoiding the need to create an MJPEG codec from scratch)
Perhaps a new Folder is created for each "Record", it's a simple matter to convert image sequences to .MOV files - plus would also get around the 4GB file size limit of FAT on older camera's.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 31, 2012, 06:37:30 AM
Or raw stream of what comes out before its converted into H.264. The dat file is just the uncontainerized H264 stream.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Audionut on August 31, 2012, 06:53:39 AM
Bypassing the jpeg engine and all it's processing would be the way to go.

I don't see how it would be possible to capture the raw data.  You would never be able to record the data fast enough.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Chucho on August 31, 2012, 07:16:47 AM
In a earlier post I wrote about a register that dumped a MVIxxxx.DAT file. I found the register again it's C0F113C4 it's a Boolean register. Start recording after a couple of seconds, 10 or 20 change the the 1 value and it dumps a MVIxxxx.DAT to the memory card. I opened the .DAT file in a hex editor and searched for SPS 0x0000000167, PPS 0x0000000168, I-frames 0x0000000165 and P-frames 0x0000000161, I didn't find any.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Chucho on August 31, 2012, 08:30:30 AM
Or maybe it just stop recording before it deletes the DAT file from the card
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: g3gg0 on August 31, 2012, 09:37:23 AM
hmm 1%, you uploaded a MVI_2369.MOV, remember?
it has one I every 4 frames, QP is about 10
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: deleted.account on August 31, 2012, 02:32:15 PM
I'm not a dev but interested in the flow path and state of the data at entry into the h264 encoder.

Is the .422 data (silent pics) considered the state of the feed for the h264 encoder? I've never bothered with them are they really compressed or just subsampled to 4:2:2?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 31, 2012, 05:21:07 PM
Does QP ever go above ~10? I've been looking and it seems to go lower but not higher.

QuoteI opened the .DAT file in a hex editor and searched for SPS 0x0000000167, PPS 0x0000000168, I-frames 0x0000000165 and P-frames 0x0000000161, I didn't find any.

The one time I got a dat file was when camera crashed while recording and it didn't get deleted. I could still play it in VLC it was just a raw TS file. I think it happened when we first went over the 4GB.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: ilguercio on August 31, 2012, 05:25:45 PM
Let me get this straight, none of the things tried so far delivered a better image quality?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 31, 2012, 05:54:20 PM
Better than what you'd get without it. But its not enough. You can compress the same input as nicely as possible but you still have the same input.

I cap out at ~250-260Mbps with the static... qp slice is still -10. I'll try to up it, I think its at 0x6ce0 right by deblocking filter. It goes like 0 to 1xx (at -16 qscale). We'll see what happens.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Chucho on August 31, 2012, 06:08:16 PM
Quote from: 1% on August 31, 2012, 05:21:07 PM
Does QP ever go above ~10? I've been looking and it seems to go lower but not higher.

The one time I got a dat file was when camera crashed while recording and it didn't get deleted. I could still play it in VLC it was just a raw TS file. I think it happened when we first went over the 4GB.

TS like in MPEG2 file? I'm going to try to run a DAT file in a MPEG-2 analyser to see if I can get any information.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: ilguercio on August 31, 2012, 06:20:35 PM
I'm sticking with CBR 1.0x as mov files are already huge and i don't want to burden my old laptop with excess weight of my files.
It looks like the solution isn't so easy and it's a shame given the fact that we could have so much better footage from our cameras.
Thanks for the effort, by the way :)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 31, 2012, 07:20:15 PM
QuoteTS like in MPEG2 file? I'm going to try to run a DAT file in a MPEG-2 analyser to see if I can get any information.

No like transport stream... mpeg2 ts is different, h.264 ts (or is it es?) is different. It is raw h.264 without the . mov container.

QuoteI'm sticking with CBR 1.0x

CBR 1.0x is actually compressing your footage. There is a difference between 40 and ~90 MBps. You give up too soon, as we're starting to figure out the encoder. That CBR just modifies the data rate parameters, you're actually getting Q+ (which i seen as high as Q+24) values to maintain. Watch what it does.

I recorded static @ 5000 iso and it fails right away at q-16... requests BR over 250. Need to find a way to keep quantizer but hard cap bit rate so it doesn't overrun hw buffer in encoder. We also haven't even started to try to increase frame size.

I recorded a few videos with QP parameter modified... there is crashing and all that but I got a few that overran the buffer right away and hopefully have increased qp slice values.

*Update...

I was having trouble so I set qscale to -17 and slice QP goes down to expected values and MVR records. I had to disable all assertions but I'm sure I can find the place where its checked and just NOP it.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: ilguercio on August 31, 2012, 08:44:15 PM
So what should i do then, use 2.7x (it's my maximum setting with my 50D) for some reason?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on August 31, 2012, 09:10:46 PM
2x or setting qscale to something - is probably good.

Set slice to fixed value. CBR function can change qscale but it no longer alters the slice (0x6ce0) while recording. Not sure if its showing up in the files yet.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Leon on September 01, 2012, 01:08:37 AM
Are the .422 files uncompressed 4:2:2 YUV (which is 16 bits per pixel, I think)?  It's just that they should be 4.15MB each if that's the case...

Quote from: simonrfenton on August 31, 2012, 01:43:57 AM
Uncompressed 8-bit YUV 1920x1080@25 FPS is around 103 MB/sec, MJPEG at 7:1 would give about 15MB/sec or 120 mbit/sec.

For 4:2:0 chroma subsampling, there are 12 bits per pixel.  That would give a bitrate of 1920*1080*25*12 = 622 Mbps, or 78 MB/s.

JPEG compression ratios are based on size before chroma subsampling, so for 15MB/s you'd need a compression ratio of 10:1, which would be quite feasible, and of course much less compression could be used.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 01, 2012, 04:36:06 AM
Need to take a silent pic and check.

Here are today's results:

600D Slice locked to max(qp=10)/min(qp=2)? for CBR

https://bitbucket.org/OtherOnePercent/tragic-lantern/downloads/AE_SliceMin.bin

https://bitbucket.org/OtherOnePercent/tragic-lantern/downloads/AE_SliceMax.bin

*Set slice from menu, VBR fixed, sound with audio off and high bitrate. Deblocking filter works for ALL frames:

https://bitbucket.org/OtherOnePercent/tragic-lantern/downloads/autoexec.bin.600D.Fixes
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 01, 2012, 10:50:44 PM
This is how profile is set:

*0xC0E1000C = 41091

Other mode is for something else...

What is what?


   if arg2 == 0 /*EQ*/:
                *0xC0E1000C = 41104
                DryosDebugMsg(0x15, 0x2, '[JPCORE] JP62_OPMR3 %#lx', 0xa090) => ret_DryosDebugMsg_FF1E35E4
            if arg2 != 0 /*NE*/:
                *0xC0E1000C = 41091
                DryosDebugMsg(0x15, 0x2, '[JPCORE] H264_SPS_PPS JP62_OPMR3 %#lx', 0xa083) => ret_DryosDebugMsg_FF1E35E4



Value of 0xA083 is 786366719 or 0x2EDF00FF
0xA090 is 0.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: jgharding on September 02, 2012, 11:45:57 AM
A bit of film maker feedback here :)

I've been using CBR2.8 mode for promo shoots, it's great, at high ISO settings in real-world situation I can remove much more noise in post, as said noise not so compressed mushed into the image. Thanks, it's amazing to use so far.

With regard to GOP, I and most of those I know would always choose an I-frame option if available. There does seem to be a subtle aesthetic difference between long GOP codecs and those that compress individual frames. Also from the point of view of buffer over-runs, it sounds quite practical to simply set GOP to one (I-frame) a d then pump the bitrate to around the peak speed of the camera writer. Card space is no longer the issue really, it's just getting the best quality we can, and it seems that's on the way now! :)

If you can make a unified for this I'll poke around a bit, though I'm no coder I have quite an intuition for such things ;)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: jgharding on September 02, 2012, 11:51:21 AM
Also, I'm interested to see if other Profile levels can be used. I see that Canons are using the Baseline profile in H264, I wonder if the two levels above this, Main and High, are available to the EOS hardware encoder?

If so, the video quality for a given bit rate should increase remarkably, as should the power required to decode the video in real time. While the latter is negligible in production terms, the former could be quite wonderful, if indeed alternative profiles are both accessible and effective.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Leon on September 02, 2012, 02:44:00 PM
@1% : Why are you changing settings for slices?  Slices, AFAIK, are only used to help decoding on multi-thread decoders, which is qhy BluRay asks for 4 slices...  Increasing the number of slices decreases the quality, which is why the default is 0 in x264...

See http://forum.doom9.org/showthread.php?t=148826 (regarding BluRay compatibility):

With "Slices" they mean that each frame should consist of at least 4 slices (parts), that can be decoded/processed independently.

If a stream consists of several slices, this allows (but doesn't enforce) slice-level multi-threading.

So if an encoder relies on slice-level multi-threading, but the stream doesn't consist of several slices, that encoder will fail to decode the stream (at an acceptable speed).

I guess they included the "4 slices" requirement in the BD specs, because they wanted to make sure that even such decoders will be able to decode the BD streams.

Anyway, all the state-of-the-art H.264 decoders (ffmpeg-MT, CoreAVC, DivX H.264, etc.) use frame-level multi-treading and hence work with single-slice streams just fine.

It appears that also the hardware decoders used in "stand-alone" BD players are capable of decoding Level 4.1 H.264 streams without multiple slices.

Also it was found that slices reduce compressibility. In case of x264 the loss was around -0.1 PSNR for 4 slices (link). And: The more slices/threads, the bigger the loss!

Frame-based multi-threading not only gives more speed-up, also the loss in quality/compressibility is very small - even at high number of threads.


Or is it something else you're calling a "slice"?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Leon on September 02, 2012, 02:47:09 PM
Has anyone worked out why including audio, which is only 1.5 Mbps, and is completely uncompressed, has such a massive effect on the maximum video bitrate?  It seems really strange.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: jgharding on September 02, 2012, 02:55:05 PM
Perhaps it takes some processor grunt to thread the two together in the MOV, so disabling it altogether frees up a lot of cycles for video encoding. just a guess though ;)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 02, 2012, 05:07:48 PM
QuoteIncreasing the number of slices decreases the quality, which is why the default is 0 in x264...


Not the same thing. Jpcore slice qp is basically the quality of the input frame. All VBR/CBR does is alter this parameter... ie at Qscale +16 sliceqpd is high number  (low quality, low BR). At -16 its 112 which is the highest official quality. Camera goes down to 86/87 in theory which is like qscale -25 or something.

In the file itself slice qp delta per frame stays at -10 after 112... but BR increases from 112-87 so in theory you're getting higher than q-16.

I have a theory too that encoder just alters sliceqpd and line skip (not sure how) for a given frame. That's why at high quality with lots of complexity it kills the buffer and dies... that input frame gets bigger and bigger and it gets harder to encode it. In crop mode BR gets much higher.

All CBR does is compute slice quality on the fly from those I/P/Gopt sizes. When you lock the slice, qscale and all of those parameters don't do anything. The other Jpcore registers aren't touched by many functions and there is only one path which shows JPcore OPMR3 as SPS/PPS. Another register is frame size and another is from deblocking filter.  Not much left over.

Also I have an idea. 600D released in 2011? TMP19A43CDXBG released when, and is it the controller for SD Card? What is the highest bus speed it supports? Toshiba is one of the developers of high speed SD technology and made some of the first UHS-1 cards ... in 2009/10. Quite possible it could support higher speeds just canon didn't implement them, there were all of 4 cards on the market. What is the new chip used in the T4i? Does 5dMk3 support UHS-1, I think it has dual slot?

We have functions like:
NSTUB(0xFF3BACE8, SDCOM_ChangeBusClock)
but no list of available clocks.

Camera first does read speed test and sets parameters in 0xCD7C (aAJ_0x14320_SDIOTSK_QuickReadData. I think there are 2 options based on how the read speed goes but all of that can now be modified. The bandwith of the camera itself can go much higher than now and if SD controller supports another mode it will probably happily write as fast as the SD card can go, 1/2 the bottleneck is there.

Anyone with 5d3 look? Or know what the Toshiba chip supports or how SD driver section of fw works? Already canon had planned for 64bit file addressing, not used till T4i. If we can do something here maybe we can record at slice 87 all day.

Thoughts?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Rush on September 02, 2012, 08:12:52 PM
QuoteDoes 5dMk3 support UHS-1, I think it has dual slot?
Sadly, 5DmkIII' controller don't support UHS-I rates (that's why it is reccommended to use much faster CF). 650D is the first Canon DSLR with UHS-I support.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 02, 2012, 08:44:58 PM
So we have to look at t4i firmware/HW when available. Right now hopefully we are using 48mhz bus instead of the 24mhz bus. All I see is Mode 3 in the logs but I'm not sure if that means 48mhz or just high speed mode. I only have class 10 cards.


Also, while qscale can only be changed by 1, I wonder how much slice can be changed on the fly.  This could lead to a better CBR function that drops quality when bit rate goes past certain point. Another thing to consider.

Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: a1ex on September 03, 2012, 08:16:58 AM
QuoteIs the .422 data (silent pics) considered the state of the feed for the h264 encoder?

Yes, this is the input for the H264 encoder. You can alter some data in those buffers and it will show up in the recorded video.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: deleted.account on September 03, 2012, 03:42:25 PM
Cheers for the reply. :-)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 03, 2012, 06:08:05 PM
Size of Simple 5x5 .422 is ~3mb. If we get it down to 1.0 mb or less we could record it fairly well. Maybe could be put through jpeg path but I don't know if any of them keep the 4:2:2. At 3mb would need like 480 megabits. My card does about 19MB write.

Even written as 1 frame of jpeg, could be made as image sequence in AE and then wav sound added. If there really is intermediate jpeg step already it would be even easier, just stop it from being passed to the H264E/Movie writer. I bet overhead goes down too.

I still think some buffer overruns are because compressor stop compressing. Buffer start filling from 1%-5% to suddenly 40, 60, 75, stop. Like new frames stop being written out. How can it do that at <90mbps when it records steadily at 120-130+ for minutes at a time with quality constant.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Leon on September 04, 2012, 05:41:19 PM
QuoteSize of Simple 5x5 .422 is ~3mb

What is a "Simple 5x5"?

Has anyone seen any way of preventing the frames being upscaled from 1728x972 to 1080p.  I reckon that's not helping either.  It should also help with using micro-moire removal filters in post, etc.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 04, 2012, 05:50:37 PM
Simple 5x5 is a simple silent pic 5x5. Slice quality may only come into play when the encoder starts.


Also, quite hilarious:

http://www.learn.usa.canon.com/resources/articles/2012/ipp_ipb_all_i_compare.shtml


So 600D is 5dMK2.5? I guess this means all I is the best for quality (horrible for file size), especially with a fixed qp slice. Also explains why buffer is written immediately, there is no additional pass or prediction being done.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: unity2k on September 04, 2012, 06:44:37 PM
Don't miss the second page, that's where you'll find the punchline!

To you guys who are trying to increase our bit rates, while I cannot make a contribution besides the financial, there are those of us out here wishing you all the best of luck in finding the Magic Bullet for Magic Lantern that might allow us to film with higher quality images as a result of your hard work. Thanks for your contributions.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 04, 2012, 06:52:02 PM
Going to find/fix that 30 minute limit sooner or later. Then its 5d2.6

Also here is the .422 format from a1ex before it gets lost in the other thread:

http://www.fourcc.org/yuv.php#UYVY
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Leon on September 04, 2012, 07:01:37 PM
@1% : I think the "5x5" part only applies to Hi-Res.  If you select "Simple" it seems to completely ignore whether it is 5x5, 2x1, etc.  I just had a wee play with the options, and an 18MP Hi-Res 5x5 comes out at 34MB, so it does seem to be uncompressed with 4:2:2 subsampling – I got something muddled up before and thought the full-res pics were much smaller.

Can burst not be done at higher quality and faster?  With a UHS-1 Class 10 card, I only get 1.6 fps at only 1056x720 resolution.  If the H.264 encoder is getting 1080p (or are they 972p at this point?) .422, can these not be written directly to the card?  What's limiting?

So if the .422 files aren't compressed, and they are the "input" to the H.264 encoder, what is this jpcore slice qp setting that controls "input" quality, and why does it not seem to exist for x264?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 04, 2012, 08:01:53 PM
QuoteWhat's limiting?

The zooming.

I'm not sure if h.264 is fed from the LV buffer which is only 720x480? or from full size yuv buffer. You'd have ML graphics on the video, would you not? The simple pics come out worse quality than the encoded frames so something must be missing. If I'm not mistaken, the high-res ones clear the overlays too.

I think there are several steps to compression as there are multiple passes and mention of jpeg encoding being finished. So it could go YUV -> jpeg path -> h264E. You're not going to alter the quality of the raw YUV, just the compressor input.

The YUV has to be compressed with SOMETHING before we put it to the card. The simple frames are 3 MB. Card writes at ~20MB... raw would give you 6FPS only. The buffer is 1GB big and if not much is being used in ALL-I, the frames must be much smaller. I don't think the camera could take raw YUV, compress it with H264 and run several passes (for P frames) then write it to the card at over 30fps without some intermediate step.

x264 has slice QP parameter but it will not go below -10 on videos from the camera. But bit rate does rise if lower jpcore slice like 90/95 is used vs 100/110 and recording stops sooner filming the same thing. Q-scale -16 is only 112
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Andy600 on September 04, 2012, 09:16:27 PM
1% - random question - do you think frame size can be increased at a lower bit rate?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Leon on September 04, 2012, 09:19:36 PM
In burst mode, there isn't any zooming though, is there?  It's only managing to write 2MB per second in silent frames to my camera when I use burst mode.

Seems so strange that it would waste processing encoding to jpeg and then decoding it to send it to the H264 encoder, unless it is somehow forced by the way their hardware is wired up and communicates.  If the jpcore part is already a bit compressed, maybe that can be written directly to the card bypassing x264.  We only need a 7:1 compression overall (or 4:1 compared to 4:2:2) to get 1080p down to being 20MB/s.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 04, 2012, 10:08:36 PM
Quotedo you think frame size can be increased at a lower bit rate

I'm hoping so. Especially if we can capture the input and skip h.264 all together. There is already resizing for digital zoom and 720p/640, etc. The camera sees the full frame either way.

Quoteforced by the way their hardware is wired up and communicates

Probably this. Depends if its easier to compress a jpeg to h.264 realtime or raw YUV. It already has to encode the full frame (w/ line skip) so it would be grabbing 30MB of YUV per frame and compressing it x 30FPS + resizing + passes for P frames. That is 900MB/s. What about 60fps... 720P not that much smaller. Buffer would be full all the time. We can already see they couldn't just dump .422 to card because of speed limitations.

Someone with more knowledge of assembly+ HW encoding can correct me but this is what I've gleaned so far.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 08, 2012, 06:26:56 PM
Found new thing about CBR mode. If you're using ALL-I CBR will not adjust qscale/slice quality. This means that next qscale value is calculated from P frames not I frames. Can anyone confirm? Set slice 0, reboot, turn on debugging and see if slice quality changes in all I. Even in gop2 qscale changes, gop 1 is a default of 120, I think thats -8.


*Edit:

More good news... slice QP can be adjusted by as much as you want. For now I set it to drop by 4 when buffer gets to warning level and no errors + no buffer overruns  but quality drops down quickly so I need to find a good mathematical solution either by buffer or bitrate.

Something quick:

https://bitbucket.org/OtherOnePercent/tragic-lantern/downloads/autoexec.bin.quickcbr

If buffer warning drop slice q by 2, if BR < 75mbps raise by 1. Its working at ISO5k in crop mode but its too dark right now for me to test + fine tune. Mainly doing this for ALL-I ... I'll try in longer gops next.


In daylight it still works, ramp down is good, ramp up is a little too fast I think. 1 second polling for BR and buffer is a little too slow, would like .5 seconds. Bit rate is a little bit jaggy (follows card write) but better quantizer used overall than CBR function or VBR function and mostly stops overruns unless they happen really suddenly or scene is already too big to encode. I set buffer warning to like 50/60% and had success at ISO 80-2500.

Low points of BR are usually  above default rate of 40. Footage is overall 1GB per minute. A 7minute continues shot was ~7GB. Average BR is over 120, peaks 160-200. The drop downs aren't really noticeable in watching, I have to see if they show up in post but I doubt they will as BR at worst quantizer is still usually very high.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 09, 2012, 07:57:25 PM
V2, quicker polling (1/2 sec). Instant BR is only 1/2 of actual but now average stays around 100 in gop3 at 50% buffer warning.

https://bitbucket.org/OtherOnePercent/tragic-lantern/downloads/autoexec.bin.600D.Exfat.QuickPollingAutoSlice
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Andy600 on September 10, 2012, 07:24:36 PM
Quote from: 1% on September 09, 2012, 07:57:25 PM
V2, quicker polling (1/2 sec). Instant BR is only 1/2 of actual but now average stays around 100 in gop3 at 50% buffer warning.

https://bitbucket.org/OtherOnePercent/tragic-lantern/downloads/autoexec.bin.600D.Exfat.QuickPollingAutoSlice

This is looking promising 1% :)

I shot some trees, grass and parked cars for a few mins using ALL-I. Bit rate was around 75-130mbps (q was mainly around 10 but hit 22 according to AVInaptic) with buffer warning set on 50% (that came on a few times but kept recording fine).

I can't quite put my finger on it but I think there is an improvement in over all detail. There is a very slight increase in noise at low ISO's using ALL-I but it's very fine grain and actually looks quite nice to my eyes. I'm testing it with a new LOG PS that I'm developing and it's looking great.

I need to shoot some fast moving stuff but it's getting dark here. Might try some night shots.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 10, 2012, 07:54:39 PM
QuoteThere is a very slight increase in noise at low ISO's

Probably from deblocking filter being off?

All I pulls higher BR and writes constantly. Slice pull back only happens on buffer warning. With faster polling it ramps up right away as soon as BR falls. If you look at peaks there are some drops but I can't tell for the few frames it does it for.

Gop 3 compresses some frames so BR is more jerky (like I frame is 120 and P frame compressed to 85) but buffer overrun less likely. Q22 I think is when quality dropped but some Q22 better than Q16 or Q17 all the way in CBR.


I got streameye stuidio now so that helps to see what is going on. Seems like this is the best we can get from default encoder. Wish kunie and jason_atl would test because they have some really clear primes.

Moving car footage and buffer stopping will tell too.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Andy600 on September 10, 2012, 08:09:39 PM
I set the deblocking filter settings to -2 & -1 after reading an article about best quality but can't remember where I read it. It looks better than off for sure and the noise is similar to the grain I've seen on the C300. It's not ugly.

I can't notice any serious dip in quality when q drops but it only dropped when I panned so as you say, some moving car shots should highlight any issues.

If this is the best we can hope for I think it's not too bad at all. I need to get another card and load up the default ML so I can A/B the same shots but I'm sure there is an improvement :) (there must be if I can notice it LOL)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: FilmMan on September 10, 2012, 08:32:43 PM
Andy600 and 1%,

I'd be willing to test.  I have a 5D2 and some top prime lenses.  I have the current ML installed.  Upping the bitrate (with current build), there is a slight quality difference especially with aiding in color grading.  I usually only attain 48 and peak out maybe if I'm lucky at 70 or so.

I'm not sure if your current testing files will work the the 5D2.  Achieving 100 mbps and with more stability has definitely caught my interest.  Let me know.  I'd like to help if possible.   I appreciate your efforts.  Cheers.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Andy600 on September 10, 2012, 08:55:09 PM
Quote from: FilmMan on September 10, 2012, 08:32:43 PM
Andy600 and 1%,

I'd be willing to test.  I have a 5D2 and some top prime lenses.  I have the current ML installed.  Upping the bitrate (with current build), there is a slight quality difference especially with aiding in color grading.  I usually only attain 48 and peak out maybe if I'm lucky at 70 or so.

I'm not sure if your current testing files will work the the 5D2.  Achieving 100 mbps and with more stability has definitely caught my interest.  Let me know.  I'd like to help if possible.   I appreciate your efforts.  Cheers.

I think this is only for the 600d at the moment  :-\
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: nanomad on September 10, 2012, 08:56:43 PM
Well it should work on almost every camera as long as the correct stubs are identified
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: JasonATL on September 10, 2012, 09:14:15 PM
Quote from: Andy600 on September 10, 2012, 07:24:36 PM
This is looking promising 1% :)

I shot some trees, grass and parked cars for a few mins using ALL-I. Bit rate was around 75-130mbps (q was mainly around 10 but hit 22 according to AVInaptic) with buffer warning set on 50% (that came on a few times but kept recording fine).

I can't quite put my finger on it but I think there is an improvement in over all detail. There is a very slight increase in noise at low ISO's using ALL-I but it's very fine grain and actually looks quite nice to my eyes. I'm testing it with a new LOG PS that I'm developing and it's looking great.

I need to shoot some fast moving stuff but it's getting dark here. Might try some night shots.

I agree. This looks very promising. I could definitely see an improvement in just the plain 'ol 2.3x CBR, so I'm anxious to see what these new improvements can do. I loaded up on my 600D last night. Since I was shooting only indoor shots of nothing interesting, I couldn't say much about the improvement in quality, but the bitrate was clearly higher than what I've been able to achieve with plain 'ol CBR. It did look better than anything I've seen from my 600D, in my limited shooting.

Some observations (I'll post more as I continue testing):
1. I have a 5D Mark III and the "grain" that Andy600 describes is reminiscent of what ALL-I looks like compared with IPB on the 5D Mark III. It appears that there is more "noise" with ALL-I, but this "noise" is different than the ISO noise.
2. It isn't clear to me that ALL-I is better theoretically. My understanding is that the benefit is in ALL-I being easier to decode (and encode?) because there is no interframe dependence. At the same bitrate, it is my understanding that IPB could be better since it is having only to account for the changes from the I-frame, whereas ALL-I uses the bits available to recompress all the information in each frame. Thus, it seems to me that a there is possibly an optimum for resulting picture quality (perhaps scene-dependent) in which we can maximize the bitrate and I-frame tradeoff by setting the GOP low enough to not overrun the buffer, but high enough to make use of the relatively more effective compression from IPB. This will be the focus of my tests. I could be wrong, so it is worth testing.

Thank you 1% and others for your skills and insights in doing this. I haven't coded anything in nearly a decade, so I'm trying to contribute where possible here.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 3pointedit on September 11, 2012, 03:52:06 AM
Talk of grain change and bit depth seem contrary to the real issue, that of diminshed resolution and moire. A decent white balance should get the images close to what is needed in post, removing color correction and need for more bit depth (chroma keying excluded). and temporal compression often eliminates some gain noise anyway (free noise reduction) but at the expense of resolution?
I always thought that genuine improvement (to older cameras) would be sought by increasing/improving line structure (overcoming line skipping), is that a likely hood?

Great work BTW!
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 11, 2012, 04:09:18 AM
Quotebecause there is no interframe dependence

Yep, I frame dropped from buffer almost immediately. P frame sticks around for a while while its doing more passes, etc.

The P frames are only a little bit more compressed and really the main reason to use them is file size/buffer. 1GB a minute is hardcore no matter how you do it. Also, no B frames yet... Gop3 is where its closer to old file sizes and doesn't write almost all the time. I think the I frame has to be like 10-15% bigger for the same quality so instead of a decent 130mbps you end up at 160-170mbps and you actually do crash if the encoder slows down or the card does. Should test gop of 2,3-6... any more than that and too much is held in the buffer.

Qscale drops like a stone when the buffer warning comes on but it ramps up right away with the faster polling. You can turn on debugging and watch it. Qscale with canon functions only moved by one. With slice you can go from 128 to 87 instantly. I think canon was really going for consumers and smaller files rather than quality. They want you to buy a more expensive camera rather than using a 600D, etc. The weird thing is that they don't like offering swappable lenses on their camcorder offerings. Heavy grading will really show if the bit rate dips matter or not. If a few frames get all funky out of a whole sequence then we'll know.


Other cameras:

50D = Did not change gop.
550D = no breaking 4gb limit so you'll have all of 4 minutes of footage.
60D = untested for both limit and gop.
T3 = can probably pull it off
5d3 = different encoder, different hacks
5d2 = don't have one and nobody has really looked in on it. Encoder also slightly different. CF cards should in theory do more than 20MB/s write. A1ex has one so he'd be the guy to try it. But still have to hack the 4gb limit and 5d2 is the same generation as 550D so maybe only 32bit value for file size :(

What's left?

MJPEG might end up giving better results and actual full frame video, none of this 1600x960 crap. Although maybe native recording and resizing later might give better results for regular H.264 than in camera resized 1920x1080.  I can continually take shots at S1 which is like 2592x1728 so 2k video seems possible. Max file size per frame would be like .83MB to get 24p out of that.  .67 for 30P... not a whole lot of room. SD interface max speed at 48mhz is 25MB but no card seems to want to keep up.  How good can you get at that data rate?

Line skip is likely so it can fit the entire frame into reduced resolution without resizing the whole image and taking up CPU time, etc. It IS controllable since 600D has 1:1 crop and even zooming. But the issue will be that the frames are bigger and more complex to encode.. hence static and trees are very evil.

Right now, encoder in the hand is worth 2 in the theoretical stage.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: FilmMan on September 11, 2012, 05:09:09 AM
1%,

What you have accomplished so far is absolutely AMAZING!!!  If you could convince Alex trying his expertise on the 5D2, many, including myself would be very appreciative.  Like I mentioned prior, I'd be willing to do test video shots for you.  So if you get Alex on board for the 5D2 and he needs a "tester", I'd be willing to give it a go on my camera. 

If you would achieve 2K video, that would be unbelieveable - it would ignite the video community in such a positive way. 

By the way, I have been posting your thoughts on another board.  It has generated a very positive fan base towards you.  As things develop, this could lead to more donations potentially for the Magic Lantern community.

Cheers 
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 11, 2012, 05:55:12 AM
Thanks! Wow. I think a1ex was working on actually getting mjpeg out which would beat the H.264. There are strings with AVI in the factory functions so maybe camera supports a few containers too.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: FilmMan on September 11, 2012, 06:45:38 AM
Hopefully either the H.264 tweaking or the mjpeg method works.  Maybe both will work.  People are very supportive and so excited about your progress.   The possibility of 4:2:2 color space with 2K video (or better 1080p than what Canon is providing) is making me lose sleep.  Cheers again.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Kabuto1138 on September 11, 2012, 06:58:05 AM
I think I just crapped my pants, 2k??!!!!  BTW, I am here thanks to FilmMan.  Like he said, he put all this in another board.

For the record, all of this is Greek to me.  I use the 2.3 and love it.... but that's about where all my technical knowledge, all this Gop etc. is language from another planet.

But I can gather that we are ALL here for the main reason, that CANON has given us the short change so we have to buy at least a $13,000 dollar camera, to get a good quality file.  At that price the c300 is just 1080p Apple Pro Res 444

The 5d mark II and Mark III has the biggest sensor around(bigger than RED etc.) But we have to live with that crappy h264 BS.

I have a 5d Mark II and a Lexar professional 1000x(I heard is the fastest card around?), let me know when and how can I help, once Alex is on this.... just explain how to do it.... like you would a 5 year old  :P

1%  This is incredible.  Cheers.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Andy600 on September 11, 2012, 10:32:45 AM
The noise/grain is definitely caused by the deblocking filter. I looked more closely at what I shot yesterday and there is slight blockiness in certain parts of the shots using ALL-I (deblocking filter set -2/-1). It's actually given me a much better understanding of what it does and how it affects appearance.

I shot a little test today between ALL-I and GOP 3 with the deblock filter at 0/0. There's not much in it quality-wise but GOP3 does help slightly with detail, especially grass.

I've upload a couple of seconds of ALL-I and GOP 3 here http://wtrns.fr/GwDyNV-M4di4gHQ Can you see any difference? The grass (lower left of the frame) looks a bit better to me on the GOP3 shot. Both shots were using the 600d 1080p/24p on a Nikon 50mm 1.8d at 3x sensor crop using my own LOG PS with LUT and a little luma sharpening applied. More news on the PS soon ;)

I'm shooting at an airfield this afternoon which should be a much better test with subjects actually moving. Only got a 16gb card though  :-\

Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: deleted.account on September 11, 2012, 10:55:00 AM
Not directly related with regard to encoding but post deblocking h264 with a 'decent' deblocker + a little temporal denoising an give a 'better' grain appearance subjectively but a little loss in detail. Fine line between keeping the detail and losing sharp edges to blocks.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: jgharding on September 11, 2012, 11:40:12 AM
Thanks for the GOP3 and I-frame example. IMHO I-frame always has a more film-like cadence to movement, it just feels nicer. It should do though, since frames of celluloid don't predict one another ;) Even if it's a little blockier, I feel it's better to have no prediction smear between frames.

Is there anything I can test with 550D? I'd like to push it with some low light and human subjects... i mean victims! 
mwooohahahaha!
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: jgharding on September 11, 2012, 12:10:27 PM
Also, noise that is harder edged is easier to remove in the denoising part of post-production. If you smear codec and ISO noise noise into the image at encoding, the result is actually a less-pleasing image once the production is complete, since denoising is a mandatory part of most post -production workflows, whether it be Neat Video, DaVinci Resolve, Denoiser II etc...

In other words, if all-I has slightly sharper noise, it will have two benefits for film workflow: the film-like motion cadence, and sharp noise that can more easily be detected and removed than noise which is blurred into an image at the encoder stage.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: nanomad on September 11, 2012, 12:23:34 PM
You guys are having too much fun so the 1100d is joining the party later today. :P
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: jgharding on September 11, 2012, 12:26:31 PM
Dunno if this article on deblocking is useful to any of the smart lads here, but I found it interesting: http://x264dev.multimedia.cx/archives/443
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: JasonATL on September 11, 2012, 02:18:27 PM
I wasn't happy with any of my tests from yesterday. The wind wasn't blowing, so I had nothing moving to give me much of an idea of image quality. Light was also bad when I finally got outside. I've ordered a faster card to see what the card limitations are. Will try out more tomorrow, but probably won't have time to do serious tests until the weekend.

Once again, thanks for the incredibe progress.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: nanomad on September 11, 2012, 02:57:30 PM
Man, porting this stuff to the 1100D was easy, altough I had to make your code a little better :P

Dunno if I missed any hardcoded constats though, so I didn't dare testing it yet  :-X

Pull request: https://bitbucket.org/OtherOnePercent/tragic-lantern/pull-request/1/hack-improvements-1100d-support
(review it and tell me if it's safe to test)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: JasonATL on September 11, 2012, 03:55:51 PM
If anyone ports these features to the 550D/T2i, I have one of those in addition to my 600D and am willing to test on that, too.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 11, 2012, 04:33:27 PM
Looks safe to test. I think -6, -6 is de-blocking filter off. 0, 0 is just default.

The gop had only 2 locations. One would give an assert after your movie was done. The other is for the player so you can try to play these back in camera.

For 2k and the like, we have to see what we're actually getting. for 500D LV stream was 928x616, for 550D its 1056x704. If thats uprezzed to 1920x1080 you can do 2k easily but it won't be very real.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: nanomad on September 11, 2012, 05:40:48 PM
If you accept my pull request you'll make thing easier for everyone 1% :P

Oh, and exFAT works beautifully on the 1100D ::)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 11, 2012, 06:20:12 PM
Done and done, compiling it.

oops, didn't update.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Andy600 on September 11, 2012, 06:58:42 PM
Just back from shooting my girlfriend (haha) doing a spot of microlight flying. Shot with GOP3 and ALL-I and I can tell even looking at the ungraded footage that it's really good :) I also shot as a test for my new PS. I'll try to grade and edit it tonight and I'll point out what shots were GOP or ALL-I when I upload it. I didn't have any issues with buffer overloads. The only issue was the first shot of her actual take-off because I wasn't set up properly so the WB and exposure isn't correct but I'll try to fix that or she'll kill me  ::)

TBH I'm more and more excited by the possibilities. 1% has done a great job! This isn't the first time I've shot at the airfield but it's certainly the best looking footage I've captured.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: FilmMan on September 11, 2012, 11:40:55 PM
Andy600,

I did a quick grade of your 3 second gop3 clip.  I posted here.  https://vimeo.com/49267057
If possible could you upload and put up a link to the uncompressed version for the gop3?  I want to test and see how far the colors can be pushed. :) 

Cheers
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: JasonATL on September 12, 2012, 04:25:36 AM
Using 1% latest build...

Here are some captures from my NLE of some test shots from today. It wasn't windy, so most of these shots are from near static video.

I shot with three settings all at CBR 3.0x
GOP = 1 (ALL-I, right?)
GOP = 2
ML v2.3

I'm purposefully not telling which panel is which until below. So, if you want to guess, don't read lower than the link until after you have looked at the pictures.

https://picasaweb.google.com/102645356447404039003/ML_GOP_BR?authuser=0&authkey=Gv1sRgCIuEydWZ1ZXPsAE&feat=directlink




The "adjusted" are graded with sharpness (quite a bit, actually) and contrast added, plus a 1 pixel color blur added ahead of the sharpness (which serves to really cut down on the aliasing on these particular shots). You'll see in the ones with "bad" in the title that the middle one (GOP=1) had bitrate dropped (presumably when buffer was filling?) during the clip. In VLC, the bitrate here drops to about 30Mb/s. The left pane is the ML v2.3 and the right pane is GOP=2. Frankly, I don't see a huge difference. In the zoom shots, it appears that the GOP=2 might have a slight detail advantage over GOP=1. In all cases, I prefer the subtle improved look of GOP=1 or GOP=2 than v2.3, except of course for the few frames (very visible when playing it) in which the bit rate drops. The fact that I can sharpen these so much without them looking like crap really shows how much the increased bitrate helps.

Stupidly, I didn't think to try CBR 1.0x. I already know that CBR 3.0x (or 2.3x) is an improvement over Canon FW. So, I'm really trying to see what we have going here with these new improvements. Too early to conclude much, except that they are very promising.

Using a Sandisk Extreme (45 Mb/s) card, my bitrates according to Bitrate viewer were:
GOP=1 (avg ~160 Mb/s, peak around 179 varied from 75 Mb/s to 179 Mb/s throughout)
GOP=2 (avg ~149 Mb/s, peak around 152)
ML v2.3 (avg ~76 Mb/s, peak around 94)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 12, 2012, 05:33:35 AM
For me gop 3 seemed to perform a little better than 2 at 24p. How visible are the drops when playing?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Andy600 on September 12, 2012, 06:23:06 AM
I've only tried GOP3 and ALL-I so far. ALL-I seems better for moving images while a GOP3 setting has better detail for static shots (I think that was to be expected anyway). I've uploaded a bunch of frame grabs from the footage I shot yesterday both ungraded and with a basic LUT applied to my LOG PS (I'm doing the real grade today). These shots were from shooting ALL-I.  http://wtrns.fr/LYPZI3ca2c1fkAj

FilmMan - That 3 second clip you have was part of a huge file. I'm not sure how to trim it to keep the bit rate etc identical and I'm on a pretty slow internet connection. I will upload some untouched files that I shot yesterday once I've finished the grade and edit for my LOG PS demo movie :)

Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: xaled on September 12, 2012, 11:13:33 AM
Hi 1%,

thank you for your time and efford!

there is the reference h.264 codec implementation
http://iphome.hhi.de/suehring/tml/ (http://iphome.hhi.de/suehring/tml/) with a good manual http://iphome.hhi.de/suehring/tml/JM%20Reference%20Software%20Manual%20%28JVT-AE010%29.pdf (http://iphome.hhi.de/suehring/tml/JM%20Reference%20Software%20Manual%20%28JVT-AE010%29.pdf), concerning the codec parameters. It might be helpful in understanding the H.264 parameters and finding them in the canon implementation.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Rush on September 12, 2012, 11:18:13 AM
ALL-I shouldn't be considered as best possible and best looking. Longer GOP with same bitrate should look better, because of more efficient compression mode.
Seems like 1% have found optimal GOP=3 with "highest quality + no buffer overruns + less bitrate"

GH2 have a lot of bitrate-hacks and the latest GH2 hack (Apocalypse Now) sets GOP to 3 or 6 for best quality+bitrate, so we should consider it as the best possible too...

But will 600D handle high bitrate with GOP=6 or should we stick to GOP=3?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: deleted.account on September 12, 2012, 11:52:54 AM
Andy600, a quick look at your LOG PS stills, they look good, apart from no sky detail, but assume that was an exposure choice. Looking forward to hearing more about your PS and seeing some more.

What NLE and LUT format?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Andy600 on September 12, 2012, 01:31:30 PM
Quote from: y3llow on September 12, 2012, 11:52:54 AM
Andy600, a quick look at your LOG PS stills, they look good, apart from no sky detail, but assume that was an exposure choice. Looking forward to hearing more about your PS and seeing some more.

What NLE and LUT format?

Thanks :) The shot with no sky detail was facing the sun with no ND on the lens and not really any cloud detail to see anyway. Not the easiest thing to shoot and not really a good example of the PS. There is actually more highlight detail recovered with this LOG PS than Cinestyle in my tests and without the ugly artifacts of something like Cineplus Lightform PS. I made the basic LUT using LutBuddy in After Effects CS6 and saved it as a cube file but I'll be able to provide others. It's still a little too saturated but hey it's only a LUT.

Anyway, I'll keep this topic free of any more PS talk and let you know more info in another topic when there's something to see. It's just that I was testing it at the same time as 1%'s GOP build so made sense to mention it when I show test shots.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: deleted.account on September 12, 2012, 01:36:17 PM
Cheers, sounds good.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: JasonATL on September 12, 2012, 02:11:40 PM
Quote from: 1% on September 12, 2012, 05:33:35 AM
For me gop 3 seemed to perform a little better than 2 at 24p. How visible are the drops when playing?

Thanks. I also had GOP=3. I'll put it up on my timeline against 1 and 2 and take a look.

The drops are visible. To the casual observer, possibly not. But, of course, that's not who we're talking about here. I've posted a 5 second clip (even compressed at an average of 50 Mb/s) that you can download from Vimeo: http://vimeo.com/49305199. The drop happens right around 0:01 and you can see what I understand to be macroblocking. Would this change if I changed DblockA and DblockB?

In looking at one of my GOP=3 clips, I notice the same thing. Less noticeable in terms of the resolution loss, but clearly something "digital" happens at certain points.

Is this is a result of dropping the bitrate (and QScale) to keep the buffer from overflowing or is this something on the Canon side? I thought I understood the buffer to not come into play for GOP=1, so I wonder if it is something in the Canon algorithm. I shot a total of 30 clips and had a sheet on which I took notes of settings for each shot. But, I didn't take detailed notes of what I saw on the debug screen. I do recall seeing only a few clips in which changes were clear, but I didn't mark them. More testing this weekend...

Let me know if none of this is helpful. I'll continue testing for myself, but I don't want to muck up this thread with my observations if they aren't on point.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Andy600 on September 12, 2012, 02:26:03 PM
Jason ATL - If you change the deblocking filter to 0/0 the macro blocking should reduce. I'm experimenting with -1/-1 ATM which seems to give a slight increase in the perception of sharpness/detail and add a little fine grain. I think it's more of an aesthetic thing.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Andy600 on September 12, 2012, 02:33:22 PM
Jason - If you look at the .mov in Bitrate viewer you can see exactly where on the timeline the bitrate drops and then pixel peep the video
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: FilmMan on September 12, 2012, 03:45:06 PM
Quote from: Andy600 on September 12, 2012, 06:23:06 AM
I've only tried GOP3 and ALL-I so far. ALL-I seems better for moving images while a GOP3 setting has better detail for static shots (I think that was to be expected anyway). I've uploaded a bunch of frame grabs from the footage I shot yesterday both ungraded and with a basic LUT applied to my LOG PS (I'm doing the real grade today). These shots were from shooting ALL-I.  http://wtrns.fr/LYPZI3ca2c1fkAj

FilmMan - That 3 second clip you have was part of a huge file. I'm not sure how to trim it to keep the bit rate etc identical and I'm on a pretty slow internet connection. I will upload some untouched files that I shot yesterday once I've finished the grade and edit for my LOG PS demo movie :)

Andy600,
No problem.  Only post if time permits.  I know it takes alot of time in what your doing.  I appreaciate all your efforts and posts.  The plane shots are awesome by the way.  Cheers.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 12, 2012, 03:57:22 PM
If its blocking, deblocking filter would take care of it. I can't casually tell on my videos because I haven't found a subject where it's visible.

You can set gop to whatever you want up to 100 something. Your  buffer is the limiting factor here. I think after 3 or 6 its diminishing returns, the camera only handles a certain amount of predicted frames at a time.

The dropping function still needs some work... maybe it needs to drop less. Not sure... could make the bottom limit a setting. I was mainly experimenting with it to stop overflows. Without it camera couldn't handle filming static... with it,  holds up most of the time. If its a a big noticeable drop during grading/watching then there is a problem.

Wonder how well the 4:2:2 mjpeg holds up with the buffer. Wish a1ex would release it even if not done or perfect. Sounds like it will be a while since he is working on 5DMKIII pretty heavily.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: JasonATL on September 13, 2012, 01:45:41 AM
Quote from: Andy600 on September 12, 2012, 02:33:22 PM
Jason - If you look at the .mov in Bitrate viewer you can see exactly where on the timeline the bitrate drops and then pixel peep the video

Thanks, Andy600. I edited the post above to have Bitviewer stats and will use those from now on.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Andy600 on September 14, 2012, 12:58:49 PM
Shot some of this using GOP3 and ALL-I :) https://vimeo.com/groups/superneutral/videos/49436519 (Vimeo compression sucks)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: AriLG on September 14, 2012, 02:25:56 PM
Is the GOP thing available to 550D users, too ? or is it exclusively a 600D thing (for the time being) ?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: nanomad on September 14, 2012, 04:09:42 PM
It Will probably work on any camera but you will probably be stuck with a low video duration
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: jgharding on September 14, 2012, 04:33:49 PM
Quote from: nanomad on September 14, 2012, 04:09:42 PM
It Will probably work on any camera but you will probably be stuck with a low video duration

So will this be permanent, or do other bodies like 550D just need some specific code?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: nanomad on September 14, 2012, 04:51:40 PM
Best case: you need to find the same functions in your camera fw
Worst case: you'll need to start from scratch.
550d is probably in the middle
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: jgharding on September 14, 2012, 05:06:26 PM
Ah well, I wanted to get a 600D for the flip out screen anyway ;)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Andy600 on September 14, 2012, 05:11:06 PM
Quote from: jgharding on September 14, 2012, 05:06:26 PM
Ah well, I wanted to get a 600D for the flip out screen anyway ;)

Don't get it for the flipout screen. Get it for the sensor crop ability ;)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: AriLG on September 14, 2012, 06:24:06 PM
Quote from: Andy600 on September 14, 2012, 05:11:06 PM
Don't get it for the flipout screen. Get it for the sensor crop ability ;)
So, what you are saying is that if I shoot with my - say - Tokina 11-16, I'm effectively shooting REAL HD at 33-48 (with the X3 Crop) ?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: nanomad on September 14, 2012, 06:42:11 PM
Yes, and with no moire effect too  :o

Actually, I'd like a test on one little thing as it was a subject of discussion between me and Bart :P
The 600D is an APSC camera, so it has a "default" zoom of 1.6x
Crop mode adds 3.3x zoom.

So, let's say I've to a 200 EF lens, does this become 200x1.6x3.3 = 1000 mm in crop mode?  :o

It should be easy to test by comparing a zoom lens with a short fixed one:
1. EF 28mm 3x crop vs 150mm of the 18-200 EF-S non-crop
2.  60mm of the 18-200 EF-S 3x crop vs 200 mm of the same lens
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: jgharding on September 14, 2012, 06:52:44 PM
Hmm that's maths o'clock. I'd say 1056mm (relative to 135 full frame), as it's 200mm * (1.6*3.3)... give it a test!

It's a huge crop, good for super tele! I'd have thought the noise grains will be much larger. Given the deeper DOF aesthetic it could be useful for certain shots with that Tokina.

If you're used to thinking in Super 35 or APS-C anyway (as I am), its 660mm.

Everyone talks in terms of the 1.6 crop for APS-C, but that's only relevant if you think in terms of photographic full frame. Standard movie film has always been a different world (in general), it's only recently that Super 35 has been talked of as cropped! I think of 135 frame (5D etc.) as oversized...  :o

Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: nanomad on September 14, 2012, 07:12:27 PM
My google-fu just demonstrated that's it actually like that, you're getting a focal length of X*3.3*1.6 if you're using EF lenses
On the 550D you can even get  a *7.2*1.6 magnification if you shoot in VGA mode. Now, that's some kick-ass zoom :P

600D / 200mm / 3X crop


550D comparison footage
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: AriLG on September 14, 2012, 07:36:41 PM
Quote from: nanomad on September 14, 2012, 04:09:42 PM
It Will probably work on any camera but you will probably be stuck with a low video duration
How low ? I have seen the number 4 minutes somewhere...
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: nanomad on September 14, 2012, 07:39:39 PM
ALL-I should be approximately 1Gb per minute of video and you're limited by the 4GiB file limit
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: AriLG on September 14, 2012, 08:23:46 PM
Quote from: nanomad on September 14, 2012, 07:39:39 PM
ALL-I should be approximately 1Gb per minute of video and you're limited by the 4GiB file limit
OK...wow

We are starting to reach RAW numbers...  :)

So, another thing that makes the 600d worth buying ?...
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: FilmMan on September 14, 2012, 09:16:19 PM
Quote from: Andy600 on September 14, 2012, 12:58:49 PM
Shot some of this using GOP3 and ALL-I :) https://vimeo.com/groups/superneutral/videos/49436519 (Vimeo compression sucks)

Andy600,

Awesome.   By the way, it takes guts to fly like that.  Looks like alot of fun.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 15, 2012, 02:22:15 AM
600D in crop is also the closest to HD size. So when 4:2:2 comes it will be up there with 5dIII if it can write/process fast enough.

Other cameras have much smaller LV buffers and upress much more. So from current info 2k would be 2k from silent pic size. Right now 1080P is uprezzed from silent pic size too. Big disappointment, huh?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: FilmMan on September 15, 2012, 07:23:14 PM
1%,

1.  How stable is your bitrate program?  I inboxed Alex and if the program is stable he'd port them.  Myself and others, who have the 5D2, would love to be early adopters and test your bitrate program.  To be fair, donations would come too from myself, and others indicated they'd donate too.   

2.  Alex mentioned there needs to be alot of work for the mjpeg.  I'm hopeful but if it is uprezzed would the quality suffer?  Interesting stuff here with the 40d,  but only 1024x680. http://www.magiclantern.fm/forum/index.php?topic=1452.50

Everyone is hoping for your successes.  Lot's of new cameras coming on the scene, but these Canon cameras new and old, seem to have the horsepower but unleashing that power is the trick.  You've accomplished so much in such short order with your bitrate.  Cheers.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 15, 2012, 07:50:23 PM
Its stable enough. Only problem is the buffer saving function will drop quality really really quick.

I think MJPEG is worth it. You're already uprezzing the same thing, just losing extra color. A1ex hasn't released any code so we can't work on it without starting from scratch. Perhaps more donations for this are in order. There are a few devs at least willing to give it a go, just need a starting point. Even if you have to convert the movies before opening them its still a win. Audio is pretty much done and I don't think we can push the encoder much further.

Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: FilmMan on September 15, 2012, 08:53:59 PM
Quote from: 1% on September 15, 2012, 07:50:23 PM
Its stable enough. Only problem is the buffer saving function will drop quality really really quick.


Are you going to forward to Alex so he can port? 
(I'll donate just to try your bitrate program.  Also, I'll post on that other camera forum and ask people to donate so they can try.)

Quote from: 1% on September 15, 2012, 07:50:23 PM

I think MJPEG is worth it. You're already uprezzing the same thing, just losing extra color. A1ex hasn't released any code so we can't work on it without starting from scratch. Perhaps more donations for this are in order. There are a few devs at least willing to give it a go, just need a starting point. Even if you have to convert the movies before opening them its still a win. Audio is pretty much done and I don't think we can push the encoder much further.


I'm trying to understand here, so excuse my ignorance.  Do the cameras downsize to a specific video size then upsize thru the encorder and leave out color info?  So as Chucho stated, 5d2 is  1120x752. http://www.magiclantern.fm/forum/index.php?topic=2649.msg11595#new
For example, does the 5D2  insides downsample to that level for video (for 5D2: 1120 x 752) then uprezz to 1080p changing the 422 color space to 4:2:0 in the process?  Or am I totally out to lunch here?

Are you going to ask Alex for the code?  You were amazing on the speed with what you accomplished with the bitrate. 
Cheers

Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: JasonATL on September 15, 2012, 09:46:21 PM
1% - I have been shooting a lot of test footage, varying GOP and other parameters. I have a few observations and questions (more observations are sure to come).

I tried setting slice to 86. When recording during this, it resulted in a camera error that was a little scary. It said to pull the battery and then it tried to save the file (never did). It recovered fine, but didn't save the movie. I won't be trying slice=86 again.

As we have discussed in this thread, the buffer saving function drops the quality and this is visible to me in the footage. For a scene in which the buffer fills, then is saved, then fills again, etc., I describe this as a "breathing" or "pulsing" look, in which the footage has a regular pulse revealed when the save function kicks in. The frames during which the slice is lowered don't look so bad by themselves. The problem is that they look different and hence, there is this "pulsing" look to the footage that has regular changes.

It seems that there might be an advantage to the enhancing the buffer save function. As I understand it, it increases slice by 2 when buffer is above the warning and then drops it by 1 when bitrate < 75. The polling must be really quick on this, as the changes are quite quick. Perhaps the increment by which the slice is changed could be a function of distance of the buffer from its warning level? What I have in mind is something along the lines of:
slice = max(87, 87+x*[buffer - warning]), or
slice = max(87,87+x*sign(buffer-warning)*(buffer-warning)^2)
Perhaps x could be set so that the maximum slice value is achieved at 10% above the buffer warning? Is something like this possible? Do you think it might help?

An offset to warning could also be used to preventatively start raising slice as it nears the warning level. However, the user could do this themselves by setting a lower warning level, or if it was there might set buffer warning higher.

I know I say this in every post, but this really is great! Thanks again for your work on this. I am definitely getting the highest quality footage that I have ever seen with my 600D with this enhancement.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 15, 2012, 10:44:58 PM
Firefox ate my reply but the cliff notes are:


Mjpeg the way to go. Its a1ex's code so you have to convince him. He already sees all of this but has no 600D and isn't a big bit rate fan. You pretty much have it right tho. LV buffer is what it sees through the line skps and then it uprezzes that (process is called something in imaging, not new). Overhead will be saved from the uprezzing (probably why 720P has 60FPS). Only danger is that we put in a lot of work and its impossible hw wise, etc. Donate for that rather than polishing canon's turd of an H264 implementation.

Right now everything is 5DIII but there are a few devs willing to work on it, just we have no starting point besides the LV buffer address. A1ex some sort of implementation already and its for 5dII so probably more portable. I think he wanted to make something really good before releasing but other things took over. I think its really good already even if you can't play the videos back in camera or have to convert them before viewing, etc. All I saw is demo vids.

Other cameras are a lot of effort to port and I don't have the firmwares which take 2 days to decompile anyway. The other cameras may not have the same functions (50d gop didn't change) plus crippled by the 4gb limit which gives you 4 minutes (550D out already)... a little too short. Also don't have the cameras themselves on hand so I'd be coding blind and a simple mistake could be a headache. MJPEG I think is less camera specific.

---
I do see what you're saying with the buffer leveling function and its pretty crappy. I think miyake was going to look at this stuff and his better C skills probably help.

Quote

if (MVR_BUFFER_USAGE > (int)buffer_warning_level) // Drop Slice if buffer
     {   if ( (bitrate_mode == 1) && (qscale_slice >4) )
       qscale_slice -= 3;
   }
   if ( MVR_BUFFER_USAGE < ((int)buffer_warning_level-25) ) //Raise if BR getting low.
   { if ( (bitrate_mode == 1) && (qscale_slice < 45) && (measured_bitrate<75) && (measured_bitrate!=0) )
      qscale_slice++;
   }


That's basically all of it... if buffer gets a warning drop slice quickly until the warning goes away, if buffer is 25% under utilised raise quality. So quality drops too quickly with card write and then suddenly jumps up to max where it probably should be more conservative. Kind of a quality yo-yo. Breathing was a good analogy. This is just what I came up with in a short amount of time. There is also a whole nother scale for slice 0-79. Some are lower qualities but I think some are not. Its weird, it mirrors and wraps around in the same range (something-7F).  Could be an avenue for better control. H264 itself still doesn't want to write a quantizer better than 10 even if the input frame is bigger/better.  I mentioned 86 in the commit but it never made it here and got buried. It will just freeze your camera but.... why.. and also everything "works" like LV and camera just says busy if you try to stop recording.

Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Kabuto1138 on September 15, 2012, 11:22:27 PM
In ALEX and 1% we trust
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: miyake on September 16, 2012, 04:39:53 AM
QuoteI think miyake was going to look at this stuff and his better C skills probably help.

I want to help you. But I'm now learning a lot of your bitrate work ....
   @@@@  / ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄
   (゜д゜@ < catching up with you!
   ┳⊂ )   \____________
  [[[[|凵ノ⊃
   ◎U□◎ =3 キコキコキコ
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: JasonATL on September 16, 2012, 02:49:44 PM
1% - Thank you for the thoughtful reply.

Here's another idea of a possibly gentler decrease in qslice when buffer fills (in the same spirit as the suggested algorithm above in that it sets the slope of the qslice to depend on the severity of the buffer situation). You have probably thought of such an approach already, but I just want to encourage you not to give up!

Quoteif (MVR_BUFFER_USAGE > (int)buffer_warning_level) // Drop Slice if buffer
     {   if ( (bitrate_mode == 1) && (qscale_slice >4) )
       qscale_slice -= (int)((MVR_BUFFER_USAGE - (int)buffer_warning_level)/2 + 1); // slope depends on buffer vs. warning
   }
   if ( MVR_BUFFER_USAGE < ((int)buffer_warning_level-25) ) //Raise if BR getting low.
   { if ( (bitrate_mode == 1) && (qscale_slice < 45) && (measured_bitrate<75) && (measured_bitrate!=0) )
      qscale_slice++;
   }

The "/2"  can be /3 or /4 to make it even gentler. Given the results with -= 3 that you had before, I thought that /3 might work well, since, as written, if buffer warning level is 70% with /3 in the code, you would get -= 3 at 76% buffer, changing to -= 4 at 79% buffer. Again, user can set warning level to 60% if 70% doesn't work.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: nanomad on September 16, 2012, 03:59:32 PM
If someone can collect buffer % vs time data in different scenarios we can probably come up with a prettier function   ::)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: FilmMan on September 16, 2012, 05:00:00 PM
@1%,
Thanks for the reply.   

Here's some thoughts.

People are excited about the new GH3 having 72mbp/s All-I codec.  People wanted the quality for video. 

Now you have attained in the neighborhood of 100 mbp/s with stability (although the buffer quality drop exists, I think you and the team will figure it out in the next while).   Also, there is the 4GB limit for some cameras which gives about 4 minutes of record time.  I think for most people this won't be an issue.  I'd even be okay with 1 minute of record time. 

Once the bitrate is to your satisfaction, I can then present to others and mention to donate as this bitrate exists and the ML team wants to explore the possibility of mpeg 422?  Or I can keep talking about the current progress about the bitrate and the future mpeg possibility and encourage people to donate now?  This is very exciting stuff. 

The new Sony and Nikon have come with hdmi out in their new models.    However, an upped bitrate which you developed, may
add enough to the video quality to save people from buying new cameras?  Only time will tell.

To everyone working on this, hats off to you.  All of your efforts are very appreciated.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 16, 2012, 06:26:31 PM
QuoteIf someone can collect buffer % vs time data in different scenarios we can probably come up with a prettier function

This exactly. I've tried -2, -1, -4 and lower buffer percentages for the rise. Nothing has worked as well. /2 might work or speeding up the polling beyond 1/2 second. Instant BR can be multiplied back for accuracy. Of course this will vary with complexity/iso, etc. We can try static to test and the way it is now it can almost pull it off, much better than just qscale number and close to CBR. Also it might not need to drop quality down so far. Need to find a number where it can recover from an overfilling buffer but isn't pixelated so badly. Some more gradual leveling might be better.
Also might help to add the other #s and see what kind of qp they produce in the file.

Can try changing the gop on 550D (4mins better than 0 minutes), not sure what happened for a1ex on 60D (if it stopped writing or just gave the err70 that needs to be jumped). The other stuff is a bit harder and definitely needs fw to be looked at.

Quotepossibility of mpeg 422

Donations specifically for this purpose might encourage A1ex to release some info and that people are interested. i'm just in it for the video.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 16, 2012, 08:11:53 PM
Increased polling to 100ms. BR indicators are off, you're not getting 200-300 but you are getting a more stable BR.

I haven't pushed it because I'm not sure if its better or worse. Quality shifts aren't as extreme but also doesn't ramp up as quick. Haven't checked QPs in the files yet.

http://pastebin.com/QyCaEjek

https://bitbucket.org/OtherOnePercent/tragic-lantern/downloads/autoexec.bin.lessaggressive-faster

Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: JasonATL on September 16, 2012, 09:37:24 PM
Quote from: nanomad on September 16, 2012, 03:59:32 PM
If someone can collect buffer % vs time data in different scenarios we can probably come up with a prettier function   ::)

What did you have in mind? Here's a graph of time (x-axis), bitrate, and buffer.

https://picasaweb.google.com/lh/photo/VNwz0VaV1ZhKUV9YQEfYgBfNx8dP82C1K8bmBL14X3k?feat=directlink
(how do attach pics here?)

I shot this using TV static. However, I aimed the camera above the TV and slowly panned down to increase the amount of static in the shot. I then panned back up to decrease. I recorded my screen using a second camera and got bitrate data from Bitrate Viewer. You can see the buffer save function kick in.  Any suggested modifications of this experiment in order to collect more relevant data?

My settings are:
CBR 3.0x
DBlockA = -6
DBlockB = -6
PicPC = -7
GOP = 3
BuffWarnLevel = 60%
Averaging = OFF

1% - Let me know what other type of test or data I can help with.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: JasonATL on September 16, 2012, 10:11:11 PM
A couple of more runs with this type of test. As I said above, I used TV static to have a reliable source to cause a high bitrate. This time, I panned slowly to increase amount of TV static in the shot and paused for at least 8 seconds before increasing panning again to increase the static. Then, I looked at the bitrate graph of the resulting file.

It appears that (at least for 8 second intervals), a bit rate somewhere around 160-170 Mbps can be sustained by the buffer and my cards at GOP=3. I used a Lexar 600x SD and Sandisk Extreme (45 Mbps). Results were the same for both. On the Lexar, the buffer was at 25% during the 10 or so seconds of 160 Mbps bitrate. When bitrate went to 175 Mbps, the buffer increased steadily until it hit the warn level after about 7 seconds. This 7 seconds corresponds to my experience shooting in less controlled situations (outdoors). Using the Sandisk card, the buffer was at 30% during 8 seconds of 170 Mbps bitrate. The buffer warning and save function kicked in after 3 seconds when bitrate increased to 184 Mbps.

Graphs here:
Lexar: https://picasaweb.google.com/102645356447404039003/ML_GOP_BR?authkey=Gv1sRgCIuEydWZ1ZXPsAE#5788868294372408914
Sandisk: https://picasaweb.google.com/102645356447404039003/ML_GOP_BR?authkey=Gv1sRgCIuEydWZ1ZXPsAE#5788868296496242962

Do we know the size of the 600D available buffer? My guess would be around 200MB. If this is the case, then the save function could be written to limit the bitrate directly rather than wait on the buffer warning?

Bottom line is that, for me, it appears at the limit is around a sustained rate of 170 Mbps. It would be nice to allow short bursts a little higher, but perhaps brute force is necessary here to assure stability?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 16, 2012, 10:31:48 PM
Problem is with the 100ms polling, BR indicators are off by a little. Average bit rate also seems to fail after 4gb but we're using instant to adjust anyway. Need to fix it. CBR shouldn't make a difference as its neutered from changing quality. I set it to 2x and forgot it.  Check BR in the file.

I've stopped it from adjusting down to 21, Now it should stop at 19 DRF/qscale. Maybe less pixelation but maybe not fast enough to stop an overflow, have to test.  We're polling in 1/10s of a second now.

https://bitbucket.org/OtherOnePercent/tragic-lantern/downloads/autoexec.bin

Not sure how to graph it exactly. Wish I could keep it it under 150 real MBps as thats where my card write is consistent. Complex scenes also slow the encoder and lead to the buffer filling. Sometimes even the lowest quality can't stop it. But much better than old CBR set to 3x. I'm testing from 80-1250 ISO... this will affect it much, especially w/ static.

*seems better.

Had only 1 stop in crop mode at ISO 640 on a moving tree that would have eaten buffer before. Going to check BR/quality.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: JasonATL on September 17, 2012, 12:51:47 AM
Quote from: 1% on September 16, 2012, 10:31:48 PM
Not sure how to graph it exactly. Wish I could keep it it under 150 real MBps as thats where my card write is consistent. Complex scenes also slow the encoder and lead to the buffer filling. Sometimes even the lowest quality can't stop it. But much better than old CBR set to 3x. I'm testing from 80-1250 ISO... this will affect it much, especially w/ static.

1% - Another great improvement and the polling increase should help, too. Again, thank you for continuing to move this forward.

I think it isn't so much the complexity of the scene, but the combination of the complexity, the buffer state (how full it is), and the QS. I've done more testing. I think we're learning a lot now that can help.

In my latest test (and I've done the same experiment a few times to suggest that this is a representative result), again with the TV static and panning down to control how much, I got the following results, ending in a buffer overflow.
Bitrate viewer graph: https://picasaweb.google.com/lh/photo/bjPc-wCz4ooixLkAn3DbOBfNx8dP82C1K8bmBL14X3k?feat=directlink
You can see the video of the ML screen and debug here:


The complexity of the full screen of static that occurs at 0:17 in the recording (0:21 in the video of the screen) is not associated with an overflow because QS is already at a low number, so the bitrate only gets to about 150 there. A similar thing happens at 0:28. But, at the end, this scene (same complexity) comes at a time when QS is high, so bitrate skyrockets. In this case, QS does not start decreasing until it is too late, because at the high bitrate, the buffer is filling too fast.

The solution? The amount by which you lower qscale_slice should depend on both the buffer state and the bitrate. Why? Because the buffer will fill faster at the higher bitrate and, therefore, more drastic measures are necessary. I'm not even sure this will do it. However, it gives it more of a chance. What could also help is to use the bitrate and current buffer state to PREDICT the buffer state and therefore lower qscale_slice accordingly. It seems like this approach could reduce the chances even further of a buffer overflow.

And yes, I'm going back on my original suggestion on a gentler slope. I apologize. It was my hope that it would soften that appearance of the grain. But, I think the key is to make it only as steep or as gentle as necessary.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 17, 2012, 01:01:50 AM
Seems like that is what canon tried to do with CBR. Maybe can poll what it thinks qscale should be in non all-I modes. Also might help to lower quality faster when BR is already high and less/slower when BR is low. I'll have to figure out why its inaccurate first. Also maybe stop raising if BR is already at X amount regardless of room in the buffer.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: JasonATL on September 17, 2012, 01:29:17 AM
Quote from: 1% on September 17, 2012, 01:01:50 AM
Also maybe stop raising if BR is already at X amount regardless of room in the buffer.

I see what you are saying. If BR is, say, 120 Mbps, then the scene is obviously complex at the current QS, so why raise the QS? Makes sense to me. Again, it could even be better to combine it with the buffer level. Either or both would help with the "breathing" issue, I think. I would just want that level to be set suitably high.

Also, in regards to earlier statements regarding the utility of this whole exercise.
1. I don't think that the 4GB or 4 min limit will be of concern to most who would use this. My shots are rarely more than a couple of minutes long.
2. I think that, if this feature can be further refined (and it is darn near good enough for me already), that most will find this one of the best features of ML. People want clean HDMI out to be able to get 4:2:2 or increase bitrate. This satisfies the latter. And, seeing the results that I'm seeing, would really call into question the need for 4:2:2 for most people, I suspect.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 17, 2012, 01:51:33 AM
I do a bit of chroma key stuff and seeing the color on the demo vid was great. Also you gain some zoom out of it like in crop mode but lose the in camera uprezing. Also would kill 30 minute limit which nobody has been able to so far.

I tried some quantizers in the lower end and a lot of them freeze the camera. The ones that made videos had really low DRF like 52 or 48. So if we really wanted to put the brakes on video rate.... I should try 85-80 again and make sure there is no higher q in there as some of them froze at X5, X3, etc.

Probably the time try to port to other cameras will be when 600D is maxed out like with audio and most stuff is already discovered and bugs worked out.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: JasonATL on September 17, 2012, 02:26:13 AM
Understood. I would love to have 4:2:2 at this quality. Just saying that having both would be great!

One more point to consider. Is there a way to start a recording at a lower QS and ramp up? Your point earlier to ramp up QS only if below a specific BR made me think that this would be a "safer" way to start a recording. Without it, if there is a very complex scene when someone hits record, then the buffer will fill before the save function has a chance to kick in. This happened to me yesterday with your previous build. By starting QS below its max and quickly ramping up if BR is sufficiently low would only take a few frames at lower quality in return for possibly a more stable function.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 17, 2012, 02:36:57 AM
Thats pretty easy. You can do it manually now, just set slice to somewhere above 129 before you start recording. After you finish, it leaves it where it ended up. Could have it set back lower when recording finishes or something like that.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: JasonATL on September 17, 2012, 02:42:34 AM
Great, thanks!. I wondered what slice did, since it was always different when I finished recording. Must have missed that!
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: matt2491 on September 17, 2012, 04:23:00 AM
Just donated $25 specifically for the bitrate investigation cause! Yay!
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Kabuto1138 on September 17, 2012, 08:29:32 PM
Hey Guys, this development is perfect timing for Magic Lantern/CANON users.

With the release of the GH3 and at the price range we are working at is only a matter of time before people start looking elsewhere(the GH3 or Sony cameras) for this high bitrate capabilities out of the box.  People might loose interest in ML very soon if the canon dslr cameras don't have "competitive" bitrates.... probably the most important new feature ML should have.

keep up the great work guys!!

How do we make ALEX be part of this ASAP?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: nanomad on September 17, 2012, 08:33:36 PM
Quote from: Kabuto1138 on September 17, 2012, 08:29:32 PM
How do we make ALEX be part of this ASAP?

A time machine, probably
The guys here have done a terrific job  ;)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Kabuto1138 on September 17, 2012, 08:39:22 PM
He's done an AMAZING job.  I don't know why Canon hasn't hired him.  ML is probably responsable for a lot of sales.

Just worried that there is a window here of opportunity.  With the Black Magic Camera, Gh3, A99 etc high bitrates should be a priority?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 17, 2012, 11:14:18 PM
There are drawbacks to the other cameras like smaller sensors, lens selection and they don't take stills so well. Also more crop/deeper dof.

But canon is lagging here for sure. The competition is stiff in mid levels.

Sony - I've seen no hacks yet. Over $2k price tag... sony quality and sony reputation. no thanks.
GH3 - If its hacked up and cheap why not. Ripoff lenses but gain IS. Tiny sensor/low mp. Mostly for video.
Black Magic - If software and performance is up to snuff, price is good for a semi-pro and resolution tops most things in that range. If stills are also good and body is comfortable this camera is eating everyone's lunch.

There is simply no market for 5d3 at $4k. At $2k sure. The other cameras ~$1000 are really a good value but where does that leave canon? In the slightly above low end consumer market? Broke film makers? Once semi vendor lock with the lenses wanes they are in for a world of hurt.

The days where it was canon vs $10k+ cameras are soon if not already over. Nikon is pretty good w/stills and lenses are cheaper. Their sensors are still winning but for how long? If anything, copy us faster and release more FW versions with new features. How can you upgrade hardware, sell the camera and do everything possible to not take advantage of it?

Anyways, on topic... instant/average BR is still off so I'm having problems. After dicking with it, its still 33% off and something happens to average BR indicator after 4gb or like 3 minutes. I'm using instant but its not super accurate either (occasionally pops up to like 200-400, not real). I'm having stopping issues at high ISO like 2500. Then again 2500 looks like crap in day time. Need some math help. I'll push what I just did when I test it some more.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: JasonATL on September 18, 2012, 01:16:06 AM
Quote from: 1% on September 17, 2012, 11:14:18 PM
I'm using instant but its not super accurate either (occasionally pops up to like 200-400, not real). I'm having stopping issues at high ISO like 2500. Then again 2500 looks like crap in day time. Need some math help. I'll push what I just did when I test it some more.

1% - Just curious: How do you know that 200-400 is not real? The reason that I ask is that VLC was reporting input bitrates exactly that high in some of my early test clips (and I just haven't checked in my later clips). Bitrate viewer does not show them that high, but my version of bitrate viewer only shows average bitrate in one second intervals (is there a way to get it to report bitrate at higher frequency?).
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 18, 2012, 01:27:08 AM
I did bit rate viewer in gop mode (ctrl-G) and frame mode(ctrl-g) and never saw a bit rate above 2xx) So will have to look at VLC and see if it says anything else. Stream eye also doesn't show anything that high.

To get 400mbps you would need 2x the card write available (50MB). It would have to store the frame in the buffer and write it out in pieces somehow.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: miyake on September 18, 2012, 09:45:30 AM
Just idea.

The H264 encoding has a lot of arguments. So it's a little difficult to implement in current ML menus.
So
-add new config file for encoding parameters like encoding.cfg
- file contents described current 1%'s values and symbolic name with MS-Excel csv style.
- ML will loading the file and values into memory.
- we can choose which profile want to use
    (menu is dynamically added and we can identify symbolic name).

This idea will support dynamic changing profiles without compile environment. And menu is clean. Also the user is able to change detailed parameters, if it understanding. And more, Someone make a "Encoder config generator" on Win/Mac/WEB apps.
How do you think 1%?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: a1ex on September 18, 2012, 09:56:03 AM
The 5D3 firmware can load and parse these INI files:

http://a1ex.magiclantern.fm/bleeding-edge/5D3/H264-ipb.ini
http://a1ex.magiclantern.fm/bleeding-edge/5D3/H264-alli.ini

(the current values in those files contain the default parameters, I think).

I'm thinking to implement profiles in 5D3 too, so this idea can be reused.

Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: nanomad on September 18, 2012, 10:03:42 AM
Really? Lol
Are those official or undocumented?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: a1ex on September 18, 2012, 10:04:44 AM
I've got the field names from strings and the values from memory dumps. And yes, you can change things from there and they take effect (or err70).
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: nanomad on September 18, 2012, 11:45:28 AM
Can't we use the mvrTest function to do some controlled tests? It reads a YUV sequence and sets it as an input to the mvr Encoder. The only drawback is that it reads from the A:/ drive but we should be able to patch it easily....
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Marvin on September 18, 2012, 12:03:33 PM
Let me share with you my findings with the H264 parameters in the .ini file, I did tons of experiments with Alex back in May, during the quiet days after the "Hello World" :-)

I'll start with ALL-I.


Transform8x8Flag = 2
Profile = 100
Level = 51


It seems that these parameters can't be changed (ERR70 if modified). Profile 100 and Level 51 means "High Profile, Level 5.1" in H264 specifications, this is even more advanced than standard Bluray codec which is High Profile Level 4.1.

BitRate = 90000000

This parameter might interest most of the people here :P yes it can be changed to maximum 300000000, which is 300Mbps (specified in H264 standards), compared to original 90Mbps default by Canon. But please note that this value is currently only a cap and does not keep the video file at this bitrate. More on this later.

EntropyCodingMode = 0

Another interesting parameter, there are two modes which are "Context-Adaptive Variable Length Coding (CAVLC), Context-Adaptive Binary Arithmetic Coding (CABAC)". default 0 is CAVLC which is lower encoding efficiency but easier for playback. value of 1 enables CABAC which is said to bring 10%-15% quality increase under the same given bitrate.


IntraPicInitQP = 20
InterPicInitQP = 20
QpOffsetForB = 0


can't be changed if I remember correctly, seems related to initial Quantitization Parameter, more on this later...

MinQpI = 10
MinQpP = 10
MinQpB = 10
MaxQpI = 51
MaxQpP = 51
MaxQpB = 51


an important set of parameters, these values affect the actual video bitrate, for ALL-I, after numerous ERR70s I found out that the three "MinQp" values can be changed up to 7, and "MaxQp" values can be changed up to 20. QP stands for Quantitization Parameter which is the lower the number the better, a value of "0" means lossless encoding :-)

Video shot with these values changed/unchanged did show a noticeable difference in bitrate. Under the same scene and same exposure, I got 70Mbps compared to 40Mbps with Canon default value. Together with the previous "Bitrate" parameter set to 300Mbps I was able to get 151Mbps under ISO12800 compared to 92Mbps default.


MinBitrate = 0
MaxBitrate = 0


can't be changed.

SarWidth = 0
SarHeight = 0
AspectRatioIdc = 1
VideoFmtAndVspFlag = 81
VideoFullRangeFlag = 1
TimingInfoPresentFlag = 0


not very important parameters, doesn't affect the video much.

RateControlEnable = 2

A mysterious parameter and worth looking into, according to some documents, a value of 0 => CBR, 1 => VBR, 2 = Fixed QP. with 0 I get very small video files from 1Mbps to 5Mbps, 1 gives ERR70, 2 is Canon default which must be Canon's secret of manipulating bitrate.

ScalingMatrices = 0
pScalingMatrixAddr[0] = 0
pScalingMatrixAddr[1] = 0
pScalingMatrixAddr[2] = 0
pScalingMatrixAddr[3] = 0
pScalingMatrixAddr[4] = 0
pScalingMatrixAddr[5] = 0


Not sure about the exact meanings but don't seem to affect the video much.

For IPB there is only a few differences.

Profile = 100
Level = 41
BitRate = 10000000


IPB uses High Profile Level 4.1, the same as Bluray codec. Bitrate can be changed to maximum 62000000.


MinQpI = 10
MinQpP = 10
MinQpB = 10
MaxQpI = 35
MaxQpP = 51
MaxQpB = 51


Alex are you sure these are default values? I don't remember LOL, but for IPB, MinQp and MaxQp can be changed to 6 and 30. resulting bitrate can easily hit 62Mbps which is the upper limit set by H264 specifications.

If you look at the metadata of 5D Mark III's .MOV file, it will tell you that the video is encoded by "EOS encoder", which I think it might be talking to the above parameters directly, if a value is not present, for example, if I change "Profile" to 122 which stands for High 4:2:2 encoding, it will give an ERR70...   or if I change the MinQP to something like 2 or 3, video recording will be stuck at "BUSY".

Hope this information is helpful and can save you some time, good luck folks!
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: JasonATL on September 18, 2012, 01:28:43 PM
Miyake - I think that's a great idea! I would be willing to do what it sounds like Marvin has done for the 5D3 in testing sets of parameters for the 600D (and if anyone cares to take this to the 550D, I'll test it, too). What would be really nice is to have a menu that allows me to choose a profile to load and use (sort of like the overlays). At least for this testing phase, it could be more efficient.

Marvin and A1ex -
1. Perhaps I misunderstand, but can I change those parameters on an 5D3 without ML by using the .ini files? How does the 5D3 load them? Do I just put them on the root of the SD or CF card and it automatically loads them?
2. Before I try this, is Err70 an error that needs a battery pull? Or, does it pose a greater risk to the camera?
3. If answers to 1. and 2. are positive, then is there anything I can do along these lines or is all understood by you already?
4. Is the 5D3 the only camera that takes such an .ini file?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: a1ex on September 18, 2012, 01:30:07 PM
JasonATL: this feature is not yet enabled in alpha 1, so you need to wait.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: JasonATL on September 18, 2012, 02:00:19 PM
Thanks, A1ex. I appreciate everyone's work on this.

Just trying to offer help in any way possible. Let me know if there is anything that I can test/try that might be helpful.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: deleted.account on September 18, 2012, 02:24:14 PM
Quote from: Marvin on September 18, 2012, 12:03:33 PM
Let me share with you my findings with the H264 parameters in the .ini file, I did tons of experiments with Alex back in May, during the quiet days after the "Hello World" :-)


SarWidth = 0
SarHeight = 0
AspectRatioIdc = 1
VideoFmtAndVspFlag = 81
[b]VideoFullRangeFlag = 1[/b]
TimingInfoPresentFlag = 0


not very important parameters, doesn't affect the video much.

Hope this information is helpful and can save you some time, good luck folks!

Personally would like the 'VideoFullRangeFlag' on/off as an option for an ini file. :-)

With the fullrangeflag set on as it is with Canon MOV's it signals to the decompressing codec to squeeze luma levels 16 - 235 in any import into the NLE or any transcode / general encoder output. So it can be normalized into 0-1 ie: 0 - 255 in 8bit RGB terms.

I currently use a patched build of MP4Box to be able to switch this flag off in the h264 stream which involves a remux to mp4.

Although the feed into the camera encoder appears to be raw uncompressed JFIF ie: full luma/chroma levels over 0 - 255, the option to switch the flag off for the 32bit float < 0.0  > 1.0, so overbrights exist may be useful.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 18, 2012, 04:22:28 PM
Wish I had .ini loading and QP of 7 :) Way more parameters than Digic IV h.264.  I don't think we have enough parameters to warrant an INI yet.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: JasonATL on September 19, 2012, 02:11:06 PM
I finally edited together 20 seconds or so of a waterfall. Shot with Canon FW, GOP=1 (ALL-I), and GOP=3. The links to screen captures are below, too. This difference in quality is huge, in my opinion. I did not test the ML 2.3 CBR 2.0x or something like that on this particular shot. I'm fairly confident from experience that CBR 3.0 would have caused a buffer overflow, but maybe not.


Of course, download the original file to see this without Vimeo's compression.

GOP=3: Avg 149 Mbps, Peak 170 Mbps
GOP=1 (ALL-I): Nearly constant 169 Mbps
Canon FW: 40 Mbps



Still 1: https://picasaweb.google.com/102645356447404039003/September192012#5789851335385308482
Still 2 (300% zoom): https://picasaweb.google.com/102645356447404039003/September192012#5789851289298012066

I think that ALL-I probably looks the best overall here, but GOP=3 is very close. The increased detail you can see in the rocks, leaf, and shadows in the waterfall is visible at 100% and obvious at the 300% zoom, in my opinion.

I also did GOP=2, 4, and 6 (not shown in the video) for this same subject and don't really see a large difference among them.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: ilguercio on September 19, 2012, 02:28:27 PM
It looks better to me when it comes to compression as the big rock in the foreground is a bit less "blocky".
I suppose it won't make any difference to me, i wouldn't have seen the differences by myself.
I'd prefer having true 1080p but i know it can't happen, or just crop mode on all cameras so we can get rid of line skipping.
Does any of our users have a GHx for comparison?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Marvin on September 19, 2012, 03:28:24 PM
Nice job Jason :)

How did you manage to achieve 170Mbps? 600D uses H264 Baseline Profile Level 5.0 if I remember correctly, if you look at the specification, it says maximum datarate is set to 135Mbps.

Anyway If 600D can achieve constant 169Mbps and is stable for a few minutes then I'm very optimistic that 5D Mark III might be able to hit constant 300Mbps in ALL-I (max datarate set by H264 spec)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: JasonATL on September 19, 2012, 04:01:24 PM
Marvin - Thanks. I don't know enough about H264 to answer your question precisely. 1% has been able to change the encoder's parameters. What this could mean is that the encoder is not compliant with the profile specs or that the change in the encoder preferences abandons the profile (is that even possible?). If you are knowledgeable in this area, perhaps you could shed some light on it for us and help us better understand what parameters we can control and how they will help give us the best output here. I was intrigued by your testing with the 5D3 and look forward to being able to get a better bitrate from my 5D3.

ilguercio -I suspect that, if this was your own footage and you were editing and color grading, you would notice the difference. What I have really noticed is that I can sharpen the footage and push the colors much more than before without it "breaking down." What I mean by this is that with sharpening and color grading, the compression artifacts show up quickly. With such a high bitrate resulting in such a good image to start with, the image is more robust to changes. It also looks far less "digital" to me with a much more natural look. This subjective aspect of the picture sometimes isn't easy to pinpoint as to the exact part of the picture to blame. Finally, what I will try to demonstrate in my next video will be a more detailed image. In this case, what looks like "mush" with Canon FW turns into a very nice picture.

I understand your point about resolution. I would like to see more resolution, too. But, the key point here is that the Canon FW is not giving us the best of what (little) resolution we have due to the bitrate and encoder parameters. 1% has at least made huge progress in fixing this. I am excited about the having this on the 5D3 (and other things in development, as well). But, I am quite impressed with what I'm getting out of my 600D now.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 19, 2012, 04:24:20 PM
Hehe... we are officially out of spec according to avinaptiic. I think its a good thing.

The spec is for HW playback mostly; so if chinsy HW players can't play your raw its a good thing.

I'd love to change the profile and now I know how to override the "profile" numbers in rom but I haven't a clue what the settings mean. High or at least level 5.1 is probably in there. Maybe then QP can move past 10. I'd like to figure out how to turn off resizing if possible too. We tried a little so far but it didn't work. Might be able to manipulate some of the tables used for x/y. Hopefully this can make for 1:1 720P, etc.

I'm still wanting more. 5d3 has way higher data rate and full frame sensor. But $4k price tag vs ~$600 for 600d.


Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Andy600 on September 19, 2012, 04:28:53 PM
Thanks for the test JasonATL. Downloading the original for a better look. Personally I've found GOP3 seems to be the general sweetspot for bitrate vs quality.

EDIT: just looked at the original. The zoomed part of the video really shows the difference between standard Canon CBR and increased BR. Nice job! It will certainly help with chromakey work and when upscaling 720p footage.

Just a thought about resolution. Some guys removed the OLPF from their 5d MkIII's and the resolution was much better though obviously more susceptible to moire and aliasing. Has anyone tried this with a lower spec camera? secondhand 550d's are going very cheap these days so might be worth risking?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Marvin on September 19, 2012, 04:46:31 PM
For H264 profiles and levels, here are some information for reference:

http://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Profiles

http://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Levels

With H264 parameters, here is a table that correspond each profile and its value:

Profile_IDC:

66   Baseline
77   Main
88   Extended

(FRExt profies):

100   High
110   High 10bit
122  High 4:2:2
244   High 4:4:4
44    CAVLC 4:4:4 Intra
118  Multiview High Profile
128  Stereo High Profile

IntraProfile = 0/1, Activate Intra Profile for FRExt (0: false, 1: true, e.g. Profile_IDC=110, IntraProfile=1  =>  High 10 Intra Profile)

Levels are very simple, Level_IDC = 51 means 5.1, 41 means 4.1, just take out the dot :-)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Marvin on September 19, 2012, 04:51:42 PM
Quote from: Andy600 on September 19, 2012, 04:28:53 PM
Just a thought about resolution. Some guys removed the OLPF from their 5d MkIII's and the resolution was much better though obviously more susceptible to moire and aliasing. Has anyone tried this with a lower spec camera? secondhand 550d's are going very cheap these days so might be worth risking?

Removing OLPF won't help much about resolution, it was just an urban legend, those people who removed it have already put it back because it brings significant downsides to the image, such as serious IR pollution, focus shift and dust.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Rush on September 19, 2012, 04:56:02 PM
Quote from: Andy600 on September 19, 2012, 04:28:53 PM
Just a thought about resolution. Some guys removed the OLPF from their 5d MkIII's and the resolution was much better though obviously more susceptible to moire and aliasing. Has anyone tried this with a lower spec camera? secondhand 550d's are going very cheap these days so might be worth risking?
As I know, later tests showed that there were no improvement in detail with removed OLPF. And the image has red tint (no IR filter) which breaks correct colors.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Andy600 on September 19, 2012, 05:02:56 PM
Thanks guys. re: OLPF - I remember thinking it was a crazy idea at the time but didn't check back to see if they had stuck with it.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 19, 2012, 05:23:33 PM
Profile is not that simple. The numbers don't match. Don't know exactly where to set.

if arg2 == 0 /*EQ*/:
                *0xC0E1000C = 41104 < Normal Jpeg?
                DryosDebugMsg(0x15, 0x2, '[JPCORE] JP62_OPMR3 %#lx', 0xa090) => ret_DryosDebugMsg_FF1E35E4
            if arg2 != 0 /*NE*/:
                *0xC0E1000C = 41091 < H264
                DryosDebugMsg(0x15, 0x2, '[JPCORE] H264_SPS_PPS JP62_OPMR3 %#lx', 0xa083) => ret_DryosDebugMsg_FF1E35E4



What are some lower FRext profiles.

I find 88 and 12 being thrown around in there.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: jgharding on September 19, 2012, 05:29:07 PM
Something I've not quite grasped: what stage are picture profiles applied at? I'm looking for a way to get optimum bitrate for 4:0:0, that is to say, black and white. Would an All-|I hack and monochrome picture profile be the best way?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: JasonATL on September 19, 2012, 05:30:04 PM
Andy600 - I don't think that a 550D has a very powerful OLPF. Otherwise, we wouldn't have the moire problem that we do. Still, it could have one. Perhaps when I finally get my own 5D3 (the one I use from time to time is my wife's), I'll have the guts to take my 550D apart. (because at that point, it will be fourth instead of third). I don't believe (I use this word purposefully) that removing the filter improves true resolution on the 5D3. In the end, this shouldn't be a matter of "faith" (e.g., I shouldn't believe or not believe it). But, right now, people are using faith rather than evidence, because there has been no evidence. I have not seen a side-by-side controlled experiment nor have I seen an actual resolution chart shot with video on a camera that removed the OLPF on the 5D3. I recall that James Miller shot the same scene (one with and one without the OLPF), but on different days in different light. Does perceived resolution increase? Perhaps. True resolution? I doubt it. But, I'm willing to believe it if I see the evidence. As I said, it shouldn't be about faith.

People believed the Nikon D800 had much higher resolution than the 5D3 (we're talking video here). In my tests of shooting a resolution chart, the D800 comes in at about the same resolution as the 5D3. If any, it MIGHT have about 20 more lines. But, if the 5D3 is about 800 lines, it isn't as if the D800 is Full 1080p (and the moire you get with the D800!). My point is that perceived sharpness and actual resolution are two different things. This is one reason that I'm happy about having more latitude in sharpening the 600D (and the 5D3) footage with improved (or less?) compression.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Andy600 on September 19, 2012, 06:15:09 PM
Resolution was a wrong choice of words from me. I think there is a definite increase in sharpness without the OLPF in James Millers' test shots but the IR and dust problems caused by removing it seem a step to far.

BTW JasonATL what settings were you using for the GOP test? Did you just increase the CBR and change GOP settings? Also did you alter DBlock A/B settings?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: JasonATL on September 19, 2012, 06:47:27 PM
Quote from: Andy600 on September 19, 2012, 06:15:09 PM
BTW JasonATL what settings were you using for the GOP test? Did you just increase the CBR and change GOP settings? Also did you alter DBlock A/B settings?

On the waterfall video, I don't know (I wasn't keeping very good notes  :-[ -- I'm sure about the GOP, though, since I can infer it in Bitviewer). I have tried different settings for PicPC, DBlock A/B (with notes). There appear to be differences in what I can only describe as "grain" - and I could only see it at 300% zoom, and even then, it wasn't strong. But, I could just be seeing things. Even so, while there appeared to be differences, I would hesitate to call one "better" than the other. At the same GOP length and bitrate, there isn't a very noticeable difference to me in these settings. In the end, I couldn't determine a preference for one setting over another. Is there a setting for these that you prefer? If so, what should I look for?

I agree that GOP=3 is about the sweet spot. GOP=4 works fine and GOP=2 works fine. But, again, I see very little difference among these in the tests I've done.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Andy600 on September 19, 2012, 07:22:50 PM
Quote from: JasonATL on September 19, 2012, 06:47:27 PM
On the waterfall video, I don't know (I wasn't keeping very good notes  :-[ -- I'm sure about the GOP, though, since I can infer it in Bitviewer). I have tried different settings for PicPC, DBlock A/B (with notes). There appear to be differences in what I can only describe as "grain" - and I could only see it at 300% zoom, and even then, it wasn't strong. But, I could just be seeing things. Even so, while there appeared to be differences, I would hesitate to call one "better" than the other. At the same GOP length and bitrate, there isn't a very noticeable difference to me in these settings. In the end, I couldn't determine a preference for one setting over another. Is there a setting for these that you prefer? If so, what should I look for?

I agree that GOP=3 is about the sweet spot. GOP=4 works fine and GOP=2 works fine. But, again, I see very little difference among these in the tests I've done.

I don't really have preferred settings as such but adjusting the DBlock A & B to -1 gives a uniform fine grain look. Setting -2/-1 makes the pixels look elongated. Setting both A&B to 0 seems cleanest.

I've tried setting I factor to 2x when trying out ALL-I but it doesn't appear to do much to the actual image. Also tried setting P Factor to a higher setting than I Factor when shooting GOP 3 etc (my thinking is to bring the slice numbers closer together so the bitrate remains fairly constant?). Not sure this does much either. I'm not sure what increasing GOP0, GOP1, GOP2, GOP3, GOP4 Factor actually does? 1%?

I think we're splitting hairs TBH. Increasing bit rate and setting GOP to 3 or 1 shows a marked improvement over Canon FW defaults which I think is a good enough reason to use it and like you stated you can push the image a little further in post.

I've read a lot about ALL-I and GOP on the GH2 forums and the general consensus is that regardless of image quality gains or losses, shooting ALL-I gives a more filmic look to motion but there isn't much in it if you keep GOP under 6.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 19, 2012, 07:35:18 PM
If you lock sliceqpd CBR/VBR doesn't do a thing. The most they did to begin with was setting slice based on predictions or fixed value. So CBR was just raising the sliceqpd when you set higher numbers for the prediction and thus you got higher bit rates. Notice how it tries to set Q+24 when sliceqpd is a low number. In all I it seems to be broken and not do anything. Probably because prediction is done from P frames.

You can still change gop and use old CBR/VBR just set slice to 0 and reboot.

Upping picqpy boosts luma quality by a good %. PicQPC is for chroma quality and while you can change it its calculated off picqpy so the best PicQPy has the best PC too. All we could go was 1 up from default, others froze camera. It shows when you play back the movies and the camera can't (luma goes crazy), that's not from gop.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Andy600 on September 19, 2012, 07:44:40 PM
Ah, that kind of makes sense now. So it's probably better to not change factor settings and just set BR and GOP?

I've been using PicPC set to 0. Didn't go any higher. I actually didn't try to  ::)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 19, 2012, 07:55:09 PM
If you turn picpc to 0 then py will be 20 vs default of 26. From H.264 spec this was a quality increase.

Best set by slice but then you're at risk of breathing and buffer underruns. Depends on what you're shooting and stability needs. When BR is reported correctly and the quality dropping is spot on it should be better Q than CBR hands down.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Andy600 on September 19, 2012, 07:57:47 PM
Quote from: 1% on September 19, 2012, 07:55:09 PM
If you turn picpc to 0 then py will be 20 vs default of 26. From H.264 spec this was a quality increase.

Best set by slice but then you're at risk of breathing and buffer underruns. Depends on what you're shooting and stability needs. When BR is reported correctly and the quality dropping is spot on it should be better Q than CBR hands down.

Got it :)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 3pointedit on September 20, 2012, 02:34:12 AM
I see that Marvin referred to encoding parameters for anamorphic and scale, earlier. I wondered if higher frame rates (above 60fps) could be achieved at lower resolution (SD 640x480) but in 16x9 aspect, that is an anamorphic version of SD?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 20, 2012, 05:05:03 AM
What is the highest LV fps you get? That is probably best we're going to do. If anything is found it will for sure be implemented.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: JasonATL on September 20, 2012, 08:04:58 PM
Some real world experience with this: My daughter's class just took a field trip to horse stables this morning. In 3 hours, I shot about 45 minutes or more of video (clips about 20 seconds each). Used GOP=3.  It was a sunny, windy day. No test shot... just shooting as I normally would. No overflow once. The buffer save function only had to kick in a few times and in the real footage, I can't tell. It looks like my avg bitrate was around 80-90 Mbps, with some peaks in the 130 range. I'll probably edit the footage this weekend and post the video. I wouldn't have been able to do this with the old CBR. Worked great!
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: JasonATL on September 22, 2012, 12:15:08 AM
More photos of test shots of leaves blowing on trees - exciting stuff, I know  ;).  I shot these today during some wind. To me, the difference between the Canon FW and either GOP=1 and GOP=3 is visible without pixel peeping on the 300% zooms. My wife noticed on the raw video. I didn't bother posting another vid, as I think the captures tell the story.

https://picasaweb.google.com/lh/photo/jEy4yTlS6DyEXEu46gA9DBfNx8dP82C1K8bmBL14X3k?feat=directlink
https://picasaweb.google.com/lh/photo/Q-Oio1E4HegePZwbTqVt2xfNx8dP82C1K8bmBL14X3k?feat=directlink
https://picasaweb.google.com/lh/photo/vDQ2AsTA_cw9hamymrO-xBfNx8dP82C1K8bmBL14X3k?feat=directlink
https://picasaweb.google.com/lh/photo/uDL8Pka3l0N2o1gWm9hdWxfNx8dP82C1K8bmBL14X3k?feat=directlink
https://picasaweb.google.com/lh/photo/qvH_zxX9KWx1AxLp147P4BfNx8dP82C1K8bmBL14X3k?feat=directlink
https://picasaweb.google.com/lh/photo/mzN3khs9lDsxkgt5zt-VeRfNx8dP82C1K8bmBL14X3k?feat=directlink

Without 3x crop mode:
GOP=3 BR: Avg=139 Mbps, Peak=207 Mbps
GOP=1 BR: Avg=128 Mbps, Peak=129 Mbps
Canon FW BR: Avg=45 Mbps, Peak=138 Mbps

With 3x crop mode:
GOP=3 BR: Avg=121 Mbps, Peak=168 Mbps
GOP=1 BR: Avg=115Mbps, Peak=128Mbps
Canon FW BR: Avg=46Mbps, Peak=100 Mbps

1% a couple of things happened while shooting these tests that I've not experienced before. First, the recording would stop with a "camera stopped recording" message without a buffer overflow. I was watching the debug info (Slice was around 122 in many cases) and the buffer % was in the 20% to 30% range. Nothing obvious shows up on the bitrate (it was in the 120's when many of the clips stopped). Also, this happened 30+ seconds into the footage. In other words, the buffer save function seemed to be effective and not the cause of most of my stopped recordings.

Second, (this might be related). The recording time turned red. What does this mean? In some cases, it turned red just before the recording stopped. In other cases, the recording would continue with the time in red for 30 seconds or more. Is this a sensor or processor temp warning? If so, we have another element to consider. This was the first time I had let the recordings go for a long time. If it is a digic temp warning, then perhaps the polling rate could be affecting it. Also, I didn't seem to have the stopped recording issue the one or two times that I had rebooted and then failed to turn debug on. To the extent that the debug adds overhead, perhaps turning off GlobalDraw might help?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 22, 2012, 02:07:05 AM
Global draw doesn't affect BR and stoppages much. I think the problem is you're over 4gb (hence the red) and the BR measuring and buffer functions are getting inaccurate or the card is slowing down. This has to be fixed along with making BR more accurate. The math is set up for 1s and we're doing 1/10ths of a second. I've experienced it as well and will have to test more when work winds down a bit. You have to fix the over 4gb clips too with g3gg0's movie fixer before you edit them.

*Try to do 1gb write test and see where it ends up, mine drops a bit from 20 so probably a complexity increase will affect things more at that point until card recovers.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: FilmMan on September 22, 2012, 03:09:08 AM
Jason,

There is a noticeable difference.  I downloaded Trees 3x3 and Trees 3x1, both ungraded versions.  Quick color grade and added sharpening.  The Canon FW degraded more(especially noticeable in the leaves) whereas GOP1 and GOP3 held significantly better (virtually intact).  Thanks for posting!  Cheers.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: FilmMan on September 22, 2012, 03:12:21 AM
PS.  I felt GOP 3 had the edge over GOP 1 too. 
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: JasonATL on September 22, 2012, 03:53:01 AM
Quote from: 1% on September 22, 2012, 02:07:05 AM
Global draw doesn't affect BR and stoppages much. I think the problem is you're over 4gb (hence the red) and the BR measuring and buffer functions are getting inaccurate or the card is slowing down.

Nope, not over 4gb. No clip was longer than 1.5 GB.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 22, 2012, 05:36:27 AM
Must be another bug with the time function. Wonder if elapsed time is accurate at all. Is that what you're using or time till 4gb or the other one? I'll have to see why it turns red exactly. For sure related to how fast its counting now and any possible inaccuracy from msleep.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: JasonATL on September 22, 2012, 02:43:39 PM
I was using elapsed time. It seemed accurate.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 22, 2012, 11:39:11 PM
The only thing that is supposed to make it red is going over 4gb and since its counting bit rate wrong it probably thinks thats where it is when its actually like 1 or 2 gb in. After that BR is completely wrong and quality lowering function thinks you're doing 50mbps when you're actually at 150 and stuff like that. I'll have to see whats actually in MVR_BYTES_WRITTEN.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 23, 2012, 01:24:45 AM
Try this:

https://bitbucket.org/OtherOnePercent/tragic-lantern/downloads/autoexec.bin.BetterBRI


Made BR not update so fast but buffer still does. I think the problem was a little conceptual.. don't know why it took 3 cans of "steel reserve" for it to dawn on me.  We were getting megabits / 10ms, lol. Buffer save works with new peaking but indicator will be off from slow redraws. Also has full screen magic zoom but doesn't work in crop mode. There is some danger of stoppages in the first second while BR is calculated. Maybe through sheer luck its working better for me from cursory testing.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: JasonATL on September 23, 2012, 02:20:23 PM
Thanks, 1%! I'll give it a shot later today, I hope, and report back.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: JasonATL on September 23, 2012, 11:59:12 PM
I haven't shot anything challenging with this new build yet. But, I wanted to give folks a heads up in case they try this beta (any version, probably). I had left the audio on accidentally when engaging CBR on this. I got an Err70. Recovered fine, but Err70 several times without writing a movie file. ML log was written. So, be sure you have disabled audio in the Canon menu before testing this! Otherwise, this version is working - again, haven't shot anything that would tax the bitrate yet.

Also, I edited the video that I shot last week. It isn't "wow look at this picture quality" material. It was me handholding my 600D with my Zeiss 50mm 1.4 and chasing my daughter's classmates around a horse stable, trying to get exposure on the fly (weighing overexposure against seeing into shadows - literally). Also, the color grade is not one that necessarily wows (I was going for a look). But, I shot it all with 1%'s bitrate enhancement and I will say that, unlike working with normal footage, never did the grading or sharpness get to a point that I thought constituted a breakdown. I was able to push sharpening far further than normal. https://vimeo.com/49988361
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 24, 2012, 01:22:08 AM
Did you leave record separate wav on + canon audio. Thats how I can reproduce it. It initializes audio stuff before hand so you have to pick 1 or the other.

I didn't user proof this one but I took out the 68% stuff since BR measuring is ok. Won't increase quality if BR is already at 115Mb/s. Can make this customisable or increase it if its too low. Also drops faster if your already above 100.

https://bitbucket.org/OtherOnePercent/tragic-lantern/downloads/autoexec.bin.600D.MoreStopProof
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Kabuto1138 on September 24, 2012, 02:47:16 AM
Looking pretty good!!

JasonATL, have you noticed any banding on the raw footage?

thanks!!
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: JasonATL on September 24, 2012, 02:04:44 PM
Thanks again, 1%. And thank you for saying, "user proof" rather than "idiot proof"! The latter might have been more accurate when referring to me, I think.  ;)

We're supposed to have some wind here today, so perhaps I'll be able to test out with some blowing leaves again, since I have experienced how this particular feature has behaved in this circumstance.

I haven't noticed any in the raw footage. Do you ask because you see it in the edited footage or shots? If so, perhaps I'm just not seeing it.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Kabuto1138 on September 24, 2012, 07:23:25 PM
Hey Jason, I didn't see any banding on the footage.  I just wanted to know because just THAT improvement... is AMAZING... no frigging banding!!

Thanks!!

Can't wait to try this eventually in my 5d Mark II!!
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 24, 2012, 11:18:13 PM
Only time I got banding was when editing a completely under-exposed iso 5k test clip shot in super neutral. Between SN and marvell's advanced... best footage ever. No profile/neutral doing to much color crushing for me.

Hehe, anyone can make the audio mistake. Thats why FPS override audio (from ML main) turns off canon audio itself to make sure. I don't have time to wait for that and leave canon audio off 100% of the time since its easier turning it on/off from audio menu and this way it doesn't fight with canon's functions. I haven't checked sync accuracy very thoroughly but they start at the same time so should be good.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: JasonATL on September 25, 2012, 12:20:49 AM
1% - Still working with this latest build and tried GOP=3. It looks to be absolutely awesome. The buffer save is clearly working more effectively now. I gave it quite a workout for about 15 minutes. I'm trying to look at the footage now to see if I can see the difference. My first pass through, I couldn't see when the save function kicks in - so this is great! No more "breathing". I tried to overflow the buffer many times and haven't been able to yet. The recording didn't stop on me in my few tries.

Also, I assume you're seeing this: peak bitrate at 250 Mbps on the i-frames! Craziness!
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 25, 2012, 05:11:11 AM
If only that 250 was consistent.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: unity2k on September 25, 2012, 05:32:08 AM
Guys, for some of us following this, you are doing some intriguing, amazing things on the 600D. I look on with envy. Looking at JasonATL's comparison shots between GOP3 and Canon FW, specifically the leaves that are crisp and clean on the GOP3 while the Canon FW shots are full of jpeg artifacts and mushy edges. These developments appear to be a giant leap forward.

What's missing from this dialogue is, what's it going to take to give more of us the opportunity to start working with this version of ML? I'm using a couple of 550D's and am more than willing to potentially sacrifice the older one in order to give GOP3 a run.

I'd imagine I could get the code and give it a try to port it to my camera, but what about a compiled version we can try out?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 25, 2012, 05:56:03 AM
I can try to turn off assertions and and just change the gop on 550D but you'll only get ~4 minutes of video since there is no limit hack. Dunno why other 550D people haven't tried it. All you have to do is ifdef/ifndef out the cache hack parts. Its working on 1100D too. Its not so much that your camera would break, just that the hack wouldn't produce the greatest results.

Its not just the gop, its the gop combined with direct quality control (technically above qscale limits). That part, I have no idea how it works on other firmwares.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: unity2k on September 25, 2012, 06:00:05 AM
Thanks 1% for the quick answer. As others have already said, I too do not care about the 4 minute limit. When filming nature elements very rarely do I need a segment of that length. Plus, I'm just as interested in doing some green screen testing.

So, when and where can I download it from you?

You've made my night!
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Andy600 on September 25, 2012, 12:16:19 PM
Hi 1%

I've been trying out your latest build and I'm getting some pretty wild BR fluctuations especially when setting GOP greater than 1. BR is  dropping as low as 2000 and seems to happen when I fast pan or move the camera quickly. It's very noticeable on footage.

I've also achieved sustained BR at around 80-90mbit ALL-I but can't remember the settings doh :( Can the movie logging function be modified to save BR/Encoder settings?

In your tests what have you found to be the best settings for sustaining a higher than FW BR? I'd be very happy with sustaining 80mbit but stopping it from dropping much below that. It's nice to know it can peak above 200mbit+ without stopping but in the end the footage will only be as good as the lowest sustained BR.


Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 25, 2012, 04:15:08 PM
Haven't been getting that but I have to check it out. Shouldn't be dropping that low unless you cover the lens or something. Logging would kind of kill it as would have to write to the card too.

Pan around with debug and see what quality is when you get that drop. Did a quick before work test. I did get a really low BR drop but that was when lens passed my door frame (black, no detail). I didn't see quality move for me when I pan, its all 87. But BR did fluctuate as scene changed. Lots of blur and rolling shutter... are you shooting 30p?

Just spun around at 30P and only get drops when hitting things like heavy motion blur scene/empty sky/overexposure. I'm not seeing Q move for these, just camera compresses the scene. Gop3. Post a frame grab and BR graph.

QuoteBR at around 80-90mbit ALL-I

Low ISO like 100 or 200.

Theres not that much to set at this point, just gop to 1 or 3 and the slice takes care of itself.

This is yesterdays merges, still have to do todays:

https://bitbucket.org/OtherOnePercent/tragic-lantern/downloads/autoexec.bin.MostRecent1
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Andy600 on September 25, 2012, 05:42:53 PM
Quote from: 1% on September 25, 2012, 04:15:08 PM
Post a frame grab and BR graph

I would but I've since reformatted the card. I think it might possibly have been a read/write error on my card because one of the .mov files was corrupt on playback. If it happens again I'll be sure to get a framegrab and the BR graph. Gonna try your latest binary. Thanks as ever! :)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 25, 2012, 05:51:14 PM
Yea, all this writing has got to be killing the cards. Oh well, thats what they're for. My patriot card seems broken in at this point. Lowest quality is like 129 which is like qscale +17 or so... 2k bit rates come from lower q's that we shouldn't be going to in this implementation. I made some of the lowest quality vids ever at QP 51.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Andy600 on September 25, 2012, 06:40:42 PM
I'm using a Patriot card too and it's not bad for the money but it's starting to fall apart... literally ;D

Your new Bin seems good :)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 25, 2012, 07:34:28 PM
I got 120mb average over 5minutes on iso 80.   :o
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Andy600 on September 25, 2012, 09:00:28 PM
Quote from: 1% on September 25, 2012, 07:34:28 PM
I got 120mb average over 5minutes on iso 80.   :o

Not bad ;)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 25, 2012, 09:05:25 PM
Drove around too when I went to get smokes and only got one stop (a first record) even with trees. BR was a little loopy but I have to see if its visible. I did get 1 frame at 16248kbps somewhere in the begining but haven't watched the footage yet. Also a 234Mb/s peak (in about 25 frames all over 210).

heheh, actually can't play any clips back on my development laptop at all unless really small. even a 400mb clip slows to a crawl.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Andy600 on September 25, 2012, 09:09:12 PM
On a Patriot card?  :o

My card seems to fall over at about 150mbit (previous builds). I'll try and push it harder tomorrow. Doing some night shots atm to tweak another PS and testing high BR/High ISO for noise performance.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 25, 2012, 09:12:11 PM
Yea, patriot 64gb UHS-1. It won't constantly write above 150 but sometimes it will burst, usually that would produce a stop before but now it just drops quality until card can catch up.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: unity2k on September 26, 2012, 12:20:31 AM
Took me a moment to figure out what settings were affecting what, and I am yet to start making notes as to results as I'd like to know my way around the settings, but, after some false starts, I've had success on my 550D.

I've looked back at some very detailed videos shot in the Grand Canyon on this camera, they are fairly consistent with bitrates between 45,000kbps and 50,000kbps, checked a bunch of others recorded with Canon FW, same bitrates. But now I have this new video I shot outside of some trees with a good wind kicking them around; bitrate reading is 139,793kbps. The 34 second 1920x1080 24fps video is 566MB in size - WOW!

Subsequent tests I have seen consistent bitrates of 166,817kbps with the buffer hanging around 30%. All of these tests were performed on a Pretec 64GB SDXC1 which claims to be a Class 16! Tried using the same settings on a SanDisk 64GB SDXC1 Class 10 and the only consistent numbers I get are at about 65,000kbps, better than Canon FW but not as amazing as compared to what I can record on the Pretec.

I've had some Error 70's, but only on the SanDisk card. Removing the battery cleared the issue. Also had each card become corrupted requiring reformatting.

Anyone care to explain the Encoder Options, or point me to somewhere I can understand what they do?

I do know that if I can have everything set to 3.0x while recording to the Pretec card. While on the SanDisk if I set P factor through D2 factor at 2.5x and GOP0,1,and 4 at 1.0x, with GOP3 factor = 2.5x, I get the results stated above. I haven't changed JP Slice.

And maybe I'm misunderstanding something, but how do I record all I-frames?

Amazing work guys. As soon as I can I'll setup both my 550D's to record the same scene, one with Canon FW and the other with this bitrate magic, I'll post what I see.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 26, 2012, 12:34:08 AM
Change gop size in "CBR" menu where you would go to change qscale/cbr ,etc. Slice doesn't work for you yet. I didn't prune the menus. All I is 1. That is the critical part. Make sure gop changes. Otherwise you'll just get buffer overruns as it tries to hold like 12+ frames in the buffer.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: unity2k on September 26, 2012, 12:57:03 AM
On previous tests I had set GOP in the CBR menu to 3.

With your tip I set it to 1 with all encoder values at 3.0x. On the Pretec card I recorded 29 seconds at 104,813kbps with only about 15% buffer fill, mostly less. A second longer test of 46 seconds was 98,612kbps. Both videos are of leaves in a strong wind.

On the SanDisk I'm getting similar bitrates, but I also had a few Error 70's and the buffer was quickly moving up just before the crashes.

3 log files were created, here's a part of the first one:

Tue Sep 25 15:37:38 2012
  37965: 14643.497 [MR] 1 : W=0x4a9b7120, E=0xfa0340, W=0x4a079a50, E=0x536650
  37966: 14661.078 [MR] mvrExpStarted
  37968: 14663.853 [MR_MOV] AddVideo : (0x4a9b7120,0x81934,0x00000000,0x0,1)
  37969: 14663.914 [MR] mvrFrameWrite (Size = 0x81934, Next = 0x4aa38a54)
  37970: 14663.956 [MR] mvrEncodeDone (Size = 0x81934, Next = 0x4aa38a54)
  37976: 14675.570 [LV] PCLV JPEG SIZE (1056 704)
  37977: 14685.342 [MR] mvrEncode (Addr = 0x46000080)
  37978: 14685.418 [MR] 1 : W=0x4aa38a54, E=0xf1ea0c, W=0x4a079a50, E=0x536650
  37979: 14692.497 [MR] mvrWriteDone2 (Size = 0x102cb0, Next = 0x4a6b2d50)
  37981: 14692.650 [MR_MOV] RequestWrite : End(32)
  37982: 14692.756 [MR_MOV] RequestWrite : Start(33)
  37984: 14692.826 [MR_MOV] WriteSampleData : #1 - Buf=0x4a6b2d50, Size=0x103570
  37985: 14702.432 [MR] mvrExpStarted
  37987: 14705.981 [MR] mvrFrameWrite : Frame = 72, Pad1 = 8
  37988: 14706.056 [MR_MOV] AddVideo : (0x4aa38a54,0x76bec,0x00000000,0x0,1)
  37989: 14706.168 [MR] mvrFrameWrite (Size = 0x76be4, Next = 0x4aaaf640)
  37990: 14706.212 [MR] mvrEncodeDone (Size = 0x76be4, Next = 0x4aaaf640)
  37991: 14706.327 [MR] mvrEncodeDone (Sec = 3, Frame = 72)
  37993: 14706.451 [MR] mvrEncodeDone (Pack = 36, 0xb507f0, 0x22949d8 - 0x238cef0)
  37994: 14706.478 [MR] mvrGetBufferUsage : Frame=8, Sound=0
  38000: 14707.252 [MR] mvrDecideAndAddStorageCBR (0xb507f0)
  38004: 14708.728 [LV] PCLV JPEG SIZE (1056 704)
  38005: 14710.734 [LV] lvAEResult
  38006: 14710.820 [LV] Release FaceAEL and FaceALOLock 9385
  38010: 14718.370 [LV] PROP_LV_EMD_DRIVE_RESULT AfState:0
  38011: 14726.839 [MR] mvrEncode (Addr = 0x44000080)
  38012: 14726.915 [MR] 1 : W=0x4aaaf640, E=0xea7e20, W=0x4a079a50, E=0x639300
  38014: 14734.950 [LVFD] LVCPF_SortPrimaryFace( Input Num = 0 )
  38015: 14735.064 [LVFD] OutputFace( ID=593, (240, 165), Size=25, Num=0 )
  38016: 14744.147 [MR] mvrExpStarted
  38018: 14747.800 [MR_MOV] AddVideo : (0x4aaaf640,0x824e8,0x00000000,0x0,1)
  38019: 14747.861 [MR] mvrFrameWrite (Size = 0x824e8, Next = 0x4ab31b28)
  38020: 14747.905 [MR] mvrEncodeDone (Size = 0x824e8, Next = 0x4ab31b28)
  38024: 14752.328 [LVCAE] lvcaeGetAllWbGain(sync)(10)
  38025: 14752.633 [LV] ====== lvCompleteALOGetSeed [0x408cd840]
  38027: 14753.790 [LVCLR] Out = (0, -1, -1, 0, 0, -1, -1)[0]
  38030: 14759.003 [LV] PCLV JPEG SIZE (1056 704)
 

Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 26, 2012, 01:53:51 AM
Does the movie come out at gop3 though? or gop 1?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: unity2k on September 26, 2012, 03:39:16 AM
How can I tell if it is GOP 1 or 3? The video's I recorded at 3 work fine as do the one's I set at GOP 1, or is there something I'm missing?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 26, 2012, 03:44:24 AM
Mediainfo will tell you N=1 or N=3 or N=12/15 that is gop. Also can count distance between I frames in bit rate viewer.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: unity2k on September 26, 2012, 03:53:43 AM
N=1! 103Mbps 23.976fps
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: JasonATL on September 26, 2012, 03:56:33 AM
Andy600 - I don't see the problem that you indicated, so I'm guessing that it is the suspected card problem.

Here are my results (all at GOP=3):
The buffer save function has worked flawlessly in real conditions. I was able to overflow the buffer by using TV static and moving the camera to a blank wall and then quickly to the static at 60% warning. I could not overflow it at 50% warning. In short, the save function seems much more robust than before.

Here are two comparison shots.
Full: https://picasaweb.google.com/lh/photo/8AgT16svWHKXAOzcNPd8BxfNx8dP82C1K8bmBL14X3k?feat=directlink
Zoomed 300%: https://picasaweb.google.com/lh/photo/I38AfErPsBMBzwdyzCZJbBfNx8dP82C1K8bmBL14X3k?feat=directlink

On these images from left to right:
Left is when the bitrate (and QS) is high (wind wasn't blowing much). BR avg=160 Mbps; peak=250 Mbps Slice=87
Middle is when the bitrate was dropped (but after wind has calmed a bit). BR avg=90 Mbps; peak=129 Mbps Slice=127
Right is Canon FW: BR avg=40 Mbps; peak=155 Mbps
Note that Left and Middle are from the same clip, just at different points based on the bitrate.

To me, the Canon FW is clearly inferior at full resolution (I think it was reporting Q-01 or Q+00). I can barely make out the difference between the other two. The difference between the middle and left is apparent in the 300% zoom of the footage, but the middle is still superior to the Canon FW.

In playing the video, I still cannot tell by watching when the buffer save function kicks in or lets up.

An oddity: Slice nearly always goes to 126 or 127. I had it go to -127 a few times. Also, when changing slice, I only had it stop at something other than 126 or 127 on the way down. I wonder If it is always going to 127 anyway, then why not just jump it there in the first place? Also, could there be a user setting for the min slice (or max QS)? This might allow the user to set it to a level that is less likely to get to the overflow (and perhaps might make it more workable for slower cards?). Given the frame captures above, I'd wonder if I could tell the difference between slice=120 and slice=87? And, if slice=120 saved me from getting to 127, all the better? I'd bet that 190 Mbps I frames would be nearly indistinguishable from 250 Mbps, but I could be wrong. Edit: I take that back, I want 250 Mbps when I can get it ;).
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on September 26, 2012, 04:22:07 AM
-128 is the next step down, took it out and it was too low. Now it only comes up till you get to 115mbps and that isn't necesarily slice 87.

You can see what slice is set for CBR mode if you set jp slice to 0 and reboot. Then turn on debug and start recording. After the first time it will start hacking cache if debug is set on so you have to reboot to see again. Doesn't change if you're just using CBR.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: KahL on October 01, 2012, 05:43:55 PM
I know I'm a complete newb to this topic, but I've been reading off and on for the past few weeks.
How can I get a copy of this latest ML build to test out on my end? I'm really excited about the potential quality jump this has, especially for post work.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on October 01, 2012, 06:01:18 PM
Bins are in my repo. should be links throughout the thread.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: bart on October 01, 2012, 07:01:52 PM
Great work!
Maybe an idea is to make a script that switches settings every half second and tries all settings at big value intervals. Then point it to windy leaves or water or high detailed landscapes like forests with wind. Publish resulting video and let people decide what looks best.

Also if you need a frontpage article just let me know.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on October 01, 2012, 07:55:31 PM
I'd hope to eventually get some of this stuff into ML tree with better coders getting on it. 550D gop changed so can implement some of the stuff into that too, just need FW locations. Andy's tests already show its better than CBR in terms of Q..  way worse for size.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: unity2k on October 01, 2012, 08:02:33 PM
1% So my testing shows great bitrates for busy scenes, what I don't know how to do yet is get increased bitrates when filming more static shots. For example, filming a greenscreen shot with little action, the bitrate between regular fw and GOP3 are nearly identical. I understand this may just be efficiency of compression and uniformity of elements in the image, but the edge artifacts in busy shots using GOP3 are so clean, I would expect that I could get cleaner edges in non-busy shots too, but they are about the same.

I'll be the first to admit I may be a bit dense here and this is exactly what should be happening, but also think, maybe I'm missing something. Any advice?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on October 01, 2012, 08:06:51 PM
550D is missing the quality settings. With that you'll get higher BR. Right now you're mainly getting buffer benefits but CBR is still doing the Q setting.

Heh, before its all good to go, need to fix the audio sync issue. Then we have a 100mbps source w stereo audio and sync. Then see which cameras work this way and make the cache hacks for them.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: AriLG on October 01, 2012, 09:12:42 PM
What is to be "lost" (feature wise in regard to bitrate control) with the 550D (comparing with the 600D) ?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on October 01, 2012, 11:28:08 PM
Control of slice, which is basically 1/2 of it.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: AriLG on October 01, 2012, 11:33:50 PM
So that makes it camera specific ...  :(
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on October 01, 2012, 11:53:37 PM
Pretty much and 550D build has asserts disabled so its not proper.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Leon on October 04, 2012, 02:26:27 AM
@ 1% :  What confuses me is the difference between Q-value (which I'm assuming is effectively the same thing as quantiser value in x264) and this new "slice" value, which doesn't seem to exist in x264, or at least not as a user-settable value.  Can you explain the difference and why both values exist?  Can you change one without changing the other?

If slice is like Q-value, could we establish a "safe" value for a certain GOP, e.g. by incrementing the slice quality value and record TV static noise until it can't cope reliably, then using a value that was safe?  Of course this wouldn't give you the "best" quality for low-complexity scenes (since you could have pushed slice harder then), but I'm guessing they'd still look better than the firmware default, and you'd still have "best" quality for high-complexity scenes, where you most need it.  It would also give you your upper limit (or lower limit, whichever way you look at it), which you should never want to go beyond.

(Personally, I'm ultimately hoping to see a feature where I can use something vaguely equivalent to Constant Rate Factor in x264, where very high bitrates are available for complex scenes, without overrunning the buffer, but lower bitrates can automatically be used for low-complexity scenes/frames.  Whilst super-high bitrates may be useful sometimes, I won't want to use them all the time.)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Leon on October 04, 2012, 02:33:57 AM
Also - sorry if I'm missing something - people seem to be comparing the new encoding to the original FW at CBR 1.0x, but would it not be much more appropriate to be comparing it to CBR 3.0x / 2.0x or whatever the card can manage?  (We already know that CBR 2.0x is better than CBR 1.0x.)  Can people see a difference between the new encoding and "normal" CBR 3.0x?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: JasonATL on October 04, 2012, 03:24:44 AM
Quote from: Leon on October 04, 2012, 02:33:57 AM
Also - sorry if I'm missing something - people seem to be comparing the new encoding to the original FW at CBR 1.0x, but would it not be much more appropriate to be comparing it to CBR 3.0x / 2.0x or whatever the card can manage?  (We already know that CBR 2.0x is better than CBR 1.0x.)  Can people see a difference between the new encoding and "normal" CBR 3.0x?

Leon - one problem with trying to compare CBR 3.0x with the new approach is that CBR 3.0x will fail/stop as soon as the picture becomes suitably complex - which is exactly when the high bitrate will have the most benefit. On a static (no motion), low light frame, I am confident that the new approach is at least as good as CBR 3.0x. I was never able to consistently record CBR 3.0x on a complex daylight shot. The highest reliable rate (with no sound) was around 2.3x or 2.4x for me.

So, for me, not only is this new bitrate approach getting superior picture quality - a subjective comparison to what I could get with 2.3x CBR, and objective compared to Canon FW - but, it is superior in that it is far more stable, allowing for its practical use. The video of the horse stables (link posted in a post of mine above) simply would not have been possible using CBR 3.0x and possibly not even stable at 2.3x. I shot that video without the camera/buffer ever failing in 3 hours of shooting.

Finally, I'll add that the highest average bitrate that I recall seeing with CBR 3.0x or 2.3x was around 95 Mbps. The video that I am shooting with 1%'s enhancement is averaging 120 Mbps consistently in shots that are a little complex. Indoors, I don't think it gets that high, unless you are at a very high ISO. My point is, the quality settings are driving the bitrate that high. They couldn't get that high with CBR, because it would fail.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on October 04, 2012, 05:01:18 AM
It goes Qscale (-16 to 16) - canon parameter then
Slice QP (several values at 1 qscale in CBR, usually 1 value at fixed qscale)
Frame QP (55-10) - depends on above. Slice from X-X is one step of QP.

Only the last one matches x264.

Quotecould we establish a "safe" value for a certain GOP

Unfortunately not really, complexity will always generate a different bit rate in different conditions. The only thing you can consistently measure is data rate/buffer. Sometimes you can push it all the way up and get 40mbps, sometimes you get down to the middle and its still 90 or 100. Also, Slice 87 is above Q-16 and rates still get higher despite Frame QP not going below 10. Its not 1:1 x264 or even H.264 spec. Picqpy/PC should technically go higher than it does and yet camera freezes if set that way. Seems like canon implemented what they could, how they could.

Quoteby incrementing the slice quality value and record TV static noise until it can't cope reliably

The problem here is that static barely gets by at low quality. Everything else will look terrible. The win with this was being able to lower quality to that point before a buffer overrun. Static is only good as a worst possible scenario.

Quote.  Whilst super-high bitrates may be useful sometimes, I won't want to use them all the time.)

I have the same problem because the files get HUGE and fill up the card rather quickly. Kinda limits what you keep vs delete. The only other option is stock "CBR" or VBR.  If you leave debug off you can set slice to 0 and reboot. This will leave it using the X values and prediction.

Turning on debug will let you view what it does for 1 clip but then slice gets reset and next recording will have it overriden + cbr blocked. Leaving it off is just like stock ML except with higher picqpy/PC and settable deblocking filter. You can then play with the different X values and alter how it does prediction.

Also lengthening the gop increases the # of P frames which gives you more compression at the expense of keeping frames in the buffer longer and a little quality loss.

QuoteConstant Rate Factor in x264

Canon couldn't implement it and still can't. The best they did was a way more conservative version of this approach except based on frame prediction. I don't think their hardware encoder is capable of staying within X bit rate without just dumping quality till it calms down. 5DIII may be different but DigicIV doesn't seem to have CRF capability.

The profiles may be bullshit themselves too as we are super violating the baseline spec with all of these mods. The player can neither handle the higher bit rates or increased chroma/luma values... as seen by the undulating pixelated colors on playback. The higher profile on 5dIII may simply be an implementation of B frames and ability to set higher frame QP values.

Speaking of that, the only other real tweak would be to push that number past 10 but I think its set internally or by the Jpcore values which I haven't understood yet. I have seen Frame Q of 51 and its terrible, 10 is this good, 0 or 1 or 2 should be huge but perfect.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Leon on October 05, 2012, 12:13:14 AM
QuoteCBR 3.0x will fail/stop as soon as the picture becomes suitably complex

Sorry, I should have said: CBR 3.0x with GOP set at 3 or whatever GOP makes it most stable.  That would be interesting to compare, because it might not be significantly better than high CBR values with GOP, deblocking, etc, tweaked.

Quote from: 1% on October 04, 2012, 05:01:18 AM
It goes Qscale (-16 to 16) - canon parameter then
Slice QP (several values at 1 qscale in CBR, usually 1 value at fixed qscale)
Frame QP (55-10) - depends on above. Slice from X-X is one step of QP.

So, are you saying you can adjust each of these three independently without changing the value of the other two?  I had been assuming that Qscale -16 to 16 was just part of the standard CQP 0-51 scale taken and rescaled to be over -16 to 16.  No?

Quote0 or 1 or 2 should be huge but perfect.

lol, yeah, let's not go there.  You'd be way better off dumping raw 422 data by that point!

BTW, is A1ex's MJPEG any nearer public light?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on October 05, 2012, 04:45:04 AM
Mjpeg is getting there slowly.

They are all independent values but qscale sets -> sliceqpd and that determines the frame QP value. Might as well forget qscale as it can only be adjusted by 1 and is fairly limited. Direct control over frame QP would be nice or mapping slice values to frame qp. There are I think several slice values per 1 CQP, would have to test this more.

QuoteYou'd be way better off dumping raw 422 data by that point!

Not feasable. :( any compression is good or would have been done. But I'm sure some scenes can handle values <10.

QuoteCBR 3.0x

Cbr doesn't mean anything, it just sets slice qp based on prediction, nothing else. It took 1 or 2 NOP'ed calls to neuter it completely. Slice 112 or 115 is something like Qscale -16/cbr 3.0x, etc.

Its not even that good at stopping the buffer overruns.

What I can do is make the buffer save customizeable so it stops raising quality at like X mbps and will drop 2, 3, and 4 slices at X values. Would help people with slower cards or if you want more compression/more dropping/less dropping, etc.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on October 06, 2012, 07:51:46 PM
New buffer save and bigger audio buffers.

https://bitbucket.org/OtherOnePercent/tragic-lantern/downloads/autoexec.bin.600D.NewBuff.BiggerAudBuffer
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Andy600 on October 06, 2012, 08:47:59 PM
Hi 1%,

Thanks as ever for your work!

I'm a bit confused by the additional slice menu. Do we need all these extra settings to be user adjustable? I like the idea of the min/max BR setting. For me its a case of the simpler the better.

I would be happy with locking DBlock at 0 and just having GOP and BR min/max settings being user adjustable. I confess I still don't fully understand slices etc

A

Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on October 06, 2012, 09:19:56 PM
Below Min BR is where it starts to raise quality by 1. Max is about where it will start dropping by 1. Both of these run every second to try to smooth things out before it gets to the warning level.

If the buffer warning goes off it drops quality by 1 at "drop by 1 bit rate" and by 3 at the drop by 3 rate.

Point of adjusting it is so that you can customize it to your card. I can now use it on my slow card too.
I actually wanted to let people choose min/max slice but only implemented that in the config file.

Slices are simple, its just like qscale. Lower values mean higher quality, higher values mean lower quality. You don't really have to mess with the slice setting besides 0 and non 0.

I can try for less settings but then people with slower/faster cards are out of luck. The way it was going before it would raise to 87 immediately and then the buffer would fill again and it was an up/down yoyo on really difficult scenes.

Really I should make a guide at some point once its all good and tested. The big question though is if quality has dropped since the functions are less aggressive.

Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 3pointedit on October 08, 2012, 02:53:57 AM
If allowing many parameters is used to suit individual cards, could the camera benchmark a card and set quality to best capacity?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on October 08, 2012, 08:35:51 PM
Things change between all-i and gop3, also I'm not confident enough to just run a test and call it a day. Cards degrade over time until reformatted. I haven't done mine until recently and it was barely doing 80mbps from all of the testing. Its just 4 settings in a menu. Before we can do something so finite I'll have to do more testing and so will others. I don't know if what I did is better on video, just better at stopping overruns.

Also noticed: br indicator for all I seems off by a bunch.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Rush on October 08, 2012, 09:37:20 PM
I am sorry, if I want to turn off bitrate autocontrol in latest builds - should I just set Slice to 0?
I was playing with settings and faced problem with low bitrate blockiness in the middle of recorded video, so I want to switch to old method of bitrate control.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on October 08, 2012, 09:58:57 PM
Yes that would work. I'm working on fixing that actually. I don't want to use that -128. I think its good for stopping overages but causes dreaded blockiness in the middle.

* this is also the reason why I  need to put that min/max slice

https://bitbucket.org/OtherOnePercent/tragic-lantern/downloads/autoexec.bin.SliceMin.6

This should have indicators under control and no longer go to "-128". See if its still blocky.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Rush on October 08, 2012, 10:53:47 PM
Thanks for reply, 1%.
I changed Slice to 0, but when I start recording it still tries to avoid buffer fill and lower bitrate to blocky image. And Slice becomes non zero value.
Is there any other way to disable bitrate autocontrol?
I would like to have old manually controllable bitrate with buffer overflows, to be sure that my shots will not suffer bitrate drops.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on October 09, 2012, 12:07:07 AM
It will change to non 0 if you turn debug on.

If you leave debug off CBR prediction works.

So set to 0, reboot, leave debug off.
I named the debug variable the same as the variable that sets it. Have to fix that and turn off showing I/P, etc when in slice mode too.

What slice does it get really blocky at? I'm hoping to fix the blockiness so need input. Maybe add constant slice like qscale mode too.


If you hate buffer save, you can now record at fixed quality buffers be damned. Also fixed that debug setting slice in cbr issue. Still working on getting wav to be skip free.

****Canon CBR fixed for real. It was still going back to slices after a little while. I couldn't tell until I put the indicator.
https://bitbucket.org/OtherOnePercent/tragic-lantern/downloads/autoexec.bin-10-11-12
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: JasonATL on October 12, 2012, 04:20:02 PM
1% - I just wanted to put a note of thanks (again) here. I sincerely appreciate your continued work on this.

I have had limited time in the last couple of weeks to do more testing/experimenting, but hope to do more soon.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: aace on October 18, 2012, 02:13:31 AM
I'm very interested in this. I've been using the nightly builds. To use this am I replacing the autoexe.bin file with these or should I just wait for it to be implemented into a main release? I'd like to try this but not at the expense of my camera.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on October 18, 2012, 02:52:06 AM
Haven't broken my camera yet.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: AriLG on October 18, 2012, 12:13:08 PM
Quote from: aace on October 18, 2012, 02:13:31 AM
I'm very interested in this. I've been using the nightly builds. To use this am I replacing the autoexe.bin file with these or should I just wait for it to be implemented into a main release? I'd like to try this but not at the expense of my camera.
It's only for 600D, IIUC (or not ?... :))
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on October 18, 2012, 04:40:06 PM
I've released a couple of non-600d builds but the good stuff is for the camera I have + have firmware to.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: reddono on October 18, 2012, 05:52:49 PM
Quote from: 1% on October 18, 2012, 04:40:06 PM
I've released a couple of non-600d builds but the good stuff is for the camera I have + have firmware to.


Sorry, does this mean that the magic lantern team doenst believe in your experiment? I dont see the reason for not include your work in the night build...
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on October 18, 2012, 06:14:27 PM
I've never made a pull request.  Also use a lot of cache hacks. Few other cameras break 4gb limit. I also use movie remap. These things probably keep it from being integrated. I just pull ML changes to my code tree so its not like you're missing anything official. Also playback issues in camera since videos violate baseline spec on gop, bitrate and picqpy/pc.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: nanomad on October 18, 2012, 06:20:00 PM
Cache hacks are the primary reason. They may cause unwanted side effects :/
That being said, the only cameras this hack will work decently are the 1100D, the 600D and (possibily) the new Digic V. The other lack the ability to overcome 4Gigs of footage
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on October 18, 2012, 06:39:13 PM
If MJPEG gets working untethered (right now makes 0 size jpegs that don't show in ram) at movie YUV sizes H.264 will be obsolete anyway. Camera has AVI player in fw and it runs all the time. Then if separate wav stops stretching/skipping we have a full replacement for canon's consumerish/gutted implementation. Also dunno how fast it updates, LV fps > recording fps it seems. On eos utiity its a little jerky but that's because its going over USB. Copying the pics might give 60FPS in "1080"
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: CaptainHook on October 26, 2012, 11:47:47 PM
Hey guys, is there any other place MJPEG is discussed on this forum, or is it just in this thread? I've read quite a bit of the thread but not all of it..
Is it discussed anywhere why Alex isn't that interested in higher bitrates etc (deeper in this thread maybe)? Is it because MJPEG is a better option?
We have 2 5Dmk3's (and a mk2) and i'm really interested to know what might be possible to improve the video quality. Alpha 2 is really great so far though.
Thanks for all the hard work! :)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Rush on October 26, 2012, 11:53:23 PM
CaptainHook, Topic: (M)JPEG encoder (http://www.magiclantern.fm/forum/index.php?topic=2803.0)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: CaptainHook on October 27, 2012, 12:13:20 AM
Thank you!
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: skydragon on November 07, 2012, 05:26:37 PM
QuoteReally I should make a guide at some point once its all good and tested.

1% - Is there any way you could write up a quick user guide in this thread, or at least a basic settings template to use that gave maximum video quality, along with a few comments as to what does what.

I downloaded your dev work, but couldn't make out what to do in the UI, so stopped ;-)  I'd really like to try the All-I mode, or whatever you recommend as best from your tests.

Reason i ask is I'd really like to try and improve the quality of video on my 600D. I've tried doing some tests with ML 2.3 and I find it very difficult to see any difference between CBR x1.0 and CBR x 2.8. I've tried putting screen grabs side by side and doing zoom in's but there appears to be little or no difference between the two bitrates in use (which is not what I was expecting).

Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on November 07, 2012, 06:17:46 PM
QuoteHere is an explanation of the settings:


From the top:

Canon CBR: All of the numbers that make up Canon Predictive CBR. Set all to 1 x value or alter as you see fit. See what QPs show up in the files.

Slice CBR - Bit rate based on frame quality.

Jp Slice - The "Quality" of each frame
Lock Slice - Disabled/Enabled - Set slice to "JP slice" value and don't change it.


Taper rates - Enabled/Disabled. When bit rate is lower than min BR raise slice by 1. When BR is above Max BR drop slice by one. Hence, taper the rate to where you want.

Drop By 1 - When buffer starts overflowing.. as set by BuffWarnLevel drop slice by 1 at this rate until it stops.
Drop by 3 - When buffer starts overflowing drop by 3 above this bit rate, its an emergency!

Old Bit rate menu:

CBR - Use slice CBR or canon "cbr". VBR, use the qscale value

DblockA/B - Set deblocking filter. -6,-6 is off, you can look this up its just like H264

PicPC- Raise chroma offset. Negative values are better. Caveat: at 0 you raise luma offset which is technically  up a level from stock. You can't raise both or camera freezes so its  made this way.

Gop- "flush rate", group of pictures... 1 is all I, 3 is a good mix. 4 is being used on 7d. Raising it higher creates more predictive frames which is better on compression but harder on your buffer.

Debug- Show values of slice or canon CBR on the screen.
Averaging - Scale average values in canon CBR - they go higher.

You can use Canon CBR, Slice CBR or Qscale to control rates. The settings for each mode work in that mode. So changing X values in slice or qscale will do nothing.


What I'm using: MinBR :80 Max:125
Drop By 1, 85, drop by 3 100, tapering enabled. I turn off deblocking filter and leave picqPC at 0.
I mostly shoot gop3. Gop 1 is nice but rates are very high and writing is pretty much continuous.


MaX quality: Slice Lock: Enabled, Slice 87, picPC :0

This will shoot a video of 100% QP10 frames if your card can take it.



You won't really notice if you're not doing post and delivering to youtube. If you're mixing with "pro" footage, grading, and delivering to Bluray or theater you will.

Some ISOs produce X bit rate at max quality, you can't artificially squeeze more data out. You're not getting 200mbps at ISO 80 unless the scene is very complex.

Buffer warning level is very important. I set it at 50%. In gop 3 or 1 if buffer climbs above like 30% camera is starting to get behind in writing out the frames.

Hope this makes some sense.




Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: skydragon on November 07, 2012, 08:02:12 PM
I've just done a (non-scientific) test.

I arranged some objects, lit them with a LED panel and using a 600D on a fluid head tripod, with a sandisk 95MB/sec SD card and using ISO160, 1/50th, f6.4 i shot some video. I used a neutral picture profile with in-camera sharpening turned down to minimum.

Firstly here is a screenshot from the scene i shot,  (apologies I just gathered some objects together quickly to use)

(http://farm9.staticflickr.com/8478/8164633298_df77135de1_b.jpg)

Firstly I shot a video panning slowly from right to left (about a six second pan) using ML 2.3 and CBR x1.8. The resulting video was an average bitrate of 61678kbps

Then using 1%'s version of ML, I shot another video using the exact same conditions and panning at a very similar speed, but at All-I and an average bitrate of 103320 kbps.

On playback afterwards, on a 23" PC monitor (frustratingly) I could not see any visual difference between the two clips.

I then did a screen capture of a mid point frame from each video and in photoshop zoomed to 300% and compared a similar area from each video, to see if closer scrutiny would reveal a difference.

The ML2.3 CBR x1.8 is on the left, the 1% ML is on the right. I can't really see much or any appreciable difference.

(http://farm9.staticflickr.com/8208/8164582479_0cf072e317_b.jpg)

I'm really puzzled why cranking the bitrate up and making the capture All-I appears to make no perceivable viewing difference to the video quality...

NB; I'm not saying that in other lighting conditions or scenes there might not be benefits in having a higher bitrate or All-I, but I'm struggling to see how to improve 600D video for 'general' usage.

Is it perhaps that the (less than 1080HD) resolution used internally within the 600D before up-scaling to 1080 is the quality bottleneck....or is it that the 600D's H264 encoder is at or near it's quality limit and not much more can be squeezed out of it.

Any ideas?

Thanks,
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on November 07, 2012, 08:30:26 PM
You encode X at the best quality, you still have X. Compare with silent pic of the YUV buffer (big one, not the small 1024 one). Upscaling is one factor, limited real resolution is another. We're only messing with the encoder, not input.

Canon CBR dropping quality on scenes to keep it around 40-60mbps is where it gets worse, this kinda prevents that. Try QP of 127 (or higher) locked and you'll see a difference. With larger sized frames and weaker compression the grading gets a little bit better or at least easier. The only other things to do to improve quality are 1:1 sizing to kill the upscaling or 4:2:2 color. Encoder is as good as it gets unless we make it record at like QP0 or something... even then its still going to look like the YUV frame.

Also, the point of compression is to use visual tricks to make the difference not noticeable. Like a 320kb/s MP3 vs a 128kbps one can you really tell THAT much of a difference? But in a normal workflow you'll re-compress again and its better to start with more data than less, even if you end up throwing some of it away.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: skydragon on November 07, 2012, 09:17:19 PM
Thanks 1%. Understood that a higher bitrate may mean easier post.

Does this all mean that irrespective of encoding tweaks, the internal resolution of the 600D will mean that this is as good as video quality will get?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on November 07, 2012, 09:43:52 PM
For now this is where its at. Better than everything except 5DIII and dedicated video cameras.

Also, one thing I left out: Set slice to 0 to use canon CBR.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: skydragon on November 09, 2012, 09:57:33 PM
QuoteBetter than everything except 5DIII and dedicated video cameras

Just talking about pure 1080P HD Video quality (not build quality or camera ergonomics or features) don't you think a hacked Panasonic GH2 can produce 'better' HD video that an EOS currently can ?

I know the 'soft' video look a Canon EOS camera produces can be very nice for some uses, but at the end of the day although the EOS file output is 1080p, you could argue it isn't actually a 1080 HD video at all, due to the internal camera resolutions and upscaling etc.

I love my 600D but really wish there was a way of getting better sharper, higher resolution video out of it...
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: hjfilmspeed on November 13, 2012, 09:54:16 PM
Quote from: skydragon on November 09, 2012, 09:57:33 PM
Just talking about pure 1080P HD Video quality (not build quality or camera ergonomics or features) don't you think a hacked Panasonic GH2 can produce 'better' HD video that an EOS currently can ?

I know the 'soft' video look a Canon EOS camera produces can be very nice for some uses, but at the end of the day although the EOS file output is 1080p, you could argue it isn't actually a 1080 HD video at all, due to the internal camera resolutions and upscaling etc.

I love my 600D but really wish there was a way of getting better sharper, higher resolution video out of it...

This topic would be good to read up on:  http://www.magiclantern.fm/forum/index.php?topic=3492.0
and this one too: http://www.magiclantern.fm/forum/index.php?topic=3489.0
At this point i totally agree. I just want want amazing detail from the canon dslr video. I dont think its gunna happen though for under $7k from canon.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 3pointedit on November 14, 2012, 10:08:30 AM
Ha, but it can happen from a sub $1k sony handycam...
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: skydragon on November 15, 2012, 05:35:40 PM
I'm sad to say I've now defected over to a Panasonic GH2 ;-)

The build quality is poor compared to a Canon, but the 1080p 24fps AVCHD video, hacked to >60Mbps is simply amazing quality. Comparing it side-by-side with a friends 5d MkII or my 600D it's a massive improvement in resolution, not just a subtle change.

...I'll come back to Canon as soon as they make a DSLR with proper 1080p video rather than the current 'soft' upscaled offering.

I really do miss the great features that ML provides, it really does make using a DSLR so much easier and it's one of the downsides of using a GH2 rather than an EOS.

Hope to return back to Canon, in a few canon model generations time, if and when they get their act together ;-)

in the meantime thanks and best wishes to the guys at ML.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on November 18, 2012, 05:59:49 PM
Should compare slice 87 600D vs GH2, would be interesting to see. I thought GH2 uprezzed too but maybe that was only GH1.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: KahL on November 21, 2012, 06:27:20 AM
Crap, I am SO lost, guys. How do I setup and install this version?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: driftwood on November 21, 2012, 04:53:07 PM
Seems to be some good work going on here. Where are we at in terms of encoder patching? I could be of some help - haven't ML'd my 5DmkII , 7D or 60D yet but you never know...

The 5DMKII, 7D 60D are h264 Profile IDC baseline level 5 which isn't a brilliant starting point... but according to the spec allows a maximum video bitrate of 135,000. Max decoded picture buffer of 110,400 at 1920x1080. (Having said that, if you look at the h264 stream / PPS there could the usual baseline 66 constraints on these three cameras)

Have you found a way of raising the decoder picture buffer yet (dpb/cpb)? Youve got to stop the buffer underuns/overruns and tests need to be done on buffer analysis / stream analaysis software (like elecard or CodecVisa, etc...)

Have any Quantisation scaling matrices been found ? Are they being employed?

With the 5DMKIII being level 5.1 High profile (thats slightly higher than the new Pany GH3 you should be able to do something. You've got adaptive 8x8/4x4 transform, scaling tables, cabac or cavlc, in-loop deb locking etc...

Here's the stock bitrates of the GH3 in ALL-I 72Mbps .mov mode.

Bit Rate = 71680000 (69999 = bit_rate_value_minus1/SchedSelIdx/*0*...)
CPB Size (the coded picture buffer) = 64512000 (62999 = cpb_size_value_minus1(SchedSelIdx/*0*...)

Intitial QP=20/ QP=20...

Actually, someone email me a very small All-I file / dropbox me. [email protected]

Hmmm... need to get hold of a MKIII.



Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: hjfilmspeed on November 21, 2012, 07:13:52 PM
Wow is this the Drifwood that made the gh2 hacks hard round the world? (Not sarcastic I'm genuinely excited)
I'd be more then happy to send u a clip when I get a chance. I'm desperate for better sharper video from the mark III at this point!
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: CaptainHook on November 21, 2012, 07:18:48 PM
It would be pretty epic if Driftwood started to help out with ML !!
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: a1ex on November 21, 2012, 07:32:08 PM
Driftwood from this video? http://vimeo.com/17722077 :)

For 5D Mark III, this post (http://www.magiclantern.fm/forum/index.php?topic=1565.msg11808#msg11808) summarizes our findings pretty well.

Default encoder settings:
H264-alli.ini (http://a1ex.magiclantern.fm/bleeding-edge/5D3/H264-alli.ini)
H264-ipb.ini (http://a1ex.magiclantern.fm/bleeding-edge/5D3/H264-ipb.ini)

(not sure what all these things mean, some can be changed, others give err70)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Shizuka on November 21, 2012, 07:34:33 PM
Quote from: driftwood on November 21, 2012, 04:53:07 PM
Seems to be some good work going on here. Where are we at in terms of encoder patching? I could be of some help - haven't ML'd my 5DmkII , 7D or 60D yet but you never know...

The 5DMKII, 7D 60D are h264 Profile IDC baseline level 5 which isn't a brilliant starting point... but according to the spec allows a maximum video bitrate of 135,000. Max decoded picture buffer of 110,400 at 1920x1080. (Having said that, if you look at the h264 stream / PPS there could the usual baseline 66 constraints on these three cameras)

Have you found a way of raising the decoder picture buffer yet (dpb/cpb)? Youve got to stop the buffer underuns/overruns and tests need to be done on buffer analysis / stream analaysis software (like elecard or CodecVisa, etc...)

Have any Quantisation scaling matrices been found ? Are they being employed?

With the 5DMKIII being level 5.1 High profile (thats slightly higher than the new Pany GH3 you should be able to do something. You've got adaptive 8x8/4x4 transform, scaling tables, cabac or cavlc, in-loop deb locking etc...

Here's the stock bitrates of the GH3 in ALL-I 72Mbps .mov mode.

Bit Rate = 71680000 (69999 = bit_rate_value_minus1/SchedSelIdx/*0*...)
CPB Size (the coded picture buffer) = 64512000 (62999 = cpb_size_value_minus1(SchedSelIdx/*0*...)

Intitial QP=20/ QP=20...

Actually, someone email me a very small All-I file / dropbox me. [email protected]

Hmmm... need to get hold of a MKIII.

For the non-5D3 cameras, there's nothing that really can be done:
QSM isn't available in baseline profile, and maxDPB is just a specification specifying how many reference frames can be used for video. For canon's encoder, it only uses the previous frame to be predicted from, ie. 1 reference frame.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on November 21, 2012, 07:44:10 PM
I don't know how far they actually implement the profiles. Seems like they just add features the profile should have and call it a day. The bit rates and some other stuff  violate baseline.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Kabuto1138 on November 21, 2012, 10:20:52 PM
So, is high bitrates for the 5d2 etc. officially not happening?

Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: hjfilmspeed on November 22, 2012, 12:51:18 AM
Driftwood and ML! shhhoooooooo weeeee this is exciting!

Is it possible to have bit rate options on the 5d3 alpha versions that run off the card? i like the idea of running my off the card. seems allot safer.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Audionut on November 22, 2012, 06:12:41 AM
Quote from: a1ex on November 21, 2012, 07:32:08 PM
Default encoder settings:
H264-alli.ini (http://a1ex.magiclantern.fm/bleeding-edge/5D3/H264-alli.ini)
H264-ipb.ini (http://a1ex.magiclantern.fm/bleeding-edge/5D3/H264-ipb.ini)

(not sure what all these things mean, some can be changed, others give err70)

"Transform8x8Flag = 2"  8x8 DCT increases quality and implies High Profile.  God only knows what that flag changes though.  Would need to adjust the setting and analyze the encoded video.

"EntropyCodingMode = 0"  CABAC.  http://en.wikipedia.org/wiki/Entropy_encoding

"IntraPicInitQP = 20 - InterPicInitQP = 20"  Initial encoded QP?  Lower values would increase quality but increase stress on the buffer.

"QpOffsetForB = 0"  QP offset for B-frames?  http://mewiki.project357.com/wiki/X264_Settings#pbratio

"MinQpI = 10 - MinQpP = 10 - MinQpB = 10 - MaxQpI = 51 - MaxQpP = 51 - MaxQpB = 51"  Min and Max Quantizer to use for encoding.  MinQP should be reduced to 0, but might put to much stress on the buffer.  MaxQP should be left alone.  MinQP could be reduced to 0 if used in conjunction with,
"MinBitrate = 0 - MaxBitrate = 0"

"RateControlEnable = 2"  Different rate control methods?  Again, would need to adjust and analyze the encoded video.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: driftwood on November 22, 2012, 02:40:09 PM
Yes, its Driftwood from GH2 settings, anyone gotta problem with that!?

Yeah, judging by baseline 66 work I did on the GX1 camera there's not much one can do with any of the canons prior to the 5DmkIII - sure you can get higher bitrates and change GOP but most baseline profiles are constrained. See constrain flags in the h264 SPS set.

So the 5Dmk3 has distinct possibilities: Gonna have to sell the other canons to buy one ;-)


Level 5 mode:

I would perhaps leave the initial QPs alone - anything lower than 16 causes problems. There is also minimum QP for each P_slice type at 10 - the GH3 has something similar I believe. Subsequently, in these recording modes a scaling matrix is not being used/implemented. As a whole Scalers work best at coding the luma frequencies better - ring artefacts etc... but Canon's encoder interp. probably deems them unneccesary. Are the recordings when analysed in SPS/PPS using QP_delta offsets like =0 (no change in QP from previous prediction) or other values (e.g.-4, -6, -8, -10) that indicate a change of QP?

QpOffsetForB = 0 For B frames no QP offset variation is being employed?


Transform8x8Flag = 2 probably means adaptive 8x8/4x4 transform on residuals - ie most of the picture is coded with 8x8 and some of the luma residuals are receiving 4x4. My 'hunch' is if it were =0 all luma residual samples are receiving 4x4 block transform, or =1 are luma samples are in residual 8x8 block transform. I would leave that alone if this was confirmed as the case. A quick check with stream analysis will confirm transform being used.

EntropyCodingMode = 0
You need to establish the order of the setting e.g. does = 0 mean cavlc, 1= cabac, 3= Exp Golomb vlc - one or the other. (therefore probably 0, 1 or 2 are the available settings)

RateControlEnable = 2 - needs experimenting (first with =0, then =1) Could be some sort of adaptive rate control schemes. Could be any of the following:-

=0 - Original quadratic rate control scheme based on JVT spec
=1 - Extension of quadratic scheme for all Intra and IBsBsBs... coding.
=2 - Basic extension of quadratic scheme to better support hierarchical codingstructures
=3 - Extension of quadratic scheme with slice type separation

-------
Level: 4.1 modes:

IntraPicInitQP = 28
InterPicInitQP = 28

I see no reason why these cant be improved (changed to =20 as a starting point )  with tests on added bitrate to correct rate control.

I really need the developer to find a patch in their firmware/ROM/RAM loadables for coded picture buffer (surely there's something in there?) as just adding bitrate requires more buffer bitrate (whats the ceiling?) In the GH2 ptools we have it as a 'bottom' setting, 'Top' setting being bitrate. Also anything regarding Frame buffers (and sizes), and frame size in bits?

Can someone load up a 5dmk3 PPS produced from elecard or any other h264 stream analysis software?
And can someone please email me a few small sized original .mov recordings from the mk3?

more to come...
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: hjfilmspeed on November 22, 2012, 06:01:28 PM
ill send ya some stuff. just shot a bunch.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: hjfilmspeed on November 22, 2012, 06:30:31 PM
ok driftwood just sent you a bunch of clips but i have no idea if they went through. they are probably to big for email.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: driftwood on November 22, 2012, 08:01:27 PM
You can always upload to a file share site or use dropbox to share sample files.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: hjfilmspeed on November 22, 2012, 10:00:28 PM
Quote from: driftwood on November 22, 2012, 08:01:27 PM
You can always upload to a file share site or use dropbox to share sample files.

Did you get em? sent them here.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: driftwood on November 22, 2012, 10:26:16 PM
Not yet  - do you have my correct email? Link me. Itd be easier.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on November 23, 2012, 12:36:48 AM
Quotethere's not much one can do with any of the canons prior to the 5DmkIII

Its not really so much baseline as my build already violates it according to avi-naptic. Its that all you can set in the older canons is gop, slice quality, de-blocking filter and qpy and qpc offset. There is a set of parameters sent directly to the "JPCORE" encoder but I haven't been able to figure them out, its just a bunch of numbers put into registers.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: hjfilmspeed on November 23, 2012, 03:56:24 AM
Quote from: driftwood on November 22, 2012, 10:26:16 PM
Not yet  - do you have my correct email? Link me. Itd be easier.

[email protected]

Is this correct?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: hjfilmspeed on November 23, 2012, 04:04:18 AM
just sent you a dropbox with 1080 30p all-i and ipb 1080 24p all-i and ipb 720 60p all-i and ipb Mr Driftwood. should be 6 clips when there done syncing
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: driftwood on November 23, 2012, 12:25:36 PM
ok thanks
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: DTSET123 on November 24, 2012, 09:57:08 AM
Driftwood- I'm so glad see you join the game  :D
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Marvin on November 24, 2012, 11:59:35 AM
@driftwood

please take a look at this post:
http://www.magiclantern.fm/forum/index.php?topic=1565.msg11808#msg11808

some tests have already been done, might save you some time :)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: driftwood on November 24, 2012, 01:31:20 PM
Good work Marvin:

Can you experiment with the two patches ratecontrol= in ciombination with maxbitrate / minbitrate ?

Change ratecontrol value to =1 then =0 together with entering values into min/max bitrate.

I'm still looking for a value for cpb ncoded picture buffer/dpb. Maybe Maxbitrate is like GH2's Top settings (declared bitrate) and Minbitrate could be a reinterpret of 'Bottom' settings which we worked out to be the CPB size.

We need higher coded picture buffer settings for adjusting the bitrate higher - ie they work in unison.

Failing this, somewhere there must be something in the code that declares a cpb. We need to find it.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Marvin on November 26, 2012, 05:32:25 PM
Quote from: driftwood on November 24, 2012, 01:31:20 PM
Good work Marvin:

Can you experiment with the two patches ratecontrol= in ciombination with maxbitrate / minbitrate ?

Change ratecontrol value to =1 then =0 together with entering values into min/max bitrate.

I'm still looking for a value for cpb ncoded picture buffer/dpb. Maybe Maxbitrate is like GH2's Top settings (declared bitrate) and Minbitrate could be a reinterpret of 'Bottom' settings which we worked out to be the CPB size.

We need higher coded picture buffer settings for adjusting the bitrate higher - ie they work in unison.

Failing this, somewhere there must be something in the code that declares a cpb. We need to find it.

both resulted in ERR70 when i tried
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: driftwood on November 27, 2012, 04:50:54 AM
Together with testing out the new GH3 (love it!) I'm analysing 5dmkiii mov files to rip out SPS, etc... report back here soon.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on December 09, 2012, 03:38:09 AM
Found a side effect of gop changing.

Canon has a sync function when recording with audio enabled. If gop is changed sync makes an overrun error and then error 70s the camera. So if your recordings w/audio are erroring, turn gop back to default of 12 or 15 or 30 depending on fps. Don't know if can be patched out yet. Maybe also related to separate wav sync issues? Does this apply to 7d too?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: driftwood on December 13, 2012, 03:38:02 AM
I havent forgotten this thread. I hope to have a MKIII in my hands v soon to test.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: wenen on December 13, 2012, 11:37:01 AM
Quote from: 1% on December 09, 2012, 03:38:09 AM
Found a side effect of gop changing.

Canon has a sync function when recording with audio enabled. If gop is changed sync makes an overrun error and then error 70s the camera. So if your recordings w/audio are erroring, turn gop back to default of 12 or 15 or 30 depending on fps. Don't know if can be patched out yet. Maybe also related to separate wav sync issues? Does this apply to 7d too?

So what settings do you recommend for the 600D now?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: DTSET123 on December 13, 2012, 01:43:29 PM
@driftwood
Saw your short GH3 vs GH2 movie- funniest thing i have ever seen! Actors did a good job, well done!
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on December 13, 2012, 04:10:44 PM
[quote[So what settings do you recommend for the 600D now?[/quote[

If you're shooting built in audio. You have to go back to stock gop. If you're shooting just video with audio off you can use whatever you want like gop 3, etc.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: driftwood on December 18, 2012, 03:03:45 AM
Ok Here's a look at the h264 encoder found inside the MKIII in-depth. Thanks to hjfilmspeed for supplying the test files.

First and foremost the encoder uses only 4x4 transform (along with the standard 16x16) and employs High Profile Level 5.1 for All-I and High Profile Level 4.1 for IPB. Everything is encoded using CAVLC (All-I and IPB) and NO scaling matrices are found to be used if my analysis using Elecard is correct.

Contrary to the Canon 5DMKiii encoder, the new Panasonic GH3 encoder uses Adaptive 8x8 and 4x4 transform and is thus 12% more efficient. It's All-I rec modes chooses High Profile lvl 5.0 CAVLC to encode - IPB employs CABAC High Profile level 4.1. All the GH3 modes use the same scaling matrix algorithms. So you can see there are marked differences between the GH3 and the 5DMK3.

I will attach graphs and sequence parameter sets for each of the Canon 5DMKIII modes shortly.

The dropbox link to the folder/files is;-

https://www.dropbox.com/sh/zqayfoniqo1s6dz/GO9zr30AMb

if you can't see the dropbox files for some reason (though Im sure sharing the link should work) email me at [email protected]
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: wenen on December 19, 2012, 01:05:36 PM
Quote from: 1% on December 13, 2012, 04:10:44 PM
[quote[So what settings do you recommend for the 600D now?[/quote[

If you're shooting built in audio. You have to go back to stock gop. If you're shooting just video with audio off you can use whatever you want like gop 3, etc.

Btw one more question: Do you have any new setting for the 600D to max out the quality? Have you done any comparison to the GH2?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: hjfilmspeed on December 19, 2012, 09:21:04 PM
Ha Ha Glad to be of assistance drifftwood (tho i feel i hardly did anything compare to what all you guys are doing). I know nothing about how all this works but i do know a little bit about video and cinema. if theres anything else i can do to push forward the quality of video from the 5d3 or the all-i codec in general let me know!
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on December 20, 2012, 12:58:51 AM
QuoteDo you have any new setting for the 600D to max out the quality

I have no GH2 (so can't compare) but Q87 in crop mode is camera max. Can't really do anything else to improve quality, just need a fast card to handle what is there. The only other improvements won't be H.264. As shown above, canon cameras are missing most encoder tweaks/settings even on 5dIII/6D/M.  With UHS-1 on 6D and similar slice QPD functions we'll see what DigicV can do.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: wenen on December 20, 2012, 12:35:39 PM
Quote from: 1% on December 20, 2012, 12:58:51 AM
I have no GH2 (so can't compare) but Q87 in crop mode is camera max. Can't really do anything else to improve quality, just need a fast card to handle what is there. The only other improvements won't be H.264. As shown above, canon cameras are missing most encoder tweaks/settings even on 5dIII/6D/M.  With UHS-1 on 6D and similar slice QPD functions we'll see what DigicV can do.

Ok I have my camera at Q87 and gop1 now. Shuld I use CBR 3.0x or VRB -16 settings?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on December 20, 2012, 09:13:45 PM
CBR and VBR and quality by slice are all mutually exclusive. CBR3x will just increase frame prediction sizes and hopefully pick a better quality. You want lock slice and slice set to 87 in "CBR" mode, the x won't matter (just keep it over 1x), it is not used. Old CBR only gets used when slice is 0. If you see Q scale indicators on the screen then its in CBR or VBR. if you see slice then you're in slice mode.
Q87 is above qscale -16 (this is like slice 112)

There is a guide a few posts back on what the settings do.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: wenen on December 21, 2012, 02:10:10 PM
Watching my difference test.

200% zoom

(http://img209.imageshack.us/img209/3728/mvi66670000013.png) (http://imageshack.us/photo/my-images/209/mvi66670000013.png/)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on December 22, 2012, 05:49:00 PM
Yea its not super dramatic. Main difference is when grading you have more robust footage and you get to capture fine details. Also gop1 has different motion blur. Sometimes you max out your scene. YUV buffer isn't modified so its still compressing the same image... just less. Also a couple of P frames sometimes helps like gop3. Tests were done months ago on fine details. You're scene is slightly under so a lot of stuff is going to get crushed and hidden already.

Waiting to try this on the 6D where there is bandwidth to spare and UHS-1. Hoping to crack 250Mbps but we'll see. You also get the other features like named bracketing/remap/etc.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Yoshiyuki Blade on January 04, 2013, 03:05:36 AM
First post on the "new" forums, but I've been following ML for a while and donated a little a while back. Thanks for all your hard work, everyone! I've only recently found this thread and it seems very exciting. I'm curious about where you guys are headed with this now.

The most recent posts seem to focus more on the h.264 side of things, but a couple months ago, there was crazy talk about implementing an mjpeg encoder directly with the raw 422 data, which was wonderful to imagine. It had me thinking about how much goes into the pipeline of image processing at the moment you press that record button. Where do you suppose would be the best place to min/max for video quality?

With my 5D2, I found that the silent pics written while recording (1872x1080) don't really represent the actual resolution being recorded. I've done subjective tests with various resolutions listed in the 422toimage converter xml files and it's probably 1720x974. If bypassing the upscale process to 1080p is possible, that alone would probably help a lot in increasing the overall quality of the video. Less bits is required for the same quality and less resolution to encode frees up more cpu performance. And boy does audio take up a ton of overhead! With audio on, the video bitrate can only go up to the 60 megabits before crashing, but can go up to the 120s with it off. I can record at Q-16 constantly in fairly busy scenes that way. What's the deal with that lol.

I also noticed that the difference between the (converted) raw 422 image and when I downsample it to 420 is very negligible. However, I haven't tested it on bright vivid reds yet, where perceiving the difference in chroma resolution is most apparent. I wonder if trading performance for chroma resolution is worth the extra cost. Anyway, thanks again guys. I'm not literate in programming or understanding the low level stuff that goes on, but I do like to get into the nitty gritty technical details of video as much as possible.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on January 06, 2013, 07:46:32 PM
Canon does use adaptive 8x8

H264.ini: Transform8x8Flag = 2

Also scaling matrices can be chosen.

ScalingMatrices = 0
pScalingMatrixAddr[0] = 0
pScalingMatrixAddr[1] = 0
pScalingMatrixAddr[2] = 0
pScalingMatrixAddr[3] = 0
pScalingMatrixAddr[4] = 0
pScalingMatrixAddr[5] = 0

You just looked at default encoded videos. This needs more investigation.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: feureau on January 18, 2013, 04:50:35 PM
Quote from: 1% on December 09, 2012, 03:38:09 AM
Found a side effect of gop changing.

Canon has a sync function when recording with audio enabled. If gop is changed sync makes an overrun error and then error 70s the camera. So if your recordings w/audio are erroring, turn gop back to default of 12 or 15 or 30 depending on fps. Don't know if can be patched out yet. Maybe also related to separate wav sync issues? Does this apply to 7d too?

I've been playing with flush and GOP on the 7D. I know this much:

When GOP is set to 1, and audio recording turned on, you have to set the flush to match the video framerate. For instance, if you shoot at 24p, the flush has to be set at 24, or it will stop recording after a few moments.

On the ML alpha 2 for 7D, the flush can be set to maximum of 50. If I use 720p and shoot at 50p, with GOP at 1, with audio recording on, it will record up to the full 4 gig.

But when the 720p is at 60, since flush can't be set at 60, with GOP at 1, the recording will fail after a few seconds.

The GOP number doesn't matter much in regards to erroring, but the flush is key.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on January 18, 2013, 05:38:05 PM
On 600D its one setting. Gop is actualy fps/2 per default even on 6D.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: g3gg0 on January 18, 2013, 07:07:42 PM
Quote from: 1% on December 09, 2012, 03:38:09 AM
Found a side effect of gop changing.

Canon has a sync function when recording with audio enabled. If gop is changed sync makes an overrun error and then error 70s the camera. So if your recordings w/audio are erroring, turn gop back to default of 12 or 15 or 30 depending on fps. Don't know if can be patched out yet. Maybe also related to separate wav sync issues? Does this apply to 7d too?

didnt notice this post earlier.
yeah thats the reason why 7D alpha 2 disables audio when video hacks are active.
didnt patch the MOVW_AddAudio routine yet.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on January 19, 2013, 01:18:08 AM
If you figure it out, let me know. I can probably patch the assert but even if I do it might just mute/stop audio anyway.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: hjfilmspeed on January 28, 2013, 08:00:17 PM
At this point canon is starting to make me sad. Have you seen the s35 1080p all-i from the 1dc. its tack sharp!

http://www.eoshd.com/content/9548/bbc-freelance-cameraman-tests-super-35mm-1080p-on-the-canon-1d-c

I feel like if the 1dc is shooting s34 4k with dual digi 5s why cant we get sharper s35 1080p from the single digi 5 cams like the 5d3.  I would so take the s35 crop if it ment sharper 1080. Sorry i know ML cant change the deep deep proporties of canon dslrs but i feel the need to voice this one anyway. I LOVE YOU ML! but i strongly dislike canon (coming from a 5d3 owner)   
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: RenatoPhoto on January 28, 2013, 08:17:58 PM
Quote from: hjfilmspeed on January 28, 2013, 08:00:17 PM
I LOVE YOU ML! but i strongly dislike canon (coming from a 5d3 owner)

Me too!
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: wenen on February 02, 2013, 03:26:41 PM
Quote from: RenatoPhoto on January 28, 2013, 08:17:58 PM
Me too!

Me too :)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: DTSET123 on February 03, 2013, 09:21:48 AM
me too
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on February 03, 2013, 04:41:36 PM
i don't dislike how they made the digicV encoder... I do dislike that they didn't put lower tier features into a more expensive camera. ie crop mode.

I have to try this moire out that everyone is talking about. Time to get out the striped shit because its not doing it to bricks.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 3pointedit on February 03, 2013, 11:13:26 PM
"Time to get out the striped shit..."

Wow you are dedicated, I just use a shirt! ;)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: clkvang on February 06, 2013, 10:04:52 PM
Is there a current release of this firmware for me to test out? Kinda interested to see what the results are for real world video versus the still examples posted on this thread.

Saw a few links on downloading, but not sure which was the current firmware everyone was testing.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on February 06, 2013, 10:54:18 PM
For 600D - top is newest
https://bitbucket.org/OtherOnePercent/tragic-lantern/downloads

6D  - top is newest
https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/downloads

for ml stock just do the nightlies
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Kabuto1138 on February 23, 2013, 02:58:14 AM
Hey guys,

So is this dead?  what happened to Driftwood, i know he was helping out with the 5d3 but haven't heard from him a while. :'(
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on February 23, 2013, 04:06:41 AM
Not dead just H264 is at max on 600D. I could use help from driftwood on figuring out how to play back the 4x4 transform videos, I can't decode them.

Next step is MJPEG with custom buffer sizes to stop moire, etc.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: wenen on February 23, 2013, 12:10:56 PM
Quote from: 1% on February 23, 2013, 04:06:41 AM
Not dead just H264 is at max on 600D. I could use help from driftwood on figuring out how to play back the 4x4 transform videos, I can't decode them.

Next step is MJPEG with custom buffer sizes to stop moire, etc.

Good job! work on :)

Btw what is EOSMovieFixer.exe?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on February 23, 2013, 07:08:36 PM
For fixing atoms on movies over 4gb
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Kabuto1138 on February 25, 2013, 11:35:39 AM
Thanks 1%.  Mjpeg sound pretty awesome. 

All other cameras seem to be getting better Bit Rates through hacks except the Canons.  But Mjpeg would GREAT!
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Rush on February 25, 2013, 11:49:31 AM
Quote from: Kabuto1138 on February 25, 2013, 11:35:39 AMAll other cameras seem to be getting better Bit Rates through hacks except the Canons.  But Mjpeg would GREAT!
Why "except Canon"? Thanks to ML developers we have control over Canon bitrate and we can control GopSize too! It is awesome.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on February 25, 2013, 04:43:42 PM
Well GH2/GH1 - software encoder so easier to change everything.
Nikon - No hacks that I know of, like 24mbit video
Sony - Has some hacks for like 42mbit

I'd say next to GH2 we are pretty good, only held back by the canon resizing.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Kabuto1138 on February 25, 2013, 08:26:47 PM
My apologies,

Didn't mean to saying anything negative about ML.  I don't I would still hace a Canon camera if it wasn't for ML.   

I thought they had bitrate hacks for the nex?  maybe not. 

When I try to increase the bitrate with ML 2.3 it barely does anything, maybe 10 mbs extra? if I'm lucky.  Sometimes recording stops, even with a fast card, I rather keep it at 1.4x just to be safe. 

But I think Mjpeg would amazing.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Rush on February 25, 2013, 09:05:44 PM
Quote from: Kabuto1138 on February 25, 2013, 08:26:47 PM
When I try to increase the bitrate with ML 2.3 it barely does anything, maybe 10 mbs extra? if I'm lucky.  Sometimes recording stops, even with a fast card, I rather keep it at 1.4x just to be safe. 
ML unlocks upper limit for bitrate - it can be at 150 mbits and more. But it have some limits - bad buffer work (g3ggo have some workarounds; turning off audio will help too).
For me, at stock GOP, after turning off sound recording I can squeeze up to 80-100 mbps video (twice more than stock!) to get rid most of compression artefacts which is notable with high ISO.

I am sure you will see the difference: (upscaled 400% crop)
(http://cs301315.userapi.com/v301315431/4511/Bq3-5UOxxLo.jpg)
(http://cs301315.userapi.com/v301315431/451a/j8K5W1SNxU0.jpg)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on February 25, 2013, 10:24:58 PM
QuoteWhen I try to increase the bitrate with ML 2.3 it barely does anything, maybe 10 mbs extra?

That way only tricks the prediction into predicting larger frames. Then it holds all of them (at gop12) in the buffer and stops. Mine adjusts lower level quality setting and sets it up or down based on buffer usage. You have to make it make bigger frames AND write them out faster.

At some point a scene won't compress any higher. Stock is really conservative though, it had to to work on < class 10 cards.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Kabuto1138 on February 26, 2013, 01:23:12 AM

Hey Rush,

I do see the difference.  What the heck am I doing wrong.  What settings are you using for this?

thanks
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Rush on February 26, 2013, 10:32:10 AM
Quote from: Kabuto1138 on February 26, 2013, 01:23:12 AMWhat settings are you using for this?
It is not that simple with bitrate control.
First, if you record with audio - you will not be able to increase bitrate more than extra 15-20 mbps (1.3x).
But if you record without audio - you can switch to 2.0x on Class10 SD and up to 3.0x on speedy CF cards I think... (g3gg0 experimental tweaks can help with buffer overflow, but I didn't tried it by myself)

What will bitrate do to your video quality? For real, stock 45 mbps is enough bitrate for static camera scenes without a lot of movement or tiny details and when set to low ISOs - you will not see any downsides or artefacts to image cause of lack of bitrate.
But filming complex scene with movement + high ISOs (noise makes a lot of tiny movement across the frame) - requires high bitrates. 45 mbps become not enough and tiny details became washed out and blurred. It became more obvious when you start color grading and sharpening in post - you will see ugly lines and even blocks showing up as you sharpen your image.
High bitrate with ML can preserve more details on such bitrate demanding conditions.

So, the conclusion:
Stock 1.0x bitrate is usually artefact-free on low ISOs - 160 or 100 or 80 and trying to raise up bitrate will do nothing - it tops at 40-50 mbps. As you go up with ISO, you need more bitrate to avoid ugly compression lines and blocks which becomes visible after post shaprening.

Side effect - high bitrates on high ISOs will preserve not only fine details, but fine noise too - so it will look noisier that usual. But it will be easier to get rid of noise in post (NeatVideo) because it is less blotched or blurred.

You can clearly see if your compression works best and it have enough bitrate - just turn on bitrate info on screen and look at QScale value. Top bitrate quality is Q-16, and as it goes to 0 and to +... it means that your image is suffer of lack of bitrate.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Kabuto1138 on February 26, 2013, 11:57:49 PM
Thanks Rush!
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Yoshiyuki Blade on February 27, 2013, 01:38:28 AM
Yeah, recording without audio is the key factor in increasing video bitrate tremendously. As a matter of fact, when I fool around with camera settings and record indoors, I can just keep the quality at constant Q-16 and it won't go anywhere near my card's limit (~120 Mbps). It does exceed that limit outdoors though. There's just too much high frequency detail to maintain Q-16 with my card.

With audio on, I can't really exceed 60 Mbps. But with audio off, indoor bitrates typically range from 70-90 Mbps so it makes a sizable difference. With this much bitrate at our disposal, I wonder how much more quality can be achieved with mjpeg. Could it exceed what Q-16 of the H.264 encoder can do at similar/lower bitrates? I'm eager to see how things pan out.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on February 27, 2013, 02:00:50 AM
Q16 isnt really the tops. They are made up canon numbers. The real scale is a little bit bigger. Look at the 422 files for what input looks like. Mjpeg should look like that. 422 color is the main benefit.

600D can do the wav separately and 6D seems to just record audio with whatever bit rate you throw at it.

Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: wenen on March 16, 2013, 12:01:57 PM
Some good information here about stuffs http://www.youtube.com/watch?feature=player_embedded&v=Fs1TnHjGljw
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Yoshiyuki Blade on March 16, 2013, 01:25:29 PM
One thing about what the guy in the video mentioned that bothered me is that he somehow made a connection between a a bayer filter and the various types of chroma subsampling of the YUV color space (technically YCbCr). I noticed someone else make that connection in another video too so I don't know whether I'm misunderstanding something,  whether they're oversimplifying things or even misunderstanding it themselves.

Afaik, Bayer filters capture individual RGB colors and interpolate them since each pixel doesn't have all 3 color channels dedicated to it. Well, it's also called RGBG or some different order depending on the pattern because there are 2 green pixels for every 1 red and 1 blue in a 2x2 pixel block. Anyway, the raw image from the sensor can be "demosaiced" internally or via a RAW photo editor and can be encoded in RGB. I don't see how this has anything to do with 4:4:4, 4:2:2 and 4:2:0; that's from a completely different color space. Unlike RGB, where each pixel has its own RGB channel, each with its own luma level, YUV has 1 channel dedicated to luma and 2 channels dedicated to chroma, which makes it much more efficient. It also allows images to have a lower chroma resolution than luma, exploiting the human eye's weaker sensitivity to chroma resolution over luma resolution. Something like that doesn't happen in RGB. They're also different enough that they can't be losslessly converted between one another. A full 8-bit RGB picture can be converted to YUV 4:4:4 and back and look very close, but there will be rounding errors.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Rush on March 17, 2013, 07:36:31 PM
Just got SanDisk 95MB/s and tried to get maximum quality on 600D with the help of GOP-control.

What I get:

GOP1 - is the way to go for high bitrates, but this compression is not very effective - and it don't use buffer at all. 600D is limited to stable 160 mbps with it.
GOP12 - effective but it can't go higher than 100 mbps - because of high buffer use. Not stable.
GOP3 - is the platinum! It still limited to stable 160 mbps just like GOP1 (I think that it is ~170mbps SD interface limit of 600D), but is more effective than GOP1.
For example - it can go to 160mbps and compress with QScale -16 quality, while GOP1 can compress only QScale -12 quality with same bitrate. It uses buffer, but very stable and limited not to buffer, but to ~170mbps of SD interface.
I'd tried GOP6 - but I see that it hits buffer limit, not SD interface limit (GOP4 is as good as GOP3, but it is limited to buffer, not interface. There is thin edge between GOP3 and GOP4 - between interface and buffer limits)

And the most impressive for me was stable audio recording with 160 mbps video! GOP3 ftw! 8)

Two words about compression effectiveness:
Seems like GOP3 is only 10-15% less effective than stock GOP12 (handles same QScale with only 10-15% higher bitrate)
But GOP3 160mbps bitrate limit is ~50% higher than no-audio StockGOP12 (100mbps) and ~150% (!!!) higher than audio StockGOP12 (60mbps)!
So stable 160mbps of GOP3 is like 140 mbps of stock GOP12 +audio, which don't cause recording stops!

Thank you, developers! (is 1% - the main developer of GOP control? thank you!)
I have a question - how can I set my bitrate limit, so it will switch to lower -QScale if hits 165mbps limit?
CBR 3.0x drops QScale when bitrate is higher than 155mbps, not 165 mbps - but still is pretty nice working (I am so happy that it is stable, with audio, 155mbps!).
It seems that with CBR 3.0x 155mbps limit optimal will be GOP4 - slightly more effective and hits buffer limit only with 170mbps, so CBR 3.0x will do the work not to hit it. But I am not sure about GOP4 buffer fill stability, GOP3 will be more safe to use.

And another question - when GOP-control will be available in ML nightly builds? It is very useful and looks stable.

p.s. separate WAV recording don't work for me - it just fills buffer. But I don't need it now - happy with GOP3 audio recording.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on March 17, 2013, 09:06:37 PM
You have to have canon audio disabled for separate wav, just override on.

I have to rewrite the whole thing for it to work with the nightlies, the menu system changed.

Plan to add back the good stuff and leave out some extra stuff like all the canon cbr tinkering. I use gop3 too, I think its most effective because you still make some P frames. 600D video is eating digiV. :)


g3ggo also has separate flush and gop controls but I don't have the addresses for 600D, would be nice to try it and see what it does.

Quotehow can I set my bitrate limit, so it will switch to lower

If you use slice control it will drop when the buffer goes over the buffer warning level. Drops by 1 at x rate and then by 3 at x rate for when its getting really overloaded. The leveling eases it off slowly if over a certain rate. There are more detailed infos in this thread some posts back.

If you're just recording fixed qscale value its not set up to move either way... that is for fixed qp recording.

Right now mainly working on 6D since it still needs help and the only really missing from here are some bug fixes and image effects like peripheral correction, etc.






Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Rush on March 17, 2013, 10:24:21 PM
Quote from: 1% on March 17, 2013, 09:06:37 PM
You have to have canon audio disabled for separate wav, just override on.
Yes, I tested it with disabled Canon audio. It just fills buffer...

QuoteIf you use slice control it will drop when the buffer goes over the buffer warning level. Drops by 1 at x rate and then by 3 at x rate for when its getting really overloaded.
I tried this bitrate control with my Class10 card in october, but it worked really bad for me - it always dropped bitrate to <40 mbps so the blocks appears (but my card was able to record 80-90 mbps stable)... I did hate how it worked... Do you use it yourself? Is it works good for you?

I think that  GOP-settings should be tweaked according to SD card speed.
On slow cards it better to use longer GOPs (6-9). On fast cards - GOP3 or GOP4 will be the top quality.

To determine which GOP is best for your card, you should:
step 1) determine speed of card - turn off audio, switch to GOP1 mode + QScale-16, slowly increase complexity of scene and watch for bitrate and buffer values. (increase ISO to increase complexity) remember which value of bitrate causing red buffer load - it is max bitrate for your card.
step 2) determine which GOP is the best - turn off audio, switch to GOP12 + QScale-16, slowly increase complexity of scene and watch for bitrate. If video stops before max bitrate from step 1), then lower GOP value and repeat step 2). If video stops when bitrate hits max bitrate (from step 1) - then it is your card's optimal GOP value.
step 3) now set to you optimal GOP value and switch to CBR mode. Increase CBR value just before buffer starts to became red.
step 4) now repeat steps 2-3, but with audio turned on to get optimal GOP with audio.

For SanDisk 95mbps which can handle >160mbps best settings:
CBR 3.0x, GOP3 or GOP4. This will work with and without audio.

For my Kingston 100x Class 10 card, which can handle 110 mbps max:
w/o audio: CBR 2.2x, GOP8. Decreasing GOP will lead to lack of card speed, increasing GOP will lead to buffer limit.
with audio: it hard to determine... short GOPs surprisingly unusable cause of buffer overflows. long GOPs is better but don't hit 110 mbps cause of buffer... seems that on slow cards it hard to keep both high bitrate video and audio...
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on March 17, 2013, 11:23:20 PM
I use it for myself, don't have problems with the audio filling the buffer. It doesn't use that buffer. But I can't record audio on with different gops. After a little bit I get an error 70 because the audio funcitons weren't patched. My only issue with separate wav was syncing it and skips before it used dma memcopy.

If fixed qscale works for you, use it. Camera sets quality by slice, the other 2 functions are just abstractions. 5d3/6d just use qp directly now.

Also crop mode rates > regular rates.





Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Rush on March 18, 2013, 08:37:23 AM
Quote from: 1% on March 17, 2013, 11:23:20 PMIf fixed qscale works for you, use it. Camera sets quality by slice, the other 2 functions are just abstractions. 5d3/6d just use qp directly now.

Also crop mode rates > regular rates.
Ah, I see.. That is because higher ISO noise in crop mode. ISO noise have great impact on required bitrate to maintain all moving tiny noise dots.

Can I use fixed slice? To avoid autoadjusting of bitrate.
I heard that slice quality can be higher than QScale-16. Is it true?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on March 18, 2013, 02:19:12 PM
Yep, you can use fixed slice. Just set the lock to on and turn off leveling. Crop mode buffer isn't just noise, its bigger too.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: xaled on March 21, 2013, 02:12:23 PM
Hi,

Is it "normal", that when you play recorded videos with changed GOP and QScale parameters in the camera (600d) you get all kind of strange color splashing over the screen?

If I play the same file on the pc, everything is fine.   
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: nanomad on March 21, 2013, 02:50:24 PM
Yes, the decoder is not really that good
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: xaled on March 21, 2013, 06:31:30 PM
Ok, thanks, that's actially kind of obvious afterwards.
For a second I thought my cam was somehow broken.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Rush on March 26, 2013, 10:01:53 PM
I found bug in high bitrate handling.
It works like a charm with enabled sound up to 160 mbps stable. But works only in 25p mode! If I switch to 24p - buffer just fills in a few seconds. 24p works with enabled sound only with stock bitrate. Once I disable sound - high bitrates works nice.
But high bitrates in 25p mode works great both with and without sound! It is strange for me.
1%, what are your thoughts about it? (can I ask you to make new .bin with latest changes?)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on March 26, 2013, 10:16:43 PM
I want to rewrite the whole thing and try to use engio writes to set the slice instead of just changing it in memory... its still WIP since I've been working on 6D.... BUT sound works in 25p? That is good, need to find out what is different and make sound work in all modes. Audio functions aren't patched in theory it shouldn't be working anywhere so this is actually a good find.

Wow... holy shit... it really does. 25P is the only one.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: N/A on March 27, 2013, 04:08:22 AM
Whoa. Tried it on my T3i, works perfectly with sound on, averaged 161 mbps. Why do you think this is? Surely 25p would require even more data than 24p??
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on March 27, 2013, 03:45:06 PM
Hopefully I can find out and apply it to all modes... and hopefully audio stays in sync. 25P must not check it so we have to set up sound like it does all the time.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: N/A on March 27, 2013, 09:11:43 PM
Not an expert on bitrate specifics, but this is what VLC was telling me, This is 25p, audio on, slice 87, GOP1. Holy shit.

(http://s11.postimg.org/ako32luw3/Screen_Shot_2013_03_27_at_4_03_27_PM.png)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on March 27, 2013, 11:34:27 PM
I have to take a log while its doing it and compare to a log of when it mutes the audio.

I've started on the second version. Will be a little bit before everything is ported (and figured out). Audio on in all modes and digic set bit rate parameters might be the trick to get perfect videos (fixed fps override on digic V). Going to try to roll picoC into this one so we have scripting.. right now don't see too many scripts available. Hopefully can have everything both ways, cCBR, VBR and digic fixed + dynamic quality.

Getting the easy features out of the way first... hopefully audio on can work for all cameras that only do canon cbr. Have you checked if the audio stays in sync/mutes or skips? I guess DigicIV isn't tapped out yet.

*PS. This proves audio isn't being stopped by the buffer but instead canon bullshit.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Yoshiyuki Blade on March 27, 2013, 11:50:46 PM
Though this may be known already, I've found that FPS override with audio recorded separately allows for high bitrates. I don't know how high, but its much higher than default; I can maintain a constant Q-16 while recording indoors without a hitch, but the bitrate indoors rarely exceeds 100 Mbps, which is below what my card can handle.

Of course, the issue is with the sound sync. I still have no idea how to systematically sync the AV back together. I think it's mainly a problem with the true framerate that FPS override records at, which doesn't seem to be the FPS that it says it records at. Once I can figure that out, it should be cake.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: ilguercio on March 28, 2013, 01:24:29 AM
But still, is it worth bumping up the CBR or the VBR value? For most users default is ok, some of us just want the ability to control P and I frames.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on March 28, 2013, 02:41:37 AM
I dunno... i'm not really going to mess with those modes, just leave them as options. I want to set slice per frame via digic + other parameters like deblocking, etc. The old functions changed everything in memory. Maybe it will be faster to skip that step or maybe it will let you enter more/higher values.

Would be interesting to see what happens with different gop/flush rate, 7D and video hacks have options for that but I don't have the locations for 600D.

Maybe try again to fix vsync with coutt's evf state thing.

At the least you get new menu + movie mode remap :)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Rush on March 28, 2013, 08:14:36 AM
Quote from: ilguercio on March 28, 2013, 01:24:29 AM
But still, is it worth bumping up the CBR or the VBR value? For most users default is ok, some of us just want the ability to control P and I frames.
I'd posted camparison earlier. High bitrate is better - much less compression artefacts with high ISOs.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: mike.charal on April 06, 2013, 04:46:10 PM
Thanks to 1% and all the devs for your hard work and success. It looks like I'm quite late to join, but I read all 20 pages if that counts haha.

Forgive me if this is not the venue to pose this. I've been experimenting with the slice settings all night and I've found my card can sustain about 90 mbps for long periods of time. If I know what my scene complexity is and that it will not change significantly, I've had great success with locking the slice and disabling taper rates. I get a sustained 90 mbps, which is great for high ISO.

When I let the slices change depending on the buffer, I can't seem to keep them under control regardless of how I set everything up. It yoyos back and forth when the buffer fills and it's very obvious in the video that the quality drops out. In bitrate viewer, this particular video sits around 100 mbps but dips down to 20 mbps many times even though I've set the max bitrate not to exceed 85 and the min to 45.

Am I missing something obvious here? Maybe bitrate is dependent on something else entirely? As I understand it, the CBR multiplier doesn't really affect anything in slice mode. Would a faster card solve the issue of yoyoing? It's just a sandisk ultra 30 MB/s SDXC UHS-I 64GB. I just thought I'd be able to sustain a higher bitrate with that card than with the stock firmware encoding (like I said 80-90, not the 160 people are getting on here with extreme cards).

Thanks for any help! (Also, I'm set to GOP 3, deblock a/b=-1, PicQPC=-4 if that matters. GOP 3 seemed like a good balance for me too.)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Rush on April 06, 2013, 06:24:59 PM
mike.charal, do you try it with sound off? It is a problem to raise bitrate if sound is recording and it is not 25p mode.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on April 06, 2013, 08:22:14 PM
Set buffer warning level. That is what tells it to drop. I set mine to like 50%/60%
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: N/A on April 06, 2013, 09:39:01 PM
The PAL 25p setting with sound on works perfectly with slice 87 GOP 1. BUT  then under tungsten I get that weird rolling shadow look, coming from 25p and 60 hz not playing nice together I'm guessing? Need more tests but it happened every time I was using tungsten lighting.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Rush on April 06, 2013, 09:45:25 PM
Quote from: N/A on April 06, 2013, 09:39:01 PM
Yes, sure rolling shadows in NTSC country... But current implemetation of bitrate control have troubles when it is not 25p mode.
Try 24p or 30p with disabled sound...
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: N/A on April 06, 2013, 10:03:18 PM
Yeah I have been so far, but cam audio is a god send for lining up the video to recorded dialogue from the recorder. Ah well, can always have the actors clap I suppose.  ;)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on April 06, 2013, 11:00:20 PM
Try recording wav and see how it matches up. I'll figure out how to patch all at some point.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Leon on April 07, 2013, 03:09:22 AM
@ N / A :  I can't think why you would be getting rolling shadows with tungsten lighting.  Are you sure you don mean fluorescent / energy saving?  A tungsten light doesn't cool down sufficiently between the phases of the alternating current to cause a visible variation in light level, whereas fluorescent does.  In any case the frame rate shouldn't matter; it is the shutter speed that can cause problems, so try a shutter speed of 1/60 (for USA and 60Hz supplies) regardless of frame rate.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: mike.charal on April 07, 2013, 09:18:52 PM
Rush - Yeah, I had the sound disabled and was recording in 24p.

1% - Thanks for the advice. I was able to fool around with and get a better handle on the min/max bitrate settings and I raised the buffer warning level much higher and it's working pretty flawlessly, no more shooting down to slice 127 then back up to 87. Suffice it to say, I'm very excited. I've seen a lot of comparisons of shots straight out of the camera, where the differences may seem subtle, but yesterday I did a test grade on a 90 mbps medium complexity image and I noticed when I applied levels and squeezed everything in, the image remained amazingly smooth; very few (if any) compression artifacts or macroblocks were revealed. Truly amazing, thanks again!
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on April 07, 2013, 10:57:51 PM
You're welcome. V 2.0 will hopefully be faster/better.

You can see it being made here: https://bitbucket.org/OtherOnePercent/tragic-lantern-2.0

So far I got most stuff in except for bit rate. xxGB limit will be selectable. Hopefully get it all done in a couple of months. Right now its tax time.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: driftwood on April 10, 2013, 11:29:39 PM
Hi 1%
Buffer settings are generally around .70 of the bitrate settings in h264. Is there a way to set buffer bitrate yet?

Whilst on the subject of the latest encoder settings for ML - Do you have scaling tables ? Point me to the latest variables available.

Thanks

Nick
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on April 11, 2013, 01:04:53 AM
No way to set the buffer per se, just when the buffer warning goes off. H264 buffer is different than camera write buffer. I think we only see the latter (I see H264 buffer when throwing into analysis tools).

There are 4x4 and 8x8 scaling matrices in the 5d3/6D encoder. Only the 8x8 transforms work, the 4x4s for some reason don't play back. Lots of underexplored territory.

See below for what is available in new encoder:

http://www.magiclantern.fm/forum/index.php?topic=4124.0


highest rate out of 6D I got was 400Mb/s


Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on April 14, 2013, 08:15:35 PM
Seems the 4x4s get used dynamically on 6D.


quant_param          : 5
pmode                : Intra_4x4
ipred Intra_4x4      :
DC           DC           DC           DC         
DC           DC           DC           Horz       
Vert         DiagDwnLeft  DiagDwnLeft  Horz       
Vert         Vert         DiagDwnLeft  Horz       
ipred chroma         : DC


Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: kgv5 on May 04, 2013, 01:07:20 PM
Hi
sorry for the noob question but i have to ask especially the 6d owners: what do you think is the best setting of ML encoder (newest version) to get best picture quality?
Could you please give me your settings of:

ALL-I or IPB ??

bit rate
initQP
flush rate
GOP size
Config select.

I have 45 MB/s sandisk extreme card.
In the 2.3 version (in my 550d) it looks easier, here i am little bit confused.

Do you think that improvement of picture quality is noticeble?

Thank you
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on May 04, 2013, 06:11:22 PM
Off, Off
Flush 6, GOP 24
IPB with rate control config set to autoload.

For sound I just leave flush stock unless I'm recording wav with it.

ALL-I is a wash just like on 600D, frames are just too big at good qps and inferior to IPB or IBB.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Rush on May 04, 2013, 06:24:05 PM
1%, thank you very much for porting flush control to 600D! (spotted it in latest 2.0 build (https://bitbucket.org/OtherOnePercent/tragic-lantern-2.0/downloads))

So, I will further optimize my previous settings (http://www.magiclantern.fm/forum/index.php?topic=1565.msg28974#msg28974).
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on May 04, 2013, 06:41:32 PM
Still needs slice and gop :)

But an interesting thing is that with flush via cache hack and gop via video parameters it might be possible to do flush 3 and longer gop. That's not something I've done on 600D yet.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: kgv5 on May 04, 2013, 08:54:18 PM
Quote from: 1% on May 04, 2013, 06:11:22 PM
Off, Off
Flush 6, GOP 24
IPB with rate control config set to autoload.

For sound I just leave flush stock unless I'm recording wav with it.

ALL-I is a wash just like on 600D, frames are just too big at good qps and inferior to IPB or IBB.

Thank you 1%, so what bitrates are you getting with such settings?
I made a quick test 1080p 24fps IPB,  (due to mediainfo) i get about 64 mbit/s with flush 6, sound off and almost the same with flush auto sound on. In the same conditions canons stock is about 30 IPB and 50 ALL-I. Fact, it is very stable but do you think it will give visible better picture quality? Does it make sense to go higher up to 120-150 and if it does, which parameter i have to change in the first place?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on May 04, 2013, 09:09:01 PM
I get 62-80... it depends on scene complexity. 150+ is where issues start to happen with dropped frames and the like.

Picture quality is better than stock, it picks higher QP when I analyze it in streameye. Also CABAC vs CAVLC. Mostly you would notice it editing the files in post, they stretch further.

Ideally someone should test lvrec vs H264 at different qualities and find out what is "best".

To just jack up the bit rate set it to override and set a low initqp like 1-10 or 15. Target can be whatever its only loosely adhered to in fixed qp recording. You can decrease the gop and flush rate too if it stops. Just pick gops that are divisible into fps or it won't record.

Gop of 3 or 4 gives you IBB I think so you lose the P frame and I guess only compress between I frames. Slightly less load on the CPU. Not sure what it does to quality as nobody else has an IBB encoder.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: kgv5 on May 04, 2013, 09:59:45 PM
Got it! Thank you so much for your explanation, helped me a lot  :D
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: DjJuvan on May 04, 2013, 10:55:40 PM
I didn't go all the way back in topic... mostly you're talking about GOP changes on 600D.... what about 550D?  Does this settings and tragiclantern apply to this model too?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on May 04, 2013, 11:22:38 PM
I have no 550D and 550D is limited to 4gb files. 2.0 still doesn't have it (big files, not 550D) implemented but soon.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: AriLG on May 06, 2013, 09:04:52 AM
What is the difference between TL v1 and TL v2 ?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on May 06, 2013, 03:25:45 PM
TL 1- BR by sllice and extra controls.
TL -2 LV_rec + new menus - bit rate control not fully done, just have gop, flush, file size so far.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Rush on May 08, 2013, 01:55:25 AM
My review on new TL 2.0 flush rate and GOP control.

Seems like there is no reasons to use GOP 100 - it is only ~0-5% more effective than GOP 12 :(
So default GOP 12 is optimal with flush rate adjusted (I set it to 6 frames. I don't know really, but for me it look like low values reduces upper limit of bitrate because card become more busy).

New extreme high CBR modes is not possible with 600D sd's transfer limit. Maximum is 4.5x with GOP12 which is around 160 mbps.
It is strange, but with short GOPs 160 mbps SD limit is achieved by setting CBR 3.0x, not 4.5x.

And it looks like there is no any problems to playback video files incamera with changed GOP if it is under 60-70 mbps.
It wasn't possible to playback with old GOP-control.

It is sad that flush rate settings not compatible with audio recording, so for me I stick to GOP4+audio+CBR3.0-160mbps with default flush rate.. But wait! I can't use this settings with audio in tl-2.0 cause of buffer while it works perfectly with tl-1.0 in 25p! Something is bad with sound in tl-2.0...
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on May 08, 2013, 02:34:59 AM
I'm working on the audio, needs a solution for all cameras. Gop 100? Gop 24 is 1/fps, should be maximum. Gop 1 is All-I... flush is independent of gop or its supposed to be. So you can have more compressed gop of 24.. or maybe try 48, this didn't work on 6D, limit is FPS.

I think the excessive X numbers just overflow so you won't gain anything from 4.5x-20x, it seems to just encode at QP10. The extra X numbers are from going off of video hacks.

You have to analyze the videos with BR viewer (or analysis tools if you have them). Gop 100 you are writing 1 iframe for 100 p frames or somewhere about there, almost all P.

More gop in reasonable steps should mean more compression across frames with QP10. In the old version couldn't set flush and gop separately.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Rush on May 08, 2013, 11:46:31 AM
QuoteGop 100? Gop 24 is 1/fps
I didn't expected that it works, and it showed no visible difference vs GOP12 with bitrate with fixed QScale.
But it works! I checked video in VirtualDub and it showed that I frames (K frames in VD) is every 100th frame.

Hope that audio issue will be resolved. Thank you for your effort!
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: wenen on May 14, 2013, 10:26:39 PM
24p RAW on a 5D mark III (Magic Lantern) http://www.cinema5d.com/news/?p=17898#more-17898

Will it work on a 600D?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Rush on May 14, 2013, 10:35:41 PM
It works!  ;)
Good quality: http://bit.ly/YFM92c


Max possible without dropped frames is 1280x400 24p or 960x540 24p but still looks awesome (I prefer 1280x400).

1%'s builds for 600D:
https://bitbucket.org/OtherOnePercent/tragic-lantern-2.0/downloads/
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on May 14, 2013, 10:55:12 PM
I feel like even the tiny DNG raw files are better or as good as H264. The small ones will probably go to 720P at least... a 2x upres to 1080, dunno but probably close.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: wenen on May 15, 2013, 05:20:43 PM
What is lv_rec?

I can't find in the new meny how to set the camera to slice 87. Where can i find it?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on May 15, 2013, 08:36:49 PM
That stuff isn't done yet.. been working on raw which is 10x better than H264.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: N/A on May 15, 2013, 09:49:12 PM
I know it's a moot point now but I noticed with slice 87/GOP 3 video on the TL1 build that I could push the in-camera sharpening up a few notches with no discernible effect on moire or aliasing, which I found much more pleasant looking than sharpening in post.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: wenen on May 19, 2013, 03:44:55 PM
Any recommended settings for RAW with SanDisk Extreme SDHC 45MB/s 32GB card?

I have try 1280/720 25p its laggy
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on May 19, 2013, 03:59:47 PM
Something that fits under 21MB/s
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: mike.charal on May 19, 2013, 09:30:54 PM
Hey guys, I've been itching to test this out and I finally found the time, but I can't seen to find the menu item "RAW video" on the most recent 600D Tragic Lantern 2.0 build. I get a message that says "script dir missing", just in case that has something to do with it. I still get that message even when there is a script dir there. Sorry to clutter up the thread but I'd love to know if I'm missing something totally obvious here. Thanks so much in advance, I'm continually amazed by the progress here.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on May 19, 2013, 09:32:35 PM
scripts and modules go in ML folder not root
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: mike.charal on May 19, 2013, 09:46:08 PM
Should have just tried that, much appreciated.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: wenen on May 31, 2013, 04:56:03 PM
How many frames can you film? I can only do 71 frames then it stop
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on May 31, 2013, 04:57:31 PM
Depends on resolution.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: vfx_junkie on June 01, 2013, 10:58:00 AM
1%

can you tell me what are the best settings for GOP etc.

on default I can shoot on CBR x20 with no problem the quality is much better then the native x1.0 :D
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Critical Point on June 06, 2013, 11:02:58 PM
Seems to me, that everyone has gone with the raw video trend, and forgot about tweaking the H.264, but for the 600D, I think that this is the only real option for improving quality. As long as we can't get at least 1280x720@24 fps raw on the 600D, tweaking better the H.264 still remains the only real option. I was really intrested in seeing more progress on this thread.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Audionut on June 07, 2013, 05:56:50 PM
It's understandable that attention has been focused on raw recording.

It's pro's far outweigh it's cons.

This topic could become active again at some stage in the future, especially if the pipeline can be adjusted to bypass the intentional quality destroying attributes of Canons current implementation.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Critical Point on June 08, 2013, 01:24:46 AM
Yes, it kills me to see how much quality the hacks bring on GH2, and how little improvement bring on Canon. I mean Canon can now shoot at ~160 mb/s, and we get nowhere near the quality improvement that we see on a GH2 shooting at even 65 mb/s. We definitely need better H.264 tweaks, and I still believe it can be done, and I have great faith in the developers of ML that they will manage to crack this problem. AVCHD that Panasonic is using, and Canon's H.264, are related, because AVCHD is just a subset of H.264, and because of this, I know we can get way better quality from the H.264 then we're getting now, if Panasonic have done it, ML can do it too.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: driftwood on June 08, 2013, 03:50:06 AM
Now Ive got the MKIII Im gonna be dedicating some to time to the h.264 encoder settings.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on June 08, 2013, 03:55:50 AM
Welcome to the dark side, lol.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: N/A on June 08, 2013, 09:37:05 PM
Quote from: 1% on June 08, 2013, 03:55:50 AM
Welcome to the dark side, lol.
You ever look into why PAL 25p could do audio with max bitrate, but NTSC 24p couldn't? Or was that just a fluke?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on June 08, 2013, 11:43:37 PM
I tried to, when I put fixed slice back I'll try it again.

First I have to fix the crashes with EVF state and MZ + raw histo or zebra.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: N/A on June 09, 2013, 01:43:47 AM
Cool, can't wait. Too bad 2k .mov is impossible, we could probably hit it with 170 mbps.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on June 09, 2013, 02:01:27 AM
Yea, fixed sizes is all it takes :( At best we can take some other rectangle crop.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Critical Point on June 09, 2013, 03:43:28 PM
How about taking a normal ML 2.3 for 600D, and add the GOP control feature on it, and then release it as a nightly build ? ML 2.3+, or something like that.

Right now, that's all I'm caring about, that GOP control feature, because it's becoming clearer that on a 600D, the raw feature ain't going to happen for 720p@24 fps.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: N/A on June 09, 2013, 03:47:17 PM
Already done-

https://bitbucket.org/OtherOnePercent/tragic-lantern/downloads
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Rush on June 09, 2013, 03:48:04 PM
Quote from: Critical Point on June 09, 2013, 03:43:28 PM
Right now, that's all I'm caring about, that GOP control feature, because it's becoming clearer that on a 600D, the raw feature ain't going to happen for 720p@24 fps.
GOP Control is not the best way to achieve max quality of H.264.
Flush rate control do this job better, but it just lacks compatibility with audio yet...
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Critical Point on June 09, 2013, 03:50:26 PM
Flush rate control ? And what settings exactly ? I don't need audio.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: ItsMeLenny on June 09, 2013, 04:03:26 PM
So is it official that the dslrs can only record at 1080, 720, and 480.
Or are there other options somewhere in there.
I can imagine the processors only being designed for those as the optimal.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: jankyy on June 09, 2013, 05:10:14 PM
hi there,

i have been looking for instructions about setting the .h264 encoder on latest ML / 6D. unfortunately, the 'user guide' seems to refer to some older version / menu items.

could somebody tell a noob like me, what's a good (and safe!) setting / combination for shooting .h264 with approx. 80-100Mbit/sec? in 'my' encoder menu, the possibilities are:

- Bit Rate (0... 15 or more)
- InitQP (off ... 50)
- Flush Rate (auto ... 4frames... 50frames)
- GOP size (default ... 100frames)
- Autoload conf (off / on / override)
- Config select (off/h264.ini  /  CBR fixed QP   /   VBR   / Rate Control)

i don't understand how these items combine, and i can't find an explanation anywhere... thanks so much -

jan

p.s. i have ML from June 2013, a 6D and SanDisk Extreme Pro 32GB

Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Critical Point on June 09, 2013, 05:14:26 PM
Yes, yes, please tell us the best settings for H.264 (without audio). I don't understand those settings also, and I can't find any explications.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on June 09, 2013, 06:11:18 PM
QuoteI can imagine the processors only being designed for those as the optimal.

The encoder has fixed modes only... other image processors are fine.

Instructions are all in this thread... but old stuff not ported yet. So you can do flush + gop and make it record max "normal" qp all the time. No flipping or direct quality control yet.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: jankyy on June 09, 2013, 10:28:49 PM
Quote from: jankyy on June 09, 2013, 05:10:14 PM
what's a good (and safe!) setting / combination for shooting .h264 with approx. 80-160Mbit/sec? in my ML encoder menu, the options are:

- Bit Rate (0... 15 or more)
- InitQP (off ... 50)
- Flush Rate (auto ... 4frames... 50frames)
- GOP size (default ... 100frames)
- Autoload conf (off / on / override)
- Config select (off/h264.ini  /  CBR fixed QP   /   VBR   / Rate Control)

i can't find an explanation for these particular options anywhere???... thanks so much -

jan

p.s. i have ML from June 2013, a 6D and SanDisk Extreme Pro 32GB

p.p.s and how do these settings relate to the canon in-camera settings ALL-I and IPB? which overrides which? (if there's an explanation already somewhere, please hit me upon it! thx...)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: kgv5 on June 10, 2013, 03:08:02 PM

I was discussing this with 1% before, try this:

1080p IPB

- Bit Rate OFF
- InitQP OFF
- Flush Rate (auto ... for sound of 4 if you don't need a sound)
- GOP size 24
- Autoload conf: override
- Config select: Rate Control

It should give very stable IPB with 60-70 mbps bitrate which theoretically should look better than 60-70 ALL-I. :-X
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Critical Point on June 10, 2013, 03:44:17 PM
But for the maximum quality (without sound), what settings ?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: jankyy on June 10, 2013, 03:46:27 PM
Quote from: kgv5 on June 10, 2013, 03:08:02 PM
I was discussing this with 1% before, try this:

1080p IPB

- Bit Rate OFF
- InitQP OFF
- Flush Rate (auto ... for sound of 4 if you don't need a sound)
- GOP size 24
- Autoload conf: override
- Config select: Rate Control

It should give very stable IPB with 60-70 mbps bitrate which theoretically should look better than 60-70 ALL-I. :-X

thanks very much, kgv5!!!

looks like it's working well for 1920/25fps IPB, gives me ca. 61Mbit/sec, which is a good number for me to handle...
just wondering:

- is the GOP size 24 the same choice for 24fps or 25fps?
- what would be the respective settings for 1280p50 IPB? (for now, a test with the above setting gives me about 106Mbit/sec, which would still be fine for my SanDisk ExPro card i guess)

thanks - jan
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on June 10, 2013, 04:34:19 PM
Max gop is FPS so for 25 P its 25.. 50p its 50, etc.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: jankyy on June 10, 2013, 06:07:03 PM
Quote from: 1% on June 10, 2013, 04:34:19 PM
Max gop is FPS so for 25 P its 25.. 50p its 50, etc.

strange:
- it's only working with GOP size exactly 24  (both at 1920p25 and 1280p50 IPB)
- not working with GOP 25 or 4 or... it just doesn't record, and does a restart after 10secs...

anymore insight in these settings?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on June 10, 2013, 07:38:46 PM
What about 50?... maybe uneven gop size doesn't work. This is all from canon, all I can do is set it.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: jankyy on June 10, 2013, 10:06:15 PM
i just tested diefferent GOP sizes:

- 2 and 4 and 25 and 50 - they all don't work at all - in the upper right corner, A and B and the % stay at zero, after 10secs the camera restarts...
- only GOP size 24 frames seems to work.

i don't understand why; and i also don't understand the big difference in bit rate when using the 1920p25 or the 1280p50 resolution:

- 1920p25 gives approx. 60Mbit/sec
- 1280p50 gives apporox 105Mbit/sec (which is a lot of data, and takes lots more time to convert to DNxHD...)

any idea? thanks! jan
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on June 10, 2013, 10:40:38 PM
I'll check on gops.. I have to rewrite this for 50D... 50/60FPS vs 24 or 30 is why the data rate is so high.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: jankyy on June 11, 2013, 02:04:45 PM
Quote from: 1% on June 10, 2013, 10:40:38 PM
I'll check on gops.. I have to rewrite this for 50D... 50/60FPS vs 24 or 30 is why the data rate is so high.

trial and error:
- i found that as settings for GOP size: 24, 12, 6 and 3 are working, for 1920p25 IPB (other GOP sizes do not record at all)
- but the GOP size doesn't seem to affect the file size / bitrate (/quality?) at all

and still wondering: data rate for 1920*1080, 25fps and 1280*720, 50fps should be about the same; yet, using the RateControl mode produces very different data rates for these modes...

sorry to bother, i know you're very busy working on the raw option, but for the time being it would be just great to have some .h264 settings that work safely (with options that i can get my head around  ??? ). thanks again - jan
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: jgharding on June 18, 2013, 05:20:06 PM
I know most of the dev time seems to now be on raw modes, but did the 600D I frame mode ever reach a stable build that one can use to shoot with?

I'm keen to try it, but is it available in friendly form for shooting?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on June 18, 2013, 05:43:40 PM
Quotebut the GOP size doesn't seem to affect the file size / bitrate (/quality?) at all

It affects I to P ratio. Analyze the files...
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: jgharding on June 20, 2013, 07:06:23 PM
Ah yes I found Tragic Lantern.

I'm mainly interested in getting an I-frame (GOP1) with sound, so I'll fiddle about

As far as I can see GOP1 with CBR 2.3 or so should be good results? Do I need to change slice in order to get sound too though?

On the 550D I could do about 2.8 with no sound, but I'd like I-frame and sound if possible...

I shall experiment.

raw would be wonderful but current limitation make it impracticable i think.

If the data to the buffer could be cute (maybe 10-bit raw?) it'd become more practical, but for now it's H264 and the mosaic engineering anti-aliasing filter...

That filter makes the bitrate increases a lot more meaningful as far as I've been able to determine... the data isn't soaked up by false detail from aliasing.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: jgharding on June 21, 2013, 11:10:40 AM
I found that GOP1 and all the way down on the Q-scale gives good high bitrate.

Noise can be removed easily and motion is nice. I didn't touch any of the other settings.

I'm curious as to how a consistently higher rate can be squeezed out though, but obviously beyond a point the detail isn't there, so for now this seems reliable, good and has sound.

Any other tips out there for solid I-frame?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: jgharding on June 21, 2013, 11:28:15 AM
I did try the raw mode, and it is wonderful for dynamic range and colour, but obviously at such reduced resolution and sensor crop it's quite a compromise. Damn that little buffer.

I'll definitely keep a close eye on it and keep testing it though, and have a trawl back through this thread to find some more H264 settings...
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on June 21, 2013, 03:01:33 PM
Heh, the old builds I think had sraw + slice.. in theory the most free memory should be in sraw.. so does recording time go up @ slice 87?

New builds still need this, will have to see.. also have to put combine dialog timer hack + slice + sraw memory.. I'm assuming this combination might be good.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Critical Point on June 21, 2013, 10:03:29 PM
Isn't there a way to make the H.264 codec not rescale the image so many times ? I bet that is one of the major causes of poor image quality compared to a GH2.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: jgharding on June 24, 2013, 01:05:09 PM
Quote from: 1% on June 21, 2013, 03:01:33 PM
Heh, the old builds I think had sraw + slice.. in theory the most free memory should be in sraw.. so does recording time go up @ slice 87?

New builds still need this, will have to see.. also have to put combine dialog timer hack + slice + sraw memory.. I'm assuming this combination might be good.

I've missed a bit of development recently so I'll need to look up what sRaw is... I'll trawl some threads...
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: wenen on June 25, 2013, 02:29:19 PM
1% is it easy to go back to the old build from the new one?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: jgharding on June 25, 2013, 05:01:39 PM
Quote from: wenen on June 25, 2013, 02:29:19 PM
1% is it easy to go back to the old build from the new one?
Keep each Autoexec.bat file that you try out stored in different logically named folders on your PC. Then you can swap and test builds as and when you like.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on June 25, 2013, 07:19:25 PM
The encoder takes fixed sizes, so I don't think we can eliminate any scaling... at least there is crop mode.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: jgharding on June 26, 2013, 12:48:46 PM
Quote from: 1% on June 21, 2013, 03:01:33 PM
Heh, the old builds I think had sraw + slice.. in theory the most free memory should be in sraw.. so does recording time go up @ slice 87?

New builds still need this, will have to see.. also have to put combine dialog timer hack + slice + sraw memory.. I'm assuming this combination might be good.

I can limit the encoder to around 160mbps cap using slice control. With GOP1 I never fill the buffer. My sound is off, but I'll try it with it on.

Slice control really is brilliant. With:

Min BR: 130
Max BR: 160
Drop 1: 140
Drop 3: 145
Taper Rates: Enabled

PicPC: 0

I never get in trouble. Well, if I have a super complex scene like a carpet all in focus at f22 and ISO6400, and start recording at slice 87 it'll overrun and drop down to highslice, then settle. If I start such a scene at slice 120, it'll settle very quickly, capping around 160mbps and only buffer dropping to a higher slice in an absolute emergency.

These settings let me maintain maximum possible from the stock H264 method. It's a great bit of coding.

I have a SanDisk 95MBs UHS1 card and the TXi filter installed.

Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: jankyy on June 26, 2013, 09:23:59 PM
Quote from: jgharding on June 26, 2013, 12:48:46 PM
Slice control really is brilliant. With:

Min BR: 130
Max BR: 160
Drop 1: 140
Drop 3: 145
Taper Rates: Enabled

PicPC: 0

I never get in trouble.

please help me understand:
is this something that works for Canon 6D, too? where would I find these settings?

thanks - jan
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: jgharding on June 26, 2013, 10:01:09 PM
Quote from: jankyy on June 26, 2013, 09:23:59 PM
please help me understand:
is this something that works for Canon 6D, too? where would I find these settings?

thanks - jan

I'm not aware of if such an H264 control mode was implemented in 6D code at all, I'm afraid I can't answer that.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on June 26, 2013, 10:04:26 PM
6D uses configs, lets you set qp and and other things. Diff system.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Scipione205 on June 27, 2013, 06:45:50 PM
Hi all, I tried to understand something reading from the first post till the last. However I didn't understand much.


jgharding said that he ca use ISO 6400 with is 600D using certain compressions, I'm trying to understand how: I have a 600D (with no TXi mosaic filter) and I need to record in very low light, at least ISO 1250 or even 2500, but the noise is too strong (I use flaat, cinestyle and visioncolor).

I tried with the last build (the one with new menu and RAW modules) with GOP1, CBR 20x and Flush 4 (I didn't understand yet what does flush means), but I can't notice better image quality and lower noise vs default recording settings.

Can anyone write a summary wich can help to understand, so I can help doing experiments?

Thank you
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Luzestudio on June 27, 2013, 08:23:49 PM
Hi!!

I just installed the Tragic lantern version and adjusted the parameters kindly recommended by jgharding.
I'm really interested in getting the best from h.264.

Will make some tests over the weekend but for now my sd card is holding well.

Best!

Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: MD87 on June 27, 2013, 10:04:29 PM
 "GOP1, CBR 20x and Flush 4" I was also pleasantly surprised by the result of such adjustments. Better detail with a slight increase in bit rate. Not only pleases the lack of sound recording  :(
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on June 27, 2013, 10:08:05 PM
Try lower CBR.. the 20x doesn't do anything but overflow the numbers, will fixate on max "official" qp. Old version had much better H264.. yes still needs to be ported, I sound like a broken record :)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: jgharding on June 28, 2013, 01:10:59 AM
Quote from: Scipione205 on June 27, 2013, 06:45:50 PM
jgharding said that he ca use ISO 6400 with is 600D using certain compressions, I'm trying to understand how: I have a 600D and I need to record in very low light, at least ISO 1250 or even 2500, but the noise is too strong (I use flaat, cinestyle and visioncolor).

Sorry I forgot I mention denoising, I take it for granted.

I should have said: ISO 6400 is usable after denoising with Neat Video. The reason the high bitrates help is (speaking simply) the noise is encoded sharply so it's easier for Neat Video to detect and remove.

At default bitrate the noise is mushed into the picture so you can't take it out. That's why ISO6400 is useful with this hack, because the settings I used keep the bitrate cranked to 160mbps continuos.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Luzestudio on June 28, 2013, 10:06:41 AM
Short question for improved higher bitrates:

With a Sandisk 45 MB/s (http://ecx.images-amazon.com/images/I/71DBV%2BV0NRL._SL1482_.jpg (http://ecx.images-amazon.com/images/I/71DBV%2BV0NRL._SL1482_.jpg)) is enough or a 95 MB/s ( http://ecx.images-amazon.com/images/I/51Kdyjl02FL._SX385_.jpg) will do any better? I think it will do the same because the hardware, but...

For the 600D I mean.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: SDX on June 28, 2013, 10:20:07 AM
It doens't matter which you choose since the camera is the limit.
Unless you want fast transferrates for the pc-side of things.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: vanburen08 on June 28, 2013, 06:07:46 PM
I agree, I've been following this thread for awhile and would like to know a detailed explanation of how to use TL 2.0. IT has as far as GOP settings, Flush, GOP, and file size. It mentions EOSmoviefixer.exe. Do I install this alongside the autoexec? I'd really appreciate to know what settings are best with sound, and what settings are best without sound. Since I'm using the t3i, RAW at this point isn't my priority, but getting the most out of h.264 is. I love this development, because I can still shoot video for commercials, etc. without having to deal with the raw workflow. Although, if mjpeg were still possible, then that would be awesome as well, but that seems more of a long shot now.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on June 28, 2013, 06:15:47 PM
EOS fixer is just for >4gb movies... it runs on the PC.


Set the BR somewhere above 3x, flush of 3 or 4 and gop of 1 for all I if you want it. Usually non all I gop can be FPS but I'm not sure if that works, haven't tried recently.. 3, 6, 12, 15 are all ok. Mainly you want it keeping the quality set high and writing the frames out faster.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: vanburen08 on June 28, 2013, 07:21:43 PM
Thanks, that helps tremendously. And these two settings are good for sound?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on June 28, 2013, 07:44:31 PM
Sound you can't change flush rate... I don't think gop either. Wav is fine but regular canon sound is not.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: wenen on June 28, 2013, 11:19:23 PM
I just want to say that the old build with Slice 87 is awesome, 120MBs with sound on!

F*** RAW. It is not working good on 600D anyway and to big file size to have on a computer.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Scipione205 on June 29, 2013, 01:04:12 AM
I didn't udnerstand what't this 'old build' you're talking about, and what are the reccomended settings?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: N/A on June 29, 2013, 02:47:16 AM
I shot a few vids with TL 1, the slice 87/gop 1 combo is damn pretty, especially with a nice picture style. Very detailed with a natural sharpness to it.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: wenen on June 29, 2013, 01:46:03 PM
Quote from: Scipione205 on June 29, 2013, 01:04:12 AM
I didn't udnerstand what't this 'old build' you're talking about, and what are the reccomended settings?

Old: https://bitbucket.org/OtherOnePercent/tragic-lantern Settings: 25p, sound on,CBR 2,0x, DblockA/B -6,-6, picqPC 0, GOP3, JP Slice 87, Slice Lock: Enabled, MinBR :80, Max:125, Drop By 1: 85, drop by 3: 100, Taper rates: enabled

New: https://bitbucket.org/OtherOnePercent/tragic-lantern-2.0 This one have RAW
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Critical Point on June 29, 2013, 04:18:55 PM
Can someone please post the absolute best settings for H.264 without audio ?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: N/A on June 29, 2013, 11:08:17 PM
Wenen's settings but with sound off in 1080p/24p mode, taper rates disabled, dblock both to -1. After you change settings restart the camera.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Critical Point on June 29, 2013, 11:45:10 PM
And if I want to use 30 fps, is that a problem ?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: N/A on June 30, 2013, 12:58:40 AM
Never tried 30p, one way to find out.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: far.in.out on June 30, 2013, 11:40:10 AM
For me it stops after 1-2secs. WTF? How do I adjust for lower target bitrate?

TL2.0 I can go as high as 110Mbps GOP3 CBR x2.2...
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Critical Point on July 02, 2013, 04:14:39 PM
Is there a normal Magic Lantern 2.3 + the new H.264 features added on ? I mean for those settings:

30p, sound off, CBR 2,0x, DblockA/B -1,-1, picqPC 0, GOP1, JP Slice 87, Slice Lock: Enabled, MinBR :80, Max:125, Drop By 1: 85, drop by 3: 100, Taper rates: disabled.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: N/A on July 02, 2013, 04:40:46 PM
Afaik it was only available in TL 1, when I was using it I could hit a steady 160 mbps, which is pretty much max.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Critical Point on July 02, 2013, 05:18:57 PM
In that case what build should I use for those settings ?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: N/A on July 03, 2013, 08:42:13 AM
https://bitbucket.org/OtherOnePercent/tragic-lantern/downloads

Top in newest
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Critical Point on July 03, 2013, 08:12:39 PM
I got a 32 Gb Sandisk 45 MB/s SDHC card, formatted in exFat, and ML 2.3 on it running, now I just download that autoexec.bin-bitrateback file, rename it to autoexec.bin, overwrite it to the SD card, and that is all ?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: N/A on July 03, 2013, 10:19:58 PM
You got it
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: jgharding on July 04, 2013, 06:22:46 PM
Because there are a lot of starter questions in this thread now, I made a new FAQ thread:

http://www.magiclantern.fm/forum/index.php?topic=6913.0
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: AriLG on July 04, 2013, 07:24:08 PM
Quote from: jgharding on July 04, 2013, 06:22:46 PM
Because there are a lot of starter questions in this thread now, I made a new FAQ thread:

http://www.magiclantern.fm/forum/index.php?topic=6913.0
Yes, but it relates only to Tragic Lantern V1. there is a V2, with a lot less control (as of now) but there IS one...
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: vfx_junkie on July 04, 2013, 09:16:33 PM
1% can you put those options in TL 2.0 maybe ? =)
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: jgharding on July 04, 2013, 09:50:12 PM
That's right, but for now it's fine just to use that build, I gave it about 20 hours or so already and it's good and stable.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on July 04, 2013, 10:32:52 PM
It has to be rewritten because of new menus, etc.

Also remember buffer warn is also the level it will start lowering slices so if it stops on TL 1, lower the buffer warning level.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: AriLG on July 04, 2013, 10:49:43 PM
Nevermind
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: jgharding on July 05, 2013, 01:01:15 AM
Ah yes I should add that to the post about buffer warning.

I just figured it may be a good first stop for people trying to get to grips with it, hopefully it will help someone.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on July 05, 2013, 01:33:21 AM
I don't think you can get over FPS gop.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: AriLG on July 05, 2013, 09:06:41 AM
ah, ok
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: AriLG on July 11, 2013, 10:53:12 PM
Will the Bitrate development commence along with the RAW development ? are there further developments down the line ?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on July 11, 2013, 11:50:16 PM
Heh, will commence when some of these big summer projects are over. I think everyone is having the same problem, hence not so much new stuff.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Critical Point on July 12, 2013, 01:07:41 AM
Quote from: 1% on July 04, 2013, 10:32:52 PM
It has to be rewritten because of new menus, etc.

Also remember buffer warn is also the level it will start lowering slices so if it stops on TL 1, lower the buffer warning level.
But lowering the buffer warning level what does that mean ? From 70% to 95%, or from 70% to 50%, in witch way is lowering ? My thoughts are that if put to 95%, it means that only then it will reduce the video quality, right ? Only when it reaches 95% the slices will drop.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: deletedAcc.0021 on July 18, 2013, 12:48:44 AM
Quote from: 1% on June 28, 2013, 06:15:47 PM
EOS fixer is just for >4gb movies... it runs on the PC.


Set the BR somewhere above 3x, flush of 3 or 4 and gop of 1 for all I if you want it. Usually non all I gop can be FPS but I'm not sure if that works, haven't tried recently.. 3, 6, 12, 15 are all ok. Mainly you want it keeping the quality set high and writing the frames out faster.


These settings are perfect 1% ... 3X, Flush4 and GOP3.  Getting an average 85 mbps, with slightly higher for high ISO scenes.

thanks!
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Critical Point on July 18, 2013, 02:56:16 AM
30p, sound off, CBR 3.0x, DblockA/B -6,-6, picqPC 0, GOP1, JP Slice 87, Slice Lock: enabled, MinBR :130, Max:160, Drop By 1: 140, drop by 3: 145, Taper rates: disabled.

Are these the best settings for H.264 ?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on July 18, 2013, 03:46:35 AM
50% is what I use. 95% it will never lower or catch in time.

@CP

Well that is fixed max + all I so if you can handle it then its not going to go any higher. CBR doesn't do anything if not using canon CBR, anything over 1 is good.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Critical Point on July 18, 2013, 04:25:57 PM
Funny, because yesterday I was shooting with these settings from a moving car, and I never got higher bitrates than 104 mbs, but I has filming at ISO 80, maybe that is the cause. Filled two 32 Gb cards with no problems, no stops, no nothing. My buffer warning is set to 70%, but from what I see on the screen, it never gets past 8%, it goes 2-3%, 5%, back to 1%, 2-3% and so on.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Rush on July 19, 2013, 12:56:05 AM
Quote from: 1% on July 05, 2013, 01:33:21 AM
I don't think you can get over FPS gop.
But wait, I tried GOP=100 (TL v2) and it worked. Virtual dub reported 1 key-frame every 100 frames
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: jjproducciones on July 19, 2013, 02:44:17 AM
this works for the Canon 550D?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Critical Point on July 23, 2013, 03:04:50 PM
I am a little bit confused. If I understand correctly GOP 1 uses QScale -12, and GOP 3 uses QScale -16 ? And in this case, what is the best option for better video ? GOP 1 or 3 ?

I know GOP 1 is All-I, but is this more important than QScale -16 ?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on July 23, 2013, 03:33:41 PM
Gop is independent from qscale. Since canon is still controlling quality it might just be stopping at 12? On TL1 I could do 1 and the same slice value.


Lower iso will produce lower rates.. .I think at higher iso you're mainly encoding bouncing noise tho.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: ilguercio on July 23, 2013, 03:53:06 PM
I am still trying to figure out the best settings in case i wanted MAX quality and i could not care about size.
I am trying to record with:
Bitrate:20
Init QP:1
Flush rate:4 frames
GOP size:1

What does Config Select mean?
Also, why my camera locks up with these settings?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Critical Point on July 23, 2013, 05:14:16 PM
Quote from: 1% on July 23, 2013, 03:33:41 PM
Gop is independent from qscale. Since canon is still controlling quality it might just be stopping at 12? On TL1 I could do 1 and the same slice value.


Lower iso will produce lower rates.. .I think at higher iso you're mainly encoding bouncing noise tho.

If I disable the Slice control and shoot only with CBR 3.0x, with GOP 1 it goes only as high as QScale -12, but with GOP 3 it goes all the way to QScale -16. What I don't understand is what is there to gain/lose by using either GOP 1 or QScale -16, what is more important for the quality of the video ? GOP 1, or QScale -16 (GOP 3) ?

And another thing that I don't understand is what happens to the QScale under Slice control, for instance if slice set to 87, that means what in terms of QScale ?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: CinematicSyndicate on November 04, 2013, 10:03:22 PM
To respond what what someone said earlier - I have also noticed the bitrate controls are gone from the nightly build I have...yet there is a LOAD H.264.INI option that seems to do nothing when I press it. Am I missing something here?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on November 04, 2013, 11:44:36 PM
That sounds like 5DIII which is a different encoder.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: CinematicSyndicate on November 05, 2013, 03:11:50 AM
It is indeed - ideas?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on November 05, 2013, 04:33:36 AM
Compile off my repo I guess and enable the 6D controls for 5DIII.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: CinematicSyndicate on November 05, 2013, 06:58:07 AM
Gotcha. Are the bitrate controls slated to be added back to the 5D3 build in the near future?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on November 05, 2013, 04:43:31 PM
I doubt because a1ex didn't like the code and 5DIII is very raw friendly.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: CinematicSyndicate on November 06, 2013, 12:10:30 AM
It would be helpful to have a happy medium, I think, between pure RAW and crappy h.264 for footage-intensive shoots...
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: gary2013 on November 06, 2013, 12:15:53 AM
I was thinking the same thing.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Yoda on December 08, 2013, 04:02:39 PM
So if the 5d3 nitrate controls aren't coming back, anyone know of the last build with the nitrate controls in?  I really liked this feature on the 550D, and I just picked up my 5D3, and loading the latest nightly build, noted that it's gone.

Raw is awesome, and even when having a solid infrastructure to support the post-production workflow, I find it's practical to have a high bit rate control for h264/MOV.  Stuff like birthdays, events, or demo-type of footage has its uses, and fast turn around time has it's advantages. 

Cheers,

Yoda
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on December 08, 2013, 06:09:02 PM
Dunno, I can make a new one some time if you want. Just have to make 5DIII on my repo.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Yoda on December 08, 2013, 08:01:19 PM
Alex, that would be awesome if you put it back.  #hero #awesome #bitrate   ;D
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on December 08, 2013, 10:32:09 PM
lol I'm not a1ex but i can upload a snapshot.

make sure it still has bitrate controls, if not I'll look at the build config

https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/downloads/magiclantern-Tragic.2013Dec08.5D3113.zip

remember I don't have a 5DIII
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Yoda on December 09, 2013, 12:55:17 AM
Sorry bout that mate.  But I will def check it out and report back.

1%, Side note:  Do you have a thread to share where I can report back some results using the latest 5D3 nightly build?  I got some near over heating after recording just 30 seconds on RAW. (51 degree C light popped up).  I thought perhaps I might share my info in case it would help the dev effort. 

Cheers,

Yoda
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on December 09, 2013, 04:19:47 AM
I guess open an overheating thread? Easy way to do it is lots of indicators + gd on.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Yoda on December 09, 2013, 07:57:34 PM
Fyi, I loaded your build from Dec 08, no bit rate controls showed up.  Only h264.ini option was there.

On the over heating, I disabled all the global draw stuff except for live view.  Maybe I should disable that as well.  I was using FPS over-ride, and some basic parameters in exposure over ride for iso, shutter, etc, but nothing out of the ordinary.  Hmm....
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on December 09, 2013, 08:26:52 PM
FPS is very likely to do it especially at reduced FPS. Digic V alternates FPS somehow and I think its to reduce load on the sensor... otherwise why

Ok, fixed it.. its still recommended to load configs (but bitrate-5d3 doesnt work!), they are in this thread. You can probably just do fixed QP tho since its CF and all, read the other posts.

http://www.filedropper.com/magiclantern-tragic2013dec095d3113
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: wenen on December 09, 2013, 09:49:35 PM
Quote from: 1% on December 09, 2013, 08:26:52 PM
FPS is very likely to do it especially at reduced FPS. Digic V alternates FPS somehow and I think its to reduce load on the sensor... otherwise why

Ok, fixed it.. its still recommended to load configs (but bitrate-5d3 doesnt work!), they are in this thread. You can probably just do fixed QP tho since its CF and all, read the other posts.

http://www.filedropper.com/magiclantern-tragic2013dec095d3113

This build is only for 5D mark III? You have any new build for 600D and new settings?
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: 1% on December 09, 2013, 09:53:31 PM
600D encoder is 100% different. No new 600D yet.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Yoda on December 10, 2013, 06:01:47 AM
Interesting.  Ok, I'll def not do fps over ride....was only doing 24 exact, lol.

I'll load the latest build and read up.  You totally rock for the awesome support man!  :)

Cheers,

Yoda
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: g3gg0 on December 11, 2013, 12:17:30 AM
Quote from: 1% on December 09, 2013, 08:26:52 PM
Digic V alternates FPS somehow and I think its to reduce load on the sensor... otherwise why


wasnt it for dithering towards the perfect accurate frame rate?
Title: Re: Bit rate investigation
Post by: 1% on December 11, 2013, 05:18:17 PM
You're most likely right, either way it gets hotter when FPS override is on. Especially low FPS
Title: Re: Bit rate investigation
Post by: Datadogie on December 11, 2013, 10:07:52 PM
Yoda the 600d has 24, 25, 30, 50 and 60fps so no need for ML to adjust these.
Title: Re: Bit rate investigation
Post by: g3gg0 on December 12, 2013, 11:32:31 AM
Quote from: 1% on December 11, 2013, 05:18:17 PM
it gets hotter when FPS override is on. Especially low FPS

obvious as the sensels are powered longer.
Title: Re: Bit rate investigation
Post by: arrinkiiii on December 12, 2013, 12:34:01 PM
Quote from: 1% on December 11, 2013, 05:18:17 PM
You're most likely right, either way it gets hotter when FPS override is on. Especially low FPS

Yes, it's true. About 1 month i try very low fps override and the camera just warning me that is hot temperature and self disconnected, in about 15 minutes.
Title: Re: Bit rate investigation - altering ratio between P and I frame sizes
Post by: Yoda on December 15, 2013, 01:40:55 AM
Quote from: 1% on December 09, 2013, 08:26:52 PM
FPS is very likely to do it especially at reduced FPS. Digic V alternates FPS somehow and I think its to reduce load on the sensor... otherwise why

Ok, fixed it.. its still recommended to load configs (but bitrate-5d3 doesnt work!), they are in this thread. You can probably just do fixed QP tho since its CF and all, read the other posts.

http://www.filedropper.com/magiclantern-tragic2013dec095d3113

Link doesn't work?  and I'm still catching up on load configs....what is this for?  Fixed QP?

Thanks 1%,

Yoda
Title: Re: Bit rate investigation
Post by: wenen on December 23, 2013, 04:51:15 PM
1% we want a tragic lanter christmas gift to 600D :)
Title: Re: Bit rate investigation
Post by: Afterimages on December 23, 2013, 07:02:06 PM
+1000
Title: Re: Bit rate investigation
Post by: eyeland on March 02, 2014, 01:40:58 PM
I am still confused when it comes to manipulating h264..
Are there ways to utilize the "load h.264.ini" found in the ML nightlies? Or are you rather talking about a different set of options found in the TL?
Anyone had any positive results tweaking the 5Dmkiii h.264 ? Can't really figure out if this approach was abandoned due to the awesomeness of RAWvideo or simply because it didn't improve much.