7D at Halloween: The Bitrate-Monster (EXPERIMENTAL)

Started by g3gg0, October 28, 2012, 10:42:46 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

P337

Quote from: Digital Corpus on November 01, 2012, 07:58:06 AM
It's ~f/6.9 for the 7D due to the 18 MP sensor. However, with 1080p video, you're then talking about effectively a ~2.1 MP sensor so the airy disk can technically be larger. On top of that, since the video is created with line skipping, anything over f/7.1 may still in hibit resolution. One more thing I want to look into aside from the pixel-peeping effects of the VAF-7D.

Thanks Digital! You're always getting me to double check my work and learn more lol :D

You are right the Ariy disk does begin to overlap pixels around f/6.3 but it isn't really noticeable until f/8 so most people consider that diffraction limited.

Unfortunately I think with "line skipping" to down-res the image to 2.1MPs we do not benefit from an increased diffraction limit like we did with Photo mode's pixel binding method which physically makes each pixel sensor larger rather then in the line-skipping method we still have the same sized (4.3micron in this case) pixel sensors :/

P337

Quote from: akshayverma1 on November 01, 2012, 10:51:31 AM
Yes, those are single frames taken from video. Average bitrate for the newsprint was about 150. Peak was 170. For the tree, average was 210, and peak was 260.

I did some high ISO tests with well lit trees last night and got some amazing bitrates - 290 peak and 260 average. It's 35 seconds and 1.04GB. :)

Thanks Akshayverma!

Those are very valuable numbers, my fastest card is averaging about 150 and peaking at 170 too (with a flush rate of 2). 

I found the best way to max out bitrate (for testing purposes only) is to set ISO to max (ISO 128,000) with a fast shutter (250 - 4000 depending on lighting) and point your camera at something that spotmeters under 75 on the 0-255 scale; this will shoot your bitrate up to max right quick!

Set to 20x with a Flush rate of 2 my "200x CF" card (benchmarked at 21 MB/s) can record 200 Mbps for one second after filling the buffer and shutting down recording but I found I could stably record anything to the full 4GB limit at a fairly constant 160 Mbps, which is the peak limit for 3.5x :/ guess I'll need those "600x CF" cards after all lol

I'll try some 160 Mbps high noise and tree details later today lol.

Digital Corpus

Quote from: P337 on November 01, 2012, 12:19:46 PM
Thanks Digital! You're always getting me to double check my work and learn more lol :D

You are right the Ariy disk does begin to overlap pixels around f/6.3 but it isn't really noticeable until f/8 so most people consider that diffraction limited.

Unfortunately I think with "line skipping" to down-res the image to 2.1MPs we do not benefit from an increased diffraction limit like we did with Photo mode's pixel binding method which physically makes each pixel sensor larger rather then in the line-skipping method we still have the same sized (4.3micron in this case) pixel sensors :/
You're welcome. Most people refer to f/8 since it's the closest full stop measurement near the diffraction limit :). I don't currently have the facilities to test where we start loosing detail between the 18 MP sensor and the resulting 1080p video. However, this is what I've looked at so far...


Ok, so I've done some testing myself. The only consistent scene I've been able to record (partially overcast day) is a patch of grass. Detail here is similar to leaves, but with a lot less movement. Using the all I-frame firmware, I acheived a max average bitrate of ~281 Mbps using CBR @ 20x and the flush rate at 4 frames. I had ML stop recording at 60 seconds and the buffer started out at 28% and then leveled off at 22-23%.

All recording was done with a VAF-7D installed as well and sharpness set to 1. I have screens showing a significant increase in perceptable detail, but I don't have a quick enough of a connection to upload them at this time as they are ~5 MB png files. The artifacting that I previously saw in my test scene with leaves showed scaling artifacts when in-camera sharpening was set to 0. With the VAF-7D, I saw none of these artifacts, even on the grass blades.

I noticed it was a bit hard to get the bitrate above ~230 Mbps for the scene. I had only 1 issue with my CF card not writing out all of the frames and throwing an Error 70 message, but I only had ~70 MB left on the card so I'm assuming it was due to fragmented freespace. When I did this scene for testing the speed of the card w/ the bitrates, I adjusted the 12 points (not worth the words to explain why) on a logarithmic scale (e^(x*ln(20)/11)) from 1 to 20. The file sizes were as follows:
1x   333 MB
1.3x   408 MB
1.7x   531 MB
2.3x   744 MB
3x   975 MB
3.9x   1.2 GB
5.1x   1.62 GB
6.7x   1.73 GB
8.8x   1.63 GB
11.6x   1.66 GB
15.2x   1.72 GB
20x   1.62 GB (ended ~2 seconds early)

These were I-Frame btw... Based on these numbers and the ones others have produced with similar tests, it appears that the sweet spot of recording is 4x-7x. I'll see what I can do for some controlled tests in this area because it appears that the encoder is currently hitting a wall. No reason to give users options to go beyond 10x atm. When/if we establish a best-case upper limit, we could just set it there so we don't have people trying to figure out if they should push their hardware harder?
7D w/ ML | Tokina ATX 11-16 | Canon 24 mm pancake | Canon 40 mm pancake | Canon 17-55 f/2.8 IS | Sigma 150-600 Sports

feureau

Quote from: Digital Corpus on November 01, 2012, 02:27:12 PMWhen/if we establish a best-case upper limit, we could just set it there so we don't have people trying to figure out if they should push their hardware harder?

Please don't do this.

1%

Upper limit not really possible. There is a "max" quality but not set with predictive CBR, rate still varies.

g3gg0

thanks a lot for these comprehensive tests!

in the meantime i made both VBR and CBR selectable and configurable.
the camera now also displays the Q factor as known from other models.

@1%:
> You guys haven't pushed luma/chroma quality up yet
you mean QP(Y;U;V)? that is according to analysis tools exactly the same as QP (10).

one correction: you said, camera flushes after each GOP - you meant after each two GOPs, as this is one second :)
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

1%

I thought it was after each gop, maybe its 2, would have to double check.

Chroma QP and Luma QP should be separate parameters, it was on 600D. Couldn't set 10 (max 20 vs 26) or camera would freeze. I used H.264 analyzer and it never matched QP. I think they are really chroma_qp_index_offset and luma_qp_index_offset in H264.

Like: http://opencores.org/websvn,filedetails?repname=nova&path=%2Fnova%2Ftrunk%2Fsrc%2FQP_decoding.v
See: http://books.google.com/books?id=jlL1xqOry2QC&pg=PA192&lpg=PA192&dq=QPy+QPC&source=bl&ots=IOBIt7POyT&sig=vhmlA2bbHzohta62ovma4OeJViI&hl=en&sa=X&ei=LpOSUPbTOsPXyAGQroBY&ved=0CCUQ6AEwAjgK#v=onepage&q=QPy%20QPC&f=false

Also: http://thinknaturally.blogspot.com/2009/01/h264how-to-do-conversion-from-8-bits-to.html

Just had a thought, is 4:2:2 YUV being fed to the encoder or is it 4:2:0.. I think its set via digic register.

Set H264

*0xC0E10004 = 0
*0xC0E10050 = 0

YUVlossless

*0xC0E00004 = 65538
*0xC0E00050 = 0
*0xC0E0008C = 0x111121

YUV: 4:2:0

*0xC0E00004 = 65538
*0xC0E00050 = 0
*0xC0E0008C = 0x111141


Also *0xC0E1000C is interesting. Set to 41104, 41091, 8090

feureau

Quote from: g3gg0 on November 01, 2012, 04:10:04 PM
in the meantime i made both VBR and CBR selectable and configurable.
the camera now also displays the Q factor as known from other models.

Awesome! :D Awesome to the max.

mimiloveyou

Yes!!! Yes!!! Yes !!! Yes!!! Yes!!! Yes!!! Yes!!! Yes!!!

g3gg0

Quote from: 1% on November 01, 2012, 04:26:46 PMCouldn't set 10 (max 20 vs 26) or camera would freeze. I used H.264 analyzer and it never matched QP. I think they are really chroma_qp_index_offset and luma_qp_index_offset in H264.

hoho, holy shit...
tried setting  PICQPC to 1.
this results in chroma_qp_index_offset being set to 1 and QPY 50, QPU/V 39.
the 1280 x 720 video looks like shot with a 320x240 webcam.
basically every 16x16 block is "worth" a pixel.

same if i set  chroma_qp_index_offset to -1..

when setting PICQPY from 0x1A to 0x19, then the pic_init_qp_minus26 goes down by one from -1 to -2
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

feureau

Quote from: g3gg0 on November 01, 2012, 09:57:14 PM
hoho, holy shit...
tried setting  PICQPC to 1.
this results in chroma_qp_index_offset being set to 1 and QPY 50, QPU/V 39.
the 1280 x 720 video looks like shot with a 320x240 webcam.
basically every 16x16 block is "worth" a pixel.

same if i set  chroma_qp_index_offset to -1..

when setting PICQPY from 0x1A to 0x19, then the pic_init_qp_minus26 goes down by one from -1 to -2

Is this about the line skipping/pixel binning thing the canons do?

.... Does it mean that it's possible to increase resolution if you set it to the other direction? Like reversing the polarity and all that star trek-y mumbo jumbo?

Anyway, I wonder if anyone can confirm this: if you have ML on, and you're on the bitrate menu, and probably highlighting flush at around 4,6,7, or 8 and half pressing the shutter button would just shut down the camera. Probably a bug or just my 7D doing weird stuff. I've been trying to see how fast I can push one of my cards, a slow "cheap" transcend 133x card.

g3gg0

Quote from: feureau on November 01, 2012, 10:25:46 PM
Is this about the line skipping/pixel binning thing the canons do?

.... Does it mean that it's possible to increase resolution if you set it to the other direction? Like reversing the polarity and all that star trek-y mumbo jumbo?


errrhh... definitely: no! ;)
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!

1%

It didn't work setting it using the function for me.

Picqpy lowest was 20, stock is 26. any other lower would cause "busy" and camera freezes. Could only set it via address.

PicqPC let me move it +-7 with the function.

According to H.264 20 is better than 26. And picqpc is connected to the picqpy range. If I moved PC after setting py to 20 camera would freeze.

Finally seeing those terrible QP's? Can make like 8MB video out of those :)

P337

Quote from: g3gg0 on November 01, 2012, 09:57:14 PM
hoho, holy shit...
tried setting  PICQPC to 1.
this results in chroma_qp_index_offset being set to 1 and QPY 50, QPU/V 39.
the 1280 x 720 video looks like shot with a 320x240 webcam.
basically every 16x16 block is "worth" a pixel.

same if i set  chroma_qp_index_offset to -1..

when setting PICQPY from 0x1A to 0x19, then the pic_init_qp_minus26 goes down by one from -1 to -2

lol, is that cause PicQpC needs to be set by PicQpY?  Rather can changes be made directly to PicQpC without adjusting PicQpY?
(Forgive me if that question makes no sense cause I'm not entirely sure I even understand what I just said lol)

Anyway is that "720 looks like 240" trick basically what happens when setting a low CBR like 0.3x.  Should we expect a similar loss of quality when setting bitrate that low?

1%

Yes, you can adjust both independently.

720 looks like 240 is a product of really high QPs (low quality). I can do the same thing but I'm setting the parameter that controls QP directly rather than relying on CBR. When I set picqpy/pc the overall QP doesn't change. Increasing those produced higher BR for same QP.

If you set CBR to .3x it will pick QPs like 25/30/40, etc and look like pixelated blocky mess.

P337

Thanks 1%

I think I'm starting to get it; they can each be adjusted on their own but pY is still connected to pC so when you set pY to 20 (it's maximum) then is pC also pulled to it's maximum too?  Is that's why any changes to pC after pY is maxed out causes a freeze?

1%

Theoretically you should be able to change throughout whole range. Maybe on 7d you can but it messes with other things or doesn't work as intended. CBR could also be setting lower QPs because the parameter was changed. 

https://bitbucket.org/OtherOnePercent/tragic-lantern/src/0ec03c27a34f01ad4b04021cd20aec91b6b146a2/src/bitrate.c?at=unified

Is how I change it but I have hard-coded addresses.

P337

To better test these high bitrates I'm trying to decide between getting a 600x or 400x CF card.  The 600x benchmarked at an avg 640 Mbps (80MB/s) write speed, the 400x I can't find benchmarks for it but their 300x benched at 292 Mbps (36.5MB/s) write speed which is about the fastest we've seen on the 7D.


So do you think this 300Mbps limit is a camera or card speed limit? 

Has anyone achieved any peak numbers over 300 Mbps? 

Do you guys think it's worth even trying a 600x (I doubt it will be)? 
If 300Mbps really is the camera's limit then that's about CBR 6x (right?), so setting any higher values than that or using a card that benchmarks over 37.5MB/s won't make any difference.


If you have a super fast card I recommend setting CBR to 20x and Flush to 1.  Then enable spotmeter and select the 0-255 option.  Then set ISO to 128,000 and use your shutter or ND filters to underexpose to "25" on the 0-255 spotmeter.  This always shoots my bitrate up to it's max but my cards, which are "200x" 170Mbps (21MB/s)*, peak at 200 Mbps for 1 second then the buffer can't keep up** so the recording stops :/

*ML Benchmarked my card at 21.5MB/s
**my Flush rate is set to 2 (my camera throws err70 if flush=1)

1%

The gop length assert has to be patched, is it not?

You can try a faster card if it isn't too expensive... ie under 100usd. The best that you'll probably do is steady 2xx MBPs. When I patched for quality I got that at like iso 320 in daylight with trees. Most of the time BR doesn't go much above 120-160 but you're future proofing for mjpeg/yuv dumping, recording audio alongside video, etc.

Also need to see if write does go above 35mbps, on SD cameras the max is 25mbps per the controller.

mimiloveyou

   g3gg0

         "in the meantime i made both VBR and CBR selectable and configurable.
          the camera now also displays the Q factor as known from other models."



   Give news please.
   We try CBR and VBR.


mimiloveyou

 How can I do with these things.
I am not a technician.

I want to understand.
how to do.

Thank's.


Sorry for my bad english.

Calomile

Definitely take heed of the advice in the first post about overheating, my camera came perilously close at 190Mbit/s for a prolonged recording. Works fantastically well, however! Can see this being useful for getting ultra clean looking footage for twixtor etc. Great work!

P337

Quote from: mimiloveyou on November 03, 2012, 12:25:21 PM
How can I do with these things.
I am not a technician.

I want to understand.
how to do.

Thank's.


Sorry for my bad english.

It doesn't seem to be working yet, looks like only the first few frames in a recording are accepting the CBRs.

They are however making pretty good strides in the MJPEG encoder thread, looks like the 7D can take 1056x704 4:2:2 JPEGs from the buffer and record them to the card without needing to be connected to the EOS Utility through USB ;D
http://www.youtube.com/watch?v=jQLHkXL-AWU

g3gg0

here is another experimental build.

i set GOP to 4 and enabled VBR/CBR selection.
also the flush rate is now limited to 2-50.
flush rate is on both digics set via cache hacks now - this means it is less likely to abort recording after the first 100ms, although
it is likely to happen if you start recording the first time, as the drivers seem to read some filesystem information which takes some time (?)
successive recordings will fail less likely.

what do you think about that?
i think i will make the GOP selection configurable somewhen, so you can choose your favorite.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: [email protected]
ONLY donate for things we have done, not for things you expect!