Okay..
As some guys asked for thorough tests of what the high bitrate reported in this thread (http://www.magiclantern.fm/forum/index.php?topic=3103.350) really look like,
i decided to post an EXPERIMENTAL release of the 7D bitrate hack.
i repeat. EXPERIMENTAL.
what does that mean?
1) it is only intended for people with decent technical knowledge, because...
2) ... you have to be aware of the possible negative side effects like overheating ....
3) ... or some unrecoverable crash of your camera.
4) and it would only make sense to test if you know what bitrate means :)
4) although i don't think this will happen - the risk for your camera is higer as with the alpha
5) the usual "be warned" stuff i always tell you still applies ;)
okay. i have to say that, you know. :)
back to the bitrate hack.
what is it about?
with ML we can drive the bitrate up to factor 3.0x then, depending on your card speed, recording stops.
this is because of the recording buffers that fill too fast.
these buffers are cleared about ONCE per SECOND - if you set 25 fps, they will get written to card after the 25th frame.
if we would set e.g. 10x rate, the buffer would be full after half a seconds or so.
thats the reason for the recording that stops.
what does this hack do?
it flushes the buffers more often. this is configurable.
i pre-set it to 4 frames which works quite well with my 30MiB/s CF card and a rate of < 9.0x.
on 7D this also requires some cache hacks in master firmware.
porting that other models is a lot of simpler. (imho)
what do you want to test?
a) test how far your card can go up.. set bit rate higher and test high detail scenes. report what your highest stable bitrate was and your card type (with benchmark speed).
b) check if the hack is worth its effort. is the video quality good? or would we stick better with 3.0x and this hack is useless?
known issues:
- ERR70 may happen if your card is too slow and/or the flush rate is too low
- important: disable sound recording, else you will get an ERR70 too
- it seems to be not CBR, but VBR although it says CBR
i really would love to see some deep analysis of the videos and your conclusion - is it worth to get implemented in all models?
or should i just drop this code and stick to other things?
here (http://upload.g3gg0.de/pub_files/7373e7c0539b3f631f45be992a13a013/7D000203_ML.FIR) is the DL link for this experimental version.
I thought 3.0x was the maximum allowed but looks like it's nothing compared to 9.0x.
Files are going to be HUGE and i unless you're doing a short movie and have a high end computer to process the files most of the people don't want to shoot as such high bitrates. Don't get me wrong, your release has the purpose of being tested by movie guys to see if such stuff is worth. What are your early opinions?
I knew CF would stomp SD. I bet 7D can shoot max quality most of the time and never overflow.
When I was just scaling CBR eventually the numbers overflowed and turned negative. But 7d is probably more like 5dII with different MVR config.
9.0x CBR is something like 3Gb per minute, quite a lot for any "casual" application. Let's see what our users can do with that.
Looks very interesting ! Could you please port it to work on 550d/T2I ?
Yes indeed, it would be nice to test how far a stupid camera like the 1100D can go :P
8)
Can't go any higher than maximum QP of the encoder. Low end cameras are like 600D. Theoretically I could do BR that high too but SD card is too slow or not enough data unless you crank the ISO.
CBR 9x is just going to make the predictor set highest quality all the time, I tried it, values overflow.
5DII and 7D have a slightly different mvr config + faster card rates.
Ran some simple test with Lexar Professional 1000x 32GB
card Benchmark 20.8, 20, 53.8
Recording at night iso 6400
Tested the Defult values = ERR70
UPDATED October 29: After I disabled the Audio then I was able to record much longer videos. Disregard the results below.
Test flush rate (CBR, 1X, OFF, 4, 70%)
Increase flush rate to 8 = record 4 sec no playback on camera
Continue to increase flush rate @ 10 = record 4 sec and 1 sec playback on camera
Continue to increase flush rate @ 15 = record 6 sec and 1 sec playback on camera
Continue to increase flush rate @ 20 = record 8 sec and 1 sec playback on camera
It would seem that higher flush rate allows longer recording and buffers do no fill up.
Test BuffWarnLevel (CBR, 1X, OFF, 20, 70%)
Using previous best settings of Flush rate= 20
Increase BWL 90% = record 8 sec and 1 sec playback on camera
Increase BWL 100% = record 8 sec and 1 sec playback on camera
Decrease BWL 30% = record 8 sec and 1 sec playback on camera
Conclusion, BWL has no effect on recording
Test CBR begining settings ( CBR, 0.1X, OFF, 20, 90%)
Increase CBR 0.1 = record unlimited time and 1 sec playback on camera
Increase CBR 0.2 = record 15 sec and ERR 70 shut off
Increase CBR 1.4 = record 5 sec and 1 sec playback on camera
Increase CBR 2 = record 3 sec and 3 sec playback on camera
Increase CBR 3 = record 1 sec and no playback on camera - Camera locks up when trying to play
Increase CBR 4 to 9 = record 1 sec and 1 sec playback on camera
Increase CBR 10 = record 1 sec and no playback on camera
Increase CBR 11 to 20 = record 1 sec and 1 sec playback on camera
Conclusion: longest recording at cbr=1, longest playback at cbr=2
Tomorrow I will try to do some recording in the afternoon and to look at video
please make sure that audio recording is disabled
else you will face an err too
I'm using a SanDisk Extreme 8GB 60mbps card. I just shot some trees.
ISO 160
Shutter 1/250
Aperture f/11
Flush rate: 4
CBR 20.0x
Focus peaking, magic zoom, histogram, waveform and bitrate display were ON
Audio OFF
I recorded 1 minute, 42 seconds of handheld footage. Recording was stopped by me (it would keep recording for the whole 4GB). The file size is 2.38GB.
As g3gg0 rightly said, it's actually VBR. I got a maximum bitrate of 218 while the average was around 170. Although I focused it pin-sharp, the footage is blurry as is all 7D footage. Visually it looks just like ordinary 1x BR to me.
Could anyone please guide me about doing a better comparison?
Don't expect the footage to be pin sharp just because you raised the CBR value.
The problem at the basis on that fuzzyness is how the encoder scans the sensor. It's not downsampling it but skipping lines so you're not going to have amazing quality just because of the bitrate increase.
Thanks for the answer, ilguercio.
I shot some more footage, this time water coming from a tap with lots of fast-moving bubbles. The difference is clear between bitrates lower than 1x, and 1x. There's a lot of compression artifacts at lower bitrates.
I also shot a ceiling fan at 1/500s. Again, there is only minimal difference between 1x and 20x. I have tried applying an LUT (shot in cinestyle), color grading, denoising, and over-sharpening. 20x is only minimally better; it still has some artifacts on the moving blades. Is this because it's VBR?
thanks for that investigation.
has anyone an idea where the higher bit rate comes from?
e.g. are all frames effectively (or nearly) I-frames with only few P-blocks?
i have a demo version of the tool AVC Analyzer that is made especially for this kind of analysis,
but i can not analyse the files because the tool crashes with these files :(
Avinaptic will tell you the % of frames that are X QP at least.
Details were small if you look through 600D tests. Visually hard to tell until you grade/convert. Only visible when you zoom in. Rencoding to things like cineform keeps the file sizes so there is more data in there.
okay, i shot a comparison that AVC Analyzer could handle.
now some details what the higher bitrate causes.
i shot two videos of a wall with little details, high iso and high f-stop value.
one video with 1.0x and one with 10.0x
first frame (#0): both videos have an I-frame with ~500KiB size. QP of this frame is in both videos 18.
second frame (#1): is P and uses in both videos ~430KiB. It consists of ~89% Intra MBs (MacroBlock) and also both have QP 18.
third frame (#2): the things change.
both videos still have ~89% Intra-MBs. this doesnt change at all in the whole video.
but the QP and thus the size differs slowly. the 1.0x video goes up to QP 22 whereas the 10.0x video goes down to 17.
for the 1.0x this are 262KiB whereas 10.0x uses 490KiB per frame.
the QP, size | QP, size for 1.0x | 10.0x now continues like this:
25, 164299 | 16, 551711
24, 185327 | 15, 611007
24, 186271 | 14, 667463
24, 185715 | 13, 741415
24, 185215 | 12, 797839
24, 185059 | 11, 873799
24, 184827 | 10, 956451
24, 185711 | 10, 952903
24, 186119 | 10, 952615
next I-frame (#12)
24, 237231 | 10, 1000675
so now it is clear that the data is spent for QP parameter which affects quantization.
here some graphical analysis results regarding image quality:
1x (http://upload.g3gg0.de/pub_files/48511e3b8e9273afc4539ec1a4ae78d6/nitrate_7D_1x.png)
3x (http://upload.g3gg0.de/pub_files/80612bcf956098840cde0f557cd96544/nitrate_7D_3x.png)
5x (http://upload.g3gg0.de/pub_files/46f8bcf8f88bf9bdf932bdbe21551080/nitrate_7D_5x.png)
10x (http://upload.g3gg0.de/pub_files/e35590974808ca61e5461a2ed594117a/nitrate_7D_10x.png)
Looks like the higher nitrate has less denoising but with the same blurriness...I wonder if that's something good or not
no, blurriness and denoising both go down to 50%
Yep, I wonder what i was reading... any chance you can do a screenshot at 3x or something in the middle?
sure, added the results to the post above
So no lower QP than 10? In 600D when BR gets high, CBR likes to set qscale of +25. You can shoot ALL-I too and then look at frame sizes.
uhm what? qscale 25 is the worst quality. ah ok 26 is the worst, so 25 is not really better.
and thats the default value?
This high bitrate is AWESOME!!! \o/
I've only been playing with this for a couple-a hours, but it seems my Transcend 133x cards can't record more than CBR x2.5. However, my Sandisk Extreme can record at CBR x20.0 for the full 4GB. As mentioned, it seems it's actually recording on VBR, so I set my 7D to ISO 12,800 to maximize noise, forcing high bitrate with the following settings:
# Magic Lantern v2.3.NEXT.2012Oct28.7D203
Start : 2012/10/30 19:44:03
Lens name : EF50mm f/1.8 II
ISO : 12800
Shutter : 1/48.194s
Aperture : f/14.0
Focal length : 50 mm
Focus distance : 0 mm
White Balance : 1 - Sunny, Magenta 0, Blue 0
Picture Style : UserDef2 (0,-4,-2,0)
FPS : 23.976
Bit Rate (CBR) : 20.0x
CSV data:
Time,ISO,Shutter,Aperture,Focal_Len,Focus_Dist
19:44:03,12800,48,14.0,50,0
19:44:09,12800,48,16.0,50,0
19:44:10,12800,48,18.2,50,0
I'm shooting f/14.0 indoor at night. :D And the result is bleepin' awesome! I've never seen ISO 12,800 denoise so cleanly:
> http://i.imgur.com/gKEyi.jpg
Here's the original frame, for comparison:
> http://i.imgur.com/UwtiV.jpg
Compression noise so low that it's easy for the denoiser to clean sensor noise. :D I'd say it's pretty much usable, especially after some post processing.
What's weird was that the POS screen and a lot of the areas near the lights looks blown out on the screen when I shot this, but it looks just fine on the video file.
I put this particular file through AVInaptic for analysis. I'm not sure if any of these means anything (especially the errors), but in case anyway want to take a look, here's the analysis result.
My 7D's processor stabilizes at 15% with flushing at every 4 frames at CBR 20. Maybe there's room to increase the CBR (well, VBR?) even higher?
One thing I noticed with high ISO, high bitrate is that, it seems the rolling shutter becomes far more severe compared with regular bitrate, but I haven't done a comparison yet, so this is just a feeling.
I also did a stress test by shooting at 12800 ISO and I kept panning the camera around so the picture would kept changing. It records for 3min. and11sec. before it hits 4GB.
Thank you for all of your hard work, g3gg0! \o/
Let me know if there's anything you want me to try out. :D
[ About file ]
Name: MVI_6586.MOV
Date: Tue, 30 Oct 2012 20:47:25 +0700
Size: 474,530,285 bytes (452.547345 MiB)
[ Magic ]
File type: ISO Media, Apple QuickTime movie
[ Generic infos ]
Duration: 00:00:19 (19.26925 s)
Container: Apple QuickTime movie (fast start)
Creation time: Wed, 31 Oct 2012 02:44:03 +0700
Modification time: Tue, 30 Oct 2012 20:47:21 +0700
Total tracks: 1
Track nr. 1: video (avc1) []
[ Relevant data ]
Resolution: 1920 x 1080
Width: multiple of 32
Height: multiple of 8
Average DRF: 10.095238
Standard deviation: 0.755656
Std. dev. weighted mean: 0.737784
[ Video track ]
Codec: avc1
Resolution: 1920 x 1080
Frame aspect ratio: 16:9 = 1.777778
Pixel aspect ratio: 1:1 = 1
Display aspect ratio: 16:9 = 1.777778
Framerate: 23.976024 fps
Bitrate: 197005.307005 kbps
Duration (bs): 00:00:19 (19.26925 s)
Qf: 3.962559
[ Video bitstream ]
Bitstream type: MPEG-4 Part 10
SPS id: 0
Profile: Baseline@L5
Num ref frames: 1
Chroma format: YUV 4:2:0
PPS id: 0 (SPS: 0)
Entropy coding type: CAVLC
Weighted prediction: No
Weighted bipred idc: No
8x8dct: No
Total frames: 462
Drop/delay frames: 0
Corrupt frames: 0
P-slices: 423 ( 91.558 %) ##################
B-slices: 0 ( 0.000 %)
I-slices: 39 ( 8.442 %) ##
SP-slices: 0 ( 0.000 %)
SI-slices: 0 ( 0.000 %)
[ DRF analysis ]
average DRF: 10.095238
standard deviation: 0.755656
max DRF: 18
DRF<10: 0 ( 0.000 %)
DRF=10: 453 ( 98.052 %) ####################
DRF=11: 1 ( 0.216 %)
DRF=12: 1 ( 0.216 %)
DRF=13: 1 ( 0.216 %)
DRF=14: 1 ( 0.216 %)
DRF=15: 1 ( 0.216 %)
DRF=16: 1 ( 0.216 %)
DRF=17: 1 ( 0.216 %)
DRF=18: 2 ( 0.433 %)
DRF>18: 0 ( 0.000 %)
P-slices average DRF: 10.085106
P-slices std. deviation: 0.689222
P-slices max DRF: 18
I-slices average DRF: 10.205128
I-slices std. deviation: 1.264495
I-slices max DRF: 18
[ Profile compliancy ]
Selected profile: MTK PAL 6000
Resolution: 1920 x 1080 > 720 x 576
Framerate: 23.976024 <> 25
Buffer underflow: 00:00:00 (frame 0)
Buffer underflow: 00:00:00 (frame 1)
Buffer underflow: 00:00:00 (frame 2)
Buffer underflow: 00:00:00 (frame 3)
Buffer underflow: 00:00:00 (frame 4)
Buffer underflow: 00:00:00 (frame 5)
Buffer underflow: 00:00:00 (frame 6)
Buffer underflow: 00:00:00 (frame 7)
Buffer underflow: 00:00:00 (frame 8)
Buffer underflow: 00:00:00 (frame 9)
Buffer underflow: 00:00:00 (frame 10)
Buffer underflow: 00:00:00 (frame 11)
Buffer underflow: 00:00:01 (frame 12)
Buffer underflow: 00:00:01 (frame 13)
Buffer underflow: 00:00:01 (frame 14)
Buffer underflow: 00:00:01 (frame 15)
Buffer underflow: 00:00:01 (frame 16)
Buffer underflow: 00:00:01 (frame 17)
Buffer underflow: 00:00:01 (frame 18)
Buffer underflow: 00:00:01 (frame 19)
Error: Too many violations
This report was created by AVInaptic (http://fsinapsi.altervista.org) (18-12-2011) on 30-10-2012 21:17:58
Yep, +25 is pretty bad but not the worst quality. QP of 50 or so is the WORST quality but I don't think it normally can set that. Kinda why I went away from CBR, sometimes it got wonky. Here it looks to be pretty decent. Gettin QP10 with fewer cache hacks.
Like above post, high BR let really bad night footage process into something useful.
QuoteMaybe there's room to increase the CBR (well, VBR?) even higher?
Sort of, but not the CBR way. Once you hit 100% QP 10 there's not much higher to go. I think frame quality goes to 87 and highest qscale "cbr" can set is 112. But QP doesn't change over that range, I think frame gets a little bigger but that's it.
this would make sense for astrophotography
Quote from: feureau on October 30, 2012, 03:47:22 PM
This high bitrate is AWESOME!!! \o/
I would like to duplicate your results. What are the important parameters to obtain such results?
okay now found how to enable all-I on 600D using cache hacks.
will try to port that on 7D.
i ordered a maksutov mto 11-ca and want to shoot some high bit rate videos and stack them using e.g. registaxx :)
update:
all-I works with super high bit rate.
just my analyzer crashes with all videos.
but i see that all frames are I, just cannot look deeper into the details.
but test yourself:
D/L experimental All-I 7D FIR (http://upload.g3gg0.de/pub_files/c129fd81c100f3d966dadcde7d1284af/7D000203_ML.FIR)
If you're doing astro stuff, maybe messing with deblocking will help. If you had gop 4, all I is just gop1
Quote from: akshayverma1 on October 29, 2012, 11:41:25 AM
I'm using a SanDisk Extreme 8GB 60mbps card. I just shot some trees.
ISO 160
Shutter 1/250
Aperture f/11
Flush rate: 4
CBR 20.0x
Focus peaking, magic zoom, histogram, waveform and bitrate display were ON
Audio OFF
I recorded 1 minute, 42 seconds of handheld footage. Recording was stopped by me (it would keep recording for the whole 4GB). The file size is 2.38GB.
As g3gg0 rightly said, it's actually VBR. I got a maximum bitrate of 218 while the average was around 170. Although I focused it pin-sharp, the footage is blurry as is all 7D footage. Visually it looks just like ordinary 1x BR to me.
Could anyone please guide me about doing a better comparison?
It might help to know that once you hit an aperture of f/8 the pixels in the 7D start to diffract which will blur the image a bit more and more the further you go passed that threshold.
My visual test show little-to-no difference in noise between 1x or 20x. I applied denoising as mentioned before but still no improvement in final results.
Also tried the ALL-I at 20x and here I noticed a little more noise at 20x.
Tests at iso 12800
f/5.6 and 1.8
Lens 24mm sigma f/1.8
1920x1080 24fps
If anyone gets better results with cbr 20x then I would like to know how. It is very difficult for me to post images or videos since I have very limited bandwidth.
Quote from: g3gg0 on October 30, 2012, 11:24:15 PM
update:
all-I works with super high bit rate.
just my analyzer crashes with all videos.
but i see that all frames are I, just cannot look deeper into the details.
but test yourself:
D/L experimental All-I 7D FIR (http://upload.g3gg0.de/pub_files/c129fd81c100f3d966dadcde7d1284af/7D000203_ML.FIR)
This is great news, g3gg0! :D I'm trying it out as we speak.
AVInaptic says I'm getting All-I now~! :D
Out of 139 frames:
P-slices 0 ( 0.000 %)
B-slices 0 ( 0.000 %)
I-slices 139 (100.000 %) ####################
SP-slices 0 ( 0.000 %)
SI-slices 0 ( 0.000 %)One thing I noticed though, is that increasing the bitrate and using all-I, it seems that I'm seeing far more severe rolling shutter than normal...
Also:
Quote- it seems to be not CBR, but VBR although it says CBR
I think this is VBR too. If I shoot with low ISO and don't have anything moving in the frame, I'm only getting around 35-45 mbps...
Any news on recording audio too, btw? (even at reduced bitrate to save bandwidth for video?) Would be great as guidetracks.
Thanks again for all the hard work, g3gg0! :D
Quote from: RenatoPhoto on October 30, 2012, 10:32:20 PM
I would like to duplicate your results. What are the important parameters to obtain such results?
ML Movie settings:
Mode: CBR
CBR Factor 20.0x
Bitrate Info: On
Flush rate: 2
BuffWarnLevel: 50%
The processor percentage thingy at the upper right corner that shows up on the bitrate info box on the upper right stabilizes at 4% on my 7D.
Quote from: RenatoPhoto on October 31, 2012, 03:13:32 AM
My visual test show little-to-no difference in noise between 1x or 20x. I applied denoising as mentioned before but still no improvement in final results.
interesting. i can see the ISO noise a more with 10x than with 1x when playing with VLC on my computer.
also image quality analysis using a special tool says that the quality is better (factor 2, when bitrate is raised by factor 5 iirc)
sure that your video file shows that high bit rate? try using VLC and look at the statistics.
I did some tests with the All-I port.
These are jpegs at quality setting 10.
1080p at 25FPS
Bitrate: CBR 20x
1/250s, f/5.6, ISO 160
18-135mm lens at 135mm
1) Newspaper closeup: unsharpened
(http://dl.dropbox.com/u/48224378/MVI_0390%20(0.00.02.12).jpg)
Sharpened:
(http://dl.dropbox.com/u/48224378/MVI_0390%20(0.00.02.12)%20Sharpened.jpg)
2) Tree: unsharpened
(http://dl.dropbox.com/u/48224378/MVI_0388%20(0.00.02.12).jpg)
Sharpened:
(http://dl.dropbox.com/u/48224378/MVI_0388%20(0.00.02.12)%20Sharpened.jpg)
Compression artifacts still show up occasionally. All-I is improving the quality but if the bitrate could be constant there would be a major improvement.
Thanks a ton for all the hard work g3gg0!
@P337: Yes! I agree anything lower than f/8 would soften the footage, but a still image taken at the same aperture isn't that soft. I think it's because of the line skipping, as ilguercio pointed out. This time I shot at f/5.6 (which I think is the sharpest for this camera, or even f/6.3) and it's still not sharp.
Compare ALL-I vs gop-3 and 4. In 600D tests, all I resolves a little bit less detail... still better than stock tho. With 200Mbps I frames you have a bit more quality so may be better.
Glad someone else noticed the rolling shutter. It does seem to increase but I didn't know if it was better or worse than stock.
Quote from: akshayverma1 on October 31, 2012, 12:20:51 PM
I did some tests with the All-I port.
These are jpegs at quality setting 10.
1080p at 25FPS
Bitrate: CBR 20x
1/250s, f/5.6, ISO 160
18-135mm lens at 135mm
1) Newspaper closeup: unsharpened
(http://dl.dropbox.com/u/48224378/MVI_0390%20(0.00.02.12).jpg)
Sharpened:
(http://dl.dropbox.com/u/48224378/MVI_0390%20(0.00.02.12)%20Sharpened.jpg)
That's some really good sharpening work there. What did you use to sharpen these up? I can't believe there's practically no moire on these patterns. Did you use an anti-aliasing filter such as the VAF7D?
Quote from: feureau on October 31, 2012, 06:24:32 PM
That's some really good sharpening work there. What did you use to sharpen these up? I can't believe there's practically no moire on these patterns. Did you use an anti-aliasing filter such as the VAF7D?
I sharpened these with After Effects' built in unsharp mask at 200% (wanted to over sharpen a bit), radius: 1px, threshold:0.
Didn't use any anti-aliasing. I guess it's not there because the details are rather large (the pictures aren't cropped). Ink droplets on a newsprint are quite visible.
Quote from: akshayverma1 on October 31, 2012, 07:46:45 PM
I sharpened these with After Effects' built in unsharp mask at 200% (wanted to over sharpen a bit), radius: 1px, threshold:0.
Didn't use any anti-aliasing. I guess it's not there because the details are rather large (the pictures aren't cropped). Ink droplets on a newsprint are quite visible.
Hey Akshayverma,
Thanks for the tests ;D what bitrates were you averaging and peaking at for those jpegs? And I assume those were single frames taken from a video right?
Quote from: akshayverma1 on October 31, 2012, 12:20:51 PM
@P337: Yes! I agree anything lower than f/8 would soften the footage, but a still image taken at the same aperture isn't that soft. I think it's because of the line skipping, as ilguercio pointed out. This time I shot at f/5.6 (which I think is the sharpest for this camera, or even f/6.3) and it's still not sharp.
Right, line skipping will also effect it as well as the 4:2:0 subsampling and frame compressions. Even at 1:1 compression rates, about 597 Mbps, it would still only have half the chroma pixels :/ So I doubt it will ever equal to photo image quality :(
This whole thing of bitrates is a mystery to me. It is great that this additional tool has
been added to 7D and now I am trying to figure out how to make the best of it. I was really
excited with a previous post where some amazing results were presented with ISO 12800
where the denoising had improved significantly by the use of higher bitrates. Unfortunately
I ran a test myself and cannot find any such results. I import the videos to vegas 12
and denoise with Neatvideo and the results are the same shitty noise between 1x or 20x for IBP and
the noise of ALL-I seems worst compared to the standard canon video (no ML) until you get up to
10x CBR. I hope someone can provide more details on their workflow to getting such great results
in the high iso world by using the higher bitrates.
Also I have done some testing at low iso (100) with all the options and again the final results do
not convince me that using higher bitrates produces any better video. For all comparisons I select an area of
about 320x240 pixels of the whole 1920x1080 so I am looking at at 6x factor where I can see artifacts
and noise very clearly.
I also ran some test on the 5D3 and found out that the ALL-I video quality (artifacts) is worst and therefore
I will only use IBP video with my 5D3.
I don't have the video analysis tools but by looking at the video properties I found the following bitrates for
all my test subjects:
CBR IBP ALL-I
kbps kbps
No ML 97573 updated -->(54588 IBP)
1x 49541 43228
2x 88795 95941
3x 89605 150578
5x 90403 228832
10X 91294 268863
15X 96169 272493
20X 98747 243947
Tested 7D with 24mm sigma f1.8
Note that the standard 7D video has a bitrates of 97573 54588 kbps which is below 2X bitrates and above 1X for both IBP and ALL-I. In All-I video the bitrates go significantly higher but the artifacts show big time until you get above 2x and thereafter I do not see any improvements as bitrates go up.
Since the video analysis software presented in this post, is showing significant improvements, I would assume that these improvements are at such small pixel size that my eye cannot distinguish and therefore I do not see the improvements that the analysis would seem to point. Again I am looking at 6x magnification and cannot match VISUAL IMPROVMENTS TO SOFTWARE ANALYSIS NUMBERS.
Any help to clarify this mystery would be greatly appreciated.
Updated since I made a mistake which was pointed out where the standard video on 7D does not go up to 97Mbs. I must have mixed up some of the videos.
Just to be clear, is a lower number for the "flush rate" speeding up the buffer? Is Flush=1 the fastest or is it Flush=20 and what is Canon's standard flush rate?
Cause whenever I try to record with the flush rate any lower than 3 I get an err70 and when I set flush to 20 I see my buffer percentage stabilizes around 30% but when Flush is set to 4 it stabilizes around 5%.
Anyway my 1st CF card is benchmarking at 15MB/s (8, 8, 15) I set to CBR 20x and Flush to 3; it buffers out at 100 Mbps Average (125 Mbps peak instant) which is only equal to CBR 2.5x. Not a very good card for testing high bitrates but I got another one that's benchmarking at 45MB/s that I'll try later.
@P337:
it is the number of frames after that buffers should get flushed.
there is no "better" value.
if your buffer runs full (because of high bit rate), but your card could write faster, decrease the value.
lower values so are not "speeding up buffer" but clearing the buffer more often.
canon default is after [fps] frames. as said in main post - every second.
@RenatoPhoto:
what makes you think that the noise is worse?
more noise in video means that there are more details. in this case its high ISO noise (?)
thanks for testing.
Quote from: RenatoPhoto on October 31, 2012, 10:23:46 PM
...I don't have the video analysis tools but by looking at the video properties I found the following bitrates for
all my test subjects:
CBR IBP ALL-I
kbps kbps
No ML 97573 54588
1x 49541 43228
2x 88795 95941
3x 89605 150578
5x 90403 228832
10X 91294 268863
15X 96169 272493
20X 98747 243947
...
Note that the standard 7D video has a bitrates of 97573 kbps which is close to the 20X bitrate. In All-I video the bitrates
go significantly higher but the artifacts show big time until you get above 2x and thereafter I do not see any improvements
as bitrates go up.
...
Any help to clarify this mystery would be greatly appreciated.
Something is not right here; 7D with no ML isn't 97 Mbps (h.264 Baseline 5 has it capped at 50 Mbps) and the 7D doesn't have "ALL-I" without Magic Lantern and never had IPB even with ML.
Yes "All-I" on the 5D3 will likely have less detailed image quality than its IPB mode. We touched upon that in this thread: http://www.magiclantern.fm/forum/index.php?topic=3103.400
Quote597 Mbps for a true 1:1 compression of a 1080p 4:2:0 image at 24fps
35 Mbps for the average 17:1 compression with "IPP" (Which is Canon's stock settings)
86 Mbps for the average 7:1 compression used for "All-I" (Canon gives around 95 Mbps for the 5D3 "All-I" mode)
and
We've seen the 7D maxed at about 250 Mbps which is about a 2.5:1 compression!
...since All-I "does less compression work" and since most compression artifacts and glitches happens due to round off errors during compression it would be expected that All-I would reduce the opportunity for artifacts.
The problem is that All-I needs quite a bit more bandwidth than IPP and if it doesn't get it then most fine details get lost in the "All-I process". For example, the GH2 with h.264 IPB at 88 Mbps gave a near 1:1 compression look but the 5D3 with h.264 All-I at 95 Mbps didn't.
Being VERY untechnical about it, from what I've "heard":
ALL-I at 100 Mbps would look similar to IPP at 50 Mbps and IPB at 25 Mbps
@g3gg0
Ah, I see. So "Stock" would be like Flush = 24, which would flush the buffer every 24 frames which also happens to be every second when the recording is set to 24fps.
So setting the Flush to 4 would flush the buffer every 4th frame instead of every second, yeah?
So it's clearing the buffer faster than once per second now right?
Thanks for clearing that up :D
Quote from: g3gg0 on October 30, 2012, 01:43:22 PM
what is the advantage of all-I anyway?
thats the reason why i asked that in the other thread :)
so all-i is good for what now? is modifying GOP useful at all?
or will a lower GOP value reduce blocking?
maybe just make it configurable and (pro) users can choose what they need.
@P337:
exactly
Shorter GOPs are better for recording really fast or really erratic movements (like race cars) because it just records everything without having to try to "predict"your subjects movements (unlike with IPP and IPB) so there is less chance of artifacts caused by prediction errors (like ghosting).
ALL-I will record the subject's movement more accurately but that needs more bandwidth. If it doesn't have enough bandwidth, you'll still get the accurate movements but will get far more compression errors (like blockiness).
80% of the time I'd choose IPB (or IPP) but for those times you need "All-I" it's great to have, you need the bandwidth to use it though.
Also "All-I" is easier to edit and color grade cause each frame has its own pixels rather than having to depend on the frame that came before, or people can just transcode to an Intra codec in post :D
Canon defaults for gop are FPS/2... so at 24 its 12 at 30 its 15 and at 60 its 30. These 3 do not need cache hacks and just work.
QuoteSo setting the Flush to 4 would flush the buffer every 4th frame instead of every second, yeah?
Canon writes the buffer to the card after each GroupOfPictures is finished. So when you shrink gop it writes the buffer out faster. Hence all I writes every frame immediately.
Quotewill a lower GOP value reduce blocking?
I think higher bit rate should reduce blocking but you can control the deblocking filter independently, right now canon is doing it. I turn it off and don't get much blocking, go figure.
QuoteAlso I have done some testing at low iso (100) with all the options and again the final results do
not convince me that using higher bitrates produces any better video.
Then don't be convinced and enjoy your 40mbps max video. Frame size is bigger, QP is lower everything on the encoder end is better. Where do you think that extra file size comes from? My 10MP camera vs my 20MP camera stills don't convince me either... except somehow raws are 20mb vs 10mb and go further in post.
You guys haven't pushed luma/chroma quality up yet, that should add a little bit. According to H264 technical docs it was like 17% increase... the player can't play those back either.
Welp, I did some testing on rolling shutter with and without the bitrate monster.
Settings:
Lens: 18-135 IS. (non-STM version)
ISO: 12800
Shutter: 1/4000
Aperture: f/5.6
Focal length: 135mm
First up: The default firmware:
http://i.imgur.com/nS3aa.jpg
Here's with the bitrate monster at CBR 20.0x and flush at 3, which I believe is the maximum that the ML can pull out of the camera:
http://i.imgur.com/mrbcI.jpg
Here's the two, overlaid on top of each other:
http://i.imgur.com/Zkrqr.jpg
There's a slight increase of rolling shutter in the bitrate monster's but I'd chalk that one up as:
a. Margin of Error due to that I had to do this by hand one setting after another, and not at the same time.
b. Very slight sample variance.
That I'd say there's practically no difference in rolling shutter.
Another thing that I can conclude from the test is that if you look at the grain pattern on the wood part, and the nail details, you can definitely see definition in the All-I version while it's all washed out in the default firmware version. Thus concluding that ALL-I have a definite net positive benefit in image resolution. Keep in mind that this is ISO 12,800.
In summation:
a. Rolling shutter difference is negligible or insignificant
b. ALL-I has a demonstrable benefit in resolution
QED. :P
One thing I can't seem to explain is that the highlight seems to be blown out in the default firmware version, while it's more retained in the bitmonster. I'm not sure if something changes in the setting while I load ML (doubt it) or if high bitrate ALL-I somehow lets it retain more highlight detail.
UPDATE: I did some more quick test, and I'm seeing it: The bitmonster seems to be retaining highlight details much better at ISO 12800.
UPDATE2: false alarm on the highlight thing. Apparently, under certain circumstances, after you record some movies with default firmware, then load up ML, and go to movie-live-view mode with ML, it would stop down the aperture by one stop. Not sure what triggers it, it happens rather randomly.
Quote from: P337 on October 31, 2012, 01:31:40 AM
It might help to know that once you hit an aperture of f/8 the pixels in the 7D start to diffract which will blur the image a bit more and more the further you go passed that threshold.
It's ~f/6.9 for the 7D due to the 18 MP sensor. However, with 1080p video, you're then talking about effectively a ~2.1 MP sensor so the airy disk can technically be larger. On top of that, since the video is created with line skipping, anything over f/7.1 may still in hibit resolution. One more thing I want to look into aside from the pixel-peeping effects of the VAF-7D.
i got an error code at playback with the bit rate at 19.9. ISO was at 4000 with highlight tone priority. i don't know if anyone else has gotten something similar but thought i'd report on it. the highest iso i can use without the error on playback is 2500.
Quote from: P337 on October 31, 2012, 09:15:40 PM
Thanks for the tests ;D what bitrates were you averaging and peaking at for those jpegs? And I assume those were single frames taken from a video right?
Yes, those are single frames taken from video. Average bitrate for the newsprint was about 150. Peak was 170. For the tree, average was 210, and peak was 260.
I did some high ISO tests with well lit trees last night and got some amazing bitrates - 290 peak and 260 average. It's 35 seconds and 1.04GB. :)
Quote from: Digital Corpus on November 01, 2012, 07:58:06 AM
It's ~f/6.9 for the 7D due to the 18 MP sensor. However, with 1080p video, you're then talking about effectively a ~2.1 MP sensor so the airy disk can technically be larger. On top of that, since the video is created with line skipping, anything over f/7.1 may still in hibit resolution. One more thing I want to look into aside from the pixel-peeping effects of the VAF-7D.
Thanks Digital! You're always getting me to double check my work and learn more lol :D
You are right the Ariy disk does begin to overlap pixels around f/6.3 but it isn't really noticeable until f/8 so most people consider that diffraction limited.
Unfortunately I think with "line skipping" to down-res the image to 2.1MPs we do not benefit from an increased diffraction limit like we did with Photo mode's pixel binding method which physically makes each pixel sensor larger rather then in the line-skipping method we still have the same sized (4.3micron in this case) pixel sensors :/
Quote from: akshayverma1 on November 01, 2012, 10:51:31 AM
Yes, those are single frames taken from video. Average bitrate for the newsprint was about 150. Peak was 170. For the tree, average was 210, and peak was 260.
I did some high ISO tests with well lit trees last night and got some amazing bitrates - 290 peak and 260 average. It's 35 seconds and 1.04GB. :)
Thanks Akshayverma!
Those are very valuable numbers, my fastest card is averaging about 150 and peaking at 170 too (with a flush rate of 2).
I found the best way to max out bitrate (for testing purposes only) is to set ISO to max (ISO 128,000) with a fast shutter (250 - 4000 depending on lighting) and point your camera at something that spotmeters under 75 on the 0-255 scale; this will shoot your bitrate up to max right quick!
Set to 20x with a Flush rate of 2 my "200x CF" card (benchmarked at 21 MB/s) can record 200 Mbps for one second after filling the buffer and shutting down recording but I found I could stably record anything to the full 4GB limit at a fairly constant 160 Mbps, which is the peak limit for 3.5x :/ guess I'll need those "600x CF" cards after all lol
I'll try some 160 Mbps high noise and tree details later today lol.
Quote from: P337 on November 01, 2012, 12:19:46 PM
Thanks Digital! You're always getting me to double check my work and learn more lol :D
You are right the Ariy disk does begin to overlap pixels around f/6.3 but it isn't really noticeable until f/8 so most people consider that diffraction limited.
Unfortunately I think with "line skipping" to down-res the image to 2.1MPs we do not benefit from an increased diffraction limit like we did with Photo mode's pixel binding method which physically makes each pixel sensor larger rather then in the line-skipping method we still have the same sized (4.3micron in this case) pixel sensors :/
You're welcome. Most people refer to f/8 since it's the closest full stop measurement near the diffraction limit :). I don't currently have the facilities to test where we start loosing detail between the 18 MP sensor and the resulting 1080p video. However, this is what I've looked at so far...
Ok, so I've done some testing myself. The only consistent scene I've been able to record (partially overcast day) is a patch of grass. Detail here is similar to leaves, but with a lot less movement. Using the all I-frame firmware, I acheived a max average bitrate of ~281 Mbps using CBR @ 20x and the flush rate at 4 frames. I had ML stop recording at 60 seconds and the buffer started out at 28% and then leveled off at 22-23%.
All recording was done with a VAF-7D installed as well and sharpness set to 1. I have screens showing a significant increase in perceptable detail, but I don't have a quick enough of a connection to upload them at this time as they are ~5 MB png files. The artifacting that I previously saw in my test scene with leaves showed scaling artifacts when in-camera sharpening was set to 0. With the VAF-7D, I saw none of these artifacts, even on the grass blades.
I noticed it was a bit hard to get the bitrate above ~230 Mbps for the scene. I had only 1 issue with my CF card not writing out all of the frames and throwing an Error 70 message, but I only had ~70 MB left on the card so I'm assuming it was due to fragmented freespace. When I did this scene for testing the speed of the card w/ the bitrates, I adjusted the 12 points (not worth the words to explain why) on a logarithmic scale (e^(x*ln(20)/11)) from 1 to 20. The file sizes were as follows:
1x 333 MB
1.3x 408 MB
1.7x 531 MB
2.3x 744 MB
3x 975 MB
3.9x 1.2 GB
5.1x 1.62 GB
6.7x 1.73 GB
8.8x 1.63 GB
11.6x 1.66 GB
15.2x 1.72 GB
20x 1.62 GB (ended ~2 seconds early)
These were I-Frame btw... Based on these numbers and the ones others have produced with similar tests, it appears that the sweet spot of recording is 4x-7x. I'll see what I can do for some controlled tests in this area because it appears that the encoder is currently hitting a wall. No reason to give users options to go beyond 10x atm. When/if we establish a best-case upper limit, we could just set it there so we don't have people trying to figure out if they should push their hardware harder?
Quote from: Digital Corpus on November 01, 2012, 02:27:12 PMWhen/if we establish a best-case upper limit, we could just set it there so we don't have people trying to figure out if they should push their hardware harder?
Please don't do this.
Upper limit not really possible. There is a "max" quality but not set with predictive CBR, rate still varies.
thanks a lot for these comprehensive tests!
in the meantime i made both VBR and CBR selectable and configurable.
the camera now also displays the Q factor as known from other models.
@1%:
> You guys haven't pushed luma/chroma quality up yet
you mean QP(Y;U;V)? that is according to analysis tools exactly the same as QP (10).
one correction: you said, camera flushes after each GOP - you meant after each two GOPs, as this is one second :)
I thought it was after each gop, maybe its 2, would have to double check.
Chroma QP and Luma QP should be separate parameters, it was on 600D. Couldn't set 10 (max 20 vs 26) or camera would freeze. I used H.264 analyzer and it never matched QP. I think they are really chroma_qp_index_offset and luma_qp_index_offset in H264.
Like: http://opencores.org/websvn,filedetails?repname=nova&path=%2Fnova%2Ftrunk%2Fsrc%2FQP_decoding.v
See: http://books.google.com/books?id=jlL1xqOry2QC&pg=PA192&lpg=PA192&dq=QPy+QPC&source=bl&ots=IOBIt7POyT&sig=vhmlA2bbHzohta62ovma4OeJViI&hl=en&sa=X&ei=LpOSUPbTOsPXyAGQroBY&ved=0CCUQ6AEwAjgK#v=onepage&q=QPy%20QPC&f=false
Also: http://thinknaturally.blogspot.com/2009/01/h264how-to-do-conversion-from-8-bits-to.html
Just had a thought, is 4:2:2 YUV being fed to the encoder or is it 4:2:0.. I think its set via digic register.
Set H264
*0xC0E10004 = 0
*0xC0E10050 = 0
YUVlossless
*0xC0E00004 = 65538
*0xC0E00050 = 0
*0xC0E0008C = 0x111121
YUV: 4:2:0
*0xC0E00004 = 65538
*0xC0E00050 = 0
*0xC0E0008C = 0x111141
Also *0xC0E1000C is interesting. Set to 41104, 41091, 8090
Quote from: g3gg0 on November 01, 2012, 04:10:04 PM
in the meantime i made both VBR and CBR selectable and configurable.
the camera now also displays the Q factor as known from other models.
Awesome! :D Awesome to the max.
Yes!!! Yes!!! Yes !!! Yes!!! Yes!!! Yes!!! Yes!!! Yes!!!
Quote from: 1% on November 01, 2012, 04:26:46 PMCouldn't set 10 (max 20 vs 26) or camera would freeze. I used H.264 analyzer and it never matched QP. I think they are really chroma_qp_index_offset and luma_qp_index_offset in H264.
hoho, holy shit...
tried setting PICQPC to 1.
this results in chroma_qp_index_offset being set to 1 and QPY 50, QPU/V 39.
the 1280 x 720 video looks like shot with a 320x240 webcam.
basically every 16x16 block is "worth" a pixel.
same if i set chroma_qp_index_offset to -1..
when setting PICQPY from 0x1A to 0x19, then the pic_init_qp_minus26 goes down by one from -1 to -2
Quote from: g3gg0 on November 01, 2012, 09:57:14 PM
hoho, holy shit...
tried setting PICQPC to 1.
this results in chroma_qp_index_offset being set to 1 and QPY 50, QPU/V 39.
the 1280 x 720 video looks like shot with a 320x240 webcam.
basically every 16x16 block is "worth" a pixel.
same if i set chroma_qp_index_offset to -1..
when setting PICQPY from 0x1A to 0x19, then the pic_init_qp_minus26 goes down by one from -1 to -2
Is this about the line skipping/pixel binning thing the canons do?
.... Does it mean that it's possible to increase resolution if you set it to the other direction? Like reversing the polarity and all that star trek-y mumbo jumbo?
Anyway, I wonder if anyone can confirm this: if you have ML on, and you're on the bitrate menu, and probably highlighting flush at around 4,6,7, or 8 and half pressing the shutter button would just shut down the camera. Probably a bug or just my 7D doing weird stuff. I've been trying to see how fast I can push one of my cards, a slow "cheap" transcend 133x card.
Quote from: feureau on November 01, 2012, 10:25:46 PM
Is this about the line skipping/pixel binning thing the canons do?
.... Does it mean that it's possible to increase resolution if you set it to the other direction? Like reversing the polarity and all that star trek-y mumbo jumbo?
errrhh... definitely: no! ;)
It didn't work setting it using the function for me.
Picqpy lowest was 20, stock is 26. any other lower would cause "busy" and camera freezes. Could only set it via address.
PicqPC let me move it +-7 with the function.
According to H.264 20 is better than 26. And picqpc is connected to the picqpy range. If I moved PC after setting py to 20 camera would freeze.
Finally seeing those terrible QP's? Can make like 8MB video out of those :)
Quote from: g3gg0 on November 01, 2012, 09:57:14 PM
hoho, holy shit...
tried setting PICQPC to 1.
this results in chroma_qp_index_offset being set to 1 and QPY 50, QPU/V 39.
the 1280 x 720 video looks like shot with a 320x240 webcam.
basically every 16x16 block is "worth" a pixel.
same if i set chroma_qp_index_offset to -1..
when setting PICQPY from 0x1A to 0x19, then the pic_init_qp_minus26 goes down by one from -1 to -2
lol, is that cause PicQpC needs to be set by PicQpY? Rather can changes be made directly to PicQpC without adjusting PicQpY?
(Forgive me if that question makes no sense cause I'm not entirely sure I even understand what I just said lol)
Anyway is that "720 looks like 240" trick basically what happens when setting a low CBR like 0.3x. Should we expect a similar loss of quality when setting bitrate that low?
Yes, you can adjust both independently.
720 looks like 240 is a product of really high QPs (low quality). I can do the same thing but I'm setting the parameter that controls QP directly rather than relying on CBR. When I set picqpy/pc the overall QP doesn't change. Increasing those produced higher BR for same QP.
If you set CBR to .3x it will pick QPs like 25/30/40, etc and look like pixelated blocky mess.
Thanks 1%
I think I'm starting to get it; they can each be adjusted on their own but pY is still connected to pC so when you set pY to 20 (it's maximum) then is pC also pulled to it's maximum too? Is that's why any changes to pC after pY is maxed out causes a freeze?
Theoretically you should be able to change throughout whole range. Maybe on 7d you can but it messes with other things or doesn't work as intended. CBR could also be setting lower QPs because the parameter was changed.
https://bitbucket.org/OtherOnePercent/tragic-lantern/src/0ec03c27a34f01ad4b04021cd20aec91b6b146a2/src/bitrate.c?at=unified
Is how I change it but I have hard-coded addresses.
To better test these high bitrates I'm trying to decide between getting a 600x or 400x CF card. The 600x benchmarked at an avg 640 Mbps (80MB/s) write speed, the 400x I can't find benchmarks for it but their 300x benched at 292 Mbps (36.5MB/s) write speed which is about the fastest we've seen on the 7D.
So do you think this 300Mbps limit is a camera or card speed limit?
Has anyone achieved any peak numbers over 300 Mbps?
Do you guys think it's worth even trying a 600x (I doubt it will be)?
If 300Mbps really is the camera's limit then that's about CBR 6x (right?), so setting any higher values than that or using a card that benchmarks over 37.5MB/s won't make any difference.
If you have a super fast card I recommend setting CBR to 20x and Flush to 1. Then enable spotmeter and select the 0-255 option. Then set ISO to 128,000 and use your shutter or ND filters to underexpose to "25" on the 0-255 spotmeter. This always shoots my bitrate up to it's max but my cards, which are "200x" 170Mbps (21MB/s)*, peak at 200 Mbps for 1 second then the buffer can't keep up** so the recording stops :/
*ML Benchmarked my card at 21.5MB/s
**my Flush rate is set to 2 (my camera throws err70 if flush=1)
The gop length assert has to be patched, is it not?
You can try a faster card if it isn't too expensive... ie under 100usd. The best that you'll probably do is steady 2xx MBPs. When I patched for quality I got that at like iso 320 in daylight with trees. Most of the time BR doesn't go much above 120-160 but you're future proofing for mjpeg/yuv dumping, recording audio alongside video, etc.
Also need to see if write does go above 35mbps, on SD cameras the max is 25mbps per the controller.
g3gg0
"in the meantime i made both VBR and CBR selectable and configurable.
the camera now also displays the Q factor as known from other models."
Give news please.
We try CBR and VBR.
Might find some updates here:
https://bitbucket.org/hudson/magic-lantern/changesets
How can I do with these things.
I am not a technician.
I want to understand.
how to do.
Thank's.
Sorry for my bad english.
Definitely take heed of the advice in the first post about overheating, my camera came perilously close at 190Mbit/s for a prolonged recording. Works fantastically well, however! Can see this being useful for getting ultra clean looking footage for twixtor etc. Great work!
Quote from: mimiloveyou on November 03, 2012, 12:25:21 PM
How can I do with these things.
I am not a technician.
I want to understand.
how to do.
Thank's.
Sorry for my bad english.
It doesn't seem to be working yet, looks like only the first few frames in a recording are accepting the CBRs.
They are however making pretty good strides in the MJPEG encoder thread, looks like the 7D can take 1056x704 4:2:2 JPEGs from the buffer and record them to the card without needing to be connected to the EOS Utility through USB ;D
http://www.youtube.com/watch?v=jQLHkXL-AWU
here (http://upload.g3gg0.de/pub_files/874f4306b8683eb88e9eeccbb704ae14/7D000203_ML.FIR)is another experimental build.
i set GOP to 4 and enabled VBR/CBR selection.
also the flush rate is now limited to 2-50.
flush rate is on both digics set via cache hacks now - this means it is less likely to abort recording after the first 100ms, although
it is likely to happen if you start recording the first time, as the drivers seem to read some filesystem information which takes some time (?)
successive recordings will fail less likely.
what do you think about that?
i think i will make the GOP selection configurable somewhen, so you can choose your favorite.
Quote from: P337 on November 04, 2012, 12:12:30 AM
They are however making pretty good strides in MJPEG, looks like the 7D can take 1056x704 4:2:2 JPEGs from the buffer and record them to the card without needing to be connected to the EOS Utility through USB ;D
That's such a weird resolution... the 4:2:2 part is great though. I wonder if there'd be a way to record raw like this. There's a lot of buffer space for SRAW on the 7D.
updated my previous post, forgot to add the link...
Quote from: g3gg0 on November 04, 2012, 01:00:13 AM
here (http://upload.g3gg0.de/pub_files/874f4306b8683eb88e9eeccbb704ae14/7D000203_ML.FIR)is another experimental build.
i set GOP to 4 and enabled VBR/CBR selection.
Awesome, thanks g3gg0!
So in this build everything is GOP=4 even at low bitrates?
Thanks again, checking it out now ;D
So, I may not be using this right yet but I unfortunately do not see a difference in bitrate with CBR vs VBR(FW default) :/
I got a scene giving me about 40-46Mbps under FW default and CBR 1.0x, I tried the same scene all the way up to CBR 10.0x but they all still read the same 40-46Mbps range (using "display bitrate", haven't analyzed the clips yet). I'm assuming CBR should keep the bitrate at its maximum even with low detail scenes, like CBR 1.0x = 48Mbps and CBR 2.0x = 96Mbps, if that's not the case then maybe it's working as intended and I just don't understand what it's supposed to do.
Just tried the scene again with different exposure settings to give me 22-29Mbps and got the same results (every CBR setting was still only giving 22-29Mbps).
This may be unrelated and I assume just a display bug but every now and again when I first start recording the Bitrate display will start in the 700Mbps or 500Mbps range for the first frame then just drop straight down to the 20Mbps or 40Mbps range. Once, when scaling up the CBR quickly to CBR 10.0x, it started at 528Mbps then my camera immediately rebooted itself and ML had to be reinstalled, it didn't record anything but everything still works fine and when I tried again it recorded fine and started in the 22-29Mbps range.
What you're experiencing is normal. Sometimes the first rate is all crazy, it isn't real.
As for no difference between cbr 10x and 20x or whatever... some scenes just encode to X rate even at 100% quality from the camera.
Tip: Go outside... indoors it doesn't get so high (not enough complexity/light), except in crop mode which 7D doesn't have.
You shouldn't have to use CBR only, set qscale to max (do you guys have that yet?) and then camera records at fixed quality. All frames will be QP10.
There is no such thing as real CBR on canon anyways. Its all VBR just with constraints via quality/qp H264 parameters. You're not directly controlling the rate, you're just telling the camera "even if frame size is *friggin huge* pick a high QP for the next frame.
oooh, 422 (M)JPEG and 422 Uncompressed seems to work though ;-D
I enabled Jpeg A: and recorded 24 jpgs per second for one minute on my card (some dropped frames though).
also tried 422 A: but haven't counted the frames yet, I assume they're the same.
Jpgs are 98KB-623KB each (assumed 5:1 compression of 4:2:2) and .422s are 1.5MB each (1:1 uncompressed 422!)
Thanks 1%
Quote from: 1% on November 04, 2012, 02:55:51 AM
As for no difference between cbr 10x and 20x or whatever... some scenes just encode to X rate even at 100% quality from the camera.
Does that mean a static scene may just ignore "CBR" settings and base it off the first frame's complexity?
Quote from: 1% on November 04, 2012, 02:55:51 AM
Tip: Go outside... indoors it doesn't get so high (not enough complexity/light), except in crop mode which 7D doesn't have.
Well I don't have a problem maxing out the bitrate indoors, by underexposing with ISO set to 128000, which get my card's full 160Mbps but it's too noisy so I'm trying to get that max bitrate with proper exposure.
Quote from: 1% on November 04, 2012, 02:55:51 AM
You shouldn't have to use CBR only, set qscale to max (do you guys have that yet?) and then camera records at fixed quality. All frames will be QP10.
We just got Qscale, its default to -8 but says "very risky" so I won't play with it until I better understand what it's doing. All I know is -16 = highest quality and 16 = lowest quality. Since I don't see -26 on the scale I assume it's relatively safe but I'm still not sure what to look out for. Recording stopping? Overheating?
Recording stopping. Can't be any worse than CBR, it sets a Qscale of -16 too.
Its compressing a YUV scene to H.264. The more data in the YUV the higher the bit rate at X quality. So ISO 80 might not generate a lot of data but if CBR is higher it will compress it at QP 10. After that no magic way to increase bit rate.
Is Qscale 10 the same as QP 10?
How do I use MJpeg 422 ?
Thank's.
No, Qscale -16 is 10 I think.
They're continuous silent pics so press and release the shutter in jpeg or 422.
Been playing with Canon's bitrates and I think I understand it better now.
"VBR (Qscale) mode" in 7D seems to be used kinda like constant bitrates or rather "limited range bitrates". For example, recording the same scene with the same exposure settings on Qscale(+16) recorded at 1-2Mbps, (+15) was 1-3Mbps, (+14) 2-3Mbps, (+13) 2-4Mbps and all the way up to 42-76Mbps for Qscale(+16). It's like limiting the entire VBR spectrum to a smaller range, then Qscale sets that range high, low or somewhere in between. The range gets wider with higher quality settings (lower number) but the important thing is that it brings the lowest possible bitrate up.
"CBR mode" I assumed takes the bitrate "range" that is set by Qscale then multiplies it by a number of your choice (2.0x, 3.0x, etc.). CBR mode also tries to automatically set the highest Qscale it can (which is -16) but if your card or buffer is too slow is starts to scale it back, I got Qscale(-16 to -10) in CBR mode with a FlushRate of 4 but a constant Qscale(-16) with a FlushRate of 2. But what I still didn't understand is why are my bitrates in "VBR mode" with Qscale(-16) the same as "CBR mode" with Qscale(-16) set to 20.0x?
I've recorded with this card at an avg 160Mbps so I know it's not a hardware limitation.
So does "CBR mode" not multiply Qscale, does it just increase the selected Qscale's range upward?
Is most of that right so far? Thanks for everyone's work and help here ;D
Quote from: mimiloveyou on November 04, 2012, 05:31:44 AM
How do I use MJpeg 422 ?
Thank's.
Right now it's in the "Debug" Menu called "LV Dumping".
Scroll to it and press the "set button"; there are 7 choices: "Disabled", "A:/.JPEG", "B:/.JPEG", "A:/.422", "B:/.422", "A:/.JPEG/.422" and "B:/.JPEG/.422"
The second you select one it starts saving (note the red "busy" LED) your LiveView buffer as .jpgs or .422(or both). The choices with "A:" saves them to your card, I'm not sure where "B:" is but I heard that maybe for dual card cameras.
Once you plug the card into your computer you'll find all the jpgs or 422 files in your DCIM folder. Then you need to make them into an mov file, like a timelapse.
I was getting higher bitrates with the previous port. But with the new port's CBR there's very few artifacts. Noise reduction works so well :) VBR with Qscale -16 is almost as good as CBR at 20x. Can CBR be pushed to more than 20x? Also, which is better, jpeg or 422?
Quote from: akshayverma1 on November 04, 2012, 07:40:02 AM
I was getting higher bitrates with the previous port. But with the new port's CBR there's very few artifacts. Noise reduction works so well :) VBR with Qscale -16 is almost as good as CBR at 20x. Can CBR be pushed to more than 20x? Also, which is better, jpeg or 422?
I think the last build was All-I, which pushed bitrates higher but also needs a lot to avoid compression artifacts; if your card isn't fast enough you're likely to get compression artifacts. What bitrates were you getting and what's your card benchmarking at?
.422 seems to be uncompressed 4:2:2 frames while the jpegs are 4:2:2 compressed.
My jpeg files were 98-623KB, which are 15:1 to 2.5:1 compression ratios of the 1.5MB .422 files. Those compression ratios are JPEG's "Highest" and "High" Quality compression standards, so they should be clean from compression and motion artifacts ;-D Which only leaves line skipping and an 8bit color depth to worry about right?
They're good clean quality, except they are only 1056x704 so they need resizing to 1290x720(about 25% increase) or 1920x1080(almost 3x larger) so clear overlay HDMI out with external recorder will still have better quality but that requires having an external recorder :/
QuoteThose compression ratios are JPEG's "Highest" and "High" Quality compression standards, so they should be clean from compression and motion artifacts ;-D Which only leaves line skipping and an 8bit color depth to worry about right?
Hardly, you will see more blocking from jpeg compared to h264, as jpeg lacks in-loop deblocking. You will likely see more motion artifacts from jpeg as it does no sort of motion compensation. It would be better if the h264 engine could be used in place of jpeg, but it's probably impossible due to the h264 encoder's fixed input sizes.
I can not read a .422
Apparently audio recording works now if you set the flush rate to 24 (any other settings would just err70 as usual).
Great work, G3gg0! As always! Thanks for this update!
Looking forward when we can set this to All-I. :D
Speaking of audio recording, has anyone managed to record audio with Qscale at -16 or CBR factor 20.0x? All my cards crapped out at QScale-14 or -15 and CBR 16.0x-18.0x. Wondering which card can shoot max quality.
YES. YES. YES !!!!!!
Me Too.
"Great work, G3gg0! As always! Thanks for this update!"
Thank's P337 for explanations.
Quote from: Shizuka on November 04, 2012, 08:56:12 AM
Hardly, you will see more blocking from jpeg compared to h264, as jpeg lacks in-loop deblocking. You will likely see more motion artifacts from jpeg as it does no sort of motion compensation. It would be better if the h264 engine could be used in place of jpeg, but it's probably impossible due to the h264 encoder's fixed input sizes.
aww :( are you sure it's that bad? Either way this is the best JPEG could offer so I'm happy for that and if that's still not good enough there's also a full 422 option ;)
I may have miss-termed "motion artifacts", what I meant was prediction errors as these are all full frames like an intra codec.
Quote from: mimiloveyou on November 04, 2012, 08:58:19 AM
I can not read a .422
Use gimp with the plugin
http://registry.gimp.org/node/27589
Quote from: P337 on November 04, 2012, 09:30:04 AM
aww :( are you sure it's that bad? Either way this is the best JPEG could offer so I'm happy for that and if that's still not good enough there's also a full 422 option ;)
I may have miss-termed "motion artifacts", what I meant was prediction errors as these are all full frames like an intra codec.
Not sure what kind of prediction errors you are talking about. Predicted frames ("P frames") can also encode stationary information as intra blocks if motion can't be detected, ie. noise. Given the same amount of bits to work with, it's better to give the encoder the flexibility of choosing what's moved and what's stationary. Motion predicted blocks aren't [as] vulnerable to blocking artifacts (or the evil counterpart, deblocking) since they don't necessarily lie on block boundaries.
All-I is really meant for ease of editing (ie. seeking and splitting). You don't require an intermediate file for frame-exact splits with All-I files.
oh the .422/JPEG output was not intended in this experimental release.
consider it as an accident and do not rely on it.
it's also no MJPEG as someone called it. MJPEG is a lot more work than this hack.
about VBR/CBR
as far as i understand of what ive seen, the basic parameter is QScale. (maybe i'm wrong)
in vbr mode, QScale is constant and the output rate differs depending on scene complexity (=VBR)
if CBR mode is selected, there is a controller that varies QScale depending on current bitrate.
so the bitrate will be held at a constant average level.
QP = 26 - QScale
QP: the one displayed on screen when recording. its a codec output value (10 ... 42)
QScale: codec input parameter configured in ML menu (-16 ... +16)
Is there any reason that 422 JPEG recording makes sense?
Let's look at Canon's 1056x704 live view:
This resolution is acquired at the sensor by line skipping: every five rows are read from a sensor area of 5202x3465 (3:2).
The resulting 5202x693 RAW is demosaiced and rescaled to 1056x704 4:2:2 for live view.
A 5202x693 bayer sensor has a chroma resolution of 2601x396.
The final resolution is the minimum of every step above: 1056x693 luma resolution, and 528x396 chroma resolution. Vertical chroma resolution is limited by bayer line skipping.
5202x693 RAW Bayer -> 5202x693 Y / 2601x396 UV [assuming 2x2 superpixel required for chroma determination]
1056x704 422 YUV -> 1056x704 Y / 528x704 UV
We can also look at the 1728x792 recording live view:
This resolution is acquired at the sensor by line skipping: every three rows are read from a sensor area of 5202x2925 (16:9).
The resulting image, 5202x975, is demosaiced and rescaled to 1728x792.
A 5202x975 bayer sensor has a chroma resolution of 2601x487.
The final resolution is the minimum of every step above: 1728x792 luma resolution, and 864x487 chroma resolution. Again, line skipping limits our vertical chroma resolution.
5202x975 RAW Bayer -> 5202x975 Y / 2601x487 UV [assuming 2x2 superpixel required for chroma determination]
1728x792 422 YUV -> 1728x792 Y / 864x792 UV
If the video was saved as a 1728x792 4:2:0 H.264 video, we would actually lose vertical chroma resolution. But Canon upscales this to 1920x1080 before encoding it:
A 1920x1080 4:2:0 video frame contains a 1920x1080 Y (luma) and 960x540 UV (chroma) planes.
The final resolution available is 1728x792 Y / 864x487 UV, noticing that these values fit comfortably within the resolution limits in 1920x1080 4:2:0 video. Canon's 1080p is large enough to capture all of the detail in 1920x1080 4:2:0 as provided in the 1728x792 4:2:2 (since there are only ~480 real lines of color information).
to explain myself the way how the encoding works and what i am tuning, i made some screenshots.
consider this as source image that has to be encoded:
http://upload.g3gg0.de/pub_files/ec9400781e1b25f5ff9df7cde66b96a7/mpeg_frame_1.PNG
we are inspecting an I-frame now
the first step the encoder does, is finding some parameters that synthesize to something like this:
http://upload.g3gg0.de/pub_files/e4e39b57861be9c2558f171d845da599/mpeg_frame_2.PNG
it is a approximation of the image content, repesented with as few values as possible.
now there is still some work to do. the encoder calculates a delta between the synthesized image and the real one.
this is the remainder data:
http://upload.g3gg0.de/pub_files/5cbb6df0f2ee6bd96587df4f42288bfd/mpeg_frame_3.PNG
now this data is encoded too using some other algorithms and the frame is done.
now the next frame - its a P-Frame.
http://upload.g3gg0.de/pub_files/82e3692945b698a7ef30d1335bc3e868/mpeg_frame_5.PNG
the green blocks are predicted - they are just blindly copied from the frame before.
without any modification in the first step.
the red blocks are the intra-blocks, they are synthesized as all blocks in an I-Frame.
the resulting image is now a mix of copied portions from the previous frame and synthesized blocks.
if we now look at the remainder, there is a lot less data that is waiting for encoding again:
http://upload.g3gg0.de/pub_files/6c788470be9e8b8587b6817fc329d861/mpeg_frame_4.PNG
(compare that to frame 3)
thats the reason why I-frames eat so much bandwith without huge improvements.
how can I switch Audio recording of ?
Quote from: StefanKeller.AC on November 04, 2012, 11:19:26 AM
how can I switch Audio recording of ?
in canon menu. audio recording off instead of automatic.
Quote from: P337 on November 02, 2012, 04:47:24 AMIf you have a super fast card I recommend setting CBR to 20x and Flush to 1. Then enable spotmeter and select the 0-255 option. Then set ISO to 128,000 and use your shutter or ND filters to underexpose to "25" on the 0-255 spotmeter. This always shoots my bitrate up to it's max but my cards, which are "200x" 170Mbps (21MB/s)*, peak at 200 Mbps for 1 second then the buffer can't keep up** so the recording stops :/
*ML Benchmarked my card at 21.5MB/s
**my Flush rate is set to 2 (my camera throws err70 if flush=1)
What is the significance of setting the flush rate so much lower? Is it useful so that when we record at a higher bitrate we won't overflow the buffer? If that's all, I would keep the flush rate at 2-4. I didn't have a single issue in my dozen or so test recordings that were all 150-200 Mbps. Setting it to a lower flush rate requires a faster card. Not in terms of bandwidth, but latency of access. Use the buffer as a buffer and it'll behave properly.
I am using SanDisk's Extreme 16 GB cards. You can get them on sale at ~$50, normally at $60. For the price vs performance, it's hard to beat this card. Have a look here:
http://www.robgalbraith.com/bins/camera_wb_multi_page.asp?cid=6007-10294 (http://www.robgalbraith.com/bins/camera_wb_multi_page.asp?cid=6007-10294)
For comparing all of the cards. This has been the most reliable site for card performace. Caveat: Rob has retired the website. This happened before rumors of the 2.0 firmware that Canon released which drastically improved memory handling and thus performance. However, it's a solid baseline for at least another 6 months.
Quote from: 1% on November 04, 2012, 02:55:51 AMYou shouldn't have to use CBR only, set qscale to max (do you guys have that yet?) and then camera records at fixed quality. All frames will be QP10.
There is no such thing as real CBR on canon anyways. Its all VBR just with constraints via quality/qp H264 parameters. You're not directly controlling the rate, you're just telling the camera "even if frame size is *friggin huge* pick a high QP for the next frame.
Fixed quality would be fixed quantizer, which would be VBR, no? Due to general ignorance (which it looks like you guys do a really good job at helping alleviate) most think that CBR > VBR in terms of quality. However, VBR > CBR because it's much closer to assuming fixed quality for each frame. THis really is a better mode to record in.
Quote from: P337 on November 04, 2012, 03:39:05 AM
Thanks 1%
Does that mean a static scene may just ignore "CBR" settings and base it off the first frame's complexity?
Well I don't have a problem maxing out the bitrate indoors, by underexposing with ISO set to 128000, which get my card's full 160Mbps but it's too noisy so I'm trying to get that max bitrate with proper exposure.
Go find some grass. First attempts at all I-Frame encoding gave me ~230 Mbps. I haven't gotten around to 'CBR' testing yet.
Quote from: g3gg0 on November 01, 2012, 04:10:04 PM@1%:
> You guys haven't pushed luma/chroma quality up yet
you mean QP(Y;U;V)? that is according to analysis tools exactly the same as QP (10).
one correction: you said, camera flushes after each GOP - you meant after each two GOPs, as this is one second :)
at least via x.264, you could control the quantizer based off of scalar values. Do you guys have controls equivalent to deadzone_inter, deadzone_intra, and chroma_qp_offset?
Quote from: g3gg0 on November 04, 2012, 11:17:41 AM
to explain myself the way how the encoding works and what i am tuning, i made some screenshots.
...
thats the reason why I-frames eat so much bandwith without huge improvements.
It took me a long while to understand this process back in the day when I first started working with video encoding. This explanation you just gave is perhapse the best I've ever seen online. It'll be a valuable resource for those just starting out trying to figure out what mode to use.
As a followup to restate what others have said, the chief reason why all-I is required is for special effects work. Most pieces of software that do this need to cut into a video at an I-Frame and not havfing to wait 3, 4, 5, 10 frames later works much more easily without losing quality
Quote from: Digital Corpus on November 04, 2012, 12:22:45 PM
As a followup to restate what others have said, the chief reason why all-I is required is for special effects work. Most pieces of software that do this need to cut into a video at an I-Frame and not havfing to wait 3, 4, 5, 10 frames later works much more easily without losing quality
Yep. It also helps when color correcting. I like All-I.
Noticed than Magic Zoom no more flickers. Also when REC/STBY notification is set to "Red Crossout" red box does not disappears properly when recording is started. Got a "dir error" message after Err70. "Flush rate" setting does not restores after reboot. True VBR feature is very useful.
Quote from: Digital Corpus on November 04, 2012, 12:22:45 PMat least via x.264, you could control the quantizer based off of scalar values. Do you guys have controls equivalent to deadzone_inter, deadzone_intra, and chroma_qp_offset?
It took me a long while to understand this process back in the day when I first started working with video encoding. This explanation you just gave is perhapse the best I've ever seen online. It'll be a valuable resource for those just starting out trying to figure out what mode to use.
As a followup to restate what others have said, the chief reason why all-I is required is for special effects work. Most pieces of software that do this need to cut into a video at an I-Frame and not havfing to wait 3, 4, 5, 10 frames later works much more easily without losing quality
thanks :) i hope that was correct.
its just high-levle explained by watching what i see. not sure if that is the corrent process
about those quantizer parameters. i am sure we have access to them, its just hard to identify them.
we are looking for canon dev's log/assert messages to find them.
maybe the 5D3 ini file and the registers where data goes to will help.
what would you expect from the parameters you said?
yes all-i is good for post. so i think the best is to configure the GOP size.
the ones wo dont do post will benefit from better details with larger GOP groups.
the others can choose all-i for post processing.
Quote from: xmd5a on November 04, 2012, 03:15:19 PM
"Flush rate" setting does not restores after reboot.
I thought I was the only one having this issue. :D Yep, flush rate kept defaulting to 4 after restart on mine too...
Quote from: g3gg0 on November 04, 2012, 03:44:46 PM
thanks :) i hope that was correct.
its just high-levle explained by watching what i see. not sure if that is the corrent process
As far as I know it's dead on. The difference between I, P, and B frames is fairly simple. It's the algorithms used to determine which macroblock does what that makes things become very complex very quickly.
Quote from: g3gg0 on November 04, 2012, 03:44:46 PM
about those quantizer parameters. i am sure we have access to them, its just hard to identify them.
we are looking for canon dev's log/assert messages to find them.
maybe the 5D3 ini file and the registers where data goes to will help.
what would you expect from the parameters you said?
chroma_qp_offsetChroma requires less detail since the human eye is more sensitive to detail/luminosity than it is color. Thus color information does not need to be encoded with the same quality. However, green/blue screen scenes will benefit from this being increased.
As for the deadzone parameters, they affect small, fine details and grain/noise retention. They can help when shooting at high ISO to aid in not flattening ISO noise into a macroblock to help clean up the frames. For a more complete description, here is a copy & paste from the man page.
Quotedeadzone_inter=<0−32>
Set the size of the inter luma quantization deadzone for non-trellis quantization (default: 21). Lower values help to preserve fine details and film grain (typically useful for high bitrate/quality encode), while higher values help filter out these details to save bits that can be spent again on other macroblocks and frames (typically useful for bitrate-starved encodes). It is recommended that you start by tweaking deadzone_intra before changing this parameter.
deadzone_intra=<0−32>
Set the size of the intra luma quantization deadzone for non-trellis quantization (default: 11). This option has the same effect as deadzone_inter except that it affects intra frames. It is recommended that you start by tweaking this parameter before changing deadzone_inter.
Quote from: g3gg0 on November 04, 2012, 03:44:46 PMyes all-i is good for post. so i think the best is to configure the GOP size.
the ones wo dont do post will benefit from better details with larger GOP groups.
the others can choose all-i for post processing.
Yeah, I'd like this one a lot. Because P-frames are a lot less expensive to encode, it's easier to save bits and increase quality with this.
I noticed earlier comments about how the interval of when Canon's firmware flushes frames in terms of GOP size. Does this account for why an Err 70 occurs when trying to record audio under this [alpha] firmware?
Quote from: feureau on November 04, 2012, 03:52:49 PM
I thought I was the only one having this issue. :D Yep, flush rate kept defaulting to 4 after restart on mine too...
I can confirm as well with the initial CBR firmware you posted. I haven't check the all I-Frame or most recent one you posted in the past couple of days.
Quote from: Shizuka on November 04, 2012, 11:01:34 AM
Is there any reason that 422 JPEG recording makes sense?
Let's look at Canon's 1056x704 live view:
This resolution is acquired at the sensor by line skipping: every five rows are read from a sensor area of 5202x3465 (3:2).
The resulting 5202x693 RAW is demosaiced and rescaled to 1056x704 4:2:2 for live view.
A 5202x693 bayer sensor has a chroma resolution of 2601x396.
The final resolution is the minimum of every step above: 1056x693 luma resolution, and 528x396 chroma resolution. Vertical chroma resolution is limited by bayer line skipping.
5202x693 RAW Bayer -> 5202x693 Y / 2601x396 UV [assuming 2x2 superpixel required for chroma determination]
1056x704 422 YUV -> 1056x704 Y / 528x704 UV
We can also look at the 1728x792 recording live view:
This resolution is acquired at the sensor by line skipping: every three rows are read from a sensor area of 5202x2925 (16:9).
The resulting image, 5202x975, is demosaiced and rescaled to 1728x792.
A 5202x975 bayer sensor has a chroma resolution of 2601x487.
The final resolution is the minimum of every step above: 1728x792 luma resolution, and 864x487 chroma resolution. Again, line skipping limits our vertical chroma resolution.
5202x975 RAW Bayer -> 5202x975 Y / 2601x487 UV [assuming 2x2 superpixel required for chroma determination]
1728x792 422 YUV -> 1728x792 Y / 864x792 UV
If the video was saved as a 1728x792 4:2:0 H.264 video, we would actually lose vertical chroma resolution. But Canon upscales this to 1920x1080 before encoding it:
A 1920x1080 4:2:0 video frame contains a 1920x1080 Y (luma) and 960x540 UV (chroma) planes.
The final resolution available is 1728x792 Y / 864x487 UV, noticing that these values fit comfortably within the resolution limits in 1920x1080 4:2:0 video. Canon's 1080p is large enough to capture all of the detail in 1920x1080 4:2:0 as provided in the 1728x792 4:2:2 (since there are only ~480 real lines of color information).
I am very very much interested into how the 7D does it's image scaling. Where did you come across this info? I'd like to look more into it especially due to tests with resolution and what have you. The upscaling explains a little bit about the softness of a 1080p frame and gives me a good base to start some testing on with and without my VAF-7D.
I think its mostly correct, from hours of screwing with 600D.
Quoteabout those quantizer parameters.
There might be a way to access them directly, I actually hope so because then we can try QP below 10. But the parameter that both CBR and "VBR" modify is sliceqpd. It is the direct "quality" of the frame. How sliceqpd turns into a QP of 10 (or X) on the output side I don't know, it might be internal to digic.
QuoteWe can also look at the 1728x792 recording live view:
Looks like a real good interpretation on why they picked the sizes they did. So crop mode really has that little chroma resolution, wow. But does that mean the numbers won't work out to scale it to something else? Non crop HD is even more skips and a lower resolution. Looking in the logs, it says the available area is bigger, close to that 2000 size but non at 396 or 487. But also the window being used is smaller.
[GMT][WAKU] Avail WinW:H(1036:1036)->(1036:1036)
[GMT][WAKU] Avail X:Y(2920:1268)->(2920:1268) deadzone_inter- May be D1/D2 in MVR config. But not sure what low level parameter is being modified on the hardware encoder. All I found affecting Jpcore was sliceqpd/deblocking/picqpy/picqpc.
Quote from: 1% on November 04, 2012, 04:26:56 PMdeadzone_inter- May be D1/D2 in MVR config. But not sure what low level parameter is being modified on the hardware encoder. All I found affecting Jpcore was sliceqpd/deblocking/picqpy/picqpc.
Stupid question. I cannot check because I don't have the tools with me nor the time to install them on this computer, but is there nothing stored as a header in the video files for us to peak at?
Quote from: 1% on November 04, 2012, 04:26:56 PMLooks like a real good interpretation on why they picked the sizes they did. So crop mode really has that little chroma resolution, wow. But does that mean the numbers won't work out to scale it to something else? Non crop HD is even more skips and a lower resolution. Looking in the logs, it says the available area is bigger, close to that 2000 size but non at 396 or 487. But also the window being used is smaller.
[GMT][WAKU] Avail WinW:H(1036:1036)->(1036:1036)
[GMT][WAKU] Avail X:Y(2920:1268)->(2920:1268)
You kids are so pampered; back in the NTSC days, we had thirty lines of color information!
The human eye is terrible at picking out color resolution, so bayer sensors take advantage of that weakness. I wouldn't expect the "low" chroma resolution to have any visible impact. (Worst case scenario: sharp edged highly saturated objects - see footnote)
The 396/487 figures (which are estimates) are post-demosaic "chroma resolution" values, which are estimated by reducing the source resolution by two due to the assumption that a 2x2 superpixel is required to get color information for a group of pixels. The reality is somewhat murkier (think PenTile displays) - demosaicing inherently blurs in the vertical and horizontal direction. Regardless of the actual resolution, the fact that demosaicing reaches above and below the current line means that the reduced chroma resolution of 4:2:0 is probably more appropriate than 4:1:1 or 4:2:2.
TLDR: you won't find these values in the firmware because they are an estimate of the resolution; the doubled values may be lurking around though.
{footnote}
I'll use an example of 4:2:0 chroma resolution artifacts here, although it's not completely relevant. (What is relevant is a live view 4:2:2 dump - and seeing how Canon deals with chroma there)
http://upload.g3gg0.de/pub_files/ec9400781e1b25f5ff9df7cde66b96a7/mpeg_frame_1.PNG - see the red color blocking at the mop on the floor? This artifact is produced by nearest neighbor upscaling of the U and V planes to the Y plane's size. A filter to interpolate it goes a long way to reducing these artifacts. :)
Of course, the real question is: how does the Canon demosaic the sensor to 4:2:2? The cynic in me thinks that since it's going to be rescaled to video resolution anyway, there's not much reason to do a good job during the demosaic. Determining actual chroma resolution from 422 files is difficult: Canon does image processing *before* the Live View image is provided to us. I looked, I saw sharpening and scaling artifacts.
Quotehow does the Canon demosaic the sensor to 4:2:2
You can look at raw 4:2:2 dumps of what the input to the encoder is. The "while recording" sizes are exactly what is being encoded.
Just drop a .422 while recording, or in my case any time in video mode. Available image from that log is full sensor size so we can probably chose the line skips. Might be worth comparing 422 at zoomed out, 3x, 5x (much smaller), 10x (very tiny), etc.
Quote from: 1% on November 04, 2012, 06:40:19 PM
You can look at raw 4:2:2 dumps of what the input to the encoder is. The "while recording" sizes are exactly what is being encoded.
Just drop a .422 while recording, or in my case any time in video mode. Available image from that log is full sensor size so we can probably chose the line skips. Might be worth comparing 422 at zoomed out, 3x, 5x (much smaller), 10x (very tiny), etc.
I have a T2i, but I'll take a look at it. I noticed rescaling artifacts in 1056x704 modes, but that's the only size I've taken silent pics in.
Full image pipeline, as interpreted by me:
[Sensor] --lineskip--> [LineSkip RAW] --demosaic--> ?DIGIC image? --DIGICprocessing--> {YUV422}
This is used for live view
{YUV422} --video_rescaler--> {YUV422-to-H264} --h264_encode--> (H.264)
DIGICprocessing - sharpness, color, contrast, curves (Standard, cinestyle, flaat_10, etc.), rescaling
[raw data], ?unknown data?, {yuv data}, (compressed data)
I should make some pics at all sizes. 600D can do 1728x956 down to like ~640x480 YUV size. Zoomed out is I think the same size as you have while recording.
Quote from: g3gg0 on November 04, 2012, 01:00:13 AM
here (http://upload.g3gg0.de/pub_files/874f4306b8683eb88e9eeccbb704ae14/7D000203_ML.FIR)is another experimental build.
i set GOP to 4 and enabled VBR/CBR selection.
also the flush rate is now limited to 2-50.
flush rate is on both digics set via cache hacks now - this means it is less likely to abort recording after the first 100ms, although
it is likely to happen if you start recording the first time, as the drivers seem to read some filesystem information which takes some time (?)
successive recordings will fail less likely.
what do you think about that?
i think i will make the GOP selection configurable somewhen, so you can choose your favorite.
I'm loving this new version right now. VBR is very lovely. Time for some low light video =D.
Got 422s from my T2i. Turns out I was using the wrong values for FullHD live view image resolution, but it's the same idea.
1720x974 and 1056x704 - not scaled at all :)
1024x680 - scaled! (5x / 10x zoom modes) - massive bilinear resize artifacting - don't use this mode!
--
Now with a 1720x974 figure, we can take a look at how Canon does it:
5202 x 3465 sensor -> 5160 x 2922 16:9 sensor -> 5160 x 974 every-3-horizontal-lineskipped sensor
This is assuming Canon actually does full reads for each line, instead of skipping every three pixels...
I'm not sure whether the demosaic engine does vertical line skipping. If it does:
5202 x 3465 sensor -> 5160 x 2922 16:9 sensor -> 1720 x 974 every-3-horiz+vert-lineskipped sensor
That's a very familiar resolution, don't you think?
We would also see massive moire / aliasing in the vertical direction. I didn't see this, so I think Canon isn't doing vertical line skipping.
My guess is Canon thought they might have to do H+V line skipping, but did not end up needing to.
Also, ML 2.2 had FullHD silent picture. Was this feature removed in ML 2.3, or is there some other way to access it without actually recording a movie?
Tried with Sandisk Extreme CF with no issues at x20. Crisp detail improvement is AMAZING, but for any reason, soft detail looks very close to Canon's default bitrate (blocky). Can't decide between regular mode or ALL-I. Can't wait to go home and do more testing.
It stills drive me crazy that a 25mbps HDV looks better in detail and compression artifacts than a Canon's 50mpbs+ H.264.
Is this beacuse of the encoder Canon uses? settings?
Quote from: ideimos on November 06, 2012, 10:24:42 AMsoft detail looks very close to Canon's default bitrate (blocky).
maybe i can play with deblocking.
Quote from: ideimos on November 06, 2012, 10:24:42 AMIs this beacuse of the encoder Canon uses? settings?
Canon's encoder uses the bare minimum subset of H.264 in order to say they use H.264. No B-frames, no CABAC compression, only one reference frame... and worst of all, no adaptive quantization, which spends bits in areas that needs them the most.
Quote from: Shizuka on November 06, 2012, 05:21:22 PM
Canon's encoder uses the bare minimum subset of H.264 in order to say they use H.264. No B-frames, no CABAC compression, only one reference frame... and worst of all, no adaptive quantization, which spends bits in areas that needs them the most.
Makes sense. Thanks for the explanation.
Quote from: Shizuka on November 04, 2012, 09:45:48 PM...
Now with a 1720x974 figure, we can take a look at how Canon does it:
5202 x 3465 sensor -> 5160 x 2922 16:9 sensor -> 5160 x 974 every-3-horizontal-lineskipped sensor
This is assuming Canon actually does full reads for each line, instead of skipping every three pixels...
...
Aside form line skipping, there is pixel binning. Effective pixel count of the sensors is 5184x3456. However, DXO reports 5360x3515 for the 7D and DP Review states that the T2i's sensor is just a little different than the 7D's. Despite the same effective resolution, it is possible that the skipping and binning is a little different between the two.
On a similar note, according to Cambridge in Color (http://"http://www.cambridgeincolour.com/tutorials/diffraction-photography-2.htm"), the aperture range where we may start to see loss in resolution is f/19 to ~f/28. However, our sensor is not 2.1 MP so there may be some interesting results as we hit that range. This is next on my list of things to test which are currently
1) Visually perceptible difference between the 7D w/ & w/o the VAF-7D
1.1) Pixel peeping difference as well
2) Tests to try and establish the pattern for which Canon skips rows of pixels and bins them together to help more finely establish resolution information
2.1) Establish the same baseline for 720p recording, which currently suffers from horizontal scaling artifacts
3) Test theoretical diffraction limits of 2 and see how the image holds up.
Quote from: Digital Corpus on November 07, 2012, 02:29:47 PM
Aside form line skipping, there is pixel binning. Effective pixel count of the sensors is 5184x3456. However, DXO reports 5360x3515 for the 7D and DP Review states that the T2i's sensor is just a little different than the 7D's. Despite the same effective resolution, it is possible that the skipping and binning is a little different between the two.
On a similar note, according to Cambridge in Color (http://"http://www.cambridgeincolour.com/tutorials/diffraction-photography-2.htm"), the aperture range where we may start to see loss in resolution is f/19 to ~f/28. However, our sensor is not 2.1 MP so there may be some interesting results as we hit that range. This is next on my list of things to test which are currently
1) Visually perceptible difference between the 7D w/ & w/o the VAF-7D
1.1) Pixel peeping difference as well
2) Tests to try and establish the pattern for which Canon skips rows of pixels and bins them together to help more finely establish resolution information
2.1) Establish the same baseline for 720p recording, which currently suffers from horizontal scaling artifacts
3) Test theoretical diffraction limits of 2 and see how the image holds up.
The 7D sensor resolution figure is the number of addressable pixels on the array. Doesn't exclude the masked (always black) pixels. See http://lclevy.free.fr/cr2/#app for a table of sensor sizes - but note that the masking values are the Canon values, which exclude extreme edge pixels. More pixels can be extracted from the sensor dump. I have a modified version of dcraw available that can reveal these edge pixels.
I took a look anyway (it was a two-byte change) at the 7D vs. the T2i. The difference is that the 7D's sensor has 16 additional black mask pixels on the left side, and 16 additional total pixels. Imaging area remains the same.
I think there is horizontal line skipping and vertical pixel binning from my experiments. More interestingly, I saw
vertical and horizontal stepping artifacts in the 1720x974 YUV dump from my T2i, suggesting it's possible that the movie recording YUV buffer is actually upscaled from a lower resolution.
Brother doing some welding on my car (http://youtu.be/868n_wcLeHc)
Anyway we can have ML set the flag on the video files so that RGB values from 0-15 are not crushed to black?
# Magic Lantern v2.3.NEXT.2012Nov04.7D203
Start : 2012/11/12 12:57:16
Lens name : EF-S17-55mm f/2.8 IS USM
ISO : 1250
Shutter : 1/48.194s
Aperture : f/4.9
Focal length : 55 mm
Focus distance : 1180 mm
White Balance : 5500K, Magenta 0, Blue 0
Picture Style : Marvel's Adv 3.4.1 (1,-4,-3,0)
FPS : 23.976
Bit Rate (VBR) : QScale -12
CSV data:
Time,ISO,Shutter,Aperture,Focal_Len,Focus_Dist
12:57:16,1250,48,4.9,55,118
well, a picture style would help you with that :)
But that would force me to loose information :(
uhm any picture style is causing information loss.
you have 14 bits of information being packed in 8 bits where the pic style is the lookup table for transforming.
information loss is inavoidable, its just the question which information you want to loose.
HELP! I made a few tests and I don't understand why at iso 160 I get 36 mbs CBR at 20x setting, and at iso 6400 I get 208 mbs CBR, it looks like the higher I go with ISO the higher the bitrate. How should I use this?
The higher the ISO, the higher the noise, which does not compress very well hence the higher bitrate
And for example shooting in daylight how is it better? Using 20x bitrate at 160 iso? Or higher iso?
Quote from: exe on November 19, 2012, 05:12:16 AM
And for example shooting in daylight how is it better? Using 20x bitrate at 160 iso? Or higher iso?
Use 20x bitrate at 160 ISO. It's better. The low bitrate is due to the low details in your shot. Try shooting at tall grasses or lots of leaves on some trees, the bitrate will shoot up.
hmm not sure. what about banding?
wasnt shooting with some (little) noise that gets filtered in post better to prevent banding introduced during MPEG compression?
I dunno. All the banding goes away when I denoise my footage. There's very minimal banding when you shoot at low ISO anyway. OTOH, the image quality (sharpness, color, DR, etc) goes down rather quickly above 800-1600 ISO.
There's that strange banding though: in the Y channel when you turn on Highlight priority or Auto Lighting Optimizer and shoot at around ISO 3200, after you denoise the chroma, you can see there's a square pattern going vertically and horizontally all over the place.
That's normal. Canon 'could' fix it with a firmware update, and you only need black frame subtraction in a still image to fix it, but alas... I do admit that it is a tad annoying :-\
Hello.
I am mostly using 7D for downhill bikeing videos - so 720p 60fps.
I was so happy to see that i can change the bitrate, and hoping to have better quality. I have done some real world footages, but i dont see any difference in sharpnes, or quality. Settings i tried: CBR 15x, and VBR Q -12. Did I do something wrong? Thanks
You should see something in the file size.. and that means there is more info = more details.. Naybe it's hard to spot.. And the sharpness problem could be the lens.. Most of the time a cheaper lens makes a less sharp image :-)
Quote from: jphansen on December 05, 2012, 05:16:54 PM
You should see something in the file size.. and that means there is more info = more details.. Naybe it's hard to spot.. And the sharpness problem could be the lens.. Most of the time a cheaper lens makes a less sharp image :-)
So I should be satisfied because file is bigger? :D It doesn't matter to me, if file is bigger when quality is the same (by naked eye...but i dont need it to be better in some graphs and tests, i want beter video while shooting between trees and leaves!)
I am using 70-200 f/4 L IS, and Tamron 17-50 2.8. Both are very sharp in photography, and for video you dont need such a good optical quality(I have read somewhere), so i dont think this is reason.
Quote from: exe on November 19, 2012, 05:12:16 AM
20x bitrate
:D i got Canon T3i (D600) and when i use x.2.0 bitrate in video recording my camera cant stand it and it geting hot and lol hapend on end..camera freaze.. max what is quite good for my camera its x.1.6 ..what camera u use with x20 bitrate ? and it not stops ?
i use 45mb/s reading and wrighting Class 10 San Disk Card
Quote from: lukasildza on December 05, 2012, 05:32:04 PM
So I should be satisfied because file is bigger? :D It doesn't matter to me, if file is bigger when quality is the same (by naked eye...but i dont need it to be better in some graphs and tests, i want beter video while shooting between trees and leaves!)
I am using 70-200 f/4 L IS, and Tamron 17-50 2.8. Both are very sharp in photography, and for video you dont need such a good optical quality(I have read somewhere), so i dont think this is reason.
when u got two same lenght of file lets say 10 second Movie File`s but one is 1 GB and another one is lets say 100 mb.. its mean..more frames per second more resolution more bitrate more high quality..if u dont see diference u need maybe check ur ICC profile monitor .. or check color setings at ur software..and what Monitor ICC profile u use.. with wrong ICC profile and balance colours in ur computer and on camera ..when those two coplours profles are not same your colours can looks weird..( like in 8 bit frames ) so u can thing its low quality..
alsow tell me what software u use to Preview ur videos recorded with ML on your Canon..cus this can be problem to..
all video players use some codecs and they use own setings.. the best thing its to not powering UP quality of movie on your Video Player.. make player show u original colors with OUT Program Corection..cus sometimes ( mostly ) these software (video player ) corection can kill quality in movie
Edit :
i thing will be coool to have in MAgic Lantern Video SubMenue option like :))))))... AntyAliasing :))..ooo yeaaa .. this will Extreamly Make Quality SOOO high UP !
Well I meant. Bigger file = better quality.. Maybe more details in the shadows and stuff like that.. But it doesn't mean sharper image.. Thats your lens that have to do that job..
Quote from: lukasildza on December 05, 2012, 05:32:04 PM
So I should be satisfied because file is bigger? :D It doesn't matter to me, if file is bigger when quality is the same (by naked eye...but i dont need it to be better in some graphs and tests, i want beter video while shooting between trees and leaves!)
I am using 70-200 f/4 L IS, and Tamron 17-50 2.8. Both are very sharp in photography, and for video you dont need such a good optical quality(I have read somewhere), so i dont think this is reason.
Could you tell me a great menu setting bitrate for my Canon 7d?
I would like a good compromise between quality and file size.
Thank you very much.
What does a monstrous bit rate like the one's capable with this mod actually benefit in terms of image quality? Can I see some comparisons, just interested. Thanks.
Off topic: How do I increase the CBR/VBR on my 5D3 - cannot see the options anymore on the H.264 line - January 8, 2013 build.
So how difficult was it to remove the 3.0 CBR limit in the source code? I'm interested in trying this myself on my 700D. Also, does Qscale -16 set the maximum QP possible, or is there still room left to go? Lastly, it would be really nice if we could get GOP control brought into the main ML branch.