Messages

Hi,

could somebody maybe send upload me one darkframe and one or two pictures with severe vertical FPN (ISO100 pulled from Best would be a linear TIFF 16bit (maybe even including the OB zones)?

CR2 would be OK also, although I will not be able to get the optical blacks with my current software.

Thanks in advance.

[If you would consider installing the bootflag and full ML]

Hi Neil,

I am not a professional video guy. I just acquired some experience with video over the past 3 years. Recently I recorded a little bit of raw video now working to get a efficient workflow with it.  Occasionally I worked (as a sound engineer) together with a group of indie film makers at some little concerts. So I got some additional information by just watching.

If you want to go for 1280x720 anyway and have to record a rather long show with just one camera without experience I would suggest the following "safety first attempt":

Do not consider RAW video.

Use the built in h264 video capabilities and 1920x1080 resolution. No sharpening or Sharpness 1. The footage will be soft, but as you are going to downsample it to 1280x720 you will gain some sharpness. In post production you can sharpen to taste, resulting in a sharpness that was satisfying my personal needs most of the time.
Use full stop ISOs or the pulled ones (160, 320).  ISO 100-1600 do almost have the same quality on 5DIII
Restart the recording after each performance (!).
You can use the All-I function if your camera and card support it (test it before)
Use for example the "Flaat10" picture style or (if really needed!) Flaat11. If the scene is not high dynamic range you may be able to just use Neutral with contrast -3 or Flaat9 or Marvels Cinestyle.

Make sure the exposure is right and your white balance is optimal. In such a controlled environment you even could set the white balance beforehand with a grey-card on stage.
Speak to the lighting engineer (if any) to keep the front illumination at a reasonable level (i.e. bright enough that you can get a proper exposure at no higher ISO than 1600).

The advantages (compared to RAW video) are:

- You can record single performances of up to 12minutes without having to stop and restart the recording.
- You do not have to buy 2x 1000x 64GB memory cards (each lasting for only ~12mins of footage) and swap them for dumping files to the computer after each performance.
- You end up with files that are compatible to every computer. So even without post-production the farmers can view your videos. This reduces pressure and you can take some time to edit the video.
- your workflow is A LOT faster!

The disadvantages are:

- The footage is a little soft (even compared to designated video cameras at about half the price point of the 5DIII)
- You can only capture 8-10 stops of dynamic range depending on your picture style settings
- You can not use Adobe Camera Raw to grade your footage (fill lights / shadow recovery)

Possible applications for ML in your case:
You could use the MagicLantern tools like ETTR and Histogram / Vectorscope to make sure the exposure and white balance is right.
If you have Firmware 1.1.3:
You can have a look at the Alpha 3 (http://www.magiclantern.fm/forum/index.php?topic=2602.0) which will (to my knowledge) at least support you with a live video histogram. It can be loaded like a firmware update and not alter your camera. So it is safe to play around a bit. If anything strange happens just pull the battery and your camera is like before. (But of course there is no guarantee by any means). Please note that this Alpha has a strongly reduced feature set compared to full ML.
If you would consider installing the bootflag and full ML be aware that the boot flag is not yet removable and your Camera will be about 1-2sec slower in powering up!

Despite your question: 2:30 of video can get quite boring without additional camera angles / different framings.
If this is a "just for fun" work and you want to get into video it should be ok. If they rely on it maybe get some help of an advanced video guy. I suppose for a nice media coverage it would take at least 3 camera angles (one of them being a mobile camera).

I hope I could give some starting point for reseach ;-)

Works like a charm :-)
Also the freezing problem:

Thanks for the quick reply.

mlrawviewer is running as 2 threads. Both close when I hit ESC.
I am currently downloading the newest driver ...was really surprised that my current one is more than one year old. I will update this post with my results.

UPDATE:
- newest ATI driver installed: Still a pink frame and still the black window.
- I noticed that the black window does not appear when I drop the same MLV file on mlrawviewer again. It only happens with different files. Btw I have version 1.0.4 _alpha32 from the .zip as I can not build it on my own from the repository.

is now gone!

This 20bit version with soft-film=2 really solved a severe colour blocking problem I faced in a skyline HDR of Melbourne.
With the old version the sky consisted of huge pink / blue squares. Now it is all smooth.

GREAT :-)

You're right, in my euphoria I missed the fact that your averaging provided me with lower Gaussian noise component / virtually increased OB.
I just took the values you used in your Reply #584.

I repeated my test with 10pixels (which should be about 80 pixels of OB), resulting in slight (but to my eye pleasing) vertical FPN reduction.

The effect on horizontal FPN is negligible.

Below the reduction in mean horizontal pixel values distribution for different (virtual) OB sizes:

It is apparent that for OB=40 the FPN may even get worse.

I hope with this post my tests contribute at least a little for those, who are not such familiar with FPN...

[is]

Nope, answered here.

You do get an improvement, but it's small.

A low covariance is nothing special if there is a lot of Gaussian noise evident. The fact that there is some covariance tempted me to try a FPN removal.
The improvement in terms of dynamic range my be small. Nevertheless the improvement in visual quality can be significant.
But judge by yourself:
I took your sample ISO 200 pulled to 100 and split it in the top 80 and the lower 400 rows. Below you see the lower 400 without processing:

Then I did what g3gg0 just suggested. An vertical average (let's call it top) of the top 80. Below you see the result of a (floating point) subtraction of this median each line of the image pixels. After the subtraction I added Mean(top) to each pixel value to not shift the blacklevel and rounded the result to yield the new pixel values:

Now the same done in the vertical direction:


My subjective opinion is that removing the FPN can improve the visual quality significantly.

Please note that this test is just a quick&dirty one so there my be some errors induced during re-export of the data to .png format. Plus the input data is not optimal quantized as the histogram shows strong clipping at the borders:


(some typos corrected)

We can see the read noise is reduced in the Canon pulled ISO, however, the banding noise (probably from the sensor/CMOS amplifiers) has not been reduced.

However the results are great. Maybe your subject forces the viewers attention towards the banding because the wood on the left side has a structure that is very similar to the banding period. I had a look at the cardbord (former battery package) in 1:1 and could clearly read the whole text - In contrast to the Canon standard photo, where I could barely read the bigger letters.
I think for efficient banding reduction you have to subtract a darkframe. (The question is, how often one has to take a darkframe to compensate for possible changes of the banding)

[UPDATE:]

To cut a long story short: This subtle amount of digital gain alters the shape of the noise probability distribution and thus alters the standard deviation.

Long version:
As the data is already quantized for any amount of digital gain there will be gaps (at value v_gap in the histogram (as seen in your shown histograms a few pages earlier) because there is no value v_old to fulfill

Round[gain*v_old]==v_gap

In the following I assume an histogram centered at 2024 at unity digital gain.

Depending on the gap position the retrieved standard deviation may rise. This occurs the number of gaps in the histogram increases. The initial histogram has no gaps (because of unity gain). Increasing the gain increases the number of gaps inside the data. But those gaps lie outside the noise distribution. So the will not affect the computed stdev. Nevertheless the white level rises, so the retrieved dynamic range rises.
For your gain ([0xFE] 0 = 11.335 + ADTG [888x] -0.33 EV = 11.679 EV.  Digital gain 540, WL 16007, stdev 4.11) the first gap is produced within the distribution (at initial value of v_old=2023 -> v_gap=2090). Thus the histogram gets wider and the stdev rises. This bigger stdev is now divided by the white level which did only rise one incremental step, resulting in a decreased Dynamic range.

I think this explanation holds true also for initial histograms that already have gaps, because the additional gaps will appear anyway. I will try to add some graphics.


UPDATE: I am sorry, this only affects the standard deviation of the noise probability distribution. Depending on your calculation of the Dynamic range this may not change the dynamic range at all, if the calculation takes into account the median of the noise's probability distribution.
Maybe these finding at least help to figure out the exact amount of digital gain, as the positions of gaps in a full scale histogram are sharp indicators for digital gain. Maybe those "half gaps" seen in some histograms originate from digital smoothing algorithms or correction matrices?

UPDATE2: This is too obvious. I'm sure it is already stated somewhere in those 21 pages 

A little digital gain (530) always gives a little DR increase here.
ISO 400
[0xFE] 0 = 11.335 + ADTG [888x] -0.35 EV = 11.679 + digital gain 530 = 11.732 EV.

I have an idea about this. Could you estimate the gain by comparing both whitelevels?

I see! It indeed is not straightforward to run the new version from source under windows. I doubt I will manage to do it, so I have to wait for the 1.0.5 

Hi baldand,

I've seen a lot of changes in your bitbucket repo since 1.0.4. Do you have any plans to compile a new build in near future?

- removed -
I misunderstood Greg's post.

Hi again,

I am sorry if I am again posting something stupid, but Lenny reminded me of some fact I did read about the 5D III sensor.
There was a company that used an electron microscope to examine the sensor of Canons 5D MK III.
They revealed transistors that could be used for electronic pixel binning of three adjacent image pixels. The furthermore assumed this could be used in video mode to reduce aliasing. Unfortunately I could not find a proof of this speculation. Even worse is the fact that this paper was only accessible for some hours or days. I can not find it any more.

Why do I post it here?
Wouldn't binning of three pixels improve the resulting full well capacity by a factor of three? That COULD significantly improve DR in video mode.
On the other hand it would also increase the shot noise compared to one single channel. So if three binned pixels are read with the same settings (Gain & ADC) as one single this would result in WORSE DR compared to photo mode. (Which is what you seemed to see in the beginning?) If one could now somehow adjust the settings to respect the greater full well capacity it could finally increase the video DR, wouldn't it?

Again: I assume that somebody has already investigated / tested this and did not publish his results because it did not work or the information about those transistors is wrong. But since there is a tiny chance that this has not yet been tested I thought I should give it a try.

Great!
Thanks for clarification :-)

Hi everybody,

I am following this thread from the beginning, being excited about the possible improvements of your amazing find.
Is my following short conclusion correct?

The DR improvement on 5D MKIII is confirmed and the registers are identified.
Nevertheless there is no tool that could automatically tweak all those registers for a specific camera (without the possibility of damage). This tweaking may be required because every sensor is different.

Furthermore some RAW-converters will not benefit from the improvements because their whitelevel is fixed. But ACR seems to work.

In case there already is a module for 5DMKIII that can be used for testing and does not involve setting all those registers manually, please tell me where to find it :-) I would love to help testing.

Have I missed something here or does this feature not work in 5Dm2?
For any of my lenses (restored camera settings, new ml 8th of feb) - the dot-tune module iterates for eternity. It does not seem to change anything between the beeps and results always ends up at 0 (no change).

What have I missed? (RTFM - is there a manual somewhere?)

Regards
Joakim

To identify possible mistakes you should describe your setup completly.

What Lens, what target distance, what target?
Focussed in Liveview? Lens set to M afterwards? Doublechecked focus in LiveView?
Does it beep everytime for every AFMA-setting?

You can gain one stop of light by overriding the shutter speed to 1/25sec. (But be aware of some motion-blur)
Try to expose such that the highlights are just below clipping (RAW-Histogram).
I tend to use ETTR on the brightest Part of my Scene to optimize exposure before each shot.

For Noise-Reduction NeatVideo works great. It uses also temporal filtering so you do not loose as much detail as by just the ACR Noise reduction. But the differences are not huge.

Thanks for the quick reply.

mlrawviewer is running as 2 threads. Both close when I hit ESC.
I am currently downloading the newest driver ...was really surprised that my current one is more than one year old. I will update this post with my results.

UPDATE:
- newest ATI driver installed: Still a pink frame and still the black window.
- I noticed that the black window does not appear when I drop the same MLV file on mlrawviewer again. It only happens with different files. Btw I have version 1.0.4 _alpha32 from the .zip as I can not build it on my own from the repository.

Hi everybody,

first of all: Great tool!! I never even dreamed of being able to preview MLV on my old Thinkpad T400 in realtime.

unfortunately I encountered a little but annoying problem:
This happens after closing the viewer (clicking the red cross on Windows or using ESC-key). When I afterwards drop another MLV onto the .exe the window appears, but is black. RAM usage increases and after about 10sec mlrawviewer closes.
If I drop the file again, it will be played without problems.

I noticed when I double click the .exe between each MLV (of course producing an error in the log) the next MLV I drop will be played without issues.
Surprisingly the black window problem does not create an entry inside the error log.
This happens on an ATI MobilityRadeon Maybe the OpenCL / OpenGL is not shut down cleanly??

Another very little issue: I notice pink bars of about 10 pixels at the right and bottom edge of the footage during playback. I haven't seen them on my desktop PC with NVIDIA GPU ( but I will double check).

Hi,

i recorded a lot of MLV footage. Now I am going to update my current PC to cope with those huge data rates and the preprocessing that hast to be done.

My question (especially for your converter): Will I benefit from a powerful graphics card like Geforce GTX 770 or is it just the CPU that matters?
I think at least ACR can use the GPU, can it?

I suppose a SSD for the footag is a must? Or will a 7200rpm be good enough? (max. 1 video + transitions in PP CS5 maybe PP CC later...)

Hm, can you give a specific example?

I realize the cuts are in sync (although they are not always on the same beat)
It is the action that does not match the beat.

1:17 could be an opportunity to sync the jumping and the beat?

Maybe you sometimes (when they hold each other on the ground?) could slow down the music to add some tension?
I'm not very experienced in this topic of films.
Have a look at some Kung-Fu or Boxing films and concentrate on how the music syncs to the fight and the sound effects add tension / thrill.
Depending on the message you want to transport real sounds or Foley can also be useful (pain? action? The sound of the hands in the end?) If there are specific sounds, the viewer will search for  their origin and concentrate on this element. On the other hand your soundtrack is already rather dense, so there is not much room left without thinning it out. So I have no real advice here.

Hi

Im using the Feb 03 build with the sound module added, but i cant see my raw histogram anymore. Any ideas why? Its def switch on in the ML menu...

Also im getting skipped frames and it only sometimes lets me get continuous at 1920 x 1080 which never used to happen with RAW

E

5DIII?

Which image sizes do you use?
Switching to RAW only and automatic card swap increased my buffer from 22 frames to about 60 frames 1080p. Since then MLV works like a charm.
(Before I had RAW (CF) + JPEG S (SD) or JPEG S (CF) + JPEG S (SD) which I was used to from my 7D)

Hi Brasilius,

having read your comment I wanted to figure out why nobody replied, so I watched the video rather critical.
I am very sorry to write some negative feedback. Take it as suggestions. I can not do better or want to bash you. It is yust what came to my mind:

The soundtrack makes massive use of phasing / chorus effects. In the beginning the width of the sound stage jumps from a narrow phasing to a huge chapel with loads of bass when the acoustic guitar plays. This irritated my ears so I had to stop the video.
This might be because i use KEF LS50 in near field configuration which are very phase accurate. But there is high risk that this effect may be similar on other loudspeakers.
I checked with headphones. Here the effect is much more subtle because each ear gets it's dedicated signal.
Later when the bass drum comes in it is easier to listen because it defines some constant sound stage.

The footage is of good quality!
Unfortunately the music does not support all your / cuts. I also have the feeling your video lacks a "red line" or story for the viewer to follow.
Maybe you could introduce one or two main characters in the beginning. They could then practice hard (with all the others of course) to have a fight in the end. After that they shake hands and show they are friends. (just a very simple suggestion).
Maybe you intended to introduce the guy that sits in the train, but somehow I loose him during the clip.

My suggestions are now open for discussion. Maybe somebody else has different opinions?

[With 44.1kHz audio sampling rate MLV is contineous for me, while with 48kHz it skips after about 100-500frames.]

The biggest issue I have is it not being able to run 1920x1080 @ 25fps.  Any advice on settings that people recommend for getting the best bang for the buck on the 5d 3?
Hope the above info is useful to someone?

I struggled with se same problem. Back then i was able to record contineously and suddenly only one in a while a cont. recording worked.
Yesterday I realized that I changed the audio sample rate to 48 kHz recently. I switched back to 44.1kHz and viola:

With 44.1kHz audio sampling rate MLV is contineous for me, while with 48kHz it skips after about 100-500frames.
Build: Jan-11 5DIII 128GB KB

I made every tests with CF only buffers=0 and buffer fill method=0 so both OFF. Card spanning also OFF.

skips again at 4095MB! buffers: 0 fill: 0, spanning OFF (Global Draw does not matter)