Recording a 'border' to allow for digital image stabilization in post?

[saf34]

Hi, so, I have an idea and wanted to ask anyone who has already got their hands dirty with coding if it would be feasible.

The idea: Alex's 1x1 1080p 3x crop mode with correct preview, but have it record an extra 5% (say) of pixels from the sensor.  Why?  Then you could use stabilization in post without needing to zoom.  It would effectively be like digital stabilization in camera.

This might not be of interest to everyone, but it strikes me there are really two ends of the spectrum on how to use ML.  (1) seeking highest resolution in some way, and willing to compromise on preview, continuous recording, overexposed areas. (2) Using it as a Super 16 HD or FF camera with 14-bits and seeing what extra features can be squeezed out.

Personally, I use it almost all the time as (2) and it is amazing.  Am curious if the stabilization 'border' would be possible though!

What do you guys think?
Stewart

[Bruno Italiano]

Hi,
I use 5D3 with 5k in 14bit lossless. With stabilisation DJI Ronin MX. Sigma Art 20mm I don't edit any digital stabilisation. I don't know what else Magic Lantern could offer. Don't forget all of the developers didn't get any money 😇

[saf34]

Hi Bruno, please don't misunderstand: I'm asking as it is a feature I would consider *contributing* to if it were feasible.  I don't take for granted the ML community's work, and I'm not asking anyone to do this for me.  Just if there is a limitation based on the data streams (no point me going down a dead-end path).

Yes, using a gimbal is great, but there is a reason digital IS exists in recent cameras too.

Stewart

[theBilalFakhouri]

Hello, cool idea

-Note: 1080p 1:1 (x3 crop) on 5D3 uses default 1080p preview configuration, in this mode we are not touching the preview, just the sensor Binning mode we are using 1x1 instead of 3x3.

It's possible to increase RAW resolution but increasing RAW resolution (in particular width resolution) will break the preview. We can fix broken LiveView but this only works in x5 mode. Attempting to fix preview in 1080p mode will freeze LiveView (I did try it on 700D, I am assuming 5D3 would have same result, I can make a quick try to confirm that later).

That means we need to reverse engineer LiveView and understand how preview works on 5D3 just to achieve this simple goal; I mean a lot of time and effort are required .
We might not be able to achieve this goal anytime soon (or maybe *at all). I am working on 5D3 preview from time to time, but there is no real progress, can't promise anything. I often have to stop working for months.

On 700D we can already control preview in x5 mode, and I could made 1920x1280p 1:1 preset with correct preview (instead of default movie crop mode 1800x1012). Well, I still can't control preview in 1080p on 700D, attempting to play with preview configuration in 1080p mode freezes LiveView.

I can give some info about how to start reverse engineering preview on 5D3, a lot of work/time are required, let me know if you are interested

*Simply there are no active developers who are working on these old models (DIGIC 4/5), and in general the interest now is to port ML to new models (DIGIC 6/7/8).

[saf34]

Thanks theBilalFakhouri, got it.  I had my fingers crossed it was simply that no-one had ever thought the standard preview was useful while accessing more of the sensor - but your explanation clears that up perfectly.  Not a straightforward thing to do then!

Ultimately then it might be that there's not a huge amount of benefit over putting an overlay guide that blacks out 5% of the top, bottom and sides of the standard preview for framing and upscaling that 5% in post.    Or using an existing higher resolution with compromised preview but again with the overlay, and downscaling.

I'm definitely up for at least trying some reverse engineering of the liveview though.  Any starting pointers very welcome!  And goes without saying: all your previous work is super-appreciated.
 

[Skinny]

It's an interesting idea, but I don't really understand why do you need that 5%. Just compose your shots keeping in mind that stabilize will crop just a little... is it really making any difference? and 5% is a lot, usually when I use warp stabilizer if it goes more than 101-102% zoom then the footage is probably too shaky to be used, you will see rolling shutter artefacts and motion blur where it shouldn't be... and significant quality loss because of zooming in.
Anyway sometimes after stabilizing it requires only 100.5% zoom but sometimes as high as 102-103% and you can not really predict it. So what exact margin do you need then, if it's different all the time?

And... when you really inspect your footage you will find that most camera shakes happens is in vertical direction. So (in theory) you can increase your vertical resolution, for example to 1920x1200 when you need only 1080. Then after stabilization turn off automatic zoom and crop it manually but only watching when left and right sides looks ok, and up and down sides will be with black borders flying around, just crop it to 1080. This should significantly reduce zooming-in, maybe at least two times.
I never tried it though because Premiere doesn't like when you use clips with different aspect ratios, so you need to nest everything and I'm too lazy honestly

But the more I use digital stabilisation, the more I want to buy a steadicam. I just always see rolling shutter or motion blur artefacts. And overall less quality image after digital crop-zoom.

[saf34]

Thanks Skinny, interesting to hear your take on it - and super useful point on using vertical resolution to reduce the necessary crop in post.

To answer the question: the 5% was an arbitrary number that was 'enough' really, rather than how much I'd want to zoom.  To explain: the idea came from the question "suppose we regard the standard 1080p preview as the only 'usable' monitoring experience for a particular use case.  Then, what extra features, however small, can be squeezed out".  I have a series of films that are all handheld, incognito and require good on camera monitoring.  And so I wondered about whether it was possible to have the standard preview while still filming higher resolutions outside of the preview. 

Regarding stabilization, it was really all just about eliminating the need for zoom.  If using a translation-based stabilizer in post - as I often do - then the zoom is the only aspect that 'reduces' the resolution below 1080p.  Perhaps it's a really negligible point - it probably is - but I was curious nonetheless, as I always work in post with the zoom disabled and watch the 'dance' of the black bars.  The other practical point is that I monitor from the LCD screen when using 1080p (it's always to film agile moments) and so framing with stabilization in mind means a smaller image.  Which again, is okay, but I wondered.

Finally, there's something about 1080p which feels like a lower zoom bound these days.   For context, I have a RED Komodo, two 5Ds with ML and a couple of DJI gimbals.  It's probably just psychological, I admit, but there's something about knowing these 5Ds have the capacity to do 3.5K but having to zoom to less than 1080p for handheld stabilization that would be fun to get around.  Yes I can stick the 5D on a gimbal, but it changes the experience in a variety of ways, not least the incognito way of filming you lose. 

In retrospect, if this were easy to do then I guess someone here would have already set up a super 16 preview while filming 3.3K or similar.  Still... just a thought!

[theBilalFakhouri]

Sorry for the delay, my notes has been posted here.

TL;DR: A lot of time and effort would be required, but I think it's possible.