Pre-recording for the raw_rec module?

[bouncyball]

Haha fantastic!

mlv_lite's getting better and better. Only thing I'm missing there is optional audio recording .

Since I always use back button focusing (af-on, not half shutter) on my 5d3, I noticed that pressing af-on also triggers recording. I want to focus by af-on while camera in rec hold mode and start recording only after I push and hold the shutter button. Is that possible?

bb

[Danne]

Working nicely on the eos m too . Of course audio included would be a killer.
Shameless question about possibilities to change settings like iso and shutter while in pause mode. Is it possible?
Exciting times.

[a1ex]

Disable small hacks and Canon's user interface functionality should be restored (at the price of a minor speed drop). Maybe it should be disabled by default, since recording speed is no longer critical with 10- and 12-bit recording.

Changing ISO and shutter works on 5D3 without disabling the hacks (even while recording), but has some UI quirks.

[Danne]

Actually disabling Use SRM memory got rid of the BUSY state signal while hold and then the settings could be change with or without global draw on. Real nice! I left the small hack on. 

[a1ex]

Hm, I have SRM memory enabled and I don't have any BUSY screen. Tried both global draw on and off, AF on and off. I'm currently on 1.2.3.

[Danne]

5D mark III 1.1.3
Eosm202
Tried again and also whith the compilations from the experimentals section as opposed to my own compiling builds but still BUSY state once rec is on and while in hold state. When ending recording BUSY stays for a second or so before dissapearing from the screen. I thought it was normal behaviour. BUSY state is always on while recording regular raw and after stopping it leaves after a second or so. Same for mlv_lite and mlv_rec. I could check the behaviour on a non experimental build and report back.
The BUSY state signal works the same on eosm, just tested.

*update
BUSY state behaviour on nighly_builds/experimental_builds/mlv_lite/mlv_rec/rec_trigger and so on, related to SRM memory.

[bouncyball]

@a1ex: I want to repeat my question. Is it possible to focus while in pause mode with AF-ON button without triggering recording. I always use back button to focus instead of half press shutter.

@Danne: I also do not see BUSY message with or without SRM. (fw. 1.1.3)

[a1ex]

Telling the difference between AF and half-shutter is a bit tricky. The silent picture module uses a heuristic - if in AF mode, it waits for a few hundreds of ms to see whether the lens starts autofocusing. If yes, the half-shutter press is ignored.

Logging the MPU messages might give some more clues.

So, it's possible to implement it, but the trigger will be delayed and will require a long half-shutter press (unless another way to tell the difference between the two buttons is found).

[bouncyball]

@a1ex: Understood. Thank you for the explanation.

@Danne: Actually I was wrong about BUSY on screen message. It just half hidden under overlay but is there all the time while recording if SRM is on.

Also "Use SRM memory" menu item help/hint tells us:"Side effect: will show BUSY on the screen and might affect other functions"

[Danne]

Actually I was wrong about BUSY on screen message.

Darn, here I thought I was having something special going on with my 5D 
I guess the busy state is normal then and will interfere with setting changes working rec_trigger also with 1.2.3?

[mk11174]

Hey, not around much anymore, BUT, I really would like to say I am loving the way this feature progressed! I find the 1 frame with pre record for 2 seconds very useful, VERY COOL! Other options work just fine as well! Glad Alex kept going on this one, love how it works with my DIY half shutter press button for my 700D. I will surely test this during storm season, should work fine, already tested in flash, and it captures them all just as good as Bolt Trigger module does. But this will be perfect as well for capturing nature events for sure or any kind of action that is hard to capture without record constantly, LOVE IT!!!

[Kharak]

Hey, I am not much around either these days.

Could I suggest you rename this thread to 'Pre-Recording for Raw_Rec Module'? Its confusing to search for this on the forum when the thread does not correspond with the name of the feature.

So I downloaded latest nightly build yesterday, very eager and surprised by this new feature. Very very happy with it.

Though I had troubles getting any Module to Link as it said. I suppose it was because I had the Crop_rec module on there aswell, not sure? Deleted everything of the SD card and then it worked, used to be I just copy paste and overwrite whatever was on. Is Crop_rec Module not compatible with the latest nightly ? I mean for MLV_Rec aswell ? I don't remember if Crop_rec was only Experimental. Crop_Rec modules is also a Killer feature that I really enjoy working with. anyways, back on topic.

Could it be possible to adjust the desired amount of Pre-Recording by scrolling the shutter wheel in increments of 8-16 frames or so or just 1 second at a time? And that it showed more precisely how much is pre-recorded, preferably in X.X secs or even Frames? I had an issue today when I set it at 5 seconds, trying to film waves breaking over this ship I am shooting on. First of, it counted 3 seconds and when I hit record again a message popped up everytime saying something like "Buffer can not reach desired amount" and it would cancel the recording, so had to set it to 2 Secs. I don't remember exactly the message because when I was done shooting and trying to replicate the problem for this post, it worked with the 5 Seconds e.g. Counting to 3 secs and hitting record worked.. So I don't know what was going on there today. All settings were the same when I tried to replicate.

Which brings me back to why it would be nice to have clearer information on how much I am Pre-Recording. When it says "3 Secs" pre-recorded? Is that precisely 3 secs e.g. 3x 24 Frames or is it 3 secs and some more frames until it reaches 4x24 frames and it says 4 Secs ? I assume the buffer is not always holding the exact same amount of frames depending on background processes and other settings on camera?

Oh! and one more question, will this feature be seen in MLV_Rec in the future? for sound and metadata recording ?

Anyways, thank you so much for you're continued efforts with these cameras. My MK III still feels on par with anything else coming out these days.

Latest Nightly: March 12 -  2017
1920x1080 14 bit
MK III

Kharak.

EDIT: What steps could I take to increase the amount of Buffering in 14 bit ?

[a1ex]

You might be able to use it with crop_rec if you copy just mlv_lite.mo from the nightly, and you keep the old files from the crop_rec build. Didn't try, but if it doesn't work, I can sync the branches. The crop_rec branch is experimental because of the backend (last time I took it outside, i.e. many months ago, I had ERR70).

The error message was probably just the buffer getting full. When you select a large amount, it reserves 16MB for future buffering; maybe I should reserve some more. At such small buffer sizes, it can quickly run out of steam if the card write speed isn't known very well (it's measured on the fly), or if unexpected speed drops occur (they do). Though I don't remember noticing this message when testing (maybe I didn't push the settings far enough).

The display is currently rounded. I agree a more exact display could be useful, especially on cameras with less memory.

The buffer size will become less of an issue with reduced bit depths. I'm thinking to merge the current state into nightly and just hide the reduced bit depth menu on models with known issues, as it seems to work pretty well on recent models (except in-camera playback, which require raw_twk).

[a1ex]

I think I've got a way to limit the max memory available for pre-recording without causing the regular recording to run out of steam.

How it works? The amount of memory reserved for recording (that is, not available for pre-recording) was fixed to 16MB before, and now it depends on both resolution and write speed. I expect recording will continue after pre-recording, so I reserve a buffer large enough to allow recording 500 frames with 90% of the measured write speed, but not more than half of the RAM and not less than 10 frames. That should give continuous recording, with buffer usage peaking somewhere near maximum, and then recovering.

That also means the pre-recording buffer size drops quickly as your required write speed approaches your card write speed.

There is an exception: if the trigger is set to "Half-shutter: 1 frame", there's little need for buffering after pre-recording. In this case, the pre-recording buffer is maxed out (using nearly all the available RAM).

The amount of memory available for pre-recording can now be seen in menu, if your selection exceeds the limit. Caveat: you need to record a test clip first.

Experimental build available; please report.

[Kharak]

Reporting in:

Tested your new experimental mlv_lite with latest nightly and it worked on the first try. I set it at 5 seconds and it buffered to 3 secs and I hit record again and it worked.

I then did as you suggested, added this latest mlv_lite to the latest crop_rec build (I was apparently running an older version) and it too worked flawlessly. Though on very first try, it buffered to 2 secs and then on second try and every shot after that it was back to 3 secs.

And nice with the finer increments by the way, I used 5 Secs on theses tests because thats what failed on my shoot earlier this day. Maybe it failed last time because I just switched from mlv_rec to mlv_lite and the buffer somehow gets "confused" or some code stuff happens there. Just thought it might be a detail worth mentioning for the record, though it seems its resolved now anyways.

Thank you for your quick response and fix.

I have just seen that there is a Proxy recording module or build. How would that fair with Pre-Recording? I would personally not mind if the H264 file is not Pre-Recorded if it had to be so, but could this also somehow be conjoined? I have not tested it on a stand-alone basis or even read through the entire thread, so not sure how stable it is.

EDIT:

MK III
Lexar 1066x 128GB
14 bit

[mk11174]

Working nicely here, love the grayed out amount of how many seconds you can get when selecting a to high of a pre-buffer, very nice addon!

Man, if this had motion detection and luminance detection this would be the ultimate module, not that it isn't already, but damn, this is working very nice for those addons to be of use.

[janvkem]

Installed magiclantern-crop3x.2017Mar21.5D3113 and all seems to work (only checked briefly). Thanks a lot for combining it with the 10 bit recording! I really like the pre rec option!

[vstrglv]

I also installed magiclantern-crop3x.2017Mar21.5D3113. It works fine. Many thanks for compiling.
I have some questions.
1. If i load mvl_rec.mo and mvl_lite.mo and activate mvl_rec there is no sound recording.
2.If I load mvl_lite.mo and raw_rec.mo there is no difference which of them to activate. Both work as mlv_lite.

[g3gg0]

@a1ex

The assumption that size differs only by 48 bytes turned out to be wrong. Difference can vary. Here are the sample logs from various MLVs.

When using crop_rec there are no NULL blocks in MLV at all, Frame size also varies. ML version used to record these crops: magiclantern-crop3x.2017Jan13.5D3113.zip.

This somehow OT here b/c all of them recorded by mlv_rec (audio is on) not raw_rec/mlv_lite (to be inspected yet).

Here is excerpt from ordinary non crop MLV log recorded back in 2014:

what you might be seeing is the feature of mlv_rec to embed periodic/onevent blocks into the "DMA-alignment-padding" area of VIDF blocks.
this area would else have been just waste. so i decided it to place some low rate data in it - like RTCI and stuff.

iirc this is done by modifying frameSpace. (today i would simply use NULL blocks for that)

[bouncyball]

@g3gg0

Thank you for the explanation. I suspected it was caused by some alignment-padding.

Just one more thing: why crop_rec MLVs do not have any NULL blocks?

[a1ex]

There is no such thing as crop_rec MLVs, as crop_rec simply reconfigures the video mode. The MLVs are still made by either mlv_rec or mlv_lite.

Since crop_rec does not (yet?) alter the resolution of the original buffer (it simply changes the sensor scanning parameters), I don't expect it causing any changes in the file structure.

If you are talking about mlv_lite, NULL blocks are used either after the main headers (to make sure their size is multiple of 512, for write speed reasons), or when the 4GB limit is auto-detected the hard way (after a write failure), to discard the incomplete frame that might have been written.

Frame padding (to ensure EDMAC alignment and block size multiple of 512) is done the old-school way (frameSpace and blockSize slightly larger than actual payload, so there may be unused data both before and after the frame - whose size is xRes * yRes * bpp/8). I agree using NULL blocks would have been nicer (just needs extra care if the skip amounts are smaller than the size of a NULL block).

[bouncyball]

There is no such thing as crop_rec MLVs, as crop_rec simply reconfigures the video mode. The MLVs are still made by either mlv_rec or mlv_lite.

Yes I know sorry about incorrectly formulated question. I meant "MLVs written while using crop_rec.mo".

If you are talking about mlv_lite, NULL blocks are used either after the main headers (to make sure their size is multiple of 512, for write speed reasons)

Not mlv_lite, I was asking g3gg0 about mlv_rec. Usually MLVs recorded by mlv_rec have lot of NULLs in it. But when used in conjunction with crop_rec, NULLs are simply not there. Tried with lot of MLVs with same result.

And again sorry about OT in this thread It just g3gg0's answer triggered that question.
Mlv lite MLVs have always simple and straight structure (I know about EDMAC alignment and block size multiple of 512 for speed)

Thanks for info
bb

[g3gg0]

right. it was due to the fact back then we were just happy to get it running with the level how things were understood.
in such cases sometimes you don't see clearly

today there is a lot better understanding of camera internals like EDMAC and stuff.
so if the raw file format for ML would be defined today, it would have some smaller things done different.

in general i think it is extensible enough to satisfy all (current) needs, correct me if i am wrong

[g3gg0]

Mlv lite MLVs have always simple and straight structure (I know about EDMAC alignment and block size multiple of 512 for speed)

yep, thats why (at work) i always say that things have to get implemented twice.
first to get from the idea to some concept that works and is usable.

and second some other person going over it, understand the initial goals, the solutions and
implement the same thing again with the experience gained from the first iteration.

but yeah, OT a bit


edit: especially with mlv_rec i really didn't know what will happen. some niche thing probably?
so some things were quite experimental but scalable. obviously sacrificing performance a few per cent too much.

[a1ex]

Updated to address some comments from dfort (see the PR page):

I started out setting the Pre-record for 1 second and the Reg trigger to Half-shut:1 frame. Every half-shutter recorded a second instead of just 1 frame.

Ok--don't really need pre-record for this so I turned that off and now I'm getting 2 frames per half-shutter press. Not sure if I'm doing something wrong but it looks like all the settings are telling me that it should record only 1 frame.

Also added fractional time display while recording.