Magic Lantern Forum

Developing Magic Lantern => General Development => Topic started by: mk11174 on April 11, 2016, 03:10:19 PM

Title: Pre-recording for the raw_rec module?
Post by: mk11174 on April 11, 2016, 03:10:19 PM
I was trying to film dogs catching Frizbee the other day, and was thinking, I wonder if there's a way to add a buffer record to Raw_Rec module, like have it record to buffer but not to file for the certain amount of frames the user chooses as a buffer, then when the action you want happens, you press the shutter and it records the buffer to video file? Might be a neat option to add to this module for wildlife or sports, especially if it will probably now save the camera settings you use in the video file as well. Or is this to much clutter for the module, maybe a separate module might be needed? I have been away from ML community for awhile and my coding skills are way to rusty to even know if this is possible or not. Thanks
Title: Re: Pre-recording for the raw_rec module?
Post by: a1ex on April 11, 2016, 03:43:13 PM
Pre-recording - indeed, would be a very nice feature. And it would provide a solid backend for your older experiments like the lightning trigger.

Currently, you have something similar in the silent module (the "end trigger" option)
- hold the shutter halfway
- when the action happens, release it
- it records 1-2 seconds of video (MLV/DNG) right before the shutter release

Problem: you are limited by how many frames fit in the memory. I have it updated locally to use both SRM and shoot memory buffers (part of an unrelated experiment), so I'll commit that for a quick workaround if it helps.

In any case, pre-recording shouldn't be very hard to implement in raw_rec. Basically it would be something like this:
- before pressing the button:
   - don't enqueue frames to the writing queue
   - after say half of the available memory gets used, discard the oldest frame and mark that slot as free

- after you press the button:
  - send all the frames to the writing queue (it will start saving frames at full speed)
  - keep recording normally

(I'm even tempted to try it, since I wanted to implement that for a long time)
Title: Re: Pre-recording for the raw_rec module?
Post by: nikfreak on April 11, 2016, 04:11:58 PM
Sounds like Panasonic's 4K Photo feature which they btw extended with a so called "post focus" feature:



Title: Re: Pre-recording for the raw_rec module?
Post by: mk11174 on April 11, 2016, 05:00:37 PM
Oh yes, the frames limitation, I remember that, and every camera model would be different from what I remember.

And yeah, might help out with lightning trigger if I had a way to remote trigger, lol, good thing about lightning one, is I can hide in car when big bolts get close and the detection code will do the unsafe triggering for me.  :P

But with all these cell phone apps having these type of features, this addition to the DSLR camera would be great since there quality is way way better then a cell phone sensor.

Hopefully the frame limitation does not keep it from progressing if you decide to try to implement it, but I do understand that part.

And yes, very similar to End Trigger, only bad thing with End Trigger feature is you have to wait it out as each frame writes and goes through the previews and it doesn't include the camera settings in the file if I remember correctly, its been awhile since I was in testing mode, maybe I missed an update?

Thanks for the interest and reply!  :)
Title: Re: Pre-recording for the raw_rec module?
Post by: a1ex on April 11, 2016, 05:22:10 PM
Metadata is saved if you use MLV as file format, and previews aren't any slower than the ones from mlv_play.

Updated the silent picture module with SRM buffers, so it now uses all the available RAM.
Title: Re: Pre-recording for the raw_rec module?
Post by: mk11174 on April 11, 2016, 06:04:13 PM
Cool, sounds interesting, I will check it out and see how it works from how I remember it working or compare to old end trigger before I copy new one over, just need to hope my VBOX still works and I remember how to do all the stuff I use to mess with to compile.  :-[

As for preview, I would prefer no preview at all, just record buffer and wait for next shot, but wondering if it needs time to write anyway before you can record next shot, so maybe the preview is a must to slow the user down?

Am anxious to see if I get more frames with your new update though, Thanks!

Update: On 700D for silent pictures End Trigger, it went from 25 frames in buffer to 41! Nice update!
Title: Re: Pre-recording for the raw_rec module?
Post by: a1ex on April 11, 2016, 11:53:46 PM
Looks I've got the pre-recording (https://bitbucket.org/hudson/magic-lantern/commits/529f0b4926fa) mode working as well in raw_rec.

Bonus points: it merges cleanly with mlv_lite as well :D
Title: Re: Pre-recording for the raw_rec module?
Post by: mk11174 on April 12, 2016, 02:21:24 AM
You got to be kidding me!!!! Very cool, will have to try this out for sure! Thanks!  :)

Update: pre-record works nicely, I like how it shows the amount of seconds capable of pre record, so if u pick 10 seconds,  it will show you while recording if its possible,  and if not, it will show how much you can get for the fps and resolution camera is set for.

I see its pre record time amount doubles pretty much with srm on, then if turned off, so that's obviously working good too.

At first I was confused cause when I started recording, I waited a bit, saw up top the pre record amount in seconds,  then I pressed shutter to save, but it started writing, then kept recording until I pressed shutter. I was thinking it was going to stop capturing and save all frames from pre buffer up to where I pressed shutter, eventually I figured out how u set it up and got use to it.

Pretty cool you got it coded so quickly.

I was using 700d for testing by the way.
Title: Re: Pre-recording for the raw_rec module?
Post by: a1ex on April 12, 2016, 04:06:03 PM
Quote from: mk11174 on April 12, 2016, 02:21:24 AM
At first I was confused cause when I started recording, I waited a bit, saw up top the pre record amount in seconds,  then I pressed shutter to save, but it started writing, then kept recording until I pressed shutter. I was thinking it was going to stop capturing and save all frames from pre buffer up to where I pressed shutter, eventually I figured out how u set it up and got use to it.

Yeah, one of the harder steps was finding a help text that would fit in two lines, about 70 chars each. Can you suggest a better one?
Title: Re: Pre-recording for the raw_rec module?
Post by: mk11174 on April 12, 2016, 04:24:50 PM
No, you did fine on help file, I was to anxious to try it, I didn't even know there was help text, lol.  :-[

Yours makes perfect sense though.
Title: Re: Pre-recording for the raw_rec module?
Post by: a1ex on May 06, 2016, 03:26:33 PM
Shameless bump:

https://bitbucket.org/hudson/magic-lantern/pull-requests/728/pre-recording-feature-raw_rec/diff
Title: Re: Pre-recording for the raw_rec module?
Post by: Lars Steenhoff on July 21, 2016, 05:32:00 PM
so is it possible to use this ?

is it part of a nightly build?

Thanks
Title: Re: Pre-recording for the raw_rec module?
Post by: Walter Schulz on July 21, 2016, 07:34:44 PM
As long as a pull request's status is "open" it is not part of "main" branch (-> "unified") and main branch will be compiled to "official" nightly builds.
Therefore: Yes, you can use it if you compile it for yourself and no, it is not part of nightly build.
Title: Re: Pre-recording for the raw_rec module?
Post by: a1ex on February 11, 2017, 06:31:45 PM
Added to experimental builds (http://builds.magiclantern.fm/experiments.html).
Title: Re: Pre-recording for the raw_rec module?
Post by: Lars Steenhoff on February 11, 2017, 07:39:50 PM
Thanks for the update!

I just tested and works well on my canon 5d mk3
Super cool feature to have.

:)
Title: Re: Pre-recording for the raw_rec module?
Post by: janvkem on February 13, 2017, 08:41:26 PM
Great feature! Excellent for unpredictable action. Am I impolite if I'd ask for a 10-bit capable version?  :)
Title: Re: Pre-recording for the raw_rec module?
Post by: RenatoPhoto on February 15, 2017, 02:22:15 PM
Oh yea! Yet another magic bullet to add to my arsenal.. I will have to test this soon..
Thanks A1ex mk11174!
Title: Re: Pre-recording for the raw_rec module?
Post by: Frank7D on February 17, 2017, 04:43:23 PM
Tested on 7D.

~ Used 2 sec delay
~ Video was 136 frames (0-135) but was reported as 873 frames by mlv_play (and by MlRawViewer). Is it possible it is counting the time from the first button push?
~ Last frame (135) has a pink band running across the very bottom of the image
~ The two anomalies do not make the feature unusable.

This is a wonderful feature. If it could be added to mlv with 10/12 bit recording it would be even better.
Title: Re: Pre-recording for the raw_rec module?
Post by: Frank7D on February 18, 2017, 05:43:45 PM
Here's the 136 frame raw file that is seen as 873 frames:
https://www.dropbox.com/s/rx1gnyd04m3serk/M17-1121.MLV?dl=0 (https://www.dropbox.com/s/rx1gnyd04m3serk/M17-1121.MLV?dl=0)
Title: Re: Pre-recording for the raw_rec module?
Post by: bouncyball on February 28, 2017, 04:20:45 PM
Hehe nice one :). Actually I have lot of MLV files (older/newer ones, recorded by mlv_rec not raw_rec) where real frame number is less than reported from the header(s). Differences like 2-3 frames are common. With multichunk MLVs even more (7-25 that I've seen myself). The files are all normal - no bad/corrupted frames nothing special but frame count is off. So I'm not sure about frame count until I index the MLV. However 737 frames difference is really strange ;).

Little bit OT... I also experienced the behavior when VIDF block size differs by 48 bytes from all other VIDF blocks. This always happening when VIDF is exactly before of after RTCI  (RTCI block exactly 48 bytes long). I guess it's related to EDMAC async transfers.

bb
Title: Re: Pre-recording for the raw_rec module?
Post by: a1ex on March 11, 2017, 07:51:39 AM
Quote from: Frank7D on February 17, 2017, 04:43:23 PM
~ Video was 136 frames (0-135) but was reported as 873 frames by mlv_play (and by MlRawViewer). Is it possible it is counting the time from the first button push?

Thanks, fixed. The bug was present in the regular mlv_lite as well, but the difference wasn't that big (and went unnoticed since mlv_dump, and probably most other converters, only use the header counter for info purposes).

Quote from: Frank7D on February 17, 2017, 04:43:23 PM
~ Last frame (135) has a pink band running across the very bottom of the image

Could not see it with mlv_dump; have a screenshot after conversion?

Quote from: bouncyball on February 28, 2017, 04:20:45 PM
Little bit OT... I also experienced the behavior when VIDF block size differs by 48 bytes from all other VIDF blocks. This always happening when VIDF is exactly before of after RTCI  (RTCI block exactly 48 bytes long). I guess it's related to EDMAC async transfers.

Not sure how to reproduce this one. On the attached example, I checked with:

mlv_dump M17-1121.MLV -v | grep VIDF -A 2 | grep Size | uniq

Result:

    Size: 3629056

Title: Re: Pre-recording for the raw_rec module?
Post by: bouncyball on March 11, 2017, 11:31:40 AM
@a1ex

The assumption that size differs only by 48 bytes turned out to be wrong. Difference can vary. Here are the sample logs (http://nic.caucasus.net/ml/) from various MLVs.

When using crop_rec there are no NULL blocks in MLV at all, Frame size also varies. ML version used to record these crops: magiclantern-crop3x.2017Jan13.5D3113.zip.

This somehow OT here b/c all of them recorded by mlv_rec (audio is on) not raw_rec/mlv_lite (to be inspected yet).

Here is excerpt from ordinary non crop MLV log recorded back in 2014:

Block: VIDF
  Offset: 0x0a628c60
    Size: 3629824
    Time: 2416.261000 ms
   Frame: #0048
    Crop: 152x132
     Pan: 146x133
   Space: 992
Block: RTCI
  Offset: 0x0a99f060
    Size: 48
    Time: 2436.508000 ms
     Date:        05.03.2017
     Time:        11:37:11 (GMT+0)
     Zone:        ''
     Day of week: 0
     Day of year: 63
     Daylight s.: 0
Block: VIDF
  Offset: 0x0a99f090
    Size: 3629264
    Time: 2458.010000 ms
   Frame: #0049
    Crop: 152x132
     Pan: 146x133
   Space: 432
Block: VIDF
  Offset: 0x0ad15260
    Size: 3629312
    Time: 2499.301000 ms
   Frame: #0050
    Crop: 152x132
     Pan: 146x133
   Space: 480
Block: VIDF
  Offset: 0x0b08b460
    Size: 3629824
    Time: 2541.135000 ms
   Frame: #0051
    Crop: 152x132
     Pan: 146x133
   Space: 992

Regards
bb
Title: Re: Pre-recording for the raw_rec module?
Post by: a1ex on March 11, 2017, 02:30:38 PM
Understood, so it's not related to mlv_lite.

Anyway, thanks for testing feedback. I've pushed a fix and now the frame counter from header matches the actual number of frames from the file. Merged into unified as well, since it seems pretty solid to me.

This development opened the door to another interesting feature - recording triggers using half-shutter. See this PR for details:

https://bitbucket.org/hudson/magic-lantern/pull-requests/819/half-shutter-triggers-for-raw-recording/diff
Title: Re: Pre-recording for the raw_rec module?
Post by: Danne on March 11, 2017, 04:14:39 PM
Wow. This is a major feature. Just tested and it works really good. There is a bunch of scenarios where this will be a game changer, nature filming for instance. Never miss a moment.
For anybody who wants to play with this put the mlv_lite.mo into your ML folder from the experimental section.
https://builds.magiclantern.fm/jenkins/job/rec-trigger/30/artifact/modules/mlv_lite/mlv_lite.mo

1 - In RAW video select Rec trigger and one of the options. Just tested Half-shut: start/pause and Half-shut: hold and both works. Briefly tested.
2 - Hit rec and then press half shutter either holding it down or simply press and release depending what is selected. When done hit rec again to end recording.
Title: Re: Pre-recording for the raw_rec module?
Post by: Danne on March 12, 2017, 08:38:35 AM
I really dig this rec_trigger. Did some more testing and it seems pretty solid already. Lacks 10/12-bit right now. The 1-frame shut is great to work with.
Tested working also with crop_rec module. Ran 1:1 (centred x5 zoom) solid working as well as 3x3 720p (1xwide) all three rec_trigger modes. Also tested with HDR filming enabled. Working.
Title: Re: Pre-recording for the raw_rec module?
Post by: bouncyball on March 12, 2017, 11:54:02 AM
Haha fantastic!

mlv_lite's getting better and better. Only thing I'm missing there is optional audio recording ;).

Since I always use back button focusing (af-on, not half shutter) on my 5d3, I noticed that pressing af-on also triggers recording. I want to focus by af-on while camera in rec hold mode and start recording only after I push and hold the shutter button. Is that possible?

bb
Title: Re: Pre-recording for the raw_rec module?
Post by: Danne on March 12, 2017, 03:52:22 PM
Working nicely on the eos m too :). Of course audio included would be a killer.
Shameless question about possibilities to change settings like iso and shutter while in pause mode. Is it possible?
Exciting times.
Title: Re: Pre-recording for the raw_rec module?
Post by: a1ex on March 12, 2017, 04:58:25 PM
Disable small hacks and Canon's user interface functionality should be restored (at the price of a minor speed drop). Maybe it should be disabled by default, since recording speed is no longer critical with 10- and 12-bit recording.

Changing ISO and shutter works on 5D3 without disabling the hacks (even while recording), but has some UI quirks.
Title: Re: Pre-recording for the raw_rec module?
Post by: Danne on March 12, 2017, 05:14:30 PM
Actually disabling Use SRM memory got rid of the BUSY state signal while hold and then the settings could be change with or without global draw on. Real nice! I left the small hack on. 
Title: Re: Pre-recording for the raw_rec module?
Post by: a1ex on March 12, 2017, 05:18:29 PM
Hm, I have SRM memory enabled and I don't have any BUSY screen. Tried both global draw on and off, AF on and off. I'm currently on 1.2.3.
Title: Re: Pre-recording for the raw_rec module?
Post by: Danne on March 12, 2017, 05:36:24 PM
5D mark III 1.1.3
Eosm202
Tried again and also whith the compilations from the experimentals section as opposed to my own compiling builds but still BUSY state once rec is on and while in hold state. When ending recording BUSY stays for a second or so before dissapearing from the screen. I thought it was normal behaviour. BUSY state is always on while recording regular raw and after stopping it leaves after a second or so. Same for mlv_lite and mlv_rec. I could check the behaviour on a non experimental build and report back.
The BUSY state signal works the same on eosm, just tested.

*update
BUSY state behaviour on nighly_builds/experimental_builds/mlv_lite/mlv_rec/rec_trigger and so on, related to SRM memory.
Title: Re: Pre-recording for the raw_rec module?
Post by: bouncyball on March 12, 2017, 06:40:37 PM
@a1ex: I want to repeat my question (http://www.magiclantern.fm/forum/index.php?topic=17069.msg181259#msg181259). Is it possible to focus while in pause mode with AF-ON button without triggering recording. I always use back button to focus instead of half press shutter.

@Danne: I also do not see BUSY message with or without SRM. (fw. 1.1.3)
Title: Re: Pre-recording for the raw_rec module?
Post by: a1ex on March 12, 2017, 06:54:52 PM
Telling the difference between AF and half-shutter is a bit tricky. The silent picture module uses a heuristic - if in AF mode, it waits for a few hundreds of ms to see whether the lens starts autofocusing. If yes, the half-shutter press is ignored.

Logging the MPU messages (http://www.magiclantern.fm/forum/index.php?topic=17596.0) might give some more clues.

So, it's possible to implement it, but the trigger will be delayed and will require a long half-shutter press (unless another way to tell the difference between the two buttons is found).
Title: Re: Pre-recording for the raw_rec module?
Post by: bouncyball on March 12, 2017, 07:07:40 PM
@a1ex: Understood. Thank you for the explanation.

@Danne: Actually I was wrong about BUSY on screen message. It just half hidden under overlay but is there all the time while recording if SRM is on.

Also "Use SRM memory" menu item help/hint tells us:"Side effect: will show BUSY on the screen and might affect other functions" :P
Title: Re: Pre-recording for the raw_rec module?
Post by: Danne on March 12, 2017, 08:17:17 PM
QuoteActually I was wrong about BUSY on screen message.
Darn, here I thought I was having something special going on with my 5D  :P
I guess the busy state is normal then and will interfere with setting changes working rec_trigger also with 1.2.3?
Title: Re: Pre-recording for the raw_rec module?
Post by: mk11174 on March 12, 2017, 10:11:25 PM
Hey, not around much anymore, BUT, I really would like to say I am loving the way this feature progressed! I find the 1 frame with pre record for 2 seconds very useful, VERY COOL! Other options work just fine as well! Glad Alex kept going on this one, love how it works with my DIY half shutter press button for my 700D. I will surely test this during storm season, should work fine, already tested in flash, and it captures them all just as good as Bolt Trigger module does. But this will be perfect as well for capturing nature events for sure or any kind of action that is hard to capture without record constantly, LOVE IT!!!
Title: Re: Pre-recording for the raw_rec module?
Post by: Kharak on March 14, 2017, 03:06:37 PM
Hey, I am not much around either these days.

Could I suggest you rename this thread to 'Pre-Recording for Raw_Rec Module'? Its confusing to search for this on the forum when the thread does not correspond with the name of the feature.

So I downloaded latest nightly build yesterday, very eager and surprised by this new feature. Very very happy with it.

Though I had troubles getting any Module to Link as it said. I suppose it was because I had the Crop_rec module on there aswell, not sure? Deleted everything of the SD card and then it worked, used to be I just copy paste and overwrite whatever was on. Is Crop_rec Module not compatible with the latest nightly ? I mean for MLV_Rec aswell ? I don't remember if Crop_rec was only Experimental. Crop_Rec modules is also a Killer feature that I really enjoy working with. anyways, back on topic.

Could it be possible to adjust the desired amount of Pre-Recording by scrolling the shutter wheel in increments of 8-16 frames or so or just 1 second at a time? And that it showed more precisely how much is pre-recorded, preferably in X.X secs or even Frames? I had an issue today when I set it at 5 seconds, trying to film waves breaking over this ship I am shooting on. First of, it counted 3 seconds and when I hit record again a message popped up everytime saying something like "Buffer can not reach desired amount" and it would cancel the recording, so had to set it to 2 Secs. I don't remember exactly the message because when I was done shooting and trying to replicate the problem for this post, it worked with the 5 Seconds e.g. Counting to 3 secs and hitting record worked.. So I don't know what was going on there today. All settings were the same when I tried to replicate.

Which brings me back to why it would be nice to have clearer information on how much I am Pre-Recording. When it says "3 Secs" pre-recorded? Is that precisely 3 secs e.g. 3x 24 Frames or is it 3 secs and some more frames until it reaches 4x24 frames and it says 4 Secs ? I assume the buffer is not always holding the exact same amount of frames depending on background processes and other settings on camera?

Oh! and one more question, will this feature be seen in MLV_Rec in the future? for sound and metadata recording ?

Anyways, thank you so much for you're continued efforts with these cameras. My MK III still feels on par with anything else coming out these days.

Latest Nightly: March 12 -  2017
1920x1080 14 bit
MK III

Kharak.

EDIT: What steps could I take to increase the amount of Buffering in 14 bit ?
Title: Re: Pre-recording for the raw_rec module?
Post by: a1ex on March 14, 2017, 06:15:36 PM
You might be able to use it with crop_rec if you copy just mlv_lite.mo from the nightly, and you keep the old files from the crop_rec build. Didn't try, but if it doesn't work, I can sync the branches. The crop_rec branch is experimental because of the backend (last time I took it outside, i.e. many months ago, I had ERR70).

The error message was probably just the buffer getting full. When you select a large amount, it reserves 16MB for future buffering; maybe I should reserve some more. At such small buffer sizes, it can quickly run out of steam if the card write speed isn't known very well (it's measured on the fly), or if unexpected speed drops occur (they do). Though I don't remember noticing this message when testing (maybe I didn't push the settings far enough).

The display is currently rounded. I agree a more exact display could be useful, especially on cameras with less memory.

The buffer size will become less of an issue with reduced bit depths. I'm thinking to merge the current state into nightly and just hide the reduced bit depth menu on models with known issues, as it seems to work pretty well on recent models (except in-camera playback, which require raw_twk).
Title: Re: Pre-recording for the raw_rec module?
Post by: a1ex on March 15, 2017, 12:27:47 AM
I think I've got a way to limit the max memory available for pre-recording without causing the regular recording to run out of steam.

How it works? The amount of memory reserved for recording (that is, not available for pre-recording) was fixed to 16MB before, and now it depends on both resolution and write speed. I expect recording will continue after pre-recording, so I reserve a buffer large enough to allow recording 500 frames with 90% of the measured write speed, but not more than half of the RAM and not less than 10 frames. That should give continuous recording, with buffer usage peaking somewhere near maximum, and then recovering.

That also means the pre-recording buffer size drops quickly as your required write speed approaches your card write speed.

There is an exception: if the trigger is set to "Half-shutter: 1 frame", there's little need for buffering after pre-recording. In this case, the pre-recording buffer is maxed out (using nearly all the available RAM).

The amount of memory available for pre-recording can now be seen in menu, if your selection exceeds the limit. Caveat: you need to record a test clip first.

Experimental build available; please report.
Title: Re: Pre-recording for the raw_rec module?
Post by: Kharak on March 15, 2017, 01:26:13 AM
Reporting in:

Tested your new experimental mlv_lite with latest nightly and it worked on the first try. I set it at 5 seconds and it buffered to 3 secs and I hit record again and it worked.

I then did as you suggested, added this latest mlv_lite to the latest crop_rec build (I was apparently running an older version) and it too worked flawlessly. Though on very first try, it buffered to 2 secs and then on second try and every shot after that it was back to 3 secs.

And nice with the finer increments by the way, I used 5 Secs on theses tests because thats what failed on my shoot earlier this day. Maybe it failed last time because I just switched from mlv_rec to mlv_lite and the buffer somehow gets "confused" or some code stuff happens there. Just thought it might be a detail worth mentioning for the record, though it seems its resolved now anyways.

Thank you for your quick response and fix.

I have just seen that there is a Proxy recording module or build. How would that fair with Pre-Recording? I would personally not mind if the H264 file is not Pre-Recorded if it had to be so, but could this also somehow be conjoined? I have not tested it on a stand-alone basis or even read through the entire thread, so not sure how stable it is.

EDIT:

MK III
Lexar 1066x 128GB
14 bit
Title: Re: Pre-recording for the raw_rec module?
Post by: mk11174 on March 17, 2017, 07:29:15 AM
Working nicely here, love the grayed out amount of how many seconds you can get when selecting a to high of a pre-buffer, very nice addon!

Man, if this had motion detection and luminance detection this would be the ultimate module, not that it isn't already, but damn, this is working very nice for those addons to be of use.
Title: Re: Pre-recording for the raw_rec module?
Post by: janvkem on March 24, 2017, 12:25:03 AM
Installed magiclantern-crop3x.2017Mar21.5D3113 and all seems to work (only checked briefly). Thanks a lot for combining it with the 10 bit recording! I really like the pre rec option!
Title: Re: Pre-recording for the raw_rec module?
Post by: vstrglv on March 24, 2017, 08:19:11 PM
I also installed magiclantern-crop3x.2017Mar21.5D3113. It works fine. Many thanks for compiling.
I have some questions.
1. If i load mvl_rec.mo and mvl_lite.mo and activate mvl_rec there is no sound recording.
2.If I load mvl_lite.mo and raw_rec.mo there is no difference which of them to activate. Both work as mlv_lite.
Title: Re: Pre-recording for the raw_rec module?
Post by: g3gg0 on March 24, 2017, 09:15:03 PM
Quote
@a1ex

The assumption that size differs only by 48 bytes turned out to be wrong. Difference can vary. Here are the sample logs from various MLVs.

When using crop_rec there are no NULL blocks in MLV at all, Frame size also varies. ML version used to record these crops: magiclantern-crop3x.2017Jan13.5D3113.zip.

This somehow OT here b/c all of them recorded by mlv_rec (audio is on) not raw_rec/mlv_lite (to be inspected yet).

Here is excerpt from ordinary non crop MLV log recorded back in 2014:

what you might be seeing is the feature of mlv_rec to embed periodic/onevent blocks into the "DMA-alignment-padding" area of VIDF blocks.
this area would else have been just waste. so i decided it to place some low rate data in it - like RTCI and stuff.

iirc this is done by modifying frameSpace. (today i would simply use NULL blocks for that)
Title: Re: Pre-recording for the raw_rec module?
Post by: bouncyball on March 25, 2017, 11:16:54 AM
@g3gg0

Thank you for the explanation. I suspected it was caused by some alignment-padding.

Just one more thing: why crop_rec MLVs do not have any NULL blocks?
Title: Re: Pre-recording for the raw_rec module?
Post by: a1ex on March 25, 2017, 01:15:23 PM
There is no such thing as crop_rec MLVs, as crop_rec simply reconfigures the video mode. The MLVs are still made by either mlv_rec or mlv_lite.

Since crop_rec does not (yet?) alter the resolution of the original buffer (it simply changes the sensor scanning parameters), I don't expect it causing any changes in the file structure.

If you are talking about mlv_lite, NULL blocks are used either after the main headers (to make sure their size is multiple of 512, for write speed reasons), or when the 4GB limit is auto-detected the hard way (after a write failure), to discard the incomplete frame that might have been written.

Frame padding (to ensure EDMAC alignment and block size multiple of 512) is done the old-school way (frameSpace and blockSize slightly larger than actual payload, so there may be unused data both before and after the frame - whose size is xRes * yRes * bpp/8). I agree using NULL blocks would have been nicer (just needs extra care if the skip amounts are smaller than the size of a NULL block).
Title: Re: Pre-recording for the raw_rec module?
Post by: bouncyball on March 25, 2017, 05:08:52 PM
Quote from: a1ex on March 25, 2017, 01:15:23 PM
There is no such thing as crop_rec MLVs, as crop_rec simply reconfigures the video mode. The MLVs are still made by either mlv_rec or mlv_lite.
Yes I know :) sorry about incorrectly formulated question. I meant "MLVs written while using crop_rec.mo".

Quote from: a1ex on March 25, 2017, 01:15:23 PM
If you are talking about mlv_lite, NULL blocks are used either after the main headers (to make sure their size is multiple of 512, for write speed reasons)
Not mlv_lite, I was asking g3gg0 about mlv_rec. Usually MLVs recorded by mlv_rec have lot of NULLs in it. But when used in conjunction with crop_rec, NULLs are simply not there. Tried with lot of MLVs with same result.

And again sorry about OT in this thread :) It just g3gg0's answer triggered that question.
Mlv lite MLVs have always simple and straight structure (I know about EDMAC alignment and block size multiple of 512 for speed)

Thanks for info
bb

Title: Re: Pre-recording for the raw_rec module?
Post by: g3gg0 on March 25, 2017, 05:15:56 PM
right. it was due to the fact back then we were just happy to get it running with the level how things were understood.
in such cases sometimes you don't see clearly :)

today there is a lot better understanding of camera internals like EDMAC and stuff.
so if the raw file format for ML would be defined today, it would have some smaller things done different.

in general i think it is extensible enough to satisfy all (current) needs, correct me if i am wrong :)
Title: Re: Pre-recording for the raw_rec module?
Post by: g3gg0 on March 25, 2017, 05:20:23 PM
Quote from: bouncyball on March 25, 2017, 05:08:52 PM
Mlv lite MLVs have always simple and straight structure (I know about EDMAC alignment and block size multiple of 512 for speed)

yep, thats why (at work) i always say that things have to get implemented twice.
first to get from the idea to some concept that works and is usable.

and second some other person going over it, understand the initial goals, the solutions and
implement the same thing again with the experience gained from the first iteration.

but yeah, OT a bit :)


edit: especially with mlv_rec i really didn't know what will happen. some niche thing probably?
so some things were quite experimental but scalable. obviously sacrificing performance a few per cent too much.
Title: Re: Pre-recording for the raw_rec module?
Post by: a1ex on March 27, 2017, 12:03:49 PM
Updated to address some comments from dfort (see the PR page):

Quote
I started out setting the Pre-record for 1 second and the Reg trigger to Half-shut:1 frame. Every half-shutter recorded a second instead of just 1 frame.

Ok--don't really need pre-record for this so I turned that off and now I'm getting 2 frames per half-shutter press. Not sure if I'm doing something wrong but it looks like all the settings are telling me that it should record only 1 frame.

Also added fractional time display while recording.
Title: Re: Pre-recording for the raw_rec module?
Post by: bouncyball on March 27, 2017, 06:30:40 PM
@g3gg0: :D
Title: Re: Pre-recording for the raw_rec module?
Post by: vstrglv on March 29, 2017, 07:51:59 PM
mvl_lite.mo from magiclantern-crop3x.2017Mar21.5D3113 supports 10 and 12bit, but mvl_lite.mo from Exp. Latest Build (2017-03-27 23:39) not. Why? 5D3 1.1.3
Title: Re: Pre-recording for the raw_rec module?
Post by: a1ex on March 29, 2017, 09:01:43 PM
Because the 10- and 12-bit recording is not yet in mainline.

Feedback is mixed, and I don't use it in practice, so I could not judge whether it's OK to merge it or not in this state. Sorry.
Title: Re: Pre-recording for the raw_rec module?
Post by: vstrglv on March 29, 2017, 10:10:26 PM
Thank you alex. It's a pitty, because 10 and 12bit reduce bit rate threshold for CF card.
If i load mvl_rec.mo and mvl_lite.mo from Exp. Latest Build (2017-03-27 23:39) and activate mvl_rec there is  sound recording now. 
Title: Re: Pre-recording for the raw_rec module?
Post by: a1ex on March 29, 2017, 10:34:13 PM
The two modules are not meant to be loaded together (as any symbols exported by both will conflict somehow, but the linker doesn't warn about it and I don't really know how it's handled). I should probably enforce this somehow.

The mlv_lite module from the crop_rec branch also has the H.264 proxy included, and it's faking mlv_snd status in order to show the audio meters (since you can use the H.264 stream for audio). This is the source of conflict.

The conflict with mlv_snd is an obvious one, but I bet there are more.

Not sure what's the best way to fix, as this method (exporting symbols and letting the linker figure it out) allows some communication between modules, but it has its limits.
Title: Re: Pre-recording for the raw_rec module?
Post by: dfort on April 09, 2017, 07:28:14 AM
I've been playing around with rec-trigger. There's a special mlv_lite module in the Experiments download page (https://builds.magiclantern.fm/experiments.html) that should work on all platforms. If you configure it a certain way you can shoot single frames which is great for timelapse and animation. The way to do it is by setting the Pre-record to OFF and the Rec trigger to Half-shut:pre only.

(https://c1.staticflickr.com/4/3829/33883291796_f1b145914f.jpg)

An issue I had was that the counter shows time in minutes, seconds and tenths of seconds: MM:SS.1/10sec. so with a standard frame per second video rate you need to take a few exposures before the counter advances.

(https://c1.staticflickr.com/3/2901/33883293366_a66b3e4b51.jpg)

I tried changing the display so it uses a timecode like, MM:SS.FF. I thought it was pretty cool so I submitted a pull request (https://bitbucket.org/hudson/magic-lantern/pull-requests/823/display-counter-in-timecode-format-mm-ssff/diff) before testing all the possible variations of the Pre-record options. a1ex found all sorts of problems with my code so I started going further and further down the rabbit hole, changing variable names and adding menu items until I got something that might be useful. It is still quite experimental so I stuck it in the Advanced menu.

(https://c1.staticflickr.com/3/2846/33794848631_e2464cc263.jpg)

As you can see I left the time display that was adapted from the ffmpeg time unit syntax (https://trac.ffmpeg.org/wiki/Seeking#Timeunitsyntax) as the default and added the option to display the counter in Timecode format or in Frames. This probably isn't SMPTE approved timecode because it is based on the record frame rate instead of the playback frame rate but it does advance every time a frame is saved and it displays pretty much the same minutes and seconds as the ffmpeg version.

If the camera is set for NTSC 30fps (actually 29.976 more or less) the frame counter goes from 00 to 29 then it advances the seconds counter. Likewise at 25fps it goes from 00 to 24 and at 24fps it goes from 00 to 23, just like timecode. This should help for those cases when you are shooting one-frame-at-a-time and need to be more accurate than 1/10sec.

(https://c1.staticflickr.com/3/2852/33883285736_18f66a54da.jpg)

If anyone wants to try it out I put a test build of the mlv_lite module which should work on all platforms in my bitbucket download area (https://bitbucket.org/daniel_fort/magic-lantern/downloads/). Look for magiclantern-mlv_lite-rec-trigger-experiment.

Feel free to post comments on the pull request:

https://bitbucket.org/hudson/magic-lantern/pull-requests/823/display-counter-in-timecode-format-mm-ssff/diff

Title: Re: Pre-recording for the raw_rec module?
Post by: DeafEyeJedi on April 09, 2017, 08:45:14 AM
Thanks for this much antcipated Timecode implementation, @dfort and I guess I wasn't supposed to try this together with an Non-CPU lens info experimental build (2017-04-04) because I am getting this message below running on 5D3.123 and not sure if this is to be expected?

(https://c1.staticflickr.com/3/2882/33082479784_2698b22181.jpg) (https://flic.kr/p/SpomEu)

Perhaps I should try w Nightlies instead?
Title: Re: Pre-recording for the raw_rec module?
Post by: dfort on April 09, 2017, 07:05:02 PM
Interesting find @DeafEyeJedi. I only tried it with a nightly build.

Best to look at one experiment at a time.
Title: Re: Pre-recording for the raw_rec module?
Post by: X-RAY on August 02, 2017, 06:54:11 PM
Hey guys,
can anyone please explain how to set up the raw video for lighting photography? I played around with it and its not clear to me how especially the rec trigger setting works.
I'd like to record the last second when I hit a cable remote trigger and I thought that works when I set to "Half Shutter" but it doesn't do anything. I can only start/stop record on the back button of the 5DIII.

And is it possible to lower the fps to get more seconds pre-recorded (in the buffer)? When I use fps override it seems to alter the shutter-speed also. Can I set the fps independent to the shutter speed?

Thanks!
Title: Re: Pre-recording for the raw_rec module?
Post by: Walter Schulz on August 02, 2017, 07:21:09 PM
Some options are available for some cams only. See https://builds.magiclantern.fm/features.html
The option you are looking for is "MOVIE_REC_KEY" in Movie tab and 5D3 is the only cam *not* supported. If there is a "REC key" item in your Movie tab I want to call it a bug.
However there is an item "Sticky HalfShutter" in Prefs tab -> Misc key settings and that's a different ball park.

Have you tried using IR remote control to stop video recording?
Title: Re: Pre-recording for the raw_rec module?
Post by: X-RAY on August 09, 2017, 03:03:32 PM
Hey there, didn't know that the 5DIII doesn't support the option.
You mentioned the "Sticky HalfShutter" but how does that help in my case?!

And does anybody know how to set a small framerate like 10fps and shoot with a separate shutterspeed like 1/50?
Title: Re: Pre-recording for the raw_rec module?
Post by: Walter Schulz on August 10, 2017, 01:50:01 PM
I don't own a 5D3 and cannot test it.
But - as said above - 5D3 is the only cam not supporting "Half-Shutter" for movie start/stop and therefore I have serious doubts about this option inside this module working at all (on 5D3).

"Sticky Half-Shutter" does indeed not help for your problem.

Have you tried IR remote trigger in movie mode?
Title: Re: Pre-recording for the raw_rec module?
Post by: a1ex on August 10, 2017, 02:05:09 PM
Quote from: X-RAY on August 02, 2017, 06:54:11 PM
Hey guys,
can anyone please explain how to set up the raw video for lighting photography? I played around with it and its not clear to me how especially the rec trigger setting works.
I'd like to record the last second when I hit a cable remote trigger and I thought that works when I set to "Half Shutter" but it doesn't do anything. I can only start/stop record on the back button of the 5DIII.

I've only tested it using the half-shutter button from the camera (not with a cable remote). Should work out of the box - can you show your raw video settings?

With the rec trigger enabled, the LiveView button (labeled as start/stop on 5D3) would only prepare and end the recording session; to record any frames, you would have to press half-shutter.

The "Sticky HalfShutter" Walter is talking about, does not apply here; this is a special build of mlv_lite that has a half-shutter trigger for recording (e.g. record one frame, pre-record 1 second before the trigger and so on). It's also available in the crop_rec_4k builds. It's not yet in unified because... I didn't use it for more than a few very short test clips before committing the code, so right now I have little or no idea how it performs.
Title: Re: Pre-recording for the raw_rec module?
Post by: X-RAY on August 16, 2017, 12:18:39 AM
Quote from: Walter Schulz on August 10, 2017, 01:50:01 PM
Have you tried IR remote trigger in movie mode?
Sorry, but I dont have one to try out.

Quote from: a1ex on August 10, 2017, 02:05:09 PM
I've only tested it using the half-shutter button from the camera (not with a cable remote). Should work out of the box - can you show your raw video settings?

With the rec trigger enabled, the LiveView button (labeled as start/stop on 5D3) would only prepare and end the recording session; to record any frames, you would have to press half-shutter.

The "Sticky HalfShutter" Walter is talking about, does not apply here; this is a special build of mlv_lite that has a half-shutter trigger for recording (e.g. record one frame, pre-record 1 second before the trigger and so on). It's also available in the crop_rec_4k builds. It's not yet in unified because... I didn't use it for more than a few very short test clips before committing the code, so right now I have little or no idea how it performs.
Ohh, now that you told me and as I tested the options again I understand it. The start/stop just like starts the container ... after that I can start recording with the half shutter. Right, that works fine. :-) And now I also understand the prerecording especially the option with 1-frame / pre only. So thats the setting I need to choose when shooting lightnings. So that I can record the maximum amount of frame buffer BEFORE pressing the Half Shutter button.

So now my only question remaining is how to get the fps down independent to the shutter speed. Alex, do you have a tip on that? :-)
Title: Re: Pre-recording for the raw_rec module?
Post by: a1ex on August 16, 2017, 08:20:35 AM
For FPS, the low light mode will scale the shutter speed, while the other modes will not.

Unfortunately, only low light is compatible with high-res modes on 5D3 (at least for now). You should, however, be able to adjust the shutter speed after dialing the FPS.
Title: Re: Pre-recording for the raw_rec module?
Post by: X-RAY on August 18, 2017, 11:17:57 PM
Quote from: a1ex on August 16, 2017, 08:20:35 AM
For FPS, the low light mode will scale the shutter speed, while the other modes will not.

Unfortunately, only low light is compatible with high-res modes on 5D3 (at least for now). You should, however, be able to adjust the shutter speed after dialing the FPS.
Great ... you know your code! :-D I set exact fps and after that the shutter speed can be set separate without a problem. Now I can set it up exactly as I want. thx
Title: Re: Pre-recording for the raw_rec module?
Post by: Francois_lune on May 28, 2021, 01:29:58 PM
Hi there,

I am using the Pre-recording (half-shutter) function to digitalize Super 8 films. A projector is sending a signal to the camera for each frame allowing me to record Raw videos fully synchronized.

However I am asking myself how this function works exactly.

The projector is running at 8 fps. The timing of the capture start to be accurate when the fps of the camera is a least double. Around 18fps is fine. Under this value, the timing is bad.

I don't really understand why. Does it mean that the function only "check" if the half-shutter is pressed at the frequency of the fps set on the camera?

In my case I though that the fps would not play any role as the actual recording speed is set by the projector.