crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView

[a1ex]

Recorded a test clip, about 6.5 minutes, 5 chunks, 14-bit lossless 1920x1080, mlv_lite; it plays fine in mlv_play.

mlv_dump from crop_rec_4k:

Code: [Select]

mlv_dump M04-0015.MLV

 MLV Dumper
-----------------

Mode of operation:
   - Input MLV file: 'M04-0015.MLV'
   - Verify file structure
File M04-0015.MLV opened
File M04-0015.M00 opened
File M04-0015.M01 opened
File M04-0015.M02 opened
File M04-0015.M03 opened
File M04-0015.M04 not existing.
Processing...

Reached end of chunk 1/5 after 2261 blocks
Got a new RAWI, throwing away average buffers etc.
Got a new RAWI, throwing away previous frame buffers etc.

Reached end of chunk 2/5 after 2267 blocks
Got a new RAWI, throwing away average buffers etc.
Got a new RAWI, throwing away previous frame buffers etc.

Reached end of chunk 3/5 after 2303 blocks
Got a new RAWI, throwing away average buffers etc.
Got a new RAWI, throwing away previous frame buffers etc.

Reached end of chunk 4/5 after 2307 blocks
Got a new RAWI, throwing away average buffers etc.
Got a new RAWI, throwing away previous frame buffers etc.

Reached end of chunk 5/5 after 765 blocks
Processed 9847 video frames at 25.00 FPS (393.88 s)
Done

No errors in valgrind. No obvious issues near frame 336 either.

Can you upload the buggy file somewhere?

[Danne]

Been a bit messy with testing. Conclusion:
mlv_dump from crop_rec_4k branch is getting segmentation fault 11 on all files tested today.

My version of ml-dng mlv_dump was causing the error before which is separate from any of the other issues. Tested a newer build of mine and it works processing the correct amount of dng files. Correct frame processing metadata is showing as well. mlv_rec seems all fine. Sorry for the detour.
Tested with a 3:21min MLV with accompanying proxy and it worked perfectly. And I mean perfectly matched from first to last frame. Over 4000 dng files.

About the 336 frame issue I don´t have a clue how to create it. I think Kharak is using 5D mark III 1.2.3 version.

Recorded a test clip, about 6.5 minutes, 5 chunks, 14-bit lossless 1920x1080, mlv_lite; it plays fine in mlv_play.

I think mlv_rec together with mlv_play is broken.

[Danne]

This change in makefile causes segmentation fault 11 in mlv_dump:
https://bitbucket.org/hudson/magic-lantern/commits/9ce033951922df7a721bfe48a665d9d03656a7d8?at=crop_rec_4k

[Ali Oliya]

Hi

cr2hdr app is unable to extract few of my MLV files that exceeds 4GB limit (MLV and M00, M01....). it only creates 1 pink frame. any help with that?

[Danne]

Did you check posts above yours?
Anyway. I renamed the app to Switch and hopefully fixed that issue of yours. Test and report back.
http://www.magiclantern.fm/forum/index.php?topic=15108.0

[Ali Oliya]

@ Danne

worked like charm. Thanks a lot!

[Kharak]

About the 336 frame issue I don´t have a clue how to create it. I think Kharak is using 5D mark III 1.2.3 version.

I run 123 yes.

Sorry. for the big file: 
https://mega.nz/#!pQwT0BQT!r2P1GdlBTgXBcQrx3g4Cqao_pVUO6TLv_cOxZt-zKmQ 751 MB

here is one example of the repeat frame.

And I can basically reproduce this all the time.

Frame 336 and 338 are the same.

Just to be clear, I think what I wrote about reproducing the issue can be misinterpreted. I don't mean that it is always frame 336 and 338 on every MLV. I mean that almost every MLV has repeated frames, but at random. I read my post and see it could be interpreted as 336 and 338 are always the repeated frames.

But it is always 2 frames apart. I thought it was more random, like some 1-5 frames apart, but so far I see it is always 2 frames apart for the repeat frames. 181-183, 144-146 etc.

I tried cutting the repeat frame out to see if it goes 1-2-(1)-3-4-5-6..etc. But I believe it does overrides the 3rd frame so it is 1-2-(1)-4-5-6.. Cutting the frame out makes it pretty much unnoticeable, but it is there.

[a1ex]

I've looked at it last night, but don't have the conclusions ready - meanwhile, can you try to reproduce on 1.1.3?

[Kharak]

Yes, I will try 113

[Kharak]

@a1ex

On the experimental page you write: "Continuous: 3072x1308 @ 24p, 1920x1080 @ 45p, 1920x960 @ 50p, 1920x800 @ 60p, 4096x2560 @ 8p, 5796x3870 @ 5p."

1920x800 60p 3x3 is not continuous. Atleast not on 1.2.3. I will try 1.1.3 tomorrow, is it supposed to be continuous on 113?

[a1ex]

Just checked - it is (still) continuous on 1.2.3.

Did the first test with all the speed hacks enabled (global draw off, frozen LV, card warm up 1GB). Result: it filled the 32GB card (Komputerbay 1000x) on the first try. Details:

- started from ML defaults
- formatted CF card as FAT32 from camera (ML was running from SD)
- loaded crop_rec, mlv_lite, mlv_play, file_man
- set crop mode 50/60p 3x3
- set 60 fps in Canon menu
- enabled raw video (1920x800, 14-bit lossless - it's the default)
- noted compression rate for the test scene (52%)
- disabled global draw, enabled frozen LV and card warm up (1GB)
- rebooted
- pressed REC after the LED activity settled
- when recording stopped, rebooted again and Canon's free space warning appeared.

It was a bit on the edge though - the recording icon was orange all the time, showing ~ 70-80 seconds. The meaning of this number is: if the average write speed would drop to 95% of the measured value (this percentage is hardcoded - 100/105 - see predicted_frames_left), recording would stop after 80 seconds (or whatever that number is).

After that, ran a second test with global draw on (all others unchanged from above) and got similar results (card nearly filled with a single clip, 7 chunks, but the estimation dropped to ~ 16-50 seconds and there were 2GB of free space left). The icon was still orange throughout the test and the preview was in grayscale. Note the writing speed usually drops when the card becomes full (at least with my card).

A third test (with ML defaults, crop_rec to 50/60p 3x3) stopped after less than 10 seconds (so, Frozen LV is important).

A fourth test (back to first settings, test scene with 57% ratio - ISO 6400) stopped after ~ 30 seconds (red icon).

Didn't try on 1.1.3, but I expect it to be a little faster.

BTW - did any of you play with this speed hack ?

[Kharak]

Okey, Frozen LV explains a lot. I never use Frozen LV, its too hard to work with for me.

Yes, I think all cards slow down when 5% is left, maybe more. All my Lexar Cards do the same. For instance when I have 5 GB left, 60P is almost useless.

You are on 123. Are you unable to reproduce the Repeat frames? I still have not looked at any shots I did with Aug-26th Build (just random shots) and I see there is a newer one out now. I will try the newest Experimental before downgrading to 113. I assume you were already on the Aug 26th Build when you tried to reproduce? Maybe even the newest one ?

I just checked footage shot over a period from 14th to 19th of July and they also have plenty of repeat frames, some MLVs have several. And some even more severe a problem where about 3 frames are missing, so 1 2 1 X X X 7 8 9... etc. I cant remember what build that was shot on though, but I believe it was the one from 4th of July or thereabouts.. :/ Atleast it shows its not a recent bug introduced.

my settings:
1920x800 60p Crop mode 3x3
14 Bit lossless
Global Draw ON (But I always hold Half-Shutter right after hitting record to make it go away which increases recording times)
RAW Zebras ON
Histogram RAW Based RGB - Scaling: Log, ETTR Hint, Clip Warning ON.
Real Time Preview
Shutter Fine Tuning: +1.25
Digic Peaking - Slightly Sharper
Exposure Override ON
Movie Simulation.

if you could try with these settings again to reproduce. Perhaps the Half Shutter - Clear Overlays thing could be of interest?

And no I have not tried the Speed Hack, it seems a bit too much over my head.

[The_bald_guy]

I tested the latest proxy mode by loading the firmware: magiclantern-crop_rec_4k.2017Sep03.5D3113

Module loaded :

MLV Lite
MLV Rec
MLV Sound
MLV Play

I had a sdhc card and a CF card inside the 5D mkIII.

The proxy file has a Timecode starting at 00:07:58:00
The proxy has sound embeded

The Raw file has a timecode starting at 00:00:00:00
The frames are only aligned at the beginning if I trim 32 black frames on the H.264 file. The raw file is also shorter of 14 frames at the end.

When I recorded different clips, they all seem to have different in and out. And it made it worst to add the pre-recording to two seconds.

My point is that it's propably useless to have proxys with MLV_Lite right now since there is no time code or a consistent way to get them in sync for post production. The best improvement would be to have a consistent timecode.

[Danne]

Useless? Certainly not. TC can easily be changed in ffmpeg.
http://www.magiclantern.fm/forum/index.php?topic=15108.msg189532#msg189532

Also read around the last 30 posts here on the subject.

[dfort]

My point is that it's propably useless to have proxys with MLV_Lite right now since there is no time code or a consistent way to get them in sync for post production.

Have you tried using Danne's switch? It trims the H.264 proxy to the same length as the DNG image sequence along with matching timecode. (Danne also replied while I was still writing this post but he didn't mention his awesome program.)

Forgot to mention - 700D and EOS M will need an extra constant (easy to find).

The SRM_BUFFER_SIZE for the EOSM and 700D were pretty easy to find. An interesting difference between these cameras was that on the EOSM the SRM_BUFFER_SIZE wasn't visible until I pressed record or took a silent still while on the 700D it was available as soon as the camera booted up. Maybe this is related to the EOSM not getting into mv1080 mode until it is recording H.264?

[Danne]

Maybe this is related to the EOSM not getting into mv1080 mode until it is recording H.264?

Cross my fingers. Lossless working and just bought a sigma 10-20mm f3.5 for it filming crop_rec lossless.
Tried proxy filming as well but files came out corrupted and early stop etc...

[dfort]

Tried proxy filming as well but files came out corrupted and early stop etc...

Flashback to when mv1080 (via H.264 proxy) was almost working on the EOSM:

http://www.magiclantern.fm/forum/index.php?topic=16608.msg178438#msg178438

[a1ex]

An interesting difference between these cameras was that on the EOSM the SRM_BUFFER_SIZE wasn't visible until I pressed record or took a silent still while on the 700D it was available as soon as the camera booted up.

It should be visible as soon as the SRM memory is allocated by something in ML. The latest mlv_lite, if enabled and the camera is in movie mode, will allocate this memory during standby (as soon as the camera starts); this change is recent - the memory allocation used to be performed when pressing REC. The silent picture module will allocate the memory when taking a picture. Best guess: different settings (maybe raw video wasn't enabled at startup) or maybe the EOS M had an older mlv_lite.

There's a bunch of activity displayed if you enable the console - there you can tell when the memory is (re)allocated.

[tecgen]

SRM_BUFFER_SIZE for the 550D

https://www.dropbox.com/s/04yxfu5e98llgz8/srm_malloc_cbr_550D.jpg

Code: [Select]

srm_malloc_cbr(156510, 48000064, 1f68000)

A1ex or David, could one of you please add the following line to the consts.h header file within the platform/550D.109 folder of the crop_rec_4k branch? (My old ML fork does not yet contain the crop_rec_4k branch and I didn't found a way to sync newer branches to my repository. I guess I need to fork it again.)

Code: [Select]

#define SRM_BUFFER_SIZE 0x1F68000

Next time I try to find the SRM_BUFFER_SIZE for the 50D and 5D2 too.

[vstrglv]

May be I missed something.
I tested the latest proxy mode by loading the firmware: magiclantern-crop_rec_4k.2017Sep03.5D3113
Module loaded :

MLV_Lite
crop_rec
ettr
file_man
pic_view
1920X1080
There is a scale difference in mov and dng images. I used scale - 99,1%, Position - 967;542 in AE for mov image to fit to dng one.
mlv_dump.exe on 2017Sep03

[a1ex]

I believe the H.264 is interpolated from 1904 x about 1072. Does this give a better match?

Recording at this resolution might require only a small offset (a few pixels, if any). Didn't test - it's just what I'd expect to happen.

[vstrglv]

Thank you! No problem for an exact match, only interesting.

[a1ex]

You are on 123. Are you unable to reproduce the Repeat frames?

Still unable to reproduce. Added a check in mlv_play, to spot the issue without swapping the cards - it triggers on your sample footage, but could not get the warnings on my own footage, even with your settings and usage pattern.

Instead, this check uncovered another bug: when recording was about to stop, there was a speed optimization to save fewer frames at a time, so their buffers could be reused for other frames. The bug was: rather than writing say 7 frames instead of 10, and group the remaining 3 with whatever else will get recorded meanwhile, it was writing all the 10 frames, and then it was writing those 3 frames again (possibly grouped with others). The bug was introduced with variable frame size (required for lossless compression, but also affected uncompressed recordings). It's not present in unified. Didn't benchmark the change, but I expect it to squeeze a few more frames when recording is not continuous.

[Teamsleepkid]

sorry if this has been covered but i read almost all the posts and i don't remember seeing it. when i use compressed raw recording at 14bit my liveview is normal perfectly fine perfectly usable. when i drop to 12bit it gets darker. then when i switch to 10 bit its so dark that its unusable for framing. i know if i push the half shutter it brightens up like its normal but thats too clumsy for filming. i don't have a 5d mark iii to test it with and see if that camera is doing the same thing. you guys probably already know about this but just putting it out there in case I'm missing something... I reread the first page and i get it just wondering if anything has happened since april 1st on it. also don't think I'm trying to complain or ask for features I'm not just simply trying to talk.

[dfort]

@Teamsleepkid - you probably have Preview set to Real-time. Try the default, Auto. Yeah, it probably looks crappy but it isn't dark!