crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView

[timbytheriver]

Have updated the 16:9 graphic to show 2K(QHD).

[leozan]

ANY HOPE to see later long time update of :
ML Nightly Build and ML crop_rec_4k  for 5D3 (123)  ?
...with Latest Lua updates + fixes 2019-03-20 ?

no way to do alone , bad....sorry.
thanks

[Walter Schulz]

Please tell us in detail which of the problems you are struggling with requires a merge of crop_rec_4k and lua_fix because it cannot be handled by either version alone?

Ceterum censeo experimental build page esse delendam!

[jimiz]

Please tell us in detail which of the problems you are struggling with requires a merge of crop_rec_4k and lua_fix because it cannot be handled by either version alone?

Ceterum censeo experimental build page esse delendam!

 e98cf27: api_test.lua: test for all items from the Overlay menu
 (to make sure they can be safely turned on or off with menu.set)
 TODO: also check their submenu options

 4c6cde3: Waveform: fix crash when enabled outside ML menu
 (e.g. from Lua, https://www.magiclantern.fm/forum/index.php?topic=23673)
 Issue: initialization on first use + race condition.

 abdd1e2: api_test.lua: on EOS M, the INFO button may get us out of LiveView
 (fixes API test about ML overlay status)

 085b79c: EOS M: fixed display_idle() outside LiveView

 4aa1ba0: Zoom on HalfShutter: allow triggering when AF is enabled
 (only works when AF is not actually triggered by half-shutter, *, AF-ON etc)

 df8d088: Zoom tweaks: updated for recent models
 - option to disable the "x1" zoom (i.e. to go directly into x5/x10)
 - compatibility with touch-screen models (still rough)
 - api_test.lua updated (need to disable these tweaks in order to pass the zoom test)

 3a5bade: api_test.lua: skip checking for "Play mode actions"
 (feature not present on EOS M; requires serious cleanup)

 400d73f: EOS M: enabled rack focus and focus stacking
 (they appear to work, at least with regular EF lenses)

 0060b26: editor.lua: disabled debugging features
 (FIXME: they don't seem to work)

 252a4b7: EOS M, 100D: fix Q/SET short press outside ML menu
 (ETTR trigger and many others)

 3f5aa76: editor.lua: use the MENU key as primary method for triggering the menu
 (Q is still used, but it's not working on all models)

 b213791: EOS M: align long-press symbols for delete and Q/SET
 (minor)

 20628f1: EOS M, 100D: fix long-press symbol remaining visible after Q/SET
 https://www.magiclantern.fm/forum/index.php?topic=14828.msg210032#msg210032

 f40dacf: Arrow key shortcuts: minor cleanups
 - removed bogus ARROW_MODE_TOGGLE_KEY definition on EOS M
 - removed unused definitions on 650D and 1100D
 - require ARROW_MODE_TOGGLE_KEY if and only if FEATURE_ARROW_SHORTCUTS is defined

 9e54731: EOS M: fix PlayMain_handler stub
 7000D, 650D, 100D: comments for PlayMain_handler / PlayMovieGuideApp_handler

 7b7a9a3: Color scheme: fix conflict with fast zebras

 684ff6b: Menu: fix typo regarding FEATURE_JUNKIE_MENU
 (7469c34)

 91c84fd: exit_play_qr_menu_mode: wait until mode switch is finished
 (fixes GUI mode switching test in api_test.lua on EOS M, maybe others)

 7256e17: EOS M: fix GUIMODE_ML_MENU to allow operation outside LiveView
 (required for passing the Lua menu API tests outside LV)


sorry can't help you.
I don't know how update api_test.lua inside ML and recompile.
but seem most fox for EOS M I think....not 5d3, or meaby not...bhoooooo?!
the rest seem dead..

[leozan]

Thanks , no problem , WAS only a question, no afraid....

[jimiz]

if you like try something more "new" for 5d3 ,  look here...take time...  https://bitbucket.org/Dannephoto/magic-lantern/downloads/

[a1ex]

Quoting dmilligan:

How about a 1000% refund of the purchase price of ML? I will pay that to you personally, let me know where to send the check.

FYI - 99% of my spare time (i.e. when I'm not at work and not with my family) already goes to ML. Lately I've been focusing on other areas, such as new ports, or just researching and documenting Canon internals. I don't record video, and the pictures I've been taking for myself are sitting in the waiting queue for years (guess why). Honestly, I don't know how to do better. OK, let's say I could focus more on the video side, or on some particular camera model, but that will be at the expense of neglecting other areas of ML (which unfortunately have their own share of trolls). FYI, there are areas in ML that did not get their fair "round robin" time slice (i.e. my attention) for years.

If you think the progress is too slow, the only way to speed it up is to join the development efforts. Trolling, like you are doing here, only serves to drive me away from this topic (can't speak for others, but for me it's not funny to read such things after an intensive debugging session, or after a full day at work). It's not the first time you are complaining for what you've got for free, so please take it as a warning. Be sure to check this guide before replying.

Yeah, I could have fixed another bug in ML instead of writing this.

Ceterum censeo experimental build page esse delendam!

OK, what to do with it? Let users build from source? Just include everything in mainline without testing? Other ideas?

Anyone who thinks they can do a better job is welcome to take over maintaining ML (release management, keeping stuff up to date, making sure things work on all supported models and so on), as it's already taking 110% of my time.

[nikfreak]

... Other ideas?

Anyone who thinks they can do a better job is welcome to take over maintaining ML (release management, keeping stuff up to date, making sure things work on all supported models and so on), as it's already taking 110% of my time.

while we are offtopic and I am at 109% myself (just family and work not counting ML) I may just add my 2cents for those supported models you mentioned earlier as a proud owner of unsupported models (got it?) 
As harsh as it may sound I would announce some kind of "code freeze":

• Merge as much as possible of those open PRs (don't forget mine)
• announce dropping support for Digic IV and older cameras (yes those oldie goldies like 7D, 5D2, 50D, 60D, 1x00D)
  
• release a "final" build before dropping support for those cameras
• Do some code cleanup afterwards (yes cleanup code so anyone wanting support for those cameras has to use a code freezed branch on btbucket / github or whatever). Just point them with links over there. Anyone's free to pick up work for it.
• focus on digic 5/6/7/8 and onwards do exactly the same for digic V announcing to drop support for it but keep 5D3 because it has been built to survive. It's THE DIGIC V camera to keep / pick up.
• Afterwards focus on digic 6/7/8

Nothing will be lost but official support will be simply discontinued to free up resources for future work on new cameras. I know it's a hard decision. You mention yourself being not able to handle more cameras yourself. I guess dust spreaded all over your cameras (i know what I am talking about) but you need to get rid off 'em in terms of developing / testing ML on 'em. Just use them as a photographer but don't develop / test them. I know it's hard. You have been spending thousands of countless hours with them but I think ML is nothing w/o you as the lead developer and if you don't make the first DIGIC 6 / 7 / 8 port no one else seems to be capable to do it but everyone is expecting it atm. Once accomplishing the first working digic 6/7/8 port others will be able to follow your work. I don't think "someone" will suddenly appear and do the first DIGIC 6 / 7 / 8 port for ML. You have been asking others to pick up work for it for so long already. Maybe later "someone" could appear but I personally think that chances are much higher that most users will just switch away and use their smartphones or whatever tool to take their stills and 8k videos.



At least think about it. Something like the 5D4 would fit everyone's need for the next years.
Now let's return back to topic please 

[nikfreak]

forgot to mention: On April 1st please, as usual 

[leozan]

  well ... a little life on the topic ...   

[timbytheriver]

Should I be taking notice of all these files on my SD (ML install) card – or simply deleting them?

Thanks.

[a1ex]

If you can find a way to reproduce these errors, that would be even better.

The CRASH*.LOG files are errors coming from Canon firmware (i.e. ERR70), usually caused by ML. The others are from ML.

[timbytheriver]

@a1ex Ok, thanks. So in this example, what information should I note, and how to reproduce? :

ML ASSERT:
0
at mlv_lite.c:2712 (compress_task), task compress_task
lv:1 mode:3

compress_task stack: 1b0940 [1b09d0-1af9d0]
0x0006A01C @ ac1e5c:1b0970
0x00069878 @ 6a084:1b0940

Magic Lantern version : Nightly.2019Feb15.5D3123
Mercurial changeset   : d60f7852c937 (crop_rec_4k_mlv_snd_isogain_1x3_presets) tip
Built on 2019-02-15 15:16:20 UTC by Danne@MacBook-Pro-3.local.
Free Memory  : 211K + 2954K

[a1ex]

If you look into the source code (changeset d60f7852c937, line 2712 in mlv_lite.c), it had trouble compressing a frame to LJ92 while recording.

I'd expect a corrupted frame in the clip you have recorded when that message appeared.

[timbytheriver]

Should I be looking for source code changeset d60f7852c937, line 2712 on Danne's page – or the main ML page? Sorry to be a bitbucket noob…

[Danne]

If you look into the source code (changeset d60f7852c937, line 2712 in mlv_lite.c), it had trouble compressing a frame to LJ92 while recording.

I'd expect a corrupted frame in the clip you have recorded when that message appeared.

I have general question about corruped files. Is there a way to reduce these corrupted frames besides lowering resolution or fps, disabling global draw etc? Are there other "tweaks" on values, let´s say lj92 compression for instance which could affect outcome? Anything in mlv_lite.c or maybe something memory-related?
Another one is how much do gui code affect recording times and overhead on cpu? Disabling global draw has clearly an effect on recording times but there´s a lot of other things going on under the hood. So thinking one could go hardcore disabling almost everything but basic recording indicator but skip everything else to optimize in a maximized way.

[Danne]

Think I fixed an issue causing corrupted frames with the 48fps setting on the 5DIII:
https://bitbucket.org/Dannephoto/magic-lantern/commits/c7c0ebf6f6489347d8148b0779a79b1c4bb748ef

Downloads here:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/

Or compile yourself here:
https://bitbucket.org/Dannephoto/magic-lantern/branch/crop_rec_4k_mlv_snd_isogain_1x3_presets

[miket]

Using crop_rec_4k to digitize movie film - test results

My movie digitization machine came off the back burner recently and was fitted with a hall sensor for reliable half press triggering (each 1/2 press captures 1 frame into an mlv file).  Here are the long overdue test results.  It works great!  I have been running the projector at 10 fps without any blurred frames in the 3k crop mode at 2560 x 1920 14 bits lossless continuous recording on a 5D3 1.1.3.  Magic Lantern crop_rec_4k is amazing software for this purpose.

https://drive.google.com/open?id=184kg_8hzHTJwUDwKXIsQlGjavjSka6dy

While the resolution may seem higher than needed for 16mm movie films, I have found that it's much easier to crop off the extra pixels in post rather than trying to make fine adjustments to the camera position during capture.  Also, different movie cameras mask the image area differently with some movie cameras having image area running out to, and sometimes, beyond the film sprocket holes.  I used a file on the film mask in my projector so I can record almost to the edge of the film.

a few raw frames which are h and v flipped
https://drive.google.com/open?id=1GLMwmbsmOTAPqvqUHhfwDQg2TI9QxLPz

Timing between ML the projector film advance is the key to avoiding blurred frames. Running a capture and then looking through a few thousand frames to check for frames with blurs was not efficient so I set up a test to capture the projector main shaft position when ML captures the frame from the buffer.

example blurred frame
https://drive.google.com/open?id=1k-NNk0ZmYOVdCzdq45So0YJ3f1K7sMsD

Here is a great video about how a projector works, it may help with understanding my test video below.

How a Film Projector Works by the engineerguy
https://youtu.be/En__V0oEJsU

Running the projector at about 1/2 the Canon camera frame rate seems to be the most reliable to avoid catching the frame while the film is advancing.  The film advances within about 1/8 of a turn of the main shaft.  My projector shutter has been removed so I have the other 7/8 of a turn to capture the frame at a standstill.  I timed the hall sensor activation so it activates just before the film advances since the half shutter activation captures a frame already in the camera buffer.  Here's the video, hopefully, it makes sense!

https://vimeo.com/327622574

I find it amazing that the half-press triggering at 20fps works.  I ran a previous test in 3k crop mode at 2560 x 1920 14 bits lossless for 6000 frames (the exact number expected) but there were lots of blurred frames.

On one hand, the 3x3 pixel binning at 48fps looks very workable though the lens would need a lot more extension tubes which will complicate things.  The more macro magnification the harder it is to set and maintain focus due to even more shallow depth of field.     

Running at 10 FPS is more than enough for my project, that said, running at full projector speed of 18 to about 24fps would benefit other projects like this since the projector drive would not need to be re-worked, which is a time-consuming task.

If there are any setting that I could change to get the 3k crop mode at 2560 x 1920 14 bits lossless to run at a higher fps please make a suggestion.  I have tried FPS override, it crashes if I change to higher frame rates.

I captured a film through the projector this weekend as an example.  The mlv file was converted to an image sequence then globally graded and sharpened in Davinci Resolve.  This reel was shot by my Grandfather during the 1969 United States Grand Prix at Watkins Glen.

https://vimeo.com/327447552

[a1ex]

I find it amazing that the half-press triggering at 20fps works.  I ran a previous test in 3k crop mode at 2560 x 1920 14 bits lossless for 6000 frames (the exact number expected) but there were lots of blurred frames.

Nice! However, that's expected, as the half-shutter trigger simply selects the last captured frame in buffer, for saving. That's all it does. Image capture still runs at the LiveView frame rate, i.e. not in sync with half-shutter events.

I wanted to prepare a real half-shutter trigger that would actually start the exposure on each external event (that's called genlock, right?), but... debugging the M50 took way longer than I've initially expected.

If there are any setting that I could change to get the 3k crop mode at 2560 x 1920 14 bits lossless to run at a higher fps please make a suggestion.

That's technically possible; I "just" need to find the time to sit down and finish the bleeding edge crop_rec. Maybe Danne already has a preset for it (didn't try).

I'd expect the frame rate to go up to about 40 FPS in 2.5K according to my theory about FPS timers (3596->2560 => timer A 518->388), but for some reason I couldn't achieve that in practice, last time I tried.

[timbytheriver]

Think I fixed an issue causing corrupted frames with the 48fps setting on the 5DIII:
https://bitbucket.org/Dannephoto/magic-lantern/commits/c7c0ebf6f6489347d8148b0779a79b1c4bb748ef

@Danne. Thanks! 45fps PAL tested: seems fine (Trying NTSC 1280x720 60fps always results in crashes and super dark preview/shots with very pink highlights.) – with no crashes or corrupted frames so far. However, on this build and previous (15Feb2019) the 1920x1920 1:1 preset appears broken and only records straight mv1080. ?

Cheers.

[Danne]

I can´t reproduce your issues. Could others share if builds are working or not? Also uploaded two new version, small changes:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/

This version let´s you exit x10 zoom in 48fps mode but you´ll need to update with canon menu button for regs to reset. I noticed that Either lowering resolution before coming back to 48fps setting or lowering fps will get you out of the stuck liveview. Not sure how to temporary change fps while in x10 mode and switch back when in x1 mode but that would possibly solve this issue.

[timbytheriver]

@Danne. Thanks.

I can´t reproduce your issues. Could others share if builds are working or not? Also uploaded two new version, small changes:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/
...in x1 mode but that would possibly solve this issue.

Which issue? NTSC 60fps or 1920x1920 1:1 ?

[Danne]

There's no 1920x1920 preset among my presets. There's x3zoom(digital zoom).

[timbytheriver]

@Danne. Sorry! Confused myself there. 1280x720 NTSC 60fps throws these errors:

[miket]

@a1ex - Its a dream come true as it is, if you do find time to work on it I would love to give it a try!

I'd expect the frame rate to go up to about 40 FPS in 2.5K according to my theory about FPS timers (3596->2560 => timer A 518->388), but for some reason I couldn't achieve that in practice, last time I tried.

I'll do some testing and let you know.

Mike