Magic Lantern Forum

Developing Magic Lantern => Modules Development => Topic started by: a1ex on April 01, 2017, 11:15:41 AM

Title: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 01, 2017, 11:15:41 AM
As you probably have guessed from the latest developments (QEMU (http://www.magiclantern.fm/forum/index.php?topic=2864.125), EDMAC graphs (http://www.magiclantern.fm/forum/index.php?topic=18315.0), JPCORE (http://www.magiclantern.fm/forum/index.php?topic=18443.0), EEKO (http://www.magiclantern.fm/forum/index.php?topic=13408.msg175656#msg175656)), our understanding on how LiveView works has improved considerably. Finally, all my fiddling with QEMU, at first sight with little or no purpose for the everyday users (http://www.magiclantern.fm/forum/index.php?topic=2864.msg169970#msg169970), starts paying off.

Today, Magic Lantern proudly announces new ground-breaking features that were previously thought impossible or very hard to achieve.

We proudly present....

4K RAW Video Recording!

(http://a1ex.magiclantern.fm/2017Apr01/fishy_4k_april_fools.png)

DOWNLOAD (http://builds.magiclantern.fm/experiments.html)

Twitter (https://twitter.com/autoexec_bin/status/848102541305020416) announcement (https://twitter.com/autoexec_bin/status/848313311141789696)

(http://a1ex.magiclantern.fm/2017Apr01/4k.png) (http://a1ex.magiclantern.fm/2017Apr01/50p.png) (http://a1ex.magiclantern.fm/2017Apr01/allmodes.png)


On the 5D Mark III, you now have the following new resolutions:

* 1920x960 @ 50p (both 1:1 crop and full-frame - 3x3 pixel binning) - continuous*)
* 1920x800 @ 60p (same as above)  - continuous*)
* 1920x1080 @ 45p and 48p (3x3 binning)  - continuous at 45p
* 1920x1920 @ 24p (1:1 square crop) - continuous*)
* 3072x1920 @ 24p (1:1 crop)
* 3840x1536 @ 24p (1:1 crop) (corrupted frames at 1600)
* 4096x2560 @ 12.5p (1:1 crop) - continuous*) at 8 FPS
* 4096x1440 @ 25p (1:1 crop)
* Full-resolution LiveView: 5796x3870 at 7.4 fps (128ms rolling shutter) - continuous*) at 5 FPS!

The last feature complements the well-known full-resolution silent pictures (http://www.magiclantern.fm/forum/index.php?topic=12523.0) - the new implementation will be usable at fast shutter speeds, without the exposure gradient (http://www.magiclantern.fm/forum/index.php?topic=12523.msg120750;topicseen#msg120750) - but with rolling shutter (just like regular LiveView frames).

*) Continuous recording is achieved in 14-bit lossless compression mode (LJ92) (http://www.magiclantern.fm/forum/index.php?topic=18443.msg182074#msg182074), preview set to Frozen LV (previously known as Hacked Preview), low ISOs (about 55% ratio). With complex (highly detailed) scenes and/or higher ISO, the image doesn't compress very well, so recording may no longer be continuous.

Credits: Greg (full-width LiveView (http://www.magiclantern.fm/forum/index.php?topic=10111.msg123909#msg123909)), g3gg0 (video timer (http://magiclantern.wikia.com/wiki/VideoTimer), DIGIC registers (http://magiclantern.wikia.com/wiki/Register_Map) documentation and lots of other low-level insights).

Complete list of new video modes:
Code: [Select]
                                /*   24p   25p   30p   50p   60p */
    [CROP_PRESET_3X_TALL]       = { 1920, 1728, 1536,  960,  800 }, /* 1920 */
    [CROP_PRESET_3x3_1X]        = { 1290, 1290, 1290,  960,  800 }, /* 1920 */
    [CROP_PRESET_3x3_1X_48p]    = { 1290, 1290, 1290, 1080, 1080 }, /* 1920; 1080p45/48 <- 50/60p in menu */
    [CROP_PRESET_3K]            = { 1920, 1728, 1504,  760,  680 }, /* 3072 */
    [CROP_PRESET_UHD]           = { 1536, 1472, 1120,  640,  540 }, /* 3840 */
    [CROP_PRESET_4K_HFPS]       = { 2560, 2560, 2500, 1440, 1200 }, /* 4096 half-FPS */
    [CROP_PRESET_FULLRES_LV]    = { 3870, 3870, 3870, 3870, 3870 }, /* 5796 */

What else could you wish for?



FAQ

Where's the catch?

This is only a very rough proof of concept. It has not been battle-tested and has many quirks. Some of them may be easy to fix, others not so. In particular:

* It feels quite buggy. I'm still hunting the issues one by one, but it's hard, as Canon's LiveView implementation is very complex, and our understanding on how it works is still very limited.
* Write speeds are high. For example, 10-bit 4096x2500 at 15 fps requires 180 MB/s. 1080p45 should be a little more manageable at 111 MB/s.
* Canon preview is broken in most modes; you need to use the grayscale preview in the raw recording module.
* High-resolution modes (in particular, full-res LiveView) may cause trouble with memory management. This is very tricky to solve, as we only get 3 full-resolution buffers in LiveView, with restrictions on the order in which they must be freed, and lots of other quirks.
* Since these settings were pushed to limit, the risk of corrupted frames is high. If it happens, decrease the vertical resolution a bit (from the crop_rec submenu).
* When refreshing LiveView settings, the camera might lock-up (no idea why). Pressing MENU twice appears to fix it.

May I fine-tune the new modes?

Yes! I've included some of the knobs on the user interface. Normally you shouldn't need to touch these buttons, but if you do, you might be able to squeeze a few more pixels.

Does it work with FPS override?

Sort of. It's not reliable at this point, so it's best not to try yet.

Overheating?

During my tests, I didn't manage to get a sensor temperature higher than 60 degrees. Your mileage may vary.


Risks?

This mod changes some low-level sensor parameters that are not well understood. They were all figured by trial and error, and there are no guarantees about the safety of these changes.

As usual, if it breaks, it's your fault, sorry.

Will it work on other camera models?

I hope so; however, this is an area where I hope to get contributions from others (yes, from you). If these new features don't motivate you to look into it, I wonder what else will.

I'll explain how all this works in the coming days or weeks.

Is it difficult to port to other camera models?

So far, the 3x3 720p mode from crop_rec was ported to EOS M (rbrune) and 700D (dfort). So it shouldn't be that hard...

Will you port this to my camera model, please?

No, sorry. I have better things to do - such as, preparing the April 1st prank for next year :)

Wait a minute, didn't you say you are primarily a still photo user? Why are you even doing this?

If you look close, the usefulness for video is fairly limited, as the write speeds (and therefore the recording times) are not practical.

But the full-resolution LiveView is - in my opinion - very useful for still photo users. Although the current implementation is not very polished (it's just a proof of concept), I hope you'll like the idea of a 7.4 FPS burst mode, 100% silent, without shutter actuations.

Right now, you can take the mlv_lite module with pre-recording and half-shutter trigger: at 10 bits per pixel, you get 5 frames pre-recorded, and saved to card as soon as you touch the half-shutter button. Or, you can capture one frame for each half-shutter press, with negative shutter lag! (since the captured frame will always be pre-recorded).

And if a burst at 7.4 fps is not enough, you may also look at the 4K modes (12-15 fps).

(I know, I know, GH4 already does this, at much higher frame rates...)

The help menu for full-res LiveView says 5796x3870, but MLV Lite only records 5784x56. What's going on?

The raw recording modules have a couple of alignment constraints (e.g. can only start cropping from a multiple of 8 pixels, and the size of the cropped area (that goes into the MLV file) must be multiple of 16 bytes (that is, W*bpp/8 + H mod 16 must be 0).

To capture the full resolution, you may use the silent picture module. However, this module is not the best when it comes to memory management and buffering. Currently, you'll get an impressive buffer of 2 frames in burst mode :)

But hey - it outputs lossless DNG!

What about that lossless compression routine?

It's included, although I didn't manage to test it much. There is a lot of room for improvement, but for a proof of concept, it seems to work.



P.S. The initial announcement was disguised as an April Fools joke, just like the original crop_rec (http://www.magiclantern.fm/forum/index.php?topic=17021.0).

Twitter announcement (https://twitter.com/autoexec_bin/status/848102541305020416)

From original April Fools post:
Quote

With our latest achievements in wizardry with ARM programming and DIGIC reverse engineering, we can speak of a new era of raw video recording.

On models like the 5D Mark III, the next upcoming releases will feature an improved version of our crop_rec module (http://www.magiclantern.fm/forum/index.php?topic=17021.0) that delivers the following new resolutions:
 
* 1920x960 @ 50p (both 1:1 crop and full-frame - 3x3 pixel binning)
* 1920x800 @ 60p (same as above)
* 1920x1080 @ 45p and 48p (3x3 binning)
* 1920x1920 @ 24p (1:1 square crop)
* 3072x1920 @ 24p (1:1 crop)
* 3840x1600 @ 24p (1:1 crop)
* 4096x2560 @ 12.5p (1:1 crop)
* Full-resolution LiveView: 5796x3870 at 7.4 fps (128ms rolling shutter).

The last feature complements the well-known full-resolution silent pictures (http://www.magiclantern.fm/forum/index.php?topic=12523.0) - the new implementation will be usable at fast shutter speeds, without the exposure gradient (http://www.magiclantern.fm/forum/index.php?topic=12523.msg120750;topicseen#msg120750) - but with rolling shutter (just like regular LiveView frames).

Please understand that providing the source code for those highly DIGIC optimized routines is a bit troublesome and will need some extra legal care. After this step is taken and as soon we are finished with ensuring the product quality you are used from Magic Lantern, we will upload the code to our repository.

Consider this being a huge leap towards our next mind boggling goal:

8K RAW Video Recording!

(http://a1ex.magiclantern.fm/2017Apr01/fishy_8k_april_fools.png)

Sample DNG from 5D Mark III, to show that our proof of concept is working:

8k.dng (http://a1ex.magiclantern.fm/2017Apr01/8k.dng)

Stay tuned for more information!

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Walter Schulz on April 01, 2017, 11:22:04 AM
Who is ML's officiail bookie? I want to risk some Euro ...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: nikfreak on April 01, 2017, 12:12:57 PM
Shut up and take my money!
(https://www.awesomegifs.com/wp-content/uploads/psy-elevator-gangnam-style.gif)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: g3gg0 on April 01, 2017, 12:13:26 PM
this is wonderful! i cant wait to test it on my camera at home!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Wayne H on April 01, 2017, 12:13:52 PM
INCREDIBLE!!!!! Wonderful job alex, i really appreciate all the hard work you and and the other developers have put in over the years to get magic lantern to where it's at right now.

ULTRA HD RAW 24p on 5d mark iii wooah :) ,so the most obvious question is when will we be able to test out this new update.   

Feels like xmas. :)

Alex you are a genius.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 01, 2017, 12:24:51 PM
Incredible. It,s art.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: jagnje on April 01, 2017, 12:32:31 PM
april 1st
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: nikfreak on April 01, 2017, 12:40:17 PM
Nope, I would rather say it's the whole truth as long as no one talks about expects continuous recording:

http://www.magiclantern.fm/forum/index.php?topic=18300.0

 8) :P 8) :P :D :D ;D 8) 8)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Ilia3101 on April 01, 2017, 01:18:10 PM
KooooooooL  8) any chance of anything for the mark 2 or other cameras?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: beauchampy on April 01, 2017, 01:29:47 PM
Please guys, I can't take the heart ache. Is this an April fools? Specifically the 960px 3x3?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Levas on April 01, 2017, 01:50:51 PM
I love how you get a 8K DNG sample from a 5d mark III, there's some real magic going on there  ;D
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Walter Schulz on April 01, 2017, 02:02:02 PM
Is this an April fools?

You don't expect an answer without tricks, traps, false mirrors and double meaning, do you? No today, not today ...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Greg on April 01, 2017, 02:06:12 PM
WOW!  :o
8K is a bit noisy.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Frank7D on April 01, 2017, 08:17:34 PM
Thank you so much for everything you've done! Now, not to be demanding, but 16K is obviously doable so when can we expect it?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: nikki on April 01, 2017, 08:23:19 PM
 ..it works perfect
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 01, 2017, 11:31:34 PM
Committed some of the sources; still playing with it, but those who can compile from source can already give it a try.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hyalinejim on April 01, 2017, 11:50:36 PM
a1ex, which of the crop modes offer a real-time, colour preview?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 01, 2017, 11:56:32 PM
Only the simplest crop mode (1920x1080) has good real-time preview. The modes with higher vertical resolutions have color preview too, but it's not centered (only the top of the frame is shown).

In the regular crop_rec branch, there's also centered 5x zoom. Here I've removed it temporarily, as it increased the code complexity quite a bit, but I'll probably add it later when things will settle.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: reddeercity on April 02, 2017, 12:06:54 AM
I read all this and thought Great Joke ! :P but then I read the Source Code Holy Cow  :o :o :o :o
There goes next few days  , I'll be living in front of my computer exploring this and hopefully I can compile for 5d2  :D 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Ilia3101 on April 02, 2017, 12:11:53 AM
reddeercity do you think that's possible yet? Wouldn't it require tons more reverse engineering?
Damn it I just happen to be away from proper computers that can compile ML for a week right now so I won't be able to try it if it is possible :'(
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: beauchampy on April 02, 2017, 01:00:53 AM
Build for 5d3 available? I just spent a week shooting 50p 3x: so would love to put it through its paces
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Walter Schulz on April 02, 2017, 01:04:05 AM
Top of page -> Downloads
Download page -> Downloads (drop-down menu) -> Experiments
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: reddeercity on April 02, 2017, 01:19:01 AM
reddeercity do you think that's possible yet? Wouldn't it require tons more reverse engineering?
Will not sure yet but I'm looking in to it .
If you read this post Full-resolution silent pictures  (http://www.magiclantern.fm/forum/index.php?topic=12523.msg120497#msg120497) there's info about 5D2 and lead me to believe it maybe possible  8)
I'm currently try to compile the code now , but I get some error's .
some tweak.o stuff , I working of older code , so I think that could be the problem so I'll update to the latest source code
and try again . I don't think 4k is a reality on 5d2 but maybe 3K as the bandwidth for 4K is around 150MB from what I read.
Since we are limited to max. 80MB/s  and by your Yet Another RAW Video Calculator  (http://rawcalculator.bitballoon.com/calculator_desktop)  at 10bit 3x crop 2496x1134 @ 2:2.1 A.R. is 80MB/s
unless Lossless compression is used and that's another thing by it's self .
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 02, 2017, 01:38:02 AM
Greg's original proof of concept (full-width LiveView (http://www.magiclantern.fm/forum/index.php?topic=10111.msg123909#msg123909)) was done on 500D (same generation as 5D2 and 50D).

Sure, it probably needs a bunch of reverse engineering, but it probably boils down to changing some ADTG registers and video timers.

I've actually got these presets working in adtg_gui first. A proof of concept for 3K was committed back in 2016 (https://bitbucket.org/hudson/magic-lantern/commits/50d8f06018) (also mentioned (http://www.magiclantern.fm/forum/index.php?topic=17021.msg171809#msg171809) in the "classic" crop_rec thread).

The registers interesting for changing resolution are documented either on the Register Map wiki page, or in adtg_gui source. Interesting commits: 8bbce9c (https://bitbucket.org/hudson/magic-lantern/commits/8bbce9c), 7333869 (https://bitbucket.org/hudson/magic-lantern/commits/7333869), 4a66a48 (https://bitbucket.org/hudson/magic-lantern/commits/4a66a48), 6c62a6b (https://bitbucket.org/hudson/magic-lantern/commits/6c62a6b), 38e78de (https://bitbucket.org/hudson/magic-lantern/commits/38e78de).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: reddeercity on April 02, 2017, 02:21:58 AM
Thank you  :)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 02, 2017, 07:37:54 AM
Getting some compile error with mlv_rec and mlv_snd in crop_rec_4k branch.

Code: [Select]
Building module mlv_rec...
Updated HGVERSION
[ README   ]   module_strings.h
[ CC       ]   mlv_rec.o
mlv_rec.c: In function 'raw_rec_update_preview':
mlv_rec.c:4160:5: error: too few arguments to function 'raw_set_preview_rect'
     raw_set_preview_rect(skip_x, skip_y, res_x, res_y);
     ^
In file included from ../lv_rec/lv_rec.h:24:0,
                 from mlv_rec.c:76:
../../src/raw.h:148:6: note: declared here
 void raw_set_preview_rect(int x, int y, int w, int h, int obey_info_bars);
      ^
make[4]: *** [mlv_rec.o] Error 1

Don´t have my camera to test but I follow the error and change the line in mlv_rec to below which seems to get rid of the error code for both mlv_snd and mlv_rec.
In mlv_rec
Code: [Select]
raw_set_preview_rect(skip_x, skip_y, res_x, res_y);to this:
Code: [Select]
void raw_set_preview_rect(int x, int y, int w, int h, int obey_info_bars);Or maybe change to:
Code: [Select]
void raw_set_preview_rect(skip_x, skip_y, res_x, res_y);
?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: andy kh on April 02, 2017, 07:58:10 AM
this is wonderful. need to start looking for a used 5D mark III
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: ItsMeLenny on April 02, 2017, 09:55:45 AM
Well I guess the original 4K trolls are no longer trolls.
But if 4K is possible that must mean 1000fps is as well.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: kaco on April 02, 2017, 02:32:39 PM
Respect to a1ex and the whole magiclantern team.
No more excuses for not producing world class imagery.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: dfort on April 02, 2017, 07:19:37 PM
Is it difficult to port to other camera models?

So far, the 3x3 720p mode from crop_rec was ported to EOS M (rbrune) and 700D (dfort). So it shouldn't be that hard...

I'll take that as a compliment! :D :D :D

That reminds me, I should do a pull request for the 700D but it looks like there's more work to do now. How about getting some other cameras up to speed on the crop_rec module? By the way, don't expect 4k on anything other than the 5D3--then again maybe there will be more surprises.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 02, 2017, 08:58:03 PM
Yep - my point was that people outside the core team are definitely able to play with this code (as in, it's not pure black magic).

Took this build out for a test (nothing fancy, just taking pictures of the kids with full-res LiveView and pre-recording). Noticed a major limitation (could not set shutter speeds fast enough for daylight) and pushed a fix for that.

After the fix, modes with reduced FPS (4K and full-res LiveView) now have the usual range of 1/4000...1/30 (or 1/60) mapped to 1/15000 to 1/fps. That's right - 1/15000 exposure time with 128ms rolling shutter :D

Next annoyance was the grayscale preview, which is slow and not working at all in 10-bit. Also, lossless MLVs are not playable in mlv_play. Will look into those.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hyalinejim on April 02, 2017, 09:08:46 PM
a1ex,  if you can improve the greyscale preview you will be the king of kings
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Greg on April 02, 2017, 09:41:03 PM
These are just approximate calculations. You can initially estimate the speed of the sensor.

5D3 22MPx x 6fps = 132MPx/s
4096px x 2560px x 12.5fps = 131MPx/s

7D   18MPx x 8fps = 144MPx/s
70D 20MPx x 7fps = 140MPx/s
6D   20MPx x 4.5fps = 90MPx/s

500D 15MPx x 3.4fps = 51MPx/s  :-[
Btw, my 4 year old smartphone 8MPx x 30fps = 240MPx/s  :P

More accurate calculations - https://www.magiclantern.fm/forum/index.php?topic=12656.0
Of course we must remember about the speed of writing.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: dfort on April 02, 2017, 09:46:26 PM
Yep - my point was that people outside the core team are definitely able to play with this code (as in, it's not pure black magic).

My point too. I'd like to get some more people to start playing with the code. There's lots of information in the comments and on the forum that takes the mystery out of what's going on--though some of what you're doing sure seems like pure black magic.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Ilia3101 on April 02, 2017, 10:29:41 PM
@Greg
and the rather slow 5D2: 21.2MPx x 3.9fps = 82.68MPx/s. Does this basically tell us the limit of the sensor? So this calculation works: 82.68/23.976 = 3.45MPx limit at 24p on this camera? So I guess it could do 2560x1320(3.38MPx).
 :D how does one implement the "Greg's resolution hack" that I've been hearing about?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: vstrglv on April 02, 2017, 10:31:53 PM
Fantastic! Thank you very much!
One Q about * 1920x960 @ 50p (both 1:1 crop and full-frame - 3x3 pixel binning).
1920x960 @ 50p (3x3 pixel binning) works good, but i can not set 1920x960 @ 50p for 1:1 crop, only 1920X632.
I set Crop mode - 1920 1:1 and 50p in Canon menu.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: nikfreak on April 02, 2017, 10:39:20 PM
:D how does one implement the "Greg's resolution hack" that I've been hearing about?
http://www.magiclantern.fm/forum/index.php?topic=19300.msg182149#msg182149
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: reddeercity on April 02, 2017, 10:47:42 PM
@Ilia3101 I'm working on it  , just had some compiling issue with adtg_gui yesterday should have that resolve
today , then I can work off a1ex's links  ;D
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 02, 2017, 10:57:19 PM
1920x960 @ 50p (3x3 pixel binning) works good, but i can not set 1920x960 @ 50p for 1:1 crop, only 1920X632.
I set Crop mode - 1920 1:1 and 50p in Canon menu.

1920 1:1 tall (meaning "with increased vertical resolution", but couldn't fit that into menu)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: vstrglv on April 02, 2017, 11:31:16 PM
Thank you! It works good!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 02, 2017, 11:59:08 PM
Updated the preview with something a little more usable. It's color (though still low-res and non-realtime) and only drops to grayscale when recording speed becomes critical.

Unfortunately, g3gg0's raw_twk (https://www.magiclantern.fm/forum/index.php?topic=16682.0) routine doesn't appear to work in LiveView, so we are stuck with (slow) CPU-based previews for now :(

I can probably get faster previews in 14-bit lossless mode, by configuring the source raw stream as 16-bit (unpacked), since the compression routine accepts 10/12/14/16-bit input (http://www.magiclantern.fm/forum/index.php?topic=18443.msg181620#msg181620). For uncompressed modes, I'm afraid we currently don't have a way to get both unpacked data (for fast previews) and bit-packed (for recording) while in LiveView (so all my previous ideas (http://www.magiclantern.fm/forum/index.php?topic=5601.msg177531;topicseen#msg177531) are currently fiction).

New build posted on the Experiments page.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: mk11174 on April 03, 2017, 01:06:28 AM
Trying to see what the next step is to see if there are any modes 700d can get, first i messed with atdg tool to try to reproduce the 3x3bin mode while in 720p by changing 0x4 to 0x2 on 800c just to understand how the tool works since those values are posted. I then set the tool to show values that have been changed more then twice and screen captured every value for 1080p, 1080p x5 and x10, 720p, 720p x5 and x10, crop mode in 1080p and also photo mode just to track all the values that are changing in case it was needed.

Thats where i am at now though, not sure what values to change though, gets scary when you see the crazy images on live view that make u think u burned sensor out. Only thing i saw that did anything as far as the size was 800c changing to 0x0 makes a zoomed version but its stretched tall. Other then that another one moved the image up and down.

Dont know how to cross reference with the 5d3 cause i see Pack12(?,?)  for example and the values are not like what you see in the tool. So pretty much clueless on where to go from here??
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hyalinejim on April 03, 2017, 01:42:21 AM
Wow! That's a hell of a lot more useable and the preview is centered which is absolutely kickass.

I get 30 frames in UHD, but 3K is still continuous for me in crop_rec. The preview does drop back to greyscale about 50% of the time in 3K. The colour preview is more detailed and has a higher refresh rate... it's pretty much usable, I think.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: WonkyTuna on April 03, 2017, 02:23:44 AM
I'm trying it out but unfortunately when I try to watch it on MlRawViewer or export it to DNGs using RAWMagic I'm left with this: https://drive.google.com/open?id=0B1lko58z3g6lRTlKUWc1YzlzOFk. I tried re-downloading the build and formatting my card but I can't seem to get anything out of any of these crop modes...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: squig on April 03, 2017, 04:11:54 AM
I've found a couple of issues in both builds. The first is nothing major: when you change a resolution in the crop mode menu, say 3K to UHD, you have to exit the ML menu before you can adjust the resolution size in the raw video menu. The other issue is making me pull out what little hair I have left: the lossless compression ratio appears to have a mind of its own, it's like a recalcitrant AI. At one point it was compressing at 51% on the 1st build:

(http://i.imgur.com/6IX4XKI.jpg)

When I loaded the 2nd build it started at 85%:

(http://i.imgur.com/HB2uNHV.jpg)

So I had to scale down to:

(http://i.imgur.com/bXY36uK.jpg)

After a few seconds of recording the compression went to 70%:

(http://i.imgur.com/Ojd4ykL.jpg)

Then when I try to increase the resolution I get this [Expect xxxx frames]:

(http://i.imgur.com/NckArF4.jpg)

A battery pull gets rid of the [Expect xxxx frames], but on the 2nd build it reappears when I try to scale up the resolution. So I went back to the 1st build to see if I could record at 3.3K 51% lossless again. Initially I was able to get this setting:

(http://i.imgur.com/Mj734Xe.jpg)

That was a bit much for the card, it stopped recording after a few seconds resulting in the dreaded [Expect xxxx frames]:

(http://i.imgur.com/kR4G8cb.jpg)

Taking me back to square one where I had to scale the resolution back down:

(http://i.imgur.com/gNnZrKY.jpg)

Ideally the compression ratio would either be fixed, or manually adjustable.











Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: dmilligan on April 03, 2017, 04:44:18 AM
Ideally the compression ratio would either be fixed, or manually adjustable.
That's mathematically impossible, and not how lossless compression (https://en.wikipedia.org/wiki/Lossless_compression) works. The compression ratio you get is totally dependent on the scene, and it's even theoretically possible that the compression results in larger file sizes than the original uncompressed data. This typically happens when you give a lossless compression algorithm data that has very little redundancy (already compressed), or that is not of a type the algorithm is "tuned" for, for example feeding the compressor 10-bit raw data results in the compressed version actually being larger (http://www.magiclantern.fm/forum/index.php?topic=18443.msg176699#msg176699).

For the most part, lossless compression algorithms work by simply eliminating redundant data via more efficient encoding. Take some text for example, encoded using ASCII, each character takes 1 byte to store and therefore all characters have equal weight even ones that are rarely used. However we know that in any particular language there are characters (or even sequences of characters) that typically appear more often than others, so if we gave the characters that appear more frequently, shorter encodings (which subsequently requires less common characters to have longer encodings), then overall we can save space. However if we were to feed random gibberish to such an algorithm, it might actually take more total space the store than the original simple encoding, since the characters with longer encodings are just as likely to appear as ones with shorter encodings.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: squig on April 03, 2017, 05:15:03 AM
Understood, but from a cinematography perspective; I need to figure out what resolution will give the card enough headroom to write at that resolution continuously at least 90-95% of the time; so far it looks like that's somewhere in the 2.5-2.6K range for 24p 2.39:1, which is still great.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: squig on April 03, 2017, 09:03:55 AM
2.7-2.8K 2.39:1 looks like the sweet spot for repeatable continuous recording on the Toshiba 1066x card. By "repeatable" I mean repeatable near 100% of the time. Same goes for 3.3K anamorphic scope: 2272x1364 with a 1.5x anamorphic lens.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: GutterPump on April 03, 2017, 12:56:49 PM
Updated the preview with something a little more usable. It's color (though still low-res and non-realtime) and only drops to grayscale when recording speed becomes critical.

DAMN ! This update is very promising. I did some tests and yes it's more comfy for framing.
Thanks you for all these exciting features A1ex and all the team.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hyalinejim on April 03, 2017, 02:31:45 PM
OK, I'm pretty sure that the 14bit lossless compression varies with ISO.

In crop_rec 3K (3072 x 1286 ) I can get continuous recording at ISO 100

But if I whack the ISO up to 6400 I only get eleven seconds.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Lars Steenhoff on April 03, 2017, 03:08:39 PM
that makes sense because the more noise of 6400 iso would result in less compression
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kharak on April 03, 2017, 03:09:30 PM
Happy Aprils fools day :)

How do I go about processing these mlv files with the mlv_dump included? I can't figure out how to spit dng's out
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hyalinejim on April 03, 2017, 03:17:46 PM
I use this method on Windows:

http://www.magiclantern.fm/forum/index.php?topic=10526.msg178538#msg178538
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Launch on April 03, 2017, 03:31:14 PM
I salute the dedication but I wish squeezing every last pixel of out of the camera would be secondary to allowing audio with simple RAW. I would trade every last improvement for 1080p RAW with audio. For some reason, this worked for me once, but it just randomly stopped. It used to create a WAV file along with the RAW file.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: squig on April 03, 2017, 03:35:58 PM
100 ISO - black screen ~49%
6400 ISO - black screen ~60%
6400 ISO - busy scene, fast movement ~69%
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Levas on April 03, 2017, 03:45:18 PM
Ah that explains a lot.
High iso less compression, low iso high compression.

So for low iso shooting, compressed 14 bit is more efficiënt then shooting in 10 bit
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kharak on April 03, 2017, 03:48:03 PM
I use this method on Windows:

http://www.magiclantern.fm/forum/index.php?topic=10526.msg178538#msg178538

Thank you, exactly what I was looking for.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 03, 2017, 04:51:06 PM
I need to figure out what resolution will give the card enough headroom to write at that resolution continuously at least 90-95% of the time

Latest update estimates compression ratio on the fly, from the current scene and exposure settings. Also a few minor bug-fixes.

To get the best possible compression, all you need is to set ISO to maximum, shutter to 1/FPS, aperture wide-open and use a bright scene. You'll get close to 35% (http://www.magiclantern.fm/forum/index.php?topic=18443.msg181620#msg181620) (apparently the limit for this codec, or maybe just for Canon's implementation).

Generally, if pixel values are predictable (from their neighbors), the image will compress well. That usually means low noise levels (low ISO, shadows, fully clipped highlights) and few details (large out-of-focus and/or low-contrast areas). Note that absolute noise values are higher in highlights, and the SNR is higher too, so exposing to the right will give higher file sizes (but that shouldn't stop you from using it).

Fast motion probably gives better compression, because of motion blur (didn't test, but that's my understanding with this codec).

Invalid image data (that includes e.g. 14-bit raw data interpreted as 10-bit, or pure noise, or machine code if you want) will give compressed output larger than initial size. In current implementation, this gives memory overflow. For some unknown reason (alignment issues? no idea), setting a maximum size for compressed output causes camera lock-up at certain resolutions, with no apparent logic (see this commit (https://bitbucket.org/hudson/magic-lantern/commits/903c4828d514d3a8f2e3bced77318acdd8fffc86); minimal example here (http://a1ex.magiclantern.fm/2017Apr01/crash.c)).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: limey on April 03, 2017, 07:10:20 PM
For 1.23 only? I don't have a crop mode option in 1.13.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hyalinejim on April 03, 2017, 07:17:20 PM
Did you get the latest experimental build, and load the crop_rec module?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 03, 2017, 07:39:02 PM
I wonder if there is a potential for ordinary resolution recording but with higher frame rates.
Something like 100 FPS ?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: etienne on April 03, 2017, 10:13:35 PM
One question to the developpers do you know if the 3 things below would be possible one day:

 1/ 1080x1080  (from 3x3 bining using the 3240x3240 sensor size). for a big anamorphic s65 lens.

 2 / in the same way of the 14 bit losless mjpeg compression is an 8 bit mjpeg compression possible? so we can do low file size and get the real sharpness the 5d3 can offer (clean HDMI recorder on prores does not get any better sharpness than the in camera h264 file - whereas the raw files is always resolving much more sharpness)

 3/ finally how far would you say you are from a working preview for the mode 3K, UHD, or zoom x5 and is it even possible?

Also you probably have thought this but the basic full sensor preview with blacktape on the sceen could make it and if the 1:1 sensor readout cannot show original preview then maybe a x3 preview instead of the x5. Would not be perfect but would definitly make it usable.

Thanx for the work it's a dream. Where is the TIP JAR already?

Bests.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: magnesiumnitrate on April 04, 2017, 12:15:55 AM
I started testing 14 bit lossless with 1080P 24 because thats what I use the most. File sizes are much smaller, thats nice. Something funky is happening though. Using the histogram overlay with 80% or more of my exposure on the right half I'm getting horribly, horribly underexposed results in davinci. The live view looks great, histogram looks great, but then it hits davinici and the footage looks awful. Not sure what to make of it, but thought you guys might want to know.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Filipe Samora on April 04, 2017, 01:04:24 AM
So after a few hours playing around with A1ex's second release of the 1.1.3 firmware, here's what I'm getting with 14bit lossless option:

3520 x 1320 = max. 1314 frames - ISO within 200-400 range - scales up to UHD with no perceived loss of quality

4096 x 1862 = max. 507 frames @ 12.5 fps - Not sure how useful this would be for landscape work or slow shots. Again ISO within 200-400 range.

Previews: At the moment, not fantastic but definitely better than a black screen. I was able to rack focus with a moderate degree of success.

Camera crashes persistently when using 50/60p modes.

I'm using Komputerbay cards 64GB 1066x

IQ, even shooting at home with crap light, is just beyond words.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kharak on April 04, 2017, 01:48:08 AM
This is mighty fun to play with,

I was able to shoot up to 19 seconds at 3504x1312 at regular crop with Framed preview and Compressed Raw 14 bi, lots of movement, I was panning slowly downwards. But I only got so much recorded this one time, I never achieved the same length of recording again.

Is there a way of centering the crop? It looks weird not being centered.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hyalinejim on April 04, 2017, 02:30:09 AM
2 things I learned:

1. If you half press shutter button while recording the preview stays in colour rather than alternating between colour and greyscale. Frame rate still lags but it makes things a hell of a lot easier to know what's going on in front of the camera.

2. Filming the moire pattern of a computer screen's pixel array and refresh rate will absolutely kill the compression and stop the recording, even at lower resolutions!


Is there a way of centering the crop? It looks weird not being centered.

It should already be centered. Are you loading the crop_rec module of the latest experimental build and then selecting 3K or UHD? Framing is off in 5x but should be centred in these modes. Sometimes 3x is shifted up for me. I just restart the camera and it's fine.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: bobbyshots2 on April 04, 2017, 03:23:54 AM
Why doesnt the latest package have the mlvrec 2.0 module .. its only has light. for 5d3123.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: bobbyshots2 on April 04, 2017, 04:00:02 AM
Hi,
Ive been testing the latest build (April 03)for 5dIII123.
So far what i noticed there is no mlv snd module
or mlv_rec. Also there is no gray scale preview.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: limey on April 04, 2017, 04:09:42 AM
Yeh I figured it out. I guess I was blind while looking at the modules or I wrote the files to the wrong sd card from my lexar workflow hub.

Getting pink frames and crashes in the 50fps modes.

Getting unreadable files in some other modes like the 3k and UHD. It was just an initial test, I will test more tomorrow. I am using 1066x 256gig lexar cards. However, they aren't formatted and have a lot of shots on them. I will format one and see what difference that makes.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hindra on April 04, 2017, 06:28:06 AM
Shooting with the Canon 5d3, 1.2.3 firmware and using the latest experimental build(April 3rd 2017) with Lexar 128gb 1066x cf card.

1080p 48FPS seems to be working (It seemed to have recorded fine, but I must have deleted the file.)
3520x1320 24FPS(ISO 100-200 iso seems to be continuous and at a 5x crop mode) ISO 400 recording stops after a few seconds. In no crop mode preview was colorful noise. in 5x crop preview worked great. I recorded 1510 frames and I ended it.
3584x1320 24FPS I recorded 1443 frames and I ended it.
1080p 60FPS(buggy, stops recording quickly) gray preview was glitchy and lagged.

3k ([email protected]) I got 66 frames.
UHD ([email protected], couldnt get 3840x1600) I got 66 frames as well. It states its 120mb/sec, this might require the lossless compression with 10bit?
Both 3k and UHD frames are good, used MLV_DUMP from the page I downloaded the firmware to extract these.

I am also very tired, I will try and play around with them some more.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: budafilms on April 04, 2017, 08:32:22 AM
@A1ex, you are my king.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 04, 2017, 09:12:19 AM
Ah, a little late to yet another party once again and after playing around in the past 48 hours or so I've finally come to a conclusion to be able to confirm that 3584x1320 in crop_rec (3K 1:1) can be recorded continuously if shot in between ISO 100-200 and more often than so in 400 depending on scene.

This was all tested with both 128gb 1066x & 256gb 1066x CF cards from KomputerBay on two separate 5D3's running 2017April03.5D3123 builds.

(https://c1.staticflickr.com/3/2828/33014736263_11f475c323.jpg) (https://flic.kr/p/Sip9SK) (https://c1.staticflickr.com/4/3929/33698823611_26f5ec5f2d.jpg) (https://flic.kr/p/TkRhdc)

Seems that ISO 800 and higher doesn't get along well with 14-bit lossless compression which makes sense as previously stated by others in here.

Here's an example of what it looks like upon running this beautiful hack via LiveView from 5D3 below and watch what happens when I push it up to ISO 800 (and higher) towards the end:


I was able to convert all 14-bit lossless files with ease from @Danne's latest cr2hdr.app_4k (includes the required certain mlv_dump which was modified directly from @a1ex) so I am more than flabbergasted by the fact that all of this is storming down on us like there's no tomorrow.

Love it and will continue to play with and hopefully run into some more bugs! 

Also I can confirm that if I point the camera at a computer screen while recording and it'll crash once I focus upon the pixealated monitor (indeed this is quite strange but good find @hyalinejim) and I am actually loving the half shutter press for FULL COLOR LIVE DISPLAY or whatever that's called (plus it zooms in a bit more to be sure everything looks spot on) before recording which is very handy.

Thank you @a1ex for yet another petrifying update (in a good way!) and don't ever change who you are! Ha. 8)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: beauchampy on April 04, 2017, 09:36:31 AM

Here's an example of what it looks like upon running this beautiful hack via LiveView from 5D3 below and watch what happens when I push it up to ISO 800 (and higher) towards the end:

https://vimeo.com/211439899



I cant seem to get that lovely liveview on 113. Mine is low fps, desaturated and choppy.
Running crop_rec_4k.2017.Apr03.5D3113 and I copied your settings..
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Levas on April 04, 2017, 09:54:55 AM
I was just wondering...

Who has twixtor or something alike and creates some stunning 6K - 24p video out of that 7 fps full resolution silent video mode  ;D 8)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 04, 2017, 09:58:25 AM
I cant seem to get that lovely liveview on 113. Mine is low fps, desaturated and choppy.
Running crop_rec_4k.2017.Apr03.5D3113 and I copied your settings..

Hmmm then perhaps I should downgrade to 113 and try to reproduce this?

Did you press 5x zoom after using ML menu or before? Sometimes it goes into a weird phase which may seem like having its own quirks but a simple restart (on/off) of the camera would help at least it does oh my end.

But it could very well be a bug for 113 and I'll let you know once I've downgraded.

Thought not too long ago @a1ex made a pull request for both firmwares (113 & 123) to be merged together after each update or does this only apply to the Nightlies and not experimental?

Well I'm actually tempted to check out the good old speedy 113... Just been using 123 as of late for HDMI output on gig's.

Actually this one trick may be worth to try (Thanks @dfort for the tips) perhaps reset your settings to default (from Canon menu) not the ML version and let me know if that helps either way.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 04, 2017, 10:13:46 AM
Hi,
I don't know if it will be useful or not...

Testing latest build (April 3th crop_rec) on 5D3 113, for now only this mode because is which I find more useful for me: 1920x872 50p 3x3 (aspect ratio 2.20:1) 14bit lossless.

Yesterday I was able to record without problems in continuous with normal preview, in color.

Today, with the same settings as yesterday, it has been impossible to boot the camera. In the SD card there are text files called ASERT, one for each boot attempt. In them the following appears:
ML ASSERT:0at mlv_lite.c:1935 (compress_task), task compress_tasklv:1 mode:3
Magic Lantern version : crop_rec_4k.2017Apr03.5D3113
Mercurial changeset   : 7b0cb9532629 (crop_rec_4k) tip
Built on 2017-04-03 14:54:32 UTC by [email protected]
Free Memory  : 173K + 3807K


If I change SD card for another with 2017Feb11th 5D3113 (crop_rec proxy) all works again.

Hope it helps!!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: squig on April 04, 2017, 10:14:03 AM
Are you guys running ML off an SD card?

The best I can do @ 100 ISO is 3.3K 2.39:1. ML is on the CF card.

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 04, 2017, 10:28:34 AM

Quote
Hi,
I don't know if it will be useful or not...

Testing latest build (April 3th crop_rec) on 5D3 113, for now only this mode because is which I find more useful for me: 1920x872 50p 3x3 (aspect ratio 2.20:1) 14bit lossless.

Yesterday I was able to record without problems in continuous with normal preview, in color.

Today, with the same settings as yesterday, it has been impossible to boot the camera. In the SD card there are text files called ASERT, one for each boot attempt. In them the following appears:
ML ASSERT:0at mlv_lite.c:1935 (compress_task), task compress_tasklv:1 mode:3
Magic Lantern version : crop_rec_4k.2017Apr03.5D3113
Mercurial changeset   : 7b0cb9532629 (crop_rec_4k) tip
Built on 2017-04-03 14:54:32 UTC by [email protected]
Free Memory  : 173K + 3807K

If I change SD card for another with 2017Feb11th 5D3113 (crop_rec proxy) all works again.

Hope it helps!!


I've erased all in SD card and copied clean April3th Crop_rec_4K build.

All works ok again like yesterday.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 04, 2017, 11:01:33 AM
Same again:

ML ASSERT:0at mlv_lite.c:1935 (compress_task), task compress_task
lv:1 mode:3
Magic Lantern version : crop_rec_4k.2017Apr03.5D3113
Mercurial changeset   : 7b0cb9532629 (crop_rec_4k) tip
Built on 2017-04-03 14:54:32 UTC by [email protected]
Free Memory  : 177K + 3923K

Need to pull battery, and then camera doesn't boot anymore :(
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hyalinejim on April 04, 2017, 11:20:55 AM
Are you guys running ML off an SD card?

The best I can do @ 100 ISO is 3.3K 2.39:1. ML is on the CF card.



Yes. I was under the impression this frees up more bandwidth for recording to CF. Also, when you change CF cards your settings don't change.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: squig on April 04, 2017, 12:35:16 PM
OK cool, I'll give it a shot.

Seeing as we're all madly trying to squeeze every last MB out of the oldfella to get closer to 4K, would it be possible to add card spanning to these experimental builds?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: garry23 on April 04, 2017, 01:47:28 PM
I just downloaded the latest experimental build, but didn't get far.

I would welcome a little help, as I'm keen to test, but, I don't do video. That is I'm not up to speed on MLV etc.

If there is someone out there who has the time, maybe they would create a 1 page 'here is how to test', i.e. The steps.

In particular, how do I set up, run and process the full Res 'photo mode'. Once I know how to get things running, I can test.

Cheers

Garry
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 04, 2017, 05:52:37 PM
So what is the bottle neck for transfer speeds in the 5D3? Is it still the card reader in the camera? I thought cameras card reader was limited even with really fast cards?
Is spanning the trick for UHD?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 04, 2017, 05:55:45 PM
ML ASSERT:0at mlv_lite.c:1935 (compress_task)

This one indicates frame corruption, and should also print this (unless obscured by other messages):
Code: [Select]
                printf("\nCompressed size higher than uncompressed - corrupted frame?\n");
                printf("Please reboot, then decrease vertical resolution in crop_rec menu.\n\n");

1080p 60FPS(buggy, stops recording quickly) gray preview was glitchy and lagged.

Took a quick look - after the latest fixes (see below), the following resolutions are continuous (1.2.3, lossless ~55%, Preview: Frozen LV, with grayscale preview):

- 1920x800 60fps
- 1920x960 50fps
- 1920x1080 45fps

1920x1080 48 FPS is a bit unreliable (corrupted frames with Global Draw on, but appears OK with it off). This mode is really pushed to the limit (and that's the reason I've also added a 45-fps option). Feel free to play with the knobs from crop_rec submenu, if you think you can find the sweet spot without corrupted frames.

I would expect the 1.1.3 firmware to have a little more headroom (maybe even allow a few more pixels), but didn't test it. You can fine-tune vertical resolution from the crop_rec submenu (and if you have corrupted frames, decrease it from there).

New build posted. Short summary:

- fixes for high-FPS modes with Frozen LV (first frame was corrupted, which didn't play nice with lossless compression)
- continuous full-res recording with lossless compression (about 4-5 FPS, depending on ISO and scene complexity)
- print current FPS and rolling shutter in the FPS override menu (visible from main Movie menu tab)
- two experiments with a small risk:
   - single-buffering (risk of tearing, alongside with graphical checks)
   - use-after-free for SRM memory (to fix half-shutter quirks)
- minor fixes here and there.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: GutterPump on April 04, 2017, 06:11:34 PM
Thanks for the update A1ex, today it's my birthday, it's a nice gift from you :)

I believe this has already been said, but the best way to use the lossless 14bits is to forget to shoot in ETTR.
Do you think there is way to take the control on the compression codec ?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 04, 2017, 06:25:18 PM
In particular, how do I set up, run and process the full Res 'photo mode'. Once I know how to get things running, I can test.

Simplest way:
- crop_rec.mo, silent.mo
- crop mode: full-res LV; silent mode: simple, any file format
- press half-shutter
- for preview, also load mlv_lite.mo, enable raw recording and global draw.
- caveat: you can only review 14-bit regular DNG (pic_view) or MLV (mlv_play); for 10-bit full-res MLV, raw_twk runs out of RAM

Continuous full-res recording at 4-5 FPS:
- crop_rec.mo, mlv_lite.mo
- crop mode: full-res LV; raw recording: max resolution, lossless compression
- FPS override: 4 or 5 FPS, low light (important)
- press record
- mlv_dump foobar.mlv --dng

Fast half-shutter trigger (with negative shutter lag):
- crop_rec.mo, mlv_lite.mo, configured as above
- raw recording: enable REC trigger (half-shutter: pre only; without pre-recording enabled, that means only 1 frame)
- no need to use FPS override, as we are only saving the frames triggered by half-shutter
- start recording
- press half-shutter every now and then to capture a frame
- stop recording

Capture unpredictable action (animals, kids etc):
- crop_rec.mo, mlv_lite.mo
- crop mode: full-res LV or 4K half-FPS
- raw recording: max res, 10-bit (here, pre-recording is a bit confused by variable frame sizes, so let's not be adventurous)
- pre-recording: 1 or 2 seconds (actually "up to")
- half-shutter trigger: "hold" (which will pre-buffer up to about half of available RAM), or "pre only" (which will pre-buffer all the available RAM)
- start recording, press half-shutter etc
- caveat: preview is not working at all with 10-bit (try 14-bit)
- to review in camera: 4K 10-bit can be played back with mlv_play + raw_twk, full-res 14-bit with mlv_play only.

I believe this has already been said, but the best way to use the lossless 14bits is to forget to shoot in ETTR.
Do you think there is way to take the control on the compression codec ?

Very little. (http://www.magiclantern.fm/forum/index.php?topic=18443.msg181620#msg181620)

I wonder if there is a potential for ordinary resolution recording but with higher frame rates.

First post:
Quote
* 1920x960 @ 50p (both 1:1 crop and full-frame - 3x3 pixel binning)
* 1920x800 @ 60p (same as above)
* 1920x1080 @ 45p and 48p (3x3 binning)

In particular, 1080p48 is quite pushed to the limit, which is why I've also included a 45p preset (hopefully a bit more reliable).

1/ 1080x1080  (from 3x3 bining using the 3240x3240 sensor size). for a big anamorphic s65 lens.

Already possible on regular nightly.

Quote
2 / in the same way of the 14 bit losless mjpeg compression is an 8 bit mjpeg compression possible?

Yes... (http://www.magiclantern.fm/forum/index.php?topic=2803.msg13469#msg13469)

Quote
so we can do low file size and get the real sharpness the 5d3 can offer (clean HDMI recorder on prores does not get any better sharpness than the in camera h264 file - whereas the raw files is always resolving much more sharpness)

... but it would simply compress Canon's YUV422 buffer (with their color science, picture styles, sharpness etc).

Quote
3/ finally how far would you say you are from a working preview for the mode 3K, UHD, or zoom x5 and is it even possible?

Far, far away. The trick I've used to center the 5x zoom preview in the regular (non-4K) crop_rec branch didn't work for the new modes :(

Also, I have no idea how to configure image scaling. Adtg_gui doesn't reveal any difference between x5 and x10 zoom modes (both use identical raw buffer configurations - just the preview differs).

I can try to include the centered x5 zoom mode (which you can already use in the "classic" crop_rec builds), if that helps.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Teamsleepkid on April 04, 2017, 06:44:01 PM
Anyone trying to turn the eos m into the blackmagic cinema camera? 2.5k possible? Maybe... hopefully...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: togg on April 04, 2017, 06:51:03 PM
This is amazing for movie still photographers!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: garry23 on April 04, 2017, 07:25:36 PM
@a1ex

Many thanks for the simple insight, for a simple still photographer  ;) That is:

Quote
Simplest way:
- crop_rec.mo, silent.mo
- crop mode: full-res LV; silent mode: simple, any file format
- press half-shutter
- for preview, also load mlv_lite.mo, enable raw recording and global draw.
- caveats:
  - preview doesn't work

Because of the preview limitation, I simply composed with crop_rec and silent switched off, used ETTR, at 0%, and switched them back on after the ETTR.

Grabbed a DNG which I inspected in LR.

Fantastic!!!

OK, I know you know the limitations at the moment, eg no EXIF data, but what I step forward for silent full res photography, ie in those churches I go to etc.

One thing I have noticed is that the ETTR exposure at 0% highlights (SNRs off/0) does show over exposure. Not sure if this is real or important at this stage of the developments.

As I said, I'm keen to help test the still photography side of the experiment: I'll stay clear of the MLV world...for now.

Cheers and, once again, thanks for this new step forward for ML.

Garry
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: lostfeliz on April 04, 2017, 08:57:20 PM
I'm having trouble using mlv_dump to decode the compression. Any help appreciated. (sorry, I know this is a development forum. what I am I doing here if I can't use Terminal properly?) :)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 04, 2017, 10:13:28 PM
Assuming you're on OS X -- Have you even tried @Danne's latest cr2hdr.app which includes the modified mlv_dump yet @lostfeliz?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: etienne on April 04, 2017, 10:36:40 PM
@a1ex

Hello thanx for taking the time to answering the question this is highly appreciated. i'm on the curve to go to URSA 4.6K but this new fonctions let me think my olds 5d3's have still some life to live.

regarding the fact that mlv mjpeg 8 bit compression
Quote
would simply compress Canon's YUV422 buffer (with their color science, picture styles, sharpness etc)


Are you absolutely sure regarding the sharpness. I'm asking because this is a fact that the clean hdmi as absolutely zero additionnal sharpness from the h264 files i have tested this thousands times. But the raw definitly resolves way more sharpness similar to what produces an A7sii, gh4 or gh5 in HD so i was sure that the 5d breaks something in between what you catch with rec_mod and what you get as h264 or hdmi out.

Now what i'm undestanding from your saying is that the compress Canon's YUV422 buffer is what is sent to the hdmi out right? If it's the case it's useless for sure but can it be something else. Can you think of any other way to compress to something like 15~25% even if it means we loose all the color depth from the raw but just keep that sharpness from it. I'm insisting on this because Personnally I would be so much be happy with an MLV without any color depth additional value but which would just have the real sharpness we can get from the current raw and that could allow like 60~90 minutes on a 128GB card.

Also I want to congrats on the slowmotion mods all options are awesome i just did a 1080 60P in 2:35 flawlessly. As mentionned the 1080 48P as a few corrupted frames and its preview is also a little off towards the down of the image compared to the canon preview but appart from that we could say this is 99% usable.

Finnally. The 3K Mode and UHD are absolutely awesome. I personnally don't see how we could use over 3360x1528 with spanning and sound in the future so any preview that would be x2 would just be the ultimate thing to make this usable.

I undestand that for now it's complete unknow hope someone will find a solution one day otherwise in regards of you're sayinf:
Quote
I can try to include the centered x5 zoom mode (which you can already use in the "classic" crop_rec builds), if that helps.
Sure It's definitly better than nothing to have access to canon x5 center preview again instead of pink frame eventhough it will not help with framing but it will help to pull focus correctly durin shooting.

Bravo again keep the good work. Looking very forward to integration to MLV2 with spanning and sound enabled...

Good Luck.


 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hyalinejim on April 04, 2017, 10:57:37 PM
Hi a1ex, thanks for the new build.

I noticed that the preview framing in UHD seems to be too far up. I tried fiddling with the crop options, to no avail.

Also, for anybody concerned about low light shooting, it looks like I can get continuous 16:9 with around 2.4k horizontal resolution at ISO 3200.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 04, 2017, 11:24:03 PM
Can you attach a screenshot + DNG?

(I did the same, but couldn't notice any mismatch...)

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: photofreak126 on April 04, 2017, 11:46:15 PM
Hi! What great stuff is happening here!

One question: I try to avoid raw because of inconvenience.

The 1:1 1920 and 1:1T work awesomely fine for me with crop_rec enabled and raw disabled. It would be so good to have 1920 60p (1920x800) in non raw mode but if I choose either 720 60p or any other Canon frame rate/resolution combo, I just get the standard rates.... Is there a trick to it or is the crop_rec just for ml_raw ??
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hyalinejim on April 04, 2017, 11:54:01 PM
So this is full frame mode, just to show the framing:
(https://s13.postimg.org/8h05h652f/20170404_224558.jpg)

And this is the DNG from UHD mode:

https://www.dropbox.com/s/xoxmrxn0hcj19qo/M04-2243_000000.dng?dl=0
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 05, 2017, 12:08:23 AM
You mean, the recorded area is not centered? That's probably true, as it can only be adjusted in rough increments. But the preview should be OK.

You can fine-tune the centering (in both directions) from the crop_rec submenu. Start values for vertical centering:
Code: [Select]
            case CROP_PRESET_UHD:
                cmos_new[1] =
                    (video_mode_fps == 24) ? PACK12(4,9)  :
                    (video_mode_fps == 25) ? PACK12(4,9)  :
                    (video_mode_fps == 30) ? PACK12(5,8)  :
                    (video_mode_fps == 50) ? PACK12(12,9) :
                    (video_mode_fps == 60) ? PACK12(13,9) :

Then, you can check with raw_diag (Downloads -> Modules) using the OB zones tool, to see whether the optical black areas are clean and useful data is present in all edges.

If you (or anyone else) finds better values, I'll include them.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 05, 2017, 01:00:40 AM
WOW!!!!!  :o just saw this new development are you serious 4K on the old 5D Mark III.  Like I've said before Magic Lantern Team never sleeps, thank you @a1ex and the whole Magic Lantern Team for those awesome new features, and previously impossible features  :D
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hyalinejim on April 05, 2017, 01:14:35 AM
Actually, the preview is off as well as the file. In 3K mode framing is correct in preview:

(https://s11.postimg.org/xpb3jor03/20170404_233407.jpg)

But in 4K it's totally off (in the last build it was fine)

(https://s14.postimg.org/n9x110mb5/20170404_233433.jpg)

I tried altering CMOS[1] lo hi values. At lo 10 the image was centered vertically but with a weird overlay on half of the screen:

(https://s10.postimg.org/e8armmym1/20170404_235712.jpg)

It's a transparent overlay of what's actually beneath the frame. I tried playing with different values for CMOS[1] lo but with no luck. In the previous build this problem cropped up (excuse the pun!) from time to time, but I could restart the camera and it was back to normal. With this build UHD is vertically shifted whereas 3K is correct.

TANGENTIAL SUGGESTION: If there's some way to reliably control the positioning of the crop area, this is a great way to turn a wide angle lens into a digital shift lens, for those interested in architectural stuff and precise composition.

Another quick suggestion: It would be nice to have more control of the vertical resolution, if that's possible. Let's say I wanted to shoot in 4:3 using a 2x anamorphic. The max res I can get is 2048 x 1536, even though my card could handle a higher resolution.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: squig on April 05, 2017, 01:50:34 AM
I got that overlay when I messed with the 1920 tall positioning on the 1st or 2nd build.

I shot standard 1080p raw vs 2.8K raw at the same ISO and aperture, taking a few steps to get close to the same framing; the 1:1 2.8K crop appears to be about half a stop brighter than the 3x3 binned 1080p. Somebody else might wanna test that to confirm.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: squig on April 05, 2017, 02:46:38 AM
Getting back into the ML menu is quite tricky on the Apr04 build.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: lostfeliz on April 05, 2017, 03:18:45 AM
@DeafEyeJedi Thanks! I didn't know it'd been updated to handle the compressed MLVs.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: mk11174 on April 05, 2017, 06:16:22 AM
So, any hints on how i can port any of the high resolution options to 700d, or are the high res options just not going to work for the lower end models like the rebels?

I looked around and read about needing to look at fps timers, but also see there's mem_engio_write address that has to be found, no idea how to find that one?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 05, 2017, 07:35:23 AM
Getting back into the ML menu is quite tricky on the Apr04 build.

?! (http://www.chiark.greenend.org.uk/~sgtatham/bugs.html)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 05, 2017, 07:46:23 AM
@a1ex Can you please point me to a guide on how to use mlv_dump for mac can't seem to get it to work with the latest development. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 05, 2017, 07:50:51 AM
mlv_dump --dng INPUT.MLV
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: squig on April 05, 2017, 08:10:41 AM
?! (http://www.chiark.greenend.org.uk/~sgtatham/bugs.html)

After recording, I have to restart to get back into the ML menu.

Are you getting the Canon spotmeter appearing during recording? Busy's still showing too.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 05, 2017, 08:46:05 AM
Can't reproduce.

Canon's focus box appears for 1-2 seconds when starting recording, then again when stopping. No BUSY message (I was getting it with previous builds). Menu works fine, both before and after. Tried 3K 24p and 1080p45, starting from ML defaults. Also tried frozen LV.

If the link wasn't obvious: http://www.chiark.greenend.org.uk/~sgtatham/bugs.html
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 05, 2017, 08:52:39 AM
@Danne Can you breakdown the process for me.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 05, 2017, 08:56:50 AM
http://www.magiclantern.fm/forum/index.php?topic=19300.msg182376#msg182376
Breakdown?

If on mac
1 - Drag mlv_dump to terminal
2 - Print --dng drag your mlv file after the word --dng in terminal and press enter.

If your version of mlv_dump says permission denied do following:

Open up terminal and print chmod u=rwx and drag mlv_dump terminal and hit enter then redo steps 1 and 2.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Teamsleepkid on April 05, 2017, 09:03:34 AM
cr2hdr for mac is very easy. even I can use it. read that it has new mlv dump. id try that.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: elieka69 on April 05, 2017, 09:10:36 AM
frame and block size mismatch:0xcba800 0x20 0x88bc00

Sent from my SM-N920I using Tapatalk

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 05, 2017, 09:13:43 AM
@Danne Thanks a lot..
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: garry23 on April 05, 2017, 09:29:54 AM
In case anyone finds what I've done helpful, I offer the following as we experiment with this exciting new ML development.

This is only for the still photographers  ;)

As at the moment LV preview is not working, so composition and exposure setting are not possible, I've written the following simple script to toggle the crop and silent features on and off (only works in LV).

This way you can compose without the feature on, toggle to use the feature (I use RATE as the toggler), do a half shutter press to get your DNG (I use DNG for stills), then toggle back.

The only 'strangeness' is that after toggling you will need to go in and out of the ML menu (TRASH) to reset the shooting state. The bottom left of the ML menu will give you feedback, eg FLV.

Here is the simple script:

Code: [Select]
--[[
Simple script to toggle Experimental FRSP on and off
Garry George April 2017
http://photography.grayheron.net/
--]]

-- Declare some globals
mod_button_1 = KEY.RATE -- key to modify
toggle_state = true

function test4mod(arg)
if lv.running then
if XFRSP_menu.submenu["Turn On/Off"].value == "On" then -- be prepared to toogle X-FRSP state
if arg == mod_button_1 then  -- toggle
if toggle_state == true then toggle_state = false else toggle_state = true end
if toggle_state then
menu.set("Movie", "Crop mode",8)
menu.set("Shoot", "Silent Picture",1)
else
menu.set("Movie", "Crop mode",0)
menu.set("Shoot", "Silent Picture",0)
end
return false -- steal key press
else
return true -- do nothing
end
else
return true -- modified key works normally
end
end
end

event.keypress = test4mod

XFRSP_menu = menu.new
{
parent = "Shoot",
name = "X-FRSP",
help = "Simply toggles the X-FRSP state",
depends_on = DEPENDS_ON.LIVEVIEW,
submenu =
{
{
name = "Turn On/Off",
help = "Switches the script on/off",
choices = {"On","Off"},
}
}
}

As I say, until the previews (hopefully) get working, this little 'hack' will allow you to experiment with still experiment full res silent.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: squig on April 05, 2017, 09:59:57 AM
Can't reproduce.

Canon's focus box appears for 1-2 seconds when starting recording, then again when stopping. No BUSY message (I was getting it with previous builds). Menu works fine, both before and after. Tried 3K 24p and 1080p45, starting from ML defaults. Also tried frozen LV.

Things started to get a bit screwy yesterday after I put ML on an SD card. I'm thinking doing the install with an experimental build may not have been a great idea. I'll do it over with a nightly.

EDIT: It was sticky half shutter. All good now.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: ju64 on April 05, 2017, 10:10:55 AM
Hi,

Testing the last build (magiclantern-crop_rec_4k.2017Apr04.5D3123) on 5D 123 and after few tests camera freezed.
I had to remove the battery and now the camera doesnt' boot as usual.
The LCD stay black, back buttons don't respond, video mode no longer works, camera can only take pictures.

On the SD card I had crash log :

Code: [Select]
ASSERT: hLvJob->hJpegMemSuite
at ./Epp/Vram/VramStage.c:891, task Epp
lv:1 mode:3


Magic Lantern version : crop_rec_4k.2017Apr04.5D3123
Mercurial changeset   : e7e1fdb85f6b (crop_rec_4k) tip
Built on 2017-04-04 15:42:08 UTC by [email protected]
Free Memory  : 168K + 3151K

And assert log :

Code: [Select]
ML ASSERT:
PROP_LEN(80040002) correct:0 called:4
at ../../src/property.c:342 (prop_request_change), task ml_init
lv:0 mode:0


Magic Lantern version : crop_rec_4k.2017Apr04.5D3123
Mercurial changeset   : e7e1fdb85f6b (crop_rec_4k) tip
Built on 2017-04-04 15:42:08 UTC by [email protected]
Free Memory  : 176K + 3964K

I went back to previous working version (compressed_raw.2017Apr01.5D3123), same symptoms and I had this crash log :

Code: [Select]
ASSERT: pStruct->Data
at GuiProperty.c:1020, task PropMgr
lv:0 mode:0


Magic Lantern version : compressed_raw.2017Apr01.5D3123
Mercurial changeset   : 706183bf6459 (compressed_raw) tip
Built on 2017-04-01 11:58:46 UTC by [email protected]
Free Memory  : 170K + 3753K

And this assert log :

Code: [Select]
ML ASSERT:
PROP_LEN(80040002) correct:0 called:4
at ../../src/property.c:342 (prop_request_change), task ml_init
lv:0 mode:0


Magic Lantern version : compressed_raw.2017Apr01.5D3123
Mercurial changeset   : 706183bf6459 (compressed_raw) tip
Built on 2017-04-01 11:58:46 UTC by [email protected]
Free Memory  : 192K + 3983K

Sometime I get Err 80 too.

Any idea of what happen to my camera ?  :-[
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: seanvdeusen on April 05, 2017, 10:26:10 AM
Idk if anyone ran into this issue like I did. But if you're having trouble with dumping the files or importing once converted here's what i did.

1. Open Notepad and type in

@echo off
for %%a in ( %* ) do ( md "%%~na" 2>nul )&( "INSERT FILE LOCATION HERE" -o %%~na\%%~na_ --dng %%~na.MLV )

ALSO don't type in INSERT FILE LOCATION. Thats where you place the location of the folder you're working in. :p

2. Save that file and what ever you name it (I NAMED MINE DUMP) be sure to add .cmd at the end of it. Which will make it a CMD file.

3. Place the MLV_DUMP file into the same folder as the the previous file we just made.

4. Drag you're MLV file to that location and then drag it over the CMD you made.

5. You should now have a folder with a bunch of DNG files.

6. If you can't open the dng files and get errors in let's say After Effects. You need to deleted the The first DNG file in that folder. It's the one with just zeros at the end of it.

I'm use to magic lantern MLV files working with my normal programs. So if this is already old news i'm sorry for posting. I did however see a few questions regarding this issue. Some are getting pink dots or weird rainbow-ish frames.


Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 05, 2017, 11:03:59 AM
PROP_LEN(80040002) correct:0 called:4
at ../../src/property.c:342 (prop_request_change), task ml_init

This one sounds bad. My advice:
1) make a backup copy of your ROM if you haven't already (it's on the card, under ML/LOGS)
2) start the camera without cards and try to clear Canon settings

If that doesn't help, we'll have to investigate what happened.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: ju64 on April 05, 2017, 11:22:38 AM
Thanks a1ex

1) Backup of ROM is already done
2) How can I clear Canon settings ? LCD doesn't works...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 05, 2017, 11:25:54 AM
Got it, let's chat on IRC for further troubleshooting.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 05, 2017, 11:44:50 AM
Quote
Quote from: pc_bel on Yesterday at 10:13:46 AM

    ML ASSERT:0at mlv_lite.c:1935 (compress_task)


This one indicates frame corruption, and should also print this (unless obscured by other messages):
Code: [Select]

                printf("\nCompressed size higher than uncompressed - corrupted frame?\n");
                printf("Please reboot, then decrease vertical resolution in crop_rec menu.\n\n");

Now with new April4th build, yesterday problems are gone. Testing 1920 50/60 3x3. Camera boots every time I turn on as expected... WAIT A MOMENT !!!

While I was writting this post with camera on, I've heard the camera disconnect live view and connected again. On screen find for a moment a error message. I didn't have time to read it, just "created a LOG file" or something similar.

This is the LOG:

Wed Apr  5 11:31:33 2017
      0:    15.149 [STARTUP]
K285 ICU Firmware Version 1.1.3 ( 6.9.4 )
      1:    15.177 [STARTUP]
ICU Release DateTime 2012.05.16 14:14:46
      2:    16.597 [PROPAD] PROPAD_CreateFROMPropertyHandle DRAMAddr 0x41744000
      3:    16.886 [SEQ] seqEventDispatch (Startup, 0)
      4:    16.930 [STARTUP] startupEntry
      5:   129.073 [SEQ] seqEventDispatch (Startup, 1)
      6:   129.148 [STARTUP] startupPrepareProperty
      7:   141.147 [SND] Seq LPC fin
      8:   142.415 [RSC] hMemoryQueue (0x6A0010) hStorageQueue (0x6C0012)
      9:   144.775 [RSC] MemoryStatusMasterResultCBR
     10:   188.740 [MRK] MRK_RegisterSpaceNotifyCallback
     11:   188.883 [MRK] MRK_RegisterSpaceNotifyCallback
     12:   188.935 [FM] FM_RegisterSpaceNotifyCallback
     13:   189.503 [JOB] InitializeJobClass (ID = 858, Num = 13)
     14:   196.161 [EM] emLockControl (TYPE_JOBSTATE = 0x0)
     15:   199.904 [JOB] InitializeInnerDevelopJobClass (ID = 858, Num = 16)
     16:   199.946 [JOB] InitializeMultipleExposureShootJobClass ( Num = 27,11 )
     17:   199.981 [JOB] InitializeMultipleExposureSaveAndEndJobClass ( Num = 13 )
     18:   200.020 [JOB] InitializeHDRShootJobClass ( Num = 16 )
     19:   200.053 [JOB] InitializeHDRSaveAndEndJobClass ( Num = 33 )
     20:   200.321 [TERMINATE] SHUTDOWN init comp
     21:   200.429 [TERMINATE] Abort init comp
     22:   202.132 [STARTUP] startupPropAdminMain : End
     23:   209.359 [FM] fmPrepare : Completed
     24:   209.511 [SEQ] seqEventDispatch (Startup, 2)
     25:   209.884 [SHTRD] [MNAV] MEMNAVI_Initialize
     26:   210.049 [SHTRD] [MNAV] MEMNAVI_SetMemoryMap: 1
     27:   211.874 [ENG] [ENGIO](Addr:0x41700000, Data:0x   44000)
     28:   247.456 [SEQ] seqEventDispatch (Startup, 3)
     29:   249.164 [MC] PROP_GUI_STATE 0
     30:   250.138 [MC] [MC] PROP_HEADPHONE_VOLUME_VALUE : 0
     31:   250.286 [MC] [MC] PROP_MOVIE_PLAY_VOLUME : 5
     32:   251.295 [MC] regist master CardCover
     33:   253.856 [PRP] M:98 F:97 L:0 P:32 T:7
     34:   254.010 [STARTUP] update coded version.
     35:   254.110 [STARTUP] internal version 06:09:04:98:97:00:32:07:3c:05:d0:d0:d0
     36:   259.367 [STARTUP] startupPrepareDevelop
     37:   262.974 [PROPAD] Done Rebuild Tune
     38:   264.107 [MAC] MAC_Initialize
     39:   264.479 [MAC] K_BOARDID4
     40:   264.521 [MAC] K_BOARDID3
     41:   264.555 [MAC] K_BOARDID2
     42:   264.586 [MAC] K_BOARDID1
     43:   264.757 [MAC] Key=0x0 Board=0x0 Body=0x734bbf81
     44:   277.644 [SHTP] Regist RPC Handler Master ALO
     45:   281.440 [PROPAD] PROPAD_CreateFROMPropertyHandle DRAMAddr 0x41b2c000
     46:   393.855 [MR] mvrChangeAckCBR : Video - Mode=0, Type=1, Rate=50, GOP=1
     47:   393.981 [MR] mvrChangeAckCBR : Sound Manual(C=2,R=48000,B=16)
     48:   394.008 [MR] mvrChangeAckCBR : AE_MODE_MOVIE(3)
     49:   394.024 [MR] mvrChangeAckCBR : ISO(104)
     50:   394.178 [MR] mvrChangeAckCBR : MOVIE_REC_VOLUME(L:20 R:20)
     51:   394.228 [MR] mvrChangeAckCBR : WindCut Off
     52:   394.333 [MR] TimeCode Base:11526,11,530583872,0
     53:   394.378 [MR] TimeCode Mode:1
     54:   394.424 [MR] TimeCode DropFrame Mode:0
     55:   394.468 [MR] TimeCode Disp Mode:0
     56:   394.503 [MR] PROP_FILE_PREFIX:8W1A
     57:   394.534 [MR] PROP_USER_FILE_PREFIX:PAU_,
     58:   394.561 [MR] PROP_USER_FILE_PREFIX:PAU,
     59:   394.588 [MR] PROP_SELECTED_FILE_PREFIX:PAU_,1
     60:   402.263 [LVCOM] LiveView Gain Table Ver0.4
     61:   403.395 [HPC] ReserveHPCopyChannel (1, 116)
     62:   411.083 [MR] encdrwrapHPCopyAsyncCBR
     63:   412.885 [LVGMT] PROP_TEMP_STATUS : STATUS_NORMAL
     64:   418.280 [CAPD] LiveView Gain Table Ver0.4
     65:   421.293 [LVCOM] InitializeLiveViewDefectDetection
     66:   421.798 [LVCOM] ExecuteDefectMarge Start
     67:   421.882 [LVCOM] ExecuteDefectMarge1[0x41dd9b00:61440]
     68:   423.164 [LVCOM] A:[59] B:[9040]
     69:   427.618 [LVCOM] ExecuteDefectMarge2[0x41de8b00:61440]
     70:   428.697 [LVCOM] A:[35] B:[7713]
     71:   432.633 [LVCOM] ExecuteDefectMarge3[0x41df7b00:32768]
     72:   435.697 [LVCOM] ExecuteDefectMarge End
     73:   436.838 [LVEPP] [REMO] Jpeg 16:9 PosW: 0 PosH: 300
     74:   440.914 [PROP_AF_ASSIST_BEAM_FIRING] 0
     75:   441.074 CurrentMedia = 2
     76:   443.927 [STARTUP] startupPrepareRemote
     77:   445.087 [AUDIO] RegisterCBRSDIOCableConnect
     78:   452.462 [WFT] PROP_ADAPTER_DEVICE_ACTIVE: Can Not Change(0) Connect(0)
     79:   463.393 [RMT] QuickTransmissionEnable (0, 1, 2, 2)
     80:   463.438 [RMT] PROP_CONNECT_TARGET (0x0 <= 0x0)
     81:   463.475 [RMT] PROP_CONNECT_TARGET_WFT (0x0 <= 0x0)
     82:   463.531 [RMT] PROP_WFT_SYSTEM : 0
     83:   463.961 [PTP] PSI UnlockTransImg : Trans List is NULL
     84:   463.995 [PTP] PSI TarminateTransImgList : SessionTarget [1]
     85:   464.020 [PTP] PSI TarminateTransImgList : Trans List is NULL
     86:   464.133 [PTP] PSI UnlockTransImg : Trans List is NULL
     87:   464.158 [PTP] PSI TarminateTransImgList : SessionTarget [10]
     88:   464.180 [PTP] PSI TarminateTransImgList : Trans List is NULL
     89:   464.259 [PTP] PSI DisconnectViewFinder
     90:   464.763 [PTP] PROP_SCREEN_SAVER 0
     91:   464.926 [PTPCOM] SetPtpTransportResources:0,323a
     92:   465.189 [PTP] PSI UnlockTransImg : Trans List is NULL
     93:   465.221 [PTP] PSI TarminateTransImgList : SessionTarget [10]
     94:   465.242 [PTP] PSI TarminateTransImgList : Trans List is NULL
     95:   475.488 [WFT] Regist PropCBR PROP_WFT_BLUETOOTH
     96:   786.917 [SD] sdSendOCR: 1.8V Signaling Card
     97:   789.340 [SD] sdSendOCR: Hi-Capacity Card
     98:   795.978 [SD] CARD CAPACITY is 15193Mbyte( 31116288Sec )
     99:   812.798 [SD] SD_GetAccessMode=3
    100:   812.816 [SD] Set Hi-Speed Mode( 48MHz )
    101:   821.024 [FSU] fsuGetPart: Block(8192, 31108096, 31116288)
    102:   862.189 [FM] FileNo (DriveNo = 2, 0 <-> 157)
    103:   862.304 [FC] _FC_OpenCatalog : wNumOfDir 1
    104:   867.175 [MRK] MRK_Initialize
    105:   867.760 [MRK] MRK_PrepareMrkFileObject
    106:   870.803 [SEQ] seqEventDispatch (Startup, 4)
    107:   870.977 [DP] DP_Initialize()
    108:   871.656 [DP] Std T_PrintPage[0] T_PrintNo[0] R_PrintNo[0]
    109:   871.694 [DP] Std C_PrintNo[0] C_PrintPage[0], C_Copies[0]
    110:   871.788 [DP] Dp_SetPD_RegistApi Prev:0 New:0
    111:   872.148 [DP] PROP_CONNECT_TARGET [0x00000000]
    112:   872.268 [DP] RD_Cancel No Issue
    113:   872.501 [DP] [GYO]DpFunctionList[0x10117fff]
    114:   872.612 [DP] X FreeMem Address[0x00000000]
    115:   876.710 [DISP] HPD before InitializeDisplay !!!
    116:   876.961 [DISP] [ME] PropMgr (0) [SlaveResultCBR] (S_OFF)
    117:   877.080 [DISP] [ME] PropMgr (0) [ChangeCBR] (S_OFF)
    118:   877.103 [DISP] Display_flag = 1
    119:   927.579 [IMPP] H264E InitializeH264EncodeFor1080pDZoom
    120:   927.638 [IMPP] H264E InitializeH264EncodeFor1080p25fpsDZoom
    121:   954.964 [DP] DP_RegisterCBR() Id=1 Count:12
    122:   955.003 [DP] DP_RegisterCBR() Id=2 Count:12
    123:   955.030 [DP] DP_RegisterCBR() Id=4 Count:12
    124:   964.382 [GUI] MainEventHandler PROP_QR_DIDNOT_EXECUTE(0)(0)
    125:   966.664 [GUI] MainEventHandler PROP_ERROR_FOR_DISPLAY(0)
    126:   993.693 [LVGMT] Is PROP_OUTPUT_TYPE Real??[H:1][V:0][O:0]
    127:  1001.225 [MR] mvrChangeAckCBR : Video - Mode=0, Type=1, Rate=50, GOP=1
    128:  1002.062 [GUI] MainEventHandler PROP_LV_OUTPUT_DEVICE(0)
    129:  1003.208 [LVGMT] Is PROP_OUTPUT_TYPE Real??[H:1][V:0][O:0]
    130:  1004.069 [MAGIC] (Empty Func) MVW_RegisterXmpDataCallback
    131:  1005.200 [HDMI] HDMI CEC Initialize
    132:  1006.225 [MR] mvrChangeAckCBR : Video - Mode=0, Type=1, Rate=50, GOP=1
    133:  1007.428 [GUI] ERROR ***** ConvertHdmiVideoCodeToGuiDisplayType unknown Type(0)
    134:  1007.973 [HDMI] HPD ON
    135:  1007.997 [HDMI] ConnectHDMI
    136:  1008.023 [HDMI] InitializePortForHDMI
    137:  1008.049 [HDMI] ActiveHDMIDevice
    138:  1008.072 [HDMI] I2C_Init
    139:  1008.162 [HDMI] Pana_Init
    140:  1010.391 [SEQ] seqEventDispatch (Startup, 5)
    141:  1010.413 [STARTUP] startupInitializeComplete
    142:  1010.499 [MC] cam event guimode comp. 0
    143:  1011.314 [MC] cam event guimode comp. 0
    144:  1012.443 [DISP] BrightnessControl from GUI 1
    145:  1013.765 [DL] PROP_ACTIVE_SWEEP_STATUS (0)
    146:  1013.799 [DL] PROP_DL_ACTION (1)
    147:  1018.304 [MC] PROP_GUI_STATE 2
    148:  1021.169 [MC] PROP_GUI_STATE 2
    149:  1022.224 [GUI] MainEventHandler PROP_LV_OUTPUT_DEVICE(0)
    150:  1023.042 [MC] PROP_GUI_STATE 2
    151:  1023.944 [GUI] IDLEHandler START_WARNING_LENSLESS_MOVIE_MODE
    152:  1024.027 [MC] cam event guimode comp. 2
    153:  1034.709 [WINSYS] Zlib maxsize update 884
    154:  1034.763 [WINSYS] Zlib maxsize update 1396
    155:  1060.186 [GUI] IDLEHandler START_WARNING_LENSLESS_MOVIE_MODE
    156:  1060.259 [MC] cam event guimode comp. 2
    157:  1096.103 [GUI] IDLEHandler START_WARNING_LENSLESS_MOVIE_MODE
    158:  1096.175 [MC] cam event guimode comp. 2
    159:  1118.924 [FA] ChangeCBR(ID=0x8003000a)
    160:  1119.598 [FA] RequestFromCheckData
    161:  1119.866 [FA] MpuMonSpecificFromPartner : COM_FA_CHECK_FROM 0
    162:  1128.724 [SEQ] seqEventDispatch (Startup) : End
    163:  1133.365 [HDMI] RESET_UNCOMPLETE
    164:  1136.719 [HDMI] RESET_COMPLETE
    165:  1148.561 [HDMI] KEY LOAD OK
    166:  1148.595 [HDMI] HOT_PULG_DETECT OK
    167:  1198.979 [HDMI] EDID_PREREAD OK
    168:  1199.001 [HDMI] Rx Sense is OK
    169:  1199.621 [HDMI] pfHdmiPbCBR Done
    170:  1253.694 [HDMI] TMDS Check
 
I was pretty sure that everything was going pretty well until this happened.

Hope it helps for developement...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 05, 2017, 12:22:36 PM
Were you using a HDMI monitor? I didn't test this configuration yet.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: garry23 on April 05, 2017, 12:23:53 PM
Spreading the word  ;) http://photography.grayheron.net/2017/04/magic-lantern-does-it-again.html (http://photography.grayheron.net/2017/04/magic-lantern-does-it-again.html)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 05, 2017, 12:25:18 PM
Quote
Were you using a HDMI monitor? I didn't test this configuration yet.

Yes, a Zacuto EVF... It can be the problem?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 05, 2017, 03:20:15 PM
Trying to reproduce the live view disconnection, I let camera on... This time get this message:
CTRL SRV Stack Overflow free=84 Used=163008176
Camera off.
Camera on.
I can record continuous 1920x960 50p 3x3
Camera remains on.
Suddently get this message in top of screen (in red letters):
[103] COMPRES_TASK: TASK WARNING: Free=164 Used=3932
Camera off.
When I boot camera again, I get this message full screen:
ML ASSERT: 0 at mlv_lite.c:2006 (compress_task), task compress_task vv:1 mode:3

srm_malloc_suite(1)...
srm buffer #1: 48600064
srm_malloc_suite(f1408)
Resolution changed: 2080x1587 -> 2080x961
Black level: 2047

Compressed size higher than uncompressed - corrupted frame?
Please reboot, then decrease vertical resolution in crop_rec menu.

ML ASSERT: 0 at mlv_lite.c:2006 (compress_task), task compress_task vv:1 mode:3


I didn't change any resolution... In crop_rec menu it remains 1920x960.
I'm posting all my troubles... I have no idea if it is useful or not. If not, I will stop posting, just let me know  ;)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: garry23 on April 05, 2017, 03:29:49 PM
@A1ex

Am I right in saying the key.press(KEY.HALFSHUTTER) wont (sic) work in the experimental 4K at the moment? It doesn't seem to.

I'm trying to explore triggering the experimental FRSP(4K version) in  Lua script.

Or am I jumping the gun  ;)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 05, 2017, 04:19:56 PM
Quote
Am I right in saying the key.press(KEY.HALFSHUTTER) wont (sic) work in the experimental 4K at the moment? It doesn't seem to.

Right - will take care of this (as I want to merge lua_fix in regular nightlies anyway).

Quote
CTRL SRV Stack Overflow free=84 Used=163008176

Would be great if you can find a way to reproduce this.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: garry23 on April 05, 2017, 04:28:18 PM
@A1ex

 :) Great  :)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kyle Kearns on April 05, 2017, 07:12:41 PM
Does a 1:1 crop mean it's a 2x crop similar to a GH4?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hyalinejim on April 05, 2017, 07:24:24 PM
The references to 1:1 mean that every pixel on the sensor becomes a pixel in the image. The "crop factor" depends on the resolution you choose.

 A 1920 x 1080 1:1 central section of the sensor (5760 pixels wide) gives a crop factor of 3.

2880 x 1620 is a 2x crop factor.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 05, 2017, 07:28:37 PM
@Kyle Kearns Naw.. 3K 1:1 I believe is 1.84 -  UHD is 1.5 -  True 4K is 1.4
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 05, 2017, 09:03:28 PM
Quote
Quote

    CTRL SRV Stack Overflow free=84 Used=163008176


Would be great if you can find a way to reproduce this.

After 2 hours I'm not able to reproduce CTRL SRV...

Quote
Suddently get this message in top of screen (in red letters):
[103] COMPRES_TASK: TASK WARNING: Free=164 Used=3932
Camera off.
When I boot camera again, I get this message full screen:
ML ASSERT: 0 at mlv_lite.c:2006 (compress_task), task compress_task vv:1 mode:3

srm_malloc_suite(1)...
srm buffer #1: 48600064
srm_malloc_suite(f1408)
Resolution changed: 2080x1587 -> 2080x961
Black level: 2047

Compressed size higher than uncompressed - corrupted frame?
Please reboot, then decrease vertical resolution in crop_rec menu.

ML ASSERT: 0 at mlv_lite.c:2006 (compress_task), task compress_task vv:1 mode:3

This ML ASSERT message appears when in crop_rec mode 1920 50p 3x3, I change (in Canon menu) from 1280x720 50 All-I to 1920x1080 25 All-I before change crop_rec to OFF. I need to reboot 2 times and then all come back to normal operation...

I will try again to reproduce the CTRL SRV message :)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Doofii on April 06, 2017, 03:29:11 AM
select 14bit lossless,and "frame and block size mismatch:0xcba800 0x20 0x88bc00" shows when play back.
ive tired many times,but the problem still exists.
what could i do?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 06, 2017, 05:52:35 AM
I shot something like 12 takes.
I tried 14lossless and got a warning at the top of the screen (couldnt read due to age)
and a series of many warnings.
I removed the battery, start from scratch, default ML settings, load modules.
I set 10bit
Set the Canon video to 1280x720, NTSC.
Everything as expected.

In order to reach optimum performance, I set the Canon video back to 1920x1080, PAL.
The resolutions 3K, UHD and 4K had less height that the promised in ML forums.
e.g 3072x1728
In these resolutions, I couldnt record more than 3 secs maximum.
Surprisingly I managed to record 3568x1320 for much longer duration. <--- pretty strange not to be able to record longer for lower resolution



Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: martinhering on April 06, 2017, 09:42:55 AM
I was experimenting with the lossless lj92 compression in MLV files. I wanted to see if I can decode the image and reinterpret it as a raw cfa pattern. I came across a number of problems that could also be a problem in mlv_dump. I took the liblj92 implementation by Andreaw Baldwin found here:
https://bitbucket.org/hudson/magic-lantern/src/7a9b6805756c3b86f2174bac00433c544a976501/modules/mlv_rec/lj92.c?at=mlv_rec_lj92&fileviewer=file-view-default (https://bitbucket.org/hudson/magic-lantern/src/7a9b6805756c3b86f2174bac00433c544a976501/modules/mlv_rec/lj92.c?at=mlv_rec_lj92&fileviewer=file-view-default)

But I could not make it work. I had decoding problems due to the fact that the library is missing some support for multiple components. I then came across the implementation in tiny_dng_loader by Syoyo Fujita here:
https://github.com/syoyo/tinydngloader/blob/master/tiny_dng_loader.h (https://github.com/syoyo/tinydngloader/blob/master/tiny_dng_loader.h)

I then extracted his changes back into a simple standalone .c and .h file and tried that and viola, the decoding result was much better:
(http://i.imgur.com/5C88tC6.jpg)

I posted the extracted version of the library on GitHub:
https://github.com/martinhering/liblj92 (https://github.com/martinhering/liblj92)

Just saying, if you encounter any decoding problems in mlv_dump with the mlv_rec_lj92 branch, give my version a try.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 06, 2017, 10:08:21 AM
Quote
I shot something like 12 takes.
I tried 14lossless and got a warning at the top of the screen (couldnt read due to age)
and a series of many warnings.
I removed the battery, start from scratch, defauly ML settings, load modules.
I set 10bit
Set the Canon video to 1280x720, NTSC.
Everything as expected.

Yes, same as me. Now message is:

[76] CEC: stack warning: free=128 used=3968

I'm totally lost...
@A1ex
I'don't know what to do, so if you can point me to what to do to help, just let me know.

After the messages I can reboot and I can record continuous without corrupted frames (allways talking about crop_rec April 4th build, 1920 50p 3x3 mode)... so this is not a big problem, but I have no idea of code so no idea what all this messages mean and what to do... sorry  :-[ I'm limited to explain what happens...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 06, 2017, 10:32:12 AM
Did some changes that might help with memory corruption (new build posted).

In particular, I think I've found a way to limit compressed data size when it ends up larger than the allocated buffer. Previously, passing an arbitrary limit for buffer size simply locked up the camera at certain resolutions - it appears this limit must be aligned at 4096-byte multiples (maybe a bit less, but in any case, at least 1024).

If you still get stack overflow, please report even if you don't have a way to reproduce it. This kind of bug is usually very serious.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: garry23 on April 06, 2017, 10:59:43 AM
@A1ex

Just checking: I assume this build doesn't include the Lua fix yet? I doesn't look if it does.

Cheers

Garry
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 06, 2017, 11:05:45 AM
Just tested April 6th 113.

Can't get continuous recording like before with april4th, only 10 sec. before message "didn't make it to stimated record time" and then recording stops. (1920 50p 3x3 mode).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: GutterPump on April 06, 2017, 11:12:03 AM
This kind of bug is usually very serious.

Is this mean we could get a semi brick or definitive brick ? What is the long-term risk ?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 06, 2017, 11:19:03 AM
@pc_bel: nothing was changed on the recording speed side; I bet any difference is because of different scene complexity. Please test the stack overflow bug first.

@garry23: Lua changes are a bit too much when I'm troubleshooting a memory corruption bug. The changelog from the Experiments page is valid.

Is this mean we could get a semi brick or definitive brick ? What is the long-term risk ?

Yes, as with all ML builds. Explained a little here (http://www.magiclantern.fm/forum/index.php?topic=12627.msg170133#msg170133) and also somewhere in the Tragic Lantern threads. Currently I'm diagnosing this report (http://www.magiclantern.fm/forum/index.php?topic=19300.msg182395#msg182395), and it appears to be, indeed, caused by some bad settings reflashed by Canon code into ROM - possibly after memory corruption caused by ML.

The warnings from the download page are not there for fun - this kind of errors cannot be prevented on the current processor design, as it doesn't have a MMU (also valid for many other simple CPUs that use non-volatile storage updated frequently). Canon started to use a processor with MMU with DIGIC 7 (https://chdk.setepontos.com/index.php?topic=13014.30), but they don't seem to use its memory protection features (they simply do a flat memory mapping, except a 0x1000-byte block unique to each core).chd

Just in case, make sure you have a copy of your ROM in a safe place (any recent installer tells you how to do so).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 06, 2017, 11:23:05 AM
Quote
Just tested April 6th 113.

Can't get continuous recording like before with april4th, only 10 sec. before message "didn't make it to stimated record time" and then recording stops. (1920 50p 3x3 mode).

After reboot, I get the same message as with the april4th build:
ML ASSERT: 0 at mlv_lite.c:2006 (compress_task), task compress_task vv:1 mode:3

srm_malloc_suite(1)...
srm buffer #1: 48600064
srm_malloc_suite(f1408)
Resolution changed: 2080x1587 -> 2080x961
Black level: 2047

Compressed size higher than uncompressed - corrupted frame?
Please reboot, then decrease vertical resolution in crop_rec menu.

ML ASSERT: 0 at mlv_lite.c:2006 (compress_task), task compress_task vv:1 mode:3
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 06, 2017, 11:24:46 AM
ML ASSERT: 0 at mlv_lite.c:2006 (compress_task), task compress_task vv:1 mode:3

Fortunately, that's no longer causing memory corruption (so it's a minor bug for now). It's simply invalid image data being compressed (and output size resulting higher than initial size). The extra data is no longer written to RAM on latest build (and before, it was written in the place reserved for future frames, not on DryOS data structures or persistent setting areas - to my knowledge).

The stack overflows are worrying me, though (as they do indicate memory corruption in the middle of DryOS).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: BBA on April 06, 2017, 12:08:50 PM
Thanks for those groundbreaking efforts  :) :) !!!!!!

Don't know if it helps. If I can remember...

I got just while moving in the menu but without CF card (only SD):
white flashing "[93] audio-common-task stack warning free=180 used=3016"
then red flashing "[91] clock-task st(don't remember the end of the word) free=36 used=8156"

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: jankrueck on April 06, 2017, 12:21:16 PM
Code: [Select]
ASSERT: this->VramNv12Param.hJpegMemSuite
at ./Epp/Vram/VramStage.c:745, task Epp
lv:1 mode:3


Magic Lantern version : crop_rec_4k.2017Apr04.5D3113
Mercurial changeset   : e7e1fdb85f6b (crop_rec_4k) tip
Built on 2017-04-04 15:41:29 UTC by [email protected]
Free Memory  : 146K + 3094K



Code: [Select]
ASSERT: 0
at ./PathDrvWrap/EekoYuvToVramPath.c:606, task Epp
lv:0 mode:3


Magic Lantern version : crop_rec_4k.2017Apr04.5D3113
Mercurial changeset   : e7e1fdb85f6b (crop_rec_4k) tip
Built on 2017-04-04 15:41:29 UTC by [email protected]
Free Memory  : 158K + 3072K


this was on 4th. didn't see there is a new version.
testing up to date version atm.

looks good so far.



I have little knowledge on how you managed to achiev this update.
But I assume, there will be no way to use an external monitor on this, right?
HDMI connectors are limeted to 1080p huh?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: BBA on April 06, 2017, 12:40:10 PM
Without ML (SD card with ML removed) and only CF card (data card):
In Canon menu, the sensor cleaning gets disabled when the camera is switched on.
(I am with the 4th april build)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 06, 2017, 04:52:46 PM
For now, all I could get are these error messages:

ASSERT: hLvJob->hJpegMemSuite
at ./Epp/Vram/VramStage.c:868, task Epp
lv:1 mode:3
Magic Lantern version : crop_rec_4k.2017Apr06.5D3113
Mercurial changeset   : bc99f7502deb (crop_rec_4k) tip
Built on 2017-04-06 08:28:51 UTC by [email protected]
Free Memory  : 172K + 3779K

ASSERT: IsSuiteSignature( hSuite )
at ./PackMemory/PackMem.c:599, task RscMgr
lv:0 mode:3
Magic Lantern version : crop_rec_4k.2017Apr06.5D3113
Mercurial changeset   : bc99f7502deb (crop_rec_4k) tip
Built on 2017-04-06 08:28:51 UTC by [email protected]
Free Memory  : 172K + 3778K

ASSERT: fInitialize == TRUE
at ./System/PostPostman/PostPostman.c:243, task Epp
lv:0 mode:3
Magic Lantern version : crop_rec_4k.2017Apr06.5D3113
Mercurial changeset   : bc99f7502deb (crop_rec_4k) tip
Built on 2017-04-06 08:28:51 UTC by [email protected]
Free Memory  : 172K + 3781K




Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 06, 2017, 05:44:55 PM
Without ML (SD card with ML removed) and only CF card (data card):
In Canon menu, the sensor cleaning gets disabled when the camera is switched on.
(I am with the 4th april build)

Does clearing Canon settings help? Otherwise, do you have a screenshot, so I know what to look for?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 06, 2017, 06:04:04 PM
Day of testing!!! :)

I returned to crop_rec 113 april4th build due to the short recorded times I get with april6th (I don't know why...)  :(

Really surprised for the extremely low rolling shutter in 1920 50p 3x3 mode... but panning, I get a cut line in the top part of the frame, only with panning! it dissapears in static frames.

This video explains better:
http://ovh.to/b2isZ5B

Edited: I can't see the cut line in live view, only in processed dng's.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: jankrueck on April 06, 2017, 06:20:07 PM
Really surprised for the extremely low rolling shutter in 1920 50p 3x3 mode... but panning, I get a cut line in the top part of the frame, only with panning! it dissapears in static frames.
Edited: I can't see the cut line in live view, only in processed dng's.

same here. got this strange line, when panning.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: garry23 on April 06, 2017, 06:23:38 PM
@A1ex

I've continued 'playing around' with the latest experiments in still photography and am a bit confused. ML seems to be creating 'time dilation in the space-time continuum'  :)

I set up a long exposure test, 30s, and took a base .cr2 capture.

I then kept this exposure and used the full res LV in the experimental build.

As soon as I pressed the half shutter press, I got the saving message. I pull the card immediately.

I repeated and left the card in for a minute.

The two DNGs were the same, ie I seem to get a 30s silent in a couple of seconds.

The cr2 and the two DNGs were exposed OK, ie ETTR like and not overexposed.

I obviously don't understand how long exposures are working with the crop/silent stills.

When you have time, I would welcome any insight.

Cheers

Garry
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 06, 2017, 06:33:29 PM
I returned to crop_rec 113 april4th build due to the short recorded times I get with april6th (I don't know why...)  :(

Proof that it has shorter recording times, please! A static test scene (without varying light levels and with the same focus), 10 test runs with each build, starting from a formatted card, at the same resolution and ISO, should do the trick.

I set up a long exposure test, 30s, and took a base .cr2 capture.

I then kept this exposure and used the full res LV in the experimental build.

As soon as I pressed the half shutter press, I got the saving message. I pull the card immediately.

The two DNGs were the same, ie I seem to get a 30s silent in a couple of seconds.

The cr2 and the two DNGs were exposed OK, ie ETTR like and not overexposed.

I obviously don't understand how long exposures are working with the crop/silent stills.

Best guess: you have tried in photo mode (which I didn't try yet), with exposure simulation. So, if you have metered a 30s ISO 100, the LiveView is probably at 1/FPS and a very high ISO. An example, if FPS override was set to 5, the LiveView frame might have been exposed at 1/5 seconds, aperture wide open - even if you have selected something different in menu - and ISO as high as required to get the same brightness as with a 30-second exposure (considering the other two variables).

I'm curious to see the DNGs.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: garry23 on April 06, 2017, 06:41:25 PM
@A1ex

No EXIF so can't tell other than looking at them.

Canon and ML ISO settings remain at 100.

Tried with expo sim off. Saving straight away, i.e. Doesn't do a 30 sec capture.

Cheers

Garry
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: garry23 on April 06, 2017, 07:07:04 PM
@A1ex

Sitting here trying to think this through.

As this is coming from a video development, I guess we will be limited, at ISO100, to shutter speeds that are consistent with video framing rates.

I guess this means this still photography approach will solve fast shutter speeds, but not give ISO100 slow shutter speeds.

Or have I got it wrong?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Frank7D on April 06, 2017, 07:28:10 PM
pc_bel said:
Quote
but panning, I get a cut line in the top part of the frame, only with panning! it dissapears in static frames.

In other contexts, this was theorized to be a symptom of overloading the processor. I used to get it with greyscale ml preview on my 7D in 5x crop mode.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: BBA on April 06, 2017, 08:25:57 PM
@A1ex
Thanks for your assistance ! Don't hesitate to ask more if necessary ! I am not stuck  ;) .

a) more infos about my problem:

    a.1) without ML card (I use the SD card for ML)
    a.2) CF card in camera (formatted in camera)
    a.3) I go to Canon menu : yellow settings menu / SETUP3 / "sensor cleaning" and set "auto cleaning" to "enabled"
    a.4) I switch off the camera : the sensor cleaning screen appears as before; the red bottom led is not lit.
    a.5) I then switch on the camera : the sensor cleaning screen does Not appear; the bottom red led is lit.
    a.6) if I go again to the Canon menu : yellow settings menu / SETUP3 / "sensor cleaning", the "auto cleaning" has been set to "disabled"

b) reinit all settings : no change

    b.1) Canon menu / yellow settings menu / SETUP4 / "reinit all settings"
    b.2) the "auto cleaning" feature is actually set to "enabled" if I verify in the Canon menu
    b.3) then I get the same as above in a.4) , a.5)  and  a.6)

[EDITED 2017/04/07]
c) circumstances ? no CF card ?

Unknown

I think (not sure at all) it may be due to the fact that I tried to repeat the problem I got with no CF card in the camera (silly thing : should not be done without further investigation): the camera got stuck; no message ; I had to remove the battery.
Investigated by A1ex (thanks a lot) : no such problem when switched on with no CF card (see post #156). 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 06, 2017, 09:16:40 PM
@BBA: could not reproduce.

Side note: I've let the camera on for 10 hours, on a power adapter, with today's build, on the 3K video mode, 3072x1920 in mlv_lite menu, lossless compression, "framing" preview, in standby - so it would estimate the compression ratio every 2 seconds. Every 30 minutes, it closed LiveView for a second, and returned to LV (I could hear it when I came home).

Temperature after this experiment was 62 degrees C. The camera was enclosed in a drawer during the entire experiment.

I've now downloaded the previous build (Apr04) and will perform the same test overnight (or until it crashes).

edit: it crashed after 15 minutes or so of recording short test clips in 50p 3x3; back to Apr06, no crashes yet.

I'm doing these tests to check these reports:

I tried 14lossless and got a warning at the top of the screen (couldnt read due to age)
and a series of many warnings.

Trying to reproduce the live view disconnection, I let camera on... This time get this message:
CTRL SRV Stack Overflow free=84 Used=163008176
[...]
Suddently get this message in top of screen (in red letters):
[103] COMPRES_TASK: TASK WARNING: Free=164 Used=3932
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: garry23 on April 06, 2017, 09:28:16 PM
@A1ex

Ok I get it now  ;)

As you say on the first post

Quote
The last feature complements the well-known full-resolution silent pictures - the new implementation will be usable at fast shutter speeds, without the exposure gradient - but with rolling shutter (just like regular LiveView frames).

Photographers need to look to using the crop approach and the current FRSP, together.

I'm looking forward to trying to script these two techniques together in Lua  :)

Really exciting and fun times.

Cheers

Garry
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 06, 2017, 09:39:52 PM
Please follow the same steps as ju64 and contact me on IRC.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Walter Schulz on April 06, 2017, 11:14:36 PM
Upper right corner:
IRC: #magiclantern (http://webchat.freenode.net/?channels=magiclantern)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 06, 2017, 11:51:29 PM
Another case of bad setting flashed somehow into ROM. Reproduced the error in QEMU. Appears easy to fix, since these settings fortunately have backups (wear leveling?). I've already unbricked a similar case a while ago (also 5D3), and I've already booted the GUI in QEMU on the affected ROM.

Two failures on the same build are a bit too much to be a coincidence. For now, my advice would be to avoid the Apr04 build, even though some reports say it's faster (I'm willing to bet it's not, but I'll benchmark it, just in case).

Apr06 should be safe regarding this issue (see my tests) (http://www.magiclantern.fm/forum/index.php?topic=19300.msg182552#msg182552).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kharak on April 07, 2017, 12:09:15 AM
Thanks for the heads up!

 I'll update to Apr06.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 07, 2017, 02:00:12 AM
@a1ex Thank you for all your hard work.. I tried the April 4th build but did not encounter such problems on 1.1.3.  Will give April 6th a test drive tonight I'll post my findings. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Hans_Punk on April 07, 2017, 05:40:11 AM
Amazing work @a1ex as always!

I'm using MLV_Dump for the newest crop_rec module with higher resolutions (4K, 1080p48 etc) build - And I'm getting vertical stripes on the .dng exports.
Is this normal for now?...or am I missing something simple? I have not used MLV_Dump before now (and I'm a code ignorant idiot)
Test shots were lower ISO 100-400, various resolutions tested from April 4th build - all exhibited some vertical stripes (some more noticeable at lower ISO)

Running this code on Win 7 works for me to export the .dngs's, I just don't know if vertical stripe correction is an option yet in this version of MLV_Dump...or I simply need to somehow amend the .cmd code to 'switch on' the correction in MLV_Dump?

@echo off
md "%~n1"
"Q:\NEW MLV\mlv_dump.exe" -o %~n1\%~n1_ --dng %~n1.MLV

Link to screengrab of most prominent vertical stripes:
https://www.dropbox.com/s/md2inmxhaj604gv/M06-1750%20%2800123%29.jpg?dl=0 (https://www.dropbox.com/s/md2inmxhaj604gv/M06-1750%20%2800123%29.jpg?dl=0)

.Dng Sample files:
https://www.dropbox.com/sh/fe5nve001l2vqx5/AADpbW7eVOqafc7XP4XnhgD0a?dl=0 (https://www.dropbox.com/sh/fe5nve001l2vqx5/AADpbW7eVOqafc7XP4XnhgD0a?dl=0)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 07, 2017, 05:54:32 AM
Upload a sample file?
Shortened file:
Code: [Select]
mlv_dump -f 5 -o Output.MLV Input.MLVInput.MLV being your file with stripes. Do the command correct or you might loose the file.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: garry23 on April 07, 2017, 08:38:42 AM
@A1ex

I thought I would carry out a few experiments in photo mode looking at 'lowish' shutter speeds with the FRLV.

I decided to use an FPS twice the selected shutter speed.

It appears I can push the FPS down to 0.5 (ML says its 0.554) before it plateaus out.

I took the following combinations and all the DNGs looked OK in LR:

FPS = 2, ML:shutter = 1/10s
1, 1/4
1, 1/2
0.5(0.554), 1s

From my perspective this looks a healthy overlap with the 'normal' FRSP, which could use down to, say, 1/4s.

Cheers

Garry
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 07, 2017, 08:49:37 AM
From my perspective this looks a healthy overlap with the 'normal' FRSP, which could use down to, say, 1/4s.

Sounds reasonable - as the full-res implementation can be updated decide whether to use LiveView or the "long exposure" version, depending on the shutter speed selected in photo mode.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: garry23 on April 07, 2017, 08:53:20 AM
@A1ex

Yes: I've worked out how to do it in my Lua bracketing scripts and I will confirm things as soon as I can. ie when I can trigger with the half shutter key press.

This should give photographers a 30s to 'ultra fast' seamless capture capability.

 :)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 07, 2017, 09:50:40 AM
Quote
Two failures on the same build are a bit too much to be a coincidence. For now, my advice would be to avoid the Apr04 build, even though some reports say it's faster (I'm willing to bet it's not, but I'll benchmark it, just in case).

Ok, back to april6th...

192050p3x3, 5D3 113, ISO 100: 10 recordings 1min/each (camera didn't stop recording, I stoped recording assuming continuous).
192050p3x3, 5D3 113, ISO 400: 10 recordings 1min/each (camera didn't stop recording, I stoped recording assuming continuous).
192050p3x3, 5D3 113, ISO 800: 1 recording, camera stops recording after 10sec. (message:"didn't make it to stimated record time")
I turn off camera.
When I turn on camera, a full screen message appears like this one:
Quote
When I boot camera again, I get this message full screen:
ML ASSERT: 0 at mlv_lite.c:2006 (compress_task), task compress_task vv:1 mode:3

srm_malloc_suite(1)...
srm buffer #1: 48600064
srm_malloc_suite(f1408)
Resolution changed: 2080x1587 -> 2080x961
Black level: 2047

Compressed size higher than uncompressed - corrupted frame?
Please reboot, then decrease vertical resolution in crop_rec menu.

ML ASSERT: 0 at mlv_lite.c:2006 (compress_task), task compress_task vv:1 mode:3

So ISO was the culprit for not get continuous recording with april6th?... My fault. :-[

No STACK OVERFLOW error for now...

Edit: in this tests I got the same corrupted image that before when panning (top part of frame with motion cutted).
Can I ask if is easy to adress this, or not? That would make the recordings usable...!!!! (at least fro me).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: squig on April 07, 2017, 10:14:21 AM
Apr06 3104x1298 1600 ISO 93.4MB/s ~57%
5.26Gb
5.78Gb
Restarted
8.15Gb
25.06Gb (I got bored and stopped recording)

I was going do a comparison with Apr04 but the stack error appeared (for the 1st time) when I loaded the modules so I chickened out.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 07, 2017, 10:27:32 AM
Edit: in this tests I got the same corrupted image that before when panning (top part of frame with motion cutted).
Can I ask if is easy to adress this, or not? That would make the recordings usable...!!!! (at least fro me).

Here's a boring task: what's the number of pixels before the cut? (min/max/median would be great)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 07, 2017, 11:03:18 AM
Quote
Here's a boring task: what's the number of pixels before the cut? (min/max/median would be great)

 :-[ Sorry for that!!!... Its only thing that makes image unusuable.

I've  measured px and the pattern is very strange... min distance I've found is 7px, max is 125px, but it doesn't follow a logic. At first it seems increassing with the panning velocity, but looking close, during panning, one frame have 98px (from top to cut) the next 112px, the next 18px, one frame (in between all of this) is correct, and for example, the next have 73px...

Min I found: 7px
Max I found: 125px
Average from random 17dng:83,8px
@A1ex
Is this what you are asking for?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 07, 2017, 11:33:13 AM
Yes. I thought I'd get away with it by increasing the optical bar size, but it looks like I should re-enable double-buffering (at least for certain video modes).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 07, 2017, 11:35:27 AM
If you need to do some "mechanical-tedious" work, if you explain me how to do it, I can do it...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: BBA on April 07, 2017, 11:54:09 AM
@A1ex
Quote
Could not reproduce
A good thing for other users : if you agree, I will edit my #155 post and remove the unnecessary warning about not switching on a camera without CF card


Sensor cleaning issue:

Maybe something interesting in the Debug menu: red dot in front of "memory patches"

https://www.dropbox.com/s/ap2zant3kn9vy2e/VRAM3.PPM?dl=0
 (https://www.dropbox.com/s/ap2zant3kn9vy2e/VRAM3.PPM?dl=0)

memory patches, with a red dot:

https://www.dropbox.com/s/cafc5h767gbqgdv/VRAM2.PPM?dl=0
 (https://www.dropbox.com/s/cafc5h767gbqgdv/VRAM2.PPM?dl=0)

Could it be of any help for fixing the cleaning sensor issue of my camera ?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 07, 2017, 12:09:36 PM
Maybe something interesting in the Debug menu: red dot in front of "memory patches"

That's unrelated; it's simply telling you about some module (here, crop_rec) using ROM patches aka cache hacks (which are a bit fragile in the backend).

You can use this build to get a startup log with all the debug messages from Canon. Before the log gets saved, you should try to perform the sensor cleaning, so messages about it will get included:

https://builds.magiclantern.fm/jenkins/view/Experiments/job/startup-log/
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: BBA on April 07, 2017, 12:36:15 PM
@A1ex
Thanks a lot !
I will give it a try and report.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: GutterPump on April 07, 2017, 12:37:47 PM
And I'm getting vertical stripes on the .dng exports.

I also noticed this issue with the 14 bits lossless feature.

Even  if i change the code in the raw2dng.c file :

Code: [Select]
      if (stripes_correction_needed)
    {
        printf("\n\nVertical stripes correction:\n");
        for (j = 0; j < 8; j++)
        {
            if (stripes_coeffs[j])
                printf("  %.5f", (double)stripes_coeffs[j] / FIXP_ONE);
            else
                printf("    1  ");
        }
        printf("\n");
    }
}

to

Code: [Select]
     printf("\n\nVertical stripes correction:\n");
        for (j = 0; j < 8; j++)
        {
            if (stripes_coeffs[j])
                printf("  %.5f", (double)stripes_coeffs[j] / FIXP_ONE);
            else
                printf("    1  ");
        }
        printf("\n");
    }

For constant vertical stripe fixs with mlv_dump. (Thanks to Danne for this tip)
The fix don't appear in the command line.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 07, 2017, 12:50:46 PM
To decode, you will need a custom mlv_dump (also included). It works by simply dumping the lossless JPEG payload to DNG (that means, processing options such as bad pixel or vertical stripe fixes are not working yet).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: GutterPump on April 07, 2017, 01:16:42 PM
Sorry a1ex didnt saw this message ! Thanks for the remind.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Hans_Punk on April 07, 2017, 01:27:33 PM
Ah yes...my fault for not acknowledging that info. I should have noticed the custom mlv_dump disclaimer.

Cheers A1ex / Danne & GutterPump for reply x
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: BBA on April 07, 2017, 03:30:40 PM
@A1ex
I don't know how to save the debug msgs after a "manually started" cleaning process:
Quote
Before the log gets saved, you should try to perform the sensor cleaning, so messages about it will get included:

When I switch the camera on, the "auto sensor cleaning" is set to "disabled";
The bottom red led is lit a while and then a few times (blocks of data written to the card ?)
It ended at 14:42
I went to the Canon SETUP3 menu to start a manual immediate cleaning
The camera made some noise as if taking 2 pictures then this greyed menu item became yellow again.
I then went to "auto cleaning" item to set it to "enabled"
I then went in the ML DEBUG menu and pressed SET button with cursor on DebugMsgLog
The bottom red led was lit a very little while (but I saw no file on the card created at that time = 14:46).
Finally I switched the camera off.

The following file is generated just when the camera gets started (gets the creation time = 14:42):

https://www.dropbox.com/s/mjickmertwial3y/DM-0001.LOG?dl=0 (https://www.dropbox.com/s/mjickmertwial3y/DM-0001.LOG?dl=0)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 07, 2017, 04:10:37 PM
Diagnosed the bricking issue - it was caused by a null pointer bug in ML :(

Only Apr04 is affected - both older and newer builds are safe.

Fortunately, it's easy to recover (at least in the two previous cases). One uint32_t at an arbitrary address (between 0 and around 60MB, with a non-uniform probability distribution) might be overwritten with 0xA5A5A5A5 (and eventually end up saved into ROM at camera shutdown, when Canon code saves camera settings).

@BBA: if you believe your camera might be affected by this bug, please send me a recent (bad) copy of your ROM, alongside with an older (good) copy.

If you don't have any backup, copy the oldest files from ML/LOGS you can find on a ML card (hopefully these are "last known good" configuration).

To get a fresh copy, simply delete ML/LOGS from a ML card, and it will be re-created at next startup.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: GutterPump on April 07, 2017, 05:42:39 PM
I went to the Canon SETUP3 menu to start a manual immediate cleaning
The camera made some noise as if taking 2 pictures then this greyed menu item became yellow again.

This is normal, it's the same for me and for all 5DIII users.

infos here (http://www.canonrumors.com/forum/index.php?topic=13188.0)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: BBA on April 07, 2017, 05:59:42 PM
@ Gutterpump
Thanks for your feed !



Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Mefist on April 07, 2017, 06:11:09 PM
I get a black DNG (((((

CMD the code is:
@echo off
for %%a in ( %* ) do ( md "%%~na" 2>nul )&( "c:\RAW\mlv_dump.exe" -o %%~na\%%~na_ --dng %%~na.MLV )

The code is wrong?

I have Windows 10 x64. I drag MLV to CMD, it immediately closes, creates a folder and it has black DNG

(https://s8.hostingkartinok.com/uploads/images/2017/04/7bfaabd168d09474f10bb311ced4c840.png)


Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 07, 2017, 06:44:03 PM
@A1ex I got the nasty error explosion today using the April 6 build.
I think it was probably because I was using an odd resolution
3008x1280 or something like that. Pulled battery everything seem fine with the camera.
Tried again with 3072x what ever 2.35:1 is and it seemed to work ok.
Would you like me to report the bug.
Here is a link to what was on my card. Error codes and all.
https://www.dropbox.com/sh/8lmopv8z7p2i91f/AAA4saYTmLE7rFQ3t7nSUBCaa?dl=0
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: GutterPump on April 07, 2017, 07:00:44 PM
I get a black DNG (((((


Can you share a small part of your MLV file ? Or record a very short part of movie for get arround 10 frames.
Be sure you're using the good version of mlv_dump (https://builds.magiclantern.fm/jenkins/job/crop_rec_4k/7/artifact/modules/mlv_rec/mlv_dump.exe) too.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Mefist on April 07, 2017, 08:11:49 PM
That's what I used and settings.

(https://s8.hostingkartinok.com/uploads/images/2017/04/a0f13f057a711f0fe713aad9f9b6ffe9.jpg)(https://s8.hostingkartinok.com/uploads/images/2017/04/e8bd24d101d2c3c5dd3d648fdd1a3288.jpg)

(http://poster.radikal.ru/1704/2a/12acbdda0c84.jpg) (http://radikal.ru/video/eNEQy5t1t25)

Quote
@echo off
for %%a in ( %* ) do ( md "%%~na" 2>nul )&( "c:\RAW\mlv_dump.exe" -o %%~na\%%~na_ --dng %%~na.MLV )

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: beauchampy on April 07, 2017, 09:46:20 PM
Just testing April 6th build on 123 with HDMI.

First up, it's a bit sketchy getting signal from the HDMI - have to power on, go into Canon menu, back out again, screen flashes all kinds of weird, live view then appears. First time I tried the camera crashed.

Second up, only getting 7 seconds at 1920x800 60p, GD off, 14-bit lossless, ISO 100. Data rate is low - only about 57mb/s

Just pulling out the HDMI getting 15 seconds. Data rate is more like 67mb/s.

Strange how plugging in the HDMI seems to lower the data rate by 10mb/s?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 07, 2017, 10:19:14 PM
Successfully unbricked the two cameras that had problems earlier in this thread! (ju64 and filipe), and also solved BBA's issue.

More cameras might be affected (actually up to 817); will add an automatic check to identify them in the next build. The side effects range from nothing at all (overwriting an unused memory location) to soft-bricking (recoverable) and anything in-between (e.g. BBA's issue). The bug can overwrite any setting (aka property in Canon code), or the data structure itself. My own 5D3 was also affected, without any obvious side effects though (as I don't know where the affected setting is used in Canon code).

I'll probably stop adding features for a while and will look into ways for preventing this class of bugs.

Strange how plugging in the HDMI seems to lower the data rate by 10mb/s?

It's not strange at all, as the camera has to work 6 times as much (1920x1080 vs 720x480) to create the HDMI image (strictly talking about the LiveView buffer). Most of the work is done by the image processing chips (external to the main CPU) and the DMA (EDMAC), so most of this load is on the memory bus.

The "Frozen LV" (previously known as Hacked preview) works by stopping LiveView buffers from being updated (therefore freeing the memory bus and a bit of CPU time).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 07, 2017, 10:21:40 PM
@beauchampy

Quote
First up, it's a bit sketchy getting signal from the HDMI - have to power on, go into Canon menu, back out again, screen flashes all kinds of weird, live view then appears

Same for me, but the same thing happens to me with older (and more stable) builds... To get hdmi signal I often go to play button and back out again. In my case using a Zacuto EVF.

I get continuous recording with april6th build with ISO100 1920x960 50p GD off 14bit losseless with or without hdmi in 5Dmk3 113.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: beauchampy on April 07, 2017, 10:32:06 PM
@beauchampy

Same for me, but the same thing happens to me with older (and more stable) builds... To get hdmi signal I often go to play button and back out again. In my case using a Zacuto EVF.

I get continuous recording with april6th build with ISO100 1920x960 50p GD off 14bit losseless with or without hdmi in 5Dmk3 113.

I didn't have much problem with previous builds. Might be because right now I'm using a Leica prime with no electronics, so that might be throwing things off a bit (camera probably thinking a lens not attached).

Just tried 50p and yes, getting continuous at a steady 79MB/s @ ISO 100, 14-bit lossless with HDMI, GD off.

Bit weird how the 60p mode seems to have a much lower MB/s (57MB/s with HDMI, 67MB/s without), so I'm guessing its something a little more complex than just write speed which is the issue there. a1ex / devs  if I can help test in any way, please do let me know.

Off to Bermuda tomorrow, so I'll do some more testing of this mode there. Thanks for the suggestion :)

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 07, 2017, 10:39:28 PM
Fantastic!!!

Be aware with ISO. For me 800 is where problems start. I can get only few seconds with 800 ISO. And as I report in post #148, I get a "cutted" image in the top of frame with paning movements... So no usable for serious work...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 07, 2017, 10:41:20 PM
@a1ex I've used the April 4th build so does that mean my 5D Mark III is affected as well?  I remember when using it red numbers were flashing on the top of my screen about memory being low.   
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Mefist on April 07, 2017, 10:47:17 PM
I still do not get a good DNG. If you shoot as before then the DNG is good.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: reddeercity on April 07, 2017, 11:06:45 PM
I still do not get a good DNG. If you shoot as before then the DNG is good.
Are the frames black in any of the App's you use ? e.g. Resolve , A.E. ACR  ? Or what ever you use
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 07, 2017, 11:38:08 PM
@A1ex is it only 5Diiis that used April 4 build or could it be in the April 3 build too?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 07, 2017, 11:59:13 PM
Here's a Python script to check for sure (run it and follow the instructions):

Code: [Select]
# checks whether your camera might have been affected
# by a null pointer bug from the 2017Apr04 ML build.
# updated Apr10 2017 (ignore some false warnings)

from __future__ import print_function
import os, sys
from struct import unpack

def getLongLE(d, a):
   return unpack('<L',d[a:a+4])[0]

try:
    ROM = open(sys.argv[1], "rb").read()
except IndexError:
    print("usage:")
    print("1) Backup the contents of ML/LOGS from your card")
    print("2) Debug -> Dump ROM and RAM (to get a new ROM copy)")
    print("3) Run this command on the latest copy of your ROM:")
    print("   python %s ROM1.BIN" % sys.argv[0])
    raise SystemExit

# scan for the magic number 0xA5A5A5A5 that might have been
# written into ROM as a result of the null pointer bug
found = 0
ignored = 0
can_ignore = False
ignore_msg = ""
gaonisoy = False
for a in range(0, len(ROM)-4, 4):
    v = getLongLE(ROM, a)
    n = getLongLE(ROM, a+4)
    block_name = ""
    can_ignore = False
    if a >= 0xF40000 and a < 0xF80000:      # RING - that's where most errors were
        a0 = a & ~0xFFF                     # header of this block
        if getLongLE(ROM, a0) == 0xFFFF:    # active block? only check this one
            block_name = "active RING"
            if a == a0:
                print("Active RING block: %x-%x" % (a0, a0+0xFFF))
        else:
            block_name = "inactive RING"
            can_ignore = True               # ignore inactive RING blocks
    if a >= 0xF80000 and a < 0xFA0000:      # last debug log - ignore
        block_name = "last debug log"
        can_ignore = True
    if v == 0xA5A5A5A5 and n != 0xE5E5E5E5:
        print(hex(a), hex(v), block_name)
        found   += (0 if can_ignore else 1)
        ignored += (1 if can_ignore else 0)
    if v == 0x6E6F6167 and n == 0x796F7369:
        assert a == 0xC0004
        gaonisoy = True

print("\nErrors found: %d, ignored: %d." % (found, ignored))

if not gaonisoy:
    print("Doesn't look like a valid ROM1 dump.")
elif found > 0:
    print("Your camera might have been affected by the Apr04 null pointer bug.")
    print("Please contact Magic Lantern developers on IRC (#magiclantern on freenode)")
    print("to take a closer look.")
else:
    print("Your camera appears OK, no need to worry.")
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: etx on April 08, 2017, 03:37:29 AM
Great work to all involved! Amazing!

I'm testing this out on my 5D Mk III on 1.2.3. I'm really missing the "centered 5x zoom" option from the older crop_rec module. I was previously running crop3x.2017Jan13.5D3123 version and using the centered 5x zoom was killer for avoiding the warping from the corners of the frame.

Am I missing something, or is this something that I just have to wait on future release?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 08, 2017, 04:18:37 AM
@a1ex What if you don't know how to run that script.. Can you point me to a detailed instruction on how to run that code, would be greatly appreciated. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 08, 2017, 07:45:36 AM
Did various tests.
My interest was focused on 3K, UHD and 4K trying to reach peak performance.
Global Draw OFF
1:1 mode
(3:3 always records 30FPS)
I concluded that 14bit lossless performs better.
I also concluded that setting Canon Video to 1280x720 gives shorter vertical size and leaves room for 4K recording.
Changing Aspect Ration allows more recording time
Havent managed to playback my recordings on the camera yet, I get an innocent message.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: janvkem on April 08, 2017, 07:48:25 AM
@a1ex What if you don't know how to run that script.. Can you point me to a detailed instruction on how to run that code, would be greatly appreciated. 

This is what I just did to test it (for Windows):
1. Install python for your computer: https://www.python.org/downloads/ (https://www.python.org/downloads/)
2. Make sure to check add python x.x to PATH
3. Make a backup of your current camera's ROM (using magic lantern)
3. When done make a new folder and copy your ROM1.bin to that folder (i made E:/tmp)
4. In the same folder make a new .txt file and rename it to testrom.py --> make sure you have file extensions visible
5. Open the file in your favorite text editor (i like notepad++) and copy the code in a1ex's post
6. Open a command window in the folder you just created (press right click on the folder while holding shift key, select Open command window here and type this:

    py testrom.py ROM1.bin

7. Read and follow the output on your screen.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hindra on April 08, 2017, 08:37:30 AM
This is what I just did to test it (for Windows):
1. Install python for your computer: https://www.python.org/downloads/ (https://www.python.org/downloads/)
2. Make sure to check add python x.x to PATH
3. Make a backup of your current camera's ROM (using magic lantern)
3. When done make a new folder and copy your ROM1.bin to that folder (i made E:/tmp)
4. In the same folder make a new .txt file and rename it to testrom.py --> make sure you have file extensions visible
5. Open the file in your favorite text editor (i like notepad++) and copy the code in a1ex's post
6. Open a command window in the folder you just created (press right click on the folder while holding shift key, select Open command window here and type this:

    py testrom.py ROM1.bin

7. Read and follow the output on your screen.

Worked great, thank you. My camera appears to be ok :)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: garry23 on April 08, 2017, 09:14:32 AM
Having never done it before: how long should a ML ROM dump take?

Update: I have 'Saving 4f000000...' on the ML screen for 15mins.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 08, 2017, 09:18:50 AM
Although I had doubts in the beginning, after draining a couple of batteries, I am sure now that current build (6April) is power hungry
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hindra on April 08, 2017, 09:20:25 AM
Having never done it before: how long should a ML ROM dump take?

Near instant? I believe when you "update" firmware with ML it will backup the ROM then. I just formatted my SD card in PC, format in camera, load ML from PC onto card, update firmware to setup ML, grab the ROM1.BIN and follow instructions.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: garry23 on April 08, 2017, 09:24:50 AM
@hindra

Thanks: I'll stop the current dump and try again.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: garry23 on April 08, 2017, 09:30:11 AM
When I looked at the card, I had a copy of ROM1.bin, however, the ML screen kept the 'Saving 4f000000...' on the ML screen for 15mins, until I stopped the Cam.

Run the test: all looks ok  :)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 08, 2017, 09:34:01 AM
Sounds encouraging; I'll integrate this test in the next ML build, but I'm a bit busy this weekend.

(that also means you will need a bit of patience on IRC - I'm not online this weekend)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: vstrglv on April 08, 2017, 10:30:48 AM
I have got this:
C:\Users\XXX\AppData\Local\Programs\Python\Python36>python testrom.py ROM1.bi
n
0xf5c33c 0xa5a5a5a5
0xf5cf0c 0xa5a5a5a5
0xf5d33c 0xa5a5a5a5
0xf5df0c 0xa5a5a5a5
0xf5e33c 0xa5a5a5a5
0xf5ef0c 0xa5a5a5a5
0xf5f33c 0xa5a5a5a5
0xf5ff0c 0xa5a5a5a5
0xf6033c 0xa5a5a5a5
0xf60f0c 0xa5a5a5a5
0xf6133c 0xa5a5a5a5
0xf61f0c 0xa5a5a5a5
0xf6233c 0xa5a5a5a5
0xf62f0c 0xa5a5a5a5
0xf6333c 0xa5a5a5a5
0xf63f0c 0xa5a5a5a5
Your camera might have been affected by the Apr04 null pointer bug.
Please contact Magic Lantern developers on IRC (#magiclantern on freenode)
to take a closer look.
I have good ROM1 from magiclantern-crop3x.2017Mar21.5D3113.
Already on IRC
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pocketrubbish on April 08, 2017, 10:40:34 AM
I just put together a quick video demonstrating the new options .

https://youtu.be/9trjJKDdLO0

Pushed the saturation a bit during the grade because I wanted to see how it holds up. I'm impressed.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 08, 2017, 10:53:24 AM
Quote
Your camera appears OK, no need to worry.
Happy camper :)

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 08, 2017, 11:00:00 AM
Happy camper :)

Nice save, @Danne!  :) (will have to check mine first thing in AM after I get back home from work) -- good thing I only used the April 4th build on the 'B' cam that I just scooped up from Craigslist earlier this week and not my original baby, Ha!

Pushed the saturation a bit during the grade because I wanted to see how it holds up. I'm impressed.

Indeed, extremely impressive. What were the highest ISO's you were able to use for the night scenes @pocketrubbish?

My apologies for lack of paying attention to details in your clip as I was obviously all eyes on the static shots. Similar effect if one were to stare at a Television showing nothing but static and it just sucks you right in which means you got the job done well even if it wasnt intended for.

The reason why I asked (hope you're reading this @a1ex) is because I'm an avid fan of using DarkFraming average processes for high ISO shots and ever since this whole lossless compression 4k crop_rec stuff came out for the 5D3 -- haven't been able to figure out how or why when I apply DF avg process it just don't correspond properly and not sure if this is related to the modified mlv_dump spitting out different headers for each DNG or some sort?

I think because of this it's making it a bit trickier for the DF avg process to work properly (at least in @Danne's cr2hdr.app is what I use) so not sure if this at all possible due to the limitation factor of this new lossless compression codec in MLV 14-bit lossless or perhaps just a simply fix in the code would do the trick?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: vstrglv on April 08, 2017, 11:00:34 AM
This is what I just did to test it (for Windows):

    py testrom.py ROM1.bin
"python testrom.py ROM1.bin" works for me
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 08, 2017, 11:22:44 AM
Not sure if this would help anybody but I put in a little simplifying doubleclick command and a howto for mac users here.
https://bitbucket.org/Dannephoto/magic-lantern/downloads/ROM1test.zip

Basically the same for mac as for windows if running straight on the command line.

"python testrom.py ROM1.bin"
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 08, 2017, 11:23:09 AM
Your camera appears OK, no need to worry.

 :) :) :)

Thanks Danne
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: GutterPump on April 08, 2017, 11:49:34 AM
I'm feel quiet  8) :

(http://image.noelshack.com/fichiers/2017/14/1491644829-py.jpg)

Thanks to @a1ex for the script and thanks to @janvkem for explanations.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: RTI on April 08, 2017, 12:17:27 PM
Thx for the mini guide. "Your camera appears OK, no need to worry."
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: g3gg0 on April 08, 2017, 12:24:43 PM
@alex:
i guess its a good thing to add this checker to the module itself.
so all users get informed who potentially don't realize there are broken properties.
e.g. not crashing due to the property being bad, but weird image metadata or correction curves.

would warn them on display, telling them that the old version had a bug and this camera
seems to have a property being flipped.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Mefist on April 08, 2017, 01:14:45 PM
I just put together a quick video demonstrating the new options .

https://youtu.be/9trjJKDdLO0

Pushed the saturation a bit during the grade because I wanted to see how it holds up. I'm impressed.

How did you manage to shoot in 3072x1308 24fps at 40 seconds? Did not change the aspect ratio? I do not write 5dm3 in 3k 14bit-lossless 16:9 for more than 2 seconds
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 08, 2017, 01:29:28 PM
@A1ex Ran the Python test and my camera was not affected!!!!!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: ewinemiller on April 08, 2017, 01:52:47 PM
How did you manage to shoot in 3072x1308 24fps at 40 seconds? Did not change the aspect ratio? I do not write 5dm3 in 3k 14bit-lossless 16:9 for more than 2 seconds


16x9 would be more like 3072x1728. The best I've been able to do continuous is 3072x1536.

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Mefist on April 08, 2017, 03:29:49 PM
16x9 would be more like 3072x1728. The best I've been able to do continuous is 3072x1536.

I have 5dm3 firmware 1.1.3 CF SanDisk 64GB 160Mb/s
My settings: 3072x1308 with the aspect ratio of 2.35:1, Crop mode 3K 1:1, 14bit-lossless. If I put a resolution higher, the recording automatically stops.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 08, 2017, 03:42:57 PM
I did several 4K recordings with 14bit lossless. However I noticed some fat blotchy areas of compression, easily detectable.
I was shooting with 50mm 1.8 and compression was applied naturally in out of focus areas.
I thought 14bit lossless was visually the same as the ordinary 14bit.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: g3gg0 on April 08, 2017, 03:58:32 PM
it definitely is the same. compression is lossless.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 08, 2017, 04:15:52 PM
I was using 4K 1:1
Selection PAL/NTSC mode or 1080/720p through Canon Menu, changing Aspect Ratio, was giving me some more resources for more FPS.
Then I switched to 1920 50/60p, 1920x1080.
I pressed REC and got a warning "EDMAC timeout"
Brick camera
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Lars Steenhoff on April 08, 2017, 04:56:22 PM
Did you take out the battery before you concluded the camera is bricked?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 08, 2017, 06:56:59 PM
Not sure if this would help anybody but I put in a little simplifying doubleclick command and a howto for mac users here.
https://bitbucket.org/Dannephoto/magic-lantern/downloads/ROM1test.zip

Downloaded & ran the ROM1test (Thanks for that @Danne) and I was even more surprised by the result because I HAD USED the 2017-04-04 build.

Hmm... what gives?

(https://c1.staticflickr.com/3/2848/33914393675_a8a4fd7d52.jpg) (https://flic.kr/p/TEU8Fr)

By any chance could it be possible for some of us to have used this particular build and NOT be affected by the null pointer bug at all @a1ex?  :P

Also felt this was worth sharing after running the 2017-04-06 experimental build for 5D3.123 in case they give out hints or anything.

(https://c1.staticflickr.com/3/2894/33531220290_0eab0f61d3.jpg) (https://flic.kr/p/T63gzL) (https://c1.staticflickr.com/4/3684/33786501351_b00a979cb1.jpg) (https://flic.kr/p/TtADKn)

Some small bug I also ran into upon when selecting 45 in FPS override (w 1080p45/48 3x3 selected within crop_rec.mo) which then freezes the LiveView more than half of the time.

(https://c1.staticflickr.com/3/2810/33786952001_ec813f47b7.jpg) (https://flic.kr/p/TtCXHc)

After each battery pull I had to immediately go back into FPS override and either put it back into 48p or turn it off to keep LV from freezing up again. This is reproducible.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Vegandelight on April 08, 2017, 06:58:07 PM
This might be a stupid question but should MLVFS work with the compressed 14bit files? im just getting pinkish static when trying to view them.

Tried the script on my camera and everything seems fine, i missed the April 4th update, tried all of the other builds ;)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 08, 2017, 07:14:21 PM
@ Lars Steenhoff

If the camera doesnt respond, on/off and I need to take off the battery, is considered as brick ?
Sorry if I didnt specify clearly :D
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 08, 2017, 09:22:06 PM
@Quentin: ROM backup done, right? If not, save the contents of ML/LOGS from the oldest ML card.

Then, get a fresh copy of your (bad) ROM with the portable ROM dumper (https://www.magiclantern.fm/forum/index.php?topic=16534.0) and contact me on IRC (online for the next 2-3 hours, but busy, then tomorrow at about the same time).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 08, 2017, 10:01:53 PM
@janvkem @Danne @a1ex Thank you guy's ran the test my camera seems to be Ok  :D
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: vstrglv on April 08, 2017, 11:21:08 PM
@Quentin: ROM backup done, right? If not, save the contents of ML/LOGS from the oldest ML card.

Then, get a fresh copy of your (bad) ROM with the portable ROM dumper (https://www.magiclantern.fm/forum/index.php?topic=16534.0) and contact me on IRC (online for the next 2-3 hours, but busy, then tomorrow at about the same time).
a1ex, how to connect to you in IRC#magiclantern? I login in IRC#magiclantern with my nickname, but there is no yours nickname in the nickname list.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Vegandelight on April 09, 2017, 12:27:49 AM
(http://i.imgur.com/J7DfURv.png)
1920x960 50fps 3x3 pixel binning gives me static on all footage. Am i doing something wrong here?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: dmilligan on April 09, 2017, 01:36:53 AM
Quote
To decode, you will need a custom mlv_dump (also included). It works by simply dumping the lossless JPEG payload to DNG (that means, processing options such as bad pixel or vertical stripe fixes are not working yet).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: squig on April 09, 2017, 03:56:57 AM
Am i doing something wrong here?

Yeah, you're using MLVFS, it hasn't been updated for these experimental builds. Use this: http://www.magiclantern.fm/forum/index.php?topic=15108.0 or mlv_dump.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 09, 2017, 11:06:20 AM
I wanted to record h.264, my daughter with her new bicycle. No RAW, no anything.
Crop Mode and Raw Recording werent loaded.
The camera was recovered after a bad crash and removed the battery.
I never reset to ML defaults, or reload the modules as the usual procedure.

I shot 4 takes.
The two were MLVs.
I havent touched anything.
RAW recording enabled itself somehow
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: khamees1975 on April 09, 2017, 01:46:37 PM
Hi Alex and every one
i am so sorry but my english very bad !!!
how i can upgrade my camera ?
is't free or i have to pay or what i am new here in this forum i don't get every thing ????
thank you
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Vegandelight on April 09, 2017, 01:55:35 PM
Thanks for the response. Finally got a set of DNG's, wish they worked directly in Premiere.

Keep up the good work.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: jankrueck on April 09, 2017, 04:27:47 PM
(https://c1.staticflickr.com/3/2894/33531220290_0eab0f61d3.jpg) (https://flic.kr/p/T63gzL) (https://c1.staticflickr.com/4/3684/33786501351_b00a979cb1.jpg) (https://flic.kr/p/TtADKn)

hey @a1ex
is this any indicator for a bricked camera?
because this shows up to me, even though the script says, everthing is ok.

I have some troubles running those scripts.
not sure if I do it right. (I just take ROM1.bin from sd card and scan it via script.)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 09, 2017, 04:30:06 PM
I wanted to record h.264, my daughter with her new bicycle. No RAW, no anything.
Crop Mode and Raw Recording werent loaded.
The camera was recovered after a bad crash and removed the battery.
I never reset to ML defaults, or reload the modules as the usual procedure.

I shot 4 takes.
The two were MLVs.
I havent touched anything.
RAW recording enabled itself somehow

Was ML still in the camera?
After you pull the battery​, on the next power up the modules won't load. If you power down then power back up again, the modules will load again with your settings and all. Could this be what happened?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 09, 2017, 06:51:39 PM
After removing the battery, its was an up normal shutdown.
I havent touched anything.
Next day I turn on the camera to shoot my daughter.
I saw that the modules didnt load. Τhe setting werent reset.
I shot h.264, (I thought)
When I load the CF card into the computer, I was surprised.
There were 4 videos, two MOV and two MLV.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 09, 2017, 07:18:46 PM
@quentin  Do you remember what the record symbol looked like for any of the clips? Was it the Canon rec icon or did the cam beep then record like when it does for RAW. Also did you try processing the MLVs ? And did you test your Rom with the Python script?
You could also try recording again. Making sure that the mods are disabled for sure.
Remember that this build is bleeding edge.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 09, 2017, 07:28:40 PM
The recording icon was normal, h.264 and MLV however I didnt pay attention to the change.
I processed the MLVs, they were normal.
Nope, unfortunately I have no clue about python scripts, ROM processing etc.

I just checked the camera now.
Modules restored as the crash never happened.
Definitely I didnt reset to defaults and didnt select which modules to load.
I am familiar with this first-aid procedure and did it many many times.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: GutterPump on April 09, 2017, 07:33:15 PM
I did a try too :

https://vimeo.com/212477560

3009 x 1280  + 14 bits lossless = continuous shooting.
Just don't shot in ETTR, keep an eye on the compression percentage and check the histogram and you're done.

I used the new amazing ew ML Cinelog-C to Ektar 100 LANDSCAPE lut of hyalinejim (http://www.magiclantern.fm/forum/index.php?topic=19338.msg182770;topicseen#msg182770)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: aschille84 on April 09, 2017, 08:30:40 PM
I did a try too :

@gutterpump you make art out of a camera test. Beautiful!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: GutterPump on April 09, 2017, 09:32:47 PM
@aschille84 glad to know you like it !
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Greg on April 09, 2017, 10:50:34 PM
If anyone is interested I think we have everything to do it for 500D (Full res LV others will be useless - too slow sensor).
3 years ago I increased the width, but I did not know how to increase the height.

I suppose these are the registers (my old notes) :
Code: [Select]
ADTG[2]1172:     0x3d5 (was 0x667)
ADTG[2]1173:     0x39b (was 0x6e6)
ADTG[2]1178:     0x3d5 (was 0x667)
ADTG[2]1179:     0x39b (was 0x6e6)
ADTG[2]1214:     0x3de (was 0x622)

So look at crop_rec.c - /* some modes require additional height adjustments */
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 09, 2017, 10:55:37 PM
I,m interested. Got a couple of 500D cams at work :) Could you prepare a build for testing?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Greg on April 09, 2017, 11:04:39 PM
Sorry it will not be possible.
It's just a hint for someone like reddeercity but with 500D.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 09, 2017, 11:06:51 PM
I see. Thanks. Short on time over here.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: reddeercity on April 10, 2017, 12:50:21 AM
It's just a hint for someone like reddeercity but with 500D.
If anyone is interested I think we have everything to do it for 500D (Full res LV others will be useless - too slow sensor).
3 years ago I increased the width, but I did not know how to increase the height.

I suppose these are the registers (my old notes) :
Code: [Select]
ADTG[2]1172:     0x3d5 (was 0x667)
ADTG[2]1173:     0x39b (was 0x6e6)
ADTG[2]1178:     0x3d5 (was 0x667)
ADTG[2]1179:     0x39b (was 0x6e6)
ADTG[2]1214:     0x3de (was 0x622)

So look at crop_rec.c - /* some modes require additional height adjustments */

Very useful , thanks  :D I was wonder on this , I was going to pm you if you have any notes that could help.
I still can't reproduce you test in the 500d , registers are different on 5d2 but close thou.
my problems is the timer A can't lower it lock to 610 , and ever time I try agjusting the Headtimer
I get a frozen camera/lockup (battery pull)  so if I can get timer A to lower I think I will have it
I have the width , but compressed to the left in a 2K crop window , anyways thanks it helps .
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Greg on April 10, 2017, 02:48:30 AM
I'm afraid there may be a problem with raw buffer. Something like 10/12bit raw video with old cameras.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Teamsleepkid on April 10, 2017, 03:17:20 AM
@greg think it will be possible on eos m? I know it won't do 4K or 3k but any resolution boost would be good.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: mk11174 on April 10, 2017, 05:03:25 AM
This is where I am at with 700D so far, just trying to figure out how to change height without it getting all scrambled, was able to change width past 4096 but was getting borders so brought it down to an even number, am able to go to almost 1500 on height until it scrambles, so I am assuming other registers need to be adjusted to make it work.

Also, fps drops to 6.5fps with the register changes to make it work, was able to force it to 9fps with C0F0[6014], any higher caused some sort of overlap on image.

Here are some examples of resolutions I would get by adjusting C0F0[6804]. It seems as I adjust the 3 on the left, the height would change, the 3 on the right, the width would change, the middle 0 seems it has to stay 0. I can go much higher on the height, but as I said, it gets crazy looking, shifty and predator vision like.
Code: [Select]
4550204   1920 x 1080
4540298   1928 x 1144
4e40448   4248 x 1217   
4cc0422   4096 x 1193
45b0422   4096 x 1080  current
50b0564   5384 x 1256

(https://s29.postimg.org/we3eb6yuv/4096x1080.jpg)

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: mk11174 on April 10, 2017, 06:45:27 AM
I managed to get height going by adjusting
Code: [Select]
6804  0xc3b0422     4096x3096
713c  0x4a7 to 0xca7
8172  0x6fd to 0x8fd
8178  0x6fd to 0x8fd

But I end up with this image, not sure how to fix this????
(https://s10.postimg.org/mbk2633ax/Untitled-1.jpg)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: dfort on April 10, 2017, 08:00:03 AM
I was thinking about looking into the EOSM and 700D. Since there are others out there that are willing to fry their sensors to get 4k working I posted adtg_gui builds for these cameras.

https://bitbucket.org/daniel_fort/magic-lantern/downloads/

@mk11174 -- Did you already get 14-bit lossless compression working?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 10, 2017, 11:12:36 AM
is this any indicator for a bricked camera?

A bricked camera is easy to identify - it does not boot (in other words, it's as useful as a brick). So far, there were two cases, both successfully recovered.

However, the Apr04 null pointer bug might (or might not) have changed some of your settings somehow. Since you have not noticed any side effect yet, your camera is probably OK, or you may find out some strange behavior when using some functionality that you don't use very often. That doesn't count as "bricked", but still, those (persistent) changes that appeared because of that bug are not desirable.

By any chance could it be possible for some of us to have used this particular build and NOT be affected by the null pointer bug at all @a1ex?  :P

Of course - what happened was not deterministic.

I've just posted a new build that checks whether your camera was affected by this bug. All you have to do is to upgrade - if no message pops up, your camera was not affected by that bug at all. Otherwise, just follow the instructions.

For this particular bug, the persistent changes are easy to identify, because the value that could get written into ROM is always the same - 0xA5A5A5A5.

Note that any other code (from ML and even from Canon) can cause similar issues (and probably much harder to identify). We have patched one such issue many years ago - on most camera models, a bug in Canon's startup code overwrites one single (predictable) memory address with 0 (4 bytes). You can check it by looking for ARMLIB_OVERFLOWING_BUFFER and tracing the patched address in QEMU (e.g. set a value at startup and watch it getting overwritten). In some camera models, this address ended up right in the middle of ML code (thus causing unpredictable behavior that changed with... the amount of code before the affected spot - e.g. insert a printf and watch things working in a totally different way).

This is a limitation of the current CPU and operating system - any task is allowed to write at any memory address. On top of that, Canon code saves their settings to ROM (by reflashing), so all it takes to soft-brick a camera is a programming mistake that results in writing at the wrong memory address.

I can think of a run-time check that detects any writes to Canon's property data structures (some of these are the persistent settings that get flashed into ROM) from tasks other than (Canon's) PropMgr, but it will definitely have a noticeable overhead (impact on performance). I can also think about adapting g3gg0's mem_prot module (which, back then when g3gg0 wrote it, was way above my level of understanding), but this won't catch badly configured DMA transfers. I can also imagine a routine that would check the integrity of Canon's property data structure right before they get saved into ROM (and maybe lock up the camera if problems are detected).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: jankrueck on April 10, 2017, 12:15:21 PM
nice work @a1ex. I tested new build and now message pops up.
I was/am curious about this error/bug saying cache is locked.

I also can't get 1920x1920 working, can't get any higher than 1728px.
Before the april6 I was able to record square footage.
Not sure if you changed something, or what happend.
(http://i.imgur.com/Q5dA739.jpg)


Also If I do 3k ISO400 I get a crash when there is to much detail moving.
(http://i.imgur.com/27HLwt7.jpg)
Code: [Select]
ML ASSERT:
0
at mlv_lite.c:2036 (compress_task), task compress_task
lv:1 mode:3


Magic Lantern version : crop_rec_4k.2017Apr10.5D3113
Mercurial changeset   : 2bbf0bbbaa74 (crop_rec_4k) tip
Built on 2017-04-10 08:54:19 UTC by [email protected]
Free Memory  : 164K + 3162K


hope my data is helping you in any way. if not, just tell me to stop it :D


cheers!

ps: Is there a chance to get some reference sound back in 3k mode, to have better syncing in post?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 10, 2017, 12:37:34 PM
See this post (http://www.magiclantern.fm/forum/index.php?topic=19300.msg182619#msg182619) for the red dot and the first post for resolution:

Quote
* 1920x1920 @ 24p (1:1 square crop) - continuous*)
...
Code: [Select]
                                /*   24p   25p   30p   50p   60p */
    [CROP_PRESET_3X_TALL]       = { 1920, 1728, 1536,  960,  800 }, /* 1920 */

ps: Is there a chance to get some reference sound back in 3k mode, to have better syncing in post?

I'll probably stop adding features for a while and will look into ways for preventing this class of bugs.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: jankrueck on April 10, 2017, 12:58:47 PM
uuuhm sorry for the timewaste.. I somwhow skipped page 8 completly.
It took me a while to understand the resolution"issue"
I probably switched to 25p whithout noticing this will affect something ;)


maybe I have to get a filmclap, till you find time to add soundrecording ;)



Cheers and thanks!!!!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: mk11174 on April 10, 2017, 02:33:10 PM
Quote
Posted by: dfort
@mk11174 -- Did you already get 14-bit lossless compression working?

I have only been working with resolution changes for fun just to see what happens and of course sharing what I find.  :)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Markus on April 10, 2017, 06:30:05 PM
Tried some of the new experimental modes on Fw 123 today. Looked really good but i could not get 1920x1080 45p continious on my Komputerbay 1066x I got around 10-20sek rec time. Has anyone achieved 45p continuous on FW 123? 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 10, 2017, 07:11:00 PM
Of course - what happened was not deterministic.

I've just posted a new build that checks whether your camera was affected by this bug. All you have to do is to upgrade - if no message pops up, your camera was not affected by that bug at all. Otherwise, just follow the instructions.

Fortunately, no messages prompted up after upgrade w 2017-04-10 build though I am curious to see what the instructions would be for the affected ones.

Thanks for double checking @a1ex!  ;)

Actually decided to try this (since it was reproducible last time) again w 1080p 45/48 3x3 under Crop mode while in 60p from Canon menu together with FPS override set to 45p and can now confirm that this bug has either probably gotten worse or I'm doing something incorrectly (which I doubt) but feel free to tell otherwise.

(https://c1.staticflickr.com/3/2934/33828393021_6e5e8b3f96.jpg) (https://flic.kr/p/TximFx)

Hence the reason why I had to use an iPhone to snap this screenshot because no matter how quick I can try getting back into ML menu upon restart (after each battery pull) and getting to scroll around (both Canon or ML menus) within 5 seconds more or less was impractical. It just freezes up the LV.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 10, 2017, 07:31:41 PM
Tried some of the new experimental modes on Fw 123 today. Looked really good but i could not get 1920x1080 45p continious on my Komputerbay 1066x I got around 10-20sek rec time. Has anyone achieved 45p continuous on FW 123?

How were you able to achieve 15-20 seconds without the LV display freezing up within 5 seconds more or less @Markus?

Meanwhile gonna try defaulting Canon settings to defaults and reload a fresh 2017-04-10 experimental build again just to double check.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 10, 2017, 07:41:09 PM
For now, FPS override only works in Low Light mode for reducing the frame rate (known bug).

For 1920x1080 at 45 fps, set 50 fps in Canon menu.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 10, 2017, 07:43:06 PM
@jankrueck I got the same thing Compression crash recording @ 3K running April 6th build.   
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 10, 2017, 07:51:09 PM
@A1ex Gonna try April 10th build today :D  Thank you once again for all your hard work..
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Markus on April 10, 2017, 08:03:01 PM
No lockups here. Only hard to get cont rec on alot of the sweetspot resolutions.
Some settings of mine:
Global draw on, canon in 720p all-I 50p mode, croprec 3x3 binning 1920x1080 47/48p @ 45p. Lossless 14bit.
Without global draw i could not get a bw preview in this mode.
Anything that sticks out?

The file size estimation got confused some times when switching between modes. It also got the wrong fps when switching. I went to photo mode and back as a workaround when this happend.
 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 10, 2017, 08:28:33 PM
For now, FPS override only works in Low Light mode for reducing the frame rate (known bug).

For 1920x1080 at 45 fps, set 50 fps in Canon menu.

That did the trick. Haven't use PAL in years (I'm on NTSC land). Should have thought of that to begin with and Thanks for the tips.

So far 1920x1080 @ 45p MLV-lossless 14-bit seems to be continuous even w HDMI out to an monitor (clean native feed :D) which is a PLUS for gimbal work because obviously the 5D3 will have the grey previews while mounted on the Ronin (which will go off from 'Turn off LCD' within 5 seconds during each recordings -- useful feature from 'Powersave in LiveView' under Prefs within ML menu) and can then focus on the external monitor instead with a peace of mind!  8)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 10, 2017, 08:46:16 PM
@a1ex Just tested April 10th Build so far so good.  One thing that I've noticed is that I can record at higher resolutions for longer now for example 3520x1498 - 2.35:1 I can get over 12 sec on a Lexar 128GB 1066x card before it would record for like 2 sec on the April 6th Build great work.  Also one thing that I've noticed is LiveView in 3K mode when recording @ 2.35:1 is all grayscale now, on the April 6th Build it showed both color and grayscale. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 10, 2017, 09:08:03 PM
Any differences between Apr10 and Apr06, except the null pointer check performed at camera startup, are your own imagination (review the changes, please).

Or, do a controlled test. Same static scene, same settings (!), run 10 test clips on both builds (alternating the builds), and then draw the conclusion.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 10, 2017, 09:21:49 PM
@a1ex Cool no problem.. @ camera startup I don't see any test running am I missing something, or it just runs in the background and if anything is wrong it flashes a message. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 10, 2017, 09:49:16 PM
Upgrade camera again via Canon menu and wait for either a pop up screen notifying instructions or none will be prompt (meaning it isn't affected) @goldenchild9to5 if you haven't already.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 10, 2017, 09:52:34 PM
So far, I've got 3 users with warnings about the null pointer bug (besides the 2+1 ones already solved over the weekend). Two of the new users had a false alarm (the modified value from ROM not affecting the functionality at all, being in unused areas), and the third user had his camera successfully patched.

Updated the tests (both python script and ML build) to ignore the false warnings I've encountered.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 10, 2017, 10:19:00 PM
Awesome!!!!! @a1ex!!!!!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 10, 2017, 10:34:07 PM
Tried some of the new experimental modes on Fw 123 today. Looked really good but i could not get 1920x1080 45p continious on my Komputerbay 1066x I got around 10-20sek rec time. Has anyone achieved 45p continuous on FW 123? 

Yes. (http://www.magiclantern.fm/forum/index.php?topic=19300.msg182320#msg182320)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: vstrglv on April 10, 2017, 10:54:45 PM
Updated the tests (both python script and ML build) to ignore the false warnings I've encountered.
A1ex, where can i copy updated python script?
 Many thanks.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 10, 2017, 11:09:59 PM
@DeafEyeJedi Upgraded ML, or installed ML from the canon menu a second time Didn't see no pop-up screen all it did was install ML again.  So it seems like my camera is ok than. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 11, 2017, 02:34:18 AM
WOW.. Getting incredible images @ 3K  :D
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 11, 2017, 04:36:02 AM
Using April 10th build. Testing 3072 x 1728
Got
ML ASSERT:
RAW_IS_IDLE
at mlv_lite.c:584 (measure_compression_ratio), task shoot_task
lv:1 mode:3


Magic Lantern version : crop_rec_4k.2017Apr10.5D3113
Mercurial changeset   : 11f405b62b31 (crop_rec_4k) tip
Built on 2017-04-10 19:48:55 UTC by [email protected]
Free Memory  : 163K + 3118K

Pulled Battery then restarted and reloaded mods.
The only things that I don't like are the current gray scale that it jumps to when it records (I know its purpose is to reserve resources)
The inconsistency of clip length due to changing compression.
And I find I cant get over 4 seconds on average with a 128 GB Lexar 1066x 160MB/s card.
I know this is very experimental I am just reporting my findings so far.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: csound on April 11, 2017, 05:13:39 AM
Ran ML April 10 4K test my 5D mk3 and is indicating the problem of null pointer bug from April 4 ML 4K experimental. Reverted to latest nightly 1.2.3 2017-3-30 build which seems to work fine. Casual user of ML & not sure if I need to do anything to address potential bug?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Markus on April 11, 2017, 10:50:48 AM
Quote
Tried some of the new experimental modes on Fw 123 today. Looked really good but i could not get 1920x1080 45p continious on my Komputerbay 1066x I got around 10-20sek rec time. Has anyone achieved 45p continuous on FW 123? 

I failed to recognize how noise at higher isos gives less effective compression. 1920 1:85 45p at  iso 800 seemes to be max cont rec for me.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: togg on April 11, 2017, 01:06:26 PM
I was wondering what this means, this feature will be merged into the silent pictures module? How easy will it be to trigger the shots?


Quote
* Full-resolution LiveView: 5796x3870 at 7.4 fps (128ms rolling shutter) - continuous*) at 5 FPS!

The last feature complements the well-known full-resolution silent pictures - the new implementation will be usable at fast shutter speeds, without the exposure gradient - but with rolling shutter (just like regular LiveView frames).


Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: beauchampy on April 11, 2017, 02:15:27 PM
Shot a card full of 50p / 960 / 3x3, 113 at the beach and unfortunately every clip has a tearing across the top of the frame :-(

This was processed in cr2hdr 4k and it is the April 06th build. I have live view set to real time if that makes a difference..

https://vimeo.com/212739433

Edit: I also kept getting error messages. Recording with the error log still on the screen resulted in the old 1.67x stretch slow motion clips.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 11, 2017, 03:21:43 PM
That footage is recorded unstretched right? So cr2hdr.app shouldn,t add the compensating destretching tag to the dng file. I could maybe put in a "not stretched footage" in the menu so it would leave the tag alone.
Maybe the info is already in the rawc metadata? Don,t think compressed_raw mlv_dump keep that code yet?
Can you upload asmall mlvv sample Beauchampy?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: beauchampy on April 11, 2017, 03:24:41 PM
That footage is recorded unstretched right? So cr2hdr.app shouldn,t add the compensating destretching tag to the dng file. I could maybe put in a "not stretched footage" in the menu so it would leave the tag alone.

Yes, it's unstretched 960 50p 3x3 binning.

Do you think this is cr2hdr rather than ML?

Edit; I also tried turning the preview to auto (switched to greyscale low fps) and the problem persisted.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 11, 2017, 03:26:57 PM
Definately tag added not ML. Could you upload a short sample? In latest cr2hdr.app you can create a sample package in mlv_dump menu. Selection (12).
Latest version also combines both compressed and non compressed versions of mlv_dump so no need for any 4k version now.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: beauchampy on April 11, 2017, 03:41:52 PM
Definately tag added not ML. Could you upload a short sample? In latest cr2hdr.app you can create a sample package in mlv_dump menu. Selection (12).
Latest version also combines both compressed and non compressed versions of mlv_dump so no need for any 4k version now.

Here's a problematic sample MLV (sorry, didn't have an option 12 in mlv dump settings on my cr2hdr?)

https://drive.google.com/open?id=0B-GY59oq1k7eM2xtVVhkZFRFUEk

Is this where you upload the latest builds for mac?
https://bitbucket.org/Dannephoto/cr2hdr/downloads/


EDIT: Tried looking at the same sample file in 'Footage' and the same tearing occurs.. Not sure it is cr2hdr? :-/
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 11, 2017, 03:53:03 PM
Tag is added by mlv_dump(ml-dng version, dng.c code from dmilligan).
Could you upload the zip package? Much smaller in size. I have no good internet connection.

Quote
Is this where you upload the latest builds for mac?
yes.
https://bitbucket.org/Dannephoto/cr2hdr/downloads/
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: beauchampy on April 11, 2017, 04:08:47 PM
Tag is added by mlv_dump(ml-dng version, dng.c code from dmilligan).
Could you upload the zip package? Much smaller in size. I have no good internet connection.
yes.
https://bitbucket.org/Dannephoto/cr2hdr/downloads/

Here you go, thanks for your help!

https://drive.google.com/open?id=0B-GY59oq1k7eb2tDNm9VcEJnTmc
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 11, 2017, 04:21:34 PM
Thanks. I take a dive into this later tonight.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: beauchampy on April 11, 2017, 04:27:00 PM
Thanks. I take a dive into this later tonight.

Thanks mate I really apprecite that. Just FYI I just tried 123 and the latest April 10th build and the issue persists.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 11, 2017, 04:29:10 PM
Ok, just took a quick look
Quote
...old 1.67x stretch slow motion
I took this as you were getting stretched footage? Anyway, the tearing issue is as you say something coming from the compress raw itself not from mlv_dump. About stretching issue it all looks ok to me.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: beauchampy on April 11, 2017, 04:35:01 PM
Ok, just took a quick lookI took this as you were getting stretched footage? Anyway, the tearing issue is as you say something coming from the compress raw itself not from mlv_dump. About stretching issue it all looks ok to me.

That was only happening when my screen suddenly populated with errors and logs. I think crop_rec had crashed. When it was working properly (non stretched 3x3 50p @ 960px) I get the line tearing.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: beauchampy on April 11, 2017, 04:39:19 PM
Anyone able to get 50p 3x3 1920x960 working without the frame tearing issue I am experiencing?   :-\
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 11, 2017, 04:51:35 PM
@beauchampy I got 1920 x 1080 48p 3x3 without tearing but rec time was inconsistent.
I also got 1920 2.35:1 @ 48p without tearing. That was on the very first build

However, on the 4-10-2017 build I took about 5 or 6 clips of 3072 x 1728 @ 23.976 last night and notice that 1 frame out of all the clips had a small horizontal row of pixels that seemed like it was offset from the rest. Ill upload if I get a chance.

I was using RAWFlow with an updated MLV_DUMP to convert 14 Bit lossless to DNGS.

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 11, 2017, 05:03:14 PM
@beauchampy: known bug, but right now, figuring out how prevent the recent soft-bricks has a higher priority.

Until then, my advice would be to try the Apr03 or older builds (which are double-buffered, which avoids tearing, but Frozen LV is not working well, so recording times may be shorter).

Do not use Apr04.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Frank7D on April 11, 2017, 05:06:12 PM
As far as the frame tearing issue, I saw it on nightly builds a while back on my 7D in crop mode and eliminated it by not using the accurate preview (ml greyscale). It was suggested that processor overload was responsible.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 11, 2017, 05:14:25 PM
The 7D issue does not apply here - on 7D, the issue is caused by the preview backend (and for the moment, I don't have any solution other than disabling the previews completely).

Here, the issue is my single-buffering experiment (which gives a little more memory for high-res modes, but apparently it doesn't play nice with high-FPS modes).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 11, 2017, 05:54:18 PM
@beauchampy
I'm really interested in 192050p 3x3 mode and while I wait for a solution for the tearing issue, for me the solution is 1080p45 3x3 mode. Not same slowmotion like 50p but no tearing issue and continuous recording. Tested at ISO3200 and it works for me continuous!!!. Only framing is a little bit offset...
5Dmk3 113 Lexar 64Gb 1066x
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 11, 2017, 06:19:41 PM
Tested:
April3th build 1920x960 50p 3x3 mode NO TEARING ISSUE!!!! :) :) :) Continuous at ISO100 and 400. At ISO800 shorter recording times (aprox 20sec. for me).
5Dmk3 113 Lexar64Gb1066x
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 11, 2017, 06:28:46 PM
So I can use a little education here. Sorry Im behind. If I get the:
ML ASSERT:
RAW_IS_IDLE
at mlv_lite.c:584 (measure_compression_ratio), task shoot_task
lv:1 mode:3
where the text appears on the screen, is this a crash? I have been assuming it is and pulling the battery. Should I be doing this?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 11, 2017, 07:10:49 PM
Here's a 5D3.123 build from 2017-04-03 for those that want to try and confirm...

https://bitbucket.org/DeafEyeJedi/magic-lantern/downloads/magiclantern-crop_rec_4k_2017April03.5D3123.zip
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: JackDaniel412 on April 11, 2017, 07:41:12 PM
Hello @a1ex , I ran the ROM1 test, no error found.
I updated ML to the April 10th, in debug section show this error

(http://dcmakecreative.altervista.org/VRAM0.png)

There is a way to fix it?

Thanks!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 11, 2017, 07:58:18 PM
is this a crash? I have been assuming it is and pulling the battery. Should I be doing this?

Yes, taking the battery out after a crash is what I always do.

Unfortunately, it doesn't appear to prevent Canon code from saving corrupted settings into the ROM. Still looking into it.

edit: opening the battery door seems to be interpreted as some sort of emergency shutdown, but a clean one - settings are saved into ROM, but ML shutdown hooks (e.g. saving ML config files) are not executed. But, if you somehow hold the battery door switch pressed, and just remove the battery, the ROM is not updated.

@JackDaniel412: asked (http://www.magiclantern.fm/forum/index.php?topic=19300.msg182790#msg182790) and (http://www.rookiemoms.com/try-this-with-me-asked-and-answered-to-end-little-kid-nagging/) answered (http://www.magiclantern.fm/forum/index.php?topic=19300.msg182619#msg182619) ;)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 11, 2017, 08:29:49 PM
Re: tearing I noticed this happening after the 2017-04-06 build and still persist with 2017-04-10 build as well when shooting in 1080p 3x3 45fps (50p Canon) and could this strange pattern of corrupted (plus dropped?) frames in several MLV's that spat out be related to the tearing at all?

Because my instincts are thinking that it may have started it after I had a bunch of random corrupted footage similar to the one below (is this what you were seeing also @beauchampy?) otherwise correct me if I'm wrong.

https://vimeo.com/212794399

Re: 2017-04-03 build I am now getting corrupted frames similar to the one above when shooting in 1080p 3x3 45p until I reduce the resolution down to 1920x960p which all seems fine.

Though on the older build I seem to be able to get clean recording w full 1920x1080p in 3x3 45fps which is rather confusing. Gonna need to take a step back and check again with all of this before I lose it for good. literally.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: beauchampy on April 11, 2017, 09:15:09 PM
@beauchampy: known bug, but right now, figuring out how prevent the recent soft-bricks has a higher priority.

Until then, my advice would be to try the Apr03 or older builds (which are double-buffered, which avoids tearing, but Frozen LV is not working well, so recording times may be shorter).

Do not use Apr04.

Can confirm Apr03 build is working correctly. Thanks a1ex!

No problem buddy, you take your time. Thanks for all your hard work, these updates are amazing.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: jordanwalkert on April 11, 2017, 09:16:48 PM
Hey a1ex,
With the latest build from April 10th (the one to ignore false warnings), I am still getting a message about the null bug. I have not run into any major issues or soft bricking, but I also do not have a backup or unaffected copy of my ROM from before the 4th. Am I screwed, any solution to this without a clean ROM from before?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 11, 2017, 09:22:42 PM
In many cases, I can infer the missing values from another ROMs (so it's not a very hard requirement).

BTW, your existing copy from ML/LOGS is likely to be a good one (as it's only re-created after a format).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 11, 2017, 09:26:11 PM
Yes, taking the battery out after a crash is what I always do.

Unfortunately, it doesn't appear to prevent Canon code from saving corrupted settings into the ROM. Still looking into it.

edit: opening the battery door seems to be interpreted as some sort of emergency shutdown, but a clean one - settings are saved into ROM, but ML shutdown hooks (e.g. saving ML config files) are not executed. But, if you somehow hold the battery door switch pressed, and just remove the battery, the ROM is not updated.

@a1ex Hmmmmm So if I mod the switch with say.... tape, I can pull battery with out afflicted ROM being saved?
Should I do this as a extra precaution when heavy testing?
Would this have adverse affects on Canon code being that its an "unsafe" battery pull at that point? 
So many questions sorry.
I suppose there is always a chance I might not even know that the ROM is affected and I could clean shutdown anyway. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 11, 2017, 09:31:43 PM
@Jordanwalkert
Did you save the roms from when installing the actual april 4th build? Those should be the last uncorrepted ones if so...
looong shot
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 11, 2017, 09:37:44 PM
More testing with april3th. 5Dmk3 113
1920x960 50p 3x3.
Now I'm recording continuous at ISO3200.
I'm a little confused... Maybe it depends on the detail level of the image?...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: giarcpnw on April 11, 2017, 10:18:35 PM
I may have missed this but once you set the settings for 3k 1:1 crop, what's the diff between the regular record mode and then x5 crop mode? They seem the seem on preview but regular mode will record around 3072x1228 but x5 Crop just stops. Is it cranking the data rate or something?

And does FPS override make any difference in any of thi?


Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 11, 2017, 10:21:42 PM
Now I've got two consecutive errors... Error 70, then I turn off camera. Next an  Error 80. Camera does not show live view after both errors. After a new reboot all is apparently ok...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 11, 2017, 10:49:02 PM
Another issue in 1920x960 50p 3x3 is framing.

Here there is a composition comparing real image vs live view framing:
http://ovh.to/TpLuMh (http://ovh.to/TpLuMh)
Title: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 12, 2017, 01:06:30 AM
Can anyone else confirm that it seems when you go into Full-res LiveView (it becomes 2+ stops brighter according to the LV as well as the ML histogram) than if in any other settings within crop mode from the latest experimental build?

If so, is this normal or a bug?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 12, 2017, 01:21:31 AM
Probably it's because of the shutter speed mapping from 1/4000 ... 1/30 to 1/15000 ... 1/FPS (also used on 4K half-fps).

Suggestions for a better (less confusing) implementation?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 12, 2017, 03:47:48 AM
Here is a quick test I did using 5D Mark III in 3k - 3:1 / Build April 10th latest one.  I just Love the images that I'm getting & specially the motion, the images speak for themselves even when youtube super compresses it.  Let me know what you guy's think. 

https://www.youtube.com/watch?v=OmxEjoruetw&feature=youtu.be
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 12, 2017, 06:01:20 AM
When I find it, I am glad.
When I need it, I cant find it :(
Where is the menu option for setting the Preview ?
Thanks for your help
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: D_Odell on April 12, 2017, 11:00:50 AM
I tried today with April10th build. Since I normally shoot 1.85:1 I tried 3K 1:1 (2928 × 1580), ISO 640, shutter 1/50. But the cam can only record for 5 sec with 14 bit lossless. Is that normal? Or how to be able to record constantly with this resolution, if possible? CF card is Sandisk 160mb r/w. Thanks for al the heavy work and fun to reach new heights!

DNG: https://www.dropbox.com/s/tz79q88ozc8f5p4/M12-0944_000004.dng?dl=0
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Walter Schulz on April 12, 2017, 11:06:53 AM
Bandwidth required:
2928 x 1580 x 24 (?) x 14 / 8 = 185 MByte/s
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: D_Odell on April 12, 2017, 11:10:13 AM
Bandwidth required:
2928 x 1580 x 24 (?) x 14 / 8 = 185 MByte/s

Correct 23.976. Normal I watch ML recommendation in menu, and with these settings says 101.8MB/s at 23.976.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: mageye on April 12, 2017, 11:13:17 AM
Try recording at ISO100
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: D_Odell on April 12, 2017, 11:27:12 AM
Try recording at ISO100
Positive results, now 25 sec. Thanks! So reaching (for example) 5 min, lower resolution seems to be the only way?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: mageye on April 12, 2017, 11:37:12 AM
At the moment yes. But the noise from the ISO causes problems for the compression algorithm. I am getting a (reasonably) reliable resolution of 3648x1216 but still not completely continuous!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: D_Odell on April 12, 2017, 11:44:03 AM
At the moment yes. But the noise from the ISO causes problems for the compression algorithm. I am getting a (reasonably) reliable resolution of 3648x1216 but still not completely continuous!
I see, do you mind sharing your ML settings? Thanks!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: mageye on April 12, 2017, 12:00:04 PM
Crop mode UHD 1:1
RAW video ON, 3648x1216
Aspect 3:1
14-bit lossless


Within Canon menu:
Movie rec. size 1920 @ 24 ALL-I
Video System set to: PAL

That's it!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 12, 2017, 12:45:31 PM
Today I was doing a studio shooting for 4-5 hours, several shots.
I had my camera plugged on electricity as well as the monitor.
5d mk3, 1.1.3
Experimental Build 10 April
I used 1920x800 60FPS 16:9 14bit lossless
All manual, no AutoETTR
While I was working on the shot, to adjust lights etc, Camera crashed and saw several messages.
Turn it OFF/ON fixed the problem.
The recording was not continuous but sufficient enough to record the useful action.
I noticed that the Aspect Ratio of the monitoring wasnt right.
Later, I saw that the monitoring the colours lost saturation and the Aspect Ratio was jumping from wrong to right.
I also noticed that although the ISO was set manually to 1250, on ML monitor was still 1600.
The temperature of the camera was 40 C

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Markus on April 12, 2017, 01:19:32 PM

Monitor can't show the hacked bw preview on external monitor. And canons liveview does not give correct representation of ML enabled modes.

Iso in raw is only valid for real Iso values. 100,200,400,800 and so on. The values in between is just digital gain applied to  h264 compressed video.


Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Vegandelight on April 12, 2017, 05:18:27 PM
What kind of work flow are you guys using for the 1920x960 3x3 binning 14bit raw mode?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 12, 2017, 05:55:28 PM
What kind of work flow are you guys using for the 1920x960 3x3 binning 14bit raw mode?
In camera? Or post processing?
For post I used the RAWFlow app and just replaced the MLV_DUMP with the newest version and it processed the 14Bit Lossless just as expected. Drag n Drop.
RAWFlow - http://www.magiclantern.fm/forum/index.php?topic=13338.0
MLV_DUMP Update - https://builds.magiclantern.fm/experiments.html
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Vegandelight on April 12, 2017, 08:13:55 PM
In camera? Or post processing?
For post I used the RAWFlow app and just replaced the MLV_DUMP with the newest version and it processed the 14Bit Lossless just as expected. Drag n Drop.
RAWFlow - http://www.magiclantern.fm/forum/index.php?topic=13338.0
MLV_DUMP Update - https://builds.magiclantern.fm/experiments.html

Nice, Premiere or DaVinci Resolve? The old DNG-files opeened right up in Premiere but these compressed ones does not.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 12, 2017, 08:18:37 PM
Nice, Premiere or DaVinci Resolve? The old DNG-files opeened right up in Premiere but these compressed ones does not.
DaVinci opened them just fine. I don't know premiere.
Its and extra step but you can always export from resolve. Resolve + DNG = incredible color options!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: fkarpenko on April 12, 2017, 09:22:41 PM
Is there a chance we'll see 2k/3k/4k resolutions in 3x3? This is pretty amazing but without the massive crop factor it would be a dream come true.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 12, 2017, 09:39:15 PM
Consider this being a huge leap towards our next mind boggling goal:

8K RAW Video Recording!

Sample DNG from 5D Mark III, to show that our proof of concept is working:

8k.dng (http://a1ex.magiclantern.fm/2017Apr01/8k.dng)

All that's left is to divide the resolution by 3 on each axis ;)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 12, 2017, 09:39:41 PM
Has anyone notice an issue re: Dual-ISO 14-bit lossless MLV seems to be spitting out with correct black levels but incorrect for original 14-bit MLV's. Could this be related to the new mlv_dump not corresponding properly to cr2hdr or at least confused with each other atm?

(https://c1.staticflickr.com/4/3938/33870168051_2a8406905b.jpg) (https://flic.kr/p/TAZsWF) (https://c1.staticflickr.com/3/2926/33842109492_50757d012f.jpg) (https://flic.kr/p/TyvE7q)

Anyway here are some short samples in two of each (138 MB spat out w ease from latest cr2hdr.app -- Thanks @Danne!) and all shot within crop_rec @ 3.5k: https://mega.nz/#!OwV2DTLC!rb39cVOrAlXXuSer2M6fw3vSuvMfiSG508hewIQJvmg
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 12, 2017, 10:49:49 PM
New build posted.

Found a way to prevent Canon code from saving some of the settings at shutdown, including the settings block that caused issues earlier (reverse engineering notes here (http://www.magiclantern.fm/forum/index.php?topic=19369.msg182959;topicseen#msg182959)). While not perfect, if Apr04 had this safeguard, all the cases affected by the null pointer bug would have been caught. Still, there are ways to brick the camera, just a bit less likely to do so accidentally (and, as long as the bootloader is not erased, recoverable).

How it works: whenever a crash is identified, or whenever you open the battery door, Canon code no longer saves the usual setting groups at shutdown (RING and RASEN, if you look in the above link). There are still setting groups not covered by this safeguard (still looking into it, but so far, all the null pointer errors were in the RING group).

To test - the PAL/NTSC setting is in the RING group (and, to my knowledge, only saved at shutdown). You also have a dummy crash under Don't click me.

After a bit of battle-testing, I think this should be back-ported to all models.

Changes related to raw recording:
- double buffering is back (except at very high resolutions); didn't test much
- pre-recording indicator: mm:ss.f + ss.f
- full-res LV increases FPS when lowering vertical resolution (up to 5796x400 @ 48 fps)
- in 10x zoom, FPS override can be disabled by half-shutter to allow focusing
- minor fixes here and there (full list on the download page)

For preview with correct framing on external monitors, try Force HDMI VGA (not tested, but IIRC it used to help).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 12, 2017, 11:58:21 PM
WOW!!!!! @a1ex this is amazing! Cant wait to test!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 13, 2017, 01:00:48 AM
@a1ex Thanx for all the new updates.. Gonna give the new build a try tonight will post my results
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 13, 2017, 01:38:19 AM
Going to update to new build and found this on card from April 10 Build
ML ASSERT:
RAW_IS_IDLE
at mlv_lite.c:584 (measure_compression_ratio), task shoot_task
lv:1 mode:3


Magic Lantern version : crop_rec_4k.2017Apr10.5D3113
Mercurial changeset   : 11f405b62b31 (crop_rec_4k) tip
Built on 2017-04-10 19:48:55 UTC by [email protected]
Free Memory  : 163K + 3118K

Not sure if this needs to be reported.
Testing April 12 build now

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 13, 2017, 02:12:43 AM
Wait..... what did I do wrong here. I couldn't even get one solid rec out of the April 12 build so far. I had to pull the battery every time.
@a1ex Im sending you a message to a zip of my card its peppered with ASSERT messages.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: JADURCA on April 13, 2017, 02:17:13 AM
Hi everyone! I just saw there is a new version, in case it helps I'm sharing this 4 CRASH.LOG files from crop_rec_4k.2017Apr04.5D3123 version.

ASSERT: hLvJob->hJpegMemSuite
at ./Epp/Vram/VramStage.c:891, task Epp
lv:1 mode:3


Magic Lantern version : crop_rec_4k.2017Apr04.5D3123
Mercurial changeset   : e7e1fdb85f6b (crop_rec_4k) tip
Built on 2017-04-04 15:42:08 UTC by [email protected]
Free Memory  : 163K + 3108K

ASSERT: IsSuiteSignature( hSuite )
at ./PackMemory/PackMem.c:599, task Epp
lv:0 mode:3


Magic Lantern version : crop_rec_4k.2017Apr04.5D3123
Mercurial changeset   : e7e1fdb85f6b (crop_rec_4k) tip
Built on 2017-04-04 15:42:08 UTC by [email protected]
Free Memory  : 163K + 3107K


ASSERT: IsSuiteSignature( hSuite )
at ./PackMemory/PackMem.c:599, task RscMgr
lv:0 mode:3


Magic Lantern version : crop_rec_4k.2017Apr04.5D3123
Mercurial changeset   : e7e1fdb85f6b (crop_rec_4k) tip
Built on 2017-04-04 15:42:08 UTC by [email protected]
Free Memory  : 163K + 3107K


ASSERT: IsSuiteSignature( hSuite )
at ./PackMemory/PackMem.c:983, task RscMgr
lv:0 mode:3


Magic Lantern version : crop_rec_4k.2017Apr04.5D3123
Mercurial changeset   : e7e1fdb85f6b (crop_rec_4k) tip
Built on 2017-04-04 15:42:08 UTC by [email protected]
Free Memory  : 163K + 3108K
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 13, 2017, 03:40:56 AM
@JADURCA Dont use that build it's flawed.. make sure you back-up your ROM.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 13, 2017, 04:47:07 AM
@a1ex Just tried the April 12th build it's solid great work.  The only thing that I've noticed is when recording the minute / second green counter stays @ 00.00 it doen't show the actual recording time. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: KMikhail on April 13, 2017, 08:00:36 AM
Just wanted to give heads up to those who want most of resolution with lossless, but high ISO prevents you from it. You can sacrifice dynamic range but have a lower bandwidth requirement by using lower ISO and doing the pull in the post: the noise will be below the original gain line. Sure enough, it is worse than we'd like it, but...

Unfortunately, ISO 50 doesn't work with video, that would've basically removed one bit out of equation. Maybe there is another cheap (performance-wise) trick to lower effective number of bits, so lossless is more effective?

Here's a question: is there a way to have GD when I'm focusing, etc., but have it auto turned off (with everything else that could be turned off) during the recording?

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hindra on April 13, 2017, 10:28:30 AM
Just tested newest build from Latest Build (2017-04-12 20:31).

I got the red dot in the memory patches again, no big deal right? It seemed like most of my shots using crop_rec had a bad first frame regardless of resolution.  Shooting at 3504x1536 at 24fps I only got 157 frames with a Lexar 128gb 1066x card.

Also, something odd kept happening and it kept reverting back to 19fps with FPS override off.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 13, 2017, 11:07:58 AM
Maybe there is another cheap (performance-wise) trick to lower effective number of bits, so lossless is more effective?

Yes - adjusting the digital gain in the raw backend appears to help (as it reduces the range of the data and cuts off the noise bits). Added a (somewhat fake) lossless compression at reduced bit depths, based on this concept (theory explained here (http://www.magiclantern.fm/forum/index.php?topic=18443.msg181620#msg181620)).

Also added some suggestions about what ISO range to use with each option. See the commit for more details on how I've chosen the recommendations.

The only thing that I've noticed is when recording the minute / second green counter stays @ 00.00 it doen't show the actual recording time. 

Fixed - was broken by pre-recording updates.

@a1ex Im sending you a message to a zip of my card its peppered with ASSERT messages.

The issue seems to be in 1920x1080 48p. Hopefully fixed, but I'm seriously thinking to reduce the default resolution to something like 1040 (thus breaking the promise of 1080p48). Any objections?

Also, something odd kept happening and it kept reverting back to 19fps with FPS override off.

How to reproduce this one?



Anyway, new build posted (minor fixes + experimental 8...12-bit lossless).

Be sure to read the changelog before reporting issues ;)

Edit: just noticed the lower bit depth lossless modes only update the raw buffer every other frame, for reasons not yet understood... if you want to try, compile the sources, but you'll get files with duplicate frames (good for estimating recording times, but nothing more). If you look at a static scene (as I did before posting the build), everything appears fine.

Edit2: looks like all raw types (http://www.magiclantern.fm/forum/index.php?topic=18393.0) that include digital gain are created every other frame :( (that means, I need to find some other method for darkening the image...)

Edit3: progress narrowing down (this used to work before merging 5D3-123 into main builds)


Narrowed down to changeset 38e78de. This change fixed it! (https://bitbucket.org/hudson/magic-lantern/commits/4cce3d2adafff9ac942fcf93fa3c4c3e7c4a84f6?at=crop_rec_4k)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: nikfreak on April 13, 2017, 12:17:00 PM
Congrats, lossless 8bit support.  8)
Any easter egg planned to support other cams  ;)?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 13, 2017, 12:18:06 PM
Any easter egg planned to support other cams  ;)?

hint, hint (http://www.magiclantern.fm/forum/index.php?topic=18443.msg182989#msg182989) ;)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: beauchampy on April 13, 2017, 01:31:25 PM

The issue seems to be in 1920x1080 48p. Hopefully fixed, but I'm seriously thinking to reduce the default resolution to something like 1040 (thus breaking the promise of 1080p48). Any objections?

Ouch, so no more 16:9 slow motion? This is kinda a big deal for someone like me to *has* to deliver in 16:9 to some clients..
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 13, 2017, 01:55:55 PM
You still have the option to fine-tune the internal parameters until you get continuous recording with no errors or corrupted frames. No programming knowledge is required (just fiddle with numbers from the menu), but it can easily take hours of tinkering until you find a sweet spot.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: starimage on April 13, 2017, 02:18:18 PM
First of all, thank you so much for the tremendous work and the joy that you give us.
At me a following problem - on all last experimental insertions in a сropmode the screen becomes pink and there are diagonal strips. In the raw mode this does not happen
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 13, 2017, 02:25:49 PM
Okay so I went in for a second test. Over all here is what I am finding

3k 2.35:1 14 Bit lossless Preview auto is very stable

1920x1080 48p, 14 Bit lossless, Preview auto, out of focus scene - not stable lots of crashes even when not Rec out of focus scene

1920 2.35:1 48p, 14 bit lossless, preview auto, out of focus scene - not stable lots of crashes even when not Rec

1920 2.35:1 60p, 14 bit lossless, preview auto - seemed more stable did not crash. Pressed half shutter and it crashed

1920 2.35:1 60p, 14 bit lossless, preview real time - seamed more stable no crash

1920 2.35:1 48p, 14 bit lossless, preview real time - seemed more stable

1920 2.35:1 48p, 14 bit lossless, preview - framing crashed

There were more rec attempts in there but that is all I can remember for now. Unfortunately Ill have to stop testing for a bit. Here are my asserts

ML ASSERT:
0
at mlv_lite.c:2111 (compress_task), task compress_task
lv:1 mode:3

ML ASSERT:
0
at mlv_lite.c:2111 (compress_task), task compress_task
lv:1 mode:3

ML ASSERT:
0
at mlv_lite.c:2118 (compress_task), task compress_task
lv:1 mode:3

ML ASSERT:
0
at mlv_lite.c:2118 (compress_task), task compress_task
lv:1 mode:3

ML ASSERT:
0
at mlv_lite.c:2118 (compress_task), task compress_task
lv:1 mode:3

ML ASSERT:
0
at mlv_lite.c:2118 (compress_task), task compress_task
lv:1 mode:3

ML ASSERT:
0
at mlv_lite.c:2118 (compress_task), task compress_task
lv:1 mode:3

ML ASSERT:
0
at mlv_lite.c:2111 (compress_task), task compress_task
lv:1 mode:3

ML ASSERT:
0
at mlv_lite.c:2111 (compress_task), task compress_task
lv:1 mode:3

ML ASSERT:
0
at mlv_lite.c:2111 (compress_task), task compress_task
lv:1 mode:3

Magic Lantern version : crop_rec_4k.2017Apr12.5D3113
Mercurial changeset   : dcdf7432df74 (crop_rec_4k) tip
Built on 2017-04-12 20:31:48 UTC by [email protected]
Free Memory  : 163K + 3116K



Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 13, 2017, 02:42:03 PM
48p might be a little better in Apr13.

Otherwise, see the last 2-3 messages. Decreasing vertical resolution in the crop_rec submenu is the only way I know to reduce the frequency of these "crashes" (actually corrupted frames).
Title: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 13, 2017, 03:59:09 PM
The issue seems to be in 1920x1080 48p. Hopefully fixed, but I'm seriously thinking to reduce the default resolution to something like 1040 (thus breaking the promise of 1080p48). Any objections?

Yes. Let's keep tinkering down on this one please. 1080p 48p is crucial. Especially for HDR video kind of work.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: squig on April 13, 2017, 04:39:56 PM
Upscaling in post is a thing. I've shot stuff in 720p crop mode that I'd be comfortable projecting on a 40 foot screen. I can put nitrous in an old Honda, that doesn't make it smart. Reliability trumps  (for want of a better word) a few more pixels every time.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Vegandelight on April 13, 2017, 05:22:03 PM
I havent tried the latest build but i tried running a manual lens on the camera today, the screen doesnt turn on when in video mode. Works in photo mode.

HAs anyone experienced that?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: mageye on April 13, 2017, 05:44:39 PM
Yes.

On Samyang 35mm (manual)

I get: 'Ensure a lens is attached. Press START STOP button to activate movie shooting.'

I think it's just a lens communication thing
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Vegandelight on April 13, 2017, 05:53:04 PM
Yes.

On Samyang 35mm (manual)

I get: 'Ensure a lens is attached. Press START STOP button to activate movie shooting.'

I think it's just a lens communication thing

Thats a normal error, but when you press START STOP the screen goes black and you can hear the mirror locking up like it should. But no image.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: mageye on April 13, 2017, 06:27:53 PM
Oh. Well I get the image so who knows what's happening there?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 13, 2017, 06:33:34 PM
When it happens here, pressing MENU twice solves it (see first post).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: mageye on April 13, 2017, 06:42:46 PM
Back to the crop record.

I have noticed something which at first may appear bizarre:

Indoors I get significantly longer record times at higher resolutions than I do outdoors. This has happened repeatedly over a few days now.

I am thinking it has to do with light levels, detail and compression.

eg.

I have been able to record over 16 seconds at [email protected] inside and just about 2 - 4 seconds at the same settings outside!

(14-bit lossless compression)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 13, 2017, 07:24:37 PM
Testing April 13 build. Much more stable for sure! Compression seems to be tricky still. I'm not the biggest fan of how slow settings (shutter ISO ect) appear to change when in Framing LV but I know it serves a purpose to get proper preview with low processing power.
If we can some how cap the data rate that would be insane. But then I guess it would be lossless at that point.

Higher detail low contrast scene, F1.6 1/125 (1/98) ISO 800
1920 2.35:1 14 Bit lossless 1920 48p preview auto - fairly long rec
16:9 Did not make estimated rec time

1920 2.35:1 14 Bit lossless 1920 60p preview auto - Did not make estimated rec time

3072 2.35:1 14 Bit lossless 3k 1:1 23.976 preview auto seamed good.
16:9 Did not make estimated rec time

On same scene I changed the shutter speed F1.6 1/50 (1/98) ISO 800 then none of the above modes made there estimated rec time.
It seems that if more detail is brought out of low exposure zone, less info gets tossed which means higher data rate.

No crashes/bad frame warnings until I lowered the shutter down to 1/30 in 3072 2.35:1 14 Bit lossless 3k 1:1 23.976 preview auto. Then it crashed/bad frame warning ML assert

ML ASSERT:
0
at mlv_lite.c:2120 (compress_task), task compress_task
lv:1 mode:3

ML ASSERT:
0
at mlv_lite.c:2120 (compress_task), task compress_task
lv:1 mode:3

Magic Lantern version : crop_rec_4k.2017Apr13.5D3113
Mercurial changeset   : cf07f797b9c3 (crop_rec_4k)
Built on 2017-04-13 10:34:53 UTC by [email protected]
Free Memory  : 163K + 3115K
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hindra on April 13, 2017, 07:46:19 PM
Anyone else just try 1080p raw with the 1080p all-i proxie with the new april 13th build? Is it broken? Here is what happened to me today.

1080p raw w/ proxie broken, raw to CF and all-i to SD - Stops recording after a few seconds, camera locks up, preview still works, global draw on or off. After messing with proxie, 14bit lossless no longer working? recording at 90+mb/sec for 1080p as stated on my liveview monitor.

ML ASSERT:
0
at mlv_lite.c:2113 (compress_task), task compress_task
lv:1 mode:3


Magic Lantern version : crop_rec_4k.2017Apr13.5D3123
Mercurial changeset   : cf07f797b9c3 (crop_rec_4k)
Built on 2017-04-13 10:35:29 UTC by [email protected]
Free Memory  : 166K + 3785K


ML ASSERT:
RAW_IS_IDLE
at mlv_lite.c:592 (measure_compression_ratio), task shoot_task
lv:1 mode:3


Magic Lantern version : crop_rec_4k.2017Apr13.5D3123
Mercurial changeset   : cf07f797b9c3 (crop_rec_4k)
Built on 2017-04-13 10:35:29 UTC by [email protected]
Free Memory  : 167K + 3119K


I uninstall ML, format card in pc, then format in camera, then load ML software and reload firmware to set bootflag.

Still recording at 90mb/sec and in canon menu at format menu it shows no use of CF card. I am also now having issues recording with any of the crop_rec options. Recording stops very quickly. What did I break?  :(

EDIT:
Lossless is still working, looking at footage the 1080p raw stills are only 1.90MB
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 14, 2017, 10:53:18 AM
Cant hide my excitement.
ML makes my camera as good as cameras costing $$$$$$
and still ML moves on ...
Gratitude!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 14, 2017, 02:51:00 PM
@a1ex Tested the April 14th Build last night here are my results..

3K 2.35:1 14bit Lossless = Ok
               12bit Lossless = Liveview splits in to two screens and image is not really viewable. Recording stops quickly and compression error pops up on screen
               10bit Lossless = Same
                 8bit Lossless = Same
 
1080p 45/48fps 3x3 16:9 - 14bit Lossless = Records for a few seconds than crashes camera
                                        12bit Lossless = Same
                                        10bit Lossless = Same
                                          8bit Lossless = Same 

1080p 60fps 3x3 16:9      - 14bit Lossless = Records for longer time than 48fps, than after crashes camera
                                        12bit Lossless = Same
                                        10bit Lossless = Same
                                          8bit Lossless = Same 
 
Note after trying the slowmo functions camera turns on but ML Liveview only shows a black screen, without magic lantern sd in camera liveview works. 

@Quentin Very exciting
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 14, 2017, 03:05:16 PM
Reproduced - looks like we've got some image processing module that has to be reconfigured for correct resolution. That means, solving this would be one tiny step closer towards fixing LiveView.

(I've only tested the 8...12-bit settings in plain 1080p, without crop_rec)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 14, 2017, 03:20:34 PM
@a1ex Yeah LiveView seems to be very complicated wish I knew coding to help out. Do you think LiveView will ever be as smooth as 1080p in 3k - 4K mode?  Gonna try them in 1080p today, and I'm gonna keep testing.  All those new feature ad-ons makes this camera an elite camera.  Thank you for your hard work. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 14, 2017, 03:59:44 PM
12, 10, 8 bit Lossless!!!!!!!!!! Am I reading this correctly? Oh man. This is unreal!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: anto on April 14, 2017, 04:06:32 PM
Grazie a1ex! Magic!

Inviato dal mio HTC One utilizzando Tapatalk

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: zeus12 on April 14, 2017, 04:07:36 PM
10 bit, 8 bit compressed raw? Am I dreaming? Thanks for your ingenious work!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: GutterPump on April 14, 2017, 04:43:57 PM
Just tested the new 14 april build.

I'am able to record in crop mod with 8 10 and 12 bits lossless. (but now in crop mod, i can't use more than the 2:35 ratio "eg : 3104x1320"  and FPS are locked to 29 if we dont use FPS override. Even if we are in PAL)

This is just crazy in what the 5D mark III is become.

This mean we will really record 3.5k 4k continue.

---

Here a zip with DNG if some want check how look 8/10/12 bits lossless

https://www.dropbox.com/s/pj7db1izytkj3zj/8_10_12bitslossless_samples.zip?dl=0

Three samples from [email protected] clip
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 14, 2017, 04:56:48 PM
Quote
This mean we will really record 3.5k 4k continue.
  :o
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kharak on April 14, 2017, 05:57:24 PM
Does april 14th build run the test for the april 4th build?

My connection is so incredibly slow, if i dont have to dowload apr10 just to test for a bug and then download 14th, well that would be swell :)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 14, 2017, 06:35:29 PM
@GutterPump
You remind me of myself when I shoot lens cap  :D
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: GutterPump on April 14, 2017, 06:46:20 PM
@quentin

Haha it could have been worse, it's really the mess on my desk. But for magic lantern I make an effort.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hindra on April 15, 2017, 05:37:19 AM
Hey guys, here's a quick 23 second shot of the 8 bit lossless 3520x1322 at 60 fps with the latest April 14th 1.2.3 build. My hands are really shaky, I'm not a pro and I had my shutter at 1/80th.
ISO 200, F8, 1/80th - SHAKY HANDS - straight out of camera no corrections/lut - 60mbit vbr
Its working :)

https://vimeo.com/213302351


EDIT: 30 fps
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Teamsleepkid on April 15, 2017, 06:03:23 AM
thanks for posting the video thats pretty crazy
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 15, 2017, 06:04:02 AM
Hey guys, here's a quick 23 second shot of the 8 bit lossless ... ISO 200

Here a zip with DNG if some want check how look 8/10/12 bits lossless

https://www.dropbox.com/s/pj7db1izytkj3zj/8_10_12bitslossless_samples.zip?dl=0


Have you noticed the ISO suggestions when selecting a lower bit depth from menu? (at the bottom of screen)

Please refer to this commit (https://bitbucket.org/hudson/magic-lantern/commits/2028d73f6f34dcdaaa4c241824e538d0e0e47267#Lmodules/mlv_lite/mlv_lite.cF2996T3033) for more details, and to this article (https://theory.uchicago.edu/%7Eejm/pix/20d/tests/noise/noise-p3.html) for some theory regarding bit depth, noise and posterization.

I've considered acceptable a quantization step of twice the noise stdev, which would be just a bit better than "gradient 3-bit tonality" from Fig. 3 in the linked article (log2(256/24) = 3.3), which gives the lowest ISO usable at each bit depth. For the highest ISO, I've considered "wasteful" a quantization step smaller than noise stdev / 2.5.


3520x1322 at 60 fps

Impossible (please check the metadata with mlv_dump -v).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: iheartbmx on April 15, 2017, 06:14:41 AM
Filled a 32GB card with 3K / 2.35:1 / 23.97fps / 14bit lossless - Completely stable with zero bad frames! (I used cr2hdr v1.2 to extract DNG's on OSX Sierra and it worked perfect)

You guys are amazing!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 15, 2017, 06:27:23 AM
@a1ex
I saw ISO suggestions.
8 bit digests noise generated by high ISO ? (I read that raw is divided by 64)
Thanks in advance for the enlightenment
 and your hard work
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 15, 2017, 06:30:09 AM
Yes, at high ISOs, the lowest bits (trimmed for reducing the bit depth) are mostly noise.

At low ISOs, some of them are useful data (especially in shadows).

The last two bits are mostly noise on current sensors (so you can safely use 12-bit at low ISOs).

The posterization from ML previews is real (it matches what I'm getting when opening the DNG in ufraw, for example), so you can actually preview the loss of shadow detail.

For a visual explanation, don't forget to move the mouse over the images from the linked article (https://theory.uchicago.edu/%7Eejm/pix/20d/tests/noise/noise-p3.html) (see previous post).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 15, 2017, 06:32:24 AM
Thats incredible and gives more room to film in low light, broadening the camera capabilities.
Knowing how ETTR works, I can understand what you describe
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hindra on April 15, 2017, 07:28:52 AM
Have you noticed the ISO suggestions when selecting a lower bit depth from menu? (at the bottom of screen)

Please refer to this commit (https://bitbucket.org/hudson/magic-lantern/commits/2028d73f6f34dcdaaa4c241824e538d0e0e47267#Lmodules/mlv_lite/mlv_lite.cF2996T3033) for more details, and to this article (https://theory.uchicago.edu/%7Eejm/pix/20d/tests/noise/noise-p3.html) for some theory regarding bit depth, noise and posterization.

Impossible (please check the metadata with mlv_dump -v).

Yes, I did notice the ISO suggestions. I have no idea why I thought I was shooting at 60fps. I re-rendered at 30 fps, I also cant hit 60fps when attempting to re-create. but, 8 bit lossless works! :) I was trying to use the lowest bitrate to obtain the highest fps and I thought it was 60, I was mistaken. Thank you for those links.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: D_Odell on April 15, 2017, 09:38:32 AM
Is there any chance MLVFS will start supporting these files without using mlv_dump?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: giarcpnw on April 15, 2017, 10:17:26 AM
What would be the reason the DNGs from the April 14th build would not open in After Effects? I'm getting can not parse errors. Worked fine with April 10th. Using Footage.app on mac.  Shot 14bit lossless.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: giarcpnw on April 15, 2017, 10:26:23 AM
What would be the reason the DNGs from the April 14th build would not open in After Effects? I'm getting can not parse errors. Worked fine with April 10th. Using Footage.app on mac.  Shot 14bit lossless.

I think I answered my own question. The first frame of the sequence in all the MLVs I shot tonight were black (I assume corrupted) I removed them from the sequences and then AE and Camera Raw could load the sequence.

Something to note for this build. April 14th.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 15, 2017, 11:43:21 AM
Small MLV and steps to reproduce, please.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: GutterPump on April 15, 2017, 11:47:57 AM

Have you noticed the ISO suggestions when selecting a lower bit depth from menu? (at the bottom of screen)


Of course i noticed these suggestions, now i read your article i'm able to understand better.

I will do differents tests in low light condition with different range of ISO. eg : 12800 ISO to 800.

So, if i understand well, in this way, the noise is our friend if we need less posterization because the 8bits depth ? So it means that if we film a scene very illuminated outside with the 8bits , we have all interest to use a variable filter ND.

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 15, 2017, 01:12:36 PM
So, if i understand well, in this way, the noise is our friend if we need less posterization because the 8bits depth ?

Correct.

To be effective for reducing posterization, the noise must be applied before the quantization (therefore, analog ISO noise is fine). Adding it in post will not help.

Quote
So it means that if we film a scene very illuminated outside with the 8bits , we have all interest to use a variable filter ND.

I disagree here. Lower bit depths were added to reduce the data rate at higher ISOs (where the lowest bits are mostly noise, and because of them, the image doesn't compress well).

I don't think filming with 8 bits is actually your goal. More likely, I assume you actually want to minimize the data rate. With uncompressed data, it's easy (all frames have the same size: W x H x bpp/8). With lossless compression, you can't choose the compression ratio in advance (http://www.magiclantern.fm/forum/index.php?topic=19300.msg182229#msg182229); this is given - roughly - by the entropy of the image.

These "fake bit depth" settings (actually dividing the signal by a power of 2) are one way to reduce the entropy (by removing the least significant bits, because of integer division). That's a lossy operation (going from 14 bits to any lower value). After this step, compression is lossless (that is, the uncompressed data will be identical to what was fed to the compressor).

(That means, 14-bit lossless will give raw data identical to uncompressed 14-bit (100% identical, not one single bit different, contrary to some earlier report (http://www.magiclantern.fm/forum/index.php?topic=19300.msg182741#msg182741)), while "12-bit lossless" should be interpreted as "14-bit to 12-bit conversion - lossy by definition - followed by lossless compression", so it should have the same number of useful levels as uncompressed 12-bit. Please note the 12-bit lossless is not identical to 12-bit uncompressed - they differ by a constant value and possibly by some round-off error, and the same is true for lower bit depths.)

Back to entropy - increasing the ISO and underexposing (to keep the image brightness unchanged) will increase noise in the image (at all brightness levels). So, what you may gain by reducing the bit depth, you will lose from the added noise. Here's a quick test (static outdoor scene exposed to the right, plain 1080p, compression rates taken from ML menu):

Bits per pixel      14  12  11  10   9   8
ISO  100 1/100     61% 53% 50% 48% 46% 43%
ISO  200 1/200     62% 54% 51% 49% 47% 44%
ISO  400 1/400     63% 54% 51% 49% 47% 45%
ISO  800 1/800     65% 55% 52% 50% 48% 46%
ISO 1600 1/1600    67% 56% 53% 50% 48% 46%
ISO 3200 1/3200    70% 57% 53% 50% 49% 47%
ISO 6400 1/6250    76% 60% 55% 52% 50% 48%
ISO 12800 1/12500  79% 63% 57% 53% 50% 49%


=> there's nothing to gain by using a lower bit depth at a higher ISO + underexposure.

Of course, if your goal is to record at 8 bits, no matter what, then of course, the best thing to do is to increase ISO at 12800 (or at least 6400) for half-decent results. It's also trivial to change the code to get 6 bits, 4 bits or even one single bit, if that's your goal ;)

My advice: first expose properly (ETTR, zebras, raw histogram, whatever you prefer), and afterwards choose the optimal bit depth. Not the other way.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 15, 2017, 02:54:41 PM
Actually that post about compression needed to be in two parts
a) Why do I get these boxy big pixels compression-like in out of focus areas
b) Is 14bit and 14bit lossless visually identical

The question b) is answered already.
The question a), I guess was due to a different reason other than the bit depth

But I understand a1ex words about bit depth. Great explanation
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: VinceML on April 15, 2017, 04:28:43 PM
Hi Guys,

I explain how work and how I setup my 5D for create 3,5K video, here :

https://www.youtube.com/watch?v=kPQMNHhV41M
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Levas on April 15, 2017, 05:02:26 PM
 :o
Now with these fake bit depth options, this is getting interesting for the SD card camera's  :D

@Alex,
The red numbers in your bit depth / compression ratio table, do you mean that the serious loss of info/shadow detail is where the red numbers are ?
And if I understand correct, you're using digital gain, which cut's of the noisy bit, but the image technically stays 14 bit for the compression algorithm, but because of the noisy bit cut off, the compression rate goes even beyond the 50%.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 15, 2017, 07:21:53 PM
Correct - the red numbers match my ISO recommendations (https://bitbucket.org/hudson/magic-lantern/commits/2028d73f6f34dcdaaa4c241824e538d0e0e47267#Lmodules/mlv_lite/mlv_lite.cF2996T3033) and are below "gradient 3-bit tonality" from the third figure (labeled Fig. 18) from the noise article (https://theory.uchicago.edu/~ejm/pix/20d/tests/noise/noise-p3.html). Also added orange = wasteful (there's little or no additional image quality to gain at these settings).

If the theory is correct (which I didn't have the patience to check by pixel peeping), then it's probably best to stay near the transition line to red, on the black side (that is, 11-bit for ISO 100-800, and reduce the bit depth once you need ISO 1600 and beyond).

I believe the recommended settings are visually identical to regular 14-bit (while in 14-bit lossless, the raw data is 100% identical to uncompressed 14-bit). Again, this is not confirmed by pixel-peeping (it's just noise theory). There may be a small color cast in shadows, because the 0.5 LSB offset is not yet applied (but it's easy to fix once @martinhering helps us decode the JPEG stream (http://www.magiclantern.fm/forum/index.php?topic=18949.msg183230#msg183230)).

The estimations were done for 3x3 binning (plain 1080p); the 1:1 crop mode may require slightly different values.

At these (probably optimal) settings, the compression rate appears to be consistent (50%), at least on my test scene (it was some foliage exposed to the right, but didn't save it).

(my definition for compression rate is: saved frame size / size of a 14-bit uncompressed frame)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: giarcpnw on April 15, 2017, 07:54:41 PM
Small MLV and steps to reproduce, please.

Well, i could not duplicate it. I updated Martin's footage app this morning and they process fine. I do however remember it happened right after i tried to capture 12bit and 8 bit lossless and saw this on my screen.
This happens whenever I try to capture any lossless bit rate than other than 14bit lossless. Trying to record like this results in a ton of scrolling errors. (none of which I know what they mean) Have to power off to stop.   Pressing x5 crop WILL ALLOW capture however in this mode. 3072x1308, FPS Override 23.976

(http://i65.tinypic.com/bhyh5j.jpg)

(http://i64.tinypic.com/2n7qul5.jpg)

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 15, 2017, 07:58:42 PM
@giarcpnw: already reported, and for the moment I don't know how to solve it. Will probably disable it in crop modes with modified resolution, until I'll understand what happens.

@Quentin: I don't see "boxy big pixels compression-like in out of focus areas" in your post - I only see black text on a blueish background...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 15, 2017, 08:19:26 PM
Unfortunately I erased the videos.
But ... how do we attach an image on the post ?
Sorry for my ignorance ?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hindra on April 15, 2017, 08:20:28 PM
1080p48 at 12 and 11 bit lossless at 100 iso I get results with a split exposure. Global draw off

(http://i.imgur.com/dNVbq9P.jpg?1) (http://imgur.com/dNVbq9P)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hindra on April 15, 2017, 08:27:13 PM
Unfortunately I erased the videos.
But ... how do we attach an image on the post ?
Sorry for my ignorance ?


I upload to imgur.com and copy/paste the BBCcode link in the forum post.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 15, 2017, 08:36:07 PM
Testing last build april14th.
Tearing issue in 1920x96050p3x3 is gone (I understand because modification of buffering). It is almost perfect for me. I'm really grateful, thanks A1ex!!!... It was a dream some time ago, and now...
Thank you very very much...

Now the only downside is recording time with hi isos in 14bit lossless and incorrect framing, but I can live with that.

Thanks again!!!!!!!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kharak on April 15, 2017, 11:00:41 PM
I successfully downloaded Apr14 build, but it did not do a ROM Test at start, not that I could see atleast. Do I need to download Apr10 specifically for the ROM test ?

Anyways, with this build I have correct/centered framing at 3072x1728 14Bit Lossless 16:9. Very nice! Will actually start using this thing now.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 16, 2017, 04:57:43 AM
I successfully downloaded Apr14 build, but it did not do a ROM Test at start, not that I could see atleast. Do I need to download Apr10 specifically for the ROM test ?

Have you tried running firmware update again from Canon menu and restart camera?

If no messages prompts up afterwards then you haven't been affected from that April 4th null pointer bug.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: etienne on April 16, 2017, 06:26:09 AM
Hey

Great job for the low bit depth it opens great perspective to play around the 3.5k.

Here is my bug encouters for low bit depth 8,9,10,11:

 -  the expo is darken in the realtime preview and hdmi out. realtimre preview for 8 bit totally unusable (this is bas for focusing) framing mode is working better but focus is still not possible from there.

 - 1:1 Tall have the lower of the image not displayed correctly like mentionned for slow mode

 - non working on 3K mode and higher

Can we expect a mlv sound coming back? post synchro is so burden with just the beep :P
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kharak on April 16, 2017, 03:31:17 PM
Have you tried running firmware update again from Canon menu and restart camera?

If no messages prompts up afterwards then you haven't been affected from that April 4th null pointer bug.


Thank you, Deafeye. I did as you said, update from Canon and restart. But could not see any test being done. Thought I saw a screenshot from someone doing the test and there was like this 0-100 Counter, maybe I am mixing it up with something else.

Its just a pain for me downloading builds cause I am on Satellite internet, I get 4-5 kb/s on a good day and then there are constant dropouts which which require me to start the download all over again... Just loading this forum page takes forever, and its only text more or less..

So its safe to assume that I need to download apr10 build because it has the Rom Test ?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 16, 2017, 04:53:18 PM
https://bitbucket.org/hudson/magic-lantern/src/crop_rec_4k/src/debug.c?fileviewer=file-view-default#debug.c-288

If no errors are present, no obvious feedback is given (other than an extra LED blink, which is pretty obvious to me).

I get 4-5 kb/s on a good day and then there are constant dropouts which which require me to start the download all over again... Just loading this forum page takes forever, and its only text more or less..

Here's what helped me on a similar connection, besides obvious things such as disabling images or using a text-mode browser (mobile connection with poor signal, far away from the city, 4K/s at best):

http://lartc.org/wondershaper/

There are probably equivalents for other operating systems. The idea is to reduce download speed so you can load other pages (or use a chat client) while a long download is running.

Having a remote server accessible with VNC, with a very low image quality setting, can be also helpful (as loading a JPEG preview of a webpage can often have much lower bandwidth than its HTML+CSS+JS+images).

For a download with frequent dropouts, just leave a "wget" command open in background (as it doesn't give up easily, unlike most browsers).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: dfort on April 16, 2017, 09:59:45 PM
There seems to be lots of interest getting crop_rec on steroids working on other cameras.

The EOSM can already do basic 3x3 720p crop_rec and I got it working a while ago on the 700D but was trying to get the other options available on the 5D3 working before doing a pull request. I haven't gotten very far but since other users are also experimenting with adtg_gui, dm-spy and other ML tools to get 4K working on their cameras, maybe we should learn to crawl before attempting to fly so I posted a pull request for limited support of the crop_rec module for the 700D.

https://bitbucket.org/hudson/magic-lantern/pull-requests/824/crop_rec-for-700d/diff

Uploaded a new test build in my bitbucket download area.

https://bitbucket.org/daniel_fort/magic-lantern/downloads/
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: rubnva on April 16, 2017, 11:40:36 PM
Tested the new software update with my Canon 5D Mark III and I used this video mode: 1920x960 @ 50p (both 1:1 crop and full-frame - 3x3 pixel binning) - continuous*)
but al converted (with MLV Mystic) DNG"s are corrupt (pink pixels)

Check my MLV file here: https://we.tl/lvASYcNcRx

Some help please? :)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: mk11174 on April 17, 2017, 02:49:10 AM
On the 700D, I finally found the reg needed to fix the darkened lower half of the image when trying for higher resolution than 2048x1152.
It was ADTG2[82B6].
Best fps I can get without a freeze at this res is 10.639fps.
Don't mind the crappy image quality, room was dark and was to lazy to turn on the light for testing, LOL, was shooting 3200ISO.
(https://s4.postimg.org/xkgbmbgz1/4kfix.jpg)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 17, 2017, 06:00:14 AM
Sorry for the double post!!! But I Just tested 4-14 build 12 bit 3072 2.35:1 at 23.976 ISO 3200 with the old 5x zoom method and it seemed like it was endless. Over 20 secs for sure!!!! DNGS are clean and processed fine and all worked in resolve.
So the this will work for sure!!!! Just need to get the crop rec working. Way easier said then done no doubt. this is crazy tho. 3k 12bit lossless 24p on a 2012 5D
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 17, 2017, 08:18:42 AM
@hjfilmspeed: already reported (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183180#msg183180) and answered (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183243#msg183243) a few (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183246#msg183246) times...

Framing preview works fine and shows what you are going to get.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 17, 2017, 02:09:07 PM
Ahh sorry @a1ex missed that. But the concept is there. This is really awesome!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: treez on April 17, 2017, 03:45:28 PM
Great work...but how to convert the crop mode *.mlv file. I use MLVrawviewer but it crashes.
Any Tips or ideas?

Thnxs.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: zeus12 on April 17, 2017, 04:50:49 PM
Can anyone post their advanced settings on the crop_rec on 5D3123 so that I can get the crop from the center? Currently it looks the crop start from the extreme right side of the sensor (3K).

Thanks for all your hero members for the advancement.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 17, 2017, 09:59:12 PM
Admiration for many reasons
For revolutionary development of out of this world features , for so many cameras
Baby sitting the forums with so many users of so many experiences and variation of needs and knowledge
I surrender, I could not do that
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: BBA on April 17, 2017, 10:53:44 PM
Like in french : RESPECT !!!!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: etx on April 18, 2017, 05:01:48 AM
Can anyone post their advanced settings on the crop_rec on 5D3123 so that I can get the crop from the center? Currently it looks the crop start from the extreme right side of the sensor (3K).

Thanks for all your hero members for the advancement.

Don't use 5x mode like before. If your in regular live view you will get proper framing. It's a bit faster than before also! If you go to 5x mode the framing is off. Took me a bit to figure this out. HDMI preview isn't working yet in this mode but it's amazing none the less. I'm standing by until it does so I can use the 5D on my Movi again.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 18, 2017, 09:44:29 AM
@quentin

+1

Thank you!!!!!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: child01 on April 18, 2017, 01:55:50 PM
Can ANYONE point where is a precise step by step instruction of using mlv_dump?

Yes, I know - noob! how da hell ... and so

But I'm using ML almost since it's first build, but thankfully chmee wrote his tool very soon...
I was always using RAW2CDNG, but now I have to deal with this rocket science - command line stuff ((
And there's no clear and simple TUTORIAL here on forum or youtube video tutorial. Everything I've found is either outdated or people being sarcastic (not helping) and in most cases for MAC (why mlv_dump discussions are always mac oriented?)

A1ex is so helping and responsive. I don't get it why didn't he created a normal software with UI ...or clear and simple tutorial or list of commands for us to just copy/paste?
Why not implement at least base and commonly used commands into mlv_dump somehow? Or enable drag'n'drop functionality? anything simple
 :-\

who can help - what should I do with mlv_dump on PC to simply convert MLV to DNG?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Easygoingmartin on April 18, 2017, 05:31:54 PM
Hey guys.. i'm kinda new to ML.. i've loaded this experimental build and started trying to record in UHD. I've loaded the Rec_crop module and tried the UHD setting. Now the 1080p/50fps setting works sortof(it's not actually "slow motion" since windows still recognizes it as 29fps) the 3K/UHD/4K setting won't work. There's a weird purple/pink artifact going on when i try to film.

Would that mean i have to record in RAW? If so, what are the appropiate modules i have to activate to make it work?

Thanks jn advance!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Ryan Lightbourn on April 18, 2017, 06:55:38 PM
Is anyone getting continuous 3K raw in 4:3 aspect ratio (for 2x anamorphic shooting)? If somebody can confirm I'll pull the trigger on a used 5D3 body!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: g3gg0 on April 18, 2017, 07:58:04 PM
A1ex is so helping and responsive. I don't get it why didn't he created a normal software with UI ...or clear and simple tutorial or list of commands for us to just copy/paste?
Why not implement at least base and commonly used commands into mlv_dump somehow? Or enable drag'n'drop functionality? anything simple
 :-\

pay him 4k€ a month and he will probably do this additional work on saturday and sunday night - the only time when he usually sleeps.

ah and i forgot to mention that i implemented mlv_dump as reference implementation, tool for ML devs, users on nerd level 100 and fallback solution.
*if* there is a proper GUI frontend - go for it, else mlv_dump has to do the job :)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Walter Schulz on April 18, 2017, 08:15:47 PM
My opinion: Get rid of those experimental builds. It invites users not ready for experiments on this level.
Those wanting to go there may
a) have an environment to make their own builds
or
b) rely on a helping hand building it for them on demand.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 18, 2017, 08:35:07 PM
My opinion: Get rid of those experimental builds. It invites users not ready for experiments on this level.
Those wanting to go there may
a) have an environment to make their own builds
or
b) rely on a helping hand building it for them on demand.

Hmmm I not sure I agree when it comes to experimental builds. Although I guess I don't quite get what you mean by "rely on a helping hand building it for them on demand."

On one hand, the admins hold our hands and babysit us while testing. Time consuming to say the least.
On the other hand, some of us who have been using ML for years get a chance to learn about the inner workings of ML and learn how to become better testers.
Also, when it comes to working out the bugs, us inexperience ones are the best at breaking things because we don't listen.

However I do feel that people should not get upset or wine or cry or point fingers when they can't figure something out. I think people need to be more aware that this is all volunteer work. No pay for hours of endless coding. One should not complain especially when one has no clue what is involved with coding.   

But that is my humble "I don't even know what I'm talking about but should since I have been using ML for so long and I am being grossly off topic and I should know better about that too run on quote"  opinion. An opinion to which I feel I don't even deserve to give. Compared to Walter Schulz I am a dumb dumb head. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: GutterPump on April 18, 2017, 09:08:10 PM
Not agree, experimentals builds help to understand better the universe of Magic Lantern, it's a big help for the community, the developpement is more fast, Alpha testers are more large.

Do not delete them, the simple warning message in the top of page is enough. People have been warned from the beginning with the risks of Magic Lantern.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Lars Steenhoff on April 18, 2017, 10:57:10 PM
Experimental builds are the soul of magic lantern right now, it allows people to test lit the latest and give very valuable feedback, @walter if someone dont like them, or are scared it may harm they are free not to use them, but i for one love them.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: zeus12 on April 19, 2017, 12:48:03 AM
The experimental version of magic lantern is on the cutting edge, frustratingly cutting edge. It’s part of the struggle for the code-blind mortals who’s pulling their hairs out and ask the stupidest questions on the board. The struggles make magic lantern exhilarating,  and inspires participants to make better picture, better videos. Most of us do not post anything on the board just for the fear of sounding really really….. stupid. However, we do really appreciate the “geniuses” and generosity of those immortals such as Alex. Please bear with us.
By the way, I finally get UHD to work as well as 1080p @45fps. The DNGs were generated with Rawflow (page 13 of this board). Somehow, the first frame is always bad and the rest are splendid.
Thanks for all the immortals on the board and hope you can improve magic lantern further and let the mortals to try the experimental versions. The ‘code blind” mortals can’t contribute much to the development of magic lantern. But we will do more volunteers to places such as “soup kitchen” to feed the homeless as our appreciation.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: dfort on April 19, 2017, 09:01:18 AM
My opinion: Get rid of those experimental builds. It invites users not ready for experiments on this level.

I was posting test builds for the 12-bit (and 10-bit) RAW video development discussion (http://www.magiclantern.fm/forum/index.php?topic=5601.0) before the Experiments (https://builds.magiclantern.fm/experiments.html) page went live so I feel partially responsible for rolling that snowball.

It would be great if more users would set up development environments or at least take a good look at the code. Even if you can't compile or understand the code, there are some great comments in there that explains what the developers are doing and perhaps more importantly, what areas still need some more testing. For example, in the crop_rec.c file of the crop_rec_4k branch (to stay on topic) you'll find this:

Code: [Select]
/* max resolution for each video mode (trial and error) */
/* it's usually possible to push the numbers a few pixels further,
 * at the risk of corrupted frames */
static int max_resolutions[NUM_CROP_PRESETS][5] = {
                                /*   24p   25p   30p   50p   60p */
    [CROP_PRESET_3X_TALL]       = { 1920, 1728, 1536,  960,  800 },
    [CROP_PRESET_3x3_1X]        = { 1290, 1290, 1290,  960,  800 },
    [CROP_PRESET_3x3_1X_48p]    = { 1290, 1290, 1290, 1080, 1080 }, /* 1080p45/48 */
    [CROP_PRESET_3K]            = { 1920, 1728, 1504,  760,  680 },
    [CROP_PRESET_UHD]           = { 1536, 1472, 1120,  640,  540 },
    [CROP_PRESET_4K_HFPS]       = { 2560, 2560, 2500, 1440, 1200 },
    [CROP_PRESET_FULLRES_LV]    = { 3870, 3870, 3870, 3870, 3870 },
};

So instead of simply reporting corrupt frames, indicate the resolution where you start getting corrupt frames and how far you had to back off to get consistently clean results. If you can compile, try pushing it a little further and report how far you can go before it breaks.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 19, 2017, 09:23:24 AM
So instead of simply reporting corrupt frames, indicate the resolution where you start getting corrupt frames and how far you had to back off to get consistently clean results. If you can compile, try pushing it a little further and report how far you can go before it breaks.

For this particular snippet, vertical resolution is actually adjustable from the crop_rec submenu (no need to compile). I don't remember anyone trying to change it, even though I've mentioned (http://www.magiclantern.fm/forum/index.php?topic=19300.msg182320#msg182320) it a (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183060#msg183060) few (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183112#msg183112) times (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183115#msg183115) (including first post (http://www.magiclantern.fm/forum/index.php?topic=19300.msg182052#msg182052)).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: garry23 on April 19, 2017, 10:14:03 AM
@dfort

I agree with your 'set up a development environment', and, as you know, I wish to do this.

However, and this is a big however, not all of us are competent to do that without very clear and simple instructions.

I have tried to set up such an environment, and although the instructions I was following were good, they still assumed some knowledge I don't have. I kept getting error messages and gave up :-(

From my perspective it would be good to reset/start an ML page specifically directed at getting your own compile environment up and running, and keep Win and Mac instructions separate. Also this page should be written from a nil knowledge perspective, i.e. for 'idiots' like me.

Of course, I appreciate I'm asking a lot, i.e. someone who knows what they are doing sitting down and writing that page.

But until I can follow a simple workflow for setting up a compile environment, I flummoxed.

Cheers

Garry
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 19, 2017, 10:41:21 AM
Just start out with what´s not working in one of the compiling threads and take it from there. It´s impossible to account for every single thing that comes up when compiling although dfort´s efforts made it much easier, especially for mac users.
I´m pretty confident the solution won´t be far when out in the open.
Here are two threads if anyone is interested. There are more efforts and threads out there.
Mac
http://www.magiclantern.fm/forum/index.php?topic=16012.0
Windows
http://www.magiclantern.fm/forum/index.php?topic=15894.0
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Walter Schulz on April 19, 2017, 11:26:20 AM
I followed Mac instructions and have my own machine running for some time. Had to build a Hackintosh for that and I'm a Windows guy.
Didn't test Windows workflow, yet.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 19, 2017, 04:07:24 PM
Having involved into beta testing of 3d software in the past, in my humble experience, its good to keep an order in information coming, sort them, classify them, confirm symptoms etc. Its very hard
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 19, 2017, 04:27:55 PM
For this particular snippet, vertical resolution is actually adjustable from the crop_rec submenu (no need to compile). I don't remember anyone trying to change it, even though I've mentioned (http://www.magiclantern.fm/forum/index.php?topic=19300.msg182320#msg182320) it a (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183060#msg183060) few (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183112#msg183112) times (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183115#msg183115) (including first post (http://www.magiclantern.fm/forum/index.php?topic=19300.msg182052#msg182052)).

@a1ex I have to say, I was super nervous about changing those settings since I didn't do enough research on what they mean but Ill see if I can figure it out. Has anyone one tried tweaking those? What did you find?
 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 19, 2017, 04:58:42 PM
Simple example: if you often get corrupted frames in 1080p48, dial 1040 (or some other value lower than 1080) in the Target YRES field. Then refresh the LiveView manually (e.g. press MENU twice) and the resolution in RAW video menu should update.

Set 0 to disable the override and go back to default value.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 19, 2017, 05:31:37 PM
Simple example: if you often get corrupted frames in 1080p48, dial 1040 (or some other value lower than 1080) in the Target YRES field. Then refresh the LiveView manually (e.g. press MENU twice) and the resolution in RAW video menu should update.

Set 0 to disable the override and go back to default value.
@a1ex Ohhh cool! Do you have to match the value in the MLV Lite options?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 19, 2017, 06:18:13 PM
This sets the maximum vertical resolution available for the raw recorder (which records a cropped section from the full-size raw buffer).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 20, 2017, 06:24:41 AM
It takes 5 minutes to dial proper number.
I would prefer a number to subtract rather than setting absolute vertical resolution.
Its faster and practical.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hindra on April 20, 2017, 06:56:12 AM
I'd actually like to try and raise the vertical resolution and shorten the width to try and get the closest to 16:9 at highest resolution possible.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 20, 2017, 08:42:04 AM
@Quentin: have you tried using SET, left/right and scrollwheel?

The same controls are in many other places in ML menu, whenever you need to input a large number (check e.g. intervalometer).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 20, 2017, 08:49:28 AM
Thanks a1ex, too many things to remember/learn
Its hard to carry a manual with you :D
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: tifose on April 20, 2017, 01:07:33 PM
Whats the best Card to use?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hindra on April 20, 2017, 08:58:44 PM
Whats the best Card to use?

I'm not sure on the best card, but I have had great success with Lexar 32gb and 128gb 1066x CF cards.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Lars Steenhoff on April 20, 2017, 09:59:59 PM
same for the fastest sandisk cf 160 mb 128 gb they work great for me.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: tifose on April 21, 2017, 02:12:50 AM
Thanks guys will get one  my CF card stop working in camera
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Walter Schulz on April 21, 2017, 09:25:35 PM
If your card is working in cardreader but not in cam you might have a defective cam.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 21, 2017, 11:55:05 PM
New build:
- centered x5 zoom is back (3.5K with 8...12-bit lossless compression)
- disabled 8...12-bit lossless in incompatible modes (sorry, still couldn't figure out how to fix them)
- for bit depths between 8 and 11, the choice is now automatic, depending on ISO (see this post (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183231#msg183231))
- fixed silent pics (https://bitbucket.org/hudson/magic-lantern/commits/a9c8ad412af5c282bbad6889ce5f65d974ca3ada) (a bit sad to see them broken for 3 weeks with no bugs reported)
- minor UI fine-tunings.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: dfort on April 22, 2017, 03:59:47 AM
- fixed silent pics (https://bitbucket.org/hudson/magic-lantern/commits/a9c8ad412af5c282bbad6889ce5f65d974ca3ada) (a bit sad to see them broken for 3 weeks with no bugs reported)

Sorry I noticed that while trying to get lossless compression working on other cameras but didn't create a proper bug report--though I did point it out. I'm working on the compressed_raw branch but maybe the crop_rec_4k branch has the latest changes? Noted that  lossless.c and silent.c aren't in sync (https://bitbucket.org/hudson/magic-lantern/branches/compare/crop_rec_4k%0Dcompressed_raw#chg-modules/silent/lossless.c) between these two branches.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: tifose on April 22, 2017, 05:14:46 AM
@Walter Schulz

it is working in my Reader and 5Ds

i use to use my 5DMk3 for Raw just didn't use it in over a year not sure why this is happening any idea how this can be fix?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 22, 2017, 05:43:17 AM
Great work @a1ex you tried your best thank you for all the updates. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 22, 2017, 05:47:27 AM
@tifose You should try a different card with your camera.  Also use a flash light or light source look inside the card bay to see if any of your pins are bent, if not you may have a problem. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: csound on April 22, 2017, 05:51:16 AM
I have the null pointer bug warning coming up with April 21 build. Message says to contact ML developers on IRC Magic Lantern freenode. Tried but not sure I was at correct place to do this - Alex can you supply URL please. Unfortunately don't have old logs.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: lostfeliz on April 22, 2017, 06:26:13 AM
I've been experimenting with the 3K crop and 1080p slow motion, etc from the experimental build. Amazing stuff. It's hard to say, of course, how long it will be before usability is nailed down. Wondering if there might be a shorter term move to combine the "simple" 1080p24/30 lossless recording that's been developed along with the MLV Sound module? I would love to incorporate something like that to my everyday run and gun interviews and b-roll without the occasional random breaks in recording and the extra large files that quickly fill up my cards.

(this was posted in reverse engineering but it was recommended that I delete it and post here)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 22, 2017, 06:49:21 AM
I just tested and this is my favorite build yet!!!

Scene is well exposed fine detail @ ISO 3200
3072 x 1308 (or what ever 2.35:1) Is working extremely well in 5x zoom 12 bit lossless fps override 24.006 and framing preview. I disable the crop rec mod while recording in 5x (not sure if I should use crop rec and 5x at the same time) converting DNGs as I type.
Edit: Out of about 10,000 frames not one bad frame!!!!!!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 22, 2017, 08:51:49 AM
@hjfilmspeed: the centered x5 preset from crop_rec is for x5 zoom; all the others are for non-zoomed modes. The help at the bottom should tell you. You can also choose 23.976 for FPS override.

@csound: type IRC in your browser's search box (CTRL-F) ;)

@dfort: yes, I've looked into it because of your report, but it was my explicit request to try the bad commit, not a regular user trying it for e.g. timelapse or whatever. Also, this branch has a lot of stuff that can be broken into smaller branches and back-ported to all other models, in the main builds (already started with this PR (https://bitbucket.org/hudson/magic-lantern/pull-requests/825/prevent-canon-settings-from-being-saved/diff)).

@lostfeliz: I was also thinking to add mlv_sound compatibility to mlv_lite, just didn't look into it yet (since, for my own use, only the full-res and 4K half-FPS modes are interesting - for extracting still frames from a burst sequence, or for timelapse). That's the reason you see weird changes such as single-buffering, ability to record with only 2 buffers, half-shutter trigger etc.

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: csound on April 22, 2017, 09:20:22 AM
Quote
@csound: type IRC in your browser's search box (CTRL-F) ;)

A1ex, apologies for being a newbe clutz - but what do I do when I get to freenode?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Lars Steenhoff on April 22, 2017, 09:22:49 AM
Alex sound would be great, and if the manual lens info could be merged into it, that would be nice too.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 22, 2017, 09:24:43 AM
@hjfilmspeed: the centered x5 preset from crop_rec is for x5 zoom; all the others are for non-zoomed modes. The help at the bottom should tell you. You can also choose 23.976 for FPS override.


@a1ex OHHHHHH I didn't even see that option! That mode also lets you use 12bit and at 3200 iso im getting what seems to be continuous 3072x1308 24 FPS!!!! And its centered!!!!! and you can use the zoomed real time preview if you don't want to use the framing preview!!! No crashes so far.
The only thing I noticed is the shutter speed in at the bottom didn't seem to be registering the right number. Unless something else is going on here. The cameras shutter speed was 1/50 but when zoomed + fps override it showed 1/27 or 1/37 or something like that.
But wow .... This is impressive!!!!  THANK YOU FOR THIS!!!!
This is much more exciting then the C- log announcement. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 22, 2017, 10:00:03 AM
The shutter speed issue appears only when using the 5x preset (it appears fine without crop_rec); can you confirm?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: tifose on April 22, 2017, 01:41:40 PM
Thanks Guys it was a bad pin. back up and running :)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 22, 2017, 02:17:59 PM
The shutter speed issue appears only when using the 5x preset (it appears fine without crop_rec); can you confirm?
Yup confirmed. Just crop mode 5x preset @a1ex just tested now. 5x without crop mode fine.

I also started seeing these stuck pixels pop up in 1:1 5x liveview real time iso3200 1/50 that wouldn't be there in normal 5x zoom real time. I think its Just in the liveview. I didn't try to see if it would appear in RAW file.  Focus box was in was centered in both cases. I feel like this has been covered and I appollowgize if it was.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: BBA on April 22, 2017, 04:13:32 PM
Thanks a lot for the groundbreaking work !!! Keep going !!!!

[Deleted].
I don't know if it can help
...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 22, 2017, 05:16:51 PM
Wow FYI everyone I just did a 5div 4k vs 5diii 3k and 5d3 3k is sooooo much sharper and much more cinematic.

Same for 1080 5diii looks a lot better! I can't believe 2012 5diii has better video quality.

I guess the real test will be when 5div gets RAW video.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: BBA on April 22, 2017, 05:32:03 PM
Very nice !

Restarted everything from new download.
Loaded only the mlv_lite.mo module
Everything seems OK !
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 22, 2017, 06:13:38 PM
@a1ex Tested the latest build it's solid.. Only got one crash when I was using slowmo 45p / 48p, but when I switch to 50p / 60p mode it seems to work fine.  What's the difference between those two slowmo modes other than fps because on the 50p / 60p mode I was able to record @ 48p?   

@hjfilmspeed Awesome didn't think the 5D III could be sharper than the Mark iv in 3k, and definitely the motion in 3K is very cinematic I posted about that.  All I'm gonna say is a1ex got the keys.. keys.. keys..  Can you share a short clip of each camera?  Did you zoom in 200% - 400% to pixel peep. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 22, 2017, 06:47:31 PM
@goldenchild9to5 I'll post my quick test but before I hard conclude this I'll have to give the 5div a more suitable test.
I zoomed in and its really hard to to make a conclusion here but in both cases I gave the 5div the advantage especially with autofocus. First 2 shots are 1080 shots 3 and 4 are the higher resolution.
I tried to grade the RAW fairly.
See if you can tell what's what. I like the RAW better for sure.
If you download it it's a Cineform AVI in UHD
Also keep in mind I can only export UHD from Resolve.
https://vimeo.com/214307580
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 22, 2017, 08:38:52 PM
@hjfilmspeed You are definitely right the 5D III images looks better especially in 3K..
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 22, 2017, 09:41:08 PM
What's the difference between those two slowmo modes [slowmo 45p / 48p and 50p / 60p] other than fps because on the 50p / 60p mode I was able to record @ 48p?

FPS and vertical resolution. The 48p mode is really pushed to the limit (already discussed), while the 45p should have a little more headroom.

The cameras shutter speed was 1/50 but when zoomed + fps override it showed 1/27 or 1/37 or something like that.

Solved, it was just an oversight from my side.

Also fine-tuned the preview behavior: you can use a long half-shutter press to change between framing and real-time (e.g. if you normally use the centered real-time preview in x5 zoom, but you also want to check the framing every now and then, just press half-shutter for 1 second or so).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 22, 2017, 09:45:53 PM
@a1ex Thanks got it, didn't read up about that.   
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 22, 2017, 10:20:08 PM
Solved, it was just an oversight from my side.

Also fine-tuned the preview behavior: you can use a long half-shutter press to change between framing and real-time (e.g. if you normally use the centered real-time preview in x5 zoom, but you also want to check the framing every now and then, just press half-shutter for 1 second or so).
This is incredible! Soooo impressed!

Also I just replaced my test with a much better one

https://vimeo.com/214307580
This means Either Canons 5Div 4K is actually 3K
or
That we get to call 5D3 ML 3K  ... 4K
4 Clips
2 are 5div 1/100 f11 ISO 400 Sigma Art 35mm 4k
2 are 5Diii 1/50 f11 ISO 400 Sigma Art 35mm 3k  ML
14bit lossless using 4-21-2017 4K Crop Rec build
(I underexposed the 5Div because I was clipping highlights at 1/50 shutter)
Both rendered to Cineform UHD and both were precisely focussed on the crosshatch on the color chart.
Can you tell which is which.
Download and zoom in. It's a UHD Cineform AVI
 5Diii was quickly graded to match Canon's neutral picture style of the 5div

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 22, 2017, 10:54:27 PM
Sorry to be a party pooper but can anyone else confirm that this is reproducible (it seems to be on my end w build 2017-04-22 for 5D3.123) if one were to record in 3504x1400 resolution (and only in that resolution which is 2.50:1)?

It seems to produce a pattern of corruption after each 36th DNG frame from 14-bit lossless shot in 24p.

(https://c1.staticflickr.com/5/4183/33820054770_a4d51abbce_n.jpg) (https://flic.kr/p/TwyC1A) (https://c1.staticflickr.com/3/2934/34204130195_ca8604c418_n.jpg) (https://flic.kr/p/U7v7fF)
(https://c1.staticflickr.com/5/4172/33361447354_46a53aea0c.jpg) (https://flic.kr/p/SQ391b)

Here's an original corrupted 36th DNG: https://bitbucket.org/DeafEyeJedi/magic-lantern/downloads/5D3.123_3504x1400_corruption%3F.dng

Settings used in ML:
(https://c1.staticflickr.com/3/2834/33393407053_c0143f495d.jpg) (https://flic.kr/p/SSRWvt)

SO far in other resolutions seems to be getting better results other than the random first DNG's being either black (blank) or half screen corruptions, etc from each certain MLV's. Gotta troubleshoot some more on this to narrow down the culprit. Don't think it's random at all.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 22, 2017, 11:02:01 PM
@hjfilmspeed Great test.. This one was pretty hard to tell the difference.  Is the 5D III scenes 2 & 4?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 22, 2017, 11:30:46 PM
@hjfilmspeed Great test.. This one was pretty hard to tell the difference.  Is the 5D III scenes 2 & 4?
@ goldenchild9to5 did you download and zoom in? Which looks sharper or which do you prefer?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 22, 2017, 11:44:50 PM
@DeafEyeJedi: does it help if you reduce the vertical resolution from the crop_rec submenu a bit? (e.g. instead of 1536, try 1500, 1450, 1400)

If you can reproduce it consistently without the resolution override, try the following patches

edit: nevermind; had some hypotheses, but rejected all of them.

Reproduced.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hindra on April 22, 2017, 11:48:37 PM
I was able to replicate  the 19 FPS bug in the menu I reported a while baclk. If I forget to turn FPS Override off, turn on Crop Mode module for UHD recording, enter canon menu then go back into ML menu, when I turn off FPS Override it defaults back to 19 FPS. If FPS Override is off when I turn on Crop Mode, everything is fine.

(http://i.imgur.com/XvzW1N6.jpg)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hindra on April 23, 2017, 12:01:56 AM
This is incredible! Soooo impressed!

Also I just replaced my test with a much better one

https://vimeo.com/214307580
This means Either Canons 5Div 4K is actually 3K
or
That we get to call 5D3 ML 3K  ... 4K
4 Clips
2 are 5div 1/100 f11 ISO 400 Sigma Art 35mm 4k
2 are 5Diii 1/50 f11 ISO 400 Sigma Art 35mm 3k  ML
14bit lossless using 4-21-2017 4K Crop Rec build
(I underexposed the 5Div because I was clipping highlights at 1/50 shutter)
Both rendered to Cineform UHD and both were precisely focussed on the crosshatch on the color chart.
Can you tell which is which.
Download and zoom in. It's a UHD Cineform AVI
 5Diii was quickly graded to match Canon's neutral picture style of the 5div




1 and 3 are much sharper when viewed at 200%
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 23, 2017, 02:28:03 AM
Sorry to be a party pooper but can anyone else confirm that this is reproducible (it seems to be on my end w build 2017-04-22 for 5D3.123) if one were to record in 3504x1400 resolution (and only in that resolution which is 2.50:1)?

Narrowed down and solved; excellent catch @DeafEyeJedi!

That's how a proper bug report looks like - it contains all the details to allow others to reproduce it. This bug was quite hard to trigger, given all those reports with no corrupted frames.

Also got 4096x3072 working at 12 and 12.5 FPS (24/25 in Canon menu) and did some more fixes for very large resolutions. Even if you only get a few frames, I still find these modes useful with half-shutter trigger and pre-recording (e.g. for taking photos of really unpredictable subjects).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: dfort on April 23, 2017, 04:18:58 AM
@dfort: yes, I've looked into it because of your report, but it was my explicit request to try the bad commit, not a regular user trying it for e.g. timelapse or whatever. Also, this branch has a lot of stuff that can be broken into smaller branches and back-ported to all other models, in the main builds (already started with this PR).

Oops, sorry. I bit off more than I could chew when I came across that bug and it was just easier for me to switch to an earlier (smaller) branch. I'd like to get lossless compression working on other cameras because that seems doable while 4k on an EOSM is pie in the sky. Though I wouldn't rule it out considering how quickly ML development has been progressing lately.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: lanhl on April 23, 2017, 11:16:28 AM
Thanks for the wonderful work. I Got some problem when update to the new buuld, don't know how to fix it.

My camera is 5D3, 1.2.3

I have installed the Apr-04 build before, it works, and then update to the newest Apr-23 build, then I got this scene on top.
(http://i.imgur.com/czeNgzz.jpg)
I tried to Uninstall, and format CF card, Install again, disable all modules, but got the same scene. then I tried degrade to the earlier build, the same, But the Apr-04 and Apr-06 build works fine.

Can any body know how to fix this? Thanks a lot.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: jankrueck on April 23, 2017, 11:35:11 AM
sorry for beeing an asshole. But did you read the screenshot or any page of this tread, yet?  :D
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: bazidl on April 23, 2017, 03:21:04 PM
I'm getting frame and block size mismatch errors. I saw a few other people were getting various versions of this error as well but no responses with a possible solution. I know it's related to the 14 bit - lossless format as I don't get the message when I turn lossless off.

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 23, 2017, 05:21:45 PM
Testing April 23th build 5D3 113.
When I change to a 192050p3x3 mode with fps override on, and try to switch fps override to off, camera freezes and I get Error80. Need to pull battery.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 23, 2017, 08:24:27 PM
@hjfilmspeed Check it out Zoomed in to 200% & 400% Scenes 1 & 3 is a tiny tiny bit sharper than Scenes 2 & 4.  But here is the thing the tree behind the color checker seem a bit sharper in Scenes 2 & 4. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Ilia3101 on April 23, 2017, 09:08:55 PM
@lanhl Follow what it says on the screen. There was a dangerous bug in April 04 build so DON'T go back to it, and the new ones are detecting if that bug has affected your camera. It's Magic Lantern looking out for you ;)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Ali Oliya on April 24, 2017, 08:25:29 AM
Hi,

I used mlv_dump in mac for creating dngs. but unable to open them in After Effects CC 2017. Photoshop opens files with no problem. but AE gives me this error:  "after effects error: the file formay module not parse that file. (45::35)"
Title: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 24, 2017, 08:58:15 AM
@Ali Oliya - is the first DNG in your folder blank or corrupted in any way?

If so, try deleting it and try again which should open up fine in AE. Have had this happened to me in the past.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 24, 2017, 09:02:11 AM
I was away for some time. Trying latest build.
Way much solid!
The exclusions of many combinations makes things more targeted.
Thank you for the hard work and constructive development
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Ali Oliya on April 24, 2017, 09:11:46 AM
@DeafEyeJedi  yes it was corrupted. i deleted it. still doesn't open. but the error message changed to: After Effects error: Photoshop file format error (-1). (45 :: 35)
Title: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 24, 2017, 09:40:41 AM
Hmm would you mind using @Danne's latest cr2hdr.app to spit out original samples (#12 under mlv_dump menu) and upload the zip file for us to check out?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 24, 2017, 11:34:55 AM
@hjfilmspeed Check it out Zoomed in to 200% & 400% Scenes 1 & 3 is a tiny tiny bit sharper than Scenes 2 & 4.  But here is the thing the tree behind the color checker seem a bit sharper in Scenes 2 & 4. 
1 and 3 are mark iii
Hmmm Ill double check my lens details to see if f-stop was different. It could be possible that focal planes where not exact. I focused both on the crosshatch with autofocus. but in any event at 400% we should be seeing a HUGE difference in sharpness here. 5dIv has 1000 more pixels. Any way I don't want to go more off topic then I have.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 24, 2017, 12:18:24 PM
Any dev think if there is any possibility to get some signal through hdmi in the new full res live view mode?... just for framing and focussing, don't mind if res is low (VGA)...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Ali Oliya on April 24, 2017, 12:49:05 PM
@DeafEyeJedi - i used cr2hdr app from Danne. But it doesnt let me open the MLV file at all! .mlv file is grayed out....(same as rawmagic)

ps. i used crop rec 3k-4k 14 April expermiment build btw. so far the only software that is able to open the .mlv files from this build is MLV_Dump (for mac). (raw2cdng opens but creates pink dngs!)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 24, 2017, 01:10:34 PM
Greyed out? Could you share a screenrecording or a screenshot? Maybe the LOG.txt file?
Maybe post about it here.
http://www.magiclantern.fm/forum/index.php?topic=15108.100
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Ali Oliya on April 24, 2017, 03:13:53 PM
@Danne - yes, i can not choose the file. it can't be selected:

(https://preview.ibb.co/ebuBmQ/Screen_Shot_2017_04_24_at_5_33_34_PM.png) (https://ibb.co/j0zcRQ)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 24, 2017, 04:02:22 PM
You don´t choose files you choose the folder containing files. If you press choose in this folder a menu will pop up and you take it from there.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Ali Oliya on April 24, 2017, 04:22:48 PM
@Danne - Thank you! Sorry, it was the 1st time using this app! done. fixed. now i can open .dngs in AE without problem.

@DeafEyeJedi Thanks for your help
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 24, 2017, 06:37:12 PM
On 12 April 2017 I took several shots.
I was recording with that days latest build, 1920x800 60FPS 14bit lossless with 5dmk3.
On some of the shots I took, on the top of the frame there is a horizontal sharp edge that flickers, about 100 pixels away from top

Here is a link
https://drive.google.com/open?id=0B-d8ARtc7xwWT3BwbkdQYjdaLUE

(I am not aware if this is fixed already)
Title: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 24, 2017, 07:16:22 PM
Quote from: a1ex
Narrowed down and solved; excellent catch @DeafEyeJedi!

That's how a proper bug report looks like - it contains all the details to allow others to reproduce it. This bug was quite hard to trigger, given all those reports with no corrupted frames.

Great to hear. Thanks for the acknowledgement!

Quote from: a1ex
Also got 4096x3072 working at 12 and 12.5 FPS (24/25 in Canon menu) and did some more fixes for very large resolutions. Even if you only get a few frames, I still find these modes useful with half-shutter trigger and pre-recording (e.g. for taking photos of really unpredictable subjects).

Oh absolutely this will be useful. Especially for birding, sports photography & whatnot. This is incredible and should be a bit more forgiving than Canon's stock full burst mode which is only 6 frames per sec for 5D3.  :P

1 and 3 are mark iii
Hmmm Ill double check my lens details to see if f-stop was different. It could be possible that focal planes where not exact. I focused both on the crosshatch with autofocus. but in any event at 400% we should be seeing a HUGE difference in sharpness here. 5dIv has 1000 more pixels. Any way I don't want to go more off topic then I have.

For those who havant seen them up-close w comparisons yet in between 200% & 400% (Thanks @hjfilmspeed for sharing them!):

#1) 5D3 RAW 3K 1.85:1 (upscaled to 4K in post)
(https://c1.staticflickr.com/3/2827/34244929975_b78ee13b4d.jpg) (https://flic.kr/p/Ub7dBa) (https://c1.staticflickr.com/3/2868/34244928415_5c1066c961.jpg) (https://flic.kr/p/Ub7d9g)

#2) 5D4 MJPEG 4K
(https://c1.staticflickr.com/5/4169/34114706071_ae086da248.jpg) (https://flic.kr/p/TYAMzr) (https://c1.staticflickr.com/5/4178/34244928795_8f2fa0d461.jpg) (https://flic.kr/p/Ub7dfP)

#3) 5D3 RAW 3K 1.85:1 (upscaled to 4K in post)
(https://c1.staticflickr.com/3/2806/33403755734_c358e665fa.jpg) (https://flic.kr/p/STLYP1) (https://c1.staticflickr.com/3/2857/33403754414_df65c4273c.jpg) (https://flic.kr/p/STLYqf)

#4) 5D4 MJPEG 4K
(https://c1.staticflickr.com/3/2876/34244931825_d2e4cb6740.jpg) (https://flic.kr/p/Ub7ea4) (https://c1.staticflickr.com/5/4166/33403754644_21c9204576.jpg) (https://flic.kr/p/STLYud)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 24, 2017, 07:47:56 PM
@DeafEyeJedi the 5diii was actually 3k 3072 1.85:1 but technically upscaled at export.
Title: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 24, 2017, 08:12:14 PM
Thanks @hjfilmspeed -- fixed! (It's even more flabbergasting to know that the 5D3 still looks sharper even when upscaled to 4K)  :o
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 24, 2017, 08:31:48 PM
Thanks @hjfilmspeed -- fixed! (It's even more flabbergasting to know that the 5D3 still looks sharper even when upscaled to 4K)  :o


DeafEyeJedi Thank you for posting those crops!
Actually, you might as well call the the 5DIII 3K - 4K. Why not canon apparently seems to think it is okay. I thought to myself, maybe its the lens.. then I was like no way, the Sigma Art prime is design to resolve way high resolution.
It could be the 422 vs the .... 444 or what ever MLV 14Bit lossless is.
Regardless, this build gets better every time. Can't wait to see what the future holds!

Oh I meant to ask this, should we be using 123 vs 113 for 5d3? It was my understanding that 113 was fine unless you need hdmi features and AF @ f8. Is this still true?
Or should we be testing 123?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: giarcpnw on April 24, 2017, 08:33:59 PM
Hey all, how are you guys addressing the Vertical Lines or other pixel issues with these new builds? I'm on OSX. Martin Hering had created a great mlv to cng converter app called Footage that addressed them but he seems to be letting it time bomb in the 15th of May to pursue a revenue generating path.  So now we're back to square one.

MLVDump does not fix them.  Is there another app that i'm not aware of that does?

Thanks,

Craig
Title: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 24, 2017, 08:44:32 PM
...Oh I meant to ask this, should we be using 123 vs 113 for 5d3? It was my understanding that 113 was fine unless you need hdmi features and AF @ f8. Is this still true?
Or should we be testing 123?
AFAIK 113/123 should be identical ever since @a1ex merged these 'two firmwares under one branch' though correct me if I'm wrong since this is experimental.

...he seems to be letting it time bomb in the 15th of May to pursue a revenue generating path.  So now we're back to square one.

That's some hogshit, eh?

MLVDump does not fix them.  Is there another app that i'm not aware of that does?

Care to share some original short samples @giarcpnw? (use @Danne's cr2hdr.app (https://www.magiclantern.fm/forum/index.php?topic=15108.0cr2hdr.app) if you haven't already)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 24, 2017, 09:02:52 PM
@DeafEyeJedi thank you!!!

Also has anyone tried these settings? 4/23/2017 build
Crop Rec with 5x zoom 3.5k  enabled
And 12bit lossless with 3072 2.35:1 ?
It's very stable to me! Continuous with a good card!
And you can hit the DOF preview button or half shutter to zoom in with real time to focus!
I saw this mentioned but didn't try it yet. This is awesome!
I feel like real time framing preview of 3k is so close now!
(Coding wise it's probably not)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: giarcpnw on April 24, 2017, 09:05:26 PM
Care to share some original short samples @giarcpnw? (use @Danne's cr2hdr.app if you haven't already)

cr2hdr didn't seem to fix when i tried last week. unless it's been upgraded since.


The are DNGs from MLVdump, you can see the banding/lines in the mids. They get worse once you adjust exposure and sharpness. These were shot 14bit lossless with the April 10th build I believe. My understanding is this is inherent in the raw data and the converters are coded to adjust for it. I know there was an early post in this thread that said most of the apps or command line converters had not been adjusted to address this with the new 14bit 4k build's mlvs. Just wondering if they had yet or I missed one. Martin added it to his at my request last week, but...

https://www.dropbox.com/sh/a4158kqm85rvime/AADYDd6H8PyjYyGE1fXqKpeza?dl=0

Thanks,

C

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 24, 2017, 09:12:03 PM
Actually could you please just use cr2hdr.app (press 'm' to get in mlv_dump menu then press '12' to start sample processing). Need the originals.

Be sure to use a temporary Folder and include MLV inside since the app can only process from within Folders and not directly on the MLV file itself.

Also try to use the latest experimental builds whenever possible since it's always evolving for greatness.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: giarcpnw on April 24, 2017, 09:32:09 PM
Actually could you please just use cr2hdr.app (press 'm' to get in mlv_dump menu then press '12' to start sample processing). Need the originals.

Be sure to use a temporary Folder and include MLV inside since the app can only process from within Folders and not directly on the MLV file itself.

Also try to use the latest experimental builds whenever possible since it's always evolving for greatness.

Same folder: you'll see the zip pack.
https://www.dropbox.com/sh/a4158kqm85rvime/AADYDd6H8PyjYyGE1fXqKpeza?dl=0

I have the newest April 23rd build. I just haven't shot any new footage since the April 19th. It was there in the 19th too. These are the MLVs I have with me at the moment.

Thanks,

C

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 24, 2017, 09:35:57 PM
Oh I meant to ask this, should we be using 123 vs 113 for 5d3? It was my understanding that 113 was fine unless you need hdmi features and AF @ f8. Is this still true?

Correct.

I'm currently using 1.2.3, but that's because it has more quirks (usually, if something works on 1.2.3, it's likely OK on 1.1.3, but in the other direction, there are a lot of exceptions...)

The notes from 1.1.3 thread (http://www.magiclantern.fm/forum/index.php?topic=14704) are still valid.

AFAIK 113/123 should be identical ever since @a1ex merged these 'two firmwares under one branch' though

That means both ML versions are compiled from the same codebase (but each one is still tweaked to account for Canon firmware quirks - just like all other ML ports).

MLVDump does not fix them [vertical lines].  Is there another app that i'm not aware of that does?

I've explained here (http://www.magiclantern.fm/forum/index.php?topic=18443.msg183653#msg183653) what needs to be done for other converters (including mlv_dump). I'm not really familiar with compressed formats, and there are people in the ML community who can do this faster than me (hint, hint).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 24, 2017, 09:41:03 PM
Also has anyone tried these settings? 4/23/2017 build
Crop Rec with 5x zoom 3.5k  enabled
And 12bit lossless with 3072 2.35:1 ?

Confirmed after recording both takes for over 9 minutes each now @ ISO 3200 in 24p. Green icon all the way. Though it felt a bit toasty @ 53-54C towards the end, ha! :P
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: giarcpnw on April 24, 2017, 11:21:37 PM
Quote
I've explained here (http://www.magiclantern.fm/forum/index.php?topic=18443.msg183653#msg183653) what needs to be done for other converters (including mlv_dump). I'm not really familiar with compressed formats, and there are people in the ML community who can do this faster than me (hint, hint).



If only i had studied computer science in college instead of theater, I might be able to fix this myself. Instead, I'll have to recite shakespeare to myself while I wait for someone else with programming knowledge.

C
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 25, 2017, 01:04:13 AM
@hjfilmspeed 5D III is really a force to be reckon with, can't wait for liveview to have a smooth feed  @DeafEyeJedi great screen capture..
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: csound on April 25, 2017, 02:38:39 AM
Can someone please point me to information on procedure to fix null pointer bug from April 4 build. Screen alert on camera indicates to prepare logs and go to freenode on IRC - what happens from there? Sorry, unfamiliar with all this.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 25, 2017, 02:39:39 AM
Just tried the new Davinci Resolve 14 Beta with 5D III 3K footage man this new update is smooth.  I've noticed as well that It might be processing dng's better, seems to resolve images detail a lot better.  If anybody else used it let me know if you are experiencing the same thing.   
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 25, 2017, 02:46:24 AM
@csound Contact a1ex, he will guide you. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Oedipax on April 25, 2017, 03:06:31 AM
Quote from: goldenchild9to5
Just tried the new Davinci Resolve 14 Beta with 5D III 3K footage man this new update is smooth.  I've noticed as well that It might be processing dng's better, seems to resolve images detail a lot better.  If anybody else used it let me know if you are experiencing the same thing.   
Same experience here - realtime playback on my 3k 2.39:1 14bit CDNG files, up to a few nodes. Extremely pleased with the new Resolve so far.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 25, 2017, 04:06:50 AM
@Oedipax I'm glad that you are getting the same results so I'm not the only one.  @a1ex new build is awesome thank you will be testing more tonight.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 25, 2017, 11:05:13 AM
Quote
Quote from: a1ex

    Also got 4096x3072 working at 12 and 12.5 FPS (24/25 in Canon menu) and did some more fixes for very large resolutions. Even if you only get a few frames, I still find these modes useful with half-shutter trigger and pre-recording (e.g. for taking photos of really unpredictable subjects).

Quote from: DeafEyeJedi
Oh absolutely this will be useful. Especially for birding, sports photography & whatnot. This is incredible and should be a bit more forgiving than Canon's stock full burst mode which is only 6 frames per sec for 5D3.  :P

A dumb question maybe... but... someone are getting realtime preview in this mode? How can you focus and framing for sports and birds?... I think I'm missing something...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 25, 2017, 11:17:07 AM
A dumb question maybe... but... someone are getting realtime preview in this mode? How can you focus and framing for sports and birds?... I think I'm missing something...

Want to help? Grab ADTG_GUI and QEMU and start investigating!

All you need to do is to understand how Canon configures the image processing path for LiveView. Here's a starting point. (http://www.magiclantern.fm/forum/index.php?topic=2388.msg183168#msg183168)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 25, 2017, 11:45:01 AM
I'm taking a look to the Starting Point... and I don't know what to do... I have no knoweledge in any coding languaje. For me is like cheneese  :( :-[ I'm sorry but if I can do something more just let me know.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 25, 2017, 12:25:05 PM
The difficult part is simply understanding what Canon code does (and what we have to change to fix the preview).

Those numbers from the logs and adtg_gui are like Chinese to all of us; the trick is to fiddle with them until a tiny part of them starts to make some sort of sense. That's how we figured out how to reconfigure the sensor to 4K in the first place ;)

Some useful tricks: comparing all those numbers in different video modes (adtg_gui has a simple register diff tool), recognizing values that could look like resolutions, logging the I/O activity in QEMU, or analyzing the still photo mode first (it's simpler).

And, of course, don't give up after reading a single post. Make yourself comfortable with what's happening in the Reverse Engineering area. Some relevant threads are linked in the first post; make sure you can follow them.

Of course, you will need some basic programming knowledge.

Bottom line: the real-time preview is not implemented because... I have little or no idea about how Canon configures the image processing path.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: motionSOUL on April 25, 2017, 02:44:35 PM
Sorry don't be pissed at me, but what is the best solution to convert 14 bits lossless .mlv files?
I've red we can use cr2hdr.app_4K, but i don't seem to find a link to it, only the cr2hdr.app version. Is this application compatible with all the .mlv flavours?
Is there better way to do so? I've always used MLVMystic which is simple and elegant to use…

Thanks,

S/.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 25, 2017, 02:51:29 PM
mlv_dump is your friend
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 25, 2017, 03:04:45 PM
@A1lex
Thank you very much for the explanation!!!... I feel really dumb but I guess not everyone will know about everything. Coding is a unknown world for me. The most obvious things for you are absolutely unknown for me. :-[
I would like to do more than only download the builds, but outside testing that builds I don't know what can I do...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: beauchampy on April 25, 2017, 03:37:46 PM
Sorry don't be pissed at me, but what is the best solution to convert 14 bits lossless .mlv files?
I've red we can use cr2hdr.app_4K, but i don't seem to find a link to it, only the cr2hdr.app version. Is this application compatible with all the .mlv flavours?
Is there better way to do so? I've always used MLVMystic which is simple and elegant to use…

Thanks,

S/.

cr2hdr has been updated, so no need for the '4k' version anymore. Just grab the latest from here - https://bitbucket.org/Dannephoto/cr2hdr/downloads/
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: jimiz on April 25, 2017, 04:38:55 PM
sorry , big confusion here...2 questions please.

1) for decode dng files of this exp.release , must be used only with last release of mlv_dump (line command without interface) ? because I try anothers programs, but are not compatible, only pink and black dng.

2) any one can please tell me, the right setting for recording in 1920 50p  ?  must be select also in menu canon 1280x50p  or can be  1920 x 25 all-i) ?   must be set  FPS over to 50 ? 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 25, 2017, 04:51:28 PM
As long as you use the Experimental Build, you have to use mlv_dump. Its located in the same page to download.
From the Camera-Icon menu / Crop / 1920 50/60 3x3
You have three options.

If you select Canon's 1920x1080 25FPS, you get 1920x1080 25FPS.
If you select Canon's 1280x720, you get 1920x960 50FPS.
If you select Canon's Video System as NTSC instead of PAL, you get 1920x800 60FPS.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: jimiz on April 25, 2017, 04:59:17 PM
thanks, now is ok.
for a recording 1920x960 50p  continued , can be set 14bit loss ore better 12 ?
I'v read 1920 50p can be continued...but to me stoped later few seconds.
CF cbay 128  x 1066
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 25, 2017, 05:12:04 PM
Read the little help text at the bottom of your screen.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: motionSOUL on April 25, 2017, 05:31:05 PM
I've tested 1920x960 50p 14 bits lossless, convert the .mlv file with mlv_dump, all .dng are black for the moment, don't know why. I had no error message during the recording and stopped it myself.
I've also tried 4K at 24p, but I haven't found the way to get the 4096 x 1440 ratio it is written everywhere?
In 4K, preview isn't working, right?

Thanks and many thanks to all the team which is doing so much work to fill the gap Canon has let behind.
I'm currently finishing my second short with the help of Magic Lantern and it is great to be able to get such video quality!

S/.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: beauchampy on April 25, 2017, 05:41:52 PM
I've tested 1920x960 50p 14 bits lossless, convert the .mlv file with mlv_dump, all .dng are black for the moment, don't know why. I had no error message during the recording and stopped it myself.
I've also tried 4K at 24p, but I haven't found the way to get the 4096 x 1440 ratio it is written everywhere?
In 4K, preview isn't working, right?

Thanks and many thanks to all the team which is doing so much work to fill the gap Canon has let behind.
I'm currently finishing my second short with the help of Magic Lantern and it is great to be able to get such video quality!

S/.

Are they black if you look at them in After Effects or Resolve?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: motionSOUL on April 25, 2017, 05:58:16 PM
In After Effects / Photoshop / OSX Preview...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 25, 2017, 06:13:44 PM
Which converter did you use @motionSOUL? Latest cr2hdr.app (https://www.magiclantern.fm/forum/index.php?topic=15108.0cr2hdr.app) has been working quite flawlessly w lossless MLV's as of late.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: motionSOUL on April 25, 2017, 06:34:41 PM
No, I've used mlv_dump from the Terminal. I will test cr2hdr.app, which has just converted a 4096x1366 test file to .dng flawlessly.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: vstrglv on April 25, 2017, 09:31:24 PM
thanks, now is ok.
for a recording 1920x960 50p  continued , can be set 14bit loss ore better 12 ?
I'v read 1920 50p can be continued...but to me stoped later few seconds.
CF cbay 128  x 1066

What is better 12bit or 14bit lossless depends on an image content. It is a pity that 12bit (10-8) lossless is not available for this mode.  For 1920x960 @ 50p (both 1:1 crop and full-frame - 3x3 pixel binning) recording length depends on an image content. Sometimes 40 sec, sometimes 10. But for 40 fps (FPS override, Exact fps) I got continues rec. SanDisk, Extrime pro 160MB/s, 256GB.
BTW could not set 4096x1440 @ 25p (1:1 crop) mode. How?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 25, 2017, 09:45:36 PM
BTW could not set 4096x1440 @ 25p (1:1 crop) mode. How?

The preset is named "4K 1:1 half (https://en.wikipedia.org/wiki/One_half)-FPS". See also the first post.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: vstrglv on April 25, 2017, 10:33:01 PM
The preset is named "4K 1:1 half (https://en.wikipedia.org/wiki/One_half)-FPS". See also the first post.
Thank you A1ex! Canon menu 50fps.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: motionSOUL on April 25, 2017, 10:56:15 PM
Hello A1lex,

Do we have to do something when the Memory patches is turned red, red on Code:FF28CC3C?
It seems to have appear after my first test with the Apr23.5D3113 build when testing 1920x960 50p, I had the screen filled with error alerts, then this...

Thanks,

S/.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Savely on April 26, 2017, 01:40:21 AM
Can someone write tutorial about how to rec on these modes? Why do I get this http://joxi.ru/YmEqBpZF06jjnm.jpg instead of DNG? What mlv_dump for?

Also I don't get how card spanning is working (it seems it doesn't) for when I enable this mode then at 3200x1200 it record 4 seconds (I checked it with various superfast CF-cards), but with only one CF card it also rec 4 seconds only!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Ali Oliya on April 26, 2017, 10:11:37 AM
Quote
Can someone write tutorial about how to rec on these modes? Why do I get this http://joxi.ru/YmEqBpZF06jjnm.jpg instead of DNG? What mlv_dump for?
mlv_dump is lovely!!  make sure you use the mlv_dump in experiment page. otherwise you end up with noisy pink DNGs. been there before.  or you can try the amazing cr2hdr.app. (note: dont judge the frames by their thumbnails. open them in photoshop to see if they are converted right. and not the first frame also!)
I see you have opened the first frame which is usually pink and corrupted with this mode. have you tried other frames too?!

this is MLV_Lite with this build which is different than the old MLV. MLVRawviewer and raw2cdng can't open it or they deliver pink DNGs instead.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: BBA on April 26, 2017, 11:34:49 AM
@motionSOUL to help
Nothing to worry about : see my post #174 and A1ex reply #175 Question asked many times...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: jimiz on April 26, 2017, 06:50:13 PM
Read the little help text at the bottom of your screen.


yes , but also is write green 80 mb/s CONTINUE rec. 

after 10-12 seconds a sequence of BEEP and stop the recording...

so I try anothers crope mode...like 3k , I crope like I have 70/80 mb max writing and compare continue recording...

preview is slow and gray  , recording number  stay green....(think normal) but after 10-15 seconds ...STOP !

CF is perfect (x1066) and work very well without stop with normal raw to 90/100 mbs

I don't know why many anothers here say can recorded continued in same situations....

Title: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 26, 2017, 07:08:42 PM
What brand CF cards are you using @jimiz? Are they formatted into exFAT at the very least?

Also were you using '3.5k 1:1 center x5' within crop_rec.mo from ML menu and more importantly did you enabled FPS override to 23.976 while in x5 zoom?

Here's a screenshot from MLVFS webGUI to show that I was able to achieve continuous w 12-bit lossless in both 3072x1308 & 3072x1320 in 24p @ ISO 3200 until pressed stop.

(https://c1.staticflickr.com/3/2809/33444364344_551229871d_c.jpg) (https://flic.kr/p/SXn7ks)

This was used w KomputerBay 128GB 1066x and it was formatted to exFAT via Disk Utility on OS X prior to recording (256GB cards can be formatted in cam while keeping the exFAT).

Settings used under Movie mode within ML:

(https://c1.staticflickr.com/3/2811/33902403100_12478ee69e.jpg) (https://flic.kr/p/TDQFio) (https://c1.staticflickr.com/5/4185/33902403380_4bf81ce3fd.jpg) (https://flic.kr/p/TDQFod)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 26, 2017, 08:18:56 PM
@DeafEyeJedi has the magic settings here. I can verify most of this. But you need to make sure ML is on your SD card and you are recording to  your CF card that is formatted exfat from your computer. DeafEyeJedi settings are exactly what I have been using. It you want a smidge extra headroom you can set it to 2.35:1 which brings your vertical resolution down a tiny bit ... I think.

Edit: actually... I just got these settings to spike!
ISO 100 at some very bright trees! It went from like 89 MBs to 114 MBs I'll upload a dng

At the 11-8 bit Lossless with 3008x1280 I was able to get the same scene down to 90ish A1ex was right! (Not hard to believe) that you can get seemingly continuous settings to spike high! Very high
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Savely on April 26, 2017, 08:55:15 PM
Quote
@Ali Oliya
mlv_dump is lovely!!  make sure you use the mlv_dump in experiment page. otherwise you end up with noisy pink DNGs. been there before.  or you can try the amazing cr2hdr.app. (note: dont judge the frames by their thumbnails. open them in photoshop to see if they are converted right. and not the first frame also!)
I see you have opened the first frame which is usually pink and corrupted with this mode. have you tried other frames too?!

this is MLV_Lite with this build which is different than the old MLV. MLVRawviewer and raw2cdng can't open it or they deliver pink DNGs instead.

But where is this experiment page? I just download build pack and mlv_dump.exe but what should I do with this exe file? I actually can't understand why it so difficult to write tutorial about how to use these.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Walter Schulz on April 26, 2017, 09:10:46 PM
http://www.magiclantern.fm/forum/index.php?topic=19300.msg183363#msg183363
A user makes fun of this site and calls it "shark bay". If that is true you are pretty much bathing in Piranha infested waters right now.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 26, 2017, 09:35:03 PM
But where is this experiment page? I just download build pack and mlv_dump.exe but what should I do with this exe file? I actually can't understand why it so difficult to write tutorial about how to use these
@Savely
What modes exactly? This tutorial would be a lot to go through. I assume you read DeafEyeJedi settings and you are up to speed on the new build.
I have some that would offer even more stability depending on what you are filming.
3008x1280 11-8 bit lossless MLV Lite with preview set to auto.
Crop rec mode set to 5x zoom 3.5 k mode
Also it looks like you are having post processing issues.
If so it's easy.
For windows:
1) Download RawFlow v0.2 http://www.magiclantern.fm/forum/index.php?topic=13338.0
2) Download new MLV_dump https://builds.magiclantern.fm/experiments.html
3) Open up the Rawflow folder and delete current the MLV_Dump that came with Rawflow and replace it with the new MLV_dump you downloaded from here  https://builds.magiclantern.fm/experiments.html leave this folder open for now.
4.) download your MLVs off your card to a new folder on your computer.
5.) Open up the RAWFlow folder if you closed it.
6.) Select all your MLVs and drag and drop them right on the Rawflow.exe application icon in the Rawflow folder (note you do not have to open the exe or run it. It will do that on its own after you drag and drop mlvs on it)
7.) The app will open a command prompt window. Let it do its thing and it will place new folders in the same folder as you MLVs with DNGs for you to play with.

Hope this helps and hope this doesn't break your camera or computer. I think this workflow is similar on MAC but not sure. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 26, 2017, 10:23:02 PM
@DeafEyeJedi
sorry to double post but it appears that even 3072 x 1308 12bit lossless 23.976 fps ISO 100 under 5x zoom can cause a spike in data rate. Like I think it said 117 MB/s
@A1ex told me that if you try hard enough you can find a scene that can do this. It appears that it's not just high iso that can cause a data rate spike. Even at iso 100  12bit with a brightly exposed detailed scene that data rate can shoot through the roof. I assume if the details are bright enough to stay out of compression then this can happen. That's my guess anyway. I was able to get the scene back to 90MBs by using 11-8 bit at 3008x1280 here is the DNG that went off the charts.
https://www.dropbox.com/sh/8lmopv8z7p2i91f/AAA4saYTmLE7rFQ3t7nSUBCaa?dl=0

1920x1080 14bit lossless 24p was still okay. Not even close to danger zone.

EDIT!!!!!!! I actually tried to rec this scene at 1920x1080 14bit lossless 24p even tho MB/s was good but it gave me an ML assert
ML ASSERT:
0
at mlv_lite.c:2344 (compress_task), task compress_task
lv:1 mode:3

Then I notice that the compression ration was at 106% or something like that. @a1ex I found the scene you were talking about!
all frames are here:
https://www.dropbox.com/sh/8lmopv8z7p2i91f/AAA4saYTmLE7rFQ3t7nSUBCaa?dl=0
 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: vstrglv on April 26, 2017, 10:36:04 PM
It will be nice to have an option for manual bit lossless setting in order to reduce data rate.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 26, 2017, 10:52:31 PM
It will be nice to have an option for manual bit lossless setting in order to reduce data rate.

Not after seeing samples like (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183187#msg183187) these (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183208#msg183208), but feel free to prove me wrong (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183231#msg183231).

The automatic choice is on the gradient 3-bit tonality (https://theory.uchicago.edu/~ejm/pix/20d/tests/noise/noise-p3.html) border (any lower bit depth would result in obvious posterization).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: squig on April 27, 2017, 03:53:49 AM
Can someone write tutorial about how to rec on these modes?

You're reading it; page 23.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 27, 2017, 07:56:48 AM
WOW........ @a1ex you did it.  I just tried the new build, it recorded @ 3.5K 12bit Lossless 2.35:1 continuously.. Zero Hick-ups so far.  Converted the images with no problem they came out beautiful  :D
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 27, 2017, 08:00:40 AM
Have you actually read the change log? There were *no* changes on the camera side since latest build; only mlv_dump was updated.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 27, 2017, 09:10:01 AM
Not really was too excited to try it out  :D
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: jimiz on April 27, 2017, 07:13:20 PM
What brand CF cards are you using @jimiz? Are they formatted into exFAT at the very least?

Also were you using '3.5k 1:1 center x5' within crop_rec.mo from ML menu and more importantly did you enabled FPS override to 23.976 while in x5 zoom?

Here's a screenshot from MLVFS webGUI to show that I was able to achieve continuous w 12-bit lossless in both 3072x1308 & 3072x1320 in 24p @ ISO 3200 until pressed stop.

(https://c1.staticflickr.com/3/2809/33444364344_551229871d_c.jpg) (https://flic.kr/p/SXn7ks)

This was used w KomputerBay 128GB 1066x and it was formatted to exFAT via Disk Utility on OS X prior to recording (256GB cards can be formatted in cam while keeping the exFAT).

Settings used under Movie mode within ML:

(https://c1.staticflickr.com/3/2811/33902403100_12478ee69e.jpg) (https://flic.kr/p/TDQFio) (https://c1.staticflickr.com/5/4185/33902403380_4bf81ce3fd.jpg) (https://flic.kr/p/TDQFod)


Thanls for your really important info about! 
Cf is a  CB 128 GB x1066 tested ,formated exFAT, and seem to work very well  with normal raw 14 bit 1920 25p + audio , no stops.
In the setting you talk I become automatic stops , sometime with few beeps and error....BUT I think is some problem her of many another people....
also if you read  the setting you do, work on continued recording.... iso and  LIGHT of sky example...give a big input data that overwriting buffer and STOP the recording.
With 1920x960 50 P  14, 12 LOSS bit....begine well and suddenly break the recoding, why ? I just do few test before....if I begin recordin in the dark area/room ALL ok....when I turn to the windows light....BREAK recording, so  also if was around 60/70 Mb...the light jump over 100....  this do in all the LOSS bit.

i think this is the bottleneck which does not hold recording on some resolution, though not exaggerated and low frames.  or not?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: vstrglv on April 27, 2017, 08:53:06 PM
Not after seeing samples like (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183187#msg183187) these (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183208#msg183208), but feel free to prove me wrong (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183231#msg183231).

The automatic choice is on the gradient 3-bit tonality (https://theory.uchicago.edu/~ejm/pix/20d/tests/noise/noise-p3.html) border (any lower bit depth would result in obvious posterization).
Yes, I understand. But this build is experimental one. A person who is going to use it must understand or try to understand what's going on. But it's not easy to set a proper  exposure and sometimes the exposure is set with some shift intentionally. In these cases  the relation between ISO and bit depth is not strictly defined. So manual bit depth option will be usefull.
BTW, is there a probability to get 12-8bit lossless for other modes, espesially 45/50/60fps?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 27, 2017, 10:04:39 PM
Quote
In these cases  the relation between ISO and bit depth is not strictly defined. So manual bit depth option will be usefull.

Please show me one (1) non-trivial example where reducing the bit depth below the recommended value is useful.

Quote
BTW, is there a probability to get 12-8bit lossless for other modes, espesially 45/50/60fps?

Probability to get lower bit depths in other modes is zero with current method (I'm not aware of any situation when it could work by chance, no matter how many times you would try).

If you ask about future possibilities, I'm unable to answer (because it's something that, at the moment of writing, is beyond my understanding). Whether I'll be able to figure it out tomorrow, or next month, or within the next 10 years, I have absolutely no idea. Same for real-time previews with 4K or other similar questions.

If you ask whether the hardware can do this, I'm pretty sure it can. What I don't know is how to program it to do so. You (http://www.magiclantern.fm/forum/index.php?topic=14656.0) can (http://magiclantern.wikia.com/wiki/Register_Map/Brute_Force) help (http://www.magiclantern.fm/forum/index.php?topic=2864.125) with (http://www.magiclantern.fm/forum/index.php?topic=6751.msg71720#msg71720) that (http://www.magiclantern.fm/forum/index.php?topic=10111.0).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: vstrglv on April 27, 2017, 10:25:41 PM
Please show me one (1) non-trivial example where reducing the bit depth below the recommended value is useful.
Only for reducing data rate a little. But what is the problem with manual bit rate?

If you ask whether the hardware can do this, I'm pretty sure it can. What I don't know is how to program it to do so. You (http://www.magiclantern.fm/forum/index.php?topic=14656.0) can (http://magiclantern.wikia.com/wiki/Register_Map/Brute_Force) help (http://www.magiclantern.fm/forum/index.php?topic=2864.125) with (http://www.magiclantern.fm/forum/index.php?topic=6751.msg71720#msg71720) that (http://www.magiclantern.fm/forum/index.php?topic=10111.0).
Unfortunately I can not help, not enough knowledge...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 27, 2017, 10:30:12 PM
But what is the problem with manual bit rate?

Asked and answered. Have you actually clicked on my links?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: vstrglv on April 27, 2017, 10:44:52 PM
Yes.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: g3gg0 on April 27, 2017, 10:51:39 PM
@vstrglv:
please find the requested download here:  [this feature is only available for members having a gold subscription]
if that doesn't work, and the arguments of the developers isn't reason enough: feel free to implement it
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 27, 2017, 10:52:34 PM
So @a1ex April 27th Build has no new changes from the April 26th version.. Like before just mlv_dump changed. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: erikbaldwinson on April 27, 2017, 11:25:46 PM
Hey, installed latest build on 5D3 today as I was really hoping to record 24fps 1080p 14bit lossless on the CF and proxy files on the SD.

Unfortunately, the proxy files will only record for about 6 seconds or so before live-view locks and the battery needs to be pulled. Recording both the 14bitlossless and proxy on the CF card results in the same/similar result.

With the latest crop mode build recording 10bit 1080p on CF and proxy on SD works great!

Anychance I'm missing something or there is a quick fix for this? Cheers
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 27, 2017, 11:44:01 PM
Right, I had two commits in the raw-h264-branch that I thought were included, but looks like they were not (I've never actually tried this feature on this branch). One (https://bitbucket.org/hudson/magic-lantern/commits/d0b78dfc82) is a workaround to prevent hard lock-ups when the H.264 stream stops, and the other (https://bitbucket.org/hudson/magic-lantern/commits/9465a180dfe4) attempts to sync the file names.

Last time I've checked (a long time ago), I had good results with H.264 IPB on SD, but recording stopped very quickly with both on CF (don't remember much about ALL-I).

Build updated.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: erikbaldwinson on April 27, 2017, 11:53:13 PM
Thank you Alex!

Quick question, are there any real strong incentives to record 10bit raw instead of 14bit lossless besides possibly better stability with crop mode build?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 27, 2017, 11:58:52 PM
Currently, the only reason I could think of would be in-camera playback.

I'd actually recommend lossless compression with 12-bit (high quality, as the last 2 bits are mostly noise (https://theory.uchicago.edu/~ejm/pix/20d/tests/noise/noise-p3.html)) or less (to be tested). Keep in mind I'm not an active video user, so the recommendation is based on theory, not on practice.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: erikbaldwinson on April 28, 2017, 12:30:27 AM
Thank you for the recommendation and explanation why!

In theory and as you understand it, is there a possibility that the last 2 bits provide some type of head-room for difficult shots? Or would you say 99% not likely..
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 28, 2017, 12:38:30 AM
You can judge from these samples: http://www.eoshd.com/comments/topic/21004-12-or-10-bit-raw-magic-lantern/?page=3#comment-168058
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Janosch on April 28, 2017, 01:43:12 AM
At the very first: ML is very great stuff and is a great ingenious work! Its an overwhelming thing, can't say thanks much enough! It makes a lot of fun playing and working with it!

Right now I'm playing with ML on the high res end.

I confirm that 3.5K works continously at 2.67:1; 12bit lossless

WOW........ @a1ex you did it.  I just tried the new build, it recorded @ 3.5K 12bit Lossless 2.35:1 continuously.. Zero Hick-ups so far.  Converted the images with no problem they came out beautiful  :D

@goldenchild: are you sure that you managed real 2.35:1 with 3.5K?
3.5K @ 2.35:1 (3520x1498) only 14bit lossless is supported with this vertical resolution and here you end up at >105MB/s at 50% compression --> not continuous
Also 1320pixel is max vertical resolution for the 5x crop mode (3.5K 1:1 centered x5 as well as 5x live view crop) and this is currently the max. vert res for 12bit lossless and lower

By the way I tested 3.5K 2.39:1 at UHD crop mode to get a vert res >1320 but at 14bitlossless. Depending on the brightness of the image I manged to get 120 to nearly 200 frames in this resolution.
Here is a quick test:

https://1drv.ms/i/s!ArsIFng0eZ9dgYhqj9iycEK3sGsFSQ

3520x1472; 14bitlossless; 23.976; UHD crop mode --> ~105-110MB/s

ISO 400 --> 120 frames
ISO 320 --> 150 frames
ISO 200 --> 186 frames

as mentioned above 3520x1320 works continuous due to 12bit lossless capability, but is 2.67:1...

By the way there is a small bug:
when accidentally entering the UHD crop mode from 3.5K (where 5x lv zoom + fps override to 23.976 needs to be enabled) at a resolution of 3520+ the FPS override changes to an odd 33.. fps speed. When turning fpsoverride off it changes to 19.. fps. While turning off, the Picture Style window from Canon menu flashes up once.
To eliminate this you need to turn off crop mode und raw video, close ml menu and turn everything on again (without x5 live view zoom + fps or!)... Nothing substantial but a bit confusing.
Can anyone confirm this?

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: erikbaldwinson on April 28, 2017, 03:27:11 AM
Right, I had two commits in the raw-h264-branch that I thought were included, but looks like they were not (I've never actually tried this feature on this branch). One (https://bitbucket.org/hudson/magic-lantern/commits/d0b78dfc82) is a workaround to prevent hard lock-ups when the H.264 stream stops, and the other (https://bitbucket.org/hudson/magic-lantern/commits/9465a180dfe4) attempts to sync the file names.

Last time I've checked (a long time ago), I had good results with H.264 IPB on SD, but recording stopped very quickly with both on CF (don't remember much about ALL-I).

Build updated.

Hey Alex, unfortunately those changes you made to the latest build (2017-04-27 21:51) didn't do it. It still only records 6 seconds ALL-I or IPB before I get an "emergency stop" notification on the live view. Tried 12bit lossless, 12bit raw, and 10bit raw, makes no difference. I made sure to set record to card 2 (SD) in the canon menu each time.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 28, 2017, 02:34:00 PM
@Janosch Yep got 3.5K 12bit Lossless 2.35:1 to work continuously.  Doing more testing but so far it's been stable and continuous.   
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hyalinejim on April 28, 2017, 03:57:18 PM
How are you getting higher than 1320 vertical resolution to get 2.35 aspect ratio at 3.5k?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: jankrueck on April 28, 2017, 05:02:17 PM
Quote
By the way there is a small bug:
when accidentally entering the UHD crop mode from 3.5K (where 5x lv zoom + fps override to 23.976 needs to be enabled) at a resolution of 3520+ the FPS override changes to an odd 33.. fps speed. When turning fpsoverride off it changes to 19.. fps. While turning off, the Picture Style window from Canon menu flashes up once.
To eliminate this you need to turn off crop mode und raw video, close ml menu and turn everything on again (without x5 live view zoom + fps or!)... Nothing substantial but a bit confusing.
Can anyone confirm this?

can confirm!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 28, 2017, 05:18:38 PM
By the way there is a small bug:
when accidentally entering the UHD crop mode from 3.5K (where 5x lv zoom + fps override to 23.976 needs to be enabled) at a resolution of 3520+ the FPS override changes to an odd 33.. fps speed. When turning fpsoverride off it changes to 19.. fps. While turning off, the Picture Style window from Canon menu flashes up once.
To eliminate this you need to turn off crop mode und raw video, close ml menu and turn everything on again (without x5 live view zoom + fps or!)... Nothing substantial but a bit confusing.
Can anyone confirm this?

Trying to reproduce this on my end to no avail. Was this done on 113 or 123?

can confirm!

Really? ...How?

How are you getting higher than 1320 vertical resolution to get 2.35 aspect ratio at 3.5k?

Hmmm care to explain this @goldenchild9to5 or maybe you assumed incorrectly? Either way would you mind to elaborate a bit more or better yet share some samples?

Because atm it just stays within 2.65:1 even if you selected '2.35:1' from the ML menu:

(https://c1.staticflickr.com/5/4182/34322239625_b36e100648.jpg) (https://flic.kr/p/UhWs5a)

and yet it isn't continuous even if w FPS override enabled down to 20p.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: vstrglv on April 28, 2017, 06:40:46 PM
But how did you get 3504x1320? I can get only:
3072x1320 (~2,33:1)
3520X1320 (~2,67:1)
3584X1320 (~2,72:1)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 28, 2017, 06:54:00 PM
@vstrglv -- perhaps you didn't press magnified zoom to get into x5 mode (which is required for this mode - read the notes on bottom of LCD LiveView) while having '3.5k 1:1 centered x5' enabled from crop-mode within ML menu and then select 3520x1320 in 2.50:1 w 12-bit lossless.

(https://c1.staticflickr.com/5/4156/34165732962_ff94fa00a8_n.jpg) (https://flic.kr/p/U47j6q) (https://c1.staticflickr.com/5/4186/34165732722_57361246f1_n.jpg) (https://flic.kr/p/U47j2h) (https://c1.staticflickr.com/3/2816/34165914172_d653796838_n.jpg) (https://flic.kr/p/U48eXJ)

Finally, while highlighted on the 'Resolution 3520x1320' use your shutter scroller (top right) to fine-tune it down 3504x1320 which is 2.65:1.




Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: vstrglv on April 28, 2017, 07:09:38 PM
@DeafEyeJedi Thank you very much! I forgot about shutter scroller...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pocketrubbish on April 28, 2017, 07:17:25 PM
I'm liking the latest build, full res liveview is getting pretty darn reliable at 7.5fps, 2.35 crop. Almost continuous recording, I had a few shut off after 5 seconds. I drove up into the mountains to see how some of my older lenses handle 4k+ and made a short video.


I did notice that it would shut off right away if I was shooting right into the sun at ISO100 but I kind of expected that after reading through the thread. It's getting damn good though.

Amazing work so far!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 28, 2017, 07:25:21 PM
Hey Alex, unfortunately those changes you made to the latest build (2017-04-27 21:51) didn't do it. It still only records 6 seconds ALL-I or IPB before I get an "emergency stop" notification on the live view. Tried 12bit lossless, 12bit raw, and 10bit raw, makes no difference. I made sure to set record to card 2 (SD) in the canon menu each time.

Confirmed - it was broken by an experimental change in this branch - the way we allocate memory for the main raw buffer (during standby) conflicts with H.264. I've assumed Canon's memory layout does not change as long as you are not leaving LiveView, but it looks like it does (so our raw stream ended up overwriting the memory areas used by H.264). Looking into it.


I'm liking the latest build, full res liveview is getting pretty darn reliable at 7.5fps, 2.35 crop. Almost continuous recording, I had a few shut off after 5 seconds.

Nice to see this mode working! You should be able to reduce the memory workload (and hopefully free some resources to make them available to the card writing task) with some tricks:
- dial down the resolution in crop_rec submenu (target yres); note the FPS will increase
- re-center the image using CMOS[1] hi and lo (trial and error)
- enable FPS override in low light mode and dial it down to 7.5 fps or whatever you need
- try the "Frozen LV" preview mode for a little more writing speed.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pocketrubbish on April 28, 2017, 07:31:05 PM
Nice to see this mode working! You should be able to reduce the memory workload (and hopefully free some resources to make them available to the card writing task) with some tricks:
- dial down the resolution in crop_rec submenu (target yres); note the FPS will increase
- re-center the image using CMOS[1] hi and lo (trial and error)
- enable FPS override in low light mode and dial it down to 7.5 fps or whatever you need
- try the "Frozen LV" preview mode for a little more writing speed.

Thank you, I'll try all of that this weekend.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Janosch on April 28, 2017, 09:31:03 PM
Trying to reproduce this on my end to no avail. Was this done on 113 or 123?

113!
you may encounter it if you make 3.5K (w/ 5x zoom + fps override to 23.976 enabled) working and switch as described to the UHD crop mode...


(https://c1.staticflickr.com/5/4182/34322239625_b36e100648.jpg) (https://flic.kr/p/UhWs5a)

and yet it isn't continuous even if w FPS override enabled down to 20p.

it seems there is a problem with your cf card as 49.3 MB/s is too slow. 77.1MB/s should be no problem for continuous recording.
Maybe try to run a new cf card benchmark...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 29, 2017, 12:09:14 AM
So I decided to test the 5k mode at 6fps 10bit (not lossless) 2.35:1 and ..... Continuous. This is going to be wild for timelapsing!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: beauchampy on April 29, 2017, 02:05:21 AM
So I decided to test the 5k mode at 6fps 10bit (not lossless) 2.35:1 and ..... Continuous. This is going to be wild for timelapsing!

Just curious. Why would you choose to shoot a timelapse this way compared to shooting raw stills using the intervalometer? I don't shoot a lot of timelapse, so genuinely curious.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 29, 2017, 03:21:13 AM
@beauchampy No shutter actuations and a slightly nicer workflow IMO. The only issue would be battery life for really long lapses. The Mark iv shuts off the LCD.  But for 1fps or maybe .5 fps it would be awesome. Havent tried it yet.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: budafilms on April 29, 2017, 07:07:35 AM
Wich continous maximum resolution can I use for my new full length movie?

(I promess next Cannes festival take a picture in the end of the movie with the logo of ML)

 ;)

Off topic: I dreamed recording 4k watching by the optical view, not the viewfinder  :D
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 29, 2017, 04:36:37 PM
Hey Alex, unfortunately those changes you made to the latest build (2017-04-27 21:51) didn't do it. It still only records 6 seconds ALL-I or IPB before I get an "emergency stop" notification on the live view.

Solved. The changes were non-trivial, but since it was a memory management issue, it got higher priority.

At 1920x1080 24p 12-bit lossless, and a bit of luck (because of the variable bitrate) recording can be continuous even with both RAW and H.264 on the same card. With H.264 on SD and RAW on CF, both recorders have *much* more headroom.

Careful when the card gets full, as this event is not handled very well (and it requires more reverse engineering to figure it out).

preview is slow and gray  , recording number  stay green....(think normal) but after 10-15 seconds ...STOP !

Solved that too (although I wish I had a proper bug report). The real-time estimations (and the indicator color) were completely messed up in lossless recording modes, and just OK-ish in uncompressed mode (with estimations being far away from actual recording times).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 29, 2017, 06:11:25 PM
@budafilms I would say to stick with 3K..
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 29, 2017, 06:53:30 PM
@a1ex the change log for 29th build looks very nice!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 29, 2017, 08:05:48 PM
@a1ex Thank you for the new build, will give it a spin today. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pocketrubbish on April 29, 2017, 08:45:03 PM
Wow, the latest build is very stable and much more fool proof! Got 48p 1080 working flawlessly and the full res live view puts up no fuss at all. I really love the total frame count at the end, very useful.

Now I'm trying to decide if I should drive out somewhere this evening and do another test... maybe I'll go to the beach.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 30, 2017, 12:30:21 AM
I've cherry-picked some changes from this branch, to be included in the main builds (for which I need some help with code review). First round:

https://bitbucket.org/hudson/magic-lantern/pull-requests/827
https://bitbucket.org/hudson/magic-lantern/pull-requests/828
https://bitbucket.org/hudson/magic-lantern/pull-requests/819 (last few commits)
https://bitbucket.org/hudson/magic-lantern/pull-requests/825 (older, but no reviews yet)

Next on the list:
- compressed_raw (without all the changes required for 4K, which are mostly 5D3-specific), so you can start porting it on other cameras
- compressed_raw with lower bit depth (I'd like to keep them separate, so you can focus on plain 14-bit lossless and then move on to more complicated stuff)
- H.264 proxy (there are few changes, but currently entangled in the middle of other 4K experiments)

If you are wondering why: this branch became pretty complex (includes many changes that are not exactly related) and many of these things are not portable to other cameras, or just highly experimental. Splitting it into small logical chunks should make it a bit more manageable.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 30, 2017, 02:13:56 AM
Awesome @a1ex
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: giarcpnw on April 30, 2017, 02:18:32 AM
https://youtu.be/ZdvKALy58Es

Bunch of 3520x1320 12bit shot last night and this morning. Sadly, youtube compresses the shit out of it but, man, you should see it in prores at full size. Glorious!!

Slightly graded in AE with VisionColor Osiris LUTs.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: lostfeliz on April 30, 2017, 02:46:35 AM
Looks great, giarcpnw. The shot of your dog looks out of a feature film. I was just at that spot yesterday. Then you freaked me out because I also have the same hummingbird feeder and hedge!  :D
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: giarcpnw on April 30, 2017, 07:14:38 AM
Looks great, giarcpnw. The shot of your dog looks out of a feature film. I was just at that spot yesterday. Then you freaked me out because I also have the same hummingbird feeder and hedge!  :D

Doesn't every Angelino have an ivy hedge and HB feeder? ;-) 

Thanks, each shot went well over 30 seconds, I just didn't want to bore everyone. I'll push it further another day but it seem pretty stable for all intents and purposes. Pretty amazing @a1ex!

Now, to shoot something with it!!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: lostfeliz on April 30, 2017, 08:00:53 AM
Because the preview in 3k and above is so laggy, I hadn't thought of plugging in my monitor to it. But tonight I tried it out. The field of view is about 40% of what's on camera screen and being recorded. But it's great for focusing and no lag. If you can get used to operating with two monitors (one for composition and one for focus), you can really shoot a 3k project with a 5diii. Hell, if you had a focus puller you'd be in even better shape.

The HDMI monitor output (when mirroring, at least) is completely centered. Short of being the correct frame without jaggy lag, I'm wondering if the HDMI mirrored output could be panned as desired to frame what you want to see to keep in focus.

The cropped panning control knob does something when you use it but the screen then pop back to how they were.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: rob_6 on April 30, 2017, 08:11:35 AM
Because the preview in 3k and above is so laggy, I hadn't thought of plugging in my monitor to it. But tonight I tried it out. The field of view is about 40% of what's on camera screen and being recorded. But it's great for focusing and no lag. If you can get used to operating with two monitors (one for composition and one for focus), you can really shoot a 3k project with a 5diii. Hell, if you hades  a focus puller you'd be in even better shape.

The HDMI monitor output (when mirroring, at least) is completely centered. Short of being the correct frame without jaggy lag, I'm wondering if the HDMI mirrored output could be panned as desired to frame what you want to see to keep in focus.

The cropped panning control knob does something when you use it but the screen then pop back to how they were.

Thanks for posting this info! Just curious. Are you using 1.2.3 firmware or 1.1.3? Thanks!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: lostfeliz on April 30, 2017, 08:55:19 AM
 I was doing exposure tests and the exposure lowered dramatically when I switched from 14-bit lossless to 12-bit lossless. Settings: Crop 3.5, 3072x1320. Is this to be expected?

Rob, I'm running 1.2.3.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 30, 2017, 09:25:18 AM
The exposure stays the same; only the real-time preview gets darker. That's how it works. The non-realtime preview should be correct.

Refer to the initial commit (https://bitbucket.org/hudson/magic-lantern/commits/2028d73) and the subsequent ones.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: rudimental on April 30, 2017, 06:58:11 PM
Hello :)
I wanted to ask, if, with all the new development and discoveries you guys made lately, there is a way to improve/shorten rolling shutter ms for normal 1080p recording.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: baksh on April 30, 2017, 09:58:39 PM
Hello everyone!

Just a quick video shot in Roma last week to show that everything just works perfectly !
Thanks to the team !

For informations
//4k Settings //
- Resolution : 3072x1024 px
- Bits : 14 bpp Lossless
- Aspect Ratio : 3:1

I was able to shoot continuously.

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on May 01, 2017, 12:00:18 AM
Yes, I did some continuous flawless 3K shots. Incredibly awesome!
The 1st of April is ML special day.
I hope MLV_Sound returns some time and works in hand with video.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on May 01, 2017, 02:37:01 AM
Wow!!!!!! 29th build!!!!!!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on May 01, 2017, 02:51:46 AM
@baksh Great job.. awesome footage.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on May 01, 2017, 03:18:45 AM
Hello :)
I wanted to ask, if, with all the new development and discoveries you guys made lately, there is a way to improve/shorten rolling shutter ms for normal 1080p recording.

Already done (http://www.magiclantern.fm/forum/index.php?topic=12656.0) a long time ago (actually since 2012 (https://groups.google.com/d/msg/ml-devel/tSg5v99I7lQ/uL-XTHQ8fBQJ); we just didn't know what we had done, back then).

It might be possible to push it a little lower. With adtg_gui and enough patience, you can probably figure it out. All you have to do is to reduce FPS timer A as much as you can, but you'll also have to reconfigure some other parameters to allow this (don't know which ones - that's what you have to find out).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hindra on May 01, 2017, 03:50:21 AM
Sorry guys, with the new MLV_Dump the DNG file sizes are coming out much larger, for 3k 12 bit they are 7mb each. I am using this script to run MLV_Dump, please help me make corrections. Also, it seems to be taking much longer as well.

Quote
@echo off
for %%a in ( %* ) do ( md "%%~na" 2>nul )&( C:\MLVFS\mlv_dump.exe -o %%~na\%%~na_ --dng %%~na.MLV )

EDIT: MLV files are 16.6gb, dng output are closer to 30gb.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Levas on May 01, 2017, 10:23:28 AM
If you used compressed raw, the MLV files are always smaller. The dng files that are made by MLV_dump are not compressed, so bigger.
And as far as I know, the output in dng from MLV_dump is always in 14 bit.
So you could have a compressed 12 bit MLV file, but the dng's that come out mlv_dump are uncompressed 14 bit.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hindra on May 01, 2017, 02:54:18 PM
If you used compressed raw, the MLV files are always smaller. The dng files that are made by MLV_dump are not compressed, so bigger.
And as far as I know, the output in dng from MLV_dump is always in 14 bit.
So you could have a compressed 12 bit MLV file, but the dng's that come out mlv_dump are uncompressed 14 bit.

That makes perfect sense! Thank you for clearing that up for me.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DLM on May 01, 2017, 03:25:50 PM
Hello!

I installed the firmware version magiclantern-crop_rec_4k.2017Apr26.5D3113. I shoot 1920x1080 30p 14bit losless, it works fine, but when I open the video on the computer, in all the videos I see on the left a black bar about two pixels wide. What is the problem?

(http://thumb.ibb.co/mJ0SVk/M01_0745_000008.jpg) (http://ibb.co/mJ0SVk)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: jankrueck on May 01, 2017, 04:44:32 PM
kinda offtopic. just wanted to show some testfootage.
just wanted to thank Alex and team for their work!

Can't wait for final release and having sound recorded. keep it up!


3k 1:1 | 3072x1320 | 14bit lostless
Sandisk Pro 160mb/s 64GB card
ISO100-200 continous

3840x2160 (1650 w/o letterbox) upscaling, sharpening and some lightadjustments done in resolve 12.5.
very rough and fast, noch "real" grading or so.

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: g3gg0 on May 02, 2017, 01:34:03 AM
Sorry guys, with the new MLV_Dump the DNG file sizes are coming out much larger, for 3k 12 bit they are 7mb each. I am using this script to run MLV_Dump, please help me make corrections. Also, it seems to be taking much longer as well.

EDIT: MLV files are 16.6gb, dng output are closer to 30gb.

you now can choose between
a) output uncompressed DNGs
b) compress DNG content (bit slower bit same as uncompressed, just many tools have trouble with lossless compressed DNG)
c) pass through the original lossless data (which will be fast but skip all processing like stripe and hot pixel detection)

this can be done by adding "-c" multiple times to the commandline.

e.g.
Code: [Select]
./mlv_dump in.mlv --dng   # outputs uncompressed DNGs
./mlv_dump in.mlv --dng -c  # outputs compressed DNGs using mlv_dump's LJ92 algorithm
./mlv_dump in.mlv --dng -c -c  # outputs uncompressed DNGs by passing through unmodified lossless data
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hindra on May 02, 2017, 05:09:25 AM
you now can choose between
a) output uncompressed DNGs
b) compress DNG content (bit slower bit same as uncompressed, just many tools have trouble with lossless compressed DNG)
c) pass through the original lossless data (which will be fast but skip all processing like stripe and hot pixel detection)

this can be done by adding "-c" multiple times to the commandline.

e.g.
Code: [Select]
./mlv_dump in.mlv --dng   # outputs uncompressed DNGs
./mlv_dump in.mlv --dng -c  # outputs compressed DNGs using mlv_dump's LJ92 algorithm
./mlv_dump in.mlv --dng -c -c  # outputs uncompressed DNGs by passing through unmodified lossless data

Thank you for that. It is nice having options.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on May 02, 2017, 05:24:12 AM
@jankrueck Great job footage looks awesome.. You also captured great dynamic range, windows are well exposed & still having enough dynamic range for the interior superb. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on May 02, 2017, 09:35:00 AM
g3gg0, thanks!
Here´s latest mlv_dump for mac which beautifully applies lossless compression to dng files!
https://bitbucket.org/Dannephoto/magic-lantern/downloads/mlv_dump_compress_dng.zip

try:
Code: [Select]
mlv_dump --dng -c INPUT.MLV
And watch dng files process lossles with less than half it´s size. This works with uncompressed footage as well.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: bouncyball on May 02, 2017, 10:43:09 AM
@g3gg0

you now can choose between
a) output uncompressed DNGs
b) compress DNG content (bit slower bit same as uncompressed, just many tools have trouble with lossless compressed DNG)
c) pass through the original lossless data (which will be fast but skip all processing like stripe and hot pixel detection)
Haha, I was doing the same pass-trough stuff yesterday but you really did it a lot better (and faster) than me!!! :D plus there are also some more fixes/changes. Thank you so much!

There is a small question. When -c not specifyed e.g. 'mlv_dump -o result.mlv source.mlv', the standard output is:
Code: [Select]
Processing...
  saving: 3628800 -> 1906112  (52.53% ratio)
  saving: 3628800 -> 1906112  (52.53% ratio)
  saving: 3628800 -> 1905600  (52.51% ratio)

In case of "!verbose":
Code: [Select]
if(frame_buffer_size != (uint32_t)frame_size && !verbose)
{
   print_msg(MSG_INFO, "  saving: "FMT_SIZE" -> "FMT_SIZE"  (%2.2f%% ratio)\n", frame_size, frame_buffer_size, ((float)frame_buffer_size * 100.0f) / (float)frame_size);
}
Do we really need this info (calculated uncompressed size to already hw compressed size comparizon) during MLV->MLV pass-trough? Maybe make it somewhat optional.

bb
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: beauchampy on May 02, 2017, 10:46:24 AM
g3gg0, thanks!
Here´s latest mlv_dump for mac which beautifully applies lossless compression to dng files!
https://bitbucket.org/Dannephoto/magic-lantern/downloads/mlv_dump_compress_dng.zip

try:
Code: [Select]
mlv_dump --dng -c INPUT.MLV
And watch dng files process lossles with less than half it´s size. This works with uncompressed footage as well.

Fantastic! Look forward to seeing this with cr2hdr!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: bouncyball on May 02, 2017, 10:48:05 AM
try:
Code: [Select]
mlv_dump --dng -c INPUT.MLV
And watch dng files process lossles with less than half it´s size. This works with uncompressed footage as well.
Produced (processed and recompressed) DNGs are even a bit smaller than passed-trough ones :) I guess also becase of there is no padding bytes.

bb
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: jankrueck on May 02, 2017, 01:14:22 PM
@jankrueck Great job footage looks awesome.. You also captured great dynamic range, windows are well exposed & still having enough dynamic range for the interior superb.

thanks!  my maingoal was to test around with DR, detail and cameramoving, as most of the test I saw where handheld or tripod landscape.
the Dynamic Range is just insane. Its out of Camera. Didnt lift shadows or saved highlights at all.


cherrio!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: jimiz on May 02, 2017, 03:03:53 PM
* 1920x960 @ 50p (both 1:1 crop and full-frame - 3x3 pixel binning) last exp.4k 29 april



crash (freeze) on ML menu options, in few seconds when push upd and down select.... and must put out battery ....

any one confirm ?

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on May 02, 2017, 04:03:56 PM
@jimiz
For me that mode is no more continuous recording with the last build, I don't know why... I return to April14th, the most reliable for me... :-[
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on May 02, 2017, 04:53:59 PM
If you want the future builds to improve...

http://www.chiark.greenend.org.uk/~sgtatham/bugs.html

Before reporting any differences in recording times, please make sure they are not caused by small variations in the data rate (unavoidable with lossless compression) or by different settings (e.g. global draw, preview options).

I'm currently benchmarking the two builds with this method (http://www.magiclantern.fm/forum/index.php?topic=17091) (this script (http://www.magiclantern.fm/forum/index.php?topic=17091.msg165766#msg165766)).

Just FYI, it's not the first time people are reporting speed differences between nearly identical builds (http://www.magiclantern.fm/forum/index.php?topic=17091.msg165688#msg165688) (where the only change was something like fixing a typo in XYZ language (https://bitbucket.org/hudson/magic-lantern/commits/2a1f972cdbd87e77059539622fd4308618fa9ae6)).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: jimiz on May 02, 2017, 06:35:29 PM
@jimiz
For me that mode is no more continuous recording with the last build, I don't know why... I return to April14th, the most reliable for me... :-[

yes to me too , no possibile to rec. in 1920x920 50p  , around 20 seconds not more...little bit better in x800....but not continued.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on May 02, 2017, 06:51:24 PM
With the settings recommended in the first post, 1920x960 50p is continuous here (orange icon at first, green after ~30 seconds).

Benchmark results tomorrow (will let the script test the two builds overnight).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: g3gg0 on May 02, 2017, 07:50:33 PM
Do we really need this info (calculated uncompressed size to already hw compressed size comparizon) during MLV->MLV pass-trough? Maybe make it somewhat optional.

no, good point.
reworked a lot of stuff (so errors may appear that weren't present before).
this is one thing. got to fix that, right.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: jimiz on May 02, 2017, 08:11:42 PM
With the settings recommended in the first post, 1920x960 50p is continuous here (orange icon at first, green after ~30 seconds).

Benchmark results tomorrow (will let the script test the two builds overnight).

Thanks A1ex

to me and a friend (we have same CF   CBx1066 tested)    with 27 and 29 april , vers. 123  ,   orange icon at first 20 seconds circa  red after and stop!  no way ...tryed 20 times...
if we set 1920x800  orange icon at first 10-15 seconds...green , orange, green.... but seem to be continued...(i try more of 2-3 minutes without stop)

so I don't know if 1920x960 50p was working better in old version...this I can't say....

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on May 02, 2017, 08:30:05 PM
to me and a friend (we have same CF   CBx1066 tested)    with 27 and 29 april , vers. 123  ,   orange icon at first 20 seconds circa  red after and stop!  no way ...tryed 20 times...

Screenshot of the raw recording menu, please.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on May 02, 2017, 10:14:47 PM
I think is something related to compression and how both builds manage it. With April14th I get green icon from start to end of recording, no matter what image I'm recording. With last buid April29th, I get green icon if I'm recording a dark image with no detail, and when I point the camera towards a detailed and well iluminated image, icon changes to orange and quickly to red. Then stops.
But I can be wrong... It's just an assumption from my ignorance.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DLM on May 02, 2017, 10:47:28 PM
Hello!

Does anyone else besides me have a black bar two pixels wide on the left side of the frame on all the videos?  Firmware version magiclantern-crop_rec_4k.2017Apr26.5D3113. I shoot 1920x1080 30p 14bit losless.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on May 03, 2017, 05:08:16 AM
@ DLM

Yes!
When I use Handbrake to convert some videos to MP4, Handbrake detects 2 pixels wide column and crops it.
e.g if I shoot at 3072, it shows 3070, 2 pixels offset
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on May 03, 2017, 10:14:19 AM
With April14th I get green icon from start to end of recording, no matter what image I'm recording.

... and stops with green icon when you do this:

Quote
With last buid April29th, I get green icon if I'm recording a dark image with no detail, and when I point the camera towards a detailed and well iluminated image, icon changes to orange and quickly to red. Then stops.

As mentioned here (http://www.magiclantern.fm/forum/index.php?topic=19300.msg184055#msg184055), I've fixed the indicator, so it will actually warn you when it's going to stop.

That said, looks like the benchmark has some significant results:

Code: [Select]
mlv_lite version : ML/MODULES/RAWBENCH/APR14.MO
 854,872,940,989,924,802,930,894
 767,783,869,856,944,872,909,941
 746,973,1004,840,912,986,1088,800
 660,714,798,875,939,982,830,946
 868,929,941,983,857,960,870,846
 910,1072,929,1016,877,890,1021,924
 840,929,951,840,894,928,928,874
 651,720,926,837,850,812,1095,1078
 840,1030,804,960,910,1017,971,963
 1034,1032,961,806,894,1018,872,836
 926,1011,1074,902,960,838,864,1123
 783,949,871,890,746,760,796,853

mlv_lite version : ML/MODULES/RAWBENCH/APR29.MO
 335,727,943,761,882,929,608,781
 338,754,852,554,572,876,750,1017
 608,683,592,689,765,844,713,652
 834,733,574,804,638,560,927,853
 423,747,775,943,745,745,588,797
 505,687,1003,703,786,583,838,799
 495,597,895,872,872,572,837,926
 771,636,702,891,761,802,833,979
 710,590,905,978,889,929,810,851
 652,837,727,726,742,696,707,763
 486,655,826,729,669,882,916,697

Quartile stats: median (Q1 ... Q3)
mlv_lite version : ML/MODULES/RAWBENCH/APR29.MO:  752 frames (654 ... 852)
mlv_lite version : ML/MODULES/RAWBENCH/APR14.MO:  906 frames (840 ... 960)

P =

           1  1.3323e-15
  1.3323e-15           1

T =

           0     -8.7668
      8.7668           0

Looks like there is, indeed, a small speed regression.

Each number from the "big matrix" shows how many frames were in each test clip, and a new line means the card was formatted (and possibly other build was tested meanwhile - they are chosen randomly after filling and formatting the card). 1920x960 50p, 14-bit lossless, dark scene with artificial light (not changing during the experiment), preview set to Framing (which is not continuous here).

So, I've added some intermediate versions to the test script to narrow it down. The scene was a bit different (moved the camera), so the numbers are not directly comparable to the first run (variable bit rate):
Code: [Select]
mlv_lite version : ML/MODULES/RAWBENCH/APR23.MO
 1764,1691,1623,1565,1180,1299,1481,1307
 1173,1284,1576,1439,1482,1578,1421,1330
 1306,1110,1670,1184,1368,1439,1228,1568
 922,1360,1603,1455,1208,1201,1528,1844
 3200,3181,3199,3093,3028,3112,3215,3149

mlv_lite version : ML/MODULES/RAWBENCH/E93203.MO
 827,1134,1099,1324,1399,1475,1515,1397
 675,1121,1270,1271,921,893,1126,1088

mlv_lite version : ML/MODULES/RAWBENCH/B420A8.MO
 1596,1296,1169,1322,1292,1475,1839,1443
 1068,1895,1441,1363,1327,1360,1805,1444
 1322,1516,1703,1672,1591,1311,1185,1558

mlv_lite version : ML/MODULES/RAWBENCH/APR27.MO
 965,1490,1459,1667,1441,1234,1315,1481
 1470,1644,1536,1536,1728,2007,1773,1777

mlv_lite version : ML/MODULES/RAWBENCH/3F48DF.MO
 1349,1783,1441,1282,1444,1288,1624,1650
 1083,1368,1543,1273,1498,1680,1615,1644
 1402,1498,1477,1384,1211,1781,1110,1460
 3153,3211,3245,3260,3223,3043,3027,3162
 945,1436,1577,1595,2027,1466,1576,1322

mlv_lite version : ML/MODULES/RAWBENCH/E67FAC.MO
 627,888,1233,825,710,644,1197,1098
 531,721,833,1026,1357,1083,908,1784
 544,1034,1056,1509,1380,1109,928,1157
 892,956,985,1057,1270,1214,1147,1099
 1084,1252,1087,746,901,1042,1025,970
 1005,1235,1249,1076,1224,1305,1256,1384
 889,967,909,1542,839,1152,517,1313
 672,935,1002,1345,1249,1339,1327,1593

mlv_lite version : ML/MODULES/RAWBENCH/APR21.MO
 1346,1688,1683,1588,2244,1988,2444,1745
 1154,1398,1545,1725,1510,1630,1555,1337
 3246,3107,3236,3153,3108,3189,3275,3145

mlv_lite version : ML/MODULES/RAWBENCH/APR14.MO
 678,1752,862,1520,1394,1357,1025,1085
 1047,1553,1034,1687,1284,1459,941,1520
 1526,1709,1421,1398,1444,1701,1766,1691
 1252,1777,1760,1637,1634,1483,1654,1406
 695,1481,1520,1433,1814,1492,1360,1339

mlv_lite version : ML/MODULES/RAWBENCH/APR29.MO
 895,1055,1099,1323,1362,1265,1261,1646
 3143,3092,3185,3055,3208,3243,3207,3206
 1066,1006,1230,1388,1078,777,1045,1198
 861,1208,964,1137,968,1297,1002,1457
 763,1032,1173,1192,1227,1242,1485,1456

Quartile stats: median (Q1 ... Q3)
mlv_lite version : ML/MODULES/RAWBENCH/E67FAC.MO: 1067 frames (905 ... 1249)
mlv_lite version : ML/MODULES/RAWBENCH/E93203.MO: 1130 frames (1005 ... 1361)
mlv_lite version : ML/MODULES/RAWBENCH/APR29.MO: 1229 frames (1050 ... 1471)
mlv_lite version : ML/MODULES/RAWBENCH/B420A8.MO: 1442 frames (1317 ... 1594)
mlv_lite version : ML/MODULES/RAWBENCH/APR14.MO: 1470 frames (1312 ... 1646)
mlv_lite version : ML/MODULES/RAWBENCH/APR23.MO: 1482 frames (1303 ... 1728)
mlv_lite version : ML/MODULES/RAWBENCH/APR27.MO: 1513 frames (1450 ... 1698)
mlv_lite version : ML/MODULES/RAWBENCH/3F48DF.MO: 1521 frames (1376 ... 1782)
mlv_lite version : ML/MODULES/RAWBENCH/APR21.MO: 1735 frames (1550 ... 3127)

P =

 Columns 1 through 5:

           1     0.20463  2.3393e-05  4.2457e-09  7.9956e-09
     0.20463           1    0.063283  0.00020993    0.003231
  2.3393e-05    0.063283           1     0.55111     0.29445
  4.2457e-09  0.00020993     0.55111           1     0.53456
  7.9956e-09    0.003231     0.29445     0.53456           1
  3.5275e-10   0.0021903     0.26451    0.053254   0.0069184
  8.3596e-09  0.00014102     0.88954     0.31542     0.16273
   2.648e-11  0.00093461     0.16911    0.025252   0.0021659
  4.4409e-16  1.3427e-05   0.0054545  8.9505e-05  7.4696e-07

 Columns 6 through 9:

  3.5275e-10  8.3596e-09   2.648e-11  5.2588e-16
   0.0021903  0.00014102  0.00093461  1.3427e-05
     0.26451     0.88954     0.16911   0.0054545
    0.053254     0.31542    0.025252  8.9505e-05
   0.0069184     0.16273   0.0021659  7.4696e-07
           1     0.23071      0.7823    0.038059
     0.23071           1     0.14709   0.0032367
      0.7823     0.14709           1    0.061734
    0.038059   0.0032367    0.061734           1

T =

           0     -1.2792     -4.4339     -6.5382     -6.2902     -6.9502     -6.4563     -7.4827     -9.9692
      1.2792           0     -1.8962     -4.0996     -3.0823     -3.2171     -4.3595     -3.5022     -4.9964
      4.4339      1.8962           0     0.59938      1.0555     -1.1239     0.13954     -1.3879     -2.8799
      6.5382      4.0996    -0.59938           0     0.62454     -1.9705     -1.0173      -2.293     -4.2946
      6.2902      3.0823     -1.0555    -0.62454           0     -2.7744     -1.4153     -3.1719     -5.5061
      6.9502      3.2171      1.1239      1.9705      2.7744           0      1.2122    -0.27728     -2.1195
      6.4563      4.3595    -0.13954      1.0173      1.4153     -1.2122           0      -1.471     -3.1431
      7.4827      3.5022      1.3879       2.293      3.1719     0.27728       1.471           0     -1.9026
      9.9692      4.9964      2.8799      4.2946      5.5061      2.1195      3.1431      1.9026           0

D =

         126          78         102          86         102         102          78         102          86
          78          30          54          38          54          54          30          54          38
         102          54          78          62          78          78          54          78          62
          86          38          62          46          62          62          38          62          46
         102          54          78          62          78          78          54          78          62
         102          54          78          62          78          78          54          78          62
          78          30          54          38          54          54          30          54          38
         102          54          78          62          78          78          54          78          62
          86          38          62          46          62          62          38          62          46

It looks like the first 3 builds (in the sorted list) have the regression; that means, it must have been introduced in the earliest changeset from this group (since the previous changeset was also tested): e67fac (https://bitbucket.org/hudson/magic-lantern/commits/e67facae488444a132aaa6107de0e3eb4e3129eb?at=crop_rec_4k).

That means:
- The regression only affects the behavior when recording is about to stop (how many frames it manages to squeeze in the last moments)
- It does not affect the ability to record continuously. This piece of code starts kicking in when the buffer is about 90% full.

Is the regression caused the overhead of printf (possibly delaying the decision by a non-negligible amount of time) or it's because of the change in thresholds (resulting in smaller blocks being written to the card)? Need to restart the experiment to find out.

To interpret the statistical tests, check Frank7D's posts (http://www.magiclantern.fm/forum/index.php?topic=17091.msg166137#msg166137) and the t_test_2 (https://octave.sourceforge.io/octave/function/t_test_2.html) function description.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: festr on May 03, 2017, 11:32:46 AM
I cannot find information about missing mlv sound recording missing in the experimental branch - why I cannot record sound ?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Walter Schulz on May 03, 2017, 11:44:59 AM
Experimental page -> Notes:
Quote
mlv_lite only for now

Features page -> Modules -> mlv_lite.mo -> (link) ->
Quote
There are some caveats to what you get compared to the full mlv_rec:
[...]
2. No audio
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: g3gg0 on May 03, 2017, 11:54:14 AM
to make mlv_snd work with mlv_lite, the buffering structure must be changed to allow other modules to queue blocks.
that are only a few callbacks, but this might have a negative feedback on write speed.

currently mlv_lite is meant to be a rewrite from scratch to get the maximum write speed.
when all experimental stuff settles, maybe this will be added.

for now just let things settle.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on May 03, 2017, 12:16:38 PM
@A1ex
Thank you very much for the explanation and for the tests!!!!!!!!!

I will continue testing.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Savely on May 03, 2017, 03:13:24 PM
dear devs, is it possible to make at least preview pictures of footages? I know now you try to make h264 proxy possible, but may it be more wise to make preview pictures option and then h264 proxy?

And please, is it possible to tag processed files in RawFlow not just frame001, frame002 etc. (nightmare for postproduction) but rather MLVfileName_001, MLVfileName_002 etc?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on May 03, 2017, 03:24:49 PM
Previews are there depending on your NLE. I see thumbnails on mac just fine.
You can create any name you want in mlv_dump.
Try this for instance:
Code: [Select]
mlv_dump --dng -o MLVfileName_0000_C0000_ INPUT.MLV
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: extremelypoorfilmaker on May 03, 2017, 06:24:28 PM
just following :)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: jankrueck on May 03, 2017, 07:55:12 PM
just following :)

still waiting for the day Alex will explode because of people asking dump questions, same questions again and again or both compared into 1...

 ;D :) ;)

ps: not yours.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on May 04, 2017, 05:30:51 AM
I LOVE mlv lite sooooo much! It's perfect. Sound does not matter to me. Just use a slate and a zoom and you'll be found. Much more like film!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Savely on May 04, 2017, 05:50:41 AM
Previews are there depending on your NLE. I see thumbnails on mac just fine.
You can create any name you want in mlv_dump.
Try this for instance:
Code: [Select]
mlv_dump --dng -o MLVfileName_0000_C0000_ INPUT.MLV

on previews: but what if I have Windows(8)?

on file names: It's in mlv_dump.exe? How should I tape this code if it's window dissapeared almost instantly?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on May 04, 2017, 06:15:29 AM
@Savely

In the directory where MLV_Dump.exe lies, make a .bat file and put this in it:

@echo off
for %%a in ( %* ) do ( md "%%~na" 2>nul )&( "mlv_dump.exe" -o %%~na\%%~na_ --dng %%~na.MLV )


Drag and Drop the MLV files onto the .bat file.
It will generate Folders with the frames
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: bouncyball on May 04, 2017, 08:09:22 AM
on previews: but what if I have Windows(8)?
Take a look @ this: http://www.visualbakery.com/Tools/PhotoTools.aspx (http://www.visualbakery.com/Tools/PhotoTools.aspx)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Savely on May 04, 2017, 03:08:55 PM
@Savely

In the directory where MLV_Dump.exe lies, make a .bat file and put this in it:

@echo off
for %%a in ( %* ) do ( md "%%~na" 2>nul )&( "mlv_dump.exe" -o %%~na\%%~na_ --dng %%~na.MLV )


Drag and Drop the MLV files onto the .bat file.
It will generate Folders with the frames

Thanks man, it works! But you'll have to put mlv file directly into the folder where this .bat file is.

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on May 04, 2017, 05:25:53 PM
Its a harmless compromise
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: jimiz on May 04, 2017, 05:36:12 PM
a good basic interface, resolve all the problems,and speed all the multi clips to convert
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: lostfeliz on May 04, 2017, 08:51:36 PM
This test was moderately helpful. I wanted to see how the 3k and HD compared with a high ISO -- 1600 (high for a 5Diii, at least). Three different lenses all shot at f2.8 at 1600 ISO. The field of view for ML cropped 3k is about 1/2 of what it is for full frame HD. So for the HD version of the shot, the camera was moved from 48" away to 24" away. I down-rezzed the 3k to HD.

I didn't have the patience to get the scale exact. I kept image pretty neutral. I used ETTR for exposure. I think it helped lessen the shadow noise (I tend to underexpose ML RAW, not in a good way). Anyway, the noise isn't too awful especially since you'd most likely crush the blacks some if you were using this shot. I was expecting more noise at 1600 ISO.

https://youtu.be/qWsWEl7fb4A (https://youtu.be/qWsWEl7fb4A)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hindra on May 05, 2017, 01:50:53 AM
Hey guys, just curious. I shot some footage, about 4 takes of a local artist and its about 114gb. 3k 12bit lossless 3.5k centered mode and it all went very well. It flashed between orange and green but I also had the global draw on. Camera temp hit orange a few times but it all worked great and it is super sharp!!

My question has to do with MLV_dump.exe, I left it overnight to process the DNGs but I think it got stuck because it never got past the first set of mlv files. I had selected all the time, 32 in total. I am re-attempting to do them one clip at a time now, but I'm curious if there is limit on GB size or amount of files I can run at once.

Is this normal? Does it need to process the images more than once? Thanks for all your help guys!

Quote
MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: 'M02-2034.MLV'
   - Enforcing 14bpp for DNG output
   - Convert to DNG frames
   - Output into 'M02-2034\M02-2034_'
File M02-2034.MLV opened
File M02-2034.M00 opened
File M02-2034.M01 opened
File M02-2034.M02 opened
File M02-2034.M03 opened
File M02-2034.M04 opened
File M02-2034.M05 opened
File M02-2034.M06 not existing.
Processing...


Vertical stripes correction:
  1.00000  1.00781  1.00792  1.00864  1.00664  1.00841  1.00572  1.00777
Cold pixels : 36
Reached end of chunk 1/7 after 1235 blocks
Reached end of chunk 2/7 after 1232 blocks
Reached end of chunk 3/7 after 1230 blocks
Reached end of chunk 4/7 after 1232 blocks
Reached end of chunk 5/7 after 1237 blocks
Reached end of chunk 6/7 after 1238 blocks
Reached end of chunk 7/7 after 436 blocks
Processed 7776 video frames
Done


 MLV Dumper v1.0
-----------------

Mode of operation:
   - Input MLV file: 'M02-2034.MLV'
   - Enforcing 14bpp for DNG output
   - Convert to DNG frames
   - Output into 'M02-2034\M02-2034_'
File M02-2034.MLV opened
File M02-2034.M00 opened
File M02-2034.M01 opened
File M02-2034.M02 opened
File M02-2034.M03 opened
File M02-2034.M04 opened
File M02-2034.M05 opened
File M02-2034.M06 not existing.
Processing...


Vertical stripes correction:
  1.00000  1.00781  1.00792  1.00864  1.00664  1.00841  1.00572  1.00777
Cold pixels : 36

EDIT: It seems to be on an endless cycle just repeating and wont move on to the next clip. Even if I only select the mlv files for one clip its just in an endless loop.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pocketrubbish on May 05, 2017, 05:39:01 AM
Hmm, I've been using mlv-dump (and the dumper batch) a bunch lately and haven't had any issues. Do you have the latest version of mlv_dump.exe? And are you sure you have enough hard disk space?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hindra on May 05, 2017, 06:01:01 AM
Hmm, I've been using mlv-dump (and the dumper batch) a bunch lately and haven't had any issues. Do you have the latest version of mlv_dump.exe? And are you sure you have enough hard disk space?

Yeah I am using the latest dump, and I have plenty of HD space. All the files get extracted, I can stop mlv-dump and my files are there for 1 clip. Ive been using it fine for the past week or so this just occurred the past few days. I havent changed anything so I'm not sure whats going on.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on May 06, 2017, 08:25:44 AM
Is the regression caused the overhead of printf (possibly delaying the decision by a non-negligible amount of time) or it's because of the change in thresholds (resulting in smaller blocks being written to the card)?

Results of over 30 hours of benchmarking (Apr29 codebase with minor changes - included in the raw log - to check the effect of these changes on performance). During this time, the camera was under the control of this script (https://bitbucket.org/hudson/magic-lantern/src/raw_benchmark/scripts/extra/rawbench.lua), unattended.

Code: [Select]
mlv_lite version : ML/MODULES/RAWBENCH/ASSUMEFS.MO:  865 frames (772 ... 925)
mlv_lite version : ML/MODULES/RAWBENCH/APR29.MO:     866 frames (789 ... 900)
mlv_lite version : ML/MODULES/RAWBENCH/NOPRINTF.MO:  874 frames (795 ... 902)
mlv_lite version : ML/MODULES/RAWBENCH/ASSUME70.MO:  885 frames (840 ... 956)
mlv_lite version : ML/MODULES/RAWBENCH/ASSUME80.MO:  938 frames (866 ... 984)
mlv_lite version : ML/MODULES/RAWBENCH/ASSUME90.MO:  946 frames (846 ... 986)

The effect of printf is not significant (could not prove the version without printf is any faster):
Code: [Select]
Quartile stats: median (Q1 ... Q3)
mlv_lite version : ML/MODULES/RAWBENCH/APR29.MO:  866 frames (789 ... 900)
mlv_lite version : ML/MODULES/RAWBENCH/NOPRINTF.MO:  874 frames (795 ... 902)

P =

           1     0.63117
     0.63117           1

T =

           0    -0.48056
     0.48056           0

But, the assumed write speed (when deciding to limit the number of frames saved in a single file write call, to avoid running out of space during the call) appears to have a sweet spot between 80 and 90%:

Code: [Select]
Quartile stats: median (Q1 ... Q3)
mlv_lite version : ML/MODULES/RAWBENCH/ASSUMEFS.MO:  865 frames (772 ... 925)      ; ASSUME100
mlv_lite version : ML/MODULES/RAWBENCH/NOPRINTF.MO:  874 frames (795 ... 902)      ; ASSUME50
mlv_lite version : ML/MODULES/RAWBENCH/ASSUME70.MO:  885 frames (840 ... 956)
mlv_lite version : ML/MODULES/RAWBENCH/ASSUME80.MO:  938 frames (866 ... 984)
mlv_lite version : ML/MODULES/RAWBENCH/ASSUME90.MO:  946 frames (846 ... 986)

P =

 Columns 1 through 4:

           1      0.4984  6.2291e-07  7.7958e-10
      0.4984           1   6.194e-05  7.2981e-07
  6.2291e-07   6.194e-05           1     0.06076
  7.7958e-10  7.2981e-07     0.06076           1
  1.7948e-08  4.1919e-06     0.11062     0.86482

 Column 5:

  1.7948e-08
  4.1919e-06
     0.11062
     0.86482
           1

T =

           0    -0.67773     -5.0558     -6.2829     -5.7438
     0.67773           0     -4.0546     -5.0451     -4.6852
      5.0558      4.0546           0     -1.8798     -1.5988
      6.2829      5.0451      1.8798           0     0.17034
      5.7438      4.6852      1.5988    -0.17034           0

So, while the first place (which assumes 90% of the measured speed for the last file write calls) is clearly better (squeezes more frames) compared to last two places (which assume either 100% or 50% of the measured speed for the same calls), we cannot tell the same about the first two winners - they are tied (could not prove one is faster than the other). Same for the last two places. The sweet spot is probably somewhere near 85% (not tested, just guessed).

(http://a1ex.magiclantern.fm/bleeding-edge/raw/ovf-sweetspot.png)

However, the best thing to do for speed is to write blocks as large as we can (in other words, as few file write calls as we can), and free the buffers as soon as we know they have been written. Here's an experiment on this:
Code: [Select]
Quartile stats: median (Q1 ... Q3)
mlv_lite version : ML/MODULES/RAWBENCH/ASSUME80.MO:  938 frames (866 ... 984)
mlv_lite version : ML/MODULES/RAWBENCH/ASSUME90.MO:  946 frames (846 ... 986)
mlv_lite version : ML/MODULES/RAWBENCH/SPECFREE.MO: 1015 frames (921 ... 1090)

P =

           1     0.86482  3.7192e-14
     0.86482           1  2.8011e-13
  3.7303e-14  2.8022e-13           1

T =

           0     0.17034     -7.8429
    -0.17034           0     -7.5833
      7.8429      7.5833           0

With large blocks, and reusing image buffers *during* a file write call, performance is clearly better (possibly also helping with overall speed, as the large buffers are freed and can be reused quicker).

(this experiment is a litte risky in my opinion, as I have not tested it for data integrity, other than self-checks built in mlv_lite, so I'm not comfortable including it in the builds, but you can build it from source - diffs available in the raw log)

Number of frames recorded by each build:
Code: [Select]
mlv_lite version : ML/MODULES/RAWBENCH/ASSUME70.MO
 832,1060,989,959,869,953,898,895
 741,837,839,827,921,767,905,985
 732,703,836,944,954,949,830,800
 739,864,926,961,956,949,866,866
 840,983,870,747,856,848,972,896
 762,870,987,853,978,953,876,852
 647,880,876,917,984,895,1002,886
 555,871,966,919,882,1028,976,791
 651,939,852,868,1063,863,872,925
 737,977,989,974,958,973,855,846
 756,871,974,978,969,994,953,791
 849,860,1023,974,997,979,993,943
 765,758,964,1004,1013,942,986,824
 823,835,864,955,969,832,847,877
 744,997,844,839,1039,951,941,880
 773,884,904,985,974,619,833,778
 850,1018,947,857,942,868,1011,901
 637,990,723,930,757,837,806,872
 756,952,977,856,965,950,956,883
 647,870,828,946,938,868,832,879
 750,904,944,942,863,976,872,843
 626,760,986,975,937,755,838,986
 540,851,764,874,1036,1048,1052,836
 655,856,878,747,928,763,843,905
 626,829,953,912,966,822,997,952
 804,893,884,840,902,939,933,937
 754,943,905,988,874,962,897,952
 885,966,885,971,950,984,872,874
 843,955,848,887,871,953,861,950

mlv_lite version : ML/MODULES/RAWBENCH/ASSUME90.MO
 665,1025,986,985,1000,882,993,698
 683,992,993,918,946,1026,906,781
 463,854,720,907,982,1040,1052,834
 828,777,806,956,1001,967,879,800
 555,851,876,859,807,986,881,944
 589,986,997,979,962,1003,918,856
 762,961,945,888,916,966,998,866
 689,906,970,975,826,975,684,972
 474,898,982,1008,982,982,990,978
 582,889,963,1010,1007,1044,994,765
 782,904,978,964,1002,965,841,928
 710,966,1018,802,989,903,981,952
 882,992,974,918,1013,961,1018,1039
 762,922,963,1009,1024,961,971,970
 765,903,922,948,1011,975,903,874
 679,917,904,992,965,916,962,985
 876,902,805,865,874,761,727,966
 745,981,1061,890,737,1002,1095,783
 831,875,837,992,711,940,1031,975
 581,651,976,901,1055,1005,922,901
 659,946,978,971,997,914,816,954
 773,989,968,979,987,798,1010,964
 858,841,685,793,993,812,987,980
 654,997,983,891,1036,918,997,899

mlv_lite version : ML/MODULES/RAWBENCH/NOPRINTF.MO
 572,818,859,843,691,827,898,805
 880,920,893,799,896,794,826,981
 781,973,995,916,877,777,874,893
 776,891,907,896,799,914,894,697
 768,796,918,890,902,715,810,792
 680,824,899,901,916,860,896,718
 466,641,900,848,907,836,870,959
 678,576,891,917,698,889,921,813
 862,850,797,943,871,892,969,882
 801,879,898,898,779,905,936,768
 581,930,916,1008,892,872,895,908
 440,720,919,896,890,906,831,1016
 571,797,883,861,985,875,845,793
 686,784,906,891,877,974,869,742
 772,874,857,916,968,775,801,786

mlv_lite version : ML/MODULES/RAWBENCH/ASSUMEFS.MO
 690,1040,864,754,879,861,935,972
 523,728,833,969,935,944,971,1054
 404,819,957,836,778,845,881,514
 796,886,779,776,917,957,892,760
 361,763,1003,597,929,743,907,983
 758,868,875,859,971,979,999,934
 848,803,794,777,883,861,669,681
 612,746,879,904,774,942,972,950
 572,789,555,769,862,786,918,823
 534,792,847,859,865,910,980,738
 838,935,980,915,850,879,670,785
 561,863,954,780,809,875,940,907
 795,964,1003,891,955,870,636,813
 460,733,980,974,1037,1072,874,801
 873,908,937,838,855,801,909,891
 753,834,877,881,948,910,935,1050
 856,874,941,764,873,920,585,884
 666,638,709,947,857,947,884,906
 576,967,818,985,660,798,774,1001
 711,889,799,900,869,837,810,836
 667,790,886,871,798,872,837,886
 602,887,922,721,929,706,949,936
 358,882,724,824,882,689,876,700
 438,693,928,789,860,888,869,867
 606,826,748,782,870,908,877,858
 551,523,933,760,930,711,911,965
 674,847,732,957,958,765,888,957
 665,731,881,1012,986,933,914,995

mlv_lite version : ML/MODULES/RAWBENCH/APR29.MO
 532,631,813,903,892,905,876,1030
 768,715,803,718,949,826,893,892
 807,977,930,799,889,821,887,775
 410,803,852,984,603,777,900,883
 653,802,984,707,774,889,858,815
 703,899,707,866,793,825,841,987
 696,987,926,883,966,896,869,945
 766,900,887,789,867,788,890,897
 801,860,969,972,839,840,909,890
 682,906,793,897,894,915,911,911
 509,756,807,811,945,1003,984,780
 678,808,716,809,806,827,925,932
 501,770,852,905,951,878,632,887
 737,899,870,981,959,939,888,798
 662,886,919,760,782,684,808,947
 664,884,719,685,882,804,822,884
 861,824,795,693,991,868,814,888
 686,886,895,880,870,712,914,861
 662,888,804,828,910,966,947,817
 667,820,961,784,904,842,900,884
 561,795,873,784,949,739,756,768
 557,896,880,804,903,921,802,911
 695,865,905,863,885,911,865,885
 670,972,868,908,850,877,868,881

mlv_lite version : ML/MODULES/RAWBENCH/ASSUME80.MO
 490,483,716,909,1020,996,991,814
 663,982,999,925,983,1004,974,898
 890,923,822,1015,879,987,857,882
 911,984,960,971,952,964,974,674
 847,932,978,911,1003,985,980,998
 783,986,984,925,981,955,995,871
 872,970,871,970,788,913,1027,1086
 678,872,873,843,980,980,995,894
 766,919,971,758,806,813,1002,933
 657,891,969,974,939,968,1023,963
 650,996,989,995,975,984,982,899
 780,703,997,1014,936,937,881,834
 760,955,966,772,989,980,866,990
 926,939,972,1013,981,960,1010,767
 676,1014,925,987,1007,897,908,763
 667,1027,801,890,985,767,1001,797
 758,912,866,1015,876,976,891,770
 466,910,772,941,981,755,979,742
 889,973,941,1014,969,968,667,756
 640,908,958,994,972,977,882,888
 793,889,986,986,1004,1000,983,1069
 752,991,1029,868,884,987,906,784
 630,881,879,982,955,926,836,841
 517,772,922,1023,1034,852,898,985
 856,943,835,885,805,992,971,1006
 684,952,975,1015,863,987,996,892
 561,975,890,978,985,1019,904,975
 559,954,909,922,902,1124,982,929
 783,1019,935,1035,971,996,884,962

mlv_lite version : ML/MODULES/RAWBENCH/SPECFREE.MO
 973,1083,1029,908,1121,1011,1110,718
 794,1132,1023,853,908,1098,1114,1004
 791,1066,899,920,958,1086,1197,811
 857,1135,1034,1097,996,937,1006,921
 819,1066,970,995,1063,1021,1113,1078
 641,1095,1076,992,996,1078,1015,892
 932,1111,1003,1109,1072,974,1007,1084
 796,1009,1098,1114,1121,1000,717,881
 910,928,1012,917,1055,1085,1073,820
 617,970,1141,996,1031,1102,1022,1012
 790,1010,1072,1017,1023,1043,1034,922
 910,1140,920,1106,1116,1126,989,919
 791,772,1016,1024,1110,1123,1096,832
 802,1013,961,1097,1120,1057,1103,971
 754,984,1081,1078,1176,907,908,602
 848,1100,1115,1089,1111,1102,908,935
 1016,1106,1017,1013,1104,1045,1007,982
 713,1057,987,997,1037,954,1095,925
 992,1101,1084,1076,1068,1024,1095,920
 790,911,1128,1121,1096,1020,1010,600
 808,938,1096,978,1042,919,1099,910
 837,982,1100,1077,1080,1015,1039,917
 769,1006,1083,1091,1102,1090,915,981

Raw log (http://a1ex.magiclantern.fm/bleeding-edge/raw/RAWBENCH-4K.LOG) (includes diffs)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on May 06, 2017, 11:57:49 AM
Thank you very much for the test...
If somebody can compile the build that A1ex suggest, I will test it.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Savely on May 06, 2017, 12:29:49 PM
Does anyone have tried Dual ISO in crop_mode? It seems there are problems there. Frames (after Dual ISO Processor) comes with stripes like unprocessed dual ISO frames. At least with x5 (x3) zoom. Is there some solution?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on May 06, 2017, 12:35:22 PM
Could you post a sample file?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Savely on May 06, 2017, 04:26:32 PM
Actually it comes with stripes with, and with no zoom. Here it is:

zoom: https://yadi.sk/i/9R1QvSSJ3Hm4HD
no zoom: https://yadi.sk/i/ZVhz4gHe3Hm3Nn
download button (russian interface)http://joxi.ru/J2bJWXZFXLXed2.jpg
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on May 06, 2017, 04:51:09 PM
Ok, trying to understand what to look for.
The non crop file seems ok. After running the file through cr2hdr I get this.
(https://s18.postimg.org/72li7s4jt/Screen_Shot_2017-05-06_at_16.46.00.png)

Quote
Quote
Does anyone have tried Dual ISO in crop_mode? It seems there are problems there. Frames (after Dual ISO Processor) comes with stripes like unprocessed dual ISO frames. At least with x5 (x3) zoom. Is there some solution?
Checking your file frame_000008frame_000024_crop_mode_with_zoom.dng it doesn´t look healthy. So you´re problems are only the crop mode recording with compressed 4k modes? If yes, could you share a small MLV sample instead of a single dng file?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Savely on May 06, 2017, 07:51:18 PM
Ok, trying to understand what to look for.
The non crop file seems ok. After running the file through cr2hdr I get this.
(https://s18.postimg.org/72li7s4jt/Screen_Shot_2017-05-06_at_16.46.00.png)

Checking your file frame_000008frame_000024_crop_mode_with_zoom.dng it doesn´t look healthy. So you´re problems are only the crop mode recording with compressed 4k modes? If yes, could you share a small MLV sample instead of a single dng file?

Try to zoom it. Non crop I mean. And how did you running it through cr2hdr? I tried but it just flashing and nothing.

Thid mlv file with no zoom. https://yadi.sk/d/7nw0j1Fq3HmYpN (with zoom pretty big for half Gb mininum). As I see, problem with stripes is in unzoomed DNG's (RAW Dual ISO on crop_mode build just fine as well as Dual ISO MLV's on older builds).

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on May 06, 2017, 09:04:42 PM
The mlv file you just posted @savely converts just fine.
(https://s2.postimg.org/5d0vpzqrd/Screen_Shot_2017-05-06_at_21.04.29.png)

I´m on mac and using this. It´s a1ex cr2hdr binary working.
http://www.magiclantern.fm/forum/index.php?topic=15108.0
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on May 07, 2017, 08:30:01 PM
*update 3
Pull request made here
https://bitbucket.org/Dannephoto/magic-lantern/commits/642fb8c4d5bd7bd8169bc9b85bae11eb27495892


I´m already a big fan of the crop_rec4k mlv_dump version and the compression of dng and MLV files are great. Encountered an issue with eosm footage. Files won´t parse(open) in acr and dcraw reveals what´s wrong. Corrupted file. Other footage from eosm is working.

Testfile:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/M20-1113_samples_Issue_parsing.zip

dcraw output and terminal message:
 
Code: [Select]
Unexpected end of file(https://s29.postimg.org/r72sssd6v/Screen_Shot_2017-05-07_at_20.27.28.png)

*update.
Maybe metadata related. Other random ml builds is working with eosm. Guess I was lucky with this corrupted footage  8). Wish I knew what build I was using when filming. Have a bunch of files om my computer which comes out like this. The transcode just fine with other builds of mlv_dump.

*update 2
It seems it´s because of being 10 or 12 bit footage. Guess that´s not working at the moment with the crop_rec4k mlv_dump build.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Savely on May 07, 2017, 09:00:54 PM
The mlv file you just posted @savely converts just fine.
(https://s2.postimg.org/5d0vpzqrd/Screen_Shot_2017-05-06_at_21.04.29.png)

I´m on mac and using this. It´s a1ex cr2hdr binary working.
http://www.magiclantern.fm/forum/index.php?topic=15108.0

Can you please upload your "fine" result? Cause in crop mode build with Dual ISO I always end up with striped pics or this pattern:(http://joxi.ru/brRd9nNFJdY4z2.jpg)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: giarcpnw on May 08, 2017, 07:03:33 AM
I just shot a 3x crop of a hummingbird on a perch with a 70-200mm lens. This is just insane detail. 3072x1228 12bit lossless.

Download the movie and watch it in ProRes. Literally moving photographs.

https://www.dropbox.com/s/0vm8zsabe39vvf9/Humming3072.mov?dl=0
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: ted ramasola on May 08, 2017, 08:03:34 AM
I just shot a 3x crop of a hummingbird on a perch with a 70-200mm lens. This is just insane detail. 3072x1228 12bit lossless.

Download the movie and watch it in ProRes. Literally moving photographs.

https://www.dropbox.com/s/0vm8zsabe39vvf9/Humming3072.mov?dl=0

wow. Can I ask a favor? Would you mind sharing a single dng frame of that hummingbird that I can look at in davinci? That's beautiful.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: giarcpnw on May 08, 2017, 05:30:06 PM
wow. Can I ask a favor? Would you mind sharing a single dng frame of that hummingbird that I can look at in davinci? That's beautiful.

Here ya go.

https://www.dropbox.com/s/ebxx9xrebn04451/M07-1434_C0000_00697.dng?dl=0
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: ted ramasola on May 08, 2017, 10:15:44 PM
Here ya go.

https://www.dropbox.com/s/ebxx9xrebn04451/M07-1434_C0000_00697.dng?dl=0

Thank you, the image can still be zoomed to fill an UHD timeline in resolve. Nice.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: giarcpnw on May 08, 2017, 10:31:57 PM
Thank you, the image can still be zoomed to fill an UHD timeline in resolve. Nice.

I'm certain I can get 3520x1320. Not sure that offers a 3x crop option though, I just didn't dive into the settings that hard since i was trying to capture the little guy before he took off.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Oedipax on May 08, 2017, 10:41:43 PM
I'm certain I can get 3520x1320. Not sure that offers a 3x crop option though, I just didn't dive into the settings that hard since i was trying to capture the little guy before he took off.
3520x1320 works for me but only up to maybe 100 frames (KomputerBay 128gb 1066X CF card). But I have only tried it in the 14-bit lossless mode, 24fps. Is anyone getting longer recordings using 12-bit lossless instead?

Great looking footage, by the way!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: giarcpnw on May 08, 2017, 10:50:01 PM
I was getting continuous in 12bit...

all this...most of these scenes were in the 20-30 second range before I cut them shorter. I was using KomputerBay 64gig 1000X

https://www.youtube.com/watch?v=ZdvKALy58Es
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Oedipax on May 09, 2017, 06:28:44 PM
I realized after posting yesterday that 12-bit lossless and 10-bit lossless aren't enabled yet for the 3K/4K crop modes anyway. So 3520x1320 was continuous for you in 12-bit uncompressed? I'll have to try that out! I just assumed the larger bitrate would be prohibitive, but maybe it's something else that inhibits the 14-bit lossless from doing it continuously at that res?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on May 09, 2017, 09:18:56 PM
I'm not sure these are accurate rec times mentioned above. Possibly I guess but as far as my testing has shown, 3072 2.35:1 23.976fps override on the 5x mode in the crop rec menu @12bit lossless are continuous. Even with these settings there could be scenes that can spike the data rate so that it is not continuous. Above that resolution, I don't THINK you will see continuous. Remember, if you are using the 5x option under crop rec menu, you have to hit the zoom button to actually rec the resolution. If you don't it will just rec 1920. I'm thinking maybe that's where some confusion can happen. If your not zoomed in the 5x option in the crop mod you will not be recording higher then 1920 resolution. Also if you are using the 5x mode you will need to enable fps override since canon live view defaults to 30fps when zoomed in.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: giarcpnw on May 09, 2017, 10:44:33 PM
I shot 3520x1320 12bit continuous in the 5x centered mode continuous for 2 minutes last night. It made a 10gig (ish) mlv file but it did it. It's definitely doable.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on May 09, 2017, 11:29:17 PM
Oh I didn't realize you were using 2.66:1 ratio @giarcpnw
And you mean 12bit lossless right?
Also did you hit the zoom button before recording?

If you are in 12bit  lossless mode Try to ETTR really branchy tree tops with no highlight clipping on an over cast day and I bet your data rate will skyrocket with those settings. It did for me anyway.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: giarcpnw on May 10, 2017, 12:52:00 AM
Oh I didn't realize you were using 2.66:1 ratio @giarcpnw
And you mean 12bit lossless right?
Also did you hit the zoom button before recording?

If you are in 12bit  lossless mode Try to ETTR really branchy tree tops with no highlight clipping on an over cast day and I bet your data rate will skyrocket with those settings. It did for me anyway.

Yeah sorry. lossless. Always lossless. Can't seem to get past 3072 if 14bit lossless. I've not seen a big diff between 14 and 12 bit to try to go higher.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on May 10, 2017, 08:22:16 AM
Recording @ 3.5K 12bit Lossless is really tricky it all depends on your scene.  3072 - 12bit Lossless is working lovely.. Live View stay's full color and I'm able to focus properly.  Not sure if anybody experienced spiked data rates using the Sigma 35mm ART lens, or any of the ART lens.  I'm thinking that because those lens are soo sharp it's causing that issue.   
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on May 10, 2017, 10:39:16 AM
@goldenchild9to5

Do you get real time liveview with 3072 - 12 bit lossless?. Full color but no realtime, isn't it?

Thanks.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on May 10, 2017, 04:08:12 PM
@pc_bel no it's not realtime but I'm getting use to it.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on May 10, 2017, 10:07:47 PM
@pc_bel, @goldenchild9to5:

Have a look at the preview options; the help text has some hints for your use cases (also discussed earlier in this thread (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183548#msg183548)).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: domasa on May 10, 2017, 10:32:49 PM
Today I had crash during record:

CRASH00.LOG
ASSERT: IsSuiteSignature( hSuite )
at ./PackMemory/PackMem.c:983, task MoviePlay
lv:0 mode:3


Magic Lantern version : crop_rec_4k.2017Apr29.5D3123
Mercurial changeset   : d55423f1e270 (crop_rec_4k) tip
Built on 2017-04-29 14:31:28 UTC by [email protected]
Free Memory  : 164K + 3811K

ASSERT00.LOG
ML ASSERT:
fullsize_buffers[1] == UNCACHEABLE(raw_info.buffer)
at mlv_lite.c:1379 (free_buffers), task raw_rec_task
lv:1 mode:3


Magic Lantern version : crop_rec_4k.2017Apr29.5D3123
Mercurial changeset   : d55423f1e270 (crop_rec_4k) tip
Built on 2017-04-29 14:31:28 UTC by [email protected]
Free Memory  : 144K + 3821K

Info
I used H.264 proxy (H264->SD with ML; RAW 1920x1080 12-bit->CF)

Magic Yoom Large TL 3>1, Waveform small, Red Crossout, DIGIC peaking: Slightly sharper, Zoom with old button..
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on May 11, 2017, 05:45:09 AM
@pc_bel, @goldenchild9to5:

Have a look at the preview options; the help text has some hints for your use cases (also discussed earlier in this thread (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183548#msg183548)).
@a1ex this feature changed everything! So awesome!
Title: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on May 11, 2017, 07:54:00 AM
Some fun stuff w DarkFraming Avg Process in 12-bit lossless @ ISO 6400 from 5D3.123 shot w Helios 44-2.

Cinelog_Rec709 has been applied for comparison viewing purpose. Full album can be found here: https://flic.kr/s/aHskVk85io

Pre DF (pushed exposure up to +2.50 in ACR):
(https://c1.staticflickr.com/5/4158/34450561991_2c584c7d1a.jpg) (https://flic.kr/p/Uuh8QK)

Post DF (pushed exposure up to +2.50 in ACR):
(https://c1.staticflickr.com/5/4168/33738123384_3ddba92881.jpg) (https://flic.kr/p/TpjGEu)

Pre DF (pushed exposure up to +2.50 in ACR):
(https://c1.staticflickr.com/5/4174/34450578201_a8ffa6712b.jpg) (https://flic.kr/p/UuhdEe)

Post DF (pushed exposure up to +2.50 in ACR):
(https://c1.staticflickr.com/5/4161/34450549851_0bec0a5d1f.jpg) (https://flic.kr/p/Uuh5er)

Pre DF (pushed exposure up to +2.50 in ACR):
(https://c1.staticflickr.com/5/4179/34450580751_e333e68295.jpg) (https://flic.kr/p/Uuheqc)

Post DF (pushed exposure up to +2.50 in ACR):
(https://c1.staticflickr.com/5/4183/33738094614_51049bfdb3.jpg) (https://flic.kr/p/Tpjy7s)

Pre DF (pushed exposure up to +2.50 in ACR):
(https://c1.staticflickr.com/5/4169/33738115394_d00f1a6282.jpg) (https://flic.kr/p/TpjEhJ)

Post DF (pushed exposure up to +2.50 in ACR):
(https://c1.staticflickr.com/5/4194/34194921370_04b217357d.jpg) (https://flic.kr/p/U6FUMQ)

Such a relief that @bouncyball, @Danne, @g3gg0 and the rest of you all have made this possible to bring back this useful feature in DarkFraming Average Processing. Especially for lossless compressions.

Yes, all of this was solely done with the latest and greatest cr2hdr.app (https://www.magiclantern.fm/forum/index.php?topic=15108.0cr2hdr.app) (Thanks @Danne for standing by with all of your love & support!)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: janvkem on May 11, 2017, 10:24:20 AM
Looking good DeafEyeJedi! Do you create the darkframe by shooting (with the same settings) with the lenscap attached? Can you do it anytime after the shot (like when i go on holiday and i make the darkframes when i come back home) or does it have to be close together in time? For windows users mlv_dump can do this right?

Quote (with pics removed):
Some fun stuff w DarkFraming Avg Process in 12-bit lossless @ ISO 6400 from 5D3.123 shot w Helios 44-2.

Cinelog_Rec709 has been applied for comparison viewing purpose. Full album can be found here: https://flic.kr/s/aHskVk85io

Such a relief that @bouncyball, @Danne, @g3gg0 and the rest of you all have made this possible to bring back this useful feature in DarkFraming Average Processing. Especially for lossless.

Yes, all of this was solely done with the latest and greatest cr2hdr.app (https://www.magiclantern.fm/forum/index.php?topic=15108.0cr2hdr.app) (Thanks @Danne for standing by with all of your love & support!)
Title: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on May 11, 2017, 10:41:20 AM
Do you create the darkframe by shooting (with the same settings) with the lenscap attached?

Correct.

Can you do it anytime after the shot (like when i go on holiday and i make the darkframes when i come back home) or does it have to be close together in time?

Anytime works.

For windows users mlv_dump can do this right?

I believe so, not sure how tho.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: janvkem on May 11, 2017, 11:04:21 AM
Thanks for the reply, going to implement this into my workflow!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on May 11, 2017, 11:58:49 AM
Quote
Quote from: a1ex on Yesterday at 10:07:47 PM

    @pc_bel, @goldenchild9to5:

    Have a look at the preview options; the help text has some hints for your use cases (also discussed earlier in this thread).

Thank you!!! I didn't see that feature.
Hope it worked in 50p3x3 for correct framing.
@A1ex
It's easy to implement it in other button different from the shutter release?...
Thanks again.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on May 11, 2017, 12:19:00 PM
There are a few other buttons that send the same event (it's hard to tell the difference between them, other than waiting about 0.5 seconds to see whether it starts autofocusing or not). Use any of them ;)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on May 11, 2017, 03:24:06 PM
 :D ;D
THANKS!!!!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on May 12, 2017, 03:59:34 AM
Thanks @a1ex
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: whysodifficult on May 12, 2017, 04:08:05 AM
Hello everyone,

Could someone explain, please, what i do wrong? My brain is just exploding of complexity.

I've been trying to shoot 3k or 4k or 5k for a few days using instructions here, but everything either work 5 sec or switches to 1920 automatically. Every day it behaves differently. One day i managed to work each resolution but time was different - 3 sec mostly. Couldn't do what i've seen others shot.

Right now i tried to shoot 3,5K crop according to instructions and it allowed me to shoot only 5 sec, also preview freezes. Maybe i've done something wrong with FPS override? I tried 23.976 (from 25), Exact FPS - 23.976 and Low light - 23.980.

My goal is to find the maximum resolution and aspect ratio with reasonable time (say, from 5 min of recording).

I have 5DM3 + 1.1.3 Firmware + Sandisk 30 mb/s (exFAT) with ML (crop from 29 April) + KomputerBay 64GB 1000x (exFAT).

Thank you!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: giarcpnw on May 12, 2017, 05:50:23 PM
@whysodifficult:

Are you choosing a lossless compression? 3.5k only seems to work continuously with 12bit lossless. 3k maybe with 14bit lossless. You'll have to lower frame rates down significantly to get 4 or 5k. They are not feasible options for traditional frame rates.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on May 12, 2017, 06:30:50 PM
@whysodifficult I am currently making a video tutorial for 3k if you can give me to the end of the day
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: rob_6 on May 12, 2017, 07:06:05 PM
@whysodifficult I am currently making a video tutorial for 3k if you can give me to the end of the day

@hjfilmspeed

Really looking forward to your video tutorial! Thanks so much in advance!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on May 12, 2017, 08:09:28 PM
Been fiddling around mlv_dump code in crop_rec_4k branch and by doing two minor changes it works a little better around 10bit12bit files.
https://bitbucket.org/Dannephoto/magic-lantern/commits/b1e226a565287ae563944cc035957bd213852855?at=crop_rec_4k_ver2

What seems to work now is following:

MLV files
- Compression(lj92) of all flavours of mlv files including 10bit12bit14bit, compressed raw, regular raw. Compression level is very efficient, almost 50%.

- Decompression of mlv files into decompressed dng files or back to decompressed mlv. This works with all MLV flavours going from compressed straight to decompressed dng files. This is really great since we do not need to decompress the MLV file once it´s compressed.

DNG files
- Compression of 14bit dng files, all flavours works very good. Efficient compression around 40-50%.

Not working
- Compression of 10bit12bit files. DNG output comes out all black but compressed and with metadata present.

I did some minort changes to get 10bit12bit working at all. There is also some testing with decompressing code since the files will come out with incosistent sizes when decompressing. Depends on if coming from mlv_block or outputting dng files. Check code here.
https://bitbucket.org/Dannephoto/magic-lantern/commits/b1e226a565287ae563944cc035957bd213852855?at=crop_rec_4k_ver2

I´m trying to see what is missing around 10bit12bit now. One funny note is that if erasing bitdepth = 14; mlv_dump will output compressed 10bit12bit files fine into original 10bit12bit. Of course this will not work with vetical stripes code since it´s written in 14bit but then again it means it´s halfway working when run_compressor is set.

If any hints to these findings I´d be greatful since I hit a wall now.

Here are some testfiles by the way.
https://bitbucket.org/Dannephoto/magic-lantern/downloads/10_12_14_bit.zip
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: ju64 on May 12, 2017, 09:02:47 PM
Hi everyone,

I spent a few days trying to shoot with the crop mode 3.5K but nothing conclusive.

When I use crop mode 3.5K 1:1 centered (resolution 3504x1320 and fps overide 23.976) if I use 12 bit lossless I have the overexposed zone pink, with 14 bits lossless no problem.

Any idea where is my problem ? Did I miss something ?

Thank you !
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: whysodifficult on May 12, 2017, 11:47:05 PM
@whysodifficult:

Are you choosing a lossless compression? 3.5k only seems to work continuously with 12bit lossless. 3k maybe with 14bit lossless. You'll have to lower frame rates down significantly to get 4 or 5k. They are not feasible options for traditional frame rates.

If lossless compression means 10-12 bit instead of 14 bit, then yes, i tried it as well. Thank you for the tip!

I wonder can there be a difference between 1.1.3 and 1.2.3 firmware versions.... I'm using 1.1.3.

@whysodifficult I am currently making a video tutorial for 3k if you can give me to the end of the day

Thank you very much! I am looking forward to watch it, so as other people, i'm sure!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on May 13, 2017, 03:44:37 PM
Quote
Been fiddling around mlv_dump code in crop_rec_4k branch and by doing two minor changes it works a little better around 10bit12bit files.

@danne Thank you very much!!!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on May 13, 2017, 08:31:59 PM
@whysodifficult @rob_6
This video is not for those new to ML. Try this at your own risk. I am not responsible for any damage this video may bring if your camera explodes or bricks. Magic Lantern is not responsible either. It will probably just brick not explode though.
Magic Lantern 3072 2.35:1 12 bit Lossless RAW Recording from the CROP_REC_4K build from 4-29-2017
Once you have ML installed and the correct build on your SD card, this video should help you set up your 5DIII to record 3K almost continuously depending on the scene. Remember ML on SD and Rec to CF.
Remember this build is experimental and not for those that are new to ML. Try at your own risk.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: g3gg0 on May 13, 2017, 09:06:38 PM
Been fiddling around mlv_dump code in crop_rec_4k branch and by doing two minor changes it works a little better around 10bit12bit files.
https://bitbucket.org/Dannephoto/magic-lantern/commits/b1e226a565287ae563944cc035957bd213852855?at=crop_rec_4k_ver2

What seems to work now is following:
hi danne
thanks for your feedback and testing!

does that mean with that changes you managed to compress 10/12 bit images correctly?
made test cases for all combinations of compression/decompression/recompression/pass-thru-unprocessed-lossless to check mlv_dump behavior.

though i didn't complete that to cover 10/12 bits, just made some sloppy dev tests.

here is the test case i had for these tests:
Code: [Select]
mlv_dump_test: mlv_dump
rm *.dng
./mlv_dump -v 14bit_lossless.mlv -o mlv_dump_test_original.mlv
./mlv_dump -v 14bit_lossless.mlv -o mlv_dump_test_decompressed.mlv -d
./mlv_dump -v 14bit_lossless.mlv -o mlv_dump_test_recompressed.mlv -c
./mlv_dump -v 14bit_lossless.mlv -o mlv_dump_test_recompressed_double.mlv -c -c
./mlv_dump -v mlv_dump_test_original.mlv              --dng -o mlv_dump_test_0_exp_0
./mlv_dump -v mlv_dump_test_decompressed.mlv          --dng -o mlv_dump_test_1_exp_0
./mlv_dump -v mlv_dump_test_recompressed.mlv          --dng -o mlv_dump_test_2_exp_0
./mlv_dump -v mlv_dump_test_recompressed_double.mlv   --dng -o mlv_dump_test_3_exp_0
./mlv_dump -v mlv_dump_test_original.mlv              --dng -o mlv_dump_test_4_exp_1  -c
./mlv_dump -v mlv_dump_test_decompressed.mlv          --dng -o mlv_dump_test_5_exp_1  -c
./mlv_dump -v mlv_dump_test_recompressed.mlv          --dng -o mlv_dump_test_6_exp_1  -c
./mlv_dump -v mlv_dump_test_recompressed_double.mlv   --dng -o mlv_dump_test_7_exp_1  -c
./mlv_dump -v mlv_dump_test_original.mlv              --dng -o mlv_dump_test_8_exp_2  -c -c
./mlv_dump -v mlv_dump_test_decompressed.mlv          --dng -o mlv_dump_test_9_exp_e  -c -c
./mlv_dump -v mlv_dump_test_recompressed.mlv          --dng -o mlv_dump_test_a_exp_3  -c -c
./mlv_dump -v mlv_dump_test_recompressed_double.mlv   --dng -o mlv_dump_test_b_exp_3  -c -c
md5sum mlv_dump_test_*.dng

the result then i manually checked if that is what i expected.
all with exp_0 should have the same hash, same for exp_1, exp_2 and exp_3.
the exp_e is expected to throw an error.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on May 13, 2017, 10:10:36 PM
Hi g3gg0. Thanks for getting back on this. I'm out at the moment but will describe from memory.
As you describe 14bit compression(-c --dng) is working nicely with dng exports. When exporting 10bit12bit the dng files comes out all black.
What is working is compression of MLV files and that works for all flavours, 10bit12bit14bit.

What is adressed in my code changes is to firstly get regular uncompressed 10bit12bit working again at all since it was broken in this build. Next was to try -c with 10bit12bit but always comes out black.

My last code changes made it possible to export to uncompressed dng files coming from compressed MLV. That works with all bits and flavours.
Sorry for these messy allround tests.
If I would suggest for one priority fix is to get -c --dng option working with 10bit12bit files if possble at all?


Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: whysodifficult on May 13, 2017, 11:52:06 PM
@hjfilmspeed

Thank you very much for the video, i did everything you said. It works now!! I really appreciate it!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: D_Odell on May 13, 2017, 11:58:15 PM
Anyone else's experienced that 12-but is unavailable to choose in latest build 1.1.3? It says in mine that it's only available in photo mode.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on May 14, 2017, 12:07:22 AM
It says in mine that it's only available in photo mode.

There's no code path that could display this message in the raw video menu, and there are no lower bit depths available in photo mode.

http://www.chiark.greenend.org.uk/~sgtatham/bugs.html

When I use crop mode 3.5K 1:1 centered (resolution 3504x1320 and fps overide 23.976) if I use 12 bit lossless I have the overexposed zone pink, with 14 bits lossless no problem.

See commit messages from 2028d73 (https://bitbucket.org/hudson/magic-lantern/commits/2028d73f6f34dcdaaa4c241824e538d0e0e47267), 945aadb (https://bitbucket.org/hudson/magic-lantern/commits/945aadbefad03916a40f11762b79a82127d2a6f2) and 2aee5f8 (https://bitbucket.org/hudson/magic-lantern/commits/2aee5f851ab17becb16dfcb00bf3b41e7d510d0f).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: rob_6 on May 14, 2017, 07:40:38 AM
@hjfilmspeed

Amazing! Thank you so much for the tutorial video. Everything makes sense now! So seriously cool you made this for us! I love magic lantern! You guys are all awesome!

Rob
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pjs9505 on May 14, 2017, 10:35:51 AM
(http://thumb.ibb.co/jKTf5k/7678_re.jpg) (http://ibb.co/jKTf5k)

My 3K continuous video record settings.
HDMI Output is check focusing preview monitor (real time fps), and DSLR monitor is check frame position preview monitor (only 7 fps slow, but no crop fully size preview)

DMStorage DM38 128GB CF Card (exFAT)


HDMI output monitor will help you.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on May 14, 2017, 05:31:52 PM
Sick!!!! @pjs9505! Are you on 1.2.3 canon firmware?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: etx on May 14, 2017, 06:12:05 PM
Here are a couple quick edits I did last night. First one no real color work I was chasing my dogs around testing my 3d printed electronic follow focus I made for my Movi.

https://youtu.be/pRROVuxevuU

And one I shot yesterday at Red Bull Hart Lines in Detroit yesterday. No press pass so it was hard to get where I needed to be.

https://youtu.be/Iag4strSaEg

Thanks so much to the ML team! The raw advancements made me fall in love with my camera again. Everything was shot near 3k in 5x centered crop mode as this is the best for shooting with an external monitor on my Movi m10.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: g3gg0 on May 14, 2017, 09:06:25 PM
Not working
- Compression of 10bit12bit files. DNG output comes out all black but compressed and with metadata present.

when using windows tools for preview, or lightroom i also get black images.
but when using dcraw to convert to TIF, i get a valid image.

a1ex said that adobes tool do currently not support all types of DNG images properly, so this might be due to their libraries.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on May 14, 2017, 09:18:01 PM
Hm, dcraw outputs a valid one. Thought I tried that before. Will test on my mac soon.
Fyi if compress a 10bit file(--dng -c) and export it with bit_depth = 10; I get a crisp, compressed and fully valid 10bit dng file. Makes me speculate if upres to 14bit could be applied later after compression? Of course I do not know fully how those bits behave going up and down.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: ShootMeAlready on May 15, 2017, 03:11:09 AM
Read a number of posts, and saw a 700D reference but no real explanation for any aps-c port.
I am a 70D user, and was wondering what the difference is here? to get to 4k?
The 70D even at 3X zoom mode, would be hard pressed to get 2k for any length of time.
Is there something I am missing, and what would be a 70D (card write is still 40 MB/s) max?

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: dfort on May 15, 2017, 03:50:23 AM
saw a 700D reference

Yeah, that was me. It seems that all cameras can do the ProcessTwoInTwoOut (http://www.magiclantern.fm/forum/index.php?topic=18443.msg184464#new) thing. Basically the CR2 files are run through that lossless compression process. Then it is another thing to figure out how to tweak the sensor using adtg_gui in order to squeeze every pixel you can out of the camera. I looked into it a while ago but was in way over my head. Some guys are still trying and eventually someone should come up with something that works on the aps-c cameras.

You're right though--that maximum SD card write speed is going to keep these cameras from reaching 4K, except maybe the 7D because it has a faster CF card?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pjs9505 on May 15, 2017, 11:29:31 AM
Sick!!!! @pjs9505! Are you on 1.2.3 canon firmware?

1.2.3 Firmware.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Jared on May 15, 2017, 05:39:17 PM
Hello everyone, someone could guide me about the workflow with crop_rec under Windows, thank you very much.  8)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: D_Odell on May 15, 2017, 11:12:20 PM
Hi!

Does crop mode have snd module available? Or impossible?

Best,
David
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Walter Schulz on May 15, 2017, 11:20:43 PM
http://www.magiclantern.fm/forum/index.php?topic=19300.msg184232#msg184232
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on May 16, 2017, 02:30:15 AM
@hjfilmspeed Awesome video.. This will help the Newbies big time. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on May 16, 2017, 02:37:37 AM
@pjs9505 Great setup..
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: KMikhail on May 17, 2017, 12:12:23 PM
Hello guys.

Since I've tried April crop mode module, my shutter-release button has been acting up - it has reset itself twice to AE lock, instead of AFing. Plus microadjustment has switched itself once to a diff value as well. It has never happened to me before and at the time of malfunction the standard ML was installed. No warnings of any kind were shown upong booting with any of the April versions of the ML.

Thanks
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on May 17, 2017, 12:55:44 PM
The half-shutter function is changed by ML and can be modified without notice if you take the battery out in the middle of taking a picture initiated from ML, for example. This is documented (http://wiki.magiclantern.fm/faq#does_ml_do_any_persistent_changes_to_my_camera).

If it's not that, find a way to reproduce (http://www.chiark.greenend.org.uk/~sgtatham/bugs.html). I'd expect some similar behavior to happen after an assertion, a crash (e.g. err70) or after taking the battery out - in these cases, ML actively prevents Canon code from saving their settings at shutdown. This is a safeguard to prevent invalid settings (that could be related to the crash) from being saved into ROM (more info (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183060;topicseen#msg183060)).

Note: I've reviewed the calls related to changing half-shutter functionality (cfn_set_af_button and its wrappers) and could not find any instance where the original setting is not restored as soon as it's no longer required. So, unless you (have to) take the battery out in the middle of some photo operation (e.g. bracketing sequence), I don't see how it would happen. Nothing on the video side changes this setting.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: KMikhail on May 17, 2017, 01:43:36 PM
Thanks, much appreciated. I'll look into provided docs.

Mikhail
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: RTI on May 17, 2017, 10:42:15 PM
Hi,

I suppose it has been covered, but I haven't been able to find a direct answer to my issue, so don't hit me too hard :)
In short - the dng files that have lossless compression aren't supported natively in the latest version of  Adobe premiere (at least on my PC). I wouldn't even bother with that because Resolve does a great job, but with the latest version of Premiere i found that it handles canon raw quite good (it can be absolutely my feeling, as didn't find anything in the release notes)

Is there any way to import the dngs directly into premiere? The simple uncompressed dng files work just fine.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on May 17, 2017, 10:52:00 PM
If on mac you get compatiblity with a mlv_dump version used in cr2hdr.app (ml-dng branch + several addons)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: RTI on May 17, 2017, 11:57:31 PM
Thx Danne, I'm on PC and using the latest availabe mlv_dump from the experiments download section.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on May 18, 2017, 05:20:35 AM
If you can compile mlv_dump from this branch it should work @RTI
https://bitbucket.org/Dannephoto/magic-lantern/branch/ml-dng-unified_8_-Mr_B_compress_ver2
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: RTI on May 18, 2017, 09:05:29 AM
@Danne, thx for the input. But i have zero knowledge in coding and compiling stuff :) For now it will be easier for me to install a hackintosh.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on May 18, 2017, 10:31:07 AM
Or someone with a windows setup could help you compile a version.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: squig on May 18, 2017, 12:47:58 PM
I've been messing around with the 1.5x Iscorama shooting 4K scope 24p anamorphic (2304x1382 with a 25% upscale). With a 24mm Leica-r it's equal to 40mm full frame. Shot at 3200 ISO with ACR chroma noise reduction. Filmconvert Fuji 8553 LUT.

(http://i.imgur.com/KG169dqh.jpg)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on May 18, 2017, 02:32:09 PM
I have 16 MLV files that fail to decompress. They almost 1GB average, each.
I was shooting with 5d mk3, 1.1.3. 1920x1080/50/60 2.35:1, 14bit lossless.
I use latest mlv_dump.
I get a message

LJ92 : failed (-1)

The mlv files were random among 70 files and their order do not satisfy any logic.
The last 10 shots were faulty. I assume the problem rested there.

I can upload a mlv file on my GoogleDrive for public access.
Any clue ?
Thanks in advance
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on May 18, 2017, 03:17:47 PM
Please upload a sample file. If you dig round mlv_dump code it says following.
Error related to int ret. Frame size?

Code: [Select]
    /* this block will load an image from a MLV file, so use its reported frame size for future use */
    if(subtract_mode)
    {
        printf("Loading subtract (dark) frame '%s'\n", subtract_filename);
        int ret = load_frame(subtract_filename, &frame_sub_buffer, &subtract_frame_buffer_size);


Your whole error code snippet from mlv_dump.c
 
Code: [Select]
                           int ret = lj92_open(&handle, (uint8_t *)frame_buffer, frame_buffer_size, &lj92_width, &lj92_height, &lj92_bitdepth, &lj92_components);

                            /* this is the raw data size with 16 bit words. it's just temporary */

                            size_t out_size = lj92_width * lj92_height * sizeof(uint16_t) * lj92_components;

                            if(ret == LJ92_ERROR_NONE)
                            {
                                if(verbose)
                                {
                                    print_msg(MSG_INFO, "    LJ92: Decompressing\n");
                                    print_msg(MSG_INFO, "    LJ92: %dx%dx%d %d bpp (%d bytes buffer)\n", lj92_width, lj92_height, lj92_components, lj92_bitdepth, out_size);
                                }
                            }
                            else
                            {
                                print_msg(MSG_ERROR, "    LJ92: Failed (%d)\n", ret);
                                goto abort;
                            }
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: dfort on May 18, 2017, 03:21:49 PM
Or someone with a windows setup could help you compile a version.

Or you could cross compile a Windows version on your Mac. I compiled a Mac and Windows version from that branch, check if this works:

https://bitbucket.org/daniel_fort/magic-lantern/downloads/ml-dng-unified_8_-Mr_B_compress_ver2.zip

For now it will be easier for me to install a hackintosh.

You would still need to cross compile to get a Windows binary. In fact on a Windows system you need to set up either Cygwin or a Linux virtual machine and cross compile to get a binary that works on any Windows machine.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: dmilligan on May 18, 2017, 03:28:56 PM
Please upload a sample file. If you dig round mlv_dump code it says following.
Error related to int ret.  Frame size?

The data is corrupted. See lj92 header:

Code: [Select]
LJ92_ERROR_CORRUPT = -1
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on May 18, 2017, 03:59:03 PM
@ dmilligan

The data is corrupted.
There is no way to retrieve anything ?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: rob_6 on May 18, 2017, 05:10:54 PM
Just wanted to let you guys know that I have been recording without any crashing for quite some time with the latest build. I am using the settings recommended in hjfilmspeed's video at 3k 2.35 and it works so well! I am using a 256GB Lexar 1066x compact flash card and can record about 44-46 minutes  in 12 bit 3k reliably and it even closes the file once the card is full without crashing! Amazing work A1ex and everyone else that helped make this possible. I literally feel like I have a new camera! The only thing to be careful of is the compression and not letting the camera overexpose which seems to stop the recording due to the increased recording bitrate.

I have one question. As we know, if I half press the shutter while in the 5x crop mode it zooms in and allows me to focus with a normal live view playback. I usually half press the shutter before hitting the record button to get the focus right. But, I have been successful pressing the shutter half way while recording to get focus during the recording, too. Does pressing the shutter half way while recording put extra work on the camera which could cause the recording to stop early? So far, it seems to work fine. Just curious if anyone else has tried this technique or has any insight!

Thanks again to all!

Rob
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: g3gg0 on May 18, 2017, 07:23:40 PM
@ dmilligan

The data is corrupted.
There is no way to retrieve anything ?

without a sample we could only clearly answer: "don't know"
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: g3gg0 on May 18, 2017, 07:27:08 PM
you could also apply this (totally untested) patch - maybe it was only one frame that was faulty.
Code: [Select]
diff -r 8b31c98380d5 modules/mlv_rec/mlv_dump.c
--- a/modules/mlv_rec/mlv_dump.c Tue May 02 01:28:48 2017 +0200
+++ b/modules/mlv_rec/mlv_dump.c Thu May 18 19:26:05 2017 +0200
@@ -2958,7 +2958,7 @@
                             else
                             {
                                 print_msg(MSG_INFO, "    LJ92: Failed (%d)\n", ret);
-                                goto abort;
+                                goto skip_block;
                             }
                             
                             free(compressed);
@@ -3763,6 +3763,7 @@
                 lua_handle_hdr(lua_state, buf.blockType, "", 0);
             }
             
+skip_block:
             file_set_pos(in_file, position + buf.blockSize, SEEK_SET);
         }

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on May 18, 2017, 07:31:27 PM
...Does pressing the shutter half way while recording put extra work on the camera which could cause the recording to stop early? So far, it seems to work fine. Just curious if anyone else has tried this technique or has any insight!

I had similar thoughts. It seems to work fine on my end but sometimes I suspect that it may cause the recording to stop though not as often. Still trying to reproduce this but it could very well be related to the compression scheme from being slightly overexposed.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on May 18, 2017, 07:48:34 PM
I have no clue how to modify mlv_dump.

Here is one of the files

https://drive.google.com/open?id=0B-d8ARtc7xwWWkltLUpUcTJmams
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: stevefal on May 19, 2017, 07:23:12 PM
(4K + dual_iso)/2 = 1920x800 @ 14+ stops DR w/o aliasing?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: dmilligan on May 19, 2017, 08:05:45 PM
Not without aliasing, dual ISO introduces aliasing in the areas where the two ISOs don't overlap (highlights and shadows).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on May 19, 2017, 08:19:10 PM
A sensel cannot be read out at two different ISOs at the same time (so you are going to have some missing data in either spatial or temporal domain). In the first case you'll get aliasing; in the second one, you'll get motion artifacts.

This might be doable though: http://www.magiclantern.fm/forum/index.php?topic=19315.0
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: stevefal on May 19, 2017, 09:17:06 PM
Not without aliasing, dual ISO introduces aliasing in the areas where the two ISOs don't overlap (highlights and shadows).

I assumed that halving resolution would eliminate/reduce aliasing, since the highlight/shadow lines would no longer be doubled.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: tit_toinou on May 20, 2017, 03:58:02 PM
Thanks for this module it's awesome !
Has anyone succeeded in getting 25 FPS 2.35 4K ? I'm stucked at 12.5 FPS (I have a good CF card)..

Here's what I've done with it https://www.magiclantern.fm/forum/index.php?topic=19568 (https://www.magiclantern.fm/forum/index.php?topic=19568)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Anntria on May 21, 2017, 04:10:09 PM
Hi everyone.

I have problems with sound record on 1.2.3 firmware version, experimental build.
Got 4 modules on - crop rec, dual iso, mlv lite, mlv play

What am i doing wrong and how to solve it?

Kind regards, Anna
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Savely on May 21, 2017, 06:07:18 PM
Is there any way for Win users do decode MLVs from crop_mode to DNG sequence except RawFlow?

In Dual ISO mode in crop_mode processed through RawFlow and Dual ISO Processor files sometimes corrupt. I did some tests but it seems randomly with this pattern in resulted files: http://joxi.ru/Dr8oB9wh4Eol8A.jpg and it's RawFlow.exe bug cause DNGs from MLVP (on old, not crop_mode build though) processed through Dual ISO Processor seems good.

So again, is there some way for Windows users to decode mlv files of crop_mode to DNGs except RawFlow? (MLVP doesn't work with crop_mode MLVs)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: ewinemiller on May 21, 2017, 11:04:57 PM
There is no sound in crop_rec experimental builds. mlv lite does not support it. Use an external recorder. The Zooms are nice.

Hi everyone.

I have problems with sound record on 1.2.3 firmware version, experimental build.
Got 4 modules on - crop rec, dual iso, mlv lite, mlv play

What am i doing wrong and how to solve it?

Kind regards, Anna
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Anntria on May 22, 2017, 12:43:49 AM
There is no sound in crop_rec experimental builds. mlv lite does not support it. Use an external recorder. The Zooms are nice.

Do i understand correct that there are no time codes for crop_rec experimental builds as well?
So mixing of external sound with footage is a sort of challenge as well, or am i wrong?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: ted ramasola on May 22, 2017, 06:34:41 AM
Do i understand correct that there are no time codes for crop_rec experimental builds as well?
So mixing of external sound with footage is a sort of challenge as well, or am i wrong?

You can use a slate. or a simple clap your hands to sync it with ext recorder.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on May 22, 2017, 06:49:02 PM
you could also apply this (totally untested) patch - maybe it was only one frame that was faulty.
Code: [Select]
diff -r 8b31c98380d5 modules/mlv_rec/mlv_dump.c
--- a/modules/mlv_rec/mlv_dump.c Tue May 02 01:28:48 2017 +0200
+++ b/modules/mlv_rec/mlv_dump.c Thu May 18 19:26:05 2017 +0200
@@ -2958,7 +2958,7 @@
                             else
                             {
                                 print_msg(MSG_INFO, "    LJ92: Failed (%d)\n", ret);
-                                goto abort;
+                                goto skip_block;
                             }
                             
                             free(compressed);
@@ -3763,6 +3763,7 @@
                 lua_handle_hdr(lua_state, buf.blockType, "", 0);
             }
             
+skip_block:
             file_set_pos(in_file, position + buf.blockSize, SEEK_SET);
         }

Any chance to review the corrupted mlv file I placed on my Google Drive ?
Thanks in advance
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: g3gg0 on May 22, 2017, 09:46:00 PM
download latest mlv_dump (https://builds.magiclantern.fm/jenkins/view/Utilities/job/mlv_dump_4k/lastSuccessfulBuild/artifact/src/modules/mlv_rec/mlv_dump.exe) and add option "--skip-block 9" to your commandline
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on May 23, 2017, 04:55:36 AM
Awesome !!!
It heals the problem.
I tried it on other corrupted mlv and it works.
Thanks for your wizardry  ;)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on May 23, 2017, 04:35:39 PM
Great with all the latest "autopsy" settings @g3gg0.
--relaxed  8)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: g3gg0 on May 23, 2017, 06:26:50 PM
glad to hear it helps :)
any further suggestions are welcome.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Savely on May 23, 2017, 10:14:54 PM
Is there any way to merge DNG-sequence like AviSynth with RAW.mov do? I'm trying to shoot HDR video with crop_mode but AviSynth work just with mov files. Anyone now How to merge DNG frames with different expo to sequence with one corrected sequence?

*I'm on Windows
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on May 23, 2017, 10:52:29 PM
This is a bit of topic but regarding HDR there is one open source tool that merges dng files
https://github.com/jcelaya/hdrmerge
I tried this for a period of time but the caveat back when testing was that the tonemapping algorithms are mainly for stills. When merging movie sequences you get flicker since tonemapping will vary from image to the next.
What is not that much of topic is to elaborate about mlv_dump being able to merge hdr files coming from MLV footage. I tried some crude examples with averaging code working with consecutive frames. Bouncyball helped me out here with coding but simple averaging would produce pinkish highlights. Now averaging(tblend filter) code in ffmpeg is in another ballpark when looking at the merged footage that I played around with in MLP and cr2hdr.app. No pinks, just a good looking merged file.
Of course any of such changes would surely take a lot of time to accomplish and put into mlv_dump but since the question arised about HDR I thought I just put it out there.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: sylvain_c on May 23, 2017, 10:56:16 PM
crop_rec_4k Apr29 not working here. I have followed exactly the video tutorial, but it result in random multicolored pixels (viewed in Mlrawviewer and raw2cdng1.7.9). also tested with and without fps overide.

dng sample:
https://mega.nz/#!DocmzRJJ!-A96kskYpm7ZqzFynl_4MZH9yd0cfc9O9bNCcrPlM5k

Edit: Ok sorry, I succeed with mlv_dump.exe
Title: Issue with crop_rec_4k mlv_dump
Post by: Danne on May 24, 2017, 02:21:02 PM
I would like to focus on 10bit12bit files in recent mlv_dump again. Seems the fix is very close but if it the problem passes unnoticed now it will probably be buried into other functions adding up in time.
I test mlv_dump with the following command(latest crop_rec_4k code)
mlv-dump --dng INPUT.MLV

12bit test file
https://bitbucket.org/Dannephoto/magic-lantern/downloads/12bit.MLV


The last commit that will export 10bit12bit files correctly is this
1738cb0
https://bitbucket.org/hudson/magic-lantern/commits/1738cb06e2652d9751f9407490de507fe30ef520?at=crop_rec_4k


Next commit is where things are getting wrong with 10bit12bit footage. Compiling from this commit will cause segmentation fault 11, however the next commit partly fix this but will export frames with missing information(see examples below)
92d1b4d
https://bitbucket.org/hudson/magic-lantern/commits/92d1b4d2958975f50ba428853545f64db8a61205?at=crop_rec_4k

Following commit, "re-adding 14bpp enforcement for DNG" produces black 10bit/12bit dng files cause it won´t fill the frame entirely(see examples). Dcraw exports the file and it shows where the image information is missing.
8b31c98
https://bitbucket.org/hudson/magic-lantern/commits/8b31c98380d5dc27573886d0f84ac7ed0c5d0f2b?at=crop_rec_4k

Looking at the code changes I see a lot of changes around frame_buffer_size, read, prev_frame_size so I guess somewhere in there this could be fixed. Unfortunately I couldn´t get this right with the example file provided but hopefully a dev could take a look or give a hint on how to fix it.


dcraw -T INPUT.dng gives:

Working commit
1738cb0
(https://s4.postimg.org/cc9w4w10d/Screen_Shot_2017-05-24_at_14.21.46.png)



Not working
(https://s4.postimg.org/wide3rwnx/Screen_Shot_2017-05-24_at_14.19.39.png)

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kharak on May 24, 2017, 03:49:27 PM
Can I update from 113 to 123 Canon firmware, if I install Apr. 29 Build? Or do I have to update firmware via EOS utility? I am currently on 113... Its 2 years since I did firmware update  :-\
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Walter Schulz on May 24, 2017, 03:58:08 PM
You can install *Canon* firmware 1.2.3 using Canon menu. If you do this you have to run ML builds for *Canon* firmware 1.2.3.
ML is not firmware.
All builds should be available for 1.1.3 and 1.2.3. There is no need to upgrade Canon firmware if you don't have to for other reasons.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kharak on May 24, 2017, 04:37:10 PM
I meant if the 123 firmware is also included in the Apr 29 build. As in i could update to 123 from 113 by updating the build and simaltaneously update the firmware?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Walter Schulz on May 24, 2017, 04:40:26 PM
Best way to semi-brick/stalling your cam.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Savely on May 24, 2017, 04:54:53 PM
@Danne thanks, you clarify issue pretty well. Just to remind you and those who develop some software solutions within magic lantern - currently mlv_dump didn't work properly with Dual ISO MLVs from crop_mode. Sometimes frames corrupted with looking-like-labirynth pattern and it seems randomly.

So it seems that currently there are just one reliable (but long) workflow for HDR-video when you shoot in crop_mode - MLV to DNG-sequence then DNG-sequence to MOV then MOV with Avysinth (in HDR-workflow pack) processed to A, B and C jpg-sequence then this C-sequence again assembled in some final video format.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kharak on May 24, 2017, 05:36:06 PM
Best way to semi-brick/stalling your cam.

So the simple answer is: No. Thank you
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: g3gg0 on May 24, 2017, 10:46:15 PM
12bit test file
https://bitbucket.org/Dannephoto/magic-lantern/downloads/12bit.MLV

where does that file come from? mlv_lite?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: g3gg0 on May 24, 2017, 11:14:15 PM
reason for asking is this:
Code: [Select]
Block: RAWI
  Offset: 0x00000034
  Number: 1
    Size: 180
    Time: 212.444000 ms
    Res:  960x540
    raw_info:
      api_version      0x00000001
      height           1181
      width            1888
      pitch            2832
      frame_size       0x003B8A48
      bits_per_pixel   12
      black_level      512
      white_level      4050
      active_area.y1   26
      active_area.x1   152
      active_area.y2   1181
      active_area.x2   1886
      exposure_bias    0, 0
      cfa_pattern      0x02010100
      calibration_ill  1

Block: VIDF
  Offset: 0x0000027c
  Number: 9
    Size: 3344624
    Time: 550.688000 ms
   Frame: #0000
    Crop: 536x332
     Pan: 536x333
   Space: 0
   depth: 12 -> 14, size: 777600 -> 907200 (116.67%)
   black: 512 -> 2048
   white: 4050 -> 16200

Res:  960x540 -> 777600 bytes frame size.
but the frame seems 1181x1888 -> 3344592 bytes.

looks like the whole raw buffer was written instead of the specified frame size.
patching the fields results in proper frames.

any idea where this file came from?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on May 24, 2017, 11:43:14 PM
It's an early build put into a 600D. Wonder if it was called raw_rec(mlv_lite). Stupid of me not to download a later build when testing. Right now I don't have the camera nor the firmware version until monday.
On another note. Is recent 10bit12bit files working with latest crop_rec_4k like it should? Last I tried it wouldn't work. If so, I guess there is no issue. Don't have any cam here unfortunately to verify.
By the way. That particular file will process with earlier builds. Guess there is some other buffer detectors in previous code?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on May 25, 2017, 12:12:34 AM
Just tested with mlv files on my eos m coming from mlv_lite and regular mlv_rec and both transcodes 10bit12bit with wrong buffer(black bottom). Downloaded experimental raw_video_10bit_12bit 30 april build. Ran latest mlv_dump (crop_rec_4k) code.

Here is the thing. If I replace
Code: [Select]
if(read_size != (int)frame_buffer_size)with
Code: [Select]
if(read_size > (int)frame_buffer_size)later 10bit12bit files will process correctly. Is this the fix we could use? It still won´t work to compress 10bit12bit files(mlv_dump -c setting produces black output) but I guess they are already compressed being 10bit12bit so maybe compression for lower bits should be excluded?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: erek on May 25, 2017, 09:30:09 AM
Here's my first 3072x1320 24 fps recording:

https://www.youtube.com/watch?v=B6mEqWalnJY


so with 14-bit lossless 3K i am seeing about 6.5 minutes of record time on a 64GB CF card... does this seem accurate?  also what is a recommended Aspect Ratio?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: g3gg0 on May 25, 2017, 09:43:33 PM
Here is the thing. If I replace
then you will get crashes :)

but thanks for your testing. just pushed a fix (https://bitbucket.org/hudson/magic-lantern/commits/9b214f11098a3be752ca8a5cfe3a17584b3c23b8) that should fix that issue.
after converting bit depth, frame size was updated, but not data size.

reason for having both is obvious when compression is used. frame size wont change with or without compression, whereas the resulting binary data varies in size.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: g3gg0 on May 25, 2017, 09:48:39 PM
Here's my first 3072x1320 24 fps recording:
so with 14-bit lossless 3K i am seeing about 6.5 minutes of record time on a 64GB CF card... does this seem accurate?  also what is a recommended Aspect Ratio?
3072 * 1320 * 24 * 60 * 6.5 * 14/8 * 0.58
gives ~38GiB
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on May 25, 2017, 10:35:45 PM
@g3gg0
Code: [Select]
                        frame_buffer_size = new_size; :D
Tested, working. Thanks a million.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: lostfeliz on May 26, 2017, 03:54:40 AM
@erek, your shadows look a little noisy. maybe you underexposed or maybe you had a high ISO?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: bazidl on May 26, 2017, 09:51:46 PM
What program are you guys using post production (after exporting via mlv_dump). I would normally use Premier pro cc 2015 but it doesn't recognize the .dngs whenever I use any new feature settings...although I can view the individual .dngs fine in photoshop or any image viewer.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on May 26, 2017, 11:19:08 PM
Mlvfs or cr2hdr.app will produce cdng files
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: bazidl on May 27, 2017, 04:35:30 AM
I tried cr2hdr.app before and still got the same message as using mlv_dump. I'm prob missing a setting that allows for the proper export. Regardless just tried Mlvfs and it worked great!

Thanks for the help
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on May 27, 2017, 05:51:20 AM
I see. What settings did you use for it not to work? I would like to reproduce this issue. Do you have a sample file to test with?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on May 27, 2017, 06:37:10 AM
@g3gg0
Code: [Select]
mlv_dump -c -c is not working healthy right now. dng files comes out with 0 black and white level for instance. I narrowed the issue down to this change.
Code: [Select]
                            if(new_depth != old_depth)reverting back to this and it works again.
Code: [Select]
                            if(new_depth)

There is also one of these beauties left
Code: [Select]
LJ92: Failed
which could go into nifty --relaxed to let mlv_dump produce compressed dng files.

Code: [Select]
                                /* set new compressed size and copy buffers */
                                frame_buffer = realloc(frame_buffer, compressed_size);
                                assert(frame_buffer);
                                memcpy(frame_buffer, compressed, compressed_size);
                                frame_buffer_size = compressed_size;
                            }
                            else
                            {
                                print_msg(MSG_ERROR, "    LJ92: Failed (%d)\n", ret);
                                goto abort;
                            }
to
Code: [Select]
                                /* set new compressed size and copy buffers */
                                frame_buffer = realloc(frame_buffer, compressed_size);
                                assert(frame_buffer);
                                memcpy(frame_buffer, compressed, compressed_size);
                                frame_buffer_size = compressed_size;
                            }
                            else
                            {
                                print_msg(MSG_ERROR, "    LJ92: Failed (%d)\n", ret);
                                if(relaxed)
                                {
                                    goto skip_block;
                                }
                                goto abort;
                            }
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: g3gg0 on May 27, 2017, 09:25:34 AM
first one done, really have to rework some code to make such changes happen just in one point.

the second thing, does this ever happen?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on May 27, 2017, 09:39:06 AM
Quote
the second thing, does this ever happen?
You´re right. I´m working a version which turns on the compression switch when selecting -c -c. Not a problem in crop_rec_4k version.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on May 27, 2017, 07:03:08 PM
@g3gg0 @Danne Great job guy's.. ya'll working. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: eraizgn on May 28, 2017, 04:09:20 AM
Do I missing something? My preview like:
https://gyazo.com/ed2429dc5904467c229dab961b1c4e6d
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: extremelypoorfilmaker on May 28, 2017, 09:33:44 AM
Hi everyone,

I was fiddling around with the UHD 1:1 setting of the crop_rec module
The data I was recording was 14bit lossless, iso 1600.
It was recording continuously until I pointed the camera at the PC screen and this happened! :D

(http://i64.tinypic.com/eb2zk3.jpg)

I still have the file if any of you would like to inspect it.
p.s.
I am sure it happened because of the high ISO and the sharp change from a dark corner of my room and the PC screen.
I read that 14bit lossless works best at low iso and with not too bright scenes, therefore I believe that was what made magic lantern to do this. I restarted the camera removing the battery and all come back to normal.

p.p.s.
When that happens, what button do I press to make it go away without having to restart the camera? :D
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on May 28, 2017, 03:17:02 PM
Been testing out these option in the crop_rec_4k
* 1920x960 @ 50p (both 1:1 crop and full-frame - 3x3 pixel binning) - continuous*)
* 1920x800 @ 60p (same as above)  - continuous*)
* 1920x1080 @ 45p and 48p (3x3 binning)  - continuous at 45p

Works promising with HDR footage.

I noticed compressed raw and lossless 10/12 bits can´t be selected only 14bits lossless. Searched around but I couldn´t find out the reason why other than from a live view message and some posts over here http://www.magiclantern.fm/forum/index.php?topic=19300.msg183763#msg183763
 Is it simply not possible for 10/12bit and compressed raw with the binning modes?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on May 28, 2017, 08:17:28 PM
A potential bug. Can somebody else but me reproduce following issue.

- Set camera to mlv_lite and select 1920x960 @ 50p in the crop_rec menu. Camera set to PAL.
- Record a short sequence and stop filming. Now exit liveview closing it the normal way.
- Now try to switch into liveview again.

From now on all I get is a black screen. Only way to get back into liveview is to set 1920x960 @ 50p in crop_rec menu to off, then open up liveview again and reselect 1920x960 @ 50p again.

5D mark III (1.1.3)
crop_rec_4k.2017Apr29.5D113

*related to 720p movie mode. When in 1920 no more black screen.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on May 28, 2017, 08:24:23 PM
@Danne it happened to me as well.. can't remember which build exactly, but it was one from April.  What I did was to format my SD card again and reinstall Magic Lantern again.  Sometimes when you use 50p 60p mode something breaks in the RAW modules I think not sure but ask @a1ex. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on May 28, 2017, 08:31:02 PM
Yes, it´s related to 720p movie mode. I updated my question. Thanks for getting back.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kharak on May 29, 2017, 07:40:48 AM
Been testing out these option in the crop_rec_4k
* 1920x960 @ 50p (both 1:1 crop and full-frame - 3x3 pixel binning) - continuous*)
* 1920x800 @ 60p (same as above)  - continuous*)
* 1920x1080 @ 45p and 48p (3x3 binning)  - continuous at 45p

Works promising with HDR footage.

I noticed compressed raw and lossless 10/12 bits can´t be selected only 14bits lossless. Searched around but I couldn´t find out the reason why other than from a live view message and some posts over here http://www.magiclantern.fm/forum/index.php?topic=19300.msg183763#msg183763
 Is it simply not possible for 10/12bit and compressed raw with the binning modes?

What build are you running? I cant get continues in 1920x800 60p. Neither on 113 or 123. especially when bright and lots of details, it will even crash at times saying "compressed size larger than uncompressed" or something like that. Been using the 60p mode a lot this last week.

And a question for @A1ex . Is there possibility of getting bit higher resolutions in the 60p mode in the future? 800 vertical pixel is a bit of a squeeze, i love the look, but not every project deserves the cinemascope ;) Perhaps if you get spanning to work aswell?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on May 29, 2017, 08:09:36 AM
1920 x 960 50fps is continuous. Early stops with other ones. Did not have the time to test extensively. I'm pretty amazed about the 1920x50 fps option.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on May 29, 2017, 08:29:24 AM
I confirm the bug with the black screen when 1920x800 @ 60FPS
It takes some patience and some assistance to recover Liveview back.
I press Canon Menu a couple of times and then ML button.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kharak on May 31, 2017, 11:38:24 AM
What are the options for Windows users, to batch convert these MLV's?

I was using this Code:
Code: [Select]
@echo off
md "%~n1"
"C:\XXXXXX\Video Program\MLV_Dump on Steroids\mlv_dump.exe" -o %~n1\%~n1_ --dng %~n1.MLV

To Batch convert Selected MLV files in a folder. But after replacing the mlv_dump with latest, I can not get it to Batch Convert anymore, it will only convert one and then exit.

I don't know of any other converter for these new MLV's?

Please help, I have so incredibly many MLV's to convert.

MLVFS, does not work for me, as AE will only take in to account the changes of the First Frame from ACR.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: spe42 on June 01, 2017, 03:02:41 AM
Was testing out the most recent build and had a slew of errors when I tried recording video (there is one possible mistake I made, reinstalling ml now to see if I can reproduce error). I tried to start recording, and it did so for a few seconds, but then said card full. I didn't realize at the time, but it didn't switch over from cf to sd card, so I couldn't test to see if it would still have worked if I had switched manually. That said, since that happened I have been experiencing two kinds of errors. One, when I go to record video if I am using anything above 3k resolution a "script" will run indefinitely, saying that "mean is incorrect" and something about black level, and a range of numbers in the 2000-3000s. At the moment, I am able to reproduce this, but am unsure if reinstalling ML will fix. I have crash logs, etc. as well. The main thing that is also happening, is that I kept seeing text saying "video end not working". When I ran the self test, this message popped up quite a bit. Any thoughts? Should I provide any of the stuff I've got, crash logs and such?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on June 01, 2017, 03:01:42 PM
Quote
MLVFS, does not work for me, as AE will only take in to account the changes of the First Frame from ACR.

Same for me... Searched for a solution but unable to find it.... :(
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on June 01, 2017, 04:41:08 PM
Magic Lantern updates may generate incompatibilities with existing supporting tools. This is natural.
Developers of ML and ML tools are not gathered under the same roof and develop in synchronization.

ML developers specified in the first page that the only way to get DNG files is using mlv_dump
that is available in the download / experiments page.
Insisting to use last years tools, or older, together with today's ML build, is a waste of time.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on June 01, 2017, 04:44:54 PM
Quote
MLVFS, does not work for me, as AE will only take in to account the changes of the First Frame from ACR.
You could build xmp sidecar files of every dng and after effects will treat every dng as unique despite being a sequence.
The problem is not mlvfs but AE.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kharak on June 01, 2017, 06:35:51 PM
You could build xmp sidecar files of every dng and after effects will treat every dng as unique despite being a sequence.
The problem is not mlvfs but AE.

Hey Danne,

How do i create xmp sidecars in AE? And wont that take a lot of time? Not sure how it works, but i am imagining that ACR has to go through each and every DNG and save its xmp.. I currently have about 200k frames to process.

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on June 01, 2017, 07:19:36 PM
You can build xmp from lightroom or even faster would probably be with exiftool through scripting.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kharak on June 01, 2017, 08:53:27 PM
Was affraid you'd bring something like that up. I don't know how to do scripts or coding of any sorts. And incorporating yet another Adobe product in to my workflow is not what I want to do. I had a slow, but steadily working workflow "Send to Mlv_dump" batch convert - Transcode to Log intermediate in AE - Edit and Colour Grade in Resolve - Done.

Danne, you are good at coding. The code I posted above, it is the same I used before with the older version of mlv_dump. I assume the code is right, as it worked before. Could it be something with the mlv_dump version that is preventing it from opening the next MLV ?
 I have tried deleting all MLV_dump, change locations of the mlv_dump, tried having no Blanks in the Location to Avoid the "", which I understood was for Blanks in locations. But nothing works. I have not tried going back to the older mlv_dump, as i really like that I now can play the DNG's in MLRawViewer, its actually a Must-Have that I can do that.



Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on June 01, 2017, 10:06:25 PM
I have no idea. Don't you're command prompt tell you why it breaks?
Suggestion.
1 - try an older mlv_dump file and see if that works
2 - upload a problematic file if any.
3 - check with one of the windows threads
4 - test with latest mlv_dump and specify --relaxed which raises tolerance levels regarding broken/missing blocks.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: gnezdo.kukushki on June 02, 2017, 12:22:35 PM
Hello! Here is some test of 3k raw and slow-mo RAW!
https://youtu.be/Ei3g_-YxtUc (https://youtu.be/Ei3g_-YxtUc)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: loknar on June 02, 2017, 12:36:18 PM
I noticed, that in Experiments/"4K raw video recording" appeared builds "magiclantern-startup-log.2017May31....zip" for a lot of cameras, what is the purpose of these?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on June 02, 2017, 09:09:14 PM
Hm... looks like Jenkins found the cellar keys on the build server, and noticing that nobody was looking at him, he had a little too much wine :)

Thanks for reporting.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: loknar on June 02, 2017, 09:24:01 PM
Well, early christmas came to mind  :D
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on June 03, 2017, 12:09:37 AM
Hm... looks like Jenkins found the cellar keys on the build server, and noticing that nobody was looking at him, he had a little too much wine :)

Thought this was either a joke or a typo by Jenkins...  :P

(https://c1.staticflickr.com/5/4224/35018847766_78eea78879.jpg) (https://flic.kr/p/VmuKpm)

Anyway I went ahead and downloaded a build for 7D.203 just because...

Here are the start up logs in case they are of any help or give out hints:

https://mega.nz/#!qhNF1Q7I!qmnhGAKHLktvQ-2FiT692qjpLL9wDvji1Jv68XcGbV4
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Teamsleepkid on June 03, 2017, 03:53:30 AM
worst tease ever. I'll go back to being depressed now.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on June 03, 2017, 10:59:43 AM
There was no teaser, to begin with. It was a bug (in the build server configuration).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: jankrueck on June 03, 2017, 03:19:08 PM
Hey there.
I know you have plenty to do sir, but do you think it could be possible to throw in a very experimental sound build?
Don't care if it will be bulky to use. I have a projekt where its impossible to use a clap every time ;(

I thought about something like proxxy wave files on SD card.


sorry to annoy you again @A1ex  :P
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on June 03, 2017, 03:39:18 PM
I'd be happy to review a pull request with this feature ;)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: jankrueck on June 03, 2017, 04:58:38 PM
haha yeah. But when I say "I thought of" it literaly means "I thought of"
I have zero progamming skills beside html and css :D :D :D
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on June 03, 2017, 05:33:47 PM
Understandable, but I also have good reasons to believe the ~5000 downloads of this build were not done by you alone :)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: g3gg0 on June 04, 2017, 12:26:31 AM
I'd be happy to review a pull request with this feature ;)

having patches on my disk.
wont take too long.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: jankrueck on June 04, 2017, 12:32:30 AM
Understandable, but I also have good reasons to believe the ~5000 downloads of this build were not done by you alone :)

please help me to understand your sarcasm, as I'm not a native speaker ;)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Savely on June 04, 2017, 12:49:28 AM
I guess he mean (not native speaker though too) ir's not just you who asked something. I did too but dropped almost instantly since nobody responded. Aaand... A miracle! Now I know almost everything that once puzzled me. Actually thanks Danne for Dual ISO problem with crop_mode since it's he's tests with my footage showed me that it is not bad camera settings problem but rather flawed Dual ISO processing software
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: extremelypoorfilmaker on June 04, 2017, 08:58:13 AM
please help me to understand your sarcasm, as I'm not a native speaker ;)

What makes me laugh is, I suspect both you and A1ex have a language in common, but out of kindness, write in english so everyone can understand :D
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Walter Schulz on June 04, 2017, 09:04:26 AM
Nah. Their native languages are quite different.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: rob_6 on June 04, 2017, 10:21:24 AM
If they added sound to this build I would probably buy another 5D! Image quality is incredible! Sound is my only wish. So that's 2 of us out of 5000! :)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: g3gg0 on June 04, 2017, 10:49:40 AM
I am the german one if you refer to this language ;)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: extremelypoorfilmaker on June 04, 2017, 11:27:06 AM
I thought A1ex was german as well! :O :D
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Jazzepisode on June 04, 2017, 05:07:00 PM
In the end we are all german and everybody is just writing english, because they think the others don't understand :D

Guys I have on quick question. Most likely im bringing something up, that has been answered somewhere else, but I couldn't find it. Why is it not possible to shoot 12bit or 10 bit raw in modes with higher vertical resolution like 3k mode or 1920 60 fps mode? Is this a limitation in anyway or has it just not yet been implemented?

Anyways you are doing an awesome job here. Thank you very much!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: jankrueck on June 04, 2017, 05:33:25 PM
Na wenn das so ist, dann schreib ich mal in meiner Muttersprache :D
Also G3gg0 wie erstell ich ne pullrequest damit Alex glücklich wird?

Meinste wave als "proy" auf die SD zu schreiben hat Potential?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Savely on June 05, 2017, 05:12:32 AM
* 4096x1440 @ 25p (1:1 crop)

Is anyone could find this mode? I've check all modes and just can't find it. Is it exist?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on June 05, 2017, 06:15:03 AM
* 4096x1440 @ 25p (1:1 crop)

Is anyone could find ершы mode? I've check all modes and just can't find it. Is it exist?

Try changing Canon Video System to PAL/NTSC
or try changing 1080/720 and see how it affects your vertical size.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Savely on June 05, 2017, 08:25:47 AM
Try changing Canon Video System to PAL/NTSC
or try changing 1080/720 and see how it affects your vertical size.

Thanks, changing 1080/720 does help.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Savely on June 05, 2017, 10:45:19 AM
As I understand card_spanning doesn't work on experimental crop_mode build so we are restricted with ~100mb/s so there are no hopes for sustainable 4k at 24fps (half-fps or redused vertical resolution is not an option). Or it is possible to add SDcard speed and up a bit writing speed for sustainable 4k >5sec rec?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: etienne on June 05, 2017, 02:59:38 PM
+1 for the sound also i currently have a work around which is really cumbersome but that works with Full HD defintion.

I activate proxy record
i choose the SD card in ML menu as the prefered card for writing.
In canon menu I set the IPB as codec (all-i gives me more crashes but this is probably due to a weak SD card). SD mode 640 does works aswell.
In the End you get 2 files : 1 proxy video with sound on SD and 1 Mute MLV on CF.

The problem with this techniques is that the 2 files does not start in SYNC (h264 always start first) so you need to resync you audio with the MLV and run an export of the audio to match the MLV sequence.
It's like a clap sync without the clap but as heavy to deal with though. The benefits i that you get a proxy file you can start working with and find a little hand to do the dirty work of exporting matching audio.

The problem is that it does not work with 3K Mode as there is no preview to make image match

So if getting sound into the MLV is still hard to manage maybe enabling any kind of poor preview to the 3k mode would help to do the trick in meanwhile. or even better a way to have the proxy start in sync with the MLV
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: jankrueck on June 06, 2017, 03:24:36 PM
+1 for the sound also i currently have a work around which is really cumbersome but that works with Full HD defintion.

I activate proxy record
i choose the SD card in ML menu as the prefered card for writing.
In canon menu I set the IPB as codec (all-i gives me more crashes but this is probably due to a weak SD card). SD mode 640 does works aswell.
In the End you get 2 files : 1 proxy video with sound on SD and 1 Mute MLV on CF.

The problem with this techniques is that the 2 files does not start in SYNC (h264 always start first) so you need to resync you audio with the MLV and run an export of the audio to match the MLV sequence.
It's like a clap sync without the clap but as heavy to deal with though. The benefits i that you get a proxy file you can start working with and find a little hand to do the dirty work of exporting matching audio.

The problem is that it does not work with 3K Mode as there is no preview to make image match

So if getting sound into the MLV is still hard to manage maybe enabling any kind of poor preview to the 3k mode would help to do the trick in meanwhile. or even better a way to have the proxy start in sync with the MLV

yeah nice idea. but this realy "sounds" (haha) like too much work.
getting a clap with timecode and external recorder will make it a lot easier. (even though it is more expensive)

looking forward to any solution wich brings sound to 3k
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on June 08, 2017, 05:25:33 AM
Sound is good.... But understanding​ live view and compression are probably more important I think.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Savely on June 08, 2017, 06:45:02 AM
And no one care about card spanning idea to make (or at least, to try) crop_mode sustainable with 4096 24fps and reasonable aspect ratio? Because now we have essentially half 4k. If it's 24fps so it have some adsurd aspect ratio - like, just band throughout black screen. If we have descent aspect ratio - it's just 12fps... If we have both - it's just 3 seconds of record... 4k, fps, aspect ratio, continuance - choose one. What one can shoot with it?

Okay, I'm not pressing, but can we at least discuss this idea? It is even possible?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: MartijnR on June 08, 2017, 01:12:46 PM
Hi there, I am in the middle of a little film project (although I have some time to complete it), and using 10 bit (2017-01-12 / crop3x.2017Jan.5D113) to focus on a water droplet (around 3K). In my setup it is good to make the final film 1080 (also to have the water droplet as big as possible). It is shot inside, and at 1600 iso there is already quite some noise (but I think manageable). I would be very happy if somebody could tell me if it is worth to put the new crop version on it, since I might have to change the setup of my project and lose some time.

- Will the lossless 12 bit (or 14 bit lossless) have less noise and better image quality compared to 10 bit (nonlossless), and more than 3K at 24fps? (there is a lot of movement in the imagery so that might limit the record time)
- Will the new version also have the option to choose the 3x crop mode (in addition to the around 1.6x crop mode)?
- Will the (older) 3x crop mode have less image quality considering (I assume) it is using a smaller portion of the sensor?

I might have to do these experiments myself, but it would be great if someone already knew some of the answers and help me safe some time. Thanks!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on June 08, 2017, 03:15:28 PM
@MartijnR the new 4k crop rec mod is way nicer. Your crop area will be centered, and you can't get better preview options. It might be a little different to use though. Also it is 1:1 resolution like before. Your crop factor depends on the mode/resolution you choose.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: MartijnR on June 08, 2017, 05:12:44 PM
@hjfilmspeed Thanks for letting me know! Due to the nature of the project, this will save me a lot of time :) I have seen your tutorial (thanks) and was wondering if you would recommend the 29 april built, or just the latest built?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on June 08, 2017, 06:59:08 PM
I should probably update the tutorial. I would say try the newest build but make sure you use the newest MLV_Dump too.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: budafilms on June 09, 2017, 06:47:01 AM
@MartijnR

Wich lenses are you going to use?
Take in mind the crop factor because it's huge.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: lostfeliz on June 09, 2017, 06:52:29 AM
@MartijnR If you're shooting indoors, can't you adjust your lighting in order to lower you ISO. You should get less noise.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: MartijnR on June 09, 2017, 10:25:26 AM
@hjfilmspeed Thanks for the advice!

@budafilms Probably Zeiss 35/1.4 or 50/2 (or somewhat wider, 28/2 or 25/2 if the crop requires that), and the 100/2 for the droplet. That the crop factor is huge is a great thing, considering I need to get the droplet as big as possible in my frame. I also tried a reverse ring, but that's a bit too risky with all the water involved. I will soon start to do some testing. Cheers

@lostfeliz Thanks for your reply. I use a very nice lamp (just the one in my bathroom), with great intensity and nice character. For my film idea, I need this lamp, and to replace it with another bulb is actually a bit risky (been broken before). The intensity is more than sufficient for most shots, but when going as close as possible with the 100/2, you loose quite some light, and 1600 iso seems inevitable. I am curious how the imagery will differ with the new crop and lossless 12 bit compaired to the 10 bit nonlossless raw. Thanks.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: whysodifficult on June 09, 2017, 08:49:32 PM
Gentlemen, I've been searching everywhere but haven't found instruction on usage of MLV_DUMP on Mac. Can you kindly help me?

I'm using Footage for Mac app, it's good, but whites on video are pink there. I suppose it will not be a problem using MLV_DUMP(?)...

Thank you!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on June 09, 2017, 09:08:19 PM
That could be related to white level settings. What settings are you using in cam? Upload a sample mlv file?
You could develop with cr2hdr.app. It contains latest mlv_dump code. Mlvfs as well. Or mlv_dump binary itself of course.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: whysodifficult on June 09, 2017, 09:22:21 PM
Danne, thank you for quick reply! Is the DNG file of the video enough? https://drive.google.com/file/d/0B47FS5gpVSwOU3NxanJDaS1TNUE/view?usp=sharing

I have MLV now of 1,19 Gb.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on June 09, 2017, 09:40:14 PM
What a beautiful shot. Where is it from?
Usually white balance should be around 15000 but I see yours is at around 5000. Anyway seems a tricky shot, very bright so 15000 won´t work either. I keep a mlv_dump menu section in cr2hdr.app for manually set new white balance and also to create sample file packages so if you want to share a sample instead of the whole 1gb file you could go down the cr2hdr.app route
http://www.magiclantern.fm/forum/index.php?topic=15108.0
Once installed head over to the mlv_dump menu. There is a setting for samples and one for white balance.
You could also share the whole 1gb file if you like.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on June 09, 2017, 09:52:48 PM
@Danne:

Code: [Select]
exiftool M30-1836_C0000_00001.dng -WhiteLevel -BlackLevel
White Level                     : 5586
Black Level                     : 2047

(5586-2048)*4 + 2048 = 16200

Which means, this is a 12-bit lossless image - that's how the bit depth reduction works. The container is still 14-bit, but the usable range is restricted to 12 bits, with black level equal to whatever Canon firmware assumes (if it's 2048, that would give a valid range of 1536 ... 5586, which means 4050 levels - just below 4096).

The DNG looks just fine here, without adjusting its metadata. It's overexposed, but the highlights are white.

BTW, white balance != white level.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on June 09, 2017, 10:04:54 PM
I was fooled by the container. Thanks for getting back. Darn good view in that dng and still a lot of dynamic range to pull back but overexposed.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: whysodifficult on June 09, 2017, 10:41:40 PM
Danne, Alex,
Thank you! It's Yosemite National Park in California (rock, called Half Dome) a few days ago, i was lucky with a cloudy weather. It was much more beautiful when i drove to that point (Glacier Point), nothing but clouds around and under you and then they slowly started to disappear, you could see clear spots of the valley down there, of rocks around, and i went to parking for my camera but it was too late when i came back :)

I've just opened this DNG in Photoshop and Lightroom and it looks overexposed (so as google drive shows), but earlier i only looked at all photos in LilyView image viewer on Mac, that shows this photo NOT overexposed, it shows the same how i saw the photo on Live View screen on camera, but the whites are pink: https://drive.google.com/file/d/0B47FS5gpVSwOS1h3b214UmZILTA/view?usp=sharing

So as Finder app (native file browser in Mac) shows — pink. So as Footage for Mac app shows it in the app — https://drive.google.com/file/d/0B47FS5gpVSwOeHptZmxkNWtGQkU/view?usp=sharing , and how it converts MLV to MOV video — pink. https://drive.google.com/file/d/0B47FS5gpVSwONG9WSWlRV3kyUE0/view?usp=sharing
So i didn't even think to open DNG in Photoshop and Lightroom because everywhere else whites are pink.

I read somewhere on this forum, A1ex replied to someone on this pinkness, but i didn't understand that, so i figured that this is the temporarily downside of 3.5K. By the way, i suppose pinkness happens only in high-resolution video (crop mode)... As i recall. Also i thought that this Footage for Mac app doesn't remove this pinkness, but i read somewhere that MLV_dump does. Maybe i got it wrong. But wanted to try.

Maybe over-exposition is my fault during shooting, i'm not a professional photographer or videographer, but that's how i saw it on camera screen.

Adjusting in Lightroom shows that this photo is not fatally overexposed, all details are kept. So i don't know what to think about it (what is the reason):
https://drive.google.com/file/d/0B47FS5gpVSwOWENZWW5sZU9VV0E/view?usp=sharing

This is the MLV, video is 11 sec of static view, but you can see the clouds are slowly moving:
https://drive.google.com/file/d/0B47FS5gpVSwOOEpDN19xeXIxM0E/view?usp=sharing

It was 3.5K according to instructions in this thread above from @HJfilmspeed https://vimeo.com/217313287 so it's 12 bits but the Footage app says it's 14 bits.

Thank you.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on June 09, 2017, 11:14:57 PM
Here is a sample package of the 1gb provided from whysodifficult if anybody wants to check.
https://bitbucket.org/Dannephoto/magic-lantern/downloads/M30-1838_samples.zip

The quick look previewer in mac doesn´t get the white balance level at all. See the last example. The other two is how DaVinci resolve and adobe camera raw interprets the dng when opened up in each program.
(Added dcraw -T -w output as well).

DaVinci Resolve
(https://s13.postimg.org/kp7pzy3uv/image.png)

Adobe camera raw
(https://s13.postimg.org/ckzlv7hfr/image.png)

dcraw(tiff)
(https://s2.postimg.org/wdofc4dh5/image.png)

Quick look previewer
(https://s13.postimg.org/l4izsyps7/image.png)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on June 09, 2017, 11:18:56 PM
@Danne: the white balance looks fine to me in all 3 images. It's the white level tag that's misinterpreted.

Just for kicks: ufraw-mod with highlight recovery in HSV, EC -0.8 (linear) and +1 (soft-film):

(http://a1ex.magiclantern.fm/bleeding-edge/ufraw-mod/M30-1836_C0000_00001-08lin.jpg)

(http://a1ex.magiclantern.fm/bleeding-edge/ufraw-mod/M30-1836_C0000_00001+1.jpg)

Regular rendering (EC 0 without highlight recovery) looks pretty much identical to dcraw.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on June 09, 2017, 11:22:53 PM
Oops, word slip. I corrected my post. Meant white level. Added dcraw tiff output as well.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on June 09, 2017, 11:43:47 PM
@Danne: the white balance looks fine to me in all 3 images. It's the white level tag that's misinterpreted.

Just for kicks: ufraw-mod with highlight recovery in HSV, EC -0.8 (linear) and +1 (soft-film):

(http://a1ex.magiclantern.fm/bleeding-edge/ufraw-mod/M30-1836_C0000_00001-08lin.jpg)
(http://a1ex.magiclantern.fm/bleeding-edge/ufraw-mod/M30-1836_C0000_00001+1.jpg)

Regular rendering (EC 0 without highlight recovery) looks pretty much identical to dcraw.

Wow, that ufraw mod is doing quite a good job pulling back information. The code should be inside dcraw if you ask me. Seems to be pulling back information better than ACR. Is this possible.

Ex. ACR pulling down highlight recovery and whites to the bottom
(https://s10.postimg.org/6bq3yvdpl/image.png)

dcraw -T -H 2 -w
(https://s10.postimg.org/4xyh3keg9/image.png)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Teamsleepkid on June 10, 2017, 06:38:33 AM
i noticed this too with the footage for mac app. theres pink all over my footage. not so with cr2hdr...so I've been using that lately.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: reddeercity on June 10, 2017, 08:33:36 AM
I check out the 3.5k file that @whysodifficult posted on my PC Windows 7 Pro desktop for only WB issue .
I extracted the dng's with the latest mlv_dump.exe from the experimental download page and every thing looks right .

(https://c1.staticflickr.com/5/4218/35044374322_1fbbd1c299.jpg) (https://flic.kr/p/VoKzyo)
Test_3.5k_5d3_WIN7_FILE_EXPLORER_from_mlv_dump (https://flic.kr/p/VoKzyo) by RedDeerCityTV (https:[email protected]/), on Flickr

No adjustments , just loaded in to A.E. CS6 PC/Win7 . The WB seems to be 5500K , looks ok to me
(https://c1.staticflickr.com/5/4257/34822932030_12fe396ce3.jpg) (https://flic.kr/p/V4bCtQ)
Test_3.5k_5d3_A.E.CS6 (https://flic.kr/p/V4bCtQ) by RedDeerCityTV (https:[email protected]/), on Flickr

I was very impressed with the image , I started to push the image around with ACR and the image really pop out .

(https://c1.staticflickr.com/5/4205/35169952406_54b7316995.jpg) (https://flic.kr/p/VzRcxL)
m30-1838_00000.tiff (https://flic.kr/p/VzRcxL) by RedDeerCityTV (https:[email protected]/), on Flickr

I started with "Camera Neutral" & 7250K WB , the .XMP from A.E. is below
I also exported 50Mb Mp4 H264 @ 35Mb/s   link--> https://drive.google.com/file/d/0B3rFQHNAb91MQWxGUWNxM05WX2s/view?usp=sharing
I was amazed how much sharpness it could handle , really made the fine detail come out  (very nice)  :D
Code: [Select]
<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="Adobe XMP Core 5.3-c007 1.136881, 2010/06/10-18:11:35        ">
 <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <rdf:Description rdf:about=""
    xmlns:crs="http://ns.adobe.com/camera-raw-settings/1.0/"
   crs:Version="7.0"
   crs:ProcessVersion="6.7"
   crs:WhiteBalance="Custom"
   crs:Temperature="7250"
   crs:Tint="+10"
   crs:Saturation="+44"
   crs:Sharpness="58"
   crs:LuminanceSmoothing="40"
   crs:ColorNoiseReduction="40"
   crs:VignetteAmount="0"
   crs:ShadowTint="0"
   crs:RedHue="0"
   crs:RedSaturation="0"
   crs:GreenHue="0"
   crs:GreenSaturation="0"
   crs:BlueHue="0"
   crs:BlueSaturation="0"
   crs:Vibrance="+11"
   crs:Defringe="0"
   crs:HueAdjustmentRed="+17"
   crs:HueAdjustmentOrange="+30"
   crs:HueAdjustmentYellow="+8"
   crs:HueAdjustmentGreen="0"
   crs:HueAdjustmentAqua="0"
   crs:HueAdjustmentBlue="+1"
   crs:HueAdjustmentPurple="+18"
   crs:HueAdjustmentMagenta="+1"
   crs:SaturationAdjustmentRed="-10"
   crs:SaturationAdjustmentOrange="-6"
   crs:SaturationAdjustmentYellow="+89"
   crs:SaturationAdjustmentGreen="0"
   crs:SaturationAdjustmentAqua="-9"
   crs:SaturationAdjustmentBlue="+13"
   crs:SaturationAdjustmentPurple="-6"
   crs:SaturationAdjustmentMagenta="0"
   crs:LuminanceAdjustmentRed="-43"
   crs:LuminanceAdjustmentOrange="+18"
   crs:LuminanceAdjustmentYellow="+28"
   crs:LuminanceAdjustmentGreen="-4"
   crs:LuminanceAdjustmentAqua="-3"
   crs:LuminanceAdjustmentBlue="-100"
   crs:LuminanceAdjustmentPurple="-8"
   crs:LuminanceAdjustmentMagenta="0"
   crs:SplitToningShadowHue="0"
   crs:SplitToningShadowSaturation="0"
   crs:SplitToningHighlightHue="0"
   crs:SplitToningHighlightSaturation="0"
   crs:SplitToningBalance="0"
   crs:ParametricShadows="-48"
   crs:ParametricDarks="-100"
   crs:ParametricLights="-15"
   crs:ParametricHighlights="-75"
   crs:ParametricShadowSplit="10"
   crs:ParametricMidtoneSplit="20"
   crs:ParametricHighlightSplit="90"
   crs:SharpenRadius="+1.0"
   crs:SharpenDetail="25"
   crs:SharpenEdgeMasking="0"
   crs:PostCropVignetteAmount="0"
   crs:GrainAmount="0"
   crs:LuminanceNoiseReductionDetail="38"
   crs:ColorNoiseReductionDetail="50"
   crs:LuminanceNoiseReductionContrast="40"
   crs:LensProfileEnable="1"
   crs:LensManualDistortionAmount="0"
   crs:PerspectiveVertical="0"
   crs:PerspectiveHorizontal="0"
   crs:PerspectiveRotate="0.0"
   crs:PerspectiveScale="100"
   crs:AutoLateralCA="0"
   crs:Exposure2012="-0.55"
   crs:Contrast2012="-47"
   crs:Highlights2012="0"
   crs:Shadows2012="0"
   crs:Whites2012="0"
   crs:Blacks2012="0"
   crs:Clarity2012="+45"
   crs:ConvertToGrayscale="False"
   crs:ToneCurveName2012="Custom"
   crs:CameraProfile="Camera Neutral"
   crs:CameraProfileDigest="AF004EBA47092ECF696D06CE24881DE8"
   crs:LensProfileSetup="LensDefaults"
   crs:HasSettings="True">
   <crs:ToneCurvePV2012>
    <rdf:Seq>
     <rdf:li>0, 0</rdf:li>
     <rdf:li>50, 38</rdf:li>
     <rdf:li>73, 68</rdf:li>
     <rdf:li>107, 106</rdf:li>
     <rdf:li>130, 126</rdf:li>
     <rdf:li>176, 173</rdf:li>
     <rdf:li>209, 206</rdf:li>
     <rdf:li>255, 255</rdf:li>
    </rdf:Seq>
   </crs:ToneCurvePV2012>
   <crs:ToneCurvePV2012Red>
    <rdf:Seq>
     <rdf:li>0, 0</rdf:li>
     <rdf:li>255, 255</rdf:li>
    </rdf:Seq>
   </crs:ToneCurvePV2012Red>
   <crs:ToneCurvePV2012Green>
    <rdf:Seq>
     <rdf:li>0, 0</rdf:li>
     <rdf:li>255, 255</rdf:li>
    </rdf:Seq>
   </crs:ToneCurvePV2012Green>
   <crs:ToneCurvePV2012Blue>
    <rdf:Seq>
     <rdf:li>0, 0</rdf:li>
     <rdf:li>255, 255</rdf:li>
    </rdf:Seq>
   </crs:ToneCurvePV2012Blue>
  </rdf:Description>
 </rdf:RDF>
</x:xmpmeta>
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: whysodifficult on June 11, 2017, 01:09:59 AM
A1ex, Danne,
I didn't understand what you were writing above, of course, though it wasn't intended for me obviously.

BTW, thank you for such a huge work on making shooting video on Canon a pleasure. All the best to you!


@reddeercity, beautiful edit.

In case anybody would like to download my other 3.5K MLV's of Yosemite, i've uploaded them here:

Waterfalls — 1,3 Gb — 11 sec https://drive.google.com/file/d/0B47FS5gpVSwObWxhTTNiT0k1TW8/view?usp=sharing
(https://thumb.ibb.co/cAuJ8F/Screen_Shot_2017_06_10_at_4_31_51_PM.png) (https://ibb.co/cAuJ8F)

Mountain river — 2,2 Gb — 20 sec
https://drive.google.com/file/d/0B47FS5gpVSwOSkZYZU8tSHZ6aTQ/view?usp=sharing
(https://thumb.ibb.co/eLT6vv/Screen_Shot_2017_06_10_at_4_32_09_PM.png) (https://ibb.co/eLT6vv)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: reddeercity on June 11, 2017, 05:44:35 AM
Thanks , I'll check out the other Yosemite shots .

For all win users -- I been looking at different ways to use mlv_dump with out the cmd line interface as some have a issue with that .
So in a nut shell any of the old PC app that use mlv_dump , just replace it with the latest ver. from the experimental 4k crop rec download page.
It works with MLVBrowseSharp & MLVViewSharp MLV Viewer to view .mlv (and .raw) files on windows (C#) (http://www.magiclantern.fm/forum/index.php?topic=8447.msg77759#msg77759) FYI this is cross platform (PC & MAC) .
I really like this old app it just never got finished being develop , it very powerful you can browses to any file location and extract dng's or export as legacy .Raw .
The reason I being that up is if you do convert to .Raw app's like  MLVProducer (http://www.magiclantern.fm/forum/index.php?topic=15271.msg148538#msg148538), ML Raw Viewer 1_3_4  (http://www.magiclantern.fm/forum/index.php?topic=9560.msg132429#msg132429),  can then work with the new UHD/4k resolution's when shot with compressed
raw (it will be a bigger filer thou with uncompressed14bit) . I should mention with MLVBrowseSharp there's no thumbnail preview but does extract the dng's correctly by a right click
of the mouse hovering over the file and  MLVViewSharp there is pink highlight (works with converted .Raw , .MLV compressed raw error out) the great thing about MLVViewSharp
is you can change the raw debayering down to 1/8 of the resolution for quick playback plus many other image processing , LUTs etc. .... .
The other one I check out was MLV Converter 1.9.2 (http://www.magiclantern.fm/forum/index.php?topic=10198.msg98284#msg98284)  , replaced  mlv_dump and I could export dng's , 1080p proxy all at the same time , I can even export with Lossy  compression
which gave me a 1.35MB 3.5k file  verse a 8MB uncompressed 14bit frame . A.E. ACR see them and can be imported but not in to Resolve 12.5 . 
I tested the Lossy dng compressed to 1.35 MB pre-frame in A.E. and I could not tell the different between the compressed or uncompressed raw file.
Hope that helps a little.   
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on June 11, 2017, 11:19:09 AM
Thank you very much for this info reddeercity.

I was searching for a way to preview 14bit lossless mlv's without converting to dng's, and had no success.

Thanks!!!!!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on June 11, 2017, 12:39:16 PM
 :(
Tryed but no success.
I can't view 14bit lossless mlv's in MLVviewSharp.
mlv_dump replaced for the last version from experimental builds in same directory.
Did I miss something?...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: GutterPump on June 11, 2017, 01:04:14 PM
I did not know the existence of MLVViewSharp, thanks you reddeercity.

But do you know how extract DNG into separate folders ? All the DNG are exported in the same directory.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: reddeercity on June 12, 2017, 03:58:25 AM
Tryed but no success. I can't view 14bit lossless mlv's in MLVviewSharp. Did I miss something?...
  ..... MLVViewSharp there is pink highlight (works with converted .Raw , .MLV compressed raw error out)
@pc_bel  you need to convert .mlv to .raw legacy , unless  the developer "g3gg0" is willing to update to support compressed raw
that's the workaround at the moment . FYI you could request an update in his thread (http://www.magiclantern.fm/forum/index.php?topic=8447.msg77759#msg77759)  ;)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on June 12, 2017, 04:23:53 AM
One workaround previewing any MLV footage is going through MLVFS. Once the virtual dng files are created MlRawViever will be able to read the dng sequence. Just right click a dng file and select MlRawViewer.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: reddeercity on June 12, 2017, 05:19:58 AM
I did not know the existence of MLVViewSharp, thanks you reddeercity.
But do you know how extract DNG into separate folders ? All the DNG are exported in the same directory.
Just use MLVBrowersSharp , right click on DNG+wave then it open to a window to save , (see next image)
(https://c1.staticflickr.com/5/4197/34863292500_20ff8c061b_n.jpg) (https://flic.kr/p/V7Kuf1)
1 (https://flic.kr/p/V7Kuf1) by RedDeerCityTV (https:[email protected]/), on Flickr

Select which drive or folder you want.
(https://c1.staticflickr.com/5/4195/34441432383_18e40fb1e8_n.jpg) (https://flic.kr/p/UttkVH)
2 (https://flic.kr/p/UttkVH) by RedDeerCityTV (https:[email protected]/), on Flickr

FYI: 10-12bit uncompressed support with metadata , this is how it should work  :D
(https://c1.staticflickr.com/5/4259/34863459760_e1bcd26f30_n.jpg) (https://flic.kr/p/V7LkXN)
10_12bit_5D2+metadata (https://flic.kr/p/V7LkXN) by RedDeerCityTV (https:[email protected]/), on Flickr


Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: reddeercity on June 12, 2017, 05:31:01 AM
One workaround previewing any MLV footage is going through MLVFS. Once the virtual dng files are created MlRawViever will be able to read the dng sequence. Just right click a dng file and select MlRawViewer.
Yea I know  , but way to slow especially with UHD/4K it even lags in full HD with mlvfs .
On PC's best option for compressed raw is to convert to .raw legacy and use the existing PC apps .
There's more reasons to use MLVViewSharp then just for viewing the files .
MLVProducer developer was thinking of supporting compressed raw in his app , until then work arounds 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on June 12, 2017, 11:39:33 AM
@Danne
@reddeercity

Thank you very much!!!!.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: GutterPump on June 12, 2017, 12:16:53 PM
Just use MLVBrowersSharp , right click on DNG+wave then it open to a window to save , (see next image)
(https://c1.staticflickr.com/5/4197/34863292500_20ff8c061b_n.jpg) (https://flic.kr/p/V7Kuf1)
1 (https://flic.kr/p/V7Kuf1) by RedDeerCityTV (https:[email protected]/), on Flickr

Select which drive or folder you want.
(https://c1.staticflickr.com/5/4195/34441432383_18e40fb1e8_n.jpg) (https://flic.kr/p/UttkVH)
2 (https://flic.kr/p/UttkVH) by RedDeerCityTV (https:[email protected]/), on Flickr



Thanks for your answer, but as you can see in my picture :

(https://image.noelshack.com/fichiers/2017/24/1/1497262305-batch.jpg)

all my DNG sequence are mixed in a same folder; i see you are using windows 7, maybe this issue is only about windows 10.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: bouncyball on June 13, 2017, 10:19:14 AM
Yea I know  , but way to slow especially with UHD/4K it even lags in full HD with mlvfs .
On PC's best option for compressed raw is to convert to .raw legacy and use the existing PC apps .
There's more reasons to use MLVViewSharp then just for viewing the files .
MLVProducer developer was thinking of supporting compressed raw in his app , until then work arounds 
On a decent hardware there is no lag on PC (w/o antivirus installed) and there is no need to use legacy RAW, you can just uncompress (with appropriate mlv_dump) to MLV again and use whatever tool you desire (mlrawviewer etc)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on June 13, 2017, 10:49:39 AM
Eh, I totally forgot about decompression of MLV files although I made use of it in cr2hdr.app  8).
Thanks for the reminder.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: sylvain_c on June 14, 2017, 04:29:55 PM
Sorry to annoy you,

I don't know what I'have done with the latest build (pretty sure a bad choice in bit depth in the hurry) but half of my converted mlv's are just blacks dng (latest mlv dump).
I'm asking just to be sure, but I think this shots are lost?

A sample of the problem:
https://mega.nz/#!vxkkGQJT!8GjEWu1dzUq3yVL7siRQI_u7TuPnFIm-vDmFCELu594
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on June 14, 2017, 04:47:13 PM
Black or just darker? You´re uploaded file has a dark preview image but opens just fine in acr or when converted to a tiff with dcraw. I get those darker files as well. Not sure why but it seems related to 10/12bit and only preview.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: sylvain_c on June 14, 2017, 05:56:11 PM
Really, it open fine in acr??I was pretty upset by the black preview I come here too quickly...
(You're right, all open fine in acr, it's my viewer, picasa viewer the culprit)!
Thanks for the quick reply Danne!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on June 15, 2017, 03:08:05 PM
@g3gg0

Yesterday I did some shooting with latest build on 5dmk3
1920x800 @ 60FPS

https://drive.google.com/open?id=0B-d8ARtc7xwWQU9kVkktTW9NMEk

I cannot decompress some MLVs, here is a sample.

Use your wizardry
Thanks in advance
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Walter Schulz on June 15, 2017, 03:42:02 PM
mlv_dump  --dng m14-0847.mlv
works here.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on June 15, 2017, 03:54:54 PM
Do this.
mlv_dump --dng --relaxed
Fixes the issue on mac.

If you use cr2hdr.app you find the relaxed switch under (14) autopsy settings
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: g3gg0 on June 15, 2017, 05:43:21 PM
as i am on low bandwidth, what is the problem with the file?

which block fails?

can you extract the troublesome block using:
Code: [Select]
mlv_dump m14-0847.mlv --extract <blocknumber> --autopsy-file bad.bin
and then upload the bad.bin
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on June 15, 2017, 06:10:21 PM
Here´s a 7mb sample of the MLV file.
https://bitbucket.org/Dannephoto/magic-lantern/downloads/M14-0847_samples.zip

It´s the LJ92: Failed (-1) error.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: g3gg0 on June 15, 2017, 06:41:56 PM
thanks!

FYI
when fetching the latest mlv_dump (https://builds.magiclantern.fm/jenkins/view/Experiments/job/crop_rec_4k/) and some recent mlv_lite / mlv_rec, you can use this to determine the exact build version:

Code: [Select]
./mlv_dump <mlv_file>  --extract-type VERS --payload-only --ascii
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on June 15, 2017, 07:10:46 PM
Very nice indeed.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on June 15, 2017, 09:28:46 PM
@Walter
@g3gg0

I get -1 error.
I use the mlv_dump that is in the Experiments page.
:(
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Walter Schulz on June 15, 2017, 09:34:53 PM
mlv_dump --dng --relaxed
processes 381 out of 400 frames.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on June 15, 2017, 09:43:46 PM
mlv_dump --dng --relaxed
processes 381 out of 400 frames.

Yes, I was about to report that.
The relaxed switch does the trick.
Gratitude
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kharak on June 16, 2017, 07:42:27 PM
@a1ex / @g3gg0

Could you look in to the mlv_dump from experimental. When I convert 60P 3x3 Footage I still have a lot of Vertical Stripes in highlights, even though mlv_dump is doing Vertical Stripe Fix, I am wondering if the code is not fitting for the 60p 3x3 mode or maybe there is an offset which enhances vertical stripes?, Maybe because of the 4 dead vertical rows on the left of the frame (also a bug btw). Especially if I have lens flare or direct sunlight flaring in the lens, then the Vertical Stripes are more visible than anything I've recorded with normal 1920x1080 MLV 3x3. What could be the cause of it?

When I play the footage in mlvrawviewer and turn on Vertical stripe fix in that program, sometimes that will get completely rid of the stripes and other times it will make it worse. Its really weird to work with.

The heavier the grade, the more visible the stripes. To make MLV footage look really good, you have to push footage so much that it crosses that threshold where the stripes become really visible. Temporal Noise reduction will also at times enhance it even more, as the algorithms "think" its important information to keep.

 Here is a cropped screenshot from mlvrawviewer with a LUT pushing the image from a log curve, to show you the severity of the stripes.

https://mega.nz/#!8YIHGIJB!hzewH2JBX55vELsy2MPdp3gPSo2mTzlNKe_CQIVqu_o

 As you can see the, flaring enhances the vertical stripes a lot and in this shot its only part of the image, sometimes the entire image will be engulfed in flare and everything turns to Stripes.. I could not find my shot where I had a full flare, aber its egal, same thing.


Also, it would be really nice if you could add Allow Global Draw - OFF during recording to MLV lite as in the vanilla MLV. Because when I am pressing half shutter to remove overlays, I am getting vastly improved recording times, almost double the time and sometimes continues (depending on scene).

and another bug to report from 60p 3x3, when selecting ISO 100 the Raw Zebras become disabled and the Histogram maxes out at about 90% the way to the right, the channels don't go beyond it, leading the operator to believe that one is well within ETTR range. The non-raw spotmeter also maxes out at 96% (not sure if also at other iso's).

Also a general question, why is it when hitting record button, that the recording starts so "sluggish", like 1-2 secs after, does it need to write information to a buffer or something, is it not something that could be done before hitting record, I mean just by entering the mode and keeping it Armed for recording? Also when recording stops, it takes about 4-6 seconds before I can hit record again, maybe longer.. It is quite counter productive, when one wants to shoot a lot of bursts at times and having to wait these combined almost 10 seconds per clip. Is it something one can expect some improvements on in the future ?

123 Fir
June 3rd build (I think it was, or the one thereabouts)




Thank you for your time.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on June 16, 2017, 09:38:25 PM
Quote
https://mega.nz/#!8YIHGIJB!hzewH2JBX55vELsy2MPdp3gPSo2mTzlNKe_CQIVqu_o

In this particular case, I believe the vertical stripes are there because - just a guess - the first frame probably did not have relevant highlights to be analyzed.

Another approach that I didn't implement, but only documented the backend support for it (http://www.magiclantern.fm/forum/index.php?topic=17795.msg171595#msg171595): it would require pointing the camera to something like a gray wall and running a calibration routine that would adjust the column gain registers (8 multipliers) until the artifact disappears. Might be worth trying.

Quote
Allow Global Draw - OFF during recording to MLV lite as in the vanilla MLV

This one was a major source of confusion, so my previous answer (http://www.magiclantern.fm/forum/index.php?topic=5533.msg93763#msg93763) is still valid.

Quote
recording starts so "sluggish", like 1-2 secs after

Sorry, not sure what you are talking about. Keep in mind I'm not a regular user of this mode (actually I don't remember touching the camera in the last few weeks...)

Is it the delay between you pressing the record button and the first recorded frame? Or the lack of responsiveness on the user interface?

Quote
I mean just by entering the mode and keeping it Armed for recording?

Pre-recording does exactly this.

Of course, the preparations can be done in background, with extra care when switching video modes, or when turning off raw recording. It's doable, but will increase the code complexity (although, probably not a bad idea, since some of the preparations are done in background in order to estimate recording times).

Quote
it takes about 4-6 seconds before I can hit record again, maybe longer

Set "Show graph: Buffers" to see what happens. All the frames that were recorded to RAM before you pressed Stop are now being saved to card (about 300MB iirc).

Quote
Is it something one can expect some improvements on in the future ?

If, in the future, you will select a resolution that gives continuous recording, then yes.

If we find some way to allocate more memory (the 5D3 has 512MB), this duration will increase (along with the recording time).

One can get a minor improvement (maybe speeding this up by 1-2 seconds) by pausing LiveView during this process. The silent picture module uses this trick, so it's an extremely easy coding task for those interested.

Quote
another bug to report from 60p 3x3, when selecting ISO 100 the Raw Zebras become disabled

Will look into this one.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kharak on June 16, 2017, 11:11:16 PM
In this particular case, I believe the vertical stripes are there because - just a guess - the first frame probably did not have relevant highlights to be analyzed.

Another approach that I didn't implement, but only documented the backend support for it (http://www.magiclantern.fm/forum/index.php?topic=17795.msg171595#msg171595): it would require pointing the camera to something like a gray wall and running a calibration routine that would adjust the column gain registers (8 multipliers) until the artifact disappears. Might be worth trying.


You are exactly right, first few seconds of that clip has no flaring.

I'll take a look in to the rest of your reply tomorrow and the calibration.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: okanergoknil on June 17, 2017, 10:42:53 AM
I keep trying to install to firmware 1.2.3- 5d mark 3 however I cannot get the 4K tool. Can you please give the installation details? We tried on 2 different cameras but its not working  at all :(
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on June 17, 2017, 03:40:49 PM
Any chance to get a flashing warning when card is almost full ?
Not during recording but when idle
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kharak on June 17, 2017, 10:54:34 PM
Quote
Another approach that I didn't implement, but only documented the backend support for it (http://www.magiclantern.fm/forum/index.php?topic=17795.msg171595#msg171595): it would require pointing the camera to something like a gray wall and running a calibration routine that would adjust the column gain registers (8 multipliers) until the artifact disappears. Might be worth trying.

Read the thread again and I've been so interested in Darkframe Averaging and flatframe since the talk started here a long time ago, but as a Windows user there is no GUI converter with these options. Unfortunately MLVFS does not work properly with AE and AE is at the core of my workflow. Can you link me to a thread that explains how to type in the different codes/commands for Darkframing and Flatframing in mlv_dump, I cannot find any good relevant thread or post, there are so many search results of mlv_dump, but if you know where I should look, that would be nice, if not I'll keep looking.

Quote
This one was a major source of confusion, so my previous answer (http://www.magiclantern.fm/forum/index.php?topic=5533.msg93763#msg93763) is still valid.

Yeah, I just thought this 'Allow Global Draw - Off' was a specific "component" easy to port. I know what "Allow Global Draw - Off" means because I know what it does, but the wording of it makes no sense to me and even more confusing for newcomers.

Quote
Sorry, not sure what you are talking about. Keep in mind I'm not a regular user of this mode (actually I don't remember touching the camera in the last few weeks...)

Is it the delay between you pressing the record button and the first recorded frame? Or the lack of responsiveness on the user interface?

Pre-recording does exactly this.

Of course, the preparations can be done in background, with extra care when switching video modes, or when turning off raw recording. It's doable, but will increase the code complexity (although, probably not a bad idea, since some of the preparations are done in background in order to estimate recording times).

Yes, recording starts 1-2 seconds after pressing record. I've tried with Pre-Recording and it does counter it but only after it has made the Pre-Recording which in it self takes 1-2 seconds to start recording aswell. On top of that, I have to activate the Pre-recording and then I have to press Record again, something that in the "heat of the battle" can get in to kind of an opposite effect where I record when I think I stopped recording or I stop a recording when I want to start it (happened a bit too much for me in 24p Mode.. and I only found out when I checked the footage  ;D ) I don't know, maybe I just need to get more used to it.

But with Pre-Recording I am getting less recording time, I usually get around 400/500 to 700 Frames and with pre-record 1 sec I get 300-450's. I use the lowest amount 1 sec pre-record (60 Frames), maybe you could add 1 frame pre-recording specific for the 60/50p mode as a standard, just so that recording starts immediately as an "easy" workaround? Not sure if easy or not. The Background thing you mentioned, sounds even better.
 
I can't give any technical explanation on why the recording times are lower, maybe I've just been unlucky with the compression, I must add that I did not test it extensively. Just done a few shots with pre-record in scenes where I was expecting longer recording times than the 300 to 400s.

EDIT: Just did a few more tests, switching between Pre-Recording and no Pre-Recording. The recording time is almost halved with Pre-Recording 1 Sec.

And you not touching your camera or this mode, you really have to try out the 60p mode, it is absolutely amazing! It has given my camera a new spirit. I thought I would use it more professionally but for filming family and those speedy little kids, its amazing.

Quote
One can get a minor improvement (maybe speeding this up by 1-2 seconds) by pausing LiveView during this process. The silent picture module uses this trick, so it's an extremely easy coding task for those interested.


I don't think I would mind the LV freezing during that period, I am unable to do anything in that time anyway, so if it shortens the wait time it sounds good to me. Though I would suggest a small text appearing right before the freeze saying "Writing from Memory" or something like that, so users wont experience frustration over their camera freezing after every take and not knowing why if they didn't read this thread or pull.

Thanks again for your time.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on June 17, 2017, 11:41:12 PM
@Kharak
Darkframe processing

1 - Film a few seconds with caps lock on your lens(totally black). Let´s call it your input_darkfootage.MLV.

2 - Perform averaging on this darkframe file like so:
Code: [Select]
mlv_dump -o averaged.MLV -a input_darkfootage.MLV
3 - Now use your averaged.MLV with corresponding normal footage.( Note! Use same settings when filming darkframe as corresponding footage). in command prompt do:
Code: [Select]
mlv_dump --dng -o OUTPUT INPUT.MLV -s averaged.MLV
INPUT.MLV is your nomal filmed footage
OUTPUT is your end result(dng files).

*note
If you leave out --dng you´ll get a MLV file instead of dng´s.
Code: [Select]
mlv_dump -o OUTPUT.MLV INPUT.MLV -s averaged.MLV
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on June 18, 2017, 12:11:50 AM
Good point about flat frames - they are yet another option for correcting the vertical stripes (alongside with other defects, such as vignette or dirt spots on the sensor or on the lens), and the calibration can be done afterwards.

About Global Draw while recording - the difference I've got on a quick test is pretty small: about 450 vs 500 frames with Preview set to Frozen LV, or about 300 vs 350 frames with Preview Auto, tested in 10-bit uncompressed 1920x800 60p, on 1.2.3, with default overlay configuration - which includes CPU-intensive zebras. That's only one extra second with global draw off, and I'd expect a lower difference on 1.1.3, where zebras are hardware-accelerated.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kharak on June 18, 2017, 05:04:55 AM
@Danne,

Thank you very much for the codes.

But the averaging does not work with the 60P Mode files, I tried it on 24p 3x3 Vanilla MLV files and it worked. When I input a 60P file I get "windows has encountered an error" but Command Prompt does not exit or crash. It just generates an empty averaged.mlv

I have tried with short and longer recording times, 1 sec and 6 secs in case it was too little, but its the same. I had the range from 100 - 3200 ISO, 6 MLV files and none of them worked, not the 1 sec or 6 sec MLV's.

I have not tried Normal Lossless 24p 3x3 MLV's, to see if it has something to do with the Lossless mode, I will do that next.

I think the mlv_dump I used is the one from the 3rd or 4th of june (I dont know where to see older builds anymore with the changes on download page) , the one that is 759 KB in size.


@a1ex

I always shoot 14 Bit Lossless 60P 1920x800 with Real-Time Preview and I get about 500-700 Frames and with Pre-Recording ON it was down on 350 Max. Pressing and holding Half-Shutter almost always increase the recording time by atleast 2 seconds (real time). I say "almost" because sometimes but rarely the compression goes crazy, I have many times gotten 10-15 Seconds (real time) recording this way, where I just stop the recording because I probably got the burst shot I was aiming for. Like earlier when I was shooting Lens Cap on for the DF Averaging, first frame it crashed saying Compressed size was bigger than uncompressed. A completely black picture on ISO 100 and it could not compress it.


I have never tried 10 bit mode, but what I see from all the examples it is extremely noisy, perhaps that is why your recording times are so low, maybe the compression goes crazy from all the colour casts 10 bit footage introduces. But yeah.. 10 bit vs 14 bit, mathematically you should have higher record times than I am getting. no?

And why would the difference be lower on 113? I thought fir 113 always had higher performance than 123 because of the extra buffer.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on June 18, 2017, 07:17:21 AM
Quote
But the averaging does not work with the 60P Mode files, I tried it on 24p 3x3 Vanilla MLV files and it worked. When I input a 60P file I get "windows has encountered an error" but Command Prompt does not exit or crash. It just generates an empty averaged.mlv
I can confirm lossles 60p 1920x800 yields segmentation fault 11 when trying to create an averaged darkframe.

Workaround:
Decompress your compressed footage prior to darkframe processing. Just tested and working.
Code: [Select]
mlv_dump -d -o OUTPUT.MLV INPUT.MLV
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on June 18, 2017, 08:32:15 AM
when I was shooting Lens Cap on for the DF Averaging, first frame it crashed saying Compressed size was bigger than uncompressed. A completely black picture on ISO 100 and it could not compress it.

That was a corrupted frame.

http://www.magiclantern.fm/forum/index.php?topic=19300.msg183115#msg183115

Quote
I have never tried 10 bit mode, but what I see from all the examples it is extremely noisy, perhaps that is why your recording times are so low

I've only used it in order to get constant bitrate. With lossless compression, the recording times will vary a lot with scene contents (so it's harder to benchmark this way; a static scene or a dark frame will make things a bit more repeatable in this case, but I didn't want the extra trouble.)

Quote
maybe the compression goes crazy from all the colour casts 10 bit footage introduces.

I thought I've mentioned "uncompressed"...

Quote
But yeah.. 10 bit vs 14 bit, mathematically you should have higher record times than I am getting. no?

No.

10 < 14
10 > 14 * 60%

Quote
And why would the difference be lower on 113? I thought fir 113 always had higher performance than 123 because of the extra buffer.

The cause is lower CPU usage on both Canon's side (regardless of ML settings), and on ML's side (only when Luma-based zebras are used). Memory bandwidth usage might also be a bit higher on 1.2.3 on Canon's side (didn't check this one, as it's not straightforward to measure). Of course, absolute performance can be higher both with and without global draw, but the difference between these numbers is likely lower (that is, global draw even faster on 1.1.3) iff you use Luma-based zebras (which is the default). I didn't benchmark these cases; it's just what I'd expect to happen.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kharak on June 18, 2017, 05:04:59 PM
@Danne,

Yes, now it worked generating averaged.mlv and outputting DNG but it is a very tedious process, having to decompress every file individually, rename it, move it to the appropriate folder... So the next big question, how do I batch process this ? Some command I should know? :)

Or atleast batch the decompression part. After decompression I see all the Metadata, ISO, fstop and so on in the MLV's. That way I can sort the MLV's with the same ISO's to their appropriate folders and do the DFA from there.

After DNG conversion, the metadata is lost again. a bug ?



I thought I've mentioned "uncompressed"...

Apologies, that slipped through.

Quote
No.

10 < 14
10 > 14 * 60%

I don't understand this sorry, 10bit is smaller than 14bit, 10bit is bigger than 14bit at 60% do you mean with the lossless compression? Is there less to compress from 10 bits ?

Sorry for my silly questions all the time, but I do appreciate very much you taking your time explaining these things, so that I know what is going under the hood.

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on June 18, 2017, 05:09:27 PM
Cr2hdr.app batch processes both decompression and darkframe processing. Mac. Someone with an interest in windows scripting can do the same.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on June 18, 2017, 05:42:29 PM
Is there less to compress from 10 bits ?

http://www.magiclantern.fm/forum/index.php?topic=19300.msg183231#msg183231
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on June 18, 2017, 05:56:11 PM
By the way. With latest version of mlv_dump on steroids(Bouncyball) the -b switch will work exporting different bits. Good for comparing let's say 14bit file into 8bit dng's.
Code: [Select]
mlv_dump -b 8 --dng Input.MLV
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Savely on June 18, 2017, 09:13:04 PM
Quote
Good point about flat frames
What was the point about flat frames? It's just some coding stuff that didn't yet exist as some executable for Windows users? I'm really interesting about vertical stripes problem cause I have horrible ones (at the sky usually)

(http://image.ibb.co/cnh3Xk/M30_0921_000001.jpg)

 is there a way to get rid of it for Win users?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on June 18, 2017, 09:47:59 PM
Please use search function. Flat frame or flatframe.
The function exists in mlv_dump which is the same for both windows and mac.
Flat frame workflow isn't trivial. I experimented with it some a while ago but there is more to test out.
Here is one link
http://www.magiclantern.fm/forum/index.php?topic=17795.msg175189#msg175189
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: g3gg0 on June 18, 2017, 09:53:05 PM
1. take a "flat field" video
  a) put a white shirt over lens
  b) point to the bright sky
  c) record using the same settings as before (same exposure time not strictly needed, but would keep the ISO)
2. average that video using the -a option (e.g. mlv_dump movie.mlv -a -o flat.mlv)
3. process the problematic footage and supply this flat.mlv as flatfield (e.g. mlv_dump in.mlv -t flat.mlv -o out.mlv)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Savely on June 18, 2017, 11:29:39 PM
Danne, g3gg0 thanks a lot, I will try and will make some tests
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: g3gg0 on June 18, 2017, 11:38:30 PM
btw.
to only pick the vertical stripes instead of lens speckles and vignetting also, you can add --avg-vertical when creating the flat.mlv
didn't try any of those on my own :)

but that could also introduce some weird left/right bar due to vignetting.
probably not a good thing when you have the lens on
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: D_Odell on June 19, 2017, 02:29:30 AM
Anyone who have footage of 1920x1080 @ 45p and 48p (3x3 binning)? I have tried to pull it of but images results in black blocks or other faults. Anyone?

Edit: tried in Canon menu All-I 50p and 48p in FPS ovettige and reciewed this massages:
(http://thumb.ibb.co/is8hSk/IMG_0550.jpg) (http://ibb.co/is8hSk)

Edit 2: Changed to 45p now worked a bit better. Got 7xx frames before stoped.

Edit 3: You who managed to record 45p and used it in a NLE, how much did you slow it down? Since 48p is really simple with 24p timeline. I tried to slow it down with 53% and it seemed ok. Any thoughts?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Oedipax on June 19, 2017, 10:00:17 PM
Edit 3: You who managed to record 45p and used it in a NLE, how much did you slow it down? Since 48p is really simple with 24p timeline. I tried to slow it down with 53% and it seemed ok. Any thoughts?
I think the easiest (in Premiere / After Effects) would be to conform the clip (using the Interpret/Modify command) to 23.976 or 25p depending on your edit needs. Then you won't have to figure out the exact right clip slow-down rate and it will probably be cleaner looking than slowing the clip back down with the NLE, which runs the risk of blending frames or having to generate new ones if it's not dead-on. Hope that made sense :)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: D_Odell on June 19, 2017, 11:52:52 PM
I think the easiest (in Premiere / After Effects) would be to conform the clip (using the Interpret/Modify command) to 23.976 or 25p depending on your edit needs. Then you won't have to figure out the exact right clip slow-down rate and it will probably be cleaner looking than slowing the clip back down with the NLE, which runs the risk of blending frames or having to generate new ones if it's not dead-on. Hope that made sense :)
That made excellent help! Works good in ae also. What shutter speed do you use? At first I left it at 94, but will probably go up to 180 next time? As one can see its quite blurry, also shot from a train...:
(http://thumb.ibb.co/doMb55/Comp_1_00884.png) (http://ibb.co/doMb55)

(http://thumb.ibb.co/k5a8JQ/Comp_1_03840.png) (http://ibb.co/k5a8JQ)


Also, anyone notised the black pixel line in bottom and to the left? If so anyone made it correct?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kharak on June 20, 2017, 04:12:33 AM
@a1ex, @g3gg0, @danne

I am thinking if the 60p mode is stressing the sensor so much that it is making the Vertical Stripes more prevalent than  24p 3x3? Is that a possibility? I am saying this because normally I didn't have issues with Vertical Stripes when I was shooting 24p, sometimes I would get some in highlights after heavy grade, but after getting addicted to 60p I am seeing a lot of shots with Vertical Stripes.

And I am not seeing Vertical Stripe fix working properly with 60p footage. These shots have the same exposure and lighting across the entire recording and yet the Vertical Stripes are still there. In the first screenshot its harder to see the the lines, but when playing the recording as this is a Slow pan shot, it becomes very visible. in shot 2 its much more visible, I also zoomed 200%.

Shot 1
https://mega.nz/#!QdpETRZR!_y6mdX7ZWGkVisIamJLLAeyss9uzdsEdMtn3TgETlTc  - Screenshot from AE - ISO 200
The jpeg compression is hiding it in this shot, look furthest to the left, some remnants of the stripes are there.

Shot 2
https://mega.nz/#!wAhwzawJ!jirFfeb5WrvW6YdZ6S9ubqsXbdKTItaw1oGboUQKDh4 - Cropped - ISO 200. Here you can see them.

FYI: In this case, activating Stripe removal on the DNG's in MLRawViewer removed the stripes, but introduced big chunky vertical lines from all the shadow parts of the image.

This same shot without Darkframe Averaging makes the Vertical Stripes less prevalent, but still there, its as if the deep blacks from Darkframe averaging is enhancing the stripes or the contrast of the stripes.

My point with this post is that I think that mlv_dump stripe removal might need some optimization for 60p mode, I might be wrong and doing something wrong in the process introducing this problem myself, I'd love to be proven so. But its a big issue for the 60p mode.

Just a guess here, but when decompressing, is it also doing vertical stripe fix? and then converting to DNG the same process is done again introducing the stripes again? I don't see any text saying so when decompressing, but just a thought.

Anyone else shooting 60p having issues with Vertical Stripes?

Thank you for your time.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on June 20, 2017, 05:50:01 AM
Could you post an example MLV with stripes?
Are you sure vertical stripe code is running on all clips?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: reddeercity on June 20, 2017, 08:40:26 AM
Thanks for your answer, but as you can see in my picture :
all my DNG sequence are mixed in a same folder; i see you are using windows 7, maybe this issue is only about windows 10.
Ok  --I see the issue .
So to anyone that want to use the new mlv_dump.exe on Windows platform to batch your UHD/4K/10-12bit & compressed14bit mlv's in to separated file folder.
First I thought of making a .bat (win7) & or .cmd (win10) (really the same thing) but I had problems with importing more one file at a time
(to much code and getting too complex) . Then I considered building a simple .exe batch program , then it dawn on me why I'm I trying to
re-invent the wheel here  ::) , so after a quick search I found a Older Win's app that uses mlv_dump.exe . MLVConverter1.9.2.zip (https://www.dropbox.com/s/7qhncp7pi4t268o/MLV%20Converter%201.9.2.zip)
MLV Converter 1.9.2 for WINDOWS (https://www.magiclantern.fm/forum/index.php?topic=10198.msg98284#msg98284) You will need some other tool if you want to have it fully functional --Adobe DNG Converter , mlrawviewer (only to view file not working work compressed raw) & IrfanView .
All I did was to replace the mlv_dump that there with the new mlv_dump from the experimental download page , installed the latest  Adobe DNG Converter (to output compressed dng's)
and  IrfanView . I tested 14 mlv's files which included 10-12bit , 14bit, UHD(3520x1320) compressed raw 14bit and all exported fine in to the respected folders . The dng's import ok in to
After Effect CS6 (Wins) didn't try any other app . *Note* in the tools folder you have (3) mlv_dump's (mlv_dump , mlv_dump1 & mlv_dump2) just copy the new mlv_dump and re-name it
mlv_dump1 etc. .... that's it should work fully from there with 1080p proxy you can add a customs .xmp file from A.E. and have it export to ffmpeg prores4444 if you wish .
If you want to export Dual ISO you need to update  cr2hdr.exe to the latest windows version.
One thing windows will ask permission to run mlv_dump on every file unless you go the properties and in the general tab check "UnBlock" at least on Win7 , win10 should be the same
Hope that helps :D
Edit: update exiftool.exe also
(https://c1.staticflickr.com/5/4228/35290258021_e3e801ee01.jpg) (https://flic.kr/p/VLtNdv)
mlv raw converter1 (https://flic.kr/p/VLtNdv) by RedDeerCityTV (https:[email protected]/), on Flickr
(https://c1.staticflickr.com/5/4282/34610387673_b85bb228b0.jpg) (https://flic.kr/p/UJphrX)
mlv raw converter2 (https://flic.kr/p/UJphrX) by RedDeerCityTV (https:[email protected]/), on Flickr
(https://c1.staticflickr.com/5/4260/35290258091_b3a956b54a.jpg) (https://flic.kr/p/VLtNeH)
mlv raw converter3 (https://flic.kr/p/VLtNeH) by RedDeerCityTV (https:[email protected]/), on Flickr

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on June 20, 2017, 10:42:58 AM
@reddeercity
Thank you very much!!!

But... I'm on windows 10 now and mlv_dump processes mlv files in separate folders when I select all and use the "send to" comand found here http://www.magiclantern.fm/forum/index.php?topic=10526.msg102123#msg102123 (http://www.magiclantern.fm/forum/index.php?topic=10526.msg102123#msg102123)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on June 20, 2017, 11:27:12 AM
Anyway,
I think that using the workflow reddeercity suggest, you have a lot of advantages. Now I'm using it!!
Thanks!!!!!!!!!!!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Oedipax on June 20, 2017, 05:32:45 PM
That made excellent help! Works good in ae also. What shutter speed do you use? At first I left it at 94, but will probably go up to 180 next time? As one can see its quite blurry, also shot from a train...:
(http://thumb.ibb.co/doMb55/Comp_1_00884.png) (http://ibb.co/doMb55)

(http://thumb.ibb.co/k5a8JQ/Comp_1_03840.png) (http://ibb.co/k5a8JQ)
I usually go by the 180-degree shutter rule of thumb, where the shutter speed is 2x that of your framerate. 24fps = 1/48 (or 1/50), and so on. So if you are shooting 45P, 1/90 (or 1/94 if that's what is available) on the shutter should be a good starting place. If it's still too blurry for your liking, you could keep pushing it further like 1/120 or 1/250 and see if you like that additional clarity.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kharak on June 20, 2017, 07:40:59 PM
Could you post an example MLV with stripes?
Are you sure vertical stripe code is running on all clips?

Perhaps tomorrow or the day after I'll have access to high speed internet. I will send you a MLV.

Yes, mlv_dump says Vertical Stripe Correction and those numbers appear aswell.

Just saw that Bouncyball made a new version of mlv_dump on steroids --force stripes could be a solution?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: g3gg0 on June 20, 2017, 10:53:04 PM
tried the flat (https://www.magiclantern.fm/forum/index.php?topic=19300.msg185962#msg185962) field (http://www.magiclantern.fm/forum/index.php?topic=17795.msg175189#msg175189) videos to get rid of stripes?
tend to be the most reliable ones, as they are not just guessed from one frame and also care for speckles etc
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kharak on June 22, 2017, 02:34:26 AM
No I have not tried flat field yet.

Current conditions are not ideal to make a flat frame. But I will try it out, for sure.

On another note, the --force-stripes has done a terrific job at removing the stripes, there are stripes, but virtually invisible. But it is slow.. Very slow.. Also with no batch process I am seeing a lot of issues ahead. I have a music video coming up, that I wanted to shoot all 60p, but with so many manual steps, decompressing and sorting, converting and having to stay glued at the screen to start next process when one is done, I am feeling like this is way too much to handle. Blaaah..

Screenshot with diagonal split -
https://mega.nz/#!QFYSwZRC!cpK0HV0lpmVNR5JAtETDqA7e5rGMb7LEieT8JGeYGQs

I hope the jpeg compression will do it enough justice for you to see. I did not do darkframe subtraction on the --forced-stripe footage. Did not look in to how I could do both with the Steroids_dump
 

Do you think the flat frame would be a quicker step ? It will also require sorting, even more so I assume, each lens to its own? each f-stop to its own ? Darkframe+Flatframe sounds like the ultimate quality we can get with our Canons, but it seems overly complicated for a project shot with multiple lenses at various f-stops and having some manual lenses, it will be hard matching f-stops to the right flatframe and on top of that having to write in the commands for each shot is very tedious. For photography, it seems like a no-brainer though.

@danne
I'll have 3G access tomorrow, I'll send you the mlv.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on June 22, 2017, 07:09:29 AM
Flatframe for each lens is needed. Not sure of the ideal f-stop when it comes to flat frames. I had good results with around f5.6. I used light from a strong in door lamp. Pointing to the sky is probably better.
Since you are getting better results with --force stripes with the steroid version it indicates that mlv_dump is either using different code or isn't always on.
There are newer and older vertical stripes code so it could be an idea to compare those maybe? I think steroid version is using what is used in mlvfs(older mlv_dump code), havn't checked.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: D_Odell on June 22, 2017, 09:09:50 AM
If someone had and found a solution to crop rec: cache is locked down? How do you free it?
Thanks!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on June 22, 2017, 09:25:46 AM
Quote
cache is locked down?
What do you mean?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: D_Odell on June 22, 2017, 09:30:27 AM
What do you mean?

(http://thumb.ibb.co/k5wbOQ/IMG_0571.jpg) (http://ibb.co/k5wbOQ)

(http://thumb.ibb.co/gLOO3Q/IMG_0572.jpg) (http://ibb.co/gLOO3Q)
 Or is it nothing to worry about?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: D_Odell on June 22, 2017, 09:53:31 AM
Also the colour green is now yellow or orange in Live view. Will update how the dng:s are. Weird, when removing a perfectly balanced ND 9 filter the colours are good again.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on June 22, 2017, 12:58:40 PM
Here´s some more testing around HDR filmed like this:

Added darkframe averaged file to cleanest image possible(decompressed). Note that adding darkframe in this case probably won´t make much difference. Already got plenty light to work with.
Tried combining with an averaged flatframe file but got weird results. Didn´t test with flatframe solely.

Settings in Crop mode:
1920 50/60 3x3 (PAL) 50fps

RAW video settings:
1720x960 50fps(continuous)
Processed with after effects HDR finished result 25fps

With some trickery I use a HDR AE template in cr2hdr.app which goes through aerender(command line render engine in after effects). In main menu select (p) then (23) for a template. Your MLV files have to be transcoded into folders with dng files through cr2hdr.app.

There is a bug with 1920 50/60 3x3 setting. Once you exit liveview after filming you can´t open liveview again without getting a completely black screen. Exiting the selected mode opening up liveview and reselecting the mode will make it work again.

Some examples:
(https://s8.postimg.org/kl5e8lfid/01a.png)

(https://s8.postimg.org/l0go1m1fp/02a.png)

(https://s8.postimg.org/gfuhmohqd/03a.png)

(https://s8.postimg.org/ey91b4czp/01b.png)

(https://s8.postimg.org/dzso8tznp/02b.png)

(https://s8.postimg.org/x65vc0g5h/03b.png)

        Movie sequence checking movement and ghosting related to 50fps
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Wayne H on June 24, 2017, 06:49:45 PM
Tested the 3.5k crop mode 3584x1320 on the lowest compression setting (11-8bit) Lossless, to see how it holds up, pretty pleased with how stable it is, no dropped frames or automatic stoppage's despite shooting some highly detailed scenes, it also preserved excellent colour detail too, fantastic job Alex.

As usual liveview is a challenge to work with, but workable with some extra attention, also there one clip that is standard 1920x804 cuts very well with the crop clips, see if any of you can guess which one it is?

EF70-200
FCPX, Colour Finale

https://www.youtube.com/watch?v=Ax0BL_Ci9po&t=5s
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: bastonford on June 26, 2017, 02:30:48 PM
Hi folks - recent convert from GH4 and love it. I noticed your comment below @g3gg0

having patches on my disk.
wont take too long.

I assume that maps to this PR: https://bitbucket.org/hudson/magic-lantern/pull-requests/837/mlv_snd-and-dual_iso-support-for-mlv_lite ? If so, would have some testing done help at all? Happy compiling and checking it out on the 5d3 if that's of use.

John
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kharak on June 26, 2017, 05:11:36 PM
@A1ex

Just tested the June 19th Build, it fixed the ISO 100 Raw Zebras bug and Histogram bug. Thank you very much!

Not that I use the h264 Spotmeter for exposure, but just tested it aswell. Instead of maxing out at 96%, it maxes at 97%. Don't think it matters for the 3x3 60p mode, its Raw only right? just so you know. Personally I don't use h264, ever.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on June 26, 2017, 05:20:47 PM
Happy compiling and checking it out on the 5d3 if that's of use.

Yes, please.

Not that I use the h264 Spotmeter for exposure, but just tested it aswell. Instead of maxing out at 96%, it maxes at 97%. Don't think it matters for the 3x3 60p mode, its Raw only right?

If it's a percentage or RGB, it's YUV-based. If it's EV, it's RAW-based.

(Side note: I think I'd also like a Kelvin spotmeter)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: nikki on June 26, 2017, 09:09:43 PM
Tested the 3.5k crop mode 3584x1320 on the lowest compression setting (11-8bit) Lossless, to see how it holds up, pretty pleased with how stable it is, no dropped frames or automatic stoppage's despite shooting some highly detailed scenes, it also preserved excellent colour detail too, fantastic job Alex.

As usual liveview is a challenge to work with, but workable with some extra attention, also there one clip that is standard 1920x804 cuts very well with the crop clips, see if any of you can guess which one it is?

EF70-200
FCPX, Colour Finale

https://www.youtube.com/watch?v=Ax0BL_Ci9po&t=5s

what was you post workflow on this?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kharak on June 27, 2017, 02:45:36 AM
@a1ex

June 19th Build:

fir 123

I can not playback 14 bpp lossless - 3x3 60p in camera. Just a lot of noise and flickering.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kharak on June 27, 2017, 03:16:05 AM
(Side note: I think I'd also like a Kelvin spotmeter)

That sounds really cool, so if I understand correctly I could measure the Kelvin Temperature instead of exposure? Does any camera do that?, I never even thought about that possibility. Would it measure on a RAW basis or depending on the liveview "colours", could a Picture Profile skew it?

That would be a tremendous help in mixed lighting.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: mr.smith on June 27, 2017, 04:51:27 AM

Can I ask some questions here about 5D3's experimental build?
I don't know where to ask. If it is wrong place, I apologize.


1.
I always think that why there is no standard resolution such as below?
Are these size so difficult to build?
Probably I think almost people using ML on 5D3 wish for them below.

2880x1620(3K HD 16:9)
3072x1620(3K 1.9:1)
3840x2160(4K UHD 16:9)
4096x2160(4K 1.9:1)

Why is it only incomplete resolution now, such as 3072x1920 or 3840x1600?


2.
I want to use 4096x2560 for the stabilized footage of 3840x2160.
Is it impossible to record at 24p?
(8bit, 10bit or 12bit raw is fine. H.264 is also fine.)
Can someone please make it?


3.
I don't want to push the crop button.
Is it impossible to be crop mode from the beginning?
I think it seems save time. What do you think?


4.
I want to participate in ML development, but I don't have the knowledge of it.
What should I study for ML development?


Always thank you for development of ML.
You saved my old 5D3 camera.
I want to use it on the 4k field.

Hope your reply soon.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: bastonford on June 27, 2017, 07:43:13 PM
Yes, please.

So compiling the PR for audio (with the 11 commits to crop_rec_4k since it was opened), gives me the following on 5D3.113:

- Modules enable fine and I can enable mlv_sound in the sound menu
- Sound records in MLV videos in full-sensor mode
- Sometimes when hitting stop, there is an error message on screen stating that "sound did not stop: state 5", however I can't reliably recreate it.
- There seems to be significantly more write speed required than "advertised" when sound is on - my slower Sandisk Extreme 120 card can't keep up for more than 400 frames with sound, when it can go continuous compressed raw without. Lexar Pro 1066x card is fine at 3k 2.39 though.

I'm a bit unsure of what the outcome should be when recording sound in the new crop_rec mode. There is a WAV output at the end of the run with mlv_dump, but it is only 44 bytes on crop_rec videos. Still new around here - is mlv_dump another compilation target within the magic lantern repo? I'll get googling.

Will report back as I come across more.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Wayne H on June 27, 2017, 08:45:47 PM
what was you post workflow on this?

Hi nikki, i imported all MLV files into martin herings great mac app called FOOTAGE, corrected colour balance and then applied C Log, exported to prores 4444.

Setup a 4k timeline within FCPX, imported the 3584x1320 MLVs and used Color Finale Pro to add alexa LOG C LUT, from lut utility within color finale, then used the colour wheels and vectors to colour grade.

Exported as a 4K H.264 file and uploaded to youtube as a 4K file.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kharak on June 28, 2017, 06:32:08 PM
@a1ex

I was reading in the MLV_Play thread that disabling raw_twk should make playback work, I tried that and it did not work on 60p mode, "No decompression in this mode".
http://www.magiclantern.fm/forum/index.php?topic=9062.msg186484;topicseen#msg186484 the thread.

Does raw_twk affect recording in any way? I've just had it on all the time since lossless was released, I forgot about it, does it have any other purpose than speeding up playback? (which was it's original intent, AFAIK)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: D_Odell on June 28, 2017, 11:05:26 PM
Since I only use older lenses (nFD glas), around 30-40 years of age. Would it be better to choose a mode that uses the center parts of the sensor more than the higher part of lens glas? In my experience the UHD options render only from the top part of the sensor while center, yeah is center.

Any thoughts on which options is best to use, I mean in terms of lens qualities and DNGs to use?

Option 1: UHD 1:1 3232 x 1376
Center crop 3,5k: 3232 x 1376

From a sequence I shot today I experienced a lot of CA and bleeding. Maybe it's just the lens quality?

DNG: https://www.dropbox.com/s/ffr4mgcz0argy7o/M28-1359_001359.dng?dl=0
This one uses UHD 1:1 option.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kharak on June 28, 2017, 11:29:45 PM
All lenses normally have most sharpness in the Center, so it should always be the best option unless you want CA.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: D_Odell on June 29, 2017, 10:11:00 PM
Realised that crop-rec 3,5k doesn't cooperate as well as UHD at higher resolution. Anyone experienced the same?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: lostfeliz on July 02, 2017, 09:26:35 PM
@bastonford, so glad you're compiling this! Good luck with it. For either run & gun with a Rode VidPro or for dual system audio having the WAV recording in the 5Diii is huge help. It'll add years of usefulness to my camera.

Great to hear you're getting continuous 3k and sound.

I stopped playing with card spanning a long time ago but I always thought it'd be great if the audio could write to the SD card while the video wrote to the CF card. I'm guessing there are underlying Canon issues keeping that from happening or that it doesn't help write speeds at all.

So compiling the PR for audio (with the 11 commits to crop_rec_4k since it was opened), gives me the following on 5D3.113:

- Modules enable fine and I can enable mlv_sound in the sound menu
- Sound records in MLV videos in full-sensor mode
- Sometimes when hitting stop, there is an error message on screen stating that "sound did not stop: state 5", however I can't reliably recreate it.
- There seems to be significantly more write speed required than "advertised" when sound is on - my slower Sandisk Extreme 120 card can't keep up for more than 400 frames with sound, when it can go continuous compressed raw without. Lexar Pro 1066x card is fine at 3k 2.39 though.

I'm a bit unsure of what the outcome should be when recording sound in the new crop_rec mode. There is a WAV output at the end of the run with mlv_dump, but it is only 44 bytes on crop_rec videos. Still new around here - is mlv_dump another compilation target within the magic lantern repo? I'll get googling.

Will report back as I come across more.

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: bastonford on July 03, 2017, 09:34:31 PM
I'm still a little stuck on this - Footage and mlv_dump seem pretty convinced there is sound in there for higher resolutions looking at the metadata, but when exported it's just an empty WAV file. I'm still trying to work out if this an in-camera thing or mlv_dump that isn't doing the right thing.

I have to say though, on the 5d3, the FF 1080p 14bit is stunning quality if you de-bayer it onto a 4k timeline in Resolve, rather than upscaling the video at the end of mastering it. Obviously 4k is way slower to work with, so I tend to edit in on a DCI2k 2.39 timeline, and then just flick the timeline resolution up to the 4k option before delivery. Not saying that higher resolutions aren't necessary, but the most exciting thing for me is 14 bit compressed RAW with audio - a ~45% space saving makes a huge difference on bigger projects!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: bastonford on July 03, 2017, 10:40:43 PM
Compiling the latest state of the "crop_rec_4k_mlv_snd" branch gives me this error on all takes using the 3k 1:1 mode.

Code: [Select]
ML ASSERT:
slots[slot_index].size < max_frame_size
at mlv_lite.c:3335 (raw_video_rec_task), task raw_rec_task
lv:1 mode:3


Magic Lantern version : Nightly.2017Jul03.5D3113
Mercurial changeset   : 1ed473161338 (crop_rec_4k_mlv_snd)
Built on 2017-07-03 20:05:26 UTC by [email protected]
Free Memory  : 134K + 3091K

This is with a Lexar 1066 card, and it seems to run fine for about 8-10 seconds and then bug out with that error dumped to the card. This is mlv_sound module enabled and the module turned on in the audio menu. On the normal crop_rec branch the 3k is continuous without any issues.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: g3gg0 on July 05, 2017, 01:52:00 PM
okay interesting.

can you tell me the exact settings so i can try to reproduce?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: bastonford on July 05, 2017, 04:43:54 PM
Of course - thanks for looking into it.

5D3.113 build of commit 1ed47316133855e7f459c98ec52f4e79e4f27952

Modules enabled:
- bench
- crop_rec
- mlv_lite
- mlv_play
- mlv_sound

Sound enabled in the Audio menu

Canon settings:
1080/24P All-I

ML Movie settings:
FPS override off or exact 23.976 (doesn't seem to matter which)
Image Fine Tuning -> Shutter +0.84ms (1/48)
Crop mode either 3k or UHD 1:1
Resolution 3072 in 2.39 crop
SRM memory enabled (disabling it shortens record time for me)
14bit lossless

Edit: Global Draw in auto mode

Happy to try anything to help - just let me know! Does this work for you? What are you using to convert the MLV files?

Thanks :)

John
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Savely on July 06, 2017, 11:53:26 AM
Can someone give a clue how to make camera to rec h264 proxy on SD and not on CF card?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Jerome6405 on July 07, 2017, 05:20:18 PM
Hi there,

I tested this experimental firmware and that's amazing the video I can get from my "old" 5d mark iii, a big thanks to the community and THE developers.

My only concern is about missing mlv_snd module which is for me the reason I won't keep this firmware on my camera right now but I've seen that you are working on a subbranch which add this module, is it already somehow working or blocking issues are still to be fixed ?

I would have love to help (being myself a developer but more web/mobile oriented), I dont have the level I guess to dig into C/Perl code base.

Thanks again and if you need testers I am here.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: etienne on July 08, 2017, 04:42:48 AM
Can someone give a clue how to make camera to rec h264 proxy on SD and not on CF card?

In ML Menu under the tool icon you will find "Card Settings" in that menu you will find Preferred Card choose SD Reboot and your done.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Savely on July 08, 2017, 02:06:30 PM
In ML Menu under the tool icon you will find "Card Settings" in that menu you will find Preferred Card choose SD Reboot and your done.

Thanks, I did, but it still doesn't rec h264. So I assume this option doesn't support any crop_rec mode?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: nlp on July 12, 2017, 08:59:31 PM
Anyone know why I get blank screen playbacks for anything other 14bit?  Anything recording in 10 or 12 bit at any frame rate on playback its just black.  Thanks
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: etienne on July 13, 2017, 04:19:47 PM
Thanks, I did, but it still doesn't rec h264. So I assume this option doesn't support any crop_rec mode?

Exact it works only when realtime liveview is availaible it does work however with the crop mode enabled with the zoom (not the one inside ML)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: The_bald_guy on July 14, 2017, 05:05:29 PM
Can you direct me where I can download the update to record sound on the lastest build? I dont know how to work around bitbucket.
I basically want to record sound in regular 24p 1080p but I want the 50p 3x3 option from the latest update (I dont need sound for this option).

I just dont want to change cards each time I need sound.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on July 14, 2017, 06:31:25 PM
@The_bald_guy https://builds.magiclantern.fm/jenkins/job/crop_rec_4k/36/artifact/platform/5D3.113/magiclantern-crop_rec_4k.2017Jun19.5D3113.zip (https://builds.magiclantern.fm/jenkins/job/crop_rec_4k/36/artifact/platform/5D3.113/magiclantern-crop_rec_4k.2017Jun19.5D3113.zip)
This is for 5D3 fir 113
Oh wait sorry I'm not sure if this version records sound. I missed that you needed that.
I think you have to compile it if you want sound.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Boygetslost on July 17, 2017, 04:06:07 AM
Hi everyone
I have been using the Jun 19 crop_rec on 5d iii with kombuterbay 1106 card.

Most of the higher res crop modes stop recording almost immediately, I get maybe a second.

If there something I'm missing here? Something I haven't set up right.

Any advice would be greatly appreciated.

Thanks
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: fvanlint on July 17, 2017, 09:56:15 AM
You probably forgot to set FPS override.

Hi everyone
I have been using the Jun 19 crop_rec on 5d iii with kombuterbay 1106 card.

Most of the higher res crop modes stop recording almost immediately, I get maybe a second.

If there something I'm missing here? Something I haven't set up right.

Any advice would be greatly appreciated.

Thanks
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: D_Odell on July 17, 2017, 03:05:07 PM
Hi everyone
I have been using the Jun 19 crop_rec on 5d iii with kombuterbay 1106 card.

Most of the higher res crop modes stop recording almost immediately, I get maybe a second.

If there something I'm missing here? Something I haven't set up right.

Any advice would be greatly appreciated.

Thanks
Which settings do you use?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Boygetslost on July 17, 2017, 03:58:39 PM
This is what I have been using. On the crop modes It just starts recording then stops.
The normal modes like 1920 1080 or 1920 1220, seems to work fine.

Any ideas? Am I missing a setting? Or using a wrong build?

Thanks
Chris
(http://thumb.ibb.co/ey8bxv/IMG_6810.jpg) (http://ibb.co/ey8bxv)

(http://thumb.ibb.co/kvJ9Hv/IMG_6811.jpg) (http://ibb.co/kvJ9Hv)

(http://thumb.ibb.co/eeqBVF/IMG_6812.jpg) (http://ibb.co/eeqBVF)

photos to upload in fb (http://imgbb.com/)

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Walter Schulz on July 17, 2017, 04:41:13 PM
The settings you're showing are far from being sufficient. Compare to first post of this thread and read a1ex' remarks about this mode.
-> Frozen LV, low ISO and scene allowing good compression rate.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: rob_6 on July 17, 2017, 06:35:12 PM
@boygetslost,

Check out this video created by hjfilmspeed! It was a huge help for me! Thanks again hjfilmspeed!

Rob

@whysodifficult @rob_6
This video is not for those new to ML. Try this at your own risk. I am not responsible for any damage this video may bring if your camera explodes or bricks. Magic Lantern is not responsible either. It will probably just brick not explode though.
Magic Lantern 3072 2.35:1 12 bit Lossless RAW Recording from the CROP_REC_4K build from 4-29-2017
Once you have ML installed and the correct build on your SD card, this video should help you set up your 5DIII to record 3K almost continuously depending on the scene. Remember ML on SD and Rec to CF.
Remember this build is experimental and not for those that are new to ML. Try at your own risk.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Boygetslost on July 18, 2017, 07:21:44 AM
Thanks Rob!

Just what I needed, I have a much better understanding of it now.

Getting much better results.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Savely on July 18, 2017, 07:17:58 PM
Has anyone noticed bug on x5 crop_mode (3,5k) - sometimes center adjusted to the right. I mean you have an object almost perfectly at center and when you see a footage, it's on the right side of picture. Yet can't figure out what it depend of.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: sys73r on July 20, 2017, 02:42:19 AM
@The_bald_guy https://builds.magiclantern.fm/jenkins/job/crop_rec_4k/36/artifact/platform/5D3.113/magiclantern-crop_rec_4k.2017Jun19.5D3113.zip (https://builds.magiclantern.fm/jenkins/job/crop_rec_4k/36/artifact/platform/5D3.113/magiclantern-crop_rec_4k.2017Jun19.5D3113.zip)
This is for 5D3 fir 113
Oh wait sorry I'm not sure if this version records sound. I missed that you needed that.
I think you have to compile it if you want sound.

hey @hjfilmspeed is there any thread which shows how to compile a module? I'm looking for sound on the crop rec as well, happy to test out.  cheers.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: budafilms on July 25, 2017, 06:36:52 PM
Sound, when you shoot is a good idea.
Please, share the build.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Rcercer on July 28, 2017, 07:13:49 AM
Is  possible to use these [email protected] without RAW mode? In ALL-I maybe? Thank for your atention!!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Savely on August 02, 2017, 12:53:28 PM
Is  possible to use these [email protected] without RAW mode? In ALL-I maybe? Thank for your atention!!

If that'd be possible there wouldn't be needing for Magic Lantern at all.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: etienne on August 03, 2017, 12:16:26 AM
Any news on bringing the MLV_SOUND to work with that branch?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: sylvain_c on August 03, 2017, 06:29:43 AM
Is anybody know how to achieve 3072x1920, I can't find the good set up in the camera to go up to 3072x1320?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: g3gg0 on August 03, 2017, 09:24:48 PM
Any news on bringing the MLV_SOUND to work with that branch?

sorry, not yet. quite busy with having a life ;D
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on August 04, 2017, 06:45:39 AM
Is anybody know how to achieve 3072x1920, I can't find the good set up in the camera to go up to 3072x1320?
This could possibly help:
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: motionSOUL on August 05, 2017, 03:35:10 PM
Totally dumb question (but to better understand what is going on): what are the differences between 3.5K 1:1 centered x5 and 3K 1:1 for the end result? Why using the first one over the second or vice versa?
I have managed to record 3072x1286 - 2.39:1 files with the two with correct LiveView Preview (Frame mode / lagged) and very close framing (just few pixels differences)...

Thanks,

S/.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: etienne on August 09, 2017, 06:29:51 AM
sorry, not yet. quite busy with having a life ;D

this is totally unaccaptable!!!  So just to get an idea is it something that just needs works to be done or is it something that needs  some massive issue to be fixed and is yet still uncertain?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Walter Schulz on August 09, 2017, 07:17:24 AM
Stop trolling around, etienne.
This project is done by highly skilled people in their spare time not consumed by work, family, friends, household duties and other hobbies. For free, given to you free (as in beer). There is no schedule, time table, milestone to be fulfilled.
Top of page -> User Guide -> FAQ -> Troll questions sections

If you can't live with that you are free not to use ML but professional solutions more suitable to your needs.

Ceterum censeo experimental builds page esse delendam.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: denishogan on August 11, 2017, 01:04:18 AM
Hi All

Thanks for this new enhancements. I will give this a go this weekend ie  crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView

Has anyone tried the  "Full-resolution LiveView: 5796x3870 at 7.4 fps (128ms rolling shutter) - continuous*) at 5 FPS!", if so, are you able to share the settings.

I may be doing something wrong as when I select this option the liveview on the camera is showing a pink scrambled screen. Any ideas or suggestions?

cheers
Denis
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on August 12, 2017, 12:26:22 AM
@denishogan im not sure but it sounds like your preview might be set to canon mode or real time. It should be set to framing. I would have to double check that tho.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: etienne on August 12, 2017, 05:34:17 AM
Stop trolling around, etienne.
There is no schedule, time table, milestone to be fulfilled.
Top of page -> User Guide -> FAQ -> Troll questions sections
Hi walter,
just read that faq didn't saw mine :)

Ok Sorry if i did not make myself serious enough. Just want to assure you that my question was definitly not about the when?

but just asking (to whom have taken a look into it) is this a reasonable hope to have? or do there is evidence already that it could turn out to be something like the RT preview with a big question mark on if this is even doabble...

but rest assure i can definitly live without just as i can definitly live without even getting that answer.

Have fun & peace.
D.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: denishogan on August 12, 2017, 12:29:15 PM
Thanks hjfilmspeed

I followed your video with the exact settings but encountered 2 errors described below. I also then tried Full-resolution LiveView: 5796x3870 at 7.4 fps and got the same errors. The attached images are the settings for "full-resolution LiveView: 5796x3870 at 7.4 fps"

MY SUMMARY:
I tried the following :

1) video settings by Jonathan DeNicholas (https://vimeo.com/217313287#comment_15952088) How ro Record 3k on 5dmkIII with Magic Lantern
2) Full-resolution LiveView: 5796x3870 at 7.4 fps (128ms rolling shutter) - continuous*) at 5 FPS! - seeting at  14-bit lossless compression mode (LJ92), preview set to Frozen LV (previously known as Hacked Preview)

For both scenarios above the movie recorded and I extracted the DNGS using MLV convertor but could not open the DNGS in Photoshop CC 2017.
-The error was "Could not complete your request because the file-format module cannot parse the file." , see   attachments
- In bridge I can see each DNG file has many dots of different colors. ." ,  attachments


OTHER DETAILS
Canon 5dmkIII
Card 64gb 1066 compact flash
Using latest crop_rec_4k.2017 Jun 19.5d3123.zip
Using MLV convertor to extract DNGs

Photoshop error reading the DNG
(http://thumb.ibb.co/kKY6sF/8_12_2017_2_07_11_PM.jpg) (http://ibb.co/kKY6sF)

 What I see in BRIDGE- colored dots
(http://thumb.ibb.co/bDFKCF/8_12_2017_2_53_04_PM.jpg) (http://ibb.co/bDFKCF) 
 
(http://thumb.ibb.co/hQPUCF/8_12_2017_2_54_06_PM.jpg) (http://ibb.co/hQPUCF)
 (http://thumb.ibb.co/mv7MRa/006.jpg) (http://ibb.co/mv7MRa) (http://thumb.ibb.co/gKtHzv/005.jpg) (http://ibb.co/gKtHzv) (http://thumb.ibb.co/n9Co6a/IMG_9999.jpg) (http://ibb.co/n9Co6a) (http://thumb.ibb.co/kthSzv/IMG_0005.jpg) (http://ibb.co/kthSzv) (http://thumb.ibb.co/cLmWRa/IMG_0003.jpg) (http://ibb.co/cLmWRa) (http://thumb.ibb.co/bZF2XF/IMG_0001.jpg) (http://ibb.co/bZF2XF)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on August 12, 2017, 12:48:31 PM
For developing dng files on windows try this for lossless.
http://www.magiclantern.fm/forum/index.php?topic=10526.msg186929#msg186929
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: baccinoluigi on August 12, 2017, 12:54:50 PM
Quote
Thanks hjfilmspeed

I followed your video with the exact settings but encountered 2 errors described below. I also then tried Full-resolution LiveView: 5796x3870 at 7.4 fps and got the same errors. The attached images are the settings for "full-resolution LiveView: 5796x3870 at 7.4 fps"

MY SUMMARY:
I tried the following :

1) video settings by Jonathan DeNicholas (https://vimeo.com/217313287#comment_15952088) How ro Record 3k on 5dmkIII with Magic Lantern
2) Full-resolution LiveView: 5796x3870 at 7.4 fps (128ms rolling shutter) - continuous*) at 5 FPS! - seeting at  14-bit lossless compression mode (LJ92), preview set to Frozen LV (previously known as Hacked Preview)

For both scenarios above the movie recorded and I extracted the DNGS using MLV convertor but could not open the DNGS in Photoshop CC 2017.
-The error was "Could not complete your request because the file-format module cannot parse the file." , see   attachments
- In bridge I can see each DNG file has many dots of different colors. ." ,  attachments

Try to turn off fps override, i've never used it when i shoot in full resolution
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: denishogan on August 12, 2017, 01:28:34 PM
Thanks baccinoluigi for your prompt response. I will try that now.

I normally use Full RES DNG photo mode for extreme macro to save on my shutter count as some of these stacks take 100's to 1,000's of pics.

I am trying to do the same but using 4k video to save time. The example below is a reduced size macro. The full Res version is 10,467 pics Panorama (16,424 X 18,674 ) using magic lantern and perhaps a few actuations only. So my desire to get the 4k video Full Res working as that took nearly 2 weeks elapsed time in photo mode.


(http://thumb.ibb.co/dvOEUv/18556810_10155334927359810_1236929900883014327_o.jpg) (http://ibb.co/dvOEUv)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: denishogan on August 12, 2017, 01:44:43 PM
baccinoluigi ,

I tried turning FPS off but same issue.  DNG has coloured spots and not recognised by CC 2017

I believe it has to be on as in FULL RES we must sate the FPS as shown in the documentation ie 7.4 fps , I believe

cheers
Denis
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: denishogan on August 12, 2017, 04:15:46 PM
Thanks Danne for the link, I will try it, cheers
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: denishogan on August 12, 2017, 04:36:42 PM
Hi Danne,

I did as you suggested but nothing occured ie no DNGs created but not sure what I did was correct with your utility.

I copied the utility into the folder with the MLVs
I am not sure what to do but at the cmd prompt, I selected 2 numbers, #6 then pressed return, then selected 11 and pressed return key. See the attached black screen image. Then I pressed "R" key and pressed enter... It seemed to do something. See the second screen capture. I copied the original MLVs into a folder called Originals and created 7 folders in the red box that I number as #3 however there are no files in it.


(http://thumb.ibb.co/j62Y6a/8_12_2017_10_26_22_PM.jpg) (http://ibb.co/j62Y6a)

(http://thumb.ibb.co/msSpev/8_12_2017_10_27_52_PM.jpg) (http://ibb.co/msSpev)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on August 12, 2017, 04:45:43 PM
Just put the MLV files inside the batch_mlv folder and then double click batch_mlv.cmd :).
Then do additional settings from the menu and when finished select (r).

The automation process as follows:
MLV files are processed into folders with dng files and then the original MLV fileas are moved into a folder called A_ORIGINALS

The process is automated so user don´t need to worry about creating any folders.

Should look like this.
(https://s23.postimg.org/g3oql07rv/Screen_Shot_2017-08-12_at_16.55.27.png)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: denishogan on August 12, 2017, 04:51:10 PM
Danne

I did do that twice and it create folders but there are no DNGS in the folders

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on August 12, 2017, 04:53:59 PM
Hm, I see. Could you upload a sample file? If I can narrow down any issue I will fix it.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: denishogan on August 12, 2017, 04:54:34 PM
These are my MLV files that I put in the batch_mlv folder


(http://thumb.ibb.co/dcXQma/8_12_2017_10_52_01_PM.jpg) (http://ibb.co/dcXQma)

(http://thumb.ibb.co/fTJnzv/8_12_2017_10_53_39_PM.jpg) (http://ibb.co/fTJnzv)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: denishogan on August 12, 2017, 04:57:40 PM
Danne, how do I upload a sample MLV. I clicked attachments and nothing happend
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on August 12, 2017, 05:00:13 PM
You need a server host, google drive, dropbox, sprend or similar will do.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: denishogan on August 12, 2017, 05:04:48 PM
Thanks Danne, I will upload to dropbox, may take .
wow, its very slow here, saying 1 hour left

In the meantime, base on the settings I used, is the Aspect ratio correct for the Full Res video of 7.4FPS. Alos I used Frozen LV
When I hit the play button I cannot see anything in Liveview on the camera, just pink scrambled lines
Is that supposed to be the behaviour.

Are you able to get the Full RES video of 7.4FPS. working?

cheers
Denis


Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: denishogan on August 12, 2017, 05:08:20 PM
BTW I also tried RawFlow.exe and it extracted the DNGS from the MLV but same issue.

I am beginning to suspect perhaps my settings are incorrect in Magic Lantern so my MLV files are getting corrupted maybe,
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on August 12, 2017, 05:24:22 PM
Could be some corrupted MLV file. Let's see. Important though that you're using the latest versions of mlv_dump when processing your files.
Gonna be away from my computer for a while. Checking in tonight.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: denishogan on August 12, 2017, 05:44:43 PM
ok Thanks Danne, its 11.43pm in Perth Australi, I am going to bed but will upload the file tommorow or if I wake earlier

Much appreciated

cheers
Denis
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: denishogan on August 13, 2017, 02:10:26 AM
Hi Danne

Here is a sample mlv as requested https://www.dropbox.com/s/lyjusbyn117t4xt/M12-1934.MLV?dl=0

Are there any Canon camera ( not ML)  video settings I should set?


cheers
Denis
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on August 13, 2017, 02:30:58 AM
Downloaded the file and run the batch_mlv.cmd command. All files were exported into dng files.

Note! Try all very basic when things are cracking. For example. A more basic folder name might work instead of with spaces. Worth a shot. My script is very basic.
*Can confirm spaces in root folder is not working atm with the script. Will check into fixing this.

After double click on batch_mlv.cmd this is my view. Then I just print r and enter and files are sent to a folder.
(https://s30.postimg.org/g893xx91d/Screen_Shot_2017-08-13_at_02.35.35.png)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on August 13, 2017, 03:00:14 AM
Here´s a version which might work with spaces. Can you try @denishogan?
https://bitbucket.org/Dannephoto/batch_mlv/downloads/batch_mlv.zip
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: denishogan on August 13, 2017, 04:32:04 AM
OK I will try now, but can you please confirm if you open a DNG in photoshop?

If not then corruption perhaps
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on August 13, 2017, 04:46:49 AM
Opens fine, no corruption.
Corruption most probably happens because of when you´re using older versions of mlv_dump.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: denishogan on August 13, 2017, 04:48:28 AM
Wohoo Danne, you are a champ. it worked. So if I want to add parameters what is the syntax? at the prompt do I enter "6 11" without the quotes and press return to run parameter 6 and 11)

Once again thank you so much my friend. I am so happy.

With the Full RES video at 7.4fps, I notice that live view is scrambled with pick lines which make it hard for me to focus ( as can be seen in those example DNGs') or compose. What are your recommendations of the settings below for FUll RES? I think I am using 16.9 or 1:1 . Just charging my batteries.



(http://thumb.ibb.co/cLmWRa/IMG_0003.jpg) (http://ibb.co/cLmWRa)

img safe free image host (http://imgbb.com/)
 


Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on August 13, 2017, 04:57:33 AM
Great :). Made me fix those damn folder space issues so all good.
You can add parameters one by one. For instance 1 then enter, then select next setting and so on. When done select r and enter.
Just reselect a setting number to erase it.
Happy shooting :)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: denishogan on August 13, 2017, 06:16:48 AM
Hi Danne,

Thank you so much for your help and patience.  Thanks for the update program and effort, hugs

On a seperate note, I also found a link that there is a firmware update for the d800 that will provide 8k

Now to find the firmware. I go to Nikon site but no mention about it

Thanks once again and have a great day

cheers
Denis
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: 12georgiadis on August 13, 2017, 10:33:25 AM
I just tested H264 proxy with sound + 1080p compressed raw and it works perfectly. However h264 are longer. to sync h264 proxy + sound + 1080 raw, I tried to determine the offset. It is around 2s but not regular. Sometimes +1i or 2, sometimes -1i or two. Is there a way to have a stable offset ? If yes, a simple ffmpeg script could sync everything.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: denishogan on August 13, 2017, 11:02:09 AM
Hi All

I have shot some video on Full Res 7fps and able to extract DNGS  ( thanks to Danne)  however the liveView of the camera is showing pink and scrambled lines. Any way around this as I cannot focus or compose a shot?


(http://thumb.ibb.co/dbk7zv/IMG_0009.jpg) (http://ibb.co/dbk7zv)



(http://thumb.ibb.co/h0dSzv/IMG_0011.jpg) (http://ibb.co/h0dSzv)


(http://thumb.ibb.co/eug2XF/IMG_0012.jpg) (http://ibb.co/eug2XF)

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: mothaibaphoto on August 16, 2017, 11:42:08 AM
Hi All

I have shot some video on Full Res 7fps and able to extract DNGS  ( thanks to Danne)  however the liveView of the camera is showing pink and scrambled lines. Any way around this as I cannot focus or compose a shot?
Tripod, compose, focus, shoot.
Or buy red weapon :)