Magic Lantern Forum

Developing Magic Lantern => Modules Development => Topic started by: a1ex on April 01, 2017, 11:15:41 AM

Title: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 01, 2017, 11:15:41 AM
As you probably have guessed from the latest developments (QEMU (http://www.magiclantern.fm/forum/index.php?topic=2864.125), EDMAC graphs (http://www.magiclantern.fm/forum/index.php?topic=18315.0), JPCORE (http://www.magiclantern.fm/forum/index.php?topic=18443.0), EEKO (http://www.magiclantern.fm/forum/index.php?topic=13408.msg175656#msg175656)), our understanding on how LiveView works has improved considerably. Finally, all my fiddling with QEMU, at first sight with little or no purpose for the everyday users (http://www.magiclantern.fm/forum/index.php?topic=2864.msg169970#msg169970), starts paying off.

Today, Magic Lantern proudly announces new ground-breaking features that were previously thought impossible or very hard to achieve.

We proudly present....

4K RAW Video Recording!

(http://a1ex.magiclantern.fm/2017Apr01/fishy_4k_april_fools.png)

DOWNLOAD (http://builds.magiclantern.fm/experiments.html)

Twitter (https://twitter.com/autoexec_bin/status/848102541305020416) announcement (https://twitter.com/autoexec_bin/status/848313311141789696)

On the 5D Mark III, you now have the following new resolutions:

* 1920x960 @ 50p (both 1:1 crop and full-frame - 3x3 pixel binning) - continuous*)
* 1920x800 @ 60p (same as above)  - continuous*)
* 1920x1080 @ 45p and 48p (3x3 binning)  - continuous at 45p
* 1920x1920 @ 24p (1:1 square crop) - continuous*)
* 3072x1920 @ 24p (1:1 crop)
* 3840x1536 @ 24p (1:1 crop) (corrupted frames at 1600)
* 4096x2560 @ 12.5p (1:1 crop) - continuous*) at 8 FPS
* 4096x1440 @ 25p (1:1 crop)
* Full-resolution LiveView: 5796x3870 at 7.4 fps (128ms rolling shutter) - continuous*) at 5 FPS!

The last feature complements the well-known full-resolution silent pictures (http://www.magiclantern.fm/forum/index.php?topic=12523.0) - the new implementation will be usable at fast shutter speeds, without the exposure gradient (http://www.magiclantern.fm/forum/index.php?topic=12523.msg120750;topicseen#msg120750) - but with rolling shutter (just like regular LiveView frames).

*) Continuous recording is achieved in 14-bit lossless compression mode (LJ92) (http://www.magiclantern.fm/forum/index.php?topic=18443.msg182074#msg182074), preview set to Frozen LV (previously known as Hacked Preview), low ISOs (about 55% ratio). With complex (highly detailed) scenes and/or higher ISO, the image doesn't compress very well, so recording may no longer be continuous.

Credits: Greg (full-width LiveView (http://www.magiclantern.fm/forum/index.php?topic=10111.msg123909#msg123909)), g3gg0 (video timer (http://magiclantern.wikia.com/wiki/VideoTimer), DIGIC registers (http://magiclantern.wikia.com/wiki/Register_Map) documentation and lots of other low-level insights).

Complete list of new video modes:
Code: [Select]
                                /*   24p   25p   30p   50p   60p */
    [CROP_PRESET_3X_TALL]       = { 1920, 1728, 1536,  960,  800 }, /* 1920 */
    [CROP_PRESET_3x3_1X]        = { 1290, 1290, 1290,  960,  800 }, /* 1920 */
    [CROP_PRESET_3x3_1X_48p]    = { 1290, 1290, 1290, 1080, 1080 }, /* 1920; 1080p45/48 <- 50/60p in menu */
    [CROP_PRESET_3K]            = { 1920, 1728, 1504,  760,  680 }, /* 3072 */
    [CROP_PRESET_UHD]           = { 1536, 1472, 1120,  640,  540 }, /* 3840 */
    [CROP_PRESET_4K_HFPS]       = { 2560, 2560, 2500, 1440, 1200 }, /* 4096 half-FPS */
    [CROP_PRESET_FULLRES_LV]    = { 3870, 3870, 3870, 3870, 3870 }, /* 5796 */

What else could you wish for?


Screenshots:

(http://a1ex.magiclantern.fm/2017Apr01/4k.png) (http://a1ex.magiclantern.fm/2017Apr01/50p.png) (http://a1ex.magiclantern.fm/2017Apr01/allmodes.png)



FAQ

Where's the catch?

This is only a very rough proof of concept. It has not been battle-tested and has many quirks. Some of them may be easy to fix, others not so. In particular:

* It feels quite buggy. I'm still hunting the issues one by one, but it's hard, as Canon's LiveView implementation is very complex, and our understanding on how it works is still very limited.
* Write speeds are high. For example, 10-bit 4096x2500 at 15 fps requires 180 MB/s. 1080p45 should be a little more manageable at 111 MB/s.
* Canon preview is broken in most modes; you need to use the grayscale preview in the raw recording module.
* High-resolution modes (in particular, full-res LiveView) may cause trouble with memory management. This is very tricky to solve, as we only get 3 full-resolution buffers in LiveView, with restrictions on the order in which they must be freed, and lots of other quirks.
* Since these settings were pushed to limit, the risk of corrupted frames is high. If it happens, decrease the vertical resolution a bit (from the crop_rec submenu).
* When refreshing LiveView settings, the camera might lock-up (no idea why). Pressing MENU twice appears to fix it.

May I fine-tune the new modes?

Yes! I've included some of the knobs on the user interface. Normally you shouldn't need to touch these buttons, but if you do, you might be able to squeeze a few more pixels.

Does it work with FPS override?

Sort of. It's not reliable at this point, so it's best not to try yet.

Overheating?

During my tests, I didn't manage to get a sensor temperature higher than 60 degrees. Your mileage may vary.


Risks?

This mod changes some low-level sensor parameters that are not well understood. They were all figured by trial and error, and there are no guarantees about the safety of these changes.

As usual, if it breaks, it's your fault, sorry.

Will it work on other camera models?

I hope so; however, this is an area where I hope to get contributions from others (yes, from you). If these new features don't motivate you to look into it, I wonder what else will.

I'll explain how all this works in the coming days or weeks.

Is it difficult to port to other camera models?

So far, the 3x3 720p mode from crop_rec was ported to EOS M (rbrune) and 700D (dfort). So it shouldn't be that hard...

Will you port this to my camera model, please?

No, sorry. I have better things to do - such as, preparing the April 1st prank for next year :)

Wait a minute, didn't you say you are primarily a still photo user? Why are you even doing this?

If you look close, the usefulness for video is fairly limited, as the write speeds (and therefore the recording times) are not practical.

But the full-resolution LiveView is - in my opinion - very useful for still photo users. Although the current implementation is not very polished (it's just a proof of concept), I hope you'll like the idea of a 7.4 FPS burst mode, 100% silent, without shutter actuations.

Right now, you can take the mlv_lite module with pre-recording and half-shutter trigger: at 10 bits per pixel, you get 5 frames pre-recorded, and saved to card as soon as you touch the half-shutter button. Or, you can capture one frame for each half-shutter press, with negative shutter lag! (since the captured frame will always be pre-recorded).

And if a burst at 7.4 fps is not enough, you may also look at the 4K modes (12-15 fps).

(I know, I know, GH4 already does this, at much higher frame rates...)

The help menu for full-res LiveView says 5796x3870, but MLV Lite only records 5784x56. What's going on?

The raw recording modules have a couple of alignment constraints (e.g. can only start cropping from a multiple of 8 pixels, and the size of the cropped area (that goes into the MLV file) must be multiple of 16 bytes (that is, W*bpp/8 + H mod 16 must be 0).

To capture the full resolution, you may use the silent picture module. However, this module is not the best when it comes to memory management and buffering. Currently, you'll get an impressive buffer of 2 frames in burst mode :)

But hey - it outputs lossless DNG!

What about that lossless compression routine?

It's included, although I didn't manage to test it much. There is a lot of room for improvement, but for a proof of concept, it seems to work.



P.S. The initial announcement was disguised as an April Fools joke, just like the original crop_rec (http://www.magiclantern.fm/forum/index.php?topic=17021.0).

Twitter announcement (https://twitter.com/autoexec_bin/status/848102541305020416)

From original April Fools post:
Quote

With our latest achievements in wizardry with ARM programming and DIGIC reverse engineering, we can speak of a new era of raw video recording.

On models like the 5D Mark III, the next upcoming releases will feature an improved version of our crop_rec module (http://www.magiclantern.fm/forum/index.php?topic=17021.0) that delivers the following new resolutions:
 
* 1920x960 @ 50p (both 1:1 crop and full-frame - 3x3 pixel binning)
* 1920x800 @ 60p (same as above)
* 1920x1080 @ 45p and 48p (3x3 binning)
* 1920x1920 @ 24p (1:1 square crop)
* 3072x1920 @ 24p (1:1 crop)
* 3840x1600 @ 24p (1:1 crop)
* 4096x2560 @ 12.5p (1:1 crop)
* Full-resolution LiveView: 5796x3870 at 7.4 fps (128ms rolling shutter).

The last feature complements the well-known full-resolution silent pictures (http://www.magiclantern.fm/forum/index.php?topic=12523.0) - the new implementation will be usable at fast shutter speeds, without the exposure gradient (http://www.magiclantern.fm/forum/index.php?topic=12523.msg120750;topicseen#msg120750) - but with rolling shutter (just like regular LiveView frames).

Please understand that providing the source code for those highly DIGIC optimized routines is a bit troublesome and will need some extra legal care. After this step is taken and as soon we are finished with ensuring the product quality you are used from Magic Lantern, we will upload the code to our repository.

Consider this being a huge leap towards our next mind boggling goal:

8K RAW Video Recording!

(http://a1ex.magiclantern.fm/2017Apr01/fishy_8k_april_fools.png)

Sample DNG from 5D Mark III, to show that our proof of concept is working:

8k.dng (http://a1ex.magiclantern.fm/2017Apr01/8k.dng)

Stay tuned for more information!

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Walter Schulz on April 01, 2017, 11:22:04 AM
Who is ML's officiail bookie? I want to risk some Euro ...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: nikfreak on April 01, 2017, 12:12:57 PM
Shut up and take my money!
(https://www.awesomegifs.com/wp-content/uploads/psy-elevator-gangnam-style.gif)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: g3gg0 on April 01, 2017, 12:13:26 PM
this is wonderful! i cant wait to test it on my camera at home!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Wayne H on April 01, 2017, 12:13:52 PM
INCREDIBLE!!!!! Wonderful job alex, i really appreciate all the hard work you and and the other developers have put in over the years to get magic lantern to where it's at right now.

ULTRA HD RAW 24p on 5d mark iii wooah :) ,so the most obvious question is when will we be able to test out this new update.   

Feels like xmas. :)

Alex you are a genius.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 01, 2017, 12:24:51 PM
Incredible. It,s art.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: jagnje on April 01, 2017, 12:32:31 PM
april 1st
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: nikfreak on April 01, 2017, 12:40:17 PM
Nope, I would rather say it's the whole truth as long as no one talks about expects continuous recording:

http://www.magiclantern.fm/forum/index.php?topic=18300.0

 8) :P 8) :P :D :D ;D 8) 8)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Ilia3101 on April 01, 2017, 01:18:10 PM
KooooooooL  8) any chance of anything for the mark 2 or other cameras?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: beauchampy on April 01, 2017, 01:29:47 PM
Please guys, I can't take the heart ache. Is this an April fools? Specifically the 960px 3x3?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Levas on April 01, 2017, 01:50:51 PM
I love how you get a 8K DNG sample from a 5d mark III, there's some real magic going on there  ;D
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Walter Schulz on April 01, 2017, 02:02:02 PM
Is this an April fools?

You don't expect an answer without tricks, traps, false mirrors and double meaning, do you? No today, not today ...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Greg on April 01, 2017, 02:06:12 PM
WOW!  :o
8K is a bit noisy.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Frank7D on April 01, 2017, 08:17:34 PM
Thank you so much for everything you've done! Now, not to be demanding, but 16K is obviously doable so when can we expect it?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: nikki on April 01, 2017, 08:23:19 PM
 ..it works perfect
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 01, 2017, 11:31:34 PM
Committed some of the sources; still playing with it, but those who can compile from source can already give it a try.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hyalinejim on April 01, 2017, 11:50:36 PM
a1ex, which of the crop modes offer a real-time, colour preview?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 01, 2017, 11:56:32 PM
Only the simplest crop mode (1920x1080) has good real-time preview. The modes with higher vertical resolutions have color preview too, but it's not centered (only the top of the frame is shown).

In the regular crop_rec branch, there's also centered 5x zoom. Here I've removed it temporarily, as it increased the code complexity quite a bit, but I'll probably add it later when things will settle.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: reddeercity on April 02, 2017, 12:06:54 AM
I read all this and thought Great Joke ! :P but then I read the Source Code Holy Cow  :o :o :o :o
There goes next few days  , I'll be living in front of my computer exploring this and hopefully I can compile for 5d2  :D 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Ilia3101 on April 02, 2017, 12:11:53 AM
reddeercity do you think that's possible yet? Wouldn't it require tons more reverse engineering?
Damn it I just happen to be away from proper computers that can compile ML for a week right now so I won't be able to try it if it is possible :'(
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: beauchampy on April 02, 2017, 01:00:53 AM
Build for 5d3 available? I just spent a week shooting 50p 3x: so would love to put it through its paces
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Walter Schulz on April 02, 2017, 01:04:05 AM
Top of page -> Downloads
Download page -> Downloads (drop-down menu) -> Experiments
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: reddeercity on April 02, 2017, 01:19:01 AM
reddeercity do you think that's possible yet? Wouldn't it require tons more reverse engineering?
Will not sure yet but I'm looking in to it .
If you read this post Full-resolution silent pictures  (http://www.magiclantern.fm/forum/index.php?topic=12523.msg120497#msg120497) there's info about 5D2 and lead me to believe it maybe possible  8)
I'm currently try to compile the code now , but I get some error's .
some tweak.o stuff , I working of older code , so I think that could be the problem so I'll update to the latest source code
and try again . I don't think 4k is a reality on 5d2 but maybe 3K as the bandwidth for 4K is around 150MB from what I read.
Since we are limited to max. 80MB/s  and by your Yet Another RAW Video Calculator  (http://rawcalculator.bitballoon.com/calculator_desktop)  at 10bit 3x crop 2496x1134 @ 2:2.1 A.R. is 80MB/s
unless Lossless compression is used and that's another thing by it's self .
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 02, 2017, 01:38:02 AM
Greg's original proof of concept (full-width LiveView (http://www.magiclantern.fm/forum/index.php?topic=10111.msg123909#msg123909)) was done on 500D (same generation as 5D2 and 50D).

Sure, it probably needs a bunch of reverse engineering, but it probably boils down to changing some ADTG registers and video timers.

I've actually got these presets working in adtg_gui first. A proof of concept for 3K was committed back in 2016 (https://bitbucket.org/hudson/magic-lantern/commits/50d8f06018) (also mentioned (http://www.magiclantern.fm/forum/index.php?topic=17021.msg171809#msg171809) in the "classic" crop_rec thread).

The registers interesting for changing resolution are documented either on the Register Map wiki page, or in adtg_gui source. Interesting commits: 8bbce9c (https://bitbucket.org/hudson/magic-lantern/commits/8bbce9c), 7333869 (https://bitbucket.org/hudson/magic-lantern/commits/7333869), 4a66a48 (https://bitbucket.org/hudson/magic-lantern/commits/4a66a48), 6c62a6b (https://bitbucket.org/hudson/magic-lantern/commits/6c62a6b), 38e78de (https://bitbucket.org/hudson/magic-lantern/commits/38e78de).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: reddeercity on April 02, 2017, 02:21:58 AM
Thank you  :)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 02, 2017, 07:37:54 AM
Getting some compile error with mlv_rec and mlv_snd in crop_rec_4k branch.

Code: [Select]
Building module mlv_rec...
Updated HGVERSION
[ README   ]   module_strings.h
[ CC       ]   mlv_rec.o
mlv_rec.c: In function 'raw_rec_update_preview':
mlv_rec.c:4160:5: error: too few arguments to function 'raw_set_preview_rect'
     raw_set_preview_rect(skip_x, skip_y, res_x, res_y);
     ^
In file included from ../lv_rec/lv_rec.h:24:0,
                 from mlv_rec.c:76:
../../src/raw.h:148:6: note: declared here
 void raw_set_preview_rect(int x, int y, int w, int h, int obey_info_bars);
      ^
make[4]: *** [mlv_rec.o] Error 1

Don´t have my camera to test but I follow the error and change the line in mlv_rec to below which seems to get rid of the error code for both mlv_snd and mlv_rec.
In mlv_rec
Code: [Select]
raw_set_preview_rect(skip_x, skip_y, res_x, res_y);to this:
Code: [Select]
void raw_set_preview_rect(int x, int y, int w, int h, int obey_info_bars);Or maybe change to:
Code: [Select]
void raw_set_preview_rect(skip_x, skip_y, res_x, res_y);
?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: andy kh on April 02, 2017, 07:58:10 AM
this is wonderful. need to start looking for a used 5D mark III
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: ItsMeLenny on April 02, 2017, 09:55:45 AM
Well I guess the original 4K trolls are no longer trolls.
But if 4K is possible that must mean 1000fps is as well.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: kaco on April 02, 2017, 02:32:39 PM
Respect to a1ex and the whole magiclantern team.
No more excuses for not producing world class imagery.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: dfort on April 02, 2017, 07:19:37 PM
Is it difficult to port to other camera models?

So far, the 3x3 720p mode from crop_rec was ported to EOS M (rbrune) and 700D (dfort). So it shouldn't be that hard...

I'll take that as a compliment! :D :D :D

That reminds me, I should do a pull request for the 700D but it looks like there's more work to do now. How about getting some other cameras up to speed on the crop_rec module? By the way, don't expect 4k on anything other than the 5D3--then again maybe there will be more surprises.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 02, 2017, 08:58:03 PM
Yep - my point was that people outside the core team are definitely able to play with this code (as in, it's not pure black magic).

Took this build out for a test (nothing fancy, just taking pictures of the kids with full-res LiveView and pre-recording). Noticed a major limitation (could not set shutter speeds fast enough for daylight) and pushed a fix for that.

After the fix, modes with reduced FPS (4K and full-res LiveView) now have the usual range of 1/4000...1/30 (or 1/60) mapped to 1/15000 to 1/fps. That's right - 1/15000 exposure time with 128ms rolling shutter :D

Next annoyance was the grayscale preview, which is slow and not working at all in 10-bit. Also, lossless MLVs are not playable in mlv_play. Will look into those.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hyalinejim on April 02, 2017, 09:08:46 PM
a1ex,  if you can improve the greyscale preview you will be the king of kings
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Greg on April 02, 2017, 09:41:03 PM
These are just approximate calculations. You can initially estimate the speed of the sensor.

5D3 22MPx x 6fps = 132MPx/s
4096px x 2560px x 12.5fps = 131MPx/s

7D   18MPx x 8fps = 144MPx/s
70D 20MPx x 7fps = 140MPx/s
6D   20MPx x 4.5fps = 90MPx/s

500D 15MPx x 3.4fps = 51MPx/s  :-[
Btw, my 4 year old smartphone 8MPx x 30fps = 240MPx/s  :P

More accurate calculations - https://www.magiclantern.fm/forum/index.php?topic=12656.0
Of course we must remember about the speed of writing.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: dfort on April 02, 2017, 09:46:26 PM
Yep - my point was that people outside the core team are definitely able to play with this code (as in, it's not pure black magic).

My point too. I'd like to get some more people to start playing with the code. There's lots of information in the comments and on the forum that takes the mystery out of what's going on--though some of what you're doing sure seems like pure black magic.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Ilia3101 on April 02, 2017, 10:29:41 PM
@Greg
and the rather slow 5D2: 21.2MPx x 3.9fps = 82.68MPx/s. Does this basically tell us the limit of the sensor? So this calculation works: 82.68/23.976 = 3.45MPx limit at 24p on this camera? So I guess it could do 2560x1320(3.38MPx).
 :D how does one implement the "Greg's resolution hack" that I've been hearing about?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: vstrglv on April 02, 2017, 10:31:53 PM
Fantastic! Thank you very much!
One Q about * 1920x960 @ 50p (both 1:1 crop and full-frame - 3x3 pixel binning).
1920x960 @ 50p (3x3 pixel binning) works good, but i can not set 1920x960 @ 50p for 1:1 crop, only 1920X632.
I set Crop mode - 1920 1:1 and 50p in Canon menu.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: nikfreak on April 02, 2017, 10:39:20 PM
:D how does one implement the "Greg's resolution hack" that I've been hearing about?
http://www.magiclantern.fm/forum/index.php?topic=19300.msg182149#msg182149
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: reddeercity on April 02, 2017, 10:47:42 PM
@Ilia3101 I'm working on it  , just had some compiling issue with adtg_gui yesterday should have that resolve
today , then I can work off a1ex's links  ;D
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 02, 2017, 10:57:19 PM
1920x960 @ 50p (3x3 pixel binning) works good, but i can not set 1920x960 @ 50p for 1:1 crop, only 1920X632.
I set Crop mode - 1920 1:1 and 50p in Canon menu.

1920 1:1 tall (meaning "with increased vertical resolution", but couldn't fit that into menu)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: vstrglv on April 02, 2017, 11:31:16 PM
Thank you! It works good!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 02, 2017, 11:59:08 PM
Updated the preview with something a little more usable. It's color (though still low-res and non-realtime) and only drops to grayscale when recording speed becomes critical.

Unfortunately, g3gg0's raw_twk (https://www.magiclantern.fm/forum/index.php?topic=16682.0) routine doesn't appear to work in LiveView, so we are stuck with (slow) CPU-based previews for now :(

I can probably get faster previews in 14-bit lossless mode, by configuring the source raw stream as 16-bit (unpacked), since the compression routine accepts 10/12/14/16-bit input (http://www.magiclantern.fm/forum/index.php?topic=18443.msg181620#msg181620). For uncompressed modes, I'm afraid we currently don't have a way to get both unpacked data (for fast previews) and bit-packed (for recording) while in LiveView (so all my previous ideas (http://www.magiclantern.fm/forum/index.php?topic=5601.msg177531;topicseen#msg177531) are currently fiction).

New build posted on the Experiments page.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: mk11174 on April 03, 2017, 01:06:28 AM
Trying to see what the next step is to see if there are any modes 700d can get, first i messed with atdg tool to try to reproduce the 3x3bin mode while in 720p by changing 0x4 to 0x2 on 800c just to understand how the tool works since those values are posted. I then set the tool to show values that have been changed more then twice and screen captured every value for 1080p, 1080p x5 and x10, 720p, 720p x5 and x10, crop mode in 1080p and also photo mode just to track all the values that are changing in case it was needed.

Thats where i am at now though, not sure what values to change though, gets scary when you see the crazy images on live view that make u think u burned sensor out. Only thing i saw that did anything as far as the size was 800c changing to 0x0 makes a zoomed version but its stretched tall. Other then that another one moved the image up and down.

Dont know how to cross reference with the 5d3 cause i see Pack12(?,?)  for example and the values are not like what you see in the tool. So pretty much clueless on where to go from here??
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hyalinejim on April 03, 2017, 01:42:21 AM
Wow! That's a hell of a lot more useable and the preview is centered which is absolutely kickass.

I get 30 frames in UHD, but 3K is still continuous for me in crop_rec. The preview does drop back to greyscale about 50% of the time in 3K. The colour preview is more detailed and has a higher refresh rate... it's pretty much usable, I think.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: WonkyTuna on April 03, 2017, 02:23:44 AM
I'm trying it out but unfortunately when I try to watch it on MlRawViewer or export it to DNGs using RAWMagic I'm left with this: https://drive.google.com/open?id=0B1lko58z3g6lRTlKUWc1YzlzOFk. I tried re-downloading the build and formatting my card but I can't seem to get anything out of any of these crop modes...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: squig on April 03, 2017, 04:11:54 AM
I've found a couple of issues in both builds. The first is nothing major: when you change a resolution in the crop mode menu, say 3K to UHD, you have to exit the ML menu before you can adjust the resolution size in the raw video menu. The other issue is making me pull out what little hair I have left: the lossless compression ratio appears to have a mind of its own, it's like a recalcitrant AI. At one point it was compressing at 51% on the 1st build:

(http://i.imgur.com/6IX4XKI.jpg)

When I loaded the 2nd build it started at 85%:

(http://i.imgur.com/HB2uNHV.jpg)

So I had to scale down to:

(http://i.imgur.com/bXY36uK.jpg)

After a few seconds of recording the compression went to 70%:

(http://i.imgur.com/Ojd4ykL.jpg)

Then when I try to increase the resolution I get this [Expect xxxx frames]:

(http://i.imgur.com/NckArF4.jpg)

A battery pull gets rid of the [Expect xxxx frames], but on the 2nd build it reappears when I try to scale up the resolution. So I went back to the 1st build to see if I could record at 3.3K 51% lossless again. Initially I was able to get this setting:

(http://i.imgur.com/Mj734Xe.jpg)

That was a bit much for the card, it stopped recording after a few seconds resulting in the dreaded [Expect xxxx frames]:

(http://i.imgur.com/kR4G8cb.jpg)

Taking me back to square one where I had to scale the resolution back down:

(http://i.imgur.com/gNnZrKY.jpg)

Ideally the compression ratio would either be fixed, or manually adjustable.











Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: dmilligan on April 03, 2017, 04:44:18 AM
Ideally the compression ratio would either be fixed, or manually adjustable.
That's mathematically impossible, and not how lossless compression (https://en.wikipedia.org/wiki/Lossless_compression) works. The compression ratio you get is totally dependent on the scene, and it's even theoretically possible that the compression results in larger file sizes than the original uncompressed data. This typically happens when you give a lossless compression algorithm data that has very little redundancy (already compressed), or that is not of a type the algorithm is "tuned" for, for example feeding the compressor 10-bit raw data results in the compressed version actually being larger (http://www.magiclantern.fm/forum/index.php?topic=18443.msg176699#msg176699).

For the most part, lossless compression algorithms work by simply eliminating redundant data via more efficient encoding. Take some text for example, encoded using ASCII, each character takes 1 byte to store and therefore all characters have equal weight even ones that are rarely used. However we know that in any particular language there are characters (or even sequences of characters) that typically appear more often than others, so if we gave the characters that appear more frequently, shorter encodings (which subsequently requires less common characters to have longer encodings), then overall we can save space. However if we were to feed random gibberish to such an algorithm, it might actually take more total space the store than the original simple encoding, since the characters with longer encodings are just as likely to appear as ones with shorter encodings.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: squig on April 03, 2017, 05:15:03 AM
Understood, but from a cinematography perspective; I need to figure out what resolution will give the card enough headroom to write at that resolution continuously at least 90-95% of the time; so far it looks like that's somewhere in the 2.5-2.6K range for 24p 2.39:1, which is still great.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: squig on April 03, 2017, 09:03:55 AM
2.7-2.8K 2.39:1 looks like the sweet spot for repeatable continuous recording on the Toshiba 1066x card. By "repeatable" I mean repeatable near 100% of the time. Same goes for 3.3K anamorphic scope: 2272x1364 with a 1.5x anamorphic lens.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: GutterPump on April 03, 2017, 12:56:49 PM
Updated the preview with something a little more usable. It's color (though still low-res and non-realtime) and only drops to grayscale when recording speed becomes critical.

DAMN ! This update is very promising. I did some tests and yes it's more comfy for framing.
Thanks you for all these exciting features A1ex and all the team.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hyalinejim on April 03, 2017, 02:31:45 PM
OK, I'm pretty sure that the 14bit lossless compression varies with ISO.

In crop_rec 3K (3072 x 1286 ) I can get continuous recording at ISO 100

But if I whack the ISO up to 6400 I only get eleven seconds.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Lars Steenhoff on April 03, 2017, 03:08:39 PM
that makes sense because the more noise of 6400 iso would result in less compression
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kharak on April 03, 2017, 03:09:30 PM
Happy Aprils fools day :)

How do I go about processing these mlv files with the mlv_dump included? I can't figure out how to spit dng's out
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hyalinejim on April 03, 2017, 03:17:46 PM
I use this method on Windows:

http://www.magiclantern.fm/forum/index.php?topic=10526.msg178538#msg178538
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Launch on April 03, 2017, 03:31:14 PM
I salute the dedication but I wish squeezing every last pixel of out of the camera would be secondary to allowing audio with simple RAW. I would trade every last improvement for 1080p RAW with audio. For some reason, this worked for me once, but it just randomly stopped. It used to create a WAV file along with the RAW file.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: squig on April 03, 2017, 03:35:58 PM
100 ISO - black screen ~49%
6400 ISO - black screen ~60%
6400 ISO - busy scene, fast movement ~69%
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Levas on April 03, 2017, 03:45:18 PM
Ah that explains a lot.
High iso less compression, low iso high compression.

So for low iso shooting, compressed 14 bit is more efficiënt then shooting in 10 bit
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kharak on April 03, 2017, 03:48:03 PM
I use this method on Windows:

http://www.magiclantern.fm/forum/index.php?topic=10526.msg178538#msg178538

Thank you, exactly what I was looking for.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 03, 2017, 04:51:06 PM
I need to figure out what resolution will give the card enough headroom to write at that resolution continuously at least 90-95% of the time

Latest update estimates compression ratio on the fly, from the current scene and exposure settings. Also a few minor bug-fixes.

To get the best possible compression, all you need is to set ISO to maximum, shutter to 1/FPS, aperture wide-open and use a bright scene. You'll get close to 35% (http://www.magiclantern.fm/forum/index.php?topic=18443.msg181620#msg181620) (apparently the limit for this codec, or maybe just for Canon's implementation).

Generally, if pixel values are predictable (from their neighbors), the image will compress well. That usually means low noise levels (low ISO, shadows, fully clipped highlights) and few details (large out-of-focus and/or low-contrast areas). Note that absolute noise values are higher in highlights, and the SNR is higher too, so exposing to the right will give higher file sizes (but that shouldn't stop you from using it).

Fast motion probably gives better compression, because of motion blur (didn't test, but that's my understanding with this codec).

Invalid image data (that includes e.g. 14-bit raw data interpreted as 10-bit, or pure noise, or machine code if you want) will give compressed output larger than initial size. In current implementation, this gives memory overflow. For some unknown reason (alignment issues? no idea), setting a maximum size for compressed output causes camera lock-up at certain resolutions, with no apparent logic (see this commit (https://bitbucket.org/hudson/magic-lantern/commits/903c4828d514d3a8f2e3bced77318acdd8fffc86); minimal example here (http://a1ex.magiclantern.fm/2017Apr01/crash.c)).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: limey on April 03, 2017, 07:10:20 PM
For 1.23 only? I don't have a crop mode option in 1.13.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hyalinejim on April 03, 2017, 07:17:20 PM
Did you get the latest experimental build, and load the crop_rec module?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 03, 2017, 07:39:02 PM
I wonder if there is a potential for ordinary resolution recording but with higher frame rates.
Something like 100 FPS ?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: etienne on April 03, 2017, 10:13:35 PM
One question to the developpers do you know if the 3 things below would be possible one day:

 1/ 1080x1080  (from 3x3 bining using the 3240x3240 sensor size). for a big anamorphic s65 lens.

 2 / in the same way of the 14 bit losless mjpeg compression is an 8 bit mjpeg compression possible? so we can do low file size and get the real sharpness the 5d3 can offer (clean HDMI recorder on prores does not get any better sharpness than the in camera h264 file - whereas the raw files is always resolving much more sharpness)

 3/ finally how far would you say you are from a working preview for the mode 3K, UHD, or zoom x5 and is it even possible?

Also you probably have thought this but the basic full sensor preview with blacktape on the sceen could make it and if the 1:1 sensor readout cannot show original preview then maybe a x3 preview instead of the x5. Would not be perfect but would definitly make it usable.

Thanx for the work it's a dream. Where is the TIP JAR already?

Bests.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: magnesiumnitrate on April 04, 2017, 12:15:55 AM
I started testing 14 bit lossless with 1080P 24 because thats what I use the most. File sizes are much smaller, thats nice. Something funky is happening though. Using the histogram overlay with 80% or more of my exposure on the right half I'm getting horribly, horribly underexposed results in davinci. The live view looks great, histogram looks great, but then it hits davinici and the footage looks awful. Not sure what to make of it, but thought you guys might want to know.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Filipe Samora on April 04, 2017, 01:04:24 AM
So after a few hours playing around with A1ex's second release of the 1.1.3 firmware, here's what I'm getting with 14bit lossless option:

3520 x 1320 = max. 1314 frames - ISO within 200-400 range - scales up to UHD with no perceived loss of quality

4096 x 1862 = max. 507 frames @ 12.5 fps - Not sure how useful this would be for landscape work or slow shots. Again ISO within 200-400 range.

Previews: At the moment, not fantastic but definitely better than a black screen. I was able to rack focus with a moderate degree of success.

Camera crashes persistently when using 50/60p modes.

I'm using Komputerbay cards 64GB 1066x

IQ, even shooting at home with crap light, is just beyond words.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kharak on April 04, 2017, 01:48:08 AM
This is mighty fun to play with,

I was able to shoot up to 19 seconds at 3504x1312 at regular crop with Framed preview and Compressed Raw 14 bi, lots of movement, I was panning slowly downwards. But I only got so much recorded this one time, I never achieved the same length of recording again.

Is there a way of centering the crop? It looks weird not being centered.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hyalinejim on April 04, 2017, 02:30:09 AM
2 things I learned:

1. If you half press shutter button while recording the preview stays in colour rather than alternating between colour and greyscale. Frame rate still lags but it makes things a hell of a lot easier to know what's going on in front of the camera.

2. Filming the moire pattern of a computer screen's pixel array and refresh rate will absolutely kill the compression and stop the recording, even at lower resolutions!


Is there a way of centering the crop? It looks weird not being centered.

It should already be centered. Are you loading the crop_rec module of the latest experimental build and then selecting 3K or UHD? Framing is off in 5x but should be centred in these modes. Sometimes 3x is shifted up for me. I just restart the camera and it's fine.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: bobbyshots2 on April 04, 2017, 03:23:54 AM
Why doesnt the latest package have the mlvrec 2.0 module .. its only has light. for 5d3123.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: bobbyshots2 on April 04, 2017, 04:00:02 AM
Hi,
Ive been testing the latest build (April 03)for 5dIII123.
So far what i noticed there is no mlv snd module
or mlv_rec. Also there is no gray scale preview.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: limey on April 04, 2017, 04:09:42 AM
Yeh I figured it out. I guess I was blind while looking at the modules or I wrote the files to the wrong sd card from my lexar workflow hub.

Getting pink frames and crashes in the 50fps modes.

Getting unreadable files in some other modes like the 3k and UHD. It was just an initial test, I will test more tomorrow. I am using 1066x 256gig lexar cards. However, they aren't formatted and have a lot of shots on them. I will format one and see what difference that makes.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hindra on April 04, 2017, 06:28:06 AM
Shooting with the Canon 5d3, 1.2.3 firmware and using the latest experimental build(April 3rd 2017) with Lexar 128gb 1066x cf card.

1080p 48FPS seems to be working (It seemed to have recorded fine, but I must have deleted the file.)
3520x1320 24FPS(ISO 100-200 iso seems to be continuous and at a 5x crop mode) ISO 400 recording stops after a few seconds. In no crop mode preview was colorful noise. in 5x crop preview worked great. I recorded 1510 frames and I ended it.
3584x1320 24FPS I recorded 1443 frames and I ended it.
1080p 60FPS(buggy, stops recording quickly) gray preview was glitchy and lagged.

3k ([email protected]) I got 66 frames.
UHD ([email protected], couldnt get 3840x1600) I got 66 frames as well. It states its 120mb/sec, this might require the lossless compression with 10bit?
Both 3k and UHD frames are good, used MLV_DUMP from the page I downloaded the firmware to extract these.

I am also very tired, I will try and play around with them some more.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: budafilms on April 04, 2017, 08:32:22 AM
@A1ex, you are my king.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 04, 2017, 09:12:19 AM
Ah, a little late to yet another party once again and after playing around in the past 48 hours or so I've finally come to a conclusion to be able to confirm that 3584x1320 in crop_rec (3K 1:1) can be recorded continuously if shot in between ISO 100-200 and more often than so in 400 depending on scene.

This was all tested with both 128gb 1066x & 256gb 1066x CF cards from KomputerBay on two separate 5D3's running 2017April03.5D3123 builds.

(https://c1.staticflickr.com/3/2828/33014736263_11f475c323.jpg) (https://flic.kr/p/Sip9SK) (https://c1.staticflickr.com/4/3929/33698823611_26f5ec5f2d.jpg) (https://flic.kr/p/TkRhdc)

Seems that ISO 800 and higher doesn't get along well with 14-bit lossless compression which makes sense as previously stated by others in here.

Here's an example of what it looks like upon running this beautiful hack via LiveView from 5D3 below and watch what happens when I push it up to ISO 800 (and higher) towards the end:

https://vimeo.com/211439899

I was able to convert all 14-bit lossless files with ease from @Danne's latest cr2hdr.app_4k (includes the required certain mlv_dump which was modified directly from @a1ex) so I am more than flabbergasted by the fact that all of this is storming down on us like there's no tomorrow.

Love it and will continue to play with and hopefully run into some more bugs! 

Also I can confirm that if I point the camera at a computer screen while recording and it'll crash once I focus upon the pixealated monitor (indeed this is quite strange but good find @hyalinejim) and I am actually loving the half shutter press for FULL COLOR LIVE DISPLAY or whatever that's called (plus it zooms in a bit more to be sure everything looks spot on) before recording which is very handy.

Thank you @a1ex for yet another petrifying update (in a good way!) and don't ever change who you are! Ha. 8)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: beauchampy on April 04, 2017, 09:36:31 AM

Here's an example of what it looks like upon running this beautiful hack via LiveView from 5D3 below and watch what happens when I push it up to ISO 800 (and higher) towards the end:

https://vimeo.com/211439899



I cant seem to get that lovely liveview on 113. Mine is low fps, desaturated and choppy.
Running crop_rec_4k.2017.Apr03.5D3113 and I copied your settings..
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Levas on April 04, 2017, 09:54:55 AM
I was just wondering...

Who has twixtor or something alike and creates some stunning 6K - 24p video out of that 7 fps full resolution silent video mode  ;D 8)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 04, 2017, 09:58:25 AM
I cant seem to get that lovely liveview on 113. Mine is low fps, desaturated and choppy.
Running crop_rec_4k.2017.Apr03.5D3113 and I copied your settings..

Hmmm then perhaps I should downgrade to 113 and try to reproduce this?

Did you press 5x zoom after using ML menu or before? Sometimes it goes into a weird phase which may seem like having its own quirks but a simple restart (on/off) of the camera would help at least it does oh my end.

But it could very well be a bug for 113 and I'll let you know once I've downgraded.

Thought not too long ago @a1ex made a pull request for both firmwares (113 & 123) to be merged together after each update or does this only apply to the Nightlies and not experimental?

Well I'm actually tempted to check out the good old speedy 113... Just been using 123 as of late for HDMI output on gig's.

Actually this one trick may be worth to try (Thanks @dfort for the tips) perhaps reset your settings to default (from Canon menu) not the ML version and let me know if that helps either way.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 04, 2017, 10:13:46 AM
Hi,
I don't know if it will be useful or not...

Testing latest build (April 3th crop_rec) on 5D3 113, for now only this mode because is which I find more useful for me: 1920x872 50p 3x3 (aspect ratio 2.20:1) 14bit lossless.

Yesterday I was able to record without problems in continuous with normal preview, in color.

Today, with the same settings as yesterday, it has been impossible to boot the camera. In the SD card there are text files called ASERT, one for each boot attempt. In them the following appears:
ML ASSERT:0at mlv_lite.c:1935 (compress_task), task compress_tasklv:1 mode:3
Magic Lantern version : crop_rec_4k.2017Apr03.5D3113
Mercurial changeset   : 7b0cb9532629 (crop_rec_4k) tip
Built on 2017-04-03 14:54:32 UTC by [email protected]
Free Memory  : 173K + 3807K


If I change SD card for another with 2017Feb11th 5D3113 (crop_rec proxy) all works again.

Hope it helps!!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: squig on April 04, 2017, 10:14:03 AM
Are you guys running ML off an SD card?

The best I can do @ 100 ISO is 3.3K 2.39:1. ML is on the CF card.

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 04, 2017, 10:28:34 AM

Quote
Hi,
I don't know if it will be useful or not...

Testing latest build (April 3th crop_rec) on 5D3 113, for now only this mode because is which I find more useful for me: 1920x872 50p 3x3 (aspect ratio 2.20:1) 14bit lossless.

Yesterday I was able to record without problems in continuous with normal preview, in color.

Today, with the same settings as yesterday, it has been impossible to boot the camera. In the SD card there are text files called ASERT, one for each boot attempt. In them the following appears:
ML ASSERT:0at mlv_lite.c:1935 (compress_task), task compress_tasklv:1 mode:3
Magic Lantern version : crop_rec_4k.2017Apr03.5D3113
Mercurial changeset   : 7b0cb9532629 (crop_rec_4k) tip
Built on 2017-04-03 14:54:32 UTC by [email protected]
Free Memory  : 173K + 3807K

If I change SD card for another with 2017Feb11th 5D3113 (crop_rec proxy) all works again.

Hope it helps!!


I've erased all in SD card and copied clean April3th Crop_rec_4K build.

All works ok again like yesterday.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 04, 2017, 11:01:33 AM
Same again:

ML ASSERT:0at mlv_lite.c:1935 (compress_task), task compress_task
lv:1 mode:3
Magic Lantern version : crop_rec_4k.2017Apr03.5D3113
Mercurial changeset   : 7b0cb9532629 (crop_rec_4k) tip
Built on 2017-04-03 14:54:32 UTC by [email protected]
Free Memory  : 177K + 3923K

Need to pull battery, and then camera doesn't boot anymore :(
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hyalinejim on April 04, 2017, 11:20:55 AM
Are you guys running ML off an SD card?

The best I can do @ 100 ISO is 3.3K 2.39:1. ML is on the CF card.



Yes. I was under the impression this frees up more bandwidth for recording to CF. Also, when you change CF cards your settings don't change.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: squig on April 04, 2017, 12:35:16 PM
OK cool, I'll give it a shot.

Seeing as we're all madly trying to squeeze every last MB out of the oldfella to get closer to 4K, would it be possible to add card spanning to these experimental builds?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: garry23 on April 04, 2017, 01:47:28 PM
I just downloaded the latest experimental build, but didn't get far.

I would welcome a little help, as I'm keen to test, but, I don't do video. That is I'm not up to speed on MLV etc.

If there is someone out there who has the time, maybe they would create a 1 page 'here is how to test', i.e. The steps.

In particular, how do I set up, run and process the full Res 'photo mode'. Once I know how to get things running, I can test.

Cheers

Garry
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 04, 2017, 05:52:37 PM
So what is the bottle neck for transfer speeds in the 5D3? Is it still the card reader in the camera? I thought cameras card reader was limited even with really fast cards?
Is spanning the trick for UHD?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 04, 2017, 05:55:45 PM
ML ASSERT:0at mlv_lite.c:1935 (compress_task)

This one indicates frame corruption, and should also print this (unless obscured by other messages):
Code: [Select]
                printf("\nCompressed size higher than uncompressed - corrupted frame?\n");
                printf("Please reboot, then decrease vertical resolution in crop_rec menu.\n\n");

1080p 60FPS(buggy, stops recording quickly) gray preview was glitchy and lagged.

Took a quick look - after the latest fixes (see below), the following resolutions are continuous (1.2.3, lossless ~55%, Preview: Frozen LV, with grayscale preview):

- 1920x800 60fps
- 1920x960 50fps
- 1920x1080 45fps

1920x1080 48 FPS is a bit unreliable (corrupted frames with Global Draw on, but appears OK with it off). This mode is really pushed to the limit (and that's the reason I've also added a 45-fps option). Feel free to play with the knobs from crop_rec submenu, if you think you can find the sweet spot without corrupted frames.

I would expect the 1.1.3 firmware to have a little more headroom (maybe even allow a few more pixels), but didn't test it. You can fine-tune vertical resolution from the crop_rec submenu (and if you have corrupted frames, decrease it from there).

New build posted. Short summary:

- fixes for high-FPS modes with Frozen LV (first frame was corrupted, which didn't play nice with lossless compression)
- continuous full-res recording with lossless compression (about 4-5 FPS, depending on ISO and scene complexity)
- print current FPS and rolling shutter in the FPS override menu (visible from main Movie menu tab)
- two experiments with a small risk:
   - single-buffering (risk of tearing, alongside with graphical checks)
   - use-after-free for SRM memory (to fix half-shutter quirks)
- minor fixes here and there.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: GutterPump on April 04, 2017, 06:11:34 PM
Thanks for the update A1ex, today it's my birthday, it's a nice gift from you :)

I believe this has already been said, but the best way to use the lossless 14bits is to forget to shoot in ETTR.
Do you think there is way to take the control on the compression codec ?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 04, 2017, 06:25:18 PM
In particular, how do I set up, run and process the full Res 'photo mode'. Once I know how to get things running, I can test.

Simplest way:
- crop_rec.mo, silent.mo
- crop mode: full-res LV; silent mode: simple, any file format
- press half-shutter
- for preview, also load mlv_lite.mo, enable raw recording and global draw.
- caveat: you can only review 14-bit regular DNG (pic_view) or MLV (mlv_play); for 10-bit full-res MLV, raw_twk runs out of RAM

Continuous full-res recording at 4-5 FPS:
- crop_rec.mo, mlv_lite.mo
- crop mode: full-res LV; raw recording: max resolution, lossless compression
- FPS override: 4 or 5 FPS, low light (important)
- press record
- mlv_dump foobar.mlv --dng

Fast half-shutter trigger (with negative shutter lag):
- crop_rec.mo, mlv_lite.mo, configured as above
- raw recording: enable REC trigger (half-shutter: pre only; without pre-recording enabled, that means only 1 frame)
- no need to use FPS override, as we are only saving the frames triggered by half-shutter
- start recording
- press half-shutter every now and then to capture a frame
- stop recording

Capture unpredictable action (animals, kids etc):
- crop_rec.mo, mlv_lite.mo
- crop mode: full-res LV or 4K half-FPS
- raw recording: max res, 10-bit (here, pre-recording is a bit confused by variable frame sizes, so let's not be adventurous)
- pre-recording: 1 or 2 seconds (actually "up to")
- half-shutter trigger: "hold" (which will pre-buffer up to about half of available RAM), or "pre only" (which will pre-buffer all the available RAM)
- start recording, press half-shutter etc
- caveat: preview is not working at all with 10-bit (try 14-bit)
- to review in camera: 4K 10-bit can be played back with mlv_play + raw_twk, full-res 14-bit with mlv_play only.

I believe this has already been said, but the best way to use the lossless 14bits is to forget to shoot in ETTR.
Do you think there is way to take the control on the compression codec ?

Very little. (http://www.magiclantern.fm/forum/index.php?topic=18443.msg181620#msg181620)

I wonder if there is a potential for ordinary resolution recording but with higher frame rates.

First post:
Quote
* 1920x960 @ 50p (both 1:1 crop and full-frame - 3x3 pixel binning)
* 1920x800 @ 60p (same as above)
* 1920x1080 @ 45p and 48p (3x3 binning)

In particular, 1080p48 is quite pushed to the limit, which is why I've also included a 45p preset (hopefully a bit more reliable).

1/ 1080x1080  (from 3x3 bining using the 3240x3240 sensor size). for a big anamorphic s65 lens.

Already possible on regular nightly.

Quote
2 / in the same way of the 14 bit losless mjpeg compression is an 8 bit mjpeg compression possible?

Yes... (http://www.magiclantern.fm/forum/index.php?topic=2803.msg13469#msg13469)

Quote
so we can do low file size and get the real sharpness the 5d3 can offer (clean HDMI recorder on prores does not get any better sharpness than the in camera h264 file - whereas the raw files is always resolving much more sharpness)

... but it would simply compress Canon's YUV422 buffer (with their color science, picture styles, sharpness etc).

Quote
3/ finally how far would you say you are from a working preview for the mode 3K, UHD, or zoom x5 and is it even possible?

Far, far away. The trick I've used to center the 5x zoom preview in the regular (non-4K) crop_rec branch didn't work for the new modes :(

Also, I have no idea how to configure image scaling. Adtg_gui doesn't reveal any difference between x5 and x10 zoom modes (both use identical raw buffer configurations - just the preview differs).

I can try to include the centered x5 zoom mode (which you can already use in the "classic" crop_rec builds), if that helps.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Teamsleepkid on April 04, 2017, 06:44:01 PM
Anyone trying to turn the eos m into the blackmagic cinema camera? 2.5k possible? Maybe... hopefully...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: togg on April 04, 2017, 06:51:03 PM
This is amazing for movie still photographers!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: garry23 on April 04, 2017, 07:25:36 PM
@a1ex

Many thanks for the simple insight, for a simple still photographer  ;) That is:

Quote
Simplest way:
- crop_rec.mo, silent.mo
- crop mode: full-res LV; silent mode: simple, any file format
- press half-shutter
- for preview, also load mlv_lite.mo, enable raw recording and global draw.
- caveats:
  - preview doesn't work

Because of the preview limitation, I simply composed with crop_rec and silent switched off, used ETTR, at 0%, and switched them back on after the ETTR.

Grabbed a DNG which I inspected in LR.

Fantastic!!!

OK, I know you know the limitations at the moment, eg no EXIF data, but what I step forward for silent full res photography, ie in those churches I go to etc.

One thing I have noticed is that the ETTR exposure at 0% highlights (SNRs off/0) does show over exposure. Not sure if this is real or important at this stage of the developments.

As I said, I'm keen to help test the still photography side of the experiment: I'll stay clear of the MLV world...for now.

Cheers and, once again, thanks for this new step forward for ML.

Garry
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: lostfeliz on April 04, 2017, 08:57:20 PM
I'm having trouble using mlv_dump to decode the compression. Any help appreciated. (sorry, I know this is a development forum. what I am I doing here if I can't use Terminal properly?) :)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 04, 2017, 10:13:28 PM
Assuming you're on OS X -- Have you even tried @Danne's latest cr2hdr.app which includes the modified mlv_dump yet @lostfeliz?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: etienne on April 04, 2017, 10:36:40 PM
@a1ex

Hello thanx for taking the time to answering the question this is highly appreciated. i'm on the curve to go to URSA 4.6K but this new fonctions let me think my olds 5d3's have still some life to live.

regarding the fact that mlv mjpeg 8 bit compression
Quote
would simply compress Canon's YUV422 buffer (with their color science, picture styles, sharpness etc)


Are you absolutely sure regarding the sharpness. I'm asking because this is a fact that the clean hdmi as absolutely zero additionnal sharpness from the h264 files i have tested this thousands times. But the raw definitly resolves way more sharpness similar to what produces an A7sii, gh4 or gh5 in HD so i was sure that the 5d breaks something in between what you catch with rec_mod and what you get as h264 or hdmi out.

Now what i'm undestanding from your saying is that the compress Canon's YUV422 buffer is what is sent to the hdmi out right? If it's the case it's useless for sure but can it be something else. Can you think of any other way to compress to something like 15~25% even if it means we loose all the color depth from the raw but just keep that sharpness from it. I'm insisting on this because Personnally I would be so much be happy with an MLV without any color depth additional value but which would just have the real sharpness we can get from the current raw and that could allow like 60~90 minutes on a 128GB card.

Also I want to congrats on the slowmotion mods all options are awesome i just did a 1080 60P in 2:35 flawlessly. As mentionned the 1080 48P as a few corrupted frames and its preview is also a little off towards the down of the image compared to the canon preview but appart from that we could say this is 99% usable.

Finnally. The 3K Mode and UHD are absolutely awesome. I personnally don't see how we could use over 3360x1528 with spanning and sound in the future so any preview that would be x2 would just be the ultimate thing to make this usable.

I undestand that for now it's complete unknow hope someone will find a solution one day otherwise in regards of you're sayinf:
Quote
I can try to include the centered x5 zoom mode (which you can already use in the "classic" crop_rec builds), if that helps.
Sure It's definitly better than nothing to have access to canon x5 center preview again instead of pink frame eventhough it will not help with framing but it will help to pull focus correctly durin shooting.

Bravo again keep the good work. Looking very forward to integration to MLV2 with spanning and sound enabled...

Good Luck.


 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hyalinejim on April 04, 2017, 10:57:37 PM
Hi a1ex, thanks for the new build.

I noticed that the preview framing in UHD seems to be too far up. I tried fiddling with the crop options, to no avail.

Also, for anybody concerned about low light shooting, it looks like I can get continuous 16:9 with around 2.4k horizontal resolution at ISO 3200.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 04, 2017, 11:24:03 PM
Can you attach a screenshot + DNG?

(I did the same, but couldn't notice any mismatch...)

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: photofreak126 on April 04, 2017, 11:46:15 PM
Hi! What great stuff is happening here!

One question: I try to avoid raw because of inconvenience.

The 1:1 1920 and 1:1T work awesomely fine for me with crop_rec enabled and raw disabled. It would be so good to have 1920 60p (1920x800) in non raw mode but if I choose either 720 60p or any other Canon frame rate/resolution combo, I just get the standard rates.... Is there a trick to it or is the crop_rec just for ml_raw ??
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hyalinejim on April 04, 2017, 11:54:01 PM
So this is full frame mode, just to show the framing:
(https://s13.postimg.org/8h05h652f/20170404_224558.jpg)

And this is the DNG from UHD mode:

https://www.dropbox.com/s/xoxmrxn0hcj19qo/M04-2243_000000.dng?dl=0
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 05, 2017, 12:08:23 AM
You mean, the recorded area is not centered? That's probably true, as it can only be adjusted in rough increments. But the preview should be OK.

You can fine-tune the centering (in both directions) from the crop_rec submenu. Start values for vertical centering:
Code: [Select]
            case CROP_PRESET_UHD:
                cmos_new[1] =
                    (video_mode_fps == 24) ? PACK12(4,9)  :
                    (video_mode_fps == 25) ? PACK12(4,9)  :
                    (video_mode_fps == 30) ? PACK12(5,8)  :
                    (video_mode_fps == 50) ? PACK12(12,9) :
                    (video_mode_fps == 60) ? PACK12(13,9) :

Then, you can check with raw_diag (Downloads -> Modules) using the OB zones tool, to see whether the optical black areas are clean and useful data is present in all edges.

If you (or anyone else) finds better values, I'll include them.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 05, 2017, 01:00:40 AM
WOW!!!!!  :o just saw this new development are you serious 4K on the old 5D Mark III.  Like I've said before Magic Lantern Team never sleeps, thank you @a1ex and the whole Magic Lantern Team for those awesome new features, and previously impossible features  :D
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hyalinejim on April 05, 2017, 01:14:35 AM
Actually, the preview is off as well as the file. In 3K mode framing is correct in preview:

(https://s11.postimg.org/xpb3jor03/20170404_233407.jpg)

But in 4K it's totally off (in the last build it was fine)

(https://s14.postimg.org/n9x110mb5/20170404_233433.jpg)

I tried altering CMOS[1] lo hi values. At lo 10 the image was centered vertically but with a weird overlay on half of the screen:

(https://s10.postimg.org/e8armmym1/20170404_235712.jpg)

It's a transparent overlay of what's actually beneath the frame. I tried playing with different values for CMOS[1] lo but with no luck. In the previous build this problem cropped up (excuse the pun!) from time to time, but I could restart the camera and it was back to normal. With this build UHD is vertically shifted whereas 3K is correct.

TANGENTIAL SUGGESTION: If there's some way to reliably control the positioning of the crop area, this is a great way to turn a wide angle lens into a digital shift lens, for those interested in architectural stuff and precise composition.

Another quick suggestion: It would be nice to have more control of the vertical resolution, if that's possible. Let's say I wanted to shoot in 4:3 using a 2x anamorphic. The max res I can get is 2048 x 1536, even though my card could handle a higher resolution.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: squig on April 05, 2017, 01:50:34 AM
I got that overlay when I messed with the 1920 tall positioning on the 1st or 2nd build.

I shot standard 1080p raw vs 2.8K raw at the same ISO and aperture, taking a few steps to get close to the same framing; the 1:1 2.8K crop appears to be about half a stop brighter than the 3x3 binned 1080p. Somebody else might wanna test that to confirm.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: squig on April 05, 2017, 02:46:38 AM
Getting back into the ML menu is quite tricky on the Apr04 build.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: lostfeliz on April 05, 2017, 03:18:45 AM
@DeafEyeJedi Thanks! I didn't know it'd been updated to handle the compressed MLVs.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: mk11174 on April 05, 2017, 06:16:22 AM
So, any hints on how i can port any of the high resolution options to 700d, or are the high res options just not going to work for the lower end models like the rebels?

I looked around and read about needing to look at fps timers, but also see there's mem_engio_write address that has to be found, no idea how to find that one?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 05, 2017, 07:35:23 AM
Getting back into the ML menu is quite tricky on the Apr04 build.

?! (http://www.chiark.greenend.org.uk/~sgtatham/bugs.html)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 05, 2017, 07:46:23 AM
@a1ex Can you please point me to a guide on how to use mlv_dump for mac can't seem to get it to work with the latest development. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 05, 2017, 07:50:51 AM
mlv_dump --dng INPUT.MLV
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: squig on April 05, 2017, 08:10:41 AM
?! (http://www.chiark.greenend.org.uk/~sgtatham/bugs.html)

After recording, I have to restart to get back into the ML menu.

Are you getting the Canon spotmeter appearing during recording? Busy's still showing too.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 05, 2017, 08:46:05 AM
Can't reproduce.

Canon's focus box appears for 1-2 seconds when starting recording, then again when stopping. No BUSY message (I was getting it with previous builds). Menu works fine, both before and after. Tried 3K 24p and 1080p45, starting from ML defaults. Also tried frozen LV.

If the link wasn't obvious: http://www.chiark.greenend.org.uk/~sgtatham/bugs.html
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 05, 2017, 08:52:39 AM
@Danne Can you breakdown the process for me.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 05, 2017, 08:56:50 AM
http://www.magiclantern.fm/forum/index.php?topic=19300.msg182376#msg182376
Breakdown?

If on mac
1 - Drag mlv_dump to terminal
2 - Print --dng drag your mlv file after the word --dng in terminal and press enter.

If your version of mlv_dump says permission denied do following:

Open up terminal and print chmod u=rwx and drag mlv_dump terminal and hit enter then redo steps 1 and 2.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Teamsleepkid on April 05, 2017, 09:03:34 AM
cr2hdr for mac is very easy. even I can use it. read that it has new mlv dump. id try that.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: elieka69 on April 05, 2017, 09:10:36 AM
frame and block size mismatch:0xcba800 0x20 0x88bc00

Sent from my SM-N920I using Tapatalk

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 05, 2017, 09:13:43 AM
@Danne Thanks a lot..
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: garry23 on April 05, 2017, 09:29:54 AM
In case anyone finds what I've done helpful, I offer the following as we experiment with this exciting new ML development.

This is only for the still photographers  ;)

As at the moment LV preview is not working, so composition and exposure setting are not possible, I've written the following simple script to toggle the crop and silent features on and off (only works in LV).

This way you can compose without the feature on, toggle to use the feature (I use RATE as the toggler), do a half shutter press to get your DNG (I use DNG for stills), then toggle back.

The only 'strangeness' is that after toggling you will need to go in and out of the ML menu (TRASH) to reset the shooting state. The bottom left of the ML menu will give you feedback, eg FLV.

Here is the simple script:

Code: [Select]
--[[
Simple script to toggle Experimental FRSP on and off
Garry George April 2017
http://photography.grayheron.net/
--]]

-- Declare some globals
mod_button_1 = KEY.RATE -- key to modify
toggle_state = true

function test4mod(arg)
if lv.running then
if XFRSP_menu.submenu["Turn On/Off"].value == "On" then -- be prepared to toogle X-FRSP state
if arg == mod_button_1 then  -- toggle
if toggle_state == true then toggle_state = false else toggle_state = true end
if toggle_state then
menu.set("Movie", "Crop mode",8)
menu.set("Shoot", "Silent Picture",1)
else
menu.set("Movie", "Crop mode",0)
menu.set("Shoot", "Silent Picture",0)
end
return false -- steal key press
else
return true -- do nothing
end
else
return true -- modified key works normally
end
end
end

event.keypress = test4mod

XFRSP_menu = menu.new
{
parent = "Shoot",
name = "X-FRSP",
help = "Simply toggles the X-FRSP state",
depends_on = DEPENDS_ON.LIVEVIEW,
submenu =
{
{
name = "Turn On/Off",
help = "Switches the script on/off",
choices = {"On","Off"},
}
}
}

As I say, until the previews (hopefully) get working, this little 'hack' will allow you to experiment with still experiment full res silent.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: squig on April 05, 2017, 09:59:57 AM
Can't reproduce.

Canon's focus box appears for 1-2 seconds when starting recording, then again when stopping. No BUSY message (I was getting it with previous builds). Menu works fine, both before and after. Tried 3K 24p and 1080p45, starting from ML defaults. Also tried frozen LV.

Things started to get a bit screwy yesterday after I put ML on an SD card. I'm thinking doing the install with an experimental build may not have been a great idea. I'll do it over with a nightly.

EDIT: It was sticky half shutter. All good now.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: ju64 on April 05, 2017, 10:10:55 AM
Hi,

Testing the last build (magiclantern-crop_rec_4k.2017Apr04.5D3123) on 5D 123 and after few tests camera freezed.
I had to remove the battery and now the camera doesnt' boot as usual.
The LCD stay black, back buttons don't respond, video mode no longer works, camera can only take pictures.

On the SD card I had crash log :

Code: [Select]
ASSERT: hLvJob->hJpegMemSuite
at ./Epp/Vram/VramStage.c:891, task Epp
lv:1 mode:3


Magic Lantern version : crop_rec_4k.2017Apr04.5D3123
Mercurial changeset   : e7e1fdb85f6b (crop_rec_4k) tip
Built on 2017-04-04 15:42:08 UTC by [email protected]
Free Memory  : 168K + 3151K

And assert log :

Code: [Select]
ML ASSERT:
PROP_LEN(80040002) correct:0 called:4
at ../../src/property.c:342 (prop_request_change), task ml_init
lv:0 mode:0


Magic Lantern version : crop_rec_4k.2017Apr04.5D3123
Mercurial changeset   : e7e1fdb85f6b (crop_rec_4k) tip
Built on 2017-04-04 15:42:08 UTC by [email protected]
Free Memory  : 176K + 3964K

I went back to previous working version (compressed_raw.2017Apr01.5D3123), same symptoms and I had this crash log :

Code: [Select]
ASSERT: pStruct->Data
at GuiProperty.c:1020, task PropMgr
lv:0 mode:0


Magic Lantern version : compressed_raw.2017Apr01.5D3123
Mercurial changeset   : 706183bf6459 (compressed_raw) tip
Built on 2017-04-01 11:58:46 UTC by [email protected]
Free Memory  : 170K + 3753K

And this assert log :

Code: [Select]
ML ASSERT:
PROP_LEN(80040002) correct:0 called:4
at ../../src/property.c:342 (prop_request_change), task ml_init
lv:0 mode:0


Magic Lantern version : compressed_raw.2017Apr01.5D3123
Mercurial changeset   : 706183bf6459 (compressed_raw) tip
Built on 2017-04-01 11:58:46 UTC by [email protected]
Free Memory  : 192K + 3983K

Sometime I get Err 80 too.

Any idea of what happen to my camera ?  :-[
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: seanvdeusen on April 05, 2017, 10:26:10 AM
Idk if anyone ran into this issue like I did. But if you're having trouble with dumping the files or importing once converted here's what i did.

1. Open Notepad and type in

@echo off
for %%a in ( %* ) do ( md "%%~na" 2>nul )&( "INSERT FILE LOCATION HERE" -o %%~na\%%~na_ --dng %%~na.MLV )

ALSO don't type in INSERT FILE LOCATION. Thats where you place the location of the folder you're working in. :p

2. Save that file and what ever you name it (I NAMED MINE DUMP) be sure to add .cmd at the end of it. Which will make it a CMD file.

3. Place the MLV_DUMP file into the same folder as the the previous file we just made.

4. Drag you're MLV file to that location and then drag it over the CMD you made.

5. You should now have a folder with a bunch of DNG files.

6. If you can't open the dng files and get errors in let's say After Effects. You need to deleted the The first DNG file in that folder. It's the one with just zeros at the end of it.

I'm use to magic lantern MLV files working with my normal programs. So if this is already old news i'm sorry for posting. I did however see a few questions regarding this issue. Some are getting pink dots or weird rainbow-ish frames.


Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 05, 2017, 11:03:59 AM
PROP_LEN(80040002) correct:0 called:4
at ../../src/property.c:342 (prop_request_change), task ml_init

This one sounds bad. My advice:
1) make a backup copy of your ROM if you haven't already (it's on the card, under ML/LOGS)
2) start the camera without cards and try to clear Canon settings

If that doesn't help, we'll have to investigate what happened.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: ju64 on April 05, 2017, 11:22:38 AM
Thanks a1ex

1) Backup of ROM is already done
2) How can I clear Canon settings ? LCD doesn't works...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 05, 2017, 11:25:54 AM
Got it, let's chat on IRC for further troubleshooting.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 05, 2017, 11:44:50 AM
Quote
Quote from: pc_bel on Yesterday at 10:13:46 AM

    ML ASSERT:0at mlv_lite.c:1935 (compress_task)


This one indicates frame corruption, and should also print this (unless obscured by other messages):
Code: [Select]

                printf("\nCompressed size higher than uncompressed - corrupted frame?\n");
                printf("Please reboot, then decrease vertical resolution in crop_rec menu.\n\n");

Now with new April4th build, yesterday problems are gone. Testing 1920 50/60 3x3. Camera boots every time I turn on as expected... WAIT A MOMENT !!!

While I was writting this post with camera on, I've heard the camera disconnect live view and connected again. On screen find for a moment a error message. I didn't have time to read it, just "created a LOG file" or something similar.

This is the LOG:

Wed Apr  5 11:31:33 2017
      0:    15.149 [STARTUP]
K285 ICU Firmware Version 1.1.3 ( 6.9.4 )
      1:    15.177 [STARTUP]
ICU Release DateTime 2012.05.16 14:14:46
      2:    16.597 [PROPAD] PROPAD_CreateFROMPropertyHandle DRAMAddr 0x41744000
      3:    16.886 [SEQ] seqEventDispatch (Startup, 0)
      4:    16.930 [STARTUP] startupEntry
      5:   129.073 [SEQ] seqEventDispatch (Startup, 1)
      6:   129.148 [STARTUP] startupPrepareProperty
      7:   141.147 [SND] Seq LPC fin
      8:   142.415 [RSC] hMemoryQueue (0x6A0010) hStorageQueue (0x6C0012)
      9:   144.775 [RSC] MemoryStatusMasterResultCBR
     10:   188.740 [MRK] MRK_RegisterSpaceNotifyCallback
     11:   188.883 [MRK] MRK_RegisterSpaceNotifyCallback
     12:   188.935 [FM] FM_RegisterSpaceNotifyCallback
     13:   189.503 [JOB] InitializeJobClass (ID = 858, Num = 13)
     14:   196.161 [EM] emLockControl (TYPE_JOBSTATE = 0x0)
     15:   199.904 [JOB] InitializeInnerDevelopJobClass (ID = 858, Num = 16)
     16:   199.946 [JOB] InitializeMultipleExposureShootJobClass ( Num = 27,11 )
     17:   199.981 [JOB] InitializeMultipleExposureSaveAndEndJobClass ( Num = 13 )
     18:   200.020 [JOB] InitializeHDRShootJobClass ( Num = 16 )
     19:   200.053 [JOB] InitializeHDRSaveAndEndJobClass ( Num = 33 )
     20:   200.321 [TERMINATE] SHUTDOWN init comp
     21:   200.429 [TERMINATE] Abort init comp
     22:   202.132 [STARTUP] startupPropAdminMain : End
     23:   209.359 [FM] fmPrepare : Completed
     24:   209.511 [SEQ] seqEventDispatch (Startup, 2)
     25:   209.884 [SHTRD] [MNAV] MEMNAVI_Initialize
     26:   210.049 [SHTRD] [MNAV] MEMNAVI_SetMemoryMap: 1
     27:   211.874 [ENG] [ENGIO](Addr:0x41700000, Data:0x   44000)
     28:   247.456 [SEQ] seqEventDispatch (Startup, 3)
     29:   249.164 [MC] PROP_GUI_STATE 0
     30:   250.138 [MC] [MC] PROP_HEADPHONE_VOLUME_VALUE : 0
     31:   250.286 [MC] [MC] PROP_MOVIE_PLAY_VOLUME : 5
     32:   251.295 [MC] regist master CardCover
     33:   253.856 [PRP] M:98 F:97 L:0 P:32 T:7
     34:   254.010 [STARTUP] update coded version.
     35:   254.110 [STARTUP] internal version 06:09:04:98:97:00:32:07:3c:05:d0:d0:d0
     36:   259.367 [STARTUP] startupPrepareDevelop
     37:   262.974 [PROPAD] Done Rebuild Tune
     38:   264.107 [MAC] MAC_Initialize
     39:   264.479 [MAC] K_BOARDID4
     40:   264.521 [MAC] K_BOARDID3
     41:   264.555 [MAC] K_BOARDID2
     42:   264.586 [MAC] K_BOARDID1
     43:   264.757 [MAC] Key=0x0 Board=0x0 Body=0x734bbf81
     44:   277.644 [SHTP] Regist RPC Handler Master ALO
     45:   281.440 [PROPAD] PROPAD_CreateFROMPropertyHandle DRAMAddr 0x41b2c000
     46:   393.855 [MR] mvrChangeAckCBR : Video - Mode=0, Type=1, Rate=50, GOP=1
     47:   393.981 [MR] mvrChangeAckCBR : Sound Manual(C=2,R=48000,B=16)
     48:   394.008 [MR] mvrChangeAckCBR : AE_MODE_MOVIE(3)
     49:   394.024 [MR] mvrChangeAckCBR : ISO(104)
     50:   394.178 [MR] mvrChangeAckCBR : MOVIE_REC_VOLUME(L:20 R:20)
     51:   394.228 [MR] mvrChangeAckCBR : WindCut Off
     52:   394.333 [MR] TimeCode Base:11526,11,530583872,0
     53:   394.378 [MR] TimeCode Mode:1
     54:   394.424 [MR] TimeCode DropFrame Mode:0
     55:   394.468 [MR] TimeCode Disp Mode:0
     56:   394.503 [MR] PROP_FILE_PREFIX:8W1A
     57:   394.534 [MR] PROP_USER_FILE_PREFIX:PAU_,
     58:   394.561 [MR] PROP_USER_FILE_PREFIX:PAU,
     59:   394.588 [MR] PROP_SELECTED_FILE_PREFIX:PAU_,1
     60:   402.263 [LVCOM] LiveView Gain Table Ver0.4
     61:   403.395 [HPC] ReserveHPCopyChannel (1, 116)
     62:   411.083 [MR] encdrwrapHPCopyAsyncCBR
     63:   412.885 [LVGMT] PROP_TEMP_STATUS : STATUS_NORMAL
     64:   418.280 [CAPD] LiveView Gain Table Ver0.4
     65:   421.293 [LVCOM] InitializeLiveViewDefectDetection
     66:   421.798 [LVCOM] ExecuteDefectMarge Start
     67:   421.882 [LVCOM] ExecuteDefectMarge1[0x41dd9b00:61440]
     68:   423.164 [LVCOM] A:[59] B:[9040]
     69:   427.618 [LVCOM] ExecuteDefectMarge2[0x41de8b00:61440]
     70:   428.697 [LVCOM] A:[35] B:[7713]
     71:   432.633 [LVCOM] ExecuteDefectMarge3[0x41df7b00:32768]
     72:   435.697 [LVCOM] ExecuteDefectMarge End
     73:   436.838 [LVEPP] [REMO] Jpeg 16:9 PosW: 0 PosH: 300
     74:   440.914 [PROP_AF_ASSIST_BEAM_FIRING] 0
     75:   441.074 CurrentMedia = 2
     76:   443.927 [STARTUP] startupPrepareRemote
     77:   445.087 [AUDIO] RegisterCBRSDIOCableConnect
     78:   452.462 [WFT] PROP_ADAPTER_DEVICE_ACTIVE: Can Not Change(0) Connect(0)
     79:   463.393 [RMT] QuickTransmissionEnable (0, 1, 2, 2)
     80:   463.438 [RMT] PROP_CONNECT_TARGET (0x0 <= 0x0)
     81:   463.475 [RMT] PROP_CONNECT_TARGET_WFT (0x0 <= 0x0)
     82:   463.531 [RMT] PROP_WFT_SYSTEM : 0
     83:   463.961 [PTP] PSI UnlockTransImg : Trans List is NULL
     84:   463.995 [PTP] PSI TarminateTransImgList : SessionTarget [1]
     85:   464.020 [PTP] PSI TarminateTransImgList : Trans List is NULL
     86:   464.133 [PTP] PSI UnlockTransImg : Trans List is NULL
     87:   464.158 [PTP] PSI TarminateTransImgList : SessionTarget [10]
     88:   464.180 [PTP] PSI TarminateTransImgList : Trans List is NULL
     89:   464.259 [PTP] PSI DisconnectViewFinder
     90:   464.763 [PTP] PROP_SCREEN_SAVER 0
     91:   464.926 [PTPCOM] SetPtpTransportResources:0,323a
     92:   465.189 [PTP] PSI UnlockTransImg : Trans List is NULL
     93:   465.221 [PTP] PSI TarminateTransImgList : SessionTarget [10]
     94:   465.242 [PTP] PSI TarminateTransImgList : Trans List is NULL
     95:   475.488 [WFT] Regist PropCBR PROP_WFT_BLUETOOTH
     96:   786.917 [SD] sdSendOCR: 1.8V Signaling Card
     97:   789.340 [SD] sdSendOCR: Hi-Capacity Card
     98:   795.978 [SD] CARD CAPACITY is 15193Mbyte( 31116288Sec )
     99:   812.798 [SD] SD_GetAccessMode=3
    100:   812.816 [SD] Set Hi-Speed Mode( 48MHz )
    101:   821.024 [FSU] fsuGetPart: Block(8192, 31108096, 31116288)
    102:   862.189 [FM] FileNo (DriveNo = 2, 0 <-> 157)
    103:   862.304 [FC] _FC_OpenCatalog : wNumOfDir 1
    104:   867.175 [MRK] MRK_Initialize
    105:   867.760 [MRK] MRK_PrepareMrkFileObject
    106:   870.803 [SEQ] seqEventDispatch (Startup, 4)
    107:   870.977 [DP] DP_Initialize()
    108:   871.656 [DP] Std T_PrintPage[0] T_PrintNo[0] R_PrintNo[0]
    109:   871.694 [DP] Std C_PrintNo[0] C_PrintPage[0], C_Copies[0]
    110:   871.788 [DP] Dp_SetPD_RegistApi Prev:0 New:0
    111:   872.148 [DP] PROP_CONNECT_TARGET [0x00000000]
    112:   872.268 [DP] RD_Cancel No Issue
    113:   872.501 [DP] [GYO]DpFunctionList[0x10117fff]
    114:   872.612 [DP] X FreeMem Address[0x00000000]
    115:   876.710 [DISP] HPD before InitializeDisplay !!!
    116:   876.961 [DISP] [ME] PropMgr (0) [SlaveResultCBR] (S_OFF)
    117:   877.080 [DISP] [ME] PropMgr (0) [ChangeCBR] (S_OFF)
    118:   877.103 [DISP] Display_flag = 1
    119:   927.579 [IMPP] H264E InitializeH264EncodeFor1080pDZoom
    120:   927.638 [IMPP] H264E InitializeH264EncodeFor1080p25fpsDZoom
    121:   954.964 [DP] DP_RegisterCBR() Id=1 Count:12
    122:   955.003 [DP] DP_RegisterCBR() Id=2 Count:12
    123:   955.030 [DP] DP_RegisterCBR() Id=4 Count:12
    124:   964.382 [GUI] MainEventHandler PROP_QR_DIDNOT_EXECUTE(0)(0)
    125:   966.664 [GUI] MainEventHandler PROP_ERROR_FOR_DISPLAY(0)
    126:   993.693 [LVGMT] Is PROP_OUTPUT_TYPE Real??[H:1][V:0][O:0]
    127:  1001.225 [MR] mvrChangeAckCBR : Video - Mode=0, Type=1, Rate=50, GOP=1
    128:  1002.062 [GUI] MainEventHandler PROP_LV_OUTPUT_DEVICE(0)
    129:  1003.208 [LVGMT] Is PROP_OUTPUT_TYPE Real??[H:1][V:0][O:0]
    130:  1004.069 [MAGIC] (Empty Func) MVW_RegisterXmpDataCallback
    131:  1005.200 [HDMI] HDMI CEC Initialize
    132:  1006.225 [MR] mvrChangeAckCBR : Video - Mode=0, Type=1, Rate=50, GOP=1
    133:  1007.428 [GUI] ERROR ***** ConvertHdmiVideoCodeToGuiDisplayType unknown Type(0)
    134:  1007.973 [HDMI] HPD ON
    135:  1007.997 [HDMI] ConnectHDMI
    136:  1008.023 [HDMI] InitializePortForHDMI
    137:  1008.049 [HDMI] ActiveHDMIDevice
    138:  1008.072 [HDMI] I2C_Init
    139:  1008.162 [HDMI] Pana_Init
    140:  1010.391 [SEQ] seqEventDispatch (Startup, 5)
    141:  1010.413 [STARTUP] startupInitializeComplete
    142:  1010.499 [MC] cam event guimode comp. 0
    143:  1011.314 [MC] cam event guimode comp. 0
    144:  1012.443 [DISP] BrightnessControl from GUI 1
    145:  1013.765 [DL] PROP_ACTIVE_SWEEP_STATUS (0)
    146:  1013.799 [DL] PROP_DL_ACTION (1)
    147:  1018.304 [MC] PROP_GUI_STATE 2
    148:  1021.169 [MC] PROP_GUI_STATE 2
    149:  1022.224 [GUI] MainEventHandler PROP_LV_OUTPUT_DEVICE(0)
    150:  1023.042 [MC] PROP_GUI_STATE 2
    151:  1023.944 [GUI] IDLEHandler START_WARNING_LENSLESS_MOVIE_MODE
    152:  1024.027 [MC] cam event guimode comp. 2
    153:  1034.709 [WINSYS] Zlib maxsize update 884
    154:  1034.763 [WINSYS] Zlib maxsize update 1396
    155:  1060.186 [GUI] IDLEHandler START_WARNING_LENSLESS_MOVIE_MODE
    156:  1060.259 [MC] cam event guimode comp. 2
    157:  1096.103 [GUI] IDLEHandler START_WARNING_LENSLESS_MOVIE_MODE
    158:  1096.175 [MC] cam event guimode comp. 2
    159:  1118.924 [FA] ChangeCBR(ID=0x8003000a)
    160:  1119.598 [FA] RequestFromCheckData
    161:  1119.866 [FA] MpuMonSpecificFromPartner : COM_FA_CHECK_FROM 0
    162:  1128.724 [SEQ] seqEventDispatch (Startup) : End
    163:  1133.365 [HDMI] RESET_UNCOMPLETE
    164:  1136.719 [HDMI] RESET_COMPLETE
    165:  1148.561 [HDMI] KEY LOAD OK
    166:  1148.595 [HDMI] HOT_PULG_DETECT OK
    167:  1198.979 [HDMI] EDID_PREREAD OK
    168:  1199.001 [HDMI] Rx Sense is OK
    169:  1199.621 [HDMI] pfHdmiPbCBR Done
    170:  1253.694 [HDMI] TMDS Check
 
I was pretty sure that everything was going pretty well until this happened.

Hope it helps for developement...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 05, 2017, 12:22:36 PM
Were you using a HDMI monitor? I didn't test this configuration yet.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: garry23 on April 05, 2017, 12:23:53 PM
Spreading the word  ;) http://photography.grayheron.net/2017/04/magic-lantern-does-it-again.html (http://photography.grayheron.net/2017/04/magic-lantern-does-it-again.html)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 05, 2017, 12:25:18 PM
Quote
Were you using a HDMI monitor? I didn't test this configuration yet.

Yes, a Zacuto EVF... It can be the problem?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 05, 2017, 03:20:15 PM
Trying to reproduce the live view disconnection, I let camera on... This time get this message:
CTRL SRV Stack Overflow free=84 Used=163008176
Camera off.
Camera on.
I can record continuous 1920x960 50p 3x3
Camera remains on.
Suddently get this message in top of screen (in red letters):
[103] COMPRES_TASK: TASK WARNING: Free=164 Used=3932
Camera off.
When I boot camera again, I get this message full screen:
ML ASSERT: 0 at mlv_lite.c:2006 (compress_task), task compress_task vv:1 mode:3

srm_malloc_suite(1)...
srm buffer #1: 48600064
srm_malloc_suite(f1408)
Resolution changed: 2080x1587 -> 2080x961
Black level: 2047

Compressed size higher than uncompressed - corrupted frame?
Please reboot, then decrease vertical resolution in crop_rec menu.

ML ASSERT: 0 at mlv_lite.c:2006 (compress_task), task compress_task vv:1 mode:3


I didn't change any resolution... In crop_rec menu it remains 1920x960.
I'm posting all my troubles... I have no idea if it is useful or not. If not, I will stop posting, just let me know  ;)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: garry23 on April 05, 2017, 03:29:49 PM
@A1ex

Am I right in saying the key.press(KEY.HALFSHUTTER) wont (sic) work in the experimental 4K at the moment? It doesn't seem to.

I'm trying to explore triggering the experimental FRSP(4K version) in  Lua script.

Or am I jumping the gun  ;)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 05, 2017, 04:19:56 PM
Quote
Am I right in saying the key.press(KEY.HALFSHUTTER) wont (sic) work in the experimental 4K at the moment? It doesn't seem to.

Right - will take care of this (as I want to merge lua_fix in regular nightlies anyway).

Quote
CTRL SRV Stack Overflow free=84 Used=163008176

Would be great if you can find a way to reproduce this.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: garry23 on April 05, 2017, 04:28:18 PM
@A1ex

 :) Great  :)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kyle Kearns on April 05, 2017, 07:12:41 PM
Does a 1:1 crop mean it's a 2x crop similar to a GH4?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hyalinejim on April 05, 2017, 07:24:24 PM
The references to 1:1 mean that every pixel on the sensor becomes a pixel in the image. The "crop factor" depends on the resolution you choose.

 A 1920 x 1080 1:1 central section of the sensor (5760 pixels wide) gives a crop factor of 3.

2880 x 1620 is a 2x crop factor.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 05, 2017, 07:28:37 PM
@Kyle Kearns Naw.. 3K 1:1 I believe is 1.84 -  UHD is 1.5 -  True 4K is 1.4
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 05, 2017, 09:03:28 PM
Quote
Quote

    CTRL SRV Stack Overflow free=84 Used=163008176


Would be great if you can find a way to reproduce this.

After 2 hours I'm not able to reproduce CTRL SRV...

Quote
Suddently get this message in top of screen (in red letters):
[103] COMPRES_TASK: TASK WARNING: Free=164 Used=3932
Camera off.
When I boot camera again, I get this message full screen:
ML ASSERT: 0 at mlv_lite.c:2006 (compress_task), task compress_task vv:1 mode:3

srm_malloc_suite(1)...
srm buffer #1: 48600064
srm_malloc_suite(f1408)
Resolution changed: 2080x1587 -> 2080x961
Black level: 2047

Compressed size higher than uncompressed - corrupted frame?
Please reboot, then decrease vertical resolution in crop_rec menu.

ML ASSERT: 0 at mlv_lite.c:2006 (compress_task), task compress_task vv:1 mode:3

This ML ASSERT message appears when in crop_rec mode 1920 50p 3x3, I change (in Canon menu) from 1280x720 50 All-I to 1920x1080 25 All-I before change crop_rec to OFF. I need to reboot 2 times and then all come back to normal operation...

I will try again to reproduce the CTRL SRV message :)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Doofii on April 06, 2017, 03:29:11 AM
select 14bit lossless,and "frame and block size mismatch:0xcba800 0x20 0x88bc00" shows when play back.
ive tired many times,but the problem still exists.
what could i do?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 06, 2017, 05:52:35 AM
I shot something like 12 takes.
I tried 14lossless and got a warning at the top of the screen (couldnt read due to age)
and a series of many warnings.
I removed the battery, start from scratch, default ML settings, load modules.
I set 10bit
Set the Canon video to 1280x720, NTSC.
Everything as expected.

In order to reach optimum performance, I set the Canon video back to 1920x1080, PAL.
The resolutions 3K, UHD and 4K had less height that the promised in ML forums.
e.g 3072x1728
In these resolutions, I couldnt record more than 3 secs maximum.
Surprisingly I managed to record 3568x1320 for much longer duration. <--- pretty strange not to be able to record longer for lower resolution



Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: martinhering on April 06, 2017, 09:42:55 AM
I was experimenting with the lossless lj92 compression in MLV files. I wanted to see if I can decode the image and reinterpret it as a raw cfa pattern. I came across a number of problems that could also be a problem in mlv_dump. I took the liblj92 implementation by Andreaw Baldwin found here:
https://bitbucket.org/hudson/magic-lantern/src/7a9b6805756c3b86f2174bac00433c544a976501/modules/mlv_rec/lj92.c?at=mlv_rec_lj92&fileviewer=file-view-default (https://bitbucket.org/hudson/magic-lantern/src/7a9b6805756c3b86f2174bac00433c544a976501/modules/mlv_rec/lj92.c?at=mlv_rec_lj92&fileviewer=file-view-default)

But I could not make it work. I had decoding problems due to the fact that the library is missing some support for multiple components. I then came across the implementation in tiny_dng_loader by Syoyo Fujita here:
https://github.com/syoyo/tinydngloader/blob/master/tiny_dng_loader.h (https://github.com/syoyo/tinydngloader/blob/master/tiny_dng_loader.h)

I then extracted his changes back into a simple standalone .c and .h file and tried that and viola, the decoding result was much better:
(http://i.imgur.com/5C88tC6.jpg)

I posted the extracted version of the library on GitHub:
https://github.com/martinhering/liblj92 (https://github.com/martinhering/liblj92)

Just saying, if you encounter any decoding problems in mlv_dump with the mlv_rec_lj92 branch, give my version a try.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 06, 2017, 10:08:21 AM
Quote
I shot something like 12 takes.
I tried 14lossless and got a warning at the top of the screen (couldnt read due to age)
and a series of many warnings.
I removed the battery, start from scratch, defauly ML settings, load modules.
I set 10bit
Set the Canon video to 1280x720, NTSC.
Everything as expected.

Yes, same as me. Now message is:

[76] CEC: stack warning: free=128 used=3968

I'm totally lost...
@A1ex
I'don't know what to do, so if you can point me to what to do to help, just let me know.

After the messages I can reboot and I can record continuous without corrupted frames (allways talking about crop_rec April 4th build, 1920 50p 3x3 mode)... so this is not a big problem, but I have no idea of code so no idea what all this messages mean and what to do... sorry  :-[ I'm limited to explain what happens...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 06, 2017, 10:32:12 AM
Did some changes that might help with memory corruption (new build posted).

In particular, I think I've found a way to limit compressed data size when it ends up larger than the allocated buffer. Previously, passing an arbitrary limit for buffer size simply locked up the camera at certain resolutions - it appears this limit must be aligned at 4096-byte multiples (maybe a bit less, but in any case, at least 1024).

If you still get stack overflow, please report even if you don't have a way to reproduce it. This kind of bug is usually very serious.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: garry23 on April 06, 2017, 10:59:43 AM
@A1ex

Just checking: I assume this build doesn't include the Lua fix yet? I doesn't look if it does.

Cheers

Garry
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 06, 2017, 11:05:45 AM
Just tested April 6th 113.

Can't get continuous recording like before with april4th, only 10 sec. before message "didn't make it to stimated record time" and then recording stops. (1920 50p 3x3 mode).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: GutterPump on April 06, 2017, 11:12:03 AM
This kind of bug is usually very serious.

Is this mean we could get a semi brick or definitive brick ? What is the long-term risk ?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 06, 2017, 11:19:03 AM
@pc_bel: nothing was changed on the recording speed side; I bet any difference is because of different scene complexity. Please test the stack overflow bug first.

@garry23: Lua changes are a bit too much when I'm troubleshooting a memory corruption bug. The changelog from the Experiments page is valid.

Is this mean we could get a semi brick or definitive brick ? What is the long-term risk ?

Yes, as with all ML builds. Explained a little here (http://www.magiclantern.fm/forum/index.php?topic=12627.msg170133#msg170133) and also somewhere in the Tragic Lantern threads. Currently I'm diagnosing this report (http://www.magiclantern.fm/forum/index.php?topic=19300.msg182395#msg182395), and it appears to be, indeed, caused by some bad settings reflashed by Canon code into ROM - possibly after memory corruption caused by ML.

The warnings from the download page are not there for fun - this kind of errors cannot be prevented on the current processor design, as it doesn't have a MMU (also valid for many other simple CPUs that use non-volatile storage updated frequently). Canon started to use a processor with MMU with DIGIC 7 (https://chdk.setepontos.com/index.php?topic=13014.30), but they don't seem to use its memory protection features (they simply do a flat memory mapping, except a 0x1000-byte block unique to each core).chd

Just in case, make sure you have a copy of your ROM in a safe place (any recent installer tells you how to do so).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 06, 2017, 11:23:05 AM
Quote
Just tested April 6th 113.

Can't get continuous recording like before with april4th, only 10 sec. before message "didn't make it to stimated record time" and then recording stops. (1920 50p 3x3 mode).

After reboot, I get the same message as with the april4th build:
ML ASSERT: 0 at mlv_lite.c:2006 (compress_task), task compress_task vv:1 mode:3

srm_malloc_suite(1)...
srm buffer #1: 48600064
srm_malloc_suite(f1408)
Resolution changed: 2080x1587 -> 2080x961
Black level: 2047

Compressed size higher than uncompressed - corrupted frame?
Please reboot, then decrease vertical resolution in crop_rec menu.

ML ASSERT: 0 at mlv_lite.c:2006 (compress_task), task compress_task vv:1 mode:3
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 06, 2017, 11:24:46 AM
ML ASSERT: 0 at mlv_lite.c:2006 (compress_task), task compress_task vv:1 mode:3

Fortunately, that's no longer causing memory corruption (so it's a minor bug for now). It's simply invalid image data being compressed (and output size resulting higher than initial size). The extra data is no longer written to RAM on latest build (and before, it was written in the place reserved for future frames, not on DryOS data structures or persistent setting areas - to my knowledge).

The stack overflows are worrying me, though (as they do indicate memory corruption in the middle of DryOS).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: BBA on April 06, 2017, 12:08:50 PM
Thanks for those groundbreaking efforts  :) :) !!!!!!

Don't know if it helps. If I can remember...

I got just while moving in the menu but without CF card (only SD):
white flashing "[93] audio-common-task stack warning free=180 used=3016"
then red flashing "[91] clock-task st(don't remember the end of the word) free=36 used=8156"

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: jankrueck on April 06, 2017, 12:21:16 PM
Code: [Select]
ASSERT: this->VramNv12Param.hJpegMemSuite
at ./Epp/Vram/VramStage.c:745, task Epp
lv:1 mode:3


Magic Lantern version : crop_rec_4k.2017Apr04.5D3113
Mercurial changeset   : e7e1fdb85f6b (crop_rec_4k) tip
Built on 2017-04-04 15:41:29 UTC by [email protected]
Free Memory  : 146K + 3094K



Code: [Select]
ASSERT: 0
at ./PathDrvWrap/EekoYuvToVramPath.c:606, task Epp
lv:0 mode:3


Magic Lantern version : crop_rec_4k.2017Apr04.5D3113
Mercurial changeset   : e7e1fdb85f6b (crop_rec_4k) tip
Built on 2017-04-04 15:41:29 UTC by [email protected]
Free Memory  : 158K + 3072K


this was on 4th. didn't see there is a new version.
testing up to date version atm.

looks good so far.



I have little knowledge on how you managed to achiev this update.
But I assume, there will be no way to use an external monitor on this, right?
HDMI connectors are limeted to 1080p huh?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: BBA on April 06, 2017, 12:40:10 PM
Without ML (SD card with ML removed) and only CF card (data card):
In Canon menu, the sensor cleaning gets disabled when the camera is switched on.
(I am with the 4th april build)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 06, 2017, 04:52:46 PM
For now, all I could get are these error messages:

ASSERT: hLvJob->hJpegMemSuite
at ./Epp/Vram/VramStage.c:868, task Epp
lv:1 mode:3
Magic Lantern version : crop_rec_4k.2017Apr06.5D3113
Mercurial changeset   : bc99f7502deb (crop_rec_4k) tip
Built on 2017-04-06 08:28:51 UTC by [email protected]
Free Memory  : 172K + 3779K

ASSERT: IsSuiteSignature( hSuite )
at ./PackMemory/PackMem.c:599, task RscMgr
lv:0 mode:3
Magic Lantern version : crop_rec_4k.2017Apr06.5D3113
Mercurial changeset   : bc99f7502deb (crop_rec_4k) tip
Built on 2017-04-06 08:28:51 UTC by [email protected]
Free Memory  : 172K + 3778K

ASSERT: fInitialize == TRUE
at ./System/PostPostman/PostPostman.c:243, task Epp
lv:0 mode:3
Magic Lantern version : crop_rec_4k.2017Apr06.5D3113
Mercurial changeset   : bc99f7502deb (crop_rec_4k) tip
Built on 2017-04-06 08:28:51 UTC by [email protected]
Free Memory  : 172K + 3781K




Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 06, 2017, 05:44:55 PM
Without ML (SD card with ML removed) and only CF card (data card):
In Canon menu, the sensor cleaning gets disabled when the camera is switched on.
(I am with the 4th april build)

Does clearing Canon settings help? Otherwise, do you have a screenshot, so I know what to look for?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 06, 2017, 06:04:04 PM
Day of testing!!! :)

I returned to crop_rec 113 april4th build due to the short recorded times I get with april6th (I don't know why...)  :(

Really surprised for the extremely low rolling shutter in 1920 50p 3x3 mode... but panning, I get a cut line in the top part of the frame, only with panning! it dissapears in static frames.

This video explains better:
http://ovh.to/b2isZ5B

Edited: I can't see the cut line in live view, only in processed dng's.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: jankrueck on April 06, 2017, 06:20:07 PM
Really surprised for the extremely low rolling shutter in 1920 50p 3x3 mode... but panning, I get a cut line in the top part of the frame, only with panning! it dissapears in static frames.
Edited: I can't see the cut line in live view, only in processed dng's.

same here. got this strange line, when panning.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: garry23 on April 06, 2017, 06:23:38 PM
@A1ex

I've continued 'playing around' with the latest experiments in still photography and am a bit confused. ML seems to be creating 'time dilation in the space-time continuum'  :)

I set up a long exposure test, 30s, and took a base .cr2 capture.

I then kept this exposure and used the full res LV in the experimental build.

As soon as I pressed the half shutter press, I got the saving message. I pull the card immediately.

I repeated and left the card in for a minute.

The two DNGs were the same, ie I seem to get a 30s silent in a couple of seconds.

The cr2 and the two DNGs were exposed OK, ie ETTR like and not overexposed.

I obviously don't understand how long exposures are working with the crop/silent stills.

When you have time, I would welcome any insight.

Cheers

Garry
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 06, 2017, 06:33:29 PM
I returned to crop_rec 113 april4th build due to the short recorded times I get with april6th (I don't know why...)  :(

Proof that it has shorter recording times, please! A static test scene (without varying light levels and with the same focus), 10 test runs with each build, starting from a formatted card, at the same resolution and ISO, should do the trick.

I set up a long exposure test, 30s, and took a base .cr2 capture.

I then kept this exposure and used the full res LV in the experimental build.

As soon as I pressed the half shutter press, I got the saving message. I pull the card immediately.

The two DNGs were the same, ie I seem to get a 30s silent in a couple of seconds.

The cr2 and the two DNGs were exposed OK, ie ETTR like and not overexposed.

I obviously don't understand how long exposures are working with the crop/silent stills.

Best guess: you have tried in photo mode (which I didn't try yet), with exposure simulation. So, if you have metered a 30s ISO 100, the LiveView is probably at 1/FPS and a very high ISO. An example, if FPS override was set to 5, the LiveView frame might have been exposed at 1/5 seconds, aperture wide open - even if you have selected something different in menu - and ISO as high as required to get the same brightness as with a 30-second exposure (considering the other two variables).

I'm curious to see the DNGs.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: garry23 on April 06, 2017, 06:41:25 PM
@A1ex

No EXIF so can't tell other than looking at them.

Canon and ML ISO settings remain at 100.

Tried with expo sim off. Saving straight away, i.e. Doesn't do a 30 sec capture.

Cheers

Garry
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: garry23 on April 06, 2017, 07:07:04 PM
@A1ex

Sitting here trying to think this through.

As this is coming from a video development, I guess we will be limited, at ISO100, to shutter speeds that are consistent with video framing rates.

I guess this means this still photography approach will solve fast shutter speeds, but not give ISO100 slow shutter speeds.

Or have I got it wrong?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Frank7D on April 06, 2017, 07:28:10 PM
pc_bel said:
Quote
but panning, I get a cut line in the top part of the frame, only with panning! it dissapears in static frames.

In other contexts, this was theorized to be a symptom of overloading the processor. I used to get it with greyscale ml preview on my 7D in 5x crop mode.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: BBA on April 06, 2017, 08:25:57 PM
@A1ex
Thanks for your assistance ! Don't hesitate to ask more if necessary ! I am not stuck  ;) .

a) more infos about my problem:

    a.1) without ML card (I use the SD card for ML)
    a.2) CF card in camera (formatted in camera)
    a.3) I go to Canon menu : yellow settings menu / SETUP3 / "sensor cleaning" and set "auto cleaning" to "enabled"
    a.4) I switch off the camera : the sensor cleaning screen appears as before; the red bottom led is not lit.
    a.5) I then switch on the camera : the sensor cleaning screen does Not appear; the bottom red led is lit.
    a.6) if I go again to the Canon menu : yellow settings menu / SETUP3 / "sensor cleaning", the "auto cleaning" has been set to "disabled"

b) reinit all settings : no change

    b.1) Canon menu / yellow settings menu / SETUP4 / "reinit all settings"
    b.2) the "auto cleaning" feature is actually set to "enabled" if I verify in the Canon menu
    b.3) then I get the same as above in a.4) , a.5)  and  a.6)

[EDITED 2017/04/07]
c) circumstances ? no CF card ?

Unknown

I think (not sure at all) it may be due to the fact that I tried to repeat the problem I got with no CF card in the camera (silly thing : should not be done without further investigation): the camera got stuck; no message ; I had to remove the battery.
Investigated by A1ex (thanks a lot) : no such problem when switched on with no CF card (see post #156). 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 06, 2017, 09:16:40 PM
@BBA: could not reproduce.

Side note: I've let the camera on for 10 hours, on a power adapter, with today's build, on the 3K video mode, 3072x1920 in mlv_lite menu, lossless compression, "framing" preview, in standby - so it would estimate the compression ratio every 2 seconds. Every 30 minutes, it closed LiveView for a second, and returned to LV (I could hear it when I came home).

Temperature after this experiment was 62 degrees C. The camera was enclosed in a drawer during the entire experiment.

I've now downloaded the previous build (Apr04) and will perform the same test overnight (or until it crashes).

edit: it crashed after 15 minutes or so of recording short test clips in 50p 3x3; back to Apr06, no crashes yet.

I'm doing these tests to check these reports:

I tried 14lossless and got a warning at the top of the screen (couldnt read due to age)
and a series of many warnings.

Trying to reproduce the live view disconnection, I let camera on... This time get this message:
CTRL SRV Stack Overflow free=84 Used=163008176
[...]
Suddently get this message in top of screen (in red letters):
[103] COMPRES_TASK: TASK WARNING: Free=164 Used=3932
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: garry23 on April 06, 2017, 09:28:16 PM
@A1ex

Ok I get it now  ;)

As you say on the first post

Quote
The last feature complements the well-known full-resolution silent pictures - the new implementation will be usable at fast shutter speeds, without the exposure gradient - but with rolling shutter (just like regular LiveView frames).

Photographers need to look to using the crop approach and the current FRSP, together.

I'm looking forward to trying to script these two techniques together in Lua  :)

Really exciting and fun times.

Cheers

Garry
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 06, 2017, 09:39:52 PM
Please follow the same steps as ju64 and contact me on IRC.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Walter Schulz on April 06, 2017, 11:14:36 PM
Upper right corner:
IRC: #magiclantern (http://webchat.freenode.net/?channels=magiclantern)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 06, 2017, 11:51:29 PM
Another case of bad setting flashed somehow into ROM. Reproduced the error in QEMU. Appears easy to fix, since these settings fortunately have backups (wear leveling?). I've already unbricked a similar case a while ago (also 5D3), and I've already booted the GUI in QEMU on the affected ROM.

Two failures on the same build are a bit too much to be a coincidence. For now, my advice would be to avoid the Apr04 build, even though some reports say it's faster (I'm willing to bet it's not, but I'll benchmark it, just in case).

Apr06 should be safe regarding this issue (see my tests) (http://www.magiclantern.fm/forum/index.php?topic=19300.msg182552#msg182552).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kharak on April 07, 2017, 12:09:15 AM
Thanks for the heads up!

 I'll update to Apr06.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 07, 2017, 02:00:12 AM
@a1ex Thank you for all your hard work.. I tried the April 4th build but did not encounter such problems on 1.1.3.  Will give April 6th a test drive tonight I'll post my findings. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Hans_Punk on April 07, 2017, 05:40:11 AM
Amazing work @a1ex as always!

I'm using MLV_Dump for the newest crop_rec module with higher resolutions (4K, 1080p48 etc) build - And I'm getting vertical stripes on the .dng exports.
Is this normal for now?...or am I missing something simple? I have not used MLV_Dump before now (and I'm a code ignorant idiot)
Test shots were lower ISO 100-400, various resolutions tested from April 4th build - all exhibited some vertical stripes (some more noticeable at lower ISO)

Running this code on Win 7 works for me to export the .dngs's, I just don't know if vertical stripe correction is an option yet in this version of MLV_Dump...or I simply need to somehow amend the .cmd code to 'switch on' the correction in MLV_Dump?

@echo off
md "%~n1"
"Q:\NEW MLV\mlv_dump.exe" -o %~n1\%~n1_ --dng %~n1.MLV

Link to screengrab of most prominent vertical stripes:
https://www.dropbox.com/s/md2inmxhaj604gv/M06-1750%20%2800123%29.jpg?dl=0 (https://www.dropbox.com/s/md2inmxhaj604gv/M06-1750%20%2800123%29.jpg?dl=0)

.Dng Sample files:
https://www.dropbox.com/sh/fe5nve001l2vqx5/AADpbW7eVOqafc7XP4XnhgD0a?dl=0 (https://www.dropbox.com/sh/fe5nve001l2vqx5/AADpbW7eVOqafc7XP4XnhgD0a?dl=0)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 07, 2017, 05:54:32 AM
Upload a sample file?
Shortened file:
Code: [Select]
mlv_dump -f 5 -o Output.MLV Input.MLVInput.MLV being your file with stripes. Do the command correct or you might loose the file.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: garry23 on April 07, 2017, 08:38:42 AM
@A1ex

I thought I would carry out a few experiments in photo mode looking at 'lowish' shutter speeds with the FRLV.

I decided to use an FPS twice the selected shutter speed.

It appears I can push the FPS down to 0.5 (ML says its 0.554) before it plateaus out.

I took the following combinations and all the DNGs looked OK in LR:

FPS = 2, ML:shutter = 1/10s
1, 1/4
1, 1/2
0.5(0.554), 1s

From my perspective this looks a healthy overlap with the 'normal' FRSP, which could use down to, say, 1/4s.

Cheers

Garry
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 07, 2017, 08:49:37 AM
From my perspective this looks a healthy overlap with the 'normal' FRSP, which could use down to, say, 1/4s.

Sounds reasonable - as the full-res implementation can be updated decide whether to use LiveView or the "long exposure" version, depending on the shutter speed selected in photo mode.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: garry23 on April 07, 2017, 08:53:20 AM
@A1ex

Yes: I've worked out how to do it in my Lua bracketing scripts and I will confirm things as soon as I can. ie when I can trigger with the half shutter key press.

This should give photographers a 30s to 'ultra fast' seamless capture capability.

 :)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 07, 2017, 09:50:40 AM
Quote
Two failures on the same build are a bit too much to be a coincidence. For now, my advice would be to avoid the Apr04 build, even though some reports say it's faster (I'm willing to bet it's not, but I'll benchmark it, just in case).

Ok, back to april6th...

192050p3x3, 5D3 113, ISO 100: 10 recordings 1min/each (camera didn't stop recording, I stoped recording assuming continuous).
192050p3x3, 5D3 113, ISO 400: 10 recordings 1min/each (camera didn't stop recording, I stoped recording assuming continuous).
192050p3x3, 5D3 113, ISO 800: 1 recording, camera stops recording after 10sec. (message:"didn't make it to stimated record time")
I turn off camera.
When I turn on camera, a full screen message appears like this one:
Quote
When I boot camera again, I get this message full screen:
ML ASSERT: 0 at mlv_lite.c:2006 (compress_task), task compress_task vv:1 mode:3

srm_malloc_suite(1)...
srm buffer #1: 48600064
srm_malloc_suite(f1408)
Resolution changed: 2080x1587 -> 2080x961
Black level: 2047

Compressed size higher than uncompressed - corrupted frame?
Please reboot, then decrease vertical resolution in crop_rec menu.

ML ASSERT: 0 at mlv_lite.c:2006 (compress_task), task compress_task vv:1 mode:3

So ISO was the culprit for not get continuous recording with april6th?... My fault. :-[

No STACK OVERFLOW error for now...

Edit: in this tests I got the same corrupted image that before when panning (top part of frame with motion cutted).
Can I ask if is easy to adress this, or not? That would make the recordings usable...!!!! (at least fro me).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: squig on April 07, 2017, 10:14:21 AM
Apr06 3104x1298 1600 ISO 93.4MB/s ~57%
5.26Gb
5.78Gb
Restarted
8.15Gb
25.06Gb (I got bored and stopped recording)

I was going do a comparison with Apr04 but the stack error appeared (for the 1st time) when I loaded the modules so I chickened out.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 07, 2017, 10:27:32 AM
Edit: in this tests I got the same corrupted image that before when panning (top part of frame with motion cutted).
Can I ask if is easy to adress this, or not? That would make the recordings usable...!!!! (at least fro me).

Here's a boring task: what's the number of pixels before the cut? (min/max/median would be great)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 07, 2017, 11:03:18 AM
Quote
Here's a boring task: what's the number of pixels before the cut? (min/max/median would be great)

 :-[ Sorry for that!!!... Its only thing that makes image unusuable.

I've  measured px and the pattern is very strange... min distance I've found is 7px, max is 125px, but it doesn't follow a logic. At first it seems increassing with the panning velocity, but looking close, during panning, one frame have 98px (from top to cut) the next 112px, the next 18px, one frame (in between all of this) is correct, and for example, the next have 73px...

Min I found: 7px
Max I found: 125px
Average from random 17dng:83,8px
@A1ex
Is this what you are asking for?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 07, 2017, 11:33:13 AM
Yes. I thought I'd get away with it by increasing the optical bar size, but it looks like I should re-enable double-buffering (at least for certain video modes).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 07, 2017, 11:35:27 AM
If you need to do some "mechanical-tedious" work, if you explain me how to do it, I can do it...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: BBA on April 07, 2017, 11:54:09 AM
@A1ex
Quote
Could not reproduce
A good thing for other users : if you agree, I will edit my #155 post and remove the unnecessary warning about not switching on a camera without CF card


Sensor cleaning issue:

Maybe something interesting in the Debug menu: red dot in front of "memory patches"

https://www.dropbox.com/s/ap2zant3kn9vy2e/VRAM3.PPM?dl=0
 (https://www.dropbox.com/s/ap2zant3kn9vy2e/VRAM3.PPM?dl=0)

memory patches, with a red dot:

https://www.dropbox.com/s/cafc5h767gbqgdv/VRAM2.PPM?dl=0
 (https://www.dropbox.com/s/cafc5h767gbqgdv/VRAM2.PPM?dl=0)

Could it be of any help for fixing the cleaning sensor issue of my camera ?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 07, 2017, 12:09:36 PM
Maybe something interesting in the Debug menu: red dot in front of "memory patches"

That's unrelated; it's simply telling you about some module (here, crop_rec) using ROM patches aka cache hacks (which are a bit fragile in the backend).

You can use this build to get a startup log with all the debug messages from Canon. Before the log gets saved, you should try to perform the sensor cleaning, so messages about it will get included:

https://builds.magiclantern.fm/jenkins/view/Experiments/job/startup-log/
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: BBA on April 07, 2017, 12:36:15 PM
@A1ex
Thanks a lot !
I will give it a try and report.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: GutterPump on April 07, 2017, 12:37:47 PM
And I'm getting vertical stripes on the .dng exports.

I also noticed this issue with the 14 bits lossless feature.

Even  if i change the code in the raw2dng.c file :

Code: [Select]
      if (stripes_correction_needed)
    {
        printf("\n\nVertical stripes correction:\n");
        for (j = 0; j < 8; j++)
        {
            if (stripes_coeffs[j])
                printf("  %.5f", (double)stripes_coeffs[j] / FIXP_ONE);
            else
                printf("    1  ");
        }
        printf("\n");
    }
}

to

Code: [Select]
     printf("\n\nVertical stripes correction:\n");
        for (j = 0; j < 8; j++)
        {
            if (stripes_coeffs[j])
                printf("  %.5f", (double)stripes_coeffs[j] / FIXP_ONE);
            else
                printf("    1  ");
        }
        printf("\n");
    }

For constant vertical stripe fixs with mlv_dump. (Thanks to Danne for this tip)
The fix don't appear in the command line.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 07, 2017, 12:50:46 PM
To decode, you will need a custom mlv_dump (also included). It works by simply dumping the lossless JPEG payload to DNG (that means, processing options such as bad pixel or vertical stripe fixes are not working yet).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: GutterPump on April 07, 2017, 01:16:42 PM
Sorry a1ex didnt saw this message ! Thanks for the remind.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Hans_Punk on April 07, 2017, 01:27:33 PM
Ah yes...my fault for not acknowledging that info. I should have noticed the custom mlv_dump disclaimer.

Cheers A1ex / Danne & GutterPump for reply x
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: BBA on April 07, 2017, 03:30:40 PM
@A1ex
I don't know how to save the debug msgs after a "manually started" cleaning process:
Quote
Before the log gets saved, you should try to perform the sensor cleaning, so messages about it will get included:

When I switch the camera on, the "auto sensor cleaning" is set to "disabled";
The bottom red led is lit a while and then a few times (blocks of data written to the card ?)
It ended at 14:42
I went to the Canon SETUP3 menu to start a manual immediate cleaning
The camera made some noise as if taking 2 pictures then this greyed menu item became yellow again.
I then went to "auto cleaning" item to set it to "enabled"
I then went in the ML DEBUG menu and pressed SET button with cursor on DebugMsgLog
The bottom red led was lit a very little while (but I saw no file on the card created at that time = 14:46).
Finally I switched the camera off.

The following file is generated just when the camera gets started (gets the creation time = 14:42):

https://www.dropbox.com/s/mjickmertwial3y/DM-0001.LOG?dl=0 (https://www.dropbox.com/s/mjickmertwial3y/DM-0001.LOG?dl=0)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 07, 2017, 04:10:37 PM
Diagnosed the bricking issue - it was caused by a null pointer bug in ML :(

Only Apr04 is affected - both older and newer builds are safe.

Fortunately, it's easy to recover (at least in the two previous cases). One uint32_t at an arbitrary address (between 0 and around 60MB, with a non-uniform probability distribution) might be overwritten with 0xA5A5A5A5 (and eventually end up saved into ROM at camera shutdown, when Canon code saves camera settings).

@BBA: if you believe your camera might be affected by this bug, please send me a recent (bad) copy of your ROM, alongside with an older (good) copy.

If you don't have any backup, copy the oldest files from ML/LOGS you can find on a ML card (hopefully these are "last known good" configuration).

To get a fresh copy, simply delete ML/LOGS from a ML card, and it will be re-created at next startup.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: GutterPump on April 07, 2017, 05:42:39 PM
I went to the Canon SETUP3 menu to start a manual immediate cleaning
The camera made some noise as if taking 2 pictures then this greyed menu item became yellow again.

This is normal, it's the same for me and for all 5DIII users.

infos here (http://www.canonrumors.com/forum/index.php?topic=13188.0)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: BBA on April 07, 2017, 05:59:42 PM
@ Gutterpump
Thanks for your feed !



Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Mefist on April 07, 2017, 06:11:09 PM
I get a black DNG (((((

CMD the code is:
@echo off
for %%a in ( %* ) do ( md "%%~na" 2>nul )&( "c:\RAW\mlv_dump.exe" -o %%~na\%%~na_ --dng %%~na.MLV )

The code is wrong?

I have Windows 10 x64. I drag MLV to CMD, it immediately closes, creates a folder and it has black DNG

(https://s8.hostingkartinok.com/uploads/images/2017/04/7bfaabd168d09474f10bb311ced4c840.png)


Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 07, 2017, 06:44:03 PM
@A1ex I got the nasty error explosion today using the April 6 build.
I think it was probably because I was using an odd resolution
3008x1280 or something like that. Pulled battery everything seem fine with the camera.
Tried again with 3072x what ever 2.35:1 is and it seemed to work ok.
Would you like me to report the bug.
Here is a link to what was on my card. Error codes and all.
https://www.dropbox.com/sh/8lmopv8z7p2i91f/AAA4saYTmLE7rFQ3t7nSUBCaa?dl=0
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: GutterPump on April 07, 2017, 07:00:44 PM
I get a black DNG (((((


Can you share a small part of your MLV file ? Or record a very short part of movie for get arround 10 frames.
Be sure you're using the good version of mlv_dump (https://builds.magiclantern.fm/jenkins/job/crop_rec_4k/7/artifact/modules/mlv_rec/mlv_dump.exe) too.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Mefist on April 07, 2017, 08:11:49 PM
That's what I used and settings.

(https://s8.hostingkartinok.com/uploads/images/2017/04/a0f13f057a711f0fe713aad9f9b6ffe9.jpg)(https://s8.hostingkartinok.com/uploads/images/2017/04/e8bd24d101d2c3c5dd3d648fdd1a3288.jpg)

(http://poster.radikal.ru/1704/2a/12acbdda0c84.jpg) (http://radikal.ru/video/eNEQy5t1t25)

Quote
@echo off
for %%a in ( %* ) do ( md "%%~na" 2>nul )&( "c:\RAW\mlv_dump.exe" -o %%~na\%%~na_ --dng %%~na.MLV )

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: beauchampy on April 07, 2017, 09:46:20 PM
Just testing April 6th build on 123 with HDMI.

First up, it's a bit sketchy getting signal from the HDMI - have to power on, go into Canon menu, back out again, screen flashes all kinds of weird, live view then appears. First time I tried the camera crashed.

Second up, only getting 7 seconds at 1920x800 60p, GD off, 14-bit lossless, ISO 100. Data rate is low - only about 57mb/s

Just pulling out the HDMI getting 15 seconds. Data rate is more like 67mb/s.

Strange how plugging in the HDMI seems to lower the data rate by 10mb/s?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 07, 2017, 10:19:14 PM
Successfully unbricked the two cameras that had problems earlier in this thread! (ju64 and filipe), and also solved BBA's issue.

More cameras might be affected (actually up to 817); will add an automatic check to identify them in the next build. The side effects range from nothing at all (overwriting an unused memory location) to soft-bricking (recoverable) and anything in-between (e.g. BBA's issue). The bug can overwrite any setting (aka property in Canon code), or the data structure itself. My own 5D3 was also affected, without any obvious side effects though (as I don't know where the affected setting is used in Canon code).

I'll probably stop adding features for a while and will look into ways for preventing this class of bugs.

Strange how plugging in the HDMI seems to lower the data rate by 10mb/s?

It's not strange at all, as the camera has to work 6 times as much (1920x1080 vs 720x480) to create the HDMI image (strictly talking about the LiveView buffer). Most of the work is done by the image processing chips (external to the main CPU) and the DMA (EDMAC), so most of this load is on the memory bus.

The "Frozen LV" (previously known as Hacked preview) works by stopping LiveView buffers from being updated (therefore freeing the memory bus and a bit of CPU time).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 07, 2017, 10:21:40 PM
@beauchampy

Quote
First up, it's a bit sketchy getting signal from the HDMI - have to power on, go into Canon menu, back out again, screen flashes all kinds of weird, live view then appears

Same for me, but the same thing happens to me with older (and more stable) builds... To get hdmi signal I often go to play button and back out again. In my case using a Zacuto EVF.

I get continuous recording with april6th build with ISO100 1920x960 50p GD off 14bit losseless with or without hdmi in 5Dmk3 113.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: beauchampy on April 07, 2017, 10:32:06 PM
@beauchampy

Same for me, but the same thing happens to me with older (and more stable) builds... To get hdmi signal I often go to play button and back out again. In my case using a Zacuto EVF.

I get continuous recording with april6th build with ISO100 1920x960 50p GD off 14bit losseless with or without hdmi in 5Dmk3 113.

I didn't have much problem with previous builds. Might be because right now I'm using a Leica prime with no electronics, so that might be throwing things off a bit (camera probably thinking a lens not attached).

Just tried 50p and yes, getting continuous at a steady 79MB/s @ ISO 100, 14-bit lossless with HDMI, GD off.

Bit weird how the 60p mode seems to have a much lower MB/s (57MB/s with HDMI, 67MB/s without), so I'm guessing its something a little more complex than just write speed which is the issue there. a1ex / devs  if I can help test in any way, please do let me know.

Off to Bermuda tomorrow, so I'll do some more testing of this mode there. Thanks for the suggestion :)

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 07, 2017, 10:39:28 PM
Fantastic!!!

Be aware with ISO. For me 800 is where problems start. I can get only few seconds with 800 ISO. And as I report in post #148, I get a "cutted" image in the top of frame with paning movements... So no usable for serious work...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 07, 2017, 10:41:20 PM
@a1ex I've used the April 4th build so does that mean my 5D Mark III is affected as well?  I remember when using it red numbers were flashing on the top of my screen about memory being low.   
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Mefist on April 07, 2017, 10:47:17 PM
I still do not get a good DNG. If you shoot as before then the DNG is good.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: reddeercity on April 07, 2017, 11:06:45 PM
I still do not get a good DNG. If you shoot as before then the DNG is good.
Are the frames black in any of the App's you use ? e.g. Resolve , A.E. ACR  ? Or what ever you use
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 07, 2017, 11:38:08 PM
@A1ex is it only 5Diiis that used April 4 build or could it be in the April 3 build too?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 07, 2017, 11:59:13 PM
Here's a Python script to check for sure (run it and follow the instructions):

Code: [Select]
# checks whether your camera might have been affected
# by a null pointer bug from the 2017Apr04 ML build.
# updated Apr10 2017 (ignore some false warnings)

from __future__ import print_function
import os, sys
from struct import unpack

def getLongLE(d, a):
   return unpack('<L',d[a:a+4])[0]

try:
    ROM = open(sys.argv[1], "rb").read()
except IndexError:
    print("usage:")
    print("1) Backup the contents of ML/LOGS from your card")
    print("2) Debug -> Dump ROM and RAM (to get a new ROM copy)")
    print("3) Run this command on the latest copy of your ROM:")
    print("   python %s ROM1.BIN" % sys.argv[0])
    raise SystemExit

# scan for the magic number 0xA5A5A5A5 that might have been
# written into ROM as a result of the null pointer bug
found = 0
ignored = 0
can_ignore = False
ignore_msg = ""
gaonisoy = False
for a in range(0, len(ROM)-4, 4):
    v = getLongLE(ROM, a)
    n = getLongLE(ROM, a+4)
    block_name = ""
    can_ignore = False
    if a >= 0xF40000 and a < 0xF80000:      # RING - that's where most errors were
        a0 = a & ~0xFFF                     # header of this block
        if getLongLE(ROM, a0) == 0xFFFF:    # active block? only check this one
            block_name = "active RING"
            if a == a0:
                print("Active RING block: %x-%x" % (a0, a0+0xFFF))
        else:
            block_name = "inactive RING"
            can_ignore = True               # ignore inactive RING blocks
    if a >= 0xF80000 and a < 0xFA0000:      # last debug log - ignore
        block_name = "last debug log"
        can_ignore = True
    if v == 0xA5A5A5A5 and n != 0xE5E5E5E5:
        print(hex(a), hex(v), block_name)
        found   += (0 if can_ignore else 1)
        ignored += (1 if can_ignore else 0)
    if v == 0x6E6F6167 and n == 0x796F7369:
        assert a == 0xC0004
        gaonisoy = True

print("\nErrors found: %d, ignored: %d." % (found, ignored))

if not gaonisoy:
    print("Doesn't look like a valid ROM1 dump.")
elif found > 0:
    print("Your camera might have been affected by the Apr04 null pointer bug.")
    print("Please contact Magic Lantern developers on IRC (#magiclantern on freenode)")
    print("to take a closer look.")
else:
    print("Your camera appears OK, no need to worry.")
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: etx on April 08, 2017, 03:37:29 AM
Great work to all involved! Amazing!

I'm testing this out on my 5D Mk III on 1.2.3. I'm really missing the "centered 5x zoom" option from the older crop_rec module. I was previously running crop3x.2017Jan13.5D3123 version and using the centered 5x zoom was killer for avoiding the warping from the corners of the frame.

Am I missing something, or is this something that I just have to wait on future release?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 08, 2017, 04:18:37 AM
@a1ex What if you don't know how to run that script.. Can you point me to a detailed instruction on how to run that code, would be greatly appreciated. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 08, 2017, 07:45:36 AM
Did various tests.
My interest was focused on 3K, UHD and 4K trying to reach peak performance.
Global Draw OFF
1:1 mode
(3:3 always records 30FPS)
I concluded that 14bit lossless performs better.
I also concluded that setting Canon Video to 1280x720 gives shorter vertical size and leaves room for 4K recording.
Changing Aspect Ration allows more recording time
Havent managed to playback my recordings on the camera yet, I get an innocent message.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: janvkem on April 08, 2017, 07:48:25 AM
@a1ex What if you don't know how to run that script.. Can you point me to a detailed instruction on how to run that code, would be greatly appreciated. 

This is what I just did to test it (for Windows):
1. Install python for your computer: https://www.python.org/downloads/ (https://www.python.org/downloads/)
2. Make sure to check add python x.x to PATH
3. Make a backup of your current camera's ROM (using magic lantern)
3. When done make a new folder and copy your ROM1.bin to that folder (i made E:/tmp)
4. In the same folder make a new .txt file and rename it to testrom.py --> make sure you have file extensions visible
5. Open the file in your favorite text editor (i like notepad++) and copy the code in a1ex's post
6. Open a command window in the folder you just created (press right click on the folder while holding shift key, select Open command window here and type this:

    py testrom.py ROM1.bin

7. Read and follow the output on your screen.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hindra on April 08, 2017, 08:37:30 AM
This is what I just did to test it (for Windows):
1. Install python for your computer: https://www.python.org/downloads/ (https://www.python.org/downloads/)
2. Make sure to check add python x.x to PATH
3. Make a backup of your current camera's ROM (using magic lantern)
3. When done make a new folder and copy your ROM1.bin to that folder (i made E:/tmp)
4. In the same folder make a new .txt file and rename it to testrom.py --> make sure you have file extensions visible
5. Open the file in your favorite text editor (i like notepad++) and copy the code in a1ex's post
6. Open a command window in the folder you just created (press right click on the folder while holding shift key, select Open command window here and type this:

    py testrom.py ROM1.bin

7. Read and follow the output on your screen.

Worked great, thank you. My camera appears to be ok :)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: garry23 on April 08, 2017, 09:14:32 AM
Having never done it before: how long should a ML ROM dump take?

Update: I have 'Saving 4f000000...' on the ML screen for 15mins.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 08, 2017, 09:18:50 AM
Although I had doubts in the beginning, after draining a couple of batteries, I am sure now that current build (6April) is power hungry
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hindra on April 08, 2017, 09:20:25 AM
Having never done it before: how long should a ML ROM dump take?

Near instant? I believe when you "update" firmware with ML it will backup the ROM then. I just formatted my SD card in PC, format in camera, load ML from PC onto card, update firmware to setup ML, grab the ROM1.BIN and follow instructions.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: garry23 on April 08, 2017, 09:24:50 AM
@hindra

Thanks: I'll stop the current dump and try again.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: garry23 on April 08, 2017, 09:30:11 AM
When I looked at the card, I had a copy of ROM1.bin, however, the ML screen kept the 'Saving 4f000000...' on the ML screen for 15mins, until I stopped the Cam.

Run the test: all looks ok  :)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 08, 2017, 09:34:01 AM
Sounds encouraging; I'll integrate this test in the next ML build, but I'm a bit busy this weekend.

(that also means you will need a bit of patience on IRC - I'm not online this weekend)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: vstrglv on April 08, 2017, 10:30:48 AM
I have got this:
C:\Users\XXX\AppData\Local\Programs\Python\Python36>python testrom.py ROM1.bi
n
0xf5c33c 0xa5a5a5a5
0xf5cf0c 0xa5a5a5a5
0xf5d33c 0xa5a5a5a5
0xf5df0c 0xa5a5a5a5
0xf5e33c 0xa5a5a5a5
0xf5ef0c 0xa5a5a5a5
0xf5f33c 0xa5a5a5a5
0xf5ff0c 0xa5a5a5a5
0xf6033c 0xa5a5a5a5
0xf60f0c 0xa5a5a5a5
0xf6133c 0xa5a5a5a5
0xf61f0c 0xa5a5a5a5
0xf6233c 0xa5a5a5a5
0xf62f0c 0xa5a5a5a5
0xf6333c 0xa5a5a5a5
0xf63f0c 0xa5a5a5a5
Your camera might have been affected by the Apr04 null pointer bug.
Please contact Magic Lantern developers on IRC (#magiclantern on freenode)
to take a closer look.
I have good ROM1 from magiclantern-crop3x.2017Mar21.5D3113.
Already on IRC
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pocketrubbish on April 08, 2017, 10:40:34 AM
I just put together a quick video demonstrating the new options .

https://youtu.be/9trjJKDdLO0

Pushed the saturation a bit during the grade because I wanted to see how it holds up. I'm impressed.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 08, 2017, 10:53:24 AM
Quote
Your camera appears OK, no need to worry.
Happy camper :)

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 08, 2017, 11:00:00 AM
Happy camper :)

Nice save, @Danne!  :) (will have to check mine first thing in AM after I get back home from work) -- good thing I only used the April 4th build on the 'B' cam that I just scooped up from Craigslist earlier this week and not my original baby, Ha!

Pushed the saturation a bit during the grade because I wanted to see how it holds up. I'm impressed.

Indeed, extremely impressive. What were the highest ISO's you were able to use for the night scenes @pocketrubbish?

My apologies for lack of paying attention to details in your clip as I was obviously all eyes on the static shots. Similar effect if one were to stare at a Television showing nothing but static and it just sucks you right in which means you got the job done well even if it wasnt intended for.

The reason why I asked (hope you're reading this @a1ex) is because I'm an avid fan of using DarkFraming average processes for high ISO shots and ever since this whole lossless compression 4k crop_rec stuff came out for the 5D3 -- haven't been able to figure out how or why when I apply DF avg process it just don't correspond properly and not sure if this is related to the modified mlv_dump spitting out different headers for each DNG or some sort?

I think because of this it's making it a bit trickier for the DF avg process to work properly (at least in @Danne's cr2hdr.app is what I use) so not sure if this at all possible due to the limitation factor of this new lossless compression codec in MLV 14-bit lossless or perhaps just a simply fix in the code would do the trick?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: vstrglv on April 08, 2017, 11:00:34 AM
This is what I just did to test it (for Windows):

    py testrom.py ROM1.bin
"python testrom.py ROM1.bin" works for me
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 08, 2017, 11:22:44 AM
Not sure if this would help anybody but I put in a little simplifying doubleclick command and a howto for mac users here.
https://bitbucket.org/Dannephoto/magic-lantern/downloads/ROM1test.zip

Basically the same for mac as for windows if running straight on the command line.

"python testrom.py ROM1.bin"
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 08, 2017, 11:23:09 AM
Your camera appears OK, no need to worry.

 :) :) :)

Thanks Danne
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: GutterPump on April 08, 2017, 11:49:34 AM
I'm feel quiet  8) :

(http://image.noelshack.com/fichiers/2017/14/1491644829-py.jpg)

Thanks to @a1ex for the script and thanks to @janvkem for explanations.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: RTI on April 08, 2017, 12:17:27 PM
Thx for the mini guide. "Your camera appears OK, no need to worry."
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: g3gg0 on April 08, 2017, 12:24:43 PM
@alex:
i guess its a good thing to add this checker to the module itself.
so all users get informed who potentially don't realize there are broken properties.
e.g. not crashing due to the property being bad, but weird image metadata or correction curves.

would warn them on display, telling them that the old version had a bug and this camera
seems to have a property being flipped.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Mefist on April 08, 2017, 01:14:45 PM
I just put together a quick video demonstrating the new options .

https://youtu.be/9trjJKDdLO0

Pushed the saturation a bit during the grade because I wanted to see how it holds up. I'm impressed.

How did you manage to shoot in 3072x1308 24fps at 40 seconds? Did not change the aspect ratio? I do not write 5dm3 in 3k 14bit-lossless 16:9 for more than 2 seconds
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 08, 2017, 01:29:28 PM
@A1ex Ran the Python test and my camera was not affected!!!!!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: ewinemiller on April 08, 2017, 01:52:47 PM
How did you manage to shoot in 3072x1308 24fps at 40 seconds? Did not change the aspect ratio? I do not write 5dm3 in 3k 14bit-lossless 16:9 for more than 2 seconds


16x9 would be more like 3072x1728. The best I've been able to do continuous is 3072x1536.

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Mefist on April 08, 2017, 03:29:49 PM
16x9 would be more like 3072x1728. The best I've been able to do continuous is 3072x1536.

I have 5dm3 firmware 1.1.3 CF SanDisk 64GB 160Mb/s
My settings: 3072x1308 with the aspect ratio of 2.35:1, Crop mode 3K 1:1, 14bit-lossless. If I put a resolution higher, the recording automatically stops.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 08, 2017, 03:42:57 PM
I did several 4K recordings with 14bit lossless. However I noticed some fat blotchy areas of compression, easily detectable.
I was shooting with 50mm 1.8 and compression was applied naturally in out of focus areas.
I thought 14bit lossless was visually the same as the ordinary 14bit.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: g3gg0 on April 08, 2017, 03:58:32 PM
it definitely is the same. compression is lossless.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 08, 2017, 04:15:52 PM
I was using 4K 1:1
Selection PAL/NTSC mode or 1080/720p through Canon Menu, changing Aspect Ratio, was giving me some more resources for more FPS.
Then I switched to 1920 50/60p, 1920x1080.
I pressed REC and got a warning "EDMAC timeout"
Brick camera
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Lars Steenhoff on April 08, 2017, 04:56:22 PM
Did you take out the battery before you concluded the camera is bricked?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 08, 2017, 06:56:59 PM
Not sure if this would help anybody but I put in a little simplifying doubleclick command and a howto for mac users here.
https://bitbucket.org/Dannephoto/magic-lantern/downloads/ROM1test.zip

Downloaded & ran the ROM1test (Thanks for that @Danne) and I was even more surprised by the result because I HAD USED the 2017-04-04 build.

Hmm... what gives?

(https://c1.staticflickr.com/3/2848/33914393675_a8a4fd7d52.jpg) (https://flic.kr/p/TEU8Fr)

By any chance could it be possible for some of us to have used this particular build and NOT be affected by the null pointer bug at all @a1ex?  :P

Also felt this was worth sharing after running the 2017-04-06 experimental build for 5D3.123 in case they give out hints or anything.

(https://c1.staticflickr.com/3/2894/33531220290_0eab0f61d3.jpg) (https://flic.kr/p/T63gzL) (https://c1.staticflickr.com/4/3684/33786501351_b00a979cb1.jpg) (https://flic.kr/p/TtADKn)

Some small bug I also ran into upon when selecting 45 in FPS override (w 1080p45/48 3x3 selected within crop_rec.mo) which then freezes the LiveView more than half of the time.

(https://c1.staticflickr.com/3/2810/33786952001_ec813f47b7.jpg) (https://flic.kr/p/TtCXHc)

After each battery pull I had to immediately go back into FPS override and either put it back into 48p or turn it off to keep LV from freezing up again. This is reproducible.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Vegandelight on April 08, 2017, 06:58:07 PM
This might be a stupid question but should MLVFS work with the compressed 14bit files? im just getting pinkish static when trying to view them.

Tried the script on my camera and everything seems fine, i missed the April 4th update, tried all of the other builds ;)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 08, 2017, 07:14:21 PM
@ Lars Steenhoff

If the camera doesnt respond, on/off and I need to take off the battery, is considered as brick ?
Sorry if I didnt specify clearly :D
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 08, 2017, 09:22:06 PM
@Quentin: ROM backup done, right? If not, save the contents of ML/LOGS from the oldest ML card.

Then, get a fresh copy of your (bad) ROM with the portable ROM dumper (https://www.magiclantern.fm/forum/index.php?topic=16534.0) and contact me on IRC (online for the next 2-3 hours, but busy, then tomorrow at about the same time).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 08, 2017, 10:01:53 PM
@janvkem @Danne @a1ex Thank you guy's ran the test my camera seems to be Ok  :D
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: vstrglv on April 08, 2017, 11:21:08 PM
@Quentin: ROM backup done, right? If not, save the contents of ML/LOGS from the oldest ML card.

Then, get a fresh copy of your (bad) ROM with the portable ROM dumper (https://www.magiclantern.fm/forum/index.php?topic=16534.0) and contact me on IRC (online for the next 2-3 hours, but busy, then tomorrow at about the same time).
a1ex, how to connect to you in IRC#magiclantern? I login in IRC#magiclantern with my nickname, but there is no yours nickname in the nickname list.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Vegandelight on April 09, 2017, 12:27:49 AM
(http://i.imgur.com/J7DfURv.png)
1920x960 50fps 3x3 pixel binning gives me static on all footage. Am i doing something wrong here?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: dmilligan on April 09, 2017, 01:36:53 AM
Quote
To decode, you will need a custom mlv_dump (also included). It works by simply dumping the lossless JPEG payload to DNG (that means, processing options such as bad pixel or vertical stripe fixes are not working yet).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: squig on April 09, 2017, 03:56:57 AM
Am i doing something wrong here?

Yeah, you're using MLVFS, it hasn't been updated for these experimental builds. Use this: http://www.magiclantern.fm/forum/index.php?topic=15108.0 or mlv_dump.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 09, 2017, 11:06:20 AM
I wanted to record h.264, my daughter with her new bicycle. No RAW, no anything.
Crop Mode and Raw Recording werent loaded.
The camera was recovered after a bad crash and removed the battery.
I never reset to ML defaults, or reload the modules as the usual procedure.

I shot 4 takes.
The two were MLVs.
I havent touched anything.
RAW recording enabled itself somehow
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: khamees1975 on April 09, 2017, 01:46:37 PM
Hi Alex and every one
i am so sorry but my english very bad !!!
how i can upgrade my camera ?
is't free or i have to pay or what i am new here in this forum i don't get every thing ????
thank you
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Vegandelight on April 09, 2017, 01:55:35 PM
Thanks for the response. Finally got a set of DNG's, wish they worked directly in Premiere.

Keep up the good work.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: jankrueck on April 09, 2017, 04:27:47 PM
(https://c1.staticflickr.com/3/2894/33531220290_0eab0f61d3.jpg) (https://flic.kr/p/T63gzL) (https://c1.staticflickr.com/4/3684/33786501351_b00a979cb1.jpg) (https://flic.kr/p/TtADKn)

hey @a1ex
is this any indicator for a bricked camera?
because this shows up to me, even though the script says, everthing is ok.

I have some troubles running those scripts.
not sure if I do it right. (I just take ROM1.bin from sd card and scan it via script.)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 09, 2017, 04:30:06 PM
I wanted to record h.264, my daughter with her new bicycle. No RAW, no anything.
Crop Mode and Raw Recording werent loaded.
The camera was recovered after a bad crash and removed the battery.
I never reset to ML defaults, or reload the modules as the usual procedure.

I shot 4 takes.
The two were MLVs.
I havent touched anything.
RAW recording enabled itself somehow

Was ML still in the camera?
After you pull the battery​, on the next power up the modules won't load. If you power down then power back up again, the modules will load again with your settings and all. Could this be what happened?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 09, 2017, 06:51:39 PM
After removing the battery, its was an up normal shutdown.
I havent touched anything.
Next day I turn on the camera to shoot my daughter.
I saw that the modules didnt load. Τhe setting werent reset.
I shot h.264, (I thought)
When I load the CF card into the computer, I was surprised.
There were 4 videos, two MOV and two MLV.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 09, 2017, 07:18:46 PM
@quentin  Do you remember what the record symbol looked like for any of the clips? Was it the Canon rec icon or did the cam beep then record like when it does for RAW. Also did you try processing the MLVs ? And did you test your Rom with the Python script?
You could also try recording again. Making sure that the mods are disabled for sure.
Remember that this build is bleeding edge.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 09, 2017, 07:28:40 PM
The recording icon was normal, h.264 and MLV however I didnt pay attention to the change.
I processed the MLVs, they were normal.
Nope, unfortunately I have no clue about python scripts, ROM processing etc.

I just checked the camera now.
Modules restored as the crash never happened.
Definitely I didnt reset to defaults and didnt select which modules to load.
I am familiar with this first-aid procedure and did it many many times.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: GutterPump on April 09, 2017, 07:33:15 PM
I did a try too :

https://vimeo.com/212477560

3009 x 1280  + 14 bits lossless = continuous shooting.
Just don't shot in ETTR, keep an eye on the compression percentage and check the histogram and you're done.

I used the new amazing ew ML Cinelog-C to Ektar 100 LANDSCAPE lut of hyalinejim (http://www.magiclantern.fm/forum/index.php?topic=19338.msg182770;topicseen#msg182770)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: aschille84 on April 09, 2017, 08:30:40 PM
I did a try too :

@gutterpump you make art out of a camera test. Beautiful!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: GutterPump on April 09, 2017, 09:32:47 PM
@aschille84 glad to know you like it !
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Greg on April 09, 2017, 10:50:34 PM
If anyone is interested I think we have everything to do it for 500D (Full res LV others will be useless - too slow sensor).
3 years ago I increased the width, but I did not know how to increase the height.

I suppose these are the registers (my old notes) :
Code: [Select]
ADTG[2]1172:     0x3d5 (was 0x667)
ADTG[2]1173:     0x39b (was 0x6e6)
ADTG[2]1178:     0x3d5 (was 0x667)
ADTG[2]1179:     0x39b (was 0x6e6)
ADTG[2]1214:     0x3de (was 0x622)

So look at crop_rec.c - /* some modes require additional height adjustments */
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 09, 2017, 10:55:37 PM
I,m interested. Got a couple of 500D cams at work :) Could you prepare a build for testing?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Greg on April 09, 2017, 11:04:39 PM
Sorry it will not be possible.
It's just a hint for someone like reddeercity but with 500D.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 09, 2017, 11:06:51 PM
I see. Thanks. Short on time over here.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: reddeercity on April 10, 2017, 12:50:21 AM
It's just a hint for someone like reddeercity but with 500D.
If anyone is interested I think we have everything to do it for 500D (Full res LV others will be useless - too slow sensor).
3 years ago I increased the width, but I did not know how to increase the height.

I suppose these are the registers (my old notes) :
Code: [Select]
ADTG[2]1172:     0x3d5 (was 0x667)
ADTG[2]1173:     0x39b (was 0x6e6)
ADTG[2]1178:     0x3d5 (was 0x667)
ADTG[2]1179:     0x39b (was 0x6e6)
ADTG[2]1214:     0x3de (was 0x622)

So look at crop_rec.c - /* some modes require additional height adjustments */

Very useful , thanks  :D I was wonder on this , I was going to pm you if you have any notes that could help.
I still can't reproduce you test in the 500d , registers are different on 5d2 but close thou.
my problems is the timer A can't lower it lock to 610 , and ever time I try agjusting the Headtimer
I get a frozen camera/lockup (battery pull)  so if I can get timer A to lower I think I will have it
I have the width , but compressed to the left in a 2K crop window , anyways thanks it helps .
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Greg on April 10, 2017, 02:48:30 AM
I'm afraid there may be a problem with raw buffer. Something like 10/12bit raw video with old cameras.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Teamsleepkid on April 10, 2017, 03:17:20 AM
@greg think it will be possible on eos m? I know it won't do 4K or 3k but any resolution boost would be good.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: mk11174 on April 10, 2017, 05:03:25 AM
This is where I am at with 700D so far, just trying to figure out how to change height without it getting all scrambled, was able to change width past 4096 but was getting borders so brought it down to an even number, am able to go to almost 1500 on height until it scrambles, so I am assuming other registers need to be adjusted to make it work.

Also, fps drops to 6.5fps with the register changes to make it work, was able to force it to 9fps with C0F0[6014], any higher caused some sort of overlap on image.

Here are some examples of resolutions I would get by adjusting C0F0[6804]. It seems as I adjust the 3 on the left, the height would change, the 3 on the right, the width would change, the middle 0 seems it has to stay 0. I can go much higher on the height, but as I said, it gets crazy looking, shifty and predator vision like.
Code: [Select]
4550204   1920 x 1080
4540298   1928 x 1144
4e40448   4248 x 1217   
4cc0422   4096 x 1193
45b0422   4096 x 1080  current
50b0564   5384 x 1256

(https://s29.postimg.org/we3eb6yuv/4096x1080.jpg)

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: mk11174 on April 10, 2017, 06:45:27 AM
I managed to get height going by adjusting
Code: [Select]
6804  0xc3b0422     4096x3096
713c  0x4a7 to 0xca7
8172  0x6fd to 0x8fd
8178  0x6fd to 0x8fd

But I end up with this image, not sure how to fix this????
(https://s10.postimg.org/mbk2633ax/Untitled-1.jpg)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: dfort on April 10, 2017, 08:00:03 AM
I was thinking about looking into the EOSM and 700D. Since there are others out there that are willing to fry their sensors to get 4k working I posted adtg_gui builds for these cameras.

https://bitbucket.org/daniel_fort/magic-lantern/downloads/

@mk11174 -- Did you already get 14-bit lossless compression working?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 10, 2017, 11:12:36 AM
is this any indicator for a bricked camera?

A bricked camera is easy to identify - it does not boot (in other words, it's as useful as a brick). So far, there were two cases, both successfully recovered.

However, the Apr04 null pointer bug might (or might not) have changed some of your settings somehow. Since you have not noticed any side effect yet, your camera is probably OK, or you may find out some strange behavior when using some functionality that you don't use very often. That doesn't count as "bricked", but still, those (persistent) changes that appeared because of that bug are not desirable.

By any chance could it be possible for some of us to have used this particular build and NOT be affected by the null pointer bug at all @a1ex?  :P

Of course - what happened was not deterministic.

I've just posted a new build that checks whether your camera was affected by this bug. All you have to do is to upgrade - if no message pops up, your camera was not affected by that bug at all. Otherwise, just follow the instructions.

For this particular bug, the persistent changes are easy to identify, because the value that could get written into ROM is always the same - 0xA5A5A5A5.

Note that any other code (from ML and even from Canon) can cause similar issues (and probably much harder to identify). We have patched one such issue many years ago - on most camera models, a bug in Canon's startup code overwrites one single (predictable) memory address with 0 (4 bytes). You can check it by looking for ARMLIB_OVERFLOWING_BUFFER and tracing the patched address in QEMU (e.g. set a value at startup and watch it getting overwritten). In some camera models, this address ended up right in the middle of ML code (thus causing unpredictable behavior that changed with... the amount of code before the affected spot - e.g. insert a printf and watch things working in a totally different way).

This is a limitation of the current CPU and operating system - any task is allowed to write at any memory address. On top of that, Canon code saves their settings to ROM (by reflashing), so all it takes to soft-brick a camera is a programming mistake that results in writing at the wrong memory address.

I can think of a run-time check that detects any writes to Canon's property data structures (some of these are the persistent settings that get flashed into ROM) from tasks other than (Canon's) PropMgr, but it will definitely have a noticeable overhead (impact on performance). I can also think about adapting g3gg0's mem_prot module (which, back then when g3gg0 wrote it, was way above my level of understanding), but this won't catch badly configured DMA transfers. I can also imagine a routine that would check the integrity of Canon's property data structure right before they get saved into ROM (and maybe lock up the camera if problems are detected).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: jankrueck on April 10, 2017, 12:15:21 PM
nice work @a1ex. I tested new build and now message pops up.
I was/am curious about this error/bug saying cache is locked.

I also can't get 1920x1920 working, can't get any higher than 1728px.
Before the april6 I was able to record square footage.
Not sure if you changed something, or what happend.
(http://i.imgur.com/Q5dA739.jpg)


Also If I do 3k ISO400 I get a crash when there is to much detail moving.
(http://i.imgur.com/27HLwt7.jpg)
Code: [Select]
ML ASSERT:
0
at mlv_lite.c:2036 (compress_task), task compress_task
lv:1 mode:3


Magic Lantern version : crop_rec_4k.2017Apr10.5D3113
Mercurial changeset   : 2bbf0bbbaa74 (crop_rec_4k) tip
Built on 2017-04-10 08:54:19 UTC by [email protected]
Free Memory  : 164K + 3162K


hope my data is helping you in any way. if not, just tell me to stop it :D


cheers!

ps: Is there a chance to get some reference sound back in 3k mode, to have better syncing in post?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 10, 2017, 12:37:34 PM
See this post (http://www.magiclantern.fm/forum/index.php?topic=19300.msg182619#msg182619) for the red dot and the first post for resolution:

Quote
* 1920x1920 @ 24p (1:1 square crop) - continuous*)
...
Code: [Select]
                                /*   24p   25p   30p   50p   60p */
    [CROP_PRESET_3X_TALL]       = { 1920, 1728, 1536,  960,  800 }, /* 1920 */

ps: Is there a chance to get some reference sound back in 3k mode, to have better syncing in post?

I'll probably stop adding features for a while and will look into ways for preventing this class of bugs.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: jankrueck on April 10, 2017, 12:58:47 PM
uuuhm sorry for the timewaste.. I somwhow skipped page 8 completly.
It took me a while to understand the resolution"issue"
I probably switched to 25p whithout noticing this will affect something ;)


maybe I have to get a filmclap, till you find time to add soundrecording ;)



Cheers and thanks!!!!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: mk11174 on April 10, 2017, 02:33:10 PM
Quote
Posted by: dfort
@mk11174 -- Did you already get 14-bit lossless compression working?

I have only been working with resolution changes for fun just to see what happens and of course sharing what I find.  :)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Markus on April 10, 2017, 06:30:05 PM
Tried some of the new experimental modes on Fw 123 today. Looked really good but i could not get 1920x1080 45p continious on my Komputerbay 1066x I got around 10-20sek rec time. Has anyone achieved 45p continuous on FW 123? 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 10, 2017, 07:11:00 PM
Of course - what happened was not deterministic.

I've just posted a new build that checks whether your camera was affected by this bug. All you have to do is to upgrade - if no message pops up, your camera was not affected by that bug at all. Otherwise, just follow the instructions.

Fortunately, no messages prompted up after upgrade w 2017-04-10 build though I am curious to see what the instructions would be for the affected ones.

Thanks for double checking @a1ex!  ;)

Actually decided to try this (since it was reproducible last time) again w 1080p 45/48 3x3 under Crop mode while in 60p from Canon menu together with FPS override set to 45p and can now confirm that this bug has either probably gotten worse or I'm doing something incorrectly (which I doubt) but feel free to tell otherwise.

(https://c1.staticflickr.com/3/2934/33828393021_6e5e8b3f96.jpg) (https://flic.kr/p/TximFx)

Hence the reason why I had to use an iPhone to snap this screenshot because no matter how quick I can try getting back into ML menu upon restart (after each battery pull) and getting to scroll around (both Canon or ML menus) within 5 seconds more or less was impractical. It just freezes up the LV.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 10, 2017, 07:31:41 PM
Tried some of the new experimental modes on Fw 123 today. Looked really good but i could not get 1920x1080 45p continious on my Komputerbay 1066x I got around 10-20sek rec time. Has anyone achieved 45p continuous on FW 123?

How were you able to achieve 15-20 seconds without the LV display freezing up within 5 seconds more or less @Markus?

Meanwhile gonna try defaulting Canon settings to defaults and reload a fresh 2017-04-10 experimental build again just to double check.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 10, 2017, 07:41:09 PM
For now, FPS override only works in Low Light mode for reducing the frame rate (known bug).

For 1920x1080 at 45 fps, set 50 fps in Canon menu.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 10, 2017, 07:43:06 PM
@jankrueck I got the same thing Compression crash recording @ 3K running April 6th build.   
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 10, 2017, 07:51:09 PM
@A1ex Gonna try April 10th build today :D  Thank you once again for all your hard work..
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Markus on April 10, 2017, 08:03:01 PM
No lockups here. Only hard to get cont rec on alot of the sweetspot resolutions.
Some settings of mine:
Global draw on, canon in 720p all-I 50p mode, croprec 3x3 binning 1920x1080 47/48p @ 45p. Lossless 14bit.
Without global draw i could not get a bw preview in this mode.
Anything that sticks out?

The file size estimation got confused some times when switching between modes. It also got the wrong fps when switching. I went to photo mode and back as a workaround when this happend.
 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 10, 2017, 08:28:33 PM
For now, FPS override only works in Low Light mode for reducing the frame rate (known bug).

For 1920x1080 at 45 fps, set 50 fps in Canon menu.

That did the trick. Haven't use PAL in years (I'm on NTSC land). Should have thought of that to begin with and Thanks for the tips.

So far 1920x1080 @ 45p MLV-lossless 14-bit seems to be continuous even w HDMI out to an monitor (clean native feed :D) which is a PLUS for gimbal work because obviously the 5D3 will have the grey previews while mounted on the Ronin (which will go off from 'Turn off LCD' within 5 seconds during each recordings -- useful feature from 'Powersave in LiveView' under Prefs within ML menu) and can then focus on the external monitor instead with a peace of mind!  8)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 10, 2017, 08:46:16 PM
@a1ex Just tested April 10th Build so far so good.  One thing that I've noticed is that I can record at higher resolutions for longer now for example 3520x1498 - 2.35:1 I can get over 12 sec on a Lexar 128GB 1066x card before it would record for like 2 sec on the April 6th Build great work.  Also one thing that I've noticed is LiveView in 3K mode when recording @ 2.35:1 is all grayscale now, on the April 6th Build it showed both color and grayscale. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 10, 2017, 09:08:03 PM
Any differences between Apr10 and Apr06, except the null pointer check performed at camera startup, are your own imagination (review the changes, please).

Or, do a controlled test. Same static scene, same settings (!), run 10 test clips on both builds (alternating the builds), and then draw the conclusion.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 10, 2017, 09:21:49 PM
@a1ex Cool no problem.. @ camera startup I don't see any test running am I missing something, or it just runs in the background and if anything is wrong it flashes a message. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 10, 2017, 09:49:16 PM
Upgrade camera again via Canon menu and wait for either a pop up screen notifying instructions or none will be prompt (meaning it isn't affected) @goldenchild9to5 if you haven't already.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 10, 2017, 09:52:34 PM
So far, I've got 3 users with warnings about the null pointer bug (besides the 2+1 ones already solved over the weekend). Two of the new users had a false alarm (the modified value from ROM not affecting the functionality at all, being in unused areas), and the third user had his camera successfully patched.

Updated the tests (both python script and ML build) to ignore the false warnings I've encountered.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 10, 2017, 10:19:00 PM
Awesome!!!!! @a1ex!!!!!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 10, 2017, 10:34:07 PM
Tried some of the new experimental modes on Fw 123 today. Looked really good but i could not get 1920x1080 45p continious on my Komputerbay 1066x I got around 10-20sek rec time. Has anyone achieved 45p continuous on FW 123? 

Yes. (http://www.magiclantern.fm/forum/index.php?topic=19300.msg182320#msg182320)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: vstrglv on April 10, 2017, 10:54:45 PM
Updated the tests (both python script and ML build) to ignore the false warnings I've encountered.
A1ex, where can i copy updated python script?
 Many thanks.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 10, 2017, 11:09:59 PM
@DeafEyeJedi Upgraded ML, or installed ML from the canon menu a second time Didn't see no pop-up screen all it did was install ML again.  So it seems like my camera is ok than. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 11, 2017, 02:34:18 AM
WOW.. Getting incredible images @ 3K  :D
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 11, 2017, 04:36:02 AM
Using April 10th build. Testing 3072 x 1728
Got
ML ASSERT:
RAW_IS_IDLE
at mlv_lite.c:584 (measure_compression_ratio), task shoot_task
lv:1 mode:3


Magic Lantern version : crop_rec_4k.2017Apr10.5D3113
Mercurial changeset   : 11f405b62b31 (crop_rec_4k) tip
Built on 2017-04-10 19:48:55 UTC by [email protected]
Free Memory  : 163K + 3118K

Pulled Battery then restarted and reloaded mods.
The only things that I don't like are the current gray scale that it jumps to when it records (I know its purpose is to reserve resources)
The inconsistency of clip length due to changing compression.
And I find I cant get over 4 seconds on average with a 128 GB Lexar 1066x 160MB/s card.
I know this is very experimental I am just reporting my findings so far.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: csound on April 11, 2017, 05:13:39 AM
Ran ML April 10 4K test my 5D mk3 and is indicating the problem of null pointer bug from April 4 ML 4K experimental. Reverted to latest nightly 1.2.3 2017-3-30 build which seems to work fine. Casual user of ML & not sure if I need to do anything to address potential bug?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Markus on April 11, 2017, 10:50:48 AM
Quote
Tried some of the new experimental modes on Fw 123 today. Looked really good but i could not get 1920x1080 45p continious on my Komputerbay 1066x I got around 10-20sek rec time. Has anyone achieved 45p continuous on FW 123? 

I failed to recognize how noise at higher isos gives less effective compression. 1920 1:85 45p at  iso 800 seemes to be max cont rec for me.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: togg on April 11, 2017, 01:06:26 PM
I was wondering what this means, this feature will be merged into the silent pictures module? How easy will it be to trigger the shots?


Quote
* Full-resolution LiveView: 5796x3870 at 7.4 fps (128ms rolling shutter) - continuous*) at 5 FPS!

The last feature complements the well-known full-resolution silent pictures - the new implementation will be usable at fast shutter speeds, without the exposure gradient - but with rolling shutter (just like regular LiveView frames).


Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: beauchampy on April 11, 2017, 02:15:27 PM
Shot a card full of 50p / 960 / 3x3, 113 at the beach and unfortunately every clip has a tearing across the top of the frame :-(

This was processed in cr2hdr 4k and it is the April 06th build. I have live view set to real time if that makes a difference..

https://vimeo.com/212739433

Edit: I also kept getting error messages. Recording with the error log still on the screen resulted in the old 1.67x stretch slow motion clips.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 11, 2017, 03:21:43 PM
That footage is recorded unstretched right? So cr2hdr.app shouldn,t add the compensating destretching tag to the dng file. I could maybe put in a "not stretched footage" in the menu so it would leave the tag alone.
Maybe the info is already in the rawc metadata? Don,t think compressed_raw mlv_dump keep that code yet?
Can you upload asmall mlvv sample Beauchampy?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: beauchampy on April 11, 2017, 03:24:41 PM
That footage is recorded unstretched right? So cr2hdr.app shouldn,t add the compensating destretching tag to the dng file. I could maybe put in a "not stretched footage" in the menu so it would leave the tag alone.

Yes, it's unstretched 960 50p 3x3 binning.

Do you think this is cr2hdr rather than ML?

Edit; I also tried turning the preview to auto (switched to greyscale low fps) and the problem persisted.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 11, 2017, 03:26:57 PM
Definately tag added not ML. Could you upload a short sample? In latest cr2hdr.app you can create a sample package in mlv_dump menu. Selection (12).
Latest version also combines both compressed and non compressed versions of mlv_dump so no need for any 4k version now.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: beauchampy on April 11, 2017, 03:41:52 PM
Definately tag added not ML. Could you upload a short sample? In latest cr2hdr.app you can create a sample package in mlv_dump menu. Selection (12).
Latest version also combines both compressed and non compressed versions of mlv_dump so no need for any 4k version now.

Here's a problematic sample MLV (sorry, didn't have an option 12 in mlv dump settings on my cr2hdr?)

https://drive.google.com/open?id=0B-GY59oq1k7eM2xtVVhkZFRFUEk

Is this where you upload the latest builds for mac?
https://bitbucket.org/Dannephoto/cr2hdr/downloads/


EDIT: Tried looking at the same sample file in 'Footage' and the same tearing occurs.. Not sure it is cr2hdr? :-/
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 11, 2017, 03:53:03 PM
Tag is added by mlv_dump(ml-dng version, dng.c code from dmilligan).
Could you upload the zip package? Much smaller in size. I have no good internet connection.

Quote
Is this where you upload the latest builds for mac?
yes.
https://bitbucket.org/Dannephoto/cr2hdr/downloads/
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: beauchampy on April 11, 2017, 04:08:47 PM
Tag is added by mlv_dump(ml-dng version, dng.c code from dmilligan).
Could you upload the zip package? Much smaller in size. I have no good internet connection.
yes.
https://bitbucket.org/Dannephoto/cr2hdr/downloads/

Here you go, thanks for your help!

https://drive.google.com/open?id=0B-GY59oq1k7eb2tDNm9VcEJnTmc
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 11, 2017, 04:21:34 PM
Thanks. I take a dive into this later tonight.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: beauchampy on April 11, 2017, 04:27:00 PM
Thanks. I take a dive into this later tonight.

Thanks mate I really apprecite that. Just FYI I just tried 123 and the latest April 10th build and the issue persists.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 11, 2017, 04:29:10 PM
Ok, just took a quick look
Quote
...old 1.67x stretch slow motion
I took this as you were getting stretched footage? Anyway, the tearing issue is as you say something coming from the compress raw itself not from mlv_dump. About stretching issue it all looks ok to me.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: beauchampy on April 11, 2017, 04:35:01 PM
Ok, just took a quick lookI took this as you were getting stretched footage? Anyway, the tearing issue is as you say something coming from the compress raw itself not from mlv_dump. About stretching issue it all looks ok to me.

That was only happening when my screen suddenly populated with errors and logs. I think crop_rec had crashed. When it was working properly (non stretched 3x3 50p @ 960px) I get the line tearing.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: beauchampy on April 11, 2017, 04:39:19 PM
Anyone able to get 50p 3x3 1920x960 working without the frame tearing issue I am experiencing?   :-\
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 11, 2017, 04:51:35 PM
@beauchampy I got 1920 x 1080 48p 3x3 without tearing but rec time was inconsistent.
I also got 1920 2.35:1 @ 48p without tearing. That was on the very first build

However, on the 4-10-2017 build I took about 5 or 6 clips of 3072 x 1728 @ 23.976 last night and notice that 1 frame out of all the clips had a small horizontal row of pixels that seemed like it was offset from the rest. Ill upload if I get a chance.

I was using RAWFlow with an updated MLV_DUMP to convert 14 Bit lossless to DNGS.

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 11, 2017, 05:03:14 PM
@beauchampy: known bug, but right now, figuring out how prevent the recent soft-bricks has a higher priority.

Until then, my advice would be to try the Apr03 or older builds (which are double-buffered, which avoids tearing, but Frozen LV is not working well, so recording times may be shorter).

Do not use Apr04.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Frank7D on April 11, 2017, 05:06:12 PM
As far as the frame tearing issue, I saw it on nightly builds a while back on my 7D in crop mode and eliminated it by not using the accurate preview (ml greyscale). It was suggested that processor overload was responsible.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 11, 2017, 05:14:25 PM
The 7D issue does not apply here - on 7D, the issue is caused by the preview backend (and for the moment, I don't have any solution other than disabling the previews completely).

Here, the issue is my single-buffering experiment (which gives a little more memory for high-res modes, but apparently it doesn't play nice with high-FPS modes).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 11, 2017, 05:54:18 PM
@beauchampy
I'm really interested in 192050p 3x3 mode and while I wait for a solution for the tearing issue, for me the solution is 1080p45 3x3 mode. Not same slowmotion like 50p but no tearing issue and continuous recording. Tested at ISO3200 and it works for me continuous!!!. Only framing is a little bit offset...
5Dmk3 113 Lexar 64Gb 1066x
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 11, 2017, 06:19:41 PM
Tested:
April3th build 1920x960 50p 3x3 mode NO TEARING ISSUE!!!! :) :) :) Continuous at ISO100 and 400. At ISO800 shorter recording times (aprox 20sec. for me).
5Dmk3 113 Lexar64Gb1066x
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 11, 2017, 06:28:46 PM
So I can use a little education here. Sorry Im behind. If I get the:
ML ASSERT:
RAW_IS_IDLE
at mlv_lite.c:584 (measure_compression_ratio), task shoot_task
lv:1 mode:3
where the text appears on the screen, is this a crash? I have been assuming it is and pulling the battery. Should I be doing this?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 11, 2017, 07:10:49 PM
Here's a 5D3.123 build from 2017-04-03 for those that want to try and confirm...

https://bitbucket.org/DeafEyeJedi/magic-lantern/downloads/magiclantern-crop_rec_4k_2017April03.5D3123.zip
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: JackDaniel412 on April 11, 2017, 07:41:12 PM
Hello @a1ex , I ran the ROM1 test, no error found.
I updated ML to the April 10th, in debug section show this error

(http://dcmakecreative.altervista.org/VRAM0.png)

There is a way to fix it?

Thanks!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 11, 2017, 07:58:18 PM
is this a crash? I have been assuming it is and pulling the battery. Should I be doing this?

Yes, taking the battery out after a crash is what I always do.

Unfortunately, it doesn't appear to prevent Canon code from saving corrupted settings into the ROM. Still looking into it.

edit: opening the battery door seems to be interpreted as some sort of emergency shutdown, but a clean one - settings are saved into ROM, but ML shutdown hooks (e.g. saving ML config files) are not executed. But, if you somehow hold the battery door switch pressed, and just remove the battery, the ROM is not updated.

@JackDaniel412: asked (http://www.magiclantern.fm/forum/index.php?topic=19300.msg182790#msg182790) and (http://www.rookiemoms.com/try-this-with-me-asked-and-answered-to-end-little-kid-nagging/) answered (http://www.magiclantern.fm/forum/index.php?topic=19300.msg182619#msg182619) ;)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 11, 2017, 08:29:49 PM
Re: tearing I noticed this happening after the 2017-04-06 build and still persist with 2017-04-10 build as well when shooting in 1080p 3x3 45fps (50p Canon) and could this strange pattern of corrupted (plus dropped?) frames in several MLV's that spat out be related to the tearing at all?

Because my instincts are thinking that it may have started it after I had a bunch of random corrupted footage similar to the one below (is this what you were seeing also @beauchampy?) otherwise correct me if I'm wrong.

https://vimeo.com/212794399

Re: 2017-04-03 build I am now getting corrupted frames similar to the one above when shooting in 1080p 3x3 45p until I reduce the resolution down to 1920x960p which all seems fine.

Though on the older build I seem to be able to get clean recording w full 1920x1080p in 3x3 45fps which is rather confusing. Gonna need to take a step back and check again with all of this before I lose it for good. literally.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: beauchampy on April 11, 2017, 09:15:09 PM
@beauchampy: known bug, but right now, figuring out how prevent the recent soft-bricks has a higher priority.

Until then, my advice would be to try the Apr03 or older builds (which are double-buffered, which avoids tearing, but Frozen LV is not working well, so recording times may be shorter).

Do not use Apr04.

Can confirm Apr03 build is working correctly. Thanks a1ex!

No problem buddy, you take your time. Thanks for all your hard work, these updates are amazing.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: jordanwalkert on April 11, 2017, 09:16:48 PM
Hey a1ex,
With the latest build from April 10th (the one to ignore false warnings), I am still getting a message about the null bug. I have not run into any major issues or soft bricking, but I also do not have a backup or unaffected copy of my ROM from before the 4th. Am I screwed, any solution to this without a clean ROM from before?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 11, 2017, 09:22:42 PM
In many cases, I can infer the missing values from another ROMs (so it's not a very hard requirement).

BTW, your existing copy from ML/LOGS is likely to be a good one (as it's only re-created after a format).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 11, 2017, 09:26:11 PM
Yes, taking the battery out after a crash is what I always do.

Unfortunately, it doesn't appear to prevent Canon code from saving corrupted settings into the ROM. Still looking into it.

edit: opening the battery door seems to be interpreted as some sort of emergency shutdown, but a clean one - settings are saved into ROM, but ML shutdown hooks (e.g. saving ML config files) are not executed. But, if you somehow hold the battery door switch pressed, and just remove the battery, the ROM is not updated.

@a1ex Hmmmmm So if I mod the switch with say.... tape, I can pull battery with out afflicted ROM being saved?
Should I do this as a extra precaution when heavy testing?
Would this have adverse affects on Canon code being that its an "unsafe" battery pull at that point? 
So many questions sorry.
I suppose there is always a chance I might not even know that the ROM is affected and I could clean shutdown anyway. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 11, 2017, 09:31:43 PM
@Jordanwalkert
Did you save the roms from when installing the actual april 4th build? Those should be the last uncorrepted ones if so...
looong shot
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 11, 2017, 09:37:44 PM
More testing with april3th. 5Dmk3 113
1920x960 50p 3x3.
Now I'm recording continuous at ISO3200.
I'm a little confused... Maybe it depends on the detail level of the image?...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: giarcpnw on April 11, 2017, 10:18:35 PM
I may have missed this but once you set the settings for 3k 1:1 crop, what's the diff between the regular record mode and then x5 crop mode? They seem the seem on preview but regular mode will record around 3072x1228 but x5 Crop just stops. Is it cranking the data rate or something?

And does FPS override make any difference in any of thi?


Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 11, 2017, 10:21:42 PM
Now I've got two consecutive errors... Error 70, then I turn off camera. Next an  Error 80. Camera does not show live view after both errors. After a new reboot all is apparently ok...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 11, 2017, 10:49:02 PM
Another issue in 1920x960 50p 3x3 is framing.

Here there is a composition comparing real image vs live view framing:
http://ovh.to/TpLuMh (http://ovh.to/TpLuMh)
Title: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 12, 2017, 01:06:30 AM
Can anyone else confirm that it seems when you go into Full-res LiveView (it becomes 2+ stops brighter according to the LV as well as the ML histogram) than if in any other settings within crop mode from the latest experimental build?

If so, is this normal or a bug?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 12, 2017, 01:21:31 AM
Probably it's because of the shutter speed mapping from 1/4000 ... 1/30 to 1/15000 ... 1/FPS (also used on 4K half-fps).

Suggestions for a better (less confusing) implementation?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 12, 2017, 03:47:48 AM
Here is a quick test I did using 5D Mark III in 3k - 3:1 / Build April 10th latest one.  I just Love the images that I'm getting & specially the motion, the images speak for themselves even when youtube super compresses it.  Let me know what you guy's think. 

https://www.youtube.com/watch?v=OmxEjoruetw&feature=youtu.be
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 12, 2017, 06:01:20 AM
When I find it, I am glad.
When I need it, I cant find it :(
Where is the menu option for setting the Preview ?
Thanks for your help
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: D_Odell on April 12, 2017, 11:00:50 AM
I tried today with April10th build. Since I normally shoot 1.85:1 I tried 3K 1:1 (2928 × 1580), ISO 640, shutter 1/50. But the cam can only record for 5 sec with 14 bit lossless. Is that normal? Or how to be able to record constantly with this resolution, if possible? CF card is Sandisk 160mb r/w. Thanks for al the heavy work and fun to reach new heights!

DNG: https://www.dropbox.com/s/tz79q88ozc8f5p4/M12-0944_000004.dng?dl=0
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Walter Schulz on April 12, 2017, 11:06:53 AM
Bandwidth required:
2928 x 1580 x 24 (?) x 14 / 8 = 185 MByte/s
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: D_Odell on April 12, 2017, 11:10:13 AM
Bandwidth required:
2928 x 1580 x 24 (?) x 14 / 8 = 185 MByte/s

Correct 23.976. Normal I watch ML recommendation in menu, and with these settings says 101.8MB/s at 23.976.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: mageye on April 12, 2017, 11:13:17 AM
Try recording at ISO100
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: D_Odell on April 12, 2017, 11:27:12 AM
Try recording at ISO100
Positive results, now 25 sec. Thanks! So reaching (for example) 5 min, lower resolution seems to be the only way?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: mageye on April 12, 2017, 11:37:12 AM
At the moment yes. But the noise from the ISO causes problems for the compression algorithm. I am getting a (reasonably) reliable resolution of 3648x1216 but still not completely continuous!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: D_Odell on April 12, 2017, 11:44:03 AM
At the moment yes. But the noise from the ISO causes problems for the compression algorithm. I am getting a (reasonably) reliable resolution of 3648x1216 but still not completely continuous!
I see, do you mind sharing your ML settings? Thanks!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: mageye on April 12, 2017, 12:00:04 PM
Crop mode UHD 1:1
RAW video ON, 3648x1216
Aspect 3:1
14-bit lossless


Within Canon menu:
Movie rec. size 1920 @ 24 ALL-I
Video System set to: PAL

That's it!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 12, 2017, 12:45:31 PM
Today I was doing a studio shooting for 4-5 hours, several shots.
I had my camera plugged on electricity as well as the monitor.
5d mk3, 1.1.3
Experimental Build 10 April
I used 1920x800 60FPS 16:9 14bit lossless
All manual, no AutoETTR
While I was working on the shot, to adjust lights etc, Camera crashed and saw several messages.
Turn it OFF/ON fixed the problem.
The recording was not continuous but sufficient enough to record the useful action.
I noticed that the Aspect Ratio of the monitoring wasnt right.
Later, I saw that the monitoring the colours lost saturation and the Aspect Ratio was jumping from wrong to right.
I also noticed that although the ISO was set manually to 1250, on ML monitor was still 1600.
The temperature of the camera was 40 C

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Markus on April 12, 2017, 01:19:32 PM

Monitor can't show the hacked bw preview on external monitor. And canons liveview does not give correct representation of ML enabled modes.

Iso in raw is only valid for real Iso values. 100,200,400,800 and so on. The values in between is just digital gain applied to  h264 compressed video.


Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Vegandelight on April 12, 2017, 05:18:27 PM
What kind of work flow are you guys using for the 1920x960 3x3 binning 14bit raw mode?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 12, 2017, 05:55:28 PM
What kind of work flow are you guys using for the 1920x960 3x3 binning 14bit raw mode?
In camera? Or post processing?
For post I used the RAWFlow app and just replaced the MLV_DUMP with the newest version and it processed the 14Bit Lossless just as expected. Drag n Drop.
RAWFlow - http://www.magiclantern.fm/forum/index.php?topic=13338.0
MLV_DUMP Update - https://builds.magiclantern.fm/experiments.html
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Vegandelight on April 12, 2017, 08:13:55 PM
In camera? Or post processing?
For post I used the RAWFlow app and just replaced the MLV_DUMP with the newest version and it processed the 14Bit Lossless just as expected. Drag n Drop.
RAWFlow - http://www.magiclantern.fm/forum/index.php?topic=13338.0
MLV_DUMP Update - https://builds.magiclantern.fm/experiments.html

Nice, Premiere or DaVinci Resolve? The old DNG-files opeened right up in Premiere but these compressed ones does not.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 12, 2017, 08:18:37 PM
Nice, Premiere or DaVinci Resolve? The old DNG-files opeened right up in Premiere but these compressed ones does not.
DaVinci opened them just fine. I don't know premiere.
Its and extra step but you can always export from resolve. Resolve + DNG = incredible color options!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: fkarpenko on April 12, 2017, 09:22:41 PM
Is there a chance we'll see 2k/3k/4k resolutions in 3x3? This is pretty amazing but without the massive crop factor it would be a dream come true.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 12, 2017, 09:39:15 PM
Consider this being a huge leap towards our next mind boggling goal:

8K RAW Video Recording!

Sample DNG from 5D Mark III, to show that our proof of concept is working:

8k.dng (http://a1ex.magiclantern.fm/2017Apr01/8k.dng)

All that's left is to divide the resolution by 3 on each axis ;)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 12, 2017, 09:39:41 PM
Has anyone notice an issue re: Dual-ISO 14-bit lossless MLV seems to be spitting out with correct black levels but incorrect for original 14-bit MLV's. Could this be related to the new mlv_dump not corresponding properly to cr2hdr or at least confused with each other atm?

(https://c1.staticflickr.com/4/3938/33870168051_2a8406905b.jpg) (https://flic.kr/p/TAZsWF) (https://c1.staticflickr.com/3/2926/33842109492_50757d012f.jpg) (https://flic.kr/p/TyvE7q)

Anyway here are some short samples in two of each (138 MB spat out w ease from latest cr2hdr.app -- Thanks @Danne!) and all shot within crop_rec @ 3.5k: https://mega.nz/#!OwV2DTLC!rb39cVOrAlXXuSer2M6fw3vSuvMfiSG508hewIQJvmg
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 12, 2017, 10:49:49 PM
New build posted.

Found a way to prevent Canon code from saving some of the settings at shutdown, including the settings block that caused issues earlier (reverse engineering notes here (http://www.magiclantern.fm/forum/index.php?topic=19369.msg182959;topicseen#msg182959)). While not perfect, if Apr04 had this safeguard, all the cases affected by the null pointer bug would have been caught. Still, there are ways to brick the camera, just a bit less likely to do so accidentally (and, as long as the bootloader is not erased, recoverable).

How it works: whenever a crash is identified, or whenever you open the battery door, Canon code no longer saves the usual setting groups at shutdown (RING and RASEN, if you look in the above link). There are still setting groups not covered by this safeguard (still looking into it, but so far, all the null pointer errors were in the RING group).

To test - the PAL/NTSC setting is in the RING group (and, to my knowledge, only saved at shutdown). You also have a dummy crash under Don't click me.

After a bit of battle-testing, I think this should be back-ported to all models.

Changes related to raw recording:
- double buffering is back (except at very high resolutions); didn't test much
- pre-recording indicator: mm:ss.f + ss.f
- full-res LV increases FPS when lowering vertical resolution (up to 5796x400 @ 48 fps)
- in 10x zoom, FPS override can be disabled by half-shutter to allow focusing
- minor fixes here and there (full list on the download page)

For preview with correct framing on external monitors, try Force HDMI VGA (not tested, but IIRC it used to help).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 12, 2017, 11:58:21 PM
WOW!!!!! @a1ex this is amazing! Cant wait to test!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 13, 2017, 01:00:48 AM
@a1ex Thanx for all the new updates.. Gonna give the new build a try tonight will post my results
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 13, 2017, 01:38:19 AM
Going to update to new build and found this on card from April 10 Build
ML ASSERT:
RAW_IS_IDLE
at mlv_lite.c:584 (measure_compression_ratio), task shoot_task
lv:1 mode:3


Magic Lantern version : crop_rec_4k.2017Apr10.5D3113
Mercurial changeset   : 11f405b62b31 (crop_rec_4k) tip
Built on 2017-04-10 19:48:55 UTC by [email protected]
Free Memory  : 163K + 3118K

Not sure if this needs to be reported.
Testing April 12 build now

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 13, 2017, 02:12:43 AM
Wait..... what did I do wrong here. I couldn't even get one solid rec out of the April 12 build so far. I had to pull the battery every time.
@a1ex Im sending you a message to a zip of my card its peppered with ASSERT messages.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: JADURCA on April 13, 2017, 02:17:13 AM
Hi everyone! I just saw there is a new version, in case it helps I'm sharing this 4 CRASH.LOG files from crop_rec_4k.2017Apr04.5D3123 version.

ASSERT: hLvJob->hJpegMemSuite
at ./Epp/Vram/VramStage.c:891, task Epp
lv:1 mode:3


Magic Lantern version : crop_rec_4k.2017Apr04.5D3123
Mercurial changeset   : e7e1fdb85f6b (crop_rec_4k) tip
Built on 2017-04-04 15:42:08 UTC by [email protected]
Free Memory  : 163K + 3108K

ASSERT: IsSuiteSignature( hSuite )
at ./PackMemory/PackMem.c:599, task Epp
lv:0 mode:3


Magic Lantern version : crop_rec_4k.2017Apr04.5D3123
Mercurial changeset   : e7e1fdb85f6b (crop_rec_4k) tip
Built on 2017-04-04 15:42:08 UTC by [email protected]
Free Memory  : 163K + 3107K


ASSERT: IsSuiteSignature( hSuite )
at ./PackMemory/PackMem.c:599, task RscMgr
lv:0 mode:3


Magic Lantern version : crop_rec_4k.2017Apr04.5D3123
Mercurial changeset   : e7e1fdb85f6b (crop_rec_4k) tip
Built on 2017-04-04 15:42:08 UTC by [email protected]
Free Memory  : 163K + 3107K


ASSERT: IsSuiteSignature( hSuite )
at ./PackMemory/PackMem.c:983, task RscMgr
lv:0 mode:3


Magic Lantern version : crop_rec_4k.2017Apr04.5D3123
Mercurial changeset   : e7e1fdb85f6b (crop_rec_4k) tip
Built on 2017-04-04 15:42:08 UTC by [email protected]
Free Memory  : 163K + 3108K
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 13, 2017, 03:40:56 AM
@JADURCA Dont use that build it's flawed.. make sure you back-up your ROM.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 13, 2017, 04:47:07 AM
@a1ex Just tried the April 12th build it's solid great work.  The only thing that I've noticed is when recording the minute / second green counter stays @ 00.00 it doen't show the actual recording time. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: KMikhail on April 13, 2017, 08:00:36 AM
Just wanted to give heads up to those who want most of resolution with lossless, but high ISO prevents you from it. You can sacrifice dynamic range but have a lower bandwidth requirement by using lower ISO and doing the pull in the post: the noise will be below the original gain line. Sure enough, it is worse than we'd like it, but...

Unfortunately, ISO 50 doesn't work with video, that would've basically removed one bit out of equation. Maybe there is another cheap (performance-wise) trick to lower effective number of bits, so lossless is more effective?

Here's a question: is there a way to have GD when I'm focusing, etc., but have it auto turned off (with everything else that could be turned off) during the recording?

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hindra on April 13, 2017, 10:28:30 AM
Just tested newest build from Latest Build (2017-04-12 20:31).

I got the red dot in the memory patches again, no big deal right? It seemed like most of my shots using crop_rec had a bad first frame regardless of resolution.  Shooting at 3504x1536 at 24fps I only got 157 frames with a Lexar 128gb 1066x card.

Also, something odd kept happening and it kept reverting back to 19fps with FPS override off.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 13, 2017, 11:07:58 AM
Maybe there is another cheap (performance-wise) trick to lower effective number of bits, so lossless is more effective?

Yes - adjusting the digital gain in the raw backend appears to help (as it reduces the range of the data and cuts off the noise bits). Added a (somewhat fake) lossless compression at reduced bit depths, based on this concept (theory explained here (http://www.magiclantern.fm/forum/index.php?topic=18443.msg181620#msg181620)).

Also added some suggestions about what ISO range to use with each option. See the commit for more details on how I've chosen the recommendations.

The only thing that I've noticed is when recording the minute / second green counter stays @ 00.00 it doen't show the actual recording time. 

Fixed - was broken by pre-recording updates.

@a1ex Im sending you a message to a zip of my card its peppered with ASSERT messages.

The issue seems to be in 1920x1080 48p. Hopefully fixed, but I'm seriously thinking to reduce the default resolution to something like 1040 (thus breaking the promise of 1080p48). Any objections?

Also, something odd kept happening and it kept reverting back to 19fps with FPS override off.

How to reproduce this one?



Anyway, new build posted (minor fixes + experimental 8...12-bit lossless).

Be sure to read the changelog before reporting issues ;)

Edit: just noticed the lower bit depth lossless modes only update the raw buffer every other frame, for reasons not yet understood... if you want to try, compile the sources, but you'll get files with duplicate frames (good for estimating recording times, but nothing more). If you look at a static scene (as I did before posting the build), everything appears fine.

Edit2: looks like all raw types (http://www.magiclantern.fm/forum/index.php?topic=18393.0) that include digital gain are created every other frame :( (that means, I need to find some other method for darkening the image...)

Edit3: progress narrowing down (this used to work before merging 5D3-123 into main builds)


Narrowed down to changeset 38e78de. This change fixed it! (https://bitbucket.org/hudson/magic-lantern/commits/4cce3d2adafff9ac942fcf93fa3c4c3e7c4a84f6?at=crop_rec_4k)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: nikfreak on April 13, 2017, 12:17:00 PM
Congrats, lossless 8bit support.  8)
Any easter egg planned to support other cams  ;)?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 13, 2017, 12:18:06 PM
Any easter egg planned to support other cams  ;)?

hint, hint (http://www.magiclantern.fm/forum/index.php?topic=18443.msg182989#msg182989) ;)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: beauchampy on April 13, 2017, 01:31:25 PM

The issue seems to be in 1920x1080 48p. Hopefully fixed, but I'm seriously thinking to reduce the default resolution to something like 1040 (thus breaking the promise of 1080p48). Any objections?

Ouch, so no more 16:9 slow motion? This is kinda a big deal for someone like me to *has* to deliver in 16:9 to some clients..
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 13, 2017, 01:55:55 PM
You still have the option to fine-tune the internal parameters until you get continuous recording with no errors or corrupted frames. No programming knowledge is required (just fiddle with numbers from the menu), but it can easily take hours of tinkering until you find a sweet spot.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: starimage on April 13, 2017, 02:18:18 PM
First of all, thank you so much for the tremendous work and the joy that you give us.
At me a following problem - on all last experimental insertions in a сropmode the screen becomes pink and there are diagonal strips. In the raw mode this does not happen
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 13, 2017, 02:25:49 PM
Okay so I went in for a second test. Over all here is what I am finding

3k 2.35:1 14 Bit lossless Preview auto is very stable

1920x1080 48p, 14 Bit lossless, Preview auto, out of focus scene - not stable lots of crashes even when not Rec out of focus scene

1920 2.35:1 48p, 14 bit lossless, preview auto, out of focus scene - not stable lots of crashes even when not Rec

1920 2.35:1 60p, 14 bit lossless, preview auto - seemed more stable did not crash. Pressed half shutter and it crashed

1920 2.35:1 60p, 14 bit lossless, preview real time - seamed more stable no crash

1920 2.35:1 48p, 14 bit lossless, preview real time - seemed more stable

1920 2.35:1 48p, 14 bit lossless, preview - framing crashed

There were more rec attempts in there but that is all I can remember for now. Unfortunately Ill have to stop testing for a bit. Here are my asserts

ML ASSERT:
0
at mlv_lite.c:2111 (compress_task), task compress_task
lv:1 mode:3

ML ASSERT:
0
at mlv_lite.c:2111 (compress_task), task compress_task
lv:1 mode:3

ML ASSERT:
0
at mlv_lite.c:2118 (compress_task), task compress_task
lv:1 mode:3

ML ASSERT:
0
at mlv_lite.c:2118 (compress_task), task compress_task
lv:1 mode:3

ML ASSERT:
0
at mlv_lite.c:2118 (compress_task), task compress_task
lv:1 mode:3

ML ASSERT:
0
at mlv_lite.c:2118 (compress_task), task compress_task
lv:1 mode:3

ML ASSERT:
0
at mlv_lite.c:2118 (compress_task), task compress_task
lv:1 mode:3

ML ASSERT:
0
at mlv_lite.c:2111 (compress_task), task compress_task
lv:1 mode:3

ML ASSERT:
0
at mlv_lite.c:2111 (compress_task), task compress_task
lv:1 mode:3

ML ASSERT:
0
at mlv_lite.c:2111 (compress_task), task compress_task
lv:1 mode:3

Magic Lantern version : crop_rec_4k.2017Apr12.5D3113
Mercurial changeset   : dcdf7432df74 (crop_rec_4k) tip
Built on 2017-04-12 20:31:48 UTC by [email protected]
Free Memory  : 163K + 3116K



Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 13, 2017, 02:42:03 PM
48p might be a little better in Apr13.

Otherwise, see the last 2-3 messages. Decreasing vertical resolution in the crop_rec submenu is the only way I know to reduce the frequency of these "crashes" (actually corrupted frames).
Title: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 13, 2017, 03:59:09 PM
The issue seems to be in 1920x1080 48p. Hopefully fixed, but I'm seriously thinking to reduce the default resolution to something like 1040 (thus breaking the promise of 1080p48). Any objections?

Yes. Let's keep tinkering down on this one please. 1080p 48p is crucial. Especially for HDR video kind of work.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: squig on April 13, 2017, 04:39:56 PM
Upscaling in post is a thing. I've shot stuff in 720p crop mode that I'd be comfortable projecting on a 40 foot screen. I can put nitrous in an old Honda, that doesn't make it smart. Reliability trumps  (for want of a better word) a few more pixels every time.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Vegandelight on April 13, 2017, 05:22:03 PM
I havent tried the latest build but i tried running a manual lens on the camera today, the screen doesnt turn on when in video mode. Works in photo mode.

HAs anyone experienced that?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: mageye on April 13, 2017, 05:44:39 PM
Yes.

On Samyang 35mm (manual)

I get: 'Ensure a lens is attached. Press START STOP button to activate movie shooting.'

I think it's just a lens communication thing
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Vegandelight on April 13, 2017, 05:53:04 PM
Yes.

On Samyang 35mm (manual)

I get: 'Ensure a lens is attached. Press START STOP button to activate movie shooting.'

I think it's just a lens communication thing

Thats a normal error, but when you press START STOP the screen goes black and you can hear the mirror locking up like it should. But no image.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: mageye on April 13, 2017, 06:27:53 PM
Oh. Well I get the image so who knows what's happening there?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 13, 2017, 06:33:34 PM
When it happens here, pressing MENU twice solves it (see first post).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: mageye on April 13, 2017, 06:42:46 PM
Back to the crop record.

I have noticed something which at first may appear bizarre:

Indoors I get significantly longer record times at higher resolutions than I do outdoors. This has happened repeatedly over a few days now.

I am thinking it has to do with light levels, detail and compression.

eg.

I have been able to record over 16 seconds at [email protected] inside and just about 2 - 4 seconds at the same settings outside!

(14-bit lossless compression)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 13, 2017, 07:24:37 PM
Testing April 13 build. Much more stable for sure! Compression seems to be tricky still. I'm not the biggest fan of how slow settings (shutter ISO ect) appear to change when in Framing LV but I know it serves a purpose to get proper preview with low processing power.
If we can some how cap the data rate that would be insane. But then I guess it would be lossless at that point.

Higher detail low contrast scene, F1.6 1/125 (1/98) ISO 800
1920 2.35:1 14 Bit lossless 1920 48p preview auto - fairly long rec
16:9 Did not make estimated rec time

1920 2.35:1 14 Bit lossless 1920 60p preview auto - Did not make estimated rec time

3072 2.35:1 14 Bit lossless 3k 1:1 23.976 preview auto seamed good.
16:9 Did not make estimated rec time

On same scene I changed the shutter speed F1.6 1/50 (1/98) ISO 800 then none of the above modes made there estimated rec time.
It seems that if more detail is brought out of low exposure zone, less info gets tossed which means higher data rate.

No crashes/bad frame warnings until I lowered the shutter down to 1/30 in 3072 2.35:1 14 Bit lossless 3k 1:1 23.976 preview auto. Then it crashed/bad frame warning ML assert

ML ASSERT:
0
at mlv_lite.c:2120 (compress_task), task compress_task
lv:1 mode:3

ML ASSERT:
0
at mlv_lite.c:2120 (compress_task), task compress_task
lv:1 mode:3

Magic Lantern version : crop_rec_4k.2017Apr13.5D3113
Mercurial changeset   : cf07f797b9c3 (crop_rec_4k)
Built on 2017-04-13 10:34:53 UTC by [email protected]
Free Memory  : 163K + 3115K
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hindra on April 13, 2017, 07:46:19 PM
Anyone else just try 1080p raw with the 1080p all-i proxie with the new april 13th build? Is it broken? Here is what happened to me today.

1080p raw w/ proxie broken, raw to CF and all-i to SD - Stops recording after a few seconds, camera locks up, preview still works, global draw on or off. After messing with proxie, 14bit lossless no longer working? recording at 90+mb/sec for 1080p as stated on my liveview monitor.

ML ASSERT:
0
at mlv_lite.c:2113 (compress_task), task compress_task
lv:1 mode:3


Magic Lantern version : crop_rec_4k.2017Apr13.5D3123
Mercurial changeset   : cf07f797b9c3 (crop_rec_4k)
Built on 2017-04-13 10:35:29 UTC by [email protected]
Free Memory  : 166K + 3785K


ML ASSERT:
RAW_IS_IDLE
at mlv_lite.c:592 (measure_compression_ratio), task shoot_task
lv:1 mode:3


Magic Lantern version : crop_rec_4k.2017Apr13.5D3123
Mercurial changeset   : cf07f797b9c3 (crop_rec_4k)
Built on 2017-04-13 10:35:29 UTC by [email protected]
Free Memory  : 167K + 3119K


I uninstall ML, format card in pc, then format in camera, then load ML software and reload firmware to set bootflag.

Still recording at 90mb/sec and in canon menu at format menu it shows no use of CF card. I am also now having issues recording with any of the crop_rec options. Recording stops very quickly. What did I break?  :(

EDIT:
Lossless is still working, looking at footage the 1080p raw stills are only 1.90MB
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 14, 2017, 10:53:18 AM
Cant hide my excitement.
ML makes my camera as good as cameras costing $$$$$$
and still ML moves on ...
Gratitude!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 14, 2017, 02:51:00 PM
@a1ex Tested the April 14th Build last night here are my results..

3K 2.35:1 14bit Lossless = Ok
               12bit Lossless = Liveview splits in to two screens and image is not really viewable. Recording stops quickly and compression error pops up on screen
               10bit Lossless = Same
                 8bit Lossless = Same
 
1080p 45/48fps 3x3 16:9 - 14bit Lossless = Records for a few seconds than crashes camera
                                        12bit Lossless = Same
                                        10bit Lossless = Same
                                          8bit Lossless = Same 

1080p 60fps 3x3 16:9      - 14bit Lossless = Records for longer time than 48fps, than after crashes camera
                                        12bit Lossless = Same
                                        10bit Lossless = Same
                                          8bit Lossless = Same 
 
Note after trying the slowmo functions camera turns on but ML Liveview only shows a black screen, without magic lantern sd in camera liveview works. 

@Quentin Very exciting
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 14, 2017, 03:05:16 PM
Reproduced - looks like we've got some image processing module that has to be reconfigured for correct resolution. That means, solving this would be one tiny step closer towards fixing LiveView.

(I've only tested the 8...12-bit settings in plain 1080p, without crop_rec)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 14, 2017, 03:20:34 PM
@a1ex Yeah LiveView seems to be very complicated wish I knew coding to help out. Do you think LiveView will ever be as smooth as 1080p in 3k - 4K mode?  Gonna try them in 1080p today, and I'm gonna keep testing.  All those new feature ad-ons makes this camera an elite camera.  Thank you for your hard work. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 14, 2017, 03:59:44 PM
12, 10, 8 bit Lossless!!!!!!!!!! Am I reading this correctly? Oh man. This is unreal!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: anto on April 14, 2017, 04:06:32 PM
Grazie a1ex! Magic!

Inviato dal mio HTC One utilizzando Tapatalk

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: zeus12 on April 14, 2017, 04:07:36 PM
10 bit, 8 bit compressed raw? Am I dreaming? Thanks for your ingenious work!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: GutterPump on April 14, 2017, 04:43:57 PM
Just tested the new 14 april build.

I'am able to record in crop mod with 8 10 and 12 bits lossless. (but now in crop mod, i can't use more than the 2:35 ratio "eg : 3104x1320"  and FPS are locked to 29 if we dont use FPS override. Even if we are in PAL)

This is just crazy in what the 5D mark III is become.

This mean we will really record 3.5k 4k continue.

---

Here a zip with DNG if some want check how look 8/10/12 bits lossless

https://www.dropbox.com/s/pj7db1izytkj3zj/8_10_12bitslossless_samples.zip?dl=0

Three samples from [email protected] clip
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 14, 2017, 04:56:48 PM
Quote
This mean we will really record 3.5k 4k continue.
  :o
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kharak on April 14, 2017, 05:57:24 PM
Does april 14th build run the test for the april 4th build?

My connection is so incredibly slow, if i dont have to dowload apr10 just to test for a bug and then download 14th, well that would be swell :)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 14, 2017, 06:35:29 PM
@GutterPump
You remind me of myself when I shoot lens cap  :D
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: GutterPump on April 14, 2017, 06:46:20 PM
@quentin

Haha it could have been worse, it's really the mess on my desk. But for magic lantern I make an effort.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hindra on April 15, 2017, 05:37:19 AM
Hey guys, here's a quick 23 second shot of the 8 bit lossless 3520x1322 at 60 fps with the latest April 14th 1.2.3 build. My hands are really shaky, I'm not a pro and I had my shutter at 1/80th.
ISO 200, F8, 1/80th - SHAKY HANDS - straight out of camera no corrections/lut - 60mbit vbr
Its working :)

https://vimeo.com/213302351


EDIT: 30 fps
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Teamsleepkid on April 15, 2017, 06:03:23 AM
thanks for posting the video thats pretty crazy
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 15, 2017, 06:04:02 AM
Hey guys, here's a quick 23 second shot of the 8 bit lossless ... ISO 200

Here a zip with DNG if some want check how look 8/10/12 bits lossless

https://www.dropbox.com/s/pj7db1izytkj3zj/8_10_12bitslossless_samples.zip?dl=0


Have you noticed the ISO suggestions when selecting a lower bit depth from menu? (at the bottom of screen)

Please refer to this commit (https://bitbucket.org/hudson/magic-lantern/commits/2028d73f6f34dcdaaa4c241824e538d0e0e47267#Lmodules/mlv_lite/mlv_lite.cF2996T3033) for more details, and to this article (https://theory.uchicago.edu/%7Eejm/pix/20d/tests/noise/noise-p3.html) for some theory regarding bit depth, noise and posterization.

I've considered acceptable a quantization step of twice the noise stdev, which would be just a bit better than "gradient 3-bit tonality" from Fig. 3 in the linked article (log2(256/24) = 3.3), which gives the lowest ISO usable at each bit depth. For the highest ISO, I've considered "wasteful" a quantization step smaller than noise stdev / 2.5.


3520x1322 at 60 fps

Impossible (please check the metadata with mlv_dump -v).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: iheartbmx on April 15, 2017, 06:14:41 AM
Filled a 32GB card with 3K / 2.35:1 / 23.97fps / 14bit lossless - Completely stable with zero bad frames! (I used cr2hdr v1.2 to extract DNG's on OSX Sierra and it worked perfect)

You guys are amazing!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 15, 2017, 06:27:23 AM
@a1ex
I saw ISO suggestions.
8 bit digests noise generated by high ISO ? (I read that raw is divided by 64)
Thanks in advance for the enlightenment
 and your hard work
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 15, 2017, 06:30:09 AM
Yes, at high ISOs, the lowest bits (trimmed for reducing the bit depth) are mostly noise.

At low ISOs, some of them are useful data (especially in shadows).

The last two bits are mostly noise on current sensors (so you can safely use 12-bit at low ISOs).

The posterization from ML previews is real (it matches what I'm getting when opening the DNG in ufraw, for example), so you can actually preview the loss of shadow detail.

For a visual explanation, don't forget to move the mouse over the images from the linked article (https://theory.uchicago.edu/%7Eejm/pix/20d/tests/noise/noise-p3.html) (see previous post).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 15, 2017, 06:32:24 AM
Thats incredible and gives more room to film in low light, broadening the camera capabilities.
Knowing how ETTR works, I can understand what you describe
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hindra on April 15, 2017, 07:28:52 AM
Have you noticed the ISO suggestions when selecting a lower bit depth from menu? (at the bottom of screen)

Please refer to this commit (https://bitbucket.org/hudson/magic-lantern/commits/2028d73f6f34dcdaaa4c241824e538d0e0e47267#Lmodules/mlv_lite/mlv_lite.cF2996T3033) for more details, and to this article (https://theory.uchicago.edu/%7Eejm/pix/20d/tests/noise/noise-p3.html) for some theory regarding bit depth, noise and posterization.

Impossible (please check the metadata with mlv_dump -v).

Yes, I did notice the ISO suggestions. I have no idea why I thought I was shooting at 60fps. I re-rendered at 30 fps, I also cant hit 60fps when attempting to re-create. but, 8 bit lossless works! :) I was trying to use the lowest bitrate to obtain the highest fps and I thought it was 60, I was mistaken. Thank you for those links.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: D_Odell on April 15, 2017, 09:38:32 AM
Is there any chance MLVFS will start supporting these files without using mlv_dump?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: giarcpnw on April 15, 2017, 10:17:26 AM
What would be the reason the DNGs from the April 14th build would not open in After Effects? I'm getting can not parse errors. Worked fine with April 10th. Using Footage.app on mac.  Shot 14bit lossless.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: giarcpnw on April 15, 2017, 10:26:23 AM
What would be the reason the DNGs from the April 14th build would not open in After Effects? I'm getting can not parse errors. Worked fine with April 10th. Using Footage.app on mac.  Shot 14bit lossless.

I think I answered my own question. The first frame of the sequence in all the MLVs I shot tonight were black (I assume corrupted) I removed them from the sequences and then AE and Camera Raw could load the sequence.

Something to note for this build. April 14th.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 15, 2017, 11:43:21 AM
Small MLV and steps to reproduce, please.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: GutterPump on April 15, 2017, 11:47:57 AM

Have you noticed the ISO suggestions when selecting a lower bit depth from menu? (at the bottom of screen)


Of course i noticed these suggestions, now i read your article i'm able to understand better.

I will do differents tests in low light condition with different range of ISO. eg : 12800 ISO to 800.

So, if i understand well, in this way, the noise is our friend if we need less posterization because the 8bits depth ? So it means that if we film a scene very illuminated outside with the 8bits , we have all interest to use a variable filter ND.

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 15, 2017, 01:12:36 PM
So, if i understand well, in this way, the noise is our friend if we need less posterization because the 8bits depth ?

Correct.

To be effective for reducing posterization, the noise must be applied before the quantization (therefore, analog ISO noise is fine). Adding it in post will not help.

Quote
So it means that if we film a scene very illuminated outside with the 8bits , we have all interest to use a variable filter ND.

I disagree here. Lower bit depths were added to reduce the data rate at higher ISOs (where the lowest bits are mostly noise, and because of them, the image doesn't compress well).

I don't think filming with 8 bits is actually your goal. More likely, I assume you actually want to minimize the data rate. With uncompressed data, it's easy (all frames have the same size: W x H x bpp/8). With lossless compression, you can't choose the compression ratio in advance (http://www.magiclantern.fm/forum/index.php?topic=19300.msg182229#msg182229); this is given - roughly - by the entropy of the image.

These "fake bit depth" settings (actually dividing the signal by a power of 2) are one way to reduce the entropy (by removing the least significant bits, because of integer division). That's a lossy operation (going from 14 bits to any lower value). After this step, compression is lossless (that is, the uncompressed data will be identical to what was fed to the compressor).

(That means, 14-bit lossless will give raw data identical to uncompressed 14-bit (100% identical, not one single bit different, contrary to some earlier report (http://www.magiclantern.fm/forum/index.php?topic=19300.msg182741#msg182741)), while "12-bit lossless" should be interpreted as "14-bit to 12-bit conversion - lossy by definition - followed by lossless compression", so it should have the same number of useful levels as uncompressed 12-bit. Please note the 12-bit lossless is not identical to 12-bit uncompressed - they differ by a constant value and possibly by some round-off error, and the same is true for lower bit depths.)

Back to entropy - increasing the ISO and underexposing (to keep the image brightness unchanged) will increase noise in the image (at all brightness levels). So, what you may gain by reducing the bit depth, you will lose from the added noise. Here's a quick test (static outdoor scene exposed to the right, plain 1080p, compression rates taken from ML menu):

Bits per pixel      14  12  11  10   9   8
ISO  100 1/100     61% 53% 50% 48% 46% 43%
ISO  200 1/200     62% 54% 51% 49% 47% 44%
ISO  400 1/400     63% 54% 51% 49% 47% 45%
ISO  800 1/800     65% 55% 52% 50% 48% 46%
ISO 1600 1/1600    67% 56% 53% 50% 48% 46%
ISO 3200 1/3200    70% 57% 53% 50% 49% 47%
ISO 6400 1/6250    76% 60% 55% 52% 50% 48%
ISO 12800 1/12500  79% 63% 57% 53% 50% 49%


=> there's nothing to gain by using a lower bit depth at a higher ISO + underexposure.

Of course, if your goal is to record at 8 bits, no matter what, then of course, the best thing to do is to increase ISO at 12800 (or at least 6400) for half-decent results. It's also trivial to change the code to get 6 bits, 4 bits or even one single bit, if that's your goal ;)

My advice: first expose properly (ETTR, zebras, raw histogram, whatever you prefer), and afterwards choose the optimal bit depth. Not the other way.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 15, 2017, 02:54:41 PM
Actually that post about compression needed to be in two parts
a) Why do I get these boxy big pixels compression-like in out of focus areas
b) Is 14bit and 14bit lossless visually identical

The question b) is answered already.
The question a), I guess was due to a different reason other than the bit depth

But I understand a1ex words about bit depth. Great explanation
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: VinceML on April 15, 2017, 04:28:43 PM
Hi Guys,

I explain how work and how I setup my 5D for create 3,5K video, here :

https://www.youtube.com/watch?v=kPQMNHhV41M
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Levas on April 15, 2017, 05:02:26 PM
 :o
Now with these fake bit depth options, this is getting interesting for the SD card camera's  :D

@Alex,
The red numbers in your bit depth / compression ratio table, do you mean that the serious loss of info/shadow detail is where the red numbers are ?
And if I understand correct, you're using digital gain, which cut's of the noisy bit, but the image technically stays 14 bit for the compression algorithm, but because of the noisy bit cut off, the compression rate goes even beyond the 50%.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 15, 2017, 07:21:53 PM
Correct - the red numbers match my ISO recommendations (https://bitbucket.org/hudson/magic-lantern/commits/2028d73f6f34dcdaaa4c241824e538d0e0e47267#Lmodules/mlv_lite/mlv_lite.cF2996T3033) and are below "gradient 3-bit tonality" from the third figure (labeled Fig. 18) from the noise article (https://theory.uchicago.edu/~ejm/pix/20d/tests/noise/noise-p3.html). Also added orange = wasteful (there's little or no additional image quality to gain at these settings).

If the theory is correct (which I didn't have the patience to check by pixel peeping), then it's probably best to stay near the transition line to red, on the black side (that is, 11-bit for ISO 100-800, and reduce the bit depth once you need ISO 1600 and beyond).

I believe the recommended settings are visually identical to regular 14-bit (while in 14-bit lossless, the raw data is 100% identical to uncompressed 14-bit). Again, this is not confirmed by pixel-peeping (it's just noise theory). There may be a small color cast in shadows, because the 0.5 LSB offset is not yet applied (but it's easy to fix once @martinhering helps us decode the JPEG stream (http://www.magiclantern.fm/forum/index.php?topic=18949.msg183230#msg183230)).

The estimations were done for 3x3 binning (plain 1080p); the 1:1 crop mode may require slightly different values.

At these (probably optimal) settings, the compression rate appears to be consistent (50%), at least on my test scene (it was some foliage exposed to the right, but didn't save it).

(my definition for compression rate is: saved frame size / size of a 14-bit uncompressed frame)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: giarcpnw on April 15, 2017, 07:54:41 PM
Small MLV and steps to reproduce, please.

Well, i could not duplicate it. I updated Martin's footage app this morning and they process fine. I do however remember it happened right after i tried to capture 12bit and 8 bit lossless and saw this on my screen.
This happens whenever I try to capture any lossless bit rate than other than 14bit lossless. Trying to record like this results in a ton of scrolling errors. (none of which I know what they mean) Have to power off to stop.   Pressing x5 crop WILL ALLOW capture however in this mode. 3072x1308, FPS Override 23.976

(http://i65.tinypic.com/bhyh5j.jpg)

(http://i64.tinypic.com/2n7qul5.jpg)

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 15, 2017, 07:58:42 PM
@giarcpnw: already reported, and for the moment I don't know how to solve it. Will probably disable it in crop modes with modified resolution, until I'll understand what happens.

@Quentin: I don't see "boxy big pixels compression-like in out of focus areas" in your post - I only see black text on a blueish background...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 15, 2017, 08:19:26 PM
Unfortunately I erased the videos.
But ... how do we attach an image on the post ?
Sorry for my ignorance ?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hindra on April 15, 2017, 08:20:28 PM
1080p48 at 12 and 11 bit lossless at 100 iso I get results with a split exposure. Global draw off

(http://i.imgur.com/dNVbq9P.jpg?1) (http://imgur.com/dNVbq9P)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hindra on April 15, 2017, 08:27:13 PM
Unfortunately I erased the videos.
But ... how do we attach an image on the post ?
Sorry for my ignorance ?


I upload to imgur.com and copy/paste the BBCcode link in the forum post.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 15, 2017, 08:36:07 PM
Testing last build april14th.
Tearing issue in 1920x96050p3x3 is gone (I understand because modification of buffering). It is almost perfect for me. I'm really grateful, thanks A1ex!!!... It was a dream some time ago, and now...
Thank you very very much...

Now the only downside is recording time with hi isos in 14bit lossless and incorrect framing, but I can live with that.

Thanks again!!!!!!!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kharak on April 15, 2017, 11:00:41 PM
I successfully downloaded Apr14 build, but it did not do a ROM Test at start, not that I could see atleast. Do I need to download Apr10 specifically for the ROM test ?

Anyways, with this build I have correct/centered framing at 3072x1728 14Bit Lossless 16:9. Very nice! Will actually start using this thing now.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 16, 2017, 04:57:43 AM
I successfully downloaded Apr14 build, but it did not do a ROM Test at start, not that I could see atleast. Do I need to download Apr10 specifically for the ROM test ?

Have you tried running firmware update again from Canon menu and restart camera?

If no messages prompts up afterwards then you haven't been affected from that April 4th null pointer bug.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: etienne on April 16, 2017, 06:26:09 AM
Hey

Great job for the low bit depth it opens great perspective to play around the 3.5k.

Here is my bug encouters for low bit depth 8,9,10,11:

 -  the expo is darken in the realtime preview and hdmi out. realtimre preview for 8 bit totally unusable (this is bas for focusing) framing mode is working better but focus is still not possible from there.

 - 1:1 Tall have the lower of the image not displayed correctly like mentionned for slow mode

 - non working on 3K mode and higher

Can we expect a mlv sound coming back? post synchro is so burden with just the beep :P
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Kharak on April 16, 2017, 03:31:17 PM
Have you tried running firmware update again from Canon menu and restart camera?

If no messages prompts up afterwards then you haven't been affected from that April 4th null pointer bug.


Thank you, Deafeye. I did as you said, update from Canon and restart. But could not see any test being done. Thought I saw a screenshot from someone doing the test and there was like this 0-100 Counter, maybe I am mixing it up with something else.

Its just a pain for me downloading builds cause I am on Satellite internet, I get 4-5 kb/s on a good day and then there are constant dropouts which which require me to start the download all over again... Just loading this forum page takes forever, and its only text more or less..

So its safe to assume that I need to download apr10 build because it has the Rom Test ?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 16, 2017, 04:53:18 PM
https://bitbucket.org/hudson/magic-lantern/src/crop_rec_4k/src/debug.c?fileviewer=file-view-default#debug.c-288

If no errors are present, no obvious feedback is given (other than an extra LED blink, which is pretty obvious to me).

I get 4-5 kb/s on a good day and then there are constant dropouts which which require me to start the download all over again... Just loading this forum page takes forever, and its only text more or less..

Here's what helped me on a similar connection, besides obvious things such as disabling images or using a text-mode browser (mobile connection with poor signal, far away from the city, 4K/s at best):

http://lartc.org/wondershaper/

There are probably equivalents for other operating systems. The idea is to reduce download speed so you can load other pages (or use a chat client) while a long download is running.

Having a remote server accessible with VNC, with a very low image quality setting, can be also helpful (as loading a JPEG preview of a webpage can often have much lower bandwidth than its HTML+CSS+JS+images).

For a download with frequent dropouts, just leave a "wget" command open in background (as it doesn't give up easily, unlike most browsers).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: dfort on April 16, 2017, 09:59:45 PM
There seems to be lots of interest getting crop_rec on steroids working on other cameras.

The EOSM can already do basic 3x3 720p crop_rec and I got it working a while ago on the 700D but was trying to get the other options available on the 5D3 working before doing a pull request. I haven't gotten very far but since other users are also experimenting with adtg_gui, dm-spy and other ML tools to get 4K working on their cameras, maybe we should learn to crawl before attempting to fly so I posted a pull request for limited support of the crop_rec module for the 700D.

https://bitbucket.org/hudson/magic-lantern/pull-requests/824/crop_rec-for-700d/diff

Uploaded a new test build in my bitbucket download area.

https://bitbucket.org/daniel_fort/magic-lantern/downloads/
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: rubnva on April 16, 2017, 11:40:36 PM
Tested the new software update with my Canon 5D Mark III and I used this video mode: 1920x960 @ 50p (both 1:1 crop and full-frame - 3x3 pixel binning) - continuous*)
but al converted (with MLV Mystic) DNG"s are corrupt (pink pixels)

Check my MLV file here: https://we.tl/lvASYcNcRx

Some help please? :)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: mk11174 on April 17, 2017, 02:49:10 AM
On the 700D, I finally found the reg needed to fix the darkened lower half of the image when trying for higher resolution than 2048x1152.
It was ADTG2[82B6].
Best fps I can get without a freeze at this res is 10.639fps.
Don't mind the crappy image quality, room was dark and was to lazy to turn on the light for testing, LOL, was shooting 3200ISO.
(https://s4.postimg.org/xkgbmbgz1/4kfix.jpg)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 17, 2017, 06:00:14 AM
Sorry for the double post!!! But I Just tested 4-14 build 12 bit 3072 2.35:1 at 23.976 ISO 3200 with the old 5x zoom method and it seemed like it was endless. Over 20 secs for sure!!!! DNGS are clean and processed fine and all worked in resolve.
So the this will work for sure!!!! Just need to get the crop rec working. Way easier said then done no doubt. this is crazy tho. 3k 12bit lossless 24p on a 2012 5D
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 17, 2017, 08:18:42 AM
@hjfilmspeed: already reported (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183180#msg183180) and answered (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183243#msg183243) a few (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183246#msg183246) times...

Framing preview works fine and shows what you are going to get.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 17, 2017, 02:09:07 PM
Ahh sorry @a1ex missed that. But the concept is there. This is really awesome!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: treez on April 17, 2017, 03:45:28 PM
Great work...but how to convert the crop mode *.mlv file. I use MLVrawviewer but it crashes.
Any Tips or ideas?

Thnxs.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: zeus12 on April 17, 2017, 04:50:49 PM
Can anyone post their advanced settings on the crop_rec on 5D3123 so that I can get the crop from the center? Currently it looks the crop start from the extreme right side of the sensor (3K).

Thanks for all your hero members for the advancement.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 17, 2017, 09:59:12 PM
Admiration for many reasons
For revolutionary development of out of this world features , for so many cameras
Baby sitting the forums with so many users of so many experiences and variation of needs and knowledge
I surrender, I could not do that
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: BBA on April 17, 2017, 10:53:44 PM
Like in french : RESPECT !!!!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: etx on April 18, 2017, 05:01:48 AM
Can anyone post their advanced settings on the crop_rec on 5D3123 so that I can get the crop from the center? Currently it looks the crop start from the extreme right side of the sensor (3K).

Thanks for all your hero members for the advancement.

Don't use 5x mode like before. If your in regular live view you will get proper framing. It's a bit faster than before also! If you go to 5x mode the framing is off. Took me a bit to figure this out. HDMI preview isn't working yet in this mode but it's amazing none the less. I'm standing by until it does so I can use the 5D on my Movi again.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 18, 2017, 09:44:29 AM
@quentin

+1

Thank you!!!!!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: child01 on April 18, 2017, 01:55:50 PM
Can ANYONE point where is a precise step by step instruction of using mlv_dump?

Yes, I know - noob! how da hell ... and so

But I'm using ML almost since it's first build, but thankfully chmee wrote his tool very soon...
I was always using RAW2CDNG, but now I have to deal with this rocket science - command line stuff ((
And there's no clear and simple TUTORIAL here on forum or youtube video tutorial. Everything I've found is either outdated or people being sarcastic (not helping) and in most cases for MAC (why mlv_dump discussions are always mac oriented?)

A1ex is so helping and responsive. I don't get it why didn't he created a normal software with UI ...or clear and simple tutorial or list of commands for us to just copy/paste?
Why not implement at least base and commonly used commands into mlv_dump somehow? Or enable drag'n'drop functionality? anything simple
 :-\

who can help - what should I do with mlv_dump on PC to simply convert MLV to DNG?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Easygoingmartin on April 18, 2017, 05:31:54 PM
Hey guys.. i'm kinda new to ML.. i've loaded this experimental build and started trying to record in UHD. I've loaded the Rec_crop module and tried the UHD setting. Now the 1080p/50fps setting works sortof(it's not actually "slow motion" since windows still recognizes it as 29fps) the 3K/UHD/4K setting won't work. There's a weird purple/pink artifact going on when i try to film.

Would that mean i have to record in RAW? If so, what are the appropiate modules i have to activate to make it work?

Thanks jn advance!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Ryan Lightbourn on April 18, 2017, 06:55:38 PM
Is anyone getting continuous 3K raw in 4:3 aspect ratio (for 2x anamorphic shooting)? If somebody can confirm I'll pull the trigger on a used 5D3 body!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: g3gg0 on April 18, 2017, 07:58:04 PM
A1ex is so helping and responsive. I don't get it why didn't he created a normal software with UI ...or clear and simple tutorial or list of commands for us to just copy/paste?
Why not implement at least base and commonly used commands into mlv_dump somehow? Or enable drag'n'drop functionality? anything simple
 :-\

pay him 4k€ a month and he will probably do this additional work on saturday and sunday night - the only time when he usually sleeps.

ah and i forgot to mention that i implemented mlv_dump as reference implementation, tool for ML devs, users on nerd level 100 and fallback solution.
*if* there is a proper GUI frontend - go for it, else mlv_dump has to do the job :)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Walter Schulz on April 18, 2017, 08:15:47 PM
My opinion: Get rid of those experimental builds. It invites users not ready for experiments on this level.
Those wanting to go there may
a) have an environment to make their own builds
or
b) rely on a helping hand building it for them on demand.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 18, 2017, 08:35:07 PM
My opinion: Get rid of those experimental builds. It invites users not ready for experiments on this level.
Those wanting to go there may
a) have an environment to make their own builds
or
b) rely on a helping hand building it for them on demand.

Hmmm I not sure I agree when it comes to experimental builds. Although I guess I don't quite get what you mean by "rely on a helping hand building it for them on demand."

On one hand, the admins hold our hands and babysit us while testing. Time consuming to say the least.
On the other hand, some of us who have been using ML for years get a chance to learn about the inner workings of ML and learn how to become better testers.
Also, when it comes to working out the bugs, us inexperience ones are the best at breaking things because we don't listen.

However I do feel that people should not get upset or wine or cry or point fingers when they can't figure something out. I think people need to be more aware that this is all volunteer work. No pay for hours of endless coding. One should not complain especially when one has no clue what is involved with coding.   

But that is my humble "I don't even know what I'm talking about but should since I have been using ML for so long and I am being grossly off topic and I should know better about that too run on quote"  opinion. An opinion to which I feel I don't even deserve to give. Compared to Walter Schulz I am a dumb dumb head. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: GutterPump on April 18, 2017, 09:08:10 PM
Not agree, experimentals builds help to understand better the universe of Magic Lantern, it's a big help for the community, the developpement is more fast, Alpha testers are more large.

Do not delete them, the simple warning message in the top of page is enough. People have been warned from the beginning with the risks of Magic Lantern.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Lars Steenhoff on April 18, 2017, 10:57:10 PM
Experimental builds are the soul of magic lantern right now, it allows people to test lit the latest and give very valuable feedback, @walter if someone dont like them, or are scared it may harm they are free not to use them, but i for one love them.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: zeus12 on April 19, 2017, 12:48:03 AM
The experimental version of magic lantern is on the cutting edge, frustratingly cutting edge. It’s part of the struggle for the code-blind mortals who’s pulling their hairs out and ask the stupidest questions on the board. The struggles make magic lantern exhilarating,  and inspires participants to make better picture, better videos. Most of us do not post anything on the board just for the fear of sounding really really….. stupid. However, we do really appreciate the “geniuses” and generosity of those immortals such as Alex. Please bear with us.
By the way, I finally get UHD to work as well as 1080p @45fps. The DNGs were generated with Rawflow (page 13 of this board). Somehow, the first frame is always bad and the rest are splendid.
Thanks for all the immortals on the board and hope you can improve magic lantern further and let the mortals to try the experimental versions. The ‘code blind” mortals can’t contribute much to the development of magic lantern. But we will do more volunteers to places such as “soup kitchen” to feed the homeless as our appreciation.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: dfort on April 19, 2017, 09:01:18 AM
My opinion: Get rid of those experimental builds. It invites users not ready for experiments on this level.

I was posting test builds for the 12-bit (and 10-bit) RAW video development discussion (http://www.magiclantern.fm/forum/index.php?topic=5601.0) before the Experiments (https://builds.magiclantern.fm/experiments.html) page went live so I feel partially responsible for rolling that snowball.

It would be great if more users would set up development environments or at least take a good look at the code. Even if you can't compile or understand the code, there are some great comments in there that explains what the developers are doing and perhaps more importantly, what areas still need some more testing. For example, in the crop_rec.c file of the crop_rec_4k branch (to stay on topic) you'll find this:

Code: [Select]
/* max resolution for each video mode (trial and error) */
/* it's usually possible to push the numbers a few pixels further,
 * at the risk of corrupted frames */
static int max_resolutions[NUM_CROP_PRESETS][5] = {
                                /*   24p   25p   30p   50p   60p */
    [CROP_PRESET_3X_TALL]       = { 1920, 1728, 1536,  960,  800 },
    [CROP_PRESET_3x3_1X]        = { 1290, 1290, 1290,  960,  800 },
    [CROP_PRESET_3x3_1X_48p]    = { 1290, 1290, 1290, 1080, 1080 }, /* 1080p45/48 */
    [CROP_PRESET_3K]            = { 1920, 1728, 1504,  760,  680 },
    [CROP_PRESET_UHD]           = { 1536, 1472, 1120,  640,  540 },
    [CROP_PRESET_4K_HFPS]       = { 2560, 2560, 2500, 1440, 1200 },
    [CROP_PRESET_FULLRES_LV]    = { 3870, 3870, 3870, 3870, 3870 },
};

So instead of simply reporting corrupt frames, indicate the resolution where you start getting corrupt frames and how far you had to back off to get consistently clean results. If you can compile, try pushing it a little further and report how far you can go before it breaks.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 19, 2017, 09:23:24 AM
So instead of simply reporting corrupt frames, indicate the resolution where you start getting corrupt frames and how far you had to back off to get consistently clean results. If you can compile, try pushing it a little further and report how far you can go before it breaks.

For this particular snippet, vertical resolution is actually adjustable from the crop_rec submenu (no need to compile). I don't remember anyone trying to change it, even though I've mentioned (http://www.magiclantern.fm/forum/index.php?topic=19300.msg182320#msg182320) it a (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183060#msg183060) few (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183112#msg183112) times (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183115#msg183115) (including first post (http://www.magiclantern.fm/forum/index.php?topic=19300.msg182052#msg182052)).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: garry23 on April 19, 2017, 10:14:03 AM
@dfort

I agree with your 'set up a development environment', and, as you know, I wish to do this.

However, and this is a big however, not all of us are competent to do that without very clear and simple instructions.

I have tried to set up such an environment, and although the instructions I was following were good, they still assumed some knowledge I don't have. I kept getting error messages and gave up :-(

From my perspective it would be good to reset/start an ML page specifically directed at getting your own compile environment up and running, and keep Win and Mac instructions separate. Also this page should be written from a nil knowledge perspective, i.e. for 'idiots' like me.

Of course, I appreciate I'm asking a lot, i.e. someone who knows what they are doing sitting down and writing that page.

But until I can follow a simple workflow for setting up a compile environment, I flummoxed.

Cheers

Garry
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 19, 2017, 10:41:21 AM
Just start out with what´s not working in one of the compiling threads and take it from there. It´s impossible to account for every single thing that comes up when compiling although dfort´s efforts made it much easier, especially for mac users.
I´m pretty confident the solution won´t be far when out in the open.
Here are two threads if anyone is interested. There are more efforts and threads out there.
Mac
http://www.magiclantern.fm/forum/index.php?topic=16012.0
Windows
http://www.magiclantern.fm/forum/index.php?topic=15894.0
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Walter Schulz on April 19, 2017, 11:26:20 AM
I followed Mac instructions and have my own machine running for some time. Had to build a Hackintosh for that and I'm a Windows guy.
Didn't test Windows workflow, yet.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 19, 2017, 04:07:24 PM
Having involved into beta testing of 3d software in the past, in my humble experience, its good to keep an order in information coming, sort them, classify them, confirm symptoms etc. Its very hard
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 19, 2017, 04:27:55 PM
For this particular snippet, vertical resolution is actually adjustable from the crop_rec submenu (no need to compile). I don't remember anyone trying to change it, even though I've mentioned (http://www.magiclantern.fm/forum/index.php?topic=19300.msg182320#msg182320) it a (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183060#msg183060) few (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183112#msg183112) times (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183115#msg183115) (including first post (http://www.magiclantern.fm/forum/index.php?topic=19300.msg182052#msg182052)).

@a1ex I have to say, I was super nervous about changing those settings since I didn't do enough research on what they mean but Ill see if I can figure it out. Has anyone one tried tweaking those? What did you find?
 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 19, 2017, 04:58:42 PM
Simple example: if you often get corrupted frames in 1080p48, dial 1040 (or some other value lower than 1080) in the Target YRES field. Then refresh the LiveView manually (e.g. press MENU twice) and the resolution in RAW video menu should update.

Set 0 to disable the override and go back to default value.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 19, 2017, 05:31:37 PM
Simple example: if you often get corrupted frames in 1080p48, dial 1040 (or some other value lower than 1080) in the Target YRES field. Then refresh the LiveView manually (e.g. press MENU twice) and the resolution in RAW video menu should update.

Set 0 to disable the override and go back to default value.
@a1ex Ohhh cool! Do you have to match the value in the MLV Lite options?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 19, 2017, 06:18:13 PM
This sets the maximum vertical resolution available for the raw recorder (which records a cropped section from the full-size raw buffer).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 20, 2017, 06:24:41 AM
It takes 5 minutes to dial proper number.
I would prefer a number to subtract rather than setting absolute vertical resolution.
Its faster and practical.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hindra on April 20, 2017, 06:56:12 AM
I'd actually like to try and raise the vertical resolution and shorten the width to try and get the closest to 16:9 at highest resolution possible.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 20, 2017, 08:42:04 AM
@Quentin: have you tried using SET, left/right and scrollwheel?

The same controls are in many other places in ML menu, whenever you need to input a large number (check e.g. intervalometer).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 20, 2017, 08:49:28 AM
Thanks a1ex, too many things to remember/learn
Its hard to carry a manual with you :D
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: tifose on April 20, 2017, 01:07:33 PM
Whats the best Card to use?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hindra on April 20, 2017, 08:58:44 PM
Whats the best Card to use?

I'm not sure on the best card, but I have had great success with Lexar 32gb and 128gb 1066x CF cards.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Lars Steenhoff on April 20, 2017, 09:59:59 PM
same for the fastest sandisk cf 160 mb 128 gb they work great for me.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: tifose on April 21, 2017, 02:12:50 AM
Thanks guys will get one  my CF card stop working in camera
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Walter Schulz on April 21, 2017, 09:25:35 PM
If your card is working in cardreader but not in cam you might have a defective cam.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 21, 2017, 11:55:05 PM
New build:
- centered x5 zoom is back (3.5K with 8...12-bit lossless compression)
- disabled 8...12-bit lossless in incompatible modes (sorry, still couldn't figure out how to fix them)
- for bit depths between 8 and 11, the choice is now automatic, depending on ISO (see this post (http://www.magiclantern.fm/forum/index.php?topic=19300.msg183231#msg183231))
- fixed silent pics (https://bitbucket.org/hudson/magic-lantern/commits/a9c8ad412af5c282bbad6889ce5f65d974ca3ada) (a bit sad to see them broken for 3 weeks with no bugs reported)
- minor UI fine-tunings.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: dfort on April 22, 2017, 03:59:47 AM
- fixed silent pics (https://bitbucket.org/hudson/magic-lantern/commits/a9c8ad412af5c282bbad6889ce5f65d974ca3ada) (a bit sad to see them broken for 3 weeks with no bugs reported)

Sorry I noticed that while trying to get lossless compression working on other cameras but didn't create a proper bug report--though I did point it out. I'm working on the compressed_raw branch but maybe the crop_rec_4k branch has the latest changes? Noted that  lossless.c and silent.c aren't in sync (https://bitbucket.org/hudson/magic-lantern/branches/compare/crop_rec_4k%0Dcompressed_raw#chg-modules/silent/lossless.c) between these two branches.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: tifose on April 22, 2017, 05:14:46 AM
@Walter Schulz

it is working in my Reader and 5Ds

i use to use my 5DMk3 for Raw just didn't use it in over a year not sure why this is happening any idea how this can be fix?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 22, 2017, 05:43:17 AM
Great work @a1ex you tried your best thank you for all the updates. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 22, 2017, 05:47:27 AM
@tifose You should try a different card with your camera.  Also use a flash light or light source look inside the card bay to see if any of your pins are bent, if not you may have a problem. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: csound on April 22, 2017, 05:51:16 AM
I have the null pointer bug warning coming up with April 21 build. Message says to contact ML developers on IRC Magic Lantern freenode. Tried but not sure I was at correct place to do this - Alex can you supply URL please. Unfortunately don't have old logs.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: lostfeliz on April 22, 2017, 06:26:13 AM
I've been experimenting with the 3K crop and 1080p slow motion, etc from the experimental build. Amazing stuff. It's hard to say, of course, how long it will be before usability is nailed down. Wondering if there might be a shorter term move to combine the "simple" 1080p24/30 lossless recording that's been developed along with the MLV Sound module? I would love to incorporate something like that to my everyday run and gun interviews and b-roll without the occasional random breaks in recording and the extra large files that quickly fill up my cards.

(this was posted in reverse engineering but it was recommended that I delete it and post here)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 22, 2017, 06:49:21 AM
I just tested and this is my favorite build yet!!!

Scene is well exposed fine detail @ ISO 3200
3072 x 1308 (or what ever 2.35:1) Is working extremely well in 5x zoom 12 bit lossless fps override 24.006 and framing preview. I disable the crop rec mod while recording in 5x (not sure if I should use crop rec and 5x at the same time) converting DNGs as I type.
Edit: Out of about 10,000 frames not one bad frame!!!!!!
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 22, 2017, 08:51:49 AM
@hjfilmspeed: the centered x5 preset from crop_rec is for x5 zoom; all the others are for non-zoomed modes. The help at the bottom should tell you. You can also choose 23.976 for FPS override.

@csound: type IRC in your browser's search box (CTRL-F) ;)

@dfort: yes, I've looked into it because of your report, but it was my explicit request to try the bad commit, not a regular user trying it for e.g. timelapse or whatever. Also, this branch has a lot of stuff that can be broken into smaller branches and back-ported to all other models, in the main builds (already started with this PR (https://bitbucket.org/hudson/magic-lantern/pull-requests/825/prevent-canon-settings-from-being-saved/diff)).

@lostfeliz: I was also thinking to add mlv_sound compatibility to mlv_lite, just didn't look into it yet (since, for my own use, only the full-res and 4K half-FPS modes are interesting - for extracting still frames from a burst sequence, or for timelapse). That's the reason you see weird changes such as single-buffering, ability to record with only 2 buffers, half-shutter trigger etc.

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: csound on April 22, 2017, 09:20:22 AM
Quote
@csound: type IRC in your browser's search box (CTRL-F) ;)

A1ex, apologies for being a newbe clutz - but what do I do when I get to freenode?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Lars Steenhoff on April 22, 2017, 09:22:49 AM
Alex sound would be great, and if the manual lens info could be merged into it, that would be nice too.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 22, 2017, 09:24:43 AM
@hjfilmspeed: the centered x5 preset from crop_rec is for x5 zoom; all the others are for non-zoomed modes. The help at the bottom should tell you. You can also choose 23.976 for FPS override.


@a1ex OHHHHHH I didn't even see that option! That mode also lets you use 12bit and at 3200 iso im getting what seems to be continuous 3072x1308 24 FPS!!!! And its centered!!!!! and you can use the zoomed real time preview if you don't want to use the framing preview!!! No crashes so far.
The only thing I noticed is the shutter speed in at the bottom didn't seem to be registering the right number. Unless something else is going on here. The cameras shutter speed was 1/50 but when zoomed + fps override it showed 1/27 or 1/37 or something like that.
But wow .... This is impressive!!!!  THANK YOU FOR THIS!!!!
This is much more exciting then the C- log announcement. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 22, 2017, 10:00:03 AM
The shutter speed issue appears only when using the 5x preset (it appears fine without crop_rec); can you confirm?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: tifose on April 22, 2017, 01:41:40 PM
Thanks Guys it was a bad pin. back up and running :)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 22, 2017, 02:17:59 PM
The shutter speed issue appears only when using the 5x preset (it appears fine without crop_rec); can you confirm?
Yup confirmed. Just crop mode 5x preset @a1ex just tested now. 5x without crop mode fine.

I also started seeing these stuck pixels pop up in 1:1 5x liveview real time iso3200 1/50 that wouldn't be there in normal 5x zoom real time. I think its Just in the liveview. I didn't try to see if it would appear in RAW file.  Focus box was in was centered in both cases. I feel like this has been covered and I appollowgize if it was.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: BBA on April 22, 2017, 04:13:32 PM
Thanks a lot for the groundbreaking work !!! Keep going !!!!

[Deleted].
I don't know if it can help
...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 22, 2017, 05:16:51 PM
Wow FYI everyone I just did a 5div 4k vs 5diii 3k and 5d3 3k is sooooo much sharper and much more cinematic.

Same for 1080 5diii looks a lot better! I can't believe 2012 5diii has better video quality.

I guess the real test will be when 5div gets RAW video.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: BBA on April 22, 2017, 05:32:03 PM
Very nice !

Restarted everything from new download.
Loaded only the mlv_lite.mo module
Everything seems OK !
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 22, 2017, 06:13:38 PM
@a1ex Tested the latest build it's solid.. Only got one crash when I was using slowmo 45p / 48p, but when I switch to 50p / 60p mode it seems to work fine.  What's the difference between those two slowmo modes other than fps because on the 50p / 60p mode I was able to record @ 48p?   

@hjfilmspeed Awesome didn't think the 5D III could be sharper than the Mark iv in 3k, and definitely the motion in 3K is very cinematic I posted about that.  All I'm gonna say is a1ex got the keys.. keys.. keys..  Can you share a short clip of each camera?  Did you zoom in 200% - 400% to pixel peep. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 22, 2017, 06:47:31 PM
@goldenchild9to5 I'll post my quick test but before I hard conclude this I'll have to give the 5div a more suitable test.
I zoomed in and its really hard to to make a conclusion here but in both cases I gave the 5div the advantage especially with autofocus. First 2 shots are 1080 shots 3 and 4 are the higher resolution.
I tried to grade the RAW fairly.
See if you can tell what's what. I like the RAW better for sure.
If you download it it's a Cineform AVI in UHD
Also keep in mind I can only export UHD from Resolve.
https://vimeo.com/214307580
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 22, 2017, 08:38:52 PM
@hjfilmspeed You are definitely right the 5D III images looks better especially in 3K..
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 22, 2017, 09:41:08 PM
What's the difference between those two slowmo modes [slowmo 45p / 48p and 50p / 60p] other than fps because on the 50p / 60p mode I was able to record @ 48p?

FPS and vertical resolution. The 48p mode is really pushed to the limit (already discussed), while the 45p should have a little more headroom.

The cameras shutter speed was 1/50 but when zoomed + fps override it showed 1/27 or 1/37 or something like that.

Solved, it was just an oversight from my side.

Also fine-tuned the preview behavior: you can use a long half-shutter press to change between framing and real-time (e.g. if you normally use the centered real-time preview in x5 zoom, but you also want to check the framing every now and then, just press half-shutter for 1 second or so).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 22, 2017, 09:45:53 PM
@a1ex Thanks got it, didn't read up about that.   
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 22, 2017, 10:20:08 PM
Solved, it was just an oversight from my side.

Also fine-tuned the preview behavior: you can use a long half-shutter press to change between framing and real-time (e.g. if you normally use the centered real-time preview in x5 zoom, but you also want to check the framing every now and then, just press half-shutter for 1 second or so).
This is incredible! Soooo impressed!

Also I just replaced my test with a much better one

https://vimeo.com/214307580
This means Either Canons 5Div 4K is actually 3K
or
That we get to call 5D3 ML 3K  ... 4K
4 Clips
2 are 5div 1/100 f11 ISO 400 Sigma Art 35mm 4k
2 are 5Diii 1/50 f11 ISO 400 Sigma Art 35mm 3k  ML
14bit lossless using 4-21-2017 4K Crop Rec build
(I underexposed the 5Div because I was clipping highlights at 1/50 shutter)
Both rendered to Cineform UHD and both were precisely focussed on the crosshatch on the color chart.
Can you tell which is which.
Download and zoom in. It's a UHD Cineform AVI
 5Diii was quickly graded to match Canon's neutral picture style of the 5div

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 22, 2017, 10:54:27 PM
Sorry to be a party pooper but can anyone else confirm that this is reproducible (it seems to be on my end w build 2017-04-22 for 5D3.123) if one were to record in 3504x1400 resolution (and only in that resolution which is 2.50:1)?

It seems to produce a pattern of corruption after each 36th DNG frame from 14-bit lossless shot in 24p.

(https://c1.staticflickr.com/5/4183/33820054770_a4d51abbce_n.jpg) (https://flic.kr/p/TwyC1A) (https://c1.staticflickr.com/3/2934/34204130195_ca8604c418_n.jpg) (https://flic.kr/p/U7v7fF)
(https://c1.staticflickr.com/5/4172/33361447354_46a53aea0c.jpg) (https://flic.kr/p/SQ391b)

Here's an original corrupted 36th DNG: https://bitbucket.org/DeafEyeJedi/magic-lantern/downloads/5D3.123_3504x1400_corruption%3F.dng

Settings used in ML:
(https://c1.staticflickr.com/3/2834/33393407053_c0143f495d.jpg) (https://flic.kr/p/SSRWvt)

SO far in other resolutions seems to be getting better results other than the random first DNG's being either black (blank) or half screen corruptions, etc from each certain MLV's. Gotta troubleshoot some more on this to narrow down the culprit. Don't think it's random at all.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 22, 2017, 11:02:01 PM
@hjfilmspeed Great test.. This one was pretty hard to tell the difference.  Is the 5D III scenes 2 & 4?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 22, 2017, 11:30:46 PM
@hjfilmspeed Great test.. This one was pretty hard to tell the difference.  Is the 5D III scenes 2 & 4?
@ goldenchild9to5 did you download and zoom in? Which looks sharper or which do you prefer?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 22, 2017, 11:44:50 PM
@DeafEyeJedi: does it help if you reduce the vertical resolution from the crop_rec submenu a bit? (e.g. instead of 1536, try 1500, 1450, 1400)

If you can reproduce it consistently without the resolution override, try the following patches

edit: nevermind; had some hypotheses, but rejected all of them.

Reproduced.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hindra on April 22, 2017, 11:48:37 PM
I was able to replicate  the 19 FPS bug in the menu I reported a while baclk. If I forget to turn FPS Override off, turn on Crop Mode module for UHD recording, enter canon menu then go back into ML menu, when I turn off FPS Override it defaults back to 19 FPS. If FPS Override is off when I turn on Crop Mode, everything is fine.

(http://i.imgur.com/XvzW1N6.jpg)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hindra on April 23, 2017, 12:01:56 AM
This is incredible! Soooo impressed!

Also I just replaced my test with a much better one

https://vimeo.com/214307580
This means Either Canons 5Div 4K is actually 3K
or
That we get to call 5D3 ML 3K  ... 4K
4 Clips
2 are 5div 1/100 f11 ISO 400 Sigma Art 35mm 4k
2 are 5Diii 1/50 f11 ISO 400 Sigma Art 35mm 3k  ML
14bit lossless using 4-21-2017 4K Crop Rec build
(I underexposed the 5Div because I was clipping highlights at 1/50 shutter)
Both rendered to Cineform UHD and both were precisely focussed on the crosshatch on the color chart.
Can you tell which is which.
Download and zoom in. It's a UHD Cineform AVI
 5Diii was quickly graded to match Canon's neutral picture style of the 5div




1 and 3 are much sharper when viewed at 200%
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 23, 2017, 02:28:03 AM
Sorry to be a party pooper but can anyone else confirm that this is reproducible (it seems to be on my end w build 2017-04-22 for 5D3.123) if one were to record in 3504x1400 resolution (and only in that resolution which is 2.50:1)?

Narrowed down and solved; excellent catch @DeafEyeJedi!

That's how a proper bug report looks like - it contains all the details to allow others to reproduce it. This bug was quite hard to trigger, given all those reports with no corrupted frames.

Also got 4096x3072 working at 12 and 12.5 FPS (24/25 in Canon menu) and did some more fixes for very large resolutions. Even if you only get a few frames, I still find these modes useful with half-shutter trigger and pre-recording (e.g. for taking photos of really unpredictable subjects).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: dfort on April 23, 2017, 04:18:58 AM
@dfort: yes, I've looked into it because of your report, but it was my explicit request to try the bad commit, not a regular user trying it for e.g. timelapse or whatever. Also, this branch has a lot of stuff that can be broken into smaller branches and back-ported to all other models, in the main builds (already started with this PR).

Oops, sorry. I bit off more than I could chew when I came across that bug and it was just easier for me to switch to an earlier (smaller) branch. I'd like to get lossless compression working on other cameras because that seems doable while 4k on an EOSM is pie in the sky. Though I wouldn't rule it out considering how quickly ML development has been progressing lately.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: lanhl on April 23, 2017, 11:16:28 AM
Thanks for the wonderful work. I Got some problem when update to the new buuld, don't know how to fix it.

My camera is 5D3, 1.2.3

I have installed the Apr-04 build before, it works, and then update to the newest Apr-23 build, then I got this scene on top.
(http://i.imgur.com/czeNgzz.jpg)
I tried to Uninstall, and format CF card, Install again, disable all modules, but got the same scene. then I tried degrade to the earlier build, the same, But the Apr-04 and Apr-06 build works fine.

Can any body know how to fix this? Thanks a lot.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: jankrueck on April 23, 2017, 11:35:11 AM
sorry for beeing an asshole. But did you read the screenshot or any page of this tread, yet?  :D
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: bazidl on April 23, 2017, 03:21:04 PM
I'm getting frame and block size mismatch errors. I saw a few other people were getting various versions of this error as well but no responses with a possible solution. I know it's related to the 14 bit - lossless format as I don't get the message when I turn lossless off.

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 23, 2017, 05:21:45 PM
Testing April 23th build 5D3 113.
When I change to a 192050p3x3 mode with fps override on, and try to switch fps override to off, camera freezes and I get Error80. Need to pull battery.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 23, 2017, 08:24:27 PM
@hjfilmspeed Check it out Zoomed in to 200% & 400% Scenes 1 & 3 is a tiny tiny bit sharper than Scenes 2 & 4.  But here is the thing the tree behind the color checker seem a bit sharper in Scenes 2 & 4. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Ilia3101 on April 23, 2017, 09:08:55 PM
@lanhl Follow what it says on the screen. There was a dangerous bug in April 04 build so DON'T go back to it, and the new ones are detecting if that bug has affected your camera. It's Magic Lantern looking out for you ;)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Ali Oliya on April 24, 2017, 08:25:29 AM
Hi,

I used mlv_dump in mac for creating dngs. but unable to open them in After Effects CC 2017. Photoshop opens files with no problem. but AE gives me this error:  "after effects error: the file formay module not parse that file. (45::35)"
Title: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 24, 2017, 08:58:15 AM
@Ali Oliya - is the first DNG in your folder blank or corrupted in any way?

If so, try deleting it and try again which should open up fine in AE. Have had this happened to me in the past.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 24, 2017, 09:02:11 AM
I was away for some time. Trying latest build.
Way much solid!
The exclusions of many combinations makes things more targeted.
Thank you for the hard work and constructive development
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Ali Oliya on April 24, 2017, 09:11:46 AM
@DeafEyeJedi  yes it was corrupted. i deleted it. still doesn't open. but the error message changed to: After Effects error: Photoshop file format error (-1). (45 :: 35)
Title: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 24, 2017, 09:40:41 AM
Hmm would you mind using @Danne's latest cr2hdr.app to spit out original samples (#12 under mlv_dump menu) and upload the zip file for us to check out?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 24, 2017, 11:34:55 AM
@hjfilmspeed Check it out Zoomed in to 200% & 400% Scenes 1 & 3 is a tiny tiny bit sharper than Scenes 2 & 4.  But here is the thing the tree behind the color checker seem a bit sharper in Scenes 2 & 4. 
1 and 3 are mark iii
Hmmm Ill double check my lens details to see if f-stop was different. It could be possible that focal planes where not exact. I focused both on the crosshatch with autofocus. but in any event at 400% we should be seeing a HUGE difference in sharpness here. 5dIv has 1000 more pixels. Any way I don't want to go more off topic then I have.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 24, 2017, 12:18:24 PM
Any dev think if there is any possibility to get some signal through hdmi in the new full res live view mode?... just for framing and focussing, don't mind if res is low (VGA)...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Ali Oliya on April 24, 2017, 12:49:05 PM
@DeafEyeJedi - i used cr2hdr app from Danne. But it doesnt let me open the MLV file at all! .mlv file is grayed out....(same as rawmagic)

ps. i used crop rec 3k-4k 14 April expermiment build btw. so far the only software that is able to open the .mlv files from this build is MLV_Dump (for mac). (raw2cdng opens but creates pink dngs!)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 24, 2017, 01:10:34 PM
Greyed out? Could you share a screenrecording or a screenshot? Maybe the LOG.txt file?
Maybe post about it here.
http://www.magiclantern.fm/forum/index.php?topic=15108.100
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Ali Oliya on April 24, 2017, 03:13:53 PM
@Danne - yes, i can not choose the file. it can't be selected:

(https://preview.ibb.co/ebuBmQ/Screen_Shot_2017_04_24_at_5_33_34_PM.png) (https://ibb.co/j0zcRQ)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Danne on April 24, 2017, 04:02:22 PM
You don´t choose files you choose the folder containing files. If you press choose in this folder a menu will pop up and you take it from there.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Ali Oliya on April 24, 2017, 04:22:48 PM
@Danne - Thank you! Sorry, it was the 1st time using this app! done. fixed. now i can open .dngs in AE without problem.

@DeafEyeJedi Thanks for your help
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Quentin on April 24, 2017, 06:37:12 PM
On 12 April 2017 I took several shots.
I was recording with that days latest build, 1920x800 60FPS 14bit lossless with 5dmk3.
On some of the shots I took, on the top of the frame there is a horizontal sharp edge that flickers, about 100 pixels away from top

Here is a link
https://drive.google.com/open?id=0B-d8ARtc7xwWT3BwbkdQYjdaLUE

(I am not aware if this is fixed already)
Title: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 24, 2017, 07:16:22 PM
Quote from: a1ex
Narrowed down and solved; excellent catch @DeafEyeJedi!

That's how a proper bug report looks like - it contains all the details to allow others to reproduce it. This bug was quite hard to trigger, given all those reports with no corrupted frames.

Great to hear. Thanks for the acknowledgement!

Quote from: a1ex
Also got 4096x3072 working at 12 and 12.5 FPS (24/25 in Canon menu) and did some more fixes for very large resolutions. Even if you only get a few frames, I still find these modes useful with half-shutter trigger and pre-recording (e.g. for taking photos of really unpredictable subjects).

Oh absolutely this will be useful. Especially for birding, sports photography & whatnot. This is incredible and should be a bit more forgiving than Canon's stock full burst mode which is only 6 frames per sec for 5D3.  :P

1 and 3 are mark iii
Hmmm Ill double check my lens details to see if f-stop was different. It could be possible that focal planes where not exact. I focused both on the crosshatch with autofocus. but in any event at 400% we should be seeing a HUGE difference in sharpness here. 5dIv has 1000 more pixels. Any way I don't want to go more off topic then I have.

For those who havant seen them up-close w comparisons yet in between 200% & 400% (Thanks @hjfilmspeed for sharing them!):

#1) 5D3 RAW 3K 1.85:1 (upscaled to 4K in post)
(https://c1.staticflickr.com/3/2827/34244929975_b78ee13b4d.jpg) (https://flic.kr/p/Ub7dBa) (https://c1.staticflickr.com/3/2868/34244928415_5c1066c961.jpg) (https://flic.kr/p/Ub7d9g)

#2) 5D4 MJPEG 4K
(https://c1.staticflickr.com/5/4169/34114706071_ae086da248.jpg) (https://flic.kr/p/TYAMzr) (https://c1.staticflickr.com/5/4178/34244928795_8f2fa0d461.jpg) (https://flic.kr/p/Ub7dfP)

#3) 5D3 RAW 3K 1.85:1 (upscaled to 4K in post)
(https://c1.staticflickr.com/3/2806/33403755734_c358e665fa.jpg) (https://flic.kr/p/STLYP1) (https://c1.staticflickr.com/3/2857/33403754414_df65c4273c.jpg) (https://flic.kr/p/STLYqf)

#4) 5D4 MJPEG 4K
(https://c1.staticflickr.com/3/2876/34244931825_d2e4cb6740.jpg) (https://flic.kr/p/Ub7ea4) (https://c1.staticflickr.com/5/4166/33403754644_21c9204576.jpg) (https://flic.kr/p/STLYud)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 24, 2017, 07:47:56 PM
@DeafEyeJedi the 5diii was actually 3k 3072 1.85:1 but technically upscaled at export.
Title: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 24, 2017, 08:12:14 PM
Thanks @hjfilmspeed -- fixed! (It's even more flabbergasting to know that the 5D3 still looks sharper even when upscaled to 4K)  :o
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 24, 2017, 08:31:48 PM
Thanks @hjfilmspeed -- fixed! (It's even more flabbergasting to know that the 5D3 still looks sharper even when upscaled to 4K)  :o


DeafEyeJedi Thank you for posting those crops!
Actually, you might as well call the the 5DIII 3K - 4K. Why not canon apparently seems to think it is okay. I thought to myself, maybe its the lens.. then I was like no way, the Sigma Art prime is design to resolve way high resolution.
It could be the 422 vs the .... 444 or what ever MLV 14Bit lossless is.
Regardless, this build gets better every time. Can't wait to see what the future holds!

Oh I meant to ask this, should we be using 123 vs 113 for 5d3? It was my understanding that 113 was fine unless you need hdmi features and AF @ f8. Is this still true?
Or should we be testing 123?
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: giarcpnw on April 24, 2017, 08:33:59 PM
Hey all, how are you guys addressing the Vertical Lines or other pixel issues with these new builds? I'm on OSX. Martin Hering had created a great mlv to cng converter app called Footage that addressed them but he seems to be letting it time bomb in the 15th of May to pursue a revenue generating path.  So now we're back to square one.

MLVDump does not fix them.  Is there another app that i'm not aware of that does?

Thanks,

Craig
Title: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 24, 2017, 08:44:32 PM
...Oh I meant to ask this, should we be using 123 vs 113 for 5d3? It was my understanding that 113 was fine unless you need hdmi features and AF @ f8. Is this still true?
Or should we be testing 123?
AFAIK 113/123 should be identical ever since @a1ex merged these 'two firmwares under one branch' though correct me if I'm wrong since this is experimental.

...he seems to be letting it time bomb in the 15th of May to pursue a revenue generating path.  So now we're back to square one.

That's some hogshit, eh?

MLVDump does not fix them.  Is there another app that i'm not aware of that does?

Care to share some original short samples @giarcpnw? (use @Danne's cr2hdr.app (https://www.magiclantern.fm/forum/index.php?topic=15108.0cr2hdr.app) if you haven't already)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: hjfilmspeed on April 24, 2017, 09:02:52 PM
@DeafEyeJedi thank you!!!

Also has anyone tried these settings? 4/23/2017 build
Crop Rec with 5x zoom 3.5k  enabled
And 12bit lossless with 3072 2.35:1 ?
It's very stable to me! Continuous with a good card!
And you can hit the DOF preview button or half shutter to zoom in with real time to focus!
I saw this mentioned but didn't try it yet. This is awesome!
I feel like real time framing preview of 3k is so close now!
(Coding wise it's probably not)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: giarcpnw on April 24, 2017, 09:05:26 PM
Care to share some original short samples @giarcpnw? (use @Danne's cr2hdr.app if you haven't already)

cr2hdr didn't seem to fix when i tried last week. unless it's been upgraded since.


The are DNGs from MLVdump, you can see the banding/lines in the mids. They get worse once you adjust exposure and sharpness. These were shot 14bit lossless with the April 10th build I believe. My understanding is this is inherent in the raw data and the converters are coded to adjust for it. I know there was an early post in this thread that said most of the apps or command line converters had not been adjusted to address this with the new 14bit 4k build's mlvs. Just wondering if they had yet or I missed one. Martin added it to his at my request last week, but...

https://www.dropbox.com/sh/a4158kqm85rvime/AADYDd6H8PyjYyGE1fXqKpeza?dl=0

Thanks,

C

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 24, 2017, 09:12:03 PM
Actually could you please just use cr2hdr.app (press 'm' to get in mlv_dump menu then press '12' to start sample processing). Need the originals.

Be sure to use a temporary Folder and include MLV inside since the app can only process from within Folders and not directly on the MLV file itself.

Also try to use the latest experimental builds whenever possible since it's always evolving for greatness.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: giarcpnw on April 24, 2017, 09:32:09 PM
Actually could you please just use cr2hdr.app (press 'm' to get in mlv_dump menu then press '12' to start sample processing). Need the originals.

Be sure to use a temporary Folder and include MLV inside since the app can only process from within Folders and not directly on the MLV file itself.

Also try to use the latest experimental builds whenever possible since it's always evolving for greatness.

Same folder: you'll see the zip pack.
https://www.dropbox.com/sh/a4158kqm85rvime/AADYDd6H8PyjYyGE1fXqKpeza?dl=0

I have the newest April 23rd build. I just haven't shot any new footage since the April 19th. It was there in the 19th too. These are the MLVs I have with me at the moment.

Thanks,

C

Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 24, 2017, 09:35:57 PM
Oh I meant to ask this, should we be using 123 vs 113 for 5d3? It was my understanding that 113 was fine unless you need hdmi features and AF @ f8. Is this still true?

Correct.

I'm currently using 1.2.3, but that's because it has more quirks (usually, if something works on 1.2.3, it's likely OK on 1.1.3, but in the other direction, there are a lot of exceptions...)

The notes from 1.1.3 thread (http://www.magiclantern.fm/forum/index.php?topic=14704) are still valid.

AFAIK 113/123 should be identical ever since @a1ex merged these 'two firmwares under one branch' though

That means both ML versions are compiled from the same codebase (but each one is still tweaked to account for Canon firmware quirks - just like all other ML ports).

MLVDump does not fix them [vertical lines].  Is there another app that i'm not aware of that does?

I've explained here (http://www.magiclantern.fm/forum/index.php?topic=18443.msg183653#msg183653) what needs to be done for other converters (including mlv_dump). I'm not really familiar with compressed formats, and there are people in the ML community who can do this faster than me (hint, hint).
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: DeafEyeJedi on April 24, 2017, 09:41:03 PM
Also has anyone tried these settings? 4/23/2017 build
Crop Rec with 5x zoom 3.5k  enabled
And 12bit lossless with 3072 2.35:1 ?

Confirmed after recording both takes for over 9 minutes each now @ ISO 3200 in 24p. Green icon all the way. Though it felt a bit toasty @ 53-54C towards the end, ha! :P
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: giarcpnw on April 24, 2017, 11:21:37 PM
Quote
I've explained here (http://www.magiclantern.fm/forum/index.php?topic=18443.msg183653#msg183653) what needs to be done for other converters (including mlv_dump). I'm not really familiar with compressed formats, and there are people in the ML community who can do this faster than me (hint, hint).



If only i had studied computer science in college instead of theater, I might be able to fix this myself. Instead, I'll have to recite shakespeare to myself while I wait for someone else with programming knowledge.

C
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 25, 2017, 01:04:13 AM
@hjfilmspeed 5D III is really a force to be reckon with, can't wait for liveview to have a smooth feed  @DeafEyeJedi great screen capture..
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: csound on April 25, 2017, 02:38:39 AM
Can someone please point me to information on procedure to fix null pointer bug from April 4 build. Screen alert on camera indicates to prepare logs and go to freenode on IRC - what happens from there? Sorry, unfamiliar with all this.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 25, 2017, 02:39:39 AM
Just tried the new Davinci Resolve 14 Beta with 5D III 3K footage man this new update is smooth.  I've noticed as well that It might be processing dng's better, seems to resolve images detail a lot better.  If anybody else used it let me know if you are experiencing the same thing.   
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 25, 2017, 02:46:24 AM
@csound Contact a1ex, he will guide you. 
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: Oedipax on April 25, 2017, 03:06:31 AM
Quote from: goldenchild9to5
Just tried the new Davinci Resolve 14 Beta with 5D III 3K footage man this new update is smooth.  I've noticed as well that It might be processing dng's better, seems to resolve images detail a lot better.  If anybody else used it let me know if you are experiencing the same thing.   
Same experience here - realtime playback on my 3k 2.39:1 14bit CDNG files, up to a few nodes. Extremely pleased with the new Resolve so far.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: goldenchild9to5 on April 25, 2017, 04:06:50 AM
@Oedipax I'm glad that you are getting the same results so I'm not the only one.  @a1ex new build is awesome thank you will be testing more tonight.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 25, 2017, 11:05:13 AM
Quote
Quote from: a1ex

    Also got 4096x3072 working at 12 and 12.5 FPS (24/25 in Canon menu) and did some more fixes for very large resolutions. Even if you only get a few frames, I still find these modes useful with half-shutter trigger and pre-recording (e.g. for taking photos of really unpredictable subjects).

Quote from: DeafEyeJedi
Oh absolutely this will be useful. Especially for birding, sports photography & whatnot. This is incredible and should be a bit more forgiving than Canon's stock full burst mode which is only 6 frames per sec for 5D3.  :P

A dumb question maybe... but... someone are getting realtime preview in this mode? How can you focus and framing for sports and birds?... I think I'm missing something...
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 25, 2017, 11:17:07 AM
A dumb question maybe... but... someone are getting realtime preview in this mode? How can you focus and framing for sports and birds?... I think I'm missing something...

Want to help? Grab ADTG_GUI and QEMU and start investigating!

All you need to do is to understand how Canon configures the image processing path for LiveView. Here's a starting point. (http://www.magiclantern.fm/forum/index.php?topic=2388.msg183168#msg183168)
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: pc_bel on April 25, 2017, 11:45:01 AM
I'm taking a look to the Starting Point... and I don't know what to do... I have no knoweledge in any coding languaje. For me is like cheneese  :( :-[ I'm sorry but if I can do something more just let me know.
Title: Re: crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView
Post by: a1ex on April 25, 2017, 12:25:05 PM
The difficult part is simply understanding what Canon code does (and what we have to change to fix the preview).

Those numbers from the logs and adtg_gui are like Chinese to all of us; the trick is to fiddle with them until a tiny part of them starts to make some sort of sense. That's how we figured out how to reconfigure the sensor to 4K in the first place ;)

Some useful tricks: comparing all those numbers in different video modes (adtg_gui has a simple register diff tool), recognizing values that could look like resolutions, logging the I/O activity in QEMU, or analyzing the still photo mode first (it's simpler).

And, of course, don't give up after reading a single post. Make yourself comfortable with what's happening in the Reverse Engineering area. Some relevant threads are linked in the first post; make sure you can follow them.

Of course, you will need some basic programming knowledge.

Bottom line: the real-time preview is not implemented because... I have little or no idea about how Canon configures the image processing path.