crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView

[banertop]

Hi guys,

hate to ask, ....because i am pretty sure that someone asked this before....but i did not manage to find it......

Is it possible..., or why it is NOT possible, to hack somehow higher resolutions recording in h264 in crop mode....?

I am very happy with 5d iii raw...more than happy, actually i sold my gh4, and did not buy gh5 , just to buy 5d iii again, after all new achievement of ML team.....but, sometimes it could be nice to have decent h26h, for some projects....

Again, thanks for all your work, and all that magic you give to us 

[hjfilmspeed]

Hi guys,
hate to ask, ....because i am pretty sure that someone asked this before....but i did not manage to find it......
Is it possible..., or why it is NOT possible, to hack somehow higher resolutions recording in h264 in crop mode....? 

@Banertop I'm not sure this is the correct answer but it takes a lot of processing power to take the raw data and compress it to the very complex algorithms of h.264. The 5diii can do it at 1080p but I doubt high resolution is possible. RAW data is just RAW data taken from the sensor and sent straight to the card with maybe some metadata and other bits of info attached. It takes less battery power then h.264 too. The 10,12,14 bit compression that ML is using is not nearly as complex because it doesn't need to be.

[banertop]

Thank you very much...

i thought that it is something about processing power.....

but, the good thing is the fact that i really now rarely need h264....raw is raw! 

thanks again

[ErwinH]

Shameless cross post: https://www.magiclantern.fm/forum/index.php?topic=16650.msg192161#msg192161 sound recording with mlv_lite based on the crop_rec_4k branch.

[a1ex]

I have tried 16oct crop_rec build on 100D. No problem with RAW recording (12bit lossless most the time) at ISO100.
At ISO>100 most time "Raw detect error" (usually first ISO>100 clip correct and then further tries with this error) - was the same situation with previous builds.

Open the console (Debug menu) to see the reason you've got this error.

[dfort]

I've been trying to track down this issue and it is getting pretty complicated for me. First of all, I reported it here and here. I was trying to figure out when it first started happening and it looks like since mlv_rec has been useable on the crop_rec_4k branch--the unified branch doesn't exhibit this bug. Only the cameras that can do Movie crop mode are affected -- a.k.a. the 600D crop hack, so that should include EOSM/700D/100D/650D. I've been able to reproduce it on both the EOSM and 700D.

• Start with a fresh download or delete the ML/SETTINGS directory from the sd card.
• Turn on only mlv_rec
• Activate Movie crop mode and RAW video (MLV)
• Try a test recordings, any image size or bit option

Yes, it works but now is when it starts to get weird. Reboot the camera and try it again. It won't record with Movie crop mode turned on. Other modes work fine though.

Something is going on with the configuration files? These are the EOSM configuration files:

MAGIC.CFG

Code: [Select]

# Magic Lantern crop_rec_4k.2017Oct16.EOSM202 (8ee7858f0d7e (crop_rec_4k) tip)
# Built on 2017-10-16 20:24:23 UTC by jenkins@nightly
# Configuration saved on 2017/10/24 14:21:13
beta.warn = 24
menu.first = -4
disp.mode.x = 199

MLV_REC.CFG

Code: [Select]

# Config file for module mlv_rec (MLV_REC.MO)

mlv.video.enabled = 1
mlv.write_speed = 3911

MENUS.CFG

Code: [Select]

0 3FFFBE8F 3FE9C8FD Movie\Movie crop mode
0 3F7FBE87 3F68BA2E Movie\RAW video (MLV)

I also posted the saved settings from EOSM and 700D on my Bitbucket downloads page, file name Movie_crop_mode_bug.zip.

Yeah, not high priority because it looks like mlv_lite might soon be getting sound capabilities and mlv_rec could be phased out but I thought I'd report this anyway.

[a1ex]

No idea what's going on; a video showing the issue might help (including the steps required to bring it back to a "working" state).

If it's really one of these config entries, it should be easy to narrow down (by deleting only some of these files or entries).

[jacekp]

Open the console (Debug menu) to see the reason you've got this error.

Most of the time after switching to ISO>100 and half press shutter button (or after 1 succesfull record) whole console in non-stop messages (with different numbers in brackets)":

"Black 3/5: stdev too large (9745/100, ref 1900/100)
"Black 3/5: stdev too large (8944/100, ref 1790/100) ..."

[a1ex]

Could be either some offsets being still wrong, or some hot pixels in the OB area.

To diagnose, download raw_diag.mo and get a screenshot with the "OB zones" analysis. If that one looks good, try taking a few simple silent pictures (with silent.mo) in the affected video mode. You may have to turn off raw recording, as it preallocates the entire memory.

[jacekp]

Could be either some offsets being still wrong, or some hot pixels in the OB area.

To diagnose, download raw_diag.mo and get a screenshot with the "OB zones" analysis. If that one looks good, try taking a few simple silent pictures (with silent.mo) in the affected video mode. You may have to turn off raw recording, as it preallocates the entire memory.

Did OB Zones in ISO100 and ISO1600.




Could not take silent picture both in ISO100 (which should work) and ISO1600.
Tried silent mode: "Simple, DNG" and "Simple, Lossless DNG". Tried in photo mode 'Manual' and movie mode 'Manual'. Tried without mlv_lite and with mlv_lite but disabled "RAW recording".
Some console outputs in attachment: after camera start, after silent picture in photo mode and twice in Movie mode.



When trying  again silent mode picture i get all the time only (even in ISO 100):
"[SRM] alloc all buffers
srm_malloc_suite =>0
srm_malloc_suite(2)..."
When switch to ISO > 100 i usually get additionally my usual messages "Black 3/5: stdev too large......"
no pictures.

[a1ex]

The zones look good.

If the silent module doesn't work, try enabling Dump RAW Buffer in raw_diag. That one should work with less memory.

edit: trying to reproduce the memory issue - does the 100D really have only 1 SRM buffer? (check with Debug - Free Memory).

[jacekp]

The zones look good.

If the silent module doesn't work, try enabling Dump RAW Buffer in raw_diag. That one should work with less memory.

edit: trying to reproduce the memory issue - does the 100D really have only 1 SRM buffer? (check with Debug - Free Memory).

Free Memory:




Dump RAW Buffer (only raw_diag module enabled with no analysis). ISO 100 image and ISO 800 image (with ISO > 100 got my usual "Black 3/5: stdev too large......" and no image, but after restart succesfully got one):



hosting jpg


Workaround Note: Now I know when I can do succesfull ISO>100 image. When the Console is not throwing "Black 3/5: stdev too large......" messages I can succesfully record ISO>100. After record ISO>100 it usually throws "Black 3/5: stdev too large......" and when I swich to ISO 100 it stops throwing errors. I can then go back to ISO>100 and If I don't do many other clicks it remains without "Black 3/5: stdev too large......" and I can do second ISO>100 record...

[a1ex]

With 2 SRM buffers I'm unable to reproduce this screenshot: http://ibb.co/eD4A36

Was the free memory screenshot taken in LiveView, in the same video mode?

Can you also upload the DNG? (I'm interested in looking at the invisible areas).

[jacekp]

With 2 SRM buffers I'm unable to reproduce this screenshot: http://ibb.co/eD4A36

Was the free memory screenshot taken in LiveView, in the same video mode?

Can you also upload the DNG? (I'm interested in looking at the invisible areas).

That screenshot http://ibb.co/eD4A36 was in Photo mode. In "LiveView [Manual] Movie mode" was http://ibb.co/gj78qm . Free memory screenshot was also in "LiveView [Manual] Movie mode".

Today I made a ML reinstall and enabled only raw_diag and silent. This time I succefully made silent pictures (as usual: LiveView, [M]-Manual, MovieMode), but after silent picture in ISO>100 (1600), I got my usual "Black 3/5: stdev too large......" and couldn't make next picture, console:




DNG-s:
- previous Dump RAW Buffer in ISO800 (I have lost ISO100 image, sorry)
https://magicznyfilm.com/img/DumpRAWBuffer.DNG
- current Silent picture in ISO100
https://magicznyfilm.com/img/silent_ISO100.DNG
- current silent pictre in ISO1600
https://magicznyfilm.com/img/silent_ISO800.DNG

[a1ex]

Nothing wrong with the samples...

Are you able to compile ML? There is an "if (0)" in raw.c - turn that into "if (1)" to save all frames that caused the error.

[lostfeliz]

I've been prepping for a 3k shoot this Monday. I've been doing some exposure tests. Depending on the mode I'm in, the spot meter changes the reading of a gray card. It also changes in real terms when I measure the card in Lightroom. I'm not particularly needing these inconstancies to go away. Mainly I just want to nail my exposure so I have good highlight details but don't have to pull up shadows until they become noisy. I tested the gray card in four different settings and got sometimes some very different results.

Click on image to see the whole spreadsheet:




Now here's the strangest part: I had exposure on the 5diii set to Manual. FPS Override was on and set to 23.976. ISO was 320. AND ... AND shutter speed was set to 1/60. But when I looked at the meta-data in Lightroom, the 3k crop DNGs said they was exposed at 1/50. Is the meta-data wrong or is the shutter speed operating at a different speed than indicated on the camera? Anyone know?

Lastly, I noticed in Lightroom that one of the process options for the video DNGs was "embedded". Anyone have info on how this choice effects the image?

[a1ex]

Generally, you should look at the shutter speed printed by ML. That one is computed from low-level registers and is not rounded too much. Exact value in Expo menu, with 3 decimal places. Same from the frame rate - actual value is on the top bar, also with 3 decimal places.

If you can get the desired frame rate without FPS override, it's best to keep it off. This feature doesn't play nice with crop_rec, and it's not easy to fix (they both touch the FPS registers).

[lostfeliz]

If you can get the desired frame rate without FPS override, it's best to keep it off. This feature doesn't play nice with crop_rec, and it's not easy to fix (they both touch the FPS registers).

When I don't use FPS override, the ML settings say 29.776 when I punch in to 5x (whether the Canon settings are 1080/24 or 1080/30). And it actually records at 29.776. So, I don't know how to record 3k at 23.976 without FPS Override.

[Tgra]

Hi guys, great work on these new exciting features -as always.

I started playing with the 2.5k compressed raws the other day and tried to put it to the test: harsh exposure contrasts and low light... it held up pretty well!! Went up to iso 1600 on this and manage to bring back +0.75 usable(ish - it’s a bit mushy)  exposure with resolve. here’s the test:

I had a question on the other hand: I happen to shoot interviews sometimes and wanted to see if smaller bitrates would help reduce file size (my interviews are usually in contrôles environments and don’t require much color correction). From what I gathered, both 10bits compressions don’t have liveview -so far?- ... what I don’t understand is why the normal 10bits reduces files size by 70% (according to the menu) and the compressed 10bits only reduces it by 43% (I can’t remember the exact figures but it’s in those lines). How’s that so?

Cheers!

[Kanakori]

Hi everyone.
I've been recording in 1080p 48fps, but the liveview is too slow. Is there a way to get it playing realtime or just faster in general? I've tried disabling global draw and the only modules enabled are the raw video ones.

[Teamsleepkid]

shooting tomorrow. been using the build with sound and compressed raw but i noticed that something is wrong with it. some clips are fine others are split up and the have there own folders in resolve. the clips that are good don't look like folders they look like film strips. using danes switch to make the dngs. looking for a gold master for my shoot. probably doesn't exist but i thought id ask. thanks.

[D_Odell]

 Or is it nothing to worry about?

Still having this problem, or is it a problem? It remains locked down when using crop rec. Do anyone know if it affect performance?

[ali farahani]

When i set to Crop_rec 3k .. my  live view is with latency. With this situation i can not focus any moving objects. I s there any way to get real time liveview  in crop_rec?

[D_Odell]

When i set to Crop_rec 3k .. my  live view is with latency. With this situation i can not focus any moving objects. I s there any way to get real time liveview  in crop_rec?

Live view only work in crop rec if you shoot 3x 1080 1:1 then you have propper framing and live view. Otherwise (higher resolutions without crop) you use Frozen LV so you at least can frame CPU power and memory isn’t enough I guess..

[Wayne H]

Hi Ali, REAL TIME LIVEVIEW would indeed be the Magic Lantern holy grail, unfortunately it’s not currently possible, as for focusing you have one of two options,

You can use an external monitor and you’ll get the x10 zoom continuous feed displayed (no latency) perfect for focusing (only available in the 3.5K CROP MODE), or you can half press the shutter, and liveview will display the 10x zoom until you take your finger off the shutter button. So go into crop mode 3.5K CROP MODE, set your resolution you wish to shoot at, 3K or whatever, then compose your shot, half press shutter and then focus, finger off the shutter and hit record.