crop_rec on steroids: 3K, 4K, 1080p48, full-resolution LiveView

Started by a1ex, April 01, 2017, 11:15:41 AM

Previous topic - Next topic

0 Members and 7 Guests are viewing this topic.

D_Odell

Quote from: mageye on April 12, 2017, 11:13:17 AM
Try recording at ISO100
Positive results, now 25 sec. Thanks! So reaching (for example) 5 min, lower resolution seems to be the only way?
5D3 [size=6pt](OLPF removed)[/size] :: 1.1.3 :: Canon FD L Serie

mageye

At the moment yes. But the noise from the ISO causes problems for the compression algorithm. I am getting a (reasonably) reliable resolution of 3648x1216 but still not completely continuous!
5DMKII | 500D | KOMPUTERBAY 32GB Professional 1000x |Canon EF 50mm f/1.8 II | Samyang 35mm f/1.4 ED AS UMC | Canon EF 75-300mm f/4-5.6 III | Zoom H2 (4CH. audio recorder) | Mac OS X 10.9.2 | Photoshop CC | After Effects CC | Final Cut Pro 7

D_Odell

Quote from: mageye on April 12, 2017, 11:37:12 AM
At the moment yes. But the noise from the ISO causes problems for the compression algorithm. I am getting a (reasonably) reliable resolution of 3648x1216 but still not completely continuous!
I see, do you mind sharing your ML settings? Thanks!
5D3 [size=6pt](OLPF removed)[/size] :: 1.1.3 :: Canon FD L Serie

mageye

Crop mode UHD 1:1
RAW video ON, 3648x1216
Aspect 3:1
14-bit lossless


Within Canon menu:
Movie rec. size 1920 @ 24 ALL-I
Video System set to: PAL

That's it!
5DMKII | 500D | KOMPUTERBAY 32GB Professional 1000x |Canon EF 50mm f/1.8 II | Samyang 35mm f/1.4 ED AS UMC | Canon EF 75-300mm f/4-5.6 III | Zoom H2 (4CH. audio recorder) | Mac OS X 10.9.2 | Photoshop CC | After Effects CC | Final Cut Pro 7

Quentin

Today I was doing a studio shooting for 4-5 hours, several shots.
I had my camera plugged on electricity as well as the monitor.
5d mk3, 1.1.3
Experimental Build 10 April
I used 1920x800 60FPS 16:9 14bit lossless
All manual, no AutoETTR
While I was working on the shot, to adjust lights etc, Camera crashed and saw several messages.
Turn it OFF/ON fixed the problem.
The recording was not continuous but sufficient enough to record the useful action.
I noticed that the Aspect Ratio of the monitoring wasnt right.
Later, I saw that the monitoring the colours lost saturation and the Aspect Ratio was jumping from wrong to right.
I also noticed that although the ISO was set manually to 1250, on ML monitor was still 1600.
The temperature of the camera was 40 C


Markus


Monitor can't show the hacked bw preview on external monitor. And canons liveview does not give correct representation of ML enabled modes.

Iso in raw is only valid for real Iso values. 100,200,400,800 and so on. The values in between is just digital gain applied to  h264 compressed video.



Vegandelight

What kind of work flow are you guys using for the 1920x960 3x3 binning 14bit raw mode?

hjfilmspeed

Quote from: Vegandelight on April 12, 2017, 05:18:27 PM
What kind of work flow are you guys using for the 1920x960 3x3 binning 14bit raw mode?
In camera? Or post processing?
For post I used the RAWFlow app and just replaced the MLV_DUMP with the newest version and it processed the 14Bit Lossless just as expected. Drag n Drop.
RAWFlow - http://www.magiclantern.fm/forum/index.php?topic=13338.0
MLV_DUMP Update - https://builds.magiclantern.fm/experiments.html

Vegandelight

Quote from: hjfilmspeed on April 12, 2017, 05:55:28 PM
In camera? Or post processing?
For post I used the RAWFlow app and just replaced the MLV_DUMP with the newest version and it processed the 14Bit Lossless just as expected. Drag n Drop.
RAWFlow - http://www.magiclantern.fm/forum/index.php?topic=13338.0
MLV_DUMP Update - https://builds.magiclantern.fm/experiments.html

Nice, Premiere or DaVinci Resolve? The old DNG-files opeened right up in Premiere but these compressed ones does not.

hjfilmspeed

Quote from: Vegandelight on April 12, 2017, 08:13:55 PM
Nice, Premiere or DaVinci Resolve? The old DNG-files opeened right up in Premiere but these compressed ones does not.
DaVinci opened them just fine. I don't know premiere.
Its and extra step but you can always export from resolve. Resolve + DNG = incredible color options!

fkarpenko

Is there a chance we'll see 2k/3k/4k resolutions in 3x3? This is pretty amazing but without the massive crop factor it would be a dream come true.

a1ex

Quote from: a1ex on April 01, 2017, 11:15:41 AM
Consider this being a huge leap towards our next mind boggling goal:

8K RAW Video Recording!

Sample DNG from 5D Mark III, to show that our proof of concept is working:

8k.dng

All that's left is to divide the resolution by 3 on each axis ;)

DeafEyeJedi

Has anyone notice an issue re: Dual-ISO 14-bit lossless MLV seems to be spitting out with correct black levels but incorrect for original 14-bit MLV's. Could this be related to the new mlv_dump not corresponding properly to cr2hdr or at least confused with each other atm?



Anyway here are some short samples in two of each (138 MB spat out w ease from latest cr2hdr.app -- Thanks @Danne!) and all shot within crop_rec @ 3.5k: https://mega.nz/#!OwV2DTLC!rb39cVOrAlXXuSer2M6fw3vSuvMfiSG508hewIQJvmg
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

a1ex

New build posted.

Found a way to prevent Canon code from saving some of the settings at shutdown, including the settings block that caused issues earlier (reverse engineering notes here). While not perfect, if Apr04 had this safeguard, all the cases affected by the null pointer bug would have been caught. Still, there are ways to brick the camera, just a bit less likely to do so accidentally (and, as long as the bootloader is not erased, recoverable).

How it works: whenever a crash is identified, or whenever you open the battery door, Canon code no longer saves the usual setting groups at shutdown (RING and RASEN, if you look in the above link). There are still setting groups not covered by this safeguard (still looking into it, but so far, all the null pointer errors were in the RING group).

To test - the PAL/NTSC setting is in the RING group (and, to my knowledge, only saved at shutdown). You also have a dummy crash under Don't click me.

After a bit of battle-testing, I think this should be back-ported to all models.

Changes related to raw recording:
- double buffering is back (except at very high resolutions); didn't test much
- pre-recording indicator: mm:ss.f + ss.f
- full-res LV increases FPS when lowering vertical resolution (up to 5796x400 @ 48 fps)
- in 10x zoom, FPS override can be disabled by half-shutter to allow focusing
- minor fixes here and there (full list on the download page)

For preview with correct framing on external monitors, try Force HDMI VGA (not tested, but IIRC it used to help).

hjfilmspeed


goldenchild9to5

@a1ex Thanx for all the new updates.. Gonna give the new build a try tonight will post my results

hjfilmspeed

Going to update to new build and found this on card from April 10 Build
ML ASSERT:
RAW_IS_IDLE
at mlv_lite.c:584 (measure_compression_ratio), task shoot_task
lv:1 mode:3


Magic Lantern version : crop_rec_4k.2017Apr10.5D3113
Mercurial changeset   : 11f405b62b31 (crop_rec_4k) tip
Built on 2017-04-10 19:48:55 UTC by jenkins@nightly.
Free Memory  : 163K + 3118K

Not sure if this needs to be reported.
Testing April 12 build now


hjfilmspeed

Wait..... what did I do wrong here. I couldn't even get one solid rec out of the April 12 build so far. I had to pull the battery every time.
@a1ex Im sending you a message to a zip of my card its peppered with ASSERT messages.

JADURCA

Hi everyone! I just saw there is a new version, in case it helps I'm sharing this 4 CRASH.LOG files from crop_rec_4k.2017Apr04.5D3123 version.

ASSERT: hLvJob->hJpegMemSuite
at ./Epp/Vram/VramStage.c:891, task Epp
lv:1 mode:3


Magic Lantern version : crop_rec_4k.2017Apr04.5D3123
Mercurial changeset   : e7e1fdb85f6b (crop_rec_4k) tip
Built on 2017-04-04 15:42:08 UTC by jenkins@nightly.
Free Memory  : 163K + 3108K

ASSERT: IsSuiteSignature( hSuite )
at ./PackMemory/PackMem.c:599, task Epp
lv:0 mode:3


Magic Lantern version : crop_rec_4k.2017Apr04.5D3123
Mercurial changeset   : e7e1fdb85f6b (crop_rec_4k) tip
Built on 2017-04-04 15:42:08 UTC by jenkins@nightly.
Free Memory  : 163K + 3107K


ASSERT: IsSuiteSignature( hSuite )
at ./PackMemory/PackMem.c:599, task RscMgr
lv:0 mode:3


Magic Lantern version : crop_rec_4k.2017Apr04.5D3123
Mercurial changeset   : e7e1fdb85f6b (crop_rec_4k) tip
Built on 2017-04-04 15:42:08 UTC by jenkins@nightly.
Free Memory  : 163K + 3107K


ASSERT: IsSuiteSignature( hSuite )
at ./PackMemory/PackMem.c:983, task RscMgr
lv:0 mode:3


Magic Lantern version : crop_rec_4k.2017Apr04.5D3123
Mercurial changeset   : e7e1fdb85f6b (crop_rec_4k) tip
Built on 2017-04-04 15:42:08 UTC by jenkins@nightly.
Free Memory  : 163K + 3108K

goldenchild9to5

@JADURCA Dont use that build it's flawed.. make sure you back-up your ROM.

goldenchild9to5

@a1ex Just tried the April 12th build it's solid great work.  The only thing that I've noticed is when recording the minute / second green counter stays @ 00.00 it doen't show the actual recording time. 

KMikhail

Just wanted to give heads up to those who want most of resolution with lossless, but high ISO prevents you from it. You can sacrifice dynamic range but have a lower bandwidth requirement by using lower ISO and doing the pull in the post: the noise will be below the original gain line. Sure enough, it is worse than we'd like it, but...

Unfortunately, ISO 50 doesn't work with video, that would've basically removed one bit out of equation. Maybe there is another cheap (performance-wise) trick to lower effective number of bits, so lossless is more effective?

Here's a question: is there a way to have GD when I'm focusing, etc., but have it auto turned off (with everything else that could be turned off) during the recording?


hindra

Just tested newest build from Latest Build (2017-04-12 20:31).

I got the red dot in the memory patches again, no big deal right? It seemed like most of my shots using crop_rec had a bad first frame regardless of resolution.  Shooting at 3504x1536 at 24fps I only got 157 frames with a Lexar 128gb 1066x card.

Also, something odd kept happening and it kept reverting back to 19fps with FPS override off.
SL1 100D.100A - 5D - 7D2 - 5D3 1.2.3

a1ex

Quote from: KMikhail on April 13, 2017, 08:00:36 AM
Maybe there is another cheap (performance-wise) trick to lower effective number of bits, so lossless is more effective?

Yes - adjusting the digital gain in the raw backend appears to help (as it reduces the range of the data and cuts off the noise bits). Added a (somewhat fake) lossless compression at reduced bit depths, based on this concept (theory explained here).

Also added some suggestions about what ISO range to use with each option. See the commit for more details on how I've chosen the recommendations.

Quote from: goldenchild9to5 on April 13, 2017, 04:47:07 AM
The only thing that I've noticed is when recording the minute / second green counter stays @ 00.00 it doen't show the actual recording time. 

Fixed - was broken by pre-recording updates.

Quote from: hjfilmspeed on April 13, 2017, 02:12:43 AM
@a1ex Im sending you a message to a zip of my card its peppered with ASSERT messages.

The issue seems to be in 1920x1080 48p. Hopefully fixed, but I'm seriously thinking to reduce the default resolution to something like 1040 (thus breaking the promise of 1080p48). Any objections?

Quote from: hindra on April 13, 2017, 10:28:30 AM
Also, something odd kept happening and it kept reverting back to 19fps with FPS override off.

How to reproduce this one?




Anyway, new build posted (minor fixes + experimental 8...12-bit lossless).

Be sure to read the changelog before reporting issues ;)

Edit: just noticed the lower bit depth lossless modes only update the raw buffer every other frame, for reasons not yet understood... if you want to try, compile the sources, but you'll get files with duplicate frames (good for estimating recording times, but nothing more). If you look at a static scene (as I did before posting the build), everything appears fine.

Edit2: looks like all raw types that include digital gain are created every other frame :( (that means, I need to find some other method for darkening the image...)

Edit3: progress narrowing down (this used to work before merging 5D3-123 into main builds)


Narrowed down to changeset 38e78de. This change fixed it!

nikfreak

Congrats, lossless 8bit support.  8)
Any easter egg planned to support other cams  ;)?
[size=8pt]70D.112 & 100D.101[/size]