12-bit (and 10-bit) RAW video development discussion

[dfort]

Actually my December 12 build is using (SRM_BUFFER_SIZE - 0xF000) for the 5D2. You can check the history on Bitbucket using the Blame tool:



The way I got to that is by feedback from 5D2 users though honestly it would make more sense if we would subtracting the same amount for all cameras. I began by using the method used on the 1100D in the allocate-raw-lv-buffer branch and changed to using SRM_BUFFER_SIZE and subtracting a bit for overhead as per a1ex's suggestion:

SRM_BUFFER_SIZE would work if you allocate with srm_malloc_suite. With the malloc wrappers, there will be a small overhead from the memory backend, so... just reduce the size by 0x100 bytes or so. Didn't try, but this should be generic for any model that's going to implement full-res LiveView.

Testing on the 7D I determined that we needed to shave off 0x1000 from SRM_BUFFER_SIZE. Please run some tests on the 5D2 and let me know what works best.

Note that I started down this path assuming that 10bit/12bit was working on the 1100D because there was an experimental build posted on the downloads page but it turns out that nobody reported whether or not it was working. This whole allocate-raw-lv-buffer might turn out to be a dead end except that it seems to be working great on the 50D. Go figure.

[6D_ML]

Looks like the 6D is up and running properly. Thanks @6D_ML for the excellent bug reporting and following a1ex's instructions.

I did not expect  A1ex to release those last builds so quickly, and I lagged behind with the testing. Well done A1ex!
So far, great results for 6D video shooters:
Continuous 10-bit RAW recording 1728x724 (1.05x) @ 23.976 fps with sound.
Continuous 10-bit  RAW recording in crop mode 1920x720 (2.85x) @ 23.976 fps with sound.

Live liew during crop mode video recording does not work yet in 10/12-bit mode.

[reddeercity]

Note that I started down this path assuming that 10bit/12bit was working on the 1100D because there was an experimental build posted on the downloads page but it turns out that nobody reported whether or not it was working.
This whole allocate-raw-lv-buffer might turn out to be a dead end except that it seems to be working great on the 50D. Go figure.

That what I been using in my 3.5k/UHD thread with limited success (3584x1068),

Code Select Expand

#define RAW_LV_BUFFER_ALLOC_SIZE (2040*1267)
It was a1ex that suggested that I fellow the 1100D 10-12bit implementation
but I know I made mistakes  (sometimes the custom buffer will not load more then likely my coding skills or lack there of) I know I don't add the internals.h stuff

1100d_internals.h from10-12bit branch

Code Select Expand

/** Perfect sync using EVF_STATE **/
#define CONFIG_EVF_STATE_SYNC

Not sure if you are using this code , haven't check yet .
found this post claims to have run the Canon 1100d,  magiclantern-raw_video_10bit_12bit.2017Jan13.1100D105 build but does not say if they really used 10/12bit or just 14bit
http://www.magiclantern.fm/forum/index.php?topic=5601.msg178589#msg178589
so I would have to assume 10-12bit with RAW_LV_BUFFER_ALLOC_SIZE works on 1100d
another post from a 1100d 10-12bit build user -- http://www.magiclantern.fm/forum/index.php?topic=5601.msg178914#msg178914

Notice this allocate-raw-lv-buffer#Lsrc/raw.cT1798 for the 1100d , I know I haven't include this in my source code yet -- may you have @dfort  found them in this commit here
I guess I never fully implemented all the 1100d allocate-raw-lv state code , will update my source and try out

Edit: something interesting in vsync-lite.c at cF4T4 from this commit 1694635
I find this interesting also raw.c : Allow the backend to allocate a buffer for RAW lv if CONFIG_EDMAC_RAW_SLURP and CONFIG_BROKEN_RAW_LV_BUFFER are defined

[Danne]

Short report:

Tested 550D, following build:
https://bitbucket.org/daniel_fort/magic-lantern/downloads/raw_video_10bit_12bit.2017Dec12.550D109.zip
10bit choppy
12bit choppy
14bit working
Tried set override fps to 24, still choppy

Also tested 10/12/14bit on a 1100D, same choppy sync issues. Also a live view message about Hack error 0, expecting some other stub.
Following build:
https://bitbucket.org/daniel_fort/magic-lantern/downloads/raw_video_10bit_12bit.2017Dec12.1100D105.zip

[a1ex]

@Danne: while you have the 1100D, do you mind capturing the full set of VRAM dumps? http://www.magiclantern.fm/forum/index.php?topic=12375.0

(don't really need the external monitor ones; just the ones on internal LCD)

[Danne]

I'm on it.

[Danne]

Some progress. Seems only video mode for 1100D is mv720(1280x720):

https://bitbucket.org/Dannephoto/magic-lantern/downloads/Vram_dumps_1100D.zip

Hopefully useful. I could create as below:

3.dump live view with  5x zoom enabled (standby)
4.dump live view with 10x zoom enabled (standby)
7.dump live view in 720p video mode (standby)
8.dump live view in 720p video mode (recording)
14.enable raw video recording (via ML menu) press dump and quickly press play (to review a mlv video)
15.disable raw video, switch to photo mode (from mode dial or dedicated switch)
16.dump live view in photo mode
17.press dump then quickly press play (to review a photo)
18.For this last one, Image review must be enabled and set to at least 5 sec. (canon menu) after you press "dump VRAM" you must quickly take a picture, and the dump must be captured during image review (automatically review a photo after taken)

I got the camera and could continue tonight or tomorrow if I get specifics.

I am working with this ML build:
https://builds.magiclantern.fm/jenkins/job/raw_video_10bit_12bit/33/artifact/platform/1100D.105/magiclantern-raw_video_10bit_12bit.2017Dec18.1100D105.zip

[a1ex]

Thanks - definitely useful (as this camera has some unusual aspect ratios in its YUV buffers). There are a few 0-byte DNGs (guess: out of memory?), but I think I can fill in the gaps with what I already have.

[Danne]

Yes, noticed the 0 bytes. Occured when movie recording on. Maybe other occasions too. Glad to help.

[dfort]

Edit: something interesting in vsync-lite.c at cF4T4 from this commit 1694635
I find this interesting also raw.c : Allow the backend to allocate a buffer for RAW lv if CONFIG_EDMAC_RAW_SLURP and CONFIG_BROKEN_RAW_LV_BUFFER are defined

new-lv-buffer-detection - Yet another old experimental branch for the 1100D. There might be something there that we could try out, like we did with allocate-raw-lv-buffer. Thanks for pointing it out.

[rob_6]

@dfort,

I see that you have already done some testing with your 7D and the build that you are working on. Is there any additional testing you might need? I am happy to help if needed.

Thanks!

Rob

[djkraq]

I can't seem to find the build for the 70D.  Is it still there on the experimental page?

[dfort]

Is there any additional testing you might need?

Sure -- if you find a problem, tell us the issue, the steps to reproduce it and share any workaround you may discover.

I can't seem to find the build for the 70D.  Is it still there on the experimental page?

There is a Nightly build but if you want a 10bit/12bit experimental build you'll have to roll your own or bug @nikfreak for one.

[Stathman]

There is a Nightly build but if you want a 10bit/12bit experimental build you'll have to roll your own or bug @nikfreak for one.

There was a 10bit/12bit experimental build for 70D.
Do you know why it has been removed?

[dfort]

Don't know anything about a 70D build for 10bit/12bit - that camera isn't in the raw_video_10bit_12bit branch though it is in the crop_rec_4k branch. There isn't an official experimental build though @esas has reported on the Canon 70D topic that he has a build that seems to work with 10bit/12bit.

[Stathman]

Don't know anything about a 70D build for 10bit/12bit - that camera isn't in the raw_video_10bit_12bit branch though it is in the crop_rec_4k branch. There isn't an official experimental build though @esas has reported on the Canon 70D topic that he has a build that seems to work with 10bit/12bit.

Some days ago there was a build for 70D 10bit/12bit in the experiments section.
I have the file on my computer and this is the one i use on my 70D.
In fact, this is the reason i bought it in the first place!
Although, i will try the new build asap!
Thanks 

[dfort]

@Stathman - could you open the autoexec.bin file in a text editor and post the message at the beginning of the file? It should show where that build came from.

[Stathman]

Sure!

Magic Lantern raw_video_10bit_12bit.2017Aug21.70D112
Camera   : 70D
Firmware : 112
Changeset: 870cedc16e2d+87bd456b1128+ (raw_video_10bit_12bit) tip
Built on : 2017-08-21 08:31:05 by jenkins@nightly

[dfort]

Interesting. That looks like a merge of two branches. The Wayback Machine captured the Experiments download page the day before the build you've got and sure enough there was a 70D build there. I can't answer your question over why it went away.

[dariSSight]

Don't know anything about a 70D build for 10bit/12bit - that camera isn't in the raw_video_10bit_12bit branch though it is in the crop_rec_4k branch. There isn't an official experimental build though @esas has reported on the Canon 70D topic that he has a build that seems to work with 10bit/12bit.

Is there a update test build for 5D2, that we can test over the holiday. I apologize for my lack of coding ability, so if you can direct a non-coder in the direction of learning how to build test firmware for 5D2? I'm very willing to learn and i have the next 11 day off work.

[6D_ML]

Here is my take on comparing 10bit vs 14bit on 6D*1.1.6 Build 33.

mlv_rec module for recording RAW video (1.00x crop). →  MLV files converted into cinemaDNG (lossless) with MLV App v0.12 alpha. → DNGs to JPGs in LR 6.5. JPG conversion settings should be embedded in DNGs.

Expo settings: 1/30s, iso100, 30fps.
Image Fine-tuning:
ML Digital ISO: 0.0 EV
Black Level: 0
Shutter fine-tuning: +39 units

WB adjusted in ML EXPO tab (Auto adjust Kelvin + G/M) for 14-bit recording and remained fixed for 10bit mode. 
Click on thumb for full size JPG.


14bit / 30fps
Download DNG (https://drive.google.com/file/d/1JBd0B8n2QC-RHZ6hJaJiKvnxci4z12Ax/view?usp=sharing)


10bit / 30fps
Download DNG (https://drive.google.com/open?id=1R_mGzLH2k-4JgNHhEx-AqQl2dCopJTD-)

Is there a flaw in the above workflow or a 10-bit raw footage is not very forgiving for raising blacks and shadows compared to a 14-bit file for such a scene?
I've also shot less contrasty scenes that didn't require much shadows/blacks pushing in post, and 10/14bit footage looks indistinguishable, unless you start pixel peeping at 200%+ zoom level.

Also, I noticed that in Build 33, when I set in Pref tab, "ISO/Kelvin" - ON and "Use Set button" - ON, they do not appear in LV before or during recording. Pressing UP/DOWN/SET buttons does not change anything.

[dariSSight]

@6D_ML which build are you using and is there a 5D2 build?

[6D_ML]

@6D_ML which build are you using and is there a 5D2 build?

Build 33: http://builds.magiclantern.fm/jenkins/view/Experiments/job/raw_video_10bit_12bit/33/. It looks like 5D2 is not supported yet.

[dfort]

...if you can direct a non-coder in the direction of learning how to build test firmware for 5D2?

You don't need to know how to code in order to compile ML from source. I wrote a couple of tutorials for the methods that I use most.

Mac tutorial
Windows, Cygwin tutorial

If you are running Windows 10 there is a way to run Ubuntu on it. Here's a discussion on that topic. Cool thing about this method is that you can also run QEMU--with Cygwin you can't.

Since Danne got involved testing the 1100D I went ahead and looked up those stubs needed in the dm-spy-experiments branch. Still haven't been able to figure out a good test to run with that branch but it probably involves merging it into the experimental 10bit/12bit branch I've been using and creating some in camera logs.

@dariSSight - I'll post a 5D2 build that does this:

Code Select Expand

#ifdef CONFIG_5D2
-#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0xF000)
+#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0x1000)
#endif


That will be the Dec.22 build. What this does is to shrink the custom buffer we're using to the same amount that is working on the other cameras. Check it against the previous build, which I'll leave up for a while, and see if there is any difference between the two. Look for these unofficial test builds on my downloads page.

[dariSSight]

You don't need to know how to code in order to compile ML from source. I wrote a couple of tutorials for the methods that I use most.

Mac tutorial
Windows, Cygwin tutorial

If you are running Windows 10 there is a way to run Ubuntu on it. Here's a discussion on that topic. Cool thing about this method is that you can also run QEMU--with Cygwin you can't.
@dariSSight - I'll post a 5D2 build that does this:

Code Select Expand

#ifdef CONFIG_5D2
-#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0xF000)
+#define RAW_LV_BUFFER_ALLOC_SIZE (SRM_BUFFER_SIZE - 0x1000)
#endif


That will be the Dec.22 build. What this does is to shrink the custom buffer we're using to the same amount that is working on the other cameras. Check it against the previous build, which I'll leave up for a while, and see if there is any difference between the two. Look for these unofficial test builds on my downloads page.

@dfort Thanks for the links to Tutorials. I'm a Mac user, I'm up at 3am to jump right in, and I'll do a clean install of Dec 12 build because it gave me Choppy jumpy video when I use it last, but maybe I did something wrong but I'll test again along with Dec 22 build.Thanks Again