Author Topic: 12-bit (and 10-bit) RAW video development discussion  (Read 764907 times)

budafilms

  • Hero Member
  • *****
  • Posts: 711
  • 5D Mark a1ex
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1075 on: January 13, 2017, 04:49:02 AM »
@a1ex,
" no more trickery" means no more mlv dump black fixes?

What's next? Mpeg?

:)

dfort

  • Guest
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1076 on: January 13, 2017, 08:21:48 AM »
New builds on the Experimentals download page are ready. I took down the builds I've been hosting. These cameras do not yet support CONFIG_EDMAC_RAW_SLURP but when someone puts up a pull request I'll try to compile a build for testing.

5D2.212
7D.203
50D.109
550D.109
500D.111

Licaon_Kter

  • Hero Member
  • *****
  • Posts: 519
  • M
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1077 on: January 13, 2017, 08:55:25 AM »
So close to merger, thanks for the continous development.  :o

D_Odell

  • Member
  • ***
  • Posts: 187
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1078 on: January 13, 2017, 05:44:34 PM »
Had a shoot today. First day with latest nightly and today with Jan 11 10-12bit build. External monitor did work at first (worked without any problem on Nightly), some shots were made then later the monitor only froze when connecting to the 5D3 with that build. Anyone experienced the same? Monitor was an Small HD DP6 5,6".

Otherways the build worked great!
5D3 (OLPF removed) :: 1.1.3 :: Canon FD L Serie

ch_d

  • Freshman
  • **
  • Posts: 99
  • 5D MII
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1079 on: January 13, 2017, 06:41:13 PM »
@dfort: i would love to test a current 5D2 build (don´t know how to make a pull request  :-\)
5D MII

Sganzerla

  • New to the forum
  • *
  • Posts: 43
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1080 on: January 13, 2017, 10:21:58 PM »
I'm going to post something that may be of interest, even if I'm not sure it is related to this area of the forum, or the 'post processing' one.

Using the experimental builds today - 5D MKIII 1.2.3 13th January - I tested the crop mode 3x3 720p and FPS override at 70fps (67fps) in 10/12/14bit modes.
Using raw2cdng software with "16bit CDNG maximized" option ticked, I opened the 3 files in Photoshop (ACR) and noticed that the blue cast in the shadows is not there anymore, but there are differences related to the magenta/green cast now, being 10bit files greener than 14bit files. I don't know if it is possible to see the exact difference from the image bellow, but there is a link to download these files if anyone is interested.

10bit TOP
12bit MIDDLE
14bit BELLOW



The download link of the 3 files is here:
https://goo.gl/YE4hcN

The difference is not solved with tint adjustment, unfortunately.

By the way, I don't know if people have tried this and seen before or not, but on a side note, related to software, MLV Mystic is producing noticeable more magenta images than raw2cdng from the same 14bit files too. So I don't know if it is something software related or code related.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1081 on: January 13, 2017, 10:28:04 PM »
Any difference if you convert with latest mlv_dump from https://builds.magiclantern.fm/utilities.html ?

dfort

  • Guest
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1082 on: January 13, 2017, 11:40:53 PM »
Is there a working 5d3 123 version with mlv 10,12,14bit and crop rec version available to try? I downloaded the one from experiments but it only had Mlv-Lite module for raw.

Top of the Experimentals download page. That build includes mlv_rec.

@dfort: i would love to test a current 5D2 build (don´t know how to make a pull request  :-\)

You don't need to make a pull request to test it. I took down the builds I made for cameras that don't have CONFIG_EDMAC_RAW_SLURP which the 5D2 is one of them. @reddeercity made a very complete test of that camera so we have a good idea of what's working and what still needs to be done. If you want to do some additional testing I'd suggest getting in touch with him.

@g3gg0

There is an issue with raw_twk on the EOSM. When mlv_rec, mlv_snd, mlv_play and raw_twk are loaded and MLV Sound is turned on a "Threads failed to start" error appears on the screen and the camera must be shut down or it may even require a battery pull. This happens with any bit depth setting for picture and sound. If the raw_twk module is not loaded this will not happen.



I was unable to reproduce this on the 700D so this issue may be specific to the EOSM.

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3183
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1083 on: January 14, 2017, 01:11:41 AM »
yeah, already mentioned a few times.
the EOS M has very few threads available and some modules need threads for their work.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

Sganzerla

  • New to the forum
  • *
  • Posts: 43
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1084 on: January 14, 2017, 01:23:55 AM »
Any difference if you convert with latest mlv_dump from https://builds.magiclantern.fm/utilities.html ?

Thanks a1ex for the insight. I tought mlv_dump was a software for developers!
If I make it work, I'll come back with info. I think my English is not helping very much on this issue, but I'll try to find what is happening.

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3183
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1085 on: January 14, 2017, 01:33:48 AM »
mlv_dump is a reference implementation on how to read MLV files and how to extract the frames.
it is by far not the most powerful toy to play with MLV files, but it offers a few advanced features.

so when in doubt, use mlv_dump. if its output is unexpected then we can help :)
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7297
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1086 on: January 14, 2017, 01:46:15 AM »
Mlv_dump, the more powerful tool imo. Strong features. Now working with 10/12/14 bits without the need to specify black/white level. And also:
- darkframe handling
- flatframe handling
- Possible to parallelize exporting processing
- and more...

dfort

  • Guest
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1087 on: January 14, 2017, 03:50:55 AM »
yeah, already mentioned a few times.
the EOS M has very few threads available and some modules need threads for their work.

Sorry, I searched the forum before posting but came up with the other definition of threads.

it is by far not the most powerful toy to play with MLV...

The definition of modesty.

dmilligan

  • Developer
  • Hero Member
  • *****
  • Posts: 3218
  • 60Da / 1100D / EOSM
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1088 on: January 14, 2017, 04:35:29 AM »
threads = tasks

reddeercity

  • Contributor
  • Hero Member
  • *****
  • Posts: 2295
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1089 on: January 14, 2017, 06:15:45 AM »
i would love to test a current 5D2 build (don´t know how to make a pull request  :-\)
@ch_d no problem , I'll host the 5D2 10-12bit experimental build in my bitbucket account for the next 4-6 weeks
Or until 5D2 and older cams have liveview functional in 1:1 mode.
This build is the one I use in full mlv+audio in crop mode (2.61 crop) 2144x1076 23.976p @ 10bit + audio. (65MB/s write speed)
Please note 1:1 liveveiw is not working while recording raw bit reduction (corrupt frames) 14bit Ok .
With the build there the raw_twk.mo to play back 10-12bit in camera so it must be loaded with mlv_play & .mlv or .raw(mlv lite).
10bit_12bit_raw_twk.2016Dec01.5D2212.zip by chance if I can figure out the problem
with the liveview in bit reduction  ::) then I'll share my builds here.

If older cams e.g. 7D , 50D , 550D 500D user wish to try bit reductions with limited functionality ( I know 7D works in crop mode like the 5D2 but not sure about the rest)
I can post those cams builds also if requested , there will be from dec.1/2016 same as the one I posted for 5D2





ch_d

  • Freshman
  • **
  • Posts: 99
  • 5D MII
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1090 on: January 14, 2017, 01:28:08 PM »
1:1 not working: That´s a pity. I don´t like the crop mode (because of FOV / DOF).
5D MII

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3183
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1091 on: January 14, 2017, 02:28:32 PM »
@dfort:
yeah sorry, threads and tasks can be used as a synonym in our case, as there is no memory segregation/separation or anything between them.
threads are known as kind of light weight tasks in context of the parent task.
i should use task as the word canons os uses.


good to hear that you all like mlv_dump how it is today :)
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

GutterPump

  • Member
  • ***
  • Posts: 185
  • 5DIII/5DII/550D
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1092 on: January 14, 2017, 03:28:31 PM »
For those who use mlv_dump and who like to do things simply, I recommend this tutorial

It is easy also to make different styles of dng extraction commands in the folder "send to" by duplicating .cmd files.
It's interesting when we need to do some tests with news features.

honza123

  • Freshman
  • **
  • Posts: 65
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1093 on: January 15, 2017, 03:12:07 PM »
www.youtube.com/watch?v=__k0o0ycYk0

Test of 10 bit raw video recorded on Canon EOS 600D during daylight ISO 100.
Resolution: 1728x672, 23,97fps, ISO 100, raw_rec.mo
BUILD/dfort/: magiclantern-raw_video_10bit_12bit.2017Jan13.600D102
I recorded 23 raw video files. Three files were with a "green cast".
Command: exiftool "-BlackLevel=2032" "-whitelevel=14992" *.dng didn't fix it.
EOS 5D Mk.II

dfort

  • Guest
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1094 on: January 15, 2017, 05:47:23 PM »
Try the latest builds from the Experiments download page. I believe there were some recent changes to the way black level is handled.

chmee

  • Contributor
  • Hero Member
  • *****
  • Posts: 1031
  • raw2cdng
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1095 on: January 15, 2017, 06:17:29 PM »
@Sganzerla
as @g3gg0 said, mlv_dump is the reference, all other tools are written by "independent" coders. so, if other tools are in need of a fix, write them (as i am the coder of raw2cdng)

regards chmee

budafilms

  • Hero Member
  • *****
  • Posts: 711
  • 5D Mark a1ex
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1096 on: January 15, 2017, 08:30:35 PM »
Can someone post the link, or a list with the commands lines of Mlv_dump?

honza123

  • Freshman
  • **
  • Posts: 65
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1097 on: January 15, 2017, 08:39:42 PM »
My examples of "green cast" - black level. Original and after fix
Shot with EOS 600D:
Resolution: 1728x672, 23,97fps, ISO 100, raw_rec.mo, magiclantern-raw_video_10bit_12bit.2017Jan13.600D102,
http://www.uschovna.cz/zasilka/LIVYCU4ETGKEGXFD-ZAK/
EOS 5D Mk.II

dmilligan

  • Developer
  • Hero Member
  • *****
  • Posts: 3218
  • 60Da / 1100D / EOSM
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1098 on: January 15, 2017, 08:40:33 PM »
Can someone post the link, or a list with the commands lines of Mlv_dump?
If you run mlv_dump without any other options, it will print that for you.

RenatoPhoto

  • Moderators
  • Hero Member
  • *****
  • Posts: 1509
  • 5DM3 / 7D
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1099 on: January 15, 2017, 08:53:46 PM »
Usage: mlv_dump [-o output_file] [-rscd] [-l compression_level(0-9)] <inputfile>

Code: [Select]
Parameters:
 -o output_file      set the filename to write into
 -v                  verbose output
 --batch             output message suitable for batch processing

-- DNG output --
 --dng               output frames into separate .dng files. set prefix with -o
 --no-cs             no chroma smoothing (default)
 --cs2x2             2x2 chroma smoothing
 --cs3x3             3x3 chroma smoothing
 --cs5x5             5x5 chroma smoothing
 --no-fixcp          do not fix cold pixels
 --fixcp2            fix non-static (moving) cold pixels (slow)
 --no-stripes        do not fix vertical stripes in highlights

-- RAW output --
 -r                  output into a legacy raw file for e.g. raw2dng

-- MLV output --
 -b bits             convert image data to given bit depth per channel (1-16)
 -z bits             zero the lowest bits, so we have only specified number of b
its containing data (1-16) (improves compression rate)
 -f frames           frames to save. e.g. '12' saves the first 12 frames, '12-40
' saves frames 12 to 40.
 -A fpsx1000         Alter the video file's FPS metadata
 -x                  build xref file (indexing)
 -m                  write only metadata, no audio or video frames
 -n                  write no metadata, only audio and video frames

-- Image manipulation --
 -a                  average all frames in <inputfile> and output a single-frame MLV from it
 --avg-vertical      [DARKFRAME ONLY] average the resulting frame in vertical direction, so we will extract vertical banding
 --avg-horizontal    [DARKFRAME ONLY] average the resulting frame in horizontaldirection, so we will extract horizontal banding
 -s mlv_file         subtract the reference frame in given file from every single frame during processing
 -t mlv_file         use the reference frame in given file as flat field (gain correction)

-- Processing --
 -e                  delta-encode frames to improve compression, but lose random
 access capabilities
 -X type             extract only block type
 -I mlv_file         inject data from given MLV file right after MLVI header
 -c                  (re-)compress video and audio frames using LZMA (set bpp to
 16 to improve compression rate)
 -d                  decompress compressed video and audio frames using LZMA
 -l level            set compression level from 0=fastest to 9=best compression

-- bugfixes --
 --black-fix=value   set black level to <value> (fix green/magenta cast). if no
value given, it will be set to 2048.
 --white-fix=value   set white level to <value>. if no value given, it will be s
et to 15000.
 --fix-bug=id        fix some special bugs. *only* to be used if given instructi
on by developers.
http://www.pululahuahostal.com  |  EF 300 f/4, EF 100-400 L, EF 180 L, EF-S 10-22, Samyang 14mm, Sigma 28mm EX DG, Sigma 8mm 1:3.5 EX DG, EF 50mm 1:1.8 II, EF 1.4X II, Kenko C-AF 2X