Author Topic: 12-bit (and 10-bit) RAW video development discussion  (Read 247087 times)

MitchLally

  • New to the forum
  • *
  • Posts: 30
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1200 on: March 06, 2017, 04:43:21 AM »
Hey guys I've been getting some weird behaviour from Mark III 10bit MLV files converted to DNG with MLVFS. Probably 1 in 50 MLV files seems corrupted.

It looks like the Black Level issue that we're all familiar with – but I can't correct it using exiftool. I got the black level changed to 2048 but did not solve the issue.

What might be happening here and is there a way to fix it? Would someone care to try exiftool to confirm? Here is a DNG file (untouched) from one of the corrupted 10 bit MLVs.

https://www.dropbox.com/s/m77zmz6m6h9huik/M03-1839_000008.dng?dl=0

reddeercity

  • Hero Member
  • *****
  • Posts: 1358
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1201 on: March 06, 2017, 05:49:35 AM »
No joy with exiftool , tried with mlv_dump , results below 
Will there is problems , Can you upload the corrupted MLV file if not too big,
you skinny can it down with mlvdiag.1.0.1.zip if on pc
Code: [Select]
C:\New\test>mlv_dump -o updated.dng --black-fix=valve M03-1839_000008.dng
 MLV Dumper v1.0
-----------------
Mode of operation:
   - Input MLV file: 'M03-1839_000008.dng'
   - Setting black level to 1
   - Rewrite MLV
   - Output into 'updated.dng'
File M03-1839_000008.dng opened
Processing...
[ERROR] Invalid block size at position 0x00000000
Processed 0 video frames
[ERROR] Failed to rewrite header in .MLV file
Done

You have some odd white & black levels
Code: [Select]
Unique Camera Model             : Canon EOS 5D Mark III
Black Level                     : 418
White Level                     : 1018
Light valve seems off too
Code: [Select]
Light Value                     : 3.1

MitchLally

  • New to the forum
  • *
  • Posts: 30
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1202 on: March 06, 2017, 08:10:04 AM »
It is 8.5GB file. I'm on Mac.

I found a similarly corrupted file that is much smaller.

https://www.dropbox.com/s/rmkyyg6q3nn47yj/M03-1356.MLV?dl=0

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10019
  • 5D Mark Free
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1203 on: March 06, 2017, 08:11:30 AM »
Setting black level to 128 fixes it. There's a bug in MLVFS that doesn't let you change black level with exiftool, but didn't manage to look into it yet.

MitchLally

  • New to the forum
  • *
  • Posts: 30
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1204 on: March 06, 2017, 09:17:48 AM »
I used mlv_dump instead to convert to DNG then used exiftool black level to 2040 – seems to have fixed it.

Although now there is an issue with the top section of the frame. It looks smeared.

https://www.dropbox.com/s/ucathd53p1megkt/M03-1839_000001.jpg?dl=0

Danne

  • Hero Member
  • *****
  • Posts: 3028
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1205 on: March 06, 2017, 10:10:45 AM »
Just a note. mlv_dump supports specifying both black and white level manually.
Also had issues with mlvfs dng files and exiftool. DNG files turning black or not letting exiftool changing tags.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10019
  • 5D Mark Free
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1206 on: March 06, 2017, 01:08:18 PM »
Although now there is an issue with the top section of the frame. It looks smeared.

With mlv_dump from http://builds.magiclantern.fm/utilities.html ?

If yes, try disabling cold pixel fix (--no-fixcp) and upload a small MLV so we can look into it (even cutting the first few MB from a file will do the trick). M03-1356.MLV appears to be OK.

reddeercity

  • Hero Member
  • *****
  • Posts: 1358
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1207 on: March 06, 2017, 11:14:08 PM »
Just a little update on the digic4 cams downloads


10-12bit downloads_DigicIV
by RedDeerCityTV, on Flickr

Seams 5D2 users are very hungry for Bit reduction  148 downloads and counting  8)
I don't think even 5d3 has that many bit reduction downloads
Long Live the mighty Digic IV Cams 7 years old+ and still in the game  :P

MitchLally

  • New to the forum
  • *
  • Posts: 30
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1208 on: March 07, 2017, 07:24:09 AM »
With mlv_dump from http://builds.magiclantern.fm/utilities.html ?

If yes, try disabling cold pixel fix (--no-fixcp) and upload a small MLV so we can look into it (even cutting the first few MB from a file will do the trick). M03-1356.MLV appears to be OK.

I'm on mac – is there an updated mlv_dump for mac?


Surin Dmitriy

  • New to the forum
  • *
  • Posts: 40
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1210 on: March 11, 2017, 01:52:00 PM »
Whats going on with in the raw_twk, mlv_play and 10 12 bit??? A lot of people confirm the issue whith playback - you can only review 2-3 clips before you get black screen?
5D3

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10019
  • 5D Mark Free

Anstech

  • Just arrived
  • *
  • Posts: 3
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1212 on: March 13, 2017, 02:07:54 AM »
I'm a programmer in a very different field, and I'm just starting to try to get up to speed with understanding the ML / Digic context, so please excuse any naivety in this question:

One way that ArriRaw, amongst others, reduces its data requirements is to write the 16bit linear data from the sensor into an uncompressed raw 12bit log data stream, which - given the familiar inefficiency of linear encoding of light values - drastically reduces frame sizes.

The current 14-to- 12- and 10-bit strategy is an amazingly impressive piece of work, and I can see took years to achieve, and a lot of specialist knowledge. If I understand correctly, it's a bit-chop lossy reduction in representation precision. Obviously the raw-log-encoded method would be less of a quality drop from 14- to 10-bit, if it existed.

So before I 'go off on one' (as my old AI programming tutor used to put it when he saw me disappearing down a research rabbit hole / dead end again):

Is e.g. a 14-bit to 10-bit log-encoding of the raw uncompressed (linear) sensor data - rather than a lossy bit-chop - computationally plausible in this context? Or is it waaay too ambitious for the resources available on-camera during recording?

It's caught my imagination, but I realise it might be a silly idea if you know enough - so if that's the case, I'll be sensible and take on a far more manageable task from the 'to-do' list as a 'teach myself' project instead...

Thank you!

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10019
  • 5D Mark Free
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1213 on: March 13, 2017, 08:45:50 AM »
On CPU, even a simple memcpy is too slow. So, the only remaining possibility is finding out how to configure Canon's image processor to do that.

Relevant links here, here, here and here.

TequilaKez

  • New to the forum
  • *
  • Posts: 33
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1214 on: March 22, 2017, 04:24:14 AM »
Chiming in to add/confirm reports on 550D every 2nd frame is a copy of the last frame, except the top 3rd which looks like current frame, only shifted on X axis.

If I can contribute to testing in anyway, gimme a job! 550D is 'B' cam, but love to get more than a couple of seconds of Raw at full res.

ricbi1

  • New to the forum
  • *
  • Posts: 14
  • Shoot - Shoot - Shoot
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1215 on: March 22, 2017, 04:25:27 AM »
Anstech... if you achieved that, it would be great! Just trying to encourage coz I can't do code for my life. Absolute ZERO. I tried over the years but no, can't do sir! Thanks everyone who is coding for sharing their efforts and developments!  :)

Eugenia

  • New to the forum
  • *
  • Posts: 3
  • Filmmaker and collage artist
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1216 on: March 25, 2017, 11:44:15 PM »
Hello, first post here. :)
I have two questions, that I couldn't find answers for:

1. What is the max res of the EOS M using 10 bit, no crop, at 24p continuous?

2. What is the max res of the 5D MkII using 10 bit, no crop, at 24p continuous? Why is there no 10-12bit ZIP file to download for the 5D-MkII? I couldn't find any on the exp. download page.

thx!

far.in.out

  • New to the forum
  • *
  • Posts: 34
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1217 on: March 26, 2017, 11:05:25 PM »
Hey guys. Sorry for asking such a question but I really have no time to read all the thread, been away for too long...
What is the status on 12/10 bit raw? Is it working? Are there any downsides compared to 14bit?
I'm mainly interested in 50D as it's the camera that I have.
If there's a compiled module for my cam please give me the link. Thanks.
50D (was 600D) Helios 44M-4, Canon 18-55mm IS II

Ilia3101

  • Member
  • ***
  • Posts: 146
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1218 on: March 27, 2017, 01:06:43 AM »
Red deer city has some builds uploaded a few pages back, including 50D, I know for a fact that the 50d can't film 10/12 bit without corruption in full sensor, but maybe it works well in crop mode like the 5D2, give it a try.
edit: it's on page 46
5D2

beauchampy

  • Senior
  • ****
  • Posts: 293
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1219 on: March 27, 2017, 12:07:25 PM »
Just came back from Malta filming a client piece entirely using 12bit and 3x3 50p. Flying by the seat of my pants, but the footage is lovely and not a single corrupt frame (5d3). I'll post some results next week. As far as I am concerned, this is more than ready for commercial use IMO.

andy kh

  • Senior
  • ****
  • Posts: 292
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1220 on: March 27, 2017, 12:27:24 PM »
atleast 4 footage gets corrupted out of 10 with wrong black level.  im using 70D

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10019
  • 5D Mark Free
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1221 on: March 27, 2017, 12:50:48 PM »

beauchampy

  • Senior
  • ****
  • Posts: 293
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1222 on: March 27, 2017, 08:52:43 PM »
atleast 4 footage gets corrupted out of 10 with wrong black level.  im using 70D

I'm using cr2hdr which enables you to specify a black level. Not sure what it is on the 70D but on my 5D3 I'm using 2048.

I do have the odd clip here and there with strange pink highlight clipping. I had also specified a white level of 1163.

Danne

  • Hero Member
  • *****
  • Posts: 3028
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1223 on: March 27, 2017, 10:15:51 PM »
Could you post an example MLV which shows the corruption? You could shorten it before uploading. Use mlv_dump and open up terminal and do following:
Code: [Select]
mlv_dump -f 5 -o Output.MLV Input.MLV
mlv_dump
http://www.magiclantern.fm/modules/modules/mlv_dump.zip/mlv_dump.zip

Input.MLV being the corrupted file and Output.MLV the name of the shortened file.
Beware to do things right since a wrong command could erase the file in question.

By the way. I think black and white levels should be around this for the 70D
White level 12277
Black level   2049

andy kh

  • Senior
  • ****
  • Posts: 292
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1224 on: March 27, 2017, 11:13:25 PM »
@Danne  please check this out. i use mlvp and set the black level to 125 for all 10bit mlv raw.
https://expirebox.com/download/fcfa483143425529e11d9cd2f4c43776.html