Author Topic: 12-bit (and 10-bit) RAW video development discussion  (Read 188634 times)

MitchLally

  • New to the forum
  • *
  • Posts: 24
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1200 on: March 06, 2017, 04:43:21 AM »
Hey guys I've been getting some weird behaviour from Mark III 10bit MLV files converted to DNG with MLVFS. Probably 1 in 50 MLV files seems corrupted.

It looks like the Black Level issue that we're all familiar with – but I can't correct it using exiftool. I got the black level changed to 2048 but did not solve the issue.

What might be happening here and is there a way to fix it? Would someone care to try exiftool to confirm? Here is a DNG file (untouched) from one of the corrupted 10 bit MLVs.

https://www.dropbox.com/s/m77zmz6m6h9huik/M03-1839_000008.dng?dl=0

reddeercity

  • Hero Member
  • *****
  • Posts: 1225
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1201 on: March 06, 2017, 05:49:35 AM »
No joy with exiftool , tried with mlv_dump , results below 
Will there is problems , Can you upload the corrupted MLV file if not too big,
you skinny can it down with mlvdiag.1.0.1.zip if on pc
Code: [Select]
C:\New\test>mlv_dump -o updated.dng --black-fix=valve M03-1839_000008.dng
 MLV Dumper v1.0
-----------------
Mode of operation:
   - Input MLV file: 'M03-1839_000008.dng'
   - Setting black level to 1
   - Rewrite MLV
   - Output into 'updated.dng'
File M03-1839_000008.dng opened
Processing...
[ERROR] Invalid block size at position 0x00000000
Processed 0 video frames
[ERROR] Failed to rewrite header in .MLV file
Done

You have some odd white & black levels
Code: [Select]
Unique Camera Model             : Canon EOS 5D Mark III
Black Level                     : 418
White Level                     : 1018
Light valve seems off too
Code: [Select]
Light Value                     : 3.1

MitchLally

  • New to the forum
  • *
  • Posts: 24
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1202 on: March 06, 2017, 08:10:04 AM »
It is 8.5GB file. I'm on Mac.

I found a similarly corrupted file that is much smaller.

https://www.dropbox.com/s/rmkyyg6q3nn47yj/M03-1356.MLV?dl=0

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 9554
  • 5D Mark Free
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1203 on: March 06, 2017, 08:11:30 AM »
Setting black level to 128 fixes it. There's a bug in MLVFS that doesn't let you change black level with exiftool, but didn't manage to look into it yet.

MitchLally

  • New to the forum
  • *
  • Posts: 24
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1204 on: March 06, 2017, 09:17:48 AM »
I used mlv_dump instead to convert to DNG then used exiftool black level to 2040 – seems to have fixed it.

Although now there is an issue with the top section of the frame. It looks smeared.

https://www.dropbox.com/s/ucathd53p1megkt/M03-1839_000001.jpg?dl=0

Danne

  • Hero Member
  • *****
  • Posts: 2715
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1205 on: March 06, 2017, 10:10:45 AM »
Just a note. mlv_dump supports specifying both black and white level manually.
Also had issues with mlvfs dng files and exiftool. DNG files turning black or not letting exiftool changing tags.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 9554
  • 5D Mark Free
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1206 on: March 06, 2017, 01:08:18 PM »
Although now there is an issue with the top section of the frame. It looks smeared.

With mlv_dump from http://builds.magiclantern.fm/utilities.html ?

If yes, try disabling cold pixel fix (--no-fixcp) and upload a small MLV so we can look into it (even cutting the first few MB from a file will do the trick). M03-1356.MLV appears to be OK.

reddeercity

  • Hero Member
  • *****
  • Posts: 1225
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1207 on: March 06, 2017, 11:14:08 PM »
Just a little update on the digic4 cams downloads


10-12bit downloads_DigicIV
by RedDeerCityTV, on Flickr

Seams 5D2 users are very hungry for Bit reduction  148 downloads and counting  8)
I don't think even 5d3 has that many bit reduction downloads
Long Live the mighty Digic IV Cams 7 years old+ and still in the game  :P

MitchLally

  • New to the forum
  • *
  • Posts: 24
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1208 on: March 07, 2017, 07:24:09 AM »
With mlv_dump from http://builds.magiclantern.fm/utilities.html ?

If yes, try disabling cold pixel fix (--no-fixcp) and upload a small MLV so we can look into it (even cutting the first few MB from a file will do the trick). M03-1356.MLV appears to be OK.

I'm on mac – is there an updated mlv_dump for mac?


Surin Dmitriy

  • New to the forum
  • *
  • Posts: 25
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1210 on: March 11, 2017, 01:52:00 PM »
Whats going on with in the raw_twk, mlv_play and 10 12 bit??? A lot of people confirm the issue whith playback - you can only review 2-3 clips before you get black screen?
5D3

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 9554
  • 5D Mark Free

Anstech

  • Just arrived
  • *
  • Posts: 2
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1212 on: March 13, 2017, 02:07:54 AM »
I'm a programmer in a very different field, and I'm just starting to try to get up to speed with understanding the ML / Digic context, so please excuse any naivety in this question:

One way that ArriRaw, amongst others, reduces its data requirements is to write the 16bit linear data from the sensor into an uncompressed raw 12bit log data stream, which - given the familiar inefficiency of linear encoding of light values - drastically reduces frame sizes.

The current 14-to- 12- and 10-bit strategy is an amazingly impressive piece of work, and I can see took years to achieve, and a lot of specialist knowledge. If I understand correctly, it's a bit-chop lossy reduction in representation precision. Obviously the raw-log-encoded method would be less of a quality drop from 14- to 10-bit, if it existed.

So before I 'go off on one' (as my old AI programming tutor used to put it when he saw me disappearing down a research rabbit hole / dead end again):

Is e.g. a 14-bit to 10-bit log-encoding of the raw uncompressed (linear) sensor data - rather than a lossy bit-chop - computationally plausible in this context? Or is it waaay too ambitious for the resources available on-camera during recording?

It's caught my imagination, but I realise it might be a silly idea if you know enough - so if that's the case, I'll be sensible and take on a far more manageable task from the 'to-do' list as a 'teach myself' project instead...

Thank you!

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 9554
  • 5D Mark Free
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1213 on: March 13, 2017, 08:45:50 AM »
On CPU, even a simple memcpy is too slow. So, the only remaining possibility is finding out how to configure Canon's image processor to do that.

Relevant links here, here, here and here.

TequilaKez

  • New to the forum
  • *
  • Posts: 29
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1214 on: Yesterday at 04:24:14 AM »
Chiming in to add/confirm reports on 550D every 2nd frame is a copy of the last frame, except the top 3rd which looks like current frame, only shifted on X axis.

If I can contribute to testing in anyway, gimme a job! 550D is 'B' cam, but love to get more than a couple of seconds of Raw at full res.

ricbi1

  • New to the forum
  • *
  • Posts: 14
  • Shoot - Shoot - Shoot
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1215 on: Yesterday at 04:25:27 AM »
Anstech... if you achieved that, it would be great! Just trying to encourage coz I can't do code for my life. Absolute ZERO. I tried over the years but no, can't do sir! Thanks everyone who is coding for sharing their efforts and developments!  :)