Author Topic: 12-bit (and 10-bit) RAW video development discussion  (Read 839335 times)

reddeercity

  • Contributor
  • Hero Member
  • *****
  • Posts: 2303
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1200 on: March 06, 2017, 05:49:35 AM »
No joy with exiftool , tried with mlv_dump , results below 
Will there is problems , Can you upload the corrupted MLV file if not too big,
you skinny can it down with mlvdiag.1.0.1.zip if on pc
Code: [Select]
C:\New\test>mlv_dump -o updated.dng --black-fix=valve M03-1839_000008.dng
 MLV Dumper v1.0
-----------------
Mode of operation:
   - Input MLV file: 'M03-1839_000008.dng'
   - Setting black level to 1
   - Rewrite MLV
   - Output into 'updated.dng'
File M03-1839_000008.dng opened
Processing...
[ERROR] Invalid block size at position 0x00000000
Processed 0 video frames
[ERROR] Failed to rewrite header in .MLV file
Done

You have some odd white & black levels
Code: [Select]
Unique Camera Model             : Canon EOS 5D Mark III
Black Level                     : 418
White Level                     : 1018
Light valve seems off too
Code: [Select]
Light Value                     : 3.1

MitchLally

  • New to the forum
  • *
  • Posts: 31
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1201 on: March 06, 2017, 08:10:04 AM »
It is 8.5GB file. I'm on Mac.

I found a similarly corrupted file that is much smaller.

https://www.dropbox.com/s/rmkyyg6q3nn47yj/M03-1356.MLV?dl=0

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1202 on: March 06, 2017, 08:11:30 AM »
Setting black level to 128 fixes it. There's a bug in MLVFS that doesn't let you change black level with exiftool, but didn't manage to look into it yet.

MitchLally

  • New to the forum
  • *
  • Posts: 31
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1203 on: March 06, 2017, 09:17:48 AM »
I used mlv_dump instead to convert to DNG then used exiftool black level to 2040 – seems to have fixed it.

Although now there is an issue with the top section of the frame. It looks smeared.

https://www.dropbox.com/s/ucathd53p1megkt/M03-1839_000001.jpg?dl=0

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7655
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1204 on: March 06, 2017, 10:10:45 AM »
Just a note. mlv_dump supports specifying both black and white level manually.
Also had issues with mlvfs dng files and exiftool. DNG files turning black or not letting exiftool changing tags.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1205 on: March 06, 2017, 01:08:18 PM »
Although now there is an issue with the top section of the frame. It looks smeared.

With mlv_dump from http://builds.magiclantern.fm/utilities.html ?

If yes, try disabling cold pixel fix (--no-fixcp) and upload a small MLV so we can look into it (even cutting the first few MB from a file will do the trick). M03-1356.MLV appears to be OK.

reddeercity

  • Contributor
  • Hero Member
  • *****
  • Posts: 2303
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1206 on: March 06, 2017, 11:14:08 PM »
Just a little update on the digic4 cams downloads


10-12bit downloads_DigicIV
by RedDeerCityTV, on Flickr

Seams 5D2 users are very hungry for Bit reduction  148 downloads and counting  8)
I don't think even 5d3 has that many bit reduction downloads
Long Live the mighty Digic IV Cams 7 years old+ and still in the game  :P

MitchLally

  • New to the forum
  • *
  • Posts: 31
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1207 on: March 07, 2017, 07:24:09 AM »
With mlv_dump from http://builds.magiclantern.fm/utilities.html ?

If yes, try disabling cold pixel fix (--no-fixcp) and upload a small MLV so we can look into it (even cutting the first few MB from a file will do the trick). M03-1356.MLV appears to be OK.

I'm on mac – is there an updated mlv_dump for mac?

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7655

Surin Dmitriy

  • New to the forum
  • *
  • Posts: 45
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1209 on: March 11, 2017, 01:52:00 PM »
Whats going on with in the raw_twk, mlv_play and 10 12 bit??? A lot of people confirm the issue whith playback - you can only review 2-3 clips before you get black screen?
5D3

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564

Anstech

  • New to the forum
  • *
  • Posts: 3
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1211 on: March 13, 2017, 02:07:54 AM »
I'm a programmer in a very different field, and I'm just starting to try to get up to speed with understanding the ML / Digic context, so please excuse any naivety in this question:

One way that ArriRaw, amongst others, reduces its data requirements is to write the 16bit linear data from the sensor into an uncompressed raw 12bit log data stream, which - given the familiar inefficiency of linear encoding of light values - drastically reduces frame sizes.

The current 14-to- 12- and 10-bit strategy is an amazingly impressive piece of work, and I can see took years to achieve, and a lot of specialist knowledge. If I understand correctly, it's a bit-chop lossy reduction in representation precision. Obviously the raw-log-encoded method would be less of a quality drop from 14- to 10-bit, if it existed.

So before I 'go off on one' (as my old AI programming tutor used to put it when he saw me disappearing down a research rabbit hole / dead end again):

Is e.g. a 14-bit to 10-bit log-encoding of the raw uncompressed (linear) sensor data - rather than a lossy bit-chop - computationally plausible in this context? Or is it waaay too ambitious for the resources available on-camera during recording?

It's caught my imagination, but I realise it might be a silly idea if you know enough - so if that's the case, I'll be sensible and take on a far more manageable task from the 'to-do' list as a 'teach myself' project instead...

Thank you!

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1212 on: March 13, 2017, 08:45:50 AM »
On CPU, even a simple memcpy is too slow. So, the only remaining possibility is finding out how to configure Canon's image processor to do that.

Relevant links here, here, here and here.

TequilaKez

  • Freshman
  • **
  • Posts: 54
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1213 on: March 22, 2017, 04:24:14 AM »
Chiming in to add/confirm reports on 550D every 2nd frame is a copy of the last frame, except the top 3rd which looks like current frame, only shifted on X axis.

If I can contribute to testing in anyway, gimme a job! 550D is 'B' cam, but love to get more than a couple of seconds of Raw at full res.

ricbi1

  • New to the forum
  • *
  • Posts: 14
  • Shoot - Shoot - Shoot
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1214 on: March 22, 2017, 04:25:27 AM »
Anstech... if you achieved that, it would be great! Just trying to encourage coz I can't do code for my life. Absolute ZERO. I tried over the years but no, can't do sir! Thanks everyone who is coding for sharing their efforts and developments!  :)

Eugenia

  • New to the forum
  • *
  • Posts: 9
  • Filmmaker and collage artist
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1215 on: March 25, 2017, 11:44:15 PM »
Hello, first post here. :)
I have two questions, that I couldn't find answers for:

1. What is the max res of the EOS M using 10 bit, no crop, at 24p continuous?

2. What is the max res of the 5D MkII using 10 bit, no crop, at 24p continuous? Why is there no 10-12bit ZIP file to download for the 5D-MkII? I couldn't find any on the exp. download page.

thx!

far.in.out

  • Freshman
  • **
  • Posts: 69
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1216 on: March 26, 2017, 11:05:25 PM »
Hey guys. Sorry for asking such a question but I really have no time to read all the thread, been away for too long...
What is the status on 12/10 bit raw? Is it working? Are there any downsides compared to 14bit?
I'm mainly interested in 50D as it's the camera that I have.
If there's a compiled module for my cam please give me the link. Thanks.
EOS M (was 600D > 50D)

ilia3101

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 991
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1217 on: March 27, 2017, 01:06:43 AM »
Red deer city has some builds uploaded a few pages back, including 50D, I know for a fact that the 50d can't film 10/12 bit without corruption in full sensor, but maybe it works well in crop mode like the 5D2, give it a try.
edit: it's on page 46

beauchampy

  • Senior
  • ****
  • Posts: 302
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1218 on: March 27, 2017, 12:07:25 PM »
Just came back from Malta filming a client piece entirely using 12bit and 3x3 50p. Flying by the seat of my pants, but the footage is lovely and not a single corrupt frame (5d3). I'll post some results next week. As far as I am concerned, this is more than ready for commercial use IMO.

andy kh

  • Hero Member
  • *****
  • Posts: 539
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1219 on: March 27, 2017, 12:27:24 PM »
atleast 4 footage gets corrupted out of 10 with wrong black level.  im using 70D
5D Mark III - 70D

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1220 on: March 27, 2017, 12:50:48 PM »

beauchampy

  • Senior
  • ****
  • Posts: 302
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1221 on: March 27, 2017, 08:52:43 PM »
atleast 4 footage gets corrupted out of 10 with wrong black level.  im using 70D

I'm using cr2hdr which enables you to specify a black level. Not sure what it is on the 70D but on my 5D3 I'm using 2048.

I do have the odd clip here and there with strange pink highlight clipping. I had also specified a white level of 1163.

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7655
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1222 on: March 27, 2017, 10:15:51 PM »
Could you post an example MLV which shows the corruption? You could shorten it before uploading. Use mlv_dump and open up terminal and do following:
Code: [Select]
mlv_dump -f 5 -o Output.MLV Input.MLV
mlv_dump
http://www.magiclantern.fm/modules/modules/mlv_dump.zip/mlv_dump.zip

Input.MLV being the corrupted file and Output.MLV the name of the shortened file.
Beware to do things right since a wrong command could erase the file in question.

By the way. I think black and white levels should be around this for the 70D
White level 12277
Black level   2049

andy kh

  • Hero Member
  • *****
  • Posts: 539
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1223 on: March 27, 2017, 11:13:25 PM »
@Danne  please check this out. i use mlvp and set the black level to 125 for all 10bit mlv raw.
https://expirebox.com/download/fcfa483143425529e11d9cd2f4c43776.html
5D Mark III - 70D

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7655
Re: 12-bit (and 10-bit) RAW video development discussion
« Reply #1224 on: March 27, 2017, 11:27:59 PM »
Yes, classic black level issue. From your file after extracting with mlv_dump
Black Level                     : 7312
White Level                     : 14992

When reprocessing and specifying numbers below picture is perfect.
Black Level                     : 2049
White Level                     : 12277

In mlvp you specify 10bit levels so they differ. Above numbers are for upressed to 14bit. Visually the picture should look the same for both 10bit and 14bit after fixing.