Author Topic: uncompressed 14-bit RAW video recording  (Read 755326 times)

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3147
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #325 on: May 13, 2013, 11:30:05 AM »
I'm bummed the 5d2 doesn't save MM1 files, so raw dumping doesn't work.

it can do 64MB/s or so write speed with a 1000x card, so it can do the 1080p @ 24fps.. just need to make the raws work.

we still can set up a EDMAC to read from connection 0 which should be the CRAW source.
if we dont use fragmented memory as buffer, it could work.
in lv_rec i used fragmented memory and at fragment switching, some data seemed to get lost.

this might work on any other model, that doesnt have lv_save_raw.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

mucher

  • Member
  • ***
  • Posts: 121
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #326 on: May 13, 2013, 11:31:27 AM »
if the raw video is enable, I calculate that it will be producing 500m/s plus bitrate, which is bigger than most 4:4:4 cinema cameras I suppose, and it will be hugely demanding for the CF card and HD and post-processing. For the moment,  this raw output feature is veryvery nice, but I still wish to see than it can be compressed someway, or that 4:2:2 video file output will be better for me.

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3147
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #327 on: May 13, 2013, 11:41:32 AM »
Full 1080P test at 24fps.  This time with audio.

errrh, why has the driver a helmet on? :)
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

Andy600

  • Contributor
  • Hero Member
  • *****
  • Posts: 1857
  • Have you tried turning it off and on again?
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #328 on: May 13, 2013, 11:46:10 AM »
Ok, this is wild speculation but...

Outputting RAW data via HDMI will likely be impossible due to HDMI limitations and SD is slower compared to CF but is there a ceiling on the level of data throughput of SD (or it's controller) that would prevent higher data rates being sent to the SD card slot - through an adapter to an external recording device (i.e. an SSD) to bypass the card's own speed restrictions? I know the SSD would likely need it's own controller but we're talking pure data dumps and not debayered/cropped/formatted information.

guys?  ::)
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

ted ramasola

  • Moderators
  • Hero Member
  • *****
  • Posts: 1250
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #329 on: May 13, 2013, 11:46:25 AM »
errrh, why has the driver a helmet on? :)

falling coconuts outside the garage.
5DmkII  / 7D
www.ramasolaproductions.com
Texas

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12258
  • Emergencies only
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #330 on: May 13, 2013, 11:51:40 AM »
For 5d2, I found all stubs [...]

Did some small adjustments and seems to work very well on 5D2.

Theoretical figures for the 1000x card, at 24fps, based on Coutts' benchmark (64MB/s):

- 1880x840 in full-frame mode (with line skipping)
- 1920x840 or 2152x720 in 1:1 crop mode (zero moire)

With my 266x card, I get 21-22MB/s, so... 720x720 :)

Stedda

  • Hero Member
  • *****
  • Posts: 504
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #331 on: May 13, 2013, 12:05:07 PM »

With my 266x card, I get 21-22MB/s, so... 720x720 :)

Bring back the donation link so we can get all you guys new UDMA7 1066x Cards! :)
5D Mark III -- 7D   SOLD -- EOS M 22mm 18-55mm STM -- Fuji X-T1 18-55 F2.8-F4 & 35 F1.4
Canon Glass   100L F2.8 IS -- 70-200L F4 -- 135L F2 -- 85 F1.8 -- 17-40L --  40 F2.8 -- 35 F2 IS  Sigma Glass  120-300 F2.8 OS -- 50 F1.4 -- 85 F1.4  Tamron Glass   24-70 2.8 VC   600EX-RT X3

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3147
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #332 on: May 13, 2013, 12:06:02 PM »
Ok, this is wild speculation but...

Outputting RAW data via HDMI will likely be impossible due to HDMI limitations and SD is slower compared to CF but is there a ceiling on the level of data throughput of SD (or it's controller) that would prevent higher data rates being sent to the SD card slot - through an adapter to an external recording device (i.e. an SSD) to bypass the card's own speed restrictions? I know the SSD would likely need it's own controller but we're talking pure data dumps and not debayered/cropped/formatted information.

guys?  ::)

until now we dont understand the SD/CF hardware good enough.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

Andy600

  • Contributor
  • Hero Member
  • *****
  • Posts: 1857
  • Have you tried turning it off and on again?
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #333 on: May 13, 2013, 12:07:34 PM »
@1% I've tried just about every combination/setting and still everything is pink using the new raw module (600d). The green channel looks very weak on a histogram. DNG silentpic bursts are fine.
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

rudi

  • New to the forum
  • *
  • Posts: 5
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #334 on: May 13, 2013, 12:26:27 PM »
BIG BIG Kudos for you all!!
Just two cents from me for alle developers of ML:

My C-coding  times are about 6 years outdated, but when the sensor only has 12 Stops and and the Raw Data Rate is 14 Bit, maybe you could introduce something like a minimal Quick Compression by shifting away the 2 least significant bits. Maybe the processing power is enough for this litte bit shifting trick during copying the data from buffer to CF. This would lower the datarate with minimal loss.
On the PC/Mac/Linux side you just shift the lost bits back (with random bits) when converting to CinemaDNG or whatever.

Maybe my idea is completely off, then forget it.
Otherwise this could help to bring the datarates down a little without great visual loss...

Greetings

Rudi from slashCAM










a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12258
  • Emergencies only
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #335 on: May 13, 2013, 12:37:03 PM »
Yeah, it might work at 1 fps or so :D

The secret ingredient here is that CPU usage is almost zero; the hard work is offloaded to the DMA controllers (EDMAC). We can do only fast copying and rectangle cropping.

Look at memory benchmarks: optimized memcpy 73MB/s, 8-bit memcpy 20MB/s, EDMAC 700MB/s.


arrinkiiii

  • Hero Member
  • *****
  • Posts: 1101
  • ML ADDICTED 7D
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #336 on: May 13, 2013, 12:40:46 PM »
!!! CONGRATULATIONS !!! THIS IS AMAZING !!! AND HUGE GAME TURN !!!

...and we, the humbles mortal geeks, with a 7D (that got 2 processors), no code for we try it ?




 ;D



 

squig

  • Hero Member
  • *****
  • Posts: 518
  • Crash test dummy MK3
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #337 on: May 13, 2013, 12:42:52 PM »
errrh, why has the driver a helmet on? :)

Offers some protection in case his MK3 explodes and adds to the cinematic look.

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 3147
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #338 on: May 13, 2013, 12:44:11 PM »
...and we, the humbles mortal geeks, with a 7D (that got 2 processors), no code for we try it ?

already said: we are working on making it stable and testing what is possible.
it is not time yet to widely announce it. so there is nothing for 7D yet.
Help us with datasheets - Help us with register dumps
magic lantern: 1Magic9991E1eWbGvrsx186GovYCXFbppY, server expenses: paypal@g3gg0.de
ONLY donate for things we have done, not for things you expect!

mcnys

  • New to the forum
  • *
  • Posts: 14
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #339 on: May 13, 2013, 12:45:49 PM »
so how many frames 600d users can record now?

Kuky

  • New to the forum
  • *
  • Posts: 36
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #340 on: May 13, 2013, 12:47:29 PM »
Hi,

I'm new to this stuff, never used ML firmware before. But I have an MK3 and want to test/help on this.

I  have the 1.1.3 firmware on the mk3 and i've downloaded the mk3 zip files posted by lourenco and extracted them to an empty lexar cf card.
Then I followed the installation wiki documentation (although I see it was not written for the mk3).
When initiating the firmware update process I see a message superimposed on the menu saying something like "enable bootflag :)" and nothing happens. On restart I have the original 1.1.3 firmware.

I know it's probable trivial but can somebody help?

Thank you,
Cristian

Steven

  • Freshman
  • **
  • Posts: 52
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #341 on: May 13, 2013, 12:54:26 PM »
Just erase the bootflag fir from the card and make the card bootable with eoscard.

That's how far I got... ML is running, but I can't find the raw features in the menues!?

arrinkiiii

  • Hero Member
  • *****
  • Posts: 1101
  • ML ADDICTED 7D
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #342 on: May 13, 2013, 12:56:02 PM »
already said: we are working on making it stable and testing what is possible.
it is not time yet to widely announce it. so there is nothing for 7D yet.

I know (we, 7D users know it) this g3gg0  ;D Is just for saying that we are were, we have trust on the team :D for sure that all the people are truly amazing with this 14 bits raw scene and want to trie out this  :D

Mamamiaaaaaa  ;D


Andy600

  • Contributor
  • Hero Member
  • *****
  • Posts: 1857
  • Have you tried turning it off and on again?
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #343 on: May 13, 2013, 01:06:41 PM »
so how many frames 600d users can record now?

Not as many as on a 5d MkIII ;)

I'm still waiting for my fast SD card to arrive. On a class10 SD it only manages continuous recording at the smallest frame settings but even then it drops/corrupts some frames and everything is tinted pink atm on my 600d. At bigger frame sizes you get only slightly more frames than Silentpic DNG bursts. The guys are working hard and it's very early days, especially for the 600d. Give them time :)
Colorist working with Davinci Resolve, Baselight, Nuke, After Effects & Premier Pro. Occasional Sunday afternoon DOP. Developer of Cinelog-C Colorspace Management and LUTs - www.cinelogdcp.com

kgv5

  • Hero Member
  • *****
  • Posts: 643
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #344 on: May 13, 2013, 01:09:27 PM »
Just erase the bootflag fir from the card and make the card bootable with eoscard.

That's how far I got... ML is running, but I can't find the raw features in the menues!?

in ML menu go to M tab, press "load modules now"
now go to movie tab and go all the way down - there is RAW video option.
www.pilotmovies.pl   5D Mark III, 6D, 550D

Stedda

  • Hero Member
  • *****
  • Posts: 504
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #345 on: May 13, 2013, 01:11:31 PM »
Hi,

I'm new to this stuff, never used ML firmware before. But I have an MK3 and want to test/help on this.

I  have the 1.1.3 firmware on the mk3 and i've downloaded the mk3 zip files posted by lourenco and extracted them to an empty lexar cf card.
Then I followed the installation wiki documentation (although I see it was not written for the mk3).
When initiating the firmware update process I see a message superimposed on the menu saying something like "enable bootflag :)" and nothing happens. On restart I have the original 1.1.3 firmware.

I know it's probable trivial but can somebody help?

Thank you,
Cristian

The steps are posted on the 5D III thread half a dozen times...
5D Mark III -- 7D   SOLD -- EOS M 22mm 18-55mm STM -- Fuji X-T1 18-55 F2.8-F4 & 35 F1.4
Canon Glass   100L F2.8 IS -- 70-200L F4 -- 135L F2 -- 85 F1.8 -- 17-40L --  40 F2.8 -- 35 F2 IS  Sigma Glass  120-300 F2.8 OS -- 50 F1.4 -- 85 F1.4  Tamron Glass   24-70 2.8 VC   600EX-RT X3

Africashot

  • Senior
  • ****
  • Posts: 288
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #346 on: May 13, 2013, 01:23:26 PM »
I have just started a new thread under Hardware and Accessories for anyone to share their experience using different brands and sizes of 1000x CF cards for RAW recording, it would be nice if anyone who has one and tested it could post.
http://www.magiclantern.fm/forum/index.php?topic=5391
Thanks in advance!
ML 5D2 & T3i

mcnys

  • New to the forum
  • *
  • Posts: 14
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #347 on: May 13, 2013, 01:24:10 PM »
Not as many as on a 5d MkIII ;)

I'm still waiting for my fast SD card to arrive. On a class10 SD it only manages continuous recording at the smallest frame settings but even then it drops/corrupts some frames and everything is tinted pink atm on my 600d. At bigger frame sizes you get only slightly more frames than Silentpic DNG bursts can do but the guys are working hard and it's very early days, especially for the 600d. Give them time :)

thanks for your answer! will have to wait then :) i just really hope that it will be possible for 600d :)

Yoshiyuki Blade

  • Member
  • ***
  • Posts: 103
  • 5D2
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #348 on: May 13, 2013, 01:27:04 PM »
Did some small adjustments and seems to work very well on 5D2.

Theoretical figures for the 1000x card, at 24fps, based on Coutts' benchmark (64MB/s):

- 1880x840 in full-frame mode (with line skipping)
- 1920x840 or 2152x720 in 1:1 crop mode (zero moire)

With my 266x card, I get 21-22MB/s, so... 720x720 :)

Is it safe to assume that the 5D2 has some kind of bottleneck before reaching 1000x speeds, considering that other cameras seem to allow much higher rates at that speed? I have a 600x sandisk card and the benchmark topped out somewhere around 50 MB/s for writing. I'll probably edit this post with the benchmark.

edit:

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12258
  • Emergencies only
Re: uncompressed YUV422 and 14-bit RAW video recording
« Reply #349 on: May 13, 2013, 01:31:22 PM »
It doesn't seem safe to assume anything - the benchmarks will tell you the truth. Run them in 24p movie mode, with global draw off, otherwise the numbers may not be relevant.