Headline news

An announcement regarding the forking of Magic Lantern, and the support that can be provided for these forks on this forum.

You can find our live webchat here: [email protected]

Author Topic: uncompressed 14-bit RAW video recording  (Read 233345 times)

squig

  • Moderators
  • Senior
  • *****
  • Posts: 416
  • Crash test dummy MK3
uncompressed YUV422 and 14-bit RAW video recording
« Reply #225 on: May 12, 2013, 03:25:25 AM »
Cool! Have you tried filling up the card and if so does the MK3 start to smoke?
Reporter to Mahatma Gandhi: "Mr. Gandhi, what do you think of Western Civilisation?"   Gandhi: "I think it would be a good idea." https://vimeo.com/rollingshutter

lourenco

  • Freshman
  • **
  • Posts: 98
uncompressed YUV422 and 14-bit RAW video recording
« Reply #226 on: May 12, 2013, 03:40:09 AM »
Cool! Have you tried filling up the card and if so does the MK3 start to smoke?

I did a test to see what would happen if I filled the memory card. It records the first file fine.  It keeps recording until the memory card is full, but I am having strange results after the first file. I just sent a1ex a message about it. 
5D Mark III, CF Lexar 1000X 32GB, 24-105 F4L

squig

  • Moderators
  • Senior
  • *****
  • Posts: 416
  • Crash test dummy MK3
uncompressed YUV422 and 14-bit RAW video recording
« Reply #227 on: May 12, 2013, 04:15:42 AM »
Big congrats and thx to all the devs, this is huge for digital indie cinema. Beware the Zaibatsu Ninja.
Reporter to Mahatma Gandhi: "Mr. Gandhi, what do you think of Western Civilisation?"   Gandhi: "I think it would be a good idea." https://vimeo.com/rollingshutter

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5888
  • 600D/6D/50D/EOSM/7D
uncompressed YUV422 and 14-bit RAW video recording
« Reply #228 on: May 12, 2013, 04:37:35 AM »
Figured out how to get pic sizes... 6D its 1920 exactly. Weird, eh? Sped up to 22MB/s too when the size was corrected.

squig

  • Moderators
  • Senior
  • *****
  • Posts: 416
  • Crash test dummy MK3
uncompressed YUV422 and 14-bit RAW video recording
« Reply #229 on: May 12, 2013, 04:50:21 AM »
So is 1280x720 60p RAW now doable too? Or 48p?
Reporter to Mahatma Gandhi: "Mr. Gandhi, what do you think of Western Civilisation?"   Gandhi: "I think it would be a good idea." https://vimeo.com/rollingshutter

kgv5

  • Senior
  • ****
  • Posts: 474
uncompressed YUV422 and 14-bit RAW video recording
« Reply #230 on: May 12, 2013, 06:57:26 AM »
Figured out how to get pic sizes... 6D its 1920 exactly. Weird, eh? Sped up to 22MB/s too when the size was corrected.

You mean 1920 instead of 1840 in RAW mode? Sped up to 22 MB/s so what was the speed before correction?
5D Mark III, 6D, 550D

noisyboy

  • Moderators
  • Hero Member
  • *****
  • Posts: 613
  • 6D. 7D.
uncompressed YUV422 and 14-bit RAW video recording
« Reply #231 on: May 12, 2013, 07:45:01 AM »
Big congrats and thx to all the devs, this is huge for digital indie cinema. Beware the Zaibatsu Ninja.

Yup! Seriously well done! Amazing effort the ML wizards!  8)
Lynne, some of these people have come from Stoke!

Kabuto1138

  • Freshman
  • **
  • Posts: 84
uncompressed YUV422 and 14-bit RAW video recording
« Reply #232 on: May 12, 2013, 08:56:01 AM »
Nice guys!  great job!

This is only working in the 6D, right?

N.Mendes

  • New to the forum
  • *
  • Posts: 38
uncompressed YUV422 and 14-bit RAW video recording
« Reply #233 on: May 12, 2013, 09:33:46 AM »
Ok, Merry Christmas all..


It keeps recording until the memory card is full, but I am having strange results after the first file.

Did you try to use CF+SD ? In this case, maybe the second file will be as clean as the first?
About the aspect ratio, again, maybe "CF+SD" will allow to shoot 1920x1080 (or more?)

ps: your test is crazy, in term of detail and dynamic range, really really crazy, thanks dude..

« Last Edit: May 12, 2013, 09:36:35 AM by N.Mendes »

Africashot

  • Member
  • ***
  • Posts: 233
uncompressed YUV422 and 14-bit RAW video recording
« Reply #234 on: May 12, 2013, 09:51:02 AM »
Any chance of getting anywhere near 400 frames at 24 fps with the good old 5D2? Sorry for asking, I know you guys are working on it and I don't want to be pushy but man this is so exciting I just couldn't hold myself and if there is only the slightest chance it would be way beyond what I had ever dared to dream of!
ML 5D2 & T3i

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 6748
  • 5D Mark Free
uncompressed YUV422 and 14-bit RAW video recording
« Reply #235 on: May 12, 2013, 09:58:32 AM »
Run the benchmarks and post screenshots. I don't remember seeing any benchmarks from 1000x cards with 5D2.

Africashot

  • Member
  • ***
  • Posts: 233
uncompressed YUV422 and 14-bit RAW video recording
« Reply #236 on: May 12, 2013, 10:48:57 AM »
Run the benchmarks and post screenshots. I don't remember seeing any benchmarks from 1000x cards with 5D2.
Thanks Alex! Will try when my 1000x card gets here, shipping to Africa takes eternities... :(
ML 5D2 & T3i

arrinkiiii

  • Hero Member
  • *****
  • Posts: 937
  • ML ADDICTED 7D
uncompressed YUV422 and 14-bit RAW video recording
« Reply #237 on: May 12, 2013, 11:03:42 AM »
!!! AMAZING !!!

Anybody thinks that this will run on the 7D?

« Last Edit: May 12, 2013, 11:12:11 AM by arrinkiiii »

Andy600

  • Contributor
  • Hero Member
  • *****
  • Posts: 1297
  • Have you tried turning it off and on again?
uncompressed YUV422 and 14-bit RAW video recording
« Reply #238 on: May 12, 2013, 11:25:38 AM »
Super impressive stuff. Seriously well done guys!

@1% have you tried this on the 600d yet? I know it probably wont handle what the 6d and Mk3 can but I'm eager to know what, if any chance there is of pushing the shot length up... even if only by a few seconds.
600d & 50d / Manual lenses / Post in After Effects, Premier Pro and Davinci Resolve. Brit living in Latvia. Developer of CinelogDCP www.cinelogdcp.com

RenatoPhoto

  • Moderators
  • Hero Member
  • *****
  • Posts: 1409
  • 5DM3 / 7D
uncompressed YUV422 and 14-bit RAW video recording
« Reply #239 on: May 12, 2013, 02:06:46 PM »
Congratulations!
You are the Nobel prize winners for arm development period!
Thanks for your amazing work.
Renato
http://www.pululahuahostal.com  |  EF 300 f/4, EF 100-400 L, EF 180 L, EF-S 10-22, Samyang 14mm, Sigma 28mm EX DG, Sigma 8mm 1:3.5 EX DG, EF 50mm 1:1.8 II, EF 1.4X II, Kenko C-AF 2X

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 2148
uncompressed YUV422 and 14-bit RAW video recording
« Reply #240 on: May 12, 2013, 02:20:24 PM »
please dont ask about too many models. we are aware that 550d, 50d, 60d, 6d, 600d, 5d2, 7d, etc are also widely used.
before we start analyzing these models, we want prove our theory by making a reference implementation.

as soon this reference implementation is proven stable, we will advance to these models. ;)

current state: trying to make the code as performant, stable and portable as possible.

minimimi

  • Contributor
  • Freshman
  • *****
  • Posts: 68
uncompressed YUV422 and 14-bit RAW video recording
« Reply #241 on: May 12, 2013, 02:56:22 PM »
Just added a RAW to DNG converter: https://bitbucket.org/hudson/magic-lantern/commits/5bc3488c7a78

It's just a small command-line tool. Feel free to build a nice GUI on top of it.

For some reason, I couldn't compile it in 64-bit mode, so you may need the 32-bit compatibility libraries. Fixes are welcome ;)

With a little luck, this program may work under Windows too.


Alex, Are you forget to add this ?(when the file size reached 2Gig , then rotation the file)
I'm tried to concatinate by cat command, but something wrong. (4gig of file not supported??)
And cat need a lot of time and disk spaces. So I think each file must have footer info.

Code: [Select]
diff -r 5bc3488c7a78 modules/lv_rec/lv_rec.c
--- a/modules/lv_rec/lv_rec.c   Sat May 11 14:35:52 2013 +0300
+++ b/modules/lv_rec/lv_rec.c   Sun May 12 21:51:27 2013 +0900
@@ -649,6 +649,8 @@
                     {
                         yPos++;
                         bmp_printf( FONT(FONT_MED, COLOR_WHITE, COLOR_BLACK), 30, 20 * yPos++, "Creating next file");
+                       save_data.frameCount = 1;
+                       lv_rec_save_footer(save_data.handle, &save_data);
                         FIO_CloseFile(save_data.handle);
                         save_data.handle = NULL;
                         lv_rec_update_suffix(&save_data);

g3gg0

  • Developer
  • Hero Member
  • *****
  • Posts: 2148
uncompressed YUV422 and 14-bit RAW video recording
« Reply #242 on: May 12, 2013, 03:16:37 PM »
the lv_rec code basically is my work, alex updated it to the latest research results.
so i will answer to your question.

i wanted the split file to be really "split" just as if they were splitted by common split and merge utilities.
this means the payload is split over several files and at the end we have one footer that contains information for all frames.

simply concatenate the files to have a big movie file.

the reason for a footer is simple: in YUV mode you can process the files with the existing tools, you dont have to cut the footer away.
if i had added a header, it wont be that simple - you have to remove the header before you can use old tools.

when the tools are finished, it is as simple as specifying the first file M000000.RAW and it will autodetect that there is a .R00, R01 etc
and will virtually address them as one file.


1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5888
  • 600D/6D/50D/EOSM/7D
uncompressed YUV422 and 14-bit RAW video recording
« Reply #243 on: May 12, 2013, 03:20:07 PM »
Quote
@1% have you tried this on the 600d yet?

Almost... I got a size of 1652 but the YUV buffer is 1728... when I open the files they are still messed up. Maybe I need to pull a bigger buffer than 2k and run image analysys again... that or sync is fucked?

Have to check on 720P mode, the size might be different. If it is, should pick based on YUV buffer size... like if buffer is 1280 then raw video is XXX, etc.

Quote
the lv_rec code basically is my work,

This in the new memecpy is freaking awesome :)

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 6748
  • 5D Mark Free
uncompressed YUV422 and 14-bit RAW video recording
« Reply #244 on: May 12, 2013, 03:34:49 PM »
Heh, people got excited without even knowing the big news: g3gg0 just discovered how to use the DMA cropping routines, which just made possible RAW video recording at 1920x1080 at 24fps on 1000x cards.

Technical: we now know how to copy a cropped version of some image buffer at very high speeds (over 700MB/s), and with this trick we can save the video data the card at full speed, without being slowed down by image borders, for example.

1920x1080 RAW video now requires 83MB/s at 24fps, so it should work just fine on 1000x cards. I didn't try it.

So, I've lost my patience and rewritten the lv_rec module from scratch, to use these new routines and to experiment with different buffering algorithms. The new module is called raw_rec and outputs the same file format (RAW files).

Main changes:

- The ring buffer only uses 32MB memory blocks (maximum we can get). Reason: card benchmarks showed higher data rates for large buffers.
- Frame copying is done outside the LiveView task (not sure if it has any effect).
- When the buffer gets full, it skips some frames, rather than stopping.
- Fewer hardcoded things: should be easier to port.
- Resolution presets, from 640x320 to 3592x1320.

Just like lv_rec, this is in very early stages, so you have to compile it yourself.

Source code: https://bitbucket.org/hudson/magic-lantern/commits/54537cb85d7d

If you try it, I'd like you to look for any signs of image tearing. The source raw data is single-buffered, but it's possible to make it double-buffered if the vertical sync is less than ideal.

All credits go to g3gg0 - without his reverse engineering work on understanding the image processor, this would have been impossible.
« Last Edit: May 12, 2013, 03:43:53 PM by a1ex »

minimimi

  • Contributor
  • Freshman
  • *****
  • Posts: 68
uncompressed YUV422 and 14-bit RAW video recording
« Reply #245 on: May 12, 2013, 03:41:03 PM »
Wrote reply but alex is making a code from scratch,,,,, ignore this.........


i wanted the split file to be really "split" just as if they were splitted by common split and merge utilities.
this means the payload is split over several files and at the end we have one footer that contains information for all frames.

simply concatenate the files to have a big movie file.
I already read and understand your codes.
But , concatenate time is really long.
So I think if the all of files has footer info, we don't need time to concatenate files and tempolary disk spaces.
Also , I'm tried to concat by cat command, I don't know why, but fopen failed (3.6Gig file on 32bit linux).
(?? cat command broken files???)

the reason for a footer is simple: in YUV mode you can process the files with the existing tools, you dont have to cut the footer away.
if i had added a header, it wont be that simple - you have to remove the header before you can use old tools.

when the tools are finished, it is as simple as specifying the first file M000000.RAW and it will autodetect that there is a .R00, R01 etc
and will virtually address them as one file.
If we can cut footer , we have no problem to use old tools when each files has footer info.

It's my thinking.

Stedda

  • Supporter
  • Senior
  • *****
  • Posts: 326
uncompressed YUV422 and 14-bit RAW video recording
« Reply #246 on: May 12, 2013, 03:49:41 PM »
Heh, people got excited without even knowing the big news: g3gg0 just discovered how to use the DMA cropping routines, which just made possible RAW video recording at 1920x1080 at 24fps on 1000x cards.

True wizards! Nice work guys!
5D Mark III -- 7D   SOLD -- EOS M 22mm 18-55mm STM -- Fuji X-T1 18-55 F2.8-F4 & 35 F1.4
Canon Glass   100L F2.8 IS -- 70-200L F4 -- 135L F2 -- 85 F1.8 -- 17-40L --  40 F2.8 -- 35 F2 IS  Sigma Glass  120-300 F2.8 OS -- 50 F1.4 -- 85 F1.4  Tamron Glass   24-70 2.8 VC   600EX-RT X3

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5888
  • 600D/6D/50D/EOSM/7D
uncompressed YUV422 and 14-bit RAW video recording
« Reply #247 on: May 12, 2013, 04:01:28 PM »
Quote
Source code: https://bitbucket.org/hudson/magic-lantern/commits/54537cb85d7d

Going to try it... old LV_REC would crash in 720P raw because of fixed size.

Ok, just tested on 600D, thats the repo I have open :)... 32mb is too big to allocate here, tried 16, going to try 24. Wish the skipping was optional, sometimes you want continous frames for less time. Stopping hangs up the ML menu/camera for some reason, LV still moves and I think you can hit canon stuff. Once you switch modes LV won't go away. Have to check this out. Going to see if files are readable with the 1740 preset and try on 6D.


« Last Edit: May 12, 2013, 04:15:53 PM by 1% »

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 6748
  • 5D Mark Free
uncompressed YUV422 and 14-bit RAW video recording
« Reply #248 on: May 12, 2013, 04:13:00 PM »
Don't forget to try "don't click me" - g3gg0's demo is really cool.

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5888
  • 600D/6D/50D/EOSM/7D
uncompressed YUV422 and 14-bit RAW video recording
« Reply #249 on: May 12, 2013, 04:17:02 PM »
On 600D the demo lagged, on 6D it was perfectly fluid. Image would move back and forth... just there was no way to stop it :)

Oh man,,, the raw file is 640MB... I just recorded a few sec. 296 frames... 15MB/s @16MB malloc.. will try 24MB. Its funny but 600D/6D sd rates are very very close, maybe with the larger writing it will finally crack 30MB/s

1740 its like the files are off by 1 or 2 pixels, slightly skewed to the side. Doing img.py

600D size is 1738.

Ok, more testing: LV_REC will not record the 1738 files, they come out all pink and screwed up, why?
Raw_rec does a good job but ending is screwed up and for some reason the resolution is detected as 1740 and when manually fixed the files say unexpected end of file. Gimp can open them, they look like this:

« Last Edit: May 12, 2013, 05:08:49 PM by 1% »

 

courtesy