Author Topic: uncompressed 14-bit RAW video recording  (Read 1013189 times)

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D
Re: uncompressed 14-bit RAW video recording
« Reply #1375 on: May 21, 2013, 01:34:20 AM »
I know they have footers.. maybe only ones that stopped with the card might be missing them. Use the linux binary to convert them, its not working in windows/mac.

rodobot

  • New to the forum
  • *
  • Posts: 2
Re: uncompressed 14-bit RAW video recording
« Reply #1376 on: May 21, 2013, 07:48:31 AM »
First of all, thanks for great tools!

I am not a programmer, so this is probably not a correct way to do (In fact, I had a problem even with simple type casting ending up hardcoding..), but I hope this will help you iron out the issue with raw2dng.

First, I guessed that fseek does not support files larger than 4 gb on OSX, and after little googling, I decided to replace "fseek" with "fseeko".

I compiled and tried... failed...

Next, I guessed the second parameter of the first fseeko (i.e. -sizeof(lv_rec_file_footer_t)) is not casted correctly. After inserting printf's to see actual values, I found out that lv_rec_file_footer_t is 192.

So I replaced the parameter -sizeof(lv_rec_file_footer_t) with -192 and compiled (I don't know a correct way to cast sizeof to off_t (or long long?)).

Now, with these changes, raw2dng seems to be processing raw files larger than 4gb on OSX.

I used the file provided by zachnfine (thanks a lot!) and was able to extract 1416 files (to 001415.dng).

I have checked that 001415.dng opens normally using PhotoShop.

Again, thanks a lot for great tools and I hope this is useful.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: uncompressed 14-bit RAW video recording
« Reply #1377 on: May 21, 2013, 07:53:38 AM »
Great, can you check this changeset?

http://bitbucket.org/hudson/magic-lantern/commits/9c05522bc3ad

If it works, I'm going to update the exe on too.

mjneubrander

  • New to the forum
  • *
  • Posts: 23
Re: uncompressed 14-bit RAW video recording
« Reply #1378 on: May 21, 2013, 08:01:38 AM »
After installing build 006d890 May 19th Nightly for 5D mark III I have been infected with the 145 frame limit (at 720p) 5x zoom issue. After starting the build I enabled hack3d to test it. Next I tried the 5x zoom which stopped at 145 frames. I tested 4 previous builds (e91e2bc, f686b8c, and builds from may 15, 13) and the 145 frame zoom issue persists. I also changed sd cards and used eoscard to rebuild. Before using build 006d890, I can confirm working (centered only) 5x zoom on builds e91e2bc, f686b8c. This all lead me to believe that it changes a setting on the camera. Did I miss a fix for this? Side note digital dolly is great wouldn't mind being able to set the movement speed from a menu.

rodobot

  • New to the forum
  • *
  • Posts: 2
Re: uncompressed 14-bit RAW video recording
« Reply #1379 on: May 21, 2013, 08:03:23 AM »
I also changed fseek on Line 39 to fsseko.

But I must say that I don't know if this works on windows.

*********
I tested updated rwa2dng.c in repository, and it worked.
So I guess you don't need to replace 2nd one on Line 39 (Guess why I am not a programmer lol).

Danne

  • Developer
  • Hero Member
  • *****
  • Posts: 7295
Re: uncompressed 14-bit RAW video recording
« Reply #1380 on: May 21, 2013, 08:10:49 AM »
Great, can you check this changeset?

http://bitbucket.org/hudson/magic-lantern/commits/9c05522bc3ad

If it works, I'm going to update the exe on too.

I,d love to test this on my mac. But how  :)?

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: uncompressed 14-bit RAW video recording
« Reply #1381 on: May 21, 2013, 08:16:56 AM »
You can set the movement speed by pressing the arrow key once, twice etc. It starts at the minimal one possible right now (8 pixels at a time...)

It can be slowed down, but only by changing the converter (it needs metadata for every single frame). Not easy.

e91e2bc works and 006d890 doesn't? What exact settings?

The only backend difference is a 32MB write fix, which only makes a difference if the frame size (w * h * 14/8) is a divisor of 32MB.

mjneubrander

  • New to the forum
  • *
  • Posts: 23
Re: uncompressed 14-bit RAW video recording
« Reply #1382 on: May 21, 2013, 08:45:14 AM »
After installing 006d890 no previous build works in zoom not sure if there are newer builds to test :(.

Should I try pull the backup battery or is that pointless?

Same settings Global draw, zebras, spotmeter, histogram all off. Sound set as sync beep video set 1280x720 using a x400 card with 36Mb write (tested in camera). When I start a raw record in zoom it only shows (...) for ram instead of (....) when in non zoom.

Found the arrow key multiple press thanks ! Now if I just had smoother hands :).


a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: uncompressed 14-bit RAW video recording
« Reply #1383 on: May 21, 2013, 08:49:45 AM »
There was some slowdown in 5x after fixing the tearing (this required some more memory). Just pushed a fix that should restore the old speed (almost).

scrax

  • Contributor
  • Hero Member
  • *****
  • Posts: 1543
  • Code monkey
Re: uncompressed 14-bit RAW video recording
« Reply #1384 on: May 21, 2013, 09:08:18 AM »
Great, can you check this changeset?

http://bitbucket.org/hudson/magic-lantern/commits/9c05522bc3ad

If it works, I'm going to update the exe on too.
working here too!
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus | EOS 5D MLbeta5- EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX, PS, LR, RawTherapee, LightZone -no video experience-

mjneubrander

  • New to the forum
  • *
  • Posts: 23
Re: uncompressed 14-bit RAW video recording
« Reply #1385 on: May 21, 2013, 09:24:12 AM »
I was also able clear the 145 frame issue using the files from below. And starting a zoom raw rec from any C1, C2 ,C3 modes but not the M mode.
5D3 update files. Replace existing files. 
https://drive.google.com/folderview?id=0B7QlH_BH2m32QjRhRDhUeVBwZmM&usp=sharing
includes cf_acc: module as noted on http://www.magiclantern.fm/forum/index.php?topic=5471.msg36588#msg36588
Then as any good tester I was able to repeat the issue from the 006d890 build. And repeat the fix!

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12564
Re: uncompressed 14-bit RAW video recording
« Reply #1386 on: May 21, 2013, 09:27:49 AM »
How much free memory do you have in C modes? (in Debug menu).

I have 168MB, sometimes 88.

mjneubrander

  • New to the forum
  • *
  • Posts: 23
Re: uncompressed 14-bit RAW video recording
« Reply #1387 on: May 21, 2013, 10:12:01 AM »
Showing 168 M in both Manual and C modes for both 006d890 and e91e2bc.

Trying to pin down how this error is made can't seem to get it yet 60+ tests and configurations. It was repeatable from the memory card with 006d890 but then I rebuilt the ML on that card and it works...

noisyboy

  • Guest
Re: uncompressed 14-bit RAW video recording
« Reply #1388 on: May 21, 2013, 10:56:41 AM »
First of all, thanks for great tools!

I am not a programmer, so this is probably not a correct way to do (In fact, I had a problem even with simple type casting ending up hardcoding..), but I hope this will help you iron out the issue with raw2dng.

First, I guessed that fseek does not support files larger than 4 gb on OSX, and after little googling, I decided to replace "fseek" with "fseeko".

I compiled and tried... failed...

Next, I guessed the second parameter of the first fseeko (i.e. -sizeof(lv_rec_file_footer_t)) is not casted correctly. After inserting printf's to see actual values, I found out that lv_rec_file_footer_t is 192.

So I replaced the parameter -sizeof(lv_rec_file_footer_t) with -192 and compiled (I don't know a correct way to cast sizeof to off_t (or long long?)).

Now, with these changes, raw2dng seems to be processing raw files larger than 4gb on OSX.

I used the file provided by zachnfine (thanks a lot!) and was able to extract 1416 files (to 001415.dng).

I have checked that 001415.dng opens normally using PhotoShop.

Again, thanks a lot for great tools and I hope this is useful.

Nice one man! Would be nice to see to see this ported to the Windows .exe :)

Edit:
If it works, I'm going to update the exe on too.

*slaps forhead* reading thread properly FAIL. Sorry ;)[/i]

lourenco

  • Member
  • ***
  • Posts: 101
Re: uncompressed 14-bit RAW video recording
« Reply #1389 on: May 21, 2013, 01:09:51 PM »
Nice one man! Would be nice to see to see this ported to the Windows .exe :)

Edit:
*slaps forhead* reading thread properly FAIL. Sorry ;)

There is a new one for windows too now. It works fine over 4GB now. It was able to convert my 8GB raw movie file.
https://bitbucket.org/hudson/magic-lantern/downloads/RAw2dng.exe
5D Mark III, CF Lexar 1000X 32GB, 24-105 F4L

tihon

  • Freshman
  • **
  • Posts: 74
Re: uncompressed 14-bit RAW video recording
« Reply #1390 on: May 21, 2013, 01:19:30 PM »
Hi! Do you think that is possible to have  a raw zebra?   As i know blackmagic have it.

Thanks! I understand that all pros uses an histogram, but...
Cinema, cinema, cinema

marcb

  • New to the forum
  • *
  • Posts: 9
Re: uncompressed 14-bit RAW video recording
« Reply #1391 on: May 21, 2013, 01:20:07 PM »
Is it possible to simultaneously record H.264 proxies to an SD card while recording RAW to the CF on the 5DIII? The proxies would be very useful for offline editing and have the added advantage that sound is embedded in the file.

djoulbdx

  • New to the forum
  • *
  • Posts: 25
  • Wine before Camera - ML before Wine
Re: uncompressed 14-bit RAW video recording
« Reply #1392 on: May 21, 2013, 02:03:27 PM »
Hi,

I am aware of being boring ... I wanted to know if it will be possible on the 7D. You've explained to me the problem of slow bus between the two chips, but I can not resign myself. But I try;-)

g3gg0, Alex and the other ones : Thx.
EOS 7D ML | EOS 5 (not D) | 15-85 3.5 5.6 | 580 EX | 28-105 4.5 5.6 | 100-300 | etc...

dude

  • Member
  • ***
  • Posts: 205
Re: uncompressed 14-bit RAW video recording
« Reply #1393 on: May 21, 2013, 03:19:49 PM »
The fact that you are not able to READ (especially the first and most important page)

 (- bloody beginners and non-geeks should not touch the whole thing. wait until it is "beginner-proof". we will tell you on the website.
 - you know that you are a bloody beginner, when you read the whole thread and you still cannot get it to work.
 - DON'T be disappointed if it doesn't work or we figure out the whole thing is unstable and/or unusable
 - NO, there is no manual yet
 - NO, there is no all-in-one tool that fits every use case
 - NO, we don't have tutorials how to use it
 - NO, not all models are supported yet ;)
 - we are just at the moment testing how good it works and what we have missed and what to improve
 - you are welcome to post comparisons, experiences (both good and bad), or even deep analysis or just cool videos
 - if you are a programmer and you see potential for improvements, grab the source and support :) )
doesn t make your begging better...

djoulbdx

  • New to the forum
  • *
  • Posts: 25
  • Wine before Camera - ML before Wine
Re: uncompressed 14-bit RAW video recording
« Reply #1394 on: May 21, 2013, 03:34:45 PM »
The fact that you are not able to READ (especially the first and most important page)
 - NO, not all models are supported yet ;)

"Yet" : That's why I prefer asking for the 7D. I want to be the first  ;)
EOS 7D ML | EOS 5 (not D) | 15-85 3.5 5.6 | 580 EX | 28-105 4.5 5.6 | 100-300 | etc...

dngrhm

  • Freshman
  • **
  • Posts: 83
Re: uncompressed 14-bit RAW video recording
« Reply #1395 on: May 21, 2013, 04:09:36 PM »
What if the footer was changed into a header?  Write a default header to the file.  Write the sequence of frames. After manual stop or stop on card-full, go back and overwrite the header at the beginning of the file.

This would eliminate the issue of not having space to write the footer if the card is full.  This would also put the info needed to read the file at the beginning instead of having to fseek to the end.  It might also make it easier to fix the header if needed.

This would kill compatibility with anything anyone has already done. :(
EOS 650D + 620 | Canon EF-S 18-135mm f/3.5-5.6 STM + 55-250mm f/4-5.6 | Canon EF 50mm f/1.8
Mac OS 10.9 | PinkDotRemoval Tool | RAWMagic | DaVinci Resolve | FCP X

1%

  • Developer
  • Hero Member
  • *****
  • Posts: 5936
  • 600D/6D/50D/EOSM/7D
Re: uncompressed 14-bit RAW video recording
« Reply #1396 on: May 21, 2013, 04:29:45 PM »
I think its at the end because it provides a frame count.

dngrhm

  • Freshman
  • **
  • Posts: 83
Re: uncompressed 14-bit RAW video recording
« Reply #1397 on: May 21, 2013, 04:56:55 PM »
I think its at the end because it provides a frame count.

Yes and frame skips.  This is what makes the overwrite of the header after the recording is done necessary.  I don't see any reason in the code and how ML writes files why the file can't be closed then reopen and pass that file pointer to lv_rec_save_footer.  (I could be wrong on that) Otherwise you have to get into some tap dancing to make sure there is enough space on the card to write the footer. 
EOS 650D + 620 | Canon EF-S 18-135mm f/3.5-5.6 STM + 55-250mm f/4-5.6 | Canon EF 50mm f/1.8
Mac OS 10.9 | PinkDotRemoval Tool | RAWMagic | DaVinci Resolve | FCP X

AndreasK

  • New to the forum
  • *
  • Posts: 42
Re: uncompressed 14-bit RAW video recording
« Reply #1398 on: May 21, 2013, 05:04:25 PM »
hmm just received my 64gb transcend and I can not record 1080p25 witout skipped frames :(
when i do the benchmark i can get >100mb/sec with a different buffer. is there a way to adjust the buffer the raw encoding uses?

Habitat

  • New to the forum
  • *
  • Posts: 36
Re: uncompressed 14-bit RAW video recording
« Reply #1399 on: May 21, 2013, 05:17:01 PM »
working here too!

Scrax - think you can get it working for us mac users?



*edit scrax already got it goin on - http://www.magiclantern.fm/forum/index.php?topic=5508.0