Magic Lantern Forum

Using Magic Lantern => Raw Video => Topic started by: pravdomil on May 24, 2013, 10:16:25 PM

Title: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: pravdomil on May 24, 2013, 10:16:25 PM
Merged to a_d_'s builds
http://www.magiclantern.fm/forum/index.php?topic=5533.0 (http://www.magiclantern.fm/forum/index.php?topic=5533.0)

#########################

(https://bitbucket.org/pravdomil/ml/downloads/lv.png)

(https://bitbucket.org/pravdomil/ml/downloads/menu.png)

Features
+ selecting, deleting, playing
+ movie files are separated into 3.7GB chunks -> unlimited recording
+ raw rec is enabled by default
+ global redraw and auto wb is turned off while recording (hacked mode)
+ advanced cropmarks, that cover all outside recording rect
+ start recording by pressing joy center not LV

Future
- playback frame skipping

Know bugs
- you have to restart your camera to refresh Canon's used bar in format menu
- before playback you have to set correct dimensions of video

What is not tested
- zoom mode
- sound rec
- panning mode

4GB+ on Win
copy /B file1+file2 destfile
copy /B M0000000.RAW+M0000000.RAW002+M0000000.RAW003... OUT.RAW

4GB+ in Totalcmd
rename M0000000.RAW to M0000000.RAW001
then use totalcmd's built-in feature "join files" under file menu

4GB+ on Mac
cat file1 file2 > destfile
cat M0000000.RAW M0000000.RAW002 M0000000.RAW003... > OUT.RAW

Not having 5D2?
Well, there is no big difference between original and this module.
You can try it - no warranty!
But there is hardcored 3.7GB limit.

Download
a.d.'s ML
http://www.magiclantern.fm/forum/index.php?topic=5533.0

Download the newest raw2dng, that can process big files
http://www.magiclantern.fm/forum/index.php?topic=5404.0

Error: This ain't a lv_rec RAW file
Is mostly cause by older version of raw2dng

Instructions
http://5dfilmmaking.com/05/how-to-shoot-raw-video-on-your-5d2/

Thanks to all ML people!
No warranty!


Please help me implement 4GB+ support to raw2dng
Title: Re: Another raw_rec.mo for 5D2 (bypass 4GB limit, deleting videos)
Post by: 1% on May 24, 2013, 10:31:10 PM
I see all the buffer stars you got and I'm jealous :)


Everyone with cards worth their salt on the SD side is using xfat so cutting files is not needed and raw2dng now opens any size file.

But turning off global draw on record in hacked mode is something I'm down with. Do you know if AEWB disable affects AF button?

*I see that aewb + gd already being disabled in hacked mode according to latest.
Title: Re: Another raw_rec.mo for 5D2 (bypass 4GB limit, deleting videos)
Post by: a.d. on May 24, 2013, 10:47:55 PM
My CF Card is too slow for that, however, Good Job!
Title: Re: Another raw_rec.mo for 5D2 (bypass 4GB limit, deleting videos)
Post by: pravdomil on May 24, 2013, 11:03:03 PM
I got with Lexar 1000x 64GB ~64MB/s.
AF button works normally with AEWB disable.
Title: Re: Another raw_rec.mo for 5D2 (bypass 4GB limit, deleting videos)
Post by: 1% on May 24, 2013, 11:05:34 PM
The deleting i'll def merge... I can't just format the card to delete 1 file.
Title: Re: Another raw_rec.mo for 5D2 (bypass 4GB limit, deleting videos)
Post by: JackDaniel412 on May 25, 2013, 12:24:53 AM
WOW! The best RAW mudule I've ever tried!!!
Really great job!!!
Title: Re: Another raw_rec.mo for 5D2 (bypass 4GB limit, deleting videos)
Post by: Kabuto1138 on May 25, 2013, 12:35:06 AM
NIIICEEEEE. 
Title: Re: Another raw_rec.mo for 5D2 (bypass 4GB limit, deleting videos)
Post by: 1% on May 25, 2013, 01:01:29 AM
Pulled the crop marks from this with an on/off option. It looks cool when run with LV hacking.

Maybe add another oft requested feature.. total seconds recorded.
Title: Re: Another raw_rec.mo for 5D2 (bypass 4GB limit, deleting videos)
Post by: Hazer on May 25, 2013, 02:17:16 AM
Pravdomil, this is fantastic work.  Fastest build yet for the 5D2, by a significant margin.  And file spanning to boot.

I just did 2000 frames at 1720x960, write speeds of 64-65MB/sec on a Lexar 32GB 1000x.  That's beyond the 4GB limit.  As of yesterday I was seeing 59-60MB/sec.  Again, bravo.

Couple questions:

1.  Does this incorporate the speed enhancements from today's (the 24th) build?  Because as of a few minutes ago, that was also the fastest build yet.  63-64MB/sec on my card.  If you haven't incorporated this yet, are we likely to see even further speed improvements when you do?

2.  I'm on Mac, so I can't use totalcmd.  I just tried cat (cat raw1 raw2 > raw3) and sed (sed '' raw1 raw2 > raw3), but this didn't work.  Raw2dng said "this ain't no lv_rec".  These two methods will typically both work on binary files.  How do the spanned clips need to be joined?

Regardless it is clear that whatever optimizations are present here should certainly be merged into the standard tree.  This has the potential to make previously marginal resolutions reliably usable, and that's big.

Thanks!


Edit:  65-66MB/sec on my Komputerbay 1000x.  Hot.
Title: Re: Another raw_rec.mo for 5D2 (bypass 4GB limit, deleting videos)
Post by: Yoshiyuki Blade on May 25, 2013, 03:25:13 AM
Very nice indeed! 1880x800 lasted almost 2200 frames and 5.37 GB on my Sandisk 16GB 600x card (averaged above 58 MB/s which is a little better than it was before). I haven't checked out the frames yet though. Great job and thanks!

Just recorded a 14.8 GB clip at (almost) the usual stable resolution, which maxed out the card. Gonna see how continuous recording looks, though it'll take a while lol.
Title: Re: Another raw_rec.mo for 5D2 (bypass 4GB limit, deleting videos)
Post by: pacman829 on May 25, 2013, 09:09:49 AM
nice, will test this out in the morning
Title: Re: Another raw_rec.mo for 5D2 (bypass 4GB limit, deleting videos)
Post by: pacman829 on May 25, 2013, 09:34:53 AM
Just did a quick test with your module and build bff47c520033

Worked pretty well, I like the new crop marks

Playback wasnt working for me , it starting to kind of strobe on top of live view and when i tried to shut it off it went away but left me stuck in live view ... had to pull the battery


also no more hacked mode? was just curious if its always on now, or if the feature was just removed


thanks again for taking the time to perfect this
Title: Re: Another raw_rec.mo for 5D2 (bypass 4GB limit, deleting videos)
Post by: savale on May 25, 2013, 10:32:18 AM
Great build. I love the 4gb limit fix. I stopped manually after 4000 frames @ 1880x800 @24fps
1880x940 (1:2) I could record 417 frames.

komputerbay 64gb 1000x (algined at 4096)
Title: Re: Another raw_rec.mo for 5D2 (bypass 4GB limit, deleting videos)
Post by: bumkicho on May 25, 2013, 12:59:39 PM
Can this be used with 5D Mark III?
Title: Re: Another raw_rec.mo for 5D2 (bypass 4GB limit, deleting videos)
Post by: silvertonesx24 on May 25, 2013, 05:11:00 PM
Wow, really great work. Exactly the direction that I was looking for. Fastest build yet with a.d.'s May 24th firmware- 781 frames @ 1880x940 32gb Komputerbay 1000x. First time I used it it didn't disable crop marks and zebras on global draw, but now it seems to be working fine. Not sure if I like the joystick as raw trigger, but I guess I'll get used to it.

Not sure which part this is covered under, but it'd be nice to have a more accurate readout of space left on the card. Each time I run out of space it's a surprise- it seems as though both the Format in the Canon menu and the ML readout in the top right do not actively update and lag behind.
Title: Re: Another raw_rec.mo for 5D2 (bypass 4GB limit, deleting videos)
Post by: saint33d on May 25, 2013, 05:24:20 PM
if you half press the shutter release button it should tell you the details. Not sure if it will work in this mode though.
Title: Re: Another raw_rec.mo for 5D2 bypass 4GB, selecting, deleting, playing videos
Post by: 1% on May 25, 2013, 05:35:02 PM
The overlays fight with ML infos :(

I have GD:ON and then it turns off automatically when I start writing.
Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: pravdomil on May 25, 2013, 05:41:25 PM
next update, look above
Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: 1% on May 25, 2013, 06:16:06 PM
Ok, going to add the deleting stuff... on 600D the cropmarks caused a performance hit :( on 6D just fought with the overlays.
Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: Kabuto1138 on May 25, 2013, 07:13:08 PM
Hey Guys,

Is this working on yesterday's build only?  I tried it on today's and can't record pass 4gb or delete clips.

thanks so much!
Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: 1% on May 25, 2013, 07:58:06 PM
I added deleting... files get "deleted", file name left but file is 0KB. The playback is working great tho.

*Ok fixed...

what was the point of that loop and creating a dummy file? There are no file chunks. or is that for 3.7gb?

Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: Kabuto1138 on May 25, 2013, 11:44:39 PM
Quote from: 1% on May 25, 2013, 07:58:06 PM
I added deleting... files get "deleted", file name left but file is 0KB. The playback is working great tho.

*Ok fixed...

what was the point of that loop and creating a dummy file? There are no file chunks. or is that for 3.7gb?



Sorry for the stupid question, but I am kind of lost here.  These changes where do they go? the next nightly build?  are these new features talked about in this thread included in the next build? or do we have to put them manually?

thanks so much guys, exciting times here at ML
Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: 1% on May 25, 2013, 11:47:48 PM
Went into 600D/6D
Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: g3gg0 on May 26, 2013, 12:16:42 AM
can you try to stick to this naming scheme?
-> M0000000.RAW, M0000000.R00, M0000000.R01, M0000000.R02, etc

then it would match 8.3 and the previous lv_rec scheme.
thanks.
Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: pravdomil on May 26, 2013, 01:11:37 AM
yes, I will do it tomorrow evening.
Could you merge it to official trunk?
Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: 1% on May 26, 2013, 01:31:09 AM
I don't want cutting of gb tho and having to merge files. That's an additional step for me and 5-10, etc gb files are fine.
Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: pacman829 on May 26, 2013, 07:29:12 AM
though i'd post this here as well since i was using this build:

Also, just ran into another bug while trying to record raw with an external (Ikan VL5) Monitor hooked up to the 5d

The screenshot didnt capture the footage from liveview, but as you can see , the raw cropmarks/overlay is off to one side (panning is turned off) and when i try to play back the raw it doesnt work either  and freezes/crashes:

(http://i40.tinypic.com/20zy2o1.jpg)

When i disconnect the monitor and try to play back the footage all i get is raw "white noise"

Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: pacman829 on May 26, 2013, 07:42:09 AM
Anyone here get the dngs to work with premiere pro ?

I can use them fine in after effects , but they dont come up in premiere
Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: xNiNELiVES on May 26, 2013, 08:18:50 AM
Is anyone here using SD cards? I'm stuck with them with my 6D :'(. So angry...
Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: saint33d on May 26, 2013, 09:10:30 AM
Well seeing as this is a thread for 5d2 and 5d2 only have cf ports, ima go with no :P
Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: nannini on May 26, 2013, 05:36:41 PM
Thanks for this build. I have tried it with my 5D2 and discovered that the movie files separation into 3.7GB chunks is not working. Could someone give me a hint please where to look at to get it working?
Would be nice to have the feature of unlimited recording.

Thanks
Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: JackDaniel412 on May 26, 2013, 06:36:12 PM
2 days test of last raw_rec.mo, work really well!
Unlimited recording work and totalcmd is a useful app!

Great job!
Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: sebble on May 26, 2013, 10:18:49 PM
Where can I get the newest version of Raw2DNG for MAC?
Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: N/A on May 26, 2013, 10:35:47 PM
Raw2Dng for osx-

http://www.magiclantern.fm/forum/index.php?topic=5508.0 (http://www.magiclantern.fm/forum/index.php?topic=5508.0)
Title: Re: Another raw_rec.mo for 5D2 (bypass 4GB limit, deleting videos)
Post by: Oedipax on May 27, 2013, 01:10:34 AM
Quote from: Hazer on May 25, 2013, 02:17:16 AMRaw2dng said "this ain't no lv_rec".  These two methods will typically both work on binary files.  How do the spanned clips need to be joined?

Interesting - using cat in terminal is working for me (I've done it on 4 different clips now). Only tried it on recordings spanning two files so far, so I can't confirm it still works once you get to 3+ but I don't see why it wouldn't.

Terminal command is super simple: "cat raw1.RAW raw1.R00 > raw1-combined.RAW" (for example). raw2dng understands and decodes the output fine for me on OSX 10.8.3.

Deleting files is working as well. And I love both the joystick press to record and especially the auto-liveview disable upon recording (and black crop for unused image area). Awesome work.
Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: squig on May 27, 2013, 01:11:42 AM
Quote from: sebble on May 26, 2013, 10:18:49 PM
Where can I get the newest version of Raw2DNG for MAC???

Lay off on the big print please.
Title: Re: Another raw_rec.mo for 5D2 (bypass 4GB limit, deleting videos)
Post by: Hazer on May 27, 2013, 01:18:57 AM
Quote from: Oedipax on May 27, 2013, 01:10:34 AM
Terminal command is super simple: "cat raw1.RAW raw1.R00 > raw1-combined.RAW" (for example). raw2dng understands and decodes the output fine for me on OSX 10.8.3.

Huh.  You know what, I'm still using an older, command-line build of raw2dng and I bet that's the problem with the larger files.  I need command line because I have it running as part of a one-click script that batches a whole card to .MOV via RawTherapee, while offloading all the files to second drive.

Has anyone compiled a more recent version for OS X that handles 4GB+ files?

Thanks!
Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: saint33d on May 27, 2013, 06:22:37 AM
http://www.magiclantern.fm/forum/index.php?topic=5508.0 (http://www.magiclantern.fm/forum/index.php?topic=5508.0) is the osx version
http://www.magiclantern.fm/forum/index.php?topic=5404.0 (http://www.magiclantern.fm/forum/index.php?topic=5404.0) Win/osx/linux version

from the tweet

Magic Lantern ‏@autoexec_bin 21 May
raw2dng now works with files larger than 4GB!
Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: Hazer on May 27, 2013, 07:51:21 AM
Super, thank you!
Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: Hazer on May 27, 2013, 04:56:44 PM
On another note, now that I've had a full day to do some actual shooting with this module, I'm seeing serious frame skipping.  This is with the build from the 24th.  I haven't stepped through frame by frame, but it almost looks like one frame every second, at regular intervals.  I shot a batch of stuff yesterday which is totally unusable because of this.  Is anyone else noticing this?

I have frame skipping set to "off", so recording should simply stop if the card can't keep up.  However I'm still seeing solid write speeds in the 65-66MB/sec range, so that shouldn't be happening, and recording is definitely not stopping prematurely.  In fact, the buffer is pegged at one star most of the time at 1720x960, which is great.

One thing I'm noticing is that Live View does not disable during recording.  When not recording, the live view framing rect is an overlay but does not matte out the external portion of the screen.  When I hit record, the external portion is then matted out, but Live View continues to update inside the rect.  Is this the intended behavior?


Edit:  Never mind!  ffmpeg issue.  Frame rate must be specified before the other parameters, otherwise it assumes 25p and drops frames to fit whatever rate you specify downstream.

Carry on...
Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: a1ex on May 27, 2013, 05:00:51 PM
Quote from: pravdomil on May 26, 2013, 01:11:37 AM
Could you merge it to official trunk?

I'm trying to port the file splitting part in a portable way: if the filesystem can record more than 4GB, don't split, otherwise split.

Also, the 5D2 handles long file names, but most other cameras don't.

I don't see much point in merging the other tweaks right now, sorry.
Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: Africashot on May 27, 2013, 05:24:36 PM
For some reason I can't get the black overlays to work, what could I be doing wrong?
Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: 1% on May 27, 2013, 05:45:12 PM
I have the deleting merged... it is good. File splitting I don't need. Bars were a nice idea but didn't work in practice. Also had another fix applied for 720P zoom.. I think only 720P non zoom squeezes.
Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: a1ex on May 27, 2013, 05:57:07 PM
For deleting, there's work in progress for a file manager module.

https://bitbucket.org/minimimi/magic-lantern-filer/commits/all
Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: 1% on May 27, 2013, 06:21:21 PM
That will be the ultimate solution. Wonder if .422 should make a comeback...

there is an lv_save_sndpas... if the wonder what size that is (hd buffer or raw sized)
Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: mageye on May 29, 2013, 04:05:03 PM
Having used the cat command from terminal (OSX 10.8.3) to combine two files I then proceed to drop that file into raw2dng GUI ver.0.10

and get:

File Supported
Error: This ain't a lv_rec RAW file

This makes me sad :'(

EDIT: Oh interesting I replaced the new raw2dng GUI ver.0.10 with raw2dng GUI ver.0.5 and now its converting. I wonder what goes on there?

HAPPY AGAIN. YAY! :D
Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: ch_d on May 29, 2013, 09:58:05 PM
Getting strange "green" spots with this / latest version...
(http://oi39.tinypic.com/2whl3cw.jpg)
This is a 100% crop from a 1600x900 video.
5DMII + Transcent 1000x 16GB
Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: joshmonie on June 01, 2013, 10:38:10 PM
Hi Guys,

I just want to get it out there that I'm a total newb before I go any further.

Ok, I'm needing some help on installing this all on my 5D2. As far as I'm aware, I download the current stable ML which is v2.3, unzip it and take just the 5D2-212.fir onto the root of my CF. Where do I find the autoexec.bin and ML folder? Can I just use the autoexec.bin and ML folder from a previous build and copy/paste the raw_rec.mo file into where it should be?

I was going to try this but was worried I'd brick the camera so thought I'd get some expert advice first!

Thanks in advance,
Josh
Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: pacman829 on June 03, 2013, 07:32:34 AM
I'm curious to know of any updates, mostly for the playback frame skipping :D
Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: scrax on June 03, 2013, 08:15:42 AM
Quote from: mageye on May 29, 2013, 04:05:03 PM
Having used the cat command from terminal (OSX 10.8.3) to combine two files I then proceed to drop that file into raw2dng GUI ver.0.10

and get:

File Supported
Error: This ain't a lv_rec RAW file

This makes me sad :'(

EDIT: Oh interesting I replaced the new raw2dng GUI ver.0.10 with raw2dng GUI ver.0.5 and now its converting. I wonder what goes on there?

HAPPY AGAIN. YAY! :D
can you try with 0.9 please? 0.5 has a different raw2dng executable, if that is better will consider reuse it in next versions
Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: silvertonesx24 on June 03, 2013, 10:38:42 PM
This is great work, but doesn't seem to save any settings in the raw video menu. They have to be changed back every time I toggle the camera on-off.
Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: pravdomil on June 13, 2013, 05:34:01 PM
Continue here
http://www.magiclantern.fm/forum/index.php?topic=5533.0 (http://www.magiclantern.fm/forum/index.php?topic=5533.0)
Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: Doyle4 on July 06, 2013, 10:13:00 PM
Not sure what im doing wrong, but im on a mac and have a split file, for over an hour now i have tried everything to join, but not getting anywhere :(

Iv tried dragging the folder into terminal and using the command, nothing
tried dragging in the 2 files, then the command, nothing :(
Terminal keeps telling me theres no directory, how do i tell it where the files are etc?

Cheers
Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: togg on July 06, 2013, 10:38:32 PM
Quote from: Doyle4 on July 06, 2013, 10:13:00 PM
Not sure what im doing wrong, but im on a mac and have a split file, for over an hour now i have tried everything to join, but not getting anywhere :(

Iv tried dragging the folder into terminal and using the command, nothing
tried dragging in the 2 files, then the command, nothing :(
Terminal keeps telling me theres no directory, how do i tell it where the files are etc?

Cheers

You're doing something wrong. Be sure to copy/paste the directory also for the output file and check all the spaces etc.
Title: Re: Another raw_rec.mo for 5D2 - bypass 4GB, selecting, deleting, playing
Post by: Doyle4 on July 06, 2013, 10:40:37 PM
Quote from: togg on July 06, 2013, 10:38:32 PM
You're doing something wrong. Be sure to copy/paste the directory also for the output file and check all the spaces etc.

I am indeed! haha, killing my brain,

Could i be cheeky and ask for a screen shot of your terminal with an example please, still cant seem to get it too work, i know RAWmagic allows split files but the vertical banding is still there where i love RAW2DNG as it just works, shame splits dont