Hey.
after alex spent a lot of time to find out how we can squeeze out the last bit of performance while
writing raw video to SD and CF cards, i used the last days to think about how to structure the
raw videos to make the post processing easier and the format more extensible.
the result is our next Magic Lantern Video format (.mlv) i want you to look at.
use it on your own risk.
for users:mlv_rec: nightly download page (http://www.magiclantern.fm/downloads.html).
mlv_dump: most recent nightly download page (https://builds.magiclantern.fm/jenkins/view/Other%20tasks/job/mlv_dump/lastSuccessfulBuild/artifact/src/modules/mlv_rec/mlv_dump.exe). (binary for
WINDOWS only)
mlv_dump: or here (http://www.magiclantern.fm/modules/modules/mlv_dump.zip/mlv_dump.zip) (binaries for
WINDOWS, LINUX and OSX)
for developers:mlv file structures in C: here (LGPL) (http://www.magiclantern.fm/modules/modules/mlv_structure.zip/mlv_structure.zip)
preferred: you can export .dng frames from the recorded video using "mlv_dump --dng <in>.mlv -o <prefix>"
legacy mode: post processing is still possible with 'raw2dng'
after converting the .mlv into the legacy .raw format using mlv_dump.
for details see the description below.
see the short video i made: http://www.youtube.com/watch?v=A6pug1g-kNs
it shows a bunch of the new (user visible) features of that file format.
mlv_dump - used for debugging and converting .mlv files
- can dump .mlv to legacy .raw + .wav files
- can dump .mlv to .dng + .wav
- can compress and decompress frames using LZMA
- convert bit depth (any depth in range from 1 to 16 bits)
compression:you can get a data reduction of ~60% with 12 bit files.
downconverting to 8 bits gives you about 90% data reduction.
this feature is for
archiving your footage.
converting back to e.g. legacy raw doesnt need any parameters - it will decompress and convert transparently without any additional parameter.
parameters:
-o output_file set the filename to write into
-v verbose output
-- DNG output --
--dng output frames into separate .dng files. set prefix with -o
--no-cs no chroma smoothing
--cs2x2 2x2 chroma smoothing
--cs3x3 3x3 chroma smoothing
--cs5x5 5x5 chroma smoothing
-- RAW output --
-r output into a legacy raw file for e.g. raw2dng
-- MLV output --
-b bits convert image data to given bit depth per channel (1-16)
-z bits zero the lowest bits, so we have only specified number of bits containing data (1-16) (improves compression rate)
-f frames stop after that number of frames
-x build xref file (indexing)
-m write only metadata, no audio or video frames
-n write no metadata, only audio and video frames
-a average all frames in <inputfile> and output a single-frame MLV from it
-s mlv_file subtract the reference frame in given file from every single frame during processing
-e delta-encode frames to improve compression, but lose random access capabilities
-c (re-)compress video and audio frames using LZMA (set bpp to 16 to improve compression rate)
-d decompress compressed video and audio frames using LZMA
-l level set compression level from 0=fastest to 9=best compression
# show mlv content (verbose)
./mlv_dump -v in.mlv
# will dump frames 0 through 123 into a new file
# note that ./mlv_dump --dng -f 0 in.mlv (or ./mlv_dump --dng -f 0-0 in.mlv) will now extract just frame 0 instead of all of the frames.
./mlv_dump -f 123 -o out.mlv in.mlv
# prepare an .idx (XREF) file
./mlv_dump -x in.mlv
# compress input file
./mlv_dump -c -o out.mlv in.mlv
# compress input file with maximum compression level 9
./mlv_dump -c -l 9 -o out.mlv in.mlv
# compress input file with maximum compression level 9 and improved delta encoding
./mlv_dump -c -e -l 9 -o out.mlv in.mlv
# compress input file with maximum compression level 9, improved delta encoding, 16 bit alignment which improves compression and 12 bpp
./mlv_dump -c -e -l 9 -z12 -b16 -o out.mlv in.mlv
# decompress input file
./mlv_dump -d -o out.mlv in.mlv
# convert to 10 bit per pixel
./mlv_dump -b 10 -o out.mlv in.mlv
# convert to 8 bit per pixel and compress
./mlv_dump -c -b 14 -o out.mlv in.mlv
# create legacy raw, decompress and convert to 14 bits if needed
./mlv_dump -r -o out.raw in.mlv
baldand implemented an amazing video player that is using OpenGL and is able to convert your .raw/.mlv into ProRes directly.
even i use it as my playback tool, so consider it as the official player. ;)
see: http://www.magiclantern.fm/forum/index.php?topic=9560.0
MLV_Viewersee here for a MLV player on windows
(http://upload.g3gg0.de/pub_files/e76d16212357343b54eeac44f2e96002/MLV_Viewer.png)
(http://www.magiclantern.fm/forum/index.php?topic=8447.0)
in-camera mlv_play:the module mlv_play.mo is shipped with the pre-built binaries.
it is a plugin for file_man.mo to play .raw and .mlv files in camera.
the discussion thread for this module is there (http://www.magiclantern.fm/forum/index.php?topic=9062.0)
Drastic Preview:the guys over at drastic.tv are currently implementing the MLV format and already have a working non-open beta version. (i tried it already and i love it :) )
i am sure within the next weeks they will release a new version.
http://www.drastic.tv/index.php?option=com_content&view=category&id=42&Itemid=79
some technical facts: - structured format
- extensible layout
- as a consequence, we can start with the minimal subset (file header, raw info and then video frames)
- multi-file support (4 GiB splitting is enforced)
- spanning suport (write to CF and SD in parallel to gain 20MiB/s)
- out-of-order data support (frames are written some random order, depending on which memory slot is free)
- audio support
- exact clock/frametime support (every frame has the hardware counter value)
- RTC information (time of day etc)
- align fields in every frame (can differ from frame to frame)
the benefit for post processing will be: - files can be easily grouped by processing SW due to UIDs and file header information (autodetect file count and which files belong to each other)
- file contains a lot of shooting information like camera model, S/N and lens info
- lens/focus movement can be tracked (if lens reports)
- exact* frame timing can be determined from hw counter values (*=its accuracy is the limiting thing)
- also frame drops are easy to detect
- hopefully exact audio/video sync, even with frame drops
- unsupported frames can be easily skipped (no need to handle e.g. RTC or LENS frames if the tool doesnt need them)
- specified XREF index format to make seeking easier, even with out of order data and spanning writes
why a custom format and not reuse e.g. .mov? - other formats are good, but none fits to our needs
- hard to make frames align to sector or EDMAC sizes
- they dont support 14 bit raw bayer patterns out of the box
- even when using a flexible container, nearly all sub blocks would need custom additions
- this means a lot of effort to make the standard libs for those formats compatible
- its hard to implement our stuff in a clean way without breaking the whole format
thats the reason why i decided to throw out another format.
it is minimalistic when desired (especially the first implementation will only use a subset of the frames)
and can be extended step by step - while even the most minimalistic parser/post processing tool
can process the latest video files where all stuff is implemented.
if you are a developer (ML or even 3rd party tools) - look over it and make yourself comfortable with that format.
in case there is a bug or something doesnt make sense, please report it.
i would love to get feedback.
here is the link of the spreadsheet that is some kind of reference when designing the format:https://docs.google.com/spreadsheet/ccc?key=0AgQ2MOkAZTFHdHJraTVTOEpmNEIwTVlKd0dHVi1ULUE#gid=0
implementer's notesgreen = fully implemented
blue= implemented, but not 100%
red = not implemented yet, just defined
[MLVI] (once)
- MLVI block is the first block in every .mlv file
- the MLVI block has no timestamp, it is assumed to have timestamp value 0 if necessary
- the MLVI block contains a GUID field which is a random value generated per video shoot
- using the GUID a tool can detect which partial or spanning files belong together, no matter how they are named
- it is the
only block that has a fixed position, all other blocks may follow in
random order - fileCount field in the header
may get set to the number of total chunks in this recording (the current implementation on camera isn't doing this right)
[RAWI] (once, event triggered)
- this block is known from the old raw_rec versions
- whenever the video format is set to RAW, this block has to appear
- this block exactly specifies how to parse the raw data
- bit depth may be any value from 1 to 16
- settings apply to all VIDF blocks that come after RAWI's timestamp (this implies that RAWI must come before VIDF - at least the timestamp must be lower)
- settings may change during recording, even resolution may change (this is not planned yet, but be aware of this fact)
[VIDF] (periodic)
- the VIDF block contains encoded video data in any format (H.264, raw, YUV422, ...)
- the format of the data in VIDF blocks have to be determined using MLVI.videoClass
- if the video format requires more information, additional format specific "content information" blocks have to be defined (e.g. RAWI)
- VIDF blocks have a variable sized frameSpace which is meant for optimizing in-memory copy operations for address alignment. it may be set to zero or any other value
- the data right after the header is of the size specified in frameSpace and considered random, unusable data. just ignore it.
- the data right after frameSpace is the video data which fills up the rest until blockSize is reached
- the blockSize of a VIDF is therefore
sizeof(mlv_vidf_hdr_t) + frameSpace + video_data which means that a VIDF block is a composition of those three data fields
- if frames were skipped, either a VIDF block with zero sized payload may get written or it may be completele omitted
- the format of the data in VIDF frames may change during recording (e.g. resolution, bit depth etc)
- whenever in time line a new content information block (e.g. RAWI) appears, the format has to get parsed and applies to all following blocks
[WAVI] (once, event triggered)
- when the audio format is set to WAV, this block specifies the exact wave audio format
[AUDF] (periodic)
- see [VIDF] block. same applies to audio
[RTCI] (periodic, event triggered)
- contains the current time of day information that can be gathered from the camera
- may appear with any period, maybe every second or more often
- should get written before any VIDF block appears, else post processing tools cannot reliable extract frame time
[LENS] / [EXPO] / ... (periodic, event triggered)
- whenever a change in exposure settings or lens status (ISO, aperture, focal length, focus dist, ...) is detected a new block is inserted
- all video/audio blocks after these blocks should use those parameters
[IDNT] (once)
- contains camera identification data, like serial number and model identifier
- the camera serial number is written as HEX STRING, so you have to convert it to a 64 bit INTEGER before displaying it
[INFO] (once, event triggered)
- right after this header the info string with the length
blockLen - sizeof(mlv_info_hdr_t) follows
- the info string may contain any string entered by the user in format "tag1: value1; tag2: value2"
- tag can for example be strings like take, shot, customer, day etc and value also any string
[NULL] (random)
- ignore this block - its just to fill some writing buffers and thus may contain valid or invalid data
- timestamp is bogus
[ELVL] (periodic)
- roll and pitch values read from acceleration sensor is provided with this block
[WBAL] (periodic, event triggered)
- all known information about the current white balance status is provided with this block
[XREF] (once)
- this is the only block written after recording by processing software, but not the camera
- it contains a list to all blocks that appear, sorted by time
- the XREF block is saved to an additional chunk
- files that only contain a XREF block should get named .idx to clarify their use
- .idx files must contain the same MLVI header like all chunks, but only have the XREF block in it
[MARK] - on keypresses, like halfshutter or any other button, this block gets written for e.g. supplying video cutting positions
- the data embedded into this block is the keypress ID you can get from module.h
[VERS] (any number, usually at the beginning)
- a string follows that may get used to identify ML and module versions
- should follow the format "<module> <textual version info>"
- possible content: "mlv_play built 2017-07-02 15:10:43 UTC; commit c8dba97 on 2016-12-18 12:45:34 UTC by g3gg0: mlv_play: add variable bit depth support. mlv_play requires experi..."
possible future blocks:[BIAS][DARK][FLAT] - in-camera black and noise reference pictures can be attached here (dark frame, bias frame, flat frame)
- to be checked if this is useful and doable
[MLV Format] - the Magic Lantern Video format is a block-based file format
- every information, no matter if audio or video data or metadata is written as
data block with the same basic structure
- this basic structure includes block
type information, block
size and
timestamp (exception to this is the file header, which has no timestamp, but a version string instead)
- the timestamp field in every block is a) to determine the logical order of data blocks in the file and b) to calculate the wall time distance between any of the blocks in the files
- the file format allows multiple files (=chunks) which basically are in the same format with file header and blocks
- chunks are either sequentially written (due to e.g. 4 GiB file size limitation) or parallel (spanning over mutiple media)
- the first chunk has the extension .mlv, subsequent chunks are numbered .m00, m01, m02, ...
- there is no restriction what may be in which chunk and what not
[processing] - to accurately process MLV files, first all blocks and their timestamps and offset in source files should get
sorted in memory
- when sorting, the sorted data can be written into a XREF block and saved to an additional chunk
- do not rely on any order at all, no matter in which order they were written into a file
- the only reliable indicator is the timestamp in all headers
It sounds good!
You guys are seriously an inspiration. Time to give more donations...geez...you guys ROCK!
Wow, new big news about ML Raw... Nice job gentlemen.
Looking good! Magic Lantern getting it's own video format 8)
Have a great vacation g3gg0
feels totally right. Happy holidays
There are some great specs in there. I am keen to see that the audio situation is addressed with this format.
IF/WHEN this format is delivered it will indeed be very useful. It's certainly very promising and I will be very happy to adopt such a format that will address many of the problems that the current RAW files confront us with. The fact that it will be tailored to the Canon RAW footage is great.
One thing that I certainly hope is that software developers follow suit and support this format too. I hope that there will be enough people (developers) out there with the vision that this format deserves.
Without proper tools, 'industry' support (Adobe, Black Magic etc.) and independent developers (you know who you are ;) and thanks!) we will still be in a difficult corner. Support here is ESSENTIAL and maybe that needs a little subtle persuasion from fellow Magic Lantern users and supporters ;).
The bottom line is that I really want and hope that this works out because the benefits will be huge. This will be groundbreaking; not just for the Canon DSLR fans, but for the whole independent digital cinema movement.
Quotespecifies from which sensor row/col the video frame was copied (8x2 blocks)
Does that mean this data can be used later to apply *correct* lens correction in any RAW converter that supports it (ACR, Dxo, P1, etc)?
It would be tremendously useful to have that info stored in the .mlv file to use lens profiles for any sensor crop, even though it might needs to be padded with white/black image data to use regular lens profiles (which won't cost any extra bits if lossless compression is used with a DNG converter, right?).
Looking forward to that. :-)
I think we need a white balance value readed directly from the camera during a raw recording.
Could full EXIF data be captured to a separate file just before recording a raw file then merged with the raw file after recording ends either in-camera or in an app. This data could also then be parsed as metadata for Resolve etc
what about putting the bad pixel information into the video file?
any bad pixel handling can be done in post processing according to the information
Full Exif would be wonderful, particularly I miss the ISO tag for Raw denoising.
In-camera synced audio, omg yes!
Audio part is what I am excited about. It is neat to have a little beep sound, but when you are in a place with music/loud noise/etc., that little beep sound is so easily buried under.
Please express frame rate as a rational number: numerator/denominator instead of framerate/1000. This mitigates increasing precision errors for lower framerates.
Please add also support for markers while recording (for a lot easier life in post-prod)
(as requested here http://www.magiclantern.fm/forum/index.php?topic=6713.0 (http://www.magiclantern.fm/forum/index.php?topic=6713.0)
I updated the format a bit. Added some of the suggestions and some other stuff.
Thats still just paperwork. That has to be implemented step by step.
I would concentrate on the existing frames, if they are right for what we need before adding new ones.
It would be nice to save RAW files with LOG like the technicolor cinestyle for Canon DSLRs
Here is a nice article:
http://www.cinema5d.com/news/?p=6165
Why would this make any sense?
We are saving raw. Do in post whatever you like to do.
It would be awesome if you could reserve a few bytes for bookkeeping information such as:
time : Auto generate with internal clock
Camera # : set manually
Scene # : set manually
Slate # : set manually
Take# : manually set & edit, plus auto count and reset
Ext Sound record # : manually set & edit, plus auto count and reset
and maybe some other info that could be usefull. As was already spoken about here: http://www.magiclantern.fm/forum/index.php?topic=2497.msg60707#msg60707
An option to change the filename according to this info would also be cool, if possible.
E.g.
Scene 1 - Camera 2 - Take 4 - Slate 1 - 2013-07-21 01:05:13.RAW
well, i think you didnt look into the document i posted ;)
its already in there.
mlv_info_hdr_t 5
4 uint8_t blockType[4] INFO user definable info string. take number, location, etc.
4 uint32_t blockSize 20
8 uint64_t timestamp
4 uint32_t length
x uint8_t[] string can be structured: "tag1: value1; tag2: value2"
ohh, I definitely didn't see that. You clearly know what you are doing 8)
Awesome. It looks like a container to me. Though, I am curious how the file names are generated, because there might be multiple mlv files generated in the camera simultaneously, and written to SD/CF card/cards simultaneously and randomly, as far as I can understand, so the file might end up overwriting each other -- maybe I worried too much.
Quote from: mucher on July 22, 2013, 01:03:10 AM
Awesome. It looks like a container to me.
It is. MP4, AVI, MKV, MOV, etc, etc, are all containers.
Quote from: mucher on July 22, 2013, 01:03:10 AM
-- maybe I worried too much.
Yes :)
A reel name tag could be useful(with a default setting as date(max. 8 characters - for EDL files)). Time code would be the same as time stamp right?
For inspiration, the various metadata MediaInfo can extract, can be found in CSV files here:
http://mediaarea.net/download/binary/libmediainfo0/0.7.64/MediaInfo_DLL_0.7.64_Windows_i386_WithoutInstaller.7z
In MediaInfo_DLL_0.7.64_Windows_i386_WithoutInstaller.7z\Developers\List_Of_Parameters\
Quote from: g3gg0 on July 21, 2013, 12:26:51 PM
Why would this make any sense?
We are saving raw. Do in post whatever you like to do.
Why does Red do?
like said: as we are saving raw, it doesnt make sense.
do this in post processing please.
So the ETA for this is 1-2 months?
Quote from: g3gg0 on July 15, 2013, 10:58:23 PM
if you are an end user, wait until we looked over it. then we are going to implement some example code
to write and read this file format. and 1-2 months later it *may* be in the ML sources.
Thats what I trying to confirm on.
Yeah that dodgy g3gg0 is always making OP's with incorrect facts :o
It just seemed worded funny I don't know. I wanted to make sure. I was wondering if there is an update to this eta, that's all.
there is no hard schedule for this. it will be implemented when there is some spare time ;)
Update: (see first post)
The first implementation is done and we can record in the new format.
post processing is still done with 'raw2dng' after converting the .mlv into the legacy .raw format using mlv_dump.
(you have to compile it yourself on your computer)
see the short video i made: http://www.youtube.com/watch?v=A6pug1g-kNs
it shows a bunch of the new (user visible) features of that file format.
if you want to test recording, grab my sources from https://bitbucket.org/g3gg0/magic-lantern or use the pre-built (http://upload.g3gg0.de/pub_files/5795906e2820ed5c73514ed4311dbcd4/raw_rec.zip) ones. (5D3)
use it on your own risk.
if you can show the mlv_dump logs of your cameras, it would help me a bit.
Can i just drop the contents of raw_rec.zip in the "modules" folder?
This way i can't initialize any module.
I am on the 6D.
Quote from: ilguercio on July 28, 2013, 05:06:20 PM
Can i just drop the contents of raw_rec.zip in the "modules" folder?
This way i can't initialize any module.
I am on the 6D.
whats the error message? about _prop_get_value?
Yes, it's that one.
can you retry downloading and testing?
Wait, if it's for 5DIII only it can't work, right?
could work now on others too. just try.
It works, what is that you need? :)
Adding the pre-built modules does not work on the 50D (error linking modules). I think the problem is with the IME modules. I get compiler errors with ime_base.
600d with zip-file
Works. I get mlv-records. And now I´m searching for mlv_dump :)
Edgar
cannot compile mlv_dump
edgar@linux-dfco:~/magic-lantern/modules/raw_rec> make mlv_dump
[ GCC ] mlv_dump
In file included from mlv_dump.c:26:0:
../lv_rec/lv_rec.h:24:17: fatal error: raw.h: Datei oder Verzeichnis nicht gefunden
compilation terminated.
make: *** [mlv_dump] Fehler 1
Edgar
true, didnt fix the makefile (cant do it right now)
just run:
'gcc mlv_dump.c -o mlv_dump -I../../src'
compiles, ThankYou
but
edgar@linux-dfco:~/test> ~/raw2dng *.RAW
Error: This ain't a lv_rec RAW file
Edgar
Nice vid g3gg0 - just wondering does this version (current 2.0) have audio yet?
No stupid questions, no badgering, just THANKS for all you do!
Just for info and giving feedback, see the top of the mlv-header
File Header (MLVI)
Size : 0x00000038
Ver : v2.0
GUID : 819A0505F3E5613E
FPS : 25.000000
Frames Video: 312
Frames Audio: 0
Block: RAWI
Size: 0x000000B8
Time: 20.695000 ms
Res: 864x486
raw_info:
api_version 0x04332200
height 3905328
width 14
pitch 2048
frame_size 0x00003A98
bits_per_pixel 0
black_level 0
white_level 1734
active_area.y1 1886
active_area.x1 0
active_area.y2 0
active_area.x2 33620224
exposure_bias 1, 6461
cfa_pattern 0x00002710
calibration_ill -907
Block: INFO
Size: 0x00000010
Time: 32.126000 ms
Block: RTCI
Size: 0x00000030
Time: 37.598000 ms
Date: 28.06.2013
Time: 18:20:04 (GMT+0)
Zone: ''
Day of week: 1
Day of year: 0
Daylight s.: 0
Block: EXPO
Size: 0x00000028
Time: 38.185000 ms
ISO Mode: 0
ISO: 800
ISO Analog: 96
ISO DGain: 0 EV
Shutter: 39980 µs (1/25.01)
Block: LENS
Size: 0x00000060
Time: 38.226000 ms
Name: '70-300mm'
Serial: ''
Focal Len: 70 mm
Focus Dist: 545 mm
Aperture: f/5.60
IS Mode: 14
AF Mode: 0
Lens ID: 0x00000001
Flags: 0x00000000
Block: IDNT
Size: 0x00000058
Time: 38.441000 ms
Camera Name: 'Canon EOS 600D'
Camera Serial: '55C28963C'
Camera Model: 0x80000286
Block: VIDF
Size: 0x000B374C
Time: 131.710000 ms
Pad: 0x000000BC
Frame: #0
Crop: 584x368
Pan: 584x361
Space: 188
Block: VIDF
Size: 0x000B374C
Time: 171.778000 ms
Pad: 0x000000BC
Frame: #1
Crop: 584x368
Pan: 584x361
Space: 188
Edgar
@Habitat:
as the video description already said: no audio
@Stedda:
you are welcome
@escho:
perfect! thanks!
with this (http://upload.g3gg0.de/pub_files/6e8bacd4c62e01ef147623c6636a7634/raw_rec.zip)version, the lens id should be correct again (not just 0x00000001)
LensID is now correct (tested with exiftool).
Edgar
For ML Raw 2.0 can we please (above all additional features) move the writing/card speed information that is displayed during recording to either the upper or lower corner, but off of the image window, for HDMI?
The #1 complaint we had from EVERYONE on our 5 day shoot with the hack was that the display information was in the way of the action. Often, it ended up completely covering an actors face for the entire scene!!!
Please move it somewhere else!
It is critical for production.
Thanks.
Rather than have version 2.0 record audio, IF it is going to have Timecode, would it be possible for it to OUTPUT TC that a sound recordist could jamsync to?
This MovieSlate app for the iPhone can do it, so I would image it is possible for a 5Dmk3.
https://itunes.apple.com/us/app/movieslate-clapperboard-shot/id320315888?mt=8 (https://itunes.apple.com/us/app/movieslate-clapperboard-shot/id320315888?mt=8)
Thanks!
@AnotherDave:
TimeCode: 64 bit timestamp counters with microsecond resolution is not enough?
Display: for productional use this debug information will get removed.
My jaw literally dropped when I saw spanning support. I have a 5DMk3. How can I help?
@nigel:
implement multi-file writing support with writer threads in raw_rec.c :)
One of the simplest and maybe nicest thing to have would be to have the non recorded area blacked out, so who's recording can actually focus on what is in the frame. Maybe leave it as an option, some people might find it useful to see outside of the boundaries.
yes.
possible options: rectangle / black out / semi-transparent
Oh, the text input system looks almost seamless, you did a very good job and it works well.
:)
Two things:
1)Why does the recording size need to be a multiple of 512 bytes?
2)If i enable small hacks with your modules i get an errore when i start recording raw, it did not happen before.
Are they useless and thus disabled in your build?
Quote from: g3gg0 on July 29, 2013, 03:53:03 PM
@AnotherDave:
TimeCode: 64 bit timestamp counters with microsecond resolution is not enough?
Display: for productional use this debug information will get removed.
Will we be able to output the TC through the headphone jack (like the iPhone app does)? Maybe even jam sync with another camera through the mic input?
That is more important to me than having synced audio recorded in camera. Secondary source audio is always better, and it is good to not-have-to-be-the-sound-guy-too on a shoot.
Blackmagic Resolve can automatically sync secondary source audio if the TC information matches, so syncing in post using this method is a breeze.
@ilguercio:
thanks!
1) with the old format, where the frames had no additional information (=headers) this
would give you the best write performance (in net rate / gross rate).
in this version this is not the best of all options. just ignore it for now.
2) i didnt try that yet. i just wonder what has changed..? it should work as before.
@AnotherDave:
no audio output yet and also not planned t.b.h. as i have no clue about that topic.
plus i dont have any tools for that.
are you talking about LTC ?
Quote from: g3gg0 on July 29, 2013, 08:08:24 PM
@AnotherDave:
no audio output yet and also not planned t.b.h. as i have no clue about that topic.
plus i dont have any tools for that.
are you talking about LTC ?
Yes! :-)
LTC, essentially what I want is a module that can generate a free run (time of day based) TC as an audio signal via the headphone jack, and also read that same type of signal to synchronize the internal clock with a secondary source (jam syncing). This could allow you to sync up to a Sound Devices 788T or even a second 5D3.
Or even record the LTC output to an unused channel from a multitrack recorder.
This, to me, is far more useful than recording audio in camera.
audio reading is too hard to implement.
would a output-only version only help at all then?
Quote from: AnotherDave on July 29, 2013, 08:58:22 PM
Yes! :-)
LTC, essentially what I want is a module that can generate a free run (time of day based) TC as an audio signal via the headphone jack, and also read that same type of signal to synchronize the internal clock with a secondary source (jam syncing). This could allow you to sync up to a Sound Devices 788T or even a second 5D3.
Or even record the LTC output to an unused channel from a multitrack recorder.
This, to me, is far more useful than recording audio in camera.
I don't understand why we can't just get access to the headphone jack in these cameras and allow it to write to the sd card while the CF card does the video writing.
Quote from: g3gg0 on July 29, 2013, 09:34:09 PM
audio reading is too hard to implement.
would a output-only version only help at all then?
I'd settle for output only. :-)
Quote from: xNiNELiVES on July 29, 2013, 09:46:18 PM
I don't understand why we can't just get access to the headphone jack in these cameras and allow it to write to the sd card while the CF card does the video writing.
My thinking is this - the audio quality that the 5D3 records isn't very good. It's ok and useable, but if you're going through the trouble to shoot in uncompressed 14-bit raw... why wouldn't you want a quality of audio that is going to match with the quality of the video? That can only be done with a secondary system.
I don't think very many people record their audio into their Red cameras... and to be honest, Red is the only thing that compares to the quality of image I'm getting out of my hacked 5D3.
Quote from: AnotherDave on July 29, 2013, 09:55:12 PM
My thinking is this - the audio quality that the 5D3 records isn't very good. It's ok and useable, but if you're going through the trouble to shoot in uncompressed 14-bit raw... why wouldn't you want a quality of audio that is going to match with the quality of the video? That can only be done with a secondary system.
I don't think very many people record their audio into their Red cameras... and to be honest, Red is the only thing that compares to the quality of image I'm getting out of my hacked 5D3.
Much easier to use that horrible camera audio than a clap board even if you do have the funds and man power to record separate audio.
Quote from: aaphotog on July 29, 2013, 09:59:31 PM
Much easier to use that horrible camera audio than a clap board even if you do have the funds and man power to record separate audio.
But then why would you be shooting it in raw? If you don't have the funds or equipment? Seems to me, that if a TC syncing module for a secondary source was implemented that this would open this camera up to a wider range of productions.
Also I'm not talking about using a clapper, if your TC on your .mlv file matched the audio on your recorder you can just click 1 button in Davinci Resolve and have all of your audio synced within a second. That could only be done if the camera could output it's free run (TOD) TC via the audio jack.
Dave
Could that beepsynch work for premiere or after effects aswell?
I meant that I want the headphone jack to be able to record the audio from an external mic. I'm gonna get an H4n anyways so it doesn't really matter.
I mean it would be nice if the camera could send a signal for when the H4N (or other recorder) should stop and start recording...
Thats what I meant sorry for the confusion...
@Danne
beep sync is ok, but how do you know which clip it belongs with?
For the short I filmed on the 5D3 ML, we shot around 1,400 clips over 5 days. We were fortunate enough to have an awesome sound recordist with a 788t and a smart slate.
In resolve, all I had to do was separate the clips and .wavs by the day it was recorded, select any frame where the TC on the slate was visible, and type it in. Resolve automatically synced it from there.
If we had embedded TC out via the audiojack, I wouldn't even have had to type in the TC.
@xnine
That is probably a much more complex task, though the Zoom does have lanc... so who knows? With the zoom, and TC out, you could record the TC output sound from the headphone jack into the 1/8" at the top of the H4n and run it through a program that would read the TC track and embed it into the other tracks with recorded audio.
On a side note, it may be worth it to wait and buy an H6 - 4 xlrs and can be expanded into a 6 track recorder.
Quote from: AnotherDave on July 29, 2013, 10:07:13 PM
But then why would you be shooting it in raw? If you don't have the funds or equipment? Seems to me, that if a TC syncing module for a secondary source was implemented that this would open this camera up to a wider range of productions.
Also I'm not talking about using a clapper, if your TC on your .mlv file matched the audio on your recorder you can just click 1 button in Davinci Resolve and have all of your audio synced within a second. That could only be done if the camera could output it's free run (TOD) TC via the audio jack.
Dave
It doesn't matter WHY. That is of no concern to YOU! Not to say that I don't have the funds/equipment. The fact is, just because YOU may not find it useful, doesn't mean that no one else will. If I want to record raw because I choose to, and have the option of having a recorded audio track on my camera, is that a bad thing? I find it quite ridiculous when people say that NO ONE needs something, an assumption based on their own needs.
Quote from: aaphotog on July 30, 2013, 12:01:53 AM
It doesn't matter WHY. That is of no concern to YOU! Not to say that I don't have the funds/equipment. The fact is, just because YOU may not find it useful, doesn't mean that no one else will. If I want to record raw because I choose to, and have the option of having a recorded audio track on my camera, is that a bad thing? I find it quite ridiculous when people say that NO ONE needs something, an assumption based on their own needs.
I apologize, I meant to say... why would you want a great picture with terrible audio? TC sync would be much more useful for everyone - don't you think?
Implementing a feature that is already in the camera (recording synced audio) is great! And I hope it is possible, but if you're going for quality filmmaking why would you want to marry great pictures with bad sound when there could be a better way?
It isn't a pissing competition, I'd just like to add a MAJOR professional feature to the hack I've come to love and talk about how great it is all the time.
Maybe this can help us.
I think that would be an interesting idea VLC come to support the new ML container.
Maybe the guys from the VLC team would like to join us.
http://www.enjoythearchitecture.com/vlc-architecture
http://wiki.videolan.org/Documentation:Documentation/
http://wiki.videolan.org/Documentation:Hacker%27s_Guide/#VLC.27s_Core_.2F_libVLCcore
http://mailman.videolan.org/pipermail/vlc-devel/2011-August/081064.html
with the latest commits, i added periodic updates of LENS, EXPO and RTCI blocks.
they get embedded into the unused space between two frames, so we have no additional slowdown due to those blocks.
this requires the ML core to be compiled again, as i need the msg_queue_count function i just added. (5D3)
by the way - did some of you run speed tests and comparisons between the old format and the new one?
dont have a reliable card to say anything about speeds, but i think the write rate will be a bit lower.
Great job g3gg0! Could you maybe post a tiny mlv file here, so we can start to develop converters? :)
This is really exciting, full of great ideas and I'm really happy to see magic lantern raw looking more and more convenient and usable professionally. The custom tagging and take number is a fantastic addition, I think something like this was really needed. I'm wondering if it would be possible to have an option to name the .mlv files after their tag and take number. It would be awesome if a file would come out looking like this: "Day 1 take 1.MLV" or "Night test take 5.MLV" or "Scene 3 setup 3 take 5.MLV"
On a set a script supervisor could be keeping track of good or bad takes and then you don't even have to process the takes you know were bad, those files could just be deleted and save precious time. Really, this new format is great and I can't wait to see more, thanks so much everyone for your work.
Just installed it, and I definitely notice the write speed is slower on the 5D3 with 64GB KomputerBay 1000x cards. Card was getting 94.6/MBs previously and now only gets around 84.2/MBs which is on the edge of 4512 expected and continuous.
I'm on a mac, what can I use to convert and check the files?
Thanks.
In case you are still wanting some of this g3gg0:
File Header (MLVI)
Size : 0x00000038
Ver : v2.0
GUID : E2B601B3F770EFC9
FPS : 23.976000
Frames Video: 256
Frames Audio: 0
Block: RAWI
Size: 0x000000B8
Time: 2.933000 ms
Res: 1920x1080
raw_info:
api_version 0x0B152000
height 4797520
width 14
pitch 2046
frame_size 0x00003A98
bits_per_pixel 0
black_level 0
white_level 1928
active_area.y1 2074
active_area.x1 0
active_area.y2 0
active_area.x2 33620224
exposure_bias 1, 6722
cfa_pattern 0x00002710
calibration_ill -635
Block: INFO
Size: 0x00000010
Time: 5.250000 ms
Block: RTCI
Size: 0x00000030
Time: 7.161000 ms
Date: 29.06.2013
Time: 07:44:32 (GMT+0)
Zone: ''
Day of week: 1
Day of year: 209
Daylight s.: 0
Block: EXPO
Size: 0x00000028
Time: 7.832000 ms
ISO Mode: 0
ISO: 1600
ISO Analog: 104
ISO DGain: 0 EV
Shutter: 20845 µs (1/47.97)
Block: LENS
Size: 0x00000060
Time: 7.866000 ms
Name: 'EF35mm f/1.4L USM'
Serial: ''
Focal Len: 35 mm
Focus Dist: 88 mm
Aperture: f/0.00
IS Mode: 0
AF Mode: 0
Lens ID: 0x00000001
Flags: 0x00000000
Block: IDNT
Size: 0x00000058
Time: 7.978000 ms
Camera Name: 'Canon EOS 5D Mark III'
Camera Serial: '131EF8CF14'
Camera Model: 0x80000285
However, after tooling around with it and getting it to work, raw2dng nor RAWMagic recognize the file as a .raw :'(
EDIT: Changing the file suffix from .raw to .RAW caused raw2dng to stop denying that it exists at all, and now just returns the standard "This ain't a lv_rec RAW file."
Quote from: AnotherDave on July 30, 2013, 04:50:01 AM
Just installed it, and I definitely notice the write speed is slower on the 5D3 with 64GB KomputerBay 1000x cards. Card was getting 94.6/MBs previously and now only gets around 84.2/MBs which is on the edge of 4512 expected and continuous.
I'm on a mac, what can I use to convert and check the files?
Thanks.
My results are almost identical with the 64 GB KomputerBay card.
Quote from: AnotherDave on July 29, 2013, 09:51:54 PM
I'd settle for output only. :-)
Hey anotherdave! Maybe there's a simpler and smarter solution! There are LOADS of android / iphone apps which control the camera, so: Why not connect it through usb and the phone would act as a TC generator, allowing via the headphone jack to give TC Signal to whatever devices you want? (by using miniplug splitters of course..) I mean if there are already apps for that it means that it has been done already, and comunicating with the camera also, the pieces are there!! We just need one big smart guy to put them together ;) haha
Cheers!
About write speeds:
when you chose the largest frame format, whats the difference in the frame count?
i.e. old 300 frames, mlv 250 frames
5d2 Komputerbay 64GB:
a.d's 728e571a1276 @1872 x 1250 24fps = 260 / 295
g3gg0 25th build same set up = 230 / 250
Can you try two or three next smaller resolutions too?
Thanks!
Quote from: tonybeccar on July 30, 2013, 08:50:32 AM
Hey anotherdave! Maybe there's a simpler and smarter solution! There are LOADS of android / iphone apps which control the camera, so: Why not connect it through usb and the phone would act as a TC generator, allowing via the headphone jack to give TC Signal to whatever devices you want? (by using miniplug splitters of course..) I mean if there are already apps for that it means that it has been done already, and comunicating with the camera also, the pieces are there!! We just need one big smart guy to put them together ;) haha
Cheers!
How is that easier than generating a TC sync from the camera itself? I want this as a feature, specifically to allow this camera to be used on more professional shoots. Making it complex isn't good for the client...
5d2, Komputerbay 64GB, a.d's 728e571a1276 VS g3gg0 28th build. 1872, 24fps, GD off:
1250 @3:2
294 / 229
262 / 234
273 / 246
260 / 218
299 / 236
1054 @16:9
1110 / 913
1184 / 975
1014 / 821
1341 / 660
1215 / 1025
1569 / 683
1012 @1.85:1
a.d. is near enough continuous at this ratio
/ 2754
/ 3005
/ 3042
/ 2334 card full; reformat.
/ 700
/ 3078
Will this format store white balance info? Picture profiles?
Still not sure how to convert .mlv to .dng too :-/
@Redrocks:
thanks a lot!
@AnotherDave:
why picture profiles and wb?
Quote from: g3gg0 on July 30, 2013, 08:44:35 PM
@AnotherDave:
why picture profiles and wb?
I recently shot a project for a client with the 5d3 ML Raw Hack. It was a 5 day shoot for a short film. We filled 42 cards (6 x KB 64gb) for a total sum of 2.68TB and 1,400 clips.
I had to spend a lot of time afterward color correcting the footage for the client, adjusting the WB since non of that data was stored, and applying a look that was similar to Similaar's Flat_10 PP, which is what I was displaying for them on the monitors.
If it could store that information, it'd mean that I could just hand them footage instead of leaving them with something they don't understand.
*I wouldn't want to hand a client footage that looked as if it was shot @5500k under tungsten lighting. I know what raw is, and the colorist will know what raw is... but the client would just see it and think I didn't know what I was doing.
Also, I was shooting at -1 EV to aid in ETTR to improve my S/N ratio. If the image fine tuning data could also be included, that'd be a big help too!
Dave
I can continue to run more tests if someone can tell me how to convert the .mlv files :-/
Really excited for audio support. Go g3gg0!
Own a 5D MIII, not familiar with developing, but I'd love to help if you need testers.
Quote from: AnotherDave on July 30, 2013, 09:56:13 PM
I can continue to run more tests if someone can tell me how to convert the .mlv files :-/
this:
Quote from: g3gg0 on July 15, 2013, 10:58:23 PM
The first implementation is done and we can record in the new format.
post processing is still done with 'raw2dng' after converting the .mlv into the legacy .raw format using mlv_dump.
(you have to compile it yourself on your computer)
Quote from: AnotherDave on July 30, 2013, 09:54:12 PM
I had to spend a lot of time afterward color correcting the footage for the client, adjusting the WB since non of that data was stored, and applying a look that was similar to Similaar's Flat_10 PP, which is what I was displaying for them on the monitors.
wb makes a bit sense, but color profile not much.
no clue how to tell the DNG format that a client application has to apply a color profile that is canon specific...
5D3 autoexec.bin + modules: http://upload.g3gg0.de/pub_files/50f8e2a45595335252a79b19aba2f422/raw_rec.zip
Redrocks:
can you try the latest version with the sizes you already posted?
Can you recheck the files for me or have I messed up? I put all your new .mo files in the modules folder, left the .bin in the root and deleted the .sym. Then the camera refused to fire up. Tried the build on three cards and ended up having to insert a clean card whilst the camera was switched on to roll back to the latest regular nightly.
Quote from: g3gg0 on July 31, 2013, 12:49:43 AM
wb makes a bit sense, but color profile not much.
no clue how to tell the DNG format that a client application has to apply a color profile that is canon specific...
If I had to give a reason... it would just be for consistency.
If I needed a catch phrase... it would be "because the Red camera can do it" :-)
Dave
Tonight's build is far slower for me... I'm only getting 81.5MB/s 54ms idle, 4000-7000 frames expected, takes a lot longer to works it way up to the speed too.
5D3, KB 64 1000x, 24fps, 1080p
i can't understood why it tell me "not a chdk dng" ?
MacBook-Pro-de-david:~ david$ /Volumes/FREECOM\ HDD/Wine/drive_c/RAW/WORKFLOW/dng2raw /Volumes/FREECOM\ HDD/Wine/drive_c/RAW/WORKFLOW/out.RAW dng
reading /Volumes/FREECOM HDD/Wine/drive_c/RAW/WORKFLOW/out.RAW (size=1382573000)
read 1382573000
not a chdk dng: Undefined error: 0
ifd 0: (28804)
ifd 1: (15936)
ifd 2: (16907)
Segmentation fault: 11
and why i get this when i compile mlv_dump ?
MacBook-Pro-de-david:raw_rec david$ gcc mlv_dump.c -o mlv_dump -I../../src
mlv_dump.c:109:65: error: invalid suffix "d" on floating constant
mlv_dump.c:163:83: error: invalid suffix "d" on floating constant
MacBook-Pro-de-david:raw_rec david$
Quote from: dadinio13 on July 31, 2013, 10:25:13 AM
i can't understood why it tell me "not a chdk dng" ?
why did you specify dng ? thats a raw file.
Quote from: dadinio13 on July 31, 2013, 10:25:13 AM
mlv_dump.c:109:65: error: invalid suffix "d" on floating constant
hm will check later, maybe a copy paste error and my compiler didnt warn?
Quote from: AnotherDave on July 31, 2013, 04:19:27 AM
Tonight's build is far slower for me... I'm only getting 81.5MB/s 54ms idle, 4000-7000 frames expected, takes a lot longer to works it way up to the speed too.
5D3, KB 64 1000x, 24fps, 1080p
thanks. what is "far slower" in numbers?
Quote from: Redrocks on July 31, 2013, 02:22:23 AM
Can you recheck the files for me or have I messed up? I put all your new .mo files in the modules folder, left the .bin in the root and deleted the .sym. Then the camera refused to fire up. Tried the build on three cards and ended up having to insert a clean card whilst the camera was switched on to roll back to the latest regular nightly.
this is for 5D3 :)
you just need the module (i assume you have a recent core build)
tcc: error: undefined symbol 'msg_queue_count' [E] failed to link modules
This is with only the .mo files placed in the module folder of a.d's 728e571a1276. Can't see how I'm doing things any different than usual, but I'll see if there is something I've overlooked.
no, this symbol was introduced a few days ago.
thats the reason why i said you should use a recent ml core ;)
edit: ah forget it, i just looked up that symbol for 5D3. would have to add 5D2 first.
Quote from: g3gg0 on July 31, 2013, 01:10:41 PM
thanks. what is "far slower" in numbers?
Latest ML RAW 1 - 94.5MB/s always continuous ok
Latest ML RAW 2 - 83.3MB/s, 573ms idle
I recorded 2 mins of footage, and videotaped the display.
Here is a link to the display screen:
https://www.dropbox.com/sh/wvimd6cdqel7m3b/2Dp3_iXYyR
Hope that helps...
is this (http://upload.g3gg0.de/pub_files/3c2e37fd6ff824bd910a37351b5474e1/raw_rec.mo) one any better regarding performance?
I got pretty much the same performance on my 5D2 yesterday as with the old format. Maxed at 76MBps.
here (http://upload.g3gg0.de/pub_files/4a8662cf1d07a15450ba8c9a5d99d4be/raw_rec.mo) is some special build for the 5D3 with spanning enabled.
can you please try with your fastest SD and fastest CF cards, if you see any speedup over the version in my post before?
-> the output files will for sure be buggy, its just about writing speed
thanks!
It did something!
it always says continuous ok now...
and it says the write speed is 94.1MB (~80.4mb)
it doesn't display the name of the file it is recording though...
Speed seemed the same on my 6D, as far as i could tell.
Haven't tested it properly yet, though.
the last build has these changes:
- files are stored in card root folder
- two writer threads are started
- one thread writes to SD, the other to CF card
- one file is named .MLV, the other .M00
- frames are saved on either the one or the other card
- the order of the video frames in the .MLV / .M00 is random (well, tbh it is not random but depends on card speed)
in technical words:
- i have set up a file spanning mechanism over SD and CF
in non-technical words:
- increasing data rate by using SD and CF in parallel
the good thing:
- write rate is obviously faster than before
the bad thing:
- write rate on CF only is not as fast as alex' optimized version, but i am trying to improve that.
ah, one more (obvious, but worth mentioning) conclusion:
- this is only possible on 5D3 ;)
nice, soon I'm going to port it on 5D2. Maybe if you will reach some milestone?
I want to solve the CF speed issue.
What is the write speed CF+SD? 100+?
Do you want some help with coding?
i cant get beyond 70MB/s with my crappy card.
Will try to find a faster sd card. Awesome work by the way!
Quote from: pravdomil on August 01, 2013, 10:55:44 AM
What is the write speed CF+SD? 100+?
Do you want some help with coding?
maximum possible: 120 + 20 as alex said once.
but when you have computers, 1+1 is never 2 but 1.8.
so its less than 140MiB/s and more than 120 MiB/s under ideal circumstances.
we will see if theory matches results :)
help is always appreciated in this project - why ask? ;)
Quote from: g3gg0 on July 31, 2013, 01:07:04 PM
why did you specify dng ? thats a raw file.
hm will check later, maybe a copy paste error and my compiler didnt warn?
sorry I was tired I did not realize that I used dng2raw instead of raw2dng :-\
anyway, i still can't open the .raw file from the mlv dumper :
MacBook-Pro-de-david:~ david$ /Volumes/FREECOM\ HDD/Wine/drive_c/RAW/WORKFLOW/raw2dng /Volumes/FREECOM\ HDD/Wine/drive_c/RAW/WORKFLOW/out.RAW
Error: This ain't a lv_rec RAW filewhat version of arm none eabi are you using please?
Does it have to write to both the SD & CF? I know it will speed things up, but that also makes things even more complicated.
I love the work that everyone is doing with the hack, and to be honest... I think the quality of the images being captured is higher than most professional cameras.
Because the quality is so high, I'm hoping for a simplification of the process of recording in and syncing to raw with the ML 5D3 with version 2. I want to be able to shoot with it to get amazing image quality... but I also want to be able to be hired out with my camera to shoot in raw.
The more complicated the process gets, the less likely it is that someone will hire a DP with a hacked 5D3. The more technical knowledge you need to operate a hacked 5D3, the more it is going to be looked at as a "work around" or something "not professional".
The results of this are VERY professional, and I think the format is in the right place at the right time.
But for it to be successful, it really needs to record 1 clip to the CF card that contains ALL the possible information that is gathered while shooting. When it is opened on a computer, it needs to look as similar to what was shot in the field as possible. And, syncing the audio needs to be accomplished in 1 click via timecode.
But all of those things together... and you've got a cheap Red camera, and everyone wants to use it. Continue to make it hard to work with... people will buy or request a BMCP for shooting.
I for one like the idea with filespanning between cf and sd. Better ratios for slowmo is one benefit. Really cool work
Quote from: AnotherDave on August 01, 2013, 01:39:49 PM
Does it have to ...I'm hoping for .... I want to... I also want to... it really needs to... audio needs to...
uhm, what?
did you forget that this feature is still EXPERIMENTAL?
we still try to figure out how far can we go and what is possible.
i am happy about any recommendation about usability, but please leave us enough room for experiments.
and as the first post says, it IS an experiment.
what will be in the ML release is another story.
Quote from: AnotherDave on August 01, 2013, 01:39:49 PM
Does it have to ...I'm hoping for .... I want to... I also want to... it really needs to... audio needs to...
g3gg0 please implement 500fps to raw_rec I really need that feature to capture my breakfast also I need sound recording if something horrible will happend
but seriously - one more reason why dont share unfinished features
Quote from: g3gg0 on August 01, 2013, 12:43:17 PM
help is always appreciated in this project - why ask? ;)
well I have some ideas to raw_rec, but now I have to do some other part of ML, I will message you when I will be ready
I just think it is important, when experimenting, to keep the end user in mind.
I say something as simple as "Rec 709" to a producer with 20 year of experience... and they look at me like I have 2 heads.
Not everyone is as technologically inclined as the folks on this board, and I want to be able to use these amazing images to improve my work. The harder it gets on the backend... the less likely it is that will happen.
Dave
Quote from: AnotherDave on August 01, 2013, 03:49:20 PM
I just think it is important, when experimenting, to keep the end user in mind.
They are nice enough to share their work in progress with us and clearly mark it as so.
Keep hounding them for features they said they'd work on for the future or for your own personal use and ruin it for the rest of us.
For something that is free and they work hard on it's never enough for some of you....
Quote from: Stedda on August 01, 2013, 04:30:55 PM
They are nice enough to share their work in progress with us and clearly mark it as so.
Keep hounding them for features they said they'd work on for the future or for your own personal use and ruin it for the rest of us.
For something that is free and they work hard on it's never enough for some of you....
I'm not hounding, I'm trying to help. I'm a professional, working filmmaker who loves the quality of the image I am getting through my "hacked" 5D3 with their awesome code. I'm only asking to have changes implemented that will help with the perception that his custom firmware isn't a "hack" or something unprofessional, and can be used artistically in a production.
If they're not interested in my input, ask me to stop offering my experience and I'll just as easily stop posting.
Thanks.
Look at g3gg0 quote from you, seems at least one main DEV has the same impression.
Don't ruin it for us patient ones that are happy to get whatever they can come up with because it's 1000 times better than what we had.
Quote from: Stedda on August 01, 2013, 05:20:06 PM
Look at g3gg0 quote from you, seems at least one main DEV has the same impression.
Don't ruin it for us patient ones that are happy to get whatever they can come up with because it's 1000 times better than what we had.
I'm not a developer. I'm a DP, Director, Editor... when I say "I was hoping" what I should say is "a professional would expect" but I'm also not a jerk who thinks his ideas are the most important.
Bottom line is... this produces an image that I think is better than a Red. It allows me to work on lower budget projects that can't afford a Red, but still get amazing images.
It has its flaws though. I know it is experimental. But since we can already, rather comfortably, shoot 1080p on the 5D3 with no real issues. *I say this based on the fact that I used it for a short where we filmed 1,400 clips and 2.68TB of data with it and had no problems.
As a professional, it seems more important to me to focus on what it is missing. IE: WB information, external TC sync, Picture Profile information, easy backend processing...
If you'all don't think I should be using this thing and asking for improvement cause I am "annoying" them... fine. The BMPC 4K is only $600 more than a non-hacked 5D3... so instead of getting what professionals want to use, we can just bow to marketing.
I thought the idea was to make it professional. Was I wrong?
I get it you're special. You're a director/professional/editor and because of that you have a right to the features you demand.
I'm not going to argue with you because you're all over the place. Just remember you're using a tool that wasn't designed to do what YOU want it to do for your one specific use be happy with what you're getting for FREE or use the proper tool. As a professional you should already know this.
Quote from: Stedda on August 01, 2013, 06:25:09 PM
I get it you're special. You're a director/professional/editor and because of that you have a right to the features you demand.
I'm not going to argue with you because you're all over the place. Just remember you're using a tool that wasn't designed to do what YOU want it to do for your one specific use be happy with what you're getting for FREE or use the proper tool. As a professional you should already know this.
I just don't know why you're arguing with me over this...
There is a reason things work they way they do. I'm not trying to annoy the developer and I think what he has been able to get out this camera to do is insane!
I'm not special at all, but I'm willing to offer my experience to the developers to help them shape this into something even more awesome.
Honestly, I could be like every other professional and say "psshhh... shoot with the F55", "Yea, but it'll make your camera explode"...
I even went as far as to test this on a paid, 5 day job... had it failed, I could have been sued. But it worked perfectly, and I've been singing it's praises ever since!
It's great! Amazing! I can't believe this quality is THIS accessible for SO CHEAP! But... the display covers actors faces, and sometimes the images look like vomit before you CC then in Resolve. I think those issues need to be solved before spanning the data across 2 cards.
Why is that such a threat to you?
can you both stop that please?
5D3 users:
can you test this (http://upload.g3gg0.de/pub_files/928d2e3d617d0617e27e2d6a84f7bb6e/raw_rec.mo) version for write speeds with spanning OFF and ON (see menu)
hacks both work on my 5D3, so please try with all hacks enabled.
you should see a clean write speed display for SD and CF (upper is CF, lower one is SD)
Do I need to compile from your source tree?
Linking..
tcc: error: undefined symbol 'msg_queue_count '
[E] failed to link modules
Looking very promising thanks g3gg0, excited to see where this ends up!
Any chance of posting a compiled version of the mlv_dump utility?
I Would love to have a go at testing this end to end but I'm not set up to compile.
https://docs.google.com/file/d/0B-tM9Z6JauKNTjBRS2dIWE5LUDA/edit
MLV_DUMP
BUT you can't actually convert the .RAW converted from your .MLV with raw2dng for now.
Quote from: dadinio13 on August 02, 2013, 06:03:34 AM
https://docs.google.com/file/d/0B-tM9Z6JauKNTjBRS2dIWE5LUDA/edit
MLV_DUMP
BUT you can't actually convert the .RAW converted from your .MLV with raw2dng for now.
Cheers for that. yeah jumping the gun a bit, but I'll be prepped and ready for when we can convert them ;)
did all that tonight.
- recording a card spanning movie
- concatenating using "cat M01-xxx.M01 >> M01-xxx.MLV"
- converting to raw using "./mlv_dump M01-xxx.MLV out.raw"
- converting to dng using "./raw2dng out.raw"
- and finally making a movie from it
works without any problems here.
make sure you have the HEAD revision (the one from today 02 PM GMT+1)
So, for now I have only a 15mb/s sd card so I can,t really try the capabilities here but gave it a go. When cardspanning is on I get about 7mb/s on the sd card and around 72mb/s on the cf card. With only cf I once got 84mb/s. Small test below. For now I get less recordingspeed with cardspanning on. Might change with a faster sd-card.
1920x1080 25fps
With filespanning of
about 592 frames
maxspeed on card 84mb/s
With filespanning on
74mb/s (cf-card)
7mb/s (sd-card)
400 frames
Thanks for your efforts g3gg0
okay i see. its not that straight forward as thought.
thanks again.
I tried it again and now it seems that its the other way around. It does give a little boost with the sd-card. However I think when my cf was warmed and suddenly gave me 84mb peaks that the cf itself gave longer recording times than with spanning on. I really like to try a faster sd, should speed up things considerably. This is a great speedboost. Good work
Quote from: g3gg0 on August 02, 2013, 02:41:56 AM
5D3 users:
can you test this (http://upload.g3gg0.de/pub_files/928d2e3d617d0617e27e2d6a84f7bb6e/raw_rec.mo) version for write speeds with spanning OFF and ON (see menu)
hacks both work on my 5D3, so please try with all hacks enabled.
you should see a clean write speed display for SD and CF (upper is CF, lower one is SD)
5D3,global draw OFF
CF: komputerbay 1000x 64GB (not the fastest one, benchmarks up to 90-92MB/s), SD: Sandisk Extreme 45MB/s 64GB
Tried couple of times with 1920x1080p 30p
CF: write speeds: 83,5-84,5
SD: 17-18,5
The longest clip I managed to record was over 3000 frames, 1:43. But that was once and I didn't take screenshot. Normally many times i get over 1:10.
https://www.dropbox.com/s/id6jjgb6elrftpc/spanning%201.jpg
https://www.dropbox.com/s/asnel2b6tzb36o5/spanning%202.jpg
With "classic" build from 28th July i believe i was getting about 30 seconds or so 30p. MASSIVE IMPROVEMENT!!!
!!! EDIT - OMG, 2560x1072 24p- previously 370-400 frames.
Now i recorded 1:23, over 1900 frames :D
It was saying continous OK but stopped after 1:23.
Second time i get 2,5K 2,39:1 continous ok - almost 5 minutes !!! and probably card just filled up.
Write speed: CF up to 93MB/s, SD about 17-17,5.
EDIT 2: 2880x1080 (2,67:1) - about 7 seconds, ~300 frames.
Seems 2,5K 2,39:1 would be fully usable :D
Well...if someone finds a way to process the files.
g3gg0... I'm getting different results then the last couple of tests posted here. For me it's slower for some reason. Here are my results on 5D3 using your lastest raw_rec (with spanning support). Once the card was warmed up, I ran it 5 times for each test.
1920x1080 33 fps
Komputerbay 64gb 1000X CF
Sandisk Extreme 16gb
raw_rec V2 (CF card only)
177
173
210
157
173
Avg frames: 178
raw_rec V2 (w/Card Spanning to sd)
235
278
239
276
247
Avg frames: 255
Using most recent V1 raw_rec (commit 707c75f)
381
345
430
426
Avg frames: 383
Damn good tests.
thanks for testing.
here (http://upload.g3gg0.de/pub_files/c3915747a666422558475cc0d1eaab8f/raw_rec.mo) is some new version, but i think its not much better.
@kgv5
Nice results. Could you try 50fps in 1920 mode.? Thanks
Quote from: kgv5 on August 02, 2013, 11:00:43 PM
Seems 2,5K 2,39:1 would be fully usable :D
Well...if someone finds a way to process the files.
thanks for the tests.
as described above, it is already possible to process them.
concatenate them with any tool (even the file order should not matter)
and process them with mlv_dump
Quote from: Danne on August 03, 2013, 12:28:54 AM
@kgv5
Nice results. Could you try 50fps in 1920 mode.? Thanks
Sure.
canon menu - 1280x720 50p PAL
RAW (requires stretch 1.61x)
1920x672 16:9 360 frames
1920x646 1.85:1 380 frames
1920x544 2.20:1 continous OK
1856x650 16:9 460-480 frames
1856x624 1.85:1 700 frames
1728x606 16:9 continous
Frame numbers varies sometimes, these are more or less the best scores i get.
Big thanks for testing slowmo.
another update:
http://upload.g3gg0.de/pub_files/9f2c26461c25ba24997aa22b9a47547a/raw_rec.mo
thats giving me maximum performance on my card setup.
up to 74 + 17 MiB/s
Quote from: g3gg0 on August 03, 2013, 03:18:56 AM
another update:
http://upload.g3gg0.de/pub_files/9f2c26461c25ba24997aa22b9a47547a/raw_rec.mo
thats giving me maximum performance on my card setup.
up to 74 + 17 MiB/s
Wasn't the speed with CF cards 90mb/s+? Nevertheless its good progress. Should be plenty for 1080p etc.
Quote from: g3gg0 on August 03, 2013, 03:18:56 AM
another update:
http://upload.g3gg0.de/pub_files/9f2c26461c25ba24997aa22b9a47547a/raw_rec.mo
thats giving me maximum performance on my card setup.
up to 74 + 17 MiB/s
When i put the new raw_rec.mo to the ML folder (yesterday build) it is getting me an error (cannot load any module)., does it need other autoexec.bin?
Bug maybe: when i swith cam on, start shooting with spanning OFF, than turn spanning ON - it cannot initialize SD card. When I restart the cam and shoot spanning ON from the beginning - its OK.
EDIT
Yesterday build further testing:
1920x1280 3:2 24 and 25p - continous OK.
2048x1152 16:9 24 and 25p - continous OK
2048x1228 5:3 24 and 25p - continous OK >>> (should be great for some stabilizing in post and crop to 1920x1080)
2240x1120 2:1 24 and 25p - continous OK
When switching between resolution i get many err70, some examples:
https://www.dropbox.com/s/ezdpxgfoomya6g1/CRASH02.LOG
https://www.dropbox.com/s/oa44zy391pc2rx7/CRASH01.LOG
https://www.dropbox.com/s/2la6kiiacyaioji/CRASH03.LOG
Which symbol is missing?
Enable console messages in module menu
I just copy/paste .mo from the link to existing and working yesterday build. This is what i get when switch show console ON
tcc: error: undefined symbol 'edmac_memspy_res_unlock'
tcc: error: undefined symbol 'edmac_memspy_res_lock'
[E] failed to link modules
Every modules gives Err
Quote from: dadinio13 on August 02, 2013, 06:03:34 AM
https://docs.google.com/file/d/0B-tM9Z6JauKNTjBRS2dIWE5LUDA/edit
MLV_DUMP
BUT you can't actually convert the .RAW converted from your .MLV with raw2dng for now.
Link dosn't work for me, no permission.
Anyone able to make MLV_Dump available please?
Quote from: CoresNZ on August 03, 2013, 01:56:27 PM
Link dosn't work for me, no permission.
Anyone able to make MLV_Dump available please?
Working now.
but i still can't able to convert the out.RAW to .DNG with raw2dng...
http://upload.g3gg0.de/pub_files/92ff040055c8551996ef13e1be6c43b0/autoexec.bin
http://upload.g3gg0.de/pub_files/55f9003d60f27804ac0924fb50d14da0/5D3_113.SYM
here you have a recent 5D3 ML coer version.
so these errors should be gone
Quote from: dadinio13 on August 03, 2013, 02:42:45 PM
but i still can't able to convert the out.RAW to .DNG with raw2dng...
are you just not able to do, or is there an error message..?
raw2dng :
Error : This ain't a lv_rec RAW file.
and raw2dng for osx by scraxx tell me :
image2pipe demuxer @ 0x102021c00] Could not parse framerate: .
pipe:0: Invalid argument
i think this is a framerate detection problem.
i have the latest ml core version.
should i "make clean" mlv_dump?
g3gg0, is there a possibility to read current white balance value from the camera and write it to a raw file ? It could improve postprocessing a lot. It's not a problem to edit it manually when we've got 10 raw files, but with 100.. it takes a lot of time..
Can anyone try how many seconds is it possible to record at highest resolutions 3k 3.5k at 24fps and 18-20 fps witnout frame skip?
Thanks a lot
Its not long, 3,5k 24p is still 1-2sec, 15 gives couple seconds more.
I checked the newest version:
3,5K 3584x1320
23,976 - 1 sec (35 frames) (write speed CF 81-81, SD 17-17,5)
20 fps - 1,5 sec (39 frames)
18 fps 4 sec (70 frames)
15fps continous OK (write speed CF 101MB/s, SD 17,7MB/s)
Quote from: dadinio13 on August 03, 2013, 04:59:57 PM
raw2dng :
Error : This ain't a lv_rec RAW file.
...
should i "make clean" mlv_dump?
cannot preproduce.
neither in CF-only mode nor in CF+SD file spanning mode.
do a 'gcc mlv_dump.c -o mlv_dump -I../../src' instead of make
Quote from: kgv5 on August 03, 2013, 07:10:43 PM
Its not long, 3,5k 24p is still 1-2sec, 15 gives couple seconds more.
that is how many MiB/s? (CF+SD)
did you use the latest revision?
Quote from: marekk on August 03, 2013, 05:46:51 PM
g3gg0, is there a possibility to read current white balance value from the camera and write it to a raw file ? It could improve postprocessing a lot. It's not a problem to edit it manually when we've got 10 raw files, but with 100.. it takes a lot of time..
it's planned as many other things.
Quote from: g3gg0 on August 03, 2013, 07:26:51 PM
cannot preproduce.
neither in CF-only mode nor in CF+SD file spanning mode.
do a 'gcc mlv_dump.c -o mlv_dump -I../../src' instead of make
I just tried to recompile with this command and still the same error in raw2dng.
could you upload your mlv_dump for testing please i don't know what's wrong ???
Quote from: g3gg0 on August 03, 2013, 07:26:51 PM
that is how many MiB/s? (CF+SD)
did you use the latest revision?
The newest version:
3,5K 3584x1320
23,976 - 1 sec (35 frames) (write speed CF 81-81,5 SD 17-17,5)
15fps continous OK (write speed CF
101MB/s, SD 17,7MB/s)
Was this with all hacks enabled? (memory, small hacks and hacked preview - this is what I use for max speed).
Quote from: kgv5 on August 02, 2013, 11:00:43 PM
5D3,global draw OFF
CF: komputerbay 1000x 64GB (not the fastest one, benchmarks up to 90-92MB/s), SD: Sandisk Extreme 45MB/s 64GB
Tried couple of times with 1920x1080p 30p
CF: write speeds: 83,5-84,5
SD: 17-18,5
The longest clip I managed to record was over 3000 frames, 1:43. But that was once and I didn't take screenshot. Normally many times i get over 1:10.
Newest build: seem slightly slower, especially SD card
1920x1080 30p - recording 30-40 sec - write speeds CF similar 83-85, SD -slower 15-15,5 (previously 17-18,5)
Quote from: a1ex on August 03, 2013, 10:55:29 PM
Was this with all hacks enabled? (memory, small hacks and hacked preview - this is what I use for max speed).
memory hack ON, small hacks ON. Preview tested both canon and Hacked - no significant difference
EDIT: I have done more testing in different resolutions and framerates, SD hardly ever exceeds 15-15,5 so even 3 MB/s slower than the previous build.
Problem with initializing SD card is gone.
appreciate your work, g3gg0 - if you'd be "ein berliner" you'd get a beer or two :)
in 23.976 fps mode, i get 74 MiB/s + 17.8 MiB/s. several tries. sometimes my CF is slower, but i know it is a faulty card.
here (http://upload.g3gg0.de/pub_files/74a871a42c606402bcc7b52889228b5d/raw_rec_mlv.zip) is the complete build for 5D3 that gave me these results.
it is the latest version that also embeds level sensor data (if enabled in ML menu) and white balance info.
Quote from: g3gg0 on August 04, 2013, 02:51:01 AM
in 23.976 fps mode, i get 74 MiB/s + 17.8 MiB/s. several tries. sometimes my CF is slower, but i know it is a faulty card.
OK, for 3,5K it is right, exceeds 18.
But try 1920x1080 29,97 or even 1920x1280 29,97. Its 15-15,5. Now replace raw_rec.mo for the one from reply #107,with this i am getting 18-18,5 with all the same settings and same SD card. 3,5K also 18-18,5.
Oh wait. Does it say continuous ok?
In this case the cf card is preferred and sd might be slower.
In total the write rate should be the same.
i am wondering that it might be faster to create two different buffers for the twoslots or even two buffers to write to the same slot. just a mere thought on the write speed
two buffers wont give us any improvement imho.
to be correct, we already have n buffers, where n is up to 70 or so on 5D3.
and these buffers get assigned dynamically to CF or SD.
i know we can do another improvement.
right now i am playing with two buffering methods on both queueing and dequeueing.
some problem that is making it hard to get optimal results, is the non-censecutiveness of memory buffers.
we can allocate e.g. memory for 70 frames. but this memory is fragmented, it it not continuous.
this means we have some groups of 2, 8, 20 or any other number of frames (we call it slots btw) that are continuous.
but the best for writing speed is to write a consecutive block of nearly 32MiB at once.
buffer layout e.g.: [____] [__] [_____________] [________] [__]
queueing:
method a) pick the first available free buffer
this method walks through the list of slots and checks for a free one.
it does not care if the previous or next buffers are directly behind the free one.
method b) use alex' algorithm that tries to place frames so that we get the largest block with frames in order.
this was necessary for the old writer where the frames had to be in order
better method:
search for the largest free space and try to fill it up.
strategy behind: build the largest block with frames in any order so that we get highest transfer speed
buffer layout e.g.: [____] [__] [______x1234x___] [________] [__]
the fifth frame would get enqueued at x, even if there is a lot of other free space
dequeueing:
method a) there are two writer threads (CF and SD) that are listening on a queue.
dispatcher thread scans the buffers for the largest consecutive block and places a write job into the queue.
the next "free" thread receives this job and writes it, no matter if SD or CF.
method b) again two threads, but every thread has its own queue.
the dispatcher places the largest found block in the queue for CF, the faster device, and the smallest block in the queue for SD.
strategy: try to keep the CF thread busy with the largest blocks to achieve the highest write rate. SD doesnt suffer that much
from writing smaller blocks. (remember, ~32MiB writes give the fastest possible transfer rate. on SD the speed difference isnt that high with smaller blocks)
i think mehtod b) gives the best results.
but to get back to your idea. one possible improvement could be this:
make the dispatcher aware that it should only qeueue writes to SD from large blocks, if there is nothing else.
background: we dont want the slow SD to permanently disturb the queueing by building large blocks.
[____] [12] [3________] [________] [__]
in this case the SD algorithm will pick frame 3.
this could lead to such fragmentation:
[____] [12] [_____789_] [________] [__]
but CF would benefit from writing large blocks so SD would probably be just annoying.
so i am thinking of building a sorted list of buffers and their sizes.
SD will always try to clear the smallest buffer areas, where CF tries to clear the larger ones.
Quote from: g3gg0 on August 04, 2013, 08:18:08 AM
Oh wait. Does it say continuous ok?
In this case the cf card is preferred and sd might be slower.
In total the write rate should be the same.
Nope, continous OK has never show up. With .mo from post #107 i am consistently getting faster SD write by 2-3MB (tested mostly 1920x1080 canons 29,97). Almost always recorded for more than 1 minute.
Quote from: g3gg0 on August 04, 2013, 11:41:48 AM
two buffers wont give us any improvement imho.
to be correct, we already have n buffers, where n is up to 70 or so on 5D3.
and these buffers get assigned dynamically to CF or SD.
i know we can do another improvement.
right now i am playing with two buffering methods on both queueing and dequeueing.
some problem that is making it hard to get optimal results, is the non-censecutiveness of memory buffers.
we can allocate e.g. memory for 70 frames. but this memory is fragmented, it it not continuous.
this means we have some groups of 2, 8, 20 or any other number of frames (we call it slots btw) that are continuous.
but the best for writing speed is to write a consecutive block of nearly 32MiB at once.
buffer layout e.g.: [____] [__] [_____________] [________] [__]
queueing:
method a) pick the first available free buffer
this method walks through the list of slots and checks for a free one.
it does not care if the previous or next buffers are directly behind the free one.
method b) use alex' algorithm that tries to place frames so that we get the largest block with frames in order.
this was necessary for the old writer where the frames had to be in order
better method:
search for the largest free space and try to fill it up.
strategy behind: build the largest block with frames in any order so that we get highest transfer speed
buffer layout e.g.: [____] [__] [______x1234x___] [________] [__]
the fifth frame would get enqueued at x, even if there is a lot of other free space
dequeueing:
method a) there are two writer threads (CF and SD) that are listening on a queue.
dispatcher thread scans the buffers for the largest consecutive block and places a write job into the queue.
the next "free" thread receives this job and writes it, no matter if SD or CF.
method b) again two threads, but every thread has its own queue.
the dispatcher places the largest found block in the queue for CF, the faster device, and the smallest block in the queue for SD.
strategy: try to keep the CF thread busy with the largest blocks to achieve the highest write rate. SD doesnt suffer that much
from writing smaller blocks. (remember, ~32MiB writes give the fastest possible transfer rate. on SD the speed difference isnt that high with smaller blocks)
i think mehtod b) gives the best results.
but to get back to your idea. one possible improvement could be this:
make the dispatcher aware that it should only qeueue writes to SD from large blocks, if there is nothing else.
background: we dont want the slow SD to permanently disturb the queueing by building large blocks.
[____] [12] [3________] [________] [__]
in this case the SD algorithm will pick frame 3.
this could lead to such fragmentation:
[____] [12] [_____789_] [________] [__]
but CF would benefit from writing large blocks so SD would probably be just annoying.
so i am thinking of building a sorted list of buffers and their sizes.
SD will always try to clear the smallest buffer areas, where CF tries to clear the larger ones.
There should be an arbitration system between CF write thread and SD write thread, I guess, and, possibly, a powerful one.
A sorting list should be a good idea. My wild thought is that it might be good to keep a list of RAW file's address, and let the CF/SD write threads to sort according the list. The question is how much RAW files they can fetch each time by each of them -- then there is a need of arbitration system.
was there any discussion about handling of splitted files? in the first implementation its an unnecessary offset-wrapping to get the pics. the better approach should be "while setting resolution and fps there should be calculated, how many pics could be saved into one file, subtracting one or two pics and we're safe. starting the next file.."
will just play with the new files next hours :) regards chmee
This requires prior knowledge about whether the filesystem supports large files or not. Current approach does not require such knowledge (the writing algorithm figures it out on the fly).
@a1ex and what about to fixing it to f.e. 2GB/File? Finally, you dont work with these files, but with the files after the conversion. So, simply speaking, does it matter, if bigger or smaller? (every file could get his own header - its logically the same as the first file with another picture-offset count)
regards chmee
the new .mlv approach supports both concepts.
implemented is none of them yet ;)
currently it will break after the 4G were reached and continue a new file.
the blocks might be corrupted in this case.
i prefer to check before recording, if >4GiB files are supported and start a new file in case.
every file gets is own header with a unique ID so you dont have to pick the files by name, but
the post processing tools can detect the sequence number and which video it belongs to - even if the filenames are changed or corrupted.
(e.g. due to copying it with copy stations)
Sure this has been asked for many times, but could this format also offer the option to capture the first 5 seconds of audio only, so that just this could be used for sync purposes?
Also for the file headers, could a standard extendable format like json,XML, or better still Messagepack be used to allow all kinds of future extension ?
Also a Version opening tag would be super useful if ever a backwards incompatible change is proposed so that tools can work out what format they are decoding!
Loving the new format proposal! James
mlv_dump updated to the last commit (0342906) :
https://docs.google.com/file/d/0B-tM9Z6JauKNTjBRS2dIWE5LUDA/edit?usp=sharing
Quote from: dadinio13 on August 06, 2013, 01:16:53 PM
mlv_dump updated to the last commit (0342906) :
https://docs.google.com/file/d/0B-tM9Z6JauKNTjBRS2dIWE5LUDA/edit?usp=sharing
Can someone please provide and exe version for testing in windows?
MLV converted to RAW with mlv_dump and RAW to DNG with raw2dng (one frame) :
https://docs.google.com/file/d/0B-tM9Z6JauKNSktDTGgxQktnM2M/edit?usp=sharing
ACR doesn't show the settings i was using for wb etc... i'ts exactly like a normal dng (with ml 1.0)
or i'am stupid ?
Can you keep the first post updated with downloads?
Quote from: jc on August 06, 2013, 03:01:07 AM
Sure this has been asked for many times, but could this format also offer the option to capture the first 5 seconds of audio only, so that just this could be used for sync purposes?
this implies your request
Quote from: g3gg0 on July 15, 2013, 10:58:23 PM
- audio support
Quote from: jc on August 06, 2013, 03:01:07 AM
Also for the file headers, could a standard extendable format like json,XML, or better still Messagepack be used to allow all kinds of future extension ?
this format already is extensible and is designed to be extensible
Quote from: g3gg0 on July 15, 2013, 10:58:23 PM
some technical facts:
- structured format
- extensible layout
- as a consequence, we can start with the minimal subset (file header, raw info and then video frames)
Quote from: jc on August 06, 2013, 03:01:07 AM
Also a Version opening tag would be super useful if ever a backwards incompatible change is proposed so that tools can work out what format they are decoding!
the first data field in the file is a version string.
sure you had a look at it? ;)
but thanks for the toughts.
Quote from: dadinio13 on August 06, 2013, 04:35:55 PM
ACR doesn't show the settings i was using for wb etc... i'ts exactly like a normal dng (with ml 1.0)
or i'am stupid ?
the converter into the legacy format converts into the legacy format ;)
this means, that you dont have all the nifty features of the new format.
so you will be able to use the new format with all existing tools.
yet this is a quirk to get some footage at all. as soon i finish the main features on ML side, i will make a mlv2dng too.
or maybe
someone else is willing to do that? combine mlv_dump with raw2dng to get a mlv2dng.
but you can look at the output of mlv2dng and check for WBAL tags.
e.g. ./mlv_dump | grep WBAL -A10
some warning:
the new version will not only read MLV, but also read the M00, M01 etc when available and merge that into the output.
but i realized that it has a >4GiB bug when converting to legacy format.
will fix that tonight.
Quote from: g3gg0 on August 06, 2013, 04:56:15 PM
but you can look at the output of mlv2dng and check for WBAL tags.
e.g. ./mlv_dump | grep WBAL -A10
Oh yeah! thank you for that :)
Quote from: AnotherDave on August 01, 2013, 06:36:43 PM
I just don't know why you're arguing with me over this...
There is a reason things work they way they do. I'm not trying to annoy the developer and I think what he has been able to get out this camera to do is insane!
I'm not special at all, but I'm willing to offer my experience to the developers to help them shape this into something even more awesome.
Honestly, I could be like every other professional and say "psshhh... shoot with the F55", "Yea, but it'll make your camera explode"...
I even went as far as to test this on a paid, 5 day job... had it failed, I could have been sued. But it worked perfectly, and I've been singing it's praises ever since!
It's great! Amazing! I can't believe this quality is THIS accessible for SO CHEAP! But... the display covers actors faces, and sometimes the images look like vomit before you CC then in Resolve. I think those issues need to be solved before spanning the data across 2 cards.
Why is that such a threat to you?
You went on a job with no backup? sheesh.
can you stop that, please?
Quote from: John Kesl on August 07, 2013, 11:31:28 AM
You're a professional who went on a job with no backup? Even saying that proves you aren't.
I didn't say I didn't have a backup if the camera failed... but you don't know if there are problems with the footage until it has been downloaded, converted, and loaded into resolve. So... you can't check it until the end of the day.
Again, what is with the attitude? I'm only asking/hoping for features to solve some issues we had. These would benefit you too... even if you don't understand why.
So with mlv_dump, we're essentially just (as the name implies) dumping the additional information V2 is recording?
Is there a way to incorporate it back into the DNGs? Will that come with mlv2dng, or will mlv2dng also remove the information?
Thanks.
Are you asking to increase your post count? Of course, g3gg0 is just wasting time, all the extra metadata will be thrown away.
Quote from: a1ex on August 07, 2013, 03:20:21 PM
Are you asking to increase your post count? Of course, g3gg0 is just wasting time, all the extra metadata will be thrown away.
No. I'm asking because I want to test it. As a programmer, I'm useless... unless you want me to break a computer. But, I follow instructions well and like to test things...
Wondering when I can get my hands on some metadata! :-)
@alex:
haha. i am sure your sarcasm will be misunderstood.
waiting for a: "why are you doing it then?" a few posts later.
@AnotherDave:
let me cite myself:
Quote from: g3gg0 on August 06, 2013, 04:56:15 PM
yet this is a quirk to get some footage at all. as soon i finish the main features on ML side, i will make a mlv2dng too.
with the latest commits i added these features to mlv_dump:
- compress frames using LZMA
- decompress frames
- convert bit depth (from 14 bits to anything less than 14 bits per pixel and back)
todo:
- apply e.g. a log curve instead of just clipping
- an intelligent reconstruction when upsamping bit depth
if both features are combined you get a data reduction of ~60% with 12 bit files.
downconverting to 8 bits gives you about 90% data reduction.
this feature is better for archiving your footage.
converting back to e.g. legacy raw doesnt need any parameters - it will decompress and convert transparently without any additional parameter.
the LZMA support is tricky and not out-of-the-box yet.
download the latest 7z SDK (v9.2.2) and compile a static version of it using this makefile:
PROG = lib7z.a
CXX = g++
LIB =
RM = rm -f
CFLAGS = -c -O2 -Wall -c -D_7ZIP_ST
OBJS = 7zAlloc.o 7zBuf.o 7zBuf2.o 7zCrc.o 7zCrcOpt.o 7zDec.o 7zFile.o 7zIn.o 7zStream.o Alloc.o Bcj2.o Bra.o Bra86.o BraIA64.o CpuArch.o Delta.o LzFind.o Lzma2Dec.o Lzma2Enc.o Lzma86Dec.o Lzma86Enc.o LzmaDec.o LzmaEnc.o LzmaLib.o Ppmd7.o Ppmd7Dec.o Ppmd7Enc.o Sha256.o Xz.o XzCrc64.o
all: $(PROG)
$(PROG): $(OBJS)
ar -q $(PROG) $(OBJS)
%.o: ../../%.c
$(CXX) $(CFLAGS) $<
clean:
-$(RM) $(PROG) $(OBJS)
then add the created lib7z.a in the raw_rec makefile
- $(call build,GCC,gcc mlv_dump.c -o mlv_dump -std=c99 -m32 -O2 -Wall -I$(SRC_DIR) $(LFS_FLAGS))
+ $(call build,GCC,gcc mlv_dump.c lzma922/C/Util/7z/lib7z.a -o mlv_dump -std=c99 -m32 -O2 -Wall -I$(SRC_DIR) -Ilzma922/C/ $(LFS_FLAGS))
then uncomment the #define MLV_USE_LZMA in mlv_dump.c and recompile using 'make mlv_dump'
examples:
# show mlv content (verbose)
./mlv_dump -v in.mlv
# will dump the first 123 frames into a new file
./mlv_dump -f 123 -o out.mlv in.mlv
# compress input file
./mlv_dump -c -o out.mlv in.mlv
# compress input file with maximum compression level 9
./mlv_dump -c -l 9 -o out.mlv in.mlv
# decompress input file
./mlv_dump -d -o out.mlv in.mlv
# convert to 10 bit per pixel
./mlv_dump -b 10 -o out.mlv in.mlv
# convert to 8 bit per pixel and compress
./mlv_dump -c -b 14 -o out.mlv in.mlv
# create legacy raw, decompress and convert to 14 bits if needed
./mlv_dump -r -o out.raw in.mlv
Ran some test with provided build ccec3eeb -aug 3rd
5D3
CF is Lexar Pro 1000x 32 GB
SD is Sandisk Extreme 45 MB/s 32 GB
Preview:Hacked, Memory hack: ON, Small hack: ON, Global Draw: OFF
1920x1080 at 29.27 CF at 87 to 90 and SD at 18 with 3% idle Continuous OK
1920x1080 at 31 fps CF at 88 to 90 and SD at 18 with 1% idle Continuous OK
1920x1080 at 32 fps CF at 72 to 92 and SD at 18 with 132ms idle Continuous OK (sometimes)
1920x1080 at 33 fps CF at 87-91.2-90 and SD at 18 with 1% idle to finally Skipped frames
1728x606 at 50p fps CF 78 to 81.1 and SD 16.6 at 40% idle Continuous OK
1856x650 at 50p fps CF 80 to 82.2 and SD 16.6 at 100ms idle Continuous OK
1920x672 at 50p fps CF 72 to 83 and SD 16.6 at 100ms idle Skipped
1728x606 st 59.948 CF 64 to 74 and SD 16.3 at 100ms idle Skipped
SPANNING OFF
1920x1080 at 29.27 CF at 82 to 91.7 Skipped
1728x606 at 50p fps CF 78 to 86.4 Skipped
This is very, very nice and practical way of reaching some new benchmarks. Also it makes good use of available space on both cards!!
Used Cat to add the videos but I dont know how to make mlv_dump work so I could not go any further.
I am finally working on a Virtualbox after struggling for a long time with Cygwin.
I am not a programmer so do not know how to make the mlv_dump work!
I am now able to compile most of the new mo except ime_base.o
../../src/dryos.h:547:14 note expected char.........
make: *** ïme_base.o Error1
Congrats G3ggo Excellent work!!!
Update: Just read your new stuff... Amazing stuff!
thanks for your tests!
do you achieve same or lower rates compared to the old raw_rec version that alex made?
Quote from: RenatoPhoto on August 08, 2013, 03:14:29 AM
Used Cat to add the videos but I dont know how to make mlv_dump work so I could not go any further.
not necessary anymore with current mlv_dump
Quote from: RenatoPhoto on August 08, 2013, 03:14:29 AM
I am finally working on a Virtualbox after struggling for a long time with Cygwin.
good choice imho
Quote from: RenatoPhoto on August 08, 2013, 03:14:29 AM
I am not a programmer so do not know how to make the mlv_dump work!
I am now able to compile most of the new mo except ime_base.o
../../src/dryos.h:547:14 note expected char.........
make: *** ïme_base.o Error1
basically you have to:
cd modules/raw_rec
make mlv_dump
then it should be done.
about the compile error - can you send all messages you get when compiling the module?
Quote from: g3gg0 on August 08, 2013, 09:29:13 AM
do you achieve same or lower rates compared to the old raw_rec version that alex made?
Will run some test with spannig off to compare. I can compile the new modules except the ime_base, so I will use the complete compile that you provided instead of todays.
Quote from: g3gg0 on August 08, 2013, 09:29:13 AM
about the compile error - can you send all messages you get when compiling the module?
magiclantern@magiclantern-VirtualBox:~/G3GGo/magic-lantern/modules/ime_base$ make
~/arm-toolchain/bin/arm-none-eabi-gcc-4.7.3 -Os -mthumb-interwork -march=armv5te -D__ARM__ -Wp,-MMD,./ime_base.d -Wp,-MT,ime_base.o -nostdlib -fomit-frame-pointer -fno-strict-aliasing -I../../include -Wall -W -Wno-unused-parameter -Wno-unused-function -Wno-implicit-function-declaration -Wno-missing-field-initializers -Wno-format -Wdouble-promotion -ffast-math -fno-trapping-math -fsingle-precision-constant -fshort-double -std=gnu99 -Winline -I../../platform/all -I../../platform/all/include -I../../src -I../../picoc -DMODULE -DMODULE_NAME=ime_base -c -o ime_base.o ime_base.c
ime_base.c:39:7: error: redefinition of 'ime_base_start'
In file included from ime_base.c:15:0:
ime_base.h:62:32: note: previous definition of 'ime_base_start' was here
ime_base.c: In function 'ime_base_start':
ime_base.c:46:5: warning: incompatible implicit declaration of built-in function 'strncpy' [enabled by default]
ime_base.c:46:5: warning: pointer targets in passing argument 1 of 'strncpy' differ in signedness [-Wpointer-sign]
ime_base.c:46:5: note: expected 'char *' but argument is of type 'unsigned char *'
ime_base.c: In function 'ime_base_test_any':
ime_base.c:92:5: warning: pointer targets in passing argument 1 of 'ime_base_start' differ in signedness [-Wpointer-sign]
ime_base.c:39:7: note: expected 'unsigned char *' but argument is of type 'char *'
ime_base.c: In function 'ime_base_test_alpha':
ime_base.c:96:5: warning: pointer targets in passing argument 1 of 'ime_base_start' differ in signedness [-Wpointer-sign]
ime_base.c:39:7: note: expected 'unsigned char *' but argument is of type 'char *'
ime_base.c: In function 'ime_base_test_num':
ime_base.c:100:5: warning: pointer targets in passing argument 1 of 'ime_base_start' differ in signedness [-Wpointer-sign]
ime_base.c:39:7: note: expected 'unsigned char *' but argument is of type 'char *'
ime_base.c: In function 'ime_base_test_alnum':
ime_base.c:104:5: warning: pointer targets in passing argument 1 of 'ime_base_start' differ in signedness [-Wpointer-sign]
ime_base.c:39:7: note: expected 'unsigned char *' but argument is of type 'char *'
ime_base.c: In function 'ime_base_test_punct':
ime_base.c:108:5: warning: pointer targets in passing argument 1 of 'ime_base_start' differ in signedness [-Wpointer-sign]
ime_base.c:39:7: note: expected 'unsigned char *' but argument is of type 'char *'
ime_base.c: In function 'ime_base_test_math':
ime_base.c:112:5: warning: pointer targets in passing argument 1 of 'ime_base_start' differ in signedness [-Wpointer-sign]
ime_base.c:39:7: note: expected 'unsigned char *' but argument is of type 'char *'
ime_base.c: In function 'ime_base_test_file':
ime_base.c:116:5: warning: pointer targets in passing argument 1 of 'ime_base_start' differ in signedness [-Wpointer-sign]
ime_base.c:39:7: note: expected 'unsigned char *' but argument is of type 'char *'
ime_base.c: In function 'ime_base_init':
ime_base.c:179:5: warning: pointer targets in passing argument 1 of 'strcpy' differ in signedness [-Wpointer-sign]
In file included from ime_base.c:9:0:
../../src/dryos.h:547:14: note: expected 'char *' but argument is of type 'unsigned char *'
make: *** [ime_base.o] Error 1
Quote from: g3gg0 on August 08, 2013, 09:29:13 AM
do you achieve same or lower rates compared to the old raw_rec version that alex made?
Tested with 1920x1080 at 29.97 fps
TEST 1 Spanning enabled>
CF is Lexar Pro 1000x 32 GB
Recorded files 6 files
.MLV 4194304 KB
.M01 4194304 KB
.M02 4194304 KB
.M03 4194304 KB
.M04 4074381 KB
.M05 29269 KB
Frame skipped and stop
SD is Sandisk Extreme 45 MB/s 32 GB
recorded 1 file
.M00 4194304 KB
Total recorded before stop 21.376884.849 bytes on CF plus 4.294.967.296 bytes on SD
Total is 25.671.892.992
I noticed that % Idle SD card began at 27% and continually went down to 2% when finally stopped.
Continuous recording OK was displayed until the idle of SD card went down to 2%, it then changed to estimated frames.
------------------------
TEST 2 Spanning OFF>
CF is Lexar Pro 1000x 32 GB
Recorded files 6 files
0708.MLV 3179009 KB
0705.MLV 1809481 KB
0706.M00 347137 KB
0706.MLV 4194304 KB
0707.M00 1081573 KB
0707.MLV 41934304 KB
Frame skipped and stop
Total recorded before stop 15.161.159.680 bytes
-------------------------------
TEST 3 ALEX1 RAW_REC
CF is Lexar Pro 1000x 32 GB
Recorded files 8 files
0734.R00 1121697 KB
0734.RAW 4194304 KB
0735.R00 600729 KB
0735.RAW 4194304 KB
0736.R00 1327249 KB
0736.RAW 41934304 KB
0737.R00 342017 KB
0737.RAW 41934304 KB
frame skip and stop
Total recorded before stop 20.652.974.080 bytes
Frame skipped and stop
------------------------
In all test it is evident a slowdown of the write procedure until the frame skips and recording stops.
Quote from: RenatoPhoto on August 08, 2013, 02:03:29 PM
magiclantern@magiclantern-VirtualBox:~/G3GGo/magic-lantern/modules/ime_base$ make
~/arm-toolchain/bin/arm-none-eabi-gcc-4.7.3 -Os -mthumb-interwork -march=armv5te -D__ARM__ -Wp,-MMD,./ime_base.d -Wp,-MT,ime_base.o -nostdlib -fomit-frame-pointer -fno-strict-aliasing -I../../include -Wall -W -Wno-unused-parameter -Wno-unused-function -Wno-implicit-function-declaration -Wno-missing-field-initializers -Wno-format -Wdouble-promotion -ffast-math -fno-trapping-math -fsingle-precision-constant -fshort-double -std=gnu99 -Winline -I../../platform/all -I../../platform/all/include -I../../src -I../../picoc -DMODULE -DMODULE_NAME=ime_base -c -o ime_base.o ime_base.c
ime_base.c:39:7: error: redefinition of 'ime_base_start'
In file included from ime_base.c:15:0:
ime_base.h:62:32: note: previous definition of 'ime_base_start' was here
fixed, didnt notice due to dependencies for modules being odd.
Hi g3gg0
Many thanks for your reply!
>this implies your request : audio support
Nope, not quite .. My request was to keep just the first 5 seconds for sync support, not any more. On many cameras the data rate is too high for audio .. Tho if you mean that if audio is supported in the format then the format should be able to handle just 5 seconds then yes it does ( if however the audio time is locked to the same as length as the video capture time in format v2 two then not :-) )
>this format already is extensible and is designed to be extensible
Great but not what I was trying to imply, I meant an already existing format eg Message Pack
When you design any home baked serialisation format you might think you left room for extension but it lacks "arrays in arrays" or "a kv set in a kv set" or some other essential thing that's needed later on + you can use a pre written parser for the desktop app with no need to write your own .. If it was me designing the format I would use message pack, erlang binary terms or similar binary format
>the first data field in the file is a version string.
Oh soz .. Must haves missed it looking first time :-)
There are tons before you who have requested the 5 sec audio feature.
Maybe search a little before asking for something that's already asked for.
okay here (http://upload.g3gg0.de/pub_files/357f92f2cfa8cad36241a2d3b13d49d3/raw_rec_mlv.zip) is a new update to our raw_rec module for MLV writing.
can some of you with a 5D3 can test it again, please?
test procedure:
* disable all modules except raw_rec
* disable audio recoding (canon menu)
* disable global draw (ML menu)
* disable electronic level sensor (ML menu)
* configure raw_rec module:
* Frame Skipping: Allow
* Buffer fill method: 0, 1, 2 - can you test all of them?
* Memory/Small hacks: On
* Card spanning: Off, On - can you test all of them?
* Format CF card
* Start recording
* wait 10-20 seconds
* write down total write rate (3rd line)
with my setup i get ~96 MiB/s (77.5 + 18.5) with fill method 2. the others are a bit slower unfortunately.
Will gladly help out on testing once my 5d3 comes back from Canon (the SD card slot died!) :'(
All tests 1920x1080 29,97p (CF komputerbay 1000x 64GB, SD sandisk extreme 45MB/s)
Couldnt find electronic level off - where is it? (so its on during the tests)
Buffer fill method: 0 (there is no 1,2,3, - there is 0,1,2)
Best: spanning on --85,5 17,5 get continous OK ----total (after 0:20 - 100,2)
spaning off ----92 and more but many dropped frames, after 30sec reached ~96MB but about 100 dropped frames
Buffer fill: 1
Best: spanning on --85 17,9 get expected couple thousand frames ------ total (after 0:20 - 101,9)
spanning off ---94 , many skipped frames afer 20sec
Buffer fill: 2
Best: spanning on ---85,1 17,3 get expected couple thousand frames ------ total (after 0:20 - 103,0)
spanning off ---95 and after 20sec started to drop frames
EDIT
Additional test (spanning ON) >>> 2,5K 2560x1090 2,35:1
Buffer fill 0 - total 111,5 (average 93,3, 18,)frame skip OFF, recorded 1:47
Buffer fill 1 - total 112 (94,1 17,7) recorded 3:23 and card full but was continous OK,
Buffer fill 2 - total 112,6 sometimes 113 (94,5 18,2) continous OK
thanks again, kgv5!
is the CF only write rate comparable to alex' version or is it far off?
(komputerbay cards are between 90 and 120MB/s afaik)
if i read correctly, you get 95MiB/s with CF only and 93MiB/s on CF plus 18 MiB/s SD?
if i enable spanning, my CF rate drops by a fre megabytes per second.
yours seems quite stable.
@all:
can some of the others provide results too?
hint:
chose a resolution so that you get dropped frames for sure - so you have max data to write
My test of todays raw_rec module:
Cards: CF Komputerbay 64GB 1000x, SD Sandisk Ultra 30MB/s
Settings:
* disable all modules except raw_rec
* disable audio recoding (canon menu)
* disable global draw (ML menu)
* Frame Skipping: Allow
* Preview: Canon
span - on
1920x1080 - 25fps
buffer fill - 0
90 + 11 = 101fps
span - off
1920x1080 - 25fps
buffer fill - 0
98.5fps
-----------------
span - on
1920x1080 - 25fps
buffer fill - 1
91 + 11 = 102fps
span - off
1920x1080 - 25fps
buffer fill - 1
96.2fps
-----------------
span - on
1920x1080 - 25fps
buffer fill - 2
90 + 11 = 101fps
span - off
1920x1080 - 25fps
buffer fill - 2
97.5fps
Hi g3gg0
With the Lexar 32 CF 1000X, SD Scandisk Extreme Pro HD HC 1 95mb/s, I get the following (Recording at 1920X1080 16:9 Aspect):
1. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: OFF - Touched 94.5 mb/s
2. Memory Hack: OFF; Small Hack: OFF; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: OFF - Touched 88.2 mb/s
3. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: ON - Touched 102 mb/s FIRST 1500 FRAMES NO SKIPPING
4. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: ON - Touched 101.7 mb/s FIRST 1500 FRAMES NO SKIPPING
5. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: ON - Touched 101.9 mb/s FIRST 1500 FRAMES NO SKIPPING
Quote from: Canon eos m on August 10, 2013, 01:18:09 PM
Hi g3gg0,
With the Lexar 32 CF 1000X, SD Scandisk Extreme Pro HD HC 1 95mb/s, I get the following (Recording at 1920X1080 16:9 Aspect):
Could not find the 'electronic level off'
1. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: OFF - Touched 94.5 mb/s
2. Memory Hack: OFF; Small Hack: OFF; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: OFF - Touched 88.2 mb/s
3. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: ON - Touched 102 mb/s FIRST 1500 FRAMES NO SKIPPING
4. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: ON - Touched 101.7 mb/s FIRST 1500 FRAMES NO SKIPPING
5. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: ON - Touched 101.9 mb/s FIRST 1500 FRAMES NO SKIPPING
I was so excited to see the 5DM3 touch 102 mb/s my heart beat must have been about 1000 beats/ second ;D
One thing, I did notice the camera was warmer in hands than usual & so was the CF card.
Also, noticed the DCIM folder was empty. The root generated the following files:
M10-1619.MLV/ M10-1619.M00 to M10-1634.MLV/ M10-1634.M00 & M10-1618.MLV
Quote from: Canon eos m on August 10, 2013, 01:20:59 PM
Also, noticed the DCIM folder was empty.
The new module writes to card (SD and CF) root folder.
Thanks. How do you convert to DNG from here onwards.
raw2dng, son of batch, rawmagic.. take your pick
Test equipment:
5D M3
CF - 32Gb Sandisk extreme 60Mb/s
SD - 8Gb Kingston ultimateX 100x
Preview: none (global draw off and still in magic lantern preview mode, not Canon)
Used all settings from g33g0's description above
My camera did not feel warmer than usual, which is not hot at all.
Edit: 1920 x 1080p 24fps ofc
Buffer fill method 0:
Spanning ON:
#1 - 56 + 14 = 70 mb/s
#2 - 55.8 + 14.1 = 69.9 mb/s
#3 - 56.2 +14 = 70.3 mb/s
Spanning OFF:
#1 - 59.7 mb/s
#2 - 58.7 mb/s
#3 - 58.8 mb/s
Buffer fill method 1:
Spanning ON:
#1 - 56.6 + 14.2 = 70.8 mb/s
#2 - 54.5 + 14.3 = 68.8 mb/s
#3 - 53.2 + 14.2 = 67.4 mb/s
Spanning OFF:
#1 - 61.4 mb/s
#2 - 61.7 mb/s
#3 - 61.9 mb/s
Buffer fill method 2:
Spanning ON:
#1 - 57.6 + 14.2 = 71.8 mb/s
#2 - 57.9 + 14.2 = 72.1 mb/s
#3 - 57.2 + 14.3 = 71.5 mb/s
Spanning OFF:
#1 - 61.9 mb/s
#2 - 61.8 mb/s
#3 - 60.2 mb/s
CF Lexar Pro 1000x 32 GB
SD Sandisk Extreme 45 45 MB/s 32 GB
1920x1080 at 32 fps for frame skiping
There is a lot of variability so I set up for three consecutive test in the following sequence:
TEST 1.
Format CF Buff=0 and spanning OFF -- Run test
Format CF Buff=0 and spanning ON -- Run test
Format CF Buff=1 and spanning OFF -- Run test
Format CF Buff=1 and spanning ON -- Run test
Format CF Buff=2 and spanning OFF -- Run test
Format CF Buff=2 and spanning ON -- Run test
TEST 2
Same as above
TEST 3
Same as above
TEST PRESENTED FOR EACH METHOD -
The first number of the resulting range was obtained 15 seconds after the test began and the last number was the highest speed obtain thereafter as long as the speed continued to increase. Therefore the second number represents the highest speed in the next few seconds.
Buff=0 - spanning = OFF
test 1 = 91.9-93.3
test 2 = 90.3-91.6
test 3 = 92.1-93.4
Average Min -Max= 91.4 - 92.7
Buff=0 - spanning = ON
test 1 = 100-100.8
test 2 = 97.1-99.6
test 3 = 98-100.3
Average Min -Max= 98.3-100.2
Buff=1 - spanning = OFF
test 1 =90 - 91.5
test 2 = 90 - 91.8
test 3 = 89.7 - 91.7
Average Min -Max= 89.9-91.7
Buff=1 - spanning = ON
test 1 = 99.7 - 100.9
test 2 = 101 - 101.8
test 3 = 97.2 - 100.3
Average Min -Max= 99.3-101
Buff=2 - spanning = OFF
test 1 =92 - 93
test 2 = 89.2 - 92.3
test 3 = 91.3 - 92.1
Average Min -Max= 90.8-92.5
Buff=2 - spanning = ON
test 1 = 100.6 - 100.7
test 2 = 101.3 - 101.9
test 3 = 101.1 - 101.9
Average Min -Max= 101-101.5
My conclusions.
With spanning the clear winner is Buff 2
Without spanning maybe buff 0 or buff 2
I checked again buffer fill method = 2, this time with RenatoPhoto way:
(CF komputerbay 1000x 64GB, SD sandisk extreme 45MB/s)
For 1920x1080 32fps
Buff=2 - spanning = ON
test 1 = 100.8 - 101.1
test 2 = 101 - 101.2
test 3 = 100.9 - 101.3
What is interesting for 29.97 fps i get better write speed:
Buff=2 - spanning = ON
test 1 = 102.2 - 103.3
test 2 = 102.7 - 103.5
test 3 = 101.9 - 102.8
EDIT
The highest speed so far i get with 2880x1226 24fps (with frameskip ALLOW) total 114,9 (95,8, 18,5), this was after ~ 1 minute recording, ~400 skipped frames.
Seems like bigger frames with lower frame rate gives better performance.
EDIT2:
Checked 2880x1206 (2,39:1) but 20fps - recorded 45 seconds before frame skipped, max total write speed 116,3 (97,5, 18,6)
EDIT3: ok, ultimate test: 3584x1320 (max resolution) - for 15fps - normally 119,2-119,6, touched 120,3 MB/s 121,7 MB/s :o
https://www.dropbox.com/s/c3118snkq0uq5ny/Spanning%203-5K%2015%20fps.jpg
Recorded about 50 sec.
I am starting to worry about my cards :D
EDIT 4: Recorded 45 sec of 2240x1260 (16:9) 24fps at 112,5-113 total.
Quote from: Toffifee on August 10, 2013, 02:46:32 PM
raw2dng, son of batch, rawmagic.. take your pick
Sorry for being dumb. Cannot convert the MLV or M00 files to dng with raw2magic. Could please explain how to do that?
Quote from: g3gg0 on July 28, 2013, 04:58:21 PM
Update: (see first post)
The first implementation is done and we can record in the new format.
post processing is still done with 'raw2dng' after converting the .mlv into the legacy .raw format using mlv_dump.
(you have to compile it yourself on your computer)
see the short video i made: http://www.youtube.com/watch?v=A6pug1g-kNs
it shows a bunch of the new (user visible) features of that file format.
if you want to test recording, grab my sources from https://bitbucket.org/g3gg0/magic-lantern or use the pre-built (http://upload.g3gg0.de/pub_files/5795906e2820ed5c73514ed4311dbcd4/raw_rec.zip) ones. (5D3)
use it on your own risk.
if you can show the mlv_dump logs of your cameras, it would help me a bit.
Found this. Is there a binary for this code yet?
@all:
thanks for all the reports!
@Canon eos m:
not for windows, just a linux build.
I can't seem to figure out how to record at 2.5K It says it is not possible?
Quote from: Toffifee on August 09, 2013, 06:05:57 PM
There are tons before you who have requested the 5 sec audio feature.
Maybe search a little before asking for something that's already asked for.
A)My request was that the format allows separate audio length to video length, not the 5 sec feature it's self
B)My second point was that using a pre-established serialisation format would ease things greatly in the future. Assuming the needed number of float or int to string conversions per second is not beyond what the processor can do ontop of the copying work etc.. Just using json would even work just fine.. :-) J
here (http://upload.g3gg0.de/pub_files/fb196456f6a6389763cab895fb6067f8/mlv_dump.exe) is a windows build of mlv_dump
did not test it for 4GiB problems. can someone else try?
Quote from: g3gg0 on August 11, 2013, 12:58:06 AM
here (http://upload.g3gg0.de/pub_files/093cc390f95aa89924d529cbc7734c77/mlv_dump.exe) is a windows build of mlv_dump
did not test it for 4GiB problems. can someone else try?
So is this available with other cameras such as the 5D2?
updated the DL link of mlv_dump.exe and fixed a memory access error that triggers on bit depth change
5D2: not yet, didnt look for the new symbols yet.
5D2: autoexec (http://upload.g3gg0.de/pub_files/4ae93005266bfda5f9bfa1f1a583dd87/autoexec.bin) symbol (http://upload.g3gg0.de/pub_files/89ed3afc29a7826b323fbe2785b42526/5D2_212.sym)
can you try it?
Possible to try on the 7D?
Cannot figure out the command chain:
C:\
C:\ cd desktop
C:\desktop\cd (where the mlv_dump.exe and *.mlv files are saved)
C:\desktop\director with mlv_dump\mlv_dump.exe m10-1619.mlv
Nothing happens - what do I type in the last line?
Quote from: g3gg0 on August 11, 2013, 02:04:14 AM
5D2: autoexec (http://upload.g3gg0.de/pub_files/4ae93005266bfda5f9bfa1f1a583dd87/autoexec.bin) symbol (http://upload.g3gg0.de/pub_files/89ed3afc29a7826b323fbe2785b42526/5D2_212.sym)
can you try it?
What raw_rec.mo should we use with this? I just tried the one from post #200 in this thread and it freezes the camera. I am currently using the 2013/08/09 build from the 5D2 raw thread.
Here you can see how it freezes:
https://plus.google.com/117039326451584292689/posts/dYB7vPW2FsR
What is the command to dump an mlv file to raw file for the windows build?
Quote from: g3gg0 on August 11, 2013, 12:58:06 AM
here (http://upload.g3gg0.de/pub_files/fb196456f6a6389763cab895fb6067f8/mlv_dump.exe) is a windows build of mlv_dump
did not test it for 4GiB problems. can someone else try?
Quote from: Canon eos m on August 10, 2013, 01:18:09 PM
Hi g3gg0
With the Lexar 32 CF 1000X, SD Scandisk Extreme Pro HD HC 1 95mb/s, I get the following (Recording at 1920X1080 16:9 Aspect):
1. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: OFF - Touched 94.5 mb/s
2. Memory Hack: OFF; Small Hack: OFF; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: OFF - Touched 88.2 mb/s
3. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: ON - Touched 102 mb/s FIRST 1500 FRAMES NO SKIPPING
4. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: ON - Touched 101.7 mb/s FIRST 1500 FRAMES NO SKIPPING
5. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: ON - Touched 101.9 mb/s FIRST 1500 FRAMES NO SKIPPING
Small correction (typo error - by mistake mentioned all buffers fills as 0; whereas, these were 0, 1, 2 in that order):
3. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 0; Card Spanning: ON -
Touched 102 mb/s FIRST 1500 FRAMES NO SKIPPING
4. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 1; Card Spanning: ON -
Touched 101.7 mb/s FIRST 1500 FRAMES NO SKIPPING
5. Memory Hack: ON; Small Hack: ON; Frame Skipping: Allow; Buffer fill: 2; Card Spanning: ON -
Touched 101.9 mb/s FIRST 1500 FRAMES NO SKIPPING
Agree with kgv5 that buff 0 and 2 produce the fastest write speeds.
Quote from: g3gg0 on August 11, 2013, 02:04:14 AM
5D2: autoexec (http://upload.g3gg0.de/pub_files/4ae93005266bfda5f9bfa1f1a583dd87/autoexec.bin) symbol (http://upload.g3gg0.de/pub_files/89ed3afc29a7826b323fbe2785b42526/5D2_212.sym)
can you try it?
Well it seems like a user already did. To repeat what he said: it froze. Sorry I couldn't test, I was at traveling 300 miles by car 700 miles by plane... Will test all new builds posted. Thanks for the hard work.
5D2:
can you try this module version if it is any better?
http://upload.g3gg0.de/pub_files/c73635e82eca629043f7f5d442e82265/raw_rec.mo
Tried to get the mlv_dump for windows through parallels 8, but I couldn't get it working.
Has somebody been more successful than me?
mlv_dump.exe works in windows with comand:
mlv_dump.exe -o mlv.raw m10-2205.mlv
Where mlv.raw is the name of the new raw file
m10-2205.mlv is the name of the spanned file, it automatically finds the others i.e. M00
How to make it work:
copy the mlv_dump.exe to the directory where the .mlv file is located.
To excecute this command use the dos terminal, navegate to the same directory and type:
mlv_dump.exe -o mlv.raw m10-2205.mlv with your own file names.
It joins the spanned file.....
PROBLEM: I cannot make the raw2dng.exe (old) work. I receive the:
This ain't a lv_rec RAW file
message... when doing it in windows under dos command
raw2dng.exe mlv.raw
I have also tried in linux but I have a virtual disk size which does not allow me to work with the large files, so I can not use the linux commands until I successfully re-size my virtualbox machine.
Holy crap!! New compile 8961e22b getting 15% more speed.
Both CF and SD write faster!!!!!! Getting 110 to 115 MB/s with hacks on.
:o ok, where is it?
110 to 115 with 1920x1080 32fps??
You forgot to add -r to write it as legacy raw.
See the help.
g3gg0,
Sorry to be so difficult, but could you please post the exact command for a windows machine:
Would it be: mlv_dump.exe -o mlv.raw -r m10-2205.mlv ?
Uhm i am on mobile only.
But i am so confident that i already posted an excessive list of commandline examples...
Quote from: RenatoPhoto on August 11, 2013, 02:57:38 PM
Holy crap!! New compile 8961e22b getting 15% more speed.
Both CF and SD write faster!!!!!! Getting 110 to 115 MB/s with hacks on.
Got damn, holy balony. Gotta test this one. g3gg0 rocks this joint
Quote from: Canon eos m on August 11, 2013, 03:42:22 PM
Would it be: mlv_dump.exe -o mlv.raw -r m10-2205.mlv ?
I just tried the mlv_dump.exe -o mlv.raw -r m10-2205.mlv
and .... made the mlv.raw file
and ... drag and drop on raw2dng.exe and Yes!!!!!!!
Got dng files!
@RenatoPhoto, Where do you find this build? do you hit the extra megabytes filming raw normal 1920x1080 or in 3x crop?
Thanks
//D
In linux type
./mlv_dump -h and get the usage RULES
Usage: ./mlv_dump [-o output_file] [-rscd] [-l compression_level(0-9)] <inputfile>
Parameters:
-o output_file set the filename to write into
-v verbose output
-r output into a legacy raw file for e.g. raw2dng
-b bits convert image data to given bit depth per channel (1-16)
-f frames stop after that number of frames
-c, -d, -l NOT AVAILABLE: compression support was not compiled into this release
Quote from: Danne on August 11, 2013, 04:12:31 PM
@RenatoPhoto, Where do you find this build? do you hit the extra megabytes filming raw normal 1920x1080 or in 3x crop?
Thanks
//D
I compiled it myself. Finally able to do this with VM machine see> http://www.magiclantern.fm/forum/index.php?topic=7579.0
Yes get 15% more on cropped video mode!!!!!!!!!
Quote from: g3gg0 on August 11, 2013, 09:46:35 AM
5D2:
can you try this module version if it is any better?
http://upload.g3gg0.de/pub_files/c73635e82eca629043f7f5d442e82265/raw_rec.mo
It seems to behave in the same way as the other one for me.
@RenatoPhoto. Sweet. Did you by any chance tried this on 50fps?
Thanks
//D
Quote from: RenatoPhoto on August 11, 2013, 04:01:16 PM
I just tried the mlv_dump.exe -o mlv.raw -r m10-2205.mlv
and .... made the mlv.raw file
and ... drag and drop on raw2dng.exe and Yes!!!!!!!
Got dng files!
I cracked it too. 1973 DNG files at my disposal from a single shoot. Awesome! Long live g3gg0.
Quote from: Danne on August 11, 2013, 04:28:19 PM
@RenatoPhoto. Sweet. Did you by any chance tried this on 50fps?
Thanks
//D
Yes 1920x672 get 103 to 106 MB/s I get 1800 to 2600 frames
Quote from: RenatoPhoto on August 11, 2013, 04:44:15 PM
Yes 1920x672 get 103 to 106 MB/s I get 1800 to 2600 frames
Fantastic!
I, m not a programmer. Could youupload the compile or do I wait for g3gg0 uploading?
Thanks
Quote from: Canon eos m on August 11, 2013, 04:36:47 PM
I cracked it too. 1973 DNG files at my disposal from a single shoot. Awesome! Long live g3gg0.
In case anyone is wondering what this is all about.
A few days ago, g3gg0 has coded a new format such that the 5D Mark III can harness the power of the CF + SD at the same time for speeds touching 102-110 mb/s for long record of full HD (2.5K) RAW video that was previously impossible even with a fast card unless one was willing to live with dropped frames or compromise on the file size. The files are saved in a new format : MLV (Magic Lantern Video)
The Windows workflow (cannot say for Apple OS, Linux):
1. Download the mlv_dump.exe file from post
#228 #218 of this thread. Unzip and extract.
2. Then bring the files recorded in the mlv format (& its shadow in *.m00 format) to the same directory as the unzipped mlv_dump.exe folder (to learn how to record in the new mlv format please scan through g3gg0's posts in this thread these past few days). Note: To begin, you will first have to download the latest modules and autoexe.bin files for recording in mlv format (post #200). The new build is also in one of g3gg0's posts
3. Each footage will be generated in two formats (*.mlv and *.m00 extensions).
4. Then in C:\ type the following thread: cd [with name of main folder (root folder), example, "cd desktop" (without the inverted commas)]
Then type the name of the folder next in the path: Same "cd XXXX"
5. Repeat this process till you reach the folder where the "mlv_dump.exe" file stored
6. Lastly, type:
mlv_dump.exe -o mlv.raw -r m10-2345.mlv Note: In your case the "m10-2345" will be replaced with the name of your mlv file.
Note: mlv-dump will pick up the *.m00 on its own. The process will be slow - be patient
8. Then the file that is generated is to be brought over the raw2dng.exe file (also in the same folder) to generate DNG's
Latest compile: https://docs.google.com/file/d/0B9z8Y0rg-pu8eFE0ckFlWkpiR1U/edit?usp=sharing (https://docs.google.com/file/d/0B9z8Y0rg-pu8eFE0ckFlWkpiR1U/edit?usp=sharing)
ps. not responsible for anything.
Thanks Canon eos m. Will try it out. The version you,re referring to, Is it the upload from the post 200?
THanks
//D
The damn thing is so awesome! I just pulled out one frame from the 1973 frames that were generated. Ran it through the ACR 8.1 and used the 'Flatz1' preset that I got from www.dlwatson.net. The result just without anything else straight from the camera (with a few changes to exposure, contract, shadows) is so amazing. A result to die for.
The worry now is how to process such huge amount of RAW data! :'(
Quote from: RenatoPhoto on August 11, 2013, 05:06:43 PM
Latest compile: https://docs.google.com/file/d/0B9z8Y0rg-pu8eFE0ckFlWkpiR1U/edit?usp=sharing (https://docs.google.com/file/d/0B9z8Y0rg-pu8eFE0ckFlWkpiR1U/edit?usp=sharing)
ps. not responsible for anything.
Thanks! Noted, at my ow risk, of course :) Maybe I,ll try g3gg0 version first.
Thanks!
//D
Quote from: Danne on August 11, 2013, 05:07:03 PM
Thanks Canon eos m. Will try it out. The version you,re referring to, Is it the upload from the post 200?
THanks
//D
No. mlv_dump.exe is @ #218
Quote from: Canon eos m on August 11, 2013, 05:09:39 PM
No. mlv_dump.exe is @ #218
Yes. The MLV build it at post #200
Heh, the compression kinda wins me over.
I added the msg queueu stub for everything I have.. 50D consistently gets at least 80Mb/s with timers disabled on old raw_rec so its an easy test to see if speed is lost. I don't have a cam with dual slots.
Maybe it can be used to count MZ syncs too on cams like M and 600D where it fires too often.
Ok I tried out the 5D2 version.
1) It saves to the root of the card?
2) As soon as the video starts recording a MASSIVE lag spike occurs that lasts for a few seconds following with a extremely short record time of 96 frames at
[email protected].
3) Buffer Status/Information display doesn't disappear
4) Card spanning should be removed (because it doesn't have a dual card slot, as you know)
5) I can't write any tag, no text display comes up.
Here's the quick 96 frame video I took (I have no idea how to process these):
https://dl.dropboxusercontent.com/u/814675/M10-2217.MLV
I labeled the above clip to be take #1...
Quote5) I can't write any tag, no text display comes up.
I think you need the IME module too.
Quote from: 1% on August 11, 2013, 07:40:08 PM
I think you need the IME module too.
Ok, where is the download located?
I think you'd have to compile it or use the ones from 5DIII package? I'm not sure what/where g3gg0 released.
Quote from: 1% on August 11, 2013, 08:07:29 PM
I think you'd have to compile it or use the ones from 5DIII package? I'm not sure what/where g3gg0 released.
Well I'm an idiot when it comes to compiling, I'll guess I'll wait for gegg0 to send it...
Quote from: 1% on August 11, 2013, 05:15:19 PM
I added the msg queueu stub for everything I have.. 5
msg_queue_count for 600D, is this 0xFF1F5C80?
Edgar
It's all on the previous page...
Quote from: escho on August 11, 2013, 08:13:08 PM
msg_queue_count for 600D, is this 0xFF1F5C80?
it should be at (msg_queue_post + 0x40). at least on three different it was like that.
Yes, thats right. I didn't push to the 600D repo yet.
Quote from: 1% on August 11, 2013, 08:22:01 PM
Yes, thats right. I didn't push to the 600D repo yet.
No problem, I compile it myself
Edgar
doesn´t run on my 600D. The camera tells me an "file create error"
Edgar
yeah, currently it is hardcoded for CF card cameras.
will change of course somewhen.
Could someone here post a small mlv file from a 5D3? I need it to test code that I'm working on.
Quote from: gnarr on August 12, 2013, 07:11:08 AM
Could someone here post a small mlv file from a 5D3? I need it to test code that I'm working on.
I posted a 5D2 clip a few posts above...
Well I'm still waiting on the IME module. Should I download it from the 5D3 package? Where's this located?
Quote from: xNiNELiVES on August 12, 2013, 07:25:58 AM
I posted a 5D2 clip a few posts above...
Well I'm still waiting on the IME module. Should I download it from the 5D3 package? Where's this located?
Yes. Take the one from somempackage i uploaded the last time.
Modules are the same on every camera.
Early versions of ime had an error where the camera will suddenly pop up
The input dialog randomly after shooting.
This happens only when you entered something before.
Fixed in the latest releases.
If you have such a buggy version, just repower the camera after you enterd smth
Early versions of ime had an error where the camera will suddenly pop up
The input dialog randomly after shooting.
This happens only when you entered something before.
Fixed in the latest releases.
If you have such a buggy version, just repower the camera after you enterd smth
[/quote]
I'm gonna go out on a limb and say you used your phone ;D.
Thanks for the response, I'll get the module.
Again I have these bugs though, I don't know if you saw them earlier:
1) Videos save to the root of the card?
2) As soon as the video starts recording a MASSIVE lag spike occurs that lasts for a few seconds following with a extremely short record time of 96 frames at [email protected]. 3) Buffer Status/Information display doesn't disappear
4) Card spanning should be removed for the 5D2 version (because it doesn't have a dual card slot, as you know)
5) I can't write any tag, no text display comes up.
the latest IME modules can be found here (http://upload.g3gg0.de/pub_files/65d4d5cc43e5a2e6fe2069ba8cdd4546/ime_modules.zip).
(see also http://www.magiclantern.fm/forum/index.php?topic=6899.0)
the latest raw_rec module including win32 exe and static linux binaries is here (http://upload.g3gg0.de/pub_files/63bd95fd274e04f63b32164dfaa04ca3/raw_rec_mlv.zip).
no guarantee that the linux binaries will work on your system (it's a 32 bit ELF for 80386)
Quote from: xNiNELiVES on August 12, 2013, 09:07:44 AM
1) Videos save to the root of the card?
2) As soon as the video starts recording a MASSIVE lag spike occurs that lasts for a few seconds following with a extremely short record time of 96 frames at [email protected].
3) Buffer Status/Information display doesn't disappear
4) Card spanning should be removed for the 5D2 version (because it doesn't have a dual card slot, as you know)
1) intention, its still experimental
2) this was on 5D2, right? can you use these (http://upload.g3gg0.de/pub_files/7678c52475d088e9dd4618d93b1203be/trace.zip) both modules (dont forget to enable trace) and send me your raw_rec.txt after recording with a spike? it should be in the root folder. upload it using http://upload.g3gg0.de and send me the [D/L] link (right-click, copy link)
3) if you stop recording, it should disappear
4) uhm, just dont enable it? ;)
.mlv recording and IME modules are working on the 50D but I can't find a way to exit IME when I've input text.
I did a couple of quick tests on a 600x card at 1920x1080p (Global Draw OFF)
old raw_rec = ~344 frames
new raw_rec = ~288 frames
Buffer fill method doesn't have much effect, maybe 2 - 3 frames variance. Fill method 2 gets most frames.
.mlv files convert ok (Windows). They are written to root directory.
thanks for your test.
about IME: press Q to directly get to the OK/cancel buttons.
raw_rec:the performance is a bit worse than the legacy raw_rec module, yeah.
alex spent a lot of time to get the last frame out of the buffering mechanism.
the new module needs such optimization too.
Quote from: g3gg0 on August 12, 2013, 01:47:56 PM
thanks for your test.
about IME: press Q to directly get to the OK/cancel buttons.
raw_rec:the performance is a bit worse than the legacy raw_rec module, yeah.
alex spent a lot of time to get the last frame out of the buffering mechanism.
the new module needs such optimization too.
Thanks. It's good to see it working anyway even with the performance decrease.
Q doesn't seem to be working on the 50D but I'm using Tragic Lantern so it may have been remapped. I'll have to check.
We don't really have Q, only joy center. Its acting kind of like set, have to check how its set up exactly.
this (http://upload.g3gg0.de/pub_files/5ab9a19c34b15c21d0aaccacbb5a5269/raw_rec_mlv.zip) version of raw_rec works better on my setup.
select:
Buffer fill method: 4
CF-only buffers: 3
this gives me close to 100MB/s - sometimes :)
Quote from: g3gg0 on August 12, 2013, 03:26:35 PM
this (http://upload.g3gg0.de/pub_files/5ab9a19c34b15c21d0aaccacbb5a5269/raw_rec_mlv.zip) version of raw_rec works better on my setup.
select:
Buffer fill method: 4
CF-only buffers: 3
this gives me close to 100MB/s - sometimes :)
This one is worse on the 50D. I tried 4/3 and various other settings and could only record 240+/- frames on the same card.
@1% - could we not use the PicStyle button for Q? It's not really needed for video and I think most users tend to set PS in the ML menu anyway.
Quote from: xNiNELiVES on August 12, 2013, 07:25:58 AM
I posted a 5D2 clip a few posts above...
Well I'm still waiting on the IME module. Should I download it from the 5D3 package? Where's this located?
The file has no data in it (it's only 1 kB). But the same is happening for me now.
I just did a "clean install" on my 5D2, that is, I cleaned my CF card and installed the 5D2_Alpha_One package and updated it to the 2013/08/09 version. Then I installed raw_rec.mo from raw_rec_mlv.zip in reply #277 and autoexec and symbol from #221.
It gives me the same results as before. The camera is super slow for ~5 seconds after I press record and then fills the buffer and "stops", but in the raw rec menu it shows "stopping" forever and everything locks up after a few more seconds.
I tried turning on the debug trace, but I can't find the trace file. @g3gg0, are there any log files I can send you or something I can do to help with fixing this? Or are you still focusing on making the 5D3 version work good before looking into debugging other platforms?
Ps.
I still need a 5D3 file :) Anyone willing to upload one for me? It doesn't need to be large.
Not having much luck converting these.
This is what I'm getting back from the dos command.
E:\>rawFiles\mlv_dump.exe -o mlv.raw -r M13-0212.MLV
Operating mode:
- Input MLV file: 'M13-0212.MLV'
- Convert to legacy RAW
- Output into 'mlv.raw'
Failed to open file 'M13-0212.MLV'
Any ideas why this would be failing to open?
Quote from: g3gg0 on August 12, 2013, 11:38:26 AM
the latest IME modules can be found here (http://upload.g3gg0.de/pub_files/65d4d5cc43e5a2e6fe2069ba8cdd4546/ime_modules.zip).
(see also http://www.magiclantern.fm/forum/index.php?topic=6899.0)
the latest raw_rec module including win32 exe and static linux binaries is here (http://upload.g3gg0.de/pub_files/63bd95fd274e04f63b32164dfaa04ca3/raw_rec_mlv.zip).
no guarantee that the linux binaries will work on your system (it's a 32 bit ELF for 80386)
g3gg0 getting about 104.7 mb/s on my 5D Mark III (1920X1080p) with this raw_rec.mo version. Going to try the one you posted a while ago.
Quote from: CoresNZ on August 12, 2013, 04:39:26 PM
E:\>rawFiles\mlv_dump.exe -o mlv.raw -r M13-0212.MLV
so your raw is in E:\ ?
Quote from: g3gg0 on August 12, 2013, 03:26:35 PM
this (http://upload.g3gg0.de/pub_files/5ab9a19c34b15c21d0aaccacbb5a5269/raw_rec_mlv.zip) version of raw_rec works better on my setup.
select:
Buffer fill method: 4
CF-only buffers: 3
this gives me close to 100MB/s - sometimes :)
Touched 106.1 mb/s with the version of raw_rec.mo
Card Lexar 32GB 1000X + 32GB Scandisk 95mb/s
Camera 2032X1040p
Buffer fill: 2
CF-only buffer: 2
No dropped frames
Quote from: CoresNZ on August 12, 2013, 04:39:26 PM
Not having much luck converting these.
This is what I'm getting back from the dos command.
E:\>rawFiles\mlv_dump.exe -o mlv.raw -r M13-0212.MLV
Operating mode:
- Input MLV file: 'M13-0212.MLV'
- Convert to legacy RAW
- Output into 'mlv.raw'
Failed to open file 'M13-0212.MLV'
Any ideas why this would be failing to open?
Do you have the M13-0212.M00 file in the same directory 'rawFiles'?
you have set cf only buffers to 2?
Should be better with 3
Quote from: g3gg0 on August 12, 2013, 11:43:25 AM
1) intention, its still experimental
2) this was on 5D2, right? can you use these (http://upload.g3gg0.de/pub_files/7678c52475d088e9dd4618d93b1203be/trace.zip) both modules (dont forget to enable trace) and send me your raw_rec.txt after recording with a spike? it should be in the root folder. upload it using http://upload.g3gg0.de and send me the [D/L] link (right-click, copy link)
3) if you stop recording, it should disappear
4) uhm, just dont enable it? ;)
1)
2)3) Even if I stop recording, the buffer status never disappeared. The rest of the info disappeared after a good 15 seconds or so.
4) I don't enable it but what happens if someone did? I realize this is only an experimental copy of the new MLV Format, I guess you'll just disable it in the final versions.
Will trace the bug with the lag spike.
Newest module from #277
This gave me very good results, great work :D : Buffer fill method 3, CF-only buffers 2
1920x1080 16:9 29,97fps >>> ~109-110 (91+19) so shows continous ok.
2240x1260 16:9 24fps >>> ~113 (95+19) continous ok
2560x1090 2,35:1 24fps >>> ~115 (96+19) continous OK
I tested buffer=3 CF=2 and buff=3 Cf=3, after couple tries i had an impression, that B3CF2 speed was more stable, B3CF3 had similar speed but more variations. But its hard to say really, maybe just coincidence, when tried again it was stable, even maybe a bit faster than B3CF2.
Quote from: kgv5 on August 12, 2013, 07:06:08 PM
Newest module from #277
This gave me very good results, great work :D : Buffer fill method 3, CF-only buffers 2
1920x1080 16:9 29,97fps >>> ~109-110 (91+19) so shows continous ok.
2240x1260 16:9 24fps >>> ~113 (95+19) continous ok
2560x1090 2,35:1 24fps >>> ~115 (96+19) continous OK
I tested buffer=3 CF=2 and buff=3 Cf=3, after couple tries i had an impression, that B3CF2 speed was more stable, B3CF3 had similar speed but more variations. But its hard to say really, maybe just coincidence, when tried again it was stable, even maybe a bit faster than B3CF2.
These tests are crazy, 2.5k continuous, Jesus.
How about making a benchmark test that checks the different methods and logs it?
In a similar fashion A1ex made when RAW video first came out.
A bunch of people can do the tests and you could merge the data into one to compare the differences.
Quote from: xNiNELiVES on August 12, 2013, 07:24:10 PM
These tests are crazy, 2.5k continuous, Jesus.
Yeah, its hard to believe.
Just for testing: If someone is interested in 2,8k 2880x1216 2,39:1 but >>20fps<< - managed to record for 45 sec.
with super wide aspect 2880x1080 2,67:1 with >>22fps<< i get similar ~45sec. With the same super wide aspect but 24 fps i am getting ~6-7 sec.
EDIT: I was wondering what is real crop factor for different resolutions like 2,5k and 2,8k. 3x crop is for 1920 res i think, so for 2,5k should be about 2,3x (just like bmcc) and for 2,8k is should be about 2x, am I right?
I have got the feeling that the SD speed looks a bit low.
@g3gg0 what about a version freeze? i would begin to code a converter next days, but the things are changing almost daily. after this version freeze or a first reference mlv-file (with version-number) i could start..
regards chmee
Quote from: mucher on August 12, 2013, 07:47:59 PM
I have got the feeling that the SD speed looks a bit low.
5d3 SD controller max speed is about 20-21 i think. It is no 40-ish like 6D or 650D but slower and current versions are reaching almost the top.
Whats the 5D2's theoretical hardware limit for its CF slot max speed?
According to the chart from
http://www.magiclantern.fm/forum/index.php?topic=6215.0
it is 75 MB/s.
For 5D3 is 100 for CF +20 for SD so devs have managed to squeeze almost the full speed.
Quote from: kgv5 on August 12, 2013, 08:07:31 PM
According to the chart from
http://www.magiclantern.fm/forum/index.php?topic=6215.0
it is 75 MB/s.
For 5D3 is 100 for CF +20 for SD so devs have managed to squeeze almost the full speed.
LOL
I said that because that what the max speeds we are attaining with Raw Video. It doesn't mean its the actual limit.
ok, sorry, my mistake.
I tried some slowmo but only managed to squeeze about 75mb on cf with the latest compilation. Anybody tried it out? 50 fps 1920x672
Quote from: xNiNELiVES on August 12, 2013, 08:00:04 PM
Whats the 5D2's theoretical hardware limit for its CF slot max speed?
I saw benchmarks of the 5D Mark II reaching write speeds slightly over 90 Megabyte/s, just like the 50D.
And the 5D Mark III even reached 130 Megabyte/s. Its SD-Card controller's hardware limit is 21 Megabyte/s.
Quote from: CFP on August 12, 2013, 08:34:16 PM
I saw benchmarks of the 5D Mark II reaching write speeds slightly over 90 Megabyte/s, just like the 50D.
And the 5D Mark III even reached 130 Megabyte/s. Its SD-Card controller's hardware limit is 21 Megabyte/s.
Well if anything the 5D Mark II should be better than the 50D. They were released at the same time, except the 5D2 was around 2000 dollars more.
New test with build 54a545720 TODAY from g3gg0 repository
CF Lexar Pro 1000x 32 GB
SD Sandisk Extreme 45 45 MB/s 32 GB
TEST PRESENTED FOR EACH METHOD -
The first number of the resulting range was obtained 15 seconds after the test began and the last number was the highest speed obtain thereafter as long as the speed continued to increase. Therefore the second number represents the highest speed in the next few seconds.
For CF recording only @ 1920x1080 at 29 fps for frame skipping
Tested all CF 1 through 9 with buffer fill method = 4
CF=9
test 1 = 98.5-100.7
test 2 = 98-101.5
test 3 = 99.1-100.3
Average Min -Max= 98.5-100.8
CF=8
test 1 = 99.101.6
test 2 = 100.1-99.1
test 3 = 102.9-102.4
Average Min -Max= 100.7-101
CF=7
test 1 = 100.1-101.9
test 2 = 100.9-102.1
test 3 = 101.6-101.3
Average Min -Max= 100.9-101.8
CF=6
test 1 = 101.9-102.7
test 2 = 99.5-101.6
test 3 = 97.7-100
Average Min -Max= 99.7-101.4
CF=5
test 1 = 102.5-103
test 2 = 102.1-102.9
test 3 = 101.8-100.5
Average Min -Max= 102.1-102.1
CF=4
test 1 = 97.4-98.9
test 2 = 101.8-102
test 3 = 100.6-102.1
Average Min -Max= 99-101
CF=3
test 1 = 97.4-98.9
test 2 = 101.8-102 Cont rec ok for a little while
test 3 = 100.6-102.1
Average Min -Max= 99.9-101
CF=2
test 1 = 97.5-100.8
test 2 = 102.9-103-100 Cont rec ok for a little while
test 3 = 100.5-101.8
Average Min -Max= 100.3-101.9
CF=1
test 1 = 95.3-98.9
test 2 = 100.8-101
test 3 = 100-101.1
Average Min -Max=98.7-100.3
CF=0
test 1 = 95.9-100.5
test 2 = 98.2-100
test 3 = 102.3-102.6
Average Min -Max=98.8-101.03
My conclusions. For CF only seem like buffer 4,5,6,7,8 are the better with 5 maybe better than the others. I repeated some of the test and repeatability is not very good.
--------------------------
Next test Spaning On. Too many options.
zoom= 3x
Resolutions: 2048x1152 at 29.97
CF=0 B=4
test 1 = 113-116.2
test 2 = 114-116.2
test 3 = 116.1-116.2
Average Min -Max=114.4-116.2
CF=3 B=4
test 1 = 112.3-114
test 2 = 116-117.1 Cont rec ok for a little while
test 3 = 116.1-119.7 Cont rec ok for a little while - touched 120!!
Average Min -Max=114.8-116.9
CF=2 B=4
test 1 = 114.9-115.9
test 2 = 116.2-117.1 Cont rec ok for a little while
test 3 = 117-118.4 Cont rec ok for a little while - touched 123!!
Average Min -Max=116-117.1
CF=1 B=4
test 1 = 117-115
test 2 = 113-117.6
test 3 = 115.2-116.9
Average Min -Max=115.1-116.5
CF=4 B=4
test 1 = 116.7-117.7 Cont rec ok for a little while
test 2 = 116.8-117 Cont rec ok for a little while
test 3 = 118.1-118.3 Cont rec ok for a little while
Average Min -Max=117.2-117.6
Note: At CF=5 and above the SD card does not initialize fast enough so recording stops quickly.
Best is CF=4 B=4
Congrats g3ggo. Amazing, blazing, speeding machine!
Quote from: xNiNELiVES on August 12, 2013, 09:05:17 PM
Well if anything the 5D Mark II should be better than the 50D. They were released at the same time, except the 5D2 was around 2000 dollars more.
The 5D Mark III has the same SD-Controller like the 600D. Why do you think the 5D Mark II should have a different CF-Card than the 50D?
Canon uses the same components for multiple cameras to keep the prices low.
Quote from: chmee on August 12, 2013, 07:49:45 PM
@g3gg0 what about a version freeze? i would begin to code a converter next days, but the things are changing almost daily. after this version freeze or a first reference mlv-file (with version-number) i could start..
regards chmee
the file format was fixed before i ever started a line of code.
it didnt change since then, except two fields.
so: the converters even from the first day would have been able to read this format the raw_rec outputs today
ok. thx - but then, its a pitty, we're talking lot 'bout the spanning (code) optimizing and less of mlv and advantages, hints and optimizing ideas for a coming version. but, so the roundbell ringed and i just start next days with the tool.
regards chmee
Quote from: chmee on August 12, 2013, 10:01:12 PM
ok. thx - but then, its a pitty, we're talking lot 'bout the spanning (code) optimizing and less of mlv and advantages, hints and optimizing ideas for a coming version. but, so the roundbell ringed and i just start next days with the tool.
regards chmee
what do you want to talk about mlv?
had the proposal two weeks in this forum and there was no discussion :)
just a few feature requests.
i just propose to split the spanning-mode optimization from this thread - finally i will code and test when im back at home, and then i can ponder about the (pro's and hopefully not con's) of the fileformat.
regards chmee
@RenatoPhoto:
i read through it and must say: wow, thanks a lot.
this helps me in finding the right sizes
Quote from: g3gg0 on August 12, 2013, 10:26:29 PM
@RenatoPhoto:
i read through it and must say: wow, thanks a lot.
this helps me in finding the right sizes
no, thank you!!
Just wondering if it would be a better test to turn off skipping frames. Running each mode until the recording stops, running it again maybe four or five times and then obtaining the total GB recorded. I suppose the best setup is the one that can record the most without skipping frames, so this test should be more applicable to the needs. In most cases, if the recording speed is stable, then the fastest recording will give you the longest file. The problem is that recording speed is not constant so maybe total GB of multiple recordings may be a more reliable number to find the best settings.
Quote from: g3gg0 on August 12, 2013, 05:39:24 PM
so your raw is in E:\ ?
Yes. I have an E drive with a folder caller rawFiles in it and I have dumped all the recorded files from both cards into it along with the mlv_dump.exe.
QuoteWell if anything the 5D Mark II should be better than the 50D. They were released at the same time, except the 5D2 was around 2000 dollars more.
50D is to 5DII as 6D is to 5DIII... similar firmware and components with only a few changes $2K+ more.
@RenatoPhoto:
here (http://upload.g3gg0.de/pub_files/3b98c60717a64e509a889ab81c4d34f8/raw_rec.mo) is a version with "test mode" that tries recording with iterating buffer count and method.
every test case is ran 4 times.
enable tracing and test mode from menu. spanning option is not changed during tests.
with my setup the frame count varies more than the buffering method option changes. :(
does it look better on your camera?
Where is tracing?
edit: you mean Debug trace?
I am running it and see:
test numb with variables M , B, W, F, and (speed)
What do they mean..
When does it stop?
My comments so far:
* Each test seems very short and only represents the beginning of recording. In my recordings (not in this test) I noticed that some buffers and CF numbers begin recording fast and then slow down while other options begin slow and increase the recording speed over time.
Is there a log saved?
Looks good
Here is my first version of mlv2dng. It's not finished yet and there is still a lot to be done.
What is not working:
Spanned files (could someone please supply me with some spanned files? I don't have a 5D3 to try it).
Preview image. (it shows random noise)
Dual ISO.
It only shows very limited amount of EXIF data, as I have still to do mappings from MLV to exif tags.
probably much more.
I have a lot of cleanup to do on the code before I submit it online. Will probably be online tomorrow or the day after.
Usage is the same as with raw2dng for the moment:
mlv2dng.exe file.mlv [prefix]
=> will create prefix000000.dng, prefix000001.dng and so on.
https://dl.dropboxusercontent.com/u/3181048/mlv2dng.zip (https://dl.dropboxusercontent.com/u/3181048/mlv2dng.zip)
Quote from: 1% on August 13, 2013, 01:26:45 AM
50D is to 5DII as 6D is to 5DIII... similar firmware and components with only a few changes $2K+ more.
Yeah I'm just saying that if anything the 5D2 should be just as fast, if not faster when it comes to write speed. Am I right?
Quote from: xNiNELiVES on August 12, 2013, 07:05:05 PM
1)
2)
3) Even if I stop recording, the buffer status never disappeared. The rest of the info disappeared after a good 15 seconds or so.
4) I don't enable it but what happens if someone did? I realize this is only an experimental copy of the new MLV Format, I guess you'll just disable it in the final versions.
Will trace the bug with the lag spike.
No bugs for me.
@gnarr:
thanks for supporting!
(it is crashing with no output on my pc)
things important to know (most of that is written already i think)
- files are numbered in the file itself
- there is a fileCount field and a fileNum (fileCount isnt fixed yet on all files, so dont trust it yet. will do that later)
- no matter if it is the second file after the first file reached 4GiB, or if it is a spanning file, the file is named M00, M01, ...
- all M00, M01, ... also contain a file header (mlv_file_hdr_t)
-> you see no difference in files from card spanning and files that went over 4GiB
- the initial blocks like IDNT, INFO etc are written in the first file before recording
- some of them (RTCI, LENS, EXPO, ELVL) are written periodically too
-> so dont rely on exact positions in file
- blocks are written out of order, which means frame #9 may be in file before frame #4 and frame #6 is in another file
- look at the timestamp to know which EXPO, RTCI, LENS, ELVL etc block belong to the video frame
-> need to build an internal sorted list of all blocks before metadata can be assigned to a frame
will upload a file, takes some time.
it will appear here (https://drive.google.com/folderview?id=0BwQ2MOkAZTFHb3lmaTMzUV9uMWc&usp=sharing), when its done.
(upload finished)
Quote from: g3gg0 on August 13, 2013, 11:22:29 AM
@gnarr:
thanks for supporting!
(it is crashing with no output on my pc)
things important to know (most of that is written already i think)
- files are numbered in the file itself
- there is a fileCount field and a fileNum (fileCount isnt fixed yet on all files, so dont trust it yet. will do that later)
- no matter if it is the second file after the first file reached 4GiB, or if it is a spanning file, the file is named M00, M01, ...
- all M00, M01, ... also contain a file header (mlv_file_hdr_t)
-> you see no difference in files from card spanning and files that went over 4GiB
- the initial blocks like IDNT, INFO etc are written in the first file before recording
- some of them (RTCI, LENS, EXPO, ELVL) are written periodically too
-> so dont rely on exact positions in file
- blocks are written out of order, which means frame #9 may be in file before frame #4 and frame #6 is in another file
- look at the timestamp to know which EXPO, RTCI, LENS, ELVL etc block belong to the video frame
-> need to build an internal sorted list of all blocks before metadata can be assigned to a frame
will upload a file, takes some time.
it will appear here (https://drive.google.com/folderview?id=0BwQ2MOkAZTFHb3lmaTMzUV9uMWc&usp=sharing), when its done.
Thanks for the notes and the files :) I will test this when I get home frome work.
one question for you g3gg0.
Frame size in the raw info header shows 4797520 for this video, because the camera was configured for 2080*1318, but of course you can only record max 1920*1288, so the real real frame size is 4327680 (as reported by the video frame header). I was wondering if this was a bug or if it is supposed to be like this?
g3gg0 and chmee you guys rock!
Awesomeness x 10
new version.
Now handles spanned files and hopefully won't crash :)
With spanned files, you just point to the first file and the program automatically detects if there are more files.
https://dl.dropboxusercontent.com/u/3181048/mlv2dng.zip (https://dl.dropboxusercontent.com/u/3181048/mlv2dng.zip)
Quote from: g3gg0 on August 13, 2013, 11:22:29 AM
-> need to build an internal sorted list of all blocks before metadata can be assigned to a frame
I have still to do this. Currently it just writes metadata from the last info block that was read. But the whole metadata thing is largely unfinished yet anyways :)
gnarr - It's working here. Exif is showing correctly. :)
Quote from: gnarr on August 13, 2013, 03:01:22 PM
new version.
Now handles spanned files and hopefully won't crash :)
With spanned files, you just point to the first file and the program automatically detects if there are more files.
https://dl.dropboxusercontent.com/u/3181048/mlv2dng.zip (https://dl.dropboxusercontent.com/u/3181048/mlv2dng.zip)
crashing still.
Quote from: g3gg0 on August 13, 2013, 01:49:21 AM
@RenatoPhoto:
here (http://upload.g3gg0.de/pub_files/3b98c60717a64e509a889ab81c4d34f8/raw_rec.mo) is a version with "test mode" that tries recording with iterating buffer count and method.
every test case is ran 4 times.
enable tracing and test mode from menu. spanning option is not changed during tests.
with my setup the frame count varies more than the buffering method option changes. :(
does it look better on your camera?
how do I delete the recorded raw video from the CF card in the camera
How do I use mlv2dng.exe with dual iso? The files that after converting with mlv2dng still contain the alternating iso traces. Cr2hdr does not help with removing the iso lines.
it's not yet implemented for dual iso. You will need to go the mlv_dump -> raw2dng way.
Is mlv2dng works like raw2dng? I mean drag and drop? I cannot run the app, it says that cannot find MSVCR110.dll, got it from post #322.
Windows 7 64bit.
Quote from: kgv5 on August 13, 2013, 04:58:54 PM
Is mlv2dng works like raw2dng? I mean drag and drop? I cannot run the app, it says that cannot find MSVCR110.dll, got it from post #322.
Windows 7 64bit.
Download Visual C++ redist from Microsoft: http://www.microsoft.com/en-us/download/details.aspx?id=30679
Download the 64bit version and if that doesn't work try the 32bit one. I had the same issue. Works now :)
Drag and drop works.
Quote from: gnarr on August 13, 2013, 04:55:49 PM
it's not yet implemented for dual iso. You will need to go the mlv_dump -> raw2dng way.
Hi gnarr,
Does not work with raw2dng.exe
Actually, once the mlv2dng.exe breaks the file up into individual frames it is not possible to run these through raw2dng.exe. The files should be cured with cr2dng.exe. But since these individual frames are already dng neither of the 2 programs works with these individual frames.
Any idea how this could be cured.
Also, while recording with both CF+SD card is there a way to get both the *.MLV and *.M00 files both on one card? Right now the *.MLV files are saved by the camera on the CF card and *.M00 on the SD card.
updated DL link in main post
http://upload.g3gg0.de/pub_files/68769d85bdb2f9753de068e9783f162a/raw_rec_mlv.zip
fixed:
- mlv_dump: handling of INFO chunks (text and take number) when rewriting MLV file
- mlv_dump: incorrect check of frame size when converting back to 14bpp
So, i take it this is for the 5DIII only.
Its CF camera only right now I think. 50D should run it.
My CF cards are quite slow anyway, 50D is the camera i want to sell amongst the 3.
Quote from: 1% on August 13, 2013, 06:38:53 PM
Its CF camera only right now I think. 50D should run it.
Confirmed working on 50D but I'm getting ~15-20% less frames ATM for the same write speeds as original raw_rec. Gnarr's mlv2dng app works too. Looking forward to when metadata is working.
Quote from: kgv5 on August 13, 2013, 04:58:54 PM
Is mlv2dng works like raw2dng? I mean drag and drop? I cannot run the app, it says that cannot find MSVCR110.dll, got it from post #322.
Windows 7 64bit.
I just uploaded a version compiled with GCC, so this should be fixed now.
QuoteMy CF cards are quite slow anyway, 50D is the camera i want to sell amongst the 3.
Vs 60D? I don't think it has anything going for it. Just a 600D with a better build and missing audio.
What do you mean?
He means that if any camera you should sell it should be the 60D not the 50D.
Quote from: Andy600 on August 13, 2013, 05:06:43 PM
Download Visual C++ redist from Microsoft: http://www.microsoft.com/en-us/download/details.aspx?id=30679
Download the 64bit version and if that doesn't work try the 32bit one. I had the same issue. Works now :)
Drag and drop works.
Thank you Andy600, it works now (with 32bit version) :D
Does anybody know if somebody is making a os x app?
or updating RAWMagic for MLVs?
I know that mlv2dng is very much work in progress and just early stage but is there any possibility of batch file processing yet?
Did somebody succeed to implement this in any existing raw2dng solution?
Quote from: kgv5 on August 14, 2013, 09:26:21 AM
I know that mlv2dng is very much work in progress and just early stage but is there any possibility of batch file processing yet?
Did somebody succeed to implement this in any existing raw2dng solution?
You can actually run multiple copies of raw2dng if that is what you are asking?
I usually open 4 instances and convert that way.
Yeah, Thats what i do, thanks.
Alex vs G3ggo compiles for speed without spanning. In all of G3GGo builds used Buffer=4 and CF = 3
Lexar Pro 1000x
Frame skip OFF to see who can record longer without dropping frames.
Test 1 1920x1080 at 29.97
Alex
97-102.8
102.4-103.4
102.9-103.7
Total GB recorded 18.7 GB
G3GGo
95-103
103.5-103.4
103.2 101.9
Total GB recorded 17.2 GB
Alex is slightly faster and more stable
----------------------------------------
Test 2 Cropped 3X 2048x1152 at 29.97
Alex
84-103.2 recorded 13 seconds
103-104.7 recorded 13 seconds
81-102.8 recorded 13 seconds
Total GB recorded 4.65 GB
G3GGo
80-103.7 recorded 8 sec
75-86 recorded 5 sec
76-95.8 recorded 5 sec
Total GB recorded 3.07
Alex is slightly faster and more stable
-----------------------
Test 3 1728x606 60 fps
Alex
82-90.3 recorded 15 seconds
80-91.8 recorded 17 seconds
75-90.6 recorded 115 seconds
Total GB recorded 4.91 GB
G3GGo
73-86 recorded 9 sec
78.6-88.5 recorded 11 sec
78-87 recorded 9 sec
Total GB recorded 3.17
Alex is faster and more stable
Hopefully the G3GGo can get up to Alex speed. Of course with spanning enabled 5D3 can record significantly faster then Alex module.
Could someone post a (or send me) short dual-iso mlv file?
I think their Q/DQ methods tie up the CPU too much, these tasks are really slow to complete, and on a slow cpu the things get even worse. There might be ways to free up the CPU from these seeking jobs to let it do what CPU is supposed to do -- like compressing files. I still sturbbonly believe that the DIGIC4 is fast enough to do something like /4 an interger. :D
Quote from: Canon eos m on August 13, 2013, 03:57:46 PM
how do I delete the recorded raw video from the CF card in the camera
Use the File Browser in ML and select and highlight delete :)
Quote from: mucher on August 15, 2013, 03:59:51 PM
I think their Q/DQ methods tie up the CPU too much, these tasks are really slow to complete, and on a slow cpu the things get even worse. There might be ways to free up the CPU from these seeking jobs to let it do what CPU is supposed to do -- like compressing files. I still sturbbonly believe that the DIGIC4 is fast enough to do something like /4 an interger. :D
The cpu is currently not involved at all.
The video is copied by dma controllers and they cant divide.
Quote from: gnarr on August 15, 2013, 03:45:39 PM
Could someone post a (or send me) short dual-iso mlv file?
Dual iso support isnt implemented yet.
But here some example files: https://docs.google.com/folder/d/0BwQ2MOkAZTFHb3lmaTMzUV9uMWc/edit
Folder contains a spanning mlv, a compressed one, 10 bit uncompressed and compressed etc.
I just updated mlv2dng.
Fixed / working:
Headers are now correctly matched to frames via timestamps.
Almost all available exif and ifd info should be included in dng's.
There shouldn't be any memory leaks.
file spanning.
Little bit of error output in case it crashes.
What doesn't work / todo:
preview image. (although I'm closer to getting it right now.. will be there soon)
converting mlv to mlv for changing file compression or bit level.
converting non 14 bit mlv to dng.
converting LZMA compressed files to dng
converting dual iso mlv to dng.
more detailed error logging.
https://dl.dropboxusercontent.com/u/3181048/mlv2dng.zip (https://dl.dropboxusercontent.com/u/3181048/mlv2dng.zip)
Quote from: g3gg0 on August 15, 2013, 06:35:55 PM
Dual iso support isnt implemented yet.
But here some example files: https://docs.google.com/folder/d/0BwQ2MOkAZTFHb3lmaTMzUV9uMWc/edit
Folder contains a spanning mlv, a compressed one, 10 bit uncompressed and compressed etc.
Thanks :) this will help a lot with development.
Quote from: gnarr on August 16, 2013, 02:12:08 AM
What doesn't work / todo:
converting mlv to mlv for changing file compression or bit level.
this is not needed, as it is the mlv_dump main use case :)
Is there a version yet for the 5D2 to test run and check? :)
5D2: no
5D3: here (http://upload.g3gg0.de/pub_files/d23d152a90a1f5ab12f65a7f7ce6aa17/raw_rec.mo) the latest one with some write performance gain
getting continuous 1920x1280 at 23.976 fps with CF only :)
@g3gg0
(*) Format-Description - RAWI-Block - isnt it better to refer to the raw.h (https://bitbucket.org/hudson/magic-lantern/src/6f63166b128a0768dc6ed9089875459d929f4fec/src/raw.h?at=unified)?
(*) would it make sense to put a badpixel-block into? i havent seen it. "we" could write a camera-module to mark bad pixels via 100%/200%/400%-photoview, then saving their positions.. this data we could use in the converter-modules to recalculate the badpixels, taking into account, that they're static and marking has to be done only once..
Blocktype - BPIX
Blocksize - 10+bpixCount*4
bpix-count - uint16
array of (x/y) uint16/uint16
regards chmee
For Time Lapse Interested RAW "video" High-resolution, the speed of the card is not as important (for example frequency of 1 frame per second). I can not understand now have that opportunity. Camera 60D.
Quote from: chmee on August 16, 2013, 11:57:24 AM
@g3gg0
(*) Format-Description - RAWI-Block - isnt it better to refer to the raw.h (https://bitbucket.org/hudson/magic-lantern/src/6f63166b128a0768dc6ed9089875459d929f4fec/src/raw.h?at=unified)?
right, but i wanted to copy-paste the raw structure into the mlv header file.
not sure if i should put the 'struct raw_info' into mlv.h too, refer to the raw.h or if
i should put the fields from 'struct raw_info' in the RAWI block in the same format.
Quote from: chmee on August 16, 2013, 11:57:24 AM
(*) would it make sense to put a badpixel-block into? i havent seen it.
thanks for the hint.
at the moment there is a rudimentary method for detecting such pixels.
you can record an unlimited number of frames with lens cap on and allowed frame skipping.
using mlv_dump you can average all frames in the video (using './mlv_dump -a -o ref.mlv lenscap.mlv')
the output mlv is a single video frame with all noise (sensor and readout) and deviations (sensel offsets, dead sesels) averaged together.
using the subtract command (using './mlv_dump -s ref.mlv -o out.mlv in.mlv') , this reference frame can get subtracted from every single video frame.
this will eliminate some kinds of noise and will make your dead pixels dark.
this file we could use to reconstruct the dead pixels too.
Quote from: Nikita_MV on August 16, 2013, 01:02:13 PM
For Time Lapse Interested RAW "video" High-resolution, the speed of the card is not as important (for example frequency of 1 frame per second). I can not understand now have that opportunity. Camera 60D.
not yet. sorry.
@g3gg0
i like the idea of simple subtracting - but every time you change the resolution, you have to think of recording the lenscap-video. is there any examplevideo of your approach? practically i think there are no more than ~10badpixels on whole picture - 15 minutes of marking them and having static data for every kind of video (luckily we have the sensor-offset to do the calculations)..
but in general - there's already this approach to start immediately. and its able to neutralize noise and other sideeffects. like it.
regards chmee
Yeah it might no be the perfect solution, but a good start.
As soon we know how bad pixels can be detected, we can specify the format.
Is a simple x/y coordinate of defective pixels enough?
Or do we have to save more?
Quote from: g3gg0 on August 16, 2013, 11:10:35 AM
5D2: no
5D3: here (http://upload.g3gg0.de/pub_files/d23d152a90a1f5ab12f65a7f7ce6aa17/raw_rec.mo) the latest one with some write performance gain
getting continuous 1920x1280 at 23.976 fps with CF only :)
:)
i tried to find an example-tag inside a cr2-file - but -awakening! - there s no tag for it - so a logical conclusion is, a bad/hot-pixel-tag is only existent in DNG, because they're found/analysed only in the converter :) look into the chdk-description (http://chdk.wikia.com/wiki/Badpixel_removal).. so, finally it makes sense to give an int8 to any pixelposition - and we try to use them if we know more about hot/dead/bad-pixel- for now it's ok if we tell
0 = defective (recalculate with neighbours-data - general approach if no info about kind of problem)
if we had any information about kind of badpixel and/or differing repairalgorithm
1 = hotpixel on hi-iso
2 = hotpixel on longexposure
4 = recalculate only red
8 = rec.e only blue
16 = rec. only green
and so on.. for now i dont know for sure.
Quote from: g3gg0 on August 16, 2013, 11:10:35 AM
5D2: no
5D3: here (http://upload.g3gg0.de/pub_files/d23d152a90a1f5ab12f65a7f7ce6aa17/raw_rec.mo) the latest one with some write performance gain
getting continuous 1920x1280 at 23.976 fps with CF only :)
Here are my test results using your latest compared to Alex's latest. I ran each test 7 times and then removed the 2 outlier results...
Buffer fill method: 0
CF-only buffers: 3
Memory hack: On
Small hacks: On
1920 x 1080, 29.97 fps----------------------------
g3gg0 a1ex g3gg0 CF CF w/Span 634 1417 3002 frames
754 1446 3594
633 1622 2563
663 1638 3006
781 1635 2603
1920 x 672, 48.00 fps---------------------------
g3gg0 a1ex g3gg0 CF CF w/Span 391 751 550
442 751 642
392 755 634
389 680 682
439 745 684
For 29.97 fps and spanning enabled, I got double the frames of Alex's latest. Nice :). But without spanning, I got double the frames using Alex's.
For 48.00 fps, Alex's latest always performed significantly better.
Nice test! I really like the spanning feature and all the other possibilities of course adding the new format. Also very curious what spanning would accomplish in Alex,s raw format :).
Really nice to follow the skillfull developing going on.
//D
another bit of performance increase, but not much.
here (http://upload.g3gg0.de/pub_files/8371bd0ed7e9de8c55985de2d838cccf/raw_rec.mo)
always try to use buffering method 4.
btw, couldnt get more than 1000 frames with alex code
Quote from: g3gg0 on August 17, 2013, 12:54:43 AM
another bit of performance increase, but not much.
here (http://upload.g3gg0.de/pub_files/8371bd0ed7e9de8c55985de2d838cccf/raw_rec.mo)
always try to use buffering method 4.
New code gave me slower results for 1920x1080, and cropped digital video 2048x1152 but slightly faster rates for high speed at 1728x606 at 60 fps.
Test 1 1920x1080 at 29.97Yesterdays test G3GGo
95-103
103.5-103.4
103.2 101.9
Total recorded on three tests 17.2 GB
Today's latest code:
86-102
103-100
101-103
Total recorded on three tests 16 GB
Test 2 Cropped 3X 2048x1152 at 29.97Yesterdays G3GGo
80-103.7 recorded 8 sec
75-86 recorded 5 sec
76-95.8 recorded 5 sec
Total recorded on three tests 3.07 GB
Todays G3GGo
86-96
101-102
78-96
Total recorded on three tests 2.39 GB
Test 3 1728x606 60 fpsYesterdays G3GGo
73-86 recorded 9 sec
78.6-88.5 recorded 11 sec
78-87 recorded 9 sec
Total recorded on three tests 3.17 GB
Todays G3GGo
86-101
90-90.8
90-85
Total recorded on three tests 3.26
Slightly better speed on this videos.
Why is Alex not involved in this thread btw?
It would make sense to involve him, doesn't it?
Premature optimization is the root of all evil... :)
mlv2dng previews are working 8)
https://dl.dropboxusercontent.com/u/3181048/mlv2dng.zip (https://dl.dropboxusercontent.com/u/3181048/mlv2dng.zip)
really cool :)
btw, do you also get "failed to get properties" for serial numbers with the latest build?
okay, fixed.
(here (http://upload.g3gg0.de/pub_files/db058a6bd0bc2c3e6a3ad1c5a9850c54/raw_rec_mlv.zip) with 5D3 autoexec/sym)
i played with edmac alignment in this version. please report if you get pink frames or lock ups.
@gnarr:
can you convert the camera serial number to a 12 digit decimal?
e.g. 110091F150 (hex) is 073024008528 (dec)
in camera i wanted to prevent 64 bit integer to string conversion, so i decided to put it as hex string.
Quote from: g3gg0 on August 18, 2013, 12:46:01 AM
@gnarr:
can you convert the camera serial number to a 12 digit decimal?
e.g. 110091F150 (hex) is 073024008528 (dec)
done :)
Quote from: g3gg0 on August 18, 2013, 12:46:01 AM
okay, fixed.
(here (http://upload.g3gg0.de/pub_files/db058a6bd0bc2c3e6a3ad1c5a9850c54/raw_rec_mlv.zip) with 5D3 autoexec/sym)
i played with edmac alignment in this version. please report if you get pink frames or lock ups.
@gnarr:
can you convert the camera serial number to a 12 digit decimal?
e.g. 110091F150 (hex) is 073024008528 (dec)
in camera i wanted to prevent 64 bit integer to string conversion, so i decided to put it as hex string.
Hi g3gg0 - quick question - downloaded the most recent build for raw - but am a bit confused - is raw2dng the same as mlv2dng? With the new autoexe.bin, raw_rec.mo, etc. will the camera retain the ability to shoot mlv or will it revert to the previous raw format?
Its the same as all the previous releases.
It records .mlv, you can convert it to raw and from raw to dng.
mlv2dng is released by gnarr in separate posts.
Quote from: g3gg0 on August 18, 2013, 10:56:37 AM
Its the same as all the previous releases.
It records .mlv, you can convert it to raw and from raw to dng.
mlv2dng is released by gnarr in separate posts.
Thanks. The new version is skipping frames massively. Skipping after about 250 frames. Total output speed on both cards total is about 92 mb/s. Any idea what I could be doing wrong? Following the same workflow as before (previous MLV version).
I understand its for 5D3 only atm, but when reading trough about it, will it work the same on the Mk2 or are there features that will be Mk3 only in the format?
Quote from: Canon eos m on August 18, 2013, 11:14:03 AM
Thanks. The new version is skipping frames massively. Skipping after about 250 frames. Total output speed on both cards total is about 92 mb/s. Any idea what I could be doing wrong? Following the same workflow as before (previous MLV version).
Now 102.5mb/s with skipping after 800 frames.
Also, how do I convert from MLV to RAW. I believe that the mlv2dng will convert directly to dng format?
Is Gnarrrs mlv2dng converter supposed to work with drag and drop with the .mlv files straight out of the camera? I have some .mlv files but the cmd prompt just closes when I try?
Quote from: Danne on August 18, 2013, 12:02:51 PM
Is Gnarrrs mlv2dng converter supposed to work with drag and drop with the .mlv files straight out of the camera? I have some .mlv files but the cmd prompt just closes when I try?
Yes it is drag and drop over the .exe but some previously recorded files do not activate for me too. Maybe a change in format that the new mlv2dng does not recognize.
I see, probably because I tried with some older files. Thanks
Is there anything I can do to help the progress of this development? :)
@g3gg0 and all mlv-devs I'm a little bit confused about the data inside/behind the first VIDF-Block. Why are there more other Blocks inside it? (This VIDF-Blocksize is that high, i cant iterate thru the blocks. What did i missed?
(http://s1.directupload.net/images/130819/mxxjozd7.jpg)
red = blocksize
green = blockposition
blue = framespace
Till the VIDF-Block i iterated thru the blocks via (blockPos+blockSize)=newBlockPos
regards chmee
Don't forget that framespace is a 32bit number, so it's 0x00000F7C.
So the actual frame size is blocksize - (framspace + 32) and you start reading the frame at the framespace offset.
these blocks are leftovers from memory that hasnt been cleared.
it is just crap, ignore it.
when doing EDMAC copies of the video frames, the destination address must be aligned to 4k borders. (sometimes...)
thats what the frameSpace is for.
ignore anything what is in this space, just follow the block sizes.
btw: the blocks are called VIDF. VideoFrame
lol. of course vidf..
@gnarr
i can calculate the framesize as well via width * height *4*14 / 8 (w*h*7) [bytes]. but in general i was confused by the garbage in the edmac- alignment-space.
@g3gg0
in the raw-fileformat the picture offset was bigger than the videoframe, because of blockcopy-sizes? is it here as well? (ah, got it..) it seems, its to expect, that after the first vidf will come a NULL for this data, and then the pictures start..
Quote from: chmee on August 19, 2013, 10:46:53 PM
@gnarr
i can calculate the framesize as well via width * height *4*14 / 8 (w*h*7) [bytes]. but in general i was confused by the garbage in the edmac- alignment-space.
uhmm. do not hardcode 14 bit, please.
the format supports 1-16 bit depth.
frameSize = (width * height * bpp + 7) / 8
the frame data starts at frameOffset bytes behind the VIDF header end.
see the VIDF block as one frame, header as described in mlv.h, some frame space for alignment and finally the data.
behind the frame data there may also be some empty space.
you have to use the RAWI information to get the real frame data size.
again, look into mlv_dump.c how i handle this.
Quote from: chmee on August 19, 2013, 10:46:53 PM
@g3gg0
in the raw-fileformat the picture offset was bigger than the videoframe, because of blockcopy-sizes? is it here as well?
which pictureoffset?
(which offset) the same as you described. to optimize edmac copies to the sd/cr-card a1ex (i think) moved/saved blocks. framesize!=dataframe
thx. regards chmee
Is there any test footage I can play with? I looked through and found M13-0957.MLV but it doesn't seem to open with mlv_dump.exe
EDIT: Oops, I just found the MLV2DNG - works a treat!
Is there a compiled version of 2.0 that I can just dump on to my card? Or a quick tutorial on how to get it loaded? When I used it with the 5D3 8 August release (just replaced the raw record module) I get a link error.
Quote from: chmee on August 19, 2013, 10:46:53 PM
in the raw-fileformat the picture offset was bigger than the videoframe, because of blockcopy-sizes? is it here as well? (ah, got it..) it seems, its to expect, that after the first vidf will come a NULL for this data, and then the pictures start..
the picture offset (=frameSpace) is at maximum 4k, where the videoframe is several MiB.
about the NULL data:
when there is some metadata like LENS, RTCI, EXPO etc, i am decreasing the frameSpace and moving the VIDF header to use that gained space for these blocks.
this allows me to put additional data without losing any write performance.
here some example:
assume a block with that size, which is quite good for writing
[empty......................... ....................... ............................ .............]
now it gets filled with a VIDF only, and some frameSpace for edmac and write alignment. size stays the same of course.
[VIDF__frameSpace...........................__datadatadatadata........datadatadatadata][NULL......]
now there is some [LENS] to embed.
I remove all frameSpace to zero and put a NULL block with that size instead and insert the [LENS] block before.
[NULL.................................][VIDF__datadatadatadata........datadatadatadata][NULL......]
[LENS][NULL...........................][VIDF__datadatadatadata........datadatadatadata][NULL......]
now there is some [EXPO] and then [RTCI] to embed, too:
[EXPO][LENS][NULL...............][VIDF__datadatadatadata........datadatadatadata][NULL......]
[RTCI][EXPO][LENS][NULL.........][VIDF__datadatadatadata........datadatadatadata][NULL......]
later, when the frame was written and is going to be reused again, the header in front is written again
[VIDF__frameSpace...........................__datadatadatadata........datadatadatadata][NULL......]
and you see that at the same position where [frameSpace...] is now, previously some other blocks like the LENS etc block have been.
i could clear this area to prevent irritation, but i wanted to save CPU load and thus kept the crap there.
to be honest, i could have implemented frameSpace also using NULL, as it is redundant. but i didnt want to touch that again.
in future releases frameSpace might be set to zero and alignments (both edmac and write) could be realized with NULL blocks only.
the only advantage with frameSpace in VIDF is that i dont have to seek while updating the VIDF header from vsync cbr. but thats not much.
Quote from: sletts02 on August 20, 2013, 06:14:46 AM
Is there a compiled version of 2.0 that I can just dump on to my card? Or a quick tutorial on how to get it loaded? When I used it with the 5D3 8 August release (just replaced the raw record module) I get a link error.
see some posts earlier, there is a package for 5D3
I see most of this thread is actually for those smart enough to do coding for it(not I)
So I have a couple of questions.
with the spanning to both cf and sd cards, I have to grab the recordings off of BOTH cards and use a software that will combine the information from both cards to make 1 video?
If so, is the software to do this and allow us to edit these videos and use them(like we can the preview .raw container) available and working? How about for a non programer like myself(running mac os x)?
Quote from: g3gg0 on August 20, 2013, 09:46:52 AM
see some posts earlier, there is a package for 5D3
Found it, cheers!
I was hoping that LRTimelapse would help solve my auto exposure hard stopping problem, but it doesn't seem to have :/ Not sure why it's not working, I'd have thought it would have with you EXIF data in each file.
hmm can you look at the raw frames - do they have the right exposure value embeded, or is it wrong?
i know that there will be a few frames delay with the metadata.
going to fix this soon.
Quote from: g3gg0 on August 20, 2013, 12:51:08 PM
hmm can you look at the raw frames - do they have the right exposure value embeded, or is it wrong?
i know that there will be a few frames delay with the metadata.
going to fix this soon.
Will have a look tomorrow night! Keep up the great work g3gg0. This new format seems to be a great improvement, especially if Alex can implement the sidecar method that he has developed for POST FLICKER.
@g3gg0 thx for the hints and infos.
thinking about the definition, it would be good to write down some restrictions and/or principles. (lot of that sounds logical, but it's worth to say!)
f.i.
(*) Fileheader (file_hdr) is the first block.
(*) info-blocks must be before the first VIDF-Block (f.i. except RTC)
(*) AUDF-block will be found after every VIDF-block. (or else?)
(*) importance-notation of every block-type (f.i. mandatory, recommended, optional)
(*) every VIDF-Block has its own header, likewise splitted VIDF-Blocks
(*) VIDF-Blocks are splitted bytewise
discussion sides before
(*) filesize is depending on fileformat, at least 4GB
or
(*) files will be splitted at 4GB regardless of fileformat
module raw_rec
(*) notation of coding-datetime/version, from when mlv was supported with all mandatory blocks.
(*) notation, when a blocktype was implemented.
regards chmee
Where can we download the latest MLV2DNG? I can't find it anywhere... :-/
Quote from: Danne on August 18, 2013, 12:02:51 PM
Is Gnarrrs mlv2dng converter supposed to work with drag and drop with the .mlv files straight out of the camera? I have some .mlv files but the cmd prompt just closes when I try?
Quote from: Canon eos m on August 18, 2013, 12:18:26 PM
Yes it is drag and drop over the .exe but some previously recorded files do not activate for me too. Maybe a change in format that the new mlv2dng does not recognize.
Quote from: Danne on August 18, 2013, 12:41:21 PM
I see, probably because I tried with some older files. Thanks
This should now be fixed. Old versions of mlv files should be decoded properly.
Please report any errors you have :) Either with a PM or on this thread. (http://www.magiclantern.fm/forum/index.php?topic=7802)
https://dl.dropboxusercontent.com/u/3181048/mlv2dng.zip (https://dl.dropboxusercontent.com/u/3181048/mlv2dng.zip)
and here is a linux version:
https://dl.dropboxusercontent.com/u/3181048/mlv2dng_linux.zip (https://dl.dropboxusercontent.com/u/3181048/mlv2dng_linux.zip)
Now all we need is one for all of us in Macland... :-/
as soon as I finish downloading an OsX vmware image I will compile and post it here :)
Quote from: chmee on August 20, 2013, 07:05:08 PM
thinking about the definition, it would be good to write down some restrictions and/or principles. (lot of that sounds logical, but it's worth to say!)
yep, good point. let me paraphrase them.
[MLV Format]
- the Magic Lantern Video format is a block-based file format
- every information, no matter if audio or video data or metadata is written as data block with the same basic structure
- this basic structure includes block type information, block size and timestamp (exception to this is the file header, which has no timestamp, but a version string)
- the timestamp field in every block is a) to determine the logical order of data blocks in the file and b) to calculate the wall time distance between any of the blocks in the files
- the file format allows multiple chunks which basically are in the same format with file header and blocks
- chunks are either sequentially written (due to e.g. 4 GiB file size limitation) or parallel (spanning over mutiple media)
- the first chunk has the extension .mlv, subsequent chunks are numbered .m00, m01, m02, ...
- there is no restriction what may be in which chunk and what not
[processing]
- to accurately process MLV files, first all blocks and their timestamps and offset in source files should get sorted in memory
- when sorting, the sorted data can be written into a XREF block and saved to an additional chunk
- files that only contain a XREF block should get named .idx to clarify their use
- do not rely on any order at all, no matter in which order they were written into a file
- the only reliable indicator is the timestamp in all headers
[MLVI]
- MLVI block is the first block in every .mlv file
- the MLVI block has no timestamp, it is assumed to have timestamp value 0 if necessary
- the MLVI block contains a GUID field which is a random value generated per video shoot
- using the GUID a tool can detect which partial or spanning files belong together, no matter how they are named
- it is the
only block that has a fixed position, all other blocks may follow in
random order - fileCount field in the header may get set to the number of total chunks in this recording
[VIDF] (periodic)
- the VIDF block contains encoded video data in any format (H.264, raw, YUV422, ...)
- the format of the data in VIDF blocks have to be determined using MLVI.videoClass
- if the video format requires more information, additional format specific "content information" blocks have to be defined (e.g. RAWI)
- VIDF blocks have a variable sized frameSpace which is meant for optimizing in-memory copy operations for address alignment. it may be set to zero or any other value
- the data right after the header is of the size specified in frameSpace and considered random, unusable data. just ignore it.
- the data right after frameSpace is the video data which fills up the rest until blockSize is reached
- the blockSize of a VIDF is therefore
sizeof(mlv_vidf_hdr_t) + frameSpace + video_data which means that a VIDF block is a composition of those three data fields
- if frames were skipped, either a VIDF block with zero sized payload may get written or it may be completele omitted
- the format of the data in VIDF frames may change during recording (e.g. resolution, bit depth etc)
- whenever in time line a new content information block (e.g. RAWI) appears, the format has to get parsed and applies to all following blocks
[AUDF] (periodic)
- see [VIDF] block. same applies to audio
[RTCI] (periodic, event triggered)
- contains the current time of day information that can be gathered from the camera
- may appear with any period, maybe every second or more often
- should get written before any VIDF block appears, else post processing tools cannot reliable extract frame time
[LENS] / [EXPO] / ... (periodic, event triggered)
- whenever a change in exposure settings or lens status (ISO, aperture, focal length, focus dist, ...) is detected a new block is inserted
- all video/audio blocks after these blocks should use those parameters
...to be continued
Is an Alpha/beta in sight soon for the mark2?
Cant wait to try this new format out :)
not yet, sorry.
Quote from: Toffifee on August 12, 2013, 07:33:55 PM
How about making a benchmark test that checks the different methods and logs it?
In a similar fashion A1ex made when RAW video first came out.
A bunch of people can do the tests and you could merge the data into one to compare the differences.
Where can I find A1ex's version of the RAW video module (with the binary) or is there no such thing?
I suggest you to start looking here: http://www.magiclantern.fm/forum/index.php?board=49.0
Works great, thank you, I was able to unpack both to DNGs and to RAW. But I can't see wav file and it doesn't seems that sound is recording - how to switch it on?
Sound recording does not work(yet).
Quote from: g3gg0 on August 20, 2013, 10:20:53 PM
[MLV Format]
..
- there is no restriction what may be in which chunk and what not
[processing]
- to accurately process MLV files, first all blocks and their timestamps and offset in source files should get sorted in memory
..
- do not rely on any order at all, no matter in which order they were written into a file
- the only reliable indicator is the timestamp in all headers
[MLVI]
..
- it is the only block that has a fixed position, all other blocks may follow in random order
- whenever in time line a new content information block (e.g. RAWI) appears, the format has to get parsed and applies to all following blocks
...to be continued
@g3gg0 excellent work. i quoted some points i'm not that sure.. i'm a little bit scared about seeking thru the chunks, finding all important info blocks. it's also possible, that in spanned-mode f.i. idnt or wbal could be found in the second file - thats the freedom of the format-description - am i right?
regards chmee
Quote from: chmee on August 21, 2013, 10:13:37 PMit's also possible, that in spanned-mode f.i. idnt or wbal could be found in the second file - thats the freedom of the format-description - am i right?
thats the point. it might be at the end of the last file or in the middle of the first. anywhere.
this is the reason why there is a XREF block (defined) which contains file number and offsets to the blocks.
the entries in this block are in order, lowest timestamp first, highest timestamp as last.
it looks like this:
[XREF]
(fileNum) (fileOffset)
0 0x100
0 0x120
0 0x1A0
1 0x80
0 0x180
1 0xA0
...
which means: first block is in file 0 at offset 0x100, second in file 0 at 0x120, next in file 0 at 0x1A0, next is in file 1 at offset 0xA0.
... and so on.
of course tools have to build this index and (optionally, but recommended) save it into another file called .idx
if you have this index, reading in correct temporal order, is just going throug that XREF list.
Ran some video test which I converted directly to dng mlv2dng.exe
1920x1080 at 29.97 - Perfect
Cropped 1920x1080 at 29.97 - Some pink frames and pink noise frames
60 fps 1600x560 at 60 fps - Some pink frames and pink noise frames
same pink frames with mlv_dump and raw2dng?
Quote from: g3gg0 on August 23, 2013, 12:13:10 AM
same pink frames with mlv_dump and raw2dng?
Yes, exactly the same frame number if processed with mlv2dng or mlv_dump + raw2dng
Quote from: g3gg0 on August 23, 2013, 12:13:10 AM
same pink frames with mlv_dump and raw2dng?
I was checking out the Pre-Built Module of MLV, I only use the rec-mo file because I work on a Mac and the exe files are irrelevant to me on a Mac, but when I use the module on my Canon 5D Mark II, it freeze so I have no choice but to reboot.
Quote from: RenatoPhoto on August 22, 2013, 11:01:50 PM
Ran some video test which I converted directly to dng mlv2dng.exe
1920x1080 at 29.97 - Perfect
Cropped 1920x1080 at 29.97 - Some pink frames and pink noise frames
60 fps 1600x560 at 60 fps - Some pink frames and pink noise frames
Sorry to point this out here but the topic is related. On the EOS M, I find if you increase the fps beyond 30, the camera will throw a few pink frames first before it stabilized. Anything up to 30 of less fps is fine.
Quote from: dariSSight on August 23, 2013, 05:01:17 PM
I was checking out the Pre-Built Module of MLV, I only use the rec-mo file because I work on a Mac and the exe files are irrelevant to me on a Mac, but when I use the module on my Canon 5D Mark II, it freeze so I have no choice but to reboot.
It is not made for the Mark II, yet =)
hello,
this all sounds very interesting..
can anyone please post a link to a 5dmk3 build with the new format.. i am looking forward to test it.
thank you.
sw
Quote from: swinxx on August 24, 2013, 11:14:04 AM
hello,
this all sounds very interesting..
can anyone please post a link to a 5dmk3 build with the new format.. i am looking forward to test it.
thank you.
sw
What about a look into the very first post in this threat?
By the way: 5D2 support is much appreciated ;)
thank you.
i see the mlvdump is available for windows and linux..
is there a mac version?
thx.
Quote from: swinxx on August 25, 2013, 11:09:02 AM
thank you.
i see the mlvdump is available for windows and linux..
is there a mac version?
thx.
Try this: http://www.magiclantern.fm/forum/index.php?topic=7802.0
thx, will check that link.
is it possible that someone can provide a new raw rec mlv module? i can see that there were changes.. but the module is from 13_8..
thx.
Where is the link to the latest build? This post is getting confusing... :-/
latest build is in first post.
it contains the smpte module, which you have to disable if you dont use smpte. (audio menu or disable module loading)
Sorry, I'm a bit confused as well.
Does this new format currently support synced audio?
Any chance someone can do a Mk2 version soon to play around with? =)
Considering it seems to work quite well on the Mk3.
it supports no audio recording yet.
SMPTE (LTC coded audio) is an experimental module that outputs a sound signal that can be read and processed (or not) with special equipment.
is there a mac conversion software? would love to test this out.
Quote from: Doyle4 on August 28, 2013, 02:00:10 PM
is there a mac conversion software? would love to test this out.
Reply 420
at g3ggo: 5D3 does not compile
error:
module.o: In function `module_shutdown':
module.c:(.text+0x2328): undefined reference to `module_save_configs'
collect2: error: ld returned 1 exit status
make[1]: *** [magiclantern] Error 1
make[1]: Leaving directory `/home/magiclantern/G3GGo/magic-lantern/platform/5D3.113'
make: *** [5D3] Error 2
It's possible to add the delta of exposure value (the value in XMP line crs:Exposure2012=) to 2.0 specs?
So that the converter can recreate other than DNG files also xmp file and you can obtain a perfect deflickered files.
I am often in similar situation of this: http://www.magiclantern.fm/forum/index.php?topic=7739.0 or with some little-than-normal-speed timelapse with sunset for example
but sorry I can't offer 100$ :)
That value has to be computed from raw data, and it's CPU-intensive, so what's the point of including it in the raw format?
For sunset (especially for the last part) I use often RAW videos because I need to use values under 1 second not available in intervalometer like 2-5fps
but I really miss a lot the deflicker possibility.
It's very important to achieve a good results because the luminosity change a lot in small time.
Maybe is possible to write the raw exposure data and compute it after in the converter?
So we can lighten the CPU.
After all If I understood correctly the frame exposure data used (aperture, iso and shutter value) are already present in 2.0 format.
Thank you much.
That would be a feature in the converter, nothing to do with metadata. Not even exposure values.
Thank you,
I was think that ML compute the XMP exposure number with the difference between the exposure calculated and the exposure that camera apply.
So what raw data use ML to calculate XMP?
Can you calculate it in post without any additional data other than the standard exif?
That's how post deflicker is working. Deflicker data is just the median of the green channel (simplest implementation). If you want all channels, you need to apply WB before median.
Ok, last thing.
For the median of green channel you mean the measured green channel value and not the green channel value of the taken photo, right?
So for an implementation of post deflicker in the converter you need to have (other than the frame exposed data used (aperture, iso and shutter value)) also the measured green channel value that normally you don't have, neither in exif of CR2's files right?
Thank you.
No, you compute it in post from the pixels that you find in the video file. You can't do that in the camera...
Perfect :) , This time I hope this time I understood correctly :) :)
With an implementation of post-deflicker in the converter (without add any values other than frame exposed data used) we can obtain the SAME results of the post-deflicker correction value calculated on-camera in case of intervalometer still photo, right?
Yes, or better.
I'll actually move the post deflicker to a PC-based tool again (it will operate on raw photo sequences, no matter how you got them - even from Nikon).
Added a small proposal for the OFFS block (black correction info), and specified the units for digital ISO field in EXPO block (0.001 EV).
ok, cool.
will the EXPO update change the format as it is now?
i.e. would the parsers have to update their code to match the changes?
(it is ok, we just have to bump the version then)
I cannot check the changes as google docs is proxy-filtered.
I don't think any parser is using the digital ISO other than printing it. I've just updated the value from 1 to 1024 (so it's no longer limited to integer EV).
ok thanks :)
Quote from: a1ex on August 29, 2013, 11:39:53 AM
Yes, or better.
I'll actually move the post deflicker to a PC-based tool again (it will operate on raw photo sequences, no matter how you got them - even from Nikon).
Exciting stuff a1ex! Is there a thread that I can follow regarding this?
Happy to still put the $100 down once you've got it implemented ;)
Great work!
I'm a bit confused about the plan to implement LZMA compression (MLV_VIDEO_CLASS_FLAG_LZMA in mlv.h)
Will that be happening on the camera side in order to remove (or reduce) the CF card speed as a bottleneck? OR is it not really going to help CF card speed requirements, and it's just about storage?
Quote from: dmk on September 01, 2013, 07:20:15 AMit's just about storage?
this. its for archiving.
test footage can be compressed to 50% of the original size without any information loss.
do this on your computer and you save a lot of space for archiving.
you can also reduce bit depth to e.g. 10 bits and you get it down to 30%-40% of original size.
Oh... so it wouldn't help the CF Card speed limit to compress on the fly, i.e. time it takes to compress > time it takes to just write the data straight (or some other issue)?
Quote from: dmk on September 01, 2013, 08:41:42 PM
Oh... so it wouldn't help the CF Card speed limit to compress on the fly, i.e. time it takes to compress > time it takes to just write the data straight (or some other issue)?
sure, just tell me how i can configure the EDMAC to compress data with LZMA..
Quote from: g3gg0 on September 01, 2013, 08:53:25 PM
sure, just tell me how i can configure the EDMAC to compress data with LZMA..
Thanks for everything Team Magic Lantern has accomplished for the Independent Film Industry Community, it has helped me in very advance ways. Also I was wondering is there a light in the tunnel for 5D Mark II to receive Magic Lantern (RAW) Video format v2.0?
i am currently focusing on making MLV usable.
but i dont think that 5D2 is a big problem. it just has to be tried and bugfixed :)
Quote from: g3gg0 on September 01, 2013, 08:53:25 PM
sure, just tell me how i can configure the EDMAC to compress data with LZMA..
<joke>Tried calling Canon Customer Support but no luck</joke>
Hi,
I compiled all the latest stuff for the 7D from g3gg0/magic-lantern. I'm not able to get any results from mlv_dump other than a read out of the file structure of the .mlv.
I wonder what it is I'm doing wrong? maybe my usage is incorrect?
studio@studio-linux:~/Desktop/MLV_Dump0$ ./mlv_dump /home/studio/Desktop/MLV_Dump0/M02-2157.MLV MLV Dumper v1.0
-----------------
[i] Mode of operation:
- Input MLV file: '/home/studio/Desktop/MLV_Dump0/M02-2157.MLV'
- Verify file structure
[i] File /home/studio/Desktop/MLV_Dump0/M02-2157.MLV opened
[i] Processing...
[i] Reached end of chunk 0/1 after 1379 blocks
[i] Processed 603 video frames
[i] Done
So, I searched and found.
Quote from: RenatoPhoto on August 11, 2013, 04:17:56 PM
In linux type
./mlv_dump -h and get the usage RULES
I'll keep my post so people can see and learn from my own mistakes
When I try mlv2dng.exe I have no problems getting the .dng output.
BTW. I am having no issues recording continuous frames at 16x9 @ 1782. on a 1066x toshiba 16gb.
no pink frames. no small hacks enabled. works great. I compiled these modules for the 7D. file_man, ime_std, ime_rot, xml_cfg, raw_rec, pic_view. Raw_rec works well. Am checking the buffering strategies for speed gains. right now with my card I get upwards of 67 MB/s write speed whilst recording. the card benchmarks avg. 90MB/s.
I just ran mlv_dump successfully for linux. works great. I did a verbose dump for 7D. i put the code below.
studio@studio-linux:~/Desktop/MLV_Dump0$ ./mlv_dump -o M01-001.RAW -v -r -l0 /home/studio/Desktop/MLV_Dump0/M02-2157.MLV
MLV Dumper v1.0
-----------------
[i] Mode of operation:
- Input MLV file: '/home/studio/Desktop/MLV_Dump0/M02-2157.MLV'
- Verbose messages
- Convert to legacy RAW
- Output into 'M01-001.RAW'
[i] File /home/studio/Desktop/MLV_Dump0/M02-2157.MLV opened
[i] Processing...
File Header (MLVI)
Size : 0x00000034
Ver : v2.0
GUID : 11103347201719758039
FPS : 23.976000
File : 0 / 0
Frames Video: 603
Frames Audio: 0
Block: RAWI
Offset: 0x00000034
Size: 180
Time: 3.927000 ms
Res: 1728x972
raw_info:
api_version 0x00000001
height 1182
width 1992
pitch 3486
frame_size 0x003EDF84
bits_per_pixel 14
black_level 2053
white_level 15000
active_area.y1 26
active_area.x1 256
active_area.y2 1182
active_area.x2 1992
exposure_bias 0, 0
cfa_pattern 0x02010100
calibration_ill 1
Block: INFO
Offset: 0x000000e8
Size: 16
Time: 4.145000 ms
Block: RTCI
Offset: 0x000000f8
Size: 44
Time: 4.332000 ms
Date: 02.08.2013
Time: 21:57:35 (GMT+0)
Zone: ''
Day of week: 1
Day of year: 244
Daylight s.: 0
Block: EXPO
Offset: 0x00000124
Size: 40
Time: 5.672000 ms
ISO Mode: 0
ISO: 400
ISO Analog: 88
ISO DGain: 0 EV
Shutter: 20749 µs (1/48.20)
Block: LENS
Offset: 0x0000014c
Size: 96
Time: 5.744000 ms
Name: ''
Serial: ''
Focal Len: 50 mm
Focus Dist: 0 mm
Aperture: f/5.60
IS Mode: 0
AF Mode: 0
Lens ID: 0x00000000
Flags: 0x00000000
Block: IDNT
Offset: 0x000001ac
Size: 84
Time: 6.047000 ms
Camera Name: 'Failed to get properties.'
Camera Serial: ''
Camera Model: 0x00000000
Block: WBAL
Offset: 0x00000200
Size: 44
Time: 6.413000 ms
Mode: 9
Kelvin: 3600
Gain R: 495
Gain G: 1024
Gain B: 1024
Shift GM: 0
Shift BA: 0
Block: VIDF
Offset: 0x0000022c
Size: 2943424
Time: 74.161000 ms
Frame: #0
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x002cebec
Size: 1600
Block: NULL
Offset: 0x002cf22c
Size: 36
Block: RTCI
Offset: 0x002cf250
Size: 44
Time: 107.820000 ms
Date: 02.08.2013
Time: 21:57:35 (GMT+0)
Zone: ''
Day of week: 1
Day of year: 244
Daylight s.: 0
Block: EXPO
Offset: 0x002cf27c
Size: 40
Time: 109.232000 ms
ISO Mode: 0
ISO: 400
ISO Analog: 88
ISO DGain: 0 EV
Shutter: 20749 µs (1/48.20)
Block: LENS
Offset: 0x002cf2a4
Size: 96
Time: 109.297000 ms
Name: ''
Serial: ''
Focal Len: 50 mm
Focus Dist: 0 mm
Aperture: f/5.60
IS Mode: 0
AF Mode: 0
Lens ID: 0x00000000
Flags: 0x00000000
Block: WBAL
Offset: 0x002cf304
Size: 44
Time: 109.504000 ms
Mode: 9
Kelvin: 3600
Gain R: 495
Gain G: 1024
Gain B: 1024
Shift GM: 0
Shift BA: 0
Block: VIDF
Offset: 0x002cf330
Size: 2943164
Time: 116.090000 ms
Frame: #1
Crop: 256x120
Pan: 256x118
Space: 3804
Block: NULL
Offset: 0x0059dbec
Size: 1600
Block: VIDF
Offset: 0x0059e22c
Size: 2943424
Time: 157.921000 ms
Frame: #2
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x0086cbec
Size: 1600
Block: VIDF
Offset: 0x0086d22c
Size: 2943424
Time: 200.693000 ms
Frame: #3
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x00b3bbec
Size: 1600
Block: VIDF
Offset: 0x00b3c22c
Size: 2943424
Time: 241.556000 ms
Frame: #4
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x00e0abec
Size: 1600
Block: VIDF
Offset: 0x00e0b22c
Size: 2943424
Time: 282.823000 ms
Frame: #5
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x010d9bec
Size: 1600
Block: VIDF
Offset: 0x010da22c
Size: 2943424
Time: 324.598000 ms
Frame: #6
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x013a8bec
Size: 1600
Block: VIDF
Offset: 0x013a922c
Size: 2943424
Time: 366.565000 ms
Frame: #7
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x01677bec
Size: 1600
Block: VIDF
Offset: 0x0167822c
Size: 2943424
Time: 575.073000 ms
Frame: #12
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x01946bec
Size: 1600
Block: VIDF
Offset: 0x0194722c
Size: 2943424
Time: 616.474000 ms
Frame: #13
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x01c15bec
Size: 1600
Block: VIDF
Offset: 0x01c1622c
Size: 2943424
Time: 658.341000 ms
Frame: #14
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x01ee4bec
Size: 1600
Block: VIDF
Offset: 0x01ee522c
Size: 2943424
Time: 700.022000 ms
Frame: #15
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x021b3bec
Size: 1600
Block: VIDF
Offset: 0x021b422c
Size: 2943424
Time: 742.674000 ms
Frame: #16
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x02482bec
Size: 1600
Block: VIDF
Offset: 0x0248322c
Size: 2943424
Time: 783.859000 ms
Frame: #17
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x02751bec
Size: 1600
Block: VIDF
Offset: 0x0275222c
Size: 2943424
Time: 825.092000 ms
Frame: #18
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x02a20bec
Size: 1600
Block: VIDF
Offset: 0x02a2122c
Size: 2943424
Time: 867.115000 ms
Frame: #19
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x02cefbec
Size: 1600
Block: NULL
Offset: 0x02cf022c
Size: 36
Block: RTCI
Offset: 0x02cf0250
Size: 44
Time: 902.736000 ms
Date: 02.08.2013
Time: 21:57:36 (GMT+0)
Zone: ''
Day of week: 1
Day of year: 244
Daylight s.: 0
Block: EXPO
Offset: 0x02cf027c
Size: 40
Time: 904.250000 ms
ISO Mode: 0
ISO: 400
ISO Analog: 88
ISO DGain: 0 EV
Shutter: 20749 µs (1/48.20)
Block: LENS
Offset: 0x02cf02a4
Size: 96
Time: 904.330000 ms
Name: ''
Serial: ''
Focal Len: 50 mm
Focus Dist: 0 mm
Aperture: f/5.60
IS Mode: 0
AF Mode: 0
Lens ID: 0x00000000
Flags: 0x00000000
Block: WBAL
Offset: 0x02cf0304
Size: 44
Time: 904.589000 ms
Mode: 9
Kelvin: 3600
Gain R: 495
Gain G: 1024
Gain B: 1024
Shift GM: 0
Shift BA: 0
Block: VIDF
Offset: 0x02cf0330
Size: 2943164
Time: 908.885000 ms
Frame: #20
Crop: 256x120
Pan: 256x118
Space: 3804
Block: NULL
Offset: 0x02fbebec
Size: 1600
Block: VIDF
Offset: 0x02fbf22c
Size: 2943424
Time: 950.834000 ms
Frame: #21
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x0328dbec
Size: 1600
Block: VIDF
Offset: 0x0328e22c
Size: 2943424
Time: 992.293000 ms
Frame: #22
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x0355cbec
Size: 1600
Block: VIDF
Offset: 0x0355d22c
Size: 2943424
Time: 408.019000 ms
Frame: #8
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x0382bbec
Size: 1600
Block: VIDF
Offset: 0x0382c22c
Size: 2943424
Time: 1117.426000 ms
Frame: #25
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x03afabec
Size: 1600
Block: VIDF
Offset: 0x03afb22c
Size: 2943424
Time: 1158.761000 ms
Frame: #26
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x03dc9bec
Size: 1600
Block: VIDF
Offset: 0x03dca22c
Size: 2943424
Time: 1200.825000 ms
Frame: #27
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x04098bec
Size: 1600
Block: VIDF
Offset: 0x0409922c
Size: 2943424
Time: 1243.190000 ms
Frame: #28
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x04367bec
Size: 1600
Block: VIDF
Offset: 0x0436822c
Size: 2943424
Time: 1284.208000 ms
Frame: #29
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x04636bec
Size: 1600
Block: VIDF
Offset: 0x0463722c
Size: 2943424
Time: 1326.281000 ms
Frame: #30
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x04905bec
Size: 1600
Block: VIDF
Offset: 0x0490622c
Size: 2943424
Time: 1367.636000 ms
Frame: #31
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x04bd4bec
Size: 1600
Block: VIDF
Offset: 0x04bd522c
Size: 2943424
Time: 449.614000 ms
Frame: #9
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x04ea3bec
Size: 1600
Block: VIDF
Offset: 0x04ea422c
Size: 2943424
Time: 491.980000 ms
Frame: #10
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x05172bec
Size: 1600
Block: VIDF
Offset: 0x0517322c
Size: 2943424
Time: 533.598000 ms
Frame: #11
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x05441bec
Size: 1600
Block: VIDF
Offset: 0x0544222c
Size: 2943424
Time: 1409.272000 ms
Frame: #32
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x05710bec
Size: 1600
Block: VIDF
Offset: 0x0571122c
Size: 2943424
Time: 1450.895000 ms
Frame: #33
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x059dfbec
Size: 1600
Block: VIDF
Offset: 0x059e022c
Size: 2943424
Time: 1493.368000 ms
Frame: #34
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x05caebec
Size: 1600
Block: VIDF
Offset: 0x05caf22c
Size: 2943424
Time: 1617.806000 ms
Frame: #37
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x05f7dbec
Size: 1600
Block: NULL
Offset: 0x05f7e22c
Size: 36
Block: RTCI
Offset: 0x05f7e250
Size: 44
Time: 1652.599000 ms
Date: 02.08.2013
Time: 21:57:36 (GMT+0)
Zone: ''
Day of week: 1
Day of year: 244
Daylight s.: 0
Block: EXPO
Offset: 0x05f7e27c
Size: 40
Time: 1654.119000 ms
ISO Mode: 0
ISO: 400
ISO Analog: 88
ISO DGain: 0 EV
Shutter: 20749 µs (1/48.20)
Block: LENS
Offset: 0x05f7e2a4
Size: 96
Time: 1654.199000 ms
Name: ''
Serial: ''
Focal Len: 50 mm
Focus Dist: 0 mm
Aperture: f/5.60
IS Mode: 0
AF Mode: 0
Lens ID: 0x00000000
Flags: 0x00000000
Block: WBAL
Offset: 0x05f7e304
Size: 44
Time: 1654.463000 ms
Mode: 9
Kelvin: 3600
Gain R: 495
Gain G: 1024
Gain B: 1024
Shift GM: 0
Shift BA: 0
Block: VIDF
Offset: 0x05f7e330
Size: 2943164
Time: 1659.786000 ms
Frame: #38
Crop: 256x120
Pan: 256x118
Space: 3804
Block: NULL
Offset: 0x0624cbec
Size: 1600
Block: VIDF
Offset: 0x0624d22c
Size: 2943424
Time: 1701.822000 ms
Frame: #39
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x0651bbec
Size: 1600
~~
~~
~~
Block: NULL
Offset: 0x66299bec
Size: 1600
Block: VIDF
Offset: 0x6629a22c
Size: 2943424
Time: 24432.173000 ms
Frame: #584
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x66568bec
Size: 1600
Block: VIDF
Offset: 0x6656922c
Size: 2943424
Time: 24474.052000 ms
Frame: #585
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x66837bec
Size: 1600
Block: NULL
Offset: 0x6683822c
Size: 36
Block: RTCI
Offset: 0x66838250
Size: 44
Time: 24502.724000 ms
Date: 02.08.2013
Time: 21:57:59 (GMT+0)
Zone: ''
Day of week: 1
Day of year: 244
Daylight s.: 0
Block: EXPO
Offset: 0x6683827c
Size: 40
Time: 24504.143000 ms
ISO Mode: 0
ISO: 400
ISO Analog: 88
ISO DGain: 0 EV
Shutter: 20749 µs (1/48.20)
Block: LENS
Offset: 0x668382a4
Size: 96
Time: 24504.226000 ms
Name: ''
Serial: ''
Focal Len: 50 mm
Focus Dist: 0 mm
Aperture: f/5.60
IS Mode: 0
AF Mode: 0
Lens ID: 0x00000000
Flags: 0x00000000
Block: WBAL
Offset: 0x66838304
Size: 44
Time: 24504.488000 ms
Mode: 9
Kelvin: 3600
Gain R: 495
Gain G: 1024
Gain B: 1024
Shift GM: 0
Shift BA: 0
Block: VIDF
Offset: 0x66838330
Size: 2943164
Time: 24515.765000 ms
Frame: #586
Crop: 256x120
Pan: 256x118
Space: 3804
Block: NULL
Offset: 0x66b06bec
Size: 1600
Block: VIDF
Offset: 0x66b0722c
Size: 2943424
Time: 24557.608000 ms
Frame: #587
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x66dd5bec
Size: 1600
Block: VIDF
Offset: 0x66dd622c
Size: 2943424
Time: 24599.616000 ms
Frame: #588
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x670a4bec
Size: 1600
Block: VIDF
Offset: 0x670a522c
Size: 2943424
Time: 24641.389000 ms
Frame: #589
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x67373bec
Size: 1600
Block: VIDF
Offset: 0x6737422c
Size: 2943424
Time: 24682.607000 ms
Frame: #590
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x67642bec
Size: 1600
Block: VIDF
Offset: 0x6764322c
Size: 2943424
Time: 24724.385000 ms
Frame: #591
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x67911bec
Size: 1600
Block: VIDF
Offset: 0x6791222c
Size: 2943424
Time: 24891.196000 ms
Frame: #595
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x67be0bec
Size: 1600
Block: VIDF
Offset: 0x67be122c
Size: 2943424
Time: 24932.799000 ms
Frame: #596
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x67eafbec
Size: 1600
Block: VIDF
Offset: 0x67eb022c
Size: 2943424
Time: 24974.609000 ms
Frame: #597
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x6817ebec
Size: 1600
Block: VIDF
Offset: 0x6817f22c
Size: 2943424
Time: 24765.842000 ms
Frame: #592
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x6844dbec
Size: 1600
Block: VIDF
Offset: 0x6844e22c
Size: 2943424
Time: 24807.931000 ms
Frame: #593
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x6871cbec
Size: 1600
Block: VIDF
Offset: 0x6871d22c
Size: 2943424
Time: 24849.782000 ms
Frame: #594
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x689ebbec
Size: 1600
Block: VIDF
Offset: 0x689ec22c
Size: 2943424
Time: 25016.513000 ms
Frame: #598
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x68cbabec
Size: 1600
Block: VIDF
Offset: 0x68cbb22c
Size: 2943424
Time: 25057.653000 ms
Frame: #599
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x68f89bec
Size: 1600
Block: VIDF
Offset: 0x68f8a22c
Size: 2943424
Time: 25099.865000 ms
Frame: #600
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x69258bec
Size: 1600
Block: VIDF
Offset: 0x6925922c
Size: 2943424
Time: 25141.592000 ms
Frame: #601
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x69527bec
Size: 1600
Block: VIDF
Offset: 0x6952822c
Size: 2943424
Time: 25182.815000 ms
Frame: #602
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x697f6bec
Size: 1600
Block: VIDF
Offset: 0x697f722c
Size: 2943424
Time: 21304.286000 ms
Frame: #509
Crop: 256x120
Pan: 256x118
Space: 4064
Block: NULL
Offset: 0x69ac5bec
Size: 1600
Block: NULL
Offset: 0x69ac622c
Size: 36
Block: RTCI
Offset: 0x69ac6250
Size: 44
Time: 19186.942000 ms
Date: 02.08.2013
Time: 21:57:54 (GMT+0)
Zone: ''
Day of week: 1
Day of year: 244
Daylight s.: 0
Block: EXPO
Offset: 0x69ac627c
Size: 40
Time: 19188.471000 ms
ISO Mode: 0
ISO: 400
ISO Analog: 88
ISO DGain: 0 EV
Shutter: 20749 µs (1/48.20)
Block: LENS
Offset: 0x69ac62a4
Size: 96
Time: 19188.621000 ms
Name: ''
Serial: ''
Focal Len: 50 mm
Focus Dist: 0 mm
Aperture: f/5.60
IS Mode: 0
AF Mode: 0
Lens ID: 0x00000000
Flags: 0x00000000
Block: WBAL
Offset: 0x69ac6304
Size: 44
Time: 19188.972000 ms
Mode: 9
Kelvin: 3600
Gain R: 495
Gain G: 1024
Gain B: 1024
Shift GM: 0
Shift BA: 0
Block: VIDF
Offset: 0x69ac6330
Size: 2943164
Time: 19219.000000 ms
Frame: #459
Crop: 256x120
Pan: 256x118
Space: 3804
Block: NULL
Offset: 0x69d94bec
Size: 1600
[i] Reached end of chunk 0/1 after 1379 blocks
[i] Processed 603 video frames
[i] Done
I did something to my 50D, now I have these .MLV files and am lost in a sea of forum posts. I'm on Windows. Can someone point me to where I can get a program/script, whatever, to get me to RAW so I can RAWanizer it :) I've read just a thing exists, but have also read many problems. So don't want to download something old and gnarly. Thanks! I just hope I can find my way back to this forum thread! :)
Quote from: maxotics on September 03, 2013, 12:57:03 AM
I did something to my 50D, now I have these .MLV files and am lost in a sea of forum posts. I'm on Windows. Can someone point me to where I can get a program/script, whatever, to get me to RAW so I can RAWanizer it :) I've read just a thing exists, but have also read many problems. So don't want to download something old and gnarly. Thanks! I just hope I can find my way back to this forum thread! :)
check out mlv2dng by gnarr.
https://dl.dropboxusercontent.com/u/3181048/mlv2dng.zip
the program is command line executable on windows something like.
c:> folder/where/mlv2dng.exe c:> file/where/Mxx-xxx1.MLV > c:> folder/where/you/want/xxxxxx.DNG
I make my folder before and drag and drop it onto the DOS prompt window. works good.
mlv_dump, is available from somewhere around here it's linux, but it's got more options (as far as I can tell)
search mlv_dump. or compile it yourself from g3gg0's repository @ bitbucket.org.
Thanks John. I sent the dev a private note. Don't see why a C++ person (scripting is all this old dog is useful for anymore) couldn't make a quick Windows command utility to parse MRV into RAW. Anyway. offered to help. In the meantime, I put my 50D back to one of Andy's stable builds.
Quote from: maxotics on September 03, 2013, 05:08:11 AM
Thanks John. I sent the dev a private note. Don't see why a C++ person (scripting is all this old dog is useful for anymore) couldn't make a quick Windows command utility to parse MRV into RAW. Anyway. offered to help. In the meantime, I put my 50D back to one of Andy's stable builds.
Hey man, mlvdng.exe goes all the way to .dng no need for raw inter. Also, i'm pretty sure g3gg0 put up a mlv_dump.exe I may be wrong. but i would check developer g3gg0 threads.
for 50D I made MLV_REC and raw_rec so the choice is yours. raw_rec is still faster, except if you have a 5DIII.
Thanks for the 50D development! Looking to test out MLV... Wonder how the 50D will handle the extra info??? Thanks for the extra effort.
Quote from: maxotics on September 03, 2013, 05:08:11 AM
Thanks John. I sent the dev a private note. Don't see why a C++ person (scripting is all this old dog is useful for anymore) couldn't make a quick Windows command utility to parse MRV into RAW. Anyway. offered to help. In the meantime, I put my 50D back to one of Andy's stable builds.
[ x ] you didn't read my first post, where it says: post processing is still done with 'raw2dng' after converting the .mlv into the legacy .raw format using mlv_dump.
Did you read about Scratch? they are going to drop a Raw Player, the specs say that also dslr raw material will be readeable.
http://www.awn.com/news/business/assimilate-launches-scratch-play
@dude: the link (in the Assimilate press release) for the Scratch player has been pulled by Assimilate, apparently, but I found the new location:
http://www.assimilateinc.com/scratch-play-build (http://www.assimilateinc.com/scratch-play-build)
Cheers!
Nice, thanks!
edit-
installed it, so far it does not load the raw files
Quote from: g3gg0 on September 03, 2013, 07:46:24 AM
[ x ] you didn't read my first post, where it says: post processing is still done with 'raw2dng' after converting the .mlv into the legacy .raw format using mlv_dump.
I sent you a private message and you're now publicly saying I don't take the time to read your work? That's not very nice. I've read everything you've wrote. I can't find a mlv_dump.exe for Windows. All if you have to do is say it doesn't exist and you don't know when it will, OR, point me to it. Really, g3gg0! :)
Quote from: maxotics on September 03, 2013, 02:21:32 PM
I sent you a private message and you're now publicly saying I don't take the time to read your work? That's not very nice. I've read everything you've wrote. I can't find a mlv_dump.exe for Windows. All if you have to do is say it doesn't exist and you don't know when it will, OR, point me to it. Really, g3gg0! :)
sorry, i read forum and answer there before i check any PMs.
look in the main post and download the pre-built version.
> if you want to test recording, grab my sources from https://bitbucket.org/g3gg0/magic-lantern or use the pre-built ones. (tested on 5D3)
as i update the link when i make new versions, please use the link from the main post.
there is raw2dng.exe, mlv_dump.exe, raw_rec.mo and a few other modules in that zip.
just open the first page, check my post and search for "pre-built". this is a link to the latest version.....
Quote from: maxotics on September 03, 2013, 02:21:32 PM
I sent you a private message and you're now publicly saying I don't take the time to read your work? That's not very nice. I've read everything you've wrote. I can't find a mlv_dump.exe for Windows. All if you have to do is say it doesn't exist and you don't know when it will, OR, point me to it. Really, g3gg0! :)
Keep your personal stuff off this board!
If you read this thread you will find this on reply 218:
Quote from: g3gg0 on August 11, 2013, 12:58:06 AM
here (http://upload.g3gg0.de/pub_files/fb196456f6a6389763cab895fb6067f8/mlv_dump.exe) is a windows build of mlv_dump
did not test it for 4GiB problems. can someone else try?
and a bit lower you will find this:
Quote from: RenatoPhoto on August 11, 2013, 04:01:16 PM
I just tried the mlv_dump.exe -o mlv.raw -r m10-2205.mlv
and .... made the mlv.raw file
and ... drag and drop on raw2dng.exe and Yes!!!!!!!
Got dng files!
So try it and see. If it is not compatible with current build then wait until dev makes it available.
@g3gg0!
I think the confusion is that you assume someone is going to load your sources and naturally see the utilities, which was natural when you started your effort.
I ended up with source that creates .MLV, which I didn't realize (and still don't know exactly how that happened). You might want to edit your first post to make it clear that if someone ends up with MLV files on their camera, that they can download your zip file and get the exe. Many people like me probably use RAWanizer straight to Cineform to check what they've done, and save on time/space with the whole DNG/TIFF extraction stuff.
I may have missed things, as RenatoPhoto pointed out, but honestly tried very hard. Again, thanks for all your efforts!
I hate sounding like a broken record... buuut :P
Anyone compiling some tests for 5D2 now that 50D seems to be getting some compiles too :)
Quote from: oddname on September 03, 2013, 04:39:19 PM
I hate sounding like a broken record... buuut :P
Anyone compiling some tests for 5D2 now that 50D seems to be getting some compiles too :)
Try the 50D one https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/downloads/mlv_rec.mo should work in the 5D2 (at least for testing).
That scratch play looks interesting but ads in the free version? Come on! VLC/MPC doesn't put ads in. Quite insulting to have spyware in your media player.
@dude: If you convert a .raw file (using raw2cdng.exe rather than raw2dng.exe) to CinemaDNG, you can point Scratch to the first .dng file and it will open and play the clip. Pretty cool. Colors are a bit off but Scratch is apparently a huge application, even the player portion, and there's probably a way to insert a LUT somewhere so that the clip will look reasonably like what you shot. Just FYI, haven't played with it for very long. Cheers!
P.S. Not sure anymore where I got raw2cdng.exe, it was on the ML forum somewhere, you should be able to Google it.
Quote from: maxotics on September 03, 2013, 03:46:27 PM
@g3gg0!
I think the confusion is that you assume someone is going to load your sources and naturally see the utilities, which was natural when you started your effort.
I ended up with source that creates .MLV, which I didn't realize (and still don't know exactly how that happened). You might want to edit your first post to make it clear that if someone ends up with MLV files on their camera, that they can download your zip file and get the exe. Many people like me probably use RAWanizer straight to Cineform to check what they've done, and save on time/space with the whole DNG/TIFF extraction stuff.
well, thats the reason why i
pointed out in the main post that one has to
convert it to legacy .raw before he can use it in his normal workflow.
i also provided the binaries so one can run the tools without the hassle of cross-compiling.
as this is
no release for the masses, i define my assumption to be perfectly fine.
it is for developers, for testers, BUT NOT for productional use yet.
as the board "General Development Discussion" implies, this is yet in development and end users will get a nifty UI, workflow, guide and whatever as soon it is in "New Releases" forum.
i understand that this is not the allround-happy-packet ready for everyone to use which you seem to have expected, but it was never meant to be that.
this will follow as soon this video format and the code is going to be merged into main tree.
@g3gg0
Sorry, I made it sound like it was your fault, and that's not what I mean. I meant to say that you might not realize that your code is now seeing its way onto more and more cameras (like my 50d) and that you may want to edit the first page a bit, for people like me who don't know a thing about MLV but now have it on their camera. Again, I'm sorry if I implied that you had done something wrong. I realize it was all just a big confusion. I AM all round happy packet! Please believe me. And thank you again.
I can already see how MLV will be a big improvement with the EOS-M. Right now, I have to pick which mode to clean up the files, normal or crop mode. With your work, future software can figure that out. Again, if there is anything I can do to help you with MLV, let me know. So again, please accept my apologies and thanks!
I'm going to go out on a limb and say that MLV will be better for SD cameras in general... they already only go ~40mb so +/- 1MB won't make a difference. On 50D I struggle with 1920x1038 and 1080P where before I had idle time.
Just for fun, the 50D module allmost works on the 5d2, can setup nicely etc, but recording just locks it up.
But hey, fun to see the options etc :D
Sure you're not using hacked modes that lock the LV?
Very. Menu looks screwed up too after. Completely locked up, with brief unlocked.
Then it just stops automatically. And menu keeps saying Stopping... on the raw-post.
Biggest MLV so far is around 3meg, so I guess its locked up then.
But I didnt expect it to work 100% since its for 50D :)
Just a test
5DII missing/broken reslock? I don't have one to test but if hacked is off it shouldn't be freezing.
Tried from a scratch made card, only module loaded = mlv.
When not locked I get a datarate of 2-3MB/s so something isnt working properly for the 5D2 =/
But never expected it to work, might be made with 50D-specifics in mind?
The handful of 50D specifics only run on 50D. I'll see what happens on SD cameras.
Here's that 50D MLV file stuff I shot the other day. Thanks again g3gg0! There was some broken frames in the beginning of some of the clips, but it could have been anything.
you see, all is there ;)
Quote from: maxotics on September 04, 2013, 01:13:45 AM
Here's that 50D MLV file stuff I shot the other day. Thanks again g3gg0! There was some broken frames in the beginning of some of the clips, but it could have been anything.
Was this taken using the slightly larger camera good for all those suffering from some sort of mid life crisis?
Played some more with the 50D-module and still no go. Tried every setting, different bufferings methods, hacks on/off. All with the same results.
So pretty sure the 50D-module has some code that just doesnt wanna play with the 5d2 =)
But it was really nice to see the menu and play around (not being sarcastic, I am uber-positive)
Quote from: ItsMeLenny on September 04, 2013, 08:27:46 AM
Was this taken using the slightly larger camera good for all those suffering from some sort of mid life crisis?
It's even worse. Her friend Christine, one to the right in the group shot, is an animation major at MassArt. Emma said she saved up all her money to buy a T3i this summer. I thought, 'well, she'll appreciate what I'm doing here.' I tried to explain ML RAW but ended up as a 52-year-old man humiliated by 21-year-old eye-rolling. As you can see, however, that doesn't STOP ME from shooting anything ;) It's a lonely world, ML RAW. In the one camp you have people who don't care, in the other, those who can't face the fact that in-camera H.264, for anyone serious about film/video, is dead..oops, hope I haven't offended anyone here!
The video version doesn't do justice what what was shot. I want to say again, that in RAWanizer, adding the args -d4 (higher sharpening) and -q5 (higher quality) makes for some very nice footage, good enough for me.
I went back to one of Andy's builds, so am shooting .RAW again. However, I would like to add .MLV to my processing scripts, maybe help out a bit, but I don't want to get MLV back by doing what I did before, copying modules into the camera willy-nilly. Is there a stable build for the 50D that will output MLV? Or can someone let me know when there is one.
I still can't make sense of which forks I should follow.
You can copy both modules and just run raw_rec and leave mlv_rec disabled. Its working and non freezing on 6D (after hard path removal) and other sd cameras but I dunno if this breaks spanning for 5DIII. Slower on SD cameras too... 39.x MB/s with raw_rec... only 36-37 MB/s on MLV rec.
1% any news regarding the odd happenings with the 50D module? :)
@1%, I'd be happy to load new modules, test logs, whatever to devs. Is there a tutorial, or page, where it explains how to load these modules, what to look out for. When do I need to re-run the EOScard utility? Can I just replace the autoexec.bin file. Can you tell me who is actively developing for the 50D, or any camera. Should I, can I, use the same code base for my EOS-M and 50D. While you're here. Does God exist? Thanks!
autoexec + sym go together... one in root the other in the ML/modules folder.
After that you can load any modules with the same API (maybe we've had 3 changes total).
Not sure whats happening on 5DII, only bug I noticed is sound recording is 2 when the defaults are set so I have to fix that. I don't exactly have one to test and 50D works ok.
50D/EOSM/6D are the same base, it picks what to compile based on the model. 600D is its own but that may change.
This does MLV on SD cameras and probably CF too.
mlv_rec.mo (47 KB)
https://mega.co.nz/#!Nlp1FahY!fyd5Bwlw_uX9ehF21P_Cxk7lX9RbOrrltHmBlokMwzk
1% Thanks, oaky, so I have Andy's latest stable build. I looked, and see the sym file. So if I want to save as .MLV, I assume I need that module in my camera's ML folder?
o. Who is working on that?
o. I take it, from what you're saying, that if I put that in the folder it will become a load option on the camera? I don't need to to anything more.
o. I see builds for you. What aspect do your builds specialize in? Which fork of your should I follow?
o. I would think you would want people to test your latest-greatest creation. But I don't see an easy to follow place for that. If it exists, can you please tell me how I should approach trying out your new stuff, say.
Thanks!
TL2 is the 600D repo, everything else is in TL6D.
To follow it look at commits, you can set up a VM and compile for yourself. There is a guide on here. Andy is building 50D off a clone of my repo.
For modules, yea pretty much what you said, you add it to the dir and pick what you want loading then reboot.
Quote from: a1ex on August 29, 2013, 11:39:53 AM
Yes, or better.
I'll actually move the post deflicker to a PC-based tool again (it will operate on raw photo sequences, no matter how you got them - even from Nikon).
Hey a1ex. Just following up - is there a thread that we can follow the progress of this update? Seems great!
Quote from: sletts02 on September 05, 2013, 12:26:09 PM
Hey a1ex. Just following up - is there a thread that we can follow the progress of this update? Seems great!
As they've said before when the file format is complete and ready for the masses the tools will be also. Until then be happy with what they put out for testing.
Hey guys, Quick question.
How does one get the new magic lantern video format (.MLV) working?
Is it some plug in you through into the modules folder or is it cooked right into the nightly release or.. what? Lol.
Also, they say it adds 20mbps to the multi card spanning. Now if you have the sandisk 95mbps sd cards.. will that make it more than 20 or do they have it limited to 20?
Using a mark 3.
Quote from: Vegeta on September 06, 2013, 04:25:05 AM
Hey guys, Quick question.
How does one get the new magic lantern video format (.MLV) working?
Is it some plug in you through into the modules folder or is it cooked right into the nightly release or.. what? Lol.
Not available through nightly release. Read the first post of this thread!
Quote from: 1% on September 04, 2013, 09:05:56 PM
autoexec + sym go together... one in root the other in the ML/modules folder.
After that you can load any modules with the same API (maybe we've had 3 changes total).
Not sure whats happening on 5DII, only bug I noticed is sound recording is 2 when the defaults are set so I have to fix that. I don't exactly have one to test and 50D works ok.
50D/EOSM/6D are the same base, it picks what to compile based on the model. 600D is its own but that may change.
This does MLV on SD cameras and probably CF too.
mlv_rec.mo (47 KB)
https://mega.co.nz/#!Nlp1FahY!fyd5Bwlw_uX9ehF21P_Cxk7lX9RbOrrltHmBlokMwzk
Added the mlv_rec.mo to the 5D Mark 3 modules directory. Get the following error: "tcc: error: undefined symbol 'message_queue_count' [E] failed to link modules"
You need a build with msg_qcount and prop body ID. I think 5DIII has it in main, I added those for all in my repo. BTW, the spanning may be broken since the path isn't hardcoded to a:
Quote from: Stedda on September 05, 2013, 01:34:04 PM
As they've said before when the file format is complete and ready for the masses the tools will be also. Until then be happy with what they put out for testing.
Sorry, my misunderstanding. I thought the deflicker tool was completely separate to format 2.0 and there may have been a separate thread that I had missed :(
@RenatoPhoto
Can you explain me the logic of the Raw file value number while recording?
Sometimes it goes like this in a very logical order:
M20-1240.raw then M20-1241.raw, M20-1242.raw
and sometimes it goes like this :
M20-1250.raw and the shot after can be M21-1263.raw and not M20-1251.raw as expected. Did I miss something?
I use MovieSlate to synchronized the soundtrack and this could be helpful to understand the way it works to organize my shooting.
Thanks again to ML team for your amazing work!!!
Look at the clock ;)
1263 must be a bug, can you show a screenshot?
did you check the camera clock?
12:63 is quite a weird time :)
(or maybe it was 12:59 and you recorded four video in that minute?)
Correct; though very unlikely in normal usage.
hehe ok didnt see you already replied :)
Thanks a1ex and g3gg0 for your quick answer! I understand now. It was so obvious that I didn't get it...
As I didn't know it was the date, I put "a kind of" example and the only reason it's weird is because I thought I could have that kind of number and I didn't check the exact number... Just a little bit slow to understand simple things! Sorry but let me thank you again and again for your great job!
I reported the weird time filename in the 50D thread, I went from a 59 to a 60 with two filenames.
Quote from: g3gg0 on September 07, 2013, 01:57:50 PM
(or maybe it was 12:59 and you recorded four video in that minute?)
i got a ideas : why not use a module for copying immediatly after recording in spanning mode the .M00 file of the SD card to the CF card ?
that is possible ?
Quote from: dadinio13 on September 09, 2013, 06:27:05 PM
i got a ideas : why not use a module for copying immediatly after recording in spanning mode the .M00 file of the SD card to the CF card ?
that is possible ?
well, i had the idea of a
separate module that checks if the camera is idle for a while (e.g. 10 seconds) and then starts
copying all files from CF card to the (WiFi-)SD card for transferring them to the computer automatically.
as soon the files are being copied over to the computer, they are deleted from the cards.
this of course can be combined with copying MLV files to CF card when the camera is idle.
didnt think about that yet, thanks.
the criteria for the right moment or enough space are a playfield for not needed code. my opinion. its just one less step on the main computer (or in the file manager, if you're sure time/space is right for moving files)
regards chmee
guys, is file spanning present in the newest 5d3 nightly builds?
no, as this module is not in nightly build....
Have you sorted out how to merge format 2.0 to unified? I see the pull request but seems to be stuck, any further discussions needed? I was hoping we could get a bit of speed boost from some of Alexs work to the mlv files. That would be awesome.
Looking forward to this feature. Are there already estimates what write speeds will be possible with current card tech and what resolutions at continuous 60 fps ?
Quote from: RenatoPhoto on September 10, 2013, 04:38:53 AM
Have you sorted out how to merge format 2.0 to unified? I see the pull request but seems to be stuck, any further discussions needed? I was hoping we could get a bit of speed boost from some of Alexs work to the mlv files. That would be awesome.
the pull request is still pending and i fixed some discussion points already.
we are still not 100% sure about properties, as reading admin props (serial number) is something we never tested thoroughly.
Hi, please help:
I have downloaded the latest 5DM3 ML build from Alex's thread:
http://www.magiclantern.fm/forum/index.php?topic=7139.0
However, as soon as I load the mlv_rec.mo to the module directory, I get loading error (even with the raw_rec.mo on 'off' mode).
How can I manage to run mlv_rec.mo as an alternative to raw_rec.mo when I want?
Do you already try to format the card and install everything again? Some times for some errors it works.
QuoteI get loading error (even with the raw_rec.mo on 'off' mode).
What error? Missing symbols?
Quote from: Canon eos m on September 10, 2013, 05:11:53 PM
Hi, please help:
I have downloaded the latest 5DM3 ML build from Alex's thread:
http://www.magiclantern.fm/forum/index.php?topic=7139.0
However, as soon as I load the mlv_rec.mo to the module directory, I get loading error (even with the raw_rec.mo on 'off' mode).
How can I manage to run mlv_rec.mo as an alternative to raw_rec.mo when I want?
As far as i know Alex build does not have the mlv function yet since g3ggo work has not been merged.
You have to download or compile yourself the entire ML from this thread. g3ggo has provided raw_rec.zip for 5d3
My questtion is...
Is it only a container format, or is it more than that?
The thing is. DSLR like 60D will not be able to record RAW nearly FullHD
and more than ca.10 seconds.
But it would be so great to not record in h264.
Something like "uncompressed png, tiff, photojpg".
Not RAW, but also not h.264.
Question is... would it be possible to reencode in camera in
a new file fomat which is not exactly raw but better than h.264.
Something like 8-bit RAW instead of 14-bit RAW.
Simple to minimize the files size, so that Cameras like D60 could
record almost FullHD in "better than h.264" quality.
Quote from: Flocksock on September 11, 2013, 04:50:23 PM
Is it only a container format, or is it more than that?
MLV is a container format for raw video (primary and obvious use case) and also other formats like MJPEG, H264 etc.
other types than raw just weren't specified yet
Quote from: Flocksock on September 11, 2013, 04:50:23 PM
Question is... would it be possible to reencode in camera in
a new file fomat which is not exactly raw but better than h.264.
Something like 8-bit RAW instead of 14-bit RAW.
Simple to minimize the files size, so that Cameras like D60 could
record almost FullHD in "better than h.264" quality.
This has been discussed so many times, and answered. Please do a search before posting.
This is all great stuff (see I don't always whine ;).
Since some of the motivation for the raw container is performance, are there any back of the envelope numbers on perf / resolutions for 5dmk3raw? Also while this is wandering into dual iso territory - will the actual bit depth of data be the same?
I searched in the forum - but sometimes you need to know what to search for first ;)
And as someone hinted - will there be raw container support for sync-sound?
Thx
Quote from: mkrjf on September 13, 2013, 06:55:27 PM
This is all great stuff (see I don't always whine ;).
Since some of the motivation for the raw container is performance, are there any back of the envelope numbers on perf / resolutions for 5dmk3raw? A
Same thread here: http://www.magiclantern.fm/forum/index.php?topic=7122.msg66931#msg66931
Pretty much same conclusions I reached... on SD/CF MLV was 1-2MB/s slower... only 5DIII can use the spanning
Quote from: mkrjf on September 13, 2013, 06:55:27 PM
And as someone hinted - will there be raw container support for sync-sound?
the container is designed to get perfect sync between audio and video.
there is just no implementation for audio yet.
anyone wants to implement?
about performance:
yeah, the new format sacrifices a few MiB/s, but i think there is still some room for optimization.
but the numbers are not always that important - with the new format a lot of other advantages come up - metadata on nearly per-frame basis :)
Quote from: g3gg0 on September 13, 2013, 09:55:03 PM
the container is designed to get perfect sync between audio and video.
there is just no implementation for audio yet.
anyone wants to implement?
I love RAW but
all my videos are muted! I will not carry additional recording audio because it is too complex for wildlife videos. I really would love to have this implemented or to bring back the out of synch video-audio option. There is a lot of people like me that could use it!
Not all video recording is done in a studio!
Quote from: RenatoPhoto on September 14, 2013, 04:25:40 PM
I love RAW but all my videos are muted! I will not carry additional recording audio because it is too complex for wildlife videos. I really would love to have this implemented or to bring back the out of synch video-audio option. There is a lot of people like me that could use it! Not all video recording is done in a studio!
Then you'll have to wait. RAW video is after all still experimental.
An audio thread pertaining to the current sub-discussion: http://www.magiclantern.fm/forum/index.php?topic=7890.msg70058#msg70058
It would should also work for the H4N...
Quoteyeah, the new format sacrifices a few MiB/s, but i think there is still some room for optimization.
There definitely is... write test LV open in play mode gives 91.8MB/s... something is using the CPU/taking edmac time.
Im sure there was a build for 600d that had the mlv module with it... cant seem too find it anymore :(
So how is this module coming along? Feels like it is some time since one read news :)
Still waiting eagerly for the 5d2 version :P
@ g3g00: When I compile the modules in https://bitbucket.org/g3gg0/magic-lantern/commits/all I get the same error in all modules:
Any help would be appreciated.
magiclantern@magiclantern-VirtualBox:~/G3GGo/magic-lantern/modules/autoexpo$ make clean
[ RM ] autoexpo.o autoexpo.mo autoexpo.sym autoexpo.dep module_strings.h *.o *.d *.bmp *.dep *.sym
magiclantern@magiclantern-VirtualBox:~/G3GGo/magic-lantern/modules/autoexpo$ make
[ README ] module_strings.h
[ DIFF ] module_strings.h
/bin/sh: 1: ../module_hginfo_create.sh: Permission denied
make: *** [module_strings.h] Error 126
Make module_hginfo_create.sh executable
or change Makefile.modules like this:
diff -r e428f411e2ca modules/Makefile.modules
--- a/modules/Makefile.modules Sun Sep 15 10:55:26 2013 +0200
+++ b/modules/Makefile.modules Sun Sep 15 14:50:52 2013 +0200
@@ -26,7 +26,7 @@
module_strings.h: FORCE README.rst
$(call build,README, $(PYTHON) ../readme2modulestrings.py > $@)
- $(call build,DIFF, ../module_hginfo_create.sh >> $@)
+ $(call build,DIFF, sh ../module_hginfo_create.sh >> $@)
README.rst:
$(call build,README,echo -e "$(MODULE_NAME)\n========\n\nPlease write a one-page description for your module here.\n\nYou may also add some metadata:\n\n:Author: <your name here>\n:License: <your license here>\n:Website: <your website here>\n\n" > $@)
Edgar
Quote from: escho on September 16, 2013, 05:20:26 PM
Make module_hginfo_create.sh executable
can you provide a patch for this?
on windows i cant do that :)
Oh, Windows, I see! I didn´t realize, that there exist a operating system called Windows, too. :)
My solution is for compiling from a Linux-machine.
I´m not sure, Windows can understand bash-scripts (foo.sh) natively.
Using Windows I cannot help, because I only use Linux
Edgar
MSYS is really nice... like power shell + cmd + bash all together... it let me use wildcards in cr2hdr when normal cmd wouldn't. Would probably execute the .sh scripts.
Been out of a loop for a bit here... wondering where I can find the newest compiled version.
Things in this forum are getting a bit hard to follow...
Cant use mlv_rec.mo on 6D. Message "OldAPI". I dont known why... Using "DualIso-FullPack" at 2013-08-11. What am doing wrong? What do I need to adjust?
in the main post in this thread thread is a _very_ simple and hackish MLV viewer to check your footage
it will read uncompressed MLV files and display the frames with just a few frames per second.
this tool was programmed in C# on windows, but it uses nothing windows-specific, so it should run on any OS using mono.
please remind:
- this tool is just a PROOF OF CONCEPT
- it is not meant as productional tool
- i used it to check what is necessary to decode and view MLV files, its just my playground
- it has bugs!
- it will most likely not be continued
- i shared it as a last resort tool in case you need something like that
notes:
- shows the video in half resolution to simplify debayering
- ramps exposure up/down if there is under/overexposure (so it may not be accurate enough for some of you)
- has no white balance algorithm
- just tested on 5D3, other cameras have different bayer patterns - didnt check them yet
- it uses the coefficients from the raw info block, so color weighting should be correct
- the scene is scaled to TV black and white levels (16..235) for a better looking playback
thanks to alex for giving me a lot of hints on how to process raw data efficiently
(http://upload.g3gg0.de/pub_files/e76d16212357343b54eeac44f2e96002/MLV_Viewer.png)
Sweet :D THANKS :D
NICE!
That is awesome, allways wanted to be able to preview files before extracting etc =)
Awesome work!
Quote from: g3gg0 on September 21, 2013, 02:43:29 AM
here (http://upload.g3gg0.de/pub_files/d9fe3b7542769792920dfae6bd82fad5/mlv_view_sharp.exe) is a _very_ simple and hackish MLV viewer to check your footage
it will read uncompressed MLV files and display the frames with just a few frames per second.
this tool was programmed in C# on windows, but it uses nothing windows-specific, so it should run on any OS using mono.
Thank you g3g00, and Alex for the help!
I tried it with mlv files from 5D3 and 7D and works very well at all resolutions, cropped video, and 60 fps. Also payed spanned files!
In windows 7 64 bits I open the program and the drag the file, after a little delay the player functions properly.
I hope your pull request is merged to unified soon!
Thanks for your great work and support. ;D ;D
thanks for the feedback.
the initial delay is when the tool scans all MLV chunks and creates an index.
so it knows the order of the blocks to process.
update:
- fixed some crashing situations e.g. when pressing alt (downside: inefficient code)
- use color matrix from the raw info instead of hardcoded, so the colors should look better
- scaled colors to TV range (16..235) for a nicer look
Wow what a great tool !!!! VERY useful..
Just cleaned-up/deleted all my small test MLV videos :)
Does one dare to ask if there is a module that works on the 5D2 yet? =)
i dont know what is wrong on 5D2. there were just reports of "hanging" module.
can you explain the problem in detail?
any new pre-builts i can play with=) ?
I think with the latest commits I can only use the bigger modules on DigicV.
i updated the MLV player again - see main post.
now it supports resizing and different debayering algorithms.
parts of it are faster now, others are slower.
in the end its about the same speed as the version before ;)
Quote from: g3gg0 on September 21, 2013, 10:35:09 PM
i dont know what is wrong on 5D2. there were just reports of "hanging" module.
can you explain the problem in detail?
I can try again, can you point me to the latest build of the module that should work?
(I tried one before, and I was the one saying it froze, gave me speeds of 6-7MB/s instead of 60-70 when it works)
Quote from: g3gg0 on September 22, 2013, 01:22:33 AM
now it supports resizing and different debayering algorithms.
Where did you get the debayer code? Did you write it yourself or are you using code from say, dcraw?
Wrote it on my own. It isn't much code. Just simple math
I'm having really nice success with the latest updates found. Just have an issue with indicators roling over at 45GB and freezing on card full.
When will the format be available in the nightly builds? :) If it will.
I've also made a simple player for both MLV and RAW file sets. It uses OpenGL shaders for the demosaicing to keep CPU usage lower. It's written in python, so it works at least on Linux and Mac (command line only though). I haven't tried on Windows but it might be possible to make it work there also with little or no changes.
Link and info about the tool can be found in my blog post here: http://thndl.com/?20
Quote from: baldand on September 24, 2013, 08:40:58 AM
I've also made a simple player for both MLV and RAW file sets. It uses OpenGL shaders for the demosaicing to keep CPU usage lower. It's written in python, so it works at least on Linux and Mac (command line only though). I haven't tried on Windows but it might be possible to make it work there also with little or no changes.
Link and info about the tool can be found in my blog post here: http://thndl.com/?20
that's great news! I do look forward to it being available for windows.
Quote from: ted ramasola on September 24, 2013, 08:46:01 AM
that's great news! I do look forward to it being available for windows.
It turns out it also works fine on Windows (at least Win7) as-is using a python distro like WinPython 2.7 (See https://code.google.com/p/winpython/ )
If you install that and use it to compile the included bitunpack module it will decode a bit faster.
E.g. from the source directory something like:
c:/WinPython/python/python setup.py build_ext -c mingw32 --inplace
(though I had to remove -mno_cygwin calls from python/lib/distutils/cygwinccompiler.py first)
Then copy bitunpack.pyd from the build subsubdirectory
hi,
mlv really sounds interesting and i think we all hope that audio will make it to the final verison,
can someone post a link to a actual build with mlv for 5d mark3... have no skills to build my own one. thx.
thx.
Hi, I am following from far what's happening with the MLV format and I think it's pretty exiting, so I was wondering, do you think or hope that in the future, softwares like Davinci Resolve, After effects, Premiere, and others will read that format ? Will they debayer proxy directly!?
If that happens, it would be crazy for Canon, they'd have to hire you guys! (to make you stop that nonsense!) or sue you.
What are you thoughts on that?
Quote from: swinxx on September 25, 2013, 06:45:21 AM
hi,
mlv really sounds interesting and i think we all hope that audio will make it to the final verison,
can someone post a link to a actual build with mlv for 5d mark3... have no skills to build my own one. thx.
thx.
Likewise but for 5D2, cant find a newer one that should work besides the old 50D one =)
Quote from: Malex on September 25, 2013, 10:00:06 AM
If that happens, it would be crazy for Canon, they'd have to hire you guys! (to make you stop that nonsense!) or sue you.
What are you thoughts on that?
we dont think about any of these options
Quote from: Malex on September 25, 2013, 10:00:06 AM
I was wondering, do you think or hope that in the future, softwares like Davinci Resolve, After effects, Premiere, and others will read that format ? Will they debayer proxy directly!?
I think there's a pretty good chance that Premiere Pro and Aftereffects will once the .mlv format is fully finalized. Adobe has hinted at it and I don't think they'd have anything to lose by adding that format in much the same way they deal with RED's proprietary format.
Regarding Canon, I don't think they care. They continue to sell cameras, and I'm sure they've seen a spike in sales of the 5D3 since the ML raw came out. Given the recent offerings from Blackmagic, I don't think anyone who is serious about video would consider buying a DSLR anymore if it wasn't for ML raw, so for that I think Canon must be pretty thankful to these guys (whether they'd admit it or not).
hey guys, what's the CPU overhead whilst recording raws?
I was just wondering if it was possible to compress a frame every now and then to "improve" bandwidth. With the meta data included in MLV does it matter if the frames are written out of order?
-Ben
compression -> no
there is nearly no cpu involved, thats the reason why its possible.
no compression because it won't work or because it's too hard or because it isn't a priority?
(I know that this would have to be a wholly software implementation, thus why i was suggesting only a frame here or there)
EDIT: i missed a letter
the cpu is too slow to compress 100MiB/s in realtime
@Enliten Compression takes time. A lot more time than de-compression. No matter what your source of data. So why would you want to add the time to compress onto the time needed to read the data stream in the first place?
I think what you're asking is why can't you throw out a bunch of data you don't need while showing it, to improve speed. That sounds logical. But if you think about for a bit you'll realize that's just another type of compression (which in a sense is already done when the software translates the bytes into data that can be used by the display).
What the devs are probably thinking about is taking the time to compress the stream for easy playback (either at once, or buffering). Realize however, that it would take their time and then you'd have people complaining about load times. (Again, no free lunch).
Some people on the forum get upset if they feel you haven't done your research first, or if you ask a question you could have answered yourself if you thought about it for a bit. Some are here on the forum to get things done and others to learn. So you'll get mixed responses. I recommend going very slowly. Try not to ask any question like "it's not a priority?" that may suggest you think they're blowing certain issues off. Everyone is working on this for free, wishes they had more time, and is already upset that they can't do all they want to do. Everyone sets priorities for themselves. There is no central command.
Also, g3gg0 has put his MLV viewer up on Bitbucket. You're welcome to clone and push any ideas you have!
Hope this helps!
Quote from: maxotics on September 26, 2013, 03:33:48 PMEveryone is working on this for free, wishes they had more time, and is already upset that they can't do all they want to do.
thanks, wise words :)
(at work atm and painting powerpoint slides. fu*k, yeah, hardcore lowelevel developer)
Quote from: g3gg0 on September 26, 2013, 12:34:08 PM
the cpu is too slow to compress 100MiB/s in realtime
This result for the DIGIC 5+?
@maxotics hey, yes i did very little research going into that question.
please be aware that it is never my intent to offend, i just like problems and see a challenge behind a "no". which is why I wanted elaboration.
I did know that realtime compression isn't viable which is why I asked what the CPU overhead is and made the comment about frame numbers.
Let me throw you a hypothetical.
Let's say the devs are able to take a frame that is about to be dumped onto the card, and instead threw it onto camera memory for processing. The camera then compresses this frame and then dumps it onto the card when it is able to next (after having compressed it).
Lets say the 5th frame is the one that gets grabbed for compression. The frames would be written to card 1,2,3,4,6,7,8,9... and once compression has finished then 5,10,11. So it would be a pseudo-realtime compression. Taking a frame here or there and compressing, rather than every frame. The thing that would allow this to be completed is the new MLV format with has frame metadata, allowing the frame to be uncompressed by whatever tool supports MLV.
Of course this is only theoretical, which is why I asked the question in the first place.
Apologies for not elaborating better in the first place.
-Ben
Quote from: g3gg0 on September 26, 2013, 05:49:52 PM
( painting powerpoint slides. fu*k, yeah, hardcore lowelevel developer)
PPT work? You're going to seriously depress everyone!
BTW, I have tons of VBA code in Office (mostly Excel, Access and Word). If you ever need some VBA to do something stupid (so you can spend more time with C), PM me.
if you have a brain-to-ppt converter, that outputs manager-compatible output, let me know.
Quote from: enliten on September 26, 2013, 06:42:04 PM
Lets say the 5th frame is the one that gets grabbed for compression. The frames would be written to card 1,2,3,4,6,7,8,9... and once compression has finished then 5,10,11. So it would be a pseudo-realtime compression. Taking a frame here or there and compressing, rather than every frame. The thing that would allow this to be completed is the new MLV format with has frame metadata, allowing the frame to be uncompressed by whatever tool supports MLV.
good point, but i guess it will not work either.
points against:
- increasing cpu load / memory accesses will slow down CF DMA (concurrent SDRAM accesses)
- locking pages for the frame to compress reduces writing efficiency (one buffer less)
- the real benefit is low, as compressing is slow
- buffer sizes will vary and thus we might see a writing slowdown due to this too
but reducing bit depth to 10 bits could help a bit. on the other hand, i guess the slowdowns cannot be
compensated using some frames that are 5/7th of their original size.
but i like the idea ;)
Quote from: g3gg0 on September 26, 2013, 08:36:38 PM
points against:
- increasing cpu load / memory accesses will slow down CF DMA (concurrent SDRAM accesses)
thanks for the feedback, out of interest does the above point mean that any further reduction of CPU / memory usage will potentially increase write speeds?
Quote from: RenatoPhoto on September 03, 2013, 03:30:26 PM
Keep your personal stuff off this board!
If you read this thread you will find this on reply 218:
and a bit lower you will find this:
So try it and see. If it is not compatible with current build then wait until dev makes it available.
Thanks for this post Renato, it helped me a lot
Just popping my head in here to find out if any progress has been made getting the audio working / back in the raw builds / synched.
Thanks!
Shawn
Can someone please give me a link to the latest build? I've been out of the loop for a long while now... :-/
Is there some central place where it will be easiest to find the module?
Thanks.
Quote from: AnotherDave on October 03, 2013, 01:32:39 AM
Can someone please give me a link to the latest build? I've been out of the loop for a long while now... :-/
Unfortunately MLV has not made it yet to the unified builds so it is not available through nightly builds. You will have to compile it yourself or find a complete build in this thread that you can use.
1% has incorporated the MLV into some of his builds including 7D, and others but dont know which.
Quote from: RenatoPhoto on October 03, 2013, 02:32:46 AM
Unfortunately MLV has not made it yet to the unified builds so it is not available through nightly builds. You will have to compile it yourself or find a complete build in this thread that you can use.
1% has incorporated the MLV into some of his builds including 7D, and others but dont know which.
I know. I'm looking for the most current build that works for the 5D3... i have no idea how to compile something... I have a few days to mess around with any-all new features and I wanted to see MLV's progress on the mac.
Hello,
i intend to switch from to 5D2 to 5D3. (never used a 5D3 yet)
i read about file spanning and got confused.
i know it's a silly question but does it mean that the raw files will be recorded on both CF and SD card?
or the SD card is only used as an additional buffer?
if i buy a 1000x 64GB CF , will it be necessary to buy a 64GB SD card as well ?
thank you for your help
Quote from: moroz on October 04, 2013, 06:03:05 PM
i know it's a silly question but does it mean that the raw files will be recorded on both CF and SD card?
...
if i buy a 1000x 64GB CF , will it be necessary to buy a 64GB SD card as well ?
correct. when enabled, you get some 10MiB/s extra.
this is optional, it can be enabled if you wish.
Hold your horses there.
I was under the impression that 64gig SD cards are not preferred size-wise if you install ML on them...is that right?
I assume that the CF and SD cards can be different sizes, they don't have to match....right?
Quote from: Midphase on October 04, 2013, 07:37:05 PM
I assume that the CF and SD cards can be different sizes, they don't have to match....right?
in this case, size doesn't matter.
Is this build worth installing? Or are the nightlies better?
Quote from: g3gg0 on October 04, 2013, 07:32:56 PM
correct. when enabled, you get some 10MiB/s extra.
this is optional, it can be enabled if you wish.
sorry but i don't understand. how we can record the same file on 2 separate cards??
meaning that we'll have to empty both CF and SD when they are full?
meaning additional data transfer time?
MLV format supports multiple chunks, parallel or sequential.
dont care about that - when enabled you get multiple files on both cards.
thats your video footage to process using mlv2dng or any other supported tool.
g3gg0 thank you for your answers. but i have the impression that it might slow down the pace on the field (double data transfer rate, extra SD cards = extra budget...)
what do you think ? do you recommend MLV for a short film production? (considering that we have limited time for shooting )
then use one card only... it is an OPTION
understood!
i apologize if this question has been asked before:
concerning the stability, what do you recommend for a low budget short film? newest nightly build or the MLV v2.0 ?
thank you
Considering that .mlv is still being worked on, I think the answer is pretty clear. ;D
i updated the module.
http://upload.g3gg0.de/pub_files/1610c3f4dba6ad5dc12e84c626ddfd32/raw_rec_mlv.zip
it has some improved writing/opening/closing code, so it may show increased write performance, especially with longer takes.
source?
New raw2dng.exe is in there if anyone needs to update :)
updated the mlv fork
https://bitbucket.org/g3gg0/magic-lantern-mlv
Quote from: 1% on October 06, 2013, 12:44:12 AM
New raw2dng.exe is in there if anyone needs to update :)
What's new?
Before yesterday updating raw_rec_mlv.zip the App MLV converter for MAC OSX worked fine, now, as I convert the file it creates an empty folder.
In the camera as I use preview, it doesn't work and I see a black screen.
Does anyone have the same problem?
Quote from: g3gg0 on October 06, 2013, 01:38:05 AM
updated the mlv fork
https://bitbucket.org/g3gg0/magic-lantern-mlv
I have been compiling and testing mlv for 5D3 form this source:
https://bitbucket.org/g3gg0/magic-lantern/commits/all
Now you update:
https://bitbucket.org/g3gg0/magic-lantern-mlv/commits/all
Should I setup to compile from this new fork? Why did you not update the other fork?
Any news regarding 5D2? =)
https://bitbucket.org/g3gg0/magic-lantern/ is my main working fork.
it contains more than just the mlv format, but also some other stuff.
https://bitbucket.org/g3gg0/magic-lantern-mlv is an excerpt of the mlv related changes,
intended for merging into main repository. there is an pull request for this fork.
Quote from: g3gg0 on October 06, 2013, 02:07:17 PM
https://bitbucket.org/g3gg0/magic-lantern/ is my main working fork.
it contains more than just the mlv format, but also some other stuff.
Is it possible to compile mlv_rec from your source to cameras other than 5d3?
I've tried it for 650D but without success. Got error 1 at the end.
Quote from: sebamax on October 06, 2013, 12:41:18 PM
Before yesterday updating raw_rec_mlv.zip the App MLV converter for MAC OSX worked fine, now, as I convert the file it creates an empty folder.
In the camera as I use preview, it doesn't work and I see a black screen.
Does anyone have the same problem?
can you open these files with the MLV viewer?
what do you mean with preview and black screen?
Quote from: oddname on October 06, 2013, 01:56:37 PM
Any news regarding 5D2? =)
Quote from: oddname on September 22, 2013, 12:08:32 PM
I can try again, can you point me to the latest build of the module that should work?
(I tried one before, and I was the one saying it froze, gave me speeds of 6-7MB/s instead of 60-70 when it works)
you wanted to try it again, you said a few weeks ago ;)
Quote from: Rewind on October 06, 2013, 02:09:31 PM
Is it possible to compile mlv_rec from your source to cameras other than 5d3?
the module is not camera-dependent, you can use it for any model.
it should work with any camera on which the original raw_rec works too.
but it may require some symbols in the ML/MODULES/<model>.SYM, so you may get an error.
which error do you get? which model?
Quote from: g3gg0 on October 06, 2013, 02:18:46 PM
which error do you get? which model?
Just updated that message: 650D, Error 1.
.sym compiles fine, but when i try to make raw_rec, it gives me this:
(http://ipic.su/img/img7/fs/kiss_34kb.1381061946.jpg)
If you have a couple of mins, would you mind to describe me, what i have to do exactly to try mlv on 650d?
can you run hg update and retry?
hg pull
hg update
make clean
make
:)
Quote from: Rewind on October 06, 2013, 02:24:16 PM
Just updated that message: 650D, Error 1.
.sym compiles fine, but when i try to make raw_rec, it gives me this:
(http://ipic.su/img/img7/fs/kiss_34kb.1381061946.jpg)
I get this kind of error if I do not compile the autoexec.bin first.
So if I compile the make 5D3 and then then compile the modules, i do not get the dep error.
I tried to compile the 650D but got an error, so I compiled for 5D3 and then do the module.
Maybe that will fix it.
Quote from: g3gg0 on October 06, 2013, 02:42:57 PM
hg pull
hg update
make clean
make
I've just deleted all the folders, cloned the whole copy of this branch https://bitbucket.org/g3gg0/magic-lantern/
.sym make works, but raw_rec make gives this:
(http://ipic.su/img/img7/fs/kiss_36kb.1381063246.jpg)
yeah right, you have to recompile the ml binary too, because of the stubs.
sorry, didnt tell you that that what i wrote was meant to be executed in the root directory of the clone :)
so now it works?
Quote from: g3gg0 on October 06, 2013, 02:47:44 PM
yeah right, you have to recompile the ml binary too, because of the stubs.
Of course, first i went to the platform/650d.104 and make. It makes ok, and gives me autoexec and .sym
But then, when trying to make raw_rec module, the error still the same
wait, i think i've got it )
Thank you!
please can you add this to stubs.S for 600D
NSTUB(0xFF1F5C80, msg_queue_count)
Edgar
done, thanks
I cannot compile mlv_dump
[ GCC ] mlv_dump
In file included from ../../src/raw.h:203:0,
from ../lv_rec/lv_rec.h:24,
from mlv_dump.c:39:
../../src/menu.h:170:22: warning: padding struct to align 'help' [-Wpadded]
../../src/menu.h:259:13: warning: padding struct to align 'split_pos' [-Wpadded]
../../src/menu.h:261:1: warning: padding struct size to alignment boundary [-Wpadded]
/usr/lib64/gcc/x86_64-suse-linux/4.7/../../../../x86_64-suse-linux/bin/ld: i386:x86-64 architecture of input file `lzma/lib7z.a(LzmaLib.host.o)' is incompatible with i386 output
/usr/lib64/gcc/x86_64-suse-linux/4.7/../../../../x86_64-suse-linux/bin/ld: i386:x86-64 architecture of input file `lzma/lib7z.a(Alloc.host.o)' is incompatible with i386 output
/usr/lib64/gcc/x86_64-suse-linux/4.7/../../../../x86_64-suse-linux/bin/ld: i386:x86-64 architecture of input file `lzma/lib7z.a(LzmaDec.host.o)' is incompatible with i386 output
/usr/lib64/gcc/x86_64-suse-linux/4.7/../../../../x86_64-suse-linux/bin/ld: i386:x86-64 architecture of input file `lzma/lib7z.a(LzmaEnc.host.o)' is incompatible with i386 output
/usr/lib64/gcc/x86_64-suse-linux/4.7/../../../../x86_64-suse-linux/bin/ld: i386:x86-64 architecture of input file `lzma/lib7z.a(LzFind.host.o)' is incompatible with i386 output
collect2: error: ld returned 1 exit status
make: *** [mlv_dump] Fehler 1
If I change the makefile like this, I can compile
# large file support
# get them with $(shell getconf LFS_CFLAGS)
LFS_FLAGS=-DLARGEFILES -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
# MLV_CFLAGS=-DMLV_USE_LZMA -Wpadded -mno-ms-bitfields -std=c99 -m32 -O2 -Wall $(LFS_FLAGS) -I$(SRC_DIR)
MLV_CFLAGS=-DMLV_USE_LZMA -Wpadded -mno-ms-bitfields -std=c99 -O2 -Wall $(LFS_FLAGS) -I$(SRC_DIR)
Where is my fault ( I´m running OpenSuse 64Bit)
Edgar
i guess it is not your fault, but mine :)
you can keep the -m32 removed maybe.
can you check if compression still works?
Sorry, but next problem with my 600D
If I record a mlv-file, all is fine. Now I stop the recording.
If I want to start a new record, the camera crashes.
assert.log
ML ASSERT:
(int)size == hSuite->size
at ../../src/exmem.c:122 (shoot_malloc_suite), task raw_rec_task
lv:1 mode:20
Magic Lantern version : v2.3.NEXT.2013Oct06.600D102
Mercurial changeset : db0460be09c1+ (unified) tip
Built on 2013-10-06 08:47:30 UTC by [email protected].
Free Memory : 199K + 942K
crash.log
ASSERT: IsChunkSignature( hChunk )
at PackMemory\PackMem.c:264, task RscMgr
lv:0 mode:20
Magic Lantern version : v2.3.NEXT.2013Oct06.600D102
Mercurial changeset : db0460be09c1+ (unified) tip
Built on 2013-10-06 08:47:30 UTC by [email protected].
Free Memory : 198K + 943KEdgar
Quote from: g3gg0 on October 06, 2013, 04:18:03 PM
i guess it is not your fault, but mine :)
you can keep the -m32 removed maybe.
can you check if compression still works?
Not at the moment, because I get an sigsegfault, using mlv_dump
edgar@linux-lyl7:~/test> ./mlv_dump -r -o out.raw 1.MLV
MLV Dumper v1.0
-----------------
[i] Mode of operation:
- Input MLV file: '1.MLV'
- Convert to legacy RAW
- Output into 'out.raw'
[i] File 1.MLV opened
[i] Processing...
[i] Unknown Block: NULL, skipping
SpeicherzugriffsfehlerThe recorded file can be seen in your player
Edgar
Quote from: escho on October 06, 2013, 04:19:31 PM
Sorry, but next problem with my 600D
If I record a mlv-file, all is fine. Now I stop the recording.
If I want to start a new record, the camera crashes.
can you hg pull/hg update and recompile the module?
please also use the trace.mo and enable tracing in raw rec menu.
then send me the raw_rec.txt content.
Quote from: escho on October 06, 2013, 04:26:04 PM
Unknown Block: NULL, skipping
Speicherzugriffsfehler[/code]
can you open raw.h in src/ directory and replace the raw_info struct with this one?
/* raw image info (geometry, calibration levels, color, DR etc); parts of this were copied from CHDK */
struct raw_info {
int32_t api_version; // increase this when changing the structure
uint32_t buffer; // points to image data
int32_t height, width, pitch;
int32_t frame_size;
int32_t bits_per_pixel; // 14
int32_t black_level; // autodetected
int32_t white_level; // somewhere around 13000 - 16000, varies with camera, settings etc
// would be best to autodetect it, but we can't do this reliably yet
union // DNG JPEG info
{
struct
{
int32_t x, y; // DNG JPEG top left corner
int32_t width, height; // DNG JPEG size
} jpeg;
struct
{
int32_t origin[2];
int32_t size[2];
} crop;
};
union // DNG active sensor area (Y1, X1, Y2, X2)
{
struct
{
int32_t y1, x1, y2, x2;
} active_area;
int32_t dng_active_area[4];
};
int32_t exposure_bias[2]; // DNG Exposure Bias (idk what's that)
int32_t cfa_pattern; // stick to 0x02010100 (RGBG) if you can
int32_t calibration_illuminant1;
int32_t color_matrix1[18]; // DNG Color Matrix
int32_t dynamic_range; // EV x100, from analyzing black level and noise (very close to DxO)
};
First the raw_rec.txt
/* 00:00:35.948984 */ Resolution: 896x504 @ 25.000 FPS
/* 00:00:36.097453 */ frame size = 0xC0F00
/* 00:00:36.097725 */ slot 0: pre_align = 0x00000180, edmac_size_align = 3552, write_size_align = 256, data_start = 0x42185000, size = 0xC1E00
/* 00:00:36.097999 */ slot 1: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x42247000, size = 0xC2000
/* 00:00:36.098255 */ slot 2: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x42309000, size = 0xC2000
/* 00:00:36.098519 */ slot 3: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x423CB000, size = 0xC2000
/* 00:00:36.098778 */ slot 4: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4248D000, size = 0xC2000
/* 00:00:36.099031 */ slot 5: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4254F000, size = 0xC2000
/* 00:00:36.099285 */ slot 6: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x42611000, size = 0xC2000
/* 00:00:36.099549 */ slot 7: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x426D3000, size = 0xC2000
/* 00:00:36.099810 */ slot 8: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x42795000, size = 0xC2000
/* 00:00:36.100068 */ slot 9: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x42857000, size = 0xC2000
/* 00:00:36.100330 */ slot 10: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x42919000, size = 0xC2000
/* 00:00:36.100590 */ slot 11: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x429DB000, size = 0xC2000
/* 00:00:36.100856 */ slot 12: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x42A9D000, size = 0xC2000
/* 00:00:36.101115 */ slot 13: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x42B5F000, size = 0xC2000
/* 00:00:36.101379 */ slot 14: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x42C21000, size = 0xC2000
/* 00:00:36.101640 */ slot 15: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x42CE3000, size = 0xC2000
/* 00:00:36.101909 */ slot 16: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x42DA5000, size = 0xC2000
/* 00:00:36.102160 */ slot 17: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x42E67000, size = 0xC2000
/* 00:00:36.102421 */ slot 18: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x42F29000, size = 0xC2000
/* 00:00:36.102938 */ slot 19: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x42FEB000, size = 0xC2000
/* 00:00:36.103243 */ slot 20: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x430AD000, size = 0xC2000
/* 00:00:36.103507 */ slot 21: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4316F000, size = 0xC2000
/* 00:00:36.103775 */ slot 22: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x43231000, size = 0xC2000
/* 00:00:36.104039 */ slot 23: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x432F3000, size = 0xC2000
/* 00:00:36.104304 */ slot 24: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x433B5000, size = 0xC2000
/* 00:00:36.104563 */ slot 25: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x43477000, size = 0xC2000
/* 00:00:36.104917 */ slot 26: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x43539000, size = 0xC2000
/* 00:00:36.105186 */ slot 27: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x435FB000, size = 0xC2000
/* 00:00:36.105471 */ slot 28: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x436BD000, size = 0xC2000
/* 00:00:36.105751 */ slot 29: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4377F000, size = 0xC2000
/* 00:00:36.106049 */ slot 30: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x43841000, size = 0xC2000
/* 00:00:36.106342 */ slot 31: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x43903000, size = 0xC2000
/* 00:00:36.106630 */ slot 32: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x439C5000, size = 0xC2000
/* 00:00:36.106904 */ slot 33: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x43A87000, size = 0xC2000
/* 00:00:36.107194 */ slot 34: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x43B49000, size = 0xC2000
/* 00:00:36.107477 */ slot 35: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x43C0B000, size = 0xC2000
/* 00:00:36.107764 */ slot 36: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x43CCD000, size = 0xC2000
/* 00:00:36.108038 */ slot 37: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x43D8F000, size = 0xC2000
/* 00:00:36.108335 */ slot 38: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x43E51000, size = 0xC2000
/* 00:00:36.108621 */ slot 39: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x43F13000, size = 0xC2000
/* 00:00:36.108924 */ slot 40: pre_align = 0x00000190, edmac_size_align = 3552, write_size_align = 256, data_start = 0x4A001000, size = 0xC1E00
/* 00:00:36.109218 */ slot 41: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4A0C3000, size = 0xC2000
/* 00:00:36.109515 */ slot 42: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4A185000, size = 0xC2000
/* 00:00:36.109803 */ slot 43: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4A247000, size = 0xC2000
/* 00:00:36.110085 */ slot 44: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4A309000, size = 0xC2000
/* 00:00:36.110362 */ slot 45: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4A3CB000, size = 0xC2000
/* 00:00:36.110643 */ slot 46: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4A48D000, size = 0xC2000
/* 00:00:36.110927 */ slot 47: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4A54F000, size = 0xC2000
/* 00:00:36.111207 */ slot 48: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4A611000, size = 0xC2000
/* 00:00:36.111475 */ slot 49: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4A6D3000, size = 0xC2000
/* 00:00:36.111762 */ slot 50: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4A795000, size = 0xC2000
/* 00:00:36.112048 */ slot 51: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4A857000, size = 0xC2000
/* 00:00:36.112338 */ slot 52: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4A919000, size = 0xC2000
/* 00:00:36.112605 */ slot 53: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4A9DB000, size = 0xC2000
/* 00:00:36.112896 */ slot 54: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4AA9D000, size = 0xC2000
/* 00:00:36.113182 */ slot 55: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4AB5F000, size = 0xC2000
/* 00:00:36.113465 */ slot 56: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4AC21000, size = 0xC2000
/* 00:00:36.113736 */ slot 57: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4ACE3000, size = 0xC2000
/* 00:00:36.114018 */ slot 58: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4ADA5000, size = 0xC2000
/* 00:00:36.114305 */ slot 59: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4AE67000, size = 0xC2000
/* 00:00:36.114651 */ slot 60: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4AF29000, size = 0xC2000
/* 00:00:36.114958 */ slot 61: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4AFEB000, size = 0xC2000
/* 00:00:36.115249 */ slot 62: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4B0AD000, size = 0xC2000
/* 00:00:36.115535 */ slot 63: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4B16F000, size = 0xC2000
/* 00:00:36.115823 */ slot 64: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4B231000, size = 0xC2000
/* 00:00:36.116094 */ slot 65: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4B2F3000, size = 0xC2000
/* 00:00:36.116381 */ slot 66: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4B3B5000, size = 0xC2000
/* 00:00:36.116662 */ slot 67: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4B477000, size = 0xC2000
/* 00:00:36.117006 */ slot 68: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4B539000, size = 0xC2000
/* 00:00:36.117299 */ slot 69: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4B5FB000, size = 0xC2000
/* 00:00:36.117589 */ slot 70: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4B6BD000, size = 0xC2000
/* 00:00:36.117873 */ slot 71: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4B77F000, size = 0xC2000
/* 00:00:36.118155 */ slot 72: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4B841000, size = 0xC2000
/* 00:00:36.118427 */ slot 73: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4B903000, size = 0xC2000
/* 00:00:36.118708 */ slot 74: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4B9C5000, size = 0xC2000
/* 00:00:36.118997 */ slot 75: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4BA87000, size = 0xC2000
/* 00:00:36.119282 */ slot 76: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4BB49000, size = 0xC2000
/* 00:00:36.119563 */ slot 77: pre_align = 0x00000000, edmac_size_align = 4064, write_size_align = 256, data_start = 0x4BC0B000, size = 0xC2000
/* 00:00:36.120830 */ Building a group list...
/* 00:00:36.121105 */ group: 0 block_len: 40 block_start: 0
/* 00:00:36.121360 */ group: 1 block_len: 38 block_start: 40
/* 00:00:36.121589 */ Sorting group list...
/* 00:00:36.121811 */ group: 0 length: 40 slot: 0
/* 00:00:36.122031 */ group: 1 length: 38 slot: 40
/* 00:00:36.154540 */ WARNING: This model doesn't have 'PROPAD_GetPropertyData' defined. Reading properties not possible.
/* 00:00:36.154890 */ [IDNT] err: 1 model_data: 0x00000000 model_len: 0
/* 00:00:36.256603 */ <-- WRITER#0: write took: 61386 µs ( 12633 KiB/s), 794112 bytes, 1 blocks, slot 0, mgmt 0 µs
/* 00:00:36.317562 */ <-- WRITER#0: write took: 60839 µs ( 12754 KiB/s), 794624 bytes, 1 blocks, slot 1, mgmt 64 µs
/* 00:00:36.393092 */ <-- WRITER#0: write took: 75448 µs ( 10285 KiB/s), 794624 bytes, 1 blocks, slot 2, mgmt 70 µs
/* 00:00:36.450494 */ <-- WRITER#0: write took: 57301 µs ( 13533 KiB/s), 794112 bytes, 1 blocks, slot 0, mgmt 70 µs
/* 00:00:36.505611 */ <-- WRITER#0: write took: 54873 µs ( 14141 KiB/s), 794624 bytes, 1 blocks, slot 1, mgmt 73 µs
/* 00:00:36.715451 */ <-- WRITER#0: write took: 209972 µs ( 11087 KiB/s), 2383872 bytes, 3 blocks, slot 2, mgmt 72 µs
/* 00:00:36.849533 */ <-- WRITER#0: write took: 133968 µs ( 5788 KiB/s), 794112 bytes, 1 blocks, slot 0, mgmt 62 µs
/* 00:00:37.174537 */ <-- WRITER#0: write took: 325014 µs ( 9550 KiB/s), 3178496 bytes, 4 blocks, slot 5, mgmt 75 µs
/* 00:00:37.591594 */ <-- WRITER#0: write took: 416950 µs ( 7444 KiB/s), 3178496 bytes, 4 blocks, slot 1, mgmt 62 µs
/* 00:00:38.241110 */ <-- WRITER#0: write took: 649460 µs ( 9558 KiB/s), 6356992 bytes, 8 blocks, slot 9, mgmt 71 µs
/* 00:00:38.770845 */ <-- WRITER#0: write took: 529614 µs ( 8791 KiB/s), 4767744 bytes, 6 blocks, slot 17, mgmt 69 µs
/* 00:00:39.715733 */ <-- WRITER#0: write took: 944816 µs ( 9855 KiB/s), 9535488 bytes, 12 blocks, slot 23, mgmt 131 µs
/* 00:00:41.145812 */ <-- WRITER#0: write took: 1429778 µs ( 9226 KiB/s), 13508096 bytes, 17 blocks, slot 0, mgmt 65 µs
/* 00:00:42.582296 */ <-- WRITER#0: write took: 1436464 µs ( 9723 KiB/s), 14302720 bytes, 18 blocks, slot 40, mgmt 67 µs
/* 00:00:44.429078 */ <-- WRITER#0: write took: 1846656 µs ( 9665 KiB/s), 18276352 bytes, 23 blocks, slot 17, mgmt 69 µs
/* 00:00:46.218938 */ <-- WRITER#0: write took: 1790003 µs ( 8670 KiB/s), 15892480 bytes, 20 blocks, slot 58, mgmt 74 µs
/* 00:00:46.865935 */ <-- WRITER#0: write took: 647156 µs ( 20383 KiB/s), 13508096 bytes, 17 blocks, slot 0, mgmt 74 µs
/* 00:00:47.064886 */ --> WRITER#0: expected to terminate
/* 00:00:56.468994 */ Resolution: 896x504 @ 25.000 FPS
Quote from: g3gg0 on October 06, 2013, 04:38:06 PM
can you open raw.h in src/ directory and replace the raw_info struct with this one?
Done, bur now raw2dng doesn´t compile anymore
[ GCC ] raw2dng
In file included from ../../src/chdk-dng.c:57:0:
../../src/raw.h:200:5: error: unknown type name 'uint32_t'
../../src/chdk-dng.c: In function 'write_dng':
../../src/chdk-dng.c:628:20: warning: initialization makes pointer from integer without a cast [enabled by default]
make: *** [raw2dng] Fehler 1Edgar
add a
#include <stdint.h>
to the c file as first line
to raw.c or to chdk-dng.c ?
Edgar
Quote from: escho on October 06, 2013, 05:51:21 PM
to raw.c or to chdk-dng.c ?
Edgar
OK, to raw.h :)
I compiled the whole stuff new with this changes and now the sigsegfault has gone. mlv_dump runs and I can decode the file with raw2dng
ThankYou
Edgar
ok thats the reason why i always use stdint types "uint32_t" etc instead of the random width types "int, long, short".
this saves you from architecture trouble.
(ok i rarely care for endianess conversion in my code and always assume we work on little endian, but thats an other issue)
Quote from: g3gg0 on October 06, 2013, 04:18:03 PM
i guess it is not your fault, but mine :)
you can keep the -m32 removed maybe.
can you check if compression still works?
Compression and decompression seems to work
edgar@linux-lyl7:~/test> ./mlv_dump -c -l 9 -o compress.mlv 1.MLV
MLV Dumper v1.0
-----------------
[i] Mode of operation:
- Input MLV file: '1.MLV'
- Rewrite MLV
- Compress frame data
- Output into 'compress.mlv'
[i] File 1.MLV opened
[i] Processing...
[i] Unknown Block: NULL, skipping
saving: 790272 -> 740650 (93.72%)
[i] Unknown Block: NULL, skipping
saving: 790272 -> 748475 (94.71%)
The camera-name looks a bit strange :)
edgar@linux-lyl7:~/test> ./mlv_dump -v 1.MLV >test
...
Block: IDNT
Offset: 0x000001ac
Size: 84
Time: 28.726000 ms
Camera Name: 'ERR:1 md:0x 0 ml:0'
Camera Serial: ''
Camera Model: 0x00000000
...
Edgar
thats because of:
"/* 00:00:36.154540 */ WARNING: This model doesn't have 'PROPAD_GetPropertyData' defined. Reading properties not possible."
they usually can be found by searching for the string and checking who is being called.
PROP_GetMulticastProperty is usually named prop_get_value, thats the reason why it has the same address.
e.g. 7D:
/** corectly named canon property reading functions **/
NSTUB(0xFF0687C8, PROP_GetMulticastProperty)
NSTUB(0xFF06934C, PROPAD_GetPropertyData)
5D3:
/** corectly named canon property reading functions **/
NSTUB(0xFF123DE4, PROP_GetMulticastProperty)
NSTUB(0xFF12523C, PROPAD_GetPropertyData)
can you hg pull and recompile magic lantern again?
no need to recompile the modules.
it may be necessary to undo the stdint.h and raw.h changes you made
you then should see your camera information in the recorded movie.
Quote from: g3gg0 on October 06, 2013, 02:18:46 PM
you wanted to try it again, you said a few weeks ago ;)
the module is not camera-dependent, you can use it for any model.
it should work with any camera on which the original raw_rec works too.
but it may require some symbols in the ML/MODULES/<model>.SYM, so you may get an error.
which error do you get? which model?
Which I did, but i noticed ppl with 5d2 posted logs and videos showing the same results.
Tried again with the same results, lock-up.
Thought there was another recent build to try :)
Quote from: g3gg0 on October 06, 2013, 07:18:08 PM
can you hg pull and recompile magic lantern again?
...
you then should see your camera information in the recorded movie.
I added the prop... per hand to stubs.S, because your repos are a little bit amazed at the moment ;) . Now the IDNT-Block with camera-name is ok.
ThankYou
Edgar
I'm still getting hsuite error on 7D, second record
Replaced the raw struct too.
Quote
ML ASSERT:
(int)size == hSuite->size
at ../../src/exmem.c:122 (shoot_malloc_suite), task raw_rec_task
lv:1 mode:3
hmm so on 600D and 7D there are such errors?
i tried a bunch of videos on 7D with some older version and had no such errors. damn.
which settings?
Just normal stuff/ gd off + hacks.
I'm trying the binary and it says no info structure found...
Cool, I see MLV works on 550D, just made my first MLV clip.
6D freezes when stopping 2nd record
ML ASSERT:
(int)size == hSuite->size
at ../../src/exmem.c:122 (shoot_malloc_suite), task raw_rec_task
lv:1 mode:3
Have i f'ed something up?
https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/src/0ae870f7dbd161440955320583c5eff381fb8b60/modules/mlv_rec/mlv_rec.c
Some other hints for this camera-crash on my 600d, maybe
I can record as many videos I want, if I switch off the camera after every video.
I record a video and stop it and than switch away from videomode. Now, if i switch back to videomode, the camera tells me: busy, please wait (beschäftigt, bitte warten)
Edgar
Quote from: 1% on October 06, 2013, 09:37:00 PM
6D freezes when stopping 2nd record
Same freeze on 550D
ok there seems to be some weird shoot mem issue... hmm
Tested todays build on 5D3.
At the moment the spanning function records both files on CF card and SD is not being used.
Recorded files are ok.
Ok, I found the commits, thank you for fixing it.. I see 83 -85MB/s instant writes but seems a bit inconsistent.
When I do the buffer test:
7D FULL 1x
M 0 B 3 79.1 5GB
M 0 B 3 80.1 6GB
M 0 B 3 81.9 14GB
M 0 B 3 82.1 14GB
As card becomes more full...
M1 81.2 8GB
M1 80.1 7GB
M1 72.1 1.2GB
M1 77.4 3.2GB
M2 78.1 2.5GB
M2 79.2 4.3GB
M2 70.4 1.5GB
M3 60.5 800MB
M3 73 1.5GB
M3 74.4 1.7GB
M4 71.1 1.3GB
M4 74.5 1.9GB
M4 72.2 1.4GB
M4 75.2 2GB
Raw_rec writes a full card 2nd time recording.
Maybe this test needs to be done on 2.5K in zoom mode?
raw_rec doesn´t compile with latest sources (magic-lantern-mlv) on my 600d. I have to change raw_rec.c like this:
edgar@linux-lyl7:~/MagicLantern/magic-lantern-mlv/modules/raw_rec> diff raw_rec_original.c raw_rec.c
68,69c68,69
< #include <math.h
< #include <cropmarks.h>>
---
> #include <math.h>
> #include <cropmarks.h>
342c342
< static unsigned int raw_rec_should_preview(unsigned int ctx);
---
> static uint32_t raw_rec_should_preview(uint32_t ctx);
Edgar
https://bitbucket.org/g3gg0/magic-lantern-mlv/pull-request/2/5d2-fix
Didn´t see the pull request. Great
Edgar
thanks for the support!
merged in the necessary stuff for 5D2.
i still have the write performance drops every few seconds and no idea where they come from.
600D:
Any reasons not to append these changes to the mlv-sources? mlv_dump doesnt run wifhout for me
Quote from: g3gg0 on October 06, 2013, 04:38:06 PM
can you open raw.h in src/ directory and replace the raw_info struct with this one?
/* raw image info (geometry, calibration levels, color, DR etc); parts of this were copied from CHDK */
struct raw_info {
int32_t api_version; // increase this when changing the structure
uint32_t buffer; // points to image data
int32_t height, width, pitch;
int32_t frame_size;
int32_t bits_per_pixel; // 14
int32_t black_level; // autodetected
int32_t white_level; // somewhere around 13000 - 16000, varies with camera, settings etc
// would be best to autodetect it, but we can't do this reliably yet
union // DNG JPEG info
{
struct
{
int32_t x, y; // DNG JPEG top left corner
int32_t width, height; // DNG JPEG size
} jpeg;
struct
{
int32_t origin[2];
int32_t size[2];
} crop;
};
union // DNG active sensor area (Y1, X1, Y2, X2)
{
struct
{
int32_t y1, x1, y2, x2;
} active_area;
int32_t dng_active_area[4];
};
int32_t exposure_bias[2]; // DNG Exposure Bias (idk what's that)
int32_t cfa_pattern; // stick to 0x02010100 (RGBG) if you can
int32_t calibration_illuminant1;
int32_t color_matrix1[18]; // DNG Color Matrix
int32_t dynamic_range; // EV x100, from analyzing black level and noise (very close to DxO)
};
Quote from: g3gg0 on October 06, 2013, 04:18:03 PM
you can keep the -m32 removed maybe.
Quote from: g3gg0 on October 06, 2013, 05:38:50 PM
add a
#include <stdint.h>
to the c file as first line
Edgar
can you enable the buffer graphs and check if you also get some write rate drop every now and then?
green line = buffer fill level
red line = write rate
Yes, I get a drop in rate every 4 seconds approximately.
It is a repeatable drop, sometimes it skips on cycle and drops at 8 seconds, etc
Also I noticed it is related to idle % calculation. When the idle shows, the drop in rate becomes regular. If you push the speed so the idle % disappears also the drop in rate disappears.
For some reason I am also getting these images as if I was pressing screen capture.
wait, i didnt understand the last post.
you mean CPU idle calculation?
While recording RAW there is a speed indicator in MB/s and next to it appear an idle calculation. If I start recording at low speeds where I show a big idle then the drop becomes regular every approximately 4 seconds. If I speed up recording (by increasing frame size or fps) until idle is gone then I get less drops in speed. If I continue to get close to the maximum speed I get no drops in speed.
Have you enabled some automatic screen capture? I get bmp files and I never pressed Screen Capture.
I get best speed results with buffer fill method = 0 and CF-only buffers= 0 At high speed like 95 to 103 MB/s. (Sometimes zero drops)
Also at the 4GB mark always get drops in speed.
Can anyone test mlv format with 48fps?
On a 5d mk3 i have problems with raw and highest resolution. The writing speed is much lower than in 24fps mode. So it is not possible to record more than 500frames.
Greets re
Higher fps -> more dma bandwidth needed -> reduced write speed
this is inavoidable.
you can try to switch preview mode to hacked, which will disable the live view functionality.
this squeezes some MiB/s out
thx, will try that out.
is there a possibility to record unsqueezed one day or is this not possible?
thx.
Quote from: swinxx on October 13, 2013, 10:55:01 AM
is there a possibility to record unsqueezed one day or is this not possible?
Not possible, line skipping is required to get high fps.
Quote from: RenatoPhoto on October 12, 2013, 05:19:56 PM
Tested todays build on 5D3.
Can you upload mlv module for latest build please? (5D3). I can not compile it myself, sorry.
Quote from: vstrglv on October 13, 2013, 10:04:00 PM
Can you upload mlv module for latest build please? (5D3). I can not compile it myself, sorry.
You can get it from first page look for
pre built
Thank you very much! I tried this link before - it was very old.
Quote from: RenatoPhoto on October 13, 2013, 01:55:01 AM
Also I noticed it is related to idle % calculation. When the idle shows, the drop in rate becomes regular. If you push the speed so the idle % disappears also the drop in rate disappears.
For some reason I am also getting these images as if I was pressing screen capture.
screenshots are nothing new. they are made on stop so you can upload your buffer graph.
the idle time and similar stuff is calculated nearly every write and has no correlation to the performance drop.
see the logs: http://upload.g3gg0.de/pub_files/1ccce7e3382250447649960595a0343e/RAW_REC.TXT
Quote from: RenatoPhoto on October 13, 2013, 03:06:37 AM
Also at the 4GB mark always get drops in speed.
thats normal and its better than in the original raw_rec versions.
file open/close operations take their time.
Quote from: g3gg0 on October 14, 2013, 12:02:23 AM
the idle time and similar stuff is calculated nearly every write and has no correlation to the performance drop.
Ok, but I have seen some kind of cycle on 5D3 tests. The drop sometimes repeats every 4 seconds, sometimes every 2 seconds, sometimes every 8 seconds. It is not totally repeatable by I see some correlation. The drop disappears with high demanding rates and appears more frequent when there is some idle % showing.
I hope you can find the couse
it could be vice versa - as soon the drop rate goes down (due to some other stuff being calcuated or so) the idle time rises.
hard to tell. i guess i will have to start a tskmon trace.
Quote from: escho on October 12, 2013, 10:12:34 PM
600D:
Any reasons not to append these changes to the mlv-sources? mlv_dump doesnt run wifhout for me
Edgar
Shall I create a pull request to ML with the changes in raw.h or will this conflict with existing stuff?
Edgar
Quote from: escho on October 15, 2013, 09:26:51 PM
Shall I create a pull request to ML with the changes in raw.h or will this conflict with existing stuff?
Edgar
ah sorry, forgot to reply.
i am afraid this shotgun approach could raise some other problems :)
can you create a pull request for the main repository and check if the original raw_rec still works fine?
Quote from: g3gg0 on October 15, 2013, 10:13:09 PM
...
can you create a pull request for the main repository and check if the original raw_rec still works fine?
I changed it in my clone of ML-Repo. Raw-Video seems to run fine. I found no problems at the moment.
But
Compiling this whole stuff shows a bunch of compiler warnings like that:
../../src/raw.c:858:33: warning: initialization makes pointer from integer without a cast [enabled by default]
../../src/raw.c: In function 'raw_red_pixel_bright':
../../src/raw.c:866:33: warning: initialization makes pointer from integer without a cast [enabled by default]
../../src/raw.c: In function 'raw_green_pixel_bright':So, regarding these warnings, I don´t have the courage to create a pull request
Edgar
It just makes warnings about comparing those uint_32 to int in other parts of ML source.
Congratulations g3goo and Alex, I see you merged the new format RAW 2.0 to the main branch! I have done a quick test on the 5D3 with and without the spanning function and all seems to work properly. I will be doing more detailed testing and reporting of the new mlv_rec.mo!
This gives me a very positive feeling for the power that the ML team has, and hope that this serves as a very good example for others to follow. The cooperation of all the players to bring their knowledge merging their code (and knowledge) back to the main branch is essential to the benefit of ML Team as a whole. ;D
Only bug so far under cropped mode with preview MLGray or Hacked the LV sometimes remains frozen and requires a re-boot to enable LV.
Quote from: RenatoPhoto on October 18, 2013, 01:55:10 PM
Congratulations g3goo and Alex, I see you merged the new format RAW 2.0 to the main branch! I have done a quick test on the 5D3 with and without the spanning function and all seems to work properly. I will be doing more detailed testing and reporting of the new mlv_rec.mo!
What does it mean exactly? Is mlv in nightly builds now?
Probably soon.
Have the bugs really been sorted out for cameras other than the 5D3? Like the 7D, and 5D2.
Probably not! Will have to test... Do you want to help?
Quote from: RenatoPhoto on October 19, 2013, 02:27:06 AM
Probably not! Will have to test... Do you want to help?
I can help test the 5D Mark II, if needed.
It's there any version for the 7D of the new mlv?
its the same .mo as for the other models.
It's need to be compile? In the the first page i have download the pre_build but just find raw_rec.mo no mlv_rec.mo
Thanks =))
I would like to test mlv on 60D. How should I look for a msg_queue_count stub ?
Let me check the missing stubs tonight
Quote from: g3gg0 on October 19, 2013, 02:29:34 PM
Let me check the missing stubs tonight
So does it work for 5D Mark II?
You know this sigsegfault:
edgar@linux-lyl7:~/test> ./mlv_dump -r -o out.raw *.MLV
MLV Dumper v1.0
-----------------
[i] Mode of operation:
- Input MLV file: 'M18-2052.MLV'
- Convert to legacy RAW
- Output into 'out.raw'
[i] File M18-2052.MLV opened
[i] Processing...
[i] Unknown Block: NULL, skipping
Speicherzugriffsfehler
This error seems to appeare, if I compile mlv_rec on a linux-platform and try to run mlv_dump -r (I cannot test windows, because I don´t have windows).
A solution, that runs for me, is to change the void pointer in the raw_info struct in raw.h to uint:
/* raw image info (geometry, calibration levels, color, DR etc); parts of this were copied from CHDK */
struct raw_info {
...
unsigned int buffer; // points to image data
...
};
Now I have three questions:
1. Why do I have this sigsegfault, I told above, if I use the default pointer from raw.h and compile this whole stuff in Linux?
/* raw image info (geometry, calibration levels, color, DR etc); parts of this were copied from CHDK */
struct raw_info {
...
void* buffer; // points to image data
...
};
2. What can be changed in mlv_rec to avoid this problem? The solution with the unsigned int buffer gives a bunch of compiler warnings like this:
../../src/raw.c:842:33: warning: initialization makes pointer from integer without a cast [enabled by default]
../../src/raw.c: In function 'raw_green_pixel_dark':
3. Can we use my solution until we find a better way?, If yes, I`ll create a pull request.
Sorry, if I have to ask a little bit stupid, but I´m no programmer (but I try to learn a bit :) )
Edgar
Quote from: escho on October 19, 2013, 08:55:47 PM
You know this sigsegfault:
64 bit compiler?
can you update and try again?
Quote from: escho on October 19, 2013, 08:55:47 PM
1. Why do I have this sigsegfault, I told above, if I use the default pointer from raw.h and compile this whole stuff in Linux?
unsigned int buffer; // points to image data
void* buffer; // points to image data;
on 32 bit systems, unsigned int is *usually* 32 bits, on 64 bit systems it is usually 64 bits.
memory pointers, no matter of which type behave the same way.
this is based on
- native register width
- memory address bus width
and compilers may do anything as long C standard is followed.
this even can result in unsigned int being 16 bit and char * being 64 bit.
thus developers are encouraged to use stdint.h where possible.
it defines uint32_t (8/16/32/64) types that are reliably of desired width.
as soon you write e.g. structs that should be read by other systems, you can be sure that
the other system doesnt get into trouble and reads crap.
well, still we all only think of little endian devices, so people with big endian ARM systems will
still have trouble in getting mlv_dump etc running.
Quote from: g3gg0 on October 20, 2013, 12:41:10 AM
64 bit compiler?
can you update and try again?
Yes, 64 bit compiler. Now mlv_dump run without sigsegfault. Great!
And ThankYou for your detailed explanation. :)
Edgar
How do I use it for with my 5D Mark II?
Are there any pre-built modules working for the 5d2 that is newer than 10-3?
the latest pre built module is from 13.10.2013.
you still have to compile the core binary for your model.
Guess Ill wait for the nightlies then =P
Me and compiling stuff... nah :)
Hey guys, long time follower and have loved using magic lantern on my 50D :)
I really know nothing about compiling, code etc but ive got my camera and are willing to test what ever you want if you point me in the right direction (ie, what/how exactly to load the module?? and how you want me to test it)
Ive got plenty of time (just had my knee reco last week) would love to help out wherever possible.
Cheers
Damien
A small change is needed so that "make zip" command will include the mlv_rec.mo in the nightly builds. Should this be done so that mlv_rec.mo is tested across the different builds?
Still getting issues with random corrupt frames on the 50D and just wondering if the buffering method choices are different than those of raw_rec which I don't have any problems with?
It's frustratingly close to being useable but I can't risk dropping frames and there seems to be no pattern to where and when they appear in the footage. I'm using a KB 1000x card and tried even small frame sizes but the corrupt frames still make an appearance. I also have had shots with no crap frames.
I would love to try the 50D specific DPC to Log from Visioncolor with ACR but my mlv->raw->dng files (using freshly compiled mlv_dump.exe and raw2dng.exe) lose the metadata needed and it defaults to Canikon :( How do you convert to legacy raw without losing metadata?
Also, the new mlv player is great! but seems to only work with the first mlv selected. The next mlv's selected make the idx files ok but drop out of the player and back to liveview. Bug?
Just tried the new MLV format. However, the "mlv2dng" is not working, likely due to being on the new Mac OS 10.9. Any suggestions?
Thanks!
Nelson
Exception! Header not recognized: 'STYL'
dyld: lazy symbol binding failed: Symbol not found: __ZNKSt5ctypeIcE13_M_widen_initEv
Referenced from: /Volumes/Media1/MLV/./mlv2dng
Expected in: /usr/lib/libstdc++.6.dylib
dyld: Symbol not found: __ZNKSt5ctypeIcE13_M_widen_initEv
Referenced from: /Volumes/Media1/MLV/./mlv2dng
Expected in: /usr/lib/libstdc++.6.dylib
Trace/BPT trap: 5
Quote from: loconetter on October 25, 2013, 04:25:41 PM
Just tried the new MLV format. However, the "mlv2dng" is not working, likely due to being on the new Mac OS 10.9. Any suggestions?
i dont develop mlv2dng.
the developer was told to NOT THROW errors when an unknown block is seen, because this will happen any time.
dont know why this still happens.
Quote from: Andy600 on October 25, 2013, 04:17:07 PM
Still getting issues with random corrupt frames on the 50D and just wondering if the buffering method choices are different than those of raw_rec which I don't have any problems with?
is it better with lower frame rates?
Quote from: Andy600 on October 25, 2013, 04:17:07 PM
I would love to try the 50D specific DPC to Log from Visioncolor with ACR but my mlv->raw->dng files (using freshly compiled mlv_dump.exe and raw2dng.exe) lose the metadata needed and it defaults to Canikon :( How do you convert to legacy raw without losing metadata?
use mlv2dng
Quote from: Andy600 on October 25, 2013, 04:17:07 PM
Also, the new mlv player is great! but seems to only work with the first mlv selected. The next mlv's selected make the idx files ok but drop out of the player and back to liveview. Bug?
didnt try, will check.
Quote from: g3gg0 on October 25, 2013, 04:58:16 PM
is it better with lower frame rates?
No. I've tried down to 1fps and still get occasional bad frames
Quoteuse mlv2dng
I just tried and it converts but model shows an error in the Exif info: ERR:1 md:0x 0 ml:0 (this is needed for the Visioncolor Log DCP)
I've put CF and SD cards in my 5d3 and installed today's build - everything is working fine but I can't see the file spanning switch in the menus, my SD card seems not to be in use.
Any hint? Thanks.
upd: oops) I'm actually using raw_rec module) Am I right to think that mlv module isn't ready yet?
got two clips today on new build of 5d3 raw that came out green, looks like the other channels weren't captured, very odd...
Quote from: Andy600 on October 25, 2013, 05:08:21 PM
No. I've tried down to 1fps and still get occasional bad frames
does this also happen (even if just rarely) with the old raw_rec module?
i would be surprised, because the share the same copy code.
Quote from: heniloux on October 26, 2013, 10:59:45 PM
got two clips today on new build of 5d3 raw that came out green, looks like the other channels weren't captured, very odd...
can you upload one of these clips?
Quote from: core_32 on October 25, 2013, 08:29:50 PMAm I right to think that mlv module isn't ready yet?
it is ready. just compile and install it.
Quote from: Andy600 on October 25, 2013, 04:17:07 PM
Also, the new mlv player is great! but seems to only work with the first mlv selected. The next mlv's selected make the idx files ok but drop out of the player and back to liveview. Bug?
just committed some fixes. that should work now.
can you test it?
i added mlv_play.mo to the pre-built binaries.
if you install file_man.mo, you can browse your card's file system and open .mlv and .raw files for playback
Im getting also random pink/green clips on footage with MLV
(http://s18.postimg.org/8oadya3x1/Screen_Shot_2013_10_26_at_9_16_08_PM.jpg)
(http://postimg.org/image/8oadya3x1/)
Quote from: g3gg0 on October 26, 2013, 11:07:22 PM
can you upload one of these clips?
the whole footage looks like that?
can you use mlv_dump to write only one frame?
Change the black level.
exiftool -BlackLevel=2048 *.DNG
(or 1024 or just adjust it until you get rid of the color cast)
Quote from: arrinkiiii on October 26, 2013, 11:47:20 PM
Im getting also random pink/green clips on footage with MLV
tried playing in MLV_Viewer ?
Thanks alex, after some reading seems like changing the black level is my forte
G3gg0- the whole clip comes out with the green color cast, gonna get the HDD from the editor this afternoon to check MLV and a couple of other stuff, ill report back as soon as i have more info.
yeah, please also check the footage with MLV-Viewer.
if you can run mlv_dump, run it with -v to dump RAWI content
Quote from: g3gg0 on October 27, 2013, 10:37:52 AM
yeah, please also check the footage with MLV-Viewer.
if you can run mlv_dump, run it with -v to dump RAWI content
I checked it on MLV viewer droped the .RAW in... still green.
raw2dng.exe=green
raw2dng.app=green
rawmagic=green
blacklevel 2048 - works
Im not to familiar with mlv_dump etc
Quote from: g3gg0 on October 27, 2013, 09:57:30 AM
tried playing in MLV_Viewer ?
No, i just use mlv2dng and then open in after effects. All my clips have this pink glitch error in all footage (like in the begging of raw video) I recorded this files showing in Lv "Continuos Recording OK". But why the question if i use MLV_Viewer?
Today will try the new mlv module update (already in the nights builds and see if the pink thing show up again. With the antepenult i already got the right name of my camera in ACR. Thanks for that =))
Thanks :)
Quote from: arrinkiiii on October 28, 2013, 09:59:48 AMBut why the question if i use MLV_Viewer?
because i want you to try it and tell me if it is better there ;)
for all with green cast problems:
mlv_dump -v file.mlv > log.txt
and please send me the log (http://upload.g3gg0.de/)
It's a side effect of broken pink frames (black level is computed from the optical black bars, but if that bar contains corrupted image data in the moment when black level is updated, you'll get green images). Nothing to do with front-ends like MLV or raw_rec; it's in the raw backend (vsync or EDMAC).
yep and mlv_dump will print the RAWI frame and thus the stored black level which comes from autocalculation ;)
Yes, but the autocalculation is only done before recording (at least in raw_rec). So if it starts with a wrong estimation, it will use that for the entire movie.
i cant find a working .mlv2dng i always get an error that says error 11...logout? on shell.
i am on a mac. i wanna compare .raw to .mlv on my 7d. sometimes i try to record on .mlv and it ends up saving as .mov? any reason why?
Quote from: mrnv45 on October 30, 2013, 12:59:59 AM
i cant find a working .mlv2dng i always get an error that says error 11...logout? on shell.
i am on a mac. i wanna compare .raw to .mlv on my 7d. sometimes i try to record on .mlv and it ends up saving as .mov? any reason why?
> mlv2dng: please post in the mlv2dng thread (see first post)
> .mov: well, i guess you forgot to enable the mlv_rec module, or you forgot to enable raw recording in menu
they cant answer me they always seem to skim pass my questions or cant suggest anything for mac.
this being a post about mov_rec i figured i would ask.
and no i have it enabled. thats why i said it records one or 2 videos then the rest are .mov files. is it cause i might hit the 5x zoom?
added dng saving code to mlv_dump.
so if you can compile it yourself, call it with --dng for saving the frames as dng
cannot compile mlv_rec anymore due to mlv_dump-error:
lzma/LzmaEnc.c: In function 'LzmaEnc_Alloc':
lzma/LzmaEnc.c:1908:8: warning: variable 'btMode' set but not used [-Wunused-but-set-variable]
ar: creating lzma/lib7z.a
mlv_dump.c:78:1: warning: padding struct size to alignment boundary [-Wpadded]
mlv_dump.c: In function 'main':
mlv_dump.c:635:9: warning: format '%d' expects argument of type 'int', but argument 2 has type 'long unsigned int' [-Wformat]
mlv_dump.c:1211:33: warning: format '%d' expects argument of type 'int', but argument 2 has type 'size_t' [-Wformat]
mlv_dump.c:1211:33: warning: format '%d' expects argument of type 'int', but argument 3 has type 'size_t' [-Wformat]
mlv_dump.c:1456:33: error: 'struct raw_info' has no member named 'buffer'
mlv_dump.c:1537:37: warning: format '%d' expects argument of type 'int', but argument 2 has type 'size_t' [-Wformat]
make: *** [mlv_dump] Fehler 1
Tried with latest sources
Edgar
I've changed it to prevent wrong code from being emitted on 64-bit systems (you were casting a 32-bit integer to a 64-bit pointer, which has undefined behavior). I think it's better for the compilation to fail, rather than having to hunt for random bugs with apparently no explanation.
So, it´s the best to setup a 32bit machine with vmware, right?
Edgar
Aren't 32-bit executables compatible with 64-bit systems? I remember running them on 64-bit Windows a few years ago, and on Linux they just work (I think I've installed some 32-bit libraries and that was it). No idea about Mac.
Quote from: g3gg0 on October 28, 2013, 11:32:36 AM
because i want you to try it and tell me if it is better there ;)
Hi,
Today i recorded again some MLV with the latest tragic lantern and i have a lot of pink/corrupt frames ...and this time i have try with mlv_viewer :)
[7D]
Quote from: a1ex on October 31, 2013, 06:30:22 PM
Aren't 32-bit executables compatible with 64-bit systems? I remember running them on 64-bit Windows a few years ago, and on Linux they just work (I think I've installed some 32-bit libraries and that was it). No idea about Mac.
Not sure, which 32-bit libraries for linux you mean.
Edgar
RAW format 2.0 specification has info about mlv_wbal_hdr_t. As far I understand we can read values of wbgain_r and wbgain_b. I think we can save white balance info to DNG (AsShotNeutral) like this:
WB RGGB Levels As Shot : 1967 1024 1024 1796
1024/wbgain_r = 1024/1967 = 0,52059
1
1024/wbgain_b = 1024/1796 = 0,57015
static int cam_AsShotNeutral[] = {100000,100000,100000,100000,100000,100000};
cam_AsShotNeutral[0] = 52059;
cam_AsShotNeutral[4] = 57015;
Am I mistaken ?
> http://www.magiclantern.fm/forum/index.php?topic=3409.msg17485#msg17485
i dont really understand how to provide either the RGB multipliers or kelvin value in an useful format for DNG.
any try causes totally weird color values, because it is unclear how canon calculates the WB stuff :(
I've found this paper: http://users.soe.ucsc.edu/~rcsumner/papers/RAWguide.pdf
look at page 8, 4.3 White Balancing
Quote from: g3gg0 on October 31, 2013, 10:49:22 PM
> http://www.magiclantern.fm/forum/index.php?topic=3409.msg17485#msg17485
i dont really understand how to provide either the RGB multipliers or kelvin value in an useful format for DNG.
any try causes totally weird color values, because it is unclear how canon calculates the WB stuff :(
It doesn't answer our questions. These are:
1) what is the relationship between WBGain multipliers and RGB multipliers? (only for Custom WB)
2) what RGB multipliers should we save in the DNG when the EXIF says 2500K or any other Kelvin value? (of course, only for Kelvin WB)
there is one thing i do not understand.
when i go to downloads and see the changelog from e.g. 5d3.. i can see that there are some changes in mlv.rec
but when i download the latest nightly (e.g. 1.nov) the module is not there..
so why? i have downloaded the latest prebuild in this thread.. but all in all the changelog displays changes that are not made within that build (because the module is not included)
so is it useable?
Please forgive me if I misunderstand the question, but I think A1ex's question 1) asks about the relationship between the various arrays of "WB RGGB" (e.g. four ints) values and the actual values to scale each color channel by to white balance?
If so, then marekk is right but has the values inverted. The ratio of the WB RGGB integers is the same as what should be used for the color channel scaling values. Thus, as in marekk's example and setting the green multiplier to be 1 (since only the ratio of them matters), we get
WB RGGB Levels As Shot : 1967 1024 1024 1796
becomes
wbgain_r = 1967/1024 = 1.9209
wbgain_g = 1
wbgain_b = 1796/1024 = 1.7539
These directly applied to the color channels will neutralize the color of the light, but not correct for being in the camera's color space.
The second question is much more complicated, and I'm not sure if there's any standard way to do it. It might be different for every camera, requiring some calibration. If the "WB RGGB Levels" EXIF tag or the "Red/Blue Multiplier" tags are available those should be used instead of the Kelvin. Are there cameras where only Kelvin is available?
If you click on the link above, posted by g3gg0, you will notice this answer is not the one I'm looking for. WBGain are Canon multipliers used for custom WB, and they are different from RGGB multipliers (they look like 1/x at first sight, but they are not).
I finally compiled mlv_dump on mac but when I'm trying to convert do dng files it ends with Bus error 10 (recorded on 60D), on linux - segmentation fault.
small mlv file:
https://drive.google.com/file/d/0B-HdscXfsKpgWW5aeXF3OXdvSlk/edit?usp=sharing
Quotemlv_dump -v --no-cs --dng M02-1733.MLV
MLV Dumper v1.0
-----------------
Mode of operation:
- Input MLV file: 'M02-1733.MLV'
- Verbose messages
- Verify file structure
- Dump all block information
File M02-1733.MLV opened
Processing...
File Header (MLVI)
Size : 0x00000034
Ver : v2.0
GUID : 4679971695627853972
FPS : 24.000000
File : 0 / 0
Frames Video: 29
Frames Audio: 0
Block: RAWI
Offset: 0x00000034
Size: 180
Time: 12.857000 ms
Res: 1600x680
raw_info:
api_version 0x00000001
height 1182
width 1888
pitch 3304
frame_size 0x003B9730
bits_per_pixel 14
black_level 2048
white_level 15000
active_area.y1 26
active_area.x1 152
active_area.y2 1182
active_area.x2 1886
exposure_bias 0, 0
cfa_pattern 0x02010100
calibration_ill 1
Block: INFO
Offset: 0x000000e8
Size: 16
Time: 21.395000 ms
Block: RTCI
Offset: 0x000000f8
Size: 44
Time: 24.897000 ms
Date: 02.10.2013
Time: 17:33:40 (GMT+0)
Zone: ''
Day of week: 0
Day of year: 0
Daylight s.: 0
Block: EXPO
Offset: 0x00000124
Size: 40
Time: 25.451000 ms
ISO Mode: 0
ISO: 100
ISO Analog: 72
ISO DGain: 0/1024 EV
Shutter: 21318 µs (1/46.91)
Block: LENS
Offset: 0x0000014c
Size: 96
Time: 25.488000 ms
Name: '1-65535mm'
Serial: ''
Focal Len: 50 mm
Focus Dist: 0 mm
Aperture: f/1.40
IS Mode: 0
AF Mode: 3
Lens ID: 0x0000001B
Flags: 0x00000000
Block: IDNT
Offset: 0x000001ac
Size: 84
Time: 25.501000 ms
Camera Name: 'ERR:1 md:0x 0 ml:0'
Camera Serial: ''
Camera Model: 0x00000000
Block: WBAL
Offset: 0x00000200
Size: 44
Time: 25.547000 ms
Mode: 0
Kelvin: 6500
Gain R: 489
Gain G: 1024
Gain B: 1024
Shift GM: 0
Shift BA: 0
Block: STYL
Offset: 0x0000022c
Size: 52
Time: 25.559000 ms
picStyle: 33
contrast: -4
sharpness: 1
saturation: -2
colortone: 0
Block: NULL
Offset: 0x00000260
Size: 36
Unknown Block: NULL, skipping
Block: EXPO
Offset: 0x00000284
Size: 40
Time: 4.878000 ms
ISO Mode: 0
ISO: 100
ISO Analog: 72
ISO DGain: 0/1024 EV
Shutter: 21318 µs (1/46.91)
Block: LENS
Offset: 0x000002ac
Size: 96
Time: 4.915000 ms
Name: '1-65535mm'
Serial: ''
Focal Len: 50 mm
Focus Dist: 0 mm
Aperture: f/1.40
IS Mode: 0
AF Mode: 3
Lens ID: 0x0000001B
Flags: 0x00000000
Block: WBAL
Offset: 0x0000030c
Size: 44
Time: 5.187000 ms
Mode: 0
Kelvin: 6500
Gain R: 489
Gain G: 1024
Gain B: 1024
Shift GM: 0
Shift BA: 0
Block: VIDF
Offset: 0x00000338
Size: 1906344
Time: 35.575000 ms
Frame: #0
Crop: 216x264
Pan: 216x264
Space: 2312
Bus error: 10
does it only happen with one file or with anything you record?
just saw that you updated your post after i read it.
thanks for the sample footage, converts fine on linux and windows.
can you tell me which compiler you use? is it a 64 bit system or anything weird? (old m68k mac?)
ah forget about it.
you didnt specify any output file with -o - this causes an error.
didnt catch this case. sorry, will fix it.
it should use a default file pattern of course.
fixed, can you pull and check?
Now it works, thx
... But somenthing is wrong with camera identification:
QuoteBlock: IDNT
Offset: 0x000001ac
Size: 84
Time: 25.501000 ms
Camera Name: 'ERR:1 md:0x 0 ml:0'
Camera Serial: ''
Camera Model: 0x00000000
Quoteexiftool -a test1000005.dng
ExifTool Version Number : 9.33
File Name : test1000005.dng
Directory : .
File Size : 1892 kB
File Modification Date/Time : 2013:11:02 16:57:08+01:00
File Access Date/Time : 2013:11:02 16:57:57+01:00
File Inode Change Date/Time : 2013:11:02 16:57:08+01:00
File Permissions : rw-r--r--
File Type : DNG
MIME Type : image/x-adobe-dng
Exif Byte Order : Little-endian (Intel, II)
Subfile Type : Reduced-resolution image
Image Width : 128
Image Height : 84
Bits Per Sample : 8 8 8
Compression : Uncompressed
Photometric Interpretation : RGB
Image Description : (MLV Video without INFO blocks)
Make : Canon
Camera Model Name : ERR:1 md:0x 0 ml:0
Strip Offsets : 1536
Orientation : Horizontal (normal)
Samples Per Pixel : 3
Rows Per Strip : 84
Strip Byte Counts : 32256
Planar Configuration : Chunky
Software : Magic Lantern
Modify Date :
Artist :
Subfile Type : Full-resolution Image
Image Width : 1600
Image Height : 680
Bits Per Sample : 14
Compression : Uncompressed
Photometric Interpretation : Color Filter Array
Strip Offsets : 33792
Samples Per Pixel : 1
Rows Per Strip : 680
Strip Byte Counts : 1904000
X Resolution : 180
Y Resolution : 180
Planar Configuration : Chunky
Resolution Unit : inches
CFA Repeat Pattern Dim : 2 2
CFA Pattern 2 : 0 1 1 2
Black Level : 2048
White Level : 15000
Default Crop Origin : 0 0
Default Crop Size : 1600 680
Active Area : 0 0 680 1600
Opcode List 1 : (Binary data 28 bytes, use -b option to extract)
Copyright :
Exposure Time : 1/46
F Number : 1.4
Exposure Program : Not Defined
ISO : 100
Exif Version : 0221
Date/Time Original :
Shutter Speed Value : 1
Aperture Value : 1.0
Exposure Compensation : 0
Max Aperture Value : 1.0
Metering Mode : Unknown
Flash : No Flash
Focal Length : 50.0 mm
Sub Sec Time :
Sub Sec Time Original :
Focal Length In 35mm Format : 0 mm
TIFF-EP Standard ID : 1 0 0 0
DNG Version : 1.3.0.0
DNG Backward Version : 1.3.0.0
Unique Camera Model : ERR:1 md:0x 0 ml:0
Color Matrix 1 : 0.6719 -0.0994 -0.0925 -0.4408 1.2426 0.2211 -0.0887 0.2129 0.6051
Analog Balance : 1 1 1
As Shot Neutral : 0.473635 1 0.624
Baseline Exposure : undef
Baseline Noise : 1
Baseline Sharpness : 1.333333333
Linear Response Limit : 1
Calibration Illuminant 1 : Standard Light A
Calibration Illuminant 2 : D65
Frame Rate : 24
Serial Number :
Lens Model : 1-65535mm
Aperture : 1.4
CFA Pattern : [Red,Green][Green,Blue]
Image Size : 1600x680
Shutter Speed : 1/46
Focal Length : 50.0 mm
Light Value : 6.5
Quote from: marekk on November 02, 2013, 05:05:04 PM
Now it works, thx
... But somenthing is wrong with camera identification:
this is most likely due to property handlers not made public in your camera model.
which model is it?
this must be done in ML Core.
60D
solution 1:
can you add these two lines to your stubs.S in platforms/60D... ?
/** corectly named canon property reading functions **/
NSTUB( 0xff05b210, PROP_GetMulticastProperty)
NSTUB( 0xff05be34, PROPAD_GetPropertyData)
then rebuild ML core and try.
solution 2:
add those two lines to your .sym file on your CF in ML/MODULES/
ff05b210 PROP_GetMulticastProperty
ff05be34 PROPAD_GetPropertyData
and then try again without recompiling.
now it works, thx
Quote from: g3gg0 on November 02, 2013, 05:26:30 PM
solution 1:
can you add these two lines to your stubs.S in platforms/60D... ?
/** corectly named canon property reading functions **/
NSTUB( 0xff05b210, PROP_GetMulticastProperty)
NSTUB( 0xff05be34, PROPAD_GetPropertyData)
then rebuild ML core and try.
solution 2:
add those two lines to your .sym file on your CF in ML/MODULES/
ff05b210 PROP_GetMulticastProperty
ff05be34 PROPAD_GetPropertyData
and then try again without recompiling.
ok pushed this change. the next nightly will contain it then.
what do you mean by the next nightly..
5dmk3 had no mlv rec module added til now.. so this means that its added now? am i right?
thx.
Quote from: swinxx on November 02, 2013, 11:15:32 PM
what do you mean by the next nightly..
5dmk3 had no mlv rec module added til now.. so this means that its added now? am i right?
thx.
They talked about 60D, not about 5DIII ;)
Edgar
I know but why can i see all the changes in the changelog ??
The changes you see were not hand-picked.
Hey guys,
Hope you're all having a good weekend.
Is it possible to move the crop area inbetween frames whilst capturing video?
If so then would it be possible to alternate frame locations every second frame so they can be stitched in post.
I'm just thinking that this method could allow a lower frame rate to have a greater aspect ratio.
Cheers,
Ben
possible and would be possible to stitch in post processing simple.
What's the news on audio implementation? :)
Haven't seen any comments about it for a while
nothing new yet.
i am still waiting for someone who is going to implement it. ;)
What if it is just put in how it is or was previously? At least just for now, because shorter clips can work for dialogue or give a starting point for scratch audio for a music video. Even if they are slightly off. It may also inspire people with more knowledge to implement it properly. I sure have no idea.
Quote from: Toffifee on November 07, 2013, 08:16:16 AM
What's the news on audio implementation? :)
Haven't seen any comments about it for a while
I think it's because no one wants to make the sacrifice to make MLV a success.
It doesn't matter how good MLV is as a technical spec, and how much potential it has, it requires politics (buy-ins) from other devs and users to be a success. I don't see it happening any time soon because the simple fact is MLV requires precious bandwidth from the camera. It's a trade-off. Do I think it's worth it? Yes. But even my few attempts to help MLV gain traction have met with failure because the devs don't want to spend time on anything else but work on some new, cool feature (even devs who should want MLV to succeed). Users keep asking for new things and the devs can't say no, or group together and prioritize. Because they work for free I can't say anything without attracting a lot of sycophantic support for the devs--which is misdirected.
On a bright note, ML is a hack that should be dead (because it's a steep development learning curve), yet it lives on, and grows. So who knows, maybe someone will implement an audio solution tomorrow.
Quote from: ChadMuffin on November 07, 2013, 02:31:54 PM
What if it is just put in how it is or was previously?
That would make me VERY, VERY happy! ;D ;D
heh, 5DIII is the only cam that can even close to pull off audio. Maybe could get away with it a little if it wrote at the same time.
I did some quick Googling and see someone has done some minor hacks to a Teac. An Arduino project is possible, though the sample rate would probably be very low. Maybe some other 16/32bit DIY board? I think the ultimate solution is not to record camera audio, but to link up an audio recorder and the camera through some sort of time-code or click-track. Either
A.) Hacked Audio recorder spits out timecode that is burned into the MLV while shooting through USB, the mic input, flash connector...? (thoughts anyone?)
B.) Camera passes a click track to the audio recorder that maybe starts with an easy to synch pattern, like da, da, dum, dum, da dum, da dum. etc.
C.) MLV software on the backend synchs up both streams using timecode or clicktrack, etc.
Any of the master devs here could do this, if time and money were no object :)
For it to happen, someone would have to project manage the thing. Get a time commitment from g3gg0 on the MLV stuff, then one of the camera devs, Alex or 1% or both, and then another dev to work on the audio recorder part and finally, we can't forget about $$$ so each dev can have a complete setup for testing. Also, you want testers, documentation, and some junior devs to assist the master devs.
Well right now clapper board works for me and syncs up perfectly. I can shoot raw on 7D or 50D and then record audio on 600D or 6D, etc.
Quote from: 1% on November 07, 2013, 04:04:28 PM
Well right now clapper board works for me and syncs up perfectly. I can shoot raw on 7D or 50D and then record audio on 600D or 6D, etc.
Me too!
Maybe my point is though the devs are maxed out, stuff like this can get done, IMHO, if people show enough interest by putting in the time to organize it and make it happen. You don't need to be a dev to do project management, or raise money, or administrate bug reporting software, etc. If a user asks you for something you can get done in a day or two, and it seems interesting, you do it. But if they ask you to do something that would take weeks, why do it? You get the same thanks for the first feature as the second. And we can't forget, the dev is also doing stuff HE wants.
So if users want audio in this way they need to band together so there is more reward, challenge AND they need to make sure the dev isn't doing all the work.
but the problem is you're asking near the impossible, most cameras can't deal with writing the video alone, much less an audio stream with it. g3gg0 did this with MLV and it has issues just from the meta data being written.
Quote from: 1% on November 07, 2013, 04:27:09 PM
but the problem is you're asking near the impossible, most cameras can't deal with writing the video alone, much less an audio stream with it. g3gg0 did this with MLV and it has issues just from the meta data being written.
Sorry, I'm not making myself clear. If you could read a simple voltage spike from the mic input say, or flash, etc., then you can read 0s and 1s, right. I don't know how high a frequency you need with, but lets say it's 10 times the frame rate for easy sampling, so you'd need to read at 240bits per second. Okay, so you're reading 0s and 1s from whatever and in the mlv file you're writing either a 0 or 1 for each frame, which gives you a 24bit signature for each second.
At the same time, the recorder is also recording those 0s and 1s (click track) to say channel 4, or the channel 2, if you can live with mono.
Now, in the recorder you hack it to plug in a cable that sends out these spikes every 240 seconds (which are sampled down to 1 bit per frame). You also program the recorder to send out a specific "smoke signal" once it's started, so it sends batches of 12 0s and 1s for the first few seconds, but in a way that you can match up later.
So when you load the MLV file, it builds a signature from the encoded click track. It builds a signature from the audio file and then matches them up. Once they're matched up other software would know where the audio starts and cut it to that point and put the start point in the MLV file which the NLE (thinking big here) would ultimately use to match up the video and audio track.
I'm extemporizing here, but that's the general idea. You mount the hacked audio recorder on a cold-shoe, and maybe, if the dev is a real genius, the camera sends a signal to start and stop the audio recorder and then we're really farting through silk ;)
It might work, it might cause pink frames for non digicV :)
i updated the pre built binaries.
seems to work fine with 60 fps video modes now.
i had some pink frames in this mode before.
can you check it, if it the writing rate is better? getting ~95 MiB/s most of the time now with 24 fps.
For now I'm recording the audio beeps from the headphone output on my 5d3 to one channel of an audio recorder. Together with the time stamp of the raw file, this syncs up pretty well. Snap fingers in front of the lens when I can't use the clapper. Doable and you have a long audio recording which is nice on documentary shoots.
Still, would kill for in camera recording as well.
Yeah, running dual system is of course a good solution, but it is less ideal for us (very) guerrilla filmmakers needs to have minimal gear.
Often when I film people while travelling they freeze up because of the size of my camera and sometimes results in awkward looking shots.
And I didn't even have that much, running a 5DM3 + 24-105, small handheld rig, shotgun mic, pre-amp.
I would love to try out a bare minimum gear set, with a z-finder + videomic pro. But it kind of depends on having in-camera audio.
I can't code, but I'd be very willing to test with my 5DM3!
this time i added a lot of stuff to mlv_play, the module to view raw files in camera.
i will update the mlv_play post (http://www.magiclantern.fm/forum/index.php?topic=9062.0).
Quote from: 1% on November 10, 2013, 07:17:11 PMThen the last problem with MLV is batch tools...
using mlv_dump you can convert .mlv to .dng
anything else missing?
Hi!
I have. 3 questions:
1. Can i convert a mlv with mlv_dump with a mac system?
2. canon 5d mk3 - When will mlv be available in the nightly..?
Cause it would be great to integrate it.. And the user can decide he will enabled the module or not...
3. can i playback mlv within a mac?
Quote from: swinxx on November 10, 2013, 10:45:56 PM
Hi!
I have. 3 questions:
1. Can i convert a mlv with mlv_dump with a mac system?
2. canon 5d mk3 - When will mlv be available in the nightly..?
Cause it would be great to integrate it.. And the user can decide he will enabled the module or not...
3. can i playback mlv within a mac?
1. http://www.magiclantern.fm/forum/index.php?topic=9130.0
2. I think it might be time for mlv_rec to be included in all the nightly builds. Without it, users will continue not to use it as much and post-production development will continue to be behind.
3. I think we need a Mac version of this: http://www.magiclantern.fm/forum/index.php?topic=8447.0
But really 2. is key here, until .mlv isn't in the hands of most ML users, there won't be a whole lot of reason for 3rd party devs (including Adobe) to support it. On the Mac, the best app by far is RAWMagic, and unfortunately it ain't getting updated so it might be time to make some noise!
I've been including it for a long while now.
MLV has been available in Nightly builds for weeks if not longer.... and was available even longer as an extra download.
hmm i cant find it..
when i download todays nightly for canon5dmk3 there is only a mlv_play module, but no mlv_rec module.
thx.
Hmmm.... maybe I added mine through this thread a while back then...
Go to the first page download the prebuilt files and you're off....
I know that i can dl it here. But i just wanted to mention that it would be a great idea to add it to the nightlys
should be in nightly soon
very nice.
mlv2dng is not working here with mac 10.9 and the latest mlv_rec module for 5dmk3
thx.
link: http://www.magiclantern.fm/forum/index.php?topic=7802.msg87994;topicseen#msg87994
use ''mlv_dump --dng'' for splitting mlv files into DNGs
great worked. thank you. :)
as i am not a developer or a terminal guy, a gui would be so great
i am planning to make the commandline parameters similar to raw2dng, so you simply can replace the .exe for GUIs that call native tools.
Hi!
I,m such an illiterate when it comes to terminal in mac. Please help me out. I drag my file to the terminal which creates a directory? Then I type in "mlv_dump --dng"
Nothing gets converted? Check the picture
(http://s10.postimg.org/5n77kpfo5/Ska_rmavbild_2013_11_11_kl_17_52_30.jpg) (http://postimg.org/image/5n77kpfo5/)
hello
first i wanna say that spanned recording works great and i can record 1920x1290 continuously.
i also successfully converted those spanned files with the available mlv_dump version for the mac.
some things that would be great for future gui versions on macs:
- preview of the files - also spanned files(player)
- to be able to batch convert a bunch of files at once into different directories named after the file names (also for spanned files)
- to be able to convert dualiso shots too:)
it would also be great to implement a in camera function to delete bad takes really quick after the shot (also when recorded in spanned mode) perhaps with only one button. that would really be handy..
thx so far - really great stuff and i hope that the progress will go on
@danne
everthing great mlv_dump --dng and then take the file which should be converted from the finder and drop it in the terminal.
worked for me..;)
Hi Swinxx! Could you upload a screenshot of your terminal when you get this going? Or maybe be more specific on the mlv_dump --dng. Do you doubleclick the mlv_dump file or what? Sorry but I can,t get this rigth yet.
Thanks
//D
no,
put the dump file into the dir where you want to convert the files
open terminal
to go to the directory where you have the mlv_dump file and the file to convert type "cd" into the terminal and drag and drop the dir from the findet into the terminal.
then type "mlv_dump --dng" and take the file which you want to convert and drag and drop it into the findet.
then press enter.. this is how it works here.
Hi Swinxx. Sorry but I must be doing something wrong. It goes like this.
1 - I open terminal then type "cd".
2 - Then I drag my directory with the mlv file and the mlv_dump.
3 - write mlv_dump --dng
4 - drag the .mlv file I want to convert and hit enter.
Do I miss space or hitting enter somewhere?
I get this
(http://s10.postimg.org/glsmknted/image.jpg) (http://postimg.org/image/glsmknted/)
1. don't use spaces
2. first cd into a directory ("cd <directory>" and note the SPACE after cd)
3. then execute "./mlv_dump --dng <filename>" (ENTER THE FILENAME MANUALLY WITHOUT ANY PATH)
4. if this works, try using drag'n'drop or whatever, but first try to get it executing the most simple way as described in 1. to 3.
Thanks for helping. Not quite there though.
Got something about illegal instruction, then tried typing "sudo" which created segmentation fault? Such a noob night :)
Nice work on the browser related to the viewer by the way
pics of terminal fail
(http://s9.postimg.org/j5tn29u3v/image.png) (http://postimg.org/image/j5tn29u3v/)
(http://s9.postimg.org/faq8zpay3/image.png) (http://postimg.org/image/faq8zpay3/)
I am trying to switch from raw to mlv back again ao i just added mlv_rec.mo to MODULES folder.
Right now i am testing mlv file spanning at 2,5k 2,35:1 24fps (5d3, CF: KB64 1000x, SD sandisk 45MB/s). I dont know, if this was reported but i tried many times and expected frames number doesnt work properly, for example it shows 1200 and skips after 600 (buffer dots are somewhere in half of the scale and it suddenly stops). Couple months ago just when mlv starts i remember this was very reliable.
Next thing is "beep" sound - could we have a beep when recording starts and stops? Now there is only when stops. THanks
Quote from: g3gg0 on November 12, 2013, 12:06:41 AM
1. don't use spaces
2. first cd into a directory ("cd <directory>" and note the SPACE after cd)
3. then execute "./mlv_dump --dng <filename>" (ENTER THE FILENAME MANUALLY WITHOUT ANY PATH)
4. if this works, try using drag'n'drop or whatever, but first try to get it executing the most simple way as described in 1. to 3.
I am new to the forums, I've done a lot of searching on the web looking for quick and easy tutorials to convert my MLV files to RAW.
I've hit a road block with the "MLV_DUMP" via terminal solution.
Just wanted to say thank you "g3gg0" for posting very clear step by step directions on how to do this. I'm obviously doing it wrong..
Quick overview of my situation.
My Camera
5D Mark II
My Computer system
Macintosh OS X 10.9 "Mavericks"
I downloaded the Alpha_One_5D Mark II Magic lantern file from Nightly build website.
I've tried looking through the magic lantern settings via my 5D Mark II to see if there is a way to just shoot in RAW or DNG format directly, because I read in one forum someone suggested changing the file format from MLV to RAW. I was unable to find the setting to do just that..
I need help, and I've recorded my desktop typing in all the tips that everyone provided in this thread and I keep getting errors. I figured recording exactly what I'm doing in terminal should help everyone narrow down what I'm doing wrong in this process.
Please help!
Thank you!!
http://www.youtube.com/watch?v=A4ogodHSHww&feature=youtu.be (http://www.youtube.com/watch?v=A4ogodHSHww&feature=youtu.be)
Please set video quality to 1080P so you can read exactly what I'm typing in terminal...
Quote from: g3gg0 on November 12, 2013, 12:06:41 AM
1. don't use spaces
2. first cd into a directory ("cd <directory>" and note the SPACE after cd)
3. then execute "./mlv_dump --dng <filename>" (ENTER THE FILENAME MANUALLY WITHOUT ANY PATH)
4. if this works, try using drag'n'drop or whatever, but first try to get it executing the most simple way as described in 1. to 3.
Thanks, work fine,
I finally did
Quote from: Danne on November 12, 2013, 12:42:40 AM
Thanks for helping. Not quite there though.
Got something about illegal instruction, then tried typing "sudo" which created segmentation fault? Such a noob night :)
Nice work on the browser related to the viewer by the way
pics of terminal fail
(http://s9.postimg.org/j5tn29u3v/image.png) (http://postimg.org/image/j5tn29u3v/)
(http://s9.postimg.org/faq8zpay3/image.png) (http://postimg.org/image/faq8zpay3/)
is the binary without any parameters running fine?
Thanks for all the feedback!
Quote from: g3gg0 on November 12, 2013, 12:06:41 AM
1. don't use spaces
2. first cd into a directory ("cd <directory>" and note the SPACE after cd)
3. then execute "./mlv_dump --dng <filename>" (ENTER THE FILENAME MANUALLY WITHOUT ANY PATH)
4. if this works, try using drag'n'drop or whatever, but first try to get it executing the most simple way as described in 1. to 3.
Worked for me too. Maybe it is useful to say that mlv_dump should be in the same folder as the .mlv files.
Thanks for the great work. Hope for sound soon.
Quote from: g3gg0 on November 12, 2013, 09:25:12 AM
is the binary without any parameters running fine?
Don,t know how to check binary but tried doubleclicking the mlv_dum file which resulted in "illegal instruction"?
(http://s23.postimg.org/8rs441j3r/Ska_rmavbild_2013_11_12_kl_13_37_06.jpg) (http://postimg.org/image/8rs441j3r/)
This means that the binary is not for your processor
aha, thanks for claryfing this. I,ll stick to my virtual box for now. Thanks for helping out
for the card spanning users:
can you reduce CF only buffers to 1 or 2 and play with the buffering method?
maybe this will help. method 3 or 4 was the most reliable for my setup.
Tried CF only buffers 1 and 2 (also combined with all buffer fill methods), 1 seems to be a little bit better (10-15 sec more) than 3 or 4 but still far from performance from the beginning. With 2,5k 2,35:1 i am getting 35-45 sec (23,976). Ah.. and it seems that "expected frames" and color buffer graph also doesnt work properly - because it skips frames when theoretically recording should last longer.
yeah it's possible that the estimation isnt that acurate.
how are your data rates when you enable spanning and the camera barely shows a continuos OK?
108-109 MB/s in total. When it shows about 108,5 it estimates 2500-3500 frames but after 35-45sec suddenly decreases and frame skips when shows 800-1200.
EDIT: Could we have a 2240 horizontal resolution as it is in raw_rec? I have just noticed, that in raw rec we have 1920,2048, 2240, 2560, 2880 etc, mlv_rec has 1920, 2048, 2560, 2880.
2240 is missing and this res could give decent frame number in 16:9 with good margins for post stabilization. thanks
Quote from: g3gg0 on November 12, 2013, 01:19:59 PM
This means that the binary is not for your processor
Thanks for all the help by the way.
I'm getting the same message "cannot execute binary file"
would the MLV_dump work if I use terminal on say, another macintosh running an older version of OS X?
or would I have to use a windows machine?
are there any alternatives to converting MLV files? I've seen many videos online that show how people convert and edit their files, but it seems they leave out the mlv_dumping part and their files are already RAW..
Again, thanks for the insight and I appreciate all the tips I can get from you. I'm just eager to start editing the raw footage or "MLV" footage I shot..
Link :Watch on youtube (http://www.youtube.com/watch?v=PFaiU7rv1pg)
its working on a 2008 mac book pro running OSX Mavericks
cd desktop was to navigate to my desktop
cd mlv was to bring it to the mlv folder inside desktop
do note mlv_dump is in the same folder (idk if its necessary)
You should probably try it on a different PC. I had the binary error while in the VM.
mlv_dump has this useful option:
-f frames stop after that number of frames
Is it possible to integrate another option:
begin with frame number x and stop after frame number y
Edgar
good point, will add.
What information do you need in order to diagnose an issue? I have found a frame with a weird border and it's happened a couple of times now while using MLV_REC.MO. Want to ensure any bugs I submit have all relevant information so as to not waste any time.
what do you mean with weird border?
I need help converting my MLV's, I was blindsided by this, I Bought a 7d and spent hours digging through threads trying to find the latest nightly build so i could go shoot raw, well just got back and a discovered I shot everything in this MLV format, which sounds great, but I can't process the files :( Im on a mac and I was hoping for a little guidance, the info is speed all over this form, I gather what MLV is and I have the mlv_dumper and that i need to use terminal but just not working for me :(
Did you see the video above?
And also this?
http://www.youtube.com/watch?v=A4ogodHSHww
in this video he blindly entered "<filename.mlv>".. correct would be "filename.mlv" of course
i didnt expect that someone really doesnt understand the common computer syntax for obligatory parameters :)
Quote from: RenatoPhoto on November 22, 2013, 01:36:54 PM
Did you see the video above?
And also this?
http://www.youtube.com/watch?v=A4ogodHSHww
Yes I did, I did not work, " invalid Instruction 4 " or something to that effect, It seems the binary doesn't work on my mac. I've tried both the dumper and the mlv2dng. Nether worked on this mac. It seems I have no way to process these files. Total bummer. Even tried that app, want work on 10.7.5
Get yourself a virtualmachine for windows
Quote from: escho on November 17, 2013, 09:10:58 PM
mlv_dump has this useful option:
-f frames stop after that number of frames
Is it possible to integrate another option:
begin with frame number x and stop after frame number y
Edgar
I see your commit. ThankYou for adding :)
Edgar
you are welcome.
can you test it thoroughly?
I tested all possibilities. It runs fine.
But I noticed a 0 Byte-file:
edgar@linux-lyl7:~/test> dir
insgesamt 158688
-rwxr-xr-x 1 edgar users 154351712 19. Okt 14:36 1.MLV
-rwxr-xr-x 1 edgar users 193028 23. Nov 18:14 mlv_dump
-rw-r--r-- 1 edgar users 0 23. Nov 18:28 test
-rw-r--r-- 1 edgar users 1985280 23. Nov 18:28 test000000.dng
-rw-r--r-- 1 edgar users 1985280 23. Nov 18:28 test000001.dng
-rw-r--r-- 1 edgar users 1985280 23. Nov 18:28 test000002.dng
-rw-r--r-- 1 edgar users 1985280 23. Nov 18:28 test000003.dng
What is this file for? This file appeares without the f-option, too.
Edgar
Question: do we have any way of extracting just the metadata per frame to a file so that my shot logs can leave out the ISO/shutter/aperture information, and follow focus/zoom etc?
I've tried extracting just the metadata but it comes out as that 0 byte file.
./mlv_dump -m -o meta.mlv in.mlv
after that you can dump all metadata with
./mlv_dump -v meta.mlv
doesnt that work?
Quote from: escho on November 23, 2013, 06:49:05 PM
I tested all possibilities. It runs fine.
But I noticed a 0 Byte-file:
What is this file for? This file appeares without the f-option, too.
Edgar
thanks a lot.
that sparse file should be gone now.
OK! Has gone.
Edgar
Another thing: bad pixel remooving
I tested the bad-pixel remooving from cr2hdr with normal dngs, extracted from a raw-video-clip with raw2dng. To be able in doing so, I fooled the hdr_check in cr2hdr and looked at the debug-output-dngs: All the bad pixels are remooved! That´s really great!
Can this function be ported to mlv_dump in some way?
Edgar
can anyone please post a link with the current mlv_dump version for mac
thx
Seems that after moving thread to other place download link from the first page doesnt work anymore, i have tried many times and no luck.
btw. raw2cdng is now able to use pre-converting to mlv->raw ;) v1.4.5
http://www.phreekz.de/wordpress/2013/06/magiclantern-raw2cdng-cinema-dng/
ah nice, the first step :)
reading mlv isnt that hard.
which programming language do you use?
g3gg0 c# - lol, i use your mlv_dump.exe as preprocessor :) in german: "das rad nicht neu erfinden" - but at least, yes, i'll rewrite it natively.
raw2cdng is written in C#?
then you/we can easily add MLV support
great!!!!
Ok not gonna lie, tried to find this info elsewhere, none of the converters worked for me....
How the hell do I convert MLV to DNG/CDNG on a mac? Does the MLV dump work? And if so is there a way to batch process them in to 16 bit so I can use speedgrade cs6?
not trying to be a troll I literally cannot find any working converter and it is annoying the hell out of me.
Quote from: Danson Delta-40 on November 28, 2013, 07:31:20 AM
Ok not gonna lie, tried to find this info elsewhere, none of the converters worked for me....
How the hell do I convert MLV to DNG/CDNG on a mac? Does the MLV dump work? And if so is there a way to batch process them in to 16 bit so I can use speedgrade cs6?
not trying to be a troll I literally cannot find any working converter and it is annoying the hell out of me.
Try ¨Son of the Batch¨ to open Raw and import to DNG.
Open DNG with Da Vinci Resolve lite or Adobe Photoshop ACR.
Forget the bits. There are a lot.
Good Lantern!
Quote from: budafilms on November 28, 2013, 07:35:16 AM
Try ¨Son of the Batch¨ to open Raw and import to DNG.
Open DNG with Da Vinci Resolve lite or Adobe Photoshop ACR.
Forget the bits. There are a lot.
Good Lantern!
I've been converting RAW for a while now, its MLV I've been having a hard time with.
AGAIN: I want to convert MLV(EDIT: Convert on MACINTOSH), I already know how to convert RAW
i didnt upload binaries, as i cannot compile it for mac.
can you ask there? http://www.magiclantern.fm/forum/index.php?topic=9130.msg89058#msg89058
I downloaded the latest pre-built binary (was at http://upload.g3gg0.de/pub_files/27a867063ed87d2f079cde5f8b577a88/mlv_rec.zip), placed its autoexec.bin at the root level of the SD card and the other files in modules, and am now experiencing a problem with the magiclantern menus in my 5D3 (as well as text displayed on the liveview screen when shooting). It's now as if my camera's screen is cropping its input (looks almost like overscan), as if the buffer for the screen is larger than the screen's resolution and it's just displaying a center extraction. Even the boot screen features text cropped at the edges. And the explanatory text for ML menu options runs on top of other menu items and off the screen.
I've removed my config files and restored defaults, but that didn't help. Perhaps I was using a sufficiently-old set of ML files that I'm missing some new smaller font that would make everything fit? The text all looks huge compared to what it was with my last set of ML files (was recording raw, but didn't have MLV 2.0).
Thanks for any help or info. Here's a screenshot showing what menus look like:
(http://www.zachfine.com/forum_images/VRAM0.jpg)
You need fonts files.
It's odd --I have fonts files, was recording raw just fine before, and everything looked fine. Can someone tell me which distribution of magic lantern has the files that are compatible with the mov 2.0 autoexec and modules? The latest? Or do I need to backtrack a few versions?
Alright, got the latest 5D3 magic lantern files in there and it's all looking good now. Great to see the rapid pace of development!
I'm having a bit of trouble, likely expected with these cutting edge builds. I'd love to have magic zoom but remove the text on the screen that tells me the names of the two files that are being written, as it's covering at least 20% of the video with those names and bitrates. All I want to see is magic zoom and a timer telling me the length of the recording. Haven't discovered the right option though I've explored every possible menu as far as I can tell.
I'm finding that if I set for spanning, I can only record about 10 seconds of 1080p23.976 raw video before it skips a frame and stops. But if I turn off spanning it records continuously just fine. Maybe it's due to the speed (or lack thereof) of my SD card? My CF is a Komputerbay 1050X 128Gb that's been working great for 1080p23.976 canon raw for the past couple of months.
One feature idea: I record second system audio using a r0de videomic plugged into a zoom H1. A pretty cheap, nice=sounding combo. It'd be neat if the camera had a way of outputting a series of beeps at the start of recording that would identify the name of the filename to which the audio recording would belong. Morse code perhaps? Maybe the name of the clip in morse over the first couple of seconds of the clip and then a beep that coincides with the exact frame on the 3 second mark to use to sync? Or a beep that coincides with a little dot in the corner of the frame, fire the flash, darken the frame, something that can be used visually to line up with the beep. Anyway, just throwing that out there in case it makes sense to anyone (or in case it's already in there and I haven't noticed).
Quote from: zachnfine on November 29, 2013, 05:47:53 AM
One feature idea: I record second system audio using a r0de videomic plugged into a zoom H1. A pretty cheap, nice=sounding combo. It'd be neat if the camera had a way of outputting a series of beeps at the start of recording that would identify the name of the filename to which the audio recording would belong. Morse code perhaps? Maybe the name of the clip in morse over the first couple of seconds of the clip and then a beep that coincides with the exact frame on the 3 second mark to use to sync? Or a beep that coincides with a little dot in the corner of the frame, fire the flash, darken the frame, something that can be used visually to line up with the beep. Anyway, just throwing that out there in case it makes sense to anyone (or in case it's already in there and I haven't noticed).
did you try the SMPTE module?
I haven't yet tried the SMPTE module. I've got a 3.5mm dual-mono to stereo splitter cable ordered, and once it arrives I'll try running the camera's headphone jack into 1 channel of my zoom H1 and the videomic into the other. Maybe I can get Resolve or some other program to recognize the timecode track. I've read all the comments here and am not sure if anyone's gotten the module to work for syncing.
Hello! I can't get this working on 5d Markiii! How shall I do? I have copied the autoexect.bat and other stuff on the sd card over the old one, but the camera won't start. Am I missing something?? Thank u!
First, you make sure you're running the nighly off that SD card. Then you copy the
autoexec.bin and the rest of the files from the mlv_rec.zip (http://upload.g3gg0.de/pub_files/27a867063ed87d2f079cde5f8b577a88/mlv_rec.zip). Next step, enable the new module and go with the flow :)
If your camera won't start... go back to step 1.
Quote from: mhimhi on December 04, 2013, 11:39:16 AM
Hello! I can't get this working on 5d Markiii! How shall I do? I have copied the autoexect.bat and other stuff on the sd card over the old one, but the camera won't start. Am I missing something?? Thank u!
I'm getting some f@#$d up frames when using Card spanning:
(http://www.quinox-std.com/ml/Comp1%280-00-01-13%29.jpg)
You can download the whole snippet: https://www.dropbox.com/s/rw7vewpyv4xc9u6/Comp%201.mp4 (https://www.dropbox.com/s/rw7vewpyv4xc9u6/Comp%201.mp4)
Here's the rundown:
1920x508 @ 50fps / 2.25:1 / Global Draw On / Auto Preview / Memory hack On / Extra Hacks On (KB 32GB 1000X CF & SanDisk 32GB Extreme Pro SD)
Same settings, Card spanning Off will give me almost perfect results (just a single frame gets "busy" somewhere in the first 10-20frames)...
Ideas?
when you watch the video frame by frame in MLV Viewer, is there smth weird to see?
@geggo, yeah... the same messed up frames :(
https://www.dropbox.com/s/36suy3x3h9w8u7e/Span.zip (https://www.dropbox.com/s/36suy3x3h9w8u7e/Span.zip)
if you wanna check for yourself(mlv's)
Say Hi to mlv_rec in the nightlies:
Changelog for 2013-12-11 02:11:09 +0200
8881: Add mlv_rec to nightly builds
Quote from: jose_ugs on December 11, 2013, 10:37:20 AM
Say Hi to mlv_rec in the nightlies:
Changelog for 2013-12-11 02:11:09 +0200
8881: Add mlv_rec to nightly builds
;)
ooooh, i'll have to go get the nightly (on 550D too right?)
What is the current differences at this point in time between raw_rec and mlv_rec?
Is there some sort of comparison list for the now rather than for what its plans are?
Any chance we'll soon have the option to move the recording information to the upper part of the screen like the legacy RAW format?
yep, i am planning to port this.
Is there any audio support other than the sync beep for mlv_rec at this point?
Thanks!
sorry, there is no other audio support yet :(
What about the SMTPE module?
Quote from: jose_ugs on December 18, 2013, 10:02:07 AM
What about the SMTPE module?
what do you want to know about it?
Isn't it a good solution for audio sync (if your equipment can take advantage of it, of course)?
Quote from: jose_ugs on December 18, 2013, 12:41:58 PM
Isn't it a good solution for audio sync (if your equipment can take advantage of it, of course)?
uhm, yes it is.
the module is available with nearly all functions people wanted, but noone really checked if it is working for their workflow.
it was even included in most mlv_rec builds i uploaded here.
see this thread: http://www.magiclantern.fm/forum/index.php?topic=7764.0
do you have real equipment to play with it?
I'm waiting for a Tascam DR-60D to show up real soon & i'll be on it!!!
Offtopic: Is there a way to "follow" a thread, without having to post there? I want to watch this SMTPE thread in case of new development/new posts...
update:
- added mlv_snd which adds audio recording to mlv_rec
HowTo:
- unpack modules into ML/MODULES directory
- enable mlv_snd module
- go to "Audio" menu and enable "MLV Sound"
- record a video using mlv_rec
- test the video on your computer (mlv_dump or MLV Viewer latest revision)
when you call './mlv_dump -o <outname>', it will create a .wav
(thanks to Jarno Paananen (https://bitbucket.org/hudson/magic-lantern/pull-request/309/add-wav-header-to-audio-output-file-and/diff))
can you test it whether it works for you? tried on
5D3: works
7D: no audio recorded
Quote from: g3gg0 on December 22, 2013, 01:02:45 AM
update:
- added mlv_snd which adds audio recording to mlv_rec
you can play it using
a) my MLV Viewer or
b) when you use 'mlv_dump -o <outname>' which will create a .wav (thanks to Jarno Paananen, https://bitbucket.org/hudson/magic-lantern/pull-request/309/add-wav-header-to-audio-output-file-and/diff)
can you test it whether it works for you? tried on
5D3: works
7D: no audio recorded
g3gg0,
Is mlv_snd part of the latest nightly build? I would love to test it out for my 5D3. Thanks so much for all the hard work and updates!
I just tried on my 6D and i can see the meters but apparently i can't hear any audio recorded.
Am i doing it wrong?
I am viewing the MLV file in MLV viewer but can't hear anything.
Suggestions?
edit:
re-uploaded the package, the module was missing in the .zip.
Oh, that's what i am missing.
I thought it was in the zip file but indeed it isn't.
Am i missing something again?
mea culpa, i didnt drop it onto the .zip, but next to it.
re-uploaded, please test again ;)
updated the post before with installation instruction.
-> enable mlv_snd, go to audio menu and enable "MLV Sound". then you can record.
Still no luck.
I get FFT graphs but apparently no audio is recorded. I am playing MLV files on the viewer with everything on low so it's fast.
I also got a crash, i am trying to reproduce it.
QuoteASSERT: FALSE
at ./ASIF/ASIF.c:444, task beep_task
lv:1 mode:3
Magic Lantern version : Tragic.2013Nov09.6D113
Mercurial changeset : a315c155c985 (unified) tip
Built on 2013-11-09 17:19:15 UTC by User@PC.
Free Memory : 302K + 1694K
Also i am getting "audio failed to stop state 4" in the middle of a recording sometimes.
i hear audio in MLviewer but i don't understand how to output the DNG with audio.. any help please.. test on 5D3
What camera?
5 d 3
Quote from: ilguercio on December 22, 2013, 03:18:21 AM
I also got a crash, i am trying to reproduce it.Also i am getting "audio failed to stop state 4" in the middle of a recording sometimes.
please use a
recent version of
magic lantern.
you are using an outdated custom build.
Quote from: Jakobmen on December 22, 2013, 05:13:36 AM
i hear audio in MLviewer but i don't understand how to output the DNG with audio.. any help please.. test on 5D3
did you try mlv_dump already?
I can not believe you added AUDIO SUPPORT! :D 8)
It really works for me on 5D3 (Komputerbay 128GB, Fill:3 Buffer:4 CF & SD: Mode S1)
(Edit: CF-only: 8 works more reliable)
This is a niiice x-mas present!
PS: Test-Mode freezes sometimes using the nightly build (Dec 22 2013). This happens after about 3-5 aquisitions stopped by a skipframe.
there is no mlv_snd in my 550D
Quote from: g3gg0 on December 22, 2013, 08:37:41 AM
did you try mlv_dump already?
it works but looking for easier way.. =p
@g3gg0, andy kh: sound recording works on 550D!
Steps I took:
- Downloaded latest (22Dec) nightly build for the 550D, also your mlv_rec.zip file from the first post of this thread
- Installed nightly build on my SD card by simply copying it over the existing files
- Extracted mlv_snd.mo from the .zip file and copied it into the MODULES folder on the card
- Enabled mlv_play, mlv_rec, and mlv_send, rebooted to load modules
- Enabled "MLV Sound" in Audio menu
- Recorded a short (18 sec) clip (using mlv_rec, obviously, and not raw_rec), which was called M22-1151.MLV
- Clip played back in camera, created a file M22-1151.IDX
- Tried to get mlv_dump (also from your mlv_rec.zip file) to work, but it kept giving error message about "Missing header"
- Renamed index file (created in camera) to M22-1151.IDX.bak and now mlv_dump works with following command (assumes a copy of mlv_dump.exe has been copied to C:\) :
C:\>mlv_dump --dng "E:\raw tests\20131222\M22-1151.MLV" ## replace the path with YOUR path to the .MLV file
- After this the typical processing with Photoshop CC (for ACR, synchronizing all .dng's to the selected settings), then After Effects CC -> Adobe Media Encoder to render as H.264 .mp4 file
I did notice that the sound file was a bit longer than the .dng sequence (I was shooting at 25fps): 19.00 seconds vs. 18 seconds, 20 frames for the video. I simply assumed that the starting points were in sync and the sound file cut off a bit later than the video, and that seems to have been a correct assumption, although a longer (few minutes) test is in order to check this thoroughly.
Very nice that we have sound with raw now, excellent work! Cheers...
Quote from: NedB on December 22, 2013, 01:51:34 PM
I did notice that the sound file was a bit longer than the .dng sequence (I was shooting at 25fps): 19.00 seconds vs. 18 seconds, 20 frames for the video. I simply assumed that the starting points were in sync and the sound file cut off a bit later than the video, and that seems to have been a correct assumption, although a longer (few minutes) test is in order to check this thoroughly.
thanks for the feedback. you are right. recording starts with the first frame (to be verified)
and records about 0.5 sec longer because the buffers are already queued for filling when you stop recording.
as soon the buffers are done filling, audio recording stops
Hello
When i try to extract the dng's on mac with mlv_dump, i got this message, and i can't see the audio files :
Vertical stripes correction:
1.000 1.000 0.997 0.994 0.993 0.990 0.999 1.018
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Unknown Block: AUDF, skipping
[i] Reached end of chunk 0/1 after 766 blocks
[i] Processed 265 video frames
[i] Done
thats obvious because you dont use the latest version...
i added sound support last night, so needless to say that you need to get an updated version....
Thank you for you response, but i only found on your file the linux and windows version ... any chance to have the osx mlv_dump ?
audio works on 600D too (if I have enabled audio-recording in canon-menu).
mlv_dump creates a wav-file which I can playback, using mplayer for example
mlv-viewer crashes:
edgar@linux-uoww:~/Software/MagicLantern> mono MLVViewSharp.exe
KDE colorscheme read failure, using built-in colorscheme
Unhandled Exception:
System.IO.DirectoryNotFoundException: Could not find a part of the path "/HOME/EDGAR/AUDIOTEST/M22-1707.IDX".
at System.IO.FileStream..ctor (System.String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, Boolean anonymous, FileOptions options) [0x00000] in <filename unknown>:0
at System.IO.FileStream..ctor (System.String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, Boolean isAsync, Boolean anonymous) [0x00000] in <filename unknown>:0
at System.IO.FileStream..ctor (System.String path, FileMode mode, FileAccess access) [0x00000] in <filename unknown>:0
at (wrapper remoting-invoke-with-check) System.IO.FileStream:.ctor (string,System.IO.FileMode,System.IO.FileAccess)
at mlv_view_sharp.MLVReader.SaveIndex () [0x00000] in <filename unknown>:0
at mlv_view_sharp.MLVReader.UpdateIndex () [0x00000] in <filename unknown>:0
at mlv_view_sharp.MLVReader..ctor (System.String fileName, mlv_view_sharp.MLVBlockHandler handler) [0x00000] in <filename unknown>:0
at mlv_view_sharp.MLVViewerForm.PlayFile () [0x00000] in <filename unknown>:0
at System.Threading.Thread.StartInternal () [0x00000] in <filename unknown>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.IO.DirectoryNotFoundException: Could not find a part of the path "/HOME/EDGAR/AUDIOTEST/M22-1707.IDX".
at System.IO.FileStream..ctor (System.String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, Boolean anonymous, FileOptions options) [0x00000] in <filename unknown>:0
at System.IO.FileStream..ctor (System.String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, Boolean isAsync, Boolean anonymous) [0x00000] in <filename unknown>:0
at System.IO.FileStream..ctor (System.String path, FileMode mode, FileAccess access) [0x00000] in <filename unknown>:0
at (wrapper remoting-invoke-with-check) System.IO.FileStream:.ctor (string,System.IO.FileMode,System.IO.FileAccess)
at mlv_view_sharp.MLVReader.SaveIndex () [0x00000] in <filename unknown>:0
at mlv_view_sharp.MLVReader.UpdateIndex () [0x00000] in <filename unknown>:0
at mlv_view_sharp.MLVReader..ctor (System.String fileName, mlv_view_sharp.MLVBlockHandler handler) [0x00000] in <filename unknown>:0
at mlv_view_sharp.MLVViewerForm.PlayFile () [0x00000] in <filename unknown>:0
at System.Threading.Thread.StartInternal () [0x00000] in <filename unknown>:0
edgar@linux-uoww:~/Software/MagicLantern>
I don´t find a .idx-file anywhere.
Edgar
thanks for testing.
about mono: is this directory writeable for your current user?
which dir? mono is /usr/bin/mono on opensuse an not writeable.
Edgar
/HOME/EDGAR/AUDIOTEST/M22-1707.IDX
Quote from: g3gg0 on December 22, 2013, 05:09:54 PM
/HOME/EDGAR/AUDIOTEST/M22-1707.IDX
/HOME/EDGAR/AUDIOTEST is writeable.
/HOME/EDGAR/AUDIOTEST/M22-1707.IDX does not exist.
Edgar
g3gg0, this is simply amazing :D
After some initial test (5d3, KB64 1000x, 1920x818 25fps)
-some dropped frames after unpacking with the newest mlv_dump (first time - over 40 out of ~400, second time - 1 out of 600)
This was with card spanning ON, i didnt notice. Will check with spanning OFF.
EDIT: dropped frames were because of spanning, with OFF is OK.
-sound out of mlv_dump seems to be slower than picture, it is pitched down a lot. Tried with 25fps, no fps override.
EDIT: after some further testing it is still an issue.
I hope that this long awaited feature will now trigger devs of RAWanizer, raw magic, batchelor and others to create some form of mlv_dump with GUI. Now it is time to switch to the new format :D
Thank you this is great!
Unfortunately I can't get the audio out yet as there is no new mlv_dump for mac. I also have the problem that I have to rename any IDX file (to IDX.bak for example) otherwise I get "Missing file header".
Quote from: kgv5 on December 22, 2013, 05:37:11 PM
-sound out of mlv_dump seems to be slower than picture, it is pitched down a lot. Tried with 25fps, no fps override.
sound is recorded using 44100 samples/sec. i cannot imagine what you mean with being slower than the video.
guess your fps setting was wrong ;)
Quote from: g3gg0 on December 22, 2013, 07:42:27 PM
sound is recorded using 44100 samples/sec. i cannot imagine what you mean with being slower than the video.
guess your fps setting was wrong ;)
I mean that after unpacking with mlv_dumb and importing dngs to NLE (after effects cs6 in my case) video stream has different lenght than sound stream. I checked import settings and it is 25 like in camera so this is not an issue. When i am playing back sound is "slower" - like 70-80 percent of video stream (which gives "lower" sound). But it can be corrected of course, have to figure out by how much. Maybe i am doing something wrong so i will make more testing.
But it is great, really, i am very happy, maybe i could keep my zoom H1 and not spend more money on H6 for now. Thanks
Quote from: g3gg0 on December 22, 2013, 07:42:27 PM
sound is recorded using 44100 samples/sec. i cannot imagine what you mean with being slower than the video.
guess your fps setting was wrong ;)
44.1khz is audio for cds/mp3s... music
video has 48khz audio.
Quote from: aaphotog on December 22, 2013, 08:06:12 PM
44.1khz is audio for cds/mp3s... music
video has 48khz audio.
nevertheless the .wav has set 44100 Hz and the audio data comes with 44100 samples per second.
i just double checked that. just play the .wav in any player and it plays correctly.
if you now import that and say "wow, hey ignore the 44100 Hz from WAV header and use 48000 Hz instead!" then its not my fault.
until there is a audio format setup screen, it is 44100 Hz.
hello.
soundrecording with canon 5d mk3.. whow.. great. thx so much.. will test it as soon as possilbe.
in the meantime, can someone please compile an updated mac version of the new mlv_dump..
thx. so much.
greets. sw
OSX Users:
i recently installed a OSX machine inside a virtual box.
after some struggle regarding installation of the necessary stuff i was able to compile mlv_dump in the VM.
can you try this (http://upload.g3gg0.de/pub_files/18ee8e20ac50501adcf36e07ef599279/mlv_dump.osx) mlv_dump?
update:
if you use OSX 10.6 or 10.7 does the link above work?
if not, can you try this (http://upload.g3gg0.de/pub_files/8705642032cded4e22a3051acb0ca870/mlv_dump.osx) mlv_dump?
Quote from: g3gg0 on December 22, 2013, 08:36:08 AM
please use a recent version of magic lantern.
you are using an outdated custom build.
There is none in the nightly builds for the 6D anymore, as you might know.
@NedB thank u so much
Audio, this is good. Thanks for all your work g3ggo.
Trying the mac binary but can,t get it to act the way the former binary acts. It says its a document file rather than a unixbased file like the former binary? Do I extract as usual? Only tried with an older .MLV file, not with recorded sound though.
By the way. If someone uploads a .MLV with audio @Baldand will start working on getting it extracted through his pythonbased rawviewer.
http://www.magiclantern.fm/forum/index.php?topic=9560.msg92659;topicseen#msg92659
g3ggo! Amazing! The audio is great and is totally in sync for a full 30 second test I just completed! This is so exciting to see the audio being developed for mlv_rec. Thank you again!
Here is some feedback and questions for you g3ggo or anyone else. (I am new to mlv_rec so I apologize ahead of time if the questions are already common knowledge)
1. Like a few people mentioned above... I also get a "missing header" error message when using mlv_dump on pc unless I rename the extension of the .IDX file. It also works if I just move the .IDX file to another directory.
2. The OSX version of mlv_dump file that you just posted didn't work for me. I think Danne is right in the post above where he mentions the file is not a unixbased file and maybe that is causing some issue.
3. When recording I loose the ML overlays and I also loose the 16:9 black bar crop on the screen. This might be a known issue, but I wasn't sure if I needed to set a setting to fix it. Also, when I stop the recording the ML overlays don't come back until I hit the preview button twice.
Thanks again for all the hard work! I really appreciate it. The addition of audio is going to make my life so much easier!
can you try to exec
chmod +x ./mlv_dump.osx
Thanks g3gg0, worked! Will try to extract audio now...
Worked, the regular command extracts both audio and dng,s with your mac binary. How wonderful!
Mac-instruction
1 - download g3gg0 binary from "Reply #891"
2 - Run this command in terminal in the directory of the binary(mlv_dump.osx)to "executes" chmod +x ./mlv_dump.osx
Then run the usual commands
1. don't use spaces
2. first cd into a directory ("cd <directory>" and note the SPACE after cd)
3. then execute ./mlv_dump --dng <filename>" (ENTER THE FILENAME MANUALLY WITHOUT ANY PATH)
4. if this works, try using drag'n'drop or whatever, but first try to get it executing the most simple way as described in 1. to 3.
Thanks g3gg0!
Quote from: g3gg0 on December 23, 2013, 01:12:46 AM
can you try to exec
chmod +x ./mlv_dump.osx
g3gg0,
Yes, that fixed it! I have it working on my OSX 10.8.5. I still had to rename the IDX extension so it would ignore it, but it has the same behavior on the pc.
Thanks for the tip!
Here was the command I used to create the dng's and audio file: ./mlv_dump.osx --dng M22-1413.MLV
Dear g3gg0,
Thanks for your great surprise before this christmas.
I really want to test the mlv_snd.
But i can't find it in the updated nighty bulid.
Regards,
Sean
Thanks, The mlv_dump.osx work for me too, it extract the wav file and the audio is sync, but the converted dng are color shifted to ugly green.
I use 5d MKIII
Quote from: crazypig on December 23, 2013, 03:07:32 AM
Dear g3gg0,
Thanks for your great surprise before this christmas.
I really want to test the mlv_snd.
But i can't find it in the updated nighty bulid.
Regards,
Sean
You can get mlv_snd from the first post in this thread. Just add it to your latest nightly build. Post #874 a few pages back has a good overview.
Quote from: rob_6 on December 23, 2013, 05:50:08 AM
You can get mlv_snd from the first post in this thread. Just add it to your latest nightly build. Post #874 a few pages back has a good overview.
thank you so much
Quote from: auxmar on December 23, 2013, 03:42:42 AM
Thanks, The mlv_dump.osx work for me too, it extract the wav file and the audio is sync, but the converted dng are color shifted to ugly green.
I use 5d MKIII
I get the green too. Once I bring it into After Effects I can adjust the Temperature and Tint to make it look right. The sliders for Temperature and Tint default to their minimum value which makes the green.
Tested the new audio function in MLV on a 6d.
Got a real audio wav file, normal size, normal specs.
But when I play the audio file, there's no audio at all ? Not even noise, it's like it's recording without any real audio input...
Tested several options in the ML audio menu, like sound record: separate wav, sound record: normal, Input source: internal mic, Input source auto...
And all these options where tested with MLV sound: ON.
Anybody else tested it on 6d ?
someone can upload the osx compiled version of mlv_dump please ?
i can't compile it from where i'am
Wow! sound record working on 550D so far, pretty cool!
Yep Audio working!!!!You are the best g3gg0.
OK I tested the MLV sound recording from this topic on 5DMKII. It appears to work and then I used the linked MLV converter (the latest one) for osx.
I managed to get it to spit out a some RAW files for video and some *.wav files for the audio. The files appear to be around the expected size.
I can't play these files though. I inspected the file with a hex editor and find that they are RIFF files. I don't seem to have any application on the Mac that can deal with RIFF. :(
Does anyone know of any Mac application that can either convert or directly deal with RIFF files?
Anyone else have any joy playing/editing the sound files on the Mac?
I'm on Mac OS X Mavericks 10.9.1
@g3ggo thank q so much g3gg0 n the magic lantern team for enabling audio recording
Thanks to g3gg0, sound works perfect with 5D3 on OS X Maverick.
For OS X users:
.MLV file and mlv_dump.osx have to be in the same directory, then convert with Terminal using this command:
./mlv_dump.osx --dng NameOfTheFile.MLV
Happy holidays shooting Raw with sound ;)
Seems to work now on 6D!
I had to turn OFF the FPS override function (It said so in a message at the bottom of the display, FPS override turns off most sound function settings... ::)
When I start recording I loose the overlays and I also loose the black mask bar on the top and bottom of the image. Is this the way it is supposed to behave with mlv_rec at this point in its development? I just want to make sure I have my settings right. Can someone please let me know... Thanks.
Quote from: rob_6 on December 23, 2013, 07:08:17 PM
When I start recording I loose the overlays and I also loose the black mask bar on the top and bottom of the image. Is this the way it is supposed to behave with mlv_rec at this point in its development? I just want to make sure I have my settings right. Can someone please let me know... Thanks.
Turn the Global Draw option On inside MLV it will keep it on instead of turning it off for performance if the performance is not needed, it seems to be Off by Default in MLV
This is fantastic news! Thank you.
I am shooting on a 5D3 and noticed that there is a 5D3_111.sym as well as autoexec.bin file in your mlv_rec folder in addition to the mlv_snd module.
Do I need to copy/replace this autoexec.bin to the root directory of my SD card?
Do I need to copy/relaoce the 5D3_111.sym from your mlv_rec folder into the modules folder of the SD card?
Thanks!
When I start recording I loose the overlays and I also loose the black mask bar on the top and bottom of the image. Is this the way it is supposed to behave with mlv_rec at this point in its development? I just want to make sure I have my settings right. Can someone please let me know... Thanks.
switch on "Disable G Draw" in Raw Video(MLV)
quick demonstration on work flow. u can sync audio easily. u can also cut in premiere pro cc. watch and see how easy it is
https://vimeo.com/82579655
Quote from: andy kh on December 23, 2013, 10:16:45 PM
quick demonstration on work flow. u can sync audio easily. u can also cut in premiere pro cc. watch and see how easy it is
https://vimeo.com/82579655
>We couldn't find that page.
>Please check to make sure you've typed the URL correctly.
>You may also want to search for what you are looking for.
quick demonstration on work flow. u can sync audio easily. u can also cut in premiere pro cc. watch and see how easy it is
https://vimeo.com/82579655
Quote from: g3gg0 on December 22, 2013, 07:42:27 PM
sound is recorded using 44100 samples/sec. i cannot imagine what you mean with being slower than the video.
guess your fps setting was wrong ;)
I still have problem with sync - tried 24 and 25 fps. Sound sounds slower, i mean sloooooower.
I used goldwave and changed wav playback rate from 44,1 to 48kzh and it helped, still not perfect but much better. I have recorded 1 minute clip, in 56-58 sec the lag was about 0,5 sec. 5d3, kb64 1000x. To use wav with resolve 10 it is required to change 44,1 to 48khz anyway.
EDIT: i can fix this problem in premiere cs6 with using Rate stretch tool on the sound clip and stretch it down to match the video layer.
Is anybody have the same problem?
Quote from: kgv5 on December 23, 2013, 10:47:34 PM
I still have problem with sync - tried 24 and 25 fps. Sound sounds slower, i mean sloooooower.
can you upload such .wav?
Hi all,
been trying for a few hours now to get this to work, but I'm a bit clueless ... and feeling bad posting for help with my low postcount :/
- On my 5Dmk3, I've tried with installing yesterday's as well as today's nightly, and activated filemanager, mlv-rec, and mlv-play, then enabled mlv-rec. records just fine.
- downloaded mlv-snd from the first post in this thread, activated it as well, and enabled mlv-snd. i get the big reddish audiobars fine when recording.
- on OSX Mavericks, i installed mlv_dump.osx, made it executable and ran the --dng command: it extracts the DNG's just perfectly, but i just do not manage to get an assorted wav file.
Any pointers to where i missed an obvious step?
Quote from: seku on December 23, 2013, 11:35:53 PM
Hi all,
been trying for a few hours now to get this to work, but I'm a bit clueless ... and feeling bad posting for help with my low postcount :/
- On my 5Dmk3, I've tried with installing yesterday's as well as today's nightly, and activated filemanager, mlv-rec, and mlv-play, then enabled mlv-rec. records just fine.
- downloaded mlv-snd from the first post in this thread, activated it as well, and enabled mlv-snd. i get the big reddish audiobars fine when recording.
- on OSX Mavericks, i installed mlv_dump.osx, made it executable and ran the --dng command: it extracts the DNG's just perfectly, but i just do not manage to get an assorted wav file.
Any pointers to where i missed an obvious step?
I am obviously not an expert with my low post count either, but you might need the mlv_record module from the first post of this thread rather than the nightly version. I use the mlv_rec from this thread and I have it working.
Quote from: andy kh on December 23, 2013, 10:10:04 PM
switch on "Disable G Draw" in Raw Video(MLV)
Thanks Andy kh! That worked.
Thanks rob_6 and andy kh :)
t'was the Disable G Draw, works perfectly now, thanks a lot !
With the latest build and MLV_SND, when I record Global Draw goes off and I can't re-enable it without rebooting. The option in the MLV module is OFF too.
Any idea?
With the latest build and MLV_SND, when I record Global Draw goes off and I can't re-enable it without rebooting. The option in the MLV module is OFF too.
Any idea?
it has to be ON
Quote from: andy kh on December 24, 2013, 12:07:16 AM
With the latest build and MLV_SND, when I record Global Draw goes off and I can't re-enable it without rebooting. The option in the MLV module is OFF too.
Any idea?
it has to be ON
Sort of counterintuitive, but it worked! Thanks.
I'd just like to remind everyone that PLURALEYES WILL WORK AGAIN!!! * head explodes *
Been doing many tests and now I only get the MLV files recorded to the card. The sound is still working great, but the IDX files are no longer being created. Anyone know what could be the reason? I tried formatting my cards an reinstalling ML files and that didn't change it. I don't have a use for the IDX files at the moment so I am not complaining, just curious. :)
I am having so much fun with this sound recording development. Its like the best Xmas gift ever! Thanks again g3gg0 and the ML team!
you are welcome ;)
i updated mlv_rec and mlv_snd.
those of you who had weird audio problems (e.g. not playable wav's) please try again.
that should be fixed now.
update:
added OS X binary and updated to include this (https://bitbucket.org/hudson/magic-lantern/pull-request/312/add-time-metadata-to-dngs-exported-from/diff) fix for DNG datetime
Quote from: g3gg0 on December 23, 2013, 11:08:25 PM
can you upload such .wav?
Sure. Here it is:
wav straight from mlv_dump
https://www.dropbox.com/s/mst85gtpzutwd7o/m24-0223_frame_straight%20from%20mlv_dump.wav
the same wav from goldwave - playback changed from 44,1 to 48khz - seems quite correct.
https://www.dropbox.com/s/wuihi0t2saondxp/m24-0223_frame_48khz%20from%20goldwave.wav
Hello g3gg0,
I downloading the recent nightly build of MLV Raw, but before I could get a chance to test the audio I ran into an error.
After recording a video in MLV it will always state "stopping..." and unless i shut off the camera and turn it back on, it will always state "stopping..."
Thanks g3gg0, this is invaluable. And thank you also for advancing the whole file format stuff in the first place.
Haven't tried the new build, but already the first build worked great on 5Dm3. Sadly I am still plagued by random crashes and not shutting down until pulling the battery when using an external monitor. But this also happens with the traditional raw_rec.mo
Hi G3gg0,
If it won't take too much of your time, is it possible to put in some more horizontal video resolution options in the MLV module?
In some raw.rec builds there where options for 1792 pixels wide and 1664 pixels wide resolution.
The 1792 resolution is probably only used for 6d owners, but I happen to have a 6d and love the 1792 pixel wide clips.
Further, THANKS for the audio recording in combination with raw video!
Works on my 6d (using the tragic lantern build of 30 November), audio clips are a little bit longer than video, but it seems in sync, it only records some more audio after video recording has stopped.
no after effects, no resolve only premiere pro cc to edit multiple mlv raw files with audio. i wil be uploading a tutorial in an hour.
We are waiting!!Thanks a lot
Quote from: andy kh on December 24, 2013, 12:19:20 PM
no after effects, no resolve only premiere pro cc to edit multiple mlv raw files with audio. i wil be uploading a tutorial in an hour.
I'd waiting on pins and needles...
MLV FTW!!! @g3gg0 4 President :) Amazing what u did on the audio side too man!!! Have a fine holidays session too!
How to monitor level with this?Can anybody help?I will post a workflow tutorial for Final Cut Pro.
It goes like this >mlv dump>convert dng's to Visionlog>compressor 4.1 to export Prores 4444 with audio>Final cut pro with LUT utility for grading.
Very good quality to my eyes
Quote from: g3gg0 on December 22, 2013, 01:02:45 AM
update:
- added mlv_snd which adds audio recording to mlv_rec
Great work g3gg0! I just tested this and it worked very well... I had perfect sync when post-processed. I only recorded 25 seconds so I'll try to record something much longer to test that sync is maintained for an extended period.
no after effects, no resolve only premiere pro cc to edit multiple mlv raw files with audio
https://vimeo.com/82623320
Quote from: g3gg0 on December 22, 2013, 10:25:36 PM
OSX Users:
i recently installed a OSX machine inside a virtual box.
after some struggle regarding installation of the necessary stuff i was able to compile mlv_dump in the VM.
can you try this (http://upload.g3gg0.de/pub_files/18ee8e20ac50501adcf36e07ef599279/mlv_dump.osx) mlv_dump?
update:
if you use OSX 10.6 or 10.7 does the link above work?
if not, can you try this (http://upload.g3gg0.de/pub_files/8705642032cded4e22a3051acb0ca870/mlv_dump.osx) mlv_dump?
I've tried this in terminal on both 10.7 and 10.9 on both I am getting a response of "command not found". Am I putting in the correct command? I'm not much of a terminal user...
Quote from: AnotherDave on December 24, 2013, 04:39:35 PM
I've tried this in terminal on both 10.7 and 10.9 on both I am getting a response of "command not found". Am I putting in the correct command? I'm not much of a terminal user...
BINGO! I read a little more and realized my mistake. chmod +x ./mlv_dump.osx needs to be executed first. Second one works well in 10.7. First one says 'illegal instruction 4'
Thanks.
When I attempt to open the dng in premiere pro cc, it says 'The file appears to have no media data'....
no after effects, no resolve only premiere pro cc to edit multiple mlv raw files with audio
https://vimeo.com/82623320
When I attempt to open the dng in premiere pro cc, it says 'The file appears to have no media data
you can open cdng but not dng in premiere pro cc. watch my video how i open cdng in premire pro cc
Quote from: andy kh on December 24, 2013, 05:22:13 PM
no after effects, no resolve only premiere pro cc to edit multiple mlv raw files with audio
https://vimeo.com/82623320
this actually makes the process seem far more complex than just converting dngs in Resolve...
Quote from: AnotherDave on December 24, 2013, 06:03:21 PM
this actually makes the process seem far more complex than just converting dngs in Resolve...
AnotherDave,
Do you know of a way to link up the audio in Resolve automatically based on filename? I can align the audio manually in the timeline, but I wonder if there is a way to align the audio for the clips automatically by name.
If the developers can find a way to encode the metadata the right way, Resolve should be able to recognize it right off the bat. Unfortunately, when I open my clips they don't seem to have any of the metadata stored... I'm processing through mlv.dump.osx for 10.7...
Has anyone tested a long clip of a more thane a couple minutes? I have only tried short clips so far and they seem in sync. Are longer clips still in sync?
-m question...
What file extension should we be outputting the metadata to? What can read this metadata? Is the audio track information being encoded on the metadata track? Is this information used when processing to .dng, or does additional software need to be written to re-encorporate it?
Thanks!
Excellent work! I just started testing the new mlv-sound.mo and the MLV viewer.
So far the MLVBrowser exports (wav + dng) the dng and the wav. The sound seem good but the dng have green cast.
If I export to RAW + wav and then use raw2dng then the dngs are OK!
Finally my movies will not be muted. Thank you very much.
I will be testing this further as I have time....
Merry Chiristmas and thank you again.
Quote from: rob_6 on December 24, 2013, 09:08:43 PM
Has anyone tested a long clip of a more thane a couple minutes? I have only tried short clips so far and they seem in sync. Are longer clips still in sync?
yeah longer clips in sync
g3gg0, is there any possibility to implement 44,1 to 48khz conversion in MLV browser?
Resolve 10 dont accept 44,1, sure we can do this one by one by hand but the workflow would be much faster with ready 48khz wav out of mlv_dump. What do you think?
Cheers
Quote from: andy kh on December 24, 2013, 11:16:10 PM
yeah longer clips in sync
Hi, i still have problems with "lowered" and "slowed down" sound in my wav files (5d3). If you have everything synced all the time could you please tell me your settings in mlv tab? Thanks
Quote from: kgv5 on December 25, 2013, 01:19:06 AM
Hi, i still have problems with "lowered" and "slowed down" sound in my wav files (5d3). If you have everything synced all the time could you please tell me your settings in mlv tab? Thanks
yeah i hav done so many test. long clips short clips and everything works fine in premiere pro cc. setting in camera is same in all camera.
Quote from: andy kh on December 25, 2013, 07:23:12 AM
yeah i hav done so many test. long clips short clips and everything works fine in premiere pro cc. setting in camera is same in all camera.
Got it. If you just click on wav and open it in say windows media player - does it play normal ? Could you please check one of my previous post in this thread with dropbox links and see those two short clips and the diffetence between them? Thanks
Quote from: kgv5 on December 25, 2013, 08:22:15 AM
Got it. If you just click on wav and open it in say windows media player - does it play normal ? Could you please check one of my previous post in this thread with dropbox links and see those two short clips and the diffetence between them? Thanks
i got u. there used to be problems sometimes. that problem is no more with yesterday's nightly build. g3ggo has already fixed it.
Sampling rate selection will be added soon.
dont know if it is possible to select 48khz in camera.
Quote from: g3gg0 on December 25, 2013, 11:09:07 AM
Sampling rate selection will be added soon.
dont know if it is possible to select 48khz in camera.
Thats great, i tested many times, also with yesterday nightly build and of course newest mlv rec pre built. Still having the same problem. Converting playback rate (not just resampling) from 44,1 to 48khz in goldwave makes wav sounds correct (it shortens it a little bit). Strange thing is that nobody else is having such a problem...
48K will be great if it's possible. I am impressed with the audio recording now. It now makes 'run and gun' type filming a much more realistic possibility. The audio thing is a game changer. Again. Thanks for all the work. It's much appreciated. :)
Can we have 24-bit too? (only joking! ;) :D)
anyone recording more then 1min on mlv with audio ?
i keep getting frame skip around the sametime
Did you try setting still image quality to a low jpg setting and rawphoto off? WIth this I get continuous 25fps 1920x1080 with audio.
Quote from: Jakobmen on December 25, 2013, 07:52:50 PM
i keep getting frame skip around the sametime
video or audio frame skip?
Here's a quick bash script for anyone wishing to run the OS X edition... move MLV_DUMP.OSX to /usr/bin and rename to MLV_DUMP then run CHMOD +X /usr/bin/MLV_DUMP
Save the script as dumper.sh in the same folder as the MLV files ... I usually copy everything to my desktop. Usage of the script: ./dumper.sh <input.mlv> <folder name>
mkdir $2
mv $1 ./$2
cd ./$2
/usr/bin/mlv_dump -o $1 --dng $1
update:
- mlv_snd now allows to select desired sampling rate (48000, 44100, 22050, 11025, 8000)
Quote from: dubzeebass on December 25, 2013, 09:21:42 PM
Here's a quick bash script for anyone wishing to run the OS X edition... move MLV_DUMP.OSX to /usr/bin and rename to MLV_DUMP then run CHMOD +X /usr/binMLV_DUMP.
Save the script as dumper.sh in the same folder as the MLV files ... I usually copy everything to my desktop. Usage of the script: ./dumper.sh <input.mlv> <folder name>
mkdir $2
mv $1 ./$2
cd ./$2
/usr/bin/mlv_dump -o $1 --dng $1
not quite correct.
this:
> move 'mlv_dump.osx' to /usr/bin and rename to 'mlv_dump' then run 'chmod +x /usr/bin/mlv_dump'
thanks dude, forgot the /. i'm trying to figure out how to list all .mlv files to a text file then load each line, call the file the original file name, and export to a directory with the file name before ".MLV"
will report back.
Quote from: dubzeebass on December 25, 2013, 09:32:23 PM
thanks dude, forgot the /. i'm trying to figure out how to list all .mlv files to a text file then load each line, call the file the original file name, and export to a directory with the file name before ".MLV"
will report back.
i can do it in windows
Quote from: dubzeebass on December 25, 2013, 09:32:23 PM
thanks dude, forgot the /. i'm trying to figure out how to list all .mlv files to a text file then load each line, call the file the original file name, and export to a directory with the file name before ".MLV"
will report back.
for FILE in `ls -A1 *.MLV *.mlv 2>/dev/null`; do
BASE=`echo $FILE | cut -d "." -f1`;
mkdir $BASE;
/usr/bin/mlv_dump --dng -o ${BASE}/frame_;
done
Quote from: g3gg0 on December 25, 2013, 09:27:26 PM
update:
- mlv_snd now allows to select desired sampling rate (48000, 44100, 22050, 11025, 8000)
Perfect!
Any more settings to try I stop recording after 1 min from frame skip?
what setting do you have now? Tried 24fps?
Yes 23.976 fps.. I know I can record more then 1 min with normal raw.rec
g3gg0,
When I try to record in 3x crop mode and when I set the FPS override to 23.976 it currently disables the audio. Is there a way to have prevent the audio from being disabled now that we have have the mlv_snd module?
g3gg0, 48khz solved my problem, now sound has a perfect sync :D
I experienced some sound frame skips though, recorded over 1 minute clip 1920x1080 25fps and after 37sec sound lost its sync completelly. I will make some further testing.
Quote from: rob_6 on December 25, 2013, 10:28:55 PM
g3gg0,
When I try to record in 3x crop mode and when I set the FPS override to 23.976 it currently disables the audio. Is there a way to have prevent the audio from being disabled now that we have have the mlv_snd module?
Are you sure the audio is disabled?
The reason I say this is because when you have the FPS override turned on there is a message saying that the audio is disabled. I would record regardless of this message and just check and see if you have recorded an audio file when you extract it.
Just try it. You might be pleasantly surprised (as I was) to see that it actually does record it! ;)
If that's not the case then I really can't help you and sorry! but Merry Christmas anyway :)
Quote from: g3gg0 on December 25, 2013, 09:42:35 PM
for FILE in `ls -A1 *.MLV *.mlv 2>/dev/null`; do
BASE=`echo $FILE | cut -d "." -f1`;
mkdir $BASE;
/usr/bin/mlv_dump --dng -o ${BASE}/frame_;
done
Not working ... do I need to feed in args or is it just a batch? It makes the directory but nothing happens after that, just shows me the mlv_dump options.
sorry, missed a parameter. try this
for FILE in `ls -A1 *.MLV *.mlv 2>/dev/null`; do
BASE=`echo $FILE | cut -d "." -f1`;
mkdir $BASE;
/usr/bin/mlv_dump --dng -o ${BASE}/frame_ $FILE;
done
I got it working like this ...
for FILE in `ls -A1 *.MLV *.mlv 2>/dev/null`; do
BASE=`echo $FILE | cut -d "." -f1`;
mkdir $BASE;
mv ./$FILE ./$BASE
cd ./$BASE
/usr/bin/mlv_dump --dng -o $BASE $FILE
cd ..
done
BAH!! anything under 1920 x 960 records fine over 2mins bah.. lol
come on... cant you be ANY MORE DETAILED?!
"does not work bah" isnt something i could fix.
also i dont know how to reproduce that.
plus you could have tested it with different settings, with/without sound etc.
Last post, mad respect to g3gg0 for the help....
Do the following ...
1. Download mlv_dump.osx
2. Open a Terminal
3. Go to the folder where you downloaded mlv_dump.osx (extract if needed) and type: sudo mv ./mlv_dump.osx /usr/bin/mlv_dump
4. Type sudo chmod +x /usr/bin/mlv_dump
5. Make a file in the folder where you've moved the MLV files, save as recursive.sh, with contents:
for FILE in `ls -A1 *.MLV *.mlv 2>/dev/null`; do
BASE=`echo $FILE | cut -d "." -f1`;
mkdir $BASE;
mv ./$FILE ./$BASE
cd ./$BASE
/usr/bin/mlv_dump --dng -o frame_ $FILE
/usr/bin/mlv_dump -m -o meta.mlv $FILE
/usr/bin/mlv_dump -v meta.mlv >$FILE.txt
rm ./meta.mlv
rm ./meta.mlv.wav
cd ..
done
6. Run recursive.sh from the Terminal.
This will extract all the frames from the MLV files and extract MLV metadata into a TXT file.
7. Profit!
Quote from: mageye on December 25, 2013, 10:43:59 PM
Are you sure the audio is disabled?
The reason I say this is because when you have the FPS override turned on there is a message saying that the audio is disabled. I would record regardless of this message and just check and see if you have recorded an audio file when you extract it.
Just try it. You might be pleasantly surprised (as I was) to see that it actually does record it! ;)
If that's not the case then I really can't help you and sorry! but Merry Christmas anyway :)
Actually you are right. It does record the sound. I just assumed that the sound wasn't coming through because the sound meters disappeared and in the menu it says the sound will be disable when FPS override is activated.
Maybe there is a way to prevent the sound meters from being disabled when FPS override is active? Not a big deal, but would be nice. :)
g3gg0,
48khz update works perfectly! I no longer need to convert the sound so that it can be read by Resolve. Thanks for the update! On Christmas even! Merry Christmas to you and everyone else.
Quote from: rob_6 on December 25, 2013, 11:17:45 PM
Merry Christmas to you and everyone else.
Hear hear!
Quote from: dubzeebass on December 25, 2013, 11:12:27 PM
Last post, mad respect to g3gg0 for the help....
Do the following ...
1. Download mlv_dump.osx
2. Open a Terminal
3. Go to the folder where you downloaded mlv_dump.osx (extract if needed) and type: sudo mv ./mlv_dump.osx /usr/bin/mlv_dump
4. Type sudo chmod +x /usr/bin/mlv_dump
5. Make a file in the folder where you've moved the MLV files, save as recursive.sh, with contents:
for FILE in `ls -A1 *.MLV *.mlv 2>/dev/null`; do
BASE=`echo $FILE | cut -d "." -f1`;
mkdir $BASE;
mv ./$FILE ./$BASE
cd ./$BASE
/usr/bin/mlv_dump --dng -o frame_ $FILE
/usr/bin/mlv_dump -m -o meta.mlv $FILE
/usr/bin/mlv_dump -v meta.mlv >$FILE.txt
rm ./meta.mlv
cd ..
done
6. Run recursive.sh from the Terminal.
This will extract all the frames from the MLV files and extract MLV metadata into a TXT file.
7. Profit!
dubzeebass!
Thanks for putting this together this great script. So great for converting a bunch of files. It works perfectly! Thanks so much. Thanks to g3gg0 again too for the help!
Cheers, glad to hear! You could also add it to /usr/bin and always have it handy then since it's in the path.
:)
Glad to contribute!
haha, that script from dubzeebass totally nailed it :)
Thanks a lot!
Strange though I get 2 wav files from each conversion? One called meta.mlv.wav and the other one simply called frame.wav
with out Snd it still skips frame at 1min
most settings are off
preview is auto
disable G draw is on
any Resolution under 1920 x 960 works fine with sound
i thought i had MLV recording doing full 1920x1080 before..
i have tried all kinds of number settings for "Buffer fell method" and "CF-only buffers"
this is on a KB 64gb 1000x card
any pointers?
Quote from: Danne on December 25, 2013, 11:49:15 PM
haha, that script from dubzeebass totally nailed it :)
Thanks a lot!
Strange though I get 2 wav files from each conversion? One called meta.mlv.wav and the other one simply called frame.wav
rm ./meta.mlv
rm ./meta.mlv.wav cd ..
Add the bolded line into the script.
Updated my post with the 2ndary WAV removal.
Worked nicely :)
Can I ask you how to execute the recursive.sh script.
I copy pasted the scripttext to text editor and the saved it and changed the suffix to .sh
How do I execute this?
Thanks for helping
/D
From the terminal... I'll make a video when I get home next week.
Quote from: Jakobmen on December 25, 2013, 10:01:53 PM
Any more settings to try I stop recording after 1 min from frame skip?
I can repoduce the skip.
It seems to happen at 4096MB. Suddenly the buffer fills up and a frame is skipped. Before about 1/5 to 1/4 of the buffer is used. Nevertheless I observe a similar behavior when the recording starts. First the buffer fills up to almost 100% then writig catches up and the buffer emptys. I assume there is something happening while beginning the second chunk and writing metadata?
5DMKIII (Fill Method: 3 CF-only buffers: 8)
Nightly build December 25th.
Nightly build December 26th.
Quote from: dubzeebass on December 26, 2013, 12:33:47 AM
From the terminal... I'll make a video when I get home next week.
Ok.
I modified a command script from a member here, jerrykil, and put in your script-instruction dubzeebass and g3gg0. So if doubleclicking the commandscript from within the folder with the mlv,s it starts the conversion without having to go through terminal.
Instruction would be:
1. Download mlv_dump.osx http://upload.g3gg0.de/pub_files/c02de9a0b17fc909b5494ddbd0fd249e/mlv_rec.zip (http://upload.g3gg0.de/pub_files/c02de9a0b17fc909b5494ddbd0fd249e/mlv_rec.zip)
2. Open a Terminal
3. Go to the folder where you downloaded mlv_dump.osx (extract if needed) and type: sudo mv ./mlv_dump.osx /usr/bin/mlv_dump
4. Type sudo chmod +x /usr/bin/mlv_dump
5. Run the recursive.command file below from the same folder as the .mlv files. Need admin rights. Go to command script through terminal and write
chmod +rxw recursive.command press enter
6. run the recursive.command by doubleclicking
You can doubleclick the recursive.command script a couple of times to have them work in parallell.
commandscript https://drive.google.com/file/d/0B4tCJMlOYfird0U5dTNnR0twNE0/edit?usp=sharing
Thanks
/D
*updated recursive.command 29 dec (g3gg0)
I've re-uploaded it with the attributes fixed as some people may have had trouble due to ownership and administrative settings. Cleared all attribs with Batchmod.
https://drive.google.com/file/d/0B7VaNSfNjUYOT2s0NXp3cXA0UVU/edit?usp=sharing
I recorded 1:30 and this time no dropped frames at all. Sound perfectly synced till the end of a clip. i dont know why i had dropped frames before, both times 1920x1080 25fps. This time however i formatted my card to exfat, before was FAT32. In both cases MLV file was splitted to 4 GB and the rest. I think i was able to record all card with single file with raw-rec. Does Mlv-rec records files over 4GB?
Quote from: dubzeebass on December 26, 2013, 12:53:59 AM
I've re-uploaded it with the attributes fixed as some people may have had trouble due to ownership and administrative settings. Cleared all attribs with Batchmod.
https://drive.google.com/file/d/0B7VaNSfNjUYOT2s0NXp3cXA0UVU/edit?usp=sharing
Tried the reuploaded fix but got problem with adminrights from that one too. Updated my modified instruction. Good work on this dubzeebass
Quote from: kgv5 on December 26, 2013, 01:01:55 AM
I recorded 1:30 and this time no dropped frames at all. Sound perfectly synced till the end of a clip. i dont know why i had dropped frames before, both times 1920x1080 25fps. This time however i formatted my card to exfat, before was FAT32. In both cases MLV file was splitted to 4 GB and the rest. I think i was able to record all card with single file with raw-rec. Does Mlv-rec records files over 4GB?
Which buffer settings do you use for MLV 1080P?
Quote from: naturalsound on December 26, 2013, 01:19:08 AM
Which buffer settings do you use for MLV 1080P?
I made every tests with CF only buffers=0 and buffer fill method=0 so both OFF. Card spanning also OFF.
Quote from: kgv5 on December 26, 2013, 01:24:04 AM
I made every tests with CF only buffers=0 and buffer fill method=0 so both OFF. Card spanning also OFF.
skips again at 4095MB! buffers: 0 fill: 0, spanning OFF (Global Draw does not matter)
I have also problem with the mlv_rec, it does lot of skipping frames with or without sound, especially the first time i try to record. i have tried various setting but, still skipping, the max time a get is about 1 minute. with Raw_rec it record continuously without skip most of the time.
i have the KB 64G 1000X with 5d mk III, the bitrate when it record mlv is around 80, with Raw_rec it's around 90 .
can someone share his setting of MLV to record continuously on MK III ?
Hi and Merry Christmas! Running last night's nightly build on a 5D MkII.
However when I try to load the mlv_dump module, I get a "file err" and "ret 0x", not allowing it to load. Does anyone know what's going on and how I can fix it? Thanks!
Also, the first post of this thread is extremely confusing for a MagicLantern newbie like me... Is there a comprehensive full guide/post that could explain clearly the steps to get mlv_dump working from shooting to processing? I'd really appreciate it, even willing to pay through PayPal! (The bit-depth/file-size reduction is so tempting)
People one more question.Do I have to copy autoexec.bin from the first post aswell?I got it working with 25 dec build but i dod not copy the auto exec.bin.Am i missing something?
I have all files from mlv-rec package on my card, including autoexec.
On the 6d, every first clip, audio is recorded good.
Every clip after it, only has a very small audio file (just as long as the sync beep file would be), it contains normal audio though, but it's only 1 or 2 seconds long.
And it says "audio failed to stop, state 4".
When I turn the camera off and on, the first clip has normal audio, the next clips have the state 4 error message and a 1 or 2 seconds audio file.
Is there something in the ML menu that I need to turn on or off ?
I'm using the new MLV record and sound modules in tragic lantern build from 30 November, which is the most recent I can find for 6d.
There isn't a nightly build for 6d (does anyone know why there are no nightly builds for 6d ?)
I'm shooting on a 5D3 with 64gb 1000x KB cards. I shot about 1:30 of mlv_raw w/audio and got some stuff I hadn't seen before... On the card, I have 3 files M26-0833.mlv, M26-0833.M00, and M26-0833.M01. Why did it span my shot across multiple clips? How do I combine them?
I am using the latest MLV_REC.mo and have never seen this before...
Quote from: AnotherDave on December 26, 2013, 02:19:44 PM
I'm shooting on a 5D3 with 64gb 1000x KB cards. I shot about 1:30 of mlv_raw w/audio and got some stuff I hadn't seen before... On the card, I have 3 files M26-0833.mlv, M26-0833.M00, and M26-0833.M01. Why did it span my shot across multiple clips? How do I combine them?
I am using the latest MLV_REC.mo and have never seen this before...
NVM it's automatic... Why did it span though?
Quote from: g3gg0 on July 15, 2013, 10:58:23 PM
some technical facts:
- structured format
- extensible layout
- as a consequence, we can start with the minimal subset (file header, raw info and then video frames)
- multi-file support (4 GiB splitting is enforced)
- spanning suport (write to CF and SD in parallel to gain 20MiB/s)
- out-of-order data support (frames are written some random order, depending on which memory slot is free)
ok my problem is happening after the 4gb limit is reach it starts the next file .m00 it gives me a frame skip .. tried and few settings and it happens the same way all the time..
is the buffer already quite full when the next chunk is created?
Right when the next chunk it starts to get full then skips
problem is, i canot create another chunk how it was done in raw_rec.
alex just wrote and check how much was written.
if the write failed because of a 4G limit, then he writes the rest into another chunk.
i cant do this because the file format doesnt allow that.
i must know in advance if your setup (camera + card) allows files >4G.
maybe i can add an option "Files >4G supported" for those who know that they have exfat and a 5D3.
but this will cause for sure 10 other people asking why it doesnt work and that the option is faulty etc etc.
then there is again a lot of crying here. oh dear...
Yup my cards are exfat also
Quote from: g3gg0 on December 26, 2013, 04:25:54 PM
maybe i can add an option "Files >4G supported" for those who know that they have exfat and a 5D3.
but this will cause for sure 10 other people asking why it doesnt work and that the option is faulty etc etc.
then there is again a lot of crying here. oh dear...
I understand your concerns, g3gg0, but I would really appreciate to have that option. For me it's much more convenient having one single .mlv file per shot. People using this module (or ML by extension) should know their cards file system and its limitations.
But that isn't important. We finally have synchronous audio and I cannot thank you enough. I've been working with the 'beep' and the slate method and it was quite painful. This makes things so much easier and quicker. For those doing their dailies in Resolve and struggling, there's an amazing tutorial from Abelcine (http://www.youtube.com/watch?v=GRKoJuRjajE)(I know I should be posting this link in the postproduction section but it's much more likely to be seen here). Right now sync in Resolve can only be done manually, in order for it to do it automatically the video and audio need to have matching time codes. Anyway, manual sync it's a minor inconvenient for this awesome format. Thanks again!
Hi, how about, naming it 5dmk3-exfat-only-mode ? So everybody knows what and how and so on... Thanks for your work and reading.
Merry ... Matthias
One thing after such change is to remember not to format card in camera anymore. I didnt know that formatting in camera makes card FAT32 even if it was formatted as exfat before. So only deleting clips via browser. If someone forget this - well - he can have much trouble unless has a laptop with him.
a option to delete all in folder..
agreed on the exFAT part. and ML, you're making dreams come true !
i'd never have thought to have the possibility, but i just got a few lexar cards, and a 1TB scratch SSD for the dng files for editing... i can try a RAW workflow ...this holiday trip shall be a lot funnier than i initially thought!
I compiled myself for 5D3 including a "make clean" and make for the new modules. For some reason my modules do not always get compiled when something new gets updated.
Tested "as is" with the camera settings as have been saved by g3goo. Great results all audio is pretty well matched. I get better alignment if I move the audio forward by 1 frame.
Changed setting to Preview=Hacked and also enabled Mrmory Hack, and card spanning ON.
Tested with Card spanning ON at 1920x1080 at 29.97 fps and the audio matched again!
Tested with Card spanning ON cropped video at 1920x1080 at 29.97 and had to slide audio forward by many frames and then to shrink the audio a bit. After that it matched.
Tested with Card spanning ON 60 fps at 1600x560 the audio could not be synched by trimming or stretching or sliding, the audio was there but could not align.
Tested with Card spanning ON Tested 50 fps at 1856x650 the audio could not be synched by trimming or stretching or sliding, the audio was there but could not align.
For Post processing I always used MLVBrowseSharp and exported to DNG (with green cast) then I imported the dng and wav file to Vegas.
Noticed that the green cast on dng is not due to BlackLevels. It appears that Color Temperature and Tint information is missing. As soon as I set in ACR Temperature to 5000 and tint to 0 the colors seem appropriate.
Quote from: kgv5 on December 25, 2013, 01:19:06 AM
Hi, i still have problems with "lowered" and "slowed down" sound in my wav files (5d3). If you have everything synced all the time could you please tell me your settings in mlv tab? Thanks
Filmed some clips during the holidays and now I am editing I hear that my voice and other male voices do sound lower and slower (than they should be) and my wife and kids voices actually sounds normal ?
Could it be that there is something going on with the lower frequencies ?
I had the same problem. Newest build resolves this, just make sure sound recording is set to 48khz in the sound menu. This should help. Cheers
Sound recording! Yes!
I tested 30 seconds with 48k + 23,976fps + 1920x1080p, perfect sync so far :D
It was quite a struggle getting the .mlv files to work on a mac, but thanks to Danne's post I got it working.
Does anybody have a program or developing such as rawmagic with support for mlv?
The newest MLV browser has ability to unpack mlv to dng with wav or raw with wav. Right click on the file icon.
Where can I download the mlv browser? Can't seem to find it.
See reply 101-111
http://www.magiclantern.fm/forum/index.php?topic=8447.100
It,s a work in progress. g3gg0 hasn,t implemented audio export on mac yet I think.
Another work in progress is this fine viewer
http://www.magiclantern.fm/forum/index.php?topic=9560.msg91651;topicseen#msg91651
Quote from: kgv5 on December 27, 2013, 02:09:09 PM
I had the same problem. Newest build resolves this, just make sure sound recording is set to 48khz in the sound menu. This should help. Cheers
Thanks will check the newest build.
I see at the bottom of your posts that you also have 6d, do you use the new sound module and new MLV-rec module with it, or did you only test it on 5d mark 3?
Quote from: Levas on December 27, 2013, 08:11:11 PM
Thanks will check the newest build.
I see at the bottom of your posts that you also have 6d, do you use the new sound module and new MLV-rec module with it, or did you only test it on 5d mark 3?
No, currently tested only on 5d3.
Tested longer clips on 5D3.
1920x1080 at 29.97 fps
Preview: Hacked
Memory Hack: ON
Extra Hack: ON
Buffer: 4:3
Card Spanning: ON
Global Draw ON but nothing selected ( I use this to get proper framing without much speed lost)
Recorded for 3 min and audio was in synch. about 20,7 GB Recording stopped automatically on frame skip
Tried the same on cropped video.
Once recorded for 4 min and the audio was OK untill 2 min 10 sec. After which point the audio skipped ahead, it seems that the ordering of the frames was lost.
Same thing happened on second recording for 3.5 minutes. The audio skipped ahead at 1 min 50 sec.
The Sound recording is amazing!
Davinci Resolve have no problem to recognize 48khz, thanks g3gg0 !
My workflow: MLVBrowseSharp --- raw2cdng.1.4.8 --- dropping each WAV into his DNG's file --- premiere pro cc ---- davinci resolve 10 lite.
But when I am exporting the final timeline as XML from premiere pro cc to davinci resolve 10, I'm not getting any signal on the sound tracks (I do get all the tracks though but mute). Anyone have same experience?
More details of sound skipping during cropped video:
I counted the seconds on the recording information and then listened to hear how the numbers are scrambled:
From beginning to second 92 is OK, then it starts counting on second 152 until 211, then it skips to 231 and continues counting until the 240 (end of recording), then it picks ups from 93 till 151, then it picks from 132 until 151.
1....92
152...211
231....240
93...151
So yes definitely the audio frames are being mixed up in cropped recording.
Quote from: Eran on December 28, 2013, 02:58:05 AM
The Sound recording is amazing!
Davinci Resolve have no problem to recognize 48khz, thanks g3gg0 !
My workflow: MLVBrowseSharp --- raw2cdng.1.4.8 --- dropping each WAV into his DNG's file --- premiere pro cc ---- davinci resolve 10 lite.
But when I am exporting the final timeline as XML from premiere pro cc to davinci resolve 10, I'm not getting any signal on the sound tracks (I do get all the tracks though but mute). Anyone have same experience?
everything works fine for me in resolve 10. make sure u dont mute ur sound track in premiere pro n resolve
Quote from: RenatoPhoto on December 28, 2013, 03:03:50 AM
So yes definitely the audio frames are being mixed up in cropped recording.
can you create an index before dumping?
(run mlv_dump with -x first)
Quote from: g3gg0 on December 28, 2013, 10:55:45 AM
can you create an index before dumping?
(run mlv_dump with -x first)
Yes it creates, with MLVSharpBrowser it gives the same results.
With mlv_dump I get the following results:
F:\0-Videos\14-test\sound\cropped>mlv_dump -x M27
MLV Dumper v1.0
-----------------
[i] Mode of operation:
- Input MLV file: 'M27-1927.MLV'
- Verify file structure
- Output .idx file for faster processing
[i] File M27-1927.MLV opened
[i] File M27-1927.M00 opened
[i] File M27-1927.M01 opened
[i] File M27-1927.M02 opened
[i] File M27-1927.M03 opened
[i] File M27-1927.M04 opened
[i] File M27-1927.M05 opened
[i] Processing...
[i] Reached end of chunk 0/7 after 2422 blocks
[i] Reached end of chunk 1/7 after 3167 blocks
[i] Reached end of chunk 2/7 after 2393 blocks
[i] Reached end of chunk 3/7 after 2490 blocks
[i] Reached end of chunk 4/7 after 2484 blocks
[i] Reached end of chunk 5/7 after 2389 blocks
[i] Reached end of chunk 6/7 after 788 blocks
[i] Processed 7215 video frames
[i] XREF table contains 16133 entries
[i] File M27-1927.IDX opened for writing
[i] Done
F:\0-Videos\14-test\sound\cropped>mlv_dump --dng
MLV Dumper v1.0
-----------------
[i] Mode of operation:
- Input MLV file: 'M27-1927.MLV'
- Convert to DNG frames
- Output into 'M27-1927_frame_'
[i] File M27-1927.MLV opened
[i] File M27-1927.M00 opened
[i] File M27-1927.M01 opened
[i] File M27-1927.M02 opened
[i] File M27-1927.M03 opened
[i] File M27-1927.M04 opened
[i] File M27-1927.M05 opened
[i] File M27-1927.IDX opened (XREF)
[i] XREF table contains 16133 entries
[i] Processing...
Error: GUID within the file chunks mismatch!
[i] Processed 0 video frames
[i] Done
What's the point in building the index first?
all blocks are save in some random order, depending on the current buffer usage.
the recoding module picks the largest block to write so you get maximum write rate.
when saving .raw or single .dng with mlv_dump, it doesnt matter in which order they are written.
but the .wav must (currently) be written in the correct order.
for parsing the blocks in the correct order, you need an index file that can be created using the -x option.
all subsequent mlv_dump executions will use the index for correct processing (in terms of time order)
In case you did not see it, I get the following error:
Error: GUID within the file chunks mismatch!
First I did the mlv_dump -x
Then I used mlv_dump --dng
@g3gg0: During one of my first recordings with mlv_snd, my 550D camera locked up (got a "busy" text, sorry I don't remember the exact wording) and apparently the sound file didn't get written properly: I get a "Received AUDF without WAVI, the .wav file might be corrupt" message after all the .dng's are (correctly, it seems) extracted.
Despite this, the .dng's look fine and the length of the resulting .wav file is roughly what one would expect for a 40 second (at 25fps) clip. Of course, it is corrupt, as its metadata indicates "MPEG AAC Audio at 7350 Hz, 28kb/s" instead of the correct "PCM S16 LE Stereo 44100 Hz, 16bps (from VLC player)" as on my other clips.
Question: is it possible to rescue this .wav clip by inserting a WAVI block into the .mlv file, or by hex-editing the .wav itself to correct the header? [Update: SOLVED! See two posts down]
In any case, thanks again for raw with sound, it's a quantum leap in usability and enjoyment! Cheers.
To all ML users trying out raw with sound: I was shooting a scene with my 550D, using mlv_raw and mlv_snd. One of the clips turned out to have an ugly greenish-black tint over all the .dng's, even though I didn't consciously change any settings between the various clips of the same scene. After I tried and failed to correct this in ACR (Photoshop/After Effects), I thought of using a more scientific method. Using Phil Harvey's exiftool (Google it), I compared the metadata of a .dng from a "good" clip to the metadata of one of these "green-black" .dng's. The only difference I noticed was in the data tagged "Exif:BlackLevel", which had a value of 2046 in the "good" .dng but 2509 in the "bad" clip.
I used exiftoolGUI (Google it) to change the value of "Exif:BlackLevel" on all the "bad" .dng's to 2046, and the problem was solved: the .dng's now looked exactly like ones from the other clips I shot of this scene.
Again, I don't know whether or not I did something to cause this error, or whether it will just crop up from time to time. But if you have this symptom (.dng's with a crushed-black/greenish tint), give it a try. Cheers!
@g3gg0 (again): Again the principle of "if you formulate a question clearly enough, you almost have your answer" applies. Pasted the header (Addresses 00000000 - 0000002B) of one of the "good" audio files over the same addresses in the "bad" audio file and it worked. FYI for anyone with a similar problem (shooting with mlv_rec and mlv_snd and you have a crash or skipped frames, resulting in an error message about a possible corrupt .wav file). Cheers and I hope TLDR didn't apply too much...
the latest binaries should not need mlv_dump -x anymore.
(although it should work. can you check if you still get such GUID error?)
Updated batchscript command for mac-users(30dec 2013) Thanks to dubzeebass, g3gg0, daisermac, jerrykil
Ok.
I modified a command script from a member here, jerrykil, and put in your script-instruction dubzeebass and g3gg0. So if doubleclicking the commandscript from within the folder with the mlv,s it starts the conversion without having to go through terminal.
Instruction would be:
1. Download mlv_dump.osx http://upload.g3gg0.de/pub_files/c02de9a0b17fc909b5494ddbd0fd249e/mlv_rec.zip
2. Open a Terminal
3. Go to the folder where you downloaded mlv_dump.osx (extract if needed) and type: sudo mv ./mlv_dump.osx /usr/bin/mlv_dump
4. Type sudo chmod +x /usr/bin/mlv_dump
5. Run the recursive.command file below from the same folder as the .mlv files. Need admin rights. Go to command script through terminal and write chmod +rxw recursive.command0.2 press enter
6. run the recursive.command by doubleclicking
You can doubleclick the recursive.command0.2 script a couple of times to have them work in parallell.
commandscript https://drive.google.com/file/d/0B4tCJMlOYfirWDd2RldZVXdjck0/edit?usp=sharing
Thanks
/D
*updated recursive.command 29 dec (g3gg0)
*updated recursive.command0.2 30 dec (daisermac moving spanned files to folder)
Quote from: g3gg0 on December 28, 2013, 10:53:03 PM
the latest binaries should not need mlv_dump -x anymore.
(although it should work. can you check if you still get such GUID error?)
Compiled new build for 5D3, and downloaded new MLVBrowser
I recorded 4 minutes 4 seconds of CROPPED video 1920x1080 at 29,97 fps = 24.5 GB I stopped the recording.
The audio file is now in correct order, I counted all the way up and never missed any numbers during playback.
To aligned the video I had to shrink the audio file form 4:02,24 to 4:01,08 which is 0.48%
In other words, I had to shrink the audio file by 1.16 seconds. Everything was properly aligned at this point.
To check for alignment I hit a little crystal bottle every ten seconds and record the strike, once the file was shrunk the strike matched perfectly on every hit.
Congratulations, you definitely got the wav frame order correct!
Tomorrow I will check 50 and 60 fps.
Quote from: Danne on December 29, 2013, 02:35:59 AM
Updated batchscript command for mac-users
Ok.
I modified a command script from a member here, jerrykil, and put in your script-instruction dubzeebass and g3gg0. So if doubleclicking the commandscript from within the folder with the mlv,s it starts the conversion without having to go through terminal.
Instruction would be:
1. Download mlv_dump.osx http://upload.g3gg0.de/pub_files/c02de9a0b17fc909b5494ddbd0fd249e/mlv_rec.zip
2. Open a Terminal
3. Go to the folder where you downloaded mlv_dump.osx (extract if needed) and type: sudo mv ./mlv_dump.osx /usr/bin/mlv_dump
4. Type sudo chmod +x /usr/bin/mlv_dump
5. Run the recursive.command file below from the same folder as the .mlv files. Need admin rights. Go to command script through terminal and write chmod +rxw recursive.command press enter
6. run the recursive.command by doubleclicking
You can doubleclick the recursive.command script a couple of times to have them work in parallell.
commandscript https://drive.google.com/file/d/0B4tCJMlOYfird0U5dTNnR0twNE0/edit?usp=sharing
Thanks
/D
*updated recursive.command 29 dec (g3gg0)
Thanks Danne for your script.
I think it needs one modification in order to handle split mlv-files - the move command should be:
mv ./"$BASE".M* ./$BASE
This way not only the first .MLV gets moved but also .M00 and .M01 etc.
Dual ISO and 3X crop mode supported? And both together?
And what about 48-60fps ?
Quote from: daisermac on December 29, 2013, 01:42:13 PM
Thanks Danne for your script.
I think it needs one modification in order to handle split mlv-files - the move command should be:
mv ./"$BASE".M* ./$BASE
This way not only the first .MLV gets moved but also .M00 and .M01 etc.
Thanks. Updated the scriptinstruction, reply #1044, and put in your code. Please test it, havn,t got any spanned .mlv files at the moment.
D
Reporting 5D3 testing with 50fps and 60fps, card spanning on.
Sound file needed to be moved 4 frames forward and bingo it is synchronized both for 50 and 60 fps
Only outstanding issue is Color Temperature and Tint for dngs. They have green cast corrected on ACR: when ACR opens the dng it reads colortemperature 2000 and tint -150. This is WB error. To correct set WB to 5000 and tint 0
Reporting 5D3 testing with sound module.
During cropped mode I cannot go above 2240x1260 pixels. The LV has some sort of an error and quits working. If I disable the sound module it works OK.
Quote from: RenatoPhoto on December 29, 2013, 06:24:49 PM
During cropped mode I cannot go above 2240x1260 pixels. The LV has some sort of an error and quits working. If I disable the sound module it works OK.
thanks! will check where this comes from.
When I turn the camera on, MLV Sound & MLV Raw seem to be turned off by default. Is this the case, or do I not have a setting flipped or something?
Quote from: AnotherDave on December 29, 2013, 07:14:02 PM
When I turn the camera on, MLV Sound & MLV Raw seem to be turned off by default. Is this the case, or do I not have a setting flipped or something?
yup for me too...its like i need to go to canon audio menu to make it come back on.
7D doesn't seem to record any audio frames, dunno why I see the red buffers . Canon audio on or off. 6D I need to play with more but I did get some wavs out.
7D Log:
http://pastebin.com/8EjPPFbi
How do you stop the ADC? At first I was getting audio failed to stop 4->0. I added a stopasifdmaadc but not sure if it helps. At least on 6D it recorded more than once after that.
on 7D i guess it misses this part:
/* experimental for 7D now, has to be made generic */
/* Enable audio Device */
void SoundDevActiveIn (uint32_t);
SoundDevActiveIn(0);
this is something for the "next-gen" wave support in magic lantern.
sound recording is simply stopped by not further queueing any new audio buffers.
there are always two buffers (double buffered) and as soon one is filled, you get the callback
where you have to enqueue the second-to-next buffer.
if you dont queue a buffer during the callback, you will get another callback for the last buffer being filled.
Just baked in everything in the same command script to simplify batchconverting .mlv with audio on mac osx
Thanks to dubzeebass, g3gg0, jerrykil, daisermac
download following https://drive.google.com/file/d/0B4tCJMlOYfirU1FncGxENHhFeE0/edit?usp=sharing
1 – Extract recursive0.3.command and mlv_dump.osx in the same folder
2 – doubleclick recursive0.3.command (type password. If permission needed go to your folder in terminal and write chmod +rxw recursive.command0.3 this will give all permissions needed)
3 - doubleclick recursive0.3.command again and type in password if needed. This should place your mlv_dump.osx according to the script(mlv_dump will disappear)
4 – place your .mlv files in same folder as recursive.command.03 and doubleclick which should start converting the files to folders
I,m not a coder so feel free to critizise
That stuff is old news. I have full audio support for 7D (pel should port this back). Meters are going while its recording. Its just AK4646 with a different audio task you can't override. You have to re configure it after canon like on 600D.
I can record old style wav just fine but then I record with MLV_snd and get nothing... on 6D I'm getting 4GB wavs for some reason (with data), mlv dump just keeps going and going until that limit is reached. I can't run trace because it crashes from out of memory.
If you leave the asif going it crashes out on a bunch of cams or won't let you restart it until it times out. When I run the FFT test I also get action on it on both cameras.
Also: 6D with correct .raw file and wav (5MB) in MLV browser (first recorded clip, canon audio off, audio override)
Next clip is 4gb, 930kb resaved. At least all data seems to be there on 6D.
File Header (MLVI)
Size : 0x00000034
Ver : v2.0
GUID : 1380004378867732245
FPS : 23.970000
File : 0 / 0
Frames Video: 0
Frames Audio: 0
Ok, got 1 wav out of the 7D, correctly sized and both channels out of 5 files. Canon audio off, separate wav override. Maybe its a timing issue. All entries in the log look identical:
mlv_snd_flush_entries: 105 entries to free in queue
/* 00:01:27.710749 */ mlv_snd_flush_entries: 98 entries to free in queue
/* 00:01:27.711923 */ mlv_snd_flush_entries: entry is MLV slot
/* 00:01:27.712237 */ mlv_snd_flush_entries: NULL slot 0 entry
etc....
All clips show 0 audio frames.
*update... I hear some audio in MLV viewer but wavs are not extracted by mlv browser. Audio playback is jerky... like silence silence sound, silence silence sound, etc.
I've run into three problems with the latest (er, yesterday's) build, I figured maybe documenting them here would help development or someone could tell me what I'm doing wrong.
1. I loaded the mlv-rec and mlv-snd modules and set it for 1920x1080p23.976 and tried recording some raw video to my Komputerbay 128Gb 1050X CF card. It gets about 48-70 frames in, reports a skipped frame or two, and stops recording. Every time with the same result. I think spanning is off, and there's only files recorded to CF, but I'm not sure (need to poke around and find that setting again). Recording at the same resolution and frame rate using the normal raw recording module works flawlessly to the same card.
2. I booted the camera with mlv-rec and mlv-snd enabled, and (without recording) hit the zoom button to see a close-up view for focusing. After focusing, I didn't hit the zoom button again to pop back to the normal view and instead just hit record thinking the view would pop back to normal and recording would commence. Instead the view froze, and remained on the screen after I shut down the camera. I had to remove the battery to get it to power off.
3. It's been noted earlier in the thread, but the mac version of mlv_dump claims the .mlv file lacks a header unless the .IDX file is moved, deleted, or renamed out of the application's view. If the IDX is present, no decoding happens. When I renamed the .IDX file to end in .IDX.bak, mlv_dump was able to see the file and I could view the metadata for its contents just fine.
Quote from: 1% on December 29, 2013, 08:51:33 PM
Ok, got 1 wav out of the 7D, correctly sized and both channels out of 5 files. Canon audio off, separate wav override. Maybe its a timing issue. All entries in the log look identical:
[...}
*update... I hear some audio in MLV viewer but wavs are not extracted by mlv browser. Audio playback is jerky... like silence silence sound, silence silence sound, etc.
when adding the lines i mentioned the 7D can record 5/5 times.
maybe you missed the necessary
> SoundDevShutDownIn();
about jerky playback... well maybe it is because the video plays back with 1 fps instead of 25...?
MLVViewer is no media player, it is for previewing ;)
just export and decide by the .wav
btw. added 7D support. see bitbucket, no binaries yet.
Quote from: zachnfine on December 29, 2013, 10:18:04 PM
1. I loaded the mlv-rec and mlv-snd modules and set it for 1920x1080p23.976 and tried recording some raw video to my Komputerbay 128Gb 1050X CF card. It gets about 48-70 frames in, reports a skipped frame or two, and stops recording. Every time with the same result. I think spanning is off, and there's only files recorded to CF, but I'm not sure (need to poke around and find that setting again). Recording at the same resolution and frame rate using the normal raw recording module works flawlessly to the same card.
I just looked at the SD and CF card and found the answer to this problem -- it was trying to record everything to the SD card. I had the preferred card set to CF, so I figured I had this set properly. Maybe there's some other mechanism to use to tell ML to just never, ever try to record anything to the SD card?
OK, got the latest nightly build and the latest mlv_rec modules installed now and am ready to try again.
Is it wrong that in the ML menu, the descriptive text for mlv_snd.mo is "Adds SMPTE output functionality to mlv_rec"?
I'll give the commands a go but have the feeling it will just reset the audio settings (to 23db internal mic)... look at what registers those touch.
It seems I'm getting wav 5/5 times too, just MLV browse isn't dumping it and I never thought to try MLV player.
QuoteIs it wrong that in the ML menu, the descriptive text for mlv_snd.mo is "Adds SMPTE output functionality to mlv_rec"?
Yep.
uhm just tried - it shows the correct description.
but i realized that i put an old version into it, not the most recent one.
reuploaded.
Quote from: 1% on December 29, 2013, 11:20:47 PM
It seems I'm getting wav 5/5 times too, just MLV browse isn't dumping it and I never thought to try MLV player.
MLVBrowseSharp doesnt dump for you?
which OS? which error? did you get the latest version?
Got continuous recording working now, with possible skipped frames.
This may be something unrelated to mlv and simply a new (to me) way that the nightly ML is noting skipped frames, but I'm not sure what to make of this "71 frames / 4115 expected" text at the bottom of the frame while recording.
I've got a screenshot I'd be happy to post, but clicking the "Insert image" button in this form just inserts a couple of img tags into my post -- doesn't seem that there's an upload option. Maybe this'll work: (https://www.evernote.com/shard/s1/sh/370ae243-27cf-48dd-9b16-853ebc97ae58/e5767bbf15dc638cf8dba588652309ee/res/ed0c02ff-8f4b-495c-9dc6-73a542de586c/skitch.jpg?resizeSmall&width=832)
Or just this link: https://www.evernote.com/shard/s1/sh/370ae243-27cf-48dd-9b16-853ebc97ae58/e5767bbf15dc638cf8dba588652309ee/res/ed0c02ff-8f4b-495c-9dc6-73a542de586c/skitch.jpg?resizeSmall&width=832 (https://www.evernote.com/shard/s1/sh/370ae243-27cf-48dd-9b16-853ebc97ae58/e5767bbf15dc638cf8dba588652309ee/res/ed0c02ff-8f4b-495c-9dc6-73a542de586c/skitch.jpg?resizeSmall&width=832)
It appears to be recording fine and the buffer graph at top is not filling past the end, but that text at the bottom alternates between saying things like "00:39.941 frames. continuous OK !" and "58 frames / 5102 expected". I'm not sure how to interpret this.
When I installed the latest nightly I blew away my settings. So I'm not sure what in my previous settings caused it to be recording to SD.
Same 7D files I thought were 1/5 were 5/5 when played in MLV viewer.
Quote
which OS? which error? did you get the latest version?
Win7 x64
Downloaded again today from first post when I saw commits. Both dated 12/29
Both 6D and 7D files are coming out 0 audio frames a bunch in the log but audio is there and looks complete. The 6D files are 4GB after the first clip. Most of the 6D extracts even with 0 audio frames reported.
Quote from: 1% on December 29, 2013, 11:35:40 PM
Both 6D and 7D files are coming out 0 audio frames a bunch in the log but audio is there and looks complete. The 6D files are 4GB after the first clip. Most of the 6D extracts even with 0 audio frames reported.
just tried on 7D:
audio works fine, here a log.
Exec: 'mlv_dump.exe --batch --dng -v -o "d:\Users\g3gg0\Desktop\Neuer Ordner\M30-0051.MLV.frame_" "F:\DCIM\100EOS7D\M30-0051.MLV"'
MLV Dumper v1.0
-----------------
Mode of operation:
- Input MLV file: 'F:\DCIM\100EOS7D\M30-0051.MLV'
- Verbose messages
- Convert to DNG frames
- Output into 'd:\Users\g3gg0\Desktop\Neuer Ordner\M30-0051.MLV.frame_'
File F:\DCIM\100EOS7D\M30-0051.MLV opened
File F:\DCIM\100EOS7D\M30-0051.IDX opened (XREF)
XREF table contains 99 entries
Processing...
File Header (MLVI)
Size : 0x00000034
Ver : v2.0
GUID : 3806773218881307577
FPS : 25.000000
File : 0 / 0
Frames Video: 73
Frames Audio: 16sure you use *my* release?
I've been playing with the MLV dump compression options and I have problems when using the -b parameter. When outputting files to DNG, the images are distorted in one form or another (I can't even recognize what's in the image). Most of the time it's all white with some random bits of noise. Tried playing around with different values, but with no luck. This parameter gives just a little more compression than without it, but even that's a nice thing to have. The files were recorded on a 5D2. All the other parameters (-e -l -z) appear to work fine in the short tests I've done.
edit: spoke a little too soon. There are a few artifacts coming from somewhere else, though not as severe.
edit2: Recompressed it and it looks fine now o_O I think having the MLV Player running while its compressing isn't a good idea XD Outputting the frames crashes after a while though.
On 6d
working with the newest available build (11 december build for 6d). And the mlv-rec and mlv-snd module from yesterday (copied from the 5d build, from the download link in the 1st post.)
First clip is with audio and error free.
Every clip after the first one gives the error (audio failed to stop, state 4), in those files I can't extract a full wave file with MLV-dump.osx.
I'm I doing things right by using the 11 december build for the 6d and replacing the mlv-rec.mo and mlv-snd.mo file for the ones in available in the newest 5d build ?
so with MLV_REC whats the best format the cards should be in ? exFAT or
FAT32.. all my cards are KB 64gb 1000 exFAT
still getting skip frame when it starts to roll over to the next 4GB file
Is anyone able to record long clips? I got a 1920x1080p23.976 clip with audio recorded to my Komputerbay 1050X 128Gb card, and it appeared to be running continuously ok (except for onscreen messages about "expected frames" that I don't really understand), but recording stopped soon after the 4Gb limit was hit. My card is formatted exFat, and I can record to it with the standard raw recording for 10 minutes and beyond without issue.
I'm processing the file now, it appears to have actual audio (I listened to the wav), but I haven't yet viewed the clip to see if there are any apparent skipped frames.
BTW, here's another way to script the processing of MLV files, using tcsh:
% foreach foo ( *.MLV )
foreach? mlv_dump --dng -o $foo:r_ "$foo"
foreach? mlv_dump -m -o $foo:r_meta.mlv "$foo"
foreach? mlv_dump -v $foo:r_meta.mlv > $foo:r.txt
foreach? rm $foo:r_meta.mlv
foreach? rm $foo:r_meta.mlv.wav
foreach? end
I dropped my .MLV and .M00 file for one clip into a folder, then cd'd into that folder and typed the lines above (the "%" and later "foreach?" are prompts, no need to type those).
FAT32 FTW!
I reformatted my CF card using OS X Disk Utility to FAT32, and now I can record long stretches with no skipped frames (where onscreen was previously displayed "X Frames Expected" now it just says "continuous frames OK"), and no skips at 4Gb split points.
So some issue must be getting in the way of recording MLV 2.0 to exFat formatted CF. But Fat32 seems to work a treat.
And just like that, after I pulled the files from the CF card to hard drive and deleted them, it's skipping frames again, still formatted to FAT32.
I guess I'll shoot with the normal raw video and second system sound at a shoot this afternoon, then I'll get back to trying to make mlv2.0 work for me later. It's very exciting that sound recording has been made possible! I'm thinking even a scratch track at 8-bit 22kHz could be used to automatically sync up the second system audio later (pluraleyes/fcpx/etc).
updated mlv_dump to skip WBGain information when writing to DNG
I shot a full 12 mins @ 1920x1080 with sound on a 5D3 with the 64gb KomputerBay 1000x card in a test. Maybe this isn't new information, but I was reading that people were having issues with skipping... I didn't lose sync once.
*It is important to note, however, in a previous attempt with the sync beep on (which didn't sound at rec start) there was NO recorded audio when I processed the clip.
Quote from: AnotherDave on December 31, 2013, 02:45:09 PM
I shot a full 12 mins @ 1920x1080 with sound on a 5D3 with the 64gb KomputerBay 1000x card in a test. Maybe this isn't new information, but I was reading that people were having issues with skipping... I didn't lose sync once.
*It is important to note, however, in a previous attempt with the sync beep on (which didn't sound at rec start) there was NO recorded audio when I processed the clip.
what are your settings on MLV_REC?
after about 60-75secs it skips and i tried all 8 of my KB cards.. lol
Quote from: Jakobmen on December 31, 2013, 02:47:35 PM
what are your settings on MLV_REC?
after about 60-75secs it skips and i tried all 8 of my KB cards.. lol
Audio
1 Beep, test tones - off
2 MLV Sound - ON
- trace off
- Sample rate 48k
Movie
1 - REC indicator - Elapsed Time
2 - Gradual Expo - off
3 - FPS override - off
4 - HDR video - off
5 - Vignetting - off
6 - Image Finetunning - none
7 - RAW video (MLV) - on, 1920x1080
- disable g draw - off
- frame skip - off
- preview - ML Grayscale (I'm using a zacuto evf so it isn't grayscale over hdmi)
- start delay - off
- digital dolly - off
- card warm-up - off
- memory hack -off
- extra hacks - on
- debug trace - off
- show buffer graph - off
- test mode - off
- buffer fill method - 4
- CF-only buffers - 3
- Card spanning - off
- tag:text - none
- tag:take - off
8 - Movie Tweaks
- movie logging - off
- REC/stby - red crossout
- dim rec led - on
- force liveview - start& cpulens
Modules - file_man, mlv_rec, mlv_snd *no other modules loaded.
Sound in canon menu is on auto.
I'm not sure if ALL of these settings make a difference... but maybe it'll help trace down the issue.
I do think it is important to note that with sync beep on, several times my first frame has been garbled and no audio was recorded, despite mlv sound being on.
Warm up your card and set picture quality to jpg not raw.
I get continuous woth 24fps 1920x1080 with sound and continuous with 25fps after warmed up card
Quote from: Danne on December 31, 2013, 03:10:48 PM
Warm up your card and set picture quality to jpg not raw.
I get continuous woth 24fps 1920x1080 with sound and continuous with 25fps after warmed up card
I have picture quality set to raw not jpg.
Quote from: g3gg0 on December 31, 2013, 02:38:03 PM
updated mlv_dump to skip WBGain information when writing to DNG
Ok, I compiled new mlv_dump.exe and now MLBrowser exports dng with WB Temperature 5500 and Tint +19.
Thanks
Quote from: AnotherDave on December 31, 2013, 03:05:28 PM
Audio
1 Beep, test tones - off
2 MLV Sound - ON
- trace off
- Sample rate 48k
Movie
1 - REC indicator - Elapsed Time
2 - Gradual Expo - off
3 - FPS override - off
4 - HDR video - off
5 - Vignetting - off
6 - Image Finetunning - none
7 - RAW video (MLV) - on, 1920x1080
- disable g draw - off
- frame skip - off
- preview - ML Grayscale (I'm using a zacuto evf so it isn't grayscale over hdmi)
- start delay - off
- digital dolly - off
- card warm-up - off
- memory hack -off
- extra hacks - on
- debug trace - off
- show buffer graph - off
- test mode - off
- buffer fill method - 4
- CF-only buffers - 3
- Card spanning - off
- tag:text - none
- tag:take - off
8 - Movie Tweaks
- movie logging - off
- REC/stby - red crossout
- dim rec led - on
- force liveview - start& cpulens
Modules - file_man, mlv_rec, mlv_snd *no other modules loaded.
Sound in canon menu is on auto.
I'm not sure if ALL of these settings make a difference... but maybe it'll help trace down the issue.
tried 5 times I did get 1at 3mins also dave are you still getting .M00 files also?
everytime the file needs to break pass the 4GB size thats when it starts.
Set picture quality to jpeg? You mean in the normal Canon menu? Does that affect video recording for some reason with mlv_rec when it doesn't seem to have an impact on raw_rec?
Quote from: AnotherDave on December 31, 2013, 03:05:28 PM
- memory hack -off
- extra hacks - on
Are you sure, that memory hack shouldnt be ON?
Quote from: Jakobmen on December 31, 2013, 04:33:49 PM
tried 5 times I did get 1at 3mins also dave are you still getting .M00 files also?
everytime the file needs to break pass the 4GB size thats when it starts.
Yep, I'm still getting .m00 files, but I am not (nor did I ever) have an issue with them splitting.
Quote from: kgv5 on December 31, 2013, 06:53:27 PM
Are you sure, that memory hack shouldnt be ON?
Should it be? I don't know. It isn't in my setup and it is working.
Quote from: AnotherDave on December 31, 2013, 08:45:31 PM
Yep, I'm still getting .m00 files, but I am not (nor did I ever) have an issue with them splitting.
anotherdave,
whats your allocation unit size on your KB cards ?
the first 4GiB are saved in a .mlv, the next in a .m00, the next in .m01 and so on.
memory hack on 5D3 gives you some extra buffers
Quote from: Jakobmen on December 31, 2013, 08:53:02 PM
anotherdave,
whats your allocation unit size on your KB cards ?
I donno... how do I find that out?
When you format card it should tell you
@g3ggo
used MLV dec 29 and ML dec 30.
I can achieve continuous ok MLV with Sound On with the ff resolutions;
1728 x 972 16:9
1856 x 928 2:1
they can record 15 min in a 64 gig 1000X KB card.
tested with clapper at regular intervals to test for audio sync. -No problems with sync.
1872 x 936 - starts to skip after 8 min.
tested FPS over ride with sound- I find this nice as it will work. MLV Sound is recorded in real time.
This can be used in certain situations.
I had a minor problem when the framing guides and other info turns OFF when recording and found out that
DISABLE G. DRAW needs to be ON to keep the frame guides and other overlays on Screen.
I suggest this be RENAMED as its description and name does the Opposite and got me misled.
Also, using certain screen layouts, the buffer graphics/elapsed time/frames/datarate info that comes on during record
clashes with the overlays such as vu meters, lens info framing guide etc.
Using 16:10 HDMI t/b layout scheme will help avoid this.
On using ext HDMI monitor.- I get short record times and coruppted frames with hdmi attached.
1728 x 972 GD.OFF + hdmi monitor = 28 sec only then skips and stops.
1856 x 928 GD.OFF + hdmi monitor = 35 sec only then skips and stops has corrupt frames.
whats everyones Allocation unit size on there KB 64gb 1000 cf cards?
Quote from: ted ramasola on December 31, 2013, 11:15:05 PM
@g3ggo
used MLV dec 29 and ML dec 30.
I can achieve continuous ok MLV with Sound On with the ff resolutions;
..
thanks for the feedback!
Finished a shoot yesterday and recorded using mlv_rec . Some of the takes were longer and over 4gb in size where ML split them into .m00 .m01 etc. Every time I open MLV browse sharp and navigate to the folder where the files are located it quits. Any pointers from anyone ?
I also noted that some of the clips are 4.29gb in size FYI card is exfat formatted.
Thanks !
Quote from: Jason Montalvo on January 01, 2014, 03:15:58 PM
Every time I open MLV browse sharp and navigate to the folder where the files are located it quits. Any pointers from anyone ?
wrong thread, go there:
http://www.magiclantern.fm/forum/index.php?topic=8447.0
and redownload the latest version and retry.
Einen guten Rutsch!
I've just roughly fixed Issue #1773: global draw disable feature not reverts changes after stopping (https://bitbucket.org/hudson/magic-lantern/issue/1773/mlv_rec-global-draw-disable-feature-not)
Magic Lantern MLV (https://bitbucket.org/a_d_/magic-lantern-mlv/commits/all)
would commenting out line #1497 if (!PREVIEW_HACKED) return;
do the same?
Confirm, it does the same thing!
Quote
sure you use *my* release?
For dumping yes 100%, I can't compile the windows binaries. The best I can compile is MLV dump but still using yours from the pack.
So the functions are getting overrode by audio_configure (right before start asif) and no problems but have to check if anything has changed for 7D. 6D still isn't shutting down properly, stuck at state 4 (but message is shorter now). I tried with both shutdownin and stop asif. Maybe MLV dump needs to stop extracting if the wav is really done for 6D and others.
Hi :)
Just wanted to report my tests with MLV with Audio on the 5D2
I first have to Say Great Job Everyone! I all Cases audio was sync .
Now to the Test Numbers;
I did two different type of tests one with HDMI devices (Zacuto EVF & Atomos Ninja HDMI recorder)
And Without HDMI. Tested on Lexar 1000x 64GB.
Without > Largest Frame size Continuous record
1872x936 23.976p with Audio 19,931 frames/13:51 Min. 60GB 73.3MB/s 67° C. :o buffer method 2
Crop Mode > 2048x872 23.976p with Audio 19,930 Frames/13:50 Min 60GB about 74-75MB/s buffer method 1
With HDMI>Largest Frame size Continuous record,
1872x850 23.976p 2.20:1 with Audio 22,710/15:47 min 60GB 67.9MB/s buffer method 1
Crop Mode > 1856x938 23.976p with Audio 20,000 frames 68 MB/s buffer method 0
This on A.D. build for the 5D2 dec.30
Tried the nightly build from Jan-1 MLV, and could get larger frame size continuous with HDMI
1856x930 23.976p 2:1 19,000 frames 55GB buffer method 1
In all tests GD off and only the 5D2_212.sym, file_man.mo, mlv_rec.mo, mlv_snd.mo
Crash 2 times with hdmi plugin when I tried to record 1856x1004 and or 1872x1012, the buffer just filled up with blue color then lockup
Battery pull & reboot ok after that.
It seems on the 5D2 you are limited to about 70-73 MB/s without hdmi active
and with it active then that drops to about 65-68 MB/s & in crop mode to about 74MB/s normally that's about 76-79MB/s
and 75MB/s in non crop mode.
Its definitely usable without hdmi at this point, run & gun style ;)
The 4gb wav problem is from mlv_dump, its impossible to end it even from process manager.
4 gib wave? how long did you record?!
and to which post did you refer?
Quote from: 1% on January 02, 2014, 02:30:20 AM
The 4gb wav problem is from mlv_dump, its impossible to end it even from process manager.
Hmmm, I think that is actually a Windows OS
problem.
edit: http://en.wikipedia.org/wiki/WAV#Limitations
finally with mlv sound i m able to make a film. thank u magic lantern so much. i was never interested in magic lantern raw since there was no audio recording. now i have made a film with mlv raw
http://www.youtube.com/watch?v=2hiFKQbQ6cc&feature=youtu.be
Quote4 gib wave? how long did you record?!
Not long, actual data is a few MB. The problem is if you get an asif did not stop error in MLV sound the wav will end up being huge. So for 6D. 1st wav is fine, 2nd wav asif didn't stop and you get the giant wav. In MLVbrowse it hangs on the last frame or maybe 1 before it and keeps extracting until the 4gb limit is reached.
This is driving me nuts.. :o in rec.raw i can record fine 25gb+ raw file 5mins or more, no problem..
in rec.MLV i start getting skips when .m00 files start to be made.. tried alot of settings.
i love how multi-file support (4 GiB splitting is enforced) works but when it spilts for me it just starts to skip...
lower frame size does fix it.. lol
Quote from: Jakobmen on January 02, 2014, 06:52:29 PM
This is driving me nuts.. :o in rec.raw i can record fine 25gb+ raw file 5mins or more, no problem..
in rec.MLV i start getting skips when .m00 files start to be made.. tried alot of settings.
I had some problems at first and realized that my mlv_sound.mo and mlv_rec.mo was getting updated but not re-compiled after the update was done. First I used g3goo compile for 5D3 and then I realized that my compiling machine was the cause. I make sure I do a make clean and then make on the modules and check the date of the .mo to make sure it is the latest.
Renatophoto can you send me your files mlv_rec & mlv_sound.. ?
I sent them to you but remember that MLV is slower. For example I can record 1920x1080 at 29.97 fps on the CF card with raw_rec but with mlv_rec I have to enable card spanning to be able to record at this resolution with audio.
Quote from: RenatoPhoto on January 02, 2014, 10:31:13 PM
I sent them to you but remember that MLV is slower. For example I can record 1920x1080 at 29.97 fps on the CF card with raw_rec but with mlv_rec I have to enable card spanning to be able to record at this resolution with audio.
thanks
i guess we need faster cards.. 8)
More processor speed would be better.
@ g3g00: I noticed that the recording speed of the SD card is not showing and was wondering if it would be best to disable all unnecessary calculations to allow for the fastest recording speed? For example these speed computations could be moved to the Show buffer graph option in case someone wants to look at it.
I am shooting on a 5D3
I noticed that there is a 5D3_111.sym as well as autoexec.bin file in your mlv_rec folder in addition to the mlv_snd module.
In addition to adding the mlv_snd module to the newest nightly build...
Do I need to copy/replace this autoexec.bin to the root directory of my SD card?
Do I need to copy/relaoce the 5D3_111.sym from your mlv_rec folder into the modules folder of the SD card?
It seems that MLV.SND is finally working on the 7D. I just recorded 4 short Clips, all 4 MLVs extracted with MLVBrowseSharp in DNGs and WAV - and got 4 WAV files, each of them containing usable Sound.
My Settings:
1) For the first two files:
Sound Record: Normal
Beep, test tones: OFF
MLV Sound: ON
other Settings Default, Sound in Canon Menu on AUTO
2) Fot the other two files
Sound Record: Separate WAV
Analog Gain: 26 db
Rest unchanged
Tested with a quick DNG->JPEG convert in Lightroom and importing in Premiere, the Audio is 7 Frames longer than the Video at 25 fps (no FPS override, Canon Video set to 25p and PAL). If i Interpret the sequence as 24p, the Audio is 5 Frames too short.
Next things to test will be wether i can get continuous recording with MLV Sound on, and wether the audio stays in sync. I will shoot an analog clock with audible ticking for a Minute or two and use a flash at intervalls for easy sync checking and in case the ticking is not loud enough. This should also help to check the video timebase - perhaps i need to use fps override to get exact 25p.
Currently, the write performance lags a bit whena recording is started, display shows 200-300 Frames expected, but then catches up at up to 80 MB/s, the buffer graph goes down, and the writing Speed settles at about 70 MB/s. My card is a Transcend 64 GB, it definitely needs low Level formatting with autoformat, otherwise it really struggles to reach the necessary writing speed. buffer fill method is set at 4, extra hacks on. I did not test other settings, maybee i should to find out what works best.
Global draw and Digic Peaking are on, also the Histogram.
You'll gain a little speed with GD off. My continuous with audio is the largest 1x 16x9 size... the 5:3/3:5 are don't seem to be making it continuous + sound.
How is the alignment at 23.976? vs 24.00? Also harder to be continuous with external stereo vs the doubled internal mic. Sampling rate doesn't seem to be making a difference speed wise.
Quote from: 1% on January 02, 2014, 05:23:44 PM
Not long, actual data is a few MB. The problem is if you get an asif did not stop error in MLV sound the wav will end up being huge. So for 6D. 1st wav is fine, 2nd wav asif didn't stop and you get the giant wav. In MLVbrowse it hangs on the last frame or maybe 1 before it and keeps extracting until the 4gb limit is reached.
Experiencing the same problem with my 6d (MLV-dump.osx, so on OSX platform).
When I'm filming now, I turn my camera off and on after every single clip, just to have sound (raw video WITH sound, which is an huge improvement!)
Will there be a solution for the "sound failed to stop, state 4" error problem ?
I hope so. There are only a couple of things to do:
1. Figure out the audio state machine and get rid of the patches
2. Stop asif properly between 6D and 7D
3. Make MLV_rec stop outputting 4gb wavs and ignore the error.
Quote from: 1% on January 03, 2014, 05:50:26 PM
3. Make MLV_rec stop outputting 4gb wavs and ignore the error.
4gig wavs are not possible, this is no fix for a bug.
although it should not hang at all.
i need such a .mlv or a full dump (-v) to check whats wrong.
with some vague "does hang" reports, i cannot fix anything.
No problem, here is a log of a 2nd file... it hangs when it hits the first AUDF block and writes the wav till 4gb.
http://pastebin.com/henB4t1K
Properly ended 1st video.
http://pastebin.com/63PBASKd
From looking:
I think maybe the seconds are excessive and need to be written as ended even if audio is stopping ,etc.
I will also try to get rid of the message with SoundDevShutDownIn and SoundDevStopIn... 6D has those same functions.. on 7D it calls stopasifdmaADC (so it stops) and unpowers the internal mic, on 6D actrl functions so still state machine problems. all the beep.c wavs across cameras were endless like this and had to be re-saved... but the asif there is stopped or times out right away when you look at the DM log.
Quote from: Steve Kahn on January 02, 2014, 11:17:49 PM
I am shooting on a 5D3
I noticed that there is a 5D3_111.sym as well as autoexec.bin file in your mlv_rec folder in addition to the mlv_snd module.
In addition to adding the mlv_snd module to the newest nightly build...
Do I need to copy/replace this autoexec.bin to the root directory of my SD card?
Do I need to copy/relaoce the 5D3_111.sym from your mlv_rec folder into the modules folder of the SD card?
Same question here. Also, will mlv_snd be an included module in the nightly builds anytime soon?
Quote from: 1% on January 03, 2014, 10:33:52 PM
No problem, here is a log of a 2nd file... it hangs when it hits the first AUDF block and writes the wav till 4gb.
http://pastebin.com/henB4t1K
thanks.
nothing as simple as that.
see the log and you will notice there is one AUDF with frame #-1 (or 0xFFFFFFFF)
Block: AUDF
Offset: 0x0cbb5a60
Size: 38656
Time: 18446744073709552.000000 ms
Frame: #-1
Space: 232guess what this will cause?
Quote from: Embark on January 03, 2014, 11:31:17 PM
Same question here. Also, will mlv_snd be an included module in the nightly builds anytime soon?
Same question here.
Quote from: Steve Kahn on January 02, 2014, 11:17:49 PM
Do I need to copy/replace this autoexec.bin to the root directory of my SD card?
Do I need to copy/relaoce the 5D3_111.sym from your mlv_rec folder into the modules folder of the SD card?
you don't have to, but its the latest version i tried with.
if there is any trouble with the nightly, use my version
Quote from: g3gg0 on January 04, 2014, 01:10:08 AM
thanks.
nothing as simple as that.
should be fixed, can you try?
(re-record a new movie)
Better but wavs are coming out 1KB with these logs:
http://pastebin.com/KVTEhUwG
http://pastebin.com/4dL3FBSV
Ok, undid the fix 1/2 way and now they have sound + dump... that block never gets written for some reason otherwise.
http://nitrobits.com/file/1lvTdJN2EcZv0Og/magiclantern-Tragic.2014Jan03.6D113.zip
Quote from: 1% on January 04, 2014, 03:29:20 AM
Better but wavs are coming out 1KB with these logs:
that is absolutely correct as the mlv doesn't have any wave.
you said that the asif didn't shut down correctly.
so it doesn't start the second time and we get no audio....
Yea but I fixed that, no more error message, correct # of frames with audio in the log but then 1KB wavs as the AUDF header was never written any time. Undoing 1/2 the overflow fix made it work and not produce 4gb wavs nor 1kb wavs.
Did some further tests with Sound recording on the 7D, this time with an external mic plugged in. So far, i can get continous recording at 1728x972 and 25 fps. When i reach 1 minute, the buffer suddenly begins to fill, and the recording stops at around 1:03 to 1:04 each time (no skip allowed). This coincides with reaching the 4 GB limit and file splitting. Either my card is too slow to catch up after the momentary writing speed drop at file split, or there is still the same bug others have been reporting.
Both with and without fps override (set at 25 fps exact) the timebase of the video is 100% correct compared to a clock, at least for the 1 minute+few seconds intervall i am able to record. I imported the files in Resolve lite, and set the project at 25 fps. No worries here.
The audio is perfectly in sync, both checked against the ticking of the clock an several flashes fired next to the mic throughout the recording. I got only one video with a 6 second longer audio file than the video, this was one i recorded until the camera stopped. The other files, where i stopped just before the buffer was full, i only get a few frames to much audio. Because the sync is ok, i guess the audio recording does not stop exactly simultaneous with the video. However, compared to the hassle external sound recording witout any in camera audio for easy matching in post, i can perfectly live with that little imperfection.
Now i need to develop a workflow, i guess i will try a conversion in ACR with the VisionLog Profile into DNxHD 10 bit, and grade the footage in AE or Premiere, perhaps using VisionLog LUTs. My machine is too slow to work directly with DNGs, and i want to use ACR for conversion, as Resolve Lite has no noise reduction and feels rather non intuitive to work with. Too bad that ACR has these flicker issues, it would be easy to get a nice look in it quickly, without having to use any LUTs later.
Next thing to test will be the 24p mode, although i probably wont use it here anyway. In PAL land, 25p is the way to go, i think, unless you plan to print your work to cine film.
Quote from: 1% on January 04, 2014, 05:23:06 PM
Yea but I fixed that, no more error message, correct # of frames with audio in the log but then 1KB wavs as the AUDF header was never written any time. Undoing 1/2 the overflow fix made it work and not produce 4gb wavs nor 1kb wavs.
fixed that. you should now be able to pull my version.
@all:updated the mlv_snd module and mlv_dump which had problems since the last change to allow long path names.
Quote from: michael84 on January 04, 2014, 08:06:20 PMEither my card is too slow to catch up after the momentary writing speed drop at file split, or there is still the same bug others have been reporting.
yeah, i am aware of this. its due to the filesystem catalog being locked etc.
i am experimenting to work around this.
Quote from: michael84 on January 04, 2014, 08:06:20 PMThe other files, where i stopped just before the buffer was full, i only get a few frames to much audio. Because the sync is ok, i guess the audio recording does not stop exactly simultaneous with the video.
yes, thats the reason - audio and video cannot stop perfectly in sync.
well it could - then i would have to configure one audio buffer per frame duration. (40 ms @ 25 fps)
and this would cause a little more cpu load what i tried to prevent. (i use 200 ms buffers)
right now you have between 200 ms and 400 ms of additional audio data, which is 8 to 16 frames with 25fps.
one thing i could to is dropping the last audio frame, which would produce 200ms less audio data.
so we end up with 0 ms to 200ms of additional data. (0 to 8 frames)
but to be honest, i prefer having a bit too much data than too few ;)
hmm i have read reports that said their audio start is 2-3 frames off.
so you are sure that the audio is perfectly in sync on 7d?
I doubt any are in sync perfectly.... would be interesting to see a black frame + FFFF or something in the audio injected at the same time, begining + end.
why do you doubt?
we can not inject black and FF hardware-synced.
I doubt because all of the wavs I've had from any canon always seemed like they started at a similar time, not necessarily frame 0/1 and have been the wrong length... if slated though and sometimes stretched it would line up however. Interestingly too if one camera is recording wav and another records videos the files were much easier to sync and didn't seem to have any stretch....
I guess if its not hardware synced any kind of built in slate would be useless, probably why I had no luck going that route.
Quote from: 1% on January 04, 2014, 09:33:41 PM
I doubt because all of the wavs I've had from any canon always seemed like they started at a similar time, not necessarily frame 0/1 and have been the wrong length... if slated though and sometimes stretched it would line up however. Interestingly too if one camera is recording wav and another records videos the files were much easier to sync and didn't seem to have any
the reason why audio is recorded longer than the video is, i already explained there (http://www.magiclantern.fm/forum/index.php?topic=7122.msg94443#msg94443)
this is no bug or problem, mlv_snd just recording 200-400ms longer. there is no need to stretch.
about the start time, i dont quite understand what you wanted to explain.
When I tried to inject frames to sync with and beep into the audio, it would start at the same time but it wasn't frame 0.
i still dont understand what you wanted to say, nevermind :)
well, at least the people report that wav is in sync.
lol. just that the start is consistent but not at the exact begining of the video.
here (http://upload.g3gg0.de/pub_files/b10437c5bc8658866a51ef9a7dfb1e6c/mlv_rec.mo)is an experimental version that *could* be better regarding buffer drops on file switching.
can some of you test it?
Recorded for 2 mins with the previous incarnation on 6D, didn't get a speed drop when the file split. Still split on an exfat card.
Quote from: g3gg0 on January 04, 2014, 08:49:52 PM
hmm i have read reports that said their audio start is 2-3 frames off.
so you are sure that the audio is perfectly in sync on 7d?
At least so far i can tell with my method. I fired an EX-430 remotely while recording, and looked frame by frame wether the Flash in the picture lined up with the discharge pop in the audio track. Every time it matched exactly. So unless the pop and the visual flash did not occur at the same time (will check this later with my HDR-FX1, just to be sure), the sync has to be ok, otherwise i would certainly be able to see it. Flash duration is only one frame. There is no sync drift over 1 minute recording.
However, what does happen is that the first few frames (2-3 could be just right) are not usable due to some kind of high level noise (waveform starts with a spike then drops to normal level). This is consistent through multiple recordings. So no offset, but some few frames of unusable audio. Note that this observation is only for 25p (selected in Canon menu /with and without 25 fps override selected, not 24p test yet), so for other modes there might be an offset/sync error.
I will try the experimental mlv.rec later. Amazing to see how much progress has been made recently, with updates rolling in on a daily basis. Keep up the good work!
Quote from: g3gg0 on January 04, 2014, 11:12:28 PM
here (http://upload.g3gg0.de/pub_files/b10437c5bc8658866a51ef9a7dfb1e6c/mlv_rec.mo)is an experimental version that *could* be better regarding buffer drops on file switching.
can some of you test it?
16:9
1920x1080 after 1min in it starts the buffer bars go crazy then it skips. once with GD on and off
1856x1944 samething like above.. once with GD on and off
1728x972 recorded 5mins+ fine.. with everything on
Thanks g3ggo
Tested the new mlv.rec with my 7D, i can now record past 1 minute without frame drops. This time, sync was off by 2 frames, the audio was 2 frames before the picture, while the audio start (waveform peak) appeared to be 3-4 frames late. Now i will try again using the old mlv.rec, to see wether the sync issue is caused by the new mvl.rec, or just some inconsistency by the camera.
Help: How can open MLV file in Apple Macintosh?
http://www.magiclantern.fm/forum/index.php?topic=8447.0
"Please contact g3ggo with this info:
Exception of type "System out of memory Exeption was thrown
at mlv_view_sharpVLVHandlerBlock(Sting type, mlv_rawi_har_t header,
Byte(} raw_data, Int32 raw_pos, Int32 raw_lwngth)
???
Also, on this same day shoot about half of the mlv files had no sound for some reason. ???
7D, 1728x1036 + internal mic is continuous for me now. Nothing is happening when going over the 4gb hump. GD off though
Sorry I haven't read this entire thread (planning to tonight) - Using the 5d3 + 1000CF KomputerBay + 32GB Sandisk I'm able to extract files fine and I get raw and audio. My only question is this - when I set it to 1920x1080 in raw and have the MLV option enabled, I lose global draw (my sweet focus peaking is no more). If I toggle the MLV recording off and restart the camera everything is fine. Before you all yell at me, is there something simple I'm missing?
Thanks!
Shawn
if I would dare guess that the mkIII is the same in MLV menu as the mkII then you go to MLV menu and turn DISABLE G.DRAW to ON.
I know, its mislabeled and I was wracking my brains as I thought turning it on would do as it say, but it actually does the opposite and DOES NOT turn Gdraw off when it is ON.
Hope a developer can change this. :) I already reported this in an earlier post #1093.
I found this:
this
uint32_t dmaFlags = 0x40001000;
to
uint32_t dmaFlags = 0x20001000;
improved write speed on 7D+6D... couldn't find what the flags stand for in the wiki , the previous also caused edmac timeouts at certain 2.35:1 resolutions.
Is the latest TL build working with sound?
Improved in that aspect ratio you mean?
Yep the sound is working..
0x40001000 was what g3gg0 changed it to, I changed it back because of above problems, wondering
what the flags are about. maybe there is a better setting because there was a definite effect on speed
Quote from: 1% on January 07, 2014, 03:25:26 AM
Yep the sound is working..
0x40001000 was what g3gg0 changed it to, I changed it back because of above problems, wondering
what the flags are about. maybe there is a better setting because there was a definite effect on speed
How much?
Sorry about the stupid question but I looked a bit and find nothing related, so here it is : how do yout extract sound when you film in the .mlv format ?
Just tried for the first time and I get a .mlv and a .idx. I can extract the vid with mlv2dng but no sound.
You can convert with mlvviewer from g3gg0 http://www.magiclantern.fm/forum/index.php?topic=8447.0
or with dubzebass script here http://www.magiclantern.fm/forum/index.php?topic=9731.msg93586#msg93586
Quote from: Fukurou on January 07, 2014, 08:47:22 AM
Sorry about the stupid question but I looked a bit and find nothing related, so here it is : how do yout extract sound when you film in the .mlv format ?
Just tried for the first time and I get a .mlv and a .idx. I can extract the vid with mlv2dng but no sound.
You did turn sound on too right? It's not enough to just enable the module; you have to go to the first page and enable it too. After I did this when I did the extraction to a raw file the .wav file appeared as well magically - meaning there's no special "switch" to extract audio.
Quote from: ted ramasola on January 07, 2014, 02:21:17 AM
if I would dare guess that the mkIII is the same in MLV menu as the mkII then you go to MLV menu and turn DISABLE G.DRAW to ON.
I know, its mislabeled and I was wracking my brains as I thought turning it on would do as it say, but it actually does the opposite and DOES NOT turn Gdraw off when it is ON.
Hope a developer can change this. :) I already reported this in an earlier post #1093.
Thanks Ted - I'm sure that will be it.
On another note, I'm going to do some long recordings today with audio as well as have my Zoom H1n recording simultaneously mounted on the shoe. Then I'm going to pull it into Vegas and run Pluraleyes - if I get a sync I'll be super happy - that's all I've wanted was a reference audio with my raw....
Will report back in.
Shawn
Looks good - took an empty card and let it run until the end. No problems and the audio synched up perfectly. Now to just fine tune my settings.
Thanks guys for the wonderful update! Shawn
Quick question for the meta-data - I ran mlv-dump (-m -o) and successfully exported it to another MLV file that is meta-data only (no video) and I was able to read the information in MLV Browser. However, if I try to read that metadata in notepad it's garbled.
How can I dump the following information into a text file? Would be very handy for quick reference.
(http://www.goldcomics.com/incoming/ML.jpg)
Quote from: Shield on January 07, 2014, 08:23:17 PM
How can I dump the following information into a text file? Would be very handy for quick reference.
mlv_dump -v
Quote from: g3gg0 on January 07, 2014, 08:33:05 PM
mlv_dump -v
Still not getting a file that is readable in notepad. I'm going to setup a batch for exiftool to just grab the info from the dng's extracted from the MLV browser. What would really be slick is an option in MLV browser to "Export selected files as .dng +.wav +.txt"...I realize you're busy, but wow, things are really close. The audio sync is perfect now.
Finding that if I use ATTR (half-shutter press) it disables the mlv_snd for some reason
Quote from: Shield on January 07, 2014, 08:56:44 PM
Still not getting a file that is readable in notepad.
uhm the output of mlv_dump is readable, right? ;)
Quote from: Steve Kahn on January 07, 2014, 09:19:21 PM
Finding that if I use ATTR (half-shutter press) it disables the mlv_snd for some reason
it disables the module?
It's most likely a conflict with the beeps (turn them off).
Quote from: a1ex on January 07, 2014, 09:28:43 PM
It's most likely a conflict with the beeps (turn them off).
That was it! With beeps on after you ATTR audio doesn't register or record.
what do you mean with ATTR? ETTR?
sorry not to be clear.
yes. the ettr module (or Auto ETTR) when I initiate it with a half shutter press (HalfS DBC) and if the BEEP is on it disables sound meters in mlv_snd and no sound is recorded.
Quote from: Shield on January 07, 2014, 08:56:44 PM
Still not getting a file that is readable in notepad. I'm going to setup a batch for exiftool to just grab the info from the dng's extracted from the MLV browser. What would really be slick is an option in MLV browser to "Export selected files as .dng +.wav +.txt"...I realize you're busy, but wow, things are really close. The audio sync is perfect now.
try mlv_dump -v file.mlv > file.txt
Quote from: g3gg0 on January 07, 2014, 09:24:27 PM
uhm the output of mlv_dump is readable, right? ;)
It's mostly gibberish with a few words that are relevant, and not formatted in any easy to use way.
Quote from: bnvm on January 07, 2014, 11:14:36 PM
try mlv_dump -v file.mlv > file.txt
That works, but far too verbose. Just looking for exposure, lens, camera info. Don't need hundreds of lines that have this stuff:
Block: RTCI
Offset: 0x000002a4
Size: 44
Time: 41.330000 ms
Date: 07.01.2014
Time: 11:21:22 (GMT+0)
Zone: ''
Day of week: 2
Day of year: 6
Daylight s.: 0
Here's what I get by right clicking on my .dng file with my exiftool registry hack:
File Name : M07-1017.MLV.frame_000000.dng
Camera Model Name : Canon EOS 5D Mark III
Lens Model : EF24-70mm f/2.8L II USM
F Number : 2.8
Shutter Speed : 1/40
ISO : 2000
Focal Length : 50.0 mm
Light Value : 4.0
Exposure Compensation : 0
Frame Rate : 23.976
Date/Time Original : 2014:01:07 10:17:38
Orientation : Horizontal (normal)
Image Size : 1920x1080
QuoteHow much?
A bunch... like 3-5 MB/s and won't let you record certain sizes. Wonder if it had an effect on pink frames because I saw way less.
QuoteIt's most likely a conflict with the beeps (turn them off).
Even with full audio controls beep turns off the input... sucks I don't think the IC can record and play at the same time. I blocked beeps while recording.
So with today's commits (pre-buffering):
1728x1152 is continuous with audio if using fill method 3. 4 won't do it for some reason.
Thanks Danne and Shield. Both of you were right and it works now. :)
Quote from: 1% on January 08, 2014, 12:51:23 AM
A bunch... like 3-5 MB/s and won't let you record certain sizes. Wonder if it had an effect on pink frames because I saw way less.
Even with full audio controls beep turns off the input... sucks I don't think the IC can record and play at the same time. I blocked beeps while recording.
So with today's commits (pre-buffering):
1728x1152 is continuous with audio if using fill method 3. 4 won't do it for some reason.
This is for 7D right?
yep, 7D.
Audio doesn't seem in sync though... on a 10 min clip the wav was a little short, not long. Another one was a few frames too long only. 6D was longer when I tested but I don't think 100% in sync.
Just a heads up from my experience - I have been testing and diddling with this all day. I'm the geek type that will change just one setting and exhaustively see what difference it makes - anyway, I was fooling around with the resolution settings for the 3x crop mode, and forgot that I set it above 1920x1080 - shooting normally it was fine, but when I went above that it would lock the camera up.
Not a big deal - pull the battery and start over.
But...inexplicably this lockup disabled audio recording in the Canon menu. If you go off the wrong path as I did please be sure to verify the audio is still enabled. Got some fantastic footage of the kids tonight, and the audio file (while there) was dead silent. Went through all the menu settings and sure enough, the Canon menu had it disabled instead of the usual "auto".
Just wanted to share my experience. I would like to thank all the developers and testers; the MLV format is amazing and the ability to have accurate, synch able reference audio is a huge benefit. Thanks again guys - think how far you've come since last May.
Another side note - I let it record until the CF card filled up - it shut off "automagically" and everything was recoverable in post too.
On 6D/7D it works fine with canon audio off as long as the IC is powered (wav mode), tap on the mic and if the meters move you're good
Quotebut when I went above that it would lock the camera up.
Maybe related to the dma flags. Which resolutions did it?
i think so many people have audio sync proplem. why i dont have any problem in my 550d. i hve been shooting short films since mlv enable audio n i dont have any (repeat) any problem. everything works fine. audio sync is also very easy but i m so bad in color grading
http://www.youtube.com/watch?v=CVDQf91YmOM
atleast i shoot 50 clips a day and all audio sycn with video very well
Quote from: 1% on January 08, 2014, 06:49:59 AM
On 6D/7D it works fine with canon audio off as long as the IC is powered (wav mode), tap on the mic and if the meters move you're good
Maybe related to the dma flags. Which resolutions did it?
Oh something stupid like 2.5k - I was just bored and curious to see how long I could get it to go. I'll be sure to keep my eye on the meters.
The problem is the Magic Lantern software is too smart - it remembered that in 3x mode I was higher than 1920x1080. I'm used to software being much more stupid. I also periodically get a "Error -1" at the top, then I realized it was set to write to the SD card (Card 2). You guys have thought of just about everything; well done!
Sound was probably disabled by FPS override (and it couldn't be restored because of abnormal shutdown). It was important when recording H.264, but with raw it no longer needs to be disabled.
It gets pretty complex to handle all these tricks.
right...
so mlv_snd should enable audio again when it is enabled.
I guess this will result in a fight between the two codes.
Should just add audio to 5DIII, its AK4646 with slightly different command method.
I should also try audio on/audio off and see what happens to sync.
any idea of when you will include mlv_snd into the nightly builds?
Subbing. Thanks once again :)
Hey everyone, hoping for a bit of help here, I'm really interested in trying out the new MLV raw with audio for the 5d mk III but looking for a bit of help with installing the files.
I know there's a few others also having difficulty figuring out how to get the latest MLV developments working on their 5d Mk III's as I've seen step by step instructions for the mkII,
would anyone who's gone through the process be able to give us a step by step guide to installing these latest developments. I've got the October ML with Raw working at the moment,
and I've downloaded the modules kindly provided by G3ggo, do I just drag all the .mo files into the modules folder I currently have? Or is that totally wrong. Huge apologies in advance for asking but hopefully someone with a bit more ML experience out there won't mind walking me through it. Thanks guys!
Just to clarify, I presume I'm overwriting all the .mo files from the first post here into the 5d mk iii latest build (from 9th Jan)? If so, where does the 5d3_113.sym file go? MLV dump is the post processing program (ie. not for the SD card in the camera) is that about right? Sorry for the stupidly basic questions :-)
I think you can download the regular nightly build for your camera here http://builds.magiclantern.fm/#/
Then you just add the mlv_rec.mo and mlv_snd.mo from g3gg0 build from the first page in this thread. Conflict occurs if you load both raw and mlv recording in camera. At least this is what I have done lately.
Alternative is to download and install g3gg0 build removing the other nighly build.
Correct me if I,m wrong here
//D
Quote from: Danne on January 09, 2014, 01:51:32 PM
I think you can download the regular nightly build for your camera here http://builds.magiclantern.fm/#/
Then you just add the mlv_rec.mo and mlv_snd.mo from g3gg0 build from the first page in this thread. Conflict occurs if you load both raw and mlv recording in camera. At least this is what I have done lately.
Alternative is to download and install g3gg0 build removing the other nighly build.
Correct me if I,m wrong here
//D
That's pretty much what I did too. Seemed to work out fine for me.
Hey Danne,
Thanks for that info. Great I seem to have it all up and running. My only issue is that I have MLV audio running, but aren't seeing any options for audio under the audio tab. I'm seeing audio bars though on the main screen, but in the menus there's just three options, speaker volume, beep test and MLV sound - is this normal? on the 5d 2 mlv i know i've seen a screen with more audio options.
enable "MLV Sound" and enter the menu using Q, if you want to see the options.
Hey g3ggo!
Thanks for the help :)
I do have mlv_snd enabled in the modules, in the audio tab I have mlv sound switched to ON, when I press Q on MLV sound I only get three options, debug, trace output and sampling rate. I wasn't sure whether I should be expecting to see digital and analog output, headphone level etc as per the usual magic lantern or not? Is this right? Thanks again for all your hardwork, it's MASSIVELY appreciated!
no there are no further options to mlv_snd. it is just recording, usually from the internal mic.
Thanks for your help guys :-) I'm up and running :-) I'll let you know if I come up against any problems :-)
Sry but i cant still understand how to get mlv files to dng and that audio file too?
Edit:
Ok, found it. Bad in english so its hard to find that things.
(http://www.magiclantern.fm/forum/index.php?topic=7802.0 and http://www.magiclantern.fm/forum/index.php?topic=8447.0)
also here http://www.magiclantern.fm/forum/index.php?topic=9731.msg93586#msg93586
Here I'm again on this topic g3gg0. I installed your build and removed the other nighly build.
Now I can see the mlv_snd module but when I record (the global draw is ON), the global draw turn to off on the LV and when I stop recording I have a "error -1" on the top left. Did I miss something?
I found an answer that resolve my problem:
As Ted Ramasola said:
"if I would dare guess that the mkIII is the same in MLV menu as the mkII then you go to MLV menu and turn DISABLE G.DRAW to ON. I know, its mislabeled and I was wracking my brains as I thought turning it on would do as it say, but it actually does the opposite and DOES NOT turn Gdraw off when it is ON."
Now it seems to work fine. Thanks again!
why isn't mlv snd included in the nightly builds?
Okay a little bit of my experiences from testing today. I tested the mlv with sound, both plugged into a monitor and not on a 5d mk iii with 1050x 128gb cards. In the older RAW module I can record constant 1920x1080 @ 25fps. In this mlv module the camera only runs for a few hundred frames then cuts out, I have to record at a lower resolution 1728x972, and sometimes even then it isn't continuous. The sound however is perfectly in sync. The sound cuts out a second or so before the cam starts to skip frames. The other issue I noticed is that when connecting a monitor sometimes the monitor screen freezes up, when pressing record when this happens you see nothing but a frozen external monitor screen, but when processing the file you get a strobe type effect where you're seeing a constant flicker between the frozen screen and the thing you're recording.
The biggest issue I have is it not being able to run 1920x1080 @ 25fps. Any advice on settings that people recommend for getting the best bang for the buck on the 5d 3?
Hope the above info is useful to someone? :)
Hi, I'm wondering there's way to convert MLVtoDNG+WAV in one commnad file by Drag and Drop.
I know MLV2DNG can do it. But not convert WAV file from MLV.
Anybody know about the development?
this works http://www.magiclantern.fm/forum/index.php?topic=9731.msg93586#msg93586
Thanx Danne ;D
I use windows, though. ㅜ.ㅜ
Then use g3gg0 viewer/browser to convert http://www.magiclantern.fm/forum/index.php?topic=8447.0
I have a Camera model ID error showing up in ACR 8.3/CC/OSX 10.7.5 using the most recent mlv modules.
I've been discussing it in this post http://www.magiclantern.fm/forum/index.php?topic=9731.25 but was advised by Danne to post it here also.
Dngs created are coming into ACR with the ID/tag: 'Camera Raw 8.3 - ERR:1 md:0x 0 ml:0' and all I get offered [in ACR Camera profiles] is the embedded profile. I'd like to apply VisionLOG here.
Exiftool on the dng shows me the line:
Unique Camera Model : ERR:1 md:0x 0 ml:0
All other extraction methods on my normal raw > dng produce usable camera ID such as 'Canon 5D Mk 2' or 'Canikon'. I've updated to latest mlv_dump also.
Any help gladly received. Many thanks.
tim
5D MK2
a.d. build 24.10.2012 with added mlv modules.
OSX
UPDATE: This has now been fixed by me doing a clean install of ML on the CF card. Not sure where the problem was... :o
Dear g3gg0,
MLV is GREAT~~~~~ I am start using it now, very stable.
Just one request...
Can you make the recording information smaller like raw_rec.mo
Because I would like to use MLV for my new project.
Best regards,
Sean
Adding in the raw rec stuff like the little green camera, etc is in the works. MLV_REC just got to the point of being a viable replacement.
Quote from: MRozier on January 11, 2014, 04:45:40 AM
The biggest issue I have is it not being able to run 1920x1080 @ 25fps. Any advice on settings that people recommend for getting the best bang for the buck on the 5d 3?
Hope the above info is useful to someone? :)
I struggled with se same problem. Back then i was able to record contineously and suddenly only one in a while a cont. recording worked.
Yesterday I realized that I changed the audio sample rate to 48 kHz recently. I switched back to 44.1kHz and viola:
With 44.1kHz audio sampling rate MLV is contineous for me, while with 48kHz it skips after about 100-500frames.Build: Jan-11 5DIII 128GB KB
You should try some of the other buffering methods too. 4 isn't always the best.
And also the small size jpg settings in picture quality
Quote from: Danne on January 11, 2014, 11:56:36 PM
And also the small size jpg settings in picture quality
I believe this is not the case with 5DM3...
Try it, It makes a difference
Which one of the 4 (on 5DM3) ?
I use 3 and 4 on 7D and 6D. on 4 the 7D wouldn't record 1728x1152 continuously with audio, on 3 it did. Seemed to blast through the 1038 though.
So try each and see because I don't have a 5DIII.
5DIII + 6D also support the memory hack which should give you all of the memory that is possible. The hack only works when pic quality is raw. One of the jpegs may give you slightly more memory than raw + nothing but not too likely on these 2 bodies. Every camera has a different trick for maxing out the buffer size.
Hey everyone,
Thanks for the advice ill test that tomorrow. On another test last night I noticed that the connection of my smallhd ac7 via hdmi was causing regular issues, both in RAW rec and MLV, it certainly seemed to be slowing things down and locking things up. It's more stable in RAW only creating slight issues but totally unusable in MLV, every time I hit record the screen would lock up and I'd have to turn the cam on and off to reset.
is there a tutorial or link with instruction how to work with MLV format in windows, I've tried using cmd prompt with what little instructions I have found but from there it turns into a mess. Shouldn't be "this" difficult right?
Quote from: gravitatemediagroup on January 12, 2014, 06:22:08 PM
is there a tutorial or link with instruction how to work with MLV format in windows, I've tried using cmd prompt with what little instructions I have found but from there it turns into a mess. Shouldn't be "this" difficult right?
use MLVBrowsesharp to convert mlv raw into dng + audio or raw + audio
Quote from: gravitatemediagroup on January 12, 2014, 06:22:08 PM
is there a tutorial or link with instruction how to work with MLV format in windows, I've tried using cmd prompt with what little instructions I have found but from there it turns into a mess. Shouldn't be "this" difficult right?
I did a short tutorial for Windows with mlv+audio but on a 5d2 it should help.
Link below
https://vimeo.com/83449475
delete
I was doing some tests on sound sync and I loaded SND.mo and saw the meters come on upon checking my footage I was surprised not to see a WAV file,
I checked my cam and saw I missed turning MLV sound to ON in ML menu, - I see potential for trouble here when using this in a shoot.
Can devs tweak this to put safety nets?
Like, If MLV sound is not ON then Vu meters should not be active?
Or, you should not hear anything on your headphones when MLV sound is OFF.
or, when SND module is loaded, this should automatically turn ON MLV Sound by Default.
There is just too many "buttons to push" that the probability for error is high.
I just tried to use the whole MLV (shoot and dev) this week end. I have to say that everything works quite well! Hudge thanks to the whole team!!!!
I have a 5D MKIII with MLV_rec, 25 fps, 1920x1080, continuous recoding ok, CF card lexar 64Gb x1000 (+SD card), sound sync without any issue.
The MLVBrowseSharp.app works well too (except the preview), specially when the file is 10 Gb (split in .mlv (4.29Gb), .m00(4.29Gb), m01(1.8Gb) in the CF Card but seen as one file in MLVBrowseSharp.app) and the wav file is created as one file that is sync to the dng folder once put together in AE (with ACR)! Really great job.
The MLVViewer in the 5D is fantastic, too. My only asking (and I know it has been asked already) is: it woud be nice to have a smaller information area in the LV when recording as it was for the Raw_rec module...
Many many thanks again!!!! Your an amazing team!
Quote from: ted ramasola on January 13, 2014, 09:15:31 AM
Can devs tweak this to put safety nets?
Like, If MLV sound is not ON then Vu meters should not be active?
Yes. In audio-common.c, one has to modify SOUND_RECORDING_ENABLED (turn it into a function first) and make it aware of mlv_sound.
I'm not sure whether sound in Canon menu has to be enabled for mlv_sound, and whether this behavior is consistent on all cameras.
This kind of integration is tricky, a very large part of the audio code has to be rewritten.
in my ML audio settings I only have 3 menu items, but In the tutorial video I seen tons of other options/extra options that I don't see? Maybe these are settings that got placed or built into other settings in newer builds?
Quote from: MRozier on January 12, 2014, 05:28:50 PM
Hey everyone,
Thanks for the advice ill test that tomorrow. On another test last night I noticed that the connection of my smallhd ac7 via hdmi was causing regular issues, both in RAW rec and MLV, it certainly seemed to be slowing things down and locking things up. It's more stable in RAW only creating slight issues but totally unusable in MLV, every time I hit record the screen would lock up and I'd have to turn the cam on and off to reset.
I am using an EVF and it locks up from time to time too or gives me corrupt frames. Too bad, as my shots are so much more stable with this extra-point of contact. I wished the devs could invest some love into this, but I fear with an external monitor, we are a small minority.
Most of my shots turn out very usable. You have to make sure to switch off the camera first and wait for five seconds before switching off the EVF.
Hi there
Would anyone getting continuous shooting at 25fps at 1920x1080 on the 5d mkIII mind sharing their settings, I'm getting that no problem with raw_rec but not managing to get anywhere close with mlv.
It'd be good to know if others have had a bit of success with those settings. I'm not card spanning at the moment as I'm recording to CF cards and keeping magic lantern on the SD. Any other tips or advice from MLV users? Thanks in advance :)
I got myself a transcend 128gb 1000x card today and it actually is a little bit faster than my 64gb 1000x transcend.
On my 5d mark 3 I get continuous with audio on following
1920x1080 24p
I can actually use canon gui and set manual audio while recording. Cool!
1920x1080 25p
Continuous without audio. With audio on it,s actually continuous until it splits the 4gb chunk. Here it,s kin of borderline. Sometimes it continues sometimes it stops, skips frames.
*update. Audio in auto mode gives me full continuous 1080x1920 25p *tried again and audio auto stopped around 10gb. No luck in 25p and audio continuous
For 24p I don,t use extra hacks on for audio meter in canon gui. G.draw off but I get it continous with global draw on with focus peak on as well in 24p. Also use pictue quality set to s2
for 25p I use extra hacks on g.draw off
Tried the modified mlv.rec.mo but made no difference when 4gb split.
Thanks g3gg0
QuoteI'm not sure whether sound in Canon menu has to be enabled for mlv_sound
If you have audio controls and power the IC no, if you don't yes. Otherwise its muted/missing.
To MRozier
As I'm able to have a continuous recording with 25fps at 1920x1080 on the 5d mkIII, the settings that works for me are:
In MLV-Rec -press(Q)- Aspect ratio 16:9 / Disable G. Draw ON / Frame skipping OFF/ Preview Auto/ Start delay.... to Memory hack OFF / Extra Hacks ON / debug trace OFF (if this debug is ON, sometimes I can't have continuous recording) / the others OFF / Buffer 3 / CF buffers 3 or 4
For the sound, don't forget to turn off Beep, test tones
In Canon menu, sound is on manuel (I have no issue with sound for the moment even with 10 GB file)
As Danne already said, use only the g3gg0 builds (don't try to mix it with other module)
I use a CF card Lexar x1000 64GB (for mlv files) and SD SanDisk 95MB/s 32GB (for ML Firmware)
I don't know if it could help or not as I have no special settings indeed.
Quote from: Sthirasukha on January 15, 2014, 12:14:37 AM
To MRozier
As I'm able to have a continuous recording with 25fps at 1920x1080 on the 5d mkIII, the settings that works for me are:
In MLV-Rec -press(Q)- Aspect ratio 16:9 / Disable G. Draw ON / Frame skipping OFF/ Preview Auto/ Start delay.... to Memory hack OFF / Extra Hacks ON / debug trace OFF (if this debug is ON, sometimes I can't have continuous recording) / the others OFF / Buffer 3 / CF buffers 3 or 4
For the sound, don't forget to turn off Beep, test tones
In Canon menu, sound is on manuel (I have no issue with sound for the moment even with 10 GB file)
As Danne already said, use only the g3gg0 builds (don't try to mix it with other module)
I use a CF card Lexar x1000 64GB (for mlv files) and SD SanDisk 95MB/s 32GB (for ML Firmware)
I don't know if it could help or not as I have no special settings indeed.
How can you get audio? (wich build are you using, and the settings... thank you).
can the sound recording option work with old raw rec?
it would be great for me because its way easier to process the old format than the new one. (and you can record a little faster)
Quote from: budafilms on January 16, 2014, 06:10:26 AM
How can you get audio? (wich build are you using, and the settings... thank you).
You just quoted the settings being used ???
Reading the OP is probably a good step to getting started.
Quote from: arturochu on January 16, 2014, 06:59:33 AM
can the sound recording option work with old raw rec?
AFAIK,
if raw_rec is kept after mlv_rec becomes more stable, it will only be in a stripped down maximum speed version. In other words, don't expect audio.
While we, re on the maximum speed subject. I, ve always been curious if the spanning feature applied to raw_rec would give that final extra mb to be able to record close to continuous in 50fps in full hd :P
Testing the last 2 MLV nightlies including one from a.d. for the mkII, is it a current limitation for MLV raw with sound to stop at a little past 10 minutes?
I notice the buffer graphs seem to be managing it until past 10 minutes when it would say skip and recording would stop. tested different buffer methods at 1728 x 972 16:9.
35.9 gig seem to be the limit in my tests (40859MB on the camera lcd )
Tested another resolution, 1728 x 864 and it went past 11 min but again it stops at 40918MB as indicated on the LV screen.
Quote from: Danne on January 16, 2014, 08:26:24 AM
While we, re on the maximum speed subject. I, ve always been curious if the spanning feature applied to raw_rec would give that final extra mb to be able to record close to continuous in 50fps in full hd :P
i dont think 50fps in fullHD can be posible, but that would be awesome!
50fps is very possible, just not continuous ;)
I'm suprised you guys are getting stops with the sound, I've gotten none. If it gets continuous, it stays.
What camera and build are you using? 25fps full hd?
To Budafilms:
I'm using the g3gg0 build (mlv_rec) from the Jan 10th 2014 (my birthday... nice gift g3gg0 and thanks again!!!)
I tried to put the mlv_snd.mo in the "classical" module folder (with raw_rec.mo, Arkanoid....) but it crashed (as Danne said)
So, my module folder has exactly (no raw_rec included):
5D3_113.sym
autoexpo.mo
dot_tune.mo
file_man.mo
mlv_play.mo
mlv_rec.mo
mlv_snd.mo
pic_view.mo
silent.mo
smpte.mo
trace.mo
To (amazing) developers:
I hope you won't stop the sound module. I'have being waiting this for such a long long time! Pleaaaaase!!! Don't go back. And tell me there are a lot of people that are able to use the sound.
Heh, I've tried this on most cameras now. Even EOSM works for a while... afterwards tho it hangs :( At least I got an error message this time and it said it dumped the log. Not the fault of mlv_snd, mainly the camera.
Ok, so here's my tests in terms of filling up a 64gig card.
All tests run at 1728 x 972 16:9 all tests run with MLV SOUND ON.
5D mkII tests
Jan 16 nightly
skip/stop at 10'7" 40,800MB (41 GB)
Jan 14 by a.d.
skips/stops approx 10 min 40,859MB
Jan 8 nightly
skip/stop at 10'6" 40,721 (41.4 GB)
Jan 6 nightly - MLV_REC + Jan 14 a.d. -MLV_SND
OK CONTINUOUS FILLS UP CARD.
Jan 6 a.d. all modules.
OK CONTINUOUS FILLS UP CARD.
Jan 6 a.d -MLV_REC + Jan 14 a.d. - MLV_SND, FILE_MAN, MLV_PLAY
OK CONTINUOUS FILLS UP CARD.
7D tests.
Jan 16 TL
skips/stop at 10'11" 40,929MB (39.9 GB)
Jan 14 TL
skips/stops 10'10" 40,907MB (39.9GB)
Jan 9 TL
skips/stops 10'11" 40,929MB (39.9GB)
Jan 8 TL (incomplete audio features in Audio tab ML menu - did not run test)
Jan 7 TL
OK CONTINUOUS FILLS UP CARD.
Jan 6 TL
OK CONTINUOUS FILLS UP CARD.
So I guess something was wrong starting from Jan 8 builds?
you say from jan 8th the recording is continuous and that is a bug?
Quote from: g3gg0 on January 17, 2014, 10:11:13 AM
you say from jan 8th the recording is continuous and that is a bug?
I wrote starting with the 8th there must be something wrong.
I listed the tests from recent to older.
Tried the build from first post here and it gives me 40gb file than stops. Gonna try nightly build in a while.
5d mark 3
Quote from: ted ramasola on January 17, 2014, 10:20:12 AM
I wrote starting with the 8th there must be something wrong.
I listed the tests from recent to older.
ah okay thanks!
now i understand.
I can't fin the module for sound. Someone can place the link? thanks!
Quote from: budafilms on January 19, 2014, 04:12:42 AM
I can't fin the module for sound. Someone can place the link? thanks!
First post. If your using Tragic lantern, its in the TL NBs. A.D also included it in his builds for the 5D2.
The pre allocation commit was the 7th. I am seeing it stop early too, no message or anything. It would have ended up in the nightly on the 8th.
I'm only getting like 20 - 25gb of 1728x1152 now. Skips with no warning.
Yeah, already nailed down.
there are only a few file handlers that are possible being left open.
on 5d3 its ten files max.
I tried replacing the files of the nightly builds with the files of the mlv folder, but it doesn`t work... Could anybody, please, tell me what is the exact way to make it run... I used the 19 Jan 2014 build and then tried to add the files with replacing the existing... and tried to delete all and then placing the mlv files... Thanks.
Playing with the tragic lantern builds for 6d form 3 and 18 Januari.
Sound recording issues with mlv_rec are fixed, I can record multiple clips with sound now, THANKS !!!
BUT
Is it me or is the sound recording volume really low (compared to previous builds), I have to crank up the volume on my mac to hear the wave files.
In Canon menu I have the wind filter and Attenuator OFF, so that's not it.
In ML menu I tried analog gain on 23 db and on 32 db, not much different.
digital gain is all off.
Also the meters from ML aren't moving that much, in canon menu, the sound meters are moving a lot further on the scale...
What's happening here ?
The audio wasn't touched, just mlv_snd. Canon always uses a little digital gain with it. If you add a tiny bit just to 1 channel it will get hot pretty fast.
Probably need to do some more testing with it, and check audio settings on the previous builds I used, maybe there was some digital gain there.
(or maybe I touched some of the many many settings in the ML or Canon menu)
Digital gain in camera has no benefits on digital gain in post on computer (I guess ?), and with digital gain in camera there's more risk of "clipping" audio, so probably better to leave it off.
I was under the impression audio was recorded through canon audio menu settings? At least it is when I change audio levels manually while fiming mlv raw.
I've been looking for an mlv to dng batch converter for windows but there doesn't seem to be any. Found the mlv2dng.exe but it's not batchable. Did I miss a thread or nobody hasn't build one yet?
Thanks
Quote from: feureau on January 20, 2014, 07:21:49 PM
I've been looking for an mlv to dng batch converter for windows but there doesn't seem to be any. Found the mlv2dng.exe but it's not batchable. Did I miss a thread or nobody hasn't build one yet?
Thanks
I use the MLVBrowserSharp.exe to do batch MLV extractions.
Gary
Quote from: gary2013 on January 20, 2014, 07:27:21 PM
I use the MLVBrowserSharp.exe to do batch MLV extractions.
Gary
What is your workflow? I have the same issues. I have to manually create a DNG folder for each MLV file. Is there an easier way to batch several MLV files?
Thanks!
Steve,
If you use resolve after converting MLV to DNGs, just let the MLVbrowsersharp extract all your MLVs to one folder as resolve can automatically see them as belonging to one clip when you point resolve to the folder containing all the dngs. At first I thought I needed them to be in separate folders but I got use to it later on. I guess it's more organized if they are in separate folders though but its more convenient that theyre not in folders so resolve can show you all the Thumbnails of your clips in one window rather than when they are in individual folders.
QuoteI was under the impression audio was recorded through canon audio menu settings?
It configures audio, asif is what records it. Canon stuff runs when the menu is accessed and record start, etc. After that ML configures the settings you've set.
Quote from: gary2013 on January 20, 2014, 07:27:21 PM
I use the MLVBrowserSharp.exe to do batch MLV extractions.
Gary
Thank you \o/
I have a big issue. On MLV mode, the camera skips frames (even with skip frames option off). There are about 3-4 pink frames in the whole video of about 250 frames. The video is choppy. Video shot at 1920X1080 (Canon 25fps)
Workflow: MLV Browser; After Effects CC.
Other programs: VLC;Windows 8 on PC.
Equipment: Canon 5D3 (Sigma 35mm F1.4), Lexar CF 1000X 32GB; Scandisk SD 95kpbs Extreme Series.
I am using the 20 January 2014 build (no separate SND files imported from g3gg0's module on first page).
What am I doing wrong?
Global draw on or off? Raw_rec?
Rec_rec off. Global draw on.
Global draw can cause pink frames.
Ok.
Will try w/o.
Another thing - how can I prevent jerky video in post. 25fps looks jerky on vlc which it shouldn't. Format QuickTime.
depends how you encoded it, ae defaults to 30P for image sequences, etc
Quote from: ted ramasola on January 20, 2014, 07:59:51 PM
Steve,
If you use resolve after converting MLV to DNGs, just let the MLVbrowsersharp extract all your MLVs to one folder as resolve can automatically see them as belonging to one clip when you point resolve to the folder containing all the dngs. At first I thought I needed them to be in separate folders but I got use to it later on. I guess it's more organized if they are in separate folders though but its more convenient that theyre not in folders so resolve can show you all the Thumbnails of your clips in one window rather than when they are in individual folders.
Thanks Ted. This works well for a few mlv files but if I have many mlv files MLVbrowsersharp throw a lot of excemption errors and ultimately crashes.
For a few files though Resolve does indeed import them as individual files even though all of the DNGs are in the same folder. After Effects imports the DNGs in a similar way as Resolve.
Is there any kind of rawanizer to batch convert either into raw or dng creating subfolders for each shot? I tried to convert first with BrowseSharp to RAW, and then import them in RAWanizer, but it crashes... Also, the MLV.RAW.WAV extension make RAWanizer treat it as a RAW file...
Started a new thread on this, but perhaps it just belongs here.
Recording typical 1920x1080 video tonight; nothing too crazy. For some reason ML locked up, which is no big deal, but after the restart the Canon menu's audio became disabled so I lost some good audio tonight.
I normally notice the audio meters but tonight I didn't see them.
Here's my question - back in the day in order to use the 3x mode one had to set "FPS override" (in my case I set it to 23.976). However if you do this now, one cannot see the audio meters at the top. I have the "Raw Histobar" at the top. Once I disable the FPS override the audio meters come back.
So, do I still need to do the FPS override trick? All I care about shooting is 1920x1080 @ 24FPS in normal mode as well as 3x crop/zoom mode. In the past it was recommended to set the Canon menu for 30 FPS then dial down the FPS override to 23.976.
Also, if it's still required, is there any way to get the audio meters back with it on? When shooting handheld stuff I just set the Canon audio menu to "auto"..
Thanks guys.
Hi,
we've been using ML a lot during a large (and important) production, it has been amazing and we've been able to get footage that would otherwise not have been possible. big thanks here to everybody who contributed. this is such a game changer!
we're a swiss production company shooting with two cameras running a build from 07.jan with the snd module from 4th jan. we convert using the mlv dump on os x with the script by dubzeebass (also thanks for that) We shot about 4 TB of footage. All documentary style with sound.
two problems occured.
1) out of about 200 shots about two had bad first frames. I could not open the first dng file in photoshop and davinci would therefore not recognise the sequence.
I saw one thing, instead of the normal vertical stripes correction:
Vertical stripes correction:
1.000 1.000 1.001 0.994 1.003 0.998 1.001 1.003
MLV Dump gave me those:
Vertical stripes correction:
1.000 1.000 1 1 1 1 1 1
I tried running the script on different locations but it seems to be an issue with the MLV file.
dublicating the second frame solves the issue with the sequence of course... but the problem is still a bit weired.
2) In about 3 shots out of over 400 we had the issue of skipped frames, they are just missing. The Sound is fine.
I can not pinpoint any logic behind it. The main Problem is, that the camera won't stop and we're therefore not aware of the issue on the shoot.
I could provide the MLV Files for analysis if it would help. Non of the files with a problem was spanned)
(but I don't want to post them publicly on the forum)
We're using the Sandisk 160MB/S 128GB Cards.
Thanks!
thanks for the feedback!
very nice to see it is actually being used and people are statisfied.
i am sorry that there are dropped frames :(
sure i will have a look into this - maybe i can find something that helps in tracking down the bug.
can you upload the files to a google drive?
of course i will not disclose any content.
Quote from: looyod on January 22, 2014, 11:56:16 AM
2) In about 3 shots out of over 400 we had the issue of skipped frames, they are just missing. The Sound is fine.
I have ran into this a few times my self, seems totally random and rather rare. I am using a 5D2, I have also had some issues where the camera will not stop recording when the button is pushed. Going into the menu shows stopping... but never does (Well is does stop recording frames but never returns to normal function camera must be turned off and on). This issue is just as rare as the other.
I'm new to this so this might sound stupid to most but I am having a issue with just getting the new nightly build on my 5d3 so I can use the MLV, I dragged and drop on my SD card and load modules but I am getting a error. How do I fix this or install the new build, I'm on a Mac and I am using a Canon 5d3 that has the ML Raw on there already..
Hello to everyone.
I'd like to tell you that we shot a shotfilm in .mlv the last two days. We were shooting 1080p25 with Global Draw on and mlv_snd on a couple of Komputerbay 64GB 1000x Cards. We did produce around 400GB of footage.
We had almost no problems, only had to warm up cards everytime we switched oherwise we had some frameskipping.
Also the remaining free space on the card was incorrect for some reason.
So far I did convert all the .mlv files to .dng+wav with the batch script on osx and now I am trying to figure out how to produce proxies with audio in a proper manner.
Thanks so far!
You could use this MLVbrowsesharp to convert your mlv to raw: http://www.magiclantern.fm/forum/index.php?topic=8447.0
Then use RAWanizer to export proxies and the DNGs. http://www.magiclantern.fm/forum/index.php?topic=5557.0
Tested the Jan 25 Nightly on the 5DmkII.
With MOVIE RESTART=ON MLV raw video will not stop recording. If you press SET to stop, it will start by itself.
This issue is not present with older builds like the Jan 6th NB and by a.d.
This is suppose to be active in H264 recording and should not have an effect in Raw recording.
HDMI MONITORING TEST
Tested playback functionality.
Immediately after recording a raw clip, I went to File manager> view mlv file.
Playback of COLOR or FAST was possible and it showed on HDMI monitor.
However after EXIT of mlv playback screen went Black and no function possible.
Had to remove battery to reboot camera.
On Older build using Jan 6 nB or A.D. Playback was possible and EXIT from playback was OK,
liveview came back and additional recording was possible.
CAPACITY BEYOND 40GIG TEST
Jan 25 Nightly build will still skip and stop approx, 39GB (Canon menu), 35.9GB (PC properties, 38,582,760,448 bytes)
The Jan 6 NB and by A.D. does not have this issue and will record till your CF card is full.
NBs after Jan 6 up to the 25th will Stop at approx that size or around 10minutes. This also applies to the 7D.
...
Out today shooting some 2-4 min clips. One clip was 3:16 and the associated .wav file was 3:07. It just didn't record the last 10 seconds for some reason. I was able to sync it up with my external source so I was able to determine that was the problem. Clips just before and after this were fine, and there was no lockup or any other problems.
Would anyone like me to upload the mlv file somewhere?
Quote from: Shield on January 26, 2014, 05:45:38 AM
Out today shooting some 2-4 min clips. One clip was 3:16 and the associated .wav file was 3:07. It just didn't record the last 10 seconds for some reason. I was able to sync it up with my external source so I was able to determine that was the problem. Clips just before and after this were fine, and there was no lockup or any other problems.
Would anyone like me to upload the mlv file somewhere?
on my tests using 7D and 5DmkII using various resolutions the results are always as follows.
when recording MLV+SND, the wav file would always be
LONGER by 7-10 frames.
It would be ahead by 1-2 frames from the video so it needs to be nudged back to sync.
Knowing this and you won't have problems syncing in post. H264 was also off by 1-2 frames so this is not an ML issue.
However, if your recording stops ABRUPTLY, due to card full for instance, the VIDEO would be LONGER and the WAV file much shorter.
Quote from: ted ramasola on January 26, 2014, 06:55:20 AM
However, if your recording stops ABRUPTLY, due to card full for instance, the VIDEO would be LONGER and the WAV file much shorter.
yeah thanks for the feedback.when card gets full, I cannot clear buffers anymore.
then you lose some megabytes of audio.
not good, right. But I don't know a quick solution.
Can you monitor the file size and end recording before card full, giving a chance to clear the buffers?
With raw_rec, I create a backup.raw file for this. Whenever the card gets full, I delete that backup file and try to save the buffers again.
Last time I checked (months ago) it was working quite well.
@Audionut: this requires keeping track of all file writes since the last time Canon code reported free space. Possible, but requires some major refactoring (maybe a file i/o backend that wraps around Canon calls in a way similar to memory backend).
I filled my 64gb card three weeks ago and with Alex wizardry (I think backupsolution) I managed to extract my files. There was even a messaged asking if my card was full :).
Posted here as advised by Ted:
Ted, also to add if the raw video is taken when the sd card is the preferred card, the 5D3 hangs. This is technically a glitch.
While I understand you can set your preference and all but I am sure there is a programming way to prevent the sd card from leading when it clearly is not meant to.
My take is that it will discourage mass adoption of the ML platform small things like these are not addressed. Pretty much like texting changed for ever when predictive texting was introduced.
To add my test on the 7D, testing Jan 25 TL.
MLV Raw+Snd
It still skips and stops at approx 10 min or 40,929 MB ( 39.9GB )
Even though it recording constantly at 77.3/80/76 MB/s 9%idle.
The last Tragic Lantern version that does not stop at 10 minutes and will fill up card is Jan 7.
(This probably coincides with the Jan 6 a.d build for the 5d2-last build with no skip/stop problems)
after that up to jan 25 will skip and stop at 10min or 40gig.
Also to add an observation which I hope will help, I reverted back to Jan 7 TL and did a capacity test again.
I observed that the LV info keeps on updating constantly and the CF card activity led light keeps flashing but upon reaching approx 10min
or 40gig which is the threshold for other builds before stopping, I notice every info stop updating and the CF activity led stops flashing, like the cam froze for a few seconds,
but after maybe 2 and 1/2 seconds the activity and led flashing continuous and so does recording.
Quote from: ted ramasola on January 26, 2014, 06:55:20 AM
on my tests using 7D and 5DmkII using various resolutions the results are always as follows.
when recording MLV+SND, the wav file would always be
LONGER by 7-10 frames.
It would be ahead by 1-2 frames from the video so it needs to be nudged back to sync.
Knowing this and you won't have problems syncing in post. H264 was also off by 1-2 frames so this is not an ML issue.
However, if your recording stops ABRUPTLY, due to card full for instance, the VIDEO would be LONGER and the WAV file much shorter.
The 5d3 is different; never once had the audio (at least lately) not sync up perfectly with the video. This wasn't an issue of running out of card space, lockup, heat, anything else. Let's say in the span of 15 minutes I recorded 4 two-three minute clips all without shutting off the camera. Clips 1,2,4 were fine. Clip 3 the audio shut off about 10 seconds early. No idea why.
Quote from: ted ramasola on January 26, 2014, 12:32:49 AM
You could use this MLVbrowsesharp to convert your mlv to raw: http://www.magiclantern.fm/forum/index.php?topic=8447.0
Then use RAWanizer to export proxies and the DNGs. http://www.magiclantern.fm/forum/index.php?topic=5557.0
Thanks for the reply Ted,
I do make my proxies with Resolve right now, but that way I can't have them with audio. Rendering the individual source clips tries to render the sources (.dngs) audio, which is non existent.
As I am on OSX I couldn't try RAWnizer, but by having a look at it, the routine is not gonna give me proxies with audio either.
We were shooting with a slate, so we did synchronize by hand which was time consuming with several hundred takes.
I'll have another look at the mlv_snd module thread, maybe it's already clear how big the offset in the beginning of the clip is. That way we could produce proxies with no sound through Resolve and merge them with the appropriate audio with a ffmpeg script.
I'll keep you updated if I find a proper solution.
I just updated to the Jan 27 ML nightly build for the 5Diii with the MLV_SND module. Everything seems to be working well so far except for one thing:
When recording MLVs (Global Draw ON, all modes; Global Draw Disable OFF), I have the crop overlays, audio meters, raw histogram, etc. available to me when I first turn the camera on. But as soon as I start recording a clip, they disappear, and they do not reappear when I stop recording. I have to turn the camera off and then on again to get them back. Anyone know what is going on?
I can confirm that this was not an issue with the Jan 7 nightly build that I was running before I updated.
UPDATE: I've been able to resolve this. Turns out that I had my card on write protect, so it was reverting to my old settings every time that I turned it off. Unlocking it, turning Global Draw ON, Liveview seemed to address the issue.
Dear developers,
Is it possible to add TIMECODE in MLV file and the Audio WAVE file?
So we can auto sync the audio in Davinci Resolve...
5D3 can render timecode for h264 video.
Best regards,
Sean
Quote from: crazypig on January 27, 2014, 03:48:07 PM
Dear developers,
Is it possible to add TIMECODE in MLV file and the Audio WAVE file?
So we can auto sync the audio in Davinci Resolve...
5D3 can render timecode for h264 video.
Best regards,
Sean
Yes this would be awesome and a huge time saver.
Hello! I made a batch mlv to dng converter for windows!!! Take a look!
http://www.magiclantern.fm/forum/index.php?topic=10198.msg98284;topicseen#msg98284
Quote from: bnvm on January 27, 2014, 05:08:33 PM
Yes this would be awesome and a huge time saver.
maybe i dont understand - but the audio is already in sync to video. maybe a frame off as i've read.
also MLV has microsecond accurate timestamps per frame.
not sure what to add else...
Quote from: g3gg0 on January 28, 2014, 01:49:13 AM
maybe i dont understand - but the audio is already in sync to video. maybe a frame off as i've read.
also MLV has microsecond accurate timestamps per frame.
not sure what to add else...
let me explain ..as of now we have to sync audio manually by draging the audio below the video clip for every clips but with time code all video and audio can sync automatically so it saves lot of time
I think it is impossible because mkv gives separate frames (NLE treats them as an image sequence) plus separate sound file. Having all synced on the timeline from the beginning is possible only when sound and image are "baked in" into one file.
Black Magic Cinema Camera is using TIMECODE to sync the CinemaDNG file and Audio file in Davinci Resolve
In Black Magic Cinema Camera, you can also output the timecode to external audio recorder...and sync it in post easily.
Now ML RAW just like Black Magic Cinema Camera. Recording DNG files and a WAVE audio file....(only different is without timecode for the DNG files and WAVE file, so we can't sync it easily by software)
Because there have at least 100 footages in each video production project. If we sync each footages by manually, it take a long time.....
g3gg0~~~ Please add the TIMECODE in each MLV file...
Quote from: crazypig on January 28, 2014, 06:20:43 PM
g3gg0~~~ Please add the TIMECODE in each MLV file...
as ive already described, MLV file format has
MICROSECOND EXACT TIMESTAMPS FOR
EVERY KIND OF
INFORMATION. *
just in case i didnt emphasize it strong enough.
and as it already has this information, it doesnt make sense to add any more timing information.
when exporting into DNG, you separate that timestamps from each other.
i dont have any of the tools that you use, but cant you simply tell it to pick up the .dngs and the .wav in one directory as a single A/V pair?
*=depending on some circumstances it may happen that you are 1-3 frames off.
I think his issue is that none of the tools support any of the timecode info. Stuff like H264 can have a timecode track the NLE can read but a sequence of DNGs and a wav file do not. I'm not aware of any separate TC file format which would solve this unless the video was converted by MLV dump into video/audio/TC in one file.
thats the point.
MLV has that timecode information implicitely.
there is real time clock plus the timer value on every block start.
i dont know how the tools would recognise the information from DNG and WAV only.
but the information is there and is happily printed when you pass -v as parameter.
Quote from: g3gg0 on January 29, 2014, 08:26:11 AM
thats the point.
MLV has that timecode information implicitely.
there is real time clock plus the timer value on every block start.
i dont know how the tools would recognise the information from DNG and WAV only.
but the information is there and is happily printed when you pass -v as parameter.
Dear g3gg0,
I try to send you Blackmagic Pocket Cinema Camera CDNG and WAVE file to you tomorrow .
Maybe can find out how they paste the timecode information that can recognise in Davinci Resolve.
Really really thank for your great help.
Best regards,
Sean
@ g3gg0
I dont know if this is a known bug but i have it constantly and i think it is easy to reproduce (my camera is doing this every time)
5D3, ML build 28-01-2014 (as far as i remember it was also present in the previous builds). CF komputerbay 1000x 64BG, Sd sandisk extreme 45mb/s.
When I record with mlv_rec and have sound recording ON and when i am in the 3X crop mode camera hangs every time when frame skips and than requires battery out. Every time camera is not responding. FPS override doesnt matter. It is like that ONLY when sound recording is ON (tested with 48khz). When it is off than after frame skip camera normally comes back to idle.
This can be extremely annoying especially when somebody forget to turn on FPS override before entering crop mode - with default 30fps it skips quite fast. This can really ruin some important shot because bringing the camera to life lasts 20-30 seconds.
@kgv5:
thanks for the detailed bug report, will check tonight.
@crazypig:
hm as CinemaDNG is another story which chmee is a specialist for,
i am not sure if i can really implement it.
but i can have a look into it.
@1%
tested jan 29 nightly for 5D2 and there is something really wrong with it as it cannot record past 3 seconds max, sometimes just 1 sec, depending on the buffer method.
also the same with the 7D module you sent.
Check the other thread but I think I got it:
https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/commits/63b6cfad34ee8b6b8566f508152e8af3f23d5520
It only showed up as a problem on CF cameras.
Quote from: kgv5 on January 29, 2014, 01:16:46 PM
@ g3gg0
I dont know if this is a known bug but i have it constantly and i think it is easy to reproduce (my camera is doing this every time)
5D3, ML build 28-01-2014 (as far as i remember it was also present in the previous builds). CF komputerbay 1000x 64BG, Sd sandisk extreme 45mb/s.
When I record with mlv_rec and have sound recording ON and when i am in the 3X crop mode camera hangs every time when frame skips and than requires battery out. Every time camera is not responding. FPS override doesnt matter. It is like that ONLY when sound recording is ON (tested with 48khz). When it is off than after frame skip camera normally comes back to idle.
This can be extremely annoying especially when somebody forget to turn on FPS override before entering crop mode - with default 30fps it skips quite fast. This can really ruin some important shot because bringing the camera to life lasts 20-30 seconds.
i m sory. this is not an issue for me
Quote from: ted ramasola on January 30, 2014, 04:40:15 AM
@1%
tested jan 29 nightly for 5D2 and there is something really wrong with it as it cannot record past 3 seconds max, sometimes just 1 sec, depending on the buffer method.
also the same with the 7D module you sent.
fixed that today, in parallel to 1% ;)
is there any way to share pic here? when i import dng and audio in davinci resolve, in date created the audio and video doesnt match and i have a picture of that. i would like to share that here
Quote from: andy kh on January 30, 2014, 04:35:48 PM
is there any way to share pic here? when i import dng and audio in davinci resolve, in date created the audio and video doesnt match and i have a picture of that. i would like to share that here
The date seem the same, only the time is slightly different. I think time stamp is generated upon extraction, not sure though.
Quote from: ted ramasola on January 30, 2014, 05:40:02 PM
The date seem the same, only the time is slightly different. I think time stamp is generated upon extraction, not sure though.
yeah u r rite. when i try to auto sync only the first audio sync with all other video clips so there is only one same audio for all video clips
Dear all developer,
2 X 3 secs BMPCC video with sound
CDNG + WAVE
Please download the file from following link
http://www.sendspace.com/file/ugbt1s
http://www.sendspace.com/file/9pjqxt
two 3 secs video with sound
Best regards,
Sean
g3gg0, did you manage to reproduce mlv 3x crop mode frame skip crash bug?
just came home.
nope, could not reproduce it yesterday. recorded ~20 videos with various resolutions and all worked as expected.
but after looking at the code again i have a suspect.
i guess its a problem with mlv_snd not being able to allocate a buffer and not continuing ASIF buffering.
for that reason audio would not be able to be stopped. looks like the case that could cause such behavior.
a) can you provide me a trace of mlv_snd?
enable trace module, enable tracing in mlv_snd and upload the log where the error happened.
to make sure the logfile is written completely, poweroff the camera and wait a minute until you remove battery.
b) can you try it with the next nightly?
but it doesnt make any sense to me to use mlv_snd while allowing frame skipping.
it would throw away a few seconds of audio randomly.
g3gg0, here it is trace log:
https://www.dropbox.com/s/lwsdv6c4pfe02sb/mlv_snd.txt
Quote from: g3gg0 on January 30, 2014, 11:09:29 PM
recorded ~20 videos with various resolutions and all worked as expected.
So after frame skip in 3x crop mode your cam went back to idle and became fully operational?
my settings (5d3):
canon menu 1920x1080 25fps
mlv sound - sampling rate 48khz
movie- fps override OFF
mlv_rec- ON 1920x1080 16:9, GD OFF, frame skipping OFF, preview auto, mem hack ON, extra hack ON, buffer fill 4, CF only buff 3, spanning OFF
Sure, tomorrow i will check nightly.
uhm it seems to be something different. this trace is totally clean. :(
can you also start tracing for mlv_rec?
... deleted .
Tested Jan 30 TL.
1728 x 972 16:9 buff-2
Capacity test OK fills up Card 64gig.
Recording now continuous and no longer stop at 40gig or 10min.
Movie restart "bug"
I notice a "quirk" I did not find in the 5dmkII.
-when movie restart is ON, in MLV recording, upon manual stop of MLV raw recording H264 starts to record.
Suggestion to have it like the 5DmkII where it does not start h264 rec upon stop of MLV record. OR to disable and grayout this feature on the 7D when MLV raw video is active.
Another side effect when restart is ON is that when mlv recording has filled up CF card, instead of a proper auto stop it crashes and freezes the camera requiring a battery pull.
Recording with HDMI monitor attached
-I noticed that there is a corrupted single frame at the start of a clip and sometimes somewhere in the mid part of clip.
-Playback of MLV video via file man is OK.
Below is sample of corrupt frame in footage, either one frame at start or somewhere mid way.
(https://scontent-a-dfw.xx.fbcdn.net/hphotos-ash3/t1/1601045_637802719610130_791701745_n.jpg)
@g3gg0
(EDIT) Build from 31-01-2014 didnt fix the issue.
But I think have found something more about this, 1920x1080 in fact seems ok, i set higher resolution by mistake.
It constantly hangs with these resolution settings and sound recording ON (at least i tested those) : 2560x1320, 2240x1320, 2880x1080 (seems that FPS doesnt matter)
Tried couple times more, also with my second CF card - sandisk extreme pro 32GB 90mb/s UDMA7 - the same issue.
So please try this, i wonder if you will be able to replicate the bug.
These resolutions are not very usable anyway (very short recording) so it is not really a huge issue and it is not so urgent. It is worth solving though because it could make somebody miss the important take (like I did :( )
Here are new logs from snd and mlv:
snd next attepmpt:
https://www.dropbox.com/s/vjfc342ge9xv1st/mlv_snd%20v2.txt
mlv_rec:
https://www.dropbox.com/s/8scx3gmy214cjl2/raw_rec%20v2.txt
One more thing:
When i am using all above setting and set global draw OFF (while recording) it doesnt comes back automatically after stopping recording but i assume it should. It does comes back when i press canon menu button (canon menu ON and than OFF). Could you please take a loot at this?
Thank you, cheers
-deleted-
it seems you use an older version of mlv_rec.
can you download the latest nightly or this (http://ml.g3gg0.de/modules/mlv_rec.mo/mlv_rec.mo)one and post the log?
g3gg0, I was using latest build : 31-01-2014
Despite of that i updated mlv_rec with the one from the link you provided.
Seems that there is no change :(
Logs:
https://www.dropbox.com/s/rkkkleltt0035ui/mlv_snd%20v3.txt
https://www.dropbox.com/s/voavblu55fdrq9p/raw_rec%20v3.txt
It could be the DMA flags doing this too if its only certain resolutions.
Anyone can tell me how to access the bytes of the file header? I'm in VB, but the only readable string I get is "MLVI v2.0" which is the title of the header, after that, the remaining bytes of the header are strange symbols. How can I translate them into their actual values? I'm a rookie programmer who's developing the MLV converter for windows,
thanks!!!
Hi g3gg0, sorry for being such a noob: A few questions:
1. On the first page of this discussion you have mentioned "grab HERE the pre-built ones. (including autoexec.bin for 5D3)
use it on your own risk. this file includes mlv_dump binaries for WINDOWS, LINUX and OSX". Is this still valid? I have downloaded the Feb 1 build on my 5D3. If it is valid, could you please indicated whether only the sound moudules have to be copied or even the .sym and .bin files from the modules on the first page?
2. From your notes (changelog), could you please help me with understanding:
9157: ETTR: If CONFIG_RAW_LVIEVIEW is not defined, do not allow ETTR in LV
9156: Make sure no raw LV feature can be used unless CONFIG_RAW_LVIEVIEW is defined.
9153: Make ML compile again
Use commit 18dc1fb
tested today's nightly build. there are pink frames so i switched back to 21 jan 14 nightly build
Quote from: andy kh on February 02, 2014, 04:51:25 AM
tested today's nightly build. there are pink frames so i switched back to 21 jan 14 nightly build
if you use those builds you cant record more than 10 minutes. Use jan 6th as it will record continuously. jan 30th is also ok.
Jan 31 Tragic Lantern for 7D test.
Constants: Buffer-2 / Stills Raw-none, Jpg-L /MLV Raw+SND
1. Capacity test No HDMI /GD-on = Fills up 64 gig CF/continuous ok.
BUT contains pink/corrupt frames/ some totally black frames.
2. When HDMI monitor is connected = Pink/corrupted frames appear at random on each clip.
3. Movie Restart "Bug". when this is ON in MLV mode, upon stopping of raw recording H264 record starts. (suggest a fix like in 5D2 that it wont have this behavior)
4. it must be stressed either as a tool tip in menu that with stills set to RAW, continuous record not possible(only 4gig at most times or 28gig depending on buffer method)
5.FPS at 1fps and 0.5 fps NOW WORKS no more repeating frozen frame.
Installed the nightly Build MLV from Feb.2nd on 5D2,
A great improvement !, got 18,887 frame about 55GB.
At 1856x928 23.976p + Audio, Hacked preview, fps over ride on & I change the picture quality
in the canon menu to small raw instead of small jpeg, buffer 1,GD off, on Lexar 64GB 1000x
Then had a frame skip, but that when things when wrong.
It give kept recording , press the start/stop button it just starts a new file ,had to power off.
I also check with a short clip of about 1000 frame some thing press stop just starts a new file.
There is no way of turning it off unless you power down.
The problem is from the H264 movie auto restart, turn that off and there is no problem.
With the 55GB file a check with MLV Browser Sharp, and had a error, flash red on the clip.
error was "8 frame missing MLV ID" did play ok.
Also at 1872x936 23.976p I got 38GB about 13,000 frames buffer 1, hack preview.
In Crop mode 1920x872 23.976p 38 GB.
@reddeercity,
yeah testing the module on 7D also and the movie restart "bug/quirk" is also there. this is not present with older 5d2 builds. Even with movie restart on, the MLV rec/start stop will not trigger the h264 record start.
g3gg0 help!
I am getting these serious black regions near the shadow areas (neck and lower cheeks of my subject). Where can I upload a sample DNG to share with you.
Camera: 5DM3
Recorded MLV
Converted with MLVViewSharp
http://upload.g3gg0.de/index.php
done!
another request: how do I share the crash/ problem log for MLVViewSharp with you. See it on the screen but cannot copy it or prntscrn it. I am using Windows 8.
ok, but whats the problem with that image?
The neck and lower jaw area has dead black pixels that cannot be recovered. Basically, shadows are really difficult for me to control - also why I stopped using dual_iso for photos.
maybe we are looking at different images. there are no black pixels in that image.
i only see two red and one green hot pixels.
can you post a screenshot of what u mean?
Feb 2 TL Test on 7D with g3gg0 preallocate feature.
#1 Capacity test- vga out/hdmi monitor/GD-off = Fills up 64gig CF, continuous ok, CLEAN no problem with footage.
ISSUE : with CF card full View in File manager hangs, freezes in index 10/10 (maybe due to lack of space to generate IDX file?
#2 Movie Restart - FIXED Tested in MLV mode and in H264. Functions as advertised, less than 1 second gap between clips.
#3 Pre Allocate delay, Record start/stop time.
Feb 2 TL- 4 seconds delay.
Jan 7 TL- 22 centisecond delay (almost instant)
#4 HDMI recording at HD resolution, GD-OFF , tested with 5 clip recording.
clip 1 length 8 sec = Clean
clip 2 length 14 sec = 1st frame and at 4" corrupted.
clip 3 length 19 sec = Clean
clip 4 length 12 sec = Clean
clip 5 length 12 sec = 1st frame corrupted.
NOTE with Forced HDMI-VGA = footage is all clean no problems in footage.
The preallocate delay might be too long for some situations and uses, maybe it should have an option to use this or not.
thanks a lot!
#1 I guess the performance is as always not as good as raw_rec
#2 nice
#3 does it give you a benefit when file splitting happens? my write rate is always quite jumpy. exFAT is no available option for 7D. i think i will add an option to toggle it.
#4 uhmm forced HDMI-VGA helps? hmm is the quality good enough to leave it as option if you want to use HDMI?
HDMI VGA has the same resolution as built-in LCD (maybe even lower), so this makes sense. At full HDMI it processes 6x as many pixels for displaying.
this is another proof that the EDMAC is the bottleneck, even when HDMI output is enabled.
so the best fix would be dropping frames on HDMI display (somehow...) for every n-th frame.
YAY!
- added support for files >4GiB for exFAT..
all my cards are working i can record over 1min or 4gb..
THANK YOU! :-*
Quote from: g3gg0 on February 02, 2014, 06:37:00 PM
maybe we are looking at different images. there are no black pixels in that image.
i only see two red and one green hot pixels.
can you post a screenshot of what u mean?
g3gg0, just sent you the sample (cropped to highlight the affected area). Basically, the black pixelation is around the lips, lower jaw and neck.
Quote from: Jakobmen on February 03, 2014, 01:27:47 PM
YAY!
- added support for files >4GiB for exFAT..
all my cards are working i can record over 1min or 4gb..
THANK YOU! :-*
This made full HD cont. rec stable at 25fps on my Mk III =), Komputerbay 1050x 128gb with sound =D
Cant see cropmarks in external zacuto viewfinder though, Global draw on works though.
tested 03 feb 14 nightly build on canon 550d recording dint start when i press record button.
Quote from: Canon eos m on February 03, 2014, 02:49:30 PM
g3gg0, just sent you the sample (cropped to highlight the affected area). Basically, the black pixelation is around the lips, lower jaw and neck.
there is nothing that comes from raw video.
alex and me double checked the raw+jpeg and the only thing we could imagine is that you just
a) underexposed the image by a few stops (fact) and
b) used some post processing settings that crushed the blacks too much
this gives you the impression that there is banding or edges.
i sent you an JPEG from alex that he shifted by 3.5EV which looks better.
-> just a post processing issue that you would have with CR2 too
Quote from: g3gg0 on February 03, 2014, 12:35:06 PM
this is another proof that the EDMAC is the bottleneck, even when HDMI output is enabled.
so the best fix would be dropping frames on HDMI display (somehow...) for every n-th frame.
Would this affect the 60i hdmi stream being captured to a hard drive recorder on the 5d2?
I use the that as my backup/proxy.
@g3gg0
Quote from: g3gg0 on February 03, 2014, 10:47:18 AM
#3 does it give you a benefit when file splitting happens? my write rate is always quite jumpy. exFAT is no available option for 7D. i think i will add an option to toggle it.
#4 uhmm forced HDMI-VGA helps? hmm is the quality good enough to leave it as option if you want to use HDMI?
#3 On the 7D, as compared to Jan 7 or 30 , preallocate delay had no benefit when in comes to file splitting.
#4 forcedHDMI-VGA was used on the 5D2 before raw came in so this is useful on the field for external monitoring, though for those using external recorder not so.
BUT then ON THE 7D, they can only use this when the camera is not recording anyway since the camera would have corruption which sometimes also flashes via hdmi,
so the only way for those with external recorder to record FROM 7D is to do so ONLY on the external and not in cam.
I have tested using the external monitor to fit focus between different planes of focus while in forced hdmi-vga mode and IT IS USABLE to a degree.
Quote from: g3gg0 on February 03, 2014, 12:35:06 PM
this is another proof that the EDMAC is the bottleneck, even when HDMI output is enabled.
so the best fix would be dropping frames on HDMI display (somehow...) for every n-th frame.
Is it possible to lower resolution to 720 instead of 1080 60i rather than to skip frames? Or is HDMI specs is always at 1080?
Another "hidden feature" I noticed in crop mode is, when using AUTO preview, it give you proper framing but lowres, lowfps and in grayscale, But when I HALF PRESS the shutter, it gives you Higher res and in Color image.
I tested making half press during recording in crop mode to see if it affects recording like corrupting or skipping and it seems it does not. Is there a way we can also implement this as a preview option aside from a half press? Its useful when doing a slow pan and focus needs to be adjusted.
Quote from: ted ramasola on February 03, 2014, 10:34:43 PM
BUT then, they can only use this when the camera is not recording anyway since the camera would have corruption which sometimes also flashes via hdmi,
so the only way for those with external recorder to record is to do so ONLY on the external and not in cam.
I don't see that problem with 5d2, I record the hdmi stream at the same time as raw or mlv.
there's no issue with corruption on the save hdmi stream.
I do see the flashes via hdmi once in a while in standby mode thou, but never when recording.
haha! @reddeercity, I was editing my post when you posted. Yeah, the 5D2 does not have issues so much so I was not testing it with this build. I'm still on a.d.'s Jan 6 and 14 combo.
QuoteAnother "hidden feature" I noticed in crop mode is, when using AUTO preview, it give you proper framing but lowres, lowfps and in grayscale, But when I HALF PRESS the shutter, it gives you Higher res and in Color image.
Found this one out yesterday and it left smiling madly for half an hour :)
Quote
I tested making half press during recording in crop mode to see if it affects recording like corrupting or skipping and it seems it does not. Is there a way we can also implement this as a preview option aside from a half press? Its useful when doing a slow pan and focus needs to be adjusted.
Its dependent on GD, there is a full time color option in advanced. When you record it does lag, etc. Nobody has been really trying it either way.
Hi! Full time color in advanced? Can,t find this option on my 5d mark 3. Is it newer builds? Where exactly do you see this option?
Thanks!
//D
Quote from: 1% on February 04, 2014, 12:11:44 AM
Its dependent on GD, there is a full time color option in advanced. When you record it does lag, etc. Nobody has been really trying it either way.
Display>advanced settings ? or Movie>RawVideo>Advanced ? Can't seem to find this option. I calculated the lag with grayscale and color they seem close. With the added resolution, it would be useful than the current implementation.
EDIT: ok got it.FULL TIME COLOR PREVIEW In RAW MLV menu.
Quote from: ted ramasola on February 03, 2014, 11:10:59 PM
haha! @reddeercity, I was editing my post when you posted. Yeah, the 5D2 does not have issues so much so I was not testing it with this build. I'm still on a.d.'s Jan 6 and 14 combo.
:)
Is there an *easy* way to extract a thumbnail (aka first frame) of an mlv file? I want to implement thumbnails in my software and I'd like to know how can I do it.. I notice mlvbrowsesharp does it, kinda slow though.. any help please?
mlv_dump -f1
will dump two frames, 0 and 1
THANK YOU!
Quote from: 1% on February 04, 2014, 12:11:44 AM
Its dependent on GD, there is a full time color option in advanced. When you record it does lag, etc. Nobody has been really trying it either way.
@1% and g3gg0
This Full time color preview for Crop mode is WAY BETTER. I've tested it and it helps in getting proper focus. A user can develop mental anticipation for predicting movement for framing to compensate for frame refresh lag, BUT its difficult to anticipate and mentally interpolate to focus manually. So having a better image in color albeit slow refresh is better than grayscale and lo res image. The lag difference between grayscale and color is just about the same.
Quote from: g3gg0 on February 03, 2014, 07:52:28 PM
there is nothing that comes from raw video.
alex and me double checked the raw+jpeg and the only thing we could imagine is that you just
a) underexposed the image by a few stops (fact) and
b) used some post processing settings that crushed the blacks too much
this gives you the impression that there is banding or edges.
i sent you an JPEG from alex that he shifted by 3.5EV which looks better.
-> just a post processing issue that you would have with CR2 too
Thanks g3gg0. Heartning to know that it was something I did to cause the blackening in part of the frame. Will keep a watch on exposure henceforth. ETTR should help.
However, I did not use any post processing settings at all. The picture is straight out of the Day converter (I think MLV converter 1.4) into ACR.
Any tips on how to stay away from the blacks when shooting people.
@g3gg0 and 1%
is it also possible to implement the audio meters when FPS over ride is on in MLV raw recording since audio is now being recorded regardless of what FPS is in MLV raw?
Quote from: g3gg0 on February 04, 2014, 02:05:48 AM
mlv_dump -f1
will dump two frames, 0 and 1
Yes, into another mlv that needs to be further processed. Is there not a faster way of doing it? Can't it dump the first .dng from it?
Will something like this work to read the first frame of an image? Translate the byte array into an image basically..
http://go4answers.webhost4life.com/Example/convert-array-bytes-image-68125.aspx
Maybe bitmap is too simple.. we're talking about raw here... will it work?
Checking the nighty build magiclantern-v2.3.NEXT.2014Feb04.5D2212 with HDMI.
I had a crash/lockup --Error 70, I haven't seen that for a few mouths.
Zacuto Evf, 5D2, GD=off, buffer=0, preview=hack view, canon menu small raw=photo,
mlv+audio 1856x928, 23.976p, fps override=on, audio=4800
42GB, 14,785 frames. Skipped frame then stop recording then a few second later error 70.
Preview the clip in MLVBrowseSharp played fine off card, had errors "5 frames missing mlv id"
But ok.
I put the Log crash file in my dropbox for the developers to look at
https://www.dropbox.com/l/rLZWcQKMRtOKZK7LMe2Wjd :)
Quote
ASSERT: 0
at Memory\Memory.c:561, task RscMgr
lv:0 mode:3
Magic Lantern version : v2.3.NEXT.2014Feb04.5D2212
Mercurial changeset : 80c96bc57ed4 (unified) tip
Built on 2014-02-03 23:09:57 UTC by [email protected].
Free Memory : 126K + 4001K
Shoot maloc memory error. Maybe some failure making preallocated files?
been scouring the forums, and just have one question. Which build should I use for the most stable RAW video on a 5dmiii? Should I use the nightly builds or go with Alpha? Or something else I am not aware of? Thanks for everything you guys do! :D
Quote from: tonybeccar on February 04, 2014, 02:24:29 AM
Yes, into another mlv that needs to be further processed. Is there not a faster way of doing it? Can't it dump the first .dng from it?
well, then add --dng ;)
Quote from: 1% on February 04, 2014, 03:36:51 AM
Shoot maloc memory error. Maybe some failure making preallocated files?
any more details?
That was from his log. I haven't seen it happen yet.
Hmm.. maybe to solve the timecode problem this would help:
http://en.wikipedia.org/wiki/Broadcast_Wave_Format
Shot with sound and the new pre-buffering/allocation. The delay before start is huge. At least 3-4 seconds. Didn't see any pink frames from 30 mins of green screen or have any crashes.
Hi there,
I'm using the last build (05feb) with 5D MKIII Lexar x1000 64GB. With raw_rec I can have a file higher than 4.29 GB but I can't do the same with the MLV_Rec (1980x1200, 25 fps, GD allow, extra hack on). It stops automatically after about 1200 frame. Can someone tell me what's wrong?
I just noticed it only happens when I use the HDMI output (to an EVF Zacuto)
To resume: with MLV_Rec (+MLV_Snd) it automatically stops at about 4.29 (a little bit more, just the time to create a new MLV_01) when I use HDMI Output.
Quote from: 1% on February 04, 2014, 06:53:41 PM
Shot with sound and the new pre-buffering/allocation. The delay before start is huge. At least 3-4 seconds. Didn't see any pink frames from 30 mins of green screen or have any crashes.
With my slow card i need to wait 17 seconds...
In the begging, wend i turn the camera one and try for the first time this new version, i see some kind of pink/flash in the LV, not while recording, (like the corrupted/pink frames in the dng's but this time in the LV), after a while just didn't happen again. I did this test yesterday but if i remember well in crop mode is there the random pink/flash in the LV.
In the firsts MLV i got a pink/corrupted dng in the 13 frame/dng.
I test with GD on, buffer 4 (default)
Tomorrow i can make more test to see if the behavior still there.
I got it: in Raw MLV_rec, the option Files>4GiB (exFat) needs to be turned ON if one wants to use the HDMI output with higher than 4.29 GB recording otherwise it doesn't work.
600D was doing these flashes too when CPU load was really high. 17 seconds is insane. I've got no pink frames in any of the clips tho. Didn't record while HDMI was plugged in and definitely not with the preview up.
Quotethe option Files>4GiB (exFat)
From the code it looked like exfat was default unless you have a 5DII/50D/7D
... ok now I see a setting for "large file support", have to add it.
One more thing:
It could be great if the GB (CF card) information on the top right of the LV would be update each time the record is off or even better in real time. It only updates when one switch off then turn on the camera.
Thanks for all.
Quote from: 1% on February 06, 2014, 01:27:04 AM
17 seconds is insane
Indeed, i have measure with windows watch.
Quote from: Sthirasukha on February 06, 2014, 01:32:27 AM
It could be great if the GB (CF card) information on the top right of the LV would be update each time the record is off or even better in real time. It only updates when one switch off then turn on the camera.
Would be nice :)
5D mkII note on the mlv snd wav file for Feb 4 build by a.d.
If you stop recording manually your WAV file would "normally" be longer than the video file by 7-10 frames -which is fine.
However, if you keep on recording till Card is full, and the recording abruptly stops on its own, the resulting WAV file would be significantly shorter than the video file from 10 seconds to more than 1 minute.
Can we place safeguards on this behavior? perhaps based on CF capacity, the recording should stop safely?
QuoteIt could be great if the GB (CF card) information on the top right of the LV would be update each time the record is off or even better in real time. It only updates when one switch off then turn on the camera.
This is the problem with canon and why the pre-creation of files, etc is happening. Also why parts of the wav would be lost, etc.
But in the case of wav files, can't we let ML stop recording when there would be just enough space left to finish writing the sound from buffer to cf? My understanding is since the resulting files are almost 64gig there was not enough space left for the last "chunk" of sound data?
Does a set of instructions for checking of capacity size for ML so disruptive that to do so would affect the writing of video and sound to the buffer and card?
The size of the audio NOT recorded should be about 13,173KB or 12.8 MB for the 1'10" length there was no audio, if this can be "preallocated" before a shutoff then perhaps more or all of the audio could be recorded to CF. This is based on my recording to capacity a 64gig CF.
QuoteDoes a set of instructions for checking of capacity size for ML so disruptive that to do so would affect the writing of video and sound to the buffer and card?
Well it made 17 sec of delay... in the current form it only checks for the video buffers and not audio as far as I can tell.
while you are discussing ... that wave issue should already have been fixed. (since 3 days)
does it not work?
tried 2day's nightly build. it took 43 secs to start recording after pressing recording button. what's wrong?
wait... what? it takes one second on the 5D3.
it is only allocating a file with the size of the available RAM.
are you using SD card? which camera?
I also saw this behavior with the latest build I compiled yesterday, I don't measured it but it took several seconds (perhaps 15s-20s) between the instant I push the rec button and the instant it start effectively recording.
Actually I only have a "slow" 16Go CF card because I had to send back my flaky 1000x 64Go CF and still wait for the replacement...
[Edit] I don't want to add fuel to the fire in any way but I only tested this on the latest TL code because I'm using a 7D
[Edit] I'm also ready to join the force to do back porting from TL to ML
it takes 43-45 secs on my canon 550D
it takes 43-45 secs on canon 550D
It probably takes 1 second on 6D/5DIII and a long time on the rest. i never ran out my card to see if the wav file disappears, stayed away from that end.
Yeah the newest nightly for 600D is taking almost a minute to start recording .mlv as well. Seems like the last 3 nightly builds have this problem.
Is there a block in the header about dual iso? I opened with hex editor and I cant find one.. or at least is there a byte that says if the video file is with dual iso or not?
the delay came from the space reservation that i do to have enough room on the card even if it runs full.
it basically the same what alex did in raw_rec afaik.
it reserves the amount of space that you have as RAM in a file. in case the card runs full, the file is deleted.
fixed that yesterday. seems to be a FIO issue when seeking.
should be more usable now.
you can disable that card space reserveation just in case.
but the file is only recreated if you a) have suddenly more ram available or b) it was deleted due to a full card.
Quote from: tonybeccar on February 07, 2014, 09:11:18 PM
Is there a block in the header about dual iso? I opened with hex editor and I cant find one.. or at least is there a byte that says if the video file is with dual iso or not?
nope. wanted to switch to ml-cbr for modules signalling each other.
alhtough i didnt touch it yet :)
Quote from: g3gg0 on February 07, 2014, 09:17:04 PM
nope. wanted to switch to ml-cbr for modules signalling each other.
alhtough i didnt touch it yet :)
Hehe, a bit chinese for me :P Hope someday it could be incorporated to the header. So for example my batch converter would automatically detect them and transcode them :)
Quote from: 1% on February 04, 2014, 06:53:41 PM
http://en.wikipedia.org/wiki/Broadcast_Wave_Format
do the usual suspects (premiere etc) support BWF tags?
Premiere is supposed to since CS5... I see posts for some people having trouble syncing the TC but all of it is from 2010.
Quote
Support for all major media types
Import and export major video, audio, and graphic file formats, including FLV, F4V, MPEG-2, QuickTime, Windows Media, AVI, BWF, AIFF, JPEG, PNG, PSD, TIFF, and more.
This is from CS5.5
Newest nightly, 600D, mlv_rec with mlv_snd is such a beautiful thing. Still limited to 9 secs of movie at 1280x536 but it now does 9 secs WITH SOUND. 22khz for the record. Still have to test the other sample rates but 22khz is completely fine. No more "dead" pixels, hot/cold pixels, whatever they're called.
Two issues I'm seeing, though. Acr 6.5 in AE CS 5.5 isn't recognizing the metadata of the dng's exported with the OSX Batch Convert App (recursive0.4.command, OSX Mavericks), it displays the camera model at the top of ACR but nothing under camera profiles and no wb info. And maybe I'm missing something because I haven't been following MLV development much lately, but the camera crashes (Err70) every time I try to record in 5x crop, in both 1080p and 640x480 mode. BUT, if I record in the 600d's 3x zoom mode it runs just fine.
With Global Draw Off (mlv_rec), after recording Global Draw remains disabled. You must exit movie mode, or enable GD in mlv_rec menu and make another recording to enable GD.
With Extra Hacks On, FPS override overlay remains onscreen and shifted behind the memory buffer overlay.
tried today's nighly and there is no more delay. there are pink/corrupted frames.
Quote from: andy kh on February 08, 2014, 05:19:42 AM
tried today's nighly and there is no more delay. there are pink/corrupted frames.
what camera are you using?
Testing for Feb 6 build.
The 5D2. No start stop delay.
Cleaner display with the buffer display removed.
Movie restart function for h264 is fixed.
Only the low FPS freeze frame is not yet fixed.
On the 7D
I had pink/corrupted frames when Status - Icon is ON as well as Full Color Preview.
With those at off all clips in tests clean and audio in sync.
i use 550D
Can you give some more details?
This is fast =)) Thank you =)) Now wend press rec it's start almost immediately. Will continuous to make some more tests.
Thank you g3gg0 and vyskocil!
I try this from TL but i would like to try the "new" stuff ported from TL 2 ML, here i can find it? Need to wait until ML nightly builds for 7D are up again?
Some tests:
-Wend frame skipped, the message stay in Lv, only get out if press trash button or if wait one minute
-Wend card full and i go format it display only 160kb used
- 1 frame skipped and camera crash
-Turn off and on camera test again and it's showing the right capacity wend go to format in camera
-Full fill the card (8GB) and format again in camera and only show 160kb used. I format and everything go's as should be. (not big deal, just for knowing)
Trying things out too, and i get this file: mlv_rec.tmp in the same folder with the mlv file. Safe to delete this one?
Anyone can help with this?
I updated to the latest nightly since a month ago and I noticed card spanning doesnt work anymore.
Tried to turn off Memory hack and or Extra Hacks and still the SD remains idle at all the resolutions both 300 procent crop and 100 procent.
Tried to format both cards to Exfat and Fat32. And in the canon camerasettings tried all variations with card storage.
Is card spanning left out off on purpose?
Card spanning is working just fine, i can confirm on that, as i always shoot with it ON! Trying out the latest nightly all day and it works all right...
QuoteCard spanning is working just fine, i can confirm on that, as i always shoot with it ON! Trying out the latest nightly all day and it works all right...
That makes it even weirder. Can you post your settings if its not too much of a hassle? Card spanning used to work here too but now its constant idle, even with the the highest resolution.
My build is of 7-2-2014. Record func + card/folder sel. is on standard in the canon menu( useless info?) The two cards are now in fat32 and the SD has EOS_develop and Bootdisk flashed on it.
the last several nightly builds i've tested all freeze up after I press record, with mlv_rec and the original raw. I haven't made any changes to settings other than frame skipping.
the little camera icon pops up in the top right corner, but just stays 0:00. with a MK3
Turn off the card full save file and it starts instantly.
Now im pretty sure card spanning is resisting:
When I turn off CF-only buffers and Buffer Method on 0 it wont record at all.
Also with a 5d3
I´m trying to make the cold-pixel-fix from raw2dng working with mlv_dump: https://bitbucket.org/escho/escho-magiclantern/branch/cold-pix-fix
Doing this, I went into something, I never realized before in this way:
MLV-file 2.2MB recorded with my 600D (camera on my teleskope for recording jupiter, 5801 frames, 640 x 360 px, mlv_dump from today)
Decoding times:
./mlv_dump -r -o test.RAW *.MLV: 11s
./raw2dng test.RAW: 44s
./mlv_dump --dng *.MLV: 5min 59s
Why this very big difference of decoding times between the detour over legacy raw and direct converting to dng with mlv_dump?
Edgar
Quote from: 1% on February 08, 2014, 06:22:35 PM
Turn off the card full save file and it starts instantly.
where is that setting 1%?
Quotewhere is that setting 1%?
Beneath Card Spanning:
Reserve card space ON/OFF
Sorry to bother here again, still trying to understand the metadata. What is the Audio Frame Count for? An amount of frames for the audio and video to be in sync? If so, how does it work exactly? Thanks!
Quote from: escho on February 08, 2014, 08:21:32 PM
Why this very big difference of decoding times between the detour over legacy raw and direct converting to dng with mlv_dump?
Edgar
Oh, I understand. Disabling croma-smooth by using the --no-cs option makes the stuff fast. Didn´t know, that chrama-smooth is enabled by default. Is this wanted?
Edgar
I've noticed that ffmpeg has a GSOC suggestion which includes better DNG handling: http://wiki.multimedia.cx/index.php?title=FFmpeg_Summer_of_Code_2014#Bayer_RGB_colorspaces
How about suggesting the ffmpeg folks a MLV import GSOC implementation, if you guys thinks that MLV is mature enough by now?
Quote from: tin2tin on February 08, 2014, 09:11:59 PM
I've noticed that ffmpeg has a GSOC suggestion which includes better DNG handling: http://wiki.multimedia.cx/index.php?title=FFmpeg_Summer_of_Code_2014#Bayer_RGB_colorspaces
How about suggesting the ffmpeg folks a MLV import GSOC implementation, if you guys thinks that MLV is mature enough by now?
Excuse me.. but ffmpeg doesn't support DNGs NOW? Am I right?
I am seeing there are multiple questions varying from themes being asked on this thread.
Maybe opening a new thread would be better.
Last try here:
(http://www.alisultan.nl/magicbug/idlesdcard.JPG)
See how idle the sd card is? Volcano Kohala saw this picture and complained how idle the sd is.
Quote from: escho on February 08, 2014, 09:08:38 PM
Oh, I understand. Disabling croma-smooth by using the --no-cs option makes the stuff fast. Didn´t know, that chrama-smooth is enabled by default. Is this wanted?
Edgar
Is there a quality difference between chrome-smoothed and non-chroma smoothed? What's the difference?
Oh one other thing, I think a bug maybe... there seems to be a mismatch in the camera serial number. For example mine is: 2661200228 and inside the MLV it shows: 9E9EB164. Mlv_dump shows the same. Is it a bug or a different way of coding the number?
Quote from: tonybeccar on February 08, 2014, 09:16:06 PM
Is there a quality difference between chrome-smoothed and non-chroma smoothed? What's the difference?
Yes, should be. Enabling chroma-smooth should make the pics a little bit smoother (reducing colored noise and maybe some hot pixels). But in this case, it does nothing. I can use any chroma-smooth-option, I want, the output-dngs are always the same. No difference between them.
Edgar
Quote from: escho on February 08, 2014, 09:40:57 PM
Yes, should be. Enabling chroma-smooth should make the pics a little bit smoother (reducing colored noise and maybe some hot pixels). But in this case, it does nothing. I can use any chroma-smooth-option, I want, the output-dngs are always the same. No difference between them.
Edgar
That´s a bug in mlv_dump.c
I found it, but cannot create a pull request to fix it at the moment. Maybe later...
Edgar
Ok, so i left out my CF card trying to do raw on just the SD. Card spanning on. I lowered the resolution so it would be like 19mb/s. I rebooted to make sure card warming increases a lil speed.
After warm up,I hit REC and the module said: failed to create file. Card full?
After rebooting the camera, put in high speed shooting and the card took the shots without hesitating. I also recorded a normal movie.
Card spanning off and it recorded perfectly. So now i put the cf card in again, card spanning on, and upped the resolution to 2k.
SD card idle again.
Quote from: tin2tin on February 08, 2014, 09:11:59 PM
I've noticed that ffmpeg has a GSOC suggestion which includes better DNG handling: http://wiki.multimedia.cx/index.php?title=FFmpeg_Summer_of_Code_2014#Bayer_RGB_colorspaces
How about suggesting the ffmpeg folks a MLV import GSOC implementation, if you guys thinks that MLV is mature enough by now?
thanks, getting in contact with them.
Have I missed this somewhere in the forum, because I just scoured the entire forum for this and got nothing. Did I just make a breakthrough discovery that PPro CC works native with MLV raw files direct off the card (with a few minor glitches). I was just mucking around with all these converters and programs and thinking to myself, how great would it be if Adobe just supported MLV native right off the SD/CF card. Well, it seems that PPro CC does just that... almost 100%.
I was dragging in some CDNG clips into PPro and then I went to my Media Browser and saw my MLV RAW folder there and decided to click on it just for fun to see what happens. To my amazement PPro said "processing clips" the MLV raw clips!!! Then I then dragged those MLV raw clips into the source monitor, set in/out marks, dragged them into my timeline and edited, added effects, and exported! All without any problems or crashes and glitch free scrubbing in PPro! The only glitch is there is a weird triangle that progresses down the frame as the clip is played and I saw a few dead/cold pixels in the output file, but that might be something in my settings that caused the cold pixels, not sure.
Has this been posted already on the forum? I couldn't find it. Could this be a major breakthrough for ML developers? I sure hope so!!! Fingers crossed! :) Please chime in with any ideas or fixes for the triangle or cold pixels, because other than that PPro works native with .MLV raw files right off the card! I can't believe this hasn't been discovered already.
Here is a screen capture of PPro CC showing the MLV clips...
(http://www.dueymoore.com/mlvraw.jpg)
And the mystery arrow that progresses down the screen with the .MLV extension next to it.
(http://www.dueymoore.com/arrow.JPG)
Here is the MLV ouput to H.264/Viemo setting...
it doesnt work for me. mlv clip is not shown in the media browser
We talk about adobe Premiere pro that supports mlv out of the box?
wow...
wish that were true :)
@g3gg0 Please look at the thread I started on this topic, it definitely works, look at the videos I posted.
http://www.magiclantern.fm/forum/index.php?topic=10401.new#new
Maybe delete this post here since I already started another thread? I wasn't sure where to put it.
now i found out that its the drastic MediaReader worstation that enable premiere pro import mlv or raw directly. and the triangle is the watermark since you are using trial version. to get rid of the triangle u need to buy drastic media reader.
Quote from: escho on February 08, 2014, 10:03:09 PM
That´s a bug in mlv_dump.c
The bug wasn´t a bug. My testings were wrong. Sorry for that...
Edgar
Where to get the latest nightly build of the mlv_rec.mo?
In the OP post there is only one link that did not change over time (as far as I understand).
I have a 128GB 1050x Komputerbay card. Is it worth switching to the current RAW version, or shall I stay with the old version because only the 64GB 1000x is reliable enough?
I have a Canon 5d mkIII and I need to shoot in minimum 1080p25 (located in Europe).
Thank you.
how do i get rid of pink/corrupted frames??
i use canon 550D
While LV is paused and the screen is black, you can start H264 on accident if you have a quick trigger finger... this seems to fix it for me:
https://bitbucket.org/OtherOnePercent/tragic-lantern-6d/commits/f16e73eafa998ac4afae4990229a73909155803a
Quote from: tronics on February 09, 2014, 04:50:42 PM
Where to get the latest nightly build of the mlv_rec.mo?
In the OP post there is only one link that did not change over time (as far as I understand).
I have a 128GB 1050x Komputerbay card. Is it worth switching to the current RAW version, or shall I stay with the old version because only the 64GB 1000x is reliable enough?
I have a Canon 5d mkIII and I need to shoot in minimum 1080p25 (located in Europe).
Thank you.
You need a 1050x card that benchmarks above 110mb/sek with crystal disk mark to get stable fullhd mlv +sound stable. (Trancend usb3 udma 7 or equally fast card reader required.)
Quote from: Markus on February 09, 2014, 08:27:01 PM
You need a 1050x card that benchmarks above 110mb/sek with crystal disk mark to get stable fullhd mlv +sound stable. (Trancend usb3 udma 7 or equally fast card reader required.)
Thank you.
As I understand after multiple passes the best value is shown.
I always looked at the Write column in the first row..
114,5MB/s write with setting 9 times 500MB
111MB/s write with setting 5 times 1000MB
109.9MB/s write with setting 9 times 1000MB
109.5MB/s write with setting 9 times 2000MB
108MB/s write with 9 Times 4000MB
Sometimes there were passes that were below 110MB/s and then the last value allowed it to move above 110MB/s
So this means it does not work for me I suppose..
How much can I save without audio?
To Markus
To get the latest nightly build of the mlv_rec.mo:
http://builds.magiclantern.fm/#/
To g3gg0
I tested your last build, thank you for the new icon options in the LV (icon is great, just like the Raw_Rec option)!
And as I said in another thread:
I tested this :
1— MLVBrowseSharp / DNG+Wav
2— MLVBrowseSharp / RAW+Wav / + RAWMagic / DNG (=CDNG)
There is a small difference between both (at 200%!):
1— DNG are a bit more grainy but more regular in the low-mid-high value. (It works with ACR. Unusable with Resolve 9, maybe it is on 10, I have to try and I also have to test on Premiere CC with DrasticPreview as it seems to work)
2— DNG (=CDNG) are less clean in the low area (more weird colors in the black). The grain is less visible in the mid-high area. (Usable on both ACR and Resolve 9)
It would be more simple if the RAWMagic App could include the MLV files and not only Raw (for a CDNG option) because, even if MLVBroweSharp is a great App, I noticed the RAWMagic is really faster.
Do you think it's possible ?
Quote from: tronics on February 09, 2014, 10:16:19 PM
Thank you.
As I understand after multiple passes the best value is shown.
I always looked at the Write column in the first row..
114,5MB/s write with setting 9 times 500MB
111MB/s write with setting 5 times 1000MB
109.9MB/s write with setting 9 times 1000MB
109.5MB/s write with setting 9 times 2000MB
108MB/s write with 9 Times 4000MB
Sometimes there were passes that were below 110MB/s and then the last value allowed it to move above 110MB/s
So this means it does not work for me I suppose..
How much can I save without audio?
If you hit over 110mb/sek
sequential write there is a good chance it will be stable. My two 64gig cards hits 115mb and are stable so I would guess minimum stable results would be gotten if It benchmarkes 111 or 112mb/sek since my 110mb/sek benchmarking card seemes to be just at the edge of whats required. Sequential Write seems to be the only value of interest since ML writes everything in one large sequential raw file.
Can someone tell me where to find comparison between MLV and the regular RAW formats in magic lantern? Thank you!
Quote from: mrd777 on February 10, 2014, 09:25:30 AM
Can someone tell me where to find comparison between MLV and the regular RAW formats in magic lantern? Thank you!
Look at the very first post of this thread.
MLV is a container for both DNGs, metadata and sound, whereas the older RAW module only handles DNGs.
I found using the latest a.d. build 02/04 to be the most unstable one so far. I was recording on a 5D2 using mlv with audio at 1728 x 934 23.976 without FPS override, global draw on with magic zoom and digic peaking enabled.
I had several "Audio Failed to Stop" errors where I had to pull the battery.
I had several times when I pressed record I go a frame counter of -1 and the recording just froze. I had to turn off the camera where I got a message saying the camera didn't shut down cleanly and module loading would be skipped.
I had one card that about half way full just would not record any more, it acted like the card was full and I thought it was until I checked it on a computer. That card had some sort of .tmp file with the mlv's that caused issues why I tried to copy the files off the card, the mlv's copied just fine but the computer would not copy the .tmp file saying it was damaged and couldn't be copied. I was able to delete it though.
Out of 68 recordings of about 20 seconds long, I had 10 mlv's that were missing 1 frame, interestingly they were all right around frame 350 ranging from 349-354. I was shooting a a slower card with a max number of frames of about 450. I suspect these may have been recordings where the buffer filled up and the recording stopped automatically. All of the recordings had at least 420 frames so it is not the last frame that is getting dropped.
These are all very random so hopefully some of this info will help.
mlv_dump
Please look at this console output:
edgar@linux-uoww:~/test_with_badpix> ./mlv_dump --dng --fixcp *.MLV
MLV Dumper v1.0
-----------------
Mode of operation:
- Input MLV file: 'M03-0503.MLV'
- Convert to DNG frames
- Output into 'M03-0503_frame_'
File M03-0503.MLV opened
File M03-0503.M00 not existing.
Processing...
Vertical stripes correction:
1.000 1.000 1.000 1.003 0.998 0.998 0.996 1.005
Cold pixels : 8
I´m sitting in front of my computer, waiting for mlv_dump to have done its work. But I´m only told: "Processing...". I would like to see the progress of the processing. Verbose-output is too much info, Just a little "processing frame xxx / yyyy " would be enough.
Can this be integrated in mlv_dump, please?
Edgar
I'm loving MLV so far. Raw video WITH sound is a godsend, even with the 600d's limited resolution. I'm running mlv_rec in 640x480 mode, fps override set to 23.976, 2.39:1 rez, GD on (usually only use spot-meter though), and extra hacks on. Even with fps override on, MLV sound still captures audio at 48khz.
I left pic quality set to sRaw from an old TL build (which is RISKY, as mentioned before), and can manage to squeeze out a decent 13-14 seconds at 1280x536. Even with GD allowed I rarely have corrupted frames now, seems like only 1 or 2 on the first recording, then the rest are fine. MZ definitely causes plenty of corrupted frames though, which is to be expected.
Only bugs I'm noticing now are the previews in-camera always stop 2-4 frames away from the end on both color and B&W previews, and it seems mlv_dump for osx needs to be updated, still not getting ACR 6.7 to detect the camera model correctly.
Quote from: PressureFM on February 10, 2014, 09:45:29 AM
Look at the very first post of this thread.
MLV is a container for both DNGs, metadata and sound, whereas the older RAW module only handles DNGs.
Sorry, I meant, is there a quality difference in these formats?
no quality difference
Quote from: g3gg0 on February 10, 2014, 10:20:04 PM
no quality difference
Thank you for your reply. Much appreciated.
hey guys, i'm having some trouble with at least two mlv files, once they output the .raw or the .dng and the .wav file, the dngs are ok, they last about 1405 frames wich is about 58.6 seconds in 23.976, but the wav file that mlv dump outputs only lasts 37 seconds, any idea why this is happening? do you think i'm doing something wrong? or mlv dump has some kind of bug?
the settings i remembered using were 1920x1080 23.976 no override of any kind (shutter or fps), audio was enabled with a 48kHz sampling rate, i used the first nightly build that had audio integrated, that means the one from the 7th of february, i'm using a 5d mark iii.
any help would be greatly appreciated. (it was a paid job, tascam broke down in set and had to use audio recording)
thx in advance
cheers
@a1ex or g3gg0
Feb 11 nightly for 5d mkII, when turning off the camera, sensor cleaning will not stop, have to turn on and off again to make it stop. I think this is a behavior when update has just been made.
After making changes in the menu, shut down is now normal.
edit: problem patched by a1ex in feb 12 nightly.
Quote from: arturochu on February 12, 2014, 03:37:46 AM
hey guys, i'm having some trouble with at least two mlv files, once they output the .raw or the .dng and the .wav file, the dngs are ok, they last about 1405 frames wich is about 58.6 seconds in 23.976, but the wav file that mlv dump outputs only lasts 37 seconds, any idea why this is happening? do you think i'm doing something wrong? or mlv dump has some kind of bug?
Did you fill up your card? If so, you may have lost the audio that was in the buffer that hadn't been written to the card yet (at least, that is my understanding of what can happen in some cases).
I'm using 650D on the latest 12 feb. nightly build.
With RAW 1472x670 I can film for 32 sec. (no sound)
With MLV 1472x670 I can film for 20-21 sec. (no sound)
With MLV 1472x670 with sound I can film for 18-19 sec.
Are there any settings that I can change in order to lengthen my recording times?
And also, there's a huge difference between RAW and MLV (no sound) recording times, it's like 10 seconds. Why is that so?
I could just record only in RAW if I don't need sound or are there any quality or any else differences too?
I am new to Magic Lantern so I apologize in advance, if those questions have already been answered.
Thanks.
Quote from: Frank7D on February 12, 2014, 05:10:26 PM
Did you fill up your card? If so, you may have lost the audio that was in the buffer that hadn't been written to the card yet (at least, that is my understanding of what can happen in some cases).
sounds logic, but actually those were the first two takes on a 128gb card, and even at the end i didn't fill it up. weird problem i have since its no logic to me that the video in one take lasts more than the audio in the same take, how could the audio stop and not the video too? i'm crossing my finger its a bug in mlv_dump or i'm doing something wrong and the audio is there i just can't get it out.
thx anyway, appreciated.
I've asked for a possible implementation of MLV into http://www.libraw.org/ here http://www.libraw.org/comment/3091#comment-3091 libraw is a lib based on dcraw.
The coder replied:
QuoteThanks for info.
I've take a quick look into the specs. It looks like, there is no way to locate Nth frame without reading all previous frames, right?
In LibRaw paradigm it will result into very inefficient applications: for movie apps one need API like get_next_frame(), while photography-oriented API (with one-two frame(s) per file) is entirely different.
Without stream-oriented API, support for MLV in LibRaw will be useless.
-- Alex Tutubalin
Can any of you more tech savvy guys help me out to answer his question?
i repied with:
Yeah, the MLV format is designed to have frames appear out-of-order to be able to optimize memory block and write sizes.
Nevertheless it is simple to overcome this - there is a .idx file which contains the file numbers and offsets to VIDF blocks.
These file can be generated using mlv_dump or with the in-camera player.
If you can send me the direct link to the API the mlv routines have to fulfill, i can prepare what is necessary to index the files on startup etc.
see https://bitbucket.org/hudson/magic-lantern/src/tip/modules/mlv_rec/mlv.h?at=unified
typedef struct {
uint16_t fileNumber; /* the logical file number as specified in header */
uint16_t empty; /* for future use. set to zero. */
uint64_t frameOffset; /* the file offset at which the frame is stored (VIDF/AUDF) */
} PACKED mlv_xref_t;
typedef struct {
uint8_t blockType[4]; /* can be added in post processing when out of order data is present */
uint32_t blockSize; /* this can also be placed in a separate file with only file header plus this block */
uint64_t timestamp;
uint32_t frameType; /* bitmask: 1=video, 2=audio */
uint32_t entryCount; /* number of xrefs that follow here */
//mlv_xref_t xrefEntries; /* this structure refers to the n'th video/audio frame offset in the files */
} PACKED mlv_xref_hdr_t;
Quote from: ted ramasola on January 13, 2014, 09:15:31 AM
Can devs tweak this to put safety nets?
Like, If MLV sound is not ON then Vu meters should not be active?
Quote from: ted ramasola on February 04, 2014, 02:23:31 AM
is it also possible to implement the audio meters when FPS over ride is on in MLV raw recording since audio is now being recorded regardless of what FPS is in MLV raw?
These two should be in next nightly. If you have coding skills, I'd like you to review and try these changes:
https://bitbucket.org/hudson/magic-lantern/commits/4069c8124e1e
https://bitbucket.org/hudson/magic-lantern/commits/bbd9161c116e
I did not find an answer in the forum thus I´m asking here: would it be possible to mark all skipped frames (maybe in an extra file) in a record thus it will be possible to interpolate the missing frames in post process (e.g. using After Effects together with Framerestorer: http://aescripts.com/pt_framerestorer/).
Feb 14 5D Mark 3 build.
When I connect my external Lilliput HDMI monitor, the MLV records fine but the Zebra and MLV white frame is shifted to the left quite a bit. When playing back MLV, the No Image overlay from Canon always shows and the MLV in the background is split into pink quadrants with weird lines underneath the No Image overlay.
I thought we'd nipped that shifting problem in the bud? Thought I haven't tried my HDMI monitor in a few months so maybe not.
@dubzeebass:
do they play fine on PC?
@mario1000:
yes, thats possible. thinking about side effects atm
Quote from: g3gg0 on February 14, 2014, 06:33:17 PM
@dubzeebass:
do they play fine on PC?
Well - Mac - but yes, they do. Both with MLRawViewer 0.4 and with the script we wrote to extract DNGs.
I found that if I put the force VGA option on, the framing is right... so it's got to be something not redrawing properly when the feed is upsampled.
Hey g3gg0,
I wonder if there's some code here you guys could use which would fix that?
http://www.magiclantern.fm/forum/index.php?topic=8518.0
i did a lot of testing.... MLV is not sustaining high bit rate anymore... don't know why. I have all set properly but i cant get 30fps continuous.... :( ---- i have a komputerbay 1000x 64gb. Using RAW_REC it stays around 99MB all time nothing more... using MLV the SD gets 18.2mb all time but CF stays around 50MB 60MB nothing more.. any idea?
thanks guys!!!
BUILD magiclantern-v2.3.NEXT.2014Feb13.5D3113
also in 5x mode it gains a little bit more speed using RAW_REC goes from 98MB to 101MB...
we are unable to download latest nighly for some cameras. there is a messsgae "this buil failed"
Quote from: noix222 on February 15, 2014, 12:37:13 AM
i did a lot of testing....
Please report it properly, without vague statements. A table with build dates and average speeds would be helpful.
Test the Feb 14th MLV+audio build, from A.D. hack crop view mode & the non hack crop view mode.
We have Audio meter now with frame over ride on,
There's at lest a 3 second delay now when you push the set button to record raw, was not there
on the feb4th build. Is this a problem now ? Or is there a change in the way raw record works ?
Lost all informational overlays/text box on Lv when recording Raw.
Just a little icon with only time as information , I don't like that, there is no way of judging
how much video you recorded at what write speed, File size GB? so you don't know when raw will stop recording or how much you just recorded.
I Would like to see a option to switch back to the text information box with buffer bars instead of the little camera icon.
I switch on "show buffer graph" , this dose not work it just keep that little move icon, no buffer graph.
I have notice when you record a short clip then stop, the buffer bar shows for a spilt second .
The "Global Draw Off" only work in 2 preview mode, HDMI & ML Grayscale, in all other preview
there is a Black-box overlay.
I also notice if you use the hdmi Preview without a hdmi device attach you get wrong framing.
So if you do not want the black box for framing ( as I Do) the White Framing box is only correct for hdmi device.
If you press the delete button while recording raw you can remove the black box for framing to a dark Grey translucent overlay. ;)
Checking files, and notice "mlv_rec.tmp" on CF card, is this the reserve Space ?
For some reason after ever video clip capture ML save a screen shot of the Lv with overlays "bmp"
With the Crop view mode half shutter HI- Res preview in ML Gray preview, work like it should until you start
recording raw then the hi-res preview becomes all pinky overcast image, feb4th build was not like that it work with or without
raw record running.
Quote from: reddeercity on February 15, 2014, 08:57:27 AM
Test the Feb 14th MLV+audio build, from A.D. hack crop view mode & the non hack crop view mode.
We have Audio meter now with frame over ride on,
There's at lest a 3 second delay now when you push the set button to record raw, was not there
on the feb4th build. Is this a problem now ? Or is there a change in the way raw record works ?
thats the time it takes to generate files on the cards.
this is done before recording starts, so 4GiB splitting wont take too much time anymore.
Quote from: reddeercity on February 15, 2014, 08:57:27 AM
Lost all informational overlays/text box on Lv when recording Raw.
Just a little icon with only time as information , I don't like that, there is no way of judging
how much video you recorded at what write speed, File size GB? so you don't know when raw will stop recording or how much you just recorded.
I Would like to see a option to switch back to the text information box with buffer bars instead of the little camera icon.
I switch on "show buffer graph" , this dose not work it just keep that little move icon, no buffer graph.
I have notice when you record a short clip then stop, the buffer bar shows for a spilt second .
option "Status when recording" -> "Debug"
Quote from: reddeercity on February 15, 2014, 08:57:27 AM
The "Global Draw Off" only work in 2 preview mode, HDMI & ML Grayscale, in all other preview
there is a Black-box overlay.
I also notice if you use the hdmi Preview without a hdmi device attach you get wrong framing.
So if you do not want the black box for framing ( as I Do) the White Framing box is only correct for hdmi device.
If you press the delete button while recording raw you can remove the black box for framing to a dark Grey translucent overlay. ;)
uhm interesting
Quote from: reddeercity on February 15, 2014, 08:57:27 AM
Checking files, and notice "mlv_rec.tmp" on CF card, is this the reserve Space ?
this file is reserve space in the case when card runs full during recording
Quote from: reddeercity on February 15, 2014, 08:57:27 AM
For some reason after ever video clip capture ML save a screen shot of the Lv with overlays "bmp"
happens when you enabled buffer graphs for debugging
Quote from: reddeercity on February 15, 2014, 08:57:27 AM
With the Crop view mode half shutter HI- Res preview in ML Gray preview, work like it should until you start
recording raw then the hi-res preview becomes all pinky overcast image, feb4th build was not like that it work with or without
raw record running.
okay will check
there was a display problem with card spanning that i fixed a day ago.
if there were problems related to spanning, check again please.
Will check it immediately for spanning:
EDIT:
Still not correct display: 1920 by 1280 24p shoots 1000 frames but display says:
Total rate 48 MB /s
m15blabla.mlv: 82 MB/s
m15blabla.m00: idle
CF only buffer OFF/ Buffer fill method 4/ card spanning ON/
Maybe a glitch of my camera, waiting for other reporters.
EDIT 2:
Found a glitch regarding SD/CF in mlv:
after I record a mlv the changed file size does not appear in the CANON format menu, but it does with RAW and h264 and photoshooting. I can acces the files though with the file manager module or mlv_play.
Example: i shot a MLV with no card spanning of 1GB to the CF card.
I go to the menu of canon(SET UP1->format card) and in the CF card only 160KB is used.(which is incorrect because the files exists on the card)
EDIT 3
Ok, the correct file size appears when shutting the camera off and on again. The glitch is only on mlv.
testing today MLV magiclantern-v2.3.NEXT.2014Feb15.5D3113 shooting 30fps mlv_rec .. card spanning ON both hacks ON all rest left as it comes... i barely got 20second of footage.. i don't know why it can't hold 100MB with the CF + 20M of SD... it is giving me just 60mb - 50mb in CF and 18mb in SD. IM using a komputerbay 1000x card it holds up to 100MB with raw_rec... if mlv works we will get easily 30fps fullhd and 2k.. Also when i import DNG's converted in mlv_dump they come all green (easily fixed by adjusting white balance)... thanks guys...
Did you try to reformat your CF card on PC/MAC ? Just a though.
i did... tried even formatting ExFat, both types of formatting gave me same results... the strange thing is using raw_rec i get 100MB from the CF... but using mlv_rec it goes just around 60mb.... giving CF+SD 80mb. filming in 30fps... are you getting any better of MLV with todays build?? at some point i got continuous 30fps using mlv with card spanning... the thing is i cant find that build anymore i mixed up the folders where i save magic lantern builds.
but no pink frames at all !!! :D
Have you tried Buffer Method 4 and CF only on 0 with spanning ON? That gives me between 90 and 100 (Graph shows 45mb average but thats not correct)
i just did it now.. tried also using TEST MODE and the best number of 52mb/s came out of buffer method 3 cf only 3 spanning ON... so no changes... when i go back to ml menu the total speed with mlv_rec shows 49MB/s. raw_rec goes on 100mb with no prob .
i just saw that i don't have this modules in my folder: ime_base.mo, ime_rot.mo, smpte.mo, trace.mo
that will change something?
What does the second card read out when u have debug on?
Mine for example has B:/dcimblabla/mlv14-000 idle
Like this
(http://www.alisultan.nl/magicbug/idlesdcard.JPG)
could not find that debug window :( where?
Raw Video (MLV) press Q >Scroll to: Status when recording None/Icon/Debug set it to Debug
yes i got it now... so it shows total rate and goes just around 53mb/s :(
tested out mlv_snd works pretty good.. :D
Quote from: g3gg0 on February 14, 2014, 06:33:17 PM
@dubzeebass:
do they play fine on PC?
Did you see my response with the other thread? http://www.magiclantern.fm/forum/index.php?topic=8518.0
Is this something that can/will be incorporated into the main distro?
if you exactly name me what you want to see added?
the audio bars removed and the new bar with ETTR put there isntead?
No the overlays drawing in the right place... Stevefal got all the zebra and focus peaking etc starting at the right area
I got it working with the latest build, but I'm seeing a little percentage progress bar in the middle? What is that? It appears to reset and count up when I move the camera.
Also,.. if I switch to 1280x720 the screen gets smaller. Is there a way for me to make it fit the screen?
:) .. and of course, really amazing work. I was able to convert using MLV converter perfectly.
Noix222 I'm using a Hoodman Steel 32gb card UDMA 7 1000x card And it recorded a full minute 4gb MLV file.. and could have kept going,..not sure how long, but I'm assuming the same as the raw.
Quote from: 5Djp on February 16, 2014, 05:49:40 AM
Noix222 I'm using a Hoodman Steel 32gb card UDMA 7 1000x card And it recorded a full minute 4gb MLV file.. and could have kept going,..not sure how long, but I'm assuming the same as the raw.
30fps ? using MLV im far from it.. it needs 103MB/s to record continuous 30fps 1920x1080 according to ML menu... i get total rate of 53mb/s using mlv_rec with fps override off, both hacks on, buffer method 4, cf 3, spanning mode on, .. don't really know whats up with spanning here my cards (komputerbay 64gb 1000x) SD sandisk extreme pro 32gb - ML installed) using same card but raw_rec i can get 98to100mb/s with no problem almost continuous ... but gives me about a minute recording +- 1500 frames...
Checking the nightly build "v2.3.NEXT.2014Feb16.5D2212" "5D2" MLV Raw
Thanks :) for the option back for The "information text box" instead of just the icon
when recording Raw. I thinks its important to have this text box so you know what's happening.
I know its very important me & my workflow.
The "Frame Over Ride" has been disable when audio is enable ?
This was never a issue , I thought.
No Crop marks , when recording but I use a Evf so not a problem for me.
Seam to be a little performance lose with hdmi device attach, 1856x928 is not continuous(with Lexar 1000x)
Used to be ,back on a.d.'s build feb4 & 14th.
Hi-resolution preview in crop mode when recording is still pinky.
:)
This is for the mark iii correct? Also I'm running 24fps, maybe thats why.
magiclantern-v2.3.NEXT.2014Feb16.5D3113 and the latest MLV_rec
Yeah, I just switched mine to 30fps, and it said write speed around 92mb per sec, then it stopped because of frames skipped. So try 24fps is my only suggestion. I'm switching mine back.
Also, when switching between frame rates.. 30fps to 24fps.. you need to remove the sd card with the bootable magic lantern on it. ( ie.... remove sd card.. shut down camera,.... boot up camera and switch to 24fps,.. shut down.. add the sd card again with magic lantern... and boot it up again.) Maybe this is standard practice but I'm new to it. :)
It makes sense though, since it loads on the sd card.
Could anyone provide me a link to most up-to-date mlv-dump, mlv2dng etc directory? I'm going all around the forum and can't find such a thread. It would be great to create one (all-in-one downloads). Thanks!
After tons of work, using dng's, and premiere, .. and afterfx.. The workflow sucks.. hands down.
Any chance of extracting the raw to .exr? Its open source and followed by multiple programs. Nuke, Afterfx, Premiere..etc
I'm doing the latest build on the Canon 5D Mark iii and MLV (extracting with MLV Converter).
Quote from: g3gg0 on February 14, 2014, 06:33:17 PM
@dubzeebass:
do they play fine on PC?
@mario1000:
yes, thats possible. thinking about side effects atm
Dear g3gg0,
I found another interesting script for VirtualDub which claims to be able to fix dropped frames in a video:
https://www.youtube.com/watch?v=1kaDcFBpbdU
Would this be something which could be implemented into a post process workflow for MLV or RAW files?
Best regards
This one simply requires the converter to fill the missing frames with a copy of last good frame. Their solution is quite clever if you ask me.
So, I believe it should fit as an EasyCoding task (the only tricky part should be when rounding the timestamps). The same workaround can be used to interpolate pink frames too, but autodetecting them is a little harder.
yeah that was already requested iirc and can be done exactly how alex described.
when the last frame was older than 1.5 times the expected frame time, output again.
if(timestamp - last_timestamp > (3/fps/2))
{
output_last_frame_again();
}
with proper casting / scaling of course
probably a dumb question, but why can't I get the DNG's converted with mlv converter to open in resolve? :/
if this has been discussed already don't kill me lol, just point me in the right direction.
(just seen that other folks are having workflow issues as well)
Quote from: 5Djp on February 18, 2014, 07:00:47 AM
After tons of work, using dng's, and premiere, .. and afterfx.. The workflow sucks.. hands down.
Any chance of extracting the raw to .exr? Its open source and followed by multiple programs. Nuke, Afterfx, Premiere..etc
I'm doing the latest build on the Canon 5D Mark iii and MLV (extracting with MLV Converter).
I am sure it could be done, but it would be a fair amount of work and I bet you will have a hard time trying to find someone to work on that when there are many more pressing issues. I may be wrong but I believe mlv can dump to tiff's? If you want exr's you would have to export them form after effects, which I personally don't recommend since AE only does 32 bit- no half float option. 10 bit dpx files using a log camera profile is a good alternative and is much faster to work with than exr's even in nuke. Also if you are using nuke you can work with the dng's directly if you install the j_ops plugin, don't think it works with nuke 8 yet though.
You could also give a resolve a try, it can open the dng's and it exports much faster than AE, adobe debayer is still the best in terms of quality though.
Unfortunately the raw workflow is much more intensive, that is the price we pay.
Thanks bnvm, I've not heard of j_ops. (and waiting for .dng's to be supported in Premiere) Dng's work great in afterfx :)
Exr's would be great, since everything supports them. I'm still trying to get some of our coders at work excited about it so they can make something for me/us.
http://www.openexr.com/downloads.html
Quote from: gravitatemediagroup on February 18, 2014, 07:06:34 PM
probably a dumb question, but why can't I get the DNG's converted with mlv converter to open in resolve? :/
if this has been discussed already don't kill me lol, just point me in the right direction.
(just seen that other folks are having workflow issues as well)
Check that the catalog name containing dng files ar not to long! Try renaming to something shorter to see if that is the problem. Davinci does not like long catalog names and deep catalog structures!
tested mlv video with audio with newest nightly built from feb 20 on my canon 5d mk3 and had a problem because of inconsistent frames exposure?..
have converted the mlv (which was spanned (i always span on both sd and cf card) during recording) with different methods, with no difference (mlv_dump from 4th jan, and mlv mystic 0.3 on a mac)
when i check the converted dng frames some of the frames are brighter than the others..
i have not used any special options (no dual iso,...)
here is the link of 2 dngs:
https://www.dropbox.com/sh/xxfb3xj5r085wz8/i7ZqhCET2d
thank you
Try outside in natural light. Artificial lights may cause flicker.
@alex..
it is not rhythmical.. it happened 3-4 times in 10 sec.. so i dont think that it is from artificial light - i had no flicker in liveview..
There are no differences in black/white levels. To me, it looks like a difference in saturation.
I'm not sure how Canon's flicker correction works (whether it's applied on raw or just on jpeg). They do that on top of shutter speed matching with mains frequency.
ok, only wanted to report it.. cause it is really eye catching..
greets.
Has anyone succeeded in getting card spanning to work? reading back through earlier posts it was working and now doesn't seem to be working, see previous page.
I tested it using 1080 30p and can't get over 10 seconds, when I look at the files on the cards the SD card only has extremely small file saved, it seems to be writing to the CF fine.
Im using a 5d Mkiii with 128Gb Lexar 1000x CF and 64 Gb Sandisk Extreme Pro SD and yesterdays build
should be fixed in the next nightly.
Thank's, Awesome!
i think pink frames for 550d has been solved. i m not very sure. need to do more test. i record 4 clips and there r no pink frames in all the clips
Just wanted to say I was able to convert using MLVconverter on this fine,. to .dng's and a wav file. /// but also to use XnView to convert the .dng's to dpx.
I converterted the .dng's using XnView to .dpx and it works in Adobe premiere...
Using the free XnView www.xnview.com/en/xnview worked for dpx and it opens in Adobe Premiere CS6.
Use XnView and select all your .dng's ... right click or hit ctrl+U to do batch processing and go to "options" .. and on DPX/Cineon,... click the checkbox on ((("Write DPX in 10bits/component"))) and click "ok" and "Go" in the window below it.
...and i just checked.. it opens in Adobe Premiere CC too.
I'm a bit new to Magic Lantern and all these formats, but converting in XnView with those options seems to work in Premiere.
@5Djp
great finding. have to try. but do not forget, you re depending on the demosaicing-algorithm in xnview..
Quote from: 5Djp on February 22, 2014, 09:14:58 AM
Just wanted to say I was able to convert using MLVconverter on this fine,. to .dng's and a wav file. /// but also to use XnView to convert the .dng's to dpx.
MLVConverter isn't doing anything on my end. I click convert and nothing happens. Unable to find more on this... any help?
I have run some tests with the last nightly build for the 7D and wend i preview the files in camera and return to Lv the box have move down. Need to turn off and on the camera for return to the right frame in Lv. Im recording at 2.35:1.
The magic zoom it's not so much flickering now, it's so great. And until now didn't find any corrupted/pink frame.
Card spanning is working :) 1920x1038 30fps is shooting continuous
Settings: status when recording NONE/Memory Hack ON/Extra hack ON/Buffer Fill 3/CF only 1
Thnx g3gg0!
PS: status when recording DEBUG gives me a around 1300 frames with above settings
Quote from: tjaja on February 22, 2014, 06:47:33 PM
Card spanning is working :) 1920x1038 30fps is shooting continuous
Settings: status when recording NONE/Memory Hack ON/Extra hack ON/Buffer Fill 3/CF only 1
Thnx g3gg0!
PS: status when recording DEBUG gives me a around 1300 frames with above settings
Please can you let me know what card and camera you are using. thnx
Lexar 32GB 1000x and Sandisk Extreme 45MB/s. I also perform card warmup btw. the resolution is 1920x1038 not 1080p.
Try recording with both hacks and no buffer or debug status. it is faster. Don't know if that's some quantum mechanics voodoo trying to go gangster on my 5d, or that the buffer/debug status is making use of the SD card.
Thanks for the the heads up on the 1038, I have 2 CF cards, a Lexar 128Gb 1000X and a Lexar 1066X 64Gb my SD card is a Sandisk Extreme Pro 95MB/s 64GB, Im testing them today.
When I run a quick benchmark on the CF and SD I get:
Lexar 1000X 128 GB
SD 19.8 MB/s
CF 77.5 MB/s
Lexar 1066X 64 GB
SD 19.8 MB/s
CF 76.4 MB/s
My first question is are these results believable? If so that would lead me to believe 96-98 MB/s is a kind of maximum data write achievable even when CF and SD cards are Combined, have we reached maximum bandwidth?
If anyone has achieved higher with earlier builds, Im aware lots has changed recently, this would really interest me and lead me to explore buffer settings etc a bit more
When I use RAW_REC I can, with small hacks on, record 1920 x 1038 30p continuously at a rate of 99.6-99.8 MB/s with the camera icon green continuously with the Lexar 64Gb 1066x card
With the benchmarks on Markus's Komputerbay 1200x CF much higher than my 1066x Lexar I would think we're close to 1920 x 1080 30p using RAW_REC. That is unless the 100MB/s is a ceiling so to speak.
I have the same experience. I think it is a error of jumping between buffers thing but I am no programmer. If you turn on Buffer Graph and trie different methods sometimes the buffers aren't nearly full and still a frame skips.
From Cf-only buffers 5 and up, the SD card delays when it starts recording, but when it starts after a second or two a frame is skipped.
Anyways: maybe a new topic with only card spanning results makes the it easier to follow and discuss.
Quote from: arrinkiiii on February 22, 2014, 03:16:53 PM
I have run some tests with the last nightly build for the 7D and wend i preview the files in camera and return to Lv the box have move down. Need to turn off and on the camera for return to the right frame in Lv. Im recording at 2.35:1.
Same here, the box move down after preview. I used 16:9 ratio (7D).
Quote from: tjaja on February 22, 2014, 10:45:15 PM
I have the same experience. I think it is a error of jumping between buffers thing but I am no programmer. If you turn on Buffer Graph and trie different methods sometimes the buffers aren't nearly full and still a frame skips.
From Cf-only buffers 5 and up, the SD card delays when it starts recording, but when it starts after a second or two a frame is skipped.
Anyways: maybe a new topic with only card spanning results makes the it easier to follow and discuss.
Agreed on buffers, something doesn't seem to be functioning correctly, the code isn't maximizing the potential bandwidth for whatever reason, like yourself I am no programmer but Im happy to test.
Quote from: tjaja on February 22, 2014, 10:45:15 PM
I have the same experience. I think it is a error of jumping between buffers thing but I am no programmer. If you turn on Buffer Graph and trie different methods sometimes the buffers aren't nearly full and still a frame skips.
From Cf-only buffers 5 and up, the SD card delays when it starts recording, but when it starts after a second or two a frame is skipped.
Anyways: maybe a new topic with only card spanning results makes the it easier to follow and discuss.
the buffer details are not printed as fast as the buffer is really filled.
this could explain that behavior.
Quote from: tjaja on February 22, 2014, 07:36:58 PM
Card spanning is working :) 1920x1038 30fps is shooting continuous
Settings: status when recording NONE/Memory Hack ON/Extra hack ON/Buffer Fill 3/CF only 1
Thnx g3gg0!
Lexar 32GB 1000x and Sandisk Extreme 45MB/s. I also perform card warmup btw. the resolution is 1920x1038 not 1080p.
Try recording with both hacks and no buffer or debug status. it is faster.
I did some testing today on a 5Dmk3 using those settings. I am running on a Komputerbay 64gb 1000x and Sandisk Extreme 16gb 45mb/s but i can't get continuous 1920x1038 30fps using mlv_rec.. even though using raw_rec i get continuous 1920x1038 30fps with buffer on 99.6mb/s ... using mlv debug it shows a constant 17MB out of SD but a variable bit rate from CF, higher in the beginning, from 90mb/s to 60mb/s.... sometimes lower.... recording around 30 to 40 sec.
Make some more test with the last ML Nightly for the 7D and the bug still there. Wend you record with mlv_rec at 2.35:1 and then you preview the clips in camera with mlv_play and you return to the Lv the black screen for the 2.35:1 stay in the same position that used in the mlv_play and that is different wend you are in Lv ready for record.
For fix this is need to turn the camera off and on. It's not possible to fix this?
Did a quick test today with 1920 x 1080 30p with card spanning on, managed 15 seconds of footage thats 5 seconds more than yesterday.
With 1920 x 1038 30p I got 25 seconds. much improved., I will play with buffer fill methods later today.
I repeated this with MLV_SND and got 23 Seconds
MLV_SND, Im not sure if this is working the audio meters are not registering any levels when the module is turned on, It did with yesterday build. seems to be working again, ok so temperamental. Ill try to find out what caused it to be on/off.
One question to anyone out there who has experience with card spanning, do you have to set the Canon menu to record to both cards or does ML override that setting and as such it isn't a concern? I currently have the Canon menu set to record to both cards and files are being written to both cards.
you dont have to select anything in canon menu.
just make sure your primary card is CF
Thanks!
Hi there! I'm testing the nightly from the 15th of February in a 5D Mark III and I've had some issues with audio. I have the default MLV settings (2.35 aspect ratio) + MLV_Sound + Global Draw on while recording. The dngs are all fine, but some of the wavs stop in the middle of the recording and others are simply empty (it happens especially in the 3x crop recordings, but those are understandable). I've search around the forum to see if this was a known issue but I didn't find anything. Am I missing something?
Anyway, thanks for the great job. The new format and the audio integration are unbelievably amazing :)
PS: I tried the 23rd nightly but I had mayor issues with audio stopping and black frames in the middle of the recording (?), so I decided to try the version from the first post of this thread, which I've found to be much more stable and reliable.
5D mkII Feb 24 tests report.
1. Low FPS freeze bug is FIXED
2. Black border bug when initially turned on is still there. It is off centered and one way to correct is by cycling through magnify button.
3. In 3x crop mode, some borders and framing is not accurate.
ex. 1728 x972 16:9 portion of top and bottom cut off, 16:9 is more like 2:1, recorded image is not affected.
4. tested continuous recording resolutions, tested at least past 10 minutes.
These resolutions are based on lexar and komputerbay 32 and 64 gig 1000X cards , komputer bay 128 gig 1050x will have slightly lower performance.
1X
1872 x 850
1856 x 928
1792 x 968
1728 x 972
3X
2048 x 872
1920 x 872
1856 x 844
1728 x 972
Trying to absorb the information in this thread, I am still unsure if less frames when recording mlv as opposed to raw is to be expected or not.
In raw, I can do 48fps in 1920x672 continuous on my Komputerbay 1000x 64. With mlv, I get around 10-15 seconds depending on other settings before the first frame drop. This is without card spanning. Te mlv format is quite the upgrade imo. I don't really need sound, but the meta data surely is useful! Thanks :)
-
MLV_REC and MLV_SND working on full HD 24p with 25th build and card spanning on. 30p not quite there yet drops after 10-15 seconds
CF and SD Benchmarks are up 5 MB/s for the CF card, looks like the buffers are smoothing out, SD - 18.9MB/s CF - 84.9MB/s
I'll check the footage and sound later today to see if there are any errors
TESTING build magiclantern-v2.3.NEXT.2014Feb26.5D3113.
5d mark 3 - ALL TESTS DONE in 30fps WITH a CF Komputerbay 64gb 1000x and SD Sandisk Extreme 45mb/s.
MLV_REC settings: status recording Debug _ memory hack ON _ extra hacks ON
buffer fill method 4 _ cf-only buffers 3 _ card spanning ON _ reserve card space ON
30fps 1920x1080 16:9
1x mode_ recorded 30 sec. _ MAX rate 99.3mb/s _ CF +- 80mb/s _ SD always on 17mb/s
5x mode_ recorded 50 sec. it always skipped a frame on 50 sec and stopped _ MAX rate 101.5mb/s _ CF +- 87mb/s _ SD always on 17mb/s
30fps 1920x1038 1:85.1
1x mode_ recorded 50 sec. it always skipped a frame on 50 sec and stopped _ MAX rate 98mb/s _ CF +- 80mb/s _ SD always on 17mb/s
5x mode_ recorded continuous _ MAX rate 101mb/s _ CF +- 87mb/s _ SD always on 17mb/s
So besides CF been inconstant the only problem i can report is this 50sec stop in both resolutions.. they're kind at the same time and both stops even though little icon is green showing continuous recording... it doesn't show "its a bug report" it only skip a frame and stop...
RAW_REC settings: memory hack ON _ small hacks ON
30fps 1920x1080 16:9
1x mode_ recorded 40 sec. _ MAX rate 99.3mb/s
5x mode_ recorded 1min. _ MAX rate 101mb/s
30fps 1920x1038 1:85.1
1x mode_ recorded continuous. _ MAX rate 99.6mb/s
5x mode_ recorded continuous. _ MAX rate 101mb/s
So it's clearly that the buffer works a bit faster recording in 5x mode. Running with both modules raw_rec & mlv_rec...
CF buffer needs to be a bit more constant in MLV_REC to be able to record continuous 1920x1080 30fps in both modes 1x and 5x...
I also processed a lot of the images i tested today from both modes mlv and raw... could not find any error or pink frame... it's all good to me !
thanks for the work guys amazing.
-- g3gg0 any advice on buffer config? thanks
can anyone advice me which consumer camera should i buy for recording mlv raw? i got 550D. it can shoots 1152 X 432 continous which is very small resolution so i m planing to upgrade
Quote from: andy kh on February 27, 2014, 02:03:11 PM
can anyone advice me which consumer camera should i buy for recording mlv raw? i got 550D. it can shoots 1152 X 432 continous which is very small resolution so i m planing to upgrade
It depends on how much you want to spend...
Quote from: kgv5 on February 27, 2014, 03:08:04 PM
It depends on how much you want to spend...
i think i should go for 650D since it can record 720P. thanks
Has there been any discussion about writing Magic Lantern DNG metadata in iXML format into the MLV wav file (sidecar) for something like Blackmagic Resolve's extensive metadata use?
I've so far tested a 3 line batch script which dumps the contents of the MLV's into named folders and then uses the open source CLI utility BWFMetaEdit, found here: http://sourceforge.net/projects/bwfmetaedit/files/binary/bwfmetaedit/1.3.0/ (http://sourceforge.net/projects/bwfmetaedit/files/binary/bwfmetaedit/1.3.0/) to write the contents of an iXML file into the .wav acting as audio + sidecar.
For %%a in (*.MLV) do mkdir "%%~na"
For %%a in (*.MLV) do mlv_dump.exe -f 10 --dng --cs3x3 -o "./%%~na/%%~na_" %%a
For %%a in (*.MLV) do bwfmetaedit.exe "./%%~na/%%~na_.wav" --in-iXML="my_iXML.xml"
Contents of iXML file.
<?xml version="1.0" encoding="UTF-8"?><BWFXML><IXML_VERSION>1.5</IXML_VERSION><PROJECT>Test</PROJECT><NOTE>Blackmagic Metadata Write Test</NOTE><BLACKMAGIC-KEYWORDS>magic,lantern,test</BLACKMAGIC-KEYWORDS</BWFXML>
There are a lot of camera related metadata options for the the usual, like ISO, WB, Shutter, Camera Model, Firmware Version as well as more project related tags. Which I could work through testing the tag names and writing the data to wav.
So is it something a dev would consider implementing DNG EXIF into wav sidecar? This would also open up anyone wanting to develop a metadata entry GUI in camera to write maybe project and multicam type data via ML?
If not then I suppose there's someone writing a MLV extractor which could do the extract EXIF data with EXIFTool, write to XML file, write to wav and so on.
It's then simple in Resolve to Import Media by "Add Folders & Sub Folders To Media Pool (Create Bins)" to give each MLV folder a separate bin containing the DNG sequence and wav, then choose the "Auto Sync Audio Based On Timecode", to link the wav to the DNG sequence resulting in wav metadata sidecar contents being displayed in the Resolve Metadata Interface for any DNG or wav selected.
TESTING TODAY BUILD magiclantern-v2.3.NEXT.2014Mar02.5D3113
5d mark 3 - ALL TESTS DONE in 30fps WITH a CF Komputerbay 64gb 1000x and SD Sandisk Extreme 45mb/s.
MLV_REC settings: status recording Debug _ memory hack ON _ extra hacks ON
buffer fill method 4 _ cf-only buffers 3 _ card spanning ON _ reserve card space ON
Good news i was able to continuous record 1920x1080 30fps in 5x mode... the weird stuff is that in 1x mode the buffer performance changes a lot.. it goes down to 98mb/s max...when in 5x mode it goes around 102-103mb/s... don't know why but i guess it's close... thanks guys
This is something that has confused me greatly, I don't get why the crop mode will do full HD at 30p and 1x mode will not. I'd love to understand why the buffers operate differently under the two modes.
Developers, can you possibly shed some light on this?
Using nightly march 2nd /14 , 5d3, Mlv , 1920 x 1080 24 fps,
GD allow , Status when recording Debug , files> 4GiB (exfat),
card warm up 128 , memory hack on , buffer fill method 4,
cf buffers 3 , card spanning on , reverse card space on.
Shooting continuous with sound no problem extract perfect with
MLV Mystic. When trying FPS overide loosing sound and the camera bugs completely.
Impossible to go back to the numbers like MLV 1920x1080 continuous but it writes
in yellow now 82.9 MB/s at 23.976p expect around 0 frames at 79.8 MB/s.
( CF card transcend 1000x 64gigs) . Switched off the camera and turn on again same message
exept this time was abble too shoot again continuous but camera erraticly.Take battery off
everythnig back in place. touching the FPS overide function seems to jam every thing .
Have you tried to remove the MLV's already on the card larry77?
And buffer fill method 0-3 gives me better results, but I don't know if that is card dependant.
After two hours of testing these two resolutions and frame rate got stuck in my head.
3584x1194 15fps fills up with 112MB per seconde.
1920x1080 30fps fills up around 98MB per seconde.
As in data rate these two should be equal, but clearly a larger image written less often after each other is easier on the cards to write.
Going up with the resolution while going down with the frame rate show that behavior.
Does that have to do with random vs sequantial writing?
In Buffer method 0 where it both cards start immediatly with recording the larger images it even produce a steady line on the graph
(http://www.alisultan.nl/magicbug/VRAM37.BMP)
Removing MLV's no problem, I'll try buffer fill method 0-3 , but what about FPS overide
it wasn't a problem with raw.rec. ?
I guess it has to do with sound, for the rest , the nightly build seems very stable
and a dream to shoot with, thanks for the genius programming .
@g3gg0
started with mlv-support in raw2cdng, nearly done. but:
* (5DIII, feb/dontknowversion) examplerecording.
* in [MLVI] is stated videoFrameCount=215, but just found 212 RAWI [VIDF]-Blocks.
* for understanding audiopart - its realized simply writing RIFF-Header according to [WAVI]-Data and concatenating [AUDF]-Chunks.. right?
regards chmee
@all:
sorry for the delays, quite busy with job and family atm :)
maybe you can draw a conclusion - as far i understand some modes suffer from the last changes and some others are much more stable now.
is that correct?
@chmee:
RAWI blocks? i think you mean VIDFs. thhey should match of course.
which recording do you mean? want to check that.
i would not rely on the information in the header. if recording stops premaurely you have zero there.
this is a hint, just for comfortable previews.
you have to build RIFF header on your own - just the wav information has the same format.
then simply concatenate all audio frames (in order), right.
thx. jepp. mean [VIDF]. will upload the mlv when i'm back home, 2 days. (finally, its working. next days i will look on stripped mlv's, because as in raw the parted frames making me some headache in terms of "beautiful code")
regards chmee
Quote from: g3gg0 on March 04, 2014, 09:37:00 AM
@all:
sorry for the delays, quite busy with job and family atm :)
maybe you can draw a conclusion - as far i understand some modes suffer from the last changes and some others are much more stable now.
is that correct?
Testing new builds its evident that the speed is improving... the only strange thing to me still is that the speed recording in 1x is not as stable recording in 5x... as i said befor i am able to record full hd 30fps continuous in 5x mode, but i cant do in 1x mode... the buffer goes from around 103mb/s in 5x compared to max 98mb/s in 1x mode.
thanks guys
What settings did you use with 1080p 30fps in 3xzoom Noix222?
Quote from: tjaja on March 04, 2014, 06:27:35 PM
What settings did you use with 1080p 30fps in 3xzoom Noix222?
i wrote my tests on page 61 with builds magiclantern-v2.3.NEXT.2014Feb26.5D3113 & magiclantern-v2.3.NEXT.2014Mar02.5D3113 - got better results with the second.
5d mark 3 WITH a CF Komputerbay 64gb 1000x and SD Sandisk Extreme 45mb/s.
MLV_REC settings: status recording Debug _ memory hack ON _ extra hacks ON
buffer fill method 4 _ cf-only buffers 3 _ card spanning ON _ reserve card space ON
Is there a possibility to refresh free space field after each shot ? Now if think we need to restart camera to refresh it.
Quote from: noix222 on March 04, 2014, 06:19:00 PM
Testing new builds its evident that the speed is improving... the only strange thing to me still is that the speed recording in 1x is not as stable recording in 5x... as i said befor i am able to record full hd 30fps continuous in 5x mode, but i cant do in 1x mode... the buffer goes from around 103mb/s in 5x compared to max 98mb/s in 1x mode.
thanks guys
Very strange. In which resolution you tried? 1920x1080?
Hi,
I've been shooting some test footage using mlv_rec on a Canon Mk3 113 firmware. The build I am running is the 07/03/2014.
I've noticed a problem with the MLV files and I couldn't find anyone else in the forum that seemed to be experiencing similar issues.
When I change a camera setting during recording (ISO or aperture) it seems to create a problem when it comes to converting to DNG using all of the current MLV conversion tools on both mac and windows.
It kind of 'trips up' the converting program and causes it to stop making DNG's from the rest of the MLV at the point that the ISO or aperture adjustments were made.
Using MlRawViewer on mac I was able to skip past the point in the clip where the camera settings changed, but extracting a full usable take is not possible.
If I do not adjust any camera settings during recording I can shoot any length of clip my card allows and convert to DNG with no problems.
I have tested many times and it is definitely the change in camera settings that causes this. Does anyone else have this problem?
I hope this feedback helps the very impressive ongoing development of the MLV format.
Ben
is there any way to get rid of pink dots on 650D/700D? i know there is an application to remove pink dots but thats not the best solution so is there any other better solution? is magic lantern working on this? or this pink dots wil remain forever if so i m gonna sell my 650D and get another camera.
inside mlv_dump exists chroma smoothing, IF you convert directly into dng. see first entry in this thread.
-- DNG output --
--dng output frames into separate .dng files. set prefix with -o
--no-cs no chroma smoothing
--cs2x2 2x2 chroma smoothing
--cs3x3 3x3 chroma smoothing
--cs5x5 5x5 chroma smoothing
btw. g3gg0 a1ex and 1%
did you try to recalculate the raw-values?
(http://s14.directupload.net/images/140307/9q4rdhen.jpg)
regards chmee
Recalculation looks for me as a great idea. And doing this outsite of chroma-smoothing would make the cold pixel fix not be needed anymore, I guess? Rest the answer of the speed. Is this recalculation fast enough for to can work with it?
Edgar
It won't replace the cold pixel fix; it will simply make these pixels harder to identify.
Other than fixing compatibility issues with Premiere (any others?) I don't see the point. You'll introduce some quantization errors, you'll have a harder time if you want to denoise or subtract dark frames (because you will lose the Gaussian distribution of shadow noise), and that's pretty much it.
The Nikon guys not exactly happy with this behavior in their cameras (for example http://forum.orpington-astronomy.org.uk/index.php?topic=8917.0 )
@escho no it doesnt help with dead/coldpixels.. and yes, its fast enough. in loop theres only one multiplication and one subtraction per value.
but: most of the users won't anything other than direct working inside premiere?! it could be just a commandline-option.. by now raw2cdng is the only one is doin' that, and the guys are happy.. thats the point. linux and mac users are desperated.. all GUIs would draw on it.
and to be honest: quantization errors are the least problem.. (not to say academic)
Yes, an option makes sense. Maybe even an option for linear DNG output, already demosaiced (in this case I'd go for AMaZE, for which we already have the source code).
hello guys... just tested again same settings 1920x1080 30fps with a 5dmark3 using build magiclantern-v2.3.NEXT.2014Mar08.5D3113
Again at 1x mode CF buffer dropped a lot staying on a average of 78-75mb/s.. sometimes lower out of CF and a constant 17mb/s from SD. But at 3x mode i was able to continuous record 30fps on average buffer 102-103mb/s total, with around 85mb/s out of CF and 17mb/s out of SD. SD speeds its constant recording on both modes.
the workflow is pretty easy and i had no errors on my footage converting mlv files using MLVMystic http://secondsparkproductions.com/index.php/blog/8-mlv-mystic-mlv-raw-to-dng-batch-converter (http://secondsparkproductions.com/index.php/blog/8-mlv-mystic-mlv-raw-to-dng-batch-converter)
i also tested and really liked MlRawViewer http://www.magiclantern.fm/forum/index.php?topic=9560.msg91165#msg91165 (http://www.magiclantern.fm/forum/index.php?topic=9560.msg91165#msg91165) it previews the file and you can easily export a ProRes422 (HQ) just pressing a rec button.
i wish i could help on anything more than just testing but i have no idea how to deal with all this codes..... ! but thanks everyone involved.....
using raw_rec the difference on buffer is lower i get constant 99mb/s 1x mode and a max of 101mb/s on 3x mode.
Quote from: chmee on March 07, 2014, 10:39:30 PM
@escho no it doesnt help with dead/coldpixels..
Yes, Alex und you, you are right. Looks like my thoughts was a bit crazy yesterday :)
Edgar
Fix for dead pixels: http://www.magiclantern.fm/forum/index.php?topic=10697.msg104050#msg104050
Thank You for the link. This descripes a nice method, which can be used on some cameras, to map out bad pixels inside the camera. But this cannot work on ML-Raw-video. The newer versions of raw2dng have their own cold-pixel fix, enabled per default. And in the newer versions of mlv_dunp you can enable the cold-pixel-fix of raw2dng for MLV-clips using the commandline-option --fixcp.
Edgar
i'm not sure if the remapping is in between - kind of "everytime-remap-array-before-the-real-sensor-output". tests will show..
It works on MLV video on a 5d3
Quote from: Ben Tubby on March 07, 2014, 02:38:48 PM
I've been shooting some test footage using mlv_rec on a Canon Mk3 113 firmware. The build I am running is the 07/03/2014. I've noticed a problem with the MLV files and I couldn't find anyone else in the forum that seemed to be experiencing similar issues.
When I change a camera setting during recording (ISO or aperture) it seems to create a problem when it comes to converting to DNG using all of the current MLV conversion tools on both mac and windows. It kind of 'trips up' the converting program and causes it to stop making DNG's from the rest of the MLV at the point that the ISO or aperture adjustments were made.
Using MlRawViewer on mac I was able to skip past the point in the clip where the camera settings changed, but extracting a full usable take is not possible. If I do not adjust any camera settings during recording I can shoot any length of clip my card allows and convert to DNG with no problems. I have tested many times and it is definitely the change in camera settings that causes this. Does anyone else have this problem?
I can confirm this issue. I thought I was going crazy! If you change the shutter speed, ISO or WB during mlv recording then the resulting file will be corrupt. You can test for this by playing back in camera where playback freezes on the frame where the settings were changed, giving the message:
frame and block size mismatch: 0x375F00 0x23A9AB 0x376EB4
(for example)
None of the conversion applications can convert these files. Here's an error message from MLVBrowseSharp:
(http://i.imgur.com/xX1pLVX.jpg?1)
I did notice that this issue doesn't exist in the Feb 23rd build, but does appear from Feb 24th onwards. Hope that is helpful. Keep up the good work!
Should add that accidentally hitting the zoom button while recording in MLV will crash out ML and requires removal of the battery.
That's kind of expected, you can't change the raw size in the middle of recording.
adding audio meter for 650D possible with newer builds?
-> trying to isolate the reported crashes
just did a few tests with new build magiclantern-v2.3.NEXT.2014Mar10.5D3113 on my mark3 again shooting 30fps 1920x1080.
Im trying to get continuous 30fps 1920x1080 with mlv_rec. Today was weird it got straight 90mb/s from CF for a few seconds and soon the SD kicked in it pumped to 107mb/s continuous 30fps 1920x1080 recording in 1x mode ... booom . But that lasted just for a few more seconds when the speed dropped really quick and went back to same 98mb/s average witch is 80-81mb/s out of CF and 17mb/s on the SD. So at the end i was able to get nothing more than 30 seconds recording. I did more tests, same thing, the speed can go higher sometimes, but it is constant only around 97-98mb/s (80mb/s out of CF - 17mb/s out of SD) which doesn't really holds recording more than 30 seconds... i also tested record 3x mode.. still working pretty good it doesn't show continuous but i stopped recording after 1min.. and i could go more if i wanted too. i guess it just needs to stabilize it in 1x mode . i wish i could help.
thanks.
Is there a way to still shoot the old (Legacy?) RAW filetype on the current (early March) build? Having problems with pink vertical color stripes in MLV and people are saying they never had that problem with older RAW format.
I searched but couldn't find an answer.
use raw_rec
Pushing crop modes on the 5D2 and using MLV from ML nightlies Mar10.
I notice when using 2144x974 2.20:1 that there is a black pillar on the right side of the extracted DNGs.
This resolution is continuous at 20fps.
I'm hoping there can be a way that monitoring in crop mode can give a faster refresh with a way to get proper framing. This has so much potential.
2 things:
1 mlv_dump doesn´t compile anymor per default, using make. In the makefile isn´t "all:: mlv_dump" anymore. I would linke to have this automation back.
2. mlv_dump shows a compile-error:
edgar@linux-uoww:~/ML/escho-magiclantern/modules/mlv_rec> make mlv_dump
../../Makefile.setup:101: BAD VALUE FOR CONFIG_SITE=/usr/share/site/x86_64-unknown-linux-gnu DEFAULTING TO n
[ HOST_CC ] mlv_dump.host.o
mlv_dump.c: In function 'main':
mlv_dump.c:1397:36: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
xrefs = (mlv_xref_t *)((uint32_t)block_xref + sizeof(mlv_xref_hdr_t));
^
mlv_dump.c:2034:37: error: 'struct raw_info' has no member named 'buffer'
raw_info.buffer = frame_buffer;
^
make: *** [mlv_dump.host.o] Fehler 1
Edgar
edit:
forget to say: compiling newest build incl changeset: e86afc9 - Merged in mofig/mofig-magic-lantern/fontsdat (pull request #433)
1. 'make' or 'make all' per default compiles only the module, no PC tools. this improves automatic build scripts. please use 'make mlv_dump'
2. your default gcc is an x64 compiler. please update Makefile.user to match your x86 compiler
ThankYou for the tips. Many things seems to be changed the last few days.
1. I will do so and use make mlv_dump (have to rewrite my install-script, no problem ;))
2. Added -m32 to cflags and ldflags. Now it compiles.
Edgar
Where can I find a mlv2dng that has audio extraction? Hopefully for os x
Compilation fails on OSX Maverick after Xcode was upgraded to version 5.1 on March 10, 2014.
[Edit:] I should have added that ML autoexec.bin and modules compile OK without any issue even after the Xcode upgrade.
compilation fails only for mlv_dump.
[ HOST_CC ] mlv_dump.host.o
clang: error: unknown argument: '-mno-ms-bitfields' [-Wunused-command-line-argument-hard-error-in-future]
clang: note: this will be a hard error (cannot be downgraded to a warning) in the future
make: *** [mlv_dump.host.o] Error 1
This did not help either:
export CFLAGS=-Qunused-arguments
export CPPFLAGS=-Qunused-arguments
Removed '-mno-ms-bitfields' from modules/mlv_rec/Makefile, but ld was not happy.
MLV_CFLAGS = -I$(SRC_DIR) -D MLV_USE_LZMA -m32 -Wpadded -D _7ZIP_ST -D MLV2DNG
ld: warning: ld: warning: ignoring file ../../src/chdk-dng.host.o, file was built for i386 which is not the architecture being linked (x86_64): ../../src/chdk-dng.host.oignoring file mlv_dump.host.o, file was built for i386 which is not the architecture being linked (x86_64): mlv_dump.host.o
ld: warning: ignoring file ../lv_rec/raw2dng.host.o, file was built for i386 which is not the architecture being linked (x86_64): ../lv_rec/raw2dng.host.o
ld: warning: ignoring file lzma/lib7z.a, file was built for archive which is not the architecture being linked (x86_64): lzma/lib7z.a
Undefined symbols for architecture x86_64:
"_main", referenced from:
implicit entry/start for main executable
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [mlv_dump] Error 1
Removed -m32 from modules/mlv_rec/Makefile, no luck either.
MLV_CFLAGS = -I$(SRC_DIR) -D MLV_USE_LZMA -Wpadded -D _7ZIP_ST -D MLV2DNG
Macintosh:mlv_rec okumar$ make clean && make mlv_dump
[ RM ] mlv_rec.o mlv.o mlv_rec.mo mlv_rec.sym mlv_rec.dep module_strings.h *.o *.d *.dep *.sym hgstamp
[ RM ] mlv_dump mlv_dump.exe lzma/7zAlloc.host.o lzma/7zBuf.host.o lzma/7zBuf2.host.o lzma/7zCrc.host.o lzma/7zCrcOpt.host.o lzma/7zDec.host.o lzma/7zFile.host.o lzma/7zIn.host.o lzma/7zStream.host.o lzma/Alloc.host.o lzma/Bcj2.host.o lzma/Bra.host.o lzma/Bra86.host.o lzma/BraIA64.host.o lzma/CpuArch.host.o lzma/Delta.host.o lzma/LzFind.host.o lzma/Lzma2Dec.host.o lzma/Lzma2Enc.host.o lzma/Lzma86Dec.host.o lzma/Lzma86Enc.host.o lzma/LzmaDec.host.o lzma/LzmaEnc.host.o lzma/LzmaLib.host.o lzma/Ppmd7.host.o lzma/Ppmd7Dec.host.o lzma/Ppmd7Enc.host.o lzma/Sha256.host.o lzma/Xz.host.o lzma/XzCrc64.host.o lzma/lib7z.a lzma/Threads.w32.o lzma/LzFindMt.w32.o lzma/MtCoder.w32.o lzma/7zAlloc.w32.o lzma/7zBuf.w32.o lzma/7zBuf2.w32.o lzma/7zCrc.w32.o lzma/7zCrcOpt.w32.o lzma/7zDec.w32.o lzma/7zFile.w32.o lzma/7zIn.w32.o lzma/7zStream.w32.o lzma/Alloc.w32.o lzma/Bcj2.w32.o lzma/Bra.w32.o lzma/Bra86.w32.o lzma/BraIA64.w32.o lzma/CpuArch.w32.o lzma/Delta.w32.o lzma/LzFind.w32.o lzma/Lzma2Dec.w32.o lzma/Lzma2Enc.w32.o lzma/Lzma86Dec.w32.o lzma/Lzma86Enc.w32.o lzma/LzmaDec.w32.o lzma/LzmaEnc.w32.o lzma/LzmaLib.w32.o lzma/Ppmd7.w32.o lzma/Ppmd7Dec.w32.o lzma/Ppmd7Enc.w32.o lzma/Sha256.w32.o lzma/Xz.w32.o lzma/XzCrc64.w32.o lzma/lib7z.w32.a
[ HOST_CC ] mlv_dump.host.o
mlv_dump.c:2034:38: error: no member named 'buffer' in 'struct raw_info'
raw_info.buffer = frame_buffer;
~~~~~~~~ ^
1 error generated.
make: *** [mlv_dump.host.o] Error 1
just leaving a note. raw2cdng is supporting mlv+audio now as well.
http://www.magiclantern.fm/forum/index.php?topic=5618.msg106173#msg106173
Hello!I do not dump mlv file in raw or in dng with mlv_dump & MlRawViewer (OSX)
In MlRawViewer playing stoping on 46 frames and app hangs(Other video good play and converting on dng.
In mlv_dump this log
Mac-Pro-MacMan:Movie macman$ ./mlv_dump -f30 M07-2323.mlv out.raw
MLV Dumper v1.0
-----------------
Mode of operation:
- Input MLV file: 'M07-2323.mlv'
- Verify file structure
File M07-2323.mlv opened
File M07-2323.m00 opened
File M07-2323.m01 not existing.
File M07-2323.IDX opened (XREF)
XREF table contains 323 entries
Processing...
Reached end of all files after 323 blocks
Processed 299 video frames
Done
I don't have M07-2323.m01 on SD or CF card.
Is it possible dump to from a particular frame. Because if you miss the 46 frame video on playing in MlRawViewer.
Sorry my english. I am from Ukraine. Please help!
Recompiling nightly builds to undo a change that made mlv_rec+mlv_play completely unusable on 5D2 and other DIGIC IV cameras (to reproduce, simply record a short clip and play it back in the camera).
Issue discussed here:
https://bitbucket.org/hudson/magic-lantern/pull-request/438/raw-recording-force-line-size-to-be/diff#comment-1409162
https://bitbucket.org/hudson/magic-lantern/commits/249d4ea65d3679df6a8591ec1af9087dc6c3078f
im still testing new builds (magiclantern-Nightly.2014Mar15.5D3113) on my 5d mk 3... the speed at 3x mode is amazing it holds 103mb/s and i can continuous record 1920x1080 30fps... but again recording at 1x mode the speed drops to 97mb/s... and stops just after 25 seconds.
Noix222 if there arent any changes in the module there Will be no changes in speed. Check the changelog or set to see if there are changes in mlv_rec before comparing nightlies.
Quote from: tjaja on March 15, 2014, 03:23:27 PM
Noix222 if there arent any changes in the module there Will be no changes in speed. Check the changelog or set to see if there are changes in mlv_rec before comparing nightlies.
Fair point, I do have a question though is this reduction in write speed in 1x mode likely to be sorted out in the future? or will it stay as such?
I have a question regarding the MLV header. I'm trying to read the metadata in it, which until so far I have *almost* success. What happens is that the header is not always exact, it varies sometimes by camera model and always varies if you have 'text' or 'take' entries (also depending on the lens model length). What my converter does is to read for example byte number 34, which is the ISO, and boom I get the value. Now, if some entry from 'text' field is present (for example), all bytes in the header shift because of the 'text' length, so the ISO isn't now on byte 34 but on 40 (for example), that causes in erroneous reads of the metadata.
Does anyone know any logic or workaround for this? I'm a rookie programmer but I can manage... But I just don't understand how can it be done. How can I identify a byte that can be anywhere?
Thanks and hope someone could point me in the right direction.
Cheers!
Tony.
Quote from: tonybeccar on March 15, 2014, 06:37:10 PM
Does anyone know any logic or workaround for this? I'm a rookie programmer but I can manage... But I just don't understand how can it be done. How can I identify a byte that can be anywhere?
as the file format description suggests, you have to identify the block by its type (LENS, VIDF, AUDF, etc)
if the block you currently have is not of any interest for you, just skip the block. for that you have the blocksize field.
Quote from: tjaja on March 15, 2014, 03:23:27 PM
Noix222 if there arent any changes in the module there Will be no changes in speed. Check the changelog or set to see if there are changes in mlv_rec before comparing nightlies.
yes your right.. i do that actually.. i've seen a few mlv cleanups and etc from g3gg0 in the changelog a day before or so and because i haven't tested for about 2 or 3 days.... i thought it could have changed something. Just posting reports after 2 or 3 builds and only if there's g3gg0 or a1ex changes..
I had Just tested the 1.2.3 on my 5D3 , Thanks for the Excellent work.
The raw rec was good up to 1920x1080x25p (Normal and crop mode).
But the MLV Raw doesnt work for me more than 1 min (4GB) and then it dumps some debug info about QUED something.
a sample of Logfile that was made to the root dir :
ML ASSERT:
hdr->blockSize >= (sizeof(mlv_vidf_hdr_t) + hdr->frameSpace + frame_size)
at mlv_rec.c:1990 (process_frame), task Evf
lv:1 mode:3
Magic Lantern version : Nightly.2014Mar16.5D3123
Mercurial changeset : a9dced4a127b (5D3-123) tip
Built on 2014-03-16 13:26:55 UTC by
[email protected].
Free Memory : 165K + 3742K
thanks, will check that.
looks like the additional check catched a bug :)
Tested again the 1.2.3 on my 5D3 and some of the MLV files had
a green tint and in addition some where very dark
(compared to other files with same EXPO.)
using rec raw it was ok.
the green tint - was it using a player or after exporting with mlv_dump ?
after exporting with mlv_dump and also viewing with MLRviewer
its on all frames?
can you upload one?
or dump a few frames from the .mlv using mlv_dump and the -f parameter (specify 2 or 3)
its on all frames.
I will later try to post a sample
Quote from: gideonplus on March 19, 2014, 10:17:49 AM
Tested again the 1.2.3 on my 5D3 and some of the MLV files had
a green tint and in addition some where very dark
(compared to other files with same EXPO.)
using rec raw it was ok.
im having same issue using mlv_rec recording at both modes 1x and 3x. Raw_rec i get the same error only in 3x mode. 1x is fine.
One thing i can say about MLV, the speed is going higherrrr.... :D i got about 108mb/s using mlv and crop mode.
@g3gg0
x1 (NORMAL) and x3 lens zoomed out(with green tint and exp problem)
https://vimeo.com/89497294
did you zoom in while recording?
did you try WB using LR or similar?
The lens was zoomed out before recording.
The DNGs untouched.
Did you have memory hack on? Can you check if turning it off helps?
(this one changes video modes right before recording, so it may need a larger delay to settle)
I had the same issue, I wanted to add when I played the files back in camera they looked fine, however when viewed in MlRawViewer or converted they had the dark green color.
@a1ex
I have made some more clips with the memory hack on/off
and yes ,it is the source of the problem.
every time I turn it ON the green tint is there and stays on.
600D
I noticed this very green tint one time until now. I couldn´t reproduce this fault, so I didn´t report it. The dngs couldn´t be repaired by manual white balancing in postprocessing. But I could repair them in changing the blackpoint a bit, using exiftool. That´s just for info.
Edgar
Quote from: a1ex on March 19, 2014, 03:52:45 PM
Did you have memory hack on? Can you check if turning it off helps?
(this one changes video modes right before recording, so it may need a larger delay to settle)
i also had memory hack ON all time... and i was getting the green tint on both modes
| raw_rec 1x green 3x green | mlv_rec 1x normal 3x green |
i tested both 5dmark 3 firmware's same results. Turning it OFF solved the problem.
Suggestion: Have the Global Draw option in mlv_rec, set to on by default. Based on the number of questions in the forums, asking why GD is off when recording.
Hi. Do you have fps override enabled when changing modes with memory hack on?
If so, I think the problem resembles the one I have on 6D. This issue is no related to recording: LV is NOK also. The change of video modes (memory hack requires a set_lv_zoom to return from pause lv) with fps override (with overrides far from fps native) and shutter speed high (1/500 and beyond) is prone to sporadic LV problems. The same thing if we change video mode with the loupe.
But on 6D it's worst than green tint, there are other artifacts.
last few builds I've been hit with a few green death clips :(
Hello RAW world. I am newbie.
does this mlv record in 12 bits? or only converting after?
Now i am using bleeding edge 123, and due to slow card cant write fullhd, but if it 12bit i think it will be possible.
This module can be installed on bleedingedge or i need to reinstall with painful downgrade and other stuff?
Sorry for bad eng.
It records 14 bit. And please make some effort and spend at least couple of minutes searching the forum before posting, it is all there.
the next nightly will contain some fixes to file handling.
please try again if you had issues with corrupted files and 4GiB stopping.
@g3ggo,
Is it possible to transfer the Global draw switches Under RAW VIdeo Sub menu and Consolidate all global draw related options together with the Main Global Draw option in the Overlay tab?
It would be more user friendly if its just bundled together. Or is it because its part of the MLV module?
Hi Everyone. Does anyone know some good workflows for post processing the RAW material. (After you convert 'em to DNG). Thanks in advanced
Quote from: ted ramasola on March 24, 2014, 04:35:30 AM
Or is it because its part of the MLV module?
I assume it's simply because the module is still experimental.
Quote from: filmio on March 24, 2014, 06:28:05 AM
Hi Everyone. Does anyone know some good workflows for post processing the RAW material. (After you convert 'em to DNG). Thanks in advanced
http://www.magiclantern.fm/forum/index.php?board=54
I'm having a hard time finding an explanation on how to merge MLV 2.0 files that span CF and SD cards.
I have spent time going thru the forums, looked at MLVMystic, mlv_dump etc. and cannot find the workflow. I have no problem converting MLV to RAW to DNG, but the spanned merge is eluding me.
For instance, how does one merge the three files that are generated for every spanned MLV:
M23-2258 2.MLV
M23-2258.M00
M23-2258.MLV
ps. this isn't the 4Gb file span issue, I use exFAT to address that
Never mind on the card merge. I used Hex Fiend to manually splice them together - the same technique used to merge >4Gb files.
It might be helpful to mention this workflow somewhere and if the card merge can be automated, that would be fantastic. :)
you do not have to manually merge the MLV file chunks.
every tool has to support reading all of them.
always specify the .mlv file and they will read .m00, m01, m02 etc.
why do you have a "M23-2258 2.MLV" ?
this cannot be from the camera, you seem to have renamed the files?
when you record, the camera produces
M23-2258.MLV
M23-2258.M00
M23-2258.M01
M23-2258.M02
...
and for converting you open M23-2258.MLV in your favorite viewer/converter.
as simple as that. no need for hex editors and byte banging.
Yes, the same filename is being written to the SD and CF cards by ML, which required a rename.
CF: M23-2258.MLV 517 MB
SD: M23-2258.MLV 608 bytes
M23-2258.M00 90.7 MB
Since they are identically named, which MLV file should one import to the convertor?
Build is last night 3/23
Recording 1920x508 @ 60fps
Been having problems with MLV on the 1.2.3 firmware with the last nightly update (March 17)
Most of my vids come out green, while some are still adjustable by the white balance there are a few which go so dark that they cannot be recovered at all.
Here is an example when i open one of the frames in ACR:
https://www.dropbox.com/s/x8hogghq4y7i1j6/Schermafbeelding%202014-03-24%20om%2014.42.56.png (https://www.dropbox.com/s/x8hogghq4y7i1j6/Schermafbeelding%202014-03-24%20om%2014.42.56.png)
Even with the exposure all the way to the right it's still almost black.
Tried a few different methods to convert but none of them worked.
The file is playable in the camera with MLV view and the colors look fine then but not on my imac.
Anyone has any suggestions?
Quote from: Javenu on March 24, 2014, 11:15:24 AM
Yes, the same filename is being written to the SD and CF cards by ML, which required a rename.
CF: M23-2258.MLV 517 MB
SD: M23-2258.MLV 608 bytes
M23-2258.M00 90.7 MB
Since they are identically named, which MLV file should one import to the convertor?
Build is last night 3/23
Recording 1920x508 @ 60fps
oh, wow. didnt notice that.
then rename one .MLV to .M01 and try it with the oter .MLV
Quote from: scarluuk on March 24, 2014, 02:56:09 PM
Been having problems with MLV on the 1.2.3 firmware with the last nightly update (March 17)
Most of my vids come out green, while some are still adjustable by the white balance there are a few which go so dark that they cannot be recovered at all.
Here is an example when i open one of the frames in ACR:
https://www.dropbox.com/s/x8hogghq4y7i1j6/Schermafbeelding%202014-03-24%20om%2014.42.56.png (https://www.dropbox.com/s/x8hogghq4y7i1j6/Schermafbeelding%202014-03-24%20om%2014.42.56.png)
Even with the exposure all the way to the right it's still almost black.
Tried a few different methods to convert but none of them worked.
The file is playable in the camera with MLV view and the colors look fine then but not on my imac.
Anyone has any suggestions?
I too have the same problem, I record with this build "2014Mar23.5D3113" someone could help us please?
(http://i57.tinypic.com/282hton.jpg)
(http://i61.tinypic.com/15own5f.jpg)
Quote from: g3gg0 on March 19, 2014, 11:41:41 AM
its on all frames?
can you upload one?
or dump a few frames from the .mlv using mlv_dump and the -f parameter (specify 2 or 3)
this please
Quote from: g3gg0 on March 24, 2014, 04:07:52 PM
this please
All frames are like this
https://www.dropbox.com/s/alebpmy150ocu5l/MLV_A001_220943_C00012_f000001.dng
and this tells me when I run mlv_dump -f
(http://i61.tinypic.com/sp93ye.jpg)
mlv_dump m22-0943.mlv -o out.mlv -f 3
Quote from: g3gg0 on March 24, 2014, 04:32:47 PM
mlv_dump m22-0943.mlv -o out.mlv -f 3
to execute the command, I have 2 files
https://www.dropbox.com/s/jdwgc3nr251a6q4/out.mlv
https://www.dropbox.com/s/kxlrkc39hcvry5j/out.mlv.wav
and still have the problem
these are the files i need to check what the problem is.
will report tonight.
Quote from: g3gg0 on March 24, 2014, 05:15:20 PM
these are the files i need to check what the problem is.
will report tonight.
Thanx @g3gg0 , very grateful!
@Jared: Download exiftool and exiftool GUI (google them). Change tag "BlackLevel" to a value of 2046. Problem fixed.
We've seen this before, for various reasons. It seems to be only a metadata problem: i.e., your .dng's are actually ok, if you just change this one value. I tried it on the .dng you uploaded and it now looks normal. Cheers
i added a workaround for that black level issue. (next nightly)
i am not sure why the raw backend has trouble in detecting the correct black level so often now.
did you all use zoomed modes?
a) mlv_rec will by default fix that value to 2048. depending on your camera this may be wrong.
iirc alex told me that there may be some models which have 4096. (maybe i mixed smth up)
for this reason there is a menu option to disable that black-fix if you need it.
b) mlv_dump has an option --black-fix to override black level to 2048. for existing footage just run
'./mlv_dump in.mlv --black-fix -o out.mlv' and your footage in out.mlv will have black level set to 2048.
(for windows users: here (http://upload.g3gg0.de/pub_files/2db1eeaea665e6a8e9bb8550d3ba958f/mlv_dump.exe) is a version with the fix)
the changes will take effect with the next nightly.
I had this problem only when the memory hack was on
Hey g3gg0, is there any syntaxis in mlv_dump to start converting from a specific frame until the last one? I used March 13th version in a short and a lot of the files have a bug, in which the first frame is corrupted and mlv_dump can't extract anything.. same bug as Ted Ramasola had.
Anyway any help would be useful! Thanks!!
what was the output of mlv_dump -v again for this bug?
This is the log: https://www.dropbox.com/s/rtl91lw3ath363x/log.txt
Basically: "[ERROR] File ends in the middle of a block. Processed 0 video frames"
you can use -f a-b where a is start and b is end frame number.
but i committed a check which should skip a block if its frameSpace is invlaid.
can you try it? (exe (http://upload.g3gg0.de/pub_files/a950621e1e47531e05a7c76edcdfecf1/mlv_dump.exe))
Quote from: g3gg0 on March 24, 2014, 11:33:49 PM
you can use -f a-b where a is start and b is end frame number.
but i committed a check which should skip a block if its frameSpace is invlaid.
can you try it? (exe (http://upload.g3gg0.de/pub_files/a950621e1e47531e05a7c76edcdfecf1/mlv_dump.exe))
WOW! Alright g3gg0!!!! That fixed it! I tried with -f a-b argument and it worked, it didn't work with the other version of mlv_dump that I had. And it ALSO works normally, without -f parameter. So wow, couldn't be happier!!!! Is this mlv_dump also faster?
Cheers and trillions of thanks!!
tony, have you tested this? Should this replace the one in the tools folder of your app?
Quote from: ted ramasola on March 25, 2014, 12:32:03 AM
tony, have you tested this? Should this replace the one in the tools folder of your app?
I tested it and uploaded a new version of my converter with this version of mlv_dump. But a simple overwrite should work as well :) But it also happened that I forgot to upload the nconvert.exe for the thumbs (I think) and thumbs weren't being created. So I made a new version.
Quote from: g3gg0 on March 24, 2014, 09:30:46 PM
i added a workaround for that black level issue. (next nightly)
i am not sure why the raw backend has trouble in detecting the correct black level so often now.
did you all use zoomed modes?
a) mlv_rec will by default fix that value to 2048. depending on your camera this may be wrong.
iirc alex told me that there may be some models which have 4096. (maybe i mixed smth up)
for this reason there is a menu option to disable that black-fix if you need it.
b) mlv_dump has an option --black-fix to override black level to 2048. for existing footage just run
'./mlv_dump in.mlv --black-fix -o out.mlv' and your footage in out.mlv will have black level set to 2048.
(for windows users: here (http://upload.g3gg0.de/pub_files/2db1eeaea665e6a8e9bb8550d3ba958f/mlv_dump.exe) is a version with the fix)
the changes will take effect with the next nightly.
@g3gg0 Thank you very much, I have retrieved all the frames, you saved my life :)
grateful to all for your feedback!!!
Hi, I have similar issues with the mlv_rec (green effect, corrupt files, challenge opening some files on mlrawviewer).
In terms of reliability, IMHO, it appears that raw_rec is robuster (if there is such a word).
The biggest difficulty I face is that most files that looked okay when shooting and replay on the camera are difficult to handle on the mlrawviewer. Initially, I deleted a few files since the mlrawviewer wouldn't open these. But now I realise that I should try opening the files at least three times before deleting. It could be that the issue is with the mlrawviewer. I simply can't say.
Plus, when the wrong combination of frame size, frame rate and snd is chosen the camera hangs and the battery has to be removed. Normally, the camera should just reject the wrong choice instead of trying to record and hang due to lacking buffer. Raw_rec somehow is better at dealing with this and is more bouncier overall. The obvious advantage is mlv_raw can record sound in camera while raw_rec cannot for the same video quality.
Just yesterday, my 8 year old daughter had a recording session to cut a cd for sending to a music competition. The music teacher arranged for the recording at a professional sound studio. The studio manager told me to arrange for my own video recording. I decided to do it myself. Initially, I thought the best idea would be to record mlv or raw. But then I chickened out last moment since the studio was booked only for an hour with 3 takes allowed. We were paying too dollars to the studio plus another sizeable sum to the pianist girl. So I did not want to take a chance and went back to the tried and tested H.264@24fps.
Having said that, I am now a raw addict and mlv is my preferred format primarily because of the completeness of this package as a whole. After filling up 4TB of memory, I am still hungry for more. The 5D3 is a beast and ever hungry :-)
Overall, things are good as they are. Appreciate all the hardwork. And love every bit of the ML interface.
I dream to be a raw champion one day.
Could the unfortunate users of OSX have a compiled version of the latest mlv_dump ? Thanks a lot .
Anyone know if it's possible to go from MLV to Cinema DNG without going from MLV to RAW first?
Thanks,
Mr D
Quote from: frenchps49 on March 25, 2014, 06:06:08 AM
Could the unfortunate users of OSX have a compiled version of the latest mlv_dump ? Thanks a lot .
I think you could change the mlv dump file to a newer one and simply run this script from here
http://www.magiclantern.fm/forum/index.php?topic=9731.msg93586#msg93586
*realized you would need a mlv_dump binary for it to work with mac.
Quote from: Canon eos m on March 25, 2014, 03:56:22 AM
Hi, I have similar issues with the mlv_rec (green effect, corrupt files, challenge opening some files on mlrawviewer).
thanks for your feedback.
i think the green issue - as reported earlier - is just a result of some changes to the raw backend and maybe some timing issues in combination with the hacks.
(wasnt able to reproduce a single video with green cast after 15-20 tries the last week)
this has always been a wrong black level since then, thats the reason why mlv_rec now hardcodes that field to the most probable value.
for existing footage this can also be fixed using mlv_dump that writes you a .mlv again.
for the crash it would be helpful if you can tell me the settings so i can reproduce.
Like just now. I was shooting casual footage with my family. The camera was on 1920x1080 snd.mo on. Kept each instance below 20 secs. Things appear to work like a breeze. But there was no prior warning that the card was about to fill up. The last bit ran out the capacity and the camera drew a blank screen and the camera stopped responding. Had to pull the battery out. Likewise, when shooting beyond what the CF card can handle (in those situations when I shoot high fps with large frames sizes), I often face the camera freeze issue. Please let me know if you want me to send you the settings on the camera when it froze last.
Don't know whether it is green or not but there are some batches from the footage I got that are black (blown). Can get one of the files to you if that will help. In some cases the files just won't open with mlrawviewer.
please, those are important to me for reconstructing what is wrong.
use mlv_dump in.mlv -f [start]-[end] -o out.mlv to export the bad frames.
please add 2-3 frames in front of the problematic section.
Sorry to be a dud but do I use this code you provided in c: prompt dos shell or some place else.
Yes. Download this: http://ml.g3gg0.de/pub_files/be40d197edf4f979415584de63e4251b/mlv_rec.zip
Extract the contents to a HDD, say C: drive. Copy the MLV files to the same place on the HDD.
Open a command prompt (windows key + x on windows 8 ), navigate to the same place as the files earlier, (type "cd\" to navigate to C: drive), and then run the command line g3gg0 posted above.
Change the in.mlv and out.mlv sections to the name of your input MLV (MLV8000.MLV or whatever), and output mlv.
on the 7D, when using max resolution in crop mode, 2496 x1200 2:1 and 2496 x1134 2.20:1
using MLV sound the recording will not stop properly and will need a battery pull.
Normal stop of recording is possible with MLV Sound OFF or at lower res starting at 2496 x1062 2.35:1.
Why is mlv_dump much slower than e.g. mlv2dng or raw2dng? Is there any plans to speed it up like multi-core support?
oh, it is slower?didnt realize.
especially as it uses raw2dng code for writing dng's.
every 1st clip recorded each startup is killed with green footage, that's the pattern I notice so far with the latest build, maybe I need to turn something off or on?
Quote from: g3gg0 on March 25, 2014, 11:37:50 PM
oh, it is slower?didnt realize.
especially as it uses raw2dng code for writing dng's.
Than mlv2dng yes, like 10-15 times.
Quote from: gravitatemediagroup on March 26, 2014, 01:15:45 AM
every 1st clip recorded each startup is killed with green footage, that's the pattern I notice so far with the latest build, maybe I need to turn something off or on?
did you try the latest nightly of mlv_rec? (after 25.03.)
I check the nightly section twice a day. Last I checked today afternoon (there were no new updates after the 17th March nightly). Checked now and see the 26th March bleeding edge version. Will load today and report results on mlv_raw.
I still owe you the samples for the dark images, skipped frames. Will do this also tonight and send whatever I can.
in MLV crop mode(x3) it crashed after about 2 minutes with a lot of text writen on screen.
The LOGFILE:
ML ASSERT:
hdr->blockSize >= (sizeof(mlv_vidf_hdr_t) + hdr->frameSpace + frame_size)
at mlv_rec.c:1990 (process_frame), task Evf
lv:1 mode:3
Magic Lantern version : Nightly.2014Mar26.5D3123
Mercurial changeset : 48e7ddebc1b9 (5D3-123) tip
Built on 2014-03-26 09:05:23 UTC by
[email protected].
Free Memory : 149K + 3753K
My card is Komputerbay 64GB x1000 , No Memory Hack,GD on,FPS
1920x1080 25p
Hey... Finally I found out what was wrong!
I shot an entire 32GB CF full of nice steadicam shots, came back home, converted the MLVs to DNG... and they were all pink! Not the usual "Premiere Pro" pink... they are pink in ACR, Resolve 10... everywhere. After some trial and error I figured out it was caused by the "Fix black level" set to ON... But since (I guess?) all the data is still there... is there a way to fix it in post (as far as I understood, you can get your "green" shots to "normal" with mlv_dump... so the opposite should be possible too, right?)?
Thanks in advance! :)
Seb
@g3gg0, testing 1920x508 @ 50fps for more than 4/5minutes str8 and i must say the new builds are much faster/more stable. Goooood Job man!!! :)
Quote from: seb3dge on March 27, 2014, 12:09:45 AM
Hey... Finally I found out what was wrong!
sorry, which problem are you referring to?
(camera model etc)
what i am doing wrong?
123 bleeding edge 5d mk3 32gb transcend 1000x. Can't record 1920-1080 29.97 even 2 seconds=|
in crystaldiskmark cf show 154 for read, 114 for write
internal in camera shows 112 for read and only 72 for write.
I am confused
And now he shows me that i should expect around 0 frames and only on 1504h continious, and no info like before about 300 frames or so. only 2 option.
Quote from: GooDween on March 28, 2014, 01:15:26 PM
what i am doing wrong?
123 bleeding edge 5d mk3 32gb transcend 1000x. Can't record 1920-1080 29.97 even 2 seconds=|
in crystaldiskmark cf show 154 for read, 114 for write
internal in camera shows 112 for read and only 72 for write.
I am confused
And now he shows me that i should expect around 0 frames and only on 1504h continious, and no info like before about 300 frames or so. only 2 option.
Is the card selector set to the SD card? That one catches me every now and then.
preferred card cf, magic lantern installed on sd.
can you please:
- enable trace module (trace.mo)
- enable tracing in mlv_rec menu
- record a video
and send me the raw_rec.txt in your SD cards root?
any one having this kind of problem?
jpg: https://www.dropbox.com/s/9v0uut63qawuqhb/101.jpg [ edit: http://a1ex.magiclantern.fm/bleeding-edge/raw/101.jpg ]
original dng: https://www.dropbox.com/s/epbkuhd9j13ncgi/101.dng [ edit: http://a1ex.magiclantern.fm/bleeding-edge/raw/101.dng ]
its present in all my takes of that scene, always around the highlights, first i thought it was a debayer problem in resolve, but it happens the same in after effects and in lightrooom.
any ideas on how to fix it?
Can you upload an underexposed version of the same scene?
jpg: https://www.dropbox.com/s/kf62atk826xq9uw/underexposed.jpg
its the same dng.
underexposed == capture fewer photons, please
i dont think i have a different exposure for that scene, i really wanted those white spots blown out cause afterwards my subject come into frame and i wanted him well exposed. I could upload the original mlv file if you want.
The issue is also present in dcraw. These were cold pixels, but their value was not zero.
I've just fixed (https://bitbucket.org/hudson/magic-lantern/commits/ea59b5b6517a) the issue in raw2dng (http://www.magiclantern.fm/forum/index.php?topic=5404.0).
cc @escho and g3gg0 to port the fix in mlv_dump. I did a quick grep and couldn't find the cold pixel fix code.
thx, will try fix tomorrow.
Quote from: a1ex on March 28, 2014, 09:52:04 PM
cc @escho and g3gg0 to port the fix in mlv_dump. I did a quick grep and couldn't find the cold pixel fix code.
mlv_dump:2079 find_and_fix_cold_pixels(fix_cold_pixels, block_hdr.frameNumber);
as mlv_dump uses raw2dng.c, the next build should have a1ex' fix.
will build them tonight. done, see the main post.
I don't have tracemodule in the list.
you can take that one:
http://ml.g3gg0.de/modules/trace.mo/trace.mo
installed this, and reinstal ml now it show right info, able to record 10s 23,976 1080p
waiting for more speedy cards.
but there are no files of trace on card.
don't understand why in camera card is much slower 68mb in camera 112mb on pc.
Hi. I'm getting magenta tint on my .mlv's. Something I should know about?
Quote from: far.in.out on March 29, 2014, 07:51:15 PM
Hi. I'm getting magenta tint on my .mlv's. Something I should know about?
well, you could explain a bit more.
one frame? all frames? all videos? just some videos?
camera? module version? settings? sample image?
etc
Omg. Sorry.
50D. Latest Nightly. All videos all frames. It's not like it's corrupted, just a tint. I've seen somewhere it has been reported before. Can't find now...
BTW, legacy RAW is OK...
(http://i60.fastpic.ru/thumb/2014/0329/29/1f51b5ce650b77debb9ba9938f07eb29.jpeg) (http://i60.fastpic.ru/big/2014/0329/29/1f51b5ce650b77debb9ba9938f07eb29.png)
mlv_rec has this nice recording symbol, followed by the elapsed recording time. But if frames are skipped, which happens to me from time to time, the elapsed time isn´t shown anymore. I can only see, how many frames are skipped acually. But I would like to see both, the elapsed time and the skipped frames.
I played a bit with the code and solved this problem for me like this in mlv_rec.c
if(!DISPLAY_REC_INFO_NONE && liveview_display_idle() && should_run_polling_action(DEBUG_REDRAW_INTERVAL, &auxrec))
{
if(DISPLAY_REC_INFO_ICON)
{
int32_t fps = fps_get_current_x1000();
int32_t t = ((frame_count + frame_skips) * 1000 + fps/2) / fps;
int32_t predicted = predict_frames(measured_write_speed * 1024 / 100 * 1024);
/* print the Recording Icon */
int rl_color;
if(predicted < 10000)
{
int time_left = (predicted-frame_count) * 1000 / fps;
if (time_left < 10) {
rl_color = COLOR_DARK_RED;
} else {
rl_color = COLOR_YELLOW;
}
}
else
{
rl_color = COLOR_GREEN1;
}
int rl_icon_width=0;
/* Draw the movie camera icon */
rl_icon_width = bfnt_draw_char(ICON_ML_MOVIE, MLV_ICON_X, MLV_ICON_Y, rl_color, COLOR_BG_DARK);
/* Display the Status */
bmp_printf(FONT(FONT_MED, COLOR_WHITE, COLOR_BG_DARK), MLV_ICON_X+rl_icon_width+5, MLV_ICON_Y+5, "%02d:%02d", t/60, t%60);
if(frame_skips)
{
bmp_printf(FONT(FONT_MED, COLOR_WHITE, COLOR_BG_DARK), MLV_ICON_X+rl_icon_width+5, MLV_ICON_Y+30, "%d skipped", frame_skips);
}
}
My changes:
int32_t t = ((frame_count + frame_skips) * 1000 + fps/2) / fps;
/* Display the Status */
bmp_printf(FONT(FONT_MED, COLOR_WHITE, COLOR_BG_DARK), MLV_ICON_X+rl_icon_width+5, MLV_ICON_Y+5, "%02d:%02d", t/60, t%60);
if(frame_skips)
{
bmp_printf(FONT(FONT_MED, COLOR_WHITE, COLOR_BG_DARK), MLV_ICON_X+rl_icon_width+5, MLV_ICON_Y+30, "%d skipped", frame_skips);
}
I don´t know, whether my change is a proper solution for this problem or whether this must be done in another way. Can this feature, showing skipped frames and elapsed time, be integrated in mlv_rec, please?
Edgar
Quote from: far.in.out on March 29, 2014, 08:04:11 PM
Omg. Sorry.
50D. Latest Nightly. All videos all frames. It's not like it's corrupted, just a tint. I've seen somewhere it has been reported before. Can't find now...
BTW, legacy RAW is OK...
(http://i60.fastpic.ru/thumb/2014/0329/29/1f51b5ce650b77debb9ba9938f07eb29.jpeg) (http://i60.fastpic.ru/big/2014/0329/29/1f51b5ce650b77debb9ba9938f07eb29.png)
ok disable the black fix option in menu and try again.
Quote from: escho on March 29, 2014, 08:23:18 PM
I don´t know, whether my change is a proper solution for this problem or whether this must be done in another way. Can this feature, showing skipped frames and elapsed time, be integrated in mlv_rec, please?
added your code, thanks ;)
That was it. Thank you.
@g3gg0
old question: (old) raw_rec solved splitting files by leaving the file-split done by the OS, thus it happends, the raw-frames are splitted as well. was it changed? it seems, in mlv there are not splitted anymore, did it changed in raw_rec as well?
regards chmee
Hey.
Splitting in raw_rec was not done by the OS. raw_rec handled that on its own.
In mlv_rec this is handled differently. Every chunk contains only complete blocks.
This way you can read every file on its own, you don't need the previous one.
There is no split in the middle of a block/frame.
But you can still merge them by simply concatenating them.
@g3gg0
Does the 5Diii 123 nightly from mar 31 have your latest
changes to the MLV_REC ?
I still see this problem
ML ASSERT:
hdr->blockSize >= (sizeof(mlv_vidf_hdr_t) + hdr->frameSpace + frame_size)
at mlv_rec.c:1984 (process_frame), task Evf
lv:1 mode:3
Magic Lantern version : Nightly.2014Mar31.5D3123
Mercurial changeset : 9765bab7157b (5D3-123) tip
Built on 2014-03-30 22:53:25 UTC by
[email protected].
Free Memory : 152K + 3747K
Quote from: gideonplus on March 31, 2014, 01:27:51 PM
@g3gg0
Does the 5Diii 123 nightly from mar 31 have your latest
changes to the MLV_REC ?
I still see this problem
ML ASSERT:
hdr->blockSize >= (sizeof(mlv_vidf_hdr_t) + hdr->frameSpace + frame_size)
at mlv_rec.c:1984 (process_frame), task Evf
lv:1 mode:3
Magic Lantern version : Nightly.2014Mar31.5D3123
Mercurial changeset : 9765bab7157b (5D3-123) tip
Built on 2014-03-30 22:53:25 UTC by [email protected].
Free Memory : 152K + 3747K
me to
Its still happening? Ok then ill check.
i just hoped to throw away my file-reading-module for raw_rec. without merging files its a little bit annoying with (part@FileA) + (part@FileB) -> deltaOffset@FileB etc pp.. ok. thanks a lot, i'll rewrite the code..
regards chmee
Hi g3gg0. All my issues with mlv_rec are sorted except when the card fills up the screen of my 5DM3 throws up and error message and freezes.
Uploaded whatever the camera produced here:
Edit: The youtube upload has an error. Will try to upload again.
Quote from: g3gg0 on March 24, 2014, 09:30:46 PM
i added a workaround for that black level issue. (next nightly)
i am not sure why the raw backend has trouble in detecting the correct black level so often now.
did you all use zoomed modes?
a) mlv_rec will by default fix that value to 2048. depending on your camera this may be wrong.
iirc alex told me that there may be some models which have 4096. (maybe i mixed smth up)
for this reason there is a menu option to disable that black-fix if you need it.
b) mlv_dump has an option --black-fix to override black level to 2048. for existing footage just run
'./mlv_dump in.mlv --black-fix -o out.mlv' and your footage in out.mlv will have black level set to 2048.
(for windows users: here (http://upload.g3gg0.de/pub_files/2db1eeaea665e6a8e9bb8550d3ba958f/mlv_dump.exe) is a version with the fix)
the changes will take effect with the next nightly.
I'm sorry g3gg0, how can I prevent this dark-green-clip issue happening when using RAW_REC module? I've tried a few latest builds they all had this issue (with ~70% chance in cropped mode)
I noticed a "Black Level" option in the "image finetuning" menu, it can be set to -100..100, is this a cure? If so which value should I use?
Thanks.
you could try to patch the raw_rec footer or implant the hack into raw2dng.c.
there is no ready-to-run solution for that.
Is there any way to fix the underexposed green DNG's on mac?
Been using RAW on the 5DmkIII with 1.2.3. and the 26 march build.
thats the way to fix the .dng files you got using raw2dng.
Quote from: a1ex on October 27, 2013, 09:36:07 AM
Change the black level.
exiftool -BlackLevel=2048 *.DNG
(or 1024 or just adjust it until you get rid of the color cast)
i currently have 1.1.3 on my 5D3 and cannot reproduce this fault.
did it happen to you with the old firmware version?
Quote from: gideonplus on March 31, 2014, 01:27:51 PM
ML ASSERT:
hdr->blockSize >= (sizeof(mlv_vidf_hdr_t) + hdr->frameSpace + frame_size)
at mlv_rec.c:1984 (process_frame), task Evf
lv:1 mode:3
Magic Lantern version : Nightly.2014Mar31.5D3123
Mercurial changeset : 9765bab7157b (5D3-123) tip
Built on 2014-03-30 22:53:25 UTC by [email protected].
Free Memory : 152K + 3747K
@g3gg0
Summery of my latest tests using MLV 5d3 1.2.3:
1. The MLV using setup as follows:
1920x1080x25p (fps on/off)
GD =Allow
File>4GB =on
Extra Hack =on
Memory hack =on
Fix black level =on
Any other settings = default
with the above setup the recording is stable as long as the clip doesnt the
max capacity of the CF card (in that case it will corrupt the file).
During recording of a debug msg shows some times on the screen:
Faild queued 139 /*or other number*/ faild 1 (Reqeued)
2. You can reproduce the fault when using:
File>4GB = OFF
and by making long files (more then 2 minutes)
3. The above will also happen in case of card is formated as FAT32
(64GB in camera)
magiclantern-Nightly.2014Mar31.5D3123
I have this problem, too. I tried everything from here and can't solve it!
I ask somebody to get my example image and make some kind of video tutorial, so some people like me, who are not programmers and do not know about binnar, hexa etc etc
can do something about this problem....
I tried Exiftool GUI, and when I try to change any value, it says:
"Error reading image data -M291707S.dng"
I tried several different files and always the same.
Here is my DNG frame:
http://www.sendspace.com/file/mic5oy
Please help!
I am getting the same message Failed.queued:13 Failed:1(dequeued) The numbers seem to change each time I get it. I am using Lexas 256GB 1066X cards. Anyone can offer any help. I need to shoot in 5 days and would hate to shoot regular format after coming this far... Seems when I go to open the files in MLRawViewer, all I get is a black box, no video even though there is 40BG used on the card... I appreciate anyones thoughts... Sincerely, Joe Ray
Quote from: joeray on April 06, 2014, 06:12:56 AMSeems when I go to open the files in MLRawViewer, all I get is a black box, no video even though there is 40BG used on the card...
can you try with MLVViewSharp? if that doesnt work, please save the console output of 'mlv_dump -v'
Quote from: gideonplus on April 03, 2014, 08:28:04 PM
During recording of a debug msg shows some times on the screen:
Faild queued 139 /*or other number*/ faild 1 (Reqeued)
Quote from: djordjevla on April 06, 2014, 04:51:28 AM
I have this problem, too. I tried everything from here and can't solve it!
which camera and which firmware version?
I am using a 5DMK3 with a LEXAR 256GB and 128GB card and nightly from 3-30-2014 at 14:53. I get the same warning and when I get the warning, the video keeps recording but I loose the audio... It seems I have been loosing audio after 30 seconds or so. Hence, when I shoot a 25 min clip of a live concert, the audio is always missing even though it's turned on...
firmware version please :)
5D mark iii 1.2.3
magiclantern-Nightly.2014Mar31.5D3123
can you do me a favor and test with the old firmware version?
At this time I am 10000Km away from my camera.
I get the same error as gideonplus. Tested with Apr-9th 113 and 123 FW on 5Dmk3.
It started at about 8 mins and then before the 128Gb card was full it incresed from failed 1 to failed 5.
Then switched to (randomly chosen) magiclantern-v2.3.NEXT.2014Feb23.5D3113 and I got no error message about failed queueing.
What I observed with Apr-9th build was, that the MLV on my 128Gb card was splitted into smaller files even my CF card was exFAT (as well as the SD card on which I run ML).
With version Feb-23th I got one big file, no splitting, no error messages. Same camera, same setup, same cards.
Quote from: kaco on April 10, 2014, 09:28:01 PM
I get the same error as gideonplus. Tested with Apr-9th 113 and 123 FW on 5Dmk3.
It started at about 8 mins and then before the 128Gb card was full it incresed from failed 1 to failed 5.
Then switched to (randomly chosen) magiclantern-v2.3.NEXT.2014Feb23.5D3113 and I got no error message about failed queueing.
What I observed with Apr-9th build was, that the MLV on my 128Gb card was splitted into smaller files even my CF card was exFAT (as well as the SD card on which I run ML).
With version Feb-23th I got one big file, no splitting, no error messages. Same camera, same setup, same cards.
sorry, no file splitting also in Apr-9th. Those files were just 0 bytes files I always get when record fills the card. So ignore the last part of my comment.
BTW, I tested also Mar-23th and I also get the queue fail message. So this problem is quite "old".
I still can't solve the problem with blacklevel :(.
I tried really everything. Tried with Exiftool, GUI, PhotoME, but the main problem is that it shows BLACKLEVEL=0 and it must be lower than 0 in that case.
Also, when I put my .RAW file in raw2cdng.exe in shows me on that (black) file that it has about 5000 blacklevel (in that case in needs to be 2047).
So, my question is what is the simpliest way to change it?
Here you are one frame so please, tell me is that file ok and can that material can be returned?
It is a big problem for me cause it is my exam film and i have lot of black takes :(.
Link for one DNG frame: http://www.sendspace.com/file/gi1iko
It would be great if you can make any shooort video tutorial so I can apply that to all of my clips...
(http://s14.postimg.org/fcttiqcgx/image.jpg)
change the black level with exiftool "type exiftool -BlackLevel=2048 *.dng
but I not sure what your black level should be on 5d3.
When I tried to change your sample I got errors ,so it didn't change it so you may have to edit the black level with a hex editor,
check the link below explains how to do it.
http://www.magiclantern.fm/forum/index.php?topic=6580.msg52480#msg52480
I would check a healthy dng with exiftool and compare them to see what the different is
type: exiftool(-k -a -u -g1 -w txt).exe that will export a txt doc.
There other on the form who can help farther.
Quote from: reddeercity on April 11, 2014, 05:44:35 AM
change the black level with exiftool "type exiftool -BlackLevel=2048 *.dng
but I not sure what your black level should be on 5d3.
2048 is right for 5D3
I really tried everything and I am not so good to do anything with HEX Editor.
I have also tried to change it in Exiftool, like you wrote, but I got that error, too....
Is there any help for me?? I have some really important takes which I need to be returned :(
Thanks in advance!
Installed today on 5d MkII 212 nightlies: magiclantern-Nightly.2014Apr05.5D2212 or magiclantern-Nightly.2014Apr09.5D2212 (fresh install after older vers uninstalled)
All frames were pink with mlv_rec. (they were all right when played in camera) + most of the times, blue light turns on during recording and stays on forever.
No problem with raw_rec though..
Anyone might know where the problem is?
Thank you.
@djordjevla:
this thread is about mlv_rec and not black level problems of raw_rec.
please create a post about that issue, it just messes up the thread too much.
thanks.
Quote from: LucianParaian on April 11, 2014, 04:44:48 PM
All frames were pink with mlv_rec. (they were all right when played in camera) + most of the times, blue light turns on during recording and stays on forever.
can you show me the output of "mlv_dump -v <file-mlv>", the first few hundred lines?
(pastebin.com)
Updated to the latest version of ML for 5D3 1.1.3. yesterday, getting the same type of errors "Faild queued 139 /*or other number*/ faild 1 (Reqeued)" as mentioned by other users. I'm on SanDisk Extreme Pro 160 MB/S. By any chance, is there an explanation of the error msg available or a solution on the way? Thanks a lot for the effort.
oh sorry, thought i already described that error.
but it seems i typed it (from my phone) and i didnt submit that message.
doh!
ok then let me explain again:
that message is from the metadata thread.
mlv_rec tried to put some metadata into that (usually) free area in front of a frame.
(EDMAC needs 4KiB alignment, so there is up to 4KiB empty space in front of a frame)
but there was not enough free space in the last few frames, so the metadata
has to be requeued and to be transmitted next time.
if you get only one message (failed: 1) or now and then one more, it doesnt matter.
but if the "failed" counts up permanently and the "queued" stays constant, then something is totally wrong.
(either "queued" or "failed" count up)
so: your footage should be fine without any information loss.
if not, then we have to talk :)
Quote from: g3gg0 on April 11, 2014, 11:02:00 PM
oh sorry, thought i already described that error.
but it seems i typed it (from my phone) and i didnt submit that message.
doh!
ok then let me explain again:
that message is from the metadata thread.
mlv_rec tried to put some metadata into that (usually) free area in front of a frame.
(EDMAC needs 4KiB alignment, so there is up to 4KiB empty space in front of a frame)
but there was not enough free space in the last few frames, so the metadata
has to be requeued and to be transmitted next time.
if you get only one message (failed: 1) or now and then one more, it doesnt matter.
but if the "failed" counts up permanently and the "queued" stays constant, then something is totally wrong.
(either "queued" or "failed" count up)
so: your footage should be fine without any information loss.
if not, then we have to talk :)
Thanks. I usually got about 5-7 messages per 128Gb file. Wouldn't it be better to write this information to console, which I am able to turn off?
Yes it shouldnt pop up at all, but its not critical.
anyway, maybe its better to hide it.
Quoteg3gg0
Use mlv_dump -o /path/to/out_frame
This will write the files into specified diretory
I´m not sure:
Does this command simply copy and rename the mlv-file to the path, you specified without doing anything other?
Using mlv_dump to decode to dngs , I love the dngs to go to another path, different from the source-path. I´m doing this manually or with a little bash-script. An option for mlv_dump wouln´t be so bad for me, where I directly can specify the output-path, I write the resulting dngs in.
edit: found the solution. Tho -o option works with --dng too. This makes things much more easy for me :)
Edgar
Hi g3gg0,
I have the 10th April build of the bleeding edge on my 5DM3.
For some strange reason the frames skip massively.
Where could I send you a sample. I will record and reproduce the skips and also note the settings that I get. All I can say is that the mlv version is still a bit unstable for my use. Maybe it is me doing it all wrong.
So, please let me know where I could send in a sample with the settings at the time of the problem and I will do so.
Please ignore this post since I was trying to post the error from the mlv_dump command but failed to attach an image. Admin please delete since this is a unintended post.
Quote from: Canon eos m on April 13, 2014, 04:45:47 PM
Please ignore this post since I was trying to post the error from the mlv_dump command but failed to attach an image. Admin please delete since this is a unintended post.
Edit:
Okay, the command I type is:
C:\>mlv_dump M13-0937.mlv -f [start]-[end] -o M13-0937.mlv
The output that I get is:
Mode of operation:
-Input MLV file: 'M13-0937.mlv'
-Rewrite MLV
-Output into: 'M13-0937.mlv'
File M13-0937.mlv opened
File M13-0937.m00 opened
File M13-0937.m01 not existing
File M13-0937.IDX opened <XREF>
Processing .....
Reached end of chunk 1/2 after 0 blocks
Reached end of chunk 1/2 after 0 blocks
Processed 0 video frames
Done
which values did you insert for [start] and [end] ?
Tried all 0 to 40 for start and end respectively.
this please:
C:\>mlv_dump M13-0937.mlv -f 1-3 -o M13-0937_out.mlv
please never specify input and output the same file!
Quote from: g3gg0 on April 13, 2014, 07:33:21 PM
this please:
C:\>mlv_dump M13-0937.mlv -f 1-3 -o M13-0937_out.mlv
please never specify input and output the same file!
same error as before.
then please post the output of mlv_dump -v M13-0937.mlv on pastebin.
a) please the full log
b) make sure this is your original file as i expect the one you have used previously is overwritten as you specified the same file as input and output
Quote from: g3gg0 on April 15, 2014, 01:59:37 AM
then please post the output of mlv_dump -v M13-0937.mlv on pastebin.
a) please the full log
b) make sure this is your original file as i expect the one you have used previously is overwritten as you specified the same file as input and output
http://pastebin.com/vLCTrXTf
File name: M13-0948
Total Frames: 318
Black Frames: (58, 64,65,66,77,78,87,88,99,100,102,118,119,120,134,135,140,150,151,157,158,163,169,170,182-185,207,208,214,215,230,231,245,246,250,251,268,269,275,276,288,289,298,306,307,311-313
thats only part of the output. please dump everything.
you can e.g. dump the output of mlv_dump into a file using "> log.txt"
Quote from: g3gg0 on April 15, 2014, 12:32:53 PM
thats only part of the output. please dump everything.
you can e.g. dump the output of mlv_dump into a file using "> log.txt"
I copied everything that I could see :(
Is there a tut that I can first attend to learn how to capture the entire log and using log.txt to dump the output because it is really not proper for me to take up your time teaching basic stuff to folks like me.
All I can say now that after being following ML for about 5-6 months regularly, I am beginning to get a hang of this stuff. I know I am almost there. There are only two things that are stopping me from realizing the full potential of this magical firmware (footage wise): (i) the frame skipping and (ii) sound recording.
Once I get this done, I am ready to take on the world.
just add a > log.txt behind the command.
then it will dump the text into a file named log.txt
what do you mean with frame skipping and sound recording?
is there some trouble?
edit:
or just upload the footage somewhere, so i can look myself.
Quote from: g3gg0 on April 15, 2014, 03:32:33 PM
just add a > log.txt behind the command.
then it will dump the text into a file named log.txt
what do you mean with frame skipping and sound recording?
is there some trouble?
edit:
or just upload the footage somewhere, so i can look myself.
I meant the black frames (sorry for using the wrong terminology) and sound recording is a bit odd for me - get it sometimes and don't in other situations. But frankly, was not keeping a close watch. Will now onwards record the settings and failed attempts at recording sound if any.
Still at work. Will reach home in about an hour and will action the log.txt command and generate the dump to send to you.
Will also upload a part of the footage with black frames if I can tonight for you to see and guide.
are you sure you didnt activate card spanning and forgot to copy the video files on the other card?
Quote from: g3gg0 on April 15, 2014, 04:02:01 PM
are you sure you didnt activate card spanning and forgot to copy the video files on the other card?
Card spanning is always ON when I use MLV. Not sure what you mean by forgot to copy the video files on the other card.
Card Spanning means that the files are written to the CF AND SD card to archive higher speeds. When you convert the mlv-files you need the files from the SD card, too.
Quote from: Jockerl on April 15, 2014, 04:54:17 PM
Card Spanning means that the files are written to the CF AND SD card to archive higher speeds. When you convert the mlv-files you need the files from the SD card, too.
OH NO!
I knew what spanning was ... but didn't realise that files from both cards had to be used. Let me try to record a few clips first and use files from both cards before I bother g3gg0. Maybe, that is why I have some files that are clean with sound while others have either no sound or black frames. Not sure but let me see this was the reason.
Quote from: g3gg0 on April 15, 2014, 03:32:33 PM
just add a > log.txt behind the command.
then it will dump the text into a file named log.txt
what do you mean with frame skipping and sound recording?
is there some trouble?
edit:
or just upload the footage somewhere, so i can look myself.
smallest file I could find with the black frame problem: https://www.dropbox.com/s/a6yk1qot0u5kt18/M13-0935.MLV
A few comments on the already exceptional piece of programming mastery:
I have CF 128GB 1066x KB card and SD 128GB 600x Transcend (GD is on - peaking and histogram, only video MLV mode considered)
a) 16384K buffer is generally slower than 15625K with my CF only (up to 88 MB/s, after several minutes of warm up).
b) In burnout mode some of the 25MB+ buffers achieved 90-91 MB/s in writing.
c) Going to 30fps eats a lot of bandwidth, easily 5-15MB/s for CF.
d) The fastest speed with SD I've seen was 18.5 MB/s, 20.1 MB/s reading.
e) Turning spanning on will degrade CF performance further and as a result SD card (a tad slowed too) doesn't help a whole lot.
Is there any way to select buffers for CF and SD for independent or spanning modes?
Sometimes I think having 1:2.40 would be nice, since 1920x800 is nicely divisible by 16 and actually used by some BDs.
I am still not entirely sure how to absolutely 100% guarantee video-to-sound sync. There is usually a difference in length between two tracks. And I strongly dislike this 24 vs. 23.976 thing, though I understand where it comes from.
At this point I'd say the only thing truly missing is ability to get 30fps with spanning, as I guess it is unreasonable to assume that any single card will sustain it along with a stable MLV with sound.
thanks for your feedback.
Quote from: KMikhail on April 16, 2014, 12:21:39 AM
Is there any way to select buffers for CF and SD for independent or spanning modes?
right now -> edit source code and recompile.
its hard to make that UI-editable, but i can think of a config-file only option.
Quote from: KMikhail on April 16, 2014, 12:21:39 AM
Sometimes I think having 1:2.40 would be nice, since 1920x800 is nicely divisible by 16 and actually used by some BDs.
same as above
Quote from: KMikhail on April 16, 2014, 12:21:39 AM
I am still not entirely sure how to absolutely 100% guarantee video-to-sound sync. There is usually a difference in length between two tracks. And I strongly dislike this 24 vs. 23.976 thing, though I understand where it comes from.
yeah, the audio is just longer in the end. its totally in sync, but i can stop audio recording only in 250 ms steps or so.
so you just have to cut the end.
it however *could* change with the new audio api where it is possible to cancel while a transfer is in progress.
but consider that as highly experimental and not even ready for alpha testing.
Quote from: KMikhail on April 16, 2014, 12:21:39 AM
At this point I'd say the only thing truly missing is ability to get 30fps with spanning, as I guess it is unreasonable to assume that any single card will sustain it along with a stable MLV with sound.
yeah SD transfers while capturing video and CF card transfers will slow down everything else.
this is relateed to EDMAC/CDFMA modules which cause load on the SDRAM interface.
Quote from: Canon eos m on April 15, 2014, 05:47:44 PM
smallest file I could find with the black frame problem: https://www.dropbox.com/s/a6yk1qot0u5kt18/M13-0935.MLV
Hi g3gg0, did you get a chance to review the file I uploaded. Would really love to know what I may be doing wrong. Did search the forum for spanning related threads but did not find much to go by.
I am now thinking of experimenting with the raw_rec.mo. Have been avoiding it so far due the sound capabilities of ML. What do you think.
Quote from: g3gg0 on April 15, 2014, 04:02:01 PM
are you sure you didnt activate card spanning and forgot to copy the video files on the other card?
Hi g3gg0, sorry for the trouble and BIG THANK YOU for the help. I could not figure out the mlv_dump. But my problem is resolved since I copied all the mlv related files from the SD card into the same folder where I had copied the mlv files from the CF card. The the black frames (skipped frames) problem has vanished. Obviously, there is some data on the SD card that was missing to complete the video.
I am so feeling good and the quality from my 5DM3 is breathtaking!
Awesome! :D Awesome! ;D Awesome! :)
I just tried the latest nightly build and sometimes during recording (MLV_rec enabled), the console pops up (on a 5D3). Found no way to disable it unless I restart the camera...
this is the assert:
ML ASSERT:
hdr->blockSize > 0
at mlv_rec.c:1969 (mlv_prepend_block), task raw_rec_task
lv:1 mode:3
Magic Lantern version : Nightly.2014Apr16.5D3113
Mercurial changeset : 9058cbc13fa4 (unified) tip
Built on 2014-04-15 23:11:59 UTC by
[email protected].
Free Memory : 151K + 3815K
Quote from: g3gg0 on April 16, 2014, 12:43:31 PM
its hard to make that UI-editable, but i can think of a config-file only option.
Thanks for your answers. I actually looked up for config already, fruitlessly, so it would be very nice to have!
Quote from: g3gg0 on April 16, 2014, 09:14:05 PM
fixed (http://ml.g3gg0.de/modules/mlv_rec.mo/mlv_rec.mo), thanks.
Thank you for all your effort!
I just installed ML on my canon 60D but I can't see the raw video in video mode? What do I need to do to shoot raw?
enable the according modules inside the menu. mlv_rec and/or raw_rec
Quote from: g3gg0 on April 16, 2014, 09:14:05 PM
fixed (http://ml.g3gg0.de/modules/mlv_rec.mo/mlv_rec.mo), thanks.
Good morning. Is this module I should use going forward?
Quote from: Canon eos m on April 17, 2014, 03:35:42 AM
Good morning. Is this module I should use going forward?
It fixes the assert and is already included in the latest nightly build.
While the latest nightly build seems to work fine (no error msgs or console anymore), the mlv files can't be decoded via MLVMystic (just decoded one dng with weird colors), decoding to dng sequence with mlv dump worked, dng seq numbering is wrong (starts at frame_000000 / 01 / 02 then continues with 1180).
Hi,
Thanks for all the hard and great work!
It it delicious ;D
Latest built reports invalid header when trying to playback MLV file in camera.
I can only see the first frame after it renders through, and it is a noisy image, sometimes pink noise, other times black & white colors!
I did not try to import that file on my Mac to convert it, so it might be a in camera problem, or this could be as @MGerard wrote:
Quote from: MGerard on April 17, 2014, 01:51:49 PM
While the latest nightly build seems to work fine (no error msgs or console anymore), the mlv files can't be decoded via MLVMystic (just decoded one dng with weird colors), decoding to dng sequence with mlv dump worked, dng seq numbering is wrong (starts at frame_000000 / 01 / 02 then continues with 1180).
So I always get that invalid header message when trying to preview in camera, so I went back to a later build that works for now :)
5D Mark III
Build 5D.113
Built on: 2014-04-18 07:11:43 +0800
Regards,
Daniel
Quote from: g3gg0 on April 16, 2014, 09:14:05 PM
fixed (http://ml.g3gg0.de/modules/mlv_rec.mo/mlv_rec.mo), thanks.
Does the fix for "hdr>blockSize>0" failed assertion apply to 5D3-123 as well ? I see similar problem there after downloading the 4/16 nightly build.
thanks.
another fix (http://ml.g3gg0.de/modules/mlv_rec.mo/mlv_rec.mo) that solves the error reported by 5DanielMIII and MGerard.
thanks.
this bug depends on some block sizes and thus on some settings.
please dont use the older builds anymore.
@SergePlotkin:
please more details about the error you get and the settings/situations that lead to this error.
g3gg0,
Posted all the .err files as part of #1954 on bitbucket.
Summary: Using latest 5D3-123 build.
Old style raw seems to work fine.
If instead I enable the new mlv_rec&mlv_snd, then shortly after starting to record
I get the above assertion violation and lots of debug-related info on the screen, as if in "console mode".
At this point I have to turn off the camera.
Interestingly enough sometimes this does not happen at all - have a couple of 3-4 minutes HD recordings with no problems.
Also, somehow pressing the AF-ON button in the middle of the recording more often then not immediately results in this error.
Using Sandisk UDMA7 90MB/s 64GB flash. Recording at about 70MB/s.
Probably unrelated, but pressing "zoom" during recording completely hangs the camera and requires removal of battery.
Please tell me what other specific information will be helpful, since it is very easy for me to reproduce the above issue/issues.
thanks.
thanks, seems to be to related with mlv_snd. will check.
please try latest nightly (or from there (http://ml.g3gg0.de/modules/mlv_rec.mo/mlv_rec.mo))
g3gg0, is it safe now to start using .123 for the MLV raw?
Quote from: g3gg0 on April 19, 2014, 02:00:43 AM
please try latest nightly (or from there (http://ml.g3gg0.de/modules/mlv_rec.mo/mlv_rec.mo))
Tested the version from jenkins, also got error as SergePlotkin. Tested your build of mlv_rec, the console error has gone. But now I am missing the option to allocate space with LV off (some kind of extra hack). And on record start/stop i got message:
Hacked error at 0:
expected e3a00032, got e1a00000
disable memory hacks and extra hacks, then try again.
g3gg0,
Disabling "extra hacks" eliminates the "hacked error at 0" message.
Are these hacks important for something ?
In particular, I am worried about quickly rising temperature - reaches 50C less than 10m into shooting video.
Thanks for creating the fix so quickly !
with those hacks you can gain a few buffers more.
so you will get a few seconds extra recording time if your card isnt fast enough.
temperature is nearly the same as in video mode without recording raw.
Quote from: MGerard on April 16, 2014, 05:36:34 PM
I just tried the latest nightly build and sometimes during recording (MLV_rec enabled), the console pops up (on a 5D3). Found no way to disable it unless I restart the camera...
this is the assert:
ML ASSERT:
hdr->blockSize > 0
at mlv_rec.c:1969 (mlv_prepend_block), task raw_rec_task
lv:1 mode:3
Magic Lantern version : Nightly.2014Apr16.5D3113
Mercurial changeset : 9058cbc13fa4 (unified) tip
Built on 2014-04-15 23:11:59 UTC by [email protected].
Free Memory : 151K + 3815K
I'm using the latest nightly build and still getting this exact error. The console pops up after about 1 minute of recording and still records. I've tried recording with RAW only and not MLV (Video/Audio) and it's amazing. No errors so far on that.
Is there a setting we're supposed to change or anything we can do?
edit - I'll test the mlv_rec.mo linked. Thank you!
edit again - I replaced mlv_rec.mo and everything works. I still get the hacked error at 0 message with extra hacks on, but that's it. Thanks!
Quote from: arch on April 20, 2014, 05:38:27 AM
I'm using the latest nightly build and still getting this exact error.
latest = 20.04.
your build = 16.04.
sure that the nightly isn't more recent?
@g3gg0 - Which part of the code applies BT709/BT601 to the DNGs. I remember you did something that does the matrix transformation on the fly but can't remember where it is. I want to try something with wide gamut RGB.
Quote from: g3gg0 on April 20, 2014, 01:21:48 PM
latest = 20.04.
your build = 16.04.
sure that the nightly isn't more recent?
Sorry, I'm new here and still learning about all this. I'm using the 1.2.3 latest nightly build. Is there a different developer on that build?
Magic Lantern Version: Nightly.2014apr16.5D3123
Built on 2014-04-16 16:33:51 UTC by
[email protected]I did use the mlv_rec.mo you linked, and it works awesome except has the hacked error with 0 message (it disappears within a second or so) when hitting record and unrecord.
Thanks again and sorry for the noob questions.
okay it seems v1.2.3 was not built a few days.
then its clear why yours is older.
Quote from: Andy600 on April 20, 2014, 01:44:26 PM
@g3gg0 - Which part of the code applies BT709/BT601 to the DNGs. I remember you did something that does the matrix transformation on the fly but can't remember where it is. I want to try something with wide gamut RGB.
https://bitbucket.org/hudson/magic-lantern/src/fd63cf6052145074ee9891faea0f1a2d1d0c08bb/contrib/g3gg0-tools/MLVViewSharp/DebayerBase.cs?at=unified
there i am using the matrices in my C# rendering example.
DNG themselves dont have any RGB colorspace information, they contain RAW RGGB bayer pattern.
Hi @g3gg0
Do not know if this should be reported here as well.
I posted it to the MLV mystic thread..
But I can not convert any of my MLV files from this build;
Date: 2014-04-20 07:11:47 +0800
Program MLV Mystic says "Conversion Complete" but all the new DNG-folders are empty.
MIrawviewer too makes empty folders when trying to export straight from the player.
I am on a 5D Mark III .113
Regards,
Daniel
hi.. I'm testing a fresh build of ML with ML_REC on 5D3 1.2.3.
Setup: 1920:1080 23,976 fps, Transcend 1000x 64GB..
After restart, first movie records without issues. Next stops after 2-3 seconds. So I allowed skipped frames and it shows skipped frames (2-4) always after 2 seconds and then goes without problems. I tested also a RAW_REC and I've got no skipped frames.. Is it a problem with my CF card ?
UPDATE: I copied warm up code just after start_delay option and it looks like it works without skipped frames now. Maybe my card needs warm up just before each file ? hmm
msleep(start_delay * 1000);
if(warm_up)
{
NotifyBox(100000, "Card warming up...");
char warmup_filename[100];
snprintf(warmup_filename, sizeof(warmup_filename), "%s/warmup.raw", get_dcim_dir());
FILE* f = FIO_CreateFile(warmup_filename);
if(f != INVALID_PTR)
{
FIO_WriteFile(f, (void*)0x40000000, 8*1024*1024 * (1 << warm_up));
FIO_CloseFile(f);
FIO_RemoveFile(warmup_filename);
}
NotifyBoxHide();
}
Hi all,
I'm testing the latest 4/22 - Built on: 2014-04-22 12:35:00 -0700 for 1.2.3
It still has the console error, but I replaced the mlv_rec.mo with the one that g3gg0 linked. And the console error is gone (still has hacked error at 0 when recording / stopping with extra hacks turned on). It seems the mlv_rec.mo may not have been replaced with this build yet.
Thanks!
Testing April 24 nightly on 5D2.
When recording in MLV raw,
GD ON,
the framing/border disappears and will only reappear when recording stops.
Apr 23 did not have this behavior.
edit:
I tested clearing the cf and placing new ML folders and its OK now. Erasing the old magic.cfg must have done the trick.
the last 2 or so nightly builds for 5d3 have been train wrecks :( but the one before those was by far the most stable I had used yet. I'm not sure what was changed but on my end I don't think it was for the best. lol
can you please clearly say what doesnt work?
I think I figured it out, dual ISO was switched on. ::)
@g3gg0,
Is it possible to have the Black fix OFF on the 5D2 by default? Its bit me once, good thing it was a test and the command line setting for mlv dump to either 2048 1024 did not fix it and instead produced greenish shadows.
Quote from: ted ramasola on April 29, 2014, 01:34:57 AM
Is it possible to have the Black fix OFF on the 5D2 by default?
a) yes, good point.
b) set black level to 1700: http://www.magiclantern.fm/forum/index.php?topic=11551.msg112388#msg112388 / https://bitbucket.org/hudson/magic-lantern/commits/02c9af698af933dfea31212713d5216cb0626ff8
5D2 uses both 1024 and 1700.
The black level workaround should not be present in the first place in my opinion (I said it back then and I'm saying it again). It should be fixed in the backend.
1750 was a closer fix than the 1700. Resolve does a faster fix than ACR, adding an ARRI LUT also helps.
Its very tricky with it there with default ON, it could be costly for those used to older builds without it and just decided to update, and since it says "black FIX" they might think its important to leave on.
Quote from: a1ex on April 29, 2014, 09:34:59 AM
The black level workaround should not be present in the first place in my opinion (I said it back then and I'm saying it again). It should be fixed in the backend.
and again i totally agree and want to add that it is hard to "just fix" it in the backend if one didnt write the code.
well, i can add some waits here and there and just submit it into main branch to test if that did the job.
yet i could not reproduce it on 1.1.3 with >50 videos i recorded automatically
--black-fix doesn't appear to be working in the latest mlv_dump, with this sample (http://www.magiclantern.fm/forum/index.php?topic=11659.0).
I also tried outputting directly to DNG with --black-fix, and it doesn't appear to be working.
edit: exiftool reports the black level on a processed dng, has a value of 2048, after setting 1800.
mlv_dump --black-fix=1800 --dng -o e:\test\test.dng m28-1702.mlv
MLV Dumper v1.0
-----------------
Mode of operation:
- Input MLV file: 'm28-1702.mlv'
- Setting black level to 1800
- Convert to DNG frames
- Output into 'e:\test\test.dng'
File m28-1702.mlv opened
File m28-1702.m00 not existing.
Processing...
Vertical stripes correction:
1.000 1.000 0.987 0.983 1.013 0.999 1.005 1.001
Reached end of chunk 1/1 after 459 blocks
Processed 221 video frames
Done
Quote from: g3gg0 on April 29, 2014, 03:19:57 PM
..it is hard to "just fix" it in the backend if one didnt write the code..
could someone explain why this workaround/fix appeared? it seems it's makin a lot of users nervous..
Quote from: chmee on April 29, 2014, 07:17:33 PM
could someone explain why this workaround/fix appeared? it seems it's makin a lot of users nervous..
mlv_rec/raw_rec make use of the camera-specific raw backend in ML core.
both modules use the same way of reading raw data using this backend.
the raw backend determines raw data dimensions and black level etc.
now on 1.2.3 (and probably on 1.1.3 too, but not as often as it seems) the digic internal EDMAC
gets disturbed somehow by redirecting and the optical black area used for calibration contains crap.
this crap unfortunately leads to a wrong black level estimation that only affects mlv/raw metadata, but not image content.
this black level correction is *obviously* a workaround for a problem that is hard to pin down.
but it is sufficient to repair corrupted footage - or - to apply that fix already during recording.
my hope was that people test their current module config and workflow before actually going out on a set.
For an workaround, I'd say the option in mlv_dump should be enough.
With the current approach, in-camera workaround being on by default, the forced value is not correct for all models (and there are cameras that use more than one value - 5D2 for sure, maybe 50D too, since it's from the same generation). This is what Ted was trying to say.
Also, with the workaround enabled by default, the real issue gets hidden and it's easily overlooked.
Rather then fix this in camera, if that is very difficult or hurts performance, couldn't you use the black level detection from cr2hdr?
The files must be post processed, so it's fine to fix the issue in post IMO.
Because the blacklevel is static per body (not per recording) you could also give an Option to set it manually?
Quote from: Audionut on April 30, 2014, 12:11:07 PM
Rather then fix this in camera, if that is very difficult or hurts performance, couldn't you use the black level detection from cr2hdr?
The files must be post processed, so it's fine to fix the issue in post IMO.
When the raw stream is pink (corrupted), the OB area is gibberish. And, for speed reasons, OB is not recorded in raw videos, so we can't look at it in post.
Quote from: chmee on April 30, 2014, 12:40:29 PM
Because the blacklevel is static per body (not per recording)...
It's not.
Is there any way to log an error that flashes across the screen when I view an MLV recording for the the first time in-camera (when it's building the .idx file)? It says something about a corrupted header I believe. The error repeats even if I delete the .idx file and view the mlv again as it recreates the .idx file.
It happens on a few different builds, Feb 16 and Apr 29 for sure on 5DM2.
Quote from: jimmyD30 on April 30, 2014, 01:21:44 PM
Is there any way to log an error that flashes across the screen when I view an MLV recording for the the first time in-camera (when it's building the .idx file)? It says something about a corrupted header I believe. The error repeats even if I delete the .idx file and view the mlv again as it recreates the .idx file.
It happens on a few different builds, Feb 16 and Apr 29 for sure on 5DM2.
you talk about mlv_play?
tis header error is due to a buggy mlv_rec version you used to record the video.
can you try a recent build of mlv_rec, record a video and play in mlv_play ?
you can fix the footage using mlv_dump on your computer if necessary
@g3gg0
Yes, mlv_play. So far I've used 4 builds Feb16, Apr 24, Apr 27, Apr 29 and it happens with all of them. Also, if the .idx is present (in the same folder as the .mlv file) when using MLV Converter 1.9.2 for WINDOWS conversion stops where the error occurs, if the .idx file is not present, conversion completes, but some frames are not converted (missing), about 5 for a 60 sec video.
The error seems to consistently occur at around 45 secs (give or take) when processing 60 sec videos (I've just been using 1 min long videos for my tests, so I don't know if the error would repeat for longer videos). It's never occurred with any of my 30 sec video shots and always with my 60 sec long shots.
I don't know yet if the error is dependent upon the ML settings yet, but I'll keep playing and see what's what and report back any new findings.
I would like to help get MLV working well, so if there is anything specific you want me to try toward that end, just let me know :-)
ah i remember smth.
do you use exFAT and MLV is recording files > 4GiB?
Hey, I'm using a 5DM2, so fat32 and files capped at 4GB with auto rollover. Funny thing, even using raw_rec the error occurs at the same frame (using in camera viewer) as mlv_rec and seems to be build dependent, like the Apr 27 build always errored at frame 848 and a different build always errors at frame 835 for both raw_rec and mlv_rec.
I'm not too worried that the in-camera viewer stops playing at the frame error, but wondering if the lost frames after mlv to dng conversion is going to be a problem?
that problem will be fixed in next nightly.
was a 4GiB playback issue of mlv_play.
jimmyD30,
While updating to new nightly builds I noticed what I thought was a bug, the way I "usually" update was just to download the nightly and copy paste only the autoexec.bin and the ml folder over to my cf card containing the previous builds, however I realized that some previous settings caused what I thought was a "bug". I deleted all the files inside the settings folder in ML after copy pasting the new nightlies and the "bug" was gone.
Maybe you should also try deleting the files in your settings folder.
Try a different Convertor like Raw2CDNGv1.4.9,v1.5.0 or sharp viewer , I use the Feb16 NB for 5d2 and no problems at all with mlv or raw in these convertors. Really no problem at all with that build
"solid" :)
yes exFAT do faster than fat32 format.
the reason I use fat32 is 5dmark3 can not format card into exFAT.
he speaks about problems with mlv_play ;)
@g3gg0
Thank you! I will download the May 1st or later nightlie as the Apr 30 has the same error.
@ted
When I update different nightlies I always replace the entire ML folder (and all of its contents) and the autoexec.bin. The one thing I was concerned about though, recently they are including the .fir file in the nightlies and I was wondering if an older .fir file (which I noticed was 64.4kb from the stable release) is compatible with the newer nightlies wich comes with its own .fir file (37.8kb size)?
EDITED: Ok, Ted, I think I see what you mean now, DELETE all the existing build files then copy and paste the new ones, in case the settings file remains because the new nightlie doesn't contain one and so the old one might remain if just copy and pasting w/o deleting. Although I normally do what you suggest out of habit, I'll make sure that is the case going forward, good advice :-)
@reddeercity
Thank you for your comments, I asked you a similar question on a different thread, so feel free to answer over there (for the benefit of others as well) or disregard :-)
Ok, tried May 2 build on 5dm2 and same issue, in-camera playback stops at frame 833 on 60 sec video at 24 fps. And info says frame count should be 1447, but after processing with MLV Converter 1.9.2 only 1441 frames and audio file is produced.
Does anyone know if this is to be expected or is something wrong with my setup?
no then its fine. if you hadnt any frame skips, those 6 frames could be a display / calculation error.
Ok, thank you. In the settings I don't allow for skipped frames during recording and so far the video looks good/complete after post-processing, so like you say, maybe a software/firmware calculation error. But I can tell you the in-camera viewer and the computer converting app both report the same higher frame count, yet the output results in a few frames less than what is reported.
So, I'll keep an eye on things and if the resulting video looks fine, I'll leave it at that :-)
The frame count is written in the file header. Seems that value is a bit off.
Got reports earlier, but that only a cosmetic problem, not causing any trouble so far.
Quote from: stpaloma on May 01, 2014, 05:09:22 PM
yes exFAT do faster than fat32 format.
the reason I use fat32 is 5dmark3 can not format card into exFAT.
If you got big cards formatted in exfat and format in camera the camera format the cards in exfat, atleast Im pretty sure that is the case...
Latest build I turned on mlv_rec, and mlv_snd and its still recording a .mov file? What did I forget to turn on?
thanks
I found it,. neevermind. This is amazing for this camera,.. and I just want to say thanks guys.. so awesome!
I was shooting with the recent build from April 30 and MLV kept having an error asking me if the card was really EXFAT formatted after I had shot on a card for a while, copy files off and came back, with reformatting the card to exfat and some cards with reformatting. I had to switch back to the original RAW module to finish my shoot and had no more exfat errors.
Quote from: amsh89es335 on May 04, 2014, 07:03:22 PM
I was shooting with the recent build from April 30 and MLV kept having an error asking me if the card was really EXFAT formatted after I had shot on a card for a while, copy files off and came back, with reformatting the card to exfat and some cards with reformatting. I had to switch back to the original RAW module to finish my shoot and had no more exfat errors.
menu item:
{
.name = "Files > 4GiB (exFAT)",
.help = "Don't split files on 4GiB margins,
not supported on all models.",
.help2 = "
Ensure you formatted your card as exFAT!"
},
should have been clearly visible on the camera?
I had all those selected and double checked the formatting as exfat.
which model?
5D Mark III with 4 different kumputer bay 64gb 1000x cards and shot dozens of hours on them with the original RAW formatted exfat on my PC, but first time using them with MLV.
can you open explorer, right click your card, select properties and send me a screenshot?
(http://upload.g3gg0.de/pub_files/64b13ec9c31bb41e280ed65dd5b94a26/prop.png)
Some progress in diagnosing the black level issues:
https://bitbucket.org/hudson/magic-lantern/pull-request/484/black-level-fix/diff
Sorry if this was answered somewhere here in ML Forum but I couldn't find.
I have a question: Is that possible to put my MKIII to record as fast as I touch the record button? I ask this because several months ago this was the norm with RAW_REC and with MLV I can't find how one can skip the "freeze time" before the actual recording.
I'm aware of this behaviour since I began to use MLV early in this year but in my last work (wedding) it brought me some small problems with a few takes.
Quote from: Sganzerla on May 07, 2014, 01:48:57 AM
Sorry if this was answered somewhere here in ML Forum but I couldn't find.
I have a question: Is that possible to put my MKIII to record as fast as I touch the record button? I ask this because several months ago this was the norm with RAW_REC and with MLV I can't find how one can skip the "freeze time" before the actual recording.
I'm aware of this behaviour since I began to use MLV early in this year but in my last work (wedding) it brought me some small problems with a few takes.
For instant recording you will have to stick to raw rec. Mlv's more complex filestructure with metadata and all that gives you that delay.
slowdowns:
- memory hack
- pre-creation of files to save time when 4GiB border is reached
first can be disabled via menu.
second one, well, i can skip this one if you have chosen exFAT (>4GiB) mode. let me check.
Hi,
I'm using magiclantern-Nightly.2014Apr20.5D3113 and the old RAW mode for video (it always worked OK), but today it fucked up most of my clips, that were all green...
(http://shrani.si/t/3l/9C/4jlNrRzC/m10-1605-000000.jpg) (http://shrani.si/?3l/9C/4jlNrRzC/m10-1605-000000.jpg)
how can I fix this clips :(
raw dng
https://gofile.me/2doWQ/5FSenQkR
I even tested the newest May 8th nightly release and I still get green .dng's. Then I turned off the memory hack (I have it for slower komputerbay) and no more green .dng's. I guess the memory hack is fucked up :\
I still need to save my files done till now... how can I get fixed .dngs out of RAW files :( I tried with preview mlv_view_sharp.exe, but are green too :(
Quote from: DjJuvan on May 10, 2014, 08:11:47 PMall green...
http://www.magiclantern.fm/forum/index.php?topic=11664.0
Quote from: g3gg0 on May 08, 2014, 01:44:53 PM
slowdowns:
- memory hack
- pre-creation of files to save time when 4GiB border is reached
first can be disabled via menu.
second one, well, i can skip this one if you have chosen exFAT (>4GiB) mode. let me check.
Anything that can be done to minimize the delay would be extremly usfull. Thanks!
sorry if this is a silly question, but can MLV shoot over 1080?
Quote from: gravitatemediagroup on May 13, 2014, 04:12:52 AM
sorry if this is a silly question, but can MLV shoot over 1080?
"1250" 3:2 A.R. , at least on the 5d2.
Audio timeline is longer in Afterfx than the video, but it still seems to be synched. 1920x1080
And I just wanted to say,. "amazing!" work, by all of you for this.
Is the mlv_rec module in the Nightly Build of Magic Lantern 5D3.113 different from the one in Magic Lantern 5D3.113 or are they interchangeable ? Thanks for the answer.
i guess you mean 1.2.3 / 1.1.3.
they are interchangeable as long nothing was changed to the API.
in this case 1.2.3 is a different branch and may miss some updates to the modules therefore.
but in general: modules are never built for a single camera, they are always compatible across all cameras.
just try, if it throws an error on loading, then the API has changed.
if it loads successfully, everything should be fine.
How can I extract audio files from mlv file. I could successfully extract video .dng files with mlv_dump.
if you had mlv_snd enabled, you will get a wav too
Yes I have mlv_snd activated.
Use this
http://www.magiclantern.fm/forum/index.php?topic=10198.0
So 1.2.3 firmware works now? Nice
hi, am using 5d3 113, April 30th build, MLV. The audio files are at least 8 milliseconds leading... ex: if the video is 0:53:04, audio is 0:53:13... For every file, this is the case. In resolve, i right click in master and select "autosync based on timecode" it syncs but some footage incorrectly. I will have to do it manually. Since am using for a serious video purpose, i have lots of footage and manual syncing and trimming the extra audio is hectic. Also, this causes flickering issue in resolve after color correction. How can i rectify the audio-video time code non-alignment? thanks...
Hi
I am using 5D3 123 and still can't get out sound files. My nightly build is may 07. I have used mlv_dump as following "mlv_dump --dng M13-1313-MLV". What can be wrong?
if you had mlv_snd enabled, you will get a wav too.
Yes I have both mlv_rec and mlv_snd enabled. Status OK. I have used command: "mlv_dump --dng M13-1313-MLV". Is it OK for geting sound files too? Does the sound files will have .wav extension?
hi g3gg0
can you please get smpte to work on 5d iii on 1.2.3
thank you
what does the back button af address? sorry if this is a silly question
Hi I have now tried to use another program for converting mlv files to dng (MLVMystic) and not this software also could produce sound files. I suppose that my MLV files does not includes sound information. Do I need enable more that mlv_snd ?
Quote from: jarandov on May 16, 2014, 03:43:07 PM
Hi I have now tried to use another program for converting mlv files to dng (MLVMystic) and not this software also could produce sound files. I suppose that my MLV files does not includes sound information. Do I need enable more that mlv_snd ?
Don't expect MLV Mystic paste your audio over your video footage. Paste your audio in Davinci Resolve.
There are a software for 30U$ - I think Raw Magic - that process both files. I preferred use Da Vinci and donate the money here, that could be better software for our camera.
Hi budafilms,
I have raw magic lite but that program dos not work with mlv files. Probably only raw.
Does exist any other raw magic program?
Hi budafilms,
I want anyway go to DaVinci in next step. But I need to have both dng files and sound files from mlv first. Do i have understod conversion from mlv wrong?
Quote from: jarandov on May 17, 2014, 09:12:12 PM
Hi budafilms,
I want anyway go to DaVinci in next step. But I need to have both dng files and sound files from mlv first. Do i have understod conversion from mlv wrong?
Hi! Here are a lot of softwares. The classic one is RAW2DNG, for mac and PC.
I love Son of the Batch.
Go to Forum, RAW, Post Processing!
And try DaVinci, in a couple of weeks they going to implemente more edition tools. So, it's not going to be only a color processing. And it's free the lite version, enough for us ;)
using the May 14 and previous nightlies for the 5D2 I sometimes get an "overloaded buzz like static" in my wav files. They are quite random and I can't pin down a pattern.
At first, I thought they are only in the first recorded file after a card was formatted but recently they just pop out at random.
I also first thought that the "buzzing" static will appear only once per clip and only near the beginning but a recent shoot shows it will appear anywhere and sometimes it will happen several times per clip.
The video or the dngs have no problems.
Here's a sample waveform of a clip around 34sec long to show where it happens. This was using only the internal mic of the 5d2.
(https://scontent-a-dfw.xx.fbcdn.net/hphotos-frc3/t31.0-8/10387112_481004792030310_8985835315909797396_o.jpg)
Out of 42 clips, 17 clips have this buzzing static.
I don't get this issue in h264 recording, only from wavs using mlv.
oh wow, this happens quite often.
let me check that.
@ted ramasola
please use the official converters as well if errors like this occur. just to exclude its a bug in my tool.
Quote from: chmee on May 20, 2014, 10:05:48 AM
@ted ramasola
please use the official converters as well if errors like this occur. just to exclude its a bug in my tool.
yes, it is not your raw2cdng tool as the location for the buzz/corruption on the wav file is the same even if I extract using mlvbrowsesharp.
ok. thank you.
@g3gg0
ran some batch tests with AGC ON and AGC OFF, 6 clips each, twice.
With AGC ON 8 clips out of 12 had static mostly the loud buzzing, 3 of these had the low variety.
With AGC OFF only 2 clips out of 12 had low static and a blank portion.
Below is a screenshot illustrating the Loud, low volume, and blank kinds of corruption.
(https://scontent-a-dfw.xx.fbcdn.net/hphotos-prn2/t1.0-9/10273413_481069682023821_3704817417348968556_n.jpg)
Pardon the typo, its meant to read "LOW VOLUME"
The batch tests runs as follows;
-Format CF card
-record 10 sec(clip1)
-record 10 sec(clip2)
turnLV off/on
-record 10 sec(clip3)
-record 10 sec(clip4)
turn cam off/on
-record 10 sec(clip5)
-record 10 sec(clip6)
batch 1 AGC ON
clip1. OK
clip2. 2-loud buzz
clip3. 1-loud buzz
clip4. OK
clip5. 3-low vol buzz
clip6. 2-low vol buzz
batch 2 AGC OFF
clip1. OK
clip2. OK
clip3. OK
clip4. OK
clip5. 1-low vol buzz
clip6. 3-low vol buzz
batch 3 AGC ON
clip1. 2-loud buzz
clip2. 2-loud buzz
clip3. OK
clip4. 1-loud buzz
clip5. OK
clip6. 2-low buzz
batch 4 AGC OFF
clip1. OK
clip2. OK
clip3. OK
clip4. OK
clip5. OK
clip6. OK
Hi guys
it could be great if someone create a program for mac like Rawmagic lite ( no problem to pay for it) if the program could handle mlv files correctly. I thinking about Mac version no windows. I hope that i can see some this program in close future.
Thanks in advance.
Quote from: jarandov on May 20, 2014, 01:11:16 PM
it could be great if someone create a program for mac like Rawmagic lite ( no problem to pay for it) if the program could handle mlv files correctly. I thinking about Mac version no windows. I hope that i can see some this program in close future.
Do you need CDNG output or just DNG? http://www.magiclantern.fm/forum/index.php?topic=9560
Hi Dmilligan,
Maybe a stupid question. What is the difference between dng cdng? I have tried MlRawViewer 1.1.4 but the program could chose only between mov and dng. Not cdng. Does the program export also sound files from mlv?
Quote from: jarandov on May 20, 2014, 02:50:03 PM
What is the difference between dng cdng?
I'm not an expert, but AFAIK, the main difference is simply metadata, but some video editors (e.g. Adobe Premiere) only accept files with all the correct CDNG metadata (there may be some other issues as well including bit depth).
Quote from: jarandov on May 20, 2014, 02:50:03 PM
the program could chose only between mov and dng
Right. What I meant was, do you require CDNG output? If not, then MLRawViewer should work fine, it can output DNG but not CNDG. The fact that you don't know the difference between DNG/CDNG, tells me you likely don't require it.
Quote from: jarandov on May 20, 2014, 02:50:03 PM
Does the program export also sound files from mlv?
Read the thread ;)
Quote from: jarandov on May 20, 2014, 01:11:16 PM
Hi guys
it could be great if someone create a program for mac like Rawmagic lite ( no problem to pay for it) if the program could handle mlv files correctly. I thinking about Mac version no windows. I hope that i can see some this program in close future.
Thanks in advance.
I've been using MLVMystic. Have you tried that program? It's working awesome for me, and it extracts the audio files for me too, but you still have to match audio and video in DaVinci Resolve.
For questions, answers and info regarding raw post processing pls post here: http://www.magiclantern.fm/forum/index.php?board=54
Quote from: arch on May 22, 2014, 01:55:12 AM
I've been using MLVMystic. Have you tried that program? It's working awesome for me, and it extracts the audio files for me too, but you still have to match audio and video in DaVinci Resolve.
This has been my workflow for the last 2 months or so. So far, it's working out well.
Quote from: llirik on May 23, 2014, 01:20:23 AM
This has been my workflow for the last 2 months or so. So far, it's working out well.
do you understand the post I made above yours?
Has anyone managed to have fullhd raw recording at 30fps with 1.2.3 and sound enabled, including card spanning?
I really like the way you can set mlv up to name your files sequentially, but I think it would be incredibly useful if there was a naming option that lets you set a prefix like "Cam1" or "CamB" and then it automatically adds the timestamp from RTC to the end of the file name - so if you start recording on camera B at 12:23:21 in the afternoon, the generated file name will be: CamB_122321
I know you can do stuff like this in asset management programs, but it would be nice to skip those and dump straight to your project folder if all you need is faux timecode.
What do you think?
Hello g3gg0, thank you for all your efforts regarding MLV.
Which graphical user interface do you personally recommend for mlv_dump on Windows for batch conversion? Do you recommend your MLV Browser?
Quote from: dpjpandone on May 25, 2014, 04:44:40 PM
a naming option that lets you set a prefix like "Cam1" or "CamB" and then it automatically adds the timestamp from RTC to the end of the file name
I've been having a look through the mlvrec module tonight in my quest for a better way to sync in post. Can someone tell me if it's possible to get a higher resolution of time from "LoadCalendarFromRTC" for example, does rtc give millisecond accuracy? if so, I think we should divide by framerate and add frames to the filename/metadata, is it possible to call "LoadCalendar" right before the first frame is recorded? So that performance does not influence accuracy? If we can do this, you can let your ext. audio recorder roll for the entire shoot, and you would only have to sync the first video clip manually. After that you can use the data from RTC to set timecode on the rest of the clips.
What do you think?
i dont have any clue about SMPTE and always wait for information from those who want to have support.
regarding your use case:
which information/field in MLV then should contain the time stamp so you can make use of it?
MLV itself contains a lot, but the RTC is only 1-second accurate.
Quote from: fascina on May 25, 2014, 11:12:18 PM
Hello g3gg0, thank you for all your efforts regarding MLV.
Which graphical user interface do you personally recommend for mlv_dump on Windows for batch conversion? Do you recommend your MLV Browser?
you are welcome.
uhm i recommend the one you can work with the best.
MLVBroseSharp is just a reference application that uses mlv_dump.
Quote from: g3gg0 on May 26, 2014, 02:54:18 PM
which information/field in MLV then should contain the time stamp so you can make use of it?
I think it would be immediately useful to add the timestamp to the filename, (it's already in mlv field)
The reason is that when proxies are created (and I think most render to a proxy format like cineform or prores for editing) the mlv data is lost, but filenames are easily retained.
The timestamp is not very useful when the file is in MLV format, it's not needed until editing, so this is why I think there should be an option to add timestamp to filenames.
When editing software allows native editing on MLV format, and computer hardware that can do it efficiently exists, we can make use of it in MLV data as well
The use case is this, when I import all the converted video files into adobe premiere, I can right click on a file, and easily assign it a starting timecode. If the timestamp is in the filename, it's a matter of copying and pasting into timecode window, and then at least the clips will automatically synchronize within one second. This is a huge timesaver for cameras that do not have reference sound for sync. At least then they are in the ballpark and only have to be nudged a few frames in either direction to achieve sync.
Another advantage is that when you open a project on another computer, and it has to search for files, it will be highly unlikely that duplicate files are found, since each file will have a timestamp in the name. So your project should find all the files with ease.
Here is a video that may help you understand: http://library.creativecow.net/harrington_richard/DSLR-Timecode/1
They are showing an excellent way to do this with h.264, but this does not work with mlv -> acr -> proxy workflow, because MLV timestamp is lost in conversion.
Re: second accurate clock:
second accuracy is better than no timestamp at all, but I think we can find a workaround to get frame accurate timestamp:
Since each frame has it's own timestamp in the MLV, we must find the first frame that has an updated timestamp, and we should use that as our reference to which all other frames timestamps are derived from. if I was coding for arduino it would go like this:
previousTimestamp=LoadRTCCal;
currentTimestamp=LoadRTCCal;
if currentTimestamp != previousTimestamp;
reference = currentTimestamp;
MLVtimestamp = (reference - offset);
I know this isn't entirely correct, but I hope it is helpful.
We will need a counter that is reset when the first frame starts recording, and the counter will increment each time a new frame is captured, this will give us our offset
Once we have found our reference frame, and subtracted the offset, we should have a frame accurate timestamp.
Please let me know if you need more information, or if there is anything I can do to help.
i understand your workflow now. thanks.
this helps me in finding a good solution.
Quote from: dpjpandone on May 26, 2014, 04:45:31 PMeasily assign it a starting timecode.
in which format do you need the timestamp?
I would prefer:
UserDefinableText_DD_MM_YYYY_Hours_Minutes_Seconds_Frames
if that is too many characters, then eliminate DD_MM_YYYY and the user can enter a short form of date in UserDefinableText if needed.
i made some analysis of microsecond timer vs. RTC and found some interesting stuff.
the RTC (well, its crystal) has quite a high drift compared to the digic microsecond timer.
we dont know which one is the more accurate clock.
but comparing each other you will get this:
(http://i60.tinypic.com/1692sxs.png)
one dot per 2 seconds. Y axis shows 857 to 878 µsec "offset" to the digic µsec timer.
every dot shows when the RTC does a second-tick.
the higher jumps are due the RTC-internal offset correction that is happening every minute.
(http://i58.tinypic.com/5zr86q.png)
according to this longer trace (10 minutes) my 5D3 has approximately 1 ms drift per minute between those two clocks,
thats approximately one frame off after 30 minutes inaccuracy at 25 fps.
this means if i calibrate the two clocks on startup, which takes two seconds, and from that on
i only use the interal digic timer (which is simple to read and provides a 1µsec resolution) then after operating
the camera for 30 minutes you will have one frame offset in the timecode no matter if you recorded one or 20 clips.
is this a problem?
solutions:
- do nothing and live with that
- rerun calibration on every recording start or every 10 minutes (two seconds delay)
- add some drift correction (already tested, have a 2 msec inaccuracy and nearly no drift anymore)
the drift correction isnt that simple to do and would take 30 minutes, but that has to be done only once.
oh and i forgot to mention:
we cannot name the file in camera that long. we can only do 8.3 there.
renaming to [prefix]_[DDMMYYHHMMSSff].MLV would only be possible on PC/mac.
Quote from: g3gg0 on May 27, 2014, 12:59:49 AM
- add some drift correction (already tested, have a 2 msec inaccuracy and nearly no drift anymore)
the drift correction isnt that simple to do and would take 30 minutes, but that has to be done only once.
This would be my pick. 30 minutes
once, for best accuracy. If you're anal enough to care about accuracy, then 30 minutes is a non issue.
Quote from: g3gg0 on May 27, 2014, 12:59:49 AM
the camera for 30 minutes you will have one frame offset in the timecode no matter if you recorded one or 20 clips.
is this a problem?
This is not a problem, I wouldn't mind correcting the drift in post.
are you saying it will take 30 minutes for you to code drift correction? What can I do for 30 minutes that will make your life easier? That is of course the best solution and it is easy to choose when it is someone else's time...
For the second solution, adding 2 seconds delay before recording starts is undesirable, it already takes a while for recording to begin sometimes. I think I would rather have it drift.
well for me to implement its one day of work and testing.
for running the calibration, you have to spend the 30 minutes mentioned.
after calibration the clocks are quite in sync at the given temperature and give you a few miliseconds accuracy.
i can also calibrate the drift correction so we will have <0.1msec accuracy to the RTC - AT THIS ENVIRONMENTAL CONDITIONS (air pressure, humidity, temperature).
so i see three parts of calibration that take some time
1. [2 sec] RTC/µsec offset detection (will be different after every poweron)
2. [10-30 min] RTC/µsec drift detection (depends lets say 50% on camera and 50% on environmental conditions)
3. [2 sec] RTC correction rate detection (can be done along with step 2)
all together i guess thats 3 days effort.
This is so incredibly awesome. Whatever steps are necessary to correct the drift would be absolutely worth it. Having this frame accurate timestamp will save an incredible amount of time in post.
To get around 8.3 filenames, I guess that's exactly enough characters for HHMMSSFF and any filebrowser will show they date they were created in the details anyways.
Quote from: dpjpandone on May 27, 2014, 07:56:48 PM
HHMMSSFF
okay but there is one thing i dont really get.
according to SMPTE standards, for NTSC 29.97 fps there are two frame drops every minute, except every 10 minutes.
so if its exactly 00:00:00 in the night and i start recording, then the first frame's time should be 00:00:00:02, right?
and if i start recording at 00:00:00 plus 100 msecs, which frame number should i pick?
or can i completely ignore frame dropping for the file naming stuff?
from this: http://www.larryjordan.biz/technique-drop-frame-vs-non-drop-frame-timecode/
"The rule of drop-frame timecode is: two frames of timecode are dropped every minute, except on the tenth minute, when nothing is dropped. This means that over the course of an hour, 108 timecode labels are skipped. The exact numbering goes: 59:28, 59:29, 1:00:02, 1:00:03... (Notice two timecode labels were skipped: 1:00:00, and 1:00:01."
so if recording at 23.97 or 29.97, yes. it starts at 00:02
I have to think about the second question for a while. My first thought is to just round up to next frame for file naming. maybe I didn't think this all the way through. It's a unique system because we are using RTC as master clock.
There are some formulas for timecode math here:
http://www.andrewduncan.ws/Timecodes/Timecodes.html
I'm sorry, I was wrong, it does not start at 00:00:00:02, it starts at 00:00:00:00
00:00:00,00
00:00:00,01
... then the frames get dropped
00:00:59,28
00:00:59,29
00:01:00,02
00:01:00,03
(etc.)
yeah and now record for 24 hours then the 24th hour 24:00:00 is again a multiple of 10 minutes...
I am getting this message when trying record one continuous clip on a 64GB card: "Failed to create file. Card full?" I get this error at around 40gb or 10 minutes. The camera locks up and I have to force the restart. Is this a known issue? I am having this problem with mlv-rec on my 7D.203 and 5D3.113.
Anyone have the same problem or know if any setting to change to resolve the issue?
Quote from: g3gg0 on May 27, 2014, 10:22:58 PM
yeah and now record for 24 hours then the 24th hour 24:00:00 is again a multiple of 10 minutes...
Now I understand the problem...
I guess normal timecode does not have this problem because it is linear (not cyclic) so it never loops around and goes back to 0 like our daily clock. You start at 0 and just keep on counting. So since this is in effect a new system we are designing that is based around a cyclic master clock, I think that the choice is really up to you. What makes the most sense?
Should we abandon the idea of using RTC as master clock, and allow user to set time to 00:00:00:00 using the shutter or half-shutter button?
This would offer the following advantage:
a splitter cable can be made that links several camera's shutter ports together, you enter the timecode menu on all of them, and then press the shutter button on just one camera, and they all reset to 00:00:00:00 simultaneously. Now the cables can be removed and the cameras are free to move about and start/stop as they wish.
This is how traditional timecode works, and is perhaps a better model than my initial proposal.
got the RTC and µs timer in sync with about 200 µsec accuracy stable within 30 minutes.
still my statement is that SMPTE isnt selfexplaining. it is full of quirks and only partially consequent stuff.
it leaves too much room for the implementation that would cause trouble to someone sooner or later.
You are absolutely right. I've been thinking about this a lot. Using the RTC as reference gives us the advantage that we can turn the camera off and on and still keep our 0 reference from the beginning of the day. Does it make sense to set a certain time of day as 0 and just count up from there until we hit 99:99:99:99 then force user to set a new 0 ? I think most projects would reset at least once per day, so I think the scenario of running out of timecode is unlikely.
I don't know if you made peace with this or not, I know SMPTE is a very old spec that predates computers... Please let me know if you need additional input, I don't wish to jack the entire MLVREC thread for this timecode idea, and I know there are lots of other things you are working on, so I'l give it a rest until you bring it up again. Thanks for all your hard work!
@g3gg0
Testing May 23 NB and the static/buzzing noise I reported here: http://www.magiclantern.fm/forum/index.php?topic=7122.msg115953#msg115953
is still present.
I tested both with agc ON and agc off.
Here is a link to one wav file extracted from an MLV recording.
https://www.dropbox.com/s/3w0babsv9eqprop/M28-1205.wav
this one with agc OFF using internal mic.
In a series of batch tests, it appears 8x out of 17 clips.
please upload the .mlv as i dont get such effects.
here some example how it looks on 5d3:
(http://i59.tinypic.com/whb0i0.png)
Quote from: g3gg0 on May 28, 2014, 09:37:32 PM
please upload the .mlv as i dont get such effects.
Ok, since I already deleted the mlv for that wav, I did another batch test.
Here is the link to an approx, 10 second MLV file. https://www.dropbox.com/s/ir84sa4vnkpuu4m/M28-2329.MLV
This is the first clip recorded after doing an in-camera format.
AGC is OFF
Sound source -internal mic.
hi
just installed last nightly built
seems to work fine.
just shooted a 4min30sec with mlv_rec. No problem.
thats greta !! thank you very much.
i noticed that when i press the SET button when shooting video (mlv or raw_rec) it freeze the screen; then i am not able to turn off the camera; Is it a known issue ?
soorry for my english
, i'm best in french ; )
Having trouble deleting MLV files after playing them back in camera on the 7D, then using the mlv_play menu to delete it. There's a beep at the beginning of playback, and attempting to delete it causes it to hangs up, not exit the play menu or go to another clip, I have to press half shutter to exit it. Then an error pops up, Deleting 'A:DCIM/100EOS7D/M29-2131.MLV' failed, retrying...
If I reboot, press play again and try to delete it again, same thing. But if I reboot and delete through the file manager, it deletes fine.
Seems to only be only shots on shots done in crop mode, with MLV sound off or on.
magiclantern-Nightly.2014May30.7D203
Memory leak, possibly? Seems like it can play back the normal crop videos and delete them just fine, but anything bigger than 1728 causes it to hang.
Quote from: SirDeVoir on May 30, 2014, 01:29:34 AM
i noticed that when i press the SET button when shooting video (mlv or raw_rec) it freeze the screen; then i am not able to turn off the camera; Is it a known issue ?
which model?
did you try reset ML settings to default? (backup your ML.cfg before so we can reproduce)
Quote from: ted ramasola on May 29, 2014, 08:40:08 AM
Ok, since I already deleted the mlv for that wav, I did another batch test.
confirmed the bug, seems audio data was never written there. weird.
5D2 it is?
Quote from: g3gg0 on May 30, 2014, 09:40:49 AM
which model?
did you try reset ML settings to default? (backup your ML.cfg before so we can reproduce)
I had just downloaded the last nighty built and replaced all the files on the SD card. So i presume the settings wad to default.
and the model in 5D3, sorry....
No, overwriting ML directory will not reset customized settings. Delete ML directory first.
Quote from: Walter Schulz on May 30, 2014, 02:59:38 PM
No, overwriting ML directory will not reset customized settings. Delete ML directory first.
I deleted ML File on the SD card, the installed the new one
I restored ML default in Prefs / Config files Menu
rebooted
then loaded MLV_play MLV_rec MLV_sound modules
recorded in MLV mode.
same results if I press SET button. Screen freeze. Impossible to turn off the camera.
I'm not sure if this is necessarily a bug, let me know if I should create an issue for the following:
When monitoring RAW in hdmi with global draw enabled, the preview image is not centered vertically on the external monitor. I have a zacuto EVF with a cropmarks feature, I set the zacuto cropmarks to the same aspect as I'm shooting in, the preview image is offset too high on the external monitor. When i disable global draw, the full screen output is centered and I can frame perfectly with the zacuto's cropmarks.
Camera is 7D with may 24th nightly. I think my monitor has a frame capture feauture. I'll try to do a screengrab and post it.
Hello, I use a camera 600D ML Nightly.2014May13.600D102 mlv_rec.mo + mlv_snd.mo. But, sometimes my camera records correct sound - waw file and sometimes is wav file without any sound (flat line). I don know why. The settings of ML is same. RAW video is OK /1728x576, 11fps/. What should I do better? Thanks.
Audionut suggested I post in this thread a problem that I have encountered with mlv_rec. I get a "Failed to create file. Card full?" error message when recording a continuous clip using a 64gb card at the 40gb and 8 minute and 13 seconds mark. The camera freezes at that point. This error also happens with my 7D at 40gb and around 10 minutes of recording.
You can view the thread discussion here (http://www.magiclantern.fm/forum/index.php?topic=12060.0) for more details. Other users are seeing the same issue.
This happens when using both cameras:
-5D mark 3
-64gb Lexar Card
latest nightly build of firmware 113
-7D
-64gb Lexar Card
latest nightly build of firmware 203
g3gg0,
Do you have any idea what might be causing this? I really love using mlv_rec because I can have sound. This failure doesn't happen with raw_rec. Thanks so much for all the work you have done with mlv_rec. I appreciate it so much! Let me know if there is any other testing I could do to help narrow down the problem.
Thanks
Rob
Hi,
If you shoot crop mode (x5 zoom) with a 5D Mark III and use FPS override (for example 24 exact FPS) MlRawViewer will report 29,776 fps. Maybe it's a metadata mistake.
I verified shooting a timer watch in crop mode. It's actually using the override setting. Here is an actual file (2,2 Gb): https://drive.google.com/file/d/0B3ncQl4hSqlnR096bXNWbk1PYTQ/edit?usp=sharing (https://drive.google.com/file/d/0B3ncQl4hSqlnR096bXNWbk1PYTQ/edit?usp=sharing)
Posted also in MlRawViewer thread.
Quote from: Kuky on June 05, 2014, 04:53:59 PM
Hi,
If you shoot crop mode (x5 zoom) with a 5D Mark III and use FPS override (for example 24 exact FPS) MlRawViewer will report 29,776 fps. Maybe it's a metadata mistake.
I verified shooting a timer watch in crop mode. It's actually using the override setting. Here is an actual file (2,2 Gb): https://drive.google.com/file/d/0B3ncQl4hSqlnR096bXNWbk1PYTQ/edit?usp=sharing (https://drive.google.com/file/d/0B3ncQl4hSqlnR096bXNWbk1PYTQ/edit?usp=sharing)
Posted also in MlRawViewer thread.
Make sure you change your composition FPS. This is covered in the thread started by audionut about questions that are redundant. Also double posting. :-)
Quote from: dubzeebass on June 06, 2014, 02:10:07 AM
Make sure you change your composition FPS. This is covered in the thread started by audionut about questions that are redundant. Also double posting. :-)
Composition FPS? What composition FPS? What thread? Search does not reveal any relevant results.
BTW, AE composition settings (maybe this is what you're talking about) has nothing to do with the fact that mlv has incorrect fps metadata when shooting crop mode. Of course you can interpret footage any way you like.
The sticky to end all stickies. Read this first. (http://www.magiclantern.fm/forum/index.php?topic=11108.0)
QuoteI recorded at 23.976 FPS, but my footage is showing as 30 FPS in my rendering software?
Your rendering software is set to use 30 FPS as default. Change it!
Composition FPS? (http://bit.ly/1hnwG2e)
baldand seems to think the problem is with the header. IIRC, the correct FPS tag doesn't get written to files when using FPS override. Until that is fixed, consider the quoted Q&A as correct, even though the wording doesn't exactly describe this problem.
edit: Added a new Q&A. Thanks for the heads up.
yet i didnt see what i always ask for in this case.
the output of "mlv_dump -v" for the whole file or the first blocks at least.
Quote from: g3gg0 on June 07, 2014, 08:38:13 AM
yet i didnt see what i always ask for in this case.
the output of "mlv_dump -v" for the whole file or the first blocks at least.
http://pastebin.com/L5UwV9K2 (http://pastebin.com/L5UwV9K2)
Quote from: Audionut on June 07, 2014, 06:18:01 AM
The sticky to end all stickies. Read this first. (http://www.magiclantern.fm/forum/index.php?topic=11108.0)
Composition FPS? (http://bit.ly/1hnwG2e)
baldand seems to think the problem is with the header. IIRC, the correct FPS tag doesn't get written to files when using FPS override. Until that is fixed, consider the quoted Q&A as correct, even though the wording doesn't exactly describe this problem.
edit: Added a new Q&A. Thanks for the heads up.
Yeah, thanks for googling it for me, but as I mentioned
before it has nothing to do with the metadata tag. In fact it's just a workaround and the Q&A is relevant if the DOP and the editor are the same person. If not, my experience tells me, being a post supervisor since 1997, that many of the editors will ingest the footage, growling "garbage in, garbage out". Of course there are also editors who will take the initiative and make a phone call or will try to guess the correct fps.
Quote from: Kuky on June 07, 2014, 01:05:04 PM
but as I mentioned before it has nothing to do with the metadata tag.
My bold for emphasis.
Quote from: Kuky on June 05, 2014, 04:53:59 PM
If you shoot crop mode (x5 zoom) with a 5D Mark III and use FPS override (for example 24 exact FPS) MlRawViewer will report 29,776 fps. Maybe it's a metadata mistake.
http://www.catb.org/esr/faqs/smart-questions.html#beprecise
Quote from: Kuky on June 07, 2014, 01:05:04 PM
In fact it's just a workaround and the Q&A is relevant if the DOP and the editor are the same person. If not, my experience tells me
Hopefully from this day forth, experience will tell you, that if you are going to use
experiential software, you will take care to inform the editor of any specific needs your footage has.
G3ggo,
I would like to help with the issue of hdmi overlay misalignment (issue#1582)
I can try different values and compile and test until they line up, can you give me a hint which lines are relevant?
small info: coders and all others interested in the mlv-format can analyze the data inside a mlv with this tool.
its far from being perfect, but it helps me to get minified files for debugging.
these files should work in all converters as well.
mlvdiag 1.0.0 - you can find it here (http://www.magiclantern.fm/forum/index.php?topic=5618.msg119737#msg119737)
regards chmee
marekk' timelapse discussion moved here (http://www.magiclantern.fm/forum/index.php?topic=12479.0).
mk3 mlvraw images and raw video can be viewed in cam, but PC doesn't recognize any media on the card? sorry if this has been discussed already.
Quote from: gravitatemediagroup on July 10, 2014, 06:37:26 AM
mk3 mlvraw images and raw video can be viewed in cam, but PC doesn't recognize any media on the card? sorry if this has been discussed already.
from card reader or trough USB to camera?
Can you wrap ML .RAW files back into an MLV file?
Quote from: ansius on July 10, 2014, 12:49:59 PM
from card reader or trough USB to camera?
card reader, I have a different reader i'm about to try now.
update: different reader still no luck, going to try on a mac and a different PC. maybe a windows update is causing me a glitch.
Quote from: AgentJJ on July 10, 2014, 11:45:45 PM
Can you wrap ML .RAW files back into an MLV file?
Theoretically, but I see no point. The raw file is missing the metadata that mlv was designed to capture. Analogy: you can turn a plain text file into a rich text file, but when you do there's not magically going to appear a bunch of formatting. Might as well just leave it a plain text file.
It would be purely retroactive. I have about 3 TB of ML .RAW files that I would love to wrap up to MLV so that I could compress them and save a load of space! I could write up a tool but I was hoping the protocol might already be implemented somewhere.
Quote from: AgentJJ on July 11, 2014, 08:04:55 PM
I have about 3 TB of ML .RAW files that I would love to wrap up to MLV so that I could compress them and save a load of space!
Making them MLV would only make them larger. Why would you need them to be MLV to compress them?
The same analogy applies, why would you make a plain text file into a rich text file so that you could compress it? The rich text file is going to be larger. Just compress the plain text file.
Only because mlv_dump already compresses the MLV files down via LMZA, while I don't see anything that compresses just the RAW files down. The amount of compression via mlv_dump far exceeds the overhead in the wrapping.
Quote from: AgentJJ on July 14, 2014, 02:03:09 PM
while I don't see anything that compresses just the RAW files down
There are thousands of programs that can losslessly compress data out there (for almost any OS you can think of)
how about gzip (http://en.wikipedia.org/wiki/Gzip)? (or if you specifically want LZMA, then lzip (http://en.wikipedia.org/wiki/Lzip))
see also (http://en.wikipedia.org/wiki/List_of_file_archivers)
Quote
There are thousands of programs that can losslessly compress data out there (for almost any OS you can think of)
:-[ Of course! I'm a programmer and I forgot all about gzip ( and I use it pratically everyday with debian packages... ).
Thanks! It's the simple things I forget...
How do I check what the fPS is for a file? I'm on a mac.
./mlv_dump ????
Thank you very much
Quote from: shambhu66 on March 12, 2014, 04:41:31 PM
Compilation fails on OSX Maverick after Xcode was upgraded to version 5.1 on March 10, 2014.
[Edit:] I should have added that ML autoexec.bin and modules compile OK without any issue even after the Xcode upgrade.
compilation fails only for mlv_dump.
[ HOST_CC ] mlv_dump.host.o
clang: error: unknown argument: '-mno-ms-bitfields' [-Wunused-command-line-argument-hard-error-in-future]
clang: note: this will be a hard error (cannot be downgraded to a warning) in the future
make: *** [mlv_dump.host.o] Error 1
This did not help either:
export CFLAGS=-Qunused-arguments
export CPPFLAGS=-Qunused-arguments
Removed '-mno-ms-bitfields' from modules/mlv_rec/Makefile, but ld was not happy.
MLV_CFLAGS = -I$(SRC_DIR) -D MLV_USE_LZMA -m32 -Wpadded -D _7ZIP_ST -D MLV2DNG
ld: warning: ld: warning: ignoring file ../../src/chdk-dng.host.o, file was built for i386 which is not the architecture being linked (x86_64): ../../src/chdk-dng.host.oignoring file mlv_dump.host.o, file was built for i386 which is not the architecture being linked (x86_64): mlv_dump.host.o
ld: warning: ignoring file ../lv_rec/raw2dng.host.o, file was built for i386 which is not the architecture being linked (x86_64): ../lv_rec/raw2dng.host.o
ld: warning: ignoring file lzma/lib7z.a, file was built for archive which is not the architecture being linked (x86_64): lzma/lib7z.a
Undefined symbols for architecture x86_64:
"_main", referenced from:
implicit entry/start for main executable
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [mlv_dump] Error 1
Removed -m32 from modules/mlv_rec/Makefile, no luck either.
MLV_CFLAGS = -I$(SRC_DIR) -D MLV_USE_LZMA -Wpadded -D _7ZIP_ST -D MLV2DNG
Macintosh:mlv_rec okumar$ make clean && make mlv_dump
[ RM ] mlv_rec.o mlv.o mlv_rec.mo mlv_rec.sym mlv_rec.dep module_strings.h *.o *.d *.dep *.sym hgstamp
[ RM ] mlv_dump mlv_dump.exe lzma/7zAlloc.host.o lzma/7zBuf.host.o lzma/7zBuf2.host.o lzma/7zCrc.host.o lzma/7zCrcOpt.host.o lzma/7zDec.host.o lzma/7zFile.host.o lzma/7zIn.host.o lzma/7zStream.host.o lzma/Alloc.host.o lzma/Bcj2.host.o lzma/Bra.host.o lzma/Bra86.host.o lzma/BraIA64.host.o lzma/CpuArch.host.o lzma/Delta.host.o lzma/LzFind.host.o lzma/Lzma2Dec.host.o lzma/Lzma2Enc.host.o lzma/Lzma86Dec.host.o lzma/Lzma86Enc.host.o lzma/LzmaDec.host.o lzma/LzmaEnc.host.o lzma/LzmaLib.host.o lzma/Ppmd7.host.o lzma/Ppmd7Dec.host.o lzma/Ppmd7Enc.host.o lzma/Sha256.host.o lzma/Xz.host.o lzma/XzCrc64.host.o lzma/lib7z.a lzma/Threads.w32.o lzma/LzFindMt.w32.o lzma/MtCoder.w32.o lzma/7zAlloc.w32.o lzma/7zBuf.w32.o lzma/7zBuf2.w32.o lzma/7zCrc.w32.o lzma/7zCrcOpt.w32.o lzma/7zDec.w32.o lzma/7zFile.w32.o lzma/7zIn.w32.o lzma/7zStream.w32.o lzma/Alloc.w32.o lzma/Bcj2.w32.o lzma/Bra.w32.o lzma/Bra86.w32.o lzma/BraIA64.w32.o lzma/CpuArch.w32.o lzma/Delta.w32.o lzma/LzFind.w32.o lzma/Lzma2Dec.w32.o lzma/Lzma2Enc.w32.o lzma/Lzma86Dec.w32.o lzma/Lzma86Enc.w32.o lzma/LzmaDec.w32.o lzma/LzmaEnc.w32.o lzma/LzmaLib.w32.o lzma/Ppmd7.w32.o lzma/Ppmd7Dec.w32.o lzma/Ppmd7Enc.w32.o lzma/Sha256.w32.o lzma/Xz.w32.o lzma/XzCrc64.w32.o lzma/lib7z.w32.a
[ HOST_CC ] mlv_dump.host.o
mlv_dump.c:2034:38: error: no member named 'buffer' in 'struct raw_info'
raw_info.buffer = frame_buffer;
~~~~~~~~ ^
1 error generated.
make: *** [mlv_dump.host.o] Error 1
I'm having this error too, unable to build mlv_dump on new OSX OS.
Edit: the fix for OSX:
diff -r e12ad3d84e7b modules/mlv_rec/Makefile
--- a/modules/mlv_rec/Makefile Thu Sep 11 23:50:29 2014 +0200
+++ b/modules/mlv_rec/Makefile Sun Sep 14 23:59:14 2014 -0700
@@ -6,9 +6,9 @@
# include modules environment
include ../Makefile.modules
-MLV_CFLAGS = -I$(SRC_DIR) -D MLV_USE_LZMA -m32 -Wpadded -mno-ms-bitfields -D _7ZIP_ST -D MLV2DNG
+MLV_CFLAGS = -I$(SRC_DIR) -D MLV_USE_LZMA -m32 -Wpadded -D _7ZIP_ST -D MLV2DNG
MLV_LFLAGS =
-MLV_LIBS = -lm
+MLV_LIBS = -lm -m32
MLV_LIBS_MINGW = -lm
autoexec.bin and modules are compiled using a compiler for arm processor. mlv_dump is compiled using standard gcc compiler for Mac Osx. You should to install gcc using port.
3 days ago, ffmpeg released a new major version 2.3. This release supports mlv-demuxing. I´m waiting for this version to find the way to the repos of my OpenSuse (packman). I don´t know, what is possible with this new feature in ffmpeg, but I will find it out :)
Edgar
I get tons of dropped frames on MK3 in 5x crop.
Hmmm... I use 3x crop a bit myself, but haven't noticed any dropped frames. How are you determining this? Is there a log or are you noticing it when viewing the footage?
Also, are you allowing frame skipping when this happens and are you trying to record faster than the camera can write to the card?
In about 10 seconds of recording I have like 20+ dropped frames, more or less depending on the detial probably
. It is telling me in the top right corner while recording, not sure if it is complately accurate or not.
Hi g3gg0
few of us think mlv_dump for windows have a bug with large images such as those comes from full frame sensors. can you investigate
http://www.magiclantern.fm/forum/index.php?topic=12733.msg123171;topicseen#msg123171
EDIT: THANK YOU FOR THE NEW FIX
thanks for info. will play with that as well.
Quote from: escho on July 20, 2014, 03:18:16 PM
3 days ago, ffmpeg released a new major version 2.3. This release supports mlv-demuxing. I´m waiting for this version to find the way to the repos of my OpenSuse (packman). I don´t know, what is possible with this new feature in ffmpeg, but I will find it out :)
Edgar
found a block named "BKUP". whats that? should be mentioned in the google-docs on 1st thread-entry.
this one was a temporary solution and should not happen anymore.
just ignore that block (as any unknown block)
:) ok, found it in a todays recording. forgotten right now.
this shouldnt be in mlv files for months now :)
@g3gg0
I've been trying to improve the performance of exact playback of MLV files (on my 50D), which struggles on >~1 GB files due to out-of-sequence blocks. Canon FIO has poor performance with reading backwards, with increasing delays the further into a file (presumably it's loaded from the beginning). Consequently, increasingly more frames need to be skipped the further past >~1 GB, and eventually, around 3 GB, it's no longer possible to read block headers fast enough to catch up with the FPS timer and playback stalls.
The issue is that mlv_play reads the header of every block during playback, and all of those FIO_ReadFile calls start becoming too expensive. I've tried some sub-optimal solutions – skipping over blocks, but that skips metadata blocks too, or flushing the timer queue when too far behind, but then playback isn't exact anymore – but I think the "right" solution would be to add information to the XREF block in the generated IDX file so that it's possible to know whether a block can be skipped (VIDF) in playback without needing to read the block header (again).
Here's my implementation: https://bitbucket.org/ayshih/magic-lantern-50d/branch/mlv_play_fix3 (https://bitbucket.org/ayshih/magic-lantern-50d/branch/mlv_play_fix3)
https://bitbucket.org/hudson/magic-lantern/branch/mlv_play
I split off a byte from the unused variable empty in the structure mlv_xref_t, and this new frameType variable is set to 1 if the block is VIDF and thus skippable. This changes the format specification, but I'm hoping the ramifications will be minor because the structure variable was unused and the size of the structure hasn't changed. (Existing on-camera IDX files would need to be re-generated.) Playback starts to drag deep into a 4 GB file, but now playback completes and maintains >2 FPS.
What do you think? This frameType variable could be extended to discriminate all of the different block types, if useful.
yep, like that change :)
just one change: use it as enum and define metadata (0), video (1) and audio (2) enum values.
so it is extensible
Is 24-bit audio recording conceivably possible via mlv_rec, or is the camera (5D3) limited to 16?
Quote from: g3gg0 on August 07, 2014, 06:27:19 PM
just one change: use it as enum and define metadata (0), video (1) and audio (2) enum values.
so it is extensible
Are you recommending an actual
enum? I'm not sure how that would work since I can't include an
enum as part of the structure due to size, and it'd be clunky as a separate global. I'll use preprocessor constants to remove the magic numbers, and then I'll issue the pull request.
Edit: pull request (https://bitbucket.org/hudson/magic-lantern/pull-request/580/improved-performance-for-exact-playback-of/diff)
Quote from: stevefal on August 07, 2014, 10:23:29 PM
Is 24-bit audio recording conceivably possible via mlv_rec, or is the camera (5D3) limited to 16?
16 bit hardware limit as far as i can tell
Hi!
First of all, i am sorry for my bad english.
I have an idea about the noise reduction in mlv_dump or raw2dng.
1 Write the sample of video without a light.
2 Averages the noise(frames). Subtract noise by 2 or more frames
3 Subtract from the normal video.
It mostly constant noise.
It is convenient to do during unpacking, right in the DNG.
You can add a key to the command line.
I can not do this, because I don't know C. I understand the sources, but i can't change it ::)
I did this in Visual Basic, but with 16-bit Tiff files :) It's not comfortable, and I could not to convert them back into DNG :(
I think it would be useful
This link can be usefull.
http://www.photo.net/learn/dark_noise/
Thank you for your atention!
this is not related to the MLV file format. you can do this in post with any tool.
Quote from: g3gg0 on August 30, 2014, 11:09:04 AM
this is not related to the MLV file format. you can do this in post with any tool.
it doesn't make sense in post, only before Camera raw. In Post(if you use CR Noise reduction) it wont work.
You can already do this with mlv_dump (see the -s option), and like g3gg0 says, it's not related to the thread topic which is about the MLV file format itself, NOT about how to best post process those files (there are many other threads for that: http://www.magiclantern.fm/forum/index.php?board=54.0)
Quote from: AWPStar on August 30, 2014, 02:58:50 PM
it doesn't make sense in post, only before Camera raw. In Post(if you use CR Noise reduction) it wont work.
before raw: modifying analog or optical path => hardware mod, not ML
after raw: DNG files => can be done in postprocessing
special case: modifying ADC parameters like black level or CMOS registers => not related to your request.
and as dmilligan said:
not here, please (mlv_dump -a averages frames, mld_dump -s subtracts a reference frame)
Is there a way to do this option:
MLV_dump -a average all frames in <inputfile> and output a single-frame MLV from it.
When you already got rid of the MLV file and only have some dng's left ?
I'd like to average out a lot of DNG's for dark frame usage, but I don't have the original MLV files anymore.
Any chance to average out some dng's with mlv_dump or maybe other software :-\
MacUser...
unfortunately not easily possible without rewriting it totally :(
you could export DNG to some other raw format and use e.g. octave to average
You can use ImageJ (http://rsbweb.nih.gov/ij/index.html) (free), probably darktable (http://www.darktable.org/install/) (also free). or even photoshop.
Also DeepSkyStacker (http://deepskystacker.free.fr/)
Ah :) thanks for the other software options.
Just taking a look at some high iso clips I shot for fun a while ago, at the same time I also did some clips with lens cap on for dark frames.
Trying to figure out what is possible with the extreme iso's.
Regarding MLV large file support is there a reason you want to have all variables within mlv_rec.c? I'm thinking it would be better to set a global variable in bootflags.c where the system actually knows if it is an exFAT card- line 171 in bootflags.c ((p.type == 7) // ExFAT)
Right now mlv_rec.c just knows what camera models have support for exFAT but doesn't check.
/* not all models support exFAT filesystem */
uint32_t exFAT = 1;
if(cam_5d2 || cam_50d || cam_7d)
{
exFAT = 0;
large_file_support = 0;
}
I've updated the long unique exFAT file names that I've been using and will upload it soon for you to consider. File name examples are: 20140927_102314_ n4yl.MLV and if you've enabled camera number and take number: 20140927_102314_cam1_take3_7pbi.MLV (The last 4 characters are just randomized "0123456789abcdefghijklmnopqrstuvwxyz"). I know some people are shooting multi camera and have problems when they get the same file name.
I try to compress about 6 GB file:
mlv_dump.exe -o compress.mlv -v -e -c -b 16 -l 9 -x big.mlv
and I get about 4 GB file, good.
Than, i try to extract dng from that file:
mlv_dump.exe --dng --no-cs compress.mlv -o
I get about 2 GB of dngs and mlv_dump.exe crashes having about 2 GB memory allocated according to Task manager. My PC is 64 bit with 24 GB memory.
The file MLV_REC.TMP is there only to reserve space, so I don't need it to make my .dng files neither to proxy them. So I can delete before decompressing the raw files. Am I right?
Quote from: mothaibaphoto on February 25, 2015, 06:55:05 AM
I try to compress about 6 GB file:
mlv_dump.exe -o compress.mlv -v -e -c -b 16 -l 9 -x big.mlv
You might want to try this: https://bitbucket.org/hudson/magic-lantern/branch/mlv_rec_lj92
@dmilligan: Thanks, looks interesting, but... the main thing I can understand from comments - this is something to compress in camera (as long, as it looks like modifications of mlv_rec module). But I actually want to find a best way to backup video. I can't believe, that compressing is possible in camera without a big overhead. Anyway, I can't try this till someone will make compiled modules available.
It's not for in camera compression, that's impossible. mlv_dump shares some code with mlv_rec and they live in the same directory, that doesn't mean these changes are to support in-camera compression.
This new compression method is much faster than the current one. I implemented it in MLVFS and getting pretty close to real-time playback.
It shouldn't be that hard to compile using mingw or whatever. You don't have to setup the whole ARM toolchain to compile it.
Howdy!
First off - great work, much appreciated.
So, what I have is an image of CF card, after in-camera recording failure. Unfortunately it was a new year celebration, crossing the 00:00 mark, so I'd really like to recover it. Right after the failure I took the CF card out, to prevent overwriting. Thus, this exact image contains fragments of all files recorded earlier and not yet overwritten.
The goal: scan it for all possible mlv file headers, then scan for all frame blocks following definitions of found mlv files, then save all of them sorted according to time stamp into corresponding mlv files (with option of filling gaps - if frame is missing, repeat the previous one with correct TS, option of inserting bright/loud dummy frame to make sure I will notice it, option of not inserting anything but what was found). Would be nice to make a check for blocks (i.e. partial / erroneous should be skipped).
Surely, some of them will not workout well, but my interest is primarily focused on the recovery of the last one. I checked definition of C structs, and probably can make a parser, but it should be definitely much easier and faster to get done by Author.
Is this possible/could be made?
Thank you,
Mikhail
FIRST: make an image of the CF card using some disk imaging tools (there are some open source tools)
some weeks ago i chatted with the developer of PhotoRec who was so kind to add MLV support.
so: get photorec, it will do its best to recover your footage.
If one didnt need sound, should we just stick to .raw instead?
if your desire is higher speeds, yes. thats the big advantage of .raw.
if you want metadata with your footage, choose MLV.
i personally do not need high speeds but prefer the files to contain all possible information.
it might be interesting to convert .raw->.mlv as .mlv seems to reach a broad tool suite.
so people who need to work around the mlv-deficit "writing speed" can still post process and archive it as .mlv
+10 for raw to mlv
Yes, I of course did make image - to secure data and free-up the CF card for usage. Will think over, Thanks!
Wasn't there already a tool for that, RAW2MLV?
Does somebody know if mlv_dump can average all images in an Full-res-silent-picture MLV ?
I can't get the -a, option to work on full res silent MLV ?
This is what I type in terminal
mlv_dump -a /Users/wij/Documents/TEST/72160000.MLV
This is the output:
MLV Dumper v1.0
-----------------
Mode of operation:
- Input MLV file: '/Users/wij/Documents/TEST/72160000.MLV'
- Verify file structure
File /Users/wij/Documents/TEST/72160000.MLV opened
File /Users/wij/Documents/TEST/72160000.M00 not existing.
Processing...
Reached end of chunk 1/1 after 205 blocks
Processed 50 video frames
Floating point exception: 8
uhm i the number of averaged frames is exactly zero.
thats why there is an exception.
reason: you forgot to add an output file
He G3gg0,
Tried to put an output file in the command line...I get a segmentation fault 11 and a zero byte sized averaged.mlv
mlv_dump -o averaged.mlv -a /Users/wij/Documents/TEST/72160000.MLV
MLV Dumper v1.0
-----------------
Mode of operation:
- Input MLV file: '/Users/wij/Documents/TEST/72160000.MLV'
- Rewrite MLV
- Output only one frame with averaged pixel values
- Output into 'averaged.mlv'
File /Users/wij/Documents/TEST/72160000.MLV opened
File /Users/wij/Documents/TEST/72160000.M00 not existing.
Processing...
Segmentation fault: 11
EDIT:
The reason I'm trying this, is because I want to have a long exposure picture in daylight without needing ND filters.
Nice landscape shots, with soft tree leaves and soft clouds 8)
@g3gg0:
Just wanted to provide feedbacks regarding 70D footage. Apparently either the MLV_Dump isn't responding to 70D files nor there aren't proper color matrixes ( it thinks its a 6D when it isn't? lol ) It looks like it may be lacking either one of these....
Here are the 70D111B files in case you need to dig through them...
Short Clips:
https://mega.co.nz/#F!HtUx0CgS!jQslzxzJTC19VVwGGN75lA
Varies of Files including Dual-ISO:
https://mega.co.nz/#F!yoNQWBTa!3ohyAu9UggiyreAiFjo8nA
I've also posted some videos of what the footage looks like via Vimeo:
https://vimeo.com/126099527 (https://vimeo.com/126099527)
Hope this helps!
*UPDATE*
After busting an all nighter with an fellow ML buddy (Special Thanks to @Danne!) he finally made it possible for me to get my converted DNG's from 70D's Dual-ISO CR2's to work again (Previously wouldn't be able to read from PS) with the help of a '70D matrix command (https://drive.google.com/file/d/0B4tCJMlOYfirTG9xN0pCN0pxTGM/view?usp=sharing70D%20matrix%20command)' provided by @Danne. (Hope it's OK w me sharing this link - if not let me know!)
Another good news is the fact that now I am able to convert both MLV & RAW files with Full color matrice 6D/70D by using @Danne's latest cr2hdr-r (http://www.magiclantern.fm/forum/index.php?topic=13512.msg146326#msg146326cr2hdr-r) which works well on Mac's.
Will still need to investigate further on 70D111B Dual-ISO MLV/RAW though...
@Levas:
can you run mlv_dump with parameter -v and upload the dump?
or can you put the MLV file in my btsync folder with key AUUKDJIEUOPLOPFN5NECA2NEVCLRHPHFO?
@DeafEyeJedi:
mlv_dump didnt produce clean DNG files?
Not for Dual-ISO MLV/RAW files but for regular MLV/RAW files it works fine from a 70D111B w latest nightly.
Did you put up a dual iso sample in the folder? I could check later.
Yes, I believe I did.
Deafeye. Could you retest a non corrupted dual iso mlv/raw and extract with mlv_dump, raw2dng? Or upload a new sample file, dual iso from 70D cam. Again, check that it isn,t any corruption issue.
What are exactly the corruption issues do you recall from 70D dual ISO files? Because all of them comes out the same weird colors w flickering effect.
I'll do another new dual ISO MLV/RAW tonight for you...
Thanks.
Downloaded the file M01-1819.MLV and the file itself shows corruption. It is not mlv_dump that is the problem. Maybe share a link to the file here or in the 70D thread for the experts to look at?
@Geggo
Used the -v option, here's the output:
mlv_dump -o averaged.mlv -a -v /Users/wij/Documents/TEST/72160000.MLV
MLV Dumper v1.0
-----------------
Mode of operation:
- Input MLV file: '/Users/wij/Documents/TEST/72160000.MLV'
- Verbose messages
- Rewrite MLV
- Output only one frame with averaged pixel values
- Output into 'averaged.mlv'
File /Users/wij/Documents/TEST/72160000.MLV opened
File /Users/wij/Documents/TEST/72160000.M00 not existing.
Processing...
File Header (MLVI)
Size : 0x00000034
Ver : v2.0
GUID : 10242427998058437803
FPS : 0.500000
File : 0 / 0
Frames Video: 50
Frames Audio: 0
Block: RAWI
Offset: 0x00000034
Size: 180
Time: 500.314000 ms
Res: 5568x3723
raw_info:
api_version 0x00000001
height 3723
width 5568
pitch 9744
frame_size 0x02298AB0
bits_per_pixel 14
black_level 2047
white_level 14783
active_area.y1 52
active_area.x1 72
active_area.y2 3723
active_area.x2 5568
exposure_bias 0, 0
cfa_pattern 0x02010100
calibration_ill 1
Block: IDNT
Offset: 0x000000e8
Size: 84
Time: 503.485000 ms
Camera Name: 'Canon EOS 6D'
Camera Serial: '1354ACF6DB'
Camera Model: 0x80000302
Block: WBAL
Offset: 0x0000013c
Size: 44
Time: 503.574000 ms
Mode: 2
Kelvin: 2500
Gain R: 496
Gain G: 1024
Gain B: 624
Shift GM: -1
Shift BA: 0
Block: STYL
Offset: 0x00000168
Size: 52
Time: 503.585000 ms
picStyle: 132
contrast: 0
sharpness: 0
saturation: 0
colortone: 0
Block: RTCI
Offset: 0x0000019c
Size: 44
Time: 513.114000 ms
Date: 27.04.2015
Time: 14:33:55 (GMT+0)
Zone: ''
Day of week: 1
Day of year: 116
Daylight s.: 0
Block: EXPO
Offset: 0x000001c8
Size: 40
Time: 513.761000 ms
ISO Mode: 0
ISO: 100
ISO Analog: 72
ISO DGain: 0/1024 EV
Shutter: 445000 microseconds (1/2.25)
Block: LENS
Offset: 0x000001f0
Size: 96
Time: 513.785000 ms
Name: 'EF35mm f/2 IS USM'
Serial: ''
Focal Len: 35 mm
Focus Dist: 65535 mm
Aperture: f/11.30
IS Mode: 0
AF Mode: 3
Lens ID: 0x000000F9
Flags: 0x00000000
Block: VIDF
Offset: 0x00000250
Size: 36276944
Time: 525.316000 ms
Frame: #0000
Crop: 0x0
Pan: 0x0
Space: 0
Segmentation fault: 11
@g3gg0:
Here's the link to 70D files including Dual-ISO MLV/RAW from 70D111B (April 23)...
https://mega.co.nz/#F!6hchkCbD!zScVqL68khdbY6MylpzhNg
@levas:
fixed. try the latest version (http://magiclantern.fm/modules/modules/mlv_dump.zip/mlv_dump.zip):
@DeafEyeJedi:
i think i dont understand the problem.
so mlv_dump creates buggy matrices for dual iso mlv files?
Regular MLV/RAW seems responding to mlv_dump well when using @Danne's cr2hdr-r latest beta but then when I record Dual-ISO RAW/MLV videos then it shows all kinds of strange pinkish effects as you can see in these files I uploaded.
MLV_dump won't respond to those files maybe because it's already corrupted from inside camera while recording?
Just thought it was worth noting... That's all.
However, Dual-ISO CR2's from 70D seems to respond well but just not the videos.
Just wanted to share that with you since I wasn't sure of @A1ex hasn't updated this binary yet for the 70D files.
@g3gg0
Averaging with the newest MLV_dump works! :D
Multiple Full-res-silent-pictures -> Recording in MLV container -> averaging out to one single photo with MLV_dump = The most extreme and variable ND filters one can have 8) :D
Edit: oh yeah I forgot something, THANKS
For those who are curious or in need of files (Dual-ISO RAW, MLV, CR2, MLV/RAW, FRSP DNG/MLV) from certain models such as 70D, 5D3 & the M... Perhaps to help speed up the process on getting the binaries up to date.
Or just want to play with certain files for testing/practice purposes.
https://mega.co.nz/#F!DokxSIhC!q5iOd_rIQTRbCXA51Kyz5Q
Happy Updates!
Is it possible to specify output directory (not output file) for mlv_dump.exe?
I have gigabytes of mlv files (silent picture time lapse) that I want to export to DNG and then to Lightroom. Currently I first copy files from CF card to hard drive, then export to DNG and then to Lightroom.
It would be nice if I could point mlv_dump.exe to mlv file on CF card and specify output directory for DNG files on my hard drive.
However, I was not able to find such feature in documentation. Does it exist?
I tried something like this:
mlv_dump --dng outdir in.mlv
But it does not work and complains "Error: Failed to open file outdir"
mlv_dump --dng -o /path/to/dir/fileprefix_ in.mlv
it will automatically append the frame number
I,m really curious about this. According to below user he claims white balance is correctly translated from windows version of mlv_dump(not sure which version)
user mothaibaphoto
Quote...mlv_dump produces dng sequence with correct WB from camera.
http://www.magiclantern.fm/forum/index.php?topic=13152.msg149992#msg149992
When I run mlv_dump for mac(latest 2 may upload) dng files still have hard coded as shot neutral values?
Could this windows claim be confirmed by somebody?
Thanks
*Update
Correction here regarding mlv_dump white balance
http://www.magiclantern.fm/forum/index.php?topic=13152.msg150044#msg150044
Hi, I've done a quick search but I didn't find any answer, I started shooting MLV with a 5D3 (I used raw on 70D or 5D2) : is it normal than I have vertical lines when shooting ? Can I remove them ? It's quite disturbing but I don't see the in MLrawviewer.
Also, is MLV good for something like 48fps@maxrespossible-videos ? or should I go back to RAW just for that ? (I understand RAW had better writing speeds, I have 2 Lexar 64go 1066x)
Thanks to you all
The vertical lines are known issue. Some programs have a remove stripes function to remove them. I have suspected this issue occurs when you film raw in non native in between iso's. I haven't had this problem since only using "true" iso's.
100, 200, 400, 800, 1600 and so on. I set the camera iso setting's in canon menu so I can only use these when shooting raw.
Haven't confirmed this theory yet though.
Note you can also get vertical lines if filming at very high iso's...
Averaging in MLV_dump in combination with FRSP is a great feature.
4 frames = 1EV more into the shadows.
16 frames = 2EV more into the shadows.
With this we do not need HDR (bracketing), we get smooth movement of water or clouds.
I would like to reduce banding/FPN (vertical stripes).
I wonder how to fix it, maybe averaging function should median optical black (OB) area?
Quote from: Greg on August 12, 2015, 05:08:36 PM
I would like to reduce banding/FPN (vertical stripes).
I wonder how to fix it, maybe averaging function should median optical black (OB) area?
Capture a 'dark frame' (put lens cap on). Then subtract that .mlv from your real .mlv (there's an option to do this in mlv_dump, I think it's -s).
mlv_dump -o o.mlv -s dark.MLV light.MLV
MLV Dumper v1.0
-----------------
Mode of operation:
- Input MLV file: 'light.MLV'
- Rewrite MLV
- Subtract reference frame 'dark.MLV' from single images
- Output into 'o.mlv'
File light.MLV opened
File light.M00 not existing.
[ERROR] File 'dark.MLV' ends in the middle of a block
[ERROR] Failed to load subtract frame (4)
:o
can you provide me the dark.mlv?
The original file is 3.6GB
I should create a new single frame, or averaged this?
In mlv_dump error will be the same.
the new single frame.
can you run the mlv_dump run that produces the single frame with parameter '-v' and upload the log?
confirmed
can you try this one?
http://www.magiclantern.fm/modules/modules/mlv_dump.zip/mlv_dump.zip
(win32/linux only fixed yet)
WOW vertical stripes gone, it works.
The dark frame subtract hot pixels (works well in the shadows).
In highlights create a cold pixels.
I think the ideal solution is to average/median vertical lines in the dark frame.
redownload
--avg-vertical [DARKFRAME ONLY] average the resulting frame in vertical direction, so we will extract vertical banding
--avg-horizontal [DARKFRAME ONLY] average the resulting frame in horizontal direction, so we will extract horizontal banding
and please test by adding one of them along with option -a :)
Hello everybody, and thank you for your amazing works!!!
I was wondering if it was possible to re-create a MLV video from its DNG files.
(A MLV i shot is not working properly, but i have the whole DNG sequence and would love to be able to create a new MLV out of it.)
Thank you in advance :)
So you have DNG and want to go back to MLV to create DNG?
Yes, that is the idea :)
For having 1 single file and playing with Mlrawviewer. When and if needed i can create DNGs again.
I thought of this to. Could be a handy container for mlvfs. Could even open up other dng relate stuff from other cameras.
To be able to do this with compressed DNGs would be ideal. The compressed size but in mlv container = dream come true! ;-)
Quote from: Markus on December 20, 2015, 05:45:55 PM
The compressed size but in mlv container = dream come true! ;-)
The dream is already true see the OP:
Quote
mlv_dump
- used for debugging and converting .mlv files
- can dump .mlv to legacy .raw + .wav files
- can dump .mlv to .dng + .wav
- can compress and decompress frames using LZMA
- convert bit depth (any depth in range from 1 to 16 bits)
compression:
you can get a data reduction of ~60% with 12 bit files.
downconverting to 8 bits gives you about 90% data reduction.
this feature is for archiving your footage.
LZMA is kinda slow and doesn't compress as well, but LJ92 is superfast:
https://bitbucket.org/hudson/magic-lantern/branch/mlv_rec_lj92
I don't really know why this hasn't been merged, but I have implemented it for MLVFS.
So you can use this to compress stuff:
https://bitbucket.org/dmilligan/magic-lantern/branch/ml_dng_lj92 or mac binary (https://bitbucket.org/dmilligan/mlvfs/downloads/CompressMLV.dmg)
and MLVFS will decompress it on the fly (in realtime, in if you are limited by HDD speed, using this compression can actually make MLVFS faster).
MLVFS will also decompress LZMA compressed MLV files, but it's excruciatingly slow. MLVFS does not support the frame diff option however (MLVFS needs to provide "random access" to frames).
Awesome! I may be misunderstanding the intended workflow, but is the attached mac binary used on an MLV file straight from the camera to compress it using the aforementioned LJ92? I received an immediate notification that the compression is complete and the resulting file is zero bytes. I extracted mlv_dump and tried running it from the command line, but received [ERROR] Missing file header. I tried using the normal mlv_dump.osx, the one included in the OP, and it seemed to work fine. Am I using your version on an incorrect target file? Thank you!
Full output:
MacDesktops-iMac:Test MLV Folder macdesktop$ ./mlv_dump -c -v -o test.MLV M30-0950.MLV
MLV Dumper v1.0
-----------------
Mode of operation:
- Input MLV file: 'M30-0950.MLV'
- Verbose messages
- Rewrite MLV
- Compress frame data
- Output into 'test.MLV'
File M30-0950.MLV opened
File M30-0950.M00 not existing.
File M30-0950.IDX opened (XREF)
XREF table contains 1484 entries
Processing...
[ERROR] Missing file header
Processed 0 video frames
[ERROR] Failed to rewrite header in .MLV file
Done
Quote from: dmilligan on January 11, 2016, 06:15:54 PM
The dream is already true see the OP:
LZMA is kinda slow and doesn't compress as well, but LJ92 is superfast
Does this mean it is possible to use the LJ92 compression for mlv files? With mlv dump?
dmilligan can you make a pull request that i can merge?
Sure thing :)
@dmilligan:
I can't understand why brilliant Advanced Intervalometer collecting dust buried deep in the forum, while less practice (IMHO) things like lua and compression gets into builds.
Quote from: g3gg0 on January 11, 2016, 11:40:32 PM
dmilligan can you make a pull request that i can merge?
Quote from: dmilligan on January 12, 2016, 01:25:02 AM
Sure thing :)
Well... the merge isn't clean since so much has changed, so it will take some more time than I thought. Also might want to wait to get dng.mo merged first since it has changes for mlv_dump too, and currently it's closer to being ready.
The only thing I really changed from your branch was to support both LZMA and LJ92, that way people who may have already used LZMA can still convert their files with the most recent version.
Quote from: mothaibaphoto on January 12, 2016, 06:08:05 AM
@dmilligan:
I can't understand why brilliant Advanced Intervalometer collecting dust buried deep in the forum, while less practice (IMHO) things like lua and compression gets into builds.
+1
Quote from: mothaibaphoto on January 12, 2016, 06:08:05 AM
@dmilligan:
I can't understand why brilliant Advanced Intervalometer collecting dust buried deep in the forum, while less practice (IMHO) things like lua and compression gets into builds.
a) OT
b) where is the pull request?
@g3gg0
I wonder if it,s possible to create a HDR_average mode for mlv_dump? I recently started experimenting with an average filter in ffmpeg which blends every other picture thus delivering HDR processed files without the need for enfuse or other HDR applications.
http://www.magiclantern.fm/forum/index.php?topic=13512.msg160592#msg160592
Now how about a setting to call for in mlv_dump which would do the same? This would be very handy in conjunction with MLVFS or in ACR with slowmo footage(48, 50, 60 fps) creating averaged HDR dng files.
What do you think?
I think this is an excellent idea if it's at all possible which I believe it is.
Nice work, Danne! [emoji2]
so simply average two consecutive frames?
Yes, so it seems. Does wonders in higher fps :)
While it probably "works", simply averaging two frames seems like a suboptimal way to do HDR merging. The more noisy shadows of the darker exposure will be given more "weight" making the resulting image more noisy than it could be (hence reducing DR) and the highlights will have a strange "elbow" in their rolloff at the point where the brighter exposure starts clipping.
A better way would be to use the method employed by CeroNoise: http://www.magiclantern.fm/forum/index.php?topic=9581.0
that would simplify things, wont it?
just export .dng from mlv_dump and process them in ceronoise?
Thanks for answers.
QuoteWhile it probably "works", simply averaging two frames seems like a suboptimal way to do HDR merging.
You might be right in theory but my tests shows very clean and professional results when using the tblend average filter with FFmpeg. Enfuse tends to flicker and getting the settings right for blacks and whites is a bit of a problem. Those problems are non existent with the tblend filter.
Question is if mlv_dump averaging does something different from the tblend filter?
Quote
that would simplify things, wont it?
just export .dng from mlv_dump and process them in ceronoise?
The biggest hassle with HDR is the post production marathon. A function in mlv_dump would simplify a lot. Is the averaging on the fly right now in mlv_dump? If so, HDR filming won,t take longer than working with regular files.
Could I test this somehow on a HDR file?
Tried a comparison by running HDR and camera set to 1fps and managed to record a mlv containing only 2 dng files. Seems to be a different way of averaging in mlv_dump and FFmpeg. Maybe overkill to implement HDR in mlv_dump right now. Higher priority would be other things like pixel_fix and dng modules. Anyway, who knows :)
Original file
https://drive.google.com/file/d/0B4tCJMlOYfirbFhoUzZHYnpMd0U/view?usp=sharing
FFmpeg
(http://s9.postimg.org/3tndmq08v/Screen_Shot_2016_01_22_at_18_25_33.png)
mlv_dump
(http://s9.postimg.org/4hw85nyyn/Screen_Shot_2016_01_22_at_18_26_02.png)
@g3gg0
Hi. I,m about to implement a nice automated workflow in MLP that will utilize the average darkframe function in mlv_dump but I encounter problems.
[ERROR] File '/Users/Daniel/Desktop/Myfiles/hej.MLV' ends in the middle of a block
[ERROR] Failed to load subtract frame (4)
The problem and the answer to why it,s not working seems to be here.
http://www.magiclantern.fm/forum/index.php?topic=7122.msg152952#msg152952
Is it possible to have this fixed also for the mac binary? Would be so nice to get this working.
Original mlv file
https://drive.google.com/file/d/0B4tCJMlOYfirTGR2S21IWFlLd1U/view?usp=sharing
The same file averaged
https://drive.google.com/file/d/0B4tCJMlOYfirTFJTUFNLRTlOblk/view?usp=sharing
*update
Got my hands on a version from 31 aug 2014 which seems to work without giving the error codes. Hopefully this can be fixed in latest version too.
https://drive.google.com/file/d/0B4tCJMlOYfirRXhZY045Z0V3eW8/view?usp=sharing
Maybe not the right way to handle the problem but I bypassed the error message by changing the ERR_FILE to ERR_OK in mlv_dump.c and now it seems to work.
http://www.magiclantern.fm/modules/modules/mlv_dump.zip/mlv_dump.zip
latest version
Great! Working now :D
guess its a FRSP file?
Actually I was planning on using it also for regular movie files as well as with frsp. Is it even effective to add a darkframe together with let,s say mlv 24fps sequences?
sure. just add as many you need/want to get a good enough dark frame.
Holy Darkness ... This is incredible news @g3gg0 and Thanks for steam rolling on this fix @Danne!
Perhaps should we call you 'Zero Dark MLP'?
Hi g3gg0. Here,s my latest tribute to mlv_dump :). Tried a little footage and the averaging does wonders on higher isos. Banding and hot pixels gone.
http://www.magiclantern.fm/forum/index.php?topic=13512.msg164871#msg164871
hi guys long time that i used ML but where do i download MLV_snd for my Canon 7D thanks guys
Hi!
Want to report a problem when selecting chroma smooth in mlv_dump for mac os x. When selecting for instance
mlv_dump -f 1 --dng --cs5x5 -o hej M04-0025.MLV
It results in segmentation fault 11. Not sure if other platforms works ok?
MLV Dumper v1.0
-----------------
Mode of operation:
- Input MLV file: '/Users/Daniel/Desktop/Myfiles/M04-0025.MLV'
- Convert to DNG frames
- Output into 'hej'
File /Users/Daniel/Desktop/Myfiles/M04-0025.MLV opened
File /Users/Daniel/Desktop/Myfiles/M04-0025.M00 not existing.
Processing...
Vertical stripes correction:
1.000 1.000 0.998 1.008 0.996 0.996 0.995 0.995
Segmentation fault: 11
There,s a whole bunch of warnings when compiling mlv_dump. An excerpt.
Last login: Sat May 21 13:42:53 on ttys001
Daniels-macbook-pro:mlv_rec Daniel$ make mlv_dump
[ HOST_CC ] mlv_dump.host.o
mlv_dump.c: In function 'chroma_smooth':
mlv_dump.c:946:31: warning: passing argument 1 of 'chroma_smooth_2x2' from incompatible pointer type [-Wincompatible-pointer-types]
chroma_smooth_2x2(aux, aux2, raw2ev, ev2raw);
^
In file included from mlv_dump.c:894:0:
../dual_iso/chroma_smooth.c:2:28: note: expected 'uint32_t * {aka unsigned int *}' but argument is of type 'short unsigned int *'
#define CHROMA_SMOOTH_FUNC chroma_smooth_2x2
^
../dual_iso/chroma_smooth.c:18:13: note: in expansion of macro 'CHROMA_SMOOTH_FUNC'
static void CHROMA_SMOOTH_FUNC(uint32_t * inp, uint32_t * out, int* raw2ev, int* ev2raw)
^
mlv_dump.c:946:36: warning: passing argument 2 of 'chroma_smooth_2x2' from incompatible pointer type [-Wincompatible-pointer-types]
chroma_smooth_2x2(aux, aux2, raw2ev, ev2raw);
^
In file included from mlv_dump.c:894:0:
../dual_iso/chroma_smooth.c:2:28: note: expected 'uint32_t * {aka unsigned int *}' but argument is of type 'short unsigned int *'
#define CHROMA_SMOOTH_FUNC chroma_smooth_2x2
^
../dual_iso/chroma_smooth.c:18:13: note: in expansion of macro 'CHROMA_SMOOTH_FUNC'
static void CHROMA_SMOOTH_FUNC(uint32_t * inp, uint32_t * out, int* raw2ev, int* ev2raw)
^
mlv_dump.c:949:31: warning: passing argument 1 of 'chroma_smooth_3x3' from incompatible pointer type [-Wincompatible-pointer-types]
chroma_smooth_3x3(aux, aux2, raw2ev, ev2raw);
^
In file included from mlv_dump.c:898:0:
../dual_iso/chroma_smooth.c:7:28: note: expected 'uint32_t * {aka unsigned int *}' but argument is of type 'short unsigned int *'
#define CHROMA_SMOOTH_FUNC chroma_smooth_3x3
^
../dual_iso/chroma_smooth.c:18:13: note: in expansion of macro 'CHROMA_SMOOTH_FUNC'
static void CHROMA_SMOOTH_FUNC(uint32_t * inp, uint32_t * out, int* raw2ev, int* ev2raw)
^
mlv_dump.c:949:36: warning: passing argument 2 of 'chroma_smooth_3x3' from incompatible pointer type [-Wincompatible-pointer-types]
chroma_smooth_3x3(aux, aux2, raw2ev, ev2raw);
^
In file included from mlv_dump.c:898:0:
../dual_iso/chroma_smooth.c:7:28: note: expected 'uint32_t * {aka unsigned int *}' but argument is of type 'short unsigned int *'
#define CHROMA_SMOOTH_FUNC chroma_smooth_3x3
^
../dual_iso/chroma_smooth.c:18:13: note: in expansion of macro 'CHROMA_SMOOTH_FUNC'
static void CHROMA_SMOOTH_FUNC(uint32_t * inp, uint32_t * out, int* raw2ev, int* ev2raw)
^
mlv_dump.c:952:31: warning: passing argument 1 of 'chroma_smooth_5x5' from incompatible pointer type [-Wincompatible-pointer-types]
chroma_smooth_5x5(aux, aux2, raw2ev, ev2raw);
^
In file included from mlv_dump.c:902:0:
../dual_iso/chroma_smooth.c:12:28: note: expected 'uint32_t * {aka unsigned int *}' but argument is of type 'short unsigned int *'
#define CHROMA_SMOOTH_FUNC chroma_smooth_5x5
^
../dual_iso/chroma_smooth.c:18:13: note: in expansion of macro 'CHROMA_SMOOTH_FUNC'
static void CHROMA_SMOOTH_FUNC(uint32_t * inp, uint32_t * out, int* raw2ev, int* ev2raw)
^
mlv_dump.c:952:36: warning: passing argument 2 of 'chroma_smooth_5x5' from incompatible pointer type [-Wincompatible-pointer-types]
chroma_smooth_5x5(aux, aux2, raw2ev, ev2raw);
^
In file included from mlv_dump.c:902:0:
../dual_iso/chroma_smooth.c:12:28: note: expected 'uint32_t * {aka unsigned int *}' but argument is of type 'short unsigned int *'
#define CHROMA_SMOOTH_FUNC chroma_smooth_5x5
^
../dual_iso/chroma_smooth.c:18:13: note: in expansion of macro 'CHROMA_SMOOTH_FUNC'
static void CHROMA_SMOOTH_FUNC(uint32_t * inp, uint32_t * out, int* raw2ev, int* ev2raw)
^
[ HOST_CC ] ../../src/chdk-dng.host.o
../../src/chdk-dng.c:219:9: warning: padding struct to align 'exp_program' [-Wpadded]
int exp_program;
^
../../src/chdk-dng.c: In function 'write_dng':
../../src/chdk-dng.c:727:60: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
if (write(fd, dng_header_buf, dng_header_buf_size) != dng_header_buf_size) return 0;
^
../../src/chdk-dng.c:728:68: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
if (write(fd, thumbnail_buf, dng_th_width*dng_th_height*3) != dng_th_width*dng_th_height*3) return 0;
^
../../src/chdk-dng.c:731:68: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
if (write(fd, UNCACHEABLE(rawadr), camera_sensor.raw
what kind of file is it?
can you dump it using -v ?
Hi g3gg0!
It happens on any MLV file I test with. Confirmed the same behaviour from dfort and deafeyejedi and your included binary in first post. Here is a -v output.
Last login: Sun May 22 20:43:53 on ttys005
Daniels-MBP:~ Daniel$ /Users/Daniel/Desktop/Myfiles/mlv_dump -f 1 --cs5x5 --dng -v -o hej /Users/Daniel/Desktop/Myfiles/M04-0025.MLV
MLV Dumper v1.0
-----------------
Mode of operation:
- Input MLV file: '/Users/Daniel/Desktop/Myfiles/M04-0025.MLV'
- Verbose messages
- Convert to DNG frames
- Output into 'hej'
File /Users/Daniel/Desktop/Myfiles/M04-0025.MLV opened
File /Users/Daniel/Desktop/Myfiles/M04-0025.M00 not existing.
Processing...
File Header (MLVI)
Size : 0x00000034
Ver : v2.0
GUID : 11410529140746060919
FPS : 23.976000
File : 0 / 1
Frames Video: 450
Frames Audio: 95
Block: RAWI
Offset: 0x00000034
Size: 180
Time: 205.985000 ms
Res: 1920x1080
raw_info:
api_version 0x00000001
height 1318
width 2080
pitch 3640
frame_size 0x00493450
bits_per_pixel 14
black_level 2047
white_level 15000
active_area.y1 28
active_area.x1 146
active_area.y2 1318
active_area.x2 2078
exposure_bias 0, 0
cfa_pattern 0x02010100
calibration_ill 1
Block: INFO
Offset: 0x000000e8
Size: 16
Time: 210.371000 ms
Block: RTCI
Offset: 0x000000f8
Size: 44
Time: 0.001000 ms
Date: 04.12.2015
Time: 00:23:59 (GMT+0)
Zone: ''
Day of week: 5
Day of year: 337
Daylight s.: 0
Block: EXPO
Offset: 0x00000124
Size: 40
Time: 0.002000 ms
ISO Mode: 0
ISO: 3200
ISO Analog: 112
ISO DGain: 0/1024 EV
Shutter: 29994 microseconds (1/33.34)
Block: LENS
Offset: 0x0000014c
Size: 96
Time: 0.003000 ms
Name: '20mm F1.4 DG HSM | Art 015'
Serial: ''
Focal Len: 20 mm
Focus Dist: 51 mm
Aperture: f/0.00
IS Mode: 0
AF Mode: 0
Lens ID: 0x000000FA
Flags: 0x00000000
Block: IDNT
Offset: 0x000001ac
Size: 84
Time: 0.004000 ms
Camera Name: 'Canon EOS 5D Mark III'
Camera Serial: 'C587A29EA'
Camera Model: 0x80000285
Block: WBAL
Offset: 0x00000200
Size: 44
Time: 0.005000 ms
Mode: 9
Kelvin: 2500
Gain R: 608
Gain G: 1024
Gain B: 400
Shift GM: 0
Shift BA: 0
Block: STYL
Offset: 0x0000022c
Size: 52
Time: 0.006000 ms
picStyle: 129
contrast: 0
sharpness: 0
saturation: 0
colortone: 0
Block: WAVI
Offset: 0x00000260
Size: 32
Time: 507.541000 ms
wav_info:
format 1
channels 2
samplingRate 48000
bytesPerSecond 192000
blockAlign 4
bitsPerSample 16
Block: RTCI
Offset: 0x00000280
Size: 48
Time: 508.823000 ms
Date: 04.12.2015
Time: 00:23:59 (GMT+0)
Zone: ''
Day of week: 5
Day of year: 337
Daylight s.: 0
Block: EXPO
Offset: 0x000002b0
Size: 48
Time: 510.421000 ms
ISO Mode: 0
ISO: 3200
ISO Analog: 112
ISO DGain: 0/1024 EV
Shutter: 29994 microseconds (1/33.34)
Block: LENS
Offset: 0x000002e0
Size: 96
Time: 510.457000 ms
Name: '20mm F1.4 DG HSM | Art 015'
Serial: ''
Focal Len: 20 mm
Focus Dist: 51 mm
Aperture: f/0.00
IS Mode: 0
AF Mode: 0
Lens ID: 0x000000FA
Flags: 0x00000000
Block: WBAL
Offset: 0x00000340
Size: 48
Time: 510.473000 ms
Mode: 9
Kelvin: 2500
Gain R: 608
Gain G: 1024
Gain B: 400
Shift GM: 0
Shift BA: 0
Block: VIDF
Offset: 0x00000370
Size: 3632624
Time: 529.645000 ms
Frame: #0000
Crop: 152x136
Pan: 146x133
Space: 3792
Vertical stripes correction:
1.000 1.000 0.998 1.008 0.996 0.996 0.995 0.995
Segmentation fault: 11
Daniels-MBP:~ Daniel$
Anybody see corruption like this? I've been getting a couple of frames every recording like this. I'm using the December 5D3.123 build.
Thanks.
(https://www.dropbox.com/s/o1w36fzmptnbr4l/M18-1554_frame_004964.jpg?raw=1)
Use August 7th 2014 for 1.2.3
Use anything post 11th April 2016 for 1.1.3
Debug logging causes the pink corrupt frames (it's turned off in those builds).
QuoteUse August 7th 2014 for 1.2.3
Use anything post 11th April 2016 for 1.1.3
Debug logging causes the pink corrupt frames (it's turned off in those builds).
Very useful info, and I think it should be on the front page somewhere!
Many user don't know this and will be put off after seeing pink frames using the latest 1.2.3 nightly.
or can we remove the offending builds and put them in an archive somewhere?
This would help many users who encounter this problem
can someone please compile a new build for 1.2.3 with logging disabled :)
Edit: or is it possible to use the newest mlv-rec.mo out of the 1.1.3 nightly, which does not have the logging enabled with the latest 1.2.3 nightly?
you could use the mlv module from August 7th 2014 for 1.2.3 in the December build.
but yea a new build based on more recent branch would be welcome
Quote from: Lars Steenhoff on July 22, 2016, 02:18:17 PM
you could use the mlv module from August 7th 2014 for 1.2.3 in the December build.
but yea a new build based on more recent branch would be welcome
the issue with that module is i think it doesn't allow for a single raw clip longer than 40gb / ~8.5 mins
Greetingz ...
None of the players and converters work for me on a mac (Yosemite 10.10.5)
Any suggestions? I would love to test the quality of the .mlv files and see how they can be edited?
Kind Regards, OM
Have you tried MLVFS and MLP?
Small update to mlv_dump:
- fix cold pixel processing when saving only a subset of the frames
- cold pixel fix is now enabled by default
- fine-tuned vertical stripe fix + option to disable it
- option to scan every single frame for cold pixels (slow, but I remember some reports with moving cold pixels, for example with very bright specular highlights)
- experimental flat field (gain) correction, details here (http://www.magiclantern.fm/forum/index.php?topic=17795.msg171607#msg171607)
- minor fix to dark frame subtraction mode (proper white level and keep sub-black values)
--no-fixcp do not fix cold pixels
--fixcp2 fix non-static (moving) cold pixels (slow)
--no-stripes do not fix vertical stripes in highlights
-t mlv_file use the reference frame in given file as flat field (gain correction)
These updates were tested here (http://www.magiclantern.fm/forum/index.php?topic=17795.0).
Windows build: https://builds.magiclantern.fm/jenkins/view/Other%20tasks/job/mlv_dump/
Will ask g3gg0 to update the first post and maybe set up a Mac build.
Get some weird artifacts processing with chr2hdr DualISO dng files extracted from FRSP MLV files with this last version. Previous version(with --fixcp2 and --fix-stripes) works pretty well. On slow connection now maybe later can upload examples.
Tested developing a dualiso file. Some bug with wb(dng corruption). Dfort found this as well yesterday.
Latest mlv_dump(wrong wb)
unprocessed dng
https://drive.google.com/file/d/0B4tCJMlOYfirekEtV05PRHpzMHM/view?usp=sharing
Comparison dng(no surprises with wb) used an older build of mlv_dump
unprocessed dng
https://drive.google.com/file/d/0B4tCJMlOYfirRk5POWM4UXJJX1U/view?usp=sharing
Original MLV(dualiso) shortened(7mb)
https://drive.google.com/file/d/0B4tCJMlOYfirdTNKRldId1oxRXM/view?usp=sharing
mlv_dump(latest) compiled for mac.
https://drive.google.com/file/d/0B4tCJMlOYfirVVR1VFNOSExPLUk/view?usp=sharing
*update.
Stripes correction causing corruption in dng files.
Not sure why this is being discussed in the MLV Lite topic but for what it is worth here are my findings.
bouncyball has a version of mlv_dump that fixes the wb issues that Danne pointed out. His version is using dmilligan's dng module.
https://bitbucket.org/bouncyball/ml_dng-branch/branch/ml_dng
I thought it would be a good project to add the latest changes to this version of mlv_dump and submit a pull request to the unified branch. This way we can start migrating to the modular dng codebase a little bit at a time instead of all at once. raw2dng is a dependency of mlv_dump but it seems to be broken (wb issues) in the ml_dng branch.
https://bitbucket.org/hudson/magic-lantern/pull-requests/603/dng-module/diff
Now for the strange part. I tried to update raw2dng with bouncyball's latest changes hoping that would fix the wb issues like it did with mlv_dump but that triggered the wb issue in dual iso in mlv_dump that Danne is pointing out. Reverting back to the original (broken) raw2dng and mlv_dump is fine. I tried reverting bouncyball's changes in the unified branch but it had no effect on the mlv_dump wb issue.
So now I've got a working mlv_dump and a broken raw2dng in my mlv_dng branch and a broken mlv_dump and working raw2dng in the unified branch but there's no way to mix and match the two codebases.
Is there a way to break the dependency between mlv_dump and raw2dng? That is beyond my coding skills.
[EDIT] Here's a link to the mlv_dump I'm working on if anyone want to test it: https://bitbucket.org/daniel_fort/magic-lantern/downloads/ml_dng-test_2016-10-12.zip
And a link to my ml_dng work in progress: https://bitbucket.org/daniel_fort/magic-lantern/branch/ml_dng-mlv_dump
I can't really pin down the white balance "issue" you are referring to. (I see nothing recent about wb in the MLV lite thread).
I'm also a bit confused about your statements about a broken raw2dng, since RAW format does not record any white balance information, there's no possible way raw2dng could do anything other than simply hard code some arbitrary white balance.
Quote from: dmilligan on October 13, 2016, 12:17:22 AM
...I'm also a bit confused about your statements about a broken raw2dng...
Sorry about the confusion, this is not specific to MLV Lite and my comments should probably be moved to another thread.
There seems to be a long standing issue with mlv_dump as noted in your comment on this post (http://magiclantern.fm/forum/index.php?topic=15948.msg154844#msg154844):
Quote from: dmilligan on October 01, 2015, 01:00:13 PM
Once the DNG module PR I have open is merged, then mlv_dump will also handle WB correctly.
So it turns out that a few of us are using mlv_dump from your ml_dng branch and it indeed handles white balance properly. I take it you've been busy on other projects so I've been working on trying to move this forward so that everyone will benefit.
However, as you know mlv_dump references raw2dng when compiling and even though mlv_dump is fine, there's a problem with raw2dng in your ml_dng branch.
No point continuing the discussion on this thread. I'll post some more details in the Update mlv_dump (http://magiclantern.fm/forum/index.php?topic=17992) topic though come to think of it that also somewhat off topic--I'm having one of those days. ???
[EDIT] Just realized that I mistakenly thought this was the MLV Lite topic. Anyway, I was still off topic bringing up raw2dng.
To continue my previous post:
I'm not about WB. DNG processed with two last builds(Aug 28, 2016 11:09 PM and Sep 5, 2016 10:19 PM) have the same WB for me. But the very last build(Sep 5, 2016 10:19 PM) produce files incompatible with cr2hdr - i get corrupted image with artifacts. If I run that last build with --no-stripes option - everything is ok(except stripes :) ). Therefore, its something wrong with very last tunes of stripe fixes.
Example file
https://www.dropbox.com/s/7fitsoqbrn34v1f/corrupt.mlv?dl=0
@mothaibaphoto - I don't see those issues on your FRSP file. Running mlv_dump -v corrupt.mlv shows lots of extra metadata that doesn't seem necessary but other than that I don't see anything wrong with it. This was processed on cr2hdr with the default options:
(https://c8.staticflickr.com/6/5579/30216825711_e25a4be6f5_z.jpg)
[EDIT] Note that --fix-stripes is no longer an option in mlv_dump.
I can confirm the finding from mothaibaphoto. --no-stripes fixes the issue with the dual iso files posted before.
I can confirm this as well. Looks good to me and this was processed with @Danne's cr2hdr app.
(https://c5.staticflickr.com/6/5655/29675294004_b5d913f724_z.jpg) (https://flic.kr/p/MdiD35)
*edit*
I just notice this and wasn't sure if this was to be expected re: metadata after its been rendered for Dual-ISO process.
(https://c5.staticflickr.com/6/5715/30009096060_4c84095eb0.jpg) (https://flic.kr/p/MHNsKW)
Would this be problematic for Timelapse type of work where certain scripts requires proper tags for exiftool and such?
Deafeyejedi. You tend to forget. MLP is mlv_dump, cr2hdr nothing less. Scripts in a wrapper :)
Ah, that makes sense and Thanks @Danne -- which was why I mentioned that I used your cr2hdr app (not MLP) even tho they both use the same binaries, right?
Another one of the moments I guess... ::)
Where are the artifacts? I might not be looking close enough?
(https://c4.staticflickr.com/9/8571/30226431131_ddb7c8c361_o.jpg)Artifacts (https://flic.kr/p/N41mUk)
@DeafEyeJedi: Whats wrong with that meta and why its matters?
@dfort: Extracting with -f option brings mlv with that frames and metadata for the rest.
It's kinda feature of mlv_dump :)
Ok--able to reproduce it here. The problem is on the current unified branch. Sorry, I've been trying some things out and probably used the wrong version when I ran that first test. If you're on a Mac try this one (https://bitbucket.org/daniel_fort/magic-lantern/downloads/ml_dng-test_2016-10-12.zip) or if you can compile you can pull it from my bitbucket repository (https://bitbucket.org/daniel_fort/magic-lantern/branch/ml_dng-mlv_dump). It uses the ml_dng module from dmilligan and has all the latest from unified in it. You can find more information on this one on the update mlv_dump (http://magiclantern.fm/forum/index.php?topic=17992) topic. This version doesn't show those artifacts though there's something strange going on with the white letters on the road if you look really close. In any case I didn't try any extra option on mlv_dump or cr2hdr so see if I could get rid of them.
(https://c8.staticflickr.com/6/5777/30314097095_016eaa9bba_z.jpg)
By the way, the -f option in mlv_dump is just for extracting frames. You didn't specify which frame to pull but both frames in your MLV file exhibit the same issues.
-f frames frames to save. e.g. '12' saves the first 12 frames, '12-40' saves frames 12 to 40.
@dfort: Update mlv_dump topic is great idea and i have various workarounds for issue i encounter,
but all this has nothing to do with message i want to bring to public:
Last version of vertical stripes correction code incompatible with DualISO.
Quote from: dfort on October 14, 2016, 08:59:26 AM
something strange going on with the white letters on the road if you look really close.
Cromatic Aberrations/Purple Fringing
Quote from: dfort on October 14, 2016, 08:59:26 AM
You didn't specify which frame to pull but both frames in your MLV file exhibit the same issues.
I mean I extracted 2 frames from 300 frames sequence, this why it contains too much metadata.
Something is going on with bitbucket's issue tracker. I posted a bug report about the issues we've been discussing but it isn't showing up on the issues tracker list.
In any case, hope @a1ex and/or @g3gg0 are reading this.
Quote from: mothaibaphoto on October 14, 2016, 09:51:00 AM
@dfort: Update mlv_dump topic is great idea and i have various workarounds for issue i encounter,
but all this has nothing to do with message i want to bring to public:
Last version of vertical stripes correction code incompatible with DualISO.
Cromatic Aberrations/Purple FringingI mean I extracted 2 frames from 300 frames sequence, this why it contains too much metadata.
Updating cr2hdr is also needed. If you process your file with MLVFS or Danne's cr2hdr.app you'll see that the white balance on your shot is 2550K, not even close for a sunny daylight scene.
(https://c4.staticflickr.com/6/5775/30341218955_1c9d3aebb6_z.jpg)
That's the first thing that I saw and thought it was related to this bug report:
https://bitbucket.org/hudson/magic-lantern/issues/2612/wb-issue-with-dual_iso-in-mlv_dump
i don't understand what the problem is.
can someone explain it to me?
The problem in unified version atm is this.
http://www.magiclantern.fm/forum/index.php?topic=7122.msg173295#msg173295
It only affects dual iso files and it seems vertical stripes causing it.
The rest going on. I,d say mainly a call out for implementing ufraw white balance code the way dmilligan has been doing it in mlvfs which Bouncyball took on with ml_dng branch. Quite confusing right now with all fruits at once in the mix...
Quote from: g3gg0 on October 15, 2016, 09:05:57 PM
i don't understand what the problem is.
can someone explain it to me?
The image on this post shows the issue clearly:
http://www.magiclantern.fm/forum/index.php?topic=7122.msg173321#msg173321
uhm, thats what i get. thus my dumb question...
but not sure if the mlv_dump contains all fixes, so would have to recompile.
to fix that issue, just undo this (https://bitbucket.org/hudson/magic-lantern/commits/ab24965b3f4d45e4cba10214bc0061f36b440730) change and it should work again
Thanks @g3gg0 verified the fix works over here.
Another issue, this time one that was reported a while back in this discussion but I can't find a bug report in bitbucket on it. Segfault when applying chroma smoothing (http://www.magiclantern.fm/forum/index.php?topic=7122.msg167481#msg167481).
The fix came up in another discussion with some off topic comments.
http://www.magiclantern.fm/forum/index.php?topic=17955.msg173530#msg173530
Long story short, mlv_dump segfaults when applying chroma smoothing. There is a fix for that but it wasn't explained in the pull request so it hasn't been merged yet.
Nice find @Danne and Thanks for the reminder @dfort!
fixed. see https://bitbucket.org/hudson/magic-lantern/commits/33843737460f1b0323a7b46fc0e81198ea44ab9b
@g3gg0
When you get a chance can you please check out these mlv_dump pull requests? These are also bug fixes.
https://bitbucket.org/hudson/magic-lantern/pull-requests/763/backout-changeset/diff
https://bitbucket.org/hudson/magic-lantern/pull-requests/765/fix-to-extract-only-the-first-frame/diff
Wil mlv implement 10/12 bit mlv raw rec?
@dfort:
first: ok
second: well, alex has some comments about that and i do not exactly know the impact of this patch.
@andy kh:
i am not aware of the progress on this topic. will check.
Quote from: g3gg0 on November 09, 2016, 06:40:50 PM
first: ok
second: well, alex has some comments about that and i do not exactly know the impact of this patch.
Thanks!
Just a suggestion--in your examples on the first post you have:
Quote# will dump the first 123 frames into a new file
./mlv_dump -f 123 -o out.mlv in.mlv
That will actually dump out frames 0 through 123 which are 124 frames. Perhaps it should say:
Quote# will dump frames 0 through 123 into a new file
Note that
./mlv_dump --dng -f 0 in.mlv (or
./mlv_dump --dng -f 0-0 in.mlv) will now extract just frame 0 instead of all of the frames.
Nicely done. How good to see mlv_dump still growing.
what is missing in mlv_rec/mlv_dump for 10 bit support? (aside of post processing)
Re: 10/12-bit -- Could this be related to headers being incomplete? Not sure tho...
Quote from: g3gg0 on November 09, 2016, 10:17:11 PM
what is missing in mlv_rec/mlv_dump for 10 bit support? (aside of post processing)
There is this commit (https://bitbucket.org/hudson/magic-lantern/commits/9b2547bb092e4c3c08265fe0d5948659d3e027db?at=unified) in the raw_video_10bit_12_bit branch so mlv_dump can work with the lower bit depths.
Quotemlv_dump: fix DNG output for lower bit depths
You need to disable all processing, e.g. mlv_dump M00-1234.MLV --dng --no-fixcp --no-stripes
ah thanks. yeah, guess that was a dumb copy paste error :)
I have a small question related to MLV_DUMP, when i extract DNG from mlv_dump i can't import DNG into Premiere Pro CC 2015, but if i use the soft raw2cdng 1.7.9 it is possible.
Is this a known problem?
I use the last mlv_dump (09 dec) made for extract 10 and 12bits MLV/RAW, but i tried with standard 14bits MLV.
I don't especially use Premiere Pro for color grading with DNG, with davinci resolve i don't get any problem, i was just curious about this issue.
Here is a version of mlv_dump which is built upon dmilligan ml-dng code. The code has been worked upon even more by Bouncyball and a little by me. It applies correct white balance and needed cdng tags and also corrects any local camera model tags so acr can read and apply dcp profiles unique to every camera model.
https://bitbucket.org/Dannephoto/ml-dng-dannephoto/downloads/mlv_dump
Code here
https://bitbucket.org/Dannephoto/ml-dng-dannephoto/overview
*oops, windows user. Maybe somebody can compile this for windows.
Very interesting and thanks for reply !
I will wait a Windows build if someone could take the time to compile this.
Here is Bouncyballs downloads area. I think there,s a version of mlv_dump for windows. Have a look. Hopefully working for 10/12bit.
https://bitbucket.org/bouncyball/mlv-tools/downloads
Just tried now and yes it work on Premiere CC 2015 !
But unfortunately does not work for the extraction of MLV 12 and 10bits
Did you specify --no-stripes, and the setting for no cold pixels?
Yes of course.
--dng --no-fixcp --no-stripes
Quote from: Danne on December 24, 2016, 03:10:19 PM
Here is Bouncyballs downloads area. I think there,s a version of mlv_dump for windows. Have a look. Hopefully working for 10/12bit.
https://bitbucket.org/bouncyball/mlv-tools/downloads
Feel we've been down this path before:
From the ml_dng pull request (https://bitbucket.org/hudson/magic-lantern/pull-requests/603/dng-module/diff):
Quote from: g3gg0well, don't get me wrong. but that doesn't help ML development at all.
i just see some fork-off with a patch that says "mlv_dump dng output fixes" for example and a few "global" changes too. and when merging the branch into the main, i am quite sure it contains 64 different stray changes that will not merge nicely.
please just create a fork, incorporate a small change like "fix mlv_dump crash situation" and file a pull request.
i will then accept that pull request and the bug is fixed for everyone, not just the two people that use bouncyball's repository.
It would be helpful if mlv_dump could create valid cdng files that will import into Adobe Premiere but until ml_dng is merged it probably won't happen. (That pull request currently has some conflicts that need to be resolved.)
In the meantime I'd suggest using MLVFS (http://www.magiclantern.fm/forum/index.php?topic=13152.0) which works on Mac, Windows and Linux and it creates valid cdng files that work in Premiere.
Or download my repository and compile a fully working mlv_dump for windows and mac meanwhile. Good to have alternatives...
fixes are in progress.
updated the mlv_dump binary zip (http://www.magiclantern.fm/modules/modules/mlv_dump.zip/mlv_dump.zip) with the 10/12/14 bit branch mlv_dump.
when converting to DNG, it will upconvert frames to 14bpp. should fix issues with lower depths.
Great g3gg0. Tested 10bit files and applying chroma smooth 2x2 is working(otherwise completely corrupted). I also tried darkframe averaging with 10bit files and all seems to work in that regard as well!!. Really great. If anybody wants to test more here is a compiled mlv_dump build with g3gg0 latest inserts.
This means 10/12bit can be treated as any14bit mlv footage just as before.
*g3gg0 already provides the mlv_dump binary. Missed that one.
*I also just updated ml-dng code(used in MLP, cr2hdr.app) with these latest total awesome changes as well. Darkframe averaging with 10bit is a beauty.
how does the mac binary differ from the one i posted?
does it contain the cdng code?
Sorry, no it,s the same as you posted. I,ll remove it. However I updated a ml-dng code version which has a version of mlv_dump with cdng tags and which calculates white balance. I guess that is not in the pipeline for now :)
Thanks again for working the mlv_dump code. Filmed a lot of 10bit material and ran it with darkframe averaging. Footage looks great.
good to hear. does the dark frame really make so much difference?
nice.
I,m building a storage with darkframes. Personally I tend to not shooting exposing to the right when using darkframes, the color noise cleanup is significant. I,ll put up some footage when I have something useful to show. Probably involving a white cat...
Quote from: g3gg0 on December 25, 2016, 10:09:46 PM
...does the dark frame really make so much difference?
Absolutely it does... Kudos to @Danne for pointing this out and thanks to you @g3gg0 for your never ending magic!
At the risk of asking a total newb question, the sampling rate (1x1, 3x3, or 3x5) and bit-depth (10, 12, or 14bit) can't be chosen in-camera at data capture on a 5D2, can it? Like to lower the data rate (potentially opening up higher resolutions) or get smaller file sizes (so cards last longer)?
And yes, I was definitely late to the MLV party (not even fashionably late, but just late-late...)
@-Rogue5- 'sampling rate' can be 'chosen' on 5D2, normal full frame raw is 3x3, crop mode is 1x1, 3x5 isn't available. Also sampling doesn't affect data rate, resolution does. 10 bit and 12 bit can be used on the 5D2, you just need a build of that branch, right now you can record corruption free 10/12bit on this camera only if you use crop mode.
Follow the 10-12bit thread: http://www.magiclantern.fm/forum/index.php?topic=5601.950 (http://www.magiclantern.fm/forum/index.php?topic=5601.950) @dfort is posting builds every now and then
Also watch reddeercity's video here: http://www.magiclantern.fm/forum/index.php?topic=11205.msg177721;topicseen#msg177721 (http://www.magiclantern.fm/forum/index.php?topic=11205.msg177721;topicseen#msg177721)
Quote from: Ilia3101 on January 07, 2017, 12:28:57 AM
@-Rogue5- 'sampling rate' can be 'chosen' on 5D2, normal full frame raw is 3x3, crop mode is 1x1, 3x5 isn't available. Also sampling doesn't affect data rate, resolution does. 10 bit and 12 bit can be used on the 5D2, you just need a build of that branch, right now you can record corruption free 10/12bit on this camera only if you use crop mode.
Follow the 10-12bit thread: http://www.magiclantern.fm/forum/index.php?topic=5601.950 (http://www.magiclantern.fm/forum/index.php?topic=5601.950) @dfort is posting builds every now and then
Also watch reddeercity's video here: http://www.magiclantern.fm/forum/index.php?topic=11205.msg177721;topicseen#msg177721 (http://www.magiclantern.fm/forum/index.php?topic=11205.msg177721;topicseen#msg177721)
Awesome stuff! Thanks for the thread-links!
Very interested to see if it makes 1920x1080 possible (presuming the cause for the 1854x1044 limitation was data throughput)..
Quote from: Ilia3101 on January 07, 2017, 12:28:57 AM
3x5 isn't available.
Actually it is. 3x5 is mv720 a.k.a. 720p mode which records at 50/60 fps. It skips 3 lines horizontally 5 lines vertically so as you can imagine the aliasing is much worse at this setting.
Quote from: dfort on January 07, 2017, 12:58:08 AM
Actually it is. 3x5 is mv720 a.k.a. 720p mode which records at 50/60 fps. It skips 3 lines horizontally 5 lines vertically so as you can imagine the aliasing is much worse at this setting.
I think he was talking explicitly about the 5D2 (which didn't have 720/60p)... Though, I'd be curious to know the cause of the limitation at this point (being that the data rate is achievable.)
I think the 5D2's sensor simply can't do 60fps, but would be interesting if mv720 could be achieved, don't know what use it would be though.
Quote from: -Rogue5- on January 07, 2017, 12:42:38 AM
Very interested to see if it makes 1920x1080 possible (presuming the cause for the 1854x1044 limitation was data throughput)..
Sadly it won't, the video resolution limit is photo resolution/3, which in the 5D2's case is 5616px and 5616/3 = 1872, although 1880 is also available, probably from some hidden pixels, 1872 & 1880 can be accessed again from raw_rec in the 10/12bit branch, mlv_rec still limited to 1856.
Also crop mode allows 2144x1076
Reprting a bug.
I tried some more around adding black level through mlv_dump using the --black-fix=2048 option. Just as Renatophoto described here http://www.magiclantern.fm/forum/index.php?topic=5601.msg177842#msg177842 it will not work properly with 10bit and 12bit files. With 14bit it,s ok. What happens is following. I used the 25 dec mlv_dump build and tested on two 5D mark III files.
10bit gives
mlv_dump --black-fix=2048 --dng 10bit.MLV delivers
Black Level : 32768
12bit gives
mlv_dump --black-fix=2048 --dng 12bit.MLV delivers
Black Level : 8192
Above also occurs when --no-fixcp is set.
14bit files works with the --black-fix option just as before.
Testfiles here
10bit
https://drive.google.com/file/d/0B4tCJMlOYfirYUJxZ0JPVzJsX0k/view?usp=sharing
12bit
https://drive.google.com/file/d/0B4tCJMlOYfirckp3bFBqanNmbXc/view?usp=sharing
should be fixed with
https://bitbucket.org/hudson/magic-lantern/commits/bff6c46efcf0c0d8bd98b13faf17b98f45203589
Commented on a white level issue.
yeah, i expected that to fail t.b.h. but it was already too late :)
it should be working correctly now
also added a white level option
Cool, tested, made a comment.
Thanks @Danne & @g3gg0 for the quick fixes!
i make video RAW on my Canon 5d MK2 , and choose resolution 1600*900 25p 16:9 ,
Global Draw - OFF
Frame Skipping - OFF
Preview Option - Hacked No Prev
Status When Recording - ICON
Start Delay - 2 sec
Digital Dolly - OFF
Card Warm-up - OFF
Use SRM Job Memory - ON
Extra Hacks - ON
Debug Trace - OFF
Show Buffer Graph - OFF
Buffer Fill Method - 4
Reserve Card Space - ON
Tag Text - none
Tag Take - none
1/Correct settings?
2/And how correct resize in Adobe Premiere via importing DNG-sequence ?
3/And which pixel size choose when creating sequence in Adobe Premiere ?
i have card CF Sandisk Extreme PRO 64gb 160MB/sec
Quote from: g3gg0 on December 26, 2016, 08:53:18 AM
...does the dark frame really make so much difference?
Quote from: DeafEyeJedi on December 26, 2016, 08:53:18 AM
Absolutely it does... Kudos to @Danne for pointing this out and thanks to you @g3gg0 for your never ending magic!
Pre-DF 5D3 @ ISO 3200(https://c1.staticflickr.com/9/8314/28849933250_49cf5ecc86.jpg) (https://flic.kr/p/KXns3f)
Post-DF 5D3 @ ISO 3200(https://c1.staticflickr.com/9/8465/29059880121_92c0f55fd9.jpg) (https://flic.kr/p/LgVtVT)
Pre-DF 7D @ ISO 3200(https://c1.staticflickr.com/1/588/31972392500_178a40bb92.jpg) (https://flic.kr/p/QHhSoy)
Post-DF 7D @ ISO 3200(https://c1.staticflickr.com/1/516/31538261303_a82abbc0e1.jpg) (https://flic.kr/p/Q3VQiV)
Pre-DF 7D @ ISO 6400(https://c1.staticflickr.com/8/7778/29162807805_a10d15decc.jpg) (https://flic.kr/p/Lr21K8)
Post-DF 7D @ ISO 6400(https://c1.staticflickr.com/8/7771/29056924252_4ab33c23af.jpg) (https://flic.kr/p/LgEkfC)
Pre-DF 7D @ ISO 6400 (200% zoomed in ACR)(https://c1.staticflickr.com/9/8898/28541472544_7cdc9fa3bd.jpg) (https://flic.kr/p/Ku7vqd)
Post-DF 7D @ ISO 6400 (200% zoomed in ACR)(https://c1.staticflickr.com/9/8425/28541472854_ea4b5fa263.jpg) (https://flic.kr/p/Ku7vvy)
Flickr link to DF Avg Comparisons Album: https://flic.kr/s/aHskMcV8d2
All files were processed with cr2hdr.app (https://www.magiclantern.fm/forum/index.php?topic=15108.0cr2hdr.app) (latest mlv_dump included) Thanks @Danne & @g3gg0!
mlv_dump command line examples for Windows??? Cant find any???
I am trying to process a 10bit raw video to dng
Here is what I enter (with mlv_dump in directory)
G:\DCIM\100CANON>mlv_dump -o test02 -b 10 --dng m20-1339.mlv
PROBLEM: The output states its defaulting to 14bpp ??? not 10??? when I try with default it still reports 14 bpp
The picture is too small as file properties reports 128x94 96dpi, this is wrong it should be 1824x776
It seems the command-line is wrong ???
Blow is Windows 10 output:
MLV Dumper v1.0
-----------------
Mode of operation:
- Input MLV file: 'm20-1339.mlv'
- Enforcing 14bpp for DNG output
- Convert to DNG frames
- Output into 'test02'
File m20-1339.mlv opened
File m20-1339.m00 not existing.
File m20-1339.IDX opened (XREF)
XREF table contains 328 entries
Processing...
Cold pixels : 0
Reached end of all files after 328 blocks
Processed 304 video frames
Done
the DNG code only supports 14bpp, so the DNG will be saved as 14bpp DNG.
thus the "- Enforcing 14bpp for DNG output"
I,ve been having issues with chroma smoothing in mlv_dump. It,s always occurring in and around blown highlights. Doesn,t matter if shooting 10/12/14bit. When trying to understand what is going on I,m checking chroma_smooth.c it seems the calculations around green is what causes it. So I check against MLVFS and although it seems to be using the same chroma smooth code the output is free from corruption. Could it be related to focus pixels? Maybe those pixels needs to be handled before applying chroma smooth? The good thing about cs2x2 is that when it works it almost completely removes focus pixels. WIth darkframe subtraction it completely removes all focus pixels.
MLV file
https://bitbucket.org/Dannephoto/magic-lantern/downloads/broken_cs2x2.MLV
DNG file
https://bitbucket.org/Dannephoto/magic-lantern/downloads/broken_cs2x2.dng
Chroma smooth set to cs2x2
Examples
(https://s30.postimg.org/p4uwsa335/Screen_Shot_2017_01_29_at_10_58_22.png)
(https://s30.postimg.org/4yw074wtd/Screen_Shot_2017_01_29_at_10_58_32.png)
raw2dng appears to handle it properly.
mlv_dump --dng --no-stripes --no-fixcp broken_cs2x2.MLV
dng2raw broken_cs2x2_000002.dng
wine raw2dng_cs2x2_whitefix.exe broken_cs2x2_000002.RAW
I was just thinking it might be caused by vertical stripes and now see your settings. Didn,t run with vert stripes off before.
Gonna check later.
raw2dng appears to handle it properly.
Ok, mlv_dump produce corrupt chroma smoothing in highlights. Just tested with disabling vertical stripes and no pixels. Then tried converting the file to legacy RAW. How do I even apply cs2x2? Don,t seem to be automated so can,t check this but I take it raw2dng and chroma smooting is working as it should according to a1ex. Any ideas about what,s up with mlv_dump?
I got raw2dng chroma smooth to work and that code is working good with blown highlights. Not mlv_dump.
#define CHROMA_SMOOTH
Copy paste raw2dng chroma smooth void code into mlv_dump works so there is something fishy going on with chroma_smooth.c probably.
Looks like mlv_dump borrowed the chroma smoothing from cr2hdr (which was fine-tuned a long time ago to reduce color artifacts, including those near clipping point), rather than raw2dng (which was fine-tuned for focus dots and had fixes related to white level and shadow noise). Worth checking whether the two codes can be unified.
Yes, that would be nice. I have put in raw2dng code for the time being in mlv_dump and it works very good. What,s your take on this @g3gg0?
I noticed something strange when i want read my DNG sequence with audio into Davinci Resolve 12.5.
When extract with mlv_dump, the .wav file isnt combined with the DNG sequence.
There is no issue when it's done with mlvfs.
(https://image.noelshack.com/fichiers/2017/05/1485888141-mlvfs.jpg) (https://image.noelshack.com/fichiers/2017/05/1485888141-mlv-dump.jpg)
Is there something to change about the output file name command ?
Actually, in my folders from mlv_dump it look like this :
(https://image.noelshack.com/fichiers/2017/05/1485888381-dng.jpg)
And my command line :
@echo off
for %%a in (*.MLV) do ( md "%%~na" 2>nul )&(mlv_dump.exe -o %%~na\%%~na --dng %%~na.MLV )
Wav file needs certain metadata. It,s also working in cr2hdr.app and MLP.(thanks to dmilligan)
Thanks for this detail Danne.
How can we change the mlv_dump code for doing the same job of mlvfs about this issue ?
The code from Dmilligan for sync audio in Resolve is here (https://bitbucket.org/dmilligan/mlvfs/src/9f8191808407bb49112b9ab14c27053ae5022749/mlvfs/wav.h?at=master&fileviewer=file-view-default) but I am completely incompetent to integrate it in mlv_dump.
Are you on windows? Otherwise the other programs would work. The audio file needs to match amount of dng frames. At least cannot be longer. Second a template metadata which must match frames per second in the dng file. I do this with bwf meta edit to put in ixml info and shorten audio with ffmpeg. Both programs exists for windows if you don,t want to do it in c-code.
It,s mainly row 32-50 that needs to be put into the wav file. Framerate must be in 5 numbers(e.g 24fps will be 24000, 23.97 will be 23970)
https://bitbucket.org/dmilligan/mlvfs/src/9f8191808407bb49112b9ab14c27053ae5022749/mlvfs/wav.c?at=master&fileviewer=file-view-default
Bwfmetaedit
http://bwfmetaedit.sourceforge.net/
@g3gg0
Is is possible to add some sort of an optional extensible block to the MLV specifications? I'm not sure what the progress is on the Extended LENS block, ELNS, that you proposed on the Assign lens focal length and name for non cpu lenses (http://www.magiclantern.fm/forum/index.php?topic=18083.msg178166#msg178166) topic but since it seems possible to add blocks without breaking current MLV applications perhaps a general extensible block (xml type of thinking here) would be possible?
A good example of this would be for the crop_rec module. That module could add the crop_rec options used and applications like MLVFS, MLVProducer and Danne's apps that uses my focus pixel script can use that information to pick the right pattern to map out the focus pixels.
I'm not sure the best way to set up such a block. Here's an idea with three optional fields using xml style tags:
Block: XBLK
Offset: 0x00000168
Size: 52
Time: 1743.839000 ms
option_1: <crop_rec>CROP_PRESET_3x3_1X</crop_rec>
option_2:
option_3:
Maybe just one field would be necessary with xml style tags and perhaps using tags isn't the best solution but you get the idea. Something so that when a new feature is introduced it won't need a new block added to the MLV specification.
First we need mlv_rec to tell us we are using crop_rec right @dfort? (Realize now that is what you meant already)
And yes! Metadata for wonderful crop_rec module needed.
My understanding of how this works is that in order to have mlv_rec tell us we're using crop_rec it will require adding a field for that information. Instead of doing something specifically for crop_rec I'm proposing adding a block that won't need to be updated every time a new feature is added.
I don't know how it should be coded but basically crop_rec would pass the preset used to mlv_rec and it would be saved in the extended block.
Let's say there is some other module that uses rather specific information like in the case of astrophotography saving the sensor temperature in the MLV header so that as the DNG's are being processed it can automatically pick from a library of dark frames saved at various temperatures. Instead of having to add this to the MLV specification, which probably won't happen, this can be added to the XBLK section.
Just an idea.
FYI: http://www.magiclantern.fm/forum/index.php?topic=17021.msg173941#msg173941
That way, communication between crop_rec and mlv_rec would be done through the raw backend.
Ok, I see.
Quote from: a1ex on October 29, 2016, 12:18:44 PMFor MLV files, we could add a new block (CROP) with exactly the same structure.
It still needs a new block that isn't in the current or proposed MLV specifications (https://docs.google.com/spreadsheets/d/1ItNuLj34JlK6bZgiAJLrmG7p3iyCmCju4XYyiwjqJOM/edit#gid=0). I was throwing out an idea that would account for future needs.
previously i already implemented some stuff locally.
polished it and pushed it into branch mlv_rec_callbacks (https://bitbucket.org/hudson/magic-lantern/branch/mlv_rec_callbacks)
also modified dual_iso and mlv_snd to add support.
just wondering if there is a way (or need anymore) to tell which lines (odd/even) are DISO'd ?
so i would simply change the definition to just put the "enabled" flag there as it is now.
interface:
any module can register its need for being called via mlv_rec_register_cbr(uint32_t event, event_cbr_t cbr, void *ctx) like this:
/* register callbacks */
mlv_rec_register_cbr(MLV_REC_EVENT_STARTING, &mlv_snd_cbr_starting, NULL);
mlv_rec_register_cbr(MLV_REC_EVENT_STARTED, &mlv_snd_cbr_started, NULL);
mlv_rec_register_cbr(MLV_REC_EVENT_STOPPING, &mlv_snd_cbr_stopping, NULL);
mlv_rec_register_cbr(MLV_REC_EVENT_BLOCK, &mlv_snd_cbr_mlv_block, NULL);
or
mlv_rec_register_cbr(MLV_REC_EVENT_STARTED | MLV_REC_EVENT_CYCLIC, &isoless_mlv_rec_cbr, NULL);
those callback functions will get called for every registered event that gets fired.
so simply register your module's cbr. it gets called and you can queue your custom block like below.
void isoless_mlv_rec_cbr (uint32_t event, void *ctx, mlv_hdr_t *hdr)
{
static mlv_diso_hdr_t dual_iso_block;
mlv_set_type((mlv_hdr_t *)&dual_iso_block, "DISO");
dual_iso_block.blockSize = sizeof(dual_iso_block);
dual_iso_block.dualMode = dual_iso_is_active();
dual_iso_block.isoValue = isoless_recovery_iso;
mlv_rec_queue_block((mlv_hdr_t *)&dual_iso_block);
}
in this case 'mlv_diso_hdr_t' is already defined by mlv.h but you can define any type as long it implements 'mlv_hdr_t'
typedef struct {
uint8_t blockType[4];
uint32_t blockSize;
uint64_t timestamp;
} mlv_hdr_t;
so e.g. add this to crop_rec:
typedef struct {
mlv_hdr_t header; /* cleaner to embed default header instead of explicitly defining members */
uint8_t profileName[32];
} crop_rec_hdr_t;
and fill that block on record start.
There is a generic callback interface in ml-cbr.c (you merged it a while ago). Could it be useful?
good point. i totally forgot about this.
though it might be a bit slow for e.g. block callback when there are more registered clients.
Hi,
I'm new user of ml.
Why when I activate "raw video (mlv)" option, everyday I have another expect write speed? It's annoying, because I never know when I can record MLV continously. Please help me.
Greetings
K.
Hey guys!
Couldn't notice how much cleaner the previews of dark frame usage are. However, I'd say the whole process might be substantially improved: if only we could generate a 'book' of dark frames and, during conversion of the MLV to dng's or transcoding, an appropriate ISO/Shutter/FPS frame would be selected. The 'book' could be fairly easily generated from a continuous MLV file, where Shutter time and ISO are swept manually, let's say once in a second, or (wonderfully) generated automatically by a plug-in in camera.
I have plenty of MLV files, with ISOs all over the place (even real time changed). See my point? I'm sure I'm not alone. Plus, on top of it - I used different FPS. A tag "Dark Frame was already applied", might have a useful potential as well. Though, I'd definitely prefer to apply it to mlv and forget about it.
Surely, some of the permutations might be excessive (like fps) and at times I also was recording into modes different from an uncropped FullHD (sadly).
If the 'book' is too much to ask, then, during generation of black frame mlv's, mlv_dump can average frames into diff files, based on ISO and shutter time. Would be sweet as well!
P.S. Am I correct and MLRawViewer (1.4.3) doesn't support compressed mlv's?
Thanks for the great work!
Mikhail
Re: generating a 'book' or a storage folder full of DF files is already possible in conjunction w @Danne's latest and greatest cr2hdr.app or better yet the recent morphed mlv-dump_steroids!
A proposal for recording pixel binning / line skipping parameters and other related information:
http://www.magiclantern.fm/forum/index.php?topic=17021.msg181639#msg181639
yeah except the re-use of "internal" structs for the file format, its fine.
RAWC due to "raw crop" ?
Yeah, tried some more names, but since it's very similar to RAWI (adding some additional information about the raw buffer), that's what I've got.
simply CROP wont get the point i guess due to the binning info, same for BINN(ing)
SMPL(ling) looks a bit hmm mismatching, sounds like audio stuff.
SENS(or) could be a candidate, but then i'd embed bayer pattern info and e.g. a pixel scan order (why not?), maybe same for TOPO
maybe ARCH(itecture) for sensor/scanning architecture info or SCAN?
CAPA (capture area), REGN (region) or SATT (sensor/scanning attributes) no SCAT please
CAPI for capture info also looks okay to me
if you fall in love with one of those, good. if not, fine as well :)
better thinking a bit too long about names than later being mad because it was misleading :D
I'm not sure it's the right place but I'll link it here: seems like I found a bunch of important and strange bugs in the module as of today.
http://www.magiclantern.fm/forum/index.php?topic=19277.0
Noticed ACR won,t open files with maximum white level set to 16384 which is default when trying to push white level above this in mlv_dump(10/12bit branch and compressed_raw branch). Maximum is 16383 for acr to chew on. Not sure if all places with 16384 in mlv_dump.c should be set to 16383 so I won,t do a pull request. I did following in mlv_dump.c in compressed_raw branch. The white fix code isn,t in unified yet I think.
case 'W':
if(!optarg)
{
white_fix = 15000;
}
else
{
white_fix = MIN(16383, MAX(1, atoi(optarg)));
}
break;
case 'B':
if(!optarg)
{
black_fix = 2048;
}
else
{
black_fix = MIN(16383, MAX(1, atoi(optarg)));
}
break;
When set to 16384
(https://s11.postimg.org/z6605buar/Screen_Shot_2017-04-07_at_07.34.24.png)
Hi,
is there a reason, why mlv_dump for linux isn't auto built anymore? Last available binary version is from 26th December 2016. I'm asking because i need "--white-fix" switch that has been implemented later.
I tried to compile it myself, but I got some errors that I am unable to correct :-[.
Hi. I encountered an annoying event around the -f function in mlv_dump. For example. When extracting a selected frame from a bigger MLV file mlv_dump not only extracts the file itself but it keeps processing VIDF blocks until it reaches the end of the file. Now this causes a serious lag overhead and if you have a MLV file that´s a few gb in size it´s a way too hefty price to pay for extracting only one or a few frames.
In my scriptbased tool cr2hdr.app I run this very fine function to parallelize exports of dng frames in chunks of 4 so the extra overhead will add up hogging resources. Question naturally is if this function could work so that as soon as the last selected dng frame is processed into a dng mlv_dump itself will end VIDF processing?
example usage:
mlv_dump --dng -f 1-2 Input.MLV
I could always kill mlv_dump processing from within terminal but the much nicer solution would be to do it from mlv_dump.
*update. Did some extensive testing and realize this overhead is also caused when going from let´s say mlv_dump --dng -f 400-600 For now I will simply exclude the parallellizing in cr2hdr.app.
hi danne
yeah i am aware of that. i also fell over this issue a few times.
but when implementing the functionality, we must be aware that
blocks are sorted randomly in the file / files (yeah frame 1 could be in the last .Mxx)
so one solution could be, allow selecting frames only in cases where an index file is being used. (first do a run with -x)
this at least guarantees that the frames that appear are in order (sorted by time, frame number *should* be consecutive too).
in case of a missing frame (due to frame drop in mlv_rec) this would also work well.
for files processed without a previously created index, the old method would have to be used.
this seemed a bit odd to me in the first place.
still want this feature?
Quote from: loknar on April 20, 2017, 09:36:13 PM
mlv_dump for linux
can you try the one i added there?
https://builds.magiclantern.fm/jenkins/view/Experiments/job/crop_rec_4k/24/
It´s a little odd, I give you that. I got the idea from so-rose, a linux user here at the forum. I think the feature could be useful if the indexing won´t hog the processing which it does now. I just realised this after parallelizing mlv_dump dng extraction for about one year :P. (Sorry all cr2hdr.app users, new version already up). For bigger files this is a real showstopper I noticed so just reverted to normal --dng processing.
For other things based around the -f function I could easily kill mlv_dump indexing via kill pid processing.
Below, the simplistic script idea. Hard to tell if there will be a real speed increase and it might be too much hassle here to make it roll effectively. This also must work with all other stuff like vertical stripes and so on.
#frame extraction in mlv_dump makes multithreading possible. Thanks to so-rose for highlighting this possibility
frct=$(mlv_dump "$FILE" | awk '/Processed/ { print $2; }')
mlv_dump -f 0-$(($frct/4)) --dng $mlv -o "$O2""${BASE}_1_$date"_ "$FILE" &
mlv_dump -f $(($frct/4+1))-$(($frct/4+1+$frct/4)) --dng $mlv -o "$O2""${BASE}_1_$date"_ "$FILE" &
mlv_dump -f $(($frct/4+1+$frct/4+1))-$(($frct/4+1+$frct/4+1+$frct/4)) --dng $mlv -o "$O2""${BASE}_1_$date"_ "$FILE" &
mlv_dump -f $(($frct/4+1+$frct/4+1+$frct/4+1))-$frct --dng $mlv -o "$O2""${BASE}_1_$date"_ "$FILE"
also thought about multi core usage, but things get weird when using some ancient things like delta mode..
maybe someone could try the OSX binary too?
https://builds.magiclantern.fm/jenkins/view/Experiments/job/crop_rec_4k/24/
Here´s a tested working mlv_dump_crop_rec4k binary for mac OSX.
https://bitbucket.org/Dannephoto/magic-lantern/downloads/mlv_dump_crop_rec4k
Tested some parallell mlv_dump processing on a ssd drive to check for improvements. It sure is possible to speed up things here. Ran 4 mlv files in parallell vs lined up one by one.
4 in parallell
02 Min, 4 Sec
Lined up one by one
03 Min, 37 Sec
My test command(parallell)
mlv_dump --dng M28-2003.MLV &
mlv_dump --dng M28-2004.MLV &
mlv_dump --dng M28-2006.MLV &
time mlv_dump --dng M28-2005.MLV
A little feature request: Ability to add white balance metadata to the processed dng files via mlv_dump. AsShotNeutral information. User could add multipliers for preset wb like sunny, shady, flourouscent etc but there is also the possibility to add manually computed multipliers working with dcraw.
Maybe a --wb= switch?
Quote from: Danne on May 31, 2017, 07:55:04 PM
A little feature request: Ability to add white balance metadata to the processed dng files via mlv_dump. AsShotNeutral information. User could add multipliers for preset wb like sunny, shady, flourouscent etc but there is also the possibility to add manually computed multipliers working with dcraw.
Maybe a --wb= switch?
+1 and this is a really good idea!
does the WBAL block help anything? you can select it in camera.
or do you want to "correct" this when exporting to DNG?
Solely for the purpose to correct wb to the dng files in post.
In the dng:
As Shot Neutral : 0.5793613707 1 0.5545483871
Something like following would be quite useful imo:
example:
mlv_dump --dng --wb=0.6793613707 1 0.7545483871
@g3gg0
I wonder if there could be an abort safeguard for a certain error that can occur when creating shortened MLV files?
http://www.magiclantern.fm/forum/index.php?topic=20418.msg189471#msg189471
typically one would run something like this:
mlv_dump -f 5 -o Output.MLV Input.MLV
When both input and output are named the same the whole MLV file is emptied from content, poff.
mlv_dump -f 5 -o Output.MLV Output.MLV
doh... yeah easy to prevent
I just upgraded to 6D (from 50D), and noticed that I cannot open MLV files made with 6D, using the tools which worked fine with 50D (a year ago). I am using a recent nightly build (magiclantern-Nightly.2017Aug18.6D116), and load module mlv_rec and silent. I take full resolution silent pictures. There seem to be no issues taking the photos, the file size seem to be right, but neither mlv_dump nor MLV Mystic work for me. The Mystic would run way too fast, and there would be no DNG files produced. mlv_dump -v seems to report no errors, runs a bit longer (prints the exif info), but then again no files are generated.
Is it a known issue? How could I troubleshoot this?
Here is one of these files (5 shots; 177 MB):
http://syam.no-ip.org/pub/11370000.MLV
Perhaps I don't understand how to use mlv_dump? My command is
mlv_dump --dng 11370000.MLV
Thanks!
Block: RAWI
Offset: 0x00000034
Number: 1
Size: 180
Time: 4.339000 ms
Res: 5568x3723
raw_info:
api_version 0x00000001
height 3723
width 5568
pitch 9744
frame_size 0x02298AB0
bits_per_pixel 14
black_level 2047
white_level 14783
active_area.y1 52
active_area.x1 72
active_area.y2 3723
active_area.x2 5568
exposure_bias 0, 0
cfa_pattern 0x02010100
calibration_ill 1
a vertical resolution that is not a multiple of two? uhm... quite unusual..?
nevertheless, mlv_dump should not crash. preparing a fix.
Try:
Mac
http://www.magiclantern.fm/forum/index.php?topic=15108.0
Windows
http://www.magiclantern.fm/forum/index.php?topic=10526.msg188558#msg188558
Tried your file on mac and working.
crashed on my machine. depends on how memory layout appears to be.
reason: detect_vertical_stripes_coeffs() goes through all line pairs and if there is an uneven number of lines, it accesses beyond allocated memory
wait till the build job (https://builds.magiclantern.fm/jenkins/view/Experiments/job/crop_rec_4k/) finished, then pick your executable (.exe for windows)
Thanks, I'll try this now. I wonder why I got the odd number of pixels - perhaps it's normal for 6D?
EDIT: g3gg0: I am still getting the same behaviour, with your latest build. BTW the program doesn't crash on me - it just runs too fast, produces no error messages, but also no DNG files. I am using Windows 7 64 bit.
Danne: your mlv_dump.exe worked, thanks a bunch! I just copied your mlv_dump.exe inside my images folder, run it from there, and it worked without issues, for both small (5 shots) and large (144 shots) MLV files I made with 6D.
interesting, i ran the mlv_dump i posted too. before the patch, it crashed, after fixing it, it properly exported some frames.
are you sure the build job finished before you got the .exe?
Quote from: pulsar124 on September 24, 2017, 03:27:23 AM
« Reply #2123 on: Today at 03:27:23 AM »
Build History:
#48 Sep 24, 2017 4:56 AM
-> the build process took longer. thus you downloaded the old version of mlv_dump with code that could crash.
Indeed - I didn't realize it wasn't done yet. I just tried the newest build, and it worked. Thanks!
Hi man,
While some people still believe Magic Lantern and RAW recording on Canon DSLR cameras is hacky, unproven, and unusable in any type of "production" fashion, I'm here to show you how they aren't. You just need to know what to expect and be slightly prepared.
Here are a few things you can expect when shooting Magic Lantern RAW:
1. You will need a massive amount of hard drive storage on set. Each hour will eat up around ~350-400GBs, depending on the exact resolution & if the sound is recorded. Make sure you have a few HDDs.
2. You should have a DIT on set who does nothing but offload cards and check the MLV files, if the production has the budget or you can wrangle someone to handle this very important task.
3. Each card will only hold between 11-22 min of footage, so you will be switching cards quite a bit. Think of it like you are shooting film, and you have to plan more carefully. It may be a limitation and CAN kill time, but not if you plan around it. Bring a least 3 1066x (or faster) CF cards and\or have an organized and efficient cycle of downloading the CF cards to avoid too much downtime. *Some people swear by 1000x, some by 1066x. Depends on the brand of CF card apparently.
4. While the camera could have the occasional "crash", no need to panic, as cycling the camera on\off or pulling the battery will almost always fix it. Some models (5D Mark II, III) are far more stable than others (7D). This article is written from the perspective of working with the least stable model, and I've had only a few issues.
source from:http://www.cineticstudios.com/blog/2014/08/the-easy-way-to-use-magic-lantern-raw-straight-to-prores.html (http://www.cineticstudios.com/blog/2014/08/the-easy-way-to-use-magic-lantern-raw-straight-to-prores.html)
good luck
Removed the second link //Audionut
Using 256gb CF cards and lossless compression gives you 80min+ rec times on one card. Really good if you wanna film documentary or interviews in raw! A couple of 256gb cards and you're set for a whole day of shooting :-)
Mk3 ML raw + da vinci resolve is awesome!
Is there any way I could save only odd or even frames out from an input MLV file and output that into a new MLV file?
I need to do this to save some space of redundant duplicated frames of some MLV written by mlv_rec.mo that wrote duplicated frames in 1fps frame override mode in 1x crop on 5D2.
I tried -f switch but that only allows input of frame or frame range, not comma delimited frame numbers, -f 0,2,4,6,8 for example.
Any ideas would be appreciated. Thanks!
Quote from: Markus on January 16, 2018, 05:46:28 PM
Using 256gb CF cards and lossless compression gives you 80min+ rec times on one card. Really good if you wanna film documentary or interviews in raw! A couple of 256gb cards and you're set for a whole day of shooting :-)
Mk3 ML raw + da vinci resolve is awesome!
which kind of lossless compression? I'm a bit lost with the latest developement :) if you can point me to the right thread it would be a big help!
Just a note for anybody having trouble exporting audio recorded with mlv_snd. My MLV videos started with the "WAVI" chunk that describes the audio configuration, which is good:
# mlv_dump --visualize --skip-xref M25-1629.MLV
...
Processing...
[MLVI]
[RAWI][RAWC][IDNT][EXPO][LENS][RTCI][WBAL][VERS]++++++++[WAVI][NULL]
[VIDF]+++++++++++++++++++++++[AUDF]++++++++++++++++++++++[NULL]
[VIDF]++++++++++++++++[AUDF]++++++++[NULL]++++++++++++++++++++++++++
Reached end of chunk 1/1 after 947 blocks
Processed 592 video frames at 0.01 FPS (86558.85 s)
But for some reason the "WAVI" chunk appeared last in the "IDX" index file:
# mlv_dump --visualize M25-1629.MLV
...
M25-1629.MLV File M25-1629.IDX opened (XREF)
XREF table contains 730 entries
Processing...
[MLVI]
[RAWI][RAWC][IDNT][EXPO][LENS][RTCI][WBAL][VERS]++++++++[AUDF]
[VIDF]++++[AUDF]
[VIDF]++++[AUDF]
[VIDF]++++[AUDF]
[VIDF]++++[AUDF]
[VIDF]++++[AUDF]
[VIDF]++++[AUDF]
[VIDF]++++[AUDF]
[VIDF]++++[AUDF]
[VIDF]+[AUDF][WAVI]
Reached end of all files after 730 blocks
Processed 592 video frames at 0.01 FPS (86558.85 s)
Done
When dumping the video using mlv_dump, it seems to read the chunks in the order specified in the index, so although it encountered "AUDF" chunks throughout the video, it never wrote them to the wav file, because it hadn't seen the "WAVI" audio information chunk yet. This resulted in an empty 2kB wav file.
Adding "--skip-xref" to the mlv_dump command fixed this and I recovered my 4.6MB of audio! (i.e. mlv_dump --dng --skip-xref M25-1629.MLV)
Nice find!
@Danne,
Please can you make and update to your Batch_MLV-in-out script.
"Adding "--skip-xref" to the mlv_dump command fixed this and I recovered my 4.6MB of audio! (i.e. mlv_dump --dng --skip-xref M25-1629.MLV)"
This might be the fix for extracting all audio from lossless MLV's in your script/app.
I see what I can do. Don´t even have a windows os anywhere close. Might be able to fix it on my mac. No promises.
@Kharak
Could you test this version:
https://bitbucket.org/Dannephoto/batch_mlv/downloads/batch_mlv_io_test.zip
Baked on my mac, completely untested. menu item 18 will set skip xref if working.
Yes, i will test ASAP and report back.
Thank you!
QuoteAdding "--skip-xref" to the mlv_dump command fixed this and I recovered my 4.6MB of audio! (i.e. mlv_dump --dng --skip-xref M25-1629.MLV) Quote
I have tested this and it seems to completely solve my audio issues with Lossless 14 bit. Thank you!
Danne added a very neat option in his Batch script for --skip-xref. https://bitbucket.org/Dannephoto/batch_mlv/downloads/batch_mlv_in-out.zip
edit: cant get it to UNQUOTE this part, only top part is supposed to be in quotes
Great. Only included a function already there but good it works.
I don't know if this is the right place to ask but I've found something odd...
I got a new Zeiss lens and when I take pictures with it, my ACR (Photoshop RAW converter) can read its metadata and apply 'lens profile adjustment' automatically.
When I create video .dng files using mlv_dump (I tested MLVApp too) I can't make it apply the adjustment automatically - it is behaving as it does not have any metadata related to the lens used but the info about 'focal length', shutter, aperture, ISO, are all there!
With my Canon and Sigma lenses this problem doesn't happen, only with my new Zeiss Distagon ZE (Canon mount).
Any light on this issue?
@g3gg0 @a1ex
Noticed apple is cabling out MacOS Catalina - 64bit only. So I notice mlv_dump is 32bit only. Hm, planning on upgrading here maybe :)?
file /Users/danne/Desktop/Switch.app/Contents/mlv_dump
/Users/danne/Desktop/Switch.app/Contents/mlv_dump: Mach-O executable i386
Same story with cr2hdr...
EDIT:
Went into the makefile and changed:
# to allow x64 binaries. doesn't work on OSX, seems to SIGSEGV for some reason.
ifndef USE_X64
MLV_CFLAGS += -m32
MLV_LFLAGS += -m32
endif
to
# to allow x64 binaries. doesn't work on OSX, seems to SIGSEGV for some reason.
ifndef USE_X64
MLV_CFLAGS += -m64
MLV_LFLAGS += -m64
endif
Seems to work:
file /Users/danne/magic-lantern_jip-hop/modules/mlv_rec/mlv_dump
/Users/danne/magic-lantern_jip-hop/modules/mlv_rec/mlv_dump: Mach-O 64-bit executable x86_64
Maybe this will compile for 64bit automatically if the system is without 32bit architecture? Didn´t check since not on Catalina yet...
Just tried it on macOS Catalina, version 10.15 and it works. Needed to use the Apple compiler, I haven't gotten Homebrew gcc-5 working yet.
why gcc-5? why not the latest- 9 or 8
Good question, that's just the way I had the build environment setup before switching over to Catalina. The Apple compiler has always worked for mlv_dump and also for ML (it is mostly using the arm compiler for that) but there was a problem when building cr2hdr. In any case a quick fix for users that have updated to the new OS is working now.
Good to hear it is working. Though I probably won't update to Catalina.
I wonder if mlv_dump works with gcc-9 on macOS.
Cr2hdr needs some love here from a1ex to get into 64bit compiling. Seems g3gg0 prepared mlv_dump for 64bit architecture so it was a quick change to make it work.
Thanks for sharing the temp fix @Danne -- gonna try this out out on my MBP running Catalina.
I need to get your Switch app working again for my Dual-ISO photo work. :o
I,m starting to feel imbecile trying to get the latest revised mlv_dump to work with Switch, Compiler & MLV_App too while on Catalina... :-\
Edit:
It's been fixed (https://www.magiclantern.fm/forum/index.php?topic=15108.450It's%20been%20fixed) — Thanks @Danne & @Ilia3101!
Is it possible to record straight in dng lossless?
Quote from: andicgn on December 26, 2019, 11:52:15 AM
Is it possible to record straight in dng lossless?
Wondering this too