Record video to > 4 GiB files in H.264 mode (esp. for 5D Mark III)

[alMalsamo]

We do quite a lot of event videography, and one of the nicer features of the 5D Mark III is that the default camera firmware allows for recording 30 minutes straight without needing ML's movie restart feature.  We are very greatful to the movie restart feature for 30+ minute recordings, but this feature request is not about that.

As you know, the 5D Mark III still splits up video files into 4 GiB chunks, even when the CF or SD card uses the exFAT filesystem.  This limitation is the basis for the mlv_rec RAW video recording option labeled

Files > 4 GiB (exFAT)

This request is for the same feature in H.264 mode.  During post-production, we have noticed that even with the camera supposedly being able to "continuously" record to 29:59, there are dropped frames during the 4GiB file changes!  The first 4GiB file generally records for 16-17 minutes, after which the next file is created: in between, there are about 4 video frames that are lost in this transition, meaning the 5D Mark III is not able to truely record to 29:59 straight through!

It is frustrating because when using exFAT, there is no need to make 4GiB splits.  Ideal ML behaviour would be to auto-detect the filesystem on the recording media and simply keep recording until the 29:59 limit is reached in a single file if exFAT is used.  If this ideal behaviour is too difficult, then simply giving a menu option as with the mlv_rec's "Files > 4 GiB" option would be a huge improvement for all of us who need continuous recording with no dropped frames.


It would, at least in our case, be a killer feature of the ML software!  We are otherwise being unfairly held back by FAT-era limitations in a post-FAT world.

[gary2013]

I was wondering if it was possible for ML to keep recording those four frames or so into a memory buffer and then it records those four frames into another split file when the next clip starts recording. Or it accumulates the four frames each time it starts a new clip. Meaning, the next clip is always four frames later than real time and so on for the whole shoot to one memory card. Maybe if we did this for a long shoot, it might have accumulated at most a 28 frame delay (just guessing, but making a point) which wouldn't even be noticed after it all gets rejoined in post without any dropped frames.

[alMalsamo]

I was wondering if it was possible for ML to keep recording those four frames or so into a memory buffer and then it records those four frames into another split file when the next clip starts recording.

Are you suggesting a temporary split file for the 4 frames to be saved to a separate file when the camera creates the 2nd file?  So we would end up with 3 files in the end for a 30 minute clip, or would you suggest having ML join the files at the end of the record session to get one continuous 29:59 file at the end of it?


I am not sure the best way to implement this, but finding out how the > 4 GiB file setting for mlv_rec works may be helpful in determining how easy this is to implement for H.264.

[g3gg0]

mlv_rec -> ML code
h264 -> canon binary

-> not possible the way you explained it

[1%]

It can be patched the same way as on 600D, again the meta data would probably have to be rewritten. I thought about doing it but figured why bother since its supposed to auto split the files. Its more work to re-write the videos just for 4 frames.

[gary2013]

Are you suggesting a temporary split file for the 4 frames to be saved to a separate file when the camera creates the 2nd file?  So we would end up with 3 files in the end for a 30 minute clip, or would you suggest having ML join the files at the end of the record session to get one continuous 29:59 file at the end of it?


I am not sure the best way to implement this, but finding out how the > 4 GiB file setting for mlv_rec works may be helpful in determining how easy this is to implement for H.264.

I would say any way that can preserve non stop playback in post after rejoining the files. No dropped frames. It would be great to have ML do it in camera so we end up with one complete joined file that has no dropped frames. But if ML cannot do it all in camera, then an app we can use in post that can join the 3 files (or more, depending on the size of the memory card) into one complete, continuous file.

[gary2013]

It can be patched the same way as on 600D, again the meta data would probably have to be rewritten. I thought about doing it but figured why bother since its supposed to auto split the files. Its more work to re-write the videos just for 4 frames.

I am talking about not missing any frames when shooting long, non stop events that are splitting the files. He said he thinks it always loses four frames between each 4gb split it records. Can ML/TL record a small 4 frame clip so we then have three clips to rejoin with nothing missing? Or, can ML possibly keep recording into a buffer while it sets up to record the next clip. That buffer then feeds the writing of the second clip, always having a latency of four frames, but not a problem since in post it all gets rejoined. 

Or, maybe we can have an extra small buffer area that only accumulates every small four frames sections and when the whole event ends, the buffers records files to the card. in post, we get an app that reassemble all the clips in proper order into one continuous file. Maybe it has some sort of checking that assures it was all compiled into one with no errors and then it deletes all the splits from the hard disk to save storage space.

I do not have the knowledge of how how TL/ML actually does any of the recording. It all reminds me of the pre read ability we had in Sony VTR's years ago when digital tape recording was introduced. Now, we have other video cameras on the market that record split files and never lose any frames. I think Sony and Panasonic can do it.

[gary2013]

mlv_rec -> ML code
h264 -> canon binary

-> not possible the way you explained it

please explain why it can't happen. If I explained it wrong, then how should it be explained?

[g3gg0]

the only reasonable way would be using mlv_rec to record H264 frames, a case which was considered in the file format.

if someone comes up with a piece of code which allows me to capture H264/JPEG in LV mode without canon firmware being recording, then i can integrate that into mlv_rec.
this requires no change to the MLV format, as it was designed as container format and i designed it even for that use case.

[g3gg0]

please explain why it can't happen. If I explained it wrong, then how should it be explained?

if we know how to write files with our own code, this doesnt automatically mean that we can "add" this to a complex binary-only firmware function.
we have the source code of mlv_rec / raw_rec. we do not have the firmware source code, so changing fundamental design concepts isnt easily possible.

[alMalsamo]

the only reasonable way would be using mlv_rec to record H264 frames, a case which was considered in the file format.

if someone comes up with a piece of code which allows me to capture H264/JPEG in LV mode without canon firmware being recording, then i can integrate that into mlv_rec.
this requires no change to the MLV format, as it was designed as container format and i designed it even for that use case.

If the H.264 encoder is embedded into the proprietary Canon firmware, then it obviously cannot be modified itself as you have explained.  But is it possible to call the Canon H.264 encoder from the ML Live View mode?  Or perhaps possible to ship a custom encoder like x264 or even for other formats such as VP9 or H.265 when those formats' encoders become more mature?

I am totally unfamiliar with the ML codebase to know what is possible in terms of calling external video encoders (or even calling the Canon H.264 encoder from within mlv_rec), but it seems like > 4 GiB video files is not possible (or very difficult?) unless ML itself handles the recording of the frames.

It can be patched the same way as on 600D, again the meta data would probably have to be rewritten. I thought about doing it but figured why bother since its supposed to auto split the files. Its more work to re-write the videos just for 4 frames.

Can you elaborate on this please?  I hope you understand that missing 4 frames is the difference between true continuous recording for 30 minutes and the need to have a second camera setup on hand to transition to in post-production to hide unprofessional video skips.  I would love to hear your thoughts on the method of video recording on a 600D with ML might help to create > 4 GiB H.264 files up to 30 min, or at least save the missing 4 frames in a separate file so we can write scripts to merge the data in post.

[g3gg0]

600D allows recording to up to 30 min or more iirc

problem A:
  Recording software-limited to 4GiB
  -> solved

problem B:
  Recording software-limited to 30 min
  -> solved (on 600D only, right?)

problem C:
  H264 format doesnt allow sizes >4GiB out of the box
  -> creates buggy H264 files that need to be fixed using a tool (EOSMovieFixer)

problem D:
  Files > 4GiB only possible with exFAT
  -> many models support exFAT, but not all

problem E:
  Filesystem driver only supports < 4GiB
  -> some few models support large file offsets

[gary2013]

Thanks g3gg0, 1% and everyone here for taking the time on this topic. I hope it eventually has a solution for all the cameras, including the M.

[1%]

Recording software-limited to 30 min
  -> solved (on 600D only, right?)

Was never able to bust through the limit even after patching it. And there are tons of spots to patch, I never got them all.

[alMalsamo]

problem A:
  Recording software-limited to 4GiB
  -> solved

...but solved only for mlv_rec (which hopefully will lead to a breakthrough for H.264 recording as well).  While it is far more urgent in RAW recording mode because of the massive size of RAW video streams, it is extremely annoying that advertised "continuous" recording to 30 minutes for bodies such as the 5D Mark III are actually not continuous because they lose frames at the 4GiB file changes.

problem B:
  Recording software-limited to 30 min
  -> solved (on 600D only, right?)

I have tacitly accepted that the 30 minute limit is not going to be overcome anytime soon.  Don't get me wrong, I would love to hear that it could, but I am a least very thankful for "Movie Restart" feature to deal with this.  Expecting dropped frames every 30 minutes is a lot easier than expecting dropped frames every ~16 min *AND* ~30 min, hence this feature request asking to overcome only the first barrier.

problem C:
  H264 format doesnt allow sizes >4GiB out of the box
  -> creates buggy H264 files that need to be fixed using a tool (EOSMovieFixer)

Although I think you know what you mean when you say this, it would be ideal to be more clear with terminology for others: the H.264 video format itself has no limitations on filesize.  It is only the recording software in the non-libre Canon firmware that has this undesirable yet unmodifiable default behaviour of finishing writing to a file at 4 GiB of size... queue question about possibility of new H.264 recording mode handled entirely by ML.  Would a third-party encoder such as x264 be required to ship with ML?  If that is possible, then would this open the door to allow recording to other formats, such as VP9 or H.265?

problem D:
  Files > 4GiB only possible with exFAT
  -> many models support exFAT, but not all

I don't see this as a huge issue, since ML only supports a limited number of camera bodies.  It should be obvious that the > 4 GiB file feature for both mlv_rec and H.264 mode (if intrepid developers can solve it) should be disabled on any camera that cannot support exFAT.

problem E:
  Filesystem driver only supports < 4GiB
  -> some few models support large file offsets

I'm less educated about how the filesystem driver differs significantly from exFAT support.  Are you implying that some cameras that DO support exFAT have a fs driver that cannot support files > 4 GiB?  That seems counter-intuitive, as that is one of the basic specifications that makes exFAT implementation more flexible and desirable.


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


 If ML is able to liberate H.264 recording from the dropped frame hell the bodies currently are stuck in, it would make these bodies unbelievably useful for many people in a wide variety of use cases.  I opened a feature request to solve the glaring 4GiB limitation first, but indeed to crack the 30-min limitation (obsoleting the need for the Movie Restart hack) would be a revolutionary feature.  Full frame cameras with unlimited continuous recording in both H.264 and RAW without dropped frames is a powerful end goal.


In the meantime, tackling this highly annoying 4 GiB limitation and using Movie Restart (plus a second body to cover up the video hiccups at 30 minutes) would be of HUGE importance to event videographers!

[g3gg0]

...but solved only for mlv_rec (which hopefully will lead to a breakthrough for H.264 recording as well).  While it is far more urgent in RAW recording mode because of the massive size of RAW video streams, it is extremely annoying that advertised "continuous" recording to 30 minutes for bodies such as the 5D Mark III are actually not continuous because they lose frames at the 4GiB file changes.

no.
a) i do not talk about replacing an uint32_t with an uint64_t in some source file.
i talk about canon firmware patching. and solved means that we can patch that point in the firmware.

b) uhm, then use exFAT and enable the >4GiB menu option? or just lower your resolution?

I have tacitly accepted that the 30 minute limit is not going to be overcome anytime soon.  Don't get me wrong, I would love to hear that it could, but I am a least very thankful for "Movie Restart" feature to deal with this.  Expecting dropped frames every 30 minutes is a lot easier than expecting dropped frames every ~16 min *AND* ~30 min, hence this feature request asking to overcome only the first barrier.

this resigned position contradicts to the euphoric last words in your post:
"to crack the 30-min limitation (obsoleting the need for the Movie Restart hack) would be a revolutionary feature.  Full frame cameras with unlimited continuous recording in both H.264 and RAW without dropped frames is a powerful end goal."

i dont get what you want with 16 min and 30 min frame drop.

Although I think you know what you mean when you say this, it would be ideal to be more clear with terminology for others: the H.264 video format itself has no limitations on filesize.  It is only the recording software in the non-libre Canon firmware that has this undesirable yet unmodifiable default behaviour of finishing writing to a file at 4 GiB of size... queue question about possibility of new H.264 recording mode handled entirely by ML.  Would a third-party encoder such as x264 be required to ship with ML?  If that is possible, then would this open the door to allow recording to other formats, such as VP9 or H.265?

no, why are you stating things as facts which you have not proven?
if you want the skinny details just ask and dont try to prove me wrong.

H264 videos on canon cameras are in a quicktime container.
this container format was not designed for file sizes >4GiB until somewhen in 2000 when quicktime 4.0 was released.
stco sections were simply just 32 bit offsets. later co64 chunks were added with that quicktime version.

canon uses the standard 32 bit set of atoms like stco.
this is a legit decision as it saves from a lot of trouble. especially from users.

file size limitation was already solved in A)

additional encoders. well. we talk about a highly embedded, high volume consumer device.
do you expect an intel pentium inside?
there is nothing to upgrade. it is pure hardware.

I don't see this as a huge issue, since ML only supports a limited number of camera bodies.  It should be obvious that the > 4 GiB file feature for both mlv_rec and H.264 mode (if intrepid developers can solve it) should be disabled on any camera that cannot support exFAT.

again it is not about mlv_rec, its about H264 recording.
no exFAT = no files >4Gib.
and for people who want >4GiB H264 files, this is in fact an issue.

I'm less educated about how the filesystem driver differs significantly from exFAT support.  Are you implying that some cameras that DO support exFAT have a fs driver that cannot support files > 4 GiB?  That seems counter-intuitive, as that is one of the basic specifications that makes exFAT implementation more flexible and desirable.

i do. files >4GiB were never meant to be handled by the camera firmware, so there are restrictions.
what if you have exFAT support but you never handle files >4GiB as movie recording is limited to that?
do you then still have to support large files?



apart from all that, i still dont get what your intention with questioning teh facts is all about.

[g3gg0]

Was never able to bust through the limit even after patching it. And there are tons of spots to patch, I never got them all.

remember that, yeah. four checks at various places and still it stopped.
a problem hard to debug with 30 min per try

[1%]

I thought of making it stop short to get around that but it failed

Even on 50D/7D where that 1799 is only in a few spots.

[alMalsamo]

i talk about canon firmware patching. and solved means that we can patch that point in the firmware.

Ah, then I misunderstood what you referring to.

this resigned position contradicts to the euphoric last words in your post:
"to crack the 30-min limitation (obsoleting the need for the Movie Restart hack) would be a revolutionary feature.  Full frame cameras with unlimited continuous recording in both H.264 and RAW without dropped frames is a powerful end goal."

i dont get what you want with 16 min and 30 min frame drop.

Continuous recording with no dropped frames is the end goal, and right now on the 5D Mark III (probably the 6D as well?) there are two barriers to that end goal.  The first barrier is what this feature request covers: the 4 dropped frames when a 4GiB file finishes writing and recording moves on to the next file.  The second barrier is the forced stopping of recording at the 30 minute mark to put the camera in a different tax category.  The Movie Restart feature ML provides is a nice (but very imperfect) workaround for the latter, but yields dozens of dropped frames instead of only four.

Because the technological and "political" background for both of these barriers are discrete, I would think it best to have a feature request and subsequent discussion for both of them individually.  From my limited understandings of the inner workings of things here, I imagine the solution for having no dropped frames through both aforementioned barriers would be different as well.

no, why are you stating things as facts which you have not proven?
if you want the skinny details just ask and dont try to prove me wrong.

I actually do want as much details as possible about the situation so that I and anyone else who may be interested in the end goal of true continuous recording can become more educated about how best to overcome the current barriers.  I did not  try to prove you wrong... I was only stating the fact that there is no 4GiB limitation inherent in the H.264 (MPEG-4 AVC) video format itself.  While you are clearly much more knowledgeable than most about the issues as they relate to H.264 recording in Canon cameras, I just wanted to make clear to anyone who might have mistakenly took your words literally.

H264 videos on canon cameras are in a quicktime container.
this container format was not designed for file sizes >4GiB until somewhen in 2000 when quicktime 4.0 was released.
stco sections were simply just 32 bit offsets. later co64 chunks were added with that quicktime version.

canon uses the standard 32 bit set of atoms like stco.
this is a legit decision as it saves from a lot of trouble. especially from users.

file size limitation was already solved in A)

Thank you for being more explicit, as some of this information was new to me and helps me understand the situation on a deeper level.

additional encoders. well. we talk about a highly embedded, high volume consumer device.
do you expect an intel pentium inside?
there is nothing to upgrade. it is pure hardware.

On second thought, it is naïve to ask about support for encoding to other formats that the cameras do not have hardware acceleration support built-in.  Scratch the questions about other formats, but I am still not knowledgable enough to know if a different H.264 encoder (such as x264) would be able to take advantage of the camera's H.264 hardware encoding acceleration, or if that would even help in overcoming the barriers to continuous recording.  Insight from those more knowledgable clearing up either point would be appreciated.

again it is not about mlv_rec, its about H264 recording.
no exFAT = no files >4Gib.
and for people who want >4GiB H264 files, this is in fact an issue.

When I said I thought it wasn't a huge issue, I meant it shouldn't be an issue in terms of implementing a solution in the ML codebase.  If the barrier to writing >4 GiB H.264 files can be overcome, then making this feature unavailable to builds of ML for non-exFAT cameras should be trivial.  It should be as obvious to the users with older cameras that they can't use the >4GiB file feature with H.264 as it is that they can't use the >4GiB file feature with RAW recording.  Making the options simply unavailable to non-exFAT cameras seems to be the ideal solution in both cases.

files >4GiB were never meant to be handled by the camera firmware, so there are restrictions.
what if you have exFAT support but you never handle files >4GiB as movie recording is limited to that?
do you then still have to support large files?

Obviously Canon has chosen not to support large files, even though the underlying filesystem on the storage media may support them.  This the annoying barrier that can hopefully be overcome in H.264 mode as it was in RAW mode.  Is there a limitation on the version of the .MOV container that Canon uses as you alluded to in your post?  If so, can someone write a recording mode for H.264 that puts it into an updated .MOV container, or perhaps something else like MLV or MKV?


Any further technical analysis of how these cameras record to H.264 MOV files currently that might help to overcome the 4GiB barrier to continuous recording is very welcome!

[Audionut]

Because the technological and "political" background for both of these barriers are discrete, I would think it best to have a feature request and subsequent discussion for both of them individually.  From my limited understandings of the inner workings of things here, I imagine the solution for having no dropped frames through both aforementioned barriers would be different as well.

> 4GB recording capacity is pretty pointless without the removal of time limits, and to a lesser extent, vice versa.  The discussion about the removal of limits in H.264 encoding, is perfectly fine, contained in 1 thread.

On second thought, it is naïve to ask about support for encoding to other formats that the cameras do not have hardware acceleration support built-in.  Scratch the questions about other formats, but I am still not knowledgable enough to know if a different H.264 encoder (such as x264) would be able to take advantage of the camera's H.264 hardware encoding acceleration, or if that would even help in overcoming the barriers to continuous recording.

As has been discussed, numerous times, the H.264 encoder used by Canon, is performed by a dedicated hardware chip.  There is no changing of encoders, upgrading, downgrading, or whatever else.
Until that hardware chip is accessed and fully understood, that's all she wrote.

When I said I thought it wasn't a huge issue, I meant it shouldn't be an issue in terms of implementing a solution in the ML codebase.  If the barrier to writing >4 GiB H.264 files can be overcome, then making this feature unavailable to builds of ML for non-exFAT cameras should be trivial.

As a1ex would say, if it's trivial, we look forward to your patch 

It should be as obvious to the users with older cameras that they can't use the >4GiB file feature with H.264 as it is that they can't use the >4GiB file feature with RAW recording.  Making the options simply unavailable to non-exFAT cameras seems to be the ideal solution in both cases.

For exfat enabled cameras, yes that would be the ideal solution.  Who cares about non exfat cameras anyway!   
Oh, and actually, non exfat cameras do have > 4GB raw capabilities.  It's called file splitting.  And since it's not limited in the same manner as H.264 encoding, there is no frame loss.

[alMalsamo]

> 4GB recording capacity is pretty pointless without the removal of time limits, and to a lesser extent, vice versa.  The discussion about the removal of limits in H.264 encoding, is perfectly fine, contained in 1 thread.

I do not consider it pointless to overcome 4 GiB file limits without removing the 30 minute time limit in the 5D Mark III.  The time limit is at 29:59, but frames are already dropped well before this limit is reached.  Being able to actually have a continuous recording to 29:59 with NO dropped frames would already be an improvement. 

Furthermore, because it will probably be a different solution to recovering the lost frames at 4GiB file changes, I imagined that focusing on one barrier at a time would maybe be more efficient.  I am also open to a full-blown discussion for obtaining true continuous H.264 recording with no dropped frames, because that is an ideal end goal.

If the barrier to writing >4 GiB H.264 files can be overcome, then making this feature unavailable to builds of ML for non-exFAT cameras should be trivial.

As a1ex would say, if it's trivial, we look forward to your patch 

My statement was conditional.  I said if the 4GiB barrier can be overcome, then making it unavailable to non-exFAT cameras would be trivial.  I didn't say designing or implementing a solution in the ML codebase is trivial.

For exfat enabled cameras, yes that would be the ideal solution.  Who cares about non exfat cameras anyway!   
Oh, and actually, non exfat cameras do have > 4GB raw capabilities.  It's called file splitting.  And since it's not limited in the same manner as H.264 encoding, there is no frame loss.

Honestly, any solution that yields no dropped frames would be a huge improvement.  Even if there are muliple files to deal with in post-production, we could at least write and distribute scripts to join continuous recordings prior to editing with no quality loss.  The crux of the problem is recovering the dropped frames during recording.  The discussion would ideally be centered first around recovering dropped frames, and only secondarily about how most efficiently to join them into a continuous file that could exceed 4 GiB.

[Audionut]

Being able to actually have a continuous recording to 29:59 with NO dropped frames would already be an improvement. 

That has been possible for quite a long time.

My statement was conditional.

Got to love those conditional statements!

If you are unable to do things yourself, then you should probably not make assumptions about things being trivial.  Whether that assumption is based on conditions, or not.

[alMalsamo]

That has been possible for quite a long time.

Is is possible when recording to H.264?  Because that is what this feature request is about.

If you are unable to do things yourself, then you should probably not make assumptions about things being trivial.  Whether that assumption is based on conditions, or not.

I believe overcoming the barrier to saving dropped frames at the 4 GiB limit and implementing a solution where we could select recording to > 4 GiB files in the ML menu for H.264 mode as well is absolutely non-trivial.  I am not able to come up with a solution to this myself, not by a long shot.

However, if someone else comes up with a solution, I'd happily submit a patch to disable the option for cameras that do not support exFAT.  This is comparatively trivial, and within my abilities to lend a helping hand on.

[Audionut]

Is is possible when recording to H.264?

Drop the bitrate.

[g3gg0]

i dont see any end in this discussion, so lets stop it here.
please understand that if things were as simple to solve as you suggested, we would have done them already.

anyway, thanks for trying to help.