Author Topic: Recording RAW and H.264 at the same time  (Read 8708 times)

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10180
  • 5D Mark Free
Recording RAW and H.264 at the same time
« on: February 11, 2017, 02:34:52 PM »
I was experimenting with shooting raw video while simultaneously recording H.264 [...]

Quote
[...]it is too much of a hack[...]

Here's an attempt to make it a bit less of a hack:

http://bitbucket.org/hudson/magic-lantern/branch/raw-h264-proxy

Danne

  • Hero Member
  • *****
  • Posts: 3245
Re: Recording RAW and H.264 at the same time
« Reply #1 on: February 11, 2017, 03:43:26 PM »
Nice one. legacy RAW still alive  :P.
I tried some and 5D mark III is working really good on this one. Make sure to select SD card in canon menu as pointed out from a1ex(preferred card type CF in ML menu). I ran 1920x1080 24fps and recording proxy H.264 to sd continuously.
Now regarding eosm not so lucky. Got better files out of the hacky build from dfort. In this case here all RAW dng files are corrupted and lacks black levels. The proxy comes out choppy. Some examples here. I also reduced bitrate on this H.264 clip. It seems maybe it gets into mv1080(hard to tell on the dng. Part of my foot seems mv1080 :)))
https://drive.google.com/drive/folders/0B4tCJMlOYfirR2QtNjNveXZUUzQ?usp=sharing

Idea. How about disabling H.264 proxy once RAW starts to record in mv1080 to maybe fool eosm to stay in mv1080 mode? Ugly as hell but might lead to another step in some direction.

dfort

  • Hero Member
  • *****
  • Posts: 1799
Re: Recording RAW and H.264 at the same time
« Reply #2 on: February 11, 2017, 03:54:23 PM »
This can actually be very useful. Some users have been asking about this "feature" of shooting an H.264 along with raw video. It is a solution for a quick on set confidence check playback. If it is frame accurate it can also be sent to editorial to start the edit while the raw files are being processed.

Though it looks like we're not quite there for the EOSM but we might be getting closer?
EOSM.202 EOSM.203 EOSM2.103 700D.115 5D3.*

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10180
  • 5D Mark Free
Re: Recording RAW and H.264 at the same time
« Reply #3 on: February 11, 2017, 04:14:40 PM »
Nice one. legacy RAW still alive  :P.

Check latest commits ;)

Quote
Idea. How about disabling H.264 proxy once RAW starts to record in mv1080 to maybe fool eosm to stay in mv1080 mode?

Not sure how you would do that, other than stopping H.264 recording (which will bring you back to 720p on EOS M).

The two recorders are pretty much independent (Canon's H.264 has no idea somebody else is also recording, and ML's raw recording doesn't care what Canon's H.264 does - it simply starts and stops H.264). They both get their image data from LiveView, so they will be in sync (without doing anything special).

The start and end moments will differ though (H.264 is started first and stopped last).

Bonus - you get free sound, without any coding effort from our side.

beauchampy

  • Senior
  • ****
  • Posts: 298
Re: Recording RAW and H.264 at the same time
« Reply #4 on: February 11, 2017, 05:18:51 PM »
This is amazing! I can finally do away with my Ninja Blade for a backup recording and switch to something like a smallhd 502.

I haven't tested yet, but does it work with MLV and audio?

Danne

  • Hero Member
  • *****
  • Posts: 3245
Re: Recording RAW and H.264 at the same time
« Reply #5 on: February 11, 2017, 05:19:32 PM »
Quote
Bonus - you get free sound, without any coding effort from our side.
Yes, noticed. Major bonus!

Oki, tested short with latest commits raw_video_10bit_12bit branch. I,m on the run. Make sure to select the latest mlv_lite.mo module from a1ex.
Builds here. BLEEDING EDGE! Be careful. These build are probably up in the experimental section already.
eosm
download from experimental build section instead
5D mark 3
download from experimental build section instead

On eosm when selecting 10bit actually produces some really nice mv1080 files now. Some corruption still in files but a lot are looking very nice. Developed in cr2hdr.app(mlv_dump). Opened up in acr. Pixel maps are not correct here but apply cs2x2, should work.
shortened MLV file here:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/mv1080_10bit.MLV


*update
Seems raw_twk.mo isn,t linking symbols again?
The linking issue is related to crop_rec.mo. Probably in conjunction with the mlv_lite.mo module.
Now working.

Hans_Punk

  • New to the forum
  • *
  • Posts: 35
Re: Recording RAW and H.264 at the same time
« Reply #6 on: February 11, 2017, 05:23:58 PM »
This can actually be very useful. Some users have been asking about this "feature" of shooting an H.264 along with raw video. It is a solution for a quick on set confidence check playback. If it is frame accurate it can also be sent to editorial to start the edit while the raw files are being processed.


YES! - This would indeed make a wonderful feature. Playback is probably the major bug bear I've encountered on shoots when trying to convince others that a 5D3 ML setup is working...trying to get others to understand how amazing the slower/low res raw preview playback can be very unimpressive to people on set who simply just want to check the previous take in real-time playback as if shot regular H.264. Having the ability to know that a higher resolution raw master is being captured separately....would be totally awesome. Extra bonus being the H.264 would contain audio track for scratch sound to help sync external recorder etc.

A 100% accurate framing of H.264 playback (confidence proxy) would probably be the most useful feature for production use and editorial efficiency that I could think of right now.


Ilia3101

  • Member
  • ***
  • Posts: 200
Re: Recording RAW and H.264 at the same time
« Reply #7 on: February 11, 2017, 05:47:19 PM »
Awesome! I've tried to code this myself before(and failed).
I compiled, but it did not work on my 5D2. When proxy was enabled, it would just start h264 recording and would not allow it to stop until raw video was disabled through trash can(constantly said 'starting')
5D2

borpozky

  • New to the forum
  • *
  • Posts: 8
Re: Recording RAW and H.264 at the same time
« Reply #8 on: February 11, 2017, 06:53:39 PM »
Nice to see that this is happening.

I was nearly torn apart, when I mentioned simultaneous RAW & H.264-recording in this forum some years ago ...
Anyway, ML always surprises. It is now my failproof maintool and I love it.











Lars Steenhoff

  • Member
  • ***
  • Posts: 242
Re: Recording RAW and H.264 at the same time
« Reply #9 on: February 11, 2017, 06:55:33 PM »
Great !  thanks for the builds, will test them now

This and the crop module can work together?

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 2956
  • 5D3 / M1 / 7D / 70D / SL1
Recording RAW and H.264 at the same time
« Reply #10 on: February 11, 2017, 07:01:31 PM »
Downloaded ml-raw_video_10bit_12bit_2017Feb17.5D3123 from experimental page (https://builds.magiclantern.fm/experiments.html).

Thanks @a1ex and ran the MLV lite (w Proxy H264 enabled ON) and reordered for about 8-9 seconds before I get a message (from Canon) stating that The Movie Recording has stopped which then required a battery pull out for reboot. This is reproducible.

Question though how exactly do I tell the camera to record H264 to SD and RAW onto CF card? I checked in the preferred card menu but don't see anything in particular for the H.264 under Card Settings within ML menu.
5D3.113 • 5D3.123 • EOSM.203 • 7D.203 • 70D.112 • 100D.101

Danne

  • Hero Member
  • *****
  • Posts: 3245
Re: Recording RAW and H.264 at the same time
« Reply #11 on: February 11, 2017, 07:10:48 PM »
In canon menu select your SD card
In ML menu select preferred card type CF

Danne

  • Hero Member
  • *****
  • Posts: 3245
Re: Recording RAW and H.264 at the same time
« Reply #12 on: February 11, 2017, 07:16:55 PM »
So poking around 10bit mv1080 files created while recording proxy files on the eosm. Not really knowing what is happening here I do see the files come out corrupted and in strange order I wonder if the raw stream buffer suddenly is leaving the raw_slurp buffer while filming proxies which would be causing corruption again? Does it make any sense?

DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 2956
  • 5D3 / M1 / 7D / 70D / SL1
Recording RAW and H.264 at the same time
« Reply #13 on: February 14, 2017, 05:44:57 AM »
Thanks Danne for the tips w SD/CF selections and yes it makes sense to me re: raw_slurp buffer issues.

Also thought of something that may be worth asking in here while we're at it -- would it be possible to have our cameras trigger the recording on external recorders through HDMI out (thus using the single from H264 proxies) and this way we'll be able to have all the footage synced up w ProRes422HQ back up files on the Atomos Shogun as well as using the H264 Proxies for audio syncing and MLV RAW files (for the grade quality obviously) all together nicely?

As we all know that currently if we were to record MLV RAW then have to press record for the Atomos either before or after pressing record on the DSLR which is obviously annoying.

Not as bad as it could get but could this new feature help solve this particular issue in a way w some sort of a hack?
5D3.113 • 5D3.123 • EOSM.203 • 7D.203 • 70D.112 • 100D.101

Licaon_Kter

  • Hero Member
  • *****
  • Posts: 513
  • M
Re: Recording RAW and H.264 at the same time
« Reply #14 on: February 14, 2017, 10:00:23 AM »
M can handle this?  :o

Lars Steenhoff

  • Member
  • ***
  • Posts: 242
Re: Recording RAW and H.264 at the same time
« Reply #15 on: February 14, 2017, 10:43:55 AM »
I tested 14 bit raw on the 5d3 with h264 to the sd and after few seconds it stopped.

I see in bitbucket some new commits that may help, so will test again when there is a newer build


@DeafEyeJedi.  yes the record trigger also crossed my mind.

I have a ninja blade so will check if it triggers.

dfort

  • Hero Member
  • *****
  • Posts: 1799
Re: Recording RAW and H.264 at the same time
« Reply #16 on: February 14, 2017, 07:10:19 PM »
M can handle this?  :o

Yes it can. In fact this H.264 proxy video feature was sort of created to get around the lack of mv1080 video mode on the EOSM.

Check it out:

https://www.magiclantern.fm/forum/index.php?topic=16608.msg179969#msg179969

So not only does it work, but so does mv1080 on the M.
EOSM.202 EOSM.203 EOSM2.103 700D.115 5D3.*

ricbi1

  • New to the forum
  • *
  • Posts: 14
  • Shoot - Shoot - Shoot
Re: Recording RAW and H.264 at the same time
« Reply #17 on: February 15, 2017, 08:38:10 PM »
The fact that this works on 5d3 1.13 having h.264 proxy is incredible! Thanks to the Dev's for this. Never thought it would be possible!
 :)

budafilms

  • Hero Member
  • *****
  • Posts: 711
  • 5D Mark a1ex
Re: Recording RAW and H.264 at the same time
« Reply #18 on: February 17, 2017, 05:50:38 AM »
@A1ex
it's possible tweak the Canon zoom from 3x to 2X or less? In Canon Menu on 5d3 it's possible put 10x 5x etc.
(I mean, use the sensor data to get raw/h264, but not too cropped).

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10180
  • 5D Mark Free
Re: Recording RAW and H.264 at the same time
« Reply #19 on: February 17, 2017, 08:46:02 AM »
Not sure what are you asking; best guess: you can use the full-screen magic zoom to do a 1:1 preview on the YUV buffer (which is what goes into H.264).

On 5D3, it's only available in 1.1.3.

pc_bel

  • Member
  • ***
  • Posts: 108
Re: Recording RAW and H.264 at the same time
« Reply #20 on: February 17, 2017, 10:22:41 PM »
Tested: https://bitbucket.org/Dannephoto/magic-lantern/downloads/magiclantern-Nightly.2017Feb11.5D3113_proxy.zip
For me 14bit recording + H264 only about 30s. (CF Lexar 64Gb 1066x + SD-HC Transcend 16 Gb class10). Maybe due to a SD card not fast enought?
10 and 12 bit + H264 continuous.

Everytime I turn on camera, I need to change prefered card to SD in Canon menu. A bit anoying...

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10180
  • 5D Mark Free
Re: Recording RAW and H.264 at the same time
« Reply #21 on: February 17, 2017, 10:37:57 PM »
For me 14bit recording + H264 only about 30s. (CF Lexar 64Gb 1066x + SD-HC Transcend 16 Gb class10). Maybe due to a SD card not fast enought?

In your case, the raw stream stopped first (not the H.264 one), so the SD card speed is not an issue. It's probably just the extra overhead.

Everytime I turn on camera, I need to change prefered card to SD in Canon menu. A bit anoying...

Disable preferred card from ML menu, or set it to SD.

I can try to force the H.264 card to SD all the time (without you having to configure anything in Canon menu), if there's a good reason to do so. Currently, on this branch, RAW is always saved to CF as long as you have one in the camera.

pc_bel

  • Member
  • ***
  • Posts: 108
Re: Recording RAW and H.264 at the same time
« Reply #22 on: February 18, 2017, 07:05:24 PM »
@a1ex
Thanks for your answer!
Quote
It's probably just the extra overhead.
Can I do something to avoid this and get 14bit+H264 continuous?... Changing to a small aspect ratio?

Quote
Everytime I turn on camera, I need to change prefered card to SD in Canon menu. A bit anoying...
Solved: set preferred card from ML menu to SD. As you said if I have a CF card in camera raw is saved to CF.
Thanks!!!

erikbaldwinson

  • Freshman
  • **
  • Posts: 67
Re: Recording RAW and H.264 at the same time
« Reply #23 on: March 13, 2017, 09:21:54 AM »
Can someone provide a link where the latest firmware for this can be downloaded?

Is there the possibility of simultaneously deleting the h264 proxy and corresponding RAW file from the canon playback menu?

Thank you!

Danne

  • Hero Member
  • *****
  • Posts: 3245

marcb

  • New to the forum
  • *
  • Posts: 9
Re: Recording RAW and H.264 at the same time
« Reply #25 on: March 14, 2017, 07:40:46 PM »
This is a great idea. Tested it and it works well. However, would it be possible to have the proxy h.264 recording together with the non-lite flavour of MLV (with sound).
That would make post-production syncing a lot easier. Thanks!

larrycafe

  • Freshman
  • **
  • Posts: 65
Re: Recording RAW and H.264 at the same time
« Reply #26 on: March 16, 2017, 04:11:22 PM »
what about separating chroma channels to be recorded with RAW video? could it further cut the RAW video bitrate?

combining the chroma channels with the H264 compressed video in post, this should give better video quality for the slow write speed camera

quality between full RAW and H264, while giving longer record time perhaps.


dmilligan

  • Developer
  • Hero Member
  • *****
  • Posts: 3203
  • 60Da / 1100D / EOSM

Pyriphlegethon

  • New to the forum
  • *
  • Posts: 6
Re: Recording RAW and H.264 at the same time
« Reply #28 on: April 16, 2017, 12:10:54 AM »
As it currently stands, this isn't meant to be used as an online/offline proxy solution, right, it's more of a 'confidence check'? With the h.264 file starting before and ending after the raw, and having no option for timecode (unless with the 5D3 there is a way of ML 'reading' camera timecode I don't know about) there's really no current solution for re-conforming offline proxies to online raw in post, cierto? Not to mention filenames are different. But the h.264 file can be set to inherit timecode from the camera clock which ML has more than frame-accurate access to no?

Finishing a shoot, popping the SD card into a laptop, editing, and reconforming and processing only the needed ML clips would be quite the treat. The only camera that does this well for me currently is any DSMC2 body from Red and even then it's on the same media as the raw file which is less convenient. The lack of a simultaneous proxy solution was the one thing that kept me from buying an Ursa mini pro. Would be delightfully humorous if ML ends up being able to do this ;)

domasa

  • New to the forum
  • *
  • Posts: 38
Re: Recording RAW and H.264 at the same time
« Reply #29 on: April 30, 2017, 11:24:25 AM »
Is there the possibility of simultaneously deleting the h264 proxy and corresponding RAW file...?
It will be greate! Please do it.

Ilia3101

  • Member
  • ***
  • Posts: 200
Re: Recording RAW and H.264 at the same time
« Reply #30 on: August 02, 2017, 08:38:22 PM »
@a1ex What would it take to get h.264 proxy working on the 5D mark 2?
Could you give me a pointer if it is not a difficult task ;)
If it is difficult, then I guess it can stay broken for some more time :(

I think it would be a good feature to put in to unified if it gets to a point where it works on all cameras ;)
5D2

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10180
  • 5D Mark Free
Re: Recording RAW and H.264 at the same time
« Reply #31 on: August 02, 2017, 09:02:54 PM »
Probably just troubleshooting it.

You could break the task into simpler ones, for example:
- can you allocate memory with shoot_malloc and/or srm_malloc during H.264 recording?
- can you record H.264 while the LiveView RAW stream is active?
- if not, is there any conflict over EDMAC channels? can it be solved by using other channels?
- does the raw recording code get stuck anywhere?
- if yes, find out where (using printf, NotifyBox, LED blinks, special effects or whatever else you find easier)
- anything unusual in Canon's debug messages? (warning: in LiveView, you'll get a few MB of messages within seconds, and you may have to reduce the FPS)

Ilia3101

  • Member
  • ***
  • Posts: 200
Re: Recording RAW and H.264 at the same time
« Reply #32 on: August 02, 2017, 10:50:00 PM »
First step:
- can you allocate memory with shoot_malloc and/or srm_malloc during H.264 recording?

Ok I don't know if I'm testing correctly, but here's what I did (in the h264 proxy branch)
Added this in line 1699 of mlv_lite.c: (some original code left in for context)
Code: [Select]
    if (h264_proxy)
    {
        /* start H.264 recording */
        ASSERT(!RECORDING_H264);
        movie_start();
    }

    struct memSuite * testmem = shoot_malloc_suite(20);
    if (testmem != NULL)
    {
        NotifyBox(5000, "shoot_malloc success");
    }
    shoot_free_suite(testmem);

    //testmem = srm_malloc_suite(20);
    //if (testmem != NULL)
    //{
    //    NotifyBox(5000, "srm_malloc success");
    //}
    //srm_free_suite(testmem);
Is this the right way to test successful memory allocation in ML? I hear stdlib isn't perfect in ML, so not 100% sure.
(I individually commented out srm and shoot and recompiled twice, both allocations seem worked)

I think I'll test this one next:
- does the raw recording code get stuck anywhere?
As it seems easier than these:
- can you record H.264 while the LiveView RAW stream is active?
- if not, is there any conflict over EDMAC channels? can it be solved by using other channels?
5D2

reddeercity

  • Hero Member
  • *****
  • Posts: 1378
Re: Recording RAW and H.264 at the same time
« Reply #33 on: August 03, 2017, 07:15:08 AM »
I tried this out about 2 mouths ago or so , if I remember right I had both h264 & raw enabled at same time but it defaulted to only h264 recording instead of both .
I see if I can find that build .

Ilia3101

  • Member
  • ***
  • Posts: 200
Re: Recording RAW and H.264 at the same time
« Reply #34 on: August 03, 2017, 11:53:27 PM »
Ok it seems to get to
Code: [Select]
hack_liveview(0);and freezes there... until I go to the trashcan, disable raw video, then stop it myself.

The line right after that is
Code: [Select]
edmac_memcpy_res_lock();"edmac"... sounds familiar...
- is there any conflict over EDMAC channels? can it be solved by using other channels?
I'm guessing it must be an EDMAC channel conflict. Do you think this is the case?
... sorry to be bothering you again how do you change EDMAC channels?

EDIT: found it, changed the channels around within allowed range in the comments, did not seem to help. So I guess it might not be an EDMAC conflict.
5D2

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10180
  • 5D Mark Free
Re: Recording RAW and H.264 at the same time
« Reply #35 on: August 04, 2017, 04:33:51 PM »
You can use the "edmac" module to diagnose it. Find the free channels in standby mode (to cross-check the existing configuration), then see if anything changes while recording.

Ilia3101

  • Member
  • ***
  • Posts: 200
Re: Recording RAW and H.264 at the same time
« Reply #36 on: August 04, 2017, 04:52:08 PM »
Hmm, you mean the EDMAC thing in the debug tab?
I looked at it, but don't understand what's going on.

Here's a screenshot while recording h.264 (no RAW):



Here's it in liveview not recording:




Doesn't really make much sense to me :-\
5D2

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10180
  • 5D Mark Free
Re: Recording RAW and H.264 at the same time
« Reply #37 on: August 05, 2017, 11:41:32 AM »
Yes, that EDMAC thing was recently moved to a module (check the current nightly).

Unfortunately, the "Find free EDMAC channels" tool does not work while recording, as it goes to PLAY mode whenever the tested channel is busy (that's because in PLAY mode, all channels are free). It might work if you use PauseLiveView instead, but that one must be forwarded to a different task (because check_timeout runs on interrupt, not on a regular DryOS task).

Background info: http://magiclantern.wikia.com/wiki/Register_Map#EDMAC

On this screen, [] are channels and <> are connections.

I should probably draw a diagram, based on my current understanding (which can be found as "executable documentation" in the QEMU branch).

Ilia3101

  • Member
  • ***
  • Posts: 200
Re: Recording RAW and H.264 at the same time
« Reply #38 on: August 05, 2017, 07:52:40 PM »
I downloaded latest nightly - there was no edmac.mo, maybe there's a build error happening?

As it seems finding free channels automatically is troublesome, might have to try and hand pick some more channels next time and see if that works.
5D2

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10180
  • 5D Mark Free
Re: Recording RAW and H.264 at the same time
« Reply #39 on: August 05, 2017, 07:57:59 PM »
I downloaded latest nightly - there was no edmac.mo, maybe there's a build error happening?

Yeah, that's why these things are under review for some months, until nobody bothers to look at them :D

https://bitbucket.org/hudson/magic-lantern/pull-requests/811

12georgiadis

  • Freshman
  • **
  • Posts: 59
  • 5DmkIII - 7D
Re: Recording RAW and H.264 at the same time
« Reply #40 on: August 13, 2017, 10:34:32 AM »
I just tested H264 proxy with sound + 1080p compressed raw and it works perfectly. However h264 are longer. to sync h264 proxy + sound + 1080 raw, I tried to determine the offset. It is around 2s but not regular. Sometimes +1i or 2, sometimes -1i or two. Is there a way to have a stable offset ? If yes, a simple ffmpeg script could sync everything.

Danne

  • Hero Member
  • *****
  • Posts: 3245
Re: Recording RAW and H.264 at the same time
« Reply #41 on: August 13, 2017, 11:17:35 AM »
There is an idea from a1ex about synching H.264 and raw stream by starting off with a black frame(both raw and H.264) when H.264 starts. Can't find the link. I thought it was a cool idea.

12georgiadis

  • Freshman
  • **
  • Posts: 59
  • 5DmkIII - 7D
Recording RAW and H.264 at the same time
« Reply #42 on: August 13, 2017, 09:25:14 PM »
And the black frames appear together Both on h264 and raw ? With the same number of iframe ? It sounds interesting


Envoyé de mon iPhone en utilisant Tapatalk

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 10180
  • 5D Mark Free
Re: Recording RAW and H.264 at the same time
« Reply #43 on: August 17, 2017, 01:17:29 AM »
I should probably draw a diagram

Done - check the "edmac" branch and this topic.

12georgiadis

  • Freshman
  • **
  • Posts: 59
  • 5DmkIII - 7D
Re: Recording RAW and H.264 at the same time
« Reply #44 on: August 19, 2017, 10:13:16 PM »
Can we have more details with the black frame sync method ? If there is an auto-sync solution, that would Be awesome and very usefull with compressed raw! :) thx


Envoyé de mon iPhone en utilisant Tapatalk