Sub-millisecond syncing of raw video on two 5D3s

[Jonneh]

Through a strange (and wonderful) turn of events, I've ended up with two 5D3s in my hands, and I plan to have some fun with them for an upcoming short film. Things I'd like to play with range from high dynamic range capture, stitching for double resolution, fusing lens effects, to fusing different parts of the EM spectrum. Most of these require close to pixel-level alignment of objects in the frame for good results. In this context, I'm not too worried about parallax error since most scenes would be shot at long distance, but inter-frame synchronisation between the two cameras is going to be an obstacle.

Based on examples I've seen (e.g. https://joancharmant.com/blog/sub-frame-accurate-synchronization-of-two-usb-cameras/), I think I'll need sub-millisecond synchronisation to have acceptable alignment and avoid excessive loss of detail/double images. At 25fps, I can expect anywhere from 0 to 20ms of misalignment with close to random initiation of recording at this temporal resolution. Launching randomly (that is, random with respect to this high temporal resolution), I make it that I will have to make about 50 attempts to have a 90% chance of happening upon a sub-millisecond inter-frame difference between the two cameras' streams --- not practical.

Two questions then arise: 1) how I can know whether I've achieved this and 2) how this hit rate can be improved upon.

For 1), I can use the process described in the above link: set up a strobe synced to the frame rate and play with the duration such that I can align the resulting banding between the two cameras.

For 2), does anyone have any ideas about which of ML's features can be leveraged to get pretty close to sub-millisecond synchronisation, such that I might only require a few attempts to get a very small gap between the two streams? I'm wondering if the pre-record option in the RAW video menu combined with the recording trigger option might ready the buffer such that on launching recording with a Y-split remote release cable or similar, the latency might be low enough to consistently get low single digit offsets, for example, from whence I can simply retrigger manually until I get something acceptable.

P.S. I'm optimistically assuming that there won't be any precession of one stream in relation to the other over time, but I have no idea whether or not this will indeed be the case.

[names_are_hard]

I'm doubtful anyone has tried to do this before, or needed to care about careful sync between two cams with ML.  So the answer to most of your questions is going to be: we don't know, try it and see.

If the strobe banding reliably shows on LV preview while recording, it should be possible to fine-tune fps on one cam till it drifts into sync, and then lock the fps the same on both cams.  Might be accurate enough?

The cams have a high precision timer but it's reinitialised whenever the cam sleeps / turns off.  You could perhaps schedule recording to start with high precision (untested, I have no idea how much jitter is present when starting recording).  But even if this works it doesn't help you without a means to sync the two cams clocks.  Might be possible via USB / PTP with external hardware.

[Jonneh]

Cheers names. I'm certainly going to be jumping in as soon as I can, but your initial ideas are very helpful. I really like the frame-rate tweaking idea --- it certainly works on my mental simulator, and hopefully in practice too.

Syncing the clocks also sounds good, albeit requiring more work, and using PTP to do it sounds like it is beyond my current abilities as a programmer/hardware tinkerer. I assumed that initiation jitter would be too much for this kind of technique to help, but perhaps that is unfair.

The more I think about it, the more things seem possible with this kind of setup beyond those listed above (3D, intentional parallax shifts, double framerate recording, focus stacked video...). If it hasn't been done, perhaps that means I'm about to run into a bunch of obstacles, but I suppose we'll see.

EDIT: I don't have the second 5D3 yet, but I had a look into producing the strobe banding that should enable me to synchronise the offsets. I expected to need an Arduino, but I found a fantastic little Android app called Strobily that strobes the torch with controls for frequency (to three decimal places) and duty cycle (1% increments, equal to 400 microseconds at 25 fps ). I never managed to catch a whole band (at low duty cycles) but increasing the duty cycle makes it easy enough to catch a leading or trailing edge, which should be enough. There is some degree of jitter from frame to frame (I would estimate a standard deviation of perhaps 20% of the 10x zoom frame, but there is little to no precession over time, so three cheers to both ML (with FPS override on) and Strobily. Looking good!

[names_are_hard]

There isn't currently a GUI that will work well to sync via adjusting the frame rate.  You'd need some way to change it while LV was running.  I'm not even sure you can tweak fps while video is recording, though I expect it's possible.

PTP for clock sync likely is more work, but it isn't hard in principle.  The protocol has been reversed to a fair degree and code exists in ML repo for doing bi-directional comms to PC via PTP.  You'd need to work out some way to determine a high-precision offset between two clocks when the transport protocol has lower precision, but this isn't too hard.  Then you send that to one of the cams and now both can act at the same time.  E.g., when record signal is sent, both know to start recording when the adjusted HP timer is next at an exact round second time.

I suspect initiation jitter may be quite low.  The HP timer is microsecond precision.  Checking some old logs the granularity might be 30us.  You'd have to test it to see how the whole chain performs.

[Jonneh]

Changing the frame rate from the FPS override menu does work while recording! I did a test filming a stopwatch and ramping up the fps from 5 to 20, and the maths in the output videos comes out as expected. I haven't checked whether fine tuning also works on the fly, although I suppose the default expectation is that it would work in the same way. The FPS override function is particularly buggy, so I required several battery removals before I got the test running, but I'm sure I'll identify the precipitating combinations of factors in due course.

The proof of the various aspects of the method will be in the pudding, of course. If it doesn't pan out, or doesn't produce an accurate enough result, the way you describe the functioning of the clocks and a general route to syncing them is very encouraging. Thanks! Anything in the hundreds of microseconds should produce good results.

Not long now until the second body is with me.

[names_are_hard]

Cool, doesn't sound obviously impossible then   It's a fun project.

The PTP clock sync route I think is more upfront pain, perhaps not worth it unless you want to sync several cams.  With a beam splitter and well sync'd, slightly offset clock cams, you could do arbitrary high fps, which would be super cool.  But if you only need to sync two cams, maybe too much work getting it reliable.