This post is true24 hours have passed. So, how does it really work?
I didn't even lie very much! There is an FPGA. I'm not using it. You probably can do cool things with it, but I don't know how it works.
There is no video trickery, there are no wires.
200d has wifi. I've got Magic Lantern to use that, beam the LV data to my PC in real-time, and run object detection there. Detected object coordinates are sent back, and ML draws the boxes. I am really doing YOLO object detection, which really is a DNN algorithm.
We can do real-time machine learning processing, in a useful sense! You can do it in the field too, tethered through your phone, I've tested this and it works. The cam is not plugged into anything, it's not some HDMI trick. You could replace object detection with anything you wanted. You could upload your images as soon as you take them, in the background (but I warn you, the wifi on the cam seems quite slow, it would be about 15s per JPG).
Technical details below.
I cleaned up earlier network code from 6D and Turtius:
https://magiclantern.fandom.com/wiki/6D/Networkinghttps://github.com/turtiustrek/magiclantern_simplified_minecraft/commit/2bf635a77a17c19d5d6ef4800f80f4caf5ea9989Turned that into an okay not great quality socket / wifi header:
https://github.com/reticulatedpines/magiclantern_simplified/blob/b8727b0d5b423ee7735f425a70bfe6fddd36b2b1/src/ml_socket.hWrote a module for the cam to read LV, strip the chroma bytes to save network bandwidth (YOLO doesn't use colour), and send to the server. This is 200D only for now, but easy to port to cams that have Wifi. Network config is stored on the card.
https://github.com/reticulatedpines/magiclantern_simplified/blob/b8727b0d5b423ee7735f425a70bfe6fddd36b2b1/modules/yolo/yolo.cWrote a fairly simple server that handles the cam data and does the image processing.
https://github.com/reticulatedpines/magiclantern_simplified/blob/b8727b0d5b423ee7735f425a70bfe6fddd36b2b1/modules/yolo/yolo_server.pyThat branch will get removed when I integrate the code into dev. There's a few things that need improving first.
The server sends back detection box coords, which the module displays. The lag is almost all network related. Object detection takes 20ms (GPU accelerated). But it's taking 200-500ms to send 300kB of LV data. Sometimes it lags and takes > 3s. This is much slower than it should be, it's 802.11g so the theoretical max is around 5MB/s, and we're getting... 0.6MB/s. The 6D networking notes suggest this is limited by the priority of the Canon networking tasks, so perhaps this can be tweaked. I've tried a few things in that area but no luck so far.
OpenCV and Cuda are doing the heavy lifting. They can do a lot of things, and 500ms response time is fine for some tasks. Got an idea for something cooler, or more useful? There really are so many things we can do with off-cam CPU / GPU processing!
