File transfer off camera using MQTT

[NohSpamJose]

Hi, just got myself an R6 mark ii. It is possible to transfer files off the camera using FTP but this requires a special app and a Canon pro subscription. FTP is a flawed protocol for this use case.

I worked on IBM MQFTP (not as a coder, as a UX designer) and I've long wanted similar capability in MQTT, which if you're not already aware is a very lightweight (IoT) pub-sub client with roots in MQ.

The use case would allow a user to get their photos off their camera to a target of choice with similar transfer speed as FTP but with much greater resiliency. An MQTT solution has already been architected (I'll find the link and post it) but I think in Erlang, so would need a rewrite in an appropriate language.

The benefits of the pub-sub, queue based model is that the file is broken into smaller sections and each is sent independently and possibly with a degree of parallelism (with a common identifier and a unique order number) the MQTT target knows how many and can reconstruct the whole file as/after the parts arrive. A Mosquito server sitting in the middle (owned or a public cloud one) means that  if a part doesn't arrive, it's sent again, so the transfer can survive intermittent network connections and even where the target is not yet on/networked or the camera loses connection. It's bulletproof file transfer and I would highly recommend a look into this.

Happy to DM for details 

[NohSpamJose]

Existing solution
https://www.emqx.com/en/blog/file-transfer-over-mqtt

[NohSpamJose]

The following is outside the MagicLantern remit but would be just amazing for users

If you want to throw a NodeRed app into the mix, it could manage where and how files are stored and named, which would make a fabulous workflow.

Say a file arrives via MQTT, NodeRed subscribes to that topic and detects the file's arrival. It reads the Exif data and uses that to rename the file and store it in an appropriate place and add additional Metadata tags. It could even use AI to ident family, friend and pets and tag accordingly. Then it could import the file into a Darktable or Lightroom folder and update its database.

3rd parties or enthusiasts could build sophisticated workflows and no proprietary lock in. What's not to like!

[names_are_hard]

I think what you're saying here is "Could ML support MQTT to upload images from a cam, possibly over wifi"?

Yes, in theory, there's a proper networking stack and we know how to use it.  If you specifically mean R6ii, you're out of luck, because it doesn't run ML at all yet, and is likely hard to support generally.

Personally, I think MQTT is likely to add complexity for not much gain.  If the goal is high likelihood of successful transfer, anything TCP/IP is probably good enough.  Nobody will stop you implementing MQTT if you want

See here for me making my 200D upload images to a server, and get back object detection data, in real-time:
https://imgur.com/D4MYBfE

The major problem is currently ML has approximately one dev: me.  At this time I am focused on adding support for new cams (this is going fairly well), not adding new features.  I do generally like the idea of using wifi more on cams that support; a lot of modern cams do and the official Canon networking stuff is rather limited.

[NohSpamJose]

Thanks for your response. I take your points but would just add that using MQTT for file transfer adds robustness in a context where lots of things can go wrong, such as poor network, battery loses power/ battery change or a user choose to prioritize battery power over transfer rate/completion. The transfer can just trickle along as circumstances allow, without the FTP or TCP/IP danger that you'll have to start over - MQTT transfer would just carry on from the point it last stopped. I'd ask Canon but they'd turn it into a subscription feature. It'd be a premium feature for professionals, especially with a NodeRed customisable workflow.

[names_are_hard]

Yes, maybe worth the trades, I'd need to compare MQTT characteristics against a range of other similar protocols.  Some quick reading suggests MQTT has a 256MB file size limit (unless you use extensions), and is inefficient for transfer of large files generally - and even JPG from cam would count as a large file in this context.  Given MQTT seems mostly to be used with IoT, and often (always?) uses store-and-forward, it might not be the best fit.

I don't really know what you want, functionally.  Something about resilient transfers?  Something about NodeRed compatibility?   I don't know what NodeRed is.

If your primary concern is resilient transfers, there are many ways to do this.  Rsync would be one obvious choice and would certainly be easier to implement.  I would think with these cams, compared to IoT devices, trickling the transfer is a bad idea.  You have to keep the cam fully awake to do that.  I would guess "race to sleep" is a better strategy for us.  But again, it would need measuring.

Re FTP, this has been able to resume transfers for at least a decade, probably much longer.  Both the client and server need to support it, but we'd be implementing these.  So this may also be a valid option, again with a simpler protocol.

In any case, I will not be working on this any time in the next six months, so if you want it, you need to find a dev to commit the time.