Using the Toshiba FlashAir card for ML development

[a1ex]

I've just got this card hoping it will reduce wear and tear on my cards and card readers. As you can imagine, constantly swapping the card between camera and PC is not pleasant for any of the devices involved (card, card reader and card slot from the camera - they all suffer).

Short review: (my model is W-03 8GB)

- SLOW (1 minute for downloading a 18MB file...)
- quite hard to set up (it took me about 3 hours from unpacking to being able to copy files on it)
- some things just don't seem to work (such as internet passthrough, or manually enabling/disabling wifi)
- formatting the card removes wifi functionality (!)
+ it has a nice logo with Toshiba printed on it
+ it has documentation, developer forum, Lua scripting, all sorts of bells and whistles (too bad the basics aren't working well...)

Side note: a while ago g3gg0 got a Transcend wifi card, and he mentioned it's very slow, so I hoped this one would be better. Looks like it isn't.

How to set up on Linux

So, to save you from hours of fiddling, here's a short guide on how to get it working on Linux, to the point of being able to run "make install" - that is, to upload ML on it without any cables:

Formatting:
- make a backup copy of the files from the card, in particular:
     - DCIM/100__TSB/FA000001.JPG
     - SD_WLAN/*
     - GUPIXINF/*/*
- format card (from camera or from any other device)
- put the files you just backed up, back on the card (to restore wifi functionality)

Network setup (assuming you have a wireless router):
- put this in the config file:

Code: [Select]

APPMODE=5
APPSSID=<your network name>
APPNETWORKKEY=<your network password>
UPDIR=/
UPLOAD=1
WEBDAV=2
- power-cycle the card, so it will connect to your router, just like any other device on the network
- configure your router so it always assigns the same IP address to the card
- check the new ip (ping the card)

Uploading ML:

See below.

Old instructions:
- mount the card as WebDAV in your favorite file browser (on my system: dav://192.168.0.111/)
- find out where that location is mounted, e.g. by opening some file in a text editor and checking its path
- on my system, that path is: /run/user/1000/gvfs/dav:host=192.168.0.111,ssl=false
- put this in Makefile.user (in the directory containing ML source code)

CF_CARD=/run/user/1000/gvfs/dav:host=192.168.0.111,ssl=false
UMOUNT=gvfs-mount -u dav://192.168.0.111/ && echo unmounted

- make sure the camera is not writing to card (important!)
- run "make install" from the platform/camera directory.
- restart the camera by opening the battery door (important! this ensures the camera will no longer write to the card)
- make the card bootable if it isn't already (for example, copy ML-SETUP.FIR manually on the card and run Firmware Update)
- reboot the camera to start the latest ML you just uploaded.


"Unbricking" the card - if you have formatted it by mistake without a backup

- download Toshiba FlashAir utility from here
- install it (installation works under Wine, the utility doesn't)
- find the files you should have copied before formatting, here: c:/Program Files (x86)/TOSHIBA/FlashAirTool/default/W-02
- these files appear to work with W-03 as well; just copy them to the card, then reconfigure it

(note: I do have an Windows XP box, but couldn't manage to install that utility on it; didn't try too hard, gave up after 15 minutes or so)

TODO

- make qinstall (to only upload autoexec and maybe the modules you are working on - did I mention it's SLOW?)
- [DONE] restore Toshiba files after format (so you don't lose wifi capabilities when formatting the card from the camera)

[eduperez]

Very interesting!

Those cards do not seem too expensive, about 20€ from Amazon delivered around here, plus another 10€ more for an adapter to Compact Flash. Did you try that card with a CF adapter, by the way?

Reading your post, I understand that Windows is only needed in case someone bricks the card, and any other procedure can be performed from a Linux box; is that correct?

Many thanks!

[a1ex]

Didn't try with an adapter.

Windows is not needed, even if you brick the card; you can recover it only from Linux (you have to install their utility under Wine, but you can't run the installed program, so you have to copy some files manually).

I'm thinking at some way to execute commands remotely on the camera, maybe Lua script commands. The card has the ability to execute a Lua script either when you access a URL from a computer, or triggered when the host device (camera) writes a file on the card (not sure how it figures out when the file is completely written, though).

I wonder what happens if both camera and somebody else (Lua script, wifi interface) write to the filesystem both at the same time; the documentation doesn't say anything about it.

The firmware is not encrypted; you can just download their firmware update and run "strings" on the executable to see what's inside. Some interesting links from those who already did:

http://elm-chan.org/junk/fa/faff.html
http://hackaday.io/project/5558-reverse-engineering-toshiba-flashair-wifi-sd-card
http://www.extrud3d.com/flashair
http://geekitguide.com/wifi-sd-cards/

and also:
http://www.bunniestudios.com/blog/?p=3554

[eduperez]

Somebody already tried to use a Transcend WiFi SD card and stablish some sort of communication between a camera and the external world, by means of writing and reading specific files; there are more details at https://chdk.setepontos.com/index.php?topic=3290.1790, but apparently the camera caches the content of the card, and that breaks everything.

There is also a warning at the developers site about writing to the card from the camera and a computer at the same time: https://www.flashair-developers.com/en/documents/api/uploadcgi/

[a1ex]

Very helpful points, thanks. I did some quick tests, and here are my first thoughts:

- I believe Canon code only caches the filesystem, but not the data from the files (not 100% sure about this)
- therefore, your "armchair idea" about "FAT does not move files around, even if the contents are changed" could be useful
- to check which sectors are used by some file, you may use (source):

Code: [Select]

sudo filefrag -b512 -v -e /media/EOS_DIGITAL/TEST.LUA
- editing a small text file (less than 512 bytes) using Geany, on FAT32 filesystem, does change its location
- editing a similar small text file from the camera (using editor.lua) does change its location
- editing a similar small text file using a Lua script running on the wifi card, with the code below, on FAT32 filesystem, does not change its location:

Code: [Select]

-- code running from another script file, not test.lua
local file = io.open("test.lua", "w")
file:write("print('Hello from Lua!')\n")  -- bug: last character is not saved (?!)
file:close()
- after editing a script file from camera (using editor.lua), and invoking the script via HTTP request from PC, the latest version of the file is executed
- execution time for a simple hello world script is about 0.1-0.2 seconds (time wget -N http://192.168.0.111/hello.lua)
- after the script updates a small text file (test.lua with the above file), the updated file is displayed in ML file manager

In a nutshell, if the processes on the camera are allowed to write any files (as usual), and the wifi card is only allowed to change a small text file that does not change its location on the card, it looks like we have the building blocks for sending small Lua commands to the camera (via HTTP request, saved to a fixed file), executing it (polling the file from ML) and getting back the results (text log, optional screenshot or whatever).

[a1ex]

After a long fight with the quirks of this card (which could easily win the prize for the most unreliable device I've ever used), I've got a working proof of concept:

Code: [Select]

$ lua wifi_console.lua
> help
Special commands: exit quit snap
> 2+2
4 
> print(a)
[string ""]:1: variable 'a' is not declared
> a=5
> print(a)
5 
> a+7
12
> snap



Source code

Timings:
- after you press ENTER, script gets uploaded (confirmed by the "..." prompt) in about 0.1 seconds
- it starts executing on the camera in about 0.5...1 seconds (can be improved)
- results are received back to terminal after 1...2 seconds
- screenshot takes about 4 seconds

Tip: to see Lua errors, go to (undocumented) http://[IPofSDCard]/eva.cgi.

Anyone interested? maybe in implementing the same idea on other wifi cards?

[a1ex]

If anyone is interested in reverse engineering the firmware of this card, please find a Lua-based dumper here:

https://bitbucket.org/snippets/a1ex/nrXoA

Code: [Select]

   1001 [ BootProgram  ]:Ver08.01 13:04:44,Jun 24 2014


[Walter Schulz]

Just trolling/brainfarting around:
If I understand correctly Canon's firmware does allow non-exclusive access to memory card. Correct?

So - in theory - it might be possible to create a break-out-box for SD/CF with some additional hardware/software allowing access to file system via WiFi or whatever interface you want to have?

[a1ex]

Canon firmware is unaware of what the wifi card is doing - they both think they can write to the filesystem.

In particular, Canon code (including the FIO routines called by ML) assumes it has exclusive access to the filesystem (it has no reason to assume otherwise), so it may want to cache some stuff (for example, the file list, or even individual files). That means, it may not see the changes done to the filesystem, by the wifi card, until reboot.

The problem becomes worse when both entities want to write something to the filesystem. For example, the wifi card writes some file and updates the FAT. Canon code also wants to save something there, but it already has the file list cached, and it's unaware of the first write. It may work just fine if they touch different areas on the FAT and on the card, or the second file may overwrite (totally or partially, depending on luck) the first file, or the FAT itself can get messed up.

So, my method for uploading things only works fine if you don't write anything else to the card from the camera side (in particular, disable ML config file saving at shutdown).

I'm not sure I understood your questions though.

[Walter Schulz]

Thanks!
Brainfart totally ignored FAT issue you pointed out.

Post was caused by daydreaming about ML and PTP. Case closed for now.

[a1ex]

Alternative to opening the battery door after uploading via wifi:

Code: [Select]

    void (*FSUunMountDevice)(int drive) = (void*) 0xFF4827D4;   /* 5D3 113 */
    FSUunMountDevice(2);  /* 1=CF, 2=SD */
   
    /* after running this, upload files via wifi, then reboot the camera */


update: got a nicer way to run "make install" on the wifi card:

https://bitbucket.org/hudson/magic-lantern/pull-requests/726/helpers-to-run-make-install-on-wi-fi-sd/diff

[eduperez]

If there is a method to umount the device, there should be a method to mount it... 

[a1ex]

Yes, it is. For this purpose, I also have to restart, so why bother?

For the interactive console, it might be useful, but that's a lot more complex to get right.

[nkls]

I got one of these the other day and have been playing around with it a bit.

Some interesting stuff:

There seems to be a telnet server in the firmware which can be enabled by adding:

Code: [Select]

TELNET=1to the CONFIG file. If I connect I get a welcome message

Code: [Select]

Connected to 192.168.1.116.
Escape character is '^]'.
Welcome to FlashAir
ESC R4161 built 16:20:27, Oct  6 2014
>but I'm not able to enter any commands. Maybe there is some login required?

There is a builtin fdump utility that can be used to dump memory content at startup.

Code: [Select]

COMMAND=fdump 0x00C00000 -f fdump.dat -l 0x00100000
Sometimes this hangs the card, but I've managed to dump segments as large as 0x00200000 bytes. There is also a -m flag (mount filesystem) I haven't tried yet which might make things more stable.

The processor seems to be some Toshiba MeP processor. The bytecode can be interpreted by binutils configured with --target=mep. (link for the lazy)

Code: [Select]

binutils-mep/bin/mep-objdump -D -b binary -mmep rom.bin
00000000 <.data>:
       0:       08 d8 01 00     jmp 0x100
       4:       18 df 08 00     jmp 0x8e2
       8:       00 00           nop
       a:       00 00           nop
       c:       00 00           nop
...
      fe:       00 00           nop
     100:       00 70           di
     102:       28 59           mov $9,40
     104:       59 79           stc $9,$cfg
     106:       00 59           mov $9,0
     108:       58 79           stc $9,$rpe
     10a:       03 eb a0 41     lw $11,(0x41a000)
     10e:       b5 cc 00 10     and3 $12,$11,0x1000
     112:       b5 cb 20 00     and3 $11,$11,0x20
     116:       2a 6b           srl $11,0x5
     118:       5a 6c           srl $12,0xb
     11a:       c0 1b           or $11,$12
...
     8e2:       48 df 08 00     jmp 0x8e8
     8e6:       02 70           ret
     8e8:       f0 cf ac ff     add3 $sp,$sp,-84
     8ec:       fa 00           sw $0,($sp)
     8ee:       06 41           sw $1,0x4($sp)
     8f0:       0a 42           sw $2,0x8($sp)
     8f2:       0e 43           sw $3,0xc($sp)
     8f4:       12 44           sw $4,0x10($sp)
     8f6:       16 45           sw $5,0x14($sp)
     8f8:       1a 46           sw $6,0x18($sp)
     8fa:       1e 47           sw $7,0x1c($sp)
     8fc:       22 48           sw $8,0x20($sp)
     8fe:       26 49           sw $9,0x24($sp)
     900:       2a 4a           sw $10,0x28($sp)
     902:       2e 4b           sw $11,0x2c($sp)
     904:       32 4c           sw $12,0x30($sp)
     906:       36 4d           sw $tp,0x34($sp)
     908:       3a 4e           sw $gp,0x38($sp)
     90a:       3e 4f           sw $sp,0x3c($sp)
     90c:       1a 71           ldc $1,$lp
     90e:       42 41           sw $1,0x40($sp)
Boot entry at 0x0 and interrupt at 0x4 perhaps?

There is also this iSDIO interface available. It would be interesting to expose it to ML and see what may come out of it.

[a1ex]

Cool stuff - the SDIO interface is definitely worth exploring.

BTW, CHDK already has a remote control app using this card: https://chdk.setepontos.com/index.php?topic=12959

[Walter Schulz]

In case you missed it: Toshiba announced 4th generation FlashAir. Specs are promising. Nothing about price and availability (EDIT: Q2 it is).
http://www.toshiba-memory.com/cms/en/products/wireless-sd-cards/FlashAir/flashair-wireless-sd-card-W04.html

[a1ex]

It just becomes a pain to insert the card a dozen of times

My current workflow:

Network setup: first post.

Makefile.user (card connecting to your router):

Code: [Select]

WIFI_MAIN_NETWORK=your-network
WIFI_CARD_NETWORK=your-network
WIFI_CARD_IP=192.168.100.10

Makefile.user (your PC temporarily connecting to card, and returning to your main network after uploading):

Code: [Select]

WIFI_MAIN_NETWORK=your-network
WIFI_CARD_NETWORK=flashair
WIFI_CARD_IP=192.168.0.1


Uploading ML core (autoexec.bin):

Code: [Select]

export WIFI_SD=y
cd platform/5D3.123
make clean
make
make install                 # also installs modules and data files, but it's very slow
...
make installq                # only copies autoexec.bin and .sym - much faster
...

Uploading a module:

Code: [Select]

export WIFI_SD=y
cd modules/mlv_lite
make clean
make install
...
make install
...

Caveat: after uploading something via wifi, we *must* make sure the camera does not write anything else to the card (because it caches the filesystem, as discussed above, so it may end up overwriting what we just uploaded).

Solutions:
- Clean: Debug -> Unmount SD card (before uploading => I almost always forget doing it)
- Dirty: open the battery door without turning the camera off (so ML config is not saved at shutdown)
- Ugly: disable ML saving config files at shutdown (caveat: you will want the settings saved, sooner or later...)
- TODO: autodetection (possible, PoC described above in this thread).

Mounting the filesystem (so you can browse it remotely):

Code: [Select]

export WIFI_SD=y
make install_prepare

Note: the command "export WIFI_SD=y" must be typed only once (or every time you open a new terminal).

Tip: formatting the card from ML keeps the wifi functionality, and fixes "minor" filesystem corruptions that will appear sooner or later with this workflow (e.g. when the wifi connection drops while copying).

[a1ex]

If anyone is interested in running custom code on this card, this presentation is a must-read.

You may find it useful for any other embedded systems, including cameras or cards from other manufacturers.