PTP autoexec.bin uploading(save SD-card door project)

[miyake]

I don't wanna use SD-reader and , open the SD-card door.

https://bitbucket.org/miyake_t/magic-lanternptpdev/
Current repo's code and ptpcam combination can upload 95% of files correctly.
But it's will stop last 6k with error.(only 600D's autoexec.bin)
transferd 240k is correct values .(confirmed "od -x" and diff)


History and knowledge
-based on g3gg0's patch
-added libptp-chdk-win32(It's a GPL so we can redistribute with source)
-src/ptpcam.txt is described how ptpcam is working(how extending ptpcam by chdk guys).
-(solved)enum value is different ML and ptpcam
-(solved)current ptpcam code is not supported over 1000byte files to transfer
-(solved)current ML codes are not used callback function for receiving from USB.(If we dont use it, got hangup receive process)
-(solved)current ML codes not handle USB ack packet(Simply discarding now.)
 *it's 12byte data adding each after part of USB_BULK_WRITE data.
-USB_BULK_WRITE is sending 500byte data .
-console_printf added for debugging


I'm coding for this code 1week . But I can't solved lask 6k issue. So please help me to find bugs in this code, if you interested in this.

[ilguercio]

Why don't you compile a bin for each camera?
Since it's not good to use the same card for different cameras it won't be a big issue if we had a single file for each DSLR.

[nanomad]

The point of the unfified ML tree is not to have a tons of autoexec.bin around

[miyake]

Why don't you compile a bin for each camera?
Since it's not good to use the same card for different cameras it won't be a big issue if we had a single file for each DSLR.

I can't understand what you are advice to me .
What relation is existing about "solving PTP codes" and "each camera autoexec.bin" ?

Sorry for my poor English reading.

[SDX]

This actually is quite cool. I have thought of exact the same thing for a few days now.

What I think ilguercio means is, that he would like to have a autoexec.bin for each camera model. Maybe that would help the 6k issue on 600D?

[miyake]

@SDX
Let's solve this bug together.

BTW, I'm using 600D autoexec.bin(make 600D) , then I got 6k bug. In addition, I was transport  platform/all/autoexec.bin. It's 1.7M. But I could transport over 250k. I forgot detailed value. But I can transport about 1.5M.
Ratio is almost same , so I think the code has bug.(loop condition had bug, I guess).

[ilguercio]

The point of the unfified ML tree is not to have a tons of autoexec.bin around

Not only, it means that the code is shared among the cameras. It doesn't mean it has to be in one single file at any cost.
FW files are specific for each camera.
Anyway, a single bin file is good until it is too large that it can burden the startup sequence. I don't know if it made any difference if it was done for each camera.
SDX has understood me well, it was just a thought so it could help avoiding this bug but it isn't a solution for sure.

[1%]

Try to send file with a different name/ext? Or just after certain size it cuts off? The loader you made can then load the bin or whatever extension. I think its better to make it autoexec.ae vs AE_autoexec.bin, what do you think? Then function renames it properly.

[miyake]

Try to send file with a different name/ext? Or just after certain size it cuts off? The loader you made can then load the bin or whatever extension. I think its better to make it autoexec.ae vs AE_autoexec.bin, what do you think? Then function renames it properly.

I'm always use different file name to store transferd files. and cuts off size is different of each files(different size).So I think my code's loop has a bug , but I can't find it.

Filename has 8.3 limitation, and my code is already have function which is copy .bin and .cfg together.
So we can't use extention. and "AE_" prefix is easy to identify by human. also we can remained 5 character to use identify the file.
Now AE_XXXXX.BIN and AE_XXXXX.CFG will overwrite working autoexec.bin and magic.cfg file.
We can more consideration on prefix naming.
Therefor I'm using AE_ prefix for naming.

[SDX]

I must admit that I don't know much about PTP. But here are my ideas, what could go wrong.

• - Could it be that PTP splits up the file into chunks and you just simply forget the last one? I just skimmed the code and I don't know how the loop you are talking about exactly looks like, but could it be, that you should 'extend' it by one iteration?
• - Also - I don't know if you already mentioned that - have you checked if the error occurs when writing the file? Compare the size of the data you receive with the one you write into a file.
• - Are you 100% sure, that you send all the data?
• - Maybe you stop the transfer too early?
• - Maybe you have an error in the calculation of the size? Looks like, whatever file you send, you miss the last 9% (or factor 1.1333).
  Check if the calculation is right.

Again, I just barely know that PTP is a protocol, but have no clue on how that stuff internally works.

Edit: I just noticed:

-(solved)current ML codes not handle USB ack packet(Simply discarding now.)
 *it's 12byte data adding each after part of USB_BULK_WRITE data.
-USB_BULK_WRITE is sending 500byte data

- Maybe these 12bytes mess up your calculation of the size? I know, 12bytes on 500bytes just makes approx 2%, not 9%. Maybe not as useful as you properly hoped - I do what I can

Another Edit: (I just noticed, that the crap I previously wrote was absolutely useless)
I just found enough motivation to take a closer look at the code. It looks totally okay to me. I assume that fninfo_len is the length of some sort of header which is being skipped, right?

- I would write the received data into a buffer instead of writing it directly into the file. Doing all the writing once is faster.
- Have you tried to simply save everything you receive? Then you could send a small file, 2k or something and take a closer look at, what it ends up with.
- And finally: is context->get_data_size() with or without headers, signatures and that kind of stuff.

Btw: how are you sending files?

[miyake]

I assume that fninfo_len is the length of some sort of header which is being skipped, right?

yes. The transferd byte stream has  4byte filename length and next variable length filename, then actual transfer file data.

[SDX]

Okay

I would like to take a closer look at what you have done and what doesn't work. I just need to know one little thing: how to you send the files? Are you using ptpcam upload or your own application?

[miyake]

my repo has a customized  libptp.  So you can use ptpcam .
And it's a extended for win32 . But I think it can use on linux/Mac, maybe. Try it if you don't use win32.

Then,

Code: [Select]

$ pcpcam --chdk
<conn>  upload ./autoexec.bin B:/AE_PTP1.BIN


[SDX]

Strange issue here:
I'm not able to establish a stable connection. Most of the times I get an error like "Failed to connect". Only the first few seconds after connecting to the camera it is possible to execute something that works. Also, Windows comes up with a "overview-of-the-camera" window, which closes and opens every few seconds. I also managed to crash ptpcam.exe *proud*.

Btw: what value does get_data_size() return? Have you compared it with the size of the file? (I should be slightly bigger than your file, of course).

[miyake]

@SDX

I got hint from your comment.
Yey.Thankyou.

This is transfer file

Code: [Select]

$ ls -la autoexec.bin
-rwxr-xr-x 1 miyake None 299008 8月  25 18:49 autoexec.bin

299008byte - 7026byte = 291982byte (un-transferd 7026k)
291982byte / 500byte = 583 (500byte is USB_BULK_TRANSFER_SIZE)
583 * 12byte = 6996byte (12byte over head of transfer)


So I think
1: USB bulk header descrived it's size. We can get from "context->get_data_size (context->handle);"
      (This is actually 299008byte(miss taken: 299008+fn_len + fn) )
2: Our receiving code is uging it data size.
3: But windows?libusb? is sending 12 byte data when sending after each BULK packet.
Then
6k is remained to transfer  on our while routine. But Canon's ptp protocol implementation is not understanding additional 12 byte data. and It's will be finish the transfer. Because we already transferd 299008byte including additional 12byte data.
4: Then we got error from recv_ptp_data()

That's my guess.

I'm already read all of code which  sending all data on libptp and libusb. It's not added 12 byte.
And I confirmed transferd raw data  by hex on libptp side . But it's correct with original hex.
 So Next, we need to find how to disable 12 byte data. The libusb is using windows native function to transport by USB.
I guess, another header option will disable 12 byte data.... It's just my guess.

Anyway, If we can disable 12 byte data, we can get success to transport it.

Also, current CHDK's ptp.c has no 12byte data handring.
So I think , PC side program need some trick to disable 12 bytedata.

Will find  BULK header option

MSG added
hmm, I have no unstable connection. It's really good and stable . I'm compiled cygwin libusb. and Win7 64 bit.
The libusb driver changing by downloadable libusb-win32-bin-1.2.6.0/bin/inf-wizerd.exe.

and size is correct value on my side.
size is = filename len(4byte) + actual filename + actual file size
confirmed.

[miyake]

I found libusb/src/windows.c

LIBUSB_WIN32_DLL_LARGE_TRANSFER_SUPPORT
entry.

So we don't need to limit transport file size on our userland applications.
So we don't use BULK_MAX_SIZE(500Byte).

[Pelican]

Is this working now?

[SDX]

It was never finished.

[Pelican]

Thanks