Port CHDK file browser (file_man.mo)

Started by gerk.raisen, May 20, 2013, 10:38:45 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

a1ex

Yeah, the HaCKeD sort is slowing it down. Feel free to optimize it a little.

QuickSort or HeapSort should be pretty good. Or some other algorithm with O(n log n) complexity.

Marsu42

Quote from: a1ex on June 06, 2013, 05:48:56 PM
Yeah, the HaCKeD sort is slowing it down. Feel free to optimize it a little.

Ha, this is the exact thing computer scientists know about, it was only part of my studies and I certainly am not a hardcore algorithm programer :-p ... since the main use for the file browser probably is working with files Canon cannot handle like dng maybe it would make sense to drop them in another directory than the standard Canon DCIM that quickly fills up with cr2/jpg/txt/xmp/... ?

But no big deal if I know I have to just wait 1min or so until the directory is open :-p

platu

Just a few observations about the file manager module...

I spent a few hours shooting both raw video as well as a couple of timelapses.  The timelapses were short, only 360 shots each (using silent pics) plus associated xmp files.  In total, about 1500 files plus 20 raw files.  As others have mentioned, it takes a very long time for directories with a lot of files to open.  So long that it makes it a bit impractical to review/playback files between takes.  In order to try to solve this while I was out filming, I decided to use the move files feature to move the timelapse related files to another folder on the same CF card.  But this turns out to be a very long process as it seems to behave more like a file copy with regards to time than a move.  I suggest the following...

1.  To make this a more workable solution, is it possible for the move operation when performed within the same card only modify the file allocation table (for fat32 and exfat)?  If that's already what it's doing then disregard this suggestion.

2.  If the above is not possible, could files created via the intervalometer be automatically created in subfolders?  Each intervalometer based operation would be saved in its own subfolder.  This would allow viewing of standard raw files without the slowdown that currently exists.  This would be ideal since it would not only solve the speed issue, it would allow for logical grouping of timelapses in a way not possible at the moment.

a1ex

Rename support is very weak in DryOS (on some cameras the function is there, but it just doesn't work). So, copying is the only portable way.

You can create folders from Canon menu; ML can't help you with that (unless you are OK with ERR70 messages).

platu

Ok... thanks.  Will use a separate Canon folder for each timelapse and that should suffice.

Pelican

I've just play with the file browser on the 7D.
It has worked well.
Some suggestions:
- The module name is file manager but the name of the menu item is file browser. It could be the same.
- There is enough space between the file name and the file size so the file date could be a useful info there. The date format can set as the camera's date format using the corresponding property.

Also I've found a bug in the module loader:
- If one module loading fails all the other modules show error and not load.
EOS 7D Mark II, EOS 7D, EOS 5, EOS 100 + lenses (10mm to 300mm), 600EX, 550EX, YN600EX x 3
EOScard, EOS DSLR firmwares, ARMu, NiControl, etc.: http://pel.hu/down

Pelican

Something like this.


I couldn't figure out how to convert timestamp to struct tm...

Edit:
Done.
Download here: http://pel.hu/ML/file_man.mo
EOS 7D Mark II, EOS 7D, EOS 5, EOS 100 + lenses (10mm to 300mm), 600EX, 550EX, YN600EX x 3
EOScard, EOS DSLR firmwares, ARMu, NiControl, etc.: http://pel.hu/down

Marsu42

Doh, I should have added this feature request here, sorry, didn't see this thread: http://www.magiclantern.fm/forum/index.php?topic=7410.0

ayshih

Quote from: a1ex on June 06, 2013, 05:48:56 PM
QuickSort or HeapSort should be pretty good. Or some other algorithm with O(n log n) complexity.

Mergesort has now been implemented.  I did a few time tests on my 50D, recording both the time it took for the menu to appear and separately printing out the time for just the sort:






FilesTotalSort
900<1 s11 ms
28003 s40 ms
550010 s88 ms

Thus, the sort itself scales like N*log(N) as it should, but the rendering of the menu now completely dominates the total time and scales more like N^2.
Canon EOS 50D | 17–40mm f/4L & 70–300mm f/4.5–5.6 DO IS | Lexar 1066x

a1ex

Hm, I'd say this one is an overhead from menu_add / menu_remove. What we can try is to compact the linked list into a single array, re-allocate it, and add it to the menu structure with a single call.

For 2800 files:
- total: 2100ms
- menu_add loop: 1900ms
- FindNext loop: 200ms (most likely overhead in add_file_entry and AllocateMemory)
- go back (up one level), menu_remove loop: 1200ms
- console_printf call from menu_remove loop: 200ms (can be deleted)

Note: if you want to measure times longer than 1 second and you have a CPU-intensive loop, be careful that hardware timer overflows every second and the software task that counts the overflows may not wakeup; to avoid this, you can add some get_ms_clock_value calls in the processing loops. Or, use a stopwatch :P

I've tried to run the FindFirst/FindNext loop twice (first time just for counting the items), and first run only takes 1ms!

=> I think it now makes sense to drop the entire linked list thingie and just use a single AllocateMemory call. This should solve all our speed problems.

P.S. menu rendering can't be the problem, because once the menu is built, scrolling is fast.

ayshih

I've now done detailed time testing, and nearly all of the slowdown is due to unnecessary traversals of the menu_entry list.  I just submitted a pull request that dramatically improves the speed of the file manager.  The menu now takes under a second to open a 5500-file directory; build_file_menu takes ~230 ms and appears to scale as N now.  Menu closing is also really fast.
Canon EOS 50D | 17–40mm f/4L & 70–300mm f/4.5–5.6 DO IS | Lexar 1066x

josd

Can we open a video or a pictures with file_man ?

Marsu42

Feature request: Delete all un/zero-rated and not protected files from directory.

Rationale: When shooting bursts, I often (pre)-select the pictures by rating them or at least protecting the ones I want to review further. Problem is: How to get rid of the rest? At home, I've written a script only moves all rated/protected files from the card. But with only the camera, I'd like to have a function to delete all at once except endless clicking trash->set or set+erase ... which is also prone to human with rated, but not protected files.

Any volunteers to add this to the file browser (Alex :-))? Would it even be possible, b/c ML of course would need to be able to check for the existence of a rating or the protection status (i.e. if the file is write protected or not)?

a1ex

Protection status should be the same as DOS read-only attribute, so I guess it's easy to read.

For rating status, I guess one should parse the CR2 header somehow, similar to exiftool. Should be doable (maybe even in Lua).

Marsu42

Quote from: a1ex on October 16, 2016, 10:11:12 AM
Protection status should be the same as DOS read-only attribute, so I guess it's easy to read.

Sounds promising, b/c the "quickly protect good images and delete the rest" method is the emergency rescue if you're running out of card space. But in  these cases, you're often pressed for time and don't really want to press trash/set for a couple of minutes :-\

garry23

Forgive me for opening up this older subject, but I've only just started using the file manager.

I seem to be able to select the files I wish to move from the CF to the sd card, ie ,mlvs.

The screen then has a "move here" item which I click, but I can't work out how to select B (sd card) and complete the move.

Has anyone worked out how to move multiple files from the CF to the sd, say?