Hm, I'd say this one is an overhead from menu_add / menu_remove. What we can try is to compact the linked list into a single array, re-allocate it, and add it to the menu structure with a single call.
For 2800 files:
- total: 2100ms
- menu_add loop: 1900ms
- FindNext loop: 200ms (most likely overhead in add_file_entry and AllocateMemory)
- go back (up one level), menu_remove loop: 1200ms
- console_printf call from menu_remove loop: 200ms (can be deleted)
Note: if you want to measure times longer than 1 second and you have a CPU-intensive loop, be careful that hardware timer overflows every second and the software task that counts the overflows may not wakeup; to avoid this, you can add some get_ms_clock_value calls in the processing loops. Or, use a stopwatch

I've tried to run the FindFirst/FindNext loop twice (first time just for counting the items), and first run only takes 1ms!
=> I think it now makes sense to drop the entire linked list thingie and just use a single AllocateMemory call. This should solve all our speed problems.
P.S. menu rendering can't be the problem, because once the menu is built, scrolling is fast.