ML on EOS-M2

Started by Palpatine, September 22, 2015, 02:48:23 PM

Previous topic - Next topic

0 Members and 4 Guests are viewing this topic.

dfort

Thanks!

So -- 1 MiB = 220 bytes = 1024 kibibytes = 1048576bytes = 0x100000

~/magic-lantern/platform/EOSM2.103/Makefile.platform.default
# DryOS memory map
# RESTARTSTART is selected to be at user_mem_start
# (aka heap start / DRY_HEAP_START / malloc memory pool)
# (actually 0x00100d80 but make it a little bit higher)
#
RESTARTSTART = 0x100E00


RESTARTSTART   = (0x100E00 - 0x200000) = -0xF200

Uh oh.

I may have found the stubs needed for CONFIG_ALLOCATE_MEMORY_POOL. Got it to compile. Might be useful if we ever find some free memory.

Still don't think I've got minimal ML working. Shouldn't it boot all the way to the Canon GUI? Maybe this camera doesn't have enough memory even for a minimal ML?

Thought I'd take a baby step and try to get the Model ID working in the recovery branch. We know there's enough memory for that:



Yay!

The Camera model isn't working but it isn't working for the EOSM, 100D or 700D either. Haven't been able to figure out why.

a1ex

https://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-EOSM2/ - compiled from dfort's repo.



Build log

Will explain CONFIG_ALLOCATE_MEMORY_POOL later.

dfort

From your console output:

sed -i 's/0xFF0C1C94/0xFF0C1C90/' platform/EOSM2.103/consts.h


I was that close?

[EDIT] Minimal ML working perfectly over here now. Well, it needs the debugmsg.gdb file while the 550D doesn't but that's a QEMU mystery.


dfort

Quote from: a1ex on July 13, 2017, 12:26:27 PM
Will explain CONFIG_ALLOCATE_MEMORY_POOL later.

Turned it on:

internals.h
/** This camera loads ML into the AllocateMemory pool **/
#define CONFIG_ALLOCATE_MEMORY_POOL


and set up a special case for the EOSM2

boot-hack.c
    #if defined(CONFIG_550D)
    // change end limit to 0xc60000 => reserve 640K for ML
    *addr_AllocMem_end = MOV_R1_0xC60000_INSTR;
    ml_reserved_mem = 640 * 1024;
   
    #elif defined(CONFIG_EOSM2) // experimenting - Hello World autoexec.bin is 425K
    *addr_AllocMem_end = MOV_R1_0xC80000_INSTR;
    ml_reserved_mem = 512 * 1024;
   
    #else
    // change end limit to 0xc80000 => reserve 512K for ML
    *addr_AllocMem_end = MOV_R1_0xC80000_INSTR;
    ml_reserved_mem = 512 * 1024;
    #endif


Of course like everything I try this doesn't work right away.

However, according to this post from the "placing ML into shoot memory" topic the EOSM2 only has 256MB, same with the 1300D, so does this mean trying to get ML running on these cameras is a lost cause or should we keep trying to shoehorn it in somehow?

dfort

Made a test pull request against unified from my repository (kept in sync with the master unified branch) just to see what has changed and make it easy to find what is still left to do.

https://bitbucket.org/daniel_fort/magic-lantern/pull-requests/4/eosm2103-wip/diff

a1ex

Quote from: dfort on July 16, 2017, 01:32:15 AM
However, according to this post from the "placing ML into shoot memory" topic the EOSM2 only has 256MB, same with the 1300D, so does this mean trying to get ML running on these cameras is a lost cause [...] ?

How much memory do your other cameras have?

How much memory does the 1100D have?

dfort

Quote from: a1ex on July 16, 2017, 10:06:22 AM
How much memory does the 1100D have?

According to the "placing ML into shoot memory" topic it has only 128MB and of course it can run ML. So I started thinking, how did you do that trick? That is when I realized that I was in the Twilight Zone--in other words I was lost in another dimension, obsessing over shot memory and ignoring other options.

Quote from: a1ex on July 06, 2017, 05:24:47 PM
One option is to try the AllocateMemory (like 550D or 1100D), and another is to hunt for possibly unused memory areas (smemShowFix - FF222B68) - see http://www.magiclantern.fm/forum/index.php?topic=5071.0.

That lead me down a path where I was able to track down some more stubs that might be useful to experiment with the AllocateMemoryResource method used on the 60D and 7D:

/* here we are patching RscMgr/SRM initialisation to use less memory */
// used on 7D and 60D - found values for EOSM2 but haven't figured out how to implement it yet
// #define HIJACK_CACHE_HACK
// #define RSCMGR_MEMORY_PATCH_END          0xff0c4a8c
// #define HIJACK_CACHE_HACK_INITTASK_ADDR  0xff0c1c98


Of course this isn't working (not enough shot memory?) and am back to pondering over these options:

Quote from: g3gg0 on April 06, 2013, 01:43:18 PM
in canon's firmware we have three options how to allocate memory and where to place data.

1) malloc
2) AllocateMemory (MemoryManager)
3) AllocateMemoryResource (RscMgr, Srm)

By the way, I like looking at graphs better than lists of hexadecimal numbers and the graphs on the placing ML into shoot memory topic have disappeared when using the Chrome browser. Looking into the issue it seems that Google has been implementing stricter rules that is affecting the link to your graph: https://a1ex.magiclantern.fm/bleeding-edge/RscMgr/EOSM2.png



That graph is helpful so if you don't mind I uploaded it to Flickr in case that is happening with others who are following this topic.

EOSM2 by Daniel Fort, on Flickr

Ok--so what are we looking at here? Is that two ways of looking at the same thing? Before and after memory allocation? Something there must have caused your to come up with this:

Quote from: a1ex on May 08, 2016, 06:06:13 PM
...I'm thinking to move autoexec.bin to AllocateMemory, and use the 7MB block as general-purpose memory for ML, using some custom allocator (maybe Canon's).

It hasn't sunk into my thick skull yet and I found myself in another Catch 22 situation. In order to create the data needed for these graphs the instructions are to:

Quote from: a1ex on May 08, 2016, 06:06:13 PM
- compile the dm-spy-experiments branch, with CONFIG_DEBUG_INTERCEPT=y in Makefile.user
- put this in don't click me...

(For the EOSM2)
    void debug_intercept();
    debug_intercept();
    void (*smemShowFix)(void) = 0xff222b6c;
    smemShowFix();
    debug_intercept();


There are also some stubs to look for so I followed the excellent example by @josepvm and translated it to the EOSM2

/** Low-level allocators */

NSTUB(0x79fc, init_memory_pool) 
NSTUB(0x8080, allocate_memory_from_pool)
NSTUB(0x83ec, free_memory_to_pool)
NSTUB(0x7ea0, get_max_region_of_pool)


Now how did you run that? I've got a feeling you're 10 steps ahead of me and already have a full ML for the EOSM2 working in QEMU--or you have an EOSM2 running ML and are keeping it a secret.

a1ex

Quote from: dfort on July 16, 2017, 08:02:45 PM
Now how did you run that?

As simple as...


./run_canon_fw.sh EOSM2,firmware="boot=0" -d debugmsg -s -S & arm-none-eabi-gdb -x EOSM2/debugmsg.gdb
...
CTRL-C
(gdb) set $pc = 0xff222b64
(gdb) continue


QuoteIs that two ways of looking at the same thing?
Yes.

QuoteBefore and after memory allocation?
No.

QuoteOk--so what are we looking at here?

Besides standard C malloc (aka user_mem in DryOS), and a similar allocator named AllocateMemory (sys_mem - nope, that's a third one), which both live in a tiny space of about 12MB (the gray bar at the left - 13MB on most models), for the memory-hungry stuff, Canon code uses a bunch of hardcoded regions (which may change according to operating modes). The graph shows these regions (top: scaled to show their real size, bottom: scaled so you can read the text).

Some of these regions are managed by allocators (shoot_malloc, srm_malloc), but it's not very clear which and when. It's one thing in photo mode and another in LiveView. Consider modes like video recording, raw developing, direct print (anyone uses it?) or wifi operation, and it starts to get really complicated. The safest way is to rely on these allocators when a large amount of memory is needed, and free it as soon as you no longer need it (otherwise you may not be able to change between some of the above modes). That's why the memory backend avoids shoot/srm allocators until it has no other choice. That's also the explanation for the second way to trigger ERR70 described in the stack trace PR.

In some cases, there are memory areas that are never used by Canon code (they are marked as full-height gray bars). We can double-check whether these are really unused (by making sure nobody writes to them while operating the camera in all possible modes we can think of), and if they are really free, we can take them. Caveat: they might be using stuff that's not listed in this log (as parts of that log are also hardcoded, so there's nothing to make sure what's in the log actually matches reality - they could have printed any numbers). On 60D, a region identified as free in this graph was later found to be actually used by Canon code (see the RscMgr PR, where we ran into trouble because of that). So, take this graph with a grain of salt.

For ML, we only need about 512K to load autoexec.bin (compare that to 128MB or even 64 for the 1000D). The other stuff (modules, work memory, whatever) are allocated dynamically, which is less troublesome, as the memory backend figures out which allocators to use, and lets you configure additional ones. But the initial code (autoexec.bin) has to be loaded statically, at some predefined address (unless you relocate it).

Of course, we can move everything to modules and end up with a very small static binary. It's a little more difficult than copy/paste though. Or you can try to compile as Thumb (good luck figuring it out).

Or, we can disable some features. You can even comment out features.h, CONFIG_RAW_* and fix up whatever gives compilation errors - that way, you'll get just the menu with a few debug items inside. It's best to do that anyway, and enable features one by one - otherwise you may end up with lots of possibly broken features (as it's really hard to test every single thing - I used to do that manually years ago), and if it's not something really widely used, you get a bug report after some months or years (even if the feature is not working at all).

Exercise: draw a similar memory map for DryOS (the first ~13 MB), based on the info from this post. Find out where the minimal ML is loaded (draw the map before and after). Find out the memory pool used by AllocateMemory. Find out where it's initialized (hint: check QEMU source). Patch it.

After finishing this, you will have both the Hello World working and a very good understanding of how the memory is organized.

dfort

Excellent! Here are some highlights of the output--totaling over 3625 lines!

^C
Program received signal SIGINT, Interrupt.
0xff352318 in ?? ()
(gdb) set $pc = 0xff222b6c
(gdb) continue
Continuing.
[    PowerMgr:ff222b78 ] (80:16)
   185:  3179.520 [RSC]
[    PowerMgr:ff222b94 ] (80:16) --- Common Top ----
   186:  3179.520 [RSC] --- Common Top ----
[    PowerMgr:ff222bb0 ] (80:16) [TOP1]                  0x40C2A000
   187:  3179.776 [RSC] [TOP1]                  0x40C2A000
[    PowerMgr:ff222bd4 ] (80:16) REPLACE_IMAGE_VRAM      0x40C2A000 0x00032000 204800
   188:  3179.776 [RSC] REPLACE_IMAGE_VRAM      0x40C2A000 0x00032000 204800
[    PowerMgr:ff222bfc ] (80:16) SSS_DEVELOP_WORK        0x40C5C000 0x00038000 229376
   189:  3180.032 [RSC] SSS_DEVELOP_WORK        0x40C5C000 0x00038000 229376
...
[    PowerMgr:ff223f78 ] (80:16) INDV_WORK               0x0 0x00000000 0
   274:  3187.712 [RSC] INDV_WORK               0x0 0x00000000 0
[    PowerMgr:ff0c2994 ] (8b:16) ###exceptionhandlercbr 0xda2c 0
   275:  3187.712 [STARTUP] ###exceptionhandlercbr 0xda2c 0
< Error Exception>
TYPE        : 4
ISR         : 0
TASK IDSR   : 1835011
TASK Name   : PowerMgr
R 0         : 0
R 1         : 19980218
R 2         : 0
R 3         : 0
R 4         : 19980218
R 5         : 19980218
R 6         : 19980218
R 7         : 19980218
R 8         : 19980218
R 9         : ca18
R10         : 19980218
R11         : 19980218
R12         : 1008
R13         : b89a0
R14         : da2c
PC          : da2c
CPSR        : 600000d3
[    PowerMgr:ff0c2ac4 ] (8b:16) #####exceptionhandlercbr 0xda2c
   276:  3188.480 [STARTUP] #####exceptionhandlercbr 0xda2c
[    PowerMgr:ff0c2ad4 ] (8b:03) < Error Exception>         
[    PowerMgr:ff0c2ae8 ] (8b:03) TYPE        : 4         
[    PowerMgr:ff0c2afc ] (8b:03) ISR         : 0         
[    PowerMgr:ff0c2b10 ] (8b:03) TASK IDSR   : 1835011       
[    PowerMgr:ff0c2b24 ] (8b:03) TASK Name   : PowerMgr     
[    PowerMgr:ff0c2b38 ] (8b:03) R 0         : 0       
[    PowerMgr:ff0c2b4c ] (8b:03) R 1         : 19980218   
[    PowerMgr:ff0c2b60 ] (8b:03) R 2         : 0     
[    PowerMgr:ff0c2b74 ] (8b:03) R 3         : 0       
[    PowerMgr:ff0c2b88 ] (8b:03) R 4         : 19980218       
[    PowerMgr:ff0c2b9c ] (8b:03) R 5         : 19980218
[    PowerMgr:ff0c2bb0 ] (8b:03) R 6         : 19980218       
[    PowerMgr:ff0c2bc4 ] (8b:03) R 7         : 19980218         
[    PowerMgr:ff0c2bd8 ] (8b:03) R 8         : 19980218         
[    PowerMgr:ff0c2bec ] (8b:03) R 9         : ca18         
[    PowerMgr:ff0c2c00 ] (8b:03) R10         : 19980218
[    PowerMgr:ff0c2c14 ] (8b:03) R11         : 19980218       
[    PowerMgr:ff0c2c28 ] (8b:03) R12         : 1008         
[    PowerMgr:ff0c2c3c ] (8b:03) R13         : b89a0         
[    PowerMgr:ff0c2e7c ] (8b:03) R14         : da2c           
[    PowerMgr:ff0c2e90 ] (8b:03) PC          : da2c             
[    PowerMgr:ff0c2ea4 ] (8b:03) CPSR        : 600000d3               
[    PowerMgr:ff0c2ee8 ] (8b:16) Exception : Time 2000/1/1 0:0:3
   277:  3188.736 [STARTUP] Exception : Time 2000/1/1 0:0:3
[      DbgMgr:ff1470d0 ] (00:01) [PM] DisablePowerSave (Counter = 2)
FF0C11BC: MRC p15,0,Rd,cr1,cr0,0:      SCTLR -> 0xC005107D
FF0C11BC: MCR p15, ...          : CACHEMAINT x11722 (omitted)
FF0C11BC: MCR p15,0,Rd,cr1,cr0,0:      SCTLR <- 0xC0051079
FF0C11F0: MRC p15,0,Rd,cr1,cr0,0:      SCTLR -> 0xC0051079
...
FF0C1120: MCR p15, ...          : CACHEMAINT x1 (omitted)
FF0C113C: MCR p15,0,Rd,cr1,cr0,0:      SCTLR <- 0xC005007D
FF0C1168: MRC p15,0,Rd,cr1,cr0,0:      SCTLR -> 0xC005007D
FF0C1120: MCR p15, ...          : CACHEMAINT x1 (omitted)
FF0C1168: MCR p15,0,Rd,cr1,cr0,0:      SCTLR <- 0xC005107D


[DM] FROM Write Complete!!!
[      DbgMgr:ff147140 ] (00:01) [PM] EnablePowerSave (Counter = 1)




And trimming it down to just what we're interested in:

   186:  3179.520 [RSC] --- Common Top ----
   187:  3179.776 [RSC] [TOP1]                  0x40C2A000
   188:  3179.776 [RSC] REPLACE_IMAGE_VRAM      0x40C2A000 0x00032000 204800
   189:  3180.032 [RSC] SSS_DEVELOP_WORK        0x40C5C000 0x00038000 229376
   190:  3180.032 [RSC] SDS_DEVELOP_WORK        0x40C94000 0x00038000 229376
   191:  3180.032 [RSC] DARKCUR_COMP_WORK       0x40CCC000 0x00020000 131072
   192:  3180.288 [RSC] FENCING_WORK            0x40CEC000 0x00010000 65536
   193:  3180.288 [RSC] DCFNO                   0x40CFC000 0x00004000 16384
   194:  3180.544 [RSC] LVMARGE_P_DEF_DATA_1    0x40D00000 0x0000A000 40960
   195:  3180.544 [RSC] LVMARGE_P_DEF_DATA_2    0x40D0A000 0x0000A000 40960
   196:  3180.800 [RSC] LVMARGE_P_DEF_DATA_3    0x41FF0000 0x0000A000 40960
   197:  3180.800 [RSC] LVMARGE_P_DEF_DATA_ZOOM 0x40D14000 0x0000C000 49152
   198:  3180.800 [RSC] FILE_HEADER             0x40D20000 0x00240000 2359296
   199:  3181.056 [RSC] SAF WORK                0x40FA0000 0x00200000 2097152
   200:  3181.056 [RSC] BMPVRAM1                0x411A0000 0x00080000 524288
   201:  3181.056 [RSC] BMPVRAM2                0x41220000 0x00080000 524288
   202:  3181.312 [RSC] ENGINE_MIRROR           0x412A0000 0x00044000 278528
   203:  3181.568 [RSC] VSHADING_COMP_WORK      0x412E4000 0x000DC000 901120
   204:  3181.568 [RSC] STILL SCAR              0x413C0000 0x00075B00 482048
   205:  3181.568 [RSC] TUNEDATA                0x41435B00 0x00140000 1310720
   206:  3181.824 [RSC] TUNEDATA2               0x41575B00 0x00160000 1441792
   207:  3181.824 [RSC] FIXDATA                 0x416D5B00 0x0021E500 2221312
   208:  3181.824 [RSC] LVMARGE_P_DEF_DATA_CROP 0x418F4000 0x0000C000 49152
   209:  3182.080 [RSC] WIRELESS_WORK2          0x41900000 0x00300000 3145728
   210:  3182.080 [RSC] WIRELESS_WORK1   *      0x41C00000 0x00200000 2097152
   211:  3182.080 [RSC] ADAPTER_TRANSFER *      0x0 0x00000000 0
   212:  3182.336 [RSC] EEKO                    0x41E00000 0x001E0000 1966080
   213:  3182.336 [RSC] SHOOTING_CREATIVEFILTER 0x41FE0000 0x00010000 65536
   214:  3182.336 [RSC] EXMEM3_AREA_4           0x0 0x00000000 0
   215:  3182.592 [RSC] --- Usually Mode ----
   216:  3182.592 [RSC] MEMORY_MGR              0x42000000 0x04000000 67108864
   217:  3182.592 [RSC] ONLY MEM1 1             0x46000000 0x02000000 33554432
   218:  3182.848 [RSC] ONLY MEM1 2             0x48000000 0x02000000 33554432
   219:  3182.848 [RSC] IMGPLAY_WORK            0x4A000000 0x00A00000 10485760
   220:  3182.848 [RSC] IMGPLAY_WORK2           0x4AA00000 0x00400000 4194304
   221:  3183.104 [RSC] SS_DEVELOP1             0x4A000000 0x00E00000 14680064
   222:  3183.104 [RSC] EXMEM3_AREA_2           0x4AE00000 0x000EA000 958464
   223:  3183.104 [RSC] AVERAGE_WORK_TOP        0x4AEEA000 0x01116000 17915904
   224:  3183.360 [RSC] AVERAGE_WORK_BOTTOM     0x4C000000 0x01116000 17915904
   225:  3183.360 [RSC] SS_DEVELOP_OTHER_WORK   0x4D116000 0x00400000 4194304
   226:  3183.360 [RSC] SLIDE_SHOW_WORK         0x4D516000 0x00800000 8388608
   227:  3183.616 [RSC] CAPTURE_WORK1           0x4D600000 0x00240000 2359296
   228:  3183.616 [RSC] EXMEM3_AREA_1           0x4DD16000 0x016BA000 23830528
   229:  3183.872 [RSC] IMGVRAM1                0x4F3D0000 0x00410000 4259840
   230:  3183.872 [RSC] IMGVRAM2                0x4F7E0000 0x00410000 4259840
   231:  3183.872 [RSC] IMGVRAM3                0x4FBF0000 0x00410000 4259840
   232:  3183.872 [RSC] ---   GIS Mode   ----
   233:  3184.128 [RSC] TEMPMEM1                0x48000000 0x02000000 33554432
   234:  3184.128 [RSC] WORK                    0x4A000000 0x08600000 140509184
   235:  3184.128 [RSC] IMGPLAY_WORK            0x4A000000 0x00A00000 10485760
   236:  3184.384 [RSC] IMGPLAY_WORK2           0x4AA00000 0x00400000 4194304
   237:  3184.384 [RSC] MOVIE_REC_WORK          0x4AE00000 0x00FA0000 16384000
   238:  3184.384 [RSC] MOVIE_PLAY_WORK         0x4BE00000 0x00E00000 14680064
   239:  3184.640 [RSC] SLIDE_SHOW_WORK         0x4D516000 0x00800000 8388608
   240:  3184.640 [RSC] MOVIE_STREAM            0x4DD16000 0x01400000 20971520
   241:  3184.640 [RSC] IMGVRAM1                0x4F3D0000 0x00410000 4259840
   242:  3184.896 [RSC] IMGVRAM2                0x4F7E0000 0x00410000 4259840
   243:  3184.896 [RSC] IMGVRAM3                0x4FBF0000 0x00410000 4259840
   244:  3185.152 [RSC] EXMEM3_1_AREA           0x42000000 0x06000000 100663296
   245:  3185.152 [RSC] EXMEM3_2_AREA           0x4CC40000 0x004D6000 5070848
   246:  3185.152 [RSC] ---   HDR Mode   ----
   247:  3185.152 [RSC] TEMPMEM1                0x48000000 0x02000000 33554432
   248:  3185.408 [RSC] WORK                    0x4800000 0x0A600000 174063616
   249:  3185.408 [RSC] IMGPLAY_WORK            0x4A000000 0x00A00000 10485760
   250:  3185.408 [RSC] IMGPLAY_WORK2           0x4AA00000 0x00400000 4194304
   251:  3185.664 [RSC] MOVIE_REC_WORK          0x4AE00000 0x00FA0000 16384000
   252:  3185.664 [RSC] MOVIE_PLAY_WORK         0x4BE00000 0x00E00000 14680064
   253:  3185.664 [RSC] SLIDE_SHOW_WORK         0x4D516000 0x00800000 8388608
   254:  3185.920 [RSC] MOVIE_STREAM            0x4DD16000 0x01400000 20971520
   255:  3185.920 [RSC] IMGVRAM1                0x4F3D0000 0x00410000 4259840
   256:  3185.920 [RSC] IMGVRAM2                0x4F7E0000 0x00410000 4259840
   257:  3186.176 [RSC] IMGVRAM3                0x4FBF0000 0x00410000 4259840
   258:  3186.176 [RSC] EXMEM3_1_AREA           0x42000000 0x06000000 100663296
   259:  3186.176 [RSC] EXMEM3_2_AREA           0x4CC40000 0x004D6000 5070848
   260:  3186.432 [RSC] ---    NR Mode   ----
   261:  3186.688 [RSC] NR_MEMORY_MGR           0x42000000 0x08000000 134217728
   262:  3186.688 [RSC] COMPOSITION_WORK_TOP    0x0 0x00000000 0
   263:  3186.688 [RSC] COMPOSITION_WORK_BOTTOM 0x0 0x00000000 0
   264:  3186.688 [RSC] ---    DP Mode   ----
   265:  3186.944 [RSC] DP_SINGLE               0x42000000 0x05E00000 98566144
   266:  3186.944 [RSC] DP_MULTI                0x47E00000 0x04D94000 81346560
   267:  3186.944 [RSC] DP_CAPTURE_WORK1        0x4CB94000 0x00040000 262144
   268:  3187.200 [RSC] DP_AVERAGE_TOP          0x4DBD4000 0x01116000 17915904
   269:  3187.200 [RSC] DP_AVERAGE_BOTTOM       0x4ECEA000 0x01116000 17915904
   270:  3187.200 [RSC] --- Indev Mode ----
   271:  3187.456 [RSC] [INDVMGR]               0x0
   272:  3187.456 [RSC] YUV                     0x4AEEA000 0x0222C000 35831808
   273:  3187.456 [RSC] YUV_OUT                 0x42000000 0x0222C000 35831808
   274:  3187.712 [RSC] INDV_WORK               0x0 0x00000000 0


It is also interesting to look at an alternate version and follow along in the disassembly:

[    PowerMgr:ff223a54 ] (80:16) ---   HDR Mode   ----
[    PowerMgr:ff223a74 ] (80:16) TEMPMEM1                0x48000000 0x02000000 33554432
[    PowerMgr:ff223a94 ] (80:16) WORK                    0x4800000 0x0A600000 174063616
[    PowerMgr:ff223ab4 ] (80:16) IMGPLAY_WORK            0x4A000000 0x00A00000 10485760
[    PowerMgr:ff223ad4 ] (80:16) IMGPLAY_WORK2           0x4AA00000 0x00400000 4194304
[    PowerMgr:ff223af4 ] (80:16) MOVIE_REC_WORK          0x4AE00000 0x00FA0000 16384000
[    PowerMgr:ff223b14 ] (80:16) MOVIE_PLAY_WORK         0x4BE00000 0x00E00000 14680064
[    PowerMgr:ff223b34 ] (80:16) SLIDE_SHOW_WORK         0x4D516000 0x00800000 8388608
[    PowerMgr:ff223b54 ] (80:16) MOVIE_STREAM            0x4DD16000 0x01400000 20971520
[    PowerMgr:ff223b74 ] (80:16) IMGVRAM1                0x4F3D0000 0x00410000 4259840
[    PowerMgr:ff223d90 ] (80:16) IMGVRAM2                0x4F7E0000 0x00410000 4259840
[    PowerMgr:ff223db0 ] (80:16) IMGVRAM3                0x4FBF0000 0x00410000 4259840
[    PowerMgr:ff223dd0 ] (80:16) EXMEM3_1_AREA           0x42000000 0x06000000 100663296
[    PowerMgr:ff223dec ] (80:16) EXMEM3_2_AREA           0x4CC40000 0x004D6000 5070848


BTW--interesting side trip into the Thumb instruction set. Looks like it is possible to mix it with standard ARM instructions.

Ok--back to work:

Quote from: a1ex on July 17, 2017, 12:39:54 AM
Exercise: draw a similar memory map for DryOS (the first ~13 MB), based on the info from this post. Find out where the minimal ML is loaded (draw the map before and after). Find out the memory pool used by AllocateMemory. Find out where it's initialized (hint: check QEMU source). Patch it.

a1ex

You may wonder - why are all these messages doubled?

[    PowerMgr:ff222b94 ] (80:16) --- Common Top ----
   186:  3179.520 [RSC] --- Common Top ----


The first format is written by "-d debugmsg" (we are printing the arguments to DebugMsg and some context - task and address where it was called from).

The second format is written to UART (possibly one of these pins) by Canon code - they do that for important messages (whatever that means - we don't know much about it).

You can redirect the UART messages to a file (e.g. "-serial file:uart.log"). The bootloader menu (printed e.g. if you start from a bootable card without autoexec.bin present) also goes to the serial port - you can interact with it from the console if you use "-serial stdio". These are standard QEMU options - you can use them on any other guest OS that uses a serial port.

TODO: the serial console is also usable while running the firmware (and you can call these info functions from there - example). The emulation is probably missing an interrupt or some status bits on the UART in order to accept incoming characters.

dfort

I've amassed a boat load of data but have no idea how to draw a graph that makes any sense of it.

Feels like this is very close. Here are some highlights of where things are at (using HIJACK_CACHE_HACK):

SD LOAD OK.
Open file for read : AUTOEXEC.BIN
...
File size : 0x67A00
Now jump to AUTOEXEC.BIN!!
0010E5F4: MRC p15,0,Rd,cr1,cr0,0:      SCTLR -> 0xC005107D
0010E5F4: MCR p15,0,Rd,cr1,cr0,0:      SCTLR <- 0xC005107D
[BOOT] patching init_task from 0xff0c57a0 (-15968352)
[BOOT] autoexec.bin loaded at 100E00 - 178140.


Great -- autoexec.bin is loaded, next surprise is that the Canon GUI comes up in the QEMU window.

[     FileMgr:ff304f9c ] (24:02) PROP_TFT_STATUS = 0
[     FileMgr:ff304fb4 ] (24:03) LED Permit
[MPU] Received: 06 05 03 19 00 00  (spell #52)
[    PowerMgr:ff13826c ] (82:01) VdInterruptHandler bmp=ff136384 img=0 localWaitBmpCBR=ff136384
[    PowerMgr:ff136398 ] (82:01) WaitBmpCBR(local) pParam=0
[  DisplayMgr:ff13a118 ] (82:03) MuteOffDisplay finish Type=0
[  DisplayMgr:ff1790cc ] (9a:02) fTunOnDisp = On


Then it gets ugly:

[      init:0010137c ] task_create(ml_init, prio=1e, stack=4000, entry=101510, arg=0)
[     ml_init:ff696760 ] (23:01) sdReadBlk: st=73344, num=32, buf=0x4035e000
[     ml_init:ff6951bc ] (23:01) sdDMAReadBlk: st=73344, num=32
qemu: fatal: Trying to execute code outside RAM or ROM at 0x20000000

R00=ffffffff R01=00987c0a R02=001f3912 R03=00000000
R04=19980030 R05=00000000 R06=596f0ae0 R07=19980218
R08=1998002e R09=19980218 R10=19980218 R11=19980218
R12=00000007 R13=001f3918 R14=00101cc0 R15=20000000
PSR=60000013 -ZC- A svc32
FPSCR: 00000000
qemu: fatal: Trying to execute code outside RAM or ROM at 0x20000000


Here's the full console output: https://pastebin.com/5P45Ujtz

Homework assignment --> F-incomplete (graph missing, Hello World not working)

(gdb) set $pc = 0xFF0C6E70
# No ML
sys_mem_start   0x0018ac00
sys_mem_max        1179648
user_mem_start  0x00100d80
user_mem_max        515828

# Minimal ML
sys_mem_start   0x0018ac00
sys_mem_max        1179648
user_mem_start  0x00104580
user_mem_max        501492

# Current almost working full ML Hello World
sys_mem_start   0x0018ac00
sys_mem_max        1179648
user_mem_start  0x00100d80
user_mem_max        515828


(gdb) set $pc = 0xff0c750c
# No ML
Malloc Information (onetime type)
  Start Address       = 0x00100d88
  End Address         = 0x0017ea60
  Total Size          = 0x0007dcd8 (   515288)
  Allocated Size      = 0x00015ab0 (    88752)
  Allocated Peak      = 0x00015ab0 (    88752)
  Allocated Count     = 0x0000005a (       90)
  Free Size           = 0x00068228 (   426536)
  Free Block Max Size = 0x000681f0 (   426480)
  Free Block Count    = 0x00000002 (        2)

# Minimal ML
Malloc Information (onetime type)
  Start Address       = 0x00104588
  End Address         = 0x0017ea60
  Total Size          = 0x0007a4d8 (   500952)
  Allocated Size      = 0x00015ab0 (    88752)
  Allocated Peak      = 0x00015ab0 (    88752)
  Allocated Count     = 0x0000005a (       90)
  Free Size           = 0x00064a28 (   412200)
  Free Block Max Size = 0x000649f0 (   412144)
  Free Block Count    = 0x00000002 (        2)

# Current almost working full ML Hello World
Malloc Information (onetime type)
  Start Address       = 0x00100d88
  End Address         = 0x0017ea60
  Total Size          = 0x0007dcd8 (   515288)
  Allocated Size      = 0x00015ab0 (    88752)
  Allocated Peak      = 0x00015ab0 (    88752)
  Allocated Count     = 0x0000005a (       90)
  Free Size           = 0x00068228 (   426536)
  Free Block Max Size = 0x000681f0 (   426480)
  Free Block Count    = 0x00000002 (        2)


(gdb) set $pc = 0xff134304
# No ML
   189:  1284.352 [WINSYS] allocated=44140 max_region=412876

# Minimal ML
   185:  1840.896 [WINSYS] allocated=44140 max_region=412876

# Current almost working full ML Hello World
   185:  1042.176 [WINSYS] allocated=44140 max_region=412876

# No change in the above but when experimenting with CONFIG_ALLOCATE_MEMORY_POOL this came up:
[    PowerMgr:ff13432c ] (04:16) allocated=0 max_region=1568300


(gdb) set $pc = 0xff1343f4
# No ML
[    PowerMgr:ff134450 ] (04:02) current=44140 max_reg=412876 allocate_size=0
   185:  1667.328 [MEM] NG AllocateMemory 0
   186:  1667.584 [MEM] Total = 0x7be000, Free = 0x3148b8, MaxReg = 0x2a320c
   187:  1667.840 [MEM] Num Alloc = 4864, Num Free = 460

# Minimal ML
[    PowerMgr:ff134450 ] (04:02) current=44140 max_reg=412876 allocate_size=0
   185:  2236.928 [MEM] NG AllocateMemory 0
   186:  2236.928 [MEM] Total = 0x7be000, Free = 0x3148b8, MaxReg = 0x2a31a0
   187:  2237.184 [MEM] Num Alloc = 4864, Num Free = 460

# Current almost working full ML Hello World
[    PowerMgr:ff134450 ] (04:02) current=44140 max_reg=412876 allocate_size=0
   185:  1226.752 [MEM] NG AllocateMemory 0
   186:  1227.008 [MEM] Total = 0x7be000, Free = 0x3148b8, MaxReg = 0x2a31f0
   187:  1227.264 [MEM] Num Alloc = 4864, Num Free = 460


dfort

My previous post was about what is almost working over here. This post is about what should be working but is a little further away than "almost."

#define CONFIG_ALLOCATE_MEMORY_POOL

Quote from: a1ex on July 17, 2017, 12:39:54 AM
Find out where the minimal ML is loaded (draw the map before and after). Find out the memory pool used by AllocateMemory. Find out where it's initialized (hint: check QEMU source). Patch it.

Is that QEMU source hint in memcheck.c? There are stubs in there but only for the 500D. So other cameras don't need this or is the 500D supported better in QEMU than any other camera? Ok, lots of other interesting stuff in there including something that looks like it is triggering the error message on my HIJACK_CACHE_HACK attempt:

assert(addr < 0x20000000);

[EDIT] This also looks like it can be useful once I learn how to use it:

~/gcc-arm-none-eabi-4_8-2013q4/bin/arm-none-eabi-readelf ~/magic-lantern/platform/EOSM2.103/magiclantern -a | grep " memcpy$"
  5694: 00148c94   360 FUNC    GLOBAL DEFAULT    1 memcpy


I'm still trying to understand how AllocateMemory works and believe me I'm trying. Here I created a huge table and found that while most addresses didn't change between No ML, Minimal ML and the almost working HIJACK_CACHE_HACK build, there were some very significant changes in the AllocateMemory attempt.

   186:  1221.888 [RSC] --- Common Top ----     No ML                               Minimal ML                          Almost working full ML Hello World  AllocateMemory (not working)                             
   187:  1221.888 [RSC] [TOP1]                  0x40C2A000                          0x40C2A000                          0x40C2A000                          0x0
   188:  1222.144 [RSC] REPLACE_IMAGE_VRAM      0x40C2A000 0x00032000 204800        0x40C2A000 0x00032000 204800        0x40C2A000 0x00032000 204800        0x0 0x00032000 204800
   189:  1222.144 [RSC] SSS_DEVELOP_WORK        0x40C5C000 0x00038000 229376        0x40C5C000 0x00038000 229376        0x40C5C000 0x00038000 229376        0x32000 0x00038000 229376
   190:  1222.400 [RSC] SDS_DEVELOP_WORK        0x40C94000 0x00038000 229376        0x40C94000 0x00038000 229376        0x40C94000 0x00038000 229376        0x6A000 0x00038000 229376
   191:  1222.400 [RSC] DARKCUR_COMP_WORK       0x40CCC000 0x00020000 131072        0x40CCC000 0x00020000 131072        0x40CCC000 0x00020000 131072        0xA2000 0x00020000 131072
   192:  1222.656 [RSC] FENCING_WORK            0x40CEC000 0x00010000 65536         0x40CEC000 0x00010000 65536         0x40CEC000 0x00010000 65536         0xC2000 0x00010000 65536
   193:  1222.656 [RSC] DCFNO                   0x40CFC000 0x00004000 16384         0x40CFC000 0x00004000 16384         0x40CFC000 0x00004000 16384         0xD2000 0x00004000 16384
   194:  1222.912 [RSC] LVMARGE_P_DEF_DATA_1    0x40D00000 0x0000A000 40960         0x40D00000 0x0000A000 40960         0x40D00000 0x0000A000 40960         0xD6000 0x0000A000 40960
   195:  1223.168 [RSC] LVMARGE_P_DEF_DATA_2    0x40D0A000 0x0000A000 40960         0x40D0A000 0x0000A000 40960         0x40D0A000 0x0000A000 40960         0xE0000 0x0000A000 40960
   196:  1223.168 [RSC] LVMARGE_P_DEF_DATA_3    0x41FF0000 0x0000A000 40960         0x41FF0000 0x0000A000 40960         0x41FF0000 0x0000A000 40960         0x41FF0000 0x0000A000 40960
   197:  1223.424 [RSC] LVMARGE_P_DEF_DATA_ZOOM 0x40D14000 0x0000C000 49152         0x40D14000 0x0000C000 49152         0x40D14000 0x0000C000 49152         0xEA000 0x0000C000 49152
   198:  1223.424 [RSC] FILE_HEADER             0x40D20000 0x00240000 2359296       0x40D20000 0x00240000 2359296       0x40D20000 0x00240000 2359296       0xF6000 0x00240000 2359296
   199:  1223.680 [RSC] SAF WORK                0x40FA0000 0x00200000 2097152       0x40FA0000 0x00200000 2097152       0x40FA0000 0x00200000 2097152       0x376000 0x00200000 2097152
   200:  1223.936 [RSC] BMPVRAM1                0x411A0000 0x00080000 524288        0x411A0000 0x00080000 524288        0x411A0000 0x00080000 524288        0x576000 0x00080000 524288
   201:  1223.936 [RSC] BMPVRAM2                0x41220000 0x00080000 524288        0x41220000 0x00080000 524288        0x41220000 0x00080000 524288        0x5F6000 0x00080000 524288
   202:  1224.192 [RSC] ENGINE_MIRROR           0x412A0000 0x00044000 278528        0x412A0000 0x00044000 278528        0x412A0000 0x00044000 278528        0x676000 0x00044000 278528
   203:  1224.192 [RSC] VSHADING_COMP_WORK      0x412E4000 0x000DC000 901120        0x412E4000 0x000DC000 901120        0x412E4000 0x000DC000 901120        0x6BA000 0x000DC000 901120
   204:  1224.192 [RSC] STILL SCAR              0x413C0000 0x00075B00 482048        0x413C0000 0x00075B00 482048        0x413C0000 0x00075B00 482048        0x413C0000 0x00075B00 482048
   205:  1224.448 [RSC] TUNEDATA                0x41435B00 0x00140000 1310720       0x41435B00 0x00140000 1310720       0x41435B00 0x00140000 1310720       0x41435B00 0x00140000 1310720
   206:  1224.704 [RSC] TUNEDATA2               0x41575B00 0x00160000 1441792       0x41575B00 0x00160000 1441792       0x41575B00 0x00160000 1441792       0x41575B00 0x00160000 1441792
   207:  1224.704 [RSC] FIXDATA                 0x416D5B00 0x0021E500 2221312       0x416D5B00 0x0021E500 2221312       0x416D5B00 0x0021E500 2221312       0x416D5B00 0x0021E500 2221312
   208:  1224.704 [RSC] LVMARGE_P_DEF_DATA_CROP 0x418F4000 0x0000C000 49152         0x418F4000 0x0000C000 49152         0x418F4000 0x0000C000 49152         0x418F4000 0x0000C000 49152
   209:  1224.960 [RSC] WIRELESS_WORK2          0x41900000 0x00300000 3145728       0x41900000 0x00300000 3145728       0x41900000 0x00300000 3145728       0x41900000 0x00300000 3145728
   210:  1224.960 [RSC] WIRELESS_WORK1   *      0x41C00000 0x00200000 2097152       0x41C00000 0x00200000 2097152       0x41C00000 0x00200000 2097152       0x41C00000 0x00200000 2097152
   211:  1224.960 [RSC] ADAPTER_TRANSFER *      0x0 0x00000000                      00x0 0x00000000                     00x0 0x0000000                      00x0 0x00000000 0
   212:  1225.216 [RSC] EEKO                    0x41E00000 0x001E0000 1966080       0x41E00000 0x001E0000 1966080       0x41E00000 0x001E0000 1966080       0x41E00000 0x001E0000 1966080
   213:  1225.216 [RSC] SHOOTING_CREATIVEFILTER 0x41FE0000 0x00010000 65536         0x41FE0000 0x00010000 65536         0x41FE0000 0x00010000 65536         0x41FE0000 0x00010000 65536
   214:  1225.216 [RSC] EXMEM3_AREA_4           0x0 0x00000000                      00x0 0x00000000                     00x0 0x00000000                     00x0 0x00000000 0
   215:  1225.472 [RSC] --- Usually Mode ----
   216:  1225.472 [RSC] MEMORY_MGR              0x42000000 0x04000000 67108864      0x42000000 0x04000000 67108864      0x42000000 0x04000000 67108864      0x42000000 0x04000000 67108864
   217:  1225.472 [RSC] ONLY MEM1 1             0x46000000 0x02000000 33554432      0x46000000 0x02000000 33554432      0x46000000 0x02000000 33554432      0x46000000 0x02000000 33554432
   218:  1225.728 [RSC] ONLY MEM1 2             0x48000000 0x02000000 33554432      0x48000000 0x02000000 33554432      0x48000000 0x02000000 33554432      0x48000000 0x02000000 33554432
   219:  1225.728 [RSC] IMGPLAY_WORK            0x4A000000 0x00A00000 10485760      0x4A000000 0x00A00000 10485760      0x4A000000 0x00A00000 10485760      0x4A000000 0x00A00000 10485760
   220:  1225.728 [RSC] IMGPLAY_WORK2           0x4AA00000 0x00400000 4194304       0x4AA00000 0x00400000 4194304       0x4AA00000 0x00400000 4194304       0x4AA00000 0x00400000 4194304
   221:  1225.984 [RSC] SS_DEVELOP1             0x4A000000 0x00E00000 14680064      0x4A000000 0x00E00000 14680064      0x4A000000 0x00E00000 14680064      0x4A000000 0x00E00000 14680064
   222:  1225.984 [RSC] EXMEM3_AREA_2           0x4AE00000 0x000EA000 958464        0x4AE00000 0x000EA000 958464        0x4AE00000 0x000EA000 958464        0x4AE00000 0x000EA000 958464
   223:  1225.984 [RSC] AVERAGE_WORK_TOP        0x4AEEA000 0x01116000 17915904      0x4AEEA000 0x01116000 17915904      0x4AEEA000 0x01116000 17915904      0x4AEEA000 0x01116000 17915904
   224:  1226.240 [RSC] AVERAGE_WORK_BOTTOM     0x4C000000 0x01116000 17915904      0x4C000000 0x01116000 17915904      0x4C000000 0x01116000 17915904      0x4C000000 0x01116000 17915904
   225:  1226.240 [RSC] SS_DEVELOP_OTHER_WORK   0x4D116000 0x00400000 4194304       0x4D116000 0x00400000 4194304       0x4D116000 0x00400000 4194304       0x4D116000 0x00400000 4194304
   226:  1226.496 [RSC] SLIDE_SHOW_WORK         0x4D516000 0x00800000 8388608       0x4D516000 0x00800000 8388608       0x4D516000 0x00800000 8388608       0x4D516000 0x00800000 8388608
   227:  1226.496 [RSC] CAPTURE_WORK1           0x4D600000 0x00240000 2359296       0x4D600000 0x00240000 2359296       0x4D600000 0x00240000 2359296       0x4D600000 0x00240000 2359296
   228:  1226.496 [RSC] EXMEM3_AREA_1           0x4DD16000 0x016BA000 23830528      0x4DD16000 0x016BA000 23830528      0x4DD16000 0x016BA000 23830528      0x4DD16000 0x016BA000 23830528
   229:  1226.752 [RSC] IMGVRAM1                0x4F3D0000 0x00410000 4259840       0x4F3D0000 0x00410000 4259840       0x4F3D0000 0x00410000 4259840       0x4F3D0000 0x00410000 4259840
   230:  1226.752 [RSC] IMGVRAM2                0x4F7E0000 0x00410000 4259840       0x4F7E0000 0x00410000 4259840       0x4F7E0000 0x00410000 4259840       0x4F7E0000 0x00410000 4259840
   231:  1226.752 [RSC] IMGVRAM3                0x4FBF0000 0x00410000 4259840       0x4FBF0000 0x00410000 4259840       0x4FBF0000 0x00410000 4259840       0x4FBF0000 0x00410000 4259840
   232:  1227.008 [RSC] ---   GIS Mode   ----
   233:  1227.008 [RSC] TEMPMEM1                0x48000000 0x02000000 33554432      0x48000000 0x02000000 33554432      0x48000000 0x02000000 33554432      0x48000000 0x02000000 33554432
   234:  1227.008 [RSC] WORK                    0x4A000000 0x08600000 140509184     0x4A000000 0x08600000 140509184     0x4A000000 0x08600000 140509184     0x4A000000 0x08600000 140509184
   235:  1227.264 [RSC] IMGPLAY_WORK            0x4A000000 0x00A00000 10485760      0x4A000000 0x00A00000 10485760      0x4A000000 0x00A00000 10485760      0x4A000000 0x00A00000 10485760
   236:  1227.264 [RSC] IMGPLAY_WORK2           0x4AA00000 0x00400000 4194304       0x4AA00000 0x00400000 4194304       0x4AA00000 0x00400000 4194304       0x4AA00000 0x00400000 4194304
   237:  1227.264 [RSC] MOVIE_REC_WORK          0x4AE00000 0x00FA0000 16384000      0x4AE00000 0x00FA0000 16384000      0x4AE00000 0x00FA0000 16384000      0x4AE00000 0x00FA0000 16384000
   238:  1227.520 [RSC] MOVIE_PLAY_WORK         0x4BE00000 0x00E00000 14680064      0x4BE00000 0x00E00000 14680064      0x4BE00000 0x00E00000 14680064      0x4BE00000 0x00E00000 14680064
   239:  1227.520 [RSC] SLIDE_SHOW_WORK         0x4D516000 0x00800000 8388608       0x4D516000 0x00800000 8388608       0x4D516000 0x00800000 8388608       0x4D516000 0x00800000 8388608
   240:  1227.776 [RSC] MOVIE_STREAM            0x4DD16000 0x01400000 20971520      0x4DD16000 0x01400000 20971520      0x4DD16000 0x01400000 20971520      0x4DD16000 0x01400000 20971520
   241:  1227.776 [RSC] IMGVRAM1                0x4F3D0000 0x00410000 4259840       0x4F3D0000 0x00410000 4259840       0x4F3D0000 0x00410000 4259840       0x4F3D0000 0x00410000 4259840
   242:  1228.032 [RSC] IMGVRAM2                0x4F7E0000 0x00410000 4259840       0x4F7E0000 0x00410000 4259840       0x4F7E0000 0x00410000 4259840       0x4F7E0000 0x00410000 4259840
   243:  1228.032 [RSC] IMGVRAM3                0x4FBF0000 0x00410000 4259840       0x4FBF0000 0x00410000 4259840       0x4FBF0000 0x00410000 4259840       0x4FBF0000 0x00410000 4259840
   244:  1228.032 [RSC] EXMEM3_1_AREA           0x42000000 0x06000000 100663296     0x42000000 0x06000000 100663296     0x42000000 0x06000000 100663296     0x42000000 0x06000000 100663296
   245:  1228.288 [RSC] EXMEM3_2_AREA           0x4CC40000 0x004D6000 5070848       0x4CC40000 0x004D6000 5070848       0x4CC40000 0x004D6000 5070848       0x4CC40000 0x004D6000 5070848
   246:  1228.288 [RSC] ---   HDR Mode   ----
   247:  1228.288 [RSC] TEMPMEM1                0x48000000 0x02000000 33554432      0x48000000 0x02000000 33554432      0x48000000 0x02000000 3355443       20x48000000 0x02000000 33554432
   248:  1228.544 [RSC] WORK                    0x4800000 0x0A600000 174063616      0x4800000 0x0A600000 174063616      0x4800000 0x0A600000 174063616      0x4800000 0x0A600000 174063616
   249:  1228.544 [RSC] IMGPLAY_WORK            0x4A000000 0x00A00000 10485760      0x4A000000 0x00A00000 10485760      0x4A000000 0x00A00000 10485760      0x4A000000 0x00A00000 10485760
   250:  1228.544 [RSC] IMGPLAY_WORK2           0x4AA00000 0x00400000 4194304       0x4AA00000 0x00400000 4194304       0x4AA00000 0x00400000 4194304       0x4AA00000 0x00400000 4194304
   251:  1228.800 [RSC] MOVIE_REC_WORK          0x4AE00000 0x00FA0000 16384000      0x4AE00000 0x00FA0000 16384000      0x4AE00000 0x00FA0000 16384000      0x4AE00000 0x00FA0000 16384000
   252:  1228.800 [RSC] MOVIE_PLAY_WORK         0x4BE00000 0x00E00000 14680064      0x4BE00000 0x00E00000 14680064      0x4BE00000 0x00E00000 14680064      0x4BE00000 0x00E00000 14680064
   253:  1228.800 [RSC] SLIDE_SHOW_WORK         0x4D516000 0x00800000 8388608       0x4D516000 0x00800000 8388608       0x4D516000 0x00800000 8388608       0x4D516000 0x00800000 8388608
   254:  1229.056 [RSC] MOVIE_STREAM            0x4DD16000 0x01400000 20971520      0x4DD16000 0x01400000 20971520      0x4DD16000 0x01400000 20971520      0x4DD16000 0x01400000 20971520
   255:  1229.312 [RSC] IMGVRAM1                0x4F3D0000 0x00410000 4259840       0x4F3D0000 0x00410000 4259840       0x4F3D0000 0x00410000 4259840       0x4F3D0000 0x00410000 4259840
   256:  1229.312 [RSC] IMGVRAM2                0x4F7E0000 0x00410000 4259840       0x4F7E0000 0x00410000 4259840       0x4F7E0000 0x00410000 4259840       0x4F7E0000 0x00410000 4259840
   257:  1229.568 [RSC] IMGVRAM3                0x4FBF0000 0x00410000 4259840       0x4FBF0000 0x00410000 4259840       0x4FBF0000 0x00410000 4259840       0x4FBF0000 0x00410000 4259840
   258:  1229.568 [RSC] EXMEM3_1_AREA           0x42000000 0x06000000 100663296     0x42000000 0x06000000 100663296     0x42000000 0x06000000 100663296     0x42000000 0x06000000 100663296
   259:  1229.824 [RSC] EXMEM3_2_AREA           0x4CC40000 0x004D6000 5070848       0x4CC40000 0x004D6000 5070848       0x4CC40000 0x004D6000 5070848       0x4CC40000 0x004D6000 5070848
   260:  1229.824 [RSC] ---    NR Mode   ----
   261:  1229.824 [RSC] NR_MEMORY_MGR           0x42000000 0x08000000 134217728     0x42000000 0x08000000 134217728     0x42000000 0x08000000 134217728     0x42000000 0x08000000 134217728
   262:  1230.080 [RSC] COMPOSITION_WORK_TOP    0x0 0x00000000                      00x0 0x00000000                     00x0 0x00000000                     00x0 0x00000000 0
   263:  1230.080 [RSC] COMPOSITION_WORK_BOTTOM 0x0 0x00000000                      00x0 0x00000000                     00x0 0x00000000                     00x0 0x00000000 0
   264:  1230.080 [RSC] ---    DP Mode   ----
   265:  1230.336 [RSC] DP_SINGLE               0x42000000 0x05E00000 98566144      0x42000000 0x05E00000 98566144      0x42000000 0x05E00000 98566144      0x42000000 0x05E00000 98566144
   266:  1230.336 [RSC] DP_MULTI                0x47E00000 0x04D94000 81346560      0x47E00000 0x04D94000 81346560      0x47E00000 0x04D94000 81346560      0x47E00000 0x04D94000 81346560
   267:  1230.336 [RSC] DP_CAPTURE_WORK1        0x4CB94000 0x00040000 262144        0x4CB94000 0x00040000 262144        0x4CB94000 0x00040000 262144        0x4CB94000 0x00040000 262144
   268:  1230.592 [RSC] DP_AVERAGE_TOP          0x4DBD4000 0x01116000 17915904      0x4DBD4000 0x01116000 17915904      0x4DBD4000 0x01116000 17915904      0x4DBD4000 0x01116000 17915904
   269:  1230.592 [RSC] DP_AVERAGE_BOTTOM       0x4ECEA000 0x01116000 17915904      0x4ECEA000 0x01116000 17915904      0x4ECEA000 0x01116000 17915904      0x4ECEA000 0x01116000 17915904
   270:  1230.848 [RSC] --- Indev Mode ----
   271:  1230.848 [RSC] [INDVMGR]               0x00x00x00x0
   272:  1230.848 [RSC] YUV                     0x4AEEA000 0x0222C000 35831808      0x4AEEA000 0x0222C000 35831808      0x4AEEA000 0x0222C000 35831808      0x4AEEA000 0x0222C000 35831808
   273:  1230.848 [RSC] YUV_OUT                 0x42000000 0x0222C000 35831808      0x42000000 0x0222C000 35831808      0x42000000 0x0222C000 35831808      0x42000000 0x0222C000 35831808
   274:  1231.104 [RSC] INDV_WORK               0x0 0x00000000                      00x0 0x00000000                     00x0 0x00000000                     00x0 0x00000000 0


I've also been pouring over the code and zeroed in on this comment:

boot-hack.c
    // We shrink the AllocateMemory (system memory) pool in order to make space for ML binary
    // Example for the 1100D firmware
    // ff0197d8: init_task:
    // ff01984c: b CreateTaskMain
    //
    // ff0123c4 CreateTaskMain:
    // ff0123e4: mov r1, #13631488  ; 0xd00000  <-- end address
    // ff0123e8: mov r0, #3997696   ; 0x3d0000  <-- start address
    // ff0123ec: bl  allocatememory_init_pool

    // So... we need to patch CreateTaskMain, which is called by init_task.
    //
    // First we use Trammell's reloc.c code to relocate init_task and CreateTaskMain...


I looked over my disassemble.pl version of the 1100D (and 550D) and without giving away any more than what was already published--came up with this for the EOSM2:

EOSM2 stub:
NSTUB(0xFF0C57A0,  init_task)

ff0c57a0: push {r4, lr}
ff0c5814: b loc_ff0c3064 <-- call CreateTaskMain
...
ff0c3064: push {r1, r2, r3, r4, r5, lr}
...
ff0c30a4: ldr r1, [pc, #344] ; ff0c3204: (00c2a000)   <-- end address??
ff0c30a8: ldr r0, [pc, #344] ; ff0c3208: (0046c000)   <-- start address??
...
ff0c30c4: bl loc_ff146d3c <-- allocatememory_init_pool?


I'm pretty sure about the allocatememory_init_pool but not sure at all that I found the start and end addresses. I've been trying different addresses for ROM_ALLOCMEM_END but keep getting the same disappointing results.

consts.h
// Used in boot-hack.c with CONFIG_ALLOCATE_MEMORY_POOL
#define ROM_ITASK_START 0xff0c57a0
#define ROM_ITASK_END  0xff0cb994
#define ROM_CREATETASK_MAIN_START 0xff0c3064
#define ROM_CREATETASK_MAIN_END 0xff0c33dc
#define ROM_ALLOCMEM_END 0xff0c4a8c // 0xff0c30a8 0xff0c30a4 // kinda iffy
#define ROM_ALLOCMEM_INIT 0xff0c30c4
#define ROM_B_CREATETASK_MAIN 0xff0c5814


Had this camera been able to use the "classic" boot process (a.k.a. malloc?) we would probably be running code on the camera by now, right?

a1ex

QuoteSo other cameras don't need this or is the 500D supported better in QEMU than any other camera?

500D was the first to get menu navigation in QEMU, and right now there are still some things that only work there (such as changing display color, or the memcheck tool). Also, in some places such as photo capture, Canon code is simpler (easier to understand).

Quote
ff0c30a4:    ldr r1, [pc, #344] ; ff0c3204: (00c2a000)   <-- end address??
ff0c30a8:    ldr r0, [pc, #344] ; ff0c3208: (0046c000)   <-- start address??
...
ff0c30c4:    bl loc_ff146d3c <-- allocatememory_init_pool?

Almost there - start and end address are correct, but allocatememory_init_pool is not.

During a function call, R0-R3 are not preserved; in particular, R0 is also used for return value. Therefore, whatever was loaded in R0 and R1 are parameters for the function call that follows immediately, but not for the subsequent functions.

Therefore, allocatememory_init_pool is the one called right after loading the parameters into registers:

FF0C30AC                 BL      0x7ABC <-- this one


Note that both 0x81F0 (AllocateMemory) and 0x7ABC (allocatememory_init_pool) reference the same memory address (0x90CC8). The latter (0x7ABC) writes something to this address (the result of 0x79FC), and this call contains a solid hint about the meaning of R0 and R1: start and end. Why? Because 0x79FC receives R0 = start and R1 = end-start, so the latter must be size:

00007ABC
...                      ; nothing else touching R0=start and R1=end
00007AD0                 SUB     R1, R1, R0
00007AD4                 BL      sub_79FC


In other words, the AllocateMemory pool is from 0x46c000 to 0xc2a000. Note that, unlike my previous comments (je me suis trompé), this is not the system memory area:


(gdb) set $pc=0xFF0C77EC
System Memory Information
  Start Address       = 0x0018ac08
  End Address         = 0x002aac00
  Total Size          = 0x0011fff8 (  1179640)
  Allocated Size      = 0x00062328 (   402216)
  Allocated Peak      = 0x00064b30 (   412464)
  Allocated Count     = 0x00000051 (       81)
  Free Size           = 0x000bdcd0 (   777424)
  Free Block Max Size = 0x000bb240 (   766528)
  Free Block Count    = 0x00000003 (        3)


I have a feeling this might be where DryOS allocates stack space for its tasks. Let's check:

arm-none-eabi-gdb -x EOSM2/patches.gdb & ./run_canon_fw.sh EOSM2,firmware="boot=0" -s -S -d debugmsg,v |& grep "Current stack" | sort | cut -d ' ' -f 1-3 | uniq
...
Current stack: [18af38-18acb8]
...
Current stack: [1ed9b0-1ec9b0]


Confirmed. Finding the allocator for the system memory would be good for cameras with very little RAM (such as 1100D).

Back to our case - AllocateMemory uses a different block of memory (not sys_mem), so I'd better update the wrong comments in the source code.

Let's try to find out how much free space we have (GetMemoryInformation). From mem.c, we know this about it:

extern int GetMemoryInformation(int* total, int* free);


So, it takes 2 parameters, pointers to 32-bit integers. It will write the total amount into the first integer, and the amount of free space into the second integer.

Let's call it from gdb. Since we don't care about code execution after this function returns, we can just pick two arbitrary addresses to store the two integers (never do this on real hardware!)

(gdb) set $r0 = 4
(gdb) set $r1 = 8
(gdb) set $pc = 0xFFD25774
(gdb) b *$lr
Breakpoint 2 at 0xff352220
(gdb) c
Continuing.

Breakpoint 2, 0xff352220 in ?? ()
(gdb) x/2 4
0x4: 0x7be000 0x3148d0


Cross-checking: 0xc2a000 - 0x46c000 = 0x7be000.

So, that's what we have to patch in order to load ML.

TODO: I'd like to update 100D to use the "classic" boot process as well.

nikfreak

Quote from: a1ex on July 22, 2017, 05:51:57 PM

TODO: I'd like to update 100D to use the "classic" boot process as well.

as discussed here I failed to use AllocateMemory pool but:

Quote from: nikfreak
...I am able to load 100d into malloc but the interesting part I can only bring up ML menu with the task related changes from this PR in addition with a fixed task_dispatch_hook and undefining hijack_task_addr as instructed earlier. Otherwise ML menu wouldn't come up.

I'm ready If you want to give it another go.
[size=8pt]70D.112 & 100D.101[/size]

dfort

Quote from: a1ex on July 22, 2017, 05:51:57 PM
Almost there - start and end address are correct, but allocatememory_init_pool is not.
...
...allocatememory_init_pool is the one called right after loading the parameters into registers:

FF0C30AC                 BL      0x7ABC <-- this one


Yikes! That's the first one I tried but later dismissed it.

In order to follow along on my disassembly I'm doing this:

0x7ABC + RAM_OFFSET

Quote from: a1ex on July 22, 2017, 05:51:57 PM

00007ABC
...                      ; nothing else touching R0=start and R1=end
00007AD0                 SUB     R1, R1, R0
00007AD4                 BL      sub_79FC


In other words, the AllocateMemory pool is from 0x46c000 to 0xc2a000. Note that, unlike my previous comments (je me suis trompé)...

Je suis clueless -- Hey, Google did the translation.

Quote from: a1ex on July 22, 2017, 05:51:57 PM
Let's check:

arm-none-eabi-gdb -x EOSM2/patches.gdb & ./run_canon_fw.sh EOSM2,firmware="boot=0" -s -S -d debugmsg,v |& grep "Current stack" | sort | cut -d ' ' -f 1-3 | uniq
...
Current stack: [18af38-18acb8]
...
Current stack: [1ed9b0-1ec9b0]


Here is what I got:

arm-none-eabi-gdb -x EOSM2/patches.gdb & ./run_canon_fw.sh EOSM2,firmware="boot=0" -s -S -d debugmsg,v |& grep "Current stack" | sort | cut -d ' ' -f 1-3 | uniq
[8] 28220


Getting rid of all the tricky stuff there really isn't much to search.

./run_canon_fw.sh EOSM2,firmware=boot=0 -s -S -d debugmsg,v
DebugMsg=0x4398 (from GDB script)
Lockdown read 0
Lockdown read 0
Lockdown read 1
Lockdown read 1
Lockdown read 2
Lockdown read 2
Lockdown read 3
Lockdown read 3
Lockdown read 4
Lockdown read 4
00000000 - 00000FFF: eos.tcm_code
40000000 - 40000FFF: eos.tcm_data
00001000 - 1FFFFFFF: eos.ram
40001000 - 5FFFFFFF: eos.ram_uncached
F0000000 - F0FFFFFF: eos.rom0
F1000000 - F1FFFFFF: eos.rom0_mirror
F2000000 - F2FFFFFF: eos.rom0_mirror
F3000000 - F3FFFFFF: eos.rom0_mirror
F4000000 - F4FFFFFF: eos.rom0_mirror
F5000000 - F5FFFFFF: eos.rom0_mirror
F6000000 - F6FFFFFF: eos.rom0_mirror
F7000000 - F7FFFFFF: eos.rom0_mirror
F8000000 - F8FFFFFF: eos.rom1
F9000000 - F9FFFFFF: eos.rom1_mirror
FA000000 - FAFFFFFF: eos.rom1_mirror
FB000000 - FBFFFFFF: eos.rom1_mirror
FC000000 - FCFFFFFF: eos.rom1_mirror
FD000000 - FDFFFFFF: eos.rom1_mirror
FE000000 - FEFFFFFF: eos.rom1_mirror
FF000000 - FFFFFFFF: eos.rom1_mirror
C0000000 - DFFFFFFF: eos.iomem
[EOS] enabling code execution logging.
[EOS] loading './EOSM2/ROM0.BIN' to 0xF0000000-0xF0FFFFFF
[EOS] loading './EOSM2/ROM1.BIN' to 0xF8000000-0xF8FFFFFF
[EOS] loading './EOSM2/SFDATA.BIN' as serial flash, size=0x1000000
[MPU] warning: non-empty spell #31 (08 06 03 1f) has duplicate(s): #40
[MPU] warning: non-empty spell #41 (0a 08 01 3b) has duplicate(s): #37 #38
[MPU] warning: non-empty spell #45 (26 24 01 4e) has duplicate(s): #46

[MPU] Available keys:
- Arrow keys   : Navigation
- [ and ]      : Main dial (top scrollwheel)
- SPACE        : SET
- DELETE       : guess
- M            : MENU
- P            : PLAY
- I            : INFO/DISP
- Q            : guess
- L            : LiveView
- Shift        : Half-shutter
- B            : Open battery cover
- F1           : show this help

Setting BOOTDISK flag to 0


Maybe a problem with this Mac QEMU? Seemed to be working fine up until now. Ok--moving on.

Quote from: a1ex on July 22, 2017, 05:51:57 PM
Finding the allocator for the system memory would be good for cameras with very little RAM (such as 1100D).

I was thinking of trying an alternate booting method on the original EOSM to see if that takes care of the shutter-bug. That is if I can ever get through this.

(gdb) set $r0 = 4
(gdb) set $r1 = 8
(gdb) set $pc = 0xFFD25774
(gdb) b *$lr
Breakpoint 2 at 0xff352220
(gdb) c
Continuing.

Breakpoint 2, 0xff352220 in ?? ()
(gdb) x/2 4
0x4: 0x7be000 0x3148d0


Ok--this works but how did you know to give r0 = 4 and r1 = 8? Then there's that tricky x/2 4. I tried other values and of course came up with different results. In any case, the important part is that:

end address - start address = free space

Great psudo code heh? Now is that really free space? Free as in beer? Or--is that total space? The latest autoexec.bin file I made was:

File size : 0x686A0


So--

0x7be000 - 0x686A0 = 0x755960

Plenty of space but is it shared with the Canon firmware? Yes--but there is still enough room for everybody?

Quote from: a1ex on July 22, 2017, 05:51:57 PM
So, that's what we have to patch in order to load ML.

Great, let's do this. First I saw what was done for the 550D and 1100D (the only other cameras that uses AllocateMemory pool) and added a special case for the EOSM2:

boot-hack.c

    #elif defined(CONFIG_EOSM2)
    // change end limit to 0xc2a000 => reserve 512K for ML
    *addr_AllocMem_end = MOV_R1_0xC2A000_INSTR;
    ml_reserved_mem = 512 * 1024;   


Next I added MOV_R1_0xC2A000_INSTR for the start address of the EOSM2:

arm-mcr.h
#define MOV_R0_0_INSTR 0xe3a00000
#define MOV_R1_0xC2A000_INSTR 0xe3a0???? // mov r1, 0xc2a000


I'm kind of stuck figuring out what to plug in and if there is anything else that needs to be changed. For example RESTARTSTART is set to 0x100E00 so I'm getting this:

Now jump to AUTOEXEC.BIN!!
0010E5F4: MRC p15,0,Rd,cr1,cr0,0:      SCTLR -> 0xC005107D
0010E5F4: MCR p15,0,Rd,cr1,cr0,0:      SCTLR <- 0xC005107D
[BOOT] changing init_task from 0xff0c57a0 (-15968352) to 0x1015f8 (1054200)
[BOOT] autoexec.bin loaded at 100E00 - 181840.


I'm also getting these message that makes me think this isn't working:

ff0c8314: !!!! can not fixup jump from 0016dc9c to fd77ab1c (offset -00a7cc62)
...
ff0c8948: !!!! can not fixup jump from 0016e2d0 to fd77b150 (offset -00a7cc62)


Finally, seeing how the 500D work, it ends up like this:

Fixups=cd4730 entry=cd4740 free_space=10
[BOOT] changing sys_mem_end:
0x00cd4760:  e3a0160d      mov r1, #13631488 ; 0xd00000
0x00cd4760:  e3a018c6      mov r1, #12976128 ; 0xc60000
K270 READY
[      init:ff1d3b58 ] create_semaphore('EventProcedure', 1)
Temporary breakpoint 15 at 0xff1d3b5c


While the EOSM2 winds up like this:

Fixups=171390 entry=17135c free_space=ffffffcc

a1ex

Quote from: dfort on July 23, 2017, 02:12:25 AM
In order to follow along on my disassembly I'm doing this:

0x7ABC + RAM_OFFSET

That's exactly the reason I've listed the RAM stubs in the ROM_ADDR - RAM_OFFSET format in stubs.S - so one can just copy the ROM address and look it up in a ROM disassembly, without requiring the RAM segments to be set up. I should probably include the RAM address in comments as well - a job for the stubs formatting scripts.

Quote
Ok--this works but how did you know to give r0 = 4 and r1 = 8?

Just picked two RAM addresses that I thought they won't be used in this routine. Since in some cases, null pointers are used for optional output arguments, I've chosen non-zero pointers. Addresses 4 and 8 are actually exception vectors - Undefined instruction and Software interrupt. Our routine is not going to trigger these, and they are easy to type.

x/2 4 means 2 items from address 4, and by default, an item is a 32-bit integer. Therefore, it's printing the memory contents at 4 and 8 (our pointers).

Alternatives:
- allocate on the stack (below $sp) - probably the safest option if you debug a real target
- use some high address that's unlikely to be used (anything above 0x10000000 on a 256M camera will do, since QEMU uses 512M for all models, to keep it simple)
- find some other address unlikely to be used (e.g. in the middle of some RAM string, or some other statically allocated variable)
- pick something random and repeat until it succeeds (emulation only!)

Quote

#define MOV_R1_0xC2A000_INSTR 0xe3a0???? // mov r1, 0xc2a000


You don't need to hardcode a MOV here - it uses PC-relative addressing. Look at ff0c3204.

Will take a closer look later, also for 100D, since now I have the tools to do so :D

dfort

Quote from: a1ex on July 23, 2017, 08:36:46 AM
You don't need to hardcode a MOV here - it uses PC-relative addressing. Look at ff0c3204.

That's why my patching isn't working? It always comes up like this:

File size : 0x67D40
Now jump to AUTOEXEC.BIN!!
0010E5F4: MRC p15,0,Rd,cr1,cr0,0:      SCTLR -> 0xC005107D
0010E5F4: MCR p15,0,Rd,cr1,cr0,0:      SCTLR <- 0xC005107D
[BOOT] changing init_task from 0xff0c57a0 (-15968352) to 0x1015e0 (1054176)
[BOOT] autoexec.bin loaded at 100E00 - 181840.
Fixing from ff0c57a0 to ff0cb994


Those [BOOT] messages are handy but only available in the QEMU branch. If merging QEMU into unified doesn't break anything it would be lovely. :D

In any case, it seems to be doing the right thing.

This is still a problem--noticed I didn't give enough information in my previous post:

ff0c8314: 3b9aca00 BL ff9aca00 => fd77ab1c
ff0c8314: !!!! can not fixup jump from 0016dc9c to fd77ab1c (offset -00a7cc62)
...
ff0c8948: 3b9aca00 BL ff9aca00 => fd77b150
ff0c8948: !!!! can not fixup jump from 0016e2d0 to fd77b150 (offset -00a7cc62)


Another problem is that we're not getting to this (using the 550D example):

[BOOT] changing sys_mem_end:

dfort

Thought I'd take a detour and attempt to get AllocateMemory pool working on the 700D. Although I really wanted to do this on the EOSM to see if it would take care of the shutter-bug, the 700D plays better in QEMU and it should be easy to port any changes from the 700D to the EOSM.

Ok--just for the record, this is needed or AllocateMemory pool won't compile. This applies to both the EOSM2 and 700D as well as the 550D and 1100D. In other words, all cameras using AllocateMemory pool:

magic-lantern/platform/700D.114/Makefile.setup.default
#Makefile for 700D

CONFIG_LVAPP_HACK_RELOC = y


Here is what I found using the EOSM2 as an example:

magic-lantern/platform/700D.114/consts.h
// Used in boot-hack.c with CONFIG_ALLOCATE_MEMORY_POOL
#define ROM_ITASK_START 0xff0c5408
#define ROM_ITASK_END  0xff0ca68c
#define ROM_CREATETASK_MAIN_START 0xff0c3000
#define ROM_CREATETASK_MAIN_END 0xff0c339c
#define ROM_ALLOCMEM_END 0xff0c304c
#define ROM_ALLOCMEM_INIT 0x6fc4
#define ROM_B_CREATETASK_MAIN 0xff0c547c


And of course:

magic-lantern/platform/700D.114/internals.h
/** This camera loads ML into the AllocateMemory pool **/
#define CONFIG_ALLOCATE_MEMORY_POOL


Taking into account the different addresses between the 700D and EOSM2, the results were pretty much identical.

File size : 0x701C0
Now jump to AUTOEXEC.BIN!!
0010E920: MRC p15,0,Rd,cr1,cr0,0:      SCTLR -> 0xC005107D
0010E920: MCR p15,0,Rd,cr1,cr0,0:      SCTLR <- 0xC005107D
[BOOT] changing init_task from 0xff0c5408 (-15969272) to 0x7f3e8 (521192)
[BOOT] autoexec.bin loaded at 7EC00 - 106940.
Fixing from ff0c5408 to ff0ca68c


ff0c85cc: 3b9aca00 BL ff9aca00 => fd77add4
ff0c85cc: !!!! can not fixup jump from 000f3dec to fd77add4 (offset -00a5e408)
...
ff0c8c00: 3b9aca00 BL ff9aca00 => fd77b408
ff0c8c00: !!!! can not fixup jump from 000f4420 to fd77b408 (offset -00a5e408)

a1ex

Quote from: nikfreak on July 22, 2017, 07:23:33 PM
as discussed here I failed to use AllocateMemory pool but:

I'm ready If you want to give it another go.

Got it working in QEMU, on both 6D and 100D (they are very similar, and 700D is from the same group). EOSM2 is just a tiny bit different.

You'll probably need new-dryos-task-hooks for EOSM2 as well. Hopefully this time I'll get the confirmation from 6D and 70D in order to merge it in mainline :D

Audionut

Quote from: a1ex on July 24, 2017, 03:03:36 PM
Hopefully this time I'll get the confirmation from 6D

Need to get a compiler up and running again, then I can test.

dfort

Great news. I got the the latest 100D with AllocateMemory pool working in QEMU before running off to work this morning. Looking forward to figuring out how to apply this to the EOSM2 (and 700D and EOSM).

@Audionut -- I recently updated the Macintosh and Windows (Cygwin) tutorials so you should be able to get a development environment up and running in a matter of minutes. Of course if you are on Linux you probably already have all you need.

dfort

Still on a detour working with the 700D but got full ML Hello World working with AllocateMemory pool.


dfort

Still on detour -- this time with the original EOSM running on a real camera. (Not easy doing a Screenshot of that Free Memory screen on the EOSM.)

AllocateMemory pool


Nightly.2017Jul16.EOSM202 from downloads page (jenkins build)


And the shutter-bug is still present--ugh!

Ok--back to the EOSM2 after taking a break.

dfort

AllocateMemory pool seems to be working on the EOSM2

File size : 0x67D40
Now jump to AUTOEXEC.BIN!!
0010E5F4: MRC p15,0,Rd,cr1,cr0,0:      SCTLR -> 0xC005107D
0010E5F4: MCR p15,0,Rd,cr1,cr0,0:      SCTLR <- 0xC005107D
[BOOT] changing init_task from 0xff0c57a0 (-15968352) to 0x44c984 (4508036)
[BOOT] autoexec.bin loaded at 44C100 - 4C6940.
Fixing from ff0c57a0 to ff0c5978
...
Fixups=4b6638 entry=4b6640 free_space=8
[BOOT] changing AllocMem limits:
0x004b6680:  e59f1158      ldr r1, [pc, #344] ; 0x4b67e0
0x004b6684:  e59f0158      ldr r0, [pc, #344] ; 0x4b67e4
0x004b6680:  e59f1158      ldr r1, [pc, #344] ; 0x4b67e0
0x004b6684:  e3a0084e      mov r0, #5111808 ; 0x4e0000
: %x
READY
128K Sector FROM From BL 0xffff


Things seem to go fine and the Canon menu pops up, it pauses for a while then:

[      init:0044cb84 ] task_create(ml_init, prio=1e, stack=4000, entry=44c528, arg=0)
[     ml_init:ff696760 ] (23:01) sdReadBlk: st=256, num=32, buf=0x4035e000
[     ml_init:ff6951bc ] (23:01) sdDMAReadBlk: st=256, num=32
qemu: fatal: Trying to execute code outside RAM or ROM at 0x20000000

R00=ffffffff R01=009fbc5a R02=001f3912 R03=00000000
R04=19980030 R05=00000000 R06=597375e6 R07=19980218
R08=1998002e R09=19980218 R10=19980218 R11=19980218
R12=0000000f R13=001f3918 R14=0044d1d0 R15=20000000
PSR=60000013 -ZC- A svc32
FPSCR: 00000000
qemu: fatal: Trying to execute code outside RAM or ROM at 0x20000000


Hum, where have I seen that before? Oh yeah, when I tried it with HIJACK_CACHE_HACK. The error is nearly identical:

Quote from: dfort on July 22, 2017, 12:59:44 AM
Then it gets ugly:

[      init:0010137c ] task_create(ml_init, prio=1e, stack=4000, entry=101510, arg=0)
[     ml_init:ff696760 ] (23:01) sdReadBlk: st=73344, num=32, buf=0x4035e000
[     ml_init:ff6951bc ] (23:01) sdDMAReadBlk: st=73344, num=32
qemu: fatal: Trying to execute code outside RAM or ROM at 0x20000000

R00=ffffffff R01=00987c0a R02=001f3912 R03=00000000
R04=19980030 R05=00000000 R06=596f0ae0 R07=19980218
R08=1998002e R09=19980218 R10=19980218 R11=19980218
R12=00000007 R13=001f3918 R14=00101cc0 R15=20000000
PSR=60000013 -ZC- A svc32
FPSCR: 00000000
qemu: fatal: Trying to execute code outside RAM or ROM at 0x20000000


On the 700D it prints "Hello World" and the firmware signature. That same bit of code that's tripping up the EOSM2 looks like this on the 700D:

[     ml_init:ff74c988 ] (23:01) sdReadBlk: st=256, num=32, buf=0x402fe000
[     ml_init:ff74b3e4 ] (23:01) sdDMAReadBlk: st=256, num=32
[     ml_init:ff74c988 ] (23:01) sdReadBlk: st=320, num=32, buf=0x40306000
[     ml_init:ff74b3e4 ] (23:01) sdDMAReadBlk: st=320, num=32
...


From the looks of that message it seems like it is having a problem reading the SD card image file. The 700D saves a ROM dump but the EOSM doesn't so maybe it is a write issue? The address of that "ml_init" points to something named "sdWriteBlk: Emergency Stop3" in the firmware. Stumped again--still no "Hello World" with the full ML and of course no firmware signature yet.

These messages might be a clue:

[   CSMgrTask:ff6978ec ] (23:03) ---- SDEventHandler(ID=1:Event=8) ----
    64:   129.536 [MR] mvrChangeAckCBR : Video - Mode=0, Type=2, Rate=30, GOP=15
    82:    71.936 [PROPAD] ERROR GetPropertyData ID (1) = 0x01050038
    83:    71.936 [PROPAD] ERROR GetPropertyData ID (1) = 0x01050041
    84:    71.936 [PROPAD] ERROR GetPropertyData ID (1) = 0x01050042
    85:    71.936 [PROPAD] ERROR GetPropertyData ID (1) = 0x01050043
    86:    71.936 [PROPAD] ERROR GetPropertyData ID (1) = 0x01050044
    87:    71.936 [PROPAD] ERROR GetPropertyData ID (1) = 0x0105004F
    88:    71.936 [PROPAD] ERROR GetPropertyData ID (1) = 0x01050050
    89:    71.936 [PROPAD] ERROR GetPropertyData ID (1) = 0x01050051
    90:    72.192 [PROPAD] ERROR GetPropertyData ID (1) = 0x01050052
    91:    72.192 [PROPAD] ERROR GetPropertyData ID (1) = 0x0105010E
    92:    72.192 [PROPAD] ERROR GetPropertyData ID (1) = 0x0105010F
    93:    72.192 [PROPAD] ERROR GetPropertyData ID (1) = 0x01050110
    94:    72.192 [PROPAD] ERROR GetPropertyData ID (1) = 0x01050111
    95:    72.192 [PROPAD] ERROR GetPropertyData ID (1) = 0x0104000B
    96:    72.192 [LVCOM] InitializeLiveViewDefectDetection
    97:    72.192 [LVCOM] ExecuteDefectMarge Start
    98:    72.448 [LVCOM] ExecuteDefectMarge End
   100:    76.800 [LV]  PROP_LV_BLOCK  PROP_LV_UNBLOCKING 0
   102:    78.336 [AUDIO] RegisterCBRSDIOCableConnect
   103:    85.248 [WFT] InitializeAdapterControl END


That [WTF] seems like a good way to end this post.

dfort

Found something that may or may not shed light into what is causing the EOSM2 to crash when running the full ML "Hello, World!" I was able to reproduce the crash outside of ML:

./run_canon_fw.sh EOSM2,firmware="boot=0" -d debugmsg -s -S & arm-none-eabi-gdb -x EOSM2/debugmsg.gdb

...
[  DisplayMgr:ff1790cc ] (9a:02) fTunOnDisp = On
^C
Program received signal SIGINT, Interrupt.
0xff352318 in ?? ()
(gdb) set $pc=0xff697870
(gdb) c
Continuing.
qemu: fatal: Trying to execute code outside RAM or ROM at 0x20000000

R00=c0400000 R01=00000080 R02=19980218 R03=19980218
R04=0000021c R05=0017ee2c R06=19980218 R07=19980218
R08=19980218 R09=19980218 R10=19980218 R11=19980218
R12=19980218 R13=0018b330 R14=ff352220 R15=20000000
PSR=600000d3 -ZC- A svc32
FPSCR: 00000000
qemu: fatal: Trying to execute code outside RAM or ROM at 0x20000000


That is pretty much the error I've been getting when trying to run ML on the EOSM2 (in QEMU of course) and shows up outside of ML when calling a function that has this bit of interesting information in it:

ff6978dc: e28f2f6e add r2, pc, #440 ; ff697a9c: (2d2d2d2d)  *"---- SDEventHandler(ID=%d:Event=%d) ----"


So what is the state of SDEventHandler at this point?

(gdb) set $pc = 0xff6978ac
(gdb) c
Continuing.
[    PowerMgr:ff51d13c ] (1e:01) MapLogToPhysic: pLStorage=0x19980218
[    PowerMgr:ff51d184 ] (1e:06) MapLogToPhysic: Logical Storage Not Found(BAD_LOGICAL_STORAGE)
   185:  1118.720 [CSMGR] ERROR MapLogToPhysic: Logical Storage Not Found(BAD_LOGICAL_STORAGE
[    PowerMgr:ff6978ec ] (23:03) ---- SDEventHandler(ID=429392408:Event=128) ----
[    PowerMgr:ff697dfc ] (23:06) Invalid Event(Event=128)
   186:  1118.976 [SD] ERROR Invalid Event(Event=128)


Switching over to the 700D and calling these same functions created the same messages. So what is the point? Somehow when running ML the EOSM2 is tripping up on this while the 700D sails right through it and displays "Hello, World!" along with the firmware signature. I've been double checking suspect stubs and constants and as far as I can tell they are correct. At least when checking against similar cameras like the EOSM, 100D and 700D.

So what is going on? I don't know, I haven't been successful setting breakpoints and stepping through the code. Looks like I need to do a remedial class in GDB source code debugging.

Oh no--the dog ate my homework!



Quote from: dfort on July 16, 2017, 08:02:45 PM
...the graphs on the placing ML into shoot memory topic have disappeared when using the Chrome browser. Looking into the issue it seems that Google has been implementing stricter rules that is affecting the link...