Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Messages - a1ex

Pages: [1] 2 3 ... 377
Camera-specific discussion / Re: Canon 1100D / T3
« on: Today at 05:24:28 PM »
malloc error :((sad smiley from liveview window)

Solved. Took many of hours to rework the memory backend (found lots of other issues besides this one), but now it runs on all models in QEMU. Overall, the memory backend should be a lot more robust.

Torture test: old-style Lua (without umm_malloc) on 1100D (the camera with lowest amount of memory). It finally worked! 8)

Scripting Corner / Re: Simulated Q Button for EOSM
« on: Today at 12:08:11 PM »
Too bad this is not implemented in the build for EOS-M though.

Please try the experimental lua_fix build (it uses a long SET/Q press to open submenus), and report back if you want this feature in mainline (since I need to know how well it works in order to include it).

Camera-specific discussion / Re: Canon 7D Mark I
« on: Today at 12:03:14 PM »
7D raw was a lot more stable back then [late 2016]. I don't have this kind of stability anymore.

Old builds are still available; if you can identify a regression, please do a proper bug report.

In particular, since 2016, the main builds received almost exclusively bug fixes on the raw recording side (look at the changes). In particular, for video, the relevant changes since September 16 2016 (15eb9aa) are:
- mlv_rec: RAWC metadata, minor file I/O fix, minor speed tweak (round MLV header to 512 bytes), version metadata
- raw_rec -> mlv_lite
- file I/O backend: fixed an important bug that resulted in invalid video files (like 1 bad video out of a few hundreds)
- raw backend: use the same raw type for all cameras, autodetect resolution rather than hardcoding it, raw capture info (pixel binning factors), preview fixes.

Even when going back to January 2016, the only relevant changes (between f87b759 and 15eb9aa) are:
- mlv_rec: fixed pink frames, wrong black level, crop offsets, added RAWC metadata, refactored powersave, blocked the zoom key while recording
- mlv_snd: persistent settings, show audio meters when recording sound (and hide them when not recording)
- raw backend: black/white level fixes.

That was all on the raw recording side. Notable new features that landed in mainline in this period: Lua scripting, advanced intervalometer, 5D3.123, 700D.115, Makefile updates, various refactors, benchmarks/tests moved to modules, submenu icons, various minor updates.

Given the above, I'm very surprised about your comment ("I don't have this kind of stability anymore."), so please report the issue properly.

BTW, noticed you have been trying the 10/12-bit builds. I hope you are not talking about them - are you aware these builds are not present on the download page, not even as experimental builds, for the 7D? There is a reason for that ;)

How is the 2.0.6 build?

I have absolutely no idea - the current feedback doesn't allow me to draw a conclusion. Are you aware that we are relying on user feedback?

Camera-specific discussion / Re: Canon 6D
« on: Yesterday at 09:47:24 PM »
Not exactly; there is an option to save the log to a file (CONSOLE_DEBUG in console.c), but that's pretty much it. However, it's not a bad idea to have a larger - scrollable - buffer.

Camera-specific discussion / Re: Canon 6D
« on: Yesterday at 09:26:47 PM »
Looks OK. Same numbers when starting in other modes? (1080p, 720p etc)

Camera-specific discussion / Re: Canon 6D
« on: Yesterday at 07:49:59 PM »
If default buffer size is wrong, how can we fix it, can we check these values, or can we do some trial and error ?

Here (just needs enabling a compile-time option).

General Development Discussion / Re: placing ML into shoot memory
« on: Yesterday at 10:37:40 AM »
600D is linked earlier (you can do the math). In some cases, ML runs out of general-purpose memory and allocates from shoot_malloc (Debug -> Free Memory); whenever such memory is used for a longer time (such as, many/large modules loaded), certain operations in Canon menu can not be performed (such as switching some ISO options, or developing raw photos in camera, maybe others) and will result in ERR70.

You can tell exactly what these situations are.

Having 256K of additional memory could make the difference between having to use shoot_malloc (with the problems it has) or not.

The issue is probably subtle enough that nobody would ever bother narrowing it down to report it (they would probably just reset ML to defaults).

Edit: a quick test in QEMU, with and loaded (but no features actually used) gives error on shutdown (might or might not translate to ERR70 on real hardware, not sure):
Code: [Select]
   167: 22539.008 [MEM] NG AllocateMemory 262176
   168: 22539.008 [MEM] Total = 0x8a0000, Free = 0x81320, MaxReg = 0x2e248
   169: 22539.520 [MEM] Num Alloc = 3463, Num Free = 287
   170: 22542.080 [STARTUP] ERROR ASSERT : Memory\Memory.c, Task = GuiMainTask
   171: 22542.080 [STARTUP] ERROR ASSERT : Line 188
   172: 22542.080 [STARTUP] ERROR ASSERT : FALSE

Loading Lua alongside with a few other modules will result in shoot_malloc being used. Try changing everything you can think of in Canon menu in this configuration, and I'm sure you'll find more issues, not just the ones I know about.

Camera-specific discussion / Re: ML on EOS-M2
« on: Yesterday at 12:09:21 AM »
It probably needs to go in boot-hack.c before init_task - that way, it will affect all the ENGIO calls. However, for testing, you may want to try from "don't click me" first (it would take effect after a LiveView refresh). Might also work as an INIT_FUNC, though if these happen to run during LiveView, you may require to refresh LiveView before things start working.

Commit 90f702c (you have already used it).

I couldn't go further than that because of the memory? When I did I got "too small srm memory" on the screen and this:

This is great news - it means the safeguards regarding raw buffer size are working!

When the image size exceeds the amount (what we believe) Canon code has reserved for the raw buffer, the raw backend should allocate a full-size image. A possibly cross-platform solution (edit: committed) is something like this:
Code: [Select]

That's because SRM_BUFFER_SIZE (the largest contiguous block we can get from Canon's memory manager) matches the full-res image size (as 14-bit uncompressed; actually it's a tiny bit larger than that). You need to use a slightly smaller amount because our memory backend also has some overhead.

Yes, the two FPS timers are the only ways to control the FPS. The nearby registers might be useful to break some limits, but their exact meaning is unknown.

Another helpful change is in raw.c - by default, the optical black areas are checked for consistency (e.g. to avoid wrong black level for corrupted frames, which are usually transient). In autodetect_black_level, just set *black_mean to 2048 and *black_stdev_x100 to some nonzero value, without doing all the math - this will help previewing raw buffers where only half of the image is valid, for example.

Camera-specific discussion / Re: ML on EOS-M2
« on: February 22, 2018, 06:27:21 AM »
We have 3 ways to proceed:

1) hardcode the resolutions (and get bitten by this Canon bug in other parts of the code)
2) reallocate this buffer from somewhere else (where?)
3) patch ENGIO routines to use ADD instead of ORR, to make them able to work on unaligned buffers

I'd try the last route first. These addresses are:
Code: [Select]
FF2C6C40 (in EngDrvOut)
FF2C6CD8 (in EngDrvIn)
FF2C6D30 (in EngDrvBits)
FF2C6D3C not
FF2C6DA4 (in EngDrvOuts)
FF2C6F64 ORRNE (in engio_write)
FF2C6F94 (in EngDrvOut_mirror)
FF2C6FC4 (in EngDrvBits_mirror)
FF2C6FD0 not
FF2C6FF4 (in engio_memcpy_maybe)

Code: [Select]
FF2C6C40 02 00 80 E1                 ORR     R0, R0, R2

should be changed to:
Code: [Select]
FF2C6C40 02 00 80 E0                 ADD     R0, R0, R2

With the patch manager (patchmgr) merged:
Code: [Select]
patch_instruction(0xFF2C6C40, 0xE1800002, 0xE0800002, "engio");

The other instructions should be changed in the same way, and tested in QEMU first.

General Development Discussion / Re: placing ML into shoot memory
« on: February 22, 2018, 05:58:09 AM »
Not really; it was linked as documentation for the above procedure (to make sure some possibly free memory area is really unused). This extra memory is only available for DIGIC 4 models (they allocate 0x80000 - twice as much - to be able to align this buffer). That's not needed in my opinion - they could have just used ADD instead of ORR...

The models that would benefit even from this - small - extra memory are 1100D and 600D.

General Development Discussion / Re: placing ML into shoot memory
« on: February 21, 2018, 10:37:43 PM »
From this EOSM2 quirk:

The ENGINE_MIRROR region from the above logs is used by engio_read / EngDrvOut to store the values written to hardware registers, as these registers cannot be read back. This address must be aligned at 0x40000 bytes, but the block allocated for it might be a bit larger.

Some examples:

60D: ENGINE_MIRROR 0x5C849960 0x00080000
Actual mirror address is 0x5c880000 (rounded up near 0xff1c566c), used until 0x5c8c0000.
0x5c880000 - 0x5C849960 = 217K free before this block.
0x5C849960 + 0x00080000 - 0x5c8c0000 = 38K free after this block.

1100D: ENGINE_MIRROR 0x40FF7000 0x00080000
Actual mirror address is 0x41000000 (rounded up near 0xff1d4890), used until 0x41040000.
0x41000000 - 0x40FF7000 = 36K free before this block.
0x40FF7000 + 0x00080000 - 0x41040000 = 225K free after this block. Not bad!

EOSM2: ENGINE_MIRROR 0x412A0000 0x00044000
Actual mirror address 0x412A0000 (somebody forgot to round it up). Good luck fixing that...

550D: ENGINE_MIRROR 0x4F757200 0x00040000
Actual mirror address: 0x4f780000 (rounded up near 0xff1c1228), used until 0x4f7c0000. Buffer overflow?! (overlaps VSHADING_COMP_WORK)

The above is just theory; I did not verify whether the addresses are really unused or not. Procedure: CONFIG_MARK_UNUSED_MEMORY_AT_STARTUP, then use the camera in any mode you may think of, including taking pictures in burst mode until the buffer gets full, then look for changes in the tested memory region. See also PR #731.

Camera-specific discussion / Re: ML on EOS-M2
« on: February 21, 2018, 09:36:11 PM »
Code: [Select]
0.027.660         Evf:ff2c6f58:MMIO : [0xC0F06800] <- 0x00010013
0.027.661         Evf:ff2c6f58:MMIO : [0xC0F06804] <- 0x02D801D7

These values are OK, both set from engio_write. I could not find other instances. What happened to the shadow memory, then?!

Generally, ENGIO registers cannot be read back directly (with very few exceptions), so Canon code also writes their value into RAM (possibly to read them back later, or for debugging). On M2, this shadow memory is at 0x9696C -> 0x412a0000, and a register is written there like this:
Code: [Select]
*reg = value;
*(reg & 0x3FFFF | shadow_addr) = value;

In other words, the value written into register 0xC0F06800 would be also written to memory at 0x412a6800.

We've got a problem: 0x412a0000 & 0x3FFFF is not 0. The value written to 0xC0F26800 (EDMAC channel 0x18) will be also written to the same memory address, at 0x412a6800. When reading this back, we'll get whatever was written last (of these two registers).

It's the first model I'm aware of with this problem. 5D3 has the shadow memory at 0x29C04 -> 0x41700000. 700D: 0x31708 -> 0x41400000. For other models, the value can be found in gdb+qemu; I didn't even think about it until now, since it just worked.

Can these registers be read directly from hardware? (answer: no - they won't crash, but they will return just 0 or 1)

BTW, this probably means the ENGIO register range is from 0xC0F00000 to 0xC0F3FFFF ("only" 65536 registers?) and the other ranges are not meant to be written with these routines (engio_write / EngDrvOut). I should enforce this in the code.

Stack trace to see where this shadow memory is allocated:
Code: [Select]
0x35F20(6109f0 &"StageClass", 35f20, 19980218, 19980218)                         at [ShootCapture:ca14:19dfc8] (pc:sp)
 0xFF0D38C0(406108e8 &"ShootCapture", 0, ff0c5724, 40000)                        at [ShootCapture:35f7c:19dfa0] (pc:sp)
  0x3637C(61096c &"StateObject", 406108e8 &"ShootCapture", 0, ff0c5724)          at [ShootCapture:ff0d38ec:19df88] (pc:sp)
   0x363B4(61096c &"StateObject", 406108e8 &"ShootCapture", 0, ff0c5724)         at [ShootCapture:363ac:19df78] (pc:sp)
    0xFF1567B4(406108e8 &"ShootCapture", ff0c5724, 40000, ff1567b4)              at [ShootCapture:36434:19df58] (pc:sp)
     0xFF23398C(412a0000, 44000, 1, ff1567b4)                                    at [ShootCapture:ff156810:19df20] (pc:sp)
      0xFF4ADB84(412a0000, 44000, 1, 0)                                          at [ShootCapture:ff2339cc:19dee0] (pc:sp)
       0xFF2C6BE8(412a0000, 44000, 1, 0)                                         at [ShootCapture:ff4adb8c:19ded8] (pc:sp)

Size appears hardcoded to 0x44000 (so, 412a0000 - 0x412e4000). Doesn't help much. Can be found on the RscMgr memory map as well:
Code: [Select]
ENGINE_MIRROR           0x412A0000 0x00044000 278528

We might be able to find another buffer for this.

Camera-specific discussion / Re: ML on EOS-M2
« on: February 21, 2018, 08:26:18 PM »
Ouch, these pixels over the screen don't look good at all.

The bottom bar hack can be commented out from internals.h (it's not needed for the log).

The extra loggers are still there, and very verbose; comment them out.

Camera-specific discussion / Re: ML on EOS-M2
« on: February 21, 2018, 07:49:03 PM »
CONFIG_DEBUG_INTERCEPT=y and CONFIG_MMIO_TRACE=y in Makefile.user? Also, make clean?

Camera-specific discussion / Re: ML on EOS-M2
« on: February 21, 2018, 03:51:00 PM »
Comment out all the "extra" stuff (calls to dm_spy_extra_install/uninstall) and only record a small subset of the messages (for example, those starting with 'e' from 'evf'). The default set is too verbose to cover the entire LiveView startup (and the overhead of all these messages might be too large). Also, keep in mind CONFIG_DEBUG_INTERCEPT_STARTUP is not compatible with CONFIG_MMIO_TRACE in the io_trace_full branch (the main reason it's yet another branch). Only plain CONFIG_DEBUG_INTERCEPT works there (with manual selection from menu).

The message filtering is done in my_DebugMsg, somewhere at the beginning:
Code: [Select]
    if (fmt[0] != 'e') return;

You could also write small test sequences, such as:
Code: [Select]
static void run_test()
    SetGUIRequestMode(GUIMODE_PLAY); // go to PLAY mode
    debug_intercept();     // start logging
    SetGUIRequestMode(0);  // back to LiveView
    debug_intercept();     // stop logging

Scripting Corner / Re: Auto Hand Held Bracketing Script
« on: February 21, 2018, 03:15:53 PM »
1) Keeping in mind that some of the shots are taken at high ISO - 6400, where does the noise go?  Averaging???  Your examples look amazingly clean.

Misconceptions 101 - higher ISOs have less noise, all other exposure variables being equal (including the brightness of the output image). This was discussed many times, and Garry himself linked to a paper on this subject a few posts ago.

In particular, for 5D3, log2(34.9 / 2.5) = almost 4 stops of better shadow performance with ISO 6400, compared to ISO 100. Price to pay for that: 6 stops of clipped highlights.

When bracketing, you take the highlights from other images, possibly captured at lower ISOs and/or faster shutter speed. Problem solved. The same happens with Dual ISO.

P.S. compare these plots with the ones from Hasinoff's paper (in particular, Fig. 2b) :)

Camera-specific discussion / Re: ML on EOS-M2
« on: February 21, 2018, 02:31:23 PM »
All these logs show the GPIO range. The log has to be captured in LiveView; otherwise it won't be useful. Starting from PLAY mode is OK as long as it also goes to LiveView. Starting from LiveView may or may not be useful, depending on where this register is set. Starting into a video mode with lower FPS may help (don't start into 60p).

Lock-ups are hard to debug - some hints here.

The io_trace_full branch might also help, but for small ranges like this, I wouldn't expect major improvements.

Scripting Corner / Re: Lua Scripting (
« on: February 21, 2018, 01:38:09 PM »
Drawing on the screen is currently done by writing directly to video memory. In all other models, Canon code only redraws when it has something to redraw (with minor exceptions). Not so with 50D - here, Canon code redraws itself over and over (and for some unknown reason, it does not redraw the entire screen, just a very large part of it).

Redraw does not have visible side effects on its own - Canon code is flicker-free when redrawing itself. Try running the rectangle test at the same time.

lv_focus_status will probably fail in LiveView - if it's like 500D, it won't autofocus by pressing half-shutter. Not sure how to fix.

is_pure_play_photo_mode() -> this should return true when reviewing an image in PLAY mode (it's tested from Lua as well). This works by comparing the return value of get_current_dialog_handler() with some predefined stub (PlayMain_handler - Canon dialog function used for reviewing images). To check, print the result of get_current_dialog_handler() somewhere while in PLAY mode, and compare it with the PlayMain_handler stub.

Not sure if this parameter can be altered during image capture, and even if it does, it would have very tight timing requirements. Normally, the main CPU configures the exposure parameters, starts it, and waits until it's finished (actually it has a lot of other tasks to run during this wait).

We have no datasheet for the ADTG chip (the timing generator that handles line skipping/binning); all we know about it is what we have found by trial and error (adtg_gui), so I have no idea where to start in order to configure the chip for variable line skipping (or whether it can do that).

Camera-specific discussion / Re: ML on EOS-M2
« on: February 21, 2018, 12:54:03 PM »
Okay, somebody (some Canon code) must be overwriting the value; adtg_gui can tell what happened, but it requires some fiddling (it won't do so out of the box - too many registers out there); a better tool for diagnosing this would be io_trace (monitored range: 0xC0F06000, 0x1000).

Camera-specific discussion / Re: Canon 6D
« on: February 21, 2018, 12:31:27 PM »
Wow, that post from @Levas is from 3 1/2 years ago. It took that long to make it easy?

FYI, the original crop_rec module was announced 2 years ago, and it referenced Levas' post from the very beginning. The module was a lot simpler back then, since it only had one preset (you already know).

Best guess: the feature wasn't tempting enough for other folks to look at the source code.

Camera-specific discussion / Re: Canon 6D
« on: February 21, 2018, 01:34:35 AM »
You should probably just provide a declaration; then, to get changes into mainline, the general procedure is to provide a pull request, and other 6D owners to confirm it's working (since I'm unable to test otherwise, except for some limited stuff in QEMU, such as how the menus look like, or whether it turns off the display or not).

@dfort: yes, it's the same function. If these functions become widely used, it would make sense to add them to stubs.S, along with some generic callbacks that other modules (dual_iso, crop_rec etc) could use to override these register values. Need to think a bit about it, as it's not a straightforward change (at least not for me).

Something like this would be also welcome for the ISO tweaks (to implement them in a way that doesn't conflict with existing features - something which was holding me back, besides the complexity of that topic).

The issue can be reproduced with theBilalFakhouri's ROM in QEMU.

First step: what Canon code writes to this address range? (0x40452044 on the "good" firmware).

Code: [Select]
# this goes into 700D/debugmsg.gdb
# note: 0x40000000 is the uncacheable memory flag, so both addresses refer to the same physical memory location
watch *0x40452044
watch *0x452044

# run with:
./ 700D,firmware="boot=1" -s -S & arm-none-eabi-gdb -x 700D/debugmsg.gdb

No results that way.

Second try: use our own logging framework (-d ram). The logs obtained that way would be huge, so add this into logging.c, eos_log_mem, right after the call to should_log_memory_region_verbosely:

Code: [Select]
    if ((addr & 0x0FFFFFF0) != 0x00452040)

Code: [Select]
./ 700D,firmware="boot=1" -d ram |& grep -a -C 5 ram
[EEPROM-DMA]! [0x340000] -> [0x40450E00] (0xBB860 bytes)
[ram]   at Startup2:FF139C24:FF139C24 [0x40452040] <- 0xFF      : was 0x0; 8-bit

A-ha! This data is copied from the serial flash.

On the bad firmware:
Code: [Select]
[EEPROM-DMA]! [0x340000] -> [0x40451200] (0xBB860 bytes)

Alright, so next step would be to find out where 0x40450E00 or 0x40451200 is allocated, and how to find a pointer to it. Grep is usually good for finding needles in the haystack, so...

Code: [Select]
./ 700D,firmware="boot=1" -d ram,io,calls,tail,sflash,debugmsg |& grep -a -C 5 -i 40450E00
   call 0xFF13AB98(340000, 40450e00, bb6dc, 100000)                              at [Startup2:ff12087c:ff123344]
    call 0xFF139FFC(340000, 40450e00, bb6dc, c0820404)                           at [Startup2:ff13abfc:ff120880]
[    Startup2:ff13a028 ] (00:01) [SF] readSerialFlashWithQuad Dest 0x40450e00 Src 0x340000 Size 767708
     call 0xFF139AE8(bd000, 40450e00, bc, ffffff3e)                              at [Startup2:ff13a3f4:ff13ac00]
      [SFDMA] at Start:FF139BE8:FF139BE8 [0xC0530060] <- 0x40450E00: Transfer memory address
[EEPROM-DMA]! [0x340000] -> [0x40450E00] (0xBB860 bytes)

All of the above addresses are good candidates to be looked up in the disassembly.

Stack trace for the call to readSerialFlashWithQuad:
Code: [Select]
./ 700D,firmware="boot=1" -d callstack -s -S & arm-none-eabi-gdb -x 700D/debugmsg.gdb
(gdb) b *0xff13a028
Breakpoint 4, 0xff13a028 in ?? ()
(gdb) print_current_location_with_callstack

Current stack: [14bf60-14bb60] sp=14be88                                         at [Startup2:ff13a028:ff13ac00]
0xFF0C35B0(0, ff0c35b0, 19980218, 19980218)                                      at [Startup2:bba0:14bf58] (pc:sp)
 0xFF1232FC(14bf44, 1, 340000, 1000)                                             at [Startup2:ff0c3634:14bf28] (pc:sp)
  0xFF1206E8(450d3c, 450d64, 23b24, 0)                                           at [Startup2:ff123340:14bf10] (pc:sp)
   0xFF13AB98(340000, 40450e00, bb6dc, 100000)                                   at [Startup2:ff12087c:14bed0] (pc:sp)
    0xFF139FFC(340000, 40450e00, bb6dc, c0820404)                                at [Startup2:ff13abfc:14beb0] (pc:sp)

In particular, 0xFF1232FC is... PROPAD_CreateFROMPropertyHandle : FIX.

Wait, what?! All these years, we've been writing in the middle of Canon's property data structures without even knowing about it?! Guess we were lucky, as I believe the FIX settings block is probably not written back to the permanent memory at shutdown... (one small piece added to this puzzle).

Back to our quest: who allocates memory for this? There are some calls to alloc_dma_memory around, so let's trace them with gdb:
Code: [Select]
b *0x16e8c
  printf "alloc_dma_memory(0x%x)\n", $r0

b *0xFF1232FC
  printf "PROPAD_CreateFROMPropertyHandle(0x%x, 0x%x, 0x%x, 0x%x, 0x%x, 0x%x, 0x%x)\n", $r0, $r1, $r2, $r3, MEM($sp), MEM($sp+4), MEM($sp+8)

Code: [Select]
(./ 700D,firmware="boot=1" -d debugmsg -s -S & arm-none-eabi-gdb -x 700D/debugmsg.gdb) |& grep -a -C 50 -i 40450E00
[    Startup2:ff0c3634 ] PROPAD_CreateFROMPropertyHandle(0x14bf44, 0x1, 0x340000, 0x1000, 0x100, 0x100000, 0x1000000)
[    Startup2:ff1230a8 ] alloc_dma_memory(0x1000ff)
[    Startup2:ff1230ac ]  => 0x40450dc4
[    Startup2:ff13a028 ] (00:01) [SF] readSerialFlashWithQuad Dest 0x40450e00 Src 0x340000 Size 767708

Found where it's allocated, now to figure out where the result is stored. Or, maybe better, as I prefer to avoid messing with the property data structures - refactor with the crop_rec method for overriding these registers. That would require finding the cmos_write stub for all camera models.

Reference for the above commands: README.rst - Debugging with GDB.

Pages: [1] 2 3 ... 377