Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Messages - a1ex

Pages: [1] 2 3 ... 335
Minor progress towards integration in the main tree: added a lua_fix experimental build. It's 100% untested on this model (I've tried it on 5D3 and 500D).

In particular, for this build, I'm interested in:
- logs from and api_test.lua
- any error screens that might pop up under regular usage (with steps to reproduce)
- whether raw video is working in all video modes
- whether raw video histogram changes with digital ISO (e.g. whether ISO 160/200/250 give different raw histograms in movie mode or not)

Scripting Corner / Re: Lua Scripting (
« on: Yesterday at 10:52:06 PM »
I think I've found a fix for file_man crashing. Maybe not the cleanest way, but at least I no longer get warnings in QEMU. Ended up doing some major changes on the menu backend, such as enforcing valid names on every single menu entry (to avoid null pointer issues).

Also renamed dryos.prefix to dryos.image_prefix and made sure it's actually working (with an API test that also shows how to get the file name of the current image).

Didn't look into stack issues yet.

Scripting Corner / Re: Eclipse exposure sequence generator.
« on: Yesterday at 07:33:09 PM »
leaving the event.seconds_clock() function active was leading to semaphore errors.

That's the current state of multitasking in Lua, unfortunately. The event handlers cannot run while the main task is busy.

vibration caused by mirror slap could be largely reduced by shooting in LiveView.

ML also has a tool to address this, though I have no idea whether it works on 70D (don't remember any reports - looks like it's disabled on this camera). Apparently nobody used this feature in the last 3 years (on the cameras where it's enabled).

I think I may be uncovering issues because I am hammering the camera hard.

That's probably the best way to test, and proper bug reports are always appreciated here.

Camera-specific discussion / Re: Canon EOS M
« on: Yesterday at 06:27:35 PM »
0xFFA73C84 is a wrapper for 0xFFA880D8 (the only difference being at error handling, which never happened within our usage - yet).

For consistency, it's best to use the same stub on all models. Probably a good candidate for automation (the algorithm is in ARM-console, but that one does have some weird dependencies...)

General Development Discussion / Re: Updating via USB
« on: Yesterday at 04:13:32 PM »
1. Last time I've checked ML PTP was many years ago. To my knowledge, CHDK's is light years ahead.

2. "make ptpinstall", written many years ago, only uploads autoexec.bin.

3. Probably - contributions welcome.

See also

Camera-specific discussion / Re: Canon 6D
« on: Yesterday at 09:28:24 AM »
@ a1ex if i can help somehow im in :)

Just look up Audionut's signature.

BTW on my 6D when I view (play) the images and when i press the LIGHT button (uppers screen light button) it really displays the histogram and zebras or focus peaking dost on the image.

Yes, that button should be mentioned on the help tab.

But I noticed the histogram is not the RAW histogram, i think it is generated from the preview image..?
It would be nice to be able to see the RAW histo and even the ETTR hint whenever you want for all saved images

Until recently, that required running dcraw on the camera (very very slow); however, some days ago, g3gg0 found out how to use the hardware lossless decoder (which is what we need to decode the CR2). Still, some small things are left to reverse-engineer, such as the file name of the image being reviewed.

Finding these should be straighforward in QEMU. For emulating the 6D (which is something I'd like to do regardless of this particular feature), I need a copy of the the serial flash contents. One can get it with the sf_dump module (it needs to be updated for 6D and compiled from source). Very easy coding task.

General Help Q&A / Re: crop_rec_4k corrupted mlv's
« on: June 22, 2017, 06:48:23 PM »
works reliable so all fine

Code: [Select]
    for (int i = 0; i < 1000; i++)
        char name[10];
        snprintf(name, sizeof(name), "%03d.txt", i);
        printf("%s\n", name);
        FILE * f = FIO_CreateFile(name);
        uint32_t buf = i + '0';
        FIO_WriteFile(f, &buf, sizeof(buf));

This creates 1000 files, all of them with correct contents (tested a couple of times), although the usage of FIO_WriteFile - without the wrappers from the PR - is incorrect. So, even if we ran some simple file integrity test 1000 times, on a known-to-be-incorrect program, there is a chance that our tests won't reveal any issues, and then the same code may fail randomly on other cameras (with unknown probability).

General Help Q&A / Re: Dual ISO (am I doing it right?)
« on: June 22, 2017, 06:15:29 PM »
Try to find a subject that requires even more shadow pushing (something that has a lot of noise at ISO 100, but looks fine at 1600).

Otherwise, if the clipped highlights are not very important, it's probably better to use a single ISO.

General Help Q&A / Re: crop_rec_4k corrupted mlv's
« on: June 22, 2017, 06:08:21 PM »
I'd consider the old one as the same bug; however, its exact side effects depend on many factors (in particular, how things are aligned in memory being one important factor). That probably explains why, with the build and the combination of settings you used, the probability of getting an invalid MLV (with the same type of error) was pretty high (about 10% from your report), while nobody else reported the exact same issue (so, either the issue occurred in places that did not affect the MLV structure, or, if it was, either it wasn't reported - likely if it happened once in 1000 clips or so, or, if it was reported, we had no idea what it could have been).

These "random" errors are usually very hard to track down, but in your case, the only hypothesis that made sense (how it could happen) was that CPU was aware of the size of the NULL block (and computed it correctly), but for some reason, what was written to card (which is performed using DMA from physical memory) was wrong. Therefore, there must have been some sort of mismatch between what CPU thinks about the physical memory, and what the memory actually contains (and that's likely because the has only written some changes - the contents of that NULL block - into its own cache, but these changes did not reach the physical memory by the time the NULL block was written to card).

I knew about this quirk, and most of the code doing file I/O uses memory buffers with caching disabled (fio_malloc), but since this wasn't enforced, these calls (those for writing the MLV headers, which were used incorrectly for many years) didn't ring a bell. Now, the PR either enforces correct usage (when reading files) or just ensures everything is written to main memory before writing files (so there will be no more restrictions about what memory addresses can be used for writing).

And yes, similar issues can be in other parts of the source code (although I didn't find any by looking at the code), or in code written by new developers, who may not be aware of this quirk (as there were no warnings, and so far, on the other platforms we were familiar with, this was a non-issue - afaik, the cache coherence issues, if any, are handled by the operating system). If there are such problems in other areas, after the fix, you'll get an error text all over the screen.

Camera-specific discussion / Re: Canon 6D
« on: June 22, 2017, 12:58:40 PM »
Can you help me where can i find and install the Help files for ML ?

That's something we need to fix (help welcome), but the info you need should be in the main menu (don't press anything, just read it).

I don't have anything in the Help Menu.

Even though I don't know how exactly this menu looks on 6D (as the emulator is not there yet), I have a feeling it's not completely empty.

General Help Q&A / Re: Real-time exposure monitoring: is it possible?
« on: June 22, 2017, 12:38:19 PM »
Also answered here:

In a nutshell: it's not possible to peek into an in-progress image capture, but it is possible to do a long exposure by stacking shorter exposures. The result will be different (better in highlights, worse in shadows).

BTW, ML has tools to evaluate a long exposure before taking it (such as extended exposure simulation in LiveView, via Expo Override, or Auto ETTR compatible with Bulb mode).

Camera-specific discussion / Re: ML on EOS-M2
« on: June 21, 2017, 09:35:37 PM »
However, the MD5 values from the "original" ROM1 and the new QEMU dump don't match. Is this expected?

"boot=1" works by patching the ROM. If you do the same change in a hex editor, you'll get the second MD5.

Reverse Engineering / Re: RE contributions
« on: June 21, 2017, 10:10:08 AM »
My attempt at parsing properties: prop_diag from the recovery branch. This is currently included in the portable display test, but the same file can be compiled as a standalone program:

Code: [Select]
hg up recovery -C
cd src
gcc prop_diag.c -o prop_diag

mv prop_diag /path/to/rom/dumps
cd /path/to/rom/dumps
./prop_diag ROM1.BIN

Camera-specific discussion / Re: ML on EOS-M2
« on: June 21, 2017, 10:06:33 AM »
That comes from the way properties are stored in memory: there are many setting blocks, and only one of them is active. The property parser from the recovery branch is was not very smart (it doesn't look at whether a block is active or not - it just parses everything). So, it takes the values from the last block in memory that has them (not necessarily the active one).

It probably takes a couple of restarts after the firmware upgrade to replace all the old blocks with 1.0.3.

Updated prop_diag to only parse the active blocks (recovery branch). You can build the display test from that branch, from platform/portable.000 (comment out the ROM dumping routine and cache disabling tricks and that's the plain display test).

Tip: compiling the same branch from some camera directory will give additional capabilities (such as LED blinking), but the binary will no longer be portable.

I would love suggestions for correcting the overblown highlights

Does it help if you reduce saturation in the picture style?

Do you have a CR2 image from the same scene? (just curious to see how various raw processors handle this case)

If you want to use Canon's color science (since you use HDMI out, you have to), having a CR2 of the tricky scene means you'll be able to experiment with Canon's raw processing software (DPP) and see what parameters will render it properly.

in a live-event setting like this.

I'm not sure what this implies. Should the adjustment be done automatically, in real-time? Should the saturation go back to the initial level once the highly saturated colors disappear from the scene? If yes, some script that looks at RGB zebras and adjusts saturation (maybe also exposure and white balance) might help, but a general solution is probably difficult (at least for my limited knowledge).

Scripting Corner / Re: Eclipse exposure sequence generator.
« on: June 21, 2017, 01:16:37 AM »
Very cool - I've seen it earlier, it was linked from this thread, also eclipse-related.

My first lua script, although I have been hacking in various languages for years.

I actually like this coding style very much.

The script spends a lot of time waiting for the appropriate time to take the next image, and these two functions should (I think) pause until the target time arrives, and exit as close as possible to the beginning of the target clock time second.  Any suggested improvements?

Looks fine to me.

Currently, this event handler is done by sleeping for 200ms and checking the internal microsecond timer (which is different from RTC and they are not in sync, to my knowledge). So, there is probably a bit of jitter.

For a more precise sync to integer seconds, you could poll the RTC ( with a tighter interval. For syncing to shorter intervals, you could use dryos.ms_clock, which is derived from the internal microsecond timer (we call it "digic timer"), but that one will probably drift from the RTC a little (didn't test).

Also, since multitasking is quite tricky in our Lua implementation (and that's because of my very limited understanding of Lua internals), I prefer to keep the scripts single-threaded. From my limited tests (read: api_test.lua), event handlers appear to be working fine though.

This is takes a bracketed exposure, with $bkcount brackets, and an f-stop change of $bktstep per bracket.  Is there a more sensible way to do this?

An alternative would be to enable Advanced Bracket menu (with the menu API from Lua) and then take the first picture. Not sure if it would be any better; didn't try.

If you want to take a burst of full-size images, and you have (or can borrow) a 5D3, my suggestion would be the 4K branch, as it can also record full-res frames (with rolling shutter) at about 4-5 FPS continuously, without shutter actuation. Otherwise, if you are OK with a 1:1 crop, the 48/50/60 fps video modes might also be useful. Per-frame bracketing in LiveView is also doable (just not implemented, but the backend support is there - see e.g. HDR video). Or maybe this.

BTW, noticed this in the script:
-- I have tried replacing this with multiple calls to "camera.shoot()", but it is still considerably
-- slower than "camera.burst()", and can also crash the camera

Can you prepare a minimal example that crashes the camera?

And I also have a question: how do you know what MinShutterSpeed and MaxShutterSpeed will give a reasonably good exposure during the eclipse?

Side note: the Intervalometer Ramping module is probably capable of something similar (by ramping interval time according to real-time clock, and by enabling Advanced Bracketing with auto exposure range). Have you considered it?

General Help Q&A / Re: crop_rec_4k corrupted mlv's
« on: June 21, 2017, 12:09:39 AM »
Fix prepared:

This bug was present for many years (probably since the first versions of mlv_rec, and present in all the subsequent implementations - including mlv_lite and silent picture module). The side effect of this bug was some incorrect bytes in the MLV header, appearing with very low probability (you'd have to record many clips to get one such error). If these incorrect bytes were on metadata like exposure time or camera serial number or something like that, it's easy to see why the bug went unnoticed for so much time.

However, once the the erroneous byte appeared in the size field of a block (which is required to be correct in order to parse the MLV), invalid MLVs were written.

Side note: I had a pretty hard time creating a synthetic test that would reproduce this bug consistently - at first, I've even created 1000 files and checked all of them, and could not get a single erroneous byte.

After fiddling with it all day long, g3gg0 gave me a very good hint - the source buffer (that was going to be written to card) must be at different memory locations (different stack offsets) during the test, as the bug depends on where this buffer gets loaded into cache. Randomized the buffer location and suddenly got lots of errors, in other words, a consistent way to reproduce this bug. More details in the PR.

Camera-specific discussion / Re: ML on EOS-M2
« on: June 20, 2017, 04:25:13 PM »
The second screen should be as easy as specifying "boot=1".

Reason: the ROM dumped from EOS M2 does not have the boot flag enabled, while the ROM dumped from EOS M probably has it.

At this point, we have everything we need to port ML and debug it in the emulator. At least, things like Hello World or ML menu navigation will work without requiring additional changes to QEMU.

Reverse Engineering / Re: EDMAC internals
« on: June 20, 2017, 04:19:03 PM »
Committed the test code that outputs the raw logs used to figure out the EDMAC model, just in case anyone would like to play with it.

This code can be used for cross-checking the EDMAC behavior with our understanding on what it does, by running it on both a real camera and on QEMU - the logs should match.

PR ready.

Camera-specific discussion / Re: ML on EOS-M2
« on: June 19, 2017, 05:11:16 PM »
All OK so far - just keep following the walkthrough.

The terminal reset is intended, as it allows you to scroll to the beginning of the log. Would be nice to print the command line after clearing the screen.

Camera-specific discussion / Re: ML on EOS-M2
« on: June 19, 2017, 09:43:34 AM »
The serial flash activity looks alright. The issue is at MPU messages.

Simply adding files to the repository will not change anything - they must be included somehow in the build process. For example, if it's a .c file, generally you'll need to include the corresponding object file into the executable (example). If it's a .h file, you'll need to #include it somewhere. And once you have done that, you usually have to reference the new stuff somehow (call some functions or use some data structures from the new files), unless these files are used to replace or override some older definitions. That's generally how it works in a C program.

QEMU does something interesting here - any function or variable must be used somehow (unless it's a global API - in this case it requires a prototype - or an inline function), otherwise you get a compile error. So, if you define button_codes_EOSM2[], but you don't reference it from anywhere, you should get this:
Code: [Select]
error: ‘button_codes_EOSM2’ defined but not used [-Werror=unused-variable]
 static int button_codes_EOSM2[] = {

In this case, you don't even have to create new files or new definitions. Simply look in mpu.c to see how it's done on other models.

Camera-specific discussion / Re: ML on EOS-M2
« on: June 19, 2017, 12:10:34 AM »
Changing '|&' to '2>&1 |' still didn't give me the same results a1ex posted.

These warnings should only appear if .current_task_addr is not defined. There are lots of them, on all models.

Binary file (standard input) matches

Maybe grep gets confused by ANSI color codes. Adding ansi2txt to the pipeline, or using grep --text, may help.

Reverse Engineering / Re: UHS-I / SD cards investigation
« on: June 18, 2017, 11:54:16 PM »
Revisiting this.

At the suggestion of Ant123, I've ran ML under the Xilinx version of QEMU, which also emulates UHS. This revealed the missing bits from my previous patch: on 5D3, although it was printing a message about 96MHz, the card interface was set up in the same way as for... 24 MHz (which explains the previous benchmark results).

Took the missing register configurations from 700D and...


Now the details.

Xilinx QEMU includes UHS emulation (along with some other nice stuff) and is based on QEMU 2.6.x (at the time of writing). Currently we have patches for QEMU 2.5.0 and 2.9.0, and for the Xilinx version, it will be something in-between. If there is interest, I can commit the patches.

First, I've ran 5D3 1.2.3 from the dm-spy-experiments branch on both the camera and QEMU 2.5.0. Result.

The problem:
Code: [Select]
CSMgrTask:ff6bdb7c:23:05: Set Hi-Speed Mode( 48MHz )                ; real hardware
[   CSMgrTask:ff6bdb9c ] (23:05) Set Normal-Speed Mode( 24MHz )     ; QEMU 2.5.0

That's where the Xilinx QEMU comes in. Result: after a minor patch to the SCR structure (first field - SpecVersion - changed from 0 to 1), both the log from camera and Xilinx QEMU now have:
Code: [Select]
CSMgrTask:ff6bdb44:23:05: SD_GetAccessMode=3
CSMgrTask:ff6bdb7c:23:05: Set Hi-Speed Mode( 48MHz )

Also, besides a minor difference in handling CMD1, and the card capacity, the emulation matches the reality pretty well. There's even some nice debug info if you uncomment DEBUG_SD in hw/sd/sd.c and use the "-d sd" switch on the command line. Result - in particular, these lines:
Code: [Select]
CSMgrTask:ff6b8fec:23:01: sdSetFunction( 16776961 ) Start
CSMgrTask:000aea50:00:00: *** sd_setup_mode(0x2), from ff484738
sd: CMD16 0x00000040 state 4
sd: Response: 00 00 09 00 state 4
CSMgrTask:000aea50:00:00: *** sd_setup_mode(0x2), from ff6b8300
sd: CMD6 0x80ffff01 state 4
sd: Function default selected (fn grp 2)
sd: Function high-speed/SDR25 selected (fn grp 1)
sd: Response: 00 00 09 00 state 5
sd: CMD13 0x45670000 state 4
sd: Response: 00 00 09 00 state 4
sd: CMD16 0x00000200 state 4
sd: Response: 00 00 09 00 state 4
CSMgrTask:ff6b91c8:23:01: sdSetFunction( 16776961 ) End

Now we can analyze how the SD initialization code configures the hardware (MMIO registers). This is as easy as specifying "-d io" in QEMU's command-line. Result.

At this point I've applied the old patch and tried to figure out whether there's any obvious trouble. Result:

Without hack:
Code: [Select]
CSMgrTask:ff6bdb7c:23:05: Set Hi-Speed Mode( 48MHz )
CSMgrTask:000aeed0:00:00: *** sd_set_mode(0x1, 0x3), from ff6bdb88
(some registers)
CSMgrTask:000aeed0:00:00: *** sd_setup_mode(0x3), from ff484738
(some more registers)
CSMgrTask:ff6bc87c:23:01: sdReadBlk: st=0, num=1, buf=0x40983808

With hack:
Code: [Select]
CSMgrTask:ff6bdb5c:23:05: Set Hi-Speed Mode( 96MHz )
CSMgrTask:000aeef0:00:00: *** sd_set_mode(0x1, 0x4), from ff6bdb88
(some registers)
CSMgrTask:000aeef0:00:00: *** sd_setup_mode(0x4), from ff484738
(a few more registers, but many of them missing!)
CSMgrTask:ff6bc87c:23:01: sdReadBlk: st=0, num=1, buf=0x409837bc

In other words, sd_setup_mode(4) appears to skip some hardware configuration it might be supposed to perform. With other arguments, it sets a bunch of registers, and the argument appears to be related to SD clock speed (3 = 48MHz, 2 = 24MHz, 4 = 96MHz).

Let's see what it does on 700D. Result (also with a zoomed-in view).

Notice some additional registers on 700D (highlighted in red on the zoomed-in view). What's up with them?

The 0xC04004xx range is never set on 5D3, so these registers are probably specific to 700D hardware. Same for 0xC040063x and 0xC040064x. I didn't touch them.

The remaining registers, 0xC04006[012]x, are also set on 5D3, at other speed modes. In these other modes, their values are the same on 700D. You can see them here: sd_setup_mode(2), sd_setup_mode(4).

My hypothesis was that 5D3's SD controller is UHS-capable, but for some unknown reason (could be even problems during the initial tests), Canon decided not to include it in the firmware. As a result, some of the UHS initialization code (hopefully a small part) was optimized out.

So I've tried to take the missing register configurations from 700D.

Therefore, the patch for 5D3 1.2.3 becomes:
Code: [Select]
/* in dm-spy.c, right before dm_spy_extra_install() */
patch_instruction(0xff48446c, 0xe3a00000, 0xe3a00001, "SD 1.8V");

/* in dm-spy-extra.c */
static void sd_setup_mode_log(uint32_t* regs, uint32_t* stack, uint32_t pc)
    /* log the original call as usual */
    generic_log(regs, stack, pc);

    if (regs[0] == 4)
        MEM(0xC0400600) = 3;
        MEM(0xC0400610) = 4;
        MEM(0xC0400614) = 0x1D000301;
        MEM(0xC0400618) = 0;
        MEM(0xC0400624) = 0x201;
        MEM(0xC0400628) = 0x201;
        MEM(0xC040061C) = 0x100;
        MEM(0xC0400620) = 4;
        MEM(0xC0400604) = 3;

    /* under CONFIG_5D3_123 */
    { 0xFF4844A0, "sd_setup_mode", 1, sd_setup_mode_log },

If you decide to try it, make sure you don't have any important data on your card. Otherwise, you will be playing Russian Roulette with your data (just like with my other SD patch).

Does this apply to DIGIC 4 cameras?

I'm afraid not - the hardware configuration of these cameras is different (and a lot simpler). You now know where to look, so you can play with it, attempt to change the clock speed and report your findings.

Can the clock speed be pushed even further?

I have no idea. Feel free to play with these registers, run the benchmarks and report.

Can this be included in a module, to be used on a regular ML build?

That's hard, because the hack must be applied before the SD card gets initialized by Canon firmware (in other words, before loading any module, and also before loading the config file). So, even if we include it in ML core, it will be hard to create an option for it.

At the moment, the easiest way to try it would be a custom build. Probably best to start from the crop_rec_4k branch, as the backend support is there, and there is little reason to try this hack outside that branch.

It might be possible to switch the SD to a higher speed on the fly. Didn't investigate this approach.

How's this useful in practice?

Other than raw video recording on both CF and SD at the same time (aka card spanning), it's probably not very useful.

What's the maximum total speed (CF+SD)?

Load the benchmarks module ( to find out.

How can I get similar results in QEMU?

Take a look at this post. In a nutshell, it's the dm-spy-experiments branch compiled with CONFIG_DEBUG_INTERCEPT_STARTUP=y for the camera, and additionally with CONFIG_QEMU=y for running it under the emulator. These two will give logs that can be directly compared, and with the logging options from QEMU, you can get additional details. Then, look for CSMgrTask in the logs, compare them and try to understand what it does. Also refer to SD docs (summarized nicely by nikfreak earlier in this thread) to understand the initialization protocol. That's it. If you get stuck, just ask (here or on IRC).

If there is interest, I can commit the patch for Xilinx QEMU and write a walkthrough similar to this one.

Please note I no longer have the UHS card to do more tests (it wasn't mine), so from now on, what will happen with this hack is entirely up to you.

when I was shooting Lens Cap on for the DF Averaging, first frame it crashed saying Compressed size was bigger than uncompressed. A completely black picture on ISO 100 and it could not compress it.

That was a corrupted frame.

I have never tried 10 bit mode, but what I see from all the examples it is extremely noisy, perhaps that is why your recording times are so low

I've only used it in order to get constant bitrate. With lossless compression, the recording times will vary a lot with scene contents (so it's harder to benchmark this way; a static scene or a dark frame will make things a bit more repeatable in this case, but I didn't want the extra trouble.)

maybe the compression goes crazy from all the colour casts 10 bit footage introduces.

I thought I've mentioned "uncompressed"...

But yeah.. 10 bit vs 14 bit, mathematically you should have higher record times than I am getting. no?


10 < 14
10 > 14 * 60%

And why would the difference be lower on 113? I thought fir 113 always had higher performance than 123 because of the extra buffer.

The cause is lower CPU usage on both Canon's side (regardless of ML settings), and on ML's side (only when Luma-based zebras are used). Memory bandwidth usage might also be a bit higher on 1.2.3 on Canon's side (didn't check this one, as it's not straightforward to measure). Of course, absolute performance can be higher both with and without global draw, but the difference between these numbers is likely lower (that is, global draw even faster on 1.1.3) iff you use Luma-based zebras (which is the default). I didn't benchmark these cases; it's just what I'd expect to happen.

Pages: [1] 2 3 ... 335