Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.


Messages - a1ex

Pages: [1] 2 3 ... 357
1
BTW could not set 4096x1440 @ 25p (1:1 crop) mode. How?

The preset is named "4K 1:1 half-FPS". See also the first post.

2
General Development Discussion / Re: Canon EOS 1300D / Rebel T6
« on: Yesterday at 03:48:38 PM »
I had no luck with memory_region_rom_device_set_romd (the read callback is not called, only the write one), but I think I've found a cleaner workaround:

Code: [Select]
    void * rom_ops_arg = (void *)((uintptr_t) s | rom_id);
    memory_region_init_rom_device(rom, NULL, &rom_ops, rom_ops_arg, name, rom_size, &error_abort);

Then, just hardcode our magic numbers (model ID or whatever that is) in the first 3 bytes of ROM1. That appears to do the trick.

It doesn't log ROM reads though (which is something I wanted on all models, regardless of how the 1300D port will turn out).

Will update the patch later.

3
General Development Discussion / Re: Canon EOS 1300D / Rebel T6
« on: Yesterday at 03:05:30 PM »
Some parts of the ROM are copied to RAM - see reply #14.

You can get the RAM contents either with dd (after identifying what is copied where), or from either QEMU or GDB (they both have commands for dumping the RAM). Or, you can disassemble directly from GDB or from the QEMU monitor console.

You'll need a RAM_OFFSET in the stubs file, similar to DIGIC 5 models. It's explained in the tutorial for finding stubs.

These are helpful:
https://sourceware.org/gdb/onlinedocs/gdb/Machine-Code.html
https://en.wikibooks.org/wiki/QEMU/Monitor

4
General Development Discussion / Re: Canon EOS 1300D / Rebel T6
« on: Yesterday at 12:43:38 PM »
Thanks for the help (read: doing 99.99995% of the work)

Well, that was because the first assert was not something I'd expect new contributors to be able to figure out (as it was not present on any other model, and requires a very good understanding of the ROM layout - which I don't have yet). This doesn't usually happen with things already documented or mentioned elsewhere.

And I also happened to have a few days off :D

Edit: looks like the proper way to implement a ROM in QEMU is by using memory_region_init_rom_device. However, that one appears to handle only writes with callbacks. Go figure...

Edit2: looks like memory_region_rom_device_set_romd might do exactly what we are looking for :D

5
The difficult part is simply understanding what Canon code does (and what we have to change to fix the preview).

Those numbers from the logs and adtg_gui are like Chinese to all of us; the trick is to fiddle with them until a tiny part of them starts to make some sort of sense. That's how we figured out how to reconfigure the sensor to 4K in the first place ;)

Some useful tricks: comparing all those numbers in different video modes (adtg_gui has a simple register diff tool), recognizing values that could look like resolutions, logging the I/O activity in QEMU, or analyzing the still photo mode first (it's simpler).

And, of course, don't give up after reading a single post. Make yourself comfortable with what's happening in the Reverse Engineering area. Some relevant threads are linked in the first post; make sure you can follow them.

Of course, you will need some basic programming knowledge.

Bottom line: the real-time preview is not implemented because... I have little or no idea about how Canon configures the image processing path.

6
A dumb question maybe... but... someone are getting realtime preview in this mode? How can you focus and framing for sports and birds?... I think I'm missing something...

Want to help? Grab ADTG_GUI and QEMU and start investigating!

All you need to do is to understand how Canon configures the image processing path for LiveView. Here's a starting point.

7
Added a couple of jobs on the build server:

- QEMU-dm-spy: compiles the dm-spy-experiments branch and runs the binary in the emulator. These logs contain all Canon's debug messages, and optionally all MMIO activity. Should be useful for anyone who wants to understand the startup process.
- QEMU-boot-check: compiles ML from every camera model (from nightly) with CONFIG_QEMU=y and runs it for a few seconds in the emulator; this compilation flag enables additional debug info at startup, useful for checking the boot process (where autoexec.bin is loaded, how much memory it takes, what it does to reserve it and so on).
- QEMU-FA_CaptureTestImage: compiles a minimal autoexec.bin that calls FA_CaptureTestImage (therefore taking a full-res silent picture). All the debug messages from Canon and all the MMIO activity are logged. Might be useful for understanding the still photo capture process.
- QEMU-tests: that's the test suite for QEMU (presented earlier in this thread)

All these tests have HTML logs (actually just plain text with colors) and screenshots (where it's the case).

I'm also thinking to run some basic tests on the nightly, on those models with functional GUI (tests such as menu screenshots, load each module, check memory usage, run some simple Lua scripts). The emulation is not there yet for more complex tests (for example, we cannot take a CR2 picture or go to LiveView).

8
Oh I meant to ask this, should we be using 123 vs 113 for 5d3? It was my understanding that 113 was fine unless you need hdmi features and AF @ f8. Is this still true?

Correct.

I'm currently using 1.2.3, but that's because it has more quirks (usually, if something works on 1.2.3, it's likely OK on 1.1.3, but in the other direction, there are a lot of exceptions...)

The notes from 1.1.3 thread are still valid.

AFAIK 113/123 should be identical ever since @a1ex merged these 'two firmwares under one branch' though

That means both ML versions are compiled from the same codebase (but each one is still tweaked to account for Canon firmware quirks - just like all other ML ports).

MLVDump does not fix them [vertical lines].  Is there another app that i'm not aware of that does?

I've explained here what needs to be done for other converters (including mlv_dump). I'm not really familiar with compressed formats, and there are people in the ML community who can do this faster than me (hint, hint).

9
General Development Discussion / Re: Canon EOS 1300D / Rebel T6
« on: April 24, 2017, 06:20:27 PM »
Yes, it does (it's a side effect of my modification - one of the reasons I'm not going to commit it in this state). When setting the boot flag, MEM_WRITE_ROM writes to the first copy of the ROM (the one modelled as I/O), and the write is currently ignored.

As a workaround, try writing the bootflag in another copy of the ROM (there are a bunch of mirrored ones - any of them will update all the others). For example, .bootflags_addr = 0xFA000000 (not tested). What I've tested was changing MEM_WRITE_ROM to write at addr+ROM0_SIZE, but that's way too hackish.

Probably it's best to handle it in the ROM write handler.

10
Reverse Engineering / Re: ProcessTwoInTwoOutLosslessPath
« on: April 24, 2017, 03:23:10 PM »
There is a closed-source tool that can decode them, to my knowledge.

The decompressor is based on Baldand's LJ92 codec (from MlRawViewer), but it needs some changes. Some of these changes were shared here, so I assume one should be able to integrate them in other converters.

So, all that's required is some manpower to do the integration. No reverse engineering is required.

11
General Development Discussion / Re: Canon EOS 1300D / Rebel T6
« on: April 24, 2017, 03:12:36 PM »
Right, 60D has an exception exactly for this constant :D

Other cameras have dummy definitions (those 0x123456), so anything that checks if the current GUI mode is GUIMODE_FOCUS_MODE will be false.

In general, if you have doubts about a constant, grep the source code to see how it's used. Some of them are used as memory locations where things are written - these need additional care, as the camera bricking does happen (should be recoverable in most cases, but it's best not to get there). This should help understanding why this happens - although the only 100% sure way to prevent bricking is... executing it only in QEMU.

12
General Development Discussion / Re: Canon EOS 1300D / Rebel T6
« on: April 24, 2017, 02:37:34 PM »
Besides nikfreak's advice, here are some useful tips:

Example for compiling without features: 60D-dm-spy.patch

(I know, they are not isolated very well, as we don't turn them off very often...)

You can also use the minimal target, but that one is really minimal (useful for a lower-level version of Hello World). It uses the platform-specific files (stubs, consts) from the platform directory, a single source file for experiments (minimal.c) and a tiny graphics library (font_direct.c) - besides the loader code in reboot.c. Therefore, it's a good playground environment that does not touch the larger codebase (and does not require a lot of stubs/consts to get started).

The digic6-dumper branch also makes use of the minimal target, but a different way (using a platform-specific minimal.c - because the current boot process has to be changed significantly for newer models). I hope it's not needed for 1300D.

13
Reverse Engineering / Re: ProcessTwoInTwoOutLosslessPath
« on: April 24, 2017, 10:28:06 AM »
At the time of writing, it only applied to 5D3, from the models I've looked at.

Our logging routines are not printing [ENGIO], so the reason for larger log might have been a few more seconds allocated for the test routine. The new log doesn't appear to have any new information though.

EngDrvOut and engio_write can be taken from stubs; the others are easy to identify from strings (as they only log a debug message in case of error). Most DIGIC 4 models also log the register and the data written to it, when the call is successful.

14
General Development Discussion / Re: Canon EOS 1300D / Rebel T6
« on: April 24, 2017, 09:42:23 AM »
Modelling the ROM as IO has just gone over my head complete, so assuming I cant figure that out (yet, im learning) I might continue trying to figure out what might be being mishandled to result in the Assert call.

Yes, that was pretty difficult, as this one requires detailed knowledge of how ROM is configured, and how to do that in QEMU. I think I've figured it out last night, but had to change the ROM layout on all models. This approach appears to break other functionality (e.g. 60D no longer boots), but also gives interesting insights on how ROM reflashing is done (and allows one to implement its emulation, since the ROM addresses appear to behave like I/O during this process):

qemu-1300D.patch

A simpler approach would have been to patch the ROM manually (hardcoding the flash model ID at those addresses where the firmware expects it). Unfortunately, that appears to lock up the bootloader.

A third approach would be to patch the affected function in GDB (see e.g. 700D - patches.gdb) or in ROM (see DIGIC 6 models, but that would remove the ability to run unmodified autoexec.bin's later, since they do a checksum of the ROM at startup to ensure correct firmware version).

Anyway - currently it starts a few tasks, so you can apply the patch and start identifying stubs. Some of them are useful for debugmsg.gdb as well (e.g. DebugMsg, task_create). The current state is also enough for testing the boot process (to see whether ML is able to run code alongside the main Canon firmware, reserve memory for itself, start a task and so on). It won't display any GUI yet, but it shouldn't be hard to reach the Hello World stage without this functionality.

15
Reverse Engineering / Re: ProcessTwoInTwoOutLosslessPath
« on: April 24, 2017, 12:49:53 AM »
Looks like the 700D has the same problem as 5D3 - engio_write and others are silent. Can you include them as well? (they are in the same guide)

16
General Development Discussion / Re: Canon EOS 1300D / Rebel T6
« on: April 23, 2017, 08:01:59 PM »
My hypothesis is that it might be trying to get some sort of manufacturer ID of the flash ROM chip.

See for example the K8P2815UQB datasheet (used in 7D, according to this page). Here's the I/O and ROM activity for the 7D, when trying to change the boot flag from the FROMUTILITY menu (Serial Console in QEMU window):

Code: [Select]
Is flg written(Y=ON(0xFFFFFFFF)/N=OFF(0x00000000))? :y
[FlashIF]  at 0x00102164:001021B8 [0xC0000000] -> 0x0       : ???
[FlashIF]  at 0x0010216C:001021B8 [0xC0000000] <- 0x1000000 : ???
[FlashIF]  at 0x00102178:001021B8 [0xC0000010] <- 0xD9C50000: 'Write enable' enabled
ROM(0xf8000aaa) = 0xaa (ignored)
ROM(0xf8000554) = 0x55 (ignored)
ROM(0xf8000aaa) = 0x80 (ignored)
ROM(0xf8000aaa) = 0xaa (ignored)
ROM(0xf8000554) = 0x55 (ignored)
ROM(0xf8000000) = 0x30 (ignored)
ROM(0xf8000000) => 0x0
ROM(0xf8000000) => 0x0
... (infinite loop)

This looks similar (but not identical) to the Block Erase sequence. Probably the chip is some related model (not exactly the one listed on the wiki page).

On 1300D, the following ROM accesses are made since "K404 READY":
Code: [Select]
K404 READY
ROM(0xf9000000) => 0x0
ROM(0xf8000000) => 0x0
ROM(0xf8000000) <= 0x6 (ignored)
ROM(0xf9000000) => 0x0
ROM(0xf8000000) => 0x0
ROM(0xf9000000) => 0x0
ROM(0xf8000000) => 0x0
ROM(0xf9000001) => 0x0
ROM(0xf8000001) => 0x0
ROM(0xf9000002) => 0x0
ROM(0xf8000002) => 0x0
ROM(0xf9000000) => 0x0
ROM(0xf8000000) => 0x0
ROM(0xf9000000) => 0x0
ROM(0xf8000000) => 0x0
ROM(0xf9000000) => 0x0
ROM(0xf8000000) => 0x0

Assert: File ./Startup/Startup.c Line 220

So, my best guess is that we should model this copy of the ROM as I/O memory and fake the data somehow.

Note: in QEMU, it's generally not possible to log every single memory access, unless that memory block is configured as I/O. However, memory implemented as I/O cannot contain executable code (so we have to choose one).

Side note: I'm currently looking at Panda (a fork of QEMU), which promises the ability to log any memory access, and a lot more useful analyses (look at plugins in their manual, for example).

https://github.com/moyix/panda/blob/master/docs/manual.md
http://moyix.blogspot.com/2013/09/announcing-panda-platform-for.html
https://gist.github.com/bridgeythegeek/d7a6c449287c6e32187be2639a7920bf

17
Reverse Engineering / Re: ProcessTwoInTwoOutLosslessPath
« on: April 23, 2017, 05:59:46 PM »
I did record a dm-spy log shooting a single CR2

Can you take the same log, with interrupts and engio messages enabled? They are both disabled by default (first in dm-spy-extra.c, second in dm-spy.c).

18
General Development Discussion / Re: Porting a Canon firmware update
« on: April 23, 2017, 11:15:34 AM »
Very nice write-up (and very easy to follow!)

Some minor correction:

Didn't happen on your first try? Some times it is stubborn so try it again or switch to another card. The point of this is to get a ROM dump of the new firmware. If you want to be meticulous about it you should run an MD5 check on your file.

Here, checking the MD5 is not about being meticulous; the bootloader file I/O routines *are* really stubborn. They may corrupt the filesystem, insert a few or a lot of wrong bytes, lock up the camera, refuse to run on any or all of your cards and so on.

So far, what worked every single time was formatting the card at a much lower capacity (I've used 256MB successfully in many cases). To do this, you can write the SD image that comes with QEMU to your SD or CF card. This guide is helpful; just don't forget to unzip the SD image. This image contains the portable display test and is bootable (so, you can test it in the camera straight away).

Even in this case, checking the MD5 is mandatory. On Linux, this is as simple as:
Code: [Select]
md5sum -c *.MD5

19
General Development Discussion / Re: Canon EOS 1300D / Rebel T6
« on: April 23, 2017, 10:36:53 AM »
Sounds like you are on the right track. The assert puzzle appears harder than I've expected - I've tried to debug it yesterday and here's what I found:

The tricky subroutine is called like this:
Code: [Select]
sub_27C4(0xF8000000, &out1, &out2, &out3);

0xF8000000 is the address used for boot flags. This suggests the 3 output values are probably read from there.

The next checks (before the assert) appear to accept the following values for out1-3: C2, 25, 39 or 20 BB 19 or 01 02 19 (hex). Each of these sequences is handled with a different subroutine: 2938 / 2B0C / 2CE4; they all allocate memory, fill in some round values and call FE2B486C (which is complicated).

Inside 27C4, there are a couple of functions that call others indirectly (BX R1); these are easy to get by running the debugger and finding the value of R1 (or PC after the call); understanding where this code takes these values from is a lot harder. Here's how to debug with GDB:

Code: [Select]
./run_canon_fw.sh 1300D -d exec,io -singlestep -s -S

then you need a debugmsg.gdb file:
Code: [Select]
source -v debug-logging.gdb

b *0xFE0C1B60
b *0x27C4

continue

then, in another terminal:
Code: [Select]
arm-none-eabi-gdb -x 1300D/debugmsg.gdb

then:
Code: [Select]
(gdb) layout asm
(gdb) layout regs
(gdb) si

If you want to jump over a function, gdb may complain ("Cannot find bounds of current function"); here's a workaround:
Code: [Select]
B+ │0x27ec  mov    r0, r6
   │0x27f0  bl     0x6e50
   │0x27f4  mov    r3, r8
Code: [Select]
(gdb) tbreak *0x27f4
(gdb) c

Now, our function returns 06 00 00 (instead of one of the accepted sequences). Where does that come from?!

20
Sorry to be a party pooper but can anyone else confirm that this is reproducible (it seems to be on my end w build 2017-04-22 for 5D3.123) if one were to record in 3504x1400 resolution (and only in that resolution which is 2.50:1)?

Narrowed down and solved; excellent catch @DeafEyeJedi!

That's how a proper bug report looks like - it contains all the details to allow others to reproduce it. This bug was quite hard to trigger, given all those reports with no corrupted frames.

Also got 4096x3072 working at 12 and 12.5 FPS (24/25 in Canon menu) and did some more fixes for very large resolutions. Even if you only get a few frames, I still find these modes useful with half-shutter trigger and pre-recording (e.g. for taking photos of really unpredictable subjects).

21
@DeafEyeJedi: does it help if you reduce the vertical resolution from the crop_rec submenu a bit? (e.g. instead of 1536, try 1500, 1450, 1400)

If you can reproduce it consistently without the resolution override, try the following patches

edit: nevermind; had some hypotheses, but rejected all of them.

Reproduced.

22
What's the difference between those two slowmo modes [slowmo 45p / 48p and 50p / 60p] other than fps because on the 50p / 60p mode I was able to record @ 48p?

FPS and vertical resolution. The 48p mode is really pushed to the limit (already discussed), while the 45p should have a little more headroom.

The cameras shutter speed was 1/50 but when zoomed + fps override it showed 1/27 or 1/37 or something like that.

Solved, it was just an oversight from my side.

Also fine-tuned the preview behavior: you can use a long half-shutter press to change between framing and real-time (e.g. if you normally use the centered real-time preview in x5 zoom, but you also want to check the framing every now and then, just press half-shutter for 1 second or so).

23
General Development Discussion / Re: Experiment - Dynamic My Menu
« on: April 22, 2017, 09:04:34 PM »
Recently I had to change often a setting that was normally hidden under Advanced, in a submenu (that was mlv_lite -> raw video -> preview). This required a bit too many clicks for my taste. Of course, it's easy to move this particular setting back to main menu, but I was thinking at something more generic.

Since we already have usage counters, a submenu with more than say 8 items could auto-hide the less frequently used ones, overriding my default choice of simple/advanced options.

Is this something worth looking into?

24
General Development Discussion / Re: Canon EOS 1300D / Rebel T6
« on: April 22, 2017, 11:45:41 AM »
The second, which the GPIO handler has annotated as maybe being SD Detect for the 70D and 6D, proved more valuable.
Replacing the output of that overwrite with a 0 value skipped the SDAC and moved ahead. Whoopie!

Yep, that's the one.

Not sure what's causing the assert (didn't look much into it yet, other than noticing it depends on the output values given by sub_27C4, which is copied to RAM right before cstart - a process done on other DIGIC 5 and 6 cameras). The 1300D appears to have a few bits from the newer codebases backported on DIGIC 4.

Debugging in IDA may help:

Code: [Select]
./run_canon_fw.sh 1300D -d io -singlestep -s -S

followed by F9 in IDA.

Some useful functions:
Code: [Select]
FE0C0000 main firmware start
FE0C3A28 cstart
FE1279E8 init_task
FE0C1B60 AJ_massive_kernel_init

It also helps extracting the memory blocks copied to RAM and loading them in IDA at the copied location as additional binary files. The functions copied there will be executed from RAM.

Not sure how much it helps, but last night I've added other QEMU startup logs (from other camera models) here:
http://builds.magiclantern.fm/jenkins/view/QEMU/job/QEMU-dm-spy/

25
The shutter speed issue appears only when using the 5x preset (it appears fine without crop_rec); can you confirm?

Pages: [1] 2 3 ... 357
courtesy