Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Messages - a1ex

Pages: [1] 2 3 ... 342
Hardware and Accessories / Re: battery grip pins
« on: Today at 11:04:40 AM »
Got 2 pictures from a 600D:


These are likely the UART ports for the main CPU (ICU) and the MPU.

The first one should reveal the DryOS shell (aka DrySh or Dry-shell). There should be also some activity from the bootloader (e.g. "Now jump to AUTOEXEC.BIN").

No idea about the second one.

If anyone could hook up an Arduino/RPi/whatever to these pins and log the UART activity, it will be very appreciated.

General Chat / Re: Canon EOS 1000D problem buSY HELP!
« on: Today at 10:58:58 AM »
No, to my knowledge.

However, there is an unexplored path that could be helpful:

The UART pins for both the main CPU and the MPU are likely to be in the battery slot (at least for some models). If you can find some sort of UART activity on these pins, I'll look into it.

I hope to be able to check these pins on a 550D later in September; meanwhile, I have 2 pictures to share (will add them on the linked thread).

Reverse Engineering / Re: ProcessTwoInTwoOutLosslessPath
« on: Today at 09:28:43 AM »
Correct - ETTR also relies on the raw buffer, which was overwritten. Just pushed the fix (but didn't test this scenario).

Also went ahead and pulled ErwinH's and dfort's changes for 700D/EOSM and did some minor cleanups. Didn't pull the null pointer changes yet (want to test them in QEMU first).

Reverse Engineering / Re: ProcessTwoInTwoOutLosslessPath
« on: Yesterday at 11:21:02 PM »
Figured it out - I've been compressing the raw data in place (reusing the memory buffer); however, the overlays no longer had a valid raw buffer. Will disable that (though I might have to re-think the memory management).

Back then, I must have tested it with global draw off...

A bigger trouble I have is with full-res LiveView snapshots (crop_rec enabled and set to Full-res LV, silent pictures set to Simple). Regular (uncompressed) DNG works fine. Lossless DNG usually locks up the camera, though it's not 100% repeatable.

I might have missed the message because the LiveView screen was displaying the null pointer message which pretty much takes over the entire screen.

If that was on 5D3, I should take a look at your ROM.

Reverse Engineering / Re: ProcessTwoInTwoOutLosslessPath
« on: Yesterday at 09:35:22 PM »
Rather unnerving but I went ahead and shot some FRSP frames and lossless compression is working on that camera.

Really? I'm currently trying to take some full-res frames and I'm getting the above errors on 5D3.123...

(I've started from the current 4K experimental build, and went back to first known good build with lossless silent pictures - but that one isn't working either...)

Reverse Engineering / Re: ProcessTwoInTwoOutLosslessPath
« on: Yesterday at 07:29:29 PM »
The EOS M and 650D should be very similar. 100D uses the same pattern, only the code was refactored a bit. EOS M2 is likely similar to 100D. 70D and 6D are more like 5D3 (TwoInTwoOutLosslessPath). All these models are very good choices for low-hanging fruits.

BTW, 700D might also work with FF36B2D8 TTL_SetFlags. There are two versions of StartTwoInTwoOutJpegPath - one of them (the one I've looked at back then) has this function inline, the other one calls it.

Whatever these flags do, they also worked on 5D3 (with TTJ routines), so they are probably common to all DIGIC 5 models.

Interesting notice about doubling the width no longer needed; will look into it, but not right now.

General Development Discussion / Re: mlv_dump on steroids
« on: Yesterday at 07:01:52 PM »
Cold pixels used to be a problem before this PR; see also this analysis. I think most of them were actually pixels marked as bad by Canon firmware (they mark them as 0, so they are very easy to detect by our "cold pixel" heuristic). Some of them were not 0; they were only below the black level, although I don't remember the specific conditions one would get such artifact.

Need to double-check whether there are still cold pixels on other models (60D and 600D were troublesome, possibly also 550D and 500D, not sure about others). One has to compare a MLV (or a DNG developed with all processing turned off) from any recent build, vs any 2016 build.

Some of the "cold pixels" used to appear near very strong highlights (there were some bug reports about this for 5D3, with green artifacts: this one or this one or this one). These are dynamic and won't be fixed with default processing settings (they require scanning every frame for bad pixels, which is very slow: option --fixcp2).

General Chat / Re: Canon EOS 1000D problem buSY HELP!
« on: Yesterday at 12:10:13 PM »
The first requirement for unbricking: we should be able to run code on the camera. If you are not able to access Canon menu and run a firmware update, I'm afraid there's nothing we can do.

The second requirement: a ML port should be available for your camera. Currently, the 1000D runs a LED blinker and the portable display test, there are some VxWorks ports to start from, and some - not very complete - support in the emulator, so it's doable (at least a ROM dumper should be easy).

If you already have the boot flag enabled (very unlikely, unless you have run engelmarkus' BootDisk-Enabler in the past), you are lucky.

Reverse Engineering / Re: EDMAC internals
« on: August 17, 2017, 01:37:34 AM »
Something easier: playing back an image.

- the graph shows only the steps performed on the image processing engine
- first step: some memcpy of size 3840x1079, using connection <6> which is pass-through (probably zeroing out some buffer); note the input is a single line repeated many times.
- second step: a quick JPEG read pass (reading the embedded JPEG from the CR2 to find its metadata).
- third step: decoding the JPEG (it reads the same JPEG again, but this time outputs a 2880x960 YUV image in some unusual order); input from connection <5>, output on <3>, using JPCORE.
- fourth step: resizing the 2880x960 YUV to 1440x480 (all these sizes were in bytes, so the end result is 720x480 pixels - the displayed image). Input and output on connection <3>.

The last configuration probably shows the data coming to some connection is not automatically forwarded to the other end of that connection (exception: connections 6 and 7 will simply copy the input data to output).

A real-time connection diagram is available on the "edmac" branch, but it only shows the latest state (so, in this case of playing back an image, it will only show the last processing step, because the EDMAC channels are reused).

After some playing around, I've got a few snapshots throughout the image playback process (captured one screenshot on each StartEDmac call):

I should probably draw a diagram

Done - check the "edmac" branch and this topic.

General Development Discussion / Re: mlv_dump on steroids
« on: August 17, 2017, 01:09:31 AM »
Can you upload a dark frame for this one, so I can add it to the test suite?

For FPS, the low light mode will scale the shutter speed, while the other modes will not.

Unfortunately, only low light is compatible with high-res modes on 5D3 (at least for now). You should, however, be able to adjust the shutter speed after dialing the FPS.

Reverse Engineering / Re: ProcessTwoInTwoOutLosslessPath
« on: August 15, 2017, 08:15:19 PM »
Sorry for delay - was a bit away from ML lately, so anything that required more than a 5-minute break went to the back burner. Nice to see the progress - now the questions:

1) the null pointer warning appears to be false - both my 700D ROMs have that magic value. A sure way to tell would be to compare with a ROM from a 700D that never had ML on it - for example, the ROM backup saved during ML installation. If anyone made a backup copy of those ROMs (easily recognized by their boot flag status), they could be useful.

Detailed analysis: I get the magic value 0xA5A5A5A5 at the following offsets:
- 0xa5403c: that's in the middle of other magic values with similar patterns (e.g. 34 34 34 34 A5 A5 A5 A5 E5 E5 E5 E5 F1 F1 F1 F1). This pattern is also present in the firmware update from Canon.
- 0xc950e4, 0xc950e8, 0xc950ec, 0xc950f0. This is in the data area, not present in Canon's update and not recognized by my property parser either. Not sure what's up with it, but there are similar patterns nearby.
- 0xc95bd0, 0xc95bd4, 0xc95bd8, 0xc95bdc: same as above.
- 0xc9f1b0, 0xc9f1b4: like the first one, except it's not present in Canon's update.
- 0xfdb4b0: same as above.

2) Buffers - this was a bit of guesswork. On 5D3, I've assumed the raw buffer allocated by Canon is at least as big as the maximum resolution - the one from 5x zoom. With dm-spy-experiments I've found out a possible size for that buffer (commit 1ee9679 => 0x1cae000, so about 30 MB). Experimentally I've found out that was wrong - the buffer was overwritten by something else.

Another interesting discovery - as soon as leaving LiveView, a small part of Canon's default raw buffer (0x4d31a000) was overwritten near 0x4d600000 - about 2.9MB after its beginning. If we always stay in LiveView, things should be fine - however, we are using the "paused" LiveView for various things, so I wanted to play safe and pick a buffer that's not going to be overwritten when doing that. So I took whatever was not overwritten - found by trial and error between 0x4d600100 and 0x4ee00000: about 24 MB. Enough for 4K, but not for full-res (in 14-bit). For that (extreme) case, the only way to allocate a contiguous chunk of memory was with the SRM backend from Canon - they use it as raw buffer for still photo capture. It's very tricky to use - the buffers have to be freed in exactly the same order as allocated, there's a BUSY sign printed on the screen while this buffer is allocated, while this sign is on the screen it also locks up the UI as soon as you move a scrollwheel, and so on. For that reason, I went for something very hackish - intentional use after free for this buffer (assuming Canon code won't use or move the SRM buffers while in LiveView). It's not very robust, but it's only used with full-res LiveView (since that's the only case we need more than 25 MB for one frame).

To see the memory map, and find out what areas might be free, there is CONFIG_MARK_UNUSED_MEMORY_AT_STARTUP. Without it, the RAM is not cleared at boot, so you'll get different results (and likely not relevant) every time you boot.

Then, once you have guessed a possibly free memory region, you'll need to test it somehow to make sure nobody overwrites it (at least during LiveView). There is some test code for that in the RscMgr_memory branch, but it's nothing fancy - you could just fill the buffer with something and check whether it's still unchanged, in a loop.

RAW_LV_BUFFER_ALLOC_SIZE is how much we need for a full-res image (max W * max H * 14/8).

3) Regarding 07573f3 - that's a bit difficult to do in C (since the arrays have different lengths for different camera models). See for example mpu.c in the QEMU branch for a similar situation.

Tip: the ResLock entries do not depend on the firmware version (they probably depend only on the DIGIC generation), so you can use is_camera("5D3", "*") to cover all versions.

4) 600D LED blink means "out of memory". You could remove some features in features.h (you can even boot with an empty features.h, also comment out CONFIG_RAW* and related entries, e.g. EDMAC, and - on 600D - also remove audio-lapis.o from Makefile.setup.default). [ todo: allow compiling without features on all models, and include this on jenkins ]

Another todo for me: integrate dfort's 700D crop_rec in the 4K branch [DONE], and merge the non-4K changes (e.g. without memory hacks) into mainline (WIP in the raw_fixes branch).

Raw Video / Re: Issue with Liveview whilst using Full Res 7.4fps
« on: August 14, 2017, 05:21:58 PM »
Looks like that orange help text should be more visible (suggestions? larger font? different color?)

General Chat / Re: Histogram... bottom to top
« on: August 13, 2017, 11:12:34 PM »
I'm not aware of any software that would display a histogram in this way, but if you find it more readable than either linear or log, it could be interesting.

If you want the graph to be non-ambiguous, you could draw some tick marks that would show the Y scaling.

Here I've used some non-standard Y scaling for some plots (neither linear, nor log - it was derived from 1/x):

General Chat / Re: Histogram... bottom to top
« on: August 13, 2017, 07:32:40 PM »
Sometimes this peaks are so high, that you can't see the other colors.

That's where the log histogram comes in (on the Y axis). See p.7.

On the X axis, ML's raw histogram is always logarithmic (in EV). The YUV-based histogram (for JPG/H.264) is linear (0-255), but its input is a gamma-corrected image. See

I keep getting:

Program received signal SIGTRAP, Trace/breakpoint trap.
0xff13659c in ?? ()

on the 5D3 but it does the same thing on the Mac.

I don't have this issue on Ubuntu with gdb (gcc-arm-none-eabi-5_4-2016q2). The older gdb (gcc-arm-none-eabi-4_8-2013q4) works too (boots the GUI on 5D3), but some features from the GDB scripts (IIRC semaphores and message queues) require gdb newer than 7.7.

Tried gdb (required to run some models in QEMU, including 5D3). The one from Ubuntu repository doesn't work (it stops the emulation when it shouldn't). Tried (gcc 5.4, Linux installation tarball), but that didn't work either, as it has 32-bit binaries.

As I'm currently on a slow connection, I'll let somebody else figure it out (best guess: compile the 64-bit version from sources, or use the Windows install package). The command you need to get running is:
Code: [Select]
./ 5D3/firmware="boot=0" -s -S & arm-none-eabi-gdb -x 5D3/patches.gdb

or similar for any other camera model that has a patches.gdb file.

Camera-specific discussion / Re: Canon 750D
« on: August 10, 2017, 04:12:58 PM »
Is there a guide to follow step by step to help?

To quote from another firmware project:
It's not as simple as "copy these two or three files, then compile and it will work". I won't know what it specifically takes until I go through the trouble of doing it myself [...]

However, you can take a look at the recent ports (EOS M2, 1300D, 1200D, 100D, 70D) or the other D6 threads (80D, 7D2, 760D, 5D4, don't forget CHDK) or the sticky threads from development or reverse engineering areas. You'll find this question already answered.

This is a good overview for pre-D6 models:
This is an easy coding task for D6:
This is another one (to be done with QEMU):
This is yet another one:

Hey guys,
can anyone please explain how to set up the raw video for lighting photography? I played around with it and its not clear to me how especially the rec trigger setting works.
I'd like to record the last second when I hit a cable remote trigger and I thought that works when I set to "Half Shutter" but it doesn't do anything. I can only start/stop record on the back button of the 5DIII.

I've only tested it using the half-shutter button from the camera (not with a cable remote). Should work out of the box - can you show your raw video settings?

With the rec trigger enabled, the LiveView button (labeled as start/stop on 5D3) would only prepare and end the recording session; to record any frames, you would have to press half-shutter.

The "Sticky HalfShutter" Walter is talking about, does not apply here; this is a special build of mlv_lite that has a half-shutter trigger for recording (e.g. record one frame, pre-record 1 second before the trigger and so on). It's also available in the crop_rec_4k builds. It's not yet in unified because... I didn't use it for more than a few very short test clips before committing the code, so right now I have little or no idea how it performs.

The "splint" linked above is really simple to use., it may be worth your time:

Code: [Select]
... magic-lantern/modules/mlv_lite $ splint mlv_lite.c -I../../src -I../../platform/all -I.
/usr/include/bits/types.h:30:8: Parse Error: Non-function declaration:
    __BEGIN_DECLS : int. (For help on parse errors, see splint -help
*** Cannot continue.

How to get past that?

- find out how did g3gg0 get that nice GUI for QEMU (mine has no menus)

Figured it out - installing libgtk2.0-dev *before* building QEMU does the trick. Updated the screenshot.

The DryOS shell:

QEMU is working--great, but how are you guys mounting the card image so you can install ML?

One scriptable way is using mtools - if the image uses FAT16/FAT32. I'm currently using it in the test suite (tests/ and in this script: (which is not meant for interactive use, but for running some - usually short - test on more than one camera model, then saving a log and/or some screenshots). Without any mounting tools, we can use that script to copy ML on the card image:

Code: [Select]
# this is non-interactive (no display, but we'll ask it to save a screenshot, just for kicks)
# tip: use INCREMENTAL=1 to avoid running "make clean", and AUTOEXEC_ONLY=1 to avoid building/copying modules

# now we have ML on the card, so we can run it interactively
./ 60D,firmware="boot=1"

Note: the "boot=0" or "boot=1" (which looks that way because I couldn't figure out how to add custom command-line options in QEMU, so I've hijacked an existing one) is used to change the boot flag in the ROM. On the very first ML install, the ROM dumps have the boot flag turned off. Subsequent ROM dumps (e.g. after formatting the card) will have the boot flag enabled. For cameras that do not run ML yet, the boot flag will be disabled. Therefore, for this example I'm assuming nothing about the boot flag state and always setting it manually.

Todo: a "make install_qemu" would be handy.

I've set up this Win10 VM, which is a fresh installation with bash, visual studio and a few others preinstalled. The Linux subsystem is based on Ubuntu Xenial.

Package list is mostly OK; had to run "sudo apt-get update" before, and also had to install "zip".

To find out what GCC version you have, type:
Code: [Select]
arm-none-eabi-gcc- [TAB] [TAB]

Result: in my case it was 4.9.3.

For Makefile.user I prefer rewriting only the lines I need changed (rather than copying the entire Makefile.user.default). Therefore, I've created it from scratch (just these 2 lines):
Code: [Select]

In a nutshell:

Code: [Select]
# prepare system
sudo apt-get update
sudo apt-get install make gcc gcc-arm-none-eabi mercurial gcc-mingw-w64 python3-docutils zip
hg clone -u unified
cd magic-lantern
echo "GCC_VERSION=-4.9.3" > Makefile.user
echo "ARM_PATH=/usr" >> Makefile.user

# preparation complete, now build ML
cd platform/5D3.123
make zip

# desktop utilities
cd ../../modules/mlv_rec
make mlv_dump.exe
cd ../../modules/dual_iso
make cr2hdr.exe

Some more:
Code: [Select]
# ports in progress (100D, 70D)
hg update 100D_merge_fw101 -C # use TAB to find the exact name
hg merge unified # or lua_fix or whatever (optional)
cd ../../platform/100D.101
make zip

# 4K with sound
hg update crop_rec_4k_mlv_snd -C
cd ../../platform/5D3.123
make clean; make zip

# quick build (autoexec.bin only, without modules)
cd ../../platform/5D3.123

# recovery (portable display test, ROM dumper, CPU info...)
hg update recovery -C
cd ../../platform/portable.000

For QEMU, I had to install some additional packages and an X server (see e.g. Running Linux desktop apps on the Windows Subsystem for Linux):
Code: [Select]
# not sure if this step is needed
sudo nano /etc/apt/sources.list
# add deb-src entries (copy the deb ones and replace deb with deb-src)
sudo apt-get update
sudo apt-get build-dep qemu

# from g3gg0
sudo apt-get install zlib1g-dev libglib2.0 autoconf libtool libsdl-console flex bison
sudo apt-get install libsdl-console-dev

# needed for GUI
sudo apt-get install libgtk2.0-dev

# needed to access the virtual SD/CF images
sudo apt-get install mtools

# install QEMU
hg update qemu -C
cd contrib/qemu
# for some reason, the output from is truncated
# opening a new terminal appears to fix it (?!)
# if it still doesn't work: ./ |& tee install.log
# then open install.log in a text editor to read it

# anyway, now follow the instructions from
cd /path/to/qemu/qemu-2.5.0
make -j4
cd ..

# copy the ROMs from your camera (e.g. sdcard/ML/LOGS/ROM*.BIN -> qemu/5D3/ROM*.BIN)

# now run the emulation
export DISPLAY=:0
./ 60D,firmware="boot=0"

- find out how did g3gg0 get that nice GUI for QEMU (mine has no menus) [DONE]
- install a GUI for Mercurial [g3gg0 suggests TortoiseHg]
- check build environment for common errors and suggest how to fix / what to install if things go wrong (also on other usual operating systems)
- mount the SD image used in QEMU and install ML there [halfway done]
- sample debugging session with GDB+QEMU, maybe also IDA
- run the QEMU test suite (a large part of it won't work, as it's hardcoded for my ROMs)
- how to run a Lua script
- bonus: show how to build CHDK
- ... ?

Right - Trammell used Bitbucket when he started the project, and we had no good reason to change it.

Camera-specific discussion / Re: Canon 700D / T5i
« on: August 08, 2017, 03:09:11 PM »
I get warnings saying I might have encountered the null pointer bug.

May I take a look at your ROM? That's on the current crop_rec_4k codebase, right?

(it's probably a false warning, but let's check it first)

Pages: [1] 2 3 ... 342