Messages

26

Scripting Q&A / Re: MLU

« on: February 08, 2021, 05:20:31 PM »

Right; maybe somebody could volunteer to keep the documentation up to date. I'm not actively using Lua, and the documentation itself is not covered by the test suite, so... outdated docs are not always obvious to me.

27

Scripting Q&A / Re: MLU

« on: February 08, 2021, 10:40:01 AM »

Not sure I understand. What kind of focus info is not available outside LiveView?

Focus distance and DOF info were available in lua_fix builds since early 2018. ML asks the MPU for a refresh, once every second.

28

General Help Q&A / Re: Err 70

« on: February 07, 2021, 05:45:42 PM »

Apparently Canon code ended up without sufficient RAM to index the files on the card. I've got it a few times in the past, but couldn't reproduce it easily.

One may need to debug this in QEMU, on a full copy of the memory card.

Workaround: download the photos and format the card.

29

Scripting Q&A / Re: MLU

« on: February 07, 2021, 11:19:35 AM »

The usual way is to create the topic first, with relevant contents, of course.

Once that topic is useful enough to become sticky, it can be made sticky by a moderator.

30

Scripting Q&A / Re: MLU

« on: February 07, 2021, 10:56:09 AM »

In that case, you may enter LiveView from Lua, take the pictures, then return to regular photo mode. On some models (5D3 included), this actually approximates MLU behavior very well - the only side effects being possible delays from LiveView processing. If you capture the images in burst mode, these additional delays should be minimal.

A sticky topic with Lua API suggestions can be helpful, yes.

31

Scripting Q&A / Re: MLU

« on: February 07, 2021, 10:31:16 AM »

Mirror actuation is decided by the MPU, so... the only ways to achieve this would be:

1) regular pictures in LiveView, with Canon's silent shooting mode enabled (this won't move the mirror while taking pictures)
2) full-res silent pictures (which won't move the shutter either)

I'd rather make full-res silent pictures usable at short exposure times

32

Camera-specific Development / Re: DIGIC 8 'PowerShot' development (M50, SX70, SX740)

« on: February 06, 2021, 08:11:37 AM »

When sharing scripts that are likely to be run by regular users, I strongly recommend performing some kind of camera model / firmware version checking. CHDK do this for scripts meant to be run by users (look up "check_compat").

Most of the time, the effect of calling an invalid stub - which would usually jump in the middle of some random Canon function - will be a crash, but if one is unlucky enough, that crash might lead to a memory write that will end up written back into main ROM at camera shutdown (yes, Canon firmware does this). Or, what if - on some different camera model - that stub might be in the middle of EraseSectorOfRom? Very unlikely, but still...

I've got a fair amount of backlash in the past for recommending this timeless article to other developers, but I still believe the advice there is very good (long answer). And I have no experience unbricking recent Canon models, btw

33

General Chat / Re: M50 II Firmware onto a M50

« on: February 04, 2021, 06:29:26 PM »

No.

Even if they had the exact same hardware, flashing the firmware from a different camera model would be copyright infringement.

34

Raw Video Postprocessing / Re: UGLY clipping samples

« on: February 02, 2021, 03:59:54 PM »

Did your highlight patch get merged to UFRaw?

No, it's still over here. But now I realize the approach was way too much of a hack, so... no wonder they weren't impressed

On the other hand, the soft-film curve works really well in my opinion (OK, I'm biased). Sample images:
https://www.magiclantern.fm/forum/index.php?topic=5197.msg91513#msg91513

This post by the Oklab guy shows how much CIELAB skews purple between blue and white, definitely an outdated colour space that shouldn't be used anymore.

Excellent! You must be a few light years ahead of me on this topic

Do you apply some sort of perceptual mapping, where you not only map the out of gamut colors to the end color space, but also apply some mapping to the rest of the color to retain some form of perceptual linearity!

Right - the user shouldn't notice any sharp transitions in the colors. You could keep the "normal" (not highly-saturated) colors unchanged, and compress only the extreme ones - both the ones near the gamut limits, and the ones outside the gamut. The soft-film curve could actually be useful for that - but I'd have to experiment.

Thanks guys!
Wow, interesting links. I kind of knew about this phenomenon, but never thought it could happen so easily on a "pro" camera. And it isn't black on my clips..

It doesn't have to go all the way to black - but once the analog side of the sensor is saturated, the reference dark value will continue to grow (it's slightly influenced by the incoming light, as the electronic shutter is not perfect). If I understand well, that's a side effect of CDS (a technique commonly used in image sensors).

This shouldn't happen with a mechanical shutter - but need to double-check.

The visible outcome is that, above a certain threshold (when saturation happens), the apparent response curve of the image sensor becomes non-monotonic (negative slope). In the sample response curve below, imagine the response goes downwards at some point after reaching saturation.



(image source)

In Canon cameras, there's also some harsh clipping caused by saturation of downstream electronics (ADC, amplifiers and whatever else Canon uses in their pipeline). Some theories are buried over here:
https://www.magiclantern.fm/forum/index.php?topic=10111.

35

Feature Requests / Re: [DONE] hold SET at camera startup to load ML (load Canon firmware by default)

« on: February 02, 2021, 11:06:19 AM »

It's probably not yet integrated in crop_rec builds - but it's for sure present in lua_fix builds.

(I'll take a look when I'll revisit crop_rec again - hopefully in the next few months, but that's not a promise)

36

Duplicate Questions / Re: Option to hold SET to boot ML

« on: February 02, 2021, 10:19:36 AM »

Offtopic for io_crypt. Feature already in ML.

https://www.magiclantern.fm/forum/index.php?topic=21765.0

37

Raw Video Postprocessing / Re: UGLY clipping samples

« on: February 02, 2021, 08:53:12 AM »

That's the "black sun" effect - see e.g.:
- pco.de/.../20140412_FOM_CameraTutorial_PCO_fin_pdf.pdf p.31
- imagesensors.org/.../7-02__geurts.pdf p.3
- https://www.physicsforums.com/threads/black-sun-effect-in-cmos-sensors.799334/

For this particular image, the easiest way to render it would be to reduce the white level. If your raw processing software has a slider for that, use it. In mlv_dump, the relevant option is --white-fix. For this particular file, 3300 looks about right to me (from the default of 4050).


In this thread, we are looking specifically for sample images with extreme out-of-gamut clipping issues (extremely saturated colors that are rendered obviously wrong). Such issues can be found when using highly saturated light sources such as colored LEDs, or with concert photos/videos (with unusual lighting), or highly saturated scenes/subjects, such as flowers.

These colors are more colorful than the color space is able to encode.

Here's an example of an image that's clearly not clipped in RAW, but rendered incorrectly (clipped) in sRGB. I couldn't find the original raw file, but didn't look too hard.

https://www.dpreview.com/forums/post/62681348

The best way to deal with these issues, in my opinion, would be with some kind of gamut compression - but you'd have to carefully choose a suitable color space for that. In particular, I can tell you for sure that CIELAB is not the right color space for this purpose (long answer in the links shared earlier).

38

crop_rec and derived builds / Re: Danne's crop_rec_4k experiments for EOS M

« on: February 02, 2021, 08:10:00 AM »

General rule when optimizing presets: first minimize timer A. You'll want to stop as soon as you start getting black columns on the right side, or any other kind of image corruption.

This will minimize the rolling shutter for a given resolution. At the same time, since you'll need to increase timer B in order to keep the frame rate, this could help you squeeze a few more pixels vertically.

References:
- arbitrary resolution crop_rec PoC for 700D: https://www.magiclantern.fm/forum/index.php?topic=19300.msg205546#msg205546 (around line 930)
- my old 700D notes (same timings expected for EOS M): https://www.magiclantern.fm/forum/index.php?topic=19300.msg205464#msg205464
- Bilal's 700D notes: https://www.magiclantern.fm/forum/index.php?topic=19300.msg197870#msg197870
- Same explanation for 70D: https://www.magiclantern.fm/forum/index.php?topic=14309.msg205996#msg205996
- several other posts scattered throughout the forum

39

Raw Video Postprocessing / Re: UGLY clipping samples

« on: February 01, 2021, 09:57:02 PM »

Possibly related: https://twitter.com/CT_Bergstrom/status/1333614982647353347

The topic is of interest to me as well, btw - just not actively researching it right now.

My earlier hackish attempt - back then, I didn't know what I was doing:
https://www.magiclantern.fm/forum/index.php?topic=5197.0

Found two sample images (I can dig for more, if needed):

IMG_0307.CR2
IMG_5207.CR2

Some stuff I came across, but didn't try yet:
https://bottosson.github.io/posts/gamutclipping/
http://www.brucelindbloom.com/index.html?UPLab.html

Maybe it helps.

40

Raw Video Postprocessing / Re: MLV App 1.11 - All in one MLV Video Post Processing App [Windows, Mac and Linux]

« on: January 30, 2021, 01:09:19 PM »

Silent pictures contain the complete image buffer delivered by Canon firmware (including black bars), while MLV frames contain a cropped section, not including black bars. Last time I've checked, mlv_dump was handling both types of files correctly (edit: tested two samples linked earlier, they both work fine with mlv_dump).

The tricky part is that raw_info structure is copied directly from current LiveView configuration - that is, it includes the active area offsets for a complete LiveView frame. As the MLV video frame (from mlv_rec/mlv_lite) is cropped, its active area offsets should be updated, and - since the black bars are not normally included in the recorded image - those offsets would be normally 0.

In mlv_dump, this is currently handled as a special case: if the frame size from the RAWI header (xRes x yRes) is smaller than the complete raw frame size (raw_info.width x height), that must be a cropped section of the entire frame. Since the black bars are not included in the recorded image, active_area is assumed to cover the entire recorded frame (that is, active_area offsets must be rewritten - "zeroed out" - by the MLV converter). If the frame size from RAWI matches the one from raw_info, that means the recorded image contains the complete raw frame, including black bars, so the active area declared in raw_info is considered valid (and should be used for conversion).

The offset fields from RAWC are supposed to show where the complete LiveView frame (possibly binned/cropped by crop_rec, x5 zoom, Canon's 1080p / 720p / crop mode etc) is placed, relative to a full-res image (FRSP or CR2), but these are not implemented yet.

The position of the cropped image (saved by mlv_rec/mlv_lite), relative to the complete LiveView frame, is given by VIDF cropPosX/Y (restrictions: we can only crop multiples of 8 pixels horizontally - because of 14-bit packing - and multiples of 2 pixels vertically, in order to keep the same Bayer layout). The "pan" offsets (panPosX/Y) were meant for smooth digital panning within a complete LiveView frame frame (not implemented yet).

In any case, I don't see a good reason for changing the silent picture code - it works as expected, if you ask me. Changing the active_area offsets should have been done in mlv_rec/mlv_lite from the very beginning, rather than requiring MLV converters to rewrite these offsets - but now it's probably too late to make this change.

In other words, the active area metadata in a silent picture MLV is correct, while the one from a mlv_rec/lite MLV is invalid and must be "zeroed out" (recomputed from scratch) by the MLV converter. I don't see a good way out of this, other than using the above workaround (special case) in MLV converters.

41

Camera Emergency Department / Re: 60D not turning on

« on: January 29, 2021, 12:57:43 PM »

Yes. However, if you already have an UART dongle, I'd recommend using it during the firmware update, to see what exactly it does. Otherwise, you'll be running a firmware update blindly, so we'll have to do some guesswork if anything goes wrong.

Option 1 (recommended):
- Identify the camera's TXDICU pin (it prints a bunch of stuff at camera startup, even in current state) and connect it to the RX pin of the UART dongle. If using a FT232RL dongle, the jumper should be set 3V3. Probing the pins in the UART connector should be safe, as long as the wire is connected to RX (because RX only listens to incoming data, unlike TX).
- Keep the TXDICU -> RX connection active during the firmware update, and make sure you can record messages before confirming. Arduino's serial terminal is OK. Test it by starting the camera without card, a few times (you will need to take the battery out in order to reboot).
- Format a card, place Canon's firmware update (ideally 1.1.1, to run ML)
- Confirm the update by pressing SET. In QEMU, the firmware update only displays an OK button. (On other cameras, you'd have to press Right and then SET.)
- I'd expect some kind of progress messages on UART. A firmware update should normally take a few minutes.

Option 2 (blind):
- Format a card, place Canon's firmware update (ideally 1.1.1), confirm with SET.
- Leave the camera unattended for at least 10 minutes.


Afterwards, if the camera still doesn't boot, run the portable ROM dumper again, regardless of which option you choose.

If it helps, you can also jump on Discord or IRC for real-time assistance.

42

Camera Emergency Department / Re: 60D not turning on

« on: January 29, 2021, 12:35:50 PM »

ROM1 doesn't look good - main firmware is disabled and there are differences on the code area. It doesn't boot in QEMU.

It smells like a firmware update that was interrupted. The code area is mostly empty.

Compared to a good ROM1:
- some differences below FF010000, unknown, probably not relevant
- OK between FF010000 and FF510000
- empty (FF everywhere) between FF510000 and FF51FFFF
- valid contents from FF520000, but they appear to be from a different firmware version (ROM dumper log reports 1.1.2, ML runs on 1.1.1)
- identical from FFF20000 until the end (bootloader and some other stuff)

Have you tried to downgrade it from 1.1.2, and took the battery out in the process?

I can fix the interrupted firmware update, but I'm expecting the display to work in the current state. In QEMU, the camera asks for a firmware update. That is, if you start with a SD card not prepared for ML (e.g. a formatted card), that contains a Canon firmware update, the camera would start executing it without prompting. It does ask for confirmation before attempting to reflash, though.

43

Camera Emergency Department / Re: 60D not turning on

« on: January 29, 2021, 11:33:06 AM »

To be clear, the portable ROM dumper managed to save ROM0.BIN on the card, but didn't display anything on the main LCD? In other words, you ran it blindly?

In that case, the built-in display may be faulty. Any luck using an external monitor?

Can you let the portable ROM dumper run for a little longer? It should also save ROM1.BIN.

If the startup-log builds didn't save a log, and we've got no working display, the easiest way to get more logs is via UART. You would have to disassemble the back cover, as the connector is not easily accessible on this camera. It's under the rubber on the camera back, bottom right side.

Alternatively, I can also try to blink the debug messages using a LED, but that requires either a second camera running ML, or an Arduino with some kind of light sensor (photoresistor, phototransistor).

44

crop_rec and derived builds / Re: 3K/UHD 5D2 Raw development and Other Digic IV Cams

« on: January 28, 2021, 07:44:42 AM »

Also hoping all is well, but locking is probably not needed - the small community already formed around these builds can still continue the discussion, if you ask me.

45

Camera Emergency Department / Re: 60D not turning on

« on: January 26, 2021, 01:27:15 PM »

Portable ROM dumper worked? If yes, keep the ROMs safe or send them to me privately (don't share them here).

You may try to capture further logs with these builds:
https://builds.magiclantern.fm/jenkins/job/startup-log/
https://builds.magiclantern.fm/jenkins/job/startup-log-mpu/

If you can't get any logs, try this one:
https://www.magiclantern.fm/forum/index.php?topic=25668.msg233208#msg233208

46

Camera Emergency Department / Re: Canon 60D has Err 70/80

« on: January 26, 2021, 01:25:37 PM »

Each ERR70 should be accompanied by a crash log saved by ML - please upload those.

You may also try to capture further logs with these builds:
https://builds.magiclantern.fm/jenkins/job/startup-log/
https://builds.magiclantern.fm/jenkins/job/startup-log-mpu/

47

Scripting Q&A / Re: Full shutter press and event handler

« on: January 25, 2021, 05:31:29 PM »

This one is easy: set AF on the back button, and the shutter will fire as soon as it's pressed, without waiting for focus confirmation.

That's how camera.shoot() works - by default, it doesn't autofocus and shoudn't wait for any kind of confirmation. You can optionally ask for autofocus, and in that case, it will wait.

https://builds.magiclantern.fm/lua_api/modules/camera.html#shoot

You can also autofocus without taking a picture, with lens.autofocus():
https://builds.magiclantern.fm/lua_api/modules/lens.html#autofocus

48

Scripting Q&A / Re: Full shutter press and event handler

« on: January 25, 2021, 12:41:14 PM »

I've considered that as well. If you were wondering why ML uses the first picture (captured by the user) in order to start a bracketing sequence, or a half-shutter press in order to capture a full-res silent picture, rather than full-shutter press, the above answer still applies

For hand-held bracketing, I'd recommend some automation around the full-res LiveView functionality from crop_rec_4k and derived builds, as that would minimize the delay between two captures. Unfortunately, it's not something Lua can help with...

49

Scripting Q&A / Re: Full shutter press and event handler

« on: January 25, 2021, 12:32:51 PM »

When using the keypress event handler, I can’t seem to hijack the full shutter and unpress full shutter, ie even when I return false, the shutter still operates, both in LV and non LV mode.

You want to prevent the camera from taking a picture when user fully presses the shutter button, right?

That is, no shutter/mirror movement should be done when user presses the shutter?

If that's the case, bad news - it can't be done easily from the main ARM CPU (the one where ML runs), because this button is handled by the MPU. The decision to actuate the shutter is made on the MPU. Afterwards, the MPU transmits the event to the main CPU, which is supposed to complete the image capture process - but it has no veto rights (it can't block the shutter movement - the MPU decides this on its own).

The only way - known to me - to block the shutter button is to put the camera into BUSY mode (with that message displayed by Canon firmware). The memory allocation backend (exmem) does this when allocating SRM memory (as the camera would crash if user tries to take a picture while ML uses SRM memory). The functionality is not yet available to Lua, but it can be done if there is a very good use case for it (as usual).

I would love to have all these posts in one single place. Is this possible?

CC @ moderators, they have the power to join threads (or split them).

50

Camera Emergency Department / Re: 60D not turning on.

« on: January 25, 2021, 09:18:06 AM »

A real hardware bug!   

Rather, a colony of bugs

This is a first or what you mean, a1ex?

First time seeing such a defect? Definitely. TBH, I haven't disassembled any cameras yet, other than removing the back cover from my 5D3 for troubleshooting this issue.

However, it's not the first case of a camera that doesn't turn on because of a faulty power switch. Another recent example here.

Thanks for the pictures - shared the story on Twitter


BTW, checking the state of the power switch is not straightforward with code running on the main CPU (this switch is handled by the MPU), but the experiments mentioned earlier in this thread might provide a reasonably good heuristic for diagnosing such issues. That is, manually triggering an ERR70 (e.g. via some invalid call to AllocateMemory) should be able to save a log file from main firmware, even if the camera is started with the power switch off (or with the card/battery door open). TODO: prepare a test build and cross-check the hypothesis on different camera models.

If the camera can be tricked into saving a log file from main firmware, that means the boot process goes far enough, so the camera is at least halfway alive (both hardware and main firmware, including sane property data structures etc). That's one step further, compared to "just" being able to run the portable ROM dumper (which runs from the bootloader context, not from main firmware - see this flowchart for details).