Canon 5D Mark III / 5D3 / Firmware 1.3.4

[dfort]

It's usually spelled Excute in Canon code.

Ok--now I see it. There's a bunch of "DlgMnCardFormatExcute" all over the place.

(the obvious part is on the UI, but the presence of the bug can be found in these logs)

Obvious? Hum, how does Canon spell UI?

[nexiabass]

Just installed the newest version.
It seems to be magic.... The Q Button works as expected now!
Thank you so much 

[dfort]

Still looking for trouble. Self tests are passing.



Going from 1.2.3 to 1.3.4 check-stubs.py flags a few stubs that affect the UI but these seem to be correct.

Code: [Select]

gui_main_struct                     0x00024120 -> 0x00024120 [0x000]
dm_names                            0x00024a70 -> 0x00024a70 [0x000]
task_max                            0x00025024 -> 0x00025024 [0x000]
gui_task_list                       0x000321c8 -> 0x000321e0 [0x018] [!!!]
cf_device_ptr                       0x00036184 -> 0x0003619c [0x018] [!!!]
sd_device                           0x00039960 -> 0x00039978 [0x018] [!!!]
LCD_Palette                         0x00041510 -> 0x00041530 [0x020] [!!!]
bmp_vram_info                       0x00042d10 -> 0x00042d30 [0x020] [!!!]
additional_version                  0x00046910 -> 0x00046930 [0x020] [!!!]
camera_engine                       0xc0220000 -> 0xc0220000 [0x000]
cstart                              0xff0c1ca0 -> 0xff0c1ca0 [0x000] [!!!]
init_task                           0xff0c5298 -> 0xff0c52a0 [0x008]
_malloc                             0xff0c8178 -> 0xff0c8180 [0x008]
_free                               0xff0c81d0 -> 0xff0c81d8 [0x008]
gui_main_task                       0xff0de948 -> 0xff0de950 [0x008]
GUI_Control                         0xff0ded1c -> 0xff0ded24 [0x008]
gui_init_end                        0xff0df384 -> 0xff0df38c [0x008]

Is it one bug or several? And it is in the api_test.lua log? Feel I'm overlooking something very obvious.

[dfort]

...the presence of the bug can be found in these logs...

Took another look and of course it hit me in the face this time.

Code: [Select]

lens = table:
  name = "EF50mm f/1.8 STM"
  focal_length = 0
  focus_distance = 14080
  hyperfocal = 0
  dof_near = 0
  dof_far = 0
  af = true
  af_mode = 0
  focus = function: p

Yeah, it shows up in the UI.



Some of it seems to be fine but obviously some of those values are out of whack.



Of course I found this out right when I have to run off to an appointment.

[dfort]

Fixed! Maybe?





New test build posted.

[EDIT] Log looks better?

Code: [Select]

===============================================================================
ML/SCRIPTS/api_test.lua - 2017-11-17 23:17:32
===============================================================================

Strict mode tests...
Strict mode tests passed.

Generic tests...
camera = table:
  shutter = table:
    raw = 96
    apex = 5
    ms = 31
    value = 0.03125
  aperture = table:
    raw = 48
    apex = 5
    value = 5.599999
    min = table:
      raw = 22
      apex = 1.75
      value = 1.8
    max = table:
      raw = 80
      apex = 9
      value = 22.6
  iso = table:
    raw = 96
    apex = 8
    value = 800
  ec = table:
    raw = 0
    value = 0
  flash_ec = table:
    raw = 0
    value = 0
  kelvin = 5500
  mode = 3
  metering_mode = 3
  drive_mode = 0
  model = "Canon EOS 5D Mark III"
  model_short = "5D3"
  firmware = "1.3.4"
  temperature = 152
  state = 0
  bulb = function: p
  reboot = function: p
  shoot = function: p
event = table:
  pre_shoot = nil
  post_shoot = nil
  shoot_task = nil
  seconds_clock = nil
  keypress = nil
  custom_picture_taking = nil
  intervalometer = nil
  config_save = nil
console = table:
  show = function: p
  hide = function: p
  write = function: p
  clear = function: p
lv = table:
  enabled = true
  paused = false
  running = true
  zoom = 1
  pause = function: p
  start = function: p
  resume = function: p
  info = function: p
  stop = function: p
  wait = function: p
lens = table:
  name = "EF50mm f/1.8 STM"
  focal_length = 50
  focus_distance = 410
  hyperfocal = 15444
  dof_near = 400
  dof_far = 419
  af = true
  af_mode = 0
  focus = function: p
display = table:
  idle = true
  height = 480
  width = 720
  screenshot = function: p
  pixel = function: p
  off = function: p
  circle = function: p
  rect = function: p
  notify_box = function: p
  load = function: p
  line = function: p
  clear = function: p
  on = function: p
  draw = function: p
  print = function: p
key = table:
  last = 0
  press = function: p
  wait = function: p
menu = table:
  visible = false
  get = function: p
  close = function: p
  block = function: p
  open = function: p
  set = function: p
  new = function: p
testmenu = userdata:
  value = 0
  name = "Script API tests"
  help = "Various tests for the Lua scripting API."
  help2 = "When adding new Lua APIs, tests for them should go here."
  advanced = 0
  depends_on = 0
  edit_mode = 0
  hidden = false
  icon_type = 5
  jhidden = false
  max = 0
  min = 0
  selected = true
  shidden = false
  starred = false
  submenu_height = 0
  submenu_width = 0
  unit = 0
  works_best_in = 0
  run_in_separate_task = 0
  select = function: p
  update = nil
  info = nil
  rinfo = nil
  warning = nil
movie = table:
  recording = false
  stop = function: p
  start = function: p
dryos = table:
  clock = 18
  ms_clock = 18204
  prefix = "1Y7A"
  dcim_dir = table:
    exists = true
    create = function: p
    children = function: p
    files = function: p
    parent = table:
      exists = true
      create = function: p
      children = function: p
      files = function: p
      parent = table:
        exists = true
        create = function: p
        children = function: p
        files = function: p
        parent = nil
        path = "A:/"
      path = "A:/DCIM/"
    path = "A:/DCIM/100EOS5D/"
  config_dir = table:
    exists = true
    create = function: p
    children = function: p
    files = function: p
    parent = table:
      exists = true
      create = function: p
      children = function: p
      files = function: p
      parent = table:
        exists = true
        create = function: p
        children = function: p
        files = function: p
        parent = nil
        path = "B:/"
      path = "ML/"
    path = "ML/SETTINGS/"
  ml_card = table:
    cluster_size = 131072
    drive_letter = "B"
    file_number = 3188
    folder_number = 100
    free_space = 62705536
    type = "SD"
    path = "B:/"
    _card_ptr = userdata
  shooting_card = table:
    cluster_size = 32768
    drive_letter = "A"
    file_number = 3188
    folder_number = 100
    free_space = 85757472
    type = "CF"
    path = "A:/"
    _card_ptr = userdata
  date = table:
    year = 2017
    wday = 6
    day = 17
    min = 17
    yday = 321
    month = 11
    hour = 23
    isdst = false
    sec = 34
  remove = function: p
  directory = function: p
  call = function: p
interval = table:
  time = 10
  count = 0
  running = 0
  stop = function: p
battery = table:
  level = 37
  id = 0
  performance = 3
  time = 3996
  drain_rate = 33
task = table:
  yield = function: p
  create = function: p
property = table:
Generic tests completed.

Module tests...
Copy test: autoexec.bin -> tmp.bin
Copy test OK
Append test: tmp.txt
Append test OK
Testing exposure settings, module 'camera'...
Camera    : Canon EOS 5D Mark III (5D3) 1.3.4
Lens      : EF50mm f/1.8 STM
Shoot mode: 3
Shutter   : Ç30 (raw 96, 0.03125s, 31ms, apex 5)
Aperture  : Å5.6 (raw 48, f/5.599999, apex 5)
Av range  : Å1.8..Å22 (raw 22..80, f/1.8..f/22.6, apex 1.75..9)
ISO       : 800 (raw 96, 800, apex 8)
EC        : 0.0 (raw 0, 0 EV)
Flash EC  : 0.0 (raw 0, 0 EV)
Setting shutter to random values...
Setting ISO to random values...
Setting aperture to random values...
Please switch to Av mode.
Setting EC to random values...
Setting Flash EC to random values...
Exposure tests completed.

Testing module 'lv'...
LiveView is running; stopping...
Starting LiveView...
Setting zoom to x1...
Setting zoom to x5...
Setting zoom to x10...
Setting zoom to x5...
Setting zoom to x1...
Setting zoom to x10...
Setting zoom to x1...
Pausing LiveView...
Resuming LiveView...
Stopping LiveView...
LiveView tests completed.

Focus distance: 410
Focusing backward...
Focus distance: 8610
Focusing forward with step size 3, wait=true...
......................
Focus distance: 370
Focusing backward with step size 3, wait=true...
......................
Focus distance: 8610
Focus range: 22 steps forward, 22 steps backward.
Focusing forward with step size 3, wait=false...
......................
Focus distance: 370
Focusing backward with step size 3, wait=false...
......................
Focus distance: 8610
Focus range: 22 steps forward, 22 steps backward.
Focusing forward with step size 2, wait=true...
...
Focus distance: 370
Focusing backward with step size 2, wait=true...
...
Focus distance: 655350
Focus range: 159 steps forward, 160 steps backward.
Focusing forward with step size 2, wait=false...
...
Focus distance: 370
Focusing backward with step size 2, wait=false...
...
Focus distance: 655350
Focus range: 159 steps forward, 159 steps backward.
Focusing forward with step size 1, wait=true...
...
Focus distance: 370
Focusing backward with step size 1, wait=true...
...
Focus distance: 655350
Focus range: 1053 steps forward, 1054 steps backward.
Focusing forward with step size 1, wait=false...
...
Focus distance: 370
Focusing backward with step size 1, wait=false...
...
Focus distance: 655350
Focus range: 1054 steps forward, 1053 steps backward.
Focus test completed.

Done!


[a1ex]

Right - this time, the focus distance figures look sane. For infinity, Canon firmware returns 65535 cm; maybe this can be adjusted in Lua somehow, but that applies to all models, not just 5D3 1.3.4.

There may be (didn't test) a similar bug in another widely used feature; it can be noticed with grep.

[garry23]

...a word of caution.

Remember the DoFs in the core ML are broken because of the integer math that gets used in C.

I don't think it has been corrected yet.

All ok in Lua, hence my focus bar script returns the 'correct' DoFs based on the usual assumptions, eg thin lens formula is ok to use and Canon returns a reasonable value for the focus distance.

[dfort]

...it can be noticed with grep.

That gave me an idea, not to debug a widely used feature but to find anything that I might have missed pertaining to this 1.3.4 firmware update.

Here's one I just found and updated:

fps-engio.c

Code: [Select]

#elif defined(CONFIG_5D3)
    #define NEW_FPS_METHOD 1
    #ifdef CONFIG_5D3_123
    #define SENSOR_TIMING_TABLE MEM(0x32530)
    #elseif CONFIG_5D3_134
    #define SENSOR_TIMING_TABLE MEM(0x32548)
    #else
    #define SENSOR_TIMING_TABLE MEM(0x325ac)
    #endif


[dfort]

Combed through the source tree with a fine toothed comb and came up with a few more fixes.

• io_crypt module now works. Well, I've never actually used it but the addresses do check out if anyone wants to check it out. (Module posted)
• 5D3.134 should also have quad-buffered LV like on the 5D3.123 so I updated it in vram.c

I also found this:

qemu-helper.c

Code: [Select]

// all DIGIC V cameras require a RAM offset, take it from stubs.S
#if defined(CONFIG_650D)
#define RAM_OFFSET (0xFFA4DF58-0x1900)
#elif defined(CONFIG_5D3)
#define RAM_OFFSET 0xFF9F07C0
#elif defined(CONFIG_6D)
#define RAM_OFFSET 0xFFCC34D4
#elif defined(CONFIG_EOSM)
#define RAM_OFFSET 0xFFA68D58
#else
#define RAM_OFFSET 0
#endif

qemu-helper.c is not in the qemu branch so maybe it has been depreciated? In any case, the RAM_OFFSET in this file is only valid for the 5D3.123.

[dfort]

There may be (didn't test) a similar bug in another widely used feature; it can be noticed with grep.

Sort of hard to track down a bug with description like that. 

Here's one, still photo playback with cropmarks turned on results in the cropmark overlay to flash intermittently on the screen. This happens only with still photos, reviewing movies is fine. Well, it does flicker slightly where the cropmarks overlap with the Canon overlays but that probably can't be avoided.

I suppose that the best way to test bugs like this is to see if it can be reproduced on 1.2.3. Since I have never used the overlaying cropmarks on still photo during playback feature I'll just make a note of it and keep testing until I find a show stopper before switching over to 1.2.3. As you know downgrading from 1.3.3/1.3.4 is a bit more complicated.

Of course if any of you 2,000 downloaders could give feedback it would be greatly appreciated. (I doubt there were actually 2,000 individuals who downloaded and tried this but hey, some ML users seem to want this so let's hear from you!)

@garry23 -- Are you testing 1.3.4? Some of your scripts probably require the LUA fixes. I was thinking about porting 1.3.4 to some of the experimental branches but I would like to know that if I do put in that effort somebody is actually going to try it and report back.

[garry23]

@dfort

Watching all your hard work and experiments, but I only use 1.2,3.

As for scripting, I've 'withdrawn' my auto focus bracketing script as it has been overtaken by my focus bar script, that runs on any lens that reports focus distance, including EOSM lenses.

I've also made the focus bar highly integrated with Canon and ML, i.e. I think the UI is 'smooth'.

The focus bar gives me everything I need to focus with ML, eg correct DoF, infinity focusing and focus stacking.

I'm now looking for my next scripting challange 

One thing I haven't worked out yet is how to store user presets from a script.

Hope things are going well for you?

Cheers

Garry,

[dfort]

Speaking of focus, 1.3.3 added this:

Improves the AF controllability when shooting in Live View mode with a wide-angle lens (fixed focal length or zoom).

Not sure if this affects your lua script but it would be great to find out.

[garry23]

@dfort

Now you've got me thinking 

[garry23]

@dfort

Ok can't find anything specific, but the WA 1.3.3 fix appears to be related to AF..

My focus bar is used in manual focus mode.

Cheers

Garry

[eugene-r]

Of course if any of you 2,000 downloaders could give feedback

2000 downloaders just unable to install ML without ML-SETUP.FIR
At least me unable... I searched inet how to make card bootable, and set bootflag using gparted, but looks like this not enough.

[dfort]

2000 downloaders just unable to install ML without ML-SETUP.FIR
At least me unable... I searched inet how to make card bootable, and set bootflag using gparted, but looks like this not enough.

You can use EOScard (Windows) or MacBoot (OS X/macOS) to make card bootable but you don't need to go there because installing ML will set the boot flag on both the card and the camera. Read the first post on how to install without ML-SETUP.FIR:

To make testing easier for users to try it out, I'll be posting builds to my bitbucket repository download area until it gets into the Experiments download page. Note that this first very early build doesn't have the ML-SETUP.FIR file yet so make sure your camera bootflag is set before doing the firmware update. To go back to 113 or 123, follow the instructions on the Firmware Update/Downdate? tutorial.

[eugene-r]

Read the first post on how to install without ML-SETUP.FIR:

Note that this first very early build doesn't have the ML-SETUP.FIR file yet so make sure your camera bootflag is set before doing the firmware update. To go back to 113 or 123, follow the instructions on the Firmware Update/Downdate? tutorial.

Is there any way to set camera bootflag without firmware downgrade?

If not, I'll need to wait when ML-SETUP.FIR will appear in builds.

[Walter Schulz]

Is there any way to set camera bootflag without firmware downgrade?

No.

If not, I'll need to wait when ML-SETUP.FIR will appear in builds.

Why is that?

[eugene-r]

Why is that?

Well, finally I dare to downgrade firmware, installed ML for 1.2.3, upgraded firmware to 1.3.4 and new ML works.)

But Canon just released firmware 1.3.5

[dfort]

But Canon just released firmware 1.3.5

Ha ha! Right you are, they released it just today!

1.  Fixes a phenomenon in which standard exposure may not be obtained, or an irregular exposure may result, when Silent LV (Live View) shooting with the following TS-E lenses: TS-E 50mm f/2.8L MACRO, TS-E 90mm f/2.8L MACRO, or TS-E 135mm f/4L MACRO.

TS-E lenses? Didn't know they added electronics to tilt-shift lenses. Interesting that the 1.3.4 updater is still available. Probably a minor update that can be done from the 5D3.134 branch. Anyone want to start a new topic?

[Walter Schulz]

Of course they have electronics: Aperture is controlled by cam. "E" in TS-E actually stands for "Electronic" (other sources are telling TS-E means "Tilt-and-Shift for EOS", though).

[a1ex]

Heh - I was hoping they have finally fixed the LiveView underexposure bug with manual lenses

[Walter Schulz]

Maybe they have. Canon sometimes prefers not to tell all the changes.
And the whole cam bunch is getting this update, too.

[a1ex]

Just looked into the downgrade restriction - tried to launch firmware 1.1.3 from 1.3.4.

Code: [Select]

  1601:  5698.304 [UPDC] CheckFirVersion(80000285)(129)
  1607:  5698.560 [UPDC] FileName(5D300113.FIR)(.FIR)0x20
  1609:  5702.912 [UPDC] Find Under Viersion ModelID(113 <= 129)

Where does that 129 come from?

Code: [Select]

Breakpoint 3, 0xff731cb4 in ?? ()
(gdb) print_current_location_with_callstack
Current stack: [1a4580-1a0580] sp=1a4448                                         at [CtrlSrv:ff731cb4:ff73206c]
0xFF2F2558(0, ff2f2558, 19980218, 19980218)                                      at [CtrlSrv:de48:1a4578] (pc:sp)
 0xFF453C4C(812, 2134, 0, 2134)                                                  at [CtrlSrv:ff2f2998:1a4548] (pc:sp)
  0xFF453704(824408, 812, 2134, 0)                                               at [CtrlSrv:ff453df4:1a4518] (pc:sp)
   0xFF455158(8242d0 &"DIALOG", 812, 2134, 0)                                    at [CtrlSrv:ff453764:1a4500] (pc:sp)
    0xFF705E44(8242d0 &"DIALOG", 0, a, 2134)                                     at [CtrlSrv:ff45561c:1a44c0] (pc:sp)
     0xFF731F80(80000285, 81, 6bbec4, b8)                                        at [CtrlSrv:ff706284:1a4468] (pc:sp)
      0xFF731CB4(80000285, 81, 4074c988, ff732114)                               at [CtrlSrv:ff732068:1a4448] (pc:sp)

Looking near ff706284 (where the 129 = 0x81 appears), this is stored at 0x26A84.

Who sets this value?

Code: [Select]

[     CtrlSrv:ff18c784 ] (84:01) copyDataToStorage eventID(0x7)Data(129)size(0)
[     CtrlSrv:ff190fd4 ] (83:03) PROP_PROHIBIT_UPDATE_VESION [129]

Hardware watchpoint 4: *0x26A84

Old value = 0x0
New value = 0x81
0xff190fe0 in ?? ()
(gdb) c
Continuing.
[     CtrlSrv:ff18c784 ] (84:01) copyDataToStorage eventID(0x8)Data(9796376)size(8)
[     CtrlSrv:ff18f4d8 ] (83:03) PROP_PERMIT_UPDATE_VERSION [1.3.0]

Hm, two properties, not present in 1.1.3/1.2.3. These might be stored in the ROM - not going to change them on a physical camera without safeguards (running in QEMU first, ROM backups, maybe also this).


edit: ok, upgraded to 1.3.5 - the underexposure bug is still there.

[dfort]

Oh--so you're on 1.3.5? Planning on trying out my 1.3.4 pull request before returning to 1.1.3?

I've gone between 1.1.3 and 1.3.4 many times always resetting the custom settings and the camera seems to be no worse off. Then again I got it second hand so whatever issues it has I'll blame it on the previous owner.

In any case I've been playing with other stuff but I'll eventually get back to searching for bugs on 1.3.4 or maybe push it to 1.3.5 just for the practice.