Lua Scripting (lua.mo)

[garry23]

@A1ex

Many thanks for finding the time to initiate this 'scrub down' of the Lua documentation.

The immediate help is clear to me, eg removing references to calls/function that are not yet enabled.

Also providing guidance as to use, eg in lens.focus.

Finally, how do you suggest we manage others 'adding value'. From my experience is can be confusing if many, in parallel, are trying to edit/change a document. I say this as there will be a time (soon), when others, such as myself, may wish to add helpful text/clarification to the documentation.

Once again: many thanks for kick-starting this Lua doc update.

Cheers

Garry

[a1ex]

Finally, how do you suggest we manage others 'adding value'. From my experience is can be confusing if many, in parallel, are trying to edit/change a document.

As long as they do not touch the same sections (even if the edits are in the same file), pull requests will handle it just fine. If there are conflicts, these can be resolved manually.

These can be done from the web interface, see https://www.magiclantern.fm/forum/index.php?topic=7940.0 . It's only a matter of editing the comments.

[a1ex]

While troubleshooting this script, I've smelled some issues related to multitasking (which I didn't really understand how it's implemented in Lua, other than "it has semaphores, appears to work, g3gg0 merged it, so it's probably OK"), so I've added a simple test to api_test.lua.

Do not attempt to run it outside QEMU, unless you have a ROM backup and a camera I know how to reflash (550D or 60D).

Code Select Expand


function taskA()
    printf("Task A started.\n")
    local i
    for i = 1,100 do
        printf("Task A: %d\n", i)
        task.yield(math.random(10,50))
    end
end

function taskB()
    printf("Task B started.\n")
    local i
    for i = 1,100 do
        printf("Task B: %d\n", i)
        task.yield(math.random(10,50))
    end
end

function test_multitasking()
    printf("Testing multitasking...\n")

    task.create(taskA)
    task.create(taskB)
    task.yield(5000)

    printf("Multitasking tests completed.\n")
    printf("\n")
end


This gives camera lockup or reboot, in the best case. Happens on both nightly and lua_fix. It's not hard to imagine an unlucky sequence of events that may result in permanent camera bricking (hence the above warning).

I've got some partial success with the ThreadsTutorial page. Removed the current semaphore guards and defined the following:

Code Select Expand


#define lua_lock(L)    lua_take_sem(L, 0, 0)
#define lua_unlock(L)   lua_give_sem(L, 0)

/* create this somewhere at startup; single-script testing only */
static struct semaphore * semm = NULL;

int lua_take_sem(lua_State * L, int timeout, struct semaphore ** assoc_semaphore)
{
    take_semaphore(semm, 0);
}

int lua_give_sem(lua_State * L, struct semaphore ** assoc_semaphore)
{
    give_semaphore(semm);
}


With this, I no longer get crashes or reboots, and the error message looks like this:



The error message is not always the same, but it's along the same lines. Showing two runs (same code) on the lua_fix branch.

What I think it happens: when calling C functions, Lua calls lua_unlock before and lua_lock after the function (source); therefore, while a C function is running, another thread is free to interrupt at the wrong moment.

This invalidates the following approach:

2. Long running API calls (like camera.shoot) should "give" the semaphore while they are executing.

The ThreadsTutorial page mentions each thread (task in DryOS) needs a separate Lua state, to be created either with lua_open (from scratch) or with lua_newthread (so they can share global variables). I've found this example using the second method, but it only works on Lua 5.1, and I don't understand Lua internals well enough to port the code to a newer version.

As I have little or no experience with Lua, and zero experience with multithreaded Lua, any help is welcome.

Otherwise, I'm afraid I'll have to remove the multi-threading functionality (event handlers included). I agree this functionality is very useful, but not at the cost of totally random behavior.

@garry23: please double-check you have a backup copy of your ROM, just in case.

[dfort]

Sounds scary so I made sure I had the ROM backup ready and the emergency crew on standby before running the test. It didn't get very far, froze and needed a battery pull. Luckily the camera, an EOSM, survived unscathed.

Code Select Expand


===============================================================================
ML/SCRIPTS/API_TEST.LUA - 2017-1-27 21:05:08
===============================================================================

Strict mode tests...
Strict mode tests passed.

Generic tests...
arg = table:
  [0] = "API_TEST.LUA"
camera = table:
  shutter = table:
    raw = 104
    apex = 6.
    ms = 16
    value = 0.015624
  aperture = table:
    raw = 56
    apex = 6.
    value = 8
    min = table:
      raw = 32
      apex = 3.
      value = 2.799999
    max = table:
      raw = 80
      apex = 9
      value = 22.6
  iso = table:
    raw = 96
    apex = 8
    value = 800
  ec = table:
    raw = 0
    value = 0
  flash_ec = table:
    raw = 0
    value = 0
  kelvin = 6500
  mode = 3
  metering_mode = 3
  drive_mode = 1
  model = "Canon EOS M"
  model_short = "EOSM"
  firmware = "2.0.2"
  temperature = 198
  state = 0
  reboot = function: p
  bulb = function: p
  burst = function: p
  shoot = function: p
event = table:
  pre_shoot = nil
  post_shoot = nil
  shoot_task = nil
  seconds_clock = nil
  keypress = nil
  custom_picture_taking = nil
  intervalometer = nil
  config_save = nil
console = table:
  clear = function: p
  hide = function: p
  show = function: p
  write = function: p
lv = table:
  enabled = true
  paused = false
  running = true
  zoom = 1
  start = function: p
  pause = function: p
  resume = function: p
  wait = function: p
  stop = function: p
  info = function: p
lens = table:
  name = "EF-S17-55mm f/2.8 IS USM"
  focal_length = 17
  focus_distance = 655350
  hyperfocal = 1918
  dof_near = 1895
  dof_far = 1000000
  af = true
  af_mode = 0
  focus = function: p
  autofocus = function: p
display = table:
  idle = true
  height = 480
  width = 720
  circle = function: p
  screenshot = function: p
  line = function: p
  notify_box = function: p
  on = function: p
  rect = function: p
  print = function: p
  draw = function: p
  load = function: p
  off = function: p
  clear = function: p
  pixel = function: p
key = table:
  last = 10
  wait = function: p
  press = function: p
menu = table:
  visible = false
  new = function: p
  set = function: p
  block = function: p
  close = function: p
  open = function: p
  get = function: p
movie = table:
  recording = false
  start = function: p
  stop = function: p
dryos = table:
  clock = 19
  ms_clock = 19292
  prefix = "IMG_"
  dcim_dir = table:
    exists = true
    create = function: p
    children = function: p
    files = function: p
    parent = table:
      exists = true
      create = function: p
      children = function: p
      files = function: p
      parent = table:
        exists = true
        create = function: p
        children = function: p
        files = function: p
        parent = nil
        path = "B:/"
      path = "B:/DCIM/"
    path = "B:/DCIM/100CANON/"
  config_dir = table:
    exists = true
    create = function: p
    children = function: p
    files = function: p
    parent = table:
      exists = true
      create = function: p
      children = function: p
      files = function: p
      parent = table:
        exists = true
        create = function: p
        children = function: p
        files = function: p
        parent = nil
        path = "B:/"
      path = "ML/"
    path = "ML/SETTINGS/"
  ml_card = table:
    cluster_size = 32768
    drive_letter = "B"
    file_number = 5764
    folder_number = 100
    free_space = 31117312
    type = "SD"
    _card_ptr = userdata
    path = "B:/"
  shooting_card = table:
    cluster_size = 32768
    drive_letter = "B"
    file_number = 5764
    folder_number = 100
    free_space = 31117312
    type = "SD"
    _card_ptr = userdata
    path = "B:/"
  date = table:
    yday = 27
    min = 5
    sec = 10
    hour = 21
    isdst = false
    month = 1
    wday = 6
    year = 2017
    day = 27
  rename = function: p
  remove = function: p
  directory = function: p
  call = function: p
interval = table:
  time = 10
  count = 0
  running = false
  stop = function: p
battery = table:
function not available on this camera
stack traceback:
[C]: in ?
[C]: in for iterator 'for iterator'
ML/SCRIPTS/LIB/logger.lua:125: in function 'logger.serialize'
ML/SCRIPTS/API_TEST.LUA:32: in function <ML/SCRIPTS/API_TEST.LUA:31>
[C]: in function 'xpcall'
ML/SCRIPTS/API_TEST.LUA:31: in function 'print_table'
ML/SCRIPTS/API_TEST.LUA:77: in function 'generic_tests'
ML/SCRIPTS/API_TEST.LUA:788: in function 'api_tests'
ML/SCRIPTS/API_TEST.LUA:810: in main chunktask = table:
  yield = function: p
  create = function: p
property = table:
Generic tests completed.

Module tests...
Testing file I/O...
Copy test: autoexec.bin -> tmp.bin
Copy test OK
Append test: tmp.txt
Append test OK
Rename test: apple.txt -> banana.txt
Rename test OK
Rename test: apple.txt -> ML/banana.txt


[a1ex]

I'm not looking for testing, I'm looking for suggestions from those familiar with multithreading in Lua.

[garry23]

@a1ex

I'm afraid I will be off the map for about a month and can't afford to 'experiment' for a while, i.e. I'm taking four camera bodies away with me and need 'stability'.

I'll keep an eye on progress and will reconnect with the Lua experiments when I'm back.

Cheers

Garry

[a1ex]

Added latest Lua docs on the nightly builds page:

http://builds.magiclantern.fm/lua_api/

I still have some trouble with CSS, so a bit of help would be welcome. I'd like to keep the navigation menu from the Builds site, but it looks like there is some conflict between the two styles, so it's probably best to tweak ldoc.css somehow.

The page with navigation menu enabled is here (notice the different style once you switch to other pages):

http://builds.magiclantern.fm/lua_api/index2.html

As I'm not an expert in web design, suggestions are welcome.

[garry23]

Once again A1ex thanks for keeping the docs in sync with the 'fixes'.

BTW does lens.focus not return anything now?

Cheers

Garry

[a1ex]

It returns the same as before. The old docs had it missing as well...

[garry23]

 

[a1ex]

Found a temporary workaround for the multitasking issue.

If one task calls yield, and other Lua tasks or event handlers that may be interrupting it are allowed to complete their execution before the first task resumes, things appear to work fine.

So I've implemented a workaround that simply throws an error if two tasks call yield() at the same time (which causes the camera to crash). That's probably good enough as a temporary workaround for merging into the nightly, until a proper solution is found (which, given my lack of experience with multi-threaded programs, and also the eagerness of ML community to solve this issue, will probably take years or more).

This update touches more than just Lua - a lot of internals were fine-tuned to pass the Lua API tests, and I'd like to merge it in the nightly before proceeding with recent ports (100D and 70D), as some changes are helpful for those models as well. Therefore, please test and report your experience with the latest experimental Lua build - not just for Lua, but also for everything else.

[garry23]

@A1ex

First try at the latest experimental build.

I simply removed all the supplied Lua scripts from the ML folder and put my Landscape Bracketing script in the folder, which works on the current Nightly.

I set the script to autoload.

On restart I switched to LV to observe the console and ML 'froze' at the console announcement saying my script was loading.

Not sure where to go now.

Cheers

Garry

[garry23]

Just ran the API test and get the following error

TEST.LUA 765 assertion failed

....'globals assert'....

....765
....905
....921

....not enabling autorun error

[a1ex]

Works here. Did the camera have something to focus on?

(I thought it's obvious from the error message)

[garry23]

@A1ex

The problem I'm having (5D3) is that I can't even get the experimental build running as when I load my main script, that runs with the current nightly, the ML script menu just freezes.

What I don't know is if anything has changed between the nightly and the latest Lua Fix experiment, ie that is in my script.

[garry23]

When it freezes the following text, in red, is displayed in the top left of the LV screen

[2] ? stack overflow free = 0 used = 0

[dmilligan]

What I don't know is if anything has changed between the nightly and the latest Lua Fix experiment, ie that is in my script.

Lots and lots of stuff changed. There are many breaking changes to the Lua API, so I would expect stuff to be broken, what's concerning is the stack overflow. Looking over your script, I don't immediately see anything that would cause something like that.

[garry23]

@dmilligan

As I said above, without knowing the 'changes', I just don't know how/where to start 'debugging'.

The main issue is my script freezes ML when I try and run it, ie not autoload, simply run it (with the error posted above).

This I find strange.

[a1ex]

Confirmed. What I find puzzling: the build from the website crashes, but my local compilation (from the same changeset) works just fine.

Edit: it was indeed an issue on the build server. Recompiled after cleaning the workspace and now it's working.

Guess it's high time adding some automatic tests to the nightly server (run each build in QEMU), to catch similar issues.

[garry23]

:-)))))))

In the nicest possible way, I'm glad it was such an error.

I was beginning to feel stupid: again! 

I'll try the latest as soon as I can and provide feedback.

Cheers

Garry

[garry23]

First strangeness connected with latest Lua fix.

My script generates an error "...lens.focus only works in LV...", but I'm in LV.

Obviously I need a hint here.

Cheers

Garry

[a1ex]

Are you actually seeing a live preview (updated at roughly 30 frames per second), or are you in some other camera mode derived from LiveView (such as Canon menu entered from LiveView, or quick review after taking a picture in LiveView) ?

The former is considered LiveView by ML, but the latter is not (even if the mirror may be raised). Focus commands are only known to work in LiveView.

[garry23]

@A1ex

I run my script explicitly from Live View.

As I said before, the script ran perfectly before the last experimental build.

Cheers

Garry

[a1ex]

Best guess: in check_lens_ready, you call lens.focus repeatedly until it succeeds.

Code Select Expand


repeat
msleep(200)
until lens.focus(0,1,false)


This call is not running in LiveView - you call this function right after camera.shoot(), so it actually waits for the camera to return to LiveView after taking a picture.

Before, lens.focus returned false outside LiveView; now it raises an error if the preconditions (LiveView and AF enabled) are not met. It still returns false when the lens could not be moved (for example, reached the limit). Was the previous behavior better?

To fix this error, you could try either waiting until lv.running (or even forcing it with lv.start(), which should cover long image review settings), or wrapping lens.focus with a pcall (to catch the error and to keep the logic unchanged).

[garry23]

@A1ex

Thanks for the insight.

I got my script to run by explicitly calling lv.start before every lens.focus: as a proof of principle.

The problem is the mirror slaps up and down now, ie I'm stacking and don't need/want the mirror to move.

Do you have any further thoughts?

Cheers

Garry