ML on EOS-M2

[critix]

I got prop_diag working as a stand alone app and was able to parse ROM1.BIN but am stuck trying to figure out how to parse the RAM4.BIN, especially, "...around the known address..." As Sergeant Schultz would say, "I know nothing!"

Starting from https://www.magiclantern.fm/forum/index.php?topic=12177.msg117735#msg117735 I tried to unblock RAM4.BIN. The problem is I do not know what offset to use.
I tried with 0xF0000000 and with 0xE0000000. The RAM4.BIN.strings file resulted. No matter what offset I've tried, the files are the same. Still running disassemble.pl, so I still do not have a result. I will return with a result when it is ready.

[Edit]
Maybe I got the offset wrong. Looking in the source, I found that RAM4 is saved with offset 0x10000000. So I run the script with offset 0x10000000. Let's see what's coming out.

[dfort]

@critix - Are you using prop_diag from the recovery branch? It is quite easy to build. The documentation explains how to enter the offsets but I'm not sure what offset to use.

src/prop_diag.c

Code: [Select]

/**
 * Property parsing code inspired from g3gg0's Property Editor and Indy's parse_prop.py,
 * but rewritten from scratch :)
 *
 * This tool can also be built as a standalone program, from the "src" directory:
 *    gcc prop_diag.c -o prop_diag
 * or:
 *    make prop_diag
 *
 * Python version available on request.
 *
 * Data structure: in ROM, properties are organized in blocks, sub-blocks and sub-sub-blocks.
 *
 * Each nesting level uses the same structure:
 * [status_1 size_1 data_1    status_2 size_2 data_2    ...    status_n size_n data_n    terminator]
 *
 * - "size" covers the size of "status", "size" and "data" fields, so "data" has size-8 bytes
 * - "data" is either an array of sub-blocks [subblock_1 subblock_2 ... subblock_n terminator]
 *    or the property data itself.
 * - "status", "size" and "terminator" are 32-bit integers each.
 *
 * Consistency check: sum of size_i == buffer_len - 4.
 * There are 4 nesting levels: let's call them "class", "group", "subgroup" and "property".
 *
 * There are multiple copies of various setting blocks, and only one of them is marked
 * as "active" (status = 0xFFFF); this is probably (just a guess) used for wear leveling
 * and maybe also to save a "last known good" configuration (or that's just a side effect).
 *
 * New models may have different header structure for the main blocks;
 * in particular, status and size are no longer the first items.
 */


[critix]

No, I use:

Code: [Select]

/disassemble.pl 0x1000000 RAM4.BIN [Edit]
If you use prop_diag with the camera, then in prop_diag you have to put the offset. If you use offline, then you do not need offset.
That's why I'm trying with that script.

[dfort]

Found something interesting. Going back to something Danne commented on in Reply #426 - the LiveView freezing in Photo mode doesn't happen when FPS override is set like this:



There are probably other settings that work but this is something that users can experiment with using a test build from my downloads page.

Ok--so why would you use FPS override while in Photo mode?

Tip: this feature also works in photo mode, making LiveView usable in dark environments. Combine it with display gain.

So is there anything here we can use to resolve the raw LiveView freezing issue?

[liquidether]

My post is not about the bits and bytes behind but my experience with ML on the M2 as a user. First of all, thanks so much for your work! Maybe what I've found can help you devs flush something out?

I've been testing mostly in RAW trying to use settings that seem to work for others on the M. The quirks I've noticed that might be meaningful relate to dfort's discovery of the FPS timer setting. I don't know how that was discovered but I've noticed in testing that if I do not configure the FPS timer A to 588 then I get a black band on the right side of video with my RAW settings. The band seems to vary in width depending on the frame rate I use although it is constant during any given recording.

My settings for this were as follows:

Modules loaded: mlv_lite, mlv_play, mlv_snd, sd_uhs, lua.

I then enable my M2 with "RAW video" and "FPS override". FPS override is set at 23.976 exact. RAW is set to a resolution of 1736x474 with a 1.61 crop, 14 bit lossless. I then do the 2.5k trick of going back to Canon's LiveView (LV) interface and zoom in to 5x. LV unfreezes (previously frozen) and I can view and record live motion. I go back to the ML menu and set RAW to record at 1972x872 with a 1:2 crop. This seems to work for me and I get usable footage. Looks great (MLV App with Chroma Smooth 3x3 to eliminate Focus Pixels)! However, on the right side is the banding I mentioned:



UNLESS:

I go back into the Canon LV mode and disable the 5x zoom then go into the FPS override and set the FPS timer A to the 588 mentioned. I have not tried other values to see if bands appear elsewhere if I go higher or lower. If I go back into the Canon LV and enable 5x zoom then my recordings no longer have the black band on the right. I've tried it with other resolutions and also with 24fps exact and the black band on the right is gone. I did notice that it seems to be precise amount above 588 and not 588 itself since the timer changes depending on frame rate and zoom ratio but stays a constant +64 above. 64 sounds like an address shift of some kind.

Another thing I've noticed is that the 5x zoom does not center the image. At least on mine it doesn't. Whatever I see on in LV is of course smaller than what I record but the resulting footage shows that the LV zoom, while centered on the screen (meaning the little positioning square is smack dab in the middle of its container), is off to the left of what it's recording. So much so that it looks like there's nothing recorded to the left of what I shot and double that to the right. Like this [x  ] where "x" is what I saw on LV and [] represent frame bounds. I would have expected [ x ].


The green dotted line is what I see in LV.

Maybe that's no help at all or maybe you've encountered similar or already know this. I'm happy to test anything you'd like but will need guidance if you want log files. Trying to go through the setup on my Mac for dev work but my machine is a Jenga tower of developer tools and something's not playing right and silently failing so I'll have to figure that one out before whatever the next step will be.

[dfort]

Funny how the M2 can do close to 4k raw video but can't save a still photo with ML loaded.

What I've been doing is to keep merging Danne's EOSM branch into my bleeding edge EOSM2 branch and posting test builds. It has been getting several downloads. Good to finally get some feedback.

The black bars and FPS override are settings that can probably be tweaked. Danne knows more about what to adjust than I do -- by the way he also has an M2.

As far as the focus pixels, I have added map files for the EOSM2. They are mostly theoretical but since most of the map files are the same between the various cameras I'm pretty sure they should work. You can download them from my ML Focus Pixels repository. Look in the focus_pixel_map_files directory, the M2 files start with 80000355. Put them next to the executable file in either MLVFS or MLV App. Check out the last several posts on the Dealing with Focus Pixels in raw video topic for more information.

[liquidether]

I have the development environment configured on my machine now, am able to build but it looks like I have several tutorials to run though to get me up to speed with the typical workflow of ML development. Currently my QEMU displays a blank black screen past the date/time screen although I can see the console recording keypresses. Might have to do with my Mac or version conflicts or a myriad of other possible incompatibilities.

However, I'm not sure if QEMU is even used at this stage of development. Are you building and testing directly on the M2?

Until I'm up to speed I can only report what I'm finding as a user attempting RAW video. I'm able to record 1800x1012 in 3x crop mode in 16:9 and all the way up to the 2.5k in 5x crop mode but the 2.5k has a significant number of purple frames and is unusable. I use a 64GB SanDisk Extreme Pro with the sd hack in both cases. If I lower the resolution to 1920x872 in 5x crop (I believe it's 1:2.35) then I can record continuously. I can do these in 14bit lossless. If I use 12bit lossless then every other frame is frozen (as in either black or a repeat of the very first frame of video).

The 1800x1012 in 3x crop mode is very finicky. If I start recording immediately after the sd_uhs script completes its autorun then I have been able to record continuously without issue. However, if I wait to record for some undetermined amount of time live view freezes up. The only way I can unfreeze live view is to toggle through the 4 view modes with the Info button. There are 2 Canon GUI modes, the ML GUI then the pure settings GUI mode. I have to cycle through the settings GUI mode back to the first Canon mode and then I'll see an unfrozen live view again. However, if I don't start recording immediately it will freeze again and I have to cycle through the screens to get it back. I can play this game only two times, unfortunately. The third time all of the view modes are a blank black screen.

I've tried flipping to camera mode and back to video but once I see a blank black live view mode I have to restart the camera for video work. Fortunately I can play this game all over again after rebooting.

If there are particular files with configurations that are tedious to build and test I'm happy to build and grunt test them locally but would need to know the files and rough plan. Otherwise it will be a while before I can contribute to the freezing issue.

[liquidether]

I know you mentioned M2 development is stalled due to some issues not replicable in QEMU so I'd like to see about getting some grease on my hands. I'm a developer but just a young Padawan in this realm. I've been reading through the developer guides and have tried to repeat the porting process in this tread but…why reinvent the wheel? Plus it's taken nearly two years of your effort to get this far and I'm missing things not found in the thread partly due to my environment being different.

For instance, I'm using Ubuntu 16LTS on a VirtualBox machine because my Mac is heavily configured for other development work and I don't want to accidentally break it for my paying work. I did get QEMU up and running on the Mac but got endless logs and a blank gray UI with a red dot in the lower right then it would crash. Actually took my Mac with it which I rarely ever see. I decided on some separation.

Following the guide here https://www.magiclantern.fm/forum/index.php?topic=991.0 for installing magiclantern I have a sandboxed environment to play in. Seems to work just fine. However, I see no EOSM2 in the platform directory. So I tried merging in your branch (hg merge crop_rec_4k_mlv_snd_isogain_1x3_presets_EOSM2_dfort) but that fails with an "unknown revision" error.

I tried from scratch using your version of the bleeding edge branch (hg clone https://bitbucket.org/daniel_fort/magic-lantern/branch/crop_rec_4k_mlv_snd_isogain_1x3_presets_EOSM2_dfort).

Renaming the resulting directory "magic-lantern" lets me complete the qemu setup process. However, if I update QEMU (hg up qemu -C) I again have no EOSM2 directory. If I don't update qemu and instead update the tip (hg update tip) then I get your EOSM2 directory in the platform folder.

I then create the gcc-arm path (PATH="$HOME/gcc-arm-none-eabi-5_4-2016q3/bin:$PATH") in .bashrc since it wouldn't work in .profile. After this I'm able to run the install.sh for qemu (the one in your branch) without a problem.

I then make magic-lantern in the ml EOSM2 directory (make clean && make install_qemu).

This updates the sd.img in the new qemu-eos directory (same level as magic-lantern directory). Seems like I'm good to go.

However, running QEMU with ./run_canon_fw.sh EOSM2 results in errors concerning the format of long unsigned int values. I've tried commenting out the code it references since they're just status updates but the more I comment, the more keep coming and I doubt this is how you're building it.

So I start over and build qemu exactly according to the instructions in the link above. Then I clone your branch as before, build the EOSM2 with make clean && make install_qemu and everything seems peachy keen. I copy my ROMs (ROM0, ROM1, SFDATA) into qemu's EOSM2 directory and again try ./run_canon_fw.sh EOSM2 but get the same gray screen I got on my Mac and it kills the virtual machine with the same endless loop of "WARN [I2C] localI2C_Write : 317 (Task : Startup)". It also spits out "[AUDIO] ERROR I2C abort 0(1020000)" repeatedly. If I kill the qemu window quickly enough then I can regain control.

I've tried various .run… commands but everything results in an endless loop or runs to a crash or runs to a lockup. I'm unable to get anything in QEMU. Since I only have the EOS M2 I can't test with any other build just to see what I'm doing wrong (no ROM files for anything else). I did order an M for comparison but it's not arrived yet.

So, I'd like to get up and running with testing in QEMU but I think I might be acting above my pay grade. I'm happy to take advice.

In the meantime I keep testing with your releases. As I understand it they are just updated syncs with the bleeding edge EOS M branch and have nothing to do with getting the M2 past its live view freezing issues. Is that a fair read?

[dfort]

@liquidether - Wow, best news for the EOSM2 in a long time.

Sorry I missed your March 13 post, got hit with other things to do--like being the President of the HOA, shooting a no-budget feature film, preparing for my tax appointment, making through my wife's honey do list, you get the picture. I'm also the least qualified of the ML users with "Developer" status. When I started porting the M2 I thought it would be a nice little introduction to a fairly easy camera to get ML working. Ha!

Back on topic. I'm also having problems getting the M2 working in QEMU. It should work and it used to work in past for me. Note that the current qemu branch is set up to work with the M2, though it is missing from the "platform" directory. I'll PM you with something that you can use to check your QEMU setup.

In the meantime I've been doing most of my testing on the camera. If you clone or fork my repository you'll see lots of experimental EOSM2 branches. I've got to update some of these, like the lua_fix branch that is finally working on the EOSM and might help with the M2.

It looks like you've got the M2 working about as well as anyone. That "bleeding edge" build I keep posting is just keeping up with Danne's latest EOSM changes. It happens to be the most stable build at this time though I don't quite understand why that's so.

[dfort]

As EOSM2 Magic Lantern users (and abusers) know, I've been merging in Danne's EOSM experiments to my almost but not quite yet working EOSM2 bleeding edge branch and posting test builds. Today I merged in the latest lua_fix changes and tried running the Script API Tests. It turned up a problem in the menu test when trying to change the FPS override. Tried changing it manually and there does seem to be something weird going on.

Code: [Select]

Testing ML menu API...

ML/SCRIPTS/API_TEST.LUA:411: assertion failed!
stack traceback:
[C]: in function 'assert'
ML/SCRIPTS/API_TEST.LUA:411: in function 'test_menu'
ML/SCRIPTS/API_TEST.LUA:1573: in function <ML/SCRIPTS/API_TEST.LUA:1564>
[C]: in function 'xpcall'
ML/SCRIPTS/API_TEST.LUA:1564: in function 'api_tests'
ML/SCRIPTS/API_TEST.LUA:1603: in main chunk
I commented out test_menu() and got an error on the test_camera_take_pics(). This was a strange one because it asked to switch to M mode when the camera was already in manual mode.

Code: [Select]

Testing picture taking functions...
Please switch to M mode.
Snap simulation test...
Single picture...

ML/SCRIPTS/API_TEST.LUA:1029: attempt to call a nil value (method 'image_path')
stack traceback:
ML/SCRIPTS/API_TEST.LUA:1029: in function 'test_camera_take_pics'
ML/SCRIPTS/API_TEST.LUA:1574: in function <ML/SCRIPTS/API_TEST.LUA:1564>
[C]: in function 'xpcall'
ML/SCRIPTS/API_TEST.LUA:1564: in function 'api_tests'
ML/SCRIPTS/API_TEST.LUA:1603: in main chunk
Next issue, test_lv():

Code: [Select]

Testing module 'lv'...
LiveView is running; stopping...

ML/SCRIPTS/API_TEST.LUA:1240: assertion failed!
stack traceback:
[C]: in function 'globals.assert'
ML/SCRIPTS/API_TEST.LUA:1240: in function 'globals.test_lv'
ML/SCRIPTS/API_TEST.LUA:1578: in function <ML/SCRIPTS/API_TEST.LUA:1564>
[C]: in function 'globals.xpcall'
ML/SCRIPTS/API_TEST.LUA:1564: in function 'globals.api_tests'
ML/SCRIPTS/API_TEST.LUA:1603: in main chunk
So much for the tests that aren't working. Here is what is working:

Code: [Select]

===============================================================================
ML/SCRIPTS/API_TEST.LUA - 2019-3-23 12:38:09
===============================================================================

Strict mode tests...
Strict mode tests passed.

Generic tests...
arg = table:
  [0] = "API_TEST.LUA"
camera = table:
  shutter = table:
    raw = 101
    apex = 5.625
    ms = 20
    value = 0.020263
  aperture = table:
    raw = 48
    apex = 5.
    value = 5.6
    min = table:
      raw = 37
      apex = 3.625
      value = 3.5
    max = table:
      raw = 80
      apex = 9.
      value = 22.6
  iso = table:
    raw = 104
    apex = 9.
    value = 1600
  ec = table:
    raw = 0
    value = 0
  flash = "auto"
  flash_ec = table:
    raw = 0
    value = 0
  kelvin = 6500
  mode = 3
  metering_mode = 3
  drive_mode = 0
  model = "Canon EOS M2"
  model_short = "EOSM2"
  firmware = "1.0.3"
  temperature = 196
  gui = table:
    menu = false
    play = false
    play_photo = false
    play_movie = false
    qr = false
    idle = true
  burst = function: p
  reboot = function: p
  shoot = function: p
  wait = function: p
  bulb = function: p
event = table:
  pre_shoot = nil
  post_shoot = nil
  shoot_task = nil
  seconds_clock = nil
  keypress = nil
  custom_picture_taking = nil
  intervalometer = nil
  config_save = nil
console = table:
  hide = function: p
  write = function: p
  clear = function: p
  show = function: p
lv = table:
  enabled = true
  paused = false
  running = true
  zoom = 1
  overlays = 2
  info = function: p
  wait = function: p
  pause = function: p
  resume = function: p
  start = function: p
  stop = function: p
lens = table:
  name = "EF28-105mm f/3.5-4.5 USM"
  focal_length = 28
  focus_distance = 3900
  hyperfocal = 7424
  dof_near = 2582
  dof_far = 8093
  af = true
  af_mode = 0
  autofocus = function: p
  focus = function: p
display = table:
  idle = nil
  height = 480
  width = 720
  notify_box = function: p
  on = function: p
  off = function: p
  draw = function: p
  load = function: p
  line = function: p
  clear = function: p
  circle = function: p
  pixel = function: p
  rect = function: p
  screenshot = function: p
  print = function: p
key = table:
  last = 9
  press = function: p
  wait = function: p
menu = table:
  visible = false
  open = function: p
  new = function: p
  get = function: p
  select = function: p
  close = function: p
  block = function: p
  set = function: p
movie = table:
  recording = false
  start = function: p
  stop = function: p
dryos = table:
  clock = 13
  ms_clock = 13556
  image_prefix = "IMG_"
  dcim_dir = table:
    exists = true
    create = function: p
    children = function: p
    files = function: p
    parent = table:
      exists = true
      create = function: p
      children = function: p
      files = function: p
      parent = table:
        exists = true
        create = function: p
        children = function: p
        files = function: p
        parent = nil
        path = "B:/"
      path = "B:/DCIM/"
    path = "B:/DCIM/100CANON/"
  config_dir = table:
    exists = true
    create = function: p
    children = function: p
    files = function: p
    parent = table:
      exists = true
      create = function: p
      children = function: p
      files = function: p
      parent = table:
        exists = true
        create = function: p
        children = function: p
        files = function: p
        parent = nil
        path = "B:/"
      path = "ML/"
    path = "ML/SETTINGS/"
  ml_card = table:
    cluster_size = 32768
    drive_letter = "B"
    file_number = 1560
    folder_number = 100
    free_space = 31112320
    type = "SD"
    path = "B:/"
    _card_ptr = userdata
  shooting_card = table:
    cluster_size = 32768
    drive_letter = "B"
    file_number = 1560
    folder_number = 100
    free_space = 31112320
    type = "SD"
    path = "B:/"
    _card_ptr = userdata
  date = table:
    year = 2019
    isdst = false
    day = 23
    yday = 82
    wday = 7
    min = 38
    sec = 11
    month = 3
    hour = 12
  directory = function: p
  rename = function: p
  remove = function: p
  call = function: p
interval = table:
  time = 10
  count = 0
  running = false
  stop = function: p
battery = table:
function not available on this camera
stack traceback:
[C]: in ?
[C]: in for iterator 'for iterator'
ML/SCRIPTS/LIB/logger.lua:125: in function 'logger.serialize'
ML/SCRIPTS/API_TEST.LUA:45: in function <ML/SCRIPTS/API_TEST.LUA:44>
[C]: in function 'globals.xpcall'
ML/SCRIPTS/API_TEST.LUA:44: in function 'globals.print_table'
ML/SCRIPTS/API_TEST.LUA:90: in function 'globals.generic_tests'
ML/SCRIPTS/API_TEST.LUA:1568: in function <ML/SCRIPTS/API_TEST.LUA:1564>
[C]: in function 'globals.xpcall'
ML/SCRIPTS/API_TEST.LUA:1564: in function 'globals.api_tests'
ML/SCRIPTS/API_TEST.LUA:1603: in main chunktask = table:
  create = function: p
  yield = function: p
property = table:
Generic tests completed.

Module tests...
Testing file I/O...
Copy test: autoexec.bin -> tmp.bin
Copy test OK
Append test: tmp.txt
Append test OK
Rename test: apple.txt -> banana.txt
Rename test OK
Rename test: apple.txt -> ML/banana.txt
Rename test OK
File I/O tests completed.

Testing Canon GUI functions...
Enter MENU mode...
Enter PLAY mode...
Enter PLAY mode...
Enter MENU mode...
Enter MENU mode...
Enter PLAY mode...
Exit PLAY mode...
Enter MENU mode...
Exit MENU mode...
Pause LiveView...
Enter MENU mode...
Enter MENU mode...
Enter MENU mode...
Enter MENU mode...
Exit MENU mode...
Enter PLAY mode...
Enter PLAY mode...
Enter PLAY mode...
Enter PLAY mode...
Enter PLAY mode...
Exit PLAY mode...
Enter PLAY mode...
Enter MENU mode...
Enter MENU mode...
Enter MENU mode...
Enter PLAY mode...
Enter MENU mode...
Exit MENU mode...
Enter PLAY mode...
Enter PLAY mode...
Enter MENU mode...
Exit MENU mode...
Pause LiveView...
Resume LiveView...
Stop LiveView...
Enter PLAY mode...
Exit PLAY mode...
Stop LiveView...
Enter PLAY mode...
Enter PLAY mode...
Enter MENU mode...
Exit MENU mode...
Stop LiveView...
Start LiveView...
Pause LiveView...
Resume LiveView...
Enter PLAY mode...
Exit PLAY mode...
Pause LiveView...
Enter MENU mode...
Enter PLAY mode...
Enter PLAY mode...
Enter PLAY mode...
Enter PLAY mode...
Exit PLAY mode...
Pause LiveView...
Enter MENU mode...
Enter MENU mode...
Enter MENU mode...
Enter MENU mode...
Enter MENU mode...
Enter PLAY mode...
Enter PLAY mode...
Enter PLAY mode...
Enter PLAY mode...
Enter MENU mode...
Exit MENU mode...
Pause LiveView...
Resume LiveView...
Enter PLAY mode...
Exit PLAY mode...
Pause LiveView...
Resume LiveView...
Enter PLAY mode...
Exit PLAY mode...
Pause LiveView...
Resume LiveView...
Canon GUI tests completed.

Testing multitasking...
Only one task allowed to interrupt...
Main task yielding.
Task C started.
Task C finished.
Main task back.
Main task yielding.
Task C started.
Task C finished.
Main task back.
Main task yielding.
Task C started.
Task C finished.
Main task back.
Main task yielding.
Task C started.
Task C finished.
Main task back.
Main task yielding.
Task C started.
Task C finished.
Main task back.
Main task yielding.
Task C started.
Task C finished.
Main task back.
Main task yielding.
Task C started.
Task C finished.
Main task back.
Main task yielding.
Task C started.
Task C finished.
Main task back.
Main task yielding.
Task C started.
Task C finished.
Main task back.
Main task yielding.
Task C started.
Task C finished.
Main task back.
Multitasking tests completed.

Testing half-shutter...
Half-shutter test OK.


Testing lens focus functionality...
Focus distance: 3900
Autofocus in LiveView...
Please trigger autofocus (half-shutter / AF-ON / * ).
59...Autofocus triggered.
Autofocus completed.
Focus distance: 6820
Focusing backward...
Focus distance: 655350
Focus motor position: 10533
Focusing forward with step size 3, wait=true...
........
Focus distance: 470
Focus motor position: 293
Focusing backward with step size 3, wait=true...
....
Focus distance: 655350
Focus motor position: 10533
Focus range: 8 steps forward, 4 steps backward.
Motor steps: 10240 forward, 10240 backward, 0 lost.
Focusing forward with step size 3, wait=false...
..............................
Focus distance: 470
Focus motor position: 293
Focusing backward with step size 3, wait=false...
......................
Focus distance: 655350
Focus motor position: 10533
Focus range: 30 steps forward, 22 steps backward.
Motor steps: 10240 forward, 10240 backward, 0 lost.
Focusing forward with step size 2, wait=true...
.........................
Focus distance: 470
Focus motor position: 293
Focusing backward with step size 2, wait=true...
.....................
Focus distance: 655350
Focus motor position: 10533
Focus range: 25 steps forward, 21 steps backward.
Motor steps: 10240 forward, 10240 backward, 0 lost.
Focusing forward with step size 2, wait=false...
...............................................................................
Focus distance: 470
Focus motor position: 293
Focusing backward with step size 2, wait=false...
......................................................................
Focus distance: 655350
Focus motor position: 10533
Focus range: 79 steps forward, 70 steps backward.
Motor steps: 10240 forward, 10240 backward, 0 lost.

Focus test completed.

Testing exposure settings...
Camera    : Canon EOS M2 (EOSM2) 1.0.3
Lens      : EF28-105mm f/3.5-4.5 USM
Shoot mode: 3
Shutter   : �50 (raw 101, 0.020263s, 20ms, apex 5.625)
Aperture  : �5.6 (raw 48, f/5.6, apex 5.)
Av range  : �3.5..�22 (raw 37..80, f/3.5..f/22.6, apex 3.625..9.)
ISO       : �1600 (raw 104, 1600, apex 9.)
EC        : 0.0 (raw 0, 0 EV)
Flash EC  : 0.0 (raw 0, 0 EV)
Setting shutter to random values...
Setting ISO to random values...
Setting aperture to random values...
Please switch to Av mode.
Setting EC to random values...
Setting Flash EC to random values...
Exposure tests completed.


Testing movie recording...
Please switch to Movie mode.
Movie recording tests completed.

Done!

Sometimes I feel like I'm trying to start an old car. Pull battery, pull card, reinsert battery, start camera without card, reinsert card... but hey, this is still a new port even though this topic was started back in September 2015.

In any case, these tests should point to areas that still need some more work. Any help would be greatly appreciated.

[dfort]

Finally got the EOSM2 working in QEMU. It has been a while and I thought things were broken but they are working. It is hard as ... to get to the ML menus on this camera in QEMU and I wanted to see if I could run the lua tests so I ended up copying over the EOSM settings with the tests on autorun. Looks like it is working.

Here's what the QEMU console shows when running the strict_tests() and generic_tests():

Code: [Select]

[ module_task:00b009ec ] task_create(lua_load_task, prio=1c, stack=10000, entry=b02190, arg=0)
[MPU] Received: 06 05 03 07 60 00  (unknown - PROP_BURST_COUNT)
[MPU] Received: 06 05 03 11 02 00  (unknown - PROP_ICU_AUTO_POWEROFF)
[lua_load_task:ff14a4f0 ] register_interrupt(null, 0x94, 0xff14a344, 0x7)
[lua_load_task:ff14a4f0 ] register_interrupt(null, 0x94, 0xff14a344, 0x7)

Note that nothing is showing up on the QEMU gui but these tests did seem to complete successfully:

LUATEST.LOG

Code: [Select]

===============================================================================
ML/SCRIPTS/API_TEST.LUA - 2017-9-30 04:15:00
===============================================================================

Strict mode tests...
Strict mode tests passed.

Generic tests...
arg = table:
  [0] = "API_TEST.LUA"
camera = table:
  shutter = table:
    raw = 80
    apex = 3.
    ms = 125
    value = 0.125
  aperture = table:
    raw = 61
    apex = 6.625
    value = 9.900001
    min = table:
      raw = 41
      apex = 4.125
      value = 4.1
    max = table:
      raw = 84
      apex = 9.5
      value = 26.899999
  iso = table:
    raw = 72
    apex = 5.
    value = 100
  ec = table:
    raw = 0
    value = 0
  flash = true
  flash_ec = table:
    raw = 0
    value = 0
  kelvin = 4600
  mode = 3
  metering_mode = 3
  drive_mode = 0
  model = "Canon EOS M2"
  model_short = "EOSM2"
  firmware = "1.0.3"
  temperature = 161
  gui = table:
    menu = false
    play = false
    play_photo = false
    play_movie = false
    qr = false
    idle = false
  bulb = function: p
  shoot = function: p
  burst = function: p
  wait = function: p
  reboot = function: p
event = table:
  pre_shoot = nil
  post_shoot = nil
  shoot_task = nil
  seconds_clock = nil
  keypress = nil
  custom_picture_taking = nil
  intervalometer = nil
  config_save = nil
console = table:
  hide = function: p
  write = function: p
  show = function: p
  clear = function: p
lv = table:
  enabled = true
  paused = false
  running = true
  zoom = 1
  overlays = false
  info = function: p
  pause = function: p
  resume = function: p
  wait = function: p
  stop = function: p
  start = function: p
lens = table:
  name = "EF-S18-55mm f/3.5-5.6 IS STM"
  focal_length = 0
  focus_distance = 90
  hyperfocal = 0
  dof_near = 0
  dof_far = 0
  af = true
  af_mode = 0
  autofocus = function: p
  focus = function: p
display = table:
  idle = nil
  height = 480
  width = 720
  print = function: p
  clear = function: p
  rect = function: p
  line = function: p
  circle = function: p
  draw = function: p
  load = function: p
  pixel = function: p
  notify_box = function: p
  off = function: p
  screenshot = function: p
  on = function: p
key = table:
  last = 0
  press = function: p
  wait = function: p
menu = table:
  visible = false
  close = function: p
  set = function: p
  select = function: p
  block = function: p
  get = function: p
  open = function: p
  new = function: p
movie = table:
  recording = false
  start = function: p
  stop = function: p
dryos = table:
  clock = 3
  ms_clock = 3404
  image_prefix = "IMG_"
  dcim_dir = table:
    exists = true
    create = function: p
    children = function: p
    files = function: p
    parent = table:
      exists = true
      create = function: p
      children = function: p
      files = function: p
      parent = table:
        exists = true
        create = function: p
        children = function: p
        files = function: p
        parent = nil
        path = "B:/"
      path = "B:/DCIM/"
    path = "B:/DCIM/100CANON/"
  config_dir = table:
    exists = true
    create = function: p
    children = function: p
    files = function: p
    parent = table:
      exists = true
      create = function: p
      children = function: p
      files = function: p
      parent = table:
        exists = true
        create = function: p
        children = function: p
        files = function: p
        parent = nil
        path = "B:/"
      path = "ML/"
    path = "ML/SETTINGS/"
  ml_card = table:
    cluster_size = 32768
    drive_letter = "B"
    file_number = 1395
    folder_number = 100
    free_space = 433216
    type = "SD"
    path = "B:/"
    _card_ptr = userdata
  shooting_card = table:
    cluster_size = 32768
    drive_letter = "B"
    file_number = 1395
    folder_number = 100
    free_space = 433216
    type = "SD"
    path = "B:/"
    _card_ptr = userdata
  date = table:
    month = 9
    year = 2017
    sec = 0
    wday = 7
    day = 30
    isdst = false
    hour = 4
    yday = 273
    min = 15
  remove = function: p
  rename = function: p
  call = function: p
  directory = function: p
interval = table:
  time = 10
  count = 0
  running = false
  stop = function: p
battery = table:
function not available on this camera
stack traceback:
[C]: in ?
[C]: in for iterator 'for iterator'
ML/SCRIPTS/LIB/logger.lua:125: in function 'logger.serialize'
ML/SCRIPTS/API_TEST.LUA:45: in function <ML/SCRIPTS/API_TEST.LUA:44>
[C]: in function 'xpcall'
ML/SCRIPTS/API_TEST.LUA:44: in function 'print_table'
ML/SCRIPTS/API_TEST.LUA:90: in function 'generic_tests'
ML/SCRIPTS/API_TEST.LUA:1568: in function <ML/SCRIPTS/API_TEST.LUA:1564>
[C]: in function 'xpcall'
ML/SCRIPTS/API_TEST.LUA:1564: in function 'api_tests'
ML/SCRIPTS/API_TEST.LUA:1603: in main chunktask = table:
  create = function: p
  yield = function: p
property = table:
Generic tests completed.

Module tests...
Done!

Now my question is, is this a valid test or am I wasting time? I was planning on going through each test one at a time to see if I can get a log and maybe the QEMU console will show some issues on the tests that are failing. I'm not sure if the tests that are failing can be run in QEMU so I was going to run them on the EOSM to see if that camera can complete the tests in QEMU.

These are the tests that the EOSM2 is having problems with and the EOSM is passing -- when run in camera:

Code: [Select]

test_menu()
test_camera_take_pics()
test_lv()
It is getting late and this is time consuming so I'm just throwing this out there before getting some shut eye. Planning on giving this a go in the morning.

[a1ex]

It is hard as ... to get to the ML menus on this camera in QEMU

It still works fine over here, and menu navigation is covered by the test suite for quite some time (i.e. tested after every group of commits). For example, latest log:

Code: [Select]

Testing Canon menu...
  EOSM2: ................................. OK

Key press sequence for that test:

Code: [Select]

MENU_SEQUENCE[EOSM2]="m space space space up up space m up space m up space m up space m right space space m m m l i q q 9 0 space i i m" # starts in LiveView

Screenshots resulted from that test:


It's working on the Mac VM, too.

Some time ago I've reproduced your issue by swapping SFDATA.BIN from another camera (100D). Make sure it's not that; the first word in SFDATA.BIN is the model ID of the camera (0x80000355 for M2, 0x80000346 for 100D). Then, there comes a version number (6.0.3 for M2, 4.2.1 for 100D). If the SFDATA.BIN is the correct one, send me your ROMs (including SFDATA) so I can reproduce it.

Now my question is, is this a valid test or am I wasting time?

Generally speaking, if the log from emulator matches what you'd get on real hardware, it's probably a valid test. Things like file I/O, DryOS task switches or general-purpose computations are working very well. Once you get into LiveView, or photo capture, or even a plain half-shutter press in some cases, that's where things are starting to break.

If the log doesn't match, that usually indicates incorrect emulation (either bug in the emulator, or just not yet implemented). Some tests might be nondeterministic, for example the DryOS context switch order is no longer to match the one from real hardware, as it's not deterministic. You won't get the same order on real hardware either. There may be small differences between the two environments (such as, different card size or free space available), but these are already nitpicks.

A better way to debug these tests is to look at the line that failed:

Code: [Select]

ML/SCRIPTS/API_TEST.LUA:1240: assertion failed!
api_test.lua:1240    assert(lv.vidmode == "PH-NOLV")
lua_lv.c -> look for "vidmode" -> get_video_mode_name()
->         !is_movie_mode() && display_idle()          ? "PH-NOLV"  :      /* Regular photo mode outside LV (ready to take a picture) */

Now, "all" you have to do is to check if both return 1 in what the script expects to be "photo mode", i.e. after getting out of LiveView via lv.stop(). That function calls close_liveview(). On EOS M, it does this:

Code: [Select]

#ifdef CONFIG_EOSM
    {
        /* To shut off LiveView switch to the info screen */
        SetGUIRequestMode(21);
        msleep(1000);
    }
#else
    # other ways to exit LiveView, for other models
#endif

It's probably commit 085b79c2b (the info/shooting screen is no longer running as GUISTATE_IDLE on the M series, unlike regular DSLRs). The same issue might be present on all other mirrorless models, so maybe it's time to define CONFIG_MIRRORLESS (assuming M50 and R/RP have the same quirk).

[dfort]

Thanks for the tips to track down the last few hiccups on this camera.

I have no problems navigating through the Canon menus and the emulation seems pretty good. The frustrating part is getting to the ML menus. Not that it is impossible, just very difficult. At least I haven't found a reliable way to open them in QEMU. No issues on the actual hardware.



This is what I've got:

src/movtweaks.c

Code: [Select]

void close_liveview()
{
    if (lv)
#if defined(CONFIG_EOSM) || defined(CONFIG_EOSM2)
    {
        /* To shut off LiveView switch to the info screen */
        SetGUIRequestMode(21);
        msleep(1000);
    }
#else

Maybe I shouldn't assume that the EOSM2 is using the same SetGUIRequestMode codes as the EOSM? This is still a learning processes for me.

Back to running the lua tests in QEMU.

[dfort]

I keep coming up with a strange issue where the lua test is asking to switch to M mode when the (virtual) camera is already in M mode.



Maybe it has something to do with the rather odd double photo modes switch on the EOSM2? Note that there are two photo settings while the EOSM has just one:



The #2 is the one I have set for manual mode and seems to be the one that QEMU displays in the gui but maybe it is actually looking at the #1 settings which are not in M mode?

[dfort]

So much for the dual photo mode switch on the M2, the EOSM also has a problem figuring out what mode it is in when running the lua tests in QEMU.



It also can't verify whether or not it is in movie mode.



So this isn't just an EOSM2 emulation issue.

BTW -- On the EOSM2 the still photo mode #1 is "Basic Zone Modes" where Manual mode is not possible and #2 is "Creative Zones Modes" and is where you'll find the Manual mode.

[CuepublicStudios]

Following, I'm a total potato at coding, but ill send positive vibes? I just picked up an M2 for a killer deal and would love to make it my main shooter!

[Danne]

@alex
Could I ask you for what this is doing in state-object.c?

Code: [Select]

    #ifdef EVF_STATE
        stateobj_start_spy(EVF_STATE, stateobj_lv_spy);
    #endif
Clearly affects liveview causing freezes when RAW video is enabled. Uncommenting and no freeze. Not able to record as well obviously when not enabled.
Seems mentioned here as well:
https://www.magiclantern.fm/forum/index.php?topic=15895.msg187795;topicseen#msg187795

[a1ex]

This installs a hook in Canon's "Evf" task, allowing our code to run before and/or after each state transition.

Among others, this triggers the VSync hooks, i.e. functions called once for every LiveView frame. One of them is raw_lv_vsync, which you have found it may be related to the crash.

That could be either some incorrect stub called by that function, or some EDMAC transfer writing data into unallocated memory, or something along these lines.

EDMAC transfers appear to work; they were reported to pass the stubs API test. You could also try running the EDMAC screen test from selftest.mo, but I believe it will work.

Memory issues are a little harder to debug, but not impossible. I don't have a shortcut, but I'll be available this week starting Wednesday - if we can sync on the chat, we can debug this one together. It would probably take a couple of hours, or maybe more.

[Danne]

EDMAC transfers appear to work; they were reported to pass the stubs API test. You could also try running the EDMAC screen test from selftest.mo, but I believe it will work.

Memory issues are a little harder to debug, but not impossible. I don't have a shortcut, but I'll be available this week starting Wednesday - if we can sync on the chat, we can debug this one together. It would probably take a couple of hours, or maybe more.

Can´t say no to that offer. Friday or saturday would work here so let´s see if we can sync(); here. I´ll pm later.

Fast test:
Selftest.mo seems not to happy with ot without RAW video enabled. Two examples:

Code: [Select]

ML ASSERT:
dst == UNCACHEABLE(dst)
at ../../src/edmac-memcpy.c:142 (edmac_copy_rectangle_cbr_start), task run_test
lv:1 mode:0

run_test stack: 233af8 [233c50-22bc50]
0x0048DDA4 @ b4f880:233c00
0x0048DD40 @ 48dddc:233be0
0x0048DAD0 @ 48dd90:233bb0
0x0044C9AC @ 48db50:233b28
0x0044C478 @ 44ca08:233af8

Magic Lantern version : Nightly.2019May27.EOSM2103
Mercurial changeset   : 15f42e1abbf2+ (crop_rec_4k_mlv_snd_isogain_1x3_presets_EOSM2)
Built on 2019-05-27 13:35:28 UTC by danne@danands-MacBook-Pro.local.
Free Memory  : 372K + 1370K
Another:

Code: [Select]

ML ASSERT:
dst == UNCACHEABLE(dst)
at ../../src/edmac-memcpy.c:142 (edmac_copy_rectangle_cbr_start), task run_test
lv:0 mode:0

run_test stack: 233af8 [233c50-22bc50]
0x0048DDA4 @ b4f880:233c00
0x0048DD40 @ 48dddc:233be0
0x0048DAD0 @ 48dd90:233bb0
0x0044C9AC @ 48db50:233b28
0x0044C478 @ 44ca08:233af8

Magic Lantern version : Nightly.2019May27.EOSM2103
Mercurial changeset   : 15f42e1abbf2+ (crop_rec_4k_mlv_snd_isogain_1x3_presets_EOSM2)
Built on 2019-05-27 13:35:28 UTC by danne@danands-MacBook-Pro.local.
Free Memory  : 372K + 1384K

hm:

Code: [Select]

    /* make sure we are writing to uncacheable memory */
    ASSERT(dst == UNCACHEABLE(dst));
Branch used atm, mainly dfort ports:
https://bitbucket.org/Dannephoto/magic-lantern/branch/crop_rec_4k_mlv_snd_isogain_1x3_presets_EOSM2


EDIT:
Hm, getting similar results with my eosm so maybe not indicating anything here:

Code: [Select]

ML ASSERT:
dst == UNCACHEABLE(dst)
at ../../src/edmac-memcpy.c:139 (edmac_copy_rectangle_cbr_start), task run_test
lv:0 mode:3

run_test stack: 213d18 [213e70-20be70]
0x000E0CF4 @ b4acd0:213e20
0x000E0C90 @ e0d2c:213e00
0x000E0A20 @ e0ce0:213dd0
0x0009EC30 @ e0aa0:213d48
0x0009E558 @ 9ec8c:213d18

Magic Lantern version : Nightly.2019May27.EOSM202
Mercurial changeset   : 759e482088c8 (crop_rec_4k_mlv_snd_isogain_1x3_presets) tip
Built on 2019-05-27 09:22:45 UTC by danne@danands-MacBook-Pro.local.
Free Memory  : 215K + 3213K

[dfort]

For those EOSM2 users who are checking out the progress and trying out the builds I've been posting -- have you noticed things are getting better? It still has a ways to go but those gnarly screen freezes are gone and Danne has even started playing around with some advanced crop_rec settings. Full resolution silent pictures are still a problem but hey, one step at a time.

[garry23]

@dfort

Waiting like a praying mantis for you video guys to sort things out for us Neanderthal-like, knuckle dragging still photography types 

[Danne]

Yes a1ex of course guiding, troublechecking.
Got mv1080p rewired mode working but still a hackish state but fully capable of producing raw recordings.

[dfort]

Waiting like a praying mantis for you video guys to sort things out for us Neanderthal-like, knuckle dragging still photography types 

LOL!

Did you know that I was a still photographer from about 1973 through 1993 then worked in motion picture post production until just a couple years ago?

This is one of my cameras back when I was a photographer:



This is what a1ex and I were looking at today:



How things have changed!

We now return to your regularly scheduled forum topic.

[garry23]

 

[BlueDoo]

Joining the testing team today ! Just tested Danne's latest build:
- Anamorphic unsqueeze seems to work;
-Having trouble with both RAW modules, even in crope mode, frame freezes during recording. The result is a pretty good first frame, but then we have glitches flickering and some times the first frame coming back. No error message from ML.
-Noticing different glitch patterns between the different modes.
-Liveview sometimes freeze, esoecially when activating RAW shooting.
-Made some benchmarks: getting around 54mb/s with an Integral SD card that can write up to 100mb/s (80-90 in reality, method 4 for memory allocating, didn't try other modes for the moment)
-Had the console popping and not going away when skipped a frame during RAW shooting.
Will make more tests and note different errors.

PS: Tried different methods to push the recording resolution to its limits but couldn't get past 1800p.