Author Topic: How to run Magic Lantern into QEMU?!...  (Read 139090 times)

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12181
  • Maintenance mode
Re: How to run Magic Lantern into QEMU?!...
« Reply #400 on: April 09, 2019, 02:25:06 PM »
Narrowed down a bunch of bugs in the logging code, i.e. edge and corner cases when following DryOS or VxWorks task switches. Commit.

Whew, this was hard! I kept bumping into all sorts of edge cases... but hey, it finally passes the tests!!! I can finally follow all those task switches!

New feature: print number of instructions executed by each function (-d calls,tasks -icount 5). Very useful for code optimization - I actually wrote it one month ago for profiling the drawing code for DIGIC 6/7/8, in particular for helping chris_overseas with the Hello World code for 5D4. It was actually this feature (i.e. a simple assertion) that revealed all the above bugs. Commit.

Example - profiling Hello World on 60D.111:
Code: [Select]
call 0x1FE0FE18 bmp_printf(40201, 32, 32, 1fe589da "Hello, World!")             at [ml_init:1fe008b0:ff07b8d0] (my_big_init_task)
  call 0x1FE56018 __vsnprintf_veneer(1660b0, 7f, 1fe589da "Hello, World!", 166140)
                                                                                 at [ml_init:1fe0fe44:1fe008b4] (bmp_printf)
   -> 0xFF1DA530 vsnprintf                                                       at [ml_init:1fe56018:1fe0fe48] (__vsnprintf_veneer)
  return d to 0x1FE0FE48                                                         at [ml_init:ff1da8dc:1fe008b4]                    [icount 141]
  call 0x1FE0FD80 bmp_puts(40201, 1660a4, 1660a0, 1660b0 "Hello, World!")        at [ml_init:1fe0fe58:1fe008b4] (bmp_printf)
   call 0x1FE11D20 rbf_draw_string(96868, 32, 32, 1660b0 "Hello, World!")        at [ml_init:1fe0fdf8:1fe0fe5c] (bmp_puts)
    call 0x1FE114F8 rbf_draw_char(96868, 32, 32, 48)                             at [ml_init:1fe11d88:1fe0fdfc] (rbf_draw_string)
    ...
    return 15 to 0x1FE11D8C                                                      at [ml_init:1fe11678:1fe0fdfc] (rbf_draw_char)    [icount 12905]
    ...
    return 9 to 0x1FE11D8C                                                       at [ml_init:1fe11678:1fe0fdfc] (rbf_draw_char)    [icount 5993]
   return b8 to 0x1FE0FDFC                                                       at [ml_init:1fe121f0:1fe0fe5c] (rbf_draw_string)  [icount 116740]
  return b8 to 0x1FE0FE5C                                                        at [ml_init:1fe0fe0c:1fe008b4] (bmp_puts)         [icount 116776]
 return b8 to 0x1FE008B4                                                         at [ml_init:1fe0fe68:ff07b8d0] (bmp_printf)       [icount 116938]

 call 0x1FE0D620 info_led_blink(1, 1f4, 1f4, 218)                                at [ml_init:1fe008d8:ff07b8d0] (my_big_init_task)
  call 0x1FE56020 __msleep_veneer(1f4, 1f4, 1f4, 218)                            at [ml_init:1fe0d650:1fe008dc] (info_led_blink)
   -> 0xFF06EA08 msleep                                                          at [ml_init:1fe56020:1fe0d654] (__msleep_veneer)
   ... (other tasks) ...
 return 0 to 0x1FE008DC                                                          at [ml_init:1fe0d668:ff07b8d0] (info_led_blink)   [icount 542]

Profiling minimal Hello World on 5D4 (doesn't work out of the box yet):
Code: [Select]
call 0x1CCCB0 font_draw(64, 4b, 1, 3)                                           at [run_test:1cca48:80001737] (hello_world)
  arg5 = 1d09a4 "Hello, World!"                                                  at [run_test:1ccccc:1cca4c] (font_draw)
  call 0x1CCAA8 disp_set_pixel(64, 4b, 1, 1ccaa8 disp_set_pixel)                 at [run_test:1ccd70:1cca4c] (font_draw)
   call 0x1CD0F0 rgb2yuv422(ff, ff, ff, 1cd0f0 rgb2yuv422)                       at [run_test:1ccb00:1ccd74] (disp_set_pixel)
    -> 0x1CCF6C rgb2yuv422_rec709                                                at [run_test:1cd0f0:1ccb04] (rgb2yuv422)
   return fe80fe80 to 0x1CCB04                                                   at [run_test:1cd024:1ccd74] (rgb2yuv422_rec709)   [icount 48]
  return 80fe80 to 0x1CCD74                                                      at [run_test:1ccb5c:1cca4c] (disp_set_pixel)      [icount 89]
...
 return 0 to 0x1CCA4C                                                            at [run_test:1ccd8c:80001737] (font_draw)         [icount 555882]

It reveals how many instructions are required for the entire function, for a single pixel, where various overheads come from, and so on. Very useful for me.

I'd also like to clean it up a bit, make sure it works on all major operating systems (anyone had trouble with the install script?) and merge the current state into mainline, as pretty much all recent developments depend on this.

scrax

  • Contributor
  • Hero Member
  • *****
  • Posts: 1538
  • Code monkey
Re: How to run Magic Lantern into QEMU?!...
« Reply #401 on: April 09, 2019, 04:49:13 PM »
To be sure. I've made (in qemu branch):

Code: [Select]
hg pull
cd contrib/qemu/
./install.sh

I'm on OsX 10.13.6 and seems to work ok with clang
GCC: gcc-arm-none-eabi-7-2017-q4-major
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus | EOS 5D MLbeta5- EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX, PS, LR, RawTherapee, LightZone -no video experience-

calle2010

  • Contributor
  • New to the forum
  • *****
  • Posts: 46
Re: How to run Magic Lantern into QEMU?!...
« Reply #402 on: April 09, 2019, 11:33:41 PM »
A good oppportunity to reprovision my Vagrant environment. The install script works with latest Ubuntu Bionic

arm-none-eabi-gcc: gcc version 6.3.1 20170620 (15:6.3.1+svn253039-1build1)
gdb-multiarch: GNU gdb (Ubuntu 8.1-0ubuntu3) 8.1.0.20180409-git

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12181
  • Maintenance mode
Re: How to run Magic Lantern into QEMU?!...
« Reply #403 on: April 10, 2019, 12:08:21 AM »
Sounds good. I've tested on an older Win10 WSL VM (based on Xenial) and downloaded a newer one, hopefully based on Bionic; not expecting any surprises, other than minor usability quirks.

Also preparing to install Mojave in a VM to try that as well; High Sierra appears to work fine (well, with the "1998" Mac quirks discussed earlier in the thread).

Noticed a quirk - in the "qemu" branch I'm compiling ML with -ggdb3; this option adds debug information usable for QEMU+GDB (so you can step through ML code at source level). For some reason, this also generates slightly different binary code (aside from the additional debug information, which doesn't end up in autoexec.bin anyway). Likely minor, as this change was used in experimental builds for quite some time, without issues.

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3651
Re: How to run Magic Lantern into QEMU?!...
« Reply #404 on: April 10, 2019, 12:29:44 AM »
I'm using QEMU on Mojave, macOS 10.14.4 using an actual (not virtual) PowerBook and it seems (almost) as good as Linux. Haven't checked to see if that 1998 quirk is still there.
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12181
  • Maintenance mode
Re: How to run Magic Lantern into QEMU?!...
« Reply #405 on: April 10, 2019, 09:11:36 AM »
Installed Mojave, downloaded the zip archive of the QEMU branch (from here) with Safari (i.e. without first installing hg & co), ran the install script (cd contrib/qemu; ./install.sh) and... worked out of the box! It installed brew and other dependencies, compiled gdb 8.2.1 from source et voilĂ ! QEMU and ML development environment installed and ready to use!

Problems:
- precompiled toolchain (8-2018-q4 at the time of writing) includes gdb 8.0.x 8.2.50.20181213-git, which... doesn't work with our scripts (e.g. EOSM/debugmsg.gdb)
- guess what - gdb 8.2.1 "stable" is not working either! Why?! I remember testing 8.1 successfully before...
- back to gdb 8.1 from source -> working! Regression in GDB?!
- previous precompiled toolchain (7-2018-q2-update) uses gdb 8.1.0.20180315-git, which IS WORKING with EOSM/debugmsg.gdb, so I'll just use that in the install script.

Minor wrinkles:
- user has to modify PATH manually (can be solved, see e.g. this PR for another project)
- need to allow Terminal to interact with System Events or something like that

Mac quirks:
- how do you type a tilde on Mac?! (OK, found it at the end of a google search)
- why is scrolling in the opposite direction?! (and why it's asking me for a USB mouse in order to change this setting?!)
- how do you open a new terminal?! (clicking the icon just activates the old one)
- how do you navigate to your Home directory? how do you cut & paste a file? and so on...

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 5248
Re: How to run Magic Lantern into QEMU?!...
« Reply #406 on: April 10, 2019, 09:27:05 AM »

Mac quirks:
- how do you open a new terminal?! (clicking the icon just activates the old one)
- how do you navigate to your Home directory? how do you cut & paste a file? and so on...

- cmd + n for new terminal window while having terminal up front
- Cut not working but cmd + c(copy) then cmd + v(paste)
- Shift + cmd + g(will open 'Go to the folder' field) then paste tilde sign and enter

Other stuff:
- shift + cmd + .(punctuation) will unhide all hidden folders. Run the same command to go back to hidden mode.

calle2010

  • Contributor
  • New to the forum
  • *****
  • Posts: 46
Re: How to run Magic Lantern into QEMU?!...
« Reply #407 on: April 10, 2019, 03:55:17 PM »
- why is scrolling in the opposite direction?!

Because this direction is the same as the direction you would use on a touchpad with a two-finger-gesture to move the screen content. See the scroll wheel as a primitive one-dimensional touchpad.

The Windows scroll wheel direction comes from the "I click&drag a scrollbar to move the content" times (1990s or so :) ). The scrollbar moves into the opposite direction (down if the content moves up).
Fortunately my Logitech mouse on Windows can change this weird Windows quirk.

dfort

  • Developer
  • Hero Member
  • *****
  • Posts: 3651
Re: How to run Magic Lantern into QEMU?!...
« Reply #408 on: April 10, 2019, 04:05:39 PM »
- how do you navigate to your Home directory?

All of these work on the Mac:

Code: [Select]
cd
cd ~
cd ~/
cd $HOME
5D3.* 7D.206 700D.115 EOSM.203 EOSM2.103 M50.102

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12181
  • Maintenance mode
Re: How to run Magic Lantern into QEMU?!...
« Reply #409 on: April 10, 2019, 10:00:54 PM »
I mean, the Mac has a reputation of being easy to use, right? I get all sorts of animated popups, OK, maybe that's what users like, but... for pretty much every single basic thing (like moving a file, or going to the Home directory in the GUI, or... typing the ~ character) I need to google how to do it. And it's not the first time I've used a Mac.

Win10 has its own share of quirks (e.g. the WSL home directory is hidden quite deep in the host filesystem), but not nearly as many.

Good news - precompiled toolchain 7-2018-q2-update appears to work fine! The installation script now defaults to that on both Mac and WSL.

scrax

  • Contributor
  • Hero Member
  • *****
  • Posts: 1538
  • Code monkey
Re: How to run Magic Lantern into QEMU?!...
« Reply #410 on: April 10, 2019, 10:07:12 PM »
I mean, the Mac has a reputation of being easy to use, right? I get all sorts of animated popups, OK, maybe that's what users like, but... for pretty much every single basic thing (like moving a file, or going to the Home directory in the GUI, or... typing the ~ character) I need to google how to do it. And it's not the first time I've used a Mac.

Win10 has its own share of quirks (e.g. the WSL home directory is hidden quite deep in the host filesystem), but not nearly as many.

Good news - precompiled toolchain 7-2018-q2-update appears to work fine! The installation script now defaults to that on both Mac and WSL.
It's easy for who never used a PC before :P
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus | EOS 5D MLbeta5- EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX, PS, LR, RawTherapee, LightZone -no video experience-

scrax

  • Contributor
  • Hero Member
  • *****
  • Posts: 1538
  • Code monkey
Re: How to run Magic Lantern into QEMU?!...
« Reply #411 on: April 21, 2019, 11:59:00 AM »
I don't know if it something that need attention or not, but when emulating 600D if I press L (for live view) i got this:

Code: [Select]
   163:  8499.968 [MC] PROP_GUI_STATE 0
   164:  8499.968 [PRP] ERROR TWICE ACK REQUEST L:846
   165:  8499.968 [PRP] this->dwWaitAckID = 0x80020000(0x80040004)
   166:  8499.968 [LV] [LVAE] EP_SetControlBv() >> EP_ControlBv:1
   167:  8500.992 WARN [LVDS] First Get DTS_GetAllRandomData
   168:  8501.248 [LV] [PATH] GetPathDriveInfo[0]
   169:  8501.248 WARN [LVDS] First Get DTS_GetAllRandomData
   170:  8501.760 WARN [LVDS] First Get DTS_GetAllRandomData
   171:  8502.272 WARN [LVDS] First Get DTS_GetAllRandomData
   172:  8502.272 WARN [LVDS] First Get DTS_GetAllRandomData
   173:  8502.528 [MC] cam event guimode comp. 0
   175:  8506.880 [GUI] ERROR ***** Lv GetMovieFrameRateIcon S (81)
   176:  8507.136 [GUI] ERROR ***** Lv GetMovieZoomIcon S (88)
   177:  8507.136 [GUI] ERROR ***** Lv IsMovieZoomSetting(88)
   178:  8508.672 [GUI] ERROR ***** Lv IsMovieZoomSetting(88)
   179:  8508.672 [GUI] ERROR ***** Lv IsMovieZoomSetting(88)
   180:  8508.928 [GUI] ERROR ***** Lv IsMovieZoomSetting(88)
   181:  8508.928 [GUI] ERROR ***** Lv IsMovieZoomSetting(88)
   182:  8510.208 [GUI] ERROR ***** Lv IsMovieZoomSetting(88)
   183:  8510.208 [DISP] UpdateReverseTFT(off) Current=1 Target=1
   184:  8558.336 [GUI] ERROR ***** Lv IsMovieZoomSetting(88)
   185:  8558.336 [GUI] ERROR ***** Lv IsMovieZoomSetting(88)
   186:  8558.592 [GUI] ERROR ***** Lv IsMovieZoomSetting(88)
   187:  8558.592 [GUI] ERROR ***** Lv IsMovieZoomSetting(88)
   188:  8558.848 [GUI] ERROR ***** Lv IsMovieZoomSetting(88)
   189:  8616.704 [GUI] ERROR ***** Lv IsMovieZoomSetting(88)
   190:  8616.704 [GUI] ERROR ***** Lv IsMovieZoomSetting(88)
   191:  8616.704 [GUI] ERROR ***** Lv IsMovieZoomSetting(88)
   192:  8616.704 [GUI] ERROR ***** Lv IsMovieZoomSetting(88)
... this Lv IsMovieZoomSetting(88) continues till I get out of LV...

(I'm using last qemu branch with python3 and new toolchain)
I'm using ML2.3 for photography with:
EOS 600DML | EOS 400Dplus | EOS 5D MLbeta5- EF 100mm f/2.8 USM Macro  - EF-S 17-85mm f4-5.6 IS USM - EF 70-200mm f/4 L USM - 580EXII - OsX, PS, LR, RawTherapee, LightZone -no video experience-

kitor

  • Contributor
  • Member
  • *****
  • Posts: 106
Re: How to run Magic Lantern into QEMU?!...
« Reply #412 on: April 23, 2019, 04:37:56 PM »
I installed QEMU from qemu branch, however I don't see any definitions for M50 or R here. What do I need to do to emulate those?
Any hacks in arm-softmmu/hw/eos, or I should use some other camera for now?
EOS R

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12181
  • Maintenance mode
Re: How to run Magic Lantern into QEMU?!...
« Reply #413 on: April 23, 2019, 04:39:43 PM »
There is a patch for M50 here; IIRC it worked on R with minimal changes. Will clean them up for committing, but I think it will happen after bringing the current state into mainline.