Magic Lantern Forum

Using Magic Lantern => Tutorials and Creative Uses => Topic started by: Walter Schulz on March 11, 2019, 05:06:32 PM

Title: Flowchart "ML Pre-Start". First draft and questions
Post by: Walter Schulz on March 11, 2019, 05:06:32 PM
Hello!

Created a flowchart about what is going on *before* running ML. I think it covers most what might be described as bootload stuff.
https://wiki.magiclantern.fm/_media/ml-prestartup_draft2.png

1): @a1ex: Roughly correct or not?
2): @all: Edible or unable to swallow?

And another one for a1ex: After wake-up from sleep there is another card check slowing down wake-up. What is the cam actually doing? Again checking for bootable card or just reading Autoexec.bin and other files?

EDIT: Dang! Forgot "Check if card is bootable!" New version online.
Title: Re: Flowchart "ML Pre-Start". First draft and questions
Post by: a1ex on March 12, 2019, 10:59:48 AM
Well... having some trouble following it :P



Startup

First: camera powers on in one of these conditions:
- power switch on AND card/battery doors closed
- after Auto Power Off: the above + half-shutter press

Boot sequence (regardless of how the camera is started):
- Canon firmware (running on main CPU) has two parts: main firmware and bootloader.
- There are parts running on secondary CPUs, one of them being the MPU.
- I believe the MPU starts first.
- Canon bootloader (on main CPU) starts running (possibly started by the MPU).
- checks boot flags in the ROM (there are more of them, but we are interested specifically in the BOOTDISK flag).
- if the boot flag is enabled in the camera (ROM):
  - card present?
    - bootloader mounts card and checks its contents [ this causes startup delay ]
    - is the card bootable? (marked with EOS_DEVELOP and BOOTDISK in the boot sector)
      - camera loads autoexec.bin (possibly without turning on the LED)
      - on error (autoexec.bin not present or whatever), camera locks up (well, it enters the factory menu available via UART, but end user will only see a locked up camera). Battery out.
      - Canon bootloader un-mounts the card and runs autoexec.bin
      - autoexec.bin (our code) turns on the LED, patches Canon's main firmware to "inject" our code, and runs it
      - ML up and running
    - card not bootable:
      - Canon bootloader launches main firmware (unmodified). There was some startup delay from previous step.
  - card not present?
    - Canon bootloader launches main firmware (unmodified). No startup delay, since the bootloader did not attempt to mount the card. It only checked its presence.
- boot flag not set in the ROM:
  - Canon bootloader launches main firmware (unmodified). Factory condition, no startup delays.



Temporary wake-up

There is also a special power on (or wake up if you prefer) when the main switch is OFF:
- when card door changes its state (either open->closed or closed->open)
- when battery door changes its state (either open->closed or closed->open at least on 5D2, to be double-checked on other models)
- when a lens is attached or removed

In this case the camera:
- boots as usual (loading autoexec.bin if configured, yadda yada)
- remains on for 1-2 seconds
- shuts down
- I believe it's the MPU that tells the main CPU to shut down early (will get some UART logs later)

With a ML-enabled card, you will see a LED blink. The first thing done by autoexec.bin is turning on the LED.

Without a ML-enabled card, or without the boot flag enabled, you (the end user) won't see any activity. The camera still wakes up; you will be able to see activity on the UART pins (both ICU and MPU), or with an ammeter.

We have no control over this special wake-up (Canon hardware does it; the decision to wake up is taken outside the main processor, possibly on the MPU side). You must be careful: do not remove the card during this wake up. This is a problem on older models, where card is accessed by Canon's bootloader without turning on the LED. Otherwise, this wake-up is harmless.



Shutdown

Normal shutdown, from main power switch:
- camera powered on (main switch on, card/battery doors closed).
- user turns off from the main switch (which is just a soft button, btw).
- ML is notified and saves its settings.
- DryOS waits for remaining tasks to finish (i.e. last photo to be saved completely etc) [ TODO for me: keep the LED on during this process! ]
- Canon firmware (on main CPU) saves their settings (by reflashing the main ROM).
- Canon firmware (on MPU) saves their settings too (guess: in the EEPROM).
- camera shuts down (well, unless DryOS is left by ML in some invalid state or whatever).
- I belive it's the MPU that decides when the power is cut off (but don't have a hard proof).
- caveat: when user wants to remove the card, card door changes its state => camera wakes up for 1-2 seconds and may access the card (possibly without LED activity); you need to wait until LED stops blinking. Do not rush to remove the card!

Shutdown by opening card door:
- camera powered on (main switch on, card/battery doors closed).
- user opens card door.
- main switch remains on.
- a normal shutdown is performed (as before).
- after shutdown, when user wants to remove the card, card door is already open, i.e. it no longer changes its state => cam no longer wakes up. Safest shutdown method when running ML, if you ask me.

Shutdown by opening the battery door:
- this is an emergency shutdown; DryOS turns off ASAP (well, unless the camera is locked up).
- ML does not get notified in main builds, so it does not save its settings.
- ML assumes the camera crashed and will not load modules at next boot.
- Canon firmware (on main CPU) saves their settings (yes, by reflashing the main ROM), unless we prevent it from doing so (https://bitbucket.org/hudson/magic-lantern/pull-requests/825/prevent-canon-settings-from-being-saved/).
- Canon firmware (on MPU) saves their settings (guess: in the EEPROM). The main CPU has no veto rights on this.

Taking the battery out:
- user keeps the battery door microswitch pressed and takes the battery out.
- after a camera lockup (i.e. if screen is still open after opening battery door), the same might happen, not 100% sure.
- Canon firmware (on ICU) no longer gets a chance to save their settings.
- Canon firmware (on MPU) no longer gets a chance to save their settings.
- ML no longer gets a chance to save their settings; it assumes crash and skips module loading at next boot.

Related: Shutdown and reboot (https://bitbucket.org/hudson/magic-lantern/src/qemu/contrib/qemu/README.rst?fileviewer=file-view-default#rst-header-shutdown-and-reboot) in QEMU guide.



Software reboot

Triggered from PROP_REBOOT (Lua: camera.reboot()). Apparently handled as clean shutdown + cold boot?! (https://www.magiclantern.fm/forum/index.php?topic=23622.msg213163#msg213163)

Wake up from sleep?

There is no such thing on Canon EOS cameras. There is a setting called "Auto Power Off" in Canon menu. If used, the camera powers off completely after some timeout. When it "wakes up" (e.g. by touching half-shutter), it powers on from scratch. The only difference is the button you use for powering it on.

PROP_STARTUP_CONDITION can be used to tell the difference between a cold boot (after battery out) and a warm boot (second or third or n'th boot without taking the battery out). Maybe also waking up from auto power off; need to check.

Hope this clarifies.

TODO: write the above in a way accessible to regular users.



Edit: here's my attempt at drawing the above in a diagram. It also includes what autoexec.bin does, so it may be a bit too detailed for end users. Feel free to edit (https://www.draw.io/?lightbox=1&highlight=0000ff&edit=_blank&layers=1&nav=1&title=ml(15).xml#R7R1Zc6M487fsQ2qzD3EhMNfj5JpvajOb1CSz18sUBtlmg5GXI8f%2B%2Bk8SAgOSbdnG2EmUVCWIQ7Ra6lbfnBgXs5fPiTeffkUBjE50LXg5MS5PdF03LQP%2FI2deizO2yU5MkjAoToHFifvwP8hOauxsHgYwbdyYIRRl4bx50kdxDP2scc5LEvTcvG2MouZb594EcifufS%2Fiz%2F4RBtmUnQWWu7jwPxhOpuzVjm4XF0ae%2FzhJUB6z953oxpj%2BFJdnXtkXG2g69QL0XDtlXJ0YFwlCWXE0e7mAEcFtibbiueslVyu4ExhnMg88XD%2F9%2B%2BL9bX26Ofv7n8nw7OzLp8sz4LB%2Bnrwoh%2BVArAj3eB6ETwTHUTiJ6QXr35zAep5QZFRNfDRh%2F%2BljY4ThwcPNXhmKyxvJhbOULoBP%2BAbgzF%2BW9zKqnfBmc3wQj9K5VFtb9HqOsRshL4BJsYIy%2BNKEPCr%2BX4%2B49yftM%2FiuYmjcaYqo5tkdcVdSSXmfIXxEMO73fkq1VVu1VVu132ZbO4XxU5igeAaLnQ6Sv%2FmcSm5kk87TI91vR6%2F4xgsvRjGR%2FSq54peTNyQwKEAVoApQBagCVAGqAFWAKkAVoApQBejhbZc7Aq43DM46tcpDYvfW8OXnaZjB%2B7nnk6vPiUfeOM1mEW4BfJhmCXqsbP86GUkYRRcowvqocYn1PUhw4qVT2iGo3le3vJdGdJhk8KV2ilniP0M0g1lCNEh29ax0fzCviTtk7obnhQ%2FCHrJ7pjX3w9DSme%2BD%2BT0mVd8L2z8%2BYOb%2FDVwBrtW1J2C0bIls5xoole%2Br23v8t27aX7dWOECOaOEf3KSujPiqrdqqrdqqLdfWTpM8jsN4gg%2FpjjzzQvLv4u67MocrQBWgClAFqAJUAaoAVYAqQBWgClAFqAJUAaoAVYAqQBWgClAFqAJUAaoAVYAeHaAqMqcRmQOALheZ4wBnT5E5QDPeXJLuPuNFZG75Wrgkr8Nk9oyXQ215jrhRvD2iVDE7qq3aqq3ab6X9IQMvVaCQAlQBqgBVgCpAFaAKUAWoAlQBqgBVgCpAFaDKMbObY2ZoaHKeGVM39uSZsYYcImEwgfesiZJsiiYo9qKrxdnzJqoX99wgNGfo%2Bgdm2SurlevlGWqiH%2BMwef2TPD8wy%2BZfrDvauHxptF7rrTuYhHjwMGEnl85MivLEZ4OyWGFeL5lAdpfOhk7Gu3L6Ehh5WfjULLe7y0xYnDPs%2FPb24fLL%2Fa88UYwjjxifSJk545qbq3Tqzcnh7GVCihsPxhF69qdekg0C6IdpiOLW0iY1ft2xTSaPEUHtimEZrhEIyQN3H4QYwZdhAv2M9ovJJCHXKRMoF4zWnOgOqEa3yCKp040uIBsAXJ5sbG33ubI5nJPh1pCmacA1jBYagAi9moYxbHHoBZIo626dskfvUEi5N0O0oWsDoA81SwOmZtim28S6bTY7LKiL9bHAp7Bb03bcoW1YzlAH2rDJBI3WJBUkyvVL56karNTUOR9q6qzaT3PmgKttP3VG7adkXG067H7qAF%2FU%2BxWm3HzSatjF9K2b1hZjas4w2%2FBLJtc6LRANeAbZAaczyl2%2BxG9JgTU%2BJ5IOymnZhc2BIYfwGB0zvktR1cfPE1Ggk63GaApousHXtNnXBIilM56F9SGdvYRZTTjDrb9qVxaiGWm81hptwWxbKU9KoCult7pEJ8aheSgxT%2BdpqqFuFQLehZcEZLgJTOl4ryUUteLU6ee7L7fk0TBe7YxcpTZ9DAmyQdaGSH7UBXRtux0sAvtD7mRNIW9Yxv%2F1wEiHvHZ13DtZFyt82FrhQ2Ng9yc8CPmu6XDzcDGF%2FuNJUXwc%2FyOqbcozLhpg4XszmHj44NvtV27yOjT1yHGzbphQS40FsmzI6kCNFW%2BNok%2FntOWNOPhEPlJERK3IS9PQXyIs1ESFheBwJMKCwPojriAoq6%2FVpswU2evYuR3Vumo1lCq4vqUiBzSs2g91x7TZX9DaEN2BXbtu2c3XLNHr8LrwXmu3zckN6fLhOHbjrTqTzBYruOhwW51ROKP2Yeyb20jQvdtEV4rLa2nF1iVppXO5eiXcq4Xtr3hmM7q9EKFbUs4ut63ioeIrXHShSz4vL5tzjz5MQ%2FwebUbpLPMeIUEd4T%2FTMIKbA1BpDime3ox%2BtCTAs%2FP6S62r0eZg73d7Ho913xdtz4E1skyrKwnKbUqswyG%2FOQ91ntW7%2B8qZsXjFYfPNuetNdf0OKutU6WcHNdo7qLO9KbTRkTPcm%2B1TiFfBVw4%2FghpptGRmy%2BJl5l61Gof3B71Lf%2BnKLX8tF3AO5loVpx8aB5m17RHtyBo3gXEw6%2BZKwFdLYe3okbU5nLooh%2FPT94fbqz%2BvLgbnX37bPualS4iENtwtQlk%2Bgk1Wb3mXNYExxN2TTVa8dHl71UfYYZuG2ko%2BOtj%2B6nKz8C2nGVfs2wnjKh96qdYBNtU6DJ6cAhM6wVBETo4%2BKkIBOkA%2FaDp8gWPzdnLg9GkSBCVIb8pgMgRGQywaaJXU1I9k5MrqR%2BX63ndECND0xtIyLTBwHd02HUvXLMdtWfjkDYnNboFRBhD2pAe5vB70G6LLtWHQWLql4sn1oghGCDPeGdmya0ugca22NjY3YbB4WO1QfEVvWlqHltmYfIt3DfXreAAm4GboSLnM9jwBDG1JpuAaRyXEl%2BCsFuKLbfmrNwl9%2FP%2FGI2E4sXzsgheheJKGARTs7b9sbGjsYsuHABOnLSJN17INryvSNEDbDyjc9W27T3rU3bdCj3v1E640aa5XxsFxKeOulDL%2BDXoBvmmJNr2air%2FHs20cKgXraL5RO%2F16Q%2FpBAdw%2BmOlNBga0Lcvl5l3jBrom0oT3xg3KEJo3YuTsd1cvmfV6djA8MivogXzh%2FTNsV3aG7CMTvHjjh8gnnBM5qyhHw1zSN1eXhIMmkERWDfwNGKjAQx6E49eaULbQrAhvJq%2Fx0fz1hxcHPxJILy4V1wR7TQqTJ9LJDM4QxewY81QsRt7s2vE8Inyegf0zcZqHcZj9yLz0kfJSzEoxznLyMvRMxnU6e%2F1R3bILxr7EGFg6ucQzP0WIvDAjSik%2Bi57p6iHQ0SpCkn2ibEo%2BC1pNAoGRjOmUHPwIwnTuZf70B3nZ5sJyF3tjP8IysOy2sKwLDMWGZvHbI9D2Jy0fJiy%2FYqQNNrrgqp0zUgCkRd%2FDJVMu8Yt9BAOD9PS4nVsddzQyS2ZGsBjgGd54SKdVCnnxdweevfAiEG5EPWqyj%2FooIc6ympNvczVlimajPJVlwyLmKs%2Bgu1BRygrCJRPWHIGOAgQWi3a4TYcErr0VAu9fAK4MERImiyPj20AqjJMLaLylgh30kojAyvILSnmVE4u3i3LsMq7zd5gUUnbbCPLn7TcCOYk4TfPZLmLpImq1xUWxgH1S96NuzAFPa2ERi17ScBJ7WU6Oy6u9GnIOziWB20yPdm1eUq3CTBshpPsTVHlL%2FrvPhRoOm8kNAGiAm4deYxqAXCLoXVK4WmGSUJ14BtPUm8hHeedpUad2jvc1b0TCwzWiKFJvKOOF%2B2ds31Oqt069tFSCE6zpU5V%2F5GUZTF4HHTCFLtw95%2FRXxD66tO6aWjOuHGiuzq1GYYjH%2Fsy7hs4tx48X6QSAwAveL1swpGSdSguKkE9tQPl8IE2O7CMSCz6AN%2BgYkv0fjcfy3VzUzYzbdfFbwQu8IsyQ8YZRnmUoTjdgCVy%2FN2GWUV5HeWaMmCSSUvgI1sIx3GSgFbIRRXYU0pybURL6j2TQFwXUZG4zamWMsWS5OWOV4rTcXZv3f4ninzNKOnHo%2F7RT399afLySJIkNWh4FdxR5iz5Gnv%2BIn9YuIpTCk8qiXXS9uC1AKEnpfeUMpVOUR8FC2icmXpRnbYD6leuv6SoM4FNaxAmP8Xov7NwzGOdkXE9eGLHd%2BSkk4%2Fj%2B6dvDUvTtdQeUyKzyEp%2Fpwm5H0pnJpXQY%2FIYoTAnSgbk3TszH%2Fr5%2FORm0IrDBgUN%2FQVlt5yPNAmhVWTPdqkzAweahfFvvZQF2CADYKpRXuobQsdUFaBUS1bV2JTbpeN5WaRzgti22%2B8n9Nyy3tU73kPtf5eseW30LF9SX9xkJV%2B85Xl0%2Bk0%2FWc9TTyjeb69U23IFhLmpXtJKDZMmgciqXMomzstuOaKKqWVq%2BlUWC7ZkoNkrRa1nYFzEVRLzdRZtYJPW0gzVkFT8vj%2F12UAfphqx%2F%2BG8OYx%2FuIMmT8JBRONkqRGR3Y%2FbBIy90nStXhEV3TjARBiZae5PTh6KaCPzKQDSilUiCNGbm2%2Fl1N%2FE98wTNMQYJmqZYQY1gIihftbGayVEFWXKhF4X%2FecxWMsYrnRzKlxy5SKCXwYo4iPK55mEJ%2FZ9i1UfxOCRm5jGtQKKd5oXFN4VZFsaTdJuYrTLa6TJ5vb0%2FqUU7oTkZthftMn1CPtF6k2xXxcKaoSCP4LYAvgfm0FIdQVWYqhEPIGANXLXnznhDKdF0m3nurvp48G7J4pv2Yi3pRPhRhSV555t8a6GXPPo94pfZrIl1kNrb1ibl7zoRx4nhldAuR0Pno%2BpksH18D6XFnz9gLQbT4R3m%2FdZiqHxz7z1VAZjSkVql7HsskVqmlPfy%2FurhpChunRL3mSTdewVrKxIPPlKgJV8k1gKCZG3Rx072F2ppvZmSEDsQYmkWWU%2BIZWHjYyHEt%2FUJp114pSM9RceViAsEtbDlDAIr1eaF9exzHpJIj4dN7Gc0AIOVFCUlTRNExoGeqCZeBY2yCI1tbAfvMMRTB8M2fzaHAqtYWV9ZZAHfAwOwuNX1AeK5SiG12ihLK%2F3B%2FKaCiq3v3n9ttD6ZAyzn0NNgH6ZEZ89Z6LIFp4B1bOLK8r1wLjRoRGEMz8qlQ%2B09xOlgLlf9MVzzVRvoknpya%2FfN9cYWpwtzlrCTU67QjhYhL2iFJMYkvPKpUT9n00%2FASuDfKdBfrGCtNxTsbNk5zmGy%2FvlVrotX%2BX0V90gkt2xKc2%2BwuFYQK6GrGPN%2Br7G0WZb9PCHrw%2FeoLytP5iglYgJbP4MBCa%2Bkifi05nyahZSaM%2BhPY%2FIQTa1K8jgu0gyKCM2iMn1IxMcgp1LpKJ%2BkgzUzI%2BvG4ci4SdpLJcq3UwFyaDUDY4TFoHSRvr%2B%2FYlDl3rLGsvMYzqsFyLks2x74tepIQ4VprkCikyRhEMANUtSeCVQxpHHqNCSAqjsBzEjqbgl1YZwqVJwTZqWS919%2BbZawKNyLLOewAX9SVL2QN38V0PteEbu9YAH%2F5GnjW9xaHkeFSxK%2FPIXRE9wA%2FFvmdGUVMhiSqVuz4BnlbiIPNhxMCA%2B5J3QYwKda53RJIMqkaBpieiwO07FJfkVUXnxJlZfUi5%2BOqN9sUb%2FQkWoIqB%2FssUzQmvJ%2FUiLaks2rcsazhNRwHFISJZVfTljGxqzIWdlNMJOG6JRWUqFAFOlqVDpsEHO9VGEoVZRmA%2FFiIylnNf39QVP%2BqFmn%2BGzMIiUijP0oD8qNu17ep2RYi1GPMOEnIUx%2F6pNED27aASZXl9EQEKNRpgr0k6pXGvW7iGAtj%2FcZir2DRivvCSvlkyOJST1rFfDTtW0%2F09a2pbjthdVR4Klpt7Ydy%2BzhU2zA0TfeWHpTLq9u739cXv1%2BdXN7V%2Bk257e3D5df7n%2FtVM1cDYxWftZ6cZJ94HrPyuYSkwrdFliCXQpJLtuGyBAbP7v%2Bvm%2FTFGwBU5BDI6xwvbdwNCD4EsodeqYS9%2B1vXaiuIoV07Y5ZKbV7QLvs50zL6qIb4Bw3E7IIa%2FyIBON8pRVNjKv%2FAw%3D%3D).

(https://www.magiclantern.fm/forum/proxy.php?request=https%3A%2F%2Fa1ex.magiclantern.fm%2Fbleeding-edge%2Fml-startup.png&hash=dbdb7d1220f8c62a49b9df57edc8ff02)
Title: Re: Flowchart "ML Pre-Start". First draft and questions
Post by: scrax on March 12, 2019, 06:26:48 PM
uhmmm... wait...

I have a bootable card but EOS_DIGITAL not EOS_DEVELOP and it works ok, to be honest two of three are EOS_DIGITAL
Title: Re: Flowchart "ML Pre-Start". First draft and questions
Post by: a1ex on March 12, 2019, 09:58:38 PM
Open the raw card contents with a hex editor and you'll have a little surprise. Then look at the code (make_bootable.sh, bootflags.c etc).

I'm not talking about the label displayed by your regular Windows/Mac/Linux PC in the file browser; these will display EOS_DIGITAL in most cases.

Yes, on FAT12/16/32 EOS_DEVELOP is written in the place of the volume label, but... there are two copies of the boot sector. Canon bootloader looks at one of these two copies; most other programs look at the other copy. Yes, this means a ML-bootable FAT12/16/32 card may trigger some warnings when the filesystem is checked for errors, as the boot sectors will not be identical.

I've noticed this nitpick with this commit (https://bitbucket.org/hudson/magic-lantern/commits/aed35ab2de0a8a0e231e88530a0fe61266ad9ea6?at=recovery), and kept the non-compliant, but backwards-compatible solution for now (with different boot sectors, but card mounted as EOS_DIGITAL). I can update everything to EOS_DEVELOP (with both boot sectors identical) if you find a good reason for doing so.
Title: Re: Flowchart "ML Pre-Start". First draft and questions
Post by: scrax on March 12, 2019, 10:44:38 PM
For me is better like this because have scripts that look for /Volumes/EOD_DIGITAL to import when card connected, and I use the EOS_DEVELOP for test and stuff that don't need to autoimport, so basically is a feat  ;)

I was just curious why you talked about the EOS_DEVELOP, I didn't know all the tecnical detail and this post about camera boot up is really informative .
Title: Re: Flowchart "ML Pre-Start". First draft and questions
Post by: calle2010 on March 16, 2019, 07:25:55 AM
I like both flowcharts. Walter's is useful from a user and noob developer (read: my) perspective. Users can be educated about the high-level steps and can better understand what is going possibly wrong.

Alex's is great for developers to gain an understanding of the boot process. I already followed boot and reboot. I do not yet understand how the my_big_init_task is launched.

Both should be in documentation. Perhaps in README (Walter's chart) and HACKING (Alex's chart)?