Magic Lantern development on Windows - A detailed step-by-step-guide

Started by dlrpgmsvc, June 27, 2013, 12:39:00 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

dlrpgmsvc

[1] Create a BitBucket account (go to "https://bitbucket.org/hudson/magic-lantern" and select the "sign up" option.
    At the end, bookmark your bitbucket page and write down his web address (url). If, for example, we have chosen
    "dlrpgmsvc" as our username, we will end up with our bitbucket address as follow : "https://bitbucket.org/dlrpgmsvc")

[2] Once inside our bitbucket account, we need to "fork" the magic-lantern project. In order to do that, we must go
    again to "https://bitbucket.org/hudson/magic-lantern" (you will see that now we are logged in in bitbucket because
    in the upper right corner you will see a man-symbol, that represent you) and then push the upper right button named
    "fork". In this way, all the current and up-todate magic lantern sources are copied inside our bitbucket account: wait
    for the finish of the copy process, and at the end, if you return to your account page ("https://bitbucket.org/dlrpgmsvc"
    in our example) you will find "Magic Lantern" listed as a fork: now click on the name "Magic Lantern" and you will enter
    inside your fork (parallel derivative project), inside which you can work as you desire. But for this moment leave all
    as-is. Write down our complete fork address, that is, in our example : "https://bitbucket.org/dlrpgmsvc/magic-lantern"

[3] Download and install "Virtual Box" (go to "https://www.virtualbox.org/" and select "downloads" to the left, then select
    the version suitable for your operative system ("windows", in this case)
   
[4] Download the virtual machine of the developing environment (ubuntu) from here :
    http://nanomad.magiclantern.fm/Development%20Box/
    (go inside the VirtualBox folder and download the file : "MagicLantern.ZIP" clicking on it with the left mouse button.
    The Windows system will start to download the file.
    After the download is done, decompact it somewhere on your pc, but remember where, because we will need it in the next steps.
    The User/Root password of the ubuntu system is "123456" (without quotation marks). It will be useful in the future if the
    system will ask you for it.

[5] Launch "Virtual Box" that we previously downloaded (if warning windows will appear, push OK or IGNORE on all of them, don't
    worry about them) and then load inside Virtual Box the ubuntu developing environment that we already downloaded
    (menu "machine" then "add" and select the file "MagicLantern.vbox" that is located where you decompacted the virtual machine),
    then start this virtual machine (select it from the left column with a sigle left button mouse click, so it become with the
    blue background, and then you push the big green arrow on the upper part that point toward right named "start"). Then wait
    patiently the start of the ubuntu environment. During the loading, some windows will appear: answer OK or IGNORE to all.
    At the end of the loading process, DO NOT CLICK INSIDE the window but instead click on the upper menu bar like described
    in the next step.

[6] Activate the wired netword from the upper menu of Virtual Box (if the windows pc is connected by wire to the network)
    with this procedure:

   [ A ] Select from the Virtual Box upper menu bar of the ubuntu window the "Devices" item and then "Network cards...".
              be sure that the option "Enable network card" is checked, and inside the option "Connected to:" select
              "Card with Bridge". Under the option "Name:" will appear the name of your netword card installed on your pc.
              Under "Promiscuous mode:" select "permit all".
        [ B ] Press "OK" on the lower part and the configuration is done, and it will be retained also for the next future
              sessions.

[7] Now click inside the ubuntu window: the system will ask if you want to capture the mouse: answer affirmatively (press the
    "capture" button. Once inside the environment, enter into the folder named "magic-lantern" from the icon "Home" that is
    located in the upper left corner (double click with the left mouse button over it). In order to exit with the mouse from
    the environment, you must press the rightmost keyboard key CONTROL (CTRL), once. To return inside the environment, click
    with the left mouse button once, inside the the ubuntu window.

[8] Erase all the content of the folder where you are into (magic-lantern), all but the file "makefile.user" that is optimized
    for the ubuntu environment and must be left alone and not deleted

[9] Download inside this folder the new up-to-date sources, following this procedure :

   [ A ] Push the right mouse button inside an empty space inside the folder and select the command prompt from the
              manu that pops up

   [ B ] Issue this command: "hg clone -r unified https://bitbucket.org/hudson/magic-lantern".
              This command connects to the internet and downloads the latest Magic Lantern sources, and it generates,
              starting from the folder where you are into (magic-lantern) a new folder (magic-lantern) and it puts
              inside this new generated folder all the up-to-date sources.

        BEWARE : If the sub-folder named "magic-lantern" is yet present and it is full of the old sources and you want to
                 update them, FIRST YOU MUST DELETE this sub-folder along with all its content. Otherwise, the "clone"
                 command will give you an error. After deleting and then cloning, remember to copy inside the new created
                 sub-folder the file "makefile.user" that we saved from the deleting at previous point [8].

[10] Duplicate this new sub-folder with the up-to-date sources and name it "magic-lantern-mymodifications" (or whatever you want)
     with copy-paste. In this way, startig from the root, we have "magic-lantern" inside which we have the file "makefile.user"
     and two folders: "magic-lantern" and "magic-lanter-mymodifications"

[11] Copy the file "makefile.user" inside sub-folder "magic-lantern" and also inside sub-folder "magic-lantern-mymodifications".
     Now you will work inside the folder "magic-lantern-mymodifications" and we will leave alone "magic-lantern" subfolder as
     a reference from which to start for other modifications, generating everytime a new copy of this sub-folder for every
     modifications we would want to do.

[12] Go to the root folder "/home/magiclantern/magic-lantern",press the right mouse button inside an empty space and launch the
     command prompt. At this point, write the command "hgtk" or "thg" followed by ENTER key. A new window will open then.
     On the left column all the repositories will be listed that are inside your local environment (they are the folders that
     contain your sources). It will be listed "magic-lantern" and no more. In order to make listed also our new
     "magic-lantern-mymodifications" folder we created, you must evidence in blue (by clicking once with the left mouse button
     over it) the word "default" that is over the word "magic-lantern", that is the root, then right mouse button in an empty
     space in the left column under the word "magic-lantern": in the appearing menu select "Add repository" and then select
     "magic-lantern-mymodifications". Now your new repository is listed in the left column as well.
     Now right mouse button over "magic-lantern-mymodifications" and select "settings". In the new windows that appears, on the
     left select "commit" and on the right, at the "Username" parameter put your bitbucket username you selected (for example,
     in our case, "dlrpgmsvc" without quotation marks). then push OK in the lower right side. Now close the "Tortoise Hg"
     window (push the X in the upper right corner, like in Windows style). The username will be remembered by the system
     as long as the folder "magic-lantern-mymodifications" will not be deleted. So, this operation has to be executed only once
     at the beginning and then no more. If you do not set the "username", the "commit" operations will give you an error and
     will be aborted, so it's important not to jump this step. What is "commit" operation will be discussed later.

[13] Now we can enter into the "magic-lantern-mymodifications" sub-folder and we can start to study and modify the sources files
     that we want. Once your modifications are terminated, go to the root of the folder "magic-lantern-mymodifications", right
     mouse click inside an empty space and in the resulting menu select the command prompt and type the following command:
     "make clean" (without quotation marks) and push ENTER key on the keyboard. This clears out all previous temporary files.
     Then type "make" (alone, this time) and push ENTER. This command will compile and generate all the .BIN
     files of all the cameras models, and also it generate the unified .BIN (the one that is good for all the camera models
     but that is bigger in size). if you want to save compilation time, type "make <camera model>", for example "make 550d.109":
     in this way you will compile only the .BIN that is relative to your camera, so compiling time is shorter and the resulting
     .BIN file is smaller altogether. The .BIN files are the ones that are to be copied inside your camera, and they are the
     result of the compiling phase. They are located :
     - Into the folder "/platform/all" for the unified .BIN (the one that is valid for all camera models)
     - Into the folder "/platform/<camera model>" for the camera model-specific .BIN. For example, for the Canon 550D, the
       corresponding .BIN file is located into the folder "/platform/550d.109" or 550d.xxx where "xxx" is another number and
       correspond to the Canon firmware version of that camera model.

[14] Now we must "take out" the .BIN from our developing environment. We will use an USB memory stick. Insert it inside a free
     USB port of your computer. Windows will immediately recognize it. Now we must go to the Virtual Box upper horizontal menu
     bar of the ubuntu window and select "Devices" and then "USB Devices" : it will appear a list where, on the bottom, there is
     the name of your memory stick you have inserted some moments ago. If there isn't, go to "Machine" then "Settings" then "USB"
     and then click with the left mouse button to check the option "Enable USB controller" and then "OK" in the lower part.
     Return to "Devices" -> "USB Devices", then click once with left mouse button over the usb key name. It will disappear from
     Windows but then ubuntu will see it. Now we can copy onto this usb memory key our .BIN file we just created.
     Then detach phisically the usb key from the port, then re-insert it so Windows can see it again.
     Now access the key from Windows, and copy the .BIN file to the Hard Disk and then on the camera memory card, or directly
     on the camera memory card (connected to our computer by a card reader or by the camera and usb cable).
     I don't advise to copy directly the .BIN file from the ubuntu environment to the camera card, or in the opposite way
     (from camera to ubuntu), because linux environment makes many hidden files and folders, which interfere with Windows
     filesystem and camera filesystem. If you use a usb memory key as intermediate, you will not face any problem.

[15] When the on-camera testing of your modifications are completed succesfully, you can proceed to share your modifications
     with all the Magic Lantern community. The process consists into submitting your modifications to the Magic Lantern
     developer board. If they will agree, they will fuse your modifications inside the official Magic Lantern last "nightly build".
     After some stress-testing time, the last nightly build will become the official stable release. But this usually require
     many months or years.
     All this translates in the following operations :
     - do a "commit" in local, that will produce a listing of all the source files modified
     - do a "push" from local to our online magic-lantern fork (inside our bitbucket account) of all the source files modified
     - do a "pull request" from our online magic-lantern fork to the official online Magic Lantern repository

[16] Go to the root folder of "magic-lantern-mymodifications", right click on an empty space and fire up the command prompt.
     Then type "hg commit". A new windows will open. it's a simple editor. It asks for a single line of text describing your
     modifications. The cursor is at his correct location: you have only to write a short description about your modifications,
     for example: "third menu item modified", and then push CONTROL+X on your keyboard. it will ask if you want to save the
     file that contains the sum of all your modifications to the source files: press Y on the keyboard. Now it asks to you where
     to save and the filename: leave the default ones and press ENTER on the keyboard. You will then be returned to the command
     prompt. Done.

[17] Staying on the root of the folder "magic-lantern-mymodifications", issue the command on command prompt:
     "hg push https://bitbucket.org/dlrpgmsvc/magic-lantern" and press ENTER. Obviously, "dlrpgmsvc" must me swapped with your
     bitbucket account username. This command will update your bitbucket magic lantern fork repository.
     It will ask to you your bitbucket username and password: issue them. While you will insert username and password, not even
     asterisk will appear: nothing is echoed to the screen. This is normal. Don't worry and keep typing. Beware upcase and
     lowercase: respect them always. Then press ENTER at the end of field inputting.
     The operations done will be displayed and the number of changesets (one, that is the one generated with the "commit"
     command issued on the previous point and that contains all the modifications you have done), the number of changes you have
     done, and the number of source files you have modified. At the end you will return to the command prompt.
     Now your online repository is up-to-date with your local modifications. Close the command prompt windows with the big X
     located in the upper right corner of the command prompt window itself.

[18] Before launching a pull request, we must be aware that meanwhile we was busy to do our modifications, the official magic
     lantern repository was active and modifying itself thanks to other developers like you that sent their request and some of
     them was also yet fused inside the official repository. So, if at the time of your pull request, your local fork repository
     is not up-to-date with the official one, your pull request will not be accepted and an error will be generated !
     So, to avoid this, proceed as follow: Go to internet by using Windows, and enter your bitbucket account. In our example:
     "https://bitbucket.org/dlrpgmsvc/magic-lantern". Push now the upper button "Compare". On the left make sure you have
     your repository ("dlrpgmsvc/magic-lantern" in our example) and just below this line make sure to select "unified". On the
     right make sure you have "hudson/magic-lantern" and just below this line "unified". Just below you will see the modifications
     YOU have done compared to the official repository. If no modifications were detected, it will say "There are no changes".
     If this happens, and you have done modifications, make sure you have issued the "push" command at point [17], and you have
     done "commit" at the point [16]. After you checked your modification, now you must see how the offical repository changed
     during your modifications time. You can do that by selecting "Swap source and destination" just under the "Hudson" right
     block. Wait a while and you will see all the modifications that was done while you was busy by programming your magic
     lantern modifications. Now you HAVE to update your local repository with these modifications, otherwise you pull request
     will be refused automatically by the BitBucket system. To do so, push the "Merge" button on top of your repository right
     block. A window will appear: you can press "Merge" at bottom. Wait and you will then see "There are no changes". Ok !
     Now you can proceed with your pull request at the next point [19]: do it without waiting, or other modifications could
     appear to the official repository and so invalidating your pull request.

[19] Now, the last step: submitting our modifications to the Magic Lantern community for review and, eventually, acceptance and
     fusing. Go to internet by using Windows, and enter your bitbucket account. In our example:
     "https://bitbucket.org/dlrpgmsvc/magic-lantern". Push now the upper button "Pull Request" (the button, NOT the menu item
     in the lower part that says "Pull requests"). Verify that the source (the left part) is "dlrgmsvc / Magic Lantern" and
     "unified" and that the destination (the right part) is "hudson/magic-lantern" and "unified".
     In the lower part, put a description title about your modifications inside "Title" and a more long and extended description
     into "Description". Then push the blue button in the lower part "Create pull request". Now our request is sent.
     They will analyze your modifications, and they may ask to you some questions about: you must answer to them and follow
     your modifications discussions until they are accepted or refused.

[20] In order to follow your modifications destiny, go to "https://bitbucket.org/hudson/magic-lantern" and push the blue button
     "Pull requests (n)" where (n) is a number that represents the number of pull requests that are in this moment under
     scrutiny. Verify that, among them, there is also your modification request. At the right of each request there is a
     baloon with a number: this is the number of comments received, and a circle with a checkmark: if present, the modifications
     was accepted. It is important to follow your modifications destiny to the end, otherwise, even if ok, they could be
     refused simply because you didn't answered to a question. Once accepted, your modifications will be included in the next
     automatic nightly build. The latest nightly build can be downloaded from here : "http://nanomad.magiclantern.fm/nightly/"
     by clicking on the last file on the bottom with .ZIP extension.





[NOTE 1] From time to time, a little explosion symbol may appear on the ubuntu machine: they are updates to the operative system.
         You can do them but they are very long to install and download and... no more magic lantern compatibility is assured...
         proceed at your full risk...

[NOTE 2] In order to shut down the virtual machine retaining all the modifications to the files you have done, press on the
         blue circle in the upper part where is a little white mouse drawing, and from the resulting menu select "LOG OUT"
         and then select "SHUT DOWN". Then wait until all is closed correctly.

[NOTE 3] To open "terminal" or the "command prompt", select the menu option "Open terminal here"

[NOTE 4] To modify a source file, right click it and select "Open with Leafpad"

[NOTE 5] The source files are inside the folder named "SRC"

[NOTE 7] After finishing a set of modifications and after the modification is merged (or refused) to the main ml repository,
         and verytime you start a new modifications set, you MUST delete and recreate che subfolder "magic-lantern" by starting
         at point [8] and subsequent points. This way you will work always on the most up-to-date source files.
If you think it's impossible, you have lost beforehand

Audionut

Thanks for the guide dlrpgmsvc.  I haven't read it in it's entirety, but a couple of quick things.

Quote from: dlrpgmsvc on June 27, 2013, 12:39:00 AM
[NOTE 2] In order to shut down the virtual machine retaining all the modifications to the files you have done, press on the
         blue circle in the upper part where is a little white mouse drawing, and from the resulting menu select "LOG OUT"
         and then select "SHUT DOWN". Then wait until all is closed correctly.

I just close the virtual box session and select, "save machine state".

Also, I believe it's good practice to "make clean" after cloning and pulling.

Pelican

Thank you for the detailed step-by-step guide.

One question: with this method could you directly copy the compiled autoexec.bin to the card inside the camera?
With cygwin I couldn't compile the necessary ptp stuff.
EOS 7D Mark II, EOS 7D, EOS 5, EOS 100 + lenses (10mm to 300mm), 600EX, 550EX, YN600EX x 3
EOScard, EOS DSLR firmwares, ARMu, NiControl, etc.: http://pel.hu/down

dlrpgmsvc

Quote from: Pelican on June 28, 2013, 08:47:37 AM
Thank you for the detailed step-by-step guide.

One question: with this method could you directly copy the compiled autoexec.bin to the card inside the camera?
With cygwin I couldn't compile the necessary ptp stuff.

I have not tried cygwin because is a conplete mess.
Based on my experience, and like I wrote on this guide, I don't advice you to copy the .bin file directly to the camra card by any mean (direct usb camera tethering or card reader), because, altough you surely can do this, linux messes up the file system and also generates hidden files and directories that sometimes makes the card unusable: i was forced to low level format it on camera because even windows was unable to even see it! this is however my experience
If you think it's impossible, you have lost beforehand

dlrpgmsvc

Quote from: Audionut on June 28, 2013, 04:56:32 AM
Thanks for the guide dlrpgmsvc.  I haven't read it in it's entirety, but a couple of quick things.

I just close the virtual box session and select, "save machine state".

Also, I believe it's good practice to "make clean" after cloning and pulling.

Good ! I will add these hints, many thanks ! But can you explain what the command "make clean" do in detail, so I can add some explanations in my guide ?

Also, I found that when I work for some days on my modifications, then when it is time to make a pull request, the request is refused automatically because the sources in my fork repository are not updated with the official ml repository ones (or at least I think this is the reason...), so what can I do ? I surely cannot clone again and then redo all my modifications ! Do you know there is a method, so I can add it to my guide ? Many thanks !

Many thanks for all of your help !
If you think it's impossible, you have lost beforehand

GregoryOfManhattan

by the way, much of this guide is relevant to OSX.
so your detailed description of working within the virtualbox are completely applicable.
the virtualbox is the easiest way to have all the required build tools available.



Audionut

Quote from: dlrpgmsvc on June 28, 2013, 12:47:21 PM
Good ! I will add these hints, many thanks ! But can you explain what the command "make clean" do in detail, so I can add some explanations in my guide ?

make clean, cleans the files generated with a make.  I've also read that there have been problems with compiling the bin if you don't run a make clean first.  I always run a make clean before a make.

Quote from: dlrpgmsvc on June 28, 2013, 12:47:21 PMAlso, I found that when I work for some days on my modifications, then when it is time to make a pull request, the request is refused automatically because the sources in my fork repository are not updated with the official ml repository ones (or at least I think this is the reason...), so what can I do ? I surely cannot clone again and then redo all my modifications ! Do you know there is a method, so I can add it to my guide ? Many thanks !

Yeah, if you've made changes to your source files, you can't do a clean pull request.  I'm not an expert but I see a couple of ways to fix the problem.

Keep a clean copy of the source separate.
Delete your source and clone.
Edit the files.  IIRC, after you do an hg update, the files will be marked where the problems are.  You can then open the files and check where its marked.  You can delete your changes, save the files and run an hg update again.
I'm not entirely sure, but you should just be able to delete whatever code is required in the source file (ie: leaving your source changes intact), and run another hg update.

dlrpgmsvc

I inserted a new point at [18] describing the procedure in order to not face the synch problem between your and official repository !  ;)
If you think it's impossible, you have lost beforehand

dlrpgmsvc

[11 BIS] Attention! Last releases changed a reference.
             Modify "makefile.user" inside magiclantern main folder so that
             you modify this line as follows :
       
             GCC_VERSION=-4.7.3
   
             On the previous version there was no minus sign (-) before the version number. Now it needs to be there.
If you think it's impossible, you have lost beforehand

dlrpgmsvc

I'm searching for a fast method to transfer from virtual machine to usb memory, because everytime I insert the usb, it's immediately taken by Windows, and if I take it from virtual machine "devices" - "usb", it disconnects from Windows but when it riconnects, 99 times over 100 is re-taken by Windows again... it is quite frustrating...  :'(
If you think it's impossible, you have lost beforehand

RenatoPhoto

I am guessing that you have a problem with you USB drive which is disconnecting and connecting.  Try a different USB.

I have written another manual on how to add Shared Folders and solve other problems:
http://www.magiclantern.fm/forum/index.php?topic=7579.0

http://www.pululahuahostal.com  |  EF 300 f/4, EF 100-400 L, EF 180 L, EF-S 10-22, Samyang 14mm, Sigma 28mm EX DG, Sigma 8mm 1:3.5 EX DG, EF 50mm 1:1.8 II, EF 1.4X II, Kenko C-AF 2X

dlrpgmsvc

Quote from: RenatoPhoto on August 12, 2013, 02:56:32 PM
I am guessing that you have a problem with you USB drive which is disconnecting and connecting.  Try a different USB.

I have written another manual on how to add Shared Folders and solve other problems:
http://www.magiclantern.fm/forum/index.php?topic=7579.0

Great! Many thanks Renato! You saved my day! I will try your solutions soon! Thanks again! Indeed!
If you think it's impossible, you have lost beforehand

Phaenomena

Outstanding tutorial! I do have one problem, however. I am looking to edit the 1100D files, and despite moving the camera directory up one level as described in the readme within the unmaintained directory, it fails to build.

When I try to run make 1100D.105 like as mentioned in the tutorial, I get:

make: *** No rule to make target `1100D.105'. Stop.

I searched through some of the scripts and edited the Makefile.platform_map in the root directory of platform which seemed to contain relevant code, but that didn't work either.

PLATFORM_MAP = \
    5D2.212 \
    7D.203 \
    7D_MASTER.203 \
    60D.111 \
    500D.111 \
    550D.109 \
    5D3.113 \
    600D.102 \
    650D.104 \
    700D.111 \
    EOSM.202 \

PLATFORM_MAP_UNMAINTAINED = \
    40D.111 \
    5DC.111 \
    50D.109 \
    6D.113 \
    1100D.105 \
    100D.100 \


I moved the 1100D.105 \ from the "unmaintained map" into the other like so:

PLATFORM_MAP = \
    5D2.212 \
    7D.203 \
    7D_MASTER.203 \
    60D.111 \
    500D.111 \
    550D.109 \
    5D3.113 \
    600D.102 \
    650D.104 \
    700D.111 \
    EOSM.202 \
    1100D.105 \

...


All of the other platforms compile correctly. There are no errors, it seems to just skip past the 1100D directory. I do have a basic background in programming though this is the first time I have attempted anything like this.

Any help would be appreciated :D

ayshih

Quote from: Phaenomena on January 06, 2014, 05:34:40 AM
Outstanding tutorial! I do have one problem, however. I am looking to edit the 1100D files, and despite moving the camera directory up one level as described in the readme within the unmaintained directory, it fails to build.

When I try to run make 1100D.105 like as mentioned in the tutorial, I get:

make: *** No rule to make target `1100D.105'. Stop.


After you've moved the 1100D in the file Makefile.platform.map, I believe the correct call is simply make 1100D (i.e., don't include the ".105").

Quote from: Phaenomena on January 06, 2014, 05:34:40 AM
All of the other platforms compile correctly. There are no errors, it seems to just skip past the 1100D directory. I do have a basic background in programming though this is the first time I have attempted anything like this.

If you're referring to the fact that a top-level make call skips over the 1100D, that's because the variable "SUPPORTED_MODELS" in Makefile.user.default (or overridden by Makefile.user) does not include "1100D".  Until you change it, of course!
Canon EOS 50D | 17–40mm f/4L & 70–300mm f/4.5–5.6 DO IS | Lexar 1066x