Messages

[Reminder: never edit Bash scripts while they are running!]

Confirmed, very nice find:

Code Select Expand


time hg clone --config ui.interactive=false -U https://bitbucket.org/hudson/magic-lantern ./magic-lantern-http
real 1m20.831s

time hg clone --config ui.interactive=false -U ssh://[email protected]/hudson/magic-lantern ./magic-lantern-ssh
remote: Warning: Permanently added the RSA host key for IP address '18.205.93.1' to the list of known hosts.
real 0m26.886s

time hg clone --config ui.interactive=false -U https://bitbucket.org/hudson/magic-lantern ./magic-lantern-http2
real 1m14.051s

time hg clone --config ui.interactive=false -U ssh://[email protected]/hudson/magic-lantern ./magic-lantern-ssh2
real 0m25.652s


Updated the script (edit: defaulting to https, so it works out of the box for everyone). Reminder: never edit Bash scripts while they are running!

Uploaded hg-repos-22, but hg-repos-23 will be ready tomorrow morning (repo list is currently at September 2019).

Well, 8TB sounds excellent for collecting the downloads from all participants (in max. 2 weeks, possibly earlier), and running the scripts for deciding how to compress and archive the entire thing (which I didn't write yet, but thought about).

We've worked together on the EOS R port before, so I'm comfortable with this option.

Some not-so-good news about repo sizes: in the first sets, average size is about 5 MiB, but in the middle ones (15,16), it's closer to 15 MiB / repo. Watch out for the free space!

At 20 MiB / repo, 250.000 repos would require about 5 TB. Let's hope it won't exceed 30

My guess: many of the new repos are likely to be forks, so they should compress well if grouped together.

No, only the code and its Mercurial history.

Downloading the comments, PRs, issues and other stuff can be done with bitbucket_hg_exporter, but it took about 2 days only for the hudson repository (without the forks). It's still trying to download the same things from all ML forks, but I don't think it's going to finish before the deadline. Doing this for all Bitbucket repositories would probably take years without massive parallelization

Here's an archive of our Bitbucket PRs, issues and other stuff (also being imported by Heptapod, so it will be only for cross-checking):
https://a1ex.magiclantern.fm/bitbucket-mercurial-archive/magic-lantern/gh-pages/

But, as I didn't export the forks with that script (because it was taking too long), the PRs only contain comments and other metadata, without the code. The code is what we are downloading now

Another one in red every now and then is something like "stream ended early, expected xxxx bytes, but got xxx bytes".

And some 404: not found.

Those 404 were deleted by their owners; we can ignore them.

The "stream ended early, expected xxxx bytes, but got xxx bytes" are probably network errors, and these will very likely work on a second attempt.

If you run each script 3-4 times, it should "fix" most of these errors. First run will take a long time (let's say about 1 day, give or take), but subsequent runs should be much faster (let's say 1 hour, but varies a lot, depending on how many errors actually were in that set).

For example, after fully downloading hg-repos-03, an extra run (that doesn't download anything new, but only retries the 404's) takes under 10 minutes, but for hg-repos-01, an extra run takes about 40 minutes.

No sources downloaded? Is this to be expected? Are we downloading commits and .hg content only? A bit confused...

Right. The .hg directory is all you need to recover the sources, from any version:

Code Select Expand


hg update
hg update <changeset_id>
hg update <branch_name>
...


The working directory is therefore redundant and takes additional disk space, so we don't keep it



Here's a Python script that computes the percentages (btw, 0.01% from one set is exactly 1 repo):

Code Select Expand


from __future__ import print_function
import os, sys

try: error_repos = open("hg-clone-errors.txt").readlines();
except: error_repos = []
error_repos = [x.strip() for x in error_repos]

for i in range(30):
  fn = "hg-repos-%02d" % i
  try: repos = open(fn).readlines()
  except: continue
  repos = [r.strip() for r in repos]
  downloaded = 0
  errors = 0
  total = len(repos)
  for line in repos:
      r = line.split(" ")[1]
      if os.path.isfile(r + ".commits"):
          downloaded += 1
      elif r in error_repos:
          errors += 1
      else:
          pass
  print("%s: %.2f%% downloaded, %.2f%% errors, %.2f%% todo" % (fn, downloaded * 100.0 / total, errors * 100.0 / total, (total - errors - downloaded) * 100.0 / total))


There is a catch: this script can't tell what errors might be recoverable (with a second run), and what errors are not (the 404's and a few others). Even if this prints "complete" (0% todo), running the download script once again might bring a few more repos that didn't work on the first try.

[all of Bitbucket]

Alright folks, tonight we start backing up "all of Bitbucket" (well, all of the Mercurial repositories stored there).

Background

Bitbucket (Atlassian) are going to discontinue Mercurial support (whatever, it's their business; after all, it was a free service, so we can't complain). The ugly part is - they don't seem to bother providing an archive. No, they are going to DELETE more than 10 years of work from TENS OF THOUSANDS users.

Other source code hosts did provide an archive when shutting down, see e.g. CodePlex or Google Code. Not Bitbucket.

For this reason, I'd stay away from their current and future offerings, no matter how tempting they might be. Time to move on.

OK, they were nice enough to give us one year to migrate (they could have deleted everything right away, and they would have probably been covered by their ToS - which I didn't read). For active projects, that's probably fine (more or less). However, many of these projects are no longer maintained; their last update was several years ago.

If you believe these unmaintained projects are no longer of interest, please stop reading here.

Anyway. Migration wasn't straightforward either. To date, I'm not aware of any way to losslessly convert a Mercurial repository to Git (and I bet the Bitbucket folks are not aware either). The hg-git extension promises lossless conversion, but fails to preserve the changeset IDs. The best results I've got were with the git-remote-hg extension, which provides a way to contribute from git, but... after trying several tools, I couldn't find a way to recover the original hg repo from its git copy.



OK, so what's the plan?

I've got a list of nearly ALL Bitbucket repositories, retrieved through their API, and attempted to download them. The process was not fast - only about 50.000 repos in one week (OK, my download setup was not very optimized either). There are about 250.000 Mercurial repos (as estimated by Octobus); downloading all of these would take about 5 weeks at this rate. There are only two weeks left, so... let's parallelize!

Estimated average repo size is about 5 10-15 MiB, so the entire *raw* archive should fit on 1-2 3-5 TB of disk space.

How much can this be compressed? First observation is that Mercurial raw data is already compressed, so attempting to e.g. "tar bz2" or "tar xz" every single repo, like I did in my previous attempt, is not going to help much.

However, many of these repositories are forks, which means, plenty of duplicate data. Therefore, forks are expected to compress very well if we group them together). Here's an example from our project (hudson/magic-lantern and its forks, which I've already downloaded from Bitbucket):

- Raw archive: 16.4 GiB (471 repos out of 540 reported by the API, 35.6 MiB average, downloaded in 2.5 hours)
- Individually compressed repos (tar.xz, default settings): 14.3 GiB (compression took 2 hours, one core on i7 7700HQ)
- Archive of tarballs: 14.3 GiB (food for thought)
- All ML forks archived together (tar.xz, -9e --lzma2=dict=1536Mi): only 273 MiB (!), compression time ~ 1 hour

Archiving everything in a single file, after downloading, might also work reasonably well (todo: test on a set of 10.000 repos).

Hence, the plan:

- Stage 1 (next two weeks): download all Mercurial repos from Bitbucket and store them uncompressed (raw .hg directories)
- Stage 2: decide the best strategy for compressing all of this stuff (possibly by grouping forks together - can be automated)
- Stage 3: publish an archive, for the entire world to use (what if other open source projects missed some important bit during their migration?)

List of repos:
all-repos (huge file; all repos until June 16, 2020)
hg-repos (huge file; list of Mercurial repos, which I'll divide in smaller chunks)
[ fields: hg/git, user/repo, creation date, last updated, optional url ]

Let's divide these into manageable chunks:

Code Select Expand


split -l 10000 --numeric-suffixes hg-repos hg-repos-


hg-repos-00 (a1ex: 91.23% downloaded, 8.77% errors, 2.82 MiB average) (Levas: started)
hg-repos-01 (a1ex: 92.38% downloaded, 7.62% errors, 4.16 MiB average)
hg-repos-02 (a1ex: 92.08% downloaded, 7.92% errors, 6.81 MiB average)
hg-repos-03 (a1ex: 92.76% downloaded, 6.43% errors, 0.81% todo)
hg-repos-04 (a1ex: 31.84% downloaded, 2.16% errors, 66.00% todo)
hg-repos-05 (critix: 23.59% downloaded, 1.78% errors)
hg-repos-06 (critix: 27.30% downloaded, 2.02% errors)
hg-repos-07 (Audionut: 23.08% downloaded, 1.96% errors) (Danne: started)
hg-repos-08 (Danne: started)
hg-repos-09 (Danne: started)
hg-repos-10 (Danne: started)
hg-repos-11 (names_are_hard: started)
hg-repos-12 (names_are_hard: started)
hg-repos-13 (names_are_hard: started) (Audionut: 25.35% downloaded, 1.18% errors)
hg-repos-14 (a1ex: 68.22% downloaded, 0.37% errors, 31.41% todo)
hg-repos-15 (a1ex: 91.68% downloaded, 0.94% errors, 7.38% todo)
hg-repos-16 (a1ex: 12.01% downloaded, 0.43% errors, 87.56% todo)
hg-repos-17 (Audionut: 24.25% downloaded, 3.39% errors) (kitor: 14.37% downloaded, 0.43% errors)
hg-repos-18 (Audionut: 22.46% downloaded, 0.51% errors) (kitor: 14.57% downloaded, 0.21% errors)
hg-repos-19 (Audionut: 15.49% downloaded, 0.27% errors) (kitor: 10.17% downloaded, 0.14% errors)
hg-repos-20 (Audionut: 16.87% downloaded, 0.21% errors)
hg-repos-21 (Audionut: 20.56% downloaded, 0.26% errors)
hg-repos-22
hg-repos-23
hg-repos-24 (Edit and fix link //Audionut)

Only ML forks, as identified earlier in this thread (caveat: different file format):
ml-forks (a1ex: 471/540 downloaded; the others had errors)

The hacky download script:

Code Select Expand


#!/bin/bash
# usage: [bash] ./download_bitbucket_hg_repos.sh hg-repos-00   # or 01, 02 etc

for f in $(cat $1 | cut -d ' ' -f 2); do
  echo
  echo "Processing $f ..."

  # skip already-downloaded repos (for which we have a valid .commits file)
  if [ ! -f $f.commits ]; then
    # hg clone, don't prompt for user/password/whatever,
    # and don't update the working directory (we only need the .hg folder)
    # this may fail (404 on some repos, auth needed on others, etc)
    if hg clone --config ui.interactive=false -U -- https://bitbucket.org/$f $f; then   # HTTPS version, slower, but works out of the box
    #if hg clone --config ui.interactive=false -U -- ssh://[email protected]/$f $f; then   # SSH version, faster (thanks kitor), but requires additional setup ("You need to add your ssh public key to bitbucket. And run HG once by hand (without disabling interactive shell) to accept remote ssh pubkey.")
      # for each successfully-cloned repo, we build a list of commits (hashes only)
      # this lets us identify the contribution of every single fork
      # this may be used to decide the best compression strategy, after downloading all of the stuff
      (cd -- $f && hg log --template "{node}\n" > ../../$f.commits)
    else
      # "hg clone" failed for some reason
      # todo: report status? (404 or whatever)
      # these repos will be retried if you run the script twice
      echo "$f" >> hg-clone-errors.txt
    fi
  fi
done


I'm keeping the script very simple, to avoid potential trouble. To parallelize, you should be able to start as many instances as you want (each instance with its own repo list, of course). These instances can probably work in the same directory (not thoroughly tested, but as long as each instance processes a different list of repos, it should be fine, I think). You can stop and restart each instance as needed, by closing the terminal (CTRL-C will only stop the active process, very likely a "hg clone", resulting in a false error report).

You will need:
- a working directory with enough free space available (assume 10 20 MiB / repo on average, although it's likely less)
- one or more lists of repos (download from above).

That's it, now you are ready to run the script. Guess: 2...4 threads might help (not 100% sure).
2-3 threads are probably best, depending on how powerful your machine is. Watch out for HDD/SSD thrashing!
If your machine starts to be unresponsive, you may have too many threads running. Stop or pause some of them!

Caveat: never edit Bash scripts while they are running!

Script outputs:
- For each repo (e.g. hudson/magic-lantern):
  - user directory (hudson/)
  - project directory (hudson/magic-lantern/)
  - .hg folder (hudson/magic-lantern/.hg/ - possibly hidden by default, depending on your file browser)
  - a list of commits (hudson/magic-lantern.commits)
- For all repos:
  - hg-clone-errors.txt (hopefully obvious)

Once you start running the script, let me know what subset(s) of repos you are downloading; we all should know what everybody is downloading, what has been downloaded, and so on. Ideally, each repo (or set of repos) should be downloaded by at least two participants, just in case. I'll edit this post to keep it up to date.

Afterwards, you can create an archive of what you downloaded, with:

Code Select Expand


tar -cJvf commits.tar.xz */*.commits                                 # commit lists only
tar -cf - */*/.hg | xz -9e --lzma2=dict=1536Mi -c - > repos.tar.xz   # hg repos only (slow, RAM-intensive!)


Let's hope the resulting files will be small enough to exchange them.

Aren't others already taking care of this?
(or, aren't you supposed to port ML on the 13D Mark 7, instead of this bull***?)

Unfortunately, the well-known archive.org -- who, to their credit, saved the day countless times -- apparently didn't do a great job on this one (see earlier report from aprofiti).

These guys did better (see archive.softwareheritage.org) but...

1) they are one single point of failure (so, a second backup should never hurt)
2) we have noticed a little issue in their archive: the Mercurial changeset IDs were not kept, at least in our repo (example a few posts above)
3) the deadline is coming!


OK, OK. Why didn't you start earlier?

As you may or may not know, this is a hobby project for us. Translation: if there is time to spare, the project advances. If there is not, the project stalls (or... disappears). And, as surprising as it may sound, we also have to eat from time to time (alongside with our families). The "default" way to put food on the table is to get a job, which can have the side effect of not leaving much time for hobbies (especially during these difficult times).

As mentioned earlier, I actually took a 2-week holiday in order to perform this migration and to catch up with other hobby projects (no travel plans or anything like that). That's when I've started to research this issue and noticed the need for a good backup of all those repos (not only ours).

The good side: after one week of messing around, I think I've got a plan that has at least some chances to work

So, let's try!

Awesome!

Will prepare scripts and file lists for tonight, then. Repo list reached 2018, btw

You look up the changeset ID (6c6f37e9adfc). That identifies the exact commit you were working with (the exact version, if you want).

You also need to identify the repository containing this changeset (in this particular case, Danne's), clone it, "hg update 6c6f37e9adfc -C" and compile.

If there is a + on the changeset ID, that means there were local changes that were not yet committed. These changes are also included in the binaries, both for autoexec.bin and for modules (see this PR, but link will be broken after July 1st).



Besides the main repo, I'm also downloading all ML forks I could identify^*), and planning to provide an archive. I can also identify unique commits from each fork, to review them later.

^*) I'm still trying to download all Mercurial repositories from Bitbucket, to make sure I don't miss anything, but only managed to get ~ 40.000 repos out of 250.000 in one week, and there are only 2 weeks left. There is also https://archive.softwareheritage.org/, but... in their archive, the changeset IDs do not match (example: their unified vs bitbucket).

Maybe some parallelization is needed on my side

Anyone with good network connectivity, some hundred(s) of GiB of free space, able to run Bash scripts, recent Python (3.7+) and willing to leave their computer(s) running for the next two weeks? I'll provide a list of repos^*) (divided among all participants, if that makes things easier) and download scripts. You will also need to keep an eye on the scripts, as Mercurial sometimes crashes or asks for usernames or whatever, but I'll try to sort it out.

^*) Still unable to get the list of repos created after ~ 2017.

Got it, so it's only accessible to me and Heptapod admins for now.

Will keep you posted.

Some progress regarding foss.heptapod.net: I have applied for hosting here and they have approved the request. They already imported the main repository, issue list and "internal" open pull requests (i.e. those that did not come from forks).

A bit of surprise: our case is handled by... the author of this article:

http://www.legi.grenoble-inp.fr/people/Pierre.Augier/mercurial-as-a-great-version-source-control-management-tool-in-academics.html

I had no idea he's actually a team member there

Now, we should all review their workflow, which is described here:
https://heptapod.net/pages/faq.html#workflow
https://octobus.net/blog/2019-09-04-heptapod-workflow.html

Existing contributors are already able to login with their Bitbucket and/or Github credentials, but I'm not sure if the repository is visible to them yet. Let me know what happens if you try to login with this link:
https://foss.heptapod.net/magic-lantern/magic-lantern



Also made some progress with the git workflow, using the git-remote-hg extension:

Code Select Expand


hg clone https://bitbucket.org/hudson/magic-lantern
git clone hg::magic-lantern/ magic-lantern-git
cd magic-lantern-git/
git checkout branches/unified
# edit something
# commit
git push


Et voilà, the new commit appears on the right branch in the (local) Mercurial repository.

So, there is hope for lowering the barrier for contributors who prefer Git

With the script from reply #40 (yes, by modifying bitbucket-hg-exporter).

I have yet to get a complete list, as the script kept stopping somewhere in the middle (edited it a few times to address this, but...)

Will upload mine once it's complete (guess: 1-2 days).

Edit: this time it worked better, currently at about 190000 hg and 600000 git repos. They appear to be reported by their creation date (currently at summer 2016).

Edit2: crashed again somewhere at 2017 (about 207000 hg + 825000 git).

[although this may not hold true for complex histories.]

hg-fast-export can't handle unnamed heads

Author of bitbucket-hg-exporter recommends hg-export-tool in the FAQ.

Currently trying a conversion with hg-git, as they say it should be lossless:

The Hg-Git plugin can convert commits/changesets losslessly from one system to another, so you can push via a Mercurial repository and another Mercurial client can pull it. In theory, the changeset IDs should not change, although this may not hold true for complex histories.

Code Select Expand


hg clone https://bitbucket.org/hudson/magic-lantern
mkdir magic-lantern-git
cd magic-lantern-git
git init
cd ..
cd magic-lantern
hg up unified
hg bookmarks hg
hg push ../magic-lantern-git


Result: only the "unified" branch got exported, as "hg". Fixme: gitk crashes on that repo. Cloning back to Mercurial appears to work, but changeset ID was not preserved.

Unified tip on Bitbucket: 7a3b5fa Ghost image: further cleanups and fixes
Unified tip after conversion to git and back to hg: 12839:68c3b9b53f35 Ghost image: further cleanups and fixes

Last commit with the same changeset ID:

77:4845885c4d68 (default) Added tag before-dryos-split for changeset b3ac1159ee7c

Not good. Conversion to git is definitely not lossless, at least not with this tool.

The others:
- git-hg and git-hg-again: both failing with:

Code Select Expand


Traceback (most recent call last):
  File "/usr/local/lib/git-hg/fast-export/hg-fast-export.py", line 405, in <module>
    options.statusfile,authors=a,sob=options.sob,force=options.force))
  File "/usr/local/lib/git-hg/fast-export/hg-fast-export.py", line 318, in hg2git
    if not verify_heads(ui,repo,heads_cache,force):
  File "/usr/local/lib/git-hg/fast-export/hg-fast-export.py", line 299, in verify_heads
    (_,_,_,_,_,_,branch,_)=get_changeset(ui,repo,h)
  File "/usr/local/lib/git-hg/fast-export/hg2git.py", line 71, in get_changeset
    node=repo.lookup(revision)
  File "/usr/lib64/python2.7/site-packages/mercurial/localrepo.py", line 1663, in lookup
    node = scmutil.revsymbol(self, key).node()
  File "/usr/lib64/python2.7/site-packages/mercurial/scmutil.py", line 618, in revsymbol
    raise error.RepoLookupError(_("unknown revision '%s'") % symbol)
mercurial.error.RepoLookupError: unknown revision '��{b~3v���0<�a�a��'


Epic fail.

hg-export-tool: it appears to handle branches one by one. Stopped at "/raw_recc-added-121-and-11751-aspect-rati-1382109783386". TODO: try names_are_hard's mappings.

Sourcehut's invertbucket:
For the repository, it worked nearly out of the box. Had to do a minor edit, as my Bitbucket username is a1ex, but the repository is stored on hudson.

Code Select Expand


printf "\n%s\n" "Fetching your Bitbucket repos..."
next="$bbapi"/repositories/"hudson"


So, if all else fails, at least we have a working Mercurial mirror:
https://hg.sr.ht/~ml/magic-lantern

Having some hiccup when cloning it back, from two different networks, but found a workaround:

Code Select Expand


hg clone -r 1000 https://hg.sr.ht/~ml/magic-lantern
cd magic-lantern/
hg pull -r 5000
hg pull -r 10000
hg pull -r 15000
hg pull
hg update unified
hg id # expecting 7a3b5fa3f4c6


Checked the changeset IDs from the tip of unified, crop_rec_4k and digic6-dumper, and they look OK to me.

Migrating the issue tracker, on the other side, didn't work:

Code Select Expand


Migrate issues for hudson/magic-lantern? [y/n] y
Imported https://todo.sr.ht/~ml/magic-lantern/null
Importing comments...
parse error: Invalid numeric literal at line 1, column 10
Failed to import comment:
parse error: Invalid numeric literal at line 1, column 10


Only half of it, i.e. what will be deleted. I assume most of these repos are small (guess: 10MiB on average?), so I guess the entire thing should fit on an external HDD that's sitting around. If it won't fit, I can delete large blobs (can be automated), attempt some deduplication (skip clones that bring no contributions on their own), or skip large repos that already found a new home (looking them up can probably be automated as well).

Or, at least, it could be a fun exercise. So far, it looks like this:

Code Select Expand


for f in $(cat all-repos.txt | grep '^hg ' | cut -d ' ' -f 2); do
  echo $f
  if [ ! -f $f.tar.bz2 ]; then
    if hg clone -U https://bitbucket.org/$f $f; then    # some repos may have been deleted?
      tar -cvjSf $f.tar.bz2 $f
      rm -rf $f
    fi
  fi
done


So far: 400 repos downloaded in ~ 40 minutes, total 1.7 GiB, less than 5 MiB/repo average.

Status update:

- my fix was merged into bitbucket-hg-exporter, so you no longer have to patch the source code

- the "big" script downloaded the entire hudson repo (took about 50 hours, give or take) and started to download the smaller forks (15 out of 540 in ~ 10 hours? it might finish before the deadline if we are lucky ). I'll probably stop it and just "hg clone" the forks, without saving the metadata.

- the "small" script ran for 20 hours and identified 330782 git repos and 161318 hg ones. It appears to have stopped prematurely for who-knows-what reason, so I'll try to figure out what happened. Thinking to "hg clone" each of the Mercurial ones, as a low-priority background task, and provide an archive somewhere (just in case Software Heritage misses anything).

I'll give that a try since it sounds useful whatever the solution ends up being.

It will fail out of the box, but if you apply my patch (linked above), it should work.

Current status: been running that script for ~ 48 hours, about 100.000 files (2 GiB) downloaded, still running. Several error messages about issue attachments, but downloading these attachments manually does work. Example:

Code Select Expand


Thread 0 (a1ex): hudson/magic-lantern: ERROR: Unexpected response code 500 for endpoint repositories/hudson/magic-lantern/issues/2753/attachments/CRASH00.LOG
Thread 0 (a1ex): hudson/magic-lantern: ERROR: Unexpected response code 500 for endpoint repositories/hudson/magic-lantern/issues/2753/attachments/CRASH01.LOG
Thread 0 (a1ex): hudson/magic-lantern: ERROR: Unexpected response code 500 for endpoint repositories/hudson/magic-lantern/issues/2753/attachments/log000.log
Thread 0 (a1ex): hudson/magic-lantern: ERROR: Unexpected response code 500 for endpoint repositories/hudson/magic-lantern/issues/2753/attachments/log001.log


Some attachments were successfully downloaded, others not. Didn't investigate yet.



The other script (list of all Bitbucket repos) identified 36700 repos at the time of writing. Will re-check tomorrow morning.

[all]

Another important task is, in my opinion, an archive of all current Mercurial repositories (not necessarily related to ML). Some of them might be unmaintained projects, that could be still interesting for someone. I don't currently have a list of these repos [...]

Here's a quick hack to get a list of all the repositories stored on Bitbucket (replace the "main" of bitbucket_hg_exporter):

Code Select Expand


def main():
    #~ project = MigrationProject()
    # no auth needed
    # force pagelen=100 (default 10) for speed reasons
    status, json_data = bbapi_json("https://bitbucket.org/api/2.0/repositories/", None, {'pagelen': 100})
    while True:
        for repo in json_data["values"]:
            print(repo["scm"], repo["full_name"], repo["created_on"], repo["updated_on"], repo["website"])
            # many other parameters available:
            #~ for k,v in repo.items():
                #~ print(k, v)
        sys.stdout.flush()
        if 'next' in json_data:
            next = json_data['next']
            status, json_data = bbapi_json(next, None, {'pagelen': 100})
            while json_data is None or "values" not in json_data:
                print("retrying", file=sys.stderr)
                status, json_data = bbapi_json(next, None, {'pagelen': 100})
        else: break


Not very fast; about 1000 repos / minute at first sight. Let's see where this ends up. Octobus found about 250.000 Mercurial projects; this could be useful for cross-checking their list. A second backup of all these repos (plain "hg clone" without metadata) shouldn't hurt

edit: updated the script to retry failed requests.

It's from the point of view of someone who already knows Mercurial, and finds Git more complicated.

Right. I happen to know Mercurial, and got used to it while developing ML, but have nearly zero experience with Git, and I somehow doubt I'm the only one in this community. But, surely there are many others who started by learning Git and find it easier to work with.

Old habits die hard, on both sides, of course, which is why I'm looking for ways to provide a dual workflow. So far, I've found these:
- hg-git (the main repo should be git; not sure how "lossless" the conversion will be)
- git-hg-again (bidirectional hg-git? you use "git hg clone/fetch/push/pull" to interact with a Mercurial repo)
- git-hg (the main repo should be hg; you use "git-hg clone/fetch/pull/checkout/push" for interacting with it)
- Git-hg Mirror (for syncing a hg and a git repo? works either as web service or as self-hosted Windows service)

See also this Stack Overflow page.

TODO: evaluate these.

Regardless of whether Mercurial is technically better or worse, I will say that because Git has won, and Mercurial is slowly dying, there may be a benefit to picking a Git hosting provider.

It also feels like accepting the defeat. Though, I wouldn't exclude a git mirror on github/gitlab and a hg mirror on heptapod (most likely), keep them in sync somehow, and accept contributions from both sides.



One important thing to note about Heptapod is that, at least for now, they don't support personal forks. So, the development is more or less centralized, and all the contributors will have to be granted development access.

It works somehow like this: anyone with the Developer permission can publish draft changes, using the so-called topics (some kind of lightweight branches, at first sight closer to git branches, I guess). On these draft changes, history is editable, so I should be able to polish them as I see fit, before making them "public" (or "set in stone").

There's also support for the evolve extension, apparently another thing worth trying. In the past, I used to try not only to have readable code, but also readable commits (i.e. to squeeze all those "typo/fixup" commits before making them public, as I used to read the code history quite often). But this was time-consuming; curious to see whether the evolve extension can help.

There's also hg absorb in recent Mercurial (I wasn't aware of it), which also seems to fit my previous workflow pretty well. Apparently it's an alternative to evolve, but I need to actually try these things before drawing a conclusion.

[all]

I am unsure if you really need to have a backup from all forks.

Some of these forks might have useful commits that were not (yet?) submitted as pull requests; possibly fixing some personal issue of the author. Good luck identifying them manually. Yours is one out of over 500

But if you have a backup of all these repos, you could write a script to identify those commits that are not present in the main repository. These commits can be reviewed later; maybe some of them could end up useful. Maybe some of them were referenced here on the forum, and forgotten. If one of us will revisit such a discussion, there will be a way to find those old commits in the archive.

Mine for e.g. might be skipped if 70D and 100D got merged.

These two are already in the main (hudson) repository, and on the main download page. Just not in "unified" yet.

So, they can be skipped without further action

Idea: I suggest merging PRs (read: not all) as much as possible either before or short after leaving bitbucket.

I wouldn't do any further development (including PR merging) before finding a new home for the repository (whether it will be heptapod, hg.sr.ht, github, gitlab, kallithea or whatever).

Even if I'm currently on holiday, I won't be available 24/7 for this task, so I still need to prioritize. Having copies of all the data before it's deleted, is essential in my opinion. Deciding what to merge, what needs further cleanup, running tests and so on, can be done afterwards, without time pressure.

Another important task is, in my opinion, an archive of all current Mercurial repositories (not necessarily related to ML). Some of them might be unmaintained projects, that could be still interesting for someone. I don't currently have a list of these repos, and I'm not aware of any existing efforts in this direction (my google-fu is not the best), but already started to research the topic. Edit - found some:
- https://octobus.net/blog.html (which brings quite a few bonus points for heptapod)
- https://archive.softwareheritage.org/browse/origin/directory/?origin_url=https://bitbucket.org/hudson/magic-lantern
- https://archive.softwareheritage.org/browse/origin/directory/?origin_url=https://bitbucket.org/coutts/5dplus

And yes, I'd rather complete that before the July 1st deadline, and focus on PRs afterwards, as time permits.

Please don't make THE decision to selfhost a repo.

No decision has been made yet. I'd appreciate some arguments for and against the alternatives mentioned throughout this thread (not just the self-hosting ones, but *all* of them).

I personally think it's time to move to git as it seems to be more widely adopted

I wouldn't rush to move to git either. One particular reason for sticking with Mercurial is that, throughout the forum and the documentation, there are several places mentioning Mercurial changeset IDs. While I was actively developing, I used to click these links very often, so, in my opinion, they are valuable. If we switch to another Mercurial hosting, updating these links would be doable with a find/replace. Is there a relatively straightforward way to do the same with a Git clone of our existing repository? If yes, how?

Also, besides my lack of experience with git, here's a possibly more objective reason:
http://www.legi.grenoble-inp.fr/people/Pierre.Augier/mercurial-as-a-great-version-source-control-management-tool-in-academics.html

Keep in mind that article was written before Bitbucket announced their plans to delete Mercurial repos.

BTW, if there is a way to provide a hg+git workflow, I'd look into that.

[Important]

Found this: https://github.com/philipstarkey/bitbucket-hg-exporter

It exports a Bitbucket repository and all of its forks, as JSON, including issues, comments and so on. Requires login, but is able to download any repo where you have read access (i.e. any public repos). Only one thread per user allowed (there are rate limits). Optional export to Github (didn't try, but there is an example here).

Important: to use the script, you will need to make this change, or wait for my PR to be merged upstream.

Currently trying to export the main repository (hudson/magic-lantern) and all of its forks. Been running for over 36 hours, still downloading, about 500 MiB downloaded, no idea how much is left (hours? days? weeks? maybe more?). Cannot download other repos until this is finished.

Forks identified by the script (download in progress; duplicates removed from the log):

Code Select Expand


Finding all forks of hudson/magic-lantern
Finding all forks of a1ex/magic-lantern
Finding all forks of arm_indy/magic-lantern-for-600d-t3i
Finding all forks of magiclantern/magic-lantern
Finding all forks of yyfrankyy/magic-lantern
Warning: repository yyfrankyy/magic-lantern might have been deleted (404).
Finding all forks of dbaleckaitis/magic-lantern
Finding all forks of arm_indy/magic-lantern-for-60d
Finding all forks of piersg/magic-lantern-pg-fooling
Finding all forks of deti/magic-lantern
Finding all forks of alins/magic-lantern-for-500d
Finding all forks of maclema/ml500d
Warning: repository maclema/ml500d might have been deleted (404).
Finding all forks of roaldfre/magic-lantern-for-500d
Finding all forks of adijiwa/magic-lantern-for-500d
Finding all forks of sudharsh/magic-lantern
Finding all forks of sztupy/magic-lantern-for-60d
Finding all forks of nmargono/magic-lantern
Finding all forks of KholdStare/magic-lantern-unified-experiments
Finding all forks of roaldfre/magic-lantern-unified
Finding all forks of qianc/magic-lantern
Finding all forks of teegee/magic-lantern
Warning: repository teegee/magic-lantern might have been deleted (404).
Finding all forks of sztupy/magic-lantern
Finding all forks of up4/magic-lantern
Finding all forks of 600dplus/magic-lantern-for-600d_old
Finding all forks of CMatters/magic-lantern
Finding all forks of gheja/magic-lantern-gheja-dev
Finding all forks of supermaz/magic-lantern-maz-60d
Finding all forks of OtherOnePercent/tragic-lantern
Finding all forks of chrisball/magic-lantern
Finding all forks of Sticks/magic-lantern-stickshack
Finding all forks of dmgerman/ml
Finding all forks of Sticks/magic-lantern-menu-system
Finding all forks of j_l/magic-lantern
Finding all forks of jplxpto/magic-lantern
Finding all forks of ubbut/magic-lantern
Finding all forks of 600dplus/magic-lantern-for-600d
Finding all forks of 0xAF/magic-lantern-5dc-port-wip
Finding all forks of 600dplus/magic-lantern-5dc-port-wip-updated
Finding all forks of raymondlo84/magic-lantern
Finding all forks of maxchen/magic-lantern
Finding all forks of rufustfirefly/magic-lantern-5d3
Finding all forks of 600dplus/magic-lantern-finding-flash-prop
Finding all forks of 600dplus/magic-lantern-features_custom_modes
Finding all forks of rufustfirefly/magic-lantern
Finding all forks of teepean/magic-lantern
Finding all forks of OtherOnePercent/tragic-lantern-6d
Finding all forks of salsasepp/tragic-lantern-6d
Finding all forks of Mayocomplet/tragic-lantern-6d
Finding all forks of Mayocomplet/tragic-mayo-6d
Finding all forks of GregoryOfManhattan/tragic-lantern-6d
Finding all forks of andy600/tl50d
Finding all forks of miguelbrito/tragic-lantern-7d
Finding all forks of M1ke_M1ke/tragic-lantern-6d-m1ke-fork
Finding all forks of feureau/tragic-lantern-6d
Finding all forks of anthony_molesphini/tragic-lantern-6d
Finding all forks of vinlyanimate/raw-6d
Finding all forks of vinlyanimate/raw-60d-110
Finding all forks of EdK/tragic-lantern-6d
Finding all forks of hjdaniel_sun/tragic-lantern-6d
Finding all forks of 600dplus/magic-lantern-fixes
Finding all forks of 600dplus/magic-lantern-600d-photoinfodisp
Finding all forks of maxchen/magic-lantern-unified
Finding all forks of 600dplus/magic-lantern-fix
Finding all forks of magicpriest/magic-lantern
Warning: repository magicpriest/magic-lantern might have been deleted (404).
Finding all forks of sl0w0rm/magic-lantern-af-ptp
Finding all forks of OtherOnePercent/tragic-lantern-2.0
Finding all forks of datadogie/tragic-lantern-2.0
Finding all forks of igor1111111/tragic-lantern-2.0
Finding all forks of grishkavideo/tragic-lantern-2.0
Finding all forks of barbara_brockhauser/tragic-lantern-2.0
Finding all forks of BrianPrz/tragic-lantern-2.0
Finding all forks of 600dplus/magic-lantern-600d-flexinfo
Finding all forks of 600dplus/magic-lantern-menu-shortcuts
Finding all forks of 600dplus/magic-lantern-auto-audio-override
Finding all forks of simmsa/magic-lantern
Finding all forks of trsaunders/magic-lantern
Finding all forks of 600dplus/magic-lantern-5dc-update-from-0xaf-wip
Finding all forks of raeez/magic-lantern
Finding all forks of ak_ss/magic-lantern
Finding all forks of marekk17/60d-silent-dng
Finding all forks of marekk17/magic-lantern
Finding all forks of minimimi/magic-lantern-filer
Finding all forks of marekk17/magic-lantern-1
Finding all forks of marekk17/magic-lantern-2
Finding all forks of smeangol/smeangols-sneaky-magic-lantern
Finding all forks of 600dplus/magic-lantern-600d-raw-feat
Finding all forks of marekk17/magic-lantern-3
Finding all forks of marekk17/magic-lantern-4
Finding all forks of dhilung/magic-lantern
Finding all forks of flameeyes/magic-lantern
Finding all forks of GregoryOfManhattan/magic-lantern
Finding all forks of krashnik/magic-lantern
Finding all forks of sonic74/magic-lantern
Finding all forks of minimimi/magic-lantern-filer-multiselect
Finding all forks of freelyfree/cigam-nretnal
Finding all forks of mucher/magic-lantern
Warning: repository mucher/magic-lantern might have been deleted (404).
Finding all forks of marekk17/magic-lantern-5
Finding all forks of marci1/magic-lantern
Warning: repository marci1/magic-lantern might have been deleted (404).
Finding all forks of marci1/magic-lantern-marci1
Warning: repository marci1/magic-lantern-marci1 might have been deleted (404).
Finding all forks of williamh/magic-lantern-5d3
Warning: repository williamh/magic-lantern-5d3 might have been deleted (404).
Finding all forks of g3gg0/to-be-deleted
Finding all forks of davespana13/magic-lantern
Warning: repository davespana13/magic-lantern might have been deleted (404).
Finding all forks of HKServ/magic-lantern-hksfork
Finding all forks of HKServ/magic-lantern-notcc-hksfork
Finding all forks of sc1ence/magic-lantern
Finding all forks of minDark/magic-lantern-unificat
Finding all forks of pravdomil/magic-lantern-hack
Finding all forks of reddeercity/magic-lantern-hack
Finding all forks of niklasRde/magic-lantern
Finding all forks of emc/magic-lantern
Finding all forks of 600dplus/magic-lantern-600d-60d-1100d-memcheck
Finding all forks of marekk17/magic-lantern-6
Finding all forks of marekk17/ml_60d
Finding all forks of marekk17/magic-lantern-7
Finding all forks of marekk17/magic-lantern-8
Finding all forks of YMP/magic-lantern-afma-improvement
Finding all forks of YMP/magic-lantern-afma-improvement-1
Finding all forks of canon_amateur/magic-lantern
Finding all forks of aluker/magic-lantern
Finding all forks of frantony/magic-lantern.makeall
Finding all forks of pbernardi90/magic-lantern
Finding all forks of frantony/magic-lantern.checkpatch
Finding all forks of sblest/magic-lantern
Finding all forks of theunkn0wn/magic-lantern
Warning: repository theunkn0wn/magic-lantern might have been deleted (404).
Finding all forks of stealth_tramp/magic-lantern
Finding all forks of frantony/magic-lantern.check-bot
Finding all forks of masgro/magic-lantern
Finding all forks of circuitman/magic-lantern
Finding all forks of miguelbrito/magic-lantern
Finding all forks of shumonsaito/magic-lantern-holy-grail-hack
Finding all forks of g3gg0/magic-lantern-mlv
Finding all forks of nicksta/magic-lantern-mlv
Warning: repository nicksta/magic-lantern-mlv might have been deleted (404).
Finding all forks of nicksta/magic-lantern
Warning: repository nicksta/magic-lantern might have been deleted (404).
Finding all forks of v8rrc/magic-lantern-v8rrc
Finding all forks of v8rrc/magic-lantern-1
Finding all forks of aham1234/magic-lantern
Finding all forks of marekk17/magic-lantern-9
Finding all forks of Chucho/magic-lantern-70d
Warning: repository Chucho/magic-lantern-70d might have been deleted (404).
Finding all forks of qmatrxp/magic-lantern
Finding all forks of guysmilez/magic-lantern
Finding all forks of keith_clinton/magic-lantern
Finding all forks of g3gg0/magic-lantern-module-hgdiff
Finding all forks of hope_young/magic-lantern
Finding all forks of OliRi/magic-lantern
Finding all forks of stevefal/magic-lantern-rawrec-cropmarks
Finding all forks of robin051/magic-lantern
Finding all forks of maxotics/maxotics-fork-magic-lantern
Finding all forks of tgregoire/magic-lantern-1100d
Warning: repository tgregoire/magic-lantern-1100d might have been deleted (404).
Finding all forks of chchrlam/magic-lantern
Finding all forks of a1ex/magic-lantern-dual-iso-preview
Finding all forks of g3gg0/magic-lantern-mlv_rec
Finding all forks of idealsceneprod/magic-lantern
Warning: repository idealsceneprod/magic-lantern might have been deleted (404).
Finding all forks of bnvm/magic-lantern
Warning: repository bnvm/magic-lantern might have been deleted (404).
Finding all forks of sodapopodalaigh/magic-lantern
Finding all forks of swinxx/ml_a
Finding all forks of M1ke_M1ke/magic-lantern-m1ke-fork
Finding all forks of M1ke_M1ke/magic-lantern-m1ke-fork-2
Finding all forks of marekk17/magic-lantern-10
Finding all forks of alextutubalin/magic-lantern
Finding all forks of marekk17/magic-lantern-11
Finding all forks of marekk17/magic-lantern-12
Finding all forks of marekk17/magic-lantern-13
Finding all forks of marekk17/magic-lantern-14
Finding all forks of marekk17/magic-lantern-15
Finding all forks of marekk17/magic-lantern-16
Finding all forks of monty87/magic-lantern
Warning: repository monty87/magic-lantern might have been deleted (404).
Finding all forks of mr3wan/magic-lantern
Finding all forks of bwilliams18/magic-lantern
Warning: repository bwilliams18/magic-lantern might have been deleted (404).
Finding all forks of dmilligan/magic-lantern
Finding all forks of Audionut/magic-lantern
Finding all forks of minimimi/magic-lantern-raw2dng_vc
Finding all forks of gregausina/magic-lantern
Finding all forks of knumat/magic-lantern
Finding all forks of kfullman/magic-lantern
Finding all forks of baldand/baldand-7d-magic-lantern
Finding all forks of nagyistoce/baldand-7d-magic-lantern
Finding all forks of HalSup/magic-lantern
Finding all forks of LebedevRI/magic-lantern
Finding all forks of jordancolburn/magic-lantern
Finding all forks of jordancolburn/magic-lantern-1
Finding all forks of somiic/magic-lantern
Warning: repository somiic/magic-lantern might have been deleted (404).
Finding all forks of somiic/magic-lantern-7d
Warning: repository somiic/magic-lantern-7d might have been deleted (404).
Finding all forks of rudison/magic-lantern
Finding all forks of houz/magic-lantern
Warning: repository houz/magic-lantern might have been deleted (404).
Finding all forks of romandvorak/magic-lantern-fork
Finding all forks of ayshih/magic-lantern-50d
Finding all forks of Phaenomena/magic-lantern
Finding all forks of feureau/magic-lantern
Finding all forks of mk11174/t5i
Finding all forks of mk11174/t5i-backup
Finding all forks of escho/escho-magiclantern
Finding all forks of jjwchoy/magic-lantern
Finding all forks of bnvm/magic-lantern-1
Warning: repository bnvm/magic-lantern-1 might have been deleted (404).
Finding all forks of bnvm/magic-lantern-2
Warning: repository bnvm/magic-lantern-2 might have been deleted (404).
Finding all forks of bnvm/magic-lantern-3
Warning: repository bnvm/magic-lantern-3 might have been deleted (404).
Finding all forks of bnvm/magic-lantern-4
Warning: repository bnvm/magic-lantern-4 might have been deleted (404).
Finding all forks of bnvm/magic-lantern-5
Warning: repository bnvm/magic-lantern-5 might have been deleted (404).
Finding all forks of adambernacki10/magic-lantern
Finding all forks of mofig/magic-lantern-mofig
Finding all forks of frenchiefilms/phil
Finding all forks of nanomad/phil
Finding all forks of viniciusatique/magic-lantern-mymod
Finding all forks of nviennot/magic-lantern
Finding all forks of vyskocil/magic-lantern-vyskocil
Finding all forks of viniciusatique/magic-lantern_1
Finding all forks of viniciusatique/movie-restart-5d3
Finding all forks of marekk17/ml123
Finding all forks of spaynix/magic-lantern
Warning: repository spaynix/magic-lantern might have been deleted (404).
Finding all forks of PhotoCat/magic-lantern
Finding all forks of mk11174/700d113
Finding all forks of arcxyz/magic-lantern-eos-1100d
Finding all forks of samontab/magic-lantern
Finding all forks of vehnae/polar-alignment-for-magic-lantern
Finding all forks of mofig/mofig-magic-lantern
Finding all forks of OtherOnePercent/backflow2
Finding all forks of Pelican/7d203
Finding all forks of mk11174/magic-lantern
Finding all forks of Pelican/7d.205
Finding all forks of nandoide/rec_plus
Finding all forks of nanomad/tragic-lantern-1100d
Finding all forks of g3gg0/magic-lantern-plot
Finding all forks of kuhn90/magic-lantern
Finding all forks of dingolina/magic-lantern-learn
Finding all forks of Cuntstructor/magic-lantern
Finding all forks of mlvisit/magic-lantern
Finding all forks of MarsBlessed/magic-lantern
Finding all forks of fabiomoreira716/magic-lantern
Finding all forks of mealcardga/magic-lantern
Finding all forks of holicowdev/magic-lantern
Finding all forks of hicom150/magic-lantern
Finding all forks of error454/magic-lantern
Finding all forks of rcdude/magic-lantern
Finding all forks of jpaana/ml
Finding all forks of felipe0328/magic-lantern
Finding all forks of Audionut/ml-fork
Finding all forks of Audionut/user-builds
Finding all forks of bennir/magic-lantern
Finding all forks of berlincount/magic-lantern-old
Finding all forks of Pelican/magic-lantern
Finding all forks of sodapopodalaigh/my-magic-lantern
Finding all forks of a_d_/magic-lantern-stubs
Warning: repository a_d_/magic-lantern-stubs might have been deleted (404).
Finding all forks of stevefal/magiclantern-mlvrec-video
Finding all forks of gkaizer/magiclantern
Warning: repository gkaizer/magiclantern might have been deleted (404).
Finding all forks of ggjjl1/magic-lantern
Finding all forks of Audionut/cleanup
Finding all forks of vance_lear/magic-lantern
Finding all forks of kfprod/5d3-mlv_rec-naming
Finding all forks of andy600/andy50d2
Finding all forks of Clark_Kent_/magic-lantern
Finding all forks of mk11174/magic-lantern-fullres
Finding all forks of mk11174/magic-lantern-fullres2
Finding all forks of berlincount/ml-100d-old
Finding all forks of mk11174/650d
Finding all forks of kurt_selander/magic-lantern
Finding all forks of zloe/zloe-magic-lantern
Finding all forks of mk11174/menu_flickering
Finding all forks of golem72/magic-lantern-frsp
Finding all forks of dpjpandone/magic-lantern
Finding all forks of spoonfly/magic-lantern
Finding all forks of SydTHackerr/magic-lantern
Finding all forks of dpjpandone/magic-lantern-1
Finding all forks of kichetof/magic-lantern
Finding all forks of dpjpandone/magic-lantern-2
Finding all forks of mk11174/bolt_trigger
Finding all forks of pravdomil/magic-lantern-lvexpo
Finding all forks of pravdomil/magic-lantern-sndcheck
Finding all forks of kfprod/mlv_rec-exfat-naming-suggestion
Finding all forks of Shiftx07/magic-lantern
Finding all forks of johnnyrun/magic-lantern
Finding all forks of dlrpgmsvc/magic-lantern-ok
Finding all forks of kunchenguid/magic-lantern
Finding all forks of roachhd/magic-lantern
Finding all forks of Vy7ka/magic-lantern
Finding all forks of r3dsm0k3/magic-lantern
Finding all forks of coutts/magic-lantern
Finding all forks of alessiocredendino/magic-lantern
Finding all forks of sundus2y/magic-lantern
Finding all forks of gmanfredax/magic-lantern
Finding all forks of nestor_cano/magic-lantern
Finding all forks of dietsche/magic-lantern
Finding all forks of nikfreak/magic-lantern
Finding all forks of jkruser/magic-lantern
Warning: repository jkruser/magic-lantern might have been deleted (404).
Finding all forks of smokegrafix/magic-lantern
Finding all forks of Dannephoto/magic-lantern_nikfreak
Finding all forks of OtherOnePercent/magic-lantern
Finding all forks of koenkooi/magic-lantern
Finding all forks of flykeysky/magic-lantern
Finding all forks of Dan0974/magic-lantern
Finding all forks of BrianPrz/magic-lantern
Finding all forks of maits18/magic-lantern-install-scrip5
Finding all forks of maits18/summon-arm
Finding all forks of rbrune/magic-lantern
Finding all forks of jwkerr/magic-lantern
Finding all forks of JLGW/magic-lantern
Finding all forks of csghone/magic-lantern
Finding all forks of drlapser/magic-lantern
Finding all forks of sodapopodalaigh/magic-lanterns
Finding all forks of niklastisk/magic-lantern
Finding all forks of rs68/magic-lantern-rs68-tests
Finding all forks of monograph/dof_display
Finding all forks of Pelican/recovery
Finding all forks of monograph/dof_display_development
Finding all forks of mario_pich/magic-lantern
Finding all forks of brisma_meihar_arsandi/brismaarsandi
Finding all forks of milank/magic-lantern
Finding all forks of chris_miller/ml-5d3-133
Finding all forks of Maqs/magic-lantern
Finding all forks of Audionut/mlv
Finding all forks of iancmcleod/magic-lantern
Finding all forks of homerobono/magic-lantern
Finding all forks of mk11174/700d114
Finding all forks of mk11174/magic-lantern-unified
Finding all forks of garry23/depth-of-field-with-diffraction
Finding all forks of juuh42dias/magic-lantern
Finding all forks of HilliBilli/magic-lantern
Finding all forks of fjriosp/1200d-dev
Finding all forks of DennisNeumann/magic-lantern
Finding all forks of garry23/final-dof
Finding all forks of garry23/colour-warnings-with-dof
Finding all forks of gyedafe/magic-lantern
Finding all forks of FabiLAP/magic-lantern
Finding all forks of cbob/magic-lantern
Finding all forks of lucashigor/magic-lantern
Warning: repository lucashigor/magic-lantern might have been deleted (404).
Finding all forks of kookiyoung/magic-lantern
Finding all forks of Licaon_Kter/magic-lantern
Finding all forks of buttbucket/magic-lantern
Finding all forks of ewinemiller/magic-lantern
Finding all forks of dinkin/magic-lantern
Finding all forks of 0th4rw4/magic-lantern
Finding all forks of andum1/magic-lantern
Finding all forks of nagyistoce/magic-lantern
Finding all forks of Marsu42/ml-pull
Finding all forks of ilias_kotsovolos/magic-lantern
Finding all forks of dpjpandone/magic-lantern-3
Finding all forks of dpjpandone/dpjpandone-eosm-fixes
Finding all forks of dpjpandone/eos-m-dpjpandone
Finding all forks of McGv/magic-lantern
Finding all forks of ziquan_xu/magic-lantern
Finding all forks of daniel_fort/magic-lantern
Finding all forks of danilaswift/magic-lantern
Finding all forks of Dannephoto/magic-lantern_dfort
Finding all forks of guang_feng_ren/magic-lantern
Finding all forks of racerxdl/magic-lantern
Finding all forks of SwaroopB/magic-lantern
Finding all forks of lurnid/magic-lantern
Finding all forks of roshansoni/magic-lantern
Finding all forks of Audionut/test
Finding all forks of hulasamsquanch/magic-lantern
Finding all forks of iremar_junior/magic-lantern
Finding all forks of co_ikebanat/magic-lantern
Finding all forks of jacobbieker/magic-lantern
Finding all forks of dinkin/magiclantern
Finding all forks of Yongyuth-P/magic-lantern
Finding all forks of VertigoPix/magic-lantern
Finding all forks of Licaon_Kter/magic-lantern-bulb_nd-info
Finding all forks of Licaon_Kter/magic-lantern-rec_ui_fix
Finding all forks of Licaon_Kter/magic-lantern-beta-info-button
Finding all forks of rongjuan_xie/magic-lantern
Finding all forks of houssi4ever/magic-lantern
Finding all forks of waza57/magic-lantern
Finding all forks of Dannephoto/magic-lantern_waza57
Finding all forks of reddeercity/magic-lantern_reddeercity_5d2-50d_4k-crop_rec
Finding all forks of tecgen/magic-lantern-old
Finding all forks of sirleneeraphael_per_golo/raphael
Finding all forks of piers92/magic-lantern
Finding all forks of lumenmikie/magic-lantern
Finding all forks of jayspear/magic-lantern
Finding all forks of marceljuchelka/magic-lantern
Finding all forks of everest20/magic-lantern
Finding all forks of Audionut/ml1
Finding all forks of am2rican5/magic-lantern
Finding all forks of dnlit/magic-lantern
Finding all forks of dnlit/magic-lantern-1
Finding all forks of xlee2004/magic-lantern-old
Finding all forks of tecgen/magic-lantern
Finding all forks of matias82/magic-lantern
Finding all forks of Q111Q/magic-lantern
Finding all forks of mk11174/1200d
Finding all forks of matias82/1200d
Finding all forks of rs68/magic-lantern-rolands-dm-spy-expriments-ros
Finding all forks of rs68/magic-lantern-rolands-debuging-bricked-5d2
Finding all forks of SinSpy/magic-lantern
Finding all forks of pangliang/magic-lantern
Finding all forks of sabujas/magic-lantern
Finding all forks of anDyIII/magic-lantern
Finding all forks of mothaibaphoto/magic_lantern_vold_fork
Finding all forks of Dasa36/magic-lantern
Finding all forks of seed93/magic-lantern
Finding all forks of mk11174/1200d_101
Finding all forks of mk11174/1200d_new
Finding all forks of blinkythebear/1200d_new
Finding all forks of sergiodelatorre/magic-lantern
Finding all forks of Dave_Raynor/magic-lantern
Finding all forks of apchen_stu/magic-lantern
Finding all forks of MAX_XIE/magic-lantern
Finding all forks of debmalyaroy/magic-lantern
Finding all forks of anhhung1303/magic-lantern
Finding all forks of josepvm/magic-lantern
Finding all forks of josepvm/magic-lantern-1
Finding all forks of josepvm/magic-lantern-2
Finding all forks of pbenco/magic-lantern
Finding all forks of Letshowgoes/magic-lantern
Finding all forks of carlosigcimd/magic-lantern
Finding all forks of greenbox/magic-lantern
Finding all forks of wildstray/magic-lantern
Finding all forks of andy600/ml
Finding all forks of reddeercity/magic-lantern
Finding all forks of reddeercity/magic-lantern_10-12bit
Finding all forks of George-Thurman/magic-lantern-stripped-and-confused
Finding all forks of Dannephoto/magic-lantern
Finding all forks of ccritix/magic-lantern-old
Finding all forks of ArcziPL/crop_rec_70d
Finding all forks of daniel_fort/magic-lantern-danne-fork
Finding all forks of nydahl98/magic-lantern_danne
Finding all forks of Jip-Hop/magic-lantern
Finding all forks of Dannephoto/magic-lantern_jip-hop
Finding all forks of stereozont/magic-lantern
Finding all forks of STNMPTWZero/magic-lantern
Finding all forks of DeafEyeJedi/magic-lantern
Finding all forks of andy600/magic-lantern
Finding all forks of andreykuban/magic-lantern
Finding all forks of Tafaube/magic-lantern
Finding all forks of eNnvi/magic-lantern
Finding all forks of Eosm_Dev/magic-lantern
Finding all forks of fgodfrey/magic-lantern-eosm-2.0.3
Finding all forks of rapido7097/magic-lantern-750d
Finding all forks of JohanVrn/ml
Finding all forks of Xav_L/magic-lantern
Finding all forks of pankajc8726/magic-lantern
Finding all forks of SinusPi/magic-lantern-cr2hdr-append-suffix-support
Finding all forks of reddeercity/magic-lantern_5d2-d4_4k-crop_rec
Finding all forks of JohanVrn/ml_test
Finding all forks of martinhering/magic-lantern
Finding all forks of bwaggo/magic-lantern
Finding all forks of Sleeperon/kirill
Finding all forks of shadowfax/magic-lantern
Finding all forks of emklap/magic-lantern
Finding all forks of qqluqq/magic-lantern
Finding all forks of chris_miller/ml-fork
Finding all forks of garry23/dof-focus.c-fix
Finding all forks of garry23/focus.c-dof-fix
Finding all forks of garry23/dof-fix-for-focus.c
Finding all forks of garry23/dof-changes-to-focus.c
Finding all forks of a1ex/ml
Finding all forks of johncheshire/magic-lantern-fork
Finding all forks of ehoutsma/magic-lantern
Finding all forks of Dannephoto/magic-lantern-erwinh
Finding all forks of xlee2004/magic-lantern
Finding all forks of larssteenhoff/magic-lantern
Finding all forks of Licaon_Kter/magic-lantern-1
Finding all forks of tizzleh/magic-lantern
Finding all forks of nikfreak/ml
Finding all forks of kichetof/magic-lantern-1
Finding all forks of nikfreak/ml-isoresearch
Finding all forks of Audionut/debug-lossless
Finding all forks of aprofiti/magic-lantern
Finding all forks of es_as/magic-lantern
Finding all forks of Dannephoto/magic-lantern_es_as
Finding all forks of t3r4n/magic-lantern
Finding all forks of sombree/magic-lantern
Finding all forks of sombree/magic-lantern-1
Finding all forks of shorst/magic-lantern
Finding all forks of bagaric/magic-lantern
Finding all forks of blubbblubb/magic-lantern
Finding all forks of monograph/adding-space_left
Finding all forks of DieHertz/magic-lantern
Finding all forks of drevilish/magic-lantern
Finding all forks of ids1024/magic-lantern
Finding all forks of andreasb242/magic-lantern
Finding all forks of samuel-carvalho/magic-lantern
Finding all forks of samuel-carvalho/magic-lantern-1
Finding all forks of samuel-carvalho/magic-lantern-2
Finding all forks of samuel-carvalho/magic-lantern-3
Finding all forks of samuel-carvalho/magic-lantern-4
Finding all forks of drewcollins/magic-lantern-dc
Finding all forks of gertoe/magic-lantern
Finding all forks of samuel-carvalho/magic-lantern-5
Finding all forks of thenickdude/magic-lantern
Finding all forks of bilal_fakhouri/magic-lantern-1
Finding all forks of bilal_fakhouri/magic-lantern-2
Finding all forks of mkacvinsky/magic-lantern
Finding all forks of mothaibaphoto/crop_rec_4k_mlv_snd_fork
Finding all forks of bouncyball/mlv_dump-fixes
Finding all forks of kitchentowel/magic-lantern
Finding all forks of calle2010/obsolete-magic-lantern
Finding all forks of Gr3g01/ml-cmos-cal
Finding all forks of daniel_fort/ml-cmos-cal
Finding all forks of Dannephoto/ml-cmos-cal
Finding all forks of ccritix/magic-lantern-200d
Finding all forks of noreboul/magic-lantern
Finding all forks of ccritix/magic-lantern
Finding all forks of Dannephoto/magic-lantern_critix
Finding all forks of Audionut/autoexpo
Finding all forks of 600dplus/magiclantern_eyefi_trick_cleanup
Finding all forks of ArcziPL/lossless_70d
Finding all forks of LeoYuanjieLi/magic-lantern
Finding all forks of vcrparts/magic-lantern-photobooth-hack
Finding all forks of calle2010/magic-lantern-77d
Finding all forks of Levas_EOS/magic-lantern
Finding all forks of kitorPL/magic-lantern
Finding all forks of PAPPPmAc/magic-lantern
Finding all forks of geojeff/magic-lantern
Finding all forks of DirkHussen/magic-lantern
Finding all forks of MisterGun/magic-lantern
Finding all forks of jmheder/vxworks_canon40d
Finding all forks of nydahl98/magic-lantern
Finding all forks of mrcmraja/magic-lantern
Finding all forks of nikil_ragav/magic-lantern
Finding all forks of ccritix/magic-lantern-card-fmt
Finding all forks of Ant123/magic-lantern-1



TODO:

1) All other Mercurial repositories from past and current contributors, that were not direct forks of hudson/magic-lantern.

There are clones of the magic-lantern repository that are not detected as forks (so, the automated script doesn't detect them). We need to identify those.

There are also standalone projects related to ML, such as ml-focus-pixels. We need to identify those as well.

Downloading all Mercurial repositories from the above 338 users (as identified by the script) might be close enough, and can be automated by hacking the Bitbucket exporter script. I can try to do that.

I need help identifying the contributions from users that were not identified in the above list. For example, xaint (https://bitbucket.org/xaint/magic-lantern-astro-module/) is one of them. OK, that repository is Git, but the point is - there might be ML-related contributions that were not direct forks of the main repository, AND their authors might not have a direct fork of the main repository.

Already downloaded:
- https://bitbucket.org/daniel_fort/ml-focus-pixels/ and its fork https://bitbucket.org/Dannephoto/ml-focus-pixels-danne/



2) All other interesting Mercurial repositories from Bitbucket, not related to ML, in particular, unmaintained projects (these will be lost forever, unless archive.org already crawled them).

As with the above, you can help by suggesting repository URLs.

Even better: if you know a list of all public Mercurial repositories, or if you can ask Bitbucket staff, or your favorite search engine, to supply such a list, or if you know anyone else trying to archive all Mercurial repos from Bitbucket, your help will be very appreciated



To check whether a Bitbucket repository is Mercurial or Git, click on Commits, then click on any commit, then - View raw commit. If it begins with HG changeset patch, it's a Mercurial repository. If it begins with an e-mail subject, it's a Git repository. If you know an easier way, please chime in.

If you can back up the repositories on your own, I'd also appreciate. What if my backups will fail?

@reddeercity: mrcrowl.hg is a Mercurial GUI for Visual Studio Code. Not relevant here. It's not a Mercurial hosting service, and it's not going to stop Bitbucket from deleting Mercurial repositories.

Got another suggestion for Mercurial hosting: foss.heptapod.net (fork of GitLab):
https://heptapod.net/

Test account (just created; empty for now):
https://foss.heptapod.net/ml/

According to their docs, it imports Bitbucket repositories, including issues and open pull requests:
https://heptapod.net/pages/tuto-bb-import.html

To create the repository, there's no option on the user interface; I need to apply with a public discussion on their issue tracker: https://heptapod.net/a-public-heptapod-for-free-and-open-source-software.html

I have no problems doing that, as long as there will be some consensus. There is also an option to self-host (as an alternative to Kallithea, mentioned earlier).

BTW, I happen to have a two-week holiday (starting tomorrow), hopefully enough to solve this issue.

Highly recommended, and - at least at first sight - very well explained, even for those not familiar with the topic.

One small problem (on our end): those graphs should be displayed on the camera screen, in real-time, not just on the computer monitor, in a simulation

To some extent, yes. I have proof of concept code for touchscreen emulation (now to remember how to run it).

Let's try to get a diagnostic log:
https://builds.magiclantern.fm/jenkins/job/startup-log/
https://builds.magiclantern.fm/jenkins/job/startup-log-mpu/

While the LED is blinking, you need to trigger some touchscreen events (for example, try navigating Canon menu with your finger).

@alex. I am building a directory tree C1, C2, C3, C4, and C5 upon installing ml firmware instead of including a prebuilt SETTINGS folder with subfolders. Nowhere in the chain can I get those folder to show up before I do a full restart of my camera.

Look at config_load() and config_preset_scan().

However, I recommend a different approach: include subfolders in the ML zip file (either empty, or with some default presets to get started for various use cases). Empty folders will be preserved when restoring ML after format. edit: nope, this won't work nicely when upgrading; you could end up with a mix of included presets and user's custom settings.

With your approach, if the user renames existing presets, new folders will be created, eventually overflowing the list. The side effect may be that some of the user's presets will disappear from the menu.

You could decide at the end of config_preset_scan(), i.e. after seeing how many subfolders you have, maybe check which ones (if any) were renamed, and then decide whether new presets have to be created, and how many.

Here we go:

Nitpick: I find it confusing that this feature arrived in the Movie menu. It affects all ML settings, not just video ones.

Man, you are moving fast!

Fast and furious. I'm a bit worried about merging those changes into mainline...

Custom setting directories are already implemented; they can be tied to the shooting mode selected at startup (but the mode dial is not present on EOS M, so usage is a bit non-intuitive), or to a key the user has to press at startup:
https://www.magiclantern.fm/forum/index.php?topic=8499

You can also create custom setting directories manually (as subdirectories under ML/SETTINGS on the card), and they will appear in the menu. They actually work as independent presets, but they only cover ML settings (not Canon ones). Switching to another preset requires rebooting the camera.

Changing arbitrary settings without reboot, even from Lua scripting, requires a major rework of the entire ML codebase. Long answer:
https://www.magiclantern.fm/forum/index.php?topic=19932.0

If you place 2 cameras on the left and right side of your monitor, sync them (not sure how), stereo-match their images and compute the middle view, could this work?

Optical flow could be used for morphing each of the two images into the other, with intermediate steps:
https://www.magiclantern.fm/forum/index.php?topic=20999.0

That particular procedure is not fast enough for real-time video, but it can probably be sorted out (lower resolution, GPU implementation etc). If you can capture (or find online) two images from left and right, ideally with similar FOV and DOF, I can compute the middle image to see if the approach has any chances of working.

[rolling shutter becomes huge with these 2 registers override]

2 registers bump the gain, without dynamic range penalty!
combined they give about 1.66 stop of extra light, without sacrificing dynamic range, works on all iso's.
Not sure what the trickery is done in the background, but rolling shutter becomes huge with these 2 registers override.

Confirmed on 5D3 - really interesting. It appears to reduce the frame rate as well.

ADTG[23]: default 0xF00, setting to 0x200 or some other magic values will cut the frame rate in half*, doubling the exposure time.
ADTG[24]: default 0x2d1e, setting to 0x1d1e will cut the frame rate in half*, doubling the exposure time.
* changing both registers together will result in 1/3 of the original frame rate (e.g. 24 -> 8 fps), and 3x exposure time.

This could explain the results, as the dynamic range does not change significantly with shutter speed.

In addition, changing any of those 2 registers in photo mode appears to have no effect (possibly because of the mechanical shutter), but changing both of them together results in garbled image (possibly timing issues because of slower readout). With FRSP, changing any or both of those 2 registers results in garbled image (?!)

Definitely worth a closer look, in particular at higher ISOs sorry, false alarm