Bitbucket set to remove Mercurial support

[names_are_hard]

I think this affects ML.  Maybe not, they talk about Bitbucket Cloud so perhaps we're on a service where they're not retiring Mercurial?

https://bitbucket.org/blog/sunsetting-mercurial-support-in-bitbucket
"Mercurial features and repositories will be officially removed from Bitbucket and its API on June 1, 2020"

I don't like their decision to delete repos.  Putting them in a read-only mode would have been a lot kinder.

There are tools to migrate to Git (so I guess you keep your history?) but I know from personal experience that building ML has dependencies on having hg installed on your system.  It wasn't hard to remove these.

Long term this is probably good for ML - almost no-one uses Mercurial and needing to learn it must put some people off ML.  Short term it's annoying to migrate!

[dfort]

Yep, just got the email from Atlassian.

After much consideration, we've decided to remove Mercurial support from Bitbucket Cloud and the API. Mercurial features and repositories will be officially removed from Bitbucket and its API on June 1, 2020.

What used to be a very fragmented version control software market has rapidly matured. Mercurial usage on Bitbucket is steadily declining, and the percentage of new Bitbucket users choosing Mercurial has fallen to less than 1%. At the same time, Git has become the standard. According to a Stack Overflow Developer Survey, almost 90% of developers use Git, while Mercurial is the least popular version control system with only about 3% developer adoption.

Given that Git has emerged as the version control system of choice for the industry, we have decided to focus our priorities and roadmap on building the best possible experience for the market and our users.

Read on to learn more about what led to this decision, timeline, and migration resource support.

[Danne]

Aw, hell...

[names_are_hard]

Meh, annoying.  Could also move to another provider that supports Mercurial, I suppose.  Not my call.

[a1ex]

Aw, hell...

Indeed, unlearning ~ 10 years worth of workflow is not going to be funny

Not only that, but there's a lot of useful info on Bitbucket (discussions on issue tracker, pull requests, various forks) that will be deleted, even if we move to some other Mercurial host (correct me if I'm wrong). These will have to be migrated (manually copy/pasted?) somewhere, too. How? Maybe web.archive.org to the rescue?

Migrating to git also means the existing commit hashes (used all over the place in documentation and discussions) will no longer work. Updating those is going to be a PITA, just like the Dropbox links used a lot in the past (still unsolved as of today).

I actually had a few attempts at learning Git, but found it a lot more complicated than Mercurial. I'm not completely against switching to Git, but I need some very good reasons to do so.

"ML has dependencies on having hg installed" -> this one is likely easy to fix. Still a PITA, will take at least a couple of hours, but doable.

In any case, no matter what path we'll choose, this change will result in lots of time wasted (time that would have otherwise been spent on development), and plenty of headaches.

I actually hope there will be some miracle at Bitbucket, about reversing this decision...


Discussions across the web:
- https://community.atlassian.com/t5/Bitbucket-articles/What-to-do-with-your-Mercurial-repos-when-Bitbucket-sunsets/ba-p/1155380
- https://www.reddit.com/r/programming/comments/csy2tf/bitbucket_kills_mercurial_support/
- https://news.ycombinator.com/item?id=20745393

[nikfreak]

'nuff time til June May 2020. I am sure there will be a way to either archive everything (read-only mode) or migrate almost everything.

[Danne]

'nuff time til June May 2020. I am sure there will be a way to either archive everything (read-only mode) or migrate almost everything.

Well it's not gonna migrate overnight, that's for sure. Tragic. Personally I built so many tools around mercurial and now it feels like a strong grown forest and it's source trees, branches, the works is just killed.
And then git, sigh...
I guess nothing lasts forever. Except for maybe eos m 

[names_are_hard]

Finding the best compromise between least painful and most information preserved may be time consuming though

I am trying Github's import function, apparently that is quite reliable.  I'll update when it's finished with what things worked well / badly.  In theory one option would be hg-git, allowing Github (or Bitbucket, but you'd still need to migrate the repo, for which Github has a function already and Bitbucket doesn't...) for hosting, and devs can use hg or git cli tools:
https://hg-git.github.io/

In the community thread, Atlassian staff seem quite convinced they won't change their minds.  They even rule out developing an auto-migration tool for your repo type on Bitbucket.  Sometimes angry internet people get stupid decisions changed, so we can wait and see.

Personally I prefer Git...  but really only because I've never learnt Mercurial.  Reading around it seems they have basically identical functionality these days, just accessed in different ways.  But they are both complicated tools and relearning is pain, so keeping the ability to use hg tools seems valuable.

[dmilligan]

Well, hg always annoyed me b/c I use git at work and I am more used to it, but this move by atlassian is pretty sketchy. I’d see this as an opportunity to dump them altogether and moving to another hosting provider. Although as a1ex points out it does suck to loose all that history. Maybe they will relent as google did with nest recently. We should all probably complain loudly, just in hopes of keeping options open.

[names_are_hard]

Attempt one to import to Github failed, saying there were import errors, but no description beyond that.  Not very useful...

I'll give it another try and if that fails probably mess around with the Atlassian recommended CLI tools for import / export.

[names_are_hard]

Using hg-fast-export I was able to get this to work.  This also had a problem (maybe the same as Github), because Mercurial allows branch names with a leading "/" and Git doesn't.  You need to specify a branch mapping file, I used one with the contents like this:

"/raw_recc-added-121-and-11751-aspect-rati-1382109783386"="raw_recc-added-121-and-11751-aspect-rati-1382109783386"
"/added-1792-resolution-1390499711758"="added-1792-resolution-1390499711758"

Then it worked fine.  In theory it has preserved the hg hashes as I chose the option to "Annotate commits with the hg hash as git notes in the hg namespace".  Haven't tested thoroughly, but you can get what look like useful mappings of Git commits to hg hashes via: git notes --ref hg list

I then did: git push --all
This may not strictly be needed, but I don't understand the branch model of Mercurial so wanted to save all of them.  There are 274 open branches???

Anyway, the result is here:
https://github.com/reticulatedpines/magiclantern/tree/unified

Overall, pretty easy, and the results look sensible to me.  But I'm not qualified to test hg tools against that repo using hg-git.  If anyone competent is interested in testing that, let me know if you run into any problems.  It does show that you can take all the history out at least.  The same procedure should work for moving from hg to git while remaining on Bitbucket, too.

I've also found a script for moving Bitbucket issues to Github, I'm trying that out.

[names_are_hard]

The issue migration is not horrible, could use a little work.  Here's an example:
https://github.com/reticulatedpines/magiclantern/issues/2

One obvious ugly part, it thinks I'm the reporter for all issues.  It's also using a dumb heuristic to link the Bitbucket account of people in the tickets to a Github account.

More subtle, while it preserves the links to the hg commits, these link to Bitbucket so will shortly become useless.  This is fixable, I would need to modify the migration script to understand the hash mappings made by hg-fast-export, probably pretty easy, I can see where in the code this would happen.

So, fairly hopeful if the Git route is chosen!  We can automatically keep the commit history of all branches, we can keep all the issues, and we can map the hg commits to git commits (probably including doing this automatically for issues at least).

[kitor]

Then it worked fine.  In theory it has preserved the hg hashes as I chose the option to "Annotate commits with the hg hash as git notes in the hg namespace".  Haven't tested thoroughly, but you can get what look like useful mappings of Git commits to hg hashes via: git notes --ref hg list

Yup, that should work. We are migrating SVN workflow to Git and protocol v1 gets overwhelmed if you have too many tags in your repo (and we are creating a ton of them every day). Git notes was found to be one of better solutions for this problem - you can search them easily.

[chris_overseas]

This might be an easier option?

https://sourcehut.org/blog/2019-08-21-sourcehut-welcomes-bitbucket-refugees/

[names_are_hard]

Some kind of Mercurial hosting is probably the better option - the main devs apparently prefer Mercurial, plus the migration may well be easier.  I do note this from Sourcehut, https://sourcehut.org/alpha-details/ :

Payment is optional now but will be required later

I was testing the Git side just for feasibility   Well, that and I am using Git for my fork because I have zero Mercurial experience, but that shouldn't factor in to ML future choices.  (If I get enough progress for my fork to be useful, I will merge those change back into Mercurial ML - hopefully without hg eating my files.  Again.)

[Luther]

This might be an easier option?

https://sourcehut.org/blog/2019-08-21-sourcehut-welcomes-bitbucket-refugees/

@chris_overseas is right. Sourcehut is awesome. You don't need to pay, you can self host too. If git is too complicated, see also Pijul and Game of Trees

[ilia3101]

We have less than 6 months!!! Any updates?

[names_are_hard]

No change from Atlassian - they still plan to delete everything, and they still don't offer a migration tool from for Hg -> Git within Bitbucket.

[Danne]

A disgrace. But are we still staying on git within bitbucket or moving to another platform? Would be good to know at least this.

[names_are_hard]

Personally I'm staying on git.  I have a fork (ish) on both Github and Bitbucket.  Keeping the code is easy, I also copied the ticket history.  But I haven't looked at copying the wiki - does anyone know if Atlassian plan to nuke the wikis?  Are they even stored using Mercurial?

[a1ex]

I'd go with a different host; even if Bitbucket might look familiar, their decision to delete all hg repositories and metadata is not to be taken lightly.

Back then, I've created a test account on Sourcehut (https://hg.sr.ht/~ml/magic-lantern), but didn't play with it yet. The workflow seems quite different, no pull requests, I don't fully understand it atm edit: found their Mercurial guide. One interesting feature is that anyone can submit issues via e-mail, without having to log in (https://todo.sr.ht/~ml/magic-lantern).

Other suggestions welcome.

Also considering self-hosting; Kallithea looks pretty similar to what we are used to. In this case, we will need a regularly updated mirror on some external host, as backup. Maybe a dual hg+git repository could also help.

Unfortunately, time for hobbies is still an issue on my side, so... no progress at all

Worst case - we'll end up hosting our code on archive.org...

[Danne]

All for a platform move.
Meanwhile I wonder if porting temporalily to be able to migrate over time if it takes a while to find the better solution?
Not verified this idea but tempted to test:
https://jrummell.com/bitbucket-hg-git-migration
https://www.yosoygames.com.ar/wp/2019/09/migrating-repos-from-mercurial-to-git/

[names_are_hard]

Danne, I already did a test migration: https://github.com/reticulatedpines/magiclantern - it has the full code history and some of the ticket history.  Don't know how to migrate out the wiki.

Three months left.

[Danne]

Yeah, time is running up. Was it straight forward? I will need to take a look soon into this.

[names_are_hard]

Scroll up - I document things.  Pretty easy.  I pushed a fix upstream for one of the migration tools.  You'll still need the branch mapping file I think.

It won't *build*, notably - Magiclantern decided to make the build depend on you having hg tools installed and requiring the branch to be managed by Mercurial.  Seems insane to me.  The changes to remove those dependencies are here:
https://bitbucket.org/stephen-e/ml_200d/commits/1f856253bf5ad0103a69c9269a71808b92b34042
https://bitbucket.org/stephen-e/ml_200d/commits/3dd99d006ca8fe1a52ef7281dd2ef64360793dc4  (not all of it, only one file change is relevant)

A cleaner way of doing that would be to check for existance of hg or git and stamp the build with that if so, otherwise just build.