Author Topic: Bitbucket set to remove Mercurial support  (Read 13617 times)

names_are_hard

  • Contributor
  • Senior
  • *****
  • Posts: 294
  • 200D idiot
Re: Bitbucket set to remove Mercurial support
« Reply #25 on: April 28, 2020, 04:57:14 PM »
Deadline got extended:

Quote
Due to the impact of COVID-19, we are extending the deprecation date by 30 days to give you more time to transition. Mercurial features and repositories will be officially removed from Bitbucket and its API on July 1, 2020.

Two months left.

heder

  • Contributor
  • Member
  • *****
  • Posts: 129
  • No time for caution
Re: Bitbucket set to remove Mercurial support
« Reply #26 on: June 03, 2020, 11:03:23 AM »
And now the end is near and so i face the final curtain ..

Remember to download all the bits before the Atlassian shredder tear down our world  >:( 
Embedded SW engineer. Current Cameras: Canon 20d, 40d, 350d

names_are_hard

  • Contributor
  • Senior
  • *****
  • Posts: 294
  • 200D idiot
Re: Bitbucket set to remove Mercurial support
« Reply #27 on: June 03, 2020, 03:33:03 PM »
I imported all branches and history into Github a while back:
https://github.com/reticulatedpines/magiclantern

Didn't copy all the issues across, I should try and find time for that.

aprofiti

  • Contributor
  • Member
  • *****
  • Posts: 194
Re: Bitbucket set to remove Mercurial support
« Reply #28 on: June 03, 2020, 05:58:51 PM »
I imported all branches and history into Github a while back:
https://github.com/reticulatedpines/magiclantern
Looks like it's update up to 3 Jul 2018, I can't find most relevant branches (crop_rec_*, manual_lens_info, vxworks...) into your repo unfortunally.
@names_are_hard Can you check please? May be usefull as backup.

Still wishing we can get more good PR merged into codebase as possible before sunsetting to avoid loosing some works from cummunity...

names_are_hard

  • Contributor
  • Senior
  • *****
  • Posts: 294
  • 200D idiot
Re: Bitbucket set to remove Mercurial support
« Reply #29 on: June 03, 2020, 06:26:05 PM »
@aprofiti, thanks, all the branches I'd needed were always in both so I hadn't noticed any differences.  I don't know why crop_rec isn't there, I'd expect it to be, looking at Hudson.  But, Mercurial means "branch" means something different to Git "branch", so maybe it's expected and I just don't understand Mercurial well enough.  It's baffling to me that there are so many different live branches in Hudson repo.

I guess it's some problem with the scripts I used to import Mercurial bitbucket to Git github.  I'll see if I can get it working better but I don't have high confidence.

I don't have the knowledge of Mercurial or the desire to learn it, in order to make a Mercurial copy of existing Hudson ML.  If anyone wants that to exist, they should make one.

It is up to date past 2018, I guess you were looking at a particular branch that hasn't been updated since 2018.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12467
Re: Bitbucket set to remove Mercurial support
« Reply #30 on: June 05, 2020, 09:13:42 PM »
Got another suggestion for Mercurial hosting: foss.heptapod.net (fork of GitLab):
https://heptapod.net/

Test account (just created; empty for now):
https://foss.heptapod.net/ml/

According to their docs, it imports Bitbucket repositories, including issues and open pull requests:
https://heptapod.net/pages/tuto-bb-import.html

To create the repository, there's no option on the user interface; I need to apply with a public discussion on their issue tracker: https://heptapod.net/a-public-heptapod-for-free-and-open-source-software.html

I have no problems doing that, as long as there will be some consensus. There is also an option to self-host (as an alternative to Kallithea, mentioned earlier).

BTW, I happen to have a two-week holiday (starting tomorrow), hopefully enough to solve this issue.

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 6968
Re: Bitbucket set to remove Mercurial support
« Reply #31 on: June 05, 2020, 09:15:15 PM »
Much appreciated a1ex.

reddeercity

  • Contributor
  • Hero Member
  • *****
  • Posts: 2267
Re: Bitbucket set to remove Mercurial support
« Reply #32 on: June 06, 2020, 05:14:28 AM »
yea a1ex !
Also how able this mrcrowl.hg ? I just installed this on my Visual Studio Code app (wins7) and now I installed Mercurial 5.4.1
"Integrated Mercurial source control" so all you need is visual studio code (free) visualstudio-cross-platform-downloads
OS supported is macOS , Windows x64 & Linux x64 . As far as I can see it has all the hg commands  , thou it's seems to be just for github
here the documentation https://code.visualstudio.com/docs/editor/versioncontrol
just a idea to help  :)


DeafEyeJedi

  • Hero Member
  • *****
  • Posts: 3397
  • 5D3 | M1 | 7D | 70D | SL1 | M2 | 50D
Bitbucket set to remove Mercurial support
« Reply #33 on: June 06, 2020, 05:02:44 PM »
Excellent find, @alex! I am so glad I came across this thread. I was about to go insanity and worried shitless about our history within this community. I’m on board w you guys to keep us all thriving to the fullest of our abilities.

I actually like your ideas @reddeercity. Will take a look into it.
5D3.113 | 5D3.123 | EOSM.203 | 7D.203 | 70D.112 | 100D.101 | EOSM2.* | 50D.109

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12467
Re: Bitbucket set to remove Mercurial support
« Reply #34 on: June 07, 2020, 10:12:00 AM »
Found this: https://github.com/philipstarkey/bitbucket-hg-exporter

It exports a Bitbucket repository and all of its forks, as JSON, including issues, comments and so on. Requires login, but is able to download any repo where you have read access (i.e. any public repos). Only one thread per user allowed (there are rate limits). Optional export to Github (didn't try, but there is an example here).

Important: to use the script, you will need to make this change, or wait for my PR to be merged upstream.

Currently trying to export the main repository (hudson/magic-lantern) and all of its forks. Been running for over 36 hours, still downloading, about 500 MiB downloaded, no idea how much is left (hours? days? weeks? maybe more?). Cannot download other repos until this is finished.

Forks identified by the script (download in progress; duplicates removed from the log):
Code: [Select]
Finding all forks of hudson/magic-lantern
Finding all forks of a1ex/magic-lantern
Finding all forks of arm_indy/magic-lantern-for-600d-t3i
Finding all forks of magiclantern/magic-lantern
Finding all forks of yyfrankyy/magic-lantern
Warning: repository yyfrankyy/magic-lantern might have been deleted (404).
Finding all forks of dbaleckaitis/magic-lantern
Finding all forks of arm_indy/magic-lantern-for-60d
Finding all forks of piersg/magic-lantern-pg-fooling
Finding all forks of deti/magic-lantern
Finding all forks of alins/magic-lantern-for-500d
Finding all forks of maclema/ml500d
Warning: repository maclema/ml500d might have been deleted (404).
Finding all forks of roaldfre/magic-lantern-for-500d
Finding all forks of adijiwa/magic-lantern-for-500d
Finding all forks of sudharsh/magic-lantern
Finding all forks of sztupy/magic-lantern-for-60d
Finding all forks of nmargono/magic-lantern
Finding all forks of KholdStare/magic-lantern-unified-experiments
Finding all forks of roaldfre/magic-lantern-unified
Finding all forks of qianc/magic-lantern
Finding all forks of teegee/magic-lantern
Warning: repository teegee/magic-lantern might have been deleted (404).
Finding all forks of sztupy/magic-lantern
Finding all forks of up4/magic-lantern
Finding all forks of 600dplus/magic-lantern-for-600d_old
Finding all forks of CMatters/magic-lantern
Finding all forks of gheja/magic-lantern-gheja-dev
Finding all forks of supermaz/magic-lantern-maz-60d
Finding all forks of OtherOnePercent/tragic-lantern
Finding all forks of chrisball/magic-lantern
Finding all forks of Sticks/magic-lantern-stickshack
Finding all forks of dmgerman/ml
Finding all forks of Sticks/magic-lantern-menu-system
Finding all forks of j_l/magic-lantern
Finding all forks of jplxpto/magic-lantern
Finding all forks of ubbut/magic-lantern
Finding all forks of 600dplus/magic-lantern-for-600d
Finding all forks of 0xAF/magic-lantern-5dc-port-wip
Finding all forks of 600dplus/magic-lantern-5dc-port-wip-updated
Finding all forks of raymondlo84/magic-lantern
Finding all forks of maxchen/magic-lantern
Finding all forks of rufustfirefly/magic-lantern-5d3
Finding all forks of 600dplus/magic-lantern-finding-flash-prop
Finding all forks of 600dplus/magic-lantern-features_custom_modes
Finding all forks of rufustfirefly/magic-lantern
Finding all forks of teepean/magic-lantern
Finding all forks of OtherOnePercent/tragic-lantern-6d
Finding all forks of salsasepp/tragic-lantern-6d
Finding all forks of Mayocomplet/tragic-lantern-6d
Finding all forks of Mayocomplet/tragic-mayo-6d
Finding all forks of GregoryOfManhattan/tragic-lantern-6d
Finding all forks of andy600/tl50d
Finding all forks of miguelbrito/tragic-lantern-7d
Finding all forks of M1ke_M1ke/tragic-lantern-6d-m1ke-fork
Finding all forks of feureau/tragic-lantern-6d
Finding all forks of anthony_molesphini/tragic-lantern-6d
Finding all forks of vinlyanimate/raw-6d
Finding all forks of vinlyanimate/raw-60d-110
Finding all forks of EdK/tragic-lantern-6d
Finding all forks of hjdaniel_sun/tragic-lantern-6d
Finding all forks of 600dplus/magic-lantern-fixes
Finding all forks of 600dplus/magic-lantern-600d-photoinfodisp
Finding all forks of maxchen/magic-lantern-unified
Finding all forks of 600dplus/magic-lantern-fix
Finding all forks of magicpriest/magic-lantern
Warning: repository magicpriest/magic-lantern might have been deleted (404).
Finding all forks of sl0w0rm/magic-lantern-af-ptp
Finding all forks of OtherOnePercent/tragic-lantern-2.0
Finding all forks of datadogie/tragic-lantern-2.0
Finding all forks of igor1111111/tragic-lantern-2.0
Finding all forks of grishkavideo/tragic-lantern-2.0
Finding all forks of barbara_brockhauser/tragic-lantern-2.0
Finding all forks of BrianPrz/tragic-lantern-2.0
Finding all forks of 600dplus/magic-lantern-600d-flexinfo
Finding all forks of 600dplus/magic-lantern-menu-shortcuts
Finding all forks of 600dplus/magic-lantern-auto-audio-override
Finding all forks of simmsa/magic-lantern
Finding all forks of trsaunders/magic-lantern
Finding all forks of 600dplus/magic-lantern-5dc-update-from-0xaf-wip
Finding all forks of raeez/magic-lantern
Finding all forks of ak_ss/magic-lantern
Finding all forks of marekk17/60d-silent-dng
Finding all forks of marekk17/magic-lantern
Finding all forks of minimimi/magic-lantern-filer
Finding all forks of marekk17/magic-lantern-1
Finding all forks of marekk17/magic-lantern-2
Finding all forks of smeangol/smeangols-sneaky-magic-lantern
Finding all forks of 600dplus/magic-lantern-600d-raw-feat
Finding all forks of marekk17/magic-lantern-3
Finding all forks of marekk17/magic-lantern-4
Finding all forks of dhilung/magic-lantern
Finding all forks of flameeyes/magic-lantern
Finding all forks of GregoryOfManhattan/magic-lantern
Finding all forks of krashnik/magic-lantern
Finding all forks of sonic74/magic-lantern
Finding all forks of minimimi/magic-lantern-filer-multiselect
Finding all forks of freelyfree/cigam-nretnal
Finding all forks of mucher/magic-lantern
Warning: repository mucher/magic-lantern might have been deleted (404).
Finding all forks of marekk17/magic-lantern-5
Finding all forks of marci1/magic-lantern
Warning: repository marci1/magic-lantern might have been deleted (404).
Finding all forks of marci1/magic-lantern-marci1
Warning: repository marci1/magic-lantern-marci1 might have been deleted (404).
Finding all forks of williamh/magic-lantern-5d3
Warning: repository williamh/magic-lantern-5d3 might have been deleted (404).
Finding all forks of g3gg0/to-be-deleted
Finding all forks of davespana13/magic-lantern
Warning: repository davespana13/magic-lantern might have been deleted (404).
Finding all forks of HKServ/magic-lantern-hksfork
Finding all forks of HKServ/magic-lantern-notcc-hksfork
Finding all forks of sc1ence/magic-lantern
Finding all forks of minDark/magic-lantern-unificat
Finding all forks of pravdomil/magic-lantern-hack
Finding all forks of reddeercity/magic-lantern-hack
Finding all forks of niklasRde/magic-lantern
Finding all forks of emc/magic-lantern
Finding all forks of 600dplus/magic-lantern-600d-60d-1100d-memcheck
Finding all forks of marekk17/magic-lantern-6
Finding all forks of marekk17/ml_60d
Finding all forks of marekk17/magic-lantern-7
Finding all forks of marekk17/magic-lantern-8
Finding all forks of YMP/magic-lantern-afma-improvement
Finding all forks of YMP/magic-lantern-afma-improvement-1
Finding all forks of canon_amateur/magic-lantern
Finding all forks of aluker/magic-lantern
Finding all forks of frantony/magic-lantern.makeall
Finding all forks of pbernardi90/magic-lantern
Finding all forks of frantony/magic-lantern.checkpatch
Finding all forks of sblest/magic-lantern
Finding all forks of theunkn0wn/magic-lantern
Warning: repository theunkn0wn/magic-lantern might have been deleted (404).
Finding all forks of stealth_tramp/magic-lantern
Finding all forks of frantony/magic-lantern.check-bot
Finding all forks of masgro/magic-lantern
Finding all forks of circuitman/magic-lantern
Finding all forks of miguelbrito/magic-lantern
Finding all forks of shumonsaito/magic-lantern-holy-grail-hack
Finding all forks of g3gg0/magic-lantern-mlv
Finding all forks of nicksta/magic-lantern-mlv
Warning: repository nicksta/magic-lantern-mlv might have been deleted (404).
Finding all forks of nicksta/magic-lantern
Warning: repository nicksta/magic-lantern might have been deleted (404).
Finding all forks of v8rrc/magic-lantern-v8rrc
Finding all forks of v8rrc/magic-lantern-1
Finding all forks of aham1234/magic-lantern
Finding all forks of marekk17/magic-lantern-9
Finding all forks of Chucho/magic-lantern-70d
Warning: repository Chucho/magic-lantern-70d might have been deleted (404).
Finding all forks of qmatrxp/magic-lantern
Finding all forks of guysmilez/magic-lantern
Finding all forks of keith_clinton/magic-lantern
Finding all forks of g3gg0/magic-lantern-module-hgdiff
Finding all forks of hope_young/magic-lantern
Finding all forks of OliRi/magic-lantern
Finding all forks of stevefal/magic-lantern-rawrec-cropmarks
Finding all forks of robin051/magic-lantern
Finding all forks of maxotics/maxotics-fork-magic-lantern
Finding all forks of tgregoire/magic-lantern-1100d
Warning: repository tgregoire/magic-lantern-1100d might have been deleted (404).
Finding all forks of chchrlam/magic-lantern
Finding all forks of a1ex/magic-lantern-dual-iso-preview
Finding all forks of g3gg0/magic-lantern-mlv_rec
Finding all forks of idealsceneprod/magic-lantern
Warning: repository idealsceneprod/magic-lantern might have been deleted (404).
Finding all forks of bnvm/magic-lantern
Warning: repository bnvm/magic-lantern might have been deleted (404).
Finding all forks of sodapopodalaigh/magic-lantern
Finding all forks of swinxx/ml_a
Finding all forks of M1ke_M1ke/magic-lantern-m1ke-fork
Finding all forks of M1ke_M1ke/magic-lantern-m1ke-fork-2
Finding all forks of marekk17/magic-lantern-10
Finding all forks of alextutubalin/magic-lantern
Finding all forks of marekk17/magic-lantern-11
Finding all forks of marekk17/magic-lantern-12
Finding all forks of marekk17/magic-lantern-13
Finding all forks of marekk17/magic-lantern-14
Finding all forks of marekk17/magic-lantern-15
Finding all forks of marekk17/magic-lantern-16
Finding all forks of monty87/magic-lantern
Warning: repository monty87/magic-lantern might have been deleted (404).
Finding all forks of mr3wan/magic-lantern
Finding all forks of bwilliams18/magic-lantern
Warning: repository bwilliams18/magic-lantern might have been deleted (404).
Finding all forks of dmilligan/magic-lantern
Finding all forks of Audionut/magic-lantern
Finding all forks of minimimi/magic-lantern-raw2dng_vc
Finding all forks of gregausina/magic-lantern
Finding all forks of knumat/magic-lantern
Finding all forks of kfullman/magic-lantern
Finding all forks of baldand/baldand-7d-magic-lantern
Finding all forks of nagyistoce/baldand-7d-magic-lantern
Finding all forks of HalSup/magic-lantern
Finding all forks of LebedevRI/magic-lantern
Finding all forks of jordancolburn/magic-lantern
Finding all forks of jordancolburn/magic-lantern-1
Finding all forks of somiic/magic-lantern
Warning: repository somiic/magic-lantern might have been deleted (404).
Finding all forks of somiic/magic-lantern-7d
Warning: repository somiic/magic-lantern-7d might have been deleted (404).
Finding all forks of rudison/magic-lantern
Finding all forks of houz/magic-lantern
Warning: repository houz/magic-lantern might have been deleted (404).
Finding all forks of romandvorak/magic-lantern-fork
Finding all forks of ayshih/magic-lantern-50d
Finding all forks of Phaenomena/magic-lantern
Finding all forks of feureau/magic-lantern
Finding all forks of mk11174/t5i
Finding all forks of mk11174/t5i-backup
Finding all forks of escho/escho-magiclantern
Finding all forks of jjwchoy/magic-lantern
Finding all forks of bnvm/magic-lantern-1
Warning: repository bnvm/magic-lantern-1 might have been deleted (404).
Finding all forks of bnvm/magic-lantern-2
Warning: repository bnvm/magic-lantern-2 might have been deleted (404).
Finding all forks of bnvm/magic-lantern-3
Warning: repository bnvm/magic-lantern-3 might have been deleted (404).
Finding all forks of bnvm/magic-lantern-4
Warning: repository bnvm/magic-lantern-4 might have been deleted (404).
Finding all forks of bnvm/magic-lantern-5
Warning: repository bnvm/magic-lantern-5 might have been deleted (404).
Finding all forks of adambernacki10/magic-lantern
Finding all forks of mofig/magic-lantern-mofig
Finding all forks of frenchiefilms/phil
Finding all forks of nanomad/phil
Finding all forks of viniciusatique/magic-lantern-mymod
Finding all forks of nviennot/magic-lantern
Finding all forks of vyskocil/magic-lantern-vyskocil
Finding all forks of viniciusatique/magic-lantern_1
Finding all forks of viniciusatique/movie-restart-5d3
Finding all forks of marekk17/ml123
Finding all forks of spaynix/magic-lantern
Warning: repository spaynix/magic-lantern might have been deleted (404).
Finding all forks of PhotoCat/magic-lantern
Finding all forks of mk11174/700d113
Finding all forks of arcxyz/magic-lantern-eos-1100d
Finding all forks of samontab/magic-lantern
Finding all forks of vehnae/polar-alignment-for-magic-lantern
Finding all forks of mofig/mofig-magic-lantern
Finding all forks of OtherOnePercent/backflow2
Finding all forks of Pelican/7d203
Finding all forks of mk11174/magic-lantern
Finding all forks of Pelican/7d.205
Finding all forks of nandoide/rec_plus
Finding all forks of nanomad/tragic-lantern-1100d
Finding all forks of g3gg0/magic-lantern-plot
Finding all forks of kuhn90/magic-lantern
Finding all forks of dingolina/magic-lantern-learn
Finding all forks of Cuntstructor/magic-lantern
Finding all forks of mlvisit/magic-lantern
Finding all forks of MarsBlessed/magic-lantern
Finding all forks of fabiomoreira716/magic-lantern
Finding all forks of mealcardga/magic-lantern
Finding all forks of holicowdev/magic-lantern
Finding all forks of hicom150/magic-lantern
Finding all forks of error454/magic-lantern
Finding all forks of rcdude/magic-lantern
Finding all forks of jpaana/ml
Finding all forks of felipe0328/magic-lantern
Finding all forks of Audionut/ml-fork
Finding all forks of Audionut/user-builds
Finding all forks of bennir/magic-lantern
Finding all forks of berlincount/magic-lantern-old
Finding all forks of Pelican/magic-lantern
Finding all forks of sodapopodalaigh/my-magic-lantern
Finding all forks of a_d_/magic-lantern-stubs
Warning: repository a_d_/magic-lantern-stubs might have been deleted (404).
Finding all forks of stevefal/magiclantern-mlvrec-video
Finding all forks of gkaizer/magiclantern
Warning: repository gkaizer/magiclantern might have been deleted (404).
Finding all forks of ggjjl1/magic-lantern
Finding all forks of Audionut/cleanup
Finding all forks of vance_lear/magic-lantern
Finding all forks of kfprod/5d3-mlv_rec-naming
Finding all forks of andy600/andy50d2
Finding all forks of Clark_Kent_/magic-lantern
Finding all forks of mk11174/magic-lantern-fullres
Finding all forks of mk11174/magic-lantern-fullres2
Finding all forks of berlincount/ml-100d-old
Finding all forks of mk11174/650d
Finding all forks of kurt_selander/magic-lantern
Finding all forks of zloe/zloe-magic-lantern
Finding all forks of mk11174/menu_flickering
Finding all forks of golem72/magic-lantern-frsp
Finding all forks of dpjpandone/magic-lantern
Finding all forks of spoonfly/magic-lantern
Finding all forks of SydTHackerr/magic-lantern
Finding all forks of dpjpandone/magic-lantern-1
Finding all forks of kichetof/magic-lantern
Finding all forks of dpjpandone/magic-lantern-2
Finding all forks of mk11174/bolt_trigger
Finding all forks of pravdomil/magic-lantern-lvexpo
Finding all forks of pravdomil/magic-lantern-sndcheck
Finding all forks of kfprod/mlv_rec-exfat-naming-suggestion
Finding all forks of Shiftx07/magic-lantern
Finding all forks of johnnyrun/magic-lantern
Finding all forks of dlrpgmsvc/magic-lantern-ok
Finding all forks of kunchenguid/magic-lantern
Finding all forks of roachhd/magic-lantern
Finding all forks of Vy7ka/magic-lantern
Finding all forks of r3dsm0k3/magic-lantern
Finding all forks of coutts/magic-lantern
Finding all forks of alessiocredendino/magic-lantern
Finding all forks of sundus2y/magic-lantern
Finding all forks of gmanfredax/magic-lantern
Finding all forks of nestor_cano/magic-lantern
Finding all forks of dietsche/magic-lantern
Finding all forks of nikfreak/magic-lantern
Finding all forks of jkruser/magic-lantern
Warning: repository jkruser/magic-lantern might have been deleted (404).
Finding all forks of smokegrafix/magic-lantern
Finding all forks of Dannephoto/magic-lantern_nikfreak
Finding all forks of OtherOnePercent/magic-lantern
Finding all forks of koenkooi/magic-lantern
Finding all forks of flykeysky/magic-lantern
Finding all forks of Dan0974/magic-lantern
Finding all forks of BrianPrz/magic-lantern
Finding all forks of maits18/magic-lantern-install-scrip5
Finding all forks of maits18/summon-arm
Finding all forks of rbrune/magic-lantern
Finding all forks of jwkerr/magic-lantern
Finding all forks of JLGW/magic-lantern
Finding all forks of csghone/magic-lantern
Finding all forks of drlapser/magic-lantern
Finding all forks of sodapopodalaigh/magic-lanterns
Finding all forks of niklastisk/magic-lantern
Finding all forks of rs68/magic-lantern-rs68-tests
Finding all forks of monograph/dof_display
Finding all forks of Pelican/recovery
Finding all forks of monograph/dof_display_development
Finding all forks of mario_pich/magic-lantern
Finding all forks of brisma_meihar_arsandi/brismaarsandi
Finding all forks of milank/magic-lantern
Finding all forks of chris_miller/ml-5d3-133
Finding all forks of Maqs/magic-lantern
Finding all forks of Audionut/mlv
Finding all forks of iancmcleod/magic-lantern
Finding all forks of homerobono/magic-lantern
Finding all forks of mk11174/700d114
Finding all forks of mk11174/magic-lantern-unified
Finding all forks of garry23/depth-of-field-with-diffraction
Finding all forks of juuh42dias/magic-lantern
Finding all forks of HilliBilli/magic-lantern
Finding all forks of fjriosp/1200d-dev
Finding all forks of DennisNeumann/magic-lantern
Finding all forks of garry23/final-dof
Finding all forks of garry23/colour-warnings-with-dof
Finding all forks of gyedafe/magic-lantern
Finding all forks of FabiLAP/magic-lantern
Finding all forks of cbob/magic-lantern
Finding all forks of lucashigor/magic-lantern
Warning: repository lucashigor/magic-lantern might have been deleted (404).
Finding all forks of kookiyoung/magic-lantern
Finding all forks of Licaon_Kter/magic-lantern
Finding all forks of buttbucket/magic-lantern
Finding all forks of ewinemiller/magic-lantern
Finding all forks of dinkin/magic-lantern
Finding all forks of 0th4rw4/magic-lantern
Finding all forks of andum1/magic-lantern
Finding all forks of nagyistoce/magic-lantern
Finding all forks of Marsu42/ml-pull
Finding all forks of ilias_kotsovolos/magic-lantern
Finding all forks of dpjpandone/magic-lantern-3
Finding all forks of dpjpandone/dpjpandone-eosm-fixes
Finding all forks of dpjpandone/eos-m-dpjpandone
Finding all forks of McGv/magic-lantern
Finding all forks of ziquan_xu/magic-lantern
Finding all forks of daniel_fort/magic-lantern
Finding all forks of danilaswift/magic-lantern
Finding all forks of Dannephoto/magic-lantern_dfort
Finding all forks of guang_feng_ren/magic-lantern
Finding all forks of racerxdl/magic-lantern
Finding all forks of SwaroopB/magic-lantern
Finding all forks of lurnid/magic-lantern
Finding all forks of roshansoni/magic-lantern
Finding all forks of Audionut/test
Finding all forks of hulasamsquanch/magic-lantern
Finding all forks of iremar_junior/magic-lantern
Finding all forks of co_ikebanat/magic-lantern
Finding all forks of jacobbieker/magic-lantern
Finding all forks of dinkin/magiclantern
Finding all forks of Yongyuth-P/magic-lantern
Finding all forks of VertigoPix/magic-lantern
Finding all forks of Licaon_Kter/magic-lantern-bulb_nd-info
Finding all forks of Licaon_Kter/magic-lantern-rec_ui_fix
Finding all forks of Licaon_Kter/magic-lantern-beta-info-button
Finding all forks of rongjuan_xie/magic-lantern
Finding all forks of houssi4ever/magic-lantern
Finding all forks of waza57/magic-lantern
Finding all forks of Dannephoto/magic-lantern_waza57
Finding all forks of reddeercity/magic-lantern_reddeercity_5d2-50d_4k-crop_rec
Finding all forks of tecgen/magic-lantern-old
Finding all forks of sirleneeraphael_per_golo/raphael
Finding all forks of piers92/magic-lantern
Finding all forks of lumenmikie/magic-lantern
Finding all forks of jayspear/magic-lantern
Finding all forks of marceljuchelka/magic-lantern
Finding all forks of everest20/magic-lantern
Finding all forks of Audionut/ml1
Finding all forks of am2rican5/magic-lantern
Finding all forks of dnlit/magic-lantern
Finding all forks of dnlit/magic-lantern-1
Finding all forks of xlee2004/magic-lantern-old
Finding all forks of tecgen/magic-lantern
Finding all forks of matias82/magic-lantern
Finding all forks of Q111Q/magic-lantern
Finding all forks of mk11174/1200d
Finding all forks of matias82/1200d
Finding all forks of rs68/magic-lantern-rolands-dm-spy-expriments-ros
Finding all forks of rs68/magic-lantern-rolands-debuging-bricked-5d2
Finding all forks of SinSpy/magic-lantern
Finding all forks of pangliang/magic-lantern
Finding all forks of sabujas/magic-lantern
Finding all forks of anDyIII/magic-lantern
Finding all forks of mothaibaphoto/magic_lantern_vold_fork
Finding all forks of Dasa36/magic-lantern
Finding all forks of seed93/magic-lantern
Finding all forks of mk11174/1200d_101
Finding all forks of mk11174/1200d_new
Finding all forks of blinkythebear/1200d_new
Finding all forks of sergiodelatorre/magic-lantern
Finding all forks of Dave_Raynor/magic-lantern
Finding all forks of apchen_stu/magic-lantern
Finding all forks of MAX_XIE/magic-lantern
Finding all forks of debmalyaroy/magic-lantern
Finding all forks of anhhung1303/magic-lantern
Finding all forks of josepvm/magic-lantern
Finding all forks of josepvm/magic-lantern-1
Finding all forks of josepvm/magic-lantern-2
Finding all forks of pbenco/magic-lantern
Finding all forks of Letshowgoes/magic-lantern
Finding all forks of carlosigcimd/magic-lantern
Finding all forks of greenbox/magic-lantern
Finding all forks of wildstray/magic-lantern
Finding all forks of andy600/ml
Finding all forks of reddeercity/magic-lantern
Finding all forks of reddeercity/magic-lantern_10-12bit
Finding all forks of George-Thurman/magic-lantern-stripped-and-confused
Finding all forks of Dannephoto/magic-lantern
Finding all forks of ccritix/magic-lantern-old
Finding all forks of ArcziPL/crop_rec_70d
Finding all forks of daniel_fort/magic-lantern-danne-fork
Finding all forks of nydahl98/magic-lantern_danne
Finding all forks of Jip-Hop/magic-lantern
Finding all forks of Dannephoto/magic-lantern_jip-hop
Finding all forks of stereozont/magic-lantern
Finding all forks of STNMPTWZero/magic-lantern
Finding all forks of DeafEyeJedi/magic-lantern
Finding all forks of andy600/magic-lantern
Finding all forks of andreykuban/magic-lantern
Finding all forks of Tafaube/magic-lantern
Finding all forks of eNnvi/magic-lantern
Finding all forks of Eosm_Dev/magic-lantern
Finding all forks of fgodfrey/magic-lantern-eosm-2.0.3
Finding all forks of rapido7097/magic-lantern-750d
Finding all forks of JohanVrn/ml
Finding all forks of Xav_L/magic-lantern
Finding all forks of pankajc8726/magic-lantern
Finding all forks of SinusPi/magic-lantern-cr2hdr-append-suffix-support
Finding all forks of reddeercity/magic-lantern_5d2-d4_4k-crop_rec
Finding all forks of JohanVrn/ml_test
Finding all forks of martinhering/magic-lantern
Finding all forks of bwaggo/magic-lantern
Finding all forks of Sleeperon/kirill
Finding all forks of shadowfax/magic-lantern
Finding all forks of emklap/magic-lantern
Finding all forks of qqluqq/magic-lantern
Finding all forks of chris_miller/ml-fork
Finding all forks of garry23/dof-focus.c-fix
Finding all forks of garry23/focus.c-dof-fix
Finding all forks of garry23/dof-fix-for-focus.c
Finding all forks of garry23/dof-changes-to-focus.c
Finding all forks of a1ex/ml
Finding all forks of johncheshire/magic-lantern-fork
Finding all forks of ehoutsma/magic-lantern
Finding all forks of Dannephoto/magic-lantern-erwinh
Finding all forks of xlee2004/magic-lantern
Finding all forks of larssteenhoff/magic-lantern
Finding all forks of Licaon_Kter/magic-lantern-1
Finding all forks of tizzleh/magic-lantern
Finding all forks of nikfreak/ml
Finding all forks of kichetof/magic-lantern-1
Finding all forks of nikfreak/ml-isoresearch
Finding all forks of Audionut/debug-lossless
Finding all forks of aprofiti/magic-lantern
Finding all forks of es_as/magic-lantern
Finding all forks of Dannephoto/magic-lantern_es_as
Finding all forks of t3r4n/magic-lantern
Finding all forks of sombree/magic-lantern
Finding all forks of sombree/magic-lantern-1
Finding all forks of shorst/magic-lantern
Finding all forks of bagaric/magic-lantern
Finding all forks of blubbblubb/magic-lantern
Finding all forks of monograph/adding-space_left
Finding all forks of DieHertz/magic-lantern
Finding all forks of drevilish/magic-lantern
Finding all forks of ids1024/magic-lantern
Finding all forks of andreasb242/magic-lantern
Finding all forks of samuel-carvalho/magic-lantern
Finding all forks of samuel-carvalho/magic-lantern-1
Finding all forks of samuel-carvalho/magic-lantern-2
Finding all forks of samuel-carvalho/magic-lantern-3
Finding all forks of samuel-carvalho/magic-lantern-4
Finding all forks of drewcollins/magic-lantern-dc
Finding all forks of gertoe/magic-lantern
Finding all forks of samuel-carvalho/magic-lantern-5
Finding all forks of thenickdude/magic-lantern
Finding all forks of bilal_fakhouri/magic-lantern-1
Finding all forks of bilal_fakhouri/magic-lantern-2
Finding all forks of mkacvinsky/magic-lantern
Finding all forks of mothaibaphoto/crop_rec_4k_mlv_snd_fork
Finding all forks of bouncyball/mlv_dump-fixes
Finding all forks of kitchentowel/magic-lantern
Finding all forks of calle2010/obsolete-magic-lantern
Finding all forks of Gr3g01/ml-cmos-cal
Finding all forks of daniel_fort/ml-cmos-cal
Finding all forks of Dannephoto/ml-cmos-cal
Finding all forks of ccritix/magic-lantern-200d
Finding all forks of noreboul/magic-lantern
Finding all forks of ccritix/magic-lantern
Finding all forks of Dannephoto/magic-lantern_critix
Finding all forks of Audionut/autoexpo
Finding all forks of 600dplus/magiclantern_eyefi_trick_cleanup
Finding all forks of ArcziPL/lossless_70d
Finding all forks of LeoYuanjieLi/magic-lantern
Finding all forks of vcrparts/magic-lantern-photobooth-hack
Finding all forks of calle2010/magic-lantern-77d
Finding all forks of Levas_EOS/magic-lantern
Finding all forks of kitorPL/magic-lantern
Finding all forks of PAPPPmAc/magic-lantern
Finding all forks of geojeff/magic-lantern
Finding all forks of DirkHussen/magic-lantern
Finding all forks of MisterGun/magic-lantern
Finding all forks of jmheder/vxworks_canon40d
Finding all forks of nydahl98/magic-lantern
Finding all forks of mrcmraja/magic-lantern
Finding all forks of nikil_ragav/magic-lantern
Finding all forks of ccritix/magic-lantern-card-fmt
Finding all forks of Ant123/magic-lantern-1


TODO:

1) All other Mercurial repositories from past and current contributors, that were not direct forks of hudson/magic-lantern.

There are clones of the magic-lantern repository that are not detected as forks (so, the automated script doesn't detect them). We need to identify those.

There are also standalone projects related to ML, such as ml-focus-pixels. We need to identify those as well.

Downloading all Mercurial repositories from the above 338 users (as identified by the script) might be close enough, and can be automated by hacking the Bitbucket exporter script. I can try to do that.

I need help identifying the contributions from users that were not identified in the above list. For example, xaint (https://bitbucket.org/xaint/magic-lantern-astro-module/) is one of them. OK, that repository is Git, but the point is - there might be ML-related contributions that were not direct forks of the main repository, AND their authors might not have a direct fork of the main repository.

Already downloaded:
- https://bitbucket.org/daniel_fort/ml-focus-pixels/ and its fork https://bitbucket.org/Dannephoto/ml-focus-pixels-danne/



2) All other interesting Mercurial repositories from Bitbucket, not related to ML, in particular, unmaintained projects (these will be lost forever, unless archive.org already crawled them).

As with the above, you can help by suggesting repository URLs.

Even better: if you know a list of all public Mercurial repositories, or if you can ask Bitbucket staff, or your favorite search engine, to supply such a list, or if you know anyone else trying to archive all Mercurial repos from Bitbucket, your help will be very appreciated :D



To check whether a Bitbucket repository is Mercurial or Git, click on Commits, then click on any commit, then - View raw commit. If it begins with HG changeset patch, it's a Mercurial repository. If it begins with an e-mail subject, it's a Git repository. If you know an easier way, please chime in.

If you can back up the repositories on your own, I'd also appreciate. What if my backups will fail?

@reddeercity: mrcrowl.hg is a Mercurial GUI for Visual Studio Code. Not relevant here. It's not a Mercurial hosting service, and it's not going to stop Bitbucket from deleting Mercurial repositories.

nikfreak

  • Developer
  • Hero Member
  • *****
  • Posts: 1182
Re: Bitbucket set to remove Mercurial support
« Reply #35 on: June 07, 2020, 11:17:42 AM »
Please don't make THE decision to selfhost a repo. I personally think it's time to move to git as it seems to be more widely adopted but ofc having backups of all repos is a good idea. I am unsure if you really need to have a backup from all forks. Mine for e.g. might be skipped if 70D and 100D got merged.

Idea: I suggest merging PRs (read: not all) as much as possible either before or short after leaving bitbucket. Everyone wanting the "old" aka "nightly stable" builds might download it from ml website. This might be a chance to "force" community as well as newbies to participate on the new platform and speed things up.
70D.112 & 100D.101

aprofiti

  • Contributor
  • Member
  • *****
  • Posts: 194
Re: Bitbucket set to remove Mercurial support
« Reply #36 on: June 07, 2020, 01:00:33 PM »
More repository with useful work not identified by the script:

https://bitbucket.org/coutts/1000d_dev/
https://bitbucket.org/coutts/5dplus/
https://github.com/400plus/400plus/ (ML Related)

2) All other interesting Mercurial repositories from Bitbucket, not related to ML, in particular, unmaintained projects (these will be lost forever, unless archive.org already crawled them).

Tried some times ago using Wayback machine to save PR's source code, but unfortunately was failing due to JS based views  and also noticed even no screenshoots in many years for pages in main repo...

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12467
Re: Bitbucket set to remove Mercurial support
« Reply #37 on: June 07, 2020, 01:28:11 PM »
Quote
I am unsure if you really need to have a backup from all forks.

Some of these forks might have useful commits that were not (yet?) submitted as pull requests; possibly fixing some personal issue of the author. Good luck identifying them manually. Yours is one out of over 500 :)

But if you have a backup of all these repos, you could write a script to identify those commits that are not present in the main repository. These commits can be reviewed later; maybe some of them could end up useful. Maybe some of them were referenced here on the forum, and forgotten. If one of us will revisit such a discussion, there will be a way to find those old commits in the archive.

Quote
Mine for e.g. might be skipped if 70D and 100D got merged.

These two are already in the main (hudson) repository, and on the main download page. Just not in "unified" yet.

So, they can be skipped without further action :)

Quote
Idea: I suggest merging PRs (read: not all) as much as possible either before or short after leaving bitbucket.

I wouldn't do any further development (including PR merging) before finding a new home for the repository (whether it will be heptapod, hg.sr.ht, github, gitlab, kallithea or whatever).

Even if I'm currently on holiday, I won't be available 24/7 for this task, so I still need to prioritize. Having copies of all the data before it's deleted, is essential in my opinion. Deciding what to merge, what needs further cleanup, running tests and so on, can be done afterwards, without time pressure.

Another important task is, in my opinion, an archive of all current Mercurial repositories (not necessarily related to ML). Some of them might be unmaintained projects, that could be still interesting for someone. I don't currently have a list of these repos, and I'm not aware of any existing efforts in this direction (my google-fu is not the best), but already started to research the topic. Edit - found some:
- https://octobus.net/blog.html (which brings quite a few bonus points for heptapod)
- https://archive.softwareheritage.org/browse/origin/directory/?origin_url=https://bitbucket.org/hudson/magic-lantern
- https://archive.softwareheritage.org/browse/origin/directory/?origin_url=https://bitbucket.org/coutts/5dplus

And yes, I'd rather complete that before the July 1st deadline, and focus on PRs afterwards, as time permits.

Quote
Please don't make THE decision to selfhost a repo.

No decision has been made yet. I'd appreciate some arguments for and against the alternatives mentioned throughout this thread (not just the self-hosting ones, but *all* of them).

Quote
I personally think it's time to move to git as it seems to be more widely adopted

I wouldn't rush to move to git either. One particular reason for sticking with Mercurial is that, throughout the forum and the documentation, there are several places mentioning Mercurial changeset IDs. While I was actively developing, I used to click these links very often, so, in my opinion, they are valuable. If we switch to another Mercurial hosting, updating these links would be doable with a find/replace. Is there a relatively straightforward way to do the same with a Git clone of our existing repository? If yes, how?

Also, besides my lack of experience with git, here's a possibly more objective reason:
http://www.legi.grenoble-inp.fr/people/Pierre.Augier/mercurial-as-a-great-version-source-control-management-tool-in-academics.html

Keep in mind that article was written before Bitbucket announced their plans to delete Mercurial repos.

BTW, if there is a way to provide a hg+git workflow, I'd look into that.

names_are_hard

  • Contributor
  • Senior
  • *****
  • Posts: 294
  • 200D idiot
Re: Bitbucket set to remove Mercurial support
« Reply #38 on: June 07, 2020, 05:49:42 PM »
That Mercurial article reads as entirely subjective to me.  It's from the point of view of someone who already knows Mercurial, and finds Git more complicated.  I know more Git and trying to learn Mercurial I found that more complicated.  I never used SVN, perhaps that makes a difference, I know Mercurial has more similarities to it.

That said, it's easy for me to copy it to Git and work there, so I don't think the choice of SCM is a big deal.  When needed I can update a local Hg copy, do the work to make an Hg PR and then go back to Git.  No problem.  So, I defer to whoever controls the repo on what they want to use.  I'm happy Alex is able to spend some time preserving it!

Regardless of whether Mercurial is technically better or worse, I will say that because Git has won, and Mercurial is slowly dying, there may be a benefit to picking a Git hosting provider.  All the big providers are Git and a bigger provider is less likely to disappear like Atlassian's Mercurial hosting.  Something allowing hg and git tooling but backed by Git feels a nice compromise to me - but again, it's not my call and I don't think it's a big deal.  Mercurial can never entirely die, it's open-source and distributed version control.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12467
Re: Bitbucket set to remove Mercurial support
« Reply #39 on: June 07, 2020, 08:30:30 PM »
It's from the point of view of someone who already knows Mercurial, and finds Git more complicated.

Right. I happen to know Mercurial, and got used to it while developing ML, but have nearly zero experience with Git, and I somehow doubt I'm the only one in this community. But, surely there are many others who started by learning Git and find it easier to work with.

Old habits die hard, on both sides, of course, which is why I'm looking for ways to provide a dual workflow. So far, I've found these:
- hg-git (the main repo should be git; not sure how "lossless" the conversion will be)
- git-hg-again (bidirectional hg-git? you use "git hg clone/fetch/push/pull" to interact with a Mercurial repo)
- git-hg (the main repo should be hg; you use "git-hg clone/fetch/pull/checkout/push" for interacting with it)
- Git-hg Mirror (for syncing a hg and a git repo? works either as web service or as self-hosted Windows service)

See also this Stack Overflow page.

TODO: evaluate these.


Quote
Regardless of whether Mercurial is technically better or worse, I will say that because Git has won, and Mercurial is slowly dying, there may be a benefit to picking a Git hosting provider.

It also feels like accepting the defeat. Though, I wouldn't exclude a git mirror on github/gitlab and a hg mirror on heptapod (most likely), keep them in sync somehow, and accept contributions from both sides.



One important thing to note about Heptapod is that, at least for now, they don't support personal forks. So, the development is more or less centralized, and all the contributors will have to be granted development access.

It works somehow like this: anyone with the Developer permission can publish draft changes, using the so-called topics (some kind of lightweight branches, at first sight closer to git branches, I guess). On these draft changes, history is editable, so I should be able to polish them as I see fit, before making them "public" (or "set in stone").

There's also support for the evolve extension, apparently another thing worth trying. In the past, I used to try not only to have readable code, but also readable commits (i.e. to squeeze all those "typo/fixup" commits before making them public, as I used to read the code history quite often). But this was time-consuming; curious to see whether the evolve extension can help.

There's also hg absorb in recent Mercurial (I wasn't aware of it), which also seems to fit my previous workflow pretty well. Apparently it's an alternative to evolve, but I need to actually try these things before drawing a conclusion.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12467
Re: Bitbucket set to remove Mercurial support
« Reply #40 on: June 07, 2020, 08:35:39 PM »
Another important task is, in my opinion, an archive of all current Mercurial repositories (not necessarily related to ML). Some of them might be unmaintained projects, that could be still interesting for someone. I don't currently have a list of these repos [...]

Here's a quick hack to get a list of all the repositories stored on Bitbucket (replace the "main" of bitbucket_hg_exporter):
Code: [Select]
def main():
    #~ project = MigrationProject()
    # no auth needed
    # force pagelen=100 (default 10) for speed reasons
    status, json_data = bbapi_json("https://bitbucket.org/api/2.0/repositories/", None, {'pagelen': 100})
    while True:
        for repo in json_data["values"]:
            print(repo["scm"], repo["full_name"], repo["created_on"], repo["updated_on"], repo["website"])
            # many other parameters available:
            #~ for k,v in repo.items():
                #~ print(k, v)
        sys.stdout.flush()
        if 'next' in json_data:
            next = json_data['next']
            status, json_data = bbapi_json(next, None, {'pagelen': 100})
            while json_data is None or "values" not in json_data:
                print("retrying", file=sys.stderr)
                status, json_data = bbapi_json(next, None, {'pagelen': 100})
        else: break

Not very fast; about 1000 repos / minute at first sight. Let's see where this ends up. Octobus found about 250.000 Mercurial projects; this could be useful for cross-checking their list. A second backup of all these repos (plain "hg clone" without metadata) shouldn't hurt :)

edit: updated the script to retry failed requests.

names_are_hard

  • Contributor
  • Senior
  • *****
  • Posts: 294
  • 200D idiot
Re: Bitbucket set to remove Mercurial support
« Reply #41 on: June 07, 2020, 09:10:45 PM »
A dual workflow would be a nice bonus, but, given the low volume of commits to ML I think keeping things easier for the few devs that know it well is the more important goal.  If you find a way that I can avoid learning Mercurial, I will be grateful, but this is a goal far down the list!  Functionally, Git and Hg are almost identical, but learning the one you don't know is pain and minimising dev pain is the important goal I think.  Even though that means I have to suffer :P

Quote
It also feels like accepting the defeat
My personal opinion only: sort of.  There's so much more corporate money being poured into secondary Git based tools that even if Hg is better, the Git experience may be better overall.  Things like hosting, integration into build systems, UI, documentation, etc.  OS/2 was better than Windows at the time, but Windows is better than OS/2 now.  If OS/2 had won, current OS/2 would be better than current Windows - but it didn't win, therefore Windows got all the money and dev time.  Windows is functionally better even though many design decisions of the internals are worse.  Picking the better option isn't defeat.  I don't think this is very relevant to ML!  I just find it interesting.

For preservation, this looks useful, it claims it can make a local backup of "pull request history, issue trackers, wikis, forks, attachments and commit comments":
https://github.com/philipstarkey/bitbucket-hg-exporter
It is designed for compatibility: "Issues can also be imported to GitHub, and commit hashes, issues links, pull-request links, URLs (including to source code), usernames, etc. will be rewritten to point to GitHub and/or the generated BitBucket archive"

I'll give that a try since it sounds useful whatever the solution ends up being.


The Octobus archive should cover ML as I understand it:
https://www.softwareheritage.org/2020/04/23/rescuing-250000-endangered-mercurial-repositories/

But we can't rely on it, since we cannot test it until after the deletion: "After June 30th, once BitBucket deletes the archived result, we will make them available to the public."

Just saw your edit, the repo archives are live, that's good.  But, it "only" list 119 branches?  I think the real number is over 300.  Their snapshot date says Oct 2019 which is odd, but I don't think would make that much difference.

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12467
Re: Bitbucket set to remove Mercurial support
« Reply #42 on: June 07, 2020, 09:31:26 PM »
I'll give that a try since it sounds useful whatever the solution ends up being.

It will fail out of the box, but if you apply my patch (linked above), it should work.

Current status: been running that script for ~ 48 hours, about 100.000 files (2 GiB) downloaded, still running. Several error messages about issue attachments, but downloading these attachments manually does work. Example:
Code: [Select]
Thread 0 (a1ex): hudson/magic-lantern: ERROR: Unexpected response code 500 for endpoint repositories/hudson/magic-lantern/issues/2753/attachments/CRASH00.LOG
Thread 0 (a1ex): hudson/magic-lantern: ERROR: Unexpected response code 500 for endpoint repositories/hudson/magic-lantern/issues/2753/attachments/CRASH01.LOG
Thread 0 (a1ex): hudson/magic-lantern: ERROR: Unexpected response code 500 for endpoint repositories/hudson/magic-lantern/issues/2753/attachments/log000.log
Thread 0 (a1ex): hudson/magic-lantern: ERROR: Unexpected response code 500 for endpoint repositories/hudson/magic-lantern/issues/2753/attachments/log001.log

Some attachments were successfully downloaded, others not. Didn't investigate yet.



The other script (list of all Bitbucket repos) identified 36700 repos at the time of writing. Will re-check tomorrow morning.

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 6968
Re: Bitbucket set to remove Mercurial support
« Reply #43 on: June 07, 2020, 09:40:56 PM »
Nice script :).
+1 for both mercurial and git.
+10 for keeping mercurial.
I created some curl "repo_ripper" downloading all forks from a bitbucket user based upon knowing the name of the main repository but the script mentioned seems taking care of it all.
Again, relieved that back up download is being done and checking into the possibility of keeping mercurial alive.
Wish my vacation would start already...

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12467
Re: Bitbucket set to remove Mercurial support
« Reply #44 on: June 08, 2020, 04:33:27 PM »
Status update:

- my fix was merged into bitbucket-hg-exporter, so you no longer have to patch the source code :)

- the "big" script downloaded the entire hudson repo (took about 50 hours, give or take) and started to download the smaller forks (15 out of 540 in ~ 10 hours? it might finish before the deadline if we are lucky :P). I'll probably stop it and just "hg clone" the forks, without saving the metadata.

- the "small" script ran for 20 hours and identified 330782 git repos and 161318 hg ones. It appears to have stopped prematurely for who-knows-what reason, so I'll try to figure out what happened. Thinking to "hg clone" each of the Mercurial ones, as a low-priority background task, and provide an archive somewhere (just in case Software Heritage misses anything).

ilia3101

  • Moderators
  • Hero Member
  • *****
  • Posts: 924
Re: Bitbucket set to remove Mercurial support
« Reply #45 on: June 08, 2020, 05:02:30 PM »
- the "small" script ran for 20 hours and identified 330782 git repos and 161318 hg ones.

Are you downloading all of bitbucket?

a1ex

  • Administrator
  • Hero Member
  • *****
  • Posts: 12467
Re: Bitbucket set to remove Mercurial support
« Reply #46 on: June 08, 2020, 06:41:39 PM »
Only half of it, i.e. what will be deleted. I assume most of these repos are small (guess: 10MiB on average?), so I guess the entire thing should fit on an external HDD that's sitting around. If it won't fit, I can delete large blobs (can be automated), attempt some deduplication (skip clones that bring no contributions on their own), or skip large repos that already found a new home (looking them up can probably be automated as well).

Or, at least, it could be a fun exercise. So far, it looks like this:
Code: [Select]
for f in $(cat all-repos.txt | grep '^hg ' | cut -d ' ' -f 2); do
  echo $f
  if [ ! -f $f.tar.bz2 ]; then
    if hg clone -U https://bitbucket.org/$f $f; then    # some repos may have been deleted?
      tar -cvjSf $f.tar.bz2 $f
      rm -rf $f
    fi
  fi
done

So far: 400 repos downloaded in ~ 40 minutes, total 1.7 GiB, less than 5 MiB/repo average.

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 6968
Re: Bitbucket set to remove Mercurial support
« Reply #47 on: June 08, 2020, 10:35:16 PM »
Pretty stoked it´s actually possible to download ALL mercurial repositories. I did something in that direction to grab my own repos and have them all downloaded into a folder I call repo_ripper. Goes something like this:
Code: [Select]
list=$(printf '%s\n' $(grep -o -E 'repo-list--repo-name" href=".{0,50}' <<< $(curl -L "https://bitbucket.org/Dannephoto/"workspace/projects/PROJ) | cut -d '/' -f3 | cut -d '"' -f1))
mkdir -p repo_ripper
cd repo_ripper
while read -r line; do
    hg clone https://bitbucket.org/Dannephoto/$line
done <<< "$list"

names_are_hard

  • Contributor
  • Senior
  • *****
  • Posts: 294
  • 200D idiot
Re: Bitbucket set to remove Mercurial support
« Reply #48 on: June 09, 2020, 03:14:56 AM »
@Danne that's what a public repo means :)

@aprofiti - thanks to you noticing some branches seemed missing in my import, I rechecked things and worked out at least some of the problems.  This is certainly a better import (maybe it's even correct):
https://github.com/reticulatedpines/magiclantern_hg_02

hg-fast-export can't handle unnamed heads (why is this even a thing? The Mercurial branch model is so confusing).  I think the mistake I made last time was to use the --force option.  While this ignores errors enough to let the script proceed, perhaps it still can't export the broken parts?  Maybe even their children too?  This time, I allowed the script to error, it was always to do with unnamed heads, I then ran this on each revision number listed in the error:
Code: [Select]
#!/bin/bash

hg update -r $1
hg branch "$1_fix_unnamed_head"
echo "$(hg log -r $1 --template '{desc}')"
hg commit -m "Naming branch from commit: $(hg log -r $1 --template '{desc}')"

Git doesn't allow branches starting with "/", and something in hg-fast-export also doesn't like spaces in branch names, this was the mapping file I ended up using:
Code: [Select]
"/raw_recc-added-121-and-11751-aspect-rati-1382109783386"="raw_recc-added-121-and-11751-aspect-rati-1382109783386"
"/added-1792-resolution-1390499711758"="added-1792-resolution-1390499711758"
"crop hack disable"="crop_hack_disable"
"refs/heads/Windows build_tools"="refs/heads/Windows_build_tools"
"Windows build_tools"="Windows_build_tools"
"refs/heads/MinGW x-compile"="refs/heads/MinGW_x-compile"
"MinGW x-compile"="MinGW_x-compile"

Here's the script arguments I used, should anyone else want to do similar:
Code: [Select]
../hg-fast-export.sh -r /home/user/path/magiclantern_hg_clone/magic-lantern -B /home/user/path/branch_mapping.txt -n --hg-hash

Danne

  • Contributor
  • Hero Member
  • *****
  • Posts: 6968
Re: Bitbucket set to remove Mercurial support
« Reply #49 on: June 09, 2020, 08:20:08 AM »
@Danne that's what a public repo means :)
I know what public means. Still stoked about potent scripts swiping clean.