Messages

I also have some of those:

Code Select Expand

Processing JannisKhammas/railbound-frontiers ...
applying clone bundle from https://api.media.atlassian.com/file/blahblahblah
adding changesets
adding manifests
adding file changes
transaction abort!
rollback completed
abort: stream ended unexpectedly  (got 196 bytes, expected 4096)

I wonder if restarting script will get them?

started with a dash... caused trouble

Still it will be fun to work with those later  (my favorite bash test on newcomers is to create " -rf" directory in / and ask them to remove it from root account)

Script has a problem (this failed on repos 19)

Code Select Expand

Processing -1/objectlistview ...
hg clone: option -1 not recognized
hg clone [OPTION]... SOURCE [DEST]

make a copy of an existing repository

options ([+] can be repeated):

This is not the only example of username starting with "-". Simple wrapping variables in quotes doesn't work, those would need to be properly escaped.

Status:
Primary:

Code Select Expand

hg-repos-17: 43.58% downloaded, 1.34% errors, 55.08% todo
hg-repos-18: 38.51% downloaded, 0.57% errors, 60.92% todo
hg-repos-19: 29.07% downloaded, 0.34% errors, 70.59% todo
hg-repos-22: 4.22% downloaded, 0.10% errors, 95.68% todo
hg-repos-23: 2.83% downloaded, 0.23% errors, 96.94% todo
hg-repos-24: 9.56% downloaded, 0.31% errors, 90.12% todo
I'm a little worried about 22 and 23 lagging behind 19 slowed due to an error above (hg decided to show interactive help via less...)
I'm thinking about moving 2nd half of 17-19 to secondary location to leave bandwidth free for 22-24. I had to limit it to 60mbit/s to keep everything working and it's saturated all the time

secondary:

Code Select Expand

hg-repos-20: 8.14% downloaded, 0.12% errors, 91.74% todo
hg-repos-21: 11.43% downloaded, 0.08% errors, 88.49% todo

Also noticed the .hg/store/data directory has many file names resembling the ones in the source code, and got a crazy idea: what if we group similar or identical files together? Would the compressor find repeated patterns easier?

IIRC not possible (at least easily) on classic "on the fly" compressors like gzip. But afair squashfs does block-level deduplication. Thus grouping forked repos and squashing them in one file may be the way to go. mksquashfs / unsquashfs should be available in every distro...

[e @ 11PM] To not spam. I added 22-24:

Code Select Expand

hg-repos-17: 35.56% downloaded, 0.99% errors, 63.45% todo
hg-repos-18: 34.18% downloaded, 0.47% errors, 65.35% todo
hg-repos-19: 27.13% downloaded, 0.27% errors, 72.60% todo
hg-repos-22: 0.14% downloaded, 0.00% errors, 99.86% todo
hg-repos-23: 0.02% downloaded, 0.00% errors, 99.98% todo
hg-repos-24: 0.24% downloaded, 0.00% errors, 99.76% todo
So far it took about ~500 gigs.

[e @ 11:30PM] And just added 20 and 21 on secondary location:

Code Select Expand

hg-repos-20: 0.06% downloaded, 0.00% errors, 99.94% todo
hg-repos-21: 0.39% downloaded, 0.00% errors, 99.61% todo

[Never, ever access WSL filesystem this way! This will mess up linux FS. Microsoft points that in documentation. MS dev blog about this]

You use WSL, right?

Code Select Expand

kitor@kitor-p70:kitor$ cat /etc/debian_version
bullseye/sid
kitor@kitor-p70:kitor$ systeminfo.exe | grep -e "^OS Version"
OS Version:                10.0.18363 N/A Build 18363
kitor@kitor-p70:kitor$ ls ~/.ssh | grep id_rsa
-rw------- 1 kitor kitor 2.6K Feb 15 17:02 id_rsa
-rw-r--r-- 1 kitor kitor  569 Feb 15 17:02 id_rsa.pub

If you did some werid stuff like symlinking your windows home dir to WSL one, then yep - this will work as you described (won't work) as you try to write linux permissions to NTFS partition over virtual filesystem.

Also, replying to your older post: https://www.magiclantern.fm/forum/index.php?topic=24420.msg228084#msg228084

The default home folder in ubuntu is located at:

Code Select Expand

C:\Users\xxxx\AppData\Local\Packages\CanonicalGroupLimited.UbuntuonWindows_79rhkp1fndgsc\LocalState\rootfs\home\username

Replace (xxxx) with with your windows 10 user. Replace (username) with whatever username you used when installing ubunutu.

I would suggest to cut the username folder and paste it wherever you want to store these downloaded repos. Then drop a symbolic link back into the original ubuntu home location.
I use a link shell extension to make that task easier.

Never, ever access WSL filesystem this way! This will mess up linux FS. Microsoft points that in documentation. MS dev blog about this
Use file explorer to access \\wsl$ with file explorer - this is the only official and safe way [1] [2]

This path that you provided is stored on Windows side using some kind of trickery to keep linux system working properly (and thus keep permissions working). If you alter it from Windows, you will experience all sort of problems, similar to ones you described.

If you want to access Windows path from Linux, use /mnt/<letter>/*. If you want to symlink it, do it using ln -s on WSL (linux symlink on linux side). If you need to access linux FS from windows - use mentioned \\wsl$ share.



[1] You can mount it as an network share by the way. So your linux FS will be visible as disk with assigned letter in explorer.
[2] https://devblogs.microsoft.com/commandline/whats-new-for-wsl-in-windows-10-version-1903/

How you check the percentage?
I'm running 17, 18, 19 since midnight.

[e] Ok, I missed the python script

Code Select Expand

hg-repos-17: 14.37% downloaded, 0.43% errors, 85.20% todo
hg-repos-18: 14.57% downloaded, 0.21% errors, 85.22% todo
hg-repos-19: 10.17% downloaded, 0.14% errors, 89.69% todo

You need to add your ssh public key to bitbucket. And run HG once by hand (without disabling interactive shell) to accept remote ssh pubkey.

I just wonder, ssh clone wouldn't be faster? I think that http is overhead here.

quick test (ofc 1 run is not enough to prove anything):

Code Select Expand


# time hg clone --config ui.interactive=false -U https://bitbucket.org/hudson/magic-lantern ./magic-lantern-http
requesting all changes
adding changesets
adding manifests                                                                                                     
adding file changes                                                                                                   
added 18025 changesets with 40250 changes to 4090 files (+60 heads)                                                   
new changesets 4d0acc5c0792:f7947b627e33

real    1m20.908s
user    0m30.178s
sys     0m2.235s

# time hg clone --config ui.interactive=false -U ssh://[email protected]/hudson/magic-lantern ./magic-lantern-ssh
requesting all changes
adding changesets
adding manifests
adding file changes                                                                                                   
added 18025 changesets with 40250 changes to 4090 files (+60 heads)                                                   
new changesets 4d0acc5c0792:f7947b627e33

real    0m46.857s
user    0m26.173s
sys     0m3.698s



Yup, I just saw that those had no links. Anyway, I won't start before moon (still 4 hours) so Alex may come to rescue 

[e]Just started hg-repos-17 for test. If those 22-25 won't be available, I'll run 17-21 over night.

Well, 8TB sounds excellent for collecting the downloads from all participants (in max. 2 weeks, possibly earlier), and running the scripts for deciding how to compress and archive the entire thing (which I didn't write yet, but thought about).
At 20 MiB / repo, 250.000 repos would require about 5 TB. Let's hope it won't exceed 30

I'm setting up env for those:

hg-repos-22 (TBD)
hg-repos-23 (TBD)
hg-repos-24 (TBD)
hg-repos-25 (TBD)

I'll run it over night. I have 100/50 connection there, so not great, not terrible; on 2nd location I can setup about 2TB (with 100/100 connection) for side works if needed.

Well, 8TB sounds excellent for collecting the downloads from all participants (in max. 2 weeks, possibly earlier), and running the scripts for deciding how to compress and archive the entire thing (which I didn't write yet, but thought about).
At 20 MiB / repo, 250.000 repos would require about 5 TB. Let's hope it won't exceed 30

Primary has dual E5-2630 v2 and 96 gigs of ram, so it's fine for some heavy lifting. One limitation is that I need to keep bandwidth heavy things over night / cap it to half during day as it hosts all of my production sites.
On primary I can add 4TB over local gigabit network - as I have cold spare laying around (unfortunately all local storage bays are full of disks).

If somebody wonders, this is my homelab

Alex, if you want some env to experiment on, ping me on PM.

As I'm in the hurry and just noticed this thread leaving home - is there's anything still to be downloaded? I have ~8TB free space on my server.

https://www.canonrumors.com/canon-announces-100-firmware-update-for-stop-motion-photography/

Wow, $100 for increase in LV resolution. With no HDMI output, and no focus peaking in many scenarios, so camera is basically useless for other things than stop-motion.

When you steal information from website, it's nice to at least provide original source.

https://kameratrollet.se/canon-write-speeds/

[CFexpress to SATA]

I think it's about sata, not ata ... The last standard of sata is sata3 (750 MB/s)

In case of R5 it's not even SATA (CFast), but Canon skipped directly to PCI Express / NVME (CFexpress).

Oh yeah lol, I miss understood. However there are many examples doing CFexpress to SATA on YouTube. Shouldn't be a big deal.

Are you sure? CFast to SATA do exists, as CFast uses SATA protocol. But CFexpress is NVME (PCI Express), that's why I wonder if there will be similar trick possible.
But even if possible - probably a little harder due to PCIe signalling requirements.


By the way - there are three types of CFexpress cards - A, B, C. They differ in physical dimensions and number of PCIe lines. I can't find any information which one will be supported here, but I bet for type A, which is physically similar to SD card.

I just got reminded about CF-to-SATA thread. And I was going to post here that finally we will be able to attach HDDSSD to our cameras, as there are CFast to SATA adapters (CFast is in mostly a fancy connector for SATA).

And then I checked again and it's CFexpress  Didn't even notice that there's a new standard already. PCIe v3 x4 / NVME - I wonder if someone will make  m.2 NVME -> CFexpress adapter 
They are a couple of CFexpress -> (m.2 | pcie) adapters, and while I can't find any photo of "the other side" of pcbs, all PCIe traces seems to run directly between connectors.

What does this mean? Is it a good new?

That's why I suggested some time ago to replace "Magic Lantern Rescue" header with different string, at least for new developments where there's no real ML build - as this may suggest otherwise. Not that I experienced it when posted similar shots for EOS R on twitter  .

"Custom code test" would get people a little less excited.

https://www.usa.canon.com/internet/portal/us/home/explore/product-showcases/cameras-and-lenses/eos-r5
https://www.canonrumors.com/canon-announces-more-mind-blowing-specs-for-the-canon-eos-r5/

Just one quote:

Internally recorded and uncropped 8K RAW Recording up to 29.97 fps with Dual Pixel CMOS AF, and HDR-PQ Recording (H.265) capability.

Well, price will hurt for sure.

[EOS 5DS R]

I guess our two new colleagues can't see the difference between EOS 5DS R and EOS R5.

MLV on 40D  Even if that's single frame, congrats. I'll check it just for fun, after whole covid crap ends (dad has pristine 40D with only ~10k frames).
Wish I had enough knowledge to move R forward, but with time that I can spent, I'll maybe finally upgrade my cam to latest FW and update the stubs...

Aside from technical improvements, I'll be switching for sure in future (unless they screw something really bad). Missing thumb wheel was hard for me to live with, even if I got used to after a year
I just fear the pricing, as if those are not just marketing words, I believe it will be more expensive that R was.

Maybe he formatted card to FAT32.

As for Twitter on home page, I accidentally posted this in EOS M thread 

The code is there, but it's commented - so it was intentional.
Maybe it stopped working? I checked and current widget embed code is slightly different than one on site: https://publish.twitter.com/

Does anyone know if ML for EOS R is expected to enable 10bit recording in 1080p over HDMI? I love the look of 10bit in C-log, but I don't have the hardware for 4K.

ML for R (or anything Digic 7+) is not expected to exist any time soon, and you're asking about specific features...

So far we have only proven that it's possible to run custom code.
I guess @a1ex that  "Magic lantern rescue" is misleading and should be replaced with another name for early development ports. Just a string swap may keep people a little less excited.

Can ML currently allow this on the EOS R?

Is ML for R mentioned anywhere?

I need to upgrade (and dump) 1.3 and 1.4 and hopefully update stubs for 1.4. Yup, I'm still on 1.2.

This info was published before those firmware were even mentioned elsewhere, so I assumed it was an error.
But yes, that makes sense.