Messages

I'd prefer these LUA questions to remain on the forum, so they are easy to find.

Trying to join all of the threads will be problematic, as they will be sorted by date, resulting in a mixed jumble of a thread with no logical structure.

I also actually prefer the questions to all remain separate, as I think it's easier to search for thread titles, and being able to keep the (strictly related) questions contained in one thread for easy reading. Just my 2c.

I found two previously used IP's in the range blocked because spam reasons. The related IP lists were added fairly recently, so they may have worked previously.

IPs used in error messages doesn't match any of the blocked ranges though. If you want to shoot me a PM with the IP's you have trouble with I can look into it further.

Hopefully you were greeted by a message such as this:

Your IP has been marked as a spam host. If you believe this is in error please message a global moderator. You might also try Discord: https://discord.gg/uaY8akC

If not, it may be an older block that hasn't had it's message updated.

These blocks are all manual. Until the forum software gains some form of automated spam list protection, the lists will remain manual based (so may become outdated). Given the level of spam that has been reduced by aggressively blocking IP ranges lately (and yes, I have been nuking VPN hosts if they host spam bots), the IP block list is only going to grow larger, not smaller. Again, PM's will help with IP's being blocked as I might be able to weed out false positives.

Because your post is a troll question.
https://wiki.magiclantern.fm/faq#any_progress_on_xyz

also experience: its always easy to say: "meh, i would have done it that way". but the answer to this is - "you haven't done it at all, so get cope with it "

Yes, and this goes both ways (new devs <=> old devs), which is a point I was trying to make.

refactoring the build system

This is a perfect example (the other tasks you have taken are also probably good examples, sorry I haven't been following along to closely). The build system is as it is, because it works. But you've seen an opportunity to refactor the code and jumped straight into it. You made a comment on discord along the lines of "well, no one is complaining about my proposed changes, so...."

That was after a Q&A session where you sought clarity on a few details of the build system. This to me is exactly how things should proceed.

Far to often, it seems to end at a position where someone ends up asking for permission to proceed (do you think I should do this? Is it ok if I do this?). This is asking to ask.

I've learnt a lot about the code layout and how it works with Qemu, boot process on cams, etc.

This is the value IMO. Reaching an end goal of providing something useful, is obviously icing on the cake, but learning along the way also has it's value. If you're focus is entirely on ML, in the least, you are now in a better position for ML in the future (I reply to you, but it applies to everyone).

This project is not a corporate. There are no expectations to deliver results. You won't lose your job because you spent 2 months working on something, but didn't deliver an end product. If you had some fun along the way, learned some new things along the way, perfect.

I am not waiting around for guidance.  But I could do better with guidance, and be of more benefit to ML, not just what I find interesting.

You've made good points regarding organization. I expect these to be addressed when the fiscal problem is sorted as by nature, the project will be required to become more organized. Just like the bitbucket issues forced a requirement to focus on issues related to that, so will the fiscal hosting force a requirement into other areas.

But.... What you find interesting, shouldn't in my opinion, be dismissed as if it doesn't help ML, or only helps to a lesser extent. A typical user sees no value in the work you have done at all, because it doesn't solve "when will ML be available for" or, "when will feature XYZ be available". But the value is there, if for no other reason, then you had fun doing what you did.

The complexity provides ample opportunity. This is a multi-faceted project with plenty of avenue to assist.

Things are fragmented. People come and go, available time ebbs and flows, new challenges constantly appear. Take the closure of bitbucket for example, that being a significant burden. Having to find another suitable host, work with the new changes, etc etc. All of this diverts resources (time) away from the main project, to tasks that don't exactly drive the project forward, and yet are inherently necessary due to circumstances.

Asking to ask, while apparently polite, is, to some extent, just another burden. If you think you can assist in a specific area, take charge of that task, take the burden of that task off of the hands of another community member. Don't be afraid to step on toes, because after all.....

Much of what looks like rudeness in hacker circles is not intended to give offense. Rather, it's the product of the direct, cut-through-the-bullshit communications style that is natural to people who are more concerned about solving problems than making others feel warm and fuzzy.

We all make mistakes, it is part of the learning process.

You may venture down a path for 3 days working on some specific thing, at which point after some discussion it inevitably ends with having to cease completing that task. Would that be considered to be a waste of time? Possibly, but asking to ask could enviably lead to a months worth of waiting for someone to answer. Which avenue wasted more time? Surely in the 3 days being spent working on a task, something was learned, yes? What do we learn by waiting a month for someone to say no, or giving up on waiting.......

I've lost count of the amount of people who have appeared, proclaimed to be in a position to help, been offered assistance, then disappeared. I don't mean to criticize those people at all, I just simply ask to respect my healthy load of skepticism towards people who ask to help.

Some background reading on issues that have impacted main development:

https://www.magiclantern.fm/forum/index.php?topic=24420.0
https://www.magiclantern.fm/forum/index.php?topic=24548.0
https://www.magiclantern.fm/forum/index.php?topic=25717.0

There is still development work happening, simply that it's not very noticeable on first glance. Look at the popular threads in the camera specific section and join us on discord
Until all of the linked issues are fully resolved, I don't expect any changes to the current workflow.

Ideally we would like to have a stable build of ML, however, feedback is often lacking and bugs can go unnoticed by the dev team for months, if not longer. There are efforts to mitigate these problems, search "qemu".

https://builds.magiclantern.fm/experiments.html

Can we safely assume it's for the default branch "unified"?

My recollection is that a branch always had the branch name in the link, so yes, it would be safe to assume that IMO.

Otherwise, replacing it with "unified" could also work.

I added three IP addresses manually to the blocklist, from which about 100 posts had come, and that by itself has stopped over 19,000 requests in a few days.

Those IP's are from a company that supplies shared hosting, VDS/VPS and dedicated servers. Those IP's also triggered in feeds on https://otx.alienvault.com/

There is no need for members to be browsing this site from a dedicated hosting server, so I've nuked that entire IP range.

The amount of spam registered accounts and posts is rather crazy.

The forum appears to have some form of spam account protection, but the last hit on that is from back in early 2017, and thus appears to no longer be working. Possibly around the time that manual post approval became a thing? The current solution does work well for catching spam accounts and posts, but suffers from the issues in the OP and the amount of manual intervention required.

I've just finished updating IP blocklists on my router firewall and instantly thought about what can be done with this forum, and found this post . Some form of automated filtering is needed imo. I have manually blocked some IP ranges previously which seems to help.

This site contains an automatically updated feed of numerous IP lists used for filtering. Some interesting ones for a forum are probably this one (free) and this one (not free (the self contained service (the IP list is free))).

There's also a free spam blocker mod for SMF that's probably useful.
I like the option "Blacklist of IP's/Ranges added to the ban list by Spam Blocker" which looks similar to what my firewall does, which allows the import of multiple lists and presumably has some form of auto update.


Some more resources that may be helpful.
Adds some new settings based on post counts to help curb link spammers.
Prevent Adding Signature Images And Links.

https://www.magiclantern.fm/forum/index.php?topic=8431.0

edit: the above link will lead to a module and script helpful for timelapses, but doesn't really address your initial issue.

In either case, the car driving past will I guess effectively ruin that exposure. If you want to rely on Auto_ETTR, I would think that "midtone/shadow SNR limit" may work well here: https://www.magiclantern.fm/forum/index.php?topic=5693.0

Which links were you after specifically. The pre-compiled module is available in the main ML downloads, the source code has moved over to here: https://foss.heptapod.net/magic-lantern/magic-lantern/-/tree/branch/unified/modules/autoexpo

The img link is broken. The direct link in your other (deleted post) lead to a broken page.

Perhaps you could just link the whitepaper.

The "chat archive bot" should take care of archive issues, right?

Hi @Danne,

There was a fairly long list of unapproved posts in this thread that you may want to visit and address. Beginning https://www.magiclantern.fm/forum/index.php?topic=9741.msg230819#msg230819

Cheers.

Heated debate is fine, but for personal attacks (not necessarily from you), yes, there will be a need to issue bans.

Somehow or another this thread turned into a decade old debate on ii vs iii.

If the debate can't at least stick to merit, I'll close the thread and issue bans for personal attacks.

Being the development section of the board, I've always been of the preference of this section being strictly segregated to development discussion.
Ideally, I think the initial revival post is not suited to this thread. It's a general (non-developmental) discussion post tethered to this thread by the word temperature.

There has been a clear push by the userbase asking to be careful with stifling discussion, and given the current climate of ML, especially the development of the project, this needs to be respected. Discussion needs to take precedence over procedure.

IMO, the forum police comment is valid. There is a process in place to report concerns to mods.

I've made effectively no progress since my last post. I got a 7zip command line working to store and move to the synced storage drive, but it is also very slow.

If memory serves correctly, on a folder with repository 07 and 21, it ran for at least 1 day and only made it to M in the alphabet. I had to halt the process, which then required deleting the folders already stored before restarting 7zip, another massive process in itself.

This is still accurate for me.

Code Select Expand

hg-repos-05: 37.86% downloaded, 3.05% errors, 59.09% todo
hg-repos-07: 92.83% downloaded, 7.17% errors, 0.00% todo
hg-repos-13: 94.49% downloaded, 5.51% errors, 0.00% todo
hg-repos-17: 97.09% downloaded, 2.91% errors, 0.00% todo
hg-repos-18: 67.72% downloaded, 1.16% errors, 31.12% todo
hg-repos-19: 58.56% downloaded, 0.46% errors, 40.98% todo
hg-repos-20: 99.33% downloaded, 0.67% errors, 0.00% todo
hg-repos-21: 99.81% downloaded, 0.19% errors, 0.00% todo
hg-repos-24: 47.42% downloaded, 0.71% errors, 51.87% todo

i'm here:

Code Select Expand

hg-repos-05: 37.86% downloaded, 3.05% errors, 59.09% todo
hg-repos-07: 92.83% downloaded, 7.17% errors, 0.00% todo
hg-repos-13: 94.49% downloaded, 5.51% errors, 0.00% todo
hg-repos-17: 97.09% downloaded, 2.91% errors, 0.00% todo
hg-repos-18: 67.72% downloaded, 1.16% errors, 31.12% todo
hg-repos-19: 58.56% downloaded, 0.46% errors, 40.98% todo
hg-repos-20: 99.33% downloaded, 0.67% errors, 0.00% todo
hg-repos-21: 99.81% downloaded, 0.19% errors, 0.00% todo
hg-repos-24: 47.42% downloaded, 0.71% errors, 51.87% todo

My issue atm is this:


Don't be fooled by the number of items remaining, windows is still counting the total number of items.

I tried zipping the folder up, got 18 hrs in (50%) and the linked online drive disconnected. 
Google file sync opens a good number of connections, but given the small size of most of the files the upload speed is still limited.
I need to find some way to select the content of the top folder and individually compress each folder within the top folder. That will give me some redundancy from a crash, and increase the size of the files being uploaded which will increase the overall upload speed. Otherwise I'm going to be limited to how many more repo sets I can get in the next 10 days.

I'll be able to finish what I've got going, but otherwise I'm effectively out of HDD space until data is uploaded.

Code Select Expand

hg-repos-05: 0.41% downloaded, 0.01% errors, 99.58% todo
hg-repos-07: 92.83% downloaded, 7.17% errors, 0.00% todo
hg-repos-13: 52.67% downloaded, 2.44% errors, 44.89% todo
hg-repos-17: 68.06% downloaded, 2.16% errors, 29.78% todo
hg-repos-18: 64.46% downloaded, 1.10% errors, 34.44% todo
hg-repos-19: 20.84% downloaded, 0.20% errors, 78.96% todo
hg-repos-20: 59.69% downloaded, 0.49% errors, 39.82% todo
hg-repos-21: 99.81% downloaded, 0.19% errors, 0.00% todo
hg-repos-24: 0.08% downloaded, 0.04% errors, 99.88% todo


Been some stopping and starting as I've needed to rearrange data (and trying to get SSH working yesterday). Advantage is the repos are being checked multiple times (for data already downloaded).
Found some space on another HDD to get 2 more repos going (05/24).
As soon as 07 is finished I can begin to upload 07 & 21 and start another 2 repos. edit:Done and uploading

I can add a user to my gsuite account (for a month or 2) if a central host is appealing and needed. Not sure if google will like multiple IP's uploading on the one user account, but I guess there's only one way to find out. There's a documented 1TB space per user, but in practice it's unlimited.

edit: 37 Mbps (actual) upload = 15GB / hour. This will take some time.

What is the best script command to delete duplicate material?

For instance I have some hg-repos-20 on a HDD from when I was initially downloading everything to a single drive. I have since begun downloading hg-repos-20 to another drive, and I would like to delete any fragments of hg-repos-20 from the initial drive.

Can someone confirm any compression being used before I begin uploading to a hosting server.

I need the output of the script to a different location then the WSL Ubuntu install location.

The WSL Ubuntu is installed on a 120gb SSD, which is clearly not suited to this task.

I need to be able run multiple scripts to multiple different HDD locations to be able to keep the WAN connection saturated without thrashing the HDD.

Couldn't get ssh working here.

ssh very strict on permissions, and needs either 400 or 600 permission on the key file.

chmod no worky on symbolic linked files.

Tried all manner of windows permission settings and can only get to 444.

Tried different folders within the ubuntu home folder, but ssh-add won't stop bitching to me about file/folder permission outside of the username folder. Yet, ssh-keygen works fine ffs. Can create keys, but then can't add them.

edit: oh, and I deleted the symbolic link username folder from the home folder, added a new folder, and ubuntu still wants to link to the symbolic linked folder. Maybe a restart will fix it. Will finish downloading what is started first, then play again.

Currently at:

Code Select Expand

hg-repos-07: 23.08% downloaded, 1.96% errors, 74.96% todo
hg-repos-13: 25.35% downloaded, 1.18% errors, 73.47% todo
hg-repos-17: 24.25% downloaded, 3.39% errors, 72.36% todo
hg-repos-18: 22.46% downloaded, 0.51% errors, 77.03% todo
hg-repos-19: 15.49% downloaded, 0.27% errors, 84.24% todo
hg-repos-20: 16.87% downloaded, 0.21% errors, 82.92% todo
hg-repos-21: 20.56% downloaded, 0.26% errors, 79.18% todo

Guess I should update the scripts.