I would appreciate, if someoe explain me about ML and TL and their difference. Thanks.
Magic Lantern (ML) is the main branch of development and Tragic Lantern (TL) is a separate branch that has been doing some development on specific cameras like the 600D/6D/50D/EOSM/7D.
Since the developer of Tragic Lantern (1%) has not taken the effort to merge his code back to the main branch now there are some builds in the main branch that lack the functionality added to those specific cameras worked on TL.
Nightly builds are done from the main branch, and therefore are not up to date with the latest improvements done in Tragic Lantern.
I am not a programmer but I assume that the task of bringing TL into ML will be nearly impossible without the cooperation of 1%.
http://www.magiclantern.fm/forum/index.php?topic=9510.msg78084#msg78084
http://www.magiclantern.fm/forum/index.php?topic=7503.msg83832#msg83832
I've been running TL on multiple 600D's for quite some time for commercial work and haven't experienced any issues. I believe 1% is a competent coder and I'm pretty sure he wouldn't release anything that would jeopardize our cameras.
But, that said, after reading the two topics that dmilligan posted above, I've decided to heed a1ex's warning once again.
I've been known to do some stupid things while recording, including turning off the camera instead of hitting the stop button to end the recording, so as a precaution and to hopefully save the lives of my cameras I'll revert back to using the nightlies.
I could probably just get by with using 2.3, but there are so many new features and the new look make my little 600d's power houses on the job.
I'd love to see some of TL features merged into the main branch, especially the video hacks and maybe the advanced bitrate control. So, hopefully these guys will work it out.
Although I agree that keeping the code unified is important, I'm a little disappointed at the tone of some comments directed towards 1% who, like a1ex, g3gg0 and other devs spend a lot of time working on the code and, importantly, spends his time working on cameras that would otherwise not be as developed as say the 5D MkIII.
As a1ex pointed out in another thread, there is an etiquette to open source that, in an ideal world, should be followed so that the knowledge is being shared in both directions. This is actually happening with Tragic Lantern as the code is freely available in 1%'s repo(s) on Bitbucket and the main issue being that 1% is not himself pushing everything back upstream for review. This (I think) is for a couple of reasons. The first is that 1% works at his own pace and is often quick to implement/change and rework ML code across several camera bodies and second, some of the TL code is classed (by a1ex) as potentially dangerous and will never be approved.
While 1%'s approach may be a little disruptive I think it's also very important to point out that his input on several cameras has been quite significant and rather than criticizing him for not playing ball when it comes to pushing back code (for whatever reason) it should be acknowledged that he is a valuable asset to the Magic Lantern development team and perhaps he just needs a little help from knowledgeable individuals who can push his work back to the main trunk, enabling him to continue doing what he does. This is something I think might be possible for someone at my level although I do need some guidance in how to do it.
As someone who only compiles a fork of Tragic Lantern, I admit I have little knowledge of the actual code and have probably not helped matters when uploading development builds without highlighting the potential dangers to users, however, I personally have had no issues when running Tragic Lantern builds on my 600D and 50D (although I know the dangers are real). After reading comments made by a1ex in another thread I can now see a couple of areas where I can contribute more by removing some developer specific features and adding clearer warnings to the 50D alpha builds so that users will understand better.
I guess what I'm trying to say here is that everyone is on the same team and although devs may differ in their approach and working methods this shouldn't be cause for animosity and ill-feeling. Words taken at face value on a public forum can often be misinterpreted and misunderstood and we, especially the non-coders, shouldn't be adding fuel to the fire potentially, making these valuable and knowledgeable developers think that they are not appreciated..... because they truly are!
Agree with Andy600.
Look in the 50D thread: back then, when GregoryOfManhattan was active, he was pushing everything to main tree, and it was pretty nice, we were exchanging ideas, test results and so on. After he left, I don't remember seeing anything pushed back to the mainline. Basically, the main 50D builds seem abandoned (also EOS M, 6D, and 7D, and the 600D is not too far from that), and soon I'll probably have no other choice than disabling these nightly builds, like I already did with the EOS M.
QuoteThe first is that 1% works at his own pace and is often quick to implement/change and rework ML code across several camera bodies
Okay, but more than one year without a single pull request seems a little too much for me. It no longer feels like collaboration, but like competition.
Quotesome of the TL code is classed (by a1ex) as potentially dangerous and will never be approved.
Not in the current form, but most of the stuff can be reworked in an acceptable way. For example, I'm trying hard to ensure that, when feature X is disabled, it really does nothing (not patching Canon code, not altering data structures and so on). Also, when you have to disable a Canon safety check (assert), it should be used as a last resort (so look for some alternative method first). If you really have to use it, you have to make sure it won't cause harm (write a description, something I can understand, not just obfuscated cache_fake calls that are impossible to review). Showing that it seems to work is not enough for me, because I've had enough bad experiences with problems happening with very low probability (which are hardest to diagnose). You know, race conditions, buffer overflows... stuff like that.
Quoteperhaps he just needs a little help from knowledgeable individuals who can push his work back to the main trunk, enabling him to continue doing what he does. This is something I think might be possible for someone at my level although I do need some guidance in how to do it.
I recommend taking one thing at a time (for example, display filters on camera X, or FPS override improvements on camera Y). Ideally you should be able to simply pull a changeset or a group of changesets that does just that (but this is impossible to do without the cooperation from 1% - you should try to do only one thing in a changeset, not 3 unrelated things). So, the alternative approach is to take the diff between ML and TL (easy with the rdiff extension), pull the relevant lines, apply these changes manually (or if you feel confortable to edit a huge patch directly, that's fine too), describe what that change does and submit a pull request.
When raw capability first came to be, you had to run TL to get this feature on the 600D. There were links in the main raw forum to TL as the source for the 600D. I think those threads gave it credibility as to being safe to run. Other than one small post by a1ex about sraw, I never knew of any other potential hazards in the code, so I continued to use it.
And, if you look at the number of downloads, there are a lot of people either running or at least tried TL, who I'm sure are not aware of these issues or even this discussion.
I recommended TL to a lot of people including friends, which I now feel obligated to point out A1ex's concerns to them and let them make the decision whether to continue using it or not.
Like Andy600, I too am guilty of compiling code for other users from 1%'s fork and sharing it with others, and (knock on wood), no one has bricked or harmed there cameras.
In no way is this meant to be a derogatory statement aimed at 1%, I have nothing but respect for the man and I wish I had his skills.
I too hope they can all work together to get this concern ironed out.
My hat off and respect to all of the coders!!!! :-* :-* :-*
But if I have the option, I will take the (free) car with a hand brake. ;D ;D
Its not competition, ML still drives most (like 90%) of the features... I never made a pull request because I have no clue how.. I know how to bring them in but not push them out. The diff seems like the best idea. Plus previous laptop only had a 40GB so with all ida files/arm console files/etc I was just out of room. Most of my time is spent adding ML stuff, fixing conflicts and testing stuff.. But this machine can actually play back the H264 video from 6D/600D and I have 128GB to work with.
Quote from: 1% on October 22, 2013, 12:33:05 AM
Its not competition, ML still drives most (like 90%) of the features... I never made a pull request because I have no clue how.. I know how to bring them in but not push them out. The diff seems like the best idea. Plus previous laptop only had a 40GB so with all ida files/arm console files/etc I was just out of room. Most of my time is spent adding ML stuff, fixing conflicts and testing stuff.. But this machine can actually play back the H264 video from 6D/600D and I have 128GB to work with.
Cool dude, get your code tweaked so it passes ML safety inspection and get it in the main repo. You have too many good features not to include them in ML.
It pretty much is ML with some tweaks and extras. Someone merging back would be great too... its a bit of time to do new stuff + fix merges + test on like 3-4 cameras and then merge it back after that.
I.e, I can work on audio support for EOSM or I can learn pull requests, make a new repo, add everything to merge back and then fix the conflicts when merging it back into my source, etc.
Excellent!
By the way I sent kisses since I cannot send money since you took the PayPal link down. Ha ha ;)
You could probably use a new laptop! Although I could only contribute a few bucks!
I've used TL since I discovered the glory that is Magic Lantern. Even bugged 1% a few times when I first joined and he (or she?) has always been super cool. I've trusted the ML and TL builds on my own work and paying shoots, a few of which being pretty damn tense (ever shoot a music video with a room full of drunk, coke-money rappers and marijuana smoke so thick that it looked like a got-damn fog machine was on?). I'm no coder, but I've done my fair share of various videography and photography jobs. When it comes down to it, all of these guys are revolutionizing the cinematography world and deserve the utmost respect.
With all that said, where's my damn slice 87, 1%?!
;D
Lol yea, that needs to be ported. Its easy for shit to get put on the back burner with so many incoming changes.
Quote from: 1% on October 22, 2013, 12:33:05 AMIts not competition, ML still drives most (like 90%) of the features... I never made a pull request because I have no clue how.. I know how to bring them in but not push them out.
Well, if that's the issue (is it?), then here's the solution:
1. Log in to https://bitbucket.org/OtherOnePercent/tragic-lantern-6d
2. Click on "Create pull request" (that's the button on the top right, even I found it, and that's saying something)
3. Alex will merge this request asap, then take some days's work to revert the stuff he doesn't like and ask 1% if in doubt about something (don't update tl during this phase)
4. When Alex is finished, merge the main ml tree again - done.
5. Repeat procedure every few weeks whenever you feel you've done some significant & stable work on 6d/7d/eosm.
Quote from: a1ex on October 21, 2013, 08:44:40 AM
It no longer feels like collaboration, but like competition.
Strictly from a user's perspective my fear is that if 1% leaves (real life issues do happen) or gets less active the 6d port will be dead in the water if the tl/ml codebases have forked too far alex or the main ml devs might not be willing or able to merge it back. And without intimate ml knowledge it isn't possible to just resolve conflicts when merging the main ml repo into the 6d tl fork.
1% we love you!!! I am thankful for all you have done for 6d users!!!
I think its one hell of a pull request to merge the whole thing at once. It would be conflict city. I don't want to lose my notes either but they should definitely not go into main.
Quote from: 1% on October 23, 2013, 11:59:20 PM
I think its one hell of a pull request to merge the whole thing at once.
I'm 100% sure alex is very open to attempt to solve it, I don't know how large the code differences really are by now - but as you and alex are very intimate with the code it shouldn't be impossible to merge it either via mercurial or as a diff, ml is large, but not that large. But it's great the discussion moves from the "will we do it" to the public "how will we do it" stage :-)
Quote from: 1% on October 23, 2013, 11:59:20 PM
It would be conflict city.
Indeed, this shows that the merge attempt is beyond overdue. However you do it, +1 for doing it now before it gets even worse :-o ...
... and I really have to mention again that you're really doing great work and my 6d would be near useless without you, so please take this post as a user's wish for continued 6d ml success.
Quote from: 1% on October 23, 2013, 11:59:20 PM
I don't want to lose my notes either but they should definitely not go into main.
That's notes as in code annotations with // and /* */ ? Hmmmm, isn't there some editor or similar approach that can manage annotations separate from the main code file? I doubt it you're the only coder in history facing this problem, so there have to be solutions for this.
Yep, but all solutions for moving notes don't back them up to the repo :)
A1ex wrote so much of ML, I read through all the ML code to figure things out and incorporate them, sometimes by hand. In fact all code I can find to see if I can squeeze out one more feature. Its just a lot of work to merge it all, I guess one upside is that now he only has to do 5DII and 5DIII and the rest can be ported down to other cameras.
Also removing the movie remap on 600D was the reason to fork in the first place. It has worked pretty well there and its probably the only cam that both needs its and has it working without major hiccups. 60D from where I looked at the FW is a mix of 600D + 7D so kinda old and new. I can see how it was problems for cams < 600D hence its never enabled there.
But yea, not going anywhere any time soon. ML is basically needed to get anything constructive out of these cameras.
Quote from: 1% on October 24, 2013, 02:22:06 AM
Its just a lot of work to merge it all
Well, on reflection I all should probably comment on this that I hope TL & ML don't stay separate forks forever, but of course my main interest is to get a working 100% 6d port and getting all ml features merged into this, wherever the repo actually is :-)
I hope very much that some individuals with the willingness and resources tackle the merge of TL mutually with 1%, because fragmentation over time is a bad, bad thing. Especially with projects like these.
Yeah, it would be really nice if the GOP/Slice controls for H.264 would be ported to ML. If not, I'm happy with TL, for me it's working very well, no issues of any kind.
For GH2 for example, there are a ton of hacks, all different, who needs a certain thing, can install a certain hack, TL and ML are the same way.
Just a friendly reminder: it's been almost two months and no sign of improvement.
Let's try something else:
1) During the next two months I will not commit anything to ML code base other than accepting patches and maybe tweaking them.
2) I will move any ML ports that have not been maintained in the last few months into a "unmaintained" directory (that is, no more nightly builds for them). Anybody is free to jump in and maintain these ports. When the situation improves, I'll move them back.
3) I will create a separate forum section for third party modifications (TL, a.d.'s 5D2 builds, g3gg0's experiments, Marsu42's tweaks and so on). Basically, anything that lives on its own and is not (yet) maintained by the ML team. Eventually, all the good stuff from these should get merged into mainline, and until then, they should have a chance of getting tested by the adventurous users without competing for attention with the main ML nightly builds.
Discuss.
Quote from: 1% on October 22, 2013, 01:14:30 AM
... I can work on audio support for EOSM or I can learn pull requests, make a new repo, add everything to merge back and then fix the conflicts when merging it back into my source, etc.
That is a no brainer. Audio support for the EOSM. :)
I appreciate everything you, Alex and all the other devs do for the rest of us and I hope this all gets worked out somehow. No matter what Canon camera anyone owns, we all have a common goal and hopefully success.
I hope "everyone" understands the smiley face and the sincerity of my words after that.
Gary
Quote from: gary2013 on December 11, 2013, 08:15:46 AM
That is a no brainer. Audio support for the EOSM.
In other words: screw the project health, I just want features!
You missed to notice the irony, a1ex ...
If you asked me, I would prefer a consolidiation/merge/unification phase beginning asap.
Q: How can we (as in "non-developers and not going to change status") support you (as in "devs") to make this happen?
Ciao
Walter
Quote from: a1ex on December 11, 2013, 07:16:50 AM
Just a friendly reminder: it's been almost two months and no sign of improvement.
You're talking about TL, or about ML patches in general?
Quote from: a1ex on December 11, 2013, 07:16:50 AM
1) During the next two months I will not commit anything to ML code base other than accepting patches and maybe tweaking them.
Certainly worth a try, every idea that generates more user patches is a good one - I admit it's very easy to lean back and watch you do it, you're just to good at at and know everything about the code - but it is bound to generate the current situations with you and g3gg0 doing all the work.
Quote from: a1ex on December 11, 2013, 07:16:50 AM
2) I will move any ML ports that have not been maintained in the last few months into a "unmaintained" directory (that is, no more nightly builds for them). Anybody is free to jump in and maintain these ports. When the situation improves, I'll move them back.
+1, worth a try, certainly makes sense for 6d/... included in TL. As I stated in the 70d thread, unmaintained builds will damage the public ml image, and anyone who is able to can still compile them instead of using pre-compiled nightlies if I understand your idea correctly.
Quote from: a1ex on December 11, 2013, 07:16:50 AM
3) I will create a separate forum section for third party modifications (TL, a.d.'s 5D2 builds, g3gg0's experiments, Marsu42's tweaks and so on). Basically, anything that lives on its own and is not (yet) maintained by the ML team. Eventually, all the good stuff from these should get merged into mainline, and until then, they should have a chance of getting tested by the adventurous users without competing for attention with the main ML nightly builds.
What I think would be important and what I already suggested in another thread is to move new modules (including some small core additions if required for them to access model-specific consts/values) or even features to the nightly builds much sooner, but label them as "staging" or "experimental" just like with Linux... then add an option to the module loader or ml options to (dis)allow these non-tested modules or parts as this might improve the "features vs. stability" situation.
As for my part in the future, I'll have been using my new 6d for the last months and only hacking ML as really necessary for it to work on the 6d, and I will be rather busy for at least the next half year - so I don't see me bringing my currently personal-only "link ml/canon settings to dial" and "hotkeys" module to a public stage I'm afraid to say as coding takes me much longer than an experienced dev :-\
Except for the occasional comment in Bitbucket or here I should manage to renew the auto_iso module repo, I've been using the new metering code for the last months (changed from alex' Av/Tv, and new with M+EC) and I can say it works fine... and I always see requests on Canon Rumors from people wondering why their camera refuse to do auto iso with flash on.
Edit: Thinking about it, the most important thing seasoned devs can do is the backend stuff (module system, menu system (submenus! :-)) and code advice and review ... leaving more of of the feature-crazyness and model maintaining to newer users/contributors, because I admit something is easily requested, but if you really want it you will probably sit down and code it yourself :-o
Quote from: Marsu42 on December 11, 2013, 09:04:49 AM
You're talking about TL, or about ML patches in general?
About TL. Of course, some of my decisions will also touch the other unmaintained ports (since I'm not trying to point fingers at TL or anyone else, but I'd like to fix all these things in the same way).
Quote
What I think would be important and what I already suggested in another thread is to move new modules (including some small core additions if required for them to access model-specific consts/values) or even features to the nightly builds much sooner, but label them as "staging" or "experimental" just like with Linux...
I'll try to address this and combine it with my "feedback matrix" idea. Hopefully it will turn out well.
Quote
Edit: Thinking about it, the most important thing seasoned devs can do is the backend stuff (module system, menu system (submenus! :-)) and code advice and review ...
Indeed, and move towards a clean and documented API. I have a lot of ideas here, but I need to sit down and get a clear picture first.
Quote from: Walter Schulz on December 11, 2013, 08:59:44 AM
You missed to notice the irony, a1ex ...
I'm not a native English speaker, so I'm not sure what I missed.
Quote
Q: How can we (as in "non-developers and not going to change status") support you (as in "devs") to make this happen?
Not sure, maybe by encouraging it and not letting ML rot for months? I'm not going to solve this alone, and if things won't get better, I'll extend my inactivity period until people will realize this should be a community effort and not 1-2 people doing most of the work.
Quote from: a1ex on December 11, 2013, 08:32:06 AM
In other words: screw the project health, I just want features!
Everyone knows what I meant by seeing the smiley face. No English required. Of course I meant it as a joke. ::)
The thing is, A1ex, users like me can just test features and give you feedbacks, contribute with donations and not much more. I am too uneducated to start coding now and when i tried it was always because you explained me what to do (and it was never something complicated, just basic things) .
I have been a lurker for months now, i moved country and i am having a hard time even taking a couple of decent pictures. Still, i like the project and i have been a supporter of it since the first 50D port started to take shape.
So, again, as i said and as you know, you developers can agree on what things to do and how to do them but what is exactly that you expect from users?
Most of the users haven't understood the principle behind ML and they will hardly do in the future.
Myself and others are surely willing to hear any suggestion that you can give us, provided that is something that we can actually do.
Quote from: a1ex on December 11, 2013, 07:16:50 AM
3) I will create a separate forum section for third party modifications (TL, a.d.'s 5D2 builds, g3gg0's experiments, Marsu42's tweaks and so on). Basically, anything that lives on its own and is not (yet) maintained by the ML team. Eventually, all the good stuff from these should get merged into mainline, and until then, they should have a chance of getting tested by the adventurous users without competing for attention with the main ML nightly builds.
This is a great idea. It will help to keep discussion specific to these features separate.
2 and 3 mostly done. Will leave the rest of cleanup to the moderators.
Quote from: Walter Schulz on December 11, 2013, 08:59:44 AM
Q: How can we (as in "non-developers and not going to change status") support you (as in "devs") to make this happen?
This is not directed at you Walter, but at the community in general (you are one of the most helpful folks on this forum, keep up the good work ;) ):
Test, as in
really test. As if you were QC working for a software company. Don't just use ML in normal situations and submit problems when things go wrong (and don't just do another ML RAW vs. H264 test, there are more than plenty of those out there). Try to break a feature. Think of as many possible scenarios as you can. Throw everything you can think of at a feature to break it. Try every value of every setting. Try it in extremely unusual scenes or lighting. Write down your results like a scientist doing an expirement. Then share your results, even if nothing went wrong. It's also helpful for devs to know when something actually works. Test for the sake of testing, with specific intention, not for the sake of making your 'budget short film'.
If you notice a problem, figure out how to repeat it and try to isolate it to a specific build (or if you can compile, isolate it to a specific changeset). Looking at the change log on bitbucket is very helpful in doing this (look for commits related to the feature and check those builds first). This requires no coding ability at all, and can save devs a lot of time, b/c they will know exactly where in the code the problem lies. Even I can figure out bugs that would normally be way above my skill level if you tell me exactly the changeset that is causing it, in fact most of the very few patches I've submitted, I've found like this.
This has been reiterated by a1ex time after time, I'm sure he's tired of saying it. It's clear that very few people actually end up reading the 'how to report bugs' links he posts.
A good example of the lack of this being done is the 600D 'overheating' issue. Countless people have reported this issue. Not one has done anything to help resolve it. Simple scientific-like experimentation can easily help resolve the exact source of the problem without any coding skill at all (e.g. monitor the temp of the camera overtime, both the display temp and with an actual thermometer, try different builds, use the stable build as a control group, etc.). The nightly builds are for testing, nobody actually seems to be doing this, or their idea of 'testing' is: 'use in a production enviroment and complain when something goes wrong'
I think the general response to the troll question "when a stable build?" should not be a link to the faq, but something like: "As soon as
you provide us with clear and concise testing and bug reports for all ML features"
Quote from: a1ex on December 11, 2013, 09:56:03 AM
Indeed, and move towards a clean and documented API.
+1
My main stumbling block in terms of trying to help out with development is not lack of coding skill, but rather lack of specific knowlege of the ML code base. I have enough spare time to do some basic coding, fix minor bugs, and add little features here and there, what I don't have time to do is review and understand hundreds of thousands of lines of rather convoluted (understandable, it is a hack after all, and an embedded/RT system) and poorly documented (no excuse for this) code. For example the multilevel submenu thing, I feel confident that I have the coding skill required to implement it myself, the problem is I don't understand menu.c well enough (which I've tried, b/c I'd really like to get it done for my module, I usually just stare at menu.c for 30mins and then give up). At a glance, I have no idea what functions do, when they're called, etc. Sure I could probably eventually figure it out if I took the time, but I don't have that kind of time. A few simple comments before each function (even private functions) would be extremely helpful (like description, parameters, outputs, prereq's, where it's called from, task, etc., and put these comments in a standard format for automatic docs generation, e.g. doxygen).
It sounds to me like, at least to some extent, 1% has done this and that's what he is refering to when he talks about his "notes". If that is the case, then I actually would welcome 1%'s "notes" into the main repo (perhaps they just need to be cleaned up somewhat?). IMO, the more comments and documentation that's in the code, the better it is, as long as it's not 'incorrect' information or in 1%'s own language/shorthand. I haven't reviewed the TL fork code b/c I don't use any of those cams, so IDK, maybe I should. If I manage to get my hands on a 6D (which is very tempting now that it's only $1,500), I am more than willing to help merge the TL fork.
I've been trying to clean up my own repo and learn as much as I can about mercurial. I've learned some good practices and plan on sharing them (in a new thread). Following these tips I think will help everyone who maintains a fork manage their own personal tweaks as well as making it easy to get stuff merged back into the mainline.
Quote from: dmilligan on December 11, 2013, 04:16:32 PM
Test, as in really test. As if you were QC working for a software company. Don't just use ML in normal situations and submit problems when things go wrong (and don't just do another ML RAW vs. H264 test, there are more than plenty of those out there). Try to break a feature. Think of as many possible scenarios as you can. Throw everything you can think of at a feature to break it. Try every value of every setting. Try it in extremely unusual scenes or lighting. Write down your results like a scientist doing an expirement. Then share your results, even if nothing went wrong. It's also helpful for devs to know when something actually works. Test for the sake of testing, with specific intention, not for the sake of making your 'budget short film'.
If you notice a problem, figure out how to repeat it and try to isolate it to a specific build (or if you can compile, isolate it to a specific changeset). Looking at the change log on bitbucket is very helpful in doing this (look for commits related to the feature and check those builds first). This requires no coding ability at all, and can save devs a lot of time, b/c they will know exactly where in the code the problem lies. Even I can figure out bugs that would normally be way above my skill level if you tell me exactly the changeset that is causing it, in fact most of the very few patches I've submitted, I've found like this.
This has been reiterated by a1ex time after time, I'm sure he's tired of saying it. It's clear that very few people actually end up reading the 'how to report bugs' links he posts.
A good example of the lack of this being done is the 600D 'overheating' issue. Countless people have reported this issue. Not one has done anything to help resolve it. Simple scientific-like experimentation can easily help resolve the exact source of the problem without any coding skill at all (e.g. monitor the temp of the camera overtime, both the display temp and with an actual thermometer, try different builds, use the stable build as a control group, etc.). The nightly builds are for testing, nobody actually seems to be doing this, or their idea of 'testing' is: 'use in a production enviroment and complain when something goes wrong'
+1, this should be sticky.
Quote
I think the general response to the troll question "when a stable build?" should not be a link to the faq, but something like: "As soon as you provide us with clear and concise testing and bug reports for all ML features"
Agree, updated the FAQ (http://wiki.magiclantern.fm/faq?&#when_will_you_release_the_next_version).
Quote from: dmilligan on December 11, 2013, 04:16:32 PM
My main stumbling block in terms of trying to help out with development is not lack of coding skill, but rather lack of specific knowlege of the ML code base. I have enough spare time to do some basic coding, fix minor bugs, and add little features here and there, what I don't have time to do is review and understand hundreds of thousands of lines of rather convoluted (understandable, it is a hack after all, and an embedded/RT system) and poorly documented (no excuse for this) code.
My main stumbling block is skill, but a lack of documentation doesn't help.
To be fair, the new code base that I look at comes with excellent descriptions, such as,
Quoteelse
{
/* image is overexposed */
/* and we don't know how much to go back in order to fix the overexposure */
/* from the previous shot, we know where the highlights were, compared to some lower percentiles */
/* let's assume this didn't change; meter at those percentiles and extrapolate the result */
int num = 0;
float sum = 0;
float min = 100000;
float max = -100000;
for (int k = 0; k < COUNT(percentiles)-1; k++)
{
if (diff_from_lower_percentiles[k] > 0)
{
float lower_ev = raw_to_ev(raw_values[k+1]);
if (lower_ev < -0.1)
{
/* if the scene didn't change, we should be spot on */
/* don't update the correction hints, since we don't know exactly where we are */
ev = lower_ev + diff_from_lower_percentiles[k];
/* we need to get a stronger correction than with the overexposed metering */
/* otherwise, the scene probably changed */
if (target - ev < correction)
{
float corr = target - ev;
min = MIN(min, corr);
max = MAX(max, corr);
/* first estimations are more reliable, weight them a bit more */
sum += corr * (COUNT(percentiles) - k);
num += (COUNT(percentiles) - k);
//~ msleep(500);
//~ bmp_printf(FONT_MED, 0, 100+20*k, "overexposure fix: k=%d diff=%d ev=%d corr=%d\n", k, (int)(diff_from_lower_percentiles[k] * 100), (int)(ev * 100), (int)(corr * 100));
}
}
}
}
/* use the average value for correction */
correction = sum / num;
if (num < 3 || max - correction > 1 || correction - min > 1 || correction > -1)
{
/* scene changed? measurements from previous shot not confirmed or vary too much?
*
* we'll use a heuristic: for 1% of blown out image, go back 1EV, for 100% go back 13EV */
float overexposed = raw_hist_get_overexposure_percentage(GRAY_PROJECTION_AVERAGE_RGB | GRAY_PROJECTION_DARK_ONLY) / 100.0;
//~ bmp_printf(FONT_MED, 0, 80, "overexposure area: %d/100%%\n", (int)(overexposed * 100));
//~ bmp_printf(FONT_MED, 0, 120, "fail info: (%d %d %d %d) (%d %d %d)", raw_values[0], raw_values[1], raw_values[2], raw_values[3], (int)(diff_from_lower_percentiles[0] * 100), (int)(diff_from_lower_percentiles[1] * 100), (int)(diff_from_lower_percentiles[2] * 100));
float corr = - log2f(1 + overexposed*overexposed);
/* with dual ISO, the cost of underexposing is not that high, so prefer it to improve convergence */
if (dual_iso)
corr *= 3;
correction = MIN(correction, corr);
/* we can't really meter more than 10 EV */
correction = MAX(correction, -10);
}
I'm still to stupid to understand how the code is working, but with descriptions like so, I'm sure it wouldn't take much strain on the grey matter to work it out.
Quote from: dmilligan on December 11, 2013, 04:16:32 PM
what I don't have time to do is review and understand hundreds of thousands of lines of rather convoluted (understandable, it is a hack after all, and an embedded/RT system) and poorly documented (no excuse for this) code.
There are extremely hackish passages with large blocks of code commented out for one reason or another - but I don't find the core code I looked at and use (including prop-stuff and lens.c, focus.c, gui.c, menu.c, shoot.c, hdr.c, ...) to be that horrible - the main issue that takes time is to understand how ML uses the Canon DryOS functions and props at all.
But I certainly feel that the devs adding core code could be *much* more verbose when commenting what the core code its doing where and why, I know that after coding for ml for a while these things seem trivial and self-explanatory, but they aren't for new contributors.
Quote from: dmilligan on December 11, 2013, 04:16:32 PMI think the general response to the troll question "when a stable build?" should not be a link to the faq, but something like: "As soon as you provide us with clear and concise testing and bug reports for all ML features"
Thanks for your good wrap up and guide for users to improve ml w/o doing coding, but as to the "stable" part in all honesty ML is a currently moving target, so asking "When will you stop adding features and stabilize what it is right now" is not necessarily trollish. Yes, the devs need precise bug reports, but also yes, there is no timeframe or milestone-schedule for the widely used 6d/5d3 at all so I can understand if people new to ML are wondering.
Edit: I'm far too removed from the dev process, but a Linux-like model with a "merge window" and a
bugfix phase sounds reasonable to me, contributors would know about when they should have additions ready, and users would have a rough idea about when the next "as stable as it gets" release is to be expected.
Quote from: Audionut on December 11, 2013, 05:57:04 PM
To be fair, the new code base that I look at comes with excellent descriptions, such as,
Agreed, that is a great example of how code
should look. One of the reasons I think modules are a lot easier to develop is that there is a lot of well documented example code like that to go by. If the entire code base was that well documented, I wouldn't be complaining about it ;)
The main issue is that the module system is not yet fully mature, i.e. not all of the things needed by modules have been implemented. So that means if I need to do something not currently possible with a module, I have to dig into the core and implement it myself (or beg a1ex and wait).
Quote from: Marsu42 on December 11, 2013, 06:06:04 PM
"When will you stop adding features and stabilize what it is right now" is not necessarily trollish.
Yes, but that's not what anybody actually ever really asks. I too can understand why ppl would like to know this. I think that if ppl actually did more helpful testing and bug reports, we would end up a lot closer to a stable version, devs would be more likely, and have more time to actually do a feature freeze and a stable release. What incentive does a1ex have to do that when nobody even tries a feature for months (http://www.magiclantern.fm/forum/index.php?topic=8650).
Quote from: dmilligan on December 11, 2013, 06:44:33 PM
The main issue is that the module system is not yet fully mature, i.e. not all of the things needed by modules have been implemented. So that means if I need to do something not currently possible with a module, I have to dig into the core and implement it myself (or beg a1ex and wait).
I don't see the the need the a hard division between "module" and "core" devs, if you need some core patches, well, just go ahead as I know you did in lens.c - I know it's required for model-specific things, and it's most likely stay this way.
Btw, ot: What I currently want for modules (hint, hint, hint :-)) are submenus and array config variables, this has to be done by someone capable who knows the backend.
Quote from: dmilligan on December 11, 2013, 06:44:33 PM
nobody even tries a feature for months
But in the cases I remember being removed, it were rather obscure features, and the reason for nobody reporting the bug is that they simply weren't used. ML might be "over-featured" in some areas, though I'm against preemptive removal since you never know who uses what.
If ETTR qualifies as "obscure feature", no comment...
I think people interested in porting TL to ML should donate to 1% and when enough money are raised so that this project is properly funded, 1% will spend the required time to do the coding. You can not expect the man to spend many, many hours of he's free time just so that we all be happy, when he is not getting payed. That's how this problem should be presented and understood.
1% should have a link for donations to this project alone, and when enough money are raised so that he's effort, time and energy spent are properly funded, then it will be done. Raising a few thousands dollars for porting TL is not such a big challenge for this community.
Quote from: Marsu42 on December 11, 2013, 07:10:06 PM
But in the cases I remember being removed
Quote from: a1ex on December 11, 2013, 07:25:16 PM
If ETTR qualifies as "obscure feature", no comment...
I was thinking of some features you recently removed like "auto burst pic quality"... as for ettr being broken (was it?) and nobody noticing this might prove that people do not track every nightly but only update once so often?
Yes, ETTR was removed for a while from 600D (see here (https://bitbucket.org/hudson/magic-lantern/commits/4380d31e6230c4e5e5d367d95c9ebd0d5591aa97) and here (https://bitbucket.org/hudson/magic-lantern/commits/b9077b82ed761801dccc8361e7aa4fa3e32ebe86)).
Actually, it's not ETTR that was removed, but all the raw photo overlays. But since ETTR depends on them, the module no longer linked correctly and got disabled too.
Quote from: Critical Point on December 11, 2013, 07:57:51 PM
I think people interested in porting TL to ML should donate to 1% and when enough money are raised so that this project is properly funded, 1% will spend the required time to do the coding. You can not expect the man to spend many, many hours of he's free time just so that we all be happy, when he is not getting payed. That's how this problem should be presented and understood.
1% should have a link for donations to this project alone, and when enough money are raised so that he's effort, time and energy spent are properly funded, then it will be done. Raising a few thousands dollars for porting TL is not such a big challenge for this community.
So then you would have 1 dev that's getting paid for working on ML, and the other dev's that are working for free doing the same job.
that wont work.
Quote from: darkstarr on December 11, 2013, 10:15:50 PM
So then you would have 1 dev that's getting paid for working on ML, and the other dev's that are working for free doing the same job.
It wouldn't work anyway, 1% doesn't have the same level of coding and ml expertise as alex (I'm sure 1% does agree). 1% isn't the 6d "maintainer" but is mainly interested to get a "quick and dirty" working 6d distro and happens to share his repo with the rest of us. I'm very grateful for that because I couldn't afford a 5d3, but the current TL/ML problems lie elsewhere than people not getting paid.
Quote from: Marsu42 on December 14, 2013, 04:42:08 PM
I'm very grateful for that because I couldn't afford a 5d3
Dang ... I wish I'd researched the status of ML + 6d + TL before purchase. I'm half considering returning the much more cost effective 6d (YMMV) and applying the cost to a 5d3.
But I'm near the threshold of WOW (wrath of wife).
For me, the 6d without a reliable, valid, trustworthy ML is almost unusable. (and I have yet to tackle video, which I understand is benefitted by ML even more than still photographers such as myself)
I'd love for ML to be Rock Solid, mature software, but it ain't. That's not a complaint ... I've come to accept the development speed of "rolling releases".
I am Very Hesitant to "try" TL, and my impression is that a1ex (and 1% ?) would concur that TL has some (many?) unsafe coding practices.
So now my 6d is lacking ...
- auto-tune micro-focus adjustment
- ETTR
- Dual-ISO
- Intervalometer
- Extended Bulb
- etc.
BW: I'm getting ready for a "road trip" to western Kansas for full-moon panos of grain elevators. I'm considering leaving the 6d behind, and "making do" with my 5d2. The 6d without a non-bricking ML just is lacking too much. Bummer.
I didn't realize how much I've come to rely on ML. (and THANKS to the devs!!!!!)
Quote from: Marsu42 on December 11, 2013, 08:43:20 PM
people do not track every nightly but only update once so often?
That's very much my practice. At first, I was so amazed at what ML provided, that I was inhaling everything I could find. "drinking the Kool-Aid"?
But real life intrudes. Got commitments to get pictures taken and p.p.'ed and printed. Oh ... that?
And I'm a non-professional, retired hobby'ist.
On average, I probably download and install ML every other week, or so. Then grapple with What's New and What's Changed.
"May you live in interesting times" ... to be a Canon photographer.
Quote from: l_d_allan on December 14, 2013, 06:37:38 PM
I didn't realize how much I've come to rely on ML. (and THANKS to the devs!!!!!)
Same with me, that's why in spite of alex' warning I added TL to my 6d after trying to shoot one day with the crippled auto iso (only iso 400 with flash) and w/o the other usability enhancements of ML.
Many people are using TL, and none seems to have bricked the camera yet as 1% always points out ... the problems is rather that the more the peer-reviewed ML codebase and 1%'s TL code differ, the more likely it'll be to break sooner or later due to the code degenerating... but it's not at that stage yet.
Quote from: Marsu42 on December 11, 2013, 06:06:04 PM
But I certainly feel that the devs adding core code could be *much* more verbose when commenting what the core code its doing where and why,
Not sure I completely agree with you. Comments can get "stale" very quickly, especially with very dynamic code like ML. Bad comments can be worse than no comments.
My experience was that some keys to maintainable code with several people involved:
- Meaningful names for variables, parameters, functions, subroutines, modules, etc. I started programming with Fortran-66 in 1968 with 7 or 8 letter names. Those days are loooooong gone.
- Little or no "cascaded logic" which hurts debugging ... if (isOk(true, 42, -42(isLate(42, isObfuscated(true, 42))))) {
- Code so that the debugger is the most helpful ... simple statement per line so you can see as much as possible about what's happening (but I had a tendency to be over dependent on debuggers ... back in the day)
- Let the compiler do the optimizing
- KISFTSSM ... Keep it simple for the simple/stupid maintainers
- D Knuth: premature optimization is the root of much software evil
Note that I have never glanced at ML code. I suspect it is far about average wrt code quality, but that may be "wishful thinking" on my part.
There's no excuse for not having a brief description including parameter descriptions and output of every function.
Quote from: l_d_allan on December 14, 2013, 09:56:03 PM
Not sure I completely agree with you. Comments can get "stale" very quickly, especially with very dynamic code like ML. Bad comments can be worse than no comments.
That is a completely invalid point IMO. The potential for bad documentation is no reason to not document at all. Large swaths of the core code changes very little anyway. Some particular functions that I wonder what they do were written years ago and not changed since.
Quote from: l_d_allan on December 14, 2013, 09:56:03 PM
Meaningful names for variables, parameters, functions, subroutines, modules, etc. I started programming with Fortran-66 in 1968 with 7 or 8 letter names. Those days are loooooong gone.
No amount of good variable naming can make blocks of assembly easy to understand.
Quote from: l_d_allan on December 14, 2013, 09:56:03 PM
Code so that the debugger is the most helpful ... simple statement per line so you can see as much as possible about what's happening (but I had a tendency to be over dependent on debuggers ... back in the day)
If we had a full emulator working this might be a valid point. But there's just no way to 'debug' much of the ML code (e.g. explain to me how to run a debugger on a boot-loader running as a hack on an embedded system), again this is 'real time' stuff, debugging is quite a different beast than what you are probably used to with traditional programming.
Quote from: l_d_allan on December 14, 2013, 09:56:03 PM
Let the compiler do the optimizing
That's not always practical when you've got extremely limited resources and you're in a 'real time' scenario. Sometimes you have to specifically prohibit the compiler from optimizing things (i.e. values stored in hardware registers, etc.).
Maybe you should actually have a look at the code base before you render opinions about it. This is a hack that runs in a real-time environment and calls reverse engineered code. This is very very different from traditional software programming.
Quote from: a1ex on December 11, 2013, 07:16:50 AM
Discuss.
Just a perspective from a doc filmmakers view:
I think it sounds like a great idea to consolidate and simplify! From a typical end user point of view, magic lantern is still a hard to get into because the potential is so great, but the technical barriers to start using it is challenging for many people (just my perspective ofcourse). Not necessarily the steps involved to actually use it, but the sum of the uncertainty of how stable it is, what build to use, and then the steps involved to "install" it. And that keeps it from getting into the hands of many users (who could contribute to the project - mostly feedback and donations ofcourse).
I think the project would attract a larger userbase if it was possible to make the ML releases into something that separates the more stable functions and the "in development" functions:
Maybe I'm way off here, but what if it was possible to make e.g. a 5D3 build (or whatever cam) with core functionality like focus peaking and zebras (maybe some other stable functions) that is ok to indicate as "more stable". ALL other functions like RAW etc can be marked as experimental/alpha (e.g. clearly separated in the menus), so people know when they move from the more stable to the other functionality. In that scenario, maybe it would be possible to brand it a beta release? (using myself as a case, I would definetly jump onboard using ML, seeing a beta tag, just knowing that focus peaking and zebras were beta-stable)
I'm just thinking like this because I know from my own experience that I'm hesitant to use ML at all, with so many builds, different statements about what build to use, if the released alphas (which are quite old now) is safer to use etc. etc.
I'm sorry if these ideas are of limited value, just some thoughts from an end user. Also a huge thanks to all contributors in the project. I'm speechless of what's beeing achieved, and I really hope it will reach an even bigger audience soon :)
Quote from: evero on December 20, 2013, 12:06:35 PM
I think the project would attract a larger userbase if it was possible to make the ML releases into something that separates the more stable functions and the "in development" functions:
+1
+1
This requires thorough feedback from you about what works well and what not. See the posts from dmilligan from this thread.
Quote from: a1ex on December 20, 2013, 08:27:43 PM
This requires thorough feedback from you about what works well and what not. See the posts from dmilligan from this thread.
I totally understand you and dmilligan. And this is the classic egg/chicken problem, because the reality for me (and I guess a lot others) is that I don't dare testing ML at this point. It's not clear which release (is safe enough) to use for the 5d3, and using nightly builds sounds to dangerous for a guy who's nervous to harm the camera.
So even though I want to help, how can I give any testing/feedback, when I don't even dare to install it... Because of this I'm probably not in the target group for ML for now (and I totally understand that). But knowing the positive vibe around the ML project, and the already long and impressing track record, my idea was this approach with labeling a few basic features "beta reliable" in a release. By doing that a new user base would dear to engage in the project, providing feedback etc. (but again, this is just from a user perspective - from what I now read, I understand this is easier said then done ofcourse)
Hm.... People are talking about this API/documentation thing
I'm a business analyst (and getting my feet wet as a QA analyst for experience sake) in IT for a major retail chain. Pretty much my entire job is documentation. While I'm not familiar with actual coding (yet!), I do have a good amount of experience under me in documentation of features, specs, etc. Wondering if maybe I can at least hash out the overall doc and get it flowing, and then some if the devs can provide all the detail where necessary?
Maybe it'd make sense to start with 2.3? Then for every stable release I can have the changelog in the doc, documenting everything?
I don't know; just something I'm throwing out there. If no one still has 'formerly' documented all the bits of code, if be more than willing to help :) might even get started right now. (Unless any objections?)
http://www.magiclantern.fm/forum/index.php?topic=6594
http://www.magiclantern.fm/forum/index.php?topic=3820
I prefer having it edited on the repository, because from this format I can generate PDF, HTML, wiki and in-camera BMP versions.
Quote from: a1ex on December 22, 2013, 06:57:43 PM
http://www.magiclantern.fm/forum/index.php?topic=6594
http://www.magiclantern.fm/forum/index.php?topic=3820
I prefer having it edited on the repository, because from this format I can generate PDF, HTML, wiki and in-camera BMP versions.
Ahhh, nice I see. Cool!
These are my personal opinions, nothing more.
First, for end-users: No firmware modification can be guaranteed to be safe. That said, ML is probably as safe as it can be. TL isn't as aggressive in trying to be safe. That's not to say it will irreversibly brick your camera; hasn't happened (reported) to anyone yet, but past performance is no guarantee of future results.
You may have read about the possibility of a merge. Do understand, this isn't a simple process. If two people start with the same book, and they proceed to rewrite certain sentences, paragraphs, add chapters, tables, diagrams, and so forth, the books will look similar, but you can't just swap pages between them. That's what the merge would be; trying to bring some of the pages from the second book, into the current revision of the first. And making sure there's no missing references, e.g. "see explanation in the next chapter" wont work, if that chapter isn't also merged.
Deciding whether to run ML, TL, or anything else, is about risk acceptance; decide what your level of risk tolerance is, and whether the features are worth it to you. If you run either, you can help yourself (and everyone else) by reading the guidelines for bug reports. Bug reports are like a meal; fresh food, good flavor, adequate portions, and some dessert will help keep them going, and remind them you care.
Second, to the devs. Open source projects are typically a labor of love, and heartache is inevitable. Watching a project grow beyond your personal use, especially over significant time, expansion, and contribution of others, it carries a lot of rewards, and frustrations. Hindsight is a cruel mistress, and everyone's a critic. Reading this thread, I found significant insights into some of the frustrations that are at play; the statement of them felt particularly candid, and I believe that's an important and useful thing.
As developers, you're giving time, but it's backed by critical knowledge. We, as the community, can't afford to lose you; it's unlikely you'd ever be replaced. Most users want to help, but are untrained in how to do so; often, they'd rather be quiet, than be unhelpful. Of course, there's those who are unaware or ignorant of the realities of a project of this scale. The single most important thing I can suggest, is an open and ongoing dialog; candid and forthcoming. And remember, many more would try to help, if they knew their efforts wouldn't be counter-productive.
tl;dr: Thank you, good luck, stay strong, and don't be shy; communication is crucial to success of any project that has more than one person involved, be it collaborators or users.
Well said ;)
Quote from: dmilligan on December 11, 2013, 04:16:32 PM
Test, as in really test. As if you were QC working for a software company. Don't just use ML in normal situations and submit problems when things go wrong (and don't just do another ML RAW vs. H264 test, there are more than plenty of those out there). Try to break a feature. Think of as many possible scenarios as you can. Throw everything you can think of at a feature to break it. Try every value of every setting. Try it in extremely unusual scenes or lighting. Write down your results like a scientist doing an expirement. Then share your results, even if nothing went wrong. It's also helpful for devs to know when something actually works. Test for the sake of testing, with specific intention, not for the sake of making your 'budget short film'.
What could be better for the Devs and easier for people to contribute to is producing test scripts. A simple and clearly documented set of steps with the results easily quantifiable.
Something like this (a very basic example).
- Enable the raw.rec module [pass|fail]
- Enter the Movie menu and then enable Raw Video (when in Live View mode - pick resolution suitable for you card) [pass|fail]
- Render video and inspect it to check for bad frames [pass|fail]
While you sometimes can't go wrong with free-form testing, a more structured approach means you can measure the test coverage and split this activity up so many people can contribute and still have some control.
As new features are added, or defects found, new tests should be created - essentially moving to a Test Driven Development model.
As the website is PHP based there will be off the shelf applications that could be deployed to the site to automate this easily, or just use Goggle Docs and track it in a spreadsheet.