Tips for helping the development team

[Audionut]

Hopefully this thread will turn into a knowledge base of how us mere mortals can help the development team in ways other then providing code.

Test, as in really test. As if you were QC working for a software company. Don't just use ML in normal situations and submit problems when things go wrong (and don't just do another ML RAW vs. H264 test, there are more than plenty of those out there). Try to break a feature. Think of as many possible scenarios as you can. Throw everything you can think of at a feature to break it. Try every value of every setting. Try it in extremely unusual scenes or lighting. Write down your results like a scientist doing an expirement. Then share your results, even if nothing went wrong. It's also helpful for devs to know when something actually works. Test for the sake of testing, with specific intention, not for the sake of making your 'budget short film'.

If you notice a problem, figure out how to repeat it and try to isolate it to a specific build (or if you can compile, isolate it to a specific changeset). Looking at the change log on bitbucket is very helpful in doing this (look for commits related to the feature and check those builds first). This requires no coding ability at all, and can save devs a lot of time, b/c they will know exactly where in the code the problem lies. Even I can figure out bugs that would normally be way above my skill level if you tell me exactly the changeset that is causing it, in fact most of the very few patches I've submitted, I've found like this.

This has been reiterated by a1ex time after time, I'm sure he's tired of saying it. It's clear that very few people actually end up reading the 'how to report bugs' links he posts.

A good example of the lack of this being done is the 600D 'overheating' issue. Countless people have reported this issue. Not one has done anything to help resolve it. Simple scientific-like experimentation can easily help resolve the exact source of the problem without any coding skill at all (e.g. monitor the temp of the camera overtime, both the display temp and with an actual thermometer, try different builds, use the stable build as a control group, etc.). The nightly builds are for testing, nobody actually seems to be doing this, or their idea of 'testing' is: 'use in a production enviroment and complain when something goes wrong'

http://www.chiark.greenend.org.uk/~sgtatham/bugs.html

Anybody who has written software for public use will probably have received at least one bad bug report. Reports that say nothing ("It doesn't work!"); reports that make no sense; reports that don't give enough information; reports that give wrong information. Reports of problems that turn out to be user error; reports of problems that turn out to be the fault of somebody else's program; reports of problems that turn out to be network failures.

There's a reason why technical support is seen as a horrible job to be in, and that reason is bad bug reports. However, not all bug reports are unpleasant: I maintain free software, when I'm not earning my living, and sometimes I receive wonderfully clear, helpful, informative bug reports.

In this essay I'll try to state clearly what makes a good bug report. Ideally I would like everybody in the world to read this essay before reporting any bugs to anybody. Certainly I would like everybody who reports bugs to me to have read it..................

Answering a developers questions directly.

When a developer asks you to check if your problem happens when shooting at shutter speeds faster then 1/200th (for example), the developer doesn't care if your neighbours dog had 13 kittens and they photograph better at 1/10th!  Be specific with your answers that relate to the questions asked by the developer.  If you don't understand the request, ask questions.  Questions imply learning!  Discussing the current political climate on pluto or any other manner of irrelevant discussion, simply wastes a developers time and slows the bug finding process.  ie:  Your problem will remain.

[deletedAcc.0021]

A good example of the lack of this being done is the 600D 'overheating' issue. Countless people have reported this issue. Not one has done anything to help resolve it. Simple scientific-like experimentation can easily help resolve the exact source of the problem without any coding skill at all (e.g. monitor the temp of the camera overtime, both the display temp and with an actual thermometer, try different builds, use the stable build as a control group, etc.). The nightly builds are for testing, nobody actually seems to be doing this, or their idea of 'testing' is: 'use in a production enviroment and complain when something goes wrong'

Myself and my crew have been using the nightlies in a production environment in probably some of the most extreme situations these cameras will ever see for testing purposes.  And when we report issues (like overheating and shutting down), it gets dismissed because someone else proved the claims invalid during a controlled test. 

I use my cameras daily for business, not once a month for short films, and I've been testing the nightlies since the beginning of the year to help out the ML team. I film and edit daily and don't have much time to devote to specific ML controlled testing, so I report what I find in the field.  If that doesn't help, then I won't waste my time.

I just hope that the dismissing of any issue, because someone said they couldn't reproduce it in a controlled test doesn't screw up someones professional shoot. 

[Audionut]

And when we report issues (like overheating and shutting down), it gets dismissed because someone else proved the claims invalid during a controlled test.

Do you have suggestion for debugging issues that aren't reproducible in controlled tests?

I just hope that the dismissing of any issue, because someone said they couldn't reproduce it in a controlled test doesn't screw up someones professional shoot.

Professional shooters should consider using a paid product with the expected personal support that comes with it.  Of course, there are many paid products that have much less support then that offered by the Magic Lantern team and it's users, so YMMV!

[deletedAcc.0021]

Professional shooters should consider using a paid product with the expected personal support that comes with it.  Of course, there are many paid products that have much less support then that offered by the Magic Lantern team and it's users.

I never expected any support from ML.  My reports are only made to help out the ML team.

When 2.3 was released, the claim was that it was considered "stable enough for production use", so I thought that was the whole idea behind testing the nightlies.  To make sure they are stable enough for production use also.

[deletedAcc.0021]

Do you have suggestion for debugging issues that aren't reproducible in controlled tests?

Yes, work with the people who report issues.  Just because you can't reproduce it doesn't mean it doesn't exist.  I think production shoots are much more reliable than controlled testing anyway. I mean what better way to know if a build is stable, than to put it though hours and hours of constant use.  Starting and stopping the camera, changing environment, different camera settings.

I've been talking with Alex privately about this and told him I would be testing the concerned build in another commercial shoot.

[dmilligan]

It's not that you're issue is being dismissed b/c it couldn't be reproduced. It's that there's absolutely nothing a developer can do, as much as he would like too, if he can't reproduce the issue. He still wants to fix your issue, but it is impossible to fix a bug without reproducing it.

I think production shoots are much more reliable than controlled testing anyway.

You can't be serious. You are basically saying the scientific method is useless.

I mean what better way to know if a build is stable, than to put it though hours and hours of constant use.  Starting and stopping the camera, changing environment, different camera settings.

That's what you'd be doing with controlled testing, except you're going to be doing it in a much more methodical and efficient manner, you're going to cover significantly more use cases, you're focus is going to be on the testing itself, carefully observing and recording everything that is going on.

Do you use every value of every setting in a production environment? No, then how are you going to know that every value of every setting does what it's supposed to do?

[deletedAcc.0021]

You can't be serious. You are basically saying the scientific method is useless.

Please quote me before putting words in my mouth, because I don't see where I said that. 

What I said was real life conditions tend to be better testing grounds than controlled situations.  It happens in every product manufactured.  How many automotive manufacturers recall cars every year for problems that manifest themselves after the vehicle was released to the public, and not on controlled testing grounds.  Toyota offered a million bucks to anyone that can determine why some of their vehicles speed out of control and kill the occupants. ... seems they can't reproduce the problem either.

Or software where operating systems need constant updating for issues not reproduced until it was packaged and sold to the public (and I'm not talking security breaches).

What I DID say was, "testing ML in a real world environment will or can determine if the code is stable and not have issues".   Although, proper laboratory testing is very useful to determine the exact cause of a problem.

It's not that you're issue is being dismissed b/c it couldn't be reproduced. It's that there's absolutely nothing a developer can do, as much as he would like too, if he can't reproduce the issue. He still wants to fix your issue, but it is impossible to fix a bug without reproducing it.

I understand this, but the bug report was removed (dismissed) like the problem doesn't exist.   I would like to see the report from the dev who did the testing that determined the problem doesn't exist and warrants removing the bug report.   I know there is a problem, I experienced it, and obviously others have too, so there must be something going on.

Either way,  I reported an issue I know to be true trying to help out.  Investigate further or not.  As far as I'm concerned then the issue is closed.

[g3gg0]

both of you are correct.
the best approach is to test code on all levels, which means using:

- static code analysis (haha!! have fun running polyspace on ML core )
- automated unit tests (e.g. on PC)
- test procedures (on the desk)
- field tests (under real-world conditions)

static code analysis
quite important but pointless for register and bit banging code everywhere in ML.
for sure possible for algorithms and maybe some stuff like modules.

unit tests
due to the nature of magic lantern (highly invasive on the system, nearly no abstraction layers), unit tests are only possible in rare cases.
modules for example are (virtually) platform independent and don't (should not) contain any model-specific hacks.
this makes modules a candidate for automated unit tests.
wanted to do that for mlv_rec, but didnt come further than making a concept in my head.
(yeah, C# again )

test procedures
are important to cover most of the obvious bugs.
you cannot catch all cases of errors with it.
good thing it can partially be automated due to in-camera button faking.

field tests
best but most expensive testing. requires time and effort.
bad thing about it - you get reports like "doesnt work" or "crashes" and you have no more details.
this is the time then to go all the way upward by finding out how to reproduce, then making test procedures, setting up unit tests and analyze the code.


its that simple to get good quality software

[deletedAcc.0021]

Nice explanation g3gg0 ...

[dmilligan]

Please quote me before putting words in my mouth, because I don't see where I said that. 

I did quote you. Saying "production shoots are much more reliable than controlled testing" is like saying the scientific method is useless.

What I said was real life conditions tend to be better testing grounds than controlled situations.

No, that is completely wrong. The whole idea behind testing is that you actually cover every situation that occurs in normal "real life" situations and then some. You seem to think that what I refer to as 'testing' somehow involves fewer conditions/test cases/scenarios than "real life". That is not what I'm talking about at all. Proper testing always covers more cases than "real world" testing ever would. You are always putting way more stress on the software than you ever would in a real world situation.

It is far more advantageous to be methodical about what you are doing, than to just have testing be a side effect of what you are doing. The idea is that you do everything that you can think of that is possible to do. How could that possibly not cover everything that could happen in a "production situation".

In a typical scenario, finding a bug is 10% of the work, finding the source of the bug is 80% of the work, and coding the solution is 10% of the work. I should know, I maintain and test software for a living.

So many people think that since they can't code, all they can do is find bugs and report them. But there is so much more that you can do for ML if you really want to help. That is the whole reason for the OP. Because even though you can't code you can still do a large part of the work, because you can help narrow down the source of the bug, if you are really willing help, and you are methodical about how you test the software, even if you can't code a lick. Developer's time is extremely limited, there are only a few and they are working in their spare time. Everything that you can do to narrow down the source of a bug and reproduce it is time saved by them having to do basically the same thing.

I understand this, but the bug report was removed (dismissed) like the problem doesn't exist.   I would like to see the report from the dev who did the testing that determined the problem doesn't exist and warrants removing the bug report.   I know there is a problem, I experienced it, and obviously others have too, so there must be something going on.

Either way,  I reported an issue I know to be true trying to help out.  Investigate further or not.  As far as I'm concerned then the issue is closed.

If you really want to help resolve the temperature issue, why don't you try doing some of things I suggested and post your results. The burden of proof is on you, not the devs. It's your job to prove to the devs that this is real problem, it's not their job to prove to you that it isn't.

[Audionut]

The burden of proof is on you, not the devs. It's your job to prove to the devs that this is real problem, it's not their job to prove to you that it isn't.

Exactly!  You must provide a way for the developer to reproduce what you are experiencing.  Developers develop code, they do not mind read!

When you can't reproduce the problem consistently, the bug can still be reported, but consider the manner in which you report it.

"Hi, I have this issue that seems to crop up every now and then, and need some help solving it". 

"I have problem you must fix"!   

[dmilligan]

Here's a great example of exactly what I'm talking about. Edgar has really helped nail down what is really going on with the 'overheating' issue by doing some very simple, methodical, scientific tests:

I  don´t know, how many sensors for measuring temperature are in an EOS-body. So I don´t know, whether the temp, shown in the exifs is from the same sensor than the temp, ML measures via efic_temp.

If i compare both, efic_temp and exif-temp, it is not the simple polynomial relationship between both, I thougt before (celsius = efic_temp - 128). This simple relation is true in a small temperaturerange from let´s say 25 to 35°C exif-temp. But with highter temp, efic_temp grows much more than the exif-temp. This difference can approximately be described with a linear funktion f(x).

I did some tests with my 600D and plotted the values with gnuplot to see, what happens. I will try to work out this function (must search first, how to do this).

What does that mean? That means, the high temperatures, shown with ML, are not correct. They are lower. Exemple: ML-temp: 83°C --> exif-temp 61°C!

So, all the hope of some photografs, they can prepare fried eggs on their camera-body, has gone. They have to buy a frying pan 

Edgar

The following function fits best to my 600D-values:

Code Select Expand

exif-temperature = 7/12 ML-temperature +12

This is the diagramm:


cmostemp von seescho auf Flickr

I will do some more test to confirm this .

Edgar

If you really want to help ML, this is great example of the kind of stuff you can do, even if you can't code.

The next step would be using an IR thermometer to compare the 'actual' temperature of the sensor to these readings.