Messages

I had to modify my previous post as the step length change observed is due to a tiny rotation of the focal length barrel.
I will try to show the variation of the step length according to the focal length.
It also occurred during the following (2d repeatability) test : the barrel has to be at mechanical end.

https://www.dropbox.com/s/ub68gt6l0u0u9ae/70-200f28ISII%40200_2.log?dl=0

It should be noted that the number of steps is very high (more than 8000) and that these are very tiny step length (3 fine steps) for step-size=1.

There were "problems" with the EF70-200 f2.8L IS II USM too, like with the EF 100 f2.8L IS USM. It doesn't seem to be due to the lens itself as I suspected before with "electrical false contacts".

This tests @200mm seems special too: lots of steps (high precision) and different step-lengths for the two directions A focal length change has modified the step length between the two directions (nothing unknown):
- macro to infinity: 4 fine steps, then 3 fine steps,
- infinity to macro : only 3 fine steps,
though the step length graph is very clean (no outliers) !
Repeatability tests will maybe shed some light...

https://www.dropbox.com/s/7dmhieso4hc4b54/70-200f28ISII%40200_1.log?dl=0

[ask help from specialists ...in our community...or from academic people...]

I am not a specialist in optics but I will try to help. If you think I am wrong, don't hesitate to tell it !
For an authorized advice, I would ask help from specialists ...in our community...or from academic people...



As you know, real lenses are complex optical instruments.
For me, the thin lenses formula is to be considered as a simplified but convenient transform from the object field to another field not far from the image field ; in that meaning, the transform can be used with different parameters (like another focal lens value), and adapted according to our best needs.
I also thought of the  « focus breathing », where the « apparent » focal distance changes when the focus distance changes....Cine lenses avoid that problem, but most picture lenses don't when they use inner focus. In this case, even if the lens has only one focal distance in it's name, it is only valid when focusing to infinity.

But one must take care of the fact that this simplified model can lead to errors depending on what we want to infer from it and with which precision.

For instance, the actual object distance will be an approximate without calibration as a1ex explains it in his previous (#63) post.

Anyway, I think there will always be an uncertainty relative to the precision of the actual position of the hardware, even if the software has done it's job. This is true if we can only work in open loop (cf. focus backlash, motor errors...) and will always limit precision.

All depends on what the community wants to do. Is it:
1) positioning the lens at a given target position
2) focus stacking
3) display more/very accurate current focus distances in Lv

For each choice:
1) positioning the lens at a given target position
- If the target position is determined through auto focusing and if we just want to go back to that target, the position counter is just sufficient. The precision may go from one step to maybe one half of a step (step-size=1) (which is to be converted to the object field).
- If the target position is given by the object distance, IMHO, it will be difficult, at least without calibrating the lens.
2) focus stacking
The Canon focus_distance can be used to build a continuous function of the position counter (this may be a step forward over the discontinuous stair function available now). According to DOF calculations, the position of the shots needed can be calculated. Then we have to move the focus position to each of the calculated positions as in 1) above.
3) display more/very accurate current focus distances in Lv
Display more accurate focus distance may be possible (not near infinity), but very accurate display will need a lens calibration as a1ex explained in his previous post.

@A1ex
Here are still the results of a few tests:

1)Last repeatability tests for the EF 17-40 @ 28mm

https://www.dropbox.com/s/pksd0mw62ts2mg5/17-40f4L%4028_2.log?dl=0

https://www.dropbox.com/s/vvvo0onl5nvlzlg/17-40f4L%4028_3.log?dl=0

2) First test for EF70-200mm f/2.8L IS II USM @ 70mm 
(after a run with the same values at all steps) Next time I will switch console on !

https://www.dropbox.com/s/kekr4h78aa77goo/70-200f28ISII%4070_1.log?dl=0


Thanks for the disassembly paper of the 50mm 1.8 II !
The DC motor + optical encoder are there with a lot of little gears !
There is not so much mechanical backlash though...

For information purposes, here is a comparison chart between the stairs functions obtained:
- in function of the actual position counter value with actual varying steps (red)
- in function of the number of steps as if the step length were constant(blue).

https://www.dropbox.com/s/9w0vtwmp4h1nyvk/100MacroIS_5th_test.jpg?dl=0

Notes:

X axis : position counter during the scan (the scan starts at macro mechanical end = 0) (in sub-steps)
This lens has many sub-steps : about 14426 sub-steps for about 3866 steps.

Y axis : focus distance in the image field (absolute value in mm)( the thin lenses formula has been used to project distances from the object field to the image field)

BLACK points : the same Y scale has used to display the step length in sub-steps. The scaling formula is : displayed_length = 110 + step_length*10. The values have a visual trend : go down from the macro end to a minimum and then, a little upwards near infinity end. The reason (if any) is not known.There are many outliers especially near infinity end (unknown reason heretoo).

RED points : The focus distance in the image field is displayed in function of the actual position counter values : at infinity, the distance between the focal lens centre and the sensor is equal to the lens focal length = 100mm. The position counter value giving the position of the focal distance is not known.

RED curve : it is possible to ask LibreOffice to calculate automatically a trend curve : a 2d degree polynomial has been chosen : the formula and the regression coefficient (least squares fitting) have been displayed near the curve. Actually, the curve should be forced to pass on the point (x,y) where x = lens focus image ( in french = n° du sub-step correspondant à la distance focale image) , and y = lens focal distance ( in french = distance focale image de l'objectif) but as written above, the position counter value corresponding to the focal distance is not known. Note : the coefficients of x and x2 seem tiny but the values of x and x2 are big !

BLUE points : for comparison purposes, the focus distance in the image field is displayed as if the step-length was constant from macro to infinity end (which it is not the case : see black points above) : the mean value of the step length has been calculated and taken as constant step length.

BLUE curve : a 2d degree polynomial curve has been fitted to the points automatically.
The curvature/concavity of the blue curve is less then the curvature of the red one.

Does the loop stop when reaching the end, or it just stays in some infinite loop?

It stays in some infinite loop. As the lens is moving, it is possible to see when the end is reached. The sound changed too. I left it continue a little while and then switched the camera off.

The step count difference between the two attempts (1275 vs 1424) is pretty high

I wonder: as far as I can remember (Youtube video ?), the piezo electric motor present in this lens is not a ring motor which would directly move the helicoïdal lens barrel without intermediate piece. Moreover there are other external moving parts as the protruding lens, the manual focusing ring in front that is rotating : more friction ?
I wonder if this is not only an electronical counter not related to the actual position of the lens barrel.

but your experiment looked quite linear

I think I am misleading people with that picture: the red and yellow are only my attempts to adjust a straight line via the least squares method (linear regression); the only measured points correspond to the steps of the decreasing blue stair function.
I know what happened : this picture has been plotted against the number of steps (step-size=1) ! so, with the assumption that the step length was constant (at that time (july), I had not yet used the position counter : I just counted the number of steps); as the length step decreases towards the infinity end, the curve gets "compressed". My apologies  !!! I am working on another comparison picture.
Note : I only used the "thin lenses" formula : 1/p + 1/p' = 1/f where p, p' and f are in the same unit (mm), p is the focus distance, p' the image distance = distance from lens optical center to the sensor and f the focal distance.

Here are the results of the last test of the 17-40@17mm:

https://www.dropbox.com/s/82je0peg7tchqzw/17-40f4L%4017_3.log?dl=0

I have tested @ 28mm (in between):

https://www.dropbox.com/s/24bxn4obfyv09uf/17-40f4L%4028_1.log?dl=0

If you like, I can also test an EF 70-200 f/2.8 L IS II USM (quite a wonderful one)...I have never dared to put it in test  .

After some homework, here are the 2 first test results for the 17-40 @ 17mm :

https://www.dropbox.com/s/0fj4rinhm6z56xc/17-40f4L%4017_1.log?dl=0

https://www.dropbox.com/s/j0prhh637yrwh94/17-40f4L%4017_2.log?dl=0

I have tested the 50mmf/1.8 II after commenting out the first loop and turning the "front barrel" to mechanical end (macro):
Note : this lens does not report the distance to ML
Note : when reaching the infinity mechanical end, the sound changes from louder to fainter indicating the motor does not work anymore; only the mechanical end protection (micro switch?) maybe.
I have left it working a few times at the mechanical end.
I guess the position conter counts and stops counting when the end protection enters in action because the same position should have been repeated more often.
The step length varies but there is no focus distance reported.

https://www.dropbox.com/s/qwp7drl7euz5n2z/50f18II_MI_1.LOG?dl=0

https://www.dropbox.com/s/z9tczx173ajycmg/50f18II_MI_2.LOG?dl=0

Many thanks for those informations as well as the very interesting paper on the Kalman filter (I would like to look at closely).

The next test for the 100mm Macro IS was difficult to get done. During the first run, the position counter and focus distance were stuck on the first values for the whole length of he file. I had to try and find out what went wrong. I think there are false electrical contacts. I still don't know where actually (a Gray encoder ?). Maybe this lens needs servicing.

Anyway, I could finally get a correct run (but the results for this lens will have to be taken with greater care given the troubles we have had with it) :

https://www.dropbox.com/s/d7ujp5buqwoff1m/100MacroIS_5th_test.LOG?dl=0

This morning, I had time to do the 2 repeatability tests with the EF 17-40mm f/4 L USM @ 40mm with a rest time between them (to be sure, as I don't know about the temperature of the friction surfaces) :

https://www.dropbox.com/s/75oety1f5fh6w77/17-40f4L%4040_2.log?dl=0

https://www.dropbox.com/s/ctohymzgak84f8o/17-40f4L%4040_3.log?dl=0

and do a first try with the EF 50mm f/1.8 II : I think it went in an infinite loop with the first loop on lens.focus command (step-size=3) as there is nothing in the log file:

https://www.dropbox.com/s/6l8xo5lxjr5r1vt/50f18II_0.LOG?dl=0

The 100mm will need more time.

The Extradelay in Lens.focus() has been set to 500 ms.

You can find here the repeatability tests for the 35mm f/2 IS

https://www.dropbox.com/s/8waglc2nc96pyhf/35f2IS_2.LOG?dl=0


https://www.dropbox.com/s/nevqdg1stxqugjp/35f2IS_3.LOG?dl=0


and the first one for the zoom EF17-40mm F/4 L USM @ 40mm

https://www.dropbox.com/s/xmaf82vmuhz4us3/17-40f4L%4040_1.log?dl=0


I will retry the test of the 100mm Macro IS, but with 500ms delay, Lv will probably shut down I am afraid : I will try to dim Lv display,...

The hysteresis was present in my first test as well.

Yes, you are right (of course).
I made a mistake at that time : I thought it was like the clipping at each end (which is less than a step in length)...

I made :
- a second test with the 100 mm L macro IS USM (switch on full range as for the first test, same delay conditions = extra-delay set to 100ms) :

https://www.dropbox.com/s/k3xpj93srzyvvye/100MacroIS_2d_test.LOG?dl=0

- a first test with the 35mmf/2 IS USM (extra-delay=100ms too).

https://www.dropbox.com/s/vlziuqejrkew2pd/35f2IS_1.LOG?dl=0

What should be the strategy to increase the delay ?
What do you think of a loop for testing the value of the position counter ?
Note : the tests I have made till now are not a reference as I use very large delays to avoid problems (500ms for scans and seconds for the tests I have made with the 24mm to reach a target from random starting positions) which is unusable in practice.

Very nice find, the hysteresis !

Would have had problems for positioning the transition more precisely if moving back and forth. Would like to see with step-size=2 if the change of the step length at the transition position is still valid in the other direction.

For the 24mm, it should be somewhere near 45 fine steps ? (to "visually" make the curves best match each other) which is significant in relation to the step length of 7 fine steps.

Anyway, I can share the logs for the 2d and third test for the EF24mm f/1.4L II USM.

2d test: starting in-between (macro and infinity)

https://www.dropbox.com/s/kbj7irdaqab94jd/24f14II_2.log?dl=0

3d test : starting near infinity end (done after a rest time)

https://www.dropbox.com/s/0txw7b3rsa4z1r8/24f14II_3.log?dl=0

I will retry the test with the 100mm Macro (Fyi, it was in full range the first time)
You are right, I don't think the delay would be the cause of this sort of trouble.

Your script is very fast, but the position counter does follow and finally returns the good values even if they arrive later.

Many thanks for your answer !

I just had time to run two first tests : at first sight, the delay could be too short.

EF100mm f/2.8L Macro IS USM :

https://www.dropbox.com/s/uhb53tpjc16pzpf/100MacroIS.LOG?dl=0


EF24mm f/1.4L II USM :

https://www.dropbox.com/s/d8b082myficge0n/24f14II.log?dl=0

@a1ex
I'm back.
I will do that.

Here are the lenses I can use (some L and some primes/zooms) :
EF 24mm f/1.4 L II USM
EF 35mm f/2 IS USM
EF 100mm f/2.8 L Macro IS USM
EF 50mm f/1.8  II USM
EF 17-40 f/4 L USM
Which ones is it better to choose ? (the step lengths seem more constant with L)
How many times do you think I should repeat the same experiment (to see...) ?

If needed, I can share some results of experiments:
Homing strategies
I have tried some "homing" strategies to each end (go to the mechanical end and sync hardware and software : absolute position = relative position + offset - hw_error ), but as far as I have seen (EF24L step-size=3), the precision was not so good, even for a L lens (I felt it was better not to do that, even less systematically, even when the target is near to one end ; actually, it is not needed in normal camera precision use (AF) but only as a protection ). My last thinking about homing was to assume that there were sort of "reading errors" of a given level (dispersion) at both ends that we have to accept without any change to the offset I had taken at the beginning of the script. After a given number of moves (heuristic to be found), it could be interesting to home because the hw_error would be of a higher level than the acceptable reading errors...given that hw_error could also subtract themselves.
Transitions
I am in sync with you. These are the nearest source of information we have; (from experiments), I have seen "metastability" (taking the value of focus_distance at successive moments, it changed and went back to it's first value, not very bad : the only problem could be it can take more time in the worst cases (for a few step-size=3 the lens can take a little while to stabilize: the question is to wait just when needed and detect); I have not tested it but the position counter changes could be checked in this case until they don't move anymore, even more if lens.focus is used with wait argument=false); systematic errors could be eliminated by minimizing the sum of the differences between the current readings of transition positions and previously saved positions; the remaining errors ? random?
I have not yet tried to position the transitions at a higher precision then the step-size=1 when scanning.

@Garry23
Thanks!

You put the finger on a good question.
You are a passionate to a certain level like many of us. I have seen an expression somewhere "for the theory of it". I think this is part of the answer.

Give me some more time for a better answer, but you are right : it can not be possible to do much better things than what the lenses have been designed for.

IMHO, from the little I know of the many different lenses, I think it will be possible to interpolate to get better precision for stack focusing or lens positioning, maybe in the infinity end.
I think there is a group on bitbucket to enable more precision via interpolation : this is a good thing.

Step-size 1 is precise : an important thing is to know where to stop.
It is where the position counter will give you something :
- you will not be "limited" by the "step length" of each step of the step function as you can be now... (still to be proved for landscape, as you say).
- you will know if it is better to move the lens one step further, or if it is better to stop where you are : this already reduces the error to a fraction of a step-size 1 step.
The hardware will never allow us to go to the sub-step precision level.

I have tried to figure out on a 45° target with the EF24mm f/1.4 L II USM lens at full aperture as close as possible to reduce the DOF.
The software gives a 0 error but on the hardware side, on the pictures taken of the target, it is very difficult to see.
That's why I turned to a macro lens.
I will share the results here with pictures.

@Garry23
Thanks for sharing!
I am always interested in what you do.

Would you like to tell me a word about the optimise function ?
I think I understand. Quite interesting !

I see you have used the step-size 2 to go faster : very interesting too.
I wonder if you preferred not to use step-size 3 (or, maybe, for next time ?).

@a1ex

Thanks!
I think I understand you (tell me if I am wrong) : with the "lenses equation", it is possible to do the transform from the object to the image field. Here is what I have tried in this direction:

https://www.dropbox.com/s/adbpq6ryatrobs9/EF100mmf28LISUSMAv56_3.jpg?dl=0

I thought the "infinity end" of the step function seemed "not linear enough" (if I can say that).
I am not sure there is a non linearity in the helicoidal barrel (in cylindrical coordinates, z = k*theta , transforming the rotation of the ring motor (theta) into a linear move (z) ) ?
I don't think so...
At least, it should give a more easy way to handle the infinity end.

I would like to ask you if you have any info on the C data structure above?
I wonder what is called "lens_rotation" and "Lens_step" ?

Modifications:

In my previous post, the mentioned EF50mm F/1.8 II USM should be better replaced by the EF 35mm f/2 IS USM : I already have a graph of the small variations of the step length to show you the problem of what variation should or should not be taken into account:


https://www.dropbox.com/s/dmx38e0ovu9jgeq/EF35mm_SCAN_dom2.jpg?dl=0


@Garry23
I think you can be interested by the following picture.
I can share the following picture  , but take it, as always, with a grain of salt  .

It shows the results of scans of the EF24mm f/1.4L II USM lens with step-sizes 1, 2 and 3.
Each point represents the value of the lens.focus_distance in function to the position of the lens achieved with lens.focus() moves of step-size 1 (blue) 2(red) 3(yellow) beginning at macro end.
For red and yellow colors, the points are connected by straight line segments. For the yellow points it is like a linear interpolation. The graph is limited to an interval between positions 2000 to 4000.


https://www.dropbox.com/s/61awuv8q0y1u9gl/EF24mmf2ISUSM_SCANS123.jpg?dl=0


What seems interesting (To be checked) to me is the yellow curve which goes from point to point as if it gave the positions for the pictures of a focus stack (with a big grain of salt as IMHO it does not change with the aperture value) . As you used step-size 2 in your optimized HFD script (for which I thank you for sharing : I will have too check how you found the best aperture), maybe you can go one step further ?

@knoodrake
Thank you for your initiative !

I wonder if what I have written is understandable...(sub-steps, step-size,...).
If you have any question, don't hesitate to ask!

I was asking myself why the step-lengths were changing and how one would have to do to store the values to use them in a Lua script (companion file).
When the step-length is constant on the whole range, it is rather simple...but when it changes from time to time, what data structure should I conceptually set up to store where a new step-length has to be taken into account. With some lenses like the EF 50mm f/1.8 II USM EF 35mm f/2 IS USM there are (little) changes on top of that. What are the real changes and the (possible) random changes (sort of hiccup) ?

The simultaneous variation of the lens.focus_distance and the step-length gives a clue to make a distinction between real changes (coupled with a change in the focus_distance value) and the random ones (when there is no simultaneous change of the focus_distance).
This simplifies  to a limited number of intervals on which the step-length should remain the same.

This simultaneous variation gives better consistance to the step values as such : they acquire sort of real existence : they are specific to a given lens and have been decided when the lens was designed. With the EF 100mm f/2.8L Macro IS USM, the corresponding match between the different scales and the same length of the steps is really astonishing. It is difficult to speak of a coincidence.

There is still a conceptual problem as where the step-length changes, one would have to store two values : the value when going to macro end and the other value when moving to infinity end.

I will have to dig into lens.c and lens.h to try to find in the prop_lv data structure if there is some integer equal to the step-length I have seen.

Code Select Expand


struct prop_lv_lens
{
        uint32_t                lens_rotation; // float in little-endian actually
        uint32_t                lens_step; // float in little-endian actually
        uint32_t                off_0x08;
        uint32_t                off_0x0c;
        uint32_t                off_0x10;
        uint32_t                off_0x14;
        uint32_t                off_0x18;
        uint32_t                off_0x1c;
        int16_t                 focus_pos;  /* see lens_info.focus_pos */
        uint16_t                off_0x22;
        uint32_t                off_0x24;
        uint32_t                off_0x28;
        uint16_t                focal_len;      // off_0x2c;
        uint16_t                focus_dist;     // off_0x2e;
        uint32_t                off_0x30;
        uint32_t                off_0x34;
        uint16_t                off_0x38;
} __attribute__((packed));


I think there should be something interesting here.

There seems to be a strong correlation between the position of the changes in the length of the step (expressed in sub-steps) and the position of the changes of the value of the focal distance. I mean they seem to change at the same position when scanning the lens from macro end to infinity end.

The values of the focus distance (as displayed on the liveview screen or given by lens.focus_dist in Lua) come from a data structure: they are read somewhere if I remember correctly. I would not be surprised if the values of the step length could be found there too, or nearby in the same data structure.

Being at a given position, one could then know what is the length (in sub-steps) of the step he can do and be able to calculate where he will get if he chooses to move with a given step-size. Companion files (with those step lengths data for each lens) would be unnecessary.

For those who are not afraid of spreadsheets, I put here under, the LibreOffice sheet I started to study the sub-steps scales of the EF 100mm f/2.8L Macro IS USM : with a little switch on the side of the lens it is possible to choose between a macro scale (30-50cm) a longer scale (50cm to infinity) and a full scale. The sheet « compar 30-50 and 50-infty scales » shows a beautiful correspondance : the scales can be superposed with an offset of 588 sub-steps.


https://www.dropbox.com/s/xw2zdd58ee7dm8p/EF100mm30_50macroA.ods?dl=0


What do you think ?

I meant it can be useful to find errors before execution time.
Luac maybe is one way to do that but it is not the only one.
This may be better discussed in the "Lua scripting" topic.

Don't change all your tutorials just for that.
IMHO they are very well made for now.. Thanks for that.

I think the brew instructions you shared are well enough for people having such particular needs.

Here I can share with you the results of the moves back and forth of 6 steps (of step-size=2) apart from each other.
The camera is nearer to the target (almost the nearest focus distance).


https://www.dropbox.com/s/lu5v9vw57dgjuyv/7411_7415.log?dl=0

https://www.dropbox.com/s/o24pw6pdswhq55s/_GOR7411_CROPPED.jpg?dl=0

https://www.dropbox.com/s/rtnlropnagcjstj/_GOR7412_CROPPED.jpg?dl=0

https://www.dropbox.com/s/4b93tfgkgv9c22y/_GOR7413_CROPPED.jpg?dl=0

https://www.dropbox.com/s/9gjxyt5rlb7cfj4/_GOR7414_CROPPED.jpg?dl=0

https://www.dropbox.com/s/0fvok825ezroko5/_GOR7415_CROPPED.jpg?dl=0


Next time, I hope I'll be ready for moves from random starting points back to the same target using step-sizes= 3 for approaching, then 2 and 1 for precision.
I have simulated the algo which gives spot on results (error=0).
But the reality will be different for sure !

Would someone mind sharing an example of the keyframes used for a day to night or night to day timelapse ?
I know one case is not the other...
If you don't mind, I would also like to know : the location, the azimuth, the elevation and the date.
Don't worry, it doesn't need to be exact: I would be very thankful with only (very) rough approximates  .

Maybe off topic ?
For new developers like me, it is interesting to detect as many errors as possible before run time in camera.

Some 40 years ago, the teacher was speaking of ?modula? as a language intended to be coded as easily as writing (sort of code as easy to write as pseudo code).
This is much further : it becomes some sort of artwork. Very nice !