Messages

I don't have a Windows machine to test on.  Swap is disk space reserved to swap memory to if ram is exhausted.

Currently I am trying to run export under valgrind, but it might not be practical, it's so slow

Top during dark frame subtraction export:

Code Select Expand


1395499 username       20   0 1710964 267988  22812 R 888.7   0.8  51:24.36 ffmpeg                                                                                 
1395388 username       20   0   18.0g  15.2g  53684 R 324.6  48.5  18:32.42 mlvapp


The amount of reserved mem steadily increases through the export.  Peaked at 20GB.  No crash here, this machine is fat.  It would surely crash if you had less ram + swap.  I would expect it's easy to observe memory going up during the export - should it be doing this?  Exported file is 7.7GB.  Maybe we're keeping a reference to each frame, something like that?  So they don't get garbage collected during the export?  Will dig a bit deeper.

Masc - thanks for the reply.  It built nice and easily.  Had to fiddle around a bit with config (never used it before), am now trying to repro dark subtraction crash.

Only complaint so far: it expects ffmpeg binary to be in the same dir as mlvapp, and the error message isn't very good if it's missing "encoder ffmpeg missing".  I have ffmpeg in system path, but not next to mlvapp.  Had to use strace to realise it was opening with AT_FDCWD, and then copy the binary into my build location.  Is this deliberate?  Maybe people need to use custom ffmpeg versions sometimes?  Maybe it's fixed by a proper install (I just ran make and started mlvapp from there).  It would be nicer for me if when ffmpeg is not found, it tried to use the one in system path.  Maybe that would be a bad default for other people, I don't know.

Okay, probably not a leak using all your allowed process memory...  I'm not great at diagnosing real software on Windows, just the stupid stuff malware does.  I can try and look for problems in the Linux version, they might apply on both.  The behaviour of malloc failing being persistent seems odd to me unless all mem is exhausted.  Did you try an ASAN build?

Is this the right place to get current version?
https://github.com/ilia3101/MLV-App

Do you free all memory as you go?  Any chance of memory leaks?

You really should check the return value from malloc / calloc in all cases, before using the memory.  You could do this by wrapping / redefining malloc, if you don't want to change every call site.  You can attempt to retry, but generally you are screwed if malloc fails (and some systems, e.g. default Linux, will have malloc never fail at point of allocation - the failure will occur only when you attempt to use the obtained pointer).  You can at least fail gracefully with more information (at least on platforms that have a malloc that will ever fail).

Because Linux has "optimistic" malloc behaviour, and you only see this behaviour on Windows, I wonder if you are sometimes allocating very large amounts, and not using them?  You could log the peak size allocated (some static global, wrap malloc with a logging call and output the peak value as you go).

You could try building with ASAN, that may give you more information (including memory leaks).  You could measure peak memory usage - Windows probably has some per process memory limit which you could be hitting even though the system has memory left.

What would demonstrate support for (or lack of) DebugMsg in Qemu 4?

I'd never expect emulated hardware to match benchmark numbers on a physical cam.  What are you trying to check for?

Qemu 4 also looks to have UHS-I support.

This looks how I'd expect, and still shows rolling shutter.  They've chosen scenes with fast horizontal pans, but no vertical pans.  Distortion of objects is still visible, but it's a different kind of distortion.

It might be nice to have this as an option within ML, it needs someone to mess around with registers to get a portrait resolution.

This will not and cannot remove rolling shutter effect.  That effect will always be present on a sensor that is scanned one line at a time.

Reorienting the scan direction relative to the scene can change the appearance of the distortion.  Will it be a pleasant change?  Depends on the scene.

Has been suggested several times before.  It's an interesting idea.  Likely needs someone to mess around with registers to see if any cam will record in "portrait raw".  Might work.

Cool   Good luck with the new cam!

Okay.  Mine is basically: sounds like it would be interesting to test it.

They describe the algorithm, I didn't spend much time on it but it looks like there's enough info to implement it.  If someone was willing to spend the time to do that, it would be possible to compare the methods.

What kind of discussion were you hoping for?

Turns out it is enough not to commit for 5 days to become inactive dev My 750D is sad.

I'd forgotten you had that one   In my defence, last commit to platform dir for that one was Dec, and you have too many cams   I keep meaning to pick up a 760D but haven't found a good 2nd hand one.

I like the M50 a lot as a target cam.  One significant point I forgot to mention previously: the uart is harder to access:
https://www.magiclantern.fm/forum/index.php?topic=7531.msg235437#msg235437

Most D678X "non powershot" cams have easily accessible port, under one of the rubber grips.  Although the socket may not be soldered.  We know the typical parts needed to solder it if you want.

Join usssss, join usssss.  On D678X work.  I am baised, I'm the most active dev there at present.  Walter's list seems reasonable.  I might add the M50, similar prices to M200, you may find one cheaper.  200D is D7 and in your price range.  More details follow

What cam to play with depends on what kind and level of challenge you would enjoy.  The newer D4 cams should be easiest to port, and I think they'll benefit from ML a lot, so it's not a bad choice.  Maybe they're faster than old D4, maybe have better SD interface?

D78 are very similar internally and we have ML working to menus with some small features enabled.  D7 is quite stable on my 200D.  D8 on my 850D is stable, but other D8 cams seem to suffer from frequent unpredictable crashes, cause unknown (*possibly* EVF related, since 850D doesn't have one?).  These are dual-core cams, which is fun; interesting cache behaviour and you can schedule tasks on the different cores.  We have two active devs with a 200D so that could make sense, it's not an expensive cam and we understand it fairly well.  But all D78 cams seem fairly similar, you can likely port our code quickly (a few days, we've done this multiple times and can help).  These are ARMv7-A.

D6 are dual-cpu, not dual-core.  No shared memory, some form of inter-core IPC.  ARMv7-R.  We can run code on these but not as much work has been done, no active devs happen to have D6.  We try to keep our code functional on D6 but don't have good ways to test.  It would be helpful to the group effort to have an active dev with D6, but it's the cam we can give the least help with.  5D4 is D6 so that's a cool target that would be aided by a working port on any D6 cam.  750D is D6 and cheap.

MLVApp uses QT Creator I think?  It should be relatively easy for users to profile their usage and see where the pain is coming from.  I've never done this with QT Creator but I'm a big fan of profiling if you're trying to optimise.

Also, I just realize that AMD CPU and GPU is not well supported by low level compilers

This is not true.  The article you linked to was about a specific *Intel* library that *Intel* deliberately made to be poorly optimised for AMD CPUs.  It has no relevance to "low level compilers".  Only one specific Intel library.  MLV App isn't even using that library!  Compilers optimise just fine on AMD.

Benchmarking is always quite tricky.  Since we can run MLVApp anywhere, when you're interested in MLVApp performance, it's the obvious best choice to use.  Ideally, script running MLVApp, always with the same input file, processing options etc.  That way you can get reproducible results, and you can share the script with other people so they can test on their systems.  It would be cool to see a set of results across a lot of different systems.

I remember that MacOS later versions use Linux kernel

MacOS has never had a Linux kernel.

Better avoid AMD cpus and video cards. Something I was not aware before.

Please don't give completely unfounded advice.  This is just untrue.  It's probably the *opposite* of true for MLVApp, which is heavily threaded; current Zen based AMD CPUs normally beat Intel (and Apple M1) based systems in multi-threaded workloads.

Don't guess.  Don't assume benchmarks *on a different piece of software* will be representative.  Benchmarking is notoriously complicated.  Benchmark using MLVApp on different systems; it's free and available on all OSes!  If somebody gives me an MLV file and instructions, I'll bench on my modern 8c/16t AMD Linux system.

There just aren't enough people with deep ML knowledge and free time to edit the wiki

This was a simple edit and took me less than ten seconds.  I'd say the time consuming part of improving instructions is checking them thoroughly while keeping aware of possible sources of confusion.  Finding the problem here was slower than fixing it.

I just don't quite understand how a very experienced developer can't google such a simple thing

The terms were quite generic.  People are tired sometimes, or don't have much time.  There are many reasons.

It simply seems to be an error of omission within the installation instructions, that can be remedied by adding one or two senteces.

Yes, probably.  I know "just submit a patch" is cliched for open source projects, but you may not be aware that everyone with a forum account gets a wiki account; you can submit the change you want.

Coming from an almost purely open source experience for the last 20 years, it is apparent that a majority of the devs (and almost all of the users) run proprietary OS's and proprietary software.  Any consideration for Linux and the BSD's seem like an afterthought here.

This isn't the impression I get at all, so I'm genuinely curious as to why you do.  I agree about users, but that's to be expected.  I also don't think it's very important what devs run, I'm just curious why you get that impression.  I run Linux and I'm *probably* the most active dev currently?  Alex was linux I think and I'm fairly sure he's the biggest single contributor historically.  The founder was using some linux flavour I believe.

My opinions are somewhere in the middle.  Noobs are always going to ask stupid questions, then after a while they'll learn how to do things sensibly.  But there's always more noobs joining, going through this process.  You will never succeed in making 100% of new joiners read everything before asking questions.  You'll never make docs that everyone finds clear, either.  That said:

I try to give polite answers to questions, in a way that lets people know how to ask it sensibly the next time.  I take this attitude so that the *next* generation of noobs is more likely to do things the sensible way.  I know it won't work 100%, but I hope it will approach the limit over time.  Bear in mind you're not just replying to the questioner, you're also replying to the much larger audience of lurkers.  Giving a positive, helpful impression seems better to me than a negative one.  Yes, it's frustrating answering the same questions every time - but that specific person is asking it for the *first* time, so they don't understand why they're now getting shouted at.  We can do better than that.

I think that replying in a hostile, or grumpy way is *understandable* for people that have been here a long time (or worked with similar attitudes from noobs in different circumstances), but not *helpful*.  It gives a hostile impression about ML forums, discord etc, and is off putting.  I definitely get annoyed sometimes.  Sometimes I even reply when I'm annoyed, although I try to simply wait it out.  It's okay to let someone else answer if you're not in the right mood.

Re the specific question of documentation; we should try to improve it so it makes the most sense to the most people.  It's unfortunate that very early ML docs called ML firmware, and it's unlucky that we use Canon's firmware update *menu* to do something that's not a firmware update; it's pretty obvious why this can cause confusion.  So I'd say the docs want to be extra clear in this part (there have been many changes in this area to improve it).

Re dev attitudes: ML seems fairly normal to me.  Most devs don't really want to answer user questions but know they have to.  There's an unusually wide difference between the technical skills required to make ML and required to use it, and sometimes that causes friction.  I think we could encourage a more polite, welcoming culture and this would make new users happier.  This can be as simple as replacing "You clearly haven't read the FAQ" with "Please read the FAQ and if you have questions after that, just ask", etc.  It's not hard to be polite, and importantly your answer is being read by 99 people that *didn't ask a stupid question*.  Even if you think the noob should have done better (but we were all noobs...), what about the lurkers?  They didn't do anything wrong.

It's interesting that you think most devs use proprietary OS - I'm fairly sure most code was written using Linux.  At some point a lot of work was put in to allow building on Win and Mac but this was a later addition.  Perhaps it enabled another generation of cross-platform devs?  The core devs have normally been Linux users I believe.  It looks like there was a phase with quite a few contributions from a Mac user.  I've only been working with this code for about 3 years so certainly don't know the history very well.

It took me a long time to get round to it, but thanks a lot for the code srsa!  After a lot of ARM manual reading, and working from your examples, I can now remap memory on 200D (and very probably all D78X).



Code will live here when I tidy it up and push: https://github.com/reticulatedpines/magiclantern_simplified/tree/mmu_investigation

EDIT: this commit has a working PoC for 200D, easy to adapt for other cams: https://github.com/reticulatedpines/magiclantern_simplified/commit/2ee5ccd0751d37bd03335f52f23e8673938893b4

That's a great summary, thank you!  We could probably remake an autoexec.bin from that info.

Should we document this better?  I don't really know what went wrong and how it was fixed, but it seems common enough that we should have a proper recovery process.