It seems there are two types of users - those who occasionally don't want ML, and those who occasionally do. I am sympathetic to the latter, as my first experience with ML was like some describe. There are so many mysterious settings, many of which can screw up your shooting. At very least you fear they can. It takes time to get over that hump.
Assuming it's possible, how about a option in Preferences:
Preferences
Override startup >
Hold SET to unload ML (default)
Hold SET to load ML
If in the 'Hold SET to load' mode, it could display a prompt while bailing, that says, "Hold SET during power-up to load Magic Lantern". This would address the issue of users locking themselves out.
So far, I also use ML only for video. If I had this option, I would use 'Hold SET to load' (opt-in) most of the time, and 'Hold SET to unload' during video projects.