Author Topic: Canon 100D / SL1 (Read 579693 times)

a1ex · « **Reply #1075 on:** January 31, 2018, 09:11:40 AM »

Looking what's between these commits (hg diff -r 3ce63f4 -r 41e7183). Basically, integration of 100D and 70D, with the experimental backends they use (new DryOS hooks , qemu). Nothing obvious. That means the problem happened before including 100D/70D/M/700D/etc in crop_rec_4k.

Trying to narrow between these two - first changeset in-between is 659d35238605. That has the terminateShutdown stuff which can be commented out, and also requires DEFAULT_RAW_BUFFER_SIZE and RAW_LV_BUFFER_ALLOC_SIZE. For these, you can copy the 5D3 definition (it will only attempt to allocate if the default buffer is large enough, so the value of RAW_LV_BUFFER_ALLOC_SIZE won't be used if DEFAULT_RAW_BUFFER_SIZE is large enough - at least between these two changesets).

Also compile with make -j1, if you have enabled parallel build lately; this is old stuff before the Makefile updates.

It takes a while, but if "hg bisect" is to be believed, it takes "only" 7 tests with manual patching like that.

Another idea: does the latest 10/12-bit build (Jan31) work on EOSM/700D without this issue? It has the memory allocation stuff in raw.c backported from crop_rec_4k.

IDA_ML · « **Reply #1076 on:** January 31, 2018, 11:17:29 AM »

Oh my God, Dfort, this sounds scary! I hope, with A1ex's help you will be able to bring your trusted EOS-Ms back to life again. I keep my thumbs pressed that everything works out well for you!

I have a question to A1ex. Is it safe to use the 100D with the latest crop_4k builds, especially those working so well with MLV_lite now? Or, there is a chance that this might brick the 100D, 700D or some other camera model? Please let us know if using these builds is safe.

Thanks.

a1ex · « **Reply #1077 on:** January 31, 2018, 12:30:21 PM »

Well, guess what all the recent QEMU stuff is good for - it allowed me to understand a lot better how these cameras work, and how to diagnose them if anything goes wrong. I'd say the risk is very low, and very unlikely to cause unrecoverable damage; however, it is at least theoretically possible to erase the bootloader by jumping to the wrong program counter in the ROM. The risk is probably the same with any recent builds.

There is a lot of room for improvement, but all that requires low-level reverse engineering, feedback from knowledgeable users, and I'd even try to intentionally brick my own cameras to test the effectiveness of the safeguards (I already tried with simple scenarios I knew how to recover from). I have a couple of ideas to try in QEMU, such as writing random noise into Canon's data structures that are going to be written into ROM, and then preventing the reflashing process that takes place at shutdown. On 5D3, this is done to some extent - for example, Canon settings are not saved if you had to take the battery out or if you got some assertion or ERR70.

One idea I did not explore yet: a way to prevent the flash chip from being rewritten with standard Canon routines, no matter what they are trying to do (maybe by setting some write protection register, if the flash chip has any). Being able to emulate the flash would help, but still, the hardware might behave a bit differently, and unfortunately this can be model-specific; to find out what it does, one still has to run some risky experiments that could possibly result in camera brick (maybe recoverable with JTAG, but I have no experience with it). If the flash chip used by your camera is not listed on the Datasheets page, consider taking (or googling) high-resolution photos of your camera mainboard and attempting to identify it.

IDA_ML · « **Reply #1078 on:** January 31, 2018, 02:44:09 PM »

Quote from: a1ex on January 31, 2018, 12:30:21 PM

however, it is at least theoretically possible to erase the bootloader by jumping to the wrong program counter in the ROM. The risk is probably the same with any recent builds.

Well, but this also applies to Canon's firmware too, doesn't it? If what you are saying is true then the risk that it happens with Magic Lantern installed is the same as with just the Canon's firmware installed, isn't it? Please correct me if I am wrong.

As far as the latest builds are concerned, I have been playing around with Danne's latest build:

https://bitbucket.org/Dannephoto/magic-lantern/downloads/magiclantern-Nightly.2018Jan27.100D101_audio.zip

and it works so well with all losslessly compressed modes, with synchronous sound, that it really is a dream come true for me. Yesterday, I got almost half an hour of continuous recording with a 64 GB card at 8 ... 11 bit LL with sound at 1736x976 resolution and at 24 fps and did not have a single corrupt frame! Maximum chip temperature reached was just 56 deg. C in that experiment, at about 23 deg. C room temperature. Unfortunately, the battery ran out of juice at the 28-th minute, so I couldn't check if the camera will stop at the 30-th minute but even if it does, being capable of recording almost an hour oh high-quality RAW-video on a cheap 128 GB SD card is a real achievement for this little and inexpensive camera!

a1ex · « **Reply #1079 on:** January 31, 2018, 03:35:59 PM »

Of course, it applies to Canon firmware as well, but at least their code is well tested (I'm pretty sure they have dedicated testing teams, unlike ML, which is developed on evenings / weekends / holidays, and obvious bugs that make certain camera models or operating modes unusable are often reported after some months/years; and even when they are, we may not always know where to start looking to fix them). However, I did recover a couple of cameras that never had ML and still got soft-bricked (by third party apps via USB, humid conditions - a couple of 60D's - or who knows what else).

The issue it a limitation of the memory protection system (it allows any task to write anywhere in memory). This design also gives us complete control over the firmware, limited only by our knowledge on how Canon code/hardware works, so it's both a blessing and a curse.

Side note: DIGIC 7 models have a full-fledged MMU; from what I could tell, it's not used for isolating tasks, but could be interesting to see if it can be; that would hopefully prevent the side effects of such programming mistakes.

IDA_ML · « **Reply #1080 on:** January 31, 2018, 04:16:54 PM »

Thank you, A1ex. It's good to know these things.

dfort · « **Reply #1081 on:** January 31, 2018, 07:53:51 PM »

Quote from: a1ex on January 31, 2018, 09:11:40 AM

Nothing obvious. That means the problem happened before including 100D/70D/M/700D/etc in crop_rec_4k.

The EOSM and 700D have been in the crop_rec_4k branch since the beginning, right? Of course they weren't really supported in that branch--that's why there are so many changesets where these platforms don't compile properly. The changes only applied to the 5D3. Still, the very early versions work fine on the EOSM as long as we limit ourselves to the uncompressed options.

Of course you know this much better than I. My reference to, "I Know What You Did Last Summer," was because the EOSM was working fine until the 5D3 changes killed it last summer. Then the EOSM came back to life and started haunting... Ok, you get the idea.

I'm going to have another run at hg bisect to find which commit killed the EOSM and that should also be the problem on the 100D/700D/70D/etcD.

This is a rather strange bug to track down because uncompressed recording in Movie Crop Mode does work--after the first clip. What might give a clue is that the counter that was changed from the funky camera icon to a green, orange or red box doesn't show up on the first clip. It goes directly to the "Movie recording stopped automagically" error message. Next clip--all fine.

a1ex · « **Reply #1082 on:** January 31, 2018, 08:14:58 PM »

If it fails right away, it might be some memory allocation issue. Does this mode work fine on 700D and EOS M with the latest 10/12-bit experimental build?

Also, can you record a video of the camera screen, showing the error, with the latest crop_rec_4k build and the console open? If the 10/12-bit build exhibits the issue, please record a clip with that one as well (as it will print fewer messages - just in case the crop_rec_4k might be flooding the console with lots of stuff).

dfort · « **Reply #1083 on:** January 31, 2018, 08:58:48 PM »

No problems with the 10/12-bit experimental build.

Here is the crop_rec_4k build in action with the console open. This is the current Jenkins build from the downloads page so the issue is somewhat different than I previously described.

Here's the link in case the YouTube tags aren't working--sometimes that happens.

a1ex · « **Reply #1084 on:** January 31, 2018, 11:04:59 PM »

Got a hypothesis. Can you check changesets b041efa and e33b082? They both seem to compile on EOSM.

dfort · « **Reply #1085 on:** January 31, 2018, 11:27:20 PM »

Both of those changesets have the bug.

Taking a break from hg bisect. Only about 3 more to go!

dfort · « **Reply #1086 on:** February 01, 2018, 02:18:41 AM »

This is what hg bisect says:

Code: [Select]

The first bad revision is:
changeset:   15849:d70032d274ea
branch:      crop_rec_4k
user:        alex@thinkpad
date:        Wed Aug 16 17:50:49 2017 +0300
summary:     edmac-memcpy: experiment - use connection 7, since 6 appears to be used

Not much to it:

src/edmac_memcpy.c

Code: [Select]

 /* both channels get connected to this... lets call it service. it will just output the data it gets as input */
-uint32_t dmaConnection = 6;
+uint32_t dmaConnection = 7;
Add a comment to this line

So I took the latest crop_rec_4k, changed dmaConnection back to 6 and viola! Issue resolved. Pretty cool.

I have no idea what this is and how it affects other the other cameras.

OlRivrRat · « **Reply #1087 on:** February 01, 2018, 02:53:13 AM »

   The last 2 "Nightlys" will not Rec FRSP TimeLapses to Disc. They Appear to go through the process

of doing the Correct Number of Exposures @ the Correct Interval but No MLV or DNGs on Disc.

   Builds are >

magiclantern-Nightly.2018Jan27.100D101_audio

magiclantern-Nightly.2018Jan24.100D101_audio_hiccupfix

   Previous Build >

magiclantern-Nightly.2018Jan23.100D101

Does Produce MLV on Disc.

   Also > During the Process, on the 2 Latest Builds, the LV Displays the actions quite nicely as Opposed

to the Jan23 Build which gives the impression that it is not real sure it knows what it's doing. Or Maybe It's

Preoccupied Saving Files to Disc.

dfort · « **Reply #1088 on:** February 01, 2018, 08:41:58 AM »

@OlRivrRat - Are you trying to save FRSP lossless compressed DNG's? There's an issue with that, though there shouldn't be an issue with uncompressed DNG's and MLV's.

Oh wait--these aren't really Nightly builds:

Quote from: OlRivrRat on February 01, 2018, 02:53:13 AM

Builds are >

magiclantern-Nightly.2018Jan27.100D101_audio

magiclantern-Nightly.2018Jan24.100D101_audio_hiccupfix

These are some experiments to get audio working in mlv_lite. They are still a work in progress and yeah, everything else should be working but why are you using audio test builds if you are shooting timelapse?

Danne · « **Reply #1089 on:** February 01, 2018, 09:10:44 AM »

Here is a version with the fix from dfort for silent dng shooting:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/magiclantern-Nightly.2018Feb01.100D101_audio.zip

These versions should be treated as temporary WIP versions. Feedback is welcome.

IDA_ML · « **Reply #1090 on:** February 01, 2018, 01:15:45 PM »

Hello Dfort and Danne,

I have now briefly tested your latest build:

https://bitbucket.org/Dannephoto/magic-lantern/downloads/magiclantern-Nightly.2018Feb01.100D101_audio.zip

All I can say is that this is my favorite build sofar. All options of the "RAW video" mode work fine, with sound. I love also the lossless FRSP mode of the silent module which allows taking single DNGs at the full sensor resolution and the file size is about 18 MB - fantastic !!! If A1ex adds the trigger options with IS working on stabilized lenses to this specific build, this would be perfect for taking hand held single FRSPs at low light with stabilized lenses.

NOT WORKING:
==========

1) In-camera playback of the 10 and 12-bit UNcompressed clips recorded in the "RAW video" mode - black screen with the frame counter working. The clips get recorded fine though, and with sound!

2) The "RAW video (MLV)" mode is very unstable. I get all the time "Threads failed to start" and "Audio failed to stop" messages. It takes several times camera restarting and battery pulls to have a clip recorded, NO sound. Keeping in mind that the RAW video mode is working so well with sound, I think that for now the "RAW video (MLV)" mode is redundant until it gets fixed, (see my post #1064 for a suggestion on that).

------------------------------------------------------------------

I will keep testing and if I find something else, I will report again.

Congratulations, guys, an important milestone for ML on the 100D has been reached with this build!

Question:
=======

What are "temporary WIP versions"?

OlRivrRat · « **Reply #1091 on:** February 01, 2018, 05:47:41 PM »

DFort wrote >    "Are you trying to save FRSP lossless compressed DNG's?"    Answer > No

         "there shouldn't be an issue with uncompressed DNG's and MLV's."   Answer > Appears as though there is
            But would not have been surprised to find out it was Pilot Error.

         "Oh wait--these aren't really Nightly builds:These are some experiments to get audio working in mlv_lite.
         They are still a work in progress and yeah, everything else should be working but" Hence the reason I put Nightlys in Quotes.
            Perhaps maybe I or someone should have called them Experimental Nightlys.

          "why are you using audio test builds if you are shooting timelapse?"

             Answer > I had the Jan27 Build "@Bat" & had no reason to think that all available Modules might not be working properly,

         Especially when it Appeared on LV to be functioning Much More Elegantly than the Jan23 Nightly.

            Will give the Feb01 "Nightly"Experimental Build a try here shortly.

dfort · « **Reply #1092 on:** February 01, 2018, 06:20:11 PM »

Quote from: IDA_ML on February 01, 2018, 01:15:45 PM

What are "temporary WIP versions"?

If you think of the Experimental builds as "Beta" versions, the WIP versions are "Alpha" versions.

The issues you mentioned sound like raw_tweak problems. I believe you need raw_tweak to playback 10/12bit in camera and raw_tweak can cause some of the issues you reported. Of course if you aren't using raw_tweak....

IDA_ML · « **Reply #1093 on:** February 01, 2018, 09:59:27 PM »

Quote from: dfort on February 01, 2018, 06:20:11 PM

If think of the Experimental builds as "Beta" versions, the WIP versions are "Alpha" versions.

Does this mean that we will see this build in the Nightlies very soon?

dfort · « **Reply #1094 on:** February 01, 2018, 10:19:24 PM »

No, it means I didn't check my grammar before posting.

There is no release schedule. Things go into the Nightlies when the main developers feel like it is ready. Some of the builds you are testing don't even have a pull request so they aren't even under consideration yet. Sort of like pre-Alpha.

Danne · « **Reply #1095 on:** February 02, 2018, 09:44:58 AM »

New version uploaded:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/magiclantern-Nightly.2018Feb02.100D101audio.zip

Almost missed dfort and A1ex narrowed down the movie crop mode issue here so now that one is fixed too. Great guys:
https://www.magiclantern.fm/forum/index.php?topic=16040.msg196634#msg196634

So this wip prealpha version now works with:
mlv_lite + audio on all compression modes (ErwinH)
full buffer stops also when filming in 5x zoom mode
frsp
movie crop mode + audio

Testing mlv_rec on this prealpha build it´s working but only when mlv_rec and mlv_snd is enabled. Problem starts when enabling other modules such as mlv_play. This module won´t link so if you also add mlv_lite the mlv_play will link and work BUT that is when audio fails to stop for mlv_rec. My guess the module fiesta has to be linked or separated from each other.

Tested regular crop_rec_4k and mlv_rec + snd works when mlv_lite and mlv_play is enabled so the problem is within the prealpha set up. Seems the linking set up with ErwinH mlv_lite version with sound changes interferes with mlv_rec audio.

a1ex · « **Reply #1096 on:** February 02, 2018, 11:54:15 AM »

New version uploaded, too - except mine is untested on 100D

(well, I did try it in QEMU, but that only shows that it boots, can browse the menu, can load modules and other basic stuff like that)

Included mlv_snd, Lua, silent picture updates and hopefully all recent fixes (please let me know if I've missed any).

Danne · « **Reply #1097 on:** February 02, 2018, 12:19:42 PM »

Yes, noticed. Great! I will do some more tests later but inially(tests on 100D).
- mlv_lite sound working(It breaks mlv_rec sound functionality with mlv_lite+mlv_play added however mlv_rec now seems more and more redundant)
- Lua seems working https://bitbucket.org/Dannephoto/magic-lantern/downloads/LUATEST.LOG

Vaguely tested
Ok, so did brief tests on silent dng files, somebody else need to take on. Regular silent dng files seems to work. Still issues with lossless. Fullres didn´t save picture. Liveview window prints something about compression error -3. Take this test with a grain of salt. Was in a hurry.

Regarding:

Quote

latest memory updates for raw backend (please report back on 70D and 100D, will require compiling from source)

Compiling from crop_rec_4k source or allocate-raw-lv-buffer?

a1ex · « **Reply #1098 on:** February 02, 2018, 12:27:45 PM »

Easiest from crop_rec_4k, as it includes everything from allocate-raw-lv-buffer.

That -3 is a good hint - in lossless.c, near line 353, can you print MEM(WR1_Address) as hex?

I've reverted ErwinH's changes to mlv_snd to avoid breaking compatibility with mlv_rec, but they need to be loaded only one at a time (that's a limitation of the callback "system" used in modules, but g3gg0 is working on something more generic to address this).

OlRivrRat · « **Reply #1099 on:** February 02, 2018, 06:00:15 PM »

@Alex who wrote "New version uploaded, too - except mine is untested on 100D"

Will be happy to test if You would be so kind as to say where it can be found.

News:

Author Topic: Canon 100D / SL1 (Read 579693 times)