Author Topic: Canon 1100D / T3 (Read 1124432 times)

Danne · « **Reply #1450 on:** January 29, 2018, 02:00:48 PM »

According to your question:

Quote

Can you try this on a MLV with more than 2 frames?

It´s about 60mb in size so I assume it´s more than two shitty frames in there:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/M29-1336.MLV
Enjoy your wine...

a1ex · « **Reply #1451 on:** January 29, 2018, 02:18:23 PM »

Hm, other than frame #2 (corrupted) and possibly #3 (broken metadata), all other frames are clean.

Can you run the stubs test (selftest.mo) from both the 10/12-bit build and the lua_fix build? Also EDMAC screenshots and other related stuff from selftest.mo and edmac.mo.

Danne · « **Reply #1452 on:** January 29, 2018, 02:27:11 PM »

Later, work atm.

starbase64 · « **Reply #1453 on:** January 29, 2018, 04:24:29 PM »

Quote from: a1ex on January 29, 2018, 11:03:50 AM

There is an experimental raw video build, but other than a few snapshots from Danne, I have no idea how it works in practice => please report back. The silent picture module requires the LiveView raw backend enabled, so try it from the 10/12-bit raw video build.

If it works fine or if I can solve it easily based on user feedback, I'll consider integrating this feature into mainline.

this works fine, but display goes off and not on again.

possible to combine silent pic with advanced bracketing and intervalometer?

Danne · « **Reply #1454 on:** January 29, 2018, 04:28:18 PM »

dfort · « **Reply #1455 on:** January 29, 2018, 06:19:21 PM »

Quote from: Danne on January 29, 2018, 04:28:18 PM

Stubs API test starts and run but after a while falls into an error.

Interesting, I just ran the Stubs API test on the EOSM2 thinking it might show what I need to work on and it failed on LoadCalendarFromRTC, same as your 1100D test. Ran it on the EOSM and it also failed at the same spot. All tests passed on the 700D.

a1ex · « **Reply #1456 on:** January 29, 2018, 11:19:01 PM »

The above log shows successful result on LoadCalendarFromRTC; the problem is at the next test

Got this in QEMU:

Code: [Select]

       m0 = MALLOC_FREE_MEMORY => 0x43148
[Pass] p = (void*)_malloc(50*1024) => 0xc1b40
[Pass] CACHEABLE(p) => 0xc1b40
       m1 = MALLOC_FREE_MEMORY => 0x36938
       _free(p)
       m2 = MALLOC_FREE_MEMORY => 0x43148
[Pass] ABS((m0-m1) - 50*1024) => 0x10
[Pass] ABS(m0-m2) => 0x0
       m0 = GetFreeMemForAllocateMemory() => 0x6f24c
ASSERT : Memory\Memory.c, Task = run_test, Line 188
  1862:  7111.168 [MEM] NG AllocateMemory 262144
  1863:  7111.168 [MEM] Total = 0x8b0000, Free = 0x6f1ec, MaxReg = 0x239d4
  1864:  7111.936 [MEM] Num Alloc = 3296, Num Free = 293

The error is actually on the next line - it's unable to allocate 256K of memory...

a1ex · « **Reply #1457 on:** January 30, 2018, 12:47:28 PM »

The lua_fix and 10/12-bit experimental builds also have the latest changes for 1100D; any luck with these?

The silent picture module should work too.

Danne · « **Reply #1458 on:** January 30, 2018, 02:35:57 PM »

Tested 10/12-bit experimental
Silent DNG works and MLV has the correct size although giving malloc errors, check pics below. As I managed to stop the file before the camera buffer ran out it´s also possible to transcode the 10bit file with mlv_dump and the files looks really nice.

Stubs api test is working, ends with following message:
"test complete 11756 passed. 3 failed"
STUBTEST.LOG
https://drive.google.com/open?id=1AftiepRPtHyTs2iKOzcauhGLGH94uUUN
Some other log:
https://drive.google.com/open?id=1X88I13ruQ_jKapWnR1R0ijZI9fE5KFFe

Silent DNG:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/90630000.DNG

MLV:
https://bitbucket.org/Dannephoto/magic-lantern/downloads/M30-1622.MLV

Screenshots when recording 10bit

starbase64 · « **Reply #1459 on:** January 30, 2018, 04:07:42 PM »

can someone please update the download links in the first post...

Tested lua fix build 30.01.2018 and works fine for me. THANKS.

a1ex · « **Reply #1460 on:** January 30, 2018, 10:49:39 PM »

Done.

Reminder: nobody ran the Lua tests on this camera yet. They were linked earlier in this thread ~ 1 month ago.

Updated 10/12-bit builds with memory management from crop_rec_4k and attempted some integration to raw_video_10bit_12bit. Will it work or will it break raw video on all other cameras?

Danne · « **Reply #1461 on:** January 31, 2018, 08:51:18 AM »

Quick dirty test:
magiclantern-raw_video_10bit_12bit.2018Jan31.1100D105
Tested two 10bit MLV recordings: Data corruption at slot 51(file is recorded only two files extracted as before)
Tested on 12bit recording: Data corruption at slot 61(file is recorded only two files extracted as before)

Tests following this:
https://www.magiclantern.fm/forum/index.php?topic=14828.msg194706#msg194706
magiclantern-lua_fix.2018Jan30.1100D105
LUATEST.LOG
https://drive.google.com/file/d/1SivdU_wQi4Et1Mc4r5ygAF0lNbDIRDY3/view?usp=sharing
benchmarks memory
(in liveview window)
malloc error

(sad smiley from liveview window)

Should test my 100D with the latest changes in raw_video_10bit_12bit branch....

Quote

Updated 10/12-bit builds with memory management from crop_rec_4k and attempted some integration to raw_video_10bit_12bit. Will it work or will it break raw video on all other cameras?

magiclantern-raw_video_10bit_12bit.2018Jan31.100D101
Tested a 100D with the latest changes And ran test recordings on 10/12/14 bit for mv1080, movie crop mode 5x zoom, all worked.

a1ex · « **Reply #1462 on:** January 31, 2018, 09:20:36 AM »

Data corruption: regression or nondeterministic issue also present in previous build? (this will take some more recordings to figure out)

Can you test the Jan31 build on something else than 100D?

Lua test locked up at testing Canon GUI functions? That's odd - this test is just flipping GUI modes back and forth (MENU, PLAY, LiveView, photo mode). This test also works in QEMU...

Danne · « **Reply #1463 on:** January 31, 2018, 09:38:03 AM »

Quote

Data corruption: regression or nondeterministic issue also present in previous build? (this will take some more recordings to figure out)

Was not an issue with earlier build. Only "Hack error at 0" Could test a few more files to see if I can reproduce data corruption.

Ran a few files on a 600D with this build and all files were recorded nicely:
magiclantern-raw_video_10bit_12bit.2018Jan31.600D102

Don´t have my 5D mark III available, maybe someone else could test on that one.

Update*
Data corruption issues appears also on januay 30th build when stopping manually or camera automatically stops filming. When checking the MLV files there is a corrupted frame in the later part of the file more or less always present an all files.

Danne · « **Reply #1464 on:** January 31, 2018, 09:57:49 AM »

Tested 31th january build on a faster SD card and 10bit was continuous with max resolution1424x480.
Data corruption seems gone. Maybe it was a slow/bad card. Will do more takes with the faster card...Yep, data corruption seems vanished.
Two noob questions. Turning off the beep? Is there a manual mode setting for when recording movies? Cannot find it.

How about lossless...

Some more tests:
- regular mlv_lite shooting works fine(mv720, only mode available on this cam)
sample: https://bitbucket.org/Dannephoto/magic-lantern/downloads/M31-1208_samples.zip
- 5xzoom reaches 2160x748 but needs some shaving at the top part it seems:

2160x748(5xzoom)
sample: https://bitbucket.org/Dannephoto/magic-lantern/downloads/M31-1212_samples.zip

1600x748(5xzoom)
sample: https://bitbucket.org/Dannephoto/magic-lantern/downloads/M31-1209_samples.zip

Fullres silent DNG:
Seems problematic.
Sample: https://bitbucket.org/Dannephoto/magic-lantern/downloads/90640003.DNG

starbase64 · « **Reply #1465 on:** January 31, 2018, 10:56:41 AM »

Quote from: starbase64 on January 29, 2018, 04:24:29 PM

this works fine, but display goes off and not on again.

same problem on 70D, display is off after take a photo

a1ex · « **Reply #1466 on:** January 31, 2018, 12:12:31 PM »

@Danne: actually I was thinking at other models with low memory (600D, 650D, 700D, EOSM) and I remembered you had something from this list. These updates were taken from crop_rec_4k, initially written from 5D3, so that test is unlikely to give any surprises.

Can you also take a screenshot of the console (Debug menu - Show console) right after starting the camera in movie mode and enabling raw video, and another one right after turning it off?

Lossless compression on DIGIC 4 is hard - my attempt on 60D (some months ago) was unsuccessful. I'll revisit it at some point, but not right now - my short-term goal is to integrate the current stuff into mainline (QEMU, Lua, 10/12-bit raw video, 100D, 70D, 1200D etc).

The silent DNG looks fine to me (I have something to address the overexposure, work in progress, stay tuned).

@starbase64: if you are talking about the full-res silent picture, it should behave the way you want in the lua_fix build.

starbase64 · « **Reply #1467 on:** January 31, 2018, 12:29:07 PM »

Quote from: a1ex on January 31, 2018, 12:12:31 PM

@starbase64: if you are talking about the full-res silent picture, it should behave the way you want in the lua_fix build.

cool, THANKS

Danne · « **Reply #1468 on:** January 31, 2018, 12:33:53 PM »

On the 1100D:
With RAW video enabled

RAW video disabled

a1ex · « **Reply #1469 on:** January 31, 2018, 01:11:46 PM »

Looks fine. I've got another test for you to try.

Start from the allocate-raw-lv-buffer branch and apply this patch:

Code: [Select]

diff -r 23524a6ed2f9 src/config-defines.h
--- a/src/config-defines.h
+++ b/src/config-defines.h
@@ -47,3 +47,3 @@
     /** warning: it will slow down boot by a few seconds */
-    //~ #define CONFIG_MARK_UNUSED_MEMORY_AT_STARTUP
+    #define CONFIG_MARK_UNUSED_MEMORY_AT_STARTUP
 
diff -r 23524a6ed2f9 src/raw.c
--- a/src/raw.c
+++ b/src/raw.c
@@ -139,6 +139,6 @@
 #ifdef CONFIG_1100D
-#define CONFIG_ALLOCATE_RAW_LV_BUFFER
+#undef CONFIG_ALLOCATE_RAW_LV_BUFFER
 #define DEFAULT_RAW_BUFFER MEM(MEM(0x4C64))     /* how much do we have allocated? */
-#define DEFAULT_RAW_BUFFER_SIZE 0               /* is this really overwritten by other code? needs some investigation */
-#define RAW_LV_BUFFER_ALLOC_SIZE (3906*968)
+#define DEFAULT_RAW_BUFFER_SIZE 10*1024*1024    /* is this really overwritten by other code? needs some investigation */
+//#define RAW_LV_BUFFER_ALLOC_SIZE (3906*968)
 #endif
diff -r 23524a6ed2f9 src/mem.c
--- a/src/mem.c
+++ b/src/mem.c
@@ -927,4 +927,4 @@
 
-#define MEMORY_MAP_ADDRESS_TO_INDEX(p) ((int)CACHEABLE(p)/1024 * 720 / 512/1024)
-#define MEMORY_MAP_INDEX_TO_ADDRESS(i) ALIGN32((i) * 512 * 1024 / 720 * 1024)
+#define MEMORY_MAP_ADDRESS_TO_INDEX(p) ((int)CACHEABLE(p)/1024 * 720 / 128/1024)
+#define MEMORY_MAP_INDEX_TO_ADDRESS(i) ALIGN32((i) * 128 * 1024 / 720 * 1024)

Then take two screenshots of the Free Memory dialog:
- in movie mode, without any modules loaded
- in movie mode, with mlv_lite enabled (raw video active).

Danne · « **Reply #1470 on:** January 31, 2018, 02:19:22 PM »

nothing loaded

mlv_lite RAW video loaded

a1ex · « **Reply #1471 on:** January 31, 2018, 03:27:30 PM »

Wait a minute, that means the raw buffer is not overwritten by anything else. Then, where is all this allocation thingie coming from? Maybe back then it didn't work for some other reason, and we thought it's some sort of memory corruption? How do the same two screenshots look in x5 zoom?

(BTW, the screenshots are easier to compare if you don't add variable-sized white borders; try "convert foo.ppm foo.png")

Danne · « **Reply #1472 on:** January 31, 2018, 04:39:04 PM »

A couple of jpegs 5x zoom

Nothing loaded

loaded

a1ex · « **Reply #1473 on:** January 31, 2018, 06:29:19 PM »

That's clean, too - we may not even need this allocation stuff. Can you check the photo mode as well? This time, the two states would be:
- global draw off (camera started that way, not turned off afterwards)
- global draw on, with any raw-based overlay enabled (such as raw histogram, raw zebras or raw spotmeter)

Can you define DEFAULT_RAW_BUFFER_SIZE as 8MB (8*1024*1024 - from counting the pixels in the screenshots I get about 8.7), keep CONFIG_MARK_UNUSED_MEMORY_AT_STARTUP (so it would re-check this size, just in case) and try recording? This time, the assert should no longer get triggered.

If successful, can you try this configuration in all the video modes? (photo mode, video mode, x5 zoom, whatever else this camera has).

TODO: I'm not really sure I understand how this raw buffer allocation is done in Canon code, and whether the results are repeatable; will experiment on 60D and see what I get.

Danne · « **Reply #1474 on:** January 31, 2018, 07:59:16 PM »

In photo mode, liveview open:

global draw off

global draw on

I added this:

Quote

Can you define DEFAULT_RAW_BUFFER_SIZE as 8MB (8*1024*1024 - from counting the pixels in the screenshots I get about 8.7), keep CONFIG_MARK_UNUSED_MEMORY_AT_STARTUP (so it would re-check this size, just in case) and try recording? This time, the assert should no longer get triggered.

To my patched version from here:
https://www.magiclantern.fm/forum/index.php?topic=1009.msg196592#msg196592

And files comes out clean and nice without errors. But I don´t get errors with today´s raw_video_10bit_12bit either even without the fix. I had a slow card before which I believe corrupted files. 5x zoom still comes out with a fishy top border like described here:
https://www.magiclantern.fm/forum/index.php?topic=1009.msg196582#msg196582

Tets only done i 14bit. I was thinking I could try the same changes in raw_video_10bit_12bit but it will not compile:

Code: [Select]

../../src/raw.c:146:2: warning: #warning FIXME: using dummy DEFAULT_RAW_BUFFER_SIZE [-Wcpp]
 #warning FIXME: using dummy DEFAULT_RAW_BUFFER_SIZE
  ^
../../src/raw.c: In function 'raw_lv_realloc_buffer':
../../src/raw.c:544:39: error: 'DEFAULT_RAW_BUFFER' undeclared (first use in this function)
         if (raw_lv_buffer != (void *) DEFAULT_RAW_BUFFER)
                                       ^
../../src/raw.c:544:39: note: each undeclared identifier is reported only once for each function it appears in
make: *** [raw.o] Error 1
danands-MacBook-Pro:1100D.105 danne$

Anybody else tired of rescale and convert PPM files one by one by the way? ffmpeg does the trick:

Code: [Select]

for i in *.PPM ; do 
    ffmpeg -i "$i" -pix_fmt rgb24 -vf scale=500:-1 $(basename "${i/.PPM}").tif
done

News:

Author Topic: Canon 1100D / T3 (Read 1124432 times)

dfort