Messages

1

Raw Video Postprocessing / Re: MLV App 1.14 - All in one MLV Video Post Processing App [Windows, Mac and Linux]

« on: Today at 03:03:39 PM »

Cool, probably SSE4.1 then.  I think 64 bit builds should work fine for you so long as they don't include that.  And 32 bit OS *can* use SSE4.1 (and all other SSE), just not your core2duo because it is super old

Even before no feature used it in win32 version, just the compiler flag was enabled

The compiler will optimise using sse4.1 ops when it feels like it, can happen in any code.

The sophisticated fix would be something like: use CPUID to detect features at runtime, build multiple object files, some with sse4.1, some without, and swap function pointers / load different DLL or .so based on CPUID results.  Probably quite annoying work to do.

CPUs from about 2008 should have SSE4.1 and 4 is a big upgrade from prior versions, I expect you'll see noticeable perf drop with it disabled (have to test to be sure, of course).  How much do you want to support 15 year old hardware?

2

Raw Video Postprocessing / Re: MLV App 1.14 - All in one MLV Video Post Processing App [Windows, Mac and Linux]

« on: Today at 08:24:18 AM »

I don't know about 32 vs 64, but this is what I'm refering to, in platform/qt/MLVApp.pro:

Code: [Select]

QMAKE_CFLAGS += -O2 -fopenmp -msse4.1 -mssse3 -msse3 -msse2 -msse -D_FILE_OFFSET_BITS=64 -std=c99 -ftree-vectorize

I don't know Qt well at all, could be wrong.

3

Raw Video Postprocessing / Re: MLV App 1.14 - All in one MLV Video Post Processing App [Windows, Mac and Linux]

« on: Today at 07:23:34 AM »

Some Core2Duo don't have SSE4.1 support, looks like MLV App builds with that.  0xc000001d is illegal instruction, so it's plausible as a cause.

4

General Help Q&A / Re: 500D/60D are not complete turned off when camera is turned off

« on: Yesterday at 04:25:05 PM »

Some camera peripherals drain battery even when cam is "off"; I have a lens adapter that does this.  Sometimes, ML can crash on camera shutdown and still be using power.  I would expect this to be rare on 500D and 60D.

There is a test you can run that may help.  Turn cam off, remove card, remove battery, replace battery; leave cam off.  If the battery still drains, it is definitely not ML causing it.

5

General Chat / Re: Hardware reset/unbrick a 6D

« on: June 27, 2022, 03:18:01 AM »

That's a very strange looking firmware number.  How do you know it had that version?  It's a lot of numbers to remember - do you happen to have logs / photos showing the number?

Did you attempt to install ML at any point?  This may have saved a backup ROM that could be useful for recovery.

Most (all?) EOS DSLR have a serial port that can in some cases be used to reflash ROMs.  I don't know the location for the 6D, it may be on the forums somewhere already.

6

Camera-specific Development / Re: DIGIC 7 development (200D/SL2, 800D/T7i, 77D, 6D2)

« on: June 26, 2022, 06:18:47 AM »

Using MMU patches, I hooked mpu_send() and mpu_recv().  This allowed me to obtain a nice clean log of early MPU messages from 200D.  The same technique will be easy to adapt to all other D7, D8, DX cams.

This should greatly help in improving emulation.  So far I've only logged the very early messages, but in theory this can also log MPU traffic while the camera is exercised, to see how different camera functions work.  Note that the messages as seen by these functions are missing a container layer, so e.g. these two lines (one from my code, one from Qemu logging) are the equivalent message:

Code: [Select]

mpu_send: 06 01 27 00 64 00
[MPU] Received: 08 06 01 24 00 01 00 00  (PROP_CARD2_STATUS - spell #7)
Adding this extra layer for Qemu use looks easy.

The build is from this commit: https://github.com/reticulatedpines/magiclantern_simplified/commit/4d037140ad1c3ea72c16d508de65ddcc13f98a8d

Log (after slight manual cleanup, UART log lines get interlaced sometimes): https://pastebin.com/VmRdvQ9E

7

Raw Video Postprocessing / Re: MLV App 1.13 - All in one MLV Video Post Processing App [Windows, Mac and Linux]

« on: June 25, 2022, 07:26:37 PM »

Haha, yes, I was surprised by that one.  Maybe it was always called in the right order?  Then you'd set the later items wrong, but overwrite with correct values?

8

Raw Video Postprocessing / Re: MLV App 1.13 - All in one MLV Video Post Processing App [Windows, Mac and Linux]

« on: June 25, 2022, 09:50:53 AM »

Yes, all compilers like to complain about different things!  And I agree it can be tricky to find fixes that make all of them happy.

I see you fixed the missing breaks in that big switch statement, that seemed a real problem

9

Raw Video Postprocessing / Re: MLV App 1.13 - All in one MLV Video Post Processing App [Windows, Mac and Linux]

« on: June 24, 2022, 02:51:09 AM »

Made this PR for a tiny fix: https://github.com/ilia3101/MLV-App/pull/243

Are you interested in fixes around compiler warnings?  E.g., this one:

Code: [Select]

MainWindow.cpp: In member function ‘void MainWindow::startExportPipe(QString)’:
../../src/mlv/macros.h:84:41: warning: macro expands to multiple statements [-Wmultistatement-macros]
   84 | #define setMlvDontAlwaysUseAmaze(video) (video)->use_amaze = 0; (video)->current_cached_frame_active = 0
      |                                         ^
MainWindow.cpp:1788:54: note: in expansion of macro ‘setMlvDontAlwaysUseAmaze’
 1788 |         if( !ui->actionAlwaysUseAMaZE->isChecked() ) setMlvDontAlwaysUseAmaze( m_pMlvObject );
      |                                                      ^~~~~~~~~~~~~~~~~~~~~~~~
MainWindow.cpp:1788:9: note: some parts of macro expansion are not guarded by this ‘if’ clause
 1788 |         if( !ui->actionAlwaysUseAMaZE->isChecked() ) setMlvDontAlwaysUseAmaze( m_pMlvObject );

The macro will expand to look like this:

Code: [Select]

if( !ui->actionAlwaysUseAMaZE->isChecked() )
{
    m_pMlvObject->use_amaze = 0;
}
m_pMlvObject->current_cached_frame_active = 0;

This feels like a bug to me?  Presumably you want both statements to execute only if the condition is met? I would recommend converting the macro into a function.  It looks like a function, and performs the job of a function.  Just make it be a function.

I don't know if you have some normal process for checking and fixing compiler warnings so I haven't spent any time on this - but there are quite a few compiler warnings that looks like bugs.

10

Raw Video Postprocessing / Re: MLV App 1.13 - All in one MLV Video Post Processing App [Windows, Mac and Linux]

« on: June 19, 2022, 07:42:47 PM »

Q:Does compiling with MinGW_32 makes MLVApp version x32 too? and same for MinGW_64, makes MLVApp x64?

Depends on build system.  Both compilers should be able to make 32 and 64 bit output files.  You can inspect the exe or the running process to find out what they've done.  Task Manager, should be a Platform column (maybe not visible by default?).  A 32 bit process will be limited to 2GB mem so this can be quite relevant.  Do you see the same crashes via WSL?

I saw the leaks from a 64 bit mlvapp (but on Linux).

Can you share a session file with me?  I can edit to use my MLV files.  If that doesn't repro then sharing your clips might help, hopefully not needed (exporting a 3 min clip takes several hours so I would like to avoid this!).

11

Raw Video Postprocessing / Re: MLV App 1.13 - All in one MLV Video Post Processing App [Windows, Mac and Linux]

« on: June 19, 2022, 07:36:34 PM »

I ran a non-dark frame export with a bunch of different options turned on and didn't get anything suspicious.  These tests take about 20 minutes to export a 2s clip, so I can't be bothered doing them if there's not a decent chance it will find something.  If you have something fairly reproducible and can share e.g. a session file so I can copy it, I'm quite happy to try it.

Since you're on Mac, Valgrind isn't well supported (it used to work okay, Apple broke it with Big Sur).  ASAN via clang does some of the same things, I strongly recommend you try it (introduce some buffer overflow bugs to test it in action).  According to Stack Overflow, Apple clang doesn't support leak checking via ASAN, so you'll want to get llvm from Brew or similar.  Then you need to change build options to include "-fsanitize=address", and when running, use "ASAN_OPTIONS=detect_leaks=1".  ASAN is faster than Valgrind, but not as thorough.

I've got two small fixes to current code that are unrelated to export, I'll PR them later.

12

Camera-specific Development / Re: Canon 5D Mark IV

« on: June 19, 2022, 07:20:44 PM »

Hi everyone! I have a 5DIV and I'm a software eng. How can I help? Really want ML on my 5DIV

Hi - somehow I missed this message.  If you're still interested, we have a testable build (with very limited features) for 5D4, but it crashes early on.  Probably, you will want to get UART access to your cam (not hard for this model) in order to debug.  No active devs have access to a 5D4 with UART.

It should look like this, under the thumb grip:
https://www.magiclantern.fm/forum/index.php?topic=7531.msg224730#msg224730

Some cams have the socket soldered, some don't.  We have Gerber files for ordering a flexible PCB to connect to the socket.  We know the socket part if yours doesn't have it and you want to add it.  Otherwise, you can use whatever probes work for you to contact the right pin.  You only need one pin connected to get debug output.

13

Raw Video Postprocessing / Re: MLV App 1.13 - All in one MLV Video Post Processing App [Windows, Mac and Linux]

« on: June 19, 2022, 05:14:27 PM »

Nice, much better:

Code: [Select]

==1745753== LEAK SUMMARY:
==1745753==    definitely lost: 7,140 bytes in 32 blocks
==1745753==    indirectly lost: 116,660 bytes in 156 blocks
==1745753==      possibly lost: 7,961,728 bytes in 30 blocks

There were several other leaks besides the big one I listed before, these are also fixed - makes sense, you free a bunch of related things in the latest change.

Most of the remaining leaks are a single block per size.  Often that means you created something once at the start and never free it.  That's fine if you want it to exist until the program exits.  It's nicer to explicitly free on exit, just so leak checkers don't FP on it.  Not important beyond that.

Bilal, does that change stop your crash?  This commit: https://github.com/ilia3101/MLV-App/commit/faddb3e1b5a1cec8c73b85252728fe031b6b23d3

Oh yeah, re ffmpeg, I am only doing make, not make install.  So perhaps it's not expected that ffmpeg gets copied for me?

14

Raw Video Postprocessing / Re: MLV App 1.13 - All in one MLV Video Post Processing App [Windows, Mac and Linux]

« on: June 19, 2022, 03:55:08 PM »

Cool, glad if it seems helpful testing.  Bilal gave nice repro instructions so I was trying to do that for dark frame export, yes.  Haven't tried anything else yet.

Yes, high memory usage via "top".  Not very scientific.

Your change seems the right kind of thing to me, I tried a quick hack addition of freeing raw_rgb_current_frame as part of dl_free() but that seg faulted, so I didn't mention it.  I didn't know where cleanup should live   Testing now.

15

Raw Video Postprocessing / Re: MLV App 1.13 - All in one MLV Video Post Processing App [Windows, Mac and Linux]

« on: June 19, 2022, 02:52:26 AM »

I made a much shorter clip, hoping it would still show the leak.  It did, and ran faster.

Code: [Select]

==1685697== 182,476,800 bytes in 55 blocks are definitely lost in loss record 511 of 511
==1685697==    at 0x483877F: malloc (vg_replace_malloc.c:307)
==1685697==    by 0x1DC83C: openMlvClip (video_mlv.c:1876)
==1685697==    by 0x2114A1: df_load_ext (darkframe.c:57)
==1685697==    by 0x1DFE3C: applyLLRawProcObject (llrawproc.c:172)
==1685697==    by 0x1D9C30: getMlvRawFrameFloat (video_mlv.c:308)
==1685697==    by 0x1D879A: get_mlv_raw_frame_debayered (frame_caching.c:305)
==1685697==    by 0x1DA0C5: getMlvRawFrameDebayered (video_mlv.c:439)
==1685697==    by 0x1DA190: getMlvProcessedFrame16 (video_mlv.c:465)
==1685697==    by 0x160B8F: MainWindow::startExportPipe(QString) (MainWindow.cpp:2572)
==1685697==    by 0x170093: MainWindow::exportHandler() (MainWindow.cpp:8128)
==1685697==    by 0x17D688: MainWindow::on_actionExport_triggered() (MainWindow.cpp:6596)
==1685697==    by 0x26D5C2: MainWindow::qt_metacall(QMetaObject::Call, int, void**) (moc_MainWindow.cpp:1764)

Valgrind thinks the allocation to rgb_raw_current_frame is not always being freed, and because "definitely" lost, we are getting to a state where there are no references to that block of mem.  That suggests we overwrite the pointer.

Code: [Select]

1874     /* For frame cache */
1875     video->rgb_raw_frames = (uint16_t **)malloc( sizeof(uint16_t *) * video->frames );
1876     video->rgb_raw_current_frame = (uint16_t *)malloc( getMlvWidth(video) * getMlvHeight(video) * 3 * sizeof(uint16_t) );
1877     video->cached_frames = (uint8_t *)calloc( sizeof(uint8_t), video->frames );

Hacked in some quick printf debugging around alloc free of rgb_raw_current_frame and got this:

Code: [Select]

initMlvObject hit
rgb_raw_current_frame alloc'd
freeMlvObject hit
rgb_raw_current_frame free'd
rgb_raw_current_frame alloc'd
rgb_raw_current_frame alloc'd
rgb_raw_current_frame alloc'd
rgb_raw_current_frame alloc'd
rgb_raw_current_frame alloc'd
rgb_raw_current_frame alloc'd
rgb_raw_current_frame alloc'd
rgb_raw_current_frame alloc'd
rgb_raw_current_frame alloc'd
rgb_raw_current_frame alloc'd
rgb_raw_current_frame alloc'd
rgb_raw_current_frame alloc'd
rgb_raw_current_frame alloc'd

That's truncated a lot.  Looks like it allocates to rgb_raw_current_frame every frame that's exported, and never frees them.

16

Raw Video Postprocessing / Re: MLV App 1.13 - All in one MLV Video Post Processing App [Windows, Mac and Linux]

« on: June 19, 2022, 01:28:50 AM »

Probably a mem leak.  It took over two hours to run the test:

Code: [Select]

==1650919== HEAP SUMMARY:
==1650919==     in use at exit: 2,168,469,999 bytes in 10,637 blocks
==1650919==   total heap usage: 3,537,763 allocs, 3,527,126 frees, 62,845,029,066 bytes allocated
==1650919==
==1650919== LEAK SUMMARY:
==1650919==    definitely lost: 1,769,939,527 bytes in 4,387 blocks

Now I need to run it again with more logging, which will make it take longer.

17

Raw Video Postprocessing / Re: MLV App 1.13 - All in one MLV Video Post Processing App [Windows, Mac and Linux]

« on: June 18, 2022, 10:44:44 PM »

Valgrind has found one likely error so far, conceivably related to the dark subtraction issue, though I'd guess probably not.  It's an easy fix in code to test.

Here, in dng.c, we ROR32 over a pointer into a uint16 buffer.  This can read 2 bytes past the end of buffer (this *probably* won't crash on Windows, which typically has readable bytes after the allocated space on the heap.  I think if you exactly hit the end of a page boundary maybe it doesn't).  Probably we should check the buffer is 4 byte aligned earlier on?  Sometimes these type of errors are FPs by valgrind when it doesn't understand the asm for the function, but ROR32 over 16 bit buffer feels likely to be real to me.

Code: [Select]

703         uint32_t uncorrected_data = *((uint32_t *)&packed_bits[bits_address]);
 704         uint32_t data = ROR32(uncorrected_data, rotate_value);

Valgrind dump so people unfamiliar can see how useful it is:

Code: [Select]

==1650919== Thread 35:
==1650919== Invalid read of size 4
==1650919==    at 0x1FC885: dng_unpack_image_bits._omp_fn.0 (dng.c:704)
==1650919==    by 0x4890DE5: ??? (in /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0)
==1650919==    by 0x5DEFEA6: start_thread (pthread_create.c:477)
==1650919==    by 0x6231DEE: clone (clone.S:95)
==1650919==  Address 0xc676e0e is 967,678 bytes inside a block of size 967,680 alloc'd
==1650919==    at 0x483AB65: calloc (vg_replace_malloc.c:760)
==1650919==    by 0x2114F1: df_load_ext (darkframe.c:102)
==1650919==    by 0x2119BC: df_validate (darkframe.c:256)
==1650919==    by 0x15A5E2: MainWindow::on_lineEditDarkFrameFile_textChanged(QString const&) (MainWindow.cpp:9728)
==1650919==    by 0x26D5C2: MainWindow::qt_metacall(QMetaObject::Call, int, void**) (moc_MainWindow.cpp:1764)

Shows you which buffer was used badly, including where it was allocated.  Learn to use valgrind if you're debugging C or C++! (works on anything but especially useful in these languages).

18

Raw Video Postprocessing / Re: MLV App 1.13 - All in one MLV Video Post Processing App [Windows, Mac and Linux]

« on: June 18, 2022, 09:32:31 PM »

I don't have a Windows machine to test on.  Swap is disk space reserved to swap memory to if ram is exhausted.

Currently I am trying to run export under valgrind, but it might not be practical, it's so slow

19

Raw Video Postprocessing / Re: MLV App 1.13 - All in one MLV Video Post Processing App [Windows, Mac and Linux]

« on: June 18, 2022, 09:05:06 PM »

Top during dark frame subtraction export:

Code: [Select]

1395499 username       20   0 1710964 267988  22812 R 888.7   0.8  51:24.36 ffmpeg                                                                                 
1395388 username       20   0   18.0g  15.2g  53684 R 324.6  48.5  18:32.42 mlvapp

The amount of reserved mem steadily increases through the export.  Peaked at 20GB.  No crash here, this machine is fat.  It would surely crash if you had less ram + swap.  I would expect it's easy to observe memory going up during the export - should it be doing this?  Exported file is 7.7GB.  Maybe we're keeping a reference to each frame, something like that?  So they don't get garbage collected during the export?  Will dig a bit deeper.

20

Raw Video Postprocessing / Re: MLV App 1.13 - All in one MLV Video Post Processing App [Windows, Mac and Linux]

« on: June 18, 2022, 09:00:05 PM »

Masc - thanks for the reply.  It built nice and easily.  Had to fiddle around a bit with config (never used it before), am now trying to repro dark subtraction crash.

Only complaint so far: it expects ffmpeg binary to be in the same dir as mlvapp, and the error message isn't very good if it's missing "encoder ffmpeg missing".  I have ffmpeg in system path, but not next to mlvapp.  Had to use strace to realise it was opening with AT_FDCWD, and then copy the binary into my build location.  Is this deliberate?  Maybe people need to use custom ffmpeg versions sometimes?  Maybe it's fixed by a proper install (I just ran make and started mlvapp from there).  It would be nicer for me if when ffmpeg is not found, it tried to use the one in system path.  Maybe that would be a bad default for other people, I don't know.

21

Raw Video Postprocessing / Re: MLV App 1.13 - All in one MLV Video Post Processing App [Windows, Mac and Linux]

« on: June 18, 2022, 10:34:05 AM »

Okay, probably not a leak using all your allowed process memory...  I'm not great at diagnosing real software on Windows, just the stupid stuff malware does.  I can try and look for problems in the Linux version, they might apply on both.  The behaviour of malloc failing being persistent seems odd to me unless all mem is exhausted.  Did you try an ASAN build?

Is this the right place to get current version?
https://github.com/ilia3101/MLV-App

22

Raw Video Postprocessing / Re: MLV App 1.13 - All in one MLV Video Post Processing App [Windows, Mac and Linux]

« on: June 17, 2022, 05:03:00 PM »

Do you free all memory as you go?  Any chance of memory leaks?

You really should check the return value from malloc / calloc in all cases, before using the memory.  You could do this by wrapping / redefining malloc, if you don't want to change every call site.  You can attempt to retry, but generally you are screwed if malloc fails (and some systems, e.g. default Linux, will have malloc never fail at point of allocation - the failure will occur only when you attempt to use the obtained pointer).  You can at least fail gracefully with more information (at least on platforms that have a malloc that will ever fail).

Because Linux has "optimistic" malloc behaviour, and you only see this behaviour on Windows, I wonder if you are sometimes allocating very large amounts, and not using them?  You could log the peak size allocated (some static global, wrap malloc with a logging call and output the peak value as you go).

You could try building with ASAN, that may give you more information (including memory leaks).  You could measure peak memory usage - Windows probably has some per process memory limit which you could be hitting even though the system has memory left.

23

Reverse Engineering / Re: UHS-I / SD cards investigation

« on: June 12, 2022, 08:24:56 PM »

What would demonstrate support for (or lack of) DebugMsg in Qemu 4?

I'd never expect emulated hardware to match benchmark numbers on a physical cam.  What are you trying to check for?

24

Reverse Engineering / Re: UHS-I / SD cards investigation

« on: June 12, 2022, 08:01:37 PM »

Qemu 4 also looks to have UHS-I support.

25

Feature Requests / Re: 1190 x 2800

« on: June 04, 2022, 10:28:53 PM »

This looks how I'd expect, and still shows rolling shutter.  They've chosen scenes with fast horizontal pans, but no vertical pans.  Distortion of objects is still visible, but it's a different kind of distortion.

It might be nice to have this as an option within ML, it needs someone to mess around with registers to get a portrait resolution.