Messages

Dual ISO now works on 200D.  So that's nice.  Process is similar to other cams, but the data format for ADTG / CMOS registers appears different, here are some examples:

Code Select Expand


0b400000 // ISO 100
0b444440 // 1600
0b440400 // Dual ISO, 100/1600
0b455550 // 3200
0b466660 // 6400
0b466660 // 12800, so presumably this is boosted further elsewhere and is non-native ISO


I'm currently assuming this means the data is now 16 bit, and b4 is the register index.  The lower and upper 4 bits are used by some commands.

It would contain image data and would be displayed.  But I'm happy if you don't want to do this.  Just buy some cloud storage, really.  It'll be $10 a month or less and won't be randomly removed in the future like this Photos offer, and you can simply upload MLV without any of this fuss.

It's exactly what your current plan is: store the content of a filetype that's not covered, by splitting the data into multiple files that are covered.  You're only "storing images".  Both are equally against the spirit.

You've misunderstood; you don't need to know anything about the internals of MLV for this approach.  Just slice it up so each piece is small enough to fit in the image data of a DNG (or any raw format Photos supports).

Any tips on how I can get started on implementing this? This type of software development is pretty far out of my wheel house

Any file can be considered a long list of numbers.  The image data in a raw file is also a long list of numbers (the pixel data).  Chop up the file you care about into the correct sized pieces, and copy the numbers into the raw (in the correct place for it to be the "image data").  Repeat until you run out of source file.  Name each generated raw file sequentially or whatever.

Somebody always does something like this when there's an "unlimited" storage deal going, and the deal always gets shut down in a year or two (especially when somebody starts using it for storage of unsavory content).  Amazon Photos may last longer since it's not free.

EDIT: in case it's not clear - I am recommending you don't do this.  Just buy some cloud storage, it'll be much less hassle and won't get shut down in the future randomly.

Since your game plan is to abuse Amazon Photos to store video, and you already expect to do work after downloading the "photos" to reconstruct the videos *and* since Amazon Photos allows storing raw images with no alterations:

Chunk the video up into pieces and store each piece as b&w data in a series of raw images.  Perfect reproduction of any file.  Easy to automate.

crop_rec_4k_mlv_snd is very close to lua_fix, lua_fix was merged into it "only" 2 years ago:
https://foss.heptapod.net/magic-lantern/magic-lantern/-/network/branch%2Fcrop_rec_4k_mlv_snd
https://foss.heptapod.net/magic-lantern/magic-lantern/-/compare/branch%2Fcrop_rec_4k_mlv_snd...branch%2Flua_fix?from_project_id=321

Merging these would be easy, so merging to mine shouldn't be too hard; I've already done the hard lua_fix + unified merge.

But that's enough cluttering up this SD card thread!  Let me know if you get the merge working, and I'll let you know if I do

Sure, I can do tests later, but I am not very interested in unified branch which is the main one you are using in magiclantern_simplified.

Nope - unified is too old.  Mine is unified + lua_fix + qemu + some other ones (I forget...  there are so many open branches on official, I wanted to get away from that).  I want one long-lived branch only (and stable / release branches, no work happens there).  Other branches are for feature work only, before merging into dev (the main branch).  I nuked a lot of history because it was too confusing to understand, hundreds of branches.  I made that magiclantern_hg_02 repo to keep all history from all branches preserved in case it was needed.

I think lua_fix is similar to unified.

I don't really know, because I don't know of any documentation for what the different branches are for (other than commit messages), but in practice, lua_fix was the dev branch for several years.  It has many changes that are unrelated to lua.  I think the intention was to merge to unified, but this never happened.

I think we need to import crop_rec_4k_mlv_snd in magiclantern_simplified repo, then we need to find a way to merge dev branch and crop_rec_4k_mlv_snd correctly, but I think these two branches have a lot of differences, things might break, if that works I have no problem to work in your main branch.

I can look at this and see how different they are.

For now I think the best way is to stay in my fork of magiclantern_hg_02, and port the new D45 crop mode stuff in one place, and probably merging other branches if it make sense, then importing the updated crop_rec_4k_mlv_snd to magiclantern_simplified then dealing with merging it to e.g. dev branch (unless if you have other plans).

That would also work for me.  I warn you that the different official branches are painful to merge.  I had to inspect several 40k line diffs.  Fun.  Make sure you have some free time

[actually]

Anyway, I said before I would rewrite my 650D/700D/100D code in a clean way and make the repo has native git support (no needs for hg / mercurial), and actually re-writing in progress and it might take some time:

https://github.com/bilalfakhouri/magiclantern_hg_02/commits/crop_rec_4k_mlv_snd

My goal to include the latest achievements for all DIGIC 5 models in a clean way in one branch (includes 100D/650D/700D/5D3/EOS M and probably 6D and 70D). Beside a basic support for DIGIC 4 models.

My main ML repo has git support already.  The commits that do that work are here:
https://github.com/reticulatedpines/magiclantern_simplified/commit/4a29a347f29276d4bec0876412cec4bb12b5a28c
https://github.com/reticulatedpines/magiclantern_simplified/commit/3163d41cee35fb54b4308e1f32318ff87de53645
(I think there might be a third commit near that time, too.  But that should be most of it).

In theory, that repo should work on Digic 4 and 5 cams.  It's been tested, a little...  I'd be interested to learn if your modules work with it - but test in qemu first!  You'll need to copy your module code to my repo and build there, because I made changes to how modules are compiled.  The official repo has a serious bug that can make modules jump to incorrect address when doing calls, triggering bad crashes.

Do you know if crop_rec_4k_mlv_snd is very different from lua_fix?  My repo has lua_fix already (and some other branches, merged together), so you may find it's not that hard to merge your changes into mine directly.  You're welcome to take the hg removal commits whatever you choose.  I think even better would be to recognise if the repo is hg *or* git, and work correctly for both.  It's nice to have the commit tag in the build info screen, which mine currently doesn't do.

Ah well, thanks for considering it

Reviewing can be quick, but can be slow.  The cache hack change is small and easy, I think an hour or less.  The MMU work is larger and more complicated, several hours minimum, so might take a few days of spare time.

Sounds like I will have to more carefully consider my own code - at least it's good to know.

Cool, nice proof of concept.  SD frequency is a better name.  I like it being part of benchmarking generally, unrelated to sd_uhs module.

We probably want to check if the reg addresses are the same for D678X, if they're not it will get a bit more complicated (either use is_camera(), or a getter function in ML, so different generations can read from different locations?).

I think naming them as the addresses is quite confusing, maybe reg_C0400614 style?  sd_frequency should probably be static, no need to make it global.  I don't see any point in making the reg reads volatile, they're only used once.

Thanks, nice summary.  I agree, nicer to read it back from the cam, to detect what is really being used, not what we think should be in use.

Is this as simple as reading back the ADTG regs, does that give us the correct info?

(I've seen some screenshots now and Mode and Global Draw are already listed, I didn't realise that)

Makes sense to me.  Mode and global draw are easy I think.  OC freq I don't know how to query, I haven't worked on that code at all.  Read back register values I guess?  Or maybe there's a Canon API?  I'd need someone to show me where the OC setting code is.

Hello!  It's hard to find people with the ML knowledge to do code reviews, so perhaps devs on here would like to arrange something?  We could swap reviews for changes.

I have two recent changes that I'd really appreciate a review for:
- module changes that make cache hack usage compatible with Digic 678X (and fix a bug in the build system that affects all ML repos): https://github.com/reticulatedpines/magiclantern_simplified/compare/dev...d45_cache_fix
- working but minimal framework for MMU memory patches on Digic 678X cams: https://github.com/reticulatedpines/magiclantern_simplified/compare/dev...mmu_investigation

The MMU commits are kind of ugly and need cleaning up before merging, I'm not asking for a full review there but some validation that the general approach is sound.  Testing on cams isn't required, although the D45 branch should make builds that work on older cams (this has had some testing: thanks Walter!).

I'll review your weird hacks in exchange!

This is lucky imaging, yes?  Raw image quality will be quite a bit higher, but if you have to lower fps to 30 you may get more atmospheric disturbance.  I think you'll have to try it and see.

If you can upgrade to a 650d you should be able to do 640x640 at 50fps.  With lossless compression 800 something, with SD overclocking perhaps as high as 1280x1280 60fps.  I am sure the 650d would be much less than your scope and mount, plus you'd get some back if you sell the 60d.

Cool, probably SSE4.1 then.  I think 64 bit builds should work fine for you so long as they don't include that.  And 32 bit OS *can* use SSE4.1 (and all other SSE), just not your core2duo because it is super old

Even before no feature used it in win32 version, just the compiler flag was enabled

The compiler will optimise using sse4.1 ops when it feels like it, can happen in any code.

The sophisticated fix would be something like: use CPUID to detect features at runtime, build multiple object files, some with sse4.1, some without, and swap function pointers / load different DLL or .so based on CPUID results.  Probably quite annoying work to do.

CPUs from about 2008 should have SSE4.1 and 4 is a big upgrade from prior versions, I expect you'll see noticeable perf drop with it disabled (have to test to be sure, of course).  How much do you want to support 15 year old hardware?

I don't know about 32 vs 64, but this is what I'm refering to, in platform/qt/MLVApp.pro:

Code Select Expand


QMAKE_CFLAGS += -O2 -fopenmp -msse4.1 -mssse3 -msse3 -msse2 -msse -D_FILE_OFFSET_BITS=64 -std=c99 -ftree-vectorize


I don't know Qt well at all, could be wrong.

Some Core2Duo don't have SSE4.1 support, looks like MLV App builds with that.  0xc000001d is illegal instruction, so it's plausible as a cause.

Some camera peripherals drain battery even when cam is "off"; I have a lens adapter that does this.  Sometimes, ML can crash on camera shutdown and still be using power.  I would expect this to be rare on 500D and 60D.

There is a test you can run that may help.  Turn cam off, remove card, remove battery, replace battery; leave cam off.  If the battery still drains, it is definitely not ML causing it.

That's a very strange looking firmware number.  How do you know it had that version?  It's a lot of numbers to remember - do you happen to have logs / photos showing the number?

Did you attempt to install ML at any point?  This may have saved a backup ROM that could be useful for recovery.

Most (all?) EOS DSLR have a serial port that can in some cases be used to reflash ROMs.  I don't know the location for the 6D, it may be on the forums somewhere already.

Using MMU patches, I hooked mpu_send() and mpu_recv().  This allowed me to obtain a nice clean log of early MPU messages from 200D.  The same technique will be easy to adapt to all other D7, D8, DX cams.

This should greatly help in improving emulation.  So far I've only logged the very early messages, but in theory this can also log MPU traffic while the camera is exercised, to see how different camera functions work.  Note that the messages as seen by these functions are missing a container layer, so e.g. these two lines (one from my code, one from Qemu logging) are the equivalent message:

Code Select Expand


mpu_send: 06 01 27 00 64 00
[MPU] Received: 08 06 01 24 00 01 00 00  (PROP_CARD2_STATUS - spell #7)

Adding this extra layer for Qemu use looks easy.

The build is from this commit: https://github.com/reticulatedpines/magiclantern_simplified/commit/4d037140ad1c3ea72c16d508de65ddcc13f98a8d

Log (after slight manual cleanup, UART log lines get interlaced sometimes): https://pastebin.com/VmRdvQ9E

Haha, yes, I was surprised by that one.  Maybe it was always called in the right order?  Then you'd set the later items wrong, but overwrite with correct values?

Yes, all compilers like to complain about different things!  And I agree it can be tricky to find fixes that make all of them happy.

I see you fixed the missing breaks in that big switch statement, that seemed a real problem

Made this PR for a tiny fix: https://github.com/ilia3101/MLV-App/pull/243

Are you interested in fixes around compiler warnings?  E.g., this one:

Code Select Expand


MainWindow.cpp: In member function 'void MainWindow::startExportPipe(QString)':
../../src/mlv/macros.h:84:41: warning: macro expands to multiple statements [-Wmultistatement-macros]
   84 | #define setMlvDontAlwaysUseAmaze(video) (video)->use_amaze = 0; (video)->current_cached_frame_active = 0
      |                                         ^
MainWindow.cpp:1788:54: note: in expansion of macro 'setMlvDontAlwaysUseAmaze'
1788 |         if( !ui->actionAlwaysUseAMaZE->isChecked() ) setMlvDontAlwaysUseAmaze( m_pMlvObject );
      |                                                      ^~~~~~~~~~~~~~~~~~~~~~~~
MainWindow.cpp:1788:9: note: some parts of macro expansion are not guarded by this 'if' clause
1788 |         if( !ui->actionAlwaysUseAMaZE->isChecked() ) setMlvDontAlwaysUseAmaze( m_pMlvObject );


The macro will expand to look like this:

Code Select Expand


if( !ui->actionAlwaysUseAMaZE->isChecked() )
{
    m_pMlvObject->use_amaze = 0;
}
m_pMlvObject->current_cached_frame_active = 0;


This feels like a bug to me?  Presumably you want both statements to execute only if the condition is met? I would recommend converting the macro into a function.  It looks like a function, and performs the job of a function.  Just make it be a function.

I don't know if you have some normal process for checking and fixing compiler warnings so I haven't spent any time on this - but there are quite a few compiler warnings that looks like bugs.

Q:Does compiling with MinGW_32 makes MLVApp version x32 too? and same for MinGW_64, makes MLVApp x64?

Depends on build system.  Both compilers should be able to make 32 and 64 bit output files.  You can inspect the exe or the running process to find out what they've done.  Task Manager, should be a Platform column (maybe not visible by default?).  A 32 bit process will be limited to 2GB mem so this can be quite relevant.  Do you see the same crashes via WSL?

I saw the leaks from a 64 bit mlvapp (but on Linux).

Can you share a session file with me?  I can edit to use my MLV files.  If that doesn't repro then sharing your clips might help, hopefully not needed (exporting a 3 min clip takes several hours so I would like to avoid this!).