Author Topic: Apertus MLV Format Discussion  (Read 376 times)

Ilia3101

  • Contributor
  • Hero Member
  • *****
  • Posts: 582
Apertus MLV Format Discussion
« on: September 12, 2019, 03:34:37 AM »
As we know, the Apertus project will be making some improvements and extensions to the MLV format. To add features such as support for non linear raw data for example...

So this can be the place where Apertus and Magic Lantern can discuss any changes that will be made to the MLV format. Anyone can contribute or just follow the discussion.

I was talking to Fares on the Apertus Telegram group and we decided it is a good time to start a thread.

Quote from Fares:
Quote
we need to spend quality time with ML community and Axiom community to build next version of MLV to be generic as possible while maintaining the goal of the format -to be very computationally friendly on the camera side-
5D2, MLV App

Luther

  • Freshman
  • **
  • Posts: 89
Re: Apertus MLV Format Discussion
« Reply #1 on: September 12, 2019, 04:34:50 AM »
Apertus has some freedoms ML doesn't have. For example, using Zstandard instead of LJ92 could possibly offer smaller sizes, but would require FPGA programming, which ML can't do. Or expansion of the metadata to include more accurate color related informations (spectral data, for example). Or add Cooke /i metadata for cinema lenses.
So in order to maintain compatibility, Apertus will have to be limited to what Canon cameras can do. I don't think this is a wise step, as Apertus can go much further/better due to the nature of not being limited by any company/hardware...

Ilia3101

  • Contributor
  • Hero Member
  • *****
  • Posts: 582
Re: Apertus MLV Format Discussion
« Reply #2 on: September 12, 2019, 01:23:52 PM »
They are going to use LJ92. But I think it's ok to add stuff canon cameras can't do, as long as it stays optional. For example, the Axiom sensor can shoot non linear raw for extended dynamic range, and for that they want to add a block to describe the curve.

Or expansion of the metadata to include more accurate color related informations (spectral data, for example).

You have read my mind! This was actually going to be one of my main suggestions, I want expanded colour metadata. The current raw_info structure only has space for one camera matrix. I want there to be a better way of adding colour data like matrices, as well as a way to include spectral data for the sensor (optionally of course).
5D2, MLV App

Fares Mehanna

  • Just arrived
  • *
  • Posts: 1
Re: Apertus MLV Format Discussion
« Reply #3 on: September 13, 2019, 04:23:23 PM »
Thank you Ilia for opening the thread.

Sorry for my late replay. I was revise my knowledge about MLV.

I was working in the last 3 months in GSoC, I was working in implementing LJ92 in the FPGA for axiom beta and micro. I have a good knowledge about MLV during and before GSoC.

Luther is correct, Axiom can build its own hardware so possibilities are endless, but still our main priority is to be friendly as possible in the receiver end, since any heavy processing could limit the FPS.

Side info: currently, Axiom cameras do not store footage internally, so RAW data will be transferred and stored in a recorder. a computational friendly format will allow the recorder to use less power and to be more compact while handling more FPS.

Anyway, I will start by pointing to a few things Axiom currently need.

1. how bayer pattern is represented in the frame, for example if the sensor bayer pattern is RGGB, the data can be represented in the frame as
R G R G R G R G ...
G B G B G B G B ...
or
R G G B R G G B ...
R G G B R G G B ...
Currently Axiom frames are represented in the second style which is not supported in MLV.
 
2. HDR modes in Axiom beta include three modes
 - Even and odd lines with different exposure times - not ISO as ML do.
 - PLR mode which allow non-linear Raw data.
 - In the future Axiom can utilize its high fps and global shutter to merge several frames together with different gain or exposure times.

The current "mlv_diso_hdr_t" expect dual iso only, but we need more information in all the three modes.
You can check Supragya Raj work to understand more about PLR.

3. It is not currently used in the RAW pipeline but it might be used in the future, log encoded pixels instead of linear values, check the work done on this area here Log Curve Simulation Paper, it might be implemented as alpha, beta and gamma or as LUT.

I believe that is what needed to currently store the RAW data out of the camera.
but in the future I think we might need much more possible ways to describe data in the frame, like if we are storing each channel separately, or if we slice the frame horizontally or vertically into several slices. those tricks can help in the FPGA side.

Those are what in my mind right now, I will keep researching in possible areas that we might need and I will also post this thread in Apertus irc so interested developers could contribute.

using Zstandard instead of LJ92 could possibly offer smaller sizes, but would require FPGA programming

Interesting point, I was working in LJ92 since it is supported in MLV and DNG, and I did not really know about Zstandard, the concept of Asymmetric numeral systems is interesting. maybe someone in the future or me going to implement it (:

rexorcine

  • New to the forum
  • *
  • Posts: 26
Re: Apertus MLV Format Discussion
« Reply #4 on: September 13, 2019, 11:49:28 PM »
Is there anything anyone reading can think of that MLV should or shouldn't be doing besides what's already been described?
. . .

Ilia3101

  • Contributor
  • Hero Member
  • *****
  • Posts: 582
Re: Apertus MLV Format Discussion
« Reply #5 on: Yesterday at 02:04:09 AM »
Other than the non linear block, not much else I can think of myself. This thread should be useful if anything comes up though, I'm assuming there must be something.

And as MLV is getting upgrades, we may as well add a couple of other things. Such as better colour metadata - in the case of AXIOM where the sensor is upgradable and could be almost anything (in theory), so it is important to be able to specify colour info, at least a matrix or two.

And I think it would be nice to add the ability to include spectral data, as that may be available for some sensors too.

More from Fares:
Quote
Fares Mehanna, [12.09.19 00:11]
Hi Ilia. As far as I can remember the way you handle the color specs matrices currently is hardcoded in MLV App for canon cameras, so I think it is a good idea to add it to MLV format to generically handle different sensores without hardcode them.

Fares Mehanna, [12.09.19 00:14]
The other area that needed some work in MLV format is how the sensor data is arranged in the frame, for example Axiom store frame data as RGGB in the same line, and in the future it can encode every channel alone. so despite the fact that the sensor is RGGB pattern, the data in frame can be in different order than the normal  R G R G R G .... \n G B G B G B ...

Edit: his post has finally come through here!
5D2, MLV App