Messages

I have got the feeling that the SD speed looks a bit low.

[at once]

two buffers wont give us any improvement imho.
to be correct, we already have n buffers, where n is up to 70 or so on 5D3.
and these buffers get assigned dynamically to CF or SD.


i know we can do another improvement.
right now i am playing with two buffering methods on both queueing and dequeueing.

some problem that is making it hard to get optimal results, is the non-censecutiveness of memory buffers.
we can allocate e.g. memory for 70 frames. but this memory is fragmented, it it not continuous.
this means we have some groups of 2, 8, 20 or any other number of frames (we call it slots btw) that are continuous.
but the best for writing speed is to write a consecutive block of nearly 32MiB at once.

buffer layout e.g.: [____] [__] [_____________] [________] [__]

queueing:
method a) pick the first available free buffer
this method walks through the list of slots and checks for a free one.
it does not care if the previous or next buffers are directly behind the free one.

method b) use alex' algorithm that tries to place frames so that we get the largest block with frames in order.
this was necessary for the old writer where the frames had to be in order

better method:
search for the largest free space and try to fill it up.
strategy behind: build the largest block with frames in any order so that we get highest transfer speed

buffer layout e.g.: [____] [__] [______x1234x___] [________] [__]
the fifth frame would get enqueued at x, even if there is a lot of other free space


dequeueing:
method a) there are two writer threads (CF and SD) that are listening on a queue.
dispatcher thread scans the buffers for the largest consecutive block and places a write job into the queue.
the next "free" thread receives this job and writes it, no matter if SD or CF.

method b) again two threads, but every thread has its own queue.
the dispatcher places the largest found block in the queue for CF, the faster device, and the smallest block in the queue for SD.
strategy: try to keep the CF thread busy with the largest blocks to achieve the highest write rate. SD doesnt suffer that much
from writing smaller blocks. (remember, ~32MiB writes give the fastest possible transfer rate. on SD the speed difference isnt that high with smaller blocks)

i think mehtod b) gives the best results.

but to get back to your idea. one possible improvement could be this:
make the dispatcher aware that it should only qeueue writes to SD from large blocks, if there is nothing else.
background: we dont want the slow SD to permanently disturb the queueing by building large blocks.

[____] [12] [3________] [________] [__]

in this case the SD algorithm will pick frame 3.
this could lead to such fragmentation:
[____] [12] [_____789_] [________] [__]

but CF would benefit from writing large blocks so SD would probably be just annoying.

so i am thinking of building a sorted list of buffers and their sizes.
SD will always try to clear the smallest buffer areas, where CF tries to clear the larger ones.

There should be an arbitration system between CF write thread and SD write thread, I guess, and, possibly, a powerful one.

A sorting list should be a good idea. My wild thought is that it might be good to keep a list of RAW file's address, and let the CF/SD write threads to sort according the list. The question is how much RAW files they can fetch each time by each of them -- then there is a need of arbitration system.

I did some quick tests in my garden with silent pics burst this afternoon.
https://vimeo.com/71649487

The SilentPic Mode is beautiful, knocking H.264 out, but only it stops after 1-2 seconds, I really hope to get the jpg output from LV back as the Alpha2 used to do, but couldn't find the setting everywhere I looked. May somebody show me the way?

i am wondering that it might be faster to create two different buffers for the twoslots or even two buffers to write to the same slot. just a mere thought on the write speed

Comparing with SSD, CF card prices are ridiculously high, can I understand that CF cards are more difficult or more expensive to produce?

britom, thanks. I finally got it loaded.

arrinkiiii, thanks for solving the red light blinking issue.

Unable to boot twice, solved the issue easily by pulling the battery out, probably because I hot-pulled the CF card.

But the Qscale factor shrinks to -8 (used to be -16 plus GOP1), or it will stop recording in several seconds. Still unable to save that "Video Hack" option. But it is good enough for now.

Thanks again. G3gg0, Audionut, Pelican, britom, arrinkiiii, ...

I have no luck with Audionut's latest build @reply 721 neither.

I guess that I am one of those "luna"s.  I load Audionut's autoexec.bin with Alpha 2 into 7D, nothing happens. I mean really nothing, the ML didn't load into camera. I download everything in Audionut's 7D build directory and put them all in CF card's root directory. And start up the camera again. Nothing happens, again. I don't visually see that my camera is damaged neither.

The UDMA interface has restricted the bandwidth, I believe.

I remember somewhere that I have seen someone written a module for output through HDMI. But the HDMI recorder must present itself to be an external drive to the camera also.

I think the only possible solution for fuller resolution RAW video is to write to an external drive, if the raw would not be compressed.

Awesome. It looks like a container to me. Though, I am curious how the file names are generated, because there might be multiple mlv files generated in the camera simultaneously, and written to SD/CF card/cards simultaneously and randomly, as far as I can understand, so the file might end up overwriting each other -- maybe I worried too much.

I was able to download the full-res dng file, and open it with GIMP 2.8, and didn't find much artifacts, and, when shrinked to 1920x1280, the picture was beautiful, but I couldn't get that much DR as the whitepaper did.

Couldn't wait to see when this good thing, plus raw video, plus 10 bit downbit raw video, will come to 7D. And I had a dream last night shooting that Dual-raw video with 7D. This is serious stuff.

I personally appreciate Alex's great job and his daring thought, which I think quite fun, and the whitepaper is really well written(though a little beyond me). However, I do notice that this setting only works with iso100, if it can be extended to other ISOs will be a genuine plus.

I am wondering if it is possible to convert directly out of CF card.

CF card>>software>>HDD DNG>>Resolve Lite>>avid DNX.mov>>PR CS6

70D is released and I have noticed that it has one exciting video capture feature of ALL-I encoding for me. Did a little search of ALL-I, and it brought me back to this thread. The greatest G3GG0 has already built-in this feature into 7D ML, test run a little and it doubles my file size. Can't wait to see when there will be an autoexec.bin released.

7D's color and DR is seemingly fine, it only lacks details. How much I wish that Canon can give 7D stock 4:2:2 output.

Maybe we can use some precalculated values. It is seemingly that DIGICs has accuracy problems when multiply too many times. But I am not sure how it will do in dividing, probably it will be fine. The equation changing to 10bit, x * 2^10 / 2^14, can actually turn into two parts x * ( 2^10 / 2^14), and 2^10 / 2 ^14 equals to 1/16, so the equation can change to x* 1 / 16, and that equals to x / 16, and likewise, changing to 12bit means x/4

I would like to report a bug(or not) in this 7D Alpha2. While using LV>>jpg mode, which exists in the debug menu, the jpg sequence starts to record LV000001.jpg things like that. If I change battery or something else that will turn off the camera, the next time I tried to record jpg sequence through LV, as per the functions above, the new jpg sequence records in the same name LV0000001.jpg as the last ones recorded before turn off/restart the camera, so, eventually, the last jpg sequence was deleted, partially replacing by a new jpg sequence.

Another thing. My Lexar 800x CF is not fast enough to record the LV>>jpg jpg sequence. The jpgs were about 205k big, 24fps means that 205 x 24 = 4920k per second, times 8 into bit translate into around 40mb/s, the Lexar 800x CF should be fast enough to handle. I don't know what I have done wrong with the Alpha2 or the camera or the CF card.

Nope, LUT things will be just y = lut [ x ], and that's all.

And about the calculations you mention - they are easily done (like g3gg0 said) with bit shifting:
Raw14b: 13598 = 0011 0101 0001 1110
Shift 2 bits right =>12bit = 0000 1101 0100 0111 = 3399
13598 * 2^12 / 2^14 = 13598 * 4096 / 16384 = 3399.5 = 3400

So no need for calculations at all. Rsh solves it on cpu-elementary level.
If you need better rounding (like said), add half of the bits (2bits = 4, so add 2) to number before shifting:
(13598+2) = 13600
13600 Rsh 2 = 3400

As per vicnaum. If this works, there will be no loss in DR, just lose lum levels because of that 10bit's less levels, but I understand very little of that hex level calculation and I have no time to check its validity.  And if vicnaum is right, there is no calculations to do except moving bits in HEX level, which sounds very promising to me.

Will someone kindly tell me what is the interval of the jpg dump(which is located in the debug menu)?

I cannot find that things mentioned in EOS SDK

I am not expert in computing, but as far as I have known, the LUT things looks very good, but that sounds that we will have a lot of codes like: if x<y0 and if x>y1, x=y2, these kinds of codes are known to stall the CPU to a halt. d's method should be more suitable for CPU to do, and it already working real time reportedly by several one here who have actually loaded it into camera and tried, and we only don't know why the program halts. Maybe d can mercifully display his source code, maybe there will be someone who can look closely to see if there is something more we can do. d's method sound more straight-forward to me, and easier. To achieve better rounding and accuracy probably, soundingly to me, we only need to change from (x * 12) / 14 to (x * 2^12) / 2 ^ 14, and the cpu might be able to collaborate with that.

As the rounding issue, I don't know what these DIGICs will do, probably it will delete everything behind the decimal point to speed up running. But the difference can be like 2394 or 2395, can still be far more accurate than the human eyes can tell.

Another thought.

In x * (2^12) / (2^14), one can only multiply first, or every value will become 0 if one divide by 2^14 first. So the worse scenerio can be this:

The largest number in 14bit data times 2^12, and then divide by 2^14, can be pains-takingly slow, but worth a try.

the largest value: 2^14 * 2^12 / 2 ^ 14, that means 16384 * 4096(that is 67108864
)/ 16384

And we can use precalculated values in the programs too, like instead of using x * (2^12) / (2^14) for easier reading, we can always use x * 4096 / 16384 instead.

As a last resort, we can use unsigned int instead of int, can we?

ARM might possibly be fast enough, because int is pretty fast to run, usually 1-a few CPU cycles in x86 world, and if we use unsigned int, it can be even faster.

The bad news is that every 4 value in 14bit data will change to the same value in 12bit form, but then I realize that 12bit world has 4 times less value than 14bit world anyway.

What do you say masters?

Hello, Masters,

I just did a simple math, to see which part will change its value while converting from 14bit to 12bit or 10bit, and I use this formula:

We will try to find out which brightness value in 14bit world, after converting to 12bit by using int formula like x * 2^12 / 2^14, will change to the smallest value, which either 1 or 0 in int world:

int x; //the value to be found out

formula:

x * (2^12) / (2^14) = 1

the result:

x = 4

So any value in the original 14bit data smaller than 4 will be converted into 0 in the final 12bit data format, which totally consists 4 values: 0, 1, 2, 3

That means that in the original 14bit data, only brightness level from 0, 1, 2, 3 will be converted into value 0 in the final 12bit format. And these 4 values only consists only minimum fraction comparing with 14bit world, which is about 0.024%(4 / 2^14) of all the values in 14bit world, and its final form, which is total black level value = 0. What the good news is this part of data will not be transfer by HDMI interface(I heard that HDMI usually will not transfer brightness level from 0 - 5 in all the 0 - 255 values to the final equipments of display (like TV, projectors...).

To see the 10 bit scenerio:

formula:

x * (2^10) / (2^14) = 1

x = 16

There  will be totally 16 values converted into 0 while using int formula x * (2^10) / (2^14), that consists 0.098% of all the values in the 14bit data format, and these values might be neglected by HDMI interface anyway.

So, I think that using int formula x * (2^12) / (2^14),  or x * (2^10) / (2^14) into 10bit respecfully,  might be very viable methods.