Messages

What you ask is also technically possible, though it feels quite dirty to code something like this. Do you have an idea how much H.264 will slow down the raw recording?

That's why I was mentioning the 17th build example. I'm able to record 1920x840 and 1080p IPB H.264 at the same time without getting skipped frames for a long time, though sometimes it does skip a single frame from time to time (Transcend 64gb 1000x).

Anyway, regarding the real-time playback, the black and white wouldn't be a problem, performances can be checked without color 

Hi there,

I'm going to use the RAW recording on a 5D Mark III in a videoclip soon and I've been talking with the director about the new format, that I love, and the creative possibilities it opens both in production and in post. The only reservations we have is that we cannot check the files on camera, I've shown her the playback option currently available and we agreed we cannot check-review the performances with that.

I'm not asking for a better RAW playback option since it's already a miracle the one we have but maybe there's a better-easier option for this kind of use. In earlier versions, before the start/stop button implementation for RAW recording (which I like a lot) there was the "possibility" of recording RAW and H.264 at the same time. In fact, I evaluated the option for my project and it worked pretty well in 1920x840 for example. The problem is that in that early versions (17th May, I believe) there was not the option to stop before skipping frames and I'm worried I might have unintended jump cuts without noticing.

So the request is if there could be an option for recording h.264 simultaneously in the SD card or, if thats a code nightmare, to the CF card (even at low bitrate/low res in VGA, though the framing would change on that one) so it can be used as a on set review option for production.

I'll completely understand a "won't fix" on this one but I wanted to submit the suggestion. In any case, once ML is ported to 1.2.1 we could use the regular on set monitor option, or even something as the ninja; so this would be a temporary "fix".

I've just tried it and I get a lot of "Not found errors", no preview and no files. I've unzipped and put it in a folder with the .RAW. Drag and drop on the processor and follow the prompts with no luck. I'm running Windows 7 64bits.
Thanks for the work, though.

You mean that ACR and Cineon "neutralize" the magenta cast somehow by balancing the channels? That's interesting to know, I though it was a metadata handling error. I'll have to try and balance the channels then for this workflow.

-l90 -- SI-2K curve
-l400 -- Protune 12 stop optimized
-l900 -- Protune 13 stop optimized

While the default and -l400 both store 13+ stops, it is just a matter of emphasis, and how much you need to dig into the shadows, are the mild expense of midtones and highlights.

Thanks, David, I'll try those. I was using cineon since it's the only one which is not giving me the pink highlights after recovery (with 28th Build in 5D Mark III). I input the files directly into Resolve so I guess I cannot take advantage of the protune-encoding/cineon(or other options)-decoding since Resolve ignores them.

I think Resolve reads cineform metadata in a different way, I've tried to input the R G B color matrix you posted but colors were off so I'll try the log options and see how it goes. Let me know if you need any specific feedback/tests for the utility, happy to help.

@DANewman Really great utility!!! Thanks a lot! I've tried the "raw2gpcf -444- -c -q5" with and without the cineform curve (MOV files for Resolve). The most information I get is with the -c command (without it the highlights are completely blown out). It's fast and seems quite reliable but color noise is worse than in ACR (debayer process I guess?) and the "highlights recovery" of the curve creates some posterization.

Nevertheless, thanks again and I (we) would love to have a GUI and, specially, a batch tool.

EDIT: Could you explain the log curve options? I mean the -lX command. I've no idea what value to use or what the effects are but I would like to try the Protune curve with this tool.

Last build (016ea7d ) seem to have fixed the issue for me!!!
I have only made a couple test but there's nothing I can see. Thanks a lot, devs!

It turns out that lv_af_raw enables something that looks very similar to a vertical edge detection algorithm instead of defect correction (at least on the 650D) 

https://dl.dropboxusercontent.com/u/3904468/DNG/AF-SERVO.DNG

I think that's exactly the issue we were describing!

A couple crops (without any other adjustments)
This is type 17 (NORMAL):
https://dl.dropboxusercontent.com/u/21789470/ML%20PIXELS/Type%2017%20CROP.jpg
This is type 19 (ISSUE):
https://dl.dropboxusercontent.com/u/21789470/ML%20PIXELS/Type%2019%20CROP.jpg

I'm amazed with the exposure change, it remembers me to the 1/3 reported difference between the H.264 and the HDMI output in the new 1.2.1 firmware.

Test done!

Summary:
Several images show image with color pixels but they are not the issue described.

I've found two RAW types which match exactly the problem (they are in caps): 19 y 83. They are also 1/3 more exposed that the ones marked as "NORMAL". They are equal images for what I can see (the two have same exposure and same hot pixels)

I've found 6 types which shows a normal image with no problems, they seem to be equal among them: 17, 35, 37, 39, 81, 99
*The ones marked as "Color points" are color points covering the whole image in which appears to be the sensor pattern.

I've saved the recorded frames in case you need them for reference. I've double-checked the NORMAL and ISSUE type numbers.

Code Select Expand

1 Horizontal color lines
2 Color points
3 Image tearing mostly Green/magenta
4 Color points but with some form from the objects photographed
5 Normal image with a few color pixels
6 Color points but with all the form from the objects photographed – Green/magenta
7 Normal image with a few color pixels
8 Normal image with a few color pixels – less exposure
9 Normal image with a few color pixels
10 Color points but with all the form from the objects photographed – Green/magenta
11 Normal image with a few color pixels
12 Color points but with all the form from the objects photographed – Green/magenta
13 Normal image with a few color pixels
14 Color points but with all the form from the objects photographed – Green/magenta
15 Normal image with a few color pixels
16 Color points but with all the form from the objects photographed – Green/magenta
17 NORMAL IMAGE, nothing I can see
18 Color points but with all the form from the objects photographed – Green/magenta
19 IMAGE SHOWING THE ISSUE DESCRIBED – Hot pixels along contrasted lines
20 Color points but with all the form from the objects photographed – Green/magenta
21 Horizontal color lines
22 Color points
23 Horizontal color lines
24 Normal image with a few color pixels
25 Color points but with all the form from the objects photographed – Green/magenta
26 Normal image with a few color pixels
27 Color points but with all the form from the objects photographed – Green/magenta
28 Normal image with a few color pixels
29 Normal image with a few color pixels – more exposure
30 Color points but with all the form from the objects photographed – Green/magenta
31 Normal image with a few color pixels
32 Color points but with all the form from the objects photographed – Green/magenta
33 Rubbish + horizontal lines
34 Color points
35 NORMAL IMAGE, nothing I can see
36 Color points but with all the form from the objects photographed – Green/magenta
37 NORMAL IMAGE, nothing I can see
38 Color points but with all the form from the objects photographed – Green/magenta
39 NORMAL IMAGE, nothing I can see
40 Image tearing mostly Green/magenta
41 Weird horizontal lines-points
42 Color points
43 Image tearing mostly Green/magenta
44 Color points but with some form from the objects photographed
45 Image tearing mostly Green/magenta
46 Color points but with some form from the objects photographed
47 Image tearing mostly Green/magenta
48 Color points but with some form from the objects photographed
49 Rubbish + Color points
50 Color points
51 Image tearing mostly Green/magenta
52 Color points but with some form from the objects photographed
53 Horizontal color lines
54 Color points
55 Horizontal lines
56 Image tearing mostly Green/magenta
57 Image tearing mostly, some magental and vertical stripes one info+one black pattern
58 Color points but with some form from the objects photographed
59 Image tearing mostly Green/magenta
60 Color points
61 Image tearing mostly Green/magenta
62 Color points but with some form from the objects photographed
63 Image tearing mostly Green/magenta
64 Color points but with some form from the objects photographed
65 Horizontal color lines
66 Color points
67 Image tearing mostly Green/magenta
68 Color points but with some form from the objects photographed
69 Normal image with a few color pixels
70 Color points but with all the form from the objects photographed – Green/magenta
71 Normal image with a few color pixels
72 Normal image with a few color pixels – less exposure
73 Normal image with a few color pixels
74 Color points but with all the form from the objects photographed – Green/magenta
75 Normal image with a few color pixels
76 Color points but with all the form from the objects photographed – Green/magenta
77 Normal image with a few color pixels
78 Color points but with all the form from the objects photographed – Green/magenta
79 Normal image with a few color pixels
80 Color points but with all the form from the objects photographed – Green/magenta
81 NORMAL IMAGE, nothing I can see
82 Color points but with all the form from the objects photographed – Green/magenta
83 IMAGE SHOWING THE ISSUE DESCRIBED – Hot pixels along contrasted lines
84 Color points but with all the form from the objects photographed – Green/magenta
85 Horizontal color lines
86 Color points
87 Horizontal color lines
88 Normal image with a few color pixels
89 Color points but with all the form from the objects photographed – Green/magenta
90 Normal image with a few color pixels
91 Color points but with all the form from the objects photographed – Green/magenta
92 Normal image with a few color pixels
93 Normal image with a few color pixels – more exposure
94 Color points but with all the form from the objects photographed – Green/magenta
95 Normal image with a few color pixels
96 Color points but with all the form from the objects photographed – Green/magenta
97 Rubbish + horizontal lines
98 Color points
99 NORMAL IMAGE, nothing I can see
100 Color points but with all the form from the objects photographed – Green/magenta


EDIT: I used your autoexec.bin with the other files from 22nd build and the RAW module loaded.

I put a DNG comparison in another post http://www.magiclantern.fm/forum/index.php?topic=5413.msg39600#msg39600 . If you zoom in the big "N" or in other letters the pixels are massive.

I'll run the tests and post them ASAP

Okay, I'll fix it if you do the pixel peeping part.

I don't know if you need more tests or if 1% have taken care of that. I'll be happy to help with the pixel peeping 
As I side note, to keep all the test data in this post, it doesn't happen either in 19th May Build.

My latest! Comparing H264 Technicolor Cinestyle to ML Raw. Highlight recovery, curves tweaked slightly in Adobe Camera Raw. No other correction applied, and no post sharpening applied. Didn't touch cinestyle H264 footage at all because frankly, it's no longer pleasurable to tweak compressed footage. Camera settings were identical for each set of shots.

Shot on May 22nd build, on 128GB Komputerbay 1000x.

My write speeds have gotten slower with the May 22nd. I can no longer hit 70MB peak at all and struggle to maintain 64MB/sec needed for 1920 x 840. Most clips have at least one dropped frame. I use CF workaround which seems to help. Global draw fully off.

Vimeo is set to 1080p, so please give it a second to load and then hit fullscreen. The parrot shots and the clocktower shots are my favorites in terms of showing off the power of the extended DR and color corrective capabilities.

As other user (stevefal) reported, this video shows the pixel issue (1:03, right side of the zipper, a big area including hair). I'm sorry but those don't look like dead pixels to me. We would be talking about a lot of dead pixels and very poor testing/manufacturing which I think it¡s not the case. It happens along all the video, if you pay close attention to almost any area, hot blinking points appear: in 0,54 the letters have tons of them. I think some of them even move with the letters.

I insist on the issue because I believe they are not hardware defects but data corruption/noise but I might be wrong.

So you can record 1920x1080 with no dropped frames on the Transcend 32?

I can record 1920x1080 24fps with both of my new Transcend 64gb 1000x but there's a little trick. The maximum sustainable speed I get is 82,9 MB/s which is just enough for recording without dropping frames.

When you put these cards on the camera and try to do 1080 24 right away they just won't. Not even recording several files. But, if you record a couple GBs, I go all the way up to 4GB, in 1920x960 and them you switch to 1080 they are just fine after that. I've recorded several 16GB files without a problem. If you switch the camera on/off, you take out the card or you don't record for a couple minutes you might have to warm them up again.

Of course they don't do 1080p 25fps because they max out at 82,9 MB/s. My cards are formated in exFAT (Quick format, 128 bytes units). Global Draw On/Off doesn't seen to affect much (~0,2 MB/s in the worst case scenario, which for these cards you should watch out). The benchmark doesn't show these speeds, though.

My guess is that they need to heat up to work to the peak. So, AndreasK, I hope it works for you too. Let me know how it goes!

EDIT: I'm using last lourenco build (22nd May).

I do it depending of the scene. The default 25 value is fine but you have to try and use what looks good to the eye. I don't usually mask (or just a little). Personally, I keep the noise reduction in 10 just for cleaning up fix pattern noise and go to 25-30 if I'm lifting the shadows a lot.

Sharpening and noise reduction of ACR are great but, for video, it's always better to keep those settings in lower values and do the final sharpening/denoising step in the grading software or the editor of your choice. Premiere's sharpen is pretty good, can run in real time during playback and it's non-destructive so you can change it and see what values you want.

Neat Video works great with 5D3 RAW (also with H.264), it provides temporal NR and keeps detail pretty well. Though I would only use it if you have a lot of fixed pattern or visible color noise, the regular RAW grain can be taken care of with ACR's luminance noise slider.

Good plan.

A mate of mine came around tonight with his spanky new MK3 and I spent like an hour trying to get ML running on it and every attempt failed. It had 1.1.3 on it but I even re-installed it, I tried reformatting cards, tried the pre raw alpha, everything I could possibly think of. The same cards all ran ML on my MK3. I'm perplexed. He was devastated.   

I would suggest the possibility of Canon blocking ML after the RAW hack, but it's practically impossible for them to have done it that fast (I mean in logistic terms). Have you tried re-installing 1.1.3 or reseting camera settings?

I have been using (2) 1000x 32GB Transcend cards and it will work solid 1080p 24fps. That's with everything disabled (no global draw). Also, if I go into crop mode I have to scale down to 1920 x 840 in order for it not to skip frames. Does anyone know if the Lexar can do full 1080p @ 3x?

Thanks,

AJ

If the card is able to record the bitrate in normal mode, the crop mode issue should be software related. I've seen a fix in the code related to the zoom and double buffering, I guess the next build will help with that.

HI! Is it a 1000x card and have you tried it on 24p at 1080p? I,m about to buy one but it doesn,t seem to work that good?

I'm receiving two of those in a couple days, I'll post the results when I test them. I've seen a couple videos done with those and people said they worked well. In the other hand, I asked James Miller and he said it buffered fast in 1080p (unsure if 24p or 25p). Anyway, I'll let you know what happens.

Build "006d890" - Works great but I saw a fix related to DaVinci Resolve (removing thumbnails - "89e62d8 ") and I wanted to report that it's not working. Resolve still sees the files as 128x96 and doesn't interpret them right. ACR or PS doesn't recognize the camera id either, which was another fix in the code if I remember right. Magenta cast is definitely fixed though it didn't saved (or read) the WB I set on camera.

I was trying to figure out how to make the .DNGs fit the Cinema DNG standard, which is the only thing Resolve reads (allegedly a DNG 1.1 Subset), and I ran into the white paper from Adobe. I'm sure you're already familiar with that document but you have the link below.

http://wwwimages.adobe.com/www.adobe.com/content/dam/Adobe/en/devnet/cinemadng/pdfs/CinemaDNG_Format_Specification_v1_1.pdf (Page 24, B.4)

I'm not sure what's going on. Maybe somehow ML is taking over to such an extent that the Camera OS doesn't know there's activity going on, so it goes to sleep? It happens every time.

Look if "Power saving" is set to "Disable" on Canon menu. If it is, that shouldn't happen.

Yea big difference when uprezzing in AE vs selecting the bigger file in PS. WTF is ae doing? And it won't let you choose a diff size in ACR like PS does.

PS actually reads the 240 ppi of the RAW images but I think After Effects defaults to the 72 ppi of video. It should explain why the images don't upres equally:
After effects is resizing 1920x1080 72 ppi -> 4K (or whatever) 72 ppi
Photoshop should be using 1920x1080 240ppi -> 4K 72 ppi (supposing you are using video resolution)

I run it for 20 minutes. It got a max of 42MB/s which is not what it usually gets when recording RAW (tried at 1280x720).
I repeat it was a 5D Mark III, Lexar 16GB 400x CF.

Code Select Expand

Buffer size experiment
ML v2.3.NEXT.2013May17.5D3113, 3046238719b2+ (unified) tip
Mode: playback, Global Draw: OFF
CF LEXAR ATA FLASH CARD LEXAR ATA FLASH CARD                   
10312704 314
15885312 407
80896 271
8949760 412
28661760 418
16264192 413
24521728 417
16424960 415
13148160 414
15619072 414
1658880 400
21001216 414
8548352 414
19760128 418
5965824 412
24906752 417
13968384 415
16770048 415
4542464 412
13063168 414
33263616 420
30312448 414
16713728 419
3887104 412
21841920 415
12846080 415
16192512 418
3333120 411
6719488 413
17313792 419
23374848 415
2865152 410
32488448 416
18588672 419
4186112 412
821248 392
3661824 411
18359296 415
19875840 415
22754304 416
13665280 385
24957952 414
28555264 417
33442816 415
10708992 414
20242432 413
8774656 383
11216896 413
23156736 413
12750848 413
12369920 412
2157568 404
15279104 415
29943808 411
30321664 412
19004416 413
268288 347
23778304 303
467968 261
1905664 258
32368640 402
19829760 272
26281984 413
24153088 417
11364352 413
32160768 415
5944320 412
24118272 418
28511232 416
31526912 418
15956992 415
1732608 404
13881344 416
6106112 416
17367040 415
21693440 416
30835712 389
1139712 399
13698048 415
19388416 414
1532928 404
16333824 419
15963136 415
23540736 420
23770112 416
7317504 414
6019072 414
13303808 415
32039936 419
13137920 416
24311808 417
25442304 419
8986624 414
25421824 420
2720768 410
18526208 415
12448768 415
6514688 418
6632448 414
17897472 415
26569728 416
22568960 415
11912192 418
22007808 415
30651392 414
29615104 420
18367488 409
21049344 418
15778816 415
24951808 414
5359616 412
23476224 418
17125376 413
3318784 409
12782592 413
30965760 388
1974272 315
32021504 363
24463360 297
28246016 338
23352320 312
27161600 222
25191424 417
10841088 319
22846464 318
19111936 388
26234880 341
32610304 415
14385152 418
18029568 414
11080704 415
3933184 413
22414336 418
8314880 415
27988992 418
20073472 419
23487488 416
30233600 419
25299968 415
2406400 408
28059648 416
2145280 408
27503616 419
20705280 416
28731392 421
2458624 409
19662848 416
6198272 417
21655552 417
32225280 416
21788672 420
16669696 417
12007424 416
4321280 413
14878720 420
10118144 416
2836480 411
21092352 420
17086464 416
1707008 406
17509376 417
28161024 420
18457600 417
16290816 420
10758144 416
17766400 416
28011520 418
19835904 383
11879424 415
18191360 387
12581888 414
15906816 418
6525952 413
30545920 418
934912 395
6738944 412
18920448 415
27171840 416
19916800 412
28598272 397


I'm running it on a 5D Mk III with a 400x 16GB Lexar CF card, I don't know if with that speed the benchmark would be of any help but it can't harm 
I have more than half charge on the battery so it can take a while.