Raspberry Pi High Quality Camera

Started by katrikura, May 01, 2020, 05:16:17 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

katrikura

Raspberry Pi has released a new camera, which can be used with your hardware. The camera uses a sony IMX477-AACK sensor, which has the following characteristics:

It uses IMX477, a BSI & stacked 1/2.3" 12MP sensor.

It's capable of full 12MP readout at 60fps with 10bit ADC, and 40fps with 12bit ADC.

https://www.sony-semicon.co.jp/products/common/pdf/IMX477-AACK_Flyer.pdf

https://www.raspberrypi.org/blog/new-product-raspberry-pi-high-quality-camera-on-sale-now-at-50/

cmh

Yeah I saw that this morning, this is super interesting

yourboylloyd

LOLOLOLOLOLOLOLOLOLOL I have to buy this thing asap!
Join the ML discord! https://discord.gg/H7h6rfq

Walter Schulz

Proportion reminds me of this gem:

And yes, the cam is mounted!

a1ex

Looks promising, and I see a lot of potential. Not for replacing DSLR cameras, but for things like machine vision, security cameras or... just a half-decent webcam.

Quote from: yourboylloyd on May 01, 2020, 04:31:12 PM
LOLOLOLOLOLOLOLOLOLOL I have to buy this thing asap!

Obviously, that picture was made for marketing reasons. This sensor is small: 6.29x4.71 mm active area. Compare to Canon's 22.5x15 mm (APS-C) or 36x24 mm (full frame). A 3x crop on your favourite APS-C camera (EOS M, 700D, 600D etc) is going to have roughly the same size (on the sensor: 7.5x5 mm). Here's a calculator: https://www.scantips.com/lights/fieldofview.html

FYI, the C mount accepts 1" sensors, or even larger:
Quote from: https://en.wikipedia.org/wiki/C_mount
C-mount lenses are built for the 8 mm and 16 mm film formats and the 1/3", 1/2", 2/3", 1", and 4/3" video formats, which corresponds to a range of image circles approximately from 5 to 22 mm in diameter.

But yes, you will be able to swap lenses between this thing and your Canon DSLR (with adapters). In both directions if you use an EOS M.

Anyway, don't underestimate the power of RPi user community. Pretty sure they have a quite a few highly skilled - active - developers ;)

https://www.techrepublic.com/article/raspberry-pi-why-sales-have-rocketed-in-the-middle-of-the-coronavirus-outbreak/




Edit - found some more details:
- Camera-Guide.pdf
- 2028x1520 at 0.1-50 fps (2x2 binning)
- 4056x3040 at 0.005-10 fps (even if the sensor is faster, probably the bottleneck is on the RPi side)
- long exposures up to 200s

Wanted: complete sensor documentation, not just the 2-page datasheet :)

Luther

Quote from: a1ex on May 01, 2020, 05:59:44 PM
Not for replacing DSLR cameras, but for things like machine vision
I'm contributing to a dataset for a computer vision project that would highly benefit from it. I wonder if it's able to output Raw or if they have the firmware open sourced to tweak compression parameters. One issue we are having with the dataset images is that most cameras have too much compression and this affects quality of object labeling. Outputing raw would not just improve overall labelling but also open to possibilities of pre-processing those images (to decrease haze on low visibility scenarios, for example).


2blackbar

any dng samples ? Id buy it if it
shoots raw video
https://www.google.com/amp/s/amp-blog.robertelder.org/recording-660-fps-on-raspberry-pi-cameravut all they have is crap resolutions,probably cause card reader is a bootleneck, they should start with that, and provide highest res info sheet in typical fps like 30 25 24 60 , not 660 with 64 pixels height.
I guess resolutions in 24 or 30 fps cant be too high , suspiciously they dont talk about it which could mean theyre poor not even hd.
I hope im wrong tho.

---
ok its 30 fps max in hd nobinning but crop and higher res or nocrop is 15 fps max, binned 2x nocrop mode is max 1640*1232 4:3  or 15:9 aspect ratio and 40 fps  info in pdf is vague , unless someone proves otherwise .
Ill pass until its going to be higher than 1080 in 24fps.

Or crop from fullres to lower one to achieve 24ps maybe? Maybe worth a try

Frayfray


2blackbar

This is their 720p...

Looks like they skip 4 and leave 4.
I would ask them why, but its so unreliable at the moment anyways, exposure issues, dcraw issues with white balance, no way to control shutter speed by simpl switches.
It needs a lot of work, and considering that it was started in 2015, i dont have high hopes.
I wonder how 2x binning looks like and whats the max res at 24fps but still , controlling exposure is tricky, there is gain instead of iso and its controlled by command line so yo ucant change stuff mid recording.

jpegmasterjesse

I can only imagine this is going to lead to some incredible experiments.

Magic Lantern port?  :P

a1ex

Quote from: Frayfray on May 02, 2020, 04:14:51 PM
There are other options toshiba TC358840 Camera Serial Interface Converter Chipset (4K HDMI to MIPI CSI -2) https://toshiba.semicon-storage.com/content/dam/toshiba-ss/ncsa/en_us/docs/product-brief/assp/14A04_TC358840_ProdBrief.pdf

That chip could be used to feed the video signal from a DSLR into a Raspberry Pi. Not sure why you'd want to do this.

Quote from: 2blackbar on May 02, 2020, 07:55:42 PM
This is their 720p...
Looks like they skip 4 and leave 4.

FYI, in 720p50/60, your Canon DSLR records one line and skips 4 (5x3 column binning / line skipping).

Exceptions:
- 5D3 (with full 5x3 binning, rather than line skipping)
- crop_rec with custom binning factors (3x3, 1x3 etc)

Quote from: 2blackbar on May 02, 2020, 07:55:42 PM
I would ask them why, but its so unreliable at the moment anyways, exposure issues, dcraw issues with white balance, no way to control shutter speed by simpl switches.
It needs a lot of work, and considering that it was started in 2015, i dont have high hopes.
I wonder how 2x binning looks like and whats the max res at 24fps but still , controlling exposure is tricky, there is gain instead of iso and its controlled by command line so yo ucant change stuff mid recording.

Well, yes. The question was whether the RPi can capture raw images. There is a proof of concept for that, and a rather good one in my opinion. Yes, it's not polished for serious use, but it's functional, and - for the intended audience of RPi, i.e. developers, hardware tinkerers, tech students - it's probably enough to start the ball rolling.

To repeat this important point:

Quote from: a1ex on May 01, 2020, 05:59:44 PM
Not for replacing DSLR cameras

BTW, the image you are looking at was taken with the *previous* generation of RPi cameras. Compared to that, the newly announced camera is a noticeable improvement, in my opinion (but don't forget I'm terrible at pixel peeping).

https://twitter.com/hashtag/ShotOnRaspberryPi

And you are no longer stuck with the default optics; you can use any other C/CS-mount lenses (heck, there were EOS M users adapting such lenses for filming), or - with adapters - pretty much any DSLR lenses.

Quote from: jpegmasterjesse on May 04, 2020, 04:28:25 AM
I can only imagine this is going to lead to some incredible experiments.

Yep.

I wonder how it compares with something like OpenMV H7plus (2592x1944 15fps, 1080p30, 720p60, 3.6x2.7mm active area, M12 mount) for real-time machine vision applications (visual servoing). In particular, I happen to be looking for a low-cost vision platform for CNC machining, where I'd like to get about 0.01mm positioning accuracy, or even micron levels if possible (with subpixel processing, of course), but the machine is relatively slow, so 3-4K resolution at 10-15 FPS, with optional cropping or pixel-binning for a higher frame rate, might be adequate. A "classic" 640x480 industrial camera is not going to do the trick. Are there other alternatives in this price range?

I'd get them both (RPi cam and OpenMV) for a comparison, but my budget is a little tight atm.

I could also consider an EOS M, but...
1) Good luck running computer vision algorithms on its general-purpose CPU (as Canon's image processing hardware is a big mystery even for us)
2) Exchanging the result with motor controllers or other hardware, is also tricky. UART to Arduino-like? USB to RPi? In any case, a second microcontroller board is needed.

Or, I could offload the processing to a RPi Zero, via USB, but I expect very large lags and very low resolution; maybe OK for a webcam, but not practical for visual servoing.

Quote from: jpegmasterjesse on May 04, 2020, 04:28:25 AM
Magic Lantern port?  :P

If this camera proves interesting enough, it might happen sooner or later. Don't look at me, but at the target audience of RPi :)

Edit: some arguments against the RPi cam:
https://twitter.com/marcan42/status/1088472549715918848

Frayfray

Quote from: a1ex on May 04, 2020, 08:10:16 AM
That chip could be used to feed the video signal from a DSLR into a Raspberry Pi. Not sure why you'd want to do this.

maybe i misunderstood i thought the post above me was talking about 1080 or better res. and yeah i know that it is hdmi input i did say there are other options, but if you can capture a 4k 30/60 fps for $200- 250 from many new dslr or mirrorless on cheaper media for long recordings that would be great . i did reach out to the guy who wrote the raw script and maybe if possible to have a working model on pi4 and thank you for posting those links cause this has been on my mind for a while, i was not even sure that you could record raw on ras. pi, there was a board put out by orange pi RK3399 but i think that it was using older chip ,tc358749bg so no 4k,but i found this board from uk B112 4kp60 HDMI to CSI-2 bridge https://auvidea.eu/product/70513/  4k/30 4:2:2 @ 10 bit wow  i hope that people start experimenting soon not you alex i know that your very busy and i appreciate what you done for our cameras     

Luther

Quote
I wonder how it compares with something like OpenMV H7plus (2592x1944 15fps, 1080p30, 720p60, 3.6x2.7mm active area, M12 mount) for real-time machine vision applications (visual servoing).
I think that's the resolution for still photography, no? Video capture seems to be 640x480px (at 60 FPS) (couldn't find in the official documentation - didn't try hard to be fair). It has Raw output though:
https://www.youtube.com/watch?v=8FVoSF34zNM

Quote from: a1ex on May 04, 2020, 08:10:16 AM
And you are no longer stuck with the default optics; you can use any other C/CS-mount lenses (heck, there were EOS M users adapting such lenses for filming), or - with adapters - pretty much any DSLR lenses.
Yeah, that's something awesome about these cameras. You could put a 12mm f/1.8 for less noise in autonomous driving applications, for example.

For anyone interested, we are building a 10K image dataset for autonomous driving that will be available for free to everybody and will be used in OpenPilot/Comma.ai:
https://github.com/commaai/comma10k

a1ex

Quote from: Frayfray on May 04, 2020, 11:34:55 AM
yeah i know that it is hdmi input i did say there are other options, but if you can capture a 4k 30/60 fps for $200- 250 from many new dslr or mirrorless on cheaper media for long recordings that would be great .

Ah, implementing a custom HDMI recorder. Aren't there already good options on the market? (no experience with any of them, but I had the impression this market is already saturated).

Assuming the RPi has the processing power to handle a 4K stream (the more powerful ones might have, but I have no experience with them), it could make sense. Found these things, apparently doing exactly what you mentioned:
https://eu.mouser.com/_/?P=1y9f2kt (B110 using the TC358840)
https://auvidea.eu/product-category/csi2bridge/hdmi2csi/
https://eu.mouser.com/datasheet/2/864/B10x_technical_reference_1.4-1130369.pdf (B101 and B102)
https://auvidea.eu/download/manual/JN30/JN30_technical_reference_1.0.pdf (mentions the B112)
https://www.raspberrypi.org/forums/viewtopic.php?t=216903
https://www.youtube.com/watch?v=-r0ZN7LeSlM


Quote
there was a board put out by orange pi RK3399

Right, this one has HDMI IN (specs unknown): http://www.orangepi.org/Orange%20Pi%20RK3399/

Quote from: Luther on May 04, 2020, 02:59:29 PM
I think that's the resolution for still photography, no? Video capture seems to be 640x480px (at 60 FPS) (couldn't find in the official documentation - didn't try hard to be fair). It has Raw output though:
https://www.youtube.com/watch?v=8FVoSF34zNM

The H7 is, indeed, limited to 640x480. The H7plus has OV5640, which is capable of 2592x1944 at 15 FPS. I don't expect this board to process such resolutions in real-time, at this frame rate, but working on cropped areas (regions of interest) might be doable.

Both have the option of a 640x480 global shutter module, or a FLIR Lepton module (thermal camera).

Quote
For anyone interested, we are building a 10K image dataset for autonomous driving that will be available for free to everybody and will be used in OpenPilot/Comma.ai:
https://github.com/commaai/comma10k

Very nice! Isn't that the self-driving car startup of George Hotz?

Luther

Quote from: a1ex on May 04, 2020, 03:37:30 PM
The H7 is, indeed, limited to 640x480. The H7plus has OV5640, which is capable of 2592x1944 at 15 FPS. I don't expect this board to process such resolutions in real-time, at this frame rate, but working on cropped areas (regions of interest) might be doable.

Both have the option of a 640x480 global shutter module, or a FLIR Lepton module (thermal camera).
I see, I mixed up the Plus with the normal one. That's a very impressive resolution for such a small/cheap camera.
Quote
Very nice! Isn't that the self-driving car startup of George Hotz?
Yes, exactly. The project is all open source, couldn't believe it when I first saw. And it works very well most of the time.

Frayfray


Luther

That's so awesome @Frayfray ! Very good quality. This would be very useful for autonomous driving projects or drone projects (maybe port it to smaccmpilot).

Frayfray

i would be interested in drone applications but with all the other projects going on right now they have to wait a little ,i checked with local store already sold out on this camera ! i think lots of good things are going to happen once people get there hands on few of these and to mount them in side a car is a very good idea ... :)

cmh

If if it's the Raspberry Pi VideoCore IV gpu that is hindering the capabilities of the Sony IMX477 maybe another board will do the job.
The Nvidia Jetson Nano paired with the Camera Module v2 (IMX219-200) can achieve 3840x1848 at 28 fps if I'm not mistaken.
I would stick a Kern Paillard 36mm f/2.8 D mount lens on that thing.
I haven't done much research tbh.



2blackbar

jestson nano cam faq
Quote1.14 Is it possible to get RAW Bayer frames from the camera?

Current camera driver and firmware doesn't support getting RAW Bayer frames from camera. We can customize the camera driver and the firmware according to your requirement. Kindly write your requirements to [email protected]. They will help you out on customization request.

garry23


cmh

Quote from: 2blackbar on May 15, 2020, 11:49:33 AM
jestson nano cam faq

This is from e-consystems and concerns their camera (e-CAM55_CUMI0521_MOD an AR0521 sensor).
The Jetson nano works with the v2 IMX219 sensor out of the box.

cmh

https://leopardimaging.com/product/csi-2-mipi-modules-i-pex/csi-2-mipi-modules/li-imx477-mipi-cs/
It's 4 times the price and this is pretty much the same as the hq camera. It works on a old Jetson nano tx1 https://elinux.org/Jetson/Cameras

The driver is not open source and you'd probably need their API unfortunately (but CUDA tho plus it has 2 interfaces and you can hook up 6 with an adapter).

So maybe it would works with ease with the 50 bucks hq camera.

edit: leopard imaging's camera works with a tx2 too.
People are working on upstreaming the Broadcom Unicam CSI-2 and ISP peripheral drivers to support libcamera, that will include the hq camera.



Luther