Open source USB C camera with C mount lens, MIPI Sensor, Lattice FPGA, USB 3.0 | Hacker News Reader

Open source USB C camera with C mount lens, MIPI Sensor, Lattice FPGA, USB 3.0

circuitvalley.com

482 pointspicture97 comments

Fantastic open source project. I was involved in similar development a decade ago. The camera was in development hell will 1.6 million € wasted. There were permanently changing product managers, project managers and contractors. No end in sight. And nowadays somebody publish their million euro development on GitHub. Thank you!

In your opinion, what caused the development of that product to be hell? Bad/clueless management? Poor engineering hires?

Was it a scrappy startup or a large established corporation?

Asking since I've also experienced this and seems to be prevailing in this industry and yet I feel it's relatively easily preventable.

I had another case of the same development hell and I feel one coming in my current job soon. It’s easy for middle management to draw yearly plans in the slides. However reality can bite hard with silicon bugs, unusable libraries or even bugs in development tools.

This camera disaster was in a big old company. They wanted to get mpg pictures straight from the camera to show on other device. Half million € was gone between planing and first buggy prototype. A company was hired to design it externally. This external company generated design with matlab rendering it non-debuggable.

I think it’s managerial thing. I was screaming many times as simple developer not to hit own leg with an axe. But I felt like I was barking. Manager has always last word and there is a need for convincing to do technically sane thing. Now I just do what I was told. Even when technically it makes no sense. I don’t need additional tensions with insulted manager (anymore).

This looks wonderful!

I wish it could be mass produced, with some minor polish. I would throw money at it instantly. If the author is reading this, have you considered crowd sourcing it? It's great that everything is open source, but putting one together is above my skill level.

I'm desperately looking for a hackable webcam that isn't a huge privacy risk. Adding network connectivity to this project probably wouldn't be a large endeavour.

Actually, is there any open webcam for home use that doesn't seen all the recordings to some 3rd party? I'm interested too.

There are plenty of cheap-ish commodity cameras that use ONVIF to stream video over a LAN. That by itself doesn't guarantee that they aren't also exfiltrating the data somewhere, but you can put them on a firewalled VLAN.

you can make a webcam from a raspberry pi (a case is harder)

or you can plug in one or more usb webcams, some of which have good cases.

Some of the random-commercial-thing-from-China home security cams have open-source replacement firmwares.

At a guess, this would likely cost in the high-hundreds to thousands of dollars if "mass" produced.

"mass" production is really expensive, as you have to add design-for-manufacturability and design-for-testing, and account for yield loss and customer support (how happy would you be if your expensive camera appeared DOA?).

At that price-point, are there other devices on the market that already fulfill the need?

> Adding network connectivity to this project probably wouldn't be a large endeavour

lol

> At that price-point, are there other devices on the market that already fulfill the need?

No there aren't. Webcams in 2022 all have miniscule lenses/sensors paired with decade-old tech and none of the big players seem interested in producing anything higher quality.

https://reincubate.com/support/how-to/why-are-webcams-bad/

If you want high-quality video capture, your options are pretty much just (1) plug in a "real" $$$$ DSLR/mirrorless camera which will be fiddly and prone to overheating, or (2) use a cell phone as an IP camera, which will add noticeable latency and is still a tiny sensor compensated for by AI.

If this was for sale fully assembled I'd easily spend a few hundred on it. I don't care about networking, just give me plain old USB.

There is another option: You can buy relatively high quality PTZ cameras designed for low-end broadcast (HoW and legislative mostly). Some will output to USB, at the higher end you’ll need to capture SDI, NDI, or 2110.

I have a coworker with this setup and it’s easily the best video quality I’ve ever seen on a teams call even in his fairly dark room.

I’d ask you to find and share the model number they used, but I feel like it would be OOS by the time I remembered to look, then come back at 3-6X the price.

https://www.photo4less.com/product/bzgear-ptz-full-hd-1080p-...

The “professional video” category has quite a few

The insta360 sells at about €300-350 including VAT. This is $1000 EX.

Is the video quality that much better? Or is there some other feature that justifies the price?

A lot of PTZ cameras have fairly large sensors compared to webcams or phones, and unlike a phone or DSLR they’re designed to be operated continuously so they shouldn’t overheat.

> is there some other feature that justifies the price?

Physical lens zoom on most of those, with decent lens sizes, but that's not that helpful for a Zoom call, unless you've got a very weird setup.

PTZ = pan tilt zoom

SDI is https://en.wikipedia.org/wiki/Serial_digital_interface

NDI and 2110 are video-over-IP standards. https://renewedvision.com/blog/guide-to-video-standards-sdi-...

HoW I can't figure out.

House of Worship maybe

> low-end broadcast (HoW and legislative mostly)

I like how the original comment was kind of slagging HoW and legislative to the lower realms of quality. For the most part, that's going to be true, but there's definitely edge cases. I run in weird circles having grown up in HoW production and moving into cinematic production. One of the megachurches in town is one of Sony's flagship install locations. They always have the top end equipment that even some production houses can't get yet. The amount of money that flows through these places is flabbergasting. Their control rooms are better equipped than some of the live sports trucks I've been in. The only thing they are missing is the crew of slo-mo operators. They have a couple of seats for shaders, they have chyron ops, they have technical director, director, multiple sound boards (one for record, one for stage, one for broadcast), and just an incredible amount of gear overall. Yes, this is said from a state of jelly.

Wasn’t try to slag HoW at all[*]! Just a lot of cheap broadcast PTZs are built with more price sensitive customers in mind, who often come from legislative, corporate or house of worship. You can absolutely find high-end sophisticated productions in all those fields, but the field also has much more of a low-end (both in terms of cost and in terms of production complexity) than you’ll normally find in most more traditional broadcast verticals.

* I actual have a love for HoW as a vertical because they’re much more willing to try wild or experimental things with their production since they often have a greater tolerance for things going wrong and often have entire days of downtime between shows to tinker. This is probably part of why they’re a Sony flagship - willingness to run beta software! I know of at least one other vendor who has churches as beta testing sites…

There are a number of churches like this in the USA, yes.

But before anybody freaks out too much, it's probably worth pointing out that the median church size is 75 people.

I'm curious what the median church "experience" is of those that attend. One could imagine a single 50 million member church and two churches of 75, which would lead to a median church size of 75 but a median experience of 50 million fellow congregants.

The mean size is quite a bit larger, meaning that many people are in much larger churches, like the one discussed above.

I think in this context it's "house of worship."

A much much more expensive option?

PTZ is really nice but for normal webcam use doesn't add much.

Consumer webcams aren't the only class of device that you can plug into computers to capture images/videos.

>> At that price-point, are there other devices on the market that already fulfill the need?

> No there aren't.

There are tons of cameras that meet this need. They're "machine vision" or "scientific" cameras, google for those. Edmund Scientific has a good selection. They're ridiculously expensive for what you get if you're comparing against a webcam or a surveillance camera since they're very similar to those for >10x the price. That's the price of low volume assembly, proper support, et cetera.

Astronomical cameras and microscope cameras are affordable. Some of them mm support sensors that uses them even supports Sony's proprietary protocol.

A lot of sbc boards support MIPI CSI (iirc even RPI). Nvidia robotic modules has IO for Sony's LVDS (driver not provided though).

That's also the price of selling to a market that can afford to spend gazillions on crap cameras.

You can buy an IMX477 board with CSI output and a CSI->USB board, ~$100 combined.

Not quite a fully assembled solution but all you have to do is screw the two boards together with a few standoffs.

I haven't had a lot of success in this regard. Do you have any links I could review to do this?

For example (I just emailed Arducam about this):

The IMX477/R sensor supports Full (16:9) Binning 1080P @ 2028(H) × 1128(V) with 240hz/fps availability in a drive mode. However, no IMX477/R package I've ever found - including those from Arducam - support a CSI->USB board or similar that can drive this mode, only allowing 100 FPS maximum on most packages. (ex: https://www.uctronics.com/arducam-high-quality-complete-usb-...)

I am looking to buy a product involving the IMX477/R sensor that can drive this mode or similar 240 FPS modes, but cannot find it for the life of me despite the datasheet openly supporting this drive mode.

Forget the USB boards, they all work to the best of the USB / UVC limitations but they aren’t great. You want to hook your camera sensor up to a processor with a CSI input.

Image sensors are really just high speed ADC clocking out data, whether the data is useful or not is up to the viewer. What I’m getting at is forget FPS and think data rates.

So, you need a processor that is that can handle that data rate. The napkin math says 2K@240FPS has the same data rate as 4K@60FPS. All you need is a processor or module that can handle it and you’re golden. There are demos of the RPI pushing 1000fps (at a significant lower resolution) but you get the point.

Off the top of my head, NVIDIA jetson, imx8m mini plus, the latest allwinner V series, and a whole host of unobtainable in less than 10k qty parts all supported this. The rest is up to you.

Also, RPi is out even with 4 lanes because the CSI input doesn’t support the speed required on each lane.

You seem knowledgeable in this area. Do you know how image sensor support works? Can you attach any camera with CSI to the SBCs that you listed?

Short answer - No. The MIPI CSI-2 (CSI has been deprecated) interface is not plug and play. It requires some tuning of the handshaking between the image sensor and the receiving SOC. I think the RPI + HQ camera + libcamera is the best place to start to learn a full camera system since libcamera is open source. The software stack for a camera is a pretty large undertaking. The ISP calibration and AAA algorithms are generally closely held by various camera companies.

I've written raw frames to memory using the RPi HQ cam and RPi4 and processed them in ffmpeg/dcraw for example, so I'm familiar with that methodology. I can take it from there if I can find an IMX477 sensor (or similar) plus 4+ MIPI lanes to get full drive mode.

Any idea where to start? I am proficient enough in scripting/coding to pick up a new platform like the Jetson/Nvidia products, but no idea where to find the correct sensor+board package to actually give me the data rates I need.

Looking for 240fps/hz @ 720p in some form of greyscale, for the record.

Yeah, no, you don't need a DSLR. A point and shoot with a good sensor is more than enough. My Sony RX100 runs all day without issue and acts as a fantastic webcam via a cheap HDMI to USB adapter. Total cost used < $500.

I use an entry level mirror less DSLR, the canon M50. And I think it was about $350 and it can connect directly via USB.

> or (2) use a cell phone as an IP camera which will add noticeable latency

You can use them directly via USB using Camo or OBS - no IP needed, and that changes the latency equation.

Also, the new iOS continuity webcam feature is essentially zero latency - it's better than my good webcams.

If your need is video conferencing, relatively recent ‘insta360 link’ is _very_ good. I’ve ditched my mirrorless+HDMI setup in favor of it and haven’t seen any drop in quality.

Companion app is pretty shit though, but they seem to be working on it.

You could probably pick up those big old Cisco video conferencing cameras, which take 12V power, emit up to 1080p video over HDMI (which goes into whatever generic HDMI capture card you have) and are controlled over RS232 using Sony VISCA commands.

The cameras themselves aren't especially "hackable" but then they don't need to be, because everything going in and out is extremely well-documented.

If you just want a working camera, there are tons of industrial USB cameras on the market but probably all of them are proprietary.

If you want an open source and open hardware camera there's nothing on the market AFAIK. Of course when people discover that open source costs 10x more than proprietary they are usually no longer interested.

Industrial USB cameras use either USB3vision or proprietary protocols and are deeply cursed either way. They’re very much on the other end of the spectrum from UVC plug and play.

Hasn't been my experience working with ximea, point grey, and realsense— all these cams are at least some degree of plug and play on Linux. Obviously you end up with the vendor SW if you want more control over the shutter or something, but basic video stream capture is pretty universal.

Same, it pretty much worked without a fuss, we did a cool proof of concept demo where we used it to auto target a gimbal on a drone using an nvidia jetson running ubuntu. We got the entire software stack working together within a day or so, linux just picked up the industrial camera without a problem, but getting more control over its output required some more work. There's a whole little industry standard for high res video streams over gigabit ethernet or usb 3 that the vendors have published libraries for you can just link in your C++ project.

I am the Developer behind this camera. If you buy from mouser and try to make just 1 then Only part would be around 200 USD. If you make hundred in bath of new improved hardware, You can look at BOM cost under 100 USD. I am working on new version of this camera with totally new enclosure and PCBs as well. You will be able to buy this soon enough.

Where can we follow the development of this new project?

The real question is, how much better is the end result compared to the Pi HQ camera that sells for half as much?

That depends how stretchable your price point is. I can greatly recommend Basler cameras from my endeavours with those at work, complete C/C++/Python SDK and extremely thoroughly documented. USB and IP, HFR and High Resolution.

After working with cheapish cameras before, I won't ever look back.

There's no way stuff like this will ever be mass produced, but it's actually not that difficult to assemble this.

You'd need:

- Find components from Mouser/DigiKey/etc

- Order PCBs w/ a stencil from JLCPCB/PCBWay

- USB-C Mini Hot Plate

- Pinecil/TS100/TS101 soldering iron w/ a set of fine tips

- Soldering paste, regular solder, flux, IPA, old tooth brush, magnifying glass w/ light

- Steady hands

Then you apply solder paste over the stencil to the top side with all the chips, put components on, and reflow on the hot plate. You solder the rest of the components on the back with soldering iron and flux.

Overall the whole set is gonna cost you ~300 bucks (provided the components are not crazy expensive). But you'll have a pretty advanced electronics setup AND an open-source camera.

I have done lots of fine pitch SMT hand assembly at home with stencils etc. But I do see this has lots of BGA chips and I'm not sure how easy those are (I've never done them).

BGA seems scary, but it’s not bad: with the proper amount of solder it “self-aligns” just like any other SMD component. Even a little more precisely because of all the contact points.

AFAIK, reflowing a BGA chip without fucking it up is difficult without a temperature-controlled reflow station.

Soldering BGA is the easiest part, You just need good hotair. While soldering connectors are hardest. They melt , do not self alging and many other problems

Yeah, reflowing BGA is a whole different beast. You have to completely remove and reapply all solder but then that’s not really “reflowing” that’s just assembling.

I am the Developer behind this camera. I have assembled cameras the exact way you have described. If you buy from mouser and try to make just 1 then Only part would be around 200 USD.

I was wondering about this comment on the article:

> IMX335 is also MIPI CSI sensor, So it can be added, I just did a quick search and found sensor itself is not available for purchase but you can buy IMX335 based cameras pretty cheap and get camera sensor out of it.

Do you desolder the sensor (and how would you do that)? Or does the sensor still needs to be on its original PCB?

Would be great to see this n a crowdsupply campaign.

Consider the OpenMV[1] camera, I have several of these and find them very useful for everything from simple vision control for robots to automated face detection. I turned one into a license plate reader and had it identify plates that drove down my street. All very versatile.

[1] https://openmv.io/products/openmv-cam-h7-plus

OpenMV seems to be targeting completely different use cases where the image never leaves the camera (because OpenMV doesn't have any usable I/O to get video out of the board).

I wonder if it could be adjusted so that it can be ordered assembled through a place like JLCPCB. Then you could get the PCBs, order your choice of 3D printed enclosure, add the lens, assemble it and be done.

I'm desperately looking for a hackable webcam that isn't a huge privacy risk

RPi is no good for that? If they are available.

The OpenCV OAK series of cameras are fun to play with: https://store.opencv.ai/

I am the Developer behind this camera. Camera as is not very productionable because of manufacturing related issues.

There will be Third revision of this camera coming in few months. I am working on absolute new hardware with lots of improvement in image quality.

This is really nice, and the documentation and testing results are extensive.

https://github.com/circuitvalley/USB_C_Industrial_Camera_FPG...

Um, but I could find only a single actual photo and zero videos made with this camera. So in my definition, that's sparse testing results.

I think it's early days yet. A work in process, and maybe not the last iteration.

I am developer behind this camera. Its a second revision of hardware and also second revision or firmware. I have been doing MIPI CSI FPGA stuff from quite sometime. Current state of firmware is stable. Where work need to be done isimage quality in terms of color correction , Whitebalance, noise reduction other similar things.

He includes ~5 minutes of the cameras working at various resolutions and frame rates in the project video.

See: https://youtu.be/dwXs9RB6uD0?t=2621

This is a very cool project especially that it seems to be the only low latency "consumer grade" camera available. 4 lines of latency are quoted, this is great considering other alternatives (rpi cam, other sbcs + csi camera) have latencies from 50ms up to 150ms which makes them borderline unusable for many uses such as dynamic fpv, fast acting machine vision etc.

This is a pretty impressive project, though the results (https://www.youtube.com/watch?v=dwXs9RB6uD0) are somewhat disappointing. From what I see toward the end of the video, the image is blurry with poor colour rendition, although it is fantastically low latency.

I suspect you're currently limited by the optics you're using, lack of white-balance and low CRI lighting. Looking forward to seeing further tests!

Also related: Take a Raspberry Pi Mini, a Raspberry Pi HQ camera, and install this firmware

https://github.com/showmewebcam/showmewebcam

and it turns your Pi into a UVC-compatible USB webcam with a C mount lens. This firmware also boots instantly, and is also safe to unplug and plug without logging in and issuing a shutdown command. There are even some 3D print housings for this combination on Thingiverse.

I really wish we had something similar with a larger e.g. APS-C or 35mm full frame sensor though. The Pi "HQ" camera is more like shit-Q compared to a modern consumer camera.

I am testing out arducam, my biggest problem is finding an outdoor friendly enclosure or any enclosure that i can mount on a window or wall. My options seem to be messing with non-conductive resin or repurposing existing webcam enclosures. Also, the quality and fps isn't impressive and I still have to figure out how to disable the IR lights that are a deal breaker for window mounting. Tips welcome.

I have no idea if this camera is any better but will try to compare specs.

I was looking for a "Buy" button. I didn't care about the other stuff. I wish there was one.

I have a bunch of industrial cams, which are pretty similar and pretty good, but open design would be huge for what I do. What I want:

- Quality 1080p capture at 30fps (25 is fine too)

- USB Webcam

- C-mount

- Compressed output, so I can connect a few without saturating USB

ELP makes many modules like this, and they're <$100 and work well. Other brands do too, which have been a mixed bag.

> ELP makes many modules like this, and they're <$100 and work well.

Slight off topic but do you have any recommendations on cameras with motorized focus that can be controlled electronically? For example with OpenCV's CV_CAP_PROP_FOCUS flag?

I have no recommendations there matching 100%, but one of these might work:

Many (perhaps most) off-the-shelf webcams do allow you to control focus, but aren't C-mount, and you're stuck with their lens and their preprocessing.

For what I do, webcam preprocessing makes it almost useless for my applications. Almost all are auto-something, and I don't need exposure or white balance changing once I've calibrated lighting and a machine vision pipeline. Many do nominally have manual exposure and white balance controls, but I've had poor luck finding ones where I have consistency at the level of a module like an ELP.

If you're willing to hack or forgo OpenCV integration, it's ridiculously easy to control C-mount lenses with electronic focus control. That's what I did when I needed to do this. You can look on eBay for something like a vintage Computar TV Zoom Lens (and documentation on Computar's web site still existed when I did this). Prices vary, but when contractors rip out ancient surveillance systems, these can be had for a song.

Or you can splurge, and get a 20x zoom f/1.8.

The ones I bought, it's literally just applying a voltage to a pair of wires to make it zoom or focus in one direction or the other. You just need a few GPIO pins, and to step up the voltage to what it needs (I'm inclined to say 12V, but it might be 5V; it's been a while -- but from not recalling, you can see how easy this was; if this was a week project, I'd remember).

What you really want is a module with the same interface as something like the Computar, which I know exist, but I've never used, so I can't recommend them one way or the other.

And it'd be easy to hack into OpenCV if you really wanted.

Thanks for the pointers indeed controlling the focus motorized control might be easier and more guaranteed result than a software/driver only solution, I wasn't really considering that before. Thanks again!

Great project. Speaking of which, did anyone have the opportunity to get their hands on Samsung's ISOCELL sensor ICs, or any camera-modules that contain them?

https://semiconductor.samsung.com/image-sensor/

E.g. the ISOCELL HP3 has 200 megapixels, 0.56μm pixelsize and 120 fps at 4K resolution.

This is great work. In the same light is anyone aware of ip67 cameras with integrated lens? I see someone posted the opencv camera above, but I’d be interested in more options.

The openmv h7 and arducam boards have tempted me, but getting housing made for them is difficult.

Hot. I hope the year of modular phone will come so I can pick up a decent sensor for a budget phone for my mom instead of $1200+ with a less than an inch of sensor!

Looks like you can get e.g. a Xiaomi 11T Pro for $425 on Amazon. You can't do much better than that for a mobile camera sensor (Isocell GN2), and I can't imagine any modular phone that will be cheaper once you've upgraded the camera.

Sony produces new sensors at a relatively faster pace compared to the camera development cycle. For years, I've been looking for ways of testing these sensors before developments boards would come to market. This project is super interesting and great inspiration for testing new sensors.

Not too no diminish this in any way, but https://www3.elphel.com/ has been doing open source hard- and software since a while.

Elphel Camera cost a insane amount of money and Reputation of the company after google incidence, I would rather not talk about.

Honest question: what Google incidence?

(I know that Elphel cameras were used for iterations of Google StreetView and Googles book scanning project.)

They were initially used then for certain reason they were dropped.

Still interested; do you know more about the reason? Is this the incidence you mention?

Fantasic work.

Thank you.

I am the Developer behind this camera. Happy to answer any questions you have regard this project.

How can we get USB-C (or USB3) connectivity with 720p@240fps? The IMX477 theoretically can do this but due to 2 lane limitation on the Jetson and the RPi this is infeasible (plus write speeds saturate bandwidth, but you can dump frames to DDR RAM first).

Have had a ton of problems trying to figure this out.

720p@240FPS would be little hard because of limitations on the USB controller side. because of 100Mhz and 32bit limit you have possibility of only 400Mbytepersecnd and with few % overhead you have may be get around 200 FPS of color UYV image. if you are ok without colors then you can get to 400FPS @ 720p

> 720p@240FPS would be little hard because of limitations on the USB controller side.

What would be the theoretical limit of the sensor itself?

Could you extend the project to work as a high speed camera? Perhaps using a Firewire ip core implementation instead of USB?

There are only few options, 1. Use USB controller Chip that support 10Gbit or more. 2. Use 10Bit or more ethernet, Optical or copper. 3. Use PCIe 4. Use onboard storage. 5. Use HDMI or something custom and have another receiver (They are called frame grabber and everybody hate them).

Right now there are not many controllers on market that can do more than 5Gbit

Most useable solution is to use Ethernet , I would say optical.

Typically what cameras like the Edgertronic do are have 8-16 GB of onboard DDR3-4 RAM and write raw frames directly to memory on a trigger or a loop/trap. Then when you have X amount of frames you want, process them using ffmpeg/gstreamer using onboard hardware acceleration methods and write to an SD card / network drive / attached SSD or similar. Simply bind X GB of RAM to a memdisk and write to that path in Linux, for example.

I am fine doing this and have done it using raspiraw and know the Edgertronic platform / source code very well. The problem is that sensors and MIPI lanes on commercially-available products are complete trash with even worse documentation.

B&W is fine by me, color not necessary. Also fine with PoE and 10GBE if required. Seems crazy that the IMX477 is capable of so much more than what people are currently drawing out of it.

Any pointers on where to start? I've done raspiraw frame dumping with the RPi HQ camera but it's just going to be a non-starter given all the issues it has.

Nice project! I've used Crosslink-NX and Cypress FX-3 on various MIPI camera projects as well. Did you notice that Radiant does not control the timing of the data from the mixel hardmacro to your FPGA logic? (click on one of the pins in the physical viewer to see this). I ended up adding a ring of flip-flops and physically locking them near the top edge of chip to get consistent timing.

Looking at code: why are you not using the byte aligner built into the hardmacro?

FPGA ISP i am using with this project is improved version of ISP that i made for lattice machxo3 FPGA , those FPGA like most FPGAs do not have any MIPI hard PHY. If you want to port this ISP to Xilinx you would not find hard PHY in many FPGAs and you would need a byte aligner. That is why Byte Aligner was implemented and left enabled in there for the sake of portability to other FPGA it does not hurt (except for may be very very small performance in very edge case or some FPGA resource consumption).

I had many issues with Crosslink NX part. I never specifically got/or noticed the issue you have mentioned.

Reply on news.ycombinator.com