Taking over a Dead IoT Company
blog.kchung.co
blog.kchung.co
My apologies for the downtime, I wasn't expecting much traffic today since I submitted the post to HN yesterday but I've started scaling my server now!
People's stuff will work again thanks to you and you should feel great about it!
I wouldn't discount your ability to bring this back to market. Would consult a lawyer to see what implications taking over the branding and APIs have. But clearly this has a big market.
[0] https://hackaday.com/2023/01/09/iot-archaeology-leads-to-api...
EDIT: Here is the specific tweet: https://twitter.com/NYCTRAINSIGN/status/926106932573810688
It's a shame because there is a small but sustainable business here.
Why would you think that a company using a $25 Adafruit HAT that could be found for a magnitude less elsewhere would be using cheap LED matrices?
Put another way, what from the article gives you enough confidence to say that they would have used cheaper parts?
There are various "HUB75" panels of different pitch, brightness, LED type, matrix size, etc. Also, there's variations like "HUB75E" with 5 address lines instead of 4. Address lines are typically A/B/C, A/B/C/D, or A/B/C/D/E. And there's also "HUB08", "HUB12" and other standards.
Once the new server was up, did any signs from the field connect to it?
Do you know how many signs are out there actively looking for the mothership?
> Good question! No signs connected to the server until I reached out to some other sign owners to try out my instructions.
I do not know how many signs are out there. I imagine most people would have just unplugged their sign after the company's API vanished since any data would be stale and useless.
I've had a few projects like this where I list out the BOM then go for 'reasonable business success profit' and scratch my head at exactly this line:
> At even just 60k per founder, with the profit per sign sold being maybe $400, you would probably need to sell ~600 signs at full price per year to create enough revenue to run payroll.
Hardware is an exponent to the value of the internal reward, but the root of all business profit. We need a better funding mechanism for this style of development - more like the hardware lifecycle - wasn't Bolt doing something like that?
Even your calculations are way too optimistic.
Giving it little thought, I can say that NYC would have at best 10k sales of this item. In total... including any knockoffs.
The product may have scaled to be able to sell to London, Paris, Tokyo, Soul, Beijing, Shanghai, Hong Kong and Singapore. But this is the total addressable market for this item.
Let's say you have 2 engineers in US working on this full time - that's 500k in salaries(that includes all associated staff).
In NYC you would be able to charge $300, but no other major metro area has that kind of disposable income. In London the max price would be $170, Paris and others - $150... And boom, you can't even make a product to cover development costs.(not to mention any operating costs)
Cool art project and commercial viability clash perfectly here. This item is at best - an art piece, that should cost as such($3000+).
USA, the land of start ups!
That's 2 days of median disposable income.
For comparison:
San Francisco, U.S. - Disposable income after rent: $4,710
Zurich, Switzerland - Disposable income after rent: $4,626
Chicago, U.S. - Disposable income after rent: $3,298
Boston, U.S. - Disposable income after rent: $3,188
New York City, U.S. - Disposable income after rent: $3,157
Sydney, Australia - Disposable income after rent: $2,615
Melbourne, Australia - Disposable income after rent: $2,485
Oslo, Norway - Disposable income after rent: $2,342
Copenhagen, Denmark - Disposable income after rent: $2,285
Wellington, New Zealand - Disposable income after rent: $2,075
Depending on your list and method, New York could be 44th on global purchasing power index:
https://www.numbeo.com/quality-of-life/rankings.jsp?title=20...
Or 15th:
https://defence.pk/pdf/attachments/screen-shot-2019-11-27-at...
Anyway, "no other major metro area has that kind of disposable income" seems inaccurate.
With easy to use editors (e.g. easyEDA) and cheap and fast PCB production/assembly (even for small quantities), there is no reason to use overized MCUs/mini computers like the Pi and expensive modules e.g. from Adafruit.
Also, BOM components selection is key. For the same functionality, e.g. a multiplexer there are many different chips available -often at very different price points. Availability in times of chip shortages are also key to keep an eye on.
Technically this product is very simple and the BOM costs they had were a magnitude too high. Too bad because it seemed that the market was ready and they could have become a very successful business.
There is if you don't have the expertise. Of course, arguably if that's the case, you shouldn't be developing such a product anyway. But as the post detailed, by switching to a Pi Zero and removing the Adafruit HAT, the BOM costs would already drop significantly. I think if you're smart about it, you absolutely can build a viable product this way. Certainly not as profitable, but possible.
Eventually I switched some things up and it got much easier.
Definitely not for scaling, but for deploying prototypes this approach is awesome! Eg use the Pi to create some prototypes, get user feedback, launch a kickstarter, do what you said. I worked at robotics startup that did this, worked super well for our first robots. Podcast about it: podcasts.apple.com/us/podcast/erik-schluntz-cobalt-robotics/id1649845905?i=1000584727151
Genuinely curious, as I've typically used things like the Pi for projects like this but have been genuinely curious what it would take to strip it down to something less powerful and achieve the same result.
https://raspberry.piaustralia.com.au/products/raspberry-pi-p...
Since I am a Go developer, I like using TinyGo with these type of boards, which means I get concurrency (goroutines) as well, although sadly Tinygo doesn't yet support the Wifi module. However, I've used other boards with Wifi, and once you're connected to WiFi you can poll APIs and do anything else you do on the internet. No OS required!
As an example of a complete, stand-alone program, I wrote a digital clock in TinyGo. (https://github.com/doctor-eval/clocky) - you plug the Pico into a 7-segment display and it operates just like an old school digital clock.
Sadly it can't (yet) connect to the internet due to missing wifi support in Tinygo, and it's running on a pre-W Pico, but you get the idea.
The cheapest you can get is about $5(Espressif assembled board), with full LED matrix controller.(that's the cost of manufacturing, though)
a full blown Linux stack is $10+
This screeches to a halt, the moment that you put into perspective. My typing this is equivalent to a few dollars, because most people here have at least $50 per hour engineers. Designing PCBs for existing modules(ex. ESP32) takes a very long time(and a lot of knowledge) and it's not just "put this chip here and have software figure it out".
Missed opportunity by this startup.
Once the new server was up, did any signs from the field connect to it?
It's amazing to me how quickly the company loaded itself up with staff. Being kinda familiar with ESP32 development, I thought at the beginning that the company would have 4-6 people - 1-2 technical, 1-2 production, 1 business/sales/marketing, 1 admin. It seems like they opted to target the luxury nostalgia market with the $600 price point rather than a more value-driven $300, and wrapped themselves in an aura of success rather than growing it naturally. The descriptions on the resumes of the former marketing staff juxtaposed with the business reality was...quite a contrast.
It's especially sad as the basic idea was good and seems sustainable at a lower price point - with a metal/plastic housing it might have been an easy sell to businesses near subway stations, for example, and developing cosmetically different versions for other large cities would be quite feasible. I wonder what made them take the 'growth corporation' approach rather than crowdfunding the prototype > product route, which was fairly well established even 5 years ago. That offers a fairly clear roadmap for new ventures and is sufficiently familiar to consumers that innovators can do some market segmentation and have a cushion of patience to get them though the design to manufacturing transition.
A great case study for anyone thinking about launching a product for fun or profit.
Also seemingly not a lot of people paid 600$ for the sign, with a lot going for a lot cheaper than that.
But they were not aiming at the $600 price-point. That's just the estimated price-point that OP estimated from the BOM. They were actually selling at much cheaper than that, which probably contributed to their collapse.
As someone who would love to buy one of these and customize it for myself does anyone have experience or recommendations with any of these?
The other path is to buy plain led matrix modules and use some SBC to directly control what's displayed. More fun, but more work. This particular sign uses qty 2, 4mm pitch, 32x64 pixel panels, side by side. Like this one: https://www.aliexpress.us/item/2251832064290423.html?spm=a2g... Lots of choices in pitch, single/three/full color, indoor or outdoor brightness, etc. Though you need to ensure your choice of driver (see paragraph below) supports it. They have different refresh rates, data sequence, and timing requirements.
Then, you pick what to drive it with. Lots of choices there. There are libraries and hardware level-shifter adapters for a variety of things like a Parallax Propeller, Raspberry Pi, BeagleBone Black, etc. With various advantages and disadvantages. Fwiw, the Rpi is usually trickier as you're bitbanging from a non-real-time OS. The Parallax and BeagleBone have better ways to be quasi real time.
The panels themselves don't take data and hold it. You have to refresh the display constantly, typically on a 1/8 or 1/4 duty cycle.
You can drive smallish, single color matrix displays with a cheap MCU. Or there's also the approach of using addressable LEDs instead of these more "dumb" matrix modules, like arranging the smart strips into a matrix. Moves the refresh logic down into each LED.
But let’s start with a simple, quick hit of dopamine: There’s a web-based installer [1] (it uses web serial, so Chrome is required) for a couple varieties of clock designed for a 64x64 HUB75 panel and an ESP32. $20 in parts from AliExpress if you already have a suitable 5V power supply.
To use that site, all you need is that aforementioned 5V power supply—4A or more recommended, 10A if you’re planning to drive a full panel of pixels on an outdoor panel at full brightness—a 64x64 HUB75 matrix panel, and an ESP32. Everything except power is run directly to the ESP32. Many panels even come with a 16-pin female IDC connector for the panel, with individual DuPont-style female connectors at the other end that you can just slip over the pins on the ESP32.
The source code for each clock (repo is linked from the web page) is a great starting point for making your own versions.
The panels, as others have mentioned, are shockingly cheap from the usual sources like eBay [2] or AliExpress (search for “HUB75 panel” or “p3 64x64”), but even those in a hurry can get a 64x64 panel for under $40 from Amazon [3].
[0]: https://github.com/mrfaptastic/ESP32-HUB75-MatrixPanel-DMA [1]: https://clockwise.page [2]: https://www.ebay.com/sch/i.html?_nkw=HUB75%20Panel [3]: https://www.amazon.com/dp/B0B3F7WKJ1/
This would reduce cost to:
2x panels
1x colorlight card
1x power supply
Misc cables / case
Edit: The card can handle an 8x8 matrix of panels ( IIRC ) and the cards can be chained together to make larger displays. I dont recall the max size. The biggest I’ve made is 5x5 for a xmas display.
Edit: Ahh, that's a great option...assuming that reverse engineered way to send frames is stable, etc.
The display is done by sending the card network frames ( someone did a reverse engineering of the protocol, i dont know the details ). You can send the display anything you want.
https://hkubota.wordpress.com/2022/01/31/winter-project-colo...
great tutorials too https://learn.adafruit.com/rgb-led-matrix-cube-for-pi
We also made an arcade game with the LED screen.
Not the cheapest route but it’s relatively easy and fun to get it working
The sign wasn't the most problematic "mininum viable product" I've seen. If it were a rapidly developed first prototype, I'd even be impressed.
I would bet they spent significantly more than $3 on the packaging and miscellaneous BOM items. I was going to say the wiring harness alone would be much more than that, but it looks like most of it is included in the Adafruit panel kit.
The WayBack Machine has it https://web.archive.org/web/20230109144459/https://blog.kchu...
As does Google
https://webcache.googleusercontent.com/search?q=cache:71rSDb...
Even at $600 - there's a screenshot saying the company had three engineers and a marketer.
If your 4 employees earn $25k per annum, with the $150 parts cost you'd have to sell 222 signs at $600. And if the employees were on $100k, that's 888 signs a year.
That's a heck of a lot of signs, at the price. And you'd have to keep hitting those sales targets so long as you wanted the API to stay up.
It would make much more sense if the business was a one-engineer passion project - someone who had a day job and did a few hours a week maintaining this as a side project would have much more achievable sales targets!
833 subscribers at $10/month? 8,333 subscribers at $1/month?
If the HWE leaves for a bit, same thing. If he's making units production stops, but maybe the SWE can fill in since these dont look like difficult builds.
The third one designed the wooden case and the manufacturing process for it ... /s
One engineer makes a mess because there is nobody to push back. Two engineers either fight or achieve full echo chamber. Three balances some power dynamics, but you definitely don't need 3 FTEs. In fact for this project you probably find 3 of your friends' friends to sit together for 4 hours twice a week and slam out some code or designs, in exchange for a little money and some equity. You even get to write off food and beer as a business expense.
I actually built something very similar and all in my spare time. I've been wondering how realistic making a side-business out of it is, but if you are a single engineer that has to code the firmware, server, build the displays and market it all... it's pretty tough (especially in your spare time).
Hardware is a tough business and managing cash flow & inventory requires you to be a lot more deliberate than a pure SaaS business. Fortunately we had a lot of examples going in of how not to do things.
I just took a look around the tidbyt website and it looks like a fairly well polished product.
- "I hacked a Joy-Con controller to have a Capacitive Trackpad", https://news.ycombinator.com/item?id=34329927
- "Smartknob – an open-source input device", https://news.ycombinator.com/item?id=30646371
- "DVD Laser Scanner Microscope", https://news.ycombinator.com/item?id=26012652
- "A simple 11.2 GHz radio telescope", https://news.ycombinator.com/item?id=26078761
- "Home-Built Scanning Tunneling Microscope", https://news.ycombinator.com/item?id=26740968
(twitter @bettercaltrain)
Hint: if your product plans involve a raspberry pi, you probably fucked up.
It's more common in the pre-built devices to have a controller that's using a main CPU plus a FPGA or CPLD. The better hobbyist drivers use things like a beaglebone black and it's onboard "PRU" that can drive real time from memory it shares with the ARM cpu and Linux.
Also, since it's pulling train data, you need more smarts or cycles than a typical MCU to pull via WiFi/https, transform text into pixmaps, etc. Though there are examples of beefy MCUs doing all this well, like an ESP32. Or examples of small MCUs driving a small single color matrix.
Edit: Yes, you could even use an ESP8266, though 32x128x3 plus X bits of PWM brightness would mean being limited to drawing from storage rather than in-memory manipulation. I'd jump right to the WROOM type devices with PSRAM.
It's fine for prototype at a very early stage, absolutely not fine for something you ship to a user.
Am assuming here that they're adopting strategies to minimize/work around the problems you mention.
An integrated combination or even a board design offering from either of the two mentioned manufacturers would be awesome and really help bring down the barrier to entry for hobbyists that want to try moonlighting as entrepreneurs
Hardware academy is also good, where you can ask questions and meet other folks who build electronics products.
https://predictabledesigns.com/academy/
Happy to talk and give pointers, email in my profile.
Edit: I've no affiliation with hardware academy
However, for small-quantity, high-quality PCBs at a great price, once you have the design done, it's hard to beat OSHPark and OSHStencils for solder stencils. I highly recommend both.
"At boot time, the Config Server will pull the latest configuration from an HTTP server. In addition, the Config Server will connect to an AWS IoT Core endpoint to receive real time config updates from an MQTT server."
So, perhaps to bootstrap information they didn't have, like the current configuration? It sounds like they sold different types of signs with different resolutions, and also whatever train station setup config there was. Maybe a one-time exploit to upload that info from the sign itself.
There was only one type of sign but it did come in various different cases.
You are correct that with domain control I am able to serve content to any sign but the content will only be loaded once at boot time. Any future updates would have needed to come from their defunct AWS IoT connection (ignoring full restarts).
Using the exploit I remove the connection to AWS IoT and update some of the code to better connect it to the recreated API so users can update their signs in mostly real time.
There are only three parts in this product that should have any significant cost to them:
* the display itself,
* the wooden enclosure,
* the packaging the product comes in,
If there are switches, potentiometers or encoders I am really partial to well working ones and these are not so cheap. It is nice to have but at least they make the product feel more expensive so it is not a total waste.
Everything else can be done dirt cheap. I don't think chinese are the best way to learn product design but they are pretty darn good at saving every last penny (or yuan in their case). I found it illuminating to study some of the devices of chinese origin (as opposed to ones that are only manufactured in China).
Low volume (I was doing all soldering myself!), high sticker price, not especially profitable for me.
But makes it so much easier to design when you don't have to compromise so much and you don't have to be very inventive trying to save on BOM. And it is fun to be able to just do whatever you fancy.
I think starting mass market production for price conscious clients with no experience is just asking for problems.
It's effectively a picture frame. There are plenty of standard box frame profiles they are very similar to what they have built. If it was me, I would have explored working with a picture frame company for final assembly, many are used to "assembling" products with extra components (think of stuff on Etsy). The frames are simply cut with a framing guillotine and stapled together.
My estimate would be frame (including custom cut back panel), and final assembly would be in the region of $10-$15 at a US/UK framers. (Excluding electrical BOM)
Really what blocked me the most was CE/FCC conformance. I have done a lot of research into it and I couldn't figure out a way to get it without a massive investment upfront ($10k+). If anyone has experience in that area I would love to talk, my contact info is in my profile.
Who knows, maybe I will get it to a point where it can be sold eventually. I'll certainly write up about my learnings from the project at the very least.
my problem was squeezing the MTA API parser into the ESP32 flash...
as long as you use off-the-shelf / already conforming modules, like the esp wroom etc, the cost for FCC compliance testing goes down into the thousands. it's still a line item, but over a run of 1000-2000 units, it turns into something that costs about a buck per board :)
Ahh, I wouldn't do that. The way I've done it is by pushing as much of the API handling logic off the ESP32 and onto a standalone server. The ESP32 is effectively a dumb client which takes in instructions like "WRITE westminster 5 min" from the server and draws the appropriate text/pixels/etc.
> as long as you use off-the-shelf / already conforming modules, like the esp wroom etc, the cost for FCC compliance testing goes down into the thousands. it's still a line item, but over a run of 1000-2000 units, it turns into something that costs about a buck per board :)
yeah... it's $1 per board unless none of your boards sell :)
when kickstarter was first taking off, I loved using it to springboard only the projects that would at least break even: if there wasn't enough demand, nothing happens, but as long as the minimum is high enough to cover all the costs, it worked out pretty well
@dom96 thats cool! Ping me on the Nim discord sometime if you want. I've not done FCC but did do some hardware certification stuff. For low volume products there possible ways around full FCC certs I believe. IIRC, you can sell things as "kits". I wonder if theres also exemptions for "prototype builds" too.
The store that made the LED strip caught wind of it some how and they cut out a wooden enclosure for me to put the thing in, with the tube line names etched into the wood, which completely blew me away.
My dad kept on telling me at the time to sell it as a product but even then I knew that not being a hardware guy and not really having any experience building products it would be a dumb, expensive move - and stories like this reaffirm that this was the right thinking.
Not saying it can't be done, it's just someone with more electronics expertise is way more qualified than a guy throwing together a python script on a Pi zero with retail components.
would love to see the Pi zero and hacked together code on there!
I guess this more often the case than people realize (especially with such hobby level hard- and software), but it actually is a huge security risk. Because once you are on a Raspyberry Pi in a customers network, there are no limits..
Insane.
Get some digital picture frame out of Shenzhen, like this one, that wholesales for $30 - $50.[1] That thing has WiFi, a 4-core CPU, and runs Android. Reprogram it to just run your application. Sell for maybe $75-$100. The manufacturer will add your logo and do custom packaging if you order at least 50.
There are even cheaper discrete LED signs that talk WiFi, but those would be kind of intense for home use. Those you sell to shops near stations, so people can shop or eat while watching the train arrival info.
[1] https://www.alibaba.com/product-detail/IPS-Display-7-8-10-13...?
Does the cost scale somewhat linearly with number of LEDs or horizontal x vertical resolution in pixels?
$0.01 per LED
About $2 per 100x100 mm board if you order about 100 of them.
About $1 per above board for assembly service.
Total: $6 per 15x15 LED board of 100x100 mm.
Total: $12 per 30x30 LED board of 100x100 mm.
I have no idea about how many would you actually need.
As for the power supply, each LED draws 12 mA. So you are going to need quite a beefy supply.
15x15 grid = 2.7 A @ 5 V
30x30 grid = 10.8 A @ 5 V
You can also buy P5, P10 and other sizes. They number tells you how many millimeters between each physical pixel.
I use P10s for my Christmas display, they look good from about 15feet away or more.
Maybe if i play devils advocate it will add some value: People have castigated the high RPi cost (among others) and they are probably right about it, but there is something remarkable about been able to ship even a small number of devices with a full blown computer and a unique UI.
Maybe the more fundamental problem was not so much the hardware cost but not shipping enticing enough software to get people excited about the device?
Nice article overall :)
This company is a great example of why I don't.
Mad respect for the author though. It'd be cool enough to just get one sign working, but to take over the API and domain is pretty awesome.
https://ukdepartureboards.co.uk/ is the British equivalent of this, and it seems to do everything right (don’t own one, but have seen on Twitter)- optional subscription, and nice looking hardware (seems like they contracted out for that). Though at the minute I suppose there are barely any trains running here :)
You could buy a phone (moto e20), same size as their small display at 90e vs 140e for the board above...
Sell a nice android app, subscription and be done with it.
Phone packs a lot more features.
Last time I looked they were missing CE conformance, which afaik is pretty illegal. Surprised they haven't been found out yet.
The allure of having one be able to add features "with just a bit of code" is catnip to product managers. They seriously cannot seem to resist going there.
I can believe someone might prefer software for prototyping, but manufacturing? Well, I've seen prototypes enter production, so yes, but still, ugh.
Kind of reminds me of a small company I used to work for where I noticed shortly after joining that they 1. had no source control or reproducible builds, and 2. ship Debug builds with symbols and no optimizations to customers, because they couldn't get the Release configuration to run without crashing.
1: https://web.archive.org/web/20180107132644/https://www.devsh...
But hardware startups are brutal. Kickstarted hardware is torture.
Even experienced and well capitalized tiny teams go through this. A friend did one, rapidly realized the only option was to somehow get Chinese manufacturers excited about a small run of a strange product. Then two years later was trying to QA injection molds from across the ocean and switched manufacturers two times.
They delivered, but only after years of what must have felt like crawling over broken glass.
Maybe the NYC Train Sign team realized they couldn’t profit from the sign itself, but, unwilling to let the viral moment pass, were going to use it to establish themselves in consulting. You know… sell pickaxes.
Startups fail all the time, and hardware startups are a minefield.
If they had succeeded then we'd be singing their praises on how they started off with a scrappy product built from off the shelf parts and then managed to productionise it and outsource the manufacturing to china...
I modified the adafruit python library to read lines from Redis
then I have another script that makes the API calls, has a REST server, etc and publishes text to reds queues.
What a red flag... they basically had a backdoor to everyone's home network.
On reset it should still verify the server cert when it tries to call home, but since you have access to the image you can replace that (if they didn't stash it somewhere securely, that is).
The sign did use AWS IoT for real time configuration updates however initial configuration was pulled from their HTTP server. Using the vulnerability I describe in the article I just remove the connection to AWS IoT.
This is just straight up gaslighting. “That thing you said you did isn’t going to work”.
If you had read the post, you would have seen that they did, in fact, not “do things correctly”.
If you needed to update the CA, you'd need a firmware upgrade, served with a certificate signed by something chained to the old CA. And then, more likely than not, you'd want to do further updates on a new hostname, because it's hard to do a single hostname and send the right certs to the right clients.
This is a giant pain, and I don't blame people for using public CA infrastructure instead. Especially if your company goes bust, who cares?