Man who built ISP instead of paying Comcast expands to hundreds of homes (2022)
arstechnica.com
arstechnica.com
(Huge kudos for this project in general)
That said, I know our page isn't particularly lightweight anyway, I've been pretty focused on expansion efforts and haven't had much time to update & work on the site.
The backend is an azure appservice running on a single low-spec worker at the moment, wasn't planning on enabling scaling before doing a hard-launch but here we are.
This nerdsniped me.
10 minutes later, and TIL: MPD is some sort of streamed-MP4 format; dash-mpd-cli is a xplatform Rust utility binary that can download this to an MP4, just given the MPD URL in dev tools.
However I keep getting 1.5 MB and 500 KB for the two videos, no matter window width. Chrome on macOS arm64, 16" MBP.
I'm curious what your environment is, if you don't mind sharing
(also, trivia for audience: last week I saw a tweet that palantir.com was doing over 100 MB worth of videos, and of course, A) they are B) they're poorly compressed, as much as 10x the bitrate they need to be.)
In the left panel, at the end of the video you'll see two important numbers: 28.5MB transferred, 40.7MB resources. "Transferred" is the (compressed) size of everything downloaded over the wire, 40.7MB is the ultimate (uncompressed) size of those.
I don't show it in that video but you can filter by "initiator" to see that the video files are the lion's share of this.
If you used Ayva's fiber internet that video would download instantly =D
However, what is indeed slow is the initial load, and the lack of CDN for their static assets (css etc). When the HN effect started eating their resources, these static assets are what hurt their load time the most.
I dunno, (social) economics seem pretty sweet to me.
I suspect a many would jump at the opportunity as rural folks tend to be quite resourceful.
I've done this for churches for campus point to point. Don't have enough installers, but about 30% have volunteers that got it done or mostly done. Some we just did the Cat-5 terminations for them.
Does spending 30k per household connected make any sense?
The backup is great and shockingly resilient, nobody ever notices when it's in use and it'll usually still get us 300-600mbps of throughput on average. I'd say we typically see about 5-10% of the network revert to the backup radio during a major blizzard/hailstorm, otherwise everything seems to do quite well with normal snow & rain.
Over the lifetime of the structure (which can be upwards of 100 years)? Probably, just like the road, water, sewer, and electrical service to it. It’s another utility.
If it was already affordable to connect those houses there wouldn't have been a need for federal funding (not to say that I think tax dollars should be spent that way or that the program was run well).
That said, some ISPs have been known to collect hundreds of billions in taxpayer money on the promise of providing service only to just pocket that money without providing it anyway so for consumers, the real world impact of ISPs losing that money isn't so clear.
1: http://irregulators.org/wp-content/uploads/2017/05/BookofBro...
2: A more recent example from 2024: https://www.washingtonpolicy.org/publications/detail/the-42-...
Nationwide telecomms did that with funds for fiber to the premises as well. I have no idea why these telecomms can take the money and just refuse to provide the services.
Rural programs are really successful, until the money runs out. Cities get screwed though — I served on a commission with my city on this topic and the only feasible competition at scale, Verizon, refused to engage unless the city allowed them to pick individual blocks to deploy, which means no servitude for underrepresented people.
This was 20 years ago now, but the service was very reliable using Motorola radios. Relatively low bandwidth (4-6mbps, not bad for the day) but you could on a good day do that at a few miles out.
p.s self-plug: for our senior year capstone we are working on a secure/private home router firmware. Since you are in this space (tangentially) and local, I would love to chat with you
God I wish that was me. Xfinity has a raised middle finger where the privacy policy should go.
It's not crazy, but it's longer time horizons than a business would ever care about.
That's not much different than what it costs to connect them to power grid or the water and sewage system.
Side note, truly inspirational an something I would love to do in my little village in Ohio.
I was very pleased (before making an offer on the house) to find out about the cable internet as I have worked remote for 11 years. But the ISP is Spectrum, which has historically been a very shady business (like most Cable companies, ISP's).
I also run a Mofi router with a SIM card as a back up network.
Year 1: $52/mo Year 2: $74/mo Year 3: $98/mo
I have made no changes to my account.
This is why we need choices for ISP's.
Also, this is a highly highly resource-dependent website. Consider a scale back. It'd be a funny tongue-in-cheek thing if you made it super encumbered and say "Our customers can load this page just fine!", but it's counter-intuitive for everyone else haha
The first information is fine (say, speed) but when I switch to latency the graph does not change (and BTW it's not readable on mobile)
Same for the coverage
Does long-range ubiquiti just work because you’re in the mountains?
I’m down in much-flatter Columbia South Carolina. Would a similar setup even be physically feasible?
We had a company burying fiber, but their installers almost blew up a gas line during 5 o’clock traffic and it really pissed city council off.
Someone, somewhere says that they built something for a local community and suddenly Joe from Sydney and Marie from Bordeaux are on the site, discussing its tech stack and comparing the pricing in Wakanda.
Great site.
I see all your replies to people; have you considered that running a video on a website is just completely unnecessary?
Video background and scroll to animate stuff was cool...in 2013. I think people appreciate a faster loading page with up front information these days.
But you do you I guess! :)
I have 1GB Sonic bidirectional fiber with unlimited data and could get 10GB if I wanted. The head of Sonic points out that long-haul prices have decreased over the years, and there's no real need for usage limits.
> Sonic has the best customer service of any company ever encountered, and it's not even close. The few times I've had to contact them for assistance, I've been very quickly connected with someone clearly _very_ technical who was able to grok my problem immediately and give clear, cogent, respectful debugging advice and perspective. I do not exaggerate when I say I would gladly pay double their current rate just for the peace of mind of knowing that I can depend on them if I ever need their support again. Not that I often do, because their baseline connectivity/speed is also great.
>
> ...yes, I know I look like a shill/bot. I don't care. They're genuinely just that good, and I will happily advocate for them until that ever changes.
Properly good, human and effective customer service is so rare these days that I tend to trust hyperbolic reviews like these more than others, believing that the person honestly feels that way, having been there myself.
Well, aside from increasing your margins...
The installer was super nice and great at their job.
Their service is so good I have not had an excuse to talk with anyone else.
Many of my neighbors have switched from Comcast, who I was with for more than 10 years, and hated every second of it. Only AT&T is worse than Comcast, but they are both bottom dwellers.
* https://www.youtube.com/watch?v=ASXJgvy3mEg
* Slides: https://nlnog.net/static/live/nlnog_live_sep_2020_jared.pdf
* 2020: https://news.ycombinator.com/item?id=24424910
* He posted some replies in that discussion: https://news.ycombinator.com/user?id=jaredmauch
Involves trenchers. Also, at NANOG, "Starting a Telephone Company in 2019, or How I Built Fiber to my Neighbors":
My story is in the opposite direction.
We and everyone else in the neighborhood had symmetrical 1 Gbps installed about 15 years ago. We all paid the ISP for the top tier of full capacity.
During Covid decided to take inventory of our actual bandwidth needs.
Anything that can be deferred doesn’t count. Gone from instant bandwidth requirements are all cloud backups, OTA OS upgrades and apps updates. They need to complete overnight. Overlapping is not a requirement.
Videos are automatically played at 480p or less on iPhones, 720p or less on iPads and 1080p or less on HDTV. We purposely didn’t buy 4K TV because at our viewing distance has no benefits whatsoever. Aggregate peak bandwidth required here is 25 Mbps at a stretch. That is also enough for my wife to work from home.
We don’t deal with large datasets or raw videos over the internet.
So we found ourselves with one cable connected TV and the usual assortment of mobile devices connected to one WiFi 4 1x1 hotspot. At 70 Mbps we never noticed any loss of quality in our digital lifestyle.
After about ten years we replaced the hotspot with one capable of WiFi 5. An overkill but needed the extra port.
Eventually convinced the ISP to lower our subscription to the lowest available tier of 200 Mbps. We don’t notice any difference. We could afford the extra bandwidth. But don’t see the benefits of it.
That's also why a lot of supposedly fast ISPs absolutely crumbled when COVID hit. A lot of people started doing video calls in the morning/afternoons, which suddenly sent latency-sensitive, bidirectional, high-bandwidth data to every corner of the network. Upload speeds collapsed, gigabit networks were struggling to hit a couple hundred mbps, and DSL providers downgraded their customers to 2005 in terms of attainable network speeds.
For that reason, I think ISPs may as well offer 10gbps as a default. Their customer base is not going to make use of that capacity anyway. Only when downloading a new game, or doing a backup, or uploading a video file somewhere, does that bandwidth become a necessity. If you remove the cap on the bandwidth side, all of that capacity will remain available for a longer period of time for all of the other people in the neighbourhood.
Some cellular providers used the same reasoning for their plans here a few years back: there were no 4G speed caps, just upload/download as fast as you can, because if you're done doing your file transfer quicker, you're clearing the airwaves for other users. Of course, you'd still pay for those hefty bandwidth caps, charging >€1 per GB per month to rake in the cash.
I am not sure maximizing throughput to gigabit and beyond is materials efficient though. Fig 33 shows energy efficiency is squarely on the FTTH side anyway: https://europacable.eu/wp-content/uploads/2021/01/Prysmian-s...
A software update completing download in an hour instead of one minute often doesn’t lead to any practical difference. The same number of users is being served but the latter requiring gigabit class CPEs.
Alas it does offer some up selling opportunities. As promoting Cat 6 or even 7 cables to home users.
You're thinking with a technical mind, but don't forget the business, marketing, legal and support.
Why have "unlimited" speed as baseline while you can charge some 10 times more for the privilege?
Also don't forget that if you sell "10gbps internet" you might have to legally guarantee a percentage of that. Or you have to explain how networking works to everyone who complains they never see 10gbps on Speedtest.net.
Also the more you offer, the more expensive your modem has to be.
My parents have now 150/30 Mbps. Totally unaware of it.
extra bandwidth if good to have just in case and will save time on large downloads.
ISPs perhaps decided it was commercially more convenient, for the industry at large, to cross subsidize demanding applications by providing everyone with gigabit class connections and CPEs.
Guess what? It's still not enabled. AT&T only did it because there was a risk that Google Fiber will do it in our city. Unfortunately, IIUC, Google never could overcome local regulations and abandoned the project. So, AT&T didn't care to light up their fiber (that was already in the ground & ready to go!!!).
Comcast doesn't offer any cable in my location neither.
I've been seriously tempted to do it myself too, but doubt I'll ever have time for that - mostly to overcome the local bureaucracy to get all the permits...
Huge respect to Jared!
Man who built ISP instead of paying Comcast $50K expands to hundreds of homes - https://news.ycombinator.com/item?id=32411493 - Aug 2022 (427 comments)
%PASTLINKS=32411493%
And someone forgot to turn them off for all users and they were discovered ... HACKED! haha
In most US places, utility companies own the poles and it’s ridiculously expensive to lease space. Urban areas with competitive ISPs usually have government owned poles or leases for streetlight arbs that allow them to string fiber.
I believe the standard explanation is that most of the colonists were British (already a high-wage country at the time), and you really had to pay skilled labor to get them to leave & settle on a new continent. Plus labor mobility between the proto-state governments of the time (Virginia, Massachusetts, etc.)
Internet pricing is a scam in the USA.
Obviously no excuse to claim it is unlimited, but if the major US cable companies speeded up moving to true FTTH it would really save them a lot of trouble in the long run.
I was on 1000/40 for most of my history with them ($100+$50) now I have 2000/100 ($150+$50). I would be fine with 40Mbps upstream unlimited; the issue is not the throttling but the threats resulting from bait-and-switch.
I switched to fiber a few years back. But at one point during covid, my cable modem upstream was getting less than a megabit (I was paying for 500/30.)
Compare that to docsis 3.1 - maybe 200mbit/sec between hundreds of users.
DOCSIS 4.0 is better but just basically brings the split much closer to the home, in which case you'd probably be better off finishing the job with FTTH instead of having thousands of expensive active DOCSIS terminals every few hundred metres.
Also, you can easily upgrade capacity on FTTH, just add a new OLT and you can run that side by side with the old one while you upgrade everyone's ONTs.
Regardless you're missing the point. DOCSIS has maybe 100-200mbit/sec of upstream shared between hundreds of homes - this will vary depending on config, and keep in mind a lot of that will be used for TCP ACKs from the downstream. So you probably have say 50-100mbit/sec "real" upstream available, or less than 1mbit/sec per subscriber.
If you hammered 40mbit/sec hard you are using nearly the entire 'usable' upstream of your entire cable modem segment.
DOCSIS is just massively inferior to FTTH for this reason.
I remember paying about 10$ for a proper gigabit in Russia. Probably a perk of living right next to an exchange point.
I'm paying £40 a month for symmetric gigabit to my home, which is a house, in a suburb full of other houses, with no apartment blocks in sight.
Typically the US only has one. Two if you are exceedingly lucky.
https://www.techdirt.com/2024/11/07/16-u-s-states-still-ban-...
Considering the company I worked for didn't even serve the community I lived in but multiple startups wanted to provide us fiber service but ran into all the road blocks my telco and others pushed, I was less than thrilled knowing the company i worked for was actively trying to block providing better internet to older neighborhoods.
Buncha jerks.
I was in one of the earlier roll-outs and the change was amazing! It actually did happen during the pandemic, and at one point we had 4 people doing video calls from the house and Xfinity's fastest service couldn't keep up (because of outbound bandwidth limits), but the asymmetrical gigabit fiber wasn't breaking a sweat.
Now 3+ years later, CenturyLink (q.com) is finally starting to come through and lay down fiber. Those tools should have been laying it 20 years ago.
These rules don't keep people like the subject of this post from deploying community ISPs.
(I think the rules are very dumb but also the economics of small-scale municipal ISPs are not good at all; part of the reason they're dumb is that in reality a typical suburb has absolutely no hope of competing on price with Xfinity, Verizon, and AT&T.)
See also: banking, healthcare
https://coloradosun.com/2023/05/24/municipal-internet-sb-152...
However, I really hope that more small ISP's get their shit together from a cybersecurity perspective. They are generally completely apathetic on the subject.
(Quick but AI-sourced summary: https://chatgpt.com/share/68004134-f9b0-8008-9b77-f43d1750c5... )
Ubiquiti has made huge leaps and bounds in the last year in terms of software and hardware. Realistically Ubiquiti Network could be used at the enterprise ISP level.
- 48 port 25G SFP28 switch with hot-swap power (https://store.ui.com/us/en/category/all-switching/products/e...)
- Gateway / router with 25G SFP28 WAN and LAN with hot-swap power (https://store.ui.com/us/en/category/cloud-gateways-enterpris...)
Websites like these tend to win subscribers. My ISP was the same when I subscribed.
This costs $500 in Australia in the inner city.
I’m on Leaptel 1000/400 for $166/m and it’s fantastic. Fingers crossed they offer the nbn multi gig speeds later this year.
I'd pay that. I am stuck with 2mb ADSL living in the centre of a major city in Scotland.
(tbf when I was more central dundee it was closer to 20/10).
I'm in the Silicon Valley and have multiple ISP options (although AT&T is the only 1000/1000 option).
I guess our prices stay low because if they went too high it would motivate their competitors to move in.
I’m with Zen and paying them, I think, £65/month for their 1.6Gbps package (they don’t offer 1.8 for some reason).
I am only paying for 500Mbps but could get 2Gbps if I wanted.
In the case of the linked article project, however, I suspect there's limited impact from Starlink. The linked article was a subsidy project. The subsidy is to provide fiber in places that don't have landline Internet options, so it's still eligible in spite of Starlink. With subsidies, Starlink isn't hard to compete with due to its price (customers would be rather irrational not to switch to a $55/mo service that was comparable to their $80-$120/mo Starlink).
maybe you don't have starlink?
the variance in latency is dogshit, especially in the evening when everyone in the area hops on netflix :) and i don't even play online games.
Maybe you haven't had a crappy boutique WISP? They're genuinely awful usually.
My house isn't served by any landline provider (frustratingly, it's also _almost_ in a city, so there's no hope for subsidy access like the linked article), so over the years I've had a mix of corporate WISP (Rise), boutique WISP, and LTE as well as Starlink. I also work with a fairly large geographically distributed deployment of Starlinks professionally. Overall I would rate Starlink as excellent in the past ~9months or so.
It's been a pretty rapid improvement - a year ago I'd say Starlink was fairly, to use your phrasing, "dogshit," but now I'd say it's better than all competing wireless solutions I've used for moderate throughput needs (<100mbit) when the whole package is considered.
Good, well engineered fixed point to point ISM wireless can certainly be much faster and a decent amount lower latency but is a pain to install correctly, and licensed microwave is obviously superior at the massive overhead of, well, licensed.
On the other hand, Starlink is so easy - set the service address, plug it in, and a few minutes later it's Just Working with a nice /56 worth of IPv6 addresses.
In my experience WISPs usually take weeks or more to install, have terrible neteng, struggle with IPv6 and even public IPv4, deliver frequent backhaul outages, and often end up with bad enough buffer bloat to somehow deliver inferior latency to Starlink. For semi-mobile solutions like construction trailers, command centers, or even sensor telemetry use cases where LTE isn't available and you don't need a whole LoRa setup, Starlink is a no brainer compared to a WISP. For residential service it's certainly a last resort, but a superior last resort to all but the very best WISPs.
One thing we have noticed is that the plan prioritization is VERY aggressive in oversubscribed regions, though. The Mobile and Lite plans get quite aggressively chopped off compared to the Residential/Enterprise/Priority plans.
If they have half a clue regarding marketing and networking, they are doing fine. Starlink doesnt offer Layer 2 or Managed WAN options (Possibly Vocus is bringing these projects out at some stage on their behalf)
In dense areas, starlink underperforms. In larger cities Fibre is beloved. Theres a wedge, where WISPS are king and still are king, where the density is just right.
That said, if you are running a really shitty wisp, and you dont have any business links or complex services. And half your customer base just bailed for starlink, you will likely fold. But honestly, WISP as an industry can do without the cowboys.
This is the entire reason that the US forced Bell/AT&T to allow any company to lease a connection to their network. Without that, it would have been flatly impossible for any new provider to compete. There is simply no way that anyone could have ever built out a network to compete with Bell.
You can probably lease a connection to the fiber network in the same way (I haven't checked, but I assume common carrier applies), but if there's no fiber to the addresses you want to service you're SOL unless you want to front tens of thousands of dollars per customer. No ROI on that for many, many years.
Your only recourse there is to additionally stand up DSL/DOCSIS as a 'last mile' connection between the customer and your fiber. At additional unbelievable expense.
This may be true in some areas, but it wasn't in mine. I didn't have to agree to anything; it was negotiated with the town, not me. I presume they used the existing utility/sewer easements.
If it was $50/mo, and 20 customers, that's only $1k/mo, which I'm not sure would cover a fiber backhaul...
They probably should have found a small telco or three to buy for expertise on pole bureaucracy.
Responding to your parent...
> You can probably lease a connection to the fiber network in the same way (I haven't checked, but I assume common carrier applies), but if there's no fiber to the addresses you want to service you're SOL unless you want to front tens of thousands of dollars per customer. No ROI on that for many, many years.
I would assume mandatory line sharing doesn't apply; the FCC walked back almost all of the 1996 Telecom Act line sharing; telcos and cablecos may well have designed their fiber to the home in ways to thwart what limited regulation was present anyway. If you're near 'commercial' fiber though, lots of that is available for lease.
-Trenching and directional boring equipment is expensive. Setting poles to run overhead fiber on is expensive.
-Restoration of green space, asphalt, and concrete are all expensive
-Easements are time consuming to negotiate if there isn’t a utility easement available to use
Satellite is - once you've got the infra up in the air - very straightforward, with the downside that your satellite ISP is likely owned/operated by an unregulated billionaire nutcase that will turn off your access if he doesn't like you any more (c.f. Ukraine front line). It's hard to do that with regulated fiber backhaul, but not impossible.
I've seen a few wireless ISPs mentioned here before, which can be a nice hub/spoke model - run fast fiber to a community, but distribute via wireless (note, not WiFi) to homes and businesses within range.
I'd definitely love to see more community-run ISPs in the World, it's how the Internet should work, really.
Is this actually true?
https://en.wikipedia.org/wiki/Starlink_in_the_Russian-Ukrain...
> In 2022, Elon Musk denied a Ukrainian request to extend Starlink's coverage up to Russian-occupied Crimea during a counterattack on a Crimean port, from which Russia had been launching attacks against Ukrainian civilians; doing so would have violated US sanctions on Russia. This event was widely reported in 2023, erroneously characterizing it as Musk "turning off" Starlink coverage in Crimea.
This is the level of political bias that exists where facts have to be suppressed to fuel the narrative and the agenda.
https://www.theguardian.com/technology/2023/sep/07/elon-musk...
[EDIT: Deleted quoted text from above, because I missed (!!!) the retraction from the cited primary author ]
> Update: on 9 September 2023, Walter Isaacson said his biography’s claim about Starlink and Crimea was based on “mistaken” information [see footnote]
The footnote:
> This article was amended on 14 September 2023 to add an update to the subheading. As the Guardian reported on 12 September 2023, following the publication of this article, Walter Isaacson retracted the claim in his biography of Elon Musk that the SpaceX CEO had secretly told engineers to switch off Starlink coverage of the Crimean coast.
Says a lot about how shit the competition is if they’d rather take a chance on a startup
Do keep in mind, however, that residential service is usually provided by GPON or XGPON. Instead of a single fiber running from every single customer to the ISP, optical splitters are used so that up to 128 customers can share a single fiber. The fiber is run at 50Gbps or 100Gbps or whatever, and the individual customers share that using TDMA. That does somewhat limit the available headroom. Speaking just for myself, I would be entirely satisfied by a 1Tbps slice of that 100Tbps of available bandwidth.
For one, Starlink won't get anywhere close to 1gbps
https://mynews4.com/on-your-side/ask-joe/ask-joe-usda-cuts-t...