Hacker News Reader

Comment by geozimm | Hacker News Reader

All these things are related. Back in the 1990s I worked with others to develop DSL technologies (ADSL, HDSL, VDSL) for telcos. Why not fiber, some ask? oh, fiber materials may be sand, but installed fiber is sand, energy, and labor. Installation costs trump everything - use what you have for the physical medium.

While we (PairGain) also made point-to-point private links, it was clear then, and even moreso now that most DSL involves a service provider managing the network - with specialized skills, practices and setup.

In the 2010's I was spending time with the Ethernet folks, and in particular industrial folks with large plants. I ended up chairing IEEE Std 802.3cg which developed the 10 Mb/s 1km technology. Not really a speed increase - more an application refocus. As the networking world developed, many realized that converging networks above the physical layer added network complexity and therefore setup and ongoing operating costs... So we now also have SPE - pure Ethernet at DSL-like distances...

Similar tech with different use models, enabling connectivity for whatever...

I'm not Al Gore - I won't claim to have invented the internet. I just an engineer who happened to have a hand in both of these technologies, and am still pleased to see that people use them.

View on HN

What a blast from the past, DSL was an exciting technology.

I did the config and networking of DSLAMs for the first private test installs of Paradyne (Hotwire) DSL hardware in a western state around ~1998-1999, a couple years before DSL really hit the mainstream.

The telecos were slow to move on the technology and didn't do their own centralized rollouts until several years later. We took full advantage of that lead time time to win away a lot of ISDN customers with much lower prices and much faster service. We also saw end of that advantage coming and got out of the business before we had to compete directly with the telecos who could operate at scale and do bundled pricing of the line + service that was harder to compete with. At the time high speed internet over cable was also a while off for the general public.

We also had a tip from a teleco employee that we could use DC Signalling channels / dry copper pairs instead of regular phone circuits (no dial tone, they were meant for alarm service). They cost about 1/2 to 1/3 of the price.

At one point we petitioned to get the local telco to dig up the sidewalk and install several hundred more copper pairs into the building.

It was really interesting to watch the internet access landscape change so quickly.

Cool to see you here on HN. I hadn't heard the name PairGain since I worked for a small corporate ISP in the early 2000s. We'd recommend PairGain modems for clients who needed seriously high-speed links of 2Mbps! This was just before ADSL and SHDSL were rolled out en masse, or at least well before they were reliable enough for corporate use. We had to organise a special installation of a direct copper line from their premises to ours. I guess they just patched them together at the exchange? It was a pretty small catchment area. Fun times!

The tariff in my locality allowed for dry copper pairs to be installed ("burglar alarm circuits") and some of my Customers took advantage of that along w/ PairGain devices to get high speed links between sites serviced out of the same central office.

Relevant article: http://helices.org/commentary/dry_copper_pair.html

I can't tell exactly from a quick scan of Cringeley's commentary... but I get the impression that a "dry copper pair" the single-pair POTS equivalent of "dark fiber"... with the critical caveat that a dry copper pair can link two points only if they are serviced by the same central telephone company switch.

A leased loop is any arrangement where a customer pays the telco for a dedicated loop for continuous use. A dry loop is a type of leased loop that isn't connected to any telco equipment, just spliced together to create a continuous circuit (a typical or 'wet' leased loop has at least battery and sometimes dial tone from the telephone switch). It's perfectly possible to get a leased line that spans telco offices, in which case your loop has to be spliced to an interoffice or toll lead (long distance line). In the days before heavy use of multiplexing, toll leads were scarce and so interoffice leased lines were extremely expensive.

Dry loops were often sold as burglar alarm circuits because one of the most common uses was for burglar alarm communicators that often operated on polarity reversal - meaning that they applied a potential to the pair and just swapped its polarity when an alarm condition occurred. Of course there were more sophisticated burglar alarm communicators at the time that used telegraphy techniques, but these usually ran on private networks since they could share a bus in a way that was not typical of telco infrastructure (most of these were basically Gamewell systems even if not made by Gamewell proper). In the early days of burglar alarm monitoring, if the monitoring service didn't have a private network (typical in urban areas) they usually installed their monitoring equipment or a multiplexing system at each telephone exchange in the covered area, allowing for more economical dry loops within a single office. Actually this pattern continued well into the '90s with some burglar alarm services using DSL-like high-frequency digital communicators that interacted with a monitoring system that had to be connected to the line card at the telephone exchange.

Thanks, that explains it quite nicely.

1990s AppleTalk networks could work over a single pair. I ran a network with hundreds of Macs for a tech company. Ten buildings in a small campus.

Still remember the day the fire-alarm techs punched down over one of the Macs in the executive office. Not much of a network signal when there's 24 Volts on the same line.

Fiber doesn’t conduct electricity, a wire does. I remember reading somewhere to be careful with running a wire over any long distance between two houses that are not connected to the grid with the same meter. Something to do with phase or different potential, can’t remember, but the point was it could fry your equipment. Fiber sounds a lot safer to me.

The biggest problem is lightning strikes hitting the cable. The phase wouldn’t matter - and even different earthing wouldn’t matter as long as you only connect any shield at one end.

But fibre is so much more versatile if you’re running new cables (unless you want to run power)

... also lightning strikes hitting the ground somewhere in the area. That can induce large voltage differences between the power grounds on the buildings. The voltage difference might be large enough to arc across any isolation provided by the physical network implementation.

My neighbor's house was hit by lightning last year and it took out the ethernet ports on two of my devices. Nothing else was affected though. Those devices still work. They're on a PoE switch (though not using PoE) so that may have been part of the cause.

Then you just install a lightning arrestor on your comms circuit - I have a mast way up the hill to provide our connectivity here, and use one on the Ethernet line down to prevent issues (like a house fire) from a strike.

There's really no "just" concerning lightning protection. You can just add some protection, to code or above, but it may not work. Nature can be unforgiving.

You have to assume a direct hit by lightening will fry your hardware, period, full stop - proper grounding and lightning protection however will mean that the hardware does not catch fire.

There are lightning protectors that will absorb a direct lightning strike. Most antennas on hilltops and tall buildings have them. They take lightning strikes routinely. Here's some ARRL material on lightning protection.[1]

It's not difficult, but it's not miniature. A classic design was a soup-can sized device with a coax connector on each end and a hulking big ground connection on the can. Inside was a spark gap with dime-sized silver contacts, and a few turns of copper busbar as an inductor to smooth out the spike that got past the spark gap. That goes where the cable enters the building. Similar units today tend to be smaller. There will still be serious metal boxes.[1]

You need a serious ground. As in hulking big copper cable to a long ground rod. Grounding to a pipe is no longer allowed; there might be plastic pipe somewhere in the system, either now or in the future.

The next stage is a "central office protector". This is a gas tube with three terminals - both sides of the line, and ground. So it's an enclosed spark gap in an inert gas. An overvoltage will ionize the gas and short it to ground. Telco central offices have one of those on each line. They're plug-in devices that sometimes have to be replaced.

There's a lot of obviously fake stuff on eBay and Amazon in this area. 2D logos superimposed on curved surfaces, even. There's a standard, UL 497B. If it doesn't have that certification, don't buy.

[1] https://www.polyphaser.com/search?Category=Data+Surge+Protec...

[1] http://www.arrl.org/lightning-protection

Well, Howdy!

I'm a licensed radio amateur since 1996, I've spent about 20 years working in the cellular/telecom/two way radio industry, and I've done Motorola R56 inspections (as well as other proprietary grounding standards).

I respectfully disagree, a direct lightning strike almost certainly will take out gear at a cell site, even when properly grounded. Similarly a direct strike to telco cable will certainly fuse the 16-20ga wire in the cable itself at the first point its near a ground. Carbons, Glass Tubes, and other similar hardware will protect you in the event of a nearby strike (like to a lightning rod on a tower, or building) - but wont save you if the infrastructure is struck itself.

Generally the point of lightning protection systems is to well ground the tower, to draw the lightening away - so the tower and grounding system can protect the equipment - that isn't a direct strike by what I'm saying here - a direct strike would be if it struck the antenna itself.

Thats the perspective I have from cleaning up from strikes at well grounded and protected tower sites.

Yes, few antennas really need to remain operational against direct hits. Nor do they usually need to be the highest thing on the tower.

Data cables aren't usually up that high, fortunately. Power cables, though, are. In some areas high tension towers carry a ground wire between the peaks of the towers for lightning protection. It's impressive to see those systems take repeated direct hits without the lights even flickering. I've seen that in Florida.

Worst case is probably is an AM broadcast station where the tower is isolated from the ground at the base. WSM in Nashville TN is like that. They had a pipe ground vaporized and windows blown out in a lightning strike in December 2019. They lost the tower lighting and some transmission components were damaged, but they apparently stayed on the air.

The Empire State Building takes about 25 lightning hits a year. I wonder what their lightning protection looks like.

Clearly you have never seen a phone line fried by a direct lightning strike before: the cable vaporizes and blows up the ground over top of it. Direct strikes are rare enough that most people will never see them. On overhead lines the neutral and communications strand are grounded and will take most of the strike over the twisted pair communications cable. Underground cables have some benefits by being in non-conductive conduit, but none of that matters if it's a direct strike. All insulators will break down in a strong enough electric field.

There's also the issue of ground bounce. A lightning strike near a house will feed back into the telecommunications and power equipment via the ground rod/plate. I've had experience with plenty of modems getting fried over the years. Some places are just lightning magnets.

Last summer the local cable company replaced all their cables in town in order to begin offering digital cable and internet service. I was flabbergasted that they they spent all that money on linemen but still ran coax rather than fiber. As far as I can tell talking to their linemen, its not even FTTN, just FTT-central-office.

Assuming its not run by morons, which I'll accept is a bit of a stretch for a cable company, there must be some other reason to not run fiber for new installations.

Cost and Fragility.

You can push around 1GHz of bandwidth on the normal hardline-feed line with taps system cable uses, each node is designed to pass by a certain number of households.

Coax is.. cheap, forgiving, easy to terminate, and inexpensive to replace - Fiber is more expensive, unforgiving, and much harder to terminate.

Regarding the termination: our local fiber provider handles the termination with some optical precision connector (forgot the name). Both to the sunken-in-sidewalk multiplexer and in the home to the optical termination point (both gpon). So for mass deployments fiber connections do not require fibre welds are not required.

I have to see it play out in practice and I'm not a fan of the idea that one telco controls controls (ie stifles competition) in a gpon scenario. The conduit has recently been placed in our street, so "soon"...

I'm still a fan for cost reasons of FTTN, because I think with Coax in the last mile, you can deliver fantastic performance, so long as you're not also trying to delivery video too.

Furthermore if you actually run Coax in duct for buried circuits, its easy to replace with fiber later.

Our telco converted all its infra to FTTN via fiber. So, I've actually have fiber connection up to the front of my building, then it's terminated and distributed via VDSL to the street.

I have a 50/8 mbps connection at home and, it gives all the performance it can give. The telco keeps the speeds a bit higher to handle VDSL overhead, so we have a real 50/8 mbps IP connection at premises.

I'd rather not rewire my home and use existing equipment (which can handle 350mpbs), rather than bringing in fragile fiber into the home.

This sounds like post-hoc rationalization.

I have Sonic fiber in SF. 1Gbps symmetric, over a "fragile fiber" run directly into my home. It works quite well. The drop cable is pre-made in standard lengths with weatherproof connectors. The glass is embedded in a large-ish diameter substrate that resists sharp bends naturally so the installers don't need to take special care to prevent losses, just don't try to force the cable to bend beyond what it wants to do (very different from your standard fiber patch cables in a switch room). It is robust enough you could cable staple it to a wall without issue. Terminates in a tiny ONT that gives me Ethernet on my side.

They're deploying 10Gbps for all new installs and I'm eagerly awaiting my upgrade. No change to the fiber itself are required, just swapping equipment on both ends. This same fiber can do 100Gbps in the future if the need arises, possibly more. No coax plant can come close. The fact that an independent ISP can do this for $40/month and make money at it proves the economics.

There is no reason not to run fiber unless you're more focused on rent extraction than investing in your business... at least in suburbs and cities. (See ATT's public comments and focus on milking wireless while dis-investing in physical plant as an example of goosing profits because they don't face real competition in most of their service area).

Does Sonic own their own fiber network?

Yes.

They own some fiber. They'll also sell you service using AT&T's fiber if that's what's there.

Sorry I should have been clearer: I am referring to "Sonic Fiber" which is their own infrastructure, not any kind of resold service which they do offer in areas where they have not deployed their own fiber infrastructure.

In the Bay Area they have fiber in SF, South SF, Daly City, Redwood City, Berkley, Oakland, San Carlos, Burlingame, Albany, Brentwood, and probably more. They are rolling it out mostly in areas with poles at the moment because the cost of burying is quite high.

They've proposed "microtrenching" in SF to reach areas with all underground utilities but so far the city hasn't been very interested and it isn't cost-effective for them to dig up the sidewalks to install conduit. I hope that will change at some point.

> This sounds like post-hoc rationalization.

Uhm, no, it's not.

I'm using a 1gbps symmetric connection at the office, for the last decade or so. The network speed is limited by my NIC and the cabling to the switch. The network at the office has more bandwidth (we're the network backbone).

We have fiber ran into our apartment buildings. My apartment's termination box is at the wall across my flat door. On the other hand, the speed I can get from that fiber is not higher than the current VDSL offerings, and they both cap at 100mbps downstream (upstream is probably limited at the same speeds with VDSL). Since the FTTN box is also outside, the speeds and stability from that VDSL connection is rock solid.

For no apparent speed advantage, I need to terminate a thick fiber, and need to run it in the open across the house, drilling walls in the process, or move my house's whole internet infrastructure near it. Both are illogical given the floor plan of my house.

Then comes the equipment part. Again, I'll need to change my core router at home and change everything (I have a mesh network at home), or cascade it to ISPs fiber router, which is another box, more cables, and more management. If the ISP allows me to use my own router, I'd need a media converter from fiber to copper. Which's again more cables, more boxes, more management.

As a result, I'd rather use my house's in-wall cabling to get the speeds I'm happy with instead of getting a shiny (pun intended) technology with no speed advantage.

If the speeds offered here changes over time, I can re-evaluate my choices, but as the xDSL technology gets better, I'm guessing that it'll keep the same speeds with fiber offerings, at least for residential stuff in my area. So, I project that I can upgrade my network speeds at least for a decade without changing my network equipment or cabling.

https://www.chorus.co.nz/tools-support/broadband-tools/broad...

New Zealand has fibre to the home for the majority of a country the side of the eastern seaboard of the US, with only 5 million people, and a lower per-capita GDP.

It's doable. It's just a question of wanting to.

Its just not as efficient, you can deliver similar classes of service by HFC networks.

Australia and New Zealand have helpfully agreed to provide a case study in which works best. Mixed fibre and copper in Australia is slower and ended up costing dramatically more than it was supposed to.

I think it's just inertia. At higher speeds you have to do so much signal processing going over copper, that it iss costing more and more energy compared to optical. Which is limited to 2.5 Watts per port and end of the fiber, at least in common prosumer facing gear. While you can push up to 80KM(or meanwhile even more!) in one run with such stuff, depending on the used fibre and wavelength.

Speaking about optical modules in this formfactor: https://en.wikipedia.org/wiki/Small_form-factor_pluggable_tr...

I was initially a skeptic of the cost of Fibre-To-The-Premises, but in hindsight for New Zealand (≈Oregon) I think a national rollout of fibre was very effective[3][4].

AFAIK fibre is more resilient to catastrophes (like earthquakes[1][2] in Christchurch or California), and having a high speed residential fibre network definitely helped during Covid.

I am uncertain what you mean by “efficient”. Perhaps link to something that backs up your opinion?

[1] https://az659834.vo.msecnd.net/eventsaircancprod/production-...

[2] https://www.researchgate.net/profile/Sonia-Giovinazzi-2/publ...

[3] https://news.ycombinator.com/item?id=29791864

[4] https://news.ycombinator.com/item?id=25339491

When AT&T did a fiber overlay in San Jose (GPON), they had neighborhood cables that were factory terminated to go from wherever to each pole. The linemen had to setup the pulleys? and pull the cable through, and attach them, but they didn't have to terminate them. When they ran a line to my house, they did do a field termination, but it seemed like they had a tool and it went quickly (mine was the tech's first time, or at least pretty early, but it only took a minute or so).

Of course, for repair work, if a tree or something breaks a multi fiber bundle, splicing is going to be a lot harder than coax. Probably harder than traditional phone lines, but I think PON is setup so that you don't need to care which wire connects to which other wire, and if you do that for end-user traditional phone lines, you'll have a big mess, so you'd really want to match the wires up before you junction them. Coax is just one big wire, so way simpler.

DOCSIS is capable of 3gbps over coaxial cable and there is more room to improve, so fiber doesn't have a significant advantage in terms of possible bandwidth. Because DOCSIS was designed to operate on the cable network, which was designed to reach a very large number of homes, the architecture of the system tends to be more cost-effective than fiber. The fiber equivalent of a cable-like topology, and what is used by fiber ISPs, is PON, but PON is actually relatively limited in terms of both range and number of service points compared to DOCSIS on cable. A typical PON installation requires more field equipment to serve the same customers at the same rate compared to DOCSIS.

This is made worse by the issue of power distribution: the field equipment for DOCSIS consists of distribution amplifiers which are powered over the coaxial cable itself, allowing the battery-backed power supplies to be placed at convenient locations. There's not really any equivalent of this for PON, so extending PON networks beyond a single loop (depends a lot but typically a few KM and <100 customers) requires an OLT which is relatively large and needs separate power provisioned. You can put OLTs in serving area cabinets but this is costlier compared to cable equipment.

Another major factor is the customer premises: most homes already have coaxial cable distribution installed that is either compatible with DOCSIS 3 or can be made compatible with DOCSIS 3 by replacing the distribution amplifier or passive tap, which is a fairly cheap and fast operation. Installing PON to customers requires getting fiber to their house, and then either an outdoor ONT (troublesome from a maintenance perspective) and ethernet into the building or fiber into the building. The equipment here doesn't necessarily cost much but the labor of running new lines into customer homes is substantial and makes signing up new customers much higher-friction.

Most of the time when a cable provider upgrades to introduce DOCSIS 3 for digital voice and internet they aren't really replacing any cable anyway, just distribution amplifiers and nodes. The cost of this work is significantly lower than running new cable and it doesn't require new pole attachment agreements etc.

In general, in urban environments with existing cable TV plant there are few upsides to fiber. Urban fiber in existing areas is usually only cost effective when it's a new ISP competing with the cable company.

Finally, most DOCSIS networks are in a process of transitioning from CMTS (cable modem termination system, the upstream end) in the cable headend to a compact serving area CMTS at each amplifier point. This is called "Node+0" architecture, meaning there is a fiber node and then zero distribution amplifiers before the customer. One of the nice things about the cable "HFC" or hybrid fiber/coaxial network is that it is relatively easy to make this transition progressively as you sign up additional customers, since CMTS nodes have been made very small. PON is less forgiving this way and requires more up-front capacity planning, especially since network expansion means the permitting process on relatively large curb cabinets or enclosures.

Lots of comments about lightning, but I believe you are referring to a "ground loop" where the ground/neutral at one site is offset from that of a different site tens or hundreds of meters away. This is why all commonly used RJ-45 Ethernet connections use differential signalling and have the tx/rx pairs isolated by transformers. (Note that PoE Ethernet can still create similar problems if one is not careful.)

Here's a relevant news article about a lawsuit related to this phenomenon:

https://easyreadernews.com/edison-stray-voltage/

https://www.dailybreeze.com/2013/03/19/redondo-beach-woman-a...

https://www.nydailynews.com/news/national/woman-wins-4m-verd...

Magnetic coupling removes any issues with potentials, and if it's not AC at the same frequency as what's in the wall, the phase is irrelevant.

I'd like to add a few asterisks:

Magnetic coupling removes issues with potentials _up to the dielectric breakdown voltage_. Ethernet magnetics are considered high potential components, and even the entry level options will isolate at least 1.5kV, but fault events often exceed that figure.

Magnetic coupling removes issues with _common mode_ potentials. If the + and - side of a pair are both a thousand volts away from the pair on the secondary side, no problem. If a wire pair suddenly measures a thousand volts across… well, Ethernet transformers are typically wound 1:1.

_Ideal_ magnetic coupling removes issues with potentials. Ideal transformers behave as above, but real transformers have parasitic effects, particularly winding capacitance. Fast transients (including ESD) can and do capacitively couple across the transformer from primary to secondary.

Magnetics are important but do not solve the problem on their own. It is possible to design and manufacture electrically robust copper Ethernet systems – for a given definition of robustness (typically defined as passing some specific EMC test) – but even then real world electrical faults can and do destroy robust Ethernet systems. Fiber has none of these concerns.

All very true; good clarifications. In context my point was that it would be totally safe to run a copper Ethernet cable between two houses on separate electrical grids (or even an un-earthed battery powered computer).

Inductive pickup (foreign voltage) from even fairly long runs (20kf +) of well maintained copper is usually much less less than half a volt, measuring either T-R, or T-G/R-G. It's supposed to be floating to ground - nothing on a twisted pair should be ground referenced, if you do have voltage to ground, you have a short.

If it was not floating, you'd get atmospherics, hum, and other issues that you saw in old fashioned ground return systems.

Indeed, measurement of voltage and continuity of T-G and R-G is a standard way to check for faulty cable

Unless the telco cut the outer shielding on the twisted pair cables everywhere a terminal was installed. That was one of the remediations the incumbent had to undertake here when they started deploying 50Mbps VDSL2 FTTN service. Oops.

Any long run like this would use optoisolaters.

Al Gore never claimed that, btw, but he did recognize its importance and provided political support, even Vint Cerf said so: https://www.snopes.com/fact-check/internet-of-lies/

Why ruin a good meme with FACTS! People

absolutely - a good meme takes on a life of its own...

Right. Don't want to mess up the AlGoreithms that run the interwebz!

there is an entire multiverse fan fic where Al Gore won the presidential election and we're all living in a post-scarcity society.

I want to live there.

My jar of mayo costed $5.50 with tax yesterday.

"During my service in the United States Congress, I took the initiative in creating the Internet"

Sounds like we're playing word games at this point.

Imagine that, a politician playing word games.

>Al Gore never claimed that, btw,

GORE: “During my service in the United States Congress, I took the initiative in creating the Internet. I took the initiative in moving forward a whole range of initiatives that have proven to be important to our country’s economic growth and environmental protection, improvements in our educational system.”

There are three ways to read the first sentence, two of which take the presumption that Al Gore invented the internet.

Snopes is so half-wrong that it's getting to be easy to ignore.

In other words , he did say that, he just didn't mean for people to interpret his ambiguous statement in that fashion.

aside : the word of the day is initiative.

Gore absolutely deserves to be dinged for speaking ambiguously in a way that leans towards greatly inflating his contribution to the field.

But at the same time, anyone who's ever read or written a résumé will readily recognize "took the initiative in creating" as CV-speak for "I assisted in some capacity, however minor." I expect the actual number of people misled by Gore's self-puffery to have been small to nonexistent.

My ex's step father always said, "Well, if Al Gore invented the internet, I paid for it."

Which, as the senior official at DARPA who authorized funding for the project, he kinda did. ;)

Dr Z! I've recently started at Tunstall Healthcare and you are frequently referenced when refering to SPE. I'm the first person they've had working full time on SPE, so hoping to make some progress :)

> While we (PairGain) also made point-to-point private links, it was clear then, and even moreso now that most DSL involves a service provider managing the network - with specialized skills, practices and setup.

Mmm. This brings back memories of running private DSLAMs in a campus environment as a transitional pre-Ethernet stopgap in the early 2000s. 95% of the DSL lines worked great. The remaining 5% were a never-ending nightmare of troubleshooting and sudden breakage.

We didn't want to be DSL experts, we just wanted the hardware to work well-enough for long enough to make it all go away.

I actually ran a private DSLAM to get internet between upstairs and downstairs in my house only a few years ago (not a joke, G.fast).

I remember seeing your guys HDSL hardware in the field what seems like a million years ago. (for those without a telecom background) It was pretty common as a way to extend T1's without conditioned lines or repeaters then.

Interestingly enough, I'd done direct t-spans's inside of a building over house cable in a situation that was too long (and too poor of cable) to do Ethernet.

Awesome to have your input on this. I'd love to talk to you in person to learn more. Reach out at josh@botblox.io

This is nice, but how long is it going to take until we get multi-drop over single pair back?

You know ... a technology that we could do back in 1986?

It seems like 10baseT1-S is never going to actually come out.

>oh, fiber materials may be sand

what does sand means in this context please ?

Fiber optics cables are made of glass whose primary constituent is silicon dioxide and/or quartz which are the primary constituents of sand.

oh okay, thx

And therefore should be cheap in a naive estimate

To be fair, fiber cable itself is actually pretty cheap in the scheme of things. It's the installation and termination that account for the bulk of the cost

Reply on news.ycombinator.com