Hacker News Reader

Comment by ryandrake | Hacker News Reader

As someone who has been in the private sector his whole life, and not had much experience with government/military work, it’s always fascinated me how often these projects end up way over budget, way WAY late, and fail to meet every requirement. I’ve always had this morbid curiosity to see first hand how messed up everything has to be at these places in order to fail so badly and so consistently. I’d love to understand how companies can win contract after contract and yet be so bad at delivering. Reading these articles feels like watching car accidents.

View on HN

I worked on a DARPA contract on military radio technology. I’m not surprised at all why these things go over budget.

Imagine awarding a contract to build a self-driving car. Do you think that contract could be delivered on budget and on time? That’s like every military contract. They’re for bespoke stuff that doesn’t exist in the market. The military looks at what’s out there, adds a bunch of wishlist items, and then puts out a contract for that.

Think about how the private sector develops new products. They don’t award a fixed price contract. They throw a bunch of money at startups. Most of those efforts fail. What is the development cost of every successful product, accounting for those failures? In the military, all the hiccups and restarts and course corrections and dead ends that would kill a startup, and be written off by investors, instead get accounted for in the final cost of the contract.

Unlike self-driving car, radio is a solved problem. Marine and commercial transceivers from Icom and Yaesu are cheap and rugged. Ham radio operators sometimes even use transceivers build from kits, e.g. Elecraft. Adding encryption should be trivial.

You can't say that "radio is a solved problem" without knowing the military's requirements for these radios.

For example, we were working on radios that could self-organize into networks on frequencies they picked dynamically and keep on the same frequencies while facing a changing RF environment because they were in a convoy driving through the desert with a high powered jammer randomly stomping on everything.

And these aren't fanciful requirements. You're in a country whose government you just toppled--you can't count on the civilian frequency allocations. The military really doesn't want to keep track of every squad's frequency assignment manually. The radios are always on the move, in often challenging terrain (mountains, etc.). The enemy has jammers and countermeasures. Even if not every radio needs every one of these capabilities, the military needs an overall communications architecture that accommodates all of these functions.

Partially solved. Most Ham operators don't have to deal with adversaries. These radios need to stand up to jamming and eavesdropping. They also need to handle a (fairly complex) protocol automatically and in a dynamic configuration. Imagine a cell network with no tower, each phone is capable of becoming the leader. And multiple networks can exist in the same physical space. All traffic has to be encrypted to protect against eavesdropping, and frequency hopping and other techniques have to be incorporated to become jamming resistant.

APRS and packet radio existed since the '90s. They handle routing and multiple participants, and are rather resilient to interference.

There is no encryption and no frequency hopping not because it's difficult to do, but because the goal is to be heard and understood by anyone who listens.

APRS does not have the reliability necessary for commercial use, let alone military use. Packet radio is pretty trivially jammed, handles routing poorly and interference almost not at all.

> Adding encryption should be trivial.

Encryption is by no means the only additional feature, at least frequency hopping is crucial to avoiding jamming. I can imagine there are other requirements I can't immediately think of.

Doing secure encryption of multicast radio is anything but trivial. You have packet loss, the group is constantly changing and for narrowband technologies you typicaly have small enough MTU that normal cryptographic constructions have larger per-packet overhead than what is left for the useful payload (this is also issue for many long-range IoT radio layers)

Modern radios are software-defined, or at least controlled by a microprocessor, so frequency hopping or even spread spectrum shouldn't be much of a problem.

That's like me going up to a Netflix engineer and saying "why do you get paid so much? You've got the video already, just send it to the device; shouldn't be much of a problem."

Well, the difference would be that I actually built radios, and at least some of them worked. It's really that simple, even without SDR. You already have a microcontroller that reads rotary encoder attached to the tuning knob and changes voltage that goes to the VCO (or divisor in PLL). All you need is to control it from CSPRNG.

You will need to spend some effort on getting synchronization right, but it's a lot less complicated than, say, building a CRUD app.

Congratulations, you made a few working radios with the most basic feature set.

Now make a radio that works in sub-zero and 100+ degree temperatures, survives water, dust, and firearm exposure, being dropped dozens of feet, is resistant to broad-spectrum jamming, can handle secure encrypted communications with a large set of ever-changing participants, has a range of dozens of miles and days of battery life, is of a size and weight portable enough to be carried by a soldier for hours or days alongside their normal gear, and which can be field-repaired by a soldier with a few days of training.

And those are just the high-level requirements. It's a lot more complicated than a CRUD app, which even a non-programmer can do in an hour.

Link to CELab post on Mil Std 810 - Gunfire shock.https://celab.com/en/laboratory/environmental-tests/mil-std-...

The application portion has a nice explanation on the effects of gun fire on systems. Keep in mind that "gunfire" covers everything from a 9mm handgun to a 30mm Gatling gun.

What is firearm exposure? Working next to a firearm? Working after being shot with a firearm?

Any cell phone seems to do the former just fine, and I don't know of any radio, military or not, that can do the latter.

What do you think these radios are doing? They aren't just voice radios, you do realize that right?

I was talking about the tactical radios discussed in the article (2km range).

If you want the radio to let you watch video feed from a drone above, show a map with locations of other soldiers, or update Facebook status via satellite link, that may be a bit more work.

Imagery and location data with others are among the feature set of JTRS. So yeah. A bit more work.

Guess how we know you've never designed or used a military-grade radio.

Try the following experiment. Take the best 2meter hand held radio to the Dayton hamvention. Pick a simples frequency, talk to your buddies on this frequency. You will hear crosstalk, imd, phantom signals, dropouts. Now take that situation and put it in a hostile, battle situation. Not at all solved.

Watch the "Pentagon Wars - Bradley Fighting Vehicle Evolution" video: https://youtu.be/aXQ2lO3ieBA

It's a great example of how trying to design a product that works well for every stakeholder in every scenario leads to a Frankenstein of a product that doesn't work well for anybody. This happens all too often in government (typically military) contracts.

Hilarious. Reminds me of what it's like working with a client to build a website. Build it!!

For the requirements, these are written by an office in the military with say a dozen people in it. They will each serve in there for two to ten years then move to someplace completely different. None of them will be experts in radio design. They are going to listen to what their "customers" want, which will be an incrementally better version of what they already have. They are going to listen to what radio salesmen want to sell them. They are going to listen to what some futurist tells them is possible. And then they will write a specification.

The resulting spec may or may not be possible to implement. It may have pairs of requirements which require great cost to implement both, where the sane thing to do would be to relax one slightly, but the spec is cast in stone and that will not happen. It may have specs which could be vastly improved at small cost, but that also will probably not happen.

In industry, ideally, there would be a series of give-and-take rounds between marketing, the product designers, the engineers, and manufacturing which will find the efficient solutions to the conflicts and identify cost effective enhancements. This can't happen in military procurement because to engage in a design process with a company or companies would be unfair to the other bidders. They might engage a company to help them write the requirements, but it would be one which does not bid on the final contract and does not enable the iterative design cycle.

As a procurement system, it can fail spectacularly and is almost guaranteed to preclude optimum solutions, but it also deters cronyism and snake oil peddlers. If it seems like it always fails, remember that you never hear of the contracts which come in on time, on budget, and deliver a usable product.

Strict quoting procedures that don't allow incremental learning while building new things, warring factions continually changing requirements on the client side and unlimited money mean that the projects almost can't be delivered, they are doomed to be overpriced and underdelivered before the development even starts.

Disclaimer: At my previous job, I was several degrees disconnected from SCA/radio contracts (worked on test sets for those radio systems).

Being over budget, late, and not meeting requirements seems to be an infectious quality. However, it's not like many companies are set up to deal with winning these contracts. Do you choose the company that's more likely to have a more complete product but possibly be unable to support it in a couple of years? Or the one that can do decades of support (including replacing obsoleted hardware) but has a reputation for being late, overbudget, and half done?

This is a big issue. There are only a few companies operating in this specific field. You aren't going to find any startups interested in it because making a barebones radio is a multiyear process (to hit all the requirements laid on them from software capabilities to hardware durability).

The established players also have a tendency to layoff or move the experienced engineers after a major update and leave only cheap, inexperienced engineers in their place. So the next big cycle will have mostly people who don't know what they're doing accepting changes that cannot actually be done (in time or budget, if at all) but they don't know better.

I don't think it's "you aren't going to find any startups interested" as much as "you aren't going to find DoD interested in trusting startups to be around". No one gets fired for giving a billion dollar contract to General Dynamics.

Big enterprise software contracting is similarly dysfunctional.

Every defense software requirements document:

1. Make it do exactly the same thing that the previous version did, right down to replicating the bugs.

2. Then add a unicorn.

I'm amazed/confused that stuxnet worked out as well as it did.

"The Pentagon Wars" is a decent film presentation of this farce.

I've heard about stories about government agencies and such second hand from family members. To me, it sounds like it breaks down to this because the projects that they are running are not subject to risk of failure. There isn't a small company that is going to absolutely die if a deliverable doesn't meet the market's demands at a certain point in time. Just don't stop throwing money and resources at the project until it is done no matter what.

Government projects are about as efficient as similarly-sized projects in the private sector. You just never hear about private company’s failures because the public has less of a stake in it. Journalists focus on people wasting your money.

The laws requiring publication of bidding documents and damning reports on cost overruns are also mostly specific to public projects. Who knows how much money Apple invested in self-driving technology? Try sending a Freedom of Information Act request to Uber inquiring on their self-driving car efforts.

It is a little more complex. The small company realizing money will run out will start to look at options. The obvious option is cut back requirements/scope until they can get something - anything - on the market to bring in money. If they can slow the rate of losses they might be able to finish eventually. The other option is to show their investors current progress and ask for more money - this is a hard sell but it sometimes works.

In government the first is not allowed - the contract is specified. However you can ask for more and if you have made progress they will sometimes grant you "that little bit more" - remember ultimately they want what you are under contract to make and if you fail they have to start over again with someone else.

As somebody else has remarked, your perception is somewhat skewed by survivorship bias: yes, it seems that these cost overruns are due to a lack of “risk of failure”, but one should really compare it to the overall manner by which capital is allocated and products delivered to market, meaning that one should not be blind to all the startups that get funded and fail to deliver. I'm not saying the two systems are equivalent, but this is the information you must keep in mind to compare them fairly.

There's because the lowest bidder won on an ever expanding requirement list.

Reply on news.ycombinator.com