State paid $22K each for Internet routers
wvgazette.com
wvgazette.com
Amortized over 10 years, I would have chosen the 3945 everywhere versus sticking 2921s in some places (the ISR that would have been an alternative) and 3945s in others. Single Security Policy. Single IOS update Policy. Zero doubt as to what features will run in a particular location.
I think what most people have difficulty with is that they are comparing this decision to roll out a state communications infrastructure with the fact that they can go connect a $60 linksys wrt54G in their house and serve a dozen people without breaking a sweat. And get wireless as well! The issues involved in scaling that across the state, while looking to the future, and managing all that gear is a different challenge though.
I don't see any huge scandal here.
Not to mention spending $14k on upgrades for many places that don't need that particular upgrade. I don't care if it's a little simpler to buy all the same, it's absurd to spend millions and millions of taxpayer dollars on something that is completely unnecessary.
http://yro.slashdot.org/story/12/05/11/1937235/west-virginia...
When I handed over network engineering, though, our last purchase order for networking equipment (with around 500 employees) - was north of $500K (including very hefty Cisco Discounts) (This isn't including the Data Center infrastructure - by then we had colo space in three data centers - just the corporate infrastructure for three buildings)
It's hard to explain without living through it - but the decisions that make sense when you are small, and you aren't paying your network engineers $150K/year (fully loaded), and you aren't trying to figure out how to handle pager duty, and warehouse and deal with RMAs, and support various types of hardware, and worry about rotating inventory, and dealing with upgrade cycles, and manage security, and patch levels, and remote administrations - not even considering future feature enhancements and performance (75 mbits sounds great today - but what about the future?) - when you get to scale, the CapEx (the capital cost of the hardware) starts to have fewer consequences on a technology investment (particularly over 10 years) than the other elements. Not to mention that there are also political issues (Layer 8) associated with differing levels of services/features for small population centers versus large population centers.
Yes - this does result in seemingly ludicrous situations like a 4 computer library being run on a ISR capable of running 350 mbits/second without breaking a sweat - but in 5-8 years from now, some technology administrator for the state will take over that infrastructure, and I can guarantee you that they won't be thinking "My God, we're over provisioned on our networking equipment" - but instead, "Thank goodness I have a few more years of runway before we have to replace all this gear."
I realize I haven't fully fleshed out the argument as to why it might make sense to put these hefty branch routers into smaller locales, but hopefully it doesn't sound silly.
Hey - at least they didn't roll out 6509s in all these libraries. Now that would be something I could rant against. :-)
This is true of all sorts of "edge" infrastructure - Skype clients, IM clients, Laptops, backups (now that we have backblaze/crashplan), etc...
Deploying at scale loses all sorts of that efficiency, and should be reserved for "Core" things like your exchange server, filer, and networking. For everything else - try to empower your users - they'll probably be better at it than you will be doing it centrally.
I could probably defend either decision (Going heterogenous 2921 (small) + 3945 (big) vs homogenous 3945 across the state) - but I know the one that would let me sleep easy for the next 10 years.
Also - one thing I've learned about networking sites (real world experience, two companies that went from a dozen to 500+) is that once you are able to satisfy one person with sufficient pipe - that same bandwidth is usually sufficient for the 20-50 people. The reality is that need for bandwidth is very bursty.
Future proofing is one thing. This is like buying a 747 for a route that serves a dozen passengers a flight to prepare them for the future.
In other words, they'll be in the exact same position that they were evidently trying to avoid from the start.
Shame on you for justifying this nonsense in any way shape or form. This is textbook waste ala bureaucratic laziness.
It's not broken, it's not "fixable", it's just the nature of it all.
Let me put it to you this way, if they purchased $60 dollar routers, there would be a scandal about the incompetence and lack of forward vision at the top of HN right now.
I think it's kind of a no-brainer that this is a prime example of incompetence in government.
I would be surprised if there wasn't somebody in their area that does the same.
I don't think he said that. For one, cheaper hardware doesn't imply increased management costs; for two, the more expensive hardware didn't come with a support contract included in the price (at least, it's not mentioned in the article); for three, his comment reads to me that they could have optionally put the money towards employing people, to put people back to work -- i.e., it was one example of a better way to spend the money. I don't see anything in his comment that implies, "cheaper hardware and some IT-smarts...".
Hell, a college kid could do the job.
And this BS attitude is part of the reason why the US has one of the worst internet infrastructures in the western world. Let's just use what is adequate for today and save some pennies, sure cable internet (or 640K) should be enough for everybody.
Not to mention that you're suggesting micromanaging and needlessly complicating the municipality IT infrastructure for marginal gains (compared to IT staff wages). Not to mention ignoring the "economy of scale" and mass-ordering benefits.
The IT staff wages will still have to be paid because hardware needs support regardless of the unit price.
If you break the uniformity of the order, to lower priced for some and higher priced for others (for which the lower priced won't do) then you break the economy of scale benefit.
Instead of buying 100 same machines, you then buy a 100 machines of A/B/C etc types. You still buy bulk (within every machine class), but not as "bulk" as before.
Sure, the savings from buying bulk the more expensive unit may not totally offset the savings of buying units with different price points, according to each deployment's specific needs. But uniformity has a lot more benefits, too.
>The IT staff wages will still have to be paid because hardware needs support regardless of the unit price.
Sure, and nobody argued that. What we said is that since the price of the hardware is a drop in the bucket (marginal) compared to the total cost (mainly the IT staff costs), then it doesn't matter. Paying 30,100,000 compared to 30,020,000 is not much of a difference (this one is 0.2%, but even if it was 5-10% it would not matter much).
Also there will be savings in the staff training (the have to only learn one unit), fixing costs, etc.
The most important reason though, is that we are very bad at anticipating future needs in internet infrastructure. Better get something that has a lot of headroom for future needs.
It is surely more cost effective to use the cheaper router, and replace them when necessary than to buy the uber-model that will still be state of the art in 10 years.
Another thing to realize - they probably get one chance to upgrade their infrastructure like this. Sticking a 3945 in now means they can save money on upgrades for the next 10+ years.
Fix the process.
"Our main concerns were to not have something that would become obsolete in a couple of years," Gianato said. "Looking at how technology evolves, we wanted something that was scalable, expandable and viable, five to 10 years out. We wanted to make sure every place had the same opportunity across the state."
Wow. I've spent some time in West Virginia libraries, and yes the internet connection was slow, but it's not due to the router. And in most towns, libraries and schools are not going to be serving anyone outside their walls with some kind of WAN. Best case will be public wireless near the building.
And how is it that the "West Virginia Homeland Security chief" is "leading the state broadband project"? Seems like bureaucratic overreach, technical ignorance, and budget authority all wrapped in.
Why not spend $50 for a router for libraries and something less than $22k per school?
I think part of the reaction to this story is people coming to terms with how much real network equipment costs. It's definitely not cheap. You're paying for features, service, long-term support, upgradability, reliability, etc. Some of these routers will probably be in use 15 years from now. I've still got a Cisco 2500 series router chugging away. Next year will be it's 20th birthday. This stuff is built to last.
Unless there is some big backbone being built that can service a mostly small-town and very mountainous state, spending $20k (each!) now so that your routing infrastructure within schools and libraries won't be too slow in a couple years is kinda crazy.
http://nerdfeed.needfeed.com/blog/2012/02/our-network-has-a-...
If I needed to cover 1,000 widely scattered locations for the next decade, there is no way I would try to do that with a bunch of consumer-grade stuff. Especially when trying to manage a bunch of existing T1s and a conversion to fiber.
And then quietly replace it (after dumping the config and replicating the configuration) with a $100 half-depth Supermicro server bought off Ebay, running OpenBSD?
And then walk out, like the Grinch after he artfully puts Cindy Lou Who to bed, with a nice new 3945 for my own use?
EDIT: forgot to mention, I need a CF card reader also to get the config off the Cisco flash.
Earlier: "Gianato acknowledged that he didn't heed Dunlap's advice or wait for an evaluation.
'The routers already had been bid out,' Gianato said. 'I think John was looking at our needs now, not looking at our needs into the future."
---
So not only does Gianato admit he doesn't know anything (which means he's not even suffering from the Dunning-Kruger effect), but he also admits that he's not sure what his IT advisor was thinking about, but, eh, it doesn't really matter because the people who are the experts obviously don't think about the long-term and I'm not going to bother asking them about it anyway, because as we all know, bids are binding contracts. Right.
There is no excuse for this insanity.
Out government should be small. It should perform a few basic functions like making sure we are secure, our trading interests are promoted and our banking and financial markets are healthy. It should educate our children to the best standards in the world. Finally it should make sure we are energy independent and our food and water is uncontaminated. That's pretty much it.
I think we agree on the idea that government should be small, or at least as small as possible. And we agree on the roles it should play. But I don't buy into the argument that government is inherently bad at allocating resources efficiently and private businesses are inherently good. I've seen way way too many examples that contradict the latter. I think it ultimately comes to individual situations and the choices of individuals, what information is available to them, their goals, where their financial interests pull them.
It's not a democrat/republican or big/small government thing. It's about corruption of the system.
Out of your comment, I agree most with that our government needs to help educate our children. Without an educated population, things only get worse.
While the republicans are certainly guilty of lots of spending and many things, democrats are clearly the party that is guilty of more spending and even bigger government.
Getting the money / corporate influence out of government would be the place to start.
There are a few components to this particular screw-up. 1 is big government spending (with little oversight) and 2 is one idiot in WV with a big budget who failed miserably at spending it.
"1 is big government spending (with little oversight)"
The obvious problem here isn't "big government" spending. It's spending without oversight. This problem occurs all the time everywhere, in sectors both public and private. To claim that private businesses are immune to this by nature of their ability to fail and thus inherently a better mechanism than government is an absolute farce.
Take ownership of your government. To view the elimination of government as a solution to bad governance is both a lazy approach to problem solving and toxic to proper governing.
Nobody said anything about elimination of government, just less of it.
I'm sorry to burst your bubble, but our current presidents and congress members do what will appease the general population in the very near term, no matter what they are supposed to do according to their party lines. They are all equally guilty of spending.
You know, the TSA in the airports, which has a larger budget than the FBI? Guess who created and nurtured that entity? Your small government friends. And, who spent so much money on the wars in Iraq and Afghanistan that it is bankrupting the country. We can't even get involved in another war right now because we simply can't afford it. It is questionable if we will ever be able to lead a war like that again, due to budget constraints.
Then, we have the wall street bailouts. Which, by the way, aren't republicans supposed to be the only friends of wall street? Not if letting the banks fail mean that you are going to have to line up for the bread line tomorrow morning.
The big government people aren't exactly cherubs either. How many RIAA lawyers have been appointed to the Justice Department by the current administration? I think it's 5, so far. Enjoy additional Internet censorship.
Sorry to be such a downer, but things are just getting worse and the current two party system is a complete facade. I'm envious of countries like Germany that are seeing a surge in pirate party support. Unfortunately, it's going to take decades to undo this mess and, frankly, a large majority of baby boomers are going to need to expire naturally before the voting population comes back into balance.
Let me guess... military spending doesn't count as "more spending". The $800 billion allocated for 10 years of stimulus is a paltry sum compared to the $700+ billion we intended to spend every year on military, and we all know that money is spent very efficiently.
Meanwhile, federal spending has increased every year under both parties for the last two decades, so you don't really have a leg to stand on with regard to your "democrats are the big government spenders" FUD.
Just because there is waste in defense, doesn't mean there isn't more elsewhere. Defense is one of the few things are government is actually tasked with. All the other shit we waste money on isn't part of what they are supposed to be doing. Roads and armies, that's all I expect. I don't need the government to feed me or wipe my ass.
gps and the internet are two of the most notable and visible examples with astronomical infrastructure costs. no private company would undertake a risk like this yet both have substantially improved society. the microwave and digital cameras are two others that a private company may have eventually created.
this isn't an argument for military spending, just the fact that without a government body that focuses entirely on applied research and development the military is the next best thing.
As for your core point about the role of government... just look at the top 10 economies of the world. Most are not libertarian paradises and they're doing OK. It seems to me both models could probably work if executed properly. I doubt one model is inherently right and one is inherently wrong. Most likely some hybrid is the best solution for everyone.
When you reach a certain size, and you have many competing groups, all with their agenda with the same company, it becomes difficult to allocate resources correctly, especially in a domain that is not considered "core of business" (IT being a common case, sadly).
licenses for clearcase cost enormous amounts. yet clearcase is missing many of the features found in open-source (and free) distributed version control systems. meanwhile, there is no free alternative to the routers, which may actually be necessary at some of the installed locations.
this doesn't include the huge server costs to run clearcase after you buy the license, which aren't necessary for distributed version control systems like git or mercurial. and big companies often spend more money buying other IBM tools to integrate for which free alternatives exist...
the amount of money spent on this purchase is fucking peanuts compared to what many large private sector companies spend on IBM software that lacks most of the features found in your favorite open-source alternative.
I disagree with the sidebar that it's mostly Democrats trying to take our money. Both parties do it, just with different priorities and in different ways, with all the accepted terms used to discuss it.
A little known fact in the tech community is that a $7,800 router provides exactly 16.016 times more opportunity to a student than a $487 router.
I guess someone involved in the decision got some kind of kickback, gift, or will have a nice cushy job offer waiting for them at Cisco when they get fired as they should be.
Remember this article next time some liberal ask to raise taxes and justifies this by saying how stimulus money is important, and why everyone should have the same opportunities (quoted in the article - as if a 22k router gave any kind of opportunity to people browsing the net in a public library in WV).
And cry when they get trashed at the end of their lifecycle without having ever been connected to optic fiber.
Unless I misread, the article is talking about West Virginia. I would hardly call WV a bastion of liberal thinking.
Never attribute to malice (or political leaning) that which is adequately explained by stupidity (or incompetence)
One of West Virginia's two current Democrat senators, Joe Manchin, was governor at the time of the stimulus and appointed Mr. Gianato.
The executive branch of the federal government in 2009-2010 was controlled by the Democrats, along with both houses of congress. The federal stimulus was passed with no Republican support.
http://www.nytimes.com/2009/01/29/us/politics/29obama.html
The entire chain of custody from the grant through request for bids through the eventual approval was controlled by Democrats. The only people involved in this fiasco without a (D) after their name are those people who will have their taxes raised in order to pay for these routers, because of course they were not paid for with existing revenue.
This is very much a failing of Democrat administrations at a federal, state and local levels.
Governments that waste money just raise taxes, borrow more money, or print more money. Corrections when they do come are often catastrophic in nature.
I love West Virginia, and there is so much more important the state government could do with this kind of money if they're going to take it outside of "Homeland Security". As in, better health care or safety for the huge population of coal miners.
i have been - in my time (2000? 2001? 2002? 2003? 2004? ...) - involved into selling the national austrian television their first video streaming solution (think: free (as in free beer) windows media streaming server) - it took man-years. man-years cost money.
I could further elaborate on the reasons why this matters to me, but I'm just relieved that it works OK now.
I'm not a programmer or technical person, I'm an end user who uses R, LaTeX and processing now and again. I hack an awesome spreadsheet. I have a degree and I do a professional job.
I could be a manager in a public service or an elected representative. Perhaps in a small rural authority.
Given the lack of consensus in this discussion thread, and evidence in the article of a similar lack of consensus between the various offices involved, how am I supposed to reach a decision?
Is there a case for some kind of planning toolkit or requirements estimation software? Is there an opportunity here?
PS: I and other colleagues did once have to help a senior manager spend out a £250k capital grant in 10 days. The idiotic spending deadline was due to delay in award of funds in a competitive funding round. We did ok but could have done better with another month or two to think through detailed requirements.
I would have to make a spreadsheet (good at those) of all the sites and then include a rating for current traffic and then expected traffic growth over the life of the project. Yes, a library with 4 PCs is not going to have a lot of growth...
1. You ask people who have already done this, like hotels who deploy thousands of networked sites. 2. You hold a bakeoff at a few dozen sites and see what works.
I noticed some went for a 5 year sort of rolling provision making assumptions about increased traffic (but certainly no gold plating like in the original article) and others just bought what would work now and dealt with changes as they arose.
I have to say that the former strategy seemed to provide a more uniform system with less 'catches'. People don't share budgets so I have no idea which came in cheaper.
No you fool! These are designed to handle large loads, not protection from obsolescence!
I would still like to see a breakdown of the costs and what services are provided with the routers before I call it a total waste though. If it was just the routers then it was a waste. If it includes other things like a long term warranty & onsite service, then maybe not so much.
I remember a few weeks ago PB made a comment that at least social networks and alike make it easier to spread the news, which eventually will lead to greater good (http://news.ycombinator.com/item?id=3890543). At least that's true, but doesn't feel like it's enough.
No. This is not okay. No twisted sense of reason that only you have makes it okay. A $60 consumer router that needs to be upgraded every year is still cheaper than this. Even with support costs. People have been rolling out tiered infrastructure for years.
I imagine the government worker that signed off on this purchase worrying a lot more about a dollar increase in the local lunch special than millions spent on equipment.
Deleted comment
You haven't mentioned the article once.
What do you think the probability of expanding a small library from four users to 50 is in this economic climate? Did you read that the stimulus was for the installation of fiber? Instead the state got routers equipped with T1 modules.
In this story, at least 20 million of the money seems to be completely wasted. (Higher capacity router does not mean better internet access for 4-computer lib.)
Wait, not finished. So our corrupt official has 2 million in his hands. He may buy a car or a house, but probably he would spend it for something useful. So much of the value would return to the market. This is also true for the vendor company. But in the US case, the value is gone. it is like you burned the cash.
Moral of the story: stupidity is more dangerous than corruption. You may want to have a corrupt offical than a stupid one.
Finally, that stupid would never have survived in the relatively corrupt but wild political system of my country.
* Yes, I realize this would bankrupt half the military contractors as well.
You are correct on the military contractors as well though. Lots of government waste and incompetence there as well.
Assuming that the government has some rules that requires them not to deploy soho or open source products and they have to spend the money on some big brand company: how come Juniper was not taken into consideration (or other known network vendors)?
I would argue that Juniper could provide cheaper equipment of simmilar of higher spec (let's ignore for a moment if that is oversized or not) - and it becomes even cheaper if you consider the simpler licensing terms and upgrade support.
Isn't the government supposed to run open tenders? Is cisco mandatory in the US?
Anyone familiar with telecom and living in PA, WV, other states that have Verizon as the ILEC, knows this.
A librarian will notice a computer missing, but an expensive router replaced with a cheap one won't even be noticed for years.
somehow i doubt these sites have multiple uplinks, so they certainly don't need BGP routers. even if they're doing IPSec between some locations this can be handled by an old piece-of-junk $100 computer running BSD or nix, e.g. with isakmpd and pf.
stuff like this is why america's infrastructure is crumbling: dodgy contractors looking for an excuse to upcharge municipalities for irrelevant gear and services.
Anything more than that for a moderately large library or most schools is an outlandish, shameful waste of money and equipment.
There's a lot to be said for uniformity in deployment, but that's why you would define, say, three tiers of supported hardware instead of kitting out everyone with top-of-the-line.