Hacker News Reader

That seems like a pretty silly argument to me. The library with 4 computers would be just fine with the $60 router. How is it any harder to manage that gear compared to managing the completely unneeded equipment that the local employees have no clue how to run.

Not to mention spending $14k on upgrades for many places that don't need that particular upgrade. I don't care if it's a little simpler to buy all the same, it's absurd to spend millions and millions of taxpayer dollars on something that is completely unnecessary.

I'm not really sure where it came from, but the Slashdot summary states that a lot of these things were never even un-boxed. If true, that's an incredible waste and far-outweighs any convenience or stability benefits gained by using homogeneous hardware.

http://yro.slashdot.org/story/12/05/11/1937235/west-virginia...

Interestingly enough - I can see both sides of the argument. When I was building the IT infrastructure for SSN, I ran the company's IT and it's dozen employees on a $75 linksys (and a half-dozen Poweredge Dell Server) for the first 5-6 months. Eventually, we grew to around 20 employees, picked up a data center colo-cabinet for our customer apps and needed a bit more robustness, so I purchased a $300 used Cisco 2621xm, a couple T1 WICs for $500, and ran the company for the next year on that 3 mbit pipe - the company got to around 50 employees before we moved. For the first two years of its life I spent a grand total of $875 on routing hardware and 50 employees were able to get their work done (including code pushes to our data center) over that 3 mbit pipe.

When I handed over network engineering, though, our last purchase order for networking equipment (with around 500 employees) - was north of $500K (including very hefty Cisco Discounts) (This isn't including the Data Center infrastructure - by then we had colo space in three data centers - just the corporate infrastructure for three buildings)

It's hard to explain without living through it - but the decisions that make sense when you are small, and you aren't paying your network engineers $150K/year (fully loaded), and you aren't trying to figure out how to handle pager duty, and warehouse and deal with RMAs, and support various types of hardware, and worry about rotating inventory, and dealing with upgrade cycles, and manage security, and patch levels, and remote administrations - not even considering future feature enhancements and performance (75 mbits sounds great today - but what about the future?) - when you get to scale, the CapEx (the capital cost of the hardware) starts to have fewer consequences on a technology investment (particularly over 10 years) than the other elements. Not to mention that there are also political issues (Layer 8) associated with differing levels of services/features for small population centers versus large population centers.

Yes - this does result in seemingly ludicrous situations like a 4 computer library being run on a ISR capable of running 350 mbits/second without breaking a sweat - but in 5-8 years from now, some technology administrator for the state will take over that infrastructure, and I can guarantee you that they won't be thinking "My God, we're over provisioned on our networking equipment" - but instead, "Thank goodness I have a few more years of runway before we have to replace all this gear."

I realize I haven't fully fleshed out the argument as to why it might make sense to put these hefty branch routers into smaller locales, but hopefully it doesn't sound silly.

Hey - at least they didn't roll out 6509s in all these libraries. Now that would be something I could rant against. :-)

This is a great thought - when talking about a growing business. Comparing it to a public library in a very small town (just over 6k) doesn't make any sense though. For the money, the library can buy a new router every year for next couple of centuries and keep up with current technology. Or we could have bought them a router and then given them $21,940 for new computers, books, facility upgrades etc.

Remember - these decisions weren't being made by the library, or for a 500 person company, they were part of a $126 million dollar infrastructure upgrade for over 1000 sites for the state of west virginia. The point I was trying to make was that the decision that makes sense for that library (individually - say the local librarian was given $25,000 and given freedom to spend it however they chose - they would probably do exactly as you suggested - drop $60 on a linksys and the rest on computers/books), no longer makes sense when you are rolling out $126 million worth of equipment. The most realistic alternative decision would have been to downgrade the library from a $7800 Cisco 3945 to a $2800 Cisco 2921 (an ISR with less capacity, but reasonably similar features and management). The Library would then have been on a 75 mbit/sec platform for the next 10 years. They wouldn't have received any of the savings to spend on books/computers, etc... They would have just gotten less capable networking equipment.

I could probably defend either decision (Going heterogenous 2921 (small) + 3945 (big) vs homogenous 3945 across the state) - but I know the one that would let me sleep easy for the next 10 years.

That's pretty circular logic. If they were given a $60 Linksys then it wouldn't be a $126M investment. It'd be a $1M investment.

These problems with managing at "scale" btw, are one of the reasons why consumerization of IT is so popular. The reality is that individual decisions made at small scale, can sometimes be orders of magnitude more efficient than by the enterprise. Example - Our company doesn't have a policy regarding mobile device OS upgrades - which means they don't have to worry about standardizing on a particular mobile device, or test "authorized apps" (or even have authorized apps) against the OS upgrade, or manage the OS updates, etc... Basically, a few early adopters will do an OTA upgrade, watch it for a few days, and if their device doesn't brick or crash more frequently, and the forums seem to indicate it's a good version - they'll send out a note to an internal alias and everybody else will just upgrade their device. Absolutely zero corporate overhead required.

This is true of all sorts of "edge" infrastructure - Skype clients, IM clients, Laptops, backups (now that we have backblaze/crashplan), etc...

Deploying at scale loses all sorts of that efficiency, and should be reserved for "Core" things like your exchange server, filer, and networking. For everything else - try to empower your users - they'll probably be better at it than you will be doing it centrally.

The problem with this argument is that the blanket purchase only provides giant imagined future capacity for the smallest users. If they will actually need this quality of a router in 5-8 years, then the big users will definitely need even bigger ones. But apparently the state is OK with the biggest users not having that much breathing room. So why do the smaller ones need it?

The first thing that comes to my mind is that in 5-8 years, as the State of west Virginia starts to run out of headroom in the larger locations, they can selectively upgrade that 5% or so that require something larger, while leaving the other 95% on the previous iteration.

Also - one thing I've learned about networking sites (real world experience, two companies that went from a dozen to 500+) is that once you are able to satisfy one person with sufficient pipe - that same bandwidth is usually sufficient for the 20-50 people. The reality is that need for bandwidth is very bursty.

And in 5-8 years, there will be more powerful equipment available at lower prices for that 5% of infrastructure. And you still have 1100 5-8 year old routers at all your locations that might not be capable of running whatever physical layer link we're using 5-8 years from now.

Future proofing is one thing. This is like buying a 747 for a route that serves a dozen passengers a flight to prepare them for the future.

> The first thing that comes to my mind is that in 5-8 years ... they can selectively upgrade that 5% or so that require something larger, while leaving the other 95% on the previous iteration.

In other words, they'll be in the exact same position that they were evidently trying to avoid from the start.

This smug, platitudinous self-serving drivel is all that a network executive needs to know to misspend millions of dollars. You conveniently neglect to mention remote network management facilities that will handle configuration and monitoring of the routers that could have been purchased instead of a fully loaded 3945, including the $487 CISCO router that was mentioned in the article. How do you justify this preposterous expense? "What if the state decides on an unprecedented expansion of library facilities?" In this economic climate? "What if homeland security needs emergency facilities and this is the only site available?" You didn't read the article where two identical enterprise level routers were installed in the same little neighborhood. What about the efforts to engage ham radio operators--West Virginia is full of them. So much for the big picture. You live in a world where automating the administrative aspects of networking hasn't kept up with the network devices themselves. At least that's what you tell your superiors. It's as if remote network management facilities don't exist and you're still upgrading IOS on routers state-wide by hand with TFTP. Don't tell me: you support one-size-fits all cable management for the state too.

Yes exactly, homeland security will need emergency facilities for the next terrorist attack and they can conveniently use the libraries. It's a good thing they had an ex-homeland security person to manage these tough decisions and prepare the libraries for the coming internet armaggedon.

The thing is that these libraries are not going to grow to several hundred seats in the next 5-10 years, like the company you worked for. They will remain at their level of 5-20 seats. As such there is no need, nor will there ever be, for such strong infrastructure in a small communal library. They'd easily be able to make do with buying a 100$ router every second year.

This is pure, unadulterated nonsense. There are plenty of ways to manage everything you mentioned with a tiered approach to the IT needs of the municipalities of the state without resorting to grossly over accommodating every location. Hell, they could've even used some of that money for regional IT management positions, putting people back to work.

Shame on you for justifying this nonsense in any way shape or form. This is textbook waste ala bureaucratic laziness.

Ten years of IT wages seems a bit more costly than some fancy hardware, particularly when you remember the pension.

Does that in any way change his argument that the money could have been put to better use?

I don't think it does. In the example put forth in the article, the library had only four computers! In the case of just this one library, they could have had: ipads, kindles, more computers, more books and other media.

I think it's kind of a no-brainer that this is a prime example of incompetence in government.

They would also need to hire someone to manage those devices which would cost maybe 2x the router year after year. It'd suck too (not going to get a competent person in that position).

My company provides free I.T. services, support, and sometimes equipment to non-profits and other community organizations. We would be happy to be the on-call techs for stuff like that.

I would be surprised if there wasn't somebody in their area that does the same.

Yes, because his argument was that it could be done with cheaper hardware and some IT-smarts (aka salaried employees managing it).

> ...and some IT-smarts...

I don't think he said that. For one, cheaper hardware doesn't imply increased management costs; for two, the more expensive hardware didn't come with a support contract included in the price (at least, it's not mentioned in the article); for three, his comment reads to me that they could have optionally put the money towards employing people, to put people back to work -- i.e., it was one example of a better way to spend the money. I don't see anything in his comment that implies, "cheaper hardware and some IT-smarts...".

Some dude is checking (or should be checking) on the PCs in that library anyway.

Hell, a college kid could do the job.

>This is pure, unadulterated nonsense. There are plenty of ways to manage everything you mentioned with a tiered approach to the IT needs of the municipalities of the state without resorting to grossly over accommodating every location. Hell, they could've even used some of that money for regional IT management positions, putting people back to work. Shame on you for justifying this nonsense in any way shape or form. This is textbook waste ala bureaucratic laziness.

And this BS attitude is part of the reason why the US has one of the worst internet infrastructures in the western world. Let's just use what is adequate for today and save some pennies, sure cable internet (or 640K) should be enough for everybody.

Not to mention that you're suggesting micromanaging and needlessly complicating the municipality IT infrastructure for marginal gains (compared to IT staff wages). Not to mention ignoring the "economy of scale" and mass-ordering benefits.

The benefits from economies of scale and mass-ordering still apply to much lower priced hardware.

The IT staff wages will still have to be paid because hardware needs support regardless of the unit price.

>The benefits from economies of scale and mass-ordering still apply to much lower priced hardware.

If you break the uniformity of the order, to lower priced for some and higher priced for others (for which the lower priced won't do) then you break the economy of scale benefit.

Instead of buying 100 same machines, you then buy a 100 machines of A/B/C etc types. You still buy bulk (within every machine class), but not as "bulk" as before.

Sure, the savings from buying bulk the more expensive unit may not totally offset the savings of buying units with different price points, according to each deployment's specific needs. But uniformity has a lot more benefits, too.

>The IT staff wages will still have to be paid because hardware needs support regardless of the unit price.

Sure, and nobody argued that. What we said is that since the price of the hardware is a drop in the bucket (marginal) compared to the total cost (mainly the IT staff costs), then it doesn't matter. Paying 30,100,000 compared to 30,020,000 is not much of a difference (this one is 0.2%, but even if it was 5-10% it would not matter much).

Also there will be savings in the staff training (the have to only learn one unit), fixing costs, etc.

The most important reason though, is that we are very bad at anticipating future needs in internet infrastructure. Better get something that has a lot of headroom for future needs.

When smart and experienced people tell you that things work differently in large orgs and govs, they don't primarily mean that in a negative way (...that the system is broken), they mean that as a matter of fact (...that the system is completely different).

It's not broken, it's not "fixable", it's just the nature of it all.

Let me put it to you this way, if they purchased $60 dollar routers, there would be a scandal about the incompetence and lack of forward vision at the top of HN right now.

If you are putting routers in a school that might serve a few thousand students and teachers, and a branch library that might be serve 5 patrons, you really can't afford to adopt a heterogeneous hardware strategy just to make admin easier.

It is surely more cost effective to use the cheaper router, and replace them when necessary than to buy the uber-model that will still be state of the art in 10 years.

Bucket system needs into three categories (large, medium, small) and find three routers that best manage those needs. Hire a full-time government worker to facilitate categorization of each installation location and manage the installation. Replace upgrade as needed.

Yes - that's actually a reasonable alternative. I'm hoping that someone did the math, and looked at Option #1: $7800 Cisco 3945, $2800 Cisco 2921, $900 Cisco 1921 vs Option #2 - $7800 Cisco 3945 across the board) and then compared it with the costs of having to prematurely replace those 2921s/1921s when they realized the 75 mbits/second wasn't what it was in 2012 (or some service modules with cool feature (Telepresence?) wouldn't work in the 1921)

Another thing to realize - they probably get one chance to upgrade their infrastructure like this. Sticking a 3945 in now means they can save money on upgrades for the next 10+ years.

Government money also seems to sometimes dry up as priorities are shifted - who says there'll be money over the next 10 years for upgrades.

I think you meant homogeneous, as in all the routers being the same?

Yes! sorry, I meant homogenous. Even a small state like West Virginia is too big and the needs are too diverse for a one size fits all approach.

The nonsense about $60 linksys devices was addressed in the article. Did you bother to read it? CISCO quoted a $487 router. You expect the taxpayer to foot the bill because you're too lazy to keep track of which router you've installed at each location? I deal with bureaucrats like you all the time. This is the last place I would have expected to find them.

The bureaucrats I deal with would buy the various routers at best prices and then proceed to build a software system to track which router was installed where that would cost > top of the line routers and would be built by a company friendly to the person making the decision. Initial costs would be low to win the bid but many "required changes" would ramp the bill up. They would create software jobs and get optimal router pricing - if that's not efficiency I don't know what is !

Those systems exist--CISCO has been selling them for years. Or have we identified another area ripe for disruption?

CISCO also sells Monitoring-aaS, which will do discovery, monitoring and upgrade for $xxx/device/yr.

I can assure you that any existing software would fail to meet the requirements (hint - primary requirement being related to/friends with/lobbying the decision maker).

You're right if nepotism is a requirement. But if we could suspend our cynicism long enough to consider available enterprise products (which is a tremendous strain admittedly for anyone who agrees with Paul Graham about "enterprise" computing) then as a comment below points out, CISCO sells Monitoring as a service and there are other vendors that provide this.

The other thing is that the bids had to be submitted very quickly or they risked not getting any funding at all. In that light, they can't really be blamed for not doing research into the most efficient allocation of money! (To be clear, I blame the stupid bid process that rewards rushed decisions and punishes anyone who stops to think.)

Yes, I have been there, bid by fax with 6h turnaround in one case but that was only a few thousand.

Fix the process.

It hardly seems like managing gear would be all that difficult at this kind of margin. Hire someone full-time at 100K annually who manages the systems and upgrades when necessary.