My hell as a 911 operator
nypost.com
nypost.com
* I build big call centers for a living
* I'm a really big Telecom Nerd
You might be wondering why something as important as 911 is not automated or accessible via any other mechanism; why is it that the most critical service in the country runs on a system that hasn't materially changed since 1984?
For one, the Public Switched Telephone Network (PSTN) is extremely reliable, authentic because it's logically addressed, and ubiquitous. The only real flaw is the addressing, which, because of Caller ID, can be faked, but that's another story (See Wikipedia on Swatting).
So, there's a huge movement to reshape 911 service to take advantage of modern things like the internet, SMS, GPS and all different kinds of connectivity. The way 911 works right now is sort of a hodgepodge all over the place depending on which vendor the municipality has selected for the Public Service Access Point. There is not one standardized 911 system across the country, which may or may not be a good thing depending on your perspective.
So everyone can agree fixing 911 is a great idea, BUT why hasn't it happened already? I can point to a couple potential reasons:
1) It's a really big contract; supposing PSAPs get standardized across the board, we're talking about many millions, potentially billions of dollars. That means lobbying and negotiation and time (oh by the way Sprint is currently leading the charge but I find it unlikely that the US will let a foreign-owned conglomerate manage their backbone emergency services).
2) The current vendors see nothing wrong with the existing solution, and have projects like what has happened in New York to stand against the tide of change. Their service contracts provide a big incentive to keep those boxes in those cities.
So I think there's a lot of reasons why 911 is as bad as it is, most of it isn't technical, but political, as with most old ridiculous infrastructures.
It's amazing how not planning for obsolescence almost automatically results in political conflict down the road.
https://en.wikipedia.org/wiki/London_Ambulance_Service#Syste...
As with everything gov, too much money leads to under the table deals.
It's ridiculous that we have 200 people constantly updating code for my sound card in Linux at 2 different projects, but no one can give out time to improve a vital system like this which eats billions of my taxes.
...just because someone went to college with someone else or somebody got golf membership somewhere for free.
I just don't have any lobbying money :).
All we needed to fix the problems laid bare in 2000 was a foolproof way to encode ballots. Open source software running on a PC connected to a card punch machine would have solved it.
What we got was something that undermined the whole voting process. And at the cost of millions.
Again and again and again, our decision making system allows a minority to waste our money.
The only fix, in my opinion, is giving you the right to choose who represents you.
And I don't mean the right to pick from two alternatives chosen by someone else, and not even getting that unless you are in the majority.
If you picked your lawyer or doctor from two preselected choices and didn't even get that guy half the time, wouldn't you conclude that someone else was in control?
So Here's a brief overview of what we're doing and why we're doing it this way.
Asterisk doesn't scale without a lot of cruft and it's not accessible. At the high end of the market there's only gigantic switches from companies like Cisco/Alcatel etc; and those are $$$. So we see a market space inbetween cheapo asterisk and high-end cisco.
The crux of what we're doing is a lot of work in AMQP (RabbitMQ) to disintermediate the underlying infrastructure from the services. The point is that we don't ultimately want to care what underlying components you run on your network, only that you have components that can be consumed. Think of it as TelecomSDN from a management perspective and a distributed switch from an infrastructure perspective.
The stack as we distribute it today is:
* Kamailio (Border Controller)
* RabbitMQ (AMQP messaging bus)
* FreeSWITCH (Open-source switch)
* BigCouch (Distributed database with Dynamo Quorum to help deal with Network Partitions)
That's the gist of it; all on CentOS and all open-source. We may close source some modules as we go forward, but right now just about everything is open. Using a few clicks in AWS you can run an infrastructure capable of competing against AT&T.
There's a lot of telecom infrastructure that exists which can route calls. There aren't a lot of telecom infrastructures that have built-in fault tolerance and linear scaling.
EDIT: I should note that we've developed custom modules for both Kamailio and FreeSWITCH for our use case. We had to write a multi-threading module for Erlang for FreeSWITCH, as an example.
Will it scale down to a Raspberry Pi? I realise I'm completely not your target market but I need something that does VoIP better than my router does it.
We're currently trying to build the world's smallest cluster; seven raspberry pi's will hopefully support 100 phone calls or so with no single point of failure.
If you're looking for something single server you want freeswitch by itself. That thing is a beast if you just want call processing and don't need carrier grade reliability. We made a frontend called BlueBox which is a very popular distribution; we're revamping it for Cluecon (hopefully we'll have it out by then) so if you're going to try this I'd recommend waiting until the new version comes out.
Lastly, there's a lot of research going on at FreeSWITCH about running it by itself on the Pi. I believe it is workable now, you'll probably find a bunch of info with a quick google search.
If you look at the way 911 works, there's no location information unless you use the cellular radio (and this assumes your carrier is using the cell tower to anchor your location). If you use IP to call, let's say via an application like RingCentral, how does the 911 service know whether you're in San Francisco or New York? It doesn't.
This isn't pie in the sky fantasy, this is actually how all over the top VoIP connections work. You pick a location and that's where your 911 calls originate. It doesn't matter whether you're using your SIP client in New York, your 911 call will still go to a San Francisco 911 center because your office address is in SF. That's not good.
So IMHO the right approach is a top-down rewrite and rolling that sucker out nationwide. Normally, we'd see one vendor start to dominate over other vendors, and while there has definitely been some consolidation in the 911 market, the fact that these services are federally mandated (Read: Allocated budget automatically) there's not a huge incentive for municipalities to shop around. These systems are essentially free as far as they're concerned.
Part of the reason we're not on IP is because the federal dollars can't be legally spent on IP in many places (gotta love lobbying). In short, the correct approach, IMHO, is IP-enabled PSAPs with GPS information for location combined with reform efforts in congress. The logically addressed world of the phone network is a fantasy and our emergency services should model reality instead of our dreams.
This is almost always an extremely bad idea:
http://www.joelonsoftware.com/articles/fog0000000069.html
I have no inside knowledge, but wouldn't be surprised at all if many of the problems of the new NYC 911 system is due to a decision to rewrite the code from scratch, losing much of the knowledge and bug fixes embedded in the old system's source code.
We're talking about software from 1984 in some cases although it's all littered with security holes, bad interfaces and slow response times.
In the case of NY, I could agree with you, but I would say that the problem was significantly overcomplicated. There are better systems that could be used to deliver something like this. I think that, yes, sometimes you do need to do a top-down rewrite, especially when everything is wrong.
I do agree with your point that the NY incident is likely attributable to poor software design, but I disagree with the conclusion that any rewrite would suffer from the same malady.
CHP was traditionally where all cellular 911 calls went, too, but that got improved over time.
The iDen network is basically analog, so transactions like presence and push to talk are binary and the state is always transmittable. The same cannot be said of IP. This is a problem we've spent a lot of time working on and I can guarantee none of the Big Telcos have a solution yet. So the answer is, the iDen network's PTT is basically analog and easy to manage whereas the LTE network is IP but difficult to manage. They want people off of iDen because running two networks is a massive waste of money.
You mean the Integrated Digital Enhanced Network is analog, wait what?
The actual problem why noone has come up with a solid way to support PTT on cell phones other than iDEN phones is because iDEN is a rare breed of radio system called a Cellular Trunked Radio System, it was designed to provide a trunked radio system that could scale to large footprints. Unfortunately it never caught on with anyone but Nextel and a few other providers, most likely because of the need to totally replace other systems. Project 25 did not have this issue because agencies could gradually transition from analog conventional and analog trunked systems to digital Project 25 trunked radio systems or hybrids thereof.
So, iDEN was intended to primarily support trunked radio like features, Direct Connect and Group Connect are two of those, several bolt on upgrades later provided us with Nationwide Direct Connect and Nationwide Group Connect. This is why it performs so well at those (infact, from what I have seen, outperforming existing trunked radio systems).
That's really interesting; could you possibly recommend some additional resources where I could read further. Something beyond the wikipedia entries? Thanks again!!
There are Github repositories:
https://github.com/GSA
https://github.com/GSA-OCSIT
https://github.com/cfpb
https://github.com/opengovplatform
Got two out-of-the-blue emails today asking
"who is behind http://CFPB.gov's great graphic design."
I'm proud to say: civil servants" [1]
[1]: https://twitter.com/matthewburton/status/359747466339418113
Having worked in both, I can say that both have their strengths and weaknesses. Where you would fault contractors though, I would generally fault the contracts themselves, or the faults of the contract overseers. Government contracting is adversarial, and where contractors have made major gaffes, in most of those cases you'd find that it was the government itself to blame. There's a reason it takes several hundred thousand dollars to be the lowest bid for standing up a standard Wordpress website, and I promise you, it isn't because the contractors don't know Wordpress.
This mentality is probably true for most systems that are built for the internal use of an organization rather than as a product that has to compete on the open market. For example, software built by IT departments for use inside large corporations is notoriously badly designed and hard to use.
On the flip side it isn't always management.. I've seen developers create forms that should be simple with really weird layouts on screen that make absolutely no sense at all.
Let's get these boxes checked... while every stakeholder wants to see our project fail because they'd rather waste a few million dollars and get a proof of concept so that they can then hand the contract off to the vendor they wished had won the bid in the first place. (Note, this happens to ALL vendors, because every group has a different favorite.)
Let's get these boxes checked... while having to justify every single change to the environment to the Enterprise Architecture Board ("Why do you need to use Python? Can't you build it in Java?") as well as to the Change Control Board ("Why do you need to open port 80 to this server? Can't we simply proxy that through our dated and oversaturated proxies? Why won't your load balancing work without sticky session support?")
Let's get these boxes checked... even though the government has given us three conflicting visions of the project and now refuses to pay us because we're behind schedule because we cannot meet all the new feature requirements that have arisen.
Etc., etc. It's a hellish game.
Basically, the problem is that their system crashes. Uptime should have been specified in the contract.
Poland:"what is the address? what's happening" Bam, dispatch!
The trip from RFP to signed contract is maddening and has probably driven more than one government worker to an alcoholic stupor, wishing for the sweet embrace of death.
On a side note, I'm no longer a government employee. My name is Greg and I'm an alcoholic....
Government is a place that is ripe for disruption, but the walls around it are so high, I don't know how to really breach them. There is a huge machine in place of contractors and lobbyists which essentially maintain a strict control over all IT contracts that come out of the public sector. Massive corporate entities with an entrenched web of influencers at every level of government, ensuring that only a select few even have an opportunity to realistically compete for the jobs. The amount of overspending that could be eliminated with the disruption of those entities probably represents a significant portion of local, state, and national budgets, not to mention the likely better solutions that could be presented by folks more accustomed to working efficiently.
I'm not necessarily saying the 20 somethings who were going to make "Snapchat for dogs" could do a better job as government contractors. But certainly someone who designed infrastructure for Twitter might have some interesting perspective on how to manage 911 traffic.
Most government employees are unemployable in the marketplace, so it's in their interest to follow orders, at whatever cost, to keep their jobs.
The best thing for young professionals is to fight the system intelligently for those that have the resources to do so, or avoid it entirely.
Edit: As is with NYC's 911 boondoggle, a SF startup could've built out an amazing system at a fraction of the price. But that wouldn't be the government thing to do. It has nothing to do with building a quality product.
Citation needed.
If you actually have evidence, I'll bite, but my anecdotal evidence tends to indicate otherwise. Generally, issues are caused by good people being hamstrung by ridiculous rules, not by bad people.
My anectodal observation is that this may be true for some individuals, this could be said of any large institution. I encountered many capable and intelligent individuals in my time contracting for the government.
The problem I see more has to do with the legal mandates in place, and the labyrinth of regulation and processes. Unless you are quite savvy to how the machine works, it's nearly impossible to get a contract. And the steps to learning how to navigate these processes is pretty lengthy and would make even the most resilient individual completely jaded with the whole thing. Finding a way to unwind the over-regulation in place seems to me to be the only way this market can be opened to true competition.
This statement is so blatantly false that I have a hard time believing that you are engaged in honest discussion when you state it.
Tackling the problem of a system that must be 100% reliable, synchronous, with complete uptime from day 1 is very very difficult, and not many startup environments will prepare you for it. A lot of systems that are scaling decently now (twitter was used as an example early on) had extreme early growing pains.
In fact, I think it is impossible to build such a system without significant upfront cost, because a lot of the cost should go into the testing phase. It's almost as though you'd want to create a testing call center that receives realistic calls and needs to route them correctly, and have it running for months while the software is iterated on. A lot of effort should go into making the testing center as realistic as possible, and a lot of time should go into teasing out the inevitable bugs.
Also, replaying traffic from the old system into the new seems like a viable way to test out this sort of system. Or, they could have run both systems in parallel, and just sent a percentage of the traffic to the new system and slowly ramped up. I wonder why they chose to just cut over rather than doing a full and seemingly irreversible cutover.
Not that many startups would have done things that way, since the nature of startups tends towards the green field. But plenty of more-established tech companies would have, I imagine.
Cronyism and incompetence may or may not be inversely correlated with success, but the market does not select against those traits, specifically.
To abuse a metaphor: as a business, you don't have to outrun the bear. You don't even have to outrun the other guys. And businesses are, in the end, run by collections of irrational-in-aggregate human beings.
Some good points were raised about this, and what I should've said instead is that it would be extremely difficult to suddenly absorb a lot of displaced government workers into private industry. But aside from the protections they receive of near-absolute job security in any market condition, I subscribe to the thesis that as government expands, rights are lost.
Apt description of Enterprise class business's. or any large organization of humans.
Also, this system (as most things people blame gov for) was made not by a government but by private sector capitalist companies. Maybe who've should quit outsourcing so much to these fuckups.
This program seems aimed at bringing in some of those kinds of people to government agencies: http://www.whitehouse.gov/innovationfellows
I think I've seen a few of them posting on HN, but I can't remember who. They seem to be especially pushing to get things like data.gov working in a modern way.
if anyone wants to help us fix the broken procurement policies that lead to systems like the one described in the article, let's talk too :)
Not everyone can start an 8(a) firm but we could get legislation passed that creates a more competitive way for any small business to work with the federal government.
We certainly had the technical capabilities to disrupt the process, but the process itself isn't even self-aware enough to know it's the problem..
I have had this attitude before, but I am not sure it's a healthy way to look at the problem. What you were flabbergasted by was the inefficiency of bureaucracy. But this is not a government problem per se: large companies can also become bureaucracies; large militaries tend to have similar tiering and divisions to bureaucracies. I think we sometimes think of bureaucracy acausally -- "it's just always been that way, it always will" -- when the truth seems more subtle: something like "bureaucracy is a scaling choice."
I really think that bureaucracies are born as a sort of divide-and-conquer approach to scaling a social system. So it's something like an O(n * log n) total overhead, where n is some abstract quantity of resources; and what you see on-the-ground when interacting with the government is (overhead)/(resource), which grows like O(log n). And it seems to have unbelievable overhead just because n is very large. The pain point which drives this scaling choice is probably free-rider problems. If you can't address free-rider problems, then the potential exploiters grow at least linearly with the number of resources you command, and your scaling is O(n^2) at best. So maybe in some sense bureaucratic solutions are "efficient" -- as crazy as that sounds.
In other words, if you ask "Why do people get shackled by these strict rules?", there are two tones (meanings) you might intend: one is despair ("surely we can do better!") the other is curious ("really, I want to know why that rule is in place!"). Usually the curious tone reveals some sort of free-rider threat. That nice guy who drives the schoolbus is constrained by bureaucracy; he can't hug or praise a student who tells him that she's feeling sad today because kids are picking on her for her perceived smartness; he must act as a robot driver. But why do those rules exist? Probably due to a fear of predators who might become able to lurk in the blurred lines. And the fear of predators is created by the scale of the school-system, which makes it hard for the human discretion of administrators to protect the system from having pockets of abuse without spending even more money than bureaucracy would cost.
Don't get me wrong, if an alternative system of government can do O(n), I'd love to see how it deals with free-riders. I'm really interested in crypto-anarchism. I'm just cautioning you that maybe the status-quo -- which yes, at this scale involves big companies with political connections paralyzing the process -- protects against a lot of people just running off with the money in exchange for the government-contract equivalent of vaporware.
How the system is supposed to work: Contractor does a small increment of work, delivers it, and gets paid by client based on the time they put in.
Exceptions:
1. Contractor doesn't deliver anything, or the quality is so bad that bad faith is assumed. Response: Contractor gets paid for claimed work on increment (which is limited by how much the contractor is authorised to do). Contractor doesn't get any further work. The contractor doesn't want this since they lose the income from future iterations.
2. Contractor does work, but client refuses to pay. Response: Client doesn't do any more work for the contractor. Contractor's loss is limited because the iteration is short. The client doesn't want this because they lose the benefits of the contract on the next iteration.
3. Contractor does work, but it doesn't fully meet the quality / scope requirements of the client, but the client can see progress is being made. Client pays the contractor, and the contractor and client work together to ensure that the next iteration moves in the direction the client wants.
Truer words have never been said! Try tendering or contracting at EMC...
This does not have to be a very complicated system at all. It's basically a real-time workflow system. Information comes in, is assigned a type and other data, and depending on the values is routed to various other people, which also tag and update it. In a way, it's almost like a bug-tracking system. Tickets get made. Tickets get worked on. Tickets get resolved.
Now combine that 10K problem with a large rollout, lots of administrative BS, and $2 Billion? It's wonder the damn thing works at all. A group of hackers could cobble together something over a week with better uptime and more reliability than I'm inferring from this article.
The technical complexity of the problem has nothing at all to do with the political and legal maneuvering required to get and execute a large government contract.
Or a large contract with any organization, as far as I can tell. In universities, the whole education-ware thing is a mess, with huge piles of contractor dollars going to junk like Blackboard, who manage to sell things to high-level administrators. By far the best education-ware software I've used is stuff that was developed for almost nothing by comparison: 1) a course-specific tool a student made as a senior project; and 2) a wiki. In industry, I have some third-hand knowledge of a large procurement project at a petrochemical company that spent somewhere in the 9 or 10 figures on document digitization and didn't deliver any document digitization.
I've worked both, and government is the big leagues. I've seen private organizations squander a few hundred million bucks here or there on disasters, but government projects can easily break into the billions or tens of billions of bucks -- and drag on for years. There's just much more money available, many more players who all want some random benefit from the project, and random rules to stick to. But you're right: this is a function of large numbers of really smart people all trying as hard as they can to do the best job possible. That's the craziest part about it. It's a systemic problem, not one that you fix by just somehow getting smarter people involved or buying a new project management tool.
I ask because I started playing that game when it was cutting edge, and finished about the time it was "very easy" (early to mid-90s).
I actually had a part-time job scanning large technical diagrams through a sheet-fed scanner in the late '90s, for a different company (a NASA contractor). It was technically not what I was hired for; the main reason they needed to bring in someone part-time is that they needed additional staff to help "debug" a space toilet. So my main job was using it, and recording some basic data (pH, etc.) in a logbook whenever I did. Since I was sitting around for a few hours a day in between toilet debugging, scanning technical diagrams seemed to be the main other thing that needed doing. Seemed like a pretty ad-hoc process, but I don't know what that one was like on the backend.
So basically SharePoint?
Sounds like a perfect storm.
I'd love to see an itemization for $88 Million and how many vacation homes were bought.
Government is not as bad as you think. It's worse.
Lowest bidder really causes a lot of problems. My company recently bid for a contract for an EU agency.
We were not chosen, because the winning bid offered senior staff at a rate of 0.1 euro per hour.
The bloody idiots at Government procurement calculated the average as (senior+junior)/2 and awarded the contract to these crooks with the flawed bid as their average was the lowest.
It's not unreasonable to assume they will debit enough extra junior hours to offset the senior time.
I am not even allowed to tell media about this, by contract. I don't think it's a rare event.
Just my €0.02.
I would say very rarely does it come down to simply being the lowest bidder or nepotism.
10 years ago I was briefly employed by a contractor for a state government that was working on a system a bit like this, although not hardly so life critical. The original system had been written and customized by the one of the big names like Lockheed or CSC and was pretty good as these things go ... but the state put continued maintenance out for bid and gave it to another company. With no transition, and really counterproductive budgeting.
I.e. around a year previously another guy was hired to do what I did, and let go after a month or two because of money.
What was I hired to do? The most difficult job of software archaeology I've ever attempted, not counting one code base that was impossible (https://news.ycombinator.com/item?id=6092960). The original programmers, probably on site, used SCCS (!) ... until they stopped some months before producing the binaries running on the system.
Other minor details: the system ran on DEC Alpha hardware and their version of UNIX(TM), and of course by then it was a doomed system with no migration path for binaries. The state budgeted the project and made its milestones based on estimates made by another consulting company, and those were not based on reality.
I bailed when it became clear there was no possible way to meet the state's demands, e.g. a point when I realized that if there was a way to recreate the production binaries I wasn't going to be able to find it, and they were assuming the sources were in a lot better shape.
They should. I think suing them over the deaths of hundreds of people and having the company that made this system go bankrupt is the very least they can do. Then the secret service can come in and crack some more necks. They got paid 88 million US dollars. For that you can make a system that Just Works or be honest up front if you can't get it done (on time).
If you could "bubble up" a "good enough" 911 system, it probably would be reliable and probably would scale. But the contract will be given to the company which will say they can provide a big custom solution on a deadline. It's almost worse if they believe it themselves. A dishonest vendor would at least understand the scale of the problem(s).
It's a lot of questions I wish more journalism would answer. On another note. It's interesting to think about open source. As some have mentioned, it's difficult to imagine the feasibility of actually doing an open source version and getting it adopted (though that may be because I don't know how EMS stuff is structured). Though I think all would benefit from a source code release of this system so we could actually crowdsource a better friggin' version because this one clearly isn't doing its job...since May.
Don't get me wrong, I see things get to user testing phases and we have reasonable ways of logging bugs and problems, but almost every project I've seen had kept on trucking to meet an arbitrary deadline regardless of how poorly any of the testing goes, then the system hits the floor, chaos ensues and everyone runs around in a panic until someone managers to staple together a solution.
Kind of depressing that it looks to be the same in the government with calls that are a little more important than ecommerce sales.
I want to say that the guy that is in charge of the 911 phone network spoke a few times at a call center conference I recently went to.
And the reason for this is simple..the one person who can pull the plug on a project of this magnitude, won't. Simply because s/he doesn't want to have to answer to having already dropped 88 million with nothing to show for it. There becomes a single person to blame at this point, whereas a failed system that gets implemented, suddenly allows for MANY people to be thrown under the bus.
I will say, though, that in the case of most projects I've worked on that are failing at some point, we do have something to show for it. We just have things that need to be fixed. Even a totally failed try is something you know doesn't work.
It seems like people equate admitting any failure as admitting total failure.
It's one of the only reasons I like my job. My boss is comfortable saying "this didn't work, that's fine, let's figure out why it didn't work and make something that does."
I recently had a project to reduce a certain call-type to our call centers. We implemented a new system in one part of the company after doing some research into the call drivers, and did a follow up a few months later. My followup showed nothing had changed. I don't have any problem saying that. It just means we need to look into which part of the plan failed, why, and how we can fix it. Either way we'll have a better understanding of the problem. I'd rather do that than fudge the followup, keep the problem, and pretend everything was fixed.
The taxpayer is much less forgiving about "failures" and tend to trot out pitchforks and torches at the drop of a hat. They're much more guided by perception than actual facts. So for them, there's very little room for a distinction to be made between utter failure, failure, almost failure, not quite a failure, maybe a failure, no failure...
Maybe I shouldn't say my boss is 'comfortable' with failure. He's just got the fortitude to deal with the realities of most situations and the track record to back it up.
Most of the managers and bosses I know of deal with projects in exactly the same way as this 911 project looks like it was handled. They push push push and don't accept any hint of failure because they think the perception others have of them would be unfavorable if they admitted some failure and tried to fix it. Of course, in this case it's the perception of the management and upper management members, and not a voting public. I have to say, though, their opinions don't seem much less fickle at times than the voting public.
In an anecdotal way, it doesn't seem like public works/government projects are immune to delays and push-backs. Seems like it might depend on the type of project as to whether there was public backlash?
I'm really just spitballing about something I'm not well-informed of at this point.
The main idea being that there's not a whole lot of forgiveness in the corporate environment I work in, but I stick with the boss I have because he's got the reputation and fortitude to own problems, when necessary, instead of trying to underbus everyone.
I wonder if Intergraph employed a testing plan quite as thorough.
Development cycle diagram: http://i.imgur.com/RaBSNHN.png (from the below PDF, page 4)
The entire paper: http://procon.bg/system/files/28.18_Lawford_Wassyng.pdf
They give rookie operators two months training, and after that just throw ’em in. They’re not ready.
Two months training is not enough training? This is implausible. At its core, they're answering the phone, determining the reason for a call and typing information from the call into a computer. They're not driving out to a site and performing cardiac bypass on people having a heart attack.
The whole thing seemed targeted toward eliciting an emotional response rather than a rational one.
>Two months training is not enough training?
I've worked in customer service and (while that's not exactly any way on par with what these operators go through) there's an entire volume of stuff to keep in mind before going in. Being an operator involves engaging in a fundamentally unnatural mindset that's often completely antithetical to what your normal responses would be. No matter who's screaming, shooting, dying, it's still your job to remain calm, clear and transfer as much information as possible to emergency workers.
Try listening to a few 911 calls. There are tons on the interwebs (LiveLeak has a whole lot). Be prepared to lose sleep for a day or two.
I did some contract work on network gear in a rural Ohio county's 911 center a couple of years ago and spent a few days around the dispatch center. I have a ton of respect for the work that the dispatchers do. I listened to them stay calm, cool, and collected while taking calls from people who were completely freaked-out (angry, injured, scared, etc).
The job is certainly a lot more than just answering the phone.
I was the CTO of a large publicly traded insurance company that had 3 call centers across the country and thousands of sales reps and customer service agents. I designed user interfaces and system architecture that enabled them to rapidly quote and service prospective customers and existing customers. I spent 1 day per week in one of our call centers every week with a headset on listening in on calls eight hours per day. I did that for 10 years.
I know call centers and I most likely know them better than you do. I think considering someone being given two months training "throwing them in" to be hyperbole.
The callers at 911 centers (and I'm guessing you still haven't listened to any of these calls) have to handle different systems (with no ability to modify the UIs) contact different services, handle vital data and filter it out from non-vital ones when anything critical lost would mean deaths (as it had been).
E.G. To become familiar with police, emergency medical and fire codes. Respond to caller situations appropriately, calmly and above all else, cogently. This is a far greater burden than any insurance operator will ever face in the line of work.
It's most definitely not hyperbole.
- Where are you calling from? (Shockingly, this one is absurdly difficult to get from some people)
- What's the emergency? (Often times, they don't even know)
I suspect 911 calls are no less complex than scheduling people for a doctor's appointment.
But training people to deal with crashes and exceptions is very hard, especially if it is under pressure. Often, this sort of training only takes places as catastrophes happen. Imagine you're an operator, and you have to use a system that's made of failures and bugs, you're exhausted from working doubles, and you can't debug anything: when it fails you have to rely on someone else.
(I have never worked in 911 operator facilities, only industrial operations.)
Say, did you hire the programming team that wrote the software they are using?
This attitude is one of the reasons why so many people have such terrible experiences calling in to companies. I'm talking about companies that have nothing to do with real life emergencies and life threatening consequences.
Management tends to think "they just answer phones, so we can hire anyone, put them through a week of training and bam, they should be good to go, right?"
I work in a pretty standard e-commerce environment. I was in training for awhile, and now some data analysis and software development. It's like just about anything else. Predict how long you think it will take to competently train someone and triple it. It's not easy to get people ready to handle the majority of the calls you get in a call center, and it's really really hard to get them to handle the absurd number of one-offs you might have. Again, that's without even considering the number of call types and the weight of the calls I'd imagine 911 operators get, all while having to stay polite and calm.
Two months for a 911 operator in a major city like NY sounds kind of nutty to me, especially if the training is in the format I'm expecting it would be, and not something totally out of the box.
You've missed the mark pretty terribly on this. The reason why so many people have such terrible experiences calling in to companies is because of substantial training.
People in call centers today are trained to give you poor customer service. I can't tell you how many scripts I've seen that are written like:
1. If customer asks for X, tell them we can't do that
2. If customer insists, explain we can't do X because of A, B and C.
3. If customer still insists for X, tell them we can't do X but we are willing to do Y
4. If customer still insists for X, tell them we will call them back and then don't call them back.
5. If customer calls back for X, give them X
And these types of scripts are written for things where a company is legally obligated to give their customer X.
Sure, there are training issues but by far you are receiving poor customer service because the company doesn't really want you calling in the first place and they're training you not to call.
What you're saying is definitely a problem in some (maybe most) call centers. Companies tend to see call centers as places where money goes to die, instead of a potential treasure trove of information about their customers and what's going wrong or right with their strategies. I think that does lead to what you're talking about. Everything becomes about low balling the customer. If they have a return, you try to give them ten bucks to keep it. If they think the item is broken, you tell them why it's normal for the item to be that way. I'll be the first to admit that I underplay that portion because my financial situation is fairly good and I'd leave a company that acted that way. It's misguided and doesn't do anything for building your brand.
I'd take that a step further, though, and say that your scenario suffers from another big problem. Some call centers place a huge priority on wrote scripting and the customer ends up feeling like they never stopped talking to the IVR. Some reps can put life into a script, but most can't, and you end up with a really sterile, impersonal experience that screams "you're a statistic to us."
My personal experience has been, the more you can let your reps drop the script, the higher your CSAT/NPS scores will climb (obviously to a point). Of course, people have guidelines, but I tend to think most of those go too far. If you're in a brick and mortar retail shop, you don't usually have someone hiding behind a shelf marking down whether or not you used the customers name 2.3 times during the interaction. And I should mention that this is going to change depending on the type of business. An e-commerce site processing a return isn't going to be quite as mired in legal scripting as a financial institution laying out the terms of a loan.
As a final note, I'll get back to the 911 operator context. Yes, the article is written in a way to elicit a response. I'm not going to hold it up the standards of a well-respected peer-reviewed journal article. I still absolutely think you're wrong with that one, for a lot of reasons others have spoken on. It's not in any way implausible or hyperbolic to think a 911 call center would take two months to adequately train a rep, especially if they're using fairly standard training practices.
"911"
"You have to send the cops over!"
"What happened?"
"It's my parents ..."
If they don't triage properly, they'll be sending the police to deal with angry kids whose parents won't give them their pocket money. Or they'll not send the police to a violent confrontation.
Also, you have to consider the kind of people who make a lot of 911 calls. It might be a drug user getting threatened by a dealer. It could be a drunk guy, who's had an accident. It could be something really embarrassing ("I fell on it"). Doing triage can be quite difficult, when the caller doesn't really say what has happened, but they need to find out enough to justify sending help.
This system is disgraceful, and endemic of cheap outsourced computer systems that places like EDS have been getting away with for a long time.
When signing up for a phone, you must pick a 911 provider. The cost is fixed, so you're not choosing based on saving money, but based on quality of service. Each provider would be audited and would publish response times, etc.
You could choose the government program described in this article, or go with someone more technologically advanced.
If someone needs to call 911 for me, I want to know it's regulated and up to quality - not based on whether the person who happened to pick their provider actually bothered to research a good one, etc
Mandated monopolies have a way of discouraging innovation and depressing quality.
Call centers might be bad sometimes regionally, but that doesn't mean they can't work in principle. For example, in my home country (Germany), there is a public dispatch system for all emergency services and after having had to use it repeatedly for a range of things from routine to disaster response, I have to say it works really well. And I wouldn't say Germany is exactly the paragon of efficiency when it comes to public services either. So with a little bit of political will, it's achievable - as it should be.
Some links: (Australian) Victorian Police Corruption http://www.smuggled.com/Inte1.htm
NZ Parliamentary Record: http://www.vdig.net/hansard/content.jsp?id=76345
"This year [2000] we see the very two people who were involved in the police force who were making all the recommendations about Intergraph, go to work for Intergraph."
Let's presume we're interfacing with public services here, just for shits and giggles. What happens when grandma Jean forgets to pay the bill one month? Does she lose access to those public services when she needs them most because her 911 is redirecting to a busy tone?
2. Proponents of government services always bring up cases where grandmas or disabled or poor people somehow suffer because they don't have enough money/physical abilities/brains to handle things. I agree those cases exists, but it very often is presented as if half of the society is in someway dysfunctional and requires government help (even if that's indeed the case, we should rightly see this situation as abnormal and fix it, not feed it). But the truth is, in a society there's indeed always a % of people who are dysfunctional. My guess is, it should be no more than 5-10%. Those people can be helped by voluntary donations and charities. Beyond that, we got people who made their choices in life (not buying insurance, eating unhealthy, drunk driving etc.). While I'm not necessarily claiming those people should not be helped when they need it, but the decision to help or not should lie with the body that actually does the job: that is, if a drunk uninsured driver is brought to a hospital, it is up to the charity that sponsors it and the hospital managers or doctors to decide whether to help him or not. By leaving this decision up to the government we simply incentivize all sorts of bad behaviors at the expense of people who have no intention of supporting it.
2. So, in your world, the guy who seemed to have alcohol on his breath can be left to die at a hospital. Sure. I think that's poor example, you'd be hard pressed to find doctors who would refuse to save a life in such a case. The reason I trot out grandma is because it is the poorest and weakest in our society that need protection the most, and your system would leave all that in the hands of some charity... hand-waving away all the issues that creates.
2. In my world what's most likely to happen is that this guy would be helped immediately, but the hospital stuff later will report the incident, the guy will lose his driving licence forever, will be unable to buy alcohol and will be required to pay to the hospital over time. I'm not gonna go into detail of how it is achieved without a government, but it's quite very possible and would even be enforced more effectively.
So, yes, my main point is: forced universal anything is bad and immoral. Everyone should know exactly how and where their money go and be free to decide what to do with their money. If I dislike how my charity works and if I dislike that it doesn't help drunk drivers, I switch to a different one which does. If everyone in a society agrees to pay for universal healthcare voluntarily without a threat of IRS or similar agency showing up if they don't pay, then I have no problem with it.
It's interesting how, when this particular viewpoint comes up, the hypothetical individual is always deeply irresponsible, borderline malicious. But the person making the argument never is.
> "Now, say I'm a Canadian citizen too: would you give me the same amount of respect and not require me to pay tax money for the things I have no intention of supporting and allow me to only pay to my insurance company?"
No, because this is a fundamental philosophical difference that runs to the core of who we are. I value the collective, and am willing to make sacrifices to my individual in order to advance the whole.
As a Canadian citizen myself, stay on your side of the border, please.
No, I may be the most irresponsible person in the world. The difference is, I'm not asking you or anyone else to pay for it.
> No, because this is a fundamental philosophical difference that runs to the core of who we are. I value the collective, and am willing to make sacrifices to my individual in order to advance the whole.
So would you then personally come to my house with a gun and demand money and escort me to jail if I politely refuse to pay, explaining my reasons? Will you do this, knowing my family is going to suffer? Will you look my kids in the eyes and say "Your daddy is a criminal because he refuses to pay for what I think is right".
> As a Canadian citizen myself, stay on your side of the border, please.
Please do not assume everyone here is from the US. I'm not talking about a specific country, I'm talking about a principle. US is socialist in many ways too.
Sure you are. The moment you're out of your financial depth you'll scream like a little girl and recant.
You presume that a private system yields some kind of low-cost alternative to single-payer systems. That's not the case, and it's so far from true in the US that it's laughable.
You are the one who needs to look into his kids eyes and explain why they can't have critically needed health care. It should start with "Daddy can't do math", and end with "and so my beliefs were more important than your well-being".
We don't throw people in jail or point guns at them, and frankly even if you weren't paying income tax you would be paying tons of sales tax that you can't avoid. Worst case scenario you'd be forced into bankruptcy, tax debt is dischargeable up here, unlike in the US.
I'd leave out this part.
edit: or to be clear, I'd be as likely to say that as I would be to say to the children of a burglar, "I'm arresting your father because he wanted you to have nice things."
Here's how Canadians (and I'm one of them) think about health care: Some of us do crazy, unhealthy stuff, and some don't. We all have the same healthcare. Be free, do your thing. Live your life the way you want to. It costs way too much to have different healthcare for different people.
Some of you "land of the free" people really don't get it.
Being free to leave your job, and free to live your life that way you want to (without actuarial oversight) are pretty important.
Those are way more important to me than the last nth percent of optimization I might be able to do on my own personal wallet by taking on all the idiocy of privatized, customized health care.
As a counterpoint, I refer you to all the other counties in the US where receiving this vital service isn't contingent on your bills being paid. I can see these people losing their home to a fire because they failed to acquire home insurance, but I can't see how moral people would stand by and let a house burn to the ground over an unpaid bill.
On the other hand, in this case they had already deployed, and it would have been nicer if there had been SOME price that they had thought of as the "fee" for service to non-subscribers. 5k, 10k, etc: better than losing the house. Once you advertise something like that, though, people will stop subscribing, and decide to gamble that they won't have a catastrophe. The price has to either be high enough to discourage that, or high enough that the fire department can cover its operating costs by servicing X fires per year.
Many rural U.S. counties have no standing fire fighters. Fires are fought by volunteers, often using donated equipment. There is nothing immoral about a town offering enhanced fire service to subscribers outside the town.
Similar arrangements are also common between ambulance companies and subscribers. Old people with a fixed income consider it a great deal to pay $50 a year or so for discounted service.
Well, you could live on a boat in international waters, or renounce the use of all forms of currency and live on pure barter, however you seem to be asking for a state that cannot raise revenue.
There are some of them around the world, but you might want to hire a small military detail if you want to live there, as funnily enough they have a few security issues.
Your argument is that it is unethical for someone to use force to compel someone to do something against there will, right? Well, that works great as long as everyone shares your ethical views. Unfortunately, not everyone does, and there are people in this world who would happily kill you and take all your property if they could. It doesn't even have to be all that many people - if just 0.1% of the population believes "might makes right" and you come in contact with >1000 people, you're probably dead.
The reason we have the state is so that we can realize certain economies of scale in protection. We entrust a monopoly on the use of physical force to the state; in return (in lawful democratic regimes), it agrees to use that force only when just. That way we can trust that we can go about our business without being randomly killed by a dude on the street, because they know that they are in for a world of hurt if they kill someone. There have been times (eg. the Wild Wild West in America, or post-Gorbachev looting & unrest in the Soviet Union) where the state has not had that monopoly on force, and it usually ends up being very difficult to conduct business during those times.
A state is just a name for people accomplishing certain things on their own; its not actually a real thing with independent existence.
Next you will probably tell me: but how can you compare taxation to slavery? Well, both are done by human beings to other humans against their will because the majority implicitly agrees. In both cases, if a person resists, he is punished. Tell me, how is it moral? The only difference is the degree to which the freedom is restricted and fruits of labour are confiscated. If you believe a person made his money in a dishonest way, it is justified that this person is tried and convicted. However, if you agree a person made his money honestly, by benefiting other people and creating value, what moral right on earth do you have to take any of his earned money for any purpose? He didn't rob anyone, he created value, out of nothing. Not you, not any other person has any moral right to confiscate any part of it.
If you want to create an alternative economic system that requires no physical territory or external inputs to enable it to run and therefore can exist without taxation, then feel free.
But let's say I agree. I'm gonna switch to Bitcoin and start my own country. Problem is, no one's gonna let me occupy a territory I choose even though it may not be used at all (!) by a government or any of the citizens. The moment I start building my house in the woods, someone's gonna notice and send people in suits to evict me. What's the justification for that? No one uses the land anyway! Why is some government more entitled to it than I am?
As for entitlement to territory, the only entitlement to territory that really counts is the ability to keep hold of it. If you want an anarchist utopia, you may have problems keeping it for any length of time unless you are fairly inaccessible or fairly martial.
This isn't particularly fair, admittedly.
ultra reliable, super scalable
You're right! And then we can make a one-size-fits-all Retail Bank program that every bank will use. And a one-size-fits-all Policy Administration System for insurance companies. Once we've solved those problems we can move on to a one-size-fits-all Dispatch system for every company that has employees in the field.
Then we can wake up and realize that there's a reason why this hasn't happened in the last 50 years. Any system can accommodate maybe 80% of a particular niche's needs. But I'll be damned if you can get business users to agree on that final 20%. One says "black", the other says "white" and then you have some oddball that says, "potato."
And then there's the whole continent of minefields in regard to disparate devices, patents, trademarks, proprietary system interfaces, etc.
If you do take this on, best of luck to you - but don't quit your day job.
Linux is open source... Office Libre... Go convince a public office to switch to Ubuntu..
Saving money is often at the bottom of their focus.. and then you have competing interests releasing figures that show it is more costly to switch to free alternatives.
Public spending is a corrupt cycle.
Yes, First they ignore you, then they laugh at you, then they fight you, then you win.
Sorry, I know it's trite, but there's some truth there. Being ignored for years is not necessarily a reason not to do something. In some cases it's the very best reason to do it.
[EDIT] - would those people down-voting please explain why?
Maybe it could be started by a person who works (or worked) at the dispatch center, and is also a programmer (there must be at-least one of those in the world too).
The point is that NYC laughing/ignoring/fighting you is not a good enough reason not to start. (and who's to say that NYC would do any of those things, given the current state of affairs)
http://www.wired.co.uk/news/archive/2013-03/25/ubuntu-china
As a proof of concept goes, the world's most populous country is pretty good.
It's actually an incredibly well studied case of an IT project failure. It's studied as a textbook case in just about every IT project management course in the UK.
Interesting that NYC is repeating what sounds like exactly the same mistakes.
http://en.wikipedia.org/wiki/London_Ambulance_Service#System...
(or just Google for "london ambulance service case study")
Even though their old system was some DOS based thing from 1991, it worked. It was fast and simple. Now they have some new, enterprisey, WPF + EF behemoth that is buggy and slow.
Sure, new software will almost always have more bugs than the system it is replacing; the old system, whatever its flaws (perceived or actual), was battle-hardened. It was probably written in a time when speed was more important than the technology stack it used.
These people don't care so much about a nicer looking interface. They need something that works and stays out of their way so that they can do their job.
I can only think of one example: the UK's Government Digital Service (who built www.gov.uk). Admittedly not a mission-critical project that saves lives, but they actually managed to bring the hundreds of different government websites under one site, re-wrote and re-rorganised the content so it was easier to find and read, and actually improved considerably on what was in place before.
It's an exception rather than the rule though to most government or local council projects in the UK. Any other examples of success? (There must be some)
http://www.intergraph.com/ - Flash header on company's homepage hangs on Chrome. Not a good start.
Which of course is an even worse sign.
" © Copyright <SCRIPT> <!-- var year=new Date(); year=year.getYear(); if (year<1900) year+=1900; document.write(year); //--> </SCRIPT> "
The operators should've had final say on production worthiness. The arrogance of top-down "solutions" that don't work.
Other observation: results are often inversely proportional to budget, up to a point. Spending instead of thinking is not a solution.
Does anyone know starting salaries for air traffic controllers?
Pay follows a "public school teacher" trajectory, or at least like the trejectory in my district. Such that you'll start your career eating ramen and living in your car, but by retirement you'll be hauling in $100K or so.
The FAA workforce is gray. No other way to say it. So you tend to see ridiculous averages like your local TRACON might pay on average $80K. However as a noob you'd only get $30K or so... you need 30 years in to get rich.
Most pilots get to know ATC people for obvious professional reasons. You can probably get interesting anecdotes from a private pilot.
Also more than a few ATC personnel ARE private pilots. Helps to understand the other side of the radio, so to speak.
I am told there is a narrow window where you're "too old" to be an ATC and get mandatory retirement, yet still young enough to be a CFI, and this is where gray haired flight instructors come from. I do not know if this is true or not. In other words mandatory retirement age for CFI > ATC but not by much.
Starting pay for a 911 operator is about $26,000
Yea, obviously not as high a priority as stock exchange market where milliseconds matter!
Glad I don't trust my life on such a broken system.
Answer me please!
Windows programmers and their 'Just reboot the machine' mindset should NEVER be allowed anywhere near a mission critical application.
Why? Here is your answer, straight from the original article:
=======
... Intergraph keeps telling us, “Shut your computer down and reboot. It’ll just take a minute.”
How many lives can you lose in a minute? ...
=======
I have seen Java-based systems running on Linux that stall/hang and run agonizingly slowly under what should not be demanding levels of use. Bloated, heavyweight software stacks exist on all platforms, and they are favorites in government contractor architectures.
911 call centers have to interface with many different computer systems (for example the National Criminal Information center, department of motor vehicles, FBI CJIS, in house databases, court records systems, mapping, computer aided dispatch, weather information, etc.
Many of these systems (the NCIC, and CJIS have very strict requirements on what can and can not connect to their systems, and on top of that who can use a terminal that connects to their systems).
Its not an easy task. However comparing it to twitter is just ... wow
Tight coupling of many separate systems is a classic anti-pattern in software.
And yes 911 operators do have access to NCIC and FBI CJIS. They often are running people involved in incidents when cars are en route. Knowing who they are dealing with often dictates the number of cars that respond, the officers tactics when responding, etc. Both of those databases have nationwide info that states share on warrants, convictions, drivers licenses, etc
[1] http://highscalability.com/blog/2013/7/8/the-architecture-tw...
I do agree, a well designed and implemented system running on Oracle and Windows can certainly handle something like 911 dispatch and do it well.
You (and I) truly believe that the telcos have proof positive location on our landlines and cells, and this data is collected whole hog by the govt, but BUT somehow they cant locate a caller having a heart attack on long island.
You know, the govt has the systems it deserves, and I really dont care to hear whining about pencil and paper.
It takes no longer to write down on pencil and paper a 911 call and theres a mandatory handoff anyway.
This has nothing to do with the folks working the phones not being up to par. They are. It has everything to do with big telcos and govt contractors holding taxpayers hostage.
We dont need them. Use the pencil and paper, make a phone call to the next dept. That takes the same amount of time as data entry.
Grow up, be a human being and stop pretending that life and death situations are some variant of farmville.
Operators had to record 911 calls with pen and paper
I know I work all day on a computer but I wouldn't consider using pen and paper a hindrance.
Having to record things on pen and paper is hardly sensationalism. It's more or less a worse case scenario in the 911 dispatching arena...