Inside Team Romney’s whale of an IT meltdown
arstechnica.com
arstechnica.com
To build Orca, the Romney campaign turned to Microsoft and an unnamed application consulting firm.
This sounds like a lot of failed IT projects in large corporations -- dreamed up by upper management types who don't know anything about tech, then farmed out to consultants and tech vendors for the actual implementation. The consultants and vendors then either proceed to wander in the weeds expensively due to lack of direction, or actively seize on the client's ignorance and take them for a ride.
The problem is that the people nominally in charge of the project, the upper management types, don't have the experience or expertise to know they're getting taken for a ride until it's too late. So there's no way to hold the contractors accountable, or to get the train back on the rails once it's jumped off.
In my experience, this is 100% correct.
At least this mess has been exposed to some degree. There's something of a saying around these kind of projects to the effect of "This project -WILL- succeed".
The subtext being that that no matter what the facts are, the project will be promoted publicly as a success.
It's incredibly disheartening to see an abortive mess transformed into a sparkling success via fake dashboards, paid press and the like.
http://www.buzzfeed.com/rubycramer/messina-obama-won-on-the-...
http://swampland.time.com/2012/11/07/inside-the-secret-world...
Sounds like they spent quite a bit of time and money on that. Nice article, by the way, though it's obviously a bit of a puff piece.
On one hand, management consultants are typically working with the executives of a company, and usually one of those executives will have been a former consultant (the executive boardrooms of companies are filled with them). Not to mention that both consultants and executives usually have the same set of "skills."
On the other hand, one former management consultant I spoke to told me that executives would often hire consultants to tell them what they already knew. The only purpose of bringing the consultants on-board was so that their recommendations would serve as political validation in case they had to explain their decision (i.e. if they failed).
They are hired in order to protect against liability. They are paid money to say what clients want to hear (but in an official "report" and "power point format") and then if shit hits the fan executives can always point to "But look we hired the best of the best, these experts all agreed we should go this, so it is clearly not our fault".
Mitt Romney’s campaign handed out more than $200,000
in bonuses last month to senior staffers, according
to new disclosure records filed Thursday.
Richard Beeson, Romney’s national political director,
received a $37,500 payment on Aug. 31 in addition to his
salary, according to records filed with the Federal
Election Commission.
When I first heard about this, I remarked to a friend "what the hell are they doing giving out bonuses? The bonus should be that your candidate gets elected!"
The Orca thing was obviously an epic fail, but that doesn't mean other people didn't deserve bonuses - maybe their part of the job was just fine.
He probably also had other unknown milestones throughout the campaign in an effort to keep staffers motivated.
For that reason, we had two Technical Rehearsals. The purpose was to test both our systems and our procedures, and identify how they could be refined for the real thing. It was someone's job to actually design a huge number of expected and unexpected issues so that tech operations teams at each venue could be tested as to their response. Issues ranged from broken printers to network meltdowns to broadcasting failures, etc. We had two Technical Rehearsals for hopefully obvious reasons. And before the Technical Rehearsals, we also had the test events (i.e. world class but lower profile sporting competitions held at the Olympic venues to see whether we fell short in putting up a world class competition event).
We also had Disaster Rehearsals to test our disaster recovery systems and procedures, though I wasn't personally involved in those. I did get a string of text messages when it was happening though. "Primary datacentre flooding, starting emergency processes to switch to secondary datacentre now." etc. Never mind that nobody could ever believe that the primary datacentre could get flooded in a million years. The attitude was that any disaster was possible, and we had to prepare for the impossible.
If you have one shot only, and that shot is important, you need to have a large process committed to making sure that one shot goes well. And that process needs to be managed by someone experienced in that type of thing.
Also, the Ars article seemed to imply they wanted to keep how it worked a secret, so big testing like this would have tipped Obama what they were doing.
Even if you supported Romney, this has to give you some pause about the man's management skills and who he hired to run things.
-You want a CMS? Use SharePoint!
-How about site for delivering BI dashboards? Use SharePoint!
-Large, mostly static public facing site? Use SharePoint!
/Ugh
The solution? Instead of the project administering Sharepoint, the admin was handed off to the division of the corporation that specializes in administering Sharepoint. (It's a very large company.) Now we have two-factor authentication and double logins to upload monthly status reports. And still nobody can actually communicate.
Of course, after Tuesday, they're all looking for work now.
The POTA doesn't actually personally work on many projects. He relies on people to get them done. If Romney couldn't get something as crucial as this right, then how the heck would he have gone running the country?
There's a big difference between micromanaging and verifying that your critical infrastructure is in place when you launch.
If his job were micromanaging, then "Hey, he wasn't in the room" would excuse him of responsibility, but that is not the case. Since his job would be pretty much the greatest opposite of micromanaging I can conceive of, there can be no excuse. The failure of his campaign is his failure.
That still seems a bit unclear to me actually. Not sure how to effectively reword this.
The United States has the following federal executive departments:
* Agriculture
* Commerce
* Defense
* Education
* Energy
* Health and Human Services
* Homeland Security
* Housing and Urban Development
* Interior
* Justice
* Labor
* State Transportation
* Treasury
* Veterans Affairs
Now there is literally no way you can micromanage you way through this. Nobody, and I mean nobody can possibly single handedly do it.
However, you can delegate. Part of that is to choose honest brokers and managers who know their area well enough to ensure that their departments work well.
In this situation it is your job to delegate to the right people. If you are the President of the United States of America and you aren't able to do this with your own campaign to get elected, you must take responsibilty for your failure and accept a great deal of responsibilty for the fact you did not get elected to high office.
Harsh, but being the POTA is not a job everyone can do. I know I couldn't.
I mean, wasn't he expecting to see the results?
Wait, what?
There's a difference between people taking responsibility for what their staff do (thus, he should take responsibility for it) and people actually being responsible for what their staff do - especially if those staff are not staff but sub-contracted service providers.
To provide some scale, Romney's campaign employed approximately 500 staff. The Federal government employs more than 4 million.
And if we start counting how many projects endorsed, sponsored and initiated by the federal government failed recently, we could be here all night.
http://www.cbsnews.com/8301-31727_162-20115038-10391695.html Read the docs yourself and try to explain how comes Holder says he did not hear about F&F until 2011 when these documents are from 2010 and are addressed to him and his immediate subordinates. Only one explanation is logical - Eric Holder flatly lied to the Congress. And got away with it. That's the coverup, and if seeing it with your own eyes is not a proof, then I don't know what could prove it.
Do you realize the reason why Mexican drug gangs buy guns in Arizona? It's because the gun laws there are extremely, extremely weak, to the point where random 18-year-olds can walk out of the door with $20,000 of guns, with no oversight. Who keeps the gun laws this weak? Republicans. Who then cynically try to frame Obama for a problem they created.
Even if David Voth ever asked anyone to sell a gun to a bad guy-- and if you read the Fortune article, it seems pretty clear that he probably never did-- so what? Are we going to start prosecuting undercover narcotics agents for dealing drugs? This whole thing just shows such a tremendous lack of logic that it's just... gah.
The especially ironic thing is that since the witch hunt, the ATF's seizures of weapons has dropped 90%. There will be a lot more Brian Terrys as a result of this stupidity.
Now back to F&F. If you took a moment to educate yourself, you'd learn the whole point of F&F was to CIRCUMVENT the gun laws and allow the cartels to buy massive amount of guns from dealers that otherwise they would not be able to buy. The dealers signalled the ATF multiple times, but where told to shut up and sell. The rank agents asked to arrest the buyers and take the guns, but were told to stand down and not interfere since it is an operation to track the guns into Mexico, so everything is going fine and the guns should "walk". In fact, there was no way to track them and no tracking actually happened - the guns just disappeared. Blaming gun laws for the sales that were specifically protected by the government agency whose purpose is to monitor gun sales - well, yes, mind-bogglingly stupid. Using government agency fuckup to prove that we need more government involvement - mind-bogglingly stupid. Trying to sell guns to Mexico without actually being able to find them and without telling anybody on Mexican side a thousand guns is heading their way - mind-bogglingly stupid. Blaming Republicans for the mess - I'd say it takes a lot of effort to be this stupid, but apparently it does not. It comes naturally to some - whatever the problem is, Bush is at fault. Doesn't require any effort or any thinking at all. Saves a lot of mental energy that can be spent on blaming Bush for more stuff.
The whole F&F operation was a poorly executed screwup. It should have never happened and whoever thought it is a good idea should be reassigned to issuing parking tickets. However, the worst part is not this - anybody who follows the current events knows that for government projects screwup is a normal mode of operation. The worst is that AG Holder lied to the Congress about it - brazenly, openly and with full understanding that he will get away with it - and with President's help he did. For me and you, lying under oath would mean jail time. For him, according to you, it's just fine and if anything is wrong, the Republicans are at fault anyway.
>>> The especially ironic thing is that since the witch hunt, the ATF's seizures of weapons has dropped 90%.
Given that the purpose of F&F was specifically NOT to seize weapons that could be seized - relating the inevitable blowup of F&F to the lower seizures, is, well, you know... gah.
"Quite simply, there's a fundamental misconception at the heart of the Fast and Furious scandal. Nobody disputes that suspected straw purchasers under surveillance by the ATF repeatedly bought guns that eventually fell into criminal hands. Issa and others charge that the ATF intentionally allowed guns to walk as an operational tactic. But five law-enforcement agents directly involved in Fast and Furious tell Fortune that the ATF had no such tactic. They insist they never purposefully allowed guns to be illegally trafficked. Just the opposite: They say they seized weapons whenever they could but were hamstrung by prosecutors and weak laws, which stymied them at every turn. Indeed, a six-month Fortune investigation reveals that the public case alleging that Voth and his colleagues walked guns is replete with distortions, errors, partial truths, and even some outright lies. Fortune reviewed more than 2,000 pages of confidential ATF documents and interviewed 39 people, including seven law-enforcement agents with direct knowledge of the case. Several, including Voth, are speaking out for the first time."
We found that the lack of seizures and arrests was primarily attributable to the pursuit of a strategic goal shared by both the [Phoenix] ATF and the U.S. Attorney’s Office -- to eliminate a trafficking organization -- and the belief that confronting subjects and seizing firearms could compromise that goal.
This tactic was significantly different from previous tactic, e.g. in Project Gunrunner - where straw purchasers were arrested at the moment they handed off the guns. Here, they were allowed to proceed - to great dissatisfaction of the field agents. Whoever though up this was thinking that this would lead up the chain to eliminate the bigger organisation - but due to multiple failures in the execution - such as ATF agent tracking the buyer going on vacation and the buyer just disappearing with the guns - it did not happen and guns just disappeared in Mexico to the hands of drug cartels.
Nothing to do with gun laws. Anti-guns bunch would have to look in other place to feed their political agenda.
In a meeting on Jan. 5, 2010, Emory Hurley, the assistant U.S. Attorney in Phoenix overseeing the Fast and Furious case, told the [ATF] agents they lacked probable cause for arrests, according to ATF records. Hurley's judgment reflected accepted policy at the U.S. Attorney's Office in Arizona. "[P]urchasing multiple long guns in Arizona is lawful," Patrick Cunningham, the U.S. Attorney's then–criminal chief in Arizona would later write. "Transferring them to another is lawful and even sale or barter of the guns to another is lawful unless the United States can prove by clear and convincing evidence that the firearm is intended to be used to commit a crime." (Arizona federal prosecutors referred requests for comment to the Justice Department, which declined to make officials available. Hurley noted in an e-mail, "I am not able to comment on what I understand to be an ongoing investigation/prosecution. I am precluded by federal regulation, DOJ policy, the rules of professional conduct, and court order from talking with you about this matter." Cunningham's attorney also declined to comment.)
It was nearly impossible in Arizona to bring a case against a straw purchaser. The federal prosecutors there did not consider the purchase of a huge volume of guns, or their handoff to a third party, sufficient evidence to seize them. A buyer who certified that the guns were for himself, then handed them off minutes later, hadn't necessarily lied and was free to change his mind. Even if a suspect bought 10 guns that were recovered days later at a Mexican crime scene, this didn't mean the initial purchase had been illegal. To these prosecutors, the pattern proved little. Instead, agents needed to link specific evidence of intent to commit a crime to each gun they wanted to seize.
Yep, sounds like this is all the fault of some renegade ATF agent. Nothing to do with the laws in place, no sir.
Given that the purpose of F&F was specifically NOT to seize weapons that could be seized - relating the inevitable blowup of F&F to the lower seizures, is, well, you know... gah.
Here's a deep thought for you. If "Fast and Furious" was really the cause of those guns getting to Mexico, then why are guns still getting to Mexico, now that "Fast and Furious" is over? And why did so many of them get there before the operation started?
Nope, it is not. Random 18-year olds do not resell thousands of guns to Mexican drug cartels. The guys that did were known to ATF and sellers repeatedly alerted ATF - the agency that by law is designed to stop it - about them. The system worked - until it did not, because ATF thought they would be geniuses to allow the guns to "walk".
>>> The ATF, by law, can't even set up a database of who bought what gun.
By which law? This sounds like complete baloney - ATF knew who bought the guns. They just didn't intervene because this was the plan - to let the guns "walk", not because some law prevented them from knowing who bought what.
>>> If "Fast and Furious" was really the cause of those guns getting to Mexico
What you mean "if"? It is the fact that F&F was the reason of those guns getting to Mexico. If F&F was not underway, these buyers would have been stopped and the guns were confiscated before they reach Mexico.
>>> why are guns still getting to Mexico,
Because there are other ways of getting guns to Mexico, duh. You'd make a fine defense lawyer: "If my client sold these drugs on the street, as this footage and this undercover officer suggests, how comes drugs are still sold on the street when my client already had been arrested?". Too bad this defense makes no sense at all - of course ATF is not the only channel Mexican gangs can smuggle guns. That does not diminish the fact those specific guns were smuggled with full ATF cooperation and help, which should not have happened. And this does not diminish the fact AG lied about it after the fact that this ill-conceived mess of an operation was exposed. They knew it's a mess, they lied to cover their asses - and they got away with it. And Obama shares the blame for it.
Really, with this kind of argument I would not be so quick to throw around the "stupidity" word if I were you. I'd first learn to figure out difference between "X caused this specific instance of Y" and "X caused all instances of Y ever".
If the state DA isn't will to prosecute, then what would you suggest the ATF agents do? And if the ATF agents know that guns are being purchased, but then cannot arrest those individuals because they do not have probable cause, why would you call that "allowing the guns to walk"?
http://features.blogs.fortune.cnn.com/2012/06/27/fast-and-fu...
While I get your analogy, it simply isn't the way campaigns actually work. The candidate is the CEO - he's making the big decisions based on data fed to him or her by his campaign staff.
I voted for Romney and yeah, this does have me reconsidering just how competent he is.
I dislike Romney and did not vote for him (or Obama), but I don't think this should necessarily be seen as a major strategic failure. Was this project going to get Romney the 1 million + additional voters he needed? It's hard to think that Romney believes that.
Several accounts state that Orca positively thwarted traditional Election Day GOTV efforts. Campaigns think those matter enough to put considerable effort into them. So yes, this could well have made the difference, and its failure does have me rethinking Romney's actual management ability.
Unless the contract (to build this application) had some kind of clause like "This thing must work perfectly or all the money will be refunded" then this is a great contract to get. Let us consider this cynically for a moment. I wish I could land contracts like this one. Can you imagine building an app that the client will only need for 1 day, and you still get paid even if the thing does not work? The chance for fraud is huge. Who would even allow this? The people pitching this must have been some very talented salespeople (perhaps they were convincingly able to argue that they were such huge Republican supporters that if they lost then that would be punishment enough, as it would be so very sad for them to see Obama win?)
I've worked with a lot of startups and clients and the staff always needs training and you never, ever know what all the bugs are until you've gotten some feedback from the staff who are using it for real. To suggest that a software project is done the day you first give it to the staff is idiotic -- that's simply the moment you move to real world testing.
Some developers might suggest that a really good suite of unit tests might provide enough insurance, but that is not true -- I've worked with non-technical staff who tend to confuse bad UI decisions with software bugs, and if a UI decision is bad enough, I think it is fair to classify it as a bug. If some_important_action is crucial to the staff, and none of the staff can find the link to some_important_action, then the UI decision that hid it is effectively as bad as a Fatal Exception, but no amount of unit tests can tell you that the staff finds the UI incomprehensible.
Anecdotally, this is exactly how large consultancies, particularly those who specialize in serving the public sector, operate.
1) Deploy an army of "talented" salespeople to establish favorable executive-level relationships by exploiting every available loophole to provide trips, gifts, dinners and sporting events. Once the top has been won over, everyone below will fall in line.
2) Write contracts meeting the barest minimum interpretation of the requirements. Design them to specifically exploit uninformed procurement departments which are better equipped to source millions in copier paper or bolts than complex software.
3) Farm out any actual deliverable to the cheapest sub available nationwide. Ideally, you will have hidden travel and per diem for the talent in #2 leaving you free to fly in folks from the cheapest parts of the country.
4) Upon encountering any serious issue once work is underway simply scapegoat the current PM and shuffle in a new one.
5) Go to #1 employing the latest project as a reference that likely won't even be properly checked. In the event that diligence is performed, feed them to one of your friendly executives.
Seems the IT staff was out of their depth but there were some very serious and very fundamental issues if the users were confused as it sounds.
I say it as a joke but a barbed one: this guy thinks he was fit to manage a nation when he couldn't knock out a web app for 30000 users!?! Really?
I doubt Romney even knew the details of this mess. I'd be surprised if he knows it now.
As is typical of many "high level" exec's; Romney relied on the wrong people.
As a side note, an almost do-nothing president would be the best kind. (And, as a side note to the side note, that was what was intended when the office was created.)
This brings up a whole nother can of worms - electoral cyber warfare. What if the Dems, or some sympathizer group, say some local Anonymous, decided to disrup the GOP's centralized GOTV operation?
They could point LOIC at the IP receiving all the real-time field reports, set up hidden wireless jammers around Boston Garden, and any number of other hacks and exploits to disrupt this centralized system.
I use the GOP and Anonymous as an example here, but it could work in any direction. It's one thing for a campaign to lose due to its own incompetence and negligence, but quite another to lose due to outside interference and being the target of cyberwarefare.
But that's hard, and nobody's going to know anyway, so let's just roll with it.
DDOS defense: crashing
realistic traffic levels: more than the 3 ppl on the coding team
Intercepting field reports? By the sounds of it, Team Obama had all the info they needed and this just opens the door to serious legal problems.
In general I haven't encountered too many Republicans in software development. Libertarians, yes. Republicans, no.
Why am I asking... I am in from Czech Republic, where I can't imagine anyone knocking door over door and asking people to vote. Actually, this behavior is so connected here to Jehova's Witnesses (yeah) that it would have only the negative effect.
Similarly, I can't imagine any candidate seriously considering calling random people and persuading them to vote. Again, this behaviour is so connected to annoying salesperson (most often mobile operators) that it would hurt the candidate.
And I don't think our democracy suffers because we don't have that in here.
"Get out the vote" operations (GOTV) are real and sometimes effective in states where the outcome is expected to be close. In this election there were a small number of these "swing" states where both parties thought they had a chance to win.
In such states, if the early indications are that the voting is close, the party organization can redouble its efforts to contact known supporters (contributors, or merely people who are registered to that party in the public records, or people who have signed a petition or gotten on the party mailing list somehow) and persuade them to vote.
(This is because in the USA, unlike some countries, it is not mandatory to vote, and rarely do even half the eligible voters do so. So if you can persuade a person who was not planning to vote, to do so, that is a net gain of 1 vote.)
The purpose of this "Orca" system was to collate the reports from volunteers watching the polling places and identify precincts where a few extra votes might make a difference.
You can see how serious the effort was, in that the Republicans had mustered 37,000 volunteers willing to do this sort of thankless work: to stand around polling places, politely asking people how they had voted and phone in the counts; or, sit in an office and make telephone calls to people who do not want to be called and politely urge them to go and vote. That is a large effort, and apparently wasted by poor planning.
But the biggest difference is the per-state first pas the post mechanism, which is what creates the phenomenon of "swing states" where relatively few voter decide everything, and almost any expense of effort by campaigns is worthwhile.
So basically it is UK (less Northern Ireland), US, Canada, Australia and a few others. An Italian friend once said to me that if anyone knocked on his door asking how he voted he would stall them at the door, whilst trying to jump out the back window and get away over the hills...
http://www.slate.com/articles/news_and_politics/victory_lab/...
What the debacle brings to mind is the consequences of putting "idea guys" in charge of software development when there is a hard deadline.
"Orca had been conceived by two men—Romney's Director of Voter Contact Dan Centinello and the campaign's Political Director Rich Beeson."
I imagine them saying, "We shall have an app," while thinking, "How hard can it be?" While the developers were rationalizing, "Well, a web app is technically an app...and a lot easier to complete than an iPhone app, an Android app, and the web app we will still need to build for the old people with a Windows desktop."
Good thing for Democrats they didn't know about the open source ushahidi platform.
What I get from the article is a classic tale of crappy IT management, the like of which I've seen and heard of several times.
Also, from the article, project failed. Failure is regularly equated with having made mistakes.
No. No, it is not.
> Failure is regularly equated with having made mistakes.
And what mistake did Windows make here? Keep the baseless insults inside your head, please.
If you use or have used StackOverflow or Newegg, realize that you too are using Windows. Projects' success and failure is much more correlated with the folks running them than the technologies they use.
http://blog.stackoverflow.com/2008/09/what-was-stack-overflo... http://highscalability.com/blog/2011/3/3/stack-overflow-arch...
This was a meatspace failure, more than anything else.
I base my statement partly on that experience.
(and at Twitter when they fail whaled for months on end with Rails - it wasn't Rails fault - it was the application of Rails in too high a volume deployment)
You can host plenty of high volume things on Microsoft's code – Azure, Bing, etc. have proven scale. We can argue about cost, ease of use, etc. but a decent team should have no trouble making a serious service. The problem here is that they didn't hire a decent team – from the sounds of it, it was the usual executive-level schmoozing and managerial incomplete you see throughout large organizations of every type. They could have used HN'a favorite stack and failed just as hard.
I will not shut up about the ground truth I and many other people know about Microsoft and their software.
I'm guessing you wrote this post using autocompetence?
Yes.
Calling it "an app" however was a bad call. From a practical standpoint, that is. From the standpoint of "an idea guy" and a "communications expert" it was of course a requirement.
Of course, that web app needs to be working - but the choice itself is completely fine. Given the circumstances, I would seriously consider going the same direction if I had to do an app like that.
1.) 11 database servers. 1 app server 2.) Not redirecting http to https 3.) not stress tested or apparently tested in any way 4.) users got their first taste of the system the morning of the election.
wikipedia: Equivocation ("to call by the same name") is classified as both a formal and informal logical fallacy. It is the misleading use of a term with more than one meaning or sense
1. Developers overestimate their capabilities 2. Developers underestimate timelines
3. Client overestimates developer throughput 4. Client underestimates man hours required for any given feature.
5. Someone expects someone else to do QA. 6. Project manager assigns everything an ASAP priority
And this is just to start.
Snark postscriptum: Also, next time, GOP, you might wanna hire some IT guys who make more than $12 an hour.
In September they admitted to wholesale copying of a key Obama fundraising page, with Zac Moffat blaming the issue on "junior staff confusion":
http://www.buzzfeed.com/zekejmiller/romney-campaign-appears-...
The copy was lifted in its entirely. The similarity in the design is more subjective. (They were so similar that a lot of the discussion online speculated about a shared vendor -- which isn't the case. Quick Donate was a custom product internal to the Obama campaign.)
Romney campaign seems to have had serious problems in a lot of their internal processes when it came to tech/web. (AMERCIA)
Having access to data is one thing...having the logistics and middle managers who have good insights is the other. I think it's safe to say that no matter how much money and resources these operations have, it doesn't mean anything without campaign operatives who have a real IT/data background.
Lawsuit incoming in 3...2...1...
I can't imagine the billionaires who funded the campaign and the various Super PACs are going to be pleased to find out Romney likely lost because of a failed GOTV effort.
http://fivethirtyeight.blogs.nytimes.com/2012/11/08/as-natio...
https://docs.google.com/spreadsheet/ccc?key=0Au5WjgbQnYrSdFd...
Romney had 37,000 field volunteers, plus phone bank. I don't know how many were in each of the four tipping point states, but had each of them been able to bring in between 10 and 20 voters who didn't vote (of the 2 million extra who voted for McCain in 2008 but were absent this year), Romney could have won.
Instead it sounds like due to the massive Orca failure, they all gave up and went home.
(and fwiw, I say this as someone who voted for Obama. Not sour grapes here, just assessing the IT project failure.)
In Scotland we have 3rd generation systems (one system for all elections vertically, local government, Scottish Parliament, Westminster Parliament, European Parliament, and horizontally, 2011, 2008, 2007 back to the 80s).
The US parties be roughly considered as having 2 and half generation systems, so you might expect a difference at the top end of that.
I don't follow US elections that closely but you can work the numbers out.
Several dry runs and stress tests would have helped them alot.
That kind of thing might work in the soft field of persuasion, but is pretty useless in creating a complex technical system. (By know nothing-ism of the Republicans, I'm referring to the birtherism, beligerent rhetoric about complicated diplomatic issues, anti-science views about global warming, trying to paint Obama as a socialist, in general the whole "Guns, God, & Gays" type of messaging).
for those of you that dont know, Narwhal is an animal, but its also a internet meme. Obviously a very nerdy reference by Obamas team.
The fact that the Romney team had to name their software as a killer for a Narwhal goes to show that they wanted to win at any cost, unfortunately when that happens you sometimes lose sight of the goal, which in this case is to have a working system.
Please read the article before you downvote or comment.