Mysterious startup Shadow under scrutiny after Iowa caucus meltdown
venturebeat.com
venturebeat.com
I worked as a Precinct Captain for one of the Caucuses in Iowa last night, and I didn't personally have any issues. But I heard from colleagues who definitely did. Also, yesterday we got an email with a 7-page google doc instructing us on how to reset the browser caches on our phones to avoid issues with data carry-over from the Mock Caucuses that we had been holding.
It's entirely possible that my ability to follow directions and my status as Not-an-Octegenarian-Caucus-Volunteer is all that kept me out of trouble.
As a software guy, all I can say for sure is that I'm overwhelmingly grateful that we kept paper backups so we don't have to rely on this software.
This is most likely the case. This is very much an issue in my day to day with applications like SAP CRM, some people get it, some people learn it, some never get it.
Four days ago when the register dug into this they were refused access to test the app[1], they were not given mockups or any information about the UI[1], they weren't told who developed the app[1], they weren't told how the contract was given out to develop the app[1], they weren't told why the app was thought to be necessary[1], and they were told the app was verified by a third party testing firm but that who that was couldn't be disclosed[1].
Iowa moving forward with this app was extremely ill-advised and I'm actually rather sad that the register didn't raise more hell on receiving all these red flags.
1. A red flag...
"As a [computer/sw/hw] guy, all I can say for sure is that I'm overwhelmingly grateful that we kept paper backups."
This is a bunch of amateur app coders, managed by amateur product designers and sales people, who got paid by clueless amateur voting registrars, to make an app that was rushed, not tested at scale, deployed without training, and allowed to get Iowa into this situation by someone not treating it with the seriousness it deserved.
No more, no less. Just normal human incompetence.
If you're just arguing that it's irrelevant because vote totals weren't hacked, fine. That's a dispute over word choice of what it means to "meddle." Russia also hacked into and downloaded data from voting registration systems, which should cause us to re-evaluate our voting systems, but again, that didn't impact vote computations or anything like that.
Anyways, I never said they didn’t do anything. I said, “here’s what I thought: x. Was I wrong?” My use of “(read: modify)” was to clarify that when I said meddle in that instance, I meant actual modification; I was referencing what the parent comment had said.
For those who did, I think what happens is that it's hard to tell which people are and are not operating in good faith and sometimes people mistake one for the other. But it sounds like you genuinely weren't sure whether Russia had actually hacked into the voting machines. Only into the Podesta, the DNC, and into voter registration systems. There's no evidence to suggest they changed tabulations (and by that I mean they almost certainly didn't), so that's why people haven't made that claim.
We have over 3000 different voting systems - one for each county. Some counties are staffed better than others. They use different types of machines. In some ways, that's a strength. But in others, it's a weakness. There should really be much more discussion of the issue nationally and what needs to be done.
Thankfully(?), Georgia (the state) is in the middle of some lawsuits (somewhat) regarding their lack of a paper trail in their elections.
Disclaimer: not an app developer!
That said, this was a pretty pathetic failure to scale, and I don't use harsh language like that lightly. I assume we're going to find out someone's brother or nephew runs the company because, for whatever reason, politics loves nepotism.
Like, that's "LAMP stack in 2001 hosting someone's blog about cacti wearing sweaters in Duluth" levels of scale.
It needs to record something like a dozen numbers from each of the ~1,700 caucuses in the state. You could have used Excel as the database here with no problem.
There is books written on how to test an new system/feature but in general you have 3 stages:
The first is an automated internal integrity test(sometimes called unit test) where you throw mock data at the systems independent component, in order to test for known classes of edge cases and race conditions.
The second step is implementation where the ops team tries to yank network and hardware resources away form underneath the app until something breaks in order to see if it fails in a sensible way that don't lead to irrecoverable data corruption.
The third stage is acceptance testing which on proprietary solutions is often the only one the end user is 110% in control of where you both test the app with actual user performing standard procedures and an team of specialist trying and break it using whatever knowledge they have on the systems design.
All tree stages have it's own skill set and is often performed by different teams and it's not uncommon to see specialist companies brought in doing part of the QA process for really important new systems but for the most part the entire things just fade into the the daily routine as an newer ending feedback loop.
But I have to agree with you. The owner of Shadow's parent company has close ties to the Hillary campaign, a lot of the Shadow employees worked for the Obama campaign, Hillary campaign, and for the DNC.
The scandal here is a bunch of noobs got an extremely important contract through cronyism, and they bungled it horribly.
As far as I am concerned they stole the nomination. The DNC was biased against Bernie Sanders and they ratf*ed him every chance they had. Bernie had staff at most of the caucus locations and they kept their own independent tally to ensure there was no funny business.
The bias was so obvious. It damaged Hillary's campaign. I was so disgusted I was tempted to vote for Trump ( I didn't! ), and I am pretty sure a LOT of people tuned out after the circus was over, and didn't bother to vote in the election. It was so obvious the head of the DNC had to step down in shame.
And not to say the shadow bungle was not damaging. It robbed the candidate who won from being able to deliver a victory speech. Instead... everyone delivered a victory speech just in case xD
https://www.thestreet.com/politics/acronym-shadow-iowa-caucu...
My suspicion: The company is all designers and product managers, and they shipped it overseas to the lowest bidder. That is unconfirmed, though.
This is related to the TRANSMISSION of the vote count to the Iowa Dem Party, not the count itself. There was a separate issue from app issues where the different rounds of voting weren't matching up in some precincts. Paper backups of the vote still exist.
I like what you did there.
The founder, Gerard Niemira and Krista Davis, have a great pedigree. Niemira was formerly at kiva.org. Davis was on the Clinton campaign tech team. These are not incompetent individuals.
Now, you may be right that this is all incompetence. However, your contention that these are 'amateur coders', managed by 'amateur product designers', getting paid by clueless 'amateur voting registrars', is completely ridiculous. The leadership team is clearly professional and tech-savvy..
I'm going to disagree with this premise.
And yet they failed miserably.
I'm so sick of this attitude. Leadership never takes the fall, it is always the underlings who get the blame.
Their important high visibility project failed. The leadership failed. When you fail, that means you did not perform in a competent way.
Like, if I sold someone a piece of software that I knew hadn't been tested and told them it worked great, that is malicious behavior, whether my motive was to make a quick and easy buck, to eat ice cream instead of doing actual work, or to ensure that my customer failed. The behavior is malicious independent of motivation.
I get your point though, I think we're just arguing semantics.
Probably one of the candidates who had a decent chance of winning the Iowa primary, but was not the most likely one to be picked. I don't like throwing accusations without concrete support for them, so I will just say that the whole involvement of Buttigieg's with the app, as well as his announcement of victory last night (way before votes got actually counted), seems at least suspicious.
Though it is just as likely as simple incompetence, probably even less, so i am leaning more towards incompetence, but I cannot fully dismiss the alternative in the light of all the recent DNC events.
Those of us who work in software understand that bugs happen even when you follow best practices. But if you don't even try to follow best practices for mission critical software, then roll out in production for the first time at a marquee event with millions watching, that's on you.
Exactly. And this kind of incompetence may be expected from someone who is actually an amateur good at selling themselves. However, the leadership is clearly not comprised of amateurs. Thus, this level of incompetence is professional, almost criminal, neglect.
Some folks take "fake it 'til you make it" to an art form, and they keep right on faking it even after they've made it.
A real-time vote transmission system that requires data integrity and validation is not quite the same as a consumer website, or an email marketing system.
That's not at all what this was or sounds like. This was a data collection app, from everything I've read. It is meant to replace a phone call.
"...Two people who work for Acronym, speaking on the condition of anonymity because they did not want to risk their jobs, acknowledged that the app had problems. It was so rushed, they said, that there was no time to get it approved by the Apple store. Had it been, it might have proved far easier for users to install.
Instead, the app had to be downloaded by bypassing a phone’s security settings, a complicated process for anyone unfamiliar with the intricacies of mobile operating systems, and especially hard for many of the older, less tech-savvy caucus chairs in Iowa.
The app also had to be installed using two-factor authentication and PIN passcodes. The information was included on worksheets given to volunteers at the Iowa precincts tallying the votes, but it added another layer of complication that appeared to hinder people..."
https://www.nytimes.com/2020/02/04/us/politics/iowa-caucus-s...
What the involved people are is highly connected which is rarely the same as competent as demonstrated by the antics of the current crop of in office politicians, and it's a huge part of the reason why crazy outsiders like Trump have a base at all.
The stupid idea that people who's only achievement is climbing the rungs of an failed organization is super cometent and whose ultimate failure could only have been due to nefarious actions of supernaturally competent foreign enemies is silly and seen as such by an huge sway of the actual public as the kind of nonsense you would expect from North Korea or Russia.
One can be a competent presidential candidate and still lose. That is the point of democracy. Clinton's campaign could be competent while still more people in the right places want Trump to be president.
Competent people fail all the time for reasons outside of their control but it's also true that incompetent people can rise to the top of an organization based on luck, connections or by being associated with someone else. So you need something more detailed then having been a part of an failed bureaucracy to justify claims of extreme competency in running software projects.
Yes, that is exactly the argument. Just because you happened to be a member of some noteworthy organizations does not mean you are competent. People drift from golden parachute to golden parachute all the time, especially at an executive level where you get to benefit from people assuming "well he's an executive so obviously he knows what he's doing" and blame your underlings for your failures.
Is this a joke?
There's nothing suggesting competence here and everything suggesting casual nepotism.
If we want to talk what things are generally true, sure, I'd put money on incompetence without any context, but there IS context to this.
As you pointed out, this is not just "normal human incompetence", these are layers and layers of incompetence.
I can almost see the name used in countries that have legitimate shadow governments, but in the US this is a really bad name akin to having cavalry in the name when focusing on the reservations (yes, someone did, they though it was cute and they did prove unhelpful). Names are a first impression and this is a really bad one.
This app had no reason to exist - there isn't a reason why they couldn't use almost any other approach to collect the results for publishing - there are very few numbers to keep track of and people are only looking for provisional numbers on election night anyways.
The process needs to be dumbed-down--kindergaten style. Give everone a jellybean to put in their candidate's jar, and if some dummy eats his jellybean instead of depositing it...
So yeah, getting rid of the caucus (which is undemocratic anyway) and letting people just vote for who they want is the solution. Fixing the app doesn't fix that people were too confused in certain precincts to count it correctly.
I think they'll likely go back to just transmitting via phone, though.
It's the good old pork barrel and an proud American tradition that overstayed it's welcome with the general public but remains a big part of America's political economy.
It's kind of part of the general back to the 90ies mentality of the DNC, yes there absolutely no need for an custom app to do this especially given it's a caucus and not a closed box election, and as there is no reason for keeping the raw data secret doing the counting process, an signed message on any platform(it could and should be public) would be sufficient security.
Heck telefaxes with an off the shelf OCR business grade solution at the other end would have been fast effective and likely cheaper but would have meant less opportunity for the Iowa party bosses to mingle with the well connected Washington apparatus.
This just smell of vanity project where a bunch of low power politicians felt their status would increase with an successful app launch only to have the whole thing blow up in their faces.
That's not social media advertising, that really smacks of straight up voter fraud and abuse to me.
I'm working with a vendor right now that works with Bloomberg, even though my org hasn't endorsed yet. (We wall that Bloomberg consultant off from our work. If we couldn't work with firms that worked with any of the candidates, then we couldn't hire ANYBODY.)
The votes are recorded on paper. This had a calculator tool (that actually functioned correctly) to count the delegates. The part the failed was the transmission of results to the Iowa Dem Party. The back-up (in addition to paper) was the phone hotline they normally use, but that got overloaded. Additionally, the overall caucus process for reporting -- unrelated to the tech -- was more convoluted this time around and the numbers didn't match for the different rounds of voting. This bigger problem caused the party to have to go back to the paper backups in order to verify results. That's what took so long.
I'd be shocked if anybody actually disputed any of the results. Every campaign keeps their own records of as many precincts as possible and they're announced to everybody attending. Plus, there's the paper. And people record video of these events on their phones these days, too.
In short, this wasn't a voting system, as you claim. It was a reporting system. What's the vendor they used for the phone reporting system? Are you upset that you don't know the names of the employees who run that phone hotline system? I'd be fine with a requirement for that for reporting systems, but nobody has ever asked for it previously for past elections so we shouldn't be shocked we don't have it now.
Also, when you said it had "links to Clinton," I took it as you had a concern about that. Glad we're both in agreement that it's actually quite normal for vendors that Dem Party would and should use. It's normal - encouraged, even! - for Democratic vendors to work with Democratic candidates.
> To make that process more efficient, the Iowa Democratic Party gave local managers the Shadow app to input results. But results were delayed and backup measures — such as calling a hotline — also failed, according to the New York Times.
From the article, I definitely read that as the app is the way that results were reported. I also read it as the phone and paper ballots were the backup systems.
The app absolutely is involved in voting, even if it's not the most authoritative resource, nor should it be. This was discussed last year (for example) in: https://www.desmoinesregister.com/story/news/politics/2019/0...
> What's the vendor they used for the phone reporting system?
Dunno, we should find out, and then find out why that wasn't correctly implemented either. Who's brilliant idea was it to accept election results over the same line which served the helpdesk?
> Are you upset that you don't know the names of the employees who run that phone hotline system?
Yes, you should be too.
That's my point. It's about how caucus managers tell the Iowa Democratic Party how many votes there were, for who, in each round. It has nothing to do with how people actually vote, which is done by walking into certain corners of the room and filling out pieces of paper.
I want a comprehensive, independent audit of all of the things that went wrong, done by an entity that every single campaign would have veto power over selecting. I don't care to know the names of the people who built the phone system, personally, since it'd be part of my recommended audit, but have no problem with a rule that makes them public moving forward because why not. I don't like the idea of making the lives of developers miserable when this seems like a problem at EVERY LEVEL -- at the Iowa Dem Party, DNC, individual campaigns that didn't object, owners of this company, and sure, the developers, too.