Difficulty level in replicating this dataset from secretary of state rolls?
The problems listed below are pretty exact: huge data sets, lots of cleaning and normalizing, and the snail mail/cd problem is real. Additionally, I'd note that ~40% of the states [somehow] charge for the data...it takes six digits to get a snapshot of all 50 states - and certain states (looking at you FL) say that they do not store the historical, meaning you have to connect with the local BoE's to aggregate the data.
A part of me [now] wants to open source this because of the DNC's actions.
Was my thought exactly. Aggregate, then open.
It's not obvious to me how open access to this data would be intrinsicly bad, but I suppose that relies on the assumption that it's equally available to all parties (which may not (definitely not?)) be the case.
any pointers to the statuatory/regulatory guidelines would be appreciated.
I've chatted with a two different lawyers over here in Ohio...and both have advised strict security and election/campaign use only.
Even then, you only have a snapshot, because the states typically don't keep historical data. What this means is that your dataset won't be as good as someone who's been collecting this data for years, and thereby knows things you won't like where someone used to live, how often they voted there, who recently dropped off the registered voter rolls, etc.
In this case, even this data wouldn't be enough, because the Sanders team had made likely hundreds of thousands of contacts with voters, and recorded what issues they cared about and who they planned to vote for. This data, which they personally collected, is now inaccessible to them.
edit: expounded
And also means that Clinton's campaign had access to the Sanders' campaign data.
That is pure conjecture and I'm not even sure the Sanders campaign would agree with you considering they have fired a staffer over this.
It isn't clear if this is true.
"in fact they reported it"
VAN has not stated the issue was reported by the Sanders campaign. The claims that the Sanders campaign had reported an earlier issue are refuted in the OP, which states they had reported issues with another vendor's software.
It is possible the bug was abused:
"The database logs created by NGP VAN show that four accounts associated with the Sanders team took advantage of the Wednesday morning breach. Staffers conducted searches that would be especially advantageous to the campaign, including lists of its likeliest supporters in 10 early voting states, including Iowa and New Hampshire. Campaigns rent access to a master file of DNC voter information from the party, and update the files with their own data culled from field work and other investments. After one Sanders account gained access to the Clinton data, the audits show, that user began sharing permissions with other Sanders users. The staffers who secured access to the Clinton data included Uretsky and his deputy, Russell Drapkin. The two other usernames that viewed Clinton information were “talani" and "csmith_bernie," created by Uretsky's account after the breach began. The logs show that the Vermont senator’s team created at least 24 lists during the 40-minute breach, which started at 10:40 a.m., and saved those lists to their personal folders. The Sanders searches included New Hampshire lists related to likely voters, "HFA Turnout 60-100" and "HFA Support 50-100," that were conducted and saved by Uretsky. Drapkin's account searched for and saved lists including less likely Clinton voters, "HFA Support <30" in Iowa, and "HFA Turnout 30-70"' in New Hampshire. Despite audit logs, Weaver said at the news conference that NGP VAN has told the campaign that no Clinton data was printed or downloaded."
http://www.bloomberg.com/politics/articles/2015-12-18/sander...
The Sanders campaign had in October reported an unrelated software issue in a non-VAN system.
They did not report THIS issue to the DNC or NGP VAN, but claim they were gathering information about the breach for the purposes of reporting. Based on my reading of the OP, the breach was discovered by NGP VAN employees.
We are social creatures; we have a deep need to align ourselves with groups of others. And the evidence points to our having a deep need to argue as well.
Nationbuilder, a sorta-competitor to NGP, has put together a national voter file and it's reasonably priced. https://elections.nationbuilder.com/about/faq
The DNC/NGP voter file, however, is significantly enhanced - for one, it's got a lot of phone numbers, which most states don't include in their lists. There's a lot of other survey and consumer data associated.
For a national voter file that's good enough to use for say a City Council race anywhere in the country, where all you want to know is "who is likely to vote in this non-presidential election", it's doable but very expensive. For anything serious, it's pretty much outside of capabilities of anyone but the parties and some of the very big SuperPACs/very big orgs.