The City of Seattle Accidentally Gave Me 32M Emails for $40
mchap.io
mchap.io
Like, do people know that by emailing their local government their email address is now free for scammers to request under FOI? Could I request this data myself, then start emailing them scam emails "I know you contacted us in June, could you call me on 555-1223 etc"
This seems totally against the spirit of FOI
Data privacy laws in the US are unfortunately minimal. The bigger problem comes from imbalance -- if the government and corporations have lists of names, people need them to in order to be able to work together and organize.
If you don't think this information should go out in FOIA requests, the tool to accomplish that is data destruction. Government could wipe old emails once no longer relevant.
To meet federal requirements, information about procuring equipment with certain grants must be maintained for 10 years. Caseworker notes for a minor who is a ward of the state may be required to be kept for 20 years after the 18th birthday.
If a record is deemed in scope and topical, an employee could be committing a crime by deleting that email. As a result, the easy answer is retain.
Moreover, the requester is not required to give a reason for the request.
https://www.muckrock.com/place/united-states-of-america/wash...
https://www.rcfp.org/washington-open-government-guide/ii-exe...
Also, of course Seattle could reject the request, they could simply say: "Without an explicit court order to release this information we will not do so", and that would be that. It would then be upon the petitioner to ask the courts to force the release of the information requested, if the petitioner felt his rights had been violated. In the present situation the city is opening itself up to liability because of the privacy of all the people they have exposed (and more so because of the mistake). FOI does not mean 'every piece of data the government has should be released to the requester', the goal is increased transparency of government, not privacy violation of citizens using the FOI requests as an end-run around any kind of privacy law.
There is a tension between those two and typically the legal branch will determine where exactly the line is, when in doubt: go to court.
Because the FOI law exists, the city does not open itself to liability in releasing records, except when it accidentally releases records that are mandated to be private, which I'm not even sure is the situation here.
Increased transparency is almost always a tradeoff with privacy. I don't disagree with you that the law may be abused for commercial or malicious intent, but it is up to the legislature to propose a bill that curbs FOI. Until then, the government cannot just deny valid requests because they don't approve of the requester or the requester's purported motives.
That's the key bit right there. So, if you are not sure - and they are also not sure - then they could ask for a ruling before releasing. Err on the side of caution is good practice when it comes to releasing data.
I just looked at the dataset and it is full of information that I would normally consider to be private, which private citizens contact which government officials and when is in principle not something that should be disclosed to all callers in a format of their choosing.
What's to stop you from asking for stuff that infringes other people's privacy? I'm all for a more open government but 'anything goes' FOI requests are only a little bit less dangerous than non-transparency.
There is some middle ground to be found here.
> So, if you are not sure - and they are also not sure - then they could ask for a ruling before releasing. Err on the side of caution is good practice when it comes to releasing data.
Again, that is simply not how the law works. Some years ago, elected Washington state legislators and the governor decided the law should make these tradeoffs between transparency and privacy. And until subsequent legislators get together and decide otherwise, that is the law of the land. Washington government agencies do not have discretion to reject requests based on requester identity or motivation, period, nor can they make up their own reasons for exemptions.
The "middle ground" has already been decided -- that's ostensibly how the law got written and signed in the first place. Your line of argument would allow literally any government employee to make arbitrary rejections -- the law was codified to prevent exactly that situation.
Your concerns are no different than concerns raised about freedom of speech and the press (and of course, the right to bear arms, but let's not follow that tangent for now) -- e.g. "I'm all for people being able to express themselves, but what if those people say incredibly hurtful and damaging things?". The legislature can pass laws that limit those rights (e.g. defamation laws), and courts interpret whether those laws follow the Constitution, but it is not up to the executive branch (i.e. government agencies) to ignore the law because they disagree with it.
No, but they should decide based on the data requested. And in this case the data requested is none of the requesters business since it involves the privacy of other citizens.
Which definitely could be in contravention of other laws and in cases like that judges usually get to decide which weighs heavier. If I were a civil servant faced with a request that releases information that I felt would infringe on some other law I would definitely not decide to be the one to make the call and release it without a sign-off.
There isn't just one law at work here.
I agree with this -- of course a request can be rejected if it requests something that is explicitly exempted in the law. The metadata of emails to public agencies is currently not exempt from Washington state law.
> And in this case the data requested is none of the requesters business since it involves the privacy of other citizens.
OK, but that is not your or the state government's decision to make. The law does not allow for the government to make a unilateral judgment on whether something is "none of the requesters business" -- isn't it patently obvious how this could be abused?
> If I were a civil servant faced with a request that releases information that I felt would infringe on some other law I would definitely not decide to be the one to make the call and release it without a sign-off.
Sure, if you don't know the law exactly (most employees don't), then you consult your agency's FOI officer, who would then tell you whether the request is valid. If it is valid, and you decide to reject it anyway, you'd probably be fired (I don't think most state FOI laws provide criminal penalties for violating FOI).
I'm not a Washington historian, but I'm assuming the FOI law was passed because legislators had actual scenarios and use-cases in mind. For example, being able to request the emails sent and received by a government employee is useful if you want to know who contacted that employee about an issue, such as a regulatory enforcement action. Maybe there are clear-cut cases where a received email is obviously not work or issue-related, such as emails from that employee's mom. But what if the mom is herself a lobbyist or other influential official? Or how about an email from a guy talking about going golfing and getting a few beers? Is that just personal? What if the guy emails every week about going golfing on his dime, and the guy happens to be a businessperson waiting for regulatory approval on some project?
Apparently, the myriad of ways for unwanted behavior to be expressed via email are so plentiful that legislators decided to err on the side of transparency, because it would be too easy for officials to shut down requests and deny transparency all but to those with the means to sue (usually, corporations and journalists). It is not up to a civil servant to decide otherwise; likewise, the law protects the civil servant from liability for following a lawful request.
BTW I am not being hyperbolic here there was a case where this happened when I worked for BT - someone as a favor looked someone's new address up for a friend which resulted in a murder.
> someone as a favor looked someone's new address up for a friend which resulted in a murder.
A government employee who looks up someone's address for a friend is not covered by FOI laws. Just as FOI doesn't protect cops who use the DMV database to look up other cops they like/hate:
- https://www.wired.com/2012/11/payout-for-cop-database-abuse/
- https://www.sun-sentinel.com/local/broward/fl-pines-police-o...
Consider the example you brought up -- it is against the law for a state employee to send a friend that kind of information, and I imagine that that law exists because politicians feared that kind of murderer scenario. How exactly does that murderer use a cache of email metadata and redacted messages to go after his victim?
I have similar assumptions, but what about the less technically inclined citizens?
Moreover, I wouldn't be surprised if the ploy described by the OP:
> Could I request this data myself, then start emailing them scam emails "I know you contacted us in June, could you call me on 555-1223 etc"
would work on me. Really, it would take me checking DKIM and/or SPF to notice such an email. And from this story it seems likely the city of Seattle doesn't actually implement DKIM or SPF.
This is highly country specific. For the marriage record, I checked the laws in Germany, and (except for your own records) you have to present a "legal interest", which seems to be stricter than a "legitimate interest" (i.e. probably you need the information to enforce your rights, not just because you want to do genealogy). I'm pretty sure the others would count as particularly sensitive personal data too.
You can also not be on the role but you cant vote in that case.
That's because you're not corresponding with case officers and police officers.
When governments in the EU started digitizing their data like 20 years ago, it used to be that lots of personal data would end up published on official websites, either in the form of scanned PDFs that Google would gladly OCR and index, or in directly readable formats. Since then, the EU has cracked down on all of that, and you can no longer search for someone's phone number in order to get their full name, address, date of birth and ID. Even data that used to be available just 10 years ago, now has been removed.
(Ignoring that the GDPR doesn't even apply to governments)
The people of this state do not yield their sovereignty to the agencies that
serve them. The people, in delegating authority, do not give their public
servants the right to decide what is good for the people to know and what is
not good for them to know. The people insist on remaining informed so that
they may maintain control over the instruments that they have created. This
chapter shall be liberally construed and its exemptions narrowly construed
to promote this public policy and to assure that the public interest will be
fully protected. In the event of conflict between the provisions of this
chapter and any other act, the provisions of this chapter shall govern.
But then I implemented one on a busy mail system, and I started seeing credit card numbers very regularly. I mean, consistently. Even after people have been told why their emails are being bounced and no, I give an "ETA on when it will be fixed".
1) They don't realise email is not secure
2) When you explain point 1, all of the other solutions seem like too much hassle so they email anyway.
3) You can tell your customers not to email you CC numbers, you can even refuse them, but they will keep sending them
The Payment Card Industry Data Security Standard (PCI DSS) that you have to agree to follow in order to be allowed to process credit cards requires secure storage of all the cards you store--not just the cards that you intended to store or just cards that come in through channels you intended for receiving cards.
This is a serious issue to take into account when choosing your chat system, ticketing system, and email system because you can't just ignore those wayward credit cards. If you choose systems whose developers did not consider this and failed to provide good tools for finding and redacting unrequested, unwanted sensitive information you will end up having to hack such tools into the system yourself.
https://www.theverge.com/2015/2/10/8013531/jeb-bush-florida-...
As it turned out, the cache of thousands of emails contained things that are hard to anticipate. Including people emailing their SSN, or talking about their employment/medical issues; the former is possible to filter out computationally, the latter would require manual review and judgment. Bush was criticized, and in response, he took down the emails temporarily until employees could clean them up. But AFAIK, he didn't do anything illegal, because he mirrored exactly what was available from the state official archive which, again AFAIK, did not alter its copy.
This is similar to AOL's release of its search logs. It thought anonymizing user identities would provide anonymity, but they did not realize that some users write very personal things into the search box: https://en.wikipedia.org/wiki/AOL_search_data_leak
You fill out the form, how do you transmit it back to HR? You're probably going to email it. Very few companies have a "secure document transfer system" and very few people understand the risks of emailing personal information.
From a FOI POV, email is tough because it straddles a line between “record”, deliberative material and conversation. Everyone hates sharing email because it is always trouble.
In this case, they didn’t have a good process in place and nobody did a privilege or other review. The hint there is the police material — most police records are trivially made exempt from foi in most places.
But to your point, there are many categories of communication with government where there is literally no expectation of privacy. If you email the zoning board something, you should fully expect to see the entire email in a public record somewhere.
I worked at a polling company out of college owned by a Standford professor. My first task: After a poll is finished online, match that with voter records (using emails and addresses).
My first question was: "Well, that is a cool idea, but, there is no way the government would release a huge database of every california voter and their party affiliation. Let alone, the users entering in online poll information would extend that database to include their actual vote. There is no way this is possible.... right?"
Standford professor's response: "Do you want it in CSV?"
https://www.washingtonpost.com/news/monkey-cage/wp/2015/05/1...
This, and ‘because that’s how we’ve always done it’ are probably the main reasons party affiliation is part of voter registration.
As to "why" the government should feel obligated to subsidize the process -- because the government has the ostensible goal of facilitating fair and proper elections, and presumably the primary process -- which is not Constitutionally-enshrined -- is a net benefit to the general election, at least compared to selection-by-party-convention. In the future, political parties may decide that it's better to have open primaries, but that's orthogonal to the government providing the voting infrastructure and logistics.
Many include
1) Last time you voted
And
2) what your party affliction is
I think you are right in the UK. The US law is different and seems to allow this sort of broad request. I have been told before when filing in the US that others may request my contact information, and I have seen lists of FOIA requests received via the FOIA including contact information for requesters.
In the UK, requests need to be fairly narrow as I recall. And the time frames in which the request will be processed also are narrow: 20 business days as I recall. If it would take longer than that you probably would be asked to narrow the request. This is good for me as I typically request individual documents, not huge swarths of data. I requested a classified UK technical report and received a redacted copy within a month as I recall. Much faster than in the US.
Yes, bad actors can find malicious uses for a dataset. Not sure what that has to do with FOI. Are you suggesting that people who email the government expect their email to remain private, even as that email may be forwarded to a number of agencies and employees?
It is done but not legal and hard to prove.
[0] https://www.mcall.com/news/watchdog/mc-nws-open-records-busi...
Florida, which has fairly broad open records laws, at least makes this extremely clear:
http://www.leg.state.fl.us/Statutes/index.cfm?App_mode=Displ...
> Any agency, as defined in s. 119.011, or legislative entity that operates a website and uses electronic mail shall post the following statement in a conspicuous location on its website:
> Under Florida law, e-mail addresses are public records. If you do not want your e-mail address released in response to a public records request, do not send electronic mail to this entity. Instead, contact this office by phone or in writing.
The same exact thing happened. They thanked me and then their lawyers nicely asked me to clone my hard drive and sign a bunch of shit.
It was not fun at all. A lot of them thought that I hacked something.
How they reacted to your kind action is sad, and depressingly common. I hope you told them to pound sand, and contacted whoever the data protection authorities were in your state. There needs to be much more aggressive enforcement of HIPPA and similar data protection laws, CYA bull like you encountered should not be happening.
Article I ran across: https://info.townsendsecurity.com/bid/74330/Does-HIPAA-Requi...
People in the medical field, who deal with HIPAA all the time, pronounce it as if it was spelled HIPPA. It's a short step from there to actually spelling it HIPPA.
It feels to me like they shouldn't have much of a leg to stand on.
The laptop that I connected thier SSD to was my coding box, so I deleted all the code and secure erased the free space before they cloned it. They gave me shit about it too, because their forensic people saw days without any file activity. When I told them that it was because I removed my IP they responded with "why don't you think you need to do that? We can keep your data from falling into the wrong hands".
Maybe if I was some kind of an activist I would have tried to fight it.
You found an drive, tried to return it and then we’re subject to an illegal search? Or you consented to a search that would have otherwise been illegal?
Also, googled you. Here is a quote from your website: "I'm an american entrepreneur, inventor and __activist__" :)
One problem is that the metadata should have only contained anonymized entries for the email addresses of the counterparties of the Seattle.gov addresses, the article leaves this unclear.
Another potential problem is that if a case of corruption or nepotism is identified that has not been passed to the authorities for review that the author suddenly finds himself in the possession of data that can be used to blackmail some fairly powerful people, in fact there might be fish at a higher than city level government in the trawl because there have to be links between Seattle officials and state officials.
Yet another problem is that the addresses most likely contain the names of private individuals (including employees) as well, and I am not quite sure what to think of that but feel that the city has no business releasing that in cleartext.
A better way for amateur sleuths and the city government to work together to battle corruption would be to release only anonymized data to protect the identities of the people working for the city, for instance by releasing only hashes of the email addresses, for instance a hash@hash format where the hash for all Seatle domains is released to the requester. All the relevant analysis could still be done, and if something interesting was found it could be released to law enforcement who in turn should have then used a judge to order de-anonymization of those entries they are interested in.
The issue with hash@hash is that it is still possible to see whether a given person sent an email. Moreover, there are probably similar issues as with hashed_known_hosts as described in [1]. In short, the space of possible emails might be small enough to just brute-force search for all e-mails.
In that case you already have their email, and you know what hash and salt were used. It's game over at that point afaic, nothing will stop you from reversing all of the email addresses.
Even just seeing the graph laid out would allow you to infer who some of the players are. In general, to release such information on the assumption that it will be impossible to reverse it is irresponsible, and I would have loved for the city to recognize this and to get a judge to sign off on the release.
Agreed, hence the need for more than a plain hash. Note that technically, a 'salt' is unique per user and generally doesn't need to be kept secret. It really only applies to storing passwords.
What I suggested is more like a pepper [1], but in this use-case, you could use the same pepper for every address. Alternatively, you could just generate UUIDs for each address and publish those, but that requires a lookup in the UUID table for every e-mail. (Just like salted hashes would require a lookup to the salt for every e-mail).
" Where the salt only has to be long enough to be unique, a pepper has to be secure to remain secret (at least 112 bits is recommended by NIST), otherwise an attacker only needs one known entry to crack the pepper. "
If you use e.g. a 128 bit pepper, anyone trying to brute-force that based on a known email-hash combination would need to brute force 128 bits.
Further, the idea that sleuths (amateur seems pejorative in this case) are working with city governments to battle corruption does not hold up to Matt's (or other journalists) experience. By and large the governments only provide the data because they are required to, and we have made sure they are required to by representative legislative action.
Had the IT dept. in Seattle not made an obvious mistake this would likely have not been a story at all and the data would have been an interesting data set for informed democratic functions.
The request contains the names of private individuals through BCC and CC headers requested and exactly when they communicated with which government officials.
This is a normal expectation to the point where there are lots of rules about what public servants can even use private email addresses for. This was a big deal of course in the 2016 campaign.
This is likely a cultural difference, I view the data that Matt requested as mine because that governmental agent is acting in my name.
Yes, exactly this. History has shown that government employees and officials, being human, are reluctant to dig into, nevermind snitch about matters that may impugn their own colleagues. That a random citizen, or even a journalist, could convince law enforcement to go to a judge based on trends found via analysis of anonymized hashes is unrealistic. Nevermind that it effectively denies the benefits of transparency to anyone who isn't trained in data science.
Sender or Created by: "Herring, Kaya" <Kaya.Herring@seattle.gov>
Recipients in To line: Ortiz, Piper; Jones, Raphael
Recipients in Cc line: Valdez, Khloe
Recipients in Bcc line:
Sent: 3/23/17 18:08
More importantly, information that can be used to blackmail a public servant is IMO information that should be kept public. Blackmail is only useful if that information is kept private between a blackmailer and victim. Put it all on WikiLeaks and suddenly blackmail holds no weight because the blackmailer lost his leverage. If the information is of public interest and is severe enough that it is blackmail worthy, the entire public deserves to know it.
No thanks, that's an invitation to further corruption. Let me put that in context; recently a group of people were pushing a local authority to adjust its policy on the public release of arrestee's photos (aka mugshots). The group felt it was being selectively used for political purposes which the police and city council denied.
FOIA requests revealed that the authorities were unhappy with their public image as it related to law enforcement and decided to use mugshots as part of a social media campaign to change public perception. Can't have a social media campaign without any content, so people were arrested on nonsense charges that were subsequently dropped, so as to generate mugshots that could be publicized.
Thanks to the FOIA requests members of the public were able to confront the city council with specific names and dates of the people who drafted, approved, and implemented this policy, which information will also be central to future litigation on the topic.
And the best/brightest tech workers gravitate to the higher private salaries.
It’s not a technical problem - those are easier.
(OneBusAway is the name of the app that shows information to end users and the name of the API service that developers can query.)
I know that this service is widely available and free because I've had an API key for years and have (accidentally) pummeled the service with requests and have heard nary a peep from Sound Transit asking me for money.
Ever since the major overhaul of data in mid-2015 that tried to fix the "ghost bus" problem (a bus would show as being on the wrong trip pair) and the "never-ending terminal" problem (arrival times for stops very near a route's terminal would show as many minutes delayed or early until the bus actually started moving), I've been quite pleased with the quality of real-time data.
There are always going to be inaccuracies, though. At peak, Sound Transit / OneBusAway are trying to track a couple thousand vehicles across many miles in unpredictable traffic and weather. That it works at all is a minor miracle in my eyes.
In a similar vein, do you think the municipality wages are competitive with those from the tech companies?
I could see how being a tech-hub would function to draw a lot of talent out of a municipality.
A search for "(Fuck|Shit|Bitch)" can go a long way.
For what it's worth, I used to work at an investment bank spending 30hr/week diving through logs with unix tools, so finding interesting information quickly is something I've learned to do quickly.
At the first indication that the data was not what you requested and contained more than you - or they - bargained for you should have stopped looking at it and alerted both the sender and the relevant data protection authorities in so far as those are a functioning entity where you live to tell them they have an 'accidental disclosure' on their hands. Essentially your blog post documents something that is pretty strong proof you are not able to deal with confidential information properly.
I guess if all that is in those records I’m going to commend the Seattle IT department for their ethics at least.
It’s amazing what people think they can put in emails/messages and have stay secure...
One citizens communications with the city should not automatically result in disclosure of the fact that that person communicated with the city to other citizens.
The fact that a communication took place in itself is information, and correlated with things like timestamps and who in the city was contacted a large amount of sensitive information will leak out.
In this case, it took about a week for them to take it seriously. Like last time, it wasn't until I was explicit that they took it seriously.
There are a number of moving parts here:
- The disclosure was clearly not the intended result
- The recipient could - and in fact did - realize this
- The recipient was in contact with the sender
- The recipient had some easy means to redress the situation
Given all of the above, if you then dig in and start looking at the data I think you are crossing a line. At a minimum a legal professional should have been consulted before further examination of the data, once it became obvious something was wrong.
In the end it would have been down to a judge to decide whether that crosses the line in a criminal sense but I would be loathe to find out the hard way. Pick your battles and all that.
The idea that the OP is at fault for looking at data which the city had already published has no basis in law.
Having legally downloaded the data he looked at it. There's no crime there either.
This simply isn't true. If someone accidentally sends you information that you know you shouldn't be privy to, you should delete it. Unless perhaps you are Nelson Muntz.
From this I can accurately deduce that you really were talking to a person in Seattle. /local-in-joke.
Pretty amazing story; Seattle, collectively, always tends to mean well, but so often they stumble.
> So, I deleted the files.
Isn't it great to live in a country where we have generic felonies that governments can apply to just about anything involving a computer and ruin your life?
Land of the "free" and the home of the 'fraid.
Note: I am a Seattle resident and I expect nothing less.
The agency processing the FOIA request would get the export control status from the third party I contacted. The third party's software highlights export controlled documents with a red and highly noticeable statement. It would be difficult to believe they thought this was present when it was not.
The request also had actually been transferred several times because no one believes they have the authority to release the document I requested. The other agencies had ample opportunity to reject the request for being export controlled, but none did.
There are some other reasons that I will omit for brevity.
This makes me think that the export controlled claim was a lie meant to kill the request. Most people would stop at what they told me, but I thought it was worth verifying.
I can't remember how many times I'd point the person I was attempting to request the information from directly to their sample version or official space where they said they were the holders of said datas I was looking for which also provided the instructions to specifically call them to request the data. I know from my POV (as a tech worker) it may seem silly to expect anything like that from them but what I was asking for was akin to an excel spreadsheet full of information they absolutely do have and it required no legwork, no generation of new materials, no gathering of data from multiple ancient sources.
It was all material / datas each individual municipality was making money, hand over fist, every month of every year -- the sample datas most municipalities had was evidence of that. I was basically asking them for the collection of the entire data, publicly traded info, and most were convinced they didn't have it. A product of laziness is what I'd chalk it up to because the individuals and municipalities that were awesome to work with and more than helpful seemed to take pride in their work and getting said data was easier than the majority of things involving interfacing with the government at any level usually winds up being like.
I find it constructive in that once again it is demonstrated that the narrative that we're governed by rational, competent people and that we should trust and respect our government is very much a mirage, and the 180 turn in the tone of discussions suggests managerial malice running on top of the front line bureaucratic incompetence.
I'm completely disgusted and fed up with corruption.
I don't want to end up bone saw murdered, and it feels more likely every day.
The alternative is that the email address is correct and Matt is redirecting his domain to another Matt Chapman, which would be totally hilarious.
> Funny enough, in the middle of that question, my internet died and interrupted the call for the first time in the six months I lived in that house. Odd. It came back ten minutes later, and I dialed back into the conference line, but the mood of the call pretty much 180’d.
I find that when strange things happen like this, they’re hardly coincidence. Did you run a traceroute after the disconnect anywhere? Did you see an IP address change? If so, was it a significant change in the CIDR block it was within?
Also, once they realized they had left the room of course they would continue to discuss the case and it is obvious they had to consider all possibilities, including the recipient releasing the information to others, hence the 180.
If true, that would be a far bigger news item than the rest of your story.
My residential connection here is pretty good, even so it goes up and down at least once every week or so whenever some firmware update is pushed to the router.
I work from home, so my internet going down is a big deal for my livelihood and all that. I'm not saying that something suspicious happened, but I figured it was an interesting thing to happen. You're frankly thinking into it too much.
I think that was my line.
You're saying the natural default behavior is to assume the worst about someone and draw a conclusion in their absence, as opposed to suspending discussion briefly while trying to get the person back on the phone? That seems like a very bad-faith approach to negotiation or discussion, given that the legal liabilities are something that were so easy to identify in advance.
And here in my country, I needed a court order to get at least an acknowledgement of my FOI request.
And now I'm petition court intervention to get the FOI processed in accordance with the law.
There have been a huge number of papers using this dataset and there are not many other datasets of its type or size available and despite its age is one of the best we have. If people are aware of legally released datasets with a similar size and content I would be interested to hear about them.
https://mchap.io/a-tale-about-requesting-chicagos-mayors-off...
>Seattle's first response included a bit of gobsmackery that I’ve almost become used to
Brit here. I'm always amused that 'gobsmack' and its derived words are still used these days, more so across the Atlantic.
Roughly translated: lost for words, typically for a short time.
Excel can only handle about 1 million rows, right?
Instead of reporting data breaches turn it into a torrent and make it public.
There is a surprisingly little spam. Either that is about to change, or spam didn't get included in the FOIA.
Consider being on the other side of this, due to a careless mistake the data for many people is exposed on a random strangers hard drive. Asking for an independent third party verification is reasonable.
Bringing lawyers in the mix was also unnecessary. And if more people follow in the authors actions then the state level FOIA laws may be put at risk over the long term.
Based on timing of emails this would leak the list of people working on the case, maybe informants and put them at risk.
We all lost our collective shit when NSA said they're only collecting metadata. Metadata is Data.
IT estimated the requestor fees to be $21k/year assuming 10TB of data.
Clearly, IT were estimating the costs of releasing all email text, because FOIA mistakenly changed the words "please provide the following information:" to "including metadata:"
Once they figured out the request was for header information only, the city came back with an estimate of under $60.
And in many cases metadata is just as useful as the payload, in some cases even more useful.
They don't have a choice. Seattle IT is so underfunded that hands are tied because there isn't any resourcing.
On one hand, you have to respond to all of these requests (and rightfully so, as it's the law.) On the other, you have no money for your department because it has no funding because the citizens didn't want to spend the money.
The person who did this isn't malicious. Just very overworked and did a data pull wrong. They probably didn't give a shit to check because their job kinda sucks and they have too much to do already.
City Light and the new meters/new billing system are great examples, all the new power meters have no encryption, and use FSK for modulation. Asking City Light about this got me a response that FSK was the encryption, and the gal was dumbfounded when I pointed her to the Wikipedia article on FSK.
On the billing side, an Oracle salesman ran off with over $100 million in city funds for what is essentially a CRUD app, and the worst part is they didn't bother to customize this system, just forklifting this in place and letting the chips fall where they may. The prior billing system had quite a bit of data validation and business logic that has yet to be implemented or replicated on this new system. The same actions when you call customer service now take significantly longer.
Both these vendors fleeced the city for broken, insecure systems, and neither is having to face the music for it. Worst part is, eventually someone may attempt a fairly trivial exploit of either system, which could wreak havoc in our city.
Edit: though I shall add that it unfortunately is sometimes not possible to build a new culture without firing incumbents. That one is a really unfortunate situation. It's not the fault of low-skilled IT workers if they are enabled and rewarded for poor skills and attitudes. But if an organization wants to make good cultures, it sometimes requires hard decisions. Still don't blame the workers, just like I don't blame factory workers that get automated. It's just an unfortunate thing that can happen that most people don't deserve.
If I write a piece of software which is technically capable of meeting its requirements if you read the manual carefully enough, but in practice the intended users can't figure out how to do so, that piece of software is no good.
Similarly if the market is in principle providing IT vendors who are capable of providing a decent service, but in practice the purchasers can't figure out which ones they are or how to make them do so, the market has failed.
Edit: and let me say that I posited that the issue isn't market forces. The issue is lack of expertise at decision making levels. Even if there were zero market, it wouldn't stop people from doing it wrong.
Just as you have to build software for the users you have not the users you feel you deserve, we need a service industry that works for the service-commissioning agents we have.
https://en.m.wikipedia.org/wiki/Frequency-shift_keying
> The demodulation of a binary FSK signal can be done using the Goertzel algorithm very efficiently, even on low-power microcontrollers.
The source links to the TI page for the MSP430, named because it originally sold for $4.30. While the original link is dead, an internet search reveals many side and college projects using this technique.
How much of this is voters voting for politicians who are good at what is essentially a popularity contest instead of politicians who are capable of signing decent contracts? As long as voters don't care enough to change their voting habits this will continue to happen.
I feel like they should have offered to compensate the author for his time in their initial request - if someone wanted to perform forensic scans on my hard drives it would be a huge inconvenience.
Worst case, they'd want to know exactly what data was at risk, which would trigger more demands for forensic audits and more duty to notify.
Allow a third-party forensics company hired and beholden to a presumed-hostile counterparty unfettered access to your hard drives because of their own lack of care or incompetence? Hell no!
You are asking for leniency on the side of the officials, yet do not seem to be willing to apply the same standard to the requester. Yes, it could have been handled better, but that applies equally to both sides. Anybody that does FOI requests that have the potential to retrieve a lot of sensitive data due to mis-understandings or mistakes (which is pretty much all of them) should handle the data carefully until they have verified upon receipt that it is what it should be and that the data is not somehow more sensitive than intended.
The author did ok in that respect, could have still done better and the city would have been served better by refusing the request as stated until order by a judge to release it based on the grounds that it is an overbroad request, which will result in the release of privacy sensitive information if fulfilled.
Not really, for the same reason they never should have sent the excess data in the first place... Why should he give up his privacy to some 3rd party company to help cover up their mistake?
I do want to recognize that not all local governments behave the way described. When I was recently called for jury duty, I discovered a vulnerability in the city's jury duty online portal that would've let anyone get the PII of anyone ever called for jury duty via that system. I immediately called the county IT department, and they took me very seriously and thanked me for the report. They later emailed me back to tell me they worked with the vendor to close the vulnerability. I was extremely impressed with their professionalism and wish that all local governments could be so responsive.
Asking for it may be reasonable, demanding it certainly isn't. You can't demand that somebody gives a third party access to their hard drive.
First, it's completely unreasonable from a privacy aspect. Given the level of personal data most people store or process on their computers, it is even less reasonable than asking someone to have a third party dig through their house to "verify" that they don't have something.
Second, it's completely unreasonable because it's pointless against a malicious actor - they could have cloned the disk before deleting the data and there would be no way to detect it.
It does make sense only to confirm that the data wasn't accidentally left on the disk due to an insecure erasure method. When dealing with someone competent with computers, the proper solution (which they ultimately arrived at) is to have him describe the method he used to delete it, possibly ask him to verify it (e.g. via some form of "dd | grep"), and that's it.
In fact, a known security check was actually bypassed in this case: the email review, reserved for the content of the email, causing the whole problem in the first place.
It seems to me imperative that they actually deliver up to the amount authorized. Ideally exactly the amount, but never more.
He requested metadata and they sent actual email content, kind of a big difference there.
I would think it’s pretty cool to retrieve this massive amount of data that I wouldn’t otherwise get to play with.
What do you mean "people that know way more"? He made a simple request for email metadata, spelled out each field he was interested in. He didn't tell the city how to do it.
Are you saying that the author knew more about how to retrieve email metadata than the actual Seattle IT staff that administer the mail system? And what bothers you most is that the author knew more about how to fulfill his request than the people that run the mail system?
Wrong. Dumping a vast pile of irrelevant information at the last possible moment to obscure something embarrassing is a very common tactic in commercial litigation.
Source: 100s of FOIA requests to various WA government agencies.
https://www.rcfp.org/browse-media-law-resources/news/city-mu...
It's because Washington agencies are required to cover reasonable attorneys fees for their opponents after losing open records lawsuits (one of the factors in our FOIA laws)
So when Author sent the request to Seattle, they have this above cited example (and 100s of others across the the State) where a mistake could create a lawsuit the costs this loads of money.
Did you know that the burden is on the agency to establish that its denial of inspection is proper?
Did you know that the court could award you an amount between $5 and $100 a day for each day that access to the records was denied.
So, if they don't give you everything you ask for you can sue. And it's easy (relatively) to win in WA for that because of our FOIA laws, then the agency has to pay for the lawyers and a penalty for delay of the records. For emails that means $5 * (Days of Delay) * (Number of Records).
In short, if Seattle fucked up this FOIA request, denied or delayed -- that could have cost them millions of dollars.
The author didn't understand that and (like a fool) blames the city and city-workers.
Sorry, but that sounds like bullshit. The Washington law provides for agencies to take reasonable time on a request, especially one a request that is complicated and broad. In fact, unlike the FOI law for federal and other states, the Washington law does not proscribe the number of days that an agency must respond by, only that they be made "promptly":
http://app.leg.wa.gov/RCW/default.aspx?cite=42.56.520
The city of Shoreline did not have to pay out $500K "because of mistakes they made on a FOIA request", not according to what you posted:
> The City of Shoreline will have to reimburse $438,555 to cover the plaintiffs' costs as Washington agencies are required to cover reasonable attorneys fees for their opponents after losing open records lawsuits. Shoreline also agreed last year to pay a $100,000 statutory penalty after the court found that the city violated the state public records act.
They paid $438K for fighting the request for seven years. They paid an additional $100K penalty because they were have found to violated the law. They did not pay for "mistakes", at least not mistakes in good faith.
Another example in WA, the kind of thing that scares public sector employees.
I have direct personal experience with this, and direct knowledge of others receiving payout from government (in WA) for similar violations (almost had to start another suit this month).
Your narrow interpretation is splitting hairs. The danger is real to the government workers executing these requests
Edit: two more easy to find examples
https://www.aclu.org/news/judge-fines-tacoma-police-departme...
www.spokesman.com/stories/2016/mar/24/justices-uphold-502000-public-records-fine-against/?amp-content=amp
In your original comment [0], you suggested that employees feared of making innocent mistakes that would lead to open records lawsuits. None of the examples you've provided describe that situation. Instead, they involve agencies (and their lawyers) who have decided to refuse a request and fight it out in the courts. What does that have to do with being a danger to employees who handle these requests?
This means that these employees frequently send more, faster to avoid a big public issue.
The point is that individual people feel pressure and make decisions based on these articles (and many not so public cases) that in retrospect are not that great.
And then some blogger makes foolish claims, as if it was incompetence rather than fear.
I'll not comment further
> The point is that individual people feel pressure and make decisions based on these articles
That is literally the situation of every public servant -- as just about any police officer will tell you. The difference with FOI is that the law provides ample protection for government employees to take their time to get it right, and every investigative journalist I've ever worked with puts up with those delays -- it's only when the delay goes into months/years such that it's tantamount to a rejection that legal action is threatened, because the lawsuit itself takes months to resolve.
The only example lawsuits you've found were ones in which the agencies refused to fulfill the request. Until you can show a single instance in which a state employee, or even an agency, was punished because they were late while trying to respond in good faith, I don't think we should assume you know what you're talking about when you claim the author has "no background/understanding" of WA's public records laws.
It's especially absurd that you're trying to argue that the mistake the city of Seattle made in his case was done out of hurried fear, when the author provides correspondence that shows he and the city emailed back-and-forth from April to August before they sent him the data. A technical screwup (via internal miscommunication) is the most plausible explanation by far, as no one in the IT or FOI office had any reason to rush this request.