Congress votes to make open government data the default in the United States
e-pluribusunum.org
e-pluribusunum.org
Is this an open source project? Or what’s the way for licensing to use with US data?
Of course, there are exceptions: the PDFs that are often provided by the prosecution as part of the discovery process are prohibitively difficult to deal with, and should be considered a violation of Brady vs. Maryland, IMO.
In regards to modern day transparency requirements, it seems like laws should include a reasonableness clause.
Making records available to the public but requiring them to be hand photocopied vs. making them available in electronic form in a custom format.
Both open. But two very different magnitudes of effort.
https://aws.amazon.com/about-aws/whats-new/2018/11/introduci...
The rest kind of it kind of just comes down to using good software engineering practices to help keep yourself sane. Find useful abstractions for common tasks you need to perform and build a library around them, make sure that your data processing pipeline is designed with enough flexibility to handle inputs in different formats so that adding or modifying parsing logic becomes trivial, etc.
[0] https://www.xpdfreader.com/pdftotext-man.html [1] https://pymupdf.readthedocs.io/en/latest/
- Page 44: "Our economy has grown at a steady rate despite changes in economic policy". I expected to see a lot more fluctuation on this data.
- Page 30: "There have been more suicide gun deaths than homicide gun deaths every year since 1981". This is crazy to me given how much we hear about gun homicide being a problem in this country.
- Page 29: Crime rate has declined but "The number of incarcerated persons has increased by 330% since 1980".
[1] https://static.usafacts.org/public/resources/USAFactsReport2...
Barring some calamity, this is pretty much as expected. Tariffs impact a very small percentage of the economy with large size, and the rest of it with a small overhead, much like a fed interest rate hike. Even with this hawkish fed, there hasn't been anything overly harmful to the economy from a policy perspective.
> This is crazy to me given how much we hear about gun homicide being a problem in this country.
Anything that's politicized gets this special treatment. Kid kills brother with car, blurb in local newspaper. Kid kills brother with gun, national news and Tweets from Presidential candidates.
Texting and driving is as bad as drinking and driving when it comes to number of deaths (dwarfing gun deaths as well), yet people do it like it's no big deal. Most cities still only levy small fines (in comparisons to DUIs) for doing it. Not all accidental death is created equal.
>Page 29: Crime rate has declined but "The number of incarcerated persons has increased by 330% since 1980"
Welcome to the US where the war on drugs gives us authoritarian level incarceration rates.
Every election cycle, I continue to be disppointed. Even if they were crazy in every other regard, I would probably still vote for them. It’s like, Step 1 towards doing anything meaningful in regards to poverty, education, criminal justice reform, et cetera.
https://github.com/gsa/datagov-deploy
https://github.com/ViderumGlobal/datagov-ckan-cloud-provisio...
True, I guess HGTG applies well here:
“But the plans were on display…” “On display? I eventually had to go down to the cellar to find them.” “That’s the display department.” “With a flashlight.” “Ah, well, the lights had probably gone.” “So had the stairs.” “But look, you found the notice, didn’t you?” “Yes,” said Arthur, “yes I did. It was on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard.”
I think this was one of my biggest shocks doing work for the government, collecting public data, payed by tax funded grants. Public data isn't for the public.
We went into this project with all these starry eyed dreams of making a public online database and freely posting everything we collected, with maps and interactive tools, status reports. It was part of our grant proposal.
Then reality came and we found out public data meant a government password protected database with access fees where our data would be available to people willing to pay for it or we'd lose our funding. The data were for companies or individuals willing to pay the government not for the public.
This still doesn't sit well with me nearly 6 years later. That was never what we wanted out of that project and it wasn't what was planned or accepted when we wrote our proposal.
Ideally, I'd prefer the data be free, but if the fee was (mostly) nominal, I'd consider that almost as good...
Out of curiosity, what’s the argument here for how public data being open by default is an important step toward fostering societal equity?
https://www.scribd.com/doc/263776138/The-Social-Impact-of-Op...
It's a link to a site that wants you to agree to try a monthly subscription before you can download anything at all.
Especially in the context of a discussion about public data, that's an important distinction.
If you can, please provide an URL to the actual content?
Having said that, here's a link to the version hosted on the Sunlight Foundation's domain: http://assets.sunlightfoundation.com.s3.amazonaws.com/policy...
You can also verify what I said about the Scribd version by checking out the original press release/announcement here: https://sunlightfoundation.com/2015/05/05/a-new-approach-to-...
Make the raw data available for those of use who want to write machine-parsing algorithms, and also make it available in human-readable and easily digestible form for the broader public.
For instance, consider the statistics on accidents between cars and bikes in California. You can get the numbers yourself from the government. When someone says that more accidents are adjudged to be because the bicyclist is at fault, you can reference the truth and ruin the credibility of the person making the assertion, thereby allowing political advancement of your own cause. No one can use the technique against you because you are capable of acquiring the knowledge and won't make wrong assertions. Only other people will make them.
Having verifiable true information over someone is power. It's better non-democratized so long as I fall within the circle of power.
The more data everyone can use, rather than data that can be owned and commoditized or utilized only by specialists is a good thing.
1. public information should be open by default to the public in a machine-readable format, where such publication doesn’t harm privacy or security
I'm sure literally everything that they wish to keep opaque will declared to be covered under one or both of these incredibly vague categories and nothing will significantly change. Is there any elaboration in the bill that defines what they can call a matter of privacy or security? Even if there is, it wouldn't matter much because how are people going to tell if they keep it locked down in the first place? And they would not risk any sort of real blowback for abusing this and getting caught. Tell me there have not been far, far worse scandals that resulted in no consequences for the perps and cowed silence from the public. I don't think they're hiding the X-Files in there or anything, but this won't magically cause a more transparent, just, or equitable government unless it has serious teeth and tight language.
And 2. federal agencies should use evidence when they make public policy
Somehow I wonder if the data from the Kansas experiment will be taken into consideration and turned into public policy by this current administration, or if they will cherry-pick evidence selectively to justify only wildly unpopular legislation because someone (possibly an industry with a conflict of interest?) contrives some p-hacked research to back it up. Just because something is scientific doesn't necessarily mean it's good government. It is often so, but I'm always very wary when they trot out a bill with lots of bold language touting justice and democracy, truth, stuff like that. If the US legislature passes a bill called "protect innocent puppies from being kicked in the name of god and freedom" you can be 95% sure that this bill will enable a great wave of puppy-kicking despite its holy name.
Define equity here?
>Legislative definitions of a federal agency are varied, and even contradictory, and the official United States Government Manual offers no definition. While the Administrative Procedure Act definition of "agency" applies to most executive branch agencies, Congress may define an agency however it chooses in enabling legislation, and subsequent litigation, often involving the Freedom of Information Act and the Government in the Sunshine Act. These further cloud attempts to enumerate a list of agencies.
Let me know what you can't find or request via https://www.data.gov/data-request/
> By changing law A related to B, we expect the increase of C to be at least D in the next E months.
If not achieved, the change is reversed/reduced. Hopefully that would allow experimentation without taking the risk of creating a system too bad in case the implementation or policy isn’t good enough.
Just dreaming here :)
Since every law is a trade-off, having both positive and negative effects, I wish each law would enumerate the expected/possible positive and negative effects. In other words, I want the trade-off to be explicit and for the lawmakers to express why they believe the positive effects outweigh the negatives ones.
Some personal favorites among BigQuery public datasets include NOAA GHCN[0], the Census Bureau's Zip Code Tabulation Area [1], and FEC Campaign Finance [2].
[0] https://console.cloud.google.com/marketplace/details/noaa-pu... [1] https://console.cloud.google.com/marketplace/details/bigquer... [2] https://console.cloud.google.com/marketplace/details/bigquer...
I wish I could do more business with my gov't (State and Local) through the internet.
"Copies of documents cannot be ordered through this website, by email or over the telephone." Only fax and snail mail... https://www.dos.ny.gov/corps/faq_copies.page.asp
As long as the data is accessible and reasonable easy to get, where journalists and data scientists who really care about the data can get it, then I think we're in a good place.
Having a data review process with automated integrity, confidentiality, and quality checks is not terribly difficult.
But having a prototocol to export the pdf to csv is also dead easy for confirming only the data relevant is included. ASCII is just as “easy” as scan, but it requires training clerks to be data-oriented rather than document.
All I want for Christmas: accountability for the DoD budget, which for 2019 will be $717 Billion, the majority of the USG's discretionary spending.
In 2015, an audit of the budget revealed $125 Billion in wasteful spending, and this was covered up. In 2016, the Office of Inspector General for DoD said that the Army made 6.5 Trillion in wrongful adjustments to its 2015 accounting. https://en.wikipedia.org/wiki/United_States_Department_of_De...
We still do not know how many contractors the DOD employs, or how much money they are paid, because the numbers are just not recorded. We do know, though, that the numbers and budgets we do have are often inaccurately reported, according to the DOD OIG.
People whine a lot about paying taxes, but the politicians that always complain about taxes are extracting record amounts of tax money for a military that is mismanaged, doesn't do its accounting properly, can't build modern fighting vehicles, and doesn't record basic information like how many people they employ.
If you're interested in informing the public on legislation, have experience on the hill or UI/UX experience hit me up. I'm just some dude with an idea who lives in a terminal. Money is secondary.
But I'll be the first to admit I know very little about how the Federal Reserve works.
It's almost like they want to cover up the systemic disempowerment of the American electorate.
Despite all that, the Federal Reserve is not a government institution, so such legislation would be similar to singling out your business to post all information in machine readable format online.
The biggest problem with this terrible binary format is that metadata can leak a great deal that should have not been released. So this leads to PDF output of word/excel.
The next area is that especially local government offices have no way of setting up a data portal. I'm working on this right now, where the only way to get data out of Bloomington,IN is to do FOIA requests every week/month over the data you want. This absolutely should be available via a portal, and not locked behind "in person, mail, fax at cost of .10$ a page".
public information should be open by default to the public in a machine-readable format, where such publication doesn’t harm privacy or security federal agencies should use evidence when they make public policy
The word “should” is used in both one and two. If my time in the government taught me anything, it’s that “should” is only slightly stronger than “may”. If an instruction says “shall”, then it is required to be done.
Please, could we end this obsession with backronyms in congress. Perhaps some congressperson could create a suitably titled backronym act to rid us of backronym acts. This should just be the one singular "Government Data Act" and it should be amended as needed to cover any law changes around "Government Data".
(struggled with the NYM so gave up and went with something childish...)
There is no way around it that these companies have to work with the government to secure public interests from 21st century threats. Just today I read an article about how black hat hackers are targeting outdated industrial control systems more vigorously than ever before. The government on its own without technical upgrades cannot face down this problem in its current condition. Which is why opening up data is a beneficial thing.
Openness of data is a double edged sword. It will make malicious agents' job easier to have as much data as possible in a consistently machine readable format, but it will also help those on the other side.
If tech is one of the things that can bolster and improve government, tech needs to work in the optimal environment. Which is one with open data.
Open data will bring innovation and accountability.
Great move
the cynic in me just figures that this moves the goal post such that special interest groups will adapt to produce the right evidence for the desired outcomes
I would rather have bogus evidence in the official record, that I can analyze and challenge, than no evidence at all.
There is no "innovation" going on here. It is the data owned by the citizens, doesn't take that much effort to release it - they are collecting the data anyway, all they have to do is make it available to the general public. Still, a step in the right direction
US government has much more room for improvement (because of its complex structure (federal vs state agencies, etc), large population and so on) - This is very different from pretty much the rest of the world.
I am saying by opening data, us will be fostering innovation.
Does that mean they won't open data that affects monetary policy or that they will only open data that affects monetary policy?
Either way that seems huge.
The submitted post includes a link [0] to an article he wrote a couple days back, which provides more context on "How did open government data get into the US Code?", including the nitty-gritty of how the original bill was proposed in the last session but ultimately left out of legislation. Howard writes that the legislation was "one of the primary legislative priorities for me during my years as a senior analyst and then deputy director at the Sunlight Foundation"
[0] https://e-pluribusunum.org/2018/12/20/senate-passes-evidence...
Full disclosure? Odd brag?
Deleted comment
It makes legislation easier to access. And eventually easier to engage with.
Basically, the implied subtext is that regulators should not put any restrictions on industry unless the evidence is completely unambiguously in favor of the regulation. However, science is never 100% certain even at its best. And you can always drag up some study that contradicts the strong consensus of the field, whether by fluke or intentional design of the study. In other words "evidence based policymaking" has been euphemism for "must give alternative facts equal weight".
net worth + annual earnings+ board positions
of US Senator/congressman spouses+children ?
The bill had large majority bipartisan support (which makes me suspect it's toothless...)
I'm surprised this needs explaining.
What would this mean in practice?
Take Khashoggi as a contemporary example. Undoubtedly there have been countless heavily classified conversations weighting the pros and cons of any action against Saudi Arabia or MBS as a result of this. How we are extremely dependent on Saudi Arabia for reasons outside the scope of this post, and they know this. And this is all happening at a time when Saudi:US relations souring would greatly stand to strengthen the geopolitical position and power of nations such as China and Russia. In my opinion it's extremely likely we will do nothing, but that's because it's better than doing something. If this was not classified, social media would throw a nonstop hissy fit. Not because they actually care or think we're making the wrong decision, but because it's an incredibly easy way score those imaginary points and followers which are the hottest commodity since sliced bread. Another issue here is that knowing the mechanisms of our decision making here, and how close we came to 'breaking', would be incredibly valuable information to nations such as Saudi Arabia which they could then use to exploit the US.
So while I do agree with you that classification is very often abused, I also think there are indeed vast amounts of information that must be classified. People are not mature enough to impartially process our decision making processes, and the details of such processes would provide invaluable information to other nations which could then be used to exploit the US.
That can lead to money, but doesn't have to. Wielding power is its own ability.
Look no further than HN. Power is granted by how many "imaginary points" you have. Downvote, flag, vouch, and I'm sure more - there's power wrapped up in imaginary points.
And I'm sure someone would also argue that money is also imaginary points, that everyone accepts as 'monetary power's. Of course, this is orthogonal to 'karma', 'likes', or other names of the same thing.
I’m sorry but speak for yourself, not me. This is a democracy. Don’t tell me I can’t know about things because I want to score imaginary internet points.
Democracy and secrecy are incompatible. No man is above the law, and laws are not secret.
“The masses can’t be trusted” is nothing but the talk of a despot.
For this specific case, a loud and public debate over our country's disastrous relationship with the Saudi royal family is long overdue. Embarrassment, and unseemly noise are a pretty low price to pay to do the right thing and save more lives in, for instance, Yemen.
Our elected officials and government employees don't suddenly gain insight they did not have as common citizens. They just have access to more information.
These people, the ones who are part of the security apparatus, are not in any position to make moral choices for the rest of us. They are highly immoral actors. I want to see them removed from government. The idea that they are capable of making good decisions in secret, without public input or oversight, is belied by the two decades of war, torture, assassination, and general chaos that they have actively fostered.
The problem, really, is that all of the secret policy conversations about Saudi Arabia have been about how to get them to continue buying weapons from American arms merchants, a policy greatly to the detriment of the American public, which would, in general, favor a policy more like: disengage from the Middle East, stop supporting torture, and transition away from fossil fuels. If we had spent $6 trillion on that instead of funding pointless wars and building up the Saudi torture state, we'd be much better off.
Let's stop having unaccountable, immoral people decide in secret what is good for America, and get back to letting the American public decide. We're better at it.
[1] https://www.news.com.au/world/middle-east/inside-yemens-secr...
American arms merchants employ the American public. People get jobs. Taxes are paid. Suppliers and subcontractors benefit too.
A detriment would be if Saudi Arabia bought from China or Russia, enriching people in those nations instead of the American public. We'd also have far less influence over Saudi Arabia... really, they could be a lot worse.
Another angle: Maria Butina's husband is a Republican from South Dakota.