Hacker News Reader

Comment by baldfat | Hacker News Reader

It's all about the format:

My city's school district makes images of the spreadsheets and then publishes them in a pdf. USELESS

Happily, this bill actually does care about the format of the data:

"The policy and procedures shall include ... technical requirements with the goal of making electronic data sets available to the greatest number of users and for the greatest number of applications, including, whenever practicable, the use of machine readable, non-proprietary technical standards for web publishing"

"The Treasurer shall consider various means by which to develop a set of universal data formatting standards to effectuate the purposes of this act, including working with other State departments and agencies, and contracting, if deemed necessary, with nonprofit organizations, commercial vendors or third party groups for this purpose. If such standards are developed and adopted by the Treasurer, they shall be the format that each State department and agency will use to provide existing and future electronic data sets"

> the use of machine readable, non-proprietary technical standards for web publishing

Wouldn't that cover PDF, even though it might be deeply inappropriate for it's content? PDF has several ISO standards, and is most definently machine readable.

PDF should also meet the goal as well:

> available to the greatest number of users and for the greatest number of applications

Huge number of PDF clients, such as PDF.js that powers Firefox's reader.

> If such standards are developed and adopted by the Treasurer, they shall be the format that each State department and agency will use to provide existing and future electronic data sets

Knowing how much government uses PDF... It is probably incoming.

But it makes little/no sense for spreadsheet data. Great for graphs and prose, however.

PDF is really a container and can have just about anything inside of it.

https://www.w3.org/2013/04/odw/odw13_submission_52.pdf

PDF with csv attachments is a good compromise but I rarely see that. Usually it is PDF or Excel and the PDF is a nightmare to munge.

While deplorable from an accessibility standpoint, at least the data is available and out in the open to the public. One small step in the right direction.

EDIT: Tabula[1] is an open source tool to extract tables from PDFs into CSV/Excel format in case one really needs to process data trapped in PDF documents.

[1] http://tabula.technology

Does Tabula do OCR? That's what would be required for tables embedded as images.

Unfortunately, it doesn't. From https://github.com/tabulapdf/tabula:

> Caveat: Tabula only works on text-based PDFs, not scanned documents. If you can click-and-drag to select text in your table in a PDF viewer (even if the output is disorganized trash), then your PDF is text-based and Tabula should work.

Tabula is great!

In their defense, that's still better than most academic publications!

> My city's school district makes images of the spreadsheets and then publishes them in a pdf. USELESS

You should be able to sue them for misuse of public funds. Dedicating tax money to convert a useful format (CSV, spreadsheet, etc) into a useless or at least less useful one (ex: PDF) is a clear violation.

It isn't a violation if there isn't any policy or law against it.

Had the same problem with NJ. I FOIA'd Chris Christie's schedules and got some of the original PDF reports (which was fine, they were readable with some Python library) and some SCANS of the original PDF reports, even with OCR they were useless.

Reply on news.ycombinator.com