Hacker News Reader

Comment by mildbyte | Hacker News Reader

Absolutely, having ability to download the actual data and keep it is always going to be important. We want to facilitate access to data and think it should be available from the source. But, there will inevitably be fragmentation, so it's valuable to have a service available to catalog and aggregate it and make it available over a single protocol.

For what it's worth, we run PostgREST [0] on top of the DDN, so you can get your query results in JSON and CSV files. For example:

    $ curl -sSH "Content-Type: text/csv" https://data.splitgraph.com/cityofchicago/covid19-daily-cases-deaths-and-hospitalizations-naz8-j4nc/latest/-/rest/covid19_daily_cases_deaths_and_hospitalizations | wc -l

    172

It's limited to 10k rows but we might have an ability in the future to "order" a CSV dump asynchronously and place in a destination (like an S3 bucket) of choice.

[0] https://postgrest.org/en/latest/

View on HN

Any chance you’re willing to enable CORS on these endpoints? (Access-Control-Allow-Origin: * response header.)

Hey! Big fan of your work.

For the REST API endpoints (e.g. [0] [1]), we do set the Access-Control-Allow-Origin header to *, but for GET requests we do not send it at all. I'm not sure if this is the same as setting it to star. We can set it explicitly though; we do want to enable this use case (eventually we'll have quotas).

Soon we're releasing a Web UI for writing SQL queries. Part of that will include an HTTP (or websocket maybe) endpoint that takes a SQL query and returns the rows, so you'll be able to execute SQL queries directly from the browser.

As an aside, we'd love to get this working with Observable! Let us know anything we can do to help. You can email me at miles@splitgraph.com or join the Discord and we can figure it out :)

[0] Docs: https://www.splitgraph.com/cambridgema-gov/covid19-cumulativ...

[1] Endpoint (powered by PostgREST): https://data.splitgraph.com:443/cambridgema-gov/covid19-cumu...

Not setting it is equivalent to a deny

Ok, so this should be working now, for all types of requests. Let us know if it's not.

If you're testing with the same endpoints, you may need to give the cache a few minutes to evict the old values missing the header.

It works, thanks! https://observablehq.com/@mbostock/hello-splitgraph

Did you mean to use the "Accept" header? "Content-Type" describes the format of the request body.

Yeah, sorry, got too excited! It will return JSON by default and CSV with Accept: text/csv:

    $ curl -sH "Accept: text/csv" https://data.splitgraph.com/cityofchicago/covid19-daily-cases-deaths-and-hospitalizations-naz8-j4nc/latest/-/rest/covid19_daily_cases_deaths_and_hospitalizations?select=lab_report_date,cases_total,deaths_total | head -n5
    lab_report_date,cases_total,deaths_total
    2020-03-01,0,0
    2020-03-02,0,0
    2020-03-03,0,0
    2020-03-04,0,0

Very nice!

Reply on news.ycombinator.com