U.S. National Park Service API
nps.gov
nps.gov
> The executive budget process consists of three main phases: development of the President’s budget proposal, submission and justification of the President’s budget proposal, and execution of enacted annual appropriations and other budgetary legislation.
https://crsreports.congress.gov/product/pdf/R/R47092
I just think it's important to remind people whenever possible that the United States President isn't a king, isn't all-powerful, and isn't even a Prime Minister. The role is deliberately separated from the legislative branch, and as such cannot introduce legislation of any form.
It's not quite true. They don't want to do all of that work, so they do generally follow the executive branch's requests. But Congress has the ability to cut anything they want -- and to add in items that the department doesn't want. (e.g. https://publicintegrity.org/national-security/congress-funds...)
Thus, the annual ritual of the dead-on-arrival pronouncement. It's especially mandatory when the President and Speaker are of opposing parties.
The median software engineer salary in the US is something on the order of $120k/yr. The GS pay scale tops out at just under $160k before locality is taken into account and that's only a small single-digit percentage increase at best, but nobody is getting hired as a GS15 to do development work.
You will make more in the private sector for sure but "hundreds of thousands" is going to be the minority by far.
Edit: After a bit more research there's a separate executive pay schedule (not that different than the private sector after all) but it's not super clear to this outsider what determines which schedule you get on other than probably just defaulting to GS.
Also 18F has special hiring authorities with 2 year appointments that are not restricted to the GS pay scales for the very issues you bring up.
> https://join.tts.gsa.gov/compensation-and-benefits/
That is the page where you find jobs for 18F, nothing there indicates anyone at 18F is not on the GS scale.
It's important to note that that is GS15, step 10, with the maximum locality adjustment (which I think is just Alaska but California is not far behind). GS15 step 1 is going to be $123k before any locality adjustment so still very likely under $150k/yr unless you're in one of the highest cost areas. Alaska is 32%; most midwest states are 17%.
Money is important, obviously, but it's not always the only thing that motivates people.
All information provided as part of the NPS API roadmap is provided for informational purposes only, is general in nature, and is not intended to and should not be relied upon or construed as a binding commitment. Please do not rely on this information in making decisions, as the development, release, and timing of any products, features, or functionality are subject to change.
Maintenance mode since 2017 is fine. I'm planning to write the interface with Jquery UI.
Campgrounds is also only providing static data. An "API" should show campsite availability and provide endpoints to book campsites to be useful as an API.
I was hoping to have API access to live bear collar tracking or some such. I'd love to create a system that auto-alerts me if a bear's motion path is intersecting my hike.
nps-public-data.nps_public_data
https://console.cloud.google.com/bigquery?hl=en&project=nps-...
If you like it I’ll update the batch job to keep it fresh
- National Park of American Samoa (est. 1988) is missing any designation
- Sequoia and Kings Canyon are listed as the same park (parkCode: seki)
Then it made me wonder, since AWS/cloudflare already have ML products and have data stored in edge caches already, if they don’t/couldn’t train directly from the caches?
^^ Change republican to politician
Biden granted more oil and gas drilling permits than Trump in his first 2 years in office https://news.yahoo.com/biden-granted-more-oil-and-gas-drilli...
With PUblic Datasets, the account making queries pays for the queries. NPS only pays for the storage (which is minimal).
With this API, NPS has to pay for every call to the API. That’s not cheap.
With BigQuery they just copy the data in via CSV and Big Query handles the indexing & query engine.
Open source tools that will present a simple, read-only REST API over an SQL db with little to no custom code exist (so do proprietary tools, often from DB vendors and sometimes as part of SQL db products.) Same with NoSQL or sorta-SQL storage solutions.
The idea that they have to write a bunch of boilerplate code to do this is false. They might choose to do that, but its defintely not necessary.
> Authentication, throttling, threat prevention, encoding, etc etc.
Again, open source canned solutions that take a little bit of configuration exist for many of those, and some of them are likely shared services that they just point at whatever service needs them.
> to access public data
Keyword is access. Hosting on AWS is an implementation detail that doesn't block the end consumer from accessing the data.
* AWS
* Comcast for their internet service
* Apple for their laptop
* A number of software providers for their development tools.
But asking the customer to pay google to query the data is crossing the line?
Site hosting is not a customer cost.
The rest you list are costs orthogonal to this service.
> But asking the customer to pay google to query the data is crossing the line?
Yes.
Why are you arguing for a US government agency to require its citizens to pay for access to data which they have already paid for by funding said agency?
That's the difference.
That's fundamentally a lot worse than the government paying hosting costs to one particular vendor for a commodity service.
People have quite literally died over the issue of public access to public data. It’s quite an important belay point to arrest the deterioration of the spirit of open networks.
What?
Taxes are, generally, money.
That was what was suggested upthread.
Requiring the user to have certain capacities to access data, where those capacities are provided by a number of competing vendors (and some by free, gratis and/or libre sources) is a very different thing.
So are you ok with some chinese APP company making 50 crappy NPS themed apps and having taxpayers pay for the backend?
Addressed in a comment in another subthread, which I know you are aware of since you responded to it, too: https://news.ycombinator.com/item?id=39086270
I will make that trade every day of the week if it means access continues to be through a standard protocol (HTTP) and not beholden to any particular vendor.
That said, I hate Medium with a passion and that things like the Netflix tech blog are hosted there.
On that note, what would the processing entail? Processing the get request and packaging the entire dataset into a REST object right? Or is it a more complex API that lets you run queries against the dataset? For that matter wouldn’t downloading a CSV also have to be packaged into a REST object?
So someone can host a ripoff NPS app on the App Store and taxpayers now pay for content hosting?
You can access for free, but if you abuse or break the TOS your access is revoked. Done
Then use their API to populate a BigQuery public dataset and make available to all.
Otherwise, perhaps we, as outside observers, need to consider the possibility that those whom made the decisions to provide this service as such did so for reasons which we may not be aware.
nps-public-data.nps_public_data
parks
people
feespasses
I am perfectly fine with it being considered part of the basic, taxpayer-supported functions of government agencies to be providing the public with relevant data.
If there is a concrete abuse or wildly disproportionate cost problem in particular areas, that may need to be addressed with charges for particular kinds or patterns of access.
[0] Douglas Adams, Hitchhiker's Guide to the Galaxy
I'd be pretty happy with sqlite dumps too.
I don't really have an issue with the REST, though. I wouldn't be surprised if this was just a standard and cheap to set up Django+REST libraries stack. Yeah, the compute costs are higher than transferring static files, but I'd be shocked if this was taking enough QPS for the difference in cost to make a meaningful difference.
I get wanting the government to be responsible, but this veers a bit too far into Brutalist architecture as an organizational principle.
What’s spammy about my post? I have asked people to focus on costs when they make general statements like “all govt data should have a REST api”.
I think we can dispense with the risky argument, because this API has existed for years without issue.
When you make the argument that "X is too expensive," the onus is on you to prove it's expensive in a relative sense, not simply in an absolute sense. Saving $100 matters if you're spending $1000; it probably doesn't matter if you're spending $10m. Feel free to convince us: make some estimates, crunch some numbers, and look at existing NPS IT spending and see if they seem ballpark reasonable. Otherwise you're just banging on about a left-field solution that almost no one wants because it's putatively cheaper (but by how much, you can't say).
Sure the concept of rest APIs is mature, the but each implementation is untested.
People here clearly don’t like a querier pays model and that’s fine. But should NPS still reinvent the wheel across the SDLC to serve this data? I think there’s a compelling argument in there.
REST API compute is very expensive when you include compute costs, transfer fees and admin costs to keep it up.
Not to mention the cost to implement a bespoke API and deal with security issues.
All to make CSV available!
The goal of this should be for everyone to have access and lower barriers to entry, not put bureaucracy in the way of access and de facto suppress use by open source projects because each user would need their own API key unless someone publishes one.
An API key is about the simplest possible way to achieve that, and appears to be perfectly adequate in this case.
What do you suggest? SAML?
HTTP is an "API" that has no API keys and all the public web servers in the world seem to manage this without any trouble.
> What do you suggest? SAML?
No authentication required by default -- it's public data. Just impose a reasonable rate limit by IP address and require registration only if someone has a legitimate reason to exceed that.
Um, no. That’s just not true.
Yes you did. When you logged in, they gave you an API key in the form of a cookie that you include with every request.
And it's run at a loss by Y Combinator, which is very, very wealthy. And even hackernews has to pay for cloudflare and mods, on top of hardware, hosting, and traffic.
You can read this website (i.e. make queries against its database) without logging in. Moreover, the main thing the cookie does is not some kind of rate limiting or denial of service protection, it's assigning your username to your posts so that others can't impersonate your account. Various image boards exist that even allow you to post without logging in and they seem to be fine with it.
Yeah, but the sentence I replied to was "nobody signed up for an API key in order to make posts". That claim was false. Being able to read the website is a totally different topic.
It was not. A login cookie isn't an API key. It serves a different purpose, which you can observe on the services that do have an API key and then separately require some other credentials to make posts as a particular user account.
Here's a good way to distinguish them. If I want to make my own app (in this context a web browser), do I have to maintain some intermediary servers that the app makes requests through in order to keep my, the app developer's, API key a secret from the users who are using the app? No, the user only needs their own user account, and only for the things that require a user account, and the service expects for each user to have their own account, rather than each app.
It was. Google "what is an api key", and the first result is
> An application programming interface (API) key is a code used to identify an application or user and is used for authentication in computer applications.
Yes, as you argue, it is indeed used to indentify multi-user applications. It is also used to identify users. It is not as narrow as you thought. Learning something new is a good thing! I'll be abandoning this thread now. If you need to get the last word, go ahead. If you need a victory, then fine- I was wrong all along, you win.
https://news.ycombinator.com/item?id=39094541
Which says:
> A login cookie isn't an API key.
If the first result is authoritative then I guess that sorts it.
But your link was from this site:
https://www.fortinet.com/resources/cyberglossary/api-key
Which is confusing because it also says:
> API keys cannot be used for secure authorization because they are not as secure as authentication tokens. Instead, they identify an application or project that calls an API.
> API keys are generated by the project making a call but cannot be used to identify who created the project.
> API keys are used to identity projects, not the individual users that access a project.
Which certainly implies that API keys identify applications or projects. But it's not that confusing because when the first definition says "user" what it means in context is the application developer.
Using the same definition out of context would lead you to believe that, for example, your browser's user agent string is an API key. It's a code (i.e. symbols) that identifies an application or user (browser fingerprinting) and is used for authentication in computer applications (some sites may require you to authenticate again if your browser fingerprint changes too much). So clearly that definition is too broad without context. If you allow a loose enough definition of "code" it would make your screen resolution an API key because it can be used for fingerprinting in the same way.
> A login cookie isn't an API key.
You.... googled your own comment, and cited it as evidence that my google result was wrong?
I guess I'm done here.
Incorrect. Most large web sites invest in DDOS protection e.g. Cloudflare.
Cloudflare DDOS protection as an example is a lot more sophisticated than merely counting requests per source IP (https://developers.cloudflare.com/ddos-protection/about/how-...).
But API keys aren't any good for that anyway because if someone is just trying to overload your service by brute force, they can send requests regardless of whether the keys are valid and still use up all your bandwidth sending error responses or your CPU/memory opening new connections prior to validating the API keys, and to avoid that you'd still need some kind of DDoS protection.
Where they actually do something is where you're doing accounting, because then if someone wants to send you a million requests, you don't block them, you just process them and send them a bill. Maybe you block them if they reach the point you don't expect them to be able to pay. But if it's a free service that anybody can sign up for as many times as they want then that doesn't do any good because the price is $0 and a rate limit per key is avoided by signing up for arbitrarily many more keys.
Which allows that website (or app) to operate with minimal resources, e.g. by a non-profit or open source project, instead of having to be a for-profit entity which needs some underhanded way to generate revenue in order to display the "free" data.
The NPS Data API is open and accessible to all developers
who wish to use NPS data in their projects.
Limits are placed on the number of API requests you may
make using your API key. Rate limits may vary by service,
but the defaults are:
Hourly Limit: 1,000 requests per hour
For each API key, these limits are applied across all
developer.nps.gov API requests. Exceeding these limits
will lead to your API key being temporarily blocked from
making further requests. The block will automatically be
lifted by waiting an hour.
Rate limits are straight forward to implement per-IP address without having any other information about anyone. The sort of person willing to bypass them by using a thousand IP addresses is the same sort of person who would sign up for a thousand API keys using fake names. How are you supposed to rate limit by API key if "anyone" can get an API key? You'd need to use some means to rate limit how many API keys someone could request, which was the original problem.
IP-based rate limiting is extremely effective because it bifurcates the internet into IP addresses controlled by the attacker and ones that aren't. The attacker can only issue requests at a rate of rate limit per IP address times number of IP addresses (or IPv6 blocks) they control. Then the IP addresses under their control get denied while the IP addresses not under their control, i.e. all of the other users, are unaffected.
This only becomes a problem if they control on the order of millions of IP addresses, but then you're dealing with a sophisticated criminal organization and are probably screwed anyway.
Yes, by definition.
Apparently, you did not review the "Disclaimer" link I provided. In it is the following:
Not all information or content on this website has been
created or is owned by the NPS. Some content is protected
by third party rights, such as copyright, trademark,
rights of publicity, privacy, and contractual restrictions.
The NPS endeavors to provide information that it possesses
about the copyright status of the content and to identify
any other terms and conditions that may apply to use of the
content (such as, trademark, rights of privacy or publicity,
donor restrictions, etc.); however, the NPS can offer no
guarantee or assurance that all pertinent information is
provided or that the information is correct in each
circumstance. It is your responsibility to determine what
permission(s) you need in order to use the content and, if
necessary, to obtain such permission.
Perhaps there is a need for use to be "tracked" in order to ensure legal agreement to the Terms of Use?
And that's exactly how they're used as well. They need a method to track the usage of these services because there is often a cost involved with providing them. You also need a way to block or rate limit usage that is not IP bound.
As an example, when Yr[0] opened up their APIs for free world-wide weather forecast it quickly spiralled out of control. I don't recall the specifics of it, but in short a major phone manufacturer started using their APIs on their phones and it took down the service because of the increased load. They could have solved it by just adding more hardware, things like this is highly cacheable, but when you're dealing with tax payers money you generally don't want to subsidise for-profit companies. So you implement a token and tell them to implement their own caching layer on top of it, and everyone is happy.
I don't see how you'd solve something like that with anything other than a token. The methods you've mentioned in other posts simply don't work when a couple of hundred million phones ping your API every time they unlock their phone and it refreshes the weather widget. It also create no incentive for the developers to do things right, like not checking for updates every time the user does something, even though the initial request also came with a TTL and cache-control header that clearly states when this would be updated again.
The for-profit company is happy, anyway. They get free data and you've priced the competition out of the market.
What things like this are really useful for is to create the app equivalent of weather.gov. Most for-profit "repackage government data" websites and apps are ad-laden spyware that will spin your CPU at 100% and shovel every byte of data they can hoover up into a data warehouse that sells to anyone with a buck while doing little more than displaying the government data.
If you want to create an open source one which is free and promises not to track the user, you can, but then you need the data. If you end up with millions of users, who has more resources to set up caching servers, some individual idealist with zero revenue or the United States Government?
This shouldn't even be a question. The government has to operate infrastructure that can handle millions of users for many other reasons. This should be something they're experienced in, and something like this should just fit into a slot in existing infrastructure. This is what it's for. If all you want is to provide the data for various scummy middlemen to wrap in ads and spyware then why is it an API at all instead of a static data dump / live feed with the latest changes?
And they'll also be happy to disregard all your wishes for them to implement their own caching layer and if you have no way to block this kind of activity they absolutely will do it. As demonstrated in the example I gave you.
> If you want to create an open source one which is free and promises not to track the user, you can, but then you need the data. If you end up with millions of users, who has more resources to set up caching servers, some individual idealist with zero revenue or the United States Government?
Me - as a taxpayer - isn't really keen on paying for everyone to build their application on top of it. If you create an open source application you can always tell the users how to obtain such a token.
> This shouldn't even be a question. The government has to operate infrastructure that can handle millions of users for many other reasons. This should be something they're experienced in, and something like this should just fit into a slot in existing infrastructure. This is what it's for. If all you want is to provide the data for various scummy middlemen to wrap in ads and spyware then why is it an API at all instead of a static data dump / live feed with the latest changes?
Again - why should I as a taxpayer have to pay for that? For me, the taxpayer, the service is just as available and usable, even if I have to request a token to use the service. How do you propose you'd limit how a service can be consumed without some kind of token? We've already established that your other solutions doens't work. The alternative is likely just to not provide the service at all, which seems like a net loss for everyone involved, both for for-profit business, taxpayers and open source developers.