Passing Pointers
blog.filepicker.io
blog.filepicker.io
Urls as pointers are not needed in a world where Facebook controls everything and has access to everything by id in it's backend.
A world where services provide and consume urls is a world where it doesn't matter what server something is on, everyone can participate.
The concept the OP is going for is 'deferred work' which is to say not to pass around data that isn't going to be used. And that is indeed a noble goal, but you must have a way to vet that the pointer you passed still points to the thing you thought it did, or you will find out what so many C programmers have discovered about caching pointers, bad bad bad idea.
In the C caching pointers case, the issue is that the system makes no guarantees about the "live"-ness of any prior pointers, whereas on the web this is entirely possible and encouraged (See oft-cited post about "Cool URIs don't change)
How can that ever be a realistic expectation over anything but the immediate and short term? Not to mention that in the overwhelming majority of cases the "provider" of a URL is not the party responsible for maintaining it.
I have to agree with the parent - unless the recipient of the URL [continuously] validates the source, she is simply asking for trouble down the road.
Trusting a website doesn't mean you have to trust them indefinitely. You can trust that the URL will be kept alive for a certain length of time - minutes, hours, days, etc - and deal with them accordingly.
If you want long term storage, you can always GET the content and store it locally. But if you only get a copy of the data, you can't do everything else (PUT back updated data, poll for changes, etc).
A loooong time ago there was a web site that hosted comments (...) The comment site got sold or acquired and someone put up malware on every single inlink. Blammo armed and dangerous.
And for all you know you can get malware from a data copy as well. Using URLs is not a reason to disregard basic security practices, like verifying the content that you receive from the other service.
The concept the OP is going for is 'deferred work' which is to say not to pass around data that isn't going to be used. And that is indeed a noble goal, but you must have a way to vet that the pointer you passed still points to the thing you thought it did
Which you have: the documentation says the URLs are valid for 4 hours.
Truth. Every time I ping from my server and get ridiculously low response times, I have to pause to think before thinking "Thank you AWS".
Bandwidth will always get used up. "64k is enough"...LOL.
A pointer is handy and convenient until the resource it points to disappears.
A 501 sounds like the closest error code, and I'd say having both asynchronous and synchronous modes of locking might be useful. Synchronous just holds the connection open (certain frameworks don't mind long-lived connections) while an asynchronous method might pass in a callback_url in the request to be hit when the file is ready, in the case of lockage.
(NB: to be honest I'm not too sold on the demand for locking vs ovewriting, I guess I threw it in the list of [things that files can do]. Might be interesting to see this need evolve as files move to the "cloud" though.)
EDIT: While I'm at it, a PUT method for creating files could be cool too, to let people use filepicker without the JS widget.
// oops, missed the link. meant to reply to sibling comment
return &"Read only data";I think there are two use cases for URLs as pointers, transient and persistent. In the transient case, you use the URL as a temporary representation, pass it around to the relevant hops, and then fetch it wherever the final resting place may be. In the persistent case (say you're building a web app where users upload content and you want a permanent link), there needs to be somewhere in the line that a service "guarantees" that the link will remain alive and true so that the resource it points to sticks around. So in our case, what we do is if the developer asks for a persistent link, we will take a snapshot of the content and persist it encrypted on our CDN for them, making sure the link stays alive
As well, you'd need to deal with abuse issues. If the 3rd party is going to host a copy, you need to make sure it can't be used as a DOS attack against the URL's original host, for example.
The web is chaotic by nature. That is its strength, but it's also a weakness when it comes to data longevity. Without some kind of standard for guaranteeing data persistence that reaches critical mass (such as over 80% of content hosters implementing it), this will be a very tough nut to crack.
Another problem I see is legal. DMCA takedowns are so easy to do that a dead URL is only a lawyer letter away, even if their claim is completely invalid. Now suddenly a single takedown will affect potentially hundreds of sites instead of just one (in the case of URL pointers rather than data copies).
If a content site is DDOSed, suddenly the damage is magnified due to all of the other sites depending on a single point of failure.
When will I be able to keep a "this is in use here" record for a file that is stored else where. Whilst way better than download-upload on broadband / cell; it still seems dumb to copy in the first place, even if it is on internet backbone.
Anyway, apps that don't need security should be as you describe already.