I generally proscribe to the "Technocratic Utopianism" side of information politics, so I've been offended by needing to register to get data. My first thought on the matter was to allow for use without registering, but do something to rate-limit w/ a warning message if one group starts to get abusive ... probably based on individual IPs or IP ranges.
... and then I remembered this little problem I'm having.
What if someone uses the API in the client software, rather than doing something to cache it on the front end? Then you've got tens of thousands of requests coming from all over the place, and no way to figure out who the proper contact is.
In my case, people really like
http://umbra.nascom.nasa.gov/images/latest_eit_304.gif . They have for years. We get people direct linking it to their websites all the time. When it got really bad in the past (as we were behind a 200Mbps pipe until ~3 years ago, 100Mbps only ~4 years ago), we'd find contact info for the website, and ask them to cache the image every couple hours and serve it themselves. And it generally worked.
But for years now, I've been trying to track down what software acts as a web browser and sets the user agent to 'CompanionLink', with no referrer. I've contacted companies with related names and products, and no one admits responsibility.
The other day's stats for the server :
272463 total requests
87417 requests for that image
82335 requests for that image by 'CompanionLink'
82338 requests by 'CompanionLink' total
(those other 3 'CompanionLink' hits are for other 'latest' type image ... which is new .. the last time I looked into it, we had no other requests w/ that signature for the 30 days we're allowed to keep our logs).
I'm guessing was written into some default for ... something ... but what, I have no idea. At its peak, it was responsible for 3/4 of the hits to our webserver, with one IP address requesting it multiple times a minute. (that telescope only takes one image per filter every 6 hrs these days ... and the spacecraft only gets one downlink a day).
I've been sending an HTTP 406 status code for it for years now, and it still keeps coming.
I'm torn -- I think that for it to be 'open' data, people should be able to get it without registering. But then there's also this aspect, which has been frustrating me for years.
So, my thought on a middle ground:
- Have some API key process that's instantaneous (no waiting for a human to review it)
- Don't ask for contact info, but have them acknowledge that if they start hogging resources or become otherwise abusive, you might have to rate limit them or cut them off.
- If you find someone's mass-registering APIs or similar ... then you can block them or put some throttling on it.
- During the API signup, offer them some way to check for status changes (either leave an e-mail address to be contacted at, or a page they can check to see if they've been blocked)
In the past, when we've blocked someone, we could generally figure out where they were from, or they'd contact us to ask what was up ... and we could work if there was a better way to get what they wanted (eg, add new functionality so they didn't have to send thousands of requests, physically ship drives w/ the data to them, or get them to hit a mirror of the data on an alternate network) ... but when you start getting DDoS'd from abuse, it hurts all of the other people who might be using your API.