I wanted to inquire what is generally considered fair use of the API and of downloading stuff via third party applications.

I am developing a browser extension for keeping favorites, which benefits from getting detailed info about posts for better search, and i also have an implementation for local caching for the images, to reduce server load and improve performance.

How many requests per minute/hour/something are ok, and is it generally ok to download images programmatically for purpose of caching and/or archiving?

Fits into a couple units. We only have 2-3 servers total.

Downloading one image at a time isn't an issue. If that was enough to kill the site, the site would be long dead. If you were only downloading one image at a time, it probably wasn't you.

The issue started all the way back before the 14th, so if you only started developing your tool around the 25th, it wasn't you.

When I said that someone was hitting the site hard, I meant hard.



API requests don't really hit the site very hard since we use Solr for all tag searching. I'd say that an API pull every 5 seconds is pretty reasonable if it's something that's going to be continuous.

Please do not continuously archive the site. Our Terms of Service prohibits automatically downloading the site in such a way. However, caching images that the user has or is likely to load should be fine as long as you keep your number of simultaneous downloads low.

Just so you know, though, almost all of the load our server gets from loading images is from thumbnails. While it may speed up how fast it loads for the user, caching the actual images doesn't really provide any noticeable benefit to the site. Just throwing that out there since you said "to reduce server load".

According to our Terms of Service:

"You will not use any automated process to retrieve or index any portion of the site or site contents, with the exception of public search engines."

Exceptions might be given to a few users with special permission from the admin. If it is a low-load tool then we might not even notice. One request/5 seconds at most should work.

I usually throttle my downloading experiments to download serially and with delays in between to not hammer the servers, so it wasn't me either :D

So i take it downloading favorited posts is not really ok, or at least not according to the ToS by the letter. That is a bummer, i don't want to violate the ToS intentionally, but i also like tinkering D: The caching was especially important for using the favorites in a VR gallery viewer, not really practical to display a few hundred images as a cloud without cached copies.

The part about load being mostly thumbs is interesting, i cached those too in my tests, but mostly for completeness' sake :) It does make a lot of sense though that they produce way more throughput than the images when i think about it.

Downloading favorited posts is fine. Since a user must initiate the action every time (by favoriting the post), we don't really consider it an automated process. I was more referring to continuously scraping whole tags or the site as a whole.

Ah, i see, that is great then :D I do indeed have no blanket downloading of any kind in mind, just favorites marked by the user and related tag metadata to calculate some metrics like percentage of favorites compared to total posts from some artist and similar. Browsing still happens on the site (hence browser extension and not dedicated viewer).

My app (A Booru client) Mignori connects to different Boorus to browse them.

First, I have no idea if it is my fault, as I don't send any headers identifying my client. Regardless, Gelbooru is the most popular site used by my app. I'm worried it might be my fault since people use Mignori a lot and Gelbooru is the most used server... We don't scrap the site for tags or the site as a whole, maybe it isn't my fault after all!

I realize the answer might be a big NO but is there a way to request access to the API back? I actually thought of a way to make money via usage of the API without requiring users to pay for "premium" accounts like Danbooru does. I will share my idea but I will appreciate it if we can work a way for Mignori to get access to the Gelbooru API back. In the meantime I direct people to a different Boorus that have more or less the same content as Gelbooru.

Log into the user's account. That is the only way currently. It will be updated with access keys in the future.

Gotcha, that sounds like a good idea. Thanks.

Is usage of the API going to need an account in the future then? I don't imagine how I would generate access tokens otherwise.

It will require access to an account, but not necessarily the actual username and password. You will probably just need to supply a unique API key registered to the user.