Yeah, when I write webscraper code (which isn't very often) I put those kind of thoughts into it too, as I don't want to overburden a website with a massive DoS attack. I usually set up a queue with timers and either evenly spaced time delay between queries, or a random time delay within a range to blend in with other traffic more, and I try to err on the side of caution and underburden, handle errors, and try to detect unexpected scenarios and bail so an error doesn't send the script into a loop pounding a site and getting the same error over and over etc.

I'd have to think about it more but my first thought is to break it down into at least 2 operations. The first is finding potential nicknames to check for the existence of a shared wishlist, and the second is querying the wishlist and caching the results. I would probably start by scanning one page worth of threads in the General Discussion thread and then start walking the thread topics by queing and loading them one at a time in most recent post to least recent order of thread, and also most recent page to oldest page within a thread with the site configured to maximum posts per page for making less requests to the webserver.

I'd put a healthy time delay of a good 30s or more in between queries during testing until I got any kinks ironed out, then I'd probably set it to load pages with a random delay between loads varying from 17-37 seconds or so to more simulate human-like behaviour. The reason for the odd numbers instead of rounding to 10 is that it shows up in logs looking more random. :) If the bot gets banned for excessive queries, just increase the delays in a binary fashion to find the sweet spot to dodge the software detection. That puts it outside of the range considered problematic by a given server's configuration and thus "reasonable" IMHO.

Then the max post in each thread should be tracked in a database so subsequent scans don't redownload already processed data and waste more resources too. Next I'd probably sort the username list in order of most frequent poster to least frequent under the premise that someone who is more frequently posting is more interactive in the community and their wishlist is potentially more relevant (depending on what the purpose of gathering the data is to be used for of course). Then I'd probe for their wishlists and make a note of whether or not they had a public list or not and timestamp it in a database along with the result (yes/no). If they have a public wishlist I'd yank the data that is desired and remember the timestamp of that also.

For subsequent probes I would check the last-probe timestamps and only re-probe a given wishlist if N amount of time elapsed since the last probe. For people who do not have a public wishlist I would probably pick an arbitrary time interval to re-probe in case they open their wishlist public at some point in the future. I do like to get fancy and smart with this sort of thing too, so I'd probably keep a running count of how many times they were probed and did not have their list public, then use exponential fallback to degrade probing for them again. So for example, probe once per week by default and increment a counter for their "total probes resulting in non-public/not available count", then wait some multiple of that before the next probe. ie: 1 week then 2 weeks, 3 weeks, 4 weeks perhaps up to some maximum interval, or atlernatively double it each time 1/2/4/8. The purpose of the incremental or exponential fallback of course is to try to greatly reduce the load on the remote webserver on data that is predictably going to be false anyway, but to not give up entirely in case it does show up at some point. If someone makes their list public after a dry spell, reset the probe counter to 0 so it is probed regularly as long as it is public.

Anyhow, just some random ideas... I hope that makes some sense... No time to hack away at it myself though... WITCHER 3!!! :)