Interesting. Looks like a rather aggressive Tor crawler has found VNDB's Onion address.

Or someone's using the Tor address to bypass the regular rate limits.

I suspect that was the Phobos search crawler, seeing how many VNDB pages can be found in their search index now.

It also doesn't respect the noindex meta tag. :blobcatdead:

Show thread

@ayo Probably the latter. There aren't that many Tor crawlers. Does it impersonate torbrowser or something else?

@wasamasa The UA is that of torbrowser, indeed. Behavior is that of a stupid follow-all-links crawler. I kinda expect someone who'd intentionally bypass rate limits would use a more targeted or custom crawler.

Sign in to participate in the conversation

A lonely little town in the wider world of the fediverse.