Interesting. Looks like a rather aggressive Tor crawler has found VNDB's Onion address.
Or someone's using the Tor address to bypass the regular rate limits.
I suspect that was the Phobos search crawler, seeing how many VNDB pages can be found in their search index now.
It also doesn't respect the noindex meta tag.
@ayo Probably the latter. There aren't that many Tor crawlers. Does it impersonate torbrowser or something else?
@wasamasa The UA is that of torbrowser, indeed. Behavior is that of a stupid follow-all-links crawler. I kinda expect someone who'd intentionally bypass rate limits would use a more targeted or custom crawler.
A lonely little town in the wider world of the fediverse.