Forum:Yandex

ShoutWiki — express yourself and be heard!
Jump to navigation Jump to search

Hello. Is it true that ShoutWiki banned page indexing in Yandex in the robots.txt file? --Sdobnikov (talk) 21:09, 4 April 2023 (UTC)

Yes. Not sure how you heard about that, but, yes.
It isn't a decision I took lightly, but, the Yandex crawlers don't respect Crawl-delay, and were absolutely flooding the servers with crawl requests during the period of really bad site instability during the upgrade. I spent a few hours actively trying to set up an account to their webmaster tools to adjust the crawl rate in that, but, setting up a new account requires verifying an account via SMS, which never arrived (on multiple numbers on various attempts) and it got to a point I hit the limit on re-requests for the day.
When I'm in a stronger signal area for long enough to sit down and try again, I'll see if I can sort that. --Lewis Cawte (talk) ShoutWiki Staff 21:19, 4 April 2023 (UTC)
As an update, we think we're in a better place now, although we've still not managed to get the SMS verification from Yandex, so we've re-enabled it for now and will be monitoring it closely. --Lewis Cawte (talk) ShoutWiki Staff 22:42, 18 April 2023 (UTC)