Yandex ignore le délai d'exploration?
J'essaie de contrôler le Yandex Bot et je suis près de le bloquer.
Après 2 mois d'ajustements robots.txt, il continue de marteler mon serveur à partir de plusieurs IP.
User-Agent: *
Crawl-delay: 4
User-agent: Yandex
Crawl-delay: 6
Sur une semaine, le délai moyen est de 7,6 secondes (secondes/quantité de lignes access_log). Mais il y a toujours des pics chaque jour où il continue de marteler pendant des minutes en utilisant différentes adresses IP.
Ici et exemple: 336 requêtes de page (sans compter les fichiers/scripts, etc.) de 12 IP en 276 secondes (~ 1,2 sec)
5.255.253.3 - - [27/Apr/2019:10:43:14 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:43:15 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:43:16 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:43:16 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:43:17 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:43:18 +0200] "GET …
37.9.113.7 - - [27/Apr/2019:10:43:18 +0200] "GET …
37.9.113.156 - - [27/Apr/2019:10:43:18 +0200] "GET …
141.8.132.26 - - [27/Apr/2019:10:43:19 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:43:19 +0200] "GET …
95.108.181.94 - - [27/Apr/2019:10:43:19 +0200] "GET …
87.250.224.50 - - [27/Apr/2019:10:43:20 +0200] "GET …
87.250.224.55 - - [27/Apr/2019:10:43:21 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:43:22 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:43:23 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:43:24 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:43:24 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:43:25 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:43:28 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:43:28 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:43:29 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:43:31 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:43:43 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:43:44 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:43:45 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:43:46 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:43:47 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:43:48 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:43:48 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:43:49 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:43:50 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:43:51 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:43:52 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:43:53 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:43:54 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:43:54 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:43:55 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:43:56 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:43:56 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:43:57 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:43:58 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:43:59 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:43:59 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:00 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:44:01 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:44:01 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:44:02 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:03 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:44:04 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:04 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:44:05 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:44:06 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:06 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:07 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:44:08 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:10 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:44:10 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:10 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:11 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:44:12 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:44:12 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:14 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:44:15 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:16 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:17 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:44:17 +0200] "GET …
141.8.142.118 - - [27/Apr/2019:10:44:18 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:19 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:44:19 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:21 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:44:21 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:21 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:44:22 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:23 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:44:24 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:24 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:25 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:44:25 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:44:25 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:26 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:44:27 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:44:27 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:28 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:29 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:44:30 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:31 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:44:32 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:33 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:44:34 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:35 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:44:36 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:37 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:44:37 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:44:37 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:38 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:44:38 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:44:39 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:39 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:44:40 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:41 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:44:42 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:42 +0200] "GET …
141.8.142.118 - - [27/Apr/2019:10:44:42 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:43 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:44:43 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:44 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:44:45 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:46 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:44:47 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:48 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:44:49 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:50 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:44:51 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:51 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:44:52 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:53 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:44:55 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:55 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:44:56 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:44:57 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:44:59 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:45:00 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:45:00 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:45:02 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:45:03 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:45:05 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:45:05 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:45:05 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:45:07 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:45:07 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:45:07 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:45:08 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:45:09 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:45:10 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:45:10 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:45:11 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:45:12 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:45:13 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:45:13 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:45:14 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:45:15 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:45:15 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:45:16 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:45:17 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:45:17 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:45:17 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:45:19 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:45:20 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:45:22 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:45:23 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:45:23 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:45:24 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:45:25 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:45:26 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:45:26 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:45:27 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:45:29 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:45:29 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:45:30 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:45:31 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:45:32 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:45:32 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:45:33 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:45:33 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:45:33 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:45:34 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:45:35 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:45:36 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:45:37 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:45:37 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:45:39 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:45:39 +0200] "GET …
87.250.224.119 - - [27/Apr/2019:10:45:39 +0200] "GET …
141.8.132.26 - - [27/Apr/2019:10:45:40 +0200] "GET …
141.8.142.118 - - [27/Apr/2019:10:45:40 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:45:40 +0200] "GET …
87.250.224.119 - - [27/Apr/2019:10:45:41 +0200] "GET …
141.8.132.26 - - [27/Apr/2019:10:45:41 +0200] "GET …
141.8.142.118 - - [27/Apr/2019:10:45:41 +0200] "GET …
213.180.203.40 - - [27/Apr/2019:10:45:42 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:45:43 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:45:44 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:45:44 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:45:45 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:45:46 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:45:48 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:45:49 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:45:49 +0200] "GET … "Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexMobileBot/3.0; +http://yandex.com/bots)"
141.8.132.29 - - [27/Apr/2019:10:45:51 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:45:51 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:45:51 +0200] "GET …
141.8.132.26 - - [27/Apr/2019:10:45:52 +0200] "GET …
141.8.142.118 - - [27/Apr/2019:10:45:52 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:45:53 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:45:53 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:45:54 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:45:54 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:45:54 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:45:56 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:45:56 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:45:57 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:45:57 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:45:59 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:45:59 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:46:02 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:46:02 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:46:03 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:46:05 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:46:05 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:46:06 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:46:07 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:46:08 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:46:08 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:46:08 +0200] "GET …
87.250.224.119 - - [27/Apr/2019:10:46:09 +0200] "GET …
141.8.132.26 - - [27/Apr/2019:10:46:09 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:46:10 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:46:10 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:46:10 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:46:12 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:46:13 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:46:14 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:46:15 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:46:17 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:46:17 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:46:18 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:46:19 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:46:20 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:46:22 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:46:22 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:46:24 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:46:24 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:46:24 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:46:25 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:46:27 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:46:27 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:46:28 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:46:28 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:46:29 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:46:29 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:46:30 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:46:32 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:46:32 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:46:32 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:46:33 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:46:34 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:46:35 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:46:35 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:46:35 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:46:36 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:46:36 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:46:36 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:46:37 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:46:38 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:46:38 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:46:39 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:46:40 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:46:41 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:46:42 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:46:43 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:46:44 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:46:46 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:46:48 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:46:48 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:46:48 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:46:50 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:46:50 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:46:50 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:46:51 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:46:52 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:46:52 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:46:53 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:46:53 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:46:54 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:46:55 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:46:55 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:46:55 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:46:56 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:46:56 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:46:56 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:46:58 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:46:59 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:47:01 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:47:02 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:47:03 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:47:04 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:47:05 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:47:06 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:47:07 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:47:08 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:47:10 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:47:11 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:47:11 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:47:12 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:47:13 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:47:13 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:47:14 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:47:15 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:47:15 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:47:16 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:47:17 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:47:18 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:47:19 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:47:19 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:47:20 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:47:20 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:47:21 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:47:22 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:47:22 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:47:23 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:47:23 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:47:23 +0200] "GET …
87.250.224.119 - - [27/Apr/2019:10:47:24 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:47:26 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:47:27 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:47:27 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:47:28 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:47:28 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:47:29 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:47:29 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:47:30 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:47:31 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:47:33 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:47:34 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:47:34 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:47:35 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:47:36 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:47:37 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:47:37 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:47:37 +0200] "GET …
213.180.203.61 - - [27/Apr/2019:10:47:38 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:47:39 +0200] "GET …
141.8.132.29 - - [27/Apr/2019:10:47:39 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:47:48 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:47:50 +0200] "GET …
5.255.253.3 - - [27/Apr/2019:10:47:50 +0200] "GET …
Ce n'est qu'un extrait d'une "visite" de 40 minutes avec 1547 demandes de page par 18 adresses IP (~ 1,57 sec).
Lorsque Google et les robots Bing sont actifs en même temps, j'obtiens une utilisation élevée du processeur.
Yandex ignore également mon Crawl-delay règle? Comment puis-je le faire pour ne pas marteler le serveur comme ci-dessus?
Si vous pensez que yandex n'obéit pas au Crawl-delay directive essayez de vous inscrire à Yandex Webmasters et validez votre domaine. Une fois la validation terminée, vous pouvez définir manuellement le taux d'exploration sous Indexation -> Taux d'exploration
Pensez à lire cet article sur les webmasters Yandex Guide des outils pour les webmasters Yandex