Bot and crawler requests

Hello,

I see a phenomenal amount of requests constantly running on my WordPress site (https://valois-cyclotourisme.fr). I created a robots.txt file but it doesn’t seem to have much effect.

I sometimes get the message Internal Server Error and I see HTTP 500 errors in the logs.

At certain times it becomes almost systematic, then it calms down a bit and can return.

A full copy of the site on my local PC works without any problem.

My site is running WordPress version 6.9.4 and PHP 8.3.

I looked through the plugins but didn’t find anything except “Events Manager – OpenStreetMaps,” which I have since uninstalled.

If anyone has ideas or advice, I’m all ears.

Thanks.

Hello,

You are on a shared‑hosting plan and your goal is to reduce the load imposed by certain bots so that the site stays available (and not necessarily a plugin that would also consume heavy resources on each request).
For that you need to act at the .htaccess file level to return HTTP status codes such as 429, 503, 403 https://fr.wikipedia.org/wiki/Liste_des_codes_HTTP
And only on specific criteria like the user‑agent or the IP address range, for example.
The advantage of directives in this file is that your site does nothing; the server handles it directly, which is much faster.

Have you looked at the hosting access logs and identified a harmful bot for the site?

cloudflare > bot fight mode + AI crawler blocking.

It's the simplest, free solution to implement.

Cloudflare is listed as compatible up to version 6.9.0; I’m on 6.9.4. Can I try it, or is it risky?

I constantly see bot requests: amazonbot, dotbot, semrush, mj12bot, petalbot, etc.

It never stops, several per second...

No, the CDN, not the plugin.

CDN: Content Delivery Network?

a CDN network is a group of geographically distributed servers that cache content close to end users

Is that it?

You need to transfer your domain (the NS records) to Cloudflare, then properly configure the Cloudflare account to protect your site.
This is not a plugin to install on your site; it’s an infrastructure component to add upstream of your hosting.

I’d like to do the same, but how...?

If you don’t have a very technical background (no problem), I’d recommend providing your raw access logs (available in your OVH manager) and chatting with an LLM (Claude, ChatGPT, Lechat …) and asking it to generate .htaccess rules for you.

Note, I'm a techie, and I use AI extensively to crank out scripts, generate patterns, etc...
It saves a crazy amount of time.

Well yeah, same :slight_smile:
Claude Code.

Me too :wink:

ok, thanks, understood.