A bit of story ๐
Lately, my websites have been dealing with massive amount of requests. There are literally hundreds of thousands of request each month for about three consecutive months. Well, trafic's a good thing but pardon me for saying that traffic seems.. "questionable", at best ๐ค.
Fortunately for me, I host my sites on CloudFlare Pages with does not charge for network egress. It just my curiosity that causes me to dig deeper on this topic.
At first I thought it was DDOS until I see the analytics. The requests were different in location and also consistent for three months. Three months. Who in their right minds would DDOS my sites for three months in a row?? โ๏ธ
I mean it's fine if you want to attack like NASA or whatever, but come on now..
It just so happens that I stumbled across this video by Matt KC that then I started thinking about it seriously:
So I dug a little deeper and found out how this "Bytespider" bots have been wrecking people's nerve on the internet. Just look at this post by @generosus on the Wordpress forum from 9 months ago: (link: https://wordpress.org/support/topic/psa-bytedance-and-bytespider-bots-recommend-blocking/)
Or this one in the phpBB forum from 2020: (https://www.phpbb.com/community/viewtopic.php?t=2550501)
Well, I might be a bit late to the party since the anomaly has only occured to me lately. Nevertheless, here's how to block these bots with CloudFlare WAF in case you experiencing the same issue or just want to be careful.
Here goes ๐ก๏ธ
Well first of all of course you need a CloudFlare account and I will assume you already have one.
Step 1: Go to your dashboard and click on your site
For this tutorial I will go with justforvan.com.
Step 2: On the sidebar, Go to "Security" then "WAF"
Pretty straightforward I guess. Just click on "Security" first then choose "WAF".
Step 3: Click "Create Rule"
On the middle pane that uncover after you done step 2, click the blue button "Create rule".
Step 4: Fill in the rule
Fill it like so:
Section | Content | Modifiable? |
Rule name | Byte bots | YES |
Field | User Agent | NO |
Operator | contains | NO |
Value | Bytedance | NO |
Choose action | Block | NO |
After that, press the OR button on the side:
Then in the new row, fill it like this:
So the entire page will be like:
Or in tabular form:
Rule name
Section | Content | Modifiable? |
Rule name | byte bots | YES |
Field | User Agent | NO |
Operator | contains | NO |
Value 1 | Bytedance | NO |
Value 2 | Bytespider | NO |
Choose action | Block | NO |
Step 5: Deploying the rule
And yes finally just click the sweet blue button of Deploy โ .
Closing ceremony
And yes that's how you do it. Pretty straightforward I suppose. Goes pretty well when you serve your site through Cloudflare, like your Angular site that is deployed on Pages.
Hope this helps and I will be seeing you next time! ๐๐
Special mentions
Big thanks for the amazing people behind these posts:
https://community.cloudflare.com/t/blocking-user-agents/523190
https://stackoverflow.com/questions/57908900/what-is-the-bytespider-user-agent
https://wordpress.org/support/topic/psa-bytedance-and-bytespider-bots-recommend-blocking/
https://www.phpbb.com/community/viewtopic.php?t=2550501
https://www.johnlarge.co.uk/blocking-aggressive-chinese-crawlers-scrapers-bots/#comment-11823