Mitigating Residential Proxy Scrapers: Lessons from a Real-World WAF Implementation

Web scraping has undergone a major technological evolution. In the past, blocking a bad bot was as simple as identifying requests coming from major cloud hosting providers (like DigitalOcean, AWS, or Hetzner) and blacklisting their ASNs.

Today, scrapers have weaponized residential proxy networks. By routing automated scripts through millions of residential IP addresses leased from consumer ISPs worldwide, these scrapers blend in perfectly with real human traffic.

In this article, we will share a practical case study based on our own implementation of detecting these bots, crafting a resilient Cloudflare Web Application Firewall (WAF) rule, and solving a critical DevOps “loopback” issue that broke our static site build pipeline.

Part 1: How to Detect Residential Proxy Scrapers in Your Logs

Unlike datacenter bots, residential proxies cannot be blocked by ASN alone because they use legitimate consumer networks (such as Comcast, Charter, Jio, or KPN). However, they leave telltale behavioral signatures in your HTTP access logs.

Here are the three primary indicators we used to identify a coordinated residential scraper campaign:

1. Coordinated Timing (Burst Attacks)

Residential proxy bots rotate IPs constantly to avoid rate limits. In our log analysis, we noticed a pattern where requests using the exact same User-Agent (e.g., Firefox/148.0) hit our server from completely different countries (Taiwan, Germany, Sweden, US) within a 5-minute window. A real user does not teleport across continents in seconds.

2. Isolated Static Asset Fetching

When a human browses a website, their browser first requests the HTML page. Once loaded, the browser parses the HTML and immediately sends subsequent requests to fetch CSS, JS, fonts, and images.

The scrapers, however, bypassed the HTML pages entirely. They directly requested raw .webp images and static assets from our CDN domain. Because they did not load the page first, these requests arrived with an empty Referer header.

3. User-Agent and Client Hints Mismatch

Modern Chromium-based browsers send User-Agent Client Hints (UA-CH) like sec-ch-ua-platform. Bots that spoof their User-Agent strings to look like a desktop Chrome user often forget to modify these client hints. For instance, if the User-Agent claims the client is Windows, but the sec-ch-ua-platform header says “Android” (revealing a mobile emulator), it is a guaranteed bot.

Also Read: The New Era of Web Development: What Has Changed?

Part 2: Writing a Precise Cloudflare WAF Rule

To stop these scrapers from downloading our static images directly from our CDN domain (cdn.yourdomain.com), we drafted a custom Cloudflare WAF rule.

The goal was simple: challenge or block any request to the CDN domain that does not have a referer matching our site.

An optimized WAF expression to achieve this looks like this:

(http.host eq “cdn.yourdomain.com” and not http.referer contains “yourdomain.com” and not http.request.uri.path contains “favicon.ico”)

We applied a Managed Challenge to this rule. When a real user browses the website, their browser sends the page referer, bypassing the rule. If a user pastes the raw image URL in their browser directly, they will solve a silent JavaScript challenge automatically. But automated scraper scripts will fail the challenge and get blocked.

Also Read: Why We Chose Astro to Build Our Modern Publishing Platform

Part 3: The DevOps Catch — Solving the Build Pipeline Failure

Why did the build fail?

Our site is built using a modern Static Site Generator (Astro). At compile-time, the build server runs a Node.js process that fetches remote images from cdn.yourdomain.com to determine their dimensions (width and height) to prevent layout shifts (CLS).

Because the build container runs headless:

1. It sends requests with a node User-Agent.
2. It sends requests with an empty Referer (since it is running on the backend compiler).
3. The WAF intercepted the build container, issued a Managed Challenge, and since Node.js cannot solve JS challenges, the request timed out.

The Solution: Whitelisting Loopback Requests

To prevent the WAF from breaking our build pipeline while still keeping scrapers out, we had to add loopback exemptions to our CDN rule.

We updated the rule to exempt:

• The Build Platform’s ASN: Since our build ran on Cloudflare Pages, we bypassed Cloudflare’s own network ASN (13335).
• The Compiler User-Agent: We bypassed the default Node.js User-Agent (“node”) specifically for the CDN domain.

The updated, production-ready WAF rule looks like this:

or (http.host eq “cdn.yourdomain.com” and not http.referer contains “yourdomain.com” and not ip.src.asnum eq 13335 and not http.user_agent eq “node” and not http.request.uri.path contains “favicon.ico”)

With this change, the compiler fetches dimensions smoothly, the build completes successfully, and our images remain shielded from automated scrapers.

Key Takeaways for Webmasters

1. Don’t rely solely on ASN blocks: Residential proxies bypass IP blocklists. You must check for anomalies like referers, client hint mismatches, and request headers.
2. Managed Challenges over Hard Blocks: Instead of flat-out blocking suspicious requests, use Cloudflare’s Managed Challenge. It is seamless for real humans and fatal for bots.
3. Audit your build pipeline before WAF deployment: If your framework (like Next.js or Astro) fetches images from your own domains at compile time, ensure your build servers are whitelisted.