Blocking Bad Bots: Protect Your Site Performance

Operating rule: block only automated traffic with evidence of business, security, or reliability cost. Monitor the rest until the evidence is strong enough to block without harming search crawlers, customers, accessibility tools, uptime checks, or partner integrations.

Bad bot management means measuring automated HTTP traffic, separating useful automation from abusive automation, and applying the least disruptive control that reduces resource waste, fraud, scraping, or analytics contamination. It is not search-index hiding.

A bad bot is automation that uses valid site functionality in an unintended way and causes measurable harm. OWASP's Automated Threats to Web Applications project includes scraping, credential stuffing, carding, spam, vulnerability scanning, denial of inventory, and metric skewing. Behavior matters more than whether the user agent says bot.

Allow verified search crawlers, social previews, uptime monitors, accessibility tools, approved SEO tools, feed readers, payment callbacks, and authorized security scanners. Treat scrapers, credential stuffers, card testers, fake account creators, comment spammers, click fraud bots, inventory hoarders, vulnerability scanners, and denial-of-service clients as bad when evidence confirms harm.

When Bot Blocking Applies

Bot blocking applies when automated traffic creates one or more measurable problems:

• Origin CPU, memory, database load, bandwidth, queue depth, response time, error rate, checkout completion, or page availability gets worse during bot activity.
• Search crawlers receive slow responses, 5xx errors, 429 responses, blocked resources, or crawl traps.
• Scrapers extract proprietary content, pricing, listings, inventory, lead data, or gated material.
• Login, password reset, checkout, form, account creation, coupon, search, or API endpoints receive automated abuse.
• Analytics, advertising, conversion tests, inventory, or ranking signals become unreliable because bot sessions are counted as human sessions.

Do not block when the only evidence is an unfamiliar user agent, foreign IP, one traffic spike, dislike of AI crawlers, or a general desire to reduce automation. Use monitoring, rate limits, robots.txt policy, terms language, or content licensing decisions first.

Decision Criteria: Block, Challenge, Rate Limit, or Monitor

Use a decision table before writing rules. Each rule needs evidence, action, expected result, and rollback condition.

Escalate in this order: monitor, rate limit, challenge, block. Jump to block only for known malicious payloads, failed crawler verification, credential attack infrastructure, obvious vulnerability scanning, or active availability problems.

Measurable Outcomes

Define success before enforcement. Useful controls change measurable outcomes within a fixed observation window:

• unwanted automated origin requests fall 50% to 90% without reducing verified crawler activity
• affected p95 and p99 response times return to the pre-incident baseline
• 5xx, 429, timeout, database saturation, and queue-depth metrics fall during bot periods
• Google Search Console Crawl Stats shows stable Googlebot activity and no new fetch failures
• login, checkout, form, coupon, or account-creation abuse falls without more support complaints
• analytics exclude bot sessions from human conversion, bounce, session duration, and campaign reporting
• hosting, bandwidth, WAF, search, or API costs fall when bot traffic caused the cost

Do not use blocked requests alone as the success metric. A rule can block many requests while also blocking search engines, users behind carrier-grade NAT, VPN users, disabled users, or monitoring systems.

Evidence to Collect First

Collect at least 24 hours of normal traffic plus the full incident window when possible. For seasonal sites, compare the same weekday and hour from the previous normal period.

Required log fields:

• timestamp, source IP, ASN, method, host, path, query, status, response time, bytes, cache status, upstream status, user agent, referrer, country, region, and edge data center
• TLS fingerprint, JA3/JA4 fingerprint, bot score, WAF action, and rule ID when exposed
• account ID, session ID, API key, or token hash for application endpoints

Useful evidence sources:

• Nginx or Apache access logs; CDN and WAF logs from Cloudflare, Fastly, Akamai, AWS WAF, Sucuri, Imperva, HUMAN, DataDome, or Wordfence.
• Google Search Console Crawl Stats; Google Analytics 4, server-side analytics, or warehouse queries.
• host or dig for crawler DNS verification; WebPageTest, Lighthouse, k6, Grafana, Datadog, New Relic, or CloudWatch for latency, saturation, and errors.

Primary references:

• OWASP Automated Threats to Web Applications for bot-abuse categories.
• Google's crawler verification documentation for reverse DNS, forward DNS, and published IP ranges.
• Google's robots.txt documentation for the limitation that robots.txt manages cooperative crawler traffic and does not enforce secrecy.
• Cloudflare rate limiting documentation for expression, action, counting characteristics, period, threshold, and mitigation duration.

Focused Tests Before Enforcement

• Replay 50 to 200 sampled legitimate requests through the proposed rule in log-only mode.
• Verify Googlebot-like traffic with reverse DNS and forward DNS before allowlisting or blocking.
• Test a cached public page, uncached page, login page, checkout page, form endpoint, and API endpoint.
• Confirm each challenge or block records rule ID, action, timestamp, and matched condition.
• Confirm rollback by disabling the rule without deploying application code.

Identification Process

1. Baseline requests per minute, response time, cache hit ratio, error rate, crawl activity, conversion rate, and endpoint failures.
2. Segment static assets, public pages, search, login, checkout, forms, APIs, admin paths, and webhooks.
3. Find outliers by IP, ASN, fingerprint, user agent, session, token, country, and path.
4. Prioritize expensive paths: database queries, search queries, checkout calculations, login checks, inventory checks, PDF generation, and uncached rendering.
5. Verify claimed crawlers. For Google, run reverse DNS, require an expected Google hostname, then run forward DNS back to the original IP; at scale, match published crawler IP ranges.
6. Label good automation: verified search crawlers, social previews, uptime monitors, accessibility tools, owned security scanners, vendor callbacks, and partner systems.
7. Label bad automation against OWASP categories: scraping, credential stuffing, carding, spam, vulnerability scanning, denial of service, denial of inventory, or skewing.
8. Choose the least disruptive control: monitor uncertainty, rate-limit volume, challenge suspicious browser claims, use WAF rules for attack patterns, and block verified abuse.
9. Deploy in log-only mode when supported; review what would have been blocked.
10. Enforce narrowly by endpoint, method, behavior, bot score, verified-bot flag, ASN, IP range, fingerprint, or token. Avoid broad country or user-agent-only blocks unless the business accepts the false-positive risk.
11. Measure against the baseline after 1 hour, 24 hours, and 7 days.
12. Document owner, date, evidence, exact condition, action, expected outcome, exceptions, rollback trigger, and review date.

Controls and Alternatives

Robots.txt

robots.txt is a policy file for cooperative crawlers. It can reduce legitimate crawl load, but it is not security because malicious bots can ignore it. For content that must not be accessible or indexed, use authentication, authorization, noindex, password protection, or removal.

Caching and Static Rendering

Caching is the safest first control for public content. Move bot cost from origin to edge with CDN caching, full-page caching, static rendering, stale-while-revalidate, and cache keys that avoid unbounded query-string variation. This does not stop scraping; it removes the performance problem.

Rate Limiting

Rate limiting caps request volume by IP, session, token, path, fingerprint, ASN, country, or another characteristic. Each rule needs a period, threshold, action, and mitigation duration. Cloudflare documents periods such as 10 seconds, 60 seconds, 300 seconds, 3600 seconds, and 86400 seconds, depending on plan. Start above measured human and verified crawler behavior. For public pages, a soft challenge after 60 to 300 uncached requests per minute from one source is a common starting point; login or checkout thresholds should be lower.

Waf Managed Rules

A WAF should stop known attack signatures, malicious payloads, vulnerability scans, protocol anomalies, and high-risk bot categories before they hit the application. Keep exceptions for legitimate tools, webhook providers, preview renderers, and search crawlers.

Bot Management Products

Dedicated bot-management systems use device signals, behavioral signals, IP reputation, browser checks, TLS fingerprints, and challenge outcomes. Use them for credential stuffing, carding, scraping, scalping, or denial-of-inventory attacks that simple WAF and rate-limit rules cannot handle. They do not replace application-level fraud controls.

Captcha and Javascript Challenges

CAPTCHA and JavaScript challenges reduce simple automation but add friction. They can fail for privacy tools, assistive technology, strict browsers, low-power devices, corporate networks, and users with JavaScript disabled. Use them as step-up controls, not blanket requirements.

Application-Level Controls

Some bot problems belong in application logic: login throttling by account and IP, password reset throttling, account lockout, coupon attempt limits, inventory reservation expiry, cart hold limits, payment velocity rules, signed API requests, nonce validation, and server-side form validation. Network controls cannot fully protect workflows whose abuse depends on business logic.

Risks and Failure Modes

False positives are the main risk. Rules based only on country, IP, VPN use, user agent, or JavaScript support can block customers, search crawlers, accessibility tools, uptime monitors, payment callbacks, partner systems, privacy tools, and corporate networks.

Specific failure modes:

• Search damage: do not rate limit or challenge verified crawlers unless abuse is confirmed and the SEO impact is accepted. User-agent matching alone is insufficient.
• Performance regression: complex .htaccess rules, application middleware, bot-score lookups, origin fingerprinting, and uncached challenge pages can add latency. Prefer edge enforcement when evidence is available there.
• Analytics contamination: past bot sessions can still distort reports, experiments, attribution, and conversion calculations. Mark the incident window and avoid using it for growth decisions.
• Adaptation: IP blocklists decay, residential proxies hide sources, headless browsers execute JavaScript, CAPTCHA-solving services exist, and distributed attacks stay under per-IP thresholds.
• Legal and privacy risk: document device fingerprints, behavioral biometrics, personal data, retention period, legal basis, vendor role, and user disclosures.
• Operational risk: every blocking rule needs an owner, expiration or review date, alerting, exceptions, and rollback path.

Maintenance Schedule

Review weekly during active abuse and monthly during stable periods:

• top blocked rules by request count and path
• verified crawler traffic, crawl errors, and Search Console Crawl Stats
• origin response time, error rate, cache hit ratio, and hosting cost
• login, checkout, form, API, and search abuse metrics
• support tickets mentioning access denial, CAPTCHA loops, checkout failure, or blocked pages
• exception lists for crawlers, monitors, vendors, partners, and internal tools
• rules that can be narrowed, moved to log-only mode, or deleted

Practical Rule Examples

Starting patterns, not universal thresholds:

• Public scraping: challenge or rate limit a source above 120 uncached HTML pages in 5 minutes with no verified-bot status.
• Login abuse: rate limit failed logins by account and IP, then require step-up verification after 5 to 10 failures in 10 minutes.
• Search abuse: cache common queries, limit high-cardinality patterns, and challenge clients issuing hundreds of unique searches per minute.
• Vulnerability scanning: block /wp-admin, /.env, /phpmyadmin, /xmlrpc.php, traversal strings, SQL injection payloads, and known exploit probes when those paths do not belong to the application.
• Fake Googlebot: block or challenge requests claiming Googlebot when reverse and forward DNS verification fail.

FAQ

Should Robots.txt Be Used to Block Bad Bots?

No. Use robots.txt for crawl preferences. Use WAF rules, rate limits, authentication, authorization, caching, or application controls to enforce behavior.

Should All AI Crawlers Be Blocked?

That is a content policy decision, not a performance rule. Decide based on licensing, attribution, product strategy, and server cost. If the problem is excessive request volume, rate limiting is more precise than a blanket block.

Is Ip Blocking Enough?

No. IP blocking helps during a confirmed incident, but bots rotate through cloud providers, residential proxies, mobile networks, and compromised hosts. Combine IP evidence with behavior, endpoint, fingerprint, account, token, or ASN evidence.

What Should Be Monitored After a New Rule?

Monitor blocked requests, challenges, origin response time, 5xx rate, 429 rate, conversion rate, login success rate, form completion, support tickets, Search Console crawl activity, and verified crawler logs. Roll back if legitimate traffic is blocked or search crawling degrades.

Conclusion

Bad bot blocking is an evidence-based traffic control system: measure traffic, verify good automation, identify harmful automation, choose the least disruptive control, enforce narrowly, and review the outcome. The goal is not to eliminate every bot. The goal is to prevent automated traffic from degrading performance, search crawling, security, analytics, costs, and business workflows.