Tools: Protecting Your Website with Cloudflare: Security, Performance, and Reliability [Part 2]

AI Crawl Control ## Rate Limiting ## Saving and deploying the rule ## Setting Up Cloudflare Turnstile: Step-by-Step Guide Cloudflare is an integral service for the modern web, delivering performance, integrity, and security on the edge of the network. In Part 1 of this series, the basic infrastructure deployment domain set-up, onboarding of Cloudflare, domain routing, and origin server preparation were covered. If you haven’t read Part 1 yet, it’s recommended to start there to understand the baseline architecture and configuration before moving forward: 👉 Read Part 1: Cloudflare Fundamentals & Setup As this guide has grown to cover multiple layers of infrastructure, it makes sense to continue with a focused follow-up. In Part 2, we’ll dive into edge-level security and traffic control, including: This part is all about hardening your application at the edge while keeping performance fast and user experience smooth. Why controlling AI crawlers matters AI crawlers are no longer limited to traditional search engines. Many modern bots are designed to collect content for AI training, assistants, or large-scale analysis, often without providing direct value back to your site. Uncontrolled AI crawling can lead to: At the same time, not all crawlers are bad. Search engine bots are critical for discoverability and SEO. The challenge is allowing the right bots while blocking or limiting the rest. This is exactly where Cloudflare’s AI Crawl Control becomes essential. Navigating to AI Crawl Control in Cloudflare To access AI Crawl Control: This section provides a centralized view of all detected crawlers accessing your site, along with: Blocking specific AI crawlers Within the Crawlers tab, Cloudflare lists individual bots such as AI trainers, assistant crawlers, and large-scale data collectors. To block a specific crawler: Once blocked, Cloudflare enforces this rule at the edge, preventing requests from reaching your origin server. This helps reduce unnecessary load and protects your content from being harvested for unintended use. You can selectively block: Keep search engine crawlers allowed (critical for SEO) Search engine crawlers should almost always remain allowed, as they play a fundamental role in how your site is discovered and ranked. Bots such as Googlebot, Bingbot, and other verified search engine crawlers are responsible for indexing your content, updating search rankings, and driving organic traffic to your site. Blocking these crawlers can have serious consequences, including reduced visibility in search results, ranking drops, and even complete removal of pages from search engine indexes. For most websites, restricting search engine crawlers should only be done with a very specific and well-understood reason. Rate limiting is a critical defense mechanism for protecting websites and APIs from abuse, brute-force attempts, scraping, and excessive automated traffic. By limiting how frequently a client can make requests within a defined time window, you can prevent malicious or misconfigured clients from overwhelming your application—while still allowing legitimate users to operate normally. In Cloudflare, rate limiting is enforced at the edge, meaning abusive traffic is stopped before it reaches your origin, reducing load, latency, and infrastructure costs. Navigating to Security Rules To create a Rate Limiting rule: This section is where Cloudflare allows you to define how incoming requests are evaluated and mitigated using custom rules, managed rules, and Rate Limiting rules. Creating a Rate Limiting rule Inside Security rules: This opens the rate limiting rule builder, where you define what traffic to monitor, how often it’s allowed, and what action Cloudflare should take when limits are exceeded. Configuring the rate limiting rule Start by giving your rule a clear, descriptive name. For example: A meaningful name makes future maintenance and troubleshooting much easier. When incoming requests match… This section defines which requests the rule applies to. Using a wildcard is ideal for APIs or grouped endpoints, as it applies the rule to all matching paths under /api/. This section defines how much traffic is allowed before Cloudflare takes action. This means a single client (based on IP, by default) can make up to 10 requests within 10 seconds. Any additional requests beyond this threshold will trigger the configured action. Choose what Cloudflare should do once the rate limit is exceeded. Blocking immediately stops matching requests and prevents Cloudflare from evaluating additional rules for that request, making it an effective mitigation for abusive behavior. This setting defines how long the block remains active. After the duration expires, the client is allowed to send requests again—unless the rate limit is exceeded once more. Once all fields are configured: The rate limiting rule becomes active immediately and is enforced at Cloudflare’s edge across all incoming traffic that matches your criteria. To validate the rate-limiting rule, we can intentionally send multiple rapid requests to the same endpoint to simulate abusive behavior. Once the defined threshold is exceeded (for example, more than 10 requests within 10 seconds), Cloudflare immediately blocks further requests from the same client. At this point, the client receives Error 1015 – You are being rate limited, indicating that Cloudflare has temporarily restricted access. This confirms that the rule is working as expected and that excessive or abusive traffic is being stopped at the edge before reaching the origin server. Cloudflare Turnstile provides bot protection without traditional CAPTCHAs, making it both user-friendly and secure. In this section, we’ll walk through the complete setup process, from creating a Turnstile widget in the Cloudflare dashboard to applying security rules and integrating it into your application. Navigate to Turnstile in Cloudflare Dashboard Create a New Turnstile Widget Choose how Turnstile behaves for users: For most applications, Managed mode offers the best balance between security and user experience. Once configured, click Create. Retrieve Site Key and Secret Key After successful creation, Cloudflare displays: Important: Store the Secret Key securely. It should never be exposed in frontend code. Client-Side Integration Add the Turnstile script and widget container to your HTML page: Apply Turnstile Using Security Rules To enforce Turnstile on specific routes or actions: This ensures Turnstile is triggered only where it’s needed, such as login pages or sensitive APIs. Cloudflare provides far more than a single layer of protection—it offers a comprehensive security ecosystem designed to defend modern applications against a wide range of threats. In this article, we explored how Turnstile delivers frictionless bot protection without degrading user experience, and how it can be combined with other Cloudflare features such as custom security rules, managed challenges, and rate limiting to build a stronger defensive posture. When used together, these controls allow you to protect sensitive endpoints, reduce abusive traffic, and handle sudden spikes or malicious behavior with confidence. Instead of relying on one mechanism, Cloudflare enables a layered security strategy—balancing usability, performance, and protection. By thoughtfully configuring these features, teams can stay ahead of automated abuse while keeping applications fast, accessible, and resilient at scale. Templates let you quickly answer FAQs or store snippets for re-use. Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment's permalink. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse CODE_BLOCK: /api/* Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: <script src="https://challenges.cloudflare.com/turnstile/v0/api.js" async defer></script> <div class="cf-turnstile" data-sitekey="YOUR_SITE_KEY"> </div> Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: <script src="https://challenges.cloudflare.com/turnstile/v0/api.js" async defer></script> <div class="cf-turnstile" data-sitekey="YOUR_SITE_KEY"> </div> CODE_BLOCK: <script src="https://challenges.cloudflare.com/turnstile/v0/api.js" async defer></script> <div class="cf-turnstile" data-sitekey="YOUR_SITE_KEY"> </div> - AI crawl control and bot behavior management - Rate limiting for APIs and sensitive endpoints - Turnstile for user-friendly request validation - Increased bandwidth and infrastructure costs - Higher origin load and degraded performance - Unwanted content harvesting or dataset creation - Reduced visibility into who is accessing your content and why - Log in to your Cloudflare dashboard - Select the domain you want to manage - From the left-hand navigation menu, click AI Crawl Control - Open the Crawlers tab - Crawler name and organization - Category (Search Engine Crawler, AI Crawler, AI Assistant, Archiver, etc.) - Request volume and recent activity - Allow / Block controls per crawler - Locate the crawler in the list - Review its category and request behavior - Click Block in the Action column - AI training crawlers - AI assistant fetchers - Data aggregation bots - Log in to the Cloudflare dashboard - Select your domain - From the left-hand menu, go to Security - Click Security rules - Click Create rule - From the dropdown, select Rate limiting rules - Field: URI Path - Operator: Use Wildcard if you want to match a group of URLs Use Equals if you want to target a single, exact path - Use Wildcard if you want to match a group of URLs - Use Equals if you want to target a single, exact path - Use Wildcard if you want to match a group of URLs - Use Equals if you want to target a single, exact path - Requests: 10 - Period: 10 seconds - Action: Block - For duration… - Duration: 10 seconds - Review the rule settings - Click Deploy - Log in to your Cloudflare dashboard. - Select your account or domain. - From the left navigation panel, go to: Application Security → Turnstile - This section is where all Turnstile widgets are managed. - Click the “Add widget” button under Turnstile widgets. - Provide a Widget Name (e.g., turnstile-test) to identify it later. - Under Hostname Management, click Add Hostnames. - Select an existing hostname or add a custom hostname where Turnstile should be active. - Click Add to confirm the hostname selection. - This ensures Turnstile challenges are only valid for the specified domain(s). - Managed (Recommended) Cloudflare automatically decides whether interaction is needed based on request risk. - Non-interactive Shows a loading-style challenge without user input. - Invisible Runs entirely in the background with no UI. - Site Key – used on the client side (frontend) - Secret Key – used on the server side for verification - The script loads asynchronously to avoid blocking page load. - Turnstile automatically runs risk analysis in the background. - Legitimate users usually see no visible challenge. - Bots or suspicious traffic are blocked or challenged. - Navigate to Security → Security rules. - Click Create rule → Custom rules. - Define matching conditions (e.g., URI Path equals / or /api/*). - Set the Action to Managed Challenge. - Choose rule order (usually First for critical paths). - Click Deploy. - Use View analytics on the widget to track human vs bot traffic. - Monitor challenge success rates and request patterns. - Adjust rules or widget mode if necessary. - Cloudflare’s analytics help fine-tune protection without degrading user experience.