POST /accounts/{account_id}/browser-rendering/crawl

Account ID.

Default value: "application/json"

Cache TTL default is 5s. Set to 0 to disable.

Default value: 5

Whether to render the page or fetch static content. True by default.

Valid values:

• false

Source of links to crawl. 'sitemaps' - only crawl URLs from sitemaps, 'links' - only crawl URLs scraped from pages, 'all' - crawl both sitemap and scraped links (default).

Default value: "all"

Unix timestamp (seconds since epoch) indicating to only crawl pages that were modified since this time. For sitemap URLs with a lastmod field, this is compared directly. For other URLs, the crawler will use If-Modified-Since header when fetching. URLs without modification information (no lastmod in sitemap and no Last-Modified header support) will be crawled. Note: This works in conjunction with maxAge - both filters must pass for a cached resource to be used. Must be within the last year and not in the future.

Check options.

Cookie name.

Maximum number of levels deep the crawler will traverse from the starting URL.

Default value: 100000

Check options.

Default value: { "width": 1920, "height": 1080 }

Waits for a specified timeout before continuing.

Adds a <script> tag into the page with the desired URL or content.

Formats to return. Default is html.

Default value: [ "html" ]

Check options.

Default value: 30000

Default value: "domcontentloaded"

Maximum number of URLs to crawl.

Default value: 10

Additional options for the crawler.

Default value: { "includeSubdomains": false, "includeExternalLinks": false }

Include external links in the crawl job. If set to true, includeSubdomains is ignored.

Default value: false

Include links to subdomains in the crawl job. This option is ignored if includeExternalLinks is true.

Default value: false

Exclude links matching the provided wildcard patterns in the crawl job. Example: 'https://example.com/privacy/**'.

Include only links matching the provided wildcard patterns in the crawl job. Include patterns are evaluated before exclude patterns. URLs that match any of the specified include patterns will be included in the crawl job. Example: 'https://example.com/blog/**'.

Options for JSON extraction.

Optional list of custom AI models to use for the request. The models will be tried in the order provided, and in case a model returns an error, the next one will be used as fallback.

Authorization token for the AI model: Bearer <token>. Not needed for workers-ai models.

AI model to use for the request. Must be formed as <provider>/<model_name>, e.g. workers-ai/@cf/meta/llama-3.3-70b-instruct-fp8-fast.

Schema for the response format. More information here: https://developers.cloudflare.com/workers-ai/json-mode/

Adds a <link rel="stylesheet"> tag into the page with the desired URL or a <style type="text/css"> tag with the content.

Provide credentials for HTTP authentication.

Only allow requests that match the provided regex patterns, eg. '/^.*.(css)'.

URL to navigate to, eg. https://example.com.

Only allow requests that match the provided resource types, eg. 'image' or 'script'.

The maximum duration allowed for the browser action to complete after the page has loaded (such as taking screenshots, extracting content, or generating PDFs). If this time limit is exceeded, the action stops and returns a timeout error.

Block undesired requests that match the provided resource types, eg. 'image' or 'script'.

Wait for the selector to appear in page. Check options.

Valid values:

• true

Valid values:

• true

Attempt to proceed when 'awaited' events fail or timeout.

Block undesired requests that match the provided regex patterns, eg. '/^.*.(css)'.

List of crawl purposes to respect Content-Signal directives in robots.txt. Allowed values: 'search', 'ai-input', 'ai-train'. Learn more: https://contentsignals.org/. Default: ['search', 'ai-input', 'ai-train'].

Default value: [ "search", "ai-input", "ai-train" ]

Maximum age of a resource that can be returned from cache in seconds. Default is 1 day.

Default value: 86400