Skip to main content
POST
Create Crawl Job

Headers

x-api-key
string | null

Your API key for programmatic access. Send it in the x-api-key header; you can create and manage keys from the Developers page.

Body

application/json
url
string
required

URL to a page of documentation that the agent will have access to in order to interact with users.

max_depth
integer
default:3

Maximum depth for crawling (1-5), defaults to 3.

Required range: 1 <= x <= 5
max_pages
integer
default:1000

Maximum number of pages to crawl (1-10,000), defaults to 1000.

Required range: 1 <= x <= 10000
pattern
string | null

If set, only URLs that match this pattern are included.

Maximum string length: 2048
sitemap_urls
string[] | null

List of URLs to crawl from sitemap (optional, overrides automatic URL discovery).

parent_folder_id
string | null

If set, the created document or folder will be placed inside the given folder.

enable_auto_sync
boolean
default:false

Whether to enable auto-sync for this URL document.

auto_remove
boolean
default:false

Whether to automatically remove the document if the URL becomes unavailable. Only applicable when auto-sync is enabled.

minimum_frequency_days
integer | null

Minimum frequency (in days) at which the underlying eligible documents are refreshed. The actual interval may be shorter, never longer. Defaults to 7, tightened to the parent folder's frequency if that is stricter. Only applicable when auto-sync is enabled.

Required range: 1 <= x <= 180

Response

Successful Response

id
string
required
type
enum<string>
default:discovery
required
Available options:
discovery,
sitemap
root_folder_id
string
required
status
string
required
created_at
integer
required
folder_path
KnowledgeBaseFolderPathSegmentSummaryResponseModel · object[]

The folder path segments leading to the root folder, from root to parent folder.