Skip to content
Clarvith

ClarvithBot

HTTP requests from Clarvith Actors use ClarvithBot/1.0 (+https://clarvith.com/bot; public job-board lookup for search and reference; not for AI training) . This page lists what gets fetched and how errors are handled.

Hosts and paths

  • Documented Greenhouse, Lever, Ashby, and Workable job-board API endpoints (see Actor README).
  • Company sites for discovery: homepage and, when robots.txt allows, paths such as /careers, /jobs, /company/careers.
  • robots.txt for each host before other URLs (RFC 9309).

robots.txt

  • 2xx response: rules are followed (up to five redirects on that file).
  • 4xx: treated as no restrictions for that host.
  • 5xx, timeout, or unreachable: host is skipped for the run.
  • 429: host pauses 60 seconds and slows down; repeated 429 may disable the host.

Rate limits

Per-host pacing in the shared HTTP client:

  • Lever and Workable API hosts: 1 request per second.
  • Greenhouse and Ashby API hosts: 3 requests per second.
  • Other hosts (company sites): 1 request per second unless configured otherwise.

Concurrent fetches share these limits per host. Workable may be disabled for the rest of a run after a second 429.

Opt out

Email [email protected] with the hostname and your role to request blocking or discuss policy. robots.txt remains the primary control.