ClarvithBot
HTTP requests from Clarvith Actors use ClarvithBot/1.0 (+https://clarvith.com/bot; public job-board lookup for search and reference; not for AI training)
. This page lists what gets fetched and how errors are handled.
Hosts and paths
- Documented Greenhouse, Lever, Ashby, and Workable job-board API endpoints (see Actor README).
-
Company sites for discovery: homepage and, when robots.txt allows, paths such as
/careers,/jobs,/company/careers. -
robots.txtfor each host before other URLs (RFC 9309).
robots.txt
- 2xx response: rules are followed (up to five redirects on that file).
- 4xx: treated as no restrictions for that host.
- 5xx, timeout, or unreachable: host is skipped for the run.
- 429: host pauses 60 seconds and slows down; repeated 429 may disable the host.
Rate limits
Per-host pacing in the shared HTTP client:
- Lever and Workable API hosts: 1 request per second.
- Greenhouse and Ashby API hosts: 3 requests per second.
- Other hosts (company sites): 1 request per second unless configured otherwise.
Concurrent fetches share these limits per host. Workable may be disabled for the rest of a run after a second 429.
Opt out
Email [email protected] with the hostname and your role to request blocking or discuss policy. robots.txt remains the primary control.