Crawl a website to markdown
Reads a website breadth-first from url, following its own links and the URLs in its sitemap, and returns every page as clean markdown (or text / HTML). By default the crawl stays under the start URL's folder (scope=site for the whole site); include / exclude narrow it further. Pages the site's robots.txt disallows are not read. Each item of pages has url, final_url, depth, found_in (start, link or sitemap), HTTP status, title, description, language, author, published date, the content under the key named by format, word_count, links_count and internal_links_count; a page that could not be read has error. totals gives the counts (pages read and failed, URLs discovered, queued and skipped by reason) and skipped lists up to 100 skipped URLs with the reason (robots, filtered, not_html, out_of_scope, depth). One call reads up to 50 pages within about 75 seconds; when time runs out first, truncated is true and next_urls holds the URLs still queued (crawl them with further calls). Content is capped at 100,000 characters per page. Every response includes credits_used: the credits this call cost. Cost: 1 credit per 2 pages of max_pages (default 10 = 5 credits).
Parameters
| name | type | description |
|---|---|---|
urlrequired | string | Start URL (a bare domain works too). Example: https://docs.python.org/3/tutorial/ |
max_pages | integer | Pages to read at most (1-50). Example: 5 Default: 10 |
max_depth | integer | Link depth from the start URL (0 = the start page only; sitemap URLs count as depth 1). Default: 20 |
include | string | Only read pages whose path matches: comma-separated globs (* = any characters except /, ** = anything), or one regular expression between slashes. |
exclude | string | Skip pages whose path matches (same syntax as include). |
scope | path | site | path: only pages under the start URL's folder; site: the whole site. Default: path |
use_sitemap | boolean | Also queue the URLs listed in the site's sitemap. Default: true |
format | markdown | text | html | Format of each page's content (the content key has the same name). Default: markdown |
country | string | Two-letter country to read the site from. Default: US |
Every endpoint also takes fresh, raw, fields (see lean responses and caching).
Example
curl "https://api.api.sh/web/v1/crawl?url=https%3A%2F%2Fdocs.python.org%2F3%2Ftutorial%2F" \
-H "x-api-key: $APISH_KEY" import os, requests
res = requests.get(
"https://api.api.sh/web/v1/crawl",
params={"url": "https://docs.python.org/3/tutorial/"},
headers={"x-api-key": os.environ["APISH_KEY"]},
timeout=120,
)
res.raise_for_status()
data = res.json() const url = new URL("https://api.api.sh/web/v1/crawl");
url.searchParams.set("url", "https://docs.python.org/3/tutorial/");
const res = await fetch(url, { headers: { "x-api-key": process.env.APISH_KEY } });
if (!res.ok) throw new Error(`${res.status} ${await res.text()}`);
const data = await res.json(); Response
A JSON object with the fields described above, plus raw (the source's own data), cached, fetched_at and credits_used. Errors return {"success": false, "error": "…"} with the HTTP status (see errors). A lookup that ran but found nothing (404) costs the price above; refused calls and errors on our side are free (what's charged).