🗺️Crawling & Sitemaps
Get the Full URL Tree of Any Website
Understanding a website's structure means crawling it. denkbot.dog's /crawl endpoint explores a site and returns a nested URL tree — not just a flat list of URLs, but a structured hierarchy showing how pages connect. The dog maps the territory.
What you'd use this for
Content inventories, site architecture analysis, migration planning, documentation site mapping, and building navigation menus from existing content.
How it works
example
curl -X POST https://api.denkbot.dog/crawl \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://docs.example.com",
"maxPages": 100,
"maxDepth": 4
}' | jq '.tree'Questions & Answers
Is the response a flat list or a nested tree?+
Nested tree. Each page has a children array containing pages linked from it.
What if the same URL appears at multiple depths?+
We deduplicate. Each URL appears once in the tree at its shallowest discovered depth.
Does it crawl the full site automatically?+
Up to maxPages (500 max). Set maxPages high to get comprehensive coverage.
Related topics
Crawl Entire Websites via API
Crawl any website and get a full nested URL tree. Up to 500 pages, respects robots.txt, stays on-domain by default.
Extract Sitemaps from Any Website
Automatically find and parse XML sitemaps. Returns all URLs with lastmod, changefreq, and priority. No XML parsing required.
Extract All Internal Links from a Website
Extract all internal links from any webpage or crawl an entire site's link structure. Returns hrefs, anchor text, and more.
Parse XML Sitemaps via API
Parse XML sitemaps and sitemap index files automatically. Returns structured JSON with all URLs, lastmod, changefreq, and priority.
Ready to start fetching?
€19/year. Unlimited requests. API key ready in 30 seconds.