🗺️Crawling & Sitemaps
Parse XML Sitemaps via API
XML is fine. Parsing XML is annoying. Parsing nested sitemap indexes recursively is a Saturday afternoon gone. denkbot.dog handles all of this — point it at a domain or sitemap URL and get back a clean JSON array. The dog parses the XML. You use the data.
What you'd use this for
SEO tooling, content syncing, building search indexes from sitemaps, change detection via lastmod dates, and automated content discovery.
How it works
example
# Parse a sitemap directly
curl -X POST https://api.denkbot.dog/sitemap \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{ "url": "https://example.com/sitemap_index.xml" }' \
| jq '.urls[] | select(.lastmod > "2025-01-01") | .loc'Questions & Answers
Does it follow sitemap index files?+
Yes. It recursively fetches all child sitemaps.
What if the sitemap URL is wrong?+
We also try common paths like /sitemap.xml, /sitemap_index.xml before giving up.
What if the sitemap is gzipped?+
Gzip decompression is handled automatically.
Related topics
Crawl Entire Websites via API
Crawl any website and get a full nested URL tree. Up to 500 pages, respects robots.txt, stays on-domain by default.
Extract Sitemaps from Any Website
Automatically find and parse XML sitemaps. Returns all URLs with lastmod, changefreq, and priority. No XML parsing required.
Scrape SEO Data from Any Website
Extract title tags, meta descriptions, canonical URLs, heading structure, and all SEO-relevant data from any URL via API.
Fetch and Parse robots.txt via API
Fetch the robots.txt of any domain and understand crawling rules. denkbot.dog respects robots.txt by default.
Ready to start fetching?
€19/year. Unlimited requests. API key ready in 30 seconds.