Add Web Scraping to Your OpenAI Agent

OpenAI function calling lets your model decide when to fetch a URL. Define a scrape function, hand it to your model, and GPT-4 will call it whenever it needs live web data. denkbot.dog returns structured JSON with title, text, links, and metadata — exactly what a language model needs to reason about a page.

Get API Key →Read the Docs

What you'd use this for

GPT-4 research assistants, o1 agents that browse docs to answer questions, custom ChatGPT-like tools that need live web access, Assistants API agents with browsing capabilities.

How it works

example

import openai, httpx

def scrape(url: str, render_js: bool = False) -> dict:
    r = httpx.post("https://api.denkbot.dog/scrape",
        headers={"Authorization": f"Bearer {DENKBOT_API_KEY}"},
        json={"url": url, "renderJs": render_js, "format": "json"}, timeout=30)
    return r.json()

tools = [{
    "type": "function",
    "function": {
        "name": "scrape",
        "description": "Fetch content from a URL. Returns title, text, and links.",
        "parameters": {
            "type": "object",
            "properties": {
                "url": {"type": "string", "description": "URL to fetch"},
                "render_js": {"type": "boolean", "description": "Render JavaScript first"}
            },
            "required": ["url"]
        }
    }
}]

response = openai.chat.completions.create(
    model="gpt-4o", messages=[{"role": "user", "content": "Summarize https://example.com"}],
    tools=tools)