---
url: /orchestration/recipes/web-scrape.md
---

> Documentation index: https://developer.civitai.com/llms.txt — every page on this site has a markdown version at its URL + `.md`.

# Web scrape

Fetch and render a single web page and get its content back as LLM-ready markdown (and/or raw HTML and links). The orchestrator exposes this via the `webScrape` step, backed by a self-hosted headless-browser scraper, so JavaScript-rendered pages work too.

::: warning
Swap `https://example.com` for the page you actually want to scrape.
:::

## The request shape

Every request is a single `webScrape` step on [`SubmitWorkflow`](https://developer.civitai.com/orchestration/reference/operations/SubmitWorkflow.md):

```json
{
  "$type": "webScrape",
  "input": {
    "url": "https://example.com",
    "formats": ["markdown"]
  }
}
```

## Operations

### Scrape

```http
POST https://orchestration.civitai.com/v2/consumer/workflows?wait=30
Authorization: Bearer <your-token>
Content-Type: application/json

{
  "steps": [{
    "$type": "webScrape",
    "input": {
      "url": "https://example.com",
      "formats": ["markdown", "links"]
    }
  }]
}
```

#### Parameters

| Field | Default | Allowed | Notes |
|---|---|---|---|
| `url` | — ✅ | absolute http(s) URL | The page to scrape. |
| `formats` | `["markdown"]` | `markdown`, `html`, `links` | Which content representations to return. |

## Reading the result

```json
{
  "status": "succeeded",
  "steps": [{
    "name": "0",
    "$type": "webScrape",
    "status": "succeeded",
    "output": {
      "markdown": "Example Domain\n==============\n\nThis domain is for use in...",
      "html": null,
      "links": ["https://iana.org/domains/example"],
      "title": "Example Domain",
      "description": null,
      "statusCode": 200
    }
  }]
}
```

Only the requested `formats` are populated. Very large pages are truncated to keep outputs bounded. Identical scrapes are cached for a while.

To scrape many URLs from a search, combine with [webSearch](https://developer.civitai.com/orchestration/recipes/web-search.md) and a `repeat` step:

```json
{
  "steps": [
    {
      "$type": "webSearch",
      "name": "search",
      "input": { "query": "civitai lora training guide", "limit": 3 }
    },
    {
      "$type": "repeat",
      "input": {
        "for": { "$ref": "search", "path": "output.results", "as": "result" },
        "template": {
          "$type": "webScrape",
          "name": "page",
          "input": { "url": { "$ref": "result", "path": "url" } }
        }
      }
    }
  ]
}
```

## Runtime

| Variant | Per-call wall time | `wait` recommendation |
|---|---|---|
| Static page | 1–5 s | `wait=30` |
| JavaScript-heavy page | 5–20 s | `wait=30` |

For anything that may outlast your HTTP client's timeout, submit with `wait=0` and use webhooks — see [Results & webhooks](https://developer.civitai.com/orchestration/guide/results-and-webhooks.md).

## Cost

Preview the exact charge with `whatif=true` ([Payments (Buzz)](https://developer.civitai.com/orchestration/guide/submitting-work.md#payments-buzz)).

| Operation | Buzz |
|---|---|
| Scrape | 1 |

## Troubleshooting

| Symptom | Likely cause | Fix |
|---|---|---|
| Step `failed` with an upstream error | The site blocked the request or timed out | Heavily bot-protected sites can't be scraped by the self-hosted stack — there is no stealth/anti-bot layer. |
| `markdown` empty but `statusCode: 200` | Page renders entirely client-side after load events | Retry; some pages need a moment — or request `html` and parse yourself. |
| `400` on submit | `url` not an absolute http(s) URL | Include the scheme, e.g. `https://`. |

## Related

* [`SubmitWorkflow`](https://developer.civitai.com/orchestration/reference/operations/SubmitWorkflow.md) · [`GetWorkflow`](https://developer.civitai.com/orchestration/reference/operations/GetWorkflow.md)
* [Web search](https://developer.civitai.com/orchestration/recipes/web-search.md) — find pages first, optionally scraping every result
* [Chat completion](https://developer.civitai.com/orchestration/recipes/chat-completion.md#server-executed-web-search) — let the model fetch pages itself via `civitai:web_search`
* Full parameter catalog: the `WebScrapeInput` schema in the [API reference](https://developer.civitai.com/orchestration/reference.md)