Can Claude Scrape Websites? Methods, Access, and Checks

Claude can read web content in several ways depending on which tools you give it access to. It can fetch and read a page by its URL through a fetch tool, control a browser to navigate and extract content from dynamic sites, or run code that calls a structured data API or parses HTML. Each approach has different access requirements and different risks.

Before you set any of this up, check the site's terms of service and robots.txt. Some sites prohibit automated access. Some expose an official API specifically for the data you need, which is more reliable and less likely to break. Tools like fenbs are useful for keeping the source links, research questions, and reviewed findings from a scraping project in one place that carries across sessions.

Three ways Claude reads web content

The method you choose depends on what you need. A static article or a product page with content in the initial HTML is easy to read with a fetch tool. A single-page application that loads data after the initial response needs a browser. Structured data from a site that publishes an API is best retrieved through that API.

• Fetch tools: request a URL and return the page text or HTML. Fast, simple, and works well for static pages.
• Browser automation: open a real browser, navigate, click and wait for dynamic content to render. More reliable for JavaScript-heavy sites.
• Structured extraction: call an official API, parse a sitemap, or use a data service that provides clean structured output.

For what Claude is and how tool use works in general, the main overview covers the basics.

Fetch tools: reading a page from its URL

A fetch tool is the simplest option. When Claude has one available, it can request a URL and receive the text content of the response. MCP servers such as Fetch provide this capability: you install the server, connect it to your Claude environment from Integrations, and Claude can then call it with a URL.

Fetch tools work well for reading documentation, news articles, product listings, or any page whose important content is in the initial HTML. They do not render JavaScript, so a page that loads its main content through a script after the initial response will appear empty or incomplete.

Keep a record of every URL you read. Storing source links in fenbs alongside the findings they support means the next session can verify a fact without re-fetching the page, and you can trace each conclusion back to its source.

Browser automation for dynamic pages

For sites that render content through JavaScript, a browser automation approach works better. Claude can drive a browser through a computer use API or through a browser automation MCP server. The browser opens the page, executes the scripts, and Claude can then read the rendered content, click navigation elements, or interact with forms.

Browser automation is slower than a fetch request and uses more resources. It is also more likely to trigger bot-detection measures on sites that actively try to prevent automated access. Use it when a fetch tool returns incomplete content and there is no official API available.

Anthropic's computer use API lets Claude control a browser on a desktop environment. Check Anthropic's documentation↗ (opens in a new tab) for the current capabilities and platform requirements before building a workflow around it.

Structured extraction with an official API

When a site publishes an API for its data — a news outlet, an e-commerce platform, a social network, a government data portal — use the API instead of scraping the HTML. The API gives you clean, structured output that is less likely to break when the site redesigns its pages, and it is explicitly permitted by the publisher.

Claude can call an API through a tool that wraps the HTTP request, or you can write a short script and ask Claude to run it. Either way, the API is the more stable and more respectful approach when one is available.

Check the site's terms before you scrape

Most sites have a robots.txt file at their root that lists which paths automated agents may and may not access. Checking it takes seconds and tells you the publisher's intent. Their terms of service may say more about automated access, commercial use of the data, or re-publication.

Ignoring those rules is not just a legal question. It is also a practical one: sites that detect automated access may block your IP, throttle your requests, or serve misleading content. Working with a site's permitted access paths is more reliable than working around its defenses.

Save sources and findings reliably

A scraping workflow that does not save its sources is hard to verify and hard to continue. When Claude reads a page and pulls out facts, ask it to record the URL, the date it was read, and the key finding in a task note or context note. If a later session needs to re-check the fact, the source is already on record.

For research that spans multiple sessions, keep a list of questions you started with, the sources you read, and what you concluded from each. That record lets you hand the project to a different assistant or return to it yourself without re-reading every page.

How to do this with fenbs

1. Create a free account and set up a project for your research or data-collection task.
2. Add your research questions as tasks in To Do, so the assistant knows what it is trying to find.
3. Connect your AI assistant over MCP from Integrations, and also configure whichever fetch or browser tool fits your target sites.
4. After reading each source, ask Claude to add the URL and the key finding to the task notes or context notes in your project.
5. Mark each research question as complete or add a follow-up task when the answer is not yet settled, so Where we left off reflects the current state.

fenbs does not fetch or scrape pages itself. It keeps the sources, findings, and tasks that make a multi-session research project manageable. For related questions, see the guides on whether Claude can look at websites and whether Claude can make websites.