Skip to content
MCP server

Web Content & URL Tools

Pay-per-call MCP tools to read web pages, audit SEO and check domain, email and SSO setup

Install
Pricing
Paid
Platforms
API, Web
Runs
Remotely
Version
1.0.0
Listing updated
2 days ago

About Web Content & URL Tools

Web Content & URL Tools is a paid MCP server for web search and scraping. It runs as a hosted remote server, so there is nothing to install locally.

Built on Apify, this remote server bundles dozens of deterministic web checks behind one connection and bills per successful call. The registry entry says 26 tools while the Apify page counts 32 to 38, so expect the exact number to move.

The tools fall into groups: page reading (to_markdown, get_metadata, extract_structure, detect_technologies), network and security checks (trace_redirects, check_ssl, grade_security_headers, lookup_domain), crawl helpers (parse_robots, parse_sitemap, read_feed, check_broken_links), and diagnostics for email authentication and single sign-on (check_email_domain, analyze_saml_metadata, check_oidc_provider). Composite tools combine several checks, for example audit_competitor_website, verify_domain_security, audit_ai_search_readiness and check_url_phishing_risk. There are also developer utilities such as cron and regex helpers and Hyperliquid perpetuals market data.

Prices are per call, from $0.001 for simple checks to $0.003 for page tools and $0.015 for batch conversion. You connect either with an Apify API token as a bearer header at the Apify standby URL, or through Apify's OAuth gateway at mcp.apify.com.

Install

Run in a terminal

claude mcp add --transport http web-content-url-tools https://avalonai--web-content-url-tools-mcp.apify.actor/mcp

Add to Settings → Connectors → Add custom connector

https://avalonai--web-content-url-tools-mcp.apify.actor/mcp
Install in CursorOpens Cursor, which asks you to confirm. Or add it by hand:

Add to ~/.cursor/mcp.json

{
  "mcpServers": {
    "web-content-url-tools": {
      "url": "https://avalonai--web-content-url-tools-mcp.apify.actor/mcp"
    }
  }
}
Install in VS CodeOpens VS Code, which asks you to confirm. Or add it by hand:

Add to .vscode/mcp.json

{
  "servers": {
    "web-content-url-tools": {
      "type": "http",
      "url": "https://avalonai--web-content-url-tools-mcp.apify.actor/mcp"
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "web-content-url-tools": {
      "serverUrl": "https://avalonai--web-content-url-tools-mcp.apify.actor/mcp"
    }
  }
}

Add to your client's MCP settings (most use this shape)

{
  "mcpServers": {
    "web-content-url-tools": {
      "type": "http",
      "url": "https://avalonai--web-content-url-tools-mcp.apify.actor/mcp"
    }
  }
}

Tools this server provides (24)

Taken from the project documentation; the running server may differ.

to_markdown
Clean Markdown of a page's main content with word counts and language.
get_metadata
Title, description, canonical URL, Open Graph and Twitter tags.
extract_structure
Heading outline, links, images, tables and JSON-LD.
detect_technologies
CMS, framework, CDN, analytics and payment tools.
trace_redirects
Every redirect hop to the final URL.
check_ssl
Certificate issuer, expiry and trust.
grade_security_headers
A+ to F grade for security headers and cookies.
parse_robots
robots.txt rules per user agent.
parse_sitemap
URLs from a sitemap or index.
read_feed
RSS, Atom or RDF feed as JSON.
lookup_domain
WHOIS/RDAP, DNS records and domain age.
audit_seo
On-page SEO score with prioritized fixes.
check_broken_links
Checks up to 100 links.
check_email_domain
Grades MX, SPF, DMARC, DKIM, MTA-STS, TLS-RPT and BIMI.
analyze_saml_metadata
Inspects SAML metadata.
check_oidc_provider
Checks OpenID Connect discovery and JWKS.
audit_competitor_website
One-stop SEO, tech stack and metadata audit.
verify_domain_security
TLS, security headers and email authentication check.
audit_ai_search_readiness
Checks AI crawler permissions, llms.txt and JSON-LD.
extract_company_leads
Extracts business emails, phones and social profiles.
check_url_phishing_risk
Scores phishing risk from 0 to 100.
audit_prompt_injection_risk
Detects indirect prompt injection and hidden text.
batch_to_markdown
Converts up to 10 pages in parallel.
crawl_and_map_site
Bundles up to 10 pages from a sitemap or internal links.

What it does

Pages to Markdown
Convert a URL, a batch of up to 10 pages or a sitemap crawl into LLM-ready Markdown.
SEO and AI-search audits
Score on-page SEO and check robots.txt rules for AI crawlers, llms.txt and JSON-LD.
Domain and TLS checks
Inspect certificates, security-header grades, WHOIS/RDAP and DNS records.
Email authentication
Grade SPF, DMARC, DKIM, MTA-STS, TLS-RPT and BIMI with fixes.
SSO diagnostics
Analyze SAML metadata and OpenID Connect discovery and JWKS.
Prompt-injection scan
Detect hidden text and instruction overrides on a page before it reaches an LLM.

Good for

  • Convert documentation pages to Markdown for a RAG pipeline
  • Audit a competitor site's SEO and tech stack
  • Check whether a link looks like phishing
  • Debug SPF and DMARC for a sending domain
  • Verify a SAML or OIDC provider's configuration

Pricing

Paid

Pay per call from $0.001; Apify free plan includes $5 of monthly usage

Prices change. Check apify.com before you buy.

Web Content & URL Tools FAQ

How is it billed?

Per successful call: $0.001 for checks like redirects and SSL, $0.003 for page tools, $0.015 for batch_to_markdown, on top of an Apify account.

How do I connect?

Use the Apify standby URL with your Apify API token as a Bearer header, or the OAuth route via mcp.apify.com.

Does it use a headless browser?

The README says no browser is involved.

How many tools are there?

The registry entry lists 26; the Apify listing mentions 32 to 38 as tools are added.

Which clients work?

Claude Desktop, Claude Code, Cursor, VS Code, Windsurf, n8n and any client that supports remote servers with headers.

Is Web Content & URL Tools free?

No. Web Content & URL Tools is a paid product.

How do I add Web Content & URL Tools to Claude Code?

Run this command in your terminal: claude mcp add --transport http web-content-url-tools https://avalonai--web-content-url-tools-mcp.apify.actor/mcp

What are the best alternatives to Web Content & URL Tools?

The closest alternatives to Web Content & URL Tools are REA, Scrapling MCP Server, Firecrawl MCP Server, Google Ads + Meta Ads + SEO/GEO MCP and SearXNG Search.

Alternatives to Web Content & URL Tools

  1. REA

    MCP server

    CLI and MCP server that lets coding agents reverse engineer apps and binaries

    Runs locally
    • Open source
    • 464K GitHub stars
    • Developer tools
  2. MCP server for AI clients to scrape sites with HTTP and stealth browser sessions

    Runs locally
    • 17 tools
    • Open source
    • 104K GitHub stars
    • Web search and scraping
  3. MCP server that lets AI agents search, scrape and interact with web pages

    • 3 tools
    • Remote
    • Freemium
    • 7.8K GitHub stars
    • Web search and scraping
  4. Hosted MCP connector for managing Google Ads, Meta Ads, GA4 and Search Console from chat

    • 8 tools
    • Remote
    • Freemium
    • 8.1K GitHub stars
    • Marketing and SEO
  5. SearXNG Search

    MCP server

    MCP server for web search and page reading through a SearXNG instance

    Runs locally
    • 3 tools
    • Open source
    • 1.4K GitHub stars
    • Web search and scraping
  6. Agent Roast

    MCP server

    Offline review that flags agent changes which weaken tests, assertions or CI

    Runs locally
    • Open source
    • 4 Hacker News points
    • Developer tools