Web Content & URL Tools
Pay-per-call MCP tools to read web pages, audit SEO and check domain, email and SSO setup
- Pricing
- Paid
- Platforms
- API, Web
- Runs
- Remotely
- Version
- 1.0.0
- Listing updated
- 2 days ago
About Web Content & URL Tools
Web Content & URL Tools is a paid MCP server for web search and scraping. It runs as a hosted remote server, so there is nothing to install locally.
Built on Apify, this remote server bundles dozens of deterministic web checks behind one connection and bills per successful call. The registry entry says 26 tools while the Apify page counts 32 to 38, so expect the exact number to move.
The tools fall into groups: page reading (to_markdown, get_metadata, extract_structure, detect_technologies), network and security checks (trace_redirects, check_ssl, grade_security_headers, lookup_domain), crawl helpers (parse_robots, parse_sitemap, read_feed, check_broken_links), and diagnostics for email authentication and single sign-on (check_email_domain, analyze_saml_metadata, check_oidc_provider). Composite tools combine several checks, for example audit_competitor_website, verify_domain_security, audit_ai_search_readiness and check_url_phishing_risk. There are also developer utilities such as cron and regex helpers and Hyperliquid perpetuals market data.
Prices are per call, from $0.001 for simple checks to $0.003 for page tools and $0.015 for batch conversion. You connect either with an Apify API token as a bearer header at the Apify standby URL, or through Apify's OAuth gateway at mcp.apify.com.
Install
Run in a terminal
claude mcp add --transport http web-content-url-tools https://avalonai--web-content-url-tools-mcp.apify.actor/mcpAdd to Settings → Connectors → Add custom connector
https://avalonai--web-content-url-tools-mcp.apify.actor/mcpAdd to ~/.cursor/mcp.json
{
"mcpServers": {
"web-content-url-tools": {
"url": "https://avalonai--web-content-url-tools-mcp.apify.actor/mcp"
}
}
}Add to .vscode/mcp.json
{
"servers": {
"web-content-url-tools": {
"type": "http",
"url": "https://avalonai--web-content-url-tools-mcp.apify.actor/mcp"
}
}
}Add to ~/.codeium/windsurf/mcp_config.json
{
"mcpServers": {
"web-content-url-tools": {
"serverUrl": "https://avalonai--web-content-url-tools-mcp.apify.actor/mcp"
}
}
}Add to your client's MCP settings (most use this shape)
{
"mcpServers": {
"web-content-url-tools": {
"type": "http",
"url": "https://avalonai--web-content-url-tools-mcp.apify.actor/mcp"
}
}
}Tools this server provides (24)
Taken from the project documentation; the running server may differ.
- to_markdown
- Clean Markdown of a page's main content with word counts and language.
- get_metadata
- Title, description, canonical URL, Open Graph and Twitter tags.
- extract_structure
- Heading outline, links, images, tables and JSON-LD.
- detect_technologies
- CMS, framework, CDN, analytics and payment tools.
- trace_redirects
- Every redirect hop to the final URL.
- check_ssl
- Certificate issuer, expiry and trust.
- grade_security_headers
- A+ to F grade for security headers and cookies.
- parse_robots
- robots.txt rules per user agent.
- parse_sitemap
- URLs from a sitemap or index.
- read_feed
- RSS, Atom or RDF feed as JSON.
- lookup_domain
- WHOIS/RDAP, DNS records and domain age.
- audit_seo
- On-page SEO score with prioritized fixes.
- check_broken_links
- Checks up to 100 links.
- check_email_domain
- Grades MX, SPF, DMARC, DKIM, MTA-STS, TLS-RPT and BIMI.
- analyze_saml_metadata
- Inspects SAML metadata.
- check_oidc_provider
- Checks OpenID Connect discovery and JWKS.
- audit_competitor_website
- One-stop SEO, tech stack and metadata audit.
- verify_domain_security
- TLS, security headers and email authentication check.
- audit_ai_search_readiness
- Checks AI crawler permissions, llms.txt and JSON-LD.
- extract_company_leads
- Extracts business emails, phones and social profiles.
- check_url_phishing_risk
- Scores phishing risk from 0 to 100.
- audit_prompt_injection_risk
- Detects indirect prompt injection and hidden text.
- batch_to_markdown
- Converts up to 10 pages in parallel.
- crawl_and_map_site
- Bundles up to 10 pages from a sitemap or internal links.
What it does
- Pages to Markdown
- Convert a URL, a batch of up to 10 pages or a sitemap crawl into LLM-ready Markdown.
- SEO and AI-search audits
- Score on-page SEO and check robots.txt rules for AI crawlers, llms.txt and JSON-LD.
- Domain and TLS checks
- Inspect certificates, security-header grades, WHOIS/RDAP and DNS records.
- Email authentication
- Grade SPF, DMARC, DKIM, MTA-STS, TLS-RPT and BIMI with fixes.
- SSO diagnostics
- Analyze SAML metadata and OpenID Connect discovery and JWKS.
- Prompt-injection scan
- Detect hidden text and instruction overrides on a page before it reaches an LLM.
Good for
- Convert documentation pages to Markdown for a RAG pipeline
- Audit a competitor site's SEO and tech stack
- Check whether a link looks like phishing
- Debug SPF and DMARC for a sending domain
- Verify a SAML or OIDC provider's configuration
Pricing
Paid
Pay per call from $0.001; Apify free plan includes $5 of monthly usage
Prices change. Check apify.com before you buy.
Web Content & URL Tools FAQ
How is it billed?
Per successful call: $0.001 for checks like redirects and SSL, $0.003 for page tools, $0.015 for batch_to_markdown, on top of an Apify account.
How do I connect?
Use the Apify standby URL with your Apify API token as a Bearer header, or the OAuth route via mcp.apify.com.
Does it use a headless browser?
The README says no browser is involved.
How many tools are there?
The registry entry lists 26; the Apify listing mentions 32 to 38 as tools are added.
Which clients work?
Claude Desktop, Claude Code, Cursor, VS Code, Windsurf, n8n and any client that supports remote servers with headers.
Is Web Content & URL Tools free?
No. Web Content & URL Tools is a paid product.
How do I add Web Content & URL Tools to Claude Code?
Run this command in your terminal: claude mcp add --transport http web-content-url-tools https://avalonai--web-content-url-tools-mcp.apify.actor/mcp
What are the best alternatives to Web Content & URL Tools?
The closest alternatives to Web Content & URL Tools are REA, Scrapling MCP Server, Firecrawl MCP Server, Google Ads + Meta Ads + SEO/GEO MCP and SearXNG Search.
Alternatives to Web Content & URL Tools

REA
MCP serverCLI and MCP server that lets coding agents reverse engineer apps and binaries
Runs locally- Open source
- 464K GitHub stars
- Developer tools

Scrapling MCP Server
MCP serverMCP server for AI clients to scrape sites with HTTP and stealth browser sessions
Runs locally- 17 tools
- Open source
- 104K GitHub stars
- Web search and scraping

Firecrawl MCP Server
MCP serverMCP server that lets AI agents search, scrape and interact with web pages
- 3 tools
- Remote
- Freemium
- 7.8K GitHub stars
- Web search and scraping

Google Ads + Meta Ads + SEO/GEO MCP
MCP serverHosted MCP connector for managing Google Ads, Meta Ads, GA4 and Search Console from chat
- 8 tools
- Remote
- Freemium
- 8.1K GitHub stars
- Marketing and SEO

SearXNG Search
MCP serverMCP server for web search and page reading through a SearXNG instance
Runs locally- 3 tools
- Open source
- 1.4K GitHub stars
- Web search and scraping
Agent Roast
MCP serverOffline review that flags agent changes which weaken tests, assertions or CI
Runs locally- Open source
- 4 Hacker News points
- Developer tools