mcp-llms-txt-explorer MCP Server

Use This MCP server To

Check if a website hosts llms.txt or llms-full.txt files Validate the syntax and content of llms.txt files on websites Retrieve structured metadata about websites implementing llms.txt List and filter known websites compliant with the llms.txt standard Integrate website llms.txt data into LLM context workflows Automate discovery of new websites supporting llms.txt for AI agents

README

MCP LLMS.txt Explorer

A Model Context Protocol server for exploring websites with llms.txt files. This server helps you discover and analyze websites that implement the llms.txt standard.

Features

Resources

Check websites for llms.txt and llms-full.txt files
Parse and validate llms.txt file contents
Access structured data about compliant websites

Tools

check_website - Check if a website has llms.txt files
- Takes domain URL as input
- Returns file locations and validation status
list_websites - List known websites with llms.txt files
- Returns structured data about compliant websites
- Supports filtering by file type (llms.txt/llms-full.txt)

Development

Install dependencies:

pnpm install

Build the server:

pnpm run build

For development with auto-rebuild:

pnpm run watch

Installation

Installing via Smithery

To install mcp-llms-txt-explorer for Claude Desktop automatically via Smithery:

npx -y @smithery/cli install @thedaviddias/mcp-llms-txt-explorer --client claude

Installing Manually

To use this server:

# Clone the repository
git clone https://github.com/thedaviddias/mcp-llms-txt-explorer.git
cd mcp-llms-txt-explorer

# Install dependencies
pnpm install

# Build the server
pnpm run build

Configuration with Claude Desktop

To use with Claude Desktop, add the server config:

On MacOS: ~/Library/Application Support/Claude/claude_desktop_config.json On Windows: %APPDATA%/Claude/claude_desktop_config.json

{
  "mcpServers": {
    "llms-txt-explorer": {
      "command": "/path/to/llms-txt-explorer/build/index.js"
    }
  }
}

For npx usage, you can use:

{
  "mcpServers": {
    "llms-txt-explorer": {
      "command": "npx",
      "args": ["-y", "@thedaviddias/mcp-llms-txt-explorer"]
    }
  }
}

Debugging

Since MCP servers communicate over stdio, debugging can be challenging. We recommend using the MCP Inspector, which is available as a package script:

pnpm run inspector

The Inspector will provide a URL to access debugging tools in your browser.

License

This project is licensed under the MIT License—see the LICENSE file for details.

mcp-llms-txt-explorer FAQ

How does the mcp-llms-txt-explorer validate llms.txt files?

It parses the files and checks their syntax and compliance with the llms.txt standard to ensure correctness.

Can I filter the list of websites by the type of llms.txt file?

Yes, the server supports filtering known websites by file type such as llms.txt or llms-full.txt.

What input does the check_website tool require?

It requires a domain URL to check for the presence and validity of llms.txt files on that website.

How can this server help with LLM context integration?

By providing structured, validated data about websites' llms.txt files, it enables LLMs to access real-time context from compliant sites.

Is this server limited to any specific LLM providers?

No, it is provider-agnostic and can be used with any LLM platform like OpenAI, Claude, or Gemini.

Can this server discover new websites with llms.txt files automatically?

It lists known compliant websites and can be integrated into workflows to automate discovery and updates.

What kind of structured data does the server return?

It returns file locations, validation status, and metadata about the websites implementing the llms.txt standard.