An llms.txt generator that points AI to your best pages.
SignPost is an llms.txt generator for Claude. It writes the llms.txt file that tells AI assistants what a site is and which pages matter, a quick win for generative engine optimization. A collector script reads robots.txt and the sitemaps, visits the pages politely, and records each page’s title, description, heading and indexability. Claude then picks the pages worth reading, writes one factual line for each in the llmstxt.org format, and the script rechecks every link.
01See it work
SignPost, in real use.
3 screens from real runs of SignPost. Every value shown is what the tool returned; click a screen to open it full size.
02How it works
Read the site politely, then choose what matters.
A collector script gathers the facts; Claude does the choosing and the writing; the script checks the result.
- 01Read the maprobots.txt and every sitemap, following sitemap indexes.
collect.py https://example.com - 02Visit pagesOne at a time with a pause, honouring robots.txt: title, description, H1, canonical, noindex.
- 03Write llms.txtClaude picks the pages worth reading and writes one factual line for each.
- 04Check every linkStatus and redirects for each link, and the file's structure.
collect.py --check llms.txt
03The basics
What is llms.txt, and how it differs from robots.txt.
llms.txt was proposed in September 2024 as a short Markdown map of a site for AI systems. It is a proposal, not a standard, and AI companies have not said they rank by it. It is cheap to publish and useful to any assistant that reads it.
| robots.txt | sitemap.xml | llms.txt | |
|---|---|---|---|
| Says | What crawlers may fetch | Every page that exists | Which pages matter, and what each is |
| Format | Plain text rules | XML | Markdown |
| Read by | Every well-behaved crawler | Search engines | AI assistants and tools that choose to |
| Size | A few lines | Every URL, often thousands | Usually 20 to 80 links |
| Overrides the others? | Yes: a blocked page stays blocked | No | No: it never lets a blocked crawler in |
04The format
The llms.txt format, line by line.
The format is small. SignPost writes it exactly and checks the structure when it rechecks the links.
| Part | Looks like | Rule |
|---|---|---|
| Title | # Example Outdoors | Required. One H1, the name of the site or organization. |
| Summary | > Example Outdoors sells hiking gear ... | A blockquote with one or two sentences of fact. |
| Notes | Plain paragraphs | Optional context, such as who the site is for. |
| Sections | ## Products | H2 headings that group the links. |
| Links | - [Pricing](https://example.com/pricing): Plans and what each includes. | One per line: a name, the URL, then a short factual note. |
| Optional | ## Optional | Links an assistant can skip when it is short of room. |
05An example
An llms.txt example, checked link by link.
The draft SignPost wrote for dudelemon.com from ten collected pages, and the result of checking it. All seven links answered 200 without a redirect.
# Dude Lemon
> Dude Lemon designs, builds and runs custom software, and makes its own products
> for customer service, reviews, product recalls and online selling.
## Products
- [RecallPilot](https://dudelemon.com/product-recall-portal-development): Product recall software: consumer recall websites, portals and recall management.
- [ConvertPilot](https://dudelemon.com/convertpilot): AI agent, chatbot and unified inbox for website chat, email, WhatsApp and social messages.
- [ConvertPilot for Shopify](https://dudelemon.com/convertpilot-shopify-app): The ConvertPilot AI agent and live chat, installed as a Shopify app.
- [ReviewMankey](https://dudelemon.com/reviewmankey): Review management that brings reviews from Google and other sites into one place.
- [Lakaz Fresh](https://dudelemon.com/lakazfresh): A multi-vendor marketplace designed, built and operated by Dude Lemon.
## Guides
- [Software buyer guides](https://dudelemon.com/guides): How to choose customer support, AI search visibility, donation and compliance software.
## Optional
- [Home](https://dudelemon.com/): Services, products and how a project starts.06Install
Install SignPost.
Install SignPost in Claude Code from the Dude Lemon marketplace, or download it for the Claude apps. Its script needs Python 3.8+ and internet access.
/plugin marketplace add https://plugins.dudelemon.cloud/marketplace.json/plugin install signpost@dudelemonClaude uses the skill when a task matches its description. You can also call it directly with /signpost:signpost.
07Inside
What is in the skill folder.
SKILL.md- How to choose pages, the exact llms.txt structure, writing rules for descriptions, and where the file goes.
scripts/collect.py- Reads robots.txt and sitemap indexes, visits up to a set number of pages one at a time, honours robots.txt, and outputs JSON.
collect.py --check- Rechecks every link in a finished llms.txt for status and redirects, and checks the file’s structure.
08Ask in your own words
Say what you need. SignPost does the exact part.
Claude reaches for SignPost when a request matches it. You can also ask for it by name.
- Write an llms.txt for our site.
- Is our llms.txt still accurate? Check every link.
- Which of our pages should AI assistants read first?
09Works well with
Related tools.
FAQ
SignPost, answered
Anything we have not answered here, ask us directly. A senior engineer replies.
What is llms.txt?
A plain Markdown file at the root of a site that gives AI systems a short, curated map: what the organisation is, and links to the pages worth reading with one line on each. It complements robots.txt and the sitemap; it does not replace them.
Does llms.txt override robots.txt?
No. If robots.txt blocks a crawler, llms.txt does not let it in. The skill reminds you to check that the AI search crawlers you want are allowed.
How many pages should it list?
Usually 20 to 80. Products, pricing, documentation, key policies and the strongest evergreen articles, with secondary material under an Optional section.
Is SignPost free?
Yes. SignPost is free and MIT licensed.
How do I install SignPost?
In Claude Code run /plugin marketplace add https://plugins.dudelemon.cloud/marketplace.json, then /plugin install signpost@dudelemon. In the Claude apps, download https://plugins.dudelemon.cloud/skills/signpost.zip and upload it under Customize, Skills.
Should llms.txt list every page?
No. That is what the sitemap is for. llms.txt is a curated short list: products, pricing, documentation, key policies and the strongest evergreen pages, with secondary material under Optional.