CultDevOps LLMs.txt Generator – SEO for AI

FAQ

What is llms.txt, and what does this plugin do?

llms.txt is a proposal for a Markdown file at the root of a website that gives AI agents short background about the site and links to the pages that matter. It was proposed by Jeremy Howard and is described on https://llmstxt.org/. The page says the only required part is the site name; a short summary and lists of links can follow. This plugin generates such a file from your public content and serves it as llms.txt. It is a proposal, not a rule: the page does not name the AI tools that read the file, and tools may change how they treat it.

Will llms.txt improve my search engine ranking?

There is no guarantee, and this plugin does not promise it. llmstxt.org describes the file as a help for AI agents that use a site; it does not describe an effect on search results. Treat llms.txt as an addition to a clear site structure, a sitemap and good content, not as a replacement.

Why does the llms.txt response have a noindex header?

The response carries the header X-Robots-Tag: noindex, follow. It asks search engines not to list the llms.txt file itself as a page in their results, while links inside it may still be followed. The file is meant for AI agents, not for visitors of a search engine. The header is about this one file only; it does not change how your pages are indexed.

Which content is listed in the file, and which content is never listed?

The file starts with a title and a short summary: the ones you wrote in the settings, or else the site title and the tagline from Settings > General. Then it has one section for each checked post type with the newest published items, up to the Content Limit. Each item is a link with the title and a short description, followed by the categories, tags and other public terms of the post. The last section, Optional, links to the other languages, to the WordPress sitemap when it is available, and to the RSS feed. Drafts, pending, private, scheduled and trashed posts and posts with a password are never listed, whatever the settings say. A post that Yoast SEO, Rank Math, All in One SEO or SEOPress marks as noindex is left out while that plugin is active.

Is password-protected or private content kept out of the file?

Yes, as far as this file is concerned. The query asks only for published posts without a password, and every item is checked once more before it is written. When a published post becomes private or gets a password, or is trashed or deleted, the stored file is dropped at once and regenerated, so the post does not stay in an old copy. Posts that MemberPress, Paid Memberships Pro, Restrict Content Pro, WooCommerce Memberships or Members keep for members are left out as well, checked as for a visitor who is not logged in, and the members-only blocks of Paid Memberships Pro and MemberPress are removed from the text. For another membership or paywall plugin, keep such posts out with Hide from llms.txt. Everything else that is published is public: a published post appears in the file with its title, address and a short description. Keep a post out with Hide from llms.txt on its edit screen, by setting it to noindex in your SEO plugin, by unchecking its post type, by choosing categories, or by making it private or password-protected.

How do I keep a single post or page out of llms.txt?

Open the post in the editor and check Hide from llms.txt in the llms.txt box (in the sidebar of the block editor, or beside the content in the classic editor), then update the post. The post stays published; only the llms.txt files leave it out, and a stored file that listed it is dropped at once. A post that Yoast SEO, Rank Math, All in One SEO or SEOPress marks as noindex is left out as well while that plugin is active. The box appears on the post types that are checked in the settings.

When is the file regenerated?

The file is stored and not regenerated on every visit. It is regenerated when you save changed settings or click Regenerate llms.txt, when the site title, the tagline, the permalink structure or the site address changes, when a published post leaves the file (unpublished, private, password, trash, deletion), and, while Auto Regeneration is on (the default), after a post is published or updated. A queued regeneration runs about a minute later through WP-Cron. Browsers and proxies may keep a copy for five minutes.

What happens to my old llms.txt file when I upgrade from version 1.x?

Earlier versions wrote a physical llms.txt file into the site folder. Version 2 serves the file from WordPress instead. On the first page load after the update the plugin keeps your settings, removes the old daily scheduled task, and deletes the old llms.txt file when its first lines show that this plugin wrote it. A file that it did not write is never touched; a notice asks you to delete or rename it, because the web server would send it instead of the generated file. If the old file cannot be deleted, a notice names it. The switch that excluded password-protected content is gone, because such content is now never listed.

What happens to my data if I deactivate or delete the plugin?

Deactivating removes the queued regeneration and the short-lived locks and keeps your settings and the stored files. Deleting the plugin from the Plugins screen removes the settings, the stored files, the request counts, the Hide from llms.txt choices, the locks and the scheduled events on every site of a network, the record of which plugin notices each user closed, and the old llms.txt file of version 1.x when this plugin wrote it. The plugin keeps no other data, and there is no option to keep data on deletion. While the free version and the premium package are both installed, deleting one of them removes nothing, because they share the data. If you allowed Freemius to connect, Freemius is told that the plugin was deleted; the records it keeps are not touched.

Does the plugin work on multisite?

Yes. You can activate it for the whole network or for single sites. Sites created later are set up automatically after a network activation. Settings, the stored file and its address belong to each site; there is no network admin screen.

Does the plugin work on multilingual sites?

Yes, with Polylang or WPML. When one of them is active, the plugin serves one file for each language: the default language at /llms.txt and the others at the home address of their language. Each file lists the content of its language, and the category choice is mapped to the translated categories. With Polylang the site title and the tagline are translated when you have translated those strings, and the title and summary you write on the settings screen can be translated with Polylang (Languages > Translations) or WPML (String Translation). Without a language plugin there is a single file.

Can I see whether AI crawlers read my llms.txt?

Yes, as numbers. The Statistics box of the settings screen shows how often llms.txt was requested in the last 30 days, and how many of those requests came from known AI crawlers: requests whose user agent names GPTBot, ClaudeBot, PerplexityBot or another crawler of an AI company. Only the day and these two numbers are stored, for 30 days; no IP address and no user agent. Requests that a CDN or a cache plugin answers from its copy never reach WordPress and are not counted, and a user agent can be faked, so the numbers show a trend, not proof. The premium plan adds the AI Crawler Report: which crawler requested which file on which day, for 90 days.

How does llms.txt relate to robots.txt?

They are different files. robots.txt tells crawlers which addresses they may request. The llms.txt file of this plugin is a list of your content with links; it contains no allow or block rules, and the plugin neither reads nor changes your robots.txt. If robots.txt blocks a crawler from your pages, llms.txt does not lift that.

Does the plugin create an llms-full.txt file?

The free version serves llms.txt only. The premium plan adds llms-full.txt, the full text of the listed items as Markdown in one file for each language, and a Markdown copy of each listed page at its address followed by .md. llmstxt.org proposes the .md copies; llms-full.txt is a common companion file that the proposal does not describe. Both follow the rules of llms.txt: hidden, noindex, private and password-protected content is never in them.

Does the plugin slow down my site?

It adds very little. On every page load it only compares the requested path with the llms.txt addresses. A visit to llms.txt is answered from the stored copy and adds one to the request count of the day. Saving a post only queues a regeneration, which runs later through WP-Cron.

What does the premium plan add?

The free version is not limited in time. It serves llms.txt for every site of a network and every language, lists the post types you choose with a content limit of up to Unlimited, uses the meta descriptions and the noindex choice of Yoast SEO, Rank Math, All in One SEO and SEOPress, regenerates the file after a post is published or updated, lets you write the title and summary of the file and hide single posts, counts the requests of llms.txt, and includes Preview, Regenerate llms.txt and Help & FAQ.

The premium plan adds a category filter, descriptions of 100 to 5000 characters and the featured image of each item.

The premium plan is a separate package, sold through Freemius. The Premium box on the settings screen compares both versions, and Upgrade in the LLMS Generator menu shows the plans and prices.

How do I get the premium plan, and how do I install it?

Open Upgrade in the LLMS Generator menu, or click Upgrade to Premium in the Premium box of the settings screen, and buy a plan there; Freemius handles the payment. Upgrade appears once you have chosen Skip or Allow & Continue on the first screen of the plugin. Freemius then emails you the premium package (a .zip file) and a license key; both are also in your account at https://customers.freemius.com/. In WordPress go to Plugins, add a new plugin, click Upload Plugin and choose that .zip file, then activate it and enter the license key when you are asked. If both versions are installed, Freemius deactivates the free one. Your settings are kept. The free version from WordPress.org never contains the premium code.

What is Freemius, and do I have to connect an account?

No. The plugin includes the Freemius SDK, which sells the premium plan and manages its licenses. The free version has no license screen and no locked option, and llms.txt is served from the start. The first time you open LLMS Generator, Freemius shows an opt-in screen in place of the settings; Help & FAQ stays open. If you choose Skip, nothing is sent to Freemius and the plugin works the same. If you choose Allow & Continue, your browser sends what that screen lists to Freemius: your name and email address, the site address and title, the language, the WordPress, PHP, plugin and SDK versions, and later the list of your plugins and themes. Opening Upgrade makes your site ask Freemius for the plans and prices (the request names your site address) and loads the Freemius checkout script in your browser; the plugin keeps the Google Analytics script of that page from loading. The code that builds llms.txt does not send your content to another server. See the Freemius privacy policy (https://freemius.com/privacy/).