By 10xSearch Editorial
An estimated 43% of the public web runs on WordPress, yet in our audits of 220 small- and mid-market WP sites between January and March 2026, only 11% were being actively cited by ChatGPT, Perplexity, or Gemini in their target query sets. The gap is almost never content quality. It’s a predictable stack of technical issues - JavaScript-dependent rendering, malformed schema, blocked AI user agents, bloated page weight, missing llms.txt, and builder markup that buries the actual text - that makes a WP site functionally invisible to the crawlers and retrieval layers behind modern AI answer engines.
This is the playbook we use to diagnose and fix it. Ten measurable issues, how to detect each one in under five minutes, and the exact change that moves a page from "invisible" to "cited."
The Core Problem: AI Crawlers Are Stricter Than Googlebot
Googlebot has spent a decade learning to tolerate messy WordPress sites. It executes JavaScript, waits for lazy-loaded content, infers structure from ambiguous markup, and forgives a lot. AI crawlers - OpenAI's GPTBot, Anthropic's ClaudeBot, PerplexityBot, Google-Extended, Amazonbot, and Bytespider - are more conservative. Most of them fetch the raw HTML response, do limited or no JavaScript execution, rely heavily on structured data, and skip pages that block them in robots.txt or return thin server-rendered bodies.
That means the two biggest AI visibility killers are things Googlebot hides from you:
1. Content that only appears after JavaScript runs shows up fine in Google Search Console but is invisible to GPTBot.
2. Blocking or failing to opt in to AI user agents in robots.txt silently removes you from the training and retrieval corpus used by every major LLM.
Traditional SEO tools won't surface either issue. A Lighthouse score of 95 means nothing if GPTBot never parses the body of your post.
The 10-Point WordPress AI Visibility Diagnostic
Run this against any WP site. Each row is independently measurable and fixable.
| # | Issue | How to Detect (5-minute test) | Fix |
|-|---|-----------|--|
| 1 | JS-rendered content missing from HTML | Fetch your post with a command-line tool using the GPTBot user agent string, then count paragraph tags in the response - if the paragraph count is far lower than what you see in the browser, your content is client-rendered | Switch to a server-rendered or hybrid theme; disable “defer all scripts” on content blocks; move AJAX-loaded sections to PHP output; avoid Elementor Pro dynamic loops for primary article text |
| 2 | robots.txt blocks AI crawlers | Open your site’s robots.txt in a browser and look for references to GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Amazonbot, or Bytespider | Explicitly allow the agents you want to be cited by. If using a plugin like Yoast or Rank Math to generate robots.txt, override via the plugin filter or place a physical robots.txt file in the site root |
| 3 | REST endpoint blocked | Visit yoursite.com/wp-json/wp/v2/posts?per_page=1 - a 401/403/404 response blocks retrieval tools that prefer the JSON feed | Keep the REST API open for GET requests on public post types; restrict only sensitive routes (users, settings) via a security plugin or REST authentication filter |
| 4 | No Article / FAQPage schema | Paste the URL into Google’s Rich Results Test or Schema.org validator; look for `@type: Article` and `FAQPage` where appropriate | Use Rank Math or Yoast SEO to auto-emit Article schema; add FAQPage schema on posts with Q&A sections (Rank Math’s FAQ block handles this automatically) |
| 5 | Missing or malformed Organization schema | View source and search for `”@type”:”Organization”` - must include `name`, `url`, `logo`, `sameAs` with social profiles | Deploy once in your theme header via Rank Math’s Local SEO module, Yoast’s Organization settings, or a hand-rolled JSON-LD block in `header.php` |
| 6 | Canonical + noindex conflicts on valuable pages | Crawl with Screaming Frog in “GPTBot” UA mode; filter for `noindex` or self-referencing canonicals that point elsewhere | Remove `noindex` from category/tag archives that rank; audit SEO plugin defaults - Yoast historically noindexes attachment pages and some archives |
| 7 | No `/llms.txt` file (emerging standard) | Fetch `https://yoursite.com/llms.txt` - 404 means you have no AI-readable site summary | Add a plain-text `llms.txt` (a proposed convention, gaining adoption fast in 2026) at the root with your business description, primary topics, key URLs, and a short factual bio. See What Is an llms.txt File and Why Every Business Needs One |
| 8 | Page weight over 3 MB or LCP > 4s on mobile | PageSpeed Insights mobile run; check Total Page Weight and Largest Contentful Paint | Install WP Rocket (or LiteSpeed Cache if on LiteSpeed); enable delayed JS, critical CSS, image lazy-loading with native `loading=”lazy”`; serve images as WebP; strip unused Elementor widgets |
| 9 | Elementor/builder divs burying semantic HTML | View source of a post and count `
` and ` ` tags inside the main content - if the article body is almost entirely ` ` with no headings, crawlers see mush | Use native Gutenberg blocks for article body (Heading, Paragraph, List, Table); restrict Elementor to layout sections like hero and CTA; ensure H1 is unique and H2/H3 hierarchy is clean | | 10 | Thin or "author: Admin" content | Check post length (aim > 1,200 words for cluster content), verify a real author byline with `Person` schema | Set a proper author profile with bio, photo, `sameAs` links; publish under a human name, not “Team” or “Admin”; add author schema via Rank Math’s author profile or a custom JSON-LD block | If a page fails even four of these ten, assume it is effectively invisible to ChatGPT, Perplexity, and Gemini - regardless of its Google ranking. Why WordPress Specifically Struggles
WordPress's strength - its plugin ecosystem - is also why most WP sites underperform in AI search.
Page builders render content through JavaScript templates. Elementor, Divi, Beaver Builder, and WPBakery all dynamically assemble DOM on the client. A light JS crawler sees an empty shell and skipping structured text. When GPTBot fetches your carefully written post, it gets a skeleton of empty `
Security and caching plugins routinely block crawlers they don't recognize. Wordfence, Solid Security (formerly iThemes Security), and Cloudflare’s “Bot Fight Mode” are tuned to block unfamiliar bots. GPTBot, ClaudeBot, and PerplexityBot often fall into that bucket by default unless explicitly whitelisted. Sites we audit regularly return 403s to AI crawlers while Googlebot is passed cleanly.
SEO plugins default to partial schema. Yoast and Rank Math both emit Article schema on posts, but FAQ, HowTo, Product, and author-level schemas often require manual activation. Many sites stop at the defaults and wonder why AI engines can’t confidently cite them.
Lazy loading is too aggressive. The combination of WP Rocket’s lazy-load feature plus native `loading=”lazy”` plus a CDN’s own lazy-load can delay text content past the point a non-JS-rendering crawler gives up.
Thin category and tag archives get indexed instead of cornerstone posts. Without careful configuration, AI retrieval tools scrape low-value archives that have no original prose - and your flagship 3,000-word guide never gets indexed because it’s buried behind a builder layout.
None of these are problems WordPress can't solve. They are problems WordPress won't solve for you by default.
robots.txt - The Fastest, Most Ignored Fix
Most site owners have never edited robots.txt with AI agents in mind. The current AI crawler landscape (as of April 2026):
| User Agent | Operator | Purpose | Respects robots.txt? |
|----|----|---|-------|
| `GPTBot` | OpenAI | Training data for ChatGPT | Yes |
| `OAI-SearchBot` | OpenAI | Live ChatGPT Search retrieval | Yes |
| `ClaudeBot` | Anthropic | Training data for Claude | Yes |
| `Claude-User` | Anthropic | Live user query fetches | Yes |
| `Claude-SearchBot` | Anthropic | Search indexing for Claude | Yes |
| `anthropic-ai` | Anthropic | Older/legacy UA, now deprecated | Yes |
| `PerplexityBot` | Perplexity | Live retrieval for Perplexity answers | Yes |
| `Google-Extended` | Google | Controls use of your content in Gemini training; does not affect Google Search | Yes |
| `Amazonbot` | Amazon | Alexa and retrieval | Yes |
| `Bytespider` | ByteDance | Doubao / TikTok search | Inconsistent |
| `cohere-ai` | Cohere | Training and retrieval | Yes |
A healthy starting configuration for a WP site that wants to be cited across all major AI engines should explicitly allow each of the major AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended) to access the site, while retaining the standard WordPress admin protections for the wildcard agent. Your sitemap reference should also be included so crawlers can find every indexable URL on the site.
The file should live as a physical text file in the WordPress root directory (this overrides any plugin-generated version). After deployment, verify the file by fetching it in a browser with the URL yoursite.com/robots.txt and confirming each explicit allow line is present.
Schema That Moves the Needle on WordPress
Schema is how AI engines confirm what a page is, who wrote it, and why they should trust it. For WordPress, focus your energy on five schema types and skip the long tail.
| Schema | Where It Lives | Plugin That Handles It Well |
|---|------|----------|
| Article / BlogPosting | Every post | Rank Math (auto), Yoast SEO (auto) |
| FAQPage | Posts with Q&A sections | Rank Math FAQ block, Yoast FAQ block |
| Organization | Site-wide (header) | Rank Math, Yoast, or custom JSON-LD in theme |
| Person (author) | Author pages and author JSON-LD on posts | Rank Math author schema, or custom via `functions.php` |
| Product | Commerce pages | WooCommerce + Rank Math PRO or Yoast WooCommerce SEO |
Two configuration habits that materially improve citation rates:
- Fill every field in your SEO plugin's schema settings. Rank Math in particular unlocks deeper schema when you complete the “Titles & Meta” and “Schema” sections of the plugin. Most sites leave half of it blank.
- Validate every template. Google’s Rich Results Test and Schema.org validator both flag issues GPTBot will silently ignore. Fix warnings, not just errors.
We also recommend installing WP Rocket for perf (it has the cleanest LCP impact among WP caching plugins in our 2026 benchmarks) and Imagify or ShortPixel for image compression. Both indirectly boost AI visibility because crawlers that time out don’t come back.
Before/After: A 2026 Case Study
One of our February 2026 engagements, a WordPress-hosted B2B SaaS blog, came in with a textbook version of the problem. The numbers below are anonymized but real.
Starting state (Feb 3, 2026):
- 142 published posts, average 1,400 words
- Traditional Google organic traffic: 38,000/mo
- Cited mentions across ChatGPT, Perplexity, and Gemini (measured across a set of 120 target queries): 7
- Average page weight: 4.2 MB
- AI crawlers blocked by Wordfence (GPTBot + ClaudeBot both receiving 403)
- No FAQPage schema, no llms.txt, no Organization schema
- Built on Divi with primary article text inside Divi modules (div soup in view-source)
What we changed over six weeks:
1. Whitelisted GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended in Wordfence and Cloudflare
2. Rewrote robots.txt at the filesystem level with explicit allows
3. Migrated article body content from Divi modules to native Gutenberg blocks (kept Divi for hero and CTA sections only)
4. Installed Rank Math PRO and enabled Article, FAQPage, Organization, and Person schema
5. Added Person schema for three named authors, replacing the generic "Admin" byline
6. Published an `llms.txt` with business description, primary topics, and 12 cornerstone URLs
7. Installed WP Rocket, enabled delayed JS and critical CSS; dropped average page weight to 1.6 MB
8. Fixed 22 pages with accidental `noindex` flags (SEO plugin defaults had misfired on pillar content)
Ending state (Mar 17, 2026):
- Traditional Google organic traffic: 41,200/mo (+8%)
- Cited mentions across the same 120-query set: 46 (+557%)
- Average page weight: 1.6 MB
- 0 crawler 403s over a 14-day monitoring window
The traditional SEO lift was a nice-to-have. The AI citation lift is what paid for the engagement three times over in the first quarter after launch.
What To Actually Do This Week
If you only have a few hours, prioritize in this order:
1. Audit robots.txt and unblock AI crawlers (30 minutes). Highest-leverage fix with the fastest payoff.
2. Validate your schema. Run Rich Results Test on five cornerstone URLs; fix errors, add FAQPage where Q&A sections already exist (2-3 hours).
3. Publish an `llms.txt` (30 minutes). A static text file at the root with business description, primary topics, and links to your best content.
4. Test a cornerstone post with the GPTBot user agent (15 minutes). Use any command-line fetch tool or browser extension to request the post as GPTBot would, then compare the word count of the raw response to the same post rendered in a browser. A 60%+ word-count delta tells you the content is JS-dependent.
5. Swap out a page builder's body content for Gutenberg blocks on your top three posts (half a day per post). This is the slowest fix but has the largest per-page impact.
If you want the full diagnostic run across your site and want to see exactly which pages are and aren't being cited by each major AI engine, that's what we do.
Frequently Asked Questions
Does WordPress itself hurt AI visibility, or is it the configuration?
It's the configuration. Vanilla WordPress with the default Twenty Twenty-Four theme and a completed SEO plugin performs well in AI search. The issues come from page builders that render content via JavaScript, security plugins that block unfamiliar crawlers, and SEO plugins that are installed but not fully configured. Strip those problems out and WP is a strong platform for AI visibility.
Which SEO plugin is best for AI search on WordPress - Rank Math or Yoast?
Both are capable. Rank Math (in our 2026 benchmarks) emits more schema types out of the box, including FAQ, HowTo, and Video, and its built-in schema templates are more granular. Yoast is stronger on WooCommerce integration and has a cleaner UX. Either is fine; the real gain is making sure every field is actually filled in, not which plugin you picked.
Will blocking AI crawlers in robots.txt protect my content from being used to train models?
Partially. Major crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) respect robots.txt. A few - notably some ByteDance crawlers - have been less consistent. If your goal is privacy, a robots.txt disallow is a reasonable first step but not a legal guarantee. If your goal is AI visibility, you want the opposite: explicit allows.
Do I need llms.txt if I already have strong schema and a sitemap?
Yes. `llms.txt` is different from both. It's a short, human-authored summary designed for LLM retrieval - a cheat sheet that tells AI engines exactly what your site is and which URLs matter. It takes 30 minutes to write and shows up consistently in the training and retrieval pipelines of the major engines. Skipping it is leaving citation share on the table.
Is Elementor or Divi incompatible with AI search?
Not incompatible - but not ideal for primary content. Both are fine for layout sections (hero, CTA, footer). The issue is putting your article body inside builder modules, because many builders render prose through JavaScript templates that light AI crawlers don't execute. Use native Gutenberg blocks for the body copy; use builders for the wrapping layout.
How quickly do fixes translate to AI citations?
Faster than SEO. Because AI engines combine training-time knowledge with live retrieval (RAG), fixes like unblocking GPTBot, adding schema, and publishing llms.txt can surface in Perplexity and ChatGPT Search within days to weeks. Full integration into training-set-based answers takes longer (typically 3-6 months).
How do I measure AI citations on my WordPress site?
Manually, test a set of target queries across ChatGPT, Perplexity, Gemini, and Google AI Mode monthly - track mentions, positioning, and sentiment. For automated tracking across hundreds of queries and multiple engines, use an AI visibility monitoring platform. (This is one of the things we build on the 10xSearch platform.)
The Bottom Line
WordPress sites are not inherently invisible to ChatGPT. They become invisible through an accumulation of default plugin settings, builder-dependent rendering, and robots.txt rules written in the pre-AI era. The fixes are not exotic - they are measurable, repeatable, and available to any site operator who knows where to look.
The sites we see winning AI citations in 2026 aren't the ones with the biggest content budgets. They are the ones that systematically closed every gap on the diagnostic above and kept their stack clean.
Start with robots.txt today. Validate your schema this week. Publish llms.txt before Friday. Then work down the list.
Related reading from 10xSearch:
- The Real Estate Agent's Guide to Getting Found by AI Search Engines
- GEO vs SEO - Why Traditional Search Optimization Isn't Enough Anymore
- The AI Search Visibility Guide
- How ChatGPT Makes Business Recommendations
Ready to find out where your WordPress site stands? Book a free AI Visibility Audit and we’ll run the full 10-point diagnostic on your site - robots.txt, schema, render path, page weight, llms.txt, the works - and show you exactly which pages are and aren’t being cited by ChatGPT, Perplexity, Gemini, and Google AI Mode.