Before you assume an AI answer engine cannot see your site, check the public path in order: can a crawler reach the page, can it read the useful answer, can it verify who wrote it and can a buyer take the next step? Access alone does not produce a citation or a sale.
Takeaways
- 01Access is necessary but never a citation guarantee
- 02Check public documents before buying new tooling
- 03Repair one broken layer in order: access, extraction, verification, conversion
The eight checks
Run these checks on the one page that should answer a buyer question. Do not start with the homepage unless the homepage is truly the answer page.
| # | Check | Inspect | Pass condition | First repair if it fails |
|---|---|---|---|---|
| 01 | Public URL | Open the exact page in a private browser window | The page loads without login or a challenge | Remove accidental access restrictions from the public page |
| 02 | HTTP result | Inspect the response with a basic request or browser network panel | The intended page returns a real success status | Fix redirects, server errors or soft-404 routing |
| 03 | robots.txt | Open /robots.txt at the root domain | Your stated crawling policy is explicit and internally consistent | Decide the policy first; then align rules with it |
| 04 | XML sitemap | Open /sitemap.xml | The target URL is listed and the document resolves | Regenerate the sitemap from live routes |
| 05 | Rendered content | View page source or a no-JS response | The useful answer and headings are present in initial HTML | Add server rendering or static prerendering |
| 06 | Canonical URL | Inspect the canonical tag | It points to the public preferred URL | Replace relative, stale or conflicting canonicals |
| 07 | Answer structure | Read the first screen and headings | One direct answer, steps or table appear before filler | Move the practical answer higher on the page |
| 08 | Verification | Look for author, date, sources and scope | A reader can identify who says what and when | Add dated evidence and a visible byline |
The four-layer repair order
Do not repair all eight checks at once. The first broken layer determines the next move.
| Layer | What it means | Fast test | Do not mistake it for |
|---|---|---|---|
| Access | A public crawler can request the page | robots.txt, status, public URL | A promise of ranking or citation |
| Extraction | The page contains a usable direct answer | Raw HTML, headings, tables | A long page with more words |
| Verification | A reader can verify source, date and author | Byline, source links, scope | Decorative trust badges |
| Conversion | A buyer knows the next useful action | One visible CTA and destination | More buttons or a forced email gate |
Rule: if access is broken, do not spend time on schema or prompt testing yet. If the answer is unreadable, do not add more crawl rules. Fix the first broken layer.
Read robots.txt as a policy document
robots.txt is public. It tells crawlers which paths you allow or disallow according to your chosen policy. It is not a universal switch for what an external system will cite, remember or use.
For a small site, keep the file simple:
- state the default policy clearly;
- avoid contradictory blocks for the same paths;
- keep the sitemap URL current;
- review any AI-specific user-agent rules with the same care as a general crawl rule;
- change a rule only when it matches your content and training policy.
BEARTH uses a public robots.txt and a generated sitemap. That is an implementation example, not a universal recommendation to allow every crawler.
Make the answer available before the interaction
A common failure mode is a page that looks correct after JavaScript loads but ships an empty shell or a loading state in the first response. A buyer may wait; a crawler or answer system may not reconstruct the same page.
For each page that should be discoverable, verify that the initial HTML carries:
- a distinct title and description;
- the actual first answer, not only a loading placeholder;
- headings that describe the buyer's question;
- a canonical public URL;
- one visible author, date or source trail where claims need verification.
Static prerendering is often enough for a content site. It is not required because it is fashionable; it is useful when the public answer needs to exist before browser JavaScript runs.
Use llms.txt carefully
An llms.txt file can describe a site's important public pages in one compact document. It does not compel an AI system to ingest, rank or cite the site. Treat it as a readable directory, not a visibility guarantee.
A useful file should:
- name the business and its scope clearly;
- link to the primary offer, evidence page and important guides;
- avoid inflated authority claims;
- reflect the same pages that are actually public and linked in the sitemap;
- receive an update when a core page or product changes.
Keep a 30-minute repair log
| Time | Record | Example |
|---|---|---|
| 00:00 | Target page and buyer question | “How do I test AI visibility as a solo operator?” |
| 00:05 | First broken layer | Extraction: no direct answer above the fold |
| 00:15 | Single repair | Add a 3-step answer and a decision table |
| 00:25 | Re-test condition | Same URL, same buyer prompt, same surface |
| 00:30 | Decision | Ship, repair again or deprioritize |
This log prevents a technical checklist from becoming an endless infrastructure project.
What not to claim
Do not say that a crawler allow rule guarantees an AI citation, a schema block guarantees traffic or a directory file guarantees model training. These systems use their own retrieval and quality decisions. The honest value of this checklist is smaller and more useful: it shows whether your own public site is blocking, hiding or weakening the answer you want to publish.
Use the 60-second AI visibility audit when you need to choose which exposure layer affects revenue first. Use this checklist after the audit when the first repair points to access or extraction.
FAQ
- Does allowing a crawler guarantee an AI citation?
No. Allowing access only removes one possible barrier. Citation and recommendation depend on the system, query, source quality and many factors outside a site's direct control.
- Should every small site create an llms.txt file?
Only if you can keep it accurate and it adds a clear directory of public pages. It should not replace a sitemap, useful HTML or source transparency.
- What is the first page to inspect?
Choose the single page that should answer a real buyer question and lead to your offer. A homepage is often a poor test page because it has too many jobs.
- What if the page needs JavaScript to work?
Keep the interaction, but make the meaningful public answer available in initial HTML when search discovery matters. Static generation or server rendering are common ways to do that.
- Should I block AI training crawlers?
That is a business and content-policy decision. Make it deliberately, document it consistently and do not confuse a training preference with search visibility or citations.
Updated 11 October 2026. This is a public-site inspection checklist, not legal advice or a guarantee about any crawler, search engine or AI answer system.