Is Your Website Falling Behind?
Search is changing. People are no longer relying solely on traditional search results to find businesses, products and information. They are increasingly using AI assistants and agentic search tools that can research options, compare providers, summarise content and, in some cases, complete actions on the user’s behalf.
This does not make SEO obsolete. In fact, it makes a technically sound, clearly structured website even more important.
AI systems need to discover, access, interpret and trust information before they can use it in an answer. If a website is difficult to crawl, slow to render or unclear about the organisation and services it represents, it is less likely to become a useful source for either conventional search engines or AI agents.
Agentic AI Still Depends on Retrievable Information
Many AI search experiences use retrieval augmented generation, commonly known as RAG. Instead of producing an answer entirely from a model’s existing knowledge, the system retrieves current information from external sources and uses it to support its response.
Google has confirmed that its generative search features use its existing Search index and core ranking systems to retrieve relevant pages. It can also use “query fan out”, where a single question is expanded into several related searches before an answer is assembled. Google’s guidance therefore remains clear:
“foundational SEO continues to matter for visibility in generative search”
This means a website must first pass several technical stages:
- The crawler must be able to reach the page.
- The content must be rendered correctly.
- The page must be indexable.
- The system must understand what the page is about.
- The information must be considered relevant and reliable.
- Individual facts must be easy to retrieve and connect to the user’s question.
A failure at any of these stages can reduce the likelihood of the website being used as a source.
Technical Optimisation Creates the Foundation
Technical SEO gives search engines and AI retrieval systems reliable access to a website’s content. It removes barriers that could prevent important information from being discovered or correctly interpreted.
This includes ensuring that:
- Important pages return a successful 200 status code.
- Valuable content is not blocked through robots.txt.
- Pages are not unintentionally marked noindex.
- XML sitemaps contain current, canonical URLs.
- Canonical tags identify the correct version of each page.
- Redirect chains, broken links and duplicate URLs are minimised.
- Page content is available in the rendered HTML.
- JavaScript does not prevent essential information from loading.
- Internal links create clear relationships between services and topics.
- Mobile layouts are accessible and easy to use.
- Pages load quickly and remain visually stable.
- HTTPS is used consistently across the website
AI agents may access websites in different ways. Some use search indexes, while others inspect the page’s HTML, Document Object Model or accessibility tree. A technically inaccessible website is therefore not simply offering a poor user experience. It may also be withholding information from the systems expected to retrieve it.
SEO Helps AI Systems Decide What Is Relevant
Crawlability alone is not enough. A website must clearly communicate its purpose, subject matter and authority.
Good SEO creates this clarity through descriptive page titles, logical headings, focused service pages, internal linking and genuinely useful content. It also encourages consistency between the claims made on the website and information found elsewhere online.
A strong page should answer questions such as:
- Who provides the service?
- What exactly is being offered?
- Who is it intended for?
- Where is the service available?
- What does it cost?
- What qualifications or experience support the claim?
- When was the information last reviewed?
- What should the visitor do next?
This explicit information is particularly useful for agentic searches. A broad request such as “find a dental practice near Dundee offering teeth whitening with evening appointments” contains several requirements. A website that presents its location, treatment, opening hours and booking information clearly gives a retrieval system more evidence to work with.
Content should also include first hand knowledge, original insight and verifiable details. Repeating generic information found across hundreds of competing websites gives an AI system little reason to choose one source over another.
The Role of Schema and JSON-LD
Schema markup is a standardised vocabulary used to describe the entities and information contained on a webpage. JSON-LD is the format generally recommended by Google because it is comparatively straightforward to implement and maintain. Google Search Central explains that structured data helps Google understand page content and can make pages eligible for richer search appearances.
For example, JSON-LD can state that:
- A company is an Organization.
- A clinic is a Dentist or LocalBusiness.
- A person is a Person.
- A page describes a Service.
- A piece of content is an Article.
- A page contains a VideoObject.
- An event has a date, location and organiser
- A product has a price, brand and availability
This creates a machine readable description of the information already visible to users.
Correct schema can also connect related entities through stable @id values. An article can reference its author, the author can be connected to the organisation, and the organisation can be linked to its address, logo, website and verified social profiles. This helps machines build a more coherent understanding of the people, businesses and services represented by the website.
However, schema must be accurate. It should describe information that is genuinely present on the page and must not be used to manufacture authority. Incorrect types, false reviews, inconsistent addresses or outdated pricing can create contradictions rather than clarity.
Structured data should therefore be:
- Relevant to the page.
- Consistent with visible content.
- Complete where reliable information is available.
- Connected through reusable entity identifiers.
- Updated whenever the page changes.
- Checked with Google’s Rich Results Test and the Schema.org validator
Schema does not guarantee inclusion in an AI answer. Google also states that no special schema is required for generative search. Nevertheless, correctly implemented structured data remains valuable because it reduces ambiguity and supports the wider search ecosystem.
What Is llms.txt?
The proposed llms.txt standard provides a concise, Markdown formatted guide to a website’s most important content. It is normally placed at:
https://www.example.com/llms.txt
Rather than acting as a replacement for robots.txt or an XML sitemap, it can provide AI systems with a curated overview of the website and links to useful resources. The original llms.txt proposal describes it as a way of supplying background information, guidance and links to more detailed Markdown files.
A typical file may contain:
# Example Company
> A short description of the company, its audience and its services.
## Core Services
– [Service One](https://www.example.com/service-one.md): Description of the service.
– [Service Two](https://www.example.com/service-two.md): Description of the service.
## Company Information
– [About](https://www.example.com/about.md): Company history and credentials.
– [Contact](https://www.example.com/contact.md): Locations and contact details.
This can help compatible systems locate relevant information without processing every navigation element, script and design component on the main website.
It is important to be realistic about its current value. llms.txt is a proposed convention rather than a universal standard, and Google says it does not use llms.txt as a visibility or ranking signal in Search. Creating one will not compensate for weak content or poor technical SEO.
Its potential value lies in making information more portable and accessible to other tools that choose to support it.
OKF and Markdown Knowledge Files
Open Knowledge Format, or OKF, extends the same general principle: providing agents with clean, structured knowledge in Markdown files.
HTML pages are designed for people and may contain navigation, tracking scripts, interactive components, advertisements and large amounts of layout code. Markdown removes much of this presentational overhead and exposes the content in a compact, predictable structure.
This can offer several technical advantages:
- Fewer tokens are required to process the same information.
- Headings and sections are easier to identify.
- Links retain meaningful context.
- Tables and lists remain machine-readable.
- Content can be divided into focused knowledge files.
- Files can be version controlled and updated systematically.
- Agents can retrieve individual documents without parsing an entire website
An OKF style knowledge directory could include separate Markdown files covering services, products, locations, policies, frequently asked questions and company credentials. A manifest or index file can then explain how those documents relate to one another.
These files should be generated from the same approved content source as the main website. Maintaining separate, manually written versions creates a risk that prices, dates, services or company details will become inconsistent.
Like llms.txt, OKF style Markdown should currently be treated as a supplementary accessibility and interoperability layer, not a replacement for SEO. Google has explicitly stated that special AI files and Markdown are not required for inclusion in its generative Search features.
A Layered Approach to AI Visibility
The strongest approach is not to choose between SEO, schema and AI-readable files. Each serves a different purpose:
| Layer | Primary purpose |
|---|---|
| Technical SEO | Makes pages accessible, crawlable and indexable |
| On-page SEO | Communicates relevance, context and user value |
| Quality signals | Support confidence, authority and trust |
| Schema and JSON-LD | Describe entities and facts in a standard vocabulary |
llms.txt | Offers compatible agents a curated content directory |
| OKF Markdown files | Provide compact, structured knowledge for agent retrieval |
| APIs and data feeds | Supply current transactional or frequently changing data |
The underlying facts must remain consistent across every layer. A service price shown in HTML, JSON-LD and Markdown should not differ. The organisation name, address, author details and availability information should also be synchronised.
Preparing for an Agentic Web
There is no single file or piece of markup that guarantees inclusion in an agentic AI result. AI visibility begins with the same fundamentals that have always supported strong organic performance: technically accessible pages, useful content, clear entities and trustworthy information.
Schema and JSON-LD strengthen this foundation by making facts easier for machines to classify. llms.txt and OKF style Markdown files may provide an additional retrieval layer for agents that support them, but they should be implemented with realistic expectations.
Our objective is not to optimise a website for one particular model. It is to create a reliable, structured and accessible source of information that can be understood by people, search engines and emerging AI agents alike.
Contact Interphase to discuss how we can optimise your website to keep your business visible in a fast moving Agentic AI and SEO landscape. Get in touch by calling us on 01382 221777 or completing our contact form.
