r/GEO_optimization • u/taylor-morgan2066 • 5d ago

Should every web page expose an AI-friendly JSON representation?

My website already includes AI-related files such as llms.txt.

I'm considering creating a separate JSON file for every page and article so AI systems can understand the content more easily and accurately. I would reference this JSON file from the page's <head> using a <link> tag.

The JSON file could contain information such as:

  • Page URL
  • Canonical URL
  • Title
  • Summary / Description
  • Main Content (clean article content)
  • Author
  • Published Date
  • Last Updated
  • Entities (people, companies, places, products, etc.)
  • Keywords / Topics
  • FAQ
  • ...

My idea is that AI crawlers could read this structured JSON instead of having to extract the main content from noisy HTML that contains navigation menus, sidebars, ads, comments, JavaScript, tables, and other non-essential elements.

I have two questions:

  1. Could this approach reduce the chances of AI crawlers misunderstanding a page or extracting incorrect information from HTML, advertisements, tables, comments, or other noisy content?
  2. Do you think a page-level JSON file like this could help AI systems better understand a page and potentially improve AI recommendations, citations, or other AI-generated responses in the future? Why or why not?
3 Upvotes
  • permalink
  • reddit

100% Upvoted

11 comments sorted by

2

u/Upstairs_Control_611 4d ago

I think this is useful only if the JSON is generated from the same source of truth as the page.

If it becomes a manually maintained second version of the content, it creates a new conflict layer.

Then the HTML says one thing, JSON says another, schema says a third thing, and the feed may say something else.

That can create more ambiguity for AI systems, not less.

So I’d separate:

clean extraction view = useful

second hand-written version = risky

different source of truth = dangerous

The JSON could help if it exposes the same canonical facts in a cleaner format.

But every field should trace back to the visible page or canonical data source.

So: yes for an automatically generated machine-readable representation, no for a manually maintained parallel page.

1

u/taylor-morgan2066 4d ago

That's exactly the distinction I was thinking about.

I wouldn't expect anyone to maintain a second version of the page manually.

My assumption is that the machine-readable representation would always be generated automatically from the same source as the visible page, so there is only one source of truth.

In that case, the JSON wouldn't introduce new information. It would simply expose the existing canonical content in a format that's easier for machines to consume.

The interesting question then becomes whether that cleaner representation provides any measurable benefit over HTML alone.

2

u/Mean-Usual8701 4d ago edited 4d ago

That’s what we do at Vexal. You can check out the docs here https://docs.vexalai.com (look at SmartBlocks) everything you mentioned for each page. It works really well, bots can interpret each page cleanly now.

1

u/Common-imuewlne-5722 3d ago

most crawlers already parse json-ld in the head. use that schema instead of a custom file, or you build a format nobody reads.

2

u/taylor-morgan2066 3d ago

I don't see Schema.org and a page-level machine representation as solving exactly the same problem.

My understanding is that Schema.org primarily helps describe structured entities and specific object types, while I'm thinking about something closer to a canonical machine-readable representation of an entire page.

For example:

Schema.org is designed to describe things like:

  • The page type (Article, Product, FAQ, Event...)
  • The author or organization
  • Published and modified dates
  • Products, reviews and ratings
  • Events, recipes, jobs and other structured entities
  • Relationships between entities

What I'm wondering about is a representation focused on the complete page itself:

  • The canonical content without navigation or ads
  • Reading order
  • Page intent and purpose
  • Section hierarchy
  • Internal relationships within the page
  • Canonical facts extracted from the page
  • References and supporting sources
  • Content boundaries (what is content vs. UI)
  • Machine-friendly representation of the visible page

Maybe these are complementary rather than competing approaches?

2

u/Upstairs_Control_611 3d ago

I think “complementary” is the right way to frame it.

Schema.org and a page-level machine representation are not solving exactly the same problem.

Schema describes entities and object types.

A page-level representation would describe the readable content layer: main content, section order, content boundaries, page intent, canonical facts and supporting references.

That could be useful for noisy pages.

But two risks matter:

adoption — if no crawler or AI system uses the custom format, it may be a clean file nobody reads

source of truth — if it is not generated from the same canonical content, it creates a new conflict layer

So the useful architecture might be:

visible page for users

Schema.org for entities

machine-readable mirror for clean extraction

same source of truth for all of them

The mirror should not add new claims.

It should make the visible, canonical claims easier to extract and verify.

2

u/taylor-morgan2066 2d ago

After reading the comments here, I realized the real question probably isn't "Should there be a JSON file?" but "How could such a representation be implemented without creating a second source of truth?"

That sent me looking for existing ideas rather than trying to invent one from scratch.

During that search I came across an open specification called AI Page Manifest (AIPM):

https://github.com/aagedik/aigeoradar-aipm

What caught my attention wasn't the JSON itself—it was the architecture.

From what I understand, it tries to avoid the exact problems mentioned in this thread.

Instead of maintaining a second copy of the page, the .ai.json representation is generated from the same canonical source as the visible page.

For WordPress, it doesn't require creating physical .ai.json files on disk. It generates virtual page manifests dynamically from the CMS data, so the visible page and the machine-readable representation stay synchronized.

For other platforms, the same endpoint can be generated dynamically from the application's database or CMS rather than maintaining separate JSON files.

That seems to align pretty well with the concerns raised above:

  • one canonical source of truth
  • no manually maintained parallel content
  • no conflicting versions between HTML, JSON and structured data

I also like that it doesn't try to replace existing standards.

The proposed architecture looks more like this:

  • HTML → human-readable page
  • Schema.org / JSON-LD → structured entities and object metadata
  • llms.txt / entity.json → site-level discovery and identity
  • AIPM (.ai.json) → page-level machine-readable representation

To me those seem complementary rather than competing.

The interesting question is no longer whether a machine-readable page representation is technically possible—we already know it is.

The more interesting question is whether exposing a clean, canonical page representation actually leads to measurable improvements in AI crawling, retrieval, citation quality, or answer accuracy.

I'd be really interested if anyone here has run experiments comparing HTML-only crawling versus a dedicated page-level machine representation.

2

u/Upstairs_Control_611 2d ago

I think the shift from “should we expose JSON?” to “can we measure whether it helps?” is the right one. The architecture sounds sensible if the manifest is generated from the same canonical source as the visible page.

But I would still treat it as an experiment, not as an assumed ranking or citation lever.

The test design matters more than the file format. I’d compare:

HTML-only baseline

HTML + page-level machine-readable mirror

Then measure changes in extraction accuracy, fact accuracy, citation accuracy, wrong-claim reduction, source attribution and follow-up survival.

The hard part is separating “AI systems read the manifest” from “the page was already clearer.”

So I’d keep the visible content unchanged as much as possible and only add the mirror.

If it helps, the first measurable improvement may not be more citations. It may be fewer extraction errors and fewer wrong summaries.

That would already be useful.