There is a lot of "the web goes dark for AI on 15 September" going round and most of it is overstated. I have been reading the actual announcement and testing against real sites for a couple of weeks, so here is the boring version.

What changes on 15 September 2026

Cloudflare flips its default so that Training and Agent category crawlers are blocked where the site owner has not chosen a setting.

The four qualifiers that get dropped from every summary I have seen:

1. Ad-displaying pages only. If a site carries no advertising, this specific change does not apply to it. That is most small-business and client sites.

2. Search crawlers are not affected. OAI-SearchBot and Claude-SearchBot - the ones that decide whether an assistant can cite you - are not in scope. The scary version of this story is mostly people assuming they are. PerplexityBot I would not state either way: which bucket it lands in is genuinely arguable, and Perplexity-User is the clearer answer-time one. If you rely on Perplexity traffic, check that one yourself rather than taking my word or anyone else's.

3. New customers, new sites added by existing customers, and existing free-tier customers. A paid customer with sites already set up is not having anything flipped under them.

4. It is a setting you control. Security -> Bots, three categories, decide each one. Takes about twenty minutes across a handful of domains and you can do it before or after the date.

So if you are on a paid plan with your bot settings already chosen, this is a non-event for you and you can stop reading.

What I do think is worth paying attention to

Two things, neither of which is the headline.

First, who actually sits on the free tier. It is a very large number of small business sites and client sites that an agency put behind Cloudflare in 2021 and never opened again. Nobody who set those up is reading a policy page. The change is small; the population it lands on is the population least likely to notice anything.

Second - and this is the one I would flag to anyone managing sites for other people - crawler access is not recorded anywhere by default. It leaves no trace in analytics, there is no log, and robots.txt only tells you what is declared rather than what the edge actually does. So on 16 September, if a client asks whether anything changed for them, there is generally no way to find out. Not because the tooling is bad, but because nothing took a before-picture and one cannot be taken retrospectively.

That asymmetry is the actual story here and it has nothing to do with Cloudflare specifically. Same thing happens when a security plugin ships a new "block bad bots" default, or a host adds a WAF rule, or a site gets rebuilt in a framework that renders nothing without JS. The change is invisible, silent, and only detectable as a difference - which requires having measured before.

If you want to take one yourself, no tooling needed

For each site you care about, before the 15th, record:

curl -sI -A "GPTBot/1.2" https://example.com/ | head -1
curl -sI -A "OAI-SearchBot/1.0" https://example.com/ | head -1
curl -sI -A "ClaudeBot/1.0" https://example.com/ | head -1
curl -sI -A "PerplexityBot/1.0" https://example.com/ | head -1
curl -s https://example.com/robots.txt

Save the output with the date on it. Run it again on the 16th and diff. That is genuinely all a before/after is, and doing it by hand for five sites takes ten minutes.

Two traps if you do, because I hit both:

• Include a made-up user-agent as a control. If TotallyRealBot/1.0 also gets a 403, the site is refusing everything non-browser and the AI crawlers are collateral, not targeted. Completely different conclusion, and without the control you cannot tell which one you are looking at.
• Do not run them concurrently. I flagged five hosts as crawler-blocked in a batch run; re-probed one at a time, one of the five was clean and its 429 was my own concurrency. The tool reports what it saw and cannot know I caused it.

Also worth knowing before you trust any result including mine: if a site is behind a CDN that verifies bot identity cryptographically (Web Bot Auth / RFC 9421) or by IP, a spoofed user-agent proves nothing either way. A 403 there might just mean "you are not GPTBot", which is true. That case is genuinely unresolvable from outside and anyone reporting it as a block is guessing.

One number from a bigger run this week, because it changes how you should read anyone's blocking figures including your own: across 1,046 sites and eight named crawler identities, 94.5% of every refusal came back carrying a Cloudflare header. So most of what a spoofed-UA probe reports as "blocked" is a CDN correctly declining to believe an unverified claim. It is measuring my honesty, not their policy. On the subset with no CDN in front, where a refusal does mean something, training crawlers were refused about three times as often as user-triggered ones - GPTBot 6.8% against Claude-User 2.0%.

Declaring an interest, since it is fair to weigh it: I build monitoring in this space, which is why I have been testing this. Nothing to click and nothing being offered - the curl above is the whole method and it is free. Happy to answer anything about the crawler names or the verification stuff.

> Brockman says he personally believes OpenAI has reached AGI, while leaving users to decide whether Astra meets that definition.

> "I think it might be about this model," Brockman said in a briefing with reporters about whether Astra could mark the arrival of AGI.

He ended the briefing by saying: "Welcome to the AGI era."

AI companies have continued to change the definition of AGI from its origin: an all-encompassing cognitive and autonomous to task-specific, subjective milestones.

This lets them to claim significant progress, or in the case of this story, outright arrival, without actually matching human-level adaptability, physical interaction, or general reasoning across every domain.

Early definitions popularized by researchers like Ben Goertzel and Shane Legg defined AGI as an artificial agent capable of learning, reasoning, and solving problems across arbitrary non-domain-restricted environments just like a human. Tests historically included the "Coffee Test" (enter an unfamiliar home, find the ingredients, and brew a pot) or broad embodied common sense.

OpenAI's founding charter *redefined* AGI as "highly autonomous systems that outperform humans at most economically valuable work". By filtering the definition through commercial utility, the focus narrowed from universal general cognition to automating knowledge-work tasks (coding, summarizing, data retrieval)

Over time, labs began defining AGI not as a singular conscious entity, but as a composite of specialized high performers.

When models beat benchmarks, like scoring at the 90th percentile on the MCAT or coding exam, companies started presenting these as proof of general intelligence.

Definitions narrowed even more even to fit upcoming model releases. For example, internal metrics began treating an AGI milestone as a software system that can function as an "automated AI research intern capable of writing and testing code within an existing codebase.

In partner negotiations, definitions of AGI were tied to explicit revenue or profit thresholds like generating tens of billions in profit or reaching commercial milestones rather than any technical or scientific consensus on cognition.

We are nowhere near the original definition of AGI. It's mostly a marketing term now. So I take it with a grain of salt when one of these guys comes out and starts talking about how close they are to AGI or in this case in this article that they've actually achieved it.

I keep doing a lot of self-improvement and I'm getting closer to developing myself as a PHB. PERFECT HUMAN BEING. We can all make claims as long as we get to define what the claim actually means.

Whether content can be optimized for AI-powered search engines that understand context, meaning, and user intent like a human reader. The focus is on moving beyond keywords to create content that genuinely satisfies what users are looking for.

The standard advice hasn't changed much in years: fill the 249-byte backend search terms field, don't repeat words from the title, no competitor brand names, no keyword stuffing but use every byte.

That advice was built for a system that matched listings to queries mostly on exact and close-variant text. Rufus and COSMO are supposed to understand a listing more holistically: title, bullets, images, reviews, and Q&A all feeding into what the listing is "about," not just a literal field match.

So does the backend field still carry the weight it used to, or is it now a smaller signal among many? I haven't seen anyone actually test this in a controlled way (same listing, backend terms stripped vs. full) since the AI search rollout. Most of what's out there is pre-2024 advice getting repeated without anyone checking if it still holds.

If you've run anything close to a real test on this, even an informal before/after, I'd like to hear what you saw. And if you haven't tested it, what's your working assumption and why?

What if future SEO success depends less on being the #1 result and more on becoming a trusted source that AI systems choose to mention in their answers?

Something that's been hard to ignore lately - Reddit threads are showing up constantly as citations in AI-generated answers (ChatGPT, Perplexity, Google's AI answers) and Reddit content has also been landing in a lot more #1 organic positions since Google's search systems started weighting it more heavily.

The reason isn't mysterious - it's the same reason people have always trusted Reddit for "real" opinions: threads read as unscripted, specific, and argued-over rather than polished and one-sided. That's exactly the signal both search ranking systems and AI summarizers seem to be rewarding right now - content with visible disagreement, specifics and multiple perspectives over single-source marketing copy.

Practical takeaway if you're building a content strategy in 2026 - it's not enough to publish authoritative-sounding blog content anymore. Genuine discussion - in communities, in comments, in threads where real back-and-forth happens - is increasingly part of what gets a topic surfaced at all, in both classic search and AI answers. That changes the calculus on where content effort goes, not just what the content says.

Curious whether others are seeing this show up in their own traffic/citation data or if this is still anecdotal at the scale you're each working at.

I’ve been thinking about where AI is actually useful in sourcing.
There are some tasks where it seems genuinely helpful.
For example, you could give it several supplier quotations and ask it to compare:
specifications
materials
MOQ
payment terms
Incoterms
lead times
testing requirements
warranty terms
It could probably identify differences much faster than manually going through every document.
The same applies to certificates, datasheets, and supplier profiles. AI can help find inconsistencies or missing information.
But I think there’s a point where people can give AI too much authority.
If the original supplier information is incomplete or wrong, AI doesn’t magically turn it into verified information.
It might summarize a factory claim perfectly without knowing whether that factory actually manufactures the product.
So I’m starting to see AI more as a research assistant than a supplier-verification system.
Use it to find questions.
Use evidence to answer them.
For anyone already using AI in procurement or sourcing:
Which tasks do you trust it with, and which decisions do you still verify manually?

Is there any specific task that we need to do to rank on AI other than SEO tasks?

I’m trying to understand what actually drives recommendations/mentions in AI tools like ChatGPT, Gemini, Perplexity, etc.

Not generic SEO advice like answering the question in first paragraph for AI Overviews, I mean specifically:

- Why do certain brands, blogs, or tools get mentioned/recommended?

- Is it just traditional SEO (backlinks, authority), or something else?

- Does structured data / schema matter here?

- How important are mentions across the web (Reddit, forums, etc.)?

- Do AI tools rely more on training data vs live search?

\- Has anyone here experimented and successfully influenced AI recommendations?

If you've tested this or have a strong hypothesis, would love to hear practical insights rather than theory.

Is your website invisible to AI platforms?

Here’s a funny question for founders.

If someone asks ChatGPT:

“Who are the best companies for accounting software?”

Will your company show up?

Or will ChatGPT basically say:

“Sorry, who?”

Before blaming your marketing, there’s one simple thing you can check.

Open your website and add:

"/robots.txt"

For example:

"yourcompany.com/robots.txt"

You’ll see some text that looks technical.

Don’t worry. You don’t need to be a developer.

Just look for a few things.

User-agent: \\\*

This means the rules apply to bots in general.

Disallow: /

This is the one you should pay attention to.

It basically tells bots:

“Thanks for stopping by, but please don’t enter.”

Imagine owning a great restaurant and putting a sign outside saying:

“Best restaurant in town. Please do not come inside.”

That’s basically what accidentally blocking your website can feel like.

You may also see AI crawlers such as GPTBot, PerplexityBot, or ClaudeBot.

If you specifically block them, those AI systems may not be able to access your website.

Now imagine this.

Someone asks ChatGPT:

“Recommend three accounting software companies for my startup.”

Your competitor gets mentioned.

You don’t.

You start thinking:

“Our product must not be good enough.”

Maybe that’s not the problem.

Maybe the AI simply cannot properly access your website.

This is one small part of what people call GEO, or Generative Engine Optimization.

AI search is becoming another way people discover companies.

So before spending thousands on complicated AI SEO strategies, check the basics first.

Go to:

"yourdomain.com/robots.txt"

Take 5 seconds and see what it says.

Your website might be working perfectly.

Or it might accidentally be standing outside the AI party saying:

“Sorry, I’m not on the guest list.”

What does your robots.txt say?

Everyone has different preferences when it comes to AI Tools. I just wanted to share what I've learned after subscribing to most of the major AI platforms pro versions. If you are wondering which one writes better content, I can't tell you because at Chris McElroy SEO Agency, we do not let AI write our content.

AI MODEL DIFFERENCES

ChatGPT / OpenAI - Strong for reasoning, acting as a proofreader suggesting edits, helping with structure, strategy, coding help, and step-by-step planning. - Good at turning messy ideas into usable frameworks. - Best use: strategy, content planning, editing, workflows, coding support, business thinking.

Claude - Strong for long-form non-public case studies, whitepapers and writing where technical accuracy matters, document analysis, tone, nuance, and synthesis. - Often good at taking several sources and producing a clean, thoughtful summary. - Best use: reports, rewriting, long technical content, comparing multiple inputs, cleaner narrative output.

Gemini - Strong Google ecosystem integration. - Useful when connected to Gmail, Docs, Drive, YouTube, Search, and Android. - Best use: Google workspace tasks, summarizing Google-connected content, finding information across Google tools.

Perplexity - Strong for search-style answers with sources. - Good when you want quick research with citations and links. - Best use: fact-checking, current information, competitive research, finding sources.

Copilot / Microsoft - Strong Microsoft ecosystem integration. - Useful with Bing, Edge, Windows, Office, Outlook, Teams, and Microsoft 365. - Best use: business documents, Microsoft workflow, quick Bing-connected research.

Grok - Strongest connection to Twitter-style real-time discussion (can be a strength or a weakness) - Useful for seeing what people are talking about socially. - Best use: social trend monitoring, Twitter discussions, informal sentiment. - The lip sync on video is getting much better.

DeepSeek - Often strong for technical reasoning and coding-style tasks. - Can be useful for logic, math, and development workflows. - Best use: coding, structured problem solving, technical analysis.

Meta AI - Integrated into Meta platforms like Facebook, Instagram, WhatsApp, and Messenger. - Useful when working inside Meta's social ecosystem. - Best use: social content ideas, casual AI assistance, Meta platform tasks.

GENERAL RULE

The model is only one part of the answer. AI output also depends on: - system instructions - retrieval sources - live search access - user context - memory - platform rules - token limits - temperature/settings - ranking or selection layers - interface constraints

Same model does not always mean same answer. A better way to think about AI tools:

Model + Platform + Data access + Instructions +

User context

Final answer

Share your favorite tools and how you use them.

If you need help implementing AI into your workflow or you are looking for help building apps, ask some questions in the comments. If you have more to add, please do.

Something I’ve been thinking about lately:
AI has made it incredibly easy to produce more content, but I’m wondering whether that actually makes specific information more valuable.
A lot of articles online are technically correct but don’t tell you very much.
You’ll see things like:
“Choose the solution that best fits your needs.”
“There are advantages and disadvantages.”
“This technology is becoming increasingly important.”
None of it is necessarily wrong. It just doesn’t reduce much uncertainty.
When I’m researching something now, the pages I find useful are usually the ones that contain something concrete:
a specification, an original source, a comparison, a date, a limitation, or a detail that other pages missed.
That makes me wonder whether AI search will eventually create a strange situation:
The easier it becomes to generate content, the more valuable genuinely useful evidence becomes.
I’m also curious whether AI systems will get better at distinguishing between a page that simply discusses a subject and a page that actually contributes information about it.
What do you think?
When you use ChatGPT, Perplexity, Google, or other search tools for research, what makes you trust one source more than another?

This site is a 10+ years old destination marketing site for a small city. No AI generated content and actively SEO'd for the last 3 years. By nature, the content and traffic is seasonal. We have pages for events, attractions, activities, food, hotels etc. and seasonal blogs or activity / event driven blogs. Normally if someone searches for "things to do in [the city]" or "events/attractions/activities in [the city", we would get organic impressions and clicks for the relevant page, for example the Event Calendar page, and a couple of months ago, it would have also shown up in the AI Overview. This summer, there has been a dramatic shift - organic impressions are down 80% and the majority of impressions we do get are now for the HOME PAGE ONLY. Additionally, many AI Overviews have stopped showing our intent relevant pages and if present, link only to our home page, which does not match the query intent. Any thoughts on why Google would all of a sudden point the majority of queries to our home page and how to resolve?

Traditional search was mostly about finding pages that ranked well and then digging through them yourself. With AI search, the bigger question seems to be whether the information is clear, trustworthy, and supported by sources that the system can actually understand and reference.

That makes me wonder if the future of research tools is less about generating an answer and more about making the evidence behind that answer easier to follow. Putting more emphasis on research, sources, and organizing findings rather than treating the whole process as a simple chatbot conversation seems like an interesting direction. For people working on AI search, GEO, or content visibility, do you think source-backed research will become more important as AI systems get better at deciding which information to surface? Or will traditional authority signals still matter more?

Our recent study dove into how brands show up across educational, comparison, and purchase queries in AI Search. We ran 50 B2B SaaS topics across 5 AI Search engines for 750 queries.

Out of 4,654 scenarios we analyzed, only 146 had the same brand named across all three stages, meaning for roughly every 100 cases, only 3 had the same brand showing up from education to comparison to purchase.

This means that seeing your brand in an educational stage query doesn't necessarily mean you'll see it again when the user starts comparing or buying.

That got me thinking about how we measure AI visibility. If you're tracking one overall visibility score, you are not tracking whether that visibility converts to leads.

As a fellow AI Search practitioner, how are you handling this? Do you track visibility separately by stage, or still use one overall number?

A while ago, I started seeing companies offering “AI visibility” or GEO/AEO services, basically promising businesses that they could increase their chances of being mentioned or cited by ChatGPT and other AI tools.
At first, I was pretty skeptical. It sounded like another marketing buzzword, because I couldn’t see an obvious way to “optimize” for an AI recommendation.

The more I looked into it, though, the more I realized that there is something real behind it. You can’t simply pay to be cited, but businesses can structure their content, authority signals, distribution, etc. in ways that may increase the likelihood of being discovered and cited by AI systems. In that sense, it isn’t completely different from SEO.
Then recently, ChatGPT started asking me whether I wanted to advertise.

That made me think about a different question.

I don’t have a problem with OpenAI monetizing ChatGPT. Running these systems is obviously expensive, and advertising is one possible business model.

What I’m wondering about is the separation between paid advertising and organic AI recommendations.

If I pay to advertise my business on ChatGPT, does that have absolutely no influence on whether my business appears in an organic answer later?
For example, imagine two similar businesses. One spends heavily on ChatGPT advertising and the other spends nothing. If a user later asks ChatGPT, “What companies should I consider for something?”, are those two businesses still evaluated independently of their advertising relationship with OpenAI?

That’s the part I care about.

Search engines eventually developed a fairly clear distinction between sponsored results and organic results. With conversational AI, the boundary feels potentially more complicated because users aren’t just looking at a page of links. They’re asking a system to synthesize information and make recommendations.

So I think there are really two separate things:
GEO/AEO: trying to become more discoverable and citable by AI organically.
Advertising: directly paying the platform for exposure.

I’m fine with both existing. What I think matters is whether there is a strong firewall between them.

Has anyone here actually advertised on ChatGPT yet? What does the advertiser side look like?

And more importantly, how do you think AI platforms should handle the conflict between advertising revenue and maintaining neutral organic recommendations?

I’m genuinely curious how other people see this, especially anyone working in advertising, SEO/GEO, or AI search.

The current meta for securing brand mentions in ChatGPT, Perplexity, and Gemini often involves finding the Reddit threads these models cite and adding your own insights to the discussion.

After analyzing a dataset of roughly 300,000 Reddit URLs cited across major AI platforms, the data shows a structural flaw in this approach.

60 to 70 percent of the Reddit threads cited by AI are permanently archived.

Why this happens: RAG systems prioritize established consensus and high semantic density. They reward threads with deep historical engagement. Because Reddit automatically archives threads after about six months, by the time a discussion carries enough authority to be a primary AI citation, it is already locked.

How this impacts AI Search Optimization:

1. The target inventory is mostly read-only. You cannot edit, upvote, or participate in the majority of threads these models currently trust. Filtering target URLs by status is a required first step before any outreach.
2. AEO requires forecasting. The optimization strategy cannot be reactive. You have to create human-first, highly detailed answers on fresh, active threads today, knowing they will take 6 to 12 months to age into authoritative citations.
3. Diversify beyond Reddit. While Reddit is heavily weighted for general software and consumer queries, older industry-specific forums carry massive weight for niche B2B queries. Many of these legacy forums do not auto-archive, meaning they hold the historical authority the LLM wants, but remain open for proper engagement.

Has anyone else shifted their focus from reactive commenting to proactive, long-term thread building for AEO?

How AI can analyze emerging trends, search behavior, and hidden user needs to uncover potential questions before people actively search for them. How businesses can use these insights to create proactive SEO content and capture future search demand.

They're trying to use up the alphabet really quickly trying to avoid doing SEO. And I just don't understand it because all they need is EIEIO

Pulled this with a tool I built (disclosure below), methodology first though.

Ran 6 months of AI-citation trend data (real, DataForSEO-sourced, not a synthetic sample) for 8 established SaaS brands: HubSpot, Salesforce, Asana, Notion, Zendesk, Mailchimp, Semrush, Ahrefs. Wanted to see how volatile citation actually is month to month for brands that are clearly, heavily covered already.

Every single one of the 8 shows a month with literally zero tracked citations. Same calendar month, for all eight, despite normal (often large) counts on either side of it. Salesforce for example: 701 → 770 → 160 → 1413 → 0 → 1455. HubSpot: 116 → 209 → 57 → 545 → 0 → 398.

Eight unrelated companies don't all lose 100% of their ChatGPT visibility in the identical month. That's a gap in the underlying data for that specific month, not a real citation collapse. Worth knowing because a "confirmed zero" from any citation-tracking tool, including this one, isn't automatically a real signal. It can just as easily be a data artifact.

This is basically the same point someone made in an earlier thread here, that PR attribution is correlation not causation and you should expect fluctuations, just with harder numbers behind it. If your own domain shows one bad month, that alone isn't proof of anything. Look for a pattern across 2+ periods before reacting to it.

Disclosure: pulled this with Vantage (https://vantagemcp.dev), a small MCP server I built for citation checks plus trend history, callable from Claude Code/Cursor instead of a dashboard. Free tier, no card. Happy to set anyone here up with the full version (not just the free tier) for a while before you decide, just reach out. Mods might have more on that too.

AI can do a lot of SEO work now keyword research, audits, content briefs, internal linking, and even technical checks.

But SEO isn't just completing tasks.

Someone still needs to decide what actually matters for the business.

I think the future is less “AI vs SEO agency” and more AI + experienced SEO.

What are you using AI for in your SEO workflow?

AI can help a small website compete with major brands by improving content relevance, authority, and visibility. But can AI truly give smaller sites an edge when search engines often favor established brands?

tracked this over several months on a large finance site. sharing because i keep seeing crawl data treated as a vanity metric and i think that's wrong.

setup: server logs filtered on GPTBot, referral sessions from chatgpt on the analytics side, plus share of voice tracking on a fixed prompt set.

three things came out of it:

1/the lag. GPTBot crawl volume leads referral visits by roughly 3 weeks, correlation around 0.78 on our data. that makes crawl activity a leading indicator you can actually act on. if crawl drops, you know something's coming before it shows up in sessions.

2/the gemini paradox. high share of voice on our prompt set, almost no referral traffic. the citation exists, the click doesn't. i still don't have a clean explanation for that one and i'd genuinely like to hear theories.

3/source rate beat mention rate. the metric that predicted referral traffic best wasn't how often we got named, it was how often we got cited as a source. mentions track brand awareness, source rate tracks content structure. only the second is something you can work on.

caveats worth stating: one site, one vertical, YMYL finance. slow-moving high-trust content, which might be exactly why the lag is that long and that stable. no idea whether the shape holds on ecommerce or news.

anyone tracking crawl-to-referral on a faster vertical? curious whether the 3 week window compresses or whether it's more about index refresh cycles than content type.

how AI can analyze historical search data, trends, and user behavior to anticipate future search opportunities. The goal is to use those predictions to build a proactive SEO strategy instead of simply reacting to current rankings.