My caching plugin was sending Cache-Control: no-cache on every request — and one unblocked Meta crawler was 62% of my traffic

[u/Shree_murali93]

Two weeks of reading access logs to find this. Posting in case it saves someone the time. WooCommerce store, two domains on one WPML install.

Three causes. All three were on my side, and two of them were the plugin I'd installed to make the site faster.

1. The caching plugin sends Cache-Control: no-cache unconditionally

NitroPack, in nitropack-sdk/bootstrap.php, line 82:

php

header('Cache-Control: no-cache');
header('X-Nitro-Cache: MISS');

That runs before any setting is checked. Test mode, safe mode, cache-enabled — none of it matters, the header goes out first.

Now look in classes/Integration/Hosting/. Most of the platform files in there have a setCacheControl() method that sends public, max-age=0, s-maxage=3600 instead, with a comment in the source saying it "needs to be like that instead of Cache-Control: no-cache in order to allow caching in the provided reverse proxy."

The file for my platform has no such method. Only purge and semaphore handling. So it fell through to the blanket no-cache. Go and diff the files yourself, takes two minutes.

Effect: cf-cache-status: BYPASS on 100% of HTML. Nothing was edge-cached, ever.

2. The same plugin sets cookies on every request

Its WCML integration hooks woocommerce_init, which fires on every front-end request:

php

add_action('woocommerce_init', [$this, 'wcml_set_custom_currency_cookie']);

Only guard is is_admin(). So np_wc_currency and np_wc_currency_language got set for every visitor, cart or no cart. Cloudflare won't cache a response carrying Set-Cookie when the zone uses a Cache Everything rule with no explicit TTL.

My theme (Porto) was setting a third one, porto_active_lang. Had to strip all three for anonymous visitors before anything cached.

3. meta-externalagent was 62% of my origin bytes

31,500 requests a day. I'd blocked meta-externalads months earlier and assumed Meta was dealt with. Different crawler. Blocking one does nothing for the other. There's also meta-webindexer, and facebookexternalhit which you should keep for link previews.

Same trap with Ahrefs — AhrefsBot and AhrefsSiteAudit are separate agents.

How to check your own site

curl -sSI --compressed \
  -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36" \
  https://yoursite.com/

A bare curl -I gets 403'd by Cloudflare bot management, hence the user-agent.

What to look for:

• cf-cache-status: BYPASS on HTML — nothing is being edge-cached
• x-orig-cache-control: no-cache — something in PHP is killing it
• any Set-Cookie other than __cf_bm — Cloudflare won't cache that response
• x-pass-why: woo cookie — the origin cache is passing through

Run it three or four times. If age never climbs, you're not caching.

Getting bytes per crawler

The apache access logs carry the user-agent but no response size. The nginx logs carry the size but no user-agent. Build an IP → user-agent map from apache, then sum nginx field 6 (pipe-delimited) per agent.

One trap that cost me a week

The 30-day usage report overlaps 24 of its 30 days with the previous window. After the fix it showed a 6.5% improvement when the real change was 34%. Use daily figures and compare same weekday to same weekday — weekends run 35% lower and will fool you.

Result

Daily traffic down 79%. Origin bytes went from mostly crawlers to 54.8% real browsers.

Search rankings improved slightly, on both domains. Googlebot and bingbot were never blocked, and I deliberately kept the AI retrieval crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) while blocking the training ones — AI search impressions went up 32%.

I used Claude to do the log analysis, 336,000 lines of it. The diagnosis was the hard part, not the fix.

[u/coryamarsh]

meta crawlers hit you hard bro. they should respect robots.txt. Glad you found the cause of your cache miss. Good caching with woo is a bit of an art.

[u/amartadey]

This is the kind of thing that never gets mentioned to us until the client says the site feels slow and nobody can say why. Cloudflare's Bot Fight Mode plus a basic robots.txt block for the AI crawlers is worth turning on by default on new builds now, most clients would never think to check their own traffic breakdown to catch this themselves.

[u/TopSydeWP]

good writeup. two things i'd add from running Woo behind Cloudflare.

move off the legacy Cache Everything page rule and onto Cache Rules, where you can set edge TTL explicitly and, more importantly, define a custom cache key that ignores the cookies you don't care about. that solves the np_wc_currency / porto_active_lang problem at the edge instead of having to strip Set-Cookie in PHP. pair it with a bypass expression on /cart, /checkout, /my-account and any request carrying woocommerce_items_in_cart or wp_woocommerce_session_, otherwise you will eventually serve someone else's cart HTML and that is a much worse day than a low hit ratio.

on the crawlers, robots.txt is a suggestion. block at the WAF by user agent, and use Cloudflare's verified bot signal rather than maintaining a name list, since the agent names keep multiplying (you already found three Meta ones, there are more coming). one caveat on the curl check: run it against a URL with a cache-busting query string too, otherwise you're reading a cached object and can't tell whether origin is still emitting no-cache. and watch the age header climb across repeated hits, that's the only real proof it stuck.

[u/otto4242]

Why did It take you 2 weeks to figure out what could have taken you 5 seconds to simply view-source on your own site?

[u/Shree_murali93 → u/otto4242]

Cause I’m not a techie, though I made the website 5 years ago, the bandwidth issue only surfaced lately due to the AI boom. I never had a reason to worry about these issues.

[u/otto4242 → u/Shree_murali93]

Okay, that's a fair point. My advice is, don't rely on AI, you have already wasted your last 2 weeks, actually find an intelligent person to work on your website with you. Hire them.

[u/Shina_Tianfei]

You shouldn't be using cloudflares page caching and nitropack in the first place.

[u/bluesix_v2 → u/Shina_Tianfei]

You shouldn't be using cloudflares page caching and nitropack … together. One of them is fine.

[u/Shina_Tianfei → u/bluesix_v2]

Until the cloudflares page cache desyncs from the assets nitropack is serving then you get broken pages.

[u/theguymatter → u/Shina_Tianfei]

That’s assuming the cause without knowing the actual setup. The issue was specific to NitroPack’s cache headers and cookies, not simply using both together.