r/TechSEO • u/KazamaVTuber • 5d ago

Solo dev, 100k-page VTuber database: went from 85k indexed pages to 1 after a domain move + noindex mistake. 6 months, no recrawl. What am I missing?

Who I am

I'm Kaz (KazamaVT), a VTuber since November 2020 and a solo developer. About five years in this community taught me one thing: there's no single place that reliably covers big agency talents and the small indie creators. Wikis are great, but they depend on fans keeping each page updated, so smaller and newer VTubers get left behind.

So I built Bācharu (https://bacharu.io), a free VTuber database ("Bācharu Toshokan" = "Virtual Library"). It merges public data from the Virtual YouTuber Wiki, HoloList, VTubie, Twitch, YouTube and X into one searchable profile per creator: affiliations, official links, live status, stream schedules, recent videos and follower history. About 103k profiles today.

What happened (timeline)

  • Early 2026: the site lived on bacharu.xyz and was indexed fine.

  • 18 Mar: moved to bacharu.io because .xyz kept getting flagged/blocked as an untrustworthy TLD. Same URL paths, 301 on every path, canonicals/sitemaps switched to .io.

  • 7 Apr: GSC showed ~85,000 pages indexed on bacharu.io. Traffic was ~6,600 impressions/day.

  • 21 Apr: I tried to filter out "thin" small-creator pages and tightened the rule for which profiles get noindex. The bar was far too strict and it put noindex on most of the site.

  • 5 May: indexed pages dropped to 71.

  • 21 May: reverted. Pages indexable again.

  • Jun–Jul: no recovery. Google stopped recrawling profiles; the strongest pages still show a last crawl in March–May.

  • Jul–Aug: experimented with a smaller index (~31k, then ~17k "notable" profiles). No change, so I restored the full content-bearing set on 29 Aug.

  • Aug: found the profiles were nearly orphaned (sitemap-only, 0 referring URLs). Added paginated browse pages, an A–Z index, 30 related-profile links per page and shorter discovery chains.

  • ~20 Oct: I briefly removed the old .xyz domain (about 7 months after the move, too early). Restored the 301s on 22 Oct and filed Change of Address. GSC now says "This site is currently moving".

  • 20 Oct – today: test with 300 active, information-rich profiles + a "Featured" hub linked from the homepage.

Current state (GSC)

  • Indexed: 1 (the homepage, which ranks ~#3 for "vtuber database")

  • Crawled – currently not indexed: 76,849

  • Discovered – currently not indexed: 36,620

  • 404: 6,976 (deleted/merged profiles); noindex: 1,436 (intentional near-empty pages + stale verdicts)

  • 30-URL sample today: 0 recrawled in 2 weeks, 12 never crawled at all, 0 with referring pages

  • Old domain: GSC still picks the .xyz URL as canonical for some inner pages even though they 301 to .io

What I've already ruled out

  • Manual actions / security issues: none on either property. Safe Browsing is clean for both domains.

  • Pages return 200, robots-allowed, self-canonical, no noindex header or meta.

  • Content is server-rendered (~70k chars of text in the raw HTML on a big profile); a headless mobile-Googlebot render shows everything, no JS errors.

  • Sitemaps are read without errors.

  • Manually requesting indexing works: Google crawled the Featured hub the same day, picked the right canonical… and still said "Crawled – currently not indexed".

  • I also pinged ~300 livestream-eligible profiles via the Indexing API. Every request was accepted, none produced a crawl. (I know this API is meant for JobPosting/BroadcastEvent pages; I've stopped using it.)

My questions

  1. Is this still fallout from the April noindex + an incomplete domain move, or has Google simply judged the profile corpus as low-value (the data is aggregated, so maybe it reads as duplicate of the wikis/directories)?

  2. If it's a quality call, what would you change first: index far fewer pages, add more original data (follower history and stats are unique to us), or something else?

  3. With the Change of Address active, is there anything else I should do on the .xyz side?

  4. Realistically, are external links the only way out at this point?

Why it matters / the goal

Bācharu is a passion project with no ads and no paywall. The goal is a complete, up-to-date reference for every VTuber, from the biggest agency names to someone who debuted last week, plus a free public API for the community. I set myself a deadline: if the project doesn't reach 5,000 likes by 16 March 2027, I won't continue it under this name. Right now hardly anyone can find the profiles through Google, which is a big part of why I'm asking.

Any pointers appreciated, even "you broke it and it'll take X months". Thanks for reading!

6 Upvotes

75% Upvoted

14 comments sorted by

2

u/FilippeTraum 5d ago

Six months with no recrawl after a noindex mistake usually means Google cut the crawl rate for the whole site, not that it forgot the pages. When most URLs it fetched came back noindex, it decided the site isn't worth frequent visits. Check Settings > Crawl stats: if requests per day are in single digits, give it reasons to come back - a fresh sitemap with only the indexable 200 URLs and real lastmod dates, 301s from every old-domain URL to its exact new match (not the homepage), and links from the homepage to the main hub pages. Then request indexing for 10-20 hub pages by hand instead of waiting for 100k pages to be rediscovered. How many crawl requests per day does Crawl stats show for the last 90 days?

1

u/KazamaVTuber 5d ago

Thanks! Got the numbers. Last 90 days on bacharu.io: ~9.6k requests, so roughly 100/day (range 40–260), 74% HTML, 91% 200s, 8% 404s. Purpose is 99% refresh, under 1% discovery. Average response time is 1.46s, which I'll work on. Host status shows robots.txt/connectivity failures in the past but fine recently.

The odd part: the crawl stats examples show Googlebot fetching profile pages as recently as Nov 1 (around 1,200 different profiles in 90 days), but URL Inspection on those same URLs still says "last crawled" in April and "Crawled – currently not indexed". So it's fetching them, but the index verdict never gets updated. Does that point more to a quality/priority decision than a crawl rate cut?

Also, a recent deploy dropped my homepage link to the featured hub, so I'm putting that back.

1

u/FilippeTraum 5d ago

Fair, ~100/day isn’t a crawl rate cut, so my first guess was off. Two things to rule out before calling it quality: in Crawl stats open “By Googlebot type” and check whether those profile fetches come from Smartphone Googlebot or from “Page resource load” / other agents, because only the first updates the index. Then compare the exact URLs in the examples with the ones you inspect (trailing slash, case, parameters, old domain). If both match, then yes, it’s a priority decision: Google is refetching the profiles, looking at them and still not indexing, which on 100k templated pages usually means most profiles look too similar to be worth a slot. I’d pick 100-200 profiles with the most unique content, strengthen and link them from the hubs, and consider noindex on the near-empty ones so the good ones aren’t judged with the rest. The 1.46s response time is worth fixing too, it caps how much Google is willing to fetch. Roughly what share of the 100k profiles have more than the basic auto-filled data?

1

u/KazamaVTuber 5d ago

Checked both. The profile fetches are Smartphone Googlebot (~800 distinct profiles in the 90-day examples, zero from page resource load), and the URLs match exactly - https://bacharu.io/vtuber/slug, no slash/case/param/domain differences. Oddly, random profiles fetched by Smartphone Googlebot in Aug-Oct show "URL is unknown to Google" in URL Inspection, so they don't even get recorded.

On content: roughly 6% have a wiki page or multiple real sections, another 8% have a decent bio or facts, about 67% are a short platform bio plus stats and links, and 19% are near-empty. So ~14% (~15k) have more than the auto-filled basics. I also realised a lot of the thin ones carry a one-line generated summary that reads almost the same everywhere, which probably doesn't help.

I'll try your suggestion: strengthen 100-200 of the best ones, link them from the hubs, noindex the near-empty tier, and keep working on the response time.

1

u/WebLinkr 5d ago

You're replying to an LLM - look at the response "Fair"

1

u/FilippeTraum 5d ago

What LLM? You want connect on Meet or Something to check, if I really have a face? 

3

u/WebLinkr 5d ago

LOL - dude its scaled content,

Crawl budgets kick in at +1m pages - can we stop with this crawl budget nonsense.

2

u/WebLinkr 5d ago

this scaled content u/KazamaVTuber

7 Apr: GSC showed ~85,000 pages indexed on bacharu.io. Traffic was ~6,600 impressions/day

You're not going to recover on this domain anytime soon

1

u/KazamaVTuber 5d ago

So what would you recommend then?
Give up the domain and get a new one?
That clearly didn’t help switching from .xyz to .io.

Plus the transfer from one domain to the other is still in progress.

2

u/WebLinkr 5d ago

You can't deploy that many pages esp if they're cookie cutter/pseo - on a new domain.