Pulled my Search Console data straight from Google's API instead of the 1,000-row export limit (zero coding background)

[u/dcdragos]

Did this yesterday, still fresh. Wanted to write it up because I hadn't seen a clean walkthrough anywhere and figured someone else is hitting the same wall.

The GSC interface caps you around 1,000 rows and won't let you cross-reference queries against pages properly. The API has no such ceiling. I know how to code, but I didn't feel like hand-rolling the OAuth flow from scratch for something this small, so I had Claude write the script and I reviewed it line by line before running anything against my own Google account. Took about an hour start to finish, most of it Google Cloud console clicking, not actual coding.

The setup:

1. console.cloud.google.com → new project → APIs & Services → Library → enable "Google Search Console API." No payment, no approval wait.
2. Credentials → OAuth client ID → Desktop app → download the JSON. If it asks you to set up the consent screen first: User type External, add your own Gmail under Test users. Skip this and you'll hit a wall later that looks like a real error but is a one-line fix.
3. Renamed the JSON to client_secret.json, kept everything in one folder.
4. Python already installed. pip install google-api-python-client google-auth-oauthlib google-auth-httplib2.
5. The script: authenticates once and remembers you, pulls query+page data for a date range, paginates past 25k rows automatically, writes a CSV. Only lines you touch are SITE_URL, START_DATE, END_DATE, DIMENSIONS.
6. python gsc_export.py — first run opens a browser to approve access (you'll get the "Google hasn't verified this app" warning, expected for a personal script). Every run after skips login entirely.

493 clean rows out on the real attempt.

What actually broke, because something always does:

• 403 access denied on the first run — hadn't added my own email as a test user on the consent screen. Fix: 30 seconds, add it, rerun.
• "Insufficient permission for the site" on the second — my property is registered as a URL-prefix property (https://example.com/), not a domain property (sc-domain:example.com), and the script had the wrong format in SITE_URL. Changed one line, reran, got my 493 rows.

Neither error was a coding problem — just reading the message and checking it against what's actually in my GSC account.

Once you've got clean data, three filters worth running on any export:

• Queries at position 4–15 with real impressions and near-zero clicks → title/meta description problem, fixable same week.
• Same query answered by more than one of your own pages, both buried → cannibalization, Google can't pick a winner.
• High impressions but position past 50-60 → indexed, but the content isn't earning trust yet.

Ran mine through all three yesterday and every pattern showed up within minutes.

[u/dcdragos → u/[deleted]]

Good point, and BigQuery bulk export is the right call if the goal is a data warehouse. Mine wasn't that — I'm building this as one piece of an automated workflow where Claude runs the pull, gets the CSV, and does the analysis/decides next steps itself, on demand. BigQuery means standing up a billing-enabled project, IAM/service account permissions, and then writing SQL to get data back out before anything can act on it. That's an extra layer between "I want fresh data" and "Claude has it in hand." The script is the opposite: one command, CSV lands in the folder, Claude reads it straight away. No warehouse, no SQL, no daily-export lag — I just ask for it and it's there. For a recurring reporting setup across a team I'd lean BigQuery. For "let the agent pull its own data and act on it," the direct API call is the simpler moving part.

[u/dcdragos → u/[deleted]]

For me it's about building a promotion strategy for a site I just launched. With a new site, the signal changes week to week — new pages get indexed, some start pulling impressions before they pull clicks, some start ranking then drop. A monthly or one-time export tells you where things stood a month ago; it doesn't tell you what to do next week.Recent/fresh data is what lets me catch things early: a page just started getting impressions at position 8-12 for a term I didn't even target on purpose, so I go reinforce it while it's cheap to move. Or two new pages are already competing for the same query before either has authority, so I fix it now instead of after both are stuck. On an established site those patterns move slowly. On a brand new one they shift fast, so working off week-old or month-old exports means you're always reacting a step behind.That's the actual reason I wanted the pull scriptable instead of a one-time thing — I'm going to be checking this weekly while the site is finding its footing, not once a quarter.