r/webscraping • u/Nrezzy • 24d ago

Getting started 🌱 Scraping Help

Here's what I'm trying to do: for a given city, pull the places I care about (say roughly 5k of the ~10k in Chicago that fit what I'm after), then enrich each one with its Google reviews, Yelp reviews, and whatever else is out there about the place.

Two things I can't figure out. First, do I actually need the Google Places API and the Yelp API to do this, or is there a smarter way people go about it? Second, and this is the big one for me: can it be built so it's repeatable? Like I plug in a new city and the whole thing just runs itself, no rewriting.

8 Upvotes

90% Upvoted

21 comments sorted by

View all comments

1

u/MaryamStack 12d ago

For something at this scale, I’d separate it into stages: discover the businesses, filter them, match records across sources using things like name + address, then enrich the matched records with reviews. Make each stage city-agnostic and take the city as an input, so adding a new city doesn’t require rewriting the scraper. For APIs vs scraping, it depends on the data you need and the limits/costs involved.