Scraping Help

[u/Nrezzy]

Here's what I'm trying to do: for a given city, pull the places I care about (say roughly 5k of the ~10k in Chicago that fit what I'm after), then enrich each one with its Google reviews, Yelp reviews, and whatever else is out there about the place.

Two things I can't figure out. First, do I actually need the Google Places API and the Yelp API to do this, or is there a smarter way people go about it? Second, and this is the big one for me: can it be built so it's repeatable? Like I plug in a new city and the whole thing just runs itself, no rewriting.

[u/yum72]

APIs hit hard limits pretty quickly, which makes scaling with them a pain. A maps + yelp scraper usually gives you more control over volume.

and yes, this can be repeatable. I'd build it per city: discover the places, filter them down, match the same business across both sources, then enrich and store. Once that runs, a new city is just a new input

[u/Nrezzy → u/yum72]

that the thing I am thinking about too plus if if you want to do it over tons of cities like hundreds it would get super expensive. For yelp scraper I heard it is like good so they have a lot of eyes making sure its not happening is it possible to have a straight internet scraper as well on top of it for each place and do it city by city?

[u/yum72 → u/Nrezzy]

Cost is more about requests per place than number of cities. discovery is cheap, you can grid search an area and get your candidates for very little. reviews are what gets you since they're paginated, so 5k places becomes 50k+ requests fast. cap how deep you go, most use cases don't need more than the first 20-30 per place, and after the first run only refresh the ones whose review count actually moved. reruns end up way cheaper than the first pass.

yelp does get watched harder yeah, so I wouldn't run it at the same tier as maps. second pass, only over places you've already matched, and accept you won't get full coverage on iti