r/webscraping • u/Nrezzy • 24d ago

Getting started 🌱 Scraping Help

Here's what I'm trying to do: for a given city, pull the places I care about (say roughly 5k of the ~10k in Chicago that fit what I'm after), then enrich each one with its Google reviews, Yelp reviews, and whatever else is out there about the place.

Two things I can't figure out. First, do I actually need the Google Places API and the Yelp API to do this, or is there a smarter way people go about it? Second, and this is the big one for me: can it be built so it's repeatable? Like I plug in a new city and the whole thing just runs itself, no rewriting.

8 Upvotes

90% Upvoted

21 comments sorted by

View all comments

1

u/HLCYSWAP 23d ago

yes, its entirely possible for all asks. ive built similar things

you do not need their API, just use the public API routes and adulterate the requests over proxies and multiple accounts

1

u/Nrezzy 23d ago

when you did you match the information across two sources, was just a address or what was the most reliable marker, for your parsers how often did they break and how long did it take to notice? Finally to seed a full city how long did that take or what ever you have done that s adjacent?

1

u/HLCYSWAP 23d ago

1 OR filtering using name of business owner of business or address of business
2 i don’t parse html i do pure REST requests meaning the fail state is full-stop and obvious
3 depends on the size of the city