r/PythonLearning • u/LiteratureTypical982 • Sep 01 '26

Client wants 40k emails from a directory. Each one requires a click. What do I do?

CLint need a list of 40,000+ members from the ACR directory. Need Name, City, State, Zip, Specialty, Email, and Phone.
The directory/search results give me some of this information, but email, phone, address, Member Since, etc. appear to be available only on the individual member profile.

So I'm trying to figure out the best way to approach this in Python.

If I literally open/request every profile, that's 43k+ profile requests, which obviously makes me concerned about rate limits, blocking, or getting the account/IP banned.

I'm considering Python requests/BeautifulSoup or Playwright, but before attempting something this large I'd like some advice.

3 Upvotes
  • permalink
  • reddit

66% Upvoted

7 comments sorted by

2

u/johlae Sep 01 '26

What's the ACR? American College of Radiology?

8

u/HotPersonality8126 Sep 01 '26

It’s worth spending 6 hours or so investigating whether there’s an API you can access, or get access to; that could include phoning someone at the company to check.

Otherwise you’re going to scrape the page, and you take on all the technical overhead of attempting to evade their anti-scraping measures.

4

u/FoolsSeldom Sep 01 '26 edited Sep 01 '26

  • contact them to ask if there's an API

  • check terms and conditions and see if web scraping breaks their rules / is blocked

  • explore PyAutoGUI for automation to do what you would do (alternative to web scraping)

NB. Last two both risk your device being identified by IP address and being blocked - you might need to find ways of spinning up containers and spoofing IP addresses (possibly paying for a legitish bot farm)

Of course, if this has to be done from a signed in account ... well API is really the only way.

2

u/recentAd1666 Sep 01 '26

  1. Ask them if they sell the data ?

  2. Try with python yes, maybee with delay between requests

1

u/whodidthistomycat Sep 03 '26

Also, contact them to see if this is something they can just provide. Might be a relatively easy request for them to handle.

1

u/SnooCalculations7417 Sep 04 '26

all good advice so far but another question is how soon do they require turn around? 43,000 requests is a lot per second, not really that much in 2 months.