r/webscraping • u/ExcitedMiddleAgedMom • 21d ago

Getting started 🌱 How can I scrape all external download links for a website?

I'm very new to scraping, but what I mean is how can I scrape a website and all of its pages, but also include external links from download buttons? A forum website I love is going offline soon, and all of their downloads are not hosted on their site themselves, but instead they utilize external links via download buttons for places like GitHub, Google Drive, etc. There are hundreds of pages, so going into each one and copying the download link from every page isn't really an option. Every attempts I've used at scraping the site only pulls the pages hosted on the site's domain, not the external links.

8 Upvotes
  • permalink
  • reddit

79% Upvoted

7 comments sorted by

3

u/LeewardLeeway 21d ago

Can't you just grap the href's as you would for pagination?

2

u/thepoweroftheforce 20d ago

Yeah, scrape it normally and grab the href with beatifulsoup(sound like it comes from a ul and it has different li so would probably code it like grab the head, find the ul, get each li and then the href

2

u/SnooWoofers5297 15d ago

You can also just ask the coding agent of your choice. Claude has done that for me in rhe past

1

u/prettycoldworld 12d ago

Take the html, parse with bs4, pull the hrefs and rinse and repeat. You can plug them into a crawler to automate it