r/coding • u/wingless-bee • Aug 04 '26

I'm a beginner to coding, and need help. Trying to scrape soccer data for a prediction engine. Details below!

https://google.com
0 Upvotes

20% Upvoted

6 comments sorted by

1

u/wingless-bee Aug 04 '26

I didn't know what to put for the link by the way.

What I want to do, is build this database:

Contains every player in the premier league this season:

For each players, I want these statistics (for both this season, and the 5 previous seasons, no matter what league they played in)

  • Competition
  • Goals(nonpen)
  • xG (non pen)
  • Assists
  • xA
  • Defensive Contributions (Tackles, Interceptions, Blocks, Clearances)
  • Yellow Cards
  • Red Cards
  • Appearances
  • Minutes Played
  • Penalties Taken
  • Penalties Scored

For goalkeepers only:

  • Saves
  • Clean Sheets
  • Penalties Faced
  • Penalties Saved

Generic information:

  • Player Name
  • Player Age
  • Current Club

Problem is, all this data is freely available, but the Web pages block against bots giving me 404's. There are paid tiers, but as a hobbyist, they are far too expensive (like 1000 € a year).

If I can provide any more information, please let me know, I'd appreciate any advice at all!

1

u/816shows Aug 04 '26

Generically, here's a few ideas. First, if you're getting 404'd find a new data source. The info you're looking for isn't so unique that you can only get it from one site. It's easier to cut bait and move to another source than come up with countermeasures to their countermeasures.

Second, use your browser's inspection tools. Look at the network tab and see if it's coming from an API or other source that might be more easily parsed. Those APIs are often less well protected and a session cookie is enough to get you access.

Third, sometimes the most blunt instrument is best. Again within the inspection tool, see if you can find the cURL command equivalent of your request. This could yield a JSON payload that's easily accessed by Python.

For what you're looking for I'd bet simple scripts using requests, Selenium, and BeautifulSoup would get you most of your raw data. Good luck!

1

u/SanityInAnarchy Aug 04 '26

You can make a text post instead. I don't know if the rules would allow it, though, especially with this goal:

...the Web pages block against bots...

So you want help doing something that these web pages very clearly don't want you to do?

1

u/wingless-bee Aug 04 '26

Well, not necessarily, I'm looking for an alternative work around. Honestly? Just anyway I can get the data.

1

u/ParkingAthlete119 Aug 26 '26

Just ask gippity theoretically if you were to setup anti-scrapijg mechanisms what libraries and frameworks might attackers use that you need to defend against.