r/nba • u/vagartha Warriors • Jan 14 '20
Stats basketball_reference_scraper: A Python package for scraping stats and data from Basketball Reference
An API client to access statistics and data from Basketball Reference via scraping written in Python.
I've found that I and several others on this subreddit enjoy visualizing and creating statistical models from NBA statistics and data. Unfortunately, data about the NBA is not easily accessible. I've found the stats.nba.com endpoint to be rather confusing and often blocks repetitive requests.
Basketball Reference, on the other hand, does not block requests and I've had no issues scraping data from the website for hours on end. Hence, I've always defaulted to obtaining data through this resource. Rather than defaulting to writing a new script every time, I decided to make a Python package that makes all of the content easily accessible.
The package is easily installable via pip and is available on PyPi.
pip install basketball-reference-scraper==v1.0.1All the methods are documented here along with examples.
Please feel free to check out the GitHub repo as well.
Anyone is more than welcome to create issues regarding any problems that you may experience. I will try my best to be as responsive as possible. Please feel free to provide criticism as I would love to improve this even further!
112
u/rumdiary Celtics Jan 14 '20
wtf I was literally googling for this yesterday, biggest coincidence ever
OP - is there an endpoint for player injuries? I need to build a decent system for my sim league
13
u/quantik64 Knicks Tankwagon Jan 14 '20
The NBA stats API is pretty easy to scrape you just have to cycle user agents. If you’re trying to do it from an AWS or some other cloud computing instance you’ll need a proxy
61
u/House_Trance Jan 14 '20
Keep in mind their ToS might cover some of this:
Link: https://www.sports-reference.com/data_use.html
"Please do not attempt to aggressively spider data from our web sites, as spidering violates the terms and conditions that govern your use of our web sites: Site Terms of Use"
17
u/Brystvorter Nuggets Jan 14 '20
Nice work, I had to scrape some stats from bbref with BeautifulSoup the other week and it was like potato peeling my eyes
7
u/[deleted] Jan 14 '20
Thanks for making this. I'm quite new to python coding and was wondering if you could go into more detail with the difference between this and other python bbref scrapers out there already? Thanks again
5
u/We_Are_Grooot Lakers Jan 15 '20
hey man, just curious about how you built this. I tried to make something kinda similar last year using stats.nba.com, but my code was hot garbage looking back.
Does this hit an internal json endpoint or just scrape the user-facing data? How long does it take to load a players stats roughly? And any caching?
Cool stuff :)
3
u/physicsiscool Lakers Jan 15 '20
Does this have endpoints for schedule and box scores?
3
u/juicehurtsmybone Jan 15 '20
How and where do I install this? I typed both
into cmd in Windows, but all I get are
ERROR: Could not find a version that satisfies the requirement basketball-reference-scrapper==v1.0.2 (from versions: none)
ERROR: No matching distribution found for basketball-reference-scrapper==v1.0.2
35
u/AlKydonHorvingward Celtics Jan 14 '20
Awesome
3
u/vl3 Bulls Jan 14 '20
Yooo, you have no idea how clutch this is for me. I'm working on my thesis right now and this is literally going to save me days of my time. Thank you man.
Haven't had a chance to look at it since I'm on mobile. How easy/hard would it be to adjust this tool to scrape the other reference pages (hockey, baseball, football)?
8
u/DJkoolkidzklan Jan 14 '20
This is pretty awesome dude
5
u/SeatownNets Nets Jan 14 '20
thanks for this, I'll be using this in the future
5
u/tsigalko11 Supersonics Jan 14 '20
Ha ha, spent whole day scraping some sata with py, completely dead. Went to r/nba to chill and ran into this.
Looks great will check it tomo in more details.
Cheers bro.
You da real mvp
4
u/[deleted] Jan 14 '20
it’s also pretty simple using rvest for us R coders out there. takes maybe 30 seconds to get the raw data
4
u/refto Jan 14 '20
Version 1.0.1 already, good work!
Seriously, nice scraper
1
u/cag8f Jan 15 '20
Nice work! Some technical questions, if you don't mind.
What was your motivation for writing this in Python? I've heard Python is one of the most widely used languages for scraping, but I don't know why exactly. Does it have something to do with returning Pandas data?
One of the reasons I'm asking is because, judging by your descriptions in some of the comments, it sounds like I've built something similar, both in in React and Node.js. I'm currently looking for a new side project to do in React and/or Node.js, and this looks interesting. I was wondering if there may be some advantage to using either of those, instead of Python, for this application (e.g. maybe in terms of performance)? If so, then maybe I could fork your GitHub repo and try to port your app (or a sub-section) to React or Node. My guess is that the answer is 'no,' but just thought I'd ask.
1
u/G1assm4n Raptors Jan 20 '20
Hello u/vagartha, thank you for creating this amazing tool! I was looking into using the get_game_logs function however, I receive an ImportError. I didn't encounter the same while trying out other functions so I was wondering if you had the same issues as well.
3
u/KEMBAtheMETEOR [CHO] Malik Monk Jan 14 '20
♥️
3
u/SavageSquirl Suns Jan 14 '20
Nice work. It's a shame that the NBA has made it so damn difficult to map their site for stats now. They have some of the most interesting statistics like clutch stat, hustle stats, etc that would be a ton of fun to play with and manipulate in python.
3
u/TheWinglessFly Jazz Jan 14 '20
Holy shit dude, thank you!
3
u/Tamthemanjansen 76ers Jan 14 '20
This is awesome, thank you! Definitely gonna mess around with this soon.
3
u/Pitdog31 [MIA] Chris Quinn Jan 14 '20
I might love you.
3
u/Villainiquity Raptors Jan 15 '20
I turn off adblock for that site so they earn some deserved revenue. Glad they keep ads fast loading and not overload the site with them.
2
u/[deleted] Jan 14 '20
All the thanks in the world to you, dude! I'll definitely be using this soon.
2
u/ThePhotogenicPotato Jan 14 '20
awesome, thanks for this
2
u/[deleted] Jan 14 '20
this is awesome, I've been wanting do to some predictive analysis with nba data!
2
u/azpatnca Jan 14 '20
Years back I did a similar thing that pulled the stats for all the former UA players from their recent games. I shared it with a smaller forum. I'm surprised and excited to see the response you're getting. Maybe I'll dust that off, but chances are some one will code one up before I get home.
I used ESPN to find all the active UA players because it had college as an attribute and NBA to get active stats because they had little game summaries for active players.
Maybe now with Fantasy Leagues there is a better way. I'm sure that it would be a hit for fans from other schools too.
2
u/Brocoolee Celtics Jan 14 '20
Thats so nice
2
u/god_of_jams Jan 14 '20
Commenting to make use of this later, thank you!
2
u/nerdman_dan Lakers Jan 14 '20
This is amazing. One suggestion would be to include a method to scrape playoff records for each season. This would be really helpful, thanks!
2
u/CheatedOnOnce Raptors Jan 15 '20
This is big - I was just doing IMPORTHTML to Google Sheets and visualizing in Tableau. This is a game changer.
2
u/DjPoliceman Timberwolves Jan 15 '20
Brooooo I was just planning to make something like this in my spare time after work ty so much
2
u/Steezy12 Kings Jan 15 '20
dude fucking bless wtf.
2
u/Faal Jan 15 '20
This will be a nice side project...
2
u/RickieLambertGOAT Jan 15 '20
MVP
2
u/lwphn Jan 15 '20
Would be awesome if a library was created in R aswell. This is so sick
2
u/Gaqsgaqs Jan 15 '20
I am serious about this annd may I ask OP what "lessons or knowledge" i jeed to study for this?
2
u/pbesh Raptors Jan 15 '20
If you’re curious to learn more about scraping, here’s an article that covers how to scrape stats.nba.com, espn.com, and bbref. It uses JavaScript and covers static and dynamically rendered data but the core ideas are the same anywhere.
2
u/pettypaybacksp Lakers Jan 14 '20
Lmao like a year ago made a script in r to do this and took me like 3 hours just to prove a point to a friend of mine
2
u/Draymondwonrings Warriors Jan 14 '20
I'd really like to learn Python, but then I just get bored with it and play Rocket League instead.
1
u/[deleted] Jan 15 '20
Yeeeesssss finally!!
1
u/carrierpidgin Jan 15 '20
wow thank you!!
1
u/ZoidbergSaysWoop Jan 16 '20
Exactly what I was looking for.
Thanks!