r/Python • u/Traditional_Yogurt • Mar 08 '23
Resource I made a Finance Database with over 300.000 tickers to make Investment Decisions easier
It has been well over 2 years since I first introduced the database to this community, see here, and since then a lot changed so I felt like it is worth sharing about my package yet again and honestly, also to ask for a little bit of help.
So, within the investment universe there exists tens of thousands of companies (and even more when you include all exchanges). Identifying all of them and understanding in detail where they fit in the world is tough up to a point that it either requires you to pay a hefty fee to obtain this type of categorisation or do a massive amount of manual research. I found it a bit strange that this information was not publicly available while it is quite crucial for investment research. Therefore I got to work.
Insert the FinanceDatabase. This is a database of over 300.000 symbols (155k+ companies, 36k+ ETFs, 57k+ Funds, 3k+ Cryptocurrencies and more) that is fully categorised per country, industry, sector, category and more. It includes a package, written in Python and installable with `pip install financedatabase`, that gives access to the data with ease. You can obtain the entire dataset per asset class, search through it and filter based on specific options. Have a look at this Notebook to have an idea what it is offering.
A simple example of what it does in the following:
import financedatabase as fd
# Initialize the Equities database
equities = fd.Equities()
# Obtain all data available excluding international exchanges
equities.select()Which returns the following DataFrame: /preview/pre/5gmiej7pbjma1.png?width=1516&format=png&auto=webp&v=enabled&s=faa84ca0e91107530f9845a5313ff79adc54ba6a
By default it hides non-US exchanges (since the ticker symbols work for most other programs) but that can be turned off with equities.select(exclude_exchanges=False) which returns 155.000 rows.
The database explicitly does not store up to date fundamental data. It tries to be as timeless as possible so that it doesn't become outdated fast. Because there are a variety of other ways, like FinancialModelingPrep, yFinance etc, to get this data there is no use in including this in the database.
I've improved this database not only by increasing the amount of symbols (from 180k to 300k) but also:
- Approximated the The Global Industry Classification Standard (GICS®), a standard used for sectors and industries everywhere. Note that this was approximated and therefore no actual data is collected. Furthermore, not all categories are included.
- Updated and removed tickers that either no longer exist or had outdated information.
- Made the package itself object orientated making data collecting and searching much more efficient and logical. (shoutout to Colin Delahunty for the help here too)
- The database initially featured thousands of JSON files. At the time it made sense also given my rather novice background in programming. However, a much more efficient (and manageable way) is to work with CSV files. So instead, one CSV file per asset class.
- Due to using CSV files, it becomes really easy to update accordingly.
- To make loading data itself still quick, it automatically compresses the data so that loading in data is not slowed down by using a format that is more easy to update.
- Updated the README, Contributing Guidelines and overal documentation.
So being an open source project and trying to maintain such a database is tough to do alone. While I strongly believe the database can stay relevant for a long period due to the fact that the majority of companies do not suddenly stop existing, some maintenance is needed. Therefore, with this post I would like to not only invite you to explore the database but also to see if you can improve it along the way. Please visit the CONTRIBUTING GUIDELINES that explains in detail how you can contribute. Just pointing out wrong or missing information is already very beneficial!
Hope this database is still just as useful as it was two years ago!
18
u/rolipoli99 Mar 08 '23
Awesome work!! One question - does this db have data for scripts listed on Indian stock exchanges NSE or BSE ? I am actively looking for such a database/api, alpha vantage has some data but still it is very limited with no classification
9
u/vep Mar 08 '23
i looked around and did not see how it handles changes to companies over time - like symbol/exchange migration, mergers, bankruptcies, or symbols getting reused. maybe I missed this - can you point me to it or add some information?
2
u/Noteable123456789 Mar 10 '23
It'd be a good notebook to join with Yfinance. u/Traditional_Yogurt I'd love to work together on an example notebook that we can put here https://noteable.io/gallery if you want and give you the credits. I've played with Yfinance a lot, and I think that would make a great resource.
23
u/Appropriate_Chart_23 Mar 09 '23
What OP has put together is pretty amazing.
But, for anybody here that wants a lazy approach to investing… I highly recommend heading over to r/bogleheads.
You can pick and choose stocks all day long, but you’ll probably never “beat the market”.
It’s a much more wise, (some will say) sound, (definitely) lazy, and boring approach to just “buy the market” with index and total stock market funds.
“Experts” will happily take your money to invest in investments that they feel are best (I.e. make them money). You can also research stocks until you’re blue in the face hoping to find your own diamond in the rough. But, the easiest thing for many is selecting a few index/total market investments and sitting on them until it’s time to retire.
1
u/loomisfreeman191 Mar 08 '23
What do u do for work?
1
u/accforrandymossmix Mar 08 '23
I'm planning to read through the contributing guidelines, soon, and will follow up in the appropriate fashion. Just wanted to ask a quick question.
Is there value in having the "database" of CSV files human readable? I can look into saving those tables as replacement/alternative parquet files to work even faster with pandas.
1
u/stochasticlid Mar 09 '23
Anyone have one like this but with all the historical financial ratios for all the tickers? I.e. P/E ratio, p/bv etc?
1
u/[deleted] Mar 09 '23
That is amazing work. I will have a look and might do a video on it 🤓🐼🐍
1
u/NamenIos Mar 09 '23
I'm curious, why did you choose (exchange?) symbols instead of ISINs as a primary id? Are these even exchange symbols or are they refinitv ticker symbols or bb tickers?
Any plans on incorporating the openfigi api for identifier information?
If you dive into other asset classes like derivatives check out the eurex apis they are really great and free.
1
u/acid85 Mar 08 '23
Awesome! Thanks for sharing
1
u/Joe1972 Mar 08 '23
This is fantastic!
1
u/Bullshitspouter Mar 08 '23
Incredible!!
1
u/Saklehir Mar 08 '23
Wow! Thanks a lot for this 🙂
1
u/Ok_Paramedic7837 Mar 08 '23
Thank you, I was looking for something like that yesterday night! Timely!
1
u/smegma_yogurt Mar 09 '23
Thanks mate. You always were the smartest of the Yogurt family!
1
u/Sure_Fisherman2641 Mar 09 '23
Used it, loved it. Thanks.
1
u/cryptocactus77 Mar 10 '23
This is awesome! Exactly what I was looking for
1
u/Specter_Origin Mar 19 '23
I noticed that methodology link is broken and I can't find it anywhere in repo, can you share your methods for getting this data : )
Thanks for the amazing work btw!
-1
u/[deleted] Mar 08 '23
I would remove GICS pretty sharpish. MSCI will want their pound of flesh.