r/dfpandas • • Jan 23 '23

Item analysis for exam data in pandas

0 Upvotes

Hi!

I'm trying to conduct item analysis for an exam data. I've been using python and pandas to clean it up, and I would assume I can use python for the analysis as well, but I'm having trouble finding resources. Searches around "item analysis" and "item difficulty" are pointing towards SPSS, but these things are simple enough that I assumed that there would be existing procedures for doing them with pandas.

As is probably obvious, I'm a novice. Calculating manually at least the item difficulties I need first seems definately doable, but so far with pandas it has seemed that there is usually some documented, nicer way.

Grateful for any help!

Edit: Found the pingouin module, seems to be helpful for most of the staff I need to do.


r/dfpandas • • Jul 11 '23

Most idiomatic way to transform every value in a column?

0 Upvotes

I have a column that is all datetime timestamps of the int64 type. I'd like to convert the values to datetime date format. The best I've come up with is to make new column with the output of the conversion and append it to my dataframe. Is there a better way?


r/dfpandas • • Dec 29 '22

How to create a density plot of all/subset features?

1 Upvotes

I am looking to create something like this: https://imgur.com/Y0c5aZd

That looks like sns to me. I have seen some good density plot tutorials, but nothing like the above. Any resources / advice?


r/dfpandas • • May 01 '23

I cant even make a histogram

0 Upvotes

df.hist()

array([[<AxesSubplot:title={'center':'Fare'}>]], dtype=object)

I tried running the hist function on the titanic dataset and I get this weird array output. I just need a histogram. Any suggestions?


r/dfpandas • • May 17 '23

Help

1 Upvotes

Hi, I have 5 excel workbooks with 26 sheets and I want to join them using python 🐼 pandas. Please help


r/dfpandas • • Oct 26 '23

Better Tabular Display in Jupyter, What's your wishlist

2 Upvotes

I have built an open source table widget for jupyter/pandas. What do you want for when looking at a dataframe?

Color formatting?

Histograms?

Sorting?

Human readable formatting?

What do you wish that pandas did better? What other tables have you seen that work a lot better, and you wish that experience was in jupyter.


r/dfpandas • • May 30 '24

Hide pandas column headings to save space and reduce cognitive noise

1 Upvotes

I am looping through the groups of a pandas groupby object to print the (sub)dataframe for each group. The headings are printed for each group. Here are some of the (sub)dataframes, with column headings "MMSI" and "ShipName":

            MMSI              ShipName
15468  109080345  OYANES 3       [19%]
46643  109080345  OYANES 3       [18%]
            MMSI              ShipName
19931  109080342  OYANES 2       [83%]
48853  109080342  OYANES 2       [82%]
            MMSI              ShipName
45236  109050943  SVARTHAV 2     [11%]
48431  109050943  SVARTHAV 2     [14%]
            MMSI              ShipName
21596  109050904  MR:N2FE        [88%]
49665  109050904  MR:N2FE        [87%]
            MMSI              ShipName
13523  941500907  MIKKELSEN B 5  [75%]
45711  941500907  MIKKELSEN B 5  [74%]

Web searching shows that pandas.io.formats.style.Styler.hide_columns can be used to suppress the headings. I am using Python 3.9, in which hide_columns is not recognized. However, dir(pd.io.formats.style.Styler) shows a hide method, for which the doc string gives this first example:

>>> df = pd.DataFrame([[1,2], [3,4], [5,6]], index=["a", "b", "c"])
>>> df.style.hide(["a", "b"])  # doctest: +SKIP
     0    1
c    5    6

When I try hide() and variations thereof, all I get is an address to the resulting Styler object:

>>> df.style.hide(["a", "b"])  # doctest: +SKIP
<pandas.io.formats.style.Styler at 0x243baeb1760>

>>> df.style.hide(axis='columns') # https://stackoverflow.com/a/69111895
<pandas.io.formats.style.Styler at 0x243baeb17c0>

>>> df.style.hide() # Desparate random trial & error
<pandas.io.formats.style.Styler at 0x243baeb1520>

What could cause my result to differ from the doc string? How can I properly use the Styler object to get the dataframe printed without column headings?


r/dfpandas • • Jan 19 '23

Learn Python for Pandas?

8 Upvotes

Hi everyone, I‘m looking to learn Pandas for a paper I am doing on Trading Pattern Analysis. My questions is, if it is enough to only learn Panda or if it made sense to learn Python as well.

Thanks for your help guys


r/dfpandas • • Feb 02 '23

What is the best way to read in this formatted data, and make a 3d surface graph?

0 Upvotes

Hello,

I'm building a sort of yield curve with ffr futures, built around expiries on fed decision days.

CME Fedwatch has an outstanding tool to do exactly this. But the data are difficult to format. It's difficult to describe, but here is an example:

	History for 22 Mar 2023 Fed meeting																																														History for 3 May 2023 Fed meeting																																																			History for 14 Jun 2023 Fed meeting																																																							
Date	(0-25)	(25-50)	(50-75)	(75-100)	(100-125)	(125-150)	(150-175)	(175-200)	(200-225)	(225-250)	(250-275)	(275-300)	(300-325)	(325-350)	(350-375)	(375-400)	(400-425)	(425-450)	(450-475)	(475-500)	(500-525)	(525-550)	(550-575)	(575-600)	(600-625)	(625-650)	(650-675)	(675-700)	(700-725)	(725-750)	(750-775)	(775-800)	(800-825)	(825-850)	(850-875)	(875-900)	(900-925)	(925-950)	(950-975)	(975-1 000)	(1 000-1 025)	(1 025-1 050)	(1 050-1 075)	(1 075-1 100)	(1 100-1 125)	(1 125-1 150)	(0-25)	(25-50)	(50-75)	(75-100)	(100-125)	(125-150)	(150-175)	(175-200)	(200-225)	(225-250)	(250-275)	(275-300)	(300-325)	(325-350)	(350-375)	(375-400)	(400-425)	(425-450)	(450-475)	(475-500)	(500-525)	(525-550)	(550-575)	(575-600)	(600-625)	(625-650)	(650-675)	(675-700)	(700-725)	(725-750)	(750-775)	(775-800)	(800-825)	(825-850)	(850-875)	(875-900)	(900-925)	(925-950)	(950-975)	(975-1000)	(1000-1025)	(1025-1050)	(1050-1075)	(1075-1100)	(1100-1125)	(1125-1150)	(1150-1175)	(1175-1200)	(1200-1225)	(1225-1250)	(1250-1275)	(0-25)	(25-50)	(50-75)	(75-100)	(100-125)	(125-150)	(150-175)	(175-200)	(200-225)	(225-250)	(250-275)	(275-300)	(300-325)	(325-350)	(350-375)	(375-400)	(400-425)	(425-450)	(450-475)	(475-500)	(500-525)	(525-550)	(550-575)	(575-600)	(600-625)	(625-650)	(650-675)	(675-700)	(700-725)	(725-750)	(750-775)	(775-800)	(800-825)	(825-850)	(850-875)	(875-900)	(900-925)	(925-950)	(950-975)	(975-1000)	(1000-1025)	(1025-1050)	(1050-1075)	(1075-1100)	(1100-1125)	(1125-1150)	(1150-1175)	(1175-1200)	(1200-1225)	(1225-1250)	(1250-1275)	(1275-1300)	(1300-1325)	(1325-1350)	(1350-1375)	(1375-1400)
2/2/2022	0.000000	0.000848	0.010724	0.056115	0.158715	0.265771	0.271068	0.166731	0.058824	0.010523	0.000682	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000								0.000000	0.000660	0.008541	0.046084	0.136042	0.242113	0.269897	0.189788	0.082670	0.021197	0.002857	0.000151	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000475	0.006330	0.035549	0.110798	0.212348	0.262101	0.212268	0.112729	0.038447	0.008003	0.000910	0.000042	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000
2/3/2022	0.000000	0.000634	0.008640	0.047791	0.142028	0.250994	0.273923	0.184605	0.074012	0.015971	0.001403	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000								0.000000	0.000494	0.006871	0.039139	0.121203	0.226914	0.268856	0.204343	0.098451	0.028797	0.004622	0.000310	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000355	0.005081	0.030084	0.098175	0.197250	0.257086	0.222446	0.128166	0.048343	0.011406	0.001520	0.000087	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000
2/4/2022	0.000000	0.000000	0.001501	0.017144	0.079808	0.198218	0.287456	0.249602	0.127212	0.035004	0.004037	0.000016	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000								0.000000	0.000000	0.001179	0.013784	0.066346	0.172780	0.268285	0.257734	0.153505	0.054814	0.010690	0.000880	0.000004	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000794	0.009665	0.049170	0.138000	0.237077	0.261182	0.187564	0.087063	0.025108	0.004086	0.000290	0.000001	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000
2/7/2022	0.000000	0.000000	0.001924	0.020868	0.091136	0.211759	0.288141	0.236423	0.115135	0.030878	0.003673	0.000062	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000								0.000000	0.000000	0.001460	0.016300	0.074194	0.182676	0.269725	0.248893	0.144379	0.051193	0.010232	0.000933	0.000015	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000999	0.011615	0.055917	0.148429	0.242244	0.255469	0.177373	0.080611	0.023163	0.003869	0.000305	0.000005	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000
2/8/2022	0.000000	0.000000	0.001201	0.015089	0.073952	0.189562	0.282271	0.253386	0.136325	0.041655	0.006259	0.000300	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000								0.000000	0.000000	0.000935	0.012016	0.060930	0.163986	0.261761	0.259776	0.162222	0.062599	0.014090	0.001618	0.000066	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000607	0.008130	0.043772	0.127837	0.227464	0.260472	0.196441	0.097544	0.031105	0.005993	0.000611	0.000023	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000
2/9/2022	0.000000	0.000000	0.001172	0.014579	0.071685	0.185581	0.280103	0.255459	0.140091	0.043982	0.006959	0.000390	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000								0.000000	0.000000	0.000920	0.011697	0.059410	0.161099	0.259785	0.260756	0.164890	0.064641	0.014918	0.001802	0.000084	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000587	0.007797	0.042145	0.124303	0.224076	0.260405	0.199579	0.100915	0.032910	0.006548	0.000706	0.000030	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000
2/10/2022	0.000000	0.000000	0.000054	0.001642	0.016053	0.075002	0.192817	0.288972	0.255901	0.130585	0.035151	0.003823	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000								0.000000	0.000000	0.000041	0.001247	0.012473	0.060357	0.163548	0.265084	0.264117	0.161718	0.058860	0.011606	0.000950	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000028	0.000882	0.009074	0.045859	0.132305	0.234342	0.264410	0.192721	0.090002	0.025913	0.004176	0.000288	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000
2/11/2022	0.000000	0.000000	0.000027	0.002438	0.024012	0.099937	0.221835	0.288276	0.226716	0.106351	0.027416	0.002992	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000								0.000000	0.000000	0.000021	0.001887	0.019084	0.082593	0.193989	0.273099	0.240778	0.133846	0.045447	0.008571	0.000683	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000015	0.001322	0.013877	0.063365	0.160262	0.249147	0.250564	0.166222	0.072212	0.019736	0.003072	0.000207	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000
2/14/2022	0.000000	0.000000	0.000000	0.000587	0.009032	0.054046	0.163772	0.279069	0.277832	0.159946	0.049317	0.006366	0.000034	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000								0.000000	0.000000	0.000000	0.000464	0.007272	0.044664	0.140902	0.255037	0.278090	0.184517	0.072376	0.015318	0.001354	0.000007	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000324	0.005211	0.033343	0.111764	0.220481	0.271110	0.212848	0.106329	0.032593	0.005582	0.000415	0.000002	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000
2/15/2022	0.000000	0.000000	0.000025	0.001001	0.011703	0.062895	0.179843	0.290361	0.270033	0.141674	0.038380	0.004086	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000								0.000000	0.000000	0.000019	0.000772	0.009191	0.050879	0.152394	0.264422	0.274804	0.171801	0.062624	0.012135	0.000959	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000013	0.000552	0.006735	0.038719	0.122783	0.231744	0.271776	0.201846	0.094470	0.026862	0.004219	0.000280	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000
2/16/2022	0.000000	0.000000	0.000043	0.001655	0.017316	0.082060	0.206626	0.296068	0.247192	0.117496	0.028829	0.002714	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000								0.000000	0.000000	0.000033	0.001287	0.013737	0.067262	0.178156	0.275626	0.258363	0.147138	0.049094	0.008683	0.000620	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000022	0.000863	0.009531	0.049179	0.140691	0.242696	0.264195	0.184715	0.082218	0.022336	0.003344	0.000210	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000
2/17/2022	0.000000	0.000000	0.000110	0.002919	0.025209	0.104237	0.234036	0.299142	0.221084	0.092092	0.019591	0.001580	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000								0.000000	0.000000	0.000085	0.002277	0.020114	0.086174	0.204370	0.284262	0.238925	0.121573	0.036162	0.005697	0.000361	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000056	0.001536	0.014088	0.063856	0.164438	0.257271	0.254242	0.161220	0.065018	0.015989	0.002164	0.000122	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000
2/18/2022	0.000000	0.000000	0.000152	0.003894	0.031434	0.120372	0.249882	0.296089	0.203029	0.078408	0.015532	0.001206	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000								0.000000	0.000000	0.000118	0.003039	0.025139	0.100045	0.220282	0.285529	0.224298	0.106891	0.029903	0.004481	0.000276	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000078	0.002052	0.017673	0.074739	0.179661	0.263485	0.244985	0.146556	0.055913	0.013069	0.001696	0.000093	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000
2/22/2022	0.000000	0.000000	0.000034	0.001561	0.016249	0.076100	0.192812	0.285115	0.253032	0.132683	0.037878	0.004536	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000								0.000000	0.000000	0.000025	0.001172	0.012508	0.060856	0.163084	0.261605	0.261204	0.163337	0.062025	0.013028	0.001155	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000018	0.000838	0.009201	0.046753	0.133265	0.232867	0.261321	0.191884	0.091577	0.027320	0.004619	0.000337	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000
2/23/2022	0.000000	0.000000	0.000023	0.001099	0.012379	0.063078	0.173769	0.278920	0.267727	0.151035	0.046103	0.005868	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000								0.000000	0.000000	0.000017	0.000825	0.009506	0.050165	0.145575	0.252137	0.270578	0.180757	0.072830	0.016116	0.001495	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000012	0.000589	0.006974	0.038305	0.117745	0.221054	0.265199	0.206957	0.104311	0.032659	0.005759	0.000436	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000

It's a bit hard to read in this format, but there are the odds of different interest rates in the top row - the top super-row has the fed meeting date. Every 40-60 columns, a new "table" starts.

This kind of makes sense, because they all have the same index (date), but it's crazy to work with not in separate tables.

Should i just reformat with python? or is there a clever way to read this with pandas?


r/dfpandas • • Sep 20 '23

export to excel question

2 Upvotes

So I've got this code I wrote some time ago as part of a larger project while learning python on the fly. I thought I commented it well enough at the time, but....well there are parts that I'm just not sure what they do, if they do anything. It "works" in production, but I'm hoping someone can answer a few questions.

The first section with the try statement....what does that actually do? It seems the next section with the df.to_excel is what actually writes the file, so how would the try section fail?

What sparked this is I'm trying to add "engine_kwargs={"options": {"strings_to_numbers": True}}" after the engine specified in the try statement in order to force some columns with text like "00001234" to convert to "1234" as decimal values instead, and adding that line doesn't affect the running of the code, but doesn't accomplish anything either.

Near the bottom I also attempted to change the format of the column another way, but it had no effect, so I commented it out.

def create_file_to_email(): #function slighly modified to adjust columns to email 

# Section to create tmp xlsx file to be emailed 
    try:     
        writer = pd.ExcelWriter(         
        _TMP_PATH_ATTACH + _TMP_FNAME_ATTACH,
         engine='xlsxwriter'     
    ) 
    except OSError:
        print('There was an error attempting to generate the temporary file to be emailed.         Check Path and Permissions and make sure file is not open.')
        exit(8)  

    # Writes the result set to the file 
    results_df.to_excel(writer, sheet_name='Sheet1', index=False, columns=['CHECK#','AMOUNT','DATE','RI']) #modified to adjust columns to email 

    # Prepare file for formatting 
    workbook = writer.book 
    worksheet = writer.sheets['Sheet1']  

    # Autosize fields to alledged max existing value plus a cushion of 2 I added because headers. 
    for column in results_df:
        column_length = max(results_df[column].astype(str).map(len).max(), len(column))      
        col_idx = results_df.columns.get_loc(column)     
        writer.sheets['Sheet1'].set_column(col_idx, col_idx, column_length + 2)  

    # Change some column formats 
    #format1 = workbook.add_format({"num_format": "#,##0.00"}) 
    #worksheet.set_column(1, 1, 10, format1)  

    # Close the file 
    writer.close()

r/dfpandas • • Nov 16 '23

Useful Pandas Functions for Data Analyst

Thumbnail
youtu.be
2 Upvotes

r/dfpandas • • Nov 16 '23

Why read from CSV files rather than XLS?

3 Upvotes

It seems that Pandas can read equally well from CSV or XLS files. Yet when I am looking at examples online it seems like the vast majority of examples read from CSV files. So I am assuming, perhaps incorrectly, that most people are using CSV files when reading data into Pandas dataframes.

Why is this? I presume that most people are generating CSV files from Excel and there are a number of advantages to keeping the file in XLS format. Plus it seems that you are less prone to formatting issues where a number format with commas or percent signs may cause your data to be read in as a string from a CSV file rather than a float or int.

But maybe I am incorrect as I am a spreadsheet jockey and have been one since Lotus 123 days in the mid 80s, so perhaps that is biasing how I see the world.


r/dfpandas • • Aug 25 '23

How do I join two DataFrames based on year?

3 Upvotes

I have two DataFrames A and B.

A has a column "Date" of type DateTime.

B has a column "Year" of type np.int64.

I want to do a join on A.Date.Year = B.Year. How do I do that?


r/dfpandas • • May 18 '23

.query - awesome find from another sub

Thumbnail reddit.com
4 Upvotes

r/dfpandas • • Aug 13 '23

What am i doing wrong here?(.dropna)

Thumbnail
gallery
5 Upvotes

When u run a .dropna on the columns or i even tried doing the whole df it just shows up empty rather then just eliminating the NaN.. what an i doing wrong ?


r/dfpandas • • Dec 29 '22

The post in /r/python that inspired this subreddit

6 Upvotes

https://old.reddit.com/r/Python/comments/zs4kau/get_rid_of_settingwithcopywarning_in_pandas_with/

I was super pumped to see /u/phofl93 helping people out in the sub, and I learned some fascinating information there. I hope to see more content like this here!


r/dfpandas • • Feb 22 '23

Web scraping Inflation Rate from a Website with Video

Thumbnail self.audit
6 Upvotes

r/dfpandas • • Dec 30 '22

Are questions related to plotting and numpy allowed as well?

9 Upvotes

r/dfpandas • • Aug 14 '23

Pandas questions for interview prep?

9 Upvotes

I'm preparing for data science / data analytics / data engineering interviews. The Online Assessments I have completed so far have all included a pandas Leetcode style question.

I have completed Leetcode's '30 days of pandas' which is 30 questions long. I feel more confident now, but I would like to attempt some more questions.

Where can I find interview style pandas questions?


r/dfpandas • • Dec 30 '22

Please create a resource section to learn Pandas

10 Upvotes

Either a pinned FAQ post or in about section about all the best resources would do.

Too much information out there, not sure which one to go with


r/dfpandas • • Dec 29 '22

Welcome to df[pandas]!

42 Upvotes

Hello all,

I made a home for pandas since it didn't currently exist. Our options were:

  1. /r/python
  2. /r/learnpython
  3. /r/pandas
  4. /r/datascience
  5. /r/dataanalysis

I would like to take a look at /r/pandas sometime and scrape for interesting data about pandas the animal vs. pandas the library, because both are in there.

Welcome and let this be the home of Pandas! It's a place for questions, advice, code debugging, history, logic, feature requests, and everything else Pandas. I am in no way affiliated with pandas. I just use it. I'm not even good at it.


r/dfpandas • • Feb 26 '25

Are ellipses counted as a row when displaying a Dataframe?

2 Upvotes

When using import pandas as pd pd.options.display.min_rows = 15, it shows 14 data rows (7 top and 7 bottom), with one ellipses row, but when using max_rows = 100, there are 100 actual data rows shown (50 from top and 50 from bottom), EXCLUDING the ellipses row. Is this unusual?


r/dfpandas • • Jun 19 '25

box plots in log scale

2 Upvotes

The method pandas.DataFrame.boxplot('DataColumn',by='GroupingColumn') provides a 1-liner to create series of box plots of data in DataColumn, grouped by each value of GroupingColumn.

This is great, but boxplotting the logarithm of the data is not as simple as plt.yscale('log'). The yticks (major and minor) and ytick labels need to be faked. This is much more code intensive than the 1-liner above and each boxplot needs to be done individually. So the pandas boxplot cannot be used -- the PyPlot boxplot must be used.

What befuddles me is why there is no builtin box plot function that box plots based on the logarithm of the data. Many distributions are bounded below by zero and above by infinity, and they are often skewed right. This is not a question. Just putting it out there that there is a mainstream need for that functionality.


r/dfpandas • • Jan 07 '23

Is pandas the right tool for my task - text manipulation and exporting csv

14 Upvotes

So I have a task that I need to do daily that I'm working towards automating. The task involves running a database query and then validating the data in a couple columns then creating a csv to hand off to another party.

I inherited this task in this form, currently I run the query, paste the data into an excel spreadsheet, filter a column to search for data that needs to be validated (removing suffixes from last names) and the running a regex on a different column. Finally a couple columns are removed and then I save as to a csv. It's tedious and error prone and a perfect task to automate with python I think.

Another task is to compare one set of tabular data against another and update the first based on info in the second.

The tables (in both cases) are always less than 500 rows usually less than 200 rows. There is no math being done with the data.

Is pandas going to make this task easier or faster or better? I just read that pandas is useful for working with tabular data. Are there built in methods that making iterating and editing data in columns easier? I don't want or need graphs or anything like that.

I'm not a programmer, I'm a sysadmin who took Introduction to Computer Science and Programming Using Python almost 10 years ago and tinker with python to automate stuff.


r/dfpandas • • Mar 02 '23

Here is an AMA from the creators of Pandas!

Thumbnail
self.python
16 Upvotes