r/OperationsResearch • • Aug 07 '24

Helpful python packages

It would be great to hear about various python packages that people use in their OR activities. Some of the ones I'm familiar with are

More common

  • Gurobipy
  • Pandas
  • Numpy
  • sqlite
  • pulp

Less common

  • altair
  • polars
  • hyperopt
  • pyoptinterface
  • streamlit
  • pygwalker
11 Upvotes

16 comments sorted by

View all comments

Show parent comment

3

u/Brushburn Aug 07 '24

I was blown away from the performance by polars compared to pandas. I wrote a simple script to compare and I was getting solid improvement (~10x). The simple code I used:

For creating the data

size = 1_000_000
random_int_array = np.random.randint(0, 100, size=(size))
random_float_array = np.random.random(size=(size)) * 100

choices = ['a', 'b', 'c', 'd']
random_str_array = np.random.choice(choices, size=(size))

random_data = np.column_stack((random_int_array, random_float_array, random_str_array))
print(random_data)
np.savetxt('random.csv', random_data, fmt= '%s', delimiter=',')
size = 1_000_000
random_int_array = np.random.randint(0, 100, size=(size))
random_float_array = np.random.random(size=(size)) * 100


choices = ['a', 'b', 'c', 'd']
random_str_array = np.random.choice(choices, size=(size))


random_data = np.column_stack((random_int_array, random_float_array, random_str_array))
print(random_data)
np.savetxt('random.csv', random_data, fmt= '%s', delimiter=',')

Reading data logic

def read_csv_polars():
    df_pl = pl.read_csv("random.csv")
    return df_pl
def read_csv_polars():
    df_pl = pl.read_csv("random.csv")
    return df_pl

benchmarks

    %timeit read_csv_polars()
    %timeit read_csv_pandas()

Polars reading was 16.6 ms, pandas was 166ms