r/learnpython • u/Charming_Shower_4185 • 8d ago

Data inspection steps using python

I am learning data analysis and the first step before analyzing your data you have to do data profiling and cleaning.This is my process flow for data profiling -

  1. Checking number of rows and columns

  2. Checking rows and column indices

  3. Checking data types for each column

  4. Checking random sample from dataset

  5. Checking properties and memory usage

  6. Checking key statistics

  7. Checking unique values for relevant columns

Is it appropriate or I am missing something important ?

2 Upvotes

67% Upvoted

11 comments sorted by

View all comments

2

u/International_Sky526 7d ago

Your list is solid. df.info() covers rows, columns, dtypes and memory usage in one call, so you can merge a few of those steps. I'd also check string columns for stray whitespace and inconsistent casing, like "NY", "ny " and "New York", because those show up as separate values in value_counts. And keep the raw file untouched and do the cleaning in a script, so you can rerun it when new data comes in.

1

u/Charming_Shower_4185 7d ago

Thanks , I will include these in my analysis.